Tate Britain project uses AI to pair contemporary photos with paintings

Seated against a deep red backdrop, gazing intently at hand-held mirrors, two eunuchs in sparkling saris inspect their appearance before Raksha Bandhan celebrations in the red light district of Mumbai.

The photograph from the Reuters news agency is an arresting contemporary scene, but a new Tate Britain project is aiming to inspire deeper reflections with images from its own collection of paintings.

Launching on Friday, Recognition is the winner of 2016’s IK prize – an annual award, this year supported by Microsoft, for a project that embraces digital technology to explore and showcase Tate’s collection of British art.

This year, the challenge was to do it with artificial intelligence.

The team behind the winning project, from the Italy-based communication research centre Fabrica, say their inspiration came from an intriguing conundrum: how can you apply rational thinking to a subject like art?

Their answer is provocative. Recognition matches stunning photographs from the 24/7 news cycle with centuries-old artworks, and presents them online.

“The team have created and trained a ‘brain’ to a point where it is simulating certain human attributes and unleashed it online – and it is creating a gallery,” said Tony Guillan, the producer of the IK prize at Tate Britain.

After scanning through about 30,000 digitised artworks from the gallery, the system pairs the evocative photograph of the two eunuchs with a mellow scene from the brush strokes of Sir Peter Lely, the principal painter to Charles II. Two women, straight-backed and smiling slightly are seated, one holding a stringed instrument, the other resting her hand on the elaborate folds of her satin dress.

It is a sumptuous 17th-century work, rich in details. But presented next to the contemporary photograph, the comparison can provoke new artistic questions - for example about gender identity and the notion of glamour.

“Without knowing it, [the system] has created subjective meaning,” said Guillan. “By asking the question ‘how do computers work and think?’ you ask the exact same question of humans.”

Guillan said he hoped the project would encourage discussion about how we depict scenes and individuals. “News always presents itself as this mimetic, glass window on to the world, but of course photojournalism is an art form often; it is a mode of communication – and so is art and painting,” he said.

With £15,000 in prize money and £90,000 to produce their vision, the four-strong team at Fabrica have spent months developing the project, which harnesses a burgeoning form of artificial intelligence known as machine learning. It will be available in full online and there will be a small exhibition at Tate Britain.

“Computers are very good at mathematical operations,” said Andrea Vedaldi, an associate professor of engineering science at the University of Oxford, who was not involved in the project. “When things start to be very difficult is when you don’t really know how to express, in mathematical terms, what it is you want to do. For example, if you want to recognise a dog in an image ... it is not very easy to come up with the equations of a dog.”

Machine learning takes a different approach: show a system enough images of different canines in different poses, the theory goes, and the system will figure out what a dog is. “This process of going from specifics to the abstract concept, this is what is really challenging,” said Vedaldi. While progress in machine learning has been dramatic in recent years, he said, creating systems that truly “understand” images was still a work in progress.

It is a challenge that scientists and engineers such as Vedaldi continue to grapple with, developing and exploring powerful systems, including new tools based on a branch of machine learning called deep learning.

Recognition takes four different approaches to match images, based on technology developed by Microsoft and a team of AI specialists at the France-based company Jolibrain. “It can look for objects, like cups and saucers, it can look for faces, it can look for composition within an image – reading lines and colours – and it can look at the context that is attached to an image, so metadata, titles and things like that,” said Isaac Vallentin of Fabrica.

Now trained, the system will spend three months continuously analysing new photographs from Reuters, comparing each to thousands of digitised paintings, sculptures and other works. Where one or more matches can be found, the best – as selected by the technology – is entered into a searchable online gallery.

Visitors to the website will also be able to explore details of how the system made each match, including the strength of the resemblance across each of the four approaches, together with particular features it has recognised, such as the age and gender of a face. “We are trying to really honestly represent how the software itself is coming to its conclusions,” said Vallentin, adding that the system would also be able to create a sentence to explain, to a limited degree, each match.

The small, three-section display at Tate Britain will offer further insights, with visitors able to compare the matches they would make to those generated by the machine. The results will then be scrutinised. “We just want to take these two datasets in the end and find the connections, the similarities,” said Vallentin. “It is really an experiment for us.”

The team also hope the project will showcase the positive side of artificial intelligence – a technology that experts including the physicist Stephen Hawking have raised safety concerns about.

“Technology is an empowering tool,” said Vallentin. “You can create really meaningful things and really helpful things, and beautiful things with technology.”

• The IK prize 2016 takes place online and at Tate Britain from 2 September – 27 November.