Tate Britain project uses AI to pair contemporary photos with paintings

IK prize-winning system matches images from the 24/7 news cycle with centuries-old artworks and presents them online

Seated against a deep red backdrop, gazing intently at hand-held mirrors, two eunuchs in sparkling saris inspect their appearance before Raksha Bandhan celebrations in the red light district of Mumbai.

The photograph from the Reuters news agency is an arresting contemporary scene, but a new Tate Britain project is aiming to inspire deeper reflections with images from its own collection of paintings.

Detail from Sir Peter Lely’s Two Ladies of the Lake Family (1660)
Detail from Sir Peter Lely’s Two Ladies of the Lake Family (1660), paired with the Mumbai scene by the Recognition software. Photograph: Tate

Launching on Friday, Recognition is the winner of 2016’s IK prize – an annual award, this year supported by Microsoft, for a project that embraces digital technology to explore and showcase Tate’s collection of British art.

This year, the challenge was to do it with artificial intelligence.

The team behind the winning project, from the Italy-based communication research centre Fabrica, say their inspiration came from an intriguing conundrum: how can you apply rational thinking to a subject like art?

Their answer is provocative. Recognition matches stunning photographs from the 24/7 news cycle with centuries-old artworks, and presents them online.

“The team have created and trained a ‘brain’ to a point where it is simulating certain human attributes and unleashed it online – and it is creating a gallery,” said Tony Guillan, the producer of the IK prize at Tate Britain.

After scanning through about 30,000 digitised artworks from the gallery, the system pairs the evocative photograph of the two eunuchs with a mellow scene from the brush strokes of Sir Peter Lely, the principal painter to Charles II. Two women, straight-backed and smiling slightly are seated, one holding a stringed instrument, the other resting her hand on the elaborate folds of her satin dress.

It is a sumptuous 17th-century work, rich in details. But presented next to the contemporary photograph, the comparison can provoke new artistic questions - for example about gender identity and the notion of glamour.

“Without knowing it, [the system] has created subjective meaning,” said Guillan. “By asking the question ‘how do computers work and think?’ you ask the exact same question of humans.”

Guillan said he hoped the project would encourage discussion about how we depict scenes and individuals. “News always presents itself as this mimetic, glass window on to the world, but of course photojournalism is an art form often; it is a mode of communication – and so is art and painting,” he said.

With £15,000 in prize money and £90,000 to produce their vision, the four-strong team at Fabrica have spent months developing the project, which harnesses a burgeoning form of artificial intelligence known as machine learning. It will be available in full online and there will be a small exhibition at Tate Britain.

“Computers are very good at mathematical operations,” said Andrea Vedaldi, an associate professor of engineering science at the University of Oxford, who was not involved in the project. “When things start to be very difficult is when you don’t really know how to express, in mathematical terms, what it is you want to do. For example, if you want to recognise a dog in an image ... it is not very easy to come up with the equations of a dog.”

Machine learning takes a different approach: show a system enough images of different canines in different poses, the theory goes, and the system will figure out what a dog is. “This process of going from specifics to the abstract concept, this is what is really challenging,” said Vedaldi. While progress in machine learning has been dramatic in recent years, he said, creating systems that truly “understand” images was still a work in progress.

It is a challenge that scientists and engineers such as Vedaldi continue to grapple with, developing and exploring powerful systems, including new tools based on a branch of machine learning called deep learning.

Recognition takes four different approaches to match images, based on technology developed by Microsoft and a team of AI specialists at the France-based company Jolibrain. “It can look for objects, like cups and saucers, it can look for faces, it can look for composition within an image – reading lines and colours – and it can look at the context that is attached to an image, so metadata, titles and things like that,” said Isaac Vallentin of Fabrica.

Now trained, the system will spend three months continuously analysing new photographs from Reuters, comparing each to thousands of digitised paintings, sculptures and other works. Where one or more matches can be found, the best – as selected by the technology – is entered into a searchable online gallery.

Visitors to the website will also be able to explore details of how the system made each match, including the strength of the resemblance across each of the four approaches, together with particular features it has recognised, such as the age and gender of a face. “We are trying to really honestly represent how the software itself is coming to its conclusions,” said Vallentin, adding that the system would also be able to create a sentence to explain, to a limited degree, each match.

The small, three-section display at Tate Britain will offer further insights, with visitors able to compare the matches they would make to those generated by the machine. The results will then be scrutinised. “We just want to take these two datasets in the end and find the connections, the similarities,” said Vallentin. “It is really an experiment for us.”

The team also hope the project will showcase the positive side of artificial intelligence – a technology that experts including the physicist Stephen Hawking have raised safety concerns about.

“Technology is an empowering tool,” said Vallentin. “You can create really meaningful things and really helpful things, and beautiful things with technology.”

  • The IK prize 2016 takes place online and at Tate Britain from 2 September – 27 November.


Nicola Davis

The GuardianTramp

Related Content

Article image
Alan Turing: the short, brilliant life and tragic death of an enigma

Codebreaker and mathematician Alan Turing's legacy comes to life in a Science Museum exhibition

Maev Kennedy

20, Jun, 2012 @4:48 PM

Article image
‘Some people feel threatened’: face to face with Ai-Da the robot artist
Self-portraits by ultra-realistic android go on show at Design Museum in London

Mark Brown Arts correspondent

18, May, 2021 @1:09 PM

Article image
Monet's UK parliament paintings to feature in Tate Britain exhibition
Works will go on show as part of first major exhibition charting the stories of French impressionists who were ‘exiled’ in Britain

Mark Brown Arts correspondent

18, Jul, 2017 @3:45 PM

Article image
Follow that robot: the hi-tech tour guides taking over Tate Britain at night

Art lovers all over the world can log on to the Tate website and control four robot guides that were test-driven by Space Oddity astronaut Chris Hadfield

Maev Kennedy

12, Aug, 2014 @12:42 PM

Article image
Tate acquires British impressionist painting Le Passeur for £1.5m
Tate Britain puts William Stott of Oldham’s 1881 work Le Passeur (The Ferryman) on public display after buying it for the nation

Mark Brown Arts correspondent

05, May, 2017 @1:18 PM

Article image
Artist and Empire at Tate Britain: cryptic paintings of violent imperialism
Exhibition features a piece by Elizabeth Butler now seen as an indictment of 19th century wars in Afghanistan – but viewed as a depiction of hero in its time

Mark Brown

20, Jul, 2015 @4:16 PM

Article image
Dreamers and disrupters: the best art and architecture of autumn 2018
Old and new masters reveal their radical edge, Assemble unveil their Goldsmiths galleries, Fernand Léger seeks utopia, and photography focuses on the facts

Jonathan Jones, Sean O'Hagan , Adrian Searle and Oliver Wainwright

27, Aug, 2018 @5:00 AM

Article image
'I've seen more self-aware ants!' AI: More Than Human – review
Barbican, London
This crowd-pleasing show throws robots, shiny toys and interactive paintings at viewers. But there’s little sign of creativity amid the chaotic overload

Jonathan Jones

15, May, 2019 @3:14 PM

Article image
Artists prepare for BP protest at Tate Britain

Good Crude Britannia will picket Tate Britain's summer party on Monday, as the gallery celebrates 20 years of BP's sponsorship

John Vidal

24, Jun, 2010 @10:19 AM

Article image
Incoherent, creepy and deceptively gorgeous: six leading British artists making art with AI
Artificial intelligence is creating increasingly sophisticated images. But what does it mean for the artworld? Gilbert and George, Gillian Wearing, Mat Collishaw, Elizabeth Price, Polly Morgan and Lindsey Mendick found out

Jonathan Jones

01, Dec, 2022 @6:00 AM