Exploring Art with Open Access and AI: What's Next?

With nearly half a million objects online, it can be challenging for users to realize the full scope of The Met’s collection. How can technology help users make connections and experience the breadth and depth of The Met's Open Access artworks?

To this end, the Met recently conducted a successful hackathon with Microsoft and MIT. Participants created five prototypes that used artificial intelligence (AI) to explore new connections to art. Since then, we've continued to delve into artificial intelligence and machine learning with Microsoft. In particular, we collaborated with the Applied AI team at Microsoft on a proof-of-concept project called Art Explorer. This tool uses Azure Search's cognitive search capability to enrich a user's experience of The Met's Open Access artworks by uncovering and providing new access points into our collection.

An artificial intelligence feature in Art Explorer produces visually similar works of art from The Met Open Access collection. Above: Vincent van Gogh. Olive Trees, 1889. Below, left to right: Unknown Artist. Demons Fighting over an Animal Limb, late 17th century. India. Robert Frederick Blum. View from the Artist's Window, Grove Street, ca. 1900. Vincent van Gogh. Cypresses, 1889 

Enhanced Cognitive Discovery

The Art Explorer includes an AI feature that presents the user with visually similar works of art. For example, a user viewing Van Gogh's Olive Trees will also see other works depicting trees with similar colors, such as Van Gogh's Cypresses and Robert Frederick Blum's View from the Artist's Window, Grove Street. But they will also encounter a seventeenth-century Indian painting, Demons Fighting Over an Animal Limb! We would not normally associate Van Gogh's work with either demons or India, but looking at both images side by side, we can see similarities in the color and shape of the trees depicted in both works. This is the power of AI—it can detect patterns not readily noticed by humans.

Improved Tagging

The Art Explorer also features another special AI skill: object detection. Here, keyword tags provide the user with information about the subject matter depicted for each object. With the help of a wonderful human work force, The Met recently added keyword tags to over 275,000 objects in the online Open Access collection. The "depicted subject matter" tagging function gives users a new access point into the collection, allowing them to see how a subject of interest, such as dogs or mirrors, may have been depicted across time, geography, and different types of objects. But there are still over 20,000 objects that need to be tagged. As we continue to add more digitized objects to the collection, we need to weigh the cost, time, effort, and accuracy of tagging our objects. While AI tag prediction is not always perfect, it promises to be a more efficient and less costly way to move forward. It may also be possible to train AI algorithms with our existing tags to improve computer-vision models and thereby produce more accurate results.

Data Enrichment

The Art Explorer proof of concept also has other ways to enrich existing content using AI cognitive skills. For works in The Met collection by known artists, the tool pulls in artist bios from other online sources, primarily Wikipedia. Ideally, we would like to provide bios written by our own staff, but given the large number of artists in the collection (over 50,000), this is not feasible. With Art Explorer's cognitive search, artist bios can be automatically linked from existing content.

Visual Connections

Finally, the network graph feature in Art Explorer allows users to see visual connections between works of art. Users can view works with similar subject matter, similar artists, similar time periods, or works from a similar medium. This interactive graph invites users to delve deeper into the collection and to see connections they may never have discovered on their own.

Screenshot of the network graph feature in Art Explorer, which suggest artworks with similar characteristics

Although we have not yet implemented these cognitive search capabilities on our website, it is exciting to see the possibilities new AI technology can offer in making our collection one of the most accessible and discoverable on the internet. Our Open Access program was launched just two years ago, but seeing all the creative uses of our data in such a short period has been inspiring. From creative data visualizations of our collection, to experiments with generative adversarial networks (GANS) and our most recent collaboration with Microsoft, it's been rewarding to see a new audience of developers and technologists be so inspired by art and so eager to create new ways of seeing our collection.

The Met, Microsoft, and MIT held a two-day hackathon session to explore how artificial intelligence could connect people to art. Learn about this collaboration and the prototypes it generated.