The Met x Microsoft x MIT
In December, The Met, Microsoft, and Massachusetts Institute of Technology (MIT) collaborated in a two-day hackathon session to explore how artificial intelligence could connect people to art. Utilizing the API images, data, and a new keyword data set, our goal was to imagine and develop scalable new ways for global audiences to discover, learn, and create with one of the world's foremost art collections through artificial intelligence.
Microsoft provided its Garage space at the New England Research & Development (NERD) center, a group of engineers and its AI platform including pre-built APls such as Azure Cognitive Services, conversational Al, and Azure Machine Learning. MIT provided a select group of students and faculty led by MIT Open Learning and the Knowledge Futures Group, and The Met provided curatorial staff, digital staff, and researchers. The Met, Microsoft, and MIT teams broke out into small groups and worked for two days to create design concepts and initial prototypes.
Through this collaboration, we hope to demonstrate the growing potential of the Open Access program (and the subject keywords data-set in particular), while scaling the global reach and relevance of The Met collection. By investing in this collaboration together, The Met, Microsoft, and MIT highlight the potential of artificial intelligence and open data to empower people globally through art.
AI Prototypes
Participants at The Met x Microsoft x MIT hackathon event in December. Photo by Natasha Moustache / Listen
The Met, Microsoft, and MIT teams broke out into small groups and worked for two days to create design concepts and initial prototypes. Each team brought a new perspective on how to use artificial intelligence to connect people and art. Here are five of the AI prototypes that were developed through this cross-disciplinary collaboration.
Artwork of the Day uses Microsoft AI to find the artwork in The Met collection that will resonate with you today.
Every day, Artwork of the Day will return a new entry point into The Met collection through an artwork that is relevant in the context of world events and your current circumstances. It does this by analyzing open data sets including your location, weather, news, and historical data. This analysis is specific to you—no two people will receive the same artwork of the day—and so delivers a level of personalization and scalability that would be near impossible to replicate through human endeavor.
Artwork of the Day could work anywhere in the world and in any language. By transforming it into an API, we also envision it being integrated into alarm clocks, bathroom mirrors, and even refrigerators so as to integrate personalized doses of art into everyone’s day.
Team:
Spencer Kiser, Lead Developer, Digital Department, The Met
Noelle LaCharite, Applied AI and Cognitive Services, Microsoft
Isaac Lau, Undergraduate Student, MIT
Participants at The Met x Microsoft x MIT hackathon event in December. Photo by Natasha Moustache / Listen
Gen Studio uses Microsoft AI to allow you to visually and creatively navigate the shared features and dimensions underlying The Met's Open Access collection. Within the Gen Studio is a tapestry of experiences based on sophisticated generative adversarial networks (GANs) which allow you to explore, search, and even be immersed within the latent space underlying The Met's encyclopedic collection.
One such experience titled Generist Maps places you within a map-like model representing the inferred structure underlying existing artworks in The Met collection. You can explore an iterative sequence of dreamlike images—created by AI and choices that you make—interpolate between artworks in the collection. Artworks are recombined in new ways, synthesizing possibilities that draw on the styles, materials, and forms found in The Met collection, and surfacing unfamiliar artworks that share similar visual characteristics. Once you have found an inspiring piece you can use another tool in the studio, Generative Explorations, to find its closest visual matches within The Met collection and so discover new areas of interest. You can even explore related works and topics through an immersive visual search experience.
We envision Gen Studio becoming a far-reaching platform that develops appreciation of the immense historical depth and scale of The Met collection, and which identifies new perspectives on the visual relationships between cultures and the production of individual artworks.
Team:
Mark Hamilton, Software Engineer, Microsoft Applied AI
Chris Hoder, AI Project Manager, Microsoft Applied AI
Matthew Ritchie, Dasha Zhukova Distinguished Visiting Artist at the MIT Center for Art, Science & Technology, MIT
Sarah Schwettmann, Brain and Cognitive Sciences, Knowledge Futures Group, MIT
Kim Benzel, Curator in Charge, Department of Ancient Near Eastern Art, The Met
Julie Arslanoglu, Research Scientist, Department of Scientific Research, The Met
Casey Hong, Software Engineer, Microsoft
Dalitso Banda, Software Engineer, Microsoft
Manon Knoertzer, Program Manager, Microsoft
Karthik Rajendran, Data Scientist, Microsoft
Botong Ma, Graduate Student, Electrical Engineering and Computer Science, MIT
SJ Klein, Underlayer, Knowledge Futures Group, MIT
Luke Hewitt, Graduate Student, Brain and Cognitive Sciences, MIT
Maddie Cusimano, Graduate Student, Brain and Cognitive Sciences, MIT
Wilson Lee, Senior Software Engineer, Microsoft
Alonso Salas Infante, MIT Student Extern with Microsoft Garage
Gillian Belto, MIT Student Extern with Microsoft Garage
Darius Bopp, MIT Student Extern with Microsoft Garage
Diana Nguyen, MIT Student Extern with Microsoft Garage
Elaine Zhang, MIT Student Extern with Microsoft Garage
My Life, My Met uses AI to turn your Instagram feed into a work of art.
My Life, My Met will use Microsoft AI to analyze your posts from Instagram and substitute the images with the closest matching Open Access artworks from The Met collection. Photos of your restaurant dinner may be replaced by Ryūryūkyo Shinsai's print of Sushi (Vinegared Fish and Rice) Food, or your celebration of a recent football win may be replaced by a late fifth-century B.C. Terracotta askos in the form of a duck.
My Life, My Met enables you to bring art into the everyday interactions of your life. With over 400,000 images available through The Met's Open Access program, and over ninety-five million photos and videos shared across Instagram each day, the possibilities to explore the visual relationships between 5,000 years of human history and today's everyday life are endless!
Team:
Chris Hoder, AI Project Manager, Microsoft Applied AI
Mark Hamilton, Software Engineer Microsoft Applied AI
Karthik Rajendran, Data Scientist, Microsoft
Storyteller uses Microsoft AI to choose artworks in The Met collection that illustrate any story you might want to tell, or any conversation you might be having.
Storyteller uses voice recognition AI to follow the discussion and share artworks from The Met collection that resonate with the stories being told. The story can be personal, whimsical, or a serious oral history. After the person or group finishes the story or discussion, they can share the Storyteller thread of artworks through social media, have the AI generate a tour based on the artworks that were selected, or even compile the works into a ready-to-print book.
Storyteller can work in over sixty languages, and we envision it existing in living room, dining room, and bedroom technologies around the world, ready to enrich any story at any time. Imagine reading "Cinderella" to your child at night, and watching as it is illustrated by artworks from The Met collection….
Team:
Emile Molin, Head of Design, The Met
Griff Mann, Michel David-Weill Curator in Charge, Department of Medieval Art and The Cloisters, The Met
Kurt Behrendt, Associate Curator, Department of Asian Art, The Met
Johann Diedrick, Senior Developer, Digital Department, The Met
Chris Stone, Senior Software Engineer, Microsoft
Azra Akšamija, Associate Professor, MIT Art, Culture, and Technology Program (ACT) and Department of Architecture, Director of the Future Heritage Lab, MIT
Lara Ramez Baladi, Visiting Scholar, MIT Art, Culture, and Technology Program (ACT) and MIT Open Documentary Lab, MIT
Matt Zumwalt, Researcher and Reporter, Protocol Labs (Friend of MIT)
Participants at The Met x Microsoft x MIT hackathon event in December. Photo by Natasha Moustache / Listen
Tag, That's It! partners humans with machines to increase the accessibility of The Met collection for the millions of people interacting across the Wikimedia platforms.
Subject keyword data is integral to helping people discover and connect with works of art most relevant to them. This requires accurately identifying which artworks contain which objects of interest: fashion, games, children, chess, calligraphy and, of course, cats. Tag, That's It! is a crowdsourcing means of fine-tuning subject keyword results generated by an AI model. By adding Met keywords into Wikidata and using Microsoft AI to generate more accurate keywords, The Met collection can be enriched by the global Wiki community and contribute to the corpus of all human knowledge on the Wikimedia platforms.
We envision being able to fine-tune the AI model to provide predicted subject keywords for artworks that the model has never encountered, and thus scale globally to work with any museum collection.
Team:
Andrew Lih, Wikipedia Strategist, Wikimedia DC – Outreach and GLAM
Jennie Choi, General Manager of Collection Information, Digital Department, The Met
Nina Diamond, Managing Editor and Producer, Digital Department, The Met
Samaneh Kazemi Nafchi, Software Engineer, Microsoft
JS Tan, Software Engineer, Microsoft
Patrick Buehler, Principal Data Scientist, Microsoft
Related Articles
Learn more about The Met's Open Access initiative.
The Met's Open Access initiative is made possible through the continued support of Bloomberg Philanthropies.