Scenes from a Salon on Artificial Intelligence

When you think of artificial intelligence and innovative technology, do you think of museums?

The potential for AI has increasingly become a focus across the museum sector, from experiments by technologists and artists to its use improving visitor services and operations. The Met’s Open Access collection, containing over 406,000 images of artworks from around the world, has laid the foundation for important advancements in artificial intelligence. We first published a mass of images and structured data, free to use and machine-accessible via our API in 2018. Over the last few years, we’ve been working with researchers, scientists, and contemporary artists to consider and advance how AI impacts art and creativity. And in what seems like a different time for us all, last February, we convened a big think about this topic, inviting some of our prior collaborators to speak alongside other experts in machine learning and AI.

In the spirit of fostering experimentation and community, we curated the evening as a digital salon. Our aim was to have experts across disciplines intermingling and discussing how new technology can help make meaningful connections between The Met’s online collection and people, as well as provide new pathways for technologists, researchers, and creators. Our panelists included Stefano Corazza from Adobe, Eva Kozanecka from Google AI, Serge Belongie from Cornell Tech, and contemporary artist Matthew Ritchie. Each of our panelists have used machine learning in their work, and we asked them to consider AI’s impact on art—whether as a tool, a new medium, or as a form of discovery within an art-making process.

The interest in this panel was infectious, and the number of guests at our community session was more than double the number of guests we’d originally invited. On salon day, the room teemed with academics (from Cornell Tech, California Institute of Technology, University of Edinburgh, New York University, Columbia University, and Ithaca University), digital experts from museums (The Cooper Hewitt Smithsonian Design Museum, The Jewish Museum, The Whitney Museum of American Art), and other technologists (from companies such as Microsoft, Google, and Instagram), to name only a few.

From left: Stefano Corazza, Eva Kozanecka, Serge Belongie, Matthew Ritchie, and moderator Sofie Andersen

As the discussion unfolded, each panelist shared their “a-ha” moments regarding AI; some were positive, but others have them pause. Corazza realized that machine learning could democratize art—anyone with access to machine-learning software could explore the complex web of art history and even create new works of art, all without an advanced degree. On the other hand, Belongie realized that machine learning would eliminate certain face-to-face research interactions that are very human in nature, and which often lead to new discoveries. The panel reflected on that absence.

Our discussion raised as many questions as it answered. Each panelist acknowledged that artificial intelligence is still in its nascent stages. Ritchie and Belongie both talked about being able to see the technology in the art-making process, similar to seeing technology’s hand in the advent of daguerreotypes in the nineteenth century or the first uses of handheld video cameras. Technology often ushers in unforeseen forms of art, and artificial intelligence is no different. Might AI enable artists to more effectively create what their mind’s eye envisions? Or does the tool get in the way and impede creativity?

We’re pleased to share an edited excerpt from our discussions:

The Met’s First Forays in Artificial Intelligence

The Met recognizes this new technology’s potential, and the Museum has eagerly experimented with AI to explore its impact on art.

The Museum’s digital images and catalogue data form the foundations for our work in machine learning. Catalogue data contains important information about artworks, such as the year they were made, the materials they were made from, and the artists' names. In early 2017, we made this information, along with images for 375,000 artworks, available to download online through our Open Access initiative. In late 2018, we also launched an API, which allows researchers to parse this massive archive. Inherent in this initiative was a bit of the unknown and a firm belief that expanding access would spark new connections to art.

The unknown quickly began to resolve when a research team at Microsoft reached out in late 2018 to explore our online collection using their latest artificial intelligence package, Azure. We wanted to know what AI could do using our images and data, so we agreed to explore this together in a hackathon. We also invited MIT researchers and Ritchie, MIT's artist-in-residence, to join Microsoft and our team of curators and digital experts. We developed seven prototypes over two days, ranging from an AI that envisions new art objects, to a voice-controlled AI that tells stories about artworks, to yet another AI that suggests artworks based on news and the day's weather.

Since those first forays into artificial intelligence, we’ve continued to explore how machine learning and AI can be used in connection with The Met’s Open Access collection. We have continued to work with Microsoft in a search prototype that highlighted new connections between art objects. We also participated in a data-science challenge, with Cornell Tech and Kaggle, featuring how AI could use fine-grain attributes to predict an object’s culture and era. And we also collaborated with Wikipedia teams to begin building AI models that will help us accurately surface additional keywords and tags so that our art works are even more discoverable. We are also indebted to many collaborators across the museum sector for continuing to push forward the conversation, including The Cooper Hewitt’s Interaction Lab and The Cleveland Museum of Art.

There are so many unknowns and new discoveries to come. Our first digital salon set the stage for more in the future, to further the Met’s mission to connect people with creativity, knowledge, and ideas. We believe this interdisciplinary approach across art, media and tech is necessary to build on these conversations, challenge our assumptions, and examine issues specific to art and cultural institutions that we may encounter.

Open Access Meets Wikimedia at The Met

Machine learning challenge uses Met Open Access collection