New Perspectives on Storytelling, Artificial Intelligence, and Art

The author meets with other participants in the Storyteller group. From left to right: Emmanuel Awa, Kurt Behrendt, Lara Baladi, Matt Zumwalt. All photos by Natasha Moustache / Listen

Making the artworks in The Met collection meaningful for the public is one of my main goals as a curator, although it usually takes the form of exhibitions, publications, or lectures. When I go to exhibitions with family and friends, I find that viewing artworks with others leads to great discussions about how we can see the world in new and unexpected ways. So when I was invited to participate in a Met x MIT x Microsoft hackathon with the goal of leveraging artificial intelligence (AI) to interface with The Met's Open Access online collection of more than 406,000 images, I accepted, eager to see how we can creatively leverage digital and web-based technology to inspire and support new kinds of conversations and interactions with art. With this in mind, I headed off to Cambridge, MA, together with other curators and digital colleagues from The Met, to meet with engineers, researchers, students, designers, artists, and technologists from Microsoft and MIT.

Left: Hackathon participants conversing on the first day (from left to right: Samaneh Kazemi Nafchi, Nina Diamond, Julie Arslanoglu, Griff Mann, Matthew Ritchie). Right: The Storyteller group gathers around a table to brainstorm (from left to right: Emmanuel Awa, Kurt Behrendt, Lara Baladi, Matt Zumwalt, Azra Akšamija, Griff Mann)

On the first evening, we all gathered over dinner to discuss how AI might be used in innovative ways. The plan was to break up into small groups and over the following two days build something useful. As I was getting ready for bed, I got an idea. I remembered a game we play in my family, where we pass a story from one person to the next. Each person improvises, and the story develops in unexpected ways. What would happen if we told stories like that while AI was listening in using voice recognition? As our personal narratives unfolded, the AI could reach into the massive data set of The Met collection and stream back relevant artworks. Drawing artworks from all periods across the world seemed like an inspirational, whimsical way to encounter The Met collection. Most importantly, it would be a way to interact with the collection that would be both personal for the storyteller and meaningful for the group listening. Instead of involving individuals in a solitary, inward-looking interaction with a tool on the web, it would be a platform that invited interaction and conversation. So late that night I drafted a short statement and pitched it as an idea the next morning when we all gathered to begin work.

Curator Griff Mann talks to Azra Akšamija, an artist and architectural historian, with Matt Zumwalt looking on. Behind them is the reproduction of a painting by Peter Paul Rubens from The Met collection

The flexibility of this approach appealed to a range of people from Microsoft, MIT, and The Met. Joining my group, they brought a host of new ideas, technical approaches, and possible outputs. We decided to call our project "Storyteller," and began brainstorming how it might be used in innovative ways. Storyteller could offer people the opportunity to create and preserve oral histories complemented by visually rich imagery from The Met collection. Since the tool would be driven by the spoken word, translation software could open up the platform to people speaking over sixty languages. Voice recognition technology could unlock the door for people with disabilities, and storytellers could dig into The Met collection to learn more about the artworks they discovered through their personal narratives. The AI could even generate a tour of the Museum so that a group of people contributing to or listening to a story could come to the Museum to see their artworks on display. Another output would assemble all of the relevant information about the artworks and produce a print-on-demand catalogue, a coffee-table memento of one's own personal story. Last but not least, the opportunity to share one's own story or to access someone else's story through social media clearly had tremendous potential; it could, for example, make the oral histories mentioned above freely available. Audiences could also enjoy collections of works created by different voices: a scholar, a curator, or perhaps a favorite YouTube star.

Clockwise from upper left: Azra Akšamija designs on a whiteboard. Johann Diedrick, a senior developer at The Met, and Chris Stone, senior software engineer at Microsoft, work on their laptops. Kurt Behrendt, the author, shares the Storyteller protoype as the hackathon concludes

I was fascinated to see how our group succeeded in just a couple of days in making this idea a technological reality. Superficially linking voice recognition software to the collection was straightforward, but figuring out how best to reach into the database to get relevant results was an issue fraught with complexity. I found myself especially interested in the question of how artificial intelligence could be made to understand an image and relate it to a story—a topic that extends far beyond the scope of our initial project and where it stands today as a prototype. Equally fascinating was seeing how the graphic website interface of the Storytelling project could be organized to allow someone to use the platform intuitively to create something that was truly a creative, personal expression.

Poster displayed behind the podium at the hackathon, showing the logos of the three participating institutions above a painting by Winslow Homer in The Met collection

On February 4, participants in the Met x Microsoft x MIT hackathon presented the five prototypes in the Great Hall of the Met Fifth Avenue. Looking back, I realized this project has changed how I think of artworks across the Museum. These diverse objects are so much more than simple traces of a distant past. Just thinking about how they might fit into a modern narrative transforms their meaning and the contexts that I might imagine for them. The Met truly offers us the possibility of bringing artworks to life and seeing the world from new perspectives.

Storyteller Project Participants

Azra Akšamija, Associate Professor, MIT Architecture, Future Heritage Lab, MIT
Lara Baladi, Lecturer, Art, Culture, and Technology Program, Ida Ely Rubin Artist in Residence, Centre for Art, Science & Technology, MIT
Kurt Behrendt, Associate Curator, Department of Asian Art, The Met
Johann Diedrick, Senior Developer, Digital Department, The Met
Griff Mann, Michel David-Weill Curator in Charge, Department of Medieval Art and The Cloisters, The Met
Emile Molin, Head of Design, The Met
Chris Stone, AI CAT, Software Engineer, Microsoft
Matt Zumwalt, Researcher and Reporter, Protocol Labs – Friend of MIT

The Met, Microsoft, and MIT held a two-day hackathon session to explore how artificial intelligence could connect people to art. Learn about this collaboration and the prototypes it generated.