Adventures In Evaluating Participatory Exhibits: An In-Depth Look At The Memory Jar Project

A man walks into a museum. He shares a story. He creates a visual representation of his story. He puts it on the wall.

How do you measure the value of that experience?

Two years ago, we mounted one of our most successful participatory exhibits ever at the Santa Cruz Museum of Art & History: Memory Jars. Over three months, about 600 people filled mason jars with personal memories and put them on display. Better yet, the graduate student who led this project, Anna Greco, documented the whole project and did in-depth analysis of the visitor contributions. This post shares some of the highlights from the project and from Anna's research. I strongly recommend checking out her entire thesis [pdf] if you want to know more.

THE BASICS

The Memory Jar project was simple. We filled a small gallery from floor to ceiling with shelves of Mason jars. We invited visitors to "bottle up" a memory in a jar, using craft materials to fill the jar with evocative objects and a hand-written label to tell their story. There were no written instructions, just a mural that suggested what to do and labels that prompted people for their name and memory. The project was linked to a larger exhibition, Santa Cruz Collects, about why local folks collect things. We realized that the most valuable things many of us collect are intangible--our memories--and the Memory Jar project was born.

From the beginning, we observed pretty amazing experiences happening with the Memory Jars. People were spending a long time working on them. Some of the stories were quickies, but others were powerful and personal. We started with 400 jars and assumed we wouldn't fill them all. Instead, we had to do a rush order on more jars halfway through the project.

Two years later, this project is still one of the most fondly remembered participatory experiences at the museum--by visitors and staff. There was something special going on in that gallery. What was it?

THE RESEARCH

The challenge, of course, was to figure out how to evaluate the experience in a way that would help us identify the power of the project. We wanted to know whether the project truly had emotional resonance, and if so, how we could identify and document that.

Anna Greco did research in three ways: through in-person interviews with participants, surveys with participants, and observational analysis of the jars themselves. These methods revealed that the majority of participants had a meaningful experience with the memory jars that stuck--even in followup a year after the initial project.

I think most of us are familiar with interview and survey-based evaluation methods. I want to instead highlight the work Anna did to analyze the content of the jars observationally, which got at the question of emotional resonance in a more quantitative way.

Anna did two types of quantitative evaluation of the jars:

• she analyzed the jar contents, looking at how full the jars were. This was used as a proxy for time and creative energy spent on the creation of jars.
• she analyzed the text on the jar labels for length, for emotional content, and for intimacy. This was used to evaluate the amount of emotional energy dedicated to the jar activity.

In each of these analyses, Anna created a coding scheme to categorize the jars.

For the fullness of the jars, Anna created a seven-point scale, going from empty to full to full+ (additional decoration or objects on the outside of the jar) by percentage of jar full. It was fairly simple to identify whether a jar was 1/4 full or 3/4 full or had stuff popping out the top of it. The result here was surprisingly linear, with more than half of the jars full or full+. People used the stuff and they used it to the fullness of their ability. This could also be an argument for larger jars if we repeat the project.

For the content of the labels, Anna used three strategies.

• She counted the number of words in each label. This was an easy (if time-consuming) proxy for engagement. The average label had 17 words, though the maximum was 105. Again, this indicates a high level of engagement, especially given the size of the shipping labels provided.
• She created a numeric scale for the "intimacy" of each label. This was created with the help of Dr. Lauren Shapiro, a psychologist and former museum intern. Lauren and Anna created a scale of one to five where each level had specific elements to indicate intensity of the story shared, using signifiers like specificity of a memory, places, names, direct quotes, or medical information. 70% of the labels were a 3, 4, or 5, with only 4% at a 1. People got intimate, sharing intense stories of loss, special moments, and potent memories.
• She created a coding schema for "emotions" expressed in each label. Lauren helped Anna create a manual for language cues to signify any of nine emotions: happiness, love, gratitude/awe, sadness, pride, anger, fear, confusion, and mixed. 36% of the labels were happy, closely followed by 32% that demonstrated no clear emotion. Labels in the "no emotion" were "reporting" memories without explicit emotional language, as in "I remember going to the beach with my friends." Among the remainder, love, mixed emotions, gratitude, and sadness were the most frequent.

Intimate? Emotional?
Coding is useful but complicated
when the goal is to capture feeling.

We learned a few things from this process:

• Creating a coding scheme for text analysis is complicated, but it's worth it. Especially with such a large number of jars, it was really valuable to be able to distill the diversity of stories via a few key axes of intimacy and emotion. Unsurprisingly, developing the coding schemes to be able to be applied fairly consistently by people without a lot of specific pelatihan led to imperfections. But just going through the process helped us understand how we COULD quantify this kind of extremely qualitative content. You can check out Anna's coding scheme manuals on pages 67-69 of her thesis.
• Intimacy was the most useful indicator for this project, but still really complicated to measure objectively. If we were doing this again and had time to either code by emotion or intimacy but not both, I would choose intimacy. The intimacy measure was the clearest signifier of how people were using the Memory Jars and what stories they chose to tell. That said, we still had plenty of debate about what qualified as more or less intimate as the schema was being developed. Mention AIDS with no context, and you shot up to a 5 on intimacy. But tell an incredibly detailed story about your childhood, and you were likely to end up somewhere like a 3. When Anna did followup surveys, it became clear that many stories were more intimate to individual jar-makers than their coded labels reflected. It's arguable that the much simpler measure of word count is sufficient as a proxy; high intimacy stories had an average word count of 38, compared to the overall average of 17. If you use more words, you are probably going deeper into your story, which typically involves more commitment, intensity, and maybe--intimacy.   
• There are quantitative ways to measure creative participation. We keep trying to find ways to assign numbers to different kinds of participatory projects at the MAH. None of them are perfect, but all of them are useful in moving towards better design and better yardsticks for our work. The Memory Jar project allowed us to experiment with a more robust approach to quantitative evaluation of participation, and it got us interested in other ways to do it in simpler projects. For example, check out this quantitative method we use for comment boards. 
• Evaluating participatory experiences exposes new questions we actually care about, which are often different from what we thought we wanted to know. In trying to get a grasp of the kind of emotional resonance of the Memory Jars, we started having some interesting discussions about design and the end goals for our work. Would it be "better" if all the jars were a 5 on intimacy? If there were as many sad jars as happy ones? Just having this data opened up new ways of talking about what we are trying to achieve with our work. One of the goals we've stumbled into regarding participatory projects is the diversity of content presented. For some projects, ones that are designed to invite everyone to engage, we want to make sure the project absorbs a diversity of content in terms of language, emotion, intensity/intimacy, and creativity, so that every visitor can find their place in the project. In other projects, we actually want to "gate" the experience so that only people who are willing to devote X amount of time or intention will complete the project. Or we want to make it as simple and breezy as possible. Looking at the mix of what happened with the Memory Jars got us thinking about the ways we design for different kinds of outcomes, when we want diverse outcomes and when we want more focused sets.

What approaches have you used or considered for evaluating participatory projects? Please share your stories in the comments.