Write annotation questions

Reading your story texts and compiling observations about them is a good way to draw more patterns out of the data you collected. Mixed-methods data analysis combines qualitative (text-based) and quantitative (numbers-based) techniques. Annotation questions help you with the qualitative part of that equation. For every project there are bound to be some annotation questions that can provide useful insights. It's usually worth spending at least a little of your time exploring what annotation can add to your project.

You can start out a choice-type annotation question (of the types "select," "radiobuttons," or "checkboxes") with an empty answer list. Just start reading stories (on the Annotate stories page). When you read a story, if you don't see a good answer to the question in the list, click the "Add New Answer" button under the list. Keep doing this. As you read each story, look through your growing list to see if you already have a matching answer for that story.

After a while, you will notice that your list is getting long. Stop and go back to this page (Write annotation questions). Select the question and look at your answer list. Look for answers that are substantially similar to each other. For example, say you're marking themes. You might have added "the system is rigged" for one story and "you can't trust the system" for another. Use the Change button next to both answers to merge them into a new, common answer.

Be careful doing this. You are changing the tags on the stories as well as in the question. After you merge two answers, you will lose the distinction between them. See below for more information on using this bulk-change feature.

Let's go through each field on the page.

Question text: not connected to the stories. The text of an annotation question isn't connected to your stories in any way. It is only used to display the question to you. Changing it won't have any effect on your stories.

Question short name: connected, but you can change it. For each story, answers to questions are stored using each question's short name as a lookup index. When you change the "Short name" field for an annotation question, NarraFirma reaches into each story and changes each lookup index that links it to that question. So you can go ahead and change a question's short name. Just be aware that you are also changing your stories when you do this.

Question type: connected, and you can't change it (usually). The type of a question determines how the data connected to that question is stored for each story.

• boolean - a single answer, "yes" or "no"
• checkbox - a single value, true or false
• checkboxes - a series of answers from a fixed list, each with a value of true or false
• text or textarea - a single free-form text
• select or radiobuttons - a single answer from a fixed list
• slider - a single number

There are a few type changes that will not lose any data. You can change a "text" question to a "textarea" question and back. You can change a "select" question to a "radiobuttons" question and back. Most other type changes will result in data being lost. It is best to set a question's type at the start of your annotation and not change it.

Checkbox label, slider labels: not connected to the stories. As with the question text, these labels are only for you to see. They have no effect on your stored data.

Choice question answers: connected, but you can change them. For each story, answers to questions with fixed answer lists (of the types "select", "radiobuttons", and "checkboxes") are stored using the answer names in the list (e.g., "positive experience" or "first-hand story"). When you change any of the answers for these questions, NarraFirma reaches into each story and changes the answers that are stored there.

This bulk-change feature is powerful but dangerous. You can use it to develop sets of answers that grow and change as you read your stories. But be careful. When you merge two answers into one, for example, you can't easily un-merge them later. You might want to backup your story collection (which will include all of your annotations) before you make any far-reaching changes. That's why there are "Export story collection" and "Export project" buttons at the bottom of this page.

Maximum number of checkbox choices: only affects new answers. For a "checkboxes" question, if you change the maximum number of checkboxes allowed after you start answering your questions, the new maximum will be used from then on, but it won't affect any of your previous answers.

When you edit an annotation question, at the bottom of the panel you will find a button that says Copy a question from a template. Click that button to see some annotation questions that have been used on PNI projects in the past. (You can find more annotation questions in Working with Stories.)

But those are just ideas for questions. The best place to find good annotation questions is in the stories you collected. First, read through your stories without any questions in mind. After a while, you'll start to notice that some words, concepts, and topics keep coming up. Or you might notice that certain things vary among the stories you read -- people keep mentioning varying degrees of trust, for example. When you begin to see trends in the texts of your stories, consider whether any of those trends might make a good annotation question. When you find yourself wanting to mark the appearances of words or concepts you keep seeing in your stories, you've found a good candidate for an annotation question.

For example, the "Scope" annotation question often works well with stories about communities or organizations. Some storytellers focus on small scopes (themselves and one or two other people) while other storytellers talk about larger scopes (dozens or hundreds of people). Thus the question "How many people are mentioned in this story?" can sometimes create useful patterns when it is juxtaposed with questions your storytellers answered, like "Did the people in this story behave responsibly?"

Yes. An annotation question should be a question whose answers anyone can see and agree with. A question whose answers are likely to represent one person's perspective or opinion won't make a good annotation question. Consider, for example, these three questions:

1. What does this story say to you about trust?
2. How much trust is evident in this story?
3. To what extent did the storyteller mention trust?

The first question is clearly a question of perspective, because the same story might say two very different things about trust to two different people. The second question sounds better, with its "evidence," but it's still a question of interpretation. The third question is a good annotation question. Anyone who answers it should be able to justify and defend their answer by showing evidence drawn from the story itself.

One way to test annotation questions is to pretend that the people who told the stories are being presented with a report based on those questions. Would they feel that the questions fairly and objectively represent what they said? Or might they feel that their words could be twisted by the use of the question?

Here's another set of questions. Which of these would make good annotation questions?

1. Do you agree with the perspective of the person who told this story?
2. Who do you think would agree with the perspective put forth in this story?
3. To what extent do the views put forth in this story align with this community's official policy?

Again, only the third question would make a good annotation question, because its answer could be explained by juxtaposing the words of the story with the words of the community's official policy. There might be some interpretation involved in deciding how well those words aligned, but at least it would be possible to make the comparison.

Whoever is doing the catalysis part of your project. That might be you, or it might be you plus a group of people who are helping you to do catalysis, or it might be you plus a group of people you are helping to do catalysis.

In any case, answering annotation questions is part of the catalysis process (not the collection process), so it is a form of qualitative research. This means that whoever is answering the annotation questions should follow standard guidelines for doing sound qualitative research. Ask yourself these questions about the process you are using to answer your annotation questions:

• Is your process credible? Is it repeatable? Is it transferable? Is it connected to you and your way of seeing things, or is it independent of your influence?
  • One way to ensure credibility is to have two or three people answer each annotation question. If you have a lot of stories, choose a subset and have two or three people answer the questions for just that subset. Then compare their answers. If the answers are very different, why is that? Can you see a pattern?
  • If you are doing catalysis by yourself, show ten or twenty of your stories to someone who knows nothing about your project. Then show them how you answered your annotation questions about those stories. Ask them how they would answer the questions. Would their answers be the same as yours? If not, why not? What would happen if you changed the questions?
  • If you can't show your stories and answers to anyone, pretend to show them to someone. Pick out someone in your mind who often disagrees with you. What would that person say about your answers? Would they agree?
• Is your process transparent? Can you explain it in detail? Can you provide proof that it is grounded in what people actually said? One way to ensure transparency is to avoid using your own words to describe what people said. Instead, rely on verbatim exerpts from your stories. You can use verbatim excerpts to check your process as you go. If you can't point to a place where someone said something that explains each answer, don't mark that answer. Better yet, take notes on why you answered the questions the way you did. Be ready to show your work to anyone who wants to know how you answered the questions.

If you look on the internet for "qualitative research standards" you can find additional guidelines for making sure your process is rigorous (credible, dependable, reliable, transferable), ethical, and transparent. The last thing you want to do is give people a voice and then distort what they say.