Our Blog

Coding: What to Do With a Sea of Open-Ended Responses

Regardless of whether it’s in a community forum about smart grocery shopping or a quick survey about power outages, online consumers weigh in with strong opinions about what they want and like.

It might feel like diving into an ocean full of sharks, but when you explore verbatim consumer responses, you can often identify pain-points around a product, program, or concept before the blood hits the water.

Unsurprisingly, knowing how to navigate and interpret these raw responses takes keen attention to detail and a deft touch. The following are steps and tips to translate these open-ended survey responses into the viable data and business initiatives.

Step 1: Identify how you’ll use the collected data

When respondents are given the opportunity to write unscripted responses to questions with little structure imposed by the researcher, the results can vary in terms of length and scope. Rather than analyzing a hard number of boxes that have been checked off, the output is words.  

Do you treat responses as qualitative or quantitative data?

Having a strategy in place as you format your questionnaire is the key to generating data that can be analyzed correctly. To ensure that responses are targeted and concise, a little structure goes a long way for open-ended questions and makes it much easier to analyze the resulting data quantitatively. Consider the following:

• Does the question provide a context that will keep responses focused?
• Are there directions about how detailed responses should be?
• Is the survey formatted to only allow responses of a certain length?

Evaluating what you really want to gain from the open-ended question is essential. If it’s included simply to cover all your bases, or you already know that funding won’t cover analysis and use of the data, is the question really necessary?  

An open-ended question is meant to provide a space for complementary or new ideas to be raised. Consider whether or not the data will be used to its fullest to augment your findings.

Step 2: Choose the appropriate open-ended question 

Open-ended questions can be framed and contextualized in any number of ways, which creates diversity in the scope and depth of responses. To aid this process it is important to consider what format will best serve your needs.  

• The extension: “Other, please specify” comes at the end of a list of explicit options that frame and contextualize the response. Use of this question is considered a best practice in market research and ensures all options are covered.
• The substitution: “Please describe your behavior in the following situation:” using this format often provides detailed and accurate information about consumer behavior in a specific context. 
• The expansion: a closed questioned, followed by an open question. Asking why or how can give illuminate or expand upon quantitative questions.
• General: “In your own words…” Asking for any additional comments in regards to their experience to the topic of the survey offers an opportunity for new ground to be covered.

From a consumer’s point of view, these kinds of questions represent a chance to give an unscripted, fully fleshed-out answer to some burning issue they have in mind. From a researcher’s point of view, discovering the specific meaning, relevance, and value of the response is part and parcel of developing additional insights regarding the topic of the survey.

Step 3: Coding and analysis

After fielding the survey and gathering the appropriate number of responses, it is time to code. This is where a lot of researchers sink or swim—proper coding is a rigorous and time-consuming process that necessitates an appropriate methodology to prevent problems and in-built biases.

• The preliminary analysis. Do the comments provide enough additional information for them to be analyzed? If the open-ended questions yield little more than the closed questions within the survey, a better strategy might be to consider how they fit into the study overall.

  If formal analysis is not warranted, a best practice is to report within the results that the responses to the general open question did not provide additional information to the closed questions.

• Responder bias. As open-ended responses are gathered, it is important to consider who is providing the content. Are females answering in greater numbers? Does annual income or education make a difference?

  Keep your respondent quotas balanced. Sometimes a certain respondent segment does not answer as much or as often. This creates a non-response bias. Check daily if need be to make sure you are getting the appropriate numbers for each segment. If you need to invite more people to participate, do so.

• Create a coding framework. This framework will organize the content of the comments into thematic, quantifiable elements that are composed of issues not within the scope of the close-ended survey questions.

  It is the coder’s responsibility to consistently categorize responses, which involves accurately interpreting comments that may be grammatically incorrect, vague, or incomplete. If multiple coders have worked on the project, cross-checking and comparisons are vital in order to keep categorizations consistent.

• Analyzing the codes. Each code or thematic element is then treated as a variable in a quantitative or qualitative analysis.

  Whether the open-ended question provides the meat of your data or was an afterthought, use the new information to improve your product or service. These are real thoughts from real consumers that can outline your weakness and strengths.

Use open-ended responses to land a bigger fish

Open-ended questions catch those important issues that closed questions miss. They encourage consumer engagement and can reveal pain-points you were previously unaware of. They can also reveal or validate areas where you are succeeding.

Maybe your consumers want easier access to a great service or product. Maybe your customer service protocol needs to be improved. You don’t know until you ask.

Do you have any suggestions or tips for using open-ended questions?