Cognitive interviewing for DCE survey testing

Cognitive interviews reveal how respondents actually interpret your survey - which is often very different from how you intended it.

This article explains cognitive interviewing techniques for DCE survey testing, how to conduct them, and how to use the results to improve attribute descriptions and choice task design.

Ben White, 07.07.2026

The interpretation gap

You write an attribute description that seems perfectly clear. A respondent reads it and understands something completely different. This interpretation gap is ubiquitous in survey research and particularly dangerous in DCE research, where misunderstood attributes produce systematically biased utility estimates.

Standard pilot testing - counting completions and estimating preliminary parameters - cannot detect interpretation gaps. A respondent who misunderstands an attribute but consistently applies their misinterpretation will produce internally consistent data that passes all quantitative quality checks. Only direct probing of their understanding reveals the problem.

What cognitive interviews reveal

Cognitive interviews use verbal protocols - think-aloud or retrospective probing - to understand how respondents read, interpret, and respond to survey questions. Think-aloud interviewing asks respondents to verbalise their thought process as they complete the survey. Retrospective probing asks specific questions after the survey is complete.

In DCE research, cognitive interviews are particularly useful for testing: comprehension of attribute definitions; interpretation of level labels (especially for probabilistic or technical attributes); the realism of the choice task scenarios; and the decision-making process (are respondents making genuine trade-offs or applying simplifying heuristics?).

TLDR Quick links

Choice experiment tutorial

Ask on the forum

Open a free account

Conducting cognitive interviews for your DCE

Step 1: Recruit 5–10 respondents from your target population. Cognitive interviews are intensive - 45–90 minutes each - so small samples are standard. Select respondents who represent the diversity of your target population, not just the most articulate or engaged.

Step 2: Choose think-aloud or retrospective probing. Think-aloud is more naturalistic but requires trained interviewers. Retrospective probing - asking specific questions after each section - is easier to standardise and can be conducted remotely via video call.

Step 3: Prepare your probe questions. For each attribute in your DCE, prepare probes: 'What does [attribute name] mean to you in the context of these scenarios?' and 'How did you use [attribute] when making your choices?' For level labels: 'What would [level label] mean in practice for someone in your situation?'

Step 4: Conduct the interviews. Provide the survey link and ask the respondent to complete it while sharing their screen. Record the session (with consent) for detailed review.

Step 5: Identify and fix interpretation problems. Compile the themes that emerge across interviews. Prioritise problems that multiple respondents share. Revise attribute descriptions, level labels, or task instructions and retest with a second round of interviews if necessary.

Worked example - cognitive testing a health DCE

Cognitive interviews for a Type 2 diabetes treatment preference study reveal two systematic interpretation problems: (1) respondents interpret 'probability of hypoglycaemia' as the probability they will personally experience hypoglycaemia during treatment, not the population probability based on clinical trials - the description is revised to use 'out of 100 patients like you' framing; (2) the level 'weekly injection' is interpreted by some respondents as weekly visits to a clinic (inconvenient) rather than self-injection at home (manageable) - the level label is revised to 'weekly self-injection at home'.

After revision, a second round of 5 cognitive interviews confirms both issues are resolved. The revised attribute descriptions are used in the pilot study without further problems.