Adding consistency checks to a patient preference DCE
Consistency checks detect respondents who are not reading the choice sets. They are a standard data quality measure in health preference research.
What consistency checks are, how to design them, and how to use them to identify and exclude inattentive respondents.
Knowledge Base -> Experiment Design -> Health
Ben White, 07.07.2026
The problem
A consistency check is a repeated choice set testing whether respondents choose the same alternative when presented with the same or a dominated choice set twice. Respondents who fail are not engaging with the choice task.
Inattentive respondents bias WTP estimates. If the bias is non-random - if certain patient subgroups are more likely to be inattentive - the bias affects not just mean WTP but the distribution of preferences across the sample.
Why this matters in health preference research
Regulators expect consistency checks in patient preference studies submitted for regulatory purposes. The FDA patient preference guidance specifically references internal validity tests. Studies without them face questions about data quality.
Consistency checks provide a quantitative exclusion criterion - a defensible, pre-specified rule rather than subjective judgement about data quality.
TLDR Quick links
Adding consistency checks in SurveyEngine
Step 1: Design the consistency check choice set. One alternative must clearly dominate the other - better on all or most attributes. Any respondent who does not choose the dominant alternative is flagged.
Step 2: Position the repeat task at least 3-4 tasks after the original presentation to reduce memory effects.
Step 3: Pre-specify the exclusion rule. The standard approach excludes respondents who fail 1 of 1 checks, or 2 of 2 if two are included.
Step 4: Report exclusions. Document the number and characteristics of excluded respondents. Systematic differences from retained respondents must be discussed.
Step 5: Conduct sensitivity analysis. Report results both including and excluding consistency check failures.
Worked example - oncology side-effect preference
A patient preference study includes two consistency checks. The first presents an alternative with better efficacy, fewer side effects, and lower cost against one worse on all three. The second repeats a choice set from the main experiment.
Of 220 respondents, 18 fail at least one check. Sensitivity analysis shows WTP estimates are 12% higher in the excluded group, suggesting mild positive bias from inclusion. Primary results exclude consistency check failures.
References
Lancsar, E. and Louviere, J. (2008). Conducting discrete choice experiments to inform healthcare decision making. PharmacoEconomics, 26(8), 661–677.
de Bekker-Grob, E.W., Ryan, M. and Gerard, K. (2012). Discrete choice experiments in health economics. Health Economics, 21(2), 145–172.
Ready to add consistency checks to your patient preference study? Log in to SurveyEngine and follow this tutorial.
Or Contact us at support@surveyengine.com — we're glad to help.