The science of attention checks

Imperfectly attentive respondents are often more representative of the real world than attentive ones. Excluding attention check failures carries severe risks for external validity.

A review of the scientific literature on attention checks - the two main types, what failing one actually means, the side effects of IMCs, best practice design, and how to use attention check data without biasing estimates.

Carol Iskiwitch (adapted), 08.07.2026

Types of attention checks and what failing one means

Ward and Meade (2023) outline two categories of attention checks. Bogus items have a commonly-agreed-upon correct answer - answering incorrectly is considered a sign of inattentiveness. Instructional manipulation checks (IMCs) provide a real question but also contain instructions (perhaps in the middle or end) to ignore the question and provide a specific response. A third type - instructed response items (IRIs) - hides instructions within a single label of a grid response item.

Failing an attention check does not always mean the participant did not notice it. Curran and Hauser (2019) and Silber et al. (2022) document that participants sometimes pass attention checks by chance and fail them deliberately. Attention exists on a spectrum - a single failed check is a noisy signal, not a definitive classification. Anduiza and Galais (2016) found that those who failed an IMC were younger, less educated, and less interested in the survey topic - more similar to the average member of the general population than those who passed.

The case for and against exclusion

Inattentive participants are less compliant with study tasks and provide lower quality self-report data (Maniaci and Rogge, 2014). Ward and Meade (2023) encourage the inclusion of attention checks, particularly in high-risk situations. The concern is legitimate.

But excluding failures has unclear benefits and clear downsides. Berinsky et al. (2014) point out that inattention is a condition in the real world - for survey experiments, excluding less attentive respondents artificially inflates treatment effects. The same logic applies to DCE studies: if attention level correlates with the preference parameters of interest, exclusion changes the policy-relevant estimate. This must be assessed empirically, not assumed.

TLDR Quick links

Ward and Meade (2023) - annual review of careless responding

Curran and Hauser (2019) - recommended attention check items

Anduiza and Galais (2016) - IMCs and satisficing

Best practices for designing and using attention checks

Favour bogus items over IMCs. IMCs - with their hidden and contradictory instructions - can lead participants to think there is more than meets the eye in a survey, increasing systematic thinking and influencing responses on complex reasoning tasks (Hauser and Schwarz, 2015). For standard survey questions the effect is small, but for DCE choice tasks the influence is uncertain. Anduiza and Galais (2016) go as far as suggesting researchers may be better off without IMCs at all.

Think of attention as a spectrum rather than a binary pass/fail. Include multiple attention check items spread through the survey and treat them as a scale (Curran and Hauser, 2019; Muszyński, 2023). Ward and Meade (2023) particularly recommend this for high-risk situations - extremely long or repetitive surveys.

If you must use a single screener question to exclude participants in real time, position it within the first few minutes of the survey. Use questions validated in previous research rather than novel items. Curran and Hauser (2019) recommend combining the strongest items from multiple previously-tested scales.

Adjust for your target population. Previous research may be less relevant if your population differs significantly from the original study population in knowledge or culture. Conduct minimal cognitive interviewing with target population members to check for culturally-bounded understanding of the check items.

On analysis - do not automatically exclude all attention check failures. Excluding inattentive respondents reduces noise but decreases sample size, and the net effect on statistical power depends on which influence dominates. More importantly, non-probabilistic online surveys over-represent educated, interested respondents - removing those who fail attention screeners removes those more similar to the average population member and risks severely biasing estimates.

How to use attention check data

Berinsky and colleagues (2014) suggest that researchers can balance the goals of internal and external validity by presenting results conditional on different levels of attention. Presenting stratified results and considering how the culled sample affects findings allows researchers to benefit from screener questions while avoiding the drawbacks.

In practice: run the analysis on the full sample, then on the subset passing all attention checks, then on the subset failing at least one. If the conclusions do not change materially across the three groups, the attention level of respondents is not a significant source of bias in the study.