Checking survey data before main fieldwork

Simulation and manual data checking before fieldwork are the only reliable way to confirm a complex survey is collecting data correctly.

How to use simulations and manual data review to verify a survey instrument before main fieldwork - covering simulation setup, data verification criteria, and the final manual check protocol.

Ben White, 09.07.2026

Why manual testing is not sufficient

Previewing a survey as a single respondent identifies display and logic errors that affect typical response paths. It does not identify problems that only appear in aggregate data - incorrect treatment allocation proportions, derived variables that fail for specific input combinations, branching logic that routes a minority of respondents incorrectly.

These aggregate problems are invisible until you have data from multiple respondents following different paths. By main fieldwork that data is real respondent data - expensive to collect and impossible to recollect if the instrument was wrong.

Simulation produces a synthetic dataset for verification

A simulation runs automated respondents through the survey, producing a complete dataset without real human participants. The dataset can be inspected for correct treatment allocation, expected branching proportions, plausible derived variable values, and any unexpected patterns that indicate errors in the survey logic.

Simulation is not a substitute for human review - automated respondents cannot evaluate whether question wording is clear or whether the task is cognitively reasonable. But it is the only efficient way to verify aggregate data behaviour before fieldwork begins.

TLDR Quick links

Pilot study design for a DCE

Soft launch and fieldwork monitoring

The verification process

Step 1: Create a simulation copy. Never run simulations on the live project. Copy the project, append '_sim1' to the name, and run all simulations on the copy.

Step 2: Remove or simplify conditions that require human knowledge. Screener questions that require genuine eligibility, captchas, and complex branching conditions that cannot be satisfied by random input need to be adjusted in the simulation copy to allow all paths to be traversed.

Step 3: Run a minimum of 100 simulated respondents. Check that treatment allocation proportions match the design. Check that branching proportions are consistent with the intended logic. Check that derived variables fall within expected ranges across the full distribution of input values.

Step 4: Complete the survey manually twice, using different response paths. Pay particular attention to back-navigation and re-entry of earlier pages. Check that partial completion and resumption works correctly if the survey supports save-and-return.

Step 5: Get sign-off. The principal investigator should review the simulation data and confirm in writing that the instrument is collecting data as intended before main fieldwork begins. This is mandatory for regulated studies and good practice for all studies.

Catching a derived variable error in simulation

A transport SP study uses a pivot design. Simulation with 100 automated respondents reveals that 3 respondents received a choice set with a negative travel cost - an impossible value resulting from a boundary condition in the pivot calculation. The error affects respondents reporting very low reference costs.

A minimum cost constraint is added to the pivot calculation and the simulation is rerun. All 100 respondents receive plausible attribute levels. The correction takes two hours. Finding the same error in main fieldwork, after 200 real respondents had been collected, would have required excluding the affected respondents and assessing whether the remainder of the dataset was sufficient for analysis.