Pilot study design for a discrete choice experiment

A pilot study is not optional in DCE research. It is the only way to identify problems with your design before they contaminate your main dataset.

This article explains how to design a DCE pilot study, what to test, how many respondents you need, and how to use pilot results to improve your main study design.

Ben White, 07.07.2026

Why pilot studies are essential

A DCE pilot study conducted with 30–50 respondents can identify problems that would contaminate data from 300–500 respondents in the main study. These problems include: attributes that respondents do not understand or interpret inconsistently; levels that are perceived as unrealistic or implausible; choice tasks that are too cognitively demanding; and survey instructions that are unclear.

Without a pilot, you discover these problems too late - after collecting data that cannot be salvaged. Pilot studies are the most cost-effective quality control measure available to DCE researchers.

What to test in a DCE pilot

Ideally you've done a cognitive pre-test beforehand so respondents understand all attributes and levels as intended. Nevertheless, it may be worth including 'throwaway' cognitive questions just for the pilot. Most of the work is verifying that the analysis produces expected results and if you are using D-efficient designs, using the results to update the priors and experiment design.

Starting with the models. With 30–50 respondents, you can estimate preliminary MNL parameters. These are valuable for two reasons: they confirm the design is identifying parameters, and they can be used as priors in a Bayesian D-efficient design for the main study.

You'll also look at the edges now you are live: Did the panel linking work? Are there patterns in the comments that need to be addressed? Are the models over dominated by one attribute? Is any of the data looking suspect? Are there high dropout rates.

A well designed experiment should go to main unchanged and with confidence.

TLDR Quick links

Choice experiment tutorial

Ask on the forum

Open a free account

Conducting a DCE pilot in SurveyEngine

Step 1: Deploy the survey to a small sample. In SurveyEngine, set up the pilot as a separate project or use the same project with a pilot flag. Target 30–50 respondents who are representative of your main study target population.

Step 2: Possibly add cognitive interview questions. After each choice task or at the end of the DCE, add questions asking respondents about their understanding of specific attributes. For example: 'In your own words, what does the Reliability attribute mean in these choice tasks?'

Step 3: Monitor completion rates and times. In SurveyEngine's fieldwork dashboard, track completion rates and average completion times in real time. Flag unusually fast or slow completion times for review.

Step 4: Estimate preliminary parameters. Export the pilot data and estimate a simple MNL model. Check that all parameter signs are in the expected direction and that the model converges. Large standard errors on specific parameters may indicate design issues.

Step 5: Iterate before the main study. Use pilot results to revise attribute descriptions, adjust level ranges, simplify task instructions, or regenerate the design with updated Bayesian priors. Document all changes between pilot and main study.

Worked example - pilot for a medication preference study

A pilot for a patient preference study on rheumatoid arthritis medications recruits 40 patients through a patient advocacy organisation. The pilot identifies two problems: (1) patients interpret the 'probability of remission' attribute as the probability that they personally will achieve remission, not the population probability - the attribute description is revised to clarify this; (2) the choice tasks take an average of 4.2 minutes each, suggesting excessive cognitive burden - the number of attributes is reduced from 6 to 5 by combining two correlated attributes.

Preliminary parameters from the pilot are used as Bayesian priors to generate a more efficient main study design, reducing the required sample size from 200 to 175.