Quality
as Standard
Every study fielded on SurveyEngine is quality checked.
Three stages, running from before the field opens to after the data is delivered, with an optional signed attestation on top.
How Is Quality Assured
- Prevention - quality measures are built into the studt
- Live On-demand - Standard citable quality reports with no setup effort.
- Expert Review - domain knowledge and experience validates the approach
- Attestation - Provision of Independent Signed Attestations
Benefits of This Approach
- Transparency - All data, quality reports and their scripts are available during data collection.
- Citable Methods All Quality methods use citable published methods that may be independently reproduced.
- No arbitrary Scores Quality recommendations use P-values to determine whether an anomalous behaviour is expected by chance.
- Actionable Flagging - Each respondent with anomalous behaviour is identified, with reasons supported by concrete statistics.
- Guidance, not adjudication - We flag, quarantine and recommend. You set the final exclusion criteria and you decide what stays in the dataset
What Checks are Done?
- Completion Time per page against expected
- Irrelevant segment modelling - does behaviour differ by irrelevant segements like browser or start time
- Geolocation - Does the respondent locaiton match the cohoer
- Response Patterns - are responses similar to other respondents
- Diurnal Activity - how likely are the respondents to be participating outside waking hours
- Incidence Rates - are incidence rates climbing, indicating feed-forward fraud
- Standard Speeder checks
- Domain and Study Expert review
Data quality checking is included in every SurveyEngine project. It is not a module, an upgrade, or something you have to know to ask for.
It is part of how we run fieldwork, and it is priced into the cost per interview rather than quoted as an extra.
Why it is standard?
- because the alternative is finding out too late
Quality problems in survey data are not hard to find. They are found late. Finding them takes somebody who understands the study looking closely at the data, and that person is expensive, busy, and in practice only looks once the data has been delivered. By then the field is closed and every remaining option is a poor one.
Making the checks automatic and standard removes the part that fails: nobody has to decide whether this study is worth looking at, and nobody has to find the time.
What runs, and when it runs
Each stage catches something the others cannot, and each happens at a point where a different set of remedies is still available.
Before and during field
Stage 01
Prevention
Controls built into the instrument itself, so problem responses are caught or deterred as they arrive rather than analysed afterwards.
- Virtual private network (VPN), proxy, Tor and relay detection
- Automated agent and bot signatures
- Cookie-based duplicate and re-entry control
- Private browsing detection
- Geolocation against the target country
- Hidden attention and numeracy items
- Study-specific eligibility controls, including documentary evidence where a study requires it
What it gives you
Problems are kept out of the dataset rather than argued about later, and because the controls are written into the study specification before the field opens, the decision to apply them cannot be said to have been made after seeing the results.
During field, on demand
Stage 02
Automated read
A standard quality report that runs on any study without configuration, at any point, including while the field is still open.
- Timing, reading speed and page-level behaviour
- Screener to main-study consistency
- Network and duplicate signals
- Choice-task behaviour variation
- Diurnal response anomalies
- Comparison between recruitment sources on the same study
- Reported as probabilities against that study's own distribution, with published methods you can reproduce
What it gives you
The same consistent read on every study, without requiring domain, study or data experts. Early enough that remedies are still available. Transparent export of methods, citations and quality script, and ultimately a list of individual respondent who failed the tests and why.
After data collection
Stage 03
Expert review
A an expert familiar with study looks at what was flagged, but also brings study and domain knowledge to the analysis that and applies the judgements no automated check can make.
- Whether the observed prevalence is plausible for that population
- Whether a respondent knows what a patient with that condition would know
- Whether a shared address is a clinic, a household or something else
- Which anomalies to ignore (e.g. IP addresses in a clinical setting) and which require further analysis.
- Each flagged respondent classified valid, invalid, or needing further work, with the reason recorded
What it gives you
Context no algorithm has, and a reason attached to every exclusion, so a decision can be explained to a sponsor months later rather than defended from memory.
On Request - A signed attestation
Where a sponsor, an ethics committee, a regulator or a journal needs more than the data, we can provide a signed record of the quality work: which checks were run, what they found, what action followed, who decided, and the final number of validated completes.
It is signed by the person who did the review, not by a director, so it says something about the work rather than about the company. We do not certify that every respondent is genuine, because nobody honestly can. We state what was checked and what was found.
What it gives youA document that can go into a file and be defended to an auditor, an ethics committee or a sponsor, rather than an assurance that has to be taken on trust.
What each stage still lets you do
The value of a finding depends almost entirely on when it arrives.
| Stage | When | What is still possible |
|---|---|---|
| Prevention | Before a response is recorded | The response is blocked, quarantined or never enters the sample |
| Automated read | Any time, including mid-field | The supplier replaces those respondents under a live contract, usually at no cost to you. Quotas and sourcing can be changed while it still matters |
| Expert review | After collection, before delivery | Respondents excluded and replaced, or the dataset delivered with the exclusions documented |
| Attestation | On delivery, or later on request | The record exists to be put in a file and defended |
Two things worth knowing before you ask
The survey runs on SurveyEngine. The recruitment mix stays yours.
Most of what makes this work is captured structurally while the study is running, which is why the instrument has to be on our platform. Who recruits is a separate question. The quality layer sits across whatever mix a study needs: direct, patient advocacy group, physician referral, panel, or your own preferred supplier. Where a design allows it we recommend two suppliers measured against the same benchmark, because a difference between them is itself a finding.
Thresholds are agreed before the field opens.
Trigger values go into the study specification at set-up, not once the data is in. Choosing a cut-off after you can see which respondents it would remove cannot be defended in an audit. We will propose values calibrated from distributions across completed studies, and you confirm them as part of scoping.
Cost - and what that means for a quote
The quality work is part of the fieldwork service rather than a line you can decline. It accounts for roughly 15 per cent on top of the recruitment cost, and it sits inside the cost per interview.
That is why a SurveyEngine cost per interview looks higher than a quote from a supplier who programmes a survey and hands it back. The comparison is incomplete rather than unfavourable. On request we will break a quote into separate lines so that the comparable part is visibly comparable and the optional work is visibly optional.