Designing a Questionnaire That Survives Peer Review

Data analysis gets all the attention, but the quality of a quantitative study is decided much earlier, at the moment the questionnaire is drafted. If a question is ambiguous, double-barrelled or leading, no amount of clever modelling will repair the answers it produces. Reviewers know this, which is why the instrument section is often where a manuscript is first challenged.

Turn constructs into measurable items, not opinions. Begin with your conceptual framework and list each construct you claim to measure. For every construct, find items from published, validated scales wherever possible and cite the source; borrowing a tested instrument is a strength, not a weakness. When you must write your own items, keep each one to a single idea, use the everyday vocabulary of your respondents, and avoid negatives inside a positively worded scale. One question should never ask two things at once.

Match the response format to the analysis you intend to run. A five- or seven-point Likert scale gives you ordinal data that behaves well in composite scores; a yes or no answer gives you a binary variable suited to chi-square or logistic regression; an open income box gives you continuous data but also invites refusals and rounding. Decide in advance which variables must be continuous, because converting a bracketed income range back into a usable number after data collection is impossible.

Establish validity and reliability before the full survey, not after. Send the draft to two or three subject experts and record their judgements as a content validity check. Then pilot the instrument with twenty to thirty respondents who resemble your sample, and compute Cronbach's alpha for every multi-item scale. Values around 0.70 and above are usually acceptable; below that, look at the item-total correlations and remove or rewrite the weakest items. Report all of this in your methodology, since it is exactly what a reviewer looks for.

Pilot for logistics as well as wording. Time the interview, watch where respondents hesitate, and note which questions enumerators have to explain. Fix the sequence so that easy demographic items come first and sensitive items come late, once trust exists. Add explicit codes for refused and not applicable, so that missing data can later be distinguished from genuine zeros.

A questionnaire is an instrument in the literal sense: it either measures accurately or it does not. Spending two extra weeks on design and piloting is the cheapest insurance available to a researcher, and it usually saves months of defensive explanation later.