How to Write Survey Questions That Get Honest, Useful Answers
Johannes
CEO & Co-Founder
14 Minutes
June 5th, 2026
The wording of a single question can shift measured opinion by more than 20 percentage points on the same underlying topic. Pew Research Center's methodology team has documented this repeatedly across decades of national surveys. That is not noise. That is the difference between understanding what your respondents actually think and measuring the question you accidentally asked them. Most survey errors are not technical. They are linguistic. This guide gives you the full system for writing questions that measure what you intend, starting from the moment you define the objective and ending at pilot launch.
What you are actually measuring: attitudes, behaviors, and knowledge
Before writing a single question, identify which type of variable you are trying to capture. The three types require different approaches.
Attitudes are opinions, preferences, evaluations, and feelings. They are measured with rating scales, agree-disagree formats, or forced-choice comparisons. Most customer and employee surveys measure attitudes.
Behaviors are what people do or have done. Behavior questions are most accurate when anchored to a specific recent time period ("in the past 7 days") rather than a general estimate ("how often do you usually"). Pew Research Center found that general behavior estimates produce substantially less reliable data than specific recent recall.
Knowledge is what people know as facts. Knowledge questions require clear correct and incorrect answer options and should not be confused with opinion questions.
Mixing these up is one of the most common mistakes in survey design. "Do you think you exercise enough?" is an attitude question. "How many times did you exercise in the past seven days?" is a behavior question. They measure different things and require different formats.
The 7 types of survey questions
Each question type measures a different construct. Using the wrong type for your variable is a source of measurement error that no amount of analysis can fix.
| Question type | Best for | Good example | Avoid using for |
|---|---|---|---|
| Likert scale (5 or 7 points) | Attitudes, agreement, frequency | "I find the product easy to use" (Strongly disagree to Strongly agree) | Factual yes/no questions |
| Rating scale (0 to 10) | NPS, eNPS, relationship tracking | "How likely are you to recommend us to a friend or colleague?" | Short single-session feedback |
| Multiple choice | Categorization, single selection | "Which channel do you use most often?" | Questions where respondents may have multiple valid answers |
| Binary (yes or no) | Factual screening, gating | "Did you attend the onboarding session?" | Nuanced attitudes where intensity matters |
| Ranking | Priorities, forced tradeoffs | "Rank these five benefits from most to least important to you" | Lists longer than 5 to 7 items |
| Open-ended | Qualitative discovery, unexpected insights | "What is the one thing we could improve?" | High-frequency pulse surveys |
| Matrix | Multiple items with the same scale | "Rate each feature: very satisfied to very dissatisfied" | More than 5 to 7 rows (causes satisficing) |
Practical ratio: Use 70 to 80 percent closed-ended questions for benchmarking and trend tracking. Use 20 to 30 percent open-ended for qualitative discovery. Cap open-ended at two or three per survey. For a deeper treatment of open-ended question design, see our open-ended survey questions guide.
The step-by-step process for writing a survey question
Most bad questions are written in one pass without a defined process. This four-step method catches most problems before launch.
Step 1: Define the decision the question is meant to inform. Write one sentence describing the action you will take based on the answer. "If more than 40% of users rate onboarding as difficult, we will rebuild the first-run flow." If you cannot write that sentence, you do not need the question.
Step 2: Draft the question in plain language. Write the question as if you were asking a colleague in conversation. Do not worry about format yet. Focus on whether the question clearly asks for one thing.
Step 3: Assign a question type and response format. Match the format to the construct. If you are measuring intensity, use a rating scale. If you are measuring category membership, use multiple choice. If you are measuring a factual behavior, use a frequency scale with a concrete time anchor.
Step 4: Test for the 10 common mistakes (covered in the next section). Read the question aloud. Ask: does the wording push the respondent toward any answer? Does it ask about two things? Does it assume something the respondent may not have experienced?
After drafting, run five cognitive interviews before launch (see the cognitive interviewing section below).
The 10 most common survey question mistakes
Each of these mistakes quietly corrupts data. The before-and-after examples show what the fix looks like in practice.
1. Double-barreled questions
Asking about two things at once produces an answer you cannot interpret.
| Before | After |
|---|---|
| "How satisfied are you with the speed and accuracy of our support?" | "How satisfied are you with the speed of our support?" + "How satisfied are you with the accuracy of our support?" |
Signal: Any "and" or "or" in your question stem is a warning sign.
2. Leading questions
Framing that implies a preferred answer measures the framing, not the respondent's view.
| Before | After |
|---|---|
| "How much did you enjoy our excellent new onboarding?" | "How would you describe your onboarding experience?" |
| "Don't you agree that longer product tours are confusing?" | "How would you rate the length of the product tour?" |
3. Loaded or emotionally charged language
Words like "failed," "only," "merely," or "excessive" push responses before the respondent has a chance to form a judgment.
| Before | After |
|---|---|
| "How frustrated were you by the checkout errors?" | "How would you describe your checkout experience?" |
4. Vague quantifiers
Words like "often," "usually," "regularly," and "satisfied" mean different things to different people. Replace them with concrete anchors.
| Before | After |
|---|---|
| "Do you use the product often?" | "How many times did you use the product in the past 7 days?" |
| "Are you generally satisfied?" | "On a scale of 1 to 5, how satisfied are you with [specific thing]?" |
5. Double negatives
Two negations in a question force respondents to parse logic rather than answer from experience.
| Before | After |
|---|---|
| "Do you disagree that we should not remove this feature?" | "Should we keep this feature? Yes or No." |
6. Jargon and technical language
A question with unfamiliar terms is a question respondents cannot answer accurately.
| Before | After |
|---|---|
| "How would you rate our omnichannel CX platform?" | "How would you rate your experience using our product?" |
7. Recall bias
Asking people to remember events from months ago returns mostly guesses. Memory degrades sharply past 30 days for minor events.
| Before | After |
|---|---|
| "Thinking about your experience this year, how satisfied have you been?" | "Thinking about your most recent interaction with us, how satisfied were you?" |
8. Acquiescence bias (covered in depth in the next section)
Agree-disagree formats invite yes-saying. Respondents are measurably more likely to agree with a statement than to disagree, regardless of content.
| Before | After |
|---|---|
| "I find the product useful. Strongly agree to Strongly disagree." | "Which best describes the product? Extremely useful / Somewhat useful / Neither useful nor not useful / Somewhat not useful / Not useful at all" |
9. Social desirability bias
Questions about sensitive behaviors, socially valued traits, or anything with a "right" answer produce inflated or deflated responses. Pew Research Center documents that respondents routinely overstate charitable giving and church attendance, and understate alcohol use and tax non-compliance.
Fix: Make surveys anonymous for sensitive topics. Frame questions to normalize a range of responses. Include response options that make it easy to give the honest answer.
10. Question order effects
Questions earlier in a survey prime how respondents answer later ones. Pew Research Center found a 10-percentage-point shift in general satisfaction ratings depending on whether specific dissatisfaction questions came before or after the overall rating.
Fix: Put general questions before specific ones. Put sensitive questions at the end. Group questions by topic so respondents stay in one mental frame.
Scale selection: 5-point, 7-point, and 10-point
The scale you choose affects both the data quality and how respondents behave on mobile. Here is what the research actually supports.
5-point Likert. The standard for operational surveys. Fast to complete on mobile, easy to analyze with top-2-box percentages, and straightforward to benchmark over time. Appropriate for most customer and employee satisfaction tracking.
7-point Likert. Academic research by Krosnick and Fabrigar (1997, published in Wiley's Survey Measurement and Process Quality) shows 7-point scales yield slightly higher reliability than 5-point when measuring attitude intensity, because the additional points allow respondents to distinguish "somewhat agree" from "agree." Use 7-point for research surveys where fine-grained differences matter.
0 to 10 (NPS scale). The 11-point scale is the standard for Net Promoter Score. Its main advantage is sensitivity: it distinguishes promoters (9 to 10) from passives (7 to 8) from detractors (0 to 6). See our NPS question examples for proven phrasing and common pitfalls.
The neutral midpoint question. Neutral midpoints ("neither agree nor disagree") capture genuine indifference but also attract satisficers who want to avoid committing. Remove the midpoint when you need to force a directional answer. Keep it when genuine neutrality is a real, meaningful position for your respondents.
Unipolar vs bipolar scales. Unipolar scales measure presence versus absence (0 = none at all, 5 = extremely). Bipolar scales measure two opposing ends (1 = very dissatisfied, 5 = very satisfied). Use bipolar for constructs that have a real opposite; use unipolar for constructs that do not.
| Scale | Points | Best use | Mobile | Reliability |
|---|---|---|---|---|
| Likert | 5 | Operational surveys, benchmarking | High | Good |
| Likert | 7 | Research surveys, attitude studies | Medium | Slightly higher |
| Rating | 0-10 | NPS, eNPS | Medium | High for NPS |
| Binary | 2 | Screening, factual gating | Highest | Limited nuance |
The satisficing problem: why respondents stop thinking
This is the most important phenomenon in survey design that most practitioners do not know by name.
Jon Krosnick's research on survey response strategies, published in Applied Cognitive Psychology (1991), introduced the concept of satisficing in surveys: respondents giving the first acceptable answer rather than the most accurate one. Instead of carefully going through all four cognitive stages (comprehension, retrieval, judgment, response), satisficers shortcut the process.
Satisficing behaviors include:
- Picking the first reasonable option
- Straight-lining through matrix questions (selecting the same response for every row)
- Choosing the midpoint without evaluating the question
- Agreeing with every statement
Krosnick found that satisficing increases with survey length, question difficulty, and low respondent motivation. A survey that takes more than 10 minutes is almost certainly producing satisficed answers in the back half.
How to counter satisficing:
- Keep surveys under 15 questions
- Use specific, concrete response options rather than vague anchors
- Ask for specific recent behavior ("in the past 7 days") rather than general estimates
- Vary question formats to prevent pattern-matching
- For matrix questions, limit rows to 5 to 7 and randomize item order
- Place your most important questions in the first half of the survey
For high-stakes research where satisficing is a serious concern, forced-choice formats and ranking questions produce more thoughtful responses than agree-disagree formats.
Acquiescence bias: the problem with agree-disagree questions
Acquiescence bias is the tendency for a subset of respondents to agree with statements regardless of content. Pew Research Center's methodology documentation notes this is especially pronounced among less educated respondents and in interviewer-administered surveys.
Research estimates that 10 to 40 percent of respondents show measurable acquiescence patterns. In a 1,000-respondent survey, that is 100 to 400 people whose responses do not reflect their actual views.
Why agree-disagree formats are the main culprit. When you present a statement and ask respondents to agree or disagree, you are creating an asymmetric choice. Agreeing is the socially easier option. Disagreeing feels confrontational to many respondents, even in anonymous surveys.
Three ways to counter acquiescence bias:
-
Forced-choice format. Instead of "I find the product easy to use (agree/disagree)," ask: "Which best describes the product? Very easy to use / Somewhat easy to use / Neither easy nor difficult / Somewhat difficult to use / Very difficult to use." The respondent must pick a position on the scale rather than simply agreeing or not.
-
Balanced opposing statements. Pew Research Center uses a format where half the sample sees one framing and the other half sees the opposite, then compares results. For recurring questions, this identifies acquiescence effects.
-
Reverse-coded items. In multi-item scales, include some items worded in the opposite direction. If a respondent is straight-lining (always agreeing), reverse-coded items will show internal inconsistency in their data, which you can flag or exclude.
Survey length, order, and flow
How you arrange questions matters as much as how you write them.
The funnel approach. Start with broad, easy, engaging questions. Move to specific, more demanding questions in the middle. End with sensitive items and demographics. This mirrors natural conversation and minimizes early dropout. Qualtrics and the Institute of Education Sciences both recommend this structure in their survey design guides.
Open-ended questions last. Open-ended questions take more effort and produce higher dropout. Place them at the end so that respondents who leave have already answered your closed-ended benchmarks.
Group by topic. Switching topics repeatedly increases cognitive load. Cluster related questions together so respondents can stay in one mental frame without re-orienting between questions.
Primacy and recency effects. In self-administered surveys, respondents tend to pick options at the top of a list (primacy effect). Randomize option order for multiple-choice items where order is not meaningful. Do not randomize ordinal scales.
Sensitive questions last. Income, politics, health, and demographics go at the end. If respondents drop off, you keep their substantive answers.
Survey length targets:
| Survey type | Recommended length | Completion time |
|---|---|---|
| Post-purchase or post-event | 3 to 5 questions | Under 2 minutes |
| Customer satisfaction | 5 to 10 questions | Under 5 minutes |
| Employee engagement | 10 to 15 questions | 5 to 7 minutes |
| Research or academic | Up to 20 questions | Under 10 minutes |
For tactics on getting people to actually complete surveys, see our guide to increasing survey response rates.
Cognitive interviewing: testing questions before launch
Cognitive interviewing is the single most effective pretesting method available, and the most underused. It originated in the US Census Bureau's questionnaire design laboratory and has been documented extensively by Gordon Willis in Cognitive Interviewing: A Tool for Improving Questionnaire Design (Sage, 2005).
The method is simple: recruit five people from your target audience, sit with each one individually, and ask them to think aloud as they read and answer each question. You listen, not for their answers, but for how they process the question.
What to listen for:
- Pauses longer than three seconds (comprehension failure)
- Rephrasing the question before answering (they understood something different from what you wrote)
- Clarifying questions ("Do you mean...?" or "Are you asking about...?")
- Expressing uncertainty ("I'm not sure what you want here")
- Giving an answer that does not match the format ("Well, it depends...")
Willis's research shows that five cognitive interviews surface roughly 80 percent of comprehension problems. The problems that emerge are almost always fixable: a vague term, an ambiguous pronoun, an assumption embedded in the question, a response option that does not cover a common situation.
The five-session process:
-
Recruit five representative respondents. Do not use colleagues. Use people who match your actual target audience in education, familiarity with the topic, and the context in which they would normally take your survey.
-
Run concurrent think-aloud sessions. Ask each participant to say every thought out loud as they read and answer. Do not interrupt or clarify unless they are completely stuck.
-
Document every hesitation and misinterpretation. Keep a log for each question, noting what respondents said or asked.
-
Analyze patterns across the five sessions. Any comprehension problem that appears with two or more respondents needs to be fixed.
-
Revise and repeat if needed. If you rewrote more than a third of the survey, run another round of five interviews.
Cognitive interviewing is not expensive. Five 30-minute sessions can be done in a single day. The cost of not doing it is collecting a full data set built on questions respondents did not understand.
How to pilot test before full launch
Cognitive interviews catch comprehension problems. Pilot testing catches statistical problems.
After cognitive interviews and revisions, launch the survey to 20 to 50 real respondents before full deployment.
What to check in pilot data:
-
Item nonresponse rate. Any question skipped by more than 5 percent of respondents has a problem. Either the question is threatening, it assumes an experience not everyone has had, or the response options do not cover the respondent's situation.
-
Completion time. Compare the expected completion time to the actual median. If respondents are finishing faster than expected, they may be satisficing. If they are taking much longer, the survey is more cognitively demanding than you thought.
-
Straight-lining in matrix questions. Check whether any respondents gave identical answers to every item in a matrix. More than 5 to 10 percent straight-lining indicates the matrix is too long or the items are too similar to discriminate.
-
Response distribution. A question where 95 percent of respondents pick the same answer is not discriminating. It is either too easy, too loaded, or measuring something everyone in your sample shares. Cut it or rewrite it.
-
Open-ended response quality. Read the open-ended responses. If they are mostly "N/A," "none," or single-word non-answers, the open-ended question is either poorly positioned, too vague, or placed too early in the survey.
After reviewing pilot data, make final revisions. Then launch to the full sample. Do not reopen the questionnaire mid-field, since any changes after data collection begins make the pre- and post-change responses incomparable.
Start Writing Better Survey Questions Today
The fastest way to test your questions is to run them. Formbricks is a free, open-source survey tool that lets you deploy surveys in minutes, with no engineering help needed. It supports every question type covered in this guide and is self-hostable for teams with strict data privacy requirements.
How to get started:
- Sign up at formbricks.com (free, no credit card required)
- Start from a template or build from scratch using the question types in this guide
- Set targeting rules to reach the right audience at the right moment
- Launch, collect responses, and analyze results in real time
For related reading:
- Open-ended survey questions: when and how to use them
- How to increase survey response rates
- Survey distribution methods compared
- NPS question examples that actually work
- Customer satisfaction survey questions
- Employee engagement survey questions
Frequently Asked Questions
Try Formbricks now
