In this post I’m going to cover the pros and cons of randomizing the question order in your employee survey, and why … well … it depends.
In the end though the intent is to balance clarity and ease-of-use on the part of the respondent, against any response biases that might be caused by your choice of which way to go.
What do I mean by “randomize the order”?
I’m talking here about whether or not you provide all your questions in a particular order by some form of grouping; like a “question type” or “driver”. (By “driver” I mean topics like Compensation, Engagement, Management, etc.).
What I’m NOT talking about, and what you should never, EVER do in an Employee Survey?
- DON’T randomize the question order for each respondent – everyone should see the same survey. Unless you are talking about very large sample sizes, randomizing the question order just adds a new level of uncertainty to the results. (“But what about “Early Question Bias” you say? Happy to discuss this at length, but in this case the cons outweigh the pros.)
- DON’T randomize or change the scales or options. Conceptual ideas of “positive” and “negative” should always flow in the same direction throughout the survey. I could write an entire post on why this is so – but in short it would again add a level of confusion and uncertainty to your results. (… and anyone who makes the argument that “it forces the respondents to concentrate” please send them to me – I will have words with them.)
- DON’T spread the demographic questions throughout the survey (ex. Location, Department, Age, Gender, etc.). Keep them in one place if you can. (At the beginning or end? I’ll address that in another post asap as “it depends”.)
Doing any of the above might make the survey results suspect from a “stats/validity” point-of-view.
The pros and the pros
So let’s get into it … now that we know we are ONLY talking about the pros and cons of randomizing the order of the non-demographic questions in employee surveys. Which is best?
I generally have a preference, which I’ll discuss in a sec – but in truth it really depends on a case-by-case review of the pros and cons below that touch on aspects of psychology, normal human behavior, accuracy of memory (“recall bias“), and some basic stats. (I will get into the many types of survey bias in a later post in excruciating detail – so you might what to skip that one – I won’t take it personally.)
PROS of randomizing the question order:
- Randomizing might avoid “context bias”. Are there concerns that the respondents will get in an intellectual or conceptual “rut” in their thinking by putting certain questions together?
- It might avoid some degree of “favor bias”. If I like my manager, and all my “my manager” questions are grouped together, I might barely read the questions and give them all a “6” (“Strongly agree”).
- It might avoid “pity bias” or “let’s even things out bias”. If questions on my manager or colleague are grouped together, some people will make a calculation as to whether they are being “too nice” or “too harsh” overall. If I have answered negatively to the last 3 questions regarding HR then I have to give them SOME positive feedback right? They’re such nice people after all.
- Spreading the question groups out can mitigate “question order bias”. As we are all human, the first questions in a survey will always get more consideration on the part of the respondent than the last.
PROS of NOT randomizing the question order:
- You can now use “question order bias“ to your advantage by placing the most important questions first.
- It can make the survey a bit easier to read overall.
- Grouping the questions can provide clarity of “concept area” and “context” when there are multiple questions on the same subject. You can even add question area titles on the survey page like “About my manager…” to save them having to read it over and over, and to save the respondent the intellectual energy of having to revert back to the context. (This may lead to favor or pity bias as above though.)
What do WE do?
Though we’ve done both given the particulars of the survey and respondent group – we generally lean toward not randomizing the questions, but grouping them by driver (ex. “benefits”) or area (ex. “The senior leadership team …“)
Every survey is a little different though – and our forte is customization after all – so we ourselves might group or not group depending on what we think is best given the above.
Something I’m missing here? Totally disagree with me and want to vent? … or just want to chat on this fascinating topic? Give me a shout if you have any questions or comments.
Next up: “Employee Survey Pro Tip Series 18 – What “Demographic” questions should I use, and where should I put them in the survey?.