Roger Hart

Technical Author - Red Gate Software

Survey design: just try to write clearly

Published Monday, January 11, 2010 11:45 AM

Most surveys suck. Think about how many you've seen and just sighed, or clicked away from in frustration, or struggled to understand. They suck. Partly, this is because survey design is hard - requiring sound statistical and research methodological experience, and a grasp of information design.

Partly. You can make surveys suck a whole lot less by doing two really, really basic things:

  • Think about how your user thinks, and if possible don't make them do it at all
  • Sort out the writing

The first one is the really, really important one. The second one is just getting it across.

Ask a silly question

Yes, it's a pathological example, but it's quite closely based on a survey I came across last week.

The survey designer here clearly wants nice, clean, quantitative data about users' creature nibbling predilections. Who wouldn't. Problem is, they don't seem to care about how much cognitive load they impose to get it. Most folks don't have quantitative breakdowns of all their behaviours on the tips of their tongues, and in this case they're being asked for one in a madly counterintuitive way. It takes too long to parse the question, much less answer it.

I'm going to take a wild guess and suggest that this kind of thing typically happens when people lose sight of a vey well-worn usability cliché: remember your user isn't you.

For surveys this means the information you want isn't necessarily something you can ask them for directly. They are likely to understand things differently, and almost certain not to share your grounding assumptions.

Terminology

Make sure you're phrasing things in a way the users think about them. (Why are the number of legs important, and what does that mean about implicit categories?). For instance, being British, I might be expecting a totally different kind of answer to most Americans if I asked a question about "football". More usefully, do the way your users experience your business, and the way you talk about it match?

Here at Red Gate, two of  our business divisions are SQL Developer Tools and DBA Tools. If we asked somebody, say, "What is your most used Red Gate DBA tool?", they'd be well within their rights to answer "SQL Data Compare". That's a fairly toothless example (and the "wrong" answers might themselves be interesting) but you get the idea. 

Use their terminology over yours. And if that means you have to do some thinking at the end, tough. If the question isn't instantly easily intelligible, you'll get lousy data.

Again, thinking is your job, not the users'.

(As a side-note, psychometric tests are a fascinating counterexample. Many are riddled with weirdly obscure assumed categories, requiring you to decide if you're more "Mauve and insouciance" or "goal-oriented and butterscotch". Personally, they just make me incredibly angry. Presumably some subtle wizardry is going on)

Instructions

We've got a smug adage in technical communications: if it's hard to explain, it's probably hard to do.

A survey question that runs into multiple sentences is probably badly-phrased or overloaded, or both. If it really needs to be that complicated, make sure you use line breaks, and every trick you know to make it easier to read.

In our bad example, "Choices  must add up to 10" is foregrounded, it's the first thing you read. Which, for a second leaves you with no idea of what choices or why. It's just in the wrong place - its predicate hasn't yet been established. It's also by no means the most important part of the instructions.

In practice, you'd re-write the whole thing:

It's not quite the same question, sure, but it's massively simpler, and it's phrased in terms people can easily visualise.

Also, concrete units that readily relate to users everyday experiences make questions much simpler to answer. If we'd asked them, for instance, to estimate goat consumption in tonnes per financial year, or assign twenty-two billion points rather than ten, we'd have lost them even more.

Quick wins

Sometimes I edit surveys. The big scary stats stuff isn't quite my field, but this is what I do to the words:

  • Paragraphs and line breaks
    These are critical for ease of reading. People skim web text, so don't make it dense. Most of what I do when somebody asks me to edit a survey is breaking up the text.
  • Don't say please
    Get right to the point in the fewest words possible. Avoid "Please supply your email address so we can contact you". "Email (optional):" will do.
  • No maths
    If you must make me think, don't make me add up. Try and re-frame the question.
  • Keep it on one page
    Or few, rationally grouped pages.
  • Keep it short
    Every extra question raises the dropout rate. It's probably also worth keeping each page's content above the fold.
  • Make sure questions aren't leading (unless they need to be)
    You can get some real gems from those "Any other feedback" style free-text fields. But you mostly get nothing. It can be worth adding some gentle suggestions.

Any other tips for making surveys less soul-destroying?

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Brian Harris said:

I eat zebra twice-daily. Where's the option for that?

Nice blog, though, Roger.

BTW, on another point of usability, I wish the "You need to sign in to comment on this blog" message contained a hyperlink to the sign-in page. Don't make me scroll to the top of the page and find the option!
January 11, 2010 6:36 AM
 

Roger Hart said:

There is no curse in Entish, Elvish or the tongues of men for the loathing I feel for this blogging platform, it's true.

I'm trying it with anonymous comments enabled, but requiring approval.

The problem is having to manually click through around four thousand spam posts.
January 11, 2010 6:39 AM
 

Phil Factor said:

I tried to enable anonymous comments for my blogs, and loved the genuine comments I got. Then, the sheer stupid work of taking out the spam got to me in the end. I ended up disabling anonymous comments entirely. Hell, the subscription gives you a free Ebook every month. It is like forcing people to accept Christmas presents. Tony and Richard are currently working on a big update of the blogging platform.

On the subject of surveys, I tremble at the thought of the problems that real psychologists and sociologists have in getting meaningful data from a survey or a psychometric test. The process is a minefield, and you are faced with a number of effects that are going to guarantee that the results are going to be, at best, misleading. The idea that anyone can create a useful survey whilst remaining innocent of the many hazards is as sensible asking for a volunteer in the local pub to take your appendix out.

There are suppliers of Zebra meat in Britain. (from farmed stocks). It is worth persevering with finding a good supplier  because Zebra is very lean, low fat and tastes delicious. Zebra has a very light flavour for a red meat.
January 11, 2010 11:48 AM
 

Roger Hart said:

It is indeed delicious. There's a pub here in Cambridge - The Geldart, on York st - that serves Zebra occasionally, as well as other more novel meats. They have aided me greatly in my quest to chew my way through the Earth's legally edible fauna.
January 14, 2010 10:37 AM
 

timothyawiseman@gmail.com said:

One thing that annoys me is questions that seem unrelated such as:
"Have you seen someone wearing glasses in the last week?"

I understand that they are supposed to be screeners to weed out people not taking the survey seriously.  I can see how some form of weeder for people just answering randomly is needed for serious psychological research, but I would hope they would only be used where actually needed.  Even when needed, I would think that using something more subtle and that at least appeared relevant to the survey would be more useful and less insulting to the users.
January 14, 2010 5:55 PM
 

uberVU - social comments said:

This post was mentioned on Twitter by RMH40: New blog post - Survey design: just try to write clearly http://digs.by/19kf
January 19, 2010 9:42 AM

What do you think?

(required) 
(optional)
(required) 


















<January 2010>
SuMoTuWeThFrSa
272829303112
3456789
10111213141516
17181920212223
24252627282930
31123456
Microsoft Office Communications Server 2007 R2 – Part II
  Once you have set up Office Communication Server 2007 R2 to provide IM within the rganisation, the... Read more...

Mission Critical: Database Design
 There is nothing like a checklist to make sure you've completed all the tasks in designing a database,... Read more...

SQL Server Intellisense VS. Red Gate SQL Prompt
 Fabiano Amorim is hooked on today's Integrated Development Environments with built-in Intellisense, so... Read more...

Doug Crockford: Geek of the Week
  Doug Crockford is the man behind JavaScript Object Notation (JSON). He is a well-known critic of XML... Read more...

Raw Materials: Mirror, Mirror, on the Desk
 Seeing ourselves as we see ourselves. Read more...