Surveys That Work: A Practical Guide for Designing Better Surveys

by Caroline Jarrett

Surveys That Work explains a seven–step process for designing, running, and reporting on a survey that gets accurate results. In a no–nonsense style with plenty of examples about real–world compromises, the book focuses on reducing the errors that make up Total Survey Error—a key concept in survey methodology. If you are conducting a survey, this book is a must–have.

Genres: Design, Nonfiction, Research

368 pages, ebook

First published July 1, 2021

Reviews

[Annas Jiwa Pratama]

I occasionally do surveys and often gets questions from my colleagues on how to do them. Surveys are tricky. I feel like it’s a polarizing method: a non-insignificant number of professionals trust it too much, or they avoid it at all costs. And perhaps within the portion that trusts it, some takes this method too, I guess, unseriously, or take the research process for granted? That produces a lot of surveys that feels off, or reports that are difficult to read. I know I’m guilty of more than a few of those.

In the past few weeks, I’ve been assaulted by the nagging feeling that I’ve been kinda winging it. My answers and decisions often felt like it was increasingly based off guesses and intuition, rather than something solid. There was reasoning, sure, and maybe good ones. But regardless of how good they were, I can’t shake the feeling that I was losing my foundations.

Thus, I start my quest to relearn this craft from different angles. I found this book as a recommendation from this talk, and I feel that it was an excellent find. The book had mainly two very good things going for it: its focus on a central framework and its excellent final chapter on what to do with limited time.

The book’s built around the framework of the ‘survey octopus’, which is basically a kind of hierarchy based on three major aspects of a survey research (the research goals or questions, the sample, and the results), broken down into several choices we need to answer while planning and conducting it. At a glance, it’s bordering on a Science Diagram that Looks Like Shitposts, but it makes sense. (the gaps between the arms are where errors occur!). Each chapter refers to this invertebrate, which tentacle it represents. I feel like framing the whole book like this is important for intuition-building, and the octopus itself is a pretty good anchor to workshop or scrutinize your own work.

There's a lot of really good introductory-level chapters on a lot of the specifics of crafting survey research. (I dig Jarrett’s chapters on writing questionnaires and building it for her emphasis on testing. I’m a fan of cognitive interviewing but its difficult to squeeze it into tight deadlines or scant resources, so it’s a nice reminder/reorientation.) Jarrett ties all of them together in her final chapter: The Least You Can Do. What I don’t think I usually see in curricular handbooks on research or surveys is this practical example of timelines, and a concrete guide to plan your survey end to end. Here, Jarret breaks down a whole survey project within the confines of three different time limits: an hour, a day, and a whole week. This is kind of a smart way to summarize everything she had written in terms of what we usually care the most in these situations: what to prioritize, what can we skimp, and how do you trim. I’ve never had the misfortune of having to make a survey within a day, much less an hour, but glad to know I’ll have something to hold on to if/when my luck runs out.

Tangents
- This is the last read of 2022. Glad it was a good one. Definitely gonna make a summary doc for myself within a few weeks!
- Makes me think of wanting to double down and pursue study in methods. And reminds me of how cool papers about stuff like “what is the optimal number of options for likert scales” are, or “can we make cognitive interviewing fast and cheap?”. They seem fun and very practical.
- My favorite watches in 2022 are Turn A Gundam and Andor. Loran is such a good boy.

[Audree Fletcher · Rating 5 out of 5]

As a research method, survey comes with an inherent truthiness that makes us inclined to believe surveys irrespective of their quality. This means people can find themselves basing important decisions on shonky research - so in the wrong circumstances, a poor survey can be more than just a waste of time and money, it can be dangerous.

Fortunately Caroline’s book is an easy-to-read, step-by-step guide to better surveys - from figuring out your research goals all the way through to results reporting.

My favourite parts of the book, though, are the little extras in there. Caroline provides a solid critique of some established gold standards - NPS, statistical significance, even Likert scales.

And I love the checklists and acceptance criteria, along with the pragmatic guides for survey improvement (sized for every time-budget). These really reduce the book down to its most actionable essence, and turn it from a book I’ve learned loads from into a reference book I’ll refer back to regularly.

A great investment for anyone without a quant research background.

[Chris Sharp · Rating 5 out of 5]

There's a moment nearing the end of this clear, concise primer on survey methodology where Caroline makes the case for moving on from that anachronistic topic/subtopic way of presenting your survey information towards a model described as assertion/evidence. I guarantee you've seen it at least once before; if you're like me, you were probably wowed by it and wondered what made the idea presented so clear.

Well, it was at that point I realised this whole book was written in this way.

It's definitely influenced the three surveys (running from Light Touch to Big Honkin) I've conducted since I started reading it, and it'll be my go-to reference recommendation for anyone in my team looking to branch out into survey use.

I also appreciate the evidence base under it; how it has been written not to let overreferencing get in the way of the points, but allows for someone to perform further reading if interested.

[forissima · Rating 5 out of 5]

Awesome book, tons of practical examples and suggestions for more reading.

[Alyssa · Rating 5 out of 5]

Easily the best book I've read for work. It's a little silly in the right ways, it makes so much sense of struggles I've had and why certain things work well in survey design. And I truly learned so much and was able to immediately make a better survey. I am so glad someone recommended this book. It's top tier. And every book rec in it is also good so far! Absolutely worth my time. If you need a survey book, this is the one. Especially if you've read a bunch of articles and don't feel like your surveys are going any better. I even went to a bigger city library to borrow it and bought a copy for myself, and I complained very little once I started reading it Jarrett's website is a helpful companion too! Lots of good stuff. I had minor issues with it, but nothing that can't be solved by looking into her book recs and using the language I now have to better research specific topics. I will say the octopus diagram is goofy and only endeared me to it more, but there might be things like that that don't speak to you. But I recommend giving it a shot if you need this sort of resource!

[Duncan Bell · Rating 5 out of 5]

I love this book. It has taken an area of user research that I had little or no “real” knowledge of, but quite happily still did; and made it something I feel I can do well and get great insights from.

In case you think this is too much of a tome, I’ve used it effectively at two start-ups - places where we were time short and needed to learn quickly. The book is well structured and easy to access, meaning you can find the advice you need quickly (for example, today I was looking for some tips on introductions). But the overall method is what brings the most value AND if you’re really time short, chapter eight has step-by-step what to do when you’ve got a day / a week schedules.

Overall, this is brilliant as a comprehensive guide to a method you can use time and again. I will never not user test and pilot my surveys!

[Yong Heng · Rating 5 out of 5]

A survival manual if you need to grasp survey research quickly. But don't expect explanations for hypothesis testing.

[Camille · Rating 3 out of 5]

This is a useful introduction to survey development, but readers may benefit from supplementing it with (at least slightly) more technical books.

[Angie Helgeson · Rating 1 out of 5]

I hated it. I couldnt even make it through the last chapter 'the least you can do'

Now I need to figure out how to get rid of this.

[ichbinkreativ · Rating 2 out of 5]

Better structured and more concise: “Surveying the User Experience: Design and Analysis of Surveys for UX and Customer Research” by Jeff Sauro, James R. Lewis