Sending Customer Development Surveys
A friend of mine just asked me for some advice on sending surveys. This is the list I came up with.
Sending surveys is an important part of early customer development; it helps you test a hypothesis and delivers you “perception” data. You can track how a user interacts with your service, it’s harder to track how they perceive it without surveys.
Early on in development of a consumer facing product, I’d recommend sending out simple surveys at short intervals (1-4 weeks) to a subset of your userbase. Below is the advice I gave my friend, if I’m missing anything, please leave it in the comments (Hattip to Hiten Shah, Leonard Speiser, and Sean Ellis who heavily influenced my thinking on this through tweets, posts, and conversations on this topic).
Hey,
First thing is too check out this: http://venturehacks.com/articles/measure-fit (not shocking that Venture Hacks has the go to resource, is it?)
Then check out the tool that video is about: www.survey.io
But here’s my advice:
- Order your survey intentionally. Use early questions as eventual filters. If your second question is “how bad would you feel if you couldn’t use this product” that helps you sort later questions (i.e. my power users think this is the key feature, everyone else thinks it’s something else).
- When evaluating the data, you don’t want to optimize for the largest segment, you want to optimize for the segment that’s most engaged.
- Don’t ask any questions without understanding how you’d apply the data you’re collecting.
- Ask some open ended questions. The open-ended everything survey recommended in the post is a great way to start. But I like to have less than half my questions require typing — and it’s usually just an “anything else you think we should know?”. You get much higher response rates. But, there’s definitely a time and a place: open-ended questions are really useful for messaging questions and for early “discovery” surveys. Those questions also allow you to learn a lot more about the user (and how committed they are — you can tell a lot from the length and quality of their response).
- Ask for the ability to followup by phone, and do phone followups with every person who says yes. You’ll learn a lot in that conversation — and you’ll develop deep relationships with potential customers.
- On the subejct of developing relationships, provide an option for opting-in to the “elites” club — let them self select into beta testing groups. These elites can often become marketing assets. Yelp did a great job of this. David Barrett at Expensify is also doing this well right now. Survey’s aren’t just data, build a marketing asset.
- Ask for their reactions to the product, optional and freeform (limit text length if you would like to use in materials or on twitter). Ask “can we use this for marketing purposes?” Another idea: followup and get a small photo, name, link, etc. — use these assets to personalize the testimonials when you put them online.
- Don’t put explanatory text in front of questions. It’s tempting to try and put people in the right “frame” — it hurts you in the long run. Don’t alter the answers they want to give you.
- Short surveys win. <10 questions. <5 mins to complete. half that is much better. Don’t write an SAT test.
View Comments, Comment or Ping
Recurly
With the crucial importance of metrics in everything from product design to brand management, these tips are awesome. It's not enough to just throw together a survey with the same questions everyone else is asking their customers. The questions _must_ provide data that drives real _action_. Eric Ries had a great post on this (and avoiding distracting “vanity metrics). You can find it here: http://bit.ly/6O5MCb
In our surveys of our beta users, we kept most of the questions short and to the point (most multiple choice but we allowed for optional input). We were excited to see such a high response rate- and the input helped us smooth a rough spot out in our design flow.
Good luck out there, and good testing! :)
Jan 5th, 2010
expensify
Great suggestions! I'd add the following:
- Make it easy for them, even if it's hard on you.
- Don't gather data, start conversations.
- Let them set the topic.
In particular, though this seems counterintuitive and against pretty much everything you'd read: I'd strongly, strongly recommend *against* sending out surveys. They have so many problems, some of which include:
- They typically require you to click a link. As any web developer knows, people are loathe to click links; assume at all times that no matter how valuable you think a link should be, 90% of people just won't click it. Assume you have a very tiny “click budget”; I find getting two clicks out of someone (“Reply”, and “Send”) is hard enough.
- They're impersonal. The very notion of being surveyed is de-humanizing. It conveys “you're just a datapoint to me”. Instead, send your questions from a real person, with a real name attached, in the actual language you'd use as if you had just naturally written them a personal message and are genuinely interested in their response. Because if you're not genuinely interested, that will be apparent and people won't bother.
- They're optimized for the receiver, not the giver. The whole point of a survey is to normalize data into a way that is easy to read. It makes the giver do all the work in parsing messy real-world concerns and confusions into a multiple choice bucket. The real world is too complex for that; it's actually *harder* to fill out a survey than to write a multi-page email describing your problem, because you're never forced to choose between options that don't quite fit.
- They only tell you what you ask, but that's rarely what you need to hear. The only filed in the survey that really matters is the open-ended one; everything else is a distraction.
- They get out of date. Surveys take time to create; the questions you want to ask change over time. Even if the survey was perfect for you yesterday, it probably won't be today. Just leave it open ended and people will always tell you relevant information, without needing to update the template.
Anyway, I swear by the “just send an email and let them do the talking” approach. It's worked incredibly well for me. And it's a lot more fun to talk with users directly than to just stare at charts and wonder “is this data even statistically relevant?”
Jan 5th, 2010
Tyler Willis
Anyone who doubted my initial comment that you are killing it in this field is officially rebuked :)
I agree, I find direct email communication with people to be a fantastic way to do in-depth development. I wish I'd done more of my early development surveys at Involver in this format (I also wish I'd done more of them in general!). Do you send these types of emails to all users? How do you decide who to send these to?
I still use surveys often when making a specific choice (i.e. how do we prioritize building feature a against building feature b) and dealing with a sizable userbase. I love surveys here because compiling email responses into a sorted data set gets VERY time consuming at scale; it can often add days to your decision-making time.
Food for thought: embedding survey-style questions directly into your product could be a great way to get well sorted and actionable data without wasting your “click budget.”
Jan 5th, 2010
expensify
*Awesome* idea on putting the survey inline to the product. For example, we currently export to QuickBooks and are trying to decide which one to add next; adding a “Where else should we export to?” question right there sounds like a brilliant idea.
As for who to send and when, we actually built up an automated system that is triggered based on certain activity signatures. Any user who hits that pattern will get the notification. (I'm about to add an improvement to only send each notification once per user, in case you hit the same pattern multiple times.)
It's actually taken a lot of trial and error to refine, but the results are well worth the effort.
Jan 5th, 2010
Lance Walley
We did a short survey early on to help see what kinds of businesses our customers are running or planning, so that we can meet their needs.
We asked 4 very simple questions, plus 1 free-form text comment field. The email was short, personal, and invited them to email me directly if they wanted to.
Response was fantastic! 27% filled out the survey, about one-third of those filled in text comments. A few emailed me.
I followed up each one with an email myself, whether or not they asked, and that led to many great email and telephone conversations.
Communicating with customers and “near customers” is really enjoyable and educational, not to mention something they really dig, too!
Don't over-survey and keep your surveys short and sweet (and useful, of course). Make the email that leads to the survery personal and also short & sweet, you much assume the recipients have a busy day!
Jan 5th, 2010
Ben Casnocha
The only question that matters: http://www.amazon.com/Ultimate-Question-Driving...
Jan 6th, 2010
Reply to “Sending Customer Development Surveys”