Select Page

You don’t need feedback from Everyone

Published December 6, 2017

A version of this post is published on Sidebench and medium.

(or how to write a participant screener for user testing)

You have a new concept for a product and then it’s all about building it so the customers will come, right? Well…

In the startup phase, you need a quick and inexpensive way to get feedback before you spend thousands of hours and dollars in building the actual product.

User testing is a design research method that puts a prototype of your product in the hands of actual people — more specifically, people who represent the goals and behaviors of the users you designed for.

Seeing how these users fumble or succeed with an early version of your product will highlight valuable insights and issues to fix — such as how the information is structured, any roadblocks they encounter, or whether the concept solves the actual problem at all.

These aren’t the users you’re looking for

Now that you have a few screens or clickable prototype, you might be going around asking your friends and family what they think. It’s likely that most of them tell you how cool it is that you’re in tech, leaving you with fluff for feedback. A few of them may be critical of the whole concept, leaving you doubtful or defensive.

The trick is to NOT look to everyone for feedback.

You need to screen out the people (and the noise) in order to efficiently narrow in on the applicable insights that will improve your design.

You need to write a screener!

What is a screener?

A screener is a survey you send out to potential user testers that screens for the people with the right qualities to bring into your user test sessions. For the most effective user test results, you’ll need to focus on people who would actually use your product because it solves some pains of theirs.

I’ll walk you through the steps to write that screener — which is usually hosted on a survey platform such as Google Forms, Typeform, SurveyMonkey, or through specialized recruitment platforms such as UserInterviews.com or UserTesting.com, etc.


Step 1 — Refine your requirements

Review your product’s user personas and requirements — so you know who you want and who you don’t.

In an example scenario for a music discovery app, we want users who are:

  • 18–35 years old
  • Browses apps like Youtube, Spotify, Google Music, Itunes, Pandora
  • Spends 1–2 hours a day finding new music

Step 2 — Start broad and narrow down. Don’t use leading questions.

User testers are usually paid for their time. That incentivizes some testers to pass as many screeners as possible, by selecting answers they think you want. In order to best screen out people who do not fit your requirements, we will have to start broad.

For example, is question A or B easier to guess the desired answer? [brackets denote whether to accept or terminate the user and is not shown]

A) Are you interested in music?

  • Yes [continue]
  • No [terminate]

B) What activities have you participated in the last week (select all that apply)?

  • Exercised
  • Cooked
  • Played Sports
  • Arts and Craft
  • Played/browsed Music [continue]
  • Played Video games
  • Browsed News sites
  • Outdoor recreational activities (ie. camping, etc)
  • None of the above [terminate]

Your screeners will need to stay away from leading questions, where an unqualified participant can easily guess the answers.

After users select questions, you can narrow it down even further by asking clarifying questions:

You indicated that recently “Played/browsed Music.” How often do you play or browse for new music?

  • Not at all [terminate, all else can continue]
  • 0–2 hours per week
  • 3–5 hours per week
  • 6–8 hours per week
  • 9+ hours per week

And so on…


Step 3 — Filter out by behaviors instead of collecting demographic information

As you can read in the line of questioning, the screener is used to filter out participants instead of collecting data. User data should have been collected beforehand to form the user personas. If narrowing down by demographics is important, use questions with multiple options so that the participants cannot guess through your screener.


Step 4 — Be quick to follow up

You don’t want to miss out on scheduling the user tests with those that do fit your requirements. Send an email with information for:
  • What the study is about
  • What the tester is expected to do and for how long
  • What the compensation is
And a few time slots that you have available — I use MixMax, a free gmail plugin, to insert calendar time slots directly into the email. The quicker you’re able to collect test data, the more you’ll know about how your product would perform with actual people.

Side note, it could be beneficial to have non-users review your product

In some cases, reviewing your product with some non-users may also yield valuable insights:
  • If they are related industry insiders or veterans, such as doctors reviewing software that their receptionist (the actual user) will use to handle patient paperwork
  • If they are a subject matter expert offering specialized feedback, such as back-end data architect reviewing whether your prototype’s structure is technically feasible
  • If you are really dying to know what your friend’s dog-walker’s twice-removed cousin’s mother thinks (just kidding)

Recap & Resources

Step 1 — Refine your requirements Step 2 — Start broad and narrow down. Don’t use leading questions. Step 3 — Filter out by behaviors instead of collecting demographic information Step 4 — Be quick to follow up

Other resources on screeners:

Written by

Dianne L Chen

I make healthy eating simple @aricoachapp — the Mindful Meal Planner for workout goals. Previously scaled design teams @sidebench & @GLOLeadership. USC ✌︎

No Results Found

The page you requested could not be found. Try refining your search, or use the navigation above to locate the post.