I’m often asked about the demographic makeup of our CUTGroup testers, specifically around age, income, and race. The fact is we never collect demographic information when testers sign up, and the reason is because it’s just not that important.
In all of the CUTGroup tests we’ve conducted, we have always seen a very diverse group of testers participate not only in demographic characteristics, but also technology experiences. We value these experiences over demographics especially in understanding how the technology we are testing works and fits into a person’s life.
In the CUTGroup, we have created a simple sign-in process to be accessible to the largest group of people, we use screening questions to find testers with specific characteristics relevant to the tech we’re testing, and we ask in all tests about both online and offline experiences.
Sign-up process
When testers sign up for CUTGroup, they fill out a simple sign up form either on our website or through text message. We do not gather information about demographics, but instead ask questions that relate to the devices they own and the ways they connect to the Internet. We use this information to make sure we test on a range of devices, and sometimes segment by device type (Android vs. iPhone).
Having a simple form online to join the CUTGroup is the most straight-forward and accessible way anyone can join. We added a text message sign-up process so people who do not have regular access to the Internet could also easily join. Adding demographic questions to the sign-up process is not only unnecessary at this point, but it would make some people feel uncomfortable to answer those questions and possibly end up not joining.
Screening Questions
For each test, I work with the developers, project managers, and organization staff to determine what are the best screening questions to ask. Sometimes we ask questions to segment further such as for the Chicago Public Schools website test we were looking for CPS parents, but then asked additional questions such as school type and grade levels to get a diverse group of testers. Other times, we will ask how people typically get information to learn whether or not they use other websites or if they rely on other offline resources.
If necessary, we use the screening questions to capture demographic information if it is relevant to the test. In the mRelief test, for example, it was required to ask demographic information because we were interested in testing with residents who were currently enrolled in social services or could be qualified for social services.
By asking demographic questions during the screening phase, this allows testers to voluntarily give us more information. If they do not feel comfortable with answering the questions for a specific test, they do not have to, and they can still be part of the CUTGroup. Letting people choose how they interact with us is important and we want our testers to feel comfortable in these interactions.
Test Questions
During a CUTGroup test, we like to start the test with some background questions that go into more detail about the person’s experiences around a topic. We use this as a way to gather more information about the tester, but also start the conversation between tester and proctor.
In our test of Chicago Cityscape, a website that “makes neighborhood, property, and construction development data accessible to all,” we asked questions like about whether or not they’re interested in what is being built or torn down in their neighborhood and how they find out about those changes. Or how important certain pieces of information (building/zoning information, permits, or tax information) are to them when learning about a property?
These questions types give us context for the rest of the test so we better understand the relevance of the technology to the tester and if this is something that works for their specific needs.
Examples
Here are a few examples of what we learn of our testers through all processes of a CUTGroup test.
Slide from Sonja Marziano’s OpenIndy Brigade Presentation
In April 2015, we tested the Chicago Public Schools website to better understand how their redesign to be more mobile friendly was actually working. We tested with one parent, “Meekmeek” who has lived in Chicago her entire life and has one child in Kindergarten. She never used the CPS website before, and generally relies on other parents, social media, and parent/teacher conferences to get school information. This test was the first time she interacted with the CPS website before and she tested on her Android HTC Desire Fire 10 Boost Mobile phone, which is how she typically connects to the Internet. During this test, she experienced very slow load times on some pages, and decided to download PDFs instead of waiting. She also experienced trouble on her device using the school locator tool, which later was fixed by the CPS development team.
Slide from Sonja Marziano’s OpenIndy Brigade Presentation
In July 2015, we tested the Ventra transit app to help the Chicago Transit Authority (CTA) team find issues and make changes before opening up the app to a larger beta testing group. One tester, “Blind transit rider,” gave invaluable feedback about tags and labeling of this app so that it could be more easily used on screen readers. “Blind transit rider” uses public transit multiple times a day on CTA busses, trains, and PACE. She uses an iPhone and voiceover commands to navigate websites and apps. She was excited for the Ventra app because it provides her with “independence” while on the go.
Final thoughts
The success of the CUTGroup is that we continue to have a diverse group participate in our tests without segmenting by demographics. This would not be possible if we didn’t try to invite everyone. We believe that everyone is “right” for the CUTGroup regardless of the technology we test because building community is the first priority, and testing the technology comes later. Naturally, we build a CUTGroup that is diverse by being accessible and providing opportunities for people to participate without excluding them based on demographic information. At the end of a test, and over many tests, we continue to build a more complete picture of the tester by learning things about who they are, what they do, and how they use technology in their everyday lives.
