Civic User Testing Group (CUTGroup): Remarks at Code for America 2015 Summit

Today, I will talk about the Civic User Testing Group (CUTGroup) during the 2015 Code for America Summit. In 2013, Dan O’Neil presented at the CfA Summit about the CUTGroup as a model for changing the relationship between government and residents. Since that summit two years ago, we have doubled the number of CUTGroup testers from 511 to over 1,000 testers, we have tested sixteen websites and apps, we have expanded to all of Cook County, and we continue to add processes to engage with people in the CUTGroup. The work is never done.

I now run the CUTGroup project for Smart Chicago. Here are my thoughts about how not only to run a CUTGroup, (we lay this out in detail in the CUTGroup book and blog posts) but how to sustain a CUTGroup by leading with community engagement.

The Civic User Testing Group (CUTGroup) is a new model for UX testing, digital skills, and community engagement. What makes the CUTGroup the CUTgroup is the merging of these three components into one experience. To build a CUTGroup, you need to devote time to all these components equally.

The CUTGroup is a central program for Smart Chicago because it cuts across our three areas of focus: access, skills and data. Access: we conduct the majority of our tests in public computer centers and libraries in the community. Skills: for the tester who is introduced to new technology, and for the developer who learns ways to design tests and engage residents. Data: we help improve existing technology and encourage the creation of better technology.

The CUTGroup is the community of people in Chicago and all of Cook County who come together in libraries and public computer centers to test and have conversations around technology. In Chicago, we have testers from every part of the city, all 50 wards and all 77 community areas. We are now reaching the rest of Cook County.

These residents are paid to test websites and apps to help create better technology. We pay every CUTGroup member $5 for signing up, and then $20 when they participate in a test. As of today, we have done 19 tests that cover a wide range of topics such as schools, transportation, social services, neighborhood information, and more.

The motto below is key to this work. It removes the idea that residents do not understand technology as well as technologists. Instead, it permits testers to participate, give their feedback and show that their ideas are a valuable part of the process.

We talk about the methods and processes in the CUTGroup book to help other cities run their own CUTGroup.

Asking how to start or run a CUTGroup is important, but I think the question below is more important.

The answer is easier than we think.

So, when is it not a CUTGroup?

It’s not a CUTGroup when you are only testing civic apps. In other words, it’s not a CUTGroup if you put the tech first. The “civic” in Civic User Testing Group has to describe the group before it describes the user testing. At Smart Chicago, we are beginning to move away from the language of “testing civic apps” because there are more important criteria in determining what to test. We determine whether or not to test technology based on these criteria:

Interest and desire to do CUTGroup testing and talk with residents (commitment to be part of the process!)
The technology reaches a large and diverse group of residents and could have or has an impact on their lives
Willingness to listen and then respond to the feedback and make changes

Not everyone who seeks CUTGroup testing might see that the tech they created is a “civic app,” and that is ok.

A CUTGroup needs to lead with engaging with people around technology and that becomes easier when you know that people want to help make tech better. They want their voices heard and they know so much about how they use technology and how they want technology to work for them.

Invite

The first step to engaging through UX testing is inviting and recruiting people to join the CUTGroup and recruit from inside your networks as well as outside of it. It is crucial to open the CUTGroup to everyone and, then once they are in the group, to build one-on-one relationships with new testers. That way, no matter how they were recruited – by a friend, from a flyer in a library, from an organization – they are on the same level with all of the testers in the group. It does not matter how they get to you, but how they are included in the CUTGroup experience.

I cannot emphasize the statements below more. Recruit anywhere and everywhere. If we only recruited from one place or one group, or stayed with the same method of recruitment, we would miss so many people who want to join and participate. The tech we test cannot define who is in our CUTGroup.

Here are a couple of very specific ways that we try to be inclusive in CUTGroup.

First, we use physical gift cards because it’s the closest thing to cash that we can get. We want to give testers something that they can use anywhere they normally would shop. Even though physical gift cards cost more than digital gift cards we use them because they are easier to use and can be used anywhere. We also cannot assume that people shop online in their normal day-to-day.

Recently, we incorporated text messaging into the CUTGroup process to reach people who do not have regular access to the internet. Out of our 1,000+ CUTGroup members today, 29% of our testers said their primary form of connecting to the Internet is either via public wifi or their phone with data plan. Testers will be able to sign up for CUTGroup and receive text notifications when new tests come up and being to respond to participate.

Ask

The next step centers on how we communicate with our testers and how we keep in contact with them. First, we never share information with our CUTGroup testers about anything other than CUTGroup. This maintains a relationship that only centers on the program that they signed up to be a part of.

We keep regular and open communication about upcoming tests, and explain the reasons behind why or why not we picked them for a specific test. If they are not selected for a test, we still want to share what the test was about so that they feel included in what is happening and can still interact with the website on their own.

When testers sign up for CUTGroup, we do not gather information about demographics, and prefer to ask questions that relate to their devices and the ways they connect to the Internet. This keeps our sign-up form simple and easy to fill out. Also, we are more interested in learning about the tester’s non-technical and technical experiences, and we value learning about these experiences more than gathering demographic information. We do not have a database of testers’ age, race, income levels, or education. These things are just not as important.

We do design the screening questions to gather better information about testers for that specific test. If necessary, this is when we capture information about demographics.

This allows testers to voluntarily give us more information. If they do not feel comfortable with answering the questions for a specific test, they do not have to, and they can still be part of the CUTGroup. Letting people choose how they interact with us is important and we want to feel comfortable in these interactions.

Listen

When it comes to the CUTGroup test, it is hard to listen and leave expectations aside. When you bring in developers and project managers, they have their own assumptions of how the test will go (and so will you). However, the unexpected solutions that testers come up with can sometimes be the most valuable part of UX testing. It is important to step back, listen, and not tell testers why they are right or wrong or what they should have done. This is hard because we want to help if they are getting stuck. What is better is to really focus on understanding how they are doing the task you asked of them.

We generally test in branches of the Chicago Public Library because it allows us to visit new neighborhoods and reach new people. We are very lucky to have such a great resource in the Chicago Public Library since there is a library in every neighborhood. It would be really easy for us to test in our own offices, but meeting people in their community allows new people to participate in testing. We love going to them.

It is important to talk about the person’s experiences not just about how they use technology. We want to ask how testers get information, how they do things in their normal day-to-day, and how they see technology fitting into those experiences.

Once testing begins, we keep asking questions to learn more about what testers are doing and their expectation. We ask them why they clicked on that button/link or why they are using that feature. Their responses help us gather insight not only about the tech being tested but also tech in general. Their UX experience is influenced by other tech they know and learning about those expectations helps understand how to build tech better.

We also always ask testers for very specific ways to improve the website or app. “Tell us anything and everything you can” is a phrase I use often because I never want testers to feel their improvements are too big or too small. I want to hear all of their ideas. We consistently ask this question across tests, and it gives testers to contribute to the creation.

Another big part of our work is teaching developers how to incorporate user testing in the development of their technology. By requiring that the developer be part of the process, we put them in front of testers to see and hear for themselves how real people would use their tech.

Respond

The last piece of engaging with the CUTGroup is making the changes that the testers suggested.

Once a test is completed, I spend a lot of time working on the analysis. I take a look at each question and provide quantitative and qualitative data. We want to see how easy it was for testers to finish a task as much as we want to understand their expectations of the technology and its functions. We also want to learn about whether or not they think this website is for them, if they like it, and if they would use it again.

The last step is the hardest, and not always completely in our control, but my goal is always to find better ways to help developers make changes on their website or app. This is a process of collaboration and there needs to be a commitment made at the beginning of the process that the developers, project managers, or organization staff will make some changes based on the conversations they had with CUTGroup testers.

After that happens, I want to show testers what they helped to create and show them that we listened.

It all comes back to making change happen because if we are inviting, asking, listening, but not responding — the test is hollow. We are not respecting our testers and their suggestions and it comes full-circle to our motto. If it doesn’t work for them… it doesn’t work.