Last Tuesday, Smart Chicago Executive Director Daniel X. O’Neil spoke at the OpenGov Hack Night about the Civic User Testing Group.
CUTGroup
The Launch of the CUTGroup Book
Today we’re launching a website and book, The CUTGroup Book, http://www.cutgroupbook.org/, centered around propagating the components, tools, and methods of our Civic User Testing Group (CUTGroup) program.
The CUTGroup is a set of regular Chicago residents who get paid to test civic apps. We started the program in February 2013 and it has grown to include 800 people. We’ve learned a lot and we want to share.
We cover the hardware and software you need, methods for tester and developer recruitment, test design, location scouting, and results analysis. We show detailed budgets, exact website configurations, complete text of recruitment emails, the raw results of every test we’ve conducted, and all the other nuts and bolts it takes to make a CUTGroup in your city.
You can download a PDF of the book for free or order it as a real book on Amazon. Also read about the contributions of dozens of people who made this thing happen.
All hail the people of CUTGroup!
Gift Cards for CUTGroup
The Code for America Chattanooga team recently expressed interest in learning more about the operations of CUTGroup to begin their own civic user testing group. Smart Chicago is dedicated to openly documenting our work for everyone to use. The Chattanooga team reviewed our documentation, we talked further and realized we were missing information about gift card mechanisms. In this blog post, we will share what we learned about gift cards.
Once a resident signs up to be part of the CUTGroup, we send them a $5 VISA gift card. If and when they are chosen to test a civic app, we give them a $20 VISA gift card.
When purchasing gift cards for your program, there are three main considerations: type of cards, cost and fees associated per card, and quantity and expiration date.
Types of Gift Cards
There are different types of gift cards out there – prepaid VISA of Mastercard gift cards, store-issued, bank-issued, online, etc.
We considered Amazon gift cards as an option; these gift cards do not have card processing fees, no expiration dates, you can choose small face values (as low as $0.15), and they can be sent directly from e-mail. This is a very convenient option, but by choosing this service we would assume that CUTGroup members have regular access to their e-mail, know how to use Amazon, and know how to access and use the gift card for their purchase. If someone is not an Amazon user, this could be a barrier to joining the CUTGroup. We want to include everyone, and this just does not work.
Other gift card types also include specific store- or bank-issued gift cards. Sites like ScripSmart can provide comparisons between gift cards, and can give you an idea of what you need to ask about before purchasing your own cards. Again, our goal is to provide the most flexible currency possible, so this does not work for us.
We purchase VISA gift cards specifically because they can be broadly used in different locations. It is important that CUTGroup spans all different types of residents in Chicago, and these gift cards should fit in the normal course of these residents’ lives. By purchasing these cards, we are spending more than face value on fees, and have to take time to mail them out, but the value of accessible gift cards is worthwhile to the goals of our organization and this program.
Costs and Fees
It can be hard to find companies that offer Mastercard or Visa gift cards in values smaller than $20. In addition, there are a number of costs and fees associated with each gift card. With our first vendor, we spent around $10 for each $5 sign-up card that we sent out. Here is a list of our costs associated with a 100 card order:
- Face-value of card: $5.00 per card
- Card processing fee: $3.95 per card
- Credit card processing fee: $1.00 per card
- Shipping Fee: $21.95
- Total: $10.17 per card
The card processing fee can be higher or lower depending on the quantity of cards purchased, how you are planning to pay, and lastly the length of expiration date.
With this in mind, we researched and found an option to lower our fees and have a longer expiration date through a new vendor, Awards2Go Visa Award Card. We were able to lower the cost per card to approximately $7.07 for every $5 sign-up card. Here are the new costs associated with a 100 card order:
- Face-value of card: $5.00
- Card processing fee: $1.75
- Credit Card processing fee (1% of total order): $5.00
- Shipping Fee: $27.00
- Total: $7.07 per card
Expiration Date
Last October, we learned that we still had a lot of gift cards that were about to expire and some were already expired. With the gift cards that expired, we lost out because the cost of the fees to “restock” the cards would be higher than the value of the cards we would receive. We also had 118 $20 and 103 $5 gift cards that were going to expire at the end of November. If we sent back these cards to the vendor, we would only receive $10 for each $20 gift card.
We thought of some creative ways to use the cards including a refer a friend campaign, and a remote CUTGroup test that allowed many testers to participate. Still, some of our testers received cards late (after the expiration date), and we had to send new cards to our testers. We now have gift cards that have expiration dates set for 8 years later, with the value decreasing from the card after 13 months of receiving the cards (-$2.50/month). This gives CUTGroup participants a longer time frame to use the cards, but at the same time we still have to be mindful of the quantities we purchase to ensure we can use them before they lose value.
Final Thoughts
The success of CUTGroup operations is based on the quality of engagement with our residents and the open communication we have about our process. When our gift cards were about to expire, we told our CUTGroup members so they knew. When gift cards came too late, they e-mailed us to let us know, and we sent out a new card. These everyday conversations let us build a better community with Chicago residents around data and technology, and help us be better at what we do.
CUTGroup is an important program to Smart Chicago because it cuts across three of our areas of focus on access, skills, and data. Not only does it allow residents to connect around data and technology, it also creates meaningful communication between developers and residents. We will continue to share our processes around our programs in hopes that our experiences are useful to everyone.
The Glories of User Testing: Chicago Early Learning
In late 2012, after a short development process, we launched an initial version of our project called Chicago Early Learning.
An active regime of testing
Then, in the ensuing months, we actively listened to regular Chicago residents, dutifully noted their feedback, and directly changed the site so that it worked better for our target audience of Chicago parents and guardians looking for early childhood education. Here’s a review of the process and the results.
Changes to the site based on user feedback
Softer, less map-y design
We completely overhauled the look & feel of the site, making it softer and more rounded. Lots of users we talked to had childcare needs in different areas of the city or related to different parts of their lives (home, work, and relatives, for instance). For this reason, we also moved away from a stark search box and toward explanations of how to approach the site. We added explanatory text that short-circuited the most common question. We also moved from book imagery to a crayon/ marker in the logo to better reflect the programs that parents were looking for.
More prominent text feature
The text feature, which allows residents to text a zip code to a special number and receive a set of nearby location, was very popular in testing. We did notice that texting was hidden in the navigation, so we added a paragraph highlighting the feature. We also made the text phone number easier to see and share by giving it a separate page with a separate URL.
Improved search that helped you along
We saw that many people started off their search with a location in mind, whether it was a school or a neighborhood. We moved away from a pure address search and now pre-populate the search box as the user types. This short-circuits the search process and makes people immediately feel like this is a place that has what they’re looking for. The “Browse by community” function provides another way for people to dive in without putting in an exact address.
Improved filtering for more user control
We found in testing that people did not know how to easily drill down into search results and they very rarely used the filtering feature. We made the filtering more prominent and took up much more screen real estate with details of the search results. Previously, the user had to click on a particular item on the map to reveal details. An overall insight we observed from testing was that the map is not the thing—the details of early learning centers was the thing. We changed the interface to reflect this.
Better comparisons– more locations, easier to share
One thing we heard loud and clear from parents was that they wanted to be able to compare more than two locations. In response, we completely changed the comparison system—changing it to a more recognizable star / favorite system, displaying starred items in a grid, and giving the user the flexibility to easily add and remove locations.
Admin tool for management beyond the spreadsheet
An important milestone in this reporting period is the creation of an easy-to-use admin tool to manage all of the locations. Previously, the site was run by a “magic spreadsheet” that was difficult to manage. The Django admin interface to the rescue!
The admin search tool allows you to drill down quickly
And location detail pages are managed through a simple web form
Conclusion
The process of engaging Chicago residents with this tool has been very rewarding. Since we started this project– and in part based on what we learned here– we started the CUTGroup, a set of regular Chicago residents who get paid to test civic apps. This kind of back-and-forth helps developers, government, and residents communicate with each other and make our lives better.
Civic User Testing Group as a Model in Changing the Relationship Between Government and Residents (#cfasummit)
Here’s the presentation I’m giving today at the Code for America Summit in the context of the “Changing the Relationship Between Government and Residents” panel. Panel description as follows:
Civic technology often aims to change way governments and citizens interact — but how well is it working and what are we learning from these efforts? This session will focus on how cities are changing the way they relate to residents and enabling new modes of civic participation and engagement.
Looking forward to the conversation with Catherine Bracy, Matthew McNaughton, and Nick Bowden. Here’s my text, slides & links:
[VIDEO] Super-Specific CUTGroup Feedback on Go2School App
We are preparing our full take on CUTGroup tests we did on Tom Kompare‘s Go2School, a website that allows you to explore travel options to your Chicago Public School., but I wanted to share some great cuts we took from one of the testers from Tuesday, May 28th at the Kelly Library in the Englewood neighborhood. Take a watch— it’s really worthwhile. If you view the video on Vimeo, you can skip direct to each part.
CUTGroup #2, Tester #3 Public wifi searcher with one child and sometimes one additional child at another location
CUTGroup #2, Tester #3 Public wifi searcher with one child and sometimes one additional child at another location from Daniel X. O’Neil on Vimeo.
0:04: No broadband at home: Heavy user of public wi-fi
1:10: Homepage map confusing: Because it has nothing to do with where she lives.
1:47: Thought there would be more information about the school: Very much likes having custom school info on the site.
2:38: Her usual route is simple. But sometimes she takes a friend’s child to a different school via a more complex route
3:44: Tries that route: Notwithstanding my instinct to do it for her.
4:00: Starting over was a bit of a challenge: Back button shenanigans and back-space/ deleting the entire length of the pre-populated school name.
4:46: The other school she’s interested in is not a CPS school: “So this is useless to my now”
5:30: But she tries a workaround: With the “Explore Map” link.
6:19: Then she found a bug: Noble Street Charter School was geocoded incorrectly in the original data.
6:41: Back to the workaround: In finding a route to a Catholic school.
7:17: Issue with the time picker: She wants to enter the time she should leave in order to get there on time, rather than entering the time she needs to arrive (which is how the system is designed).
7:40: Then somehow the back button was invoked. Don’t really know how. She recovers and again enters the time she thinks she should leave to be there by 8AM.
8:12: Did not know she had to pick a day. And did not readily see the error condition. Also: still wants to enter *departure* time rather than arrival.
9:58: And now the error re: arrival time becomes evident. And that was a bummer.
10:48: This was an excellent user test. Very grateful.