For our fifth Civic User Testing session, we tested ChicagoWorksforYou.com, a Smart Chicago project. This was our first remote CUTGroup test and it took place on November 25, 2013.
ChicagoWorksforYou provides citywide information about city service requests. Users are able to get a ward-by-ward view of service delivery in Chicago, learn about the top service requests made on a given day, view photos of requests, and learn more about the process of submitting service requests.
This Web site uses data directly from the City of Chicago’s Open311 API, and shows the days with the most service requests going back to January 1, 2008.
Through this test, we wanted to find answers to these key questions:
- How do users use the site? What do they have difficulties with?
- Are users interested in the Web site’s content?
- Do users want to share information from the Web site through social media/e-mail?
- Will regular Chicago residents find this Web site interesting?
Segmenting
On November 23, 2013, we sent out an e-mail to all 565 CUTGroup participants asking them if they would be available to provide feedback through a remote test. We asked some screening questions to gather extra information, but the responses did not influence their participation.
We received 116 responses in one day and asked 90 random respondents to do the test. In addition, we asked 5 willing testers to do a test via screen share. We randomly chose our respondents, and compiled a group of testers from all areas of Chicago, and had a variety of responses to our questions.
Why 95 testers?
- We see this as a broadly useful tool that we want everyone to use, and therefore, wanted to get as big of a response as possible
- We saw this as an opportunity to tell a large group of Chicago residents about this Web site
- We had a large amount of gift cards that were about to expire, and instead of them going to waste, we wanted to reach a large group of people while gaining valuable feedback
Test Format
We ask participants to visit the Web site for 20 uninterrupted minutes on one day (November 25, 2013) and answer questions about the Web site through an online form. We asked “yes” or “no” questions to gather quantitative results, open-ended questions to see what users were interested in when visiting the site, and finally asked users to click on specific links of the Web site, and discuss their experience.
We wanted to conduct a screen share 5 people through Google Hangouts to monitor their actions, but due to technical problems, we were unable to do the screen share. In the future, we would like to provide more technical instructions on how to screen share and give extra time.
Since this was our first remote test, we were interested in seeing how many people would be interested in doing this type of test, and see how thorough and clear the responses were without a proctor. We were pleasantly surprised at the thoroughness of tester’s responses without a proctor being present.
We still value in-person tests, and believe they are an opportunity to convene participants from all areas of Chicago. We gain valuable responses in person and are able to record a tester’s actions and reactions.
Here are a couple of responses we heard from our testers specifically about the remote test:
“An online test is a better form of testing a website or app. It is done within the comfort of one’s home, with flexible times. It allows more people to participate and allows for a more natural environment.” –Tester #77, Elizabeth07
“I do like the remote survey better than the one I had to go to the library for. The particular public library I had to go to was in a very dangerous area and I didn’t know before I went…Other than that it has been a pretty good experience being a part of the CUTGroup thus far and I’m definitely willing to give my input on multiple websites.” – Tester #46, 3rd Year Student
Results
At the end of the day, 84 users tested the Web site and gave us feedback. We learned that most testers were interested in learning about service requests in their own ward, and see how their Alderman is doing. When visiting their ward’s page, testers responded positively to having their Alderman’s contact information in one place. 82% of testers said they would use this Web site again.
How do users use the site? What do they have difficulties with?
- The most common difficulty testers had was finding their own ward. A lot of testers do not know their ward, and relied on other Web sites to find or double-check their ward number. Testers want to search by their address or neighborhood. In response, we want to add a link to this Tribune Apps Boundary Look-up Tool.
Github issue
Are users interested in the Web site’s content (city services)?
- Testers are interested in submitting 311 requests directly from the site. While there are links to submitting requests on each of the “Services” pages, this needs to be a bigger feature and one of the first things people see when they visit a service type.
A lot of testers are interested in the Photos page. However, only 1 person clicked on “Photos” first when they visited the site. When directed to the “Photos” link, 89% of people said “Yes” they liked the “Photos” page. Below is a screenshot of the “Photos” page.
Here are some steps to make the “Photos” page more prominent:
- Make the “Photos” page the first thing people see on the header: Github
- Take advantage of social media sharing features, and tweet about photos on a regular basis
- Give users the option to organize the photos by ward: Github
- Create a call to action to allow users to submit 311 requests from “Photos” page
Do users want to share information from the Web site through social media/e-mail?
- 48% of testers said “Yes” they would share pages on Twitter, Facebook or E-mail. We learned that sharing information through social media is based a lot on a user’s general use of social media. Some testers do not use social media, or do not like the “Share” feature on any Web site. Testers were more likely to share this Web site with people in their own ward
Will regular Chicago residents find this Web site interesting?
- 82% of testers said “Yes,” they would use this Web site again on their own. The other 18% (15 testers) said they would not use the site again, but 6 of those testers responded and told us they like the Web site. Here is a responses from two of these testers:
“I like the website, but I don’t know if I’ll use it again. I’ve never reported any of these things before. That being said, I never knew how to or really thought about it. I might be more likely to know that I know this exists. Especially because I could actually see if my request was followed up on and taken care of. That would be worth the effort. I do wish there was a submit request button more prominently located. though. Those are kind of buried and I didn’t initially realize I could do that.” – Tester #63, Aspiring Batgirl
“I like the website, but it doesn’t have much of a purpose for me, unfortunately. It’s collecting and displaying data, but as a resident, I am most interested in submitting issues or seeing if an issue has already been submitted.” – Tester#56, Ketchup Ketchup
- Most testers were not interested in specific data points, and some testers even found the data and graphs difficult to understand. Even so, most testers were able to access information about requests and equate that to how their ward and Alderman is doing
- We learned that testers want more discrepancy in the graphs and gradients of the map, and generally how the data is displayed
- Some testers mentioned that they were concerned with recent requests that were open or closed and thought those requests were more relevant than past requests
Final Report
Here is a final report of the results with the key highlights from our CUTGroup test, followed by each tester’s responses, and copies of our e-mail campaigns and the questions we asked:
The raw test data can be found below with the complete answers from every tester.
