For our twelfth Civic User Testing Group session, we tested the Roll with Me, a website that helps residents find accessible transit directions in Chicago. This test had two components— a week-long remote test with an in-person discussion at our offices – The Chicago Community Trust at 225 N. Michigan Ave in the Loop.
Mohammad Ouyoun developed Roll with Me during an internship with Smart Chicago and as part of the Civic Works Project. We sat down with Mohammad to see what he wanted to learn from the test. We focused on individuals who have difficulties with stairs, and Mohammad saw use of this website for people that have temporary needs such as parents with strollers, or individuals using carts or luggage. Here are the things that we really wanted to know:
- How does Roll with Me work in user’s normal day-to-day? Is it easy to use on different devices?
- How important are the alerts for users?
- How does Roll with Me compare to other user’s methods of finding transit routes? What features are users interested in having?
Segmenting
On January 8, we sent out an email to all of our 837 CUTGroup participants. We wanted to know if they would be willing to answer a short remote questionnaire 2 or 3 times during a week, and then be able to come to an office in the Loop for an in-person discussion on January 20, 2015. We asked some screening questions to gather information. We wanted to focus on testers who have difficulties with stairs, and ended up with 7 testers who said they did. Since we were looking for more testers, we also reached out to an additional 8 testers who were heavy users of public transit to provide feedback about this app in comparison to other transit apps and websites.
We ended up having 14 testers participate in this test from different neighborhoods in Chicago. Here is a look at the neighborhoods distribution of the testers:
Screening Questions
We heard form 95 CUTGroup participants through our callout for testers. We received a lot of good information just from the screening questions. Here is a look at what we learned:
- 67% of CUTGroup respondents use public transit on a daily basis.
- 18% of respondents use public transit to travel to the suburbs on a regular basis (2 or 3 times a week)
- 23% of respondents specifically mentioned using CTA tools (such as CTA Bus tracker) as their primary way of getting transit route information. 21% of participants mentioned Google Maps as their primary way to search transit routes.
Test Format
We asked testers to participate in multiple test parts over a week’s time. We were happy with the level of detail respondents provided in the quantitative remote portion of the test .
14 of the 15 testers we invited answered Part I’s questions via an online form. A couple of days later, 11 of those 14 testers also completed Part II questions. Then on January 20, 2015, 10 of the 14 testers were able to make it to the in-person section.
We wanted to have an in-person discussion that was casual and focused on qualitative responses. This was not a traditional focus group. The goal of this discussion was for Mohammad to ask questions directly to testers and get feedback. Through this format, we were able to discuss very specific features that testers were looking for.
Results
How does Roll with Me work in user’s normal day-to-day? Is it easy to use on different devices?
We learned that 71% of testers normally check directions on their mobile device. When talking about how it was to find directions, 50% of testers mentioned that it was simple, easy, or straight-forward.
Only 5 testers tried the “Find Me!” feature. 64% of testers did not try to use it and did not see it during their use of the website. Currently the “Find Me!” feature is not working, but once it is fixed, testers thought it should be larger and more prominent.
3 testers experienced problems with searching for their address due to the fact that their addresses came up in different cities rather than defaulting to “Chicago, IL.”
Also, testers were frustrated that when searching for directions they could not go back and edit the directions, but instead had to retype all of the information again.
How important are the alerts for users?
The majority of testers found the alerts to be very useful information, but a few testers thought that they were distracting from the route information. 3 testers thought it would improve the website if the alerts were moved from the top, center location.
Wheelchair CTA rider (#4) says, “I EXTREMELY like that the accessible info is available for the whole CTA system on the same page.”
During the in-person discussion, Writer Rider (#12) explained that she liked having all of the alerts because it brings attention to accessibility needs and the fact that routes might be unavailable due to construction or maintenance.
How does Roll with Me compare to other user’s methods of finding transit routes? What features are users interested in having?
During the first part of the questions, 5 testers (36%) wanted to see a map associated with their directions. 7 testers (50%) wanted to be able to input a custom travel times to get directions. The 15-minute intervals were not sufficient for testers.
For the second part of the test, we asked how important these features were to all of the testers:
How important is it for Roll with Me to display your directions on a map?
5 Very Important 9% (1)
4 Important 55% (6)
3 Neutral 27% (3)
2 Slightly Important 0% (0)
1 Not Important 9% (1)
How important is selecting a custom departure time when finding directions?
5 Very Important 55% (6)
4 Important 36% (4)
3 Neutral 9% (1)
2 Slightly Important 0%
1 Not Important 0%
How important is selecting a custom arrival time when finding directions?
5 Very Important 55% (6)
4 Important 27% (3)
3 Neutral 18% (2)
2 Slightly Important 0%
1 Not Important 0%
Roll with Me for Other Uses
We wanted to also see if Roll with Me could be used for residents who do not have mobile disabilities.
Blue Line + Buses Only (#10) says, “I think the name discourages people who might not be in a wheelchair but might appreciate accessible travel accommodations (ppl. uncomfortable taking stairs, who can’t go far distances on foot, etc.)”
How useful do you think Roll with Me is for people without mobile disabilities? Such as people with strollers, carts, or luggage.
5 Very Useful 27% (3)
4 Useful 27% (3)
3 Neutral 36% (4)
2 Slightly Useful 9% (1)
1 Not Useful 0%
Currently, Roll with Me does not specifically target these groups of people, but some testers did see the use of Roll with Me for people without disabilities.
Next Steps
Directions need to be easier to find.
We heard a few suggestions from testers to make finding directions easier:
- Find me! Feature needs to be fixed and more prominent on the website.
- Testers tried using cross streets to find transit routes, which either produced incorrect results or did not show any results. One suggestion might be to provide examples of how the address should be written on the homepage.
- A lot of testers faced the same issue that when they started to type in their address, addresses started to show for other cities before Chicago. Since this app is limited to accessible transit routes in Chicago, testers wanted it to default to Chicago without having to type in city and state info. In addition, there was a bug when testers did not type in the full address.
Adding additional features
- 90% of testers thought it was “important” or “very important” to have custom arrival and departure times when finding transit times.
- 63% of testers thought it was “important” or “very important” to have a map to display directions.
We learned that most testers use other websites, such as the CTA Bus Tracker or Google Maps, and are looking for similar features in Roll with Me.
Final Report
Here is a final report of the results with the analysis of the questions we asked, followed by each tester’s responses, and copies of other questions we asked:
The raw test data can be found below with the written answers from every tester.
The Connect Chicago website, a resourced maintained by Smart Chicago showing free computer access and digital skills locations across the city, is featured in Putting Learning on the Map: Visualizing Opportunity in 21st Century Communities. Published at New America, an organization “dedicated to the renewal of American politics, prosperity, and purpose in the Digital Age” and authored by Lindsey Tepe, Policy Analyst in their Education Policy Program, here’s how they frame the report:
Our Chicago School of Data project is highlighted in the 2014 Annual Report of the Chicago Community Trust. Here’s a snip:
