CUTGroup Test

For our eighth Civic User Testing Group session, we tested the Waitbot app, which is a place where you can find wait time estimates for all sorts of things, including transit, restaurants, airports, and more. This test had an in-person and a remote component to it. The in-person test took place at one of the Connect Chicago locations – Chicago Public Library Clearing Branch at 6423 W. 63rd Place in the Clearing neighborhood.

Through this test, we were interested in finding answers to these questions:

• What makes users download an app? Delete an app?
• Do users want to use Waitbot on a daily basis? Why or why not?
• What features do users want?
• What other wait categories would users want to see?
• Do users want to share wait time information on social media?

Segmenting

On March 5, we sent out an e-mail to all of our 749 CUTGroup participants. We asked them if they would be willing to test a wait time estimates app on March 12, 2014. We asked some screening questions to gather information, and chose our group of participants based on a diverse selection of answers and also device types.

We were interested in having about 15 participants from different Chicago neighborhoods, but only had 6 testers come to test in-person. A lot of testers could not come due to a combination of weather and distance, so we reached out to 4 more testers to do the test remotely.

In the end, we had 10 testers participate in the test, although 1 tester could not get the app to load and could not fully participate in the test.

Here is a look at which neighborhoods our testers came from:

View CUTGroup 8: Waitbot Participants in a full screen map

 Test Format

For the in-person test, proctors were able to work with testers one-on-one. Testers looked at the app on their own devices and provided feedback, while the proctors wrote down notes. After the test, we sent out additional (and optional) questions to see if testers were using the app and how they liked the app in their own neighborhood.

For the remote test we asked testers to use the app on their own and provided questions to lead them through the test.

In the end, we got great responses from both types of tests.

Results

Before this test, testers generally associated apps that estimate wait times with transit apps. 9 out of 10 testers used Web sites or apps to check wait times. Some apps that testers use for wait times include CTA Bus Tracker, Transit Tracks, and Transit Stop.

Even during the test when testers were offered wait times for other establishments, most testers (7 out of 10) thought the Transit page was still the most useful, and the other three pages (Restaurants, Airports, Emergency Medical) were equally considered the least useful page.

In addition, after looking at the Transit page, testers were not drawn to do one thing over another, and there was not a logical course of action of what to do next.

What makes users download an app? Delete an app?

5 out of 10 testers mentioned that they would download or keep any app if they routinely used it and if it was easy to use. Other considerations included whether it was a low-cost or free, and if it had good recommendations or reviews from others.

Do users want to use Waitbot on a daily basis? Why or why not?

67% of testers (6) liked the Waitbot app and said they would use the app again after the test. Testers who did not like the app did not have many options populate for them and did not think they would use all of the features.

78% of testers (7) found the Transit page the most useful.  In terms of using Waitbot on a daily basis, transit is a key way for users to use Waitbot on a daily basis.

What features do users want? How can Waitbot be improved?

• Only 1 tester liked the Map view option over the List view. The majority of testers liked the list view because it gave more information and was better organized than the map view. It was in general easier to use
• There was confusion regarding the color coding and what the colors meant in terms of wait time. 1 tester thought that the color referred to the CTA line
• Testers are interested in seeing more details and information for each page. In addition, besides the Transit page, testers did not have many options and a lot of places had “No report.” Testers were not immediately aware what this referred to

“Add Wait” was not particularly a well understood feature of the Waitbot app.

• Two testers chose it after looking at the “Transit” option because they were curious of what it did
• 4 out of 9 testers did not clearly understand what this feature would be used for. One tester saw this option as only for the establishments to add, not necessarily residents. Another tester clicked on “Add wait time” while on the Emergency Medical page and it gave him a wait time screen
• 3 out of 9 testers said it seems easy to use, and was straight-forward

A tester, I like Cacti (#4), said he would add wait times:

“If there’s enough data and it’s actually helping people. When I am at a restaurant and the line is really long, it’s a way to complain about it.”

What other wait categories would users want to see?

Of the categories “Coming Soon,” the top categories testers were interested in included:

• Government Services
• Tourist Attractions
• Barber Shop
• Nightlife

Other categories, testers wanted to see included:

• DMV
• Grocery Store lines
• Movies
• Gas Stations

Do users want to share on social media?

Most testers were not interested in sharing this type of information on social media. One tester would share on Facebook only if it was automatically connected, while another tester said he would not do it unless there was an incentive. Only 3  out of 10 testers would share on social media.

Conclusion

When testing the Waitbot app, testers liked the Transit page and the fact that it populated with nearby options. There was some confusion with color-coding and testers wanted added features such as route display. However, testers overwhelmingly liked this page.

One tester, My eyes are dried out (#10), explains why he doesn’t like the Waitbot app in general, but thought that the Transit page was the most useful:

“The Swiss army knife is useful and practical. Then the impostors ‘improved on it,’ making it bigger and more cluttered with useless features. Sometimes I feel app creators want to entice a large crown, instead of just perfecting one good thing.”

The Transit page is the one that testers would use consistently, while some testers might check Restaurant wait times. Airports and Emergency Medical categories are both categories that people would not base their decisions on before using. Lastly, testers said that the restaurant category did not have enough information about the establishments, or no wait times were represented. The majority of testers liked the app (67%), and thought it was an interesting concept, but without more wait time information or categories that users would check to base future decisions on, it is difficult to state if users would use Waitbot on a regular basis.

Final Report

Here is a final report of the results with key highlights from our CUTGroup test, followed by each tester’s responses, and copies of our e-mail campaigns and the questions we asked:

The raw test data can be found below with complete answers from every tester.

 

For our seventh Civic User Testing Group session, we wanted to get feedback on the EveryBlock iPhone app after EveryBlock relaunched on January 23, 2014. This in-person test took place at one of the Connect Chicago locations – Chicago Public Library Mayfair Branch at 4400 W. Lawrence Avenue in the Mayfair neighborhood.

EveryBlock lets you follow neighborhoods and get information about what is happening around you. From the EveryBlock Web site:

“Our goal is to help you be a better neighbor by giving you frequently updated neighborhood information, plus tools to have meaningful conversations with neighbors.”

We were interested in answers to these questions:

• Are there any problems using the iPhone app? Does the app crash?
• Do users have an easy time posting or commenting on the app? Do they find things they want to comment on?
• Do users like the different pages in the app?
• Would users use this app and in what scenarios might they post or comment? Would they use it “on the go”?

Segmenting

On February 5, we sent out an e-mail to 269 CUTGroup participants who said they had an iPhone as their primary or secondary device. We asked them if they would be willing to test a neighborhood app on February 10. We also asked some screening questions to gather extra information, and chose our group of participants based on a diverse selection of answers.

Here is a look at the distribution of iPhone users in our CUTGroup:

View iPhone Users for CUTGroup 7 in a full screen map

We were interested in having about 15 participants from different Chicago neighborhoods. We had 12 testers who came from neighborhoods across Chicago including Albany Park, Hermosa, Edgewater, Uptown, Logan Square, Auburn Gresham, and more. The furthest a tester traveled from their home location was 15.3 miles.

View CUTGroup 7 Participants in a full screen map

Test Format

This in-person test was the first opportunity we had for every tester to be paired with a proctor. In previous tests, some testers were paired with proctors, while others would complete answer questions about the Web site or app through an online form.

Smart Chicago and EveryBlock staff (including a developer who worked on the app) acted as proctors for this test. We understood that there might risks in having EveryBlock staff ask questions to testers since biased responses might be captured. We asked that all EveryBlock staff not introduce themselves as staff or indicate that they were associated in any way to the app.

We received both positive and negative feedback from testers even with this format. We value this opportunity for developers to be involved in the CUTGroup process. The open communication and active involvement between developers and residents are key in developing Web sites and apps to improve lives in Chicago.

Results

Are there any problems using the iPhone app? Does the app crash?

92% of testers liked the EveryBlock iPhone app, and most testers thought the app was easy to use. 4 out of 12 testers (33%) experienced a bug when using the app.

After signing up or opening the app, two testers experienced bugs on the first “Timeline” Page.

• Victoria P Test (#4) did not have anything populate on the “Timeline” page. Content appeared in mobile web but not in the app. Victoria P Test was able to see content on “Near You Now” page
• Software Breaker (#10) had an error message appear saying there is a problem connecting to EveryBlock

The app crashed when Amateur Student (#12) clicked on the New Post icon. In addition, the app crashed when Application Developer (#2) hit “Account Settings.” Once she made a post, the app did not redirect her to a new page, but instead went to a black page.

Do users have an easy time posting or commenting on the app? Do they find things they want to comment on?

New Comments:

• In regards to making a comment, 58% of testers found a post they would want to comment on. Most testers thought it was easy to make a comment and was similar to other apps
• 42% of testers did not want to make a comment. Some testers were not comfortable making comments, while others addressed privacy issues in making comments. One tester found a post that she might comment on, but the post was not formatted for comments
•  One tester explained that they “feel like not much of a contributor.” For many testers, the decision not to make a comment is due to personal use of apps and Web sites. Two testers specifically were concerned about privacy on EveryBlock. A suggestion from a tester is clarity and options to change how names will appear on posts and comments

New Posts:

• 75% of testers liked the new post interface, and when asked what testers would want to post about half of the testers mentioned specific neighborhood news: neighborhood concerns, crime warnings, traffic, weather, favorite places, garage sales, missing pets, events

Do users like the different pages in the app?

67% of testers thought that “Timeline” was the most useful page. As mentioned before, two testers did experience problems with this page, and this number could be even higher if these issues did not appear.

• Testers liked this page the most because it showed the most relevant information. Testers were also interested in the “Near You Now” page when “on the go” to learn more about other neighborhoods
• One tester wanted to add more zip codes to follow through the app to create a more personalized experience

Would users use this app and in what scenarios might they post or comment? Would they use it “on the go”?

83% of testers liked the option of making posts on the go, 17% of testers were not sure if they would like it or thought they might use it rarely. The majority of testers thought this was a convenient option, and wants to comment on things while they are happening. One tester commented that she is a Twitter user and is used to this concept.

What do users want to see in the EveryBlock app? 

While 75% of testers used EveryBlock, none of the testers ever used the iPhone app. Therefore, some testers had knowledge of the EveryBlock Web site and could compare these experiences. Testers who were familiar with the Web site were interested in having these features:

• Search feature
• Filters or an ability for users to choose what they want to see
• Bookmark feature to read posts later
• User Profile – more apparent and have incentive to contribute and “gain status” in the community
• More information on Community Guidelines & Privacy
• Address finder and map capabilities to help find locations

Here are some other suggestions:

• Testers who did not use EveryBlock in the past were still interested in personalizing their experience.  These testers also wanted to see features like filtering, search bars, and an option to delete irrelevant posts from the feed
• Two testers would like to see dates and times on posts to see if posts are the most relevant
• 67% of testers were interested in sharing posts on social media
• More information on privacy is important to testers. There should be options to see how much personal information is being posted, and to change privacy features within the app

 

One tester was shocked to find the post above during the test about a break-in because it included the poster’s full name and the address. This tester was concerned for her own privacy and was interested in knowing how her information would appear before she posted or commented.

Conclusion

Testers generally had positive experiences using the EveryBlock iPhone app. The biggest takeaway from this test is that users are interested in more features to filter and find the news most relevant to them. Testers were able to point out features from the EveryBlock Web site that they would want to see on the iPhone app. Including these features would allow users to have an experience that matches their experience on the EveryBlock Web site.

Final Report

Here is a final report of the results with the key highlights from our CUTGroup test, followed by each tester’s responses, and copies of our e-mail campaigns and the questions we asked:

The raw test data can be found below with the complete answers from every tester.

Videos

Photos

For our sixth Civic User Testing Group  session, we focused on the editor feature of OpenStreetMap.org. This in-person test took place at one of the Connect Chicago locations – Chicago Public Library Rogers Park Branch at 6907 N. Clark Street in the Rogers Park neighborhood.

OpenStreetMap (OSM) is a Web site that is built by a community of mappers that contribute local knowledge and information to a map for everyone to use. Anyone can sign up, add information and edit the map.

OpenStreetMap is open data, and you are free to use it for any purpose as long as you credit OpenStreetMap and it’s contributors.

We were interested in answers to these questions:

• How do users use the site, and what can be improved from an UX standpoint?
• Why does the general audience want (or not want) to use the OSM editor?
• Do people see themselves as potential long-term users?
• Is this useful to general internet users? Do general internet users want to contribute? If so, how?

Segmenting

On January 13, 2014, we sent out an e-mail to 551 CUTGroup participants, who want to participate in-person, asking them if they would be willing to test a mapping Web site on January 22. We asked some screening questions to gather extra information, and chose our group of participants based on a diverse selection of answers.

We were interested in having  about 15 participants from different Chicago neighborhoods. Testers ended up coming from neighborhoods across Chicago including Rogers Park, Pullman, Albany Park, West Town, Logan Square, South Chicago, and more.

View CUTGroup 6 Participants in a full screen map

We ended up having 16 participants, and for 15 of these testers this was their first CUTGroup test. One tester traveled 23.45 miles from their home to participate in this test.

Results

16 testers provided their feedback regarding OpenStreetMap, and we learned a lot not only about the functionality of the map editor, but also about people’s feelings on the concept of editing a map. Some testers liked the idea of contributing knowledge for others to use, while other testers thought the concept of making live changes to a map was “scary” or “dangerous.” Here are a couple of very telling comments we got:

“I believe in power of people and having a significant contribution to these things. Gives a sense of community and add value in the sense of belonging” – Tester #5, kirehernan

“I personally would but I would not want others to have the same access as I would due to the lack of restrictions.” – Tester #7, B

How do users use the site, and what can be improved from an UX standpoint?

Walkthrough:

• We learned that providing information through a walkthrough and the welcome page are important for users to understand how to map
• 5 testers (31%) were never prompted to do the walkthrough or the walkthrough would not load
• Out of the 11 testers who did at least some of the walkthrough, only 5 had positive experiences. 6 testers thought some parts were difficult or confusing

Here are some suggestions we learned:

• It is important that users have an easy access point to the walkthrough. Instead of having the walkthrough prompt being based on what the user clicks, the walkthrough option needs to be always accessible by the user. This will allow users to access information when they want and go back to it if they need more assistance
• Some walkthroughs did not work properly: difficulty creating a line, highlighted areas did not line up correctly, “zoom in to start mapping” was difficult to see
• It might be helpful to consider breaking up the walkthrough into different steps. This will allow users to choose what they want to learn and serve their needs better for the type of information they want to add (ex: add a path, add an area, add information, add a note, etc)
• 2 testers thought that a video might be easier alternative to follow

Search Function:

• 11 testers (69%) liked the search feature, while 5 testers (31%) did not like the search function
• Testers became frustrated that when their search was not in the visible map area that they had to choose “Search Worldwide” 
• When searching, testers received sometimes too many options that were irrelevant to what they were looking for, or could not find their search term. Testers are interested in having a clearer way of searching for locations. Additionally, it would be helpful to understand why some results may not appear, and provide a prompt for the user to add/edit information

Editor:

• 8 testers mentioned some difficulty in editing the map. Difficulties were in (1) search feature – some testers could not find what they were looking for, (2) finding or choosing the correct identifying tag, and (3) some thought editing was confusing. Out of these 8 testers, 5 said that they finished the walkthrough
• 12 testers (75%) liked the edit features, while 4 testers (25%) said they did not like the editor features

Why does the general audience want (or not want) to use the OSM editor?

14 testers (88%) said they liked the Web site, and 11 testers (69%) said they would use the map editor again. Here are some reasons why testers want to use OSM editor:

• Update outdated data
• Contribute to a neighborhood’s visibility
• Feel the need to customize or make a map more personal
• Add information about safe pedestrian and bike paths

In addition, two testers mentioned that they wanted OSM to have a feature to see the history of a location to know what it used to be and what it is now.

A couple of testers do not feel like they would want to edit because they believe that maps should accurate in the first place, and that is the responsibility of the map-maker.

Do people see themselves as potential long-term users?

Testers seemed interested in adding information to locations that were important to them, or updating outdated information. This would happen on a case-by-case basis, and therefore, it might be hard to tell if testers see themselves as long-term users. Testers were not interested in mapping outside of their own realm of knowledge.

Is this useful to general internet users? Do general internet users want to contribute? If so, how?

At this time, OSM editor is a series of tools that might be difficult for some users to use. That being said, this test has informed us that most people do want to contribute, but in different ways. Some testers want to update text information, others want to add paths, and others might want to add a note.

The note feature might be more successful as a more prominent option for users. Here is a comment from a tester about the welcome page:

“Good overview. I’m curious about editing vs just adding a note. Why one over the other? Are they substantially different?” –Tester #15, Snarky

Conclusion

When testers were asked about things they noticed on the homepage, only 1 tester mentioned the “Start Mapping” button. Since testers are interested in contributing knowledge through the editor in different ways, there needs to be different points of engagement for users. OSM might want to consider prompting users to add information in different ways throughout the experience. In addition, the suite of editor features might be cumbersome for the general audience, and different levels of editing will serve a larger audience.

Final Report

Here is a final report of the results with the key highlights from our CUTGroup test, followed by each tester’s responses, and copies of our e-mail campaigns and the questions we asked:

For our fourth Civic User Testing session, we tested EatSafe.co, a Web site developed by Hoy Publications. This in-person test took place at one of the Connect Chicago locations – Chicago Public Library Hall Branch at 4801 S. Michigan Ave in the Grand Boulevard neighborhood.

EatSafe.co is a website that shows you details of food inspections near you. It uses your current location to show establishments near you that have food inspection history. You can also search establishments such as restaurants, schools, or grocery stores.

EatSafe.co uses data collected from the City of Chicago Data Portal, which provides raw data of recent food inspections. It is a product of Hoy Chicago. We really enjoyed working on them to get with regular Chicago residents and hear directly from them. Here’s what Hoy Managing Editor Fernando Diaz had to say:

Partnering with the CUT Group was the most effective research experience I’ve ever been a part of. We met real Chicagoans who were generous with their time and feedback. And among the highlights is that we have all of the results for further evaluation and incorporation into future iterations of our project”

Hoy Managing Editor Fernando Diaz working with  CUTGroup members – CUTGroup #5 – Eatsafe.co – Hall Library

We were interested in finding answers to these key questions:

• What is the first thing users do when they visit the Web site? Do users allow the site to use their current location?
• What do users have difficulties with when using the Web site? How can this Web site be improved?
• What is the user’s overall experience?
• Are Chicago residents interested in food inspection history?

Segmenting

On November 6, 2013, we sent out an e-mail to all 541 CUTGroup participants asking them if they would be willing to test a food inspection Web site on November 21. We asked some screening questions to gather extra information, but the responses did not influence their participation.

We were interested in having 20 participants from different Chicago neighborhoods. Testers ended up coming from neighborhoods across Chicago including Chicago Lawn, West Town, Edgewater, and more.

Results

In the end, we had 18 responses captured via online form, and 1 user’s test was captured via video. We learned that most testers were interested in learning about food inspection history, and believed that learning about food inspections would impact their future choices.

What is the first thing users do when they visit the Web site? Do users allow the site to use their current location?

• We found out that testers were not sure where to start on this Web site, and only half of testers allowed the Web site to use their current location
• There was not a clear first thing that most testers did when visiting the Web site
• Most testers saw this as a site to look up restaurants even though there is information about many different establishment types including grocery stores or schools

What do users have difficulties with when using the Web site? How can this Web site be improved?

The screen shot below is an example of an establishment’s results. Testers had difficulties searching for a specific establishment and deciphering the results.

Here are some suggestions we learned from our testers:

• When clicking a violation, testers wanted an explanation of the violation, and not be led to a list of establishments with the same violation
• Testers were interested in better explanations of the inspection results
• Some of the testers thought that too many results came up when they searched for a specific establishment, and wanted to search by other fields such as zip code or neighborhood
• Testers want to see different search results on a map, and be able to search by moving the map’s view
• Testers are looking for consistent color-coding and violation legends between pages; testers want a better understanding of the colors used, the thumbs-up or thumbs-down symbol, and the number next to each violation

What is the user’s overall experience?

• 56% of all testers liked the Web site. The other 44% of the testers who did not like the Web site still thought this was an excellent idea or a useful tool, but did not like the graphics or found it particularly easy to use

Are Chicago residents interested in food inspection history?

• More than 83% of testers  thought that learning about an establishment’s food inspection history would impact future choices
• Testers want a quick way of of learning about inspection results and better explanations of inspections and violations

Final Report & Videos

Here is a final report of the results with key highlights from each question we asked, followed by each tester’s responses, and copies of our e-mail campaigns and the questions we asked:

The raw test data can be found below with the complete answers from every tester.

Here are videos from our testers: