For our tenth Civic User Testing Group session, we tested the Build It! Bronzeville app, which is a mobile game app that uses GIS technology to help residents guide development in their neighborhood as they complete quests (i.e. enhance safety, visual appeal, and foot traffic). This in-person test took place at one of the Connect Chicago locations – Chicago Public Library Chicago Bee Branch at 3647 S. State Street in the Bronzeville neighborhood.
Build It! Bronzeville was a Winner of CNT’s 2013 Urban Sustainability Apps Hackathon competition, and this CUTGroup test was a part of the prize package. If you are building applications to make lives better for residents of Chicago you can sign up here for a CUTGroup test of your app. This is one way Smart Chicago supports development of innovative civic applications through developer resources.
Through this test, we were interested in finding answers to these questions:
- How do users play games? How frequently?
- How would users feel about registering ?
- How do people want to contribute in the game (check-ins/receipts)?
- Do users see a community impact from this game now?
- Are they having fun?
Segmenting
On June 20, we sent out an e-mail to all of our 764 CUTGroup participants. We asked them if they would be willing to test a neighborhood game app on June 25, 2014. We asked some screening questions to gather information, and chose our group of participants based on a diverse selection of answers. The app was available only to Android users, and we segmented testers by Android phone and tablet users. We wanted to focus on getting testers in the Bronzeville or South Loop areas, but were open to other neighborhoods.
We confirmed 14 testers but only 6 testers ended up coming to this test. There was not a clear reason for why testers did not show up for this test. The furthest a tester traveled from their home location was 11.02 miles.
Here is a look at which neighborhoods our testers came from:
View CUTGroup 10: Build it! Bronzeville in a full screen map
Test Format
This was an alpha-test of the app which focused on three levels of a game. We wanted to see if testers understood the game, if they liked the design and playing, and if they like the element of sharing receipt data to get in-game benefits.
For this test, it was important that Smart Chicago and part of the Team Build it! proctored the testers one-on-one to monitor tester’s actions and take notes on commentary and bugs. We advised the developer and team not to tell testers that they helped develop the app in order to receive unbiased advice from our testers.
Results
This CUTGroup test was the first time Smart Chicago tested an app in the early stages of development, and it was also the first time we tested a game app. We were excited to learn about how testers play games and what aspects they enjoyed the most. We were also very interested to learn how the community connection of this game affects if users are more inclined to play.
All of our testers brought Android phones to the test, and 5 out of the 6 testers were able to test the Build it! Bronzeville app.
How do users play games? How frequently?
We learned that the majority of testers play games on a daily basis, and normally play while they are commuting. On average, testers had 3 game apps on their mobile devices. Two testers mentioned they like to play puzzle games, while other games that were mentioned include: Threes, Piano Tiles, Farmers vs. Zombies, Candy Crush, Angry Birds.
Only 1 tester mentioned that they do not play any games on their mobile device due to a lack of time.
When we asked testers if they liked to connect to social media, 3 testers (50%) said they connect with social media on mobile games, while the other 3 testers (50%) chose not to. Privacy was a main reason to not connect with social media, since testers do not want the app to post about game play on those outlets.
3 out of 6 testers (50%) mentioned that they stop playing a game when there is nothing new or challenging in the game.
How do users feel about registering?
Two testers had problems connecting to the game on their app – MBA Girl (#4) was using a Samsung S4, while Spoung45 (#5) was using a Samsung Note 3. Spoung45 used the developer’s device to complete the test.
All of the testers (5) who were able to complete the test thought that signing up on the app was easy. There were three recommendations from testers to make signing up easier or better:
- Text should be bigger
- Password should be seen as she types and change to symbols
- It would look better to be embedded in the screen rather than being a pop-up
After registering, two testers had an issue with their keyboard remaining stuck on the screen and having to close the app and restart.
How do users want to contribute in the game (check-ins/receipts)? Is there any hesitation with privacy when users are asked to share receipts?
Testers prefer to contribute to the game by checking in at local businesses versus taking photos of the receipts.
3 out of 5 testers (60%) said they would share receipts by taking photos. 1 testers who did not want to share receipts thought this was a “weird” option, while another tester did not think this benefit was fair due to individual habits and if youth want to play they might have receipts to share.
All of the testers said they normally keep their receipts on them.
4 out of 5 testers (80%) would check-in at local businesses to get in-game benefits. Both testers who would not share receipts were willing to check-in at local businesses. 1 tester who did not want to contribute by checking in said it was because she normally plays games in transit.
Do users see a community impact from this game? Do users enjoy this element? Is community involvement important to their game play?
The majority of testers liked the community improvement element of the game, but 2 testers wanted more context about why the game is set in Bronzeville and an understanding if the game levels match real, existing Bronzeville blocks.
“I always wanted to see how neighborhoods interact with each other and how money might be taken from one neighborhood to another.” – Music Lover (#1)
When Telephober (#6) saw the description for sharing receipts, she thought that sharing receipts would be “totally worth it” since it “helps the community.”
Nathalie (#2), wanted not only to contribute to what gets built in a neighborhood but also hear feedback from residents:
“The levels were sort of limiting… there’s not really a feedback loop from citizens… I would like to know if residents are happy with two salons on the same block.”
3 out of 5 testers (60%) said they would be more inclined to play the game if players are rewarded by spending at local businesses. Casual Gamer (#3) really disliked the receipt part of this game, and thought that the developers should “get rid of the creepy receipt scanning issues.” Casual gamer is interested in a story behind the levels and making money in the game versus to helping the community.
When testers were asked what they decided to build, 3 out of 5 testers (60%) built building based on community needs.
By adding more context to each level about the neighborhood in connection with other neighborhoods through Chicago, and an understanding of real lot vacancies in the neighborhood would provide users a well-rounded understanding of each level and the purpose of the game in general.
Are users having fun?
4 out of 5 testers (80%) thought the levels were fun, and were reminded of other games such as Cityville and Roller Coaster Tycoon. Testers preferred more realistic graphics seen in level 2 over the very basic and simple graphics in level 1. Here is a look at the graphics in level 2:
Music Lover (#1) describes this game as “Edutainment – Educating and entertaining at the same time.” She likes that this game could prompt someone to think about their neighborhood and about what it does and does not have.
Next Steps
View and graphics
Testers preferred the more realistic graphics seen in level 2 over the basic/simple graphics of level 1. During the levels, most testers had difficulties tapping the lots, and clear boundaries of lots should be added. Most testers were also unaware that they could move around on the map, and wanted to zoom out to see the full block.
Bug fixes
Since this was an early test of the game, there were a number of bugs that testers faced, specifically:
- Samsung S4 and Samsung Note 3 both could not load the app
- Keyboard stayed open after signing up and could not be removed without backing out of game entirely
- Lots would not “pop” when tapped
- When taking a photo of a receipt, screen remained stuck on “sending data” action
More information
Overall, testers enjoyed playing Build it! Bronzeville, but wanted more background information about the game. Providing context about why this game is in Bronzeville and the spending at neighborhood businesses in the tutorial would give users a firmer understanding about the purpose of the game.
In order to make the game easier to use, more tutorials or descriptions of icons and features should be added:
- All icons should be described including the money and people status bars
- Better information about how money is generated
- More information about the building types
- A story or information about overall goal of game
- Information in relation to whether or not these are blocks in Bronzeville
- Tutorial for gem store once money runs out
Build it! Bronzeville is an opportunity for users to think about what their neighborhood lacks, and offer suggestions on how to make their neighborhood better. By providing a story and information about the type of information gathered, will build a strong community element for users.
Final Report
Here is a final report of the results with notes from each CUTGroup test session, followed by each tester’s responses, and copies of our e-mail campaigns and the questions we asked:
The raw test data can be found below with the written answers from every tester.
