We conducted our ninth Civic User Testing Group as a part of a grant from the John S. and James L. Knight Foundation to build communication strategies to engage with targeted communities through Foodborne Chicago, an app that searches Twitter for tweets related to food poisoning and helps report these incidents to the Chicago Department of Public Health.
Here are the outcomes we will achieve through the Knight grant:
This project will result in improved communications strategies for targeting key cultural groups on social media. The team will conduct research activities to identify the best approaches for communicating with these groups, implement and test new strategies in Food Bourne software and release a report with the findings of the research.
Cory Nissen, a statistician who wrote the Twitter classification code for this project, mapped out Foodborne Chicago reports and showed that there is less of a response from the South side of Chicago. That is to say, of all of the people we tweet, those from the South side are less likely to click and complete a report. Through this test, we were interested in learning more about how people use Twitter and if there are differences among communities and networks. We are taking the information we learned from this test to make changes to Foodborne Chicago and change our communication with Twitter.
Here is a list of questions we wanted to answer through this test
- How do users feel about responding to a random tweet?
- Does the form require too much information?
- Would users feel better or worse if there was more Chicago Department of Public Health or 311 visibility?
- Why might residents on the South side of Chicago not respond as often as residents from other Chicago neighborhoods?
Segmenting
On April 9, we sent out an e-mail to all of our 754 CUTGroup participants. We asked them if they would be willing to test an app on April 14, 2014, that helps report food poisoning incidents to the Chicago Department of Public Health. We asked some screening questions to gather information, and chose a group of participants who are Twitter users, and were primarily from South side neighborhoods. This in-person test took place at one of the Connect Chicago locations – Chicago Public Library Blackstone Library at 4904 S. Lake Park Avenue in the Kenwood neighborhood.
View CUTGroup #9 – Foodborne Chicago Test Participants in a full screen map
Test Format
We did a focus group-style test with three 30-45 minute sessions with 5 people in each session. We had a couple of different topics to discuss including introductions, general Twitter use, Foodborne Chicago tweets, the Foodborne Chicago form users have to fill out, and a general overview of the project.
First, we had testers answer questions via a form so they had a chance to form their own opinions before the group discussion. Next, we asked testers to tell us their answers and then we asked additional questions as part of a group discussion. We had some very interesting in-depth conversations, not only about Foodborne Chicago, but also about Twitter and social media use in general.
Here is the script I used during the test. I asked most of these questions in each test session, but I did choose to leave some questions out when the topic was already covered by tester’s responses.
We videotaped and audio recorded the tests to take notes from later.
Results
How do users feel about responding to a random tweet?
Foodborne Chicago uses computers and code to search Twitter for tweets related to food poisoning in Chicago. Then real humans review the tweets and @reply back to people with a link to the Foodborne Chicago page and form. We were interested how users might react to receiving a random tweet from someone they do not know. We gained a number of key insights.
Twitter is used as a private communications network for the majority of our testers. Twitter is used as a private means of communication, even though their tweets are public. Twitter users tend to connect with people they know or who are in their networks. Most testers were not sure about responding to people they did not know because most of the time they tend to be spammers.
Twitter handle & logo helps. We showed testers the above tweet, which was a commonly used response to user’s food poisoning tweets. A lot of people reacted strongly and were uncomfortable clicking the link for multiple reasons, including the fact that:
- “This might help” was not descriptive
- Testers did not know what foodborne.smartchicagoapps.org led to
- Testers were surprised that someone they didn’t know would tweet at them about food poisoning
Even though this was the case, multiple testers remarked that Foodborne Chicago still looked legitimate because of “Chicago” being part of the name, and the “clean and simple logo.”
Tweets should have three elements: compassion, question, and an official element. We shared two additional responses (see above) that Foodborne Chicago has tweeted. Some testers felt more likely to respond to strangers if they pose a question rather than a statement. Overwhelmingly, testers would be more likely to respond to the first option. Here are some responses we got about this tweet
Things that Matter says that this response
“…ties the app, situation, and solution to the Chicago Health Department, which is important.”
Spouse of Food Poison Victim says,
“it sounds authoritative”
Foodlover definitely would choose the first one because
“it shows compassion and ties it to a public health issue.”
Does the form require too much information?
When we created this test, we thought that the form might have too many fields and users would not want to fill it out. We learned that users have little issue with the form, and users liked that it was simple and did not require too much information. Bakunin thought that it was his “civic duty” to fill out this form to make sure others do not get sick. While we were not asking too much information on the form, testers still had question about our privacy policy and the process.
Would users feel better or worse if there was more Chicago Department of Public Health or 311 visibility?
Testers were interested in seeing a stronger connection to the City of Chicago, the Chicago Department of Public Health and 311. During the test, we directed testers to look at the Q&A page that gives information about what happens after the form is completed and the privacy policy. The majority of testers liked that by submitting this form they would be creating a 311 request and would receive a tracker number. One tester, B, told us:
“I wasn’t aware that this existed, I always call 311. I never bothered to go to Web site. [It] confirms that this is official.”
Testers thought that seeing relevant logos on the Web site would emphasize these connections to the Chicago Department of Public Health and 311. Testers did indicate that language about “The City” on the form was not descriptive, and they wanted more information about the process.
Why do residents on the South side of Chicago not respond as often as residents from other Chicago neighborhoods?
The most essential answer to this question is that we are not in their network. Twitter users see Twitter as a way to connect with people they know, or are within in their circles whether that be their friends, their work, or their community. From this test, we learned that we need to find ways to become part of more networks by engaging with more people on our topic. By connecting with individuals and organizations and communicating what Foodborne Chicago does, we will be able to grow our network and become a part of other people’s.
Alpha Man thought we could improve Foodborne Chicago by:
“Advertise this information out to the community so that people know about it.”
We are interested in connecting with community groups and other organizations that are trusted in these targeted networks to help facilitate information and increase the credibility of Foodborne Chicago.
Next Steps
We are taking steps to make changes to increase submissions of the forms:
Change the Tweet
We immediately changed the tweet language to include compassion and an official component – the Chicago Health Department. We also added a Twitter card in order to provide more detail about information being sent through the Chicago 311 service.
URL Update
We also changed the URL, by removing the smartchicagoapps, to www.foodbornechicago.org. By including the smartchicagoapps.org component of the Web site, testers were unsure about clicking the link.
Be more Official
Here are some ways we are thinking of doing this on the Web site:
- We changed the header information to include Chicago Department of Public Health
- We are considering ways to incorporate the relevant logos on the homepage
More about Process
Testers felt more comfortable with the form and the process of providing their information after reading the Q&A page. We made a few steps to make this clearer:
- We added a link to our privacy policy right on the form
- We changed the language on the form from “How can the City reach you?” to “How can the Chicago Department of Public Health reach you?” in order to be clearer about who would contact users
- We are planning to make changes to the home page to show the process in a couple of easy steps
- We want to add data about the number of food inspections resulting from Foodborne to header on homepage because testers wanted to see a tracker of quantitative data as a way to show impact of what this site has done
Being part of more Networks
The final step is figuring out ways to be part of more people’s networks so that they feel comfortable clicking the link and submitting the form. We are interested in learning about common organizations or individuals that people have in common to see about who we can reach for a large impact. By connecting with these organizations or individuals, we can build credibility in different communities. Lastly, over the course of the grant, the Foodborne Chicago team will also :
- Interview several community leaders connected to the part of the city that has been unsuccessfully contacted, in conjunction with Smart Chicago Collaborative
- Implement new strategies for interaction on social media, based upon feedback from the community leaders and focus groups
- Collect key metrics and visualize outcomes
- Publish a paper, in conjunction with the City of Chicago Dept. of Public Health and The Smart Chicago Collaborative that document the outcomes of the study
Final Report
Here is a final report of the results with notes from each CUTGroup test session, followed by each tester’s responses, and copies of our e-mail campaigns and the questions we asked:
The raw test data can be found below with the written answers from every tester. Notes from the group conversations can be found in the final report.
