For our thirteenth Civic User Testing Group session, we tested mRelief, a website that helps residents check their eligibility for social services benefits in Chicago and Illinois. This in-person test took place at one of the Connect Chicago locations – Chicago Public Library King branch at 3436 S. King Drive in the Douglas community area.
The mRelief team was interested in testing these things:
- Language: We heard that language is very important in understanding if someone is eligible for social services. We wanted to see if testers understood terms such as: gross income (what to and not to include), value of assets, household size, etc. The mRelief team also uses pop-overs and wanted to see if testers use them and if they were helpful.
- SMS: We wanted to test the text-messaging platform to see if users found it easy to use and how long each session took to receive and send messages
- Website usability: In addition to testing SMS, we wanted to see how testers liked the mRelief website and how easy it was to use.
Segmenting
On January 21, we sent out an email to all of our 834 CUTGroup participants. We wanted to know if they would be available for an in-person test on January 29 for about 30-45 minutes. We asked some screening questions to gather information. We wanted to focus on testers who were currently enrolled in or qualified for social services programs. We also wanted to focus on residents of the Southside of Chicago
We ended up having 11 testers participate in this test. In addition, 1 individual from the library wanted to be involved in the CUTGroup, and therefore, we tested with him, but did not include him in the results because he would not have qualified for the test.
Screening Questions
We heard form 71 CUTGroup participants through our callout for testers. We received a lot of good information just from the screening questions. Here is a look at what we learned:
- 90% of CUTGroup respondents said they have a smartphone
- 33% of respondents are currently enrolled in social services programs
Test Format
Due to the personal nature of discussing social services benefits with testers, we decided early on that we wanted to do one-on-one proctoring. We had a total of 7 people from Smart Chicago and mRelief help with welcoming testers and proctoring.
This was the first test that we were not only testing a website, but also wanted to test the text messaging platform. Testers were asked to go through both processes, and we wanted to see how they compared and if testers found one platform easier to use than the other.
Results
8 out of the 11 testers were currently enrolled in social services benefits, and all of these testers were enrolled in Food Stamp benefits. The 3 testers who were not currently enrolled seemed to have qualified for social services benefits based on their household size and income.
Language
We learned from this CUTGroup test that testers found the form(s) when checking their eligibility to be a simple process. 7 out of 11 testers (64%) mentioned that the questions were easy to understand or straightforward. Although most of the language was very clear, however, only 1 tester clicked on a pop-over to find out more information. Therefore, any question that requires additional help text should be added after the question.
In addition, some testers did not notice that some questions might ask for monthly income vs others that ask for annual income. These are key pieces of information that should be made clearer in the question format.
SMS
Out of 11 testers, 10 testers (91%) do not pay per text message.
Only 5 out of 11 testers (45%) thought that the questions were “easy” or “straight-forward” when going through the text messages. The biggest thing we heard was that testers were “frustrated” or “annoyed” that they were unable to fix a mistake and had to start from the beginning.
Lastly, some testers received messages out of order, and some terms were split up between messages (ex: “Medicaid”).
1 of our testers, Simple One (#9), experienced a delay between text messages that lasted anywhere from 1 to 2 minutes. She thought it was “annoying” that she did know when the end of the questions would be. 3 testers thought the process was pretty fast!
Website usability
When reviewing the homepage, 6 out of 11 testers (55%) mentioned the graphics. 3 testers felt that the icons were very intuitive and understood that they connected to the social services benefits. The other 3 testers did not think the icons were as intuitive. 4 testers thought the graphics were icons that they could click on.
Most of the testers had a clear direction after reviewing the homepage. The majority (9 testers) clicked “Get Started” while the other 2 testers clicked on “Programs.” It was great to see that testers had a clear path in using the website that got them to the most important part — checking their eligibility for social services.
When checking eligibility, we noticed that a lot of testers liked the idea of the calculator to calculate income, but most testers did not use if because they had already answered the questions before coming across the calculator located at the bottom of the screen.
Documents & More Information
We wanted testers to tell us how important it would be for mRelief to provide additional information about documents that they might need to bring, or money that they would receive from the benefit. We hoped this would be a help for mRelief to decide what the next steps for the website might be. Here is what we learned:
How important is it for you to know what documents you need to prepare for an interview with a caseworker to receive public assistance?
5 Very Important 100% (11)
4 Important 0%
3 Neutral 0%
2 Slightly Important 0%
1 Not Important 0%
Is it important for you to know why you do or don’t qualify for public assistance?
5 Very Important 82% (9)
4 Important 18% (2)
3 Neutral 0%
2 Slightly Important 0%
1 Not Important 0%
How important is it for you to know how much money you will receive from public assistance before going to file with a caseworker?
5 Very Important 46% (5)
4 Important 9% (1)
3 Neutral 9% (1)
2 Slightly Important 18% (2)
1 Not Important 18% (2)
How important is it for you to know how to file an application for public assistance?
5 Very Important 91% (10)
4 Important 0%
3 Neutral 0%
2 Slightly Important 0%
1 Not Important 9% (1)
Final Report
Here is a final report of the results with the analysis of the questions we asked, followed by each tester’s responses, and copies of other questions we asked:
The raw test data can be found below with the written answers from every tester.
