Month: September 2015

For our seventeenth Civic User Testing Group (CUTGroup) session, we tested the Ventra Chicago mobile transit app. The Ventra app allows riders to manage their Ventra account, buy mobile tickets for use on Metra, get notifications about their account, and other features.

This was an exciting opportunity for our CUTGroup testers to be the first to test the new app and provide feedback that could affect many public transit riders. This is a snippet about the test from the Chicago Tribune:

The CTA, Metra and Pace said they are working with a consultant with expertise on testing apps, Smart Chicago Collaborative’s Civic User Testing Group, to iron out undisclosed issues with the Ventra mobile-ticketing app, which is designed to let commuters purchase fares and manage their Ventra accounts from their smartphones.

“We get to make a first impression once, and we want the Ventra app to make a great first impression on CTA, Metra and Pace customers,” said Michael Gwinn, CTA director of fare systems.

This test had two parts:

• A remote test where CUTGroup testers downloaded the beta version of the app and then used it in their daily lives
• An in-person test where testers gave direct feedback and performed specific tasks in-app.

These two parts allowed us to collect information about app usage away from the test environment, and also have great conversations with our testers about what was or wasn’t working for them.

Here is what Tony Coppoletta, Manager, External Electornic Communication of the Chicago Transit Authority said about what the Ventra app team wanted to learn from this CUTGroup test:

The CTA has an incredibly varied set of use cases and we look forward to how all our riders can benefit from this app’s feature set. In addition to some public outreach to invite people to test our app before public release, we want to try and learn from the diverse and experienced testers the CUTGroup offers would be an excellent addition to our real-world user testing of the app. The focus is to trap bugs, ensure transactions happen as expected, good usability and a solid UX overall.

Segmenting

On June 28, we sent an email to all of our 880 CUTGroup members inviting them to complete an initial questionnaire about this new test opportunity. We wanted to know if they would be able to participate in a remote test component between July 7 and 15, and then participate in an in-person session either on July 15 or July 16, 2015. We asked a lot of screening questions to gather information.

We wanted to know what device they had, what software version they were using, and whether or not they had a Google Play or iTunes account. This was important, because they would have to be able to download the test version of the app. We also asked a lot of questions about using Ventra such as whether they had an account, if they load passes or value, if they use public transportation and what modes of transportation they use.

Screening Questions

During our initial call-out for testers, we heard from 91 CUTGroup members. We received a lot of good information just from the screening questions. This is what we learned about their main modes of public transit:

95% of respondents use CTA trains
90% of respondents use CTA buses
44% of respondents use the Metra
25% of respondents use Pace

We were looking for around 30 testers for this test, and wanted about half of the group to be Android users and the other half  iPhone users. We were also interested in getting a mix group of Ventra users and testers who ride different modes of public transit (CTA buses, trains, Metra, and Pace users).

Test Format

This test was the first time we told testers in our initial call-out email what app we would be testing. Normally we do not disclose this until the time of the test to avoid having people visit the website and app before the test.  However, we had a lot to tell our testers about the instructions and updates they were going to receive from Ventra about downloading the test version, so it made sense to give them all of the information upfront. We also had to make it very clear that testers were going to use or create a real Ventra account and make real purchases.

We offered testers a total of $40 for helping us, instead of the normal $20 per test. The reason for doing this was because they were doing two big parts  1) downloading the beta version and using it remotely, and then 2) participating in the in-person session. We respect our testers’ time and feedback, and we understood how worthwhile it was to have them participate in both pieces of the test.

Once testers were selected for a test, we emailed them to check their availability for the in-person session, and gave them instructions about how to get the beta version on their devices. Here is a part of the email we sent:

Thanks for your interest in our CUTGroup test of the new Ventra mobile transit app. You have been chosen to participate in this test. Here are some details about next steps:

• If you have an iPhone, please download the TestFlight app from the iTunes store. You will need this app before you can download the Ventra app.
• If you have an Android device, please make sure you have a Google+ account. You will need to join Google+ before you can download the Ventra app.

Early next week you will get an email invitation directly from Ventra to download the app. Even though this is a CUTGroup test, the fares you buy in the app are real—when you you load fare or tickets in the app, your chosen payment method will be charged and the fare will go into your Ventra Card’s transit account or toward real mobile tickets for Metra. Also, you will receive emails from Ventra giving you extra info or updates.

We understood that by asking testers to download the TestFlight app or get the beta version through their Google+ account that this would be adding extra steps that our testers might not be familiar with. We communicated regularly with testers to make sure they were able to download the app and to assist if any trouble arose. Only 1 tester could not download the app remotely, but was still invited to participate in the in-person session.

Remote test

Testers were invited to start using the app in their day-to-day as soon as they received the app invitation. We sent testers questions asking them about their experience, but then they were also invited to submit to Ventra’s bug report form directly, which some testers did. From the remote test, we wanted to gather an understanding of what the testers did first (create a new account, login to an existing account, etc), whether or not they added value, while also getting some feedback about their experience,  how easy they thought it was to use, and ways the app could be improved.

In-person test

After the remote test, we held two in-person sessions at The Chicago Community Trust on July 15 and 16, 2015 to have more in-depth conversations with testers about their experiences and to watch how testers completed specific tasks such as use “Trip Tools,” register a new card, purchase a Metra ticket. We asked testers to bring their own devices to complete part of the test, but we also had test devices (both Android and iPhone) available to test making purchases or creating a new account. By incorporating the test devices, we were to test more tasks, but we understood that testers might not feel as comfortable using these new types of devices.

Results

Remote Test

27 testers completed Part 1 of the CUTGroup test, which was a remote test where testers used the Ventra app in their normal day-to-day routine. During the remote test, only 8 testers added CTA transit value or passes to their account, while only 5 testers purchased Metra tickets. Here are some comments from testers who did not make a purchase in the remote test:

ChgoKC says, “I tried to add value to the account, but it asked for my Ventra password, which I entered (it’s 4 digits), and it said the password had to be 8 digits. I tried my password for my bank account, but that was ‘incorrect.’ So not sure what password it’s looking for.”

Gr8fl4CTA said, “I haven’t made a purchase because I have a monthly pass. I do not like that you can only add $5 to the card. There should be an option to either fill out the amount-especially since most rides are $2.25 or $2.50. People shouldn’t be forced to put $5 on their card if they don’t need all of that.

Here are some responses from testers who made a purchase during the remote test:

CTA/Metra Commuter/Evil Genius says, “Purchasing in general is easy. Multiple times, I’ve purchased a ticket while walking to the train. I love the ability to pay with my Ventra auto-reloading balance.” 

Frequent CTA / Occasional Metra says, “Purchasing the Metra tickets was very easy – I only purchased one-way tickets so far using the app, but it seemed very straightforward in choosing which stops and identifying the ticket type I wanted. The conductor that checked my mobile ticket on UP-N was educated on the system and he had no trouble accepting the ticket; that was my only concern about using the Ventra app to buy a Metra ticket.”

Systems engineer CTA rider says, “I added a 30 day pass to an existing transit account and it went very smoothly. I’m also REALLY, REALLY impressed with how split tenders for payments were implemented. It’s really intuitive and I really like that feature.”

Three testers mentioned that sometimes the app was slow to load. Additionally, 3 testers either expected or wanted Near Field Communication (NFC) capabilities. Otherwise, testers had a wide range of responses when describing their overall experience using the app. Here are some improvements we heard:

• Show password
• Be able to transfer value from one card to another
• Improve the to back button so it does not kick the user out of the app
• Route planning in Trip Tools

When asked about the visual design of the app, the majority of testers (67%) thought it was “appealing” or “very appealing.” Here are some useful responses from testers about visual design and how it might be improved: 

CTA rider AC would change the color of the cursor from white to black or another darker color. CTA Rider AC says “It is really annoying that I can’t see where the cursor is.”

Occasional CTA rider says, “The alerts in the Trip tools are a very light font and not very easy to read.”

When asked about how easy it was to use the Ventra app, 22 testers said it was “easy” or “very easy” to use.

In-person Session

26 testers came to the in-person sessions (14 came on Day 1, while 12 came on Day 2). Two of these testers did not complete the remote questionnaire, but still provided great feedback during the in-person section.

Trip tools

There was a lot of conversation around Trip Tools and ways to improve this feature. A lot of testers saw this as a feature that would get them to use the app on a regular basis. Testers shared with us by using the phrase “Trip Tools” they were expecting more route planning features similar to using Google Maps. For example, one tester specifically called out wanting to see a map during this part of the test. Testers thought this feature would help them choose bus and train routes to get to locations, not only tell when the next train or bus is coming. Changing the name of this feature might assist with people’s expectations of the tool.

9 out of 26 testers found the Trip Planning feature to be similar to or on par with other websites or apps that they use. Here are some responses from testers:

Brown line rider / occasional UP-N Metra rider says that this is “Similar to what you’d get on the CTA website only this app looks more this decade.”

CTA rider (KB) says, “It’s good, has about the same functionality that many other apps have, and the alerts here are easier to find” because they are shown more prominently.

9 out of 26 testers thought the Trip Planning features on Ventra were better than other websites or apps they currently use.

CTA rider (AC) says, “I trusted it more because it was affiliated with Ventra.”

Daily CTA Rider (MW) says, “Actually easier than my other apps! And I think that may be because of simplicity and consistency on the screen and not use of a lot of different colors.”

6 out of 26 testers thought this feature was worse than other tools that they use. Here are their comments:

Frequent CTA / Occasional Metra says, “This is stop driven. I think in terms of routes. The other apps think in terms of route and not stops. Other apps locate you on a map and tell you what is around. Helps you find the bus and route you need to take.”

Riding the 20 every day says, “It’s similar, but the lack of map function is a problem.”

Metra tickets

Testers were asked to purchase Metra tickets on the test devices. 31% of testers said that they travel on the Metra once or twice a month. Out of our group, 5 testers (19%) had never used the Metra before. Here are the top things that testers liked about the process of purchasing a Metra ticket.

• Split payment
• Using transit value as payment
• Billing info saved
• Stops are in order of location on the line

A recommendation for the Ventra app team would be to test the Metra ticket functionality with residents who use the Metra as their primary form of public transit. Since our testers live in Chicago, most of them tend to rely on CTA busses or trains.

Overall

24 out of 26 testers (92%) said they liked the Ventra app during the in-person test due to how easy it to use (7 testers mentioned this), the Metra functionality (7 testers), the convenience (5 testers) and the visual design (3 testers).

Brown line rider / occasional UP-N Metra rider said that this app is going to replace the Ventra website for him.

Metra rider88 says, “It was really convenient!” Normally Metra rider88 has to go down to the vending machine at the train station to add value via credit card and this can be very inconvenient due to the lines that form.

Frequent CTA / Occasional Metra says, “I will ride Metra more often because of the app. I wouldn’t have usually considered it as an option.”

Blind Transit rider says, “It has the promise to provide me with access on the go to the information my Ventra pass has. [This app] can be more accessible and useful than any other available option as a blind transit rider.”

18 testers (69%) said that they believe the Ventra app fits into their day-to-day routine. Here are some responses from testers who said, “Yes” the Ventra app fits into their day-to-day routine:

ChgoKC says, “It’d be similar to the parking apps, where you’d keep it around and put on the home screen of your phone.”

Mischievous Metra Maverick rides the Metra daily and says this “eliminates the need to look for a soggy ticket that is ruined halfway through the month.”

Geek Sheek says, this will probably be “one of his top five apps.”

Frequent CTA Rider JH, Occasional CTA Rider and CTA Savant  do not think that the Ventra app fits into their routine because they are looking for use of NFC technology so that they can use their phone at the turnstiles to get on trains or busses.  In addition, not all testers might use this app because their transit value gets loaded automatically and they might not ride the Metra or add additional value regularly. Gr8fl4CTA is not sure if she will keep the Ventra app because the trip planning does not work well enough, while TA is not sure because she knows her daily value and would not need to use trip tools often. TA says, though, “It is handy for adding a value and getting a pass.”

Updates

When we are onboarding new developers for a test, we emphasize the importance of the CUTGroup motto: “If it doesn’t work for you, it doesn’t work”.  Sometimes it can be a challenge for developers to invest the time to actually make changes based on feedback. That was not the case here. Here’s some of the updates they’ve told us they’ve made or are working on based on the feedback from testers:

• “Trip Tools” has become “Transit Tracker” to more accurately represent what this feature does. The addition of being able to do trip planning is something that the Ventra app team expects to do in the future. Also as soon as the user clicks on “Transit Tracker” their “Favorites” and “Nearby stops” are right at the top of this page
• Accessibility continues to be a priority, and one CUTGroup tester, Blind Transit Rider thoroughly helped the Ventra app team with their app’s accessibility functionality
• There is now a “Show Password” button to help people enter their passwords, and the cursor is now a dark blue that helps testers know that their cursor is in the correct field. The Ventra app team also wants to make the password requirements more prominent when creating an account
• The back button will be improved in order to be more in line with what Android users expect
• More information is going to be added for Metra in the Transit Tracker feature to give better information about when the next train is arriving

Tony Coppoletta, of the Chicago Transit Authority, said

Our work with the CUTGroup has proved to be an incredibly valuable experience as part of our test plan for the Ventra app—both through the thoughtful feedback we received via the remote test and in affording us an opportunity to sit down face-to-face with a diverse range of riders of CTA, Metra and Pace and learn about users’ experiences together combining an open dialogue and structured testing.

CUTGroup is now a community of more than 1,000 residents in Chicago and all of Cook County who work together to make lives better through technology. This test has been an example of how these testers can be an integral part of changes to technology. It is exciting to see changes being made based on the direct feedback from testers.

CTA Rider says, “I love being able to help contribute to the development of this product.”

Route 66 Book Stalker liked testing something that is “important and impacts a lot of people.”

Final Report

Here is the final report with a look at all of responses for each question that we asked, followed by all of the testers’ responses to our questions.

Here is the raw test data: