Return to UX/IA

Description Start-up in need of UX research on new web app.

 

Issues that may impede progress

  • Virtually no funds
  • Unknown project
  • limited staff

Process Requirements:

Facility to conduct study:

We found a shared office hosting space in West Seattle. The Office Junction was willing to allow us access on a weekend. This is typical off hours for the office so we had use of the entire office to ourselves and at a discounted rate.

Facility Cost:

The two day rental fee was 180 each day.

Equipment and Equipment costs

required for UX study:

  • LapTop
  • Projector
  • 2 Laptops
  • TechSmith Screen Grab software (or similar) installed on laptops

Staff Option: UX study group (Various students working in Usability program) from Seattle Central College. The project was marketed as a "workshop" in a "real world environment". The students were allowed to add the study to their resume showing actual results from a real world client. This required some more engagement on the part of the client but it was worth the end result and the substantial savings to the client.

The students were able to come out of college with an actual study under their belts. 

Process

Introduction

I‘m the founder of a company called GoodRoadNetwork. We offer tour promotion and booking services to musicians and venues across the US. We had a US launch coming up for a product we had been working on for some time and we knew we had usability issues. The product was developed quickly, and with our limited resources, there was no time for proper usability testing. The product was extremely complex and much bigger than originally anticipated.

The issues we faced around conducting a UX study

  • Need users to test: We were really new and had no marketing budget. That meant we had a really small user base. This required us to find and qualify additional potential users.
  • Needed staff to conduct study: As we could not afford your typical UX study and had limited funds to pull from and reward our users, we had to come up with innovative ways to conduct an actual user study to improve our low aggregation numbers. This meant we needed alternative means to get qualified usability specialists together to work on the project.
  • COST of professional facility and proper team. Again - Any professional usability study would be far outside of our existing budget.
  • Additional notes We had limited resources with which to pull from to raise money, meet people, connect with users.

Gaining an audience and testers

We put a call to people we knew that taught at the local colleges and universities.  We had previously done a series of lectures in support of this field of work with Seattle Central so I knew that the students were hungry for real world opportunities. We offered to present (do a lecture) for a chance to recruit volunteers at the end of our discussion. we took the GoodRoadNetwork team to one of the colleges and conducted a lecture on what it meant to work in the tech sector. After the lecture we asked the attending audience if any of those attending would be interested in helping us out on a ux study. We mentioned that it would be a chance to show real world experience on a resume as a usability team.

We committed to following up with any student that wanted to contribute their findings in a follow up UX report.  Initially the study was meant to gather ux minded people to document the study, assist on witnessing the findings, and essentially discuss the project with the client in a relaxed setting. The determination with the client was that any help was better than no help.  In addition - If we could not get enough Artists and/or band members together to test - we still had people that were unfamiliar with the product that could engage and, potentially, bring up questions around UI functionality that may not have made sense. These people had a “First-eyes-on-product” overview.

Result: We had a surprisingly positive response from the initial workshop request and confirmed about 25 people over the two day study.

Equipment and location.

Without the funds to hire a staff & UX lab, we looked to shared office space and conference rooms. Shared office space won out because it meant we could separate the users from the testers. We found Office Junction in West Seattle to meet our needs nicely and they were willing to give us the weekend (a time they were traditionally closed), which meant we would have the whole facility without interruption… AND - As part of the service they had a projector that we could use - Even better.

We blasted our social networks with request for study participants and bribed bands by showing up to their gigs in person and talking to them about our service. We reached out to all our friends and anyone that would listen. We even got a shout out from the West Seattle Herald on their social networks!

Once we had a few interested parties, we scheduled two successive Saturdays to conduct the study. We used MailChimp to create an email list to communicate with the testers and subjects all together so that we could keep everyone engaged in a singular group.

We had no problems getting a couple laptop’s together and I installed TechSmith screencast software to record the sessions. In addition, we signed up for a temp account with Blackboard.com, which allowed us to screen share with multiple users. (You don’t need blackboard - Any meeting software that allows you to share your screen would be sufficient).

Day before the actual interaction, we communicated to everyone the details of the following day’s events and went over general process of what to expect.

DAY OF OPERATION:

We brought snacks for people on both days because we knew each study would last over 8 hours a day. I set up the equipment and marked down on a dry erase board in the conference room, what we wanted the observers to focus on and take away from the study.

In one office I set up two chairs and a laptop that I logged in to the Wi-Fi. I created an event in Blackboard online that people could join and observe. I set up the tech software to record the screen and made sure the video and audio worked.

In a conference room down the hall I connected another laptop to a projector that we used to sign in to the blackboard event and observe. We turned off the audio so that we couldn’t interrupt the broadcast which meant the second laptop in the conference room could only observe what was happening on the first laptop.

Diagram:

Room Layout

With this set-up we could observe everything that was happening on the screen in office one (Including a face grab of the individual using the laptop). In the conference room we projected those events on to the wall. The rooms were far enough apart that they couldn’t hear comments or discussions happening outside of each room.

The moderator conducted the study in the office with the user being tested. The people in the conference room were able to view it in real time, take notes, and form observations about how each person was using the product.

Immediately, we saw results as people in the conference room watched people go through the product. They found themselves laughing, groaning, and occasionally calling out to the screen. This was the exact reason we needed to have significant separations between the two rooms. In truth, we could have conducted the entire review online - However, we would not be able to make sure people were engaged and could not discuss the findings as a group as easily.

Take away:

First Saturday

We found that volume suppression is extremely important because even though the observers knew better… they sometimes couldn’t help but forget that others might hear them. The distance between the office and the conference room proved to be brilliant.

Because they were still in the same building. It was really cool for the test subjects to be able to come back and talk with the students. This wasn’t normal - But it proved to be a good interaction in this scenario as the students acting as observers got to ask follow up questions about the service and clarify the understanding on certain activities. This was not planned - Instead - it was a happy accident. ALSO - The bands got to promote themselves and feel like rock stars in front of an audience. I will chalk this up as a win.

Second Saturday

There was something I read when the second Saturday started, that reminded me that Murphy’s law would not be forgotten. Most of our bands cancelled last minute. We instead took turns engaging students and ended the day by reviewing 2 of our 6 bands that showed up.

This was an interesting study and a huge win for the students as it really helped to highlight the difference between watching someone use a system, and engaging the system as a user. Even though the students had some experience through watching people use this product for 8 hours of testing the previous week… once they were actually in front of the computer - Everything changed.

The students were really engaged as observers and some were even able to engage as moderators.

Follow up

Once the study was done, we had 16 hours of video to go through and evaluate. I spent an entire day uploading all the video into a cloud drive shared with the students. We then asked any of the students interested in compiling a UX analysis, to show the findings of the two day study in a report. We linked to several templates of ux findings and found that several of the students were extremely happy to follow up.

ANALYSIS & POST MORTEM REPORT

The best part of all of this is that we had an extremely beneficial study for the cost of snacks and $360 for the ux space rental. This beats the 10-15K for a professional UX lab. Obviously - this isn’t a replacement for a professional study. HOWEVER - this is a great way for small companies with limited budgets to get on top of their usability and see results rather quickly comparatively.

Things to Consider

We had volunteers that came as observers from the college. If we had to pay them for their service - this could have cost much more. Lunches for everyone in the form of a couple large pizzas and snacks totaled less than $100

Expectation:

  • We used volunteers from the college as observers ideally you would want your staff that was involved in creating the product to be the observers as it will help them to see the problems in real time.
  • Generally each person that is testing gets some sort of compensation for their time. This will substantially increase the cost as that compensation is usually $50-100 per user and with a one hour test - you are looking at 1000 to 1600 for two days.
  • The moderator is not typically an employee. Expect to pay a UX researcher for this time.

NOTE: This was NOT a professional review. However, it was a very big win with minimal cost.

Client Feedback

We at GoodRoadNetwork ere able to vastly improve our web site by eliminating some problems that we didn’t even know we had. When everything was said and done we were able to classify the issues and prioritize the fixes:

  • P1 (Catastrophic fail)
  • P2 (Major Road Block)
  • P3 (Cosmetic and minor issues)

Noteworthy:

Several of our students completed follow up reports with recommendations as to how the pain points could be alleviated through improved design patterns. This proved really useful and they were able to put the case studies online as part of their portfolio. We only asked that they changed the name and not display logo’s for the design process. While we were extremely open - I do not imagine most companies would openly and willingly share that type of information with the public. This was something I agreed to as part of my commitment to Seattle colleges and giving back to extended education of students in the tech sector.