Phase 3 · Evaluation Results
For our study, the user population was a rather specific demographic. A general description of a participant would be a mid to late twenties male with experience playing or coaching football. There were a few outliers, of course, but this quality of the participants was nearly uniform.
Aside from this age and gender statistic, there were other differing qualities. Most users had regular access to a computer, but they had varying degrees of familiarity with computers. The level of participation in football as a sport also varied from those who played in little league to those who coach at the collegiate level.
Before the analysis, there was a generally positive trend in regard to coaching software. Seven of the eight participants strongly agreed that a playbook was vital to coaches and players. This is, of course, to be expected. Similarly, the same ratio of participants agreed or strongly agreed both that they would be willing to adopt a virtual playbook and that such software can offer coaches the ability to better communicate plays to their players than paper play books.
Thankfully none of the subjects were dropped from participating. Aside from minor nuances, such as joking and getting sidetracked with the subjects of sports and computers in general, the users' input remains intact.
Results
While all of the participants did agree or strongly agree that they would be willing to adopt a digital playbook, the responses dipped into neutrality and disagreement when it came to the suggestion that software might offer coaches the ability to better explain plays than the traditional paper playbook. Recall from the background survey that the same questions were answered with more positivity prior to using the software.
While no disagreement occurred for the final question, two participants slipped into neutrality when asked whether they would likely use a future version of iCoach in place of traditional paper playbooks.
The most obvious conclusion is that some aspect of the process of completing the assigned tasks changed the perspective of the users in a negative way. The specific reason for this is varied, based on the short response given by each user.
One user, for example, claimed "There were some kinks that needed to be ironed out, but the proposed functionality is good." This places the burden more on the quality of the prototype, while acknowledging the potential of a full-fledged application. Similarly, one user stated, "I wasn't sure why the players were different sizes. I had trouble deleting routes." This issue is specific to the prototype. These flaws, while minor in the eyes of the developers, seemed to have a distinct effect on the users. It can't be determined with any degree of certainty that the concept itself would create a negative experience.
Problems
The problems expressed by the users were varied to say the least. The majority of the complaints, however, had more to do with the shortcomings of the prototype than the philosophy behind the finished product. There were complaints on the implementation of certain features (such as the "Create a Play" interface) which had more to do with the prototype than the finished product it intended to represent. One user noted, "A little bit clunky to use. Maybe should auto-align the player icons when making a play. I'm OCD." The prototype's version of the playbook is somewhat bare-bones. By reading between the lines, however, certain issues can be revealed which would also apply to the finished product.
An example of such an issue is the process of returning to the opening tutorial. One user explained, "I felt overall it was a good app. It has a good concept at its core. It has some flow issues. It's hard to get back to the home screen, for example, and read the tutorial again. I couldn't figure it out.. so it was sort of figure it out as you go along." This feature wasn't implemented for the prototype, but is certainly something of importance to consider for the final design moving forward.