< Back || Journal Entry #17 ||

Repent! Repent! For the end is near!

Well, I'm leaving to go back to Lethbridge tomorrow, 'cause I get kicked out of rez on Monday for the start of the next term. That's it, folks, that's all she wrote. What's been happening in the last little while (some of this stuff happened before last week, but I forgot to mention it before):

* I've started writing my final report, which I have to do for the CDMP. Basically, it's a summary of my activities this summer, and what I've learned. I tried to put particular emphasis on what I learned about human research in CS, in the hopes that it'll be useful to future (or past, or present) members of CHISEL. It'll be posted up on the site eventually, but you should read this bit, I'm really quite proud of it. It's a list of mistakes that you should not make when doing a user study:

1. “It’s for the good of science” is not a good way to recruit people. People are busy. They typically require an incentive to give up their valuable time.

2. Once people have agreed to participate in your study, do not assume that they will participate in your study. Again, people are busy. Even when a modest incentive is offered, the effort that it takes to actually leave one’s house and arrive at a meeting place on time is significant.

3. Though it may be easy for you to wave away your prototype’s shortcomings, the same will not be true for participants. Some-hing so simple as not implementing a drag-select can affect the results of your study tremendously. Construct a succinct-as-possible to test your specific hypothesis. Make sure that every aspect of your interface works well. Not only does this approach eliminate the Prototype Effect, but also mitigates result invalidation due to extraneous factors. This is just good scientific method.

4. Do not assume that participants will follow the instructions given them. Asking someone to read and assimilate a ten-page document, no matter how well-written, is simply not an option. Providing click-by-click instructions on how to open a Zip file will result in an successfully opened Zip file about 50% of the time. Alleviate these tendencies by providing video or in-person assistance as often as possible.

6. Once you’ve come up with a study design, stick to it. It is surprisingly easy to answer “No, it doesn’t really matter” when a participant asks you if they absolutely have to complete a section of the study. This is a good way to invalidate your results. Conducting the study in an “official” environment can lessen this phenomenon.*

7. Just because subjects do not exhibit the reactions that you were hoping for does not mean that their results are not valuable. Again, this is just good scientific method. Even completely off-track responses can provide interesting qualitative evidence.

8. Unless you are holding a participant’s 100% completed study in your hand, do not assume that they will complete the study.

* If you are consistently running into this problem, perhaps it’s time to consider changing your study design. Asking a participant to organize a group of photos in three slightly-different environments seems like a very good idea on paper. By the third round, however, participants are not likely to be enthusiastic about the task at hand.

* I made up a pack for present/future CHISEL members including all the media that Trish and I created/used during the course of this study. This included all the logos and stuff that I made, flyers, handbills, the CHISEL timer, and the ethics board proposal. It should be up on the CHISEL page soon.
* Later today I have one last participant interview for this round. Wish me luck.
* Trish and I have decided to continue recruiting/testing participants into the fall, seeing as we're not going to get into CHI anyhow, and we recently learned that we can indeed offer monetary compensation to participants. *That* should help a lot.

Well, fill you in on the final end next week. Till then, toodle-oo.

posted by Kim Hansen on August 18 at 3:00 PM MST