It's a Jungle Out There: Practical Considerations for Evaluation in the City

Kellar, M. , Reilly, D., Hawkey, K., Rodgers, M., MacKay, B., Dearman, D., Ha, V., MacInnes, W.J., Nunes, M., Parker, K., Whalen, T. & Inkpen, K.M. (2005). It’s a Jungle Out There: Practical Considerations for Evaluation in the City. In Proceedings of the CHI 2005 Extended Abstracts, Portland, OR. 1533 – 1536. [poster]

This paper that although traditional methods of evaluation of mobile and ubiquitous computing research my be difficult to apply in dynamic and unpredictable environments like cities, the challenges are surmountable and field research can be crucial component of evaluation.

I am interested by the range of issues that the experimental control and the ability to observer behaviors, because they highlight the external factors that impact both research and adoption of mobile, ubiquitous computing technologies:

• Software: interrupted session due to bad Bluetooth connectivity, software failures prevented one participant pair from using the handheld at one point
• Material: battery power had to be carefully managed during the long study days.
• Weather: Bright sunlight made it difficult at times to view the handheld displays
• Audio and video: it was difficult to capture quality audio recordings due to background noise, which was in general far worse than that encountered during feasibility testing and pilots.