T8 Wizard of Oz Evaluations: Safe and Sound

Janet Read, Johanna Höysniemi
University of Central Lancashire, UK / University of Tampere, Finland

This tutorial is suitable for practitioners and researchers who want to know more about the suitability of Wizard of Oz evaluations for interactive products. The presenters have experience of delivering tutorials about evaluation methods at Interact2004, CHI2004, and IDC2004; but this is a new tutorial that covers a single specialist topic.

The emphasis in the tutorial is on producing sound data in a safe way. We begin with an introduction to Wizard of Oz studies, and then look at scenarios for which this method of evaluation is particularly useful. We discuss the value of the collected data and describe some scenarios for which a Wizard of Oz study might not be appropriate.

In the second half of the tutorial we focus on the methods that should be applied when carrying out a Wizard of Oz study; examples from our own work are used to illustrate different approaches. The tutorial closes with a discussion of the ethical concerns that need to be held in mind when doing this sort of study and we propose some tips and hints to minimise the risks.

Attendees can expect to participate by way of small group exercises, a practical exercise and a whole group discussion.