next up previous contents
Next: Results Up: No Title Previous: A Minimal Device-Independent Text

Subsections

   
Design of the Tests

In this chapter we will describe the experiment that was conducted to evaluate MDTIM. The goal of the experiment was to generate some hard data on the speed, error rate and typical learning rate of MDTIM. We also tested various input devices to see whether MDTIM writing skill gained on one device transfers to the others.

Input devices

Five different input devices were used. For most of the experiment we used Cirque EasyCat touchpad. To see whether MDTIM writing skill is device-independent we used four other input devices. These were Gravis PC Gamepad, Microsoft IntelliMouse, CH Products Trackball pro trackball and a regular QWERTY keyboard.

Next we give more details on the input devices and the way they were used in the experiment.

EasyCat touchpad

The Cirque EasyCat touchpad is shown in figure 5.1. The EasyCat was chosen because it is small enough (6,75 x 8,5 x 1,25 cm) to be conveniently held in one hand while the fingers of the other hand are used for input.

The EasyCat has two buttons that can be used as mouse buttons, but it can also emulate the mouse buttons through tapping the surface. Tap on the brighter colored area in the upper right corner of the active surface emulates the second mouse button. A tap anywhere else emulates the first mouse button. Dragging with button emulation happens through a double tap where the finger is left down on the second tap. EasyCat acts as though the first button was pressed until the finger is lifted again. If the finger drifts close to the edge of the touchpad during a drag, the finger may be lifted and placed elsewhere on the pad without breaking the dragging mode. These standard features of the EasyCat work conveniently with MDTIM and thus we did not bother to disable them. The dragging mode allows the user to utilize a kind of CapsLock feature. This is convenient if the user holds the touchpad in an orientation that does not allow pressing the buttons easily.

EasyCat is designed to be used while resting on a desk. In our test, however, the touchpad was held in the non-dominant hand to simulate using a small portable device.


  
Figure: Cirque EasyCat Touchpad [Cirque1999].
\begin{figure}
\begin{center}
\epsfig{file=figures/easycat.eps,width=10cm}\end{center}\end{figure}

Gravis PC gamepad

Gravis PC Gamepad is a small handheld game controller with four buttons and one tilting eight-direction control hat with a detachable stick. The Gamepad can be turned around and has a switch that reverses the controls for left handed users. In our test the stick was attached to the direction control hat. And thus the gamepad was actually a joystick.

Like the touchpad, the gamepad was also held in the hands of the user. The most common style adopted by the users was to envelope the gamepad in both hands and use the dominant hand to control the joystick while the thumb of the non-dominant hand was held over the buttons.

Microsoft IntelliMouse

Microsoft IntelliMouse has two buttons and the scroll-wheel that has recently become a standard feature in most new mice. We however needed only the old standard mouse functionality with one button.

The dominant hand was used to operate the mouse. A mouse does not suit well for use without a desktop and thus it was used on a desk.

CH Products Trackball Pro

As seen in figure 5.2 the Trackball Pro is a typical trackball construction with a big ball mounted on a box with buttons. The buttons are located symmetrically and their functions can be reversed so that the trackball is equally suited for left and right handed operation.


  
Figure: CH Products Trackball Pro [CH Products1999].
\begin{figure}
\begin{center}
\epsfig{file=figures/chtbpro.eps,width=12cm}\end{center}\end{figure}

While some trackballs are small enough for desk-free operation, the Trackball Pro is not one of them. Thus the trackball experiments were done while the trackball rested on a desktop. Again, the dominant hand was used for operating the trackball.

QWERTY keyboard

The only decision with the standard PC keyboard was which five keys to use for MDTIM. We chose the arrow keys that are in the inverted-T formation between the main key block and the numeric keypad. For the modifier key we used the right shift.

The keyboard was yet another desk-bound device. Like with the other devices, the dominant hand was used for operating the keyboard.

Software

For gathering data on the performance of users writing with MDTIM we wrote a program that implements MDTIM and logs the relevant events so that they can be analyzed later. The test program was written in C++ and it uses the Win32 APIs directly. We used the mouse driver supplied with the Microsoft IntelliMouse for all mouse-like devices. For the remaining two devices we used the drivers included in the Windows-95 distribution.

Figure 5.3 shows the interface that was visible to the test subjects during the test sessions. The largest window within the main window is a ''working space`` where the software displays the current MDTIM character during the input. Above the large child window we have two smaller windows. The upper of these displays the phrase that the user is supposed to write using MDTIM. The lower window shows what the user has written. The tall window on the right shows the tail of the data written to the log file.


  
Figure 5.3: The MDTIM test program.
\begin{figure}
\begin{center}
\epsfig{file=figures/program.eps,width=14cm}\end{center}\end{figure}

As can be seen in figure 5.3 the user can see both the actual input and MDTIM interpretation of that input. The actual input is the thin line and the MDTIM interpretation is the thick black direction sequence that we are already familiar with from chapter 4.

Although the idea was to test MDTIM in a form as simple as possible without any fancy signal processing, dictionary support for recognition, or other external means of improving the performance, some measures were necessary to make it usable at all. First the joystick data is gathered with some filtering as discussed in chapter 4. Second, the mouse data needed some filtering too. A minimum threshold was set for interpreting the difference between two consecutive coordinates to be a direction. Without filtering out the tiny movements the result would be an uncontrolled stream of directions as soon as the user touches the touchpad or mouse. When the acceleration was set close to minimum in the mouse driver settings, a good threshold seemed to be around three pixels. This filtered out most of the small unintentional movements while reacting soon enough when the user actually tried to do something.

While MDTIM may allow eyes-free operation with some input devices, this is not the case for novice users using a touchpad. Thus, as described above, the software gives visual feedback. While the benefits of auditory feedback may not be as clear as they are with the visual feedback, the software also emits a click through the Windows' sound system when a character is recognized.

Procedure

Goals

The experiment had two main goals. First, it was a demonstration of the MDTIM. Secondly, it was a longitudinal study that had two goals. Firstly, the rate of learning typical to MDTIM was be determined. Secondly, we attempted to gain a good estimate on expert speed and accuracy.

Test subjects

Because the experiment had a longitudinal nature, the number of sessions per subject was rather large. Given limited resources this means that the number of subjects had to be rather small. Acquiring unlimited resources for this project would not have been a good idea either given the inexperience of the experimenter. The result might have been a big mess instead of a controlled experiment.

Five volunteers were chosen from the staff of the Department of mathematics, statistics and philosophy in the University of Tampere. All were between 23 and 29 years of age. Our sample was clearly biased towards the best young brains in town at least in terms of mathematical ability. Clearly a sample like this cannot be claimed to represent the population in general. However, we did not have much choice. Ten sessions, preferably on different days during a two-week period, is too much to ask from a complete stranger without any compensation. Thus we chose to accept the warped sample to get the test done at all. Because the test subjects worked in the same building, the test sessions at the beginning or end of the day were easy to arrange.

Sessions

The first and last sessions lasted approximately one hour and the eight sessions in between lasted slightly over 30 minutes. All sessions had a 30 minute MDTIM practice session using a Cirque EasyCat touchpad. During the first session typing and handwriting speeds for all subjects were measured. During the last session MDTIM skill transfer test was done using Gravis Gamepad joystick, Microsoft Intellimouse mouse, arrow keys of a QWERTY-keyboard, and a CH Products Trackball pro trackball. Each input device was used for five minutes without any formal familiarization.

We tried to schedule the sessions for consecutive days. However to accommodate the test subject's summer holiday schedule we sometimes conducted two sessions each day with a minimum of five hours between the sessions. Weekends caused a two day break. Few one day breaks were also allowed if the test subjects had more important things to do.

Session 1

The test subjects were told that the goal of the experiment is to evaluate the writing method, not to test the subjects. They were also told that they may stop the experiment at any time if they for any reason wish to do so.

The first measured task was the typing and handwriting speed test. All subjects were presented with the beginning of the test phrase file in one window and were asked to type as much of it as possible into another window in five minutes. They were asked to avoid or correct all errors while writing as fast as they could.

The handwriting speed was measured using the same data, paper, and a ballpoint pen. This time the subjects were asked to write as fast as possible while keeping the text legible by most people. They were especially reminded to write all characters including punctuation.

Two subjects started with the typing test and three with the handwriting test. The purpose of these tests was to get a rough idea of the writing speed of the subject using two familiar methods for comparison with MDTIM.

Before starting the first 30-minute MDTIM practice session, the subjects were told and asked to try out the following features of the touchpad-MDTIM combination.

1.
Tapping causes the next recognized character to be upper case.

2.
Double-tap with finger left down on the second tap causes a Caps-lock effect.

3.
The left button on the touchpad can be used for the upper-case function too.

4.
MDTIM recognizes movements only if they are large and fast enough. Slowly moving the finger on the pad does not cause MDTIM direction input.

5.
Beyond the minimum threshold the length or speed of the strokes do not matter.

6.
For best performance fairly large and fast strokes should be used.

7.
MDTIM characters never have repeating directions.

8.
Repeating directions can be used for mid-character correction by waiting for the 70 ms timeout to occur before repeating the direction.

9.
The backspace character can be used to erase text.

Additionally the visual representation of the characters in the reference chart was explained until the subjects seemed to understand it.

The rest of the session 1 was just like sessions 2-9.

Sessions 2-9

The subjects were given the MDTIM reference chart (appendix B) and asked to start writing. They were also told that they should complete each phrase as fast and with as few errors as possible and that they should only rest between phrases. All questions were answered, but the test supervisor did not interfere in any other way. Each training session lasted roughly 30 minutes including all rest periods between phrases.

Session 10

Session 10 began with the already familiar 30 minute training session. After that the users went through four five minute sessions using different input devices. The order of the devices was balanced to avoid bias due to fatigue or undesired skill transfer between the new devices.

Test data

For the MDTIM practice session all 451 phrases shorter than 30 characters were extracted from the fortune cookie databases delivered with RedHat Linux 5.2. These phrases were presented to the subjects in random order. The phrases contained both upper and lower case characters of the Latin alphabet along with numbers, most types of parentheses, and other characters found on standard keyboards.

Often researchers are satisfied with a small subset of characters such as lower or upper case alphabet and space (for an example see MacKenzie's paper mackenzie_soft_keyboard). While this approach does give a good basis for comparing the input methods, the input speeds and especially the learning rates are unrealistically good. The non-alphabet characters may be rare, but if it takes the user a minute to figure how to input the @ character (or to deduce that it cannot be written at all), the average input speed may be reduced significantly. The more important result of a larger character set may become evident in the speed of learning and ability to retain the learned character sets. MacKenzie and Zhang recognize this fact in their 1997 paper on Graffiti mackenzie97. For these reasons we chose not to artificially limit the size of the character set. The test data is exactly what we found in the fortune databases.

Using a wider set of characters is expected to make the results look bad in comparison to figures measured for other writing method using fewer characters. However, having many characters including many relatively rare characters probably brings the measured speeds and learning rates closer to what we can expect in real-world situations. Therefore, we consider the wider character set justified even at the cost of some difficulty in comparing the results with figures measured for other writing methods.

Recorded data

During the MDTIM practice sessions most relevant events such as mouse, keyboard, and joystick input, recognized MDTIM directions, completed test phrases, etc. were timestamped and recorded into a file. The timer used has a resolution of one millisecond. However, because the events were timestamped when Windows 95's message passing architecture had delivered the messages to the logging procedure rather than when they were generated, the accuracy of the timer is not the limiting factor. Queuing delays greater than 50 ms should be rather rare and thus the events happened between 0 and 50 ms before the recorded time. These log files allow us to extract almost any features of the writing activity.

In addition to the automatic recording of events, we used a notepad and a pen to record user remarks and behavior during the tests.


next up previous contents
Next: Results Up: No Title Previous: A Minimal Device-Independent Text
poika@cs.uta.fi