SNIF-ACT: Instrumentation and Protocol Analysis

Our method begins with the construction of ecologically valid tasks—that is, tasks that resemble what people do in real, culturally significant situations. Controlled laboratory experiments are then conducted using tasks derived from this task database. The laboratory experiments collect data using an eye tracker, logging software that collects all user interactions with a WWW browser, and video recordings of think-aloud verbal protocols. These data are coded by automatic means and by hand into a comprehensive trace of states and events representing the interaction of user with the WWW. Computational models of user cognition and perception are then developed to simulate—as accurately as possible—the observed user-WWW interactions.

Performance on the tasks was recorded using an instrumentation package that included: (a) WebLogger, which is a program that tracks user keystrokes, mouse-movements, button use, and browser actions, (b) an eye tracker, and (c) video recordings that focused on the screen display. Keystrokes, mouse-movement, browser controls, and browser actions were recorded using our WebLogger system. Eye-movements are handled by our WebEyeMapper system. At experiment-time, a user works on a specific task with a WWW browser, while data are collected from the user as shown in the figure below.

As the user performs various actions with the browser, including navigating to different pages and scrolling within pages, these events are recorded on videotape, by the WebLogger program, and by an eye tracker resulting in the databases enumerated in the figure. WebLogger instruments the WWW browser and records all significant events and display states. [Details of the WebLogger program is described in this paper.]

A fragment of the WebLogger event log is shown below.

One basic problem is that eye tracking data must be mapped onto data recorded by WebLogger in order to determine on what content the user was visually focused at any given time, even though the user might be scrolling the window or moving it on the screen. This is called the points-to-elements mapping problem and is solved by the WebEyeMapper program as shown below.

Every 1/60th second, the eye tracker records the point-of-regard of the eye, the inferred x,y screen point at which the eye is gazing, In order to analyze data from a user’s browsing session, WebLogger launches an instance of Internet Explorer and maintains a pointer to the instance of Internet Explorer. Using WebLogger event log data, raw eye tracker data, and content from WebLogger's content-saving feature, WebEyeMapper begins a "playback" of a browsing session. WebEyeMapper maintains an analyst-controlled simulation clock to coordinate the replay of WebLogger events and eye fixations. As the simulation clock advances, WebEyeMapper directs Internet Explorer to load the same Web pages that the user was viewing at the time indicated by the simulation clock, and directs Internet Explorer to alter its scroll position, window position, and window size as the user did at experiment-time. In this manner, WebEyeMapper restores the display state of the browser to the same state, moment-by-moment, as the user viewed it at experiment-time. WebEyeMapper can then take eye fixation points, align them in time with the simulation clock, align them in space with the browser window, and determine what is rendered in the browser at the time of each fixation. For each fixation, WebEyeMapper writes the fixation start time and duration, screen, window, and scroll system coordinates, element fixated, and element text fixated, to a database.

The Web Protocol Transcript shown below starts with a selection of interactions recorded by the WebLogger and adds to these (a) transcribed data, including viewed URLs, eye movements, verbalizations, and observed actions, which are presented side-by-side with (b) a model coding of the inferred cognitive action that is associated with the data.

Videotapes of users thinking aloud provide additional data about users’ goals and subgoals, attention, and information representation. WebLogger and WebEyeMapper data are used to produce a Web Behavior Graph A Web behavior graph is an application to WWW behavior of a problem behavior graph by Newell and Simon, (1972) and visualizes user behavior as a search through a problem space. Each box in the diagram represents a state in one of several problem spaces. Each arrow depicts the execution of an operator, moving the state to a new state. Double vertical arrows indicate the return to a previous state, augmented by the experience of having explored the consequences of some possible moves. Thus, time in the diagram proceeds left to right and top to bottom.

The WBG is particularly good at showing the structure of the search. Color surrounding the boxes in the diagram represents different Web sites. Oval boxes are distinguished in order to show hit lists from a search. An X following a node indicates that the user exceeded the time limits for the task and that the task was therefore a failure. A loop has been drawn around different problem spaces, showing how the users pass from one problem space into another as the operators in one become less effective. Evident in the figure is the hub-and-spoke structure of the behavior, in which the user follows a trail out from the hit list of a search until that trail goes cold, retreats to the hit list page, and finds another link to try.

More details on WBGs and Web Protocol Transcripts are presented in Card et al. (2001).