![[Logo]](macshapa.gif){kind=small}
| Return to MacSHAPA home page |
![[new]](dbnew.gif){kind=small}
We are using MacSHAPA to code online reasoning in scientific laboratories. We video- and audiotape laboratory meetings, transcribe the meetings, and import the transcriptions into MacSHAPA. We then code (a) reasoning of the scientists, and (b) interactions of the scientists along a whole host of dimensions. For reasoning we have been looking at Induction, Deduction, Visual Reasoning, Risk assessment etc. We also code scientists' reactions to unexpected and expected findings and look at whether new hypotheses are proposed, the data is explained away etc. For analogical reasoning we code all Sources, Targets, and Mappings and relations among analogies. We code our data line by line and then use various aggregate variables that collapse over our line-by-line codings. For interactions we look at whether there are agreements, challenges, disagreements, supporting arguments etc. The goals of this research project and a summary of our initial findings are available in the following paper:
Dunbar, K. (1995) How Scientists Really Reason: Reasoning in Real-World Laboratories. In R.J.Sternberg, & J. Davidson (Eds.) The nature of Insight. Cambridge MA:MIT Press, pp 365-395.
A team of researchers from robotics, computational mechanics, human-computer interaction, surgery, and bioengineering are developing advanced planning, simulation, and execution technologies for the next generation of computer-assisted surgical robots. We have targeted hip replacement surgery, one of the most common procedures in orthopaedic surgery, because a robot can mill accurate holes in the femur and pelvis to receive the artificial implants. There are many HCI challenges in this project, and one of them is to provide tools and procedures for collecting data about the position of a bone during surgery that will allow accurate registration of the robot without additional surgical trauma to the patient or additional time under anesthesia. We are using MacSHAPA to encode videotapes of current surgeries to supply a baseline of the amount of time taken to do different procedures in the surgery. We have videotapes of four different views of conventional manual surgeries and surgeries using a robot currently in clincal trial. It is difficult to watch all four views at the same time for evidence of when a procedure starts or stops, but using MacSHAPA with QuickKeys (tm) makes it much easier than with previous equipment. We have not done any sophisticated analysis as yet, but use the timeline report format extensively to communicate with the other research team members about the analysis.
Air Operations Division of the Aeronautical and Maritime Research Laboratory uses MacSHAPA as a tool in the human factors engineering analysis of military aircraft systems. A critical requirement of any mission or systems analysis is the development of a task description and activity taxonomy of the mission or system under investigation. Task descriptions are generated through the analysis of video and audio recordings of operator behaviour in operational and simulated aircraft environments. Data at the task analysis level is entered into the MacSHAPA database. MacSHAPA's data analysis capabilities are then typically used to identify any differences in system behaviour (at the task level) that may have occurred in missions where performance was measured as good or bad. MacSHAPA is also typically used to identify common sequences of behaviour that are repeated, indicating areas in which redesign would lead to improved system performance. This approach has been used in studies of Long Range Maritime Patrol Aircraft and Seahawk Helicopter operations.
The Advising Workbench (AWB) is a project under way at University of Illinois to develop software that will support faculty members, professional academic advisors, and undergraduate students during the student advising process. During AWB development, we have used the PICTIVE method in videotaped participatory design sessions to elicit evaluative comments and software requirements from beta testers who have between one and five hours' experience with an AWB prototype. We use MacSHAPA to review and annotate the videotapes. Specifically, we use the VCR control functions to navigate each timestamped videotape, and we enter annotations into a text variable in the MacSHAPA spreadsheet. Having the timestamped database linked to the videotape makes it easy to navigate videotapes when we return to them after a break of several days. We then use the Query Language to perform searches for keywords in the text variable, and to create summary Listings of spreadsheet cells in which a keyword is used that can then be pasted or exported into Excel (tm) spreadsheets in which we collate results over several design sessions.
Sanderson, P.M., Iozzo, N., Buberel, J., & Au, I. (1995). The Advising Workbench: Participation-based development of a software environment to support student advising. To appear in Proceedings of the Human Factors and Ergonomics Society 1995 Annual Meeting, San Diego, CA, 9-13 October.
We use MacSHAPA to study motor skill acquisition in infants. To investigate infants' exploratory procedures and strategy choices, we test infants encountering practical but novel locomotor problems--going up and down slopes or stairs, and crawling and walking over slippery surfaces, gaps, bridges, etc. We then use MacSHAPA to code exploratory activity (looking, touching, testing positions) and locomotor behaviors (crawling, walking, sliding, etc.). For some experiments, we use relatively crude category codes and durations codes (latency, various locomotor positions, etc.). For other experiments, we micro-code movements (onset and offset of each limb movement during crawling, etc.). MacSHAPA is terrific for coding in multiple passes and for checking inter-rater reliability. In most studies, each researcher codes only one variable across all the tapes, and other researchers follow behind, building on the previous codes. We've found time-line reports and transitions reports especially useful. Because all of our studies require aggregation over many cases, MacSHAPA's analyses are not well-suited for our purposes. [MacSHAPA's analyses perform sequential analyses on one data set at a time--PS] However, we've mastered MacSHAPA's query language and have found many useful ways to dump data into Excel (tm) spreadsheets and from there into Systat (tm), where such analyses can be performed.
We have begun to use MacSHAPA to analyze video and keystroke protocol data collected from expert programmers while they are doing their own work. Our focus is on modeling the encoding and use of episodic memory during a programming session. Conventional spreadsheets and ASCII editors, and unintegrated video, allowed us to analyze a few minutes of protocol in the detail necessary for building a computational model. But these tools seemed prohibitively ineffective when we faced the task of characterizing encoding and use of episodic memory over several hours of protocol. MacSHAPA has provided much useful power. Its video-manipulation capabilities proved invaluable for identifying and reviewing a large number of episodes with various characteristics. Its integration in the Macintosh environment has allowed simple cut-and-paste importation of our existing transcriptions. We hope to make use of MacSHAPA's inherent importation routines to incorporate our time-stamped keystroke data. Finally, we plan to use MacSHAPA's report formats in presentations of our high-level study of episodic memory.
The Human Computer Interaction Institute (HCII) of the School of Computer Science at Carnegie Mellon University is engaged in a large scale study of the effects and usability of wearable computer systems in collaborative environments. We are using MacSHAPA to analyze video data from an initial experiment where we are studying communication and coordination of professional aircraft maintenance students using the wearable systems to aid in inspecting and repairing aircraft propeller components. For this study, a participant "worker" physically handled and worked with the aircraft parts and tools, while a remotely connected "helper" provided guidance to the "worker" about task completion. In addition to video data of participants, we are analyzing audio transcripts of the collaborative partners' discussion super-imposed over the video.
MacSHAPA appears particularly well suited for examining communication between the worker and helper using different levels of technology (paper documents with audio support, on-line shared hypertext documents with audio support, and shared hypertext documents with both audio and video support, where the remote helper sees the aircraft propeller and worker's hands via video.) For the communication studies, we are using MacSHAPA to review, annotate, and code a nominally defined communication variables from the videotapes. Using MacSHAPA in conjunction with QuickKeys (tm) we are able to code the communication between the helper and the worker participants in real-time providing frequency, patterns and duration of conversations in an easily accessible manner. When completed, the coded data will likely be formatted and initial analysis will be done. Then data will be exported to a statistical package (to be determined). MacSHAPA's flexibility in reformatting data and the reports generated by MacSHAPA provide a wealth of information for analysis.
I'm a post-doc at CMU doing work in HCI. I am using MacSHAPA to help in a project about how people use highly interactive computer systems. MacSHAPA was extremely useful in segmenting and characterizing the behaviors of people playing a videogame, which was a crucial part of the study. Without MacSHAPA, the work would have taken much longer. MacSHAPA allowed me to do an exploratory video analysis that would not have been feasible before. Preliminary results of the work appeared in the proceedings of CHI '95. I also must mention that if I had a program like MacSHAPA when I was working on my dissertation, I would have been done at least six months sooner!
Bauer, M.I., & John, B.E. (1995). Modeling time-constraining learning in a highly interactive task. Proceedings of the ACM Conference on Human Factors in Computing Systems (ACM SIGCHI '95), pp. 19-26.
Leon Segal. IDEO Product Development. (Work performed at NASA Ames Research Center) lsegal@ideo.com
I investigated the connection between aircraft cockpit design and crew coordination at NASA Ames Reserach Center. Central to the investigation was whether pilots monitor each other's activities as part of the on-going exchange of information within the cockpit. In order to test whether the action of one pilot's reaching for a control may predict a cross-cockpit look from the other pilot, MacSHAPA was used to perform a Lag Sequential Analysis on a data stream of cockpit activity. This analysis focused on whether a pilot tended to look at their crewmember's display immediately after that crewmember manipulated that display; thus, the analysis focused on events that follow each other immediately, without any intervening events (1st order transitions). The findings showed that, indeed, the interaction of one pilot with a control elicits a head-turn from the other pilot. It seems that beyond the use of verbal communication, pilots use activity information to maintain the flow of information in the cockpit.
Dal Vernon Reising and Penelope Sanderson. Department of Mechanical and Industrial Engineering, University of Illinois at Urbana-Champaign. dreising@psych.uiuc.edu
For his M.S.I.E. thesis at University of Illinois, Dal Vernon Reising completed a study on how people reason about multiple faults. The goal was to determine when, and why, multiple faults with interacting symptoms impose a greater cognitive load on subjects than other faults. Subjects were asked to diagnose different multiple faults in a simulated binary adder, talking out loud as much as they could, and the sessions were audiotaped or videotaped. The data logs from the experiments were imported into MacSHAPA, tests and diagnoses going into separate columns ("Test" and "Diagnosis"). Using MacSHAPA's query language, we added additional variables showing the binary adder output ("Actual Result") and the correct output if all had been normal ("Normal Result"). This provided the complete problem-solving context and made it easier to see what was happening--for instance, to see that subjects initially follow a stereotyped sequence for testing but that they spend most time in the later tests of each trial. Reising used the VCR control to watch the relevant parts of the videotapes and entered comments ("Notes") and even brief verbatim transcription ("Subjects' Comments") whenever the subject seemed overloaded or confused. Instances of overload or confusion were counted using keyword searches. These findings allowed Reising to draw conclusions about the effects of certain kinds of prior experience on troubleshooting performance.
Reising, D. C. (1993). Diagnosing multiple simultaneous faults. Proceedings of the 37th Annual Meeting of the Human Factors and Ergonomics Society, 524-528. Santa Monica, CA: Human Factors and Ergonomics Society.
Visual scanning is a critical component of flight skills. In spite of the fact that different scan patterns are typically taught in flight training programs, there is little evidence regarding how these patterns actually differ between skill levels, and how particular scan patterns may be symptomatic of difficulties in the acquisition of flight skills. Our current research examines the relationship between the visual scanning behavior of novice and expert pilots, their control actions and the state of flight variables in an IFR desktop flight simulator. By collecting these three classes of variables, we can begin to model the complex contingencies between them and study how these contingencies differ as a function of expertise. MacSHAPA provides a convenient and powerful set of tools to explore this kind of data set. Using the Query language, we have begun to categorize and "discretize" these data, revealing patterns which are currently being studied further. Content Reports give concrete numbers (fixation durations, total time for a specific control movement, etc.) to further aid in classification, while the Cycles and Transitions Reports support the generation of additional datasets to analyze. Results from this project will be available in the near future.
We have been using MacSHAPA to examine crew problem-solving techniques and information gathering in simulation scenarios. For each crew, we have created a file that contains data on flight parameters, such as heading, altitude, button presses, etc. (imported directly from the simulator data files), as well as transcripts of dialogue and codings of behaviors. In order to do comparisons among crews, we copied all of the coded variables into one variable, and have been able to compare information seeking and trouble shooting relevant to each sequential event in the scenario (e.g., What did the crews look at between the first cue and subsequent confirmation of the malfunction? From whom or what did they seek information?).
At AT&T Global Information Solutions, an initial use of Exploratory Sequential Data Analysis (ESDA) with MacSHAPA was an analysis of gate agent behavior at a major airline's hub, investigating the conflicts between customer service requirements and airline operational procedures. The analysis looked at observable events, such as customer-agent interaction, use of artifacts, human-computer interaction, time management, and required flight management procedures. MacSHAPA permitted us to look at changes in behavior patterns over time and showed how customer service and flight management tasks varied as time deadlines approached. A current project is centered in the fast food environment and is using ESDA techniques to study how technology is used with their just-in-time methodologies for food production. The analysis is focused on team situation analysis, co-operative work, technology usage, and customer-operator dialogues. Among other things, the results should show how information about store status is conveyed over time and how product buffers are depleted when the flow of information fails.
The Cognitive Models of Situation Assessment project investigates the situation assessment processes used by U. S. Navy submarine commanders in locating enemy submarines and merchant vessels. Ten expert submarine Commanding Officers (CO) were presented with various scenarios using a high-fidelity simulation of a submarine command center. All COs talked aloud while executing the scenario. Videotapes were collected and computer files saved. We have successfully imported pre-MacSHAPA transcriptions into MacSHAPA and added timestamps to match the MacSHAPA cells to parts of the videotape. Newer data have been transcribed directly into MacSHAPA with the MacSHAPA cells being timestamped as they are created.
Our analyses encode behavior as operators, goals, and schemas. In MacSHAPA, each of these is a separate variable. Both operators and goals are MacSHAPA predicates whereas schemas are MacSHAPA matrixes. MacSHAPA helped clarify agreement and disagreements via a modified Delphi method. We took three independent encodings (operators only) and copied them all into a single comparisons spreadsheet. The comparisons calculation report showed us which categories had substantial agreement and which had considerable disagreement. Also, having three encodings side by side helped us resolve differences in understanding and disagreements in vocabulary. We have made extensive use of the facilities for editing predicates and have used the query language for more extensive transformations. While we have just begun to use the MacSHAPA report facilities, already the cycle reports have changed the way we interpret our data and are being used to help guide the encoding of goals and subgoals. Likewise, we have begun using MacSHAPA built-in facilities for measuring interrater reliability.
This work has been jointly sponsored by the Office of Naval Research (ONR), grants#: N00014-95-1-0175 and N00014-91-1-1143 to Wayne D. Gray, Dr. Susan Chipman, Scientific Officer and by Naval Undersea Warfare Center's (NUWC) Independent Research (IR) Program, as Project A10328. The IR program is funded by ONR; the NUWC program manager has been Dr. Kenneth M. Lima (Code 102).
[Approved for public release: Distribution is unlimited]
The 1995 Concept of Operations Exercise (COOPEX'95) was an experiment investigating how the many systems and subsystems of the proposed New Attack Submarine (NSSN) and the human crew would work together. It was conducted in the Combat Systems Test Laboratory at the Naval Undersea Warfare Center Division, Newport, R.I. A testbed was created using current technology, commercial, off-the-shelf, hardware; and both commercial and advanced development modules of software. A special observation platform was constructed on three sides of the testbed to allow both real-time observation and videotaping of all scenarios. During the COOPEX, members of current submarine crews, retired submariners, and development engineers, acted as the NSSN crew, in roles within their areas of expertise (i.e., sonarmen used the sonar equipment, officers served in the officer roles, etc.). Several new pieces of equipment were of special interest, but the purpose of the exercise was to test the interactions among all players, human and machine during a number of realistic scenario situations that were selected to stress different capabilities.
Evaluation methods included concurrent observations by six trained observers using MacSHAPA on Macintosh Powerbooks (various models), video taping for retrospective viewing and analysis, and opportunities for the players to contribute their subjective judgments. Player judgments were collected both by responses to focused questionnaires at the end of each scenario and by discussions, comments, and participation in evaluation sessions. The six observers were system engineers and/or current or former submariners. Many had done observations in the context of training submarine crews. Each observer was assigned to a particular subsystem area of the testbed. Most of the observers were not experienced Macintosh users. They were trained in observation techniques and in how to use the Macintosh Powerbooks and MacSHAPA during the week prior to the start of COOPEX. They were provided with a template spreadsheet that included one predicate variable for structured observations and one text variable for free-form comments.
During the runs, each scenario event was cued so that all observations would be synchronized and could later be compared with the video tape. Observers made extensive use of the text formatted variables. As this made classification of the observations difficult, queries were used to search for key words such as equipment or activity names, and insert cells into a new predicate variable. For example, an entry such as "... comms (communications) at CWS (command workstation)..." would be categorized as both coms(...) and CWS(...), with the same time stamp. This has facilitated analysis, which is currently under way.
[Approved for public release: Distribution is unlimited]
We are using MacSHAPA to assist in the analysis of verbal protocol data (transcribed from audio tapes and imported into MacSHAPA) of students solving open-ended engineering design problems. In addition to audio tapes, we have video-taped some students solving problems and plan to use the video interface in the future. Currently we are segmenting the protocols into codable cells then coding the verbal data using four variables. The first gives an indication of the cognitive process of the subject (i.e. reading, calculating, making assumptions, etc.), the second gives and indication of the information being processed (specific to the nature of the problem), the third gives an indication of the item being designed (specific to the nature of the problem), and the last gives an indication of the step in the engineering design process. We hope to answer such questions as: do "good" and "poor" students differ in their approach to solving open-ended design problems? Do freshmen and seniors differ in their approach? Do student subjects differ from experts? Which approaches lead to better quality solutions (specifically following the design process steps versus taking opportunistic paths, Doing more analysis and evaluation of alternative solutions? Asking for more information as opposed to making assumptions? etc.) We expect the Timeline, Content, and Cycles reports in MacSHAPA to be most valuable in this analysis. This experiment is part of a larger effort aimed at studying how engineering students approach open-ended design problems in order to better understand how to teach this subject. The goal of this larger research effort is to determine appropriate educational interventions to improve students' ability to learn and apply engineering design concepts.
The Alexandria Digital Library is a software testbed under development to deliver comprehensive library services for browsing and retrieving maps, satellite imagery, and other georeferenced data that is distributed across local and wide area networks including the INTERNET. MacSHAPA is being used to evaluate the process by which users learn the Alexandria interface. The testbed software includes functions to generate a timestamped interactive log of what menu items, tools, and buttons they work with. These logs are directly imported into the MacSHAPA spreadsheet. This allows analysis of time lapses, which may indicate periods of inactivity, or confusion or frustration with the interface. These lapses are identified through MacSHAPA timeline reports or through the development of an appropriate query. By using a longitudinal subject testing design, timestamps may also indicate improved clarity of interface tools and changes in the learning curve as the interface is refined. Intercoder reliability functions in MacSHAPA provide comparisons between different types of users, different data archives, and different versions of the interface. MacSHAPA's video capability will be used once we start videotaping subjects as they work with the interface. We chose MacSHAPA in order to synchronize timestamps in the interactive logs with timestamps on the videotape.
MacSHAPA is being used within the Department of Psychology at University of Nottingham by research staff and students. In a recent project we used MacSHAPA in an investigation of air traffic controller decision making processes. Specifically, we wanted to examine how air traffic controllers responded to increased traffic demands.
Transcribed data was imported into the MacSHAPA spreadsheet. A coding scheme was developed iteratively, based upon a hierarchical task analysis of ATC. This preliminary coding scheme consisted of low level, observable actions, such as the air traffic controllers writing on a paper flight strip. The low level actions were grouped into functional tasks, such as accept aircraft into sector. Finally, trigger events were coded, such as an aircraft pilot contacting the controller. Hence, the MacSHAPA spreadsheet contained four columns of data: transcriptions, observed actions, functional tasks, and trigger events. MacSHAPA's Comparisons Report was used to show that another coder could use this scheme with 99% agreement.
A variety of statistics were carried out on the coded data. Some of these-- for example we used log linear analysis--required that data be ported into specialist statistical packages. We found the statistics within MacSHAPA to be useful, and on the whole easy to use. In particular we found that Fisher's Cycles gave us a powerful representation of common sequences of events within our data.
We found that as task demands increase, (a) controllers will postpone low priority actions, (b) between-controller communication has an increased importance in decision making, and (c) communication between controllers becomes more direct. MacSHAPA's flexible Import and Export functions allowed us to complement MacSHAPA with other specialist software. Powerful word processors (e.g., Emacs) were used to facilitate routine data manipulation. We do not have access to a VCR that can be directly controlled by MacSHAPA. However, we have found that CVideo (from Envisionology, Inc: +1 415 677-7956) provided a reliable alternative to operating a VCR through Abbate VTK. CVideo reduces the time required to transcribe and timestamp video tapes. The transcribed data can then be imported into the MacSHAPA spreadsheet.