What we've been up to in the Cognitive Modeling unit since 1993

http://www.psychology.nottingham.ac.uk/staff/ritter/cmu-projects.html

Frank E. Ritter

21 Oct 1998


Aims

Our aims are to understand more clearly how learning occurs. This is done by creating and testing cognitive models that learn, and, as necessary step, to make model creation, explanation, evaluation, and improvement more routine.

Better theories of learning will help design better computer interfaces and instructional materials, amoung a host of other applications.


Theoretical Background

Since January 1993, we have been busy working on creating cognitive models, and working on ways to create them more routinely. Progress has been made on several fronts, which we will note here. Interestingly, the web itself has played a substantial role.

The name Cognitive Modeling unit (or CMu) is used because we are "too small to be a group", yet we have had some cohesion. The unit has gotten larger, more a trio now, having had full-time RAs and usually one or more post-graduate students. We also have hosted a lot of visitors, over 15 so far, who stay from a day or a semester. Sam Marshall was the first RA. Gordon Baxter is the current and has been here longest. Gary Jones has recently graduated with a PhD (May, 1999). Two students have graduated with a MSc in Intelligent Knowledge-based Systems. The latest visitor is Moritz Baumann, from the Technical University of Berlin, which we have several informal ties.

We have started to work with Fernand Gobet, but our work comparing different models that learn has not reached a point where we have a report available.

The work can be broken into several components that are taken up in turn. They include:

These topics are taken up in turn below, orgainized into a single file for convenience of printing and reading. A list of informal notes too small or too numerous to get listed here is also available.



MODELS THAT LEARN


Aims

To create models that learn, in a variety of ways and in a variety of tasks.


Theoretical Background

We've created several models using these tools and processes. Several of these models can be seen directly as models of decision making, some them explicitly so. In most cases, we tend to test them using sequential data. We have used these models to understand the data and to provide theories of behavior that are not always visible without a a theory of how the task is preformed sufficient to perform the task.

A model of diagnostic, graphical reasoning

Pete Bibby and Sam Marshall created a model in Soar that diagnosed faults in an electrical circuit. Along with Shara Lochun, we tested its sequential predictions with protocol data because the model matches aggregate data fairly well. A paper is available showing that while the model now learns at about the same rate as subjects do, comparison with the protocol data shows that the rate of learning within a trial is not the same. This has allowed us to understand where and how learning occurs on a finer level.

Modeling How and When Learning Happens in a Diagrammatic Reasoning Task

A model that learns through interaction and reflection

With a variety of people we've been working on creating models that interact with their environment and learn from doing so. The best example of this is a model that learns by interacting with a very simple ATC simulation to land a plane. The model moves an eye (fovea) and hand (mouse) around a screen, and after the plan crashes, learns through reflection how to avoid this. This work has shown that interaction is a crucial aspect of problem solving, and that future work in this area should pay heed to reflection and interaction.

A summary of some of the lessons we have learned was presented at the ACT-R workshop.

Bass, E. J., Baxter, G. D., & Ritter, F. E. (1995). Using cognitive models to control simulations of complex systems. AISB Quarterly, 93, 18-25.
 
Ritter, F. E., Jones, G., Baxter, G. D., & Young, R. M. (1998). Lessons from using models of attention and interaction. In Proceedings of the 5th ACT-R Workshop Proceedings, 117-123. Psychology Department, Carnegie-Mellon University.

Miscellaneous models

We've worked on several miscellaneous models. These include a model that solves job shop scheduling tasks, a description of how and where emotions could be included in cognitive models, and a reimplementation and generalization of a model of physics problem solving. Theoretically, these models should provide a basis for creating further models. Practically, there remain problems in doing so.

Nerb, J., Krems, J., & Ritter, F. E. (expect 11/1998). Rule learning and the powerlaw: A Soar model and empirical results. For submission to Kognitionswissenschaft (Cognitive Science) special issue on cognitive models and cognitive architectures.

Nerb, J., Krems, J., & Ritter, F. E. (1993). Rule learning and the power law: A computational model and empirical results. Using a computer model to examine learning and the power law. In Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society ,765-770. Hillsdale, NJ: LEA.

Ritter, F. E. (1993). Three types of emotional effects that will occur in cognitive architectures. Workshop on architectures underlying motivation and emotion, The University of Birmingham 11-12 August. Also presented as a colloquium at the MRC-APU in Cambridge, October, 1993. and at EuroSoar7 and at North American Soar 14.

Ritter, F. E., & Baxter, G. D. (1996). Able, III: Learning in a more visibly principled way (Tech. Report No. 40. 8 pages.). ESRC CREDIT, Dept. of Psychology, U. of Nottingham. In the Proceedings of the 1st European Cognitive Modeling Workshop, November, 1996. Includes new Soar interface and two models.

A model of the Tower of Nottingham

We created a model that not only solves the Tower of Nottingham, but also learns and shows appropriate behavior at several levels of development. This work should help distangle whether development is best understood as changes to the cognitive architecture, or as the result of learning. This work was first started with Josef Nerb, but is now the PhD work of Gary Jones. At the start of this work, he was funded to visit major cognitive modeling sites in the US. A review of this trip has been reported in an online ONR newsletter. At this point in time, the model matches adult data fairly well, after being modified, it matches children data almost as well, and it shows that several theories of what develops (such as Piaget's) are either incomplete or wrong.

Jones, G., & Ritter, F. E. (1998). Simulating development by modifying architectures. In Proceedings of the Cognitive Science Society. 543-548. Madison, WI: Lawrence Earlbaum.

Jones, G., & Ritter, F. E. (1998). Initial explorations of modifying architectures to simulate cognitive and perceptual development. In Proceedings of the Second European Conference on Cognitive Modelling. 44-51. Nottingham: Nottingham University Press.

Ritter, F. E., Nerb, J., Kindsmüller, M. (1994). Steps towards a series of models for a developmental task. Overheads included in the Proceedings of the EuroSoar 8 Workshop. 95-99. Graduate School of Experimental Psychology, U. of Leiden.

 
Jones, G., & Ritter, F. E. (1997). Modelling transitions in children's development by starting with adults. In Proceedings of the European Conference on Cognitive Science, 62-67, Manchester, UK.

A model of two person games in Act and Soar

I have started to create a model of learning in two person games using Soar to compare with Dieter Wallach's ACT-R model. Currently, his model fits better. We've learned a bit about each other's architectures and about architectures in general by doing this.

Ritter, F. E., Wallach, D. P. (1998). Models of two-person games in Act-R and Soar. In Proceedings of the 2nd European Cognitive Modelling Conference. 202-203. Thrumpton (UK): Nottingham University Press.


TOOLS FOR CREATING MODELS


Aims

We have worked on several tools for creating and testing models. These can be organized under several subheadings: tools to tie models to simulated worlds, and tools for creating models.


Theoretical Background

Tying models to simulations

We have been working for several years on how to tie models, particularly Soar and Act-R type models to simulations. This is necessary in order to understand how perception influences behavior, and to provide a more complex set of stimuli to our models. Currently, we are having the eye and hand implemented in a commercially available graphical rapid prototyping language (SLGMS). In 1998 we expect to use it to allow a model to see and interact with an interface. We have learned that perception influences cognition, not only slowing it down in a general way, but also, for example, by restricting its behavior by precluding knowing everything at once.

Baxter, G. D., Ritter, F. E., Jones, G., & Young, R. M. (resubmitted Oct. 1998). Extending User Interface Management Systems to Support Cognitive Models as Users.

Ritter, F. E., & Major, N. P. (1995). Useful mechanisms for developing simulations for cognitive models. AISB Quarterly, 91(Spring), 7-18.

Ong, R., (1994). Mechanisms for routinely tying cognitive models to interactive simulations. MSc thesis.

Ong, R., & Ritter, F. E. (1995). Mechanisms for routinely tying cognitive models to interactive simulations. In HCI International '95. Osaka, Japan: July 1995.

Baxter, G. D., & Ritter, F. E. (1997). Model-computer interaction: Implementing the action-perception loop for cognitive models. In: The 1st International Conference on Engineering Psychology and Cognitive Ergonomics, October 1996, Stratford-upon-Avon. 215-222. Ashgate.

Rassouli, J. (1995). Steps towards a process model of mouse-based interaction. MSc. Thesis, U. of Nottingham.

Baxter, G. D., & Ritter, F. E. (1996). A simple set of generic perceptual and motor capabilities for use by cognitive models in SLGMS (Tech. Report No. 36). ESRC Credit, Psychology, U. of Nottingham.

Embedding models in tools

Perhaps the simplest way to create a model is to embed it in a tool. With Sarah Nichols, we have created an extension to an existing, free spreadsheet (called Dismal) for creating keystroke-level models. These can be useful models, but they are simple enough that 2nd year undergraduates can learn to apply them within a 5 week practical class. The spreadsheet, including the keystroke model extension are available on the web in my public FTP area. This work shows that simple enough models can be made available to help interface designers in their work. This tool has been used to create alias sets for two releases of a software package.

Nichols, S. & Ritter, F.E. (1995) A theoretically motivated tool for automatically generating command aliases. In Proceedings of CHI '95. 393-400.

Tools for directly creating cognitive models

A group of people, including Randy Jones (at the U. of Michigan), Tony Kalus (at the U. of Portsmouth/U. of Michigan), Gordon Baxter and I, along with input from people like Richard Young, have been working on an improved interface for Soar. Last year we used an early version in a cognitive modeling class at Nottingham. An preliminary version was presented at a workshop. When this is released, it will be called the Tcl/Tk Soar Interface. This work has shown that it is possible to make model creation, understanding, and debugging easier.

Ritter, F.E. and M. Bishop, (1997). The effect of abbreviation on time and errors: Even novices can profit from aliases. Unpublished mss.
 
Ritter, F. E., Jones, R. M., & Baxter, G. D. (1999). Reusable models and graphical interfaces: Realising the potential of a unified theory of cognition. In U. Schmid, J. Krems, & F. Wysotzki (Eds.), Mind modeling &emdash; A cognitive science approach to reasoning, learning and discovery. 83-109. Lengerich: Pabst Scientific Publishing.

Ritter, F. E., & Baxter, G. D. (1996). Able, III: Learning in a more visibly principled way (Tech. Report No. 40. 8 pages.). ESRC CREDIT, Dept. of Psychology, U. of Nottingham. In the 1st European Cognitive Modeling Workshop, November, 1996. Includes new Soar interface and two models.

A revised and extended version is available as Ritter, F. E., Jones, R. M., & Baxter, G. D. (1999). Reusable models and graphical interfaces: Realising the potential of a unified theory of cognition. In U. Schmid, J. Krems, & F. Wysotzki (Eds.), Mind modeling -- A cognitive science approach to reasoning, learning and discovery. 83-109. Lengerich: Pabst Scientific Publishing.

 
(FTP preliminary graphic displays (TSI) and a model of means-ends analysis that learns (AbleIII))


TOOLS FOR TESTING MODELS AND FOR ANALYZING DATA


Aims

After a model has been created, the next step is to test how well its predictions match human behavior. This is often difficult and time consuming, so automatic and semi-automatic tools are desirable.


Theoretical Background

As part of my PhD thesis, I've started to codify the steps in testing the sequential predictions of process models. This was written up as a more accessible journal article, and is a slightly longer tech report version is available for those looking for more details.

Ritter, F. E., & Larkin, J. H. (1994). Using process models to summarize sequences of human actions. Human Computer Interaction's special issue on Exploratory Sequential Data Analysis. 9 (3&4). 345-383.
The corresponding technical report abstract.

Once you have a cognitive model in hand, you may wish to test it. The first step in this process is often to gather protocol data.

Dismal was created as a spreadsheet for manipulating psychology data, including protocol data. Work on dismal has also generated some general utilities for GNU-Emacs. A more complete explanation is available as well http://www.psychology.nottingham.ac.uk/staff/ritter/papers/dismal/dismal.html.

 
Ritter, F. E., Fox, D., & (revised about every 6 months since 1991). Dismal, a spreadsheet for GNU Emacs. Older versions are available from the Elisp library at THE Ohio State University, and the most current version is available from my Psychology FTP site. It has been accepted to be part of the GNU Emacs distribution.

Ritter, F. E., Lochun, S., Bibby, P. A., & Marshall, S. (1994). Dismal: A free spreadsheet for sequential data analysis and HCI experimentation. In A. Trapp & N. Hammond (Eds.), Computers in Psychology '94, 62-63. York (UK): CTI Centre for Psychology, U. of York. Reprinted in Psychology Software News, Vol. 5 No. 2 (November 1994), Computers in Teaching Initiative Centre for Psychology, U. of York. pp. 57-58.

Ritter, F. E. (May, 1991). The simple-menu package. Available from The Ohio State University Elisp archives on archive.cis.ohio-state.edu as file pub/gnu/emacs/elisp-archive/interfaces/simple-menu.el.Z.

MacShapa is another program for editing protocol data. While it is not programmable like Dismal, it is a broader program, it can be hooked up to code video data more directly, and is is easier to use for most tasks. We recommend it in a review we did, and an initial analysis of some Air traffic control data was done using it.

CREDIT has set up a MacShapa lab for analyzing sequential data. Either for its own sake or as a precursor to building a process model. It includes the ability to transcribe directly from video tape to computer files, and enough memory to do this fairly easily. If you would like more details or to use it locally, please contact me.

 
Arnold, M., Ritter, F. E., & Kuk, G. (1995). MacSHAPA Review. 6(1). 18-20. CTI Psychology Software Review. Also referenced on the MacShapa example uses page and at the Psychology Computers in Teaching Initiative site.
 

Arnold, M., Ritter, F. E., Kuk, G. (1995). Prioritizing attention in air traffic control. Part of the MacShapa web pages.


A TUTORIAL FOR TEACHING MODELING


Aims

An essential aspect of cognitive modeling is understanding the modeling language.


Theoretical Background

To this end, with Richard Young we have created instructional materials called "The Psychological Soar Tutorial", or PST. This tutorial has been presented several times with Richard Young to external audiences: EuroSoar 7,8,9,10 workshops, 1993-1996 (UK, Wales, Netherlands, Germany); AISB Spring symposium series, 1994, 1996, 1997; HCI '94; HCI International 1995 (Japan). An extended version of it was presented at the Autumn School on Cognitive Science '97 run by the Graduiertenkolleg (graduate college) in Intelligence in Humans and Machines at the University of Freiburg (Germany). A review of it has been published in AI and Simulation of Behavior Quarterly, and this review has been updated once.

A short report on it was presented at the AISB'99 Workshop on teaching cognitive science to undergraduates, and a paper is in their proceedings.

A report on this is in preparation (a summary was published in CTI Psychology Software News, and is likely to be on their web server), and I can add you to the future distribution list if you wish. We will report that the web is not a panacea for developing course ware. The document we created I hoped would replace lectures and the lecturers. Except for remote students who don't have a choice, this was not the case. The web-based document did provide help with students' revision, and it made the material seem more real to them, such that they took it more seriously. This work showed that the next serious step for making modeling easier is an interface for programming and understanding models, and then higher level modeling tools such as a higher level language for creating models.

It is used as a case study in the U. of Notitnhgam's new lecterure's course.

Ritter, F. E., & Young, R. M. (1994). Practical introduction to the Soar cognitive architecture: Tutorial report. AISB Quarterly, 88, 62.
 
Ritter, F. E., Jones, G., & Young, R. M. (1996). Report on Tutorial 1: Introduction to the Soar cognitive architecture. AISB Quarterly, 95, 18.

Ritter, F. E. (1997). WWW presentation of overheads & exercises CTI Psychology Software News, 7(2), p 46.

The Psychological Soar Tutorial (v.9) for Soar 7.0.2 (with Richard Young and Gary Jones),
(FTP the whole Web tutorial code, Mac version), or (FTP the MacSoar to go with it), (FTP the whole PST9 Web tutorial code, Unix version).


MATHEMATICAL THEORIES AND ANALYSES OF LEARNING


Aims

In addition to creating full process models, it will often be useful (and more practical) to create more abstract models. These should be easier to use and apply than process models, but will often be based on models that do the task of interest.


Theoretical Background

With George Kuk, we have looked at how to create summaries of problem sub-task transitions in the area of air traffic control.

Kuk, G., Arnold, M., & Ritter, F. E. (in press). Using event history analysis to model the impact of workload on an air traffic tactical controller's operations. Ergonomics.

A preliminary version of the analysis is available as well, and is directly based on Martin Arnold's third year project based on data he collected at DRA Malvern.

Arnold, M., Ritter, F. E., Kuk, G. (1995). Prioritizing attention in air traffic control. Part of the MacShapa web pages.

The rate at which performance on task improves as the task is repeated is fairly similar across tasks. The improvement generally follows what is called a power-law. Based on an idea in my master's thesis, we looked at the power-law of learning in detail by separating the data into different strategies, where we found that each strategy improves with a power-law rate. This is a step forward in understanding how learning occurs.

Delaney, P. F., Reder, L. M., Staszewski, J. J., & Ritter, F. E. (1998). The strategy specific nature of improvement: The power law applies by strategy within task. Psychological Science. 9(1). 1-8
Abstract.


GENERAL SUPPORT FOR COGNITIVE MODELING


Aims

Cognitive modeling requires infrastructure. We've worked on setting up some infrastructure, locally and on a larger time and geographic scale.


Results

Ritter, F. E. (1995). Review of "Soar: An architecture in perspective". Philosophical Psychology, 8(3), 301-305.

Lehman, J., Newell, P., Altmann, E., Ritter, F., McGinnis, T. (1994). The Soar introduction video (11 min.). The Soar Group, School of Computer Science, Carnegie-Mellon University.

The Soar FAQ (with Gordon Baxter) It is updated quarterly and stored at the U. of Nottingham Soar site, and mirrored at the U. of Michigan.
 
Transfer times to the US can be slow at times. At an ad-hoc meeting of Soar users in the UK, we decided to set up a European mirror of the major Soar software and paper archives. These are updated nightly and stored at the University of Nottingham Soar Web site. The site includes information about how to set up similar sites.

Organizer and editor of proceedings, EuroSoar 7 Workshop, U. of Nottingham, November 1993.

Cognitive Modeling, The First European Workshop on, 14 - 16 of November, 1996; Berlin, Germany. It should be of interest for researchers in the fields of artificial intelligence, cognitive psychology and computer linguistics. The second one was held at Nottingham in 1998.


SUPPORT FOR THIS WORK

Support for this work has come from the DRA, the Enterprise in Higher Education, ONR/Europe, and from the UK Economic and Social Research Council through The Credit Centre.


Frank E. Ritter
School of Psychology .
University of Nottingham
Nottingham, NG7 2RD UK
+44 (115) 951-5292
+44 (115) 951-5324

Frank.Ritter@nottingham.ac.uk


The Credit Centre
Psychology Department Staff
Psychology Department Home Page
University of Nottingham Home Page


URL of this document: http://www.psychology.nottingham.ac.uk/staff/ritter/cmu-projects.html
Author: Frank E. Ritter