Syllabus for IST 402 Emerging Technologies:
The Development and Application of Models of Human Performance

Please note, this is a live document. Changes announced in class and on the list server will be incorporated from time to time. Announcements in class and their mirror here are the definitive version.

1. COURSE OVERVIEW

An emerging technology, oddly enough, is the humanistic technology of simulating human behaviour. This and related technologies have been called expert systems and artificial intellegence. They are currently being used as opponents in computer games and have been started to be used to test interfaces as surrogate users. They are an interesting intersection between technology and human psychology. The National Resarch Council (Pew & Mavor, 1998) and research councils in the US, UK, and Australia expect this area to become increasingly important. There are a wide range of possible applications including video games, military simulations, knowledge-based computer-based tutors, and assistive agents [Norling on agents and why they need to be more human: ABC Radio, 18 M, [mp4] [mp3]].

To explore this new technology and its science base, this class has three components. The first, short, component examines the application of models of human performance firstly as scientific theories, but also their use in training environments, in operations research, in design, and as opponents in computer games. This section is based on reading and discussion.

The second component examines how to create such models. Students will study how to create such models using an existing cognitive and AI architecture, Soar. Students will be introduced to the ideas of using published descriptions of human behavior (i.e., journal articles and handbooks), as well as how data is gathered when published materials are not available. Tools for analysing the data to help create models will also be covered. This component results in a model created by the student or a team within a somewhat constrained task, dTank. A prototypical final project is to build a model of your own play in this game.

The final component examines how such models in general and the student's model in particular can be tested, validated, and thus can be believed and improved. If the dTank environment is used, a competition between models will be staged, both as models of intelligence (do they win?), and as models of humans (do they act like humans?). Thus questions related to the philosophy of science are also discussed.

This course requires an Instructor technology room on a Monday afternoon and a student technology room on a Wed afternoon. The student room will have PCs. You should also be able to install the software on your home machine.

The official pre-requisites are IST 210 and IST 220 (or such). You should also have an interest in modeling human behaviour, and then secondarily, some knowledge of human behaviour or some knowledge of the technology to model it (or both). Knowledge of human behaviour can come from IST 331 or two psychology courses. Knowledge of the technology means Lisp or JAVA or expert systems or agent architectures. Students will be paired where possible to create pairs with complementary knowledge. If you have questions about prerequisites, please see me.

2. COURSE OBJECTIVES

This course should change the way you think about human behaviour, providing a theory of cognition that can be extended to help predict human behaviour for use in applications such as video games and interface testing.

This course provides a balance between theory and practice, which are tightly intertwined in this area. Basic and more advanced readings will introduce the student to current thinking about facts, theories, and ways to model data. A small group project, drawing on the different backgrounds students bring to the program, will support integrating these various pieces of knowledge and applying them. The course includes working in small groups, speaking, and other non-traditional processes and outputs, but processes and outputs that researchers use in this area.

At the conclusion of this course, students will be able to:

• Understand cognitive architectures and unified theories of cognition and their role with respect to agents, expert systems, human-computer interaction, and psychology, and their applications.
  
• Understand better in a quantitative and qualitative way some of the most relevant aspects of human behavior with respect to modeling their interaction with computer interfaces.
  
• Understand a particular cognitive architecture, Soar, well enough to create models in it.
  
• Understand and be able to compare the predictions of a simulation to appropriately chosen and gathered data. (This data may be gathered from the literature as well as from study participants).
  
• Be able to write about these topics.
  

3. COURSE ORGANIZATION

3.1 The IST 402 Web Site. This course has an active web page that contains the syllabus, assignments, links to useful sites, and other valuable material (such as how to correctly prepare assignments, citations templates, and other academic and recreational information). We will post late-breaking information and updates to the web page. This page can currently be found at uniform resource locator (URL) acs.ist.psu.edu/ist402, and later will be available through links from the IST home page via course listings.

3.2 The IST 402 Listserv. Each section has a mandatory listserv that we will use to post course and class information, conduct on-line discussions, and share information. You are encouraged to use your PSU account, and not a hotmail or yahoo account that cannot receive attachments.

If you are in Section 4 (Ritter) you need to subscribe to the class email list.

If you add late, or if you are not on it, you can join by sending mail to ist402-4-F04-subscribe-request@lists.psu.edu .

Once you have subscribed, you can then send mail to the class at ist402-4-F04@lists.psu.edu

If you send mail to me about this course, please include "ist402" in the subject, as this will help a filter bring it to my attention more quickly.

3.3 Required Texts / Materials

Stuff on Soar (read, in order, until you know Soar)

The Psychological Soar Tutorial, available online

Ritter, F. E. (2003). Soar. In L. Nadel (Ed.), Encyclopedia of cognitive science. vol. 4, 60-65. London: Nature Publishing Group. [A006.pdf]

Laird's online tutorial notes, chapter 1

Lehman, J. F., Laird, J. E., & Rosenbloom, P. S. (1996). A gentle introduction to Soar, an architecture for human cognition. In S. Sternberg & D. Scarborough (Eds.), Invitation to cognitive science, vol. 4 Cambridge, MA: MIT Press.

Soar manual

Newell, A. (1992). Précis of "Unified theories of cognition". Behavioral and Brain Sciences, 15, 425-492. Responses Introduction to Newell

Stuff on the software (you will need all of these, and all are free)

Herbal [a Description of Ontologies, to help with understanding]

The Soar 8 distribution.

Those who like SoarceForge, can also find Soar there

Fix to the TSI to support Herbal

You may find VisualSoar helpful, as well as lots of things in the Soar FAQ

dTank-Soar game, available locally (source code is in the jars!)

Stuff on psychology and writing [one copy per group]

(ABCS) The ABCS of HCI. Ritter, F. E, Gilmore, D., & Churchill, E. 2004. Available from Kinko's (on the corner-ish of Atherton and College, ph. 238-2679) at cost, for about $20.00. approximately 210 pages.

Publication Manual of the APA (available at the PSU Bookstore) as a guide to referencing, citing, and the formatting of papers and manuscripts in general. Each pair should have access to a copy of this, or to Strunk and White's "The elements of style."

Gopen's paper on the science of writing (password protected). Also see APA guide to online references, available online.

3.4 Required readings

3.4.1 The importance and applications of models (read two)

Anderson, J. R., Conrad, F. G., & Corbett, A. T. (1989). Skill acquisition and the LISP tutor. Cognitive Science, 13(4), 467-505.

Laird, J., & van Lent, M. (2001). Interactive Computer Games: Human-level AI's Killer Application, ai.eecs.umich.edu/people/laird/papers/AI-games.pdf.

Hudlicka, E., & McNeese, M. D. (2002). User affective and belief states: Assessment and user interface adaptation. Journal of User Modeling and User Adapted Interaction, 12, 1-47.

3.4.2 Testing and validating models (read all)

Ritter, F. E., & Larkin, J. H. (1994). Using process models to summarize sequences of human actions. Human-Computer Interaction, 9(3), 345-383.

Ritter, F. E. (2004). Choosing and getting started with a cognitive architecture to test and use human-machine interfaces. MMI-Interaktiv-Journal's special issue on Modeling and Simulation in Human-Machine Systems. 7. 17-37. [in English, abstract in German]

3.4.3 Creating and building models (read all)

Ritter, F. E., Lehtinen, E., & Nerb, J. (accepted for publication). Putting things in order: Collecting and analysing data on learning. In F. E. Ritter, J. Nerb, T. O'Shea, & E. Lehtinen (Eds.), In order to Learn: How ordering effects in machine learning illuminates human learning and vice versa

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press. Their appendix on collecting protocols.        Their 1980 paper as Precis of book [Read appendix, rest is optional]

Ritter, F. E., Langley, P., & Nerb, J. (accepted for publication). Rules of order: Process models of human learning. In F. E. Ritter, J. Nerb, T. O'Shea, & E. Lehtinen (Eds.), In order to learn: How ordering effects in machine learning illuminates human learning and vice versa Kidlington, UK: Pergamon.

Nuxoll, A., & Laird, J. (2003). Soar Dogma. (how to write better, safer, faster Soar code.)

The Soar Frequently Asked Questions list (acs.ist.psu.edu/soar-faq) and the ACT-R FAQ (acs.ist.psu.edu/actr-faq).

3.5 Optional Texts and Interesting Resources on Writing

Agre on writing

Related Douglass class

Comments on writing

How to write an abstract by Mary-Claire Van Leunen

Grant, D. A. (1962). Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 69(1), 54-61.

Paschler

Pew, R. W., & Mavor, A. S. (Eds.). (1998). Modeling human and organizational behavior: Application to military simulations. Washington, DC: National Academy Press. nap.edu/catalog/6173.html (click on the cover of the book on the left to read it online.)

Ritter, F. E., Shadbolt, N. R., Elliman, D., Young, R., Gobet, F., & Baxter, G. D. (2003). Techniques for modeling human performance in synthetic environments: A supplementary review. Wright-Patterson Air Force Base, OH: Human Systems Information Analysis Center (HSIAC), formerly known as the Crew System Ergonomics Information Analysis Center (CSERIAC). (click on the cover PDF file to read it online, or on Table of contents to get individual chapters.)

Sanderson, P. M., McNeese, M. D., & Zaff, B. S. (1994). Handling complex real-world data with two cognitive engineering tools: COGENT and MacSHAPA. Behavior Research Methods, Instruments, & Computers, 26(2), 117-124. [email me for macshapa software for Mac]

4. EVALUATION

You earn your grade but it will be assigned by me. The criteria for each assignment will be discussed in detail, as will the grading scheme. Each written assignment will be evaluated on how well it addresses the questions posed, the clarity of thinking, the organization and presentation of the material, the quality of writing, and its timeliness. 

Your grade will be based on 100 possible points. You earn points with each assignment (see below). As a maximum scale (i.e., cutoffs may be lowered): A: 100-74, A-: 73-70, B+ 69- 67, B: 66- 64, B-: 63- 60, C+: 59- 57, C: 56- 50, D: 49- 40, F: 39- 0.  (The cutoffs for each grade is the lower number, without rounding.)

Your learning will be assessed in several ways. Please consult the schedule to see when papers/ assignments are due and exams scheduled. You will receive more written instructions for each assignment well in advance of the due date. Here is a brief summary of each:
   

Assignment	Weight		Due Date
Labs Tips on doing them Notes on writing and doing labs Marking scheme	30%	You will do a variety of labs/homeworks. Each lab writeup is nominally 5-10 points, 66 points total including an extra 5 extra credit points and the initial 1 point project writeup. 60 points will be taken to be the maximum lab grade (i.e., you can miss 6 points and get a perfect score). This score may be modified/moderated/adjusted by self and team evaluations.	Typically Wednesdays, as below
Current Affairs and Additional Readings Assignment	(5 points added to lab grade)	Once during the semester your group may find an additional resource that addresses or relies upon topics covered in class. In one page or less, you will comment on how that resource relates to the class. You will share both the article and your comments with the class. You must arrange this before the end of October.	Once, varies by group
Mid-Term Exam	35%	In class, 70 points below.	November 2004
Project Example templates: RTF word5	35%	72 total points make this up, which you can keep the 2 as a bonus. Examples from 1999, from a smaller version of this course.	December 2004
Total	100%

5. COURSE CONDUCT

• Classes will start on time and end as scheduled. Please take your seat prior to the start of class.
• You should attend each class and actively participate in the discussions during class. University policy on class attendance is applied.
• If you are uncomfortable with public speaking, or if English is not your native language, we must meet in the first two weeks of school to establish ways to make you more comfortable in speaking and interacting with your peers. I am happy to do this; I have been there myself.
• For every hour of lecture, I anticipate that you will need to budget about 3 hours of out-of-class time. This implies that you need to budget about 140 hours of out-of-class time over the course of the semester. This time estimate is a guide and you may need to budget more or less. For example, if the material is new to you or difficult to comprehend, it will require more of your time. 
• You are responsible for all the readings, even if the material is not explicitly covered in class. You should read the class materials prior to class and be prepared to discuss and ask questions about the readings and assignments. You should also re-read the material after class as not every topic will be covered during class time. Many passages in the text may need to be read several times to gain clarity. Also, taking notes on the material you are reading and reflecting on the reading and these notes will help you better understand the issues, concepts and techniques that are being presented.
• All work must be completed and turned in at the start of class on the assigned date. No late work will be accepted. Late means after the class has begun. Note that a computer's failure is not an excuse (it represents poor planning on your part). If you miss a deadline, a written explanation of a university recognized excuse and written documentation (e.g., a doctor's note) must be handed to me at the end of a lecture.
  Assignments that are simply late can be turned in for feedback but 0 marks.
• All assignment should be double-spaced (or 1.5 spaced where appropriate), on 8.5"x 11" or A4 paper. All pages should have 1" margins. Papers should be stapled and collated. Please do not use report covers; they will not be returned. Your group number and names should be on the cover, as well as an abstract (where appropriate).
• Carefully proofread your work. Mistakes include spelling, grammatical errors, and other typos. You should assume that your reader is about as smart as you, not smarter. You must also show your work, even if you just note 'by inspection'. The marker will want to know that you know how to get the answer.
• I expect individual work should be just that -- it should be done by you, alone.
• I expect group work should be just that -- from all of the group. If I become aware that you are not contributing to your group equally, I will intervene.
• Students who participate in University-sanctioned events (such as athletics) must make prior arrangements and give ample notice.
• The official language of this course is English.
• Requests for regrading must be turned in with this form.

6. IST 402 CLASS SCHEDULE (subject to revision)

	Date	In Class	Read/Prepare	Due	points
1	1-Sep-04 w	Intro - Course, emerging technologies, new uses for old important technologies, UTCs, Cognitive psych review		Handin-OSs, courses
2	8-Sep-04 w	Applications	two readings from 3.4.1	1 para on each reading	1
3	13-Sep-04	PST1	Preface/ABCS1 ht.s8 Further reading on Soar
4	15-Sep-04	Soar / Stevenson / Protege	Ritter '03, soar video ABCS3 Protege	Ontology of project/agent knowledge	1
5	20-Sep-04	PST2 pst exercises	ht-imp.s8 ABCS4 Laird1
6	22-Sep-04	Soar / Herbal Viewer Herbal Language (Cohen)	ABCS5	PST1-hwk 1 para on project	10 1
7	27-Sep-04	PST3	ABCS6
8	29-Sep-04	Model development	Ritter & Larkin 94 ABCS-task anal.	PST2-hwk	10
9	4-Oct-04	Eaters1 / dTank (Schenck+)
10	6-Oct-04	Testing models/why	More from 3.4.2	PST3-hwk	10
11	11-Oct-04	dTank
12	13-Oct-04	Data for developing	Ericsson & Simon ABCS	1 page on project	1
13	18-Oct-04	dTank
14	20-Oct-04	Creating models	readings from 3.4.3
16	25-Oct-04	HERBL2	ABCS9
17	27-Oct-04	Example model	(project design)
18	1-Nov-04	Creating models	two readings from 3.4.3 and ABCS11	dTank Hwk1 (comes by email)	10
19	3-Nov-04	Review	Example midterm Another example midterm
20	8-Nov-04	dTank2	ABCS10	dTank2 HWK (do another dTank task)	10
21	10-Nov-04	midterm in class		Group feedback	70
22	15-Nov-04	Video games, hedonics, software engineering		Outline of project report	1
23	17-Nov-04	Slack		dTank HWK3	10
24	22-Nov-04	Project - writing techniques (Buchanon?)	Agre/APA Manual/Strunk/advanced Word
	26-Nov-04	*Thanksgiving*
25	29-Nov-04	Future directions	SOAR report
26	1-Dec-04	Project work
27	6-Dec-04	Reports discussion	ABCS-A4	talk in class	5
28	8-Dec-04	Hand in		Project	65
					209 pts, keep 200

Model the effect of your learning, starting with a naive model, and then a less naive model.

Model how you learn, starting with a naive model, and then with the learning component turned on, show that the model learns like you do (considerably harder).

Extend the dTank interface in some interesting way, and show how it is easier for you and for the model.

Set up two opponents for people to train against. Show how one is a better sparing partner, either for some of your friends, or for your model to learn from.

Extend SegMan to see what users see in the 3d display.

All projects above, if extended, could be turned into an honors thesis.

Graduate students taking this course for independent study credit (or if approved as a 597 course) will need to modify both the task (multiple interfaces or tasks) and the model (multiple strategies or individual differences).

7. Labs for IST 402

The laboratory portion of IST 402 provides students with the chance to become familiar with using the concepts, software, and data about modeling people. It is absolutely essential for understanding the material and will be useful for passing the exam and doing the labs.

You have been put into small groups to do your labs because we believe this generally leads to better learning. That means that you must turn in one lab report per group, that conferring within your group is not a violation of academic policy or of ethics on the lab section of this course, and that conferring with other groups *is* a violation of academic policy and ethics if it results in reports that are noticeably similar without citation.

The best way is to work on the lab and then meet to discuss and proofread the report. The worst way is to have each member of the group do (and thus learn) one of the sections. This will result in a noticeably inferior product.

As we explore these topics, we will also practice skills in working together, analytical skills, and information problem-solving approaches. 

We have arranged time for group meetings that may need a lab.

8. Relevant University Policies

PSU STATEMENT ON ACADEMIC INTEGRITY

Academic Integrity: According to the Penn State Principles and University Code of Conduct:

Academic integrity is a basic guiding principle for all academic activity at Penn State University, allowing the pursuit of scholarly activity in an open, honest, and responsible manner. In according with the University's Code of Conduct, you must not engage in or tolerate academic dishonesty. This includes, but is not limited to cheating, plagiarism, fabrication of information or citations, facilitating acts of academic dishonesty by others, unauthorized possession of examinations, submitting work of another person, or work previously used without informing the instructor, or tampering with the academic work of other students.

Any violation of academic integrity will be investigated, and where warranted, punitive action will be taken. For every incident when a penalty of any kind is assessed, a report must be filed This form is used for both undergraduate and graduate courses. This report must be signed by both the instructor and the student, and then submitted to the Senior Associate Dean.

In this course, academic integrity needs two explanations. The first is that you are expected to contribute to your group. This means making time in your schedule for the group's meetings, being in touch via email, and preparing for those meetings. The second explanation is that your group's writeups are to be done only by your group. This means that while you can discuss the problems with others, and we agree that this is a good thing, the writing up needs to be solely by your group. If you have a question as to whether or not two writeups are too similar, then use the following standard:

If a pair of writeups are sufficiently similar that you can tell by reading them that the two groups worked together, then they are too similar.

By all means, talk to your colleagues, get help if necessary, but prove to us that, in the end, you understand what you are doing, and you can and must express it in your own words. If you and your group don't understand the material well enough to write it up on your own, and you need to copy, then four things are lost: Your integrity, useful feedback to us on how you are doing, your ability to perform well on the exam, and ultimately, your knowledge.

Note to students with disabilities

It is Penn State's policy to not discriminate against qualified students with documented disabilities in its educational programs. If you have a disability-related need for modifications in your testing situation, your instructor should be notified during the first week of classes so that your needs can be accommodated. You will be asked to present documentation from the Office of Disability Services (located in 105 Boucke Building) that describes the nature of your disability and the recommended remedy. You may refer to the Nondiscrimination Policy in the Student Guide to University Policies and Rules 1999.

As each student is an individual with specific needs, academic accommodations are provided on an individual basis based on the student's documentation. A reasonable accommodation is a modification or adjustment to a course, program, service, job, activity, or facility that provides the qualified individual with a disability to have an equal opportunity. An equal opportunity provides the means to attain the same level of performance or to enjoy benefits that are available to students without disabilities. For more information about services for individuals with learning disabilities, please contact the Office for Disability Services at (814) 863-1807.

Americans with Disabilities Act: IST welcomes persons with disabilities to all of its classes, programs, and events. If you need accommodations, or have questions about access to buildings where ISTactivities are held, please contact us in advance of your participation or visit. If you need assistance during a class, program, or event, please contact the member of our staff or faculty in charge.

An Invitation to Students with Learning Disabilities: It is Penn State's policy to not discriminate against qualified students with documented disabilities in its educational programs. If you have a disability-related need for modifications in your testing or learning situation, your instructor should be notified during the first week of classes so that your needs can be accommodated. You will be asked to present documentation from the Office of Disability Services (located in 116 Boucke Building, 863-1807) that describes the nature of your disability and the recommended remedy. You may refer to the Nondiscrimination Policy in the Student Guide to University Policies and Rules.

Affirmative Action & Sexual Harassment: The Pennsylvania State University is committed to a policy that all persons shall have equal access to programs, facilities, admission, and employment without regard to personal characteristics not related to ability, performance, or qualifications as determined by University policy or by Commonwealth or Federal authorities. Penn State does not discriminate against any person because of age, ancestry, color, disability or handicap, national origin, race, religious creed, gender, sexual orientation, or veteran status. Direct all inquiries to the Affirmative Action Office, 211 Willard Building.