Syllabus for IST 402 Emerging Technologies:
The Development and Application of Models of Human Performance

Please note, this is a live document. Changes announced in class and on the list server will be incorporated from time to time. Announcements in class and their mirror here are the definitive version.

1. COURSE OVERVIEW

An emerging technology, oddly enough, is the technology of simulating human behaviour. This and related technologies have been called expert systems and artificial intellegence. They are currently being used as opponents in computer games and have been started to be used to test interfaces as surrogate users. They are an interesting intersection between technology and human psychology. The National Resarch Council (Pew & Mavor, 1998, 2007) and research councils in the US, UK, and Australia expect this area to become increasingly important and they invest in it. There are a wide range of possible applications including video games, military simulations, knowledge-based computer-based tutors, and assistive agents [Norling on agents and why they need to be more human: ABC Radio [mp4] [mp3]].

To explore this new technology and its science base, this class has three components. The first, short, component examines the application of models of human performance firstly as scientific theories, but also their use in training environments, in operations research, in design, and as opponents in computer games. This section is based on reading and discussion.

The second component examines how to create such models. Students will study how to create such models using an existing cognitive and AI architecture, Soar. Students will be introduced to the ideas of using published descriptions of human behavior (i.e., journal articles and handbooks), as well as how data is gathered when published materials are not available. Tools for analysing the data to help create models will also be covered. This component results in a model created by the student or a team within a somewhat constrained task, dTank. A prototypical final project is to build a model of your own play in this game, but this project can be student-interest driven.

The final component examines how such models in general and the student's model in particular can be tested, validated, and thus can be believed and improved. If the dTank environment is used, a competition between models will be staged, both as models of intelligence (do they win?), and as models of humans (do they act like humans?). Thus questions related to the philosophy of science are also discussed.

This course requires an Instructor technology room on a Monday afternoon and a student technology room on a Wed afternoon. The student room will have Mac/PCs. You should also be able to install the software on your home machine, and many students will choose to do this.

The official pre-requisites are IST 210 and IST 220 (or such). You should also have an interest in modeling human behaviour, and then secondarily, some knowledge of human behaviour or some knowledge of the technology to model it (or both). Knowledge of human behaviour can come from IST 331 or a psychology course. Knowledge of the technology means production systems, Lisp, or JAVA, or expert systems or agent architectures. Students will be paired where possible to create pairs with complementary knowledge. If you have questions about prerequisites, please see me.

2. COURSE OBJECTIVES

This course should change the way you think about human behaviour, providing a theory of cognition that can be extended to help predict human behaviour for use in applications such as video games and interface testing.

This course provides a balance between theory and practice, which are tightly intertwined in this area. Basic and more advanced readings will introduce the student to current thinking about facts, theories, and ways to model data. A small group project, drawing on the different backgrounds students bring to the program, will support integrating these various pieces of knowledge and applying them. The course includes working in small groups, speaking, and other non-traditional processes and outputs, but the processes and outputs used are those that researchers and developers use in this area.

At the conclusion of this course, students will be able to:

• Start to be a more self-aware learner, and be aware of different types of publications.
• Understand cognitive architectures and unified theories of cognition and their role with respect to modeling human behavior, agents, expert systems, human-computer interaction, and psychology, and their applications.
  
• Understand better in a quantitative and qualitative way some of the most relevant aspects of human behavior with respect to modeling their interaction with computer interfaces.
  
• Understand a particular cognitive architecture, Soar, well enough to create models in it.
  
• Understand and be able to compare the predictions of a simulation to appropriately chosen and gathered data. (This data may be gathered from the literature as well as from study participants).
• Be able to use the technologies: Soar, Eclipse, Herbal, and Word
  
• Be able to write about these topics.
  

3. COURSE ORGANIZATION

3.1 The IST 402 Web Site. This course has an active web page that contains the syllabus, assignments, links to useful sites, and other valuable material (such as how to correctly prepare assignments, citation templates, and other academic and recreational information). We will post late-breaking information and updates to the web page. This page can currently be found at uniform resource locator (URL) acs.ist.psu.edu/ist402, and later will be available through links from the IST home page via course listings. You should bookmark it.

3.2 The IST 402 Listserv. This course has a mandatory listserv that we will use to post course and class information, conduct on-line discussions, and share information. You are encouraged to use your PSU account, and not a hotmail or yahoo account that cannot receive attachments.

If you are in Section 2 (Ritter) your PSU account should be automatically subscribed. If you want to use a different account, you will need to subscribe to the class email list by sending email to Changkun.

Once you have subscribed, you can then send mail to the class at L-ist402-FA11-S1@lists.psu.edu

If you send mail to me about this course, please include "IST402" in the subject, as this will help a filter bring it to my attention more quickly.

3.3 Required Texts / Materials

Stuff on Soar (read, in order, until you know Soar) [passwords available from TA and Ritter]

The Psychological Soar Tutorial, available online

Ritter, F. E. (2003). Soar. In L. Nadel (Ed.), Encyclopedia of cognitive science. vol. 4, 60-65. London: Nature Publishing Group. [A006.pdf]

Laird's online tutorial notes, chapter 1 [from the Soar installation]

Lehman, J. F., Laird, J. E., & Rosenbloom, P. S. (1996). A gentle introduction to Soar, an architecture for human cognition. In S. Sternberg & D. Scarborough (Eds.), Invitation to cognitive science, vol. 4 Cambridge, MA: MIT Press.

Soar manual

Newell, A. (1992). Précis of "Unified theories of cognition". Behavioral and Brain Sciences, 15, 425-492. Responses Introduction to Newell

Stuff on the software (you will need all of these, and all are free)

Summary installation notes

Herbal [a Description of Ontologies, to help with understanding] Herbal tutorial

The Soar 9 distribution.

Those who like SoarceForge, can also find Soar there

You may find the Soar FAQ helpful.

dTank-Soar game, available locally

Stuff on psychology and writing [one copy of each per group]

(ABCS) The ABCS of HCI. Ritter, F. E, Baxter, G. D., & Churchill, E. 2011. Available from LuLu.com at cost, for about $11.00 plus shipping. approximately 400 pages.

Publication Manual of the APA (available at the PSU Bookstore) as a guide to referencing, citing, and the formatting of papers and manuscripts in general. Each pair should have access to a copy of this, or to Strunk and White's "The elements of style." Abebooks.com is another place to get this book.

3.4 Required readings

3.4.1 The importance and applications of models (read two)

Anderson, J. R., Conrad, F. G., & Corbett, A. T. (1989). Skill acquisition and the LISP tutor. Cognitive Science, 13(4), 467-505.

Laird, J., & van Lent, M. (2001). Interactive Computer Games: Human-level AI's Killer Application, ai.eecs.umich.edu/people/laird/papers/AI-games.pdf.

Hudlicka, E., & McNeese, M. D. (2002). User affective and belief states: Assessment and user interface adaptation. Journal of User Modeling and User Adapted Interaction, 12, 1-47.

Pew, R. W., & Mavor, A. S. (Eds.). (2007). Human-system integration in the system development process: A new look. Washington, DC: National Academy Press. http://books.nap.edu/catalog.php?record_id=11893.

Pew, R. W., & Mavor, A. S. (Eds.). (1998). Modeling human and organizational behavior: Application to military simulations. Washington, DC: National Academy Press. nap.edu/catalog/6173.html (click on the cover of the book on the left to read it online.)

Ritter, F. E., Shadbolt, N. R., Elliman, D., Young, R., Gobet, F., & Baxter, G. D. (2003). Techniques for modeling human performance in synthetic environments: A supplementary review. Wright-Patterson Air Force Base, OH: Human Systems Information Analysis Center (HSIAC), formerly known as the Crew System Ergonomics Information Analysis Center (CSERIAC). [local cleaner copy]

** Or, paper of your choice, approx. 20 pages, or a whole article, whichever is shorter. **

3.4.2 Testing and validating models (read all)

Ritter, F. E., & Larkin, J. H. (1994). Using process models to summarize sequences of human actions. Human-Computer Interaction, 9(3), 345-383.

Ritter, F. E. (2004). Choosing and getting started with a cognitive architecture to test and use human-machine interfaces. MMI-Interaktiv-Journal's special issue on Modeling and Simulation in Human-Machine Systems. 7. 17-37. [in English, abstract in German] [local copy]

Grant, D. A. (1962). Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 69(1), 54-61.

3.4.3 Creating and building models (read all)

Ritter, F. E., Lehtinen, E., & Nerb, J. (2007). Putting things in order: Collecting and analysing data on learning. In F. E. Ritter, J. Nerb, T. O'Shea, & E. Lehtinen (Eds.), In order to Learn: How ordering effects in machine learning illuminates human learning and vice versa

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press. Their appendix on collecting protocols.        Their 1980 paper as Precis of book [Read appendix, rest is optional]

Nerb, J., Ritter, F. E., & Langley, P. (2007). Rules of order: Process models of human learning. In F. E. Ritter, J. Nerb, T. O'Shea, & E. Lehtinen (Eds.), In order to learn: How ordering effects in machine learning illuminates human learning and vice versa Kidlington, UK: Pergamon.

Nuxoll, A., & Laird, J. (2003). Soar Dogma. (how to write better, safer, faster Soar code.)

The Soar Frequently Asked Questions list (acs.ist.psu.edu/soar-faq) and the ACT-R FAQ (acs.ist.psu.edu/actr-faq).

3.5 Optional Texts on modeling and Interesting Resources on Writing

Agre on writing

Related Douglass class

Comments on writing

How to write an abstract by Mary-Claire Van Leunen

Paschler on testing (simple) models

Sanderson, P. M., McNeese, M. D., & Zaff, B. S. (1994). Handling complex real-world data with two cognitive engineering tools: COGENT and MacSHAPA. Behavior Research Methods, Instruments, & Computers, 26(2), 117-124. [email me for macshapa software for Mac]

4. EVALUATION

You earn your grade but it will be assigned by me. The criteria for each assignment will be discussed in detail, as will the grading scheme. Each written assignment will be evaluated on how well it addresses the questions posed, the clarity of thinking, the organization and presentation of the material, the quality of writing, and its timeliness. 

Your grade will be based on 100 possible points. You earn points with each assignment (see below). As a maximum scale (i.e., cutoffs may be lowered): A: 100-74, A-: 73-70, B+ 69- 67, B: 66- 64, B-: 63- 60, C+: 59- 57, C: 56- 50, D: 49- 40, F: 39- 0.  (The cutoffs for each grade is the lower number, without rounding.)

Your learning will be assessed in several ways. Please consult the schedule to see when papers/ assignments are due and exams scheduled. You will receive more written instructions for each assignment well in advance of the due date. Here is a brief summary of each:
   

Assignment	Weight		Due Date
Labs Tips on doing them Marking scheme	30%	You will do a variety of labs/homeworks. Each lab writeup is nominally 5-10 points, 66 points total including an extra 5 extra credit points and the initial 1 point project writeup. 60 points will be taken to be the maximum lab grade (i.e., you can miss 6 points and get a perfect score). This score may be modified/moderated/adjusted by self and team evaluations.	Typically Wednesdays, as below
Additional Readings and Resources	(5 points added to lab grade)	Once during the semester your group may find an additional resource that addresses or relies upon topics covered in class. In one page or less, you will comment on how that resource relates to the class. You will share both the article and your comments with the class. You must arrange this before the end of October. Paragraphs on the readings, up to 20 of them at 1 pt/reading, can also be used this way. There may also be an opportunity to participate in a study for extra credit.	Once, varies by group
Mid-Term Exam	30%	In class, 70 points below.	November 2011
Attendance	5%	In class, score pro-rated with 3 allowed unexcused absences
Project Example templates: RTF word5	35%	71 total points make this up, which you can keep the 1 as a bonus.	December 2011
Total	100%

5. COURSE CONDUCT

1. Classes will start on time and end as scheduled. Please take your seat prior to the start of class.
2. You should attend each class and actively participate in the discussions during class. University policy on class attendance is applied.
3. If you are uncomfortable with public speaking, or if English is not your native language, we must meet in the first two weeks of school to establish ways to make you more comfortable in speaking and interacting with your peers. I am happy to do this; I have been there myself.
4. For every hour of lecture, I anticipate that you will need to budget about 3 hours of out-of-class time. This implies that you need to budget about 140 hours of out-of-class time over the course of the semester. This time estimate is a guide and you may need to budget more or less. For example, if the material is new to you or difficult to comprehend, it will require more of your time. 
5. You are responsible for all the readings, even if the material is not explicitly covered in class. You should read the class materials prior to class and be prepared to discuss and ask questions about the readings and assignments. You should also re-read the material after class as not every topic will be covered during class time. Many passages in the text may need to be read several times to gain clarity. Also, taking notes on the material you are reading and reflecting on the reading and these notes will help you better understand the issues, concepts and techniques that are being presented.
6. All work must be completed and turned in at the start of class on the assigned date. No late work will be accepted. Late means after the class has begun. Note that a computer's failure is not an excuse (it represents poor planning on your part, we recommend a USB drive or tools such as dropbox). If you miss a deadline, a written explanation of a university recognized excuse must be handed to me at the end of a lecture.
   Assignments that are simply late can be turned in for feedback but 0 marks.
7. All assignment should be double-spaced (or 1.5 spaced where appropriate), on 8.5"x 11" or A4 paper. All pages should have 1" margins. Papers should be stapled and collated. Please do not use report covers; they will not be returned. Your group number and names should be on the cover, as well as an abstract (where appropriate).
8. Carefully proofread your work. Mistakes include spelling, grammatical errors, and other typos. You should assume that your reader is about as smart as you, not smarter.
9. I expect individual work should be just that -- it should be done by you, alone.
10. I expect group work should be just that -- from all of the group. If I become aware that you are not contributing to your group equally, I will intervene.
11. Students who participate in University-sanctioned events (such as athletics) must make prior arrangements and give ample notice but will be excused.
12. The official language of this course is English.
13. Requests for regrading must be turned in with this form.

6. IST 402 CLASS SCHEDULE (subject to revision)

	Date	In Class	Read/Prepare	Due	points
1	22aug	Intro - Course, emerging technologies, new uses for old important technologies, UTCs, Cognitive psych review		Hand in: OSs, courses
2	24aug	End of intro	Install/test software Summary installation notes	Software test (in class)	1
3	29aug	Applications of models	two readings from 3.4.1	1 para on each reading	2
4	31aug	Soar/ PST1	Further reading on Soar Preface/ABCS1 hts9.soar	1 para on each reading	1-2
5	7sep	Soar	Ritter '03, soar video ABCS3		1-2
6	12sep	PST2 pst exercises	ht-imps9.soar ABCS4 Laird1	PST1-hwk readings	10 1 1-2
7	14sep	Soar / Herbal	ABCS5 [from now, all ABCS up to now due] Eclipse	Exercise on Eclipse (in class)	1
8	19sep	PST3	ABCS6	1 para on project PST2-hwk	10 1
9	21sep	Model development	Ritter & Larkin 94 ABCS12(TA)		1-2
10	26sep	Herbal /dTank		PST3-hwk	10
11	28sep	Testing models/why	More from 3.4.2		1
12	3oct	Herbal/dTank			10
13	5oct	Data for developing	Ericsson & Simon 93A Ericsson & Simon 80 (optional) ABCS4-11	1 page on project	1pp
14	10oct	Herbal/dTank		Using Basic Herbal homework vacuum cleaner jar
15	12oct	Creating models	readings above in 3.4.3		1
16	17oct	Herbal/dTank	ABCS12		1
17	19oct	Example model	(project design)	dTank Hwk1 (email it in)	10
18	24oct	Herbal/dTank
19	26oct	Creating models	two readings from 3.4.3 and ABCS11		10 1
20	31oct	Review	Example midterm Another example midterm ABCS17.4	dTank2 HWK	1
21	2nov	midterm in class +Group feedback			60
22	7nov	Discussion of projects		Outline of project report	1
23	9nov	Video games, hedonics, software engineering
24	14nov	Project - writing techniques	Agre/APA Manual/Strunk/advanced Word	dTank HWK3 (do another dTank task)	10
25	16nov	Future directions	SOAR report and other futures reports		1
	21-23nov	*Thanksgiving*
26	28nov	Project work
27	30nov	Reports discussion	ABCS-A4	presentations in class	5
28	5dec	Reports discussion		presentations in class
29	7dec	Hand in		Project	65
					>209 pts, keep 200

Model the effect of your learning, starting with a naive model, and then a less naive model.

Model how you learn, starting with a naive model, and then with the learning component turned on, show that the model learns like you do (considerably harder).

Extend the dTank interface in some interesting way, and show how it is easier for you and for the model.

Set up two opponents for people to train against. Show how one is a better sparing partner, either for some of your friends, or for your model to learn from. 

All projects above, if extended, could be turned into an honors thesis.

Graduate students taking this course for independent study credit (or if approved as a 597 course) will need to modify both the task (multiple interfaces or tasks) and the model (multiple strategies or individual differences).

7. Labs for IST 402

The laboratory portion of IST 402 provides students with the chance to become familiar with using the concepts, software, and data about modeling people. It is absolutely essential for understanding the material and will be useful for passing the exam and doing the labs.

You have been put into small groups to do your labs because we believe this generally leads to better learning. That means that you must turn in one lab report per group, that conferring within your group is not a violation of academic policy or of ethics on the lab section of this course, and that conferring with other groups *is* a violation of academic policy and ethics if it results in reports that are noticeably similar without citation.

The best way is to work on the lab and then meet to discuss and proofread the report. The worst way is to have each member of the group do (and thus learn) one of the sections. This will result in a noticeably inferior product.

As we explore these topics, we will also practice skills in working together, analytical skills, and information problem-solving approaches. 

We have arranged time for group meetings that may need a lab.

8. Relevant University Policies

PSU STATEMENT ON ACADEMIC INTEGRITY

Academic Integrity: According to the Penn State Principles and University Code of Conduct:

Academic integrity is a basic guiding principle for all academic activity at Penn State University, allowing the pursuit of scholarly activity in an open, honest, and responsible manner. In according with the University's Code of Conduct, you must not engage in or tolerate academic dishonesty. This includes, but is not limited to cheating, plagiarism, fabrication of information or citations, facilitating acts of academic dishonesty by others, unauthorized possession of examinations, submitting work of another person, or work previously used without informing the instructor, or tampering with the academic work of other students.

Any violation of academic integrity will be investigated, and where warranted, punitive action will be taken. For every incident when a penalty of any kind is assessed, a report must be filed This form is used for both undergraduate and graduate courses. This report must be signed by both the instructor and the student, and then submitted to the Senior Associate Dean.

In this course, academic integrity needs two explanations. The first is that you are expected to contribute to your group. This means making time in your schedule for the group's meetings, being in touch via email, and preparing for those meetings. The second explanation is that your group's writeups are to be done only by your group. This means that while you can discuss the problems with others, and we agree that this is a good thing, the writing up needs to be solely by your group. If you have a question as to whether or not two writeups are too similar, then use the following standard:

If a pair of writeups are sufficiently similar that you can tell by reading them that the two groups worked together, then they are too similar.

By all means, talk to your colleagues, get help if necessary, but prove to us that, in the end, you understand what you are doing, and you can and must express it in your own words. If you and your group don't understand the material well enough to write it up on your own, and you need to copy, then four things are lost: Your integrity, useful feedback to us on how you are doing, your ability to perform well on the exam, and ultimately, your knowledge.

Note to students with disabilities

It is Penn State's policy to not discriminate against qualified students with documented disabilities in its educational programs. If you have a disability-related need for modifications in your testing situation, your instructor should be notified during the first week of classes so that your needs can be accommodated. You will be asked to present documentation from the Office of Disability Services (located in 105 Boucke Building) that describes the nature of your disability and the recommended remedy. You may refer to the Nondiscrimination Policy in the Student Guide to University Policies and Rules 1999.

As each student is an individual with specific needs, academic accommodations are provided on an individual basis based on the student's documentation. A reasonable accommodation is a modification or adjustment to a course, program, service, job, activity, or facility that provides the qualified individual with a disability to have an equal opportunity. An equal opportunity provides the means to attain the same level of performance or to enjoy benefits that are available to students without disabilities. For more information about services for individuals with learning disabilities, please contact the Office for Disability Services at (814) 863-1807.

Americans with Disabilities Act: IST welcomes persons with disabilities to all of its classes, programs, and events. If you need accommodations, or have questions about access to buildings where ISTactivities are held, please contact us in advance of your participation or visit. If you need assistance during a class, program, or event, please contact the member of our staff or faculty in charge.

An Invitation to Students with Learning Disabilities: It is Penn State's policy to not discriminate against qualified students with documented disabilities in its educational programs. If you have a disability-related need for modifications in your testing or learning situation, your instructor should be notified during the first week of classes so that your needs can be accommodated. You will be asked to present documentation from the Office of Disability Services (located in 116 Boucke Building, 863-1807) that describes the nature of your disability and the recommended remedy. You may refer to the Nondiscrimination Policy in the Student Guide to University Policies and Rules.

Affirmative Action & Sexual Harassment: The Pennsylvania State University is committed to a policy that all persons shall have equal access to programs, facilities, admission, and employment without regard to personal characteristics not related to ability, performance, or qualifications as determined by University policy or by Commonwealth or Federal authorities. Penn State does not discriminate against any person because of age, ancestry, color, disability or handicap, national origin, race, religious creed, gender, sexual orientation, or veteran status. Direct all inquiries to the Affirmative Action Office, 211 Willard Building.