[Based on an anonymous form used in Psychology at the U. of Nottingham. Revised by Frank Ritter, Aug 2019]
There are some of the points you should consider when setting up a psychological or user study. The list is not exhaustive, and, as well, the points do not all apply in every case. Check that you understand each idea on the list well enough to use it properly in your study, or to know why you need not. Ask if you need more details.
* Reasons for doing it. Will it be (at least some of) the flowing: original, useful, interesting, elegant, unusual, publishable, theoretically important, good experience? Tells us something we didn't know before.
* Literature review. Be selective and critical: errors and omissions in the existing story are your point of departure.
*Written statement of your aims and success criteria.
* Adequate rational linking theory to experiment/analyses/etc.
* Suitable alternative theories or hypotheses for the experiment to select between.
* Satisfactory conclusion possible for each possible outcome.
* Suitable (statistical) analysis chosen and understood in detail in advance
* Elegance and simplicity of design.
* Written statement of hypotheses.
* Fallback plan. You might want to include some ancillary measures and hypotheses in case the main predictions fail.
* Backward check. Ask yourself whether what you want to know at the end will be decided best by the kinds of results you will get, whether those will be shown most clearly by the analysis you intend to do, and so on. Keep checking "backwards" through your plan, including measures, procedure and selection of objects, making sure that each stage is optimally served by the one before.
* Materials checked. All materials should be agreed with an instructor before they are used. It is all too easy to create something that seems to be what you need, but (a) turns out to be impossible to analyse properly, or (b) gives an impression of sloppy, careless work to the people it is sent to.
* No other know threats to validity that need to be allowed for.
* Systematic sampling of suitable population. Do your hypotheses apply to a particular group defined by age, gender, class, occupation, disability, handedness, or other specific features?
* Explicit selection and exclusion criteria for subjects. It may help to have a written list and include it in your write-up.
* Sufficient number of subjects. This will depend in part on the analysis you plan to do. There are guidelines for use with multivariate descriptive statistics like factor analysis. With analysis of variance, a 'power analysis' allows you to calculate the number of subjects you would need in order to get a particular level of significance from a predicated effect size.
* Response rate. With postal questionnaires, for example, your sampling may be invalidated if only a small proportion of people reply.
* Ethics. Discuss this with your teacher or supervisor if you have any doubts. Participants should only be used with their "full, free and informed consent". This is for your protection as well as theirs. They should not be deceived, or put at risk of any harm, or upset. Always remember, as psychologists we are here to help people, not to make them help us.
* Confidentiality. Some of the information people provide may be confidential. They should know and agree in advance to the use that will be made of it, and the forms in which other people may see it.
* The Data Protection Act or similar US legislation. You study must comply with this if any computerized records are to be made of data concerning individuals who could subsequently be identified.
* Details of participants recorded. You are likely to need a note of the age, gender and occupation of each subject at least in order to describe your sample adequately, unless it is an anonymous survey.
* Adequate briefing. Keep a record of what the subjects are told.
* Adequate de-briefing. What did the subjects notice during the experiment? What did they think it was about? Was the procedure confusing, or boring? Had anything worried them? Were there questions they wanted to ask?
* Pilot study. Try the procedure on a few people informally, to get the "bugs" out. Take special note of their comments and advice afterwards.
* Possible practice. Fatigue and order effects allowed for.
* All anomalies, unexpected occurrences and variations in procedure noted for further consideration.
* Appropriate independent and dependent variable.
* Levels of measurement. Nominal, ordinal, interval and ratio scales have different properties, and different statistical possibilities associated with them.
* Validity considerations and checks. There are various kinds including internal and external, construct, content, discriminant, predictive, and so on. Some concern the measures themselves, and others the logic and relevance of the study more generally.
* Reliability checks. There are various kinds including inter-observer, test-retest, split-half, and so on.
* Data systematically recorded. Data should be kept together with dates and other details of each rial in notebooks containing your name and address. Take all sensible precautions against later misunderstanding or loss of you data. You may want to make 'back-up' copies, especially with certain kinds of computer file.
* Appropriate statistical tests. These will depend on the level of measurement, the number of subjects, the kind of hypotheses you are testing, and the structure of the experiment. Make sure you know the difference between random and fixed effects, within-subject and between-subject comparisons, nested and crossed conditions, and so on, Do not do statistics just for the sake of it. Decide what you want to find out from your data, and pick the right tools for the job.
* Appropriate format and conventions. Most studies need 'introduction', 'method', 'results' and 'discussion' sections and a reference list. Aim for the format and style of a journal article. Cite reference using the proper conventions.
* Economy clarity and organizations. Avoid unnecessary waffle, especially in introduction and discussion sections: they are not essays.
* Adequate explanation. Your write-up should be enable someone else to repeat your procedure, and to understand why you did it the way you did.
* Modesty. Do no oversell yourself. Very few result are "astonishing" or "overwhelmingly significant". About all, never draw strong conclusions from weak evidence!
* Proofreading. Check your work through carefully, and remove errors of typing, spelling, grammar, punctuation, and sense before you hand it in.
* Acknowledgments. These may be appropriate for your supervisor, subjects, organizations who gave you access, technicians, computer stuff, staff, or anyone else who was a help to you.