Experimental Methodology
- Theories
- Empirical Thinking/The Role of Experiments
- Critical Thinking/Evaluating Experiments
- Statistical Tests
- Should you read Stanovich?
- Experimental Reports
Theories
- The Role of Theories
- General Criteria for Evaluating Theories
The Role of Theories
- "A theory is a set of principles that can be used to account for,
predict and even control certain phenomena."
- Most theories make causal claims.
- Science can be viewed as a continuous process of formulating, evaluating,
reformulating, and abandoning theories.
- All theories are wrong!
- Theories can lead us astray, so it is important to be critical of
theories.
- Gravity
- Disease
General Criteria for Evaluating Theories
- Is it consistent with observable facts?
- Sufficiency (Not Uniqueness)
- Does it explain everything that needs explaining?
- The fact that one theory is sufficient doesn’t mean that other
theories aren’t sufficient as well.
- Parsimony
- The fewer assumptions a theory makes, the better!
- Generality and Completeness
- Does it explain one domain or many?
- Does it explain all of the domain, or just some of it?
- Power/Falsifiability
- Could it explain anything, including the impossible?
- Can it be disproven?
Empirical Thinking/The Role of Experiments
- Let’s Make a Deal!
- What should you do?
- Ask Marilyn
Let’s Make a Deal!
- Montie Hall gives you a choice of 3 doors.
- Behind one is a car, behind the other 2 are goats.
- You choose door A.
- Montie shows you that there is a goat behind door B, and offers to let
you switch to door C.
What should you do?
- No Time to Prepare
- Keep door A?
- Switch to door C?
- Time to Prepare
- Reason it out?
- Ask an expert?
- Something else?
Ask Marilyn
- Who is Marilyn Vos Savant?
- The "Three Doors" Column in PARADE Magazine
- The Experts’ Response
- Who’s right?
Always Switch
Critical Thinking/Evaluating Experiments
- Five Questions
- Psychokinesis (PK) Example
- Does this constitute evidence for PK?
- Some Things that Go Wrong with Experiments
Five Questions
- What exactly is the claim am I being asked to believe?
- What evidence supports the claim?
- Are there any alternative interpretations of the evidence?
- What additional evidence do I need in order to evaluate the alternatives?
- What is the most reasonable conclusion?
Psychokinesis (PK) Example
- "PK is the ability of some people to bypass their motor mechanism
and, by volition, reach directly into space and time outside themselves to exert a
force."
- Evidence for PK?
- A California televangelist gave a sermon in which he advised viewers on
methods for identifying Satanists. One was to watch for a hand signal they use to identify
one another.
- During a subsequent telecast, an "experimenter" and 4
"trained witches" tried to use PK to force the televangelist to make the sign
himself.
- During the 30 minute live broadcast, he made the sign 23 times!
Does this constitute evidence for PK?
- What exactly is the claim am I being asked to believe?
- What evidence supports the claim?
- Are there any alternative interpretations of the evidence?
- What additional evidence do I need in order to evaluate the alternatives?
- What is the most reasonable conclusion?
Some Things that Go Wrong with Experiments
- Operational Definitions of Cause (IV) and Effect (DV)
- Reliability
- Validity
- Baseline Measures/Control Groups
- Correlation versus Causality
- The Directionality Problem
- The Third Variable Problem (Confound)
- Randomization and Matching
- Procedural Artifacts
- Replication and Converging Operations
- Generalizability
Statistical Tests
- You don’t need to know statistics to make it through this course.
- It is helpful to understand the following at a conceptual level:
- Correlation Coefficient (r)
- t-test (t)
- ANOVA (F)
Should you read Stanovich?
- Yes, if you don’t feel comfortable with the following concepts:
- Independent Variable (IV)
- Dependent Variable (DV)
- Operational Definition
- Reliability and Validity
- The Directionality Problem
- The Third Variable Problem (Confound)
- Randomization and Matching
- Replication and Converging Operations
- Generalizability
- Otherwise, you needn’t bother!
Experimental Reports
- Abstract
- Introduction
- Method
- Participants
- Materials
- (Design and) Procedure
- Results
- Discussion
- References
The End!
|