Research Methods and Analysis …or “How to run a study” Gale M.

Research Methods and Analysis or “How to run a study” Gale M. Lucas, PhD

Overview of “how to run a study” Choose your variables, make them concrete – IV, DV Choose your method – Correlational or experimental Analyze your data

Choose your variables, make them concrete What do you want to study? Variables (or constructs) Operational definitions – A clear, measurable definition of a construct

Our example Belonging is a fundamental need, like hunger But people feel need to different degrees Construct: Need to belong (NTB)

Our example Operational definition: Need to Belong (NTB) Scale (Leary et al., 2001) 1 not at all characteristic of me 2 slightly characteristic of me 3 moderately characteristic of me 4 very characteristic of me 5 extremely characteristic of me I try hard not to do things that will make people avoid or reject me. I need to feel that there are other people I can turn to in times of need. I want other people to accept me. I do not like being alone. I have a strong "need to belong". It bothers me a great deal when I am not included in other people's plans. My feelings are easily hurt when I feel that others do not accept me.

Practice Come up with operational definitions for each of the following: – Trust – Aggression – Depression – Specify operational definition for IV and DV

IVs and DVs IV: Independent Variable (“manipulated”) – Variable that you think causes the effect on another variable – Variable that is manipulated in an experiment DV: Dependent Variable (“measured”) – The outcome variable (you think it has effect on) – Variable measured in an experiment Warning: some foreshadowing

Overview of “how to run a study” Choose your variables, make them concrete – IV, DV Choose your method – Correlational or experimental ? – Depends on your question

Our example What would that hunger make us do? – And socially sensitive/aware – Could ask two questions here

Our example “Is NTB associated with social sensitivity?” NTB Sensitivity Correlational “Does NTB cause heightened sensitivity?” NTB Sensitivity Experimental

Correlational vs experimental Recall, there was something different about IVs and DVs in experimental ? Correlational ? – Measure IV and measure DV – Our example? Experimental – Manipulate IV and measure DV – Our example?

Is NTB associated with sensitivity? What kind of research tests this question? Correlational research Picket, Gardner, and Knowles – NTB scale – DANVA: Number of facial expressions correctly identified r .38 What does that mean?

Correlation Coefficient -- r Indicates strength of relationship (0.00 to 1.00) Correlation coefficient rr 0.50 0.63 Indicates direction of relationship (positive or negative)

Problems with correlational research Only shows there’s an association We can’t say NTB causes social sensitivity, why not ? It could be some other variable causing both – Third variable problem Third Variable NTB Sensitivity Sensitivity could cause NTB – Directionality problem NTB Sensitivity

Problems with correlational research Third variable problem NTB Social sensitivity Motivation

Problems with correlational research Directionality problem NTB Social sensitivity

Problems with correlational research and there will always another third variable that you didn’t think of! So, conduct experimental research! – What makes an experiment? What do you have to do? – Manipulate IV: Randomly assign participants to conditions that only differ on IV Our example? Internal validity!

Experimental NTB research Picket, Gardner, and Knowles – What did they have to do here? – Randomly assign participants to one of two conditions that differ on IV: belonging Measure social sensitivity

Experimental NTB research 120 Social sensitivity 100 80 60 40 20 0 Control Rejection NTB causes an increase in social sensitivity!

Benefit of experimental research Experimental research wins! Right? Do we only care about outcomes these kinds of manipulations can effect? What important outcomes could NTB effect? Think big! How might hunger affect us?

Benefits of correlational research Age-adjusted mortality rate Real world outcomes that really matter External validity! 0.2 Males Females 0.15 0.1 0.05 0 Low Belonging High

Problems with experimental research Hard to make manipulations powerful enough to impact these kinds of outcomes Would you really want to? – Ethical operationalizations? – Ethically affect those outcomes? Even if didn’t have ethics . – Can’t practically manipulate some things

How do you ensure it is ethical? Institutional Review Board – Minimize harm, maximize benefit – Informed consent

Problems with experimental research Hard to create conditions that only differ on one variable – If conditions differ on more than one variable, those other differences are potential threats to internal validity what things differ between rejected & control conditions other than increased need to belong? – – – – – Pain Negativity Internal Mood validity! Anxiety These could be alternative explanations » If “rejected” were more socially sensitive, could be .

Recap Which should you choose when? Correlational research – Important, “hard to manipulate” real-life outcomes – Could not ethically or practically manipulate Experimental research – Make causal claims Variable Outcome – Can manipulate cleanly Internal validity!

Overview of “how to run a study” Choose your variables, make them concrete – IV, DV Choose your method – Correlational or experimental Analyze your data

Our example What did you experience when you completed the survey on qualtrics? What effect am I trying to testing for? What kind of study was this?

Data has been collected Population of students From the raw data it is impossible to tell if the groups differ! Sample Belonging Life alone 4 5 6 1 3 6 2 7 7 3 2 7 4 6 7 2 5 52 3 7 5 7 3 7 6 7 2 6 1 7 4 2 5 3 1 2 2 2 4 1 1 3 4 2 56 1 4 6 5 1 2 7 7 2

so run a statistical test Which one? It depends. . On whether you’ve run a correlational or experimental study, right? – Calculate a correlation coefficient, r, if we ran a correlational study, right? – WRONG! It depends on “type” of IV and DV

Overview of “how to run a study” Choose your variables, make them concrete – IV, DV Choose your method – Correlational or experimental Analyze your data – Depends on “type” of measurement

“Type” of IVs and DVs Both IV and DV can either be “categorical” or “continuous” – “groups” vs “how much?” Think continuous, think “continuum” 1 2 3 4 5 6 7 – Ask yourself: “can I put these in order?”

Continuous IV (e.g., how much) Continuous DV (e.g., how much) Categorical DV (e.g., yes/no) Categorical IV (e.g, exp vs control)

Continuous IV (e.g., how much) Continuous DV (e.g., how much) Categorical DV (e.g., yes/no) Categorical IV (e.g, exp vs control) t-test

Continuous IV (e.g., how much) Continuous DV (e.g., how much) Categorical DV (e.g., yes/no) correlation Categorical IV (e.g, exp vs control) t-test

Continuous IV (e.g., how much) Continuous DV (e.g., how much) Categorical DV (e.g., yes/no) correlation Categorical IV (e.g, exp vs control) t-test or ANOVA

Continuous DV (e.g., how much) Categorical DV (e.g., yes/no) Continuous IV (e.g., how much) Categorical IV (e.g, exp vs control) correlation or regression t-test or ANOVA

Continuous DV (e.g., how much) Categorical DV (e.g., yes/no) Continuous IV (e.g., how much) Categorical IV (e.g, exp vs control) correlation or regression t-test or ANOVA chi-square test

Continuous DV (e.g., how much) Categorical DV (e.g., yes/no) Continuous IV (e.g., how much) Categorical IV (e.g, exp vs control) correlation or regression t-test or ANOVA chi-square test or loglinear analysis

Continuous IV (e.g., how much) Categorical IV (e.g, exp vs control) Continuous DV (e.g., how much) correlation or regression t-test or ANOVA Categorical DV (e.g., yes/no) logistical regression chi-square test or loglinear analysis

What test would you use? Let’s go back: – Recall Picket, Gardner, and Knowles’ r .38 – Was that correct? Why? – What test here? 120 Social sensitivity 100 80 60 40 20 0 Control Rejection

What test would you use? to analyze the data from our study? Let’s run that test! what if I asked participants to provide NTB scores instead of assigning to conditions? – Which test? – What kind of study/conclusions?

Could we publish?

Should we publish? Why might we NOT want to publish our results? Did I get IRB approval? Concerns about our sample – Too small “N” (sample size) – Findings may not generalize to other population – Demand characteristics Validity of “mood” scale is questionable

Should we publish? Would we need to replicate first? – Replication crisis – However, many replication studies are underpowered and/or done back-to-back with other replications in long sessions

Summary of “how to experiment” Choose your variables, make them concrete – IV, DV Choose your method – Correlational or experimental Analyze your data – Depends on “type” of measurement

What test would you use? Practice question: – I ran a study to test whether people with high need to belong smile more – What do you need to know to figure out what test I should use? – IV: NTB scores – DV: count of smiles during 5 minute conversation – If you had just high vs low NTB? – If you had high, med, low NTB? – Are these experimental or correlational?

What test would you use? Practice question: – I ran a study to test whether people with high need to belong smile more – What do you need to know to figure out what test I should use? – IV: NTB scores – DV: smile or not in photo – If you had high vs low NTB?

More practice: Which test? 1a. Do people with higher need to belong friend more people on Facebook? 1b. Does the need to belong cause you to friend more people on Facebook? 2a. Is the type of car that you drive associated with hitting the cone (or not) on a driver’s course? 2b. Does the type of car that you drive cause to hit the cone (or not) on a driver’s course? 3a. Is working out associated with less anxiety? 3b. Does working out cause a reduction in anxiety? Is the answer to a and b the same or different for each question? If not, what is different between a and b?

Solutions in experimental research There are ways to help ensure internal validity 1. Control condition similar to experimental The conditions are very much alike, helps to control for some factors (currently alone vs with others, etc) – But still might differ on more than rejection: pain, negativity, mood, anxiety

Solutions in experimental research 2. Other additional conditions that manipulate these confounds – Pain – Negativity – If they don’t increase social sensitivity but rejection condition does – Mood? – Anxiety?

Solutions in experimental research 3. Measure alternative explanation – – – – Pain Negativity Mood Anxiety – If the rejection condition doesn’t increase these compared to control – Even if it does, can control for them statistically, and if still find a difference in social sensitivity

Solutions in experimental research 4. Measure variable intended to manipulate – If experimental condition is higher on that variable – Manipulation check – Mediation analysis 7 Belonging needs 6 5 Rejection 4 3 manipulation Belonging needs 2 1 0 Control Rejected Sensitivity

Solutions in experimental research 5. Different manipulations across studies – If always find difference – Multiple ways to increase NTB Reliving task Cyberball Living alone Chatroom More! Convergent validity!

Solutions in experimental research 6. Combine with other ways of asking the question Does NTB cause heighten social sensitivity? A) NTB scores associated with sensitivity B) Heighten need (starve) increases sensitivity C) Reduce need (feed) decreases sensitivity? And, if its like hunger, you could also ? Convergent validity!

Solutions in experimental research Lucas et al. Priming acceptance to “satiate” – Acceptance - e.g., together went us they there – Control - e.g., pretty was chair the skiing Social sensitivity 7 Social sensitivity 6 5 4 3 2 1 0 Acceptance Control

Questions? [email protected]