Methodology Matters: Doing Research in the Social and Behavioral

Methodology Matters: Doing Research in the Social and Behavioral Sciences Joseph E. McGrath Gary Suh Vesna Memisevic 1 Fall 2007 Informatics 231 Human-Computer Interaction

Outline 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 2 Introduction Some Basic Features of the Research Process Substantive Domain Conceptual Domain Methodological Domain Research Methods as Opportunities and Limitations Research Strategies: Choosing a Setting For a Study Quadrant I: The Field Strategies Quadrant II: The Experimental Strategies Quadrant III: The Respondent Strategies Quadrant IV: The Theoretical Strategies Some Strategic Issues Fall 2007 Informatics 231 Human-Computer Interaction

Outline 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 3 25. Study design Comparison techniques Baserates The Correlation Question The Difference Question Randomization and “True experiments” Sampling, allocation and statistical inference Validity of findings Potential classes of measures in social psychology Strengths and Weaknesses of Types of Measures Techniques for Manipulating Variables Selection, direct intervention, induction Conclusion Fall 2007 Informatics 231 Human-Computer Interaction

Introduction 4 Doing research – the systematic use of some set of theoretical and empirical tools to try to increase our understanding of some set of phenomena or events Some of the tools with which researchers in the social and behavioral sciences go about doing research (strategy, tactics, and operations) Fall 2007 Informatics 231 Human-Computer Interaction

Some Basic Features of the Research Process Always involves bringing together three sets of things: (a) some content that is of interest, (b) some ideas that give meaning to that content, and (c) some techniques or procedures by means of which those ideas and content can be studied. 5 Fall 2007 Informatics 231 Human-Computer Interaction

Some Basic Features of the Research Process These three sets of things more formally, as three distinct, though interrelated domains: (a) The Substantive domain, from which we draw contents that seem worthy of our study and attention; (b) The Conceptual domain, from which we draw ideas that seem likely to give meaning to our results; and (c) The Methodological domain, from which we draw techniques that seem useful in conducting that research. 6 Fall 2007 Informatics 231 Human-Computer Interaction

Some Basic Features of the Research Process 7 Fall 2007 Informatics 231 Human-Computer Interaction

Substantive Domain Phenomena and Patterns of phenomena are the object of our study The phenomena of interest involve the states and actions of some human systems and the conditions and processes that give rise to and follow from those states and actions. 8 Fall 2007 Informatics 231 Human-Computer Interaction

Substantive Domain 9 Example – an individual casting a vote in a county election Fall 2007 Informatics 231 Human-Computer Interaction

Conceptual Domain Properties of the states and actions of those human systems that are the focus of study Relations refer to any of a variety of possible ways in which two or more elements can be connected Examples – causal connections, logical relations, chronological relations 10 Fall 2007 Informatics 231 Human-Computer Interaction

Conceptual Domain 11 Examples – attitude, cohesiveness, power, social pressure, status Fall 2007 Informatics 231 Human-Computer Interaction

Methodological Domain Basic sets of elements or “tools” by which social and behavioral scientists systematically gather empirical information: Measuring Manipulating Controlling Distributing Impact 12 Fall 2007 Informatics 231 Human-Computer Interaction

Methodological Domain Measuring – – 13 For assessing the state or magnitude of some property of some actors-behaving-in-context so that the researcher can determine what value or level that feature has for each “case” to be studied Examples – questionnaire, rating scale, personality test, instruments for observing and recording communications, techniques for assessing the quality of some products resulting from individual or group task performance Fall 2007 Informatics 231 Human-Computer Interaction

Methodological Domain Manipulating – – 14 Techniques for manipulating some property of an actor-behavior-context Experimental manipulation – making a feature have one particular predetermined value or level for certain “cases” to be studied and another specific preordained value or level for certain other “cases,” so that the effect of differences in that property can be assessed by comparing those two sets of “cases” Fall 2007 Informatics 231 Human-Computer Interaction

Methodological Domain Manipulating – – 15 Social psychologists have tried to manipulate features of the systems they study by a number of techniques, such as: (a) giving instruction to participants (b) imposing constraints on features of the environment (c) selecting materials for use (d) giving feedback about prior performances (e) using experimental confederates (More is said about techniques for manipulating variables near the end of this chapter) Fall 2007 Informatics 231 Human-Computer Interaction

Methodological Domain Controlling – – A set of techniques for controlling the impact of features that are important but that you are not going to measure or manipulate in a particular study These include: Techniques for experimental control – you make certain features take the same predetermined value for all cases in the study Techniques for statistical control – you try to nullify the effects of variations in a given property within a study by “removing” those variations by statistical means Techniques for distributing the impact – so that such impact can be taken into account in interpretation of results 16 Fall 2007 Informatics 231 Human-Computer Interaction

Methodological Domain Distributing – – 17 Impact Techniques for distributing the impact of a number of features of the system and its contextwithout directly manipulating or controlling any one of them-so that such impact can be taken into account in interpretation of results Randomization – the most prominent means; refers to procedures for the allocation of “cases” among various conditions within the study Fall 2007 Informatics 231 Human-Computer Interaction

Research Methods as Opportunities and Limitations (a) (b) (c) (d) 18 Methods enable but also limit evidence. All methods are valuable, but all have weaknesses or limitations. You can offset the different weaknesses of various methods by using multiple methods. You can choose such multiple methods so that they have patterned diversity; that is so that strengths of some methods offset weaknesses of others. Fall 2007 Informatics 231 Human-Computer Interaction

Research Methods as Opportunities and Limitations The fundamental principle, in behavioral and social science is that credible empirical knowledge requires consistency or convergence of evidence across studies based on different methods. 19 Fall 2007 Informatics 231 Human-Computer Interaction

Research Strategies: Choosing a Setting For a Study 20 When you gather a batch of research evidence, you are always trying to maximize three desireable features or criteria: A. Generalizability of the evidence over the populations of Actors. B. Precision of the measurement of the behaviors that are being studied (and precision of control over extraneous factors that are not being studied). C. Realism of the situation or Context within which the evidence is gathered, in relation to the contexts to which you want your evidence to apply. Fall 2007 Informatics 231 Human-Computer Interaction

Research Strategies: Choosing a Setting For a Study Although you always want to maximize all three of these criteria, A, B and C simultaneously, you cannot do so. This is the fundamental dilemma of the research process. 21 Fall 2007 Informatics 231 Human-Computer Interaction

Research Strategies: Choosing a Setting For a Study 22 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant I: The Field Strategies 23 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant I: The Field Strategies 24 The two research strategies in quadrant I are the Field Study and the Field Experiment. Field study – the researcher sets out to make direct observations of “natural”, ongoing systems, while intruding on and disturbing those systems as little as possible. Field experiment – also works within an ongoing natural system as unobtrusively as possible, except for intruding on that system by manipulating one major feature of that system. Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant I: The Field Strategies The essence of both of the strategies in quadrant I, the field study and the field experiment, is that the behavior system under study is “natural”, in the sense that it would occur whether or not the researcher were there and whether or not it were being observed as part of a study. 25 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant II: The Experimental Strategies 26 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant II: The Experimental Strategies 27 Laboratory experiment – the investigator deliberately concocts a situation or behavior setting or context, defines the rules for its operation, and then induces some individuals or groups to enter the concocted system and engage in the behaviors called for by its rules and circumstances. Experimental simulation – the researcher attempts to achieve much of the precision and control of the laboratory experiment but to gain some of the realism (or apparent realism) of field studies. Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant II: The Experimental Strategies The two strategies in Quadrant II, in contrast to those of Quadrant I, involve concocted rather than natural settings. The laboratory experiment and the experimental simulation are strategies that involve “actor-behavior-context” systems that would not exist at all were it not for the researcher’s interest in doing the study. 28 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant III: The Respondent Strategies 29 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant III: The Respondent Strategies Sample survey – the investigator tries to obtain evidence that will permit him or her to estimate the distribution of some variables, and/or some relationships among them, within a specified population Examples – public opinion surveys on voting intentions, political preferences, buying intentions 30 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant III: The Respondent Strategies 31 The strategies of Quadrant III concentrate on the systematic gathering of responses of the participants to questions or stimuli formulated by the experimenter, in contrast to the observation of behaviors of the participants within an ongoing behavior system Studies are usually done under “neutral” conditions of room temperature, lighting, chair comfort to nullify any effects of the behavior setting or context on the judgments that are the topic of study. Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant IV: The Theoretical Strategies 32 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant IV: The Theoretical Strategies Formal theory – the researcher focuses on formulating general relations among a number of variables of interest Computer simulation – a complete and closed system that models the operation of the concrete system without any behavior by any system participants 33 Fall 2007 Informatics 231 Human-Computer Interaction

Quadrant IV: The Theoretical Strategies The inclusion of these two strategies reminds us of the importance of the theoretical side of the research process. One of the more powerful general strategies for research is the simultaneous use of one of the theoretical strategies (say, the formulation of a general theory) and one of the empirical strategies (for example, a laboratory experiment). 34 Fall 2007 Informatics 231 Human-Computer Interaction

Some Strategic Issues 1. 2. 35 Does the material, as presented, properly reckon with the strengths and weaknesses of the research strategies it encompasses? To what extent is the research evidence on each problem or topic based on use of only a single research strategy, and therefore limited by the weaknesses of that strategy; and to what extent is that body of evidence based on use of multiple, complementary strategies, with agreement or convergence among the findings attained via the different strategies? Fall 2007 Informatics 231 Human-Computer Interaction

Study design 1. 2. 3. 36 Gathering observations Aggregation and partitioning Comparison on the data set Fall 2007 Informatics 231 Human-Computer Interaction

Study design Comparison depends on: What’s included in the study (what phenomena, what properties, what model of treatment for variables) What system works (which substantive system we study, paradigms, strategies) What conceptual relations have been posted (which properties are linked) What 37 comparison techniques are available Fall 2007 Informatics 231 Human-Computer Interaction

Comparison Techniques Three base forms: Baserates (how often?) Correlations (are properties related; do they occur together?) Differences (comparison or difference) 38 Fall 2007 Informatics 231 Human-Computer Interaction

Base rates 39 How often Y occurs in the general case, as a basis for deciding whether the rate of Y in some particular case is “notably” high or low Problems - difference in interpretation - various political, economic and social issues Fall 2007 Informatics 231 Human-Computer Interaction

The Correlation Question 40 Is there covariation in the values of two properties or features of system? Correlations: High – Low; Positive – Negative; Zero Linear or nonlinear relation between two or more variables Can asses conceptual relations that imply covariation between two or more variables, but cannot asses any conceptual relations that are causal in their implications Fall 2007 Informatics 231 Human-Computer Interaction

The Difference Question 41 Whether Y is present under condition where X is present and whether Y is absent when X is absent? Samples are separated in two groups, one with members which have performance X, and another with members who don’t have performance X. Some set of tasks is given to both of groups, and average task performance is compared for both groups. Groups must be comparable on factors that might affect task performances! Informatics 231 Human-Computer Interaction

Randomization and True Experiments 42 Randomization using random assignment procedure to allocate cases to conditions (in order to remove artifacts and observations that happen by chance) In order for the study to be called a “true experiment”, study must include some randomization Random allocation procedure doesn’t guarantee an equal distribution of any of the potential factors among the conditions being compared Fall 2007 Informatics 231 Human-Computer Interaction

Randomization and True Experiments 43 Possible problems: – Reduce the scope of study, as some variables are hold constant, and therefore experimental variables will occur only at a few levels – Reduce the realism of context of your study, designed the tasks serves us - not the participants' - purposes. Fall 2007 Informatics 231 Human-Computer Interaction

Sampling, Allocation and Statistical Inference 44 The way we choose cases which will be included in our study (from larger population of potential cases) effect credibility of the evidence resulting from the study How to choose right nature of the “random sample” population? Fall 2007 Informatics 231 Human-Computer Interaction

Sampling, Allocation and Statistical Inference 45 We need to use random sample in that way that we can apply results of the study to the population of which cases constitute a random sample. Fall 2007 Informatics 231 Human-Computer Interaction

Sampling, Allocation and Statistical Inference 46 Sampling case procedure: which cases from larger population will be included in study. Allocation case procedure: which condition every given cases (already selected as a part of the study) will be assigned to Random refers to procedure, not outcome! Fall 2007 Informatics 231 Human-Computer Interaction

Sampling, Allocation and Statistical Inference 47 Size of the samples? The larger the number the more distributed those cases will approach the idealized random distribution Uneven distribution doesn’t occur often if only chance is operating. Fall 2007 Informatics 231 Human-Computer Interaction

Validity of Findings Four different types of validity (Cook & Campbell): Statistical conclusion validity Internal validity Construct validity External validity 48 Fall 2007 Informatics 231 Human-Computer Interaction

Validity of Findings Statistical conclusion validity: – – Internal validity: – – – 49 Difference arisen just by chance? Relationship between cause and effect How close can you come to asserting that the present of X caused the altered level of Y values? Difference in Y associated with a difference in X does not necessarily imply a causal role for X Are there other factors which may have been covary with X and they, rather than X, might have produced the change in Y Fall 2007 Informatics 231 Human-Computer Interaction

Validity of Findings Construct validity: – – – External validity: – – 50 How well defined are the theoretical ideas in our study? How clearly understood are the conceptual relations being explored? How well are mapping of concepts and relations How confident you can be that your findings will hold upon replication or how general are your our findings. How confident you can be that you can make predictions ? Fall 2007 Informatics 231 Human-Computer Interaction

Potential Classes of Measures In Social Psychology 51 Fall 2007 Informatics 231 Human-Computer Interaction

Strengths and Weaknesses of Types of Measures Self reports Observations by visible observer Observations by hidden observer Archival records of public behavior Archival records of private behavior Trace measures Reactivity High potential (Some participants are aware that their response will be recorded, that may influence how they response – they might try to make good impression, to give socially desirable answers or to try to help investigation) High Low High Low None Versatility Versatile to potential contents and to the population to which they would apply to Can only be used on overt behavior (not feelings, expectations, etc) Med-Low Low Low Low (many concepts not available) Costs Low High High Medium Medium High (sometimes) Dross Rates (discarded info.) Low High (per observer hour bases) High(per observer hour bases) High High Very high (links to concept sometimes very week) Time Low High High Medium Medium High Vulnerable to observer error Ethical concerns, Vulnerable to observer error No possibility to cross validate No possibility to cross validate Errors 52 Fall 2007 Informatics 231 Human-Computer Interaction

Techniques for Manipulating Variables Techniques for manipulating variables are not very well specified Three classes of techniques: Selection Direct intervention Inductions Manipulating variables are use to create an experimental manipulation of features of a situation 53 Fall 2007 Informatics 231 Human-Computer Interaction

Selection 54 Advantage: Selection is the most convenient means to make sure that all cases of a given conditions are alike on a certain variable and that all cases of another comparison condition differ o that variable Disadvantage: conclusion drawn is more or less unreliable unless all other potential factors are removed (uncertainty in nature of variable manipulated (additional meaning)) Fall 2007 Informatics 231 Human-Computer Interaction

Direct intervention Advantages: – – – – Disadvantages: – – 55 Surest way of achieving a definite and specific variable manipulation Permits random allocation (distributing impact of other factors and variables, that we are not studying) Nor likely to be costly or time consuming Low dross rate Applicable only for relatively overt and tangible variables May suffer from reactivity effects and experimental demands Fall 2007 Informatics 231 Human-Computer Interaction

Induction Three major forms 1. 2. 3. Advantages: – – Fairy versatile in context of variables Normally low in cost and time Disadvantages: – – – 56 Use of misleading instruction to the participants Use of false feed back Use of experimental confederates Involves deception (raises ethical concerns) Risk of detection Risk of experimental demands Fall 2007 Informatics 231 Human-Computer Interaction

Conclusion Results depend on methods. All methods have limitations. Hence, any set of results is limited. It is not possible to maximize all desirable features of method in any one study; tradeoffs and dilemmas are involved. Each study (each set of results) must be interpreted in relation to other evidence bearing on the same questions. 57 Fall 2007 Informatics 231 Human-Computer Interaction

Questions? 58 Fall 2007 Informatics 231 Human-Computer Interaction