Research Design

1. Rules of the Scientific Method
2. General Procedures of the Scientific Method
3. Research Methods
   1. Naturalistic Observation
   2. Survey Methods
   3. Case History Method
   4. Test Methods
   5. Experimental Method
4. The Role of Statistics
   1. Significance of Differences
   2. Noncausal Relationships
   3. Statistics as a Tool

Homework

1. Universe is assumed to be orderly. So, events have specific causes.
2. It is publicly verifiable. Ex. You can go to the library.
3. It is repeatable for a given lab, as well as across different labs.

1. Ask a question about the world.
2. Operationally define the relevant terms.
   Operational definition - a concept defined by how it is measured (thus, it typically includes numbers). Ex. "Hunger" - 24 hours of food deprivation.
3. Choose a research method. We will talk about several.
4. Collect & statistically analyze the data.
   The latter is what this course is all about.
5. Report the results publicly (i.e., publish or present it).

These differ in the kinds of information about behavior they yield, as well as in the types of behavior to which they are best suited for studying. We will look at five different methods. Note that they are not mutually exclusive.

1. Naturalistic Observation
   Also called systematic observation or the observational method. It is a systematic method for observing behavior as it naturally occurs. Some important issues include:
   1. Unobtrusiveness - subject is unaware they're being observed.
   2. Naturalness - subject is "at home."
   3. Systematic Recording - behavior is measured (or counted) somehow. For example, frequencies (how many), durations (how long), and/or latencies (how long until) might be recorded for operationally defined behaviors. Procedures such as time sampling (where behavior is sampled at regular intervals) might be used. We might want to compute reliabilities to see if the different observers are in agreement about what is being measured.

2. Surveys
   • Includes Questionnaires & Interviews.
   • Require careful consideration of:
     1. Question Structure - should not be leading.
     2. Adequacy of sampling? - Some Definitions:

        Population - the whole group we are interested in.
        Sample - the group we work with.
        

        
        
        

        Random Sample - one in which each member of the population has an equal chance of becoming a member of the sample.
        
        
   • Example of Survey Data

     Premarital Sexual Attitudes
     (from Gallup Poll data)  Sample		% saying it's wrong in:
     		1969	1973
     Nationwide		68	48
     Gender
     	Female	74	53
     	Male	62	42
     Education
     	Grade School	77	60
     	High School	69	45
     	College	56	41
     Geog. Location
     	East	65	38
     	Midwest	69	51
     	South	78	58
     	West	55	41

3. Case Studies
   • Used a lot by clinicians
   • Are several types. Two include:
     1. Retrospective - looks at past events.
     2. Longitudinal or Proactive - follows events as they occur.

4. Test Methods
   • Operationally defines variables.
   • Exs. IQ test, TMAS, BDI, ACT, SAT.

5. Experimental Method
   • Involves manipulating something we choose.
   • Is the most powerful method because it allows us to determine cause & effect.
     
   • Definitions:

     Variable

     Characteristic of a person or thing that can occur in different amounts or kinds. Ex. Performance on a test.

     Independent Variables (IVs)

     We select and manipulate these. So an IV must have at least two levels (or values the IV can take).
     Ex. Amount of sleep deprivation, where the control group is not deprived and the experimental group is 24 hours deprived (they pulled an "all nighter"). In this case, the IV has two levels or values it can take.
     A variable that is similar to an IV, but is not truly manipulated is called an ex post facto variable. Ex. gender; experimenter does not decide who will be male and who will be female.

     Dependent Variables (DVs)

     We measure these.

     Extraneous Variables (EVs)

     Variables other than the IV which can influence the DV. We worry about these. If an EV effects the groups in an experiment differentially, we do not know whether the IV or EV is resulting in the differences in the DV. In this case, we say that the results of the experiment are confounded. EVs can actually result in 3 possibilities. Only one of these is a problem (i.e., confounding).
     
     

     	Possibility	Result
     1	EV has no effect on either group	Not a problem
     2	EV effects all of the groups in the same manner	Not a problem
     3	EV effects the groups differentially (e.g., one & not the other)	Confounding - A problem.

     
     

   • Purpose: To see if changes in the IV cause changes in the DV.
     
     
   • Example: Effects of Marijuana on Memory (Weil, Zinberg, & Nelson, 1968). Numbers given are fictitious, but based on the actual data.
     
     
     • Experiment 1 (2 independent groups). The IV was marijuana with two levels.
       
       

       IV ->	Marijuana
       Levels ->	Control	Experim.
       Average words remembered from 20 5-letter nouns presented at 1 sec intervals	17	12

       
       
     • Experiment 2 (involved more than one IV and is thus called a factorial design). In Experiment 2, they included the variable (ex post facto) of prior experience with the drug.
       
       

       IV2 - Experience	IV1 - Marijuana
       	Control	Experim.
       Naive Users	18	13
       "Pot-heads"	17	18

       
       Note that, in this case, there are three research questions (there is a question for each IV as well as for their interaction):
       1. (As in Experiment 1,) Does marijuana affect memory?
       2. Is prior experience with marijuana related to its effect on memory performance?
          Notice that no mention is made of an effect, because the variable is not something we manipulated.
       3. The interaction question asks whether the effect of one variable depends on the other. That is:
          Does the effect of marijuana on memory depend on prior experience with the drug?
          
          
   • Factors and Groups: An IV can be a between groups factor (involving different groups) or a within groups factor (same group tested repeatedly). Each of these approaches has some advantages and disadvantages that we will learn about later in the semester. For now, you can always tell the number of groups in a design by multiplying out the between groups factors. If you have all within groups factors, then you only have one group. Consider some examples to help make this clearer. In these examples, let b be a between groups factor and w be a within groups factor.

   IVs	Levels(Factor)	Design	Multiplying	# Grps	Comments
   1	2(b)	1 factor between	2=	2	Exp 1 in example.
   1	3(b)	1 factor between	3=	3
   1	2(w)	1 factor within	1 grp tested 2x=	1
   2	2(b) x 2(b)	betw grps factorial	2x2=	4	Exp 2 in example.
   2	2(b) x 2(w)	mixed factorial	2=	2
   2	2(w) x 2(w)	within grps factorial	1 grp tested 4x=	1
   3	2(b) x 3(b) x 4(w)	mixed factorial	2x3=	6	3-Way factorial.

---

IV. The Role of Statistics

1. Significance of Differences
   In an experiment with two groups, there are two reasons why differences may occur.
1. IV or treatment effect.
2. Chance or sampling error.

   
   
   Statistics helps us decide whether the difference is due to the IV (significant).

Important concepts:

• Probability (p) - refers to how likely something is to occur. Probabilities range from zero through one. Thus, in the case of statistics, the observed results could be:

Improbable Due to Chance		Probable Due to Chance

• Alpha level (a) - arbitrary level chosen to separate probable from improbable.
• The Stat test - determines the p that a given difference is due to chance. If the p £ a, we say the difference is significant (or reliable).
  
  
2. Noncausal Relationships
   Variables are not always related in a causal manner. Statistical techniques are available to assess various aspects of relations between variables even when no causal relation exists.
   
   
3. Statistics as a Tool
   It should be clear from our discussion of research design that statistics is a tool of the scientific method. First, a research project is carried out. The data are then analyzed. If the research project was poorly designed, even the most brilliant statistical analysis will not provide a meaningful answer to the original research question.

---

Copyright © 1997-2012 M. Plonsky, Ph.D.
Comments? mplonsky@uwsp.edu.