It is often possible that different research studies concerning the same issue can produce very dissimilar or contradictory results.  How can this happen?  Aren't all statistics "true"?


(ABC University)

A research study conducted at ABC University shows that high cholesterol alone is not a contributing factor to heart disease.

A research study conducted at DEF University shows that high cholesterol alone is a contributing factor to heart disease.
(DEF University)

While the names of these Universities are fictitious, the controversy over the relationship between high cholesterol alone (with no other heart disease risks) and heart disease is still under investigation.

Statistics are influenced by a multitude of factors.  It is even possible that statistics can be manipulated so that they tell the story that the person using them wants to tell. 

Because of these influencing factors, it is important to understand how to evaluate statistical information. 

When viewing statistics, you should consider: 
 

  • Who collected the data?   Does the group collecting the data have an interest in how the results turn out?
  • Example:  A study on the hazards of cigarette smoking being done by a tobacco company. (may not be reliable findings - conflict of interest)
  • Is the study a recent study, or did it occur decades ago?  Could recent developments have changed the findings? 
  • Example:  Decades past, second-hand cigarette smoke was found to not be hazardous.  More recent findings prove that this is not true.
     (findings should be current)
  • What is the sample size of the study?  How many people/items were studied?
  • Example:  A study is done on the favorite color of 14 year olds.  The sample group for the study is Mrs. Smith's third period class containing 20 students.
     (too few participants to generalize a finding to all 14 year olds)
  • Is the data from a primary source?  Or has the data been "condensed" by another group? 
  • Example:  The US Census Bureau collects data on US populations.  A tabloid magazine publishes a synopsis of the findings.
     (the most reliable information comes from the original source - avoid the "Reader's Digest" condensed version by another publisher who may be interpreting the findings)
  • Do the statistics show any bias? 

Read more about BIAS below ↓
  • Example:  The study of how many people can walk a balance beam is conducted with students from a  gymnastics class.
    (the results are biased due to the very specific selection of the participants)

 

Sources of bias:

Selection bias:
     In a statistical study, it is important that the smaller group used for the study (the sample) be truly representative of the larger group to whom the findings will be directed (the population).  Preferably the sample group should be chosen at random.

Dexterity Study:  500 people are invited to a research center for an experiment in dexterity and flexibility.  100 of the people show up.  The researchers document the number of people who can clasp their right foot with their right hand behind their backs, by reaching over their right shoulders (as seen at the right).  They conclude that an amazing 62% of people can perform this act of dexterity and flexibility. 

Are these finding reliable?  What is wrong with this study?

Since only 100 of the 500 invited participants showed up for this study, this is not a representative sample.  It may also be the case that the people who were confident about their dexterity and flexibility showed up and the results are biased in their favor.

Measurement bias:
     In a statistical study, it is important that the means of gathering measured data be reliable, accurate and appropriate for the study.

Dexterity Study Weights:  The people participating in the dexterity study mentioned above were weighed to determine if participants who were not overweight were more flexible.  It was discovered after the study that the scale used for weighing had a tendency to list weights over 150 pounds inaccurately. 

What effect did this have on the results of the study?

This situation presents two problems for the study.  First, the weights are unreliable for participants weighing over 150 pounds.  Second, the question should be asked if the participants' heights were also measured.  If not, what was the definition of an "overweight" participant?