Assessing the Results of the Study 

While the importance of the direction and magnitude of results is obvious, these characteristic sometimes go unreported.  There are thousands of correlations reported in the research literature without an indication of whether they were positive or negative correlations.  There are thousands of experimental differences reported as statistically significant without an indication of the magnitude of the difference.  Qualitative research is usually reported in more detail so direction rarely goes unreported.  Qualitative analysis, however, often provides ambiguous indications of magnitude.  If it says, “the entry-level employees were initially delighted by the firm’s offer to provide tuition assistance at the local community college” does that mean all, most or many of the entry level employees?  And if the qualifier “most” is added to the statement, does that mean at least 90 percent, 80 percent, or 70 percent? 

Averages or summaries simplify complexity and that is often helpful, but they can also hide important information.  It is very easy to accompany mean values with the variance, and that adds considerable information.  Some qualitative researchers are careful to report observed variations, but others, in their effort to indicate themes and patterns, don’t provide a sense of the variation. 

While statistical significance is usually reported in quantitative research, it is widely misinterpreted by researchers and by readers.  When the samples are small (less than 100), a failure to find statistical significance may mean there is no difference in the population or it may mean that there is a modest difference that was not inferred because of the inadequate power of the test.  The power of all statistical tests is positively affected by sample size and inversely affected by the variance in the samples.  Conversely, a finding of statistical significance from large samples (greater than 1,000) can result from trivial differences.  Furthermore, it should also be noted that when 100 hypotheses are tested at the .05 level of significance, there is a good chance of finding about 5 statistically significant results by chance when there are no real differences in the population.

Most quantitative research involves multiple results, and the pattern of the results is far more important than any one by itself.  For that reason, it is important that the report indicate all the results, but that may not be the case.  Researchers will usually report all their statistically significant results, but they sometimes fail to report some or all of the results that were not significant.  The following example will illustrate how this can be grossly misleading. In a study of the effects of Math Explosion software on Hispanic youths, it might be reported that the intervention had statistically significant results for mathematical computation skills.  That seems interesting and encouraging until you learn that four other measures of math achievement did not show statistically significant results!  The full set of measures might have included a standardized achievement test with scores for computation, concepts, and problem-solving; the students’ math course grades; and their quantitative score on the SAT college admissions test. 

Qualitative researchers don’t have any device comparable to statistical significance.  That makes the pattern of their results even more important than in quantitative research.  Good qualitative research explicitly cross- verifies important results (“triangulate”) and indicates when some but not all of the evidence points in a given direction. 

How can you ascertain whether a research report indicates all the results?  Sometimes it will say that is has or has not.  Sometimes more measures are discussed in the data collection section than are reported in the results section of the report.  In addition, the following three conditions that should raise concerns about selective reporting of the results: 

1. Do all the reported hypothesis tests have statistically significant results?  Rarely are all the results significant except when five or more significance tests are conducted.
   
   
2. Are all the reported results consistent?  For instance, do all the results favor a demonstrated innovation or do all show a given social arrangement to be dysfunctional?  Since social phenomena are rarely perfectly consistent, high consistency in the results suggests selective reporting by the researcher.
   
   
3. Is there an indication that the researcher was surprised by some of the results? This is usually found toward the end of the report.  It is almost impossible to do honest research without encountering some surprises and being intrigued by them.  If the researcher found no surprises, it is possible that the whole study was skewed to support his or her pre-held opinions or that selective reporting buried unwanted results.

Key Assessment Questions

10. What is the direction, magnitude, variance, and statistical significance (when applicable) of each result? 

11. What is the pattern of results for each broad question that was addressed?

Last Update: June 29, 2000