Overview of Assessing
Research Literature

What causes invalid results and misleading conclusions in research studies?  Some are caused by incompetence, when the researchers are not adequately prepared to conduct the study.  Some are due to constraints in resources (money, staff, time) and access that force scholars to conduct research that is less thorough and rigorous than they would desire.  Some result from honest mistakes and others from carelessness. Finally some are due to duplicity--deliberately skewing the research so that it serves only to buttress preconceived opinions. 

Most research journal editors have two knowledgeable scholars review each article submitted for publication.  This is supposed to identify weaknesses and used either to request improvements or to reject the submitted article.  Ulrich's International Guide to periodicals indicates which use peer review.  University presses and commercial publishers of text books usually require peer review; most other commercial publishers do not use it.

Peer review definitely helps screen out the worst research, but there is compelling evidence that it allows considerable flawed research to reach print.  For instance, in a study of 114 research articles sampled from 44 journals, the average expert ratings of the problem statements, research procedures, data analysis, and summary and conclusion were each “mediocre” (Ward, Hall, and Schramm, 1975).

The merit of a research study can be considered like a chain hanging from a stout crossbeam, holding up a heavy pallet of knowledge.  If any one link has a major crack, the pallet is in danger of crashing to the floor.  The links include the background preparation for the study, the conceptual framework, the questions or hypotheses addressed, the contexts prevailing during the study, the interventions actually applied in experiments and demonstrations, the methodology of the study, the results, and the conclusions inferred from the study. 

Appraisal Guides

The following are some general guidelines that should help when assessing individual research studies.  The guidelines will be discussed broadly so that they are generally applicable to both quantitative and qualitative research, although not every discussion will be applicable to all research approaches.

1. Titles sometimes have more hype than substance.  Don’t assume the title of a research article accurately reflects what was studied or the results.

 
2. Try to assess the most thorough report of the research study.  Often there are three levels of reports: one intended mostly for policy makers and practitioners and focusing mostly on the interventions, results, and conclusions; one published in a research journal with moderate detail about the background, questions or hypotheses, methodology, and the results and conclusions; and one submitted to a funding agency with the most extensive detailing of the study.

 
3. Not all cracks in the chain links will cause the pallet to fall.  Whether the chain will hold the pallet depends on the strength of the links, the depth of the cracks, and the weight of the pallet.  Just as it is a mistake to assume every published research study can be relied upon, it is a mistake to assume that any shortcoming in a study makes it useless.  Careful and wise judgment is needed when assessing the merit of individual research studies.

 
4. Beware of your own biases.  Even seasoned scholars are quicker to identify the flaws in studies with results that contradict their opinions than those with results that support their opinions.  Work consciously to exercise even-handed judgment.

 
5. There are at least three levels of assessment.  These include: 
• What are the characteristics of the study that may have affected the results?  What are the implications?  These questions are important to the synthesis phase of the review. 
• Does the study validly answer the questions addressed or validly test the hypotheses posed?  This is often referred to as “internal validity.” 
• Are the results sufficiently large and/or generalizable to be important to theory, policy, and/or practice?  This is often referred to as external validity.

 
6. At the end of each Lesson A-2 through A-8 there are one or more “Key Appraisal Questions. ”  These will help you make a good assessment of the merit of each individual study that has been found during a literature search.  The answers to these questions will also be important for the third phase of a literature review: the synthesis phase.  Depending on the priority purposes of your review and the nature of the literature being reviewed, it may be important to address additional questions or sub-questions during the appraisal of each study.

 
7. When assessing multiple research studies, code answers to the appraisal questions in a matrix.  The rows in the matrix should represent the studies and the columns the appraisal questions, or vice versa. 

This is the first of eight brief lessons on assessing research literature. The other lessons are indicated below. For moderate competency in conducting literature reviews, one should master all the lessons. 

Lesson A-2: Assessing the background of the study
Lesson A-3: Assessing the questions or hypotheses addressed
Lesson A-4: Assessing the contexts of the situation studied
Lesson A-5: Assessing the interventions (when applicable) that were studied
Lesson A-6: Assessing the methodology of the study
Lesson A-7: Assessing the results of the study
Lesson A-8: Assessing the conclusions of the study 
Treasure Chest on Assessment 

Reference
Ward, A. W., Hall, B. W., & Schramm, C. F. (1975). Evaluation of Published Educational Research: A National Survey. American Educational Research Journal, 12 (2), 109-128. 

Last Update: June 29, 2000