Assessing “quality” in qualitative research

There is a very large body of literature devoted to thinking about how the “quality” of qualitative research should be assessed. From writing several decades ago in which the concepts of “validity” and “reliability” were redefined and applied to qualitative research (e.g.,  Goetz & LeCompte, 1983; LeCompte & Goetz, 1992), methodologists have argued for the rejection in qualitative inquiry (e.g., “validity”, Wolcott, 1990), re-conceptualized such terms as “validity,” (e.g., Lather, 1993; Scheurich, 2001), and proliferated new terms such as “crystallization” (Ellingson, 2009, 2011). It can be difficult for anyone new to conducting qualitative research to figure out where to begin when so much has been written. Here are a few ways to begin to think about what quality means in qualitative research. It’s useful to recognize that there is much disagreement among scholars as to how standards of evidence should be applied in qualitative inquiry (Freeman et. al, 2007). Further, the use of terminology has changed over time. For example, the application of the concepts “external reliability” and “internal reliability” is now rare in methodological writing about qualitative inquiry. Of course this also depends on the discipline in which qualitative studies are conducted, and the research design used.

Here are some strategies that newcomers to qualitative inquiry might use in thinking about the assessment of “quality” in a qualitative study.

1. Look at what established scholars in a particular discipline do in their research. What terms are used? How are these evident in research? What are the hallmarks of “quality” in published research in the top journals in the field? Frequently journal editors include articles and editorials to do with what they expect in an article reporting findings from a high-quality study. Look out for these kinds of articles in peer-reviewed journals in your discipline.
2. Look at what established writers in a particular methodological tradition do. For example, even though “narrative inquiry” encompasses a family of approaches to doing research, narrative scholars have written about what is needed to do a quality narrative study (e.g., Polkinghorne, 2007; Riessman, 2008); as have autoethnographers (Gingrich-Philbrook, 2013). Therefore, whatever research design and methodological approach one hopes to use, it is useful to examine what methodologists in that tradition write about the topic of “quality” criteria.
3. Consider how different traditions of qualitative inquiry are based on different assumptions about what counts as evidence, and what kinds of assertions might be made. When reading debates about the issue of quality, one might then consider what ontological and epistemological assumptions underlie an author’s argument. How do these differ across traditions?

The fact that there is so much literature written on the assessment of quality of qualitative research speaks to the importance of this topic. For example, I was surprised by the amount of new writing on this topic when I updated this partial list of references (see SOE Bibliography) that my colleagues and I had compiled some years ago. Clearly, researchers are still innovating on this topic, and have much more to say. For those researchers beginning a first project, and struggling to make sense of what the “right” thing to do might be, I would suggest that you start where you are. That is, think about what is valued in your discipline and among the community of scholars in which you hope to situate your own work. Once you read more in this literature, you will find that scholars change their views over time; so it is likely that your own views on this topic will not remain static.

Happy exploring!

Kathy Roulston

References

Ellingson, L. L. (2009). Engaging crystallization in qualitative research: An introduction. Thousand Oaks, CA: Sage.

Ellingson, L. L. (2011). Analysis and representation across the continuum. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed., pp. 595-610). Los Angeles, CA: Sage.

Freeman, M., deMarrais, K., Preissle, J., Roulston, K., & St. Pierre, E. A. (2007). Standards of Evidence in Qualitative Research: An Incitement to Discourse. Educational Researcher, 36(1), 25-32. doi:10.3102/0013189×06298009

Gingrich-Philbrook, C. (2013). Evaluating (evaluations of) autoethnography. In S. H. Jones, T. E. Adams, & C. Ellis (Eds.), Handbook of autoethnography (pp. 609-626). Walnut Creek, CA: Left Coast Press.

Goetz, J. & LeCompte, M. D. (1984). Ethnography and qualitative design in educational research. Orlando, FL: Academic Press.

Lather, Patti. (1993). Fertile obsession: Validity after poststructuralism. Sociological Quarterly, 34(4), 673-693.

LeCompte, Margaret D., and Goetz, Judith Preissle (1992). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52(1), 31-60.

Polkinghorne, D. E. (2007). Validity Issues in Narrative Research. Qualitative Inquiry, 13(4), 471-486.

Riessman, C. K. (2008). Narrative methods for the human sciences. Thousand Oaks, CA: Sage.

Scheurich, J. J. (2001). Research method in the postmodern. Philadelphia, PA: Routledge.