This week’s blog post is from Dr. Sean Halpin, who is a Qualitative Analyst with RTI-International, on the Genomics, Ethics, and Translational Research team. Dr. Halpin has over a decade of experience leading socio-behavioral studies across a wide range of chronic and infectious disease areas and has published numerous journal articles to do with patient care. His responsibilities at RTI include preparing research proposals, developing and executing research protocols, overseeing data collection and analysis, interpreting the research results and supporting sponsors’ strategic goals, managing operational and financial aspects of research studies, and disseminating results. Dr. Halpin has a Ph.D. in qualitative research and evaluation methodologies from the University of Georgia and an MA in developmental psychology from Teachers College, Columbia University.
Expanding the Conversation on Rigor in Qualitative Research
In my previous blog post on inter-rater reliability (Halpin, 2023), I explored the role of quantitative approaches to determining qualitative coding agreement between two or more coders in research and their limitations. That post eventually led to my published paper on inter-coder agreement (ICA) in qualitative coding (Halpin, 2024), where I argued that ICA is most appropriate for studies that meet narrow criteria—particularly those grounded in positivist traditions that use deductive coding. But what about research that doesn’t fit into that mold?
Many researchers assume ICA is the gold standard for rigor in qualitative research, but that’s not always the case. While ICA may make sense for structured, positivist studies, it isn’t necessary. Further ICA can even be misleading—especially for approaches that prioritize interpretation and meaning-making. So, let’s explore some different ways of establishing rigor by focusing on two approaches to thematic analysis—one that allows for structured coding but also accommodates interpretive approaches (Hsieh & Shannon’s [2005] Content Analysis) and one that fully embraces researcher subjectivity (Braun & Clarke’s [2022a] Reflexive Thematic Analysis).
My goal is to explore the nuance of how different approaches to showing rigor may or may not be appropriate given a researcher’s chosen analysis method. Importantly, rigor in qualitative research is not just about agreement between coders but also about transparency, reflexivity, and credibility in the analytical process.
What Does Rigor Even Mean in Qualitative Research?
Rigor is one of those words that gets thrown around a lot, but it can mean different things depending on how you approach research. At its core, rigor is about making sure your findings are trustworthy, credible, and transparent. However, the specific strategies used to achieve rigor vary based on the research paradigm and study design.
- Positivist approaches: These assume that reality exists outside of us and that research should be objective, replicable, and structured. Rigor is often associated with reliability and validity, ensuring that results can be replicated by others and that biases are minimized. Inter-coder agreement (ICA) and statistical methods of coding comparison are commonly used to establish external reliability, which is particularly important in fields where findings may directly influence decision-making, such as clinical trials (Krippendorff, 2022). Internal reliability, by contrast, is more relevant when training coders within a single research team (Cascio et al., 2019).
- Constructivist approaches: These recognize that meaning is co-created by the researcher and the participant, and therefore emphasize interpretation, reflexivity, and depth. Rigor in these approaches is assessed through trustworthiness—a concept that encompasses credibility, transferability, dependability, and confirmability (Lincoln & Guba, 1985). Statistical coding agreement methods are less relevant here, as they may give a false sense of rigor (Cook, 2012; Morse, 2020). Instead, strategies such as thick description (Geertz, 1973/2021), reflexivity (Watt, 2015), negative case analysis (Denzin, 2017), triangulation (Denzin, 2017), and member checking (Lincoln et al., 1985, 2013) are employed to ensure the credibility of findings.
Regardless of the approach, rigor requires intentionality in method selection. While saturation is often used in structured qualitative designs to determine when sufficient data has been collected (Guest et al., 2006; Francis et al., 2010), it is not appropriate for Reflexive Thematic Analysis (Braun & Clarke, 2021). Instead, researchers should select rigor strategies that align with their epistemological stance and analytic method.
Hsieh & Shannon’s (2005) Content Analysis: The Structured Approach to Rigor
Hsieh & Shannon’s qualitative content analysis provides a structured yet flexible approach to qualitative data analysis. Their framework includes three types of content analysis:
- Conventional Content Analysis: Codes emerge inductively from the data with minimal predetermined structure. This approach is used when little existing theory or research is available to inform coding categories.
- Directed Content Analysis: Researchers use existing theory or prior research to guide coding categories. This approach provides a structured analysis while allowing for new themes to emerge if the data indicates additional relevant insights.
- Summative Content Analysis: Researchers quantify the presence of certain words or phrases before interpreting deeper meanings. This approach blends quantitative and qualitative techniques, making it useful in large-scale document analysis or policy research
This range of content analysis can align with both structured, deductive coding approaches and more exploratory, interpretive methods depending on how it is applied. While Directed Content Analysis relies on predefined frameworks, Conventional Content Analysis shares similarities with thematic analysis in that it allows themes to emerge inductively from the data.
How Does Hsieh and Shannon’s Content Analysis Ensure Rigor?
- Use of Established Frameworks (in Directed Content Analysis): Researchers apply predefined categories drawn from existing theories or literature to structure coding.
- Inter-Coder Agreement (in Directed Content Analysis): Hsieh and Shannon highlight that ICA can be used when categories are predefined to maintain consistency across coders.
- Clear Documentation of Coding Decisions: Regardless of the type of content analysis used, transparency in decision-making is essential for credibility.
- Integration of Theory and Data: Especially in directed content analysis, findings are grounded in prior research, ensuring alignment with established concepts.
What’s the Downside?
Because content analysis can be applied in different ways, rigor depends on how the analysis is implemented. A heavily structured approach may miss emergent themes, while an inductive approach requires clear reflexivity and transparency to ensure trustworthiness. Researchers need to be intentional in selecting which type of content analysis best suits their research goals.
Braun and Clarke’s (2006, 2019, 2022b) Reflexive Thematic Analysis: The Meaning-Making Approach
In contrast, Braun & Clarke’s Reflexive Thematic Analysis argues that qualitative research isn’t about achieving coder agreement—it’s about engaging deeply with the data and allowing themes to develop organically. Importantly, Braun and Clarke have emphasized the importance of selecting appropriate ways of showing rigor when using their method in a recent publication (Braun and Clarke, 2024).
How Does Reflexive Thematic Analysis Ensure Rigor?
- Transparency in Coding Decisions: Researchers document how themes evolve.
- Reflexivity Journals: Researchers track how their perspectives shape interpretations.
- Thick Description: Findings are richly detailed and contextually embedded.
- Iterative Analysis: Codes and themes change as researchers engage more with the data.
Why ICA is not used in Reflexive Thematic Analysis
Since researcher subjectivity is central to Reflexive Thematic Analysis, expecting two analysts to generate identical codes doesn’t make sense. Instead, rigor comes from coherence, depth, and transparency in analytical decision-making, rather than statistical reliability measures.
Final Thoughts: Rigor is Not One-Size-Fits-All
Qualitative research is diverse, and different studies require diverse strategies for ensuring rigor. While inter-coder agreement works well for structured, deductive approaches, it doesn’t suit methodologies that prioritize interpretation and meaning-making. The key is not to force a one-size-fits-all approach but to critically evaluate what makes sense for the research at hand.
For those conducting structured, positivist-oriented research, rigor is often about consistency and reliability—ensuring that coding decisions can be reproduced and that findings hold up under scrutiny. In its directed form, content analysis (Hsieh & Shannon, 2005) aligns with this paradigm, emphasizing predefined categories and clear coding frameworks.
On the other hand, researchers working from constructivist or interpretivist perspectives emphasize depth, nuance, and researcher reflexivity. Braun and Clarke’s Reflexive Thematic Analysis (2006, 2019) provides a model where rigor comes from transparent documentation of analytical decisions, reflexivity, and engagement with the data over time. Instead of treating differences in interpretation as weaknesses, this approach acknowledges that researcher subjectivity is an essential part of the process.
At the end of the day, rigor is about ensuring research findings are credible, defensible, and meaningful. It’s not about applying the same checklist to every study but about choosing the right tools for the job. Researchers should feel empowered to make methodological choices that align with their epistemological stance, research questions, and data collection strategies.
References
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589-597. https://doi.org/10.1080/2159676X.2019.1628806
Braun, V., & Clarke, V. (2022a). Supporting best practice in reflexive thematic analysis reporting in Palliative Medicine: A review of published research and introduction to the Reflexive Thematic Analysis Reporting Guidelines (RTARG). Palliative Medicine. https://doi.org/10.1177/02692163241234800
Braun, V., & Clarke, V. (2022b). Thematic analysis: A practical guide. SAGE. https://doi.org/10.1177/1035719X2110582
Braun, V., & Clarke, V. (2024). Supporting best practice in reflexive thematic analysis reporting in Palliative Medicine: A review of published research and introduction to the Reflexive Thematic Analysis Reporting Guidelines (RTARG). Palliative medicine, 38(6), 608-616. https://doi.org/10.1177/02692163241234800
Cascio, M. A., Lee, E., Vaudrin, N., & Freedman, D. A. (2019). A team-based approach to open coding: Considerations for creating intercoder consensus. Field Methods, 31(2), 116-130. https://doi.org/10.1177/1525822X19838237
Cook, T. D. (2012). The false promise of qualitative coding reliability. Qualitative Inquiry, 18(1), 62-68. https://doi.org/10.1177/1077800411434944
Denzin, N. K. (2017). The research act: A theoretical introduction to sociological methods. Routledge.
Francis, J. J., et al. (2010). What is an adequate sample size? Operationalizing data saturation for theory-based interview studies. Psychology & Health, 25(10), 1229-1245. https://doi.org/10.1080/08870440903194015
Geertz, C. (1973/2021). The interpretation of cultures. Basic Books.
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? Field Methods, 18(1), 59-82. https://doi.org/10.1177/1525822X05279903
Halpin, S. N. (2023). Inter-rater reliability in qualitative coding: Considerations for its use. QualPage Blog. https://qualpage.com/2023/08/31/inter-rater-reliability-in-qualitative-coding-considerations-for-its-us
Halpin, S. N. (2024). Inter-coder agreement in qualitative coding: Considerations for its use. American Journal of Qualitative Research, 8(3), 23-43. https://doi.org/10.29333/ajqr/14887
Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277-1288. https://doi.org/10.1177/1049732305276687
Krippendorff, K. (2022). Content analysis: An introduction to its methodology (4th ed.). Sage.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Sage.
Lincoln, Y. S., & Guba, E. G. (2016). The constructivist credo. Routledge.
Morse, J. M. (2020). The changing face of qualitative inquiry. International Journal of Qualitative Methods, 19, 1-7. https://doi.org/10.1177/1609406920950849
Watt, D. (2015). On becoming a qualitative researcher. Qualitative Inquiry, 11(2), 161-176. https://doi.org/10.1177/1077800404270840
QualPage 