Doing Critical Realism (Part I): Reflections on Data Analysis

Introduction

            My conversations about critical realism have been characterized by a mix of enthusiasm and confusion—whether internal conversations (Archer, 2014) or ones with academic peers. Critical realism can sound like an exciting philosophy of research that rigorously separates reality from claimed knowledge of it. But it can also be a complex metatheory of sometimes unclear practical use. I lean to the view that it is complex and that its practical use may not be as clear as that of its competing alternatives, chiefly positivism and social constructivism. However, I believe critical realism has to date presented me a good philosophical framework within which I have been able to progress methodologically, rigorously, and coherently in my doctoral research. Let me begin with sharing my evolving understanding of it.

Critical realism stands on a trinity of assumptions: ontological realism, epistemic relativism, and judgmental rationalism. Let's unpack each of these. Ontologically, reality is viewed  as being comprised of three domains: an empirical domain of perceptions and subjective experiences, an actual domain of events and activities, and a real domain of causal structures and relationships (see Figures 1). A good illustration of this stratified reality is the case of a patient: their reported feelings and observable symptoms falling in the empirical domain, their diagnosed illness in the actual domain, and the causes of their diagnosed illness in the real domain. 

Figure (1): An iceberg metaphor for critical realism (Fletcher, 2017, p. 183)

Then, there is the epistemological view of relativism: that inferences of the actual and of the real may well be limited by one’s earlier training experience, intellectual and cultural traditions, and life experiences, among other factors. In the illustrative case, a paediatrician’s interpretation of symptoms of a child’s illness may be different from an oncologist’s—and most likely more accurate than a GP’s. And, there is finally judgmental rationalism, the view that a knowledge claim, before being concluded, can and should be adjudicated vis-à-vis ontologically plausible alternatives. Results of a diagnosis involving multiple tests of several potential causes may build stronger possible explanations of a patient’s illness than those based on a diagnosis conducted through testing for fewer potential causes (see Figure 2).

Figure (2): The trinity of critical realist assumptions

This trinity may at first sound complex to grasp, let alone translate into actual practice of social science research. I remember it took me several days to just learn the terms of critical realism. It also took me several weeks to understand how I can apply these views in my own research—and especially to cultivate the confidence to do so. This perceived complexity notwithstanding, I believe critical realism has thus far offered me so much in designing and implementing my research.

In this three-blog series, I wish to share reflections on the value I have found in following critical realism particularly through my data analysis. Before getting to it, it is worth sharing a bit about my doctoral research. Funded by the ESRC through NINE DTP, my research looks at the potential impact of international higher education scholarships vis-à-vis peace in Palestine. I am trying to understand how funded graduate education abroad may influence Palestinian scholarship recipients’ effective abilities to respond to conflict logics and impact. I think of these effective abilities as capabilities for everyday peace, a theoretical synthesis I am developing of the Capability Approach (Sen, 2001) and Everyday Peace Approach (Mac Ginty, 2021). As may be clear, this research requires working across the two fields of international education and peace studies. As an insider researcher, being a Palestinian scholarship recipient myself, the research also requires a higher level of reflexivity. I will be sharing reflections on this interdisciplinarity and reflexivity in a later blog of this series. For now, I reflect on how critical realism has helped plan and apply a rigorous approach to my data.

Rigor in Data Collection and Analysis

            Data collected for my research were of the participants’ perceived experiences and outcomes of their funded education abroad. While planning it, data collection looked to me like it would involve two potential risks. The first is social desirability bias (Leshem et al., 2019), which is important to consider in researching the relationship of international education to peace (see Wilson, 2015). The second is my own potential confirmation bias as an insider researcher with the same lived experiences as my participants (Nickerson, 1998). Thinking within critical realism, I reasoned that I could mitigate both of these risks by not seeking direct data of participants’ perception of the contribution of their education abroad to peace. For example, the participants did not need to report thinking that, through their education abroad, they engaged in recovery from the impact of the Israeli-Palestinian conflict (empirical domain) in order for me to be able to infer that they may have done that (actual domain). Instead, I sought data that spoke to the field of international education and whose theoretical analysis would then speak to the field of peace studies, a point I revisit in my next blog on interdisciplinarity and reflexivity. As such, my interview questions were about the participants’ perceived experiences, engagements, and outcomes of their education abroad—in academic, cultural, social, professional, civic, or other settings—rather than their perception of any relevance thereof to peace.

Analysis in the Empirical Domain

Critical realism informed this risk mitigation plan by offering the ontological recognition that one event (in the actual domain) can be experienced in different ways by different people (in the empirical domain). Based on this recognition, I first applied data-driven coding, or descriptive analysis, to my participants’ reported experiences and perceived outcomes (see Wiltshire & Ronkainen, 2021). Findings from this layer of analysis are experiential, and they remain in the empirical domain of reality. Separating this layer of analysis from the subsequent two, the actual and the real, allowed me to approach my participants’ accounts more openly, identifying and keeping patterns in the data as they appear rather than giving precedence to (my/any) theoretical interpretation thereof. To illustrate, one empirical pattern emerging from the data is that most participants (81%) shared mixed expressions of appreciating and struggling through their new multicultural environments abroad. As reflected in its phrasing, this finding is of and about the participants’ own reported subjective testimonials (see Wiltshire & Ronkainen, 2021). I operated this level of analysis using critical realist thematic analysis as my method. Critical realist thematic analysis builds on Braun and Clarke's development of thematic analysis but integrates across its steps techniques of critical realist analysis (2006, cited in Fryer, 2022; and in Wiltshire & Ronkainen, 2019). 

Analysis in the Actual Domain

Now, working in the actual domain, I am applying abductive analysis to experiential findings. By abductive analysis, I refer to the process of inferring and interpreting a particular phenomenon based on analysis of empirical data vis-a-vis theoretical concepts (Fletcher, 2017, drawing on Danermark et al., 2002). In this level of analysis, I redescribe experiential findings in terms of my theoretical framework and thereby infer functionings of everyday peace. Ongoing dialogic refinement of my framework notwithstanding, these functionings can theoretically range from disruption (questioning conflict logics) to navigation (finding pathways to act on alternative logics) and to transformation (following logics and actions that actively reconstruct a conflict legacy from one of destruction to one of rebuilding). In inferring such functionings, I have found continued usefulness of critical realism. I share two aspects of this usefulness in this blog, with more to come in later blogs.

First, given the nature of experiential findings, a theoretical search of them for such functionings may not be as (interdisciplinarily) plausible or even logical. However, the ontological recognition in critical realism of the actual as not always observable readily lends itself to warranting it. Second, given this clear layering of the analysis process, critical realism has been helpful to me in identifying what and where to apply specific measures of analytical rigor and quality. Following Wiltshire and Ronkainen (2019), I am now making theoretical inferences from experiential findings based on three main checks:

• empirical adequacy—how much and how strong is the empirical evidence based on which the inference is made;
• interpretive validity—how accurate and how fair is the re-presentation of empirical data in the claimed functioning; and
• ontological plausibility—how well can the claimed functioning be said to represent real-world occurrences. 

Any inferable functionings with high degrees of empirical adequacy, interpretive validity, and ontological plausibility would then be claimed as functionings of everyday peace that are involved in Palestinian scholarship recipients’ education abroad. Such actual findings remain as such, claimed functionings. They reflect my interpretation of the participants’ data while demarcating the two, thereby allowing a higher degree of transparency and potentially greater room for others to verify the research confirmability.

Conclusion

I believe the distinction in critical realism between different domains of reality and the resulting demarcation of descriptive from inferential analysis allowed me to mitigate against the risk of social desirability in both data collection and data analysis. For me, this demarcation also involved greater clarity as to how to engage in inferential analysis that is rigorous. Also, this clearer separability of the perceived and empirical from the inferred and actual avails to me clearer room to practice interdisciplinarity and reflexivity in the analysis, in the actual and real domains, which I discuss in my next blog. 

References

Archer, M. S. (2014). Structure, Agency and the Internal Conversation. Cambridge University Press.

Fletcher, A. J. (2017). Applying critical realism in qualitative research: methodology meets method. International Journal of Social Research Methodology, 20(2), 181-194. https://doi.org/10.1080/13645579.2016.1144401

Fryer, T. (2022). A critical realist approach to thematic analysis: producing causal explanations. Journal of Critical Realism, 21(4), 365-384. https://doi.org/10.1080/14767430.2022.2076776

Leshem, O. A., Nooraddini, I., & Witte, J. C. (2019). Surveying Societies Mired in Conflict: Evidence of Social Desirability Bias in Palestine. International Journal of Public Opinion Research, 32(1), 132–142. https://doi.org/10.1093/ijpor/edz002

Mac Ginty, R. (2021). Everyday Peace: How So-Called Ordinary People Can Disrupt Violent Conflict. Oxford University Press.

Nickerson, R. S. (1998). Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

Sen, A. (2001). Development as Freedom. Oxford University Press.

Wilson, I. (2015). Exchanges and Peacemaking: Counterfactuals and Unexplored Possibilities. All Azimuth, 4(2), 5-18.

Wiltshire, G., & Ronkainen, N. (2021). A realist approach to thematic analysis: making sense of qualitative data through experiential, inferential and dispositional themes. Journal of Critical Realism, 20(2), 159-180. https://doi.org/10.1080/14767430.2021.1894909

Author Biography

Anas N. Almassri is a PhD Candidate in Education and an incoming Teaching Assistant at the Schools of Education and of Government and International Affairs at Durham University. Anas works on international education and peace, particularly on their interplay in contexts of protracted conflict or crisis. Anas is a Student Fellow at the Durham Research Methods Centre, and he currently serves as the Editorial Assistant for Frontiers: The Interdisciplinary Journal of Study Abroad.