How to Replicate a Qualitative Review: A Replication of "A Systematic Review of Psychosocial Functioning Changes after Gender-Affirming Hormone Therapy among Transgender People"

The systematic review by Doyle, Lewis, Barreto (2023) examines the potential impacts of gender-affirming hormone therapy (GAHT) on psychosocial functioning in trans people. They offer a narrative review of a body of 46 relevant papers, where they find that gender-affirming hormone therapy is consistently linked with reductions in depressive symptoms and psychological distress. Results are more mixed for other domains, such as self-mastery and quality of life. Here, we report 5 different replication tests. In the absence of existing guidelines for how qualitative research can be replicated, we aimed to assess different parts of the systematic review and narrative synthesis in various ways. These include (1) a computational replication where we re-ran the original search strategy, a (2) robustness replication where we offered an independent narrative synthesis, (3) a direct replicability test where we reviewed new relevant papers on the topic, (4) a robustness replication with a large language model (LLM), where we provided an LLM-led synthesis and also verified the summary table, and finally (5) another robustness replication where we offered one more ‘coarse agreement’ score for another part of the original narrative synthesis. Tests (2-4) verified and successfully corroborated the results reported by Doyle and colleagues. Although we were not able to replicate the search strategy (test 1), we acknowledge that this is an uncommon replication test, which has previously been described as challenging in the meta-science literature (Koffel and Rethlefsen, 2016). Thus, overall, we believe the findings we have examined in this report to be robust. We discuss our replication tests in the context of recommendations to journals to require the provision of full, reproducible search strategies, as well as supporting initiatives like ‘living’ systematic reviews, which allow for easier updating with new data.