Values-driven causal pathways evaluations treat rigor not as control and replicability, but as how well evidence makes complex systems change visible in context. They centre evaluator reflexivity, methodological transparency, and diverse evidence, guided explicitly by values like equity, complexity, and learning utilisation. Inclusive, participatory practice and rubrics-based criteria such as triangulation, plausibility, transferability, and representativeness are used to judge causal claims credibly in dynamic, hard-to-randomise settings like philanthropy, governance, education reform, and civil society strengthening programmes.
A Webinar Summary from the Causal Pathways Initiative
When evidence falls short of explaining how and why change happens in complex social systems, evaluators need more than a randomized controlled trial. That is the core message behind a recent training webinar hosted by the Causal Pathways Initiative, led by trainers Florencia Guerzovich (Independent Consultant, Brazil) and Drew Koleros (Mathematica, USA). The session drew on a growing international network of evaluators, methodologists, and philanthropic leaders committed to making the “black box” of systems change visible.
Why Traditional Rigor Falls Short
Conventional evaluation approaches tend to equate rigor with methodological controls rooted in assumptions of stability, fidelity, and replicability. These designs work well in controlled settings, but they become brittle when applied to complex, adaptive systems where change is non-linear, context-dependent, and emergent. The webinar argued that rigor in systems-change evaluations must be redefined to encompass dimensions like complexity, inclusion, and plural causal pathways — not just statistical robustness.
This reframing draws directly on a 2025 guidance paper by Marina Apgar and Tom Aston (“How do we define and support quality and rigor in Causal Pathways evaluation?”), which underpins the entire training. The shift is significant: instead of asking “did this intervention work?”, evaluators ask “how, for whom, and under what conditions did this work?” — a question that demands richer, more contextually sensitive methods.
The Evaluator as a Variable
One of the most distinctive features of the approach is its attention to the evaluator’s own role in shaping evidence quality. The webinar identified two anchor practices: reflexivity and credibility. Reflexivity requires evaluators to actively map their own positionality and biases, reflect collectively with participants and co-evaluators, and embed this practice into analysis rather than treating it as a one-time check. Credibility, by contrast, is built through methodological transparency, articulating explicit causal pathways, attending to a broad range of outcomes, understanding contextual variation, using an iterative “bricolage” approach that draws on multiple methods, and maintaining ongoing dialogue with stakeholders.
These practices position the evaluator not as a neutral observer but as an active participant whose choices and perspectives directly shape what counts as credible evidence in a given context. This epistemological honesty is itself a quality criterion.
Values as a Structural Component
The webinar took the provocative position that values are not peripheral to evaluation design but constitutive of it. The Causal Pathways Initiative’s rigor framework is organized around three core values: equity, complexity, and learning and utilization. Equity means centering inclusion, representation, and power-sharing throughout the evaluation process. Complexity means being context-aware and non-linear, attending to both intended and unintended effects. Learning and utilization means producing iterative evidence that can immediately inform strategy.
A practical mini case study illustrated this with a U.S. evaluation of virtual and hybrid school models conducted in the wake of post-COVID narratives claiming these models “do not work”. The evaluation team surfaced their shared values through individual reflection exercises followed by collective prioritization sessions, ultimately agreeing on four criteria: collaboration with program users, elevating lived experience, critical reflection across multiple sources of evidence, and usefulness for school model improvement. These values then directly guided which evidence quality criteria were prioritized.
Inclusive and Participatory Approaches
The webinar structured participatory practice around six stages of a causal pathways evaluation: preparing, defining evaluation and learning questions, designing for causal analysis, applying methods and gathering data, sensemaking, and supporting use of findings. At every stage, the question is: who needs to be involved, whose questions are included, and how do we attend to power dynamics? This is not a checklist exercise but a genuine reorientation toward co-ownership of the evidence-generating process.
This approach aligns closely with traditions in participatory action research, but it goes further by embedding participation into the causal analysis itself rather than limiting it to problem identification or dissemination. When communities participate in sensemaking, they build ownership of findings that makes use far more likely.
Rubrics as Operationalized Values
Perhaps the most concrete methodological contribution of the webinar is the use of rubrics as structured frameworks for assessing change claims. Rubrics make explicit what aspects of performance are being assessed, define sequential levels of performance, and specify what evidence is required at each level. The webinar identified four ways rubrics strengthen causal evaluations: they help judge the overall methodological approach, ground specific evidence-based judgments, support decisions about the strength of causal claims, and operationalize stated values.
Drawing on Apgar, Aston, Snijder, and Zwollo’s 2024 article “Raising the Bar” in The Foundation Review, the webinar presented ten potential quality criteria: transparency, triangulation, plausibility, uniqueness, representativeness, transferability, ethics, responsiveness, independence, and utilization. For the school models evaluation, the team prioritized four: triangulation (using multiple lines of evidence), plausibility (coherence of the causal narrative), transferability (clarity about contextual conditions), and responsiveness and representativeness (centrality of participant perspectives in the evidence). A rubric then assigned clear performance levels to each criterion, enabling final quality ratings that could be defended publicly.
Suggested Use Cases
The approach is particularly well-suited to contexts where experimental designs are ethically or practically impossible, where change is driven by multiple interacting actors, or where the goal is to generate learning rather than accountability alone. Concrete fields of application include:
- Philanthropic evaluation of systems change strategies, where funders need to understand whether their theory of change is playing out and why, not just whether a pre-specified outcome was achieved.
- Social accountability and governance programs, where causal pathways evaluations have been applied to understand how citizen voice shapes institutional behavior across different governance contexts.
- Education system reforms, especially where adaptive or locally-led implementation means that the “program” itself varies substantially across sites.
- Civil society strengthening initiatives, where evidence of contribution rather than attribution is both more honest and more useful for ongoing learning and strategy adaptation.
The approach also offers a compelling alternative for development cooperation evaluators who work within GIZ, BMZ, or EU external action frameworks and who face the recurring challenge of evaluating interventions in politically complex, low-control environments where traditional log-frame logic produces technically compliant but strategically shallow evaluations.
Further Reading
Webinar recording: https://youtu.be/A5w_Mec5Z1U
Causal Pathways Initiative (slides, case studies, resources): https://www.causalpathways.org
BetterEvaluation Causal Pathways Resource Hub: https://www.betterevaluation.org/methods-approaches/themes/causal-pathways
Raising the Bar: Improving How to Assess Evidence Quality in Evaluating Systems-Change Efforts: https://scholarworks.gvsu.edu/tfr/vol16/iss2/11/
CLARISSA’s Quality of Evidence Rubrics: https://clarissa.global/resource/clarissas-quality-of-evidence-rubrics/
A participatory approach to exploring causal analysis (case study PDF): https://www.causalpathways.org/_files/ugd/5a867c_7c84e6119d1245059c17fd4cb65d3422.pdf
How, for Whom, and Under What Conditions: Using a Causal Pathways Approach to Support the Goals of Philanthropic Evaluation: https://scholarworks.gvsu.edu/tfr/vol18/iss2/6
Navigating Complexity: Combining Theories of Change, Typologies, and Rubrics for Effective Evaluation: https://medium.com/@florcig/navigating-complexity-combining-theories-of-change-typologies-and-rubrics-for-effective-6bcf1cf70b26
Letting Go, Without Losing Coherence: How Local Agency and Donor Accountability Can Coexist: https://twpcommunity.org/wp-content/uploads/2026/03/Letting-Go-Without-Losing-Coherence_How-Local-Agency-and-Donor-Accountability-Can-Coexist-1.pdf
What Happens When Civil Society Asks for MEL, Even When Donors Don’t: https://medium.com/@florcig/what-happens-when-civil-society-asks-for-mel-even-when-donors-dont-1d97b941fe53
Any opinions, interpretations, or conclusions expressed are solely those of the author. The texts and information presented were created with the support of artificial intelligence.