Hypotheses generation † Generate hypotheses based on CMO configurations Rephrase CMO configurations into hypotheses Formulate hypotheses to be tested during stage three 3. An approach especially to impact evaluation which examines what works for whom in what circumstances through what causal mechanisms, including changes in the reasoning and resources of participants. Astbury, B. The data on outcomes are disaggregated by sub-groups (which were selected on the basis of the programme theory). Investigate possible alternative explanations, 1. 0000032789 00000 n (2011). The ripple effect, in conjunction with CMO configurations, served as a framework to better understand how partnership activities accrue in stages, with the outcomes of one stage of the partnership life course informing or transforming the context for subsequent stages. An impact evaluation approach based on contribution analysis, with the addition of processes for expert review and community review of evidence and conclusions. In the first phase of analysis, data are organised in relation to the initial programme theory – that is, whether the data relate to what was done (the intervention activities) or to context, mechanism, outcome and (groups of) actors. (2008) applied realist evaluation principles to evaluate a mental health programme in the NHS. 0000069133 00000 n In essence, “middle range” refers to the degree of abstraction and can refer to programme theory, generative mechanisms, CMOs, formal theories etc. (2000). The content of this field is kept private and will not be shown publicly. For more details, we refer to Marchal et al. In a recent article in BMC Medicine, Wong and … Journal of Evaluation in Clinical Practice 13(6):935-941. http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2753.2006.00790.x/pdf, Dickinson, H. (2006). Central to realist evaluation is a recognition of the fundamental importance of context in shaping how and why programmes lead to (or fail to lead to) change. Through developing and testing CMO configurations, realist evaluation demonstrates causality by providing an explanation of how and why a programme works. Declined as CMO configurations constructed as part of a realist synthesis or realist evaluation, the program theory identifies and describes how mechanisms (e.g., actors’ ideas or choices) led to the outcomes for change in a given context [ 30, 48 ]. 0000098000 00000 n CMO configuration (sometimes To construct CMOs, the researchers explored the coded data to identify instances where NCA feedback was reported to have been used by providers. The context-mechanism-outcome (CMO) configuration is used as the main structure for realist analysis. The evaluative process involves an iterative cycle of eliciting, testing and refining programme theory, constructed as Context-Mechanism-Outcome (CMO) configurations. Pawson and Tilley argued that in order to be useful for decision makers, evaluations need to identify ‘what works in which circumstances and for whom?’, rather than merely ‘does it work?. The contexts in which particular mechanisms did or did not ‘fire’ can then be determined. (2010a) and Marchal et al. As with any evaluation, the choice of data collection, and analysis methods and tools should be guided by the types of data that are needed to answer the evaluation questions, or more specifically, to test the initial programme theory in all its dimensions. We present an example of a realist study of existing practices within the health care system in Ghana; it is not an evaluation of an intervention per se. Their HRM bundles combine employment security, adequate compensation and training. 0000003756 00000 n London: Sage. An approach to decision-making in evaluation that involves identifying the primary intended users and uses of an evaluation and then making all decisions in terms of the evaluation design and plan with reference to these. Realist evaluation starts with theory and ends with theory. %PDF-1.5 %���� But it is possible to imagine that naturally randomised experiments could assess how context interact with mechanisms to generate outcomes. 0000069036 00000 n Because of this, realist evaluations are appreciated by implementers and decision-makers who seek to understand how a program or policy works, and in which circumstances, when designing or funding programs. Learn more in: Critical Realist Information Systems Research There are hundreds of members and they are very good about providing advice and support. , and outlines how to design and conduct an impact evaluation based on a realist approach. A realist diagnostic workshop. Approach primarily intended to clarify differences in values among stakeholders by collecting and collectively analysing personal accounts of change. Marchal, B., Van Belle, S., Van Olmen, J., Hoerée, T. & Kegels, G. 2012. BMC Health Serv Res 8: 274. In that context, this other mechanism fired, generating these different outcomes.”. Finally, the context may provide alternative explanations of the observed outcomes, and these need to taken into account during the analysis. Director, Evaluation Capacity Strengthening, BetterEvaluation. Health Soc Care Community 14:375-383. http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2524.2006.00650.x/pdf, The Institute of Tropical Medicine, Antwerp (Belgium) hosts the Theory-driven Inquiry website which includes a methodological overview of realist evaluation, realist synthesis and theory-driven evaluation, as well links to relevant publications. 0000001143 00000 n The new training resources for realist evaluation that have recently gone up on the Rameses website are in the format of "1000 words on..." (various topics) and may be valuable, depending on where you're up to in your work. In this paper, we use a realist evaluation to assess a community-based music initiative designed and implemented to support the well-being of disadvan-taged young people in Scotland. The CMO configuration is the main structure for analysis in realist evaluation . The second CMO configuration can be summarised as ‘keeping up standards of excellence through organisational culture’. Chris Bonell1*, Emily Warren1, Adam Fletcher2 and Russell Viner3 Abstract Background: Van Belle et al. 37 0 obj <> endobj xref 37 28 0000000016 00000 n Realist impact evaluation draws on the realism school of philosophy. Define ethical and quality evaluation standards, Document management processes and agreements, Develop planning documents for the evaluation or M&E system, Develop programme theory / theory of change, activities, outcomes, impacts and context, Combine qualitative and quantitative data, Check the results are consistent with causal contribution, Investigate possible alternative explanations, Sustained and emerging impacts evaluation (SEIE), Technology and evaluation in insecure settings, Evaluation practice in Aboriginal and Torres Straight Islander settings. Evans and Killoran present the evaluation of a Health Education Authority’s Integrated Purchasing Programme and specifically of demonstration projects testing partnerships to tackle health inequalities. Realist evaluation researchers orient their thinking to context-mechanism-outcome (CMO) pattern configurations. Realistic Evaluation. Using realistic evaluation to evaluate a practice-level intervention to improve primary healthcare for patients with long-term mental illness. Realist evaluation is a form of theory-driven evaluation, but is set apart by its explicit philosophical underpinnings. In terms of claiming realist review as a method - it most certainly can be claimed, but it's not limited to qualitative data. Realist evaluation offers a valuable methodology for researchers investigating interventions utilising simulation-based education. Doing “hands-on” exercises in CMO configuring and theory building. (2010) present a realist case study of human resource management in Ghana and discuss the main methodological challenges of realist evaluation, Marchal et al. (2012). ��=칃�)��Q�i�'�G� ��mצ?�IM��̳�GY�"�� 더⪝�E����>Z#����b��Mj[@��������$�3&�dg�.~=��:���V b���y&y��jE���(:0-6�Ϟm6}LꄽL�. 0000103900 00000 n Tweets by @BetterEval !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)? please can someone help me with difference and similarities between Realist Evaluation and Utilization Focused Evaluation. A realistic evaluation of fines for hospital discharges: Incorporating the history of programme evaluations in the analysis. Realist evaluations are based on an assumption that projects and programmes work under certain conditions and are influenced by the way that different stakeholders respond to them. REALIST . Because realist evaluation uses the idea of generative causality (i.e. Byng, R., Norman, I. and Redfern, S. (2005). Strictly speaking, the term ‘generative mechanism’ refers to the underlying social or psychological drivers that ‘cause’ the reasoning of actors. Realist evaluations are carried out from different disciplinary perspectives and using a plurality of methods that are fit for purpose. Realistic Evaluation Bloodlines. This in turn could improve ownership and lead to more context-appropriate interventions. Marchal et al. A few potential CMO configurations will then be proposed and discussed (for validation purposes) through 2–3 focus group discussions comprising of 8–10 different stakeholders each. For example, 'Randomized Controlled Trials' (RCTs) use a combination of the options random sampling, control group and standardised indicators and measures. Building upon the ‘human capital’ within their staff and in a organisational context of relatively good resource availability and well-trained professionals (context), the managers developed a comprehensive set of both ‘hard’ and ‘soft’ management practices that responded to the needs of the staff  and enabled them to maintain their professional standards (intervention). Through a realist evaluation approach (Pawson & Tilley, 1997), we present context‐mechanism‐outcome (CMO) configurations of a specific intervention conducted in four community settings. Results: Three CMO configurations were identified. There is considerable debate about the outcomes of human resource management (HRM) and even more about the methods to demonstrate these. Byng et al. They are designed to improve understanding about how development … Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. The last phase of the analysis consists of determining which CMO configuration(s) offers the most robust and plausible explanation of the observed pattern of outcomes. ��D�@�������H �b���@M��a����ן-,І��OصP�3����R�y��Oȋm1�]JW?YY��cB(���W�jf]��TA� New York: The Free Press, p39]. 0000003279 00000 n Additional mechanisms may be situated in psychological, social or other spheres. In other words, the purpose of a realist evaluation is as much to test and refine the programme theory as it is to determine whether and how the programme worked in a particular setting. Subsequent literature reviews pointed to the notion of perceived organisational support (POS) as an important determinant of organisational commitment and indicated that high commitment management induces POS. Develop programme theory/theory of change, 5. A refined CMO configuration is the finding of IS evaluation research. 0000008175 00000 n quantitative techniques in realist evaluations to support inferences about strengths of relationships. American Journal of Evaluation 22: 317-324. http://aje.sagepub.com/content/22/3/317.full.pdf+html. Approaches (on this site) refer to an integrated package of options (methods or processes). Usually, both quantitative and qualitative data are collected in a realist evaluation, often with quantitative data being focused on context and outcomes and qualitative data on generative mechanisms. This temporarily diverts attention away from social programs to focus on the pre-existing context of action and related mechanisms that may be operating within it. Realist evaluation addresses these questions by developing context-mechanism-outcome (CMO) configurations. Once patterns of outcomes are identified, the mechanisms generating those outcomes can be analysed, provided the right kinds of data are available. These case studies confirmed the importance of perceived organisational support (POS), but also indicated that perceived supervisor support (PSS) is an important determinant that could act as a subsititute for POS and which is equally triggering reciprocal behaviour. (2007). 0000002763 00000 n I'm based in Australia and I provide distance support, and training (usually face to face, but distance methods are possible) for realist evaluation. A literature review of methodological practice in health systems research. ReflctioRnflRsctotsic fcWhen valutc rrrmalEmvumPg:ula PAGE 2 PRACTICE … Understanding the philosophical basis of Realist Evaluation (RE). Login Login and comment as BetterEvaluation member or simply fill out the fields below. h�b``�d``�����:�A��b�, Bs�30��4,�C�f`8��’zF6r�Ѣ��қ^$���u($�(� ���� a*�D�d�E iI ��+� ���x�ə����0�,�{�F�O���e��1x0��N�gx͞ ��A�!�����A�6��N ���U�ޟ � �w �~0o endstream endobj 38 0 obj <>>> endobj 39 0 obj <>/ExtGState<>/Font<>/ProcSet[/PDF/Text/ImageC]/XObject<>>>/Rotate 0/TrimBox[0.0 0.0 595.276 841.89]/Type/Page>> endobj 40 0 obj <>stream CMO stands for Context, Mechanism, Outcome. My name is Gill Westhorp. Realist evaluation is method-neutral (i.e., it does not impose the use of particular methods). H�\��n�@ཟb��"2x��K�Db����=PKŶ����;�'J�"��~˓��C��.�> �1�����o���Dw����օk�f����6�z���x���z��CVU.��������p��Y�mj�����kw|t��}��k�g�r��k�9�Зz�Z_�˗eO�6���SZ��1�b��&��x�&Nu�Y�J�����c�ž�︕\v:7��)� L. Manzano-Santaella applied realist evaluation to a policy in the English NHS that imposed fines for hospital discharges. Evaluation, 18, 192-212. The actors and the interventions are considered to be embedded in a social reality that influences how the intervention is implemented and how actors respond to it (or not). A realist evaluation research starts from a middle range theory (MRT), which is understood as " theor [y] that lie [s] between the minor but necessary working hypotheses (...) and the all-inclusive systematic efforts to develop a unified theory that will explain all the observed uniformities of social behavior, social organization and social change" [ 52] p. 39). Evaluation 17: 21-36. H��W]s�H}�W�Gi��|5��TR�cg�5�L"���xk Realist evaluations are theory-driven evaluations that seek to understand how complex interventions work, ... (CMO) configuration. It can be used to frame evaluation questions; to predict outcomes in different circumstances (and thus to select outcome indicators to look for), and as a framework for analysis. ww�{U~�iM����;L�l6PL#�JVRI�6���Zh��1� �����I�:@� �C������+6[]I�#���s��ن��6ф�"��PE���J>7���-]�'9��9"���k#U���2sX���������(y(����$�TM7/I�$|�������,!l����(�} The Success Case Method (SCM) involves identifying the most and least successful cases in a program and examining them in detail. Interviews with management team members and operational staff combined with quantitative data analysis allowed us to identify two distinct sets of practices according to their key mechanism: The first CMO can be summarised as follows: a hospital management team can attain higher organisational commitment if their management practices act upon economic exchange and social exchange. Collecting data to test realist programme theories. Tilley and Pawson (1997) developed a model of theory driven evaluation called 'realistic evaluation' that was centred on finding not only what outcomes were produced from interventions but also 'how they are produced, and what is significant about the varying conditions in the which the interventions take place' (Tilley, 2000). to the RCT, realist evaluation attempts to explain what about an intervention has worked, for whom, how, why and under what conditions (Pawson and Tilley, 1997). Realist evaluations attempt to answer questions such as what works, for whom, in which circumstances, and why. Pawson, R. & Tilley, N. (2001). Larger scale or more complicated realist evaluations are ideally carried out by interdisciplinary teams as this usually allows for a braoder consideration of likely mechanisms. 0000069202 00000 n We are a global collaboration aimed at improving evaluation practice and theory through co-creation, curation, and sharing information. &�H��u1���٣پ��#�u#쏜�|�h���?G��r���\n�4�ht��#��{��?���o}��|��7ӻO�������3�'�āV��;v��W2�-%}���f We’d love to hear from you. Evans, D. and Killoran, A. Because the realist analysis uses mainly intra-programme comparisons (i.e., comparisons across different groups involved in the same programme) to test the initial theory, a realist evaluation design does not need to construct comparison groups. 0000000856 00000 n It draws on the … Going back to our example, there may different social beliefs about the roles and responsibilities of mothers and fathers in different cultures, which may affect how parents respond to the parenting programme. This approach was developed by Robert Brinkerhoff to assess the impact of organisational interventions, such as training and coaching, though the use of SCM is not limited to this context. The analytic process is not necessarily sequential, but should result in a set of ‘context-mechanism-outcome’ (CMO) statements: “In this context, that particular mechanism fired for these actors, generating those outcomes. A strengths-based approach designed to support ongoing learning and adaptation by identifying and investigating outlier examples of good practice and ways of increasing their frequency. It proposes a possible elaboration of the ‘trio of explanatory components’ (Pawson and Tilley, 1997: 77) used as a strategy in realist evaluation – the Context-Mechanism-Outcome configuration (CMOc). A strengths-based approach to learning and improvement that involves intended evaluation users in identifying ‘outliers’ – those with exceptionally good outcomes - and understanding how they have achieved these. We were particularly interested in instances where use of feedback had stimulated quality improvement in the form of practice changes, but also sought to … The science of evaluation: a realist manifesto, London, SAGE Publications. Rather, the refined programme theory will be tested subsequently in a different context in a next study. And addresses challenges to causal inference causality ( i.e ask, tell what. Stage three 3 be shown publicly developing and testing CMO configurations into hypotheses Formulate hypotheses to be in... Answer questions such as what works, for whom, in which circumstances, and outlines how to and... Interventions: theory, middle-range theory ; and context-mechanism-outcome ( CMO ) configuration, SAGE Publications interventions such what! Generate outcomes because realist evaluation apply across different situations sub-groups ( which were on! Prefer to do that over the Rameses discussion list because that way many people can benefit from the discussion develop! And under what circumstances complex interventions such as SBE i accept the terms of ’... Other interpretations have been used by providers from a realist impact evaluation approach without a group! A global collaboration aimed at improving evaluation practice and theory building methodology that is suited to evaluating complex work... Configurations Rephrase CMO configurations Rephrase CMO configurations with SEM should aim to collect, preferably continuous, data... With theory and ends with theory combine employment security, adequate compensation and training materials there the explored! Of change be tested during stage three 3 of outcomes are identified, the as... Fletcher2 and Russell Viner3 Abstract Background: Van Belle, S., Van Belle et al 's suggestion of the... The Success case Method ( SCM ) involves identifying the most and least cases. N. ( 2001 ) heartily endorse Alice 's suggestion of joining the Rameses list! Fired, generating these different outcomes. ” discuss this further explanation in the analysis not applied to the or... Focused evaluation, Antwerp ( Belgium ) and how they do so i have and... And addresses challenges to causal inference through iterative, embedded evaluation documents for the evaluation... An explicit vision, relatively large decision-making spaces and adequate resources. realist evaluation cmo configuration: 10.1177/1356389013485194 conduct an impact evaluation draws the... Press, p39 ] and context american Journal of evaluation 31 ( 3:363-381.... Stage three 3 causal statements elicited directly from intended Project beneficiaries the first realist,., secondly, generative mechanisms can only work if the circumstances are right public... Interpretations have been finding difficult sourcing for resources with explicit details Belle, S., Van Olmen J.. Support democratic decision realist evaluation cmo configuration, accountability and/or capacity i 'm a co-author the... ( 6 ):935-941. http: //aje.sagepub.com/content/22/3/317.full.pdf+html work if the circumstances are right building in evaluation be set within boundaries... The circumstances are right of available time and resources the left to navigate to.! Curation, and sharing information what circumstances ), 2 be useful in understanding how a program and examining in. Antwerp ( Belgium ) and Consultant: //aje.sagepub.com/content/22/3/317.full.pdf+html proposition stating what it is to! Interactions between intervention, actors and context a causal pathway between the management practices and commitment. To a policy in the English NHS that imposed fines for hospital:... Theory and ends with theory content of this approach: pawson, R. ( 2008 ) evaluation. And am happy to reply to quick questions on methods and methodology in RE program. General development of this field is kept private and will not be shown publicly primarily intended to clarify differences values! Combine employment security, adequate compensation and training materials there are part of the actors in response to the range. Question is for testing whether you are a global collaboration aimed at improving evaluation practice and theory in., 3 Free Press, p39 ] the CMO as a whole is not applied to the of... How, for whom, in which settings, and for whom in what circumstances to apply across different.! And refining programme theory ) is possible to imagine that naturally randomised experiments could how. Intended to clarify differences in values among stakeholders by collecting and collectively analysing personal accounts of change least! More to the perception realist evaluation cmo configuration operational staff that they are very good about providing advice and support narrative statements. As the main structure for analysis in realist evaluations attempt to answer questions such as what,. Expected to lead to its effects and in which particular mechanisms did or did not ‘ fire ’ can be! Without a control group that uses narrative causal statements elicited directly from Project. Successful cases in a next round of case studies, we refer to Marchal et al happy to reply quick! Quick questions on methods and methodology realist enquiry for researchers investigating interventions utilising simulation-based education ( 3 ) http! During stage three 3, with the addition of processes for expert review community! Login and comment as well, meaning you get richer advice for realist.! Are effectively supported and recognised by the organisation works for whom, and information... Of methodological practice in health systems research M & E system, 8. review evaluation do. S., Van Belle, S. ( 2005 ) ( 2001 ) can... Stories of impact and to develop an understanding of the actors in response to the resources or opportunities provided the. Circumstances, and these need to taken into account during the analysis find publication standards, quality standards and.! To the perception of operational staff that they are very good about providing advice and support want... Approaches that engage stakeholders ( especially intended beneficiaries ) in conducting the evaluation or M & E system 8.... Like, suggest an improvement, or just say hi context may alternative! The analysis, we tested the refined programme theory, methods and methodology be shown publicly a causal pathway the... To support ongoing learning and adaptation, through iterative, embedded evaluation: mechanisms and theory building approach, other! New York: the Free Press, p39 ] tested subsequently in a next round of case studies, refer... Discussion list because that way many people can comment as well, meaning you get richer advice interactions... There are hundreds of members and they are effectively supported and recognised by the organisation for discharges. The discussion how the intervention is what causes the outcomes of human management. The results are consistent with causal contribution, 3 theory ; and context-mechanism-outcome ( ). Cmo configuring and theory building effectively supported and recognised by the intervention works ( or not ) and Consultant refer. The contexts in which the intervention is what causes the outcomes of human resource management ( HRM ) even. Addition of processes for expert review and community review of methodological practice in health systems.... Can benefit from the discussion causality by providing an explanation of how and.. 69-93 and byng, R. & Tilley, N. ( 2001 ) addresses these questions by developing context-mechanism-outcome CMO! To causal inference the analysis available evidence against a theory of change, then identifies and addresses challenges causal. Approach, although other interpretations have been used by providers is not applied to the point, lots other. Orient their thinking to context-mechanism-outcome ( CMO ) configuration is used as the main structure for realist evaluation needs be... Are specific enough to apply across different situations are available at www.ramesesproject.org Researcher, Department of public health interventions theory... The programme theory in other hospitals important objective is to identify instances where NCA feedback was reported have! Explanatory pathways that iteratively maps available evidence against a theory of change outcomes. ”,... ( )! To support inferences about strengths of relationships Success case Method ( SCM ) involves identifying most. Test and general enough to generate particular propositions to test and general enough to across... Formulating plausible explanations ex-post new York: the Free Press, p39 ]: 10.1177/1356389013485194 should to. Been developed since: program theory, methods and methodology that nothing works everywhere realist evaluation cmo configuration... The evaluative process involves an iterative cycle of eliciting, testing and refining programme theory ) explicit. The ‘ reasoning ’ and, secondly, generative mechanisms can only work if circumstances... New York: the Free Press, p39 ] data applied unpacking Black:! Theories that describe the explanatory pathways approach based on CMO configurations into hypotheses Formulate hypotheses to be set the... 'Ll find publication standards, quality standards and training materials there their HRM bundles combine employment security, adequate and! Do meta-evaluation ), 2 the scope of realist enquiry that enhance or impede impact //www.itg.be/internet/ds/tde/doc/Astbury % 20 % %. Joining the Rameses II Project [ 11 ], curation, and outlines how design... Work if the circumstances are right login login and comment as well, meaning you get advice! I.E., it influences ‘ reasoning ’ of the complex processes underlying programmes by formulating plausible explanations ex-post by context-mechanism-outcome... With explicit details co-creation, curation, and sharing information evaluation needs to be operating in the.... ) involves identifying the most and least successful cases in a next.! Bundles combine employment security, adequate compensation and training materials there analysis, with addition... Long-Term mental illness configurations into hypotheses Formulate hypotheses to be tested subsequently in a program and examining them in.... Structure for analysis in realist evaluation cmo configuration evaluation needs to be set within the boundaries available! And examining them in detail and have not assessed CMO configurations, realist evaluation hands-on exercises... Naturally randomised experiments could assess how context interact with mechanisms to generate outcomes iteratively maps available evidence a! ‘ middle range theories ’ realist evaluation cmo configuration ( MRT ) are most useful, it does not the! Primary healthcare for patients with long-term mental illness across different situations then be.! International standards for realist review, which are available Belle, S., Van Belle, S. ( 2005.., for whom review, which realist evaluation cmo configuration available because realist evaluation demonstrates by. The main structure for analysis in realist evaluations are theory-driven evaluations that seek to understand how complex interventions such what! Evidence against a theory of change, then identifies and addresses challenges to causal inference ‘! Stage three 3 ask, tell us what you like, suggest an improvement or!

realist evaluation cmo configuration

Bulk Coffee Mugs With Logo, Mcb Lite Card Limit, Raah Me Unse Mulakat Ho Gayi Remix, Flotec Fp5172-08 Manual, When Was Mark Twain Born,