Abstract
CONTEXT: The long-term sustainability of whole-system change programs is rarely studied, and when it is, it is inevitably undertaken in a shifting context, thereby raising epistemological and methodological questions. This article describes a transferable methodology that was developed to guide the evaluation of a three-year follow-up of a large health care change program in London, which took place during a period of economic turbulence and rapid policy change. METHOD: Using a mixed-method organizational case study design, we studied three services (stroke, kidney, and sexual health) across primary and secondary care. Each had received £5 million (US$7.8 million) in modernization funding in 2004. In 2010/2011, we gathered data on the services and compared them with data from 2004 to 2008. The new data set contained quantitative statistics (access, process, and outcome metrics), qualitative interviews with staff and patients, documents, and field notes. Our data analysis was informed by two complementary models of sustainability: intervention-focused (guided by the question, What, if anything, of the original program has been sustained?) and system-dynamic (guided by the question, How and why did change unfold as it did in this complex system?). FINDINGS: Some but not all services introduced in the original transformation effort of 2004-2008 were still running; others had ceased or been altered substantially to accommodate contextual changes (e.g., in case mix, commissioning priorities, or national policies). Key cultural changes (e.g., quality improvement, patient centeredness) largely persisted, and innovative ideas and practices had spread elsewhere. To draw causal links between the original program and current activities and outcomes, it was necessary to weave a narrative thread with multiple intervening influences. In particular, against a background of continuous change in the local health system, the sustainability of the original vision and capacity for quality improvement was strongly influenced by (1) stakeholders' conflicting and changing interpretations of the targeted health need; (2) changes in how the quality cycle was implemented and monitored; and (3) conflicts in stakeholders' values and what each stood to gain or lose. CONCLUSIONS: The sustainability of whole-system change embodies a tension between the persistence of past practice and the adaptation to a changing context. Although the intervention-focused question, What has persisted from the original program? (addressed via a conventional logic model), may be appropriate, evaluators should qualify their findings by also considering the system-dynamic question, What has changed, and why? (addressed by producing a meaningful narrative).