MOJ ISSN: 2379-6383MOJPH

Public Health
Research Article
Volume 2 Issue 1 - 2015
Developing Comprehensive Health Promotion Evaluations: A Methodological Review
Kozica SL1, Lombard CB1, Hider K1,2, Harrison CL1 and Teede HJ1,3*
1Monash Centre for Health Research and Implementation (MCHRI), Monash University, Australia
2Evaluation Assist, Australia
3Diabetes and Vascular Medicine Unit, Monash Health, Australia
Received:November 18, 2014| Published: December 30, 2014
*Corresponding author: Helena J Teede, Monash Centre for Health Research and Implementation (MCHRI), School of Public Health and Preventive Medicine, Monash University, Locked Bag 29, Monash Medical Centre, Clayton, Victoria, 3168, Australia, Tel: +61 3 9594 7500; Fax: +61 3 9594 7554; Email:@
Citation: Kozica SL, Lombard CB, Hider K, Harrison CL, Teede HJ (2015) Developing Comprehensive Health Promotion Evaluations: A Methodological Review. MOJ Public Health 2(1): 00007. DOI: 10.15406/mojph.2015.02.00007

Abstract

Background: Evaluation has an integral role in effective health promotion, yet few large scale health promotion interventions prospectively incorporate rigorous evaluation into their planning processes. To promote greater use of evaluation we reviewed the health promotion literature incorporating evaluations and have produced a succinct summary.
Methods: MEDLINE and SCOPUS databases were accessed to identify health promotion publications, which utilised formal evaluation. Publication were then summarised and reviewed informing on evaluation methods relevant to health promotion.
Results: Numerous health promotion evaluation levels and approaches exist some of which include objective-based, needs-based, collaborative and utilization focused. Multiple evaluation frameworks are available including the RE-AIM, Centre for Disease Control and Prevention (CDC) and Medical Research Council framework (MRC). Data collection methods often include both qualitative and quantitative methods and validated tools are limited but important. Evaluation approaches and frameworks have been applied effectively to diverse health programs with useful learnings.
Evaluation enablers include:
I. Prospective planning
II. Tailoring the evaluation to the study context
III. Engaging with evaluation target audience early
IV. Ample time to undertake the evaluation
Conclusion: Evaluation of health promotion interventions is imperative to deliver benefits of research into improved health outcomes. Understanding of evaluation concepts including levels, approaches and framework and methods is needed to facilitate consistent use of evaluation in research. Ultimately, health promotion programs require context specific adaptable evaluations. Greater opportunities exist for shared learnings to build evaluation capacity and to deliver greater health impacts.
Keywords: Evaluation; Health promotion; Evaluating health programs; Evaluation frameworks; Translational research

Abbreviations

CDC: Centre for Disease Control and Prevention; MRC: Medical Research Council Framework; RE-AIM: Reach, Efficacy, Adoption, Implementation and Maintenance; CIPP: Context, Input, Process and Product; NHMRC: National Health and Medical Research Council

Introduction

As the global prevalence of obesity and chronic diseases continues to rise, the need for effective health promotion programs is imperative. Whilst research into effectiveness of health promotion programs is needed to improve population health outcomes, translation of these research findings into policy and practice is crucial. Translation requires not only efficacy data around what to implement, but also information on how to implement it. Evaluation seeks to optimise translation by answering questions related to how to implement evidence-based interventions under real world conditions [1]. Evaluation studies the implementation strategies and adds value to efficacy outcomes [2], potentially informing on program reach, fidelity and sustainability. Hence evaluation is now recognised as an integral component of all health promotion programs [2]. Ultimately, comprehensive evaluation provides essential knowledge about program implementation practices and processes, which is vital to the translation of programs into varied setting and population contexts [3,4].
Presently, evaluation has been applied inconsistently to health promotion programs [5], limiting the translation of knowledge. Health promotion programs have further been criticised for insufficient evaluation planning [6] and scrutinised as “fail (ing) to contribute to their own quality enhancement” [5]. Barriers to evaluation include funding constraints, lack of knowledge, skills and familiarity with evaluation methodologies and poor availability of measurement tools. Information related to evaluation is often difficult to access through conventional academic literature and is more often located in non-academic grey literature. Additionally, terminology in this field is inconsistent and there is considerable overlap between areas such as implementation research and evaluation.
In this context, we aimed to review and summarise the literature and discuss planning and development of comprehensive health promotion evaluations. This is relevant to health professionals, researchers and end users, who seek insight into evaluation designs and are engaged in driving evidence into policy and practice. An enhanced appreciation of evaluation methodologies and terminology aims to provide a foundation for those new to evaluation.

Methods

MEDLINE and SCOPUS databases were used to identify publications related to the evaluations of health promotion interventions with no publications date restrictions. Techniques such as “snowballing” otherwise referred to as reference tracking were also utilised. This method involves scanning reference lists of all full text papers and using judgment to decide whether to pursue texts further [7]. A systematic review was not conducted, as it was essential to locate literature outside databases containing published journals (text books, government and non for profit websites), as this is a regular repository for evaluation literature. Publications were included and reviewed if it clearly described an evaluation planning process or methodology and had applied these methods to a health promotion intervention. Publications were excluded if it did not include a formal evaluation methodology inclusive of evaluation levels, approaches and frameworks within the context of health promotion programs.

Results

Evaluation Planning and Purpose
Literature suggests that in depth planning is critical to comprehensive evaluation. The first step in evaluation planning is to identify the purpose for undertaking the evaluation and formulating clear evaluation objectives and questions. Broadly, health promotion evaluation purposes and key questions will commonly focus on program effectiveness and/or program implementation. Evaluation is not an absolute science [8] and a program evaluation wills unlikely represent all perspectives and components of an intervention, resulting in a “balancing act” or “trade-off” between priorities [9]. Gathering information on a large number of issues may result in the evaluation lacking depth to draw a confident conclusion [10]. Deciding which aspects to evaluate will need to be informed by relevant health literature, resources allocated, stakeholder needs and determined based on the primary targets audience of the evaluation purpose of the health program [11], mandating the importance of early stakeholder engagement. Stakeholders may include funders, end users, service providers, government employees or the general public. Importantly, providing ample time to plan and conduct an evaluation is imperative. For an overview of the common phases of evaluation planning and implementation please refer to figure 1 based on the work of Øvretveit [9] and Van Marris [12].
Evaluation Concepts
Areas for consideration when developing an evaluation plan can include evaluation levels (also referred to as types), approaches, framework (also referred to as models) and data collection tools. We recommend having an understanding of these commonly used evaluations ‘concepts’ and terminology.
Evaluation levels or types: Evaluation ‘levels’ include process, formative, summative, impact and outcome evaluations. The levels are determined by the purpose of the evaluation and are influenced by the state of the program (under development, settled) and the timing of data collection (before program roll out, during implementation or post implementation) [13]. In Table 1,2 we describe evaluation levels (or types) and note that elements can overlap. Common elements assessed in a process evaluation include program reach, fidelity in relation to program protocol, program context, quality and dose delivered and received by participants [14,15]. A community based obesity prevention study in adolescents living in Tonga completed a process evaluation by research staff recording all intervention-related activities, the frequency of the activity, the reach of that activity (how many people were involved in the program) and the resources required [16]. The purpose of a formative evaluation is to improve the quality of performance or the delivery of the program prior to roll-out and is conducted either before or during the program [17]. On the contrary, summative, impact and outcome evaluations typically occur at the completion of an intervention, yet these three evaluation methods differ significantly. In the large scale “Be Active Eat Well” program which promoted healthy lifestyles in disadvantaged communities, impact and outcome evaluation results were assessed as environmental changes, policy implementation and anthropometric data two years post program commencement [18]. When used alone an impact, outcome and summative evaluation may miss the depth of the data available during intervention roll-out. However, process evaluation requires preplanning with stakeholders to enable data collection throughout program delivery. We suggest that combining a range of evaluation levels [E.g. process, summative and outcome evaluation] can strengthen capacity to translate successful health promotion interventions to an array of settings and target groups.
Evaluation approaches: An evaluation approach informs on ‘how’ we will conduct the evaluation i.e. will stakeholders be engaged to participate in the evaluation or will the evaluation be conducted by the researcher only? Each evaluation approach has a number of associated steps, guiding the processes and activities of the evaluation (Table 3). There are a number of evaluation approaches available some of which include objective-based, needs-based, theory based, collaborative, utilization focused and realistic evaluation. Typically there is a large degree of overlap between varied evaluation approaches; however the emphasis and the tasks related to each step of the evaluation varies in accordance with the nature and purpose of the evaluation [19]. Please note this is not an extensive list of approaches, rather we have included approaches that are frequently applied to health promotion evaluations.
Figure 1: Evaluation planning and implementation.

Clinical Research

Clinical research as defined by the National Institute of Health incorporates; 1) patient orientated research (direct human interaction), 2) epidemiological and behavioural studies, 3) outcomes and health service driven research [53].

Evaluation

Evaluation is a comparative assessment and comparison of an intervention of interest, against a standard of acceptability [54], utilising systematically collected data [9]. In order to determine the merit, worth or significance of the activity [55].

Evaluation Research

Is the systematic application of social research procedures for assessing the conceptualisation, design, implementation and utility of social intervention programs [56].

Implementation Research

“Implementation research is the scientific inquiry into questions concerning implementation—the act of carrying an intention into effect, which in health research can be policies, programs, or individual practices (collectively called interventions)” . The intent is to understand what, why, and how interventions work in “real world” settings and to test approaches to improve them [57].

Efficacy Research

Explored the initial impact of the intervention (whether it has done more good than harm) amongst a target population under specific conditions [1].

Translational Research

Describes the notion of moving health knowledge generated into products, practices, policies and can include knowledge exchange, transfer and mobilization [58]. Translatsional reserach is dynamic and iterative process aiming to improve health ouctomes, provide more effective health serevices and products to strengthen the health care system [59]. More simply, translational research is the transfer of basic science discoveries to represent a movement towards the goal of improved health [58].

Scaling-up

‘Scaling up’ is the notion to describe the ambition or process of expanding the exposure of health interventions to maximize population benefits, but can also refer to increasing the financial, human and capital resources needed to increase coverage [60].

Table 1: Glossary of evaluation and implementation research terms.

Process Evaluation

Measures the activities of the program including reach, implementation, satisfaction, quality and capacity of the program. It determines whether a program is delivered as intended to the target audience [8].

Impact Evaluation

Measures the immediate effect of the health intervention [61]. It measures changes in awareness, knowledge, skills, attitudes and behaviours [62].

Outcome Evaluation

Measures the longer term program impacts, inclusive of subsequent effects on health [63].

Formative Evaluation

A combination of measurements are obtained and judgements are made before or during the implementation phase of materials, methods, activities, in order to improve the quality of performance or the delivery of the program [17].

Summative Evaluation

Conducted after completion of a program and draws conclusions regarding the quality, impact, outcomes and benefits of a program [17].

Table 2: Evaluation levels (types).

Evaluation Approach

Description

Previous Application in the Health Promotion Setting

Objective-based

The effectiveness and worth of a program is based purely on whether pre-defined objectives have been successfully achieved [13].

The objectives of the Diet, Exercise and Weight Loss Trial (DEW-IT) were to determine the effectiveness of a healthy lifestyle program to reduce cardiovascular disease risk factors. Whilst, this study was not strictly described as an objective based evaluation, large elements of this study support this type of evaluation, as unintended outcomes were not explored. In this program the predefined end-points were determine as weight, blood pressure, serum lipids and fitness [64].

Needs-based Evaluation

An evaluation in which the determination of worth is identified by the communities wants or needs, which will then be addressed by the program planned [13].

A needs assessment was conducted in patients with advanced cancer to determine the most suitable delivery methods of health information. Participants were asked to complete a survey and indicated that the preferred mode of delivery of information was one-to-one education sessions [65].

Theory based/logic Development

The evaluation is based on program theory and the logical relationship between program inputs and outcomes. This approach involves having a good appreciation of the nature of the program, context and environment [66].

The Being Active Eat Well program was developed to address childhood obesity by promoting healthy lifestyles. In this programme the intervention activities (inputs) focused on capacity building, policy development and community empowerment. As a result of the program inputs the children in the intervention group had significantly lower increases in weight and BMI scores [67].

Collaborative/

Participatory Evaluation

This approach compels evaluators to include program staff, consumers and/or community members in the evaluation design and conduct of the evaluation. Stakeholders are involved in the evaluative endeavour, including interpreting and conclusions [43].

To determine the success of a community based rehabilitation program in individuals with disabilities, a participatory evaluation was conducted. Program participants, staff and managers of the program were engaged in the evaluation process by part-taking in interviews and focus groups to explore satisfaction of the service. Stakeholders reported that the program had supported the needs of the community [68].

Utilization Based

The approach is formulated on the premise of who the primary intended users and stakeholders are and how the results will be employed. Utilization-focused evaluation is highly individualised, flexible and situational [69].

A utilization focused evaluation was conducted to assess the role and effectiveness of nurse practitioners in an acute hospital. The evaluation focused on stakeholder’s satisfaction with the nurse practitioners and provided valuable information for other organizations interested in introducing nurse practitioners into their health care system [70].

Realistic Evaluation

This theory-driven evaluation focus on the context in which a program is implemented and describes the mechanism responsible for outcomes achieved. The findings highlight ‘what works for whom in a set of given circumstances’, suggesting that it is not possible to universally generalize cause-and-effect relationships.

A realistic evaluation approach was employed to determine the effectiveness of a theory driven school based program, designed to improve children’s social, emotional and physical health. The mixed-methods evaluation explored the processes used during program implementation to provide contextual information and establish relationships between program inputs and outcomes [71].

 

 

Table 3: Evaluation approaches and previous applications within the health promotion setting.
Evaluation frameworks or models: Similarly, evaluation frameworks provide detailed guidance for evaluators, as they ensure the evaluation design considers the origins and contexts of the program being examined. An evaluation framework can encourage the prioritisation of evaluation purpose and the selection of data collection tools [20]. Moreover, evaluations are program specific as there is no “one size fits all” evaluation design in evaluation [19]. Evaluators will often utilise more than one evaluation level, approach and framework in order to meet the needs and purpose of their evaluation.
Evaluation Frameworks
From the literature, the commonly applied evaluation frameworks used in health promotion research include the RE-AIM, Context, Input, Process and Product (CIPP), Predisposing, Reinforcing, Enabling, Constructs in Educational/Ecological Diagnosis and Evaluation (PROCEED, PRECEDE), Centre for Disease and Control and Prevention (CDC), Medical Research Council (MRC) and programme logic model. Here we describe briefly these evaluation frameworks, prior application and practical insights from our experiences of utilising these frameworks.
RE-AIM framework: The RE-AIM (Reach, efficacy, adoption, implementation and maintenance) framework is an adaptable and simple framework for evaluating large scale projects, which considers both the individual and population impacts of the program [21]. RE-AMI focuses on informing and optimising translation and has been applied numerous times to health promotion programs for example to a diabetes self-management programs [22] and school based physical activity programs [23]. In our experience this framework is useful because it is adaptable, easy to use and supports the development of a focused evaluation. The associated resources provide clear examples of evaluation questions, potential measurement indicators and data collection tools which are helpful for novice evaluators [24]. However, it is worth noting that within this framework ‘maintenance’ is measured at two years post program delivery; therefore consideration of evaluation timelines is essential. In comparison to other frameworks the RE-AIM is less flexible and the relationship between the five domains of the RE-AIM framework remains unclear.
CIPP framework: The Context, Input, Process and Product (CIPP) framework for evaluation is comprehensive and useful for guiding formative and summative evaluations. The context of the program refers to the evaluation environment in which the program will be implemented. The input refers to the resources and activities required to meet the objectives of the program. The final stage of the CIPP model is the product, which refers to the judgment of the effectiveness and success of the program [6,25]. This framework has been applied to health promotion programs previously and examples include a diabetes management program [26] and a behavioural intervention targeting reduction in metabolic syndrome [27]. The CIPP framework is highly flexible and adaptable and specifically useful for evaluations seeking insight into a range of factors that influence program effectiveness such as environmental and contextual factors. However, this framework provides less guidance for evaluation planning, including developing evaluation objectives and key questions, compared to the RE-AIM framework.
PROCEED/PRECEDE framework: The PROCEED/PRECEDE is a theoretically grounded and comprehensive framework, combining program planning, implementation and evaluation. It is based on the premise that a thorough social, educational and ecological assessment (Precede) should take place prior to program development and that evaluation is required to assess effectiveness (Proceed). Phase include:
i. Social assessment
ii. Epidemiological, environmental and behavioural intervention
iii. Educational and ecological assessment
iv. A. Intervention alignment
B. Administrative and policy assessment
v. Implementation
vi. Process evaluation
vii. Impact and outcome evaluation [28]
The framework has been applied to multiple health promotion programs including mental health programs [29]. The PROCEED/PRECEDE models is a structured and valuable framework for multi-strategy program evaluations and considers multiple determinants of health. However, this framework is complex and has been criticised for being a very medically based and requires significant time to prospectively plan and undertake the evaluation [19].
Centre for disease and control and prevention (organisational framework): The Centre for Disease and Control and Prevention (CDC) framework suggests that effective program evaluation is a systematic way to improve public health actions by conducting evaluations that ensure useful, feasible, ethical and accurate results. The framework outlines 6 inter-connected steps to planning and delivering an evaluation
I. Engage stakeholder
II. Describe the program
III. Focus the evaluation design
IV. Gather credible evidence
V. Justify conclusions
VI. Share lessons learnt [19]
The framework is highly reputable and has been applied to numerous health promotion program including physical activity programs [30] and tobacco control initiatives [31]. In our experience the CDC framework is very practical and easy to follow and is useful for the evaluation of simple through to complex programs and partnerships. It is particularly valuable for capacity building evaluations as it incorporates multiple stakeholder perspectives into the evaluation. Another advantage of the CDC model is that it is underpinned by four central standards for ensuring ethically sound evaluations (utility, feasibility, propriety and accuracy). However, the framework is broad and may be less useful to novice evaluators, seeking a supportive framework to assist in the development of evaluation objectives and questions. Also this framework may require significant funding and strong evaluation facilitation [19].
Medical research council (organisational framework): The Medical Research Council (MRC) framework is theory based and describes the development, evaluation and implementation. The stages involved include:
I. Development of an intervention,
II. Pilot and feasibility assessment,
III. Evaluation of the intervention,
IV. Implementation, dissemination, monitoring and long term follow up [32].
Examples of previous application include a physical activity program targeting a population at high risk of diabetes [33] and evaluation of a cardiovascular disease management program [34]. The advantages of the MRC framework are that it is highly flexible and promotes a non-linear approach to evaluation with many supportive documents to assists evaluation planning and implementation. However, the framework is broad and does not provide depth and detail around developing evaluation objectives and key questions. Similarly to the CDC framework, the MRC framework may be less useful to novice evaluators, who may prefer a more structured and supportive framework.
Evaluation Methodological Approaches
Logic model/programme theory methodological approach: A logic model is a methodological approach to program planning and evaluation, describing the assumptions and resources needed to support the activities of the program of interest [35]. A logic model typically identifies the background of the program, the resources (inputs), the activities (outputs), immediate impacts and short and long term outcomes [36]. Logic models have been applied to diverse health programs including the evaluation of community nutrition education programs [37] and a school based programs aimed at reduce the frequencies of teenage pregnancies [38]. The value of logic models is well established in health promotion and resources available to develop logic models [39]. Logic models provide a practical means to assess the relationship between program inputs and anticipated outcomes. However, a caution is that logic models may not capture all important aspects of programs and associated policies. Logic models are also time consuming requiring prospective planning to consider the underlying program theory and relationships between processes and outcomes [19].
Overall
Ultimately, evaluation methods selected will vary according to the purpose, timing, resources available, stakeholders involved in the evaluation and their prior experience conducting evaluations. Combining a range of evaluation approaches and frameworks is commonly accepted in the literature, enabling evaluators to account for potential methodological limitations. Structured evaluation frameworks may not be appropriate for all evaluations contexts, emphasising the importance of rigorous evaluation planning and the need for context specific adaptation of evaluation frameworks. However, the importance of stakeholder engagement as an integral component of evaluation conduct is recognised across the diverse evaluation frameworks, emphasising the importance of communication and engagement of stakeholders to yield useful evaluation results [40]. In planning a health promotion program, resources ideally need to be allocated for the evaluation. However, funding bodies commonly fund research but less commonly fund evaluation and advocacy is needed to address this. Providing ample funding for major evaluation strategies would likely shift evaluations from being perceived as a “minor after thought” to a major integral component of a health promotion program [5].
Data Collection Methods and Tools
A summary of the advantages and disadvantages of quantitative and qualitative data collection methods are outlined in Table 4. The emphasis of qualitative research is depth, detail and individual perceptions [10]. Quantitative research methods use standardised data collection tools to fit diverse opinions and experiences into often predetermined categories [8]. Most evaluators recognise the synergistic benefits of mixed methods, which incorporate quantitative and qualitative research [41]. Data “triangulation” provides an opportunity to integrate data in mixed-methods studies. Data triangulation protocols commonly involve initially analysing qualitative and quantitative data separately and then comparing the findings from each method, to determine whether they support or contradict each other [42]. The purpose of data triangulation is to enhance the reliability and confidence of research findings. A practical example of data triangulation in the health promotion setting could involve comparing anthropometric measurements with results from a physical activity questionnaire and focus groups (qualitative), following a specific health promotion intervention [14,43].

 

Methods

Advantages

Disadvantages

Qualitative

  • Focus groups
  •  In-depth interviews
  • Structured interviews
  • Open-ended survey
  •  Journals
  •  Participant observations
  • Case-studies
  •  Narrative evaluation
  •  Photograph [72]
  • Permits depth and detail [10]
  • Opportunity to study the meaning and experiences
  • Data collection process is not restrained by predetermined categories of analysis
  • Useful in determining unintentional effects of a health intervention [8].
  • Time consuming and laborious
  • Costly
  • Smaller number of cases
  • High quality analysis depends on the skills and integrity of the researcher [73]

Quantitative

  • Questionnaires
  • Measurements
  • Tracking templates
  • Reports
  • Can be less time consuming
  • Typically increased sample size
  • Facilities comparison among subjects and statistical rigorous of data
  • Less detailed information
  • Less sensitive
Table 4: Qualitative and quantitative evaluation methods advantages and disadvantages.
Existing validated data tools are preferable for large pre- and post-intervention surveys, as they provide validity and confidence that any change identified is likely to reflect a meaningful result. Yet, it is important to only employ validated tools if they are useful for the evaluation [44]. Currently there are limited reliable and validated evaluation tools available [14], with a clear need to develop program specific tools. However, limited large scale studies are beginning to share evaluation tools on websites for public use and descriptions of process evaluation objectives and associated data collections tools employed are increasingly being published in peer reviewed journals. This has occurred in interventions in mental health [45], weight management [16] and physical activity [46]. There are additionally some evaluation guides available online (grey literature) that incorporate evaluation templates as well as sample questions for semi-structured interviews and focus groups [47,48]. There are also guides available to describe the optimal methods of reporting evaluation outcomes, specific to the health promotion context [49]. Ultimately, data collection tools frequently require adaptation to accommodate and capture within health program processes, which are unique to an intervention, increasing sensitivity and usefulness of results [8]. Combining a range of data collection methods can partly address the inherent weaknesses of each method. Determining evaluation objectives upfront is integral to ensure that data gathering methods yield useful and appropriate information [9]. In future, increased availability of validated evaluation tools will improve evaluation standards and enable better comparability between studies [14].

Discussion

Developing a robust evaluation is challenging due to the lack of guidance in conventional published literature and the wide range of evaluation concepts across evaluation level, approach and framework. There are five available evaluation levels which include formative, process, impact, outcome and summative evaluation. Multiple evaluation approaches are available some of which include objective-based, needs based, collaborative, realistic and utilization focused evaluations. In addition to evaluation levels and approaches, numerous health promotion evaluation frameworks are available such as RE-AIM, CDC, MRC, logic model, CIPP and PREECE/PROCEDE. These evaluation components have been applied effectively to diverse health promotion programs and target populations. Commonly, evaluators will combine and use various aspects of evaluation levels, approaches and frameworks to meet their evaluation purposes. We highlight, that each evaluation will require planning and context specific adaptation to produce useful and credible results [50].
Moving forward, funders as key stakeholders are increasingly seeking knowledge about how to translate evidence into practice. Increasingly, researchers are likely to be held accountable for optimising opportunities for translating research into practice or at least for ensuring that the evidence generated during their research also informs and enables translation [51]. Subsequently, the importance of up skilling our workforce to build greater evaluation capacity has been established [52]. Evaluation is a vital step to inform translation; consequently researchers face the challenge of deciding which evaluation level, approach and framework to apply to drive translation. A structured evaluation approach has some practical limitations such as inherently being both resource and labour intensive [14]. Presently, most evaluations involve a combination of program specific questionnaires (developed and/or validated tools) and qualitative research methods, in order to ensure a robust evaluation design [14,43]. An evaluation consultant can provide guidance for making major decisions such as selecting the evaluation level, appropriate framework and data collection methods.
Broad scale evaluations of health promotion interventions are lacking and use of evaluation terminology is ambiguous and inconsistent in the literature. Many health promotion interventions may show efficacy but commonly do not have comprehensive integrated prospective evaluations, limiting the ability to translate outcomes and deliver return on research investments [5]. This is further confounded by difficulties in accessing published evaluation literature through accessible conventional academic avenues. However, utilising existing levels, approaches and framework creates a greater opportunity to critique and compare evaluations across different health interventions. Ultimately, standardisation of evaluation reporting would enhance the field across a diverse array of health settings. Opportunities for shared learnings, avoidance of repeated mistakes and efficient allocation of resources also exist in this field. Future directions require clear definition of terms in, evaluation, as well as further development of well tested evaluation tools. Improved sharing of evaluation processes, planning and tools will assist novice evaluators to more easily incorporate comprehensive evaluation methods into their research plans and will improve translation of evidence into policy and practice.

Conclusion

Comprehensive within program evaluations is integral to enable translation and scale-up of effective health promotion programs. As health promotion interventions vary significantly, the authors do not suggest a “one size fits all” in evaluation, as each evaluation needs to be tailored to the context and circumstance to ensure results provide credible and useful information. Evaluation concepts including level, approach, framework along with data collections methods and tools all require consideration. Enablers include prospective planning, providing adequate time to develop the evaluation, defining key objectives and target audiences for the evaluation and ensuring the evaluation is practical and within resources allocated. Sound evaluation focuses on the importance of real world impacts and attempts to bridge the gap between research, clinical practice and policy. Arguable evaluation should not be optional, but rather an obligatory in future health promotion research and the summary provided here will assist health promotion researchers to integrate evaluation into future health promotion programs.

Acknowledgements

This work was supported by the National Health and Medical Research Council (NHMRC). Samantha Kozica is an NHMRC postgraduate scholar, Cheryce Harrison is a National Heart Foundation Postdoctoral Research Fellow (100168) and Helena Teede is a NHMRC Practitioner Fellow.

References

  1. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL (2008) A glossary for dissemination and implementation research in health. J Public Health Manag Pract 14(2): 117-123.
  2. Kumanyika S, Jeffery R, Morabia A, Ritenbaugh C, Antipatis V, et al. (2002) Obesity prevention: the case for action. Int J Obes Relat Metab Disord 26(3): 425-436.
  3. Oldenburg B, Absetz P (2011) Lost in translation: overcoming the barriers to global implementation and exchange of behavioral medicine evidence. Transl Behav Med 1(2): 252-255.
  4. Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F (2005) Implementation research: A synthesis of the literature: National Implementation Research Network.
  5. Swinburn B, Bell C, King L, Magarey A, O Brien K, et al. (2007) Obesity prevention programs demand high-quality evaluations. Aust N Z J Public Health 31(4): 305-307.
  6. Fleming ML, Parker E (2007) Health promotion: Principles and practice in the Australian context. Allen & Urwin, New South Wales, Australia.
  7. Greenhalgh T, Peacock R (2005) Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 331(7524): 1064-1065.
  8. Hawe P, Degeling D, Hall J (1990) Evaluating health promotion: MacLennan & Petty, Sydney.
  9. Ovretveit J (1997) Evaluating health interventions: An introduction to evaluation of health treatments, services, policies, and organizational interventions. Open University Press.
  10. Patton MQ (1987) How to use qualitative methods in evaluation. Sage Publications, USA.
  11. Rossi, Lipsey, Freeman (2004) Evaluation: A systematic approach. (7th edn), Sage Publications, USA.
  12. Marris BV, King B (2007) Evaluating Health Promotion Programs Version 3.6: The Health Communication Unit, Centre for Health Promotion, University of Toronto.
  13. Owen JM (2006) Program evaluation: Forms and approaches. Allen & Unwin, Crows Nest, N.S.W Australia.
  14. Nutbeam D (1998) Evaluating health promotion-progress, problems and solutions. Health Promot Int 13(1): 27-44.
  15. Saunders RP, Evans MH, Joshi P (2005) Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 6(2): 134-47.
  16. Fotu KF, Moodie MM, Mavoa HM, Pomana S, Schultz JT, et al. (2011) Process evaluation of a community-based adolescent obesity prevention project in Tonga. BMC public health 11: 284.
  17. Green LW, Lewis FM (1986) Measurement and evaluation in health education and health promotion. Mayfield Publishing Company, Palo Alto, California.
  18. de Silva-Sanigorski AM, Bolton K, Haby M, Kremer P, Gibbs L, et al. (2010) Scaling up community-based obesity prevention in Australia: background and evaluation design of the Health Promoting Communities: Being Active Eating Well initiative. BMC public health 10: 65.
  19. Kahan B (2008) Review of Evaluation Frameworks. Saskatchewn Minstry of Education.
  20. Fink A (2004) Evaluation fundamentals: Insights into the outcomes, effectiveness, and quality of health programs. (2nd edn), Sage Publications, USA.
  21. Glasgow RE, McKay HG, Piette JD, Reynolds KD (2001) The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns 44(2): 119-27.
  22. Glasgow RE, Nelson CC, Strycker LA, King DK (2006) Using RE-AIM Metrics to Evaluate Diabetes Self-Management Support Interventions. Am J Prev Med 30(1): 67-73.
  23. De Meij JS, Chinapaw MJ, Kremers SP, Wal MFVD, Jurg ME, et al. (2010) Promoting physical activity in children: The stepwise development of the primary school-based JUMP-in intervention applying the RE-AIM evaluation framework. Br J Sports Med 44(12): 879-87.
  24. McKenzie R, Naccarella L, Stewart A, Thompson C (2007) Targeting what matters in health promotion evaluation. Evaluation Journal of Australasia 7(1): 19-26.
  25. Stufflebeam LD (2005) Cipp Model (Context, Input, Process, Product). Encyclopedia of Evaluation. Sage Publications, Thousand Oaks, USA.
  26. Clinton J, McNeill R, Irvine R, Silva M, Brown P (2009) An evaluation of the Gardening for Health and Sustainability initiative: an initiative supported by Let's Beat Diabetes. Counties Manukau District Health Board 1-44.
  27. Intarakamhang U, Patcharee D (2012) Effects of health behavioral modification program on metabolic diseases in risk Thai clients. Asian Biomedicine 6: 2.
  28. Green LW, Kreuter KW (1998) Health promotion planning: An educational and Ecological Approach. (2nd edn), Mayfield Publishing Company, Mountain View, USA.
  29. Wright A, McGorry PD, Harris MG, Jorm AF, Pennell K (2006) Development and evaluation of a youth mental health community awareness campaign–The Compass Strategy. BMC public health 6: 215.
  30. Gamez R, Parra D, Pratt M, Schmid TL (2006) Muevete Bogota: promoting physical activity with a network of partner companies. Promot Educ 13(2): 138-143.
  31. MacDonald G, Starr G, Schooley M, Yee SL, Klimowski K, et al. (2001) Introduction to program evaluation for comprehensive tobacco control programs. Centers for Disease Control and Prevention, Atlanta, GA, USA.
  32. Medical Research Council (2000) A framework for development and evaluation of RCTs for complex interventions to improve health. Medical Research Council, UK.
  33. Hardeman W, Sutton S, Griffin S, Johnston M, White A, et al. (2005) A causal modelling approach to the development of theory-based behaviour change programmes for trial evaluation. Health Educ Res 20(6): 676-687.
  34. Robinson L, Francis J, James P, Tindle N, Greenwell K, et al. (2005) Caring for carers of people with stroke: developing a complex intervention following the Medical Research Council framework. Clin Rehabil 19(5): 560-71.
  35. Cooksy LJ, Gill P, Kelly PA (2001) The program logic model as an integrative framework for a multimethod evaluation. Evaluation and Program Planning 24(2): 119-28.
  36. Helitzer D, Hollis C, de Hernandez BU, Sanders M, Roybal S, et al. (2010) Evaluation for community-based programs: the integration of logic models and factor analysis. Eval Program Plann 33(3): 223-233.
  37. Medeiros LC, Butkus SN, Chipman H, Cox RH, Jones L, et al. (2005) A logic model framework for community nutrition education. J Nutr Educ Behav 37(4): 197-202.
  38. Hulton LJ (2007) An evaluation of a school-based teenage pregnancy prevention program using a logic model framework. J Sch Nurs 23(2): 104-110.
  39. Renger R, Titcomb A (2002) A three-step approach to teaching logic models. American Journal of Evaluation 23(4): 493-503.
  40. Johnson K, Greenseid LO, Toal SA, King JA, Lawrenz F, et al. (2009) Research on Evaluation Use A Review of the Empirical Literature From 1986 to 2005. American Journal of Evaluation 30(3): 377-410.
  41. Creswell JW, Clark VLP (2007) Designing and conducting mixed methods research: Australian and New Zealand Journal of Public Health 31(4): 388.
  42. Davidson EJ (2005) Evaluation methodology basics: The nuts and bolts of sound evaluation. Sage Publications, USA.
  43. Hider K Frequently Asked Questions about Evaluation. In: SatCfC (eds.), Southern Health C-CCH, Effectiveness. Melbourne.
  44. Nakkash RT, Alaouie H, Haddad P, El Hajj T, Salem H, et al. (2012) Process evaluation of a community-based mental health promotion intervention for refugee children. Health Educ Res 27(4): 595-607.
  45. Jenkinson KA, Naughton G, Benson AC (2012) The GLAMA (Girls! Lead! Achieve! Mentor! Activate!) physical activity and peer leadership intervention pilot project: a process evaluation using the RE-AIM framework. BMC public health 12: 55.
  46. Zarinposh F (2006) Project Evaluation Guide For Nonprofit Organizations: Fundamental Methods and Steps For Conducting Project Evaluation. Canada Volunteer Initiative, Canada.
  47. de Groot F, Robertson N, Swinburn B, de Silva-Sanigorski A (2010) Increasing community capacity to prevent childhood obesity: challenges, lessons learned and results from the Romp & Chomp intervention. BMC public health 10(1): 522.
  48. Moskowitz JM (1989) Preliminary guidelines for reporting outcome evaluation studies of health promotion and disease prevention programs. New directions for program evaluation 1989(43): 101-112.
  49. Greene JC, Lipsey MW, Schwandt TA, Smith NL, Tharp RG (2007) Method choice: Five discussant commentaries. New Directions for Evaluation 2007(113): 111-127.
  50. Adams J, Dickinson P (2010) Evaluation training to build capability in the community and public health workforce. American journal of evaluation 31(3): 421-433.
  51. Suarez-Balcazar Y, Taylor-Ritzler T (2013) Moving From Science to Practice in Evaluation Capacity Building. American Journal of Evaluation.
  52. Rubio DM, Schoenbaum EE, Lee LS, Schteingart DE, Marantz PR, et al. (2010) Defining translational research: implications for training. Acad Med 85(3): 470-475.
  53. Green LW, Kreuter MW (1999) Health Promotion Planning; an Educational and Ecological Approach. (3rd edn), Mayfield Publishing Company, Mountain View, USA.
  54. Scriven M (2007) Key evaluation checklist. Evaluation Checklists Project.
  55. Rossi PH, Freeman HE, Lipsey MW (1993) Evaluation: A systematic approach. (6th edn), SAGE Publications, USA.
  56. Peters DH, Adam T, Alonge O, Agyepong IA, Tran N (2013) Implementation research: what it is and how to do it. BMJ 347: f6753.
  57. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, et al. (2006) Lost in knowledge translation: time for a map? J Contin Educ Health Prof 26(1): 13-24.
  58. Straus SE, Tetroe J, Graham I (2009) Defining knowledge translation. CMAJ 181(3-4): 165-168.
  59. Mangham LJ, Hanson K (2010) Scaling up in international health: what are the key issues? Health Policy Plan 25(2): 85-96.
  60. Slatter T (2003) Measuring healt promotion impacts: A guide to impact evaluation in integrated health promotion. Victorian Government Department of Human Services, Melbourne, Victoria.
  61. McKenzie JF, Neiger B, Smeltzer J (2005) Planning, implementing, and evaluating health promotion programs: a primer. Pearson/Benjamin Cummings, San Francisco.
  62. Naidoo J, Wills J (1994) Health promotion: foundations for practice. Bailliere Tindall, UK.
  63. Miller ER, Erlinger TP, Young DR, Jehn M, Charleston J, et al. (2002) Results of the diet, exercise, and weight loss intervention trial (DEW-IT). Hypertension 40(5): 612-618.
  64. Wong RK, Franssen E, Szumacher E, Connolly R, Evans M, et al. (2002) What do patients living with advanced cancer and their carers want to know?-a needs assessment. Support Care Cancer 10(5): 408-415.
  65. Donaldson SI, Gooler LE (2003) Theory-driven evaluation in action: Lessons from a $20 million statewide work and health initiative. Evaluation and Program Planning 26(4): 355-366.
  66. Sanigorski A, Bell A, Kremer P, Cuttler R, Swinburn B (2008) Reducing unhealthy weight gain in children through community capacity-building: results of a quasi-experimental intervention program, Be Active Eat Well. Int J Obes 32(7): 1060-1097.
  67. Sharma M, Deepak S (2001) A participatory evaluation of community-based rehabilitation programme in North Central Vietnam. Disabil Rehabil 23(8): 352-358.
  68. Patton MQ (1996) Utilization-Focused Evaluation: The New Century Text. (3rd edn), SAGE publications, London, United Kingdom.
  69. Stetler CB, Effken J, Frigon L, Tiernan C, Zwingman-Bagley C (1998) Utilization-focused evaluation of acute care nurse practitioner role. Outcomes Manag Nurs Pract 2(4): 152-160.
  70. Pommier J, Guevel MR, Jourdan D (2010) Evaluation of health promotion in schools: a realistic evaluation approach using mixed methods. BMC public health 10(1): 43.
  71. Russ-Eft D, Preskill H (2001) Evaluation in organizations: A systematic approach to enhancing learning, performance, and change: Basic Books, Arizona.
  72. Pope C, Ziebland S, Mays N (2000) Analysing qualitative data. BMJ 320(7227): 114-116.
© 2014-2016 MedCrave Group, All rights reserved. No part of this content may be reproduced or transmitted in any form or by any means as per the standard guidelines of fair use.
Creative Commons License Open Access by MedCrave Group is licensed under a Creative Commons Attribution 4.0 International License.
Based on a work at http://medcraveonline.com
Best viewed in Mozilla Firefox | Google Chrome | Above IE 7.0 version | Opera |Privacy Policy