From your readings, it is suggested that we as health providers, and, for the purposes of this class health promoters, must ask ourselves why we do what we do. Philosophically, this question can be answered in terms of why we have a desire to help people live better lives. Pragmatically, as per Dr. Martin, it has to do with why we choose the type of promotion programming that we do. Consider in this assignment being able to articulate WHY you would want to take on certain health promotion or disease prevention projects. Ideally, the key is to take on a project that is justifiable within your community versus taking on a project that you personally think is important regardless of what the community or targeted population perceives or really needs. As outlined in the readings, it is easy to place energy into an activity that is somewhat successful but that ultimately misses a larger, more important goal.
For example, a community program that teaches bicycle safety would seem like a worthy undertaking. However, using information learned in the course, a needs assessment (evidence-based literature review) could show that bicycle injuries were very low in the target community while drowning rates were very high. Thus, efforts aimed at reducing bicycle injuries would be noble but would not be the best allocation of talent and resources.
Therefore, the first assignment of this semester is for you to perform a PRELIMINARY NEEDS ASSESSMENT by performing an evidence-based literature review. Once you identify a need, then you can start creating a solution. This will specifically, later on, assist you in explaining and justifying what the needs are in your target population. Use the following process to PREPARE to do your needs assessment.
*Review Healthy People 2030 documents and the objectives from 2030 Topics and Objectives (Objectives A–Z) Link:
https://health.gov/healthypeople/objectives-and-data/browse-objectives.
(Links to an external site
.)
(Links to an external siteAlso Table 1-4 (pg. 27) (Course required textbook).*
REQUIRED COMPONENTS for Assignment #1: THE PRELIMINARY NEEDS ASSESSMENT (PLEASE READ CAREFULLY)
The assignment should be typed out on a word document in APA format. Each paragraph should consist of 5-6 complete sentences. The assignment should consist of 1,500- to 3,000-words. This assignment must be completed and submitted on a Microsoft word document.
APA format is required. 12 font size should be used, font names that should be used are Times New Roman or Cambria (not both), normal margins (no more than 1 inch), double spaced, indentation, etc. Please include a title page that consists of your name (First and Last), Panther ID number, and title of the article (APA format).In addition, include a reference page that consists of three (3) references from creditable resources.
Requirements:
1. Document your targeted Objective and your Health Behavior Change Focus. State your chosen health behavior focus and targeted objective from HP 2030 and the Leading Health Indicator it relates to that you intend to address in your health education and program.
2. Document the need for a program related to your targeted objective. Discuss the rationale or need for your choice to design a program to meet the targeted HP 2030 objective by reporting on the following: a) Report on the documented health risk(s) associated with not meeting this objective. b) State ALL of the HEALTH RISKS AND HEALTH CONDITIONS and DISEASES that are associated with the objective’s corresponding maladaptive health behavior. This is to be done by doing an evidence-based literature review using appropriate sources of secondary data.
Examples of appropriate sources include official statistical databases like National Center for Health Statistics, technical reports, scholarly journal articles, and literature review articles. A practical example of this would be to perform a literature review that provides documentation of all pertinent health risks and conditions and diseases that are associated with one of the Healthy People Objective’s that deal with tobacco use, and that describes each of these health risks and diseases (lung cancer, chronic obstructive pulmonary disease).
3. Report on the incidence and prevalence rates of the health risks and the health conditions and diseases that are associated with your chosen Healthy People 2030 Health Behavior. Report on any pertinent morbidity rates or mortality rates that are directly linked to your chosen Healthy People 2030 health behavior. Report any temporal trends of incidence and prevalence rates of the health risks and health conditions and health diseases that are directly linked to your chosen Healthy People 2030 health behavior. Locate research data and statistics from scholarly literature, trustworthy websites, and statistical evidence. You must provide incidence and prevalence rates on a national level. Examples of appropriate sources of data include official statistical databases like the census database at
www.census.gov (Links to an external site.)
, technical reports, scholarly journals, and literature review articles). An example of this would be to expound upon the prevalence AND incidence rates AND TRENDS of INCIDENCE AND PREVALENCE RATES of lung cancer and chronic obstructive pulmonary disease for one of the Healthy People 2030/2020 Tobacco Use Objectives at the national level morbidity and mortality rates associated with lung cancer and chronic obstructive pulmonary disease.
4. Report what the research and literature suggest to minimize the risk that is associated with not performing your chosen health behavior change and thus, meet the Healthy People 2030 objective. This is also to be done by doing an evidence-based literature review using appropriate sources. Examples of appropriate data sources include evidence base search engines, journals, and resources such as Cochrane, Medline, Trip Database, and any other evidence-driven journals or resources. An example would be to search the literature for “evidence-based” methods and programs that have been most effective to accomplish the desired behavior change (like increased physical activity).
FOURTH EDITION
Health Program
Planning and
Evaluation
A Practical, Systematic Approach
for Community Health
L. Michele Issel, PhD, RN
Professor of PhD Program
University of North Carolina at Charlotte
College of Health and Human Services
Charlotte, North Carolina
Rebecca Wells, PhD, MHSA
Professor
The University of Texas
School of Public Health
Houston, Texas
World Headquarters
Jones & Bartlett Learning
5 Wall Street
Burlington, MA 01803
978-443-5000
info@jblearning.com
www.jblearning.com
Jones & Bartlett Learning books and products are available through most bookstores and online booksellers. To contact Jones & Bartlett
Learning directly, call 800-832-0034, fax 978-443-8000, or visit our website, www.jblearning.com.
Substantial discounts on bulk quantities of Jones & Bartlett Learning publications are available to corporations, professional
associations, and other qualified organizations. For details and specific discount information, contact the special sales department at
Jones & Bartlett Learning via the above contact information or send an email to specialsales@jblearning.com.
Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning Company
All rights reserved. No part of the material protected by this copyright may be reproduced or utilized in any form, electronic or mechanical,
including photocopying, recording, or by any information storage and retrieval system, without written permission from the copyright owner.
The content, statements, views, and opinions herein are the sole expression of the respective authors and not that of Jones & Bartlett
Learning, LLC. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or
otherwise does not constitute or imply its endorsement or recommendation by Jones & Bartlett Learning, LLC and such reference shall not
be used for advertising or product endorsement purposes. All trademarks displayed are the trademarks of the parties noted herein. Health
Program Planning and Evaluation: A Practical, Systematic Approach for Community Health, Fourth Edition is an independent publication and
has not been authorized, sponsored, or otherwise approved by the owners of the trademarks or service marks referenced in this product.
There may be images in this book that feature models; these models do not necessarily endorse, represent, or participate in the activities
represented in the images. Any screenshots in this product are for educational and instructive purposes only. Any individuals and scenarios
featured in the case studies throughout this product may be real or fictitious, but are used for instructional purposes only.
This publication is designed to provide accurate and authoritative information in regard to the Subject Matter covered. It is sold with the
understanding that the publisher is not engaged in rendering legal, accounting, or other professional service. If legal advice or other expert
assistance is required, the service of a competent professional person should be sought.
15842-7
Production Credits
VP, Executive Publisher: David D. Cella
Publisher: Michael Brown
Associate Editor: Danielle Bessette
Vendor Manager: Nora Menzi
Senior Marketing Manager: Sophie Fleck Teague
Manufacturing and Inventory Control Supervisor: Amy Bacus
Composition and Project Management: S4Carlisle Publishing
Services
Cover Design: Scott Moden
Director of Rights & Media: Joanna Gallant
Rights & Media Specialist: Merideth Tumasz
Media Development Editor: Shannon Sheehan
Cover Image: © Lynne Nicholson/Shutterstock
Printing and Binding: Edwards Brothers Malloy
Cover Printing: Edwards Brothers Malloy
Library of Congress Cataloging-in-Publication Data
Names: Issel, L. Michele, author. | Wells, Rebecca, 1966- author.
Title: Health program planning and evaluation: a practical, systematic
approach for community health/L. Michele Issel and Rebecca Wells.
Description: Fourth edition. | Burlington, MA: Jones & Bartlett Learning,
[2018] | Includes bibliographical references and index.
Identifiers: LCCN 2017010386 | ISBN 9781284112115 (pbk.)
Subjects: | MESH: Community Health Services—organization & administration |
Program Development—methods | Health Planning—methods | Program
Evaluation—methods | United States
Classification: LCC RA394.9 | NLM WA 546 AA1 | DDC 362.12068—dc23 LC record available at https://lccn.loc.gov/2017010386
6048
Printed in the United States of America
21 20 19 18 17 10 9 8 7 6 5 4 3 2 1
© Lynne Nicholson/Shutterstock
Contents
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Introduction to the Types of Evaluation. . . . . . . . . . 19
List of Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Mandated and Voluntary Evaluations. . . . . . . 20
List of Exhibits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
When Not to Evaluate. . . . . . . . . . . . . . . . . . . . . . 21
Preface to the Fourth Edition. . . . . . . . . . . . . . . . . . . xix
Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . xxv
List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvii
SECTION I
The Context of Health
Program Development
The Public Health Pyramid. . . . . . . . . . . . . . . . . . . . . . . 21
Use of the Public Health Pyramid in
Program Planning and Evaluation. . . . . . . . 23
The Public Health Pyramid as
an Ecological Model . . . . . . . . . . . . . . . . . . . . . 23
The Town of Layetteville in Bowe County. . . . . . . . .25
1
Chapter 1 Context of Health
Program Development
and Evaluation. . . . . . . . . . . . . . . 3
History and Context. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Concept of Health. . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Discussion Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Chapter 2 Relevance of Diversity
and Disparities to Health
Programs . . . . . . . . . . . . . . . . . . 29
Health Programs, Projects, and Services. . . . . . 4
Health Disparities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
History of Health Program Planning
and Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Diversity and Health Disparities. . . . . . . . . . . . . 32
Evaluation as a Profession. . . . . . . . . . . . . . . . . . . . . . . . . 8
Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Who Does Planning and Evaluations?. . . . . . . 10
Interventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Roles of Evaluators. . . . . . . . . . . . . . . . . . . . . . . . . . 10
Planning and Evaluation Cycle. . . . . . . . . . . . . . . . . . . 11
Influences of Sociocultural Diversity
on Interventions. . . . . . . . . . . . . . . . . . . . . . . . . 38
Interdependent and Cyclic Nature
of Planning and Evaluation. . . . . . . . . . . . . . . 11
Influences of Biological Diversity
on Interventions. . . . . . . . . . . . . . . . . . . . . . . . . 39
Using Evaluation Results as the
Cyclical Link. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Approaches to Developing Programs. . . . . . . 39
Program Life Cycle. . . . . . . . . . . . . . . . . . . . . . . . . . 13
The Three Health Provider Sectors . . . . . . . . . . 43
The Fuzzy Aspects of Planning. . . . . . . . . . . . . . . . . . . 14
Paradoxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Diversity Within Healthcare Organizations
and Programs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Organizational Culture. . . . . . . . . . . . . . . . . . . . . . 44
Uncertainty, Ambiguity, Risk,
and Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Cultural Competency Continuum. . . . . . . . . . . 44
Diversity and Health Programs. . . . . . . . . . . . . . 33
Profession and Provider Diversity. . . . . . . . . . . .40
Enhancing Cultural Competency . . . . . . . . . . . 48
iii
iv
Contents
Stakeholders and Coalitions . . . . . . . . . . . . . . . . . . . . . 50
Types of Assessments. . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Organizational Assessment. . . . . . . . . . . . . . . . . 75
Discussion Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Marketing Assessment. . . . . . . . . . . . . . . . . . . . . . 76
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Needs Assessment . . . . . . . . . . . . . . . . . . . . . . . . . 76
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Community Health Assessment. . . . . . . . . . . . . 77
Workforce Assessment. . . . . . . . . . . . . . . . . . . . . . 77
SECTION II Defining the Health
Problem
57
Steps in Planning and Conducting
the Assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Form and Develop the Team. . . . . . . . . . . . . . . . 78
Create a Vision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Chapter 3 Community Health Assessment
for Program Planning. . . . . . . . 59
Involve Community Members. . . . . . . . . . . . . . 79
Defining Community. . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Define the Problem to Be Assessed. . . . . . . . . 81
Community as Context and Intended
Recipient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Defining Terms: Based, Focused,
and Driven. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Make a Decision. . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Types of Needs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Types of Strengths. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Anticipate Data-Related and Methodological
Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Approaches to Planning. . . . . . . . . . . . . . . . . . . . . . . . . 64
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Incremental Approach. . . . . . . . . . . . . . . . . . . . . . 64
Discussion Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Apolitical Approach . . . . . . . . . . . . . . . . . . . . . . . . 66
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Advocacy Approach. . . . . . . . . . . . . . . . . . . . . . . . 66
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Communication Action Approach. . . . . . . . . . 67
Define the Population. . . . . . . . . . . . . . . . . . . . . . 80
Prioritize. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Implement and Continue. . . . . . . . . . . . . . . . . . . 83
Summary of Approaches. . . . . . . . . . . . . . . . . . . 69
Chapter 4 Characterizing and
Defining the Health
Problem . . . . . . . . . . . . . . . . . . . 91
Models for Planning Public Health Programs. . . . . 69
Collecting Data From Multiple Sources. . . . . . . . . . . 91
Mobilizing for Action through Planning
and Partnership (MAPP). . . . . . . . . . . . . . . . . . 70
Public Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Community Health Improvement
Process (CHIP). . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Observational Data. . . . . . . . . . . . . . . . . . . . . . . . . 92
Comprehensive Rational Approach. . . . . . . . . 67
Strategic Planning Approach . . . . . . . . . . . . . . . 68
Protocol for Assessing Community
Excellence in Environmental Health
(PACE-EH). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Primary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Archival Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Proprietary Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Published Literature. . . . . . . . . . . . . . . . . . . . . . . . 93
In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Data Beyond Street Lamp. . . . . . . . . . . . . . . . . . . 93
Perspectives on Assessment. . . . . . . . . . . . . . . . . . . . . 71
Collecting Descriptive Data. . . . . . . . . . . . . . . . . . . . . . 94
Epidemiological Perspective. . . . . . . . . . . . . . . . 72
Magnitude of the Problem. . . . . . . . . . . . . . . . . . 94
Public Health Perspective. . . . . . . . . . . . . . . . . . . 74
Dynamics Leading to the Problem. . . . . . . . . . 94
Social Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Population Characteristics. . . . . . . . . . . . . . . . . . 96
Asset Perspective. . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Attitudes and Behaviors . . . . . . . . . . . . . . . . . . . . 96
Rapid Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Years of Life and Quality of Life. . . . . . . . . . . . . . 96
Contents
v
Statistics for Describing Health
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Path to Program Outcomes
and Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Descriptive Statistics. . . . . . . . . . . . . . . . . . . . . . . 100
Components of the Effect Theory. . . . . . . . . . 135
Geographic Information Systems:
Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Matching Levels: Audience, Cause,
Intervention, and Effects. . . . . . . . . . . . . . . . 137
Small Numbers and Small Areas . . . . . . . . . . . 101
Generating the Effect Theory . . . . . . . . . . . . . . . . . . . 138
Epidemiology Rates . . . . . . . . . . . . . . . . . . . . . . . 102
Involve Key Stakeholders. . . . . . . . . . . . . . . . . . 138
Stating the Health Problem. . . . . . . . . . . . . . . . . . . . . 102
Draw Upon the Scientific Literature. . . . . . . . 138
Diagramming the Health Problem. . . . . . . . . 102
Diagram the Causal Chain
of Events. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Writing a Causal Theory of the Health
Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Check Against Assumptions. . . . . . . . . . . . . . . 141
Prioritizing Health Problems . . . . . . . . . . . . . . . . . . . . 110
Functions of Program Theory. . . . . . . . . . . . . . . . . . . 141
Nominal Group Technique. . . . . . . . . . . . . . . . . 111
Provide Guidance. . . . . . . . . . . . . . . . . . . . . . . . . . 141
Basic Priority Rating System. . . . . . . . . . . . . . . . 111
Enable Explanations. . . . . . . . . . . . . . . . . . . . . . . 142
Propriety, Economics, Acceptability,
Resources, and Legality (PEARL)
Component. . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Form a Basis for Communication. . . . . . . . . . . 142
Prioritizing Based on Importance
and Changeability . . . . . . . . . . . . . . . . . . . . . . 114
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Discussion Questions and Activities. . . . . . . . . . . . . 117
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
SECTION III
Health Program
Development
and Planning
Make a Scientific Contribution. . . . . . . . . . . . . 143
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Discussion Questions and Activities. . . . . . . . . . . . . 144
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Chapter 6 Program Objectives
and Setting Targets . . . . . . . . 147
Program Goals and Objectives. . . . . . . . . . . . . . . . . . 147
Goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Foci of Objectives. . . . . . . . . . . . . . . . . . . . . . . . . 148
121
Objectives and Indicators. . . . . . . . . . . . . . . . . . 151
Good Goals and Objectives. . . . . . . . . . . . . . . . 154
Chapter 5 Program Theory
and Interventions
Revealed. . . . . . . . . . . . . . . . . . 123
Program Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Process Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Effect Theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Interventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Using Data to Set Target Values. . . . . . . . . . . . . . . . . 156
Decisional Framework for Setting
Target Values. . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Stratification and Object Target
Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Use of Logic Statements to Develop
Targets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Finding and Identifying Interventions. . . . . . 126
Options for Calculating Target
Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Types of Interventions. . . . . . . . . . . . . . . . . . . . . 127
Caveats to the Goal-Oriented Approach . . . . . . . . 170
Specifying Intervention Administration
and Dosage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Interventions and Program Components. . . . 130
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Characteristics of Good Interventions. . . . . . 131
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Discussion Questions and Activities. . . . . . . . . . . . . 171
vi
Contents
SECTION IV Implementing and
Monitoring the Health
Program
173
Chapter 7 Process Theory for Program
Implementation . . . . . . . . . . . 175
Budgeting as Part of Planning. . . . . . . . . . . . . . . . . . . 204
Monetize and Compute Program Costs. . . . . 204
Budget for Start-Up and Evaluation Costs. . . 205
Break-Even Analysis. . . . . . . . . . . . . . . . . . . . . . . . 205
Budget Justification . . . . . . . . . . . . . . . . . . . . . . . 207
Budget as a Monitoring Tool. . . . . . . . . . . . . . . . . . . . 209
Budget Variance. . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Organizational Plan Inputs. . . . . . . . . . . . . . . . . . . . . . 175
Types of Cost Analyses. . . . . . . . . . . . . . . . . . . . . 209
Human Resources. . . . . . . . . . . . . . . . . . . . . . . . . 177
Information Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Physical Resources. . . . . . . . . . . . . . . . . . . . . . . . . 179
Health Informatics Terminology. . . . . . . . . . . . 214
Transportation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Information Systems Considerations. . . . . . . 214
Informational Resources. . . . . . . . . . . . . . . . . . . 180
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Discussion Questions and Activities. . . . . . . . . . . . . 217
Managerial Resources . . . . . . . . . . . . . . . . . . . . . 180
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Fiscal Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . 182
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Organizational Plan Outputs. . . . . . . . . . . . . . . . . . . . 182
Organizational Chart. . . . . . . . . . . . . . . . . . . . . . 184
Chapter 9 Implementation Evaluation:
Measuring Inputs
and Outputs. . . . . . . . . . . . . . . 219
Information System. . . . . . . . . . . . . . . . . . . . . . . . 185
Assessing the Implementation. . . . . . . . . . . . . . . . . . 219
Inputs to Service Utilization Plan. . . . . . . . . . . . . . . . 185
Implementation Documentation. . . . . . . . . . 220
Social Marketing. . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Implementation Assessment . . . . . . . . . . . . . . 221
Time Line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Operations Manual. . . . . . . . . . . . . . . . . . . . . . . . 182
Eligibility Screening. . . . . . . . . . . . . . . . . . . . . . . . 185
Implementation Evaluation. . . . . . . . . . . . . . . . 221
Queuing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Efficacy, Effectiveness, and Efficiency. . . . . . . . . . . . 222
Intervention Delivery. . . . . . . . . . . . . . . . . . . . . . 189
Data Collection Methods. . . . . . . . . . . . . . . . . . . . . . . 223
Services Utilization Plan Outputs. . . . . . . . . . . . . . . . 191
Quantifying Inputs
to the Organizational Plan. . . . . . . . . . . . . . . . . . . . 223
Summary: Elements of Organizational
and Services Utilization Plans. . . . . . . . . . . . 192
Human Resources. . . . . . . . . . . . . . . . . . . . . . . . . 228
Alternative Plan Formats. . . . . . . . . . . . . . . . . . . . . . . . 192
Physical Resources. . . . . . . . . . . . . . . . . . . . . . . . . 229
Logic Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Quantifying Outputs
of the Organizational Plan. . . . . . . . . . . . . . . . . . . . 230
Business Plans. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Discussion Questions and Activities. . . . . . . . . . . . . 197
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Information Systems. . . . . . . . . . . . . . . . . . . . . . . 230
Monetary Resources. . . . . . . . . . . . . . . . . . . . . . . 230
Quantifying Inputs to the Services
Utilization Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Participants and Recipients. . . . . . . . . . . . . . . . 230
Intervention Delivery and Fidelity. . . . . . . . . . 231
Chapter 8 Monitoring Implementation
Through Budgets and
Information Systems . . . . . . . 201
Quantifying Outputs of the Services
Utilization Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
Budgets and Budgeting . . . . . . . . . . . . . . . . . . . . . . . . 201
Participant-Related Issues. . . . . . . . . . . . . . . . . . 238
Budgeting Terminology. . . . . . . . . . . . . . . . . . . . 202
Program Logistics. . . . . . . . . . . . . . . . . . . . . . . . . . 240
Coverage as Program Reach. . . . . . . . . . . . . . . 234
Contents
vii
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Evaluation and Research. . . . . . . . . . . . . . . . . . . 268
Discussion Questions and Activities. . . . . . . . . . . . . 242
Rigor in Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . 270
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Variables from the Program Effect Theory. . . . . . . 271
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Outcome and Impact Dependent
Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Chapter 10 Program Quality and Fidelity:
Managerial and Contextual
Considerations. . . . . . . . . . . . 245
The Accountability Context. . . . . . . . . . . . . . . . . . . . . 246
Program Accountability. . . . . . . . . . . . . . . . . . . . 246
Professional Accountability. . . . . . . . . . . . . . . . 246
Performance and Quality: Navigating the
Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Quality Improvement Approaches. . . . . . . . . 248
Quality Improvement Tools. . . . . . . . . . . . . . . . 248
Relevance to Health Programs. . . . . . . . . . . . . 251
Performance Measurement. . . . . . . . . . . . . . . . 252
Informatics and Information Technology. . . . 253
Creating Change for Quality and Fidelity. . . . . . . . 255
Interpreting Implementation Data. . . . . . . . . 255
Maintaining Program Process Quality
and Fidelity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Managing Group Processes for Quality
and Fidelity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Causal Factors as Independent
Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Antecedent, Moderating, and
Mediating Factors as Variables. . . . . . . . . . . 273
Measurement Considerations. . . . . . . . . . . . . . . . . . . 275
Units of Observation. . . . . . . . . . . . . . . . . . . . . . . 275
Types of Variables (Levels
of Measurement). . . . . . . . . . . . . . . . . . . . . . . 275
Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
Sensitivity of Measures. . . . . . . . . . . . . . . . . . . . 278
Threats to Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . 279
Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Reliability Concerns. . . . . . . . . . . . . . . . . . . . . . . . 280
Validity of Measures . . . . . . . . . . . . . . . . . . . . . . . 281
Contextual Considerations in Planning
the Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
Evaluation Standards. . . . . . . . . . . . . . . . . . . . . . 281
Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
When and What Not to Change. . . . . . . . . . . . 259
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Formative Evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . 259
Discussion Questions and Activities. . . . . . . . . . . . . 284
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Discussion Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . 260
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Chapter 12 Choosing Designs for
Effect Evaluations. . . . . . . . . 287
SECTION V
Outcome and Impact
Evaluation of Health
Programs
263
Evaluation Design Caveats. . . . . . . . . . . . . . . . . . . . . . 288
Considerations in Choosing
a Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
Using Designs Derived from Multiple
Paradigms: An Example. . . . . . . . . . . . . . . . . 294
Chapter 11 Planning the Intervention
Effect Evaluations. . . . . . . . . 265
Choosing the Evaluation Design. . . . . . . . . . . . . . . . 294
Developing the Evaluation Questions. . . . . . . . . . . 266
Overview of the Decision Tree. . . . . . . . . . . . . 295
Characteristics of the Right Question . . . . . . 267
Designs for Outcome Documentation. . . . . 298
Outcome Documentation, Outcome
Assessment, and Outcome Evaluation. . . 268
Designs for Outcome Assessment:
Establishing Association. . . . . . . . . . . . . . . . . 301
Identifying Design Options. . . . . . . . . . . . . . . . 294
viii
Contents
Designs for Outcome Evaluation:
Establishing Causation. . . . . . . . . . . . . . . . . . 307
Issues with Quantifying Change
from the Program. . . . . . . . . . . . . . . . . . . . . . . 339
Practical Issues with Experimental
Designs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Relationship of Change to Intervention
Effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
Designs and Failures. . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
Clinical and Statistical
Significance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Discussion Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Across Levels of Analysis. . . . . . . . . . . . . . . . . . . . . . . . 343
Statistical Answers to the Questions . . . . . . . . . . . . 345
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Chapter 13 Sampling Designs
and Data Sources for
Effect Evaluations. . . . . . . . . 315
Association. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
Sampling Realities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Sample Construction. . . . . . . . . . . . . . . . . . . . . . . . . . . 317
Hard-to-Reach Populations. . . . . . . . . . . . . . . . 318
Sample Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Calculating Response Rates. . . . . . . . . . . . . . . . 319
Sampling for Effect Evaluations. . . . . . . . . . . . . . . . . 322
Sampling for Outcome Assessment. . . . . . . . 322
Sampling for Outcome Evaluation. . . . . . . . . 324
Data Collection Methods. . . . . . . . . . . . . . . . . . . . . . . 324
Surveys and Questionnaires . . . . . . . . . . . . . . . 325
Comparison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
Prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Four Fallacies of Interpretation. . . . . . . . . . . . . 353
Ecological Fallacy. . . . . . . . . . . . . . . . . . . . . . . . . . 354
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
Discussion Questions and Activities. . . . . . . . . . . . . 356
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Chapter 15 Qualitative Methods
for Planning and
Evaluation . . . . . . . . . . . . . . . 359
Secondary Data. . . . . . . . . . . . . . . . . . . . . . . . . . . 328
Qualitative Methods Throughout the
Planning and Evaluation Cycle . . . . . . . . . . . . . . . 359
Big Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
Qualitative Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
Physical Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
Individual In-Depth Interview. . . . . . . . . . . . . 361
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
Written Open-Ended Questions. . . . . . . . . . . . 362
Discussion Questions and Activities. . . . . . . . . . . . . 330
Focus Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Observation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Data Collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
Case Study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
Chapter 14 Quantitative Data
Analysis and
Interpretation. . . . . . . . . . . . 335
Scientific Rigor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
Data Entry and Management. . . . . . . . . . . . . . . . . . . 335
Analysis of Qualitative Data. . . . . . . . . . . . . . . . . . . . . 372
Outliers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
Overview of Analytic Process . . . . . . . . . . . . . . 372
Linked Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
Sample Description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
Issues to Consider. . . . . . . . . . . . . . . . . . . . . . . . . 374
Thinking About Change. . . . . . . . . . . . . . . . . . . . . . . . 339
Presentation of Findings. . . . . . . . . . . . . . . . . . . . . . . . 375
Change as a Difference Score. . . . . . . . . . . . . . 339
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
Innovative Methods. . . . . . . . . . . . . . . . . . . . . . . 366
Sampling for Qualitative Methods . . . . . . . . . . . . . . 369
Contents
ix
Discussion Questions and Activities. . . . . . . . . . . . . 377
Reporting Responsibly. . . . . . . . . . . . . . . . . . . . . . . . . . 392
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Report Writing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Making Recommendations. . . . . . . . . . . . . . . . 394
Misuse of Evaluations. . . . . . . . . . . . . . . . . . . . . . 397
SECTION VI Additional Considerations
for Evaluators
381
Responsible Contracts. . . . . . . . . . . . . . . . . . . . . . . . . . 398
Chapter 16 Program Evaluators’
Responsibilities. . . . . . . . . . . 383
Responsible for Evaluation Quality. . . . . . . . . . . . . . 400
Ethical Responsibilities. . . . . . . . . . . . . . . . . . . . . . . . . . 383
Ethics and Planning. . . . . . . . . . . . . . . . . . . . . . . 383
Institutional Review Board Approval
and Informed Consent. . . . . . . . . . . . . . . . . . 385
Ethics and Evaluation. . . . . . . . . . . . . . . . . . . . . . 387
Organization–Evaluator Relationship. . . . . . . 398
Health Policy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
Responsible for Dissemination. . . . . . . . . . . . . . . . . . 401
Responsible for Current Practice. . . . . . . . . . . . . . . . 402
Across the Pyramid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Discussion Questions and Activities. . . . . . . . . . . . . 405
Internet Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
HIPAA and Evaluations. . . . . . . . . . . . . . . . . . . . . 388
Responsible Spin of Data and Information . . . . . . 389
Persuasion and Information. . . . . . . . . . . . . . . . 389
Information and Sensemaking. . . . . . . . . . . . . 391
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
© Lynne Nicholson/Shutterstock
List of Figures
Figure 1-1
The Planning and Evaluation Cycle
Figure 5-3
Effect Theory Example: Effect
Theory for Reducing the Rate of
Congenital Anomalies
Figure 1-2
The Public Health Pyramid
Figure 1-3
The Pyramid as an Ecological Model
Figure 2-1
Effects of Diversity Throughout the
Planning and Evaluation Cycle Stage
in the Planning and Evaluation Cycle
Figure 5-4
Two Roots of Program Failure
Figure 6-1
Connections Among Program,
Agency, and Community
Using Elements of Program Theory
as the Basis for Writing Program
Objectives
Figure 3-1
Figure 6-2
Venn Diagram of CommunityBased, Community-Focused,
and Community-Driven
Diagram Showing Relationship of
Effect Theory Elements to Process
and Outcome Objectives
Figure 3-2
Figure 6-3
Calculations of Options 1 Through
4 Using a Spreadsheet
Figure 6-4
Calculations of Options 5 Through
8 Using a Spreadsheet
Figure 6-5
Calculations of Options 9 and
10 Using a Spreadsheet
Figure 7-1
Amount of Effort Across the
Life of a Health Program
Figure 7-2
Diagram of the Process Theory
Elements Showing the Components
of the Organizational Plan and
Services Utilization Plan
Figure 7-3
Diagram of Theory of Causes/
Determinants for Neural Tube
Defects, as Contributing to Rates of
Congenital Anomalies, Using the
Bowe County Example
Process Theory for Neural Tube
Defects and Congenital Anomalies
Health Problem
Figure 7-4
Effect and Process Theory for Neural
Tube Defect Prevention Program
Figure 8-1
Theory of Causes/Determinants with
Elements of the BPRS Score: Size,
Seriousness, and Interventions
Relevance of Process Theory to
Economic Evaluations
Figure 8-2
Information System Processes
Throughout the Program Planning
Cycle
Figure 9-1
Elements of the Process Theory
Included in a Process Evaluation
Figure 9-2
Roots of Program Failure
Figure 3-3
The Planning and Evaluation Cycle
Figure 4-1
Generic Model of a Theory of Causes
Figure 4-2
Diagram of Theory of Causes/
Determinants of Receiving
Immunizations, as Contributing to
Adult Immunization Rates, Using
the Layetteville Example
Figure 4-3
Figure 4-4
Figure 4-5
Diagram of Theory of Causes/
Determinants for Deaths from
Gunshot Wounds, as Contributing
to Adolescent Death Rates, Using
the Layetteville Example
Figure 5-1
Model of Program Theory
Figure 5-2
The Effect Theory Showing the
Causal Theory Using Community
Diagnosis Elements
xi
xii
List of Figures
Figure 9-3
Examples of Organizational Plan
Inputs and Outputs That Can Be
Measured
Figure 12-2 Decision Tree for Choosing an
Evaluation Design, Based on the
Design’s Typical Use
Figure 9-4
Examples of Services Utilization
Inputs and Outputs That Can Be
Measured
Figure 12-3 Three Sources of Program Failure
Figure 10-1 List of Quality Improvement Tools
with Graphic Examples
Figure 14-1 Contributing Factors to the Total
Amount of Change
Figure 11-1 Planning and Evaluation Cycle, with
Effect Evaluation Highlights
Figure 14-2 Summary of the Three Decisions
for Choosing an Analytic Approach
Figure 11-2 Diagram of Net Effects to Which
Measures Need to Be Sensitive
Figure 14-3 Five Ways That the Rate of
Change Can Be Altered
Figure 11-3 Using the Effect Theory to Identify
Effect Evaluation Variables
Figure 16-1 Making Recommendations Related
to the Organizational and Services
Utilization Plans
Figure 11-4 Effect Theory of Reducing
Congenital Anomalies Showing
Variables
Figure 12-1 Relationship Between the Ability to
Show Causality and the Costs and
Complexity of the Design
Figure 13-1 Probability and Nonprobability
Samples and Their Usage
Figure 16-2 Making Recommendations Related
to the Program Theory
Figure 16-3 The Planning and Evaluation
Cycle with Potential Points for
Recommendations
© Lynne Nicholson/Shutterstock
List of Tables
Table 1-1
Comparison of Outcome-Focused,
Utilization-Focused, and Participatory
Focused Evaluations
Table 4-3
Global Leading Causes of DisabilityAdjusted Life-Years (DALYs) and
Years of Life Lost (YLL)
Table 1-2
Evaluation Standards Established by
the Joint Commission on Standards
for Educational Evaluation
Table 4-4
Table 1-3
Fuzzy Aspects Throughout
the Planning and Evaluation Cycle
Numerators and Denominators for
Selected Epidemiological Rates
Commonly Used in Community
Health Assessments
Table 4-5
Existing Factors, Moderating Factors,
Key Causal Factors, Mediating Factors,
and Health Outcome and Impact for
Five Health Problems in Layetteville
and Bowe County
Table 1-4
A Summary of the Healthy People
2020 Priority Areas
Table 2-1
Examples of Cultural Tailoring
Throughout the Program Planning
and Evaluation Cycle
Table 4-6
Table 2-2
Indicators Used to Measure
Race in Different Surveys
Relationship of Problem Definition to
Program Design and Evaluation
Table 4-7
Table 2-3
Professional Diversity Among
Health Professions
Criteria for Rating Problems
According to the BPRS
Table 4-8
Table 2-4
Cultural Continuum with
Examples of the Distinguishing
Features of Each Stage
Program Prioritization Based on
the Importance and Changeability
of the Health Problem
Table 4-9
Table 3-1
Three Elements of Community,
with Their Characteristics
Table 3-2
Examples of Sources of Data for
Prioritizing Health Problems at
Each Level of the Public Health
Pyramid
Summary of the Six Approaches
to Planning, with Public Health
Examples
Table 3-3
Table 4-10 Examples of Required Existing,
Causal, and Moderating Factors
Across the Pyramid
Comparison of Models Developed
for Public Health Planning
Table 5-1
Table 3-4
A Comparison of the Five
Perspectives on Community Health
and Needs Assessment
Examples of Interventions by Type
and Level of the Public Health
Pyramid
Table 5-2
Haddon’s Typology for Analyzing
an Event, Modified for Use in
Developing Health Promotion and
Prevention Programs
Comparison of Effect Theory,
Espoused Theory, and
Theory-in-Use
Table 5-3
Examples of Types of Theories
Relevant to Developing Theory of
Causative/Determinant Factors or
Theory of Intervention Mechanisms
by Four Health Domains
Table 4-1
Table 4-2
Quality-of-Life Acronyms
and Definitions
xiii
xiv
List of Tables
Table 5-4
Table 6-1
Table 6-2
Table 6-3
Table 6-4
Table 6-5
Table 6-6
Table 6-7
Table 6-8
Table 6-9
Table 7-1
Table 7-2
Examples of Types of Theories
Relevant to Developing the
Organizational Plan and Services
Utilization Plan Components of
the Process Theory
Aspects of Process Objectives as
Related to Components of the
Process Theory, Showing the
TAAPS Elements
Domains of Individual or Family
Health Outcomes with Examples
of Corresponding Indicators and
Standardized Measures
Bowe County Health Problems with
Indicators, Health Outcomes, and
Health Goals
Effect Objectives Related to the
Theory of Causal/Determinant
Factors, Theory of the Intervention
Mechanisms, and Theory of Outcome
to Impact, Using Congenital
Anomalies as an Example, Showing
the TREW Elements
Effect Objectives Related to the
Theory of Causal/Determinant
Factors, Theory of the Intervention
Mechanisms, and Theory of Outcome
to Impact, Using Adolescent
Pregnancy as an Example, Showing
the TREW Elements
Matrix of Decision Options Based on
Current Indicator Value, Population
Trend of the Health Indicator, and
Value of Long-Term Objective or
Standard
Framework for Target Setting:
Interaction of Data Source Availability
and Consistency of Information
Summary of When to Use Each
Option
Range of Target Values Derived from
Options 1 Through 10, Based on the
Data from Figures 6-3 Through 6-5
List of Health Professionals with
a Summary of Typical Legal and
Regulatory Considerations
Relationship of Test Sensitivity and
Specificity to Overinclusion and
Underinclusion
Table 7-3
Examples of Partial- and
Full-Coverage Programs by
Level of the Public Health
Pyramid
Table 7-4
Template for Tracking Services
Utilization Outputs Using Example
Interventions and Hypothetical
Activities
Table 7-5
Hypothetical Logic Model of a
Program for Reducing Congenital
Anomalies
Table 7-6
Generic Elements of a Business
Plan, with Their Purpose and
Corresponding Element of
the Process Theory and Logic
Model
Table 8-1
Formulas Applied for Options A
and B
Table 9-1
Methods of Collecting Process
Evaluation Data
Table 9-2
Example of Measures of Inputs
and Outputs of the Organizational
Plan
Table 9-3
Examples of Measures of Inputs
and Outputs of the Services
Utilization Plan
Table 9-4
Matrix of Undercoverage, Ideal
Coverage, and Overcoverage
Table 9-5
Examples of Process Evaluation
Measures Across the Public Health
Pyramid
Table 10-1 Types of Program Accountability,
with Definitions and Examples of
Process Evaluation Indicators
Table 10-2 Comparison of Improvement
Methodologies and Program Process
Evaluation
Table 10-3 Definitions of Terms Used in
Performance Measurement
Table 10-4 Partial List of Existing Performance
Measurement Systems Used by
Healthcare Organizations, with
Their Websites
Table 11-1 Three Levels of Intervention
Effect Evaluations
Table 11-2 Differences Between Evaluation
and Research
List of Tables
Table 11-3 Advantages and Disadvantages
of Using Each Type of Variable
Table 11-4 Examples of Nominal, Ordinal,
and Continuous Variables for
Different Health Domains
Table 11-5 Example Time Line Showing the
Sequence of Intervention and
Evaluation Activities
Table 11-6 Summary of Evaluation Elements
Table 12-1 Contribution of Various Disciplines to
Health Program Evaluation
Table 12-2 Summary of Main Designs and
Their Use for Individual or
Population-Level Evaluations
Table 12-3 Approaches to Minimizing Each of
the Three Types of Program Failure
Table 13-1 Probability and Nonprobability
Samples and Their Usage
Table 13-2 Comparison of Main Types
of Samples with Regard to
Implementation Ease, Degree of
Representativeness, and Complexity
of Sampling Frame
Table 13-3 Example of Data Sources for Each
Health and Well-Being Domain
Table 13-4 Interaction of Response Bias and
Variable Error
Table 14-1 Calculation of Effectiveness
and Adequacy Indices:
An Example
Table 14-2 Intervention Efficiency as a Relation
of Effect Size and Causal Size
Table 14-3 Factors That Affect the Choice of a
Statistical Test: Questions to
Be Answered
Table 14-4 Analysis Procedures by Level
of Intervention and Level of
Analysis
Table 14-5 Commonly Used Parametric and
Nonparametric Statistical Tests
for Comparison, Association, and
Prediction
Table 14-6 Main Types of Comparison Analyses
Used by Level of Analysis and
Assuming That the Variables
Are at the Same Level of
Measurement
xv
Table 14-7 Main Types of Association Analyses
Used by Level of Analysis, Assuming
That Variables Are the Same Level
of Measurement
Table 14-8 Example of Statistical Tests for
Strength of Association by Level of
Measurement, Using Layetteville
Adolescent Antiviolence Program
Table 14-9 Examples of Statistical Tests by
Evaluation Design and Level of
Measurement, with Examples of
Variables
Table 14-10 Main Types of Prediction Analyses
Used by Level of Analysis, Assuming
That Variables Are at the Same Level
of Measurement
Table 15-1 Comparison of Qualitative Perspectives
with Regard to the Basic Question
Addressed and the Relevance to Health
Program Planning and Evaluation
Table 15-2 Comparison of Major Qualitative
Perspectives with Regard to the
Method Used
Table 15-3 Summary of Key Benefits and
Challenges to Using Qualitative
Methods in Planning and Evaluation
Table 15-4 Sampling Considerations for Each of
the Qualitative Methods Discussed
Table 15-5 Summary of Types of Sampling
Strategies Used with Qualitative
Designs
Table 15-6 Example of Interview Text with Final
Coding
Table 15-7 Suggested Qualitative Methods
by Pyramid Level and Planning
Cycle
Table 16-1 Ethical Frameworks and Principles for
Planning Health Programs
Table 16-2 Comparison of Types of IRB Reviews
Table 16-3 Eight Elements of Informed Consent,
as Required in 45 CFR 46
Table 16-4 Effect of Rigor and Importance of
Claims on Decision Making
Table 16-5 List of Ways to Make Graphs More
Interpretable
Table 16-6 Examples of Dissemination
Modes, Audiences, and Purposes
© Lynne Nicholson/Shutterstock
List of Exhibits
Exhibit 2-1 Checklist to Facilitate Development
of Cultural and Linguistic
Competence Within Healthcare
Organizations
Exhibit 2-2 Checklist to Facilitate Cultural
Competence in Community
Engagement
Exhibit 7-1 Example of an Abbreviated Time Line
for a Short-Term Health Program
Exhibit 7-2 Chapter Text Paragraph Rewritten at
an Eighth-Grade Reading Level
Exhibit 8-1 Example of a Scenario Needing a
Break-Even Analysis
Exhibit 8-2 Example of a Budget Used for a
Break-Even Analysis for Bright Light
on an Excel Spreadsheet
Exhibit 8-3 Break-Even Table Shows Number
of Paying Students Needed to
Break Even
Exhibit 8-4 Example of a Budget Showing
Year-to-Date Variance
Exhibit 8-5 Types of Cost Analyses
Exhibit 9-1 Formulas for Measures of
Coverage
Exhibit 9-2 Example of Narrative Background
about Coverage and Dosage
Measures
Exhibit 9-3 Examples of Coverage Measures
Using an Excel Spreadsheet
Exhibit 9-4 Examples of Calculating Dosage
for the Congenital Anomalies
Prevention Program Using Excel
xvii
© Lynne Nicholson/Shutterstock
Preface to the Fourth Edition
The fourth edition of Health Program Planning
and Evaluation has stayed true to the purpose and
intent of the previous editions. This advancedlevel text is written to address the needs of
professionals from diverse health disciplines
who find themselves responsible for developing,
implementing, or evaluating health programs.
The aim of the text is to assist health professionals to become not only competent health
program planners and evaluators but also savvy
consumers of evaluation reports and prudent
users of evaluation consultants. To that end,
the text includes a variety of practical tools
and concepts necessary to develop and evaluate
health programs, presenting them in language
understandable to both the practicing and novice
health program planner and evaluator.
Health programs are conceptualized as
encompassing a broad range of programmatic
interventions that span the social-ecological
range, from individual-level to population-level
programs. Examples of programs cited throughout the text are specific yet broadly related to
improving health and reflect the breadth of
public health programs. The examples have
been updated once again to reflect current best
practices. Maintaining a public health focus
provides an opportunity to demonstrate how
health programs can target different levels of a
population, different determinants of a health
problem, and different strategies and interventions to address a health problem. In addition,
examples of health programs and references
are selected to pique the interests of the diverse
students and practicing professionals who constitute multidisciplinary program teams. Thus,
the content and examples presented throughout
the text are relevant to health administrators,
medical social workers, nurses, nutritionists,
pharmacists, public health professionals, physical
and occupational therapists, and physicians.
This textbook grew from teaching experiences with both nurses and public health students
and their need for direct application of the program planning and evaluation course content to
their work and to their clients and communities.
Today programs need to be provided through
community-based healthcare settings to address
broad public health issues and expand the individual to population focus. The distinction between
individual patient health and population health is
a prerequisite for the thinking and planning—in
terms of aggregates and full populations—by
students from clinical backgrounds.
In most graduate health professions programs,
students take a research methods course and a
statistics course. Therefore, this evaluation text
avoids duplicating content related to research
methods and statistics while addressing and
extending that content into health program development, implementation, and evaluation. In
addition, because total quality management and
related methodologies are widely used in healthcare
organizations, areas of overlap between quality
improvement methodologies and traditional
program evaluation approaches are discussed.
This includes ways that quality improvement
methodologies complement program evaluations.
Sometimes evaluations are appropriate; sometimes
they are not. Enthusiasm for providing health
programs and performing evaluation is tempered
with thoughtful notes of caution in the hope that
students will avoid potentially serious and costly
program and evaluation mistakes.
xix
xx
Preface to the Fourth Edition
▸▸ Unique Features
The Fourth Edition has retained the three unique
features that distinguish this text from other
program planning and evaluation textbooks: use
of the public health pyramid, consistent use of a
model of the program theory throughout the text,
and role modeling of evidence-based practice.
The public health pyramid explains how
health programs can be developed for individuals, aggregates, populations, and service delivery
systems. Use of the pyramid is also intended as a
practical application of the social-ecological perspective that acknowledges a multilevel approach
to addressing health problems. The public health
pyramid contains four levels: direct services to
individuals; enabling services to aggregates; services
provided to entire populations; and, at the base,
infrastructure. In this textbook, the pyramid is
used as an organizing structure to summarize
the content of each chapter in the “Across the
Pyramid” sections. In these sections, specific
attention is paid to how key concepts in a given
chapter might vary across each pyramid level.
Summarizing the chapter content in this manner
reinforces the perspective that enhancing health
and well-being requires integrated efforts across
the levels of the public health pyramid. Health
program development and evaluation is relevant
for programs intended for individuals, aggregates,
populations, and service delivery systems, and this
fact reinforces the need to tailor program plans
and evaluation designs to the level at which the
program is conceptualized. Using the pyramid
also helps health professionals begin to value
their own and others’ contributions within and
across the levels and to transcend disciplinary
boundaries.
The second unique feature of this text is
that one conceptual model of program planning
and evaluation is used throughout the text: the
program theory. The program theory is like a
curricular strand, connecting content across the
chapters, and activities throughout the planning
and evaluation cycle. The program theory, as
a conceptual model, is composed of elements.
Articulating each of the component elements
of the program theory sharpens the student’s
awareness of what must be addressed to create
an effective health program. One element of the
program theory is the effect theory, which focuses
on how the intervention results in the program
effects. The effect theory had its genesis in the
concepts of action and intervention hypotheses
described by Rossi and Freeman; those concepts
were dropped from later editions of their text.
We believe these authors were onto something
with their effort to elucidate the various pathways leading from a problem to an effect of
the program. Rossi and colleagues’ ideas have
been updated with the language of moderating
and mediating factors and an emphasis on the
intervention mechanisms.
Throughout the current edition of this
textbook, emphasis is given to the effect theory
portion of the program theory. The effect theory
describes relationships among health antecedents,
causes of health problems, program interventions,
and health effects. The hypotheses that comprise
the effect theory need to be understood and explicated to plan a successful health program and
to evaluate the “right” elements of the program.
The usefulness of the effect theory throughout
the planning and evaluation cycle is highlighted
throughout this text; for example, the model
is used as a means of linking program theory
to evaluation designs and data collection. The
model becomes an educational tool by serving
as an example of how the program theory is
manifested throughout the stages of planning
and evaluation, and by reinforcing the value
of carefully articulating the causes of health
problems and consequences of programmatic
interventions. Students and novice program
planners may have an intuitive sense of the
connection between their actions and outcomes,
but they may not know how to articulate those
connections in ways that program stakeholders
can readily grasp. The effect theory and the
process theory—the other main element of the
program theory—provide a basis from which to
identify and describe these connections.
Preface to the Fourth Edition
The third unique feature of this text is the
intentional role modeling of evidence-based
practice. Use of published, empirical evidence
as the basis for practice—whether clinical
practice or program planning practice—is the
professional standard. Each chapter of this book
contains substantive examples drawn from the
published scientific health and health-related
literature. Relying on the literature for examples
of programs, evaluations, and issues is consistent
with the espoused preference of using scientific
evidence as the basis for making programmatic
decisions. Each chapter offers multiple examples
from the health sciences literature that substantiate the information presented in the chapter.
▸▸ Organization
of the Book
The book is organized into six sections, each
covering a major phase in the planning and evaluation cycle. Chapter 1 introduces the fictitious
city of Layetteville and the equally fictitious Bowe
County. In subsequent chapters, chapter content
is applied to the health problems of Layetteville
and Bowe County so that students can learn
how to use the material on an ongoing basis.
In several chapters, the case study is used in the
“Discussion Questions and Activities” section to
provide students with an opportunity to practice
applying the chapter content. In recognition
of the availability of parts of the text in digital
format, each use of the Layetteville case stands
on its own in reference to the chapter’s content.
Section I explores the context in which
health programs and evaluations occur. Chapter 1 begins with an overview of definitions of
health, followed by a historical context. The
public health pyramid is introduced and presented as an ecological framework for thinking
of health programs. An overview of community
is provided and discussed as both the target
and the context of health programs. The role of
community members in health programs and
xxi
evaluations is introduced, and emphasis is given
to community as a context and to strategies for
community participation throughout the program
development and evaluation process. Chapter 2
focuses on the role of diversity in the planning
and evaluation cycle and its effects on the delivery
and evaluation of health programs. Although a
discussion of diversity-related issues could have
been added to each chapter, the sensitive nature
of this topic and its importance in ensuring a
successful health program warranted it being
covered early in the text and as a separate chapter.
Cultural competence is discussed, particularly
with regard to the organization providing the
health program and with regard to the program
staff members.
Section II contains two chapters that focus
on the task of defining the health problem.
Chapter 3 covers planning perspectives and the
history of health program planning. Effective
health program developers understand that
approaches to planning are based on assumptions. These assumptions are exemplified in six
perspectives that provide points of reference for
understanding diverse preferences for prioritizing
health needs and expenditures and therefore for
tailoring planning actions to fit the situation
best. Chapter 3 also reviews perspectives on
conducting a community needs assessment
as foundational to decision making about the
future health program. Essential steps involved
in conducting a community health and needs
assessment are outlined as well.
Chapter 4 expands on key elements of a
community needs assessment, beginning with
a review of the data collection methods appropriate for a community needs assessment. This
discussion is followed by a brief overview of key
epidemiological statistics. Using those statistics
and the data, the reader is guided through the
process of developing a causal statement of the
health problem. This causal statement, which
includes the notion of moderating and mediating
factors in the pathway from causes to problem,
serves as the basis for the effect theory of the
program. Once the causal statement has been
xxii
Preface to the Fourth Edition
developed, prioritization of the problem is needed;
four systems for prioritizing in a rational manner
are reviewed in Chapter 4.
Following prioritization comes planning,
beginning with the decision of how to address the
health problem. In many ways, the two chapters in
Section III form the heart of planning a successful
health program. Unfortunately, students generally
undervalue the importance of theory for selecting
an effective intervention and of establishing target
values for objectives. Chapter 5 explains what theory
is and how it provides a cornerstone for programs
and evaluations. More important, the concept of
intervention is discussed in detail, with attention
given to characteristics that make an intervention
ideal, including attention to intervention dosage.
Program theory is introduced in Chapter 5 as the
basis for organizing ideas related to the selection
and delivery of the interventions in conjunction.
The effect theory element of the program theory
is introduced and the components of the effect
theory are explained. Because the effect theory is
so central to having an effective program intervention and the subsequent program evaluation, it is
discussed in conjunction with several examples
from the Layetteville and Bowe County case.
Chapter 6 goes into detail about developing
goals and objectives for the program, with particular
attention devoted to articulating the interventions provided by the program. A step-by-step
procedure is presented for deriving numerical
targets for the objectives from existing data, which
makes the numerical targets more defendable
and programmatically realistic. We focus on
distinguishing between process objectives and
outcome objectives through the introduction of
two mnemonics: TAAPS (Time frame, Amount
of what Activities done by which Participants/
program Staff) and TREW (Timeframe, what
portion of Recipients experience what Extent
of Which type of change).
Section IV deals with the task of implementing a health program. Chapter 7 provides
an in-depth review of key elements that constitute the process theory element of the program
theory—specifically, the organizational plan and
services utilization plan. The distinction between
inputs and outputs of the process theory is highlighted through examples and a comprehensive
review of possible inputs and outputs. Budgeting
for program operations is covered in this chapter
as well. Chapter 8 is devoted entirely to fiscal data
systems, including key aspects of budgeting, and
informatics. Chapter 9 details how to evaluate the
outputs of the organizational plan and the services
utilization plan. The practical application of measures of coverage is described, along with the need
to connect the results of the process evaluation to
programmatic changes. Program management for
assuring a high-quality program that delivers the
planned intervention is the focus of Chapter 10.
Section V contains chapters that are specific
to conducting the effect evaluations. These chapters present both basic and advanced research
methods from the perspective of a program effect
evaluation. Here, students’ prior knowledge about
research methods and statistics is brought together
in the context of health program and services
evaluation. Chapter 11 highlights the importance
of refining the evaluation question and provides
information on how to clarify the question with
stakeholders. Earlier discussions about program
theory are brought to bear on the development
of the evaluation question. Key issues, such
as data integrity and survey construction, are
addressed with regard to the practicality of
program evaluation. Chapter 12 takes a fresh
approach to evaluation design by organizing the
traditional experimental and quasi-experimental
designs and epidemiological designs into three
levels of program evaluation design based on the
design complexity and purpose of the evaluation.
The discussion of sampling in Chapter 13 retains
the emphasis on practicality for program evaluation rather than taking a pure research approach.
However, sample size and power are discussed
because these factors have profound relevance
to program evaluation. Chapter 14 reviews statistical analysis of data, with special attention to
variables from the effect theory and their level
of measurement. The data analysis is linked to
interpretation, and students are warned about
potential flaws in how numbers are understood.
Chapter 15 provides a review of qualitative designs
Preface to the Fourth Edition
and methods, especially their use in health program development and evaluation.
The final section, Section VI, includes just one
chapter. Chapter 16 discusses the use of evaluation
results when making decisions about existing and
future health programs. Practical and conceptual
issues related to the ethics issues that program
evaluators face are addressed. This chapter also
reviews ways to assess the quality of evaluations
and the professional responsibilities of evaluators.
Each chapter in the book concludes with a
“Discussion Questions and Activities” section.
The questions posed are intended to be provocative and to generate critical thinking. At the
graduate level, students need to be encouraged
to engage in independent thinking and to foster
their ability to provide rationales for decisions.
The discussion questions are developed from this
point of view. In the “Internet Resources” section,
links are provided to websites that support the
content of the chapter. These websites have been
carefully chosen as stable and reliable sources.
▸▸ Additions to and
Revisions in the
Fourth Edition
The fourth edition of Health Program Planning
and Evaluation represents continuous improvement, with corrections and updated references.
Classical references and references that remain
state of the art have been retained.
The Fourth Edition has retained the original
intent—namely, to provide students with the
ability to describe a working theory of how the
intervention acts upon the causes of the health
problem and leads to the desired health results.
Some content has been condensed in order to
allow enough room to describe current evaluation
approaches adequately for both new and experienced practitioners. For instance, Chapter 1 now
includes participatory evaluations in addition to
outcome- and utilization-focused evaluations. In
addition to disciplines traditionally recognized
xxiii
in western medical care, Chapter 2 now includes
acupuncture and massage therapy as examples
of health professional diversity. Discussion of
the nuances of cultural competency has been
refined, in light of the continuing importance
and challenges of this area. Community strengths
have been given more attention in Chapter 3 in
recognition of the powerful potential of shifting
from a “deficit-based” to an “asset-based” perspective on health planning. Chapter 4 now devotes
greater attention to the health evaluation potential of data from social media such as Facebook
and Twitter, as well as geospatial data, including
attendant concerns about privacy, and also notes
implications of the increasingly prevalent public
rankings of community health status. Examples
of infrastructure-level interventions within
the public health pyramid have been added in
Chapter 5. Discussion of financial modeling
options in Chapter 8 now includes simulation
modeling, an exciting if also resource-intensive
option to conducting real-world experiments, which
are, of course, inevitably expensive themselves.
Chapters 9 and 15 include emerging data collection
techniques such as participant self-reports, video,
photos, and audio recordings that may make
public health evaluation more inclusive of the
people such interventions seek to serve. Chapter 13 includes updates on surveying, reflecting
the decreased numbers of people with land-line
phones, long a mainstay of health evaluations.
Options for online surveying have been updated
in Chapter 14; given the rapid evolution of big
data such as those available from social media,
billing, and medical records, discussion of this
topic has been updated in Chapter 13 as well.
Finally, Chapter 16 now includes bioethics—
the application of ethical and philosophical
principles to medical decision making—as an
increasingly salient component of responsible
health evaluation.
In sum, we have worked hard to sustain
this book’s conceptual and empirical rigor and
currency in the Fourth Edition while maintaining
accessibility for a range of health evaluators. Above
all, we hope this book is useful to our readers’
vitally important efforts to improve health.
© Lynne Nicholson/Shutterstock
Acknowledgments
We are indebted to the many people who supported
and aided us in preparing this fourth edition
of Health Program Planning and Evaluation: A
Practical, Systematic Approach for Community
Health. We remain grateful to the numerous
students over the years who asked questions
that revealed the typical sticking points in their
acquiring and understanding of the concepts
and content, as well as where new explanations
were needed. Through their eyes we have learned
there is no one way to explain a complex notion
or process. Their interest and enthusiasm for
planning and evaluating health programs was
a great motivator for writing this book.
Several additional colleagues helped fine-tune
this text. We are especially indebted to Arden
Handler at the School of Public Health, University
of Illinois at Chicago, for taking time to contribute
to this textbook. Her devotion to quality and clarity
has added much to the richness of otherwise dry
material. We remain deeply indebted to Deborah
Rosenberg, also at the School of Public Health
University of Illinois at Chicago, for sharing her
innovative and quintessentially useful work on
developing targets for program objectives. Special
thanks as well to Joseph Chen, at the University
of Texas School of Public Health, for his many
contributions to updating the literature cited
across many chapters and for his contribution
on big data. Last, but not least, thanks to Mike
Brown, publisher at Jones & Bartlett Learning, for
his encouragement and patience over the years.
xxv
© Lynne Nicholson/Shutterstock
List of Acronyms
ABCD
Asset-based community
development
DHHS
U.S. Department of Health
and Human Services
ACA
Affordable Care Act
DSM-5
AEA
American Evaluation Association
AHRQ
Agency for Healthcare Research
and Quality
Diagnostic and Statistical
Manual of Mental Disorders,
Fifth Edition
EBM
Evidence-based medicine
ANOVA
Analysis of variance
EBP
Evidence-based practice
APHA
American Public Health Association
EHR
Electronic health record
BPRS
Basic priority rating system
EMR
Electronic medical record
BRFSS
Behavioral Risk Factor Surveillance
System
FTE
Full-time equivalent
GAO
BSC
Balanced Score Card
U.S. Government Accountability
Office
CAHPS
Consumer Assessment of Health
Plans
GNP
Gross Product
GPRA
Government Performance
and Results Act
HEDIS
Healthcare Effectiveness Data
and Information Set
CARF
Commission on Accreditation
of Rehabilitation Facilities
CAST-5
Capacity Assessment of Title-V
CBA
Cost–benefit analysis
HIPAA
CBPR
Community-based participatory
research
Health Insurance Portability
and Accountability Act
HIT
Health information technology
CDC
Centers for Disease Control
and Prevention
HMOs
Health maintenance organizations
HRQOL
Health-related quality of life
HRSA
Health Resources and Services
Administration (part of DHHS)
CEA
Cost-effectiveness analysis
CER
Cost-effectiveness ratio
CFIR
Consolidated Framework for
Implementation Research
i-APP
Innovation–Adolescent Preventing
Pregnancy (Program)
CFR
Code of Federal Regulations
ICC
Intraclass correlation
CHIP
Community Health Improvement
Process
IRB
Institutional review board
CI
Confidence interval
JCAHO
CPT
Current Procedural Terminology
Joint Commission on the
Accreditation of Healthcare
Organizations
CQI
Continuous quality improvement
MAPP
CUA
Cost–utility analysis
Mobilizing for Action through
Planning and Partnership
DALY
Disability-adjusted life-year
MBO
Management by objectives
xxvii
xxviii
MCHB
NACCHO
List of Acronyms
Maternal and Child Health Bureau
(part of HRSA)
PSA
Public service announcement
QALY
Quality-adjusted life-year
National Association of City
and County Health Officers
RAR
Rapid assessment and response
RARE
Rapid assessment and response
and evaluation
RE-AIM
Reach, Effectiveness, Adoption,
Implementation, and Maintenance
model
NAMI
National Alliance on Mental Illness
NCHS
National Center for Health Statistics
NCQA
National Commission on Quality
Assurance
NFPS
National Family Planning Survey
RR
Relative risk
NHANES
National Health and Nutrition
Examination Survey
SAMHSA
Substance Abuse and Mental
Health Services Administration
NHIS
National Health Interview Survey
SCHIP
NIH
National Institutes of Health
State Child Health Insurance
Program
NPHPS
National Public Health Performance
Standards
SES
Socioeconomic status
SMART
OHRP
Office for Human Research
Protections
Specific, measurable, achievable,
realistic, and time (objective)
TAAPS
OMB
Office of Management
and Budgeting
Time frame, Amount of what
Activities done by which
Participants/program Staff
OR
Odds ratio
TQM
Total quality management
PACE-EH
Protocol for Assessing Excellence
in Environmental Health
TREW
PAHO
Pan American Health Organization
Time frame, what portion of
Recipients experience what
Extent of Which type of change
PDCA
Plan-Do-Check-Act
UOS
Units of service
PEARL
Property, economic, acceptability,
resource, legality system
WHO
World Health Organization
WIC
PERT
Program Evaluation and Review
Technique
Special Supplemental Nutrition
Program for Women, Infants,
and Children
PPIP
Putting Prevention into Prevention
YHL
Years of healthy life
PRECEDE
Predisposing, Reinforcing,
and Enabling Factors in
Community Education
Development and Evaluation
(model)
YLL
Years of life lost
YPLL
Years of potential life lost
SECTION I
The Context of
Health Program
Development
© Lynne Nicholson/Shutterstock
CHAPTER 1
Context of Health Program
Development and Evaluation
H
ealth is not a state of being that can easily
be achieved through isolated, uninformed,
individualistic actions. Health of individuals, of families, and of populations is a state in
which physical, mental, and social well-being are
integrated to enable optimal functioning. From
this perspective, achieving and maintaining health
across a life span is a complex, complicated, intricate affair. For some, health is present irrespective
of any special efforts or intention. For most of
us, health requires, at a minimum, some level of
attention and specific information. It is through
health programs that attention is given focus and
information is provided or made available, but
that does not guarantee that the attention and
information are translated into actions or behaviors
needed to achieve health. Thus, those providing
health programs, however large or small, need
to understand both the processes whereby those
in need of attention and health information can
receive what is needed, and also the processes by
which to learn from the experience of providing
the health program.
The processes and effects of health program planning and evaluation are the subjects
of this text. The discussion begins here with a
brief overview of the historical context. This
background sets the stage for appreciating the
considerable number of publications on the topic
of health program planning and evaluation,
and for acknowledging the professionalization
of evaluators. The use of the term processes to
describe the actions involved in health program
planning and evaluation is intended to denote
action, cycles, and open-endedness. This chapter
introduces the planning and evaluation cycle,
and the interactions and iterative nature of
this cycle are stressed throughout the text.
Because health is an individual, aggregate,
and population phenomenon, health programs
need to be conceptualized across those levels.
The public health pyramid, introduced in this
chapter, is used throughout the text as a tool
for conceptualizing and actualizing health
programs for individuals, aggregates, and
populations.
© Lynne Nicholson/Shutterstock
3
4
Chapter 1 Context of Health Program Development and Evaluation
▸▸ History and Context
An appropriate starting point for this text is
reflecting on and understanding what “health”
is, along with having a basic appreciation for the
genesis of the fields of health program planning
and evaluation. A foundation in these elements
is key to becoming an evaluation professional.
Concept of Health
To begin the health program planning and
evaluation cycle requires first reflecting on the
meaning of health. Both explicit and implicit
meanings of health can dramatically influence
what is considered the health problem and the
subsequent direction of a program. The most
widely accepted definition of health is that put
forth by the World Health Organization (WHO),
which for the first time defined health as more
than the absence of illness and as the presence
of well-being (WHO, 1947).
Since the publication of the WHO definition, health has come to be viewed across the
health professions as a holistic concept that
encompasses the presence of physical, mental,
developmental, social, and financial capabilities, assets, and balance. This idea does not
preclude each health profession from having a
particular aspect of health to which it primarily
contributes. For example, a dentist contributes
primarily to a patient’s oral health, knowing that
the state of the patient’s teeth and gums has a
direct relationship to his or her physical and
social health. Thus the dentist might say that
the health problem is caries. The term health
problem is used, rather than illness, diagnosis, or
pathology, in keeping with the holistic view that
there can be problems, deficits, and pathologies
in one component of health while the other
components remain “healthy.” Using the term
health problem also makes it easier to think
about and plan health programs for aggregates
of individuals. A community, a family, and a
school can each have a health problem that is
the focus of a health program intervention. The
extent to which the health program planners have
a shared definition of health and have defined
the scope of that definition influences the nature
of the health program.
Health is a matter of concern for more than
just health professionals. For many Americans,
the concept of health is perceived as a right,
along with civil rights and liberties. The right
to health is often translated by the public and
politicians into the perceived right to have or to
access health care. This political aspect of health
is the genesis of health policy at the local, federal,
and international levels. The extent to which the
political nature of health underlies the health
problem of concern being programmatically
addressed also influences the final nature of the
health program.
Health Programs, Projects,
and Services
What distinguishes a program from a project or
from a service can be difficult to explain, given
the fluidity of language and terms. The term
program is fairly generic but generally connotes
a structured effort to provide a specific set of
services or interventions. In contrast, a project
often refers to a time-limited or experimental
effort to provide a specific set of services or
interventions through an organizational structure. In the abstract, a service can be difficult to
define but generally includes interaction between
provider and client, an intangibility aspect to
what is provided, and a nonpermanence or
transitory nature to what is provided. Using this
definition of service, it is easy to see that what
is provided in a health program qualifies as a
service, although it may not be a health service.
A health program is a totality of an organized
structure designed for the provision of a fairly
discrete health-focused intervention, where that
intervention is designed for a specific target
audience. By comparison, health services are
the organizational structures through which
providers interact with clients or patients to meet
the needs or address the health problems of the
clients or patients. Health programs, particularly
History and Context
in public health, tend to provide educational
services, have a prevention focus, and deliver
services that are aggregate or population-focused.
In contrast, health services exist exclusively as
direct services. Recognizing the distinction
between health programs and health services is
important for understanding the corresponding
unique planning and evaluation needs of each.
History of Health Program
Planning and Evaluation
The history of planning health programs has a
different lineage than that of program evaluation.
Only relatively recently, in historical terms, have
these lineages begun to overlap, with resulting
synergies. Planning for health programs has
the older history, if public health is considered. Rosen (1993) argued that public health
planning began approximately 4,000 years ago
with planned cities in the Indus Valley that had
covered sewers. Particularly since the Industrial
Revolution, planning for the health of populations
has progressed, and it is now considered a key
characteristic of the discipline of public health.
Blum (1981) related planning to efforts
undertaken on behalf of the public well-being
to achieve deliberate or intended social change
as well as providing a sense of direction and
alternative modes of proceeding to influence
social attitudes and actions. Others (Dever, 1980;
Rohrer, 1996; Turnock, 2004) have similarly
defined planning as an intentional effort to create
something that has not occurred previously for
the betterment of others and for the purpose of
meeting desired goals. The purpose of planning
is to ensure that a program has the best possible
likelihood of being successful, defined in terms of
being effective with the least possible resources.
Planning encompasses a variety of activities
undertaken to meet this purpose.
The quintessential example of planning is
the development and use of the Healthy People
goals. In 1979, Healthy People (U.S. Department
of Health, Education, and Welfare [DHEW],
1979) was published as an outgrowth of the
5
need to establish an illness prevention agenda
for the United States. The companion publication, Promoting Health/Preventing Disease (U.S.
Department of Health and Human Services
[DHHS], 1980), marked the first time that
goals and objectives regarding specific areas of
the nation’s health were made explicit, with the
expectation that these goals would be met by the
year 1990. Healthy People became the framework
for the development of state and local health
promotion and disease prevention agendas.
Since its initial publication, the U.S. goals for
national health have been revised and published
as Healthy People 2000 (DHHS, 1991), Healthy
Communities 2000 (American Public Health
Association [APHA], 1991), Healthy People 2010
(DHHS, 2000), and Healthy People 2020 (DHHS,
2011), with development of Healthy People 2030
underway. Other nations also set health status
goals and international organizations, such as
the World Health Organization (WHO) and
Pan American Health Organization (PAHO),
develop health goals applicable across nations.
The evolution of Healthy People goals also
reflects the accelerating rate of emphasis on
nationwide coordination of health promotion and
disease prevention efforts and a reliance on systematic planning to achieve this coordination. The
development of the Healthy People publications also
reflects the underlying assumption that planning is
a rational activity that can lead to results. However,
at the end of each 10-year cycle, many of the U.S.
health objectives were not achieved, reflecting the
potential for planning to fail. Given this failure
potential, this text emphasizes techniques to help
future planners of health programs to be more
realistic in setting goals and less dependent upon
a linear, rational approach to planning.
The Healthy People 1990 objectives were
developed by academics and clinician experts
in illness prevention and health promotion. In
contrast, development of the goals and health
problems listed in Healthy People 2010 and
Healthy People 2020 incorporated ideas generated
at public forums and through Internet commentary; these ideas later were revised and refined
by expert panels before final publication of the
6
Chapter 1 Context of Health Program Development and Evaluation
objectives. Greater participation of the public
during the planning stage of health programs has
become the norm. In keeping with the emphasis
on participation, the role and involvement of
stakeholders are stressed at each stage of the
planning and evaluation cycle.
The history of evaluation, from which the
evaluation of health programs grew, is far shorter
than the history of planning, beginning roughly in
the early 1900s, but it is equally rich in important
lessons for future health program evaluators.
The first evaluations were done in the field of
education, particularly as student assessment and
evaluation of teaching strategies gained interest
(Patton, 2008). Assessment of student scholastic
achievement is a comparatively circumscribed
outcome of an educational intervention. For
this reason, early program evaluators came from
the discipline of education, and it was from the
fields of education and educational psychology
that many methodological advances were made
and statistics developed.
Guba and Lincoln (1987) summarized the
history of evaluations by proposing generational
milestones or characteristics that typify distinct
generations. Later, Swenson (1991) built on their
concept of generations by acknowledging that
subsequent generations of evaluations will occur.
Each generation incorporates the knowledge of
early evaluations and extends that knowledge based
on current broad cultural and political trends.
Guba and Lincoln (1987) called the first
generation of evaluations in the early 1900s “the
technical generation.” During this time, nascent
scientific management, statistics, and research
methodologies were used to test interventions.
Currently, evaluations continue to incorporate the
rationality of this generation by using activities
that are systematic, science based, logical, and
sequential. Rational approaches to evaluations
focus on identifying the best-known intervention or strategy given the current knowledge,
measuring quantifiable outcomes experienced by
program participants, and deducing the degree
of effect from the program.
The second generation, which lasted until
the 1960s, focused on using goals and objectives
as the basis for evaluation. Second-generation
evaluations were predominantly descriptive. With
the introduction in the 1960s of broad innovation
and initiation of federal social service programs,
including Medicare, Medicaid, and Head Start,
the focus of evaluations shifted to establishing
the merit and value of the programs. Because
of the political issues surrounding these and
similar federal programs, determining whether
the social policies were having any effect on
people become a priority. Programs needed to
be judged on their merits and effectiveness. The
U.S. General Accounting Office (GAO; now
called the Government Accountability Office)
had been established in 1921 for the purpose of
studying the utilization of public finances, assisting Congress in decision making with regard to
policy and funding, and evaluating government
programs. The second-generation evaluation
emphasis on quantifying effects was spurred, in
part, by reports from the GAO that were based
on the evaluations of federal programs.
Typically, the results of evaluations were not
used in the “early” days of evaluating education
and social programs. That is, federal health
policy was not driven by whether evaluations
showed the programs to be successful. Although
the scientific rigor of evaluations improved,
their usefulness remained minimal. Beginning
in the 1980s, however, the third generation of
evaluations—termed “the negotiation generation”
or “the responsiveness generation”—began. During
this generation, evaluators began to acknowledge
that they were not autonomous and that their
work needed to respond to the needs of those
being evaluated. As a result of this awareness,
several lineages have emerged. These lineages
within the responsiveness generation account
for the current diversity in types, emphases,
and philosophies related to program evaluation.
One lineage is utilization-focused evaluation
(Patton, 2012), in which the evaluator’s primary
concern is with developing an evaluation that will
be used by the stakeholders. Utilization-focused
evaluations are built on the following premises
(Patton, 1987): Concern for use of the evaluation
pervades the evaluation from beginning to end;
History and Context
evaluations are aimed at the interests and needs
of the users; users of the evaluation must be invested in the decisions regarding the evaluation;
and a variety of community, organizational,
political, resource, and scientific factors affect
the utilization of evaluations. Utilization-focused
evaluation differs from evaluations that are
focused exclusively on outcomes
Another lineage is participatory evaluation
(Whitmore, 1998), in which the evaluation is
merely guided by the expert and is actually generated by and conducted by those invested in the
health problem. A participatory or e mpowerment
approach invites a wide range of stakeholders
into the activity of planning and evaluation,
providing those participants with the skills and
knowledge to contribute substantively to the
activities and fostering their sense of ownership
of the product (TABLE 1-1).
The fourth generation of evaluation,
which emerged in the mid-1990s, seems to
be meta-evaluation, that is, the evaluation of
evaluations done across similar programs. This
trend in program evaluation parallels the trend
in social science toward using meta-analysis of
existing studies to better understand theorized
relationships and the trend across the health
professions toward establishing evidence-based
practice guidelines. This new generation became possible because of a pervasive culture of
evaluation in the health services and because
of the availability of huge data sets for use in
the meta-evaluations. An early example of the
evaluation culture was the mandate from United
Way, a major funder of community-based health
programs, for their grantees to conduct outcome
evaluations. To help grantees meet this mandate,
United Way published a user-friendly manual
(United Way of America, 1996) that could be
used by nonprofessionals in the development
of basic program evaluations. More broadly,
the culture of evaluation can be seen in the
explicit requirement of federal agencies that
fund community-based health programs that
TABLE 1-1 Comparison of Outcome-Focused, Utilization-Focused, and Participatory
Focused Evaluations
Outcome-Focused
Evaluations
7
U…
Delivering a high-quality product at a reasonable price is not enough anymore.
That’s why we have developed 5 beneficial guarantees that will make your experience with our service enjoyable, easy, and safe.
You have to be 100% sure of the quality of your product to give a money-back guarantee. This describes us perfectly. Make sure that this guarantee is totally transparent.
Read moreEach paper is composed from scratch, according to your instructions. It is then checked by our plagiarism-detection software. There is no gap where plagiarism could squeeze in.
Read moreThanks to our free revisions, there is no way for you to be unsatisfied. We will work on your paper until you are completely happy with the result.
Read moreYour email is safe, as we store it according to international data protection rules. Your bank details are secure, as we use only reliable payment systems.
Read moreBy sending us your money, you buy the service we provide. Check out our terms and conditions if you prefer business talks to be laid out in official language.
Read more