1. Why is it important for professionals to evaluate public health promotion programs?
The main value of evaluation is knowing that what you are doing is effective. If it is not effective what can be done to improve it and make it more effective. There are a lot of different stakeholders in evaluation who want to know what is going on in the program. From a stakeholder perspective they want to know it is effective. From a program provider perspective you want to know that what you are doing is working. It is always important to keep reviewing and looking at things to make sure it is functioning the way it should be. From the participant perspective you want it to be high quality and make sure it is effective. It is easy for people to do the same thing and it’s very important to go back and make sure you are adjusting for various situations.
Key stakeholders in the evaluation of public health promotion programs:
- Funding Source
- Staff or Program Providers
- Community – Not in every program but in some there is the benefit for the community/larger stakeholders
- Partners – Partners in the community or others who partner with the evaluation
2. What careers are available in the field of program evaluation?
In some cases and organizations, there are internal evaluators; school districts are a good example of this. They have so much that they need to report on like grant funding, etc. An organization that does a health program may have someone they put on their staff that does the evaluating of that program. You’ll often see this built into an organization where they have to do a lot of reporting.
There are independent consultants as well who have their own business and get grants and contracts to work with other organization to do evaluation. I actually work like that. Within the university I have a center and I partner with someone who has a grant or contract to do a program.
Usually, evaluation is built into a role, it’s not the main thing that someone does but it is a little part of the overall position because many organizations can’t afford to have a full-time employee or consultant as an evaluator. The most common things graduates of the MPH program who are savvy in program evaluation may be involved in might be being called on for insights. For instance, small non-profit organizations are applying and getting grant funding to do different projects all the time. It may be a requirement of a grant to have some program evaluations. Since organizations of this type usually have tight budgets they could bring on an MPH graduate for this project. Another area an MPH degree can be especially applicable is knowing enough about evaluation to hire an external evaluator or consultant. When you can’t afford to have a full comprehensive evaluation you may need to have enough knowledge to know what kind of help you need if you have a low budget.
3. What merits of health interventions and policies should students be aware of and look for during an evaluation?
It is most important to look at the goals and objectives of the program and keep those in mind during the evaluation. Making sure that you’re understanding the desired outcomes and checking to make sure those are happening. This is another field is the evidence-based movement. There are plenty of existing programs out there that haven’t met the evidence-based standards. During an evaluations if not having met these standards is something that has not been tested as the evaluator you should be making the suggestions on how the organization can meet those.
I am currently working with the state of Nevada with their early hearing detection program. They have received some funding from the CDC and they are looking into the lack of follow up with infant patients who they have detected hearing loss. With this funding they are trying to get a plan in place for follow-up testing within a certain amount of time. They are currently in the exploratory phases of this program and looking into the different aspects of that. They were trying to identify where the fall off was happening and then work with providers to make a better follow-up connection. Once they fine tune that process they could look at a different part of their program like data collection.
The state of Nevada did training to improve quality of data collection. Not all evaluations are like that, but the best ones are. You don’t have to just do one thing and not change anything. The whole point of a good evaluation is to continually feed information back into the program to make adjustments. The most useful kind of evaluation is to look at the organization as a whole and continually make adjustments.
The first step in research is you set-up a question you want to research. Then, you test that question and always keeping your control in mind. In evaluation, sometimes you do that kind of study, but things change in evaluation and you have to remember that you’re looking at a real-life program and you can’t necessarily control everything. It is in an action-oriented way that you are finding out how things are working. In research you are trying to study and have results that can be generalized for a population. In evaluation, it is really specific to a program and it may not work the same way somewhere else. You have to ask yourself, what adjustments might I need to make to have it be effective in a different setting? There is a push towards evidence-based programs but it can be difficult to make them work in different settings. We use many of the same methods in research and evaluation. Most evaluators come from another field and use the skills and methods learned there to apply it to all of their evaluations.
4. Do you believe that program evaluation is a valuable skill for public health professionals? If so, why?
Yes, I definitely believe it is a valuable skill. In any situation, in an evaluation, there will be that need to assess what you are doing and determine how effective it is. Those skills are useful in any facet of public health at some level. You will need to evaluate what you are doing even if it is not a health promotion program.
5. Can you give a high-level overview of the steps in the program evaluation process?
- Describe the program that is being evaluated. Oftentimes you can use a logic model, which lays out the elements of the program that will be in place. You need to ask, what do you have going into the program, who is the program for, what are the inputs and outcomes? You also need to identify the outcomes over time; the short, medium and long-term goals? And, are the activities that you are planning going to meet your outcomes?
- Develop evaluation questions using the logic model. Some questions may be: what are you hoping to learn about the program through the evaluation? Is it an outcomes-focused evaluations? Will the outcomes happen? Or you can look at a process type evaluation and ask: how well is the program being implemented? Are we implementing as planned?
- a. Determine how you will measure outcomes and deciding on methods. Will you be utilizing surveys, interviews, focus groups, observation, document review, etc.?
- Identify the timeline and when you will report out on it.
- Carry out the evaluation
- Evaluate the data to see how it can be fed into the overall program to improve it.
The planning portion can take a while and it will depend on what the budget is to decide how much time you have to dedicate to an evaluation. Sometimes the planning happens in a grant-writing phase. There will likely be multiple meetings to put together a logic model and put together the evaluation questions. Personally, I take a partnership approach to evaluation to involve the stakeholders in the process.
6. What advice do you give to MPH students in your advisor role?
I advise students, even if they don’t think they will go into program evaluation, to develop these skills and see some things in action. Seeing the job sets our graduates get I find that they often need to have some of these skills even if the position isn’t in evaluation directly. For example, in a small non-profit you will have to do a variety of things and apply these skills.
When I advise students on picking their internship, I suggest that they think about where they see themselves working and what they see themselves really enjoying. I tell students to picture where they want to go and be realistic about the picture being the long-term goa. Then, they’ll need to come up with the steps that they’ll need to take to get where they want to go. Students need to really look at what types of positions are out there and how they can use their experience to get where they want to be. There are a lot of different places you can work with an MPH, but think about what could be valuable to learn and be open to places that can help you grow.
Sometimes in internships students discover that what they thought they would love isn’t actually a good fit for them.
7. Is there anything you want to add that you feel we haven’t covered?
A lot of the methods and courses are useful in program evaluation even if it is directed toward research in the course, statistics being one. However, those may be the things you will need in program evaluation to be successful. You use different skills and gain different knowledge in internships can be applied to program evaluation.
Lastly, the American Evaluation Association is the main organization for evaluation in the U.S. They have a lot of resources on their website – www.eval.org. They have coffee break webinars that are 20 minutes a few times a month and you can typically attend for free. That is a great organization to check out if you are interested in program evaluation.