Evaluating Local Programs – 2/2/06

Q:

About $6 million of our revenue generated from our excise tax goes to our local health agencies, which are local health departments and county nursing services. The Review Committee wants us to evaluate the efforts of these local programs. Currently, we ask our local health agencies to report on process measures, such as the number of training provided, number of meetings held, and the number of people reached. I realize the CDC Best Practices recommend "community programs" as an integral part of statewide initiatives, however the document fails outline the steps to fund and evaluate these programs.
 
Here are my questions for other states:

1. What percent of their dollars are devoted to local programs?
2. Do they fund only local public health agencies or are other community programs funded?
3. Is the application process competitive or non-competitive?
4. What process do they use to evaluate local programs?
5. Are strategies and activities driven from the state program or are local program free to choose any strategies they want to work on?

A:

1. Arizona:
    
   1. We devote almost 48% of our excise tax dollars to local and community-based programs.
   2. We fund county health departments, Native American tribes, Tribal Organizations, and grassroots community-based programs.
   3. For our county health departments the application process is non-competitive due to our governmental relationship. Our community partners are sole source contracts deemed impractical to bid based upon the specific populations that they serve (i.e. African American, Asian American, Hispanic, Low SES, Native American). These are re-evaluated every year.
   4. We have a centralized evaluation process, and have a contract with one of our state universities to evaluate our programs. The evaluation is done in conjunction with our long-term planning, and currently we use a variety of instruments to assess the program's cessation and prevention services.
   5. We have finished a draft of our 5-year Tobacco Program strategic plan which will guide how we provide services. However, we use a combination of both state and local program planning to determine the services offered at the local level based upon population needs. We have an approval process in place to determine what can be conducted at the local level. We also are establishing a statewide tobacco plan with stakeholders to further guide our program services.
       
2. California:
    
   1. Approximately 50% of the budget for the California Tobacco Control Program is directed towards local programs and statewide infrastructure to support local program efforts, e.g., quitline, clearinghouse, training and technical assistance.
   2. In California approximately half of the funds available for local programs are designated for local public health departments and approximately half are designated for the competitive grants program which funds both local projects and some statewide infrastructure, e.g., quitline, clearinghouse, training and technical assistance contracts.
   3. In Californian, local health departments are funded non-competitively using a formula that is in statute. Other funds to community-based organizations and for statewide infrastructure are competitively funded.
   4. Local projects are required to designate at least 10 percent of their budget towards evaluation of their own program efforts. See the following for a description of how local program evaluation efforts have evolved in California. Tang H, Cowling DW, Koumjian, Roeseler A, Lloyd J and Rogers T. Building local program evaluation capacity toward a comprehensive evaluation (2002) in Responding to Sponsors and Stakeholders in Complex Evaluation Environments. Rakesh Mohan, David J. Bernstein, and Maria D. Whitset (eds.) New Directions for Evaluation, no. 95. Jossey-Bass. Additionally, there is a Local Program Evaluation Planning Guide that describes how local programs are evaluated. Click to access the planning guide. A hard copy can be obtained by e-mailing Karen Aschenbrenner, Library Assistant, California Tobacco Control Program. In a nutshell, in California, projects are required to write objectives that focus on intermediate outcomes, e.g., policy change, conduct an evaluation, and prepare a final evaluation report that summarizes the intervention, evaluation methods, outcomes, and makes recommendations for future efforts.
   5. The state has identified four broad priority areas and approximately 190 community-level indicators. Within this framework, local projects select the priority area and indicators which they will develop intermediate level outcome objectives and the activities that it will take to accomplish those activities. During the funding process, there is a negotiation process to strengthen objectives and activities.
       
3. Minnesota:
    
   1. 100% of state funds appropriated to the Dept. of Health are devoted to local programs ($3.28M annually). Other organizations in the state also provide funds to local programs as well as statewide activities, such as media campaigns.
   2. Funds go to local public health agencies and other community-based organizations serving either a geographic area or priority populations.
   3. Competitive
   4. Grantees report on some process measures through our reporting system, but full process or outcome evaluation of their programs is not required.
   5. Strategies are provided to them through our RFP process and related intermediate outcomes are incorporated into their grant agreement with the state. We view our "menu" of strategies as evidence-based; therefore, we have placed less emphasis on evaluation at the community level.
       
4. New Hampshire:
    
   1. About 1/3 of our dollars go toward local programs, in keeping with the Best Practices guidelines
   2. There are few local public health agencies in NH. So, we fund community coalitions with all sorts of fiscal agents, from school districts to police departments to hospitals.
   3. Competitive.
   4. Like you, we use primarily process measures.
   5. Strategies and activities are driven by the State program. This year, I borrowed Wyoming's best practices guidance to give the applicants more choices than they have had for the past few years.
       
5. New Mexico:
    
   1. 100%
   2. Fund only other community programs based on CDC Goal Areas and Best Practices.
   3. For our procurement process any award over $30,000 must be competitive. We sometimes award $30,000 or less, non-competitive funding to help build capacity in areas of need.
   4. Our contracts specify deliverables. Contractors (local grantees) report on their deliverables and activities using our on-line evaluation tool.
   5. Strategies are driven by the state program and are based on CDC Best Practices, evidence-based programming, and needs as determined by the state program. Our RFP and funding re-application specifies these strategies and priority populations
       
6. New York:
    
   1. About 60% of our program dollars fund local programs.
   2. We have moved away from funding local health departments and now fund many more community organizations than health departments.
   3. Competitive.
   4. Community programs are required to report monthly on process indicators. Local programs are required to identify at least one activity area (where they are investing significant resources) to demonstrate that the activities have caused the intended tobacco control impact. The partner works with a state staff person to agree on the evaluation plan. The main point of this exercise is to get the local program to articulate what exactly it is they hope to accomplish with their intervention, and, by holding them accountable for demonstrating that, forcing them to think whether their intervention really will have an impact (maybe they need to rethink their plan).
   5. Strategies and activities are heavily scripted. Resources are focused on a few activities so we can have an impact. We can't spread resources thinly across many activities, but must concentrate them to have an impact. Community programs used to have a fair amount of freedom to do what they thought made sense in their community. Now all partners work in a coordinated fashion to do what is best for tobacco control across the state. Our funds are not grants to local programs to do what they want. Our funds allow local partners to participate in coordinated statewide tobacco control action.
       
7. North Dakota: I am going to refer you to a report prepared in 2004 as an early assessment of our tobacco control efforts. We are preparing to release the report for fiscal year 05 shortly. The 05 will report an expanded version of cessation efforts including our quitline and policy efforts (passage of a smoke-free law) and we are adding a section on disparities. We too struggled with how to report progess and evaluate our efforts as so many of our strategies are successful because they are used in combination with other strategies. We developed an advisory committee to assist us with this effort and ending up defining what we call our "leading tobacco indicators" which are used to judge our progress and evaluate our efforts. The indicators are really more outcome in nature. The report defines our funding that is allocated to communities, tribes and school programs and describes overall how all the funding is distributed. Click to view the 2004 report. I will be happy to share the 2005 report when available.
    
   Between 75 and 80 percent of our annual funding goes out to local programs. Our MSA funding is distributed based on a funding formula defined by the state legislature. These grantees must develop a plan and budget that has to be approved by an advisory committee to the state tobacco control program. Part of our CDC funding which goes to local agencies is allocated based on a competitive grants process. Both grants programs must follow "Best Practices" in tobacco control to the degree possible. In the past year we created a combined application form that covers both grant programs since the goals are the same. We would be happy to share a copy of the form. In addition we have the grantees report twice a year and compile the results. While many of the reports are process in nature, we also track policy efforts and successes. We believe these process measures contribute and lead to changes in our leading indicator report.
    
   We are just in the process of a new evaluation of these programs where we are defining the critical success factors and outcomes so we can help to determine why some programs are successful and others are not. We are just in the process of defining the variables that will be assessed.
    
   CDC has a number of documents that we used to help us determine what areas to fund. We used their sustainability documents to help us define our priorities since we did not have enough funding for a fully comprehensive approach.
    
   Please feel free to contact Kathleen Mangskau for more information on our process. I would enjoy discussing with you as perhaps we can learn from each other.
    
8. Oklahoma – (joint response from state and foundation):
    
   1. It depends on what you consider here. If you mean just direct funding to local programs, then it would be about 40% of the budget, but this does not include what we pay for training, technical assistance, consultation and contract monitoring.
   2. We fund a variety of community agencies, and we require that a local coalition select the lead agency. This acknowledges that the county health department may not be the best fit in all communities, and it assures that the community has come together to determine the best fit for their community. In addition to the comprehensive, community-based tobacco control program, the Oklahoma State Department of Health Tobacco Use Prevention Service funds three ethnic tobacco education networks awarded through competitive bids. One contractor is an urban community college with links to a statewide health education network, the other two networks are in bid status. We also directly fund five County Health Departments to provide competent SWAT (Students Working Against Tobacco) regional coordinators. It was a great strategy by far to honor local decision-making by having the lead agency selection process for the Communities of Excellence led by each applying local coalition.
   3. The process is competitive
   4. We have a contract with the University of Oklahoma College of Public Health to evaluate our "Communities of Excellence" grantees. All grantees are required to work on certain indicators and social capital assets and the evaluator is assessing both process and outcomes for these indicators and assets.
   5. It is a combination. We require the Communities of Excellence Plus model (from TTAC) and within that the state requires that each grantee work on two indicators within each priority area. The indicators are selected by the state program. Grantees are free to select optional indicators. Also, within those indicators, grantees are free to select the activities that will best help them move the indicators. The state program provides a Program Guidelines Manual to assist them in selecting effective strategies, but the grantees have the freedom to select their own activities. However, there are core components that are essential for progress, including community mobilization, policy education, counter-marketing (paid & earned media) and surveillance and evaluation. So activities should fall within a change-strategy arena.
       
9. Virginia – (response from Foundation):
    
   1. We allocate $4 million or about 28% of our funding to local programs.
   2. We fund public health organizations, non-profits, schools, faith organizations, government organizations
   3. Our application process is competitive for program funds among all applicants
   4. We use a tiered process. For youth in 3rd grade/9 years old and higher, we have a university conduct standardized evaluation of programs using appropriate instruments designed for the various age groups. The instruments assess a variety of factors and measure statistically significant changes in five core measures: knowledge, intention to smoke, self-efficacy, actual tobacco use, and perceived benefits of remaining tobacco free. For youth programs that target younger children, we require our applicants to propose a local evaluation plan. These applicants typically use the instruments attached to the curriculum they choose from our approved list. Others hire evaluation contractors to design and implement their evaluation plan.
   5. We have two goals that applicants must choose from – one related to prevention, the other to cessation. All programs must use one of the approved programs on our Compendium of Tobacco Use Prevention Programs. The applicant creates a workplan of strategies and activities to carry out the program they selected. We allow a one-year period for the program.
       
10. West Virginia: West Virginia is currently funded at $7 million ($6.65M in State funds, $1.3M in CDC funds)
     
    1. Depends on the 'definition' of local programs, but WV funds approximately 26% of its total funding to local initiatives (i.e.- County Health Departments for Clean Indoor Air, 10 Regional Tobacco Prevention Coordinators, Local Mini-Grants)
    2. We fund both local public health agencies and other community programs
    3. The process is both competitive and non-competitive, as we do some 'continuation' funding for grantees whose programs we determine to be essential to our efforts
    4. We independently contract with the West Virginia University Prevention Research Center for evaluation, surveillance, and technical assistance ( spending 5 - 6% of our funding on evaluation. NOTE: CDC Best Practices recommends up to 10%)
    5. Both - we do have some funding to locals that is 'menued,' where projects and interventions can be chosen by the grantee.

Back to Table of Contents