Annual Program Evaluation
Annual Program Evaluation
Program evaluation is important for reviewing how a program is currently providing services and assessing how it could be improved. Programs that are not regularly undergoing a program evaluation are likely missing opportunities to improve their services and/or procedures.
We recommend that you learn about the basics of program evaluation by taking this course.
The OIB-TAC can help through resources on this site, or through individualized technical assistance. Contact us with questions related to
- updating program evaluation activities,
- setting up a program evaluation,
- interpreting information collected during a program evaluation, or
- understanding options for conducting program evaluation.
Best Practices
The OIB program manager will oversee an annual program evaluation that includes multiple measures of program efficiency and effectiveness. Evaluation measures may include a combination of file reviews, feedback from an advisory board, consumer input, empirical measures of consumer progress and outcomes, assessment of staff development and competencies, review of expenditures, programmatic outcome measures, agency procedures and records, and input from staff. Evaluations will include a description of how the agency used information from the previous program evaluation to make improvements in service delivery and/or its administration.
Program directors will conduct or obtain evaluations on an ongoing basis to monitor the program and promote quality service delivery. The program manager will work with administrators, staff, consumers, contractors, and other stakeholders, as appropriate, to collect information about consumer characteristics, outcomes, and satisfaction; services provided; referral sources; programmatic activities; service delivery methods; program governance; communication methods; staff characteristics and competencies; and financial management.
Program managers will use information from program evaluations to devise or revise programmatic goals and objectives and establish benchmarks for progress to promote effective and efficient service delivery.
Contracts with external vendors must include provisions for the contractor to collect and report specified data so that the program manager can monitor and evaluate the contractor’s performance and compliance with programmatic goals and procedures.
Acceptable Practices
Any agency that does not implement best practices will have a plan in place to move toward best practices that includes an expeditious timetable and benchmarks.
Unacceptable Practices
It is unacceptable to rely solely or primarily on measures of consumer satisfaction for program evaluation purposes, rather than using multiple assessment measures that include consumer outcomes.
Clarifying Comments
An outcome based program evaluation, also known as a summative evaluation, determines if the program is meeting its established goals. Process evaluations, or formative evaluations, assess progress of ongoing programs and provide staff with feedback regarding opportunities for improving service delivery, including what is and is not working and why. Outcome and process evaluation methods can be combined to generate a comprehensive program evaluation.
Program evaluation can confirm delivery of effective services and identify areas for improvement. Internal and external evaluations contribute to service delivery effectiveness and efficiency and in generating goals. Depending upon the agency’s strengths and areas of concern, the program evaluation may focus on specific programmatic components.
Program directors are encouraged to conduct quarterly file reviews on a sample of open cases to assess progress toward programmatic goals. Program evaluation plans, results, and response to results will be presented to and discussed with the advisory board and other stakeholders.
Consider using a logic model to facilitate a critical evaluation of the OIB program. A logic model can be a tool to
- generate alternative strategies to improve outcomes,
- clarify outcomes,
- improve communication about the process and outcomes,
- provide a focus for linking program components.
Logic models
- identify inputs (programmatic resources),
- activities (actions and tasks),
- outputs (countable products), and
- client outcomes (changes or accomplishments) that describe how a program works.
The W.K. Kellogg Foundation has created a report entitled Guiding Program Direction with Logic Models that accompanies the Logic Model Development Guide.
The Centers for Disease Control and Prevention website also provides extensive guidance and online resources.
Tips for Conducting a Telephone Satisfaction Survey
Use a script. Example: “Hi, I am calling from the Commission for the Blind to follow up on services provided. Would you have a few minutes to answer some questions that will help us know how well we are doing and how we might improve our services?”
Speak clearly and slow the pace of speaking if the consumer is having difficulty understanding you.
If the consumer says it is a bad time, ask if they could tell you when might be a good time for you to call back.
When a consumer agrees to take the survey, assure them that their answers are anonymous and that their name will not be associated with their answers. (This should be included in the script that would continue after they agree to participate.)
Since many older people receive services from numerous organizations, it is helpful to know the name of the teacher that worked directly with the consumer. Additionally, calling within a short time of the end of services is preferable so the person has not forgotten about which services were provided.
If the consumer askes, “How did you get my name and number?” Answer that you are an employee/volunteer who was provided names and phone numbers of individuals who recently completed their services with the commission for the blind for conducting this satisfaction survey.
The consumer may ask about their case closure, or be concerned that their case was closed and they do not know why. Assure them that they may call their teacher to ask these questions and suggest that they complete the survey based on the services they have received up to this point.
You may want to have a list of phone numbers for the teachers so you can provide this if the consumer wishes to contact their teacher with a question.
It may be helpful to have a list of organizations like the talking book library so you can provide this information if the consumer has a question about these commonly used programs. (e.g. The consumer might report they like the talking books, but they are receiving too many.)
The consumer may report that their case has been opened multiple times over a period of years. Instruct them to answer the questions based only on the services they have received most recently.
If a consumer gives a negative response to a question, encourage them to comment why they felt this way and record their answer on the survey.
Keep all information provided during these interviews confidential, do not share them outside of recording them on the survey.
Suggested checklist for Supervisors Observing a Site Visit
A Week Before Site Visit
___ Confirm dates and times for site visit.
___ Provide a short handout on the purpose of the site visit, if agency has more than one employee, encourage this document to be shared with all agency staff.
___ Share a projected schedule of events to occur during the site visit.
May include:
- Case file reviews
- Financial record reviews
- Interviews with staff
- Attend a sight loss support group
- Observe a group or individual instructional session
- Observe an intake or assessment
- Attend a staff meeting
Site Visit
___Greet and introduce yourself to staff, be prepared to explain the purpose of your visit in case they did not read the handout provided previously
___Be prepared to answer any questions about the purpose.
___Follow schedule of events, observe direct service without interference or comment.
What to look for:
___Are staff dressed professionally?
___Do staff conduct themselves with professionalism and respect?
___Do staff follow confidentiality protocol?
___Are forms being used appropriately?
___Are staff attentive and observant of consumer’s needs?
___Do staff allow consumers to do/demonstrate tasks?
___If not the first appointment, do staff review and follow up on instruction provided in previous appointment/s?
___Do staff provide appropriate resources in appropriate situations?
___Do staff provide information at an appropriate pace?
___Is documentation completed in a timely manner?
___Are background checks on file for the appropriate staff?
Tips
- Look at the case notes that result from the observed visit to see if information is accurately documented.
- Encourage staff by complimenting them if they do a good job with something.
- Ask staff about how they handle difficult situations
- Ask staff if they have any tips they would like to share with others across the state.
- Reinforce team work both with you and with other community agencies and other vendors.
- Provide feedback about how the vendor did in your review. (If they don’t know what they did well or what to improve, they probably won’t improve.)