Annual Program Evaluation

Any agency that does not implement best practices will have a plan in place to move toward best practices that includes an expeditious timetable and benchmarks. 

It is unacceptable to rely solely or primarily on measures of consumer satisfaction for program evaluation purposes, rather than using multiple assessment measures that include consumer outcomes. 

An outcome based program evaluation, also known as a summative evaluation, determines if the program is meeting its established goals. Process evaluations, or formative evaluations, assess progress of ongoing programs and provide staff with feedback regarding opportunities for improving service delivery, including what is and is not working and why. Outcome and process evaluation methods can be combined to generate a comprehensive program evaluation.  

Program evaluation can confirm delivery of effective services and identify areas for improvement. Internal and external evaluations contribute to service delivery effectiveness and efficiency and in generating goals. Depending upon the agency’s strengths and areas of concern, the program evaluation may focus on specific programmatic components. 

Program directors are encouraged to conduct quarterly file reviews on a sample of open cases to assess progress toward programmatic goals. Program evaluation plans, results, and response to results will be presented to and discussed with the advisory board and other stakeholders. 

Use a script. Example: “Hi, I am calling from the Commission for the Blind to follow up on services provided. Would you have a few minutes to answer some questions that will help us know how well we are doing and how we might improve our services?” 

Speak clearly and slow the pace of speaking if the consumer is having difficulty understanding you. 

If the consumer says it is a bad time, ask if they could tell you when might be a good time for you to call back. 

When a consumer agrees to take the survey, assure them that their answers are anonymous and that their name will not be associated with their answers. (This should be included in the script that would continue after they agree to participate.) 

Since many older people receive services from numerous organizations, it is helpful to know the name of the teacher that worked directly with the consumer. Additionally, calling within a short time of the end of services is preferable so the person has not forgotten about which services were provided. 

If the consumer askes, “How did you get my name and number?” Answer that you are an employee/volunteer who was provided names and phone numbers of individuals who recently completed their services with the commission for the blind for conducting this satisfaction survey. 

The consumer may ask about their case closure, or be concerned that their case was closed and they do not know why. Assure them that they may call their teacher to ask these questions and suggest that they complete the survey based on the services they have received up to this point. 

You may want to have a list of phone numbers for the teachers so you can provide this if the consumer wishes to contact their teacher with a question. 

It may be helpful to have a list of organizations like the talking book library so you can provide this information if the consumer has a question about these commonly used programs. (e.g. The consumer might report they like the talking books, but they are receiving too many.) 

The consumer may report that their case has been opened multiple times over a period of years. Instruct them to answer the questions based only on the services they have received most recently. 

If a consumer gives a negative response to a question, encourage them to comment why they felt this way and record their answer on the survey. 

Keep all information provided during these interviews confidential, do not share them outside of recording them on the survey. 

A Week Before Site Visit 

___ Confirm dates and times for site visit.

___ Provide a short handout on the purpose of the site visit, if agency has more than one employee, encourage this document to be shared with all agency staff.

___ Share a projected schedule of events to occur during the site visit.

May include: 

• Case file reviews 
• Financial record reviews 
• Interviews with staff 
• Attend a sight loss support group 
• Observe a group or individual instructional session 
• Observe an intake or assessment 
• Attend a staff meeting 

Site Visit 

___Greet and introduce yourself to staff, be prepared to explain the purpose of your visit in case they did not read the handout provided previously

___Be prepared to answer any questions about the purpose. 

___Follow schedule of events, observe direct service without interference or comment.

What to look for:

___Are staff dressed professionally? 

___Do staff conduct themselves with professionalism and respect? 

___Do staff follow confidentiality protocol? 

___Are forms being used appropriately? 

___Are staff attentive and observant of consumer’s needs? 

___Do staff allow consumers to do/demonstrate tasks? 

___If not the first appointment, do staff review and follow up on instruction provided in previous appointment/s? 

___Do staff provide appropriate resources in appropriate situations? 

___Do staff provide information at an appropriate pace? 

___Is documentation completed in a timely manner? 

___Are background checks on file for the appropriate staff? 

Tips 

• Look at the case notes that result from the observed visit to see if information is accurately documented. 
• Encourage staff by complimenting them if they do a good job with something. 
• Ask staff about how they handle difficult situations 
• Ask staff if they have any tips they would like to share with others across the state. 
• Reinforce team work both with you and with other community agencies and other vendors. 
• Provide feedback about how the vendor did in your review. (If they don’t know what they did well or what to improve, they probably won’t improve.)