11: Case Tracking and Program Evaluation

Program evaluation is the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs. The focus is on a particular program to systematically produce trustworthy and credible information that is action-oriented to assist in decision making or improve a program. Program evaluation should be part of a program’s development. Below is a graphic depiction of where evaluation fits into the development of an MDT program. 

One of the first steps to take when developing a program is to create a logic model. A logic model is simply a visual representation that depicts and describes how a program will work. A logic model is your theory of change. The factors in the logic model form the foundation for the evaluation. 

A logic model will be used throughout the life of a program and across different types of evaluation. The logic model helps ensure that you are clear from the beginning what the expected objectives and outcomes will be for the MDT and what specific changes are expected for what specific population. Below are two sample logic models. 

Once your logic model is completed, an evaluation plan can be developed. As presented in the graphic below, the two broad categories of evaluation include Formative Evaluation and Summative Evaluation. The difference between these two categories is the developmental stage of your program. Each of these broad categories has two types of evaluation that fall within each category, with distinct purposes. However, the four evaluation types relate to one another in that each one builds upon the other. In practice, often a program will utilize only one type of evaluation, but even if you choose to use just one type, it is useful to understand how these four evaluation types relate to one another. 

Below is a brief synopsis of each type of evaluation, followed by a more fulsome discussion of each.

Formative Evaluation

Formative evaluation concerns learning about your program before it has been developed or while it is a new program. As a cooking analogy, formative evaluation is when the cook tastes the soup, decides something needs to be added, and changes the soup to make it better, to improve it, before serving it to guests.

The two types of evaluation that fall under formative evaluation are:

• Needs Assessment
  • Needs assessment is a process for determining an organization's needs and gaps in knowledge, practices, or skills prior to initiating the program, as well as what resources are available in your community. Using the soup analogy, the question might be Have you looked at the recipe and then what you have in your house, and then developed a grocery list so you have everything you need to begin cooking?
• Process Evaluation
  • Here, once the program is operational but still in the early stage of development, you are asking Is the program being implemented the way I intended? Is case review being conducted according to plan? Is the process for referral of cases to the MDT being followed? Using the soup metaphor, the question might be Is the cook actually following the recipe? If the recipe calls for 1/2 t of salt, is the cook putting in 1/2 t of salt?

Summative Evaluation 

Summative evaluation is conducted after the program is well-established. In the soup analogy, summative evaluation is when the guests taste the soup. The two types of evaluation that fall under formative evaluation are:   

• Outcome Evaluation
  • Now you are asking Did the program achieve its desired outcomes?, which are outlined in the logic model. You might ask As a result of the MDT, what percentage of cases are services coordinated for older adults?; or As a result of the MDT, are more cases being referred for prosecution? Using the soup analogy, the question might be Did the people eating the soup like the soup; Did the soup satiate their hunger?
• Impact Evaluation
  • Finally, with an impact evaluation the question is What is the difference between what happened with the program compared to what happened without the program? The added value of an impact evaluation is that it lets you assess the cause of the change and therefore impact evaluations always involve the question “compared to what?” Using the soup analogy, the question might be What impact did eating the soup have on your guest’s health compared to not eating the soup? 

The material below provides additional detail for each of the four types of evaluation.

A process evaluation documents the process or implementation of a program. Here you are looking for evidence of how your program has been implemented in reality, whether it is reaching the intended audience, and is producing desired outputs in comparison to your intentions. The focus is on identifying gaps for the purpose of program improvement. To later engage in an outcome evaluation, it is imperative to understand that the program is being implemented as intended and therefore a process evaluation precedes an outcome evaluation.  

A process evaluation requires a deep dive into the program. Start by reviewing the activities and output components of the logic model (the “as intended” part of the process). Then take a walk through the program, mapping the program, and asking people about what they think about the program, for example, MDT member satisfaction. Note that you are not tying these activities to outcomes yet.  

Some questions that could be addressed by a process evaluation include:

• Whether program activities were accomplished
• Assess the quality of program components
• How well program activities were implemented
• How external factors influenced the program

Three Areas of Inquiry

The process evaluation will involve three areas of inquiry. 

1) Program Environment

Here you are describing the operating environment to enable other communities to determine whether they might achieve similar results. Factors to consider might include:

• Significant changes in the environment
• Macro-events that impact the program

2) Implementation Process

As discussed above, here you are describing the implementation process and this would involve elements such as:

• Interaction among participants (MDT members)
• The extent of participation (for example, by various agencies, units, and individuals)
• Training provided for members
• Interaction among participants and others in the community who were not involved in planning and implementing the program

3) Assessing Outputs

Here you are asking whether expected outputs were actually produced. Examples of outputs include:

• Number of cases referred to the MDT
• Number and category of case types
• Number of case review meetings
• Number and types of training provided to the MDT
• Number of joint interviews

Based on these sources of data, including interviews with key stakeholders, a set of recommendations for program improvement will emerge.

Client Satisfaction

Once your process evaluation is completed, you may want to implement a client satisfaction survey, typically an intermediary step between the process evaluation and the outcome evaluation (it is not an outcome because it does not measure the results achieved).  

The challenge associated with client satisfaction surveys is that clients do not always perceive the “process” the way professionals within the systems perceive the process. While the MDT may perceive the investigation, services, and case review as a seamless process, clients may want to rate those activities individually. Client satisfaction surveys may address whether clients:

• Feel satisfied with their interactions with various team members
• Feel satisfied with the intervention

MDT Functioning and Satisfaction

At some point after the MDT has been established, you will want to evaluate the functioning and effectiveness of your MDT. You may want to focus your evaluation on the elements of a successful MDT. These characteristics will need to be quantified for evaluation purposes. For example, items addressing team trust and cohesion might include: 

• The team has a shared interdisciplinary team philosophy
• The team has honest communication
• The team readily shares knowledge (as opposed to information), for example, through informal and formal cross training
• MDT members are comfortable exchanging information
• There is a sense of collegiality among MDT members
• MDT members share ideas and experiences
• There is a shared belief that working as a team leads to better client outcomes
• Mutual support is provided by MDT members
• MDT members feel trust and respect
• MDT members complement each other’s functions
• MDT members share resources
• MDT members enhance each other’s capacity to address the needs of their clients 

Impact evaluations are considered the gold standard of program evaluation, but they are also the most challenging to implement and typically require the expertise of an evaluation professional. An impact evaluation answers the question, What is the difference between what happened with the program compared to what happened without the program? If your objectives have been met, you can say the goals of the program have been met, but that does not necessarily mean your program caused the outcomes. To determine causality, there must be a comparison of some sort. 

The two designs typically used in an impact evaluation are 1) experimental designs and 2) quasi-experimental designs.

Experimental Designs: Comparison of Treatment and Control Group using a Randomized Control Trial

In a simple experimental design, there is a treatment group and a control group. For example, the treatment group would be clients whose cases are reviewed by an MDT and the control group would be a similarly situated group of clients whose case is NOT reviewed by the MDT (their case is processed as it normally would through APS or law enforcement). 

The critical element in an experimental design is random assignment. That is, any given individual has a 50/50 chance of being assigned to the treatment group or the control group. The purpose behind random assignment is that all differences between the two groups are eliminated. When the comparison between the two groups is made, e.g., the number of cases referred for prosecution, the difference between the two groups is truly due to the treatment and not to unknown factors. Random assignment is what allows you to talk about a program causing the desired change. 

Quasi-Experimental: Comparison of Treatment and Comparison Group using a Naturally Occurring Setting

In a quasi-experimental design, there is a treatment group and a naturally occurring comparison group. For example, because of jurisdiction or other reasons, some clients would naturally go to an Elder Justice Forensic Center (EJFC) while other clients would naturally go to the police station (or APS). In this situation, we are not manipulating where clients are seen, rather, we are just observing naturally occurring differences.  

Quasi-experimental designs are much easier to implement than designs involving random assignment. However, there are multiple factors that could account for any differences found between the two groups, reducing our confidence in causality. It may be that the area around the police station is more economically disadvantaged so if older adults seen at the EJFC fare better than those seen at a police station, it is much harder to attribute differences to the treatment (the MDT) because of these “confounding” factors (economic disadvantage) that differ between these two communities. 

Timing and Frequency of Administration of Surveys

Consider the timing and frequency of administering the survey(s). Evaluation administered at different points in the process will capture different experiences. For example, a client satisfaction evaluation administered immediately after the case has closed will capture different information than a client satisfaction survey administered six months after the case has closed.

The MDT will need to decide how frequently to administer various surveys. For example, an MDT survey could be administered annually, every six months, or after every meeting.

Instruments

There are a number of surveys that might be adapted for the purposes described above. However, there are no empirically validated measures of client satisfaction in the context of elder abuse MDTs. Where it is feasible, consider partnering with a university faculty member or graduate student who might create a survey instrument (using survey science) for your purposes.

Data Collection Plan

As part of your evaluation plan, a plan for collecting and storing data will need to be developed to ensure information is being captured that allows the evaluation questions to be answered. Several data collection plans exist. 

Utilize the information obtained from these evaluation efforts to improve your program. It may be hard to hear that all of your efforts fail to result in perfect outcomes, but keep in mind that improvement is always possible and is definitely desirable.

While starting an elder abuse MDT presents many layers of complexity, by taking the issues one at a time, your community can develop a robust and useful team that better serves the needs of older adults who have experienced abuse.  In the toolkit that accompanies this guide, there are many sample documents that you may adapt for your purposes.  Additionally, the MDT Technical Assistance Center (MDT TAC) provides consultations to assist in problem solving and identifying useful resources as you develop your team. Begin by identifying a few passionate individuals who have the ability to build relationships between agencies, identify the needs of your community, lean on the insights of the various disciplines and agencies available to you, keep the focus on ways you can best serve your clients, and let the good work begin.