|
|
CHAPTER III
PROGRAM EVALUATION
Program evaluation is a key element of the Department's strategic planning and implementation cycle. It is also a vital complement to performance measurement. Performance measurement provides information on whether a program is achieving its targets. Program evaluation tells not only what is happening in a program but why. It typically addresses a broader range of questions, is more sensitive to contextual factors, provides a richer, more complete picture of program performance (including impact), and offers greater insight about ways to improve.
All major Justice components have internal inspection and review processes designed to foster improved operations and compliance with applicable rules and regulations. Grant-making organizations (OJP and COPS) also conduct extensive financial and programmatic monitoring of the grants they award. In addition, the Department's independent Office of the Inspector General performs audits and inspections and recommends program improvements. These studies as well as those conducted by the General Accounting Office are important and useful sources of information for improving program operations and organizational effectiveness.
Although extremely useful, these various review and monitoring activities do not substitute for the conduct of formal, methodologically rigorous program evaluations that examine fundamental questions of program design, implementation and impact. Most of the formal program evaluations undertaken by the Department are focused on state and local assistance programs and are undertaken under the auspices of the Office of Justice Programs. However, several other components have evaluation programs. The FBI, for example, has in place a five-year schedule for conducting assessments of its major programs. The DEA has recently established a program evaluation unit within its Office of Inspections. BOP also has a formal evaluation program. INS is carrying out a number of evaluation studies.
For purposes of this plan, the Department relied on several studies affecting core Justice programs that have either been recently completed or are in progress. These include the ongoing study of the effects of the BOP residential drug treatment programs and the national level evaluations of Weed and Seed and other community-based programs. In addition to providing feedback on whether or not our strategies and programs are achieving the outcomes desired, program evaluations have been used to help improve program implementation and identify meaningful and appropriate performance measures.
The following table provides the current schedule of major program evaluations, organized by strategic goal. The Department's Annual Performance Plan will include any adjustments to this schedule for the performance year covered by the plan.
|
TABLE 3
| |||
|
Title |
Focus/Issues to be |
Methodology |
Estimated |
| Organized Crime Drug Enforcement Task Force (OCDETF) |
Congressionally-requested review of the OCDETF program to assess the effectiveness and efficiency with which it uses its resources. |
Program Evaluation: |
September 2000 |
| FBI's Counter-Terrorism Program |
Comprehensive review by the Office of Program Evaluation and Audit in the FBI's Inspections Division of the FBI's international and domestic terrorism programs. |
Program Evaluation: In progress |
September 2000 |
|
GOAL 2: PREVENT AND REDUCE CRIME AND VIOLENCE BY ASSISTING STATE, TRIBAL, LOCAL AND COMMUNITY-BASED PROGRAMS | |||
|
Title |
Focus/Issues to be Addressed |
Methodology Type of Evaluation |
Estimated Completion Date |
| Local Law Enforcement Block Grant (LLEBG) Evaluation | Examination of the utilization of block grant funding including decision-making models, level of innovation and effectiveness of the electronic application process. | Process Evaluation:
In progress Impact Evaluation:
|
November 2000 |
| Violence Against Women Act - Evaluation of Services, Training, Officers and Prosecutors (STOP) | Examination of STOP purpose areas including prosecution, law enforcement, victim services, services to Native Americans, and data improvement. | Impact Evaluation:
In progress |
September 2000 |
| Violence Against Women Act - Evaluation of Grants to Combat Violent Crimes Against Women on Campuses | Analysis of program characteristics and effectiveness. | Process Evaluation:
In progress |
Fall 2003 |
| Violence Against Women Act - Evaluation of the Rural Domestic Violence and Child Victimization Enforcement Grant Program | Analysis of program characteristics and effectiveness. | Process Evaluation:
In progress |
November 2001 |
| Violence Against Women Act - Evaluation of Grants to Encourage Arrest Policies Grant Program | Two-phased study of implementation and impact of program. | Process and Impact Evaluations:
In progress |
September 2001 |
| Violence Against Women Act - Evaluation of the Domestic Violence Victims Civil Legal Assistance Program | Study will document local programs funded, examine grantee planning and implementation, evaluate the need for and adequacy of special conditions pertaining to victim confidentiality and determine the effectiveness of these programs. | Process and Impact Evaluations: | Fall 2003 |
| Byrne - Evaluation of Tribal Strategies Against Violence Initiative | Study will document the processes used by tribal communities to develop and implement strategies to reduce violence at seven sites and analyze and document differences and similarities related to the development and implementation of local strategies. | Process Evaluation:
In progress |
December 2000 |
| Byrne - Evaluation of the Impact of Multi- jurisdictional Task Forces | This evaluation will attempt to assess the effectiveness of this approach to crime reduction. | Process Evaluation:
In progress Impact Evaluation:
|
September 2000 |
| COPS - Reducing Non-emergency Calls to 911: An Assessment of Four Approaches to Handling Citizen Calls for Service Evaluation | Program participation and implementation is being studied to assess the extent to which there is a reduction in the volume of inappropriate emergency calls for service as a result of 311. | Process Evaluation:
In progress |
December 2000 |
| Violent Offender Incarceration and Truth-in-Sentencing (VOI/TIS) Grants Evaluation | This evaluation will examine legislative actions, sentencing patterns, correctional populations, systems costs and crime rates in all 50 states. | Process Evaluation:
In progress |
March 2001 |
| COPS - School Resource Officer (SRO) Program Assessment | This national assessment will provide a description of various models implemented under the SRO Program and measurement of the impact of various SRO programs on selected indicators of school safety. | Process Evaluation:
FY 2000 Impact Evaluation:
|
February 2002 |
| Evaluation of the Juvenile Mentoring Program (JUMP) | The national evaluation will collect, manage, and analyze both quantitative and qualitative data, provide evaluation technical assistance, and develop reports. An interim evaluation report will be in September of 2000, and a final report in September of 2001. | Process and Outcome Evaluations:
In progress |
September 2001 |
| Evaluation of the Combating Underage Drinking Program | The national evaluation will determine how states and communities are using the Combating Underage Drinking funds and evaluate the impact of the program in communities. | Process and Impact Evaluations:
In progress |
August 2001 |
| Evaluation of the Safe Start Initiative | This evaluation will document and evaluate communities' efforts to prevent and reduce the impact of family and community violence on young children. | Process and Impact Evaluations:
In progress |
May 2005 |
| Evaluation of the Intensive Community-Based Aftercare Demonstration and Technical Assistance Program | This evaluation will assess the fidelity with which the program model is implemented and measure outcomes including several different measures of recidivism. | Process and Impact Evaluations:
In progress |
June 2001 |
| Evaluation of the Tribal Youth Program | This evaluation will examine the relationship between federal, state, local and tribal systems. | Process and Outcome Evaluations:
In progress |
June 2003 |
| National Evaluation of Title V | This evaluation will examine the viability and effectiveness of the comprehensive, locally-defined risk and protective factor focused prevention models in preventing delinquency in 12 sites in six states. | Process and Outcome Evaluations:
In progress |
July 2003 |
| Evaluation of the OJJDP Comprehensive Strategy | This process evaluation will document the lessons learned and factors associated with successful Comprehensive Strategy planning and implementation processes. | Process Evaluation:
In progress |
March 2001 |
| Evaluation of Community Assessment Centers | This evaluation will determine the degree to which two program sites, Denver, CO, and Orlando, FL, implemented the OJJDP Community Assessment Centers concept and its effect on the local juvenile justice and delinquency prevention systems and on the juveniles. | Process and Impact Evaluations:
In progress |
September 2003 |
| Safe Kids/Safe Streets Evaluation | This evaluation will (1) document and explicate the process of community mobilization, planning, and collaboration that has taken place before and during the SKSS awards and (2) determine the effectiveness of the implemented programs in achieving the goals of the SKSS program. | Process and Impact Evaluations:
In Progress |
October 2002 |
| Juvenile Accountability Incentive Block Grant Evaluation | A national study is being conducted to determine how the program has been administered, how the grants have been used, the types of programs funded as well as program access and utilization. The study will assess local satisfaction with training and technical assistance and attitudes toward the program. | Process Evaluation:
In Progress |
December 2002 |
| Victims of Crime Act (VOCA) Program Evaluation | Evaluation of victims assistance and compensation programs. | Process Evaluation:
In progress |
February 2001 |
| Victims of Crime | Evaluation of victim needs and access to assistance. | Process Evaluation:
In progress |
March 2002 |
| Community- Oriented Policing Services Program - National Evaluation | Ongoing evaluation of implementation including allocation of resources, distribution of funds, organizational structure, local hiring of officers, acquisition of technology and adoption of the COPS model by local jurisdictions. | Process and Impact Evaluations:
In progress |
October 2000 |
| COPS-Methamphetamine Project | Assessment of initiative to support enforcement, intervention and prevention efforts using community policing strategies in jurisdictions with pressing methamphetamine problems. | Process and Outcome Assessment:
In progress |
March 2001 |
| School - Based Partnerships | Local and national level evaluation of program impact, including effectiveness of applying problem solving model to specific crime and disorder problems in schools. | Impact Evaluation:
In progress |
Local: May 2001
National: December 2001 |
| Regional Community Policing Institutes | Local and national level evaluation of implementation and effects of RCPI training on police officer trainees and police agencies. | Process and Impact Evaluations:
In progress |
June 2001 |
| Problem Solving Partnerships - Program Assessment | Local and national level evaluation of impact of applying problem solving model to specific crime and disorder problems. | Impact Evaluation:
In progress |
Local: Ongoing
National: June 2000 |
| Advancing Community Policing Demonstration Centers | Local evaluations to assess active community policing laboratories and tools needed to disseminate information and assistance to other law enforcement agencies. | Process Evaluation: In progress | January 2001 |
|
GOAL 4: FAIRLY AND EFFECTIVELY ADMINISTER THE IMMIGRATION AND NATURALIZATION LAWS OF THE UNITED STATES | |||
|
Title |
Focus/Issues to be Addressed |
Methodology |
Estimated |
| New Immigrant Survey | Interagency-sponsored longitudinal study to develop program and policy-relevant information for government and public use. Study will address impact of recent immigrants on future immigration flows and naturalization, emigration, and criminal justice as well as provide data over time on the income, health, employment, net worth, and social participation of recent immigrants. | Telephone and in-person survey Process and Impact Evaluation: |
December 2004 |
| Evaluation of the IIRIRA Sponsorship Provisions | Study to determine the impact of the new sponsorship provisions on immigrant flows and characteristics, the ability of U.S. residents to bring close relatives to this country, and of the ability of benefit agencies to enforce the sponsor's commitment of support. | Administrative data and record review
Impact Evaluation:
|
March 2001 with follow up |
| Welfare Reform, the Economic and Health Status of Immigrants | Interagency-sponsored study to evaluate the impacts of welfare reform on different types of aliens in Los Angeles and New York City, including coping mechanisms and continued use of benefit programs. Results will have program and policy-relevant information for INS and the public. | Telephone and in-person survey
Impact Evaluation:
|
October 2001 |
| Evaluation of the IIRIRA Basic Pilot | Congressionally-mandated evaluation to determine how well employers participating in the pilot verify the status of all newly hired workers and the impact of the program on workload, cost, discrimination, privacy, and other factors. | Mail and in-person survey
and administrative data and record review
Process and Impact Evaluations:
|
September 2001 |
| Evaluation of the IIRIRA Citizen Attestation Pilot | Congressionally-mandated evaluation to determine how well employers participating in the pilot verify the status of newly hired alien workers and the impact of the program on workload, cost, discrimination, privacy, and other factors. | Mail survey and
administrative data and record review
Process and Impact Evaluations:
|
September 2001 |
| Evaluation of the IIRIRA Machine Readable Document Pilot | Congressionally-mandated evaluation to determine how well employers participating in the pilot verify the status of all newly hired workers and the impact of the program on workload, cost, discrimination, privacy, and other factors. | Mail and in-person survey &
administrative data and record review
Process and Impact Evaluations:
|
September 2002 |
| Evaluation of the New Legally- Enforceable Affidavit of Support | Evaluation of how well sponsors follow the requirements of the new affidavit of support and the adequacy of government decisionmaking on these forms. | Administrative data and record review
Process Evaluation:
|
June 2001 |
| Evaluation of the Immigrant Investor Regional Center Pilot Program | Evaluation of how well regional centers approved under the pilot program to attract immigrant investors engaging in certain export activities are working and what changes should be made. | Mail and in-person survey
Process and Impact Evaluations:
|
September 2001 |
| Border Coordination Initiative (BCI) Program Evaluation | An evaluative study to determine whether or not the BCI core mission initiatives are meeting the predetermined goals and objectives described in the original program plans of September 1998. | Analysis of workload and performance
data from INS' and Customs' databases, questionnaires, surveys, case studies
Process and Impact Evaluations:
|
December 2000 |
| Southwest Border Strategy Evaluation | Evaluation of (1) Are the strategies we are pursuing achieving their intended effects? (2) Are there unanticipated effects? (3) Are our goals realistic? (4) Are we collecting the right performance data? and, (5) Are there alternative strategies that might prove more successful? | Independent, non-governmental
research augmented by analyses from ongoing INS evaluations
In progress |
December 2000 |
| Customer Surveys | Survey of U.S. residents asking about the speed and courtesy of the inspections process at airports and land border crossing points; part of a multiagency survey spearheaded by the National Partnership for Reinventing Government. | Telephone survey
Process Evaluation: |
October 2000 |
| Evaluation of Use of Expedited Removal in the Institutional Removal Program | Study to determine whether use of Expedited Removal (ER) procedures with selected alien detainees in federal prisons can speed their removal. If successful, this use of ER would enhance the Service's ability to remove criminal aliens from the interior of the United States. | Administrative data and record review
Process and Impact Evaluations:
|
March 2001 |
|
GOAL 5: PROTECT AMERICAN SOCIETY BY PROVIDING FOR THE SAFE, HUMANE AND SECURE CONFINEMENT OF PERSONS IN FEDERAL CUSTODY | |||
|
Title |
Focus/Issues to be Addressed |
Methodology and
Type of Evaluation
|
Estimated |
| BOP Taft Correctional Institution in California | Cost effectiveness and operational success of government-owned but privately managed facility. | Process/Cost Effectiveness Evaluation:
In progress |
March 2003 |
| BOP Residential Drug Treatment Program | Conducted with funding and assistance from the National Institute of Drug Abuse, this study focuses on the effectiveness of residential drug treatment programs by monitoring inmates for up to three years following their release. | Impact Evaluation: In progress |
September 2000
[Preliminary report issued February 1998] |
| Private Medical Services Provided at BOP Facilities in Beaumont, Texas | Cost effectiveness and operational success of private health care services. | Process/Cost Effectiveness Evaluation:
In progress |
March 2003 |
| Assumption of Outside Medical Care Responsibilities for USMS Prisoners | Assessment of financial and operational impact on BOP of assuming these responsibilities. | Analysis of Pilot Program Data:
In progress |
September 2000 |
FY 2000 -- 2005 Strategic Plan
U.S. Department of Justice