| c | ||
|
|
CHAPTER III
PROGRAM EVALUATION
Program evaluation is a key element of the Department's strategic planning and implementation cycle. It is also a vital complement to performance measurement. Performance measurement provides information on whether a program is achieving its targets. Program evaluation tells not only what is happening in a program but why. It typically addresses a broader range of questions, is more sensitive to contextual factors, provides a richer, more complete picture of program performance (including impact), and offers greater insight about ways to improve.
All major Justice components have internal inspection and review processes designed to foster improved operations and compliance with applicable rules and regulations. Grant-making organizations (OJP and COPS) also conduct extensive financial and programmatic monitoring of the grants they award. In addition, the Department's independent Office of the Inspector General performs audits and inspections and recommends program improvements. These studies as well as those conducted by the General Accounting Office are important and useful sources of information for improving program operations and organizational effectiveness.
Although extremely useful, these various review and monitoring activities do not substitute for the conduct of formal, methodologically rigorous program evaluations that examine fundamental questions of program design, implementation and impact. Most of the formal program evaluations undertaken by the Department are focused on state and local assistance programs and are undertaken under the auspices of the Office of Justice Programs. However, several other components have evaluation programs. The FBI, for example, has in place a 5-year schedule for conducting assessments of its major programs. The DEA has recently established a program evaluation unit within its Office of Inspections. BOP also has a formal evaluation program. INS is carrying out a number of evaluation studies.
For purposes of this plan, the Department relied on several studies affecting core Justice programs that have either been recently completed or are in progress. These include the ongoing study of the effects of the BOP residential drug treatment programs, the national level evaluations of Weed and Seed and other community-based programs, and the evaluation of the effectiveness of the southwest border control strategy of the INS. In addition to providing feedback on whether or not our strategies and programs are achieving the outcomes desired, program evaluations have been used to help improve program implementation and identify meaningful and appropriate performance measures.
The following table provides the current schedule of major program evaluations, organized by strategic goal. The Department's Annual Performance Plan will include any adjustments to this schedule for the performance year covered by the plan.
|
TABLE 3
| |||
|---|---|---|---|
|
Title |
Focus/Issues to be |
Methodology |
Estimated |
| FBI's Counterterrorism Program |
Comprehensive review by the Office of Program Evaluation and Audit in the FBI's Inspections Division of the FBI's international and domestic terrorism programs. |
Program Evaluation: |
December 2001 |
|
GOAL 2: ENFORCE FEDERAL CRIMINAL LAWS | |||
|---|---|---|---|
|
Title |
Focus/Issues to be Addressed |
Methodology Type of Evaluation |
Estimated Completion Date |
| Organized Crime Drug Enforcement Task Force (OCDETF) | Congressionally-requested review of the OCDETF program to assess the effectiveness and efficiency with which it uses its resources. | Process Evaluation:
Completed. Report sent to Congress |
October 2001 |
|
GOAL 3: PREVENT AND REDUCE CRIME AND VIOLENCE BY ASSISTING STATE, TRIBAL, LOCAL AND COMMUNITY-BASED PROGRAMS | |||
|---|---|---|---|
|
Title |
Focus/Issues to be Addressed |
Methodology Type of Evaluation |
Estimated Completion Date |
| Local Law Enforcement Block Grant (LLEBG) Evaluation | Examination of the utilization of block grant funding including decision-making models, level of innovation and effectiveness of the electronic application process. | Process Evaluation: In progress
Impact Evaluation:
|
March 2002 |
| Violence Against Women Act - Evaluation of Services, Training, Officers and Prosecutors (STOP) | Examination of STOP purpose areas including prosecution, law enforce-ment, victim services, services to Native Americans, and data improvement. | Impact Evaluation:
Completed. Report sent to Congress. |
September 2000 |
| Violence Against Women Act - Evaluation of Grants to Combat Violent Crimes Against Women on Campuses | Analysis of program characteristics and effectiveness. | Process and Impact Evaluations:
In progress |
Fall 2003 |
| Violence Against Women Act - Evaluation of the Rural Domestic Violence and Child Victimization Enforcement Grant Program | Analysis of program characteristics and effectiveness. | Process and Impact Evaluations:
In progress |
November 2001 |
| Violence Against Women Act - Evaluation of Grants to Encourage Arrest Policies Grant Program | Two-phased study of implementation and impact of program. | Process and Impact Evaluations:
Report in peer review. |
June 2002 |
| Violence Against Women Act - Evaluation of the Domestic Violence Victims Civil Legal Assistance Program | Study will document local programs funded, examine grantee planning and implementation, evaluate the need for and adequacy of special conditions pertaining to victim confidentiality and determine the effectiveness of these programs. | Process and Impact Evaluations:
In progress |
Fall 2003 |
| Byrne - Evaluation of Tribal Strategies Against Violence Initiative | Study will document the processes used by tribal communities to develop and implement strategies to reduce violence at seven sites and analyze and document differences and similarities related to the development and implementation of local strategies. | Process Evaluation: Report in peer review. | June 2002 |
| Byrne - Evaluation of the Impact of Multi- jurisdictional Task Forces | This evaluation will attempt to assess the effectiveness of this approach to crime reduction. | Process Evaluation: In progress
Impact Evaluation:
|
December 2001 |
| COPS - Alternatives to 911: An Assessment of Four Approaches to Handling Citizen Calls for Service Evaluation | Program participation and implementation is being studied to assess the extent to which there is a reduction in the volume of inappropriate emergency calls for service as a result of 311. | Process Evaluation: In progress | December 2001 |
| Violent Offender Incarceration and Truth-in-Sentencing (VOI/TIS) Grants Evaluation | This evaluation will examine legislative actions, sentencing patterns, correctional populations, systems costs and crime rates in all 50 states. | Process Evaluation: Report in peer review. | June 2002 |
| COPS - School Resource Officer (SRO) Program Assessment | This national assessment will provide a description of various models implemented under the SRO Program and measurement of the impact of various SRO programs on selected indicators of school safety. | Process Evaluation:
In progress |
September 2002 |
| Evaluation of the Juvenile Mentoring Program (JUMP) | The national evaluation will collect, manage, and analyze both quantitative and qualitative data, provide evaluation technical assistance, and develop reports. Final report will be issued in September of 2002. | Process and Outcome Evaluations: In progress | September 2002 |
| Evaluation of the Combating Underage Drinking Program | The national evaluation will determine how states and communities are using the Combating Underage Drinking funds and evaluate the impact of the program in communities. | Process and Impact Evaluations: In progress | January 2002 |
| Evaluation of the Safe Start Initiative | This evaluation will document and evaluate communities' efforts to prevent and reduce the impact of family and community violence on young children. | Process and Impact Evaluations: In progress | May 2005 |
| Evaluation of the Tribal Youth Program | This evaluation will examine the relationship between federal, state, local and tribal systems. | Process and Outcome Evaluations: In progress | June 2003 |
| National Evaluation of Title V | This evaluation will examine the viability and effectiveness of the comprehensive, locally-defined risk and protective factor focused prevention models in preventing delinquency in 12 sites in six states. | Process and Outcome Evaluations: In progress | July 2003 |
| Evaluation of the OJJDP Comprehensive Strategy | This process evaluation will document the lessons learned and factors associated with successful Comprehensive Strategy planning and implementation processes. | Process Evaluation: In progress | December 2001 |
| Evaluation of Community Assessment Centers | This evaluation will determine the degree to which two program sites, Denver, CO, and Orlando, FL, implemented the OJJDP Community Assessment Centers concept and its effect on the local juvenile justice and delinquency prevention systems and on the juveniles. | Process and Impact Evaluations:
In progress |
September 2003 |
| Safe Kids/Safe Streets Evaluation | This evaluation will (1) document and explicate the process of community mobilization, planning, and collaboration that has taken place before and during the SKSS awards and (2) determine the effectiveness of the implemented programs in achieving the goals of the SKSS program. | Process and Impact Evaluations:
In Progress |
October 2002 |
| Juvenile Accountability Incentive Block Grant Evaluation | A national study is being conducted to determine how the program has been administered, how the grants have been used, the types of programs funded as well as program access and utilization. The study will assess local satisfaction with training and technical assistance and attitudes toward the program. | Process Evaluation:
In Progress |
December 2002 |
| Victims of Crime Act (VOCA) Program Evaluation | Evaluation of victims assistance and compensation programs. | Process Evaluation:
In progress |
March 2002 |
| Victims of Crime | Evaluation of victim needs and access to assistance. | Process Evaluation:
In progress |
March 2002 |
| Community-Oriented Policing Services Program - National Evaluation | Ongoing evaluation of implementation including allocation of resources, distribution of funds, organizational structure, local hiring of officers, acquisition of technology and adoption of the COPS model by local jurisdictions. | Process and Impact Evaluations: Completed. Report is on DOJ web site. | September 2000 |
| COPS-Methamphetamine Project | Assessment of initiative to support enforcement, intervention and prevention efforts using community policing strategies in jurisdictions with pressing methamphetamine problems. | Process and Outcome Assessment: Completed. Report is in peer review. | March 2001 |
| School - Based Partnerships | Local and national level evaluation of program impact, including effectiveness of applying problem solving model to specific crime and disorder problems in schools. | Impact Evaluation:
In progress |
Local: September 2002
National: September 2002 |
| Regional Community Policing Institutes | Local and national level evaluation of implementation and effects of RCPI training on police officer trainees and police agencies. | Process and Impact Evaluations: Completed | June 2001 |
| Problem Solving Partnerships - Program Assessment | Local and national level evaluation of impact of applying problem solving model to specific crime and disorder problems. | Impact Evaluation: Completed. Report is on DOJ web site. | Local: Ongoing
National: December 2000 |
| Advancing Community Policing Demonstration Centers | Local evaluations to assess active community policing laboratories and tools needed to disseminate information and assistance to other law enforcement agencies. | Process Evaluation: In progress | September 2002 |
| Office of the Police Corps and Law Enforcement Education - Police Corps | Research to provide the OPCLEE with a better understanding of the factors surrounding a police department's hiring processes, the integration of Police Corps trained officers into police departments, and the impact of these officers on the police agency and the communities they serve. OPCLEE will use these findings to assist in the development of training curriculums and retention policies options. | Impact Evaluation: Solicitation is being developed | TBD |
| OVC and BJA Victim Assistance in Indian Country; Developing and Enhancing Tribal Courts; and Indian Alcohol and Substance Abuse | Research unique challenges of developing and implementing federally-funded Victim Assistance in Indian Country programs; and developing, implementing, and enhancing tribal courts. Provide information on methods and efforts used to reduce alcohol and substance abuse in Indian Country. | Process Evaluation: Solicitation is being developed | TBD |
| BJA - Auto Theft Prevention; Watch Your Car Program | Assessment may include: 1) a review of the programmatic activities of states currently administering the program; 2) a review of the best practices; 3) a literature search to evaluative findings of other auto theft prevention initiatives; 4) the collection of programmatic and technological information that has developed from the implementation of these programs; or 5) any other steps or methods needed to collect information. | Process Evaluation: Solicitation is being developed | TBD |
|
GOAL 5: FAIRLY AND EFFECTIVELY ADMINISTER THE IMMIGRATION AND NATURALIZATION LAWS OF THE UNITED STATES | |||
|---|---|---|---|
|
Title |
Focus/Issues to be Addressed |
Methodology |
Estimated |
| New Immigrant Survey | Interagency-sponsored longitudinal study to develop program and policy-relevant information for government and public use. Study will address impact of recent immigrants on future immigration flows and naturalization, emigration, and criminal justice as well as provide data over time on the income, health, employment, net worth, and social participation of recent immigrants. | Telephone and in-person survey
Process and Impact Evaluation: |
December 2004 |
| Evaluation of the IIRIRA Sponsorship Provisions | Study to determine the impact of the new sponsorship provisions on immigrant flows and characteristics, the ability of U.S. residents to bring close relatives to this country, and of the ability of benefit agencies to enforce the sponsor's commitment of support. | Administrative data and record review
Impact Evaluation:
|
May 2002 |
| Welfare Reform, the Economic and Health Status of Immigrants | Interagency-sponsored study to evaluate the impacts of welfare reform on different types of aliens in Los Angeles and NYC, including coping mechanisms and continued use of benefit programs. Results will have program and policy-relevant information for INS and the public. | Telephone and in-person survey
Impact Evaluation:
|
May 2002 |
| Evaluation of the IIRIRA Basic Pilot | Congressionally-mandated evaluation to determine how well employers participating in the pilot verify the status of all newly hired workers and the impact of the program on workload, cost, discrimination, privacy, and other factors. | Mail and in-person survey
and administrative data and record review
Process and Impact Evaluations: In progress |
November 2001 |
| Evaluation of the IIRIRA Citizen Attestation Pilot | Congressionally-mandated evaluation to determine how well employers participating in the pilot verify the status of newly hired alien workers and the impact of the program on workload, cost, discrimination, privacy, and other factors. | Mail survey and
administrative data and record review
Process and Impact Evaluations:
|
May 2002 |
| Evaluation of the IIRIRA Machine Readable Document Pilot | Congressionally-mandated evaluation to determine how well employers participating in the pilot verify the status of all newly hired workers and the impact of the program on workload, cost, discrimination, privacy, and other factors. | Mail and in-person survey and
administrative data and record review
Process and Impact Evaluations:
|
June 2002 |
| Evaluation of the New Legally- Enforceable Affidavit of Support | Evaluation of how well sponsors follow the requirements of the new affidavit of support and the adequacy of government decision- making on these forms. | Administrative data and record review
Process Evaluation:
|
May 2002 |
| Southwest Border Strategy Evaluation | Evaluation of (1) Are the strategies we are pursuing achieving their intended effects? (2) Are there unanticipated effects? (3) Are our goals realistic? (4) Are we collecting the right perfor-mance data? and, (5) Are there alternative strategies that might prove more successful? | Independent,
non-governmental research augmented by analyses from ongoing
INS evaluations
In progress |
November 2001 |
| Customer Surveys | Survey of U.S. residents asking about the speed and courtesy of the inspections process at airports and land border crossing points; part of a multi-agency survey spearheaded by National Partnership for Reinventing Government. | Telephone survey
Process Evaluation:
|
October 2000 |
| Evaluation of Fraudulent Practitioner Pilot | To encourage undocumented immigrants who have been defrauded by unscrupulous immigration practitioners to come forward in a non-threatening setting. | In-person survey and
administrative record review.
Process Evaluation:
|
April 2003 |
| Evaluation of Alternatives to Detention | To explore alternative types of detention or alternatives to detention for immigration violators. | In-person survey and
administrative record review.
Process Evaluation:
|
April 2003 |
|
GOAL 6: PROTECT AMERICAN SOCIETY BY PROVIDING FOR THE SAFE, HUMANE AND SECURE CONFINEMENT OF PERSONS IN FEDERAL CUSTODY | |||
|---|---|---|---|
|
Title |
Focus/Issues to be Addressed |
Methodology and
Type of Evaluation
|
Estimated |
| BOP Taft Correctional Institution in California | Cost effectiveness and operational success of government-owned but privately managed facility. | Process/Cost Effectiveness Evaluation:
In progress |
March 2003 |
| BOP Residential Drug Treatment Program | Conducted with funding and assistance from the National Institute of Drug Abuse, this study focuses on the effectiveness of residential drug treatment programs by monitoring inmates for up to 3 years following their release. | Impact Evaluation: Completed |
September 2000 |
| Private Medical Services Provided at BOP Facilities in Beaumont, Texas | Cost effectiveness and operational success of private health care services. | Process/Cost Effectiveness Evaluation:
In progress |
March 2003 |
| Assumption of Outside Medical Care Responsibilities for USMS Prisoners | Assessment of financial and operational impact on BOP of assuming these responsibilities. | Analysis of Pilot Program Data:
In progress |
September 2002 |
FY 2001 -- 2006 Strategic Plan
U.S. Department of Justice