Sentinel Audit III: Status of the Federal Bureau of Investigation’s
Case Management System (Redacted - Public Version)
Audit Report 07-40
Office of the Inspector General
On March 16, 2006, the Federal Bureau of Investigation (FBI) announced that it had awarded a contract to Lockheed Martin Services (Lockheed Martin) to develop the Sentinel information and investigative case management system. The cost of the four phases of the Lockheed Martin contract totaled $305 million, and the FBI estimated that it would cost an additional $120 million to staff the FBI’s Sentinel Program Management Office (PMO), provide contractor support, and establish a management reserve for contingencies, bringing the total estimated cost of the Sentinel project to $425 million. The initial schedule for the Lockheed Martin contract called for all phases to be completed in December 2009, or 45 months from the start of work.
On June 19, 2007, the FBI announced that it had fully deployed Phase 1 of Sentinel. The goal of this first phase of the project was to provide FBI employees with a user-friendly, web-based access to information currently in the FBI’s antiquated Automated Case Support (ACS) system.18 Phase 1 features a personal workbox that summarizes a user’s cases and leads.19 It also provides user-friendly search capabilities and a squad workbox, which allows supervisors to better manage their resources and assign leads with the click of a mouse.
According to the Sentinel contract, Lockheed Martin can be rewarded for meeting established goals in four areas: project management, cost management, schedule, and technical performance. The award fee cannot exceed 11 percent of the $232.4 million total development costs for Sentinel, or approximately $26 million, and will be allocated across the four areas based on risk. This type of contract and award fee structure is common for large government IT projects.
The Sentinel project, which uses commercial-off-the-shelf (COTS) components, is intended to provide the FBI with a web-enabled electronic case management system that includes records management, workflow management, evidence management, search and reporting capabilities, and information sharing capabilities with other law enforcement agencies and the intelligence community. According to the FBI Director, “Sentinel will strengthen the FBI’s capabilities by replacing its primarily paper-based reporting system with an electronic system designed for information sharing. Sentinel will support our current priorities, including our number one priority: preventing terrorist attacks.”20
The Sentinel project follows the FBI’s unsuccessful efforts to develop an automated case management system called the Virtual Case File (VCF), which was intended to replace the FBI’s ACS system. Because of the FBI’s failed $170 million VCF project, congressional appropriations and oversight committees questioned whether the FBI could successfully develop and implement a case management system of Sentinel’s magnitude. Given the importance of the Sentinel project, the congressional appropriations committees and the FBI Director asked the Department of Justice Office of the Inspector General (OIG) to continually review and report on the progress of the FBI’s development of Sentinel.
This report is the third OIG report on Sentinel, and covers the development of Phase I of the project, planning for Phase 2, and progress made by the FBI in resolving the concerns identified in our two previous audits. The previous two reports focused on the planning for Sentinel, the FBI’s processes and controls for managing information technology (IT) projects, and the contract with Lockheed Martin to develop Sentinel.
Over the past few years, the OIG and others have reviewed various aspects of the FBI’s IT infrastructure and noted the critical need for the FBI to modernize its case management system. In previous reports, the OIG concluded that current FBI systems do not permit agents, analysts, and managers to readily access and share case-related information throughout the FBI, and without this capability the FBI cannot perform its critical missions as efficiently and effectively as it should.21
In its mission-needs statement for Sentinel, the FBI said that its current case management system must be upgraded to utilize new information technologies by moving from a primarily paper-based case management process to an electronic records system. The FBI noted that this transition would enable agents and analysts to more effectively perform their investigative and intelligence duties.
The FBI’s attempt to move from a paper-based to an electronic case management system began with the Trilogy project in mid-2001. The objectives of Trilogy were to update the FBI’s aging and limited IT infrastructure; provide needed IT applications for FBI agents, analysts, and others to efficiently and effectively do their jobs; and lay the foundation for future IT improvements. Trilogy consisted of upgrading the FBI’s: (1) hardware and software; (2) communications network; and (3) the five most important investigative applications, including the antiquated ACS. The first two components of Trilogy were completed in April 2004 at a cost of $337 million, almost $100 million more than originally planned. Among other improvements, the FBI enhanced its IT infrastructure with new desktop computers for its employees and deployed a wide area network to enhance electronic communications among FBI offices and with other law enforcement organizations.
In early 2004, after nearly 3 years of development, the FBI engaged several external organizations and contractors to evaluate the Virtual Case File (VCF), the third component of the Trilogy project. Based on critical comments by these organizations, the FBI began to consider alternative approaches to developing the VCF, including terminating the project or developing a completely new case management system. In late 2004, the FBI commissioned Aerospace Corporation to perform a study evaluating the functionality of COTS and government off-the-shelf technology to meet the FBI’s case management needs. Aerospace followed this study with an independent verification and validation (IV&V) report on the VCF, issued in January 2005, which recommended that the FBI pursue a COTS-based, service-oriented architecture.22 The IV&V report concluded that a lack of effective engineering discipline led to inadequate specification, design, and development of the VCF.
The FBI modified its approach to developing the VCF, and in late 2004 divided the project into Initial Operational Capability and Full Operational Capability segments. The Initial Operational Capability segment assessed the VCF project and involved a pilot test of the most advanced version of the VCF in an FBI field office. In February 2005, the OIG issued a report on the Trilogy project questioning the FBI’s ability to complete and deploy the VCF.23
The FBI issued a final report on the Initial Operational Capability at the end of April 2005.24 According to the report, the FBI terminated work on the VCF due to the lack of progress on its development. The FBI stated that it was concerned that the computer code being used to develop the VCF lacked a modular structure, thereby making enhancements and maintenance difficult. In addition, the FBI report said that the “marketplace” had changed significantly since the VCF development had begun, and appropriate COTS products, which were previously unavailable, were now available.
Similar to what the FBI had envisioned for the final version of the VCF, Sentinel is intended to not only provide a new electronic case management system, transitioning the FBI files from paper-based to electronic records, but also to result in streamlined processes for employees to maintain investigative lead and case data. In essence, the FBI expects Sentinel to be an integrated system supporting the processing, storage, and management of information to allow the FBI to more effectively perform its investigative and intelligence operations.
According to the FBI, the use of Sentinel in the future will depend on the system’s ability to be easily adapted to evolving investigative and intelligence business requirements over time. Therefore, the FBI has been working to develop Sentinel using a flexible software architecture that allows economical and efficient changes to software components as needed in the future. According to the FBI, a key element of the Sentinel architecture contributing to achieving this flexibility is the use of COTS and government-off-the-shelf applications software. The FBI has been working to integrate the off-the-shelf products with an Oracle database, thereby separating the applications code from the underlying data being managed in order to simplify future upgrades.
FBI agents are required to document investigative activity and information obtained during an investigation. The case file is the central system for holding these records and managing investigative resources. As a result, the case file includes documentation from the inception of a case to its conclusion. FBI agents and analysts currently create paper files in performing their work, making the process of adding a document to a case file a highly paper-intensive, manual process. Files for major cases can contain over 100,000 documents, leads, and evidence items.
Currently, the documentation within case files is electronically managed through the ACS system, which maintains electronic copies of most documents in the case file and provides references to documents that exist in hardcopy only. Upon approval of a paper document, an electronic copy of the completed document is uploaded to the electronic case file of the ACS system. However, ACS is a severely outdated system that is cumbersome to use effectively and does not facilitate the searching and sharing of information. The limited capabilities of the ACS mean that agents and analysts cannot easily acquire and link information across the FBI.
In contrast, the FBI expects Sentinel to greatly enhance the usability of case files for agents and analysts, both in terms of adding information to case files as well as searching for case information. FBI supervisors, reviewers, and others involved in the approval process will also be able to review, comment, and approve the insertion of documents into appropriate FBI electronic files through Sentinel.
Sentinel’s Phased Approach
As originally conceived, the FBI expected to develop the Sentinel project in 4 partially overlapping phases, each lasting approximately 12- to 16-months. However, at the time of our audit the FBI was reevaluating the number of phases and the capabilities each phase will deliver as it gained experience with the project’s development. Each phase, when deployed, was to result in a stand-alone set of capabilities that could be added to by subsequent phases to complete the Sentinel project. The following chart shows the FBI’s original conception of the four phases and their general timeframes.
Phase 1 introduced the Sentinel web-based portal, which provides access to data from the existing ACS system. Eventually, through incremental changes in subsequent phases, the portal will display data from a newly created investigative case management system. Phase 1 also provides a case management workbox that presents a summary of all cases a user is involved with rather than requiring the user to perform a series of queries to find the cases as was necessary when only ACS was used. The Findings and Recommendations section of this report contains a more comprehensive discussion of the Phase 1 deliverables.
Phase 2, the most ambitious and technically difficult of the phases, will begin the transition to paperless case records and the implementation of electronic records management. A workflow tool will support the flow of electronic documents through the review and approval cycles, and a new security framework will be implemented to support access controls and electronic signatures. Additionally, in Phase 2, the FBI will begin migrating data from the ACS electronic case file to Sentinel and preparing data from the Universal Index (discussed below) to be migrated to Sentinel in Phase 3.
Phase 3 will replace the Universal Index (UNI), which is used to determine if any information about a person, place, or thing exists within the FBI’s current case management system. The UNI is a database of persons, places, and things that have relevance to an investigative case. While the current UNI supports only a limited number of attributes, Phase 3 will expand the number of attributes within the information management system.25 Enhancing the attributes will allow more precise and comprehensive searching within Sentinel and increase the FBI’s ability to “connect the dots.”
Phase 4 will implement Sentinel’s new case management and reporting capabilities, and will consolidate the various case management components into one overall system. Shortly after the end of this phase, the legacy systems will be shut down and the remaining cases in the legacy ACS electronic case file will be migrated to the new case management system. In this phase, as in all the others, changes to the Sentinel portal will be required to accommodate the new features being introduced.
FBI Management Processes and Controls
In the early stages of the Trilogy project, the OIG and U.S. Government Accountability Office (GAO) recommended that the FBI establish Information Technology Investment Management (ITIM) processes to guide the development of its IT projects. In response, in 2004 the FBI issued its Life Cycle Management Directive (LCMD). The LCMD covers the entire IT system life cycle, including planning, acquisition, development, testing, and operations and maintenance. As a result, the LCMD provides the framework for standardized, repeatable, and sustainable processes and best practices in developing IT systems. Application of the IT systems life cycle within the LCMD can also enhance guidance for IT programs and projects, leverage technology, build institutional knowledge, and ensure that development is based on industry and government best practices. The LCMD is comprised of four integrated components: life cycle phases, control gates, project level reviews, and key support processes. A diagram showing how these components relate to each other and a description of the life cycle phases, control gates, and the project level reviews mentioned throughout this report are contained in Appendix 3.
The LCMD established policies and guidance applicable to all FBI IT programs and projects, including Sentinel. As we discussed in our March 2006 report on Sentinel, we believe the structure and controls imposed by the LCMD can help prevent many of the problems encountered during the failed VCF effort. Since our March 2006 report on Sentinel, the FBI has further refined its LCMD and is applying the revised directive to Sentinel.
Earned Value Management System
Earned Value Management (EVM) is a tool that measures the performance of a project by comparing the variance between established cost, schedule, and performance baselines to what is actually taking place. These variances are measured periodically to give project managers a timely perspective on the status of a project. EVM then can provide an early warning that a project is heading for trouble. EVM reporting is an important risk-management tool for a major IT development project such as Sentinel.
In August 2005, the Office of Management and Budget (OMB) issued a memorandum requiring all federal agency Chief Information Officers (CIOs) to manage and measure all major IT projects using an EVM system. Additionally, all agencies were to develop policies for full implementation of EVM on IT projects by December 31, 2005. The Department of Justice issued its EVM policy in July 2006. In response to these requirements, the FBI developed a Sentinel Program EVM Capability Implementation Plan in August 2006 and subsequently acquired a tool to implement an EVM system for the Sentinel project.
The OMB EVM memorandum also required that integrated baseline reviews (IBRs) be performed for any projects that require EVM in order to establish performance management baselines against which a project’s performance can be measured.26 Properly executed, IBRs are an essential element of a program manager’s risk-management approach. IBRs are intended to provide both the government’s and the contractor’s program managers with a mutual understanding of the project’s performance measurement baseline and agreement on a plan of action to resolve identified risks.
According to OMB guidance, the objective of an IBR is to confirm compliance with the following business rules:
The technical scope of work is complete and consistent with authorizing documents;
Key schedule milestones are identified;
Supporting schedules reflect a logical flow to accomplish the technical work scope;
Resources, including money, facilities, personnel, and skills, are adequate and available for the assigned tasks;
Tasks are planned and can be measured objectively, relative to technical progress;
Underlying performance measurement baseline rationales are reasonable; and
Managers have appropriately implemented required management processes.
Over the past few years, the OIG and other oversight entities have issued reports examining the FBI’s attempts to update its case management system. In these reports the OIG, the GAO, the House of Representatives’ Surveys and Investigations Staff, and others have made a variety of recommendations focusing on the FBI’s management of its IT projects, particularly the VCF portion of the Trilogy project, and the continuing need to replace the outdated ACS system. More recently the OIG has reported on Sentinel, the successor to the VCF project. A discussion of key points from these reports follows. (A more comprehensive description of the reports appears in Appendix 4.)
In the first OIG Sentinel report issued in March 2006, we discussed the FBI’s pre-acquisition planning for the Sentinel project, including the approach, design, cost, funding sources, time frame, contracting vehicle, and oversight structure.27 In reviewing the management processes and controls the FBI had applied to the pre-acquisition phase of Sentinel, the OIG found that the FBI developed IT planning processes that, if implemented as designed, could help the FBI successfully complete Sentinel.
In particular, the OIG found that the FBI had made improvements in its ability to plan and manage a major IT project by establishing ITIM processes, developing a more mature Enterprise Architecture, and establishing a PMO dedicated to the Sentinel project.
However, at that time the OIG identified several concerns about the FBI’s management of the Sentinel project, including: (1) the incomplete staffing of the Sentinel PMO, (2) the FBI’s ability to reprogram funds to complete the second phase of the project without jeopardizing its mission-critical operations, (3) Sentinel’s ability to share information with external intelligence and law enforcement agencies and provide a common framework for other agencies’ case management systems, (4) the lack of an established EVM process, (5) the FBI’s ability to track and control Sentinel’s costs, and (6) the lack of complete documentation required by the FBI’s information technology investment management processes.
In December 2006, the OIG released the second in a series of audit reports that examined the FBI’s development and implementation of the Sentinel project.28 This report discussed: (1) the progress the FBI made in resolving the concerns identified in the first OIG report on the planning for Sentinel, and (2) whether the contract with Lockheed Martin and the FBI’s ITIM processes and project management are likely to contribute to the successful implementation of Sentinel. The OIG found that the FBI resolved most of the concerns the OIG identified in its first Sentinel audit, although the audit reported that some aspects of those concerns as well as some new concerns identified in the second audit merited continued monitoring. Specifically, the OIG found that the FBI had made progress in: (1) establishing cost tracking and control processes, (2) implementing an Earned Value Management (EVM) system to help measure progress toward project baselines, (3) developing an IV&V plan, (4) developing information sharing capabilities, and (5) hiring more PMO staff.
Among the areas that warranted continued monitoring by the FBI, the OIG, and other oversight entities were the: (1) funding of the Sentinel project and the effect on the FBI’s operations or other projects if a reprogramming of funds was required, (2) accuracy of the estimated cost of the project, (3) availability of contingency plans for identified project risks, and (4) completion of Sentinel PMO staffing.
In May 2006, the GAO released a report critical of the FBI’s controls over costs and assets of its Trilogy project.29 The GAO found that the FBI’s review and approval process for Trilogy contractor invoices did not provide an adequate basis for verifying that goods and services billed were actually received and that the amounts billed were appropriate, leaving the FBI highly vulnerable to payments of unallowable costs. These costs included first-class travel and other excessive airfare costs, incorrect charges for overtime hours, and charges for which the contractors could not document costs incurred. The GAO found about $10 million in unsupported and questionable costs. The GAO also found that the FBI failed to establish controls to maintain accountability over equipment purchased for the Trilogy project. According to the GAO, poor property management led to 1,200 missing pieces of equipment valued at $7.6 million.
In July 2007, the GAO issued a report on the extent to which the FBI had established best practices for acquiring Sentinel and estimating the project’s schedule and costs.30 The GAO concluded that, in general, the FBI had best practices in place for acquiring IT systems, including practices for evaluating offers and awarding contracts. In contrast, our audit examined the FBI’s implementation of its policies and procedures for managing the development of Sentinel, including several related to the best practices reviewed by the GAO. Because our audit focused on how these policies and procedures were actually implemented, our findings differ from the GAO’s because we found areas of inadequate implementation.
ACS is the FBI’s current case management system. Deployed in 1995, ACS is a mainframe system.
A lead is a request from any FBI field office or headquarters for assistance in the investigation of a case.
FBI Press Release entitled FBI Announces Award of Sentinel Contract, March 16, 2006.
For a more complete discussion of the OIG’s reports on Sentinel, see the Prior Reports section on page 9.
IV&V is a standard information technology investment management (ITIM) process whereby an independent entity assesses the system as it is developed in order to evaluate if the software will perform as intended. A service-oriented architecture is a collection of services that communicate with each other. The communication can involve a simple data exchange or two or more services coordinating on an activity.
Department of Justice, Office of the Inspector General, The Federal Bureau of Investigation’s Management of the Trilogy Information Technology Modernization Project, Audit Report Number 05-07, February 2005.
Department of Justice, Federal Bureau of Investigation. Federal Bureau of Investigation: Virtual Case File Initial Operational Capability Final Report, version 1.0, April 29, 2005.
An attribute defines a property of an object within a case file. Examples of attributes are eye color, height, and nationality when describing an individual or address, floor, and room number when describing a specific location.
The performance measurement baseline is a total, time-phased budget plan against which program performance is measured.
Department of Justice, Office of the Inspector General. The Federal Bureau of Investigation’s Pre-Acquisition Planning For and Controls Over the Sentinel Case Management System, Audit Report Number 06-14, March 2006.
Department of Justice, Office of the Inspector General. Sentinel Audit II: Status of the Federal Bureau of Investigation’s Case Management System, Audit Report Number 07-03, December 2006.
U.S. Government Accountability Office. Federal Bureau of Investigation: Weak Controls over Trilogy Project Led to Payment of Questionable Contractor Costs and Missing Assets, Report Number GAO-06-306, May 2006.
U.S. Government Accountability Office, Information Technology: FBI Following a Number of Key Acquisition Practices on New Case Management System but Improvements Still Needed, July 2007.
|« Previous||Table of Contents||Next »|