Sentinel Audit II: Status of the Federal Bureau of Investigation’s Case Management System (Redacted)
Audit Report 07-03
Office of the Inspector General
On March 16, 2006, the Federal Bureau of Investigation (FBI) announced that it had awarded a contract to Lockheed Martin Services, Incorporated (Lockheed Martin) to develop the Sentinel information and investigative case management system in 4 phases. The cost of the four phases of the Lockheed Martin contract was $305 million, and the FBI estimated that it would cost an additional $120 million to staff the FBI’s Sentinel Program Office, provide contractor support, and establish a management reserve for contingencies, with the total estimated cost of Sentinel at $425 million. The initial schedule for the Lockheed Martin contract calls for all phases to be completed in December 2009, or 45 months from the start of work.
According to the contract, Lockheed Martin can be rewarded for meeting established goals in four areas: project management, cost management, schedule, and technical performance. The award fee cannot exceed [redacted] percent of the $232.4 million total development costs for Sentinel, or approximately $26 million, and will be allocated across the four areas based on risk. This type of contract and award fee structure is common for large government IT projects.
The Sentinel project, which uses commercial-off-the-shelf (COTS) components, is intended to provide the FBI with a web-enabled electronic case management system that includes records management, workflow management, evidence management, search and reporting capabilities, and information sharing capability with other law enforcement agencies and the intelligence community. According to the FBI Director, “Sentinel will strengthen the FBI’s capabilities by replacing its primarily paper-based reporting system with an electronic system designed for information sharing. Sentinel will support our current priorities, including our number one priority: preventing terrorist attacks.”10
The Sentinel project follows the FBI’s unsuccessful efforts to develop an automated case management system called the Virtual Case File (VCF), which was intended to replace the FBI’s obsolete Automated Case Support (ACS) system. Because of the FBI’s failed $170 million VCF project, congressional appropriations and oversight committees questioned whether the FBI could successfully develop and implement a case management system of Sentinel’s magnitude. Given the importance of the Sentinel project, the congressional appropriations committees and the FBI Director asked the Department of Justice Office of the Inspector General (OIG) to continually review and report on the progress of the FBI’s development of Sentinel.
This report is the second OIG report on Sentinel, and covers the progress the FBI has made in resolving the concerns identified in our March 2006 report on the planning for Sentinel, whether the FBI’s Information Technology Investment Management (ITIM) processes and project management are likely to contribute to the successful implementation of Sentinel, and the contract with Lockheed Martin to develop Sentinel. Over the past few years, the OIG and others have reviewed various aspects of the FBI’s information technology (IT) infrastructure and cited a critical need for the FBI to modernize its case management system. In previous reports, the OIG concluded that current FBI systems do not permit agents, analysts, and managers to readily access and share case-related information throughout the FBI, and without this capability, the FBI cannot perform its critical missions as efficiently and effectively as it should.
In its mission-needs statement for Sentinel, the FBI stated that its current case management system must be upgraded to utilize new information technologies by moving from a primarily paper-based case management process to an electronic records system. The FBI noted that this transition would enable agents and analysts to more effectively perform their investigative and intelligence duties.
The FBI’s attempt to move from a paper-based to an electronic case management system began with the Trilogy project in mid-2001. The objectives of Trilogy were to update the FBI’s aging and limited IT infrastructure; provide needed IT applications for FBI agents, analysts, and others to efficiently and effectively do their jobs; and lay the foundation for future IT improvements. Trilogy consisted of upgrading the FBI’s: (1) hardware and software; (2) communications network; and (3) the five most important investigative applications, including the antiquated ACS. The first two components of Trilogy were completed in April 2004 at a cost of $337 million, almost $100 million more than originally planned. Among other improvements, the FBI enhanced its IT infrastructure with new desktop computers for its employees and deployed a wide area network to enhance electronic communication among FBI offices and with other law enforcement organizations. However, despite additional funding the FBI received to accelerate completion of Trilogy, these first two phases were not completed any faster than originally planned.
In early 2004, after nearly 3 years of development, the FBI engaged several external organizations and contractors to evaluate the VCF, the third prong of the Trilogy project. Based on critical comments by these organizations, the FBI began to consider alternative approaches to developing the VCF, including terminating the project or developing a completely new case management system. In late 2004, the FBI commissioned Aerospace Corporation to perform a trade study evaluating the functionality of COTS and government off-the-shelf (GOTS) technology to meet the FBI’s case management needs. Aerospace followed this study with an Independent Verification and Validation (IV&V) report on VCF, issued in January 2005, which recommended that the FBI pursue a COTS-based, service-oriented architecture.11 The IV&V report concluded that a lack of effective engineering discipline led to inadequate specification, design, and development of the VCF.
In late 2004, the FBI modified its approach to developing the VCF by dividing the project into Initial Operational Capability (IOC) and Full Operational Capability segments. The IOC segment assessed the VCF project and involved a pilot test of the most advanced version of VCF in an FBI field office. The Project Management Executive for the FBI’s Office of Information Technology Program Management stated that the results of the pilot validated that ending the VCF project was the right decision.
The FBI issued a final report on the IOC at the end of April 2005.12 According to the report, the FBI terminated work on the VCF due to the lack of progress on its development. The FBI stated that it was concerned that the computer code being used to develop the VCF lacked a modular structure, thereby making enhancements and maintenance difficult. In addition, the FBI report said that the “marketplace” had changed significantly since the VCF development had begun, and appropriate COTS products, which were previously unavailable, were now available.
In his March 2005 testimony before the House Appropriations Committee, the FBI Director said the FBI would apply lessons learned from the VCF to develop and deploy Sentinel. The FBI has said that of the $170 million VCF project, $104.5 million was lost but that $53.3 million in contractor services and equipment could be used and $12.2 million was unspent.13
Similar to what the FBI had envisioned for the final version of the VCF, Sentinel is intended to not only provide a new electronic case management system, transitioning the FBI files from paper-based to electronic records, but also to result in streamlined processes for agents to maintain investigative lead and case data.14 In essence, the FBI expects Sentinel to be an integrated system supporting the processing, storage, and management of information to allow the FBI to more effectively perform its investigative and intelligence operations.
According to the FBI, the use of Sentinel in the future will depend on the system’s ability to be easily adapted to evolving investigative and intelligence business requirements over time. Therefore, the FBI intends to develop Sentinel using a flexible software architecture that allows economical and efficient changes to software components as needed in the future. According to the FBI, a key element of the Sentinel architecture contributing to achieving this flexibility will be the use of COTS and GOTS applications software. The FBI intends to integrate the off-the-shelf products with an Oracle database, thereby separating the applications code from the underlying data being managed in order to simplify future upgrades.
FBI agents are required to document investigative activity and information obtained during an investigation. The case file is the central system for holding these records and managing investigative resources. As a result, the case file includes documentation from the inception of a case to its conclusion. FBI agents and analysts currently create paper files in performing their work, making the process of adding a document to a case file a highly paper-intensive, manual process. Files for major cases can contain over 100,000 documents, leads, and evidence items.
Currently, the documentation within case files is electronically managed through the ACS system. The ACS system maintains electronic copies of most documents in the case file, and provides references to documents that exist in hardcopy only. Upon approval of a paper document, an electronic copy of the completed document is uploaded to the electronic case file of the ACS system. However, ACS is a severely outdated system that is cumbersome to use effectively and does not facilitate the searching and sharing of information. The limited capabilities of the ACS mean that agents and analysts cannot easily acquire and link information across the FBI.
In contrast, the FBI expects Sentinel to greatly enhance the usability of case files for agents and analysts, both in terms of adding information to case files as well as searching for case information. FBI supervisors, reviewers, and others involved in the approval process also will be able to review, comment, and approve the insertion of documents into appropriate FBI electronic files through Sentinel.
In addition to enhancing the investigative capabilities within the FBI, Sentinel is intended to serve as the pilot project in the development of the Federal Investigative Case Management System (FICMS) framework as part of the federal government’s e-government case management line of business. The FBI was named the lead agency for the FICMS initiative, which, according to a June 2005 memorandum of understanding (MOU) signed by the FBI, DOJ, and the Department of Homeland Security (DHS) Chief Information Officers (CIO), is intended to produce an architectural framework designed to: (1) bring federal law enforcement and investigative resources into a common electronic environment that promotes collaboration and optimum deployment of federal resources; and (2) create investigative case management solutions that provide state-of-the-art capabilities to collect, share, and analyze information from internal and external sources and initiate appropriate enforcement responses. According to a Senior Policy Advisor to the Department’s CIO, other federal agencies can use Sentinel’s core solution for their case management systems because of its standard set of case management tools and adaptability. Additionally, according to the FBI CIO, the Office of Management and Budget (OMB) has begun to encourage other agencies to become involved with the development of Sentinel and its interfaces in order to ensure future information sharing capability among all agencies.
Sentinel’s Phased Approach
The FBI expects to develop the Sentinel project in 4 phases, each with an approximately 12- to 16-month timeframe and overlapping by 1 to 2 months. For example, Phase 2 is anticipated to begin approximately 2 months before the end of Phase 1. Each phase, when deployed, will result in a stand-alone set of capabilities that can be added to by subsequent phases to complete the Sentinel project. The following chart shows the phases and general timeframes for Sentinel, according to the FBI.
|Conceptual SENTINEL Schedule with Capabilities
[Image Not Available Electronically]
Phase 1 will introduce the Sentinel portal, which will provide access to data from the existing ACS system and eventually, through incremental changes, support access to a newly created investigative case management system. Phase 1 will also provide a case management “workbox” that will present a summary of all cases the user is involved with, rather than requiring the user to perform a series of queries to find the cases as is currently necessary with ACS. Additionally, the FBI will acquire software to identify persons, places, or things within the case files for automated indexing to allow the files to be searchable by these categories. The FBI will also select the hardware and software that will form the foundation of Sentinel’s future service oriented architecture. Finally, the FBI will prepare the data in the Electronic Case File portion of ACS to be migrated to Sentinel in Phase 2.
Phase 2, the most ambitious and difficult of the phases, will begin the transition to paperless case records and the implementation of electronic records management. A workflow tool will support the flow of electronic documents through the review and approval cycles. A new security framework will be implemented to support access controls and electronic signatures. Additionally, the FBI will begin migrating data from the Electronic Case File to Sentinel and preparing data from the Universal Index to be migrated to Sentinel in Phase 3.
Phase 3 will replace the Universal Index (UNI), which is used to determine if any information about a person, place, or thing exists within the FBI’s current case management system. The UNI is a database of persons, places, and things that have relevance to an investigative case. While the current UNI supports only a limited number of attributes, Phase 3 will expand the number of attributes within the information management system.15 Improving the attributes will allow more precise and comprehensive searching within Sentinel and increase the ability to “connect the dots.”
Phase 4 will implement Sentinel’s new case management and reporting capabilities, and will consolidate the various case management components into one overall system. Shortly after the end of this phase, the legacy systems will be shut down and the remaining cases in the legacy Electronic Case File will be migrated to the new case management system. In this phase, as in all the others, changes to the Sentinel portal will be required to accommodate the new features being introduced.
Earned Value Management System
Earned Value Management (EVM) is a tool that measures the performance of a project by comparing the variance between established cost, schedule, and performance baselines and what is actually taking place. These variances are measured periodically to give project managers a timely perspective on the status of a project. EVM then can provide an early warning that a project is heading for trouble. EVM reporting is an important risk-management tool for a major IT development project such as Sentinel.
In August 2005, the OMB issued a memorandum requiring all federal agency CIOs to manage and measure all major IT projects using an EVM system. Additionally, all agencies were to develop policies for full implementation of EVM on IT projects by December 31, 2005. In response to these requirements, the FBI developed a Sentinel Program EVM Capability Implementation Plan in August 2005 and subsequently acquired a tool to implement an EVM system for the Sentinel project.
The OMB EVM memorandum also required that Integrated Baseline Reviews (IBRs) be performed for any projects that require EVM in order to establish performance management baselines against which a project’s performance can be measured.16 Properly executed, IBRs are an essential element of a program manager’s risk-management approach. IBRs are intended to provide both the government’s and the contractor’s program managers with a mutual understanding of the project’s performance measurement baseline and agreement on a plan of action to resolve the identified risks. According to OMB guidance on IBRs, the objective of an IBR is to confirm compliance with the following business rules:
the technical scope of work is complete and consistent with authorizing documents;
key schedule milestones are identified;
supporting schedules reflect a logical flow to accomplish the technical work scope;
resources, including money, facilities, personnel, and skills, are adequate and available for the assigned tasks;
tasks are planned and can be measured objectively, relative to technical progress;
underlying performance measurement baseline rationales are reasonable; and
managers have appropriately implemented required management processes.
Over the past few years, the OIG and other oversight entities have issued reports examining the FBI’s attempts to update its case management system. In these reports the OIG, the Government Accountability Office (GAO), the House of Representatives’ Surveys and Investigations Staff, and others have made a variety of recommendations focusing on the FBI’s management of the FBI’s Trilogy project, particularly the VCF portion of the project, and the continuing need to replace the outdated ACS system. More recently the OIG has reported on Sentinel, the successor to the VCF project. A discussion of key points from these reports follows. (A more comprehensive description of the reports appears in Appendix 3.)
In March 2006, the OIG released the first in a series of audit reports that will monitor the FBI's development and implementation of the Sentinel project.17 This report discussed the FBI’s pre-acquisition planning for the Sentinel project, including the approach, design, cost, funding sources, time frame, contracting vehicle, and oversight structure. In reviewing the management processes and controls the FBI has applied to the pre-acquisition phase of Sentinel, the OIG found that the FBI has developed IT planning processes that, if implemented as designed, can help the FBI successfully complete Sentinel.
In particular, the OIG found that the FBI has made improvements in its ability to plan and manage a major IT project by establishing ITIM processes, developing a more mature Enterprise Architecture, and establishing a Program Management Office (PMO) dedicated to the Sentinel project.
However, at that time the OIG identified several concerns about the FBI’s management of the Sentinel project: (1) the incomplete staffing of the Sentinel PMO, (2) the FBI’s ability to reprogram funds to complete the second phase of the project without jeopardizing its mission-critical operations, (3) Sentinel’s ability to share information with external intelligence and law enforcement agencies and provide a common framework for other agencies’ case management systems, (4) the lack of an established EVM process, (5) the FBI’s ability to track and control Sentinel’s costs, and (6) the lack of complete documentation required by the FBI’s information technology investment management processes.
In May 2006, the GAO released a report critical of the FBI’s controls over costs and assets of its Trilogy project.18 The GAO found that the FBI’s review and approval process for Trilogy contractor invoices did not provide an adequate basis for verifying that goods and services billed were actually received and that the amounts billed were appropriate, leaving FBI highly vulnerable to payments of unallowable costs. These costs included first-class travel and other excessive airfare costs, incorrect charges for overtime hours, and charges for which the contractors could not document costs incurred. The GAO found about $10 million in unsupported and questionable costs. The GAO also found that the FBI failed to establish controls to maintain accountability over equipment purchased for the Trilogy project. According to the GAO, poor property management led to 1,200 missing pieces of equipment valued at $7.6 million.
In February 2005, the OIG reported on the critical need to replace the ACS, finding that without an effective case management system the FBI remained significantly hampered due to the poor functionality and lack of information-sharing capabilities of its current IT systems.19 The OIG audit report concluded that the difficulties the FBI experienced in replacing the ACS were attributable to: (1) poorly defined and slowly evolving design requirements, (2) contracting weaknesses, (3) IT investment management weaknesses, (4) lack of an Enterprise Architecture, (5) lack of management continuity and oversight, (6) unrealistic scheduling of tasks, (7) lack of adequate project integration, and (8) inadequate resolution of issues raised in reports on Trilogy. The report described concerns with the cost-plus-award-fee contract as it was implemented by the FBI for Trilogy because the contract did not: require specific completion milestones, include critical decision review points, or provide for penalties if the milestones were not met.
In April 2005, the House Appropriation Committee’s Surveys and Investigations staff similarly concluded in its report that:20
In September 2004, the GAO reported that although improvements were under way and more were planned, the FBI did not have an integrated plan for modernizing its IT system.21 The GAO reported that each of the FBI’s divisions and other organizational units that manage IT projects performed integrated planning for its respective IT projects. However, the plans did not provide a common, authoritative, and integrated view of how IT investments will help optimize mission performance, and they did not consistently contain the elements expected to be found in effective systems modernization plans. The GAO recommended that the FBI limit its near-term investments in IT systems until it developed an integrated systems and modernization plan and effective policies and procedures for systems acquisition and investment management. Additionally, the GAO recommended that the FBI’s CIO be provided with the responsibility and authority to effectively manage IT FBI-wide.
FBI Press Release entitled FBI Announces Award of Sentinel Contract, March 16, 2006.
IV&V is a standard ITIM process whereby an independent entity assesses the system as it is developed in order to evaluate if the software will perform as intended. A service-oriented architecture is a collection of services that communicate with each other. The communication can involve a simple data exchange or two or more services coordinating on an activity.
Department of Justice, Federal Bureau of Investigation. Federal Bureau of Investigation: Virtual Case File Initial Operational Capability Final Report, version 1.0, April 29, 2005.
The OIG has not verified these figures, including the services and equipment the FBI said could be reused.
A lead is a request from any FBI field office or headquarters for assistance in the investigation of a case.
An attribute defines a property of an object within a case file. Examples of attributes are eye color, height, and nationality when describing an individual or address, floor, and room number when describing a specific location.
The performance measurement baseline is a total, time-phased budget plan against which program performance is measured.
Department of Justice, Office of the Inspector General. The Federal Bureau of Investigation’s Pre-Acquisition Planning For and Controls Over the Sentinel Case Management System, Audit Report Number 06-14, March 2006.
U.S. Government Accountability Office. Federal Bureau of Investigation: Weak Controls over Trilogy Project Led to Payment of Questionable Contractor Costs and Missing Assets, Report Number GAO-06-306, May 2006.
Department of Justice, Office of the Inspector General. The Federal Bureau of Investigation’s Management of the Trilogy Information Technology Management Project, Audit Report Number 05-07, February 2005.
U.S. Congress, House of Representatives, House Surveys and Investigations. A Report to the Committee on Appropriations, U.S. House of Representatives, April 2005.
U.S. Government Accountability Office. Information Technology: Foundational Steps Being Taken to Make Needed FBI Systems Modernization Management Improvements, Report Number GAO 04-842, September 2004.
|« Previous||Table of Contents||Next »|