The Federal Bureau of Investigation's Pre-Acquisition Planning for
and Controls Over the Sentinel Case Management System
Audit Report 06-14
March 2006
Office of the Inspector General
|
In March 2005, the Federal Bureau of Investigation (FBI) terminated a 3-year, $170 million effort to develop a modern case management system called the Virtual Case File (VCF) and announced a new project called Sentinel. As detailed in the Office of the Inspector General's (OIG) February 2005 audit report on the FBI's larger Trilogy Information Technology Modernization Project, the VCF project failed for a variety of reasons, including poorly defined design requirements, lack of mature Information Technology Investment Management (ITIM) processes, and poor management continuity and oversight.1 With Sentinel, the FBI is relying on improved management processes, use of commercially available components, and a four-phase approach over 39 to 48 months to develop a replacement for its obsolete Automated Case Support (ACS) system. As of February 2006, the FBI had not disclosed its specific cost estimates for Sentinel because the contract to a private information technology (IT) systems developer had not yet been awarded. However, in response to congressional inquiries, the FBI has cited a cost between $400-$500 million to develop the system. According to the FBI, a more precise cost estimate will be available once the FBI awards the Sentinel contract in calendar year 2006. The OIG performed this audit of the Sentinel project at the request of the FBI Director and congressional appropriations committees. This audit is the first in a series of audits that the OIG intends to conduct on an ongoing basis to evaluate the development and implementation of Sentinel. The objective of this first audit was to evaluate the FBI's pre-acquisition planning for Sentinel, including the approach, design, cost, funding sources, timeframe, contracting vehicle, and oversight structure. Our future audits will examine the development of the system over its four phases and assess whether cost, schedule, performance, and technical benchmarks are being met. Background to Sentinel A major objective of the FBI's IT modernization project is to replace the FBI's antiquated ACS. During a variety of OIG reviews over the past several years, we reported that ACS uses outmoded technology, is cumbersome to operate, and does not provide necessary workflow and information-sharing functions. The FBI expects that Sentinel will provide it with a web-enabled case management system that includes records management, workflow management, collected item and evidence management, and records search and reporting capabilities, all of which will replace its current paper-based case management system. The FBI intends to implement Sentinel in four phases, with each phase providing distinct capabilities until the overall project is completed in 2009. The FBI expects to complete each of the phases in 12 to 18 months, with the phases overlapping. For example, Phase II will begin about 3 months into Phase I. According to the FBI, the four phases will provide the following capabilities.
In reviewing the management processes and controls the FBI has applied to the pre-acquisition phase of Sentinel, we believe that the FBI has adequately planned for the project and this planning provides reasonable assurance that the FBI can successfully complete Sentinel if the processes and controls are implemented as intended. However, we have several concerns about the project that require action and continued monitoring: (1) the incomplete staffing of the PMO, (2) the FBI's ability to reprogram funds to complete the second phase of the project without jeopardizing its mission-critical operations, (3) Sentinel's ability to share information with external intelligence and law enforcement agencies and provide a common framework for other agencies' case management systems, (4) the lack of an established Earned Value Management (EVM) process, (5) the FBI's ability to track and control Sentinel's costs, and (6) the lack of complete documentation required by the FBI's ITIM processes. New IT Management Processes In previous reports, we were critical of the FBI's lack of ITIM processes and Enterprise Architecture (the blueprint for its current and future IT environment) in the implementation of Trilogy. We believe that these weaknesses contributed, in large part, to the FBI's past failures in developing IT systems. In this audit, we found that since the troubled Trilogy project and VCF failure, the FBI has established ITIM processes through its Life Cycle Management Directive (LCMD) and through continued work on fully defining its Enterprise Architecture. The FBI's newly created IT management processes, reviews, and controls, coupled with external oversight by the OIG, contractors, congressional committees, and others, should help the FBI identify and minimize failures to achieve cost, schedule, performance, and technical benchmarks for the Sentinel project. Life Cycle Management Directive In November 2004, the FBI established an initial Life Cycle Management Directive, which it has since refined and is applying to the Sentinel project. The LCMD governs all aspects of an IT project, including planning, acquisition, development, testing, and operations and maintenance. The FBI's LCMD contains four overlapping components: life cycle phases, control gates, project level reviews, and key support processes. Nine life cycle phases require FBI management approvals during the development, implementation, and retirement of IT projects. The approvals occur through seven control gates in which an FBI executive-level review board discusses and approves the project before it proceeds to the next control gate. The control reviews, in turn, are based on the results of project-level reviews described below. As of December 2005, the FBI's Investment Management Project Review Board (IMPRB) had approved the Sentinel project through two control gates covering three of the nine life cycle phases: concept exploration, requirements development, and acquisition planning. These three phases covered the following planning aspects of Sentinel.
The remaining life cycle phases will cover source selection where proposals are solicited and evaluated and the vendor is selected; design of the system's components and connectivity; testing of system components and the overall product; implementation and integration of the operational system, including training; operations and maintenance to support the system; and disposal of Sentinel when it reaches the end of its life cycle. The FBI completed two Sentinel control gates by the conclusion of our field work for this audit report in December 2005. The review board approved the system concept in mid-July 2005 and the acquisition plan in late-July 2005. The latter review approved documentation of the system specifications and interface controls, as well as the project approach and resource estimates. Sentinel will be required to pass through four more control gates - final design review, deployment readiness, system test readiness, and operational acceptance review - and will be reviewed by four other executive-level review boards as the project proceeds.2 The next control gate, final design review, is led by the Technical Review Board and seeks to ensure that the project design complies with technical requirements and will meet the FBI's needs. The various executive-level control gate reviews are based in part on the results of more detailed project-level reviews. The LCMD calls for the FBI's Program Management Office to conduct these project reviews. By December 2005, the FBI-wide Program Management Office had conducted two project-level reviews that fed into the two higher-level control gate reviews. The first was a mission-needs review approving Sentinel's mission requirements, and the second was a system specification review approving documents for the system specifications and the external interface controls. The system specification review was the decision point that led to development of Sentinel's acquisition plan, the allocation of the requirements to the four phases of the project, and the development of project plans to carry out the acquisition. In addition to the project-level reviews, the LCMD contains 23 key support processes that provide additional support to the development of IT projects within the FBI. Rather than being created for specific projects, these processes cover organization-wide management functions, such as strategic planning. As a result, the key support processes affect how individual projects such as Sentinel are managed within the FBI. Key support processes are also performed independently from the life cycle phases, but the deliverables associated with each key process area are integrated into the project-level and control gate reviews where applicable. In examining the implementation of the LCMD for Sentinel thus far - a vital element in providing internal management oversight and control over the project - we concluded that the FBI's ITIM processes appear to be sound and were generally being followed. We also found that the FBI successfully completed most of the documentation required for the first three phases of the nine-phase life cycle. However, as of December 2005, the FBI had not yet completed the system security plan or the verification and validation plan as required by the LCMD. Nevertheless, Sentinel was approved to proceed past the second control gate without these two plans. The FBI explained that: (1) the system security plan cannot be completed until Sentinel's vendor provides detailed information on the project's design, and (2) a separate contract will be awarded to develop an Independent Verification and Validation (IV&V) plan. The FBI further explained that the system security plan will provide detail necessary for the completion of certification and accreditation of the applications being created for Sentinel, while the IV&V plan will provide for an independent control to assess the implementation of the system according to technical and performance baselines. We believe the FBI's explanation for deferring these two plans are reasonable, given the timing of the contract for Sentinel. However, in our next audit, we will monitor whether the FBI completes the system security plan and the IV&V plan during the early stages of Sentinel's development. Risk Management The purpose of risk management is to assist the program management team in identifying, assessing, categorizing, monitoring, controlling, and mitigating risks before they negatively affect a program. A risk management plan identifies procedures used to manage risk throughout the life of the program. We found that the FBI has instituted a risk management process for Sentinel. Although Risk Review Board meetings have been held biweekly since the project began, the FBI stated that it plans to hold weekly meetings once the Sentinel contract is awarded. When the Risk Review Board identifies specific risks, they are discussed at monthly Program Management Review sessions and other Sentinel oversight meetings. Risks are categorized by severity and identified as either open or resolved. Open risks are tracked until resolved. During the initial life cycle phase of Sentinel, the FBI developed a mission-needs statement that assessed five areas for risk mitigation: (1) user acceptance, (2) implementation plan, (3) system capacity and performance, (4) data migration, and (5) infrastructure support. In addition, the Sentinel acquisition plan identified the following seven risks.
Awareness of these risks and a systematic monitoring and resolution of those risks is critical to keeping Sentinel on track. Project Oversight In addition to the management controls incorporated into its LCMD, the FBI has established two additional forms of project management and oversight for Sentinel: a Program Management Office or PMO established specifically for Sentinel, and an array of external oversight bodies. The PMO, as the FBI's direct manager of the Sentinel project, is vital to Sentinel's success. Among the many reasons for the failure of the VCF was a fragmented and ill-equipped PMO that suffered from rapid personnel turnover. Simply put, the VCF was poorly managed. A well functioning PMO can reduce the risks that threaten the successful implementation of the Sentinel project. While the FBI has established a PMO dedicated exclusively to Sentinel, this PMO has not yet been fully staffed. Without a fully staffed, stable, and capable PMO managing the project on a daily basis, Sentinel is at risk. The FBI intends for the PMO to be comprised of systems engineers, technical assistance personnel, and other subject matter experts from the FBI, other government agencies, federally funded research and development centers, and contractors. As of January 30, 2006, the PMO had 51 of the planned full staffing level of 76 employees and contractors on board. In response to our concerns about staffing, Sentinel's program manager stated that because of the pre-award spending caps the FBI placed on the program, fully staffing the PMO during the pre-award phase was premature. As a result, the program manager said the FBI is only hiring essential program management oversight personnel during this initial phase to ensure that the PMO is prepared to handle contract award activities. However, in light of the FBI's aggressive development and deployment schedule for Sentinel, it is critical for the FBI to fully staff the PMO office as soon as possible. In our opinion, the significant turnover of project management during the Trilogy project - 15 different key IT managers over the course of its life, including 10 individuals serving as project managers for various aspects of Trilogy - was a major reason for Trilogy's problems. We believe that sufficiently staffing the Sentinel PMO at the outset of the project is key to establishing the stable management staff required to properly oversee the project. At the time of our audit, the FBI was working to identify qualified candidates to fill the vacant PMO positions, many of whom will be contractor personnel. Another reason for our concern is that security clearances will be required for the staff of the PMO and, according to the FBI, obtaining the clearances may delay personnel coming onboard. In addition, it is critical for the PMO to have stable leadership. In November 2005 the FBI appointed a seasoned program manager on detail to the FBI from the Central Intelligence Agency to manage the Sentinel project. However, this program manager's current agreement calls for a 2-year detail with an option to extend to a third year. In light of the likelihood of this manager returning to the CIA before Sentinel is completed, the FBI plans to groom a successor for him. We believe that continuity in this position, or a seamless transition to a qualified successor, is critical for the success of the project. In addition, continuity in the FBI's CIO position is important. During development of Trilogy and the VCF, the FBI had five different CIOs or Acting CIOs. However, in the last several years, the FBI has had continuity in the CIO position. In July 2004, the FBI reorganized its IT resources and established the Office of the CIO to centrally manage all IT responsibilities, activities, policies, and employees across the FBI. The current CIO, who has been in his position since May 2004, now has responsibility for the FBI's overall IT efforts, including developing the FBI's IT strategic plan and operating budget, developing and maintaining the FBI's technology assets, and providing technical direction for the re-engineering of FBI business processes. External oversight organizations also play an important role in monitoring the Sentinel project and identifying problems that the FBI may not see. These groups include congressional oversight committees, the OIG, and several other outside organizations. To its credit, the FBI has enlisted the assistance of its Science and Technology Board, RAND, the Markle Foundation, and a retired corporate chief technology officer to advise the FBI on areas of information sharing and privacy, IT strategic planning and investments, and management of large IT acquisitions.4 In addition, the Department of Justice CIO and the Office of Management and Budget are also tracking the progress of Sentinel. Earned Value Management The FBI has developed a Sentinel Program Earned Value Management (EVM) Capability Implementation Plan in which the FBI and the Sentinel vendor will be required to apply EVM practices to the project. EVM is a process that coordinates work scope, schedule, and cost goals and objectively measures progress toward those goals. The Sentinel Program Management Office will use the EVM plan to measure Sentinel's performance and the performance of the vendor and will report the results to oversight entities. As of December 2005, the FBI was in the process of acquiring its EVM tool to track and manage Sentinel. Until the tool is acquired, the plan outlines a methodology for the FBI to obtain earned value measures through other applications. When acquired and implemented, the EVM tool should allow program managers to evaluate Sentinel project performance against baselines and identify potential problems with the project. Due to the importance of EVM in helping to detect problems in a project's development, we will continue to monitor the FBI's implementation of this process in our future audit work. Capability Maturity Model Integration The FBI's Statement of Work for the Sentinel project requires that bidders obtain an independent appraisal certifying that their systems development, software engineering, and integration processes are at a Level 3 or higher on the Carnegie-Mellon University's Capability Maturity Model Integration (CMMI) 5-level maturity scale. This requirement covers all vendors and any subcontractors that will contribute a minimum of 10 percent of the total Sentinel effort in developing or integrating software. Sentinel's Statement of Work also gives the FBI the right to interview the lead appraiser who conducts the assessment and obtain independent assessments during the development of the project to verify compliance with the appraised processes. We believe that by requiring vendors to perform at a CMMI Level 3, the FBI has reduced the risk of selecting vendors that are not capable of completing the Sentinel project and integrating all four project phases. Additionally, because the vendors will be independently reviewed by a CMMI appraiser, the FBI has greater assurance that the processes the vendor will use to develop Sentinel follow best industry practices. In our upcoming audit work, we plan to verify that the CMMI appraisal is conducted, review its results, and assess the appraiser's independence. Enterprise Architecture Since 2000, the FBI has struggled to develop an Enterprise Architecture to help manage its current and planned IT infrastructure and applications. The lack of a mature Enterprise Architecture was one of the reasons for the troubled Trilogy project and the failure of the VCF. However, over the past 5 years the FBI has made significant progress in establishing its Enterprise Architecture. In March 2005, the FBI completed an Enterprise Architecture report that provides a high-level snapshot of current FBI business processes and supporting IT structures and systems. The FBI has also defined its desired IT infrastructure environment, or target architecture. In addition, the FBI has completed an interim architecture report describing how Sentinel will enhance the FBI's current IT capabilities. Like most federal agencies the FBI does not yet have a fully mature architecture, but the FBI's architecture now appears to be sufficiently mature to provide the required management structures and processes needed to guide the Sentinel project and ensure its compatibility with the rest of the FBI's IT environment. Contracting The process to identify a contractor for the Sentinel project began in late June 2005, with the FBI providing information to potential bidders. In early August 2005, the FBI issued a Request for Proposals (RFP). Initially, responses were due by September 19 and the contract was to be awarded on November 15. However, because of technical questions arising from potential bidders, the FBI extended the response date to September 26 and the award date to December 31. As of February 2006, however, the contract had not been awarded and the FBI had not provided a revised award date. According to the Sentinel program manager, the award date was postponed because initial reviews by the source selection evaluation team identified a need for additional data from the companies that submitted proposals. Once the data is received, the source selection evaluation team will complete the formal review and present its results to the awarding committee. The program manager said an award date cannot be determined until the FBI receives and reviews the additional data. The Sentinel development contract will be cost-plus-award-fee in which the vendor will be rewarded for meeting established goals in four areas: project management, cost management, schedule, and technical performance. The award fee can not exceed 12 percent of the total development costs for Sentinel and will be allocated across the four areas based on the degree of risk agreed to by the FBI and the vendor at the signing of the contract. This type of contract is common for large government IT projects. In our 2005 report on the FBI's Trilogy project, we stated our concerns with the cost-plus-award-fee contract as it was implemented by the FBI in that project. The cost-plus-award-fee contract used for Trilogy did not: (1) require specific completion milestones, (2) include critical decision review points, and (3) provide for penalties if the milestones were not met. However, the FBI's improved management processes and controls should reduce the risk of such problems recurring for Sentinel because the FBI intends to establish clear milestones, impose penalties for missed milestones, and include critical decision review points. To identify a prime contractor for Sentinel, the FBI used a contracting vehicle provided through the National Institutes of Health (NIH), one of 16 government-wide acquisition contracts the FBI evaluated before narrowing the field to 5 suitable for a large IT project such as Sentinel. The FBI selected the NIH CIO Solutions Partners 2 Innovations contracting vehicle because it had 37 prime contractors and could provide a greater number of potential bidders and a greater opportunity for competition. The FBI has closely guarded information about potential contractors and costs as procurement sensitive, and has not informed the OIG of the identities of the potential contractors. However, several publications have reported that two major defense contractors have bid on Sentinel. According to the Sentinel program manager, as of February 2006 the FBI was evaluating the bids based on the following five factors:
Funding Because this first OIG audit of Sentinel was focused on the FBI's pre-acquisition planning, and given the procurement sensitive nature of cost information at this stage of the award process, the FBI did not provide us with details regarding the estimated cost of the planned four-phase Sentinel project. However, in response to a Senate Appropriations Committee inquiry in October 2005, the FBI estimated that it would cost the government between $400 and $500 million to develop Sentinel. The FBI stated that the precise cost estimate will not be disclosed until the FBI awards the contract, a decision which as mentioned previously has been postponed to early 2006. In our upcoming audit work, we plan to examine in detail the winning bidder's cost estimates. The FBI has stated, however, that it plans to fund the first two phases of Sentinel by seeking congressional approval to reprogram FBI funds through two separate requests. According to the FBI's plan, the third and fourth phases would be funded by appropriations. In accord with this plan, in September 2005 the FBI requested a $97 million reprogramming of fiscal year (FY) 2005 funds for the first phase of Sentinel. Congress approved the reprogramming in mid-November 2005. According to the FBI's submission, more than $14 million of the initial reprogramming will come from the Counterterrorism Division budget, $13 million from intelligence-related activities, and $2 million from the Cyber Division. We interviewed officials at FBI headquarters to assess the effect of this $97 million reprogramming on FBI operations. Generally, these officials said their divisions and offices can withstand the diversion of funds to Sentinel for the first reprogramming. However, we are concerned that diverting substantial funds from such mission-critical areas could begin eroding the FBI's operational effectiveness, only to be compounded by an anticipated second reprogramming. Although the FBI divisions and offices seemed confident about their ability to absorb the initial reprogramming of funds to Sentinel, they stated that a second reprogramming of the same magnitude would damage their ability to fulfill their mission. According to the FBI CIO, the FBI intends to send another reprogramming request to Congress to fund the second phase of the Sentinel program in FY 2006. The OIG plans to assess the operational impact of these reprogrammings in subsequent Sentinel audits to ensure the FBI's critical missions are not adversely affected by the reprogramming of funds to the Sentinel project. Training At the time of our audit in February 2006, the FBI had not yet developed a training plan or complete cost estimates for Sentinel training. The FBI's first reprogramming request estimated $1.2 million for training in the first phase, although the FBI recognized that total training costs over the life of the project will be substantially higher. Consequently, we recommend that the FBI develop a comprehensive training plan with more accurate cost estimates as soon as possible so that complete training costs can be included in the overall Sentinel budget. Cost Tracking In the Trilogy project, the FBI lacked an effective, reliable system to track and validate the contractors' costs. We highlighted this concern in our February 2005 report on Trilogy and the VCF. Although the FBI stated during the current audit that it was evaluating a tool to track project costs, we recommend that the FBI implement an effective method to track and control costs as soon as possible. We view the potential weaknesses in cost control over the Sentinel project as a significant project risk. Information Sharing According to the Sentinel requirements document, the FBI's ability to share information not only internally but also with its law enforcement and intelligence community partners is an important design requirement for Sentinel. In addition, according to the Senior Policy Advisor to the Department of Justice's CIO, through the interagency Federal Investigative Case Management System (FICMS) effort, Sentinel is intended to provide the core elements of a case management system that other law enforcement and intelligence agencies can adapt to meet their unique requirements. While the FBI has considered its internal needs in developing Sentinel's requirements, we are concerned that the FBI has not yet adequately examined or discussed Sentinel's ability to connect with external systems in other Department of Justice components, the Department of Homeland Security (DHS), and other intelligence community agencies. If such connectivity is not built into Sentinel's design, other agencies could be forced into costly and time-consuming modifications to their systems to allow information sharing with the Sentinel system. The FBI CIO told us that the FBI invited representatives of the DHS, Drug Enforcement Administration (DEA), and Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) to participate in the development of Sentinel's requirements. In addition, the CIO said the FBI has discussed Sentinel interface issues with the Office of Management and Budget (OMB) and the Directorate of National Intelligence (DNI). We interviewed officials from the DHS, DEA, and ATF concerning Sentinel. DHS officials told us that it reviewed the system requirements the FBI had already prepared, but that the DHS did not participate in developing them. DHS officials said that the DHS does not have enough information at this stage of Sentinel's development to assess whether Sentinel and DHS systems will be able to share information or what will be required to achieve compatibility. According to a DHS official, the DHS hopes to "piggyback" onto Sentinel and use at least parts for its own investigative case management system. In addition, the DHS said it plans to assign IT subject-matter experts to the FBI to assist in advising on and managing Sentinel, but is not certain of the specific role the personnel would play. The DEA plans to deploy its own new case management system to DEA field offices in early 2006. According to the DEA's Deputy CIO, its new case management system is not compatible with Sentinel as currently designed. To address this incompatibility, DEA officials said they plan to monitor Sentinel's development to identify any modifications in the DEA system needed to achieve compatibility with Sentinel. The ATF said it had not reviewed the requirements for Sentinel and did not know at this early stage whether it would need to modify its systems to achieve compatibility. Conclusions In our judgment, the FBI has taken important steps to address its past mistakes with the VCF in planning for the development of Sentinel. In reviewing the management processes and controls the FBI has applied to the pre-acquisition phase of Sentinel, we believe that the FBI has adequately planned for the project and this planning provides reasonable assurance that the FBI can successfully complete Sentinel if the processes and controls are implemented as intended. However, we have several concerns about the project that we believe require action and continued monitoring by the FBI, the OIG, and other interested parties. These concerns include: (1) the incomplete staffing of the PMO, (2) the FBI's ability to reprogram funds to complete the second phase of the project without jeopardizing its mission-critical operations, (3) Sentinel's ability to share information with external intelligence and law enforcement agencies and provide a common framework for other agencies' case management systems, (4) the lack of an established EVM process, (5) the FBI's ability to track and control Sentinel's costs, and (6) the lack of complete documentation required by the FBI's ITIM processes. The OIG will continue to monitor and periodically issue audit reports throughout the Sentinel project in an effort to track the FBI's progress and identify any emerging concerns over the cost, schedule, technical, and performance aspects of the project. OIG Recommendations In this initial Sentinel audit, we make seven recommendations for the FBI to help ensure the success of the Sentinel case management system. The recommendations are:
Footnotes
|
| « Previous | Table of Contents | Next » |