Finding 1:  Phase 1 Schedule, Cost, and Performance

Phase 1 of Sentinel, deployed on June 19, 2007, delivered two key components of the FBI’s new information and investigative case management system: a web-based portal to ACS data and workboxes to aid in case management. However, Phase 1 took about 2 months longer than scheduled, cost slightly more than the FBI expected, and delivered less than originally planned. Our audit found four main causes for the delay, including: (1) unrealistic schedule expectations by the FBI, (2) Lockheed Martin’s delays in staffing, (3) challenges Lockheed Martin encountered in integrating COTS software programs to work together as a system, and (4) FBI problems in assessing the project’s progress against the approved schedule.

With regard to deliverables, Phase 1 did not deliver all the commonly understood foundational components of a service-oriented architecture (a concept that we found to be ill-defined), but did deliver the enterprise service bus component that was appropriate for the first phase of the project.³¹ Phase I did not cleanse the data in the electronic case file module of ACS as originally proposed because the FBI said it would be more efficient to do so in Phase 2 of the Sentinel project. Also, the web-based portal developed in Phase 1 did not include the full range of functionality originally anticipated.

Further, as a result of a series of contract modifications, the cost of Phase 1 exceeded the amount originally budgeted for Lockheed Martin although the overall contract value of $305 million did not change. Lockheed Martin’s costs exceeded the revised contract amount of $59.7 million by approximately $4.4 million. However, Lockheed Martin and the FBI agreed that Lockheed Martin would only be paid the amount of the revised budget, a figure that included using the $2 million budgeted for award fees.³² Other factors such as the transfer of requirements to later phases of the project, not tying all deliverables to the requirements, and transferring costs from Lockheed Martin’s budget to the PMO’s budget suggest that the cost of Sentinel ultimately may be higher than originally anticipated.

Schedule Delay

The FBI and Lockheed Martin established April 19, 2007, as the completion date for Phase 1 during the Initial Baseline Review (IBR) in May 2006.³³ During the IBR, held about 2 months after the contract award, the FBI and Lockheed Martin agreed to a schedule for Phase 1, including all the program-level reviews and life cycle management reviews.

By September 2006, the time the Critical Design Review was originally scheduled to be performed, Lockheed Martin’s performance deviated from the schedule to such a degree that the FBI directed Lockheed Martin to postpone the Critical Design Review until October 2006. Specifically, several principal design documents prepared by Lockheed Martin were returned by the PMO without comment because the documents were insufficient.

Despite concerns about the completeness of these key design documents, the FBI allowed Phase 1 to pass the Final Design Review in October 2006. At that time, PMO officials told Lockheed Martin that its schedule for the remaining work was not feasible and directed Lockheed Martin to replan the schedule for the remainder of Phase 1. Lockheed Martin reconsidered the schedule for the remaining work, including the approach to the remaining tasks, the linkage between the remaining tasks, and whether additional resources could shorten the length of time needed to complete the remaining tasks. Lockheed Martin then provided a revised schedule to the FBI. The FBI approved the revised Lockheed Martin schedule, and that schedule remained in effect until March 2007 when the FBI and Lockheed Martin jointly decided that Lockheed Martin would not be able to resolve all of the deficiencies identified during testing in time to meet the Delivery Acceptance Review scheduled for April 19, 2007. A discussion on the reasons for the delays follows.

Causes for Delay

Our audit found the following four primary causes for the delay in the delivery of Phase 1: (1) an unrealistic schedule, (2) delays by Lockheed Martin in fully staffing the project with appropriately experienced personnel, (3) challenges in integrating the various COTS software components to work as a system, and (4) a progress reporting schedule that did not allow the FBI to assess the schedule impact of changes proposed by Lockheed during the course of Phase 1.

Unrealistic Schedule

Prior to formally soliciting proposals for Sentinel, the FBI created a notional schedule that broke the project into four phases, identified deliverables for each phase, and provided an estimated timeline for each phase. Various documents outline the FBI’s concept for Sentinel’s development schedule, including the following timeline.

Notional SENTINEL Schedule
with Capability Deliverables

[Image Not Available Electronically]

Source: The FBI

While the Sentinel Acquisition Plan clearly stated that potential vendors were free to submit proposals with a different number of phases or suggest changes to the deliverables identified in each phase, vendors’ proposals had to meet the FBI’s overall deadline of completion by December 2009. PMO personnel said that Lockheed Martin’s proposal closely aligned with the FBI’s notional schedule in an effort to win the FBI’s business. The PMO personnel also stated that Lockheed Martin’s proposal was completed before it familiarized itself with ACS and that instead of basing the Phase 1 schedule on the FBI’s notional concept of the project, Lockheed Martin should have based the schedule on its own estimates of the time required to achieve the tasks contained in Phase 1.

In addition to broad outlines of the project’s overall schedule, the FBI also dictated certain project milestones. For example, Sentinel’s Statement of Work specified the timing of the Contract Implementation Review, the Requirements Clarification Review, and the IBR. All three of these reviews, described in Appendix 3, were to occur within 60 days of the March 16, 2006, contract award date for Sentinel. Meeting the schedule specified in the Statement of Work would have required that Lockheed Martin staff the Sentinel project shortly after receiving the contract. The Sentinel Program Manager told us that, in retrospect, the timeframes in the Statement of Work were overly aggressive because they did not allow Lockheed Martin enough time to staff the project.

Delays in Staffing

In addition to the aggressive schedule, which gave Lockheed Martin little time to properly staff the Sentinel project, other factors further delayed Lockheed Martin from properly staffing the project. A high demand for personnel with current top secret clearances in the Washington, D.C., area also affected the staffing of the project, as did Lockheed Martin’s underestimation of the level of experience with COTS integration that its personnel needed to have. These issues contributed to both delays in staffing and also higher-than-expected personnel costs.

Almost immediately following the contract award, Lockheed Martin fell behind in its projected staffing levels. For example, by May 2006, less than 2 months into the contract, Lockheed Martin’s spending on personnel according to EVM data was $476,000 less than expected primarily because the project had yet to be fully staffed. According to the FBI, the costs for qualified personnel with top secret clearances were 25 to 40 percent higher than the cost Lockheed Martin projected due to a limited supply of qualified personnel with top secret clearances. The FBI said that shortly after the Sentinel contract was awarded, the Defense Security Service, the organization responsible for performing background investigations and granting clearances, suspended its processing of clearances for all government contractors for 6 months in an effort to clear its backlog of requests for such clearances. As a result of the suspension, the supply of cleared contractor personnel, which was already insufficient to meet demand, dwindled even more. While Lockheed Martin had fewer staff than planned, it had to pay each individual more than expected.

In addition, Lockheed Martin and the FBI also underestimated the level of COTS integration experience that personnel would need for the Sentinel project. In a January 2007 briefing to the FBI’s Associate Deputy Director, the Sentinel Program Manager said that both the FBI and Lockheed Martin based their original personnel cost estimates on the assumption that most of the development work could be completed by recent college graduates, an approach Lockheed Martin had successfully used on a large scale information technology project at the Social Security Administration. Several PMO and FBI Chief Information Office personnel said that throughout Phase 1 of Sentinel, the level of expertise required of the Lockheed Martin staff to deal with Sentinel’s COTS software was not sufficient for the project, although they said that Lockheed Martin eventually added the required expertise. FBI officials said the quality of the Lockheed Martin staff had improved during the first phase, but that additional improvements need to be made if the subsequent phases of the project are going to be successful. Others said that Lockheed Martin should have considered contracting with the software manufacturers who developed the most challenging pieces of software to help with implementation.

According to FBI officials, both the FBI and Lockheed Martin recognize that they underestimated the level of expertise necessary to work on the interface with ACS and integrate the COTS software products which make up Sentinel and a service-oriented architecture (explained in more detail later in this report). According to the FBI, at the time of contract award the division of Lockheed Martin awarded the Sentinel contract had a limited personnel pool from which to draw, and it was not successful in hiring the additional personnel needed in a timely manner. As a result, Lockheed Martin relied more heavily on subcontractors than anticipated and transferred personnel from within other parts of the company.

COTS Integration

Several PMO officials, including the Sentinel Program Manager and Lockheed Martin’s Deputy Program Manager, stated that integrating the various software modules that comprise Phase 1 into a unified system was a major challenge in Phase 1. Several variables affect how a given piece of software will perform, including: the hardware that the software is running on, the settings for that hardware, the settings for the software itself, and the addition of other software on the system and the settings of that software. According to PMO personnel, many of the individual software products can be configured in hundreds of different ways. As a result, there are hundreds of potential issues that can occur when integrating different pieces of software into a system. Consequently, identifying why a particular problem occurred within an integrated system is difficult due to the number of variables impacting the system.

The Lockheed Martin Deputy Program Manager said that while software and hardware manufacturers’ literature, demonstrations, and trade studies provide information about the functionality of their products, hands-on experience is necessary to determine if the general capabilities of a particular COTS product can be configured to implement the specific functions required by a particular customer. According to the Deputy Program Manager, it is virtually impossible to complete a design involving COTS products without hands-on experience with the product and its interfaces. Risk factors for COTS integration include the number of COTS components, the number of interfaces, the complexity of the interfaces, and the amount of knowledge and experience the integrator has with the product. Because Sentinel is using multiple software programs, and more than one software product may be capable of performing a particular function, Lockheed Martin had to decide which product to use based on the advantages and disadvantages of each choice. Most COTS products are complex enough that some amount of manufacturer support is required to use the product efficiently.

Several of these factors appear to have had an impact on Sentinel development during Phase 1 of the project. As discussed previously, Lockheed Martin did not have staff assigned to the project with significant expertise in the software components being used to build Sentinel. Also, the major COTS software manufacturers were not official Lockheed Martin subcontractors, so their input was not sought in the design phase of the project. Once Sentinel encountered problems, obtaining support from the software manufacturers was time-consuming.

Two other factors compounded the general challenge of COTS integration. First, according to the FBI CIO the FBI is using the latest software and technologies in developing Sentinel, and therefore some of the software has bugs that had not previously been identified by the manufacturer. In at least one case, the developer of the software was not aware of the bug until being notified by the FBI and Lockheed Martin. Because this was a new bug, the manufacturer had to research the cause and develop a solution before Lockheed could implement the patch to the software. Second, Lockheed Martin did not develop a COTS integration strategy to describe what approach it would take to overcome the compatibility issues of Sentinel’s various components.³⁴ The Sentinel architect and the independent verification and validation (IV&V) team both cited the lack of an integration strategy as a primary reason for the integration problems Lockheed Martin faced during Phase 1.³⁵

Assessing Progress

Sentinel PMO personnel said that the methodology used by Lockheed Martin to construct the Sentinel project’s schedule made it difficult to assess the progress of the project. Specifically, they said the schedule: (1) overused “hard constraints”, (2) contained logic problems, (3) was not updated accurately, and (4) contained a high percentage of “level-of-effort” tasks.

Hard Constraints

Lockheed Martin utilized project management software that included a program to establish the project’s schedule. Within the schedule, timeframes were set for the completion of specific tasks, and these tasks were entered into the schedule as “hard constraints.” Hard constraints are dates entered into a schedule that require a task to begin or end on a specific date, regardless of any other activity with the schedule. Additionally, if a hard constraint is entered into the schedule for a task, the schedule’s software will assume that the task met the constraint, regardless of whether or not it did. For example, if a task had a hard constraint to be completed by December 1, 2006, unless the date was updated the project software would assume that the task was completed by that date, regardless of the actual progress on the task. Also, if a task has a fixed end date entered into the scheduling software and the task is not actually completed by that date, all of the subsequent phases will not be delayed within the schedule and the completion date for the whole project will not be moved back. Because of this, hard constraints cloud an assessment of the impact of schedule slippages on the project end date.

While the Sentinel Deputy Program Manager stated that hard constraints are helpful in creating a sense of urgency about completing a task, we found that using a large number of hard constraints limited the FBI’s ability to assess the progress being made on the Sentinel project. Several PMO personnel attributed the overuse of hard constraints to Lockheed Martin’s problems hiring and retaining an experienced scheduler, an issue they now believe Lockheed Martin has resolved. At the PMO’s request, Lockheed Martin has removed many of the hard constraints from the Sentinel development schedule.

Logic Problems

PMO officials also cited problems with some of the logic of Lockheed Martin’s schedule. Specifically, they said Lockheed Martin’s schedule did not always accurately reflect the interdependence between tasks and linked some tasks that were not interdependent while failing to link others that were.

Schedule Updates

Assessing the project’s progress was also difficult because Lockheed Martin did not accurately update the schedule as required. Sentinel PMO personnel said a task’s reported rate of completion appeared to be based on Lockheed Martin management’s assessment of what it thought the FBI wanted to hear, rather than an actual assessment of the percentage of the task completed. Two IV&V reports confirmed problems with Lockheed Martin’s maintenance of the schedule, including that the schedule was not updated to reflect activities that occurred in the most recent reporting period and inaccurate reporting on the percentage of completion of tasks. Specifically, the IV&V team found that various tasks would be 99 percent complete for more than one reporting period, allowing Lockheed Martin to claim all but 1 percent of the cost of a task but not claim that the task was completed.

Level-of-Effort Tasks

An FBI policy on EVM issued by the FBI CIO in March 2006 recognizes that some portion of the development of IT systems will be characterized as level-of-effort, meaning that progress toward the completion of a task is measured by time spent on the task rather than progress toward completing the task. Tasks that do not have a defined deliverable, such as project management, are often measured using level of effort. However, because level-of-effort tasks are not tied to a deliverable, it is difficult to determine how much their completion contributes to the overall progress of a project. Therefore, in IT development projects like Sentinel it is prudent to have a schedule with as few level-of-effort tasks as possible. Recognizing this, the FBI’s EVM policy requires a project to receive approval from the Chief of the Project Assurance Unit when planned level-of-effort tasks exceed 15 percent of the total work hours for a project. The FBI’s Project Assurance Unit denied the PMO’s request to exceed the 15 percent threshold for Sentinel. However, the PMO appealed the denial to the FBI’s CIO and received his approval for the planned level-of-effort hours. As shown in the graph below, for every month since the IBR the actual percentage of level-of-effort tasks exceeded the 15 percent threshold established by the FBI. With the exception of July 2006 through October 2006, the budgeted amount of level-of-effort tasks also exceeded 15 percent of the budgeted hours. A high level of both budgeted and actual level-of-effort tasks makes it difficult to accurately assess the progress of a project toward meeting its goals.

Level–of-Effort Tasks as a Percentage
of Total Work Hours
June 2006 through April 2007

[Image Not Available Electronically]

Source: FBI data

As Phase 1 progressed, the percentage of hours budgeted for level-of-effort tasks increased. The following graph shows the increase in level-of-effort hours budgeted for June 2006 through April 2007 at the IBR to the hours budgeted in the Phase 1 budget as of March 2007.

Hours Budgeted for Level-of-Effort Tasks
June 2006 Plan vs. March 2007 Plan

[Image Not Available Electronically]

Source: FBI data

As of April 2007, spending on level–of-effort tasks accounted for 27 percent of the total Phase 1 budget. An FBI analysis of the increase in spending on level-of-effort tasks concluded that most of the increase resulted from higher than expected spending on program management costs. This, in turn, resulted from higher than expected staffing and labor costs.

Tight Schedule Affected Implementation of LCMD Processes

In attempting to meet the Phase 1 schedule, the FBI took risks that affected the quality of some of the Phase 1 deliverables and may have contributed to the schedule delays. Specifically, the FBI allowed Lockheed Martin to begin building Phase 1 before the design documentation was complete, contributing to some of the problems encountered in the testing of Phase 1. The FBI also allowed Phase 1 to progress to the next stage of testing before correcting all critical deficiencies identified in previous stages of testing.

Incomplete Design

In October 2006, the Sentinel PMO allowed Phase 1 to pass through the Critical Design Review despite hundreds of unaddressed FBI comments on the System Design Document and the Interface Design Document, key design documents that provide programmers with the specifics required to build a system to a customer’s specifications. Both the System Design Document and the Interface Design Document are supposed to be finalized and approved during the Critical Design Review.

The PMO knew of these documents’ shortcomings at the time of the Critical Design Review. Lockheed Martin had submitted both documents to the PMO 3 weeks before the Critical Design Review to allow the FBI time to review and comment on the documents. The FBI’s review concluded that the documents did not meet standards for content, completeness, consistency, and quality, and returned the documents to Lockheed Martin for additional work. The FBI directed Lockheed Martin to resubmit the documents and advised Lockheed Martin that it would require 2 weeks to provide formal written comments on the revised plans. Lockheed Martin resubmitted the documents on October 4, 2006, 1 day before the Critical Design Review. As agreed, the FBI did not complete its formal review of the documents until October 18, 2006, 2 weeks after they had been resubmitted. However, the FBI and Lockheed Martin still held the Critical Design Review on October 5-6, 2006, during which time the PMO concluded that Lockheed Martin’s design documentation was sufficient to ensure that the design presented could be produced and when built would meet the design specification.

Similarly, the Technical Review Board – part of the FBI’s Life Cycle Management process described in Appendix 3 – concluded that the system design was not complete or well documented. However, at Sentinel’s Final Design Review in October 2006, the Board granted the PMO conditional approval to proceed with the project but required that six technical issues be addressed, such as the need to provide details on the number of modules and design details. The Board’s approval authorized the development of Phase 1 of the project, while also requiring that the deficiencies found in the design documents be addressed. Of the six conditions the Board cited, four had not been resolved as of April 2007, including three that concerned design documents.

While a temporary delay would have occurred if the FBI required Sentinel’s design to be completed before development began, the time invested during the delay may have shortened the actual time spent on Phase 1. An April 2007 IV&V report stated that “Historically, most deficiencies occurred around the display of information to the Personal and Squad Workbox; prototyping and a complete System Design Document and Interface Design Document would have avoided many of these deficiencies.”

Testing

In an apparent effort to keep Sentinel on schedule, the PMO deviated from sound project management and Sentinel’s Test Evaluation Master Plan. The PMO allowed Phase 1 to pass through two program level reviews – the Product Test Readiness Review and the Site Test Readiness Review – without resolving all high priority deficiency reports as required by the Test and Evaluation Master Plan.³⁶ The IV&V contractor was repeatedly critical of Sentinel’s testing and expressed concern about the impact the testing modifications would have on the stability of the system as it moved forward.

While we believe that adherence to schedule goals is desirable, the risk of project failure appreciably rises if corners are cut and there is not adequate quantitative data to assess the risks to the project of not implementing disciplined processes in critical areas. Ineffective implementation of these processes exposes a project to the unnecessary risk that costly reworks could be required, which in turn would adversely affect the project’s cost and schedule and can adversely affect the ultimate performance of the system. Effective project management requires quantitative data or metrics to assess whether a project plan needs to be adjusted and to determine what oversight actions may be needed to ensure that the project meets its stated goals and complies with agency guidance.

Lockheed Martin’s Cost Performance

The cost of the contract awarded to Lockheed Martin to develop Sentinel is about 72 percent of the total Sentinel budget.³⁷ Therefore, Lockheed Martin’s ability to deliver its portion of Sentinel within budget is critical to the cost performance of the overall project. As the result of a series of contract modifications, the value of Lockheed Martin’s task order for Phase 1 increased from $57.2 million at the time of the integrated baseline review (IBR) to $59.7 million in March 2007. However, in June 2007 Lockheed Martin advised the FBI that it had incurred costs totaling $64.1 million in the performance of Phase 1. Lockheed Martin attributed the cost overruns to unanticipated work in interfacing with existing FBI computer systems and modifications to the FBI’s testing approach.

Specifically, Lockheed Martin said the documentation and functionality of the FBI’s Web-enabled Automated Case Support (WACS), Phoenix, and ACS systems differed from the descriptions provided in the FBI’s Request for Proposals.³⁸ As a result, Lockheed Martin was not able to reuse portions of the WACS, Phoenix, or ACS portal interfaces, as it previously assumed it could. Consequently, Lockheed Martin had to develop extensive amounts of new code to provide the functionality it believed it was going to be able to reuse from these existing systems. In addition, Lockheed Martin said it had to develop an interface control document for ACS. Lockheed Martin estimates that these two issues required 7 weeks of additional effort and cost approximately $3.4 million. The Sentinel Program Manager agreed that the ACS interface control document was not sufficient for Lockheed Martin to do its work, and that the development of a satisfactory interface control document added several weeks to the schedule.

Lockheed Martin said that changes to the FBI’s testing approach during Site Acceptance Testing and User Acceptance Testing also resulted in schedule delays and increased costs.³⁹ Specifically, the FBI’s decisions requiring the closure of all high-priority defects prior to the start of Site Acceptance Testing, increasing the scope of Site Acceptance Testing, adding unofficial User Acceptance Testing, and piloting the system before full deployment cost an additional 3 weeks and $1 million for Phase 1. FBI officials told us there was no change in the testing requirements, but to ensure that the system met the FBI’s specifications, supplementary testing was added after the system failed initial tests.

In June 2007, the FBI and Lockheed Martin agreed that Lockheed Martin would absorb approximately $4.4 million in costs it incurred in excess of the agreed-upon contract amount and that $2 million budgeted for award fees would be used to reimburse Lockheed Martin for costs incurred during the development of Phase 1.

We found that three factors obscured a precise accounting of Lockheed Martin’s cost performance, which we describe below. First, even though the FBI transferred some Phase 1 requirements to later phases of the project, it received cost reductions on Phase 1 from Lockheed Martin for deferring completion of these requirements. Second, the FBI did not adequately define all of the Phase 1 deliverables and did not tie all of the deliverables to the requirements agreed upon for Phase 1. Third, the FBI transferred $2.5 million in materials and services from Lockheed Martin’s budget to the PMO’s budget.

Deferred Requirements

Over the course of Phase 1, the FBI deferred a total of 57 high- and low-level requirements from Phase 1 to later phases. As a result of these deferrals, Lockheed Martin was required to deliver less functionality in Phase 1 than agreed upon at the time the project budget was established. However, the FBI did not require Lockheed Martin to determine the decrease in the amount of time and materials required for Phase 1 resulting from these deferrals, and none of these deferrals resulted in a decrease in the cost of Phase 1.

The majority of these requirements addressed user interface or security capabilities. For example, the capability to display a list of leads that a squad had completed in accordance with current ACS screens was deferred to Phase 2.⁴⁰ Other areas addressed by the deferred requirements include the system’s ability to format, sort, and search data.

Phase 1 Requirements Deferred to Later Phases

According to the FBI, it deferred most of the 57 requirements because it decided the requirement was outside of the scope of Phase 1, did not add value to Phase 1, would require the modification of ACS, or would duplicate a capability included in a future phase of Sentinel. FBI officials said they did not believe it was prudent to invest in upgrading ACS because Sentinel is intended to replace it. We recognize that phased projects using COTS components often transfer requirements from one phase to another and, in general, we do not disagree with the FBI’s transfer of the requirements to later phases. However, we are concerned that the FBI did not require Lockheed Martin to determine the financial impact of deferring this work in Phase 1 and adjust the cost of Phase 1 accordingly.

Phase 1 Deliverables

Throughout the Sentinel project, FBI documents, including slides from weekly briefings to the FBI Director, have shown four major anticipated deliverables for Phase 1: (1) a web-based portal to ACS, (2) a case workbox, (3) the foundational components of a service-orientated architecture, and (4) data cleansing of the electronic case file (ECF) portion of ACS. As implemented, Phase 1 delivered the key ACS portal and case workbox. Phase 1 also delivered the one component of the ill-defined foundational components of a service-oriented architecture that was appropriate for that phase, but did not provide the data cleansing of the ECF portion of ACS.⁴¹ As Sentinel progressed through the life cycle management process, the FBI’s internal technical reports noted this divergence from the original set of deliverables. For example, a technical report issued as part of the design review process noted that “The analysis is based on the ability of the project to meet the SENTINEL Phase 1 goals. Initially, Phase 1 was to include foundational components of a service-oriented architecture and ECF Data Cleansing. These two initiatives were pushed beyond Phase 1 and are not germane to this analysis.” Neither the foundational components of a service-oriented architecture nor the data cleansing of ECF data were specified in the requirements for Phase 1, so the deferral of these goals did not require the deferral of requirements. However, achieving both of these goals may potentially require significant financial and personnel resources. And as mentioned previously, deferral of these goals did not result in a corresponding decrease in the Phase 1 contract amount.

In June 2006, the FBI and Lockheed Martin held an IBR for Phase 1 of Sentinel. One of the goals of an IBR is to agree on the scope of work.⁴² At the IBR, Lockheed Martin identified the following three major goals for Phase 1.

• Portal – a single web-based user interface to the system providing access to ACS functionality, a case “workbox” that summarizes a user’s workload, enhanced search capabilities including saved queries, and a single sign-on within Sentinel.

• ACS ECF Data Cleansing – prepare the ECF portion of ACS data to be transferred to Sentinel in Phase 2.

• Service-oriented Architecture/Enterprise Service Bus – provide the governance of the service-oriented architecture and web services, and preliminary public key infrastructure (PKI) services for Sentinel system administrator login.⁴³

Sentinel Portal

Phase 1 of Sentinel delivered a web-based user interface to ACS data, giving a much more modern look and feel to ACS data and allowing users to navigate through the system using a mouse. However, the portal will not allow users to perform all of the functions in the three modules that make up ACS: ECF, Investigative Case Management (ICM), and Universal Index (UNI). Because many of the functions now performed by ACS will be performed by Sentinel starting in Phase 2, the web portal to ACS will be useful only until the completion of Phase 2. As a result, the FBI did not believe that duplicating all of ACS was cost effective and chose to include only the most frequently used functions in the Phase 1 portal. The table below summarizes the number of functions included in the Phase 1 portal and compares the ACS, the web-enabled ACS (WACS), and Phoenix.

XXXXXXXXXXXXXXXXXXX

Because some functionality can be found only in ACS, such as the uploading of documents, FBI personnel will have to continue to use ACS while Sentinel is being developed. Appendix 5 lists the functionality found in WACS, Phoenix, and Sentinel Phase 1. For those functions not available in any of these three systems, users will have to continue using ACS until the functions are available in a later phase of Sentinel. According to the Sentinel Program Manager, the FBI recognizes that some critical functions, such as uploading documents, are currently available only in ACS and those functions need to be integrated into Sentinel as soon as possible. As part of Phase 2, the FBI plans to enhance Phase 1 to allow users to upload documents.

Data Cleansing

Before Sentinel is completed, 23 million ACS records must be transferred from ACS to Sentinel. The IDP does not include any data cleansing or data migration capabilities for Phase 1. Rather, the IDP states “There are no specific requirements for migration of case data in Phase 1.” However, Lockheed Martin’s proposal included data cleansing of ECF data as part of Phase 1 in preparation for the data’s transfer, or migration, to the Sentinel database in Phase 2. As a result, the FBI added this capability to its description of the Phase 1 capabilities.

Specifically, Lockheed Martin’s proposal described the overall data migration effort as consisting of two basic steps: (1) data cleansing, and (2) data migration. The data cleansing approach described in Lockheed Martin’s proposal yields an Error Assessment Report, which documents all of the data errors found in a system. These errors are then used to build data transformation rules, which improve the quality of legacy data before the data is migrated to the system being built. The FBI agreed to Lockheed Martin’s data cleansing approach at the IBR, and the proposed scope of the data cleansing efforts were built into the integrated master schedule. However, further analysis and coordination with FBI headquarters divisions responsible for ACS and data security led to the conclusion that the full scope of Lockheed Martin’s proposed data cleansing effort could not be implemented in Phase 1 and that the actual data cleansing would have to be deferred to a later phase.

FBI personnel told us that the FBI deferred data cleansing until Phase 2 because of technical concerns the FBI had with cleansing data in advance of migrating it. Specifically, the FBI was concerned that synchronizing the cleansed data with the actual ACS records at the time of migration would be difficult. Since the FBI would continue to add and modify data between the time of the data cleansing and the time the data would be migrated to Sentinel, the two sets of data would be different and resolving those differences presents significant challenges. FBI officials also had concerns about where, or within what system, to store the large volume of data in the interim between cleansing and migration.

The final Data Migration Plan described the Phase 1 data cleansing efforts as testing and prototyping only. The goals of the Phase 1 data cleansing efforts are described below.

• Complete installation, configuration, and testing of the Phase 1 data migration environment in the FBI’s testing and quality assurance environments.

• Complete performance benchmarking of ACS data access using the FBI’s testing and quality assurance environments.

• Complete testing, integration, and validation of COTS products using the FBI’s testing and quality assurance environments.

• Complete testing, integration, and validation of the target word search filtering capabilities of two COTS products using the FBI’s testing and quality assurance environments.

• Complete error assessment report generation testing using the FBI’s testing and quality assurance environments.

• Initiate ECF data analysis and migration planning for Phase 2.

• Initiate installation, configuration, and testing of Phase 2 data migration environment in the FBI production environment.

The Data Migration Plan states that data quality assessment, data migration, and data cleansing of ACS ECF production data will not occur until Phase 2. However, the FBI did not conduct, or have Lockheed Martin conduct, a cost assessment of the reduction in scope of the Phase 1 data cleansing on Phase 1 or Phase 2. While the scope of the Phase 1 data cleansing activities decreased, there was no corresponding decrease in the cost of Phase 1. Similarly, while some data cleansing activities were deferred until Phase 2, there was not an increase in the cost of Phase 2. This change of scope occurred without a change to the Phase 1 requirements because the System Requirements Specifications does not contain any requirements that specifically address data cleansing or migration. Therefore, there were no requirements allocated to Phase 1 and no requirements deferred to Phase 2. According to the Sentinel statement of work, “The contractor shall perform all activities (extract, translate and error correction, load) necessary to migrate legacy data needed for the operation of capabilities delivered in each Phase.” Because Lockheed Martin’s proposal included ECF data cleansing in Phase 1, Lockheed Martin effectively created a derived requirement, one that was deferred until Phase 2 without any cost implications.⁴⁴

FBI officials said that while Lockheed Martin had not done any actual data cleansing during Phase 1, it had made substantial progress on the data cleansing task, including configuring and testing the data cleansing software, setting up the data cleansing facility that will be used during Phase 2, and establishing data cleansing procedures to adjudicate the resolution of problem data.

Service-Oriented Architecture

A service-oriented architecture is a collection of services that communicate with each other. The communication can involve a simple data exchange or two or more services coordinating on an activity. Common components of a service-oriented architecture include a metadata repository, governance, a service registry, and an enterprise service bus, all of which are described below.

• Metadata repository– a database of data descriptions which is intended to provide consistent and reliable access to data.

• Governance– the mechanisms and policies to control what services are available via a service-oriented architecture and to whom they are available.

• Service Registry - the infrastructure for building, deploying, and discovering services on a service-oriented architecture.

• Enterprise Service Bus– software “middleware” that connects software components and allows the components to communicate with each other.

Of these four components, Phase 1 delivered an enterprise service bus. However, none of Sentinel’s requirements specifically address the service-oriented architecture, and therefore the FBI could not validate whether Lockheed Martin delivered the foundation of a service-oriented architecture, one of the stated goals for Phase 1. The FBI’s Incremental Development Plan (IDP), which was provided to all potential Sentinel bidders as a framework from which to describe the intent of the Sentinel program, states that Phase 1 “may include an Enterprise Service Bus (ESB) and foundation services,” but does not further describe what is meant by either “foundation services” or “foundational components.” According to the FBI, the definition of “foundational components of a service-oriented architecture” was introduced during an internal review of the Sentinel design and was not specifically shared with Lockheed Martin as a goal, objective, or requirement of Phase 1. The FBI said that as a result, it had no expectation that Lockheed would specifically address the commonly recognized basic components of a service-oriented architecture in Phase 1. FBI officials said that Phase 1’s enterprise service bus and web portal are both foundational components of a service-oriented architecture and that the enterprise service bus was the only one of the four common foundational components that was applicable to Phase 1.

At the IBR, according to Lockheed Martin the scope of work for Phase 1 included governance of Sentinel’s service-oriented architecture, web services, and preliminary public key infrastructure (PKI) services for the Sentinel system administrator login.⁴⁵ The FBI and Lockheed Martin also agreed the scope of the Phase 1 service-oriented architecture would include the following: define and document the service-oriented architecture including the enterprise service bus, portal, access control framework with single sign on, ACS access, services management/governance, data protection/ distribution/ recovery, and workboxes.

However, at the time of the Phase 1 Preliminary Design Review, the PMO told the FBI’s Technical Review Board that the “core service-oriented architecture implementation design/components will be presented in Phase 2,” and “[t]he enterprise service bus description, design, architecture will be introduced during Phase 2.” In addition, the PMO said, “Please note that the bulk of service-oriented architecture information will come during Phases 2, 3 and 4.” The technical report done for the combined Deployment Readiness Review and Site Testing Readiness Review states that a service-oriented architecture “could not be completed in Phase 1” and that it was deferred to a later phase.

The System Requirements Specifications do not delineate requirements for Sentinel’s service-oriented architecture or the foundational components of a service-oriented architecture. Because the System Requirements Specifications are the source for requirements allocation by phase to supporting program documents, such as the Requirements Clarification Document (RCD), the RCD did not allocate any service-oriented architecture related requirements to Phase 1. Without any service-oriented architecture requirements, the FBI cannot test whether Phase 1 met the stated service-oriented architecture objectives.

Transferred Costs

Through a series of six contract modifications, the FBI increased the total contract value of Phase 1 by $2.5 million, from $57.2 million to $59.7 million, but the overall contract value of $305 million did not change. As expected in a project of Sentinel’s size and complexity, some of the modifications increased the scope of Phase 1, while others decreased it. However, the decreases either transferred the cost for the tasks to the Sentinel PMO budget or to the amount budgeted for Lockheed Martin’s award fee. For example, in March 2007 the FBI issued a modification which deleted tape silos (computer hardware that uses tapes to store large amounts of computer data) from the Phase 1 contract. Although the tape silos were still necessary for Phase 1, the FBI purchased silos with more storage capacity with funds from the PMO’s budget and used the funds originally allocated to the tape silos to offset the cost of additions to the scope of Phase 1. The table below shows the deletions in scope to the Phase 1 task order.

Reductions in the Phase 1 Task Order

The increases in scope to Phase 1 were for items that Lockheed Martin believed the government was obligated to provide, subsequent phases of Sentinel, the operations and maintenance of Phase 1, or requirements added by the FBI. The following table shows the additions in scope to the Phase 1 task order.

Additions to the Phase 1 Task Order

Conclusion

Phase 1 delivered the two key components of Sentinel, the web-based portal and the workboxes. However, it did not include all the deliverables planned for Phase 1. While we cannot yet assess the full impact of deferring some of the original Phase 1 deliverables to subsequent project phases, we believe that deferring these deliverables may add cost and schedule pressures to subsequent phases of the project. In addition, we question why cost adjustments did not occur in Phase 1 due to reduced requirements.

The FBI’s ITIM framework and the Sentinel PMO both establish processes that should enable the FBI to avoid the problems that occurred in the failed Trilogy project, but rigorous implementation of these processes is necessary for the FBI to reduce the risks it faces in implementing Sentinel. The schedule and cost issues the FBI encountered during Phase 1 demonstrate that inadequate implementation of any of the disciplined processes in systems development can reduce or overcome the positive benefits of other processes. For example, the FBI’s decision to allow Lockheed Martin to begin building Phase 1 without completing the design documents likely had an impact on the time required for testing and the number of defects found during testing.

The FBI is currently reexamining whether the current development approach is best for the subsequent phases of Sentinel. Other approaches feature shorter phases and inherently lower the risk of spending substantial amounts of money on a deliverable that does not meet the customer’s needs. However, shorter phases may increase the overall cost and schedule of Sentinel. In our judgment, while the FBI must continue to closely manage Sentinel’s cost and schedule, the FBI’s primary consideration in reevaluating Sentinel’s development approach should be delivering a system that meets the needs of the FBI regardless of how many discrete project phases may be needed to accomplish that goal.

Recommendations

We recommend that the FBI:

1. Reconsider the four-phase approach to developing Sentinel to limit the scope of future phases to allow them to be completed in 9 months or less.

2. Negotiate decreases in the cost of future phases if requirements are deferred in that phase.

Finding 2:  Phase 2 Planning and Management Issues

The FBI has implemented several management controls and processes that are designed to help it adequately manage the development of Sentinel and bring it to a successful conclusion. We reviewed four of these controls and processes in-depth: (1) EVM, (2) independent verification and validation (IV&V), (3) risk management, and (4) bill of materials. We found that the FBI has made significant progress in each of the four, but that additional progress needs to be made in the implementation of EVM, risk management, and the bill of materials.

In addition to these controls and processes, we found that the FBI has identified lessons learned during Phase 1 of the Sentinel project. We examined six key lessons that, if acted upon, will aid the FBI in planning Phase 2 of the Sentinel project and reduce risks. We also reviewed the status of the recommendations we made in our previous two reports on the Sentinel project and found that the FBI is taking action to resolve our concerns. We have closed 5 of the 12 prior recommendations and note that the FBI agreed with the remaining recommendations and was in the process of taking corrective action. These recommendations dealt generally with the FBI’s need to complete the required planning and general management oversight policies and procedures for Sentinel in order to help ensure its success.

Management Controls and Processes

The FBI has established four important management controls and processes that, if implemented correctly and fully, will help the FBI better manage the Sentinel project. However, the FBI needs to improve its implementation of three of the controls and processes to ensure their effective use.

Earned Value Management

As we reported in December 2006, the FBI established an EVM system for Sentinel as required by OMB, and our current audit found that the FBI is continuing to implement it. EVM helps manage project risks by producing reliable cost estimates, evaluating progress, and allowing the analysis of project cost and schedule performance trends. EVM compares the current status of a project, in terms of both cost and schedule, to the established cost and schedule baselines. Deviations between the baselines and the current status demonstrate the project’s progress and the overall level of performance, thereby enabling a level of accountability to be imposed on the project. When properly implemented and utilized, EVM allows project management to pinpoint potential problems and address them before they escalate.

According to the FBI’s EVM plan, the Sentinel PMO will use the plan to measure both its and Lockheed Martin’s earned value performance and report the results to oversight entities. The Sentinel project’s Statement of Work requires vendors and contractors to fully implement EVM in accordance with the plan, including having an EVM system of its own that complies with American National Standards Institute (ANSI) /Electronic Industries Alliance (EIA) Standard 748-A.⁴⁹ This allows the FBI to gather EVM data on the development portion of the project from Lockheed Martin through monthly electronic data transfers from Lockheed Martin.

Our review of EVM reporting from September 2006 to March 2007 showed that the FBI had continued to make efforts to implement EVM and use EVM data to help manage Phase 1 of Sentinel. Although the FBI’s implementation of EVM was consistent with the Department’s guidance, several issues decreased the effectiveness of EVM as a tool to manage the Sentinel development contract.

Reliability of EVM Data

The most significant issue was the reliability of the EVM data Lockheed Martin provided the FBI. In June 2007, the FBI rejected Lockheed Martin’s April 2007 EVM data after Lockheed Martin notified the FBI that it estimated that it had incurred approximately $64.1 million in costs during Phase 1 of Sentinel. Because the EVM baseline for Phase 1 was $59.7 million, Lockheed Martin’s estimate showed that its EVM system was not collecting accurate data on Sentinel costs as Lockheed Martin was accruing the costs – one of the primary purposes of an EVM system. While Phase 1 was behind schedule, the EVM data indicated that the cost would not exceed the baseline of $59.7 million. In the last EVM report released before the end of Phase 1, the FBI estimated that on April 19, 2007, the original planned end date, there would be enough unallocated funds to pay for an additional 5 weeks of effort by Lockheed Martin. In addition, the EVM data used for the March 2007 EVM report indicated that the project was on budget for the amount of progress made.

Changes in Performance Measure Baseline

In addition, Lockheed Martin and the FBI agreed to several changes in the performance measurement baseline, including two significant reallocations (also called “replanning”) of resources within the original baseline amount of $57.2 million.⁵⁰ To remain compliant with the ANSI/EIA Standard 748-A and OMB direction, the FBI must control changes to the performance measurement baseline. To clarify what types of changes are appropriate and define the process for authorizing changes, the Department’s Office of the CIO (OCIO) developed and distributed guidance on revisions to a project’s performance measurement baseline. Once the baseline has been established, any changes to it must be managed through a documented change control process. If a change is authorized, it should be incorporated into both the project’s budget and schedule in a timely manner. Any changes to the baseline must be recorded prior to the commencement of the new work in the baseline revision.

The Department’s guidance describes four categories of baseline revisions: (1) internal replanning, (2) customer-directed change; (3) application of management reserve; and (4) reprogramming.

• Internal replanning involves adjustments to future work in the baseline as long as the adjustments do not affect the total scope of work, baselined cost, or scheduled completion of the project. Replanning, which requires the program manager’s authorization, should not result in any changes to the total amount authorized or key schedule milestones. Replanning may include revisions such as:

• adjusting future work between work packages within the cost and schedule constraints of a single control account;

• adjusting future work between control accounts; and

• distributing undistributed budget to control accounts.

• Customer-directed changes include contract changes and modifications that typically add or remove scope to the original performance measurement baseline. Funds from contract changes and modifications should be distributed to control accounts as soon as possible. While the contracting officer authorizes the contract modification, the program manager authorizes the change to the performance measurement baseline. A customer-directed change may result in an increase to the performance measurement baseline and may require the adjustment of key milestones.

• The management reserve, a portion of a project’s budget set aside to cover any unanticipated costs of a project’s scope of work, is not part of the project management baseline and therefore its use requires a change to the baseline. The program manager can authorize the use of the management reserve, which may result in an increase to the performance measurement baseline. However, the Department CIO should authorize any changes to a project’s key milestones.

• Reprogramming revises the project baselines. This typically takes place when project performance deviates significantly from the original plan – usually the cost and time necessary to complete the project’s remaining work exceeds the budget and schedule. Reprogramming sets planned value and earned value equal to actual cost, thus eliminating any cost and schedule variances. However, reprogramming should always be accompanied by a thorough replanning of the remaining work and should never be implemented solely to eliminate current variances. Reprogramming is authorized by the Department CIO and usually results in changes to the performance measurement baseline and the adjustment of key milestones.

Customer-directed changes, applying the management reserve, and reprogramming usually change the amount of the project management baseline since these types of revisions adjust a project’s scope, cost, or schedule. Customer-directed changes and application of the management reserve do not have any effect on the cost variances or schedule variances. Reprogramming sets planned value and earned value equal to actual cost, thus eliminating any cost and schedule variances. The Department’s guidance refers to reprogramming as an extreme measure that should be applied when, due to performance issues, all key stakeholders agree that the original plan was either seriously flawed or is no longer practical. Any baseline revisions, regardless of the type, should be documented according to a change control process compliant with the ANSI/EIA-748A guidelines.

We reviewed the revisions to the Sentinel Phase 1 performance measurement baseline and found that the FBI and Lockheed Martin revised the project management baseline via internal replanning, customer directed change, and application of the management reserve. In addition, in a revision very similar to the application of management reserve, Lockheed Martin and the FBI agreed to transfer about $483,000 in potential award fees to supplement the performance measurement baseline.

The most substantial of the revisions occurred in August 2006 when, as part of a replanning effort, the FBI transferred about $15 million from the distributed budget to the undistributed budget and, through a transfer from the management reserve, increased the performance measurement baseline by $1.6 million. Almost all of the transfer to the undistributed budget was the result of a decrease in the amount budgeted for equipment. FBI officials said changes in the planned equipment and uncertainty about other equipment necessitated the transfer. This replanning effort also delayed the timing of equipment purchases and moved the planned value associated with those purchases. As shown below, the revisions had a significant impact on the project’s planned value, the basis for measuring a project’s progress against its schedule.

Original Planned Value vs. Revised Planned
April 2006 through May 2007

[Image Not Available Electronically]

Source: OIG analysis of FBI data

Compared to the original plan, these revisions reduced the amount the FBI planned to accomplish from August 2006 through February 2007. In July 2006, the EVM data showed Lockheed Martin’s progress was about 10 percent behind schedule. By October 2006,
2 months after the replanning, the EVM data showed that Lockheed Martin was only 1 percent behind schedule. If the original plan had been in effect, Lockheed Martin would have been 24 percent behind schedule. However, neither the prior cost variance nor the prior schedule variance was set to zero as part of this replanning.

As previously discussed, the FBI issued a series of four contract modifications which increased or decreased the scope of work and the contract amount. These contract modifications also required revisions to the performance measurement baseline and the respective cost accounts. In addition, the modifications funded planning for Phase 2 and preparation for Phase 1 operations and maintenance, both of which are outside of the scope of the development of Phase 1. In our judgment, frequent and numerous changes to the scope of work make EVM data less reliable. The Department’s CIO agreed and said that revisions to the performance management baseline are expected but frequent revisions make it difficult to determine what the EVM data is measuring against.

The FBI’s implementation of EVM comports with the Department’s guidance and aids the FBI in managing the project, but it does not provide all the data the OMB believes necessary for oversight purposes. OMB officials said it would be helpful to them to be able to compare Sentinel’s performance to the original baseline. As a result of OMB concerns that the FBI reprogrammed or rebaselined Phase 1 of Sentinel without the required OMB approval, we reviewed all changes to the time-phased budget used to measure Sentinel’s progress. We concluded that the FBI had not rebaselined the project, but that its frequent replanning diminished the quality and usefulness of the EVM data for higher-level oversight.

Independent Verification and Validation

Inadequate implementation of any one of the disciplined processes in systems development can significantly reduce or overcome the positive benefits of others. When this happens, it is important to act promptly to address risks so as to minimize their impact. One way to monitor the processes used in systems development is to implement an IV&V process. In September 2006, the FBI hired Booz Allen Hamilton (Booz Allen) to perform the IV&V function for the Sentinel project. Since then, Booz Allen has participated in FBI-only project meetings and joint FBI-Lockheed Martin project reviews. In addition, Booz Allen has provided written comments and recommendations on many project documents, and produced 15 project status briefings and monthly reports. Booz Allen also produced monthly reports and biweekly briefings that were sent directly to the FBI’s CIO.

These reports and briefings highlighted recent activities, upcoming events, and Booz Allen’s view of the overall status of the project, including a discussion of risks that could affect the project’s cost, schedule, or performance. The IV&V products also included recommendations and best practices. As of May 2007, Booz Allen had made over 70 recommendations based on risks and other areas it identified. Booz Allen also reported several project management and oversight weaknesses that increased the risks associated with Sentinel, including the following:

• Design Incompleteness - In December 2006, the IV&V team noted that the PMO staff made several hundred comments on System Design Document and the Interface Design Document.⁵¹ Due to the level of effort and time required to address the comments in the documents, the PMO and Lockheed Martin decided not to address these comments in the final design documentation prior to development, but to address the comments in the Software Product Specification, which describes the specifications of a system as it was actually built. Booz Allen was concerned that the developers would create Sentinel without a complete understanding of the agreed-upon design. Booz Allen recommended that the FBI categorize and prioritize the comments, and then requested that Lockheed Martin incorporate the high-priority comments into revised documents.

• Test Procedures - In December 2006, Booz Allen reviewed the test procedures submitted by Lockheed Martin in preparation for the Product Test Readiness Review (PTRR), which assesses the readiness of a system for formal testing. (See Appendix 3 for a description of a PTRR.) Booz Allen found that 90 percent of procedures were unsatisfactory for performing dry-run or system-level tests for one or more of the following reasons: unclear test objectives, inadequate text description of the test, insufficient details on the test procedures, inadequate description of the results expected from the test, or unclear user roles. Booz Allen warned that inconsistent test procedures could result in inconclusive evidence that the system met requirements and that the project could be delayed since a verified set of test procedures would be completed before proceeding to formal testing. Booz Allen made a series of recommendations to improve the test procedures and recommended that the FBI and Lockheed Martin reexamine the testing schedule to determine whether the needed improvements to the test procedures would cause a delay in the start of formal testing.

In addition, as of May 2007 Booz Allen considered the following significant issues open and was tracking their resolution.

• Testing - In April 2007, Booz Allen noted that the Sentinel system had been tested with a maximum of only 217 concurrent users, a load which Booz Allen viewed as unrealistic. Because the limited load was not representative of the user activity Sentinel would face once it was deployed, Booz Allen concluded that Sentinel was vulnerable to performance degradation – such as increased response times – during periods of high workload. Booz Allen recommended load testing with a number of users similar to the load that would be placed on Sentinel once it was deployed nationwide. Booz Allen believed this additional testing was necessary to identify and fix any performance related errors prior to the production release.

• Physical Architecture – In April 2007, Booz Allen identified the proposed hardware system as a potential vulnerability due to concerns about the system’s ability to be available to users whenever needed. Specifically, Booz Allen believed that the system did not have the redundant components needed to keep it up and running should a server fail. Booz Allen recommended enhancing the current hardware architecture to ensure higher availability, reliability, and performance of the Sentinel system.

• Earned Value Management – Lockheed Martin was not contractually required to generate variance analysis reporting at the control account level. As a result, Booz Allen reported that Lockheed Martin did not provide in-depth insight into the cost and schedule drivers for the discrete technical areas. While the PMO required Lockheed Martin to complete a “root cause” analysis when a substantial variance occurred for the program as a whole, it did not require one for variances in major control accounts. Booz Allen was concerned that a high-level analysis of the program’s variances from the plan would not allow the FBI to completely evaluate the technical risk posed by the variance or Lockheed Martin’s proposed strategy for eliminating the variance. In addition, Booz Allen had the following EVM concerns:

• Activities in the Integrated Master Schedule (IMS) without Work Breakdown Structure Alignment – Booz Allen identified several activities in the IMS without baseline dates and with no cross-reference to the work breakdown structure.⁵²

• Inaccurate Statusing (reporting of level of completion of an activity) – Booz Allen reported that the IMS showed that Lockheed Martin had claimed that some tasks were completed when they were not. For example, Lockheed Martin reported that it delivered the Security Vulnerability Risk Matrix on March 12, 2007, and recorded the task as 100 percent complete in the IMS. However, the matrix Lockheed Martin submitted was essentially a copy of the FBI template and contained no data. Booz Allen expressed concern that the ground rules for taking credit for meeting a task may only require deliverable submission and not address quality, acceptance by the FBI, and rework.

• IMS Logic Conflicts – These types of IMS constraints limit impact analysis. Activities with “hard constraints” are likely to adversely impact the critical path. Booz Allen noted that the IMS used “hard constraints” such as “Must Finish On.” The use of hard constraints created conflicts in the underlying logic of the schedule. For instance, the use of “Must Finish On” forced the scheduling software to record that a given activity finished on the “Must Finish On” date, regardless of whether that activity was finished early or had not yet been completed. Without an accurate record of when the task is completed, it is difficult to assess when tasks that depend on the completion of the first task can begin.

The IV&V process has resulted in numerous Booz Allen recommendations related to Sentinel Phase 1 development. We believe that Booz Allen has been able to identify areas of concern, explore them, develop recommendations and inform FBI decision makers in a timeframe that allows for meaningful changes to the project. See Appendix 6 for a list of Booz Allen’s IV&V recommendations.

Risk Management

The FBI has instituted a risk management process to identify and mitigate the risks associated with the Sentinel project. The risk process is managed by the Sentinel Program Manager and a Risk Review Board, which, according to the Risk Management Plan, is supposed to meet biweekly.⁵³ The most significant risks identified by the board are examined at monthly Program Management Review sessions and other Sentinel oversight meetings in accordance with the FBI’s Life Cycle Management Directive (LCMD).⁵⁴

The purpose of risk management is to assist the program management team in identifying, assessing, categorizing, monitoring, controlling, and mitigating risks before they negatively affect a program. A risk management plan identifies the procedures used to manage risk throughout the life of the program. In addition to documenting the risk approach, the plan focuses on how the risk process is to be implemented; the roles and responsibilities of the program manager, program team, and development contractors for managing risk; how risks are to be tracked throughout the program life cycle; and how mitigation and contingency plans are implemented.

According to the Sentinel Risk Management Plan, risks are to be identified, managed, and tracked throughout the life of the project. When a proposed risk is brought before the Risk Review Board, the board’s voting members decide whether or not to accept the risk as an “open” risk and, if accepted, vote on the severity the risk will have on the project’s cost, schedule, and performance and the probability the risk will occur. Risks brought before the Risk Review Board are documented in a risk register, which includes the following:

• description of the risk,
• impact on the program should the risk occur,
• phase of Sentinel affected by the risk,
• person responsible for managing the risk,
• OMB risk category,
• severity of the risk as voted by the Risk Review Board,
• probability the risk will occur as voted by the Risk Review Board,
• strategy to mitigate the risk,
• risk status,
• contingency trigger, and
• contingency plan.

The risk register lists open risks in rank order based on the risks’ probability and severity ratings. The PMO is responsible for tracking and periodically reviewing risks that are closed or resolved to prevent recurrence and to document the effectiveness and any unintended consequences of the mitigation strategy employed. Generally, Sentinel’s mitigation strategy has been to develop a series of actions that will decrease the probability a risk will turn into an issue or to reduce the severity of a risk’s impact on Sentinel.

Program risks include risks that are identified and managed by the development contractor as well as risks that can only be identified and managed by the FBI. This requires that risk management be performed by the vendor and subcontractors to identify risks from the contractor perspective, and by the FBI program management team to identify risks from the FBI’s perspective.

As of April 2007, the FBI had identified and was managing 16 open risks to the Sentinel program, including the following top-ranked risks:

• Losing key contractor work force members due to overwork;

• The source of authoritative attributes are not known or available by the end of Phase II planning;

• The difficulty in cleansing data from phased-out legacy systems may have been underestimated; and

• The difficulty in migrating data from phased-out legacy systems may have been underestimated.

Risk Management Challenges

We view the FBI’s ability to interface with other FBI IT systems from which it will extract data as a potentially significant challenge. The project manager said that during the planning stage of Sentinel, he was assured by the FBI that the documentation about ACS that Lockheed Martin needed in order to develop Sentinel existed. However, as discussed previously, Lockheed Martin had unanticipated problems with the ACS system documentation. In order for Sentinel to be able to extract information from ACS files, Lockheed Martin had to reverse engineer ACS to create satisfactory system documentation, which was then used to build the necessary interfaces between the two systems. According to the project manager, as part of the documentation re-creation Lockheed Martin had to develop new documentation for approximately 71 modules. The unexpected project took 4 weeks to design and an additional 4 weeks to develop. As a result, the Sentinel Program Manager has identified the potential lack of adequate documentation for other systems’ interfaces and business processes as a risk to the remaining phases of Sentinel. This risk has the potential to impact both the budget and schedule of the Sentinel project.

We also view the FBI’s ability to prepare the data for transfer from ACS into Sentinel as a significant challenge. Much of the work to be done for data cleansing will depend on the condition of the data in ACS, which is the data to be cleansed then migrated to Sentinel. According to the FBI, one significant problem in ACS is that the data fields do not check or correlate. This means that data in a given field may not represent the data expected in that field. For example, “XXXXX” could be entered in the field for recording the date an item was entered into ACS. If a large amount of data in ACS is in this or a similar condition, this could have a significant impact on Sentinel’s development schedule.

Migration of data from ACS to Sentinel is another potential significant challenge. There are more than 23 million records in ACS that will need to be migrated to Sentinel, and a major problem for the data migration is that ACS data is linked by serial numbers. One of the challenges will be to keep the data together during migration. In addition, the timing of this data migration will be crucial. During data migration there needs to be a minimal number of new ACS entries; otherwise migration can never be fully completed. One idea the FBI is considering is to make ACS inaccessible for a period of time during this migration. Two other options are to turn off the electronic case file (ECF) on ACS and only allow entries into Sentinel while still using the ACS user interface or to flag ECF data when it has been accessed and queue the flagged data for migration into Sentinel within 24 hours.

Our review of the April 30, 2007, risk register showed that the majority of the 16 open risks are most likely to affect the second phase of the Sentinel project.⁵⁵ As shown in the following chart, the Risk Review Board classified 15 of the 16 (94 percent) risks as having a potential impact on Phase 2.⁵⁶ Appendix 7 lists the 16 risks in order of priority as well as the phase of Sentinel they could affect.

Open Risks by Sentinel Phase

Risk Mitigation Strategies

According to the Risk Management Plan, a risk mitigation strategy should be developed for each open risk to eliminate or reduce the probability or impact of the risk on Sentinel’s success. According to the Risk Management Plan, risk mitigation strategies should include the following information:

• description of the mitigation activity (required),

• organization and individual responsible for executing the mitigation strategy (required),

• measure of effectiveness (required),

• schedule of activities with completion dates, and

• resources required and cost estimates of additional activities

The Risk Management Plan requires a mitigation strategy for any open risk assessed with at least a medium impact or probability of occurrence and calls for the mitigation strategies to be recorded in the risk register. The strategies should be reviewed and revised during team status meetings and risk management meetings.

We reviewed the April 30, 2007, risk register and found that the FBI had developed mitigation strategies for the 9 risks for which a strategy is required.⁵⁷ However, we found that mitigation strategies for all 9 were incomplete because none of them included a description of how the effectiveness of the mitigation strategy would be measured. In addition, none of the mitigation strategies described the resources required to implement the strategy or the cost of its implementation.

Contingency Plans

According to the FBI’s risk management plan, the Sentinel PMO should develop a “contingency trigger” and a contingency plan for each risk it is managing that has a probability or severity rated as at least medium by the Risk Review Board. A contingency trigger is an event that would convert a risk into an operational issue and cause the FBI to implement a risk’s contingency plan. However, we found that the April 30, 2007, risk register included a contingency trigger and contingency plan for only 3 of the 9 risks required to have a contingency plan. In addition, only 2 of the risks ranked in the top 5 risks had a contingency trigger or plan. Yet, while the FBI is not fully complying with the risk management plan’s contingency trigger and contingency plan requirements, we found that the FBI’s compliance has improved since our previous audit in December 2006.

The Sentinel Risk Manager said that the Phase 1 schedule delays had caused the Sentinel PMO to focus its efforts on avoiding further delays and that the discipline required to maintain the risk register had been diverted to other tasks. That explanation notwithstanding, we believe there should be a contingency plan developed for each major risk having the potential to result in a significant cost, schedule, or performance deviation from the project baselines.

Closed Risks

According to the Sentinel Risk Management Plan, risks that are resolved or closed will continue to be tracked and periodically reviewed to prevent occurrence and to document effectiveness and unintended consequences of the mitigation strategy employed. We reviewed the 68 closed risks in the April 30, 2007, risk register and found no evidence that the mitigation strategies for 32 of the risks had been fully implemented. We recognize that full implementation of a risk’s mitigation strategy may not be necessary to prevent a risk from occurring. However, none of the entries for the 32 risks indicated that full implementation of the mitigation strategy was not warranted. To maintain the integrity of Sentinel’s risk management process and to show that the FBI has effectively addressed all closed risks, we believe that the basis for closing risks should be clearly documented in the Sentinel risk register.

Risk Register Maintenance

The Sentinel Risk Management Plan requires that the Sentinel Risk Review Board meet at least biweekly and that the risk register be updated after each meeting. We found during the period from August 11, 2006, through May 4, 2007, the Sentinel Risk Review Board met 16 of the required 20 times. During this 8-month period, we identified 4 periods when the Risk Review Board went 3 weeks or more without meeting. The Sentinel Risk Manager told us holidays and major program reviews caused the lapses in the standard meeting schedule.

In addition, we found other problems with the maintenance of the risk register.

• The risk register did not always accurately document the date of the Risk Review Board meeting, suggesting to us that FBI personnel could not determine whether they were looking at the most recent risk information.

• Some risk registers with different dates were exact duplicates, indicating to us that the risk information had not been updated.

When the FBI does not adhere to the disciplined processes required by the Sentinel Risk Management Plan, the project becomes increasingly vulnerable to risks. For Sentinel to succeed, risks must be diligently monitored and managed, and accurate data on the risks facing Sentinel must be available to the PMO staff as well other oversight organizations both within and outside the FBI.

Risk Ownership

When a new risk is opened, the Sentinel Risk Review Board assigns an owner to that risk. Risk owners are required to: (1) draft the risk mitigation plan; (2) determine contingency triggers, (3) draft contingency plans to reduce the impact of a risk condition actually occurring, (4) report on statistics, and (5) ensure that mitigation plan is implemented. At the time of our audit, most risk owners headed one of the units that make up the Sentinel PMO. Most of the risk owners also sat on Sentinel’s risk review board. We support the idea of having one person taking a lead role in managing each risk. However, as discussed below, the process of risk ownership does not appear to be functioning as intended. Specifically, during interviews with risk owners we found that some:

• could not explain the nature of the risks they had been assigned,

• were not the most active in designing or implementing the strategy for mitigating the risk,

• thought they did not have the expertise to fulfill their role as risk owner, or

• thought they did not have the authority or capability to implement a risk’s mitigation strategy.

In addition, the responsibility for managing Sentinel’s risks is disproportionate; two people were responsible for the majority of the risks open as of April 2007. In our judgment, risks cannot be adequately managed if the personnel assigned cannot do not have the required expertise or cannot devote a sufficient amount of time to a particular risk.

Issue Tracking

According to the Sentinel Risk Management Plan, risks that are considered unavoidable, and therefore its mitigation strategy options are outside of Sentinel program management control, are to be assigned as an issue and tracked and reported accordingly. If a risk is identified as an issue by the Sentinel PMO, it is transferred from the open risk section of the risk register to the closed risk section, where it is no longer actively managed. However, the FBI has not implemented a system to track issues and their resolution. Sentinel officials told us that they are aware of the need to actively manage issues and are considering different methods to do so.

The FBI has made significant progress in implementing a program to manage the risks that threaten Sentinel’s cost, schedule, and performance. However, the effectiveness of the risk management program depends on disciplined implementation, and we found that the FBI has not always adequately implemented the processes necessary to reduce the risks that Sentinel faces. With respect to currently identified project risks, we view the FBI’s ability to interface with the systems from which it will extract data as a potentially significant challenge. In Phase 1, Lockheed Martin had unanticipated problems connecting Sentinel with ACS because there was no detailed documentation describing how ACS works. Consequently, Lockheed Martin had to create the documentation itself. The Sentinel PMO did not anticipate this task because the managers of ACS told the PMO that all of the necessary documentation existed. Because this unexpected task strained both the Phase 1 budget and schedule, the PMO is now tracking documentation for the other systems with which Sentinel must interface as a risk.

Bill of Materials

A Bill of Materials (BOM) is a document that centralizes information from numerous system documents. The BOM should list all parts and components, both hardware and software, that comprise an IT system. According to the FBI’s LCMD, a BOM should provide a complete list of all parts, assemblies, COTS equipment, and software that make up the system as well as the information required to construct new units for the system or order spare parts for it.

Lockheed Martin submitted a BOM, as required, as part of its proposal when competing for the Sentinel project. Contractually, Lockheed Martin is required to purchase the list of items on the BOM. However, as is to be expected in a project as complex as Sentinel, Lockheed Martin has needed to revise the BOM submitted in its proposal. Several allowable reasons for BOM revisions include:

• Specific models of equipment have become obsolete prior to purchase;

• Model numbers of specific equipment have changed;

• The cost of specific hardware or software has changed;

• Items were mistakenly omitted on the original BOM; and

• The design or approach of Sentinel has evolved as the project has progressed.

Because of the importance of having an accurate BOM, the Sentinel PMO established a BOM Deviation Policy. This policy establishes the criteria for what constitutes a change to the BOM and whether Lockheed Martin needs the FBI’s approval prior to making a change to the BOM. The policy defines a deviation as any difference between the original cost, model number or purchase date, or any addition or deletion of material. Lockheed Martin must obtain approval from the Sentinel Contracting Officer’s Technical Representative (COTR) for changes when the cost of materials to be purchased increases from the originally proposed cost by the lesser of 5 percent or $1,000, or there is a change to the purchase date of material by 30 days or more.⁵⁸ However, changes to the BOM resulting from Sentinel Configuration and Change Management Board proceedings do not require approval from the COTR.⁵⁹ Every month, Lockheed Martin is required to submit a BOM Deviation Report that lists all changes made to the BOM in the last month, regardless of whether FBI approval is needed.

According to the Sentinel COTR, Lockheed Martin also established its own procedures for making changes to the BOM, requiring its Engineering Review Board to approve every change request before seeking approval from the FBI. However, according to the COTR Lockheed Martin employees viewed approval by the Engineering Review Board as final and therefore did not submit all changes to the FBI as required. The Sentinel PMO has recognized that Lockheed Martin employees were not fully aware of the PMO’s policies for making changes to the BOM and has begun working with Lockheed Martin to clarify its policies.

In addition to not following the policy for making changes to the BOM, Lockheed Martin also did not submit the required BOM Deviation Reports. According to the Sentinel COTR, Lockheed Martin’s failure to submit the deviation reports was linked to confusion about the approval process for making changes to the BOM, as discussed above. Instead of BOM Deviation Reports, Lockheed Martin provided periodic equipment purchase reports to the PMO. However, these reports could not be reconciled to the BOM and did not document proper FBI approval, so the PMO could not verify whether all items on the purchase report were listed on the BOM or received FBI approval. The same was true of Lockheed Martin’s invoices.

The Sentinel PMO is aware of these shortcomings and has established a joint Lockheed Martin-FBI team to revise the PMO’s policies for making changes to the BOM. FBI officials said the revised policy would include solutions to all of the issues discussed above. In addition, at the close of Phase 1 the FBI required Lockheed Martin to submit a BOM documenting the system as it was built and reconcile that BOM to Lockheed Martin’s invoices.

In addition to the issues with which the FBI was aware, we identified another significant flaw in the PMO’s BOM deviation policy. While the policy states that a deviation is any addition or deletion to the BOM, the policy does not require FBI approval for these additions or deletions. Instead, the policy only requires approval for cost increases and changes to purchase dates for items already on the BOM. The lack of written procedures for making changes to the BOM increases the risk that the document will not provide an accurate reflection of the equipment that has been purchased or that needs to be purchased to ensure successful completion of the project. This information is necessary to control costs, manage property, and prevent unnecessary or wasteful purchases. The Sentinel PMO agreed with our concerns and said that this issue would be addressed in revised BOM policies.

According to Lockheed Martin records obtained from the FBI, changes to the BOM have been significant, resulting in a reduction of nearly $7 million from the cost of Phase 1. As shown in the following table, during Phase 1 Lockheed Martin deferred the purchase of $5.8 million worth of equipment to later phases and purchased $1.5 million worth of equipment originally scheduled to be purchased during future phases. Coupled with changes resulting from modifications in the design of Sentinel and the substitution of newer equipment not available at the time of Lockheed Martin’s proposal, the cost of materials for Phase 1 decreased by nearly $7 million.

Changes to the Bill of Materials

We concluded that the FBI needs better controls to ensure the accuracy of the equipment on the BOM and adequate cost control and property management. In addition, we found that the FBI cannot track changes made to successive versions of the BOM. Without a reliable system to track equipment purchased by the BOM to FBI property management records, the opportunity exists for the loss of system equipment. Management of project costs is also made more difficult, resulting in the increased possibility that project funds could be wasted. Finally, absent a way to document changes as the BOM is updated, it is difficult to determine whether the listed equipment is current and necessary for completion of the project.

The Sentinel Program Manager agreed with our assessment of the BOM deviation policy. He acknowledged that he has found discrepancies between the BOM and the equipment listed in the FBI’s property management system. As noted above, one of the close-out activities for Phase 1 was a line-by-line inventory of all equipment listed on the BOM.

Planning for Phase 2 and Lessons Learned

The FBI’s experience with Phase 1 has had a substantial impact on planning for the remaining phases of Sentinel. At the conclusion of our audit field work in May 2007, the FBI’s Phase 2 activities were limited to planning rather than development. The FBI and Lockheed Martin were examining the best way to divide the remaining Sentinel development tasks. To accomplish this, Lockheed Martin and the FBI were negotiating an engineering change proposal to replan the remaining work, but the FBI had not decided on the number of subsequent phases or the content of those phases. However, FBI officials said the cost estimate and overall timeframe of Sentinel completion by December 2009 had not changed. Below we discuss six lessons the FBI and Lockheed Martin learned during Phase 1 and how the FBI plans to adjust future phases to reduce risk identified by these lessons.

Complexity of COTS Integration

In our December 2006 Sentinel report, we cited the integration of COTS components into a system that meets the FBI’s needs as one of the most significant risks to Sentinel’s cost, schedule, and performance. During Phase 1, integrating COTS software proved to be a significant challenge for Lockheed Martin, one that contributed to the delay in the delivery of Phase 1. This challenge was compounded by Sentinel’s use of relatively new software and the need for personnel with security clearances.

According to the Chief of the Sentinel System Development Unit, while the complexity of integrating COTS software also will affect subsequent phases of Sentinel, it is difficult to determine the degree of the impact. In an effort to mitigate the schedule and cost risks associated with these complexities, the FBI and Lockheed Martin plan to set up a prototyping lab for Phase 2 in which engineers will be able to test early in the phase how different COTS products interact when they are integrated. This will allow engineers to gain early insight into potential problems that may occur during Phase 2 and allow Lockheed Martin enough time to adjust its design accordingly.

Interfacing with ACS is Complex

The web portal developed in Phase 1 displays data stored in ACS. To accomplish this, Lockheed Martin had to build interfaces between Sentinel and ACS, a task during which Lockheed Martin gained familiarity with the complexity of interfacing with ACS. According to Sentinel PMO officials, the documentation or blueprints for ACS were not sufficiently detailed to allow Lockheed Martin to build the interfaces necessary for Phase 1 as it initially intended. Instead, as described previously, Lockheed Martin had to reverse engineer and fix approximately 30 interfaces to build the Phase 1 portal. This unexpected additional work resulted in a significant strain on the project schedule and budget.

To minimize the cost and schedule risk associated with interfacing with ACS and other legacy systems, the FBI plans to reexamine the functionality of the subsequent phases of Sentinel and the need to interface with the FBI’s legacy systems. At a March 2007 meeting, the FBI eliminated from the project plans several interfaces with little-used FBI legacy systems. As discussed previously, the FBI and Lockheed Martin are currently restructuring the remaining phases of Sentinel and a major factor in that restructuring is the desire to minimize temporary interfaces with ACS.

Phase 1 Schedule Focused on Documents

According to the FBI’s Sentinel Program Manager, some of the deliverables in the Phase 1 schedule were documents. However, in some cases, Lockheed Martin delivered incomplete or poorly constructed documents so that it could meet due dates. Because these documents did not serve their intended function, the project was delayed.

According to the program manager, the Phase 2 deliverables will be based more on the delivery of functionality rather than the delivery of documents. For example, Lockheed Martin may not be required to deliver a document detailing the Phase 2 design if the same need can be met using computer software. While some of the Phase 2 deliverables will still be documents, their purpose will be to support a specific function. According to the PMO, documents delivered by Lockheed Martin will not be accepted unless they meet FBI standards.

Weaknesses in Phase 1 Requirements Validation

Requirements validation is the process by which FBI and Lockheed Martin personnel meet during what is called the Requirements Clarification Review (RCR) to discuss the requirements to be completed during the upcoming phase of the project. The RCR is held at the beginning of each phase. During the RCR, requirements that were identified at the beginning of the project are reviewed by both parties to determine whether the functionality provided by completion of the requirement is still necessary. If the requirement is determined to still be applicable to the project, RCR participants then break the requirement down into sub-requirements to be completed in support of the main requirement. The product of the RCR is the Requirements Clarification Document (RCD) which delineates the requirements and sub-requirements to be competed during the current phase.

FBI officials said that not enough time was devoted to requirements validation during Phase 1 and consequently some of the Phase 1 requirements were unclear or incomplete. These shortcomings in the Phase 1 requirements led to disagreements with Lockheed Martin about the design of Phase 1 and created problems during testing. To resolve the issues stemming from the Phase 1 requirements validation, the PMO and Lockheed Martin plan to began Phase 2 requirements validation well in advance of the start of Phase 2 and devote more time to the effort.

Scheduling of Design Reviews

The Phase 1 schedule did not allow enough time between the Preliminary Design Review, the Critical Design Review, and the Final Design Review. Key design documents must be developed by Lockheed Martin and reviewed by the FBI at each of these interrelated design reviews. In addition, the schedule needs to allow adequate time for the following:

• The FBI must review and comment on the documents;

• Lockheed Martin must incorporate the FBI’s comments and resubmit the documents to the FBI; and

• The FBI must verify that Lockheed Martin adequately addressed the FBI’s comments.

As discussed in the previous finding, the lack of time between design reviews led to incomplete design documents that did not incorporate all of the FBI’s technical comments. In turn, the incomplete design documents led to problems during the testing phase of the project. In the subsequent phases of Sentinel, the FBI has committed itself to allowing sufficient time between design reviews.

Construction and Maintenance of Schedule

According to the Sentinel Program Manager, the Phase 1 schedule was extremely tight from its inception. In addition, some of the tasks were scheduled out of order. Another issue that the PMO and Lockheed Martin encountered was starting and ending dates for tasks were entered into the schedule as hard constraints. This means that even if the project progressed at a faster rate than planned, tasks could not be recorded as started until the date specified. This, in effect, prevented the project from ever achieving an “ahead of schedule” status.

These hard constraints were also applied to task end dates. As a result, whether or not the tasks had actually been completed and signed off on by the PMO, on the end date identified in the schedule the tasks were automatically listed as complete. The Sentinel Program Manager said that Lockheed Martin personnel mismanaged the schedule in this way because of intense pressure from FBI and Lockheed Martin managers to stay on schedule. This misidentification of task status contributed to further delays and provided poor visibility into the project’s actual status.

For future phases, the PMO plans to work with Lockheed Martin to ensure a more accurate assessment of progress. The PMO and Lockheed Martin also plan to reduce the number of hard constraints from the schedule to improve the accuracy of progress measurement. The removal of constraints will also allow for the adjustment of task end dates in the event the project falls behind schedule. During Phase 1, when a task that affected the completion of a related task fell behind schedule, the end date of the related task remained static. This resulted in schedule compression and creation of an unrealistic schedule. The removal of hard constraints will allow for greater visibility into how delays affect the completion of other tasks and ultimately the entire phase.

Actions Taken on Previous OIG Recommendations

During our audit, we examined the FBI’s actions to address recommendations we made in our earlier audit reports on Sentinel and found that the FBI was, in general, taking action to resolve our concerns. Based on the FBI’s actions, we closed 5 of the 12 recommendations. We also noted that the FBI agreed with the remaining recommendations and was in the process of taking corrective action. Our recommendations dealt generally with the FBI’s need to complete required planning and general management oversight policies and procedures for Sentinel in order to help ensure its success.

In our March 2006 report, The Federal Bureau of Investigation’s Pre-Acquisition Planning For and Controls Over the Sentinel Case Management System, we made seven recommendations, of which four have been closed. The remaining three recommendations relate to the need to complete certain planning documents, the staffing of the Sentinel PMO, and Sentinel training. In addressing these recommendations, we found that the FBI:

• continued to work on completing its system security plan and completed its independent verification and validation (IV&V) plan, which partially closes this recommendation;

• filled 70 of 75 positions within the Sentinel PMO, with three of the vacancies tentatively filled; and

• continued work on a comprehensive training plan with schedule and cost estimates.

In our December 2006 report, Sentinel Audit II: Status of the Federal Bureau of Investigation’s Case Management System, we made five recommendations, one of which has been closed. The four remaining recommendations were that the FBI should: (1) develop contingency plans as required by the Sentinel Risk Management Plan, (2) provide experienced contractors to conduct an IV&V process throughout the Sentinel project, (3) determine the appropriate amount of management reserve for each phase of the project, and (4) fill the vacant Sentinel PMO positions needed to complete Phase 1 of the project.

In performing the fieldwork for our current audit, and as mentioned earlier in this report, we found that the FBI has improved its development of contingency plans and triggers as required by the Sentinel Risk Management Plan. With the implementation of Phase 1, there is no longer a need to address the risks related to that phase. However, we continue to have concerns regarding the FBI’s implementation of its Risk Management Plan, as well as some of the risks we noted earlier concerning the future Phases of Sentinel, and we will continue to monitor their implementation during future audit work.

Also noted earlier in this report, we found that the FBI has begun utilizing a contractor, Booz Allen, to perform IV&V of the Sentinel project. We found that the IV&V process has resulted in numerous recommendations to Sentinel Phase 1 development. Based on our examination of the FBI’s implementation of this process, this recommendation can be closed through our normal audit follow-up process.

Because each of Sentinel’s four phases will have a separate task order with its own funding, and planning for Phase 2 continued after our fieldwork ended, we were unable to determine whether the FBI is establishing a management reserve based on an assessment of the project risks for each phase and for the project overall. However, we will continue to monitor the FBI’s utilization of management reserves in future audit work. After the completion of Phase 2 planning, the methodology currently being used by the FBI to develop management reserves should become more apparent.

Due to the importance of the PMO in the oversight of Sentinel, we recommended in both of our previous Sentinel audits that the PMO complete hiring as soon as possible for the vacant PMO positions. The PMO plays a critical role in assuring that the FBI implements a case management system that meets its needs. The PMO’s contract and program execution responsibilities include: (1) cost, schedule, and performance oversight; (2) LCMD project reviews; (3) award fee evaluations; (4) primary contractor’s documentation review and acceptance; (5) requirements and risk management; and (6) budget and financial management. In light of these responsibilities, having a qualified, dedicated PMO staff focused on program execution is critical to the success of the Sentinel project.

Since our December 2006 audit the PMO has increased its planned size from 73 positions to 76 positions, reallocated positions among PMO units, and was in the process of filling 3 positions. As of May 2007, the PMO consisted of 70 of the 76 personnel identified in the FBI’s Sentinel Staffing Plan (92 percent) as required to properly oversee the project. According to the FBI, the objective in staffing the PMO is to form an integrated team of subject matter experts from government, federally funded research and development centers, and system engineers and technical assistance contractors to maximize program expertise.⁶⁰ The following table summarizes the PMO’s staffing level as of May 15, 2007, and shows the progress the FBI has made in staffing the office since October 2006.

SENTINEL PMO STAFFING REQUIREMENTS

For a more complete description of PMO staff and their duties, see Appendix 8.

Of the current vacancies, one is a government position — a Supervisory Special Agent — and one is a contractor position — an Organizational Change Management (OCM) expert. Hiring for the other position — a System Engineer — has been delayed until Phase 3. The Chief of the Business Management Unit said that a setback in filling PMO positions quickly occurred when the FBI instituted a hiring freeze as a result of the FY 2007 continuing resolution. He said that another setback to filling open PMO positions quickly is the amount of time that is required to execute and adjudicate Top Secret clearance background checks; he said that the PMO has been submitting its background check requests in expedited status.

The following table shows the changes in the number of planned staff from October 2006 to May 2007.

Changes in Sentinel PMO Staffing Requirements,
October 2006 to May 2007

The FBI Deputy CIO said he thinks the PMO is “pretty lean” in comparison to other PMOs he has seen. He said that he and the Sentinel Program Manager looked at best practices while developing the structure of the PMO, but were also mindful that while they could select whomever they wanted internally in the FBI to work at the PMO, this would also put another component of the FBI at a disadvantage by losing top personnel. For expertise in areas in which the government was weak, contractors were chosen. The FBI Deputy CIO said that he is happy with the PMO’s return on investment thus far in the project.

The FBI Deputy CIO also said that as the Sentinel project progresses, the focus areas and personnel needs of the PMO will shift. For example, Operations and Maintenance (O&M) will become more of a focus, especially by the end of Phase 2. Since this will result in the need for a different skill set than is currently available at the PMO, there will be changes in personnel. Fewer designers will be needed, and more people to maintain the system will be required. The FBI Deputy CIO said the PMO will probably start trading personnel out of the PMO as the need arises.

While we concluded that the FBI is making progress in implementing the recommendations we have made during our prior audits, we will continue to monitor the progress made by the FBI in implementing the remaining open recommendations through our audit follow-up process and through our future Sentinel work.

Conclusion

The FBI has a broad range of management controls and processes that should aid it in managing the development of Sentinel. Of the four controls and processes we reviewed in depth, we found that the FBI has made significant progress in implementing each. However, the FBI’s experience with Phase 1 demonstrates the high priority the FBI must assign to the entire range of management controls and processes over the project. During Phase 1, issues the FBI experienced with three of the four of controls – earned value management, risk management, and the bill of materials – show that additional improvements need to made to allow the FBI to adequately monitor cost, schedule, and performance of future phases of Sentinel.

Lockheed Martin and the FBI will likely face many of the issues encountered in Phase 1 during subsequent phases of Sentinel’s development. For example, even though the FBI plans early prototyping to reduce the likelihood of problems when integrating COTS components, the next phase of Sentinel will likely encounter unexpected problems integrating the various COTS components into legacy FBI systems in a way that meets the FBI’s needs. However, rigorous implementation of processes and lessons learned is necessary to minimize significant deviations from cost, schedule, technical, or performance baselines.

Recommendations

We recommend that the FBI:

3. Collect and report EVM data for both the performance measurement baseline approved at the integrated baseline review as well as the revised performance measurement baseline.

4. Reconcile the discrepancy between the costs Lockheed Martin reported for Phase 1 with Lockheed Martin’s EVM data, and develop and implement policies and procedures to prevent any future discrepancies.

5. Develop and implement effectiveness measures for all risk mitigation plans.

6. Ensure that personnel assigned to manage Sentinel risks devote sufficient time to the risk and have the experience and authority to adequately manage the risk.

7. Document and track project issues, risks that have occurred, as well as the plan to resolve those issues and their ultimate resolution.

8. Implement policies and procedures to ensure that any changes to the bill of materials receive proper authorization and that the changes can be reconciled to the bill of materials submitted in Lockheed Martin’s proposal.

9. Implement policies and procedures to ensure that materials contained in Lockheed Martin invoices can be reconciled to the bill of materials or an FBI approval for a change to the bill of materials.

31. An enterprise service bus is software “middleware” that connects software components and allows the components to communicate with each other.

32. Lockheed Martin did not receive the award fee. Instead, the FBI allowed it to transfer the $2 million budgeted for the award fee to the cost portion of the budget.

33. See Appendix 3 for an explanation of the management reviews of the project discussed throughout this report.

34. In general, the strategies for integrating COTS components include adjusting the standard configuration settings of the system’s components, modifying the system’s components, replacing problematic components, and adding additional components.

35. In September 2006, the FBI obtained the services of Booz Allen Hamilton to perform the IV&V function for the Sentinel project. See Finding 2 for a more detailed discussion of IV&V.

36. A deficiency report documents a problem identified in testing and the resolution of the problem.

37. The remaining 28 percent of Sentinel’s budget funds the PMO.

38. Web-Enabled Automated Case Support (WACS) is ACS updated with access through a web portal. Phoenix is the second generation of WACS intended to access ACS through mini-programs specifically written to run within Web browsers.

39. In general, testing ensures the system satisfies FBI requirements for utility and performance. Site Acceptance Testing executes a complete set of test procedures on the software that has actually been delivered from the contractor to the FBI and installed in order to insure consistency and functionality in the operation of the software. The purpose of User Acceptance Testing is to gain end users’ acceptance of Sentinel.

40. FBI field offices are divided into squads that have specific subject matter responsibilities such as drug trafficking or counterterrorism. A Supervisory Special Agent manages each squad and supervises the personnel assigned to the squad.

41. See page 34 for a discussion of the common components of a service-oriented architecture.

42. See the Introduction to this report for a more detailed discussion of the goals of an IBR.

43. Service-oriented architecture governance is the policies and controls used to manage any changes to a service-oriented architecture. PKI is a system of computers, software, and data that relies on cryptography to provide some aspects of computer security.

44. A derived requirement is a requirement deduced or inferred from the requirements in the statement of work.

45. Service-oriented architecture governance is the policies and controls used to manage any changes to a service-oriented architecture. PKI is a system of computers, software, and data that relies on cryptography to provide some aspects of computer security.

46. Capability Maturity Model ® Integration is a process improvement approach that provides the essential elements of effective processes. There are six capability levels, numbered 0 through 5. Each capability level corresponds to a goal and a set of practices.

47. Xxxx Xxxxxxxxxx Xxxxxxx xxxxx xx x xxxxxxxxxxx xxxx xxxxx xxxxxxx xxxxxxxxx xx xxxx xxxx xxxxxxxx xx xxxx xxx xxxxxxx xxxx xxx xxx xxxxxx xxxx xxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxxxxxx xxxxxxxxxxxx xxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxxx xxxx xxxxxxxxxxxxxxxxxxxxxx.

48. While the FBI purchased the xxxxxxxx enterprise license through the Sentinel contract, Sentinel funds were not used for the purchase.

49. ANSI/EIA Standard 748-A is the criteria selected by the OMB for EVM systems. The standard includes 32 specific criteria in five process areas necessary for a sufficient EVM system: (1) organization; (2) planning, scheduling and budgeting; (3) accounting; (4) analysis and management reports; and (5) revisions and data maintenance.

50. Replanning revises the time-phased budget for completing the work remaining in a project without any changes to the total scope of work, baselined cost, or scheduled completion of the project.

51. The System Design Document contains the design decisions made for the system as well as the system’s architectural design and the services it will utilize. The Interface Design Document details the design of and responsibility for both internal data exchanges (within Sentinel) and external data exchanges (with systems outside of Sentinel).

52. At a minimum, an integrated master schedule shows the expected start and stop dates for each event and accomplishment in a project’s integrated master plan. To aid in the day-to-day management of a program, each event or accomplishment may be broken down into multiple level hierarchy of tasks necessary to achieve the larger task. A work breakdown structure is a tool for defining the hierarchical breakdown of tasks and activities in an integrated master schedule.

53. According to the Sentinel Risk Management Plan, the Sentinel Risk Review Board will include representatives from the following: program manager, systems engineer, program manager support personnel, systems engineer support personnel, program sponsor, users, the prime contractor, sub-contractors (as determined by the prime contractor), the Project Assurance Unit, and the Sentinel risk coordinator.

54. In addition to the risk management processes cited above, the following receive briefings that include information about Sentinel risks: the FBI Director (weekly); a review team with senior representatives from the Department of Justice, OMB, and Director of National Intelligence (monthly); the FBI CIO’s Advisory Council (bi-monthly); the FBI Director’s Advisory Board (as requested); and congressional oversight committees (quarterly).

55. The April 30, 2007, risk register is the latest register that the FBI provided us.

56. Two risks were assigned to both Phases 1 and 2.

57. Of the 16 open risks in the April 30, 2007, risk register, 6 had both a probability of occurrence and severity of impact assessed below “medium” and are therefore not required to have a mitigation strategy. However, the FBI developed mitigation strategies for them anyway. One of the 10 remaining risks did not have probability or severity ratings, so we could not determine whether it required a contingency plan.

58. The Sentinel COTR is assigned to the Business Management Unit and assists the contracting officer with the technical oversight necessary to execute the Sentinel contract. Specifically, the COTR is the official recipient of all contract deliverables, coordinates the delivery of government furnished equipment and information to the prime contractor, and verifies invoices.

59. The Sentinel Configuration and Change Management Board is responsible for approving updates to any of Sentinel’s baselines. The Board adjudicates proposed changes affecting system requirements, configuration management, risk management, and documents that affect the project’s scope, schedule and cost.

60. Federally funded research and development centers are nonprofit organizations sponsored and funded by the U.S. government to assist government agencies with scientific research and analysis, systems development, and systems acquisition.