Sentinel Audit II:  Status of the Federal Bureau of Investigation’s Case Management System (Redacted)

Audit Report 07-03
December 2006
Office of the Inspector General


Findings and Recommendations

Foundation of the Sentinel Project

Sentinel Contract

The FBI is using a GWAC contracting vehicle, administered by the National Institutes of Health (NIH), to develop Sentinel. Such a contracting vehicle streamlines the acquisition process by allowing multiple government agencies to purchase services under one contract. Instead of awarding a specific contract to a vendor, the awarding agency issues a task order to the selected vendor. In the case of Sentinel, the FBI administers the task order itself. In March 2006, the FBI announced that the four-phase Sentinel project would cost an estimated $425 million, with $305 million awarded to Lockheed Martin to develop the system by December 2009 and $120 million for the FBI’s program management costs and other contractor support.

The FBI subsequently awarded Lockheed Martin a $57 million task order for Phase 1 of Sentinel, with options for $248 million more for the three additional phases and the operations and maintenance (O&M) of the system developed during the project. In addition to the cost baseline, the project has an overall schedule for which specific baselines are being established phase-by-phase. Over about 4 years, Lockheed Martin will be responsible for designing, developing, integrating, testing, deploying, operating, and maintaining Sentinel – which will be primarily based on commercial-off-the-shelf software – and will provide all the personnel, facilities, equipment, material, and support necessary to implement Sentinel.

Lockheed Martin is performing the work under a cost-plus-award-fee arrangement, similar to the one used during the Trilogy project.22 However, the FBI is providing much greater control and oversight for Sentinel compared to the weak management evident in the Trilogy project. The contract is structured to reward excellent performance by Lockheed Martin. If Lockheed Martin meets the schedule and cost targets set by the FBI, the FBI can grant Lockheed Martin award fees of up to [redacted] percent of the [redacted] Sentinel development costs, or up to nearly [redacted]. Lockheed Martin’s performance will also determine whether the FBI exercises options to award additional phases of the project to Lockheed Martin. If the FBI finds Lockheed Martin’s performance unacceptable at any stage of the project, the FBI can order Lockheed Martin to stop work on the project. If the contractor does not meet its milestones, it will be penalized by loss of the award fee.

Estimating Sentinel’s Cost

The FBI based its $425 million estimate for the total cost of the Sentinel project on: (1) an independent government cost estimate conducted on the FBI’s behalf by Mitretek Systems prior to soliciting bids for Sentinel in April 2005, and (2) the FBI’s assessment of the cost estimate contained in Lockheed Martin’s proposal.23 We reviewed the processes used to derive the $425 million estimate, noted inconsistencies in the process and the results, and concluded that the estimate is a rough approximation of Sentinel’s overall costs. The estimate is, in our view, tentative given the variances in the supporting cost estimates and the inherent complexity of estimating costs for a major IT system even before the design is finalized. However, the FBI’s CIO said he stands by the estimate. Further, we identified several Sentinel-related projects, the costs of which are not included in the overall Sentinel estimate.

Independent Government Cost Estimate

The independent government cost estimate concluded that project costs would range between $329 million and $493 million, with the most likely cost $438 million. According to the Chief of Sentinel’s Business Management Unit, this estimate is the basis for the $120 million program management portion of the FBI’s total estimate of $425 million.

The independent government cost estimate established a series of classifications to describe the work to be accomplished and the products to be acquired in the development of Sentinel. Six different techniques were used to estimate the cost of the various elements of Sentinel: parametric modeling, cost estimating relationships, analogy, engineering assessment, vendor quote, and historical data. Appendix 4 provides detailed definitions of each of these cost estimating methods. The cost-estimating method chosen for each work element depended on the availability of technical and cost data.

We reviewed the estimate and identified several concerns about its ability to provide the FBI with a reliable estimate of Sentinel’s costs. The estimate was performed concurrently with development of Sentinel’s requirements. While Mitretek Systems, the FBI’s estimating contractor, coordinated its efforts with personnel developing Sentinel’s requirements, the estimate might not accurately reflect the project’s final design specifications, which are not expected to be completed until about October 2006. Also, the estimate contains several inconsistencies. For example, some parts of the cost estimate show Sentinel’s O&M phase lasting 3 years while other parts show it lasting 2 years, resulting in a likely O&M cost range of $62 million to $87 million. If the additional inconsistencies are factored into the summary cost of the O&M phase, the O&M estimate could be as low as $53 million. Finally, the overall cost estimate does not include all of the costs in the Sentinel funding plan. For example, the estimate does not include the management, or risk, reserve or a separate Independent Verification and Validation (IV&V) contract to independently assess Lockheed Martin’s testing of Sentinel’s software, which currently account for a total of $40 million of the PMO’s $120 million estimate.24

Government’s Estimated Most Probable Cost

The FBI received proposals for the Sentinel project from [redacted] bidders. When the [redacted] proposals were received, the FBI reviewed them to determine whether the cost data within the proposals was complete, based on clear and accepted methodologies, and accurate. [redacted]25 Cost realism analysis results in the Government Estimate of Most Probable Cost (GEMPC) for the project. [redacted].26 Based on the GEMPC, FBI officials concluded that Lockheed Martin’s estimate was reasonable [redacted]

[redacted]

[redacted]

[redacted]

[redacted]

[redacted]

[redacted]

[redacted]27 Despite these differences, the FBI determined that Lockheed’s proposal was reasonable and did not pose a significant risk. According to FBI officials, the FBI resolved the issues identified in the GEMPC during its negotiations with Lockheed Martin.

[redacted]

[redacted] Due to the variability and inconsistencies of the estimates we reviewed, and the difficulty of forecasting the eventual cost of a major IT project, we could not confirm the accuracy of estimates, nor could we validate the FBI’s overall estimate of $425 million for Sentinel.

Sentinel-Related Costs

We also identified several projects and other costs, which in total exceed $25 million, that are related to Sentinel but are not considered by the FBI as direct Sentinel costs and are therefore not included in the FBI’s total estimate of $425 million. Examples of these related costs include the National Name Check system, security costs, and FBI salaries. However, as discussed previously, because of the difficulties associated with accurately estimating the total cost of such a large project, we cannot state with certainty whether Sentinel’s costs would exceed $425 million, only that the costs would be higher if the costs of the Sentinel-related projects were included.

The implementation of Sentinel will require changes to the FBI’s National Name Check system. In response to a request from a federal, state, or local agency, the National Name Check Program queries FBI records to determine whether the person named in the request has been the subject of an FBI investigation or mentioned in an FBI investigation. The data system used by the Name Check program relies very heavily on the ACS system, which Sentinel is intended to replace. The estimated cost of updating the existing name check system to work with Sentinel is over [redacted].

The FBI is also developing security through its Information Access Technology Initiative (IATI) to support Sentinel and future FBI systems. A portion of the IATI is intended to help the FBI move from a manual security classification review of documents to a more automated review. The IATI will be developed in concert with Sentinel and should be able to integrate with Sentinel and the FBI’s overall IT network as well. The purchase of any initial license for a security product used in conjunction with the IATI would be funded by the Office of the CIO (OCIO). This license will be used for testing and evaluation. If approved, Sentinel would later purchase a license for its own use of the product. While the software is critical to the security of Sentinel, the cost of the initial license is not reflected in the FBI’s Sentinel costs. The FBI is uncertain as to which of the products in development would be used by Sentinel and therefore was unable to estimate the specific costs related to Sentinel.

The salary costs of FBI employees are also not tracked as a Sentinel expense. These costs include FBI employees assigned to the Sentinel PMO, the employees who will be developed to train other employees on Sentinel use, ITOD staff assigned to Sentinel, employees who will attend Sentinel training, and the Finance Division auditors who review Sentinel invoices. While the Independent Government Cost Estimate of $438 million does not include the cost of FBI employees in the overall cost of Sentinel, other portions of the report concluded that the cost of FBI employees’ involvement in the development and implementation of Sentinel would be approximately $15.8 million.

The FBI’s position is that the separate projects discussed above are independent, enterprise-wide projects that will benefit the FBI’s overall IT structure, including Sentinel but also many other FBI projects. The CIO and the Sentinel PMO contend that the costs of such independent projects ought not be considered as Sentinel costs. While we agree that these Sentinel-related projects may not be direct Sentinel costs, in our view the scope of the Sentinel project would be larger if it was not supported by these other investments. When decision makers are considering the full cost of the Sentinel project, they should keep in mind both the direct project costs as well as the additional related costs.

Spending Plan and Management Reserve

In the FBI’s spending plan for Sentinel, developed shortly after it awarded the contract, the $425 million total project cost estimate covers the four phases of Sentinel plus 2 years of O&M after the completion of the system. Based on Lockheed Martin’s proposal, the FBI plans to pay Lockheed Martin $305 million for the development of Sentinel and its O&M expenses. The spending plan shows that the FBI will use the remaining $120 million for program management, the IV&V of the software, and management reserve. The FBI estimates that Phase [redacted], with a cost of [redacted] over [redacted] years, will be the most expensive phase as well as the most challenging. The chart below summarizes the FBI cost estimates by type of expense and project phase.

Sentinel Spending Plan by Phase

CHART REDACTED

 Source: The FBI

According to a May 2006 Lockheed Martin plan, material and equipment will be the largest cost of Lockheed Martin’s contract to develop and deploy Sentinel. As shown in the following chart, labor to develop the system and O&M of the system are the other two major cost categories; together, they represent over 50 percent of the value of Lockheed Martin’s contract.

Lockheed Martin Spending Plan
By Cost Category

CHART REDACTED

 Source: Lockheed Martin Services, Incorporated

The Sentinel PMO is responsible for ensuring that the Sentinel project is properly executed, including: (1) oversight of the program’s cost, schedule and performance, (2) Life Cycle Management Directive (LCMD) reviews; (3) award fee evaluations; (4) review and acceptance of Lockheed Martin’s documents; (5) requirements and risk management; and (6) budget and financial management.28 As shown in the following chart, the FBI estimates that the majority of the PMO’s expenses will be for the operation of the PMO itself. The primary expense of the PMO is contractors, which accounts for about 74 percent of the PMO’s 73 planned positions. The PMO’s budget is based on the requirement that all positions be filled throughout the four phases of development. However, the Chief of the Business Management Unit told us that there is no reason to fill six positions until the project approaches Phase 2, which begins in early 2007 (PMO staffing is discussed later in this report). Twenty-eight percent of the PMO’s $120 million budget is for a management, or risk, reserve. (As discussed in the EVM section of this report, Lockheed Martin also has a management reserve for Phase 1.) The management reserve is an OMB-required contingency fund used to cover the costs not known at the time a project’s cost estimate is developed. Depending on the confidence level the agency has in a project’s cost estimate, the OMB calls for management reserve of 10 to 30 percent.

Project Management Office Spending Plan
By Cost Category

Program Management: 67%, Risk Management: 28%, IV and V: 5%.
Source: OIG Analysis of FBI data

According to the Sentinel Program Manager, Sentinel’s management reserve should be 11 percent of the estimated $232.4 million development cost of the project, or about $25.6 million. The PMO determined the percentage of the management reserve based on a review of the known risks and the System Requirements Specification.29 We found that the total Sentinel management reserve of $34.1 million is about 15 percent of the development cost of the project. As shown in the following chart, the FBI’s management reserve varies by program phase from 11 percent of the development cost for Phase 2 to 32 percent for Phase 4.

Management Reserve as a Percentage of
Development Cost by Project Phase

CHART REDACTED

 Source: OIG Analysis of FBI data

The Sentinel Program Manager said he did not advocate a management reserve greater than 11 percent of development, and he expects the Phase 1 management reserve to be reduced from 15 percent to 11 percent.30 The FBI’s Deputy Assistant Director of Finance agreed that an 11 percent management reserve was sufficient for Phase 1. However, he said the Finance Division had not transferred the excess management reserve to another account because there was no current operational need for the money. Both the Chief of the Sentinel PMO’s Business Management Unit and the Deputy Assistant Director of Finance said that the amount of the management reserve for each phase was determined based on preliminary estimates of Sentinel’s cost and had not been adjusted to reflect the FBI’s contract with Lockheed Martin. The FBI’s current spending plan for Sentinel overstates the total anticipated cost of the project by $8.6 million, the difference between a management reserve of 15 percent of development costs and the 11 percent. However, FBI officials told us that over the course of the project, the management reserve will be adjusted to 11 percent of Sentinel’s development cost.

Funding Sentinel

Our March 2006 report stated that according to an FBI official the OMB required the FBI to identify the funding for each phase of Sentinel before work on that phase could begin. As a result, on September 27, 2005, the FBI submitted a $97 million reprogramming request to Congress for the first phase of Sentinel. Congress approved the request on November 15, 2005. According to the PMO’s most recent cost estimates, Phase 1 will cost $108.5 million and require funds over four fiscal years (FY) starting in FY 2006. However, Phase 1 will only require $93.4 million in FY 2006 and 2007 funds, potentially making $3.6 million of the $97 million in reprogrammed funds available to help fund Phase 2.

The President’s FY 2007 budget request includes $100 million for Phase 2 of the Sentinel project. However, whether the FBI will receive the full requested amount is uncertain because the FY 2007 appropriation has not been finalized by Congress. [redacted]. If the FBI receives the full $100 million requested in the FY 2007 budget, the FBI would need to identify an additional [redacted] to meet Sentinel’s FY 2007 funding requirements. [redacted] However, the FBI’s CIO recently told us that an FY 2007 appropriation of less than $100 million would be cause for concern and could result in an unanticipated level of reprogramming of FBI resources to fund the Sentinel project.

The FBI plans to seek additional appropriations to fund the third and fourth phases of Sentinel. [redacted] The table below shows the spending plan for Sentinel by fiscal year over the life of the project.

Sentinel Spending Plan (Millions of Dollars)

CHART REDACTED

 Source: FBI

In our first report on Sentinel, we noted that more than $14 million of the FBI’s $97 million November 2005 reprogramming would come from the Counterterrorism Division budget, $13 million from intelligence-related activities, and $2 million from the Cyber Division. During our first audit, most FBI divisions and offices seemed confident about their ability to absorb the initial reprogramming of funds to Sentinel for Phase 1. However, the officials stated that a second reprogramming of the same magnitude would damage their ability to fulfill their mission.

During this audit, we also interviewed officials at FBI headquarters to assess the impact of the $97 million reprogramming and any future reprogrammings for Sentinel. Generally, these officials confirmed that their divisions and offices can withstand the diversion of funds to Sentinel for the first reprogramming and that the successful implementation of a modern case management system would offset the operational impact of the reprogramming. These officials also said they had not received notice of the need for or amount of any future reprogrammings and therefore could not assess its potential impact. In our judgment, any reprogramming significantly above $50 million will require the FBI to carefully consider which programs and activities will be affected and how to monitor the overall impact on the FBI’s mission.

Cost Tracking and Control

For the Trilogy project, the FBI lacked an effective, reliable system to track and validate the contractors’ costs. We highlighted this concern in our February 2005 report on Trilogy and stated our continuing concern in our March 2006 report on Sentinel. Also, in its February 2006 report the GAO stated that the FBI’s poor cost controls resulted in the payment of about $10 million in questionable contractor costs, and poor property management led to missing equipment valued at $7.6 million.

The FBI has now established several layers of control to help ensure that costs are authorized in advance, verified when delivered, and validated when invoiced. The overlapping responsibilities for oversight of Sentinel’s costs include: the FBI’s Finance Division – which performs accounting, auditing, and budget monitoring; the Office of the Chief Information Officer’s (OCIO) IT Financial Management Unit – which tracks Sentinel’s costs in detail; and the Sentinel PMO’s Program Integration Unit – which tracks program and development costs and has developed policies and procedures for processing invoices, requisitioning and procuring equipment, reviewing contractor time charges, and resolving discrepancies. The Sentinel PMO’s Business Management Unit has also implemented a “change management process” to help prevent “requirements creep” that can increase project costs or schedule delays. The tracking systems and controls the FBI has implemented provide greater assurance that the FBI will be better able to monitor and control project costs for Sentinel than was the case under Trilogy.

Oversight and Control

The Finance Division’s Audit Unit has dedicated two of its six auditors to work part time on Sentinel. According to Finance Division staff, auditors periodically review a sample of invoices for Sentinel goods and services to verify that applicable procedures are being followed. The Audit Unit produces a monthly audit report, which is distributed to the Contracting Officer’s Technical Representative (COTR), the Finance Division, and FBI management, including the Deputy Director.

The Finance Division tracks Sentinel spending through the FBI’s Financial Management System (FMS). The FMS uses four categories – development contract, O&M, program management, and risk management – to track Sentinel costs. In addition, the Chief Financial Officer (CFO) has established a separate, dedicated cost code for Sentinel that allows the Sentinel PMO, OCIO, and CFO teams to jointly track and control Sentinel costs through the Budget Execution and Analysis Reporting System (BEARS), a database used to track budget information within the OCIO. BEARS tracks Sentinel equipment purchases and other expenditures by project phase based on 20 specific spending plans. BEARS extracts purchase order information from the FMS and generates reports on funds requested, amounts approved, and obligations that have not yet entered the FMS. BEARS data is used for the FBI’s EVM analyses, discussed below.

Requisitions require the approval of the Sentinel PMO, Business Management Unit Chief, the COTR, and the Office of IT Program Management’s Program Management Executive. The PMO budget analyst and the IT Financial Management Unit verify availability of funds according to the spending plans. The Office of IT Policy and Planning validates and approves the requisition requirements, and the IT Financial Management Unit enters the requisition information into BEARS.

The IT Financial Management Unit only tracks funds that have been entered into the Sentinel spending plans in BEARS. It loses visibility over Sentinel funds any time funds are transferred from Sentinel to another FBI program. For example, the Sentinel PMO had to pay for its portion of the FBI’s wireless service that supports its handheld e-mail devices. The IT Financial Management Unit transferred funds from the Sentinel account to the appropriate account. Once this transfer occurred, the Unit no longer had the capability through BEARS to determine whether the money was actually spent for the use intended. The IT Financial Management Unit has not devised a practical alternative method to track Sentinel costs not entered into the BEARS database managed by the unit.

Invoice Processing Overview

We reviewed Sentinel’s requisitioning and invoice processing procedures and found that they appeared reasonable. The contractor submits invoices to the COTR for review. The COTR verifies the invoices with the Sentinel Unit Chiefs, such as the chief of the System Development Unit, to ensure that the billed work has been performed, is within the scope of work, and is funded. The COTR returns any incorrect invoices to the vendor with comments detailing the discrepancies or the additional information required.

The Chief of Sentinel’s Business Management Unit records and tracks invoices against purchase orders; analyzes actual expenditures against planned spending by month; prepares regular reports for the COTR, Unit Chiefs, and the Program Manager regarding the availability of funds; notifies the COTR and Program Manager of any deviation greater than 5 percent from planned expenditures; revises spending plans at least quarterly; and coordinates invoices with EVM estimates.

The Program Manager or Deputy Program Manager reviews final invoices after the reviews by the COTR and unit chiefs, and is responsible for approving invoices for payment. The Contracting Officer then gives final approval and forwards the invoice to the FBI’s Commercial Payments Unit for payment.

Based on our review, the Sentinel’s policies and procedures for processing invoices, requisitioning and procuring equipment, reviewing contractor time card, and handling deviations in bills of materials should help prevent the FBI from incurring and paying for unauthorized services and materials.

Earned Value Management

Our March 2006 report on Sentinel pointed out the need for the FBI to establish an EVM process for Sentinel, which it has since done. EVM helps manage project risks by achieving reliable cost estimates, evaluating progress, and allowing the analysis of project cost and schedule performance trends. EVM compares the current status of a project, in terms of both cost and schedule, to the established cost and schedule baselines. Deviations between the baselines and the current status demonstrate the project’s progress and the overall level of performance, thereby enabling a level of accountability to be imposed on the project. When properly utilized, EVM allows project management to pinpoint potential problems and address them before they escalate. Based on our review of early EVM reporting from April to August 2006, we identified no immediate concerns with Sentinel’s cost or schedule in the first phase of the project, although Lockheed Martin was still grappling with some estimating errors that may have a future impact on the EVM results.

According to the FBI’s EVM plan, the Sentinel PMO will use the plan to measure its and the contractor’s earned value performance and report the result to oversight entities. The Sentinel project’s Statement of Work requires vendors and contractors to fully implement EVM in accordance with the plan, including having an EVM system of its own that complies with American National Standards Institute (ANSI) /Electronic Industries Association (EIA) Standard 748-A.31 This allows the FBI to gather EVM data on the development portion of the project from Lockheed Martin through monthly electronic data transfers from Lockheed Martin. The Sentinel PMO collects EVM data for the PMO portion of the Sentinel from invoices from support services contractors and BEARS, an FBI reporting system discussed previously.

The Statement of Work also included the requirement that the vendor perform an Integrated Baseline Review (IBR), where the cost and schedule baselines would be established for the project. Properly executed, IBRs are an essential element of a Program Manager's risk-management approach. IBRs are intended to provide both the government’s and the contractor’s program managers with a mutual understanding of the project’s performance measurement baseline and agreement on a plan of action to resolve any identified risks.

The Sentinel IBR started on schedule, but took somewhat longer than scheduled to complete. According to the report documenting the results of the IBR, the FBI and Lockheed Martin achieved the objectives of the IBR, and the Project Management Baseline was set for Phase 1. The IBR set the baseline budget at [redacted], not including the [redacted] (about [redacted] percent of the baseline budget) management reserve established for Lockheed Martin at the IBR. Including the management reserve, the baseline budget is $2.9 million less than the $57.2 million contracted for Phase 1. Lockheed Martin’s management reserve, which was established with the FBI’s agreement, is intended to provide Lockheed Martin with the flexibility to respond to any cost estimating errors it may have made and still stay within the contracted amount. The Sentinel Statement of Work also required that Lockheed Martin submit its EVM system to the contracting officer for review. In June 2006, the PMO’s EVM analyst reviewed Lockheed Martin’s EVM system and determined that the system complies with ANSI/EIA Standard 748, and the FBI’s contracting officer concurred.

At the time of our audit, the FBI had begun using “Winsight” software to maintain and report Sentinel’s EVM performance metrics. Sentinel’s EVM analyst prepares three EVM reports each month: one analyzing the whole program’s EVM data, one analyzing Lockheed Martin’s EVM data, and one analyzing the PMO’s EVM data.

We reviewed the EVM reports for April to August 2006. The August 2006 EVM reports show that since the schedule and costs of Lockheed Martin’s work were determined, the actual cost of work performed by Lockheed Martin exceeded the planned cost. During June, July, and August, the Lockheed Martin portion of the program was [redacted] percent, [redacted] percent, and [redacted] percent over budget respectively.

According to the June report, Lockheed Martin made an estimating error in the EVM baseline approved at the IBR. [redacted] However, according to the EVM report, Lockheed Martin officials said another estimating error should offset the excess costs accrued in June. [redacted]

However, if Lockheed Martin continues to accrue costs at the rate it did in June, the EVM report projects that Lockheed Martin’s cost for Phase 1 will be about [redacted], or approximately [redacted] more than the baseline budget of $ [redacted] (excluding Lockheed Martin’s [redacted] management reserve). Still, the projected cost is less than the $57.2 million contracted amount for Phase 1 of Sentinel. The report concluded that Lockheed Martin’s EVM data is not likely to show “a rapid and large improvement,” [redacted]

[redacted] FBI officials recently told us that Lockheed Martin has developed a plan showing how the variance in [redacted] will not have a negative impact on the cost of Phase 1.

The July 2006 EVM report also showed that the actual costs incurred by the PMO were about $1.1 million less than planned at this stage of the project. The EVM report attributes the spending variation primarily to vacancies in the PMO. The report concluded that the variance should not prevent the program from meeting its schedule or performance goals and recommended that PMO management continue to focus on filling the PMO’s vacancies (see discussion of PMO vacancies later in this report). As a result of joint Lockheed Martin-FBI decision to delay some purchases, Lockheed Martin did not receive hardware and software on the dates envisioned by the baseline schedule, causing the July EVM report on Lockheed Martin’s activities to show it being behind schedule by 10.1 percent.

The OMB requires agencies to report to it EVM variances greater than 10 percent, including what corrective actions the agency will take to remedy the variances. While the development of Sentinel depends heavily on Lockheed Martin’s performance, the Lockheed Martin EVM data is only part of the Sentinel EVM data. In July, the net schedule variance for the Sentinel program as a whole – the basis for whether it is required to report variances to the OMB – was 8.1 percent. Sentinel is on the OMB government-wide list of high-risk IT projects, meaning that Sentinel is a high-priority project, not that it is a troubled project. FBI officials said that because Sentinel is on the high-risk list, the FBI provides the OMB with monthly EVM data on the PMO’s performance and Lockheed Martin’s performance, regardless of whether or not there are any significant variances.

In our judgment, reporting from June to August 2006 shows that Sentinel’s EVM system is functioning as intended and providing FBI managers with warnings of issues that may affect Sentinel’s cost or schedule, including Lockheed Martin’s estimating errors and vacancies in the PMO. We also believe it was prudent for the FBI to allow Lockheed Martin to establish a management reserve to compensate for estimating errors. While we identified no significant immediate concerns, we are concerned about the future implications of the cost variances experienced by Lockheed Martin, especially the higher-than- expected labor rates. We will continue to monitor Sentinel’s EVM reporting to identify any concerns affecting the project baselines.

Risk Management

The FBI has instituted a risk management process to identify and mitigate the risks associated with the Sentinel project. The risk process is managed by the Sentinel Project Manager and a Risk Review Board, which meets biweekly. The most significant risks identified by the board are examined at monthly Program Management Review sessions and other Sentinel oversight meetings in accordance with the FBI’s LCMD.32

The purpose of risk management is to assist the program management team in identifying, assessing, categorizing, monitoring, controlling, and mitigating risks before they negatively affect a program. A risk management plan identifies the procedures used to manage risk throughout the life of the program. In addition to documenting the risk approach, the plan focuses on how the risk process is to be implemented; the roles and responsibilities of the program manager, program team, and development contractors for managing risk; how risks are to be tracked throughout the program life cycle; and how mitigation and contingency plans are implemented.

Program risks include risks that are identified and managed by the development contractor as well as risks that can only be identified and managed by the FBI. This requires that risk management be performed by the vendor and subcontractors to identify risks from the contractor perspective, and by the FBI program management team to identify risks from the FBI’s perspective. According the Sentinel Program Manager, PMO personnel attend and participate in Lockheed Martin’s risk management meetings. These weekly meetings are the primary reason that the Sentinel Risk Review Board continues to meet biweekly rather than weekly, as planned in the pre-acquisition phase.

According to the Sentinel Risk Management Plan, risks are to be identified, assessed, and tracked throughout the life of the project. When a proposed risk is brought before the Risk Review Board, the board’s voting members decide whether or not to accept the risk as an “open” risk and, if accepted, vote on the severity the risk will have on the project’s cost, schedule, and performance and the probability the risk will occur. Risks brought before the Risk Review Board are documented in a risk register, which includes the following:

The risk register lists open risks in rank order based on the risks’ probability and severity ratings. The PMO is responsible for tracking and periodically reviewing risks that are closed or resolved to prevent recurrence and to document the effectiveness and any unintended consequences of the mitigation strategy employed. Generally, Sentinel’s mitigation strategy has been to develop a series of actions that will decrease the probability a risk will occur or the severity of a risk’s impact on Sentinel.

As of August 2006, the FBI had identified, and was managing, 20 open risks to the Sentinel program, including the five top-ranked risks:

The severity of 9 of the 20 risks was classified as high, meaning that if the risks occurred they would have a major impact on Sentinel’s schedule, cost, or performance. One risk was classified as having a high probability of occurring. However, no high-impact risk was judged to have a high probability of occurrence. Many of these risks addressed subjects raised in our interviews of FBI personnel working on Sentinel, including successfully migrating data from ACS to Sentinel.

We view the FBI’s ability to successfully migrate data from the antiquated ACS system to Sentinel as a potentially significant challenge. If the migration were to fail or be seriously delayed, the FBI would need to try maintaining its legacy ACS system with all of its flaws. An inability to migrate the ACS data would also result in a Sentinel system that builds its data from the present day forward, without the benefit of years of investigative data compiled in the old system. Further, should ACS cease to be maintainable, that data could effectively be lost. The Sentinel Program Manager told us that the task of “cleaning” and reconciling the ACS data for migration into Sentinel is not technically difficult, and the FBI plans to use an available COTS software tool for that purpose. However, he pointed out that it will take a significant amount of work to accomplish. He also said that as a preventative measure intended to eliminate any delays in the overall project due to data cleansing, the FBI plans to cleanse data in the phase preceding the phase in which the data will be transferred to Sentinel.

Another potential risk in our opinion is the extent to which Sentinel will actually use commercial-off-the-shelf software modules as intended. A high degree of customization of the software could result in increased costs and schedule delays. The Sentinel Program Manager told us that the components for Sentinel are all off-the-shelf and little or no customization is anticipated. However, the key task will be configuring Sentinel’s various applications – such as the workflow, document management, searching and reporting, and electronic signatures – to all work together. The Program Manager noted that Lockheed Martin has successfully configured similar systems in other major projects, using some of the same software modules, including one at the Social Security Administration.

The August 2006 risk register also included 43 closed risks. Most of these risks had been closed for the following four reasons: the time for the risk to occur had passed; all the steps in the mitigation strategy had been completed; the risk was divided into multiple risks; or the risk was consolidated with another risk.

Our review of the risk register showed that the majority of the 20 open risks are most likely to affect the first two phases of the Sentinel project. As shown in the following chart, the Risk Review Board classified 15 of the 20 (75 percent) risks as having a potential impact on Phases 1 and 2.33 Of the 6 risks identified as having a potential impact on Phase 1, all but 2 were ranked within the top 10 highest priority risks. Appendix 5 lists the 20 risks in order of priority as well as the phase of Sentinel they could affect.

Open Risks by Sentinel Phase

Phase 1: 6, Phase 2: 9, Phase 3: 2, Phase 4: 2.
Source: OIG analysis of FBI data

The register also includes a statement describing the impact each risk would have on the project should it go unmitigated. We reviewed these statements and found that the consequences of the risks may affect the following aspects of Sentinel: the project’s cost and the need for additional funds, the scope of the work to be performed and the project’s requirements, the project’s schedule, the system’s functionality, and user acceptance of the system. As shown in the following chart, schedule, requirements or scope, and cost or funding are the most frequent consequence of the risks the FBI is currently managing.

Consequences of the Risks Currently
Being Managed by the Sentinel PMO

[Image Not Available Electronically]

 Source: OIG analysis of FBI data

According to the FBI’s risk management plan, the Sentinel PMO should develop a “contingency trigger” and a contingency plan for each risk it is managing that has a probability or severity rated as medium or higher by the Risk Review Board. A contingency trigger is an event that would convert a risk into an operational issue and cause the FBI to implement a risk’s contingency plan. However, we found that the risk register includes a contingency trigger and contingency plan for only 3 of the 18 risks required to have a contingency plan.34 In addition, only one of the five highest-ranked risks had a contingency trigger or plan. The Sentinel Program Manager told us that in some cases it is difficult to develop a contingency plan before the FBI’s preventive actions mitigate the likelihood or severity of the risk. Instead, he said the PMO is focusing on taking action to prevent risks from occurring and reducing the impact risks could have on the program. He also told us that many risks are temporary and as a project phase progresses the risk may become moot, at which point it is closed. If a risk occurs, the PMO said the FBI will develop corrective actions. We believe there should be a contingency plan developed for each major risk having the potential to result in a significant cost, schedule, or performance deviation from the project baselines.

Staffing of the Program Management Office

Due to the importance of the PMO in project oversight, our previous Sentinel audit raised concerns about the progress in staffing the Sentinel PMO. The PMO plays a critical role in assuring that the FBI implements a case management system that meets its needs. The PMO’s contract and program execution responsibilities include: (1) cost, schedule, and performance oversight; (2) LCMD project reviews; (3) award fee evaluations; (4) primary contractor’s documentation review and acceptance; (5) requirements and risk management; and (6) budget and financial management. In light of these responsibilities, having a qualified, dedicated PMO staff focused on program execution is critical to the success of the Sentinel project.

Since our March 2006 audit: the planned size of the PMO has decreased from 76 positions to 73 positions primarily because of less overlap in the project phases than initially anticipated; the PMO has reallocated positions among PMO units; and the PMO has filled 14 additional positions.35 As of October, 2006, the PMO consisted of 65 of the 73 personnel identified in the FBI’s Sentinel Staffing Plan (89 percent) as required to properly oversee the project. According to the FBI, the objective in staffing the PMO is to form an integrated team of subject matter experts from government, federally funded research and development centers, and system engineers and technical assistance contractors to maximize program expertise.36 The following table summarizes the PMO’s staffing level as of October 18, 2006, and shows the progress the FBI has made in staffing the office since January 2006.

SENTINEL PMO STAFFING REQUIREMENTS

Organizational Units Planned Staff (a) Staff on Board, January 2006 Staff on Board October 2006 (b)

Program Leadership

2

2

2

Direct Reporting Staff

8

6

8

Organization Change Management Team

4

2

3

Business Management

14

9

13

Program Integration

10

10

10

System Development

25

21

25

Transition

5

1

4

Operations & Maintenance

5

0

0

Total

73

51

65

Source:  The FBI

Notes:       (a)   Since January 2006, the Sentinel PMO has revised the total planned staff from 76 to 73. Also, the plan does not include individuals who are on temporary duty assignment to the project.

(b)   The number of staff on board includes three positions for which the FBI has selected candidates and is in the process of hiring.

For a more complete description of PMO staff and their duties, see Appendix 7.

The Sentinel Program Manager told us he did not intend to fill all of the PMO’s eight vacancies immediately because six positions are not needed until the project approaches Phase 2, which begins in early 2007. We agree that not filling positions until required is prudent. However, recruitment efforts need to be timed so that the six positions are filled when needed, allowing time for processing the new hires, including conducting background investigations. The FBI plans to begin recruiting for the Phase 2 positions by the end of October 2006. Moreover, even if some hiring is delayed, two current vacancies exist. Of the current vacancies, one is a government position — an intelligence analyst – and one is a contractor position – a planner. The Chief of the Business Management Unit said that government positions were the most difficult to fill because of the FBI’s hiring and background investigation processes. However, he said the steps the PMO had taken steps to expedite hiring, including interviewing applicants who had applied to an open FBI-wide job announcement for computer scientists, had been successful.

The Sentinel Program Manager said that he has gained more insight into the personnel requirements of the PMO and that these insights led him to decrease the number of planned staff by three and reallocate the planned staff among the PMO’s units. He said he made the most significant reduction, the elimination of four positions from the Transition Unit, because the current schedule has phases of the project overlapping less than originally anticipated. The following table shows the changes in the number of planned staff from January 2006 to October 2006.

Changes in Sentinel PMO Staffing Requirements,
January 2006 to October 2006

Organizational Unit Change in
Planned Staff

Organization Change Management Team

-1

Business Management

-2

System Development

+2

Transition

-4

Operations & Maintenance

+2

    Total

-3

Source: The FBI

In our opinion, the significant turnover of project management during the Trilogy project – 15 different key IT managers over the course of its life, including 10 individuals serving as project managers for various aspects of Trilogy – was a major reason for Trilogy’s problems. As of August 2006, three staff from the Sentinel PMO (five percent) had left the PMO since the project’s inception in March 2005. While the PMO has replaced all three staff, we will continue to monitor turnover of Sentinel PMO staff in future audits.

Improved Management Processes and Controls

In the early stages of the Trilogy project, the OIG and GAO recommended that the FBI establish Information Technology Investment Management (ITIM) processes to guide the development of its IT projects. In response, the FBI issued its Life Cycle Management Directive (LCMD) in 2004 after Trilogy was well underway. The LCMD established policies and guidance applicable to all FBI IT programs and projects, including Sentinel. As we reported in our March 2006 report on Sentinel, we believe the structure and controls imposed by the LCMD can help prevent many of the problems encountered in the VCF effort. Since our March 2006 report on Sentinel, the FBI has further refined its LCMD and is applying the revised directive to Sentinel.

The LCMD covers the entire IT system life cycle, including planning, acquisition, development, testing, and operations and maintenance. As a result, the LCMD provides the framework for standardized, repeatable, and sustainable processes and best practices in developing IT systems. Application of the IT systems life cycle within the LCMD can also enhance guidance for IT programs and projects, leverage technology, build institutional knowledge, and ensure that development is based on industry and government best practices.

The LCMD is comprised of four integrated components: life cycle phases, control gates, project level reviews, and key support processes. A diagram showing how these components relate to each other and a description of the life cycle phases, control gates, and project level reviews is found in Appendix 6.

LCMD Phases and Control Gates

The LCMD has established nine phases that occur during the development, implementation, and retirement of IT projects. During these phases, specific requirements must be met for the project to obtain the necessary FBI management approvals to proceed to the next phase. The approvals occur through seven control gates, where management boards meet to discuss and approve or disapprove a project’s progression to future phases of development, implementation, or retirement. As of August 2006, the Sentinel project had passed through the first four life cycle phases and is currently in the fifth phase – Design.

FBI LCMD PHASES

PHASE NAME DESCRIPTION
  1. Concept Exploration

Identifies the mission need, develops and evaluates alternate solutions, and develops the business plan.

  1. Requirements Development

Defines the operational, technical and test requirements, and initiates project planning.

  1. Acquisition Planning

Allocates the requirements among the development segments, researches and applies lessons learned from previous projects, identifies potential product and service providers, and identifies funding.

  1. Source Selection

Solicits and evaluates proposals and selects the product and service providers.

  1. Design

Creates detailed designs for system components, products, and interfaces; establishes testing procedures for a system’s individual components and products and for the testing of the entire system once completed.

  1. Development and Test

Produces and tests all system components, assembles and tests all products, and plans for system testing.

  1. Implementation and Integration

Executes functional, interface, system, and integration testing; provides user training; and accepts and transitions the product to operations.

  1. Operations and Maintenance

Maintains and supports the product, and manages and implements necessary modifications.

  1. Disposal

Shuts down the system operations and arranges for the orderly disposition of system assets

The seven control gate reviews provide management control and direction, decision-making, coordination, confirmation of successful performance of activities, and determination of a system’s readiness to proceed to the next life cycle phase. Decisions made at each control gate review dictate the next step for the IT program or project and may include: allowing an IT program or project to proceed to the next segment or phase, directing rework before proceeding to the next segment or phase, or terminating the IT program or project. The FBI’s Investment Management Project Review Board (IMPRB) – comprised of 12 representatives from each FBI division at the Assistant Director level and 4 representatives from the Office of the Chief Information Office, including the CIO – is responsible for approving an IT project’s passing through each control gate.

At the time of our previous Sentinel audit, the Sentinel project had received approval for the first two of the LCMD control gates: the System Concept on July 15, 2005, and the Acquisition Plan on July 29, 2005. As of August 2006, the Sentinel program had not requested or received approval for the third control gate. According to the Sentinel program manager, Phase 1 of Sentinel is scheduled to pass through Control Gate 3, Final Design Review, in late October 2006. Depending upon the development model employed, programs or projects may pass through the control gates more than once. Because Sentinel is being developed in phases, and the contractor must provide a system design for each phase, the project will pass through Control Gate 3 four times.

At each control gate, executive-level reviews determine system readiness to proceed to the next phase of the IT systems life cycle. Evidence of readiness is presented and discussed at each control gate review in the form of deliverables, checklists, and documented decisions. Regardless of the development model used for a particular program or project, all control gate reviews should be performed unless an agreement is made to skip or combine them. The control gate reviews also provide executive-level controls to ensure that IT projects are adequately supported and reviewed before a project receives additional funding. Appendix 6 lists the five executive-level review boards that serve as the decision authority for the control gate reviews.

The Gate 2 approval for Sentinel in July 2005 signified that the IMPRB accepted the overall project approach and cost estimate for acquiring the Sentinel system. Our previous audit showed that the FBI generally complied with the requirements of the then-current LCMD in performing the control gate reviews for Sentinel. However, two documents had not been completed at the time the control gate review was conducted: (1) the system security plan could not be developed at that time because the vendor needed to provide the project design details and, as of the date of the control gate review, the vendor had not been selected; and (2) the Independent Verification and Validation (IV&V) plan, to be implemented by a separate contractor to independently assess the implementation of the system according to technical and performance baselines, required a separate contract.

In August 2006, the Department awarded eight IV&V contracts for use throughout the FBI and parts of the Department of Justice. In September 2006, the FBI awarded a task order to Booz Allen Hamilton under one of those contracts for the IV&V of Phase 1 of Sentinel, with options for the remaining phases.37 According to the FBI, the independent contractor will monitor Lockheed Martin’s testing of the system software to ensure the software performs as intended. As an interim measure prior to the award of the FBI-wide IV&V contract, the FBI used one of the contractors supporting the PMO, Keane, Inc., to provide those services pending the availability of the independent contractors. To minimize any conflict of interest with its FBI PMO responsibilities, Keane’s activities have been limited to examining Lockheed Martin’s performance and not the FBI’s. We believe Keane is providing a useful service in helping the FBI monitor Lockheed Martin’s performance to date. However, the FBI and its oversight bodies need the assurance of a fully implemented IV&V process throughout the development of Sentinel. We believe this process should begin as soon as possible, and we intend to review the scope and results of the IV&V in our upcoming Sentinel audits.

The system security plan will provide the detail necessary for the completion of the critical certification and accreditation of the applications being created for Sentinel. Unless certification and accreditation is accomplished, Sentinel will not be allowed to operate due to security risks. According to FBI officials, it was not feasible to develop Sentinel’s system security plan prior to Sentinel’s final design, because the security plan is dependent on the design. However, as of August 2006, Lockheed Martin and the FBI had largely agreed on the design for Phase 1 of the Sentinel project, and Lockheed Martin provided the FBI with a draft of the system security plan for that phase. The Sentinel Program Manager said the plan should be completed by October 2006 when Lockheed Martin and the FBI are scheduled to finalize the design of Phase 1.

The plans for IV&V and system security are, in our opinion, crucial to ensuring the success of the Sentinel project. We will monitor the implementation of both plans in our subsequent audit work.

Project-Level Reviews

Project-level reviews help determine a project’s readiness to proceed to the next phase of the project life cycle. Each project-level review provides information to the executive-level control gates as data is developed and milestones are completed. Appendix 6 includes a list of the project-level reviews called for in the LCMD from the beginning of the Concept Exploration Phase to the end of the Design Phase.

In the Sentinel Program Management Plan, approved in August 2005, the FBI stated its intention to combine the Design Concept Review and Preliminary Design Review into a single review as part of the project’s LCMD tailoring approach. The LCMD provides for the tailoring of its requirements to meet a specific project’s needs, allowing a project to combine, streamline or eliminate events, and modify reports, documents, or deliverables. All tailoring decisions must be reviewed and approved at the Acquisition Plan Review Control Gate before finalizing them as part of the Program Management Plan. A review of the minutes from the Acquisition Plan Review indicates that the IMPRB was briefed on Sentinel’s LCMD tailoring approach.

To date, the FBI has conducted the Mission Needs Review, System Specification Review, Source Selection Acquisition Review, Contract Implementation Review, Requirements Clarification Review, combined Design Concept/Preliminary Design Review, and Critical Design Review. The FBI planned to conduct the Final Design Review in October 2006.

Based on our review of meeting minutes and documentation resulting from these reviews, it appears that the FBI is adhering to LCMD requirements in conducting these reviews and is following the schedule for producing the requisite deliverables established in the Program Management Plan.

Department Investment Review Board

In addition to the FBI’s management reviews, Sentinel has also been required to make periodic presentations to the Department Investment Review Board (DIRB). As part of the Department’s IT investment management process, the Department Investment Review Board oversees 10 to 15 of the Department’s IT investments with the greatest strategic and financial value. Periodic presentations to the Board, which includes the Deputy Attorney General and the Department’s CIO, should demonstrate adequate financial and risk management, alignment with the Department’s mission, and a sufficient return on investment. Each time Sentinel has appeared before the DIRB, the DIRB has approved the continued development of Sentinel. The Office of Management and Budget provides additional monitoring of Sentinel. For example, Sentinel is on the OMB government-wide list of high-risk IT projects, meaning that Sentinel is a high-priority project, not that it is a troubled project. Were the Sentinel project to encounter serious problems, it could be placed on the OMB watch list.

Change Management Process

The FBI has implemented a change management process to aid in controlling changes in Sentinel’s requirements that could result in cost growth, schedule delays, or performance problems. As shown in the following flowchart, the FBI evaluates the potential effect of each request for change (RFC) on project baselines. Changes that affect the cost or schedule must be approved by the System Configuration and Change Management Board and senior FBI management, up to and including the Deputy Director. According to FBI officials, the FBI Director has made it clear that the FBI’s requirements should not necessitate the customization of the commercial software being used in Sentinel. If the FBI’s business processes conflict with the capabilities of the software, the FBI is committed to changing its processes rather than the software. We reviewed five of the six RFCs and found they were approved in accordance with the FBI’s procedures.38

SENTINEL Request for Change (RFC) Process

[Image Not Available Electronically]

 Source: The FBI’s Sentinel Configuration Management Plan

However, while the FBI has established a reasonable system for limiting changes to the system’s requirements, the Sentinel PMO does not control all events that could affect Sentinel’s requirements. For example, the Sentinel PMO does not control the FBI’s legacy systems or policy changes affecting the FBI. The FBI continues to improve several IT systems that will either interface with Sentinel or be subsumed by Sentinel. These upgrades could add to the scope of Sentinel’s requirements by making more difficult the required interfaces. For example, the FBI continues to improve Guardian, an incident tracking system that Sentinel is expected to replace. According to Sentinel’s risk register, changes to Guardian may lead to changes in Sentinel’s functional or interface requirements, causing delays or cost increases. Also, changes in the FBI’s policies governing access to FBI computer systems could affect Sentinel’s requirements.

Information Sharing

Executive Order 13356 requires that federal agencies design information systems with priority given to the interchange of terrorism information among agencies and between agencies and appropriate authorities of state, local, and tribal governments. According to FBI officials, the FBI will build Sentinel to share information based on the National Information Exchange Model (NIEM), a joint project of the Departments of Justice and Homeland Security.39 The NIEM also has the support of the Director of National Intelligence. When finalized, the model will essentially become the new government-wide law enforcement and intelligence agency standard and will serve as the vehicle for future information exchange. However, because the NIEM standards have not been finalized, the FBI has not modified Sentinel’s information sharing requirements to meet the draft NIEM standards currently available. FBI officials said that Sentinel will be modified to meet final NIEM standards.

The National Information Exchange Model

Agencies are not able to exchange information if they maintain legacy systems that were not designed for information exchange. The NIEM information sharing standard, which FBI officials said should be finalized in January 2007, is intended to create a national enterprise-wide framework to facilitate information sharing across all levels of government by developing common information exchange standards.

Previously, many agencies shared information with other agencies on a strict “need-to-know” basis and therefore provided little or no access to their systems. In addition, many agencies maintained databases with applications residing on networks that could not communicate with other agencies’ networks. As a result of the September 11, 2001, terrorist attacks, information sharing became a high priority. Agencies found that they did not have enough time or resources to modify their systems fast enough to allow for real time information exchange. In an attempt to remedy the immediate problem, agencies built “bridges” to facilitate information exchange, such as Law Enforcement Information Exchange (LInX) and the Regional Data Exchange (R-DEx).40 R-DEx permits data to be accessed from another computer system and, based on security clearance and the need to know the information, the requester is permitted access to information up to the security level deemed necessary. Standards had to be developed so that information is characterized the same way, no matter what agency originates it, to facilitate the information exchange. NIEM is the effort to standardize the data.

Interagency Coordination on Sentinel

We interviewed representatives from the Drug Enforcement Administration (DEA), the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF), and the Department of Homeland Security (DHS) to determine the extent of each agency’s involvement with Sentinel and the need to retrofit their case management systems to communicate with Sentinel.

According to the DEA, two staff members participated in Sentinel coordination meetings and used these meetings to identify changes to Sentinel that would require the DEA to retrofit its case management system, Impact. The DEA is also involved with the development and usage of the NIEM information sharing standard.

The ATF told us it has had limited involvement with Sentinel. The ATF has a representative on the DIRB as a non-voting member and has another staff member who serves as the liaison with the FBI for Sentinel. The ATF is trying to avoid investing large amounts of money in its case management system until after Sentinel is developed because the ATF representative believes that modifications will be needed to its case management system, N-Force. The ATF representative said that if the FBI builds a generic system that other agencies can use, it will be good for everyone; if not, it will not be very helpful to the ATF. In response, FBI officials said Sentinel will be a flexible system that other agencies can configure to meet their needs.

According to a DHS official, a DHS representative will participate with the FBI on the FBI Change Control Board. The DHS representative stated that during the early stages of the Sentinel project, the DHS provided four of its employees and two contractors to support the Sentinel PMO in the areas of case management, system analysis, biometrics, immigration enforcement, strategic planning, and technical architecture. Similar to concerns expressed by the ATF, the DHS hopes Sentinel will not be too FBI-specific so that it will be usable by other agencies. The DHS is developing its own case management system, the Consolidated Enforcement Environment, and expects to use some of the knowledge and reusable components from Sentinel to reduce the costs of DHS’s own case management system.

Lockheed Martin’s Observations on Sentinel

During our audit, we met with Lockheed Martin’s project manager for Sentinel to obtain his perspective on how the project is progressing. The project manager stated that he is confident the project would meet its targeted budget and schedule, but that there were project risks that need ongoing attention. In his opinion, user acceptance and utilization was the most significant risk to the project. He explained that this risk is being addressed in several ways during the implementation of Sentinel. First, a prototype of the Phase 1 products were provided to agents in three field offices to obtain input on what should be added, removed, or changed. Similar assessments would be made in the future phases of Sentinel. Second, organizational change management strategies were being implemented within the FBI so that the transition from current workflows and IT systems used by agents and analysts to the new Sentinel workflow and systems would be facilitated. For example, Sentinel users will be trained as the system is brought online. This would allow users to immediately utilize the training on how to operate the system. System trainers will remain after the system is brought online in order to assist any users requiring further training or help. Finally, the project manager said that Lockheed Martin is taking steps to ensure that all of the significant workflows that will be affected by Sentinel will be addressed in planning the system. This will ensure that users will readily use the system to perform their day-to-day activities.

While the project manager viewed user acceptance and utilization as a significant risk, and Lockheed Martin is taking steps to ensure that the processes that need to be included within Sentinel are covered, we believe other risks are more significant, as discussed earlier in this report. In our view, because Sentinel will be the only FBI case file system and employees will have to use the system in order to perform their jobs, we do not believe user acceptance and utilization is a significant concern. However, a related risk, that all of the processes used by the FBI are included within the functionality of Sentinel, is a greater concern. We believe that the steps being taken by the FBI and Lockheed Martin should ensure that all of the necessary workflow processes are included within Sentinel. In future audits we will monitor whether agents and analysts are finding the new Sentinel applications to be user-friendly and include all of the required functionality necessary to perform their jobs.

Other risks the Lockheed Martin project manager identified include the control over system requirements, the migration of data from the antiquated ACS system to Sentinel, and the connectivity of all of the field offices to the Sentinel databases. He noted that the FBI is paying particular attention to the requirements of the system and making efforts to eliminate “requirements creep.” The project manager pointed out that to date the FBI has only made six requests for change. Of those requests, one involved a security item that Lockheed Martin was implementing differently than the FBI anticipated. Lockheed Martin agreed to change the way the security issue was implemented and funded the changes through its management reserve. Four of the requests for change amounted to issues that were implemented at no cost and did not affect the project schedule. Lockheed Martin is considering the sixth request, which deals with the project’s cost classification system.

The project manager told us that Lockheed Martin and the FBI are dealing with the risks involved in migrating ACS data to Sentinel. He explained that a software tool had been purchased to take the data from the ACS and “cleanse” it by determining the attributes of the data, placing the data into defined categories, and then placing the data into the correct locations in Sentinel. The significant risks of this process include the creation of rules to properly categorize the data within ACS and place it in Sentinel, and also what occurs when data is not properly cleansed. To address this risk, the software has been tested using sample case files. However, according to the project manager, until actual case file information is used, it will not be known how many of the case files will not be able to be cleansed and uploaded into Sentinel. For those case files that cannot be cleansed, a review board of Lockheed Martin and FBI personnel has been established to manually review the data and determine where it should be placed within the Sentinel system. Because no one knows how many case files will not be able to be cleansed, the time required to cleanse or review all of the ACS case file data cannot be estimated. As discussed earlier in this report, we consider the migration of data from ACS into Sentinel as a significant risk that could affect both the cost and schedule of bringing Sentinel fully online.

The last risk the Lockheed Martin project manager cited was that of the FBI’s IT infrastructure being able to adequately handle the signal traffic over its networks. With the creation of a true electronic case file system that will be used by about 15,000 agents and analysts on a continuing basis, a substantial network is required so that the information can be passed quickly within the system. According to the project manager, Lockheed Martin is not responsible within the Sentinel contract to ensure that the FBI’s entire network operates efficiently. Instead, Lockheed Martin is responsible for building the hardware and software portions of Sentinel that will be located at two sites, one as the primary site and the second as a backup site. The FBI is responsible for networking the system. We agree with Lockheed Martin that the connectivity of Sentinel is a major concern, and we will be following up on this concern in future audit work.

The project manager said that from his perspective Lockheed Martin and the FBI are working well together. Specifically, there has been significant interaction between the two groups in management meetings, including the risk boards that have been established both by the FBI and by Lockheed Martin. Working groups have also been established between the two organizations where Lockheed Martin’s teams responsible for drafting products are working with FBI staff responsible for reviewing the products, thereby providing clear communications on what is expected for each product. Overall, the project manager believed that the FBI is performing well its role as a good customer in providing direct feedback and maintaining the original requirements for the Sentinel project.

Regarding the Sentinel budget, the project manager stated that Lockheed Martin’s costs possibly could be held to under the $305 million contract amount because of two changes in the implementation of the project. First, since the time of Lockheed Martin’s proposal for the project, new hardware to house database files has come on the market that will lessen the cost of some aspects of the project. Second, the FBI reduced the requirement for the number of trainers needed by performing the training at fewer locations. The training plan originally called for about 120 trainers, but now requires only about 50 over the 6 to 7 weeks of implementation in the field for Phase 1.

Conclusion

By establishing stronger ITIM processes and an array of monitoring and control mechanisms, the FBI has positioned itself to better manage the Sentinel project and avoid the problems that occurred in the Trilogy and VCF projects. However, FBI officials agreed this does not mean that Sentinel is risk free. While the FBI has corrected or alleviated many of the concerns we raised in our March 2006 report, several areas warrant continued attention to avoid potentially serious problems as the project progresses.

As a result of management improvements and the FBI’s structuring of Sentinel into four phases, Sentinel poses much less risk to the FBI than the failed VCF project. Management improvements that reduce the risks include rigorous reviews and control gates required by the FBI’s LCMD; new procedures to track and control costs; the use of an EVM system to detect deviations from cost, schedule, or performance baselines; a change management control process; and a risk management process. Risks are also minimized by the way the FBI structured the Sentinel program, such as the use of off-the-shelf components, conducting the project in a phased approach with specific deliverables, and the establishment of firm baselines and design requirements for each project phase. Further, the FBI will adopt the new information sharing standards required by the Department, has made progress toward completing and implementing plans for system security and the IV&V of the system, and has added staff to the Sentinel PMO.

However, some of the concerns from our March 2006 report remain. These concerns include: (1) uncertainty over the funding for the project and the effect on the FBI’s operations should an unexpected level of reprogramming of FBI funds be required to continue Sentinel, and (2) the need to fill remaining vacancies in the Sentinel PMO to ensure proper FBI oversight of the project. In addition, our current review identified concerns over: (1) the uncertainty of total project cost estimates, and (2) the need for contingency plans for the risks the PMO is currently monitoring. Because the FBI has, in our judgment, only a tentative estimate of project costs, we believe the FBI needs to periodically update its cost estimate for the Sentinel project based on actual cost experience and inform Congress and the Department of any revisions to its estimate. We also believe the FBI should establish contingency plans for risks that could seriously affect the cost, schedule, or performance of the Sentinel project.

We believe the FBI’s approach to the Sentinel project and the processes and controls it has developed, if implemented and followed, provide reasonable assurance that Sentinel can be developed and deployed successfully. However, there are serious project risks such as the ability to configure all of Sentinel’s components into a seamless system and to migrate ACS data into Sentinel. Project costs and funding are also somewhat uncertain. The OIG will continue to monitor and periodically issue audit reports throughout the four overlapping phases of the FBI’s Sentinel project in an effort to track the FBI’s progress and identify any emerging concerns.

Recommendations

We recommend that the FBI:

  1. Ensure the management reserve is based on an assessment of project risks for each phase and for the project overall.

  2. Periodically update the estimate of total project costs as actual cost data is available.

  3. Complete contingency plans as required by the Sentinel Risk Management Plan.

  4. Ensure that the IV&V process is conducted through project completion.

  5. Complete hiring as soon as possible for the vacant PMO positions needed during the current phase of the project.



Footnotes
  1. The development contract under the GWAC is cost-plus-award-fee. However, all materials are cost-plus-fixed-fee and travel is cost reimbursable only.

  2. Independent government cost estimates help federal agencies budget for projects, compare contractor proposals, and evaluate the reasonableness of costs in contractor proposals.

  3. A management reserve, also known as a risk reserve, is a budgeted contingency fund used to cover costs not anticipated at the time a project’s cost estimate is developed.

  4. The Federal Acquisition Regulation (FAR) requires that cost realism analysis be performed on cost-reimbursement contracts to determine the probable cost of performance for each bidder. Cost realism analysis is the process of independently reviewing and evaluating specific elements of each proposed cost estimate to determine whether the estimated cost elements are realistic for the work performed, reflect a clear understanding of the requirements, and are consistent with the unique methods of performance and materials described in the bidder’s technical proposal.

  5. [redacted]

  6. [redacted]

  7. The LCMD, which is a set of policies applicable to all FBI IT programs and projects, contains a framework for standardized, repeatable, and sustainable processes for developing IT systems. The LCMD covers the entire IT system life cycle, including planning, acquisition, development, testing, and operations and maintenance. See Appendix 6 for a detailed description of the LCMD.

  8. A System Requirements Specification defines a system’s technical requirements in quantifiable and verifiable terms and the methods to be used to ensure that each requirement has been met.

  9. The FBI’s Finance Division, not the Sentinel PMO, controls the management reserve.

  10. ANSI/EIA Standard 748-A is the criteria selected by the OMB for EVM systems. The standard includes 32 specific criteria in five process areas necessary for a sufficient EVM system: (1) organization; (2) planning, scheduling and budgeting; (3) accounting; (4) analysis and management reports; and (5) revisions and data maintenance.

  11. In addition to the risk management processes cited above, the following receive briefings that include information about Sentinel risks: the FBI Director (weekly); a review team with senior representatives from the Department of Justice, OMB, and Director of National Intelligence (monthly); the FBI CIO’s Advisory Council (bi-monthly); the FBI Director’s Advisory Board (as requested); and congressional oversight committees (quarterly).

  12. One risk was not assigned a phase in the risk register; as a result, the chart includes a total of 19 risks rather than 20.

  13. The remaining two risks did not have probability or severity ratings, so we could not determine whether they required contingency plans.

  14. Three hires are in the process of coming on board.

  15. Federally funded research and development centers are nonprofit organizations sponsored and funded by the U.S. government to assist government agencies with scientific research and analysis, systems development, and systems acquisition.

  16. At the time our audit, all of the specific IV&V activities for Sentinel had not been determined. However, IV&V may include oversight of program management processes and assessments related to the development contractor’s performance.

  17. One RFC was approved after we completed audit fieldwork.

  18. The Sentinel statement of work, which was developed prior to the release of the draft National Information Exchange Model, requires Sentinel to be built to the Global Justice XML Model.

  19. The LInX initiative is a project designed to enhance information sharing between local, state, and federal law enforcement by providing participating law enforcement agencies with secure access to regional crime and incident data, enabling investigators to search across jurisdictional boundaries to help solve crimes and resolve suspicious events. R-DEx gives state, local, and tribal law enforcement access to federal investigative and intelligence information. R-DEx provides detectives, investigators, and analysts the ability to view the linkage across multiple cases and their jurisdictions. These links include individuals, vehicles, weapons, addresses, phone numbers or other types of links. It also allows cases to be plotted on maps in order to identify geographical patterns or links.



« Previous Table of Contents Next »