|Return to the USDOJ/OIG Home Page|
Federal Bureau of Investigation's Management of Information Technology Investments
Report No. 03-09
Office of the Inspector General
1. The FBI’s Management of IT Investments
The FBI is not effectively selecting, controlling, and evaluating its IT investments because it has not fully implemented any of the critical processes necessary for successful IT investment management. In the past, the FBI has not given sufficient attention to information technology investment management. As a result, the FBI continues to spend hundreds of millions of dollars on IT projects without having adequate selection and project management controls in place to ensure that IT projects will meet intended goals. However, since the FBI developed its ITIM Model and Transition Plan in January 2002, it has focused more management attention in this area and has made progress towards attaining a basic IT investment management foundation. Much of the progress has been in the “select” phase of the Plan, which was pilot tested in the Spring of 2002.
The ability of the FBI to completely implement the “control” and “evaluate” phases of the Plan, and achieve mature IT investment processes that can lead to enhanced mission performance, will require the FBI to increase its efforts in: (1) fully developing and documenting its new ITIM process; (2) requiring more input and participation from ITIM managers and users; and (3) further developing its project management and enterprise architecture functions. While the FBI recognizes many of these needs and has taken initial steps to address the needs, further action in these areas is needed to ensure that IT projects are developed within cost and schedule requirements, and meet performance expectations. The Trilogy project provides an example of how the non-implementation of fundamental IT investment management practices can put a project at risk of not delivering, within cost and schedule requirements, what was promised.
A. The FBI’s Progress Toward Attaining a Basic IT Investment Management Foundation
Although the FBI made measurable progress in improving its IT investment capability since it initiated a new ITIM process in early 2002, the FBI still lacks a complete foundation to build its IT investment maturity processes, and therefore is still in Stage One maturity.31 In the past, the FBI has not given sufficient management attention to IT investments. Because of the lack of management attention in the past, the FBI failed to implement the critical processes necessary to build an IT investment foundation. These critical processes include: (1) IT investment review board operation, (2) IT project oversight, (3) IT system and project identification and tracking, (4) business needs identification for IT projects, and (5) IT proposal selection.
(1) Importance of Attaining a Basic IT Investment Management Foundation
The primary purpose for attaining a basic IT investment management capability (Stage Two maturity) is to build the foundation for repeatable, successful IT project-level investment control and selection processes. Effective control processes over IT projects ensure that deviations from cost and schedule baselines can be identified and corrected. Selection processes ensure that the FBI has an effective methodology for approving only IT projects that are consistent with its needs and goals. According to the Framework, an organization can only achieve Stage Two maturity if it fully implements the following five critical processes:
To implement these critical processes, the FBI must execute a total of 38 key practices as defined in the Framework, or have alternative practices in place that are designed to achieve the same outcome.
At the start of our audit in January 2002, FBI officials told us that the Bureau was in the process of developing its new ITIM process. Although its ITIM process was still in the development stages, FBI officials told us that the FBI was executing certain key practices from Stage Two of the Framework. Additionally, the FBI officials said in March 2002 that they would pilot test ITIM processes pertaining to the selection of new IT proposals for the FY 2004 budget cycle. Further, the Plan establishes the FBI’s goal to fully attain Stage Two maturity for the FY 2005 budget cycle that starts in March of 2003, thereby establishing the foundation for enhanced investment capability.
(2) Summary of the FBI’s Progress Toward Attaining Stage Two Maturity
Based on the FBI’s responses to the self-assessment32 (and our validation of those responses), the FBI did not yet have in place any of the five critical processes associated with Stage Two maturity. However, since the FBI began pilot testing the select phase of its Plan in March 2002, it has made progress towards implementing the 38 key practices comprising the five critical processes - particularly in the area of selecting new proposals for IT projects. Specifically, at the beginning of our audit in January 2002, the FBI was only executing 4 of the 38 required key practices; however, as of June 2002, the FBI was executing 14 of the key practices. The following table provides a summary of the FBI’s progress toward implementing the key practices required for each critical process.
FBI Progress Toward Attaining Stage Two Maturity
|Critical Process||Status of ImplementingCritical Process||Total Key Practices Required||Key Practices Executed Prior to March 2002||Key Practices Executed as of June 2002|
|1. IT Investment Board Operation||Not Implemented||6||0||2|
|2. IT Project Oversight||Not Implemented||11||1||2|
|3. IT Project Identification||Not Implemented||7||1||2|
|4. Business Needs Identification for IT Projects||Not Implemented||8||2||3|
|5. Proposal Selection||Not Yet Implemented, but Substantial Progress Made||6||0||5|
|Source: OIG analyses|
For the remainder of section A of this finding, we provide detailed narratives of the FBI’s progress toward implementing each of the five critical processes. We also provide specific recommendations for expediting implementation of the critical processes and establishing more timely Stage Two maturity.
Each critical process contains core elements that provide the common framework for the process. For example, the organizational commitment element addresses the management actions that ensure the critical process is established and will endure; the prerequisites element addresses the conditions that must exist within an organization to successfully implement a critical process; and the activities element consists of the key practices necessary to implement a critical process. The key practices are the tasks within a core element that must be performed by an organization to effectively implement and institutionalize a critical process.
(3) Critical Process #1: IT Investment Review Board Operation
Depending on its size, structure, and culture, an organization may have more than one IT investment review board. The purpose of such boards is to ensure that basic policies for selecting, controlling, and evaluating IT investments are developed, institutionalized, and consistently followed throughout the organization. To establish a fully functioning investment review board, the FBI must execute the following six key practices:
The following table summarizes the FBI’s progress toward implementing fully functioning investment review boards.
FBI Progress Toward Implementing Fully Functioning Investment Review Boards (Critical Process #1)
|Key Practice||Key Practice Execution Status Prior to March 2002||Key Practice Execution Status as of June 2002|
|Organizational Commitment 1. An organization-specific IT investment process guide is created to direct each board’s operations.||Not Executed||Executed|
|Organizational Commitment 2. Organization executives and line managers support and carry out IT investment board decisions.||Not Executed||Not Executed|
|Prerequisite 1. Adequate resources are provided for operating each IT investment board.||Not Executed||Not Executed|
|Prerequisite 2. Board members understand the investment board’s policies and procedures and exhibit core competencies in using the IT investment approach via training, education, or experience.||Not Executed||Not Executed|
|Activity 1. Each IT investment board is created and defined with board membership integrating both IT and business knowledge.||Not Executed||Executed|
|Activity 2. Each IT investment board operates according to written policies and procedures in the organization-specific IT investment process guide.||Not Executed||Not Executed|
|Source: OIG analyses|
a. The FBI Has Executed Two of the Six Key Practices Associated with IT Investment Board Operation
We determined that the FBI executed two of the six key practices associated with implementing this critical process. Specifically, the FBI created an IT investment process guide containing policies and procedures to direct board operations (Organizational Commitment 1), and it created and defined three investment review boards integrating both IT and business knowledge (Activity 1).
Regarding the IT investment process guide (Organizational Commitment 1), in January 2002 the FBI issued its IT Investment Model and Transition Plan33 containing required guide elements prescribed by the Framework including:
Regarding the investment review boards (Activity 1), in June 2002 the Director approved board charters for each of the three investment review boards (the Executive Review Board, the Project Oversight Committee, and the Technical Review Board) that defined board membership and the responsibilities of board members.
The boards actually began functioning as early as March 2002, in conjunction with the FBI’s pilot testing of ITIM processes pertaining to the selection of new IT proposals for the FY 2004 budget cycle. Although board membership consists mostly of FBI managers who do not have extensive IT knowledge,35 the use of subject matter experts and reliance on the Enterprise Architecture Technical Committee36 can compensate for a lack of IT knowledge.
b. The FBI Must Execute Four of the Six Key Practices Associated with IT Investment Board Operation
Although progress has been made, the FBI does not have fully functioning IT investment boards because it still must execute four of the six key practices associated with this critical process. Specifically, the FBI must ensure that:
Regarding Organizational Commitment 2 and Activity 2, the approved charters for the investment review boards have been in effect since June 2002. Consequently, the FBI did not have sufficient data for us to assess whether managers and support staff effectively carried out board decisions and whether the boards operated according to the written policies and procedures contained in the Plan and board charters.
Regarding Prerequisites 1 and 2, in our judgment the FBI did not adequately plan sufficient time to ensure the IT investment boards operated effectively. Specifically, the FBI did not provide ample time between the initial draft of its Plan (January 25, 2002) and the March 2002 pilot testing of the select phase to adequately prepare and train IT board members. The DOJ originally instructed each component to begin developing an ITIM process in January 2001.37 In June 2001, the DOJ required each component to complete and submit to JMD an ITIM process and transition plan by the end of 2001.38 The DOJ also required each component to initiate the ITIM process for the FY 2004 budget cycle, which for the FBI began in March 2002. Consequently, the FBI had only one full month between the issuance of the Plan in late January 2002 and the initiation of the select phase of its ITIM process in early March 2002.
The ITIM Program Office Manager told us that the former FBI Chief Financial Officer would not approve the use of a contractor to assist in the development of the ITIM process earlier in the year. According to the former Chief Financial Officer, she had concerns that federal contracting regulations prohibited the FBI from using a contractor to perform a service that involves budget planning. However, following her transfer to another division in December 2001, the Information Resources Management Section received authorization to hire a contractor to assist with the development and implementation of the ITIM process.
We believe that without an ITIM contractor the FBI still had the opportunity to begin planning its ITIM process (including the training of board members) early in 2001. In fact, had the FBI better coordinated other ongoing efforts to develop processes that complement IT investment management, the FBI could have made significant strides in initiating its ITIM process during 2001 without expending additional resources. As discussed in section B of this finding, the FBI did not sufficiently incorporate (a) its enterprise architecture function (which was under development in 2001) and (b) the Project Management Process (issued in draft form in October 2001) into the development of its ITIM process. Enterprise architecture and project management not only complement the ITIM process, but also facilitate the maturation of ITIM. As discussed in section B of this finding, the FBI did not effectively utilize its internal resources when it developed its ITIM process through the use of a contractor because the FBI did not adequately consider the complementary, and potentially duplicative efforts that were already underway.
Not providing ample time resulted in inadequate training of board members and minimal preparation time to develop IT proposals. For example, Technical Review Board members had only 3 business days to review over 50 IT proposals prior to their first board meeting. FBI officials recognized these implementation issues in the Post-Implementation Review of the select phase pilot test.
In preparing board members for their duties, the FBI has thus far only provided one overview training session for board members and other users in the ITIM process. Additionally, while FBI officials have told us more ITIM training will be forthcoming, they have not provided us with any specific training plans for the future. Further, members of the Technical Review Board told us that board members, especially the Assistant Directors and EADs, do not have extensive knowledge in managing IT and must rely heavily on knowledgeable staff and other subject matter experts.
For the ITIM process to become institutionalized, the FBI must have a better training program. According to the Framework, board members should understand the board’s policies, roles, rules, and activities and be capable of carrying out their responsibilities competently. Education and training for members is needed in areas such as economic evaluation techniques, capital budgeting methods, and performance measurement strategies. The FBI’s Post-Implementation Review of the select phase pilot testing recommends “role-specific” training sessions for the following ITIM roles: (1) ITIM Liaison representatives,39 (2) Executive Review Board members, (3) Program Oversight Review Board members, (4) Technical Review Board members, and (5) ITIM stakeholders. It further recommends continuation of the overview training sessions previously provided, plus training for ITIM specific tools, such as the concept paper (containing the preliminary feasibility analysis), the OMB Exhibit 300 (containing the business case analyses), and IT proposal summaries.
FBI officials told us that time constraints were the main cause for not executing the four key practices identified above. As a result, there was insufficient time to introduce ITIM concepts to board members and other ITIM users. As mentioned above, the DOJ required each component to develop and begin implementation of an ITIM process for the FY 2004 budget cycle, which for the FBI begins in March 2002. Although FBI officials were aware of the requirement to initiate and adopt an ITIM process in January 2001, it was not until December 2001 that it began to develop its ITIM process. Had the FBI initiated more timely action to develop its ITIM process, it would have had significantly more time to prepare and train ITIM board members and other users. Without sufficient training and allocation of time to perform required tasks, the investment review boards cannot carry out their responsibilities to effectively select, control and evaluate projects.
We recommend that the Director of the FBI:
(4) Critical Process #2: IT Project Oversight
The purpose of this critical process is to ensure that the FBI’s investment review boards and project development teams provide effective oversight for its IT projects throughout all phases of the project life-cycle. IT investment boards generally review each project’s progress toward predicted cost and schedule expectations as well as anticipated benefits and risk exposure. The board members also employ early warning systems that enable them to take corrective actions at the first signs of cost, schedule, and performance slippages. Individual project development teams are responsible for meeting project milestones within the expected cost and schedule parameters.
Effective project oversight requires, among other things:
We concluded that the FBI is not effectively overseeing its ongoing IT projects. While the FBI maintained project management guidance and had three IT investment review boards in operation since March 2002, these activities have not adequately supported the FBI’s IT project oversight function. Our testing of the key practices associated with this critical process indicates that the FBI is executing only two out of the eleven key practices required to implement this critical process. The following table summarizes FBI progress toward implementing IT project oversight.
FBI Progress Toward Implementing IT Project Oversight (Critical Process #2)
|Key Practice||Key Practice Execution Status Prior to March 2002||Key Practice Execution Status as of June 2002|
|Organizational Commitment 1. The organization has written policies and procedures for project management.||Executed||Executed|
|Organizational Commitment 2. The organization has written policies and procedures for management oversight of IT projects.||Not Executed||Not Executed|
|Prerequisite 1. Adequate resources are provided to assist the boards in overseeing IT projects.||Not Executed||Not Executed|
|Prerequisite 2. Each IT project has and maintains an approved project management plan that includes cost and schedule controls.||Not Executed||Not Executed|
|Prerequisite 3. An IT investment review board is operating.||Not Executed||Executed|
|Prerequisite 4. Information from the IT asset inventory is used by the IT investment board as applicable.||Not Executed||Not Executed|
|Activity 1. Each project's up-to-date cost and schedule data are provided to the appropriate IT investment board.||Not Executed||Not Executed|
|Activity 2. Using established criteria, the IT investment board oversees each IT project's performance regularly by comparing actual cost and schedule data to expectations.||Not Executed||Not Executed|
|Activity 3. The IT investment board performs special reviews of projects that have not met predetermined performance standards.||Not Executed||Not Executed|
|Activity 4. Appropriate corrective actions for each under-performing project are defined, documented, and agreed to by the IT investment board and the project manager.||Not Executed||Not Executed|
|Activity 5. Corrective actions are implemented and tracked until the desired outcome is achieved.||Not Executed||Not Executed|
|Source: OIG analyses|
a. The FBI Has Executed Two of the Eleven Key Practices Associated with IT Project Oversight
While the FBI has project management guidance (and is therefore executing the key practice relating to the existence of project management methodology), the guidance is not being followed on a consistent basis. In fact, depending on whom we talked to, we obtained different answers as to which document represented the FBI’s official project management guidance.
For example, although IRD managers were aware that the DOJ’s System Development Life-Cycle is the FBI’s official project management methodology, they acknowledged that it is not consistently applied. Laboratory Division management officials told us that they do not follow the DOJ’s System Development Life-Cycle methodology, but rather have adopted their own project management system based on one used at the Department of Defense because it better meets their needs. CJIS Division management officials told us that although its Contract Administration Office is responsible for project management functions, they were not following any specific project methodology.
Other FBI personnel from the Information Resources Management Section told us the Project Management Process, developed by the FBI’s Inspection Division, was the FBI’s project management guidance. However, Inspection Division personnel indicated to us that the Project Management Process was still pending approval from the Director, as of June 2002. As a result, there appeared to be confusion among FBI officials as to what the official project management guidance was. As of June 2002, the Project Management Process had not been approved, nor was it being used to manage IT projects.
As previously discussed in the prior report section pertaining to the investment review board critical process, the FBI established three IT investment review boards in March 2002 (the Executive Review Board, the Project Oversight Committee, and the Technical Review Board). Although the investment review boards are operating, the boards have not yet been involved in project oversight. As the ITIM process continues to evolve, project oversight by these boards should increase accordingly.
b. The FBI Must Execute Nine of the Eleven Key Practices Associated with IT Project Oversight
Based on our analyses, the FBI does not have effective IT project oversight because it has not yet executed nine out of the eleven key practices associated with this critical process. Specifically, the FBI must ensure that:
Regarding Organizational Commitment 2, the FBI has not developed written policies and procedures for management oversight of IT projects. While the Plan provides a conceptual basis for board oversight of IT projects and the board charters define the boards’ responsibilities, the FBI does not have the specific policies and procedures in place for overseeing and controlling projects. FBI officials have acknowledged to us that the Plan was never intended to represent the complete and final policies and procedures for management oversight of IT projects. The Plan states that it is a fluid document that will need to be modified and supplemented as the pilot test is performed. As a result, FBI officials recognize that additional policies and procedures must be developed. As of June 2002, FBI officials have told us they are in the process of developing these specific policies and procedures for the control phase of the ITIM pilot test.
Regarding Prerequisite 1 (providing adequate resources to the boards), we concluded that this key practice has not been executed because as of June 2002, the FBI did not have a functioning project management office to assist the boards in overseeing IT projects. The Plan calls for a functioning project management office to assist the boards, especially the Project Oversight Committee, and consequently is a necessary resource for IT project oversight. As of June 2002, the FBI has not yet utilized its project management function to assist the Project Oversight Committee in IT investment decision-making.
The functioning project management office represents a critical resource to the Project Oversight Committee and thus to IT project oversight. In our judgment, the functioning project management office needs to have jurisdiction over IT projects throughout the Bureau, rather than limit its responsibilities to division-specific projects. Until June 2002, the FBI lacked a functioning project management office that had jurisdiction over IT projects throughout the Bureau. Rather than having a centralized project management office, independent of individual divisions, the FBI maintained three separate division-level project management offices to manage IT projects. These three separate project management functions were maintained in the IRD, CJIS, and Laboratory Divisions, contributing to inefficiencies in project coordination and the risk of “stove piping” projects. Because of its importance in supporting the ITIM process, the subject of establishing and maintaining a centralized project management office is further discussed later in this report.
Regarding Prerequisite 2, we determined that each IT project does not have an approved project management plan that includes cost and schedule controls. Personnel from the IRD project management office told us that generally IT projects with high visibility have project management plans that include cost and schedule controls. However, other lower visibility projects have less rigid controls in place. This condition developed because the IRD project management office did not uniformly enforce the development of project management plans by all IT project managers. In our judgment, projects under the IRD’s discretion have not been adequately controlled. Although personnel from the CJIS and Laboratory Divisions indicated that IT projects under their respective divisions did have management plans with cost and schedule controls, without a functioning board that approves and monitors these project management plans FBI managers have no assurance that IT projects are effectively managed in accordance with uniform standards.
Regarding Prerequisite 4, the FBI has not yet developed an IT asset inventory; consequently, the FBI’s investment review boards are not aware of all the IT projects and resources for which the boards are responsible. FBI managers told us they were in the process of developing an IT asset inventory. However, at the time of our audit they were unable to provide an estimated date for completing the inventory. Unless the investment review board members are fully cognizant of the IT projects and resources for which they are responsible, the boards cannot exercise effective oversight of ongoing IT projects. Additional details pertaining to the FBI’s plans to finalize the IT inventory are provided later in this report.
Finally, since the IT investment review boards were not involved in overseeing IT projects as of June 2002, we concluded that none of the five remaining key practices activities have been executed. These five key practices are the basic activities that investment review boards must implement to effectively oversee IT projects during the control phase. The FBI provided us documentation indicating that the Project Oversight Committee (the primary IT investment review board responsible for overseeing IT projects) met in June 2002 to discuss the FBI’s intent to pilot test the control phase of the Plan by September 2002. The documentation stated that the FBI was still working on designing the specific procedures associated with the control phase, including integrating the ITIM process with the project management office. Additionally, the FBI has only provided us with summary information on when and how the control phase of the ITIM process will be rolled out. The information lacks specific details needed to effectively implement this critical process.
FBI personnel told us that the lack of established IT investment review boards (prior to March 2002) was the main cause for ineffective project oversight. Additionally, they stated that the control phase of the ITIM process would be pilot tested by September 2002. However, the FBI has not been able to provide us with a specific timeline as to: (1) how the pilot test will be executed, and (2) details as to how the ITIM process will interface with a project management methodology. These issues are further discussed in Section B of this finding.
Without effective oversight of IT projects, FBI officials do not have adequate assurance that IT projects are being developed on schedule and within established budgets. As described in the following paragraphs, the lack of effective IT project oversight has contributed to the FBI’s problems in managing IT projects, including a lack of accountability for cost and schedule overruns, a lack of consideration for full life-cycle costs, and lost credibility with Congress.
According to a former Chief Information Officer at the FBI, the lack of effective oversight of IT projects (as a result of not having IT investment review boards and a centralized project management office) have prevented IT project managers from being held accountable for cost and schedule overruns and the ultimate performance of projects. For example, the former Chief Information Officer told us that the CJIS Division completed the Integrated Automated Fingerprint Identification System and the National Crime Information Center 2000 years behind schedule and millions of dollars over budget. He also told us that management changes in the CJIS Division have not occurred, despite these overruns. Senior FBI officials also told us that the Bureau’s budget formulation process focuses only on the acquisition costs for IT projects and not the full life-cycle costs, especially operations and maintenance costs. For example, an assessment performed by the FBI’s Inspection Division on the Trilogy project40 noted that the life- cycle cost estimate is inadequate and only focuses on the term of the contract, not the life of the project. FBI personnel told us that a lack of consideration for full project costs is not limited to Trilogy, but also applies to other IT projects. Without accountability for significant deviations from project baselines, there is a lack of incentives for project managers to adequately control and evaluate projects.
According to FBI officials, the FBI’s inability to effectively complete IT projects within budget and schedule reduced the FBI’s credibility in the eyes of Congress. The lack of credibility contributed to delays in the FBI receiving Congressional funding to upgrade its IT infrastructure. This subject, along with how Trilogy may be adversely affected because of uncertainties in determining projected costs and scheduled completion dates for project milestones, is further discussed in section C of this finding.
We recommend that the Director of the FBI ensure:
(5) Critical Process #3: IT Project and System Identification
For the FBI to make effective IT investment decisions, it must have at its disposal information about existing IT investments as well as the proposed investments being considered. The purpose of this critical process is to provide the IT investment boards the information required to fully evaluate the impacts and opportunities created by both the proposed and current IT investments. The key practices of this process require the FBI to identify and track the IT projects and systems within the organization to create a comprehensive inventory. According to the Framework, effective identification of IT projects and systems requires:
While the FBI has taken steps to identify its IT projects and systems in an IT asset inventory, it still does not have a complete IT asset inventory that is being using by the IT investment review boards for investment management purposes. As part of an enterprise architecture data repository, the FBI is developing a comprehensive inventory of its IT projects and systems. In addition, FBI officials have told us that the enterprise architecture office is primarily responsible for developing and maintaining the data repository. However, the data repository has not been completed, nor have board members used its contents during the select phase of the ITIM process that took place during the Spring of 2002. The FBI’s enterprise architecture function is further discussed in section B of this finding. The following table summarizes the key practice ratings for the IT project and system identification critical process.
FBI Progress Toward Identifying IT Projects and Systems (Critical Process #3)
|Key Practice||Key Practice Execution Status Prior to March 2002||Key Practice Execution Status as of June 2002|
|Organizational Commitment 1. The organization has written policies and procedures for identifying its IT projects and systems and collecting an inventory that includes information about the IT projects and systems that is relevant to the investment management process.||Executed||Executed|
|Organizational Commitment 2. An official is assigned responsibility for managing the IT project and system identification process and ensuring the inventory meets the needs of the investment management process.||Not Executed||Executed|
|Prerequisite 1. Adequate resources are provided for identifying IT projects and systems and collecting relevant information into an inventory.||Not Executed||Not Executed|
|Activity 1. The organization's IT projects and systems are identified and specific information about these projects is collected in an inventory.||Not Executed||Not Executed|
|Activity 2. Changes to IT projects and systems are identified and changed information is collected in the inventory.||Not Executed||Not Executed|
|Activity 3. Information from the inventory is available on demand to decision-makers and other affected parties.||Not Executed||Not Executed|
|Activity 4. The IT project and system inventory and its information records are maintained to contribute to future investment selections and assessments.||Not Executed||Not Executed|
|Source: OIG analyses|
a. The FBI has Executed Two of the Seven Key Practices Associated With Identifying IT Projects and Systems
Based on our analyses, we determined that the FBI has executed two of the seven key practices associated with this critical process. Specifically, the FBI has developed written policies and procedures for identifying its IT projects and systems in an inventory that includes information relevant to the investment management process (Organizational Commitment 1). Additionally, the FBI has designated an official responsible for managing the IT project and system identification process and ensuring that the inventory meets the needs of the investment management process (Organizational Commitment 2).
Regarding Organizational Commitment 1, we determined that the FBI has developed adequate written policies and procedures for: (a) identifying its IT projects and systems and (b) collecting information relevant to the investment management process on each project and system. Prior to December 2001, the FBI did not have written policies and procedures for identifying IT projects and systems. The FBI did, however, provide us with an electronic communication dated December 3, 2001 from the enterprise architecture staff that was distributed Bureau-wide requesting management from each division to provide information on its IT systems. The information obtained from the divisions is used by the enterprise architecture staff to develop the data repository of IT systems.
Regarding Organizational Commitment 2, the FBI has designated the Chief Architect of the enterprise architecture office with responsibility for managing the IT project and system identification process and ensuring that the inventory, when completed, meets the needs of the investment management process and ITIM managers and users. The Chief Architect currently reports to the Information Resource Management Section Chief, who reports to the Chief Information Officer.
b. The FBI Must Execute Five of the Seven Key Practices Associated with Identifying IT Projects and Systems
Although the FBI has made recent progress in identifying IT projects and systems, the FBI does not have a comprehensive IT project and system identification process because it still has not executed five out of the seven key practices associated with this critical process. Specifically, the FBI must ensure that:
Regarding Prerequisite 1, FBI managers told us that the FBI has not allocated adequate resources to ensure timely and successful completion of the IT project and system identification critical process. FBI managers from the Information Resources Management Section told us that they do not have sufficient staffing to support the ITIM process, including the enterprise architecture function. The enterprise architecture office within the Information Resources Management Section plays a key role in the ITIM process as it assists the Technical Review Board and maintains the data repository information on IT systems and projects. Further, personnel who we interviewed from the enterprise architecture office told us that limited staffing was a factor in not having the data repository completed.41
Regarding the remaining four key practices, none of those practices can be executed until the FBI completes the creation of its IT asset inventory. More importantly, the IT asset inventory will have little value to the FBI if it is not used when making IT investment decisions. Prior attempts at compiling an inventory of IT projects were used to satisfy Congressional and DOJ requests, rather than to assist the IT investment management process. For example, the FBI prepared a partial list of its information technology projects to comply with a Congressional request in August 2000.
FBI officials informed us that they anticipate the investment review boards will use the completed inventories to contribute to future investment selections and assessments. The Plan states that the FBI must establish a complete IT portfolio set as the ITIM process matures. Further, FBI personnel told us that the enterprise architecture data repository, when complete, will be available to decision-makers and other ITIM users via the FBI’s Intranet. However, we have not been provided with a specific timeframe for when the FBI expects to have a completed inventory. FBI personnel told us that the primary cause of not having a completed IT asset inventory and actively using it in the ITIM process is because of staffing shortages. While that may be a contributing factor, we concluded that the lack of centralized management over IT investments was also a limiting factor. As a result, certain divisions maintained some version of an IT inventory for the projects and systems under their jurisdiction, and there was no centralized office responsible for maintaining a uniform listing Bureau-wide.
Without a complete IT asset inventory in the ITIM process, FBI management and board members do not have adequate assurance that accurate, timely, and complete information on existing IT projects and systems is available to them. As a result, there is a risk that new IT proposals selected overlap with one of the 200 or so existing FBI applications. While the recently established review boards helped to mitigate this risk for the FY 2004 budget selection process, we believe that an IT asset inventory must be used by the boards to optimize the use of the FBI’s resources.
We recommend that the Director of the FBI:
(6) Critical Process #4: Business Needs Identification
This critical process establishes the mechanism for identifying the business needs and the associated users that drive each IT project. This critical process links the organization’s business objectives with its IT strategy and creates the partnership between the users and the IT providers. According to the Framework, effective identification of business needs requires:
While the FBI has made progress in identifying business needs for IT projects, it has not yet executed all the key practices necessary to implement this critical process. Prior to pilot testing the select phase of its ITIM process in March 2002, the FBI had been identifying users for each IT project in the Exhibit 300.42 Since pilot testing the select phase of the ITIM process beginning in March 2002, the FBI has used a concept paper along with the Exhibit 300 to identify and define business needs. In addition, the FBI has defined its general business needs and goals in its strategic plan, which is further discussed later in this report. However, as previously mentioned, the FBI has not identified all of its IT projects in an asset inventory; consequently, progress in implementing this critical process is contingent upon completing the FBI IT inventory. Also, we were not provided evidence indicating that identified users participate in project management throughout a project's life-cycle. The following table summarizes the key practice ratings for the business needs identification critical process.
FBI Progress Toward Identifying its Business Needs (Critical Process #4)
|Key Practice||Key Practice Execution Status Prior to March 2002||Key Practice Execution Status as of June 2002|
|Organizational Commitment 1. The organization has written policies and procedures for identifying the business needs (and the associated users) of each IT project.||Not Executed||Not Executed|
|Prerequisite 1. Adequate resources are provided for identifying business needs and associated users.||Not Executed||Not Executed|
|Prerequisite 2. The organization has defined business needs or stated mission goals.||Executed||Executed|
|Prerequisite 3. IT staff are trained in business needs identification.||Not Executed||Not Executed|
|Prerequisite 4. All IT projects are identified in the IT asset inventory.||Not Executed||Not Executed|
|Activity 1. The business needs for each IT project are clearly identified and defined.||Not Executed||Executed|
|Activity 2. Specific users are identified for each IT project.||Executed||Executed|
|Activity 3. Identified users participate in project management throughout a project's life-cycle.||Not Executed||Not Executed|
|Source: OIG analyses|
a. The FBI has Executed Three of the Eight Key Practices Required to Identify its Business Needs and Associated Users
We determined that the FBI has executed three of the eight key practices associated with this critical process. Specifically, the FBI has defined its business needs or stated mission goals (Prerequisite 2); the business needs for identified IT projects are clearly identified and defined (Activity 1); and specific users are identified for each IT project (Activity 2).
Regarding Prerequisite 2, we determined that the FBI has defined business needs or stated mission goals. The FBI has stated mission goals in its strategic plan. The FBI’s strategic plan has not been updated since 1998, but the Director has revised the priorities of the Bureau since the terrorist attacks on September 11, 2001. Further, the FBI is currently in the process of developing an enterprise architecture framework, which will link the FBI’s strategic plan to its business needs.
Regarding Activity 1, we determined that the business needs for each IT project are clearly identified and defined in the Exhibit 300. Prior to the initiation of the ITIM pilot test in March 2002, the FBI did not have adequate management controls in place to ensure that the business needs for each project were accurately developed in the Exhibit 300. With the ITIM process, the board reviews of the concept papers and Exhibit 300s provided assurance that these business needs were clearly identified and defined. In instances where the business needs were vague, the boards, especially the Technical Review Board, returned the concept papers and Exhibit 300s to the project sponsor for re-work. This re-work demonstrates that board review of these IT proposals was an effective control over the business needs identification process. Our review of Exhibit 300s that were ultimately recommended to the Executive Review Board for inclusion in the FY 2004 budget cycle confirmed that business needs were clearly identified and defined.
Regarding Activity 2, the FBI identified specific users for each IT project. Based on our reviews of several Exhibit 300s both before and after the initiation of the ITIM process in March 2002, we determined that the users for the IT project were identified and documented.
b. The FBI Must Execute Five of the Eight Key Practices Required to Identify its business Needs and Associated Users
Although progress has been made in identifying its business needs and associated users, the FBI has yet to execute five of the eight key practices associated with this critical process. Specifically, the FBI must ensure that:
Regarding Organizational Commitment 1, we determined that the FBI does not have written policies and procedures for identifying the business needs (and the associated users) of each IT project. The FBI has been defining business needs for IT projects in the Exhibits 300 and related concept papers. The Post-Implementation Review acknowledges that the FBI needs more formally developed policies and procedures to support the ITIM process. By formalizing these procedures in writing, the FBI reduces the risk that it will neglect to perform this practice in the future.
Regarding Prerequisites 1 and 3, FBI officials told us that adequate resources were not allocated to identifying business needs and associated users. Specifically, FBI officials from the Information Resources Management Section told us that there has not been sufficient resources dedicated to the ITIM process, including the training of ITIM users. The importance of training ITIM users in the many facets of the ITIM process cannot be underestimated. Part of the required ITIM training must include the business needs identification process. Examples of training in this critical process include organizational requirements for ongoing education, rotation of ITIM users through supported business units, and relevant conference attendance. As previously mentioned, many ITIM users have only received one training session on the FBI’s ITIM process. Additionally, the FBI has not provided us with specific plans for future training sessions that include business needs identification. As a result, these key practices have not been executed.
The ITIM training that occurred in February 2002 provided only an overview of the ITIM process, rather than role-specific training that addressed the business needs identification. The Post-Implementation Review stated that re-work of Exhibit 300s and concept papers were required after these products were submitted to the ITIM program office. This re-work was necessary because there was not a clear alignment between the IT proposal and the FBI’s strategic goals. Better training that included business needs identification may have reduced some of the re-work. Further, a more clearly defined enterprise architecture framework would have increased the IT staff’s knowledge in business needs identification.
Regarding Prerequisite 4, as previously mentioned, the FBI has not completed its IT asset inventory. Identifying all projects in an IT asset inventory is a fundamental step in having a fully developed business needs identification process. The availability of this inventory assists board members in recommending IT projects that support one or more business needs or mission goals.
Regarding Activity 3, FBI officials have acknowledged that identified users do not consistently participate throughout the project’s life-cycle. FBI officials informed us that not keeping IT system users actively involved in the creation and implementation of IT projects is a major factor in the development of multiple IT systems (including ACS) that do not effectively meet user needs. When we asked the former Chief Information Officer for other examples of systems that do not effectively meet user needs, his response was “pick one.” Clearly, this is a significant need that must be addressed by the ITIM process. The DOJ’s System Development Life-Cycle requires user participation throughout the life-cycle, but as we previously noted in this finding, the System Development Life-Cycle is not used by the FBI on a consistent basis. Board oversight of project teams should be required to ensure that users are engaged throughout the project’s life-cycle.
FBI officials told us that there has not been ample time since the implementation of the Plan to adequately train its IT staff and board members in business needs identification. A complete explanation as to why the FBI did not have ample time for training was previously discussed in section A.3 of this finding.
Although FBI officials have told us that additional training for IT staff and board members is expected to occur sometime in the future, we were not provided evidence that shows there will be any training specifically related to business needs identification. Further, we have not been provided with a timetable as to when this training will take place. In addition, an effective business needs identification process requires an organization to have a comprehensive IT portfolio and enterprise architecture, neither of which the FBI currently has. Our assessment of the FBI’s efforts to implement a basic enterprise architecture is discussed later in this report.
Without a comprehensive business needs identification process, FBI management and board members do not have adequate assurance that they are selecting IT projects that align with mission needs and priorities. Additionally, projects under development are at risk of not meeting the needs of users, as has been the case with ACS and other FBI systems.
We recommend that the Director of the FBI ensures:
(7) Critical Process #5: IT Proposal Selection
The proposal selection critical process establishes a structured methodology for selecting new IT proposals. The FBI should have this critical process fully implemented to ensure that it selects the most meritorious IT proposals to meet its mission critical needs. According to the Framework, this critical process requires:
The following table summarizes the key practice ratings for the proposal selection critical process.
FBI Progress Toward Establishing an IT Proposal Selection Process (Critical Process #5)
|Key Practice||Key Practice Execution Status Prior to March 2002||Key Practice Execution Status as of June 2002|
|Organizational Commitment 1. Executives and managers are committed to follow an established selection process.||Not Executed||Executed|
|Organizational Commitment 2. An official is designated to manage the proposal selection process.||Not Executed||Executed|
|Prerequisite 1. Adequate resources are provided for proposal selection activities.||Not Executed||Not Executed|
|Activity 1. The organization uses a structured process to develop new IT proposals.||Not Executed||Executed|
|Activity 2. Executives analyze and prioritize new IT proposals according to established selection criteria.||Not Executed||Executed|
|Activity 3. Executives make funding decisions for new IT proposals according to an established process.||Not Executed||Executed|
|Source: OIG analyses|
a. The FBI Has Executed Five of the Six Key Practices Associated With Establishing an IT Proposal Selection Process
As previously discussed, the FBI pilot tested its ITIM proposal process in March 2002. The Plan outlined a conceptual framework for selecting projects, while subsequent documents further defined the process. We determined that the FBI has executed five of the six key practices associated with this critical process. The five key practice are:
Regarding Organizational Commitment 1 and Activity 1, we concluded that in pilot testing its proposal selection process in March 2002, FBI managers were committed to and followed an established selection process for the FY 2004 budget cycle.
Prior to the initiation of the ITIM process in March 2002, the FBI did not have an established process for selecting IT proposals. Several FBI officials told us that individual divisions determined their IT needs in a “stovepipe,” without knowledge of the business needs and priorities of the Bureau as a whole. Once each division decided on its IT request, the request was forwarded to the Information Resources Management Section for a “technical” review. This review, performed by the Information Resources Management Section Chief, was designed to ensure that the request was consistent with the FBI’s existing IT infrastructure. However, without an established enterprise architecture, the review could not adequately provide assurance that the proposal aligned with the FBI’s business needs and priorities.
Once approved by the Information Resources Management Section Chief, the request was then forwarded to the Finance Division to determine if similar requests for budget enhancements were previously denied by Congress. Requests approved by the Finance Division were forwarded to a committee comprised of executive managers for final evaluation and selection. However, personnel from the Finance Division told us that it was not uncommon for the IRD, Laboratory, and CJIS Divisions to submit requests for IT projects that were duplicative but were approved anyway. This indicates that the Information Resources Management Section did not adequately perform its role in overseeing IT proposals. Additionally, according to FBI officials, the committee of executive managers did not have a formalized charter, follow approved polices or procedures, or maintain documentation detailing committee activities. Therefore, the process was not standardized or repeatable.
With the initiation of the ITIM process in March 2002, the FBI established a proposal selection process for the FY 2004 budget cycle. IT proposals were developed by the project sponsor with a preliminary feasibility analysis, referred to as a concept paper. The concept paper was submitted to the Enterprise Architecture Technical Committee for a preliminary technical review, and then forwarded to the Technical Review Board with a recommendation as to whether the project should be approved. Upon the Technical Review Board’s approval, the project sponsor was asked to prepare a more comprehensive business case analysis, which was documented in the Exhibit 300. The project proposal package, which includes the concept paper and Exhibit 300, was then submitted to the Project Oversight Committee for a business review. The Project Oversight Committee assembled the multiple requests and recommended a list of projects for the Executive Review Board’s review. The Executive Review Board selected projects for the FY 2004 budget cycle. Because this process was documented in the Plan, and enhanced with training materials, we concluded that the FBI effectively established a selection process. The following flowchart outlines the FBI's proposal selection process.
FLOWCHART OF FBI’S ITIM SELECT PHASE
Source: FBI’s training materials for the ITIM process as of February 2002.
Regarding Organizational Commitment 2, prior to the initiation of the select phase of its ITIM process in March 2002, the FBI did not have a clearly designated official to manage the proposal selection process. According to Information Resources Management Section personnel, the Finance Division managed the IT selection process. However, according to Finance Division personnel, the Information Resources Management office was responsible for managing the proposal selection process. With the onset of the ITIM process in March 2002, the FBI’s Chief Information Officer appointed the ITIM Program Manager to manage the proposal selection process. This official reports to the Information Resources Management Section Chief, who reports to the Chief Information Officer.
Regarding Activity 2, we determined that FBI IT investment board members analyzed and prioritized new IT proposals according to established selection criteria for the FY 2004 budget cycle. Projects were prioritized according to three separate areas: (1) mission fit; (2) technical criteria (including risk management and architectural assessments); and (3) financial criteria (including performance measures, cost/benefit analyses, and acquisition strategy).
Regarding Activity 3, the three IT investment review boards made funding decisions for new IT proposals according to a process established for the FY 2004 budget cycle. The Executive Review Board, chaired by the Director, had the final authority for making IT funding requests to the DOJ. The Executive Review Board members based their decisions upon recommendations made by the Technical Review Board and the Project Oversight Committee. Based on the use of an established process, this key practice has been executed.
b. The FBI Must Execute One Key Practice Associated With Establishing an IT Proposal Selection Process
Although the FBI has made substantial progress in establishing an IT proposal selection process for the FY 2004 budget cycle, in our judgment it has yet to allocate adequate resources for comprehensive proposal selection activities. Our conclusion is based upon the following observations.
Without a comprehensive proposal selection process that includes adequate resources and training, the FBI cannot ensure that it is selecting the best IT projects that meet mission-critical needs.
(8) Overriding Cause for the Lack of an FBI IT Investment Management Foundation
Although the GAO ITIM Framework was originally published in May 2000, the underlying key practices needed to implement each critical process are, in essence, tasks that are fundamental to any project management endeavor. Some of these tasks include the prerequisite conditions that must be in place in an organization to successfully implement critical processes. These tasks involve allocating resources, establishing organizational structures, and providing training. Another group of tasks include the organizational commitments that ensure critical processes will endure. These tasks involve establishing organizational policies and engaging senior management sponsorship. A third group of tasks include the activities necessary to implement the critical processes. These tasks involve establishing procedures, performing and tracking the work, and taking corrective actions as necessary.
Although these tasks are fundamental to effective project management, the majority of these tasks had not been executed by the FBI to select and manage its IT resources. Prior to the development of its ITIM process in early 2002, the FBI did not give sufficient attention to IT investment management. Organizational policies were not clearly established to ensure that critical IT investment policies endure. Additionally, there were no clearly defined, uniform procedures for project management, tracking project performance, and taking corrective actions as necessary.
Because the FBI did not fully implement any of the critical processes associated with Stage Two, the FBI continues to spend hundreds of millions of dollars on IT projects without having adequate selection and project management controls in place to ensure that IT projects will deliver their intended benefits. However, the FBI has made progress in improving its IT investment process since it initiated a new ITIM process in early in 2002. Although further action is required, the launching of the ITIM process represents improvement in the FBI’s ability to mitigate the risks that IT projects will not deliver their intended benefits. Whether the FBI can achieve further improvement depends on whether the Plan addresses the remaining key practices not being executed as well as the FBI’s ability to completely implement the Plan and fully establish its ITIM process.
B. The FBI’s Ability to Improve its IT Investment Practices
As previously noted, the FBI lacks a foundation necessary to build its IT investment capabilities, and therefore, is in Stage One maturity. However, in January 2002, the FBI developed an ITIM plan to build a foundation for selecting, controlling, and evaluating IT investments. Additionally, during the course of our audit fieldwork (from January 2002 to June 2002), the FBI initiated its ITIM process, as defined by the Plan. Consequently, the FBI made progress towards implementing the Plan, especially in the area of IT proposal selection.
Because the FBI was only in the beginning stages of implementing the Plan during our audit fieldwork, we assessed the FBI’s ability to progress through the more advanced stages of the framework necessary to improve its IT investment maturity. Our assessment of the FBI’s ability to improve its IT investment management consisted of the following four areas:
Our evaluation of these four areas, documented in the following sections, includes both the FBI’s strengths and weaknesses in each area. In our judgment, the FBI’s efforts in these areas are critical to its ability to maximize the effectiveness of its ITIM process, and ultimately improve mission performance.
(1) The Plan’s Coverage of Stage Two Key Practice Activities That Were Not Being Executed During Our Fieldwork
The FBI’s IT Investment Management Model and Transition Plan addresses the select, control, and evaluate key practice activities necessary to build an IT investment foundation. However, the Plan requires further development to ensure effective implementation. Because the Plan was intended to be a conceptual framework, it was not written to fully describe the specific policies and procedures of the select, control, and evaluate phases of the ITIM process. Without further development of the ITIM process, the FBI will have difficulty making additional progress in improving its IT investment management practices, especially in the control and evaluate phases.
a. Importance of the Plan’s Coverage of Stage Two Key Practice Activities
Because the Plan stated that its purpose is to establish and define the FBI’s Stage Two methodology necessary to build an IT investment foundation, we examined the Plan’s coverage of Stage Two key practice activities. The FBI was pilot testing the select phase of the ITIM process during our audit fieldwork. As previously noted, we determined that the FBI executed 14 of 38 Stage Two key practices, mainly in the area of proposal selection. Of the 24 key practices that were not executed, 11 specifically related to activities associated with the control and evaluate phases of the ITIM process. Although the FBI had made little progress in executing activities from the control and evaluate phases of the Plan during our fieldwork, we examined the Plan to determine whether it adequately addressed the 11 Stage Two key practices activities associated with the control and evaluate phases that were not being executed. The ability of the FBI to achieve Stage Two maturity is dependent, in part, on the adequacy of the Plan.
In JMD’s assessment of the Plan, JMD rated the Plan against elements it considered necessary to comply with GAO, OMB, and DOJ guidelines. JMD’s assessment indicated that the Plan complied with the criteria used.44 Additionally, JMD’s assessment stated that although the Plan does not fully address a few items, such as the exact criteria that will be used to select and evaluate investments, it does provide a schedule for completing these items.
Our assessment of the Plan focused on whether it addressed the Stage Two maturity key practices in the GAO ITIM Framework and our conclusions are consistent with those from JMD.
b. Results of Our Assessment of the Plan’s Coverage of Stage Two Key Practice Activities Associated with the Control and Evaluate Phases
In our judgment,the FBI’s IT Investment Management Model and Transition Plan addresses the 11 Stage Two key practice activities, on a conceptual level, that were not being executed during our fieldwork. Because the key practice activities are addressed conceptually, further development is needed to clearly define these activities and to determine how these activities can be implemented.
Our analyses (previously documented in this report) indicated that the FBI was not executing one or more key practice activities in each of the following Stage Two critical processes: (1) IT investment board operation; (2) IT project oversight; (3) IT project and system identification; and (4) business needs identification. As previously discussed, 11 of the key practice activities necessary to implement these four critical processes relate to the control and evaluate phases of the Plan. The tables below describe how the Plan addresses the key practice activities that we determined were not being executed during our audit testing.
|IT Investment Board Critical Process|
|Key Practice Activity Not Executed||How the Plan Addresses the Activity|
|Activity 2: Each IT investment board operates according to written policies and procedures in the organization-specific IT investment process guide.||While the Plan does not provide the specific written policies and procedures that the investment boards must follow, it does indicate that further development of these policies and procedures are necessary. Additionally, the Post-Implementation Review of the select phase of the ITIM pilot test recommends that additional policies and procedures be developed in a document that is independent of the Plan. Once the FBI’s ITIM policies are completely developed, this key practice can be executed when the FBI rolls-out the control and evaluate phases of the ITIM process.|
|Source: OIG analyses|
|IT Project Oversight Critical Process|
|Key Practice Activity Not Executed||How the Plan Addresses the Activity|
|Activity 1: Each project's up-to-date cost and schedule data are provided to the appropriate IT investment board.||The Plan stipulates that the functioning project management office will review status reports on cost, schedule, and performance measures. The project management office will then forward selected reports to the boards for review.|
|Activity 2: Using established criteria, the IT investment board oversees each IT project's performance regularly by comparing actual cost and schedule data to expectations.||The Plan states that the Project Oversight Committee will ensure that selected projects are meeting performance measurement objectives, risks are being appropriately managed, budgets and schedules are on track, and resource levels are adequate.|
|Activity 3: The IT investment board performs special reviews of projects that have not met predetermined performance standards.||According to the Plan, the Project Oversight Committee will perform special reviews of projects whose status reports are not meeting predetermined performance standards.|
|Activity 4: Appropriate corrective actions for each under-performing project are defined, documented, and agreed to by the IT investment board and the project manager.||The Plan states that the Project Oversight Committee will review a portfolio status report to determine if quick corrective actions can be executed to get under-performing projects back on track. When this is not possible, appropriate recommendations will be made to the Executive Review Board.|
|Activity 5: Corrective actions are implemented and tracked until the desired outcome is achieved.||The Plan gives the Project Oversight Committee the responsibility to ensure that corrective actions are implemented.|
|Source: OIG analyses|
|IT Project and System Identification Critical Process|
|Key Practice Activity Not Executed||How the Plan Addresses the Activity|
|Activity 1: The organization's IT projects and systems are identified and specific information about these projects and systems is collected in an inventory.||The Plan states that an IT investment portfolio will be built for development projects as the ITIM process is being pilot tested. An IT portfolio is expected to be completed for the full-blown ITIM roll-out during the FY 2005 budget cycle.|
|Activity 2: Changes to IT projects and systems are identified and change information is collected in the inventory.||FBI personnel told us that while there is not a written procedure to document changes to IT projects and systems, a policy will be developed when the IT asset inventory is complete. The IT asset inventory will then be updated as changes are made to IT projects and systems.|
|Activity 3: Information from the inventory is available on demand to decision-makers and other affected parties.||FBI personnel stated that the IT asset inventory, when complete, will be maintained on the FBI’s Intranet, so that relevant information will be available on demand to decision-makers and other affected parties.|
|Activity 4: The IT project and system inventory and its information records are maintained to contribute to future investment selections and assessments.||FBI personnel stated that the IT asset inventory and IT portfolio, when complete, will be updated continually to become an archive of information to be used for future investment selections and evaluations.|
|Source: OIG analyses|
|Business Needs Identification Critical Process|
|Key Practice Activity Not Executed||How the Plan Addresses the Activity|
|Activity 3: Identified users participate in project management throughout a project's life-cycle.||The Plan states that it is crucial for project team members (which must include identified users of the project) to work closely together throughout the project’s life-cycle. These project teams support the functional project management office and Project Oversight Committee.|
|Source: OIG analyses|
With the pilot testing of the select phase, the FBI further developed and refined the proposal selection process and provided training on proposal selection to ITIM users. The training materials supplemented and supported the documentation in the Plan to more clearly define the roles of ITIM users, such as IT investment review board members, project sponsors, and ITIM liaison representatives.
Even with these additional materials, the Post-Implementation Review of the select phase of the Plan (performed by the ITIM contractor) recommended that the FBI significantly expand its documentation of polices and procedures relating to the ITIM process by:
The FBI recognized that the Plan was never intended to represent its final policies and procedures for its ITIM process. The Plan states that it provides a conceptual framework for achieving Stage Two maturity, and will evolve as the FBI’s ITIM process advances to higher levels of maturity.
Without further development and refinement of the ITIM process, the FBI will have difficulty making additional progress in improving its IT investment management practices. Because the goal of Stage Two maturity is to build standardized methodologies for selecting and controlling IT investments, the FBI must have adequate documentation of these methodologies to make them repeatable and institutionalized. The Post-Implementation Review, prepared by the ITIM contractor, acknowledged the necessity for further developing and refining the Plan. In our judgment, the FBI must implement the recommendations set forth in the Post-Implementation Review prior to taking further action in pilot testing the control and evaluate phases of the ITIM process.
(2) The Amount of Participation from ITIM Users in Developing the ITIM Process
In our judgment, the Plan was written with minimal input and coordination from relevant ITIM users. The main reason cited by IRD officials46 for the limited participation from ITIM users was insufficient time allotted to develop the Plan. As a result, the institutionalization and buy-in47 of the ITIM process may have been hampered.
a. Importance of ITIM User Participation in Developing the ITIM Process
Good management practices dictate that organizations involve relevant stakeholders when attempting to implement a new management process. This involvement aids in the institutionalization of the process. Institutionalization of the ITIM process is a key goal of the Plan, which states: “[The ITIM] process applies to ALL information technology projects, from ALL business units, from ALL funding sources, whether they be new, in development or operational.”
Because of the broad applicability of the ITIM process, in our judgment the FBI should have involved representatives from throughout the Bureau when developing the Plan. In particular, individuals from the three divisions that manage major IT projects (the IRD, CJIS, and Laboratory Divisions) should have had substantial input into the creation of the Plan. Further, the Inspection Division’s Major Project Management Oversight Unit (MPMOU) has a responsibility to oversee major projects in the Bureau, including IT projects, and thus should also have been involved in creating the Plan.
b. Results of Our Assessment of ITIM User Participation in Developing the ITIM Process
We found that relevant ITIM users from the IRD, CJIS Division, Laboratory Division, and Inspection Division were not given significant input into how the Plan was developed. Our interviews with IRD personnel indicated that the FBI gave the ITIM contractor the primary responsibility to write the Plan, without requiring significant participation from ITIM users in developing the initial draft of the Plan. Additionally, we determined that while the contractor interviewed numerous individuals from the IRD, it only interviewed two people from the Inspection Division, one person from the CJIS Division, and none from the Laboratory Division.48 Further, as we discuss below, the enterprise architecture office (part of the IRD until February 2002) was not given adequate input into the development of the ITIM process. Also, the interviews that did occur outside of IRD mainly focused on the individuals’ current responsibilities for managing IT investments, rather than their insights into how the new ITIM process could be shaped to best meet the needs of the Bureau. The following paragraphs provide the perspectives of ITIM users from the IRD, CJIS Division, Laboratory Division, and the Inspection Division.
Personnel from the enterprise architecture office told us that because the FBI’s ITIM process had been developing concurrently with the enterprise architecture function, there should have been more coordination between the ITIM contractor and enterprise architecture office to increase effectiveness and reduce duplication of effort. For example, the enterprise architecture office drafted charters for a three-tiered IT investment review board structure, similar to what was ultimately written by the ITIM contractor. Additionally, the enterprise architecture office was preparing initiatives to improve the FBI’s IT investment management practices. While the enterprise architecture office was drafting board charters and other processes designed to improve the FBI’s IT investment management practices, the ITIM contractor, supervised by the ITIM Program Office, wrote the Plan without incorporating the work already accomplished by the enterprise architecture office.
Additionally, an individual from the enterprise architecture office told us that although he believed the ITIM process represents a positive step for the FBI, it must incorporate more involvement from the enterprise architecture function to ensure success of the process. He further stated that the IT investment review boards must rely more on the vast knowledge, expertise, and talents of FBI IT personnel prior to making decisions.
Further, according to a manager in the Information Resource Management Section, the Enterprise Architecture Technical Committee, which supports the Technical Review Board, has not been given the responsibility to ensure that IT proposals align with the mission of the FBI. The responsibilities of the Technical Review Board, as defined in the Plan, are focused on reviewing the technical risks of IT projects. These technical risks include compliance with the “technical architecture” or configuration management of the FBI, rather than the business architecture which shows how the business processes work together to satisfy the mission. The Plan and board charters assigned this responsibility to the Project Oversight Committee. In our judgment, because the responsibilities of the enterprise architecture office comprise both the technical and business architecture, the Enterprise Architecture Technical Committee should not only be responsible for assessing compliance with the technical architecture, but should also be responsible for assessing compliance with the business architecture. This added responsibility would provide greater assurance to FBI executives that IT proposals selected will enhance the Bureau’s capability in achieving its mission.
An official from the CJIS Division told us that he was interviewed by representatives from the ITIM contractor on one occasion to determine what role the CJIS Division had in managing IT projects. However, he was not consulted on how the FBI’s ITIM process should be created. He stated the only opportunity he had to comment on the Plan was after it was written in January 2002. His belief was that the ITIM Program Office was relying solely on the contractor to write the Plan, rather than building a Plan that has the input and buy-in from all FBI divisions.
While this official from the CJIS Division said to us that the Plan was an improvement over the FBI’s current process for managing IT investments, he was not convinced that the process could be effectively implemented without addressing other pressing issues, such as the need for: (1) standardized methodologies in configuration management, quality assurance, and IT security; (2) improved support of contractors that work on IT systems; and (3) more representation of individuals with IT technical expertise on the IT investment review boards.
An official from the Laboratory Division’s project management office told us that he first became aware of the Plan when training was announced for the new ITIM process in February 2002. Another official from the Laboratory Division told us that to his knowledge, no one from the Laboratory Division was consulted by the ITIM contractor prior to the preparation of the Plan. He told us that the Laboratory Division’s current process was working fine and not in need of change.
Additionally, Inspection Division personnel, including individuals from the MPMOU, told us (as of June 2002) they were only consulted by the ITIM contractor as to how they acquired IT, not for their project oversight role.
An official from the Information Resources Management Section cited the insufficient amount of time allotted to prepare the Plan as the main cause for the limited involvement from ITIM users. As we previously mentioned, the FBI waited until December 2001 to engage the ITIM contractor to develop the Plan, despite learning of the DOJ’s requirements to prepare a plan in January 2001. The ITIM Program Office Manager stated that the former Chief Financial Officer did not initially approve the use of an outside contractor to develop the Plan, causing a delay in hiring the contractor. The former Chief Financial Officer confirmed to us that there were initial concerns in using an outside contractor to develop a management process that affects how the IT budget is allocated and spent. Because the DOJ required initiation of the ITIM process during the FY 2004 budget cycle (which for the FBI begins in March), there was limited time between the development of the Plan (December 2001) and the initiation of the ITIM process (March 2002). In fact, the FBI only gave the contractor approximately two weeks to write the Plan because of the impending deadline to submit the Plan to JMD. As a result, FBI personnel told us that the ITIM contractor did not have ample time to include more ITIM users in the Plan’s development.
While FBI officials from the Information Resources Management Section acknowledged the ITIM contractor’s time constraints in developing the Plan, they also stated that the Plan is only a draft, and will be modified as the ITIM process is pilot tested. Additionally, because the three IT investment review boards established by the ITIM process include representatives from the major divisions that manage IT projects, officials from the Information Resources Management Section told us that there is significant opportunity for input into refining the ITIM process as it is being pilot tested.
Despite the Information Resource Management Section’s position that the pilot test provides ample opportunity for input into refining the ITIM process, in our judgment, the ITIM Program Office, along with the ITIM contractor, continues to develop the ITIM process without incorporating sufficient input from relevant stakeholders. For example, a manager from the enterprise architecture office stated to us in July 2002 that the ITIM Program Office had not requested his participation during development of the control phase of the ITIM process. This individual told us the enterprise architecture function should have a role in enhancing the control and evaluate phases of the ITIM process, but has not had the opportunity to demonstrate this role. Additionally, the process for the development of the control phase has not substantially changed from the select phase: the ITIM contractor, supervised by the ITIM Program Office, writes the policies and procedures which are then pilot tested by the ITIM users. In our judgment, this approach is not conducive to a process whose success depends on institutionalization and buy-in from ITIM users.
In our judgment, the lack of involvement by relevant ITIM users inhibits management buy-in to the ITIM process. If there had been more participation in the development of the Plan, some of the concerns stated above by key ITIM users might have been mitigated. The FBI must address these concerns to facilitate the institutionalization and buy-in the of the ITIM process, and ultimately improve its effectiveness.
(3) The Project Management Function’s Support of the ITIM Process
The FBI’s project management function needs improvement to adequately support the ITIM process, especially in the control and evaluate phases of the process. The FBI recognizes the importance of upgrading the project management function. In particular, the Plan states that the project management office must fulfill a critical role in supporting the Project Oversight Committee. In addition to the Plan, the FBI has taken other steps towards improving its project management function. Specifically, in June 2002, the FBI announced plans to create an Office of Programs Management. The Office of Programs Management will serve as a centralized project management office49 that FBI officials from this office and the Information Resources Management section expect to play a key role in implementing the ITIM process. Despite the progress being made, the FBI still has critical areas to address, such as integrating a project management methodology with its ITIM process.
a. Relationship Between Project Management and ITIM Numerous legislative mandates, including the Results Act and the Clinger-Cohen Act, require federal agencies to establish and maintain processes for managing systems throughout their life-cycle. These legislative mandates indicate that basic project management practices are essential if an organization is to ensure that its IT projects have established cost, schedule, and technical performance baselines that are monitored throughout the project’s life-cycle. Additionally, project management is fundamental to supporting an ITIM process. In particular, the control phase of an ITIM process requires an organization to have a project management function. For example, IT project oversight, which encompasses basic project management practices, must be implemented for an organization to achieve Stage Two maturity. However, the Framework does not by itself provide a comprehensive model for how an organization should develop its project management function.
According to the Framework, an ITIM process is not a substitute for good project management. While an ITIM process takes an enterprise-wide focus, good project-level management forms the foundation for successful IT investments.
In our judgment, for the FBI’s project management function to effectively support its ITIM process, the Bureau must have: (1) a fully operational centralized project management office whose responsibilities are directly integrated with the ITIM process, and (2) a standardized project management methodology that is integrated with the ITIM process. Because of the importance of these efforts, we assessed the FBI’s progress in integrating these areas with its ITIM process.
b. Importance of a Centralized Project Management Office
The Plan recommends that project teams be staffed from a “pool” of managers and developers maintained in the project management office. These project teams would not be dedicated to solely one division, function, or application; instead, these teams would work on all types of IT projects across the Bureau. According to the Plan, this approach has many benefits, including:
We concur with the Plan’s recommendations. Although the Plan does not specifically state that the project management office should be centralized (independent of any division), in our judgment, such a structure is most conducive to attaining the benefits listed above.
In addition to the above benefits, a centralized project management office can ensure that IT project teams are following a standardized project management methodology that is integrated with the ITIM process. In our judgment, this added control is especially important to the FBI since we previously concluded that the FBI’s three main divisions that manage IT projects (the IRD, CJIS, and Laboratory Divisions) have not been consistently using a standardized project management methodology.
c. Importance of a Standardized Project Management Methodology
The DOJ recognized the importance of integrating project management with the ITIM process. In January 2001, it issued DOJ Order 2880.3 to require components to manage IT investments in a way that demonstrates good stewardship, complies with applicable laws, and accomplishes the agency’s diverse mission. Among its policies, the Order required each DOJ component to establish an ITIM process that is integrated with a structured system development life-cycle methodology. While the FBI is mandated to use the DOJ’s System Development Life-Cycle methodology, we previously stated in this report that it has not been used consistently.
d. Results of Our Assessment of the FBI’s Progress in Integrating its ITIM Process with the Responsibilities of a Centralized Project Management Office
As discussed below, we concluded that the FBI has recently made progress in integrating its ITIM process with the responsibilities of a centralized project management office. Not only does the FBI recognize the importance of this integration, but it has taken major steps towards incorporating the ITIM process with the responsibilities of a centralized project management office. This progress was evidenced by: (1) how the Plan defined the role of the project management function, and (2) the FBI’s recent efforts to establish a centralized project management office.
The Plan recommends centralization of IT investment management through the use of IT investment review boards that have Bureau-wide oversight. Of the FBI’s three IT investment review boards, the Project Oversight Committee has the primary responsibility for controlling IT projects. Additionally, the Plan calls for a project management office, a subcommittee of the Project Oversight Committee, to have discretion in managing IT projects Bureau-wide.
Specifically, the Plan defines how the primary responsibilities of the project management office must be integrated with the activities of the ITIM process, particularly during the control and evaluate phases. These responsibilities include:
We were told in June 2002 that the Director of the FBI approved the creation of a centralized project management office, whose chief executive would report to the Director.50 This project management office, which would be independent of all other FBI divisions, would have the primary responsibility of managing projects in the Bureau. These projects would include, but not be limited to, information technology. The proposed mission for this new office is: “To assist the FBI in effectively managing, implementing, and deploying high-priority, complex and high risk development projects of high dollar value to successfully support the FBI’s operational mission.” To achieve this mission, this office will be:
In addition, the Office of Programs Management has the following core functions for which it will ultimately be responsible: (1) system engineering, (2) schedule, (3) budget, (4) risks, (5) contract management, (6) certification and accreditation of IT systems, (7) configuration management, and (8) quality assurance.
In our judgment, the creation of the Office of Programs Management represents a critical first step towards centralizing the project management function and improving its effectiveness. Additionally, officials from the Information Resources Management Section and the Office of Programs Management have told us that they are working together to facilitate the integration of the responsibilities of the eight core functions listed above. The ITIM process needs the full support of the Office of Programs Management to implement the control and evaluate phases of the Plan. Therefore, in our judgment, the FBI should continue its efforts to integrate the responsibilities of the Office of Programs Management with the ITIM process. Specifically, a plan should be developed that outlines activities that must be performed to complete the integration, along with reasonable suspense dates. Additionally, this plan should provide the criteria and thresholds that the Office of Programs Management will use to select IT projects for review.
e. Results of Our Assessment of the FBI’s Progress in Integrating its ITIM Process with a Standardized Project Management Methodology
We concluded that the FBI has not taken the necessary actions to integrate the ITIM process with a standardized project management methodology. While officials from the Information Resources Management Section have acknowledged to us that the ITIM process needs to be integrated with a standardized project management methodology, they have not taken sufficient action to ensure that these processes are integrated in a timely manner. This conclusion is evidenced by the Information Resources Management Section’s lack of coordination with the Inspection Division’s Major Project Management Oversight Unit (MPMOU), as previously reported in this section. Additionally, as discussed in the following paragraphs, the FBI risks duplicating efforts in managing IT projects if it implements the control and evaluate phases of the ITIM process without integrating these phases first with a standardized project management methodology.
To improve the FBI’s ability to manage projects, including IT projects, the prior FBI Director requested that the MPMOU establish a standardized project management methodology for Bureau-wide use. In October 2001, the MPMOU completed the Project Management Process and submitted it to executive management for approval. The Project Management Process, which incorporates the DOJ’s System Development Life-Cycle methodology, provides a framework that encompasses all phases of a project’s life-cycle, including planning, developing, support, and disposal.
Personnel from the MPMOU stated to us that the Project Management Process provides a mechanism to fulfill certain requirements of the ITIM process. Specifically, personnel from the MPMOU told us that the project management process facilitates the ITIM process by:
According to MPMOU personnel, given their knowledge of the FBI’s requirement to develop an ITIM process, they made repeated attempts beginning in 2001 to work with individuals from the Information Resources Management Section to develop these processes concurrently.
In November 2001, personnel from the MPMOU prepared a presentation entitled “Project Management Process Compatibility with the ITIM Process” to show appropriate individuals from the IRD the similarities between the two processes. However, according to MPMOU personnel, individuals from the IRD who were managing the development of the ITIM process never gave MPMOU the opportunity to make their presentation. In April 2002, after the development and initiation of the ITIM process, the MPMOU sent an electronic communication to the Director’s office explaining the need to integrate these processes. The electronic communication stated that integration of these processes would improve efficiencies, streamline reporting and paperwork requirements, and improve the FBI’s compliance with applicable regulations, including DOJ Order 2880.3. As of June 2002, no additional action had been taken by the Information Resources Management Section to integrate these processes.
Despite the efforts by the MPMOU to integrate the two processes, the Information Resources Management Section (with the support of the ITIM contractor) developed and began implementation of the FBI’s IT Investment Model and Transition Plan without attempting to integrate it with the Project Management Process. Until the FBI integrates these two processes, the FBI will not be in compliance with DOJ Order 2880.3. Additionally, the FBI will be unable to effectively implement the control phase and evaluate phases of the ITIM process. Further, the FBI risks inefficient use of resources as a result of the duplication of efforts that could occur if the FBI fails to integrate these processes. FBI officials from the Information Resources Management Section have acknowledged to us that they must integrate the control and evaluate phases of the ITIM process with a standardized project management methodology. Despite their recognition of this need, as of June 2002 they did not have the details of how or when this will occur.
Although the FBI has taken a critical first step in (1) centralizing its project management structure, and (2) incorporating the responsibilities of the Office of Programs Management with the ITIM process, the FBI must take further action in integrating its ITIM process with a standardized project management methodology. Without this further action, the FBI’s project management function will not adequately support the ITIM process. Consequently, the FBI risks ineffective execution of its control and evaluate phases as well as inefficient use of resources in managing its IT investments.
(4) The Enterprise Architecture Function’s Support of the ITIM Process
The FBI’s enterprise architecture function needs improvement to adequately support the ITIM process. The FBI has taken a critical first step in establishing an enterprise architecture framework with a limited amount of time and resources dedicated to this effort. Despite the progress being made, the lack of a fully developed enterprise architecture framework will hamper the FBI’s ability to advance through the ITIM maturity framework.
a. Importance of Having Support from the Enterprise Architecture Function
Enterprise architecture is the organization-wide blueprint that defines an entity’s functions and systems, including IT systems. It provides a comprehensive view (through models, narratives, and diagrams) of the interrelationships of an organization’s operations and structures and how these structures align with the organization’s mission. The Clinger-Cohen Act of 1996 recognizes the interrelationship between enterprise architecture and IT investment management by requiring federal agencies to develop an enterprise architecture.
In a review of enterprise architecture use in the federal government, the GAO stated in its February 2002 report:51
According to the Framework, achieving IT investment maturity depends not only on implementing the ITIM critical processes, but also on other good management attributes such as the effective use of human capital, training, enterprise architecture, and software management. Specifically, an established enterprise architecture supports the ITIM process by facilitating an organization’s advancement through the maturity stages of the Framework.
Achieving Stage Two maturity requires an organization to, among other things: (1) identify its IT projects and systems; (2) identify its business and user needs; and (3) select IT projects that align with those business and user needs. An organization’s enterprise architecture would assist in the implementation of this critical processes by identifying the needs between the entity’s current IT systems and processes and its target or future IT system environment.
Achieving Stage Three maturity52 is dependent on a functioning enterprise architecture framework. The Plan states that to advance to Stage Three maturity, the FBI will a need a formal enterprise architecture committee to assess the IT portfolio for enterprise architecture compliance.
Achieving Stage Four maturity requires further integration of the enterprise architecture function with the ITIM process.53 The Plan states that the FBI will have to completely integrate its enterprise architecture framework to enhance the management of its IT portfolio.
To respond to the importance of developing and overseeing enterprise architecture management in the Federal government, the GAO developed a maturity framework for enterprise architecture management that can be used in determining agencies’ development, implementation, and maintenance of these architectures. The maturity framework, developed in 2001, is based on the core elements necessary for an organization to achieve effective enterprise architecture management. These core elements are arranged into a series of five hierarchical stages based on the implicit dependencies among these elements. This framework is consistent with other maturity frameworks, including the ITIM framework. The framework’s five stages of enterprise architecture management maturity are described in Appendix 5 of this report.
To assess the status of federal agencies’ efforts to develop, implement, and maintain enterprise architectures, the GAO surveyed 116 agencies (including the FBI) in 2001 using a questionnaire that was based on the core elements of the enterprise architecture maturity framework. The GAO published the results of this survey in its February 2002 report on enterprise architecture (“INFORMATION TECHNOLOGY: Enterprise Architecture Use Across the Federal Government Can Be Improved”). The GAO indicated in the report that of the 116 agencies surveyed, 98 reported meeting the minimum criteria necessary for Stages One or Two — creating enterprise architecture awareness or building an enterprise architecture management foundation. In contrast, only five agencies reported satisfying the practices that GAO stated are needed to effectively manage enterprise architecture activities (Stages Four or Five).
The results of the GAO survey, completed by the FBI in July 2001, indicated that the FBI is in Stage One of the enterprise architecture maturity framework.54 According to the GAO, Stage One maturity is characterized by either no plans to develop and use an enterprise architecture, or plans and actions that do not yet demonstrate an awareness of the value of having and using one. While stage one agencies may have initiated some enterprise architecture core elements, these agencies’ efforts are inconsistent and unstructured, and do not provide the management foundation necessary for successful enterprise architecture development.
Specifically, the GAO reported that the FBI needed to fully establish the management foundation that is necessary to begin developing, implementing, and maintaining an enterprise architecture. While the FBI implemented most of the core elements associated with establishing the management foundation, it had not yet established a steering committee or group that has responsibility for directing and overseeing the development of the architecture.
In addition, the GAO indicated that although establishing the management foundation is an essential first step, important further steps still need to be taken for the FBI to fully implement the set of practices associated with effective enterprise architecture management. These include having a written and approved policy for developing and maintaining the enterprise architecture and requiring that IT investments comply with the architecture.
We determined that the FBI’s enterprise architecture function does not adequately support its ITIM process. Although the enterprise architecture office has provided support to the ITIM process during the pilot test of the select phase, this support needs to be enhanced. Our conclusion is based on the FBI not having a fully established enterprise architecture.
b. Results of Our Assessment of the FBI’s Progress Towards Fully Establishing an Enterprise Architecture
We concluded that although the FBI has not fully established an enterprise architecture, it is taking important steps to establish one. Specifically, personnel from the enterprise architecture office told us that a baseline architecture is being developed in a data repository, which will ultimately be maintained on the FBI’s Intranet. This data repository, when complete, will describe how all of the FBI’s IT systems align with the business processes of the Bureau. Additionally, the enterprise architecture office is developing a technical reference model that will outline the technical architecture of the Bureau’s IT systems. Also, this office is creating a commercial off-the-shelf roadmap of all commercially available hardware and software that will comply with the FBI’s technical architecture.
Despite the limited staffing of the enterprise architecture office, this office has made progress towards building a foundation for an enterprise architecture function.55 Given the importance of enterprise architecture to ensure successful IT investment management, coupled with the size and complexity of the FBI’s IT infrastructure, we concluded that additional staffing and management attention to this area is warranted.
Despite the progress of the enterprise architecture office, not having a fully established enterprise architecture framework hampers the ITIM process. As we previously mentioned, the ITIM process depends on enterprise architecture functions to fulfill critical processes in the Framework. An organization’s enterprise architecture would assist in the implementation of each of these critical processes, none of which the FBI has implemented as of June 2002. The following paragraph describes several causes for the FBI not having a fully developed enterprise architecture framework that adequately supports the ITIM process.
Personnel from the FBI’s enterprise architecture office told us that the FBI has only recently paid significant attention to developing an enterprise architecture. According to the GAO, the FBI’s lack of attention to enterprise architecture is not much different from other federal agencies. Historically, agency executives have not fully understood the value of enterprise architectures. Therefore, these tools have lacked the executive sponsorship necessary to become a funding priority. In addition, human capital expertise in this area has been scarce at federal agencies. As a result, the risk is heightened that federal agencies will proceed with investment decisions without the benefit of this architectural context and will end up with systems that limit mission performance, often after a significant and unwise use of funds. Specifically, the GAO stated in its June 2002 testimony: “The successful development and implementation of an enterprise architecture, an essential ingredient of an IT transformation effort for any organization and even more important for an organization as complex as the FBI, will require, among other things, sustained commitment by top management, adequate resources, and time.”
Because the FBI does not have a fully developed enterprise architecture, the FBI will have difficulty in achieving more mature IT investment processes such as managing its IT investments as a complete portfolio and improving the investment process through post-implementation reviews.
(5) Summary of the FBI’s Ability to Improve its IT Investment Practices
We determined that the FBI must take additional actions to improve its IT investment practices. Not only will these actions facilitate the building of an IT investment foundation (Stage Two maturity), but these actions will also be essential for any advancement beyond Stage Two. In summary, the FBI must:
The FBI’s efforts in these areas are crucial for it to successfully improve its IT investment maturity, and ultimately enhance mission performance.
C. Trilogy Case Study
To determine how the FBI’s IT investment management practices affected a major IT project, we performed a case study of the FBI’s Trilogy project. In section A of this finding, we concluded that the FBI was not fully implementing any of the critical processes necessary for successful IT investment management, including the most fundamental critical processes that are associated with the Framework’s Stage Two maturity. Because our analysis in Section A of this finding was made on an organizational level, in our case study we assessed how the FBI’s non-implementation of Stage Two critical processes affected an individual project. Next, we examined the FBI’s internal assessments of Trilogy. Finally, we assessed the FBI’s ongoing deployment of new computer hardware, software, and networks to its field offices.
We selected Trilogy for our case study because it is currently the FBI’s largest ongoing IT project, with $458 million in total appropriations as of June 2002. Trilogy’s purpose is to upgrade the FBI’s: (1) hardware and software or Information Presentation Component (IPC), (2) communication networks or Transportation Network Component (TNC), and (3) five most important investigative applications or User Applications Component (UAC). The IPC and TNC upgrades will provide the physical infrastructure needed to run the applications from the UAC portion. The UAC portion is intended to upgrade and consolidate 5 of the FBI’s 42 investigative applications. Because there are 37 other investigative applications and approximately 160 non-investigative applications that Trilogy will not include, Trilogy is only a starting point toward upgrading the FBI’s entire IT infrastructure.
When discussing the state of the FBI’s IT systems and the benefits Trilogy could bring, one Special Agent-In-Charge told us that “Trilogy must improve the FBI’s IT systems. There is just no other way that agents can continue operating with such limited abilities.” A senior FBI official stated to the Senate Judiciary Committee in July 2002 that agents must go through 12 screens just to upload one document in ACS. She further stated that the process is even more difficult because “there’s no mouse, there’s no icon, there’s no year 2000 look to it, it’s all very keyboard intensive.” While FBI officials stated that Trilogy is not intended to provide the FBI with a state-of-the-art IT system, it lays the technological foundation so that an effective information system can be built. The implementation of Trilogy is vital to enhancing the FBI IT infrastructure, and consequently to the FBI’s mission performance.
(1) Evolution of the Trilogy Project
During the 1990’s, the FBI recognized that its IT infrastructure was aging and in need of modernization. Since 1997, the FBI has proposed to Congress several projects intended to improve its IT infrastructure and office automation.
First, the Information Sharing Initiative (ISI), a four-year project with an anticipated cost of about $400 million, was presented to Congress in 1997. The project’s purpose was to upgrade the FBI’s critical hardware, software, and communications capabilities and thus facilitate the development and deployment of modern computer applications. It also would have provided secure information sharing within the FBI, and to law enforcement agencies outside of the FBI.
In November 1998, the ISI was funded by Congress with FY 1999 appropriations. However, expenditure of funds was contingent on the approval of the implementation plan and a review of it by the OMB’s IT Technology Review Board. Following the OMB’s review of the ISI plan, the FBI made minor modifications to the requirements document and acquisitions strategy. By January 2000, the FBI was ready to award the ISI contract. However, the Senate and House Appropriation Committees had not approved the implementation plan. FBI officials told us that by 1999, Congress had become increasingly concerned with the FBI’s ability to manage major IT projects on time and within budget. We were told by FBI officials that this loss of credibility was caused by previous large-scale FBI IT projects that experienced significant cost and schedule overruns. Particularly, those officials said that the Integrated Automated Fingerprint Identification System and National Crime Information Center both were completed millions of dollars over budget and years behind schedule.
Because of the FBI’s poor track record of managing major IT projects within cost and schedule, Congressional committees recommended that the FBI utilize a pilot implementation concept for ISI, which would modernize the IT infrastructure in phases. FBI officials said they resisted this concept because of concerns over having two sets of infrastructures, one old and one new. As a result, the FBI abandoned the ISI initiative.
In the Spring of 2000, the FBI prepared a project plan called eFBI, which was essentially a scaled back version of ISI. Because the project was less costly, FBI officials hoped that Congress would be more receptive to the project. The main difference between ISI and eFBI was that eFBI did not have the secure electronic information sharing capabilities included with ISI. However, press reports indicated that the FBI did not receive funding for the project when DOJ officials objected to certain proposed bidding procedures.
Because these plans to upgrade the FBI’s IT infrastructure were never approved, the FBI’s IT infrastructure had not received meaningful improvements since the early 1990’s. As a result, there was an increasing need for a Bureau-wide IT upgrade. According to FBI documentation, by September 2000:
Recognizing its credibility problems with Congress, in July of 2000, the FBI hired a new chief information officer from the private sector to outline IT management. The new chief information officer was tasked with submitting another major technology upgrade plan to Congress. That plan, called the FBI Information Technology Upgrade Plan (FITUP), was drafted and delivered to Congress in September 2000. The FITUP was intended to achieve goals similar to the ISI and eFBI projects. FBI officials told us that Congress appeared more satisfied with the FBI’s new IT management team, and consequently appropriated $379.8 million in November 2000 to fully fund the FITUP over a three-year period.
The objectives of the FITUP, as defined by the FBI, were to:
In November 2000, the FITUP was renamed Trilogy. A brief description of Trilogy’s three components (IPC, TNC, and UAC) follows.
The IPC refers to how users see and interact with information. The IPC provides new desktop computers, servers, and commercial-off-the-shelf office automation software, including a web-browser and e-mail to enhance usability by the agents. The original Trilogy plan also included the use of thin-client desktop computers. Thin-client desktop computers, according to the FITUP, utilize application software that is run from the server computer, and consequently permits the desktop computer to function with fewer hardware resources such as processors and memory. Other benefits to the thin-client strategy included less maintenance of software in field offices and timely technology upgrades to meet user needs. The FITUP further stated that the FBI sized the departmental servers to handle the processing demands imposed by the thin-client strategy.
The TNC is the complete communications infrastructure and support to create, run, and maintain the FBI’s networks. It is intended to be the means by which the FBI electronically communicates, captures, exchanges, and accesses investigative information. The TNC includes high capacity wide-area and local-area networks, authorization security, and encryption of data transmissions and storage.
The FBI combined the IPC and TNC portions for continuity when it requested contractor support, as both encompass physical IT infrastructure enhancements. The contractor for the IPC/TNC portions was signed in May of 2001. The originally scheduled completion date for these components was May 2004.
The UAC defines software-based capabilities and functions that Special Agents can use to access and analyze the information they need. The UAC is intended to provide the FBI with:
The UAC is also referred to as the Virtual Case File. The Virtual Case File is intended to replace ACS as the FBI’s primary investigative application. The goal of the Virtual Case File is to reduce agents’ reliance on paperwork to improve efficiency. The Virtual Case File is supposed to have multi-media capability that will allow agents to scan documents, photos, and other electronic media into the case file. A separate contractor was hired in June 2001 to complete the UAC portion of Trilogy by June 2004.
(2) Accelerated Deployment of Trilogy
Even before the terrorist attacks on September 11, the FBI was looking for ways to accelerate the three-year Trilogy project, given the FBI’s urgent need for improved IT infrastructure. In its Quarterly Congressional Status Report for the period between May 14, 2001 and July 6, 2001, FBI personnel stated that it had devised a plan to complete the IPC/TNC deployment in June 2003, nearly one year ahead of schedule, while the UAC deployment remained a three-year project. However, FBI officials stated they wanted to accelerate deployment of UAC.
After the terrorist attacks on September 11, 2001, the urgency of completing Trilogy increased. The FBI continued to explore options to accelerate the deployment of all three components of Trilogy. The FBI informed Congress in its February 2002 Quarterly Congressional Status Report that it devised a new plan with the contractor to complete the deployment of the IPC/TNC phases by December 31, 2002, which was nearly 18 months earlier than the originally planned completion date. Additionally, the FBI’s February 2002 report stated that the contractor for the UAC phase developed a plan to make ACS web-enabled by July 2002. Web-enablement of ACS56 was designed to put ACS in a multi-media format prior to the completion of the UAC phase in July 2004. According to its Congressional reports, the FBI could make these enhancements to Trilogy without any net increases to the project costs. The FBI would only need to have a portion of the funding earmarked for FY 2003 available by October 30, 2002.
The FBI also informed Congress in its February 2002 report, that with an additional $70 million funding for FY 2002, the FBI could further accelerate the deployment of Trilogy. This acceleration would include completion of the IPC/TNC phase by July 2002 and rapid deployment of the most critical analytical tools included as part of the UAC phase.
Congress supplemented Trilogy’s FY 2002 budget with $78 million from the Emergency Supplemental Appropriations Act of January 2002 to expedite the deployment of all three components. The Emergency Supplemental Appropriations Act increased the total funding of Trilogy from $379.8 million57 to $457.8 million. According to Trilogy documentation, the FBI obligated about $231 million as of June 2002. Trilogy’s budget by component, as of June 2002, is described in the following table.
Trilogy’s Budget by Component
|Component Area||FY||Original Plan||Revised Plan Including
|Contractor Computer Specialists||Total||-||$8.0|
|Source: FBI budget documentation|
Congress’s willingness to provide the FBI with additional funding after September 11 was not limited to Trilogy. The FBI saw an increase in funding of approximately 102 percent for IT projects from $352.8 million FY 2001 to $714 million in FY 2002.
The IPC/TNC infrastructure enhancements are being deployed in three phases in the accelerated plan. The first phase, called Fast Track, is the installation of Trilogy hardware in all of the field offices and some of the Resident Agencies. The Fast Track deployment consists of new network printers, color scanners, local area network upgrades, desktop workstations, and office automation software. FBI officials reported that by the end of April 2002, all of the 56 field offices had Fast Track completed.
We were told by FBI officials that following the completion of Fast Track, the next phase of deployment, referred to as Extended Fast Track, was initiated, and was still continuing as of June 2002. Under Extended Fast Track, the FBI: (1) installed servers and other network components at field office and resident agency sites, and (2) deployed the hardware included under Fast Track to additional resident agency sites that were not included in the first phase. Also, the FBI intended Extended Fast Track to correct any shortfalls in the distribution of hardware to the field offices that occurred in the original Fast Track deployment.
The final phase of the deployment, called Full Site Capability, represents the complete infrastructure upgrade. This phase will provide the wide area network connectivity together with new encryption devices, new operating systems and servers, and new and improved e-mail capability. According to June 2002 Congressional Testimony, Full Site Capability is expected to be completed in March 2003.
The UAC portion is also going to be deployed in two phases in the accelerated plan, release one and release two. The initial Virtual Case File release will migrate data from the current ACS and IntelPlus to the Virtual Case File. The Virtual Case File will replace ACS and serve as the backbone of the FBI’s information systems, replacing the FBI’s paper files with electronic case files that include multi-media capabilities. The first release of Virtual Case File has a targeted completion date of December 2003. This release is intended to allow different types of users, such as agents, analysts, and supervisors, to access information from their desktop computers that is specific to their individual needs. This Virtual Case File release is also intended to enhance the FBI’s capability to set and track case leads, index case information, and move document drafts more quickly through the approval process with digital signatures.
The second release is intended to upgrade three other investigative applications into the Virtual Case File. The second Virtual Case File release has a targeted completion date of June 2004. It is intended to provide agents with Audio/Video Streaming capability and content management capability. According to FBI documentation, content management should help agents access information from the FBI’s data warehouse, regardless of where in the system the information was entered, providing a single query for all of the FBI’s systems.
(3) Results of Our Assessment of Trilogy Against the Stage Two Critical Processes
The Framework provides the organization level processes necessary for effective IT investment management. As a result, the Framework’s critical processes, and in particular the Stage Two critical processes, do not necessarily ensure that individual IT projects will be effectively managed. However, it does ensure that, at a minimum, basic selection and management control processes are in place.
As discussed in Section A of this finding, Stage Two builds the foundation for successful IT investment management by establishing basic IT selection and control processes for IT projects. Stage Two is defined by the following five critical processes:
Our assessment of how Trilogy was managed in relation to each Stage Two process is described in the following paragraphs.
a. IT Investment Board Operation
According to the Framework, IT investment boards have executive decision-making authority throughout the organization. This organization-wide perspective is necessary to ensure that only the best projects are selected for development, and projects under development are being monitored with consistent policies and controls.
In section A of this finding, it was noted that the FBI did not have IT investment boards operating prior to March 2002. Because Trilogy was initiated in September 2000, it was not selected through the operation of formal IT investment boards. Additionally, because the FBI’s IT investment boards were not involved in overseeing IT projects as of June 2002, Trilogy has not been subjected to board oversight.
FBI officials have told us that most of Trilogy’s development has been managed in a “stovepipe.” One FBI official told us that the organization’s focus on Trilogy has drained the FBI of a broader view of IT. As a result, FBI personnel not involved in the management of Trilogy had little knowledge of the project’s status and progress. Although the Trilogy management structure has changed frequently, it was managed out of the IRD until March 2002. However, IRD personnel who were responsible for acquiring IT products and services through contractors on IRD IT projects were not involved in Trilogy’s acquisitions. Only members of the Trilogy management team performed these activities. Further, FBI personnel told us there was little coordination taking place with Trilogy management and contract specialists from the Finance Division or the IRD’s unit responsible for procurement of non-Trilogy IT needs. Because of the lack of coordination, there is a heightened risk that resources could be spent on potentially duplicative or non-compatible hardware, software, and systems. FBI officials have told us that the IRD is in the process of developing technical enterprise architecture that incorporates Trilogy requirements to mitigate this risk.
b. Project Oversight
The GAO Framework states that IT investment boards should monitor all projects relative to cost, schedule, and technical baselines to measure the progress of IT projects under development, and the performance of projects upon deployment. When an IT project is not performing according to expectation, the investment boards should seek corrective actions to be taken.
IT investment boards have not been involved in overseeing Trilogy. In our judgment, the lack of project oversight from IT investment review boards contributed to the FBI not having established schedule, cost, and technical baselines for Trilogy, as of June 2002.58
In terms of a cost baseline, FBI officials told us that the rapid procurement and deployment of Trilogy has prevented the project managers from performing earned value management,59 as promised in the FITUP. While FBI officials were confident they know how much money has been spent on Trilogy to date, and how much funding has been committed, they have less assurance as to whether Trilogy is on budget, over budget, or under budget.
A schedule baseline for Trilogy has never been well-established. First, FBI officials said they would complete IPC/TNC deployment in May 2004. Then, they said it could be finished in June 2003. Next, they said it would be finished by December 2002. After receiving $78 million of supplemental funding, they said it would be done by July 2002. Then, they said they could not make the July 2002 deadline and moved it to October 2002. As of June 2002, FBI officials have said deployment will probably not be complete until March 2003. Also as of June 2002, the FBI was still in the process of building a comprehensive schedule of Trilogy milestones.
In terms of a technical baseline, we previously stated that the FBI is still developing a technical architecture framework that includes Trilogy hardware and software. Personnel from the enterprise architecture office initially told us at the beginning of our audit that they were not significantly involved in ensuring that Trilogy acquisitions were compatible with non-Trilogy hardware and software. But, as of June 2002, the enterprise architecture office had developed a technical reference model, although it was not finalized.
According to the FITUP, the philosophy employed in implementing Trilogy was “to get 80% of what is needed into the field now rather than 97% later. Then we can proceed in an orderly fashion to move toward 100% in the future.” Additionally, after the events of September 11, the urgency to deploy Trilogy as quickly as possible increased. FBI management told us that risks associated with this rapid deployment were accepted. Further, they stated that given the accelerated schedule, and additional funding needed, the cost and schedule baselines could not be static.
While the events of September 11, 2001 affected the FBI’s ability to manage cost, schedule, and technical baselines, we believe the risks of not establishing such baselines puts the project at a high risk of failure. Although the overall success of Trilogy will not be determined for years to come, the FBI has already missed the July 2002 deadline to complete the IPC/TNC phase. In our judgment, this missed deadline is a further indication that increased oversight of the project is needed.
The new Trilogy project executive, hired in March 2002, has taken a different approach to managing Trilogy. She has emphasized the importance of having more structured oversight of the project. She has been developing a comprehensive schedule for all three components. Additionally, she has indicated that there are limitations to how fast Trilogy can be deployed, without risking the security of the system. In our judgment, while these actions since March 2002 represent positive changes to Trilogy’s project management function, the project’s completion time, final cost, and ultimate performance remain uncertain. Also, we concluded that for the Trilogy project management function to be effective, it must include oversight from IT investment review boards to provide much needed monitoring.
c. IT Project and System Identification
According to the Framework, IT project and system identification provides essential information to an organization as to how its IT assets (such as personnel, systems, applications, hardware, software licenses, etc.) are configured and relate to one another. Having a complete inventory of the organization’s IT assets, including documentation of the configuration and technical architecture of IT systems, helps ensure that IT investment review boards will select projects that comply with the existing architecture in place. Additionally, this process can be equated with an organization having a blueprint of what systems it tilizes, how those systems were created, and what can be done to enhance those systems.
As noted in section A of this finding, we found that the FBI did not have a comprehensive inventory of all IT assets, including complete documentation of the technical architecture of its systems. Because the UAC portion of Trilogy is focused on making significant changes to, or possibly complete replacements of, five of the FBI’s investigative systems, having documentation of the exact configuration of these systems is critical to designing the requirements for UAC. According to a senior FBI official, the FBI must know what it has before it can define the right solution to fix the problem. Not having the documentation of the configuration of these five investigative systems has caused the FBI to engage in a process of reverse engineering, which is trying to determine the structure and components of the systems after deployment. Because the FBI has to perform reverse engineering on the FBI’s five investigative systems that will be migrated to the Virtual Case File, there are limitations as to how rapidly UAC can be developed and deployed.
As of June 2002, the FBI was still defining the requirements for UAC because of the reverse engineering activities. Without knowing the exact requirements, the FBI will have difficulty establishing cost and schedule baselines for this component of Trilogy. As a result, some FBI officials told us that they believe the UAC portion of Trilogy is at significant risk of not being completed on schedule (in June 2004) or within budget.
d. Business Needs Identification for IT Projects
According to the Framework, an organization should have a systematic process for identifying, classifying, and organizing its business needs and the IT projects used to support these needs. This process should allow for the identification and definition of the business needs and specific users for all IT projects. This process can be equated with knowing where the organization wants to go, based on its mission, and the needs of its users to pursue that mission. While we concluded that the Trilogy project’s users were identified, since all users of the FBI’s systems will be affected by the IPC/TNC portion of the project, we found that the specific needs of the users, and of the FBI as a whole, were not adequately defined before Trilogy was selected and funded.
Specifically, we found that the requirements for the applications of the UAC portion were still being defined as of June 2002. Since January 2002, the FBI and the contractor were participating in a Joint Application Development planning process to define and prioritize the users’ operational requirements. This process brings users, designers, and future systems operators together to develop the applications in order to better establish operable and maintainable systems.
The Joint Application Development sessions represent a thoughtful and productive approach to ensuring that the UAC portion of Trilogy will adequately support agents’ investigative activities. However, in our judgment, this process should have been initiated from the beginning of the Trilogy project.
e. Proposal Selection
According to the Framework, proposal selection activities ensure that the right projects are selected to support the organization’s mission. The proposal selection process relies on the project and system identification process, as well as the business needs identification process, so that information contained within project proposals include sufficient documentation of the technical requirements of the projects.
While no investment boards existed at the time of Trilogy selection, it has been widely recognized by the Attorney General, FBI Director, and Congress that an investment in the upgrade of the FBI’s information technology was essential to the FBI meeting its mission goals. The FBI’s technology was outdated in terms of hardware, software, user-applications, connectivity, and data sharing abilities. There is little question of the FBI’s need to select this project. However, successful execution and deployment of the project depends on having the other control processes in place. Specifically, proposals should have adequate documentation of technical requirements and project risks.
We were told that some aspects of Trilogy that were submitted to Congress did not turn out to be technically feasible. For example, FBI officials told us that the thin-client strategy was not pursued because it was found that this type of network could not be achieved given the technical requirements of the FBI. Another example is web-enablement of the ACS, which was also discontinued when it was realized that it would require more resources than anticipated. Had a more rigorous proposal selection process been in place that required sufficient documentation of the technical requirements and risks of the project, the expending of time and resources on thin-client technology and web-enablement of ACS may have been minimized.
We have found that not implementing the critical processes associated with Stage Two maturity has contributed to missed milestones and uncertainties associated with the remaining portions of Trilogy. However, the FBI’s new Trilogy project executive has taken positive steps in establishing management controls and oversight to the project.
We recommend that the Director of the FBI ensure:
(4) The FBI’s Internal Assessments of Trilogy
The FBI had three internal assessments performed concerning the management of the Trilogy project. These assessments were done by the FBI’s Inspection Division, CJIS Division, and a contractor performing independent verification and validation work. The assessments found that the lack of baselines and general program oversight pose potential risks for the Trilogy program meeting its budget, schedule, technical, and performance goals. These assessments recommended that the FBI designate a program manager specifically for Trilogy, and that the program manager immediately take steps to establish baselines and requirements for the project.
The objective of our case study was to determine how Trilogy was being managed within Stage Two of the Framework. These assessments go beyond that objective and address additional areas of potential risk within Trilogy, such as security and configuration management. An overview of the three independent assessments (FBI Inspection Division Trilogy Risk Assessment, November 2001; Trilogy Independent Validation and Verification, December 2001; and CJIS Division Trilogy Assessment, January 2002) are presented in the following paragraphs.
a. Inspection Division Trilogy Risk Assessment
Because of the size and importance of Trilogy to the FBI, the Inspection Division’s MPMOU issued a risk assessment report on the Trilogy project to the FBI Director in November 2001. This assessment identified areas of high risk within the acquisition, financial, requirements, and overall project management of Trilogy. The areas found to be high risk included a lack of project requirements and baselines, the lack of a defined program organizational structure and program manager, and improper scheduling and cost estimates.
The report recommended that the FBI institute a short-term strategy to provide interim capabilities and a long-term strategy to restructure Trilogy. The report recommended that the short-term strategy should include a detailed plan identifying what can realistically be accomplished within a pre-determined period. It further stated that the short-term plan should have a clearly defined scope so that progress can be measured and quantified.
The MPMOU issued two follow-up letters to the Director in December 2001 and February 2002 to assess the FBI’s progress in mitigating these risks and taking action on their recommendations.
In December 2001, the Inspection Division indicated that while Trilogy management acknowledged certain project risks, Trilogy managers were willing to accept aspects of those risks and move forward. However, personnel from the Inspection Division noted that FBI senior management did hire a program manager for Trilogy in March 2002.
In February 2002, Inspection Division personnel indicated that there was then disagreement between them and Trilogy management on the level of project risk for Trilogy. The Inspection Division pointed to a CJIS review and an outside independent validation and verification report on Trilogy establishing that significant risks to the project exist, in the areas originally identified by the Inspection Division. The Inspection Division then reiterated its previous recommendation that calls for the development of a short and long-term strategy for Trilogy. Inspection Division personnel told us that Trilogy management did not sufficiently develop a short and long-term strategy for the project as was recommended.
b. Trilogy Independent Validation and Verification
The IRD hired an outside contractor to obtain an independent perspective on Trilogy. The objective of the assessment was to determine the labor requirements, level of effort, and verification and validation tasks necessary to ensure that the Trilogy acquisition meets the requirements of FBI users into the future within the established schedule and budget.60 The independent validation and verification report, issued in December 2001, disclosed risks in the Program Management of Trilogy, IPC/TNC portion, and the UAC portion of Trilogy, including a lack of program management structure and focus, a lack of formal requirements, schedules, and baselines, and changes in the UAC/IPC/TNC portions without formal changes to contracts. While we concluded that the FBI improved the Trilogy management structure through the hiring of a new project manager in March 2002, we believe that risks associated with lack of formal requirements, schedules, and baselines still remained as of June 2002.
c. CJIS Division Trilogy Assessment
Upon reviewing the Inspection Division risk-assessment, the Director requested the CJIS Division to perform an independent review of Trilogy to get another perspective on the project. The CJIS Division performed their assessment between January 3 and January 16, 2002. This assessment covered management, quality assurance, configuration management, IT security, administrative and technical requirements, and technical management. It found weaknesses similar to those identified by the Inspection Division, including a lack of clear lines of authority, no clearly designated Program Manager, a lack of authority and support in the areas of quality assurance, security, configuration management, and technical requirements, and insufficient technical reviews of Trilogy documentation. While we concluded that the FBI improved the Trilogy management structure through the hiring of a new project manager in March 2002, we believe there are still weaknesses in Trilogy’s documentation of technical requirements as of June 2002.
The three internal risk-assessments on Trilogy found significant risks associated with the management of the project. In our judgment, effective IT investment management practices, including active oversight from IT investment review boards would have mitigated these risks.
(5) Deployment of Trilogy to Field Offices
In addition to assessing the Trilogy management at FBI headquarters, we assessed the Fast Track deployment of Trilogy to five of the largest FBI field offices: (1) New York, (2) Washington, D.C., (3) Los Angeles, (4) Miami, and (5) Chicago. Our objectives were to assess the Fast Track deployment in terms of timeliness, support, and completion. Our goal was to identify current problems and recommend corrective actions, and discuss “lessons learned” for future system deployments.
In her July 16, 2002 Congressional testimony before the Senate Judiciary Committee, the FBI Project Management Executive stated that the Fast Track deployment involved the installation of Trilogy architecture at the FBI’s 56 field office locations. The installation also included as many Resident Agencies as could be completed before the second phase of the deployment (“Full Site Capability”) begins. This architecture consists of new network printers, color scanners, local area network upgrades, desktop workstations, and Microsoft office applications. She also stated that by the end of April 2002, deployment at all 56 FBI field offices was completed, and that Fast Track is continuing to deploy this architecture to the FBI’s Resident Agencies.
a. Timeliness of the Fast Track Deployment
The Fast Track deployment to the five field offices in our survey began as early as December 2001. The FBI Project Management Executive stated in her testimony that “By the end of April 2002, deployment at all 56 FBI field offices and two Information Technology Centers was completed. Fast Track is continuing to deploy this infrastructure to our resident agencies.” During our testing at five FBI field offices in June 2000, we found that implementation activities were still ongoing to correct deficiencies that occurred during the original Fast Track deployment. The FBI Project Management Executive told us that her testimony was limited to “Fast Track” and did not include ongoing activities related to “Extended Fast Track.”
Regarding the Resident Agencies, (1)FBI employees informed us that as of June 2002, deployment to the Chicago, Los Angeles, and District of Columbia Resident Agencies was underway or completed. Deployment to the Miami Resident Agencies was scheduled for August 2002, and deployment to the New York Resident Agencies was still in planning.
Regarding installation of the basic Trilogy architecture by the contractor, employees from all five field offices said the timing of the architecture installation phase of the deployment occurred either on schedule or ahead of schedule.61 Most employees interviewed (ten of eleven) said they were provided ample notice for the timing of the installation.(3) A Telecommunications Manager in the Chicago Field Office said it was one of the FBI’s smoothest “rollouts.” Personnel from the Los Angeles Field Office indicated that through careful preparation they cut the installation phase from the three weeks scheduled to just seven days. (2) Apparently, only the New York Filed Office experienced significant problems with the installation phase of the deployment. Specifically, the financial management system was left inoperable and they had to resort to pre-Trilogy processing to pay employees. Also, the FBI Intranet traffic was not reaching the FBI mainframe computer because of information being routed through too many pathways.
b. Adequacy of FBI Headquarters Support for the Fast Track Deployment
Regarding FBI Headquarters support, most employees we interviewed said they were provided with adequate planning and preparation instructions for the deployment. Employees from the New York Field Office said FBI Headquarters did not provide instructions but instead informed them to send a team to Miami to learn about the deployment, and then return to New York to plan and prepare for it. As to whether there was sufficient communication between FBI Headquarters and the field offices, four of nine employees who responded indicated that communication could have been better to adequately prepare the field offices for deployment.
Six of eleven employees who responded did not believe the FBI’s deployment strategy appropriately considered the individual needs of the field offices. Personnel from the Chicago Field Office indicated that since they had little opportunity to provide input, they had to work around the information and changing timelines received from FBI Headquarters. A supervisory computer specialist from the Los Angeles Field Office indicated the deployment was successful, in part, because they did not use the timeline provided by FBI Headquarters. Personnel from the Miami Field Office said they provided considerable information to the contractor during the survey phase that was subsequently lost. A supervisory computer specialist from the District of Columbia Field Office indicated concern that because offsite locations were not considered, there were an insufficient number of computers to deploy.
c. Adequacy of Contractor Support for the Fast Track Deployment
Eleven of the twelve employees we interviewed told us that the subcontractor for the actual installation work at the field offices was very helpful. Employees generally indicated that the subcontractor was technically competent and professional.
Regarding support from the contractor’s service support center, of ten employees who responded, three employees said they did not use the service, five employees said the support provided was inadequate, and only two said the support was helpful.
Part of the Fast Track deployment planning included the contractor conducting surveys at the field offices and resident agencies to identify existing equipment and installation requirements. The surveys were conducted in the third and fourth quarters of 2001. Regarding the accuracy of the survey work performed by the contractor, five of the nine employees who responded to our question said the surveys did not accurately identify the computer needs of personnel at the field offices.
Of nine employees who responded to our question regarding accessibility of the contractor for equipment maintenance support, six indicated that the contractor was not easily accessible.
d. Adequacy of Training Support Provided to Field Office Personnel
All employees interviewed stated that training in MS Office 2000 applications and MS Outlook was generally available before, during, and after the Fast Track deployment. All interviewees said time was made available for agents to attend this training as well as additional computer-based training available on the FBI Intranet.
However, six of ten interviewees indicated that problems existed with the Learning Management System62 available via the FBI Intranet. These six employees generally indicated that the system has not worked well from the beginning, that the system was down more than it was up, and that application problems existed.
e. Completion of Fast Track Deployment
Based on the interview results, we concluded that the Fast Track deployment for all five field offices in our sample did not provide the quantities of the desktop computers that were expected. As a result, the FBI initiated Extended Fast Track to provide the desktop computers that were not originally provided with the Fast Track deployment. According to the FBI Project Management Executive, miscommunications between FBI Headquarters and the field offices resulted in differences between the number of desktop computers delivered by FBI Headquarters and the number of desktop computers expected to be received. Additionally, the FBI Project Management Executive said shortages of fiber optic cable resulted from these miscommunications, as some field offices budgeted for the wrong amount of cable. We found that as of June 2002 (the month our interviews were conducted), some field offices did not have sufficient quantities of fiber optic cable to complete the deployment and hundreds of desktop computers still remained to be delivered.
We did determine, however, that each desktop computer delivered included the complete baseline hardware and software package specified by the fast track deployment. Additionally, we randomly selected 30 Trilogy desktop computers received by each field office and verified that the desktop computers were received, installed, and operational.
For two of the five field offices we reviewed, additional installation work remained to complete the Fast Track deployment. At the Los Angeles Field Office, we were informed that about 40 percent of the Trilogy desktop computers were not connected to servers and networked because of the shortage of fiber optic cable. Additionally, although Los Angeles received the requisite number of Trilogy printers, none of these printers were operational because of the shortage of fiber optic cable. At the District of Columbia Field Office, we were informed that only 3 percent of the Trilogy printers received were operational because the required fiber optic cables had not yet been installed.
Additionally, there appeared to be some confusion between FBI Headquarters and some of the field offices as to the actual number of Trilogy desktop computers to be deployed under Fast Track. As a result, four of the field offices had not yet received their full compliment of desktop computers as intended under the Fast Track deployment.
f. Most Significant Obstacles to Fast Track Deployment
When asked to provide what they perceived to be the most significant obstacles to the Fast Track Deployment, personnel from the five field offices provided the following responses:
g. Limitations to Field Offices Fully Utilizing Trilogy Fast Track Capabilities
When asked what are the current limitations to utilizing Trilogy Fast Track capabilities, personnel from the five field offices provided the following responses.
Based on the results of our work at the five field offices, the Extended Fast Track deployment was still ongoing as of June 2002. For two of the field offices, additional installation work remained to be completed, and for four of the field offices hundreds of desktop computers still remained to be delivered. A lack of clear communication between FBI Headquarters and the field offices contributed to the confusion over the number of desktop computers to be delivered and shortages of fiber optic cable. Additionally contractor maintenance support for the Trilogy architecture was inefficient, resulting in agents being without computers for weeks at a time. Improvements in agent and support personnel training, procurement of trouble-shooting equipment for the Trilogy architecture, and timely customization of word processing software will enhance user utilization of the Trilogy architecture.
2. The FBI’s IT Strategic Planning and Performance Measurement
A. Background on Strategic Planning
Strategic planning is used to determine and reach agreement on the fundamental results the organization seeks to achieve the goals and measures it will set to assess programs, and the resources and strategies needed to achieve its goals. Additionally, according to the GAO’s June 2002 testimony to the House Appropriations Committee:63
Strategic planning helps organizations to be proactive, anticipate and address emerging threats, and take advantage of opportunities to be reactive to events and crises. Leading organizations, therefore, understand that planning is not a static or occasional event, but a continuous, dynamic, and inclusive process. Moreover, it can guide decision-making and day-to-day activities.
The Government Performance and Results Act of 1993 (Results Act) provides for the establishment of strategic planning and performance measurement in the federal government. It seeks to improve the effectiveness, efficiency, and accountability of federal programs by establishing a system for agencies to set goals for program performance and to measure results. The Results Act requires agencies to prepare a strategic plan, annual performance plans, and annual performance reports. The strategic plan, which is the key requirement of the Results Act, identifies agencies' long-term goals. Federal agencies are required to update their strategic plan at least every three years.
While the Results Act applies to the DOJ, it does not specifically apply to components such as the FBI. However, in our judgment, for the DOJ to comply with the Results Act, the components must have strategic and performance plans that are consistent with, and support, the DOJ’s strategic and performance plans.
Annual performance plans include measurable goals that define what an agency will accomplish during a fiscal year. These plans should: (1) establish performance goals to define levels of performance to be achieved; (2) express those goals in an objective, quantifiable, and measurable form; (3) briefly describe the operational processes, skills, technology, human capital, information, or other resources required to meet the goals; (4) establish performance measures for assessing the progress toward, or achievement of, the goals; (5) provide a basis for comparing the actual program results with established goals; and (6) describe the means to be used to verify and validate measured values. There are at least two iterations of the annual performance plan. The initial annual performance plan is submitted to the OMB and is used during its review of the agency's budget request. The final annual performance plan is submitted to Congress soon after the transmittal of the President's budget.
The DOJ’s annual performance plan is comprised of two parts. The first part is a summary performance plan that provides a departmental overview and synthesis and is submitted as a stand-alone document. The second part consists of the individual performance plans of the departmental components. These component plans are prepared pursuant to guidance provided by the DOJ and are incorporated within the components’ budget submissions. Component plans should support the objectives, goals, and strategies of the DOJ's annual performance plan so that the DOJ can rely on the data provided through the component reports. In our judgment, components that do not incorporate the DOJ’s objectives, goals, and strategies in their strategic and performance plans are at a heightened risk of not allocating resources in accordance with the DOJ’s strategic priorities.
B. Strategic Planning’s Relationship to the ITIM Process
According to the Framework, the purpose of ITIM is to describe and improve the IT investment management processes so that the strategic plans and decisions that are made can and will be supported by highly effective IT investments. Similarly, performance measures created and used to guide the organization and its activities are a factor in some ITIM critical processes. However, in general, activities related to the ongoing development and implementation of performance measures are largely outside the scope of the GAO ITIM Framework.
Although strategic planning is a function that is largely independent of the ITIM process, strategic planning activities relate to the Framework’s activities at different stages of investment maturity. Specifically, the business needs identification critical process in Stage Two has a key practice that requires the organization to have defined business needs or stated mission goals. Additionally, Stage Five maturity, leveraging IT for strategic outcomes, is highly dependent on the comprehensiveness of the organization’s strategic plan. Stage Five maturity also focuses on the organization’s ability to improve strategic outcomes, change business processes to take advantage of technology changes, and learn from others by benchmarking processes. Based on the interdependencies between the ITIM and strategic planning processes, in our judgment the organization’s strategic plan should address IT investment management.
In July 2002, the DOJ released its IT Strategic Plan that included the following four goals:
To meet these goals, the DOJ is focused on four key areas that it considers to be the building blocks of the IT program: (1) IT infrastructure, (2) information security, (3) common solutions, and (4) management roles and processes. One of the strategic initiatives that comprise management roles and processes is: “Establish and implement improved investment management processes and practices.”64 Based on this focus, in our judgment the DOJ has recognized the importance of integrating strategic planning with IT management.
C. Results of our Assessment of the FBI’s IT Strategic Planning and Performance Measurement
We found that the FBI’s IT strategic planning and performance measurement is inadequate because: (1) the FBI’s strategic plan does not incorporate the ITIM process, and (2) the FBI’s strategic plan and performance plan are not consistent with the DOJ’s annual performance plan.
The FBI’s ITIM Model and Transition Plan states that the Bureau’s IT strategic plan must incorporate the ITIM process in order for it to achieve advanced IT investment maturity. However, as of the end of June 2002, the FBI did not have a current strategic plan dedicated to IT. Instead, individual divisions had program plans that included the use of IT within the particular program.
Additionally, the Bureau-wide strategic plan has not been updated since 1998. Not only does this time period pre-date the FBI’s ITIM process, but it also pre-dates the development of the Framework in 2000. Officials in the Office of Strategic Planning told us that the Office of Strategic Planning’s recent efforts have not been focused on IT.
The FBI acknowledged to us that it must incorporate strategic planning with its ITIM process, including updating its strategic plan. In our judgment, without a new strategic plan, the FBI will limit the effectiveness of its ITIM and strategic planning processes.
Further, we found that the FBI's strategic plan (from 1998) and its FY 2003 performance plan did not support the DOJ's annual performance plan relating to IT. This lack of support occurred because the FBI’s strategic and performance plans are not consistent with the strategic objectives, goals, and strategies relating to IT as the DOJ's annual performance plan. The DOJ’s FY 2003 annual performance plan includes the strategic objective to "make effective use of IT." Additionally, this strategic objective is supported by the annual goal to "expand electronic access and dissemination of information while ensuring IT security and cost effective IT investments meet programmatic and customer needs." However, both the strategic objective and the annual goal are not included within the FBI strategic plan and FY 2003 performance plan. As a result, there is a heightened risk the FBI may not be appropriately allocating resources to meet the DOJ’s strategic priorities.
The FBI must have a Bureau-wide IT strategic plan to maximize the use of its IT investments, rather than having the division-specific IT focus that is currently in place. In fact, the purpose of the FBI’s ITIM process is to move away from managing IT in division “stovepipes” to a centralized, Bureau-wide management focus. The FBI’s strategic planning process must evolve with the ITIM process to ensure the success of both functions.