- During the 2-year period ending in December 2007, the FBI spent $4.6 million in TCCF funds to implement CALEA provisions. Of this amount, the FBI paid about $4.5 million to two carriers to deploy CALEA-related solutions on over [SENSITIVE INFORMATION REDACTED] wire line networks switches. In addition, another carrier received $96,878 for costs incurred from testing software developed under 4 different agreements. We could not assess the reasonableness or cost effectiveness of these expenditures because the FBI did not base payments on independent cost data or competing price estimates. During this review period, the FBI revamped its CALEA testing team and implemented new resources to help measure and facilitate CALEA compliance capabilities of telecommunication providers and manufacturers.
Between January 2006 and December 2007, the FBI spent a total of $4.6 million in TCCF funds to implement CALEA. Table 4 presents a summary of TCCF payment activity during this period.
TABLE 4: TCCF EXPENDITURES 2006-2008
|Carrier Deployment Agreements||4,509,110|
The following sections analyze the TCCF expenditures made during the audit period ending December 2007.
The ability of law enforcement agencies to conduct authorized electronic surveillance anywhere within a telecommunication network is a central tenet of CALEA. According to internal FBI correspondence, if CALEA capabilities and dial-out solutions were not widely deployed, the ability of law enforcement to conduct surveillance would be, “limited to a few basic and costly surveillance functions.” Therefore, after spending about $452 million to pay manufacturers for costs of designing and developing CALEA software solutions under RTU licensing agreements, the FBI began assessing how best to deploy the developed technologies nationwide.
To maximize the benefits to law enforcement agencies, the FBI concentrated on activating CALEA software solutions on pre-1995 equipment used by the regional bell operating companies (RBOC). According to FBI documents, the RBOCs supported over 50 percent of all wire line switches in the United States.
We found that the FBI finalized agreements with two carriers – Verizon and BellSouth – and during our audit period paid a total of $4.5 million for costs associated with activating the RTU software CALEA features and dial-out solutions on over [SENSITIVE INFORMATION REDACTED] network switches.23 In our previous CALEA audit report, we expressed concern over the FBI’s plans to reimburse traditional wire line carriers the costs associated with deploying CALEA solutions on pre-1995 switches.24 The report recommended that since CALEA implementation was delayed significantly and communication technologies had changed drastically since CALEA’s enactment, the FBI should reexamine the benefits of activating CALEA software solutions on wire line systems. As discussed earlier in the report, however, Congress rescinded over $40 million from the TCCF in 2007. According to FBI’s last annual CALEA report, the TCCF rescission effectively ended FBI plans to establish future reimbursement agreements with additional wire line carriers.25
According to internal documents, the FBI had many discussions with Verizon officials about payments for deploying CALEA solutions developed under various RTU agreements. Since the FBI did not have an established history of reimbursing carriers for costs associated with implementing CALEA capabilities, the negotiations considered a series of cost proposals. To serve as a starting point for these negotiations, Verizon submitted a preliminary proposal detailing costs associated with activating CALEA solutions on its entire wire line network.
To determine the total reimbursable cost, Verizon first estimated the cost of: (1) upgrading equipment associated with its pre-1995 switches; and (2) deploying CALEA solutions based on security, training, management, and labor expenses to its entire wire line network. In communications with the FBI, Verizon first reported that it would cost about $[SENSITIVE INFORMATION REDACTED] to upgrade equipment and deploy CALEA solutions to its pre-1995 platforms. During subsequent negotiations with the carrier, the FBI identified certain proposed costs as unrecoverable under CALEA.26 [SENSITIVE INFORMATION REDACTED]
In October 2004, the FBI Finance Division performed an audit of Verizon’s reduced cost estimate for acceptability in negotiating a final contract price. The audit found that the $[SENSITIVE INFORMATION REDACTED] proposal appeared to be overstated and determined that the FBI should only pay Verizon for costs associated with: (1) training its personnel on new software; (2) implementing security system modifications; (3) testing upgraded software; and (4) direct costs of deploying CALEA software features, including dial-out enhancements, on three of its major platforms. As shown by Table 5, although the FBI audit did not take issue with the costs associated with activating network switches or security system development and training, the audit found that the FBI should not pay for other operating and support costs.
TABLE 5: FBI ADJUSTMENTS TO CARRIER COST PROPOSAL
|Cost Category|| Revised Verizon Estimate
|FBI Audit Adjusted Figures
|Reason for Audit Adjustment|
|[SENSITIVE INFORMATION REDACTED]|
Based on the audit results, the FBI formalized a contract to pay Verizon the costs associated with upgrading switches in November 2005. The contract stated that Verizon would be paid $[SENSITIVE INFORMATION REDACTED] for each switch activated with CALEA compliant software features.27 As shown by Table 6, the agreement based the $[SENSITIVE INFORMATION REDACTED] figure on the total cost Verizon would incur to upgrade its platforms, divided by the number of switches Verizon agreed to upgrade.
TABLE 6: VERIZON CONTRACT PAYMENT DETAILS
|Final approved cost|| [SENSITIVE
|Number of switches to upgrade|
|Cost per certified upgraded switch|
To receive TCCF funds, the agreement called on Verizon to submit quarterly invoices to the FBI that: (1) listed the specific switches activated pursuant to the agreement; and (2) certified that all listed switches, “have been modified to provide all CALEA Assistance Capabilities required.” Between April 2006 and January 2007, the FBI received [SENSITIVE INFORMATION REDACTED] invoices from Verizon certifying the upgrade of [SENSITIVE INFORMATION REDACTED] switches at a total cost of $2,901,900.28
The FBI also negotiated an agreement with BellSouth to reimburse costs associated with deploying CALEA solutions on its wire line network. According to officials at the OSCU, the FBI sought to have BellSouth deploy the same types of CALEA solutions to similar types of switch platforms as it had with Verizon.
In November 2005, the FBI formalized an agreement to pay BellSouth $1,607,210 to deploy CALEA software on [SENSITIVE INFORMATION REDACTED] network switches. According to these FBI officials, the agreed upon payment was: (1) based on performance terms negotiated during the Verizon negotiations; and (2) “in line with” the costs reimbursed under the Verizon contract on a per switch basis. Therefore, the FBI did not conduct an extensive series of negotiations with BellSouth, like it did with Verizon, before finalizing the BellSouth agreement. In April 2006, BellSouth certified that it had modified [SENSITIVE INFORMATION REDACTED] switches to provide CALEA assistance capabilities called for by the agreement and the following month invoiced the FBI $1,607,210.
Analysis and Assessment of Carrier Agreement Costs
We reviewed invoices sent by Verizon and BellSouth and found that each invoice: (1) individually listed the [SENSITIVE INFORMATION REDACTED] switches that either Verizon or BellSouth activated during the performance period of the agreements, and (2) included statements by the carriers certifying activation of CALEA compliant software on the individually identified switches. However, since the FBI did not conduct verification testing on the switches under either agreement, we spoke with carrier representatives to confirm the number of switches brought into compliance with TCCF funds. These officials told us that certifications provided to the FBI verifying the activation of CALEA software on the [SENSITIVE INFORMATION REDACTED] switches accurately reflected each carrier’s activity under the agreement.
On a “per switch” basis, we calculated that the FBI paid Verizon $[SENSITIVE INFORMATION REDACTED] per switch, while BellSouth recovered an average of $[SENSITIVE INFORMATION REDACTED] per switch. FBI officials told us that the difference in rates was based on [SENSITIVE INFORMATION REDACTED]. We reviewed records of discussions held between the FBI and carriers to assess whether these costs were reasonable. We identified internal FBI communications authorizing payments that stated that the $4.5 million paid compensated carriers for only reasonable CALEA deployment costs. However, the FBI did not provide any evidence based on independent cost data or competing price estimates to support this statement.29 Considering this and the variance in carrier size and equipment, we cannot determine whether the $4.5 million provided to the carriers was a reasonable cost to upgrade [SENSITIVE INFORMATION REDACTED] switches.
To determine the cost effectiveness, or impact, of the total $4.5 million the FBI paid Verizon and BellSouth to deploy their CALEA solutions, we asked carrier representatives how often they used the [SENSITIVE INFORMATION REDACTED] activated switches to capture surveillance results. Carrier representatives told us that they do not track surveillance intercepts by individual switch. As a result, we could not determine whether or how the carriers use their switches to conduct law enforcement surveillance intercepts.
During the audit period, the FBI paid a total of $96,878 for costs incurred by Qwest under four separate agreements finalized before January 2006. Three of the four agreements called on Qwest to test CALEA solutions developed by an equipment manufacturer, while another required Qwest to deploy and test surveillance capability solutions in anticipation of the 2002 Winter Olympic Games.
The FBI entered into three agreements with Nortel Networks Corporation (Nortel) to develop and include various CALEA solutions in its switch software. Under these agreements, Qwest worked with Nortel to test the developed capabilities and the FBI agreed to pay Qwest for planning, installing, deploying, testing, and retesting the solutions on its switches. At the conclusion of its dial-out testing, Qwest invoiced its costs under the agreements and received payments from the TCCF totaling $63,495.
In conjunction with the 2002 Winter Olympic Games, the FBI entered into an agreement with Qwest to ensure that 29 designated network switches in and around Salt Lake City, Utah, had the CALEA capabilities required for court-ordered electronic surveillance. The agreement also called on the FBI and Qwest to develop a technical plan that used a delivery network that carried court-ordered surveillance information from Qwest network switches to law enforcement agencies. In 2003, the FBI modified the agreement to allow Qwest to recover $33,383 in costs resulting from testing software on 29 designated switches.30
The FBI did not perform a formal technical review of the manufacturer agreements because Nortel did not provide cost data showing actual expenses incurred by developing CALEA solutions.31 As a result, the FBI relied on various Determination of Findings it compiled to justify the reasonableness of CALEA solution costs. In one such document, the FBI’s Operational Technology Division and the FBI’s Finance Division stated that the telecommunication industry’s reluctance to offer specific cost information stems from closely-guarded business practices. For example, telecommunication industry non-disclosure agreements often restrict parties from publicly releasing pricing data. Nonetheless, the FBI concluded that costs associated with certain CALEA solutions would be reasonable since they would result in long-term financial savings to law enforcement agencies.
According to records provided by OSCU, the CIU specifically reviewed Qwest testing costs associated with the 2002 Winter Olympic Games. However, CIU’s cost review did not consider independent cost data or competing price estimates. Therefore, we cannot offer an opinion on the reasonableness or cost effectiveness of the total $96,878 received by Qwest.
In light of the $40 million in TCCF rescissions that occurred in 2007, our audit also reviewed how the FBI has continued to work with telecommunication providers to help ensure that emerging communication technologies are CALEA compliant. Our audit found that, during the reporting period, the FBI has developed tools and resources to help facilitate and measure CALEA compliance. In addition, the FBI has hosted and attended forums and other types of meetings with law enforcement personnel, developed and updated the AskCALEA website, conducted and issued annual threat assessment surveys, and surveyed telecommunication providers regarding the status of CALEA solutions on their networks.32
To assist in its monitoring efforts, the FBI joined DOJ in its Joint Petition for Expedited Rulemaking, which asked the FCC to adopt a phase-in plan requiring carriers to provide information regarding their CALEA compliance status by certain dates.33 In its Second Report and Order, the FCC declined DOJ’s request but developed a form (Form 445) for certain service providers to use in reporting their extent of CALEA compliance to the FCC by the compliance deadline.34 Using information provided to it by law enforcement and voluntarily reported by telecommunication providers, the FBI has conducted CALEA solution tests with individual providers and trusted third parties. However, due to the rapid public emergence of packet-mode technology, such as Voice over Internet Protocol (VoIP) and broadband services, the FBI is focusing on establishing electronic surveillance solutions for these new and emerging technologies.35
One of the ways the FBI has measured the impact of CALEA is through periodic meetings, including the FBI-sponsored Law Enforcement Technical Forum, the Law Enforcement Executive Forum, and various non-FBI sponsored law enforcement meetings. FBI officials stated that these meetings give law enforcement officials opportunities to discuss problems encountered while requesting or receiving electronic surveillance.
In addition, the FBI has established the Carrier Relations Working Group (CRWG) and the Electronic Surveillance Working Group (ESWG) to address law enforcement wiretap issues. Both working groups are comprised of members of the Law Enforcement Technical Forum. The CRWG works with providers and federal, state, and local law enforcement agencies to minimize the cost of electronic surveillance, while the ESWG focuses on assisting law enforcement surveillance efforts in the face of rapidly emerging telecommunication technologies.
The FBI prepares and issues Threat Assessment Survey Reports each year from responses received from National Technical Investigator Association meetings and various training sessions hosted by federal law enforcement agencies. As shown by Table 7, the 2005 and 2006 surveys detailed various areas affecting law enforcement’s ability to conduct and receive electronic surveillance.
TABLE 7: SELECTED 2005 AND 2006 THREAT ASSESSMENT SURVEY RESULTS
|Survey Response||2005 Survey||2006 Survey|
|Top emerging technologies||[SENSITIVE INFORMATION REDACTED]||[SENSITIVE INFORMATION REDACTED]|
|Attempted surveillance on VoIP communications? (Yes/No)|
| Attempted surveillance on broadband?
| Cost of surveillance has limited intercepts performed?
|Surveillance results are not provided in a usable format? (Agree/Disagree)|
| Have investigations been hindered by provider non-compliance with CALEA?
The 2005 and 2006 surveys showed that various law enforcement agencies perceive the same types of technology – [SENSITIVE INFORMATION REDACTED] – as the “hot items” most in need of CALEA solutions. The 2006 survey revealed a four-fold increase, [SENSITIVE INFORMATION REDACTED], in law enforcement agencies conducting electronic surveillance with a VoIP provider. While the 2006 survey responses demonstrated improvement in the way providers report surveillance results, [SENSITIVE INFORMATION REDACTED] of respondents stated that investigations continue to be hindered because telecommunication providers were not CALEA compliant. According to these responses, some of the problems stemmed from a lack of provider CALEA solutions. In the end, however, law enforcement representatives reported that their greatest concern regarding electronic surveillance remained its high cost.
In December 2005, the FBI conducted a survey of telecommunications providers to gauge their CALEA compliance status.36 According to the FBI, 258 carriers responded to the survey and reported the CALEA solution status on a total of 3,858 switches.37 As shown in Figure A, providers reported that 1,530 or nearly 40 percent of their switches could not be used to produce CALEA-compliant electronic surveillance results.
Figure A also shows that telecommunication providers indicated that they did not install manufacturer updates containing the software required to effectuate a CALEA-compliant wiretap on 169 switches. Considering the total switches that had CALEA software solutions installed, providers have not yet modified or adjusted their networks to activate the software on 151 switches. Meanwhile, of the 3,538 switches that have CALEA solutions both installed and activated, the survey reported that 1,210 switches are being administered by carrier personnel that have not been trained to use the intercept software or otherwise make CALEA functionality available to law enforcement.
Considering the results of these surveys, we asked the FBI what it has done, in addition to the carrier deployment agreements, to help ensure deployment of CALEA solutions on provider networks. The FBI stated that since it can only work to enforce CALEA compliance when alerted to surveillance inadequacies by law enforcement, it has revamped its AskCALEA website to assist, in a user-friendly way, law enforcement officers and carrier personnel with surveillance issues. The website’s AskCALEA Help Desk (Help Desk) maintains a law enforcement-restricted database that details various wiretap issues and CIU remedies, as shown in Figure B.
User names and passwords provided by the Help Desk allow law enforcement personnel to search database fields to find a solution to their surveillance issue. If the inquiring law enforcement user has a new or previously undocumented surveillance problem, they can use the database to submit solution requests to the Help Desk. The CIU told us that it regularly monitors Help Desk activities and carefully reviews new requests in developing additional CALEA solutions.
The FBI also participates in several domestic and international technology standard-setting groups to: (1) encourage carriers to meet their CALEA responsibilities, and (2) promote effective liaison functions with the telecommunications industry. As a member of these groups, the FBI informs provider and manufacturer representatives about CALEA assistance capability requirements and responsibilities, especially concerning emerging technologies. Table 8 lists the various standard-setting groups with which the FBI participates.
TABLE 8: TECHNOLOGY STANDARD-SETTING GROUPS
Participants in these standard-setting groups have a vote in the decision-making process when developing new technological standards. Since the FBI has only one vote within each group, FBI officials told us that manufacturers, engineers, and private industry representatives can easily overrule their attempts to advocate guidelines that ensure newly established standards comply with CALEA prior to public release.
As a result of the FBI’s inability under CALEA to dictate specific standards in these standard-setting groups, the FBI told us that some new technologies lack adequate CALEA solutions. As a result, the FBI is concentrating its efforts on working with and testing VoIP, broadband, and other packet-mode based communication providers to develop and deploy CALEA solutions for their unique technologies. To perform CALEA solution tests more efficiently, the CIU has reorganized its in-house Solutions Verification Team (Solutions Team). During the reorganization, the Solutions Team hired additional engineers and acquired new equipment that automated testing tasks. According to CIU officials, the reorganization, staff hires, and equipment purchases have allowed the CIU to conduct many more CALEA solution tests per year.
Test Subject Selection
CALEA does not require telecommunication providers, vendors, or trusted third parties to work with the FBI to ensure their networks and systems provide compliant surveillance results. Nevertheless, FBI officials told us that they have reached out to various companies that have indicated a willingness to cooperate and work with its CIU to perform CALEA solution testing. To determine which companies are willing to work with them, FBI officials maintain a list of potentially cooperative test subjects that include companies that responded to FBI or FCC surveys or contacted the AskCALEA Help Desk for wiretap assistance. In addition, officials with the Solutions Team told us they considered other variables, as shown in Table 9, when deciding whether to approach, schedule, and test the CALEA capabilities of a certain provider or vendor.
TABLE 9: ELEMENTS CONSIDERED FOR
SOLUTIONS VERIFICATION TESTING
|1. Test Subject footprint. Regarding providers, the FBI looks at whether the provider services a large number of subscribers. For manufacturers, the FBI focuses on whether a large number of providers use the equipment.|
|2. Solution availability. According to the CIU, resources are focused on technological standards where it is feasible to use a developed CALEA solution or develop a new CALEA solution.|
|3. Known vulnerabilities. CIU officials told us that they prioritize work with providers or manufacturers to solve CALEA compliance issues that: (1) they are aware of from their work with the standard-setting groups, and (2) have reportedly affected prior or current lawful electronic surveillance.|
|4. Test Subject availability. In light of the FCC’s reluctance to mandate that carriers report on CALEA compliance activity, the FBI has to rely on providers or manufacturers that want or are otherwise willing to work with the FBI to develop or implement CALEA solutions.|
Once the FBI selects a test subject, the Solutions Team begins developing a test plan. In developing the plan, the Solutions Team reviews the technology standard against the tested feature specifications while identifying the particular surveillance need identified by law enforcement. Once drafted, the Solutions Team sends the test plan to the provider or manufacturer and once the test plan is agreed upon, the Solutions Team begins the test.
Solutions Team Test Results
From 2005 to 2007, the Solutions Team tested CALEA capabilities at 12 different providers that used emerging technologies. According to test results, the Solutions Team uncovered deficiencies in various proposed CALEA solutions. We obtained and analyzed CALEA solution problems revealed by the Solutions Team tests. Of the 50 deficiencies reviewed, we found that their impact on law enforcement could be categorized in four different ways: (1) incomplete surveillance data; (2) inaccurate surveillance data; (3) unusable surveillance data; and (4) other miscellaneous impacts, as shown in Table 10.
TABLE 10: SUMMARY OF SOLUTIONS TEAM
|Type of Issue||Test A||Test B||Test C||Test D|
|Other Miscellaneous Issues||3||0||2||3|
Incomplete Surveillance Data. The CALEA solution weaknesses that had the most detrimental impact to law enforcement were those that caused incomplete surveillance data. As identified during our analysis, incomplete surveillance was caused by a variety of different issues with the CALEA solutions. For example, one test revealed that the implemented CALEA solution was not intercepting all communications, and therefore surveillance results did not contain complete information. Another test found that the solution required that the subject’s modem and computer be active before the intercept could begin. In such cases, equipment needed to be active on the subject’s end before the FBI could initiate a court-ordered intercept.
Inaccurate or Unusable Surveillance Data. The tests also identified CALEA solutions that could result in inaccurate or unusable information. For example, in one tested case law enforcement agencies reported receiving different data on the same target. Another test revealed that a CALEA solution resulted in hard-to-read surveillance reports.
At the conclusion of a test, the Solutions Team compiles a list of issues and sends the list to the company test subject. The company test subject and the Solutions Team then work to develop solutions to the identified problems. Once these solutions are developed, the CIU compiles a Quick Reference Guide describing the issue and its solution. These guides are posted to the AskCALEA website to assist law enforcement personnel conducting electronic surveillance in a given telecommunication environment. During our audit, we found that the FBI has developed several Quick Reference Guides that are posted to the AskCALEA website.
Of the nearly $4.6 million spent between January 2006 and December 2007, about $4.5 million was paid to two carriers to deploy CALEA-related solutions. The FBI also paid $96,878 to a carrier for testing CALEA solutions on its telecommunication network. We could not assess the reasonableness or cost effectiveness of these expenditures because the FBI did not base its costs on independent cost data or competing price estimates.
Our audit also reviewed how the FBI has continued to work with telecommunication providers to help ensure that emerging communication technologies are CALEA compliant. We found that the FBI has revamped its testing group and enhanced its resources to help measure and facilitate CALEA compliance. In addition, the FBI has implemented an extensive testing program to ensure carrier compliance and capability with regard to emerging technologies.
At the end of the audit period, only $5,037 remained in the TCCF. According to a DOJ finance official, the FBI is working with DOJ to transfer the remaining funds to the DOJ Working Capital Fund and close the TCCF. Since the OIG is tracking residual TCCF funds by a recommendation made in a prior OIG report, this report makes no additional recommendations and is issued closed.38
We provided a draft of the report to the FBI for comment and review. Since the report made no recommendations, the FBI did not offer a response.
FBI officials added that discussions held with Qwest and SBC did not result in formalized reimbursement agreements. According to the FBI, Qwest and SBC deployed CALEA solutions on their networks without TCCF reimbursement
According to the agreement, the FBI could reimburse a carrier for costs either via a cost reimbursement method or a firm-fixed price method. FBI officials indicated that they preferred the firm-fixed price method for this agreement because the carrier would only receive reimbursement once it certified that it had upgraded switches under the agreement. An FBI official also told us that the firm-fixed price method is more efficient than a cost reimbursement method because the firm-fixed price method does not require the FBI to perform additional financial reviews on incurred costs.
As stated previously, the FBI audited the proposed Verizon deployment costs to assess their fairness and reasonableness. The FBI audit found that certain proposed costs were fair and reasonable. However, the FBI audit did not consider independent cost data or competing price estimates. As a result, we could not use the results of the FBI audit in our determining of whether carrier deployment costs were reasonable or cost effective.
In FY 2002, the FBI also paid Qwest $2.2 million from the TCCF for developing and implementing CALEA capabilities in the Salt Lake City area. See U.S. Department of Justice Office of the Inspector General, The Implementation of the Communications Assistance for Law Enforcement Act by the Federal Bureau of Investigation, Audit Report 04-19 (April 2004), 10.
Although the FBI requested underlying developmental costs associated with various CALEA solutions from each manufacturer, manufacturers were unable or unwilling to provide such cost information to the FBI.
In its First Report and Order, the FCC ruled that providers of Voice over Internet Protocol (VoIP) and broadband services must comply with CALEA. On May 12, 2006, the FCC issued its Second Report and Order and reaffirmed that providers of VoIP and broadband services must be CALEA compliant by May 14, 2007.
FBI officials told us that they also received copies of Forms 445 submitted to the FCC by various telecommunication providers and used, in part, results reported on these forms to set testing priorities.
The FBI estimates that there are a total about 20,000 network switches in the United States. Total survey responses reported on the CALEA-compliance status of 3,858 network switches. Therefore, the survey reported on about 19 percent of the total estimated number of network switches in the United States.