FEDERAL COMMUNICATIONS COMMISSION
Washington, D.C. 20554
UNITED STATES DEPARTMENT OF JUSTICE
FEDERAL COMMUNICATIONS COMMISSION
Washington, D.C. 20554
UNITED STATES DEPARTMENT OF JUSTICE
Introduction and Summary
The United States Department of Justice ("the Department"), pursuant to Section 271(d)(2)(A) of the Telecommunications Act of 1996(1) ("the 1996 Act"), submits this evaluation of the joint application filed by BellSouth Corporation, BellSouth Telecommunications, Inc., and BellSouth Long Distance, Inc. on February 14, 2002, to provide in-region, interLATA services in Georgia and Louisiana ("Supplemental Application").
BellSouth's Supplemental Application to the Federal Communications Commission ("FCC" or "Commission") is its second for the state of Georgia and its fourth for the state of Louisiana. BellSouth filed its initial joint application for long distance authority in Georgia and Louisiana in October 2001 ("Initial Application"). On November 6, 2001, the Department submitted to the FCC its evaluation of the Initial Application ("Georgia/Louisiana I Evaluation"), which focused primarily on the adequacy of BellSouth's operational support systems ("OSS") and the reliability of BellSouth's performance data.(2) Based on concerns raised in these areas, the Department concluded that it could not fully support BellSouth's Initial Application.(3) BellSouth subsequently withdrew its Initial Application from the Commission on December 20, 2001.
As the Department has explained, in-region, interLATA entry by a regionalBell Operating Company ("BOC") should be permitted only when the local markets in a state have been "fully and irreversibly" opened to competition.(4) This standard seeks to measure whether the barriers to competition that Congress sought to eliminate with the 1996 Act have in fact been fully eliminated and whether there are objective criteria to ensure that competitive local exchange carriers ("CLECs") will continue to have nondiscriminatory access to the facilities and services they will need from the BOC in order to enter and compete in the local exchange market. In applying its standard, the Department considers whether all three entry paths contemplated by the 1996 Act -- facilities-based entry involving the construction of new networks, the use of the unbundled elements of the BOC's network ("UNEs"), and resale of the BOC's services -- are fully and irreversibly open to competitive entry to serve both business and residential customers.
This Evaluation focuses on changes in BellSouth's OSS and progress in the auditing of BellSouth's performance metrics since the filing of the Department's Georgia/Louisiana I Evaluation. The Department recognizes that additional improvements in BellSouth's OSS have been identified and will be implemented under the direction of the Georgia and Louisiana Public Service Commissions ("PSCs"), and that final completion of the metrics audit under the auspices of the Georgia PSC should further improve the accuracy and reliability of BellSouth's performance reports. Based on this record, and subject to the Commission's review of the concerns expressed in this Evaluation, the Department recommends that the FCC approve BellSouth's application.
Both the Georgia and Louisiana PSCs have filed comments supporting BellSouth's Supplemental Application. During the past several months, the Georgia PSC has continued to work actively to create competitive conditions in the state, in part by overseeing workshops and conferences pertaining to performance measurement and enforcement, and by conducting an evaluation of the change management process.(5) Likewise, the Louisiana PSChas continued to address BellSouth's service quality performance measures and associated self-executing penalties, and conduct technical and collaborative workshops on operational issues.(6)
BellSouth asserts that its OSS are regional and, therefore, as the Department has previously explained, it is appropriate to take into consideration the OSS testing conducted by KPMG at the direction of the Florida PSC.(7) This testing has continued since the Department's Georgia/Louisiana I Evaluation was filed and, although incomplete, it has generated some results that are relevant to this application.
In assessing whether the local markets in a state are fully and irreversibly open to competition, the Department looks first to the actual entry in a market.(8) But the Department does not broadly presume that all three entry tracks -- facilities-based, unbundled network elements ("UNEs"), and resale -- are open or closed on the basis of an aggregate level of entry alone.(9)
Based on current data on CLEC entry in Georgia and Louisiana, the Department finds no basis to address issues beyond those discussed in its Georgia/Louisiana I Evaluation. The Department has not changed its positive assessment of the openness of markets in Georgia or Louisiana for the facilities-based and resale modes of entry.
In its Georgia/Louisiana I Evaluation, the Department extended the presumption of openness to the fully-facilities-based mode of competition in Georgia based on the entry recorded there.(10) Although the Department did not extend such a presumption to other modes of entry,(11) the lack of complaints from CLECs reselling BellSouth's voice services in Georgia led to a conclusion that the resale mode of entry was open to competitors.(12)
BellSouth's Supplemental Application reports that CLECs in Georgia serve approximately 913,000 lines, or approximately 18.8 percent of the total lines in BellSouth's service area, using all modes of entry, as of December 2001.(13) As to the use of UNEs, BellSouth's Supplemental Application includes evidence that employment of the UNE-platform has increased recently. According to BellSouth data, the share of all lines in BellSouth's service area served by CLECs via the UNE-platform has increased from 3.9 percent to 4.7 percent since its Initial Application.(14) CLECs' use of the UNE-platform to serve residential customers has grown from 110,927 lines, or 3.8 percent of residential lines in BellSouth's service area, to 136,306 lines, or 4.8 percent, and to serve business customers from 79,146 lines, or 4 percent of business lines in BellSouth's service area, to 91,405 lines, or 4.6 percent.(15) This growth may reflect, in part, the improvements in BellSouth's OSS since the filing of its Initial Application, as described below.
In its Georgia/Louisiana I Evaluation, the Department found that the amount of entry in Louisiana did not entitle BellSouth to a presumption of openness as to any mode of entry.(16) However,based on the absence of complaints from CLECs providing service entirely over their own facilities, the Department concluded that the Louisiana market was open to fully facilities-based competition.(17) The Department reached the same conclusion with respect to resale entry.(18) However, with respect to entry by means of UNEs, the Department indicated that BellSouth had yet to demonstrate adequate performance in serving CLECs in Louisiana.(19)
BellSouth estimates that competitors in Louisiana serve approximately 227,000 lines, or approximately 8.9 percent of the total lines in BellSouth's service area, using all modes of entry, as of December 2001.(20) Facilities-based competition for residential customers has been strengthened by the entry of Cox Louisiana Telecom, a cable telephony provider.(21) Furthermore, use of the UNE-platform has increased recently in Louisiana. According to BellSouth data, the share of all lines in BellSouth's service area served by CLECs via the UNE-platform has increased from 0.7 percent to 1.2 percent since its Initial Application.(22) CLECs' use of the UNE-platform to serve residential customers has grown from 3,119 lines, or 0.2 percent of residential lines in BellSouth's service area, to 5,145 lines, or 0.3 percent, and to serve business customers from 15,745 lines, or 1.8 percent of business lines in BellSouth's service area, to24,995 lines, or 2.9 percent.(23) Although the growth in entry via the UNE-platform in Louisiana may reflect, in part, improvements in BellSouth's OSS since the filing of its Initial Application, as described below, entry via the UNE-platform in Louisiana is still minimal.
Since the filing of the Department's Initial Evaluation a number of developments have occurred that bear upon the concerns raised in that Evaluation regarding BellSouth's OSS.
The Department's Georgia/Louisiana I Evaluation stressed the importance, for competitors using UNEs, of efficient OSS that would facilitate the automated handling of orders by the CLECs and BellSouth. Since that Evaluation was filed, several important developments have occurred that should facilitate the use of UNEs. As described more extensively below, BellSouth has (1) implemented telephone number migration ("TN migration") for UNE-platform orders, (2) altered its pre-order systems to offer a more fully parsed customer service record, (3) changed its OSS so as to reduce rejects based on a double database edit to CLEC orders, and (4) improved the electronic ordering capabilities for digital subscriber line ("DSL") competitors. Although the Department is encouraged by these developments, it notes that in implementing these changes to its OSS BellSouth appears to have violated important change management principles and that this has resulted in exceptions being opened by KPMG in the Florida OSS test.(24)
The Department's Georgia/Louisiana I Evaluation noted that BellSouth did not offer CLECs the option of ordering service based on a subscriber's telephone number (as opposed to a subscriber's complete address), a capability known as TN migration.(25) This feature is provided by other RBOCs.(26) TN migration reduces the possibility of conflicts between the addresses in an RBOC's database and the addresses provided in CLEC orders. In response to filings by CLECs, the Georgia PSC ordered BellSouth to provide TN migration by November 3, 2001, in its order recommending approval of BellSouth's Section 271 application.(27) The Georgia PSC recognized that not having to enter the customer's address "will facilitate mass-market competitive entry by reducing address-related errors."(28) The introduction of TN migration represents an important step in upgrading BellSouth's OSS.
The procedures by which BellSouth implemented this change and other software upgrades, as discussed below, however, raise questions about BellSouth's consistency in adhering to proper change management principles.(29) Because of inadequate provision for CLEC testing, problems that could cause a substantial increase in the number of rejects were discovered the day before the initial release was implemented.(30) A single day's notice that orders should be submitted differently than BellSouth had previously indicated is not nearly sufficient to permit CLECs to modify their ordering systems. BellSouth, however, did promptly act to correct the problems and since mid-November 2001 CLECs have been able to make full use of TN migration.(31)
InDecember and January there were more than 325,000 UNE-platform requests submitted region-wide by numerous CLECs.(32) As a result of the TN migration upgrade "the overall reject rate for UNE-P migration requests dropped over 35% from October 2001 to January 2002 [and] address related errors for these same requests have been reduced by over 60% during this same time period."(33) WorldCom, the leading proponent of TN migration, acknowledges that its reject rate decreased by approximately 10 percentage points.(34)
In addition to implementing TN migration, BellSouth made a software change on February 2, 2002, whereby it ceased performing address checks on two separate, and sometimes inconsistent, address databases,(35) also known as a "double edit."This change should further reduce BellSouth's rejects of CLEC orders.
In response to the Georgia PSC's order requiring BellSouth to provide "fully fielded parsed CSRs by January 5, 2002," BellSouth implemented the pre-order functionality by which customer service records are parsed on its side of the interface.(36) The Department believes that this improvement may facilitate competitive entry by lowering the operating costs of the new entrant. It should also further reduce the costs associated with mistakes that are made when CLEC service representatives are required to retype pre-order information on orders to be sent to BellSouth.
However, this improvement, like that pertaining to TN migration, was introduced without adequate testing and with defects.(37) Although BellSouth and the Georgia PSC dispute the significance of the defects in this software,(38) KPMG regards them as significant and strongly criticized BellSouth for not completing important internal testing before implementing the release.(39) As discussed more fully below, competitive harm may occur where BellSouth does not provide for smooth transitions between software releases in accordance with appropriate change control principles.
The Department's Georgia/Louisiana I Evaluation expressed concern that the lack of electronic ordering capability for certain order types could impair the ability of DSL providers to compete with BellSouth.(40) In response to a Georgia PSC order, BellSouth deployed electronic ordering for line-splitting on January 5, 2002.(41) BellSouth made electronic ordering of UDC/IDSL loops, another popular product for DSL providers, available on February 2, 2002.(42) After receipt by BellSouth, UDC/IDSL loop orders will continue to be processed manually until May 19, 2002, when BellSouth is scheduled to implement electronic processing, which will allow the orders to flow through to provisioning without manual intervention.(43) In the interim, DSL competitors will be able to use electronic systems for the generation of their orders instead of having to fax them to BellSouth.(44)
Although the introduction of TN migration, the cessation of double database edits for customer addresses, the implementation of parsed CSRs, and the improvements in electronic ordering capabilities for DSL competitors are positive developments that should permit new entrants to operate more efficiently, the changes are so recent that their full effect is not yet reflected in the performance reports. However, BellSouth's CLEC order reject rate has significantly improved since the Department's Georgia/Louisiana I Evaluation was filed, likely due to the introduction of TN migration in November. In particular, the reject rate for UNE-platform orders has fallen from 19.38 percent in September to 14.33 percent in December 2001.(45)
The Department's Georgia/Louisiana I Evaluation noted that BellSouth's performance reports indicated that its service order accuracy was substantially below the established benchmarks and that this increased the burdens on the CLECs, thereby constraining their ability to compete effectively.(46) Since that Evaluation was filed, BellSouth has implemented an intensive program to improve service order accuracy, and the data for December 2001 show it met 95 percent of the performance benchmarks for all 7 of the UNE sub-metrics and 8 of the 11 resale sub-metrics.(47) BellSouth has also agreed to include the service order accuracy measure in its Self-Executing Enforcement Mechanism ("performance enforcement plan").(48)
The Georgia PSC Comments support BellSouth's claim of improved service order accuracy, reporting that, through November 2001, BellSouth had missed the benchmark for 2 of the 3 previous months in only 4 of the 24 service order accuracy sub-metrics, and asserting that BellSouth's December 2001 performance was equally strong.(49) Birch also supports the BellSouth claim of improved service order accuracy, but suggests that these improvements have resulted from extraordinary manual efforts that may not continue once BellSouth obtains Section 271 approval.(50)
BellSouth's ability to sustain its improved performance deserves close examination. The Georgia PSC is considering requiring BellSouth to develop an electronic process to review all partially mechanized CLEC orders so as to obtain a more reliable means of measuring service order accuracy.(51) It also proposes including the revised measure in BellSouth's performance enforcement plan.(52) The Department supports this effort. BellSouth has agreed to make penalty payments under the existing measure until the service order accuracy measure is revised and incorporated in the plan.(53)
The significance of the improvements in BellSouth's service order accuracy is unfortunately obscured, however, by significant changes in how BellSouth has measured accuracy starting with November data.(54) These changes, which were made without prior approval of the Georgia PSC or notice to the CLECs,(55)raise two questions. The first is how to evaluate BellSouth's claim that service order accuracy has improved given such differences in the metrics.(56) In response to the Department's request, BellSouth has provided data showing performance over time using a consistent methodology in order to address whether improvement in service order accuracy actually has occurred.(57) The second question is how the FCC and state commissions can be expected to determine that BellSouth has continued to meet its obligations pursuant to Section 271 if BellSouth can unilaterally change metrics without notice to or input from interested parties. Such a practice substantially diminishes the value of an independent, third-party audit of the performance system. Future changes should be made only with public notice and the concurrence of the state commissions.
The Department's Georgia/Louisiana I Evaluation stressed the importance of a robust test environment so that CLECs may be able to ensure that their software interfaces correctly with BellSouth's OSS, particularly as changes are implemented.(58) The Department raised concerns regarding the degree of separation between BellSouth's test and production environments because certain CLECs claimed that the commingling of test and production orders had impeded their ability to test new software.(59) In addition, the limited availability of the system for testing software changes ("CAVE") was also a factor in constraining CLECs' testing.(60)
The Georgia PSC believes that concerns about the adequacy of the testing environment have been adequately addressed by BellSouth.(61) The Georgia PSC concludes that the CAVE test environment is sufficiently separate from the production environment even though they share the same service order processor, and states that a number of CLECs other than WorldCom have successfully used CAVE without having their test orders and production orders mixed up.(62) The Georgia PSC also reports that the concerns expressed in the Department's Georgia/Louisiana I Evaluation regarding the inability of CLECs to test EDI orders for DSL loops or to test the LENS interface have been resolved.(63)
In the period since BellSouth's Initial Application, CLECs have used the CAVE system on numerous occasions and, according to BellSouth, have made no further claims that test or production orders have been misdirected.(64) Unfortunately, CAVE's limited availability continued to create problems since it could not be used for testing the introduction of TN migration,(65) although it was available and was used by CLECs for testing the introduction of "parsed CSRs."(66) The scheduled availability of the CAVE system, however, has been substantially improved for the balance of this year and that should facilitate its effective use by the CLECs.(67)
There are two aspects to the change management process. The first relates to the procedures governing the introduction of new software, including notice of changes to be made by BellSouth in the OSS interface, exchange of specifications necessary for recoding, and adequate opportunity for both internal and CLEC testing of the proposed release. The second component of change management is the process by which proposed changes necessary to improve the incumbent's OSS are selected for development.
The Department notes that there are outstanding KPMG exceptions covering both of these areas in the Florida OSS test.(68) The Department's Georgia/Louisiana I Evaluation expressed concern that the change management process did not prompt efficient fixes for known defects or system enhancements desired by CLECs.(69) As discussed above, BellSouth's recent introductions of new OSS features providing for TN migration and parsed CSRs were marked by its failure to provide required information to CLECs regarding the effects of the upcoming changes and, apparently, by its failure to adequately perform internal testing.(70)
BellSouth and the Georgia PSC report a number of positive developments that have occurred with regard to the change management process.(71) The Georgia PSC is conducting a comprehensive, two-phase examination of the change control process ("CCP").(72) The first phase involves considering additional measures to monitor BellSouth's compliance with CCP procedures.(73) In this process, three measures were proposed by the industry and agreed to by the Georgia PSC staff: (1) a measure of the timeliness of correction of BellSouth software defects; (2) the percentage of change requests accepted or rejected within 10 business days; and (3) the percentage of change requests rejected for each reason allowed by the CCP.(74) The Georgia PSC asserts that these measures will help assure that BellSouth corrects defects and handles change requests promptly and efficiently.(75)
The second phase of the examination covers changes to the CCP itself, and the Georgia PSC is now reviewing CLECs' proposals and BellSouth's response.(76) BellSouth is supporting a number of the modifications proposed by the CLECs and has made its own proposals to address CLEC concerns regarding the scope of the CCP, the time required to implement changes, and the adequacy of the prioritization process.(77) The Georgia PSC concludes that BellSouth is making a good faith effort to improve the processes and correct the deficiencies that the Department identified in its Georgia/Louisiana I Evaluation, and notes that BellSouth has begun distributing a complete schedule for release implementation this year, as well as a report outlining the status of the "'top 15' change requests prioritized by CLECs" it is committed to implementing in 2002.(78)
BellSouth also proposed to the Georgia PSC a process by which 40 percent of its annual release capacity would be allocated for implementing CLEC change requests and/or CLEC-driven regulatory mandates.(79) The CLECs are advocating an approach that would require implementation of all their requested changes within a certain interval.(80) Resolving this issue will be difficult and may require the Georgia PSC to balance important, conflicting interests.
The Department is encouraged by the positive steps taken in the area of change management and expects BellSouth to comply with whatever approach is chosen. The record in this application supports approval, but in subsequent applications the Department will continue to review closely BellSouth's OSS and change management performance.
While commending the efforts of the state commissions to develop robust sets of performance measures, the Department's Georgia/Louisiana I Evaluation expressed concern regarding the reliability of BellSouth's performance data.(81) The Department discussed a number of problems affecting the accuracy of BellSouth's data, and noted that frequent restatements of data made it difficult to conclude that BellSouth's performance metrics were accurate.(82)
BellSouth now maintains that the stability of its performance reporting systems has improved greatly as demonstrated by the fact that it restated no performance reports between September and December 2001.(83) The Department is encouraged that performance reports are being restated less frequently and agrees that this could indicate that the current systems are becoming more stable.(84) In and of itself, however, this reduced number of restatements is not proof that the underlying problems that led to the former pattern of restatements have been resolved or that the current data are accurate.(85)
Also, BellSouth relies heavily on KPMG's Revised Interim Status Report ("KPMG Status Report") on the Georgia metrics audit as support for the accuracy of its data.(86) The KPMG Report describes those parts of the audit that are complete, the status of open issues, and the items that remain to be tested. The KPMG Status Report is useful in that it provides a more complete picture of the progress that has been made toward completing the Georgia metrics audit, but it offers no conclusions regarding the reliability of BellSouth's metrics because it was not issued for that purpose. Based on its own analysis of the open issues identified in the KPMG Status Report, BellSouth maintains that they are minor enough to suggest that the items yet to be audited will "close" successfully and should not be a cause for concern.(87) As in its Initial Application, BellSouth again argues that the three phases of the audit together provide ample grounds for concluding that its metrics are accurate.(88)
Although the number of tests KPMG reports as having been completed or deemed "satisfied" is not insignificant,(89) the audit is not complete.(90) The limited number of outstanding issues from past phases of the Georgia audit does suggest that inaccuracies discovered in the audit are being addressed and resolved,(91) but the number of open exceptions and unsatisfied test criteria in past phases do not provide a basis for predicting that other significant issues will not be discovered during the third phase. Moreover, BellSouth has made many systems changes since the first two phases of the audit which affect the majority of its metrics.(92)
On balance, the progress on the audit, coupled with the Georgia PSC's recent judgment that the data are reliable in light of its close monitoring of the audit,(93) suggests that the stability and accuracy of BellSouth's performance data are improving. However, until the Georgia metrics audit is complete or until there is additional commercial experience with the reported metrics, the Commission should not rely solely on BellSouth's performance reports in reviewing otherwise credible complaints that BellSouth is not meeting the requirements of the Act.
BellSouth's Supplemental Application demonstrates that it has made substantial progress in addressing issues previously identified by the Department. The Department recognizes that additional improvements in BellSouth's OSS have been identified and will be implemented under the direction of the Georgia and Louisiana PSCs, and that final completion of the metrics audit under the auspices of the Georgia PSC should further improve the accuracy and reliability of BellSouth's performance reports. Based on this record, and subject to the Commission's review of the concerns expressed in this Evaluation, the Department recommends that the FCC approve BellSouth's application.
March 21, 2002
I hereby certify that I have caused a true and accurate copy of the foregoing Evaluation of the United States Department of Justice to be served on the persons indicated on the attached service list by first class mail, overnight mail, hand delivery, or electronic mail on March 21, 2002.
1. Pub. L. No. 104-104, 110 Stat. 56 (1996) (codified as amended in scattered sections of 47 U.S.C.).
2. DOJ Georgia/Louisiana I Evaluation at 13-38. This Evaluation incorporates by reference the entire DOJ Georgia/Louisiana I Evaluation.
3. Id. at 38-39.
4. See DOJ Oklahoma I Evaluation at vi-vii, 36-51.
5. Georgia PSC Comments at 2-3.
6. Louisiana PSC Comments at 2-3.
7. DOJ Georgia/Louisiana I Evaluation at 6-7.
8. See DOJ Pennsylvania Evaluation at 3-4 ("The Department first looks to actual competitive entry, because the experience of competitors seeking to enter a market can provide highly probative evidence about the presence or absence of artificial barriers to entry. Of course, entry barriers can differ by types of customers or geographic areas within a state, so the Department looks for evidence relevant to each market in a state." (footnote omitted)).
9. See, e.g., DOJ Georgia/Louisiana I Evaluation at 7 ("Although the Department presumes that fully facilities-based competition is not hindered in a competitively significant manner based on the entry recorded in Georgia, the amount of entry does not justify extending such a presumption to other modes of entry in Georgia."); DOJ Missouri I Evaluation at 6-7 ("The Department presumes that opportunities to serve business customers by fully facilities-based carriers and resellers are available in Missouri, based on the entry efforts reflected in SBC's application. There is significantly less competition to serve residential customers. There also is less competition by firms seeking to use UNEs, including the UNE-platform, and there are some indications that a failure by SBC to satisfy all of its obligations may have constrained this type of competition." (footnotes omitted)).
10. DOJ Georgia/Louisiana I Evaluation at 7.
12. Id. at 10-11.
13. BellSouth Stockdale Aff. at 6 tbl.2; id. ¶ 9.
14. See DOJ Georgia/Louisiana I Evaluation at 9; BellSouth Stockdale Aff. at 6 tbl.2, Tab 6 at 1.
15. See DOJ Georgia/Louisiana I Evaluation at 8 tbl.1, 9; BellSouth Stockdale Aff. at 6 tbl.2, Tab 6 at 1.
16. DOJ Georgia/Louisiana I Evaluation at 7.
17. Id. at 12.
18. Id. at 13.
19. Id. at 12.
20. BellSouth Stockdale Aff. at 9 tbl.4; id. ¶ 12.
21. Louisiana PSC Comments at 2.
22. See DOJ Georgia/Louisiana I Evaluation at 12; BellSouth Stockdale Aff. at 9 tbl.4, Tab 8 at 1.
23. See DOJ Georgia/Louisiana I Evaluation at 11 tbl.2, 12; BellSouth Stockdale Aff. at 9 tbl.4, Tab 8 at 1.
24. KPMG FL OSS Test Exception 155 at 1-2; KPMG FL OSS Test Exception 157 at 1-2.
25. DOJ Georgia/Louisiana I Evaluation at 23-25.
26. See, e.g., FCC Texas Order ¶ 178 (SBC offers TN migration).
27. Georgia PSC Comments at 5.
28. Id. at 5-6.
29. See, e.g., WorldCom Comments at 13-16; WorldCom Lichtenberg Decl. ¶¶ 135-39.
30. See WorldCom Comments at 14; WorldCom Lichtenberg Decl. ¶ 136;see also Georgia PSC Comments at 6. On November 2, 2001, the day before the release, BellSouth had to notify CLECs to continue to include customer addresses until the software could be fixed. Georgia PSC Comments at 6. Because BellSouth was late in implementing this capability it paid $140,000 in fines to the State of Georgia. Id. at 7 n.6.
31. Id. at 7.
33. Id. at 8.
34. Id. (citing WorldCom affidavit filed in Georgia PSC proceeding).
35. Id. at 9.
36. Id. at 12. AT&T has delayed testing of the CSR functionality until the defects acknowledged by BellSouth are corrected, but several others have tested this software in CAVE. AT&TBradbury/Norris Decl. ¶ 17; BellSouth Stacy/Varner/Ainsworth Aff. ¶¶ 60-66 (describing testing by Birch and by Telcordia and Exceleron, two software vendors representing multiple CLECs).
37. See KPMG FL OSS Test Exception 157 at 1-2; see also AT&T Bradbury/Norris Decl. ¶¶ 16-42; WorldCom Lichtenberg Decl. ¶¶ 138-41.
38. BellSouth Br. at 21-22; BellSouth Stacy/Varner/Ainsworth Aff. ¶¶ 67-78; Georgia PSC Comments at 12-16.
39. KPMG FL OSS Test Exception 157 at 1-2 ("BellSouth internal release documentation for the above releases makes mention of the lack of sufficient time for appropriate pre-release testing within the release schedule. While the issue was noted, there was no apparent plan to mitigate the adverse impact of reduced pre-release testing. . . . BellSouth's incomplete internal software testing may affect a CLEC's ability to efficiently execute transactions with BellSouth, resulting in CLEC customer dissatisfaction.").
40. DOJ Georgia/Louisiana I Evaluation at 16 & n.40.
41. Georgia PSC Comments at 17.
43. Id.; BellSouth Stacy/Varner/Ainsworth Aff. ¶¶ 192-94.
44. Georgia PSC Comments at 17; BellSouth Stacy/Varner/Ainsworth Aff. ¶¶ 192-94. BellSouth still requires manual ordering of the UCL-ND product and of both ADSL loops and line sharing when conditioning is required. BellSouth Covad Issues Ex Parte at 2-3; see also Covad Comments at 4-8.
45. BellSouth Stacy/Varner/Ainsworth Aff. ¶ 106.
46. DOJ Georgia/Louisiana I Evaluation at 17-21.
47. BellSouth Br. at 25-26; BellSouth Stacy/Varner/Ainsworth Aff. ¶¶ 159-60.
48. BellSouth Br. at 26.
49. Georgia PSC Comments at 18-19.
50. Birch Comments at 7-9.
51. Georgia PSC Comments at 19.
54. See BellSouth Service Order Accuracy Ex Parte at 5-7 (describing substantive changes made to its service order accuracy measure); see also BellSouth Varner Aff. ¶¶ 63-67 (same).
55. BellSouth explained publicly the changes made to its service order accuracy measure well after the fact and appears to have provided a written explanation of the changes only when directed to do so by the Georgia PSC staff. See Georgia PSC Comments at 19 n.17; see also BellSouth Service Order Accuracy Ex Parte at 21.
56. In addition to changing the sampling methodology, BellSouth modified the reporting from a local service request ("LSR") to a service order ("SO") basis and added a large number of missing product categories (including UNE-platform, 2-wire analog loops, stand-alone LNP, UNE xDSL, and line sharing). BellSouth Service Order Accuracy Ex Parte at 3-7, 21. The change to reporting on an SO basis tends to increase reported accuracy rates for the same set of SOs, e.g., if a CLEC submits two LSRs, each of which results in two SOs, an error on one of the four SOs would yield a report of accurately handling 50 percent of the LSRs (1 of 2) but 75 percent of the SOs (3 of 4). The addition of previously omitted product types also complicates a comparison, e.g., the addition of UNE-platform tends to increase aggregate UNE performance due to the extent of their mechanization. See id. at 12 ("[UNE-P is] the largest category of fully mechanized orders[.]").
57. See BellSouth Service Order Accuracy Ex Parte at 9-11. BellSouth has provided data restating the service order accuracy metrics for September and October 2001 using its current methodology. Id.at 9. These results indicate that for those months, performance is substantially better under the new method than the performance BellSouth had reported pursuant to the old method. Id. The Georgia PSC has found that the changes in methodology were appropriate in that they have brought BellSouth's reporting into closer conformity with the Commission's Service Quality Measurement ("SQM") plan. Georgia PSC Comments at 19 n.17. The Department does not maintain that the current measure is faulty, but is concerned rather that BellSouth made significant changes without appropriate disclosure.
58. DOJ Georgia/Louisiana I Evaluation at 26-28.
59. Id. at 26-28.
60. Id. at 28.
61. Georgia PSC Comments 24.
62. Id. at 24-25.
63. Id. at 25 (noting BellSouth's deployment of Release 10.3 (December 2001) and Release 10.4 (March 2002)).
64. See BellSouth Stacy/Varner/Ainsworth Aff. ¶¶ 138-41.
65. WorldCom Georgia/Louisiana I Reply Comments at 4; see also BellSouth Stacy GA/LA I Reply Aff. ¶ 124.
66. BellSouth Stacy/Varner/Ainsworth Aff. ¶¶ 59-65.
67. See id. 143.
68. KPMG FL OSS Test Exception 88 (change control process does not allow CLEC involvement in prioritization of CLEC-impacting requests); KPMG FL OSS Test Exception 155 (BellSouth failure to provide information for minor releases in accord with change control process intervals); KPMG FL OSS Test Exception 157 (BellSouth failure to follow internal testing and quality processes).
69. DOJ Georgia/Louisiana I Evaluation at 29.
70. See supra notes 29-31, 37-39 and accompanying text.
71. BellSouth Br. at 27-31; Georgia PSC Comments at 26-28.
72. Georgia PSC Comments at 26.
77. Id. at 26-27.
78. Id. at 27.
80. Id. at 27-28; see, e.g.,WorldCom Comments at 22 (asserting that BellSouth should implement all prioritized changes within 60 weeks). BellSouth subsequently made a counterproposal. See BellSouth Stacy/Varner/Ainsworth Aff. ¶ 133.
81. See DOJ Georgia/Louisiana I Evaluation at 30-38.
82. Id. at 32-35. Although it did not know the source of all these metrics problems, the Department observed that many appeared to result from errors in software written to implement newly ordered and changed Georgia performance measures. Id. at 32. As a result, some relevant data were excluded from some measures; others included data that were being processed incorrectly. Id. at 32.
83. BellSouth Varner Aff. ¶ 25.
84. BellSouth does, however, appear to exercise some discretion in restating data, as indicated by its decision not to re-post when "the impact [of errors] is de minimus." See id. ¶ 62. Further, data is re-posted only when corrected data can be generated. See id. ¶ 20.
Moreover, the current systems are being changed. BellSouth is in the process of upgrading its Performance Measurement and Analysis Platform ("PMAP") from Version 2.6 to Version 4.0. KPMG FL OSS Test Meeting Minutes at 7-8. BellSouth maintains that this change will have minimal impact, at least in terms of the inputs and outputs, but because the upgrade includes system changes and extensive software re-coding, new data accuracy issues could arise. See id.
85. Two CLECs have provided evidence that data were re-posted to the PMAP web-site at least once during the period that BellSouth cites. AT&T Bursh/Norris Decl. ¶ 16 & Attach. 1; Network Tel. Comments at 2, Exs. 1, 2. Both claim the restated data were incorrect. AT&T Bursh/Norris Decl. ¶ 16; Network Tel. Comments at 3. Furthermore, AT&T, questioning whether the lack of data restatements is proof of a lack of problems, cites a number of apparent discrepancies that it believes should have resulted in restatements. AT&T Bursh/Norris Decl. ¶¶ 18-23.
86. BellSouth Br. at 34-36; see also KPMG Revised Interim Status Report; KPMG Interim Status Report.
87. BellSouth Varner Aff. ¶ 34.
88. BellSouth GA/LA I Br. at 24; BellSouth Br. at 34-36.
89. See KPMG Revised Interim Status Report at 1, 3 (indicating that the tests that comprise the first two phases of audit were largely complete); id. at 9, 11 (indicating that significant portions of several test segments in third phase of audit were complete).
90. See, e.g., id. at 12, 14, 17 (describing progress on specific audit components).
91. The metrics audit is a "military-style" test in which testing (and, if necessary, re-testing) continues until a criterion has been satisfied.
92. See, e.g., KPMG Revised Interim Status Report Audit III PMR 4 Data Integrity Status Summary at 1-3 (identifying new measures and measures with new disaggregations); see also supra notes 54-57 and accompanying text.
93. Georgia PSC Comments at 28-31.