UNITED STATES DEPARTMENT OF JUSTICE
February 14, 2000
INDEX OF FULL CITATIONS
UNITED STATES DEPARTMENT OF JUSTICE
SBC has filed an application for long distance entry in Texas that shows substantial progress in the development of local competition in that state. In sharp contrast to its earlier application for Oklahoma, which failed to meet even the threshold requirements of section 271,(1) this application demonstrates that the ground rules for competition have been largely established, that SBC has implemented most of the operational details of providing access and interconnection to its competitors and that numerous carriers are beginning to compete in offering a wide range of services. This progress reflects well on SBC, which has devoted substantial resources towards implementation of the requirements of the Telecommunications Act,(2) and on the Texas Public Utility Commission ("Texas PUC") and its staff, which has worked tirelessly to create an environment in which competition may emerge.
This Evaluation by the Department of Justice principally focuses on SBC's actual commercial performance in providing access and interconnection. In a great many respects, despite some continuing start-up problems, that performance appears to be adequate. In the critical area of providing unbundled loops for advanced services, however, SBC's application is clearly deficient. In this proceeding, the Federal Communications Commission ("Commission") must decide concretely how it will interpret and apply the requirements of section 271 to provide access to such unbundled loops. It is very important for the Commission to ensure that SBC satisfies those requirements, for a failure to do so will seriously retard the deployment of such services and competition in their provision. SBC has not demonstrated that it is providing non-discriminatory treatment to competitors offering xDSL services, or that its planned (but not implemented) use of a separate affiliate to provide such services will address this shortcoming. The Commission should deny SBC's application because of its deficiencies in this area.
There are other shortcomings in this application. In its recent decision approving Bell Atlantic's section 271 application for New York, the Commission found Bell Atlantic's performance in providing "hot cuts" of unbundled loops to be "minimally acceptable." As best we can determine, SBC's performance in this area falls short of that "minimally acceptable" level. Because of SBC's deficient performance, carriers seeking to use unbundled loops are constrained, and the market is not fully open to competition.
Finally, the record leaves considerable doubt about whether SBC can provide interconnection trunks in a timely manner, and whether carriers will be able to compete effectively using the UNE-platform. A careful analysis of additional commercial experience, through the entirety of the current quarter, will provide valuable evidence of whether recently implemented measures have successfully resolved problems in these areas, and whether SBC's wholesale support systems will function adequately as the volume of CLEC activity increases. Because of the limitations of time and information, and because of the critical need to protect the fairness and efficacy of the Commission's process for reviewing section 271 applications, a review of this additional experience should not be attempted in connection with the current application. But since this application should be denied in any event because of the deficiencies relating to DSL and hot cut loops, we recommend that the Commission defer judgment on interconnection and UNE-platform issues until a subsequent re-application, when it will have the benefit of evidence reflecting additional commercial experience.
The Texas PUC has shown a great commitment to open the Texas market to local competition. Through a consolidated set of arbitrations known as the "mega-arbitration" and the subsequent nearly two-year 271 review process, the Texas PUC and its staff worked carefully and extensively to define the terms, conditions and operational details necessary for the development of competition in the state of Texas. Recognizing that the appropriate offering terms are not a sufficient demonstration of market openness, the Texas PUC also contracted with Telcordia Technologies, Inc. to oversee a carrier-to-carrier readiness test of SBC's operational support systems, and the Texas PUC staff reviewed the reported commercial performance results.(3)
While providing some evidence of the functionality and capacity of SBC's operational support systems ("OSS"), Telcordia's test has significant limitations.(4) First, the Telcordia Final Report as written does not clearly identify all major problems encountered during the test, assess the impact of these problems on CLECs, or investigate the root causes of these problems. Second, the carrier-to-carrier style test used in Texas was limited primarily to the system capabilities of the two main test participants, AT&T and MCI WorldCom. Thus, unlike the test in New York, the Telcordia test was not broad enough to test the wholesale support processes for other CLECs with different target markets and business plans.(5) Third, Telcordia focused narrowly on SBC's computer systems and not on SBC's wholesale support systems generally. Consequently, Telcordia's test does not provide evidence that SBC provides adequate wholesale services overall to CLECs in Texas.
Finally, and perhaps most importantly with respect to future applications, uncertainty remains regarding the validity of some of SBC's performance reports because Telcordia reviewed only a subset of the performance measures on which SBC reports. In evaluating the actual commercial experience of SBC's competitors, the Department and the Commission place great weight on the reported performance data; the reliability of the reported data is critical. To properly validate metrics, one must verify that they are meaningful, accurate and reproducible. Meaningful metrics require clear definitions that will allow measurement of activities or processes in a way that has real-world, practical significance. Accurate metrics are faithful to established definitions in that they are correctly calculated from the proper subset of raw data using processes that ensure the data are accurately handled and transferred. Reproducible metrics can be reproduced at future dates for verification purposes because the raw data have been archived for an appropriate period in a secure, auditable form and because changes to the systems and processes used for gathering and reporting metrics are carefully controlled and fully documented.(6)
Telcordia's review does not provide an adequate basis for determining that presently reported SBC performance data are reliable. First, Telcordia did not examine whether the metrics as defined are meaningful. Second, its review is too dated and limited to ensure the accuracy of current data.(7) Third, Telcordia's reviews, both in conjunction with the OSS test Final Report and its subsequent Performance Measures Data Control Integrity Analysis, identified concerns relating to the manual handling of performance data, the integrity of data files and the auditability of performance data.(8) Thus, we find that Telcordia's reviews do not assure the accuracy and reliability of SBC's performance data.
The third-party test that was part of the record in Bell Atlantic's New York application was broad, independent and robust and played a valuable role in opening that market to competition. In comparison, the Telcordia test in Texas was far less comprehensive, blind and independent, and therefore provides much less persuasive evidence.(9)
As the Department has previously stated, in-region, interLATA entry by a Bell Operating Company ("BOC") should be permitted only when the local market in a state has been fully and irreversibly opened to competition.(10) In applying this standard, the Department examines whether all three entry paths contemplated by the 1996 Telecommunications Act -- facilities-based entry involving construction of new networks, the use of unbundled elements of the BOC's network and resale of the BOC's services -- are fully and irreversibly open to competitive entry to serve both business and residential customers. To do so, the Department looks first to actual competitive entry. The actual experience of competitors seeking to enter a market can provide highly probative evidence concerning the presence, or absence, of artificial barriers to entry. The degree to which such existing competition is broad-based determines the weight the Department places on it as evidence. For any entry path where competitively significant entry is reasonably foreseeable but broad-based commercial entry is still absent, the Department examines whether new technical and operational arrangements are available and shown to be working to support the entry mode and whether benchmarks to prevent backsliding by the incumbent have been established.(11)
In its application, SBC contends that CLECs serve 1,408,558 lines, or 12.8 percent of the total number of access lines in SBC's Texas service area.(12) We presume that SBC's estimates of the number of resale and UNE access lines (both UNE-loop and UNE-platform) are accurate, since SBC, in its role as a wholesaler, should possess reliable information about CLEC activity in these areas. However, SBC appears to have substantially over-estimated the number of lines served by facilities-based carriers, about which it has no direct evidence.(13) While SBC estimates that CLECs had 920,140 full facilities-based lines in service in November 1999,(14) we believe that a more reasonable estimate is 350,000-400,000.(15) Using our estimate, CLECs have a total of approximately 840,000-890,000 access lines, about eight percent of SBC's Texas market. A CLEC market share of eight percent of local services is significantly above the national average of five percent,(16) but the level of CLEC penetration in Texas appears to be somewhat less than that seen in New York at the time of Bell Atlantic's application, where CLECs collectively had achieved approximately an 8.9 percent market share.(17)
A substantial amount of CLEC entry in Texas is primarily by full facilities-based providers serving urban, business customers.(18) CLECs use UNE-loops and the UNE-platform to serve a small, but competitively important number of business customers. Residential customers have historically been served through resale. Many residential customers are now receiving UNE-platform based service. The record in this application does not raise concerns about competitive entry for CLECs choosing to resell SBC's retail services, but the more limited amount of competition using other modes of entry requires us to closely examine the market conditions that affect those other modes of entry.
As we explain below, the Department has concluded that markets for local services in Texas are not fully and irreversibly open to competition by carriers seeking to offer advanced services using unbundled xDSL-capable loops, or by carriers seeking to offer services using unbundled voice-grade loops. The record also raises significant doubts about the ability of facilities-based competitors to obtain adequate access to interconnection trunks, and about the ability of other carriers to compete effectively using the UNE-platform.
In the recent New York Order, the Commission stated that applicants for 271 approval should "make a separate and comprehensive evidentiary showing with respect to the provision of xDSL-capable loops . . . ."(19) The Department emphatically agrees. Such a separate showing is vitally important because DSL technology, using copper wire loops that are ubiquitous throughout the telephone network, offers one of the best prospects for broadband services. In the New York Order, the Commission set forth two methods that an applicant may use to show that it provides DSL loops to CLECs in a nondiscriminatory manner. First, the Commission indicated that the establishment of a "fully operational"(20) separate affiliate for advanced services "may provide significant evidence" of nondiscrimination.(21) Second, the Commission stated that 271 applicants could demonstrate that they are providing nondiscriminatory access to DSL-capable loops through comprehensive and accurate reports of performance measures even without a separate affiliate.(22)
SBC states that the record in this proceeding contains sufficient evidence to satisfy the Commission's separate affiliate alternative as well as performance data to prove that it is providing nondiscriminatory treatment of CLEC orders for DSL-capable loops.(24) We disagree and find that SBC cannot satisfy either requirement. SBC's performance data are fundamentally flawed in some cases, and in other cases reveal significant discrimination. SBC fails to show that its proposal to offer DSL services through a separate affiliate will be adequate to prevent a continuation of this discrimination.
As a critical threshold matter, it appears that key portions of the DSL performance data that SBC has submitted to the Commission are seriously flawed. In some cases, SBC has acknowledged important facts that preclude reliance on these performance data. In other cases, serious questions about the reliability of the data have been raised by commenters. In the context of the known inadequacies in the data, we believe these questions should be resolved before placing any reliance on the challenged data.
Performance Measure ("PM") 57 is a measure of the time within which SBC provides loop make-up (or "qualification") information in response to CLECs' requests for such information. SBC has been on notice at least since the publication of the Commission's order approving the merger of Ameritech and SBC of the appropriate way to measure these response times: "The time starts when a request is received by the CLEC [sic, ILEC] and ends when the information on the loop qualification has been made available to the CLEC."(25) Notwithstanding this clear, common sense definition, the data submitted by SBC to the Texas PUC and to the Commission measure only the time SBC's representative worked on the request, and exclude both the period of time the request was with SBC before the representative started working on it and the period of time the response remained with SBC after the representative's work was completed. Excluding these time periods will obviously understate the length of time it actually takes SBC to respond to CLEC requests for loop qualification information.(26) Although the Texas PUC stated that SBC had been ordered to fix this problem and that SBC had implemented the change, the Texas PUC apparently relied upon the faulty data when it reported in its comments that PM 57 shows SBC's statewide performance for September through November exceeded the parity requirement.(27) SBC has acknowledged to the Department that its PM 57 performance data through December, which were submitted to the Commission on February 1, 2000, also fail to capture these two important time periods.(28) Given this flaw in the data, they cannot be relied upon to show parity in this dimension of performance.
In response to the Department's questions concerning Covad's complaint that SBC had excluded most of its orders from Covad-specific reports for PM 5 ("Percent Firm Order Confirmations (FOCs) Received Within "X" Hours") and PM 6 ("Average Time to Return FOC (Hours)"), which measure SBC performance regarding FOC timeliness,(29) SBC has acknowledged that it excluded DSL FOCs from its performance measures. SBC asserted that "[t]he business rule for FOC was developed prior to August 1999, and did not contemplate DSL"(30) though in fact, the business rule does not provide for any exclusion of DSL loops.(31)
The actual experience reported by the CLECs in their comments suggests that delays in receiving FOCs for DSL orders are lengthy and that such delays impair CLECs' ability to compete.(32) Clearly, the data on FOC timeliness submitted with this application cannot be relied upon to support a conclusion that SBC is supplying DSL competitors with timely order confirmations.
Another incident that casts doubt on the reliability of SBC's performance data relates to data for PM 62 ("Average Delay Days for SWBT Caused Missed Due Dates"), which the Texas PUC has designated as a Tier 1 and 2 performance measure.(33) In reports submitted to this Commission and the Texas PUC as recently as February 1, 2000, SBC reported that the average number of "delay days" was "0.00" for September through December 1999.(34) This statistic might have led one to infer that there was little competitive significance in SBC's failure to install a DSL-capable loop at the time promised on the FOC because the average reported delay in installing such loops was insignificant. After SBC was asked to demonstrate how the reported result was mathematically possible, it filed a correction, discussed below, which shows a significant disparity between average delays for CLECs and average delays for SBC's retail operations in November and December.(35)
The performance measure for the average installation interval for DSL loops (PM 55.1) is one of the most competitively significant indicators of a DSL provider's ability to compete with SBC. SBC's reported data should not be relied upon to show nondiscrimination, absent a complete examination of the accuracy of the reported data.
The data for PM 55.1 that SBC submitted with its application failed to reflect data for a majority of the loops provisioned by SBC. After NorthPoint pointed out that none of its DSL loops had been tracked in PM 55.1, even though it had submitted more than 900 orders for DSL loops,(36) SBC acknowledged that a processing error had resulted in a substantial portion of DSL loops being reported under a DSL resale measure.(37) SBC submitted corrected data to the Commission on February 1, 2000.(38) Unfortunately, SBC has not yet issued corrected reports for individual CLECs, which are necessary for them to reconcile disputed data with SBC. Although the corrected performance data show slightly better performance by SBC, there is still reason to question the reliability of the data reported in PM 55.1, for two reasons.
First, Covad claims that SBC is improperly excluding almost half of its orders from PM 55.1, strongly disputing SBC's contention that it has properly excluded the orders because Covad requested due dates outside the available interval.(39) A resolution of this dispute should be relatively straightforward if SBC identifies to Covad (and other CLECs) the particular orders that it claims to have excluded because of out-of-interval due dates. SBC has not done so to date.
Second, the measurement is clouded by disputes over whether SBC has actually installed a working loop at the point in time when it claims to have done so. For example, if the CLEC claims the loop needed additional work, its data would show a longer installation interval than SBC's data. Agreement on the time that loops are installed can best be determined by joint-acceptance testing of loop installations. Rhythms' comments, however, state that until very recently, SBC had refused to engage in acceptance testing for DSL loops, although ordered by the Texas PUC to do so.(40) It may be necessary to wait until performance data that reflect acceptance testing are available before SBC's report on PM 55.1 can be considered reliable.
Even if the deficiencies in SBC's performance data are limited to the specific examples discussed above, those deficiencies substantially undermine SBC's claim to have provided nondiscriminatory access to DSL loops, since the deficient performance data relate to several competitively important dimensions of SBC's DSL performance. We are concerned, moreover, that those deficiencies may be symptomatic of more serious problems in the reliability of SBC's performance measurement systems and processes. The Department relies heavily on these
performance measures as an objective indicator of the quality of SBC's performance, and we regard it as critical that applicants insure the utmost reliability of such data.
SBC's DSL competitors rely principally on two types of unbundled loops: the DSL loop and the BRI ISDN loop.(41) SBC's application fails to demonstrate acceptable performance in processing orders for DSL and BRI (ISDN) loops because, as described above, its performance data regarding the provision of loop make-up information to CLECs are fundamentally flawed, and because they fail entirely to report on the return of firm order commitments for DSL. With respect to provisioning and maintenance and repair, SBC's performance reports clearly indicate discrimination with respect to both types of loops for carriers offering DSL services.(42)
Even with the limited data available, the performance reports demonstrate that SBC
is failing to provide nondiscriminatory performance, as indicated by several performance measures that the Texas PUC has deemed competition affecting.
PM 58. One of the most significant measures for DSL provisioning is PM 58 ("Percent SWBT Caused Missed Due Dates"), a high Tier 1 and 2 measure.(43) The measure shows a rate of 12.1 percent missed due dates for CLECs in December (on a total of 495 loops), compared to 6.3 percent missed due dates for SBC's retail service. This result is especially troubling because the number of missed due dates has steadily increased over the last three months as the number of CLEC orders has increased -- although the total number of loops is still far below expected commercial volumes.(44)
PM 60. PM 60 ("Percent Missed Due Dates Due To Lack of Facilities [>30 days]") measures the percentage of loops where SBC missed a committed due date by more than 30 days and attributed the miss to an absence of available facilities.(45) SBC missed 6.7 percent of due dates for CLECs (on a total of 495 loops) compared to only 0.6 percent for SBC's circuits.(46)
PM 62. PM 62 ("Average Delay Days for SWBT Missed Due Dates") measures the calendar days between the scheduled due date and the actual completion date.(47) SBC's corrected data for this measure show that SBC missed 60 due dates for December, with an average number of delay days of 6.25 for the CLECs and 4.06 for SBC.(48)
PM 55.1. PM 55.1 ("Average Installation Interval DSL") is a high Tier 1 and 2 measure of the time from completion of loop qualification to the completion of the service order.(49) For December, the average interval for conditioned loops was 14.23 days for CLECs compared to 11.50 days for SBC. This difference was not statistically significant, but, as noted previously, there are serious questions about the accuracy of the data for PM 55.1 arising from the large number of excluded orders. The results reported for December represented a significant improvement over the results for November which were 19.30 days for CLECs and 11.50 days for SBC.(50)
PM 59. PM 59 ("Percent [Trouble] Reports" ) is a high Tier 1 and 2 measure that tracks the percentage of trouble reports for newly installed DSL loops.(51) SBC tracked 398 DSL loops used by CLECs for December, which had a 15.8 percent rate for trouble reports, compared to a trouble report rate of only 5.2 percent for SBC. This performance deteriorated badly as volumes increased in December.(52)
PM 65. PM 65 ('Trouble Report Rate [For DSL Loops]") is a high Tier 1 and 2 measure of the monthly repair rate for all installed DSL loops.(53) In December 1999, SBC reported 75 trouble reports on 974 CLEC circuits, a 7.7 percent rate compared to a 4.6 percent rate for SBC's retail lines. This measure also shows decreasing performance over time as volumes rise.(54)
SBC's performance reports for BRI loops also demonstrate substantial discrimination and, as in the case of DSL loops, in several cases display a clear trend of declining performance as volumes have increased.
PM 56. PM 56 ("Percent [BRI Loop] Installations within "X" Days") is a high Tier 1 and Tier 2 measure of installation intervals for BRI loops that excludes missed due dates caused by the CLECs or their end-user customers requesting installation intervals longer than the defined standard interval. CLEC orders of one to ten BRI loops are supposed to be installed in three days.(55) For these BRI loops, SBC's on time performance has dropped from 84.6 percent in October to 71.6 percent in December.(56)
PM 58. PM 58 ("Percent SWBT Caused Missed Due Dates [for BRI Loops]"), a high Tier 1 and 2 measure, reports the percent of installations not completed on the due date.(57) For BRI loops in December, SBC reports that it was responsible for missed due dates on 23.3 percent of CLEC circuits compared to 15.5 percent for its own customers. SBC's performance on this measure also deteriorated significantly over the past several months.(58)
PM 60. PM 60 ("Percent Missed Due Dates Due To Lack of Facilities") records the percentage of loops for which SBC does not have facilities after giving the CLEC a committed due date.(59) Out of 374 circuits tracked in December, SBC missed 12 percent of CLEC due dates (45) compared to only 0.8 percent for its own customers. This problem worsened significantly as the number of circuits increased in December.(60)
PM 59. PM 59 ("Percent [Installation Trouble] Reports") is a high Tier 1 and 2 measure that tracks the percentage of trouble reports for newly installed loops.(61) This measure tracked 444 CLEC BRI loops for December which had a 20.9 percent trouble rate (93 trouble reports) compared to a trouble rate of 5.1 percent for SBC's retail lines. This performance metric appears to show deteriorating performance as volumes have increased over the last three months with the reported rate doubling between October and December 1999.(62)
PM 65. PM 65 ("Trouble Report Rate [For BRI Loops]") is a high Tier 1 and 2 measure of trouble reports for all installed loops received within the month.(63) For December, SBC reported a 19.3 percent trouble rate (116 troubles) for CLEC BRI loops, compared to a 2.5 percent rate for SBC BRI loops. Like other measures noted above, this problem worsened as volumes have increased.(64)
PM 67. PM 67 ("Mean Time to Restore (hours) -Dispatch") is a high Tier 1 and 2 measure.(65) For December the mean time to restore service for CLEC lines was 7.81 hours, compared to 5.84 hours for SBC's retail BRI lines. Performance on this measure also appears to be steadily deteriorating as volumes increase.(66)
Taken as a whole, these performance reports show a service environment in which CLECs attempting to compete against SBC's retail DSL services are seriously disadvantaged at present by SBC's inadequate wholesale performance, and may well face greater disadvantages in the future if SBC's performance continues to decline in the face of higher volumes of CLEC orders.(67)
The Commission's New York Order stated that a future 271 applicant would have the option of demonstrating nondiscriminatory treatment of DSL competitors by implementing a fully operational, separate affiliate for advanced services.(68) This application provides the first opportunity for the Commission to determine the specific requirements that will be needed to prevent discrimination against DSL competitors when a BOC chooses this alternative. It is critically important for the Commission to establish two sets of requirements in this context,(69) without which a separate affiliate structure cannot rationally be said to prevent discrimination.(70)
First, the Commission should rigorously examine the relationship that will exist between the BOC and its affiliate to ensure that the affiliate's relationship to the BOC is the same in all relevant respects as the relationship between CLECs and the BOC. SBC provides virtually no information about these matters, and thus on its face cannot be deemed to provide satisfactory evidence of nondiscrimination.
Second, whatever the ostensible relationship between SBC and its affiliate, there must be adequate mechanisms to detect, punish and deter any discrimination that may occur. An essential component of this mechanism must be meaningful, accurate and reproducible performance measures, as we have previously discussed. Absent the ability to detect discrimination through such measures, SBC could blatantly discriminate in favor of its own retail operations, whether those operations are conducted by a separate affiliate or not. As we have previously explained, SBC's performance data with respect to DSL services have serious flaws, and therefore cannot be relied upon as an effective mechanism to detect discrimination.
Apart from the two elements described above, which the Commission should require in connection with any assessment of a separate affiliate proposal, an applicant that has failed to provide nondiscriminatory treatment prior to the establishment of a separate affiliate should be required to demonstrate that the implementation of the separate affiliate structure has in fact resulted in nondiscriminatory performance. We believe it would be quite unrealistic to expect a BOC and its affiliate to be unmindful of their mutual economic interests, and quite difficult to foreclose all of the avenues by which they might seek illegitimate marketplace advantages over their competitors. A concrete demonstration of nondiscrimination, rather than a abstract promise, will provide much greater assurance that the separate affiliate structure is in fact sufficient to prevent discrimination. Such a demonstration should be a simple matter if the separate affiliate is "fully operational" and if meaningful, reliable and reproducible performance measures are in place.
SBC's affiliate, Advanced Solutions Inc. (ASI), is not yet fully operational. SBC has not demonstrated that it has actually provided nondiscriminatory performance, or even explained why the separate affiliate structure should be expected to remedy the documented current performance problems. We presume that SBC would not implement a separate affiliate that would degrade the quality of performance for its own retail operations. But unless it does so, the current discrimination could be ended only by improving the quality of performance provided to the CLECs. SBC's application offers no explanation of how this would be accomplished merely by transferring its own retail operations to the new affiliate.
In sum, we conclude that SBC has failed to establish that it is providing, or will provide, nondiscriminatory access to unbundled loops for DSL services.
The use of unbundled loops is an important component of CLECs' efforts to provide service to small and medium-sized business customers. SBC's application fails to demonstrate adequate performance in providing unbundled loops. As best we can determine, SBC's performance with regard to "hot cuts" is worse than Bell Atlantic's performance in New York, which the Commission concluded was "minimally acceptable."
SBC uses two hot cut processes in Texas. SBC states that the fully coordinated hot cut ("CHC") process is to be used for conversions of orders of 20 or more lines, or those which must be accomplished outside normal business hours.(71) CHC orders are manually processed in SBC's order processing center and require intensive coordination and communication between SBC and the CLEC during the performance of the hot cut. SBC states that the frame due time ("FDT") hot cut process is to be used for cuts of fewer than 20 lines that are taking place between 8 a.m. and 5 p.m., Monday through Friday.(72) FDT orders are capable of flowing through SBC's order processing center without manual work by SBC's representatives. FDT cuts require both the CLEC and SBC to do the necessary work at pre-arranged times, but no communication between them is required at the time of the hot cut.(73)
SBC has encouraged, if not required, CLECs to switch from CHC to FDT for smaller volume loop cuts.(74) SBC has expressed the view that CHC is too resource-intensive to support commercial levels of demand for these lower-loop-volume orders and that transition to FDT would alleviate CHC capacity constraints.(75) SBC may charge a premium if CLECs select the intensively manual CHC process.(76) In light of this, the use of CHC appears to be rapidly declining, while the use of FDT appears to be rapidly increasing. In November 1999, SBC provisioned 2375 CHC loops and 653 FDT loops.(77) In December 1999, SBC provisioned 1284 CHC loops and 1666 FDT loops.(78)
SBC's application provides performance data for CHC hot cuts, but not for FDT hot cuts, and the Texas PUC's analysis of SBC's hot cut performance appears to be based solely on CHC performance.(79) As we explain below, the currently available CHC performance data indicate inadequate provisioning and perhaps other problems. After its initial application, SBC submitted limited data on FDT performance, but those data fail to address the most serious concerns relating to FDT performance, and therefore fail to show that SBC is providing satisfactory performance.
The most apparent problems, both for CHC and FDT, arise in connection with SBC's provisioning of hot cuts. Other potential problems relate to SBC's processing of UNE-loop orders -- specifically its ability to provide timely order confirmations and rejections -- and post-provisioning failures that may lead to double billing and dropped directory listings. These problems appear to be increasing as the volume of UNE-loop orders rises.
In its recent New York Order, the Commission found that Bell Atlantic had provided adequate hot cut performance, but emphasized that:
The currently available evidence indicates that SBC's hot cut performance in Texas is not as good as Bell Atlantic's "minimally acceptable" performance in New York. As a result, CLECs are constrained in their ability to enter the Texas market using UNE-loops.(81)
In New York, the Commission concluded that Bell Atlantic completed 90 percent of hot cut orders within the benchmark time, specifically citing Bell Atlantic's completion of orders for fewer than ten loops within one hour.(82) The only directly comparable data provided by SBC are (1) "sample" data(83) which purport to show that from August through October 1999, 85.6 percent of its CHC hot cut orders were completed within one hour,(84) and (2) December 1999 data showing that 86.3 percent of all CHC hot cut loops(85) were completed within one hour, after excluding "misses" for which SBC claimed CLECs were responsible.(86)
In New York, the Commission found that fewer than five percent of Bell Atlantic's hot cuts resulted in service outages. In Texas, a joint SBC/AT&T analysis of CHC data for August, September, and October determined that service outages attributable to SBC performance occurred on 8.2 percent of AT&T's CHC orders,(87) an outage rate nearly twice as high as Bell Atlantic's rate of 4.5 percent in New York.(88)
In New York, the Commission found that fewer than two percent of hot cuts generated trouble reports (i.e., service problems reported after a hot cut had been accepted by a CLEC) within seven days. There are no precisely comparable data from Texas, but the data that are available suggest that SBC's performance is no better, and may be slightly worse, than Bell Atlantic's performance in New York: SBC's reported data indicate that 2.18 percent of CHC hot cuts generated trouble reports within ten days.(89)
As noted above, SBC has provided very little evidence of its performance in providing hot cuts through the FDT process. This failure is cause for concern in light of SBC's policy to strongly encourage the use of the FDT process, and in light of problems with that process that have been documented by commenters.(90)
AT&T's initial trial of the FDT process, in August 1999, resulted in failures on nine of 17 orders; each failure was due to SBC failing to begin the cut while AT&T had ported the number as scheduled.(91) A second trial by AT&T, in November and December, resulted in failures on 20 percent of the orders (7.7 percent of November orders and 33.3 percent of December orders).(92) AT&T's FDT data, which have been reconciled with SBC, show (1) that a significant number of AT&T's FDT orders experienced service outages and (2) that 50 percent of these outages were caused by the SBC technician not starting the cut at the scheduled time. Other commenters report similar experiences with the FDT process.(93) Given that FDT cuts are only done during business hours, problems such as these are immediately and significantly impacting to the typical, small business end-user customer, who loses phone service during the business day.
SBC has provided only limited information on the duration of FDT hot cuts, and no information on the extent of service outages. After excluding delays in completing hot cuts that it attributed to CLECs, SBC reported that 90.8 percent of FDT hot cut loops were completed within the 30-minute period contemplated under this process, and that 93.8 percent of hot cut loops were completed within one hour.(94) Service outages may result, however, from a failure to complete the hot cut by the scheduled time (whether due to a late start or an overlong duration), and from SBC's premature initiation of the hot cut. SBC's FDT data do not address these timeliness issues.(95) SBC also fails to provide any data directly measuring the extent of service outages associated with the FDT process,(96) despite the apparently well-documented complaints about these problems.(97)
In an effort to improve its hot cut provisioning performance, SBC has instituted new procedures and work groups.(98) The effects of these changes, however, have not yet been established, and therefore provide no basis for disregarding the inadequacies of historical performance reflected in the record.
SBC has experienced a disturbing number of problems in processing orders as the volume of orders has increased; there is a significant risk that these problems may become even more acute as UNE-loop orders continue to rise. Similarly, SBC has failed to resolve problems observed in its post-provisioning processing of order information.
Over one-half of the UNE-loop orders which are electronically submitted via the EDI or LEX interfaces are manually processed by SBC's Local Service Center ("LSC").(99) Manual processing may occur at several points in the order process flow.(100) SBC's systems are designed so that some types of electronically submitted orders, such as CHC hot cuts or Related Purchase Orders ("RPONs"),(101) must be manually input into SBC's back-end legacy systems by SBC representatives. In addition, some electronically submitted orders which have errors are either manually rejected by the SBC representatives, or, if the error is not the responsibility of the CLEC, are manually entered into SBC's order processing systems by SBC representatives. The use of manual processing is not in itself an indication of inadequacy in an order processing system, but systems that rely on a high degree of manual processing may experience serious problems with the timeliness and accuracy of order processing and provisioning, as well as scalability of the order processing systems.(102)
Order processing problems are reflected in a number of SBC's performance reports:
Timely Return of Order Confirmations. Performance problems seem to be increasing for each of the three categories of UNE-loop orders.(103) First, for FOCs returned via the EDI interface, SBC argues that in the six-month period from May to October 1999, it satisfied the Texas benchmark for timeliness.(104) But it was in the most recent month of that period, October, that performance fell below the benchmark, and SBC's performance was even worse in November and December, with only 85.8 percent returned within five hours.(105) Second, for FOCs returned via LEX, SBC admits that FOC returns have been chronically late, and states only that it can find no systemic explanation for missing the benchmark.(106) Third, for manually submitted orders, SBC's performance in returning timely confirmations declined from September through November. In December, SBC reported improvement in the percentage of orders returned on time, though the average time for these returns rose precipitously.(107) SBC has attributed these problems to a summer reorganization in its LSC, coupled with a period of increasing CLEC volume, a failure of its LASR GUI and then more retraining of its order processing representatives.(108)
Timely Return of Order Rejections. SBC's time to manually process rejects on electronically submitted orders(109) has been significantly increasing. Fewer and fewer of these rejects are being returned within the benchmark interval of five hours,(110) while the average time for return has increased from 6.86 hours in July to 35.65 hours in December 1999.(111) SBC explicitly attributes this continuing decline in performance to the increasing volume of electronically submitted orders, which has increased the number of orders that entail manual reject processing.(112)
Post-provisioning Order Processing. After a UNE-loop order has been processed and provisioned, additional processing of the order information through SBC's back-end legacy systems is required to generate an official completion notice (Service Order Completion, or "SOC") to the CLEC and then to change the information in SBC's billing systems so that the end-user is recognized as having moved from SBC to the CLEC. This is known as "posting," and delays may occur both before and after the SOC is generated. Timely and accurate order completion notices are important, as they are the means by which the CLEC knows to begin billing the end-user and addressing any maintenance problems experienced by the end-user.(113) SBC, however, has been returning late SOCs for UNE-loop via LEX from May through December 1999.(114) In addition to the late SOCs, UNE-loop CLECs have also experienced late posting of their orders to SBC's billing system, which SBC acknowledges, stating that it has discovered an error in the rate table that caused posting delays in UNE orders.(115) SBC asserts that it has a specialized Error Resolution Team ("ERT") "focused solely on clearing errors on orders that have been completed, but for some reason cannot post for proper billing," and is making "concentrated efforts to reduce the clearing time for errors to post to the bill."(116) Meanwhile, however, UNE-loop CLECs must deal with the apparent manifestations of these posting delays, including double-billing of end-users and the loss of directory listings.(117)
Order processing problems such as these can adversely affect the ability of CLECs to compete effectively. First, lateness or inaccuracy in order status notices requires CLECs to spend extra time and money attempting to confirm (or revise) schedules with SBC and with their customers.(118) Second, problems in processing orders may lead to problems in provisioning, though we lack sufficient information to know whether the order processing problems described here are currently a major factor in the provisioning problems that have been experienced in Texas. If a CHC order is held in error in the LSC, the central office technicians may not get the order in time to perform the pre-cut test procedures that are supposed to minimize the instances of outages upon provisioning. Similarly, if a supplemental order (for instance, a CLEC postponing a due date at the request of the end user) is not promptly processed, SBC may perform an FDT cut prematurely, before the CLEC -- having timely submitted its supplemental order -- has done its necessary work to complete the cutover in service.(119)
Our concern about those problems arises principally because of their appearance at the relatively low volumes of orders that CLECs have generated to date, and because the processing problems appear to be growing with even a modest trend of increasing order volume. These facts, combined with SBC's attribution of the problem to factors associated with manual processing, indicate a risk that the problems may significantly increase in magnitude as manually handled UNE-loop volumes increase.(120) SBC's UNE-loop order processing performance thus appears quite different from Bell Atlantic's performance in New York. There, the FCC found (1) timely return of order confirmations and rejects, (2) accurate processing of manually handled orders, and (3) successfully scaling systems so that performance (although below the benchmark levels set by the New York PSC) had improved even as volumes had risen.(121) Here, SBC's performance in the face of increasing volume is deteriorating.(122)
A CLEC in Texas that wishes to provide local telephone service over its own facilities must interconnect its network with SBC's so that telephone calls can be exchanged between the CLEC's subscribers and SBC's subscribers. The ability to obtain interconnection trunks on a reasonable and timely basis is critically important to facilities-based CLECs as they seek to maintain service quality while their subscriber base, and hence their need for interconnection trunks, increases. The record in this proceeding raises substantial doubt that SBC is providing interconnection trunks in a manner that permits CLECs to compete to their full potential.
Facilities-based CLECs have identified, in Texas PUC proceedings and again in their comments on this application, a variety of serious problems that occurred throughout 1999. These CLECs have indicated that they have been unable to obtain from SBC the number of interconnection trunks they need, in the time frames that they require, despite having followed the required forecasting and network planning processes. They indicate that SBC limited the number of trunks that a CLEC could order; refused to allow CLECs to increase their existing trunk facilities;(123) extended by months the time it takes to process CLEC trunk orders;(124) and missed the installation due dates that were eventually assigned.(125) These limitations and delays appear to have occurred in all areas of the state -- not only in the technologically sophisticated markets of Houston(126) and Dallas/Fort Worth,(127) but also smaller cities such as Amarillo.(128) The CLECs contend that these problems limited their ability to adequately serve their existing customers and to expand their service to new customers. In some cases, these problems have resulted in call blocking on the CLECs' networks,(129) customer complaints and lost or forgone sales.(130)
In this proceeding, SBC has submitted performance data from 1999 which appear largely, although not entirely, compliant with the relevant parity and benchmark standards.(131) These performance reports, however, do not refute the complaints of the CLECs. If a CLEC failed to submit an order because of limitations imposed by SBC, that limitation is not reflected in the performance reports, which capture performance only if an order is placed. Similarly, as the Texas PUC learned in November, 1999, orders that were placed but were not processed due to SBC's lack of facilities are excluded from the reported data.(132) Because of these omissions, the reported data do not provide a reliable indication of SBC's actual performance.(133)
In late 1999, the Texas PUC intervened to address the reported problems associated with interconnection trunking. After learning that SBC was excluding "held orders" from its performance reports, the Texas PUC required SBC to implement a new measure specifically to capture the percentage of held orders greater than 90 days. The new measure, PM 73.1, will be finalized at the April 2000 six-month review, and January data under this measure will first be made available in late February 2000. Texas PUC Commissioners stressed that CLECs should maintain their forecasts and place trunk orders -- regardless of any countering SBC suggestion -- so that the complained-of problems would be clearly captured in the new performance data.(135) SBC agreed to improve its trunk forecasting, data collection, data calculation and ordering processes. SBC has also committed to publish new training materials on February 15, 2000, to provide additional training of its personnel by March 1, 2000, and to mechanize its Trunk Group Service Request process by May 15, 2000 (and retrain its trunk group personnel in the interim).(136)
The Texas Commission has concluded that these changes "should result in parity performance to competitors" that should permit SBC to timely install CLEC trunks.(137) Unfortunately, though, the trunking problems have been difficult to resolve in the past, and there is no assurance that the most recent efforts will succeed.(138) While we are hopeful that the admirable efforts of the Texas PUC will resolve the difficulties of the interconnection forecasting, ordering and provisioning process, SBC should be held to a standard of demonstrated success, rather than optimistic prediction.
Commenters in this proceeding have complained of a variety of performance problems that may affect the ability of CLECs to compete effectively through the use of the UNE-platform.(139) These complaints include weaknesses in the documentation that SBC provides for its EDI interfaces; SBC's failure to adhere to its documented change management process; the absence of a stable testing environment for new interfaces; and concerns about the scalability of SBC's systems that might impede competitors' ability to operate at full volumes for mass-market services. It seems clear that SBC's performance in these areas has been flawed in a number of respects, but it is unclear to the Department at this time how significant those shortcomings have been, or more importantly, the extent to which they may seriously impede competition in the future.
There have also been disturbing allegations of service outages, such as the loss of dial tone, an inability to make outbound calls and the loss of features such as hunt groups or preferred long distance or local toll provider, at the time a customer's service is converted to UNE-platform service. Most of the CLECs using the UNE-platform in Texas affirm that some of their customers suffer from the service problems identified above at the time of conversion to the UNE-platform. Two smaller CLECs, Birch and CapRock describe but do not quantify, the problem.(140) AT&T avers that at current order volumes, three percent of its lines are affected by these service problems, while Network Intelligence, a small CLEC avers that 14 percent of its orders are affected.(141) Birch, which has been experiencing conversion problems since July 1999, did not begin to see any improvement until it filed a complaint with the Texas PUC in September.
With the involvement of the Texas PUC, SBC committed to manually monitor the problem and began weekly meetings with Birch, which are still ongoing, to deal with this and other issues. The problem, however, continues to persist for Birch.(142) The magnitude of the conversion problems may have increased recently; SBC's User Forum minutes from December 1999 state that all CLECs have "experienced a notable increase in these outages during the past several weeks."(143) This time period coincides with rising sales of UNE-platform lines and SBC's decreasing performance on manual rejects of electronically submitted orders.
SBC's process for controlling this problem, as we understand it, is to manually monitor the service orders in its back-end systems to ensure that they complete properly. Hence, there is a risk that if order volumes significantly increase, the scope of these problems may become disproportionately greater. If UNE-platform service providers attract new customers at a rate of one million lines per year, a three percent problem rate (i.e., the rate now reported by AT&T) would generate 30,000 customer problems per year or 2500 problems per month, a rate that may impose heavy demands on SBC's maintenance systems and harm CLECs' reputation in the marketplace.
We regard these service outages as a very serious potential problem, but it is unclear to us at this time the magnitude of the current problem or how likely it is to increase. There is evidence in the record that suggests that these problems are significant and may become more so in the future as order volumes increase, but both that evidence, as well as SBC's response to it, have been inconclusive in our view. Moreover, the record remains unclear as to the precise causes of the outages.
We are also concerned about the extensive amount of manual work required to process current volumes of UNE-platform orders through SBC's order processing center.(144) Current experience in New York shows that at high order volumes, systems problems that increase the manual work in the order processing center make it more difficult to timely confirm or reject manually processed orders.(145)
We believe that additional commercial experience which would be outside the scope of this application may provide valuable evidence that will clarify whether the service outages, and the other concerns noted above, will in fact operate as a serious constraint on competition using the UNE-platform, or whether, as SBC contends, these problems are minor or non-existent. Since we believe that this application should be denied for wholly independent reasons, we recommend that the Commission reserve judgment on these issues for a subsequent re-application.
This application should be denied.
February 14, 2000
CERTIFICATE OF SERVICE
I hereby certify that I have caused a true and accurate copy of the foregoing Evaluation of
the United States Department of Justice to be served on the persons indicated on the attached
service list by first class mail, overnight mail, hand delivery or electronic mail on February 14,
1. 47 U.S.C. § 271 (1996).
2. Pub. L. No. 104-104, 110 Stat. 56 (1996) (codified as amended in various sections of 47 U.S.C.).
3. The best proof that an incumbent local exchange carrier's ("ILEC") wholesale support processes work adequately is, of course, actual commercial use of the systems at substantial (that is to say, reasonably foreseeable open market) volumes. In the absence of such use, however, both the Department and the Commission have said that evidence from carrier-to-carrier testing, independent third-party testing, and internal testing may be used to demonstrate that the wholesale support processes are working and are capable of handling substantial commercial order volumes in the future. See, e.g., DOJ Oklahoma Evaluation at 29-30; DOJ Louisiana II Evaluation at 26-27 & n.50; FCC New York Order ¶ 89.
4. In offering these observations about the scope of the Telcordia test, we do not mean to criticize Telcordia, which oversaw the test in accordance with its instructions.
5. Seven CLECs were test participants in the test, and of these, only two carriers, AT&T and MCI WorldCom, devoted the resources and time necessary to actively participate in the test of the EDI interface. Telcordia Final Report at 1-2; see Transcript of OSS Testing Workshop, In re: Operations Support Testing Relating to the Investigation into Southwestern Bell Telephone Company's Entry into the InterLATA Communications Market in Texas, Public Utility Comm'n of Texas, Proj. No. 20000, at 14 (July 22, 1999), attached to SBC Brief as App. D, Vol. 3, Tab 50. Moreover, Telcordia was unable to conduct an effective test of critical services like DSL due to the limitations of the test carrier. Telcordia Final Report at 76-80. Similarly, in part because of low participation, order volumes for the order types tested were low. Id. at 12.
6. The Department emphasized the importance of these matters in a letter to SBC on March 6, 1998. That letter referenced a list of agreed-upon performance measures for use by the Department in evaluating a section 271 application. The Department emphasized that proper implementation of these measures was critical, noting that "definitional issues and other details connected with the measures themselves (such as the basis upon which due dates and start and stop times are set in particular measures) could significantly affect the meaning of the data." DOJ Performance Measures Letter at Attach. E-1 to E-2. Unfortunately, the defects in SBC's implementation of its performance measures have become apparent only as CLECs have had access to performance reports, and an opportunity to detect inconsistencies between SBC's reports and their own experiences. If SBC had conducted a more careful review of its performance measurements and processes, whether through Telcordia or otherwise, these defects could have been detected and corrected earlier.
7. Most of Telcordia's review focused on calculations made during the later stages of the reporting process on a relatively small subset of SBC's performance measures. In addition, it was based on an outdated version of the definitions of the performance measures. Telcordia Final Report at 151-53. Analysis based on the test data does not overcome these limitations because it was a small amount of data from carefully controlled, generally identifiable orders for a subset of measures and provided no information on critical measures like trunking, billing and number portability.
8. See, e.g., Telcordia Final Report at 145-46; Telcordia Technologies, Performance Measures Data Control Integrity Analysis at 3, 12 (Dec. 1999), attached to SBC Dysart Aff. as Attach. O.; SBC Dysart Aff. ¶ (archiving process to be impoved by May-June 2000).
9. FCC New York Order ¶ 100 ("[W]ere a third party test less comprehensive, less independent, less blind, and, therefore, less useful in assessing the real world impact of a BOC's OSS on competing carriers, we would not necessarily find it persuasive and may accord it less weight than we do the KPMG Final Report.").
10. See DOJ Schwartz Aff. ¶¶ 149-192; DOJ Schwartz Supplemental Aff. ¶¶ 26-60; DOJ Oklahoma Evaluation at vi-vii, 36-51.
11. As we have stated previously, the Department does not regard small market shares held by competitors, or even the absence of entry (either altogether or using a particular entry path), standing alone, as conclusive evidence that a market remains closed to competition or as a basis for denying an application under section 271. See supra note 3.
12. SBC Habeeb Aff., Attach. E at 1; see also DOJ Ex. 1: SBC's Disaggregated Access Line Data (9,624,336 SBC retail lines; 11,032,894 total lines in SBC's serving area).
13. SBC groups UNE and full facilities-based entry together under the heading of "facilities-based" entry. We make the distinction to avoid confusion.
14. SBC's estimate that CLECs had 1,086,407 "facilities-based" lines in November 1999 yields 920,140 full facilities-based lines when UNE-loop and UNE-platform lines are subtracted. See DOJ Ex. 1: SBC's Disaggregated Access Line Data; SBC Habeeb Aff., Attach. E at 1.
15. To estimate the number of full facilities-based lines, SBC multiplied the number of interconnection trunks to each CLEC switch by a factor of 2.75. See SBC Habeeb Aff. ¶ 23. Although we believe it is reasonable to use the number of interconnection trunks in order to estimate the number of CLEC access lines, SBC's factor of 2.75 appears to be much too high. A more reasonable multiplier, in our view, would be close to one; the ratio of CLEC full facilities-based lines to CLEC trunks was .97 in August and 1.04 in September. See Texas PUC Evaluation at 102 (266,734 lines in August and 317,865 lines in September); SBC Oct. 20, 1999 Habeeb Aff. ¶ 34 (274,184 trunks in August); DOJ Ex. 2: SBC's Statement of September CLEC Line Counts (306,071 trunks in early October); see also Allegiance Howland Decl. ¶¶ 4-5 (SBC overstates Allegiance's full facilities-based lines by 123% in Dallas and 190% in Houston.); AT&T Kelley/Turner Aff. ¶ 20. Our estimate uses a factor of one. See SBC Habeeb Aff., Attach. E at 1 (347,830 CLEC interconnection trunks as of December 9, 1999).
16. Federal Communications Commission, Local Competition: August 1999 Report, Press Release (Aug. 31, 1999) (CLECs' "presence remains less than 5% of the local market."), available at <http://www.fcc.gov/Bureaus/Common_Carrier/Reports/FCC-State_Link/ lec.html>, follow link <LCOMP99-1.PDF>.
17. See DOJ New York Evaluation at 9.
18. See Texas PUC Evaluation at 102; SBC Habeeb Aff., Attach. E at 1; SBC Habeeb Confidential Aff. ¶ 28, Table 3.
19. FCC New York Order ¶ 330. The Commission did not require such a showing in New York because of the unique circumstances there. SBC has not argued that such unique circumstances exist in Texas.
21. Id. ¶ 331.
22. Id. ¶¶ 333-335.
23. Id. ¶ 334.
24. SBC Brief at iii-iv, 39-44.
25. FCC SBC-Ameritech Order, App. C, Attach. A at A-34.
26. NorthPoint Comments at 11-12. In addition, NorthPoint states that the information that SWBT provides on loop make-up is often wrong. Id. at 15; NorthPoint Lewandowski Aff. ¶¶ 11-16.
27. Texas PUC Evaluation at 63-64.
28. DOJ Ex. 3 : SBC DSL E-mail at 2; see also Covad Comments at 31 (stating that Covad's actual loop qualification experience exceeds the response times reported by SBC).
29. Covad Comments at 27-28; Covad Wall Decl. ¶¶ 14-15.
30. DOJ Ex. 3: SBC DSL E-mail at 2.
31. SBC Dysart Aff., Attach. A at 12-14.
32. NorthPoint Comments at 16; NorthPoint Lewandowski Aff. ¶¶ 17, 21; Rhythms Comments at 35; Rhythms Lopez/Baros Aff. ¶ 18; Covad Comments at 27-28.
33. See infra note 43.
34. SBC 1999 Aggregated Performance Data, Measurement No. 62 ("Average Delay Days for SWBT Caused Missed Due Dates") (Digital Subscriber Line-DSL) at 271-No. 62c.
35. Compare id. with SBC DSL Delay Days Ex Parte, Tab 1 (correcting the aggregate performance data filed for PM 62 on February 1, 2000).
36. NorthPoint Comments at 2.
37. NorthPoint Mailloux Aff. ¶ 6. This incident reflects poorly on SBC's claim that Telcordia has validated its data reporting systems. SBC Dysart Aff. ¶¶ 65-78.
38. In doing so, SBC did not expressly note that the changes in the data had been made, explain why the processing error had not been previously detected or explain what steps it might take to detect other possible errors in its unaudited data reports.
39. Covad Comments at 29-31; Covad Wall Decl. ¶ 16.
40. Rhythms Lopez/Baros Aff. ¶ 22.
41. A DSL loop is a continuous copper line from the collocation site in SBC's central office to the end user, which is not equipped with "repeaters," the equipment used to increase the transmitted signal. The ISDN BRI loop may include a section of fiber optic cable and should include ISDN repeaters for long loops.
42. The Telcordia test results regarding DSL lines provide no evidence that SBC is adequately providing DSL-capable loops to its competitors. To the contrary, Telcordia advised that further testing of SBC's ability to provision these loops was needed. Telcordia Final Report at 7 (specifically suggesting further analysis of ADSL and SDSL provisioning as one of "seven next steps").
43. Tier 1 measures are deemed "customer" affecting and Tier 2 measures are "competition" affecting under the Texas performance system. See SBC Dysart Aff. ¶ 46 & Attach. H. The parity comparison for PM 58 is with SBC's DS1 service; data are supposed to be reported both on a CLEC-specific and aggregate basis. Business Rules 1.6 at A-78.
44. SBC 1999 Aggregated Performance Data, Measurement No. 58 ("Percent SWBT Caused Missed Due Dates") (Digital Subscriber Lines-DSL) at 271-No. 58c.
45. Business Rules 1.6 at A-81.
46. SBC 1999 Aggregated Performance Data, Measurement No. 60 ("Percent Missed Due Dates Due to Lack of Facilities") (Digital Subscriber Lines (DSL)), at 271-No. 60c.
47. Business Rules 1.6 at A-83.
48. Compare SBC 1999 Aggregated Performance Data, Measurement No. 62 ("Average Delay Days for SWBT Caused Missed Due Dates") (Digital Subscriber Line (DSL)) at 271-No. 62c with SBC DSL Delay Days Ex Parte at Tab 1 (Feb. 7, 2000) (correcting the aggregate performance data filed for Measurement No. 62 on Feb. 1, 2000).
49. Business Rules 1.6 at A-71.
50. SBC 1999 Aggregated Performance Data, Measurement No. 55.1 ("Average Installation Interval-DSL) (Requires Conditioning) at 271-No. 55.1c.
51. Business Rules 1.6 at A-80.
52. SBC 1999 Aggregated Performance Data, Measurement No. 59 ("Percent Trouble Reports on N.T.C Orders Within 30 Days) (Digital Subscriber Line-DSL) at 271-No. 59c.
53. Business Rules 1.6 at A-86.
54. SBC 1999 Aggregated Performance Data, Measure No. 65 ("Trouble Report Rate %") (Digital Subscriber Line-DSL) at 271-No. 65c.
55. Business Rules 1.6 at A-74 to A-75.
56. SBC 1999 Aggregated Performance Data, Measure No. 56 ("Percent Installed Within "X" Days") (BRI Loops) at 271-No. 56a.
57. Business Rules 1.6 at A-78 to A-79.
58. SBC 1999 Aggregated Performance Data, Measure No. 58 ("Percent SWBT Cuased Missed Due Dates) (BRI Loops) at 271-No. 58b.
59. Business Rules 1.6 at A-81.
60. SBC 1999 Aggregated Performance Data, Measure No. 60 ("Percent Missed Due Dates Due To Lack of Facilities") (BRI Loop) at 271-No. 60b.
61. Business Rules 1.6 at A-80.
62. SBC 1999 Aggregated Performance Data, Measure No. 59 ("Percent Trouble Reports on N.T.C Orders within 30 Days") (BRI Loops) at 271-No. 59b.
63. Business Rules 1.6 at A-86.
64. SBC 1999 Aggregated Performance Data, Measure No. 65 ("Trouble Report Rate %") (BRI Loop with Test Access) at 271-No. 65b.
65. Business Rules 1.6 at A-88.
66. SBC 1999 Aggregated Performance Data, Measure No. 67 ("Mean Time To Resotre (Hours)-Dispatch") (BRI Loop with Test Access) at 271-No. 67b.
67. The Department believes that it would be inappropriate to rely on the Texas Performance Remedy Plan ("PRP") to improve SBC's poor performance in provisioning DSL-capable loops. Texas PUC Evaluation at 65. As the Department stated in its New York Evaluation, a performance maintenance plan, such as the PRP, is intended to prevent backsliding in performance, and should not be relied upon as a bootstrap mechanism to pull poor performance into apparent compliance. See DOJ New York Evaluation at 36-40. Moreover, it is unclear whether the structure and subcaps of the PRP permit it to be an adequate deterrent to backsliding on services such as DSL. While SBC has raised the overall annual cap to $289 million, SBC Dysart Aff. ¶ 52, the underlying per occurrence and per measurement subcaps were not adjusted. As a result, any penalties may be capped at a level too low to lead to any significant, behavior affecting, payment. See Texas PUC Staff Performance Measures Evaluation at 9-10, n.4 (Tier 2 damages for three months ended August, 1999 over $5 million before application of the measurement caps, but only $389,033 after). Although SBC modified the PRP slightly in December, including increasing the Tier 2 payment for low-volume services such as DSL, SBC Dysart Aff. ¶¶ 48-50, it is not clear whether its potential liabilities in the event of poor performance were, in fact, significantly enhanced. Moreover, the DSL performance measures associated with the CLECs' interconnection agreement have yet to be defined, so it is unclear whether these CLECs are eligible to receive compensation under the PRP.
68. FCC New York Order ¶ 330.
69. We note that a separate affiliate provides no assurance of adequate performance in situations where a CLEC seeks access to unbundled elements in order to provide a service that the separate affiliate does not provide. This situation may arise when the CLEC is providing a service that is provided by a BOC, rather than its affiliate, or when the CLEC is providing a service that is not provided by either the BOC or its affiliate (though the CLEC service may be competitive with some other service of the BOC or its affiliate). See FCC SBC-Ameritech Merger Order ¶¶ 177, 197 ("For innovative entrants in particular, parity rules will not always suffice.")
70. A separate affiliate does nothing to alter any underlying incentive to discriminate. At best, a separate affiliate structure may make it more difficult to effectuate some forms of discrimination, and make it easier to detect discrimination.
71. SBC Conway Aff. ¶ 79. Many of the cuts currently being done using the CHC method are actually of lower loop volumes and within business hours, and thus would qualify to be done as FDT cuts. Id. ¶ 86; see also SBC Dysart Aff. ¶ 653 (CHC orders included in a sample of August through October data average three to four lines per order.).
72. SBC Conway Aff. ¶ 76.
73. Although SBC's Conway affidavit states that the FDT process concludes with a post-cut notification to the CLEC that the cut has been completed and the number should be ported, SBC Conway Aff. ¶ 87, the FDT process flow attached thereto makes clear that the CLECs are to automatically port the number at the scheduled frame due time. Id., Attach. J at J-2. The only post-cut contact between the CLEC and SBC is to occur if the CLEC's 30-minute and 60-minute line tests fail, in which case the CLEC may call SBC to inform it that a problem has arisen. Id. at J-6; see also AT&T DeYoung Hot Cuts Decl. ¶ 42(b) (At the designated frame due time, the CLEC sends an "activate" message to begin the number port, and SBC begins the loop cut.).
74. SBC Conway Aff. ¶ 79.
75. AT&T DeYoung Hot Cuts Decl. ¶¶ 45-47 (citing SBC letters, e-mails and statements presented to the Texas PUC).
76. Id. ¶ 44.
77. SBC Conway Aff. ¶ 79.
78. SBC Hot Cut Ex Parte at 2.
79. The Texas PUC had no data on FDT before it, and did not comment on the FDT process in its recommendation except to state that measures to capture FDT data will be addressed at the six-month PM review in April 2000. Texas PUC Evaluation at 59.
80. FCC New York Order ¶ 309 (emphasis added).
81. See Nextlink Barron Aff. ¶ 26 (SBC's service-related problems on hot cuts "directly impact NEXTLINK's ability to provision service to its end-user customers on a timely basis."); Nextlink Draper Aff. ¶ 37 & Attach. C (quantifying time and money spent to minimize the effect of premature cuts on Nextlink's end-user customers); CapRock Thompson Aff. ¶ 21 (SBC's poor hot cut performance has forced CapRock to delay its pace of market entry.); NTS Elliott Aff. ¶¶ 8, 22 ("NTS estimates that, due to SWBT's inability to correctly provision UNE cuts, NTS is submitting only about one-half the order volumes that NTS could handle if SWBT's performance were as promised."); AT&T Holtz Decl. ¶¶ 10-13 ("SWBT has failed to develop and implement provisioning processes that are robust and accurate enough to enable CLECs such as AT&T to open up our marketing channels and serve this [small business] market at commercial volumes.").
82. FCC New York Order ¶¶ 292-298.
83. SBC Dysart Aff. ¶¶ 652-56. These data are not drawn from a random sample, but rather represent performance only on orders for which SBC technicians recorded both a start and stop time for the hot cut. The orders included in the sample averaged three to four loops per order. SBC states that it used these data because "[n]ot all cutover logs during these months contained both a start and stop time, due to varying proficiency levels among technicians responsible for recording this information." SBC Hot Cut Ex Parte at 1. In theory, it is possible that this non-random sample could understate SBC's true on-time performance, but we are aware of no basis for assuming that the technicians who were less proficient at recording start and stop times would be more proficient at handling other aspects of hot cut provisioning, or that start and stop times would more likely be recorded for late-completed hot cuts than for those completed on time.
The Texas PUC cited these timeliness data in finding SBC's hot cut performance acceptable. Texas PUC Evaluation at 59. The Texas PUC voted on SBC's section 271 application at its December 16, 1999 Open Meeting, and thus was unable to consider the implications of SBC's January 21, 2000 hot cut Ex Parte with its fuller explanation of SBC's sample selection.
The number of CHC loops reflected in the sample is significantly lower than the number of CHC loops stated in its performance reports. Compare SBC Dysart Aff. ¶ 653 (showing 456 CHC loops in August 1999, 206 CHC loops in September 1999 and 398 CHC loops in October 1999) with SBC 1999 Aggregated Performance Data, Measurement No. 114 ("Percent of Premature Disconnects") at 271-No. 114 and id., Measurement No. 115 ("Percent SWBT Caused Delayed Coordinated Cutovers") at 271-No. 115a, 271-No. 115b (showing total CHCs for loop-with-LNP and loop-with-INP of 3125 in August 1999, 4134 in September 1999, 2980 in October 1999, and 1137 in November 1999). The November 1999 CHC data from the performance measures also fail to match the November 1999 CHC data included in SBC Conway Aff. ¶ 79 (2375 CHCs completed in November 1999).
84. Prior to December 16, 1999, the CHC cutover window, as understood by the CLECs and previously referenced by PUC staff, had been 60 minutes. AT&T DeYoung Hot Cuts Decl. ¶ 40(b) (SBC "must notify AT&T, within 60 minutes of the authorized cut start, that the cut has been completed."); Texas PUC Staff Performance Measures Evaluation at 54 (referencing the need for a new measure to capture cuts lasting longer than 60 minutes). At the December 16, 1999 Open Meeting, SBC proposed, and the PUC adopted as an interim measure, PM 114.1 to measure the duration of the CHC cut, referring to a 120-minute window. SBC defined the new measure to start measuring at the time when the CLEC calls SBC to start the cut, and to stop measuring at the time when the technician calls SBC's Local Ordering Center ("LOC") to report the work complete. SBC Dysart Aff. ¶ 654. SBC does not include in this reported interval any measure of the time it took to notify or attempt to notify the CLEC that the cut was complete, although it is this notification that alerts the CLEC to port the number and test the line -- necessary steps for the end-user to receive fully functional telephone service. AT&T DeYoung Hot Cuts Decl. ¶ 40 ("[I]f SWBT has completed the cut but delayed notifying AT&T, the customer will be unable to receive incoming calls throughout the entire period of delay."); see also Texas PUC Evaluation at 63-64; supra discussion pp. 12-13 (Texas PUC recognized the importance of including the time period for CLEC notification in PM 57, Average Response Time for Loop Make-Up Information, which measures the timeliness of SBC's response to CLEC requests for DSL loop make-up information.).
SBC asserts that the Department has found that a two-hour cutover window is the appropriate interval, see SBC Dysart Aff. ¶ 656; the Department, however, has never articulated such a standard.
85. SBC's December data is presented in loops, rather than in orders, and it cannot be determined from the data whether the percent of loops cutover within the relevant time period is an accurate indicator of the percentage of orders completed within that period.
86. SBC did not state how many of the more-than-one-hour hot cuts it attributed to CLEC-caused misses, and we do not know whether CLECs would agree or disagree with SBC's categorization. Without these exclusions, SBC's December data indicate that 82.2% of CHC hot cuts were completed within one hour. SBC Hot Cut Ex Parte at 2.
87. This reconciliation was performed at the request of the Texas PUC. AT&T DeYoung Hot Cuts Decl. ¶¶ 85 n.56, 87 (These outages are not captured in SBC's trouble report rate measure, which excludes outages reported before a cutover is accepted.). These data were submitted to the Texas PUC on December 16, 1999, see SBC & AT&T Royer/Van de Water Joint Aff., but were not discussed either during its Open Meeting, which occurred that morning, or in its formal comments to the Commission. See also Nextlink Barron Aff. ¶ 27 (In the last week of December 1999, 30% of Nextlink's completed hot cut orders were directly affected by non-operational facilities.).
88. FCC New York Order ¶ 302 n.961.
89. SBC Hot Cut Ex Parte at 2. These data were specifically collected for the Ex Parte. For August 1999 through December 1999, SBC's officially reported 30-day I-report rate has risen from 2.6% to 5.7%. SBC 1999 Aggregated Performance Data, Measurement No. 59 ("Percent Trouble Reports on N. T. C Orders Within 30 Days") (8.0 dB Loop) at 271-No. 59a. SBC reports I-reports for AT&T rising from 1.4% in August to 9.5% in October; and AT&T states that 70% of its trouble reports are actually filed within the first seven days after installation, resulting in a 7% 7-day I-report rate. AT&T DeYoung Hot Cuts Decl. ¶¶ 123-124.
90. It is necessary to evalute the adequacy of SBC's FDT performance not only because of the recent shift in volume from CHC to FDT, but also because SBC's statements regarding CHC capacity constraints directly implicate the scalability of the CHC process and raise doubts that even the current level of CHC performance could be maintained at higher volumes.
91. AT&T DeYoung Hot Cuts Decl. ¶¶ 59-60.
92. Id. ¶¶ 65-70.
93. NTS Elliott Aff. ¶¶ 19-21 (NTS has converted approximately 1500 lines using FDT; SBC has failed to follow the procedures on as much as 30% of these; most often SBC has cut over the end-user's service "one or more days in advance of the designated frame due time" -- often before NTS has completed its facilities work or ported the number.); CapRock Thompson Aff. ¶¶ 19-20 (CapRock has experienced a "frequent" number of premature cuts one or more days before the scheduled time and before CapRock has established its service, including porting the number, to the customer.); see also AT&T DeYoung Hot Cuts Decl. ¶ 43 (stating the risk of premature FDT cut is magnified because SBC proceeds without waiting for CLEC call).
94. SBC Hot Cut Ex Parte at 2. As noted previously, SBC lists the number of loops cutover within the window but not the number of orders completed within the window, and it cannot be determined from the data whether the percentage of loops cutover within the window is an accurate indicator of the percentage of orders completed within the window.
95. The SBC Hot Cut Ex Parte does not explain whether the start time on which it based its duration calculation is the same as the scheduled frame due time. Thus, it is unclear whether premature FDT cuts or late-started FDT cuts are reflected in the presented data.
96. In addition to service outages that may result if SBC initiates the hot cut prematurely or completes it late, outages may also occur if the hot cut is not otherwise performed correctly.
97. Although SBC nowhere addresses FDT outages upon provisioning, SBC did provide data indicating that 2.88% of FDT hot cuts generated trouble reports within ten days, again, a higher I-report rate than that accepted by the FCC in New York. SBC Hot Cut Ex Parte at 2.
98. In December 1999, as a result of the reconciliation of CHC data with AT&T, SBC and AT&T agreed to a new procedure for CHCs. SBC Conway Aff. ¶ 97. In September 1999, SBC established an FDT unit within the LOC. Id. ¶ 77. In October 1999, SBC established a committee to resolve central office issues on UNE circuits; "issues being worked on include upgrading the training on the proper use of trouble codes, proper escalation procedures and improving coordination between departmental work groups." SBC Dysart Aff. ¶ 363 (in the context of discussing SBC's 30-day I-report rate on 8 dB loops).
99. Ex parte data submitted by SBC to the Commission shows that total EDI flow-through for UNE (excluding UNE-platform) was 25.53% in September ("Total SMFIDs" 470), 29.63% in October ("Total SMFIDs" 1458) and 33.81% in November ("Total SMFIDs" 2431). SBC Reject and Flow Through Rate Ex Parte at 6 (updating SBC Ham Aff., Attach. X-2). Total LEX flow-through for UNE (excluding UNE-platform) was 31.70% in September ("Total SMFIDs" 10,285), 33.68% in October ("Total SMFIDs" 9540), and 40.17% in November ("Total SMFIDs" 11,812). Id. at 58.
100. Whether an order (an LSR) is submitted over LEX or over EDI, once it hits SBC's LASR system, the resulting process flows are identical. SBC Ham Aff. ¶ 138.
101. RPONs are often necessary with new loop orders. See Nextlink Draper Aff. ¶ 20 (referencing need for careful coordination of new loops with LNP provisioning, "[I]f . . . not successful, then the new NEXTLINK customer's first experience with our company will be a loss of service . . . [and] only NEXTLINK is held accountable in the marketplace for SWBT's underlying poor performance.").
102. See FCC New York Order ¶¶ 161-163.
103. CLECs complain that the number of late FOCs is inadequately reflected in SBC's reported performance data. Nextlink has documented SBC significantly over-reporting its timely return of FOCs to Nextlink in November and December. Nextlink Barron Aff. ¶¶ 11-21.
104. SBC Dysart Aff. ¶ 143. The Texas PUC benchmark is 95% FOCs returned within five hours, for orders submitted over EDI or LEX, regardless of whether these orders drop out for manual processing once received by SBC. Business Rules 1.6 at A-12 to A-13.
105. In October, SBC reports only 88.1% of UNE-loop (1-50) FOCs returned over EDI were returned within 5 hours; the average time to return the FOCs was 2.12 hours. In November, SBC reports 92.7% timely return of UNE-loop (1-50) FOCs over EDI, but its average time to return the FOCs had increased to 5.72 hours. In December, SBC reports only 85.8% timely return of UNE-loop (1-50) FOCs via EDI, although its average time had improved again to 2.35 hours. UNE-loop (1-50) order volume over EDI rose from 27 orders in September to 118 in October, 179 in November and 254 in December. SBC 1999 Aggregated Performance Data, Measurement No. 5 ("Percent FOCs Received Within "X" Hours") (UNE-loop, 1-50) at 271-No. 5d; id., Measurement No. 6 ("Average Time to Return FOC (Hours)") (UNE-loop, 1-50) at 271-No. 6d.
In its evaluation, the Texas PUC recognized that SBC had missed the benchmark for UNE-loop FOC return over EDI in October and November, but relied on apparently compliant results (on extremely high volume) in August to conclude that no systemic problem exists. Texas PUC Evaluation at 39. The reported results in August, however, erroneously include some 11,000 UNE-platform orders in addition to the handful of UNE-loop orders. AT&T Pfau/DeYoung Decl. ¶ 92. Thus, the reported results in August are not indicative of actual performance on UNE-loop FOC returns -- as can be seen by reviewing SBC's reported results for the following months.
106. SBC Dysart Aff. ¶ 140. SBC's performance for FOC returns via LEX has bounced around from 94% timely to 90% timely and back to 97% timely; average time has also varied from .9 hours to 2.1 hours. UNE-loop (1-50) order volume over LEX rose from 1479 orders in September 1999 to 1723 in October 1999, 2541 in November 1999, and 2142 in December 1999. SBC 1999 Aggregated Performance Data, Measurement No. 5 ("Percent FOCs Received Within "X" Hours") (UNE-loop, 1-50) at 271-No. 5b; id., Measurement No. 6 ("Average Time to Return FOC (Hours)") (UNE-loop, 1-50) at 271-No. 6b.
107. The Texas PUC benchmark is 95% FOCs returned within 24 hours for orders submitted manually to SBC. SBC Dysart Aff., Attach. A at A-12 to A-13. In September 1999, SBC returned 94.7% of its FOCs within 24 hours, but, in October 1999, SBC's benchmark performance had declined to 88.7%, and, in November 1999, SBC's performance had declined still further to 80.7%; in December 1999, the percentage of FOCs timely returned rose to 94.9%. SBC 1999 Disaggregated Performance Data, Measurement No. 5 ("Percent FOCs Received Within "X" Hours") (UNE-loop, 1-50) at 271-No. 5f. The average time to return FOCs was 37.2 hours in October 1999 (order volume 1578), improved to 23.6 hours in November 1999 (order volume 1028), but rose precipitously to 42.9 hours in December 1999 (order volume 777). Id., Measurement No. 6 ("Average Time to Return FOC (Hours)") (UNE-loop, 1-50) at 271-No. 6f.
108. SBC Dysart Aff. ¶¶ 147, 598.
109. The reject measures are not broken out by order type; the numbers which follow relate to all orders submitted via EDI or LEX which were manually rejected.
110. For Percent Manual Rejects Received Electonically and Returned in Five Hours, PM 10.1, SBC reported 81.4% timely returned in July, 59.5% timely returned in October, 65.1% timely returned in November and 69.5% timely returned in December 1999. SBC 1999 Aggregated Performance Data, Measurement No. 10.1, at 271-No. 10.1, 11.1
111. For Mean Time to Return Manual Rejects Submitted Electronically, PM 11.1, SBC reported that the average return time was 6.86 hours (on 3658 rejects) in July and rose to 9.94 hours (on 6535 rejects) in November, and all the way to 35.65 hours (on 6698 rejects) in December 1999. Id., Measurement No. 11.1, at 271-No. 10.1, 11.1.
112. SBC Dysart Aff. ¶ 160.
113. FCC New York Order ¶ 187.
114. SBC reports Percent Mechanized Completions Returned Within One Day of Work Completion, PM 7.1. This is a measure of the timeliness of completion notices and does not reflect the size or age of any backlog of pending SOCs. SBC's data show that SOCs returned via LEX have been returned late from May through December 1999, with only 87% being returned within one day of work completion in December. SBC 1999 Aggregated Performance Data, Measurement No. 7.1, at 271-No. 7.1. SBC reports that SOCs returned via EDI have been timely, in accordance with the Texas PUC's 24-hour benchmark. Id.
In addition, CLECs have presented data showing that a number of SOCs have been delayed not just for 24 hours, but for days and weeks, and that for some orders, SOCs have never been returned at all. See ICG Rowling Aff. ¶ 37 (referencing attachment that lists missing SOCs); AT&T Dalton/DeYoung Decl. ¶¶ 178-180 (The November reconciliation of CHC data revealed orders installed in the first week in August for which SOCs had not been sent.).
115. SBC Dysart Aff. ¶ 521. SBC asserts that this rate table error is an underlying cause of its repeated miss on the Billing Completeness measure (PM 17), which indicates the percentage of service orders completed within the billing cycle that post in the billing system prior to the customer's bill period. SBC adds that its performance on this measure, although below parity, is near perfect, at 98%, and has not inhibited the CLECs' ability to compete. Id. ¶ 19. SBC's performance on this measure, however, is merely an indicator of the problem but may not reflect its entire magnitude because it is a measure of the timeliness of posted service orders and does not measure the size or age of any backlog of unposted service orders. Moreover, posting delays for a particular order may be resolved within the bill period even after some end-user-affecting event has occurred.
See also AT&T DeYoung Hot Cuts Decl. ¶¶ 188-190 (AT&T analyzed its own raw data and concluded that 23% of its 8 dB loop orders had experienced posting delays greater than five days with the longest delay being 17 days; 34% of its 5 dB loops experienced posting delays greater than five days with the longest delay being 48 days. A December reconciliation meeting between AT&T and SBC referenced orders from September still sitting unposted.).
116. SBC Conway Aff. ¶¶ 55, 56.
117. See Nextlink Smith Aff. ¶ 5 (Since Nextlink began tracking the problem in November 1999, 12 customers receiving service from Nextlink have continued to receive bills from SBC. Two of these erroneously billed end-users were sent to collections by SBC, indicating that the problem had been ongoing for some time.); Nextlink Draper Aff., Attach. C (quantifying time and money spent to minimize the effect of double-billing and dropped directory listings on Nextlink's end-user customers); ICG Rowling Aff. ¶ 34 (ICG end-users have had their directory listings drop from SBC's directory assistance database.); AT&T DeYoung Hot Cuts Decl. ¶¶ 193-196 (documenting SBC continuing to bill even after customers have been converted to AT&T; end-user customers often blame AT&T for the billing confusion, damaging AT&T's reputation.).
118. Telcordia documented late and inaccurate FOC returns. Telcordia Final Report at 54 & Attach. A at A-44 to A-45 (UL-RT-O5). CLEC comments indicate that these problems are continuing. Nextlink Draper Aff. ¶¶ 31-38; Nextlink Barron Aff. ¶ 12 (stating a high rate of FOCs are placed in jeopardy status and returned to Nextlink at a later date).
119. See AT&T DeYoung Hot Cuts Decl. ¶ 43.
120. SBC recognizes that it has had problems scaling its manual processes, explaining that it has reorganized and retrained LSC representatives in an effort to "meet the growing demand for FOC." SBC Dysart Aff. ¶ 598 (specifically discussing below-benchmark performance on return of FOCs for LNP via EDI). SBC admits these adjustments to its manual processes impacted the results for September and October, but expected that "the overall long-term benefits will be seen in future months performance measures." Id. In fact, SBC's EDI performance for LNP-with-loop (1-19) declined still further in November, during which only 59.8% of the 378 FOCs were timely returned via EDI. In December, 77% of 560 FOCs were timely returned via EDI. SBC 1999 Aggregated Performance Data, Measurement No. 94 ("Percent FOCs Received Within 'X' Hours") ("Residence and Simple Loop, 1-19") at 271-No. 94c. SBC's LEX performance for LNP-with-loop (1-19), however, did improve slightly, so that in November, 91.5% of 1357 FOCs were timely returned via LEX. In December, 93.2% of 1269 FOCs were timely returned via LEX. Id. at 271-No. 94a. SBC does not appear to report an average return time for LNP FOCs.
121. FCC New York Order ¶ 177.
122. The trend in SBC's reported data is thus more important than the overall level of performance that SBC reports. This is so partly because volumes involved are still low. Moreover, as noted above, some CLECs have documented that SBC is overstating its performance on FOC return times. The record contains no reconciled data on this point.
123. See Time Warner Reeves Aff. ¶ 17 ("For most of  TWTC [Time Warner] requested additional tandem trunks to ensure that the exchange of traffic would not be blocked due to a lack of adequate facilities to the SWBT end offices. SWBT denied the majority of these requests."); see also e.spire Wong Aff. ¶ 14 ("Even in instances where e.spire has demonstrated that it is experiencing blockage and an inability to serve new customers, e.spire is given only a fraction of the capacity that it has requested.").
124. The Texas PUC established a benchmark of 20 business days for trunk installation. SBC's Service Planning process adds a month or more to the time necessary to obtain trunks, although this part of the trunk ordering process is not tracked or reported in SBC's performance metrics. CapRock Thompson Aff. ¶¶ 10-12; NTS Elliott Aff. ¶¶ 11, 14-15. SBC's failure to timely confirm a trunk service order can effectively add more unrecorded time to the provisioning interval, as a CLEC must then resubmit (supplement) its trunk order to investigate the cause of the missed FOC. NTS Elliott Aff. ¶ 13. This would cause the initial order to be excluded from the metrics covering installation intervals and missed due dates. See SBC Dec. 15, 1999 Conway Aff. ¶¶ 4, 10 ("Unless requested by the customer, LSC representatives have been instructed not to cancel orders (with the exception of rejected LSRs) and request a supplement as a means to extend the due date.").
125. SBC 1999 Disaggregated Performance Data, Measurement No. 73 (Percent Missed Due Dates) (Houston) at 271-No. 73; SBC Dysart Aff. ¶ 557.
126. In Houston, SBC limited the number of trunks Time Warner could install. In addition, SBC informed Time Warner that one specific tandem was "capped" and that Time Warner could obtain no more trunks there. After Texas PUC staff began investigating the problem, SBC personnel admitted that the tandem had additional capacity, but SBC was trying to require Time Warner to implement end office trunking. Time Warner Summitt Aff. ¶¶ 16-18; Time Warner Reeves Aff. ¶ 27.
127. In Dallas, e.spire started the ordering process for trunks in September 1999. SBC did not schedule trunk delivery until December 1999. e.spire asserts that this particular delay is typical of those in its various Texas markets. e.spire Wong Aff. ¶¶ 8-10.
128. All of NTS Communications' December 1999 trunk orders in Amarillo were held for lack of facilities. NTS Elliott Aff. ¶ 16. CapRock Communications has experienced delays in obtaining interconnection trunks in its markets. CapRock Thompson Aff. ¶¶ 8-17.
129. Time Warner Reeves Aff. ¶ 26. ("TWTC Houston experienced significant blocking from July through October 1999. For over five continuous weeks blocking occurred on TWTC's trunks every day.") e.spire also experienced blockage. See e.spire Wong Aff. ¶¶ 12, 14 (SBC refused to share the necessary network information to assist in planning to alleviate the blockage and limited trunk increases necessary to alleviate blockage.).
130. Time Warner Summitt Aff. ¶¶ 12-13 (Time Warner estimates it lost over $2 million in annual revenues due to its inability to obtain interconnection trunks. More than eight subscribers complained to Time Warner that the blocking damaged their businesses; one of Time Warner's existing customers sent its additional business to another CLEC; Time Warner turned away four other customers for fear of overloading its network during the time it could not obtain additional interconnection trunks from SBC.); see also e.spire Wong Aff. ¶¶ 11, 14 (SBC's actions "limit e.spire's ability to sign up new customers, and expand capacity for existing customers.").
131. In its evaluation, the Texas PUC pointed out a number of deficiencies of performance in prior months, including more trunk blockage than allowed by the Texas PUC standard (SBC's reported trunk blockage, disaggregated by region, was in excess of the Texas PUC's benchmark in some regions for some months, Texas PUC Evaluation at 14 n.51) and more missed due dates for delivering trunks to CLECs than to SBC itself (performance in the Houston area was out of parity for September and October). Id. at 15.
132. Id. at 6-7, 15 & n.55. ("[T]he Texas Commission became concerned that the data as collected was not accurately reflecting CLECs' ability to obtain interconnection trunks in a timely manner because SWBT was not capturing data on 'held orders,'" which are "orders that SWBT does not process due to lack of interconnection facilities."). The same effect occurs if CLECs place orders with extended due dates as a result of SBC advising them of a lack of facilities, as these orders will be excluded from PM 78.
133. We believe it would be prudent for the Commission to require additional evidence that the currently-reported data are accurate, before placing any reliance on that data. Until October 1999, SBC's performance reports reflected excessive amounts of trunk blockage as well as excessively lengthy average installation intervals, which appeared to be well over the Texas PUC's 20-business-day benchmark, and extended, at the extreme, to over 60 days.(134)
134. See PUC Staff's November 2, 1999 Evaluation of Performance at 2-3, and 45-52. ' - ¶ - - ¶¶ - ¶ ¶ ¶¶ -
135. See Dec. 16, 1999 Open Meeting Tr. at 29-31.
136. SBC Dec. 15, 1999 Leathers Aff. ¶¶ 12, 26; see also id. ¶ 19 (In reviewing the data, SBC discovered a gap in its data collection, so that data included in the preliminary report were omitted from the detail report; this gap will be closed by a correction to the data gathering program.).
137. Texas PUC Evaluation at 15; SBC Dysart Aff. ¶ 557 (describing further process changes that will address all issues).
138. An industry trunking forum was established in response to CLEC complaints about trunk blockage, and that forum has been meeting regularly since January 1999. Texas PUC Evaluation at 11-12, 14; SBC Deere Aff. ¶ 50. As noted above, however, SBC was still reporting disparate and inadequate trunk performance on key measures through September 1999, when the Texas PUC staff presented its November 2 Performance Measures Evaluation. Given this history, it is clearly premature to assume that these latest process improvements -- no matter how promising on paper -- will in fact resolve the pervasive problems which have been both alleged and observed.
139. In addition to performance problems, objections have been raised as to the legality of the non-recurring charges which SBC assesses in connection with UNE-platform orders. We leave that issue for the Commission's judgment.
140. Birch Tidwell & Kettler Aff. ¶¶ 13, 63-64, 81, 90; CapRock Thompson Aff. ¶¶ 24-29.
141. AT&T Dalton/DeYoung Decl. (3% of UNE-platform conversions from August to November 1999); Network Intelligence Burk Aff. ¶ 22 (14% of 620 orders, representing 3200 access lines, from May to December 1999).
142. Birch Tidwell/Kettler Aff. ¶ 13 & Attach. K.
143. Accessible Letter CLEC00-001, Final Minutes for December 7, 1999 CLEC User Forum 7 (Jan. 3, 2000), attached to AT&T DeYoung Aff. as Attach. 17.
144. See SWBT Local Service Center 1999 Force Model, Ex Parte Submission to the FCC (Confidential Version), CC Docket No. 00-4 (Feb. 8, 2000).
145. See New York PSC Order at 3 (increasing penalties associated with Bell Atlantic's
timeliness order processing metrics, for both manual and flow-through, because CLEC UNE-platform orders are falling out of its normal electronic OSS); DOJ Ex. 4: Letter from Bell
Atlantic to New York PSC (describing the amount of manual work necessary to coordinate the
handling of this problem with the CLECs). We are aware that Telcordia performed a "staff
scalability" analysis of SBC's order processing center work force model. See Telcordia Final
Report, Attach. I.1. Our review of that confidential model for November and December,
combined with the recent events in New York suggest that Telcordia did not take into account
the degree to which systems problems can overwhelm asserted, but not tested, manual processing