Combined DNA Index System Operational and Laboratory Vulnerabilities
Audit Report 06-32
Office of the Inspector General
The FBI received an overall positive assessment of its administration of CODIS from the CODIS administrators we surveyed. The FBI has given attention to CODIS infrastructure, development, and staffing. However, based on our analysis of the survey responses and FBI documentation, we have identified several areas in need of further improvement, including improved compliance, responsiveness, timeliness, and information sharing. In addition, the FBI needs to identify the current obstacles that prevent the CODIS Unit from achieving full staffing levels, reflect all activities in its performance measurements, and continue the progress made with the system infrastructure.
Each NDIS-participating laboratory is required by the MOU governing its participation to have an administrator who oversees CODIS operations at that laboratory. The administrator is the liaison between the FBI and CODIS users and is expected to relay necessary information to aid in compliance with NDIS participation requirements. Consequently, the CODIS administrators have an influential role in the CODIS community and have an opportunity to interact with the FBI in a way that would provide them with the experience needed to assist us in assessing the effectiveness of the FBI’s administration of CODIS. As part of our effort to assess the FBI’s administration of CODIS, we conducted a survey of 174 CODIS administrators.12
Our analysis of survey results revealed an overall positive assessment of the FBI’s administration of CODIS. However, we identified several opportunities for improvement. For example: (1) QAS compliance within the CODIS community can be improved and workloads reduced if the FBI ensures that all CODIS administrators receive QAS auditor training; (2) CODIS Unit responsiveness can be improved through sufficient staffing, tracking of information requests, and the use of other organizational tools; (3) CODIS community understanding and compliance with profile allowability restrictions can be enhanced through increased emphasis on written sources of guidance that should be available to all CODIS users; (4) Review Panel timeliness can be improved if guidance is disseminated to the appropriate members of the CODIS community who can ensure that submissions are complete; and (5) the FBI can improve information sharing through better use of the CODIS intranet website by disseminating written guidance to the CODIS community that is consistent, practical, and easy to navigate. These results are further described in the following sections.
Survey Distribution and Design
Our survey was designed to provide feedback from CODIS administrators on a variety of topics. The survey contained 46 primary questions and 25 secondary and multi-part questions, resulting in 71 total questions. (See Appendix VI for a complete listing of the survey questions and responses received.) Of the total, 26 questions allowed respondents to provide supplemental comments in which to clarify or explain their answer. Supplemental comments were generally added when respondents gave a negative answer. In total, we received 636 supplemental comments.
We developed questions from our analysis of the trends in the OIG’s former audits of CODIS laboratories, recommendations from members of the CODIS community, and the findings contained within the OIG’s 2001 audit report. In addition, the FBI provided suggestions for survey questions.
We divided the questions into seven topics, covering the major issues we identified as potential areas of weakness in the FBI’s administration of CODIS, which were applicable for comment by the administrators. Six of the seven topics contained questions in which respondents could provide additional comment. The seven topics were: (1) demographics, (2) FBI CODIS Unit responsiveness, (3) allowability of DNA profiles, (4) laboratory quality, (5) general CODIS operations, (6) NDIS Audit Review Panel, and (7) FBI guidance to the CODIS community.13
We provided administrators with 1 month (including a deadline extension) to submit their responses. In addition, we offered those states not represented in the responses received by the deadline a further opportunity to respond. We received 144 responses from 47 states, which represents an 83-percent response rate.14 Included in these responses were surveys from 49 SDIS laboratories and 95 LDIS laboratories. With such a large number of both SDIS and LDIS respondents, we believe the responses fairly represent the views of CODIS administrators within the NDIS community.
We analyzed survey results to detect commonalities of response and consensus of opinions. As part of this analysis we tabulated responses for all questions, calculated a consensus for each question, identified trends in supplemental comments, and determined if vulnerabilities were identified by the consensus responses and comment trends. The results of our analysis of the CODIS administrator survey results follow and are referenced throughout this report where applicable. The complete listing of survey questions and responses can be found in Appendix VII.
Survey Results and Analysis
While we generally note positive results below we also identify potential areas for improvement.
Demographics. We began our survey with questions that would help us ascertain the variety of experience, size of laboratories, and duties and activities of the administrators. Responses indicated that the average time the respondents had spent as a CODIS administrator was 3 to 5 years and the average size of the respondents’ DNA laboratories was 6 to 10 positions (including all staff specific to the DNA portion of their laboratory). Most respondents (65 percent) were administrators who also had casework analysis duties, and additional respondents (8 percent) were administrators who also performed casework and offender analysis duties. In addition, 13 percent were administrators who filled some other role, such as quality assurance manager or technical manager.
We found that 43 percent of CODIS administrators stated that they have not taken the FBI's QAS auditor training (survey question 5), a course that is designed to ensure a consistent understanding of the QAS and application of the FBI’s audit document, as well as an understanding of the principles and objectivity surrounding auditing.15 In our judgment, while not every administrator may need guidance on how to conduct an audit, the FBI’s QAS auditor training course would ensure that administrators are versed in QAS compliance to the degree necessary to assist their laboratories in ensuring compliance.
Further, administrators stated that one of the top reasons for contacting the CODIS Unit relates to QAS matters (survey question 6), meaning that much time and effort is expended by both the administrators and CODIS Unit staff to address QAS issues. We believe this time and effort could be minimized, freeing up time for other duties, if administrators received training in QAS compliance.
In addition, later survey results, in combination with the results of question 5, indicate that some administrators who have not taken the auditor training are still participating in the resolution of QAS audits for their laboratory. We reached this conclusion from the fact that 66 percent of CODIS administrators indicated they are involved in the QAS audit resolution process (questions 30 and 31), but only 57 percent of the administrators have taken the QAS auditor training. If CODIS administrators are to be responsible in their laboratories for handling the audit resolution process, they should have the benefit of receiving training in the accepted interpretation of the QAS and the expected documentation to establish compliance. Without that training, they could contribute to delays in the resolution process by failing to submit complete corrective action documentation or by challenging findings unnecessarily, both of which are factors that we determine hinder the timeliness of the Review Panel. (See our analysis of Review Panel timeliness in Finding III, page 62.)
Separately, our analysis of QAS audit trends in Finding II reveals trends that impact significant aspects of laboratory operations, such as chain-of-custody records and evidence storage and security. (See page 48 for additional detail.) The trends further emphasize the need for the FBI to ensure that all key members of the CODIS community, including CODIS administrators, are fully trained in compliance with the QAS.
We therefore conclude that by ensuring that administrators participate in the QAS auditor training, state and local laboratory compliance can be improved and the workload of both the administrators and CODIS Unit staff can be reduced.
FBI Responsiveness. We asked a series of questions (numbers 6 through 11) to determine how responsive the CODIS Unit has been to members of the CODIS community. According to administrators, the timeliness and helpfulness of FBI CODIS Unit staff is not a significant problem, although we noted from the overall results of the survey that there is room for continued improvement. For example, we determined that 20 respondents made a total of 28 comments regarding the FBI’s slow response time and its inaccessibility. Those comments drew attention to various issues that, if addressed, could improve the CODIS Unit’s responsiveness. According to these respondents:
Our analysis of unit staffing confirms that understaffing of the CODIS unit is an important issue (see page 29). Further, without some means of tracking information requests, the FBI cannot ensure that it responds to all requests in a timely fashion. Finally, by identifying topic-specific points of contact and enhancing information sharing through the CODIS intranet, the CODIS Unit can improve its responsiveness to the CODIS community.
Profile Allowability. NDIS Participation Requirements specify the restrictions for profiles that are permissible for inclusion in NDIS. We asked a series of questions (numbers 12 through 20) to assess the level of administrators’ understanding of those restrictions and their ability and confidence to apply that understanding in determining whether a specific profile was permissible, or allowable, for inclusion in NDIS. Results indicate that administrators are knowledgeable and confident in determining profile allowability as a routine part of their duties. However, the survey results also indicate that administrators lack confidence in whether there is consensus in the CODIS community about what is allowable and in the compliance of other laboratories in submitting only allowable profiles.
The survey results indicated that administrators did not identify themselves as solely responsible for making sure casework profiles are uploaded in compliance with NDIS requirements. As shown in Figure 7, analysts and reviewers were identified as the responsible official almost as often as administrators.
On-going guidance on profile allowability is provided primarily to CODIS administrators during national CODIS meetings where profile allowability scenarios are discussed in an open forum. The discussion sessions serve as a source of helpful guidance and clarification on profile allowability, as emphasized by the number of comments to this effect from survey respondents. However, not all analysts and CODIS users attend each national meeting. Conversely, as shown in Figure 7, only 41 percent of administrators are solely responsible for ensuring that profiles uploaded to NDIS are suitable for inclusion. Therefore, we believe this same guidance may not be communicated to the responsible staff in each CODIS laboratory. We conclude that the FBI needs to take steps to ensure that all CODIS users are provided the same guidance that is given at national meetings regarding profile allowability. Such steps could include enhancing the information sharing of the CODIS intranet, through scenarios or decision-trees accessible to all CODIS users.
The CODIS Unit Chief told us that all CODIS users are required to sign the profile allowability certification form, which specifies that they know and understand NDIS procedures governing allowability of profiles. However, since the FBI does not verify that those forms are completed as required, the FBI cannot totally rely on those certifications to ensure that all CODIS users who make profile allowability decisions are receiving the necessary guidance to ensure compliance.17
In addition, we observed that administrators primarily view a person, rather than a law, policy, or other form of written guidance, as their primary source on profile allowability matters. For example, respondents were asked in question 19, “If a member of your DNA laboratory has a question regarding whether a profile is allowable for upload to NDIS, who or what would be their most likely source for clarification?” Respondents could offer more than one reply. Out of 143 responses, as all or part of their answer, 111 respondents cited “CODIS Administrator in their laboratory” as a source of guidance; 27 respondents cited “CODIS Administrator Handbook”; and another 27 respondents cited “CODIS administrator in another laboratory.” These responses primarily identify a person, rather than a formal written document, as the source of guidance for the staff within their laboratory.
Also, the following results for questions 16a through 16c primarily indicate a person rather than a document as the final authority on what profiles are uploaded to CODIS, as shown in Figure 8.
Figure 8 – Results of Administrator Survey Question 16a – 16c
We believe the FBI must take steps to ensure that the NDIS community relies on written law or policy to ensure consistent and thorough compliance with the NDIS requirements, for consistency, reproducibility, and minimization of human error and subjectivity. See the section on “FBI Guidance” results on page 26 for additional discussion of written guidance.
Laboratory Quality. We asked CODIS administrators to comment on the operational quality of their laboratory and other laboratories with which they are familiar (questions 21 through 23). Respondents rated their own laboratory’s quality, as well as the quality of their laboratory in relation to others, fairly high. However, 8 percent of respondents stated that they were aware of a CODIS laboratory operating with what they believed to be a material weakness. Their comments revealed that they identified issues that included the inherent limitations of one‑person DNA laboratories, uninvolved off-site technical leaders, laboratories that upload profiles that have not been fully reviewed, and laboratories that emphasize quantity over quality. According to our discussions with the CODIS Unit Chief and the chairperson of SWGDAM, these weaknesses are already known and are being considered in conjunction with on-going revisions to the QAS.18 However, we recommend continued attention to these material weaknesses.
CODIS Operations. We asked administrators to assess various aspects of CODIS operations (questions 24 through 29). The CODIS administrators made it clear through their responses that the overall sentiment regarding general CODIS operations is positive. Specifically, we found that the CODIS contractor, the CODIS software, and the FBI’s current management of CODIS all received high marks from respondents, and that administrators felt there had been a fair measure of improvement in the FBI’s management of CODIS under the current CODIS Unit Chief’s leadership.
Further, respondents identified what they believe to be the most important successes of CODIS:
Respondents also identified what they believe to be the greatest challenges to CODIS in the next 5 years:
NDIS Audit Review Panel. In order for a laboratory to meet the QAS requirement for a biannual external QAS audit, the audit must be conducted using the FBI’s approved audit document by QAS auditors that have successfully completed the FBI’s auditor training.19 The FBI further requires that these external QAS audits be submitted to the NDIS Audit Review Panel. The CODIS Unit oversees the Review Panel, a group of volunteer members of the DNA community and FBI staff members who meet specific professional criteria. The Review Panel reviews all external QAS audits conducted in NDIS‑participating laboratories across the country, with the purpose of ensuring consistent and thorough application of the QAS and appropriate and complete corrective action.
We asked a series of questions (numbers 30 through 35) designed to provide insight into what experience the CODIS administrators have had with the Review Panel process, and what their comments are regarding the Review Panel’s accomplishment of its purpose. As shown in Figure 9, overall, respondents who have experience with the Review Panel process feel that it has improved compliance with the QAS.
Those who did not believe the Review Panel had improved compliance focused on the fact that individual interpretations of the QAS, rather than standardized interpretations, occur within the DNA community. Administrators also indicated that there has been improvement to Review Panel timeliness but that additional improvement is needed.21
We found that 31 percent of the respondents with experience in the Review Panel process stated that they had to supply additional corrective action documentation after their original submission to the Review Panel (question 33), which delayed the process for up to 6 months (question 34). In addition, the responses to questions 30 and 31 indicated that a large percentage (34 percent) of the CODIS administrators are not involved in the Review Panel process. Yet, based on our observations, CODIS administrators are the members of the NDIS community who often receive the guidance disseminated at national meetings regarding the Review Panel process and key factors in ensuring that a submission to the panel is complete.
We conclude from these responses that, by providing guidance to pertinent laboratory staff on ensuring their initial submission to the Review Panel is complete, one delay that undermines Review Panel timeliness can be reduced. In presenting our conclusion to the FBI, the CODIS Unit Chief stated that he understood our perspective. He subsequently asked the attendees at the 2005 National CODIS Conference for the contact information for the person in each laboratory who is responsible for the audit resolution process. The CODIS Unit Chief further stated that he would use these points‑of‑contact to develop a comprehensive mailing list to disseminate guidance or information to the NDIS community regarding the Review Panel process.
FBI Guidance. Finally, we asked administrators to provide feedback on various aspects of the FBI’s guidance to the CODIS community (questions 36 through 46). Respondents were fairly positive about the FBI’s guidance to CODIS participants on compliance with the QAS and NDIS requirements. Administrators’ perception of the FBI’s consistency in guidance was moderate, but overall, they stated that inconsistencies had limited impact on their ability to perform and comply with requirements. However, they indicated concern about the FBI-developed QAS audit guide (commonly referred to as the “audit document”) and the adequacy of the FBI’s guidance on proper use of the audit guide, as shown in Figure 10.
Figure 10 – Responses to Administrator
In the supplemental comments submitted with the responses to these questions, inconsistencies between QAS auditors were emphasized (as with question 30 in Figure 9), as were inconsistencies between the QAS auditors and other members of the DNA community. In addition, we determined that throughout the survey, 83 respondents made a total of 161 comments on inconsistencies in the way the QAS are interpreted within the DNA community. These comments identified the need for increased and improved training and improved guidance for all members of the CODIS community. See Finding III for additional conclusions regarding auditor training.
In addition, 37 of the respondents made a total of 51 comments regarding the need for the FBI to share information better by posting of guidance on the CODIS intranet website, such as frequently asked questions and common audit findings. We reviewed the contents of the CODIS website at one CODIS laboratory to assess the suggestions that were made for additional content. We found that while the current website appears to be a helpful tool for CODIS users, there are several ways that it could be enhanced to provide better guidance. For example, the website needs better tools for navigating the information it contains, such as a comprehensive table of contents or index for NDIS procedures, decision‑trees for profile allowability, and a list of frequently asked questions that direct CODIS users to the correct place within the NDIS procedures for additional guidance on various subjects. In addition, we found that some of the information on the website was not current (such as a list of upcoming QAS auditor courses that showed no entries after January 2005), and therefore was of no benefit. The FBI needs to ensure that the information is updated regularly to further encourage CODIS users to view the CODIS website as relevant and helpful to their daily activities.
When we discussed these suggested changes with the CODIS Unit Chief, he stated that the guidance the website already contains is not used as much as it could be. He added that members of the CODIS community often tell him that they are unsure of what NDIS procedures say, or that they were unaware of a change that had been highly publicized within the CODIS community months prior. We believe that while there may be those in the CODIS community who are not using the CODIS website, this should not prevent the FBI from making improvements to it to maximize the opportunity to provide written, user-friendly, relevant, and comprehensive guidance to the CODIS community.
Overall Analysis. In reviewing the overall survey responses and statements made by FBI management, we found that the FBI placed too much reliance on verbal rather than written guidance in everyday communications and in meeting discussions concerning the QAS and NDIS requirements. For example, the CODIS Unit Chief commented that he gives greater priority to phone rather than to electronic communications in everyday responses to the CODIS community, and that he is hesitant to put guidance in writing when dealing with a laboratory‑specific situation. He later clarified that answer by saying that he wants to avoid identifying specific laboratories by name or situation. However, we believe that the CODIS Unit Chief should attempt to use the interaction he has with individual labs as a means of identifying where additional guidance to the entire community is warranted. He could do this through the CODIS website or other avenues, without identifying specific labs.
Verbal communication is inherently more susceptible to misunderstandings, misapplications, and inconsistencies. For example, administrators who responded to question 44b, which asked administrators for possible causes of the inconsistencies in the FBI’s guidance to the community, stated that perceptions shifting over time are primarily to blame for the inconsistencies observed, something that does not occur with written guidance. The FBI can increase the NDIS community’s reliance on written guidance through simple practical means, such as the improvements to information sharing previously suggested, documenting guidance given to individual laboratories through written correspondence, and by disseminating that guidance wherever applicable to the overall community.
FBI management responded by saying that they view our conclusions positively, and that our work will be very helpful in identifying ways in which they can better assist the CODIS community, particularly the specific suggestions for how they can improve handling of tools like the CODIS website.
At the initiation of this audit in May 2005, the CODIS Unit was comprised of five staff: the unit chief, three program analysts, and a management assistant. An additional seven positions were vacant, two of which had been filled pending completion of security clearances. To assess the adequacy of Unit staffing, we requested and analyzed documentation from the FBI to ascertain its past handling of CODIS Program staffing and to determine its current efforts to fill the vacant positions in the CODIS Unit.
We requested staffing information for the CODIS Program since 1997, to assess the FBI’s previous efforts in staffing the Program. The information we received revealed the following:
However, beginning in February 2004, the FBI increased the CODIS Unit staff by 7 positions, bringing its full staffing level to 12. In July 2004, a business plan was submitted to FBI management requesting the creation of two new position categories in the CODIS Unit for a total of four new employees, including a paralegal specialist and three CODIS auditors. That business plan was approved in early August 2004. The CODIS Unit's FY 2005 full staffing level of 12 positions is allocated according to the organization chart contained in Figure 11 on page 32.
The seven vacant positions in the CODIS Unit include both historical positions as well as the new positions approved in August 2004. The current CODIS Unit Chief, who assumed his position in November 2003, provided the following details to demonstrate the progress made in staffing the CODIS Unit.
CODIS Program Manager Position. The CODIS Program Manager position was an existing position that was vacant. In May 2004, the CODIS Unit Chief requested that this position be advertised, which it was in June 2004. However, the posting was cancelled because of an error, and then position was put on hold because of a new hiring process. The position was not reposted until November 2004. A selection was made in February 2005, the necessary background clearance was completed, and the new CODIS Program Manager reported to duty August 22, 2005.
NDIS Program Manager Position. Another of the existing vacant positions, the NDIS Program Manager, was advertised for 2 weeks in March 2005 and again for 2 weeks in July 2005. No one applied for the March posting, and no applicants with the required experience applied for the July posting. No further action had been taken as of December 2005.
CODIS Auditor Positions. The CODIS auditor positions were approved a s new positions within the FBI on August 6, 2004, and were advertised the first 2 weeks of December 2004. From the applications received, only one applicant was considered qualified based upon the position criteria and that person was selected for the position on March 24, 2005. The background clearance needed to allow this person to report to duty was still pending as of December 2005. To fill the remaining two auditor positions, the CODIS Unit Chief requested re-advertising the positions in May 2005 but the FBI did not repost them until November 2005.
Paralegal Specialist Position. The FBI approved the new paralegal specialist position on August 6, 2004 but did not post the position until May 2005. The FBI selected an applicant in September 2005, but the background clearance for that person was pending as of December 2005.
The FBI has not taken any action on the National Missing Persons Program Manager position. In addition, as of the end of September 2005, one of the three program analyst positions was vacated. The FBI posted that position in December 2005.
In summary, as of December 2005, one clearance was completed and the new staff member reported to duty (CODIS Program Manager). In addition, one position was vacated (program analyst) and another two filled pending clearance (CODIS auditor and paralegal). Consequently, the staffing status in December 2005 was the same as it had been in May 2005, with a total of five positions filled, two positions pending clearance, and five positions vacant. Figure 11 reflects the total positions assigned to the CODIS Unit, and the status of those positions as of December 2005.
In the several years preceding 2004, the FBI failed to staff the CODIS Unit commensurate with growing demands and participation and thereby put at risk the ability of CODIS staff to properly oversee and administer the CODIS Program. However, in 2004, FBI management took action to increase CODIS staffing and provide a sufficient number of program manager positions, including a CODIS Program Manager, an NDIS Program Manager (Custodian) and a National Missing Persons Program Manager.
Yet, progress in staffing these positions has been slow. Our results at the unit level are similar to the findings in the report of the National Academy of Public Administration ( NAPA ) on the FBI's management of human capital.23 For example, the NAPA report cites the lack of a comprehensive leadership development plan for subordinate levels of management, which we found in the historical handling of the manager positions for the CODIS Program. Further, the NAPA report states that the process to hire all other types of personnel is cumbersome, costly, and untimely, and that hiring plans are inadequate. We noted similar issues for the CODIS Program in both the historical staffing data, as well as the current staffing data. For example:
Although the FBI has taken steps to provide increased staffing levels for the CODIS Unit, attention now needs to be given to filling those positions. According to our analysis of trends in the OIG CODIS laboratory audits (see Finding II), most of the findings noted pertain to compliance with NDIS requirements, which demonstrates the need for an NDIS Program Manager. Further, according to the CODIS Unit Chief and CODIS contractor staff overseeing changes to NDIS Procedures for the FBI, FY 2005 brought more changes to NDIS procedures than has occurred in a single year previously. In our judgment, the FBI must give immediate attention to the NDIS Program Manager position, in light of the need for rigorous ongoing oversight of the NDIS community's compliance with, and the maintenance of, the NDIS participation requirements.24
The Government Performance and Results Act requires agencies to develop strategic plans that identify their long-range goals and objectives and to establish annual plans that set forth corresponding annual goals and indicators of performance. Accordingly, we asked FBI officials to provide us with the documents necessary to assess the CODIS Unit’s goals, objectives, and indicators of performance.
After the CODIS Unit was established in June 2003, FBI management decided to reassess the mission, goals, and objectives of the CODIS Program. In September 2004, Laboratory Division management approved the resulting mission, goals, and objectives. According to the revised mission statement, the CODIS Unit is responsible for: (1) developing, providing, and supporting CODIS to federal, state, local, and international law enforcement agencies; (2) managing CODIS and NDIS, including providing administrative support to the NDIS and DNA-related committees and groups and telecommunications support to CODIS participants; and (3) implementing the requirements of the DNA Identification Act of 1994, through creation and management of standards, assistance with DNA‑related legislative initiatives, and coordination with DNA-related auditing organizations.
To accomplish this mission, the CODIS Unit has one primary goal: to facilitate the use of DNA technology in assisting the criminal justice community in solving crimes. To achieve that goal, the CODIS Unit outlined eight objectives:
Of these eight objectives, only two relate to finite tasks that can be accomplished at a point in time (numbers one and three). We were able to determine from information provided to us that these tasks have been accomplished. To address the on-going objectives, the CODIS Unit maintains a record of actions necessary to accomplish the objectives in a document titled “Implementing Actions.” These actions, which are specific and numerous, reflect current and planned actions. The actions also appear to be appropriate and sufficiently detailed to allow CODIS Unit management to address the objectives in an on-going manner.
The FBI has established performance measurements, setting targets for each year and then comparing actual accomplishments to those targets. Those measurements are: (1) investigations aided, (2) CODIS matches, (3) NDIS-participating labs, (4) CODIS users trained, (5) NDIS‑participating states, (6) offender profiles in NDIS, and (7) forensic profiles in NDIS. These measurements are cross-referenced with strategic plan goal numbers or areas and categories that track to the Laboratory Division’s other management documents. Figure 12 captures data provided to us by the FBI for the CODIS Unit’s performance measurements, including the goals for FYs 2003 through 2006, and the actual achievements for FYs 2003 through 2005.
Figure 12 – CODIS Unit Performance Data, FY 2003 – FY 200627
According to the data in Figure 12, the CODIS Unit has generally achieved or exceeded its goals. Further, we determined that the CODIS Unit Chief has taken steps to ensure the measurement information is accurate, including creating a new baseline for investigations aided and CODIS matches in 2004 by querying all states for confirmed data.
Overall, the combination of documents we reviewed appear to capture the mission, goals, objectives, strategies, and performance measurements for the CODIS Unit and also appear to be interlinked in a way that allows them to be meaningful and measurable.
However, we identified three activities, which are not reflected in the CODIS Unit’s performance measurements but that are an essential part of the Unit accomplishing its mission: (1) auditing of NDIS data; (2) providing training on QAS compliance; and (3) overseeing the activities of the Review Panel. The three activities comprise the CODIS Unit’s primary means of monitoring and assisting NDIS-participants’ compliance with the QAS and verifying the integrity of NDIS data. The activities are currently performed on behalf of the CODIS Unit by FBI Laboratory staff outside it. Since they also serve a crucial role in the CODIS Unit’s interaction with the NDIS community, the activities should be formalized and clearly reflected as the CODIS Unit’s responsibilities in its objectives and performance measurements. These activities are discussed in the following sections.
Integrity of NDIS Data
Currently, as part of the corrective action measures implemented in response to our previous audit of the CODIS Program, FBI staff who perform quality assurance audits at CODIS participating laboratories also review the CODIS profiles uploaded from the cases they review (generally, three to five case files are reviewed for each active DNA analyst in the laboratory). The profiles are reviewed for completeness, accuracy, and allowability. These reviews will continue more systematically once the CODIS Unit auditor positions are filled.29 However, there is no objective tracking mechanism or performance measurement to capture this activity and the role that it is intended to play in allowing the CODIS Unit to address the requirement to verify the compliance of NDIS data with applicable federal laws and regulations. We believe this activity should be reflected with both projected and actual measurements, as well as in the objectives and implementing actions maintained by the CODIS Unit.
Compliance with the Quality Assurance Standards
The DNA Analysis Unit I (DNAUI) has been conducting quality assurance auditor training courses on behalf of the CODIS Unit. The primary focus of these courses is to ensure a consistent understanding of the QAS and consistent application of the FBI's audit document. A second important function of the courses is to instill an understanding of the principles and objectivity surrounding auditing.
No performance measures or targets have been established for this activity, even though it requires a substantial amount of effort from DNAUI staff. As of November 2005, over 950 QAS auditors had been trained in these courses.30 The DNAUI Chief, who currently oversees this training, estimates that when preparation, travel, and time used to respond to questions from the DNA community are included with actual classroom instruction time, approximately 20 to 25 percent of the work year for two staff members is devoted to managing this function for the CODIS Unit. The DNAUI Chief pointed out that in addition to lacking performance measurements for this activity, there is an overarching need for FBI management to formally recognize this activity and the resources it needs. For instance, the course needs to have staff, a travel budget, resources to develop web-based instruction tools, and funding for invited guest speakers. The DNAUI Chief stated that formalizing this activity would allow the FBI to conduct training in a more effective manner by bringing improvements to the instructional process and by delivering a more uniform product across the board.
In addition, one of the staff in the DNAUI who is involved in the QAS auditor training also serves as the chairperson for the NDIS Audit Review Panel, a panel of members from the DNA community that reviews the QAS audits completed in NDIS-participating laboratories.31 The Review Panel was created in response to findings in a previous OIG audit and serves as a means for the FBI to ensure consistent and thorough application of the QAS in laboratories across the country that participate in NDIS.32 The Review Panel processed over 100 audits in 2004 and received another 80 for processing in 2005. The Review Panel chairperson must assess the records for each audit that are received by the FBI, distribute the audits to Review Panel members, consolidate their comments, follow up on any questions or requests for information with the auditee, and document the resolution of each audit. Substantial effort is required by the Review Panel chairperson to facilitate this activity on behalf of the CODIS Unit. While the Review Panel process is a crucial component of the FBI’s confirmation of NDIS‑participating laboratories’ compliance with the QAS, this activity is not reflected in the performance measurements or objectives for the CODIS Unit.
We believe that FBI management should include these activities under the CODIS Unit’s responsibility and strategic planning process (including objectives and measurements). For example, in our analysis of the FBI’s QAS auditor training and the Review Panel process reflected in Finding III, we make recommendations for improvements to be implemented by CODIS Unit management. We do not believe that these activities must be conducted by CODIS Unit staff, but we recommend that the CODIS Unit management have the authority to make changes and track performance for these activities which is commensurate with its legislated role of oversight.
While the current performance measurements for the CODIS Unit appear to be reasonable and meaningful, we believe that the three activities we identify should be formalized under the CODIS Unit’s responsibility and included in its objectives and measurements to fully reflect the Unit’s efforts to address its mission.
When we began our audit in May 2005, the FBI informed us that CODIS contractor activity, including the maintenance and operation of the CODIS system and software, was operating under a series of extensions to a contract awarded in 1997. In our judgment, the continued use of contract extensions for that length of time, without a re-evaluation of the needs of the system or the performance of the contractor, constituted a risk to the CODIS Unit’s ability to provide for the long-term planning and development of the CODIS system. Based upon this information, we collected and assessed documentation on how CODIS Unit management oversees the CODIS infrastructure, including general operations, enhancements and development, and security and safeguards.33
Current Operations and Maintenance
The contractor for CODIS operations is the Science Applications International Corporation (SAIC), which the FBI has used for previous CODIS operational contracts. In FY’s 1990 through the final contract extension that ran through November 2005, the FBI paid SAIC approximately $71 million for its work on CODIS. During our audit, the CODIS Unit Chief provided us with a copy of that final extension. We determined that it covered not just current operations and maintenance of CODIS, NDIS, and the FBI's SDIS site under SAIC, but also arranged for the relocation of the NDIS site to the FBI’s Quantico, Virginia, laboratory and the implementation of the one-time search authorized by the Justice for All Act of 2004.
In addition, in June 2005, the FBI Contract Review Board decided to authorize a new contract solicitation that would cover the operations and maintenance of CODIS once the latest contract extension expired. The Board approved the competition for a 1-year award with four additional 1‑year options. Proposals from bidding contractors were due in August 2005. The contract solicitation spelled out the tasks that should be accomplished by the contractor, the specific deliverables, and the security restrictions that should be expected and imposed on the contractor. Some of the tasks included:
The FBI awarded this contract to SAIC in September 2005. If all options are exercised, the operations and maintenance will be covered through September 2010.
We also obtained feedback about the FBI's contractor through the survey we conducted of CODIS administrators (see Appendix VII, question 26). The average response to our question about the contractor's overall performance was a 4.5 on a scale of 1-5 (with 5 being excellent), which is a positive response of SAIC’s performance.
In addition, according to the new CODIS Program Manager, the CODIS Unit will be actively seeking input from the CODIS community on whether the SAIC help desk staff is adequate to meet the community's needs. Such feedback will be crucial, because under the operations and maintenance contract, SAIC will not be performing the scope of activities that it was under the previous contract, and the help desk will be the main tool for providing service to the CODIS community.
Implementation of Legislated Expansions
The Justice for All Act, signed into law on October 30, 2004, authorized the addition of an NDIS index for DNA profiles of indicted persons and the use of a one-time search of profiles that were not previously permitted for storage in NDIS against NDIS databases.
The FBI has made changes at the NDIS level to add the indicted persons index. In addition, we asked the CODIS Unit Chief about the implementation of the one-time search provision. He stated that direct keyboard access to NDIS is not currently possible at LDIS or SDIS sites. Rather, in order to comply with the Justice for All Act, CODIS State Administrators in November 2004 agreed to a manual or batch one-time search implementation. In May 2005, the CODIS Unit published a procedure governing the searches that specifies the type of documentation that must be maintained by the states, the certifications required to complete a search, and the rules for which profiles can be searched against which databases.
Two states began completing these searches on a test basis, and with the distribution of an updated software version in November 2005, all CODIS laboratories have the capability to complete the one-time searches. The new CODIS software provides an automated mechanism for local laboratories to create one-time search files and send them to their state laboratories and then to NDIS. The CODIS software also currently allows for the designation of appropriate specimen categories and tracks which samples have already been searched, to preclude the searching of a sample more than one time, in accordance with the federal legislation. This process was demonstrated at a national CODIS meeting by staff of NDIS‑participating laboratories.
Consequently, the primary aspects of the Justice for All Act have been functionally implemented. We note that this implementation took approximately 1 year, and included safeguards to prevent improper searches from occurring.34
Further Development of CODIS
According to the CODIS Unit Chief, the FBI’s Contract Review Board determined that the development portion of the CODIS contract should be handled separately from operations and maintenance. Consequently, the CODIS Development Contract will be awarded with FY 2006 funding, with the request for proposal expected to be announced in the spring of 2006. The development contract will focus on, among other things, developing kinship analysis for missing persons capability.
In addition, an independent assessment looked at the ability of the current CODIS architecture to support the Justice for All Act and also at the need for expanded data storage due to the incorporation of additional DNA profiles.35 Findings from that assessment will be considered in developing the solicitation for bids for the development contract. Of immediate import, the independent assessment determined the Justice for All Act could be implemented and operate over the next 3 to 5 years without exceeding capacity of the current CODIS architecture.
Safeguards for NDIS data
The FBI Security Division certified and accredited CODIS in March 2005 and granted a 3-year certificate of operation. The certification and accreditation process involves detailed analysis of the components and purpose of a system and the necessary safeguards to ensure its secure and successful operation. Therefore, the CODIS system’s certificate of operation provides a measure of assurance that the technology and security have been properly scrutinized.
In addition, the FBI stated that the CODIS data is safeguarded in accordance with a system security plan – all servers are routinely backed up, systems can be restored using established back-up procedures and tapes, and additional back-up tapes are stored off-site. Also, the FBI has established a continuity of operations location at an FBI facility. The site will duplicate the NDIS site located in the FBI Laboratory and will allow continued service to the CODIS community in the event of a disaster.
Further, the FBI moved NDIS operations to the FBI's Quantico, Virginia, facility for security and enhancements. According to CODIS Unit management, the move was completed successfully using detailed specifications for stating what equipment needed to be moved and then moving it, and testing the system before and after the move was completed. Also, during that move, the NDIS hardware was upgraded, to include built-in redundancy that has resulted in faster searches.
Internal Controls over NDIS Searches
In general, the NDIS system is designed to only allow cross-searches of certain types of profiles, in keeping with legislated restrictions. For example, relatives of missing persons profiles can only be searched against unidentified human remains profiles, not against forensic or offender profiles. The NDIS procedures clearly document the limitations in place for how the NDIS databases are searched. These limitations exist only at the NDIS level. For SDIS and LDIS, state and local laboratories are permitted to set the parameters for searching profiles at each level, based upon the state or local laws that govern those activities.
We also determined that the FBI had implemented system safeguards to ensure that NDIS-participating laboratories were performing one-time searches in accordance with the Justice for All Act, specifically preventing unallowable repeat searches from occurring.36 However, the DNA Fingerprint Act, signed into law in 2006, eliminated the need for one-time searches because any profiles that could have been searched using that provision can now be added directly to NDIS for routine searches.
The FBI has taken measures to provide for the operations, maintenance, and security of the CODIS system for the near future, by providing the following:
However, continued progress is needed to ensure that the development contract process is completed as planned and that the development contract awarded allows for continued responsiveness to legislated changes to CODIS operations.
We recommend that the FBI:
Based on our analysis of the results of OIG CODIS audits completed in FYs 2004 and 2005, as well as selected external QAS audits, we determined that: (1) the FBI’s internal controls over the proper upload of forensic profiles to NDIS are inadequate; and (2) the FBI is not tracking audit findings reviewed by the NDIS Audit Review Panel to detect common and overturned findings, and therefore is unable to ensure that QAS weaknesses or misunderstandings within the community are addressed. These weaknesses leave the FBI potentially vulnerable to undetected, inadvertent, or willful non‑compliance by CODIS participants, and consequently could undermine the integrity of the CODIS Program.
The OIG CODIS laboratory audits were initially designed to support the 2001 OIG audit, The Combined DNA Index System, which included audits of eight laboratories. Since then, the OIG has completed an additional 24 CODIS laboratory audits. (See Appendix V for a complete listing.) The objective of these audits was to determine if the laboratories audited were in compliance with standards governing CODIS activities. Specifically, we performed testing to determine if the: (1) laboratory was in compliance with the NDIS participation requirements; (2) laboratory was in compliance with the QAS issued by the FBI; and (3) laboratory’s DNA profiles in CODIS databases were complete, accurate, and allowable.
Criteria used for these audits included the QAS issued by the FBI in 1998 and 1999; the NDIS Participation Requirements delineated in the participation MOU; and OIG-developed standards for profile completeness and accuracy, and timely response to CODIS matches. See Appendix IV for further details of the audit criteria for these laboratory audits.
Our analysis of trends generally focused on those audits completed in FYs 2004 and 2005.37 We included 18 audits in our review and identified 10 common findings. The findings were in three areas – compliance with NDIS participation requirements, compliance with the QAS, and proper upload of forensic profiles to NDIS.38 Figure 13 details the common findings we identified.
Figure 13 – Finding Trends from 18 OIG CODIS Laboratory Audits
Common findings occurred with greatest frequency in the two areas of review that are audited primarily by the OIG: compliance with NDIS participation requirements and the proper upload of forensic profiles to NDIS. Currently, audits performed by scientists within the DNA laboratory community do not include any analysis of compliance with NDIS participation requirements, including profile allowability restrictions (excluding those portions of the requirements that overlap with the QAS). The FBI is in the process of hiring staff auditors for the CODIS Unit who could perform audits of NDIS compliance similar to those done by the OIG. However, the CODIS Unit Chief has stated that the plan for the CODIS staff auditors is to conduct QAS audits similar to those already being performed in the DNA community, with a limited additional review of NDIS profiles.39
Further, we determined that the FBI currently relies upon the annual CODIS user certifications as the primary means of ensuring the compliance of NDIS data.40 From the trends we noted, we conclude that this reliance is insufficient for the following reasons.
We requested and received from 41 state and local laboratories throughout the CODIS community, documentation of the external QAS audit conducted at each laboratory and cleared by the Review Panel in 2004 and 2005.41 We analyzed this documentation for trends and statistics. We determined that specific facts within the documentation, such as dates the audits were submitted to the panel, were generally consistent with the FBI’s Review Panel records.42 Based on our review we found: (1) there were a total of 112 audit findings noted in the 41 audit reports, of which 11 (10 percent) were overturned after examination by the Review Panel (see Figure 14); and (2) of the 41 audit reports, 6 had no findings (15 percent), 28 shared a finding in common with another audit, and 7 had unique findings.43
We developed a matrix of the findings from the 41 external QAS audits that were selected in our sample and noted several commonalities, as shown in Figure 14. The common findings are listed by QAS section number, with a description of the specific standard and finding that was implicated in a shared finding, the number of labs that shared in that finding, and the number of overturned findings for each QAS section. (See Appendix III for a description of each QAS section.)
Figure 14 – Trends in QAS Audits Conducted and Reviewed
As shown in Figure 14, the standards with common findings cover significant aspects of a laboratory's operations, including chain-of-custody documentation, labeling of evidence and security of evidence storage (7 laboratories); completeness of case file documentation (10 laboratories); guidelines for interpretation of mixed profiles (4 laboratories); and proper monitoring of critical reagents (3 laboratories), equipment (10 laboratories), and procedures (3 laboratories).
In a few instances, we noted that some overturned findings were not communicated to the laboratories that challenged the findings. Rather, the laboratories received correspondence that notified them that they were considered to be in compliance, with no acknowledgment that the finding was overturned. For example, four laboratories challenged the finding cited against them for compliance with Standard 22.214.171.124 The correspondence received from the FBI for those laboratories did not acknowledge this finding, either to uphold or retract it. Instead, the FBI notified the laboratories that they were deemed to be in compliance with the QAS, leaving laboratory officials to conclude that the finding was overturned. The FBI should ensure that, at a minimum, correspondence with the audited laboratories clearly documents which findings have been overturned and the rationale behind that action.
In addition, we noted inconsistency with the way the Review Panel handled some findings. For example, six different laboratories were cited for non‑compliance with Standard 6.1.4. However, when the Review Panel examined the corresponding documentation, it overturned findings for the four laboratories that challenged the finding, while making no adjustment for the two laboratories that did not challenge it. We recognize that it is not the Review Panel’s responsibility to challenge findings on behalf of laboratories, but it would be appropriate, in our judgment, to directly provide the laboratories that did not challenge these findings with the information that the Review Panel had concluded in other similar instances.
Most significantly, we noted that the FBI is not formally tracking common and overturned findings. The CODIS Unit Chief stated during our fieldwork, conducted in May 2005, that his unit does not track the findings observed in the reports that go through the Review Panel, and he did not indicate any plans to do so.
However, the current Review Panel chairperson stated that she does an informal tally of findings as a means of getting a sense of where there are commonalities. The chairperson provided information to the CODIS community at a national meeting in November 2005, confirming these statements. In her presentation, she touched on the issue related to four of the overturned findings for Standard 6.1.4 that we noted in our analysis, making it clear that documentation of cleaning and decontamination in the case file is not required. She further discussed the difference between a laboratory’s compliance with accreditation standards versus the QAS, reminding QAS auditors that there can be differences between the two. She also stated that she is attempting to give QAS auditors feedback on findings that were later overturned, but this feedback is done informally rather than systematically in a written, formal context.
In addition, we determined that the previous chairperson also informally tracked overturned and common findings in the audits to provide that information to the CODIS community. In her November 2004 presentation at a national CODIS community meeting, she addressed Standard 6.1.4, as well as the underlying issue for one of the overturned findings we observed for standard 11.1.1. She also clarified the requirements of Standard 9.5, which was included in one of the trends we noted. However, these clarifications were again handled informally, rather than through written guidance or policy updates.
We concluded that while in the last 2 years the FBI Review Panel chairpersons have generally gained a sense of the areas where common and overturned findings occur, that information is not tracked systematically and completely. Without a thorough understanding of trends in common findings, the FBI cannot properly provide the CODIS community with the additional guidance needed to remedy and prevent compliance weaknesses in the trend areas, which our analysis revealed to be significant components of a laboratory’s operations.
Further, without a complete understanding of trends in overturned findings, the FBI cannot take the necessary steps to prevent QAS auditors’ continued misunderstandings of compliance in those areas, to ensure that all QAS auditors obtain feedback on their performance, and to guide QAS auditors from other organizations – such as those who audit for accrediting bodies – toward a consistent interpretation and application of the standards.
Our CODIS administrator survey results demonstrate that the FBI should track common and overturned findings. Specifically, the results to question 30, as discussed in Finding I, reveal that 13 percent of respondents did not believe that the Review Panel has improved compliance in the DNA community, because individual (or inconsistent) QAS auditor interpretations are still enforced. This sentiment was reiterated 161 times by a total of 83 respondents in comments throughout the survey, demonstrating the magnitude of the problem posed by inconsistent interpretation of the QAS.
Informing the CODIS community of common and overturned QAS audit findings serves as a valuable tool for continuing education in QAS compliance for both the FBI’s QAS auditor training courses, as well as for national meetings where compliance is discussed. By tracking findings in a manner similar to the exercise we performed, the FBI should be able to address:
Overall, we conclude that the FBI needs to develop more rigorous internal controls to ensure that it has proper oversight over compliance with NDIS requirements. Further, the FBI should track audit findings to obtain the type of information that will be beneficial to QAS auditors and audited laboratories.
We recommend that the FBI:
Previous OIG audit findings identified the need to verify the compliance of NDIS data, to ensure NDIS user compliance with NDIS requirements, and to ensure that laboratories remedy QAS audit findings. From our analysis of the FBI’s corrective actions, we determined that it has not yet implemented routine audits of NDIS profiles and still relies on self-certification in confirming NDIS user compliance with NDIS requirements. The FBI has made improvements in the oversight of QAS compliance within the CODIS community, including conducting QAS auditor training courses, the implementation of a DNA community-wide audit document, and the creation of the Review Panel to ensure consistent and thorough application of the QAS and complete and appropriate corrective action to QAS audit findings. However, we identified the need for improved Review Panel timeliness and improved consistency in training through an emphasis on written guidance.
The FBI’s corrective action approach to the OIG’s 2001 recommendation to verify the compliance of data in NDIS was two-fold: (1) the FBI began requiring FBI QAS auditors to review CODIS profiles as part of their case file reviews (this action was initiated in June 2004), and (2) the FBI began taking steps to hire staff auditors who would systematically audit the profiles contained in NDIS.
In 2004 and 2005, FBI QAS auditors completed a total of three audits during which they confirmed that the profiles uploaded to NDIS from each case they reviewed were complete, accurate, and allowable. FBI QAS auditor involvement in confirming NDIS profiles was to be a temporary measure until CODIS Unit auditors could be hired. Therefore, we assessed the FBI’s QAS auditor approach as a temporary measure and noted that improvements could still be made.
We noted that these reviews cover three to five case files per active DNA examiner in the audited laboratory. We believe such a methodology is deficient because of its limited scope. In the OIG’s audits of forensic profiles, a minimum of 50 profiles are selected randomly for review from a list of the profiles currently in NDIS. This methodology permits a review of the work of not only current but also past examiners, as well as profiles produced by another laboratory and uploaded to NDIS by the auditee. Further, this methodology ensures that for every case file OIG auditors review, an NDIS profile has been uploaded. The FBI’s methodology could miss problems with profiles that were uploaded to NDIS on behalf of another laboratory and would not assess profiles produced by any examiner not currently on staff at the laboratory. Consequently, we consider the review methodology to be inadequate.
In addition, we observed that while there is a mechanism for documenting the results of the FBI QAS auditor’s profile reviews, there is not a mechanism for documenting and tracking how many profiles are confirmed during these reviews or the frequency with which these reviews are conducted. For example, because FBI QAS auditors can look at 3 to 5 case files per active analyst in each laboratory audited, and because laboratories vary in the number of analysts employed, there is no way of knowing whether 10 or 50 NDIS profiles are reviewed in the context of a particular audit. Considering the difficulty experienced in getting CODIS auditors on staff, we believe the FBI should be tracking this information since this “temporary” measure could continue for a period of years. Records should be maintained to indicate the scope of the profile reviews that are performed to better reflect the extent to which the risk of non-compliance is being alleviated by this management control.
The CODIS Unit Chief intends for the new CODIS auditors to continue the same scope of work to verify compliance with NDIS requirements that the FBI QAS auditors currently perform. As a result, the methodology to review profiles that we consider to be inadequate will continue once permanent CODIS auditors are on staff in the CODIS Unit. Further, the CODIS Unit Chief does not intend to review any other aspect of compliance with NDIS requirements beyond the limited forensic profile review. This approach falls short of the changes intended by the OIG in the recommendations from our earlier report. We believe the intended use of the CODIS auditors is insufficient in light of the fact that the FBI is responsible for ensuring compliance of NDIS participants and that no audits, other than the OIG’s, are being conducted within the CODIS community to specifically review compliance with NDIS requirements. For example, below we note the inadequacy of the FBI’s reliance on self-certification forms to ensure user compliance with restrictions on data in NDIS. These forms serve as one example of the type of documentation that could be audited for compliance if the FBI is to reconsider its intended use of CODIS auditors.
During our prior audit, we found that 6 of 8 laboratories uploaded a total of 55 incomplete or unallowable DNA profiles to CODIS, out of the 1,308 profiles we tested. As a result of these findings, the FBI began requiring that at the beginning of each calendar year, each laboratory’s CODIS administrator ensure that each CODIS user is reminded of the categories of DNA data accepted at NDIS.46 As part of that reminder, each CODIS administrator has CODIS users at their laboratory certify they have received their annual reminder and understand and will abide by what DNA data is accepted at NDIS. An example of this form can be found in Appendix VI.
The certification or “reminder” forms are handled on a self-certification basis. Administrators sign a certification saying that the reminder forms were completed by CODIS users in their laboratory, but the signed individual forms are not submitted to the FBI. Since the certification signed by an administrator does not indicate the number or identity of CODIS users who signed the form, there is no way for the FBI to confirm that all CODIS users have completed the forms as required.
In addition, while the reminder forms were implemented as corrective action to our previous audit, one of the deficiencies noted under that audit was the FBI’s reliance upon self-certifications from CODIS participants. As previously noted, OIG CODIS laboratory audits identified that CODIS users at 6 of 18 laboratories audited in FYs 2004 and 2005 did not complete the forms as required.
We recommend that the FBI revise its current certification process to require laboratories to list CODIS users who are certified each calendar year, which would enable the FBI to ensure that all users registered for each laboratory have completed the forms. This action should be completed in conjunction with the FBI’s response to the OIG’s current related Recommendation No. 12, for greater oversight of compliance of NDIS data.
The FBI implemented various corrective action measures in response to previous OIG recommendations for greater oversight of QAS compliance and the adequacy of laboratories’ responses to QAS audit findings. Specific changes were:
We analyzed several sources of documentation regarding the adequacy of these corrective action measures, including the results of the administrator survey. The results of our analysis are stated below.
QAS Audit Document and QAS Auditor Training
According to QAS Standard 15.1, a laboratory must conduct an annual audit to determine compliance with the QAS. Standard 15.2 requires that once every 2 years, a second agency shall participate in the annual audit (referred to as “external QAS audit”). We determined that the FBI implemented a requirement as of January 2002 that if a QAS audit was to count toward meeting QAS Standard 15.2 for an external audit, the audit must be conducted by FBI-trained QAS auditors. This measure assists the FBI in ensuring that the QAS auditors in the DNA community have been provided guidance on the application of the QAS. The FBI also implemented a requirement that the audits conducted in the CODIS community be performed using the FBI’s audit document. This document contains comments and guidance on the accepted interpretation of the standards and also assists the FBI in ensuring consistent and thorough application of the QAS to CODIS-participating laboratories. Both of these measures are significant in their scope and have allowed the FBI to greatly improve the DNA community’s overall compliance with the QAS since our previous audit.
Based on the survey results and direct OIG experience with the QAS auditor training courses, we noted the need to ensure that training is based on a comprehensive written curriculum and that the supplemental guidance provided in the context of discussion sessions be documented for future reference and verification of consistency. Currently, the auditor course is based on a presentation given by the course instructors and is linked closely to what is contained in the QAS audit guide maintained by the FBI.
However, speaker notes that provide context and helpful interpretive guidance to course attendees are not available for public reference. Further, the course instructors can include extemporaneous verbal guidance regarding specific standards that is not included in the presentation materials or in the audit guide on which the training is based. The verbal guidance or explanation given in these courses can result in misunderstanding and therefore misapplied guidance. For example, in a course attended by an OIG manager, the speaker responded to a question regarding the use of contract employees for reviewing casework profiles. That answer led to confusion as to the extent of the FBI’s policy. The OIG attempted to contact various FBI personnel to clarify the point, but the incident served as an example of the misinterpretation that can occur when verbal guidance is given that is not directly linked to written guidance. The inconsistency between written and verbal guidance can impact both the QAS auditors, hindering their consistent and thorough assessment of compliance, as well as the auditees’ understanding of their obligations under the standards. Therefore, we believe the FBI needs to ensure that any significant verbal guidance given in each course is presented consistently with written guidelines.
In addition, we obtained from the FBI’s DNAUI Chief ways in which he believes the course could be improved. Particularly noteworthy was the suggestion for web-based training tools, especially since 37 respondents to our CODIS administrator survey made a total of 51 comments regarding the use of the CODIS website to offer better training and guidance resources. Based upon the support for this concept, we believe the FBI should design and implement web-based training tools as a supplement to the QAS auditor training courses being conducted. Such tools would allow those in the CODIS community who have not yet taken the QAS auditor training course to have access to the guidance and clarification they need to ensure compliance. Administrator survey results indicate that 43 percent of those who responded have not taken the QAS auditor training course.
NDIS Audit Review Panel
In January 2002, the FBI instituted a requirement that all external QAS audits performed at NDIS-participating laboratories be provided, along with corrective action documentation, to the Review Panel for examination and clearance. The Review Panel is comprised of volunteer qualified-DNA examiners who have completed the FBI’s QAS auditor training. Each audit is reviewed by four Review Panel members, two from the FBI and two from a state or local forensic DNA laboratory. The Review Panel members provide their analysis of audit findings and corrective action and forward them to the Review Panel chairperson, who consolidates members’ analyses and oversees interactions with the audited laboratory. Requests for more information or clarification come from the Review Panel chairperson. When the audit is closed (i.e., the FBI considers the laboratory to be in compliance with the QAS), correspondence to that effect is sent by the NDIS Custodian (currently the CODIS Unit Chief).
Initially, our analysis of FBI data indicated a significant backlog and delay in reviewing and closing the audits submitted to the Review Panel. In our judgment, such delays hinder the FBI’s ability to ensure that CODIS participants are currently compliant with the QAS. The CODIS Unit Chief stated that he had taken steps to improve the efficiency of the Review Panel, including a tracking system to ensure timely and complete analysis and response to audits, and assigning a chairperson to the Review Panel who can oversee it. Upon further review, we determined that improvement has been made, as reflected in Figure 15.
Figure 15 – Analysis of Improvement to NDIS Audit
As can be seen, significant improvements reduced the overall average number of days spent from receipt of the audit to close of the audit from 232 to 91 in just 1 year, while the number of audits cleared remained fairly static. Yet, we noted the following opportunities for additional improvement.
In conclusion, we believe the FBI should take action to ensure that its implementation of past corrective action measures fully addresses the weaknesses identified in the OIG’s previous audit report and to address additional needs identified in this audit.
We recommend that the FBI:
|« Previous||Table of Contents||Next »|