The story
On the 23rd of April 2015 the Director of the Department of Forensic Sciences (DFS) in Washington DC resigned and senior managers and legal counsel ‘fired’. The departure was the culmination of a series of events that began in October 2014 with a complaint about the way the DFS interprets mixed DNA profiles. Mixed profiles are those with more than one contributor.
On the 8th of May 2015 the Washington Post, under the headline “A setback for forensic science”, published an opinion piece by William Thomson, professor of criminology and law at the University of California. In it he set out the key events that led to the departure and the related suspension of DNA testing at the DFS. In Professor Thompson’s opinion at stake is the independence of the DFS as a forensic science laboratory.
The headline “A setback for forensic science” refers to the fact the DFS was established in the light of recommendation #4 in the 2009 National Academy of Science (NAS) report. The NAS report recommended the removal of administrative control of forensic science laboratories from law enforcement agencies and prosecuting authorities. According to Professor Thompson, prosecutors were not happy with the loss of control. A point supported by Jay Siegel, a member of the DFS’s Science Advisory Board who resigned over the issue.
The contention is that these bodies used legitimate differences of opinion regarding the interpretation of mixed DNA profiles as a means of gaining greater control over the operation and management of the DFS, to effect a resumption of ‘normal service’.
An alternative analysis, and that advanced by the prosecuting authorities supported by the Mayor, is that the method used by the DFS for the interpretation of mixed DNA profiles was wrong. The failure by senior management at the DFS to accept and act on that opinion made their position politically untenable.
This analysis is supported by a quality audit conducted by ANFSI-ASQ National Accreditation Board (ANAB) at the request of the Mayor and a review of evidence by experts at the request of prosecuting authorities.
This episode raises a number of major issues which might have consequences for forensic science provision beyond the District of Columbia.
Matters arising
The issues may be summarised as; political interference in the operation and management of a forensic science laboratory and an accrediting body.
The complicating factor in this episode is that the interpretation of mixed DNA profiles is a controversial area and as the DFS Science Advisory Board reported to the Mayor on April the 26th 2015:
“Within the scientific community, there is, to date, no single, universally accepted set of best practices….”.
Experts within the field know this full well. There are two fundamental approaches with world-leading and highly regarded experts supporting each approach. It could be argued that, until consensus is reached, DNA evidence derived from mixed profiles with possibly more than two contributors should not be placed before a court.
The results generated depend on the approach used and the assumptions made. The figures reported in support of the allegation of DFS error are 1 in 3,290 (DFS) and 1 in 9 (other experts). As DNA evidence goes that is hardly any difference at all. The correction of recent errors discovered in FBI DNA evidence is likely to generate far greater differences which the FBI claim are not significant.
If the Director and senior staff at the DFS have been removed by politicians using a legitimate difference of opinion within the relevant scientific community as a pretext for bringing forensic science provision back under the control of prosecutors and law enforcement agents then that indeed is a backward and worrying step; a step which puts at risk the quality of forensic evidence. In addition to the NAS report, there is a body of scholarship pointing to the closeness of the relationship between law enforcement and forensic science providers as a contributor to miscarriages of justice.
Quality matters
The second issue, perhaps of greater concern and certainly wider in scope, is the role of the accrediting body ANAB and the consequences of its involvement.
The key questions regarding ANAB are:
- Is it subject to inappropriate political interference?
- Is it a competent accrediting body (compliant with the standard ISO17011)?
- If it is not competent what effect might that have on the reliability of forensic evidence offered by other labs accredited by ANAB and the non-DNA evidence of the DFS?
ANAB is the DFS’s accrediting body. ANAB assesses compliance with an international standard identified as ISO17025. ISO17025 is the standard to which all forensic science laboratories in the developed world are accredited to or soon will be. Compliance assures the reliability of the forensic evidence offered by the DFS.
Compliance with ISO17025 essentially requires three things;
- that the laboratory management is competent,
- that the staff are competent, and
- that the methods and procedures are valid; i.e. capable of producing evidence that might be relied on.
Of particular relevance is that compliance with both ISO17025 (4.1.5 b) and ISO17011 (4.3) require measures to be in place to protect management from undue influence that might adversely affect the quality of their work. Fairness, impartiality and transparency are basic requirements in quality management.
From the 31st of October 2013 (the date on which accreditation was granted) until the report that resulted in the suspension of DNA testing was delivered on the 24th of April 2015, ANAB was satisfied that DFS remained compliant with ISO17025 and quality audits undertaken since accreditation was granted had not revealed any major issues; i.e. those that might warrant suspension of accreditation.
During the period of accreditation an accrediting body will regularly conduct quality audits, usually annually, to ensure continuing compliance. It is often the case that, as a result of these audits, minor non-compliances are identified which need to be cleared in a set period of time to maintain accreditation. In addition to quality auditors from the accrediting body, ISO17025 accredited laboratories will have a team of trained and qualified internal quality auditors conducting an agreed program of audits. The aim is for the entire quality management system (organisation, staff and methods) to be audited over the period of accreditation.
ISO17025 is seen as an essential ‘seal of approval’ by forensic science laboratories. In addition, compliance is often used as a means of preventing scrutiny by independent reviewers. Accredited forensic science laboratories claim that accredited methods should not be subject to scrutiny by an independent third party because an accrediting body (an external and independent agency) assures the validity of the methods used and consequently the reliability of evidence. This episode completely undermines that claim.
The competence of ANAB
Based on the information publicly available, the relationship between ANAB and the parties; the Mayor, prosecution authorities, DFS senior management and the DFS Science Advisory Board, is unclear. However, if the DFS and ANAB are compliant with their respective standards then the quality manual of each organisation will have those relationships clearly detailed and the measures in place to protect against undue influence and conflicts of interest. It would be very interesting to have sight of these quality manuals to see how the parties all relate to one another.
The duties and responsibilities ANAB owe to its customer are set out in the standard to which ANAB is accredited, ISO17011. As ANAB is the accrediting body for the DFS those duties and responsibilities should be towards the DFS. ANAB needs to respond to the suggestion that in its dealings with other parties it was not compliant with its own standard.
ANAB’s web page headed “ISO/IEC 17025 Accreditation for Forensic Testing Laboratories” is actually about ISO17020, a different standard.
The major evidence that calls into question the competence of ANAB is the eighteen months (and more) during which quality auditors from ANAB and internal auditors of the DFS missed non-compliances of such magnitude as to warrant suspension of accreditation.
DNA profiling evidence is often of the highest probative value and that based on mixed profiles controversial. These factors should enure that the parts of the quality management system relating to this evidence type are kept under close review by quality auditors.
One final twist
On the 11th of December 2014 it was disclosed that a romantic relationship existed between the US attorney who first complained about the DFS method of interpreting mixed DNA profiles and who is responsible for sending samples for DNA profiling and the supervisor of forensic casework at Bode Technology. Bode Technology does the DNA profiling formerly undertaken by the DFS. On the face of it this suggests a major conflict of interest. Leaving aside quality management issues, there is an old legal aphorism “not only must justice be done it must be seen to be done”; well apparently not in the District of Columbia.
Conclusions
In addition to any non-compliance with ISO17025 in respect of DNA test methods this episode raises the possibility that there may be other non-compliances of the DFS with ISO17025 and ANAB with ISO17011.
An independent review of the quality management systems of both the DFS and ANAB is urgently required.
The facts of this episode undermine the assertion that accredited methods can be accepted as valid and scrutiny by an independent third party is unnecessary.
If the DFS and ANAB have been subject to political interference in their operation and management then the reliability of forensic evidence offered by the DFS and other ANAB accredited labs must be in question. How can an expert witness from those labs claim to be independent?
If the object of the Mayor, prosecutors and/or law enforcement agents was to wrest administrative control from the DFS, contrary to the recommendation of the NAS report, then that is of concern and puts justice at risk.
Based on 25 years’ experience I am of the opinion that quality management in the USA is not in good shape and the International Standards Organisation needs to look carefully at US based accrediting bodies which seem to take an ‘a la carte’ approach to compliance.