5.14 Robustness
The LR is based upon two underlying conditional probabilities. Each of these probabilities has its own degree of reliability, which depends upon factors listed in Section 3.15. The degree to which these probabilities are reliable determines the overall robustness of an LR. The more reliable each of the probabilities is, the more robust the LR that is produced. It is worth noting that an LR which is not robust might still be the best available LR in the case circumstances, e.g. if there is limited data and expertise with which to inform the probability assignments.
Assessing the robustness of an LR is aided by making a distinction between types of information that the expert uses when constructing conditional probabilities. In the Judicial Statistics Primer (Nic Daéid et al. 2020) this information is divided into two categories:
- S: knowledge derived from robust systematic studies, ideally published, where the relevant features have been measured and studied statistically
- E: knowledge derived from personal experience, i.e. the expert’s training and professional experience in the forensic specialism.
The robustness of an LR that has been informed by the S category can be assessed using the quoted studies and data. The study design and its limitations might be relevant, as well as its applicability to the case circumstances at hand. The robustness of studies and databases can be affected by factors such as the study sample size, the precision of collected measurements, any bias in sampling methodology, etc.
The robustness of an LR that has been informed by the E category can be assessed by focussing on the experience of the expert. This could involve the expert detailing their previous experience and its relevance to the present case. This evidence could include previous proficiency or calibration tests in which the expert has participated. The robustness of these tests may be assessed by considering their study design, such as whether the tests were conducted blinded or not. This category is also more likely to be where different experts disagree on scientific matters where there is limited evidence, and there might be varying degrees of consensus within the expert’s scientific field. Disagreements between experts may be reasonable and can be presented to the court for examination.
The robustness of a particular LR can also be empirically assessed by performing a sensitivity analysis. This involves making reasonable changes to the probabilities underlying the LR and observing how the LR varies with these changes. The changes in the probabilities are made to reflect uncertainty that might reasonably have been unaccounted for. A sensitivity analysis shows how sensitive the LR is to the method or data which was used to calculate the LR. An LR whose value stays within an acceptable degree of sensitivity and is also shown to be well-calibrated is more robust than an LR that is highly sensitive and also poorly calibrated.
There are two other ways in which an expert may assign more robust LRs. The first is by assigning the LR to an interval of values rather than a single value. This might be done in situations in which the expert is unable to assign precise values for some sources of uncertainty, for example where relevant data is limited. The intervals shown in Table 5.3 are used for this purpose as well as for consistency of language. The second method is by assigning LRs conservatively. This means assigning the LR to a value that is closer to 1 (neutral), e.g. if there is a reasonable interval for the LR then a value that is closer to 1 is favoured. Taking this conservative approach ensures that the LR is protected from overestimation or underestimation that might unreasonably be biased against the prosecution or defence.