3.15 Reliable probabilities

Probability in forensic science is subjective and personal. Anyone is capable of assigning a probability to their personal uncertainties. Even if everyone does this to the best of their ability, some people are better suited to assigning probabilities to certain events than others. This supports the idea of experts versus non-experts: the expert’s probability assignment for a situation within their expertise will be more reliable than that of the non-expert. The expert has more knowledge about a field and so their best assessment is better than that of an uninformed non-expert. Reliability of probability assignments is a key part of interpreting and using expert evidence.

Factors that affect the reliability of a probability assignment include:

• assessability
  • This property describes whether sufficient information is available about the probability assignment in order to be adequately assessed. Are all assumptions stated explicitly? Is the probability explicitly stated? Are any underlying data independently accessible? Probabilities must be assessable in order to be classified as reliable or not.
• background information
  • This property refers to the reliability of the background information of a probability assignment. Is the background information that underpins the probability generally agreed in the expert’s field or is it more subjective? What reasoning has been used to include the background information? Do reasonable changes to the background information greatly affect the probability assignment?
• calibration
  • This property refers to the degree of error in expert probability assignments. For example, weather forecasters are tasked with providing probabilities for rain in a particular region of interest throughout each day. Their calibration can be measured by comparing their historical probability assignments to whether rain actually occurred or not over a long period of time. Forecasters who assign high probabilities of rain when it actually does rain and low probabilities when it actually does not rain are well calibrated. Better calibration leads to greater reliability. Experts in forensic science can demonstrate this in multiple ways, e.g. by using an empirically well-calibrated statistical model and by performing competency tests.
• validation
  • This property aims to verify the entire process of the probability assignment (and not just the quantitative measurements as in calibration above). This includes checking the consistency, calibration, and appropriateness of the probability assignment to the case at hand. Depending on the case, this could mean validating the reliability and relevance of data that has been used as well as the calibration and output of any statistical models.

None of the above properties can guarantee that a reliable probability assignment has been made under all circumstances. They only provide a general idea of what to consider when assessing reliability. Assignments in real cases require assessing the above properties as well as others not listed here.