The Devil, the Details, and the Analysis of Competing HypothesisFebruary 13, 2020
Digital Shadows’ Photon Research Team recently released a comprehensive examination of the Analysis of Competing Hypothesis (ACH) method, in a full paper: Applying the Analysis of Competing Hypotheses to the Cyber Domain.
Striving for analytical rigor in intelligence analysis is a given, but achieving it can be a challenge. To grease the wheels, we have such techniques as critical thinking and bias recognition, plus many other ways to improve the output of an intelligence assessment.
In many ways, ACH is the grandfather of all such methods. Initially developed by a veteran of the CIA, Richards (Dick) J. Heuer Jr, and outlined in the seminal Psychology of An Intelligence Analyst, the ACH method has gained traction across the intelligence sector since its introduction in the 1970s.
The basic seven-step method is as follows.
The final output from this process is similar to the one illustrated below, in that pieces of evidence are evaluated against a set of hypotheses, and the least disproved hypothesis ends up as the “winner”. In the example below, ACH takes on the fallout from a data breach of the fictitious “Organization A” and uses four pieces of evidence to consider two hypotheses, about the origin of the breach from an either internal or external source.
In the ACH table, + means the evidence is consistent with the hypothesis, ++ means it’s extremely consistent, – means it’s inconsistent, and — means it’s extremely inconsistent. We can see that Hypothesis 1 is better supported than 2, with most evidence (1, 2, and 3) behind it. And we can also see that not all pieces of data are going to be applicable to the hypotheses (in this case, 4).
ACH and Cyber Threat Intelligence
The ACH method is generally applicable to the field of intelligence as a whole, but there are caveats, and one is cyber intelligence. Heuer developed the method based on his background working on intelligence problems grounded in the physical world. “Bomb and bullet” events like the Cuban Missile Crisis were the clear stimuli for the method, and dominate his approach to it. The problem is that cyber intelligence is a few degrees more granular by its nature than conventional intelligence.
Take, for example, the origin of an AK-47 assault rifle. This is a detail rarely of relevance to the tactical response in a terrorist attack scenario. But the origin of a piece of malware is often critical to how a response develops in cyber-threat incidents, such as the Sony Pictures hack of 2014. It’s not that physical incidents don’t have a granular dimension, it’s just that the level at which details become relevant to the overall assessment is much more ‘in the weeds’ in a cyber context than in a physical context.
This shifting perspective of relevant detail that lies within cyber intelligence is the primary pitfall of applying ACH analysis to the cyber domain, but it’s also the main reason to apply it. We’ll conclude this primer with some critical areas where the Photon team feels the ACH method can show benefits in the cyber domain (see below). To explore the method further, download the full paper, Applying the Analysis of Competing Hypotheses to the Cyber Domain
- Attribution-related issues: A notoriously difficult element of cyber-threat intelligence, attribution is made even more complicated by the ambiguity of evidence associated with the discipline and the potential confusion of a crisis situation. Attribution for a cyber-threat incident is often based on a single source, such as a report from a government intelligence agency, that pins attribution on a particular threat actor/group but refuses to provide any hard evidence for the claim.
This “Take my word for it”-style approach was used in cases like the United States’ claim that North Korea was responsible for the Sony Pictures hack in 2014, the United States and United Kingdom’s attribution of the “WannaCry” worm to North Korea in 2017 and the claim that state-backed Russian hackers hijacked Iranian state hackers’ infrastructure to conduct attacks in 2019. Even if these claims were entirely correct, they’re still single-source reporting―an inherently uncomfortable basis for the conclusions of an analyst. ACH often clearly reveals this kind of dependency―a helpful thing to know if you’re presenting an assessment to a critical audience.
- Contentious issues: Aside from straight attribution issues, many aspects of cyber analysis are highly subjective. For example, in the “NotPetya” ransomware attacks, circa 2016, analysis of the malware showed that the Bitcoin address provided to make ransom payments was nonsensical. Was this because the malware’s creators made a mistake when creating the code, or was the “mistake” intended, to mask the true aim of the malware? Such examples illustrate how contentious cyber analysis can be, and in these situations ACH can add significant clarity.
- Complex issues: ACH deals with complex issues well, primarily because it’s a method that can isolate evidence. This is critical for understanding how pieces of evidence overlap and interact with each other. A good example of this is the United States’ targeted assassination of Iranian Major General Qasem Soleimani. Although this was a physical act, if you consider the cyber power of the Iranian and United States, there is potential for the conflict to spill into the cyber realm. With multiple complex dependencies, ACH allows a segregation of individual aspects of the problem.
For a deeper dive into ACH, check out the full paper below.