Combatting online crime with “needle-rich haystacks”

18 October 2016

Needle in the haystack

At Digital Shadows our analyst team is responsible for providing both tactical situational awareness and broader, strategic awareness to our clients through incident reports, intelligence summaries and specific reports. The intelligence our analysts produce is largely based on automated collection from our wide range of sources across the visible, dark and deep Internet, so a key challenge is to identify true positives and to objectively assess the available data to draw analytically supportable conclusions.

We use a range of machine based techniques to sift through data points identified through automated scans, including rules based keyword matching, regex based searches and highly targeted queries for specific entities, such as credit card numbers featuring the Bank Identification Number of a client institution. The objective is to create “needle-rich haystacks” in order to maximize our true positive to false positive ratio. Our analysts then sift this data for potential indications of data leakage, brand damaging content and both cyber and physical threats to our clients and write incident alerts to provide real-time awareness of a client’s digital shadow and any potential threats they may face. This can sometimes be a much tougher task than it sounds as the team is frequently presented with very large numbers of data points that need to be assessed (the Internet is a pretty big place), but our analyst team continuously liaises with a specialist intelligence collections team to maintain the best possible ratio of signal to noise.   

For the analyst team, finding a true positive is only half the challenge – analysts must then assess the available information to determine the nature and extent of the threat and how it could be best mitigated. Not only does this require a team with fluency in many languages, but it also requires the use of a wide range of structured analytical techniques. These include Strengths, Weaknesses, Opportunities, Threats (SWOT) analysis, Analysis of Competing Hypotheses (ACH) and Redhat analysis, to facilitate objective assessments and to ensure that every conclusion we draw is analytically sound and supported by the available evidence. We express these assessments precisely through the use of the Language of Uncertainty and source grading to ensure that our findings are clearly, concisely and accurately conveyed to our clients. In addition to these tradecraft techniques, we also use a range of specialist tools to streamline the analytical process and enable our team to dig deeper into each data point to ensure that nothing is missed. Our analysts receive training in analytical tradecraft, the use of specialist tools and the technical aspects of information security to ensure that they are equipped to handle the analytical challenges they face.

LoU

Our assessments are supported by a curated intelligence base of profiles and incident records for prominent threat actors, tactics, techniques and procedures (TTPs), criminal websites, threat actor operations and ongoing events. These profiles inform analyst decisions when assessing the severity or nature of a threat and enable our analyst team to remain familiar with the ever-changing cyber threat landscape.

Working on the Digital Shadows analyst team is challenging and requires a range of skills, including analytical tradecraft, technical awareness and a keen eye for detail, but it is consistently engaging, rewarding and enjoyable, as we constantly encounter new situations and must adapt our existing capabilities to new analytical problems in order to continually provide our clients with situational awareness of the threats they face every day.

We’re always looking for talented individuals to join our growing team; check out our careers page to find out more.