Are CyberSecurity Ratings Useful?

A Case Study of a SecurityScorecard Report.

SecurityScorecard recently published a report on the “State of the States” which they asked me to review and comment. I also took time to talk to SecurityScorecard chief data scientist, Dr. Bob Shoval, to understand how they approach their scoring and make an informed review.

Cybersecurity attacks by criminals and state sponsored actors against government or state infrastructure is reality. We’ve seen ransomware attacks against the NHS, UK state run health service, with Wannacry and more recently a US hospital group UHS with Ryuk. In 2015, the German Bundestag (parliament) was hit by an attack that led to over 16gb of data loss. In 2017, we saw state sponsored attacks on the French presidential elections. The threat on government entities is real and we should not ignore it.

A key activity in understanding the threat is preparation. Preparation revolves around understanding the weakness in the infrastructure and the vulnerable assets which we then use to build a remediation plan. A SecOps team will gather as much information about potential weakness in the infrastructure to be defended. Understanding these weaknesses is key to preparing the right detections and controls to detect an attack. I would use a scorecard or gap analysis to carry out these preparedness activities and use it as a base to then evaluate against the potential impact against the organisation to determine remedial actions or controls that need to be put into place.

The SecurityScorecard report can help in this endeavour. It provides a good insight into the external security posture of the organisation. SecurityScorecard’s methodology reflects the way that criminals and state actors might carry out reconnaissance activities (e.g. ATT&CK reconnaissance tactics); think Lockheed kill chain. Attacker techniques for reconnaissance vary and include active scanning, vulnerability identification, network identification and endpoint information gathering.

I reviewed the issues that are used to create the scoring with this in mind. The issues that are evaluated reflect common areas that an attacker would focus on during reconnaissance. They are similar to what I would use to build a gap analysis as part of a preparedness planning. While organisations may vary their data points, most will use common attack patterns or common vulnerabilities that criminals or state actors might use to carry out reconnaissance.

It is important to understand that the key aspects in reconnaissance for an attacker is target selection, information gathering and weakness identification. The attacker is trying to find where the weaknesses are and which infrastructure will be easiest to breach. SecurityScorecard takes a similar approach and gathers information on the common attack weak points and determines how predominant they are on the external facing infrastructure for the organisation or government agency; looking for those low hanging fruits or in this case vulnerabilities. Identification is carried in much similar way as an attacker might through common tools like DNS, IP attribution and what application versions are facing the internet.

Sample SecurityScorecard Detection Issues
Sample SecurityScorecard Detection Issues

The report bases the score it assigns by evaluating the number of issues it detects on the organisation’s externally visible footprint with modifiers based on severity. The score is then normalised relative to all the data points gathered to provide a letter grade. Note SecurityScorecard can provide a more detailed explanation on how this is achieved.

SecurityScorecard and similar reports provide a snapshot in time of an organisation’s external infrastructure and the possible vulnerabilities. While one can draw a risk picture from the results, it does not necessarily give an accurate reflection of the potential impact these vulnerabilities might expose. So it is important to understand your score and put into perspective or your organisational controls and capabilities.

An example might be the calculation of the endpoint security; while the issues do highlight potential weakness the measurement points may contain inaccuracies as it is easy to modify or obfuscate these values. A score like this also does not necessarily take into account organisational constraints like embedded devices that can’t be updated nor does it take into account security controls that might be deployed, e.g. sandboxing. SecurityScorecard does provide a means to apply adjustments based on the organisation’s feedback on mitigations in place.

Government infrastructure is generally and worldwide under funded when it comes to CyberSecurity. So low scores are to be expected. Governments are trying to remediate this but it is a slow and democratic process. The US government recognises that there is a need to understand the cybersecurity maturity of the infrastructure. The Cybersecurity Maturity Model Certification (CMMC) was developed as an answer to be able to to measure and benchmark the current maturity of government entities and partners.

SecruityScorecard Z-Scale Normalisation
SecruityScorecard Z-Scale Normalisation

The scores in the report are normalised against all private and public sector organisations. Personally, I would prefer to see a normalization based on similar verticals. Private companies do not have the same constraints as government agencies when it comes to spending on infrastructure and security. Attacker goals against governments are also different to those for private sector companies.

The scoring method is sound and represents a picture of the potential weakness footprint for the infrastructure. From the perspective of a SecOps engineer, this score would provide me with good starting points on where to focus my controls and improvement activities.

Principal security engineer & threat researcher providing information security engineering & incident response leadership. Spending my time researching threats!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store