Quantitative vs Qualitative Risk Analysis

While reading the NIST Guide for Conducting Risk Assessments, one line jumped out at me in § 2.3.2 Assessment Approaches; "The rigor of quantification is significantly lessened when subjective determinations are buried within the quantitative assessments". I was blown away by the succinct eloquence of something I've struggled against in academia and business since learning proper research methods. Quantitative research has a valid and powerfully useful place only when the accuracy of the quantification is near-perfect. The further from reality a conceptual numerical measurement is, the further the analysis findings drift towards false metrics . 

During a Risk Assessment for example, one valid exercise is for a business to create a quantitative analysis of the cost of replacing hardware in the event of a catastrophic failure. The number of machines in use and their respective costs are real and known numbers. This can in turn provide you with an analysis with a hard figure that you can take to an insurance company and not get overcharged or worse, not adequately control your risk. The trickery comes when these statistical tools are used to quantify the risk of something ephemeral like loss of reputation if your business is breached. 

Reputation is undoubtedly core to the value provided by any organization, and could be harmed by a breach. However breaches are generally only damaging to reputation of a business when the business is exposed to having embarrassingly poor security controls. Storing user passwords in cleartext, or a business completely fumbling their response to a breach cause customers to lose trust that you can be a reliable partner, potentially losing their business. If a business is breached because of a zero day Windows vulnerability, and attackers were only able to access encrypted customer data for a short time before the issue could be mitigated by a response team, your business reputation could theoretically increase in relation to your peers who neglected to encrypt their customer data. 

You can easily spend several hundred person-hours trying to calculate and quantify dollar amounts for each identified risk with a business. Which again may not be based on scientifically objective measurements, and can be distorted by arbitrary statistical assumptions. In instances where the risk/value ratio of an asset is even mildly ambiguous, risks should be assigned to a simple thee-tier (low-medium-high) severity and thee-tier priority matrix. Severity being judged on how much business would be lost if the vulnerability identified by the risk were to be exploited, priority is the likelihood of that happening. A report like this uses significantly less resources to create, and provides all the information security personnel need to begin deploying security controls that address your high-risk and high-priority vulnerabilities. 

Striking a healthy balance between these methods during assessments, and applying them appropriately to their use cases, is the smart and cost effective way to protect your business. While having numbers looks great in PowerPoint presentations given by metric-obsessed managers, over-reliance on quantitative methods is at best overly expensive, at worst potentially blinding leadership to the actual threats they face.