Why Your Penetration Test Severity Ratings Might Be Misleading You, and a Better Way to Think About Them

If you’ve ever received a penetration test report and felt like the severity ratings didn’t quite match your intuition about what was serious, you’re not imagining things. Severity ratings are one of the most consequential parts of any pen test report. Organizations use them to decide what to fix first, how urgently to act, and how to communicate findings to leadership. That makes it worth understanding where most scoring systems fall short, and what a more practical alternative looks like.

 

Not sure how your current pen test reports stack up? MainNerve has been delivering clear, actionable penetration testing reports for over 20 years. Reach out for a conversation. No pressure, no sales pitch.

The Problem with How Most Severity Ratings Are Built

Most commonly used scoring systems in penetration testing weren’t designed for penetration testing. They originated in vulnerability management programs and enterprise risk frameworks, environments where analysts have access to extensive organizational context: detection capabilities, compensating controls, the business sensitivity of affected systems, and organizational risk tolerance.

During a penetration test, testers typically don’t have reliable access to any of that information. They’re working from the outside in, or from a position of limited internal access, and they can observe what’s technically present in the environment, but they usually can’t verify whether a monitoring system would catch the exploit, whether a compensating control elsewhere in the network reduces the real-world impact, or how business-critical the affected system actually is.

When scoring systems require testers to factor in things they can’t verify, two problems follow. First, testers have to make assumptions, and assumptions introduce subjectivity. Second, inconsistent assumptions lead to inconsistent ratings. The same vulnerability might score differently depending on which tester writes the finding, what assumptions they make, and how conservative or aggressive their interpretation is. This is how clients end up debating severity ratings with their security team instead of fixing the things that matter.

A Framework Built for What Testers Can Actually Verify

The approach we use at MainNerve evaluates each finding based on three factors, as these are the elements a tester can observe and verify directly during any engagement.

Exposure: How reachable is the vulnerable condition to an attacker? This accounts for whether the system is accessible from the internet, whether authentication is required to interact with it, and whether access is limited to internal networks. A vulnerability that any unauthenticated user on the internet can reach is fundamentally different from one that requires an attacker to already be authenticated on an internal network. Exposure captures that difference.

Exploitability: How difficult is it to perform the attack once the vulnerable condition is reachable? Some vulnerabilities can be exploited with a single command or request. Others require complex timing, specific conditions, or multiple sequential steps to execute successfully. Exploitability measures the practical difficulty of going from “found it” to “exploited it.”

Impact: What happens if the attack succeeds? Impact captures the technical consequence of successful exploitation, whether that’s exposure of sensitive information, unauthorized modification of data, service disruption, privilege escalation, or full system compromise.

Each factor is scored from 0 to 3. The final severity score is the sum of all three, producing a total between 0 and 9 that maps to five severity levels:

That’s the whole model. Three factors, a ten-point scale, and five severity levels that align with what professional pen test reports already use.

Why Simpler Is Better Here

The instinct when building a scoring system is to add more variables. More factors mean more precision, or at least the appearance of it. In practice, the opposite is true for penetration testing. Every factor you add that requires a tester to make an assumption about something they can’t directly observe is a factor that introduces inconsistency and invites debate.

This framework produces ratings that a tester can explain in a single sentence: the rating reflects how exposed the issue is, how difficult it is to exploit, and what happens if the attack succeeds. That explanation is intuitive for technical recipients who want to understand the methodology and for non-technical stakeholders who just need to know how seriously to take a finding. It also means that two different testers looking at the same vulnerability will reach the same rating, because they’re evaluating the same observable conditions using the same criteria.

 

Want to see this framework in action? Our reports are built around findings your team can actually use, plain language, prioritized results, and remediation guidance that makes sense. Let’s talk about your next engagement.

Severity Is a Technical Metric, Not the Whole Story

One important distinction this framework makes explicit is that technical severity and business risk are not the same thing, and pretending they are is one way pen test reports lead organizations astray.

A Critical severity rating on an isolated lab machine that holds no sensitive data and is not connected to production systems represents less actual business risk than a Medium severity finding on the database that holds every customer record your organization has ever created. The severity rating tells you about the technical characteristics of the vulnerability. The business risk depends on what’s at stake if that vulnerability is exploited, and that context belongs to the organization, not the tester.

This framework is designed to give organizations a clean, honest technical metric that their internal risk owners can then layer with business context. The tester’s job is to deliver an accurate, evidence-based severity rating. The organization’s job is to apply its employees’ knowledge of asset value, regulatory exposure, and operational impact to determine how urgently to act. Keeping those two functions distinct produces better decisions than asking testers to guess at the business context they don’t have access to.

It’s also worth being direct about what a severity rating is not. It’s not a substitute for a clear description of the finding. Every finding in a penetration test report should include a plain-language description of the attack scenario, the technical details of how exploitation was performed or could be performed, and specific, actionable remediation guidance. The severity rating is a tool for prioritization. The finding description is where the real information lives.

What This Means for Organizations Reading Pen Test Reports

If you regularly receive penetration test reports, understanding how severity ratings are generated helps you use them more effectively. When you see a High or Critical finding, ask what drove that rating. Was it the exposure level, the ease of exploitation, or the severity of impact? The answer changes how you think about remediation priority. A Critical finding, driven primarily by high exposure on a system that’s easy to isolate, may warrant a faster but less complex fix than a High finding, driven by severe impact on a deeply embedded production system.

And when a severity rating feels inconsistent with your intuition about a finding, that’s worth raising with your testing team. A well-built severity framework should be easy to explain and defend. If the explanation requires a lot of assumptions about things the tester couldn’t verify, that’s useful information about how the rating was generated and whether it should be weighted differently in your remediation planning.

At MainNerve, our reports are built to communicate clearly, with severity ratings that reflect what we can verify, findings that explain the real-world attack scenario, and remediation guidance written for the people who have to implement it. If you want to talk through what that looks like in practice, we’re glad to have that conversation. Set up a consultation today.