Evaluating Pentest Quality: The Role of Auditors, Consultants, and Third-Party Risk Analysts

By Larry Kinkaid

In many organizations, penetration testing is viewed as a means to an end. Whether the driver is SOC 2, ISO 27001, HITRUST, or customer expectations in a procurement process, the rationale is often framed as, “We need a pentest because our framework or our client requires it.”

While these requirements may serve as the gateway, the real value of a pentest lies in identifying vulnerabilities, enriching an organization’s security program, and providing assurance to stakeholders. Compliance is a byproduct. The priority should always be strengthening security for the organization and the customers it serves.

But pentests vary widely in quality. Some tests provide meaningful insights and drive improvements, while others amount to little more than automated scans dressed up as a comprehensive report. For third-party auditors, consultants, and risk analysts, knowing how to distinguish between the two is critical. A low-quality pentest not only fails to improve security, but can also create new risks.

The Consultant’s Role

One major challenge for security consultants is that pentest quality cannot be evaluated solely by the report produced. The nature of the engagement matters just as much.

A strong pentest provider takes the time to understand the business context, identify the organization’s “crown jewels,” and engage in meaningful threat modeling. This relationship is about collaboration, not just delivering a lengthy report under the pentester’s letterhead.

For consultants in particular, exploring the quality of this relationship can be revealing. Asking questions about the process often uncovers whether the pentest added real value or fell short. How long did the engagement take? Did you feel like the relationship was collaborative? How was retesting managed? By asking clients questions about their experience, consultants can uncover areas of weakness that may not appear in documentation.

These conversations create opportunities to educate clients. When gaps are identified, consultants can help organizations understand what they should demand in future engagements and help mature their overall security program.

The Auditor’s Role

For auditors, assessing pentest quality is more complex.

Compliance standards are structured around conditions and criteria. For SOC 2 in particular, an auditor’s opinion is based on three components: (1) the system description, (2) the design of controls, and (3) the operating effectiveness of those controls. Neutrality expectations generally call for an agnostic stance toward pentest providers, instead placing the focus on the second point: whether the control is operating as designed.

There are circumstances in which auditors can challenge control design. If the report makes it explicit that the testing was merely an automated scan labeled as a pentest, this represents a design deficiency that should be documented. In those cases, auditors can provide significant value by highlighting the gap.

The key is defensibility. Unless the report is explicitly subpar, auditors risk overstepping if they reject a pentest based on inference alone. They are, of course, entitled to challenge the results, but concrete evidence is needed to support a finding of control design failure.

The Third-Party Risk Analyst’s Role

For third-party risk analysts, pentest quality directly impacts credibility in vendor assessments. Unlike auditors, they aren’t bound by strict rules of engagement and may rely on their overall impression of a report.

A weak or unclear pentest can raise doubts, especially if competitors provide stronger evidence or the data in question is highly sensitive—potentially lengthening sales cycles, creating unnecessary friction, or resulting in lost deals. Strong reports support smoother procurement processes and reinforce trust in the vendor’s security program.

This is an area where the cybersecurity industry overall is advancing. Customers are becoming more savvy about pentest quality, and expectations are rising. Organizations that continue to rely on low-value tests risk falling behind.

Advising Clients on Pentest Quality

The big question for consultants and third-party analysts is how to help guide their clients toward high-quality pentests. It shouldn’t have to come down to producing a report that an auditor—or even a customer—will accept. The true spirit of a penetration test is to uncover material vulnerabilities and strengthen the organization’s security program.

Selecting the right provider is one of the most effective ways to avoid quality issues. In my experience, referrals are invaluable. The best pentesters I have worked with were not discovered through marketing, but more organically through trusted professional networks.

You should also advise your clients to request draft reports and confirm alignment with established standards, like the OWASP Application Security Verification Standard (ASVS). Leveraging the ASVS allows you to find a common language. You and your clients can better understand the scope and rigor applied, and conversations can quickly move to higher-value discussions.

Organizations undergoing a penetration test should also examine the report itself to gauge its quality. Advise your clients to look for key elements like:

Detailed, actionable recommendations: Test findings should be explained in a way that allows organizations to act quickly.
Support for retesting: Retesting provides assurance that remediation steps were effective. From a compliance perspective, this step is often even more important than the initial pentest, as it validates that controls are functioning as intended.
Clear executive summaries and scope: Reports should outline the number of findings by severity and the outcomes of retesting.

In your conversations as a consultant or your reviews as an auditor, be wary of so-called pentest “certificates.” These documents are misleading. A pentest is not a pass/fail exercise, nor does it result in certification. Certificates typically lack scope, context, and meaningful information. At best, they indicate participation rather than performance.

Moving Beyond Shadowbox Pentests

Pentests should never be treated as mere check-the-box exercises. At their best, they are an essential control that identifies vulnerabilities, validates remediation, and builds trust with customers. At their worst, they are shadowbox exercises that waste resources, erode credibility, and leave organizations exposed to threats.

Auditors, consultants, and third-party risk analysts each play a role in raising expectations. Auditors can highlight deficiencies when design flaws are explicit. Consultants can educate clients and advocate for stronger engagements. Risk analysts can differentiate between reports that enhance credibility and those that undermine it.

The true purpose of a pentest is to strengthen security, provide assurance, and protect the customers who place their trust in us. Ultimately, that is the standard we should be using to measure quality.

ABOUT THE AUTHOR