Methodology

This analysis uses the SOC 2 Quality Guild's Reliability Rubric v1.0 (February 2026), licensed under CC BY-SA 4.0. The rubric defines 11 signals across 3 pillars for evaluating SOC 2 report quality. We implemented each signal as a scoring function — some purely deterministic (regex and heuristics), others AI-assisted using Claude for semantic analysis.

Scoring Formula

Each signal produces a score from 0–100and a letter grade (A ≥ 90, B ≥ 80, C ≥ 65, D ≥ 50, F < 50).
Signals are weighted and averaged within their pillar. Pillar scores are then combined:
Overall = Structure × 0.25 + Substance × 0.50 + Source × 0.25
Substance carries the highest weight (50%) because it directly measures the quality of actual audit work performed.

Structure (25% of overall)

Report formatting and AICPA compliance

S1Report StructureDeterministicWeight: 1.0

Checks that Section 1 contains all required AICPA paragraphs: Scope, Opinion, Service Organization's Responsibilities, Service Auditor's Responsibilities, Inherent Limitations, and Restricted Use.

S2Management AssertionDeterministicWeight: 1.0

Verifies Management's Assertion completeness: description accuracy, control design suitability, operating effectiveness (Type II), signature, and date.

S3Cross-Section ConsistencyAI-assistedWeight: 1.0

Analyzes cross-section consistency: company names, placeholder text, scope boundaries, and subservice organization references across Sections 1, 3, and 4.

Substance (50% of overall)

Depth of audit testing and specificity

S4System DescriptionAI-assistedWeight: 1.25 (higher)

Assesses whether the System Description names specific technologies, tools, teams, and infrastructure — or uses generic boilerplate that could apply to any company.

S5Control-Criteria MappingAI-assistedWeight: 1.0

Spot-checks whether controls logically address their assigned Trust Services Criteria. Flags illogical mappings like access controls mapped to communication criteria.

S6Control DescriptionsAI-assistedWeight: 1.0

Applies the 5-question test to control descriptions: What is done? How? Who does it? When? Where? Also checks for contradictions between controls.

S7Test ProceduresDeterministicWeight: 1.25 (higher)

The strongest signal. Measures boilerplate test language rate, exception rate (100% 'No exceptions noted' is a red flag), sample sizes, meaningful test types, and pagination fingerprints.

Source (25% of overall)

Audit firm credibility and independence

S8Firm RegistrationDeterministicWeight: 1.0

Identifies the audit firm from license numbers and known firm names. Flags known problematic entities like PAC-FIRM-LIC-47383 (Delve/Accorp).

S9CPA RatioDeterministicWeight: 1.0

Estimates the ratio of reports issued to CPAs at the firm. A single license issuing 487+ reports is an extreme outlier.

S10Firm LeadershipDeterministicWeight: 0.75

Evaluates audit firm leadership credibility. Flags firms with shell company structures or untraceable registration.

S11GRC ToolDeterministicWeight: 0.75

Scans for GRC platform mentions (Vanta, Drata, Secureframe, Delve, etc.). The platform through which the audit was conducted can indicate quality.

Attribution

The scoring rubric is based on the SOC 2 Quality Guild Reliability Rubric v1.0 (February 2026), used under the CC BY-SA 4.0 license. This analysis and tool were built by Trenta. The rubric authors are not affiliated with this project and do not endorse these specific findings.