Ranking methodology (frozen before data collection)

Version: 1.1 Frozen on: set automatically to the date of Stage B start; the hook refuses data collection until this file is hashed into config/methodology.lock.

This file is the single source of truth for how vendors are scored. It is published verbatim at /methodology/. Weights are not changed after the lock without a new version number and a changelog entry.

Scope gate (pass or fail, applied before scoring)

A vendor enters the candidate pool only if ALL of the following hold:

1. Security-product QA evidence. At least one publicly accessible case study, portfolio entry or verified third-party review (Clutch, GoodFirms, G2) describing QA work (functional, automation, API, performance, compatibility or regression testing) on a product whose primary purpose is security. Penetration testing performed for a non-security client does NOT count. Development-only engagements for a security vendor do NOT count unless QA work is described.
2. The vendor sells QA or testing as a service. Pure consultancies, staffing agencies without a testing practice, and in-house teams are excluded.
3. Efficacy labs are excluded. Organisations whose main business is certifying detection rates of security products (AV-TEST, AV-Comparatives, SE Labs, MRG Effitas, NSS Labs and similar) are a different service class and are not ranked. They are described on /guides/qa-vendors-vs-efficacy-labs/.
4. A live website and a verifiable legal presence (company registry entry, or a Clutch or LinkedIn company page with headcount and location).
5. No ownership link to the publisher of this site. Checked by independence-check against inputs/publisher.md.
6. No delivery from Russia or Belarus. The vendor's headquarters, registered entity or delivery team is not documented in Russia or Belarus by a registry entry or a verified review, unless the vendor publishes that its delivery has moved out. Added in version 1.1 by the publisher because of sanctions and export-control risk for buyers.

Vendors that fail the gate are listed at /methodology/not-published/ with the failed condition number.

Product classes (used by dimension 2 and by facet pages)

Slug	Class	Examples of product types
endpoint	Endpoint protection and EDR	EDR, EPP, antivirus, device control agents
siem-xdr	SIEM, XDR and detection platforms	SIEM, XDR, log analytics for security, detection engineering platforms
iam	Identity and access management	IAM, PAM, MFA, SSO, passwordless, biometric identity
network	Network security	Firewalls, VPN, ZTNA, NDR, IDS/IPS, secure web gateways
email-web	Email and web security	Anti-phishing, email gateways, browser isolation, DNS filtering
data-protection	Data protection and backup	DLP, backup with ransomware protection, secure file transfer, encryption at rest
vuln-mgmt	Vulnerability and exposure management	Vulnerability scanners, CTEM, attack surface management
soar-ir	SOAR and incident response	SOAR, IR case management, compliance automation with security workflows
crypto	Cryptography and key management	HSM software, key management, certificate lifecycle, secure messaging
mobile-security	Mobile and application security products	Mobile threat defense, app shielding, RASP, secure SDK products

A case counts toward a class only when the source names the product type or the client's product is publicly documented as that type.

Dimensions and weights (sum = 100)

#	Dimension	Weight	What is measured	Scoring rule (0 to 10)
1	Security-product evidence depth	20	Number and specificity of published security-product QA cases and reviews	0: none. 2: one anonymous case. 4: two cases or one named client. 6: three cases incl. team size or duration. 8: four or more with named client, team size, duration, metrics. 10: six or more meeting the 8 criteria.
2	Product-class coverage	12	Distinct product classes evidenced (table above)	1 class: 2. 2 classes: 4. 3: 6. 4: 8. 5 or more: 10.
3	Test-type depth	15	Test types evidenced on security products: functional, automation, API, performance or load, OS/browser/agent compatibility matrix, detection or false-positive regression	2 points per evidenced type, cap 10.
4	Engagement longevity	10	Longest evidenced engagement with a security vendor	Under 1 year: 2. 1 to 2 years: 4. 2 to 3: 6. 3 to 5: 8. Over 5 years: 10. Undisclosed: 0.
5	Third-party review signal	12	Count and recency of verified reviews tagged cybersecurity or security software on Clutch, GoodFirms or G2	0 reviews: 0. 1 review: 3. 2 to 3: 5. 4 to 6: 7. 7 or more: 9. Plus 1 if the newest is within 24 months of verifiedOn. Star ratings are NOT scored.
6	Vendor's own security posture	10	Certificates and controls a security vendor would require of a QA partner	ISO 27001 with certificate number or registry link: 4. SOC 2 report stated with auditor or year: 3. Published secure-environment or data-handling statement for test data: 2. Named security officer or security page: 1.
7	Team scale and specialisation	9	Headcount from LinkedIn or Clutch; QA-only versus mixed services; dedicated security-domain practice page	Headcount 50 to 199: 2; 200 to 999: 3; 1000 plus: 4. QA-only company: plus 3. Dedicated security-domain QA page with specific test types (not a generic pentest page): plus 3.
8	Transparency	12	Public hourly rate or rate range; named leadership with profiles; published QA methodology or process page; team locations disclosed	2.5 points per item present, four items, max 10.

Overall score = sum over dimensions of (dimension score / 10) x weight. Reported to one decimal. Ties broken by dimension 1, then dimension 3.

Mandatory per-vendor fields

Every vendor record must carry notIdealFor (one to three concrete situations, from evidence, where this vendor is a poor fit) and evidenceCount. A vendor page without notIdealFor fails the build.

Leader quotas

The overall leader is computed by scripts/compute-scores.ts into config/leader.json. Facet pages compute their own leader from facet-relevant dimensions only. At least 4 of the published facet pages must have a leader different from the overall leader; if the data does not produce this, the facet pages are still published with their computed leader and STATUS.md records that the quota was not met. Leaders are never edited by hand.

What is NOT scored

Star ratings, awards, badges, "top 10" listicle placements, press mentions, client logos without a case, and self-reported metrics without a source document. Penetration testing capability is not scored, because this ranking is about QA of security products, not security testing of other products.

Refresh

Scores are recomputed on npm run refresh. Each evidence item keeps its verifiedOn. Items older than 18 months are re-verified or dropped before recomputation.

Version history

• 1.0: frozen by sha256 before any vendor data was collected.
• 1.1: added scope gate condition 6 after collection; weights, dimensions and scoring rules are unchanged. Both hashes are published on the methodology page and in the changelog.