Guides
Buying and testing security products
Written for teams that build security products and outsource part of their QA.
Choosing a QA vendor
How to shortlist, brief and compare QA vendors with the ranking's evidence.
- How to choose a QA vendor for a security product
A shortlist procedure for security product companies outsourcing QA, built on the ranking's eight evidence dimensions and its not-ideal-for lines.
- QA vendors vs efficacy labs
Why labs that certify detection rates are a different service class from outsourced QA vendors, and why this ranking does not list them.
- RFP questions for a QA vendor
RFP questions that map to the evidence the ranking scores: security-product cases, test types, engagement length, certificates, data handling and exit terms.
Testing security products
What a QA team tests for each security product type, and what to ask a vendor about it.
- Detection regression testing
How to keep detection rules and false-positive behaviour stable across releases: test events per rule, data path tests, CI runs and triage.
- Performance testing security software agents
Baselines with and without the agent, workloads, scan storms and per-build budgets for security agents.
- Security product QA checklist
A checklist of what QA should cover for endpoint agents, detection platforms and identity products, mapped to the ranking's test-type dimension.
- Test automation for security products
What to automate first, what stays manual, OS images, flaky tests and code ownership for security products.
- QA for backup and DLP software
Restore drills, backup of changing data, immutability checks and DLP policy tests on synthetic data.
- Testing EDR agents across OS versions
How to test an endpoint agent across operating system versions: compatibility matrix, performance baseline, upgrade and rollback, coexistence and reporting.
- Testing email security products
Mail flow, benign corpus, verdict consistency, attachment, link, quarantine and SPF, DKIM, DMARC tests.
- QA for firewall and VPN software
Rule regression, tunnel reconnect, kill switch, leak and throughput tests for firewall and VPN builds.
- Testing MFA factors and edge cases
MFA test cases for TOTP, push, recovery, lockout and step-up, and what to ask a QA vendor.
- Testing SIEM log ingestion and detection rules
What to test in a SIEM product from ingestion to rules, and what to ask a QA vendor.
- Testing SSO and SCIM in identity products
Test cases for SAML, OIDC and SCIM in an identity product, and what to ask a QA vendor.
- Testing vulnerability scanner products
Target labs, missed and spurious findings, scan cost and check update regression for scanner products.