Testing email security products
An email security build can change the verdict on an unchanged message, and customers see the result as held business mail or a new threat in an inbox.
Research desk, cybersecurityqavendors.com. Published 2026-11-11.
Short answer
Send fixed test corpora through the product on every build, including a benign corpus for false positives, attachments, links and spoofed senders, and compare each verdict with the last release. Test quarantine and release workflows and SPF, DKIM and DMARC handling. Ask a QA vendor for a written case on a mail security product that shows verdict regression work.
Where verdicts change
An email security product sits in the mail flow, either as a gateway that relays SMTP or as a service connected to the mailbox platform's API. A change to a parser, a model or a reputation feed can change the verdict on a message that has not changed.
QA for this product type checks that the product applies its own policy the same way from one build to the next. Measuring how much malicious mail a product catches is efficacy testing, run by independent labs; the boundary is set out in QA vendors vs efficacy labs.
Build the corpora
A benign corpus holds synthetic business mail: invoices, calendar invites, newsletters, signed mail, mail with large attachments and messages forwarded through mailing lists. It is the basis of false-positive regression1: a benign message that a new build holds or rewrites fails the test.
A policy corpus holds harmless test messages the policy should act on. Include standard antivirus test files inside archives and password-protected archives, links that redirect through several hops, lookalike sender domains and display-name spoofing.
Business email compromise cases carry no payload. Write a reply-chain request to change bank details and a payment request from a spoofed executive name. The expected result for each message is the action the policy defines, not a detection score.
The test cycle
- Mail flow. Send each corpus inbound, outbound and internal through every supported deployment mode. Compare delivery, added headers and delay per message with the last release.
- Verdict consistency. Pin reputation feed and model versions in the test environment, or record their versions with each run, so a verdict change can be tied to the build. Store the verdict and action for each message per build. A changed verdict fails the build until someone records in writing whether the product, the policy, a feed or the test message is wrong.
- Attachments. Cover nested archives, encrypted archives, office files with macros, oversized files and file types the policy blocks by extension. Check that the stated action happens and that the message is delivered or held as documented.
- Links. Check URL rewriting, click-time checks, links inside attachments and links encoded in QR images. Confirm that rewritten links in benign mail still open the right page.
- Quarantine and release. Release a held message as an admin, and as an end user where the product allows it. Check that it arrives with original headers and attachments, and that the audit log records who released it.
- Authentication records. Send from test domains with SPF pass and fail, DKIM valid, broken and missing, and DMARC policies of none, quarantine and reject. Each combination should produce the action the documentation describes, including after forwarding breaks SPF.
Keep test mail separate
Authentication tests need test domains and DNS zones you control, so the team can publish any SPF, DKIM or DMARC record a case requires.
Every corpus is synthetic mail the team writes and sends from those domains. Customer mail stays out of it, even with names removed.
Verdict logs and quarantined copies of test mail can flow into a customer-facing report or a SIEM2 export. Mark every test message with a header the team can filter on, and check that the filter works before each run.
What to ask a QA vendor
Look for a mail security case that names a build where a verdict changed and how the team resolved it. Verdict replay is detection or false-positive regression, which test-type depth (dimension 3 of the methodology) counts as its own type next to functional, automation and API work.
A benign mail corpus is maintained over many releases, and the people who built it know why each message is there. Dimension 4, engagement longevity, rewards a vendor that kept such a client for years.
A published case naming the corpora and deployment modes of a mail security product counts under evidence depth, dimension 1. Its staff will handle quarantined mail, test domains and DNS access, so ask which controls from dimension 6, security posture, cover them.
There is no email and web security facet page yet; not published lists it with the threshold it missed. Storing an expected result per test item follows detection regression testing, and mail flow gates fit into the security product QA checklist.
- 1 false-positive regression: A case where a security tool starts flagging harmless activity as malicious after an update, when it did not do so in an earlier version. Source
- 2 SIEM: A system that collects log data from many sources into one place so security teams can search it, correlate events and spot attacks. Source
Questions buyers ask
Why do false positives matter in email security testing?
A benign message held by a new build is delayed or lost business mail for every customer on that build, so a benign corpus replayed per build catches the change before release.
How do you test an email security product for false positives?
Replay a synthetic benign corpus of invoices, invites, newsletters, signed and forwarded mail through each build, and fail the build when a message that was delivered before is now held.
What test cases cover business email compromise?
Payload-free messages: a reply-chain request to change bank details, display-name spoofing of an executive and lookalike sender domains. Each case expects the action the policy defines.
What does an email security test methodology include?
Fixed corpora with an expected action per message, sent through every deployment mode on each build, with verdicts stored and compared with the last release. Reputation feed and model versions are pinned or recorded with each run, so a verdict change can be tied to the build.
How do you test SPF, DKIM and DMARC handling?
Send from test domains you control with each pass and fail combination and each DMARC policy, including forwarded mail, and compare the action with the product documentation.
How should quarantine and release be tested?
Release held messages as an admin and as an end user, check that the delivered copy keeps its headers and attachments, and check that the audit log names who released it.
Is email security QA the same as an efficacy test?
No. QA checks that the product applies its own policy the same way on each build. Efficacy tests measure detection against a threat set and are run by independent labs.