A subject test sends one email with two or three subjects, split evenly between its recipients, and shows which subject gets more clicks. You set it up in the email editor — see Writing the email. This page is about reading the result.
Where the result is
On the campaign’s Performance step, General tab, inside By email, under the rows: a Subject test card for each email that has had one, named after the email. The same card is on Analytics › Email & SMS with a single campaign picked.
The subjects are lettered: A is the email’s own, B and C the ones added with Test another subject and Add a third subject. Each person always gets the same subject, on a resend too.
The table
| Column | Shows |
|---|---|
| (letter) | A, B or C. |
| Subject | The subject. One taken out of the test reads (removed from the test). |
| Sent | Messages sent with this subject in the range. |
| Click rate | Messages with at least one click, as a share of messages sent. One per clicked message, so it is not the same figure as the campaign’s Click rate card, which counts every link click. |
| Open rate | Messages opened, as a share of messages sent. Its header warns: “Overcounted: Apple Mail opens every email it receives, read or not, so the winner is judged on clicks.” |
| Conversions | People who clicked through and then reached a goal within a day. |
The leading row is highlighted with Most clicks once a subject leads with enough confidence, or Winner once one has been picked.
A person whose test subject comes out empty — for example a subject that is only ${first_name}, for someone with no first name — is sent subject A but still counted under the letter they were dealt.
The verdict line
One line over the table says where the test stands. It uses the same statistics as the site’s A/B tests.
| Line | Means |
|---|---|
| Too early to call: wait until each subject has been sent at least 100 times. | A subject still in the test has fewer than 100 sends in the range. |
| No subject is ahead on clicks yet. | The subjects are level on clicks. |
| B leads on clicks – 99% sure | The leader has reached the email’s confidence threshold. |
| B is ahead, 71% sure – not enough to call yet | A subject is ahead, but below the threshold. |
The threshold is the confidence set under Pick the winner automatically — 98% unless you changed it — and it applies even when automatic picking is off. With three subjects, the leader is compared with the other two together, and the confidence allows for having picked the best of three.
Picking the winner automatically
With Pick the winner automatically switched on in the email editor, everyone after the pick gets the winning subject. Its line reads When one subject is [98% (as A/B tests)] sure, after each has been sent [1,000] times:
| Setting | Choices | Default |
|---|---|---|
| Confidence | 90%, 95%, 98% (as A/B tests), 99% | 98% |
| Sends of each subject | 250, 500, 1,000, 2,500, 5,000 | 1,000 |
Until it picks, the report adds: The winner is picked automatically once it is 98% sure and each subject has been sent 1,000 times. Every subject still in the test must reach the sends, and the leader must be ahead and at least that sure.
Once it picks, the verdict line becomes Winner: {letter} (picked automatically on {date}, {pct}% sure) – everyone now gets {letter}.
Resuming the test
Resume the test, beside the winner line, splits recipients between the subjects again. It also switches Pick the winner automatically off, so the same subject is not picked again at once; switch it back on in the email’s editor when you want it. Resuming does not create a new version of the email.
Reading it well
- Wait for the sends. Below 100 sends of each subject the difference is mostly chance, and a small lead can turn around.
- A test compares subjects of one email. To test whole emails against each other, use the campaign’s A/B test.
- Only email types with a preheader offer a subject test; each test subject can have its own preheader.
- The table counts by send time within the range. Change the range on the Performance step to see an earlier or later stretch of the test.
Related
- Writing the Email — setting up the subject test and the automatic winner
- Reading an Email Campaign’s Report — the By email table the test sits in
- Analytics: Email & SMS
- A/B Testing — testing whole campaigns
- Troubleshooting Email Campaigns
}