How to A/B test email subject lines on your own stack

You have two ideas for a subject line and no way to know which one wins, so you pick the one that sounds better and hope. A/B testing replaces the guess with a measurement, but every marketing tool that offers it also takes your list, your metrics, and a per-contact fee for the privilege. You can run the same test on infrastructure you own, and the only genuinely hard part is the statistics, not the plumbing.

Subject-line testing is the highest-leverage A/B test in email because the subject is the one thing every recipient sees before deciding whether to open. A two-point swing in open rate compounds across every campaign you send for the rest of the year. So it is worth doing properly. Once the plumbing here is in place it powers more than subject lines, so it is worth keeping a running list of email A/B tests worth running on your own stack to feed the same harness. The catch is that "properly" means two separate problems: the engineering of splitting a list and attributing events, which is easy, and the statistics of knowing when a difference is real, which is where most people, and most tools, quietly cut corners.

This guide builds the whole thing from scratch on Cloudflare D1 and Workers: variant assignment, per-variant tracking, and an honest significance test. We will be specific about how big a list you need, because that is the number nobody puts on the marketing page. Then, at the end, the shortcut if you would rather not maintain the assignment and stats glue yourself.

How do you A/B test an email subject line?

You A/B test a subject line by randomly splitting your recipient list into groups, sending each group a different subject, tracking opens and clicks per group in your own database, and comparing the rates with a significance test before picking a winner. The mechanics are simple; the discipline is in sizing the test and not declaring a winner before the numbers can support one.

The procedure has five parts, and the rest of this article is one section per part:

1. Model the experiment in your database (the variants, the split, the assignments).
2. Assign each recipient a variant deterministically at send time.
3. Send each variant its own subject through your sender.
4. Track opens and clicks keyed by the variant.
5. Compare the rates and decide whether the difference is real.

Skipping step five is the common failure. A 22 percent open rate against a 20 percent open rate looks like a clear win until you notice each group was 150 people, at which point the difference is well inside the noise and you have learned nothing. We will come back to this with the actual math.

How do you model an A/B test in a database?

Model the experiment as one row describing the variants and the split ratio, and a variant tag on each send row recording which variant that recipient received. That is the minimum that lets you both send the right subject and attribute events afterward.

Here is a D1 schema. One experiments table holds the test definition, and the existing sends table (one row per recipient per campaign) gains a variant column:

-- The test definition: two subject lines and how the list is split.
CREATE TABLE experiments (
  id            TEXT PRIMARY KEY,
  campaign_id   TEXT NOT NULL,
  variant_a     TEXT NOT NULL,   -- subject line A
  variant_b     TEXT NOT NULL,   -- subject line B
  split_ratio   REAL NOT NULL DEFAULT 0.5,  -- fraction assigned to A
  created_at    INTEGER NOT NULL
);

-- One row per recipient per campaign. variant records the assignment.
CREATE TABLE sends (
  id            TEXT PRIMARY KEY,
  campaign_id   TEXT NOT NULL,
  recipient_id  TEXT NOT NULL,
  email         TEXT NOT NULL,
  variant       TEXT,            -- 'A' or 'B', set at assignment time
  sent_at       INTEGER
);

-- Open and click events, tagged with the variant for direct attribution.
CREATE TABLE events (
  id            TEXT PRIMARY KEY,
  send_id       TEXT NOT NULL,
  campaign_id   TEXT NOT NULL,
  variant       TEXT NOT NULL,
  type          TEXT NOT NULL,   -- 'open' or 'click'
  created_at    INTEGER NOT NULL
);

CREATE INDEX events_campaign_variant_idx
  ON events (campaign_id, variant, type);

Storing the variant directly on the events row is a deliberate denormalization. You could join back through sends every time, but tagging the event at write time makes the final aggregation a single grouped query with no join, which matters when a campaign generates tens of thousands of events in a few minutes.

How do you assign each recipient to a variant?

Assign each recipient by hashing a stable identifier into a bucket, rather than calling a random function per recipient. A deterministic hash means the same recipient always lands in the same variant for a given experiment, the split is reproducible, and you can re-run the assignment without reshuffling everyone.

A naive Math.random() < 0.5 works, but it makes the assignment unauditable: you cannot reconstruct who got what, and a retry can flip a recipient to the other variant. Hashing fixes both. Hash the recipient id together with the experiment id, take the result modulo a large number, and compare against the split:

// Deterministic A/B assignment on a Cloudflare Worker.
// Same (experimentId, recipientId) always yields the same variant.
async function assignVariant(
  experimentId: string,
  recipientId: string,
  splitRatio: number, // fraction going to A, e.g. 0.5
): Promise<"A" | "B"> {
  const data = new TextEncoder().encode(`${experimentId}:${recipientId}`);
  const digest = await crypto.subtle.digest("SHA-256", data);
  // Take the first 4 bytes as an unsigned 32-bit int, scale to [0, 1).
  const view = new DataView(digest);
  const bucket = view.getUint32(0) / 0xffffffff;
  return bucket < splitRatio ? "A" : "B";
}

When you build the send queue for a campaign, assign each recipient once and persist it:

for (const recipient of recipients) {
  const variant = await assignVariant(exp.id, recipient.id, exp.split_ratio);
  await db
    .prepare(
      `UPDATE sends SET variant = ? WHERE campaign_id = ? AND recipient_id = ?`,
    )
    .bind(variant, exp.campaign_id, recipient.id)
    .run();
}

One subtlety worth flagging: if you ever change split_ratio mid-experiment, the hash boundary moves and some recipients silently switch variants. Treat the ratio as immutable once a send starts. If you need a different split, that is a new experiment.

How do you send each variant its own subject line?

At send time, read the recipient's stored variant, pick the matching subject from the experiment row, and pass it to your sender. The body stays identical across variants; only the subject changes, which is what keeps the test clean and the result attributable to the subject alone.

async function sendForRecipient(send: SendRow, exp: ExperimentRow, env: Env) {
  const subject = send.variant === "A" ? exp.variant_a : exp.variant_b;

  // Cloudflare Workers cannot open outbound port 25 for direct-to-MX SMTP, so the
  // send goes out over an HTTPS sender API rather than nodemailer/SMTP.
  await sendEmail(env, {
    to: send.email,
    subject,
    html: renderCampaignBody(send), // identical for both variants
  });

  await env.DB.prepare(`UPDATE sends SET sent_at = ? WHERE id = ?`)
    .bind(Date.now(), send.id)
    .run();
}

That comment is not incidental. The reason you cannot lift a Node SMTP example off the internet here is that Workers prohibit outbound port 25 and Node SMTP libraries do not automatically map to the connect() API. If you are still figuring out the send path itself, sending email from a Cloudflare Worker covers it before you layer testing on top.

How do you track opens and clicks per variant?

Track per-variant engagement by serving the open pixel and click redirect from your own domain and writing each event to your events table tagged with the recipient's variant. Because you already stored the variant on the send row, the tracking endpoint just looks it up and copies it onto the event.

The open pixel is a 1x1 image behind a per-send URL; the click endpoint is a redirect that logs then 302s to the real destination. Here is the click handler, which is the one that matters most because clicks are the trustworthy signal:

// GET /c/:sendId?u=<encoded destination>
app.get("/c/:sendId", async (c) => {
  const sendId = c.req.param("sendId");
  const dest = c.req.query("u");
  if (!dest) return c.text("missing destination", 400);

  const send = await c.env.DB.prepare(
    `SELECT campaign_id, variant FROM sends WHERE id = ?`,
  )
    .bind(sendId)
    .first<{ campaign_id: string; variant: string }>();

  if (send?.variant) {
    await c.env.DB.prepare(
      `INSERT INTO events (id, send_id, campaign_id, variant, type, created_at)
       VALUES (?, ?, ?, ?, 'click', ?)`,
    )
      .bind(crypto.randomUUID(), sendId, send.campaign_id, send.variant, Date.now())
      .run();
  }

  return c.redirect(decodeURIComponent(dest), 302);
});

The open pixel follows the same pattern with type = 'open' and a transparent GIF response. The full mechanics of both endpoints, including signing tokens so recipients cannot forge events, and the honest caveats about why opens undercount, are in the self-hosted open and click tracking guide. The short version: opens are inflated by image proxies and deflated by image blocking, so for a subject-line test, opens give you more data but clicks give you more trust. Run the math on whichever you choose, but know which one you are looking at.

How many subscribers do you need for the test to mean anything?

You need enough recipients per variant that a real difference is distinguishable from random chance, and for typical subject-line lifts that number is in the thousands, not the hundreds. This is the part the marketing tools never put a figure on, so here it is concretely.

The required sample size depends on three things: your baseline rate, the size of the lift you want to detect, and your confidence level. Detecting a 2 percentage point absolute lift on a 20 percent open rate, at 95 percent confidence and 80 percent power, needs roughly 6,000 recipients per variant, so about 12,000 in the test. Want to detect a smaller, 1-point lift? The requirement roughly quadruples. Testing on clicks instead of opens, where the baseline rate might be 3 percent, needs even more, because rarer events carry more relative noise.

The practical consequence: if your list is 800 people, a subject-line A/B test will almost never reach significance, and the "winner" your dashboard shows is mostly coin-flip. That does not make testing pointless on a small list, but it changes what you do with the result. Treat it as a weak prior, accumulate the same test across several campaigns, and never let a single small-list test override editorial judgment.

How do you decide which subject line actually won?

Compare the two variants with a two-proportion z-test, which tells you the probability that the observed difference in rates could have arisen by chance. If that probability (the p-value) is below your threshold, conventionally 0.05, you call it a real difference; otherwise you call it a tie, regardless of which number is bigger.

First, the aggregation query, which is a single grouped read thanks to the variant tag on each event:

SELECT
  s.variant,
  COUNT(DISTINCT s.id) AS sent,
  COUNT(DISTINCT CASE WHEN e.type = 'open' THEN e.send_id END) AS opens
FROM sends s
LEFT JOIN events e ON e.send_id = s.id
WHERE s.campaign_id = ?1 AND s.sent_at IS NOT NULL
GROUP BY s.variant;

Then the z-test on the two open rates. This is the entire statistical core, and it is small enough that there is no reason to outsource it:

// Two-proportion z-test. Returns the p-value (two-tailed).
function abTest(
  openA: number, sentA: number,
  openB: number, sentB: number,
): { rateA: number; rateB: number; p: number; significant: boolean } {
  const rateA = openA / sentA;
  const rateB = openB / sentB;
  // Pooled proportion under the null hypothesis (rates are equal).
  const pooled = (openA + openB) / (sentA + sentB);
  const se = Math.sqrt(pooled * (1 - pooled) * (1 / sentA + 1 / sentB));
  const z = (rateA - rateB) / se;
  // Two-tailed p-value via the standard normal CDF.
  const p = 2 * (1 - normalCdf(Math.abs(z)));
  return { rateA, rateB, p, significant: p < 0.05 };
}

// Abramowitz-Stegun approximation of the standard normal CDF.
function normalCdf(x: number): number {
  const t = 1 / (1 + 0.2316419 * x);
  const d = 0.3989423 * Math.exp(-x * x / 2);
  const prob =
    d * t * (0.3193815 + t * (-0.3565638 + t * (1.781478 +
      t * (-1.821256 + t * 1.330274))));
  return 1 - prob;
}

Run it on a real but underpowered result and watch it do its job:

// Variant A: 165 opens of 750 sent (22.0%)
// Variant B: 150 opens of 750 sent (20.0%)
abTest(165, 750, 150, 750);
// => { rateA: 0.22, rateB: 0.20, p: 0.337, significant: false }

A two-point lead, and the test correctly refuses to call it: a p-value of 0.34 means there is a one-in-three chance you would see a gap that big from noise alone. Now the same 2-point lift at scale:

// Same 22% vs 20%, but 7,500 per variant.
abTest(1650, 7500, 1500, 7500);
// => { rateA: 0.22, rateB: 0.20, p: 0.0026, significant: true }

Identical rates, ten times the list, and now it is a clear, real win. This is the whole lesson of A/B testing compressed into two function calls: the rates are not the result, the significance is.

Send-to-winner: testing on a slice first

A send-to-winner test sends both subjects to a small fraction of the list, measures which won after a waiting window, then sends the winning subject to everyone else. It is the right pattern when your list is large enough that a 10 to 20 percent slice still clears the sample-size bar from earlier. On a small list it collapses, because the test slice cannot reach significance and you end up sending the "winner" of a coin flip to the majority.

If you do run it, the only additions to everything above are a phase flag on the experiment (test vs final), a scheduled job that runs the z-test after the window, and a guard that refuses to promote a winner when the result is not significant, falling back to whichever subject you marked as the default. Building that scheduler from scratch is its own small project; scheduled sends with Cloudflare Cron Triggers covers the trigger mechanics you would reuse for the measurement window.

Or skip the plumbing: A/B testing in Tinbox

Everything above is genuinely worth understanding, and on a self-hosted stack it is yours to keep: the assignments, the events, and the stats all live in your own D1 with no per-contact fee and no vendor between you and your metrics. But maintaining the variant assignment, the tracking endpoints, and the significance math across every campaign is glue you have to keep working.

Tinbox is designed to fold subject-line A/B testing into its campaign tooling on the same self-hosted Cloudflare stack you would have built by hand. The intended flow: you define the variants on a campaign, choose the split, and the campaign sender handles deterministic assignment, per-variant open and click attribution, and the winner test, all writing to the D1 database on your own Cloudflare account. The data model is the one above, so you can still query events directly or join it to the rest of your product. Owning the stack is the point: testing stays free at the margin no matter how large the list grows, which is the opposite of how per-contact pricing on a marketing SaaS treats a growing audience. And because the inbox is a real database, the same events feed your other reporting rather than living in a dashboard you cannot export.

If you want the bulk-send mechanics that sit underneath campaign testing, batched fan-out sending from a Worker is the companion piece.

Deploy this yourself: npx tinbox init <domain>, define a campaign with two subject variants, and read the per-variant results straight out of your own D1. The campaign and analytics docs cover the experiment fields and the winner-selection settings.