Experiments

An experiment splits part of a campaign's audience between two or more versions, waits, then sends the version that won to everyone else. You choose how much of the list to test on and how long to wait; Bitelio decides the winner and releases the rest.

What you can test

Anything that belongs to the message: the subject line, the body, the from address, and the sender name. A version overrides only the fields you give it — a version testing the subject keeps the campaign's body, so you don't have to restate it.

Send time is not on that list. It is a property of the moment rather than of the message: by the time a winner is known, the winning hour has already passed, so there is nothing to send the rest of the audience.

How the split works

You pick a sample — between 5% and 50% of the audience. That sample is divided evenly between your versions; the rest of the list receives nothing until a winner exists.

Which contact lands in which version is decided by a hash of their id, so it never changes. If a send is retried after an interruption, the same person stays in the same version rather than being counted in both.

Choosing what decides it

You can decide on clicks or on attributed revenue.

You cannot decide on opens, and this is deliberate. Apple Mail Privacy Protection loads a message's images on the recipient's behalf, which registers an open whether or not anyone read it. On a consumer list that is commonly half the traffic or more, and it affects both versions equally — so the difference you are trying to measure is buried under identical noise. A subject-line test measured on opens can name the wrong winner, or find no winner when there was one. Opens still appear in the results for context; they just can't decide anything.

For the same reason, revenue counts only orders attributed through a click. Bitelio also credits orders to an email that was merely opened when there was no click, which is a reasonable guess for ordinary reporting — but inside an experiment it would hand money to whichever version Apple happened to open, which is close to random.

How long it runs, and how it ends

You set a deadline and a sample size. Bitelio watches the results continuously and ends the test as soon as choosing wrongly stops being expensive — not as soon as one version is ahead.

That distinction is the whole point. "Version B is winning" says nothing about how much you lose by picking it and being wrong: a 96% chance that B is better by 0.1% is not worth acting on, and a test stopped the moment it looks good is a test that always finds a winner, whether or not one exists. Bitelio measures the expected cost of the decision instead, and concludes when that cost falls below what you set.

While the test runs you can see how many recipients each version reached, but no version is marked as winning. That is on purpose: a leaderboard invites stopping early, and stopping early is what makes A/B results untrustworthy.

If the deadline arrives without a clear answer, the control version wins and the result says no difference was detectable. That is a real result, not a failure — it means the effect, if any, was smaller than this audience could resolve. Bitelio will not invent a winner to look decisive.

Guardrails

Unsubscribes, spam complaints and bounces are tracked per version. A version that crosses your threshold is disqualified: it cannot win, and it will not be sent to the rest of the list no matter how well it performed on clicks or revenue.

Complaints matter most. They damage the deliverability of your whole project, not just this campaign, so a version that provokes them is not a winner even when it earns more — and it will not become one by running out the clock either.

A single event never disqualifies a version. A guardrail needs at least two before it acts, because one is not a rate: results are checked every few minutes, so a fresh test is usually sitting on a partly-delivered sample, and one early complaint among the first couple of hundred recipients would otherwise read as far above any sane threshold. One complaint is ordinary on any list. Two is a pattern.

If every version is disqualified, the experiment stops and the remaining audience is left unsent.

What you need to run one

A test needs enough people to fill each version. Bitelio refuses an experiment whose audience cannot reach the minimum for every version at your chosen sample size, and tells you how many contacts it would take — because an experiment that cannot conclude spends the send and teaches nothing.

Creating an experiment requires the campaigns:edit permission. Deciding a winner by hand requires campaigns:send, because releasing the remainder is a send: it mails the winning version to the rest of the list and cannot be recalled.

An experiment can only be added to a campaign that is still a draft or scheduled — not to one already awaiting approval, since an approver signed off on a specific campaign.

API reference

• POST /experiments — creates an experiment. Body: { campaignId, metric, samplePct, decideBy, lossThreshold, minExposurePerArm?, autoDecide?, variants }. Exactly one variant must be isControl. Requires campaigns:edit.
• GET /experiments/:campaignId — the experiment and its versions, including which are disqualified and why.
• POST /experiments/:id/decide — ends the test yourself. Body: { variantId }. Requires campaigns:send. A disqualified version is refused, as is one belonging to another experiment.
• DELETE /experiments/:id — removes an experiment that has not started sending.