Holdouts

A holdout takes a percentage of your contacts and sends them no marketing at all for a set period. At the end, Bitelio compares what that group spent against what everyone else spent. The difference is what your email programme was worth.

This is a different question from the one an experiment answers. An experiment asks which version of a message wins. A holdout asks whether sending at all beats sending nothing — which no amount of open rates, click rates or attributed revenue can tell you, because all three only ever look at people who received something.

What it costs

A holdout is not free, and it is not supposed to be. If your email works, the held-out group buys less, and that shortfall is real revenue you chose to forgo in order to find out how much.

That is the whole trade. Ten percent held out for a month tells you what the other ninety percent is earning you, at the price of one month of one tenth of the programme. Bitelio states the cost beside the button rather than behind it, so the decision is made with the number in view.

The percentage is between 1% and 50%, and it cannot be changed once the holdout has started — moving it mid-period would move contacts between the two groups and destroy the comparison. Above half, the "control" is most of your list: at that point you are not measuring your programme, you have paused it.

What is withheld, and what is not

Withheld: campaigns and workflow emails — marketing.

Never withheld:

• Transactional email. Password resets, receipts, order confirmations and anything else sent through the API's /v1/send reach held-out contacts exactly as they always did.
• Required topics. A topic you marked as required already bypasses the consent gate because you declared it operationally necessary. The same reasoning exempts it here.

A contact in the holdout group is not suppressed, not unsubscribed and not modified in any way. They simply do not receive marketing until the period ends. Nothing about their record changes, and nothing needs undoing afterwards.

Which contacts land in the held-out group is decided by a hash of their id, so it never changes. A send retried after an interruption puts the same person on the same side rather than counting them in both.

When the result appears

At the end, and not before.

This is not a limitation waiting to be lifted. A number available halfway through invites ending the test the moment it looks good, and a holdout ended because the number looked good is a holdout whose number means nothing. While it runs, Bitelio shows you what it is costing — how many marketing emails have been withheld so far — and nothing else. The result does not exist yet, so there is nothing to show.

You can cancel a running holdout at any time. Everyone starts receiving marketing again immediately, and no result is recorded: a period cut short measured nothing, and reporting a figure from it would be the same failure in another costume.

How to read the result

The result reads as three figures:

Contacts who received your email spent €4.10 each. Those who didn't spent €3.20. Sending was worth about €0.90 per contact, with a 93% chance it beat not sending at all.

The first two are facts: what each group actually spent, divided by how many people were in it. The third is an estimate, and deliberately not the subtraction of the other two — Bitelio pulls it toward no-difference so that a handful of unusually large orders cannot manufacture a win. It will usually be a little smaller than the arithmetic suggests, and that gap is the point.

The percentage is the probability that sending genuinely beat not sending, given what was observed. It is not a promise; it is how confident the comparison is.

"No difference was detectable" is not "email does not work"

Sometimes the answer is that no difference could be resolved. That means the effect, if there was one, was smaller than this audience over this period could distinguish from noise. It does not mean the effect was zero.

Bitelio says this rather than reporting a lift of €0.00, because a lift of zero reads as "email is worthless" — a far stronger claim than the data supports. If you get this result, a longer period or more sending inside it gives the next holdout more to measure.

What it measures

A holdout can only compare an outcome. The held-out group has no opens and no clicks, by construction — there is nothing to compare but what people actually did. There are two kinds of outcome it can read, and you pick one when you start it.

Shopify orders, which is what a holdout has always measured, and what it still measures if you choose nothing else.

A conversion goal, if you have defined one. This is how a project that does not sell through Shopify measures its programme at all: whatever event you count as an outcome — a signup, a booking, your own purchase event — becomes the thing the two groups are compared on.

The choice is fixed for the life of the holdout. Changing what is measured halfway through would make the two halves of the comparison answer different questions.

A goal holdout counts events, not attributed conversions

This is the one thing worth understanding properly, because it looks like an inconsistency and is not.

Elsewhere, a conversion goal reports attributed conversions: the ones Bitelio could trace back to an email the contact clicked or opened. A holdout does the opposite. It counts every event of that goal, from every contact, whether or not any email was involved.

It has to. The held-out group receives no email, so it has no attributed anything — its attributed total is zero before the period even starts. Comparing that against the sent group would not measure your email; it would measure the definition of the two groups, and report a "lift" equal to whatever the sent group happened to earn.

So the two reports will not agree, and should not. The campaign report answers which email got the credit. The holdout answers whether sending helped at all.

Two smaller differences

A goal holdout's revenue is gross. Events have no concept of a refund — a conversion goal is final once recorded — where a Shopify holdout subtracts refunds from each order.

Revenue counts only the goal's default currency. Conversions recorded in any other currency are left out of the measurement entirely, and the result says how many. Bitelio does not know your exchange rates, and a converted total would be a figure with no defensible date behind it — but counting them as conversions worth nothing would be worse than leaving them out, because it would tell the statistics that somebody converted and paid nothing.

What else you need

Only one holdout can run at a time per project. Two overlapping exclusions contaminate each other: part of the first holdout's control group sits inside the second's treatment, and neither measurement means anything afterwards.

Starting and cancelling require the settings:manage permission, not a campaign permission — a holdout deliberately stops marketing reaching part of your audience for weeks, which is a project-level decision. Reading the result requires analytics:view.

API reference

• POST /holdouts — starts a holdout. Body: { pct, endsAt, goalId? }. Requires settings:manage. Omit goalId to measure Shopify orders. Refused if one is already running, if the end date is not in the future, if goalId names a goal that is not yours, or if neither a goal nor a connected store gives it anything to measure.
• GET /holdouts — the current holdout and everything concluded before it, newest first. Requires analytics:view. A holdout still running carries no result fields at all — they are removed from the response, not merely hidden by the dashboard.
• POST /holdouts/:id/cancel — stops a running holdout and releases the group. Requires settings:manage. Records no result.