Workflow experiments

A split step divides the contacts reaching it between two or more branches of your workflow. From there each group lives a different journey — different emails, different waits, a different number of messages.

A branch experiment measures which of those journeys is worth more. It is the question neither an experiment nor a holdout can answer: not which subject line wins, and not whether email works at all, but whether your five-email welcome sequence beats your three-email one.

The split is useful on its own

A split with no experiment attached still splits. It is a random division of traffic, which is a perfectly good thing to have — you can send half your new subscribers down one path and half down another and simply watch what happens.

Which branch a contact lands in is decided by a hash of their id and the step's id, so it never changes. If a step is retried after an interruption, the same person goes the same way rather than being counted in both. And because the step's id is part of it, the same contacts are not the guinea pigs of every split in your account.

A branch with no outgoing connection ends the journey there. That is a legitimate treatment: "send nothing" is the holdout of this level.

What an experiment measures

Two things can decide it:

A conversion event. Did a named event — order.placed, or one of your own — fire for that contact inside their window? Binary, and available to everyone.

Revenue per contact. Everything that contact spent inside their window, attributed or not. Needs a connected store.

Everything is measured per contact, and that is the whole point

If branch A sends five emails and branch B sends three, A collects more clicks, more opens and more complaints by construction — not because it works better, but because there is more of it. Any metric counted per email would measure the number of emails.

So Bitelio counts people. One contact who bought five times is one converted person. One contact who complained about three messages in a sequence is one complaining person. Both branches are then comparable no matter how many messages each contains — including the guardrails, which is what stops a long sequence from looking cleaner than a short one purely by having a bigger denominator.

Uneven splits

By default the branches get equal shares. You can set a percentage per branch instead — 90/10, say, to expose only a tenth of your audience to a sequence you are not sure about yet.

It costs time, and the editor tells you how much. The exposure floor applies per branch, so the smallest branch is what decides when the experiment can conclude: at 90/10 with a floor of 500, the small branch needs 5,000 contacts through the split, where an even split needs 1,000. An even split also extracts the most information per contact exposed, so anything else is a deliberate trade — less exposure to a risky branch, in exchange for a much longer test.

Percentages are all-or-nothing and must add up to 100. Changing them while an experiment is running counts as editing the workflow, because it reassigns contacts who were already routed.

The window, and why a result takes weeks

Each contact carries their own window, which opens the moment they reach the split. A contact does not count until their window closes. Someone who entered yesterday has had a day to buy; someone who entered a month ago has had the full period. Mixing them would drag down whichever branch happened to receive more newcomers.

The panel shows both numbers while it runs: how many contacts have been assigned to each branch, and how many have counted so far. The gap between them is the whole reason a result takes weeks, and the panel states the earliest date a result can exist — one full window after the experiment started.

Reading it while it runs

Unlike campaign experiments and holdouts, a branch experiment shows you everything while it runs: the counts, the metric, each branch's probability of winning, the guardrail counters and any disqualification.

That is deliberate, and it is safe here. The objection to watching a test in flight — that peeking inflates false positives — applies to frequentist p-values, where the stopping rule is part of the maths. Bitelio decides on expected loss over a Bayesian posterior, a framework built for continuous monitoring: the posterior at any instant is the correct belief given the evidence at that instant. What earlier features were protecting against was the temptation to act early, not a broken calculation. You are watching a system that spends weeks emailing your people with your money, and it is your data.

The numbers refresh every five minutes.

How it ends

Sufficient evidence. A branch's expected loss falls below the threshold and every branch has passed the exposure floor. The floor is not negotiable — it is what stops a winner being declared on forty people per branch because the posterior, with that little data, said something flattering.

The deadline. If the deadline arrives with nothing decisive, the control branch stands and the result is marked inconclusive. No winner is invented to look decisive.

A guardrail. A branch whose complaint or unsubscribe rate crosses your threshold is disqualified: it cannot win regardless of how well it converts. A single event never disqualifies — one is not a rate, and among a branch's first few dozen people any isolated complaint sits far above any sane threshold. Two is a pattern.

Editing the workflow stops the experiment

A campaign freezes when it sends. A workflow does not: its steps, its connections and the templates they use are all editable while an experiment is running, and an experiment whose branches change underneath it is measuring two different things as though they were one.

So Bitelio watches for it. When anything inside a branch changes, the experiment concludes using only the contacts whose window closed before that edit — they never saw the new version, so their data is still good. A three-week experiment somebody touched on its last day keeps its result. One edited before enough contacts had finished says so, and gives you nothing.

Editing a template counts as editing the workflow. An email step points at a template, and changing that template's body changes what the branch does with the workflow itself untouched.

"No difference was detectable" is not "the branches are identical"

It means the effect, if there was one, was smaller than this audience over this period could distinguish from noise. Bitelio says that rather than reporting a difference of zero, because zero is a far stronger claim than no data supports.

Promoting the winner

If you turn on send everyone down the winning branch, then once the experiment concludes with a winner, every contact arriving at the split afterwards goes that way. Contacts already inside keep the journey they started — pulling somebody out mid-sequence would break it.

Promotion does not rewrite your workflow. The graph in the editor is always the graph you drew; the split simply stops dividing. Cancel the experiment and it goes straight back to splitting.

Only a decisive result promotes. An experiment that ends inconclusive — the deadline passed, the workflow was edited, or there was never enough evidence — leaves the split splitting, even with promotion turned on. The control branch is recorded as the outcome that stands, but nothing is sent 100% down it on a result Bitelio itself calls undecided.

Cancelling records no result. A period cut short measured nothing.

What you need to run one

• A split step with at least two branches, exactly one of them marked as the control. The control wins ties, is the outcome recorded when the deadline passes undecided, and anchors the revenue comparison.
• A connected store, if you are measuring revenue. Measuring a conversion event needs nothing extra.
• Only one experiment per workflow at a time. Two splits measuring at once cross their effects, and neither result would mean anything afterwards. You can draw as many splits as you like; only one may be measuring.

Starting and cancelling require the workflows:publish permission, not workflows:edit — with promotion turned on this changes what live contacts receive. Reading the result requires workflows:view.

API reference

• POST /workflow-experiments — starts one. Body: { workflowId, stepId, metric, conversionEventName?, windowDays, decideBy, lossThreshold, minExposurePerArm?, autoPromote?, unsubscribeRatePct?, complaintRatePct? }. Requires workflows:publish.
• GET /workflow-experiments/:workflowId — the running experiment if there is one, otherwise the most recent, with its per-branch snapshot. Requires workflows:view.
• POST /workflow-experiments/:id/cancel — stops a running experiment and releases the split. Requires workflows:publish. Records no result.