Switching CMP? How to measure the impact before and after
By The ConsentLift team · published · updated
CMP migrations are rarely decided on consent rates. Teams switch for price, for TCF support, for languages, because the group has standardised on one tool. But the consent rate is what the client will look at three weeks later, when their GA4 conversions have moved. Better to have planned for the answer.
The starting trap: two dashboards, two definitions
The reflex is to note the rate shown by the old CMP, switch, then look at the new one. It's the least reliable comparison you can make. The two tools can differ on almost everything:
- what counts as a display (every page, or once per session);
- what counts as an acceptance ("accept all" only, or partial selections too);
- the denominator behind the headline rate (consent rate or opt-in rate);
- how bots are treated.
A five-point gap between the two screens can come entirely from those differences, without a single visitor changing their mind. To measure a migration, you need the same measurement on both sides of the switch.
The protocol, in six steps
1. Measure for two weeks before
Put a CMP-independent measurement in place at least two weeks before the planned date. Two full weeks, so that each weekday appears twice: a Monday morning at the office and a Sunday evening on a phone are different audiences.
Record the three rates, opt-in, decision and consent, overall and by browser, device and page. That's your baseline.
2. Freeze what you can
A site redesign, a big campaign or a tagging overhaul in the same week as the switch, and nothing will be readable. If it can't be avoided, log the dates: a gap that lines up with a mobile campaign launch may have nothing to do with the CMP.
3. Map the categories
Each CMP has its own category names and its own way of wiring them to tags. Before the switch, build the table: which category in the old CMP fired what, and which category in the new one will have to. Check the link to Google Consent Mode signals in particular: an analytics category that no longer feeds analytics_storage will sink GA4 while the consent rate stays flat.
4. Date the switch to the hour
Log the day and time it went live. The switch day mixes both versions, so leave it out of the comparison. In ConsentLift, the date goes into the CMP Change Timeline, and the before/after comparison is built on it.
5. Wait for the re-prompt effect to fade
This is the step almost everyone forgets. The new CMP stores the visitor's choice its own way. Unless it can pick up the choices recorded by the old one (a question to ask the vendor before signing), every returning visitor sees the banner again.
For the first few weeks, banners shown and decisions both climb, and the people answering are no longer the same mix: regulars who had already chosen once, alongside new visitors. The consent rate in those weeks isn't the one the new banner will get in normal running. Where you can, compare weeks 3 and 4 after the switch with the two weeks before, rather than week 1.
6. Read the three rates, in order
Start with the decision rate. If it has dropped, the problem is technical before it's about the message: a banner that renders badly on one browser, a clash with another popup, a slower appearance. Break it down by browser and device: a drop concentrated on one segment shows up straight away.
Only then look at the consent rate: at an equal decision rate, does the new banner convince more or fewer people? That's where wording, button order and a first-layer "Reject all" come into play.
The opt-in rate sums up both. It's the number to give the client, with the other two to explain it.
How much traffic you need to conclude
A two-point gap between before and after may be real or may be chance. A statistical test settles it. As a first approximation, for an opt-in rate around 50%:
| Banners per period | Smallest detectable gap (95%) |
|---|---|
| 1,000 | about 4.4 points |
| 10,000 | about 1.4 points |
| 100,000 | about 0.4 points |
These figures assume banners are independent, which they aren't quite: one visitor can see several. Treat them as a minimum. On a small site, a migration can only be judged on gaps of several points, or over a longer period.
And a significant gap tells you the difference is real, not that the CMP caused it. That's what step 2 is for: knowing what else changed.
The migration checklist
| When | What to do |
|---|---|
| D − 15 or earlier | Independent measurement in place; baseline of the three rates by browser, device, page |
| D − 7 | Category mapping table; Consent Mode and tag wiring checked on staging |
| D − 7 | Ask the vendor whether existing choices are carried over |
| D | Switch logged to the hour; immediate display check on Safari iOS, Chrome Android, Firefox |
| D + 1 to D + 3 | Decision rate by browser: fix a localised drop now, not in a month |
| D + 14 to D + 28 | Comparison with the baseline on matching weekdays; statistical test |
| D + 28 | Client report: the three rates, before and after, with the uncertainty |
If the client asks whether the new rate is "good", our guide to what a good cookie consent rate looks like puts the published figures in context.
For agencies
A migration is often the first project where the client asks for hard numbers. It's also where the agency has most to lose if the numbers contradict each other. Having measured beforehand means answering in one page instead of three meetings. It's one of the use cases on our agencies page, alongside the pages for each supported CMP in our integrations.
Sources and notes
- Smallest detectable gap: two proportions at 50%, two-sided 5% risk, 1.96 × √(2 × 0.25 / n). ConsentLift's calculation.
- Observation period after a banner change: Cookiebot's documentation recommends one to two weeks ("Getting started with Consent Rate Benchmark").