A/B Tests: Are you sure your P2BB calculation is correct? �
complete
Nikita K.
Yesterday Paywall A was showing a P2BB of 100%, meaning there is literally zero chance of it losing. Today Paywall B is showing a P2BB of 83%. How can a 100% probability flip to 17% like that overnight? Seems like it shouldn't display 100% until it gets enough data to be statistically accurate. I almost turned off the A/B test yesterday when I saw 100%. Glad I didn't.
Adapty Support
complete
Adapty Support
Hey there,
Thanks for reaching out.
The calculation for P2BB is carried out by a statistical model which depends on the sample's nature, i.e. sample size and other statistical metrics. For example, if the sample size is very small, observing such leaps is statistically expected. When P2BB is stated as 100% the Frontend displays a "insufficient data" tag on the A/B metrics overview.
Best,