vanguard-ab-test
70,609 clients, 755,405 events. Did the Vanguard redesign work?
See code →The problem
Vanguard redesigned their digital onboarding. The experiment data is public. The interesting analysis is not computing whether there is a statistical difference — with 70,000 clients almost any difference comes out significant. The hard part is deciding which metric matters, whether the effect size justifies rollout across all segments, and what to do when the effect is not homogeneous.
The data
Public Vanguard dataset: 70,609 clients randomly split between control (original UX) and variant (redesigned UX). 755,405 web events with timestamps, onboarding steps and completion label. Available segmentation variables: age, account type, entry channel.
Approach
- Event data cleaning: deduplication, broken sessions, users appearing in both groups (exclusion)
- Hypothesis testing per metric: chi-square for completion rate (binary), Welch t-test for time in process (continuous)
- Effect size: odds ratio and Cohen's h — the p-value only says whether there is a difference, not how large it is
- Effect homogeneity analysis by cohort: age, account type, device — the average effect hides important variance
- Step-by-step conversion funnel: which exact step the variant wins or loses against control
- Tableau visualisation: interactive funnel and effect decomposition by segment
Tools
python
sqlResult
The variant improves the onboarding completion rate, but the effect is not homogeneous: it is significant for clients under 45 and practically zero for older ones. The recommendation is not "roll it out" but "roll it out selectively" — and quantify which share of the client base captures the real effect.
Next steps
The current analysis compares groups at experiment end. The natural extension is survival analysis to see on which day of the process the largest difference between groups occurs, and CUPED (variance reduction) to reduce estimator noise by leveraging users' pre-experiment metrics.