Skip to content
← projectsdata

cloud-cost-predictor

123,000 real VMs. $5.9M/month recoverable. One recommendation per machine.

123,363VMs analysed$5.9Mmonthly savings identified98.1%CPU interval coverage
See code →View pitch →

The problem

Cloud over-provisioning is an incentives problem: whoever requests the resources is not the one who pays the bill, and turning something off in production always has more political cost than economic cost. The result is idle capacity nobody reclaims. The technical challenge is twofold: not just detecting waste, but saying exactly what to do with each machine — with enough confidence for the infrastructure team to act. And doing it without killing VMs that are low on CPU but high on memory.

The data

SAP Cloud Infrastructure public dataset (CC BY 4.0, Harvard Dataverse) with 123,363 real virtual machines and 30 days of telemetry: per-VM CPU and memory, percentiles and summary statistics. Costs are calculated with real AWS EC2 on-demand prices (us-east-1) from the Bulk Pricing API, allowing waste to be expressed in dollars per machine, not just as a percentage.

Approach

  • Quantile regression with XGBoost: six models produce the 10th percentile / median / 95th percentile for CPU and memory independently — the confidence interval is part of the output
  • Asymmetric loss function (3×): penalises underestimating usage more than overestimating it; a false positive (terminating a busy VM) costs far more than a false negative
  • VIF for multicollinearity cleanup: from all available metrics, only 5 features per model are kept after removing the correlated ones
  • Dual CPU + memory models: the final recommendation is the more conservative of the two — a VM with low CPU but high memory is left alone
  • Spike detection: 7,822 VMs with z-score ≥ 2.5 in the last 7 days are routed to manual review regardless of the prediction
  • Interactive pitch deck with keyboard navigation, live Plotly charts and entrance animations — deployed as a static page

Tools

python
xgboost
plotly

Result

$5.9M/month in waste identified: 502 VMs to terminate, 20,174 to downsize, 18,619 to review. The CPU model covers 98.1% of real values within the predicted interval; the memory model, 87.6%. 68% of the fleet is left untouched — deliberate conservatism.

Next steps

The natural next step is orchestrating the pipeline with Apache Airflow (nightly refresh) and managing transformations with dbt. For the memory model, obtaining daily time series — instead of summary statistics only — would enable trend and seasonality detection, pushing coverage above the current 87.6%. Temporal cross-validation (by month, not by row) is also pending to guard against overfitting.