drive-failure-predictor

The problem

Drives fail on a bathtub curve: many failures early (infant mortality), few in the stable phase, and many again at end of life. The problem is not just detecting when the stable phase ends: it is doing so with enough lead time for the storage team to act, and with a threshold calibrated for that specific environment — because in a datacenter with 24/7 support contracts, a false negative costs far more than a false positive.

The data

Backblaze SMART dataset with 318,426 drives in real production (HDDs of different manufacturers and generations). ~10 million rows. Key variables: reallocated sectors count (S5), reported uncorrectable errors (BB), pending sectors (C5). Real failure rate: ~0.8% — the dataset is heavily imbalanced by design.

Approach

• Polars instead of pandas: processes 10M+ rows without running out of RAM — pandas OOMed on the development machine
• XGBoost for binary classification (will fail in the next 7 days / will not fail), with scale_pos_weight to compensate for class imbalance
• Strict temporal validation: train on previous quarters, test on the next quarter — dates are never mixed
• Threshold calibration by failure-cost / replacement-cost ratio: the optimal cut-off is not 0.5 when costs are asymmetric
• Feature importance analysis: the three predictive SMART attributes match exactly what the Backblaze literature identifies

Tools

python

xgboost

polars

Result

96% recall on the temporal test set (F1 0.77). The model flags drives that are going to fail with 3 to 7 days of lead time. The analysis makes explicit that precision and recall move in opposite directions as the threshold shifts — and that the right choice depends on the cost of each error in the specific environment.