etl-social-pipeline

The problem

tokpipe and instapipe solve the parsing. This project solves the next problem: running both pipelines in a coordinated way, consolidating the output into a unified data model, and automatically knowing when something fails or falls outside the norm — before opening the dashboard.

The data

Same source as tokpipe and instapipe. The difference is the unified data model: a metrics table per platform with the same columns and the same definitions, plus enough history to compute baselines.

Approach

• Prefect orchestration: each pipeline has its own task, errors trigger automatic retries with backoff, persistent failures generate an alert
• Unified data model: a single metrics table per platform, with the columns established by tokpipe and instapipe
• Per-metric anomaly detection: 4-week rolling mean and standard deviation as baseline, alert if the current week falls outside ±2σ
• Automated weekly HTML report: cross-platform comparison, anomaly flagging and context for the last 8 weeks

Tools

python

prefect

Result

A weekly report that generates itself. Anomaly detection catches reach drops or engagement spikes before you see them in the dashboard — and with the historical context the dashboard does not keep.