Spotify A&R Intelligence System

AI-powered decision support for artist discovery and music industry analytics

---

Overview • Features • Results • Quick Start • Architecture

---

Overview

A production-ready data science project demonstrating expertise in data engineering, statistical analysis, machine learning, and product thinking. This system helps record labels and streaming platforms identify emerging artists before they break into the mainstream.

What This Project Does Transforms raw Spotify data into actionable A&R insights Identifies "breakout artists" before mainstream discovery Predicts track popularity with proper leakage prevention Provides ranked prospect lists for A&R teams

What Makes It Different Data Contracts protect downstream systems Causal Analysis isolates real effects Group + Time Splits prevent leakage Strong Baselines prove genuine value

---

Key Features

Data Contract Schema validation, anomaly detection, quality gates

ML Pipeline XGBoost, SHAP explainability, baseline comparison

Breakout Detection Ranking system, Precision@10: 90%

1. Data Contract & Quality System

• Schema validation with explicit contracts
• Anomaly detection across 8.7K tracks
• Quality metrics tracking (duplicates, missing data, distribution shifts)
• Explicit failure policy: Schema violations block, quality issues warn

2. Genre Normalization Pipeline

• Handles messy categorical data (N/A, [], mixed formats)
• TF-IDF based similarity inference for missing genres
• Popularity-weighted nearest neighbors
• Coverage improved: 48.78% → 99.95%

3. Domain-Specific Feature Engineering

Feature	Formula	Purpose
Popularity Efficiency	track_pop / log(followers)	Artists punching above weight
Competition Score	Weighted album density	Market saturation
Artist Dominance	Track count / total tracks	Market presence

4. Statistical & Causal Analysis

• Stratified regression controlling for confounders
• ANCOVA tests for explicit content effects
• Polynomial regression for saturation analysis
• Bootstrap confidence intervals (n=1000)

5. Popularity Prediction

• Group + Time splits (no leakage)
• 14 online-safe features only
• SHAP explainability
• 19.9% RMSE improvement over best baseline

6. Breakout Artist Detection

• Identifies high-potential artists with low current reach
• Binary classifier with ranking evaluation
• Precision@10: 90% (9/10 top picks are true breakouts)
• Labels computed from training data only

---

Results

Model Performance

Popularity Prediction (Group + Time Split)
Model	Test RMSE	Test R²	Type
global_mean	22.32	-0.001	Baseline
artist_mean	21.33	0.086	Baseline
genre_mean	19.44	0.241	Baseline
artist_popularity_proxy	19.41	0.243	Baseline
Ridge	15.70	0.504	ML
XGBoost	15.56	0.514	Best
LightGBM	15.64	0.508	ML

Breakout Artist Detection
ROC AUC	0.992
Average Precision	0.764
Precision@10	0.900
NDCG@10	0.994

Top Breakout Prospects

#	Artist	Score	Followers	Popularity
1	JD & The Straight Chasers	0.994	1.9K	49
2	Ashwin Gane	0.994	2.7K	47
3	Revline	0.994	491	47
4	Lexie Stevenson	0.993	103	34
5	AKI	0.993	754	45
6	MetaBoy	0.992	2.6K	48
7	Sugar Jesus	0.992	4.6K	47
8	FIXL	0.992	4.0K	46
9	midnight Berlin	0.992	4.2K	42
10	A!MS	0.992	7.0K	47

Key Metrics at a Glance

Metric	Value
Tracks Analyzed	8,778
Genre Coverage	99.95%
Pipeline Runtime	21.4s
RMSE Improvement	19.9%
Breakout Precision@10	90%

---

Quick Start

Prerequisites

Python 3.9+
Virtual environment tool

Installation

# Navigate to project
cd spotify

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run full pipeline
python run_pipeline.py

Run Individual Phases

python 01_data_contract.py        # Data validation
python 02_genre_normalization.py  # Genre inference
python 03_feature_engineering.py  # Feature creation
python 04_statistical_analysis.py # Causal analysis
python 05_popularity_model.py     # ML modeling
python 06_breakout_artist_detector.py  # Breakout detection

---

Architecture

                    ┌─────────────────┐
                    │  Raw CSV Data   │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────────────┐
                    │  Data Contract Layer    │  ◄── Schema validation
                    │  (01_data_contract.py)  │      Quality checks
                    └────────┬────────────────┘
                             │
                             ▼
                    ┌────────────────────────────┐
                    │  Genre Normalization       │  ◄── TF-IDF inference
                    │  (02_genre_normalization)  │      99.95% coverage
                    └────────┬───────────────────┘
                             │
                             ▼
                    ┌──────────────────────────┐
                    │  Feature Engineering     │  ◄── 14 online-safe
                    │  (03_feature_engineering)│      features
                    └────────┬─────────────────┘
                             │
              ┌──────────────┴──────────────┐
              │                             │
              ▼                             ▼
    ┌────────────────────┐       ┌──────────────────────┐
    │  Statistical Tests │       │   ML Models          │
    │  ANCOVA, Bootstrap │       │   XGBoost + SHAP     │
    └────────────────────┘       └──────────────────────┘
              │                             │
              └──────────────┬──────────────┘
                             ▼
                    ┌──────────────────┐
                    │    Artifacts     │
                    │  Models, Reports │
                    │  Figures, CSVs   │
                    └──────────────────┘

---

Project Structure

spotify/
├── config.py                        # Central configuration
├── requirements.txt                 # Dependencies
├── run_pipeline.py                  # Master orchestrator
│
├── 01_data_contract.py             # Phase 1: Data validation
├── 02_genre_normalization.py       # Phase 2: Genre inference
├── 03_feature_engineering.py       # Phase 3: Feature creation
├── 04_statistical_analysis.py      # Phase 4: Causal analysis
├── 05_popularity_model.py          # Phase 5: ML modeling
├── 06_breakout_artist_detector.py  # Phase 6: Breakout detection
│
├── artifacts/                      # Generated outputs
│   ├── best_model_xgboost.joblib
│   ├── breakout_detector.joblib
│   ├── model_comparison.csv
│   └── top_breakout_prospects.csv
│
├── figures/                        # Visualizations
│   ├── 04_shap_summary_xgboost.png
│   ├── 06_model_comparison.png
│   └── 07_breakout_detection_eval.png
│
└── logs/
    └── spotify_pipeline.log

---

Tech Stack

Core	Python 3.9+, Pandas, NumPy
ML	Scikit-learn, XGBoost, LightGBM, SHAP
Stats	SciPy, Statsmodels, Bootstrap
Viz	Matplotlib, Seaborn, Plotly
Storage	Parquet (PyArrow)

---

Critical Learning: Leakage Detection

Initial Results (Suspicious) R² = 0.999 ← Red flag! ROC AUC = 0.997 Root Cause: Artist aggregates leaked target info Random splits allowed artist overlap Labels computed on full dataset

After Fix (Realistic) R² = 0.514 ← Credible Precision@10 = 90% Solution: Group + Time splits Online-safe features only Labels from training data

Key Insight: Near-perfect ML results are usually wrong. The fix demonstrates production-ready thinking.

---

Future Enhancements

• Streaming Pipeline (Kafka + Spark)
• Drift Monitoring (Evidently AI)
• Interactive Dashboard (Streamlit)
• API Deployment (FastAPI)
• Time-series Forecasting
• Collaborative Filtering

---

Author

Abdulrahman
Data Scientist | Machine Learning Engineer

---

_{Data sourced from Kaggle. For educational purposes.}