Lead Scoring Pipeline
Batch inference for a lead-scoring classifier on SageMaker, orchestrated with Step Functions.
- Year
- 2024
- Role
- Data and ML engineering
- Stack
- Python, SageMaker, Step Functions, AWS Glue, PostgreSQL
Contents
Overview
A batch pipeline that scores open leads for an enterprise sales team with a classifier running on SageMaker, and writes the ranked list back for the sales workflow.
The problem
Manual qualification could not keep up with lead volume, and ad hoc model runs had no schedule, no retries, and no record of what was scored.
What I built
- Built the feature pipeline on Glue 4.0 with up to 20 concurrent G.2X jobs and real time ingestion through Kinesis Firehose into S3.
- Orchestrated SageMaker batch inference with Step Functions, with retries and a dead letter path for malformed records.
- Reduced spend across Timestream, Glue, and CloudWatch by consolidating schedules and right-sizing workers.
Impact
- Ranked leads land reliably, and malformed records quarantine to a dead letter path instead of failing the batch.
- Lower spend across the data services that feed the pipeline.