LOCALYTICS · ML ENGINEER

Churn Prediction Model — Explainability & Bias Check

Time window	4 days from account invite (hard deadline) — estimated 5–7 focused hours
Format	Python, scikit-learn / XGBoost / LightGBM, sample dataset provided
Focus areas	Supervised learning, feature engineering, explainability, bias awareness

Scenario

We're providing a sample of raw events only (sessions, purchases, pushes, etc.) in JSON. We are not providing a pre-built feature set or a churn label.

This scenario sits inside the ingestion pipeline. The sample in the repo is the output of ingestion: raw events that are then stored in a data lake. Feature engineering and the churn model consume from that lake; you do not re-implement ingestion.

Your first modeling step is feature engineering: transform lake-resident raw events into RFM (Recency, Frequency, Monetary) features, define a churn label, and use that table as training data. You may add additional features if you justify them.

This mirrors the first prediction type we want live in production: churn probability feeding campaign audience selection. Your job as an ML Engineer is to train a credible model, justify the algorithm, evaluate it the way a churn / campaign-selection problem deserves, explain what drives the scores, check for uneven performance across subgroups, and propose how the model would run in production—not a one-off notebook experiment.

Getting started

Work through these in order. The sample is small on purpose; the walkthrough in data/README.md is part of the assignment, not optional background.

1. Read the schema. Start with data/dataset_schema.json for field definitions, event types, the as-of timestamp for windows, and the RFM expectation.
2. Inspect the sample. Open data/events.json (800 raw event rows, 80 customers). Confirm you understand the event shapes before you write transforms.
3. Engineer RFM features. Build a one-row-per-customer_id table from raw events. Suggested starting point:
   • Recency — days since last meaningful activity (e.g. last session) relative to as-of
   • Frequency — session (or activity) counts in lookback windows (e.g. 30d / 90d)
   • Monetary — purchase count and/or revenue in a lookback window (e.g. 90d)
   • Optional extras (push open rate, campaign clicks, support volume, etc.) if you justify them
4. Define a churn label. None is provided. Choose a definition that fits campaign audience selection, and keep feature windows from leaking the label. Defining features and the label from the same underlying activity can make a model look far more accurate than it actually is.
5. Scale the sample. Do not train or evaluate on the 80-customer raw sample alone. Write a reproducible script that generates additional synthetic customers/events that preserve the sample’s statistical structure (event mix, timing, amounts). Hundreds to a couple thousand customers is a reasonable range; justify the size and assumptions. See data/README.md.
6. Add segments if you need them for fairness. The raw schema has no profile/demographic fields. You may invent synthetic attributes (plan tier, acquisition channel, region, …) in the generation script so bias checks have something to slice on. Document that they are synthetic and state the assumptions.
7. Baseline, then model. Start with a simple heuristic or linear model. Beat it with a justified algorithm (tree ensemble is fine if the data’s shape supports it—say why). Use metrics that fit churn and campaign selection, not accuracy alone.
8. Explain and check bias. Apply SHAP (or equivalent), sanity-check that the drivers make sense, and report subgroup performance.
9. Propose production architecture. Diagram lake → features → training → scoring → campaign audience selection using the platform assumptions below (ingress, auth, observability). Do not invent a parallel platform. You do not need to ship a live scoring service.

Sample dataset

The sample dataset is data/events.json — 800 raw event rows in JSON. Start with data/dataset_schema.json for field definitions, event types, the as-of timestamp for windows, and the RFM expectation. See data/README.md for a short walkthrough.

This raw sample is intentionally small and meant to show you the event shapes, not to be your final training set. See data/README.md for guidance on scaling it up with synthetic data before you train or evaluate anything.

Observation time: use 2024-06-01T12:00:00Z as the as-of timestamp when computing recency and lookback windows.

AWS interview account

An AWS interview account will be created for you so you can complete this exercise. Access is limited to 4 days, starting when the AWS account invite is sent.

This account is shared and subject to strict budget limits. Size anything you launch conservatively (e.g. a short SageMaker training job, S3 for artifacts—avoid always-on endpoints if local training or a batch job would do). Resources that trip the budget limit may be suspended or reclaimed.

You are not required to deploy a live scoring API. A fully reproducible local training path (script + model artifact) is acceptable. If you do use the account (SageMaker training, S3 for features/artifacts, etc.), actually run the work there—a diagram-only mention of AWS is not the same as using it. Since account access lapses after 4 days, include in your submission whatever evidence (training logs, S3 paths, screenshots) an interviewer would need to confirm it worked, in case access has expired by the time we review it.

Localytics is happy to schedule a short meeting to confirm your credentials are working and to provide additional context about the project if needed. Reach out if you want to set that up.

Objectives

• Engineer features from raw events: build an RFM (Recency, Frequency, Monetary) training table (plus any justified extras), define a churn label without leakage, and keep that transform reproducible.

• Scale the sample into a training-sized dataset: write a reproducible script that generates additional synthetic customers/events preserving the real sample's statistical structure (event mix, timing, amounts), then justify the size and assumptions you chose. Don't train or evaluate directly on the 80-customer raw sample alone.

• Train a churn prediction model and justify your choice of algorithm given the data's shape. A more powerful model is not automatically the right one.

• Establish a clear baseline (a simple heuristic or model is fine) and show your approach improves on it with metrics that fit a churn / campaign-selection use case (not just accuracy). Think about class imbalance and the business cost of false negatives vs. false positives.

• Apply an explainability technique (SHAP or equivalent) to show which features drive predictions, sanity-check that the drivers make sense, and write a short interpretation a Head of Product could use.

• Check the model for bias: does it perform meaningfully worse for any subgroup implied by the available (or synthesized) features? If so, say what you'd do about it before anyone used the scores for campaigns.

• Propose how this model would run in production (training, feature refresh, scoring, monitoring) on the existing platform—as an architecture diagram that uses the ingress, auth, and observability assumptions below, not a full platform deployment.

How to submit

Fork this repository and put your complete solution in the fork. Your fork should include all of the code, artifacts, write-up, and diagrams for the exercise—do not submit materials only as separate attachments or links outside the repo.

What to submit

1. Feature Engineering & Reproducible Training: Code that transforms raw events → RFM (and any additional features), defines the label, generates the scaled synthetic dataset, trains the model, and reproduces results. Include the trained model artifact or a single script that regenerates it.
2. Evaluation Metrics: Show your metrics compared to your baseline (a simple heuristic or model). Include reasoning for why your chosen metrics are appropriate for a churn problem (class imbalance; cost of missing a likely churner vs. over-targeting).
3. Algorithm Note: A short justification of the model you chose given the data’s shape, and why it was the right level of complexity—not just the strongest library available.
4. Explainability Output: SHAP plots (or equivalent) plus a concise interpretation for a non-ML stakeholder. Call out anything that looks implausible.
5. Bias/Fairness Note: What you checked, what you found, and what you would recommend if there is a gap.
6. Architecture Diagram: A detailed diagram of the proposed production path on the existing platform (see Architecture Requirements): data lake → features → training → scoring → campaign audience selection. Include ingress/gateway (north-south flow and where rate limiting is applied), the stated auth model, OpenTelemetry → collector → centralized dashboard, auditing/security, and where an Agent tool would attach over MCP.

What we're evaluating

• Algorithm selection reasoning, not just picking the most powerful model available.

• Quality of feature engineering and label design from raw events (RFM definitions, leakage awareness, reproducibility)—not reliance on a handed feature matrix.

• Whether your evaluation approach actually fits a churn / campaign-selection problem (class imbalance, business cost of false negatives vs. false positives).

• Whether you treat explainability and bias checks as integral, not an afterthought bolted on at the end.

• How clearly you communicate model behavior and trade-offs to a non-ML audience, since this feeds a Head of Product conversation on the job.

• Sound methodology end-to-end. We're less interested in squeezing out the last 2% of AUC than in seeing a model the business could trust and act on.

Architecture Requirements

Provide a detailed architecture diagram of the proposed solution. Show how lake-resident raw events become features, how the model is trained and refreshed, how scores reach campaign audience selection, and how you would detect data-quality or score drift.

This is a proposed production design; you do not need to implement the full platform path or deploy a live scoring API. You do need to show how the model/service fits the existing ingress/gateway stack and authentication model. Treat the assumptions below as given. Do not invent a substitute platform, and do not leave ingress or auth as “TBD.”

Where this sits

• Ingestion is already in place. data/events.json represents ingestion output, which is stored in the data lake (S3 / Parquet).
• Your path consumes from the data lake. It does not re-collect or re-ingest mobile events.
• Call out auditing and security on the diagram (who can read lake data, who can request scores, what is logged).
• Show how an Agent tool would interact with this capability over MCP (diagram / short note only). You do not need to build agents in this exercise; come ready to discuss that in the follow-up conversation.

Ingress / gateway (pick one)

The platform already has these north-south options. Select one, use it in the diagram, and mark where rate limiting is applied:

• Amazon API Gateway in front of the scoring path
• EKS Ingress via the AWS Load Balancer Controller (ALB)
• SageMaker inference (real-time or serverless) fronted by API Gateway

Diagram the north-south data flow (external caller → gateway/ingress → scoring → response). Do not leave rate limiting as a footnote; put it on the path you chose.

Authentication (use this model)

Do not substitute a different identity story unless you have a specific reason and write it down.

• External callers: OIDC / JWT tokens issued by Keycloak.
• Service-to-service: mTLS at the mesh layer.
• Calls that need caller identity: scoped tokens in addition to mTLS.

Observability (use this model)

• Emit OpenTelemetry metrics — at least latency, error rate, and score distribution — to a collector.
• On the diagram, show how the collector forwards those metrics to the centralized platform dashboard.

Production readiness (diagram, not a second implementation)

The diagram should make these obvious without a separate essay: service-to-service auth (mesh mTLS + scoped tokens when identity matters), where rate limiting sits on the ingress you chose, and how OTel metrics reach the centralized dashboard. If you use the AWS interview account, you still do not need to stand up Keycloak, a service mesh, or a dashboard—those are platform fixtures you assume and draw.

Technology Stack

The following technologies are allowed and preferred:

• Well-known programming languages (Python and Terraform encouraged)
• scikit-learn / XGBoost / LightGBM (or an equivalent you justify)
• Spark/EMR/EKS
• SageMaker
• S3 (data lake for raw events and engineered features)
• IAM
• Athena
• Bedrock
• Airflow
• Parquet

Assume as already on the platform (do not rebuild them): Keycloak (OIDC/JWT for external callers), a service mesh (mTLS), an OpenTelemetry collector, and a centralized metrics dashboard. Pick one ingress option from Architecture Requirements.

While these are preferred, alternatives can be used if a valid reason is provided. Your solution does not require you to use all of these, just the ones your architecture needs.

Notes & ML Engineering Mindset

• Questions are Encouraged: We prefer clarity over guesswork. If you hit a wall, need clarification on constraints, or want to discuss a modeling trade-off, reach out—there is no penalty for asking.
• Methodology Over Leaderboard Score: A well-justified model that beats a sensible baseline, with the right metrics, is worth more than a small AUC gain you cannot explain.
• Leakage Is a Product Bug: If features and labels share the same future window, campaign targeting will look great offline and fail in production. Call out how you avoided that.
• The "So What?" Factor: Scores exist to choose who enters a campaign. For every modeling decision (label window, threshold, algorithm, fairness slice), be prepared to answer: How does this change who we target, and is that a good trade?
• Explainable Trade-offs: You will often need to balance sophistication against reliability and transparency. Explicitly state your trade-offs; a theoretically "perfect" scorer is often inferior to a reliable, explainable model the business can trust.
• Communicate Up: Write as if a Head of Product will read the explainability and bias notes. Avoid dumping metric tables with no interpretation.
• Looking Ahead: This exercise is scoped to the model, evaluation, and a production architecture on the existing platform—not agents. In the follow-up conversation, come ready to walk through your solution and discuss where an Agent / MCP component could extend it. No need to build that now.

Questions during the exercise? Reach out any time; we'd rather you ask than guess. There is no penalty for asking clarifying questions.