An agentic pipeline that critiques its own models

Problem

A global pharma client’s marketing-mix analysis was slow and expert-dependent. Every run meant an analyst manually assembling files, checking quality, engineering carryover variables, fitting models, and writing up findings — weeks of specialist time before anyone saw an ROI number. And once a run was finished, there was no easy way to ask what it had concluded.

Approach

Owned the knowledge base design and the definition of agent roles and responsibilities — what each agent knows, what it decides, and where it hands off.

The system runs as a set of specialist agents behind an orchestrator. A planner drafts the run. A data agent assembles a single modeling file from whatever the client supplied. A quality agent validates and standardizes it. A pre-modeling agent builds carryover and lag variables. A paired ideation and iteration agent proposes candidate models and scores them. A critique agent gates every major stage and sends work back when thresholds fail.

ZS owned ideation and architecture; AWS owned the build.

Outcome

Marketing teams run analyses two ways — conversationally, or through a guided step-by-step mode where they can inspect and adjust each stage. ROI estimates and model parameters flow into the client’s downstream budget optimization model, informing $10M+ in annual spend. An FAQ agent lets any user interrogate the system and its past runs without booking an analyst’s time.

Technical detail

Knowledge base

The knowledge base holds brands, channel definitions, past run results, model assumptions, client business rules, and data and database ownership. The planner, data, and pre-modeling agents all read from it; the FAQ agent answers against it directly.

Model selection

Model selection covers Bayesian and comparable approaches, scored on MAPE and R-squared, with the best-performing specification selected and its ROI carried forward.

Review & quality gates

Human review sits at every major threshold — the client’s team can approve a stage or send it back for rework, and the critique agent enforces automated quality gates before a human ever sees the output. That layering was deliberate: automated critique catches the mechanical failures so human attention is spent on judgment calls rather than error-checking.

Interface modes

The dual-mode interface was also a deliberate design choice. Chat is faster for users who know what they want. Guided mode exposes every intermediate stage for inspection and manual adjustment, which is what makes the system usable in an environment where someone may later have to explain how a number was produced.