An agentic pipeline that critiques its own models

Problem

A global pharma client’s marketing-mix analysis was slow and expert-dependent. Every run meant an analyst manually assembling files, checking quality, engineering carryover variables, fitting models, and writing up findings. That was weeks of specialist time before anyone saw an ROI number. And once a run was finished, there was no easy way to ask what it had concluded.

Approach

I owned the knowledge base design and the agent roles: what each agent knows, what it decides, and when it hands off to the next one.

The system runs as a set of specialist agents behind an orchestrator. A planner drafts the run. A data agent assembles a single modeling file from whatever the client supplied. A quality agent validates and standardizes it. A pre-modeling agent builds carryover and lag variables. A paired ideation and iteration agent proposes candidate models and scores them. A critique agent gates every major stage and sends work back when thresholds fail.

ZS owned the design and architecture; AWS owned the build. So a big part of my job was getting everyone to agree. Both sides cared how the system got built, which meant we worked through the architecture with the client and AWS before AWS started, rather than throwing a spec over the wall. One live call was whether modeling should be a single agent or split into an iterator that proposes and refines models and an executor that runs them. We split it, accepting more moving parts for a system that scales and adapts more easily.

Outcome

Marketing teams can run an analysis by chatting with it, or step by step in a guided mode where they can inspect and adjust each stage. ROI estimates and model parameters flow into the client’s downstream budget optimization model, informing $50M+ in annual spend. An FAQ agent lets anyone ask questions about the system and its past runs without booking an analyst.

Technical detail

Knowledge base

The knowledge base holds brands, channel definitions, past run results, model assumptions, client business rules, and data and database ownership. The planner, data, and pre-modeling agents all read from it; the FAQ agent answers against it directly.

Model selection

Model selection covers Bayesian and comparable approaches, scored on MAPE and R-squared, with the best-performing specification selected and its ROI carried forward.

Review & quality gates

Human review sits at every major threshold. The client’s team can approve a stage or send it back for rework, and the critique agent enforces automated quality gates before a human ever sees the output. That layering was deliberate: automated critique catches the mechanical failures so human attention is spent on judgment calls rather than error-checking.

Interface modes

The dual-mode interface was also a deliberate design choice. Chat is faster for users who know what they want. Guided mode exposes every intermediate stage for inspection and manual adjustment, which is what makes the system usable in an environment where someone may later have to explain how a number was produced.