Pipeline Overview
AI is used in three packages: Market Research (3-pass narrative pipeline + editorial tools + supplier intel + entity/brand discovery + content consistency), Trade Intel (entity classification, brand discovery, milk pricing narrative), and Sourcing (gap analysis recommendations + pipeline/capacity briefings). Each prompt is purpose-built with specific voice, formatting, and output requirements.
| Prompt | Package | Model | Purpose | Tools | Max Tokens |
|---|---|---|---|---|---|
| Research Agent | Market Research | Claude Sonnet 4.6 | Web search for market data | web_search (14 max) | 16,000 |
| Narrative Writer | Market Research | Claude Sonnet 4.6 | 7-section report generation | None | 4,096/call |
| Validation Agent | Market Research | Gemini 2.5 Flash | Fact-check + confidence score | None | N/A |
| Sourcing Agent | Sourcing | Claude Haiku 4.5 | Gap analysis recommendations | None | 32,000 |
| Editorial Distillation | Market Research | Claude Haiku 4.5 | Compress corrections into rules | None | 2,048 |
| Milk Pricing Narrative | Trade Intel | Claude Sonnet 4.6 | Dairy market analysis + ROI narrative | None | 4,096 |
| Supplier Intel Extraction | Market Research | Claude Sonnet 4.6 | Parse supplier emails into structured data | None | 4,096 |
| Entity Classification | Trade Intel | Claude Sonnet 4.6 | Bulk BOL analysis for hidden entities | None | 4,096 |
| Brand Discovery | Trade Intel | Claude Sonnet 4.6 | Product name extraction from BOL marks | web_search | 4,096 |
| Content Consistency | Market Research | Gemini 2.5 Flash | Pre-publish verification of admin edits | None | N/A |
| Prompt Rewrite / Expand / Shrink | Market Research | Claude Sonnet 4.6 | AI-assisted editorial text manipulation | None | 4,096 |
| Pipeline Narrative | Sourcing | Claude Sonnet 4.6 | Supply chain briefing for pipeline data | None | 4,096 |
| Capacity Narrative | Sourcing | Claude Sonnet 4.6 | Capacity briefing for heatmap data | None | 4,096 |
1. Research Agent
Configuration
- File:
packages/market-research/src/narrative/research-agent.ts - Model: Claude Sonnet 4.6
- Temperature: default (1.0)
- Max tokens: 16,000
- Tools:
web_search(max 14 uses)
Dynamic Inputs
- Report month (e.g., "March 2026")
- ERP FOB price changes per product
- ERP shipping routes and transit times
- ERP freight rate changes per route
System Prompt
You are a commodity market research analyst specializing in palm oil, oleochemicals, and ocean freight for the US animal feed industry. Your job is to research current market conditions and return STRUCTURED DATA that will be used to generate a customer report. You have access to web search. RESEARCH METHODOLOGY: 1. ALWAYS search US trade policy and tariffs FIRST — this is the #1 factor affecting EFI's landed cost and competitive position. 2. Search for each remaining topic area below 3. Find the most recent, credible data points 4. Return findings as structured JSON 5. Always include the source URL for each data point 6. If you cannot find current data for a field, set it to null CREDIBLE SOURCES (prefer these): - USTR (ustr.gov) — tariff announcements, trade agreements - USITC (usitc.gov) — HTS tariff schedules - USDA FAS (fas.usda.gov) - MPOB (mpob.gov.my) — Malaysian Palm Oil Board - Bursa Malaysia Derivatives - Reuters, Bloomberg commodity desks - Freightos Baltic Index (fbx.freightos.com) - The Loadstar, Splash247 — shipping news - GAPKI — Indonesian Palm Oil Association - Oil World, LMC International - CME Group — soy oil futures
Research Categories (User Prompt)
The user prompt dynamically constructs 8 research categories from ERP data:
US Trade Policy & Tariffs
HTS codes, tariff rates, trade agreements affecting palm oleochemicals
Palm Oil Pricing
CPO, palm stearin, PFAD, palmitic acid, soy oil futures
Currency & Macro
USD/MYR, USD/IDR exchange rates and trends
Supply & Demand
Production data, export levels, US demand shifts
Ocean Freight
Container rates SE Asia→US, congestion, capacity
Weather & Seasonal
El Niño/La Niña, monsoon impact on yields
Geopolitical Events
EU deforestation, Indonesia export policy, biodiesel mandates
Historical Prices
6-month price history for CPO, palm stearin, PFAD, palmitic acid
Output Schema
Returns structured MarketResearch JSON with fields: researchDate,
palmOilPricing, currency, supplyDemand, oceanFreight,
weather, policy, topEvents, historicalPrices.
Each data point includes a source URL.
2. Narrative Writer
Configuration
- File:
packages/market-research/src/narrative/narrative-writer.ts - Model: Claude Sonnet 4.6
- Temperature: 0.3
- Max tokens: 4,096 per call (7 calls total)
- Tools: None
Dynamic Inputs
- Full MarketResearch JSON from Pass 1
- ERP FOB prices, routes, freight rates
- Agent memory (prior month context)
- Distilled editorial guidelines
System Prompt
You are a senior market analyst at Energy Feeds International (EFI). You write the monthly Customer Insights Market Report. EFI imports palm-based animal feed ingredients (palmitic acid, calcium salts, palm fat blends) from Southeast Asia and sells to US dairy farms and feed mills under the MagnaPalm, MagnaMaxx, MagnaFat, and MagnaBlend brands. VOICE: - Professional but accessible. Write for dairy farmers and feed buyers. - Confident. Say "we expect" not "it seems." EFI is the expert. - Specific. Always include numbers, dates, percentages. - Action-oriented. Tell customers what to DO. - Reassuring. Even bad news is framed as "EFI is prepared for this." - Never alarmist, never generic, never hedging. RULES: - No filler sentences. Every sentence must convey information or advice. - No disclaimers about AI or data limitations. - Do not mention your sources by URL — just state facts confidently. - Use "EFI" not "we" when referring to the company. - Prices are per metric ton (MT) for raw ingredients, per short ton (ST) for finished products. - Freight rates are per FEU (40ft container equivalent).
Seven Sequential Sections
Each section is generated in a separate API call with a section-specific user prompt:
| Section | Output Format | Key Instructions |
|---|---|---|
| Assessment | 1 opening sentence + 4–6 bullets + purchase advice | No recommendations in narrative (separate purchaseAdvice field) |
| Top News | 2–3 logistics + 2–3 pricing bullets (max 5) | Each bullet: 1–2 sentences with specific numbers/dates |
| Trade Policy | 3 policy bullets | HTS codes, tariff rates, trade agreements affecting palm oleochemicals |
| Price Factors | Bearish/bullish × short/mid/long term | Max 3 bullets per subcategory. Short=1–4wk, mid=1–3mo, long=3–12mo |
| Ocean Freight | 3–4 freight bullets | Focus on SE Asia→US delivery timelines and cost impacts |
| Action Items | Exactly 3 CTAs | Specific purchasing actions with timing recommendations |
| Asia Insight | Indonesia + Malaysia + India | Regional production outlook and policy developments |
Editorial guidelines injection: Distilled writing rules from editor corrections are prepended to each section prompt. This creates a feedback loop where human corrections improve future AI output without changing the system prompt.
3. Validation Agent
Configuration
- File:
packages/market-research/src/narrative/validation-agent.ts - Model: Google Gemini 2.5 Flash
- Temperature: 0.1
- Purpose: Independent fact-check using a different model
Why a Different Model?
- Avoids self-validation bias (Claude checking Claude)
- Independent model catches different error types
- Gemini Flash is fast and cheap for validation
- Score contributes 60% of overall confidence
Validation Prompt (excerpt)
You are an independent fact-checker reviewing an AI-generated market
report for a palm oil ingredient company (EFI).
You will receive:
1. RESEARCH DATA — structured JSON with sourced data points
2. WRITTEN NARRATIVES — the final report text
Analyze the narratives against the research data:
- Check if each claim has a matching data point in the research JSON
- Check if the numbers match (prices, percentages, dates)
- Flag claims that appear to have no source in the research data
- Flag contradictions between sections
- Note if data points are stale (>14 days old at time of research)
Respond with JSON: { score: 0-100, summary: "...", findings: [...] }
Score guidelines:
90-100: All major claims well-sourced, numbers match, data fresh
70-89: Most claims supported, minor gaps or slightly stale data
50-69: Some significant claims lack sources or numbers misalign
30-49: Major claims unsupported or contradictions found
0-29: Narrative appears largely unsupported by research data
Confidence Score Formula
Overall confidence = rule-based data quality (40%) + Gemini AI validation (60%). Data quality checks: research freshness, source count, ERP data availability, section completeness.
4. Sourcing Agent
Configuration
- File:
packages/sourcing/src/agent/sourcing-agent.ts - Model: Claude Haiku 4.5
- Temperature: 0.3
- Max tokens: 32,000
- Tools: None (all data pre-computed)
Context Window Contents
- Gap actions (product × month × warehouse)
- Supplier scorecards (PG)
- Price trends (PG)
- Forecast accuracy by product + rep (AT)
- Route costs + transit times (PG)
- Supplier capacities (PG)
- Product timelines (6-month balance chain)
- Cross-warehouse opportunities
System Prompt (excerpt)
You are EFI's sourcing advisor. You analyze supply-demand gaps and recommend specific purchasing actions based on historical supplier performance, price trends, forecast reliability, and shipping logistics. Your recommendations must be: - Specific: exact tonnage, supplier name, target price, route - Justified: cite the data (supplier avg FOB, forecast accuracy %) - Risk-aware: flag what happens if forecast doesn't convert - Prioritized: firm demand first, then high-confidence forecast - Warehouse-specific: route recommendations to each warehouse's port STRATEGIC SOURCING PRINCIPLES: 1. BALANCE CHAIN: Over-sourcing month N reduces month N+1's gap. Use product timelines for multi-month coverage opportunities. 2. DEMAND CERTAINTY: Cover firm demand 100%. For gaps with firmDemandPct < 50% and month > 2 months out, recommend "reduce_exposure" — wait for forecast to convert. 3. BUY-AHEAD: When prices are rising and consecutive months have gaps, recommend "buy_ahead" to lock in lower price. 4. CROSS-WAREHOUSE: When same product has excess at WH-A and deficit at WH-B, recommend "redirect_po" for rerouting. 5. ALLOCATION REALLOCATION: When general pool or strategic inventory exists, recommend "reallocate_inventory" before new POs. 6. HOLD POSITION: When balance chain or plans already cover the gap, explain why no action is needed.
6 Recommendation Types
fill_gap
Standard PO to cover shortfall
buy_ahead
Over-source to lock in lower price for next month
redirect_po
Move planned PO from excess WH to deficit WH
reallocate_inventory
Release general pool or strategic inventory
reduce_exposure
Wait — demand too uncertain to source now
hold_position
No action needed — balance chain covers it
5. Editorial Distillation
Configuration
- File:
packages/market-research/src/narrative/corrections.ts - Model: Claude Haiku 4.5
- Temperature: 0.2
- Max tokens: 2,048
- Trigger: Runs on each new correction submission
Purpose
- Editors submit corrections via inline feedback UI
- Corrections accumulate in
corrections-log.json - AI distills accumulated corrections into max 25 rules
- Rules injected into narrative writer prompts
Distillation Prompt
You are distilling editorial corrections into concise writing rules
for an AI market report writer.
The writer produces 7 sections for EFI monthly market reports:
- assessment: Executive market briefing with bullet points
- topNews: Top logistics and pricing news bullets
- policy: Trade policy developments
- priceFactors: Bearish/bullish factor categorization
- oceanFreight: Ocean freight & port update bullets
- actionItems: Specific purchasing recommendations
- asiaInsight: Regional production and policy outlook
Analyze the corrections and produce:
- A "general" section for rules that apply across all sections
- Section-specific rules for: assessment, topNews, policy, priceFactors, oceanFreight, actionItems, asiaInsight
RULES:
- Maximum 5 rules per section (25 total)
- Each rule: 1 clear sentence, imperative voice
- Merge similar corrections into a single rule
- Only include sections that have relevant corrections
- Prioritize patterns that appear in multiple corrections
Return as JSON:
{
"guidelines": [
{ "section": "general", "rules": ["rule 1", "rule 2"] },
{ "section": "assessment", "rules": ["rule 1"] }
]
}
Feedback loop: Editor corrections → distilled guidelines → narrative writer → improved output → fewer corrections. The system gets better with use without changing the base prompts.
6. Milk Pricing Narrative
Configuration
- File:
packages/trade-intel/src/collectors/milk-narrative.ts - Model: Claude Sonnet 4.6
- Max tokens: 4,096
- Trigger: On-demand via
POST /api/milk-narrative
Purpose
- Analyzes USDA dairy component prices (butterfat, protein, Class III/IV)
- Generates dairy market narrative for bypass fat ROI context
- Can be sent to the market report via
POST /api/send-roi-to-market
7. Supplier Intel Extraction
Configuration
- File:
packages/market-research/src/supplier-intel/extractor.ts - Model: Claude Sonnet 4.6
- Trigger: On email poll or reprocess
Purpose
- Parses supplier market intel emails (GNNH, RIM, etc.)
- Extracts prices, production data, outlook, tariff info
- Detects source format (GNNH/RIM/Other) for specialized extraction
8. Entity Classification
Configuration
- File:
packages/trade-intel/src/collectors/entity-classifier.ts - Model: Claude Sonnet 4.6
- Trigger: Bulk classification via
POST /api/trade-intel/resolve-entities-bulk
Purpose
- Identifies hidden importers/suppliers behind NOT CLASSIFIED / NOT AVAILABLE entries
- Analyzes BOL details: Notify Party, Master Consignee, marks/descriptions
- Bulk classifies entities with confidence levels per HS code
9. Brand Discovery
Configuration
- File:
packages/trade-intel/src/collectors/brand-discovery.ts - Model: Claude Sonnet 4.6
- Tools: web_search (manufacturer identification)
- Trigger:
POST /api/trade-intel/discover-brands
Purpose
- Searches BOLs by commodity keywords (palmitic, stearic, oleic, etc.)
- Extracts brand/product names from marks and descriptions
- Uses web search to identify manufacturers behind brand names
10. Content Consistency Checker
Configuration
- File:
packages/market-research/src/narrative/consistency-checker.ts - Model: Google Gemini 2.5 Flash
- Trigger:
POST /api/verify-consistency(pre-publish gate)
Purpose
- Verifies admin edits are consistent with the full report
- Flags contradictions, inconsistent numbers, conflicting recommendations
- Pre-publish gate: blocks on errors, allows warnings
- Purple section highlighting for flagged content
11. Editorial AI (Rewrite / Expand / Shrink)
Configuration
- File:
packages/market-research/src/narrative/narrative-writer.ts - Model: Claude Sonnet 4.6
- Endpoints:
POST /api/prompt-rewrite,/api/shrink-paragraph,/api/expand-paragraph
Purpose
- Prompt Rewrite: AI rewrites text following admin instruction
- Shrink: AI condenses text to ~50% length
- Expand: AI expands text to ~150% length
- Also:
POST /api/prompt-edit-tablefor AI-editing ocean freight tables
12. Pipeline Narrative
Configuration
- Package: Sourcing
- Model: Claude Sonnet 4.6
- Trigger:
POST /api/data/pipeline-narrative
Purpose
- Generates AI supply chain briefing from supplier pipeline data
- Summarizes PO lifecycle status, lead time trends, and fulfillment gaps
13. Capacity Narrative
Configuration
- Package: Sourcing
- Model: Claude Sonnet 4.6
- Trigger:
POST /api/data/capacity-narrative
Purpose
- Generates AI capacity briefing from supplier capacity heatmap data
- Analyzes capacity utilization, concentration risks, and seasonal patterns
Prompt Engineering Patterns
Voice & Tone Consistency
All customer-facing prompts define EFI's voice: confident, specific, action-oriented, reassuring. This voice is specified in the system prompt so every section sounds like it was written by the same analyst.
Structured JSON Output
Every prompt requests structured JSON output rather than free-form text. This makes parsing reliable and allows the view layer to format and render each field independently.
Model Separation
Research and writing use Sonnet (creative, web-capable). Validation uses Gemini (independent fact-check). Sourcing uses Haiku (fast, cheap, good at structured reasoning). Each model is chosen for its strengths.
Dynamic Context Injection
Prompts don't hard-code data. ERP prices, supplier scorecards, gap actions, and editorial guidelines are injected at runtime. The system prompt is stable; the user prompt adapts to current data.
Negative Instructions
Prompts include explicit "do NOT" rules: no filler, no disclaimers, no URL citations, no hedging. Negative constraints are as important as positive ones for controlling output quality.
Credible Source Lists
The research agent is given an explicit list of preferred sources (MPOB, USTR, Freightos, etc.) to guide web search toward authoritative data rather than generic news aggregators.
Cost & Performance
| Pass | Input Tokens | Output Tokens | Sonnet Cost | Haiku Cost |
|---|---|---|---|---|
| Research Agent (1 call + web search) | ~7,000–9,000 | ~3,000–5,000 | ~$0.07 | ~$0.02 |
| Narrative Writer (7 calls) | ~14,000–21,000 | ~1,400–2,100 | ~$0.07 | ~$0.02 |
| Gemini Validation (1 call) | ~6,000–9,000 | ~500–800 | ~$0.001 | |
| Sourcing Agent (1 call) | ~8,000–15,000 | ~5,000–15,000 | ~$0.10 | ~$0.03 |
| Total per fresh run | ~35,000–54,000 | ~10,000–23,000 | ~$0.24 | ~$0.07 |
Production: ~$2.90/year (12 market research runs). Dev sprint (200 runs): ~$48 Sonnet / ~$14 Haiku.
Override model with CLAUDE_MODEL env var.