Generative AI has captured the imagination of businesses everywhere. While much of the early buzz has focused on marketing content, chatbots, and creative applications, its role in operations—the engine room of every organization—remains nascent.

Operational domains such as supply chain, procurement, and workforce management work on the basic tenet of data, decisions, and actions, in that order. In this context, accuracy is non-negotiable. Yet one of the known limitations of large language models is their tendency to generate confident but incorrect answers—a risk that becomes especially acute when dealing with data-driven tasks.

The real challenge, then, is not whether to use generative AI in operations, but where it belongs: identifying the points where it can complement deterministic analytics rather than replace it, adding value without compromising reliability.

Two Complementary Lines of Capability

When thinking about GenAI in operations, it helps to separate the solution into two complementary lines:

• Predictive Analytics = detect & predict.

  Algorithms that find anomalies, forecast demand, optimize scheduling, or track performance. Provide the what—the anomalies, forecasts, and optimizations.

• Generative AI = explain, communicate, act.

  Models that interpret results, enable natural interaction with data, and guide the next action. Provide the so what and the now what—explaining, contextualizing, and guiding action.

Analytics supplies the facts. Generative AI supplies the understanding. This framing is powerful because it allows us to tell a balanced story: clients can see why deterministic analytics is essential for accuracy and scale, while also appreciating where GenAI adds unique value.

Analytics Does the Math, GenAI Makes It Actionable

Analytics is the backbone of operational decision-making. It can crunch millions of rows of data to detect anomalies, forecast trends, and optimize processes. But once the results are in, the harder part begins:

• Why did this happen?

• Who needs to know?

• What should we do next?

Take margin outlier detection as an example: we can build a simple t-distribution model that takes in data at different levels (branch vs region, weekly vs month avg) and reliably identifies when costs and prices deviate from expectations. This can be accomplished using LLM but:

• much harder to do it reliably every single time

• much harder to scale dure to limitation in context length as well as latency as compared to predictive programming

Hence, the outlier detection algorithm is still in the domain of predictive AI but GenAI can step in to:

• Generate a narrative explanation of the root cause.

• Personalize the insight for different roles.

• Suggest secondary or “may-be” causes when deterministic rules don’t fit.

• Draft follow-up actions, like a buyer email or an updated pricing sheet. It can go a step further in taking automated actions on behalf of the users through custom tools

The same pattern extends to many areas of operations.

A Framework for GenAI in Operations

The use cases for generative AI can be clustered into categories that illustrate where LLMs are particularly effective, as outlined in the following. These categories can be used as brainstorming lines to explore specific, high-value operational enhancements.

Natural Language Handling

1. Classification: Mapping free-form text into structured codes or categories, even when phrasing is novel or inconsistent.

   Examples:

   • Ascertain the type of question asked in chat and use specific agent to furnish the request.

   • Routing support tickets to the correct department.

2. Extraction: Pulling entities/relations out of contracts, support tickets, call transcripts or any other corpus of text.

   Examples:

   • Convert all vendor receipts into structured data for analysis (indirect cost)

   • Extract the right information from a phone transcript from a store or an email to the buyer

   • Pull clauses (e.g., renewal terms, penalties) from vendor contracts for relational storage and retrieval

Summarization & Narrative Generation

1. Summarization: Condensing reports, notes, or feedback into concise executive briefs.

   Example:

   • Create daily digest of top outliers for a buyer

2. Explanation: Turning raw KPIs or anomalies into human-readable narratives

   Examples:

   • Generate narrative for outlier detection + root-cause analysis

   • Explaine why a forecast changed week-over-week (forecast explainability).

   • Interpret ML model results in natural language

3. Personalization: Tailoring the same insight differently for different roles

Human Interaction Layer

1. Conversational interfaces: Let business users interrogate data with natural language.

   Examples:

   • Chat interface to query data directly

2. Tutoring / onboarding: Explain what a metric means, how to interpret it, or how a process works.

   Examples:

   • Create onboarding documents (text, images and videos).

   • Interactive world models (Genie 3 by Google) for process simulation

3. Decision support: Draft action plans, emails, or reports grounded in model outputs.

   Examples:

   • Daily email to buyers about outliers

   • Draft report on supplier performance with root-cause and action steps.

Reasoning & Semantic Search Across Heterogeneous Inputs

1. Unstructured + Structured: Combine numeric facts with policy text or emails to suggest causes or actions.

2. Contextual Q&A: Answer “why” questions that blend multiple modalities.

   Examples:

   • Brainstorming/ideation for scenarios that don’t require precision (chat combining streaming data + fan feedback for a music app).

3. Semantic Search: Search through data store and documents using meaning instead of keywords and retrieve “the closest ” or “most similar” without handcrafted logic rules.

   Examples:

   • Retrieve most similar outliers within the last month

   • Return the best response out of n. This can be used as a consensus agent that asks the same question multiple times and picks out the best possible one, enhancing robustness

Agentic Actioning

1. Communication: email, text, phone

   Examples:

   • Call a store with automated recommendations, capture manager response, log it in DB.

   • Send summary emails/texts after process completion

2. Read/Write to source systems

   Examples:

   • Update pricing worksheet once buyer approves changes.

   • Push adjusted forecasts into ERP system after validation.

   • Automatically create a JIRA ticket when anomaly is detected.

Broader Operations Use Cases for Generative AI

Supply Chain Management

• Forecasting + Explanation: ML models generate forecast for certain metrics; GenAI explains why (e.g., promotion, change in distribution, shipping delyas, tariffs) and drafts a proactive communication to relevant parties.

• Exception Handling: Summarize logistics anomalies across regions and suggest corrective actions. The set of exceptions can be determined programmatically and then explained using Gen AI.

Procurement & Vendor Management

• Contract Intelligence: Extract renewal dates, penalties, or pricing clauses from supplier contracts.

• Supplier Performance Reports: Auto-generate supplier scorecards with both data and narrative context.

Workforce Management

• Scheduling: Optimization algorithms to generate the schedule; GenAI explains trade-offs in plain language (“We prioritized coverage in peak hours, leading to higher overtime for Team A”).

• Onboarding & Training: Generate role-specific guides, FAQs, and even interactive tutors for new hires.

• Employee Feedback: Summarize open-ended survey responses into themes (e.g., workload, shift fairness).

Inventory & Retail Operations

• Stock Anomalies: Gen AI can detect slow-moving inventory through tools and then take subsequent actions eg: email DC with replenishment requests.

• Store Communications: Auto-generate daily store briefs: “Top-selling items, inventory risks, and recommended actions.”

• Customer Feedback Integration: Combine structured sales data with unstructured reviews to suggest assortment changes.

Risk & Compliance

• Policy Q&A: Employees can ask, “Can I override pricing in this case?” and GenAI retrieves the right policy snippet.

• Expense Compliance: Detect expense anomalies through outlier detection algos and generative AI and then summarize and send compliance report to relevant parties.

• Issue Triage: Classify irregular transactions into potential fraud categories and explain rationale.

Beyond the Hype: A Balanced Story

The temptation is to make Generative AI the panacea to all issues facing the business. But in operations, deterministic analytics remains essential for scale, speed, and precision. Generative AI’s true role here is as the bridge between data and decision. This means:

• Turning outputs into understandable narratives.

• Exploring possibilities when the answers aren’t black-and-white.

• Guiding users toward the right next steps.

The place of GenAI in operations is not to replace analytics, but to elevate it—turning data into decisions, and insights into action. As such, the future of operations will be defined not just by more data, but by better conversations with data.

By keeping this balance, organizations can avoid overhyping AI while still unlocking its transformative potential.

Challenges in Demand Planning

Demand planning analytics typically involves several key requirements:

1. Time-Period Comparisons: Analysts need to compare forecasts across cycles or against actual values, identifying significant deviations.

2. High-Volume Data Processing: Planning data often involves millions of rows across multiple dimensions, requiring efficient filtering to identify meaningful variances.

3. Threshold Determination: What constitutes a "significant" variance involves both statistical methods and domain knowledge – for example, determining if a 10% error on a high-volume product is more important than a 50% error on a low-volume one.

4. Root Cause Analysis: Once variances are identified, planners must trace them to their sources using multiple datasets and causal reasoning.

5. Consistent Presentation: Analysis must be formatted consistently for stakeholder consumption, typically following organizational standards.

Limitations of Purely Prompt-Based Generative AI

While generative AI can conceptually handle these tasks, several limitations emerge:

1. Numerical Imprecision: LLMs aren't designed for high-precision calculations. For example, when calculating error concentrations, they might round inconsistently or make arithmetic errors.

2. Consistency Issues: Simply asking an LLM to "Find forecast errors and explain them" produces different outputs on each run, making the analysis unreliable for stakeholders.

3. Black-Box Reasoning: The inability to trace how an AI reached its conclusions undermines confidence in the analysis, especially when stakeholders ask specific questions about methodology.

4. Multi-Step Logic Challenges: Complex calculations, like calculating error concentration metrics, often get muddled when handled solely within prompts.

5. Plausible but Incorrect Explanations: LLMs can generate convincing-sounding explanations that attribute variance to incorrect business drivers, potentially misleading decision-makers.

A Structured Approach: Graph-constrained Agent Workflows with State Management

Rather than relying solely on prompts, a more effective approach uses a structured workflow with clearly defined roles:

1. Workflow Graph with Validation Cycles and Logic Loopbacks

The core innovation is implementing a cyclic workflow graph where:

Data Acquisition → Data Validation → Calculation → Analysis → Presentation
       ↑                                   |
       └───────────(if validation fails)───┘

This pattern enables powerful control loop mechanisms where agents can:

1. Self-Correct through Feedback: Agents can evaluate their own outputs and initiate remedial actions when quality thresholds aren't met

2. Implement Adaptive Sampling: Dynamically adjust data granularity based on detected error patterns

3. Enforce Quality Gates: Establish validation checkpoints that must be passed before proceeding to subsequent analysis stages

4. Create Learning Loops: Track successful analysis patterns to improve performance over time

For example, if a data query returns unexpected results, the system can retry with clarifications:

// Pseudocode example of retry logic
if (nielsen_data is invalid or empty):
  increment retry_count
  if (retry_count < 3):
    add_context("Previous query failed, please ensure it includes all required metrics")
    retry_query()
  else:
    log_error("Unable to retrieve valid Nielsen data after 3 attempts")
    proceed_with_available_data()

2. Global State Management

A well-structured global state management system is crucial for maintaining context and tracking execution. The example system uses an AnalysisState typology with these key components:

# An excerpt of our AnalysisState TypedDict:
class AnalysisState(TypedDict):
    # Tracing and debugging
    analysis_path: List[str]                  # Records each step in the execution
    
    # Analysis artifacts
    raw_data: Dict[str, pd.DataFrame]         # Original query results
    processed_data: Dict[str, pd.DataFrame]   # Calculated metrics
    summary_stats: Dict[str, Dict[str, Dict]] # Statistical summaries
    
    # Error handling
    analysis_attempts: int                    # Retry counter 
    last_analysis_error: Optional[str]        # Last error message
    
    # Data sources
    nielsen_data: Dict[str, pd.DataFrame]     # Market share data by level

This state object allows comprehensive tracking of the analysis process, making the system's reasoning transparent and debuggable. But this approach to state management requires careful consideration of memory tradeoffs:

1. Raw vs. Consolidated Data: For high-volume datasets, we store both raw query results and processed metrics, allowing for reanalysis without re-querying while maintaining access to original data

2. Summary vs. Full Data: The system strategically combines both approaches - storing full data for active analysis levels but using statistical summaries for cross-level comparison

3. In-Memory vs. Persistent Storage: Time-critical data remains in memory for fast access during the analysis workflow, while embeddings or large historical datasets might be offloaded to file system storage when not actively needed

These design decisions ensure the workflow remains responsive even with large datasets while maintaining a complete audit trail of the analysis process.

3. Task Separation

Rather than a monolithic AI system, key tasks are delegated to specialized components:

• SQL Generation: The LLM translates structured prompts into precise SQL queries, executes them, and performs initial validation to ensure completeness and correctness.

• Metric Calculation: All statistical computations are handled deterministically using pandas, ensuring precision and repeatability without relying on the LLM.

• Threshold Application: The LLM applies domain-informed, subjective thresholds to algorithmically flag significant contributors based on the calculated metrics.

• Market Context Integration: Leveraging the existing output, the LLM dynamically determines which supplementary data sources to query (e.g., Nielsen, DAR) and how to enrich the analysis with them.

• Natural Language Insights: The LLM interprets the structured data and generates contextualized, human-readable explanations to communicate root causes and patterns clearly.

For example, when calculating forecast error metrics, the system uses deterministic functions rather than asking the LLM to perform math:

// Pseudocode for error metric calculation
function compute_forecast_error_metrics(data):
  for each metric in metrics:
     data["error"] = data["forecast"] - data["actual"]
     data["abs_error"] = absolute_value(data["error"])
     total_error = sum(data["abs_error"])
     data["contribution_percentage"] = (data["abs_error"] / total_error) * 100
     data["error_direction"] = if error > 0 then "Over" else "Under"
  return data

4. Data-First Approach

The workflow prioritizes data collection and structuring before analysis:

1. Query base forecast vs. actual data

2. Calculate error metrics (MAPE, bias, etc.)

3. Identify significant contributors based on statistical thresholds

4. Pull in contextual data (Nielsen market share, DAR drivers)

5. Only then perform analysis and generate insights

This ensures the AI has all relevant information available when forming conclusions, rather than speculating based on incomplete data.

5. Analysis at the Right Stage

The AI's strengths in pattern recognition and natural language generation are applied after data is properly structured:

// Pseudocode for insight generation
function generate_level_output(level, stats, contributors, dar_data, nielsen_data):
  // Combine all preprocessed data into context
  context = {
     "level": level,
     "summary_statistics": stats,
     "significant_contributors": contributors,
     "market_drivers": dar_data,
     "market_share": nielsen_data
  }
  // Now let the LLM generate insights with all data available
  return ask_llm_for_analysis(context)

Positioning analysis at the end of the workflow creates multiple benefits:

1. Rapid Iteration on Analysis Logic: With all data pre-collected and structured, analysts can quickly modify prompts to generate different insights without re-running the entire data pipeline. For example, changing the analysis focus from "identify over-forecasted products" to "analyze base vs. incremental components" requires only prompt adjustments, not data restructuring.

2. Complex Analysis Without Complex Code: The LLM can generate sophisticated multi-factor analyses by examining relationships across various data sources. For instance, it can correlate Nielsen market share data with forecast errors to identify external market factors - all without explicit programming of these relationships.

3. Contextual Adaptation: Different business scenarios may require different analysis approaches. A structured system can adjust prompts based on detected patterns - for example, using different analysis logic when errors are highly concentrated versus widely distributed.

4. Prompt Engineering Efficiency: Since all relevant data is already in the context, prompt engineers can focus on analysis quality rather than data extraction logic. This separation allows non-technical business experts to refine analysis prompts without changing the underlying data pipeline.

5. Maximum Leverage of Context Windows: Modern LLMs with large context windows can process all relevant data simultaneously, enabling cross-referencing between different metrics and levels that would require complex logic in traditional programming.

A practical example of this approach's flexibility: when a demand planner needs to incorporate a new logic to map pipefill volume in error calculation, they can simply update the prompt template to include this dimension without modifying the data collection or calculation code. The system will leverage the already-collected metrics to generate new insights immediately.

6. Prescriptive Prompting for Structured Reasoning

Another key insight in our system is the implementation of structured causal reasoning frameworks within prompts that guide the AI through domain-specific analytical pathways.

Just as we use carefully designed workflows to process data, we apply equally rigorous structure to the reasoning process itself. This prescriptive approach ensures that AI analysis follows expert human reasoning patterns rather than taking unpredictable paths that may sound plausible but lack grounding in domain expertise.

For example, when analyzing forecast errors, we don't simply ask the AI to "explain the variance." Instead, we provide a step-by-step causal reasoning framework that mirrors how an experienced demand planner would approach the problem:

# Example of structured reasoning instructions in a prompt
prompt = """
Analyze the forecast error patterns in this data and identify likely causes.
​
For significant over-forecasting (where forecast > actual):
* First check market share: Did our share decline more than historical trend?
* Then check category performance: Did the category decline unexpectedly?
* Examine pricing factors: Were you expecting a price decrease that did not materialize?
* Assess marketing impact: Did expected marketing contribution not materialize?
​
For market dynamics impact:
* If over-forecast, negative market dynamics represents a gap between forecast and actual
* Test whether the movement year-over-year is consistent with trend/share patterns
* Check if there are other external factors reflected in the data
​
Base your reasoning on the actual data provided, not general assumptions.
Provide specific, data-backed explanations that reference the metrics shown.
"""

This prescriptive approach guides the AI to follow the same reasoning processes a human analyst would, while staying grounded in the actual data.

The Workflow in Action: A Demand Planning Example

Let's walk through how this structured approach works in practice:

1. Data Collection and Validation

The workflow begins by gathering forecast vs. actual data at different hierarchy levels. The graph's first stage queries the database for each level (brand, category, SKU), validates the results, and stores the raw data in the state:

State["analysis_path"]: 
- "initialize: created analysis queue with hierarchy levels"
- "select: starting analysis for category level"
- "calculate: starting metrics calculation for category"

Each level is processed systematically with validation to ensure data quality.

2. Metric Calculation and Contributor Identification

Once raw data is available, the workflow calculates forecast error metrics deterministically:

State["analysis_path"]:
- "calculate: completed metrics calculation"
- "identify_contributors: starting identification for category"

For significant contributor identification, the system:

• Calculates a concentration ratio to understand error distribution

• Dynamically adjusts thresholds based on this concentration

• Selects records meeting either coverage or contribution criteria

3. Contextual Data Integration

To enable root cause analysis, the workflow conditionally fetches additional data sources:

State["analysis_path"]:
- "decision: fetching DAR data"
- "dar: successfully fetched DAR data"
- "nielsen: starting data fetch for category"

These retry loops ensure robustness against temporary database issues, implementing automatic retries with increasingly detailed context:

// Pseudocode example of retry context
if (nielsen_attempt > 0):
  context = f"Previous query failed with error: {last_nielsen_error}. 
  Ensure you group by the correct category columns."

4. Analysis with Structured Reasoning

Finally, when all data is available, the analysis step applies structured reasoning:

State["analysis_path"]:
- "output: generating formatted analysis for category"

The AI is given precise instructions on how to approach the reasoning:

"First identify the direction of forecast error. If we over-forecasted, examine whether market share declined more than expected. Check the nielsen_data for category-level share trends. Then examine year-over-year patterns in the dar_data to identify if marketing, pricing, or distribution factors contributed."

This structured approach ensures the reasoning follows business logic and remains consistent across analyses.

Conclusion

Successful integration of generative AI in demand planning requires a structured approach that:

1. Separates responsibilities between deterministic calculation and AI interpretation

2. Implements validation cycles to ensure data quality

3. Maintains comprehensive state to track execution and enable debugging

4. Prioritizes data structuring before applying AI for insight generation

By combining the precision of traditional analytics with the interpretive capabilities of generative AI, organizations can produce more insightful, consistent, and reliable demand planning analyses.

The key is finding the right balance – using generative AI where it excels (pattern recognition, natural language, explanation) while relying on deterministic methods for the precise calculations that form the foundation of trustworthy analysis.

In this article, we’ll explore how to tackle this problem using Bayesian hierarchical modeling—a flexible statistical approach that allows us to “borrow” information from similar products launched in the past. This modeling framework can adapt to different levels of product grouping (or hierarchy) and incorporate a transition curve that captures the unique adoption phase of NPIs. By leveraging information from similar products within the same hierarchy, Bayesian hierarchical models allow us to create reliable forecasts for NPIs, even in the absence of historical data. Let’s dive into the concepts, approaches, and practical steps for forecasting new product demand using this approach.

The Challenge of Forecasting New Products

For established products, demand forecasting typically involves analyzing historical sales data, seasonal patterns, and external factors. However, NPIs present a different landscape, as there’s no prior sales data for the new product. This leads to several challenges:

1. Lack of Data: No historical baseline exists for identifying trends or seasonal patterns.

2. Uncertain Demand Patterns: New products often exhibit unique adoption curves, particularly in the early stages, which makes demand highly variable.

3. Difficulty in Systematizing Expert Judgement: In the absence of data, demand planners often rely on expert judgement, which can be subjective and inconsistent.

Bayesian hierarchical modeling offers a robust solution to these challenges. By designing a hierarchical model, we can pool information across similar NPIs, account for various levels of grouping (such as product category or retailer), and model product-specific variations in a systematic way.

Bayesian Hierarchical Modeling for NPIs

Bayesian hierarchical modeling is a statistical technique that allows us to build flexible models with structured relationships across different levels of data. For NPIs, this means we can leverage information from past NPIs within the same product category or retail environment to model demand. By structuring the model hierarchically, we can estimate demand patterns that are common across similar products, while also capturing nuances unique to each new product.

Why Bayesian Methods?

Bayesian methods are particularly well-suited for NPI forecasting due to several advantages:

• Borrowing Patterns from Similar NPIs: A hierarchical structure allows the model to pool information from similar new products launched in the past within the same category or retailer. This establishes a reasonable baseline for NPIs.

• Quantifying Uncertainty: Bayesian models generate full posterior distributions for each parameter, providing credible intervals that allow us to quantify the uncertainty in each forecast—a critical feature when demand is highly unpredictable.

• Flexibility in Hierarchical Structure and Transition Modeling: Bayesian models can handle one-level or multi-level hierarchies and adapt to different grouping structures, such as product categories or retailers. Additionally, they can model the initial transition phase (or adoption curve) that is common in NPIs.

Building a Bayesian Model for NPI Forecasting with Hierarchical Flexibility

In our approach, we’ve developed a single, flexible Bayesian hierarchical model that can be tailored to different demand forecasting needs based on the product hierarchy and complexity. This model is adaptable, allowing for both one-level and multi-level hierarchies, depending on the product’s structure in the organization. It also incorporates a transition curve that captures the initial adoption phase for NPIs, which is essential for modeling demand for new products without historical data.

Let’s explore how the hierarchical structure and the transition curve work within this model.

Hierarchical Flexibility: One-Level and Multi-Level Structures

The model is designed to support different levels of product hierarchy, which allows it to handle a variety of product groupings. Here’s how this flexibility works:

1. Single-Level Hierarchy: In most scenarios, products have only a single dimension —such as product category. Hence, the model can be structured with a single grouping level which makes it simpler to implement. For example, if we’re forecasting demand for a new type of snack bar launched across multiple stores, we might only need to capture variations across stores, without further grouping.

2. Multi-Level Hierarchy: When products are organized within broader categories or regions, the model can be structured to capture multiple levels of grouping. For instance, if a company launches a new line of eco-friendly cleaning products, each product might fall under broader categories (like “kitchen cleaners” or “bathroom cleaners”) or be specific to certain retailers. In this case, the model can incorporate multiple levels of hierarchy to account for both the individual product trends and the category or retailer-level patterns.

Our model’s flexibility means we can configure it based on the appropriate grouping dimensions for each forecasting scenario. By adjusting the hierarchy levels (e.g., adding or removing group levels), the model can capture the unique demand behavior of each NPI while still benefiting from shared information within similar groups.

Role of Transition Curve & Launch Date for Capturing Early-Stage Adoption

New products often exhibit distinct demand phases, particularly in the period immediately following their launch. This early demand phase might involve a rapid increase in sales, a peak, and then a stabilization period. To model this dynamic, we include a transition curve which allows us to accurately capture the adoption phase for new products.

Each product within the hierarchy—down to the most granular level—has its own transition curve that reflects its unique adoption characteristics. The parameters of the transition curve, such as the rate of adoption, the peak demand, and the stabilization period, are modeled hierarchically. This means that products belonging to the same group, like a product category or retailer, share common patterns while still allowing flexibility to represent product-specific variations.

Importantly, the launch date of the new product acts as an additional parameter for inference. It defines when the transition curve begins—essentially the point in time when the business starts putting the product in stores. For new products, the launch date is a critical input because it anchors the transition curve, ensuring the model can accurately capture the timing and shape of the demand build-up. However, for products included in the training data (past products), the launch date is not required; their observed historical data already provides the demand trajectory.

To illustrate:

• A highly anticipated product, like a gaming console, will have a sharp, early peak in demand right after the launch date, as the transition curve kicks in immediately.

• Niche products may see a slower adoption phase, with demand ramping up more gradually over time, depending on the launch date and the dynamics of their respective product category.

By treating the transition curve’s parameters (adoption rate, peak, stabilization) hierarchically, the model pools insights from similar products while retaining the flexibility to reflect individual adoption patterns. The launch date provides a starting point for the transition function, ensuring that the model can project future demand for new products based on when they hit the market.

To further strengthen the model, informative priors are set for the transition curve parameters. These priors act as a guide, particularly when data for a new product is sparse or limited. They ensure that the model converges accurately and avoids unrealistic forecasts, while still allowing room for learning as new data becomes available.

By structuring the transition curve hierarchically, the model is able to capture both the stationary demand characteristics (long-term behavior) and the dynamic adoption phase (transition behavior) for NPIs.

How the Model Works in Practice

Our Bayesian hierarchical model, implemented using CmdStanPy (a Python interface for Stan), follows a structured workflow to provide robust demand forecasts for NPIs:

1. Define the Model Structure: Based on the product’s grouping complexity, we configure the hierarchy as either a one-level or multi-level structure. The choice of grouping dimensions, such as product category or retailer, depends on what best represents the relationships between similar products.

2. Incorporate Priors and Transition Curve: The model uses Bayesian priors, informed by similar NPIs, to guide parameter estimates. It also incorporates the transition curve, with hierarchical parameters to capture group-specific adoption patterns.

3. Fit the Model Using MCMC: CmdStanPy runs MCMC (Markov Chain Monte Carlo) sampling to estimate the posterior distributions of the model parameters, capturing both group-level patterns and product-specific effects.

4. Generate Predictions with Uncertainty: The model produces forecasts with credible intervals, providing a range of likely demand outcomes that account for the uncertainty inherent in NPI forecasting.

This workflow allows us to generate demand forecasts with quantified uncertainty, making the model well-suited for NPIs where limited data is available but reliable, data-driven estimates are essential.

Benefits of a Flexible Bayesian Hierarchical Model for New Product Forecasting

Using a single, flexible Bayesian hierarchical model for NPIs provides several advantages:

1. Reliable Forecasts for New Products: By leveraging patterns from similar products launched in the past and incorporating the transition curve, we create reasonable demand estimates for new products. The transition curve parameters allow us to model the unique adoption phase of NPIs, capturing the dynamic early-stage demand that is typical for new products.

2. Quantification of Uncertainty: Bayesian models provide full posterior distributions, allowing us to estimate credible intervals for each forecast. This is particularly valuable for NPI forecasting, where demand is often unpredictable and quantifying uncertainty is critical.

3. Scalability for Complex Product Lines: The hierarchical structure of the model allows it to scale across different levels of product complexity. Whether forecasting demand for a single level (like individual products in multiple stores) or multiple levels (such as products grouped by category and retailer), the model is adaptable to large, complex product portfolios.

4. Flexibility to Add Exogenous Variables: Bayesian methods can easily incorporate additional features, such as seasonality effects or external factors, which further refine forecasts and make the model adaptable to different contexts.

Conclusion

Forecasting demand for new products without historical data is challenging, but Bayesian hierarchical modeling provides a robust and flexible solution. By designing a single model that can adapt to different hierarchy levels and incorporate a transition curve for the adoption phase, we can create reliable forecasts for NPIs, even in the absence of direct historical data.

Statistical Forecasting Methods

Statistical methods rely on historical data and mathematical models to project future demand. These methods are generally easy to implement and are widely used across industries where the demand patterns are relatively stable and predictable with clear trends and seasonality, these being retail (forecasting short-term store replenishments or regional/store-level demand of staple goods), manufacturing and energy.

Time Series Analysis: examine historical data to identify trends, cycles, and seasonality. Use Case: Forecasting sales for stable, consistent products where past data patterns are predictive of future demand. Some common techniques include:

• Moving Averages: calculates the average of a subset of data points within a sliding window over time. The idea is to reduce short-term fluctuations and highlight longer-term trends

• Exponential Smoothing: weighs recent data more heavily than older data, making it useful for products with stable but slightly fluctuating demand.

• Holt-Winters Exponential Smoothing: form of exponential smoothing used to forecast time series data that exhibits both trend and seasonality. Works well for time series with regular patterns and seasonality such as Seasonal sales forecasting for retail or e-commerce, where demand fluctuates based on holidays or recurring events.

• ARIMA (AutoRegressive Integrated Moving Average): more advanced statistical forecasting method that models the relationship between current and past values in a time series. It combines three components: AutoRegressive (AR), Integrated (I), and Moving Average (MA) to model both the trend and the pattern in the data.

Time Series Analysis with Exogenous Features: adds exogenous variables (additional external features) to the time-series data to improve forecasting accuracy. These models are useful when external factors like promotions, weather, or economic conditions influence the time series.

• ARIMAX (ARIMA with Exogenous Variables): Adds external, explanatory variables (e.g., marketing spend) to the ARIMA model.

• SARIMAX (Seasonal ARIMAX): Similar to ARIMAX, but with seasonal adjustments. It incorporates exogenous data like economic indicators to improve seasonal forecasting.

• VAR (Vector Autoregression): A multivariate model that forecasts multiple interdependent time series simultaneously, considering how they influence each other. Useful for datasets where several variables interact, like different product categories

Machine Learning (ML)-Based Forecasting Methods

ML-based methods are increasingly popular for demand forecasting because they can analyze large datasets and detect complex patterns that statistical models may miss. This is especially useful when demand is influenced by multiple factors, such as promotions, product distribution, pricing, economic conditions, or consumer behavior. However, the quality of any ML model depends heavily on the quality of the input data. In demand forecasting, having clean, structured data is a critical requirement. While deep learning techniques can incorporate unstructured data to improve forecast accuracy, such data typically serves as a supplement rather than the primary driver of model improvement.

As a result, much of the effort goes into identifying and building high-quality data inputs. Additionally, using ensemble models often comes at the cost of explainability. Techniques like SHAP can help attribute the output to the correct drivers, but as ML models become more complex, it becomes increasingly difficult to provide clear explanations for their predictions.

Here’s a breakdown of the different ways to produce ML forecasts:

Bayesian Forecasting Models: These models use Bayesian inference to update the probability of an outcome as more information becomes available. They are ideal for time series with uncertainty or irregular patterns (new product introductions or skus with little sales history)

• Prophet: Developed by Facebook, Prophet is a robust model for time series forecasting with daily, weekly, or yearly seasonality. It handles missing data and outliers well.Use Case: Retail sales forecasting with seasonality, holiday effects, and promotions.

• Orbit: Another Bayesian forecasting model similar to Prophet. It’s designed for scalable Bayesian time series forecasting and is particularly suitable for datasets with irregular or sparse patterns.

• Bayesian Ridge Regression: An ML extension of linear regression that includes Bayesian inference for time series prediction with uncertainty estimates

Gradient Tree Boosting Models: This category includes methods that apply decision trees in an iterative boosting framework. These models are powerful for predicting demand when multiple variables interact (e.g., promotions, prices, seasons). Use Case: Forecasting product sales based on many structured inputs like product type, historical demand, location and pricing.

• Random Forests: An ensemble learning model that builds multiple decision trees and averages their predictions for more accurate results.

• XGBoost (Extreme Gradient Boosting): A highly efficient, gradient-boosting algorithm that builds trees iteratively, improving each step.

• LightGBM: Another fast, efficient gradient boosting algorithm similar to XGBoost, used for time series forecasting with engineered features. It is known for handling large datasets with ease.

Neural Networks: Flexible models that can handle both structured and unstructured data, learning complex patterns from large datasets. These are not widely used for time-series prediction but can be used as part of ensemble modle to augment prediction especially from unstructured data. Use Case: Forecasting sales or demand based on historical trends, customer behaviors and time-dependent data (e.g., weekly sales fluctuations).

• Feedforward Neural Networks: A basic type of neural network that predicts outcomes by learning the relationships between input variables.

• LSTM (Long Short-Term Memory Networks): A type of recurrent neural network (RNN) designed to capture sequential patterns in time-series data.

Collaborative and Judgmental Forecasting

These strategies integrate human expertise, market knowledge, and collaboration across stakeholders and can be used standalone or as augmentation to statistical or ML methodologies discussed above. This is how industries like manufacturing have traditionally set up where they align their production schedules with anticipated demand by bringing together data from various departments (e.g., sales, marketing, and operations) to create a consolidated demand forecast.

Delphi Method: This expert-driven forecasting method is often used in healthcare, pharmaceuticals, and industries with long product development cycles. Experts provide independent demand estimates, which are then aggregated for a consensus forecast.

Collaborative Planning, Forecasting, and Replenishment (CPFR): This method is prevalent in retail and supply chain industries, where companies collaborate with suppliers and retailers to share data and co-create demand forecasts. This helps ensure products are always available where and when they are needed.

Generative AI is currently in an incubation phase, with research progressing at a rapid pace as new models are constantly emerging. However, the industry is still determining the most effective use cases for these models to drive meaningful change. This change can take various forms, such as increased productivity, unlocking new value propositions or market opportunities, streamlining automation, and enhancing decision-making and creativity.

IBP presents a significant opportunity for generative AI, particularly due to the vast amounts of data spread across various functions and the need to simulate scenarios and quickly find relevant answers. Based on experience with numerous generative AI cases, the following areas offer the greatest potential for uplift:

Providing Data Analysis in Natural Language

As generative AI models become more reliable and less prone to hallucinations, coupled with tighter control over data organization and access, a mechanism can be provided for business users to ask first-level analysis questions and receive answers quickly. Previously, this would require specialized SQL knowledge or reliance on data domain experts. By eliminating that dependency, faster responses and decision-making are enabled. However, it is crucial to implement sufficient guardrails to ensure the accuracy of the analysis. This can be achieved through several mechanisms:

• Creating an Effective and Explainable Entity-Relationship Model for Storing Data: The fewer decision blocks a model goes through, the lower the risk of errors. Structuring tables coherently for easier analysis leads to better results. Generative AI should prioritize understanding the context of questions and retrieving relevant data from tables, rather than handling complex calculations.. As a guiding principle, if retrieving the response requires a Common Table Expression (CTE), it’s often better to create the response as a view.

• Constraining the Agent: It is essential to impose restrictions on how and what data an AI agent can access. Using a SQL agent that fetches data from tables with built-in row and column access controls is more prudent than employing a Python agent that reads flat files. Granular data controls help secure information, reducing the risk of breaches or misuse of agents.

• Validation Checks on the Response: Quality controls should be applied to the response itself. These can either be rules-based or encoded as contextual instructions for the AI agent, ensuring it validates the response before providing an answer.

• User Training: It is important not to rely solely on the technology. Upskilling users and providing them with foundational knowledge of generative AI, including its limitations, can help them better navigate the augmented processes.

Creating Human-Readable Reports from Generated Analysis

Foundational models excel at natural language processing and generation, making them capable of transforming data-heavy reports into easily readable formats with the right context and guidance. This is valuable for generating insights and providing context to users who might otherwise require domain knowledge to interpret the information.

Classification and Categorization of Semi-Structured and Unstructured Data

This is a highly impactful use case for generative AI, significantly boosting productivity where traditional rules for categorization are loosely defined. Conventional rules-based or machine learning classifiers rely on structured training datasets, making them less effective with data that has fluid boundaries or is constantly evolving. Generative AI offers superior flexibility in interpreting such changes and is also easier to set up and adapt. In most cases, especially with non-specialized datasets, there’s no need for frequent model retraining.

Automation of Processes and Services

Automation is another key area where generative AI excels, and it has been a central focus for many business applications to date. By combining traditional coding techniques with AI agents, frameworks can be created to automate business processes that are often managed by SaaS models. Some examples include:

• On-Demand Infrastructure Provisioning: AI agents can be designed to handle user requests for provisioning VMs, cloud services, or user accounts. These agents have access to service accounts and the necessary permissions to interpret user requests, verify eligibility, and complete the action.

• User Onboarding: Early adoption of AI has been seen in user onboarding through chatbots that assist with process queries. There are also innovations, like Google’s tool that converts scientific papers into podcast formats. Similar audio-visual formats can be used to guide users through processes.

• Auto-Scheduling of Product Lines: Generative AI can be embedded in workflows to eliminate manual processes and reduce friction. For example, if a production line fails, a user can request rescheduling. The AI agent can initiate an approval process, and upon approval, invoke an optimizer to rebalance production. Here, the agent acts as a mediator between services, enhancing automation.

• Route Management in Logistics: Generative AI can augment GIS data to optimize warehouse locations and reduce shipping costs. By analyzing historical data, demand forecasts, and transportation routes alongside GIS mapping, AI can provide real-time analysis of logistics bottlenecks and their solutions.

However, it is worth pointing out that Gen AI is still in its infancy when it comes to traditional ML domains such as time-series forecasting, optimization and scenario planning and as such should only be used to augment existing models until such a time when foundational models become better at it.

Provided below, are a few examples where we have successfully implemented and industrialized Generative AI-driven processes.

1. Forecast Narrative

   In any planning process, demand forecasting is key and typically spans a defined time period, often updated monthly. While there are various methods for creating forecasts, many businesses are increasingly adopting machine learning-driven forecasts due to their high accuracy and bias reduction. However, these sophisticated models often lack explainability due to their black-box nature, making it challenging for demand planners to justify forecast changes. To alleviate this burden, we developed a Generative AI model that compares two forecasts and constructs a narrative explaining the differences. By structuring the model’s inputs and outputs intelligently and maintaining tight control over data, the AI could access and analyze changes between forecasts, referencing the underlying drivers to provide a plausible explanation. It then generated a complete narrative that planners could use. The key point here is that the Gen AI acts as a support tool, removing roadblocks in the process, although the planner still verifies the findings. This approach reduced the time needed to construct a narrative from a week to just two days.

2. Analyst Bot

   We developed a helper bot designed to assist business users with data queries. This bot uses a SQL agent to translate natural language queries into SQL, execute them on the server, and then present the results in a standardized format to the user. As discussed earlier, the agent first checks the user’s permissions before fetching any data, and it also runs validation checks on the responses. This greatly democratized data analysis, enabling business users to be more productive in their daily work and significantly reducing the volume of ad hoc data requests to technical teams.

3. Indirect Cost Classifier

   Indirect costs, or overhead costs, are expenses that cannot be directly linked to a specific product or service but are essential for the overall operation of a business. These costs span various activities and departments, making them difficult to track. We had an internal taxonomy for categorizing these costs, but previously, an associate had to manually review receipts collected by the client—a highly time-consuming process. By combining OCR with Generative AI, we created a system to automatically read data from receipts, identify the type of expense, and categorize it based on our taxonomy. This automation reduced the effort from days to mere minutes, streamlining the entire process. Knowing where these costs lie empowers businesses to negotiate or eliminate unnecessary expenses, ultimately enabling more effective cost management and improved control over spending.

Harnessing the accrued and ever-increasing data continues to be the biggest challenges for companies embarking on IBP transformations and it is also the most important aspect to master for successful execution. Here we delve into the factors that make mastering data so difficult and provide examples and guidance on how to overcome them.

Characteristics

Following are the major characteristics of the data powering IBP.

Accurate: IBP data is used to generating forecasts and aid in decision making and as such it needs to be precise and reliable. There is no ceiling to the damage inaccuracy in data can present.

Up to date: For the same reason as above, data needs to be real-time or near it, reflecting the current state of the business and its operations. This makes the decision-making process more agile and can also reduce communication touchpoints.

Complete: IBP data is essential to central and holistic decision making. Therefore, it needs encompass all relevant aspects of the business. These include but are not limited to financial, operational, supply-chain, procurement, and market-related information.

Granular: Operational data specific to supply chain has strict hierarchies especially as it relates to products or customers. Hence, the relevancy of data for the user can vary in granularity, from high-level summaries to detailed transactional data.

The data transformation must holistically encompass the above-mentioned aspects to truly gain insight from the information and provide tangible benefits in the decision-making process. Provided below are five necessary things an organization must do for the said data transformation.

Preparation

Having relevant data for the right users and processes is crucial precursor to enabling any kind of digital transformation. Most legacy organizations will have multiple systems and data sources. They will also have processes that have been built upon those systems and would often have dependencies on each other for normalizing the data. This often leads to situations where we define a different number based on who is looking. As s:uch, sufficient effort has to be focused on

• Disentangling these processes,

• Formalizing the source contracts, collection and refresh cadence of data

• Building out the relevant infrastructure and data pipelines

• Building out the people system and processes to manage the pipelines.

• Continuous monitoring and enhancements for robustness of the pipelines. Data will continue to grow and as such it is important that the technology evolves alongside it.

This is perhaps the most time-intensive undertaking among all mentioned. So, it is important to do it in a staggered manner that allows for quicker results while also maintaining cohesion between the different services.

For a large retail client, we upgraded their ERP system, built an enterprise data platform, and developed downstream data layers that provided a more unified view of the organization's data. This enhanced visibility not only improved operational transparency but also led to a significant increase in data-driven use cases. As a result, the client was able to make more informed decisions and optimize various business processes, driving greater efficiency across the organization.

Scoping

The most crucial aspect for reaching a consensus on numbers is to first define clearly the right sources, transformations and units of the data being utilized for a given process. This is an oft-neglected part of a build that quickly comes to the fore when problems in measurement materialize. It is important to formally acknowledge the data scopes through written records and advertise them broadly so that everyone sings from the same hymn sheet.

Following are a few of the potential hazards in data that one may encounter:

• Shipments are tracked as actualized (remove returns or unfulfilled orders) or invoiced.

• There may be multiple product id standards.

• Shipments may have multiple units of measure.

• Same data resides in two different sources. Orders are placed in SAP, but actualized shipments are tracked in Kinaxis.

• Data categorization happens in multiple systems. Customer orders are recorded in SAP, but the product hierarchy is created in Kinaxis.

• Forecasting is done for a subset of customers (say, retail)

• Data Is not refreshed at the cadence that forecasting requires.

• SMEs have incorrect definitions of data attributes and their usage.

Data Model & Platform Design

Once the scope is finalized, we can commence work on IBP specific data setup. This means taking the enterprise data and deliberately designing the pipelines and storage architecture that would allow for fit-for-purpose actions on the data by downstream IBM tasks and applications.

The data platform is a layer of fabric connecting multiple sources rather than a single component. Since the underlying storage solutions and their number change with advent of technology, you want an abstraction that can protect the application code from the changes. This can be a custom interface or a utility existing for a given language. For eg: python has fsspec, an interface for interacting with different storage solutions.

Data model refers to structure and relationships between the different data attributes that drive IBP. As described above, this data comes from disparate operational and functional sources. So, the relationships between them has to be formed based on what the usage is and where the data is shown.

For a large B2B client, we created an internal planning application that required separating data structures based on two distinct patterns of viewing the data – high granularity charts and graphs that required high level of detail in the data and low granularity ones that showed trends over longer time horizon.

Governance

Data Governance refers to framework or practices that organizations implement to ensure that their data is managed effectively, securely, and in compliance with relevant regulations and policies. When it comes to IBP applications it encompasses the following:

Role Base Access Control (RBAC)

All interactions on the applications are attributed to clearly defined user personas and each persona can be assigned to one or more users. These personas govern the rights attached to a given UI component namely read, edit, delete etc. In this manner we place access controls in place using the persona/role. Usually RBAC is integrated with an organizations IT-maintained directory services – most common of them being Microsoft Active Directory

ABAC

While the personas govern how one interacts with the application, data controls affect what information one sees. A user is assigned a specific set of attributes which act as filters and are passed down from the UI. These are then applied at the source level (database, cache, static files) and the information is then returned back to the application layer. Different applications might implement this differently, but this is usually implemented at the application level itself for off-the-shelf solutions.

Data Lineage

Data lineage refers to the ability to track and comprehend the journey of data starting from its origin and ending at its destination. It involves capturing and documenting information about where the data originated, how it underwent transformations and the various steps it went through as it moved across systems, applications and processes.

Organizations benefit from data lineage as it provides them with an understanding of how data is created, manipulated, and utilized within their systems. It helps address questions such as:

• Where did the data originate from?

• Who accessed or modified it?

• Where was it stored?

• How was it used or analyzed?

For one client, we built a custom, automated data tracer from the ground up to track data lineage across various sources, both on-premises and in the cloud. This tracer captured and stored metadata related to the flow of data, tailored to the specific storage types (such as Parquet files or relational databases). This solution allowed us to document the entire journey of the data, making it possible to visualize the flow across systems in different formats and at varying levels of detail. This visibility was critical in ensuring data accuracy, transparency, and consistency throughout the organization's IBP processes.

In conclusion, harnessing data for Integrated Business Planning (IBP) transformation is no small feat. From ensuring data accuracy and granularity to managing multiple sources and aligning stakeholders, the path to success requires thoughtful planning and execution. By addressing these challenges—through processes and technology—organizations can unlock the full potential of their data.

In today’s fast-paced business environment, a cohesive and unifying data strategy is crucial for the success of Integrated Business Planning (IBP). By integrating multiple operations, IBP offers end-to-end supply chain visibility, seamless cross-functional connectivity, and accelerates key processes such as *Sales & Operations Planning* (S&OP), *Sales & Operations Execution* (S&OE), and *New Product Development* (NPD) cycles, addressing key challenges faced by legacy organizations.

Traditional companies often operate in silos, where each department maintains its own data, benchmarks, and units of measure. This disjointed approach can lead to inefficiencies and missed opportunities, as data from one operation can significantly impact others. For example, promotional activities can dramatically influence demand forecasts, underscoring the necessity of incorporating up-to-date marketing data into demand planning.

Furthermore, as businesses grow and expand into new markets, their operations become increasingly complex, making it difficult to manage without a comprehensive, integrated planning process. This complexity is further compounded by the ever-present risk of supply chain disruptions and the growing demands for sustainability and regulatory compliance. These factors underscore the critical need for a unified, forward-thinking IBP strategy.

To achieve seamless connectivity between different systems, it is essential to conduct a thorough analysis of existing processes and technologies. While this may sound straightforward, implementation can be complex. In this article, we will draw on a previous case study to illustrate the complexities of integrating multiple planning and source data systems, while also highlighting the best practices that have emerged from our work.

A robust IBP system unifies metrics or establishes clear conversion methods between different units of measure, harmonizing data across functions for improved benchmarking and cross-functional analysis. For instance, the sales team may track product movement in cases or pallets, while the manufacturing team monitors production in individual units. Meanwhile, the finance team may focus on the monetary value of goods sold or produced. A standardized conversion method will allow each department to retain its preferred unit of measure but translate all data into a common framework. This harmonization reduces errors, increases visibility, and improves decision-making, as all teams can access consistent, comparable data for more accurate forecasting, budgeting, and operational planning.

IBP also aligns operating cycles for greater efficiency and synchronizes data refresh cadences, ensuring maximum visibility. For instance, supply planners should have access to near real-time updates on shifting demand forecasts, rather than depending on periodic or delayed data. This real-time visibility allows them to respond more quickly to changes, reduce lead times, and optimize inventory levels by adjusting supply plans proactively, ensuring that stock is aligned with actual demand.

Central to this transformation is the creation of a global data repository with functional demarcation, yet accessible across departments as needed. This de-siloed, foundational approach opens the door to new insights and use cases that were previously undiscovered. It also accelerates the development and deployment of such use cases, thus enabling to significantly lower the cost of adoption of new technologies such as Generative AI. The central repository can function as a loosely federated system tailored to specific purposes but combined to provide a holistic view of business operations. Key components include:

• Staging Layer: Stores structured and unstructured data (third-party data dumps, incoming analytic feeds, historical datasets).

• Big Data Layer: Houses semantic models for all incoming signals (shipment, consumption, inventory).

• Relational Layer: Facilitates predictive and optimization modeling (demand, supply, inventory, financial) and captures snapshots of key drivers (promotion, distribution, pricing, third-party coefficients).

• In-Memory Layer: Supports real-time reporting and complex datasets (hierarchical current forecast including adjustments, inventory levels).

The industry is now moving towards a consolidated ***lake house*** approach, which combines all the layers into a unified system. All data is funneled into a single domain following the medallion architecture: *bronze* for raw data, *silver* for cleansed and enriched data, and *gold* for high-quality, ready-for-use data. This structure improves scalability, ensures real-time insights, and simplifies data management for enhanced decision-making.

No matter the underlying architecture, the central repository acts as a conduit for data flow between all operations, creating a paradigm where data from various functions integrates seamlessly. To ensure the success of this architecture, it is crucial to establish a clear data governance structure, enforce robust security measures, provide role-based access, and continuously monitor data pipelines.

In conclusion, an overarching focus on data strategy is the cornerstone of effective IBP, driving operational efficiency and business growth. By integrating operations, harmonizing data, and ensuring real-time visibility into it, companies can make informed decisions faster and more effectively. Embracing a unified data strategy within IBP not only aligns with current business demands but also positions organizations for a future of sustained success.

Subscribe now