Harnessing Big Data and Deep Learning for Real-Time Demand Forecasting in Retail: A Scalable AI-Driven Approach

Abstract

Retailers worldwide are expanding their investments in advanced AI-driven solutions to improve demand forecasting in retail operations. Empowered by recent progress in big data technologies and deep learning algorithms, there is potential to develop a scalable and real-time intelligent system to forecast demands at SKU level. This system can help improve prediction accuracy and operational efficiency in the retail sector. Deep learning and big data models are proposed to capture complex nonlinear temporal patterns in a large number of time series with multi-scale and multi-mobility characteristics. These include (1) SoxNeT that integrates big mobility data into extreme learning machines with a deep neural network based on LSTM cells to effectively extract temporal features and spatial correlations, (2) DemandNet that applies convolutional neural networks and RNNs to model groundsill demand and investigate urban land-use context effects, and (3) EKO, a deep learning ensemble framework that combines many big data boosting and popular models with categorical embedding layers, to capture complex and dynamic spatial-temporal patterns in high-dimensional traffic data.

Several challenges need to be addressed: (1) most existing methods do not explicitly capture the seasonal effect of nonlinear combinations of many factors; (2) deep learning models are data-intensive and their training often requires cross-validation to tune hyperparameters, making them difficult to scale to many SKUs; and (3) most proposed approaches are not optimized for computational efficiency or deployment-ready. Therefore, it is important to develop a scalable deep learning algorithm that accommodates big retail data and runs efficiently, hence improving the performance of demand forecasting models. An innovative AI-driven approach is proposed that effectively forecasts demand at SKU level, benefited by the development of three scalable deep learning algorithms, including a decomposable local-global network design and an ensemble of traditional time series and CNN-based predictors. The prediction process allows for endogeneity effects and the marketing optimization problem is formulated as a sliding-horizon control strategy. Various AI algorithms are tested on a revenue management problem with purchase pre order decisions. An alternative AI framework is provided which automates the development of decision-time, dynamic pricing policies based on a class of MDP models.