Coles - SSP Forecast Ingestion
Coles leverage a data analytics process providing critical forecasting information to the business on a weekly basis.
Key figures
Their challenge
Coles leverage a data analytics process providing critical forecasting information to the business on a weekly basis.
Sales forecasts are utilised by stores around the country to understand their rostering requirements. In order to leverage the new sales forecast, the data had to be imported and the system was hitting performance constraints. This meant that the team at Coles were not able to get this in time which was causing operational challenges.
Coles engaged Ippon Australia to conduct a proof-of-concept around data batching and how this could be potentially improved without altering the downstream solution. The objectives were simple:
- Produce a small scale PoC application
- Demonstrate improved request response times
- Identify potential solutions for a potential future solution
Ippon Australia engaged with the project team to quickly understand the basic data requirements from a processing standpoint, creating a small transform script and then implemented an application which could multithread complex data driven requests in a manner expected of the downstream system. The results that came back showed there was a potential decrease of 50% to request response times indicating that greater throughput was possible.
Our solution
After the successful PoC, Coles and Ippon moved forward together to implement a solution that would meet both the needs of Coles now and potential for future.
Given the complexity of the data involved, a large amount of time went into understanding how it was structured and the overall data outcome expected.
At a high level it involved taking multiple data structures, combining and manipulating data and finally restructuring into a vastly different format. Once the data requirements were understood the team went about designing and implementing a solution:
- Using microservices to be triggered when required via APIs.
- One of the outcomes of the PoC had shown that the transform process embedded in the current solution caused a lag in the overall batch times.
- As part of the solution one service would be dedicated to transforming the data ahead of time to reduce the effort required on the overall batching process.
- Another service would exclusively batch data into the end solution as required.
- Deployed into Kubernetes to take advantage of the existing setups within Coles cloud.
Integrate with the existing downstream solution to provide the data required and then enhanced with modern cloud services to provide durability.
To provide Coles with a future path to deal with potential growth the solution was implemented to be horizontally scaled as required (and designed to be easily updated to be an event based solution). Logging and alerting were added to the solution to provide a dynamic solution for support teams (whilst also providing more traceability).
The outcome
Coles consuming greater amounts of forecasting.
The outcome after testing with production-like loads showed a clear differential:
- Existing solution batch average: 4 hours
- New solution batch average: 2 hours.
This was a great outcome, enabling Coles to load double the volume of data in the same amount of time. This provided all stores across the country with a sales projection spanning more weeks than previously available. In turn, it enabled them to plan better and operate more efficiently, saving costs and providing more accurate in-store staffing requirements. From an IT service perspective, the increased speed in the loading of data reduced the risk of service availability, allowing them to respond more quickly if required.