With the evolution of technology, businesses are increasingly adopting machine learning and data analytics to derive solutions. According to Forbes, 50% of all enterprises planned to invest more in ML and AI in 2021.
With this shift towards machine learning, data scientists often have to deal with challenges when models are in production. Last year, 38% of organizations reported that their data scientists spent 50% of their time in model deployment.
One of the challenges that occur is drift in model predictions. Model drift typically refers to shifts in predictions made by the model; hence, what the model forecasts today differs from what it previously predicted. As a result, model predictions can become less reliable in the long run. This occurrence is particularly evident in time series forecast models, where some of the factors that can affect the accuracy of predictions are:
- Shifts in consumer behavior
- Changing monetary policies
- Economic cycle changes
- Competitor Activity
So, how can we respond to drifts in forecast models? We can use either a reactive or proactive approach to ensure model reliability. However, before delving deeper into the two different strategies, we have to familiarize ourselves with a couple of terms:
Model Reliability:
Within the machine learning community, the concept of reliability covers closely related ideas of uncertainty in model predictions.
Moreover, it also encompasses failures to generalize production data. Ultimately, the reliability of model predictions at a specific point in time refers to model reliability. You can trust a reliable model’s forecasts up to a certain period before you might want to replace it.
Model Availability:
Here, model availability means the availability of a model that satisfies production requirements for model reliability. Now that we’ve learned about the abovementioned terminology, we are going to go over the two different strategies:
The Reactive Approach
One solution to drifts in model predictions is to adopt a reactive approach and establish thresholds based on forecast-related business rules.
This way, a drift event can be detected if the difference between the actual time series and forecasts crosses a particular threshold point. Consequently, that would trigger retraining existing models or further analysis in an automated setting.
Employing a reactive approach to drift events is reasonable if:
- Such events occur rarely
- Retraining expenses are low due to the low number of models in production
- Model availability does not have strict requirements
However, this approach might not be practicable in some industries because the number of models in production is high. Here, a proactive approach is more convenient.
The Proactive Approach
In industries like retail or logistics, the number of time series that need to be modeled can easily cross thousands. Additionally, these time series may be closely related and belong to the same domain, material, or geography.
In this case, responding to drift events at such a vast scale is unfeasible. That’s because responding with further analysis or model retraining can take up a lot of analysis and computational time.
We need to ensure that the model is reliable in production, has decent uptime, and does not drift away and trigger retraining. For this, we need a strategy that can foresee model drift in advance. Here, a proactive approach might be suitable. The objectives of this are to:
- Be able to meet SLA requirements related to model availability.
- Analyze these cases for root cause analysis in a pre-production setting.
- Calculate computational requirements for retraining in advance (If the frequency of these events across models is known).
A Proactive approach can provide a reasonable estimate of these events even when the model is in a pre-production state.
How to analyze forecast model reliability with survival analysis?
An excellent approach to estimating reliability in time series forecast models is through survival analysis. You can create a survival model to predict model delay and perform a proactive analysis if:
- Significant history is available on model performance
- Numerous closely related models are available
Proactive analysis helps you quantify both model availability under different scenarios and computational requirements for retraining. Moreover, it will help you pre-plan. Before we demonstrate how to use survival analysis to predict the kind of decay expected from a given set of time series, we’ll have a brief overview of how the survival model works.
What do you need to know about Survival Analysis?
Models that predict the time to an event are called survival models. Survival models fall under a branch of statistics that deals with the analysis of time to an event, called survival analysis.
For example, the time before a loan default or time before an equipment failure are examples of survival models. (Loan default and equipment failure are the “events” in these scenarios.) You can model the survival probability distribution as follows:
Where,
T: waiting time until the occurrence of an event
S(t): Probability that waiting time T is > than some time t. Meaning, probability of survival till time t.
From here, we can go to the concept of censorship. If the time the event occurs is unknown, it’s said to be censored. If the observation period of an experiment expires and the subject does not face an event (will face one in the future), it’s called right censored. Right censorship is the most relevant to our forecasting model reliability study.
Survival modeling techniques are classified broadly into two techniques:
Parametric Models:
These models make assumptions about the form of the survival probability distribution. They can be good choices if their assumptions are backed with domain knowledge.
.jpg)
