Model Monitoring, the last phase of MLOps cycle.
Machine learning models are functions that find patterns using Historical Data. These models first learn the relationship between independent and dependent variables and use that knowledge for predicting future events assuming the relationship will still be there. But we all know as a Data Scientist that ML models degrade over time in terms of performance. So, we cannot just build a good model and say that it will be reliable till eternity.
Therefore, it is important to keep an eye on your Model and whenever you see downfall trend in the performance, it is the signal that you should retrain the model to save business loss.
But the problem with relying on the performance of the model is that it needs ground truth. And waiting time for the ground truth can be huge, which results in business loss. Let’s understand this using an example –
Suppose you are building a ML model for loan approval for a Bank and it is performing good in the training time. Finally, the model has been deployed and now you are observing it for performance degradation. To calculate the performance of the model, you will be needing ground truth and ground truths come after a long period of time as until an applicant misses an EMI, he/she will not be in defaulter list. Let us say after waiting for a year, Bank is labeling an applicant defaulter/ non-defaulter. Now, if you wait for a year for the ground truths and there is degradation in the model, that means for all the loan applicants who applied in last 1 year, there is a huge risk for misclassification which results in serious business loss. So that concludes, we cannot be just dependent on the ground truth when they are delayed.
Then the question comes — How to monitor ML model without availability of Ground Truths?
What is Data Drift and how is it degrading your model’s performance in production?
Data Drift
As we previously discussed, ML models are created using Historical Data. We feed data to our model to build relationship between independent features with the outcome (label).
Let F(X) be a ML model. The only variable in F(X) is X which can be changed, where X denotes covariates. When distribution of X changes, it affects model’s performance. In model monitoring, these changes are called drift.
Data Drift can be of many types but in this section, we will focus only a type of Data Drift which is Covariate Drift.
Covariate Drift
When there is a change in the distribution of covariates (also known as input features), we call it “Covariate Drift”. In other words, when f(X) changes but f(Y|X) remains the same where f(X) denotes marginal distribution of covariates and f(Y|X) denotes conditional distribution of Y given X remains the same.
Let us understand Covariate Drift more clearly using Distribution plot –
As you can see, in 2019 the data distribution for feature 1 was bell shaped (Normal Distribution), but over time it got skewed on the left side.
One point to note here is that Covariate Drift does not imply performance degradation. It may or may not affect the model’s performance based on the severity of the drift.
Accordingly, Covariate Drift can be classified into 2 types –
1. Benign Covariate Drift
2. Malignant Covariate Drift
Benign Covariate Drift
When Covariate Drift exists but it is not affecting the model’s performance.
Consider the above example, distribution of data changes its position but not crossing the decision boundary. As a result, performance has not been hampered.
Malignant Covariate Drift
When Covariate Drift exists and it is affecting the model’s performance.
In the above example, distribution of data changes its position as well as crossing the decision boundary. As a result, performance has been hampered.
Covariate Drift Detection
To be continued…
References –
· https://towardsdatascience.com/dont-let-your-model-s-quality-drift-away-53d2f7899c09
· https://www.picsellia.com/post/what-is-data-drift-and-how-to-detect-it-with-mlops
