Effective testing for machine learning systems.for machine learning systems.ne Learning ### Production Machine Learning Monitoring [Great article](https://towardsdatascience.com/production-machine-learning-monitoring-outliers-drift-explainers-statistical-performance-d9b1d02ac158) that goes along with referenced talk. ### Testing of ML systems A few great articles and papers: * [Effective testing for machine learning systems.](https://www.jeremyjordan.me/testing-ml/) * [proceedings.mlr.press/v97/odena19a/odena19a.pdf](http://proceedings.mlr.press/v97/odena19a/odena19a.pdf) * [MLOps Chat: How Should We Test ML Models? with Data Scientist Jeremy Jordan - YouTube](https://www.youtube.com/watch?v=k0naEYedv5I&list=PL7WG7YrwYcnDBDuCkFbcyjnZQrdskFsBz&index=10&t=566s) The big takeaways can be defined as: * **Pretrained tests** * Check data shape * Check for label leakage * Identify failure modes * Check output ranges and ensure it aligns with expectations * **Post trained tests** 1. **Invariance Tests**: What perturbations should we be able to make to inputs that do not affect the model's outputs? 2. **Directional Expectation Tests**: What perturbations of the input should have a predictable effect on the model output. E.g. if we increase the square footage of a house while holding everything else constant we should *not* observe a drop in price. 3. **Minimum Functionality Tests**: 4. Investigate failure modes! Look at where the model is most egregiously failing * Ideal framework for testing: * Should be able to easily define tests via data examples * Once tests are defined, want to easily be able to get a model "report card" * **Organizing Tests** * Should organize tests around *skills* * --- Date: 20211228 Links to: Tags: References: * [# Production Machine Learning Monitoring (youtube talk)](https://www.youtube.com/watch?v=QcevzK9ZuDg) * [Effective testing for machine learning systems.](https://www.jeremyjordan.me/testing-ml/) * [How to Test Machine Learning Code and Systems](https://eugeneyan.com/writing/testing-ml/)