An important component to a successful machine learning model project is to have a solid automated testing framework. This is usually set up in the CI/CD pipeline and triggered every code push. As the code base gets larger, more complex and with more contributors adding to the code base, it will become a crucial part of your code development process. Although it may take some time initially setting it up, it will allow you to spend less time fixing code bugs in the long run.
The key word here is ‘code testing’ which is aimed to test the code functionality. It’s not model performance testing - this should be separate.
Three basic tests to have in a mature machine learning model project are unit tests, integration tests and post-deployment tests.
1. Unit tests
Unit tests are designed to test individual components of the code base. It will test that the function accepts the same input and output and will run without breaking. Unit tests allows you to easily identify which component of the code will break if some code changes. If the change was intentional, it’s just a matter of updating the unit test.
2. Integration tests
If unit tests are testing individual components of the code base, integration tests will test the components together. The idea is to set up logical components of the code to test together. For example, there may be data-preparation, model-training and model-prediction tests.
Both unit tests and integration tests should be fast and done every code push.
3. Post-deployment tests
Post-deployment tests will test the model end-to-end and usually using the same resources as you would use in production. This may not be as fast as unit tests and integration tests and so may become a bit too much to do every code push. You can choose to run this test before a code pull request instead.
Any challenges?
If you don’t have the correct environment setup, you may find that the code runs differently in different environments. This is often simply due to different code package versions that may be installed. So, an easy solution to this is to use a contained environment - and the best tool to help with that is Docker! Wrap everything in docker containers, whether you’re running the code on your local computer or on a CI/CD pipeline or on a cloud resource. This is the best way to ensure that the environment is the same.
TL;DR
For machine learning models, especially those with multiple people working on it, it’s important to have a good automated code testing framework in place. There are 3 basic tests to have: unit tests, integration tests & post-deployment tests. These three tests serve different purposes and will help ensure a smooth code development process. Spend less time on code debugging and more time on actual model development.