MLflow Experiment Tracking
Description
MLflow Tracking is the part of the open-source MLflow platform that records each run of a machine learning training script. Developers add logging calls to their code, or switch on automatic logging for supported libraries. Each run's record holds its parameters (the settings chosen, including hyperparameters, which are fixed before training), its metrics (results such as accuracy), the code version, its start and end times, and output files called artefacts, such as the trained model. Runs are grouped into experiments that can be searched and compared in a web interface, and they stay on the developer's machine unless a shared tracking server is set up. The result is a dated record of how each model was built, which a reviewer can use to see what was tried and which run produced the deployed model, but only for what was actually logged.
Example Use Cases
Transparency
A hospital group develops a diagnostic imaging model across three sites. Ahead of an external assessment, the team exports the logged runs, with the parameters, evaluation metrics, code version and time of each run, so the assessor can see every configuration tried and trace the submitted model to the run that produced it.
An energy company's data science team keeps every hyperparameter search for its demand-forecasting models on a shared tracking server. When an internal auditor asks why one configuration went live, the team shows the full set of runs searched, their metrics and when each was logged, and the run chosen for deployment.
General
A bank's fraud-detection team sees alert precision fall after a retraining. They compare the logged runs for the old and new model, covering parameters, training-dataset reference, code version and metrics, find that a feature setting changed, and note the cause on the run before the next release.
Limitations
- The record holds only what developers log, plus the default set that automatic logging picks up for supported libraries. A preprocessing choice made in a notebook and never logged is absent, and the log gives no sign that anything is missing, so the audit trail is only as complete as the team's logging habits.
- Dataset tracking stores metadata about a dataset, such as its name, a hash of its contents and where it came from, not the data itself. A logged run can show which dataset was used but cannot restore it, so reproducing a result needs the data to be versioned or retained separately.
- A tracking server is a working store, not a sealed ledger. Runs can be marked as deleted and tags overwritten by anyone with write access, so the log supports an audit only if access is restricted and the store is backed up.
- Complex multi-stage pipelines may require custom instrumentation to capture dependencies and data flow relationships effectively. Where that is not done, the record of one stage will not show what fed it.
- Artefact storage grows with every logged model and file, so heavy experimentation with large models raises storage costs. Deleting old runs to save space removes part of the record that an audit would rely on.
Resources
Research Papers
Accelerating the Machine Learning Lifecycle with MLflow
Machine learning development creates multiple new challenges that are not present in a traditional software development lifecycle. These include keeping track of the myriad inputs to an ML application (e.g., data versions, code and tuning parameters), reproducing results, and production deployment. In this paper, we summarize these challenges from our experience with Databricks customers, and describe MLflow, an open source platform we recently launched to streamline the machine learning lifecycle. MLflow covers three key challenges: experimentation, reproducibility, and model deployment, using generic APIs that work with any ML library, algorithm and programming language. The project has a rapidly growing open source community, with over 50 contributors since its launch in June 2018.