Malicious Model-File Scanning

Description

Malicious model-file scanning checks a downloaded or received model file for code that would run when the file is loaded, without loading it. Several common ways of saving (serialising) models can carry such code, above all Python's pickle format, the default for PyTorch checkpoints and the usual format for scikit-learn models, whose files can instruct the loader to import and call any Python function, so that loading an untrusted file can hand an attacker control of the machine; since version 2.6 PyTorch's own loader by default allows only a small set of known-safe functions, but some libraries switch that off to load their models. The named method is ModelScan, an open-source scanner from Protect AI, which chooses a check by file extension and reads the file as data: for pickle-based files it lists the functions the file would import and compares them with a graded list of unsafe ones, such as the os, subprocess and socket modules and Python's built-in eval and exec, and for Keras and TensorFlow files it flags Lambda layers, which hold arbitrary Python, and operations that read or write files. It reports each finding with a severity from critical to low and exits with a non-zero code when it finds any, so a pipeline can stop the file before anything loads it. The evidence is a record that no listed unsafe call was found in the files scanned; unlike data-poisoning detection it examines the saved file rather than the training data, and it shows neither that the file is safe, nor how the model behaves, nor who produced the file.

Example Use Cases

Security

An insurer's machine-learning platform team adds a ModelScan step to the job that imports models from public hubs into its internal registry. A critical or high-severity finding stops the import and opens a ticket for the security team, which decides whether the third-party checkpoint may enter the registry at all.

A university's research-computing service scans every checkpoint users upload to shared GPU storage, including PyTorch .pt files and scikit-learn models saved with joblib, before jobs on nodes shared between research groups can load them, and quarantines any file with a critical finding until its owner explains the flagged imports.

A manufacturer's operational-technology security team scans each Keras visual-inspection model delivered by an outside contractor before it is copied to factory edge devices. A flagged Lambda layer is accepted only after the contractor supplies the layer's source code and an engineer confirms that it does nothing beyond resizing and normalising images.

Limitations

It finds only what is on its list. In a peer-reviewed comparison ModelScan missed 9 of 84 known malicious models: five called functions not on its list (the authors' own example wrote files through Python's pathlib module instead of open), three loaded a further payload at run time, for example through numpy.load, and one hid its payload after an early stop instruction, where the scanner stopped reading. A clean scan shows that no known pattern was found, not that the file is safe.
Attackers adapt faster than the list. A peer-reviewed study (2026) found 22 routes by which five frameworks load pickled data, 19 of which evaded all four scanners it tested, ModelScan included, and in a 2026 preprint ModelScan detected none of 3,000 new attack files. A scan is only as current as the scanner release and unsafe list it ran with.
It checks only the files and formats it recognises, chosen by file extension, and skips the rest, so a pass can cover only some of what the loader will open. Showing that a model was checked needs the list of skipped files (ModelScan can print it) and a check that every file the loading code reads was among those scanned.
Findings need judgement. Every Keras Lambda layer is flagged, though Lambda layers are meant for ordinary pre- and post-processing, and benign models from some libraries import functions that pickle scanners treat as suspicious (one peer-reviewed study describes a widely downloaded model flagged this way). Someone able to read the flagged imports or layer code, or to question the model's author, has to decide each case.
It looks for code that runs on load, not for harmful behaviour in the weights. A model whose weights were altered to carry a backdoor, or that was trained on poisoned data, contains no unsafe call and scans clean, as does any file in a format that cannot hold code, such as safetensors; ModelScan's own documentation notes that weights can be manipulated in such formats. Backdoor and poisoning checks are separate techniques.

Resources

Software Packages

ModelScan: Protection against Model Serialization Attacks

Protection against Model Serialization Attacks. Contribute to protectai/modelscan development by creating an account on GitHub.

Tutorials

ModelScan notebook examples

Protection against Model Serialization Attacks. Contribute to protectai/modelscan development by creating an account on GitHub.

Documentations

Model Serialization Attacks (ModelScan documentation)

Related Techniques

Name	Description	Assurance Goals
Model Signing	Model signing lets the publisher of a machine-learning model attach a cryptographic signature to the model's files, so that anyone who later loads them can check who released them and that no file has changed since. Under the Open Source Security Foundation's Model Signing (OMS) specification and its reference tool, model-signing, the publisher computes a hash (a short fingerprint that changes if a single byte of the file changes) of every file in the model directory, such as weights, configuration, tokeniser and any dataset files kept with them, lists the hashes in a sorted manifest and signs the manifest, producing a detached signature file kept beside the model. The signature can be made with the publisher's own key, with a certificate from an organisation's certificate authority, or through Sigstore, which issues a short-lived certificate tied to a verified online identity, such as a person's account or a build pipeline's, and records each signing event in a transparency log, an append-only record that can be inspected afterwards. Before loading, the consumer verifies the signature against the signer it expects, recomputes every hash and rejects the model if any listed file differs or is missing or, unless that check is switched off, an unlisted file has appeared, so the evidence is a pass or fail tied to a named signer and an exact set of files. Unlike the unsigned file hashes kept by data-versioning tools, it shows who stands behind the files, and unlike a watermark, which is trained into the weights so that it survives in an altered copy, it is broken by any change; it says nothing about how the model behaves or what it was trained on.	Security
CycloneDX Machine Learning Bill of Materials	A CycloneDX ML-BOM (machine-learning bill of materials) is a machine-readable inventory of what a model is made of, written in the CycloneDX standard (ECMA-424) so that tools as well as people can check it. It applies the software bill of materials, a structured list of the components in a piece of software, to models: the model is declared as a component of type machine-learning-model with its identifiers, version, supplier, licence and file hashes, together with the base models it was derived from (its pedigree), the datasets used to train and evaluate it with their source, classification and owners, the libraries it depends on, and an optional structured model card giving its task, architecture, performance metrics, intended uses and known limitations. The BOM is written at release, by hand or with a generator that reads the model's repository and metadata, checked against the CycloneDX schema, and can be signed. Consumers then query it, for example to find every deployed model that uses a dataset or library found to be compromised, or to check a supplier's declared datasets and licences before purchase. Unlike a model card or datasheet, written for people to read, or an experiment log or data version history, which stay with the team that built the model, it is a parts list in a fixed schema that travels with the model and can be matched against vulnerability records and file hashes, but it records what its author declares and verifies none of it.	Security Transparency
Model Extraction Defence Testing	Model extraction defence testing checks whether a model offered through a query interface, such as a prediction API, can be copied by someone who can only send it inputs and read its outputs. The tester plays the attacker, spends a fixed query budget on the protected interface, then uses the answers to train a surrogate model (a copy trained on the answers) or to solve for the model's parameters, and compares the copy with the original. Fidelity is usually measured as the share of inputs on which the copy agrees with the original, together with how many queries that took. The same attack is run with and without each defence, such as limiting queries, rounding or removing confidence scores, or adding noise to outputs, to show how much each defence raises the attacker's cost. The tester also measures the effect on legitimate users, for example accuracy or response time.	Security