# Tuning models

Below we illustrate hyperparameter optimisation using the `Grid`

, `RandomSearch`

and `LatinHypercube`

tuning strategies. Also available is the tree Parzen strategy; for a complete list, see here.

MLJ tuning is implemented as an *iterative* procedure, which can accordingly be controlled using MLJ's `IteratedModel`

wrapper. After familiarizing one self with the `TunedModel`

wrapper described below, see Controlling model tuning for more on this advanced feature.

In MLJ, hyperparameter optimization, also known as model *tuning*, is implemented as a model wrapper. After wrapping a model in a tuning strategy and binding the wrapped model to data in a machine, `mach`

, calling `fit!(mach)`

instigates a search for optimal model hyperparameters, within a specified `range`

, and then uses all supplied data to train the best model. To predict using the optimal model, one just calls `predict(mach, Xnew)`

. In this way the wrapped model may be viewed as a "self-tuning" version of the unwrapped model.

For in-depth overview of tuning in MLJ, or for implementation details, see the MLJTuning documentation. For a complete list of options see the `TunedModel`

doc-string below.

## Tuning a single hyperparameter using a grid search (regression example)

```
using MLJ
X = MLJ.table(rand(100, 10));
y = 2X.x1 - X.x2 + 0.05*rand(100);
Tree = @load DecisionTreeRegressor pkg=DecisionTree verbosity=0;
tree = Tree()
```

DecisionTreeRegressor( max_depth = -1, min_samples_leaf = 5, min_samples_split = 2, min_purity_increase = 0.0, n_subfeatures = 0, post_prune = false, merge_purity_threshold = 1.0) @368

Let's tune `min_purity_increase`

in the model above, using a grid-search. To do so we will use the simplest `range`

object, a one-dimensional range object constructed using the `range`

method:

```
r = range(tree, :min_purity_increase, lower=0.001, upper=1.0, scale=:log);
self_tuning_tree = TunedModel(model=tree,
resampling=CV(nfolds=3),
tuning=Grid(resolution=10),
range=r,
measure=rms);
```

DeterministicTunedModel( model = DecisionTreeRegressor( max_depth = -1, min_samples_leaf = 5, min_samples_split = 2, min_purity_increase = 0.0, n_subfeatures = 0, post_prune = false, merge_purity_threshold = 1.0), tuning = Grid( goal = nothing, resolution = 10, shuffle = true, rng = Random._GLOBAL_RNG()), resampling = CV( nfolds = 3, shuffle = false, rng = Random._GLOBAL_RNG()), measure = RootMeanSquaredError(), weights = nothing, operation = MLJModelInterface.predict, range = NumericRange( field = :min_purity_increase, lower = 0.001, upper = 1.0, origin = 0.5005, unit = 0.4995, scale = :log), selection_heuristic = MLJTuning.NaiveSelection(nothing), train_best = true, repeats = 1, n = nothing, acceleration = CPU1{Nothing}(nothing), acceleration_resampling = CPU1{Nothing}(nothing), check_measure = true, cache = true) @624

Incidentally, a grid is generated internally "over the range" by calling the `iterator`

method with an appropriate resolution:

`iterator(r, 5)`

5-element Vector{Float64}: 0.0010000000000000002 0.005623413251903492 0.0316227766016838 0.1778279410038923 1.0

Non-numeric hyperparameters are handled a little differently:

```
selector = FeatureSelector();
r2 = range(selector, :features, values = [[:x1,], [:x1, :x2]]);
iterator(r2)
```

2-element Vector{Vector{Symbol}}: [:x1] [:x1, :x2]

Unbounded ranges are also permitted. See the `range`

and `iterator`

docstrings below for details, and the `sampler`

docstring for generating random samples from one-dimensional ranges (used internally by the `RandomSearch`

strategy).

Returning to the wrapped tree model:

```
mach = machine(self_tuning_tree, X, y);
fit!(mach, verbosity=0)
```

Machine{DeterministicTunedModel{Grid,…},…} @502 trained 1 time; caches data args: 1: Source @995 ⏎ `Table{AbstractVector{Continuous}}` 2: Source @805 ⏎ `AbstractVector{Continuous}`

We can inspect the detailed results of the grid search with `report(mach)`

or just retrieve the optimal model, as here:

`fitted_params(mach).best_model`

DecisionTreeRegressor( max_depth = -1, min_samples_leaf = 5, min_samples_split = 2, min_purity_increase = 0.0010000000000000002, n_subfeatures = 0, post_prune = false, merge_purity_threshold = 1.0) @146

For more detailed information, we can look at `report(mach)`

, for example:

`entry = report(mach).best_history_entry`

(model = DecisionTreeRegressor @146, measure = [RootMeanSquaredError @448], measurement = [0.23719236306084168], per_fold = [[0.22502102403259053, 0.22481399392244053, 0.26000934245354257]],)

Predicting on new input observations using the optimal model, *trained on all the data* bound to `mach`

:

```
Xnew = MLJ.table(rand(3, 10));
predict(mach, Xnew)
```

3-element Vector{Float64}: 0.2017418382646185 0.7122864174315473 -0.18965726477420272

Or predicting on some subset of the observations bound to `mach`

:

```
test = 1:3
predict(mach, rows=test)
```

3-element Vector{Float64}: 1.4980604607281063 0.7848367369451266 0.009831820611520034

For tuning using only a subset `train`

of all observation indices, specify `rows=train`

in the above `fit!`

call. In that case the above `predict`

calls would be based on training the optimal model on all `train`

rows.

## A probabilistic classifier example

Tuning a classifier is not essentially different from tuning a regressor. A common gotcha however is to overlook the distinction between supervised models that make point predictions (subtypes of `Deterministic`

) and those that make probabilistic predictions (subtypes of `Probabilistic`

). The `DecisionTreeRegressor`

model in the preceding illustration was deterministic, so in this example will consider a probabilistic classifier:

`info("KNNClassifier").prediction_type`

:probabilistic

```
X, y = @load_iris
KNN = @load KNNClassifier verbosity=0
knn = KNN()
```

KNNClassifier( K = 5, algorithm = :kdtree, metric = Distances.Euclidean(0.0), leafsize = 10, reorder = true, weights = NearestNeighborModels.Uniform()) @753

We'll tune the hyperparameter `K`

in the model above, using a grid-search once more:

`K_range = range(knn, :K, lower=5, upper=20);`

typename(MLJBase.NumericRange)(Int64, :K, ... )

Since the model is probabilistic, we need either to: (i) use a probabilistic measure, such as `brier_loss`

; or (ii) use a deterministic measure, such as `misclassification_rate`

, but declare `operation=predict_mode`

to ensure we are evaluating the measure on *point* predictions.

**Case (i) - probabilistic measure**:

```
self_tuning_knn = TunedModel(model=knn,
resampling = CV(nfolds=4, rng=1234),
tuning = Grid(resolution=5),
range = K_range,
measure=BrierLoss());
mach = machine(self_tuning_knn, X, y);
fit!(mach, verbosity=0);
```

Machine{ProbabilisticTunedModel{Grid,…},…} @132 trained 1 time; caches data args: 1: Source @566 ⏎ `Table{AbstractVector{Continuous}}` 2: Source @932 ⏎ `AbstractVector{Multiclass{3}}`

**Case (ii) - deterministic measure**:

```
self_tuning_knn = TunedModel(model=knn,
resampling = CV(nfolds=4, rng=1234),
tuning = Grid(resolution=5),
range = K_range,
measure=MisclassificationRate(),
operation=predict_mode);
mach = machine(self_tuning_knn, X, y);
fit!(mach, verbosity=0);
```

Machine{ProbabilisticTunedModel{Grid,…},…} @456 trained 1 time; caches data args: 1: Source @474 ⏎ `Table{AbstractVector{Continuous}}` 2: Source @265 ⏎ `AbstractVector{Multiclass{3}}`

Let's inspect the best model and corresponding evaluation of the metric in case (ii):

`entry = report(mach).best_history_entry`

(model = KNNClassifier @460, measure = [MisclassificationRate @598], measurement = [0.020092460881934567], per_fold = [[0.0, 0.02631578947368421, 0.0, 0.05405405405405406]],)

`entry.model.K`

16

Recall that fitting `mach`

also retrains the optimal model on all available data. The following is therefore an optimal model prediction based on all available data:

`predict(mach, rows=148:150)`

3-element MLJBase.UnivariateFiniteVector{Multiclass{3}, String, UInt32, Float64}: UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0625, virginica=>0.938) UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.0, virginica=>1.0) UnivariateFinite{Multiclass{3}}(setosa=>0.0, versicolor=>0.188, virginica=>0.812)

### Specifying a custom measure

Users may specify a custom loss or scoring function. Suppose, for example, we define a new scoring function `custom_accuracy`

by

`custom_accuracy(y,yhat) = mean(y .== yhat);`

custom_accuracy (generic function with 1 method)

In tuning, scores are maximised, while losses are minimised. By default, a custom measure is assumed to be a loss rather than a score, so we must also declare

`MLJ.orientation(::typeof(custom_accuracy)) = :score`

For full details on constructing custom measures, see Traits and custom measures.

Our score is deterministic; since all measures are deterministic by default, no further actioin is required. However, as in (ii) above, we must declare `operation=predict_mode`

to force our probabilistic model to deliver point predictions. We'll add a second score, `MulticlassFScore`

, to evaluate (which is ignored in determining the optimal model):

```
self_tuning_knn = TunedModel(model=knn,
resampling = CV(nfolds=4),
tuning = Grid(resolution=5),
range = K_range,
measure = [custom_accuracy, MulticlassFScore()],
operation = predict_mode);
mach = machine(self_tuning_knn, X, y)
fit!(mach, verbosity=0)
entry = report(mach).best_history_entry
```

(model = KNNClassifier @982, measure = Any[Main.ex-goof.custom_accuracy, MulticlassFScore{Float64,…} @520], measurement = [0.8856685633001422, 0.46576928643797944], per_fold = [[1.0, 0.9210526315789473, 0.918918918918919, 0.7027027027027027], [0.3333333333333333, 0.6462585034013605, 0.6083530338849488, 0.2751322751322752]],)

`entry.model.K`

5

## Tuning multiple nested hyperparameters

The `forest`

model below has another model, namely a `DecisionTreeRegressor`

, as a hyperparameter:

```
tree = Tree() # defined above
forest = EnsembleModel(atom=tree)
```

DeterministicEnsembleModel( atom = DecisionTreeRegressor( max_depth = -1, min_samples_leaf = 5, min_samples_split = 2, min_purity_increase = 0.0, n_subfeatures = 0, post_prune = false, merge_purity_threshold = 1.0), atomic_weights = Float64[], bagging_fraction = 0.8, rng = Random._GLOBAL_RNG(), n = 100, acceleration = CPU1{Nothing}(nothing), out_of_bag_measure = Any[]) @377

Ranges for nested hyperparameters are specified using dot syntax. In this case we will specify a `goal`

for the total number of grid points:

```
r1 = range(forest, :(atom.n_subfeatures), lower=1, upper=9);
r2 = range(forest, :bagging_fraction, lower=0.4, upper=1.0);
self_tuning_forest = TunedModel(model=forest,
tuning=Grid(goal=30),
resampling=CV(nfolds=6),
range=[r1, r2],
measure=rms);
X = MLJ.table(rand(100, 10));
y = 2X.x1 - X.x2 + 0.05*rand(100);
mach = machine(self_tuning_forest, X, y);
fit!(mach, verbosity=0);
```

Machine{DeterministicTunedModel{Grid,…},…} @299 trained 1 time; caches data args: 1: Source @213 ⏎ `Table{AbstractVector{Continuous}}` 2: Source @609 ⏎ `AbstractVector{Continuous}`

In this two-parameter case, a plot of the grid search results is also available:

```
using Plots
plot(mach)
```

Instead of specifying a `goal`

, we can declare a global `resolution`

, which is overriden for a particular parameter by pairing it's range with the resolution desired. In the next example, the default `resolution=100`

is applied to the `r2`

field, but a resolution of `3`

is applied to the `r1`

field. Additionally, we ask that the grid points be randomly traversed, and the the total number of evaluations be limited to 25.

```
tuning = Grid(resolution=100, shuffle=true, rng=1234)
self_tuning_forest = TunedModel(model=forest,
tuning=tuning,
resampling=CV(nfolds=6),
range=[(r1, 3), r2],
measure=rms,
n=25);
fit!(machine(self_tuning_forest, X, y), verbosity=0);
```

Machine{DeterministicTunedModel{Grid,…},…} @069 trained 1 time; caches data args: 1: Source @665 ⏎ `Table{AbstractVector{Continuous}}` 2: Source @809 ⏎ `AbstractVector{Continuous}`

For more options for a grid search, see `Grid`

below.

## Tuning using a random search

Let's attempt to tune the same hyperparameters using a `RandomSearch`

tuning strategy. By default, bounded numeric ranges like `r1`

and `r2`

are sampled uniformly (before rounding, in the case of the integer range `r1`

). Positive unbounded ranges are sampled using a Gamma distribution by default, and all others using a (truncated) normal distribution.

```
self_tuning_forest = TunedModel(model=forest,
tuning=RandomSearch(),
resampling=CV(nfolds=6),
range=[r1, r2],
measure=rms,
n=25);
mach = machine(self_tuning_forest, X, y);
fit!(mach, verbosity=0)
```

Machine{DeterministicTunedModel{RandomSearch,…},…} @580 trained 1 time; caches data args: 1: Source @384 ⏎ `Table{AbstractVector{Continuous}}` 2: Source @849 ⏎ `AbstractVector{Continuous}`

```
using Plots
plot(mach)
```

The prior distributions used for sampling each hyperparameter can be customized, as can the global fallbacks. See the `RandomSearch`

doc-string below for details.

## Tuning using Latin hypercube sampling

One can also tune the hyperparameters using the `LatinHypercube`

tuning stragegy. This method uses a genetic based optimization algorithm based on the inverse of the Audze-Eglais function, using the library `LatinHypercubeSampling.jl`

.

We'll work with the data `X`

, `y`

and ranges `r1`

and `r2`

defined above and instatiate a Latin hypercube resampling strategy:

`latin = LatinHypercube(gens=2, popsize=120)`

LatinHypercube( gens = 2, popsize = 120, ntour = 2, ptour = 0.8, interSampleWeight = 1.0, ae_power = 2, periodic_ae = false, rng = Random._GLOBAL_RNG()) @893

Here `gens`

is the number of generations to run the optimisation for and `popsize`

is the population size in the genetic algorithm. For more on these and other `LatinHypercube`

parameters, refer to the LatinHypercubeSampling.jl documentation. Pay attention that `gens`

and `popsize`

are not to be confused with the iteration parameter `n`

in the construction of a corresponding `TunedModel`

instance, which specifies the total number of models to be evaluated, independent of the tuning strategy.

```
self_tuning_forest = TunedModel(model=forest,
tuning=latin,
resampling=CV(nfolds=6),
range=[r1, r2],
measure=rms,
n=25);
mach = machine(self_tuning_forest, X, y);
fit!(mach, verbosity=0)
```

Machine{DeterministicTunedModel{LatinHypercube,…},…} @173 trained 1 time; caches data args: 1: Source @149 ⏎ `Table{AbstractVector{Continuous}}` 2: Source @628 ⏎ `AbstractVector{Continuous}`

```
using Plots
plot(mach)
```

## API

`Base.range`

— Function`r = range(model, :hyper; values=nothing)`

Define a one-dimensional `NominalRange`

object for a field `hyper`

of `model`

. Note that `r`

is not directly iterable but `iterator(r)`

is.

A nested hyperparameter is specified using dot notation. For example, `:(atom.max_depth)`

specifies the `max_depth`

hyperparameter of the submodel `model.atom`

.

```
r = range(model, :hyper; upper=nothing, lower=nothing,
scale=nothing, values=nothing)
```

Assuming `values`

is not specified, define a one-dimensional `NumericRange`

object for a `Real`

field `hyper`

of `model`

. Note that `r`

is not directly iteratable but `iterator(r, n)`

is an iterator of length `n`

. To generate random elements from `r`

, instead apply `rand`

methods to `sampler(r)`

. The supported scales are `:linear`

,`:log`

, `:logminus`

, `:log10`

, `:log2`

, or a callable object.

Note that `r`

is not directly iterable, but `iterator(r, n)`

is, for given resolution (length) `n`

.

By default, the behaviour of the constructed object depends on the type of the value of the hyperparameter `:hyper`

at `model`

*at the time of construction.* To override this behaviour (for instance if `model`

is not available) specify a type in place of `model`

so the behaviour is determined by the value of the specified type.

A nested hyperparameter is specified using dot notation (see above).

If `scale`

is unspecified, it is set to `:linear`

, `:log`

, `:logminus`

, or `:linear`

, according to whether the interval `(lower, upper)`

is bounded, right-unbounded, left-unbounded, or doubly unbounded, respectively. Note `upper=Inf`

and `lower=-Inf`

are allowed.

If `values`

is specified, the other keyword arguments are ignored and a `NominalRange`

object is returned (see above).

`MLJBase.iterator`

— Function```
iterator([rng, ], r::NominalRange, [,n])
iterator([rng, ], r::NumericRange, n)
```

Return an iterator (currently a vector) for a `ParamRange`

object `r`

. In the first case iteration is over all `values`

stored in the range (or just the first `n`

, if `n`

is specified). In the second case, the iteration is over approximately `n`

ordered values, generated as follows:

(i) First, exactly `n`

values are generated between `U`

and `L`

, with a spacing determined by `r.scale`

(uniform if `scale=:linear`

) where `U`

and `L`

are given by the following table:

`r.lower` | `r.upper` | `L` | `U` |
---|---|---|---|

finite | finite | `r.lower` | `r.upper` |

`-Inf` | finite | `r.upper - 2r.unit` | `r.upper` |

finite | `Inf` | `r.lower` | `r.lower + 2r.unit` |

`-Inf` | `Inf` | `r.origin - r.unit` | `r.origin + r.unit` |

(ii) If a callable `f`

is provided as `scale`

, then a uniform spacing is always applied in (i) but `f`

is broadcast over the results. (Unlike ordinary scales, this alters the effective range of values generated, instead of just altering the spacing.)

(iii) If `r`

is a discrete numeric range (`r isa NumericRange{<:Integer}`

) then the values are additionally rounded, with any duplicate values removed. Otherwise all the values are used (and there are exacltly `n`

of them).

(iv) Finally, if a random number generator `rng`

is specified, then the values are returned in random order (sampling without replacement), and otherwise they are returned in numeric order, or in the order provided to the range constructor, in the case of a `NominalRange`

.

`Distributions.sampler`

— Function```
sampler(r::NominalRange, probs::AbstractVector{<:Real})
sampler(r::NominalRange)
sampler(r::NumericRange{T}, d)
```

Construct an object `s`

which can be used to generate random samples from a `ParamRange`

object `r`

(a one-dimensional range) using one of the following calls:

```
rand(s) # for one sample
rand(s, n) # for n samples
rand(rng, s [, n]) # to specify an RNG
```

The argument `probs`

can be any probability vector with the same length as `r.values`

. The second `sampler`

method above calls the first with a uniform `probs`

vector.

The argument `d`

can be either an arbitrary instance of `UnivariateDistribution`

from the Distributions.jl package, or one of a Distributions.jl *types* for which `fit(d, ::NumericRange)`

is defined. These include: `Arcsine`

, `Uniform`

, `Biweight`

, `Cosine`

, `Epanechnikov`

, `SymTriangularDist`

, `Triweight`

, `Normal`

, `Gamma`

, `InverseGaussian`

, `Logistic`

, `LogNormal`

, `Cauchy`

, `Gumbel`

, `Laplace`

, and `Poisson`

; but see the doc-string for `Distributions.fit`

for an up-to-date list.

If `d`

is an *instance*, then sampling is from a truncated form of the supplied distribution `d`

, the truncation bounds being `r.lower`

and `r.upper`

(the attributes `r.origin`

and `r.unit`

attributes are ignored). For discrete numeric ranges (`T <: Integer`

) the samples are rounded.

If `d`

is a *type* then a suitably truncated distribution is automatically generated using `Distributions.fit(d, r)`

.

*Important.* Values are generated with no regard to `r.scale`

, except in the special case `r.scale`

is a callable object `f`

. In that case, `f`

is applied to all values generated by `rand`

as described above (prior to rounding, in the case of discrete numeric ranges).

**Examples**

```
r = range(Char, :letter, values=collect("abc"))
s = sampler(r, [0.1, 0.2, 0.7])
samples = rand(s, 1000);
StatsBase.countmap(samples)
Dict{Char,Int64} with 3 entries:
'a' => 107
'b' => 205
'c' => 688
r = range(Int, :k, lower=2, upper=6) # numeric but discrete
s = sampler(r, Normal)
samples = rand(s, 1000);
UnicodePlots.histogram(samples)
┌ ┐
[2.0, 2.5) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 119
[2.5, 3.0) ┤ 0
[3.0, 3.5) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 296
[3.5, 4.0) ┤ 0
[4.0, 4.5) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 275
[4.5, 5.0) ┤ 0
[5.0, 5.5) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 221
[5.5, 6.0) ┤ 0
[6.0, 6.5) ┤▇▇▇▇▇▇▇▇▇▇▇ 89
└ ┘
```

`StatsBase.fit`

— Method`Distributions.fit(D, r::MLJBase.NumericRange)`

Fit and return a distribution `d`

of type `D`

to the one-dimensional range `r`

.

Only types `D`

in the table below are supported.

The distribution `d`

is constructed in two stages. First, a distributon `d0`

, characterized by the conditions in the second column of the table, is fit to `r`

. Then `d0`

is truncated between `r.lower`

and `r.upper`

to obtain `d`

.

Distribution type `D` | Characterization of `d0` |
---|---|

`Arcsine` , `Uniform` , `Biweight` , `Cosine` , `Epanechnikov` , `SymTriangularDist` , `Triweight` | `minimum(d) = r.lower` , `maximum(d) = r.upper` |

`Normal` , `Gamma` , `InverseGaussian` , `Logistic` , `LogNormal` | `mean(d) = r.origin` , `std(d) = r.unit` |

`Cauchy` , `Gumbel` , `Laplace` , (`Normal` ) | `Dist.location(d) = r.origin` , `Dist.scale(d) = r.unit` |

`Poisson` | `Dist.mean(d) = r.unit` |

Here `Dist = Distributions`

.

`MLJTuning.TunedModel`

— Function```
tuned_model = TunedModel(; model=nothing,
tuning=Grid(),
resampling=Holdout(),
measure=nothing,
weights=nothing,
repeats=1,
operation=predict,
range=nothing,
selection_heuristic=NaiveSelection(),
n=default_n(tuning, range),
train_best=true,
acceleration=default_resource(),
acceleration_resampling=CPU1(),
check_measure=true,
cache=true)
```

Construct a model wrapper for hyperparameter optimization of a supervised learner.

Calling `fit!(mach)`

on a machine `mach=machine(tuned_model, X, y)`

or `mach=machine(tuned_model, X, y, w)`

will:

Instigate a search, over clones of

`model`

, with the hyperparameter mutations specified by`range`

, for a model optimizing the specified`measure`

, using performance evaluations carried out using the specified`tuning`

strategy and`resampling`

strategy.Fit an internal machine, based on the optimal model

`fitted_params(mach).best_model`

, wrapping the optimal`model`

object in*all*the provided data`X`

,`y`

(,`w`

). Calling`predict(mach, Xnew)`

then returns predictions on`Xnew`

of this internal machine. The final train can be supressed by setting`train_best=false`

.

The `range`

objects supported depend on the `tuning`

strategy specified. Query the `strategy`

docstring for details. To optimize over an explicit list `v`

of models of the same type, use `strategy=Explicit()`

and specify `model=v[1]`

and `range=v`

.

The number of models searched is specified by `n`

. If unspecified, then `MLJTuning.default_n(tuning, range)`

is used. When `n`

is increased and `fit!(mach)`

called again, the old search history is re-instated and the search continues where it left off.

If `measure`

supports weights (`supports_weights(measure) == true`

) then any `weights`

specified will be passed to the measure. If more than one `measure`

is specified, then only the first is optimized (unless `strategy`

is multi-objective) but the performance against every measure specified will be computed and reported in `report(mach).best_performance`

and other relevant attributes of the generated report.

Specify `repeats > 1`

for repeated resampling per model evaluation. See `evaluate!`

options for details.

*Important.* If a custom `measure`

is used, and the measure is a score, rather than a loss, be sure to check that `MLJ.orientation(measure) == :score`

to ensure maximization of the measure, rather than minimization. Override an incorrect value with `MLJ.orientation(::typeof(measure)) = :score`

.

In the case of two-parameter tuning, a Plots.jl plot of performance estimates is returned by `plot(mach)`

or `heatmap(mach)`

.

Once a tuning machine `mach`

has bee trained as above, then `fitted_params(mach)`

has these keys/values:

key | value |
---|---|

`best_model` | optimal model instance |

`best_fitted_params` | learned parameters of the optimal model |

The named tuple `report(mach)`

includes these keys/values:

key | value |
---|---|

`best_model` | optimal model instance |

`best_history_entry` | corresponding entry in the history, including performance estimate |

`best_report` | report generated by fitting the optimal model to all data |

`history` | tuning strategy-specific history of all evaluations |

plus other key/value pairs specific to the `tuning`

strategy.

**Summary of key-word arguments**

`model`

:`Supervised`

model prototype that is cloned and mutated to generate models for evaluation`tuning=Grid()`

: tuning strategy to be applied (eg,`RandomSearch()`

)`resampling=Holdout()`

: resampling strategy (eg,`Holdout()`

,`CV()`

),`StratifiedCV()`

) to be applied in performance evaluations`measure`

: measure or measures to be applied in performance evaluations; only the first used in optimization (unless the strategy is multi-objective) but all reported to the history`weights`

: sample weights to be passed the measure(s) in performance evaluations, if supported.`repeats=1`

: for generating train/test sets multiple times in resampling; see`evaluate!`

for details`operation=predict`

: operation to be applied to each fitted model; usually`predict`

but`predict_mean`

,`predict_median`

or`predict_mode`

can be used for`Probabilistic`

models, if the specified measures are`Deterministic`

`range`

: range object; tuning strategy documentation describes supported types`selection_heuristic`

: the rule determining how the best model is decided. According to the default heuristic,`NaiveSelection()`

,`measure`

(or the first element of`measure`

) is evaluated for each resample and these per-fold measurements are aggregrated. The model with the lowest (resp. highest) aggregate is chosen if the measure is a`:loss`

(resp. a`:score`

).`n`

: number of iterations (ie, models to be evaluated); set by tuning strategy if left unspecified`train_best=true`

: whether to train the optimal model`acceleration=default_resource()`

: mode of parallelization for tuning strategies that support this`acceleration_resampling=CPU1()`

: mode of parallelization for resampling`check_measure=true`

: whether to check`measure`

is compatible with the specified`model`

and`operation`

)`cache=true`

: whether to cache model-specific representations of user-suplied data; set to`false`

to conserve memory. Speed gains likely limited to the case`resampling isa Holdout`

.

`MLJTuning.Grid`

— Type`Grid(goal=nothing, resolution=10, rng=Random.GLOBAL_RNG, shuffle=true)`

Instantiate a Cartesian grid-based hyperparameter tuning strategy with a specified number of grid points as `goal`

, or using a specified default `resolution`

in each numeric dimension.

**Supported ranges:**

A single one-dimensional range or vector of one-dimensioinal ranges can be specified. Specifically, in `Grid`

search, the `range`

field of a `TunedModel`

instance can be:

A single one-dimensional range - ie,

`ParamRange`

object -`r`

, or pair of the form`(r, res)`

where`res`

specifies a resolution to override the default`resolution`

.Any vector of objects of the above form

Two elements of a `range`

vector may share the same `field`

attribute, with the effect that their grids are combined, as in Example 3 below.

`ParamRange`

objects are constructed using the `range`

method.

Example 1:

`range(model, :hyper1, lower=1, origin=2, unit=1)`

Example 2:

```
[(range(model, :hyper1, lower=1, upper=10), 15),
range(model, :hyper2, lower=2, upper=4),
range(model, :hyper3, values=[:ball, :tree])]
```

Example 3:

```
# a range generating the grid `[1, 2, 10, 20, 30]` for `:hyper1`:
[range(model, :hyper1, values=[1, 2]),
(range(model, :hyper1, lower= 10, upper=30), 3)]
```

Note: All the `field`

values of the `ParamRange`

objects (`:hyper1`

, `:hyper2`

, `:hyper3`

in the preceding example) must refer to field names a of single model (the `model`

specified during `TunedModel`

construction).

**Algorithm**

This is a standard grid search with the following specifics: In all cases all `values`

of each specified `NominalRange`

are exhausted. If `goal`

is specified, then all resolutions are ignored, and a global resolution is applied to the `NumericRange`

objects that maximizes the number of grid points, subject to the restriction that this not exceed `goal`

. (This assumes no field appears twice in the `range`

vector.) Otherwise the default `resolution`

and any parameter-specific resolutions apply.

In all cases the models generated are shuffled using `rng`

, unless `shuffle=false`

.

See also `TunedModel`

, `range`

.

`MLJTuning.RandomSearch`

— Type```
RandomSearch(bounded=Distributions.Uniform,
positive_unbounded=Distributions.Gamma,
other=Distributions.Normal,
rng=Random.GLOBAL_RNG)
```

Instantiate a random search tuning strategy, for searching over Cartesian hyperparameter domains, with customizable priors in each dimension.

**Supported ranges**

A single one-dimensional range or vector of one-dimensioinal ranges can be specified. If not paired with a prior, then one is fitted, according to fallback distribution types specified by the tuning strategy hyperparameters. Specifically, in `RandomSearch`

, the `range`

field of a `TunedModel`

instance can be:

a single one-dimensional range (

`ParamRange`

object)`r`

a pair of the form

`(r, d)`

, with`r`

as above and where`d`

is:a probability vector of the same length as

`r.values`

(`r`

a`NominalRange`

)any

`Distributions.UnivariateDistribution`

*instance*(`r`

a`NumericRange`

)one of the

*subtypes*of`Distributions.UnivariateDistribution`

listed in the table below, for automatic fitting using`Distributions.fit(d, r)`

, a distribution whose support always lies between`r.lower`

and`r.upper`

(`r`

a`NumericRange`

)

any pair of the form

`(field, s)`

, where`field`

is the (possibly nested) name of a field of the model to be tuned, and`s`

an arbitrary sampler object for that field. This means only that`rand(rng, s)`

is defined and returns valid values for the field.any vector of objects of the above form

A range vector may contain multiple entries for the same model field, as in `range = [(:lambda, s1), (:alpha, s), (:lambda, s2)]`

. In that case the entry used in each iteration is random.

distribution types | for fitting to ranges of this type |
---|---|

`Arcsine` , `Uniform` , `Biweight` , `Cosine` , `Epanechnikov` , `SymTriangularDist` , `Triweight` | bounded |

`Gamma` , `InverseGaussian` , `Poisson` | positive (bounded or unbounded) |

`Normal` , `Logistic` , `LogNormal` , `Cauchy` , `Gumbel` , `Laplace` | any |

`ParamRange`

objects are constructed using the `range`

method.

**Examples**

```
using Distributions
range1 = range(model, :hyper1, lower=0, upper=1)
range2 = [(range(model, :hyper1, lower=1, upper=10), Arcsine),
range(model, :hyper2, lower=2, upper=Inf, unit=1, origin=3),
(range(model, :hyper2, lower=2, upper=4), Normal(0, 3)),
(range(model, :hyper3, values=[:ball, :tree]), [0.3, 0.7])]
# uniform sampling of :(atom.λ) from [0, 1] without defining a NumericRange:
struct MySampler end
Base.rand(rng::Random.AbstractRNG, ::MySampler) = rand(rng)
range3 = (:(atom.λ), MySampler())
```

**Algorithm**

In each iteration, a model is generated for evaluation by mutating the fields of a deep copy of `model`

. The range vector is shuffled and the fields sampled according to the new order (repeated fields being mutated more than once). For a `range`

entry of the form `(field, s)`

the algorithm calls `rand(rng, s)`

and mutates the field `field`

of the model clone to have this value. For an entry of the form `(r, d)`

, `s`

is substituted with `sampler(r, d)`

. If no `d`

is specified, then sampling is uniform (with replacement) if `r`

is a `NominalRange`

, and is otherwise given by the defaults specified by the tuning strategy parameters `bounded`

, `positive_unbounded`

, and `other`

, depending on the field values of the `NumericRange`

object `r`

.

See also `TunedModel`

, `range`

, `sampler`

.

`MLJTuning.LatinHypercube`

— TypeLatinHypercube(gens = 1, popsize = 100, ntour = 2, ptour = 0.8., interSampleWeight = 1.0, ae*power = 2, periodic*ae = false, rng=Random.GLOBAL_RNG)

Instantiate grid-based hyperparameter tuning strategy using the library LatinHypercubeSampling.jl.

An optimised Latin Hypercube sampling plan is created using a genetic based optimization algorithm based on the inverse of the Audze-Eglais function. The optimization is run for `nGenerations`

and creates `n`

models for evaluation, where `n`

is specified by a corresponding `TunedModel`

instance, as in

```
tuned_model = TunedModel(model=...,
tuning=LatinHypercube(...),
range=...,
measures=...,
n=...)
```

(See `TunedModel`

for complete options.)

To use a periodic version of the Audze-Eglais function (to reduce clustering along the boundaries) specify `periodic_ae = true`

.

**Supported ranges:**

A single one-dimensional range or vector of one-dimensioinal ranges can be specified. Specifically, in `LatinHypercubeSampling`

search, the `range`

field of a `TunedModel`

instance can be:

- A single one-dimensional range - ie,
`ParamRange`

object -`r`

, constructed

using the `range`

method.

- Any vector of objects of the above form

Both `NumericRange`

s and `NominalRange`

s are supported, and hyper-parameter values are sampled on a scale specified by the range (eg, `r.scale = :log`

).