Thermodynamic Analytics Toolkit (TATi)

This program suite is implemented using python3 and the development mainly focused on Linux (development machine used Ubuntu 14.04 up to 18.04). At the moment other operating systems are not supported but may still work.

It has the following non-trivial dependencies:

Note that these packages can be easily installed using either the repository tool (using some linux derivate such as Ubuntu), e.g.

or via pip3, i.e.

For acor a few extra changes are required.

The last command replaces the third line in the file acor/acor.py such that the function acor (and not the module acor) is used.

Moreover, the following packages are not ultimately required but examples or tests may depend on them:

The documentation is written in AsciiDoc and doxygen and requires a suitable package to compile to HTML or create a PDF, e.g., using dblatex

Finally, for the diffusion map analysis we recommend using the pydiffmap package, see https://github.com/DiffusionMapsAcademics/pyDiffMap.

In our setting what typically worked best was to use anaconda in the following manner:

In case your machine has GPU hardware for tensorflow, replace “tensorflow” by “tensorflow-gpu”.

Henceforth, we assume that there is a working tensorflow on your system, i.e. inside the python3 shell

should print “Hello world” or similar.

Installation comes in three flavors: as a PyPI package, or through either via a tarball or a cloned repository.

In general, the PyPI (pip) packages are strongly recommended, especially if you only want to use the software.

The tarball releases are recommended if you only plan to use TATi and do not intend ito modify its code. If, however, you need to use a development branch, then you have to clone from the repository.

In general, this package is distributed via autotools, "compiled" and installed via automake. If you are familiar with this set of tools, there should be no problem. If not, please refer to the text INSTALL file that is included in the tarball.

Unpack the archive, assuming its suffix is .bz2.

If the ending is .gz, you need to unpack by

Enter the directory

Continue then in section Configure, make, install.

While the tarball does not require any autotools packages installed on your system, the cloned repository does. You need the following packages:

To prepare code in the working directory, enter

Next, we recommend to build the toolkit not in the source folder but in an extra folder, e.g., “build64”. In the autotools lingo this is called an out-of-source build. It prevents cluttering of the source folder. Naturally, you may pick any name (and actually any location on your computer) as you see fit.

More importantly, please replace “somepath” and “path to python3” by the desired installation path and the full path to the python3 executable on your system.

Structured grids such as naive full grid approaches, where a fixed number of points with equidistant spacing is used per axis, suffer from the Curse of Dimensionality, a term coined by Bellman. With increasing number of parameters $N$ the computational cost becomes prohibitive. This Curse of Dimensionality is alleviated to some extent by so-called Sparse Grids, see [Bungartz2004], where the quality bounds on the approximation are kept but fewer grid points need to be evaluated. However, they only alleviate the curse to some extent, see [Pflueger2010], and are still infeasible at the moment for the extremely high-dimensional manifolds encountered in neural network losses.

Random sequences are Monte Carlo approaches where a specific sequence of random numbers decides which point in the whole space to evaluate. Due to their inherent stochasticity these lack the rigor of the structured grid methods but neither rely on, nor exploit any regularity as the structured grid methods. Therefore, they suffer no Curse of Dimensionality with respect to their convergence rate. The rate is bounded by the Central Limit Theorem to ${O}(n^{-\frac 1 2})$ if $n$ is the number of evaluated points.

Quasi-Monte Carlo (QMC) methods use deterministic sequences, such as Lattice rules (Hammersley, \ldots) or digital nets, and are able to obtain higher convergence rates of ${O}(n^{-1} (\log{n})^d)$ at the price of a moderate dependence on dimension $d$.

In this category of sequences we also have dynamics-based sequences. As examples of Markov Chain Monte Carlo (MCMC) methods, they are closely connected to pure Monte Carlo approaches. They make use of the ergodic property of the chosen dynamics allowing to replace high-dimensional whole-space integrals by single-dimensional time-integrals, where the accuracy depends on the trajectory length.

The chain can be generated through suitable dynamics such as Hamiltonian,

or Langevin dynamics,

for positions (or parameters) $\theta$, momenta $p$, mass matrix $M$, potential (or loss) $L(\theta)$, friction constant $\gamma$ and a stationary normal random process $W$ with variance $\sigma$. Note that Langevin dynamics has both Hamiltonian dynamics and Brownian dynamics as limiting cases of the friction constant $\gamma$ going to zero or infinity, respectively.

There are also hybrid approaches where a purely deterministic sequence is randomized by a Metropolis-Hastings criterion to remove possible bias, such as Hybrid or Hamiltonian Monte Carlo, see [Neal2011]. Note that in the case of Langevin dynamics the stochasticity enters through the random process.

If we consider the function $L_{D}(\theta)$ as a potential energy function, then we may cast this into a probability distribution using the canonical Gibbs distribution $Z \cdot \exp( -\beta L_{D}(\theta))$, where $Z$ is a normalization constant and $\beta$ is the inverse temperature factor. Then, we have sampling in the typical sense in statistics where an unknown distribution is evaluated. We remark that this Gibbs measure is known in the neural networks community through the energy interpretation ([LeCun2006) of a probability distribution in relation to a certain reference energy, given by the temperature.

And indeed, dynamics-based sampling does not aim to approximate $L$ as best as possible but its Gibbs (also called the canonical) distribution $\exp{(-\beta L)}$. In our case we are only interested in particular subsets of the space, namely those associated with a small loss. As we have given ample evidence, the central challenge in sampling is the computational cost. The dynamics-based sequences allow to save computational cost in the high-dimensional spaces by incorporating gradient information. Hence, the more accurately we sample from the Gibbs distribution, the more efficiently we sample only those subsets of interest. This saving could not be obtained with a pure Monte Carlo approach.

Therefore, we focus on dynamics-based sampling for this high-dimensional exploration problem.

In principle, the underlying challenge is to assure that the sampling trajectory covers a sufficient area of the whole space for validity. However, stating when to terminate is as difficult as the exploration task itself. Hence, the best measure is to assess when a new independent state has been obtained.

When inspecting sampled MCMC trajectories, we need to assess how many consecutive steps it takes to get from one independent state to another. This is estimated by the Integrated Autocorrelation Time (IAT) $\tau_s$. For any observable $A$, we have that its variance generally behaves as $var_{A} = \frac {var_{\pi} (\varphi(X))}{T_s/\tau_s}$, where $\pi$ is the target density, $\varphi(X)$ is the function of interest of the random variable $X$, and $T_s$ is the sampling time, see [Goodman2010]. Obviously, when time steps are discrete and $\tau_s$ is measured in number of steps, then $\tau_s = 1$ is highly desirable, i. e.~immediately stepping from one independent state to the next. The IAT $\tau_s$ is defined as

The above holds also for sampling approaches based on Langevin Dynamics. There, we may use the IAT to gauge the exploration speed for each sampled trajectory $X(t)$.

Let us give a trivial example to illustrate the above with a few figures. We want to highlight in the following the key aspect about the dynamics-based sampling approach, namely sampling the probability distribution function associated with the Gibbs measure and not the loss manifold directly.

Assume we are given a very simple data set as depicted in Dataset. The goal is to classify all red and blue dots into two different classes. This problem is quite simple to solve: a line in the two-dimensional space can easily separate the two classes.

A very simple neural network, a perceptron: it would use one input node, either of the coordinate, $x_{1}$ or $x_{2}$, and a single output node with an activation function $f$ whose sign gives the class the input item belongs to. The network is given in Network. The network is chosen non-ideal by design to illustrate a point.

In Figure Loss manifold we then turn to the two-dimensional loss landscape depending on the two weights. In this very low-dimensional case we turn to the "naive grid" approach and partition each axis equidistantly. We see two minima basins both of hyperbole or "banana" shape. Here, we see that there is not a single minima but two of them. This is caused by the deliberate permutation symmetry of the two weights in the network.

Assume we additionally perform dynamics-based sampling. In the figure the resulting trajectory is given as squiggly black line. Here, we have chosen such an (inverse) temperature value such that it is able to pass the potential barrier and reach the other minima basin.

As we clearly see, the grid-based approach does not distinguish between the areas of high loss and the areas of low loss. Note that the coloring comes from a spline interpolation from the grid points. The dynamics-based trajectory on the other hand remains in areas of low loss all the time, where "low" is relative to its inverse temperature parameter. This is because the areas of low loss have a much higher probability in the Gibbs measure which our sampler is faithful to.

This quick description of the sampling loss manifolds in the context of neural networks in data science should have acquainted we with some of the concepts underlying the idea of sampling.

In the following, we will use the following notation:

The first thing in all the following example we will do is instantiate the tati class.

Although it is the simulation module, we "nickname" it tati in the following and hence will simply refer to this instance as tati.

This class takes a list of options in its construction or __init__() call. These options inform it about the dataset to use, the specific network topology, what sampler or optimizer to use and its parameters and so on.

To see how this works, we will first need a dataset to work on.

In the following we will first be creating a dataset to work on. This example code will be the most extensive one. All following ones are rather short and straight-forward.

Having created the dataset and explained how the network is set up, we know see how to evaluate the loss and the gradients.

The main idea of the simulation module is to be used as a simplified interface to access the loss and the gradients of the neural network without having to know about the internal of the neural network. In other words, we want to treat it as an abstract high-dimensional function, depending implicitly on the weights and explicitly on the dataset. To this end, the weights and biases are represented as one linearized vector. Moreover, we have another abstract high-dimensional function, the loss that depends explicitly on the weights and implicitly on the dataset, whose derivative (the gradients with respect to the parameters) is available as a numpy array, see also the section Notation.

In the following example we set up a simple fully-connected hidden network and evaluates loss and then the associated gradients.

Again, we set up the network as before, here it is a single-layer perceptron as the default value for hidden_dimension is 0. Next, we evaluate the loss $L_D(w)$ using nn.loss() and the gradients $\nabla_w L_D(w)$ using nn.gradients(), returning a vector with the gradient per degree of freedom of the network.

As we see in the above example, tati forms the general interface class that contains the network along with the dataset and everything in its internal state.

This is basically all the access we need in order to use our own optimization, sampling, or exploration methods in the context of neural networks in a high-level, abstract way.

Let us then start with optimizing the network, i.e. learning the data.

Again all options are set in the init call to the interface. These options control how the optimization is performed, what kind of network is created, how often values are stored, and so on. Next, we call nn.fit() to perform the actual training for the chosen number of training steps (max_steps). We obtain a single return structure within which we find three pandas dataframes: run info, trajectory, and averages. Of each we print the last ten values.

Let us quickly go through each of the new parameters:

For these small networks the option do_hessians might be useful which will compute the hessian matrix at the end of the trajectory and use the largest eigenvalue to compute the optimal step width. This will add nodes to the underlying computational graph for computing the components of the hessian matrix. However, we will not do so here.

After the options have been provided, the network is initialized internally and automatically, we then call fit() which performs the training and returns a structure containing runtime info, trajectory, and averages as a pandas DataFrame.

In the following section on sampling we will explain what each of these three dataframes contains exactly.

Let us have a quick glance at the decrease of the loss function over the steps by using matplotlib. In other words, let us look at how effective the training has been.

The loss per step is contained in both the run info and the trajectory dataframe in the column loss.

The graph should look similar to the one obtained with pgfplots here (see https://sourceforge.net/pgfplots).

As we see the loss has decreased quite quickly down to 1e-3. Go and have a look at the other columns such as accuracy. Or try to visualize the change in the parameters (weights and biases) in the trajectories dataframe. See 10 Minutes to pandas if we are unfamiliar with the pandas module, yet.

Obviously, we did not use a different dataset set for testing the effectiveness of the training which should commonly be done. This way we cannot check whether we have overfitted or not. However, our example is trivial by design and the network too small to be prone to overfitting this dataset.

Nonetheless, we show how to supply a different dataset and evaluate loss and accuracy on it.

Typically, as preparation to a sampling run, one would optimize or equilibrate the initially random positions first. This might be considered a specific way of initializing parameters.

However, let us first ignore this good practice for a moment and simply look at sampling from a random initial place on the loss manifold. We will come back to it later on.

Here, the sampler setting takes the place of the optimizer before as it states which sampling scheme to use. See [reference.samplers] for a complete list and their parameter names. Apart from that the example code is very much the same as in the example involving fit().

Again, we produce a single data structure that contains three data frames: run info, trajectory, and averages. Trajectories contains among others all parameter degrees of freedom $w=\{w_1, \ldots, w_M\}$ for each step (or every_nth step). Run info contains loss, accuracy, norm of gradient, norm of noise and others, again for each step. Finally, in averages we compute running averages over the trajectory such as average (ensemble) loss, averag kinetic energy, average virial, see general concepts.

Take a peep at sampling_data.run_info.columns to see all columns in the run info dataframe (and similarly for the others.)

For the running averages it is advisable to skip some initial ateps (burn_in_steps) to allow for some burn in time, i.e. for kinetic energies to adjust from initially zero momenta.

Some columns in averages and in run info depend on whether the sampler provides the specific quantity, e.g. SGLD does not have momentum, hence there will be no average kinetic energy.

Analysis involves parsing in run and trajectory files that we have written during optimization and sampling runs. Naturally, we could also perform this on the pandas dataframes directly, i.e. sampling and analysis in the same python script. However, for completeness we will read from files in the examples of this section.

The analysis functionality has been split into specific operations such as computing averages, computing the covariance matrix or generating a diffusion map analysis. See the source folder src/TAT/analysis, for all contained modules therein each represent such an operation.

In the following we will highlight just a few of them.

All these analysis operations are also possible through TATiAnalyser, see tools/TATiAnalyser.in in the repository.

This has been the quickstart introduction to the simulation interface.

If we want to take this further, we recommend reading how to implement a GLA2 sampler using this module.

If we still want to take it further, then we need to look at the programmer’s guide that should accompany our installation.

As data is read from file, this file needs to be created beforehand.

For a certain set of simple classification problems, namely those that can be found in the tensorflow playground, we have added a TATiDatasetWriter that spills out the dataset in CSV format.

This will write 500 datums of the dataset type 2 (two clusters) to a file testset-twoclusters.csv using all of the points as we have set the test/train ratio to 0. Note that we also perturb the points by 0.1 relative noise.

Similarly, for testing the dataset can be parsed using the same tensorflow machinery as is done for sampling and optimizing, using

where the seed is used for shuffling the dataset.

This will print 20 randomly drawn items from the dataset.

As weights (and biases) are usually uniformly random initialized and the potential may therefore start with large values, we first have to optimize the network, using (Stochastic) Gradient Descent (GD).

Sometimes it might be desirable to freeze parameters during training or sampling. This can be done as follows:

Note that we fix the parameter where we give its name in full tensorflow namescope: "layer1" for the network layer, "weights" for the weights ("biases" alternatively) and "Variable:0" is fixed (as it is the only one). This is followed by a comma-separated list of values, one for each component.

This call will parse the dataset from the file dataset-twoclusters.csv. It will then perform a (Stochastic) Gradient Descent optimization in batches of 50 (10% of the dataset) of the parameters of the network using a step width/learning rate of 0.01 and do this for 1000 steps after which it stops and writes the resulting neural network in a TensorFlow-specific format to a set of files, one of which is called model.ckpt.meta (and the other filenames are derived from this).

We have also created a file run.csv which contains among others the loss at each (every_nth, respectively) step of the optimization run. Plotting the loss over the step column from the run file will result in a figure similar to in Loss history.

If you need to compute the optimal step width, which is possible for smaller networks from the largest eigenvalue of the hessian matrix, then use the option do_hessians 1 to activate it.

Next, we show how to use the sampling tool called TATiSampler.

This will cause the sampler to parse the same dataset as before. Afterwards it will use the GeometricLangevinAlgorithm_2nd order discetization using again step_width of 0.01 and running for 1000 steps in total. The GLA is a descretized variant of Langevin Dynamics whose accuracy scales with the inverse square of the step_width (hence, 2nd order).

The seed is needed as we sample using Langevin Dynamics where a noise term is present whose magnitude scales with inverse_temperature.

After it has finished, it will create three files; a run file run.csv containing run time information such as the step, the potential, kinetic and total energy at each step, a trajectory file trajectory.csv with each parameter of the neural network at each step, and an averages file averages.csv containing averages accumulated along the trajectory such as average kinetic energy, average virial ( connected to the kinetic energy through the virial theorem, valid if a prior keeps parameters bound to finite values), and the average (ensemble) loss. Moreover, for the HMC sampler the average rejection rate is stored there. The first two files we need in the next stage.

As it is good practice to first optimize and then sample, also referred as to start from an equilibrated position, we may load the model saved after optimization by adding the option:

Otherwise we start from randomly initialized weights and non-zero biases.

Eventually, we now want to analyse the obtained trajectories. The trajectory file written in the last step is simply a matrix of dimension (number of parameters) times (number of trajectory steps).

The analysis can perform a number of different operations where we name a few:

There are a few more tools available in TATi. They allow to inspect the loss manifold or the input space with respect to a certain parameter set.

It can be sometimes useful to use TATi to simulate a simple harmonic oscillator. This model can be obtained by training the neural network with one parameter on one point $X=1$ in dimension one with a label equal to zero, i.e. $Y=0$, and using the mean square loss function and linear activation function. More precisely, the cost function becomes in this setting:

$L(\omega | X,Y) = | \omega X + b - Y|^2$,

where we fix the bias $b=0$.

optimizer: GradientDescent

This implementation directly calls upon Tensorflow’s GradientDescentOptimizer. It is slightly modified for additional book-keeping of norms of gradients and the virial (this can be deactivated setting do_accumulates to False).

The step update is: $\theta^{n+1} = \theta^{n} - \nabla U(\theta) \Delta t$,

optimizer: BarzilaiBorweinGradientDescent

The update step is the same as for [reference.optimizers.gd]. However, the learning rate $\Delta t$ is modified as, see [Barzilai1988],

$\Delta t = \frac{ (\theta^{n} - \theta^{n-1})\cdot(\nabla U(\theta^{n})- \nabla U(\theta^{n-1})) }{ |(\nabla U(\theta^{n})- \nabla U(\theta^{n-1})|^2 }$.

where the learning rate is bounded in the interval [1e-10,1e+10] to ensure numerical stability. If the above calculation should be invalid, the default learning rate $\Delta t$ is used.

sampler: StochasticGradientLangevinDynamics

The Stochastic Gradient Langevin Dynamics (SGLD) was proposed by [Welling2011] based on the [SGD][Stochastic Gradient Descent], which is a variant of the [GD][Gradient Descent] using only a subset of the dataset for computing gradients. The central idea behind SGLD was to add an additional noise term whose magnitude then controls the noise induced by the approximate gradients. By a suitable reduction of the step width the method can be shown to be globally convergent. This reduction is typically not done in practice.

Implements a Stochastic Gradient Langevin Dynamics Sampler in the form of a TensorFlow Optimizer, overriding tensorflow.python.training.Optimizer.

The step update is: $\theta^{n+1} = \theta^{n} - \nabla U(\theta) \Delta t + \sqrt{\frac{2\Delta t} {\beta}} G^n$, where $\beta$ is the inverse temperature coefficient, $\Delta t$ is the (discretization) step width and $\theta$ is the parameter vector, $U(\theta)$ the energy or loss function, and $G\sim N (0, 1)$.

sampler: CovarianceControlledAdaptiveLangevin

This is an extension of Stochastic Gradient Descent proposed by [Shang2015]. The key idea is to dissipate the extra heat caused by the approximate gradients through a suitable thermostat. However, the discretisation used here is not based on the (first-order) Euler-Maruyama as [SGLD] but on [GLA] 2nd order.

sampler: Geometric Langevin Algorithms_1stOrder, Geometric Langevin Algorithms_2ndOrder

GLA results from a first-order splitting between the Hamiltonian and the Ornstein-Uhlenbeck parts, see [Leimkuhler2015][section 2.2.3, Leimkuhler 2015] and also [Leimkuhler2012]. If the Hamiltonian part is discretized with a scheme of second order as in GLA2, it provides second order accuracy at basically no extra cost.

The update step of the parameters for second order GLA is BABO:

$B: p^{n+1/2} = p^n -\nabla U(q^n)\frac{\Delta t}{2}$

$A: q^{n+1} =q^n+M^{-1}p^{n+1/2}\Delta t$

$B: \tilde{p}^{n+1} = p^{n+1/2} -\nabla U(q^{n+1})\frac{\Delta t}{2}$

$O: p^{n+1} = \alpha_{\Delta t}\tilde{p}^{n+1}+ \sqrt{\frac{1-\alpha_{\Delta t}^2}{\beta}M}G^n$ where $\alpha_{\Delta t}=exp(-\gamma \Delta t)$ and $G\sim N (0, 1)$.

The first order GLA is BAO.

sampler: BAOAB

BAOAB derives from the basic building blocks A (position update), B (momentum update), and O (noise update) into which the Langevin system is split up. Each step is solved in a separate step. Hence, we perform a B step, then an A step, … and so on. This scheme has second-order accuracy and superb overall accuracy with respect to positions. See [Leimkuhler2012] for more details.

The update step of the parameters is:

$B: p^{n+1/2} = p^n -\nabla U(q^n)\frac{\Delta t}{2}$

$A: q^{n+1} =q^n+M^{-1}p^{n+1/2}\frac{\Delta t}{2}$

$O: p^{n+1} = \alpha_{\Delta t}\tilde{p}^{n+1}+ \sqrt{\frac{1-\alpha_{\Delta t}^2}{\beta}M}G^n$

$A: q^{n+1} =q^n+M^{-1}p^{n+1/2}\frac{\Delta t}{2}$

$B: p^{n+1/2} = p^n -\nabla U(q^n)\frac{\Delta t}{2}$

where $\alpha_{\Delta t}=exp(-\gamma \Delta t)$ and $G\sim N (0, 1)$.

sampler: HamiltonianMonteCarlo_1stOrder (modified Euler), HamiltonianMonteCarlo_2ndOrder (Leapfrog)

HMC is based on Hamiltonian dynamics instead of Langevin Dynamics. Noise only enters when, after the evaluation of an acceptance criterion, the momenta are redrawn randomly. It has first been proposed by [Duane1987].

This Metropolisation ensures that too large step widths will not negatively affect the sampled distribution. In contrast to the sampling through Langevin Dynamics there is no bias with respect to the step width. However, a large step width will potentially cause a higher rejection rate.

One further virtue of HMC is when using longer Verlet time integration legs (hamiltonian_dynamics_time larger than step_width) that the walker will progress much further than in the case of Langevin Dynamics. This is because it only uses the gradient. In contrast to samplers based on Langevin dynamics, it is not performing partly a Brownian motion through the injected noise. Langevin dynamics can be recovered by using just a single step for time integration (i.e. a single step is immediately followed by evaluating the acceptance criterion). This is called "Langevin Monte Carlo" which does however lack the favorable scaling properties of HMC, i.e. when using multiple steps. Random walk scales as $d^2$ in the amount of computation time for a dataset of dimensionality $d$, while HMC scales only as $d^{5/4}$. LMC on the other hand scales as $d^{4/3}$, see Neal, 2011, section 4.4 and 5.2.

However, longer legs again typically cause higher rejection rates. Therefore, they come at the price of extra computational work in terms of possibly rejected legs. There are specific rejection rates that are optimal. Typically they range between 35% ($L\gg 1$) and 43% ($L=1$). In essence, the rejection rate tells about getting sufficiently far from the initial state to a truly independent proposal state. See Neal, 2011, section 5.2 for details and in general [Neal2011] for a very readable, general introduction to HMC in the context of Markov Chain Monte Carlo methods.

Ensemble of Walkers uses a collection of walkers that exchange gradient and parameter information in each step in order to calculate a preconditioning matrix. This preconditioning allows to explore elongated minimum basins faster than independent walkers would do alone, see [Matthews2018].

This is activated by setting the number_walkers to a value larger than 1. Note that covariance_blending controls the magnitude of the covariance matrix approximation and collapse_after_steps controls after how many steps the walkers are restarted at the parameter configuration of the first walker to ensure that the harmonic approximation still holds.

This works for all of the aforementioned samplers as simply the gradient of each walker is rescaled.

Therefore, let us write a python function that works on several numpy arrays producing a single GLA2 update step by performing the four integration steps above. To this end, we make use tati.gradients() for computing the gradients $\nabla_x L(x_{n+1})$.

As the source of noise we have simply used `numpy`s standard normal distribution.

We could have used nn.momenta for storing momenta. However, this needs some extra computations for assigning the momenta inside the tensorflow computational graph. As they are not needed in the graph anyway, we can store them outside directly.

Note that we have been a bit wasteful in the above implementation but very close to the formulas in [reference.implementing_sampler].

We evaluate the gradient twice but actually only one evaluation would have been needed: The update gradients in step 3. are the same as the gradients in step 1. on the next iteration.

Hence, let us refine the function with respect to this.

Now, we use old_gradients in step 1. and return the updated gradients such that it can be given as old gradients in the next call.

Now, we ad the loop body.

Before we start, there are a few notes on how Tensorflow works internally that might be helpfup in understanding why things are done the way they are.

Tensorflow internally represents all operations as nodes in a so-called computational graph. Edges between nodes tell tensorflow which operations' output is required as input to other operations, e.g. the ones requested to evaluate. For example, in order to evaluate the loss, it first needs to look at the dataset and also all weights and biases. Any variable is also represented as a node in the graph.

All actual data is stored in a so-called session object. Evaluations of nodes in the computational graph are done by giving a list of nodes to the ‘run()` function of this session object. This function usually requires a so-called feed dict, a dictionary containing any external values that are referenced through placeholder nodes. When evaluating nodes, only that part of the feed dict needs to be given that is required for the nodes’ evaluation. E.g. when assigning values to parameters through an assign operation using placeholders, we do not need to specify the dataset

This has been very brief and for a more in-depth view into the design of Tensorflow, we refer to the programmer’s guide or to the official tensorflow tutorials.

TATi has quite a number of options that control its behavior with respect to sampling, optimizing and so on. We briefly remind of how you can request help to a specific option. Let us inspect the help for batch_data_files:

This will print a description, give the default value and expected type.

Moreover, in case you have forgotten the name of one of the options.

This will print a general help listing all available options.

Let’s first look at how to set up the neural network and supply it with a dataset.

Next, we look at how to inspect and modify the neural network’s parameters.

Now, we are in the position to evaluate our neural network. Or neural networks, in case you specified number_walkers larger than 1, see Setting parameters of each walker.

Last but not least, how to change the dataset when it was already specified?

We conclude this section with some general advice on resolving performance issues:

For longer simulation runs it is desirable to obtain an estimate after a few steps of the time required for the entire run.

This is possible using the progress option. Specified to 1 or True it will produce a progress bar showing the total number of steps, the iterations per second, the elapsed time since start and the estimated time till finish.

This features requires the tqdm package.

Tensorflow delivers a powerful instrument for inspecting the inner workings of its computational graph: TensorBoard.

This tool allows also to inspect values such as the activation histogram, the loss and accuracy and many other parameters and values internal to TATi.

Supplying a path /foo/bar present in the file system using the summaries_path variable, summaries are automatically written to the path and can be inspected with the following call to tensorboard.

The tensorboard essentially comprises a web server for rendering the nodes of the graph and figures of the inspected values inside a web page. On execution it provides a URL that needs to be entered in any web browser to access the web page.