Tutorials#

This page contains tutorials and code snippets to showcase datafold’s API. All tutorials can be viewed online. If you want to execute the notebooks in Jupyter, please follow the instructions below in section “Run notebooks with Jupyter”.

List#

Download all tutorials in a zipped file.

Data structure for time series collections TSCDataFrame (download)
We introduce datafold’s basic data structures for time series collection data. The data structures are used in model implementations and for input/output specification of models.
Uniform subsampling of point cloud manifold (download)
We show how the PCManifold data structure can be used to subsample a manifold point cloud uniformly.

Warning

The tutorial generates a large dataset with 10 Mio. samples by default. This may have to be reduced, depending on the available computer memory.
Diffusion Maps: Embedding of an S-curved manifold (download)
We use the DiffusionMaps model to compute lower dimensional embeddings of an S-curved point cloud manifold. We also select the best combination of intrinsic parameters automatically with an optimization routine.

References: scikit-learn tutorial
Manifold learning on handwritten digits (download)
We use the DiffusionMaps model to cluster data from handwritten digits and highlight its out-of-sample capabilities. This example is taken from the scikit-learn package, so the the method can be compared against other common manifold learning algorithms.

References: scikit-learn tutorial
RObust and Scalable Embedding via LANdmark Diffusion (Roseland) (download)
We use a Roseland model to compute lower dimensional embeddings of an S-curved point cloud manifold and to cluster data from handwritten digit.

References: scikit-learn tutorial 1 | scikit-learn tutorial 2
Embeddings invariant to the observables with Mahalanobis kernel (download)
We highlight how to use the Mahalanobis kernel within DiffusionMaps. The key feature of this kernel is that it can yield embeddings that are invariant to the observation function.

Warning

The implementation of Mahalanobis kernel in datafold is still experimental and should be used with care. Contributions (mainly testing / documentation) are welcome!
Jointly Smooth Functions: An Example (download)
We use JointlySmoothFunctions to learn commonly smooth functions from multimodal data. We also demonstrate the out-of-sample capabilities of the method.
Geometric Harmonics: interpolate function values on data manifold (download)
We showcase the out-of-sample extension for manifold learning models such as the DiffusionMaps model. For this we use the GeometricHarmonicsInterpolator for forward and LaplacianPyramidsInterpolator for backwards interpolation respectively.
Extended Dynamic Mode Decomposition on Limit Cycle (download)
We generate data from the Hopf system (an ODE system) and compare different dictionaries of the Extended Dynamic Mode Decomposition (EDMD). We also showcase out-of-sample predictions with a time horizon that exceeds the sampled time series in the training.
Dynamic mode decomposition with control (download)
We introduce the dynamic mode decomposition with control. In this tutorial origins from the PyDMD package. Here we use it to compare the interface and highlight that the results are identical.

References: Original PyDMD tutorial
Model-based control using Extended Dynamic Mode Decomposition (download)
This tutorial demonstrates how to use extended dynamic mode decomposition (EDMD) and a linear quadratic regulator (LQR) for controlling the Van der Pol oscillator in a closed-loop. The goal is to show how EDMD can be an effective alternative for modeling and controlling non-linear dynamic systems.

References: Templated tutorial
Dictionary Learning for enhanced Koopman Operator approximations (download)
We demonstrate the EDMD-DL method, with which it is possible to learn the dictionary (i.e. set of observable functions) from the data. In the tutorial we use an feedforward artificial network and demonstrate the method for the Duffing system.

Warning

The implementation of EDMD-DL is still experimental and should be used with care. Contributions (testing / documentation / enhanced functionality) are welcome! The notebook also requires the Python package torch to be installed separately to the datafold’s dependencies

References: Li et al. [2017]
Koopman operator model predictive control for flow control (download)
We take the 1D Burger equation with periodic boundary conditions as an example to showcase how the Koopman operator can be utilized for model predictive control (MPC) in flow systems.

References: Original code (Matlab) | Arbabi et al. [2018]
Koopman operator-based model predictive control of a bilinear motor (download)
This tutorial will demonstrate how to utilize the Extended Dynamic Mode Decomposition (EDMD) to estimate the Koopman operator in controlled dynamical systems. The nonlinear behavior of a motor engine model will be transformed into a higher dimensional space, which will result in an approximately linear evolution. This will allow the use of EDMD as a linearly controlled dynamical system within the Koopman Model Predictive Control (KMPC) framework.

References: Original code (Matlab) | Korda and Mezić [2018] (Sect. 8.2)
An example to demonstrate online dynamic mode decomposition (download)
This tutorial showcases the online dynamic mode decomposition (OnlineDMD) at the example of a simple 2D time-varying system. The performance of the online DMD is compared with batch DMD and the analytical solution of the system. Following the online update scheme the model is updated once new data becomes available, which is particularly useful in time-varying systems.

References: Original demo | Zhang et al. [2019]

Run notebooks with Jupyter#

Download files#

If datafold was installed from PyPI, …

… the tutorials are not included in the package. Download the tutorials separately from the list.
If the datafold repository was downloaded, …

… then the tutorials are located path/to/repo/tutorials/. Before executing the tutorials, please make sure that datafold is either installed
```
python -m pip install .
```
or that path/to/datafold/ is included in the PYTHONPATH environment variable
```
export PYTHONPATH=$PYTHONPATH:/path/to/datafold/repository/
```

Start Jupyter#

All tutorials are Jupyter notebooks (.ipynb file ending). The Jupyter environment and its dependencies install with

python -m pip install jupyter

For further information visit the Jupyter homepage. To open a Jupyter notebook in a web browser, run

jupyter notebook path/to/datafold/tutorial_folder

or use the target in the Makefile (if the gir repository was downloaded):

make tutorial