# Getting started¶

## Quick links¶

Source repository | Contributing and feedback | PyPI | Documentation | Tutorials | Scientific literature

### What is *datafold*?¶

*datafold* is a MIT-licensed
Python package containing operator-theoretic, data-driven models to identify dynamical
systems from time series data and to infer geometrical structures in point clouds.

The package includes:

Data structures to handle point clouds on manifolds (

`PCManifold`

) and time series collections (`TSCDataFrame`

). The data structures are used both internally and for model input/outputs. In contrast to solutions found in other projects, such as lists of Numpy arrays,`TSCDataFrame`

makes it much easier to describe many forms of time series data in a single object.An efficient implementation of the

`DiffusionMaps`

model to infer geometric meaningful structures from data, such as the eigenfunctions of the Laplace-Beltrami operator. As a distinguishing factor to other implementations, the model can handle a sparse kernel matrix and allows setting an arbitrary kernel, including the standard Gaussian kernel, continuous k-nearest neighbor kernel, or dynamics-adapted cone kernel.Out-of-sample extensions for the Diffusion Maps model, such as the (auto-tuned) Laplacian Pyramids or Geometric Harmonics to interpolate general function values on a point cloud manifold.

An implementation of the (Extended-) Dynamic Mode Decomposition (e.g. model

`DMDFull`

or`EDMD`

) as data-driven methods to identify dynamical systems from time series collection data.`EDMD`

subclasses from the flexible scikit-learn Pipeline, which allows setting up and transforming time series collection data to a more suitable feature state (cf. Koopman operator theory).`EDMDCV`

allows model parameters to be optimized with cross-validation splittings that account for the temporal order in time series collections.

See also this introduction page. For a mathematical thorough introduction, we refer to the scientific literature.

Note

The project is under active development in a research-driven environment.

Code quality varies from “experimental/early stage” to “well-tested”. Well tested code is listed in the software documentation and are directly accessible through the package levels

`pcfold`

,`dynfold`

or`appfold`

(e.g.`from datafold.dynfold import ...`

). Experimental code is only accessible via “deep imports” (e.g.`from datafol.dynfold.outofsample import ...`

) and may raise a warning when using it.There is no deprecation cycle. Backwards compatibility is indicated by the package version, where we use a semantic versioning policy [major].[minor].[patch], i.e.

major - making incompatible changes in the (documented) API

minor - adding functionality in a backwards-compatible manner

patch - backwards-compatible bug fixes

We do not intend to indicate a feature complete milestone with version 1.0.

### Cite¶

If you use *datafold* in your research, please cite
this paper published in the
*Journal of Open Source Software* (JOSS).

*Lehmberg et al., (2020). datafold: data-driven models for point clouds and time series on
manifolds. Journal of Open Source Software, 5(51), 2283,* https://doi.org/10.21105/joss.02283

BibTeX:

```
@article{Lehmberg2020,
doi = {10.21105/joss.02283},
url = {https://doi.org/10.21105/joss.02283},
year = {2020},
publisher = {The Open Journal},
volume = {5},
number = {51},
pages = {2283},
author = {Daniel Lehmberg and Felix Dietrich and Gerta K{\"o}ster and Hans-Joachim Bungartz},
title = {datafold: data-driven models for point clouds and time series on manifolds},
journal = {Journal of Open Source Software}}
```

### How to get it?¶

Installation requires Python>=3.7 with
pip and
setuptools installed. Both
packages usually ship with a standard Python installation. The package dependencies
install automatically. The main dependencies and their role in *datafold* are listed below
in “Dependencies”.

There are two ways to install *datafold*:

#### 1. From PyPI¶

This is the standard way for users. The package is hosted on the official Python package index (PyPI) and installs the core package (excluding tutorials and tests). The tutorial files can be downloaded separately here.

To install the package and its dependencies with `pip`

, run

```
python -m pip install datafold
```

Note

If you run Python in an Anaconda environment you can use pip from within `conda`

.
See also
official instructions.

```
conda activate venv
conda install pip
pip install datafold
```

#### 2. From source¶

This way is recommended if you want to access the latest (but potentially unstable) development, run tests or wish to contribute (see section “Contributing” for details). Download or git-clone the source code repository.

Download the repository

Install the package from the downloaded repository

python -m pip install .

### Contributing¶

Any contribution (code/tutorials/documentation improvements), question or feedback is
very welcome. Either use the
issue tracker or
Email.
Instructions to set up *datafold* for development can be found
here.

### Dependencies¶

The dependencies of the core package are managed in the file
requirements.txt
and install with *datafold*. The tests, tutorials, documentation and code analysis
require additional dependencies which are managed in
requirements-dev.txt.

*datafold* integrates with common packages from the
Python scientific computing stack:

- pandas
*datafold*uses pandas’ DataFrame as a base class for`TSCDataFrame`

, which captures time series data and collections thereof. The data structure indexes time, time series ID and one-or-many spatial features. It includes specific time series collection functionality and is compatible with pandas rich functionality.

- scikit-learn
All

*datafold*algorithms that are part of the “machine learning pipeline” align to the scikit-learn API. This is done by deriving the models from BaseEstimator. and appropriate MixIns.*datafold*defines own MixIns that align with the API in a duck-typing fashion to allow identifying dynamical systems from temporal data in`TSCDataFrame`

.

- SciPy
The package is used for elementary numerical algorithms and data structures in conjunction with NumPy. This includes (sparse) linear least square regression, (sparse) eigenpairs solver and sparse matrices as optional data structure for kernel matrices.

### How does it compare to other software?¶

*The selection only includes other Python packages.*

- scikit-learn
provides algorithms and models along the entire machine learning pipeline, with a strong focus on static data (i.e. without temporal context).

*datafold*integrates into scikit-learn’ API and all data-driven models are subclasses of BaseEstimator. An important contribution of*datafold*is the`DiffusionMaps`

model as popular framework for manifold learning, which is not contained in scikit-learn’s set of algorithms. Furthermore,*datafold*includes dynamical systems as a new model class that is operable with scikit-learn - the attributes align to supervised learning tasks. The key differences are that a model processes data of type`TSCDataFrame`

and instead of a one-to-one relation in the model’s input/output, the model can return arbitrary many output samples (a time series) for a single input (an initial condition).

- PyDMD
provides many variants of the Dynamic Mode Decomposition (DMD).

*datafold*provides a wrapper to make models of`PyDMD`

accessible. However, a limitation of`PyDMD`

is that it only processes single coherent time series, see PyDMD issue 86. The DMD models that are directly included in*datafold*utilize the functionality of the data structure`TSCDataFrame`

and can therefore process time series collections - in an extreme case only containing snapshot pairs.

- PySINDy
specializes on a

*sparse*system identification of nonlinear dynamical systems to infer governing equations.