Models and Equations

Below we’ll give a brief (really very brief!) intro to deep learning, primarily to introduce the notation. In addition we’ll discuss some model equations below. Note that we’ll avoid using model to denote trained neural networks, in contrast to some other texts and APIs. These will be called “NNs” or “networks”. A “model” will typically denote a set of model equations for a physical effect, usually PDEs.

Deep learning and neural networks

In this book we focus on the connection with physical models, and there are lots of great introductions to deep learning. Hence, we’ll keep it short: the goal in deep learning is to approximate an unknown function

(1)\[ f^*(x) = y^* , \]

where \(y^*\) denotes reference or “ground truth” solutions. \(f^*(x)\) should be approximated with an NN representation \(f(x;\theta)\). We typically determine \(f\) with the help of some variant of an error function \(e(y,y^*)\), where \(y=f(x;\theta)\) is the output of the NN. This gives a minimization problem to find \(f(x;\theta)\) such that \(e\) is minimized. In the simplest case, we can use an \(L^2\) error, giving

(2)\[ \text{arg min}_{\theta} | f(x;\theta) - y^* |_2^2 \]

We typically optimize, i.e. train, with a stochastic gradient descent (SGD) optimizer of choice, e.g. Adam [KB14]. We’ll rely on auto-diff to compute the gradient w.r.t. weights, \(\partial f / \partial \theta\), We will also assume that \(e\) denotes a scalar error function (also called cost, or objective function). It is crucial for the efficient calculation of gradients that this function is scalar.

For training we distinguish: the training data set drawn from some distribution, the validation set (from the same distribution, but different data), and test data sets with some different distribution than the training one. The latter distinction is important. For the test set we want out of distribution (OOD) data to check how well our trained model generalizes. Note that this gives a huge range of possibilities for the test data set: from tiny changes that will certainly work, up to completely different inputs that are essentially guaranteed to fail. There’s no gold standard, but test data should be generated with care.

Enough for now - if all the above wasn’t totally obvious for you, we very strongly recommend to read chapters 6 to 9 of the Deep Learning book, especially the sections about MLPs and “Conv-Nets”, i.e. CNNs.

Note

Classification vs Regression

The classic ML distinction between classification and regression problems is not so important here: we only deal with regression problems in the following.

Partial differential equations as physical models

The following section will give a brief outlook for the model equations we’ll be using later on in the DL examples. We typically target continuous PDEs denoted by \(\mathcal P^*\) whose solution is of interest in a spatial domain \(\Omega \subset \mathbb{R}^d\) in \(d \in {1,2,3} \) dimensions. In addition, we often consider a time evolution for a finite time interval \(t \in \mathbb{R}^{+}\). The corresponding fields are either d-dimensional vector fields, for instance \(\mathbf{u}: \mathbb{R}^d \times \mathbb{R}^{+} \rightarrow \mathbb{R}^d\), or scalar \(\mathbf{p}: \mathbb{R}^d \times \mathbb{R}^{+} \rightarrow \mathbb{R}\). The components of a vector are typically denoted by \(x,y,z\) subscripts, i.e., \(\mathbf{v} = (v_x, v_y, v_z)^T\) for \(d=3\), while positions are denoted by \(\mathbf{x} \in \Omega\).

To obtain unique solutions for \(\mathcal P^*\) we need to specify suitable initial conditions, typically for all quantities of interest at \(t=0\), and boundary conditions for the boundary of \(\Omega\), denoted by \(\Gamma\) in the following.

\(\mathcal P^*\) denotes a continuous formulation, where we make mild assumptions about its continuity, we will typically assume that first and second derivatives exist.

We can then use numerical methods to obtain approximations of a smooth function such as \(\mathcal P^*\) via discretization. These invariably introduce discretization errors, which we’d like to keep as small as possible. These errors can be measured in terms of the deviation from the exact analytical solution, and for discrete simulations of PDEs, they are typically expressed as a function of the truncation error \(O( \Delta x^k )\), where \(\Delta x\) denotes the spatial step size of the discretization. Likewise, we typically have a temporal discretization via a time step \(\Delta t\).

Notation and abbreviations

If unsure, please check the summary of our mathematical notation and the abbreviations used in: Notation and Abbreviations.

We solve a discretized PDE \(\mathcal{P}\) by performing steps of size \(\Delta t\). The solution can be expressed as a function of \(\mathbf{u}\) and its derivatives: \(\mathbf{u}(\mathbf{x},t+\Delta t) = \mathcal{P}( \mathbf{u}_{x}, \mathbf{u}_{xx}, ... \mathbf{u}_{xx...x} )\), where \(\mathbf{u}_{x}\) denotes the spatial derivatives \(\partial \mathbf{u}(\mathbf{x},t) / \partial \mathbf{x}\).

For all PDEs, we will assume non-dimensional parametrizations as outlined below, which could be re-scaled to real world quantities with suitable scaling factors. Next, we’ll give an overview of the model equations, before getting started with actual simulations and implementation examples on the next page.


Some example PDEs

The following PDEs are good examples, and we’ll use them later on in different settings to show how to incorporate them into DL approaches.

Burgers

We’ll often consider Burgers’ equation in 1D or 2D as a starting point. It represents a well-studied PDE, which (unlike Navier-Stokes) does not include any additional constraints such as conservation of mass. Hence, it leads to interesting shock formations. It contains an advection term (motion / transport) and a diffusion term (dissipation due to the second law of thermodynamics). In 2D, it is given by:

(3)\[\begin{split}\begin{aligned} \frac{\partial u_x}{\partial{t}} + \mathbf{u} \cdot \nabla u_x &= \nu \nabla\cdot \nabla u_x + g_x, \\ \frac{\partial u_y}{\partial{t}} + \mathbf{u} \cdot \nabla u_y &= \nu \nabla\cdot \nabla u_y + g_y \ , \end{aligned}\end{split}\]

where \(\nu\) and \(\mathbf{g}\) denote diffusion constant and external forces, respectively.

A simpler variant of Burgers’ equation in 1D without forces, denoting the single 1D velocity component as \(u = u_x\), is given by:

(4)\[ \frac{\partial u}{\partial{t}} + u \nabla u = \nu \nabla \cdot \nabla u \ . \]

Forward Simulations

Before we really start with learning methods, it’s important to cover the most basic variant of using the above model equations: a regular “forward” simulation, that starts from a set of initial conditions, and evolves the state of the system over time with a discretized version of the model equation. We’ll show how to run such forward simulations for Burgers’ equation in 1D and for a 2D Navier-Stokes simulation.