Models and Equations

Below we’ll give a brief (really very brief!) intro to deep learning, primarily to introduce the notation. In addition we’ll discuss some model equations below. Note that we’ll avoid using model to denote trained neural networks, in contrast to some other texts and APIs. These will be called “NNs” or “networks”. A “model” will typically denote a set of model equations for a physical effect, usually PDEs.

Deep learning and neural networks

In this book we focus on the connection with physical models, and there are lots of great introductions to deep learning. Hence, we’ll keep it short: the goal in deep learning is to approximate an unknown function

(1)\[ f^*(x) = y^* , \]

where \(y^*\) denotes reference or “ground truth” solutions. \(f^*(x)\) should be approximated with an NN representation \(f(x;\theta)\). We typically determine \(f\) with the help of some variant of a loss function \(L(y,y^*)\), where \(y=f(x;\theta)\) is the output of the NN. This gives a minimization problem to find \(f(x;\theta)\) such that \(e\) is minimized. In the simplest case, we can use an \(L^2\) error, giving

(2)\[ \text{arg min}_{\theta} | f(x;\theta) - y^* |_2^2 . \]

We typically optimize, i.e. train, with a stochastic gradient descent (SGD) optimizer of choice, e.g. Adam [KB14]. We’ll rely on auto-diff to compute the gradient of a scalar loss \(L\) w.r.t. the weights, \(\partial L / \partial \theta\). It is crucial for the calculation of gradients that this function is scalar, and the loss function is often also called “error”, “cost”, or “objective” function.

For training we distinguish: the training data set drawn from some distribution, the validation set (from the same distribution, but different data), and test data sets with some different distribution than the training one. The latter distinction is important. For the test set we want out of distribution (OOD) data to check how well our trained model generalizes. Note that this gives a huge range of possibilities for the test data set: from tiny changes that will certainly work, up to completely different inputs that are essentially guaranteed to fail. There’s no gold standard, but test data should be generated with care.

Enough for now - if all the above wasn’t totally obvious for you, we very strongly recommend to read chapters 6 to 9 of the Deep Learning book, especially the sections about MLPs and “Conv-Nets”, i.e. CNNs.

Note

Classification vs Regression

The classic ML distinction between classification and regression problems is not so important here: we only deal with regression problems in the following.

Partial differential equations as physical models

The following section will give a brief outlook for the model equations we’ll be using later on in the DL examples. We typically target continuous PDEs denoted by \(\mathcal P^*\) whose solution is of interest in a spatial domain \(\Omega \subset \mathbb{R}^d\) in \(d \in {1,2,3} \) dimensions. In addition, we often consider a time evolution for a finite time interval \(t \in \mathbb{R}^{+}\). The corresponding fields are either d-dimensional vector fields, for instance \(\mathbf{u}: \mathbb{R}^d \times \mathbb{R}^{+} \rightarrow \mathbb{R}^d\), or scalar \(\mathbf{p}: \mathbb{R}^d \times \mathbb{R}^{+} \rightarrow \mathbb{R}\). The components of a vector are typically denoted by \(x,y,z\) subscripts, i.e., \(\mathbf{v} = (v_x, v_y, v_z)^T\) for \(d=3\), while positions are denoted by \(\mathbf{p} \in \Omega\).

To obtain unique solutions for \(\mathcal P^*\) we need to specify suitable initial conditions, typically for all quantities of interest at \(t=0\), and boundary conditions for the boundary of \(\Omega\), denoted by \(\Gamma\) in the following.

\(\mathcal P^*\) denotes a continuous formulation, where we make mild assumptions about its continuity, we will typically assume that first and second derivatives exist.

We can then use numerical methods to obtain approximations of a smooth function such as \(\mathcal P^*\) via discretization. These invariably introduce discretization errors, which we’d like to keep as small as possible. These errors can be measured in terms of the deviation from the exact analytical solution, and for discrete simulations of PDEs, they are typically expressed as a function of the truncation error \(O( \Delta x^k )\), where \(\Delta x\) denotes the spatial step size of the discretization. Likewise, we typically have a temporal discretization via a time step \(\Delta t\).

Notation and abbreviations

If unsure, please check the summary of our mathematical notation and the abbreviations used in: Notation and Abbreviations.

We solve a discretized PDE \(\mathcal{P}\) by performing steps of size \(\Delta t\). The solution can be expressed as a function of \(\mathbf{u}\) and its derivatives: \(\mathbf{u}(\mathbf{x},t+\Delta t) = \mathcal{P}( \mathbf{u}_{x}, \mathbf{u}_{xx}, ... \mathbf{u}_{xx...x} )\), where \(\mathbf{u}_{x}\) denotes the spatial derivatives \(\partial \mathbf{u}(\mathbf{x},t) / \partial \mathbf{x}\).

For all PDEs, we will assume non-dimensional parametrizations as outlined below, which could be re-scaled to real world quantities with suitable scaling factors. Next, we’ll give an overview of the model equations, before getting started with actual simulations and implementation examples on the next page.

---

Some example PDEs

The following PDEs are good examples, and we’ll use them later on in different settings to show how to incorporate them into DL approaches.

Burgers

We’ll often consider Burgers’ equation in 1D or 2D as a starting point. It represents a well-studied PDE, which (unlike Navier-Stokes) does not include any additional constraints such as conservation of mass. Hence, it leads to interesting shock formations. It contains an advection term (motion / transport) and a diffusion term (dissipation due to the second law of thermodynamics). In 2D, it is given by:

(3)\[\begin{split}\begin{aligned} \frac{\partial u_x}{\partial{t}} + \mathbf{u} \cdot \nabla u_x &= \nu \nabla\cdot \nabla u_x + g_x, \\ \frac{\partial u_y}{\partial{t}} + \mathbf{u} \cdot \nabla u_y &= \nu \nabla\cdot \nabla u_y + g_y \ , \end{aligned}\end{split}\]

where \(\nu\) and \(\mathbf{g}\) denote diffusion constant and external forces, respectively.

A simpler variant of Burgers’ equation in 1D without forces, denoting the single 1D velocity component as \(u = u_x\), is given by:

(4)\[ \frac{\partial u}{\partial{t}} + u \nabla u = \nu \nabla \cdot \nabla u \ . \]

Navier-Stokes

A good next step in terms of complexity is given by the Navier-Stokes equations, which are a well-established model for fluids. In addition to an equation for the conservation of momentum (similar to Burgers), they include an equation for the conservation of mass. This prevents the formation of shock waves, but introduces a new challenge for numerical methods in the form of a hard-constraint for divergence free motions.

In 2D, the Navier-Stokes equations without any external forces can be written as:

(5)\[\begin{split}\begin{aligned} \frac{\partial u_x}{\partial{t}} + \mathbf{u} \cdot \nabla u_x &= - \frac{1}{\rho}\nabla{p} + \nu \nabla\cdot \nabla u_x \\ \frac{\partial u_y}{\partial{t}} + \mathbf{u} \cdot \nabla u_y &= - \frac{1}{\rho}\nabla{p} + \nu \nabla\cdot \nabla u_y \\ \text{subject to} \quad \nabla \cdot \mathbf{u} &= 0 \end{aligned}\end{split}\]

where, like before, \(\nu\) denotes a diffusion constant for viscosity.

An interesting variant is obtained by including the Boussinesq approximation for varying densities, e.g., for simple temperature changes of the fluid. With a marker field \(v\) that indicates regions of high temperature, it yields the following set of equations:

(6)\[\begin{split}\begin{aligned} \frac{\partial u_x}{\partial{t}} + \mathbf{u} \cdot \nabla u_x &= - \frac{1}{\rho} \nabla p \\ \frac{\partial u_y}{\partial{t}} + \mathbf{u} \cdot \nabla u_y &= - \frac{1}{\rho} \nabla p + \xi v \\ \text{subject to} \quad \nabla \cdot \mathbf{u} &= 0, \\ \frac{\partial v}{\partial{t}} + \mathbf{u} \cdot \nabla v &= 0 \end{aligned}\end{split}\]

where \(\xi\) denotes the strength of the buoyancy force.

And finally, the Navier-Stokes model in 3D give the following set of equations:

(7)\[\begin{split} \begin{aligned} \frac{\partial u_x}{\partial{t}} + \mathbf{u} \cdot \nabla u_x &= - \frac{1}{\rho} \nabla p + \nu \nabla\cdot \nabla u_x \\ \frac{\partial u_y}{\partial{t}} + \mathbf{u} \cdot \nabla u_y &= - \frac{1}{\rho} \nabla p + \nu \nabla\cdot \nabla u_y \\ \frac{\partial u_z}{\partial{t}} + \mathbf{u} \cdot \nabla u_z &= - \frac{1}{\rho} \nabla p + \nu \nabla\cdot \nabla u_z \\ \text{subject to} \quad \nabla \cdot \mathbf{u} &= 0. \end{aligned} \end{split}\]

Forward Simulations

Before we really start with learning methods, it’s important to cover the most basic variant of using the above model equations: a regular “forward” simulation, that starts from a set of initial conditions, and evolves the state of the system over time with a discretized version of the model equation. We’ll show how to run such forward simulations for Burgers’ equation in 1D and for a 2D Navier-Stokes simulation.