Fifteen Eighty Four

Author Carola-Bibiane Schönlieb discusses her book, Partial Differential Equation Methods for Image Inpainting.

How have automated PDEs changed the landscape of image ‘inpainting’?

When Marcelo Bertalmio et al has proposed his PDE inpainting method in 2000 he was one of the first ones to use powerful concepts from applied analysis to mimic the work of museum artists when physically restoring paintings. This new perspective in image inpainting opened up a completely new pool of mechanisms that has been preceded and proceded by a myriad of PDE models for image processing that turn concepts from experts in art conservation and in visual psychology,  in particular so-called Gestalt theory, into differential equations. This showed the power of mathematical modelling that is able to formalise strategies of vision experts into mathematical equations that decribe them as a combination of physical phenomena like transport and diffusion of colour in images. 

What sparked your interest in this field?

When I was a PhD student I started studying the Cahn-Hilliard equation, a fourth-order nonlinear diffusion equation that models phase separation and coarsening of binary alloys. At the time when I started my PhD a paper by Andrea Bertozzi et al came out which proposed the Cahn-Hilliard equation for the automated inpainting of binary images. I studied this equation for completely different aspects but got fascinated by this unexpected useage – described in Chapter 5 of the book. The beauty of finding physical phenomena in images and using them to define information in images has fascinated me ever since.

What are some of the latest applications for PDEs in imaging?

Partial differential equations in imaging are used in various contexts. The context of the book is using them to model structure in images and describes how to synthesise the modeled structure for filling in holes in images. As such, PDEs form a major part of image analysis and processing as images can be very well described by a few geometric features such as image edges (gradients of images), curvature (nonlinear second order content of images) and so forth, and their installation by a sequence of adaptive iterations which are modeled by PDEs. Beyond image inpainting, this has applications in the reconstruction of images from physical imaging devices such as MRT, where noisy and undersampled measurements are complemented by geometric features of instrinsic image structures to produce high resolution images of the inside of the human body, or in 3D stereovision where these structures are used to simulate 3D content from a series of 2D photographs, just to name two. On the other hand, in some applications, PDEs are crucial for describing the relationship between the measurements taken by a data collection device and physical quantities which are indirectly hidden in them. In seismic imaging, for example, images of internal structures below the earth are computed from measurements of pressure waves that are sent into earth, with the wave equation describing the propagation of these pressure waves. In an imaging technique called photo-acoustic tomography, the wave and the diffusion equation are used to connect applied optical laser energy to measured sound waves and the material hit along the way. More recently PDEs form a crucial part of non-local high dimensional data analysis where, for instance, they represent continuum models of discrete data clustering procedures such as spectral clustering or graph cut.

How is ‘Partial Differential Equation Methods for Image Inpainting’ different from similar books on the market and how can graduate students and researchers apply the concepts outlined in this book?

This book is the first one that fully covers the PDE inpainting literature from the last 20 years or so. It shows the power of applied mathematical modelling in the form of PDEs by focusing on the application of image inpainting. By providing exercises and MATLAB codes for the reader it makes the use of these equations practical and helps the reader to get a feeling of the described concepts by playing around with them on real examples.

Find out more about Carola-Bibiane Schönlieb’s new initiative CCIMI: Cantab Capital Institute for the Mathematics of Information