# Stochastic Dynamic Optimisation

## Aims

- To understand the foundations of probability theory.
- To obtain knowledge of the behaviour of martingales.
- To understand the theory of stochastic integration.
- To obtain knowledge of the behaviour of Brownian motion and Itô processes.
- To obtain knowledge of the HJB equation and its solution.

## Information

The scheme below is under proviso.

**Part I: Measure and probability**

In essence, this course studies stochastic processes, and thus requires knowledge of the modern approach to probability. The modern approach uses measure theory to define the probability space. Measure theory does not distinguish between discrete and continuous distributions, and also covers distributions which cannot be represented as (mixtures of) discrete and continuous distributions.

*Lecture 1: Spaces and σ-algebras*

metric spaces, completeness, Banach space, measurable space, Borel σ-algebra.

*Lecture 2: Measurable functions*

induced space, indicator and simple functions, limits, Lebesgue integral.

*Lecture 3: Probability spaces*

random elements, expectation, L1 space, L2 space, completeness of Lp space, independent events, independent random elements, zero-one laws.

**Part II: Martingales**

In recent decades, martingale theory has been booming in probability theory. Martingales arise as the innovative parts of stochastic processes. Martingale theory has had a unifying influence: where previously different sorts of stochastic processes, such as Markov processes, were considered independently of each other, now these can be treated from the point of view of martingale theory as well. This makes martingale theory a good basis for further techniques.

*Lecture 4: Conditional expectation*

types of conditional expectation, conditional expectation in L2, conditional expectation in L1, properties of conditional expectation.

*Lecture 5: Martingales, optional stopping, martingale inequalities*

definition, stopping times, stopped processes, optional stopping, Doob’s inequality, upcrossing inequality.

*Lecture 6: Martingale convergence, uniform integrability*

forward convergence, reversed martingale, uniform integrability, UI martingales, UI reversed martingales.

**Part III: Stochastic integration**

A semimartingale can be decomposed as the sum of an innovation part (a local martingale) and a predictable part (a well-behaved stochastic process). Semimartingales form the largest class of stochastic processes with respect to which Itô integrals can be defined. The theory of stochastic integration is specialized to Brownian motion and related processes.

*Lecture 7: Decomposition*

local martingales, Doob-Meyer decomposition, predictable (co)variation, semimartingales.

*Lecture 8: The Itô integral*

quadratic (co)variation, one-dimensional Itô formula, Doléans-Dade exponential, multi-dimensional itô formula.

*Lecture 9: Brownian motion and Itô processes*

existence, properties, paths, martingale representation, stochastic differentials, Itô diffusions, Itô processes.

**Part IV: Stochastic dynamic programming**

Stochastic integration allows the extension of the deterministic Hamilton-Jacobi-Bellman (HJB) equation so as to include semimartingales. Optimal control of a semimartingale boils down to solving the stochastic HJB equation. Some applications are discussed.

*Lecture 10: HJB equation*

partial differential equation approach: deterministic and stochastic; maximum principle: deterministic case;

*Lecture 11: Viscosity solutions*

equivalence of HJB equation and maximum principle: deterministic and stochastic; martingale and convex duality methods;

*Lecture 12: Applications*

production planning and inventory; optimal exploitation of renewable sources

### Assessment

Oral examination, in which the participant is asked to discuss a relevant paper in his/her own field. The participant selects the paper, its relevancy needs to be approved by the lecturers before the examination.

The timetable for this course can be found in the EUR course guide.

ERIM PhD candidates and Research Master students can register for this course via Osiris Student.

External (non-ERIM) participants are welcome to this course. To register, please fill in the registration form and e-mail it to the ERIM Doctoral Office by four weeks prior to the start of the course. For external participants, the course fee is 260 euro per ECTS credit.

### Materials

D.Williams, Probability with Martingales, Cambridge University Press, 2007. ISBN 0-521-40605-6.

- this text is used in parts I and II.

Morimoto, Hiroaki. Stochastic control and mathematical modeling. Applications in economics. Encyclopedia of Mathematics and its Applications 131. Cambridge University Press, Cambridge, 2010. ISBN 978-0-521-19503-4.

- this text is used in parts III and IV, and in particular contains the applications discussed in Lecture 12.

Pham, Huyên. Continuous-time stochastic control and optimization with financial applications. Stochastic Modelling and Applied Probability 61. Springer-Verlag, Berlin, 2009. ISBN 978-3-540-89499-5.

- this text is a less expensive alternative to Morimoto (2010), but does not cover the applications of Lecture 12.

### Additional info

Prerequisites include some familiarity with basic concepts from calculus and probability theory such as set theory, limits, differentiation and integration, expectation/mean and probability distributions ( in particular the normal/Gaussian distribution).