Support for AutoDiff-based optimization of coefficients for symbolic regression models with auto-regressive variables

[gkronber]

We aim to model dynamical systems using a set of differential equations for observed variables of the system. The proposed approach is to simulate the system forward in time using numerical integration (e.g. Runge-Kutta). If we want to combine this with our gradient-based optimization of coefficients then it is necessary to calculate gradients using AutoDiff over all steps of the numeric integration. For this it is probably necessary to extend the class for optimization of coefficients (aka ConstantsOptimizer) and to provide a symbol for auto-regressive variables where it is possible to set a gradient value.

E.g. for a uni-variate model: y'(t) = a * y(t)

we have:

0. y'(0) = a y(0)
1. y'(1) = a y(1) = a (y(0) + a y'(0)) = a y(0) + a²y'(0)

using a naive integration step

and:

dy'(0)/da = y(0)

which we need to calculate the dy'(1) / da

i.e.

dy'(1)/da = y(0) + 2 a dy'(0)/da = y(0) + 2 a y(0)

and so on for each time step.

[gkronber]

r15962: created branch for ticket

[gkronber]

r15964: first exploratory implementation of AutoDiff for modelling of dynamical systems

[gkronber]

r15968: #2925: added analysis method to generate results for trees and line charts. Added support for multiple target variables. Bugfixing. Added scaling to calculate NMSE as objective

[gkronber]

r15970: added expressions for latent variables and allow parameterization of the number of integration steps

[lkammere]

r16126: added constants values to tree in result view and minor bugfix when initializing problem.

[gkronber]

r16152: removed obsolete comment and a statement introduced in r16126

[gkronber]

r16153:

• added support for multiple training episodes,
• added simplification of models,
• fixed a bug in the normalization based on target variable variance

[gkronber]

r16154: bug fix for the extraction of target values (missed when introducing episodes)

[gkronber]

r16155: allow tuning individual parameter vectors for episodes (using the same structure)

[gkronber]

TODO: show line chart of integrated latent variables.

[gkronber]

r16214: added datasets from supplementary material for "distilling free-form laws ..." (oscillator, pendulum, double pendulum, ...)

[gkronber]

r16215 added sin and cos functions

[gkronber]

r16222

• added comments about parameter identification for differential equation models
• added source code of cvodes library (part of sundials) which provides functionality to calculate gradients for the parameters of partial differential equation models efficiently using the 'adjoint state method'.
• added compiled version of cvodes

[gkronber]

r16225: first working version of CVODES integration

[gkronber]

r16226: added vector access methods

[gkronber]

r16227: changed test case to linear oscillator

[gkronber]

r16245: worked on integration of CVODES library

[gkronber]

r16246: working example with forward sensitivity analysis

[gkronber]

r16248: extracted CVODES external methods into a separate class

[gkronber]

r16249: extracted Vector into a separate class

[gkronber]

r16250: preparations to switch between HL ODE solver and CVODES

[gkronber]

r16251: implemented interface to CVODES solver with forward sensitivity calculation.

[gkronber]

r16253: fixed memory leak and fixed bug in determination of integration time span

[gkronber]

r16254: made usage of x86 native library explicit.

[gkronber]

r15256: added x64 version of CVODES as well as dispatching code for x86 and x64.

[gkronber]

r16268: created a separate algorithm (similar to NLR but for parameter identification of ODE systems) for debugging of the parameter optimization and fixed a problem with the order of items in the checkedItemCollection for TargetVariables

[gkronber]

r16329: made several extensions in relation to blood glucose prediction

• added sqr function,
• added outputs for latent variables (linechart and model),
• added optimization of initial values for latent variables (for each episode separately)
• TODO: test with CVODES (so far only our own integration scheme has been tested)

[gkronber]

TODO: for testing: estimate parameters for model (including initial values for hidden variables) based on the immediately previous data points.

[gkronber]

r16386: merged changes r15972:16382 (HEAD) from trunk to branch

[gkronber]

r16395: removed unnecessary usings

[gkronber]

r16398: small changes

[gkronber]

r16399: solution class and solution view

[gkronber]

r16400: small gui improvement

[gkronber]

r16597: made some adaptations while debugging parameter identification for dynamical models

[gkronber]

r16599: changed code to use LM instead of LBFGS and removed standardization

[gkronber]

r16600: made some simplifications (Vector) to aid debugging.

[gkronber]

r16601: refactored dynamical modelling code

[gkronber]

r16602: write back optimized constants to trees

[gkronber]

r16603: scaling of residuals to target variance and update constant values directly in the tree

[gkronber]

r16604: efficiency improvements

[gkronber]

r16616:

• added crossover probability (important for multi-encoding in this case)
• re-added scaling of targets

[gkronber]

r16652: allow arbitrary number of arguments to +,-,*,/ operators, fixed bug in single-argument division

[gkronber]

r16653: added support for fixed numeric parameters as "constant variables"

[gkronber]

r16660: re-introduced partial support for latent variables

[gkronber]

r16662: merged all changes from trunk to branch (up to r16659)

[gkronber]

r16663: adapted to work with new persistence

[gkronber]

r16664: generate individual solutions for each of the trees to allow individual analysis of functions e.g. using PDP (only works without latent variables so far)

[gkronber]

r16665: fixed a problem which prevents loading persistence files

[gkronber]

r16785: fixed a bug in AQ evaluation

[gkronber]

r16786: fixed a problem with solutions for latent variables

[gkronber]

r16892: merged r16661:16890 from trunk to branch

[gkronber]

r16893: allow separate configuration of const opt steps for pre-tuning and the full ODE. Allow weighted combination of fitness using pretuning NMSE and the full ODE NMSE

[gkronber]

• r16951: added a separate result for the integrated error
• r16952: fixed plugin dependencies for execution on Hive
• r16953: made a fix in StorableProperties of SymExprGrammarBase which is necessary because of the new HEAL.Attic-based persistence.

[gkronber]

r16954:

• Add problem instance provider and instances.
• Use penalized regression splines for calculation of numeric differences (for pre-tuning).

[gkronber]

r17246: merged r17037:17242 from trunk to branch

[gkronber]

r17409: merged r17255:17402 from trunk to branch