Opened 12 months ago

Last modified 5 weeks ago

#2925 new feature request

Support for AutoDiff-based optimization of coefficients for symbolic regression models with auto-regressive variables

Reported by: gkronber Owned by: gkronber
Priority: medium Milestone: HeuristicLab 4.x Backlog
Component: Problems.DataAnalysis.Symbolic Version:
Keywords: Cc:

Description

We aim to model dynamical systems using a set of differential equations for observed variables of the system. The proposed approach is to simulate the system forward in time using numerical integration (e.g. Runge-Kutta). If we want to combine this with our gradient-based optimization of coefficients then it is necessary to calculate gradients using AutoDiff over all steps of the numeric integration. For this it is probably necessary to extend the class for optimization of coefficients (aka ConstantsOptimizer) and to provide a symbol for auto-regressive variables where it is possible to set a gradient value.

E.g. for a uni-variate model: y'(t) = a * y(t)

we have:

  1. y'(0) = a y(0)
  2. y'(1) = a y(1) = a (y(0) + a y'(0)) = a y(0) + a²y'(0)

using a naive integration step

and:

dy'(0)/da = y(0)

which we need to calculate the dy'(1) / da

i.e.

dy'(1)/da = y(0) + 2 a dy'(0)/da = y(0) + 2 a y(0)

and so on for each time step.

Change History (54)

comment:1 Changed 12 months ago by gkronber

r15962: created branch for ticket

comment:2 Changed 12 months ago by gkronber

r15964: first exploratory implementation of AutoDiff for modelling of dynamical systems

comment:3 Changed 12 months ago by gkronber

r15968: #2925: added analysis method to generate results for trees and line charts. Added support for multiple target variables. Bugfixing. Added scaling to calculate NMSE as objective

comment:4 Changed 12 months ago by gkronber

r15970: added expressions for latent variables and allow parameterization of the number of integration steps

comment:5 Changed 10 months ago by lkammere

r16126: added constants values to tree in result view and minor bugfix when initializing problem.

comment:6 Changed 9 months ago by gkronber

r16152: removed obsolete comment and a statement introduced in r16126

comment:7 Changed 9 months ago by gkronber

r16153:

  • added support for multiple training episodes,
  • added simplification of models,
  • fixed a bug in the normalization based on target variable variance

comment:8 Changed 9 months ago by gkronber

r16154: bug fix for the extraction of target values (missed when introducing episodes)

comment:9 Changed 9 months ago by gkronber

r16155: allow tuning individual parameter vectors for episodes (using the same structure)

comment:10 Changed 9 months ago by gkronber

TODO: show line chart of integrated latent variables.

comment:11 Changed 9 months ago by gkronber

r16214: added datasets from supplementary material for "distilling free-form laws ..." (oscillator, pendulum, double pendulum, ...)

comment:12 Changed 8 months ago by gkronber

r16215 added sin and cos functions

comment:13 Changed 8 months ago by gkronber

r16222

  • added comments about parameter identification for differential equation models
  • added source code of cvodes library (part of sundials) which provides functionality to calculate gradients for the parameters of partial differential equation models efficiently using the 'adjoint state method'.
  • added compiled version of cvodes

comment:14 Changed 8 months ago by gkronber

r16225: first working version of CVODES integration

comment:15 Changed 8 months ago by gkronber

r16226: added vector access methods

comment:16 Changed 8 months ago by gkronber

r16227: changed test case to linear oscillator

comment:17 Changed 8 months ago by gkronber

r16245: worked on integration of CVODES library

comment:18 Changed 8 months ago by gkronber

r16246: working example with forward sensitivity analysis

comment:19 Changed 8 months ago by gkronber

r16248: extracted CVODES external methods into a separate class

comment:20 Changed 8 months ago by gkronber

r16249: extracted Vector into a separate class

comment:21 Changed 8 months ago by gkronber

r16250: preparations to switch between HL ODE solver and CVODES

comment:22 Changed 8 months ago by gkronber

r16251: implemented interface to CVODES solver with forward sensitivity calculation.

comment:23 Changed 8 months ago by gkronber

r16253: fixed memory leak and fixed bug in determination of integration time span

comment:24 Changed 8 months ago by gkronber

r16254: made usage of x86 native library explicit.

comment:25 Changed 8 months ago by gkronber

r15256: added x64 version of CVODES as well as dispatching code for x86 and x64.

comment:26 Changed 8 months ago by gkronber

r16268: created a separate algorithm (similar to NLR but for parameter identification of ODE systems) for debugging of the parameter optimization and fixed a problem with the order of items in the checkedItemCollection for TargetVariables

comment:27 Changed 7 months ago by gkronber

r16329: made several extensions in relation to blood glucose prediction

  • added sqr function,
  • added outputs for latent variables (linechart and model),
  • added optimization of initial values for latent variables (for each episode separately)
  • TODO: test with CVODES (so far only our own integration scheme has been tested)

comment:28 Changed 7 months ago by gkronber

TODO: for testing: estimate parameters for model (including initial values for hidden variables) based on the immediately previous data points.

comment:29 Changed 6 months ago by gkronber

r16386: merged changes r15972:16382 (HEAD) from trunk to branch

comment:30 Changed 6 months ago by gkronber

r16395: removed unnecessary usings

comment:31 Changed 6 months ago by gkronber

r16398: small changes

comment:32 Changed 6 months ago by gkronber

r16399: solution class and solution view

comment:33 Changed 6 months ago by gkronber

r16400: small gui improvement

comment:34 Changed 4 months ago by gkronber

r16597: made some adaptations while debugging parameter identification for dynamical models

comment:35 Changed 4 months ago by gkronber

r16599: changed code to use LM instead of LBFGS and removed standardization

comment:36 Changed 4 months ago by gkronber

r16600: made some simplifications (Vector) to aid debugging.

comment:37 Changed 4 months ago by gkronber

r16601: refactored dynamical modelling code

comment:38 Changed 4 months ago by gkronber

r16602: write back optimized constants to trees

comment:39 Changed 4 months ago by gkronber

r16603: scaling of residuals to target variance and update constant values directly in the tree

comment:40 Changed 4 months ago by gkronber

r16604: efficiency improvements

comment:41 Changed 4 months ago by gkronber

r16616:

  • added crossover probability (important for multi-encoding in this case)
  • re-added scaling of targets

comment:42 Changed 4 months ago by gkronber

r16652: allow arbitrary number of arguments to +,-,*,/ operators, fixed bug in single-argument division

comment:43 Changed 3 months ago by gkronber

r16653: added support for fixed numeric parameters as "constant variables"

comment:44 Changed 3 months ago by gkronber

r16660: re-introduced partial support for latent variables

comment:45 Changed 3 months ago by gkronber

r16662: merged all changes from trunk to branch (up to r16659)

comment:46 Changed 3 months ago by gkronber

r16663: adapted to work with new persistence

comment:47 Changed 3 months ago by gkronber

r16664: generate individual solutions for each of the trees to allow individual analysis of functions e.g. using PDP (only works without latent variables so far)

comment:48 Changed 3 months ago by gkronber

r16665: fixed a problem which prevents loading persistence files

comment:49 Changed 2 months ago by gkronber

r16785: fixed a bug in AQ evaluation

comment:50 Changed 2 months ago by gkronber

r16786: fixed a problem with solutions for latent variables

comment:51 Changed 7 weeks ago by gkronber

r16892: merged r16661:16890 from trunk to branch

comment:52 Changed 7 weeks ago by gkronber

r16893: allow separate configuration of const opt steps for pre-tuning and the full ODE. Allow weighted combination of fitness using pretuning NMSE and the full ODE NMSE

comment:53 Changed 5 weeks ago by gkronber

  • r16951: added a separate result for the integrated error
  • r16952: fixed plugin dependencies for execution on Hive
  • r16953: made a fix in StorableProperties of SymExprGrammarBase which is necessary because of the new HEAL.Attic-based persistence.

comment:54 Changed 5 weeks ago by gkronber

r16954:

  • Add problem instance provider and instances.
  • Use penalized regression splines for calculation of numeric differences (for pre-tuning).
Note: See TracTickets for help on using tickets.