Bayesian Nonlinear Regression

[gkronber]

Currently, HL supports non-linear regression which uses the LM algorithm to fit parameters to a given non-linear model. The trained model can be used to calculate a predicted value for new inputs.

This is closely related to constants optimization in symbolic regression models, as we are using the same implementation for both.

It would be interesting to provide a feature for 'Bayesian non-linear regression'. The algorithm should determine a posterior distribution for the parameters given the data. The trained model can then be used to calculate a predictive distribution for new inputs.

It is often not possible to determine the posterior distribution analytically (depends on the structure of the model). An possible solution might be to use MCMC sampling to to estimate the posterior distribution. The samples from the posterior can than be used to calculate an estimated predictive distribution for each input.

Hamiltonian Monte Carlo seems to work well for similar tasks (see e.g. ​https://arxiv.org/pdf/1206.1901.pdf and ​http://mc-stan.org/)

[gkronber]

r15747: created a branch

[gkronber]

r15748: implemented a first version of Bayesian non-linear regression using HMC sampling

[gkronber]

r15749: normalized line endings

[gkronber]

r15750: added scaling by number of rows to make leapfrog integration independent from the number of rows