Radial Basis Function Regression

[bwerth]

Radial Basis Functions are yet another type of regression model often used in surrogate modelling. A main benefit is their extensibility to non-continuous domains.

A Radial Basis Function Regression alongside suitable Kernel functions and Distance metrics shall be added to the existing DataAnalysis solution

[bwerth]

r14385 created branch for radial basis functions regression

[bwerth]

r14386 moved RadialBasisFunctions from Problems.SurrogateProblem to Algorithms.DataAnalysis

[gkronber]

r14500: fixed a small typo and added .sln file while reviewing

[gkronber]

I tested with the 'Chemical-I' problem instance and Euclidean norm with beta=2 (see attachment). The predictions seem to be nice on training and test set but the estimated variances and the approximated error for LOO CV seems to be way off. Could you please check whether the calculation of both are correct?

[gkronber]

Run with strange LOO and variance (Line chart 95% conf)

[gkronber]

r14869: merged changesets from trunk to branch

[gkronber]

r14870: merged changesets from trunk to branch

[gkronber]

Review comments:

• LINQ is used a lot in combination with matrix operations. This is often slow because of memory allocations required for enumerators. (DONE)
• There should be an option to scale the input variables (scaling should be active by default) (DONE)
• RBF regression does not support noise. If there are duplicate x vectors, model building fails. An option would be to add a diagonal matrix to the gram matrix (leading to kernel ridge regression?) (DONE)
• I have not found a source for the calculation of variance and LOO error (DONE, fixed LOO calculation based on GPML book)
• Don't know how to best unify covariance functions and kernel functions (there is some duplication) (DONE, only use RBF 'kernels' with a beta parameter).
• The calculation of the covariance matrix takes a lot of time (10x longer than the equivalent calculation when using an equivalent covariance matrix). I suspect that the reason is the rather general implementation for distance calculation. (DONE)
• Beta should be a parameter of the algorithm instead of the kernel to make it easier to run a grid test. (DONE)
• Multiple of the implemented kernels are only conditionally positive definite. See ​http://num.math.uni-goettingen.de/schaback/teaching/sc.pdf for a definition of the kernels and valid beta-values. Additionally, it is necessary to extend the basis functions for these kernels depending on the value of beta. (Handled, by trying to use Cholesky decompose first and if it fails trying to calculate the inverse via LU decomposition).

Made a number of changes in r14872 mainly refactoring RBF model.

[bwerth]

r14883 checked and reformulated gradient functions for kernels

[gkronber]

r14884: renamed folder. RBF regression can be seen as a special case of kernelized ridge regression.

[gkronber]

r14885: renamed and moved files

[gkronber]

r14887: worked on kernel ridge regression.

• moved beta parameter to algorithm.
• reintroduced IKernel interface to restrict choice of kernel in kernel ridge regression.
• speed-up by cholesky decomposition and
• optimization of the calculation of the covariance matrix.

[gkronber]

r14888: re-added calculation of leave one out cv estimate

[bwerth]

r14891 reworked kernel functions (beta is always a scaling factor now) added LU-Decomposition as a fall-back if Cholesky-decomposition fails

[gkronber]

r14892: made some adjustments after bwerth's changes (mainly formatting)

[gkronber]

r14911: copied kernel ridge regression from branch to trunk

[gkronber]

r14936: changed namespace for KernelRidgeRegression

[mkommend]

Review comments

General

• Region names HLConstructors & Boilerplate are superflous! Just constructors is fine (comment:23)
• If too much boilerplate code has to be written omit at least unnecessary code such as empty AfterDeserialization hooks. (comment:23)

Kernels

• IKernel: Changed events for Beta and Distance (comment:24)
• KernelBase: avoid instantiating Cloner objects manually, just use Clone() (comment:23)
• Check if Beta has a value before accessing it and throw meaningful exceptions.

TBC

[gkronber]

r15155: deleted branch

[gkronber]

r15156: removed superfluous region names and empty AfterDeserialization hooks

[gkronber]

r15157: added events for BetaChanged and DistanceChanged

[bwerth]

r15158 added checks for Beta without value and according exceptions

[mkommend]

Review comments

KernelRR Model

• Do not allow direct access to the allowedInputVariables array by the according property (comment:27)
• allowedInputVariables must be cloned, because it is modifiable by the mentioned property (comment:27)
• If nothing is cloned because it is not modified, then why is the kernel function cloned? (comment:27)
• Helper functions should be static (CreateScaling, ExtractData, BuildGramMatrix) (comment:27)
• Extract model learning from ctor (comment:27)

KernelRR

• Kernel should be a constrained value parameter (create experiment support)(comment:28)

Comments

• The following remarks needn't be addressed
• Storable properties should be used instead of serializing fields directly
• Empty lines improve the readability of the source
• Use OnlineCalculators

[bwerth]

r15164 KRRModel:

• made helper functions static;
• made contents of "allowedInputVariables" immutable;
• made constructor private and added public "Create"-method that does most of the learning now;

Kernels:

• fixed inconsistency in error messages

[bwerth]

r15169 made KernelParameter a ConstrainedValueParameter; added some description to BetaParameter

[mkommend]

Reviewed and tested r15164 and r15169.

[gkronber]

Reviewed the changes and tested everything. Looks nice! r15248: fixed typo

[gkronber]

r15249: merged r14862, r14863, r14911, r14936, r15156, r15157, r15158, r15164, r15169, r15207:15209, r15225, r15227, r15234, r15248 from trunk to stable

(combined merge of #2699 and #2700)