Opened 11 months ago

Closed 3 months ago

#2699 closed feature request (done)

Radial Basis Function Regression

Reported by: bwerth Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.15
Component: Algorithms.DataAnalysis Version: 3.3.14
Keywords: Cc:

Description

Radial Basis Functions are yet another type of regression model often used in surrogate modelling. A main benefit is their extensibility to non-continuous domains.

A Radial Basis Function Regression alongside suitable Kernel functions and Distance metrics shall be added to the existing DataAnalysis solution

Attachments (1)

Radial Basis Function Regression (RBF-R).hl (4.8 MB) - added by gkronber 10 months ago.
Run with strange LOO and variance (Line chart 95% conf)

Change History (35)

comment:1 Changed 11 months ago by bwerth

  • Owner set to bwerth
  • Status changed from new to assigned

comment:2 Changed 11 months ago by bwerth

  • Status changed from assigned to accepted
  • Version changed from 3.3.14 to branch

comment:3 Changed 11 months ago by bwerth

r14385 created branch for radial basis functions regression

comment:4 Changed 11 months ago by bwerth

r14386 moved RadialBasisFunctions from Problems.SurrogateProblem to Algorithms.DataAnalysis

comment:5 Changed 10 months ago by gkronber

r14500: fixed a small typo and added .sln file while reviewing

comment:6 Changed 10 months ago by gkronber

I tested with the 'Chemical-I' problem instance and Euclidean norm with beta=2 (see attachment). The predictions seem to be nice on training and test set but the estimated variances and the approximated error for LOO CV seems to be way off. Could you please check whether the calculation of both are correct?

comment:7 Changed 10 months ago by gkronber

  • Status changed from accepted to assigned

Changed 10 months ago by gkronber

Run with strange LOO and variance (Line chart 95% conf)

comment:8 Changed 6 months ago by gkronber

r14869: merged changesets from trunk to branch

comment:9 Changed 6 months ago by gkronber

r14870: merged changesets from trunk to branch

comment:10 Changed 6 months ago by gkronber

Review comments:

  • LINQ is used a lot in combination with matrix operations. This is often slow because of memory allocations required for enumerators. (DONE)
  • There should be an option to scale the input variables (scaling should be active by default) (DONE)
  • RBF regression does not support noise. If there are duplicate x vectors, model building fails. An option would be to add a diagonal matrix to the gram matrix (leading to kernel ridge regression?) (DONE)
  • I have not found a source for the calculation of variance and LOO error (DONE, fixed LOO calculation based on GPML book)
  • Don't know how to best unify covariance functions and kernel functions (there is some duplication) (DONE, only use RBF 'kernels' with a beta parameter).
  • The calculation of the covariance matrix takes a lot of time (10x longer than the equivalent calculation when using an equivalent covariance matrix). I suspect that the reason is the rather general implementation for distance calculation. (DONE)
  • Beta should be a parameter of the algorithm instead of the kernel to make it easier to run a grid test. (DONE)
  • Multiple of the implemented kernels are only conditionally positive definite. See http://num.math.uni-goettingen.de/schaback/teaching/sc.pdf for a definition of the kernels and valid beta-values. Additionally, it is necessary to extend the basis functions for these kernels depending on the value of beta. (Handled, by trying to use Cholesky decompose first and if it fails trying to calculate the inverse via LU decomposition).

Made a number of changes in r14872 mainly refactoring RBF model.

Last edited 6 months ago by gkronber (previous) (diff)

comment:11 Changed 6 months ago by bwerth

r14883 checked and reformulated gradient functions for kernels

comment:12 Changed 6 months ago by gkronber

r14884: renamed folder. RBF regression can be seen as a special case of kernelized ridge regression.

comment:13 Changed 6 months ago by gkronber

r14885: renamed and moved files

comment:14 Changed 6 months ago by gkronber

r14887: worked on kernel ridge regression.

  • moved beta parameter to algorithm.
  • reintroduced IKernel interface to restrict choice of kernel in kernel ridge regression.
  • speed-up by cholesky decomposition and
  • optimization of the calculation of the covariance matrix.

comment:15 Changed 6 months ago by gkronber

  • Status changed from assigned to reviewing

r14888: re-added calculation of leave one out cv estimate

comment:16 Changed 6 months ago by bwerth

r14891 reworked kernel functions (beta is always a scaling factor now) added LU-Decomposition as a fall-back if Cholesky-decomposition fails

Last edited 6 months ago by bwerth (previous) (diff)

comment:17 Changed 6 months ago by gkronber

r14892: made some adjustments after bwerth's changes (mainly formatting)

comment:18 Changed 6 months ago by gkronber

r14911: copied kernel ridge regression from branch to trunk

comment:19 Changed 5 months ago by gkronber

r14936: changed namespace for KernelRidgeRegression

comment:20 Changed 5 months ago by gkronber

  • Owner changed from bwerth to mkommend
  • Version changed from branch to 3.3.14

comment:21 follow-up: Changed 4 months ago by mkommend

Review comments

General

  • Region names HLConstructors & Boilerplate are superflous! Just constructors is fine (comment:23)
  • If too much boilerplate code has to be written omit at least unnecessary code such as empty AfterDeserialization hooks. (comment:23)

Kernels

  • IKernel: Changed events for Beta and Distance (comment:24)
  • KernelBase: avoid instantiating Cloner objects manually, just use Clone() (comment:23)
  • Check if Beta has a value before accessing it and throw meaningful exceptions. (comment:25)

TBC

Last edited 3 months ago by bwerth (previous) (diff)

comment:22 Changed 3 months ago by gkronber

r15155: deleted branch

comment:23 Changed 3 months ago by gkronber

r15156: removed superfluous region names and empty AfterDeserialization hooks

comment:24 in reply to: ↑ 21 Changed 3 months ago by gkronber

r15157: added events for BetaChanged and DistanceChanged

comment:25 Changed 3 months ago by bwerth

r15158 added checks for Beta without value and according exceptions

comment:26 Changed 3 months ago by mkommend

  • Owner changed from mkommend to bwerth
  • Status changed from reviewing to assigned

Review comments

KernelRR Model

  • Do not allow direct access to the allowedInputVariables array by the according property (comment:27)
  • allowedInputVariables must be cloned, because it is modifiable by the mentioned property (comment:27)
  • If nothing is cloned because it is not modified, then why is the kernel function cloned? (comment:27)
  • Helper functions should be static (CreateScaling, ExtractData, BuildGramMatrix) (comment:27)
  • Extract model learning from ctor (comment:27)

KernelRR

  • Kernel should be a constrained value parameter (create experiment support)(comment:28)

Comments

  • The following remarks needn't be addressed
  • Storable properties should be used instead of serializing fields directly
  • Empty lines improve the readability of the source
  • Use OnlineCalculators
Last edited 3 months ago by bwerth (previous) (diff)

comment:27 Changed 3 months ago by bwerth

r15164 KRRModel:

  • made helper functions static;
  • made contents of "allowedInputVariables" immutable;
  • made constructor private and added public "Create"-method that does most of the learning now;

Kernels:

  • fixed inconsistency in error messages

comment:28 Changed 3 months ago by bwerth

r15169 made KernelParameter a ConstrainedValueParameter; added some description to BetaParameter

comment:29 Changed 3 months ago by bwerth

  • Owner changed from bwerth to mkommend
  • Status changed from assigned to reviewing

comment:30 Changed 3 months ago by mkommend

  • Status changed from reviewing to readytorelease

Reviewed and tested r15164 and r15169.

comment:31 Changed 3 months ago by gkronber

  • Owner changed from mkommend to gkronber
  • Status changed from readytorelease to reviewing

comment:32 Changed 3 months ago by gkronber

  • Status changed from reviewing to readytorelease

Reviewed the changes and tested everything. Looks nice! r15248: fixed typo

comment:33 Changed 3 months ago by gkronber

r15249: merged r14862, r14863, r14911, r14936, r15156, r15157, r15158, r15164, r15169, r15207:15209, r15225, r15227, r15234, r15248 from trunk to stable

(combined merge of #2699 and #2700)

Last edited 3 months ago by gkronber (previous) (diff)

comment:34 Changed 3 months ago by gkronber

  • Resolution set to done
  • Status changed from readytorelease to closed
Note: See TracTickets for help on using tickets.