Opened 4 years ago
Closed 3 years ago
#2699 closed feature request (done)
Radial Basis Function Regression
Reported by: | bwerth | Owned by: | gkronber |
---|---|---|---|
Priority: | medium | Milestone: | HeuristicLab 3.3.15 |
Component: | Algorithms.DataAnalysis | Version: | 3.3.14 |
Keywords: | Cc: |
Description
Radial Basis Functions are yet another type of regression model often used in surrogate modelling. A main benefit is their extensibility to non-continuous domains.
A Radial Basis Function Regression alongside suitable Kernel functions and Distance metrics shall be added to the existing DataAnalysis solution
Attachments (1)
Change History (35)
comment:1 Changed 4 years ago by bwerth
- Owner set to bwerth
- Status changed from new to assigned
comment:2 Changed 4 years ago by bwerth
- Status changed from assigned to accepted
- Version changed from 3.3.14 to branch
comment:3 Changed 4 years ago by bwerth
comment:4 Changed 4 years ago by bwerth
r14386 moved RadialBasisFunctions from Problems.SurrogateProblem to Algorithms.DataAnalysis
comment:5 Changed 4 years ago by gkronber
r14500: fixed a small typo and added .sln file while reviewing
comment:6 Changed 4 years ago by gkronber
I tested with the 'Chemical-I' problem instance and Euclidean norm with beta=2 (see attachment). The predictions seem to be nice on training and test set but the estimated variances and the approximated error for LOO CV seems to be way off. Could you please check whether the calculation of both are correct?
comment:7 Changed 4 years ago by gkronber
- Status changed from accepted to assigned
comment:8 Changed 3 years ago by gkronber
r14869: merged changesets from trunk to branch
comment:9 Changed 3 years ago by gkronber
r14870: merged changesets from trunk to branch
comment:10 Changed 3 years ago by gkronber
Review comments:
LINQ is used a lot in combination with matrix operations. This is often slow because of memory allocations required for enumerators.(DONE)There should be an option to scale the input variables (scaling should be active by default)(DONE)RBF regression does not support noise. If there are duplicate x vectors, model building fails. An option would be to add a diagonal matrix to the gram matrix (leading to kernel ridge regression?)(DONE)I have not found a source for the calculation of variance and LOO error(DONE, fixed LOO calculation based on GPML book)Don't know how to best unify covariance functions and kernel functions (there is some duplication)(DONE, only use RBF 'kernels' with a beta parameter).The calculation of the covariance matrix takes a lot of time (10x longer than the equivalent calculation when using an equivalent covariance matrix). I suspect that the reason is the rather general implementation for distance calculation.(DONE)Beta should be a parameter of the algorithm instead of the kernel to make it easier to run a grid test.(DONE)Multiple of the implemented kernels are only conditionally positive definite. See http://num.math.uni-goettingen.de/schaback/teaching/sc.pdf for a definition of the kernels and valid beta-values. Additionally, it is necessary to extend the basis functions for these kernels depending on the value of beta.(Handled, by trying to use Cholesky decompose first and if it fails trying to calculate the inverse via LU decomposition).
Made a number of changes in r14872 mainly refactoring RBF model.
comment:11 Changed 3 years ago by bwerth
r14883 checked and reformulated gradient functions for kernels
comment:12 Changed 3 years ago by gkronber
r14884: renamed folder. RBF regression can be seen as a special case of kernelized ridge regression.
comment:13 Changed 3 years ago by gkronber
r14885: renamed and moved files
comment:14 Changed 3 years ago by gkronber
r14887: worked on kernel ridge regression.
- moved beta parameter to algorithm.
- reintroduced IKernel interface to restrict choice of kernel in kernel ridge regression.
- speed-up by cholesky decomposition and
- optimization of the calculation of the covariance matrix.
comment:15 Changed 3 years ago by gkronber
- Status changed from assigned to reviewing
r14888: re-added calculation of leave one out cv estimate
comment:16 Changed 3 years ago by bwerth
r14891 reworked kernel functions (beta is always a scaling factor now) added LU-Decomposition as a fall-back if Cholesky-decomposition fails
comment:17 Changed 3 years ago by gkronber
r14892: made some adjustments after bwerth's changes (mainly formatting)
comment:18 Changed 3 years ago by gkronber
r14911: copied kernel ridge regression from branch to trunk
comment:19 Changed 3 years ago by gkronber
r14936: changed namespace for KernelRidgeRegression
comment:20 Changed 3 years ago by gkronber
- Owner changed from bwerth to mkommend
- Version changed from branch to 3.3.14
comment:21 follow-up: ↓ 24 Changed 3 years ago by mkommend
Review comments
General
Region names HLConstructors & Boilerplate are superflous! Just constructors is fine(comment:23)If too much boilerplate code has to be written omit at least unnecessary code such as empty AfterDeserialization hooks.(comment:23)
Kernels
IKernel: Changed events for Beta and Distance(comment:24)KernelBase: avoid instantiating Cloner objects manually, just use Clone()(comment:23)Check if Beta has a value before accessing it and throw meaningful exceptions.(comment:25)
TBC
comment:22 Changed 3 years ago by gkronber
r15155: deleted branch
comment:23 Changed 3 years ago by gkronber
r15156: removed superfluous region names and empty AfterDeserialization hooks
comment:24 in reply to: ↑ 21 Changed 3 years ago by gkronber
r15157: added events for BetaChanged and DistanceChanged
comment:25 Changed 3 years ago by bwerth
r15158 added checks for Beta without value and according exceptions
comment:26 Changed 3 years ago by mkommend
- Owner changed from mkommend to bwerth
- Status changed from reviewing to assigned
Review comments
KernelRR Model
Do not allow direct access to the allowedInputVariables array by the according property(comment:27)allowedInputVariables must be cloned, because it is modifiable by the mentioned property(comment:27)If nothing is cloned because it is not modified, then why is the kernel function cloned?(comment:27)Helper functions should be static (CreateScaling, ExtractData, BuildGramMatrix)(comment:27)Extract model learning from ctor(comment:27)
KernelRR
Kernel should be a constrained value parameter (create experiment support)(comment:28)
Comments
- The following remarks needn't be addressed
- Storable properties should be used instead of serializing fields directly
- Empty lines improve the readability of the source
- Use OnlineCalculators
comment:27 Changed 3 years ago by bwerth
r15164 KRRModel:
- made helper functions static;
- made contents of "allowedInputVariables" immutable;
- made constructor private and added public "Create"-method that does most of the learning now;
Kernels:
- fixed inconsistency in error messages
comment:28 Changed 3 years ago by bwerth
r15169 made KernelParameter a ConstrainedValueParameter; added some description to BetaParameter
comment:29 Changed 3 years ago by bwerth
- Owner changed from bwerth to mkommend
- Status changed from assigned to reviewing
comment:30 Changed 3 years ago by mkommend
- Status changed from reviewing to readytorelease
comment:31 Changed 3 years ago by gkronber
- Owner changed from mkommend to gkronber
- Status changed from readytorelease to reviewing
comment:32 Changed 3 years ago by gkronber
- Status changed from reviewing to readytorelease
Reviewed the changes and tested everything. Looks nice! r15248: fixed typo
comment:33 Changed 3 years ago by gkronber
comment:34 Changed 3 years ago by gkronber
- Resolution set to done
- Status changed from readytorelease to closed
r14385 created branch for radial basis functions regression