Opened 5 years ago

Closed 4 years ago

#1902 closed feature request (done)

Gaussian Process Regression Algorithm

Reported by: gkronber Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.8
Component: Algorithms.DataAnalysis Version: 3.3.8
Keywords: Cc:

Description


Change History (80)

comment:1 Changed 5 years ago by gkronber

r8323: initial import of Gaussian process regression algorithm

comment:2 Changed 5 years ago by gkronber

  • Status changed from new to accepted

comment:3 Changed 5 years ago by gkronber

r8324:

  • fixed build (be ready for the obligatory round of beers at the archery meeting)
  • set copy-local to false for new references

comment:4 Changed 5 years ago by gkronber

r8325: changed return value type for parameter properties

comment:5 Changed 5 years ago by gkronber

r8366 added linear mean and covariance function

comment:6 Changed 5 years ago by gkronber

r8368: fixed cloning of linear mean function

comment:7 Changed 5 years ago by gkronber

r8371: worked on Gaussian Process algorithm

comment:8 Changed 5 years ago by gkronber

r8372: removed unnecessary plugin references.

comment:9 Changed 5 years ago by gkronber

r8375: implemented Gaussian process regression operators and analyzers

comment:10 Changed 5 years ago by gkronber

r8396: implemented LM-BFGS algorithm and improved GPR

comment:11 Changed 5 years ago by gkronber

r8397: added random seed parameters for LM-BFGS

comment:12 Changed 5 years ago by gkronber

r8399: fixed plugin dependencies

comment:13 Changed 5 years ago by gkronber

r8403: introduced upper limit for scale of SEard covariance function

comment:14 Changed 5 years ago by gkronber

r8416: worked on sum and product covariance functions and fixed a few bugs.

comment:15 Changed 5 years ago by gkronber

r8417: added periodic covariance function

comment:16 Changed 5 years ago by gkronber

r8419: changed initialization of hyperparameter vector for GPR using a PRNG

comment:17 Changed 5 years ago by gkronber

r8421: fixed default mean and covariance function for GPR

comment:18 Changed 5 years ago by gkronber

r8455: changed calculation of gradients for covariance functions to reduce allocations of arrays

comment:19 Changed 5 years ago by gkronber

r8439:

  • implemented unit tests for mean and covariance functions.
  • implemented sum and product mean function.
  • fixed incorrect gradient calculation in CovarianceProd

comment:20 Changed 5 years ago by gkronber

r8463: improved GPR implementation:

  • implemented additional unit tests
  • fixed bugs uncovered by unit tests
  • improved performance
  • added views for sum and product mean functions
Last edited 5 years ago by gkronber (previous) (diff)

comment:21 Changed 5 years ago by gkronber

r8464: added const and noise covariance functions.

comment:22 Changed 5 years ago by gkronber

r8473

  • added line chart
  • made parameters of mean and covariance functions readable
  • removed target variable scaling
  • moved noise hyperparameter for likelihood function to the end of the parameter list
  • added methods to calculate the predicted variance
  • removed limits for scale of covariance functions and introduced exception handling to catch non-spd or singular cov matrixes
  • implemented rational quadratic covariance function
  • added unit test case from GBML book - however, it does not work as the book seemingly uses a noise-less likelihood function

comment:23 Changed 5 years ago by gkronber

r8475: fixed bug in calculation of variance in GPR model

comment:24 Changed 5 years ago by gkronber

r8484:

  • changed interface for covariance functions to improve readability
  • fixed several bugs in the covariance functions
  • fixed bugs in the line chart for Gaussian process models

comment:25 Changed 5 years ago by gkronber

r8489: removed caches in product and sum covariance functions and fixed bug in SE-ARD covariance function

comment:26 Changed 5 years ago by gkronber

r8491: fixed test cases, improved performance

comment:27 Changed 5 years ago by abeham

I tested GPR on my dataset and found that CovarianceProd, CovarianceSum, and CovariancePeriodic throw an exception in the GaussianProcessRegressionSolutionCreator. Is there anything I need to take care of in these covariance functions?

comment:28 Changed 5 years ago by gkronber

r8494:8495:

  • changed mean and covariance function parameters to simple value parameters instead of constrained value parameters
  • changed the solution creator so that it doesn't throw an exception if no model could be found.
Last edited 5 years ago by gkronber (previous) (diff)

comment:29 Changed 5 years ago by abeham

I would prefer constrained value parameters. Otherwise you can't use them in the create experiment dialog. What's the motivation for moving to a value parameter?

comment:30 Changed 5 years ago by gkronber

Replying to abeham:

I tested GPR on my dataset and found that CovarianceProd, CovarianceSum, and CovariancePeriodic throw an exception in the GaussianProcessRegressionSolutionCreator. Is there anything I need to take care of in these covariance functions?

The sum and product covariance functions need must be configured to have at least one child. The periodic covariance function only works for the 1d case (one input variable). With r8494 I changed the parameters so that it becomes more obvious that the sum and product covariance function need to be configured correctly. I also changed the solution creator so that it doesn't throw an exception.

comment:31 Changed 5 years ago by gkronber

Sum and product covariance function cannot be used in the create experiment dialog anyway.

comment:32 Changed 5 years ago by gkronber

r8562: implemented LinearARD and MaternIso covariance functions.

comment:33 Changed 5 years ago by gkronber

r8565: implemented RQard covariance function.

comment:34 Changed 5 years ago by gkronber

r8580: fixed coloring in line chart for GPR

comment:35 Changed 5 years ago by gkronber

r8582:

  • implemented a few covariance functions as parameterized named items.
  • Implemented rudimentary view for Gaussian process models.

comment:36 Changed 5 years ago by gkronber

r8583: fixed build fail.

comment:37 Changed 5 years ago by gkronber

r8592: added default ctor for HyperParameter

comment:38 Changed 5 years ago by gkronber

r8612: implemented all mean and covariance functions with parameters as ParameterizedNamedItems

comment:39 Changed 5 years ago by gkronber

r8614: removed unused file

comment:40 Changed 5 years ago by gkronber

r8615: renamed covariance functions

comment:41 Changed 5 years ago by gkronber

r8620: renamed more files. implemented scale covariance function.

comment:42 Changed 5 years ago by gkronber

r8623: implemented LS Gaussian Process classification

comment:43 Changed 5 years ago by gkronber

r8678: added masking covariance function and made necessary changes to interface and utility class.

comment:44 Changed 5 years ago by gkronber

r8679:

  • changed DiscriminantFunctionClassificationModel and DiscriminantFunctionClassificationSolution to non-abstract.
  • Classification based on GPR uses these classes to create a classification solution from a GaussianProcessRegressionModel.
  • Also added view classes for DiscriminantFunctionClassificationModel

comment:45 follow-up: Changed 5 years ago by abeham

I noticed that the solution in the run object is only a result collection, but not a DiscriminantFunctionClassificationSolution anymore.

comment:46 Changed 5 years ago by gkronber

r8682:

  • fixed bug in cloning constructor of CovarianceMask
  • removed throwing Exceptions in CovariancePeriodic (the behaviour of the periodic covariance function now matches the GPML implementation again.

comment:47 in reply to: ↑ 45 Changed 5 years ago by gkronber

Replying to abeham:

I noticed that the solution in the run object is only a result collection, but not a DiscriminantFunctionClassificationSolution anymore.

Thanks for the hint! The cloning method override is missing.

comment:48 Changed 5 years ago by gkronber

r8683: added missing cloning method override in DiscriminantFunctionClassificationSolution

comment:49 Changed 5 years ago by gkronber

r8684: fixed a bug related to the virtual call to RecalculateResults in the constructor of DiscriminantFunctionClassificationSolution

comment:50 Changed 5 years ago by gkronber

r8732:

  • added caching for variances of GaussianProcessRegressionSolution.
  • changed hyperparameter initializers to generate initial values for hyperparameters in a larger range.

comment:51 Changed 5 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from accepted to reviewing

comment:52 Changed 5 years ago by gkronber

r8733: added a unit test to create and run a GPR sample

comment:53 Changed 5 years ago by gkronber

r8740: removed assertion for test R² in GPR unit test as we are testing on a benchmark problem where the test set is randomly created.

comment:54 Changed 5 years ago by gkronber

r8771: fixed a bug in the noise covariance function

comment:55 Changed 5 years ago by gkronber

To do: check range of all input variables and deactivate if necessary.

comment:56 Changed 5 years ago by gkronber

r8827: fixed bugs concerning masking covariance function

comment:57 Changed 5 years ago by gkronber

r8829: fixed an issue that occurs in scaling if a variable is effectively constant in the training partition

comment:58 follow-up: Changed 5 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to assigned

Reviewing comments:

  • Use folders for covariance and mean functions.
  • Util.AttachValueChangeHandler is quite ugly
  • The HyperParameter class relies heavily on which method SetValue or Value.Set is called and determines according to this whether the fixed flag applies or not.
  • Util.SqrDist: Is the scale array always exactly as large as the number of column indixes?.
  • Covariance functions expose methods with column indexes which are not always forwarded to the concrete util functions (e.g., CovarianceLinear.GetCrossCovariance).

I have also tested the implementation on sample problem instances for regression and classification and did not find any bugs. However, I must admit that I didn't completely understand how the implementation works as I am lacking the basics of GP.

comment:59 in reply to: ↑ 58 ; follow-up: Changed 5 years ago by gkronber

Replying to mkommend:

Reviewing comments:

  • Util.AttachValueChangeHandler is quite ugly

Please specify the problem in more detail. Do you mean that they should not be located in the Util class?

comment:60 in reply to: ↑ 59 ; follow-up: Changed 5 years ago by mkommend

Replying to gkronber:

Replying to mkommend:

Reviewing comments:

  • Util.AttachValueChangeHandler is quite ugly

Please specify the problem in more detail. Do you mean that they should not be located in the Util class?

The "problem" is that this event registration is implemented in quite a different way as the remaining parts of HL, which makes it harder to grasp what is going on (when is the local parameter cache updated, what happens on a value change, where is the new event registered, ...).

comment:61 in reply to: ↑ 60 Changed 5 years ago by gkronber

Replying to mkommend:

Replying to gkronber:

Replying to mkommend:

Reviewing comments:

  • Util.AttachValueChangeHandler is quite ugly

Please specify the problem in more detail. Do you mean that they should not be located in the Util class?

The "problem" is that this event registration is implemented in quite a different way as the remaining parts of HL, which makes it harder to grasp what is going on

Yes it is different to the remaining parts of HL and harder to grasp.

AttachValueChangeHandler takes a ValueParameter and registers a handler that executes the action which is specified as second parameter. Because both the value of a parameter, as well as the value of the value of the parameter can change, we have to attach to both events- In case the value of the parameter is changed we have to attach the event handler to the new value.

(when is the local parameter cache updated, what happens on a value change, where is the new event registered).

AttachValueChangeHandler does not have a local parameter cache. Thus, I believe the real issue here is the caching of parameters and values in all covariance and mean functions and not the implementation of AttachValueChangeHandler.

comment:62 Changed 5 years ago by gkronber

  • r8929: moved covariance and mean functions to folders
  • r8931: fixed incorrect handling of dimension masking in CovarianceLinear (and checked all other covariance functions)
  • r8932:8933: corrected handling of length-parameter arrays in ARD functions and prevented stacking of mask covariance functions to make sure that the length-parameter and the enumerable of selected column indexes are equally long.

comment:63 Changed 5 years ago by gkronber

I thought about possible changes to remove the caching of parameter values in covariance functions and came up with a possible solution returning functions with the cached value in the closure. This would remove the necessity for event registration. It also allows us to remove the SetParameters method. What do you think about this idea?

public Func<double[,], int, int, double> GetCovarianceFunction(IEnumerable<int> columnIndices) {
      var inverseLength = inverseLengthParameter.Value.Value;
      var sf2 = scaleParameter.Value.Value;
      return (x, i, j) => {
        double d = i == j
                     ? 0.0
                     : Util.SqrDist(x, i, j, inverseLength, columnIndices);
        return sf2 * Math.Exp(-d / 2.0);
      };
    }
Last edited 5 years ago by gkronber (previous) (diff)

comment:64 Changed 5 years ago by mkommend

Is the proposed solution really working as intended? The inverseLength and scaleParameter could change its value over time, as they are optimized during the runtime. Now I have to know at which time I have to call GetCovarianceFunction to create a new closure containing the updated parameter value, if I am even aware of this issue.

Due to this difficulties, I think the event registration fits the purpose better.

comment:65 Changed 5 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from assigned to reviewing

r8982:

  • removed class HyperParameter
  • removed the parameter value caching and event handlers for parameter caching in the implemented covariance and mean functions
  • the instances of mean and covariance functions configured in the GUI are now actually templates where the structure and fixed parameters of the functions can be specified. In the templates the optimized parameters are null.
  • Templates are used in the GaussianProcessModel to create the actual covariance and mean functions as Func objects with specified parameter values (which are optimized using LM-BFGS).

comment:66 Changed 5 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to readytorelease

Reviewed new changesets r8929, r8931, r8932, r8933 and r8982. The refactored way of configuring covariance and mean functions is much cleaner, although I had doubts before I saw the first implementation.

comment:67 Changed 4 years ago by gkronber

  • Status changed from readytorelease to assigned

comment:68 Changed 4 years ago by gkronber

  • Status changed from assigned to accepted

comment:69 Changed 4 years ago by gkronber

r9104: fixed a bug occurring when the masking covariance function is used.

comment:70 Changed 4 years ago by gkronber

r9106: fixed a bug in ARD covariance functions that would occur when a parameter prior to the inverse length parameter would be fixed.

comment:71 Changed 4 years ago by gkronber

The gradients are not returned in the same order of the parameters! This should be checked for all covariance functions.

comment:72 Changed 4 years ago by gkronber

r9108: fixed bugs in ARD covariance functions (checked if parameter and gradient order matches for all functions)

comment:73 Changed 4 years ago by gkronber

r9111: fixed a calculation bug in the RQiso covariance function

comment:74 Changed 4 years ago by gkronber

GP should be a problem. Hyper-parameters are optimized, likelihood should be maximized.

comment:75 Changed 4 years ago by abeham

I think I should do the same with NCA which can also be thought of as an optimization problem.

comment:76 Changed 4 years ago by gkronber

r9211: reformulated calculation of periodic covariance function

comment:77 Changed 4 years ago by gkronber

r9357:9358: minor code improvements: removed commented code, always supply non-null columnIndizes.

Last edited 4 years ago by gkronber (previous) (diff)

comment:78 Changed 4 years ago by gkronber

r9360: implemented neural network covariance function plus test case (comparison with GPML) for Gaussian processes

comment:79 Changed 4 years ago by gkronber

  • Status changed from accepted to readytorelease

comment:80 Changed 4 years ago by swagner

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.7 to 3.3.8
Note: See TracTickets for help on using tickets.