#2591 closed enhancement (done)

Evaluation of GaussianProcessModels should be faster

Reported by: mkommend Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.14
Component: Algorithms.DataAnalysis Version: 3.3.13
Keywords: Cc:

Description

For testing purposes I have got an ensemble of six GP models trained on a dataset with 10,000 rows, 18 columns and the reevaluation of all six models takes ~ 48 seconds.

From a quick look at the implementation I suspect that lots of performance is lost due to the excessive use of iterators by enumerable methods.

Attachments (1)

Script GP Ensemble Perf.hl (8.6 MB) - added by mkommend 13 months ago.
TestScript for performance of GP evaluation

Change History (22)

comment:1 Changed 13 months ago by mkommend

  • Status changed from new to accepted

Changed 13 months ago by mkommend

TestScript for performance of GP evaluation

comment:2 Changed 13 months ago by mkommend

Timings on my hardware for reevaluation of the six models are:

49.058 48.569 48.212 48.111 48.275

comment:3 Changed 13 months ago by mkommend

r13721: Changed all GP covariance and mean functions to use int[] for column indices instead of IEnumerable<int>. Changed GP utils, GPModel and StudentTProcessModell as well to use fewer iterators and adapted unit tests to new interface.

comment:4 Changed 13 months ago by mkommend

Timings with the test script (seconds) improved by a factor of 2:

old new r13721 old/new
49.058 23.782 2.06
48.569 23.262 2.09
48.212 22.657 2.13
48.111 22.882 2.10
48.275 23.27 2.07

comment:5 Changed 13 months ago by mkommend

Summary of memory profiling with CLR profiler:

new old
Allocated bytes 2,450,780,172 16,869,022,270
Relocated bytes 113,526,450 853,352
Final Heap bytes 2,424,082,270 2,489,593,880
Gen 0 collections 38 169
Gen 1 collections 30 28
Gen 2 collections 27 26
Gen 0 Heap bytes 96,661,529 88,996,337
Gen 1 Heap bytes 25,597,736 594,243
Gen 2 Heap bytes 20,942,040 7,268,709
Large Object Heap bytes 24,605,859 100,937,162
Handles created 39,234 16,016
Handles destroyed 28,718 7,393
Handles surviving 10,516 8,623

The most significant change can be seen at the allocated bytes with a difference of ~ 14.4 GB. This difference results through ~ 300 million created instance of iterators (most RangeIterators created by Enumerable.Range). Out of the remaining ~ 2.4 GB 2.2 GB are used for calculating the Ks cross covariance matrix with 10,000² elements for each GP model.

comment:6 Changed 13 months ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from accepted to reviewing

The performance of GP has been improved with r13721 and although all unit tests pass, I am unsure whether I haven't introduced any new bugs. So please review carefully.

Last edited 13 months ago by mkommend (previous) (diff)

comment:7 Changed 13 months ago by mkommend

r13724: Reversed merged accidentally commited changes to GradientBoostingRegressionAlgorithm (r13721).

comment:8 Changed 12 months ago by pfleck

  • Owner changed from gkronber to pfleck
  • Status changed from reviewing to assigned

comment:9 Changed 12 months ago by pfleck

  • Status changed from assigned to accepted

comment:10 Changed 12 months ago by pfleck

r13784 Made the creation of a GaussianProcessModel faster by avoiding additional iterators during calculation of the hyperparameter gradients. The gradients of the hyperparameters are now calculated in one sweep and returned as IList, instead of returning an iterator (with yield return). This avoids a large amount of Move-calls of the iterator, especially for covariance functions with a lot of hyperparameters. Besides, the signature of the CovarianceGradientFunctionDelegate is changed, to return an IList instead of an IEnumerable to avoid unnececary ToList or ToArray calls.

Tested the opimization with different covariance functions and different datasets (length and number of features). After optimizing, creating a GaussProcessModel was 10% - 25% faster, depending on the covariance function and dataset. Covariance functions with more parameters (ARD covariance functions) benefitted more.

comment:11 Changed 12 months ago by gkronber

Depends on #2541

comment:12 Changed 11 months ago by pfleck

  • Owner changed from pfleck to gkronber
  • Status changed from accepted to reviewing

comment:13 follow-up: Changed 11 months ago by gkronber

Seems CovMask doesn't work anymore. Will investigate...

comment:14 in reply to: ↑ 13 Changed 11 months ago by gkronber

Replying to gkronber:

Seems CovMask doesn't work anymore. Will investigate...

Too tired, CovMask works fine...

comment:15 Changed 11 months ago by gkronber

Reviewed r13721.

comment:16 Changed 11 months ago by gkronber

r13891: removed capacity initialization in CovProd and CovSum because I believe it is more costly to calculate the capacity than to just increase the list capacity when necessary.

comment:17 Changed 11 months ago by gkronber

  • Status changed from reviewing to readytorelease

reviewed r13784. Thanks for the performance improvements!

comment:18 Changed 10 months ago by gkronber

This ticket depends on changes in #1795. Merge tickets together.

comment:19 Changed 10 months ago by gkronber

r13981: merged r13438 (#2541), r13721, r13724, r13784, r13891 from trunk to stable

comment:20 Changed 10 months ago by gkronber

Everything merged, close as done when integration tests succeed

comment:21 Changed 10 months ago by mkommend

  • Resolution set to done
  • Status changed from readytorelease to closed
Note: See TracTickets for help on using tickets.