Evaluation of GaussianProcessModels should be faster

[mkommend]

For testing purposes I have got an ensemble of six GP models trained on a dataset with 10,000 rows, 18 columns and the reevaluation of all six models takes ~ 48 seconds.

From a quick look at the implementation I suspect that lots of performance is lost due to the excessive use of iterators by enumerable methods.

[mkommend]

TestScript for performance of GP evaluation

[mkommend]

Timings on my hardware for reevaluation of the six models are:

49.058 48.569 48.212 48.111 48.275

[mkommend]

r13721: Changed all GP covariance and mean functions to use int[] for column indices instead of IEnumerable<int>. Changed GP utils, GPModel and StudentTProcessModell as well to use fewer iterators and adapted unit tests to new interface.

[mkommend]

Timings with the test script (seconds) improved by a factor of 2:

old	new r13721	old/new
49.058	23.782	2.06
48.569	23.262	2.09
48.212	22.657	2.13
48.111	22.882	2.10
48.275	23.27	2.07

[mkommend]

Summary of memory profiling with CLR profiler:

	new	old
Allocated bytes	2,450,780,172	16,869,022,270
Relocated bytes	113,526,450	853,352
Final Heap bytes	2,424,082,270	2,489,593,880
Gen 0 collections	38	169
Gen 1 collections	30	28
Gen 2 collections	27	26
Gen 0 Heap bytes	96,661,529	88,996,337
Gen 1 Heap bytes	25,597,736	594,243
Gen 2 Heap bytes	20,942,040	7,268,709
Large Object Heap bytes	24,605,859	100,937,162
Handles created	39,234	16,016
Handles destroyed	28,718	7,393
Handles surviving	10,516	8,623

The most significant change can be seen at the allocated bytes with a difference of ~ 14.4 GB. This difference results through ~ 300 million created instance of iterators (most RangeIterators created by Enumerable.Range). Out of the remaining ~ 2.4 GB 2.2 GB are used for calculating the Ks cross covariance matrix with 10,000² elements for each GP model.

[mkommend]

The performance of GP has been improved with r13712 and although all unit tests pass, I am unsure whether I haven't introduced any new bugs. So please review carefully.

[mkommend]

r13724: Reversed merged accidentally commited changes to GradientBoostingRegressionAlgorithm (r13721).

[pfleck]

r13784 Made the creation of a GaussianProcessModel faster by avoiding additional iterators during calculation of the hyperparameter gradients. The gradients of the hyperparameters are now calculated in one sweep and returned as IList, instead of returning an iterator (with yield return). This avoids a large amount of Move-calls of the iterator, especially for covariance functions with a lot of hyperparameters. Besides, the signature of the CovarianceGradientFunctionDelegate is changed, to return an IList instead of an IEnumerable to avoid unnececary ToList or ToArray calls.

Tested the opimization with different covariance functions and different datasets (length and number of features). After optimizing, creating a GaussProcessModel was 10% - 25% faster, depending on the covariance function and dataset. Covariance functions with more parameters (ARD covariance functions) benefitted more.

[gkronber]

Depends on #2541

[gkronber]

Seems CovMask doesn't work anymore. Will investigate...

[gkronber]

Replying to gkronber:

Seems CovMask doesn't work anymore. Will investigate...

Too tired, CovMask works fine...

[gkronber]

Reviewed r13721.

[gkronber]

r13891: removed capacity initialization in CovProd and CovSum because I believe it is more costly to calculate the capacity than to just increase the list capacity when necessary.

[gkronber]

reviewed r13784. Thanks for the performance improvements!

[gkronber]

This ticket depends on changes in #1795. Merge tickets together.

[gkronber]

r13981: merged r13438 (#2541), r13721, r13724, r13784, r13891 from trunk to stable

[gkronber]

Everything merged, close as done when integration tests succeed