Performance improvement of NaN/Inf-check on a double-matrix

[fholzing]

The current implementation to check whether a double[,] contains any Nan/Inf-values or not is rather time consuming.

inputMatrix.Cast<double>().Any(...) At first glance it takes about 6 seconds for a 50000x500 matrix. After consulting mkommend, a faster alternative would be preferred.

[fholzing]

There is a total of 11 occurrences in the HeuristicLab.Algorithms.DataAnalysis-project.

[fholzing]

Performance comparison

[fholzing]

After a first benchmark run the difference between the current approach (mentioned in the description) and three alternatives seems to be approximately a factor of 50.

[fholzing]

Additional note of importance: if you are ever in need of a BIG array, you have to enable gcallowverylargeobjects (see ​https://msdn.microsoft.com/de-de/library/hh285054(v=vs.110).aspx)

[fholzing]

My approach would be the following: Extend the class ObjectExtensions (Common-Project) with the Iterator-Method (as shown in the .xlsx, imho the easiest/most readable one, with very little performance-impact) and use the new extension method for all 11 occurrences.

[gkronber]

Ok, go for it but please use double.IsNaN instead of Double.IsNaN.

It would be great to know why there is such a big difference between the current code (Cast<double>) and the iterator.

[mkommend]

I suspect that it either has to do with the lambda call or more likely with boxing involved, because the Cast extension method is defined on IEnumerable instead of IEnumerable<double>.

[gkronber]

Aha, boxing seems to be a good explanation.

[fholzing]

r15783: Changed from Cast to Iterator and adapted all occurrences.

[fholzing]

FieldTest:
Random Forest Regression (Generated Testbed, 50000x500, Test/Train Datapartitions chosen with 0-20000 / 20000-40000)
M: 0,5
NoTree: 200
R: 0,2

Performance for Variable Impact View
Before optimization: ~1323 sec
After optimization: ~ 762 sec
Without check: ~ 749 sec

[gkronber]

Reviewed r15783.

• I found no other references to .Cast<double>, so this is fine.
• I would prefer the name .ContainsNanOrInfinity()

[fholzing]

r15786: Renamed ContainsNanInf to ContainsNanOrInfinity

[gkronber]

Reviewed r15786

[gkronber]

r15788: merged r15783 and r15786 from trunk to stable