Refactor Constants Optimization

[mkommend]

The code for constants optimization has grown over the years and all new functionality has been directly added in the evaluator. This should be refactored into separate classes and improvements to its functionality should be implemented. In detail the following changes should be realized:

• Optimizing constants should be separated from evaluation
• Constant optimization should use the MSE instead of the R²
• Add interfaces and static utility methods
• ...

[mkommend]

r16455: Created branch and added adapted project HL.Problems.DataAnalysis.Symbolic.Regression.

[mkommend]

r16456: Added HL.Problems.DataAnalysis.Symbolic to the branch.

[mkommend]

r16457: Extracted linear scaling terms in auto diff converter.
r16458: Readded accidently remove readonly modifier in autodiff converter.

[mkommend]

r16459: Added utility methods and copy of constant optimization evluator for refactoring.
r16460: Adapted project files.

[mkommend]

r16461: Added unit tests and refactoring.

[mkommend]

r16463: Adapted tree to autodiff converter.
r16464: Adapted new constants optimizer.

[mkommend]

r16500: Added intermediate version of new constants optimization for profiling.

[mkommend]

r16507: First stable version of new CoOp.
r16508: Reverted changes in TreeToAutoDiffTermConverter.

[mkommend]

r16509: Added new evaluator with constants optimization.

[mkommend]

r16514: Updated CoOp to handle negative R² values approprietly.

[mkommend]

r16522: Fixed bug in constants extraction when adding linear scaling coefficients.

[bburlacu]

r16525: Add benchmarks solution for testing constant optimization performance.

[gkronber]

r16676: merged r16478:16672 from trunk:HeuristicLab.Problems.DataAnalysis.Symbolic to branch:HeuristicLab.Problems.DataAnalysis.Symbolic

[gkronber]

r16677: merged r16499:16658 from trunk:HeuristicLab.Problems.DataAnalysis.Symbolic.Regression to branch:HeuristicLab.Problems.DataAnalysis.Symbolic.Regression

[gkronber]

r16678: removed project references again after merging changes from trunk

[gkronber]

r16679: changed .NET framework version of UnitTests project for compatibility with HL trunk (4.6.1)

[gkronber]

r16680: necessary changes for compatibility with new persistence

[gkronber]

r16689: changed LMConstantsOptimizer to use minlm interface of alglib instead of lsfit interface.

[gkronber]

r16690: added code in the unit tests to make behaviour of new and old ConstOpt consistent to allow direct comparison of results (+fixed a small bug in the unit tests)

[gkronber]

Currently, one unit test fails because of code in OnlinePearsonRCalculator.

It would work if we would replace it with

    public double R {
      get {
        double xVar = sxCalculator.PopulationVariance;
        double yVar = syCalculator.PopulationVariance;
        var r = covCalculator.Covariance / (Math.Sqrt(xVar) * Math.Sqrt(yVar));
        if (double.IsNaN(r)) return 0.0;
        if (r < -1.0) r = -1.0;
        else if (r > 1.0) r = 1.0;
        return r;
      }
    }

[gkronber]

r16691: improved code for benchmarking (previous code failed to account for update of trees after call of Optimize as well as early stopping in Optimize for trees with invalid outputs)

[gkronber]

Benchmark results on my notebook:

[mkommend]

r17193: Merged trunk changes into branch.

[gkronber]

r17392: merged r17344:17351 from trunk to branch

[gkronber]

r17393: fix typos in comments