SymbolicRegressionEvaluators handle invalid values differently than the models created by the analyzers

[mkommend]

If scaling is used the symbolic regression evaluators ignore invalid values during the calculation of the scaling factors. SymbolicRegressionModels however stop the calculation and perform no scaling if invalid values are encountered, which leads to a discrepancy between the current best quality and the quality values of the best training solution.

[mkommend]

r8635: Corrected handling of invalid values in the symbolic regression evaluators.

[mkommend]

The scaling of models and evaluator should ignore NaN and infinity values instead of stopping the scaling process.

[mkommend]

r8639: Changed symbolic regression evaluator and scale method in the symbolic regression model to ignore invalid (NaN & infinite) values during calculation of the scaling parameters.

[mkommend]

In addition the scaling is not performed in the same way for evaluators and models. Furthermore, the scaling parameter should be moved from the evaluator and analyzer to the problem and be discovered if needed. This ensures that the evaluator and analyzer calculate the same quality metric if configured correctly.

[mkommend]

r8664:

• Added linear scaling parameter to data analysis problems.
• Adapted interfaces, evaluators and analyzers accordingly.
• Added OnlineBoundedMeanSquaredErrorCalculator.
• Adapted symbolic regression sample unit test.

[mkommend]

Please keep in mind that the changes in r8664 result in different algorithm results for symbolic classification / regression problems, if the Pearson`s R² evaluator is used due to ignoring NaN and infinity value while the scaling parameters are calculated. Additionally, ten repetition of the symbolic regression sample show an average increase in the execution time of 10%.

[mkommend]

r8666: Corrected AfterDeserializationHook for SymbolicDataAnalysisProblem to set the ApplyLinearScaling parameter value correctly.

[mkommend]

r8667: Corrected symbolic regression sample unit test.

[gkronber]

no review comments for r8635 and r8639.

review comments for r8664:

• In ​SymbolicClassificationMultiObjectivePearsonRSquaredTreeSizeEvaluator , SymbolicClassificationSingleObjectivePearsonRSquaredEvaluator, SymbolicRegressionMultiObjectivePearsonRSquaredTreeSizeEvaluator, and SymbolicRegressionSingleObjectivePearsonRSquaredEvaluator: why is r2 set to double.NaN in case the R² couldn't be calculated? Usually we return 0.0 as worst value and I would prefer that. The change to double.NaN can have an effect in ordering of quality values.
• Why is SymbolicDataAnalysisModel.Scale(...) a static method?
• In ​SymbolicDataAnalysisEvaluator the comment "the static methods of the calculator could not be used as it performs a check if the enumerators have an equal amount of elements. this is not true if the cache is used" is confusing. Does it actually match the code?
• In SymbolicDataAnalysisEvaluator: use cache.Length instead of cache.GetLength(0)
• In SymbolicDataAnalysisEvaluator line 164: can we just reset the existing enumerator instead of creating a new one?

no review comments for r8666:8667.

[mkommend]

• Why is SymbolicDataAnalysisModel.Scale a static method?

r8972: Changed SymbolicDataAnalysisModel.Scale to a protected instance method, added the method in the classificaton and regression models and adapted all calls to the Scale method.

[mkommend]

r8973: Corrected plugin dependencies of the symbolic timeseries prognosis plugins.

[mkommend]

r8974: Adapted SymbolicDataAnalysisEvaluator.CalculateWithScaling to the review comments (removed misleading comment, used length property).

[mkommend]

Replying to gkronber:

• In SymbolicDataAnalysisEvaluator line 164: can we just reset the existing enumerator instead of creating a new one?

It is not possible to reset enumerators generated from linq statements, therefore a new one must be created.

[mkommend]

Replying to gkronber:

Why is r2 set to double.NaN in case the R² couldn't be calculated? Usually we return 0.0 as worst value and I would prefer that. The change to double.NaN can have an effect in ordering of quality values.

All evaluators were changed to return double.NaN if a problem occurs. As we normally use estimation limits this can only happen if something in the dataset is not configured correctly. Instead of continuing with the execution of the algorithm I decided to return double.Nan to indicate that there is a problem with the tree quality, which yields to an error during the selection phase of an algorithm.

[gkronber]

All of my review comments have been addressed.