Opened 5 years ago

Closed 4 years ago

#1951 closed defect (done)

SymbolicRegressionEvaluators handle invalid values differently than the models created by the analyzers

Reported by: mkommend Owned by: gkronber
Priority: high Milestone: HeuristicLab 3.3.8
Component: Problems.DataAnalysis.Symbolic.Regression Version: 3.3.8
Keywords: Cc:

Description

If scaling is used the symbolic regression evaluators ignore invalid values during the calculation of the scaling factors. SymbolicRegressionModels however stop the calculation and perform no scaling if invalid values are encountered, which leads to a discrepancy between the current best quality and the quality values of the best training solution.

Change History (20)

comment:1 Changed 5 years ago by mkommend

  • Status changed from new to accepted

comment:2 Changed 5 years ago by mkommend

r8635: Corrected handling of invalid values in the symbolic regression evaluators.

comment:3 Changed 5 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from accepted to reviewing

comment:4 Changed 5 years ago by mkommend

  • Owner changed from gkronber to mkommend
  • Status changed from reviewing to assigned

The scaling of models and evaluator should ignore NaN and infinity values instead of stopping the scaling process.

comment:5 Changed 5 years ago by mkommend

r8639: Changed symbolic regression evaluator and scale method in the symbolic regression model to ignore invalid (NaN & infinite) values during calculation of the scaling parameters.

comment:6 Changed 5 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from assigned to reviewing

comment:7 Changed 5 years ago by mkommend

  • Owner changed from gkronber to mkommend
  • Status changed from reviewing to assigned

In addition the scaling is not performed in the same way for evaluators and models. Furthermore, the scaling parameter should be moved from the evaluator and analyzer to the problem and be discovered if needed. This ensures that the evaluator and analyzer calculate the same quality metric if configured correctly.

comment:8 Changed 5 years ago by mkommend

  • Status changed from assigned to accepted

comment:9 Changed 5 years ago by mkommend

r8664:

  • Added linear scaling parameter to data analysis problems.
  • Adapted interfaces, evaluators and analyzers accordingly.
  • Added OnlineBoundedMeanSquaredErrorCalculator.
  • Adapted symbolic regression sample unit test.

comment:10 Changed 5 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from accepted to reviewing

Please keep in mind that the changes in r8664 result in different algorithm results for symbolic classification / regression problems, if the Pearson`s R² evaluator is used due to ignoring NaN and infinity value while the scaling parameters are calculated. Additionally, ten repetition of the symbolic regression sample show an average increase in the execution time of 10%.

comment:11 Changed 5 years ago by mkommend

r8666: Corrected AfterDeserializationHook for SymbolicDataAnalysisProblem to set the ApplyLinearScaling parameter value correctly.

comment:12 Changed 5 years ago by mkommend

r8667: Corrected symbolic regression sample unit test.

comment:13 follow-ups: Changed 4 years ago by gkronber

  • Owner changed from gkronber to mkommend

no review comments for r8635 and r8639.

review comments for r8664:

  • In SymbolicClassificationMultiObjectivePearsonRSquaredTreeSizeEvaluator , SymbolicClassificationSingleObjectivePearsonRSquaredEvaluator, SymbolicRegressionMultiObjectivePearsonRSquaredTreeSizeEvaluator, and SymbolicRegressionSingleObjectivePearsonRSquaredEvaluator: why is r2 set to double.NaN in case the R² couldn't be calculated? Usually we return 0.0 as worst value and I would prefer that. The change to double.NaN can have an effect in ordering of quality values.
  • Why is SymbolicDataAnalysisModel.Scale(...) a static method?
  • In SymbolicDataAnalysisEvaluator the comment "the static methods of the calculator could not be used as it performs a check if the enumerators have an equal amount of elements. this is not true if the cache is used" is confusing. Does it actually match the code?
  • In SymbolicDataAnalysisEvaluator: use cache.Length instead of cache.GetLength(0)
  • In SymbolicDataAnalysisEvaluator line 164: can we just reset the existing enumerator instead of creating a new one?

no review comments for r8666:8667.

comment:14 Changed 4 years ago by mkommend

  • Why is SymbolicDataAnalysisModel.Scale a static method?

r8972: Changed SymbolicDataAnalysisModel.Scale to a protected instance method, added the method in the classificaton and regression models and adapted all calls to the Scale method.

comment:15 Changed 4 years ago by mkommend

r8973: Corrected plugin dependencies of the symbolic timeseries prognosis plugins.

comment:16 Changed 4 years ago by mkommend

r8974: Adapted SymbolicDataAnalysisEvaluator.CalculateWithScaling to the review comments (removed misleading comment, used length property).

comment:17 in reply to: ↑ 13 Changed 4 years ago by mkommend

Replying to gkronber:

  • In SymbolicDataAnalysisEvaluator line 164: can we just reset the existing enumerator instead of creating a new one?

It is not possible to reset enumerators generated from linq statements, therefore a new one must be created.

comment:18 in reply to: ↑ 13 Changed 4 years ago by mkommend

  • Owner changed from mkommend to gkronber

Replying to gkronber:

Why is r2 set to double.NaN in case the R² couldn't be calculated? Usually we return 0.0 as worst value and I would prefer that. The change to double.NaN can have an effect in ordering of quality values.

All evaluators were changed to return double.NaN if a problem occurs. As we normally use estimation limits this can only happen if something in the dataset is not configured correctly. Instead of continuing with the execution of the algorithm I decided to return double.Nan to indicate that there is a problem with the tree quality, which yields to an error during the selection phase of an algorithm.

comment:19 Changed 4 years ago by gkronber

  • Status changed from reviewing to readytorelease

All of my review comments have been addressed.

comment:20 Changed 4 years ago by swagner

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.7 to 3.3.8
Note: See TracTickets for help on using tickets.