Opened 4 years ago

Closed 3 years ago

#2093 closed feature request (done)

Evaluator for symbolic data analysis that calculates the mean of logarithmic residuals

Reported by: gkronber Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.10
Component: Problems.DataAnalysis.Symbolic.Regression Version: 3.3.10
Keywords: Cc:

Description

In preliminary experiments, we found that this improves success rates for data sets with a few very large/small target values (e.g. Korns benchmark instances containing inverses of near zero values). In these cases the squared error or absolute error puts too much emphasis on modeling the outlier values. In such instances small variations are not stemming from noise but are actually the signal that should be modeled and a log-error evaluator would put more weight on such small values.

Change History (21)

comment:1 Changed 3 years ago by gkronber

  • Owner changed from mkommend to gkronber
  • Status changed from new to accepted
  • Version changed from 3.3.8 to branch

comment:2 Changed 3 years ago by gkronber

Created a feature development branch with r10202

comment:3 Changed 3 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from accepted to reviewing

r10203: added log residual evaluator for symbolic regression

comment:4 Changed 3 years ago by gkronber

  • Version changed from branch to 3.3.10

comment:5 Changed 3 years ago by gkronber

  • Milestone changed from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.10
  • Version changed from 3.3.10 to branch

comment:6 Changed 3 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to assigned

Despite the source code looking fine, I am unsure if this functionality should be included in trunk right now. First of all the logarithm is not defined for 0, which is not handled in the evaluator (skipping invalid values?). Second, the current implementation calculates avg(log(abs(y'-y)) which tends to -Inf for very small values (this may be an issue if one data sample was hit with a very high precision). Third, there is no warning / indication that despite linear scaling is requested, it is not performed.

comment:7 Changed 3 years ago by gkronber

r10305: changed log residual evaluator to calculate avg(log( 1 + abs(y - pred)))

comment:8 Changed 3 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from assigned to reviewing

comment:9 Changed 3 years ago by mkommend

Please put a comment / description that linear scaling is never performed for this evaluator. Otherwise everything looks fine.

comment:10 Changed 3 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to assigned

comment:11 Changed 3 years ago by gkronber

  • Status changed from assigned to accepted

comment:12 Changed 3 years ago by gkronber

r10482: added a notice that the log residual evaluator does not apply linear scaling

comment:13 Changed 3 years ago by gkronber

r10483: merged trunk changes to prepare for reintegration

comment:14 Changed 3 years ago by gkronber

r10484: reintegrated branch for log residual evaluator

comment:15 Changed 3 years ago by gkronber

r10485: deleted branch for log residual evaluator

comment:16 Changed 3 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from accepted to reviewing

Please review r10482 and r10484.

comment:17 Changed 3 years ago by gkronber

r10486: fixed build fail

comment:18 Changed 3 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to readytorelease

comment:19 Changed 3 years ago by mkommend

  • Version changed from branch to 3.3.10

comment:20 Changed 3 years ago by gkronber

Depends on #2119

Last edited 3 years ago by mkommend (previous) (diff)

comment:21 Changed 3 years ago by mkommend

  • Resolution set to done
  • Status changed from readytorelease to closed

r10509: Merged r10483 and r10486 into stable.

Note: See TracTickets for help on using tickets.