Free cookie consent management tool by TermsFeed Policy Generator

Opened 14 years ago

Closed 14 years ago

#1356 closed feature request (done)

Overfitting analyzer for symbolic regression

Reported by: gkronber Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.3
Component: ZZZ OBSOLETE: Problems.DataAnalysis.Regression Version: 3.3.3
Keywords: Cc:

Description

Follow up to ticket #1142

Change History (13)

comment:1 Changed 14 years ago by gkronber

  • Status changed from new to accepted

comment:2 Changed 14 years ago by gkronber

Copied overfitting analyzer for symbolic regression from feature exploration branch with r5192.

comment:3 Changed 14 years ago by gkronber

Introduced base class for operators that evaluate symbolic regression models on a validation set with r5197.

comment:4 Changed 14 years ago by gkronber

Changed FixedValidationBestScaledSymbolicRegressionSolutionAnalyzer to extend from SymbolicRegressionValidationAnalyzer with r5198.

comment:5 Changed 14 years ago by gkronber

Added overfitting analyzer to default operators of symbolic regression problems to make sure it also works correctly in cross-validation with r5260.

comment:6 Changed 14 years ago by swagner

  • Milestone changed from HeuristicLab x.x.x to HeuristicLab 3.3.3

comment:7 Changed 14 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from accepted to reviewing

comment:8 Changed 14 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to assigned

Reviewed the r5192, r5197, r5198 and r5260.

Comments:

  • The FixedValidationBestScaledSymbolicRegressionSolutionAnalyzer does not check if the estimation limits parameter have a value.
  • In my opinion all ValueLookupParameter and LookupParameters must have null checks before accessing their actual value.
  • The following source is too complicated:
        double correlationThreshold;
          if (OverfittingParameter.ActualValue != null && OverfittingParameter.ActualValue.Value) {
            // if is already overfitting => have to reach the upper threshold to switch back to non-overfitting state
            correlationThreshold = UpperCorrelationThresholdParameter.ActualValue.Value;
          } else {
            // if currently in non-overfitting state => have to reach to lower threshold to switch to overfitting state
            correlationThreshold = LowerCorrelationThresholdParameter.ActualValue.Value;
          }
          bool overfitting = r < correlationThreshold;
          OverfittingParameter.ActualValue = new BoolValue(overfitting);
    

I think the following is more readable and does the same thing:

      //start with no overfitting
      if (OverfittingParameter.ActualValue == null) OverfittingParameter.ActualValue = new BoolValue(false);
      double lowerCorrelationThreshold = LowerCorrelationThresholdParameter.ActualValue.Value;
      double upperCorrelationThreshold = UpperCorrelationThresholdParameter.ActualValue.Value;

      if (!(lowerCorrelationThreshold < r && r < upperCorrelationThreshold)) {
        bool overfitting = r < lowerCorrelationThreshold;
        OverfittingParameter.ActualValue = new BoolValue(overfitting);
      }

comment:9 Changed 14 years ago by gkronber

r5436: clarified code in overfitting analyzer.

comment:10 Changed 14 years ago by gkronber

r5437: added default values for upper and lower estimation limits if they are not specified where necessary.

comment:11 Changed 14 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from assigned to reviewing

Please review the changeset r5436:5437 and set the status of the ticket to ready-to-release if there are no more objections.

comment:12 Changed 14 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to readytorelease

Changesets r5436 and r5437 look fine.

comment:13 Changed 14 years ago by mkommend

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.2 to 3.3.3
Note: See TracTickets for help on using tickets.