Free cookie consent management tool by TermsFeed Policy Generator

Opened 9 years ago

Closed 5 years ago

#2361 closed feature request (done)

Adapted symbolic classification evaluators to increase the sensitivity of a model

Reported by: ehopf Owned by: mkommend
Priority: medium Milestone: HeuristicLab 3.3.16
Component: Problems.DataAnalysis.Symbolic.Classification Version: trunk
Keywords: Cc:

Description (last modified by ehopf)

Two experimental symbolic classification evaluators. Besides considering a normalized mean squared error, the first evaluator (WeightedPerformanceMeasuresEvaluator) also takes the false negative and false positive rate of the model into account. The evaluator was used in Affenzeller et al., Increasing the Sensitivity of Cancer Predictors Using Confidence Based Ensemble Modeling, EUROCAST'15. The second evaluator introduces residuals weights to the normal mean squared error.

Change History (36)

comment:1 Changed 9 years ago by ehopf

  • Owner changed from mkommend to ehopf
  • Status changed from new to accepted

comment:2 Changed 9 years ago by ehopf

r12204: Branched HL.Problems.DataAnalysis.Symbolic.Classification to implement the evaluator.

comment:3 Changed 9 years ago by ehopf

r12209: Adapted the project settings for the SensitivityEvaluator branch.

comment:4 Changed 9 years ago by ehopf

r12210: Implemented the first prototype of the new evaluator.

comment:5 Changed 9 years ago by ehopf

r12211: Simple refactoring of SymbolicClassificationSingleObjectiveWeightedPerformanceMeasuresEvaluator.

comment:6 Changed 9 years ago by ehopf

r12212: Branched HL.Problems.DataAnalysis to fix a bug in the ClassificationPerformanceMeasuresCalculator.

comment:7 Changed 9 years ago by ehopf

r12213: Adjusted the project settings of the SensitivityEvaluator branch.

comment:8 Changed 9 years ago by ehopf

r12216: Removed redundant calculations.

comment:9 Changed 9 years ago by ehopf

r12218: Added missing estimated values bound calculation and removed duplicate code. Note that the current implementation ignores the linear scaling option.

comment:10 Changed 9 years ago by ehopf

r12302: Added a NotSupported-Exception for the case that the user activates the applyLinearScaling-option. Additionally made some execution time measurements with and without invocation of the ToArray-Extension-Method after the calculation of the bounded estimated values.


Measurements: OSGA -> Symbolic Classification -> Weighted Performance Measures Evaluator -> Seed 962761388

without ToArray(): 1.00:00:54.5801344 2.00:00:53.7666428 3.00:00:53.7478323

with ToArray(): 1.00:00:53.1704467 2.00:00:52.8202198 3.00:00:52.8622417

comment:11 Changed 9 years ago by ehopf

r12309: Adjusted Release Project settings.

comment:12 Changed 9 years ago by ehopf

r12311: Removed a unnecessary method call.

comment:13 Changed 9 years ago by mkommend

  • Version changed from 3.3.11 to branch

comment:14 Changed 9 years ago by ehopf

r12409: Merged r12358 into branch.

comment:15 Changed 9 years ago by ehopf

  • Description modified (diff)
  • Summary changed from Adapted symbolic classification evaluator to increase the sensitivity to Adapted symbolic classification evaluators to increase the sensitivity of a model

comment:16 Changed 9 years ago by ehopf

r12416: Implemented a new experimental evaluator which introduces residual weights to the mean squared error.

comment:17 Changed 9 years ago by ehopf

  • Owner changed from ehopf to mkommend
  • Status changed from accepted to reviewing

comment:18 Changed 9 years ago by ehopf

r12417: Changed the type of the ModelCreatorParameter to IValueLookupParameter.

comment:19 Changed 9 years ago by ehopf

r12418: Added backward compatibility code for the WeightedPerformanceMeasuresEvaluator.

comment:20 Changed 9 years ago by ehopf

r12448: Minor naming changes.

comment:21 Changed 9 years ago by ehopf

r12449: Changed the calculation of the error in the OnlineWeightedResidualsMeanSquaredErrorCalculator. The weight is now only applied once to the error.

comment:22 Changed 9 years ago by mkommend

  • Owner changed from mkommend to ehopf
  • Status changed from reviewing to assigned

comment:23 Changed 9 years ago by ehopf

r12486: Removed a redundant check.

comment:24 Changed 9 years ago by ehopf

r12487: Fixed a bug that causes an exception when a ModelCreator is used that doesn´t implement the ISymbolicDiscriminantFunctionClassificationModelCreator interface. Note that the runtime can be further improved for those ModelCreators because of redundant calculations.

comment:25 Changed 9 years ago by ascheibe

  • Milestone changed from HeuristicLab 3.3.12 to HeuristicLab 3.3.13

comment:26 Changed 8 years ago by gkronber

  • Owner changed from ehopf to mkommend

I believe we can move this to HL 4.x backlog?

comment:27 Changed 8 years ago by mkommend

  • Milestone changed from HeuristicLab 3.3.13 to HeuristicLab 4.x Backlog

comment:28 Changed 5 years ago by gkronber

  • Milestone changed from HeuristicLab 4.x Backlog to HeuristicLab 3.3.16

Is this done and ready for trunk integration?

comment:29 Changed 5 years ago by mkommend

r16788: Merged all changes from the sensitivity branch to trunk.

Added [NonDiscoverableType] Attribute to the evaluators as these are IMHO not really useful.

comment:30 Changed 5 years ago by mkommend

  • Status changed from assigned to reviewing
  • Version changed from branch to trunk

comment:31 Changed 5 years ago by mkommend

r16789: Deleted branch for SensitivityEvaluator.

comment:32 Changed 5 years ago by mkommend

  • Status changed from reviewing to readytorelease

comment:33 Changed 5 years ago by mkommend

  • Keywords depends-2520 added

comment:34 Changed 5 years ago by abeham

  • Keywords depends-2520 removed

comment:35 Changed 5 years ago by jkarder

r17177: merged r16788 into stable

comment:36 Changed 5 years ago by jkarder

  • Resolution set to done
  • Status changed from readytorelease to closed
Note: See TracTickets for help on using tickets.