Opened 2 years ago

Last modified 18 months ago

#2361 assigned feature request

Adapted symbolic classification evaluators to increase the sensitivity of a model

Reported by: ehopf Owned by: mkommend
Priority: medium Milestone: HeuristicLab 4.x Backlog
Component: Problems.DataAnalysis.Symbolic.Classification Version: branch
Keywords: Cc:

Description (last modified by ehopf)

Two experimental symbolic classification evaluators. Besides considering a normalized mean squared error, the first evaluator (WeightedPerformanceMeasuresEvaluator) also takes the false negative and false positive rate of the model into account. The evaluator was used in Affenzeller et al., Increasing the Sensitivity of Cancer Predictors Using Confidence Based Ensemble Modeling, EUROCAST'15. The second evaluator introduces residuals weights to the normal mean squared error.

Change History (27)

comment:1 Changed 2 years ago by ehopf

  • Owner changed from mkommend to ehopf
  • Status changed from new to accepted

comment:2 Changed 2 years ago by ehopf

r12204: Branched HL.Problems.DataAnalysis.Symbolic.Classification to implement the evaluator.

comment:3 Changed 2 years ago by ehopf

r12209: Adapted the project settings for the SensitivityEvaluator branch.

comment:4 Changed 2 years ago by ehopf

r12210: Implemented the first prototype of the new evaluator.

comment:5 Changed 2 years ago by ehopf

r12211: Simple refactoring of SymbolicClassificationSingleObjectiveWeightedPerformanceMeasuresEvaluator.

comment:6 Changed 2 years ago by ehopf

r12212: Branched HL.Problems.DataAnalysis to fix a bug in the ClassificationPerformanceMeasuresCalculator.

comment:7 Changed 2 years ago by ehopf

r12213: Adjusted the project settings of the SensitivityEvaluator branch.

comment:8 Changed 2 years ago by ehopf

r12216: Removed redundant calculations.

comment:9 Changed 2 years ago by ehopf

r12218: Added missing estimated values bound calculation and removed duplicate code. Note that the current implementation ignores the linear scaling option.

comment:10 Changed 2 years ago by ehopf

r12302: Added a NotSupported-Exception for the case that the user activates the applyLinearScaling-option. Additionally made some execution time measurements with and without invocation of the ToArray-Extension-Method after the calculation of the bounded estimated values.


Measurements: OSGA -> Symbolic Classification -> Weighted Performance Measures Evaluator -> Seed 962761388

without ToArray(): 1.00:00:54.5801344 2.00:00:53.7666428 3.00:00:53.7478323

with ToArray(): 1.00:00:53.1704467 2.00:00:52.8202198 3.00:00:52.8622417

comment:11 Changed 2 years ago by ehopf

r12309: Adjusted Release Project settings.

comment:12 Changed 2 years ago by ehopf

r12311: Removed a unnecessary method call.

comment:13 Changed 2 years ago by mkommend

  • Version changed from 3.3.11 to branch

comment:14 Changed 2 years ago by ehopf

r12409: Merged r12358 into branch.

comment:15 Changed 23 months ago by ehopf

  • Description modified (diff)
  • Summary changed from Adapted symbolic classification evaluator to increase the sensitivity to Adapted symbolic classification evaluators to increase the sensitivity of a model

comment:16 Changed 23 months ago by ehopf

r12416: Implemented a new experimental evaluator which introduces residual weights to the mean squared error.

comment:17 Changed 23 months ago by ehopf

  • Owner changed from ehopf to mkommend
  • Status changed from accepted to reviewing

comment:18 Changed 23 months ago by ehopf

r12417: Changed the type of the ModelCreatorParameter to IValueLookupParameter.

comment:19 Changed 23 months ago by ehopf

r12418: Added backward compatibility code for the WeightedPerformanceMeasuresEvaluator.

comment:20 Changed 23 months ago by ehopf

r12448: Minor naming changes.

comment:21 Changed 23 months ago by ehopf

r12449: Changed the calculation of the error in the OnlineWeightedResidualsMeanSquaredErrorCalculator. The weight is now only applied once to the error.

comment:22 Changed 23 months ago by mkommend

  • Owner changed from mkommend to ehopf
  • Status changed from reviewing to assigned

comment:23 Changed 23 months ago by ehopf

r12486: Removed a redundant check.

comment:24 Changed 23 months ago by ehopf

r12487: Fixed a bug that causes an exception when a ModelCreator is used that doesn´t implement the ISymbolicDiscriminantFunctionClassificationModelCreator interface. Note that the runtime can be further improved for those ModelCreators because of redundant calculations.

comment:25 Changed 22 months ago by ascheibe

  • Milestone changed from HeuristicLab 3.3.12 to HeuristicLab 3.3.13

comment:26 Changed 18 months ago by gkronber

  • Owner changed from ehopf to mkommend

I believe we can move this to HL 4.x backlog?

comment:27 Changed 18 months ago by mkommend

  • Milestone changed from HeuristicLab 3.3.13 to HeuristicLab 4.x Backlog
Note: See TracTickets for help on using tickets.