Context Navigation

#2361 closed feature request (done)

Adapted symbolic classification evaluators to increase the sensitivity of a model

Reported by:	ehopf	Owned by:	mkommend
Priority:	medium	Milestone:	HeuristicLab 3.3.16
Component:	Problems.DataAnalysis.Symbolic.Classification	Version:	trunk
Keywords:		Cc:

Description (last modified by ehopf)

Two experimental symbolic classification evaluators. Besides considering a normalized mean squared error, the first evaluator (WeightedPerformanceMeasuresEvaluator) also takes the false negative and false positive rate of the model into account. The evaluator was used in Affenzeller et al., Increasing the Sensitivity of Cancer Predictors Using Confidence Based Ensemble Modeling, EUROCAST'15. The second evaluator introduces residuals weights to the normal mean squared error.

Change History (36)

comment:1 Changed 10 years ago by ehopf

Owner changed from mkommend to ehopf
Status changed from new to accepted

comment:2 Changed 10 years ago by ehopf

r12204: Branched HL.Problems.DataAnalysis.Symbolic.Classification to implement the evaluator.

comment:3 Changed 10 years ago by ehopf

r12209: Adapted the project settings for the SensitivityEvaluator branch.

comment:4 Changed 10 years ago by ehopf

r12210: Implemented the first prototype of the new evaluator.

comment:5 Changed 10 years ago by ehopf

r12211: Simple refactoring of SymbolicClassificationSingleObjectiveWeightedPerformanceMeasuresEvaluator.

comment:6 Changed 10 years ago by ehopf

r12212: Branched HL.Problems.DataAnalysis to fix a bug in the ClassificationPerformanceMeasuresCalculator.

comment:7 Changed 10 years ago by ehopf

r12213: Adjusted the project settings of the SensitivityEvaluator branch.

comment:8 Changed 10 years ago by ehopf

r12216: Removed redundant calculations.

comment:9 Changed 10 years ago by ehopf

r12218: Added missing estimated values bound calculation and removed duplicate code. Note that the current implementation ignores the linear scaling option.

comment:10 Changed 10 years ago by ehopf

r12302: Added a NotSupported-Exception for the case that the user activates the applyLinearScaling-option. Additionally made some execution time measurements with and without invocation of the ToArray-Extension-Method after the calculation of the bounded estimated values.

Measurements: OSGA -> Symbolic Classification -> Weighted Performance Measures Evaluator -> Seed 962761388

without ToArray(): 1.00:00:54.5801344 2.00:00:53.7666428 3.00:00:53.7478323

with ToArray(): 1.00:00:53.1704467 2.00:00:52.8202198 3.00:00:52.8622417

comment:11 Changed 10 years ago by ehopf

r12309: Adjusted Release Project settings.

comment:12 Changed 10 years ago by ehopf

r12311: Removed a unnecessary method call.

comment:13 Changed 9 years ago by mkommend

Version changed from 3.3.11 to branch

comment:14 Changed 9 years ago by ehopf

r12409: Merged r12358 into branch.

comment:15 Changed 9 years ago by ehopf

Description modified (diff)
Summary changed from Adapted symbolic classification evaluator to increase the sensitivity to Adapted symbolic classification evaluators to increase the sensitivity of a model

comment:16 Changed 9 years ago by ehopf

r12416: Implemented a new experimental evaluator which introduces residual weights to the mean squared error.

comment:17 Changed 9 years ago by ehopf

Owner changed from ehopf to mkommend
Status changed from accepted to reviewing

comment:18 Changed 9 years ago by ehopf

r12417: Changed the type of the ModelCreatorParameter to IValueLookupParameter.

comment:19 Changed 9 years ago by ehopf

r12418: Added backward compatibility code for the WeightedPerformanceMeasuresEvaluator.

comment:20 Changed 9 years ago by ehopf

r12448: Minor naming changes.

comment:21 Changed 9 years ago by ehopf

r12449: Changed the calculation of the error in the OnlineWeightedResidualsMeanSquaredErrorCalculator. The weight is now only applied once to the error.

comment:22 Changed 9 years ago by mkommend

Owner changed from mkommend to ehopf
Status changed from reviewing to assigned

comment:23 Changed 9 years ago by ehopf

r12486: Removed a redundant check.

comment:24 Changed 9 years ago by ehopf

r12487: Fixed a bug that causes an exception when a ModelCreator is used that doesn´t implement the ISymbolicDiscriminantFunctionClassificationModelCreator interface. Note that the runtime can be further improved for those ModelCreators because of redundant calculations.