Opened 5 years ago

Last modified 3 years ago

#1776 assigned feature request

Improve Ensemble Classification with new Voting

Reported by: sforsten Owned by: mkommend
Priority: medium Milestone: HeuristicLab 3.3.x Backlog
Component: Problems.DataAnalysis Version: branch
Keywords: Cc:

Description (last modified by gkronber)

New voting strategies will be implemented to improve the classification ensemble modelling. Currently only majority voting is available. Weighted voting strategies should improve the results achieved with ensemble modelling. It should be possible to change the voting strategy with an additional parameter or combo box.

Decription from #1630: The voting mechanism for ensembles of classifiers based on discriminating functions should be enhanced to weight the vote of a single classifier based on the output of its discriminating function and the selected threshold. This information should also be included into the calculation of a confidence value for each sample.

Change History (31)

comment:1 Changed 5 years ago by sforsten

  • Status changed from new to accepted

comment:2 Changed 5 years ago by sforsten

r7454: create branch for new voting strategies in classification ensemble modeling
r7455: delete project files, which have been accidently branched
r7456: create branch for new voting strategies in classification ensemble modeling (second try)
r7457: add additional project to branch "ClassificationEnsembleVoting"
r7458: branch creation complete

comment:3 Changed 5 years ago by sforsten

r7459: first implementation of different voting strategies (currently no gui elements are available to choose between the strategies)

comment:4 Changed 5 years ago by sforsten

r7464:

  • selection of a voting strategy is now possible
  • a more sophisticated strategy as discussed with gkronber will be implemented soon

comment:5 Changed 5 years ago by sforsten

r7491:

  • two new weight calculators have been implemented
  • one has been deleted, because it didn't work as expected

comment:6 Changed 5 years ago by sforsten

r7504:

  • improvements in the usage of the WeightCalculators
  • small changes in all class which inherit from the WeightCalculator class

comment:7 Changed 5 years ago by sforsten

r7531:

  • 2 more strategies have been implemented
  • major changes in the inheritance have been made to make it possible to add strategies which don't use a voting strategy with weights
  • ClassificationEnsembleSolutionEstimatedClassValuesView doesn't currently show the confidence (has been removed for test purpose)

comment:8 Changed 5 years ago by sforsten

r7549:

  • models can be selected with a check box
  • all strategies are now finished
  • major changes have been made to provide the same behaviour when getting the estimated training or test values of an ensemble

comment:9 Changed 5 years ago by sforsten

r7559:

  • minor bug fix
  • removed unused using

comment:10 Changed 5 years ago by sforsten

r7562:

  • bug fix in NeighbourhoodWeightCalculator
  • added GetConfidence method to IClassificationEnsembleSolutionWeightCalculator
  • adjusted the confidence column in ClassificationEnsembleSolutionEstimatedClassValuesView

comment:11 Changed 5 years ago by sforsten

r7596:

  • added before deleted code to restore some functionality (training and test partition of a model a stored in the .hl file again)
  • project files have been set to release

comment:12 Changed 5 years ago by sforsten

  • Owner changed from sforsten to mkommend
  • Status changed from accepted to reviewing

Comments for the review:
MedianThresholdCalculator, PointCertaintyWeightCalculator and ContinuousPointCertaintyWeightCalculator only work with two class classification.

comment:13 Changed 5 years ago by sforsten

r7729:

  • bugfix the method GetEstimatedValues of DiscriminantClassificationWeightCalculator returns real values and not class values
  • changed arguments of method DiscriminantAggregateEstimatedClassValues of DiscriminantClassificationWeightCalculator
  • added two calculators
Last edited 5 years ago by sforsten (previous) (diff)

comment:14 Changed 5 years ago by sforsten

r7866:

  • merged r7454:7863 trunk into branch to make it compatible again
  • adjusted GetConfidence of MajorityVoteWeightCalculator to be in an interval of [0, 1]

comment:15 Changed 5 years ago by sforsten

r8101:

  • added two calculators to test purposes
  • ClassificationEnsembleSolutionEstimatedClassValuesView shows the current average confidence of the correct and wrong classified samples
  • some calculators have been excluded from the project, so only relevant calculators are shown

comment:16 Changed 5 years ago by sforsten

r8177: minor changes

comment:17 Changed 5 years ago by sforsten

  • Owner changed from mkommend to sforsten
  • Status changed from reviewing to assigned

r8297:

  • Corrected namespace of IClassificationEnsembleSolutionWeightCalculator interface
  • Corrected calculation of confidence for test and training samples in ClassificationEnsembleSolutionEstimatedClassValuesView
  • Added overload method GetConfidence to IClassificationEnsembleSolutionWeightCalculator to calculate more than one point at a time (maybe additional methods for training and test confidence could improve the performance remarkably)
  • Added ClassificationEnsembleSolutionConfidenceAccuracyDependence to see how accuracy would behave, if only samples with high confidence would be classified

I reassigned the ticket to myself, because further changes have to be made, especially to improve the performance.

comment:18 Changed 5 years ago by sforsten

  • Status changed from assigned to accepted

comment:19 Changed 5 years ago by sforsten

r8329:

  • added new diagram which shows correlation of accuracy and covered samples

comment:20 Changed 5 years ago by sforsten

r8507: adapted ClassificationEnsembleSolutionAccuracyToCoveredSamples

comment:21 Changed 5 years ago by sforsten

r8508: merged r7864:8507 trunk into branch

comment:22 Changed 5 years ago by sforsten

r8510: added AUC to ClassificationEnsembleSolutionAccuracyToCoveredSamples

comment:23 Changed 5 years ago by sforsten

r8534:

  • merged r8508:8533 from trunk into branch
  • AverageThresholdCalculator and MedianThresholdCalculator can now handle multi class classification
  • changed combo boxes in ClassificationEnsembleSolutionView to drop down list

comment:24 Changed 5 years ago by gkronber

  • Description modified (diff)

comment:25 Changed 5 years ago by sforsten

r8811:

comment:26 Changed 5 years ago by sforsten

r8814:

  • improved performance of confidence calculation
  • fixed bug in median confidence calculation
  • fixed bug in average confidence calculation
  • confidence calculation is now easier for training and test
  • removed obsolete view ClassificationEnsembleSolutionConfidenceAccuracyDependence

comment:27 Changed 5 years ago by sforsten

r8862: adapted the implementation of ClassificationEnsembleSolutionAccuracyToCoveredSamples to be more similar to ClassificationEnsembleSolutionAccuracyToCoveredSamples

comment:28 Changed 5 years ago by sforsten

r8863:

  • merged r8810:8862 from trunk into branch
  • fixed exception in ClassificationEnsembleSolutionAccuracyToCoveredSamples, if no estimated values are in a partition, so nothing can be shown

comment:29 Changed 4 years ago by gkronber

What is the state of this ticket. Would it be possible to merge parts of the new functionality to the trunk soon?

comment:30 Changed 3 years ago by gkronber

  • Owner changed from sforsten to architects
  • Status changed from accepted to assigned

comment:31 Changed 3 years ago by gkronber

  • Owner changed from architects to mkommend
Note: See TracTickets for help on using tickets.