Model Comparison for Classification

[sforsten]

Similar to Regression, it shall be possible to easily create simple models for classification, which a classification model has to exceed to be accepted as good model.

0R, 1R and LDA classifiers will be created to compare them to the hopefully better classification model.

[sforsten]

r9069: initial commit of new branch

r9070: branch project HeuristicLab.Algorithms.DataAnalysis

r9071: finished preparing branch

[sforsten]

r9073: branch project Problems.DataAnalysis

r9074:

• added ZeroR and OneR classifiers
• added ConstantClassificationModel/-Solution and OneRClassificationModel/-Solution

A view to display the OneRClassificationModel has to be added.

[sforsten]

r9116: branch project Problems.DataAnalysis.Views

r9117: branch project Algorithms.DataAnalysis.Views

r9119:

• added OneRClassificationModelView
• added ClassificationSolutionComparisonView
• added several calculators (ConfusionMatrixCalculator, FOneScoreCalculator, MatthewsCorrelationCoefficientCalculator)
• fixed bug in OneR
• added StorableClass and Item attribute to several classes

[sforsten]

r9135:

• OneR now handles missing values separately
• adapted OneRClassificationModelView to show the class of missing values
• with a double-click on the row header in ClassificationSolutionComparisonView the selected solution opens in a new view
• put a try catch block around linear discriminant analysis solution (it is only shown, if it doesn't throw an exception)

ZeroR may not be a good model in the ClassificationSolutionComparisonView, because MatthewsCorrelationCoefficientCalculator will always calculate NaN and F1 score will also be NaN if the negative class is chosen by ZeroR.

[gkronber]

Can we queue this for the next release version?

[gkronber]

Discussed in architects meeting, should be integrated into the trunk.

[mkommend]

r10553: Updated classification model comparison branch with trunk changes.

[mkommend]

r10556: Updated classification model comparison branch with trunk changes (remaining changes).

[mkommend]

r10560: Fixed bugs in classification solution comparison view.

[mkommend]

r10568: Code cleanup in ZeroR classification algorithm.

[mkommend]

r10569: Reimplemented OneR classification algoithm.

[mkommend]

r10570: Added missing value handling in new implementation of OneR.

[mkommend]

Please review r10560, r10568:10570. All other changesets have already been reviewed.

For comparison reasons I left the original version of the OneR algorithm untouched.

[gkronber]

Reviewed r10560 changes are OK:

Made a more detailed review of the source code. Notes from the overall source code review:

• Adapt to work with current trunk version (export button).
• Exception occurs when running a linear discriminant analysis model (works for the trunk) cannot reproduce
• Move OneR and ZeroR out of the Algorithms.DataAnalysis.Linear folder. done
• OneR and ZeroR should not be available in the "New Item" dialog. done
• ExecutionTime for OneR should not be produced as a result. I guess this is ok.
• The file OneRTest.cs can be deleted? No, the OneRTest.cs file is the new implementation by mkommend. The file OneR.cs should be deleted

[gkronber]

Not yet ready for release

[gkronber]

Sorry

[gkronber]

r13081: changed framework version to v4.5

[gkronber]

r13082: merged changesets r10553:13081 (only on HeuristicLab.Problems.DataAnalysis.Views) from trunk to branch r13083: merged changesets r10551:13082 (only on HeuristicLab.Problems.DataAnalysis) from trunk to branch r13084: merged changesets r10551:13083 (only on HeuristicLab.Algorithms.DataAnalysis.Views) from trunk to branch r13085: merged changesets r10551:13084 (only on HeuristicLab.Algorithms.DataAnalysis) from trunk to branch

[gkronber]

r13086: made compatibility changes necessary because of trunk developments (compile fail)

[gkronber]

r13089

• deleted obsolete version of OneR algorithm (also does perform worse than mkommend's implementation in my tests)
• reused the ConstantRegressionModel as ConstantClassificationModel (OK?)
• fixed a few strings here and there

[gkronber]

r13090: moved OneR and ZeroR out of the folder for linear models, renamed OneRTest -> OneR

[gkronber]

r13091: minor changes while reviewing

Reviewed changes in

• HeuristicLab.Problems.DataAnalysis (online calculators of F1-score and Matthews correlation have been added)
• HeuristicLab.Algorithms.DataAnalysis (ZeroR and OneR algorithms have been added)
• HeuristicLab.Problems.DataAnalysis.Views (only ClassificationSolutionComparisonView has been added)
• HeuristicLab.Algorithms.DataAnalysis.Views (only the OneRClassificationModelView has been added)

TODO:

• don't calculate F1 score for multi-class problems
• terminate branch
• more detailed review of online calculators and algorithms
• add F1 score and Matthew's correlation to classification results collection
• mark ConstantRegressionModel obsolete and create new class: ConstantModel

[gkronber]

r13092: removed creatable attribute from OneR and ZeroR

[gkronber]

r13097: reverse merge of 13089 (reused the ConstantRegressionModel as ConstantClassificationModel (OK?))

[gkronber]

r13098:

• introduced new class ConstantModel (to merge ConstantRegressionModel, ConstantClassificationModel and ConstantTimeSeriesModel)
• fixed copyright statements
• tried to unify naming
• added F1 score and Matthews correlation to classification results collection

[gkronber]

r13099: fixed strings (the person is called Matthews)

[gkronber]

r13100: merged changes from the branch to trunk (btw. this branch was difficult to merge back to trunk because of it's specific structure)

[gkronber]

r13101: bug fixes (typo, duplicate result item)

[gkronber]

r13102:

• changed namespace and name of view
• calculate f1 score only for solutions for binary classification problems

[gkronber]

r13103: code simplification of ConfusionMatrixCalculator

[gkronber]

r13104: fixed a problem in Classification/ClassificationSolutionComparisonView

[gkronber]

Relevant new changes:

• r13086, r13089:13092, r13097:13104

[mkommend]

Reviewed and briefly tested all relevant changes (mostly the model comparison view, ZeroR and OneR algorithm and the new classification performance metrics). Thank you for reviewing and finishing the implementation.

The only review comment I have is to put the ConstantModel outside of the Regression folder, because it is not limited to regression anymore.

[gkronber]

r13154: moved ConstantModel class out of the Regression folder

[gkronber]

r13155: terminated the old feature branch for model comparison

[gkronber]

r13156: merged r13100:13104 and r13154 from trunk to stable