Data types and operators for classification problems

[mkommend]

Added branch for classification with r4303.

[mkommend]

Corrected classification plugin infrastructure with r4304.

[mkommend]

Updated classification branch with r4323.

[mkommend]

Created branch of HeuristicLab.Problems.DataAnalysis with r4324.

[mkommend]

Corrected probject references of HeuristicLab.Problems.DataAnalysis with r4325.

[mkommend]

Added draft version of classification with r4366.

[mkommend]

Added classification views r4367.

[mkommend]

Updated classification branch with r4391.

[mkommend]

Added SymbolicClassificationPearsonRSquaredEvaluator with r4392.

[mkommend]

Corrected DataAnalysisProblemData ctor with r4393.

[mkommend]

Updated classification views with r4394.

[mkommend]

Deleted outdated plugin Problems.Classification.Views with r4395.

[mkommend]

Adapted regression classes to work with classification plugin with r4415.

[mkommend]

Adapted classification analyzer and readded views project with r4417.

[mkommend]

Removed accidentally commited plugin file with r4450.

[mkommend]

Marked SymbolicClassificationProblem as StorableContent with r4452.

[mkommend]

Added logic to remove the test samples from the training samples with r4469.

[mkommend]

Moved DataAnalysis.Classification from branch to trunk with r4565.

[mkommend]

Corrected handling of ClassNames in the ClassificationProblemData with r4618.

[mkommend]

Deleted classification branch with r4706.

[gkronber]

Misclassification matrix is not regenerated when other parameters for the classification problem are changed (e.g. the target variable).

[gkronber]

List of class names is not regenerated when other parameters of the classification problem are changed (e.g. the target variable, or data set partition).

[gkronber]

TestSamplesStart parameter value is not adapted correctly after importing a new dataset.

[gkronber]

In the validation analyzer the accuracy of the validation-best model is calculated. This can lead to an exception when the model produces NaN estimations.

In symbolic regression NaN values are replaced by the upper estimation bound before accuracy metrics are calculated. I recommend a similar scheme for classification.

OperatorExecutionException: An exception was thrown by the operator "ValidationBestSymbolicClassificationSolutionAnalyzer": Accuracy is not defined for NaN or infinity elements

-----
ArgumentException: Accuracy is not defined for NaN or infinity elements
   at HeuristicLab.Problems.DataAnalysis.Classification.OnlineAccuracyEvaluator.Add(Double original, Double estimated)
   at HeuristicLab.Problems.DataAnalysis.Classification.ValidationBestSymbolicClassificationSolutionAnalyzer.UpdateBestSolutionResults()
   at HeuristicLab.Problems.DataAnalysis.Classification.ValidationBestSymbolicClassificationSolutionAnalyzer.Apply()
   at HeuristicLab.Operators.Operator.Execute(IExecutionContext context)
   at HeuristicLab.SequentialEngine.SequentialEngine.ProcessNextOperation()

[gkronber]

Calculation or display of the ROC curve is very slow for slightly larger datasets (>2000 samples).

[gkronber]

Please reassign me the ticket for review when the issues discussed above have been addressed. Thanks.

[mkommend]

Corrected updating of ClassificationProblemData parameters after data import with r4780.

[mkommend]

Replying to gkronber:

Misclassification matrix is not regenerated when other parameters for the classification problem are changed (e.g. the target variable).

Corrected with r4780.

[mkommend]

Replying to gkronber:

List of class names is not regenerated when other parameters of the classification problem are changed (e.g. the target variable, or data set partition).

Corrected with r4780.

[mkommend]

Replying to gkronber:

TestSamplesStart parameter value is not adapted correctly after importing a new dataset.

Corrected with r4780.

[mkommend]

Corrected adaption of the test range with r4781.

[mkommend]

Replying to gkronber:

In the validation analyzer the accuracy of the validation-best model is calculated. This can leads to an exception when the model produces NaN estimations.

In symbolic regression NaN values are replaced by the upper estimation bound before accuracy metrics are calculated. I recommend a similar scheme for classification.

OperatorExecutionException: An exception was thrown by the operator "ValidationBestSymbolicClassificationSolutionAnalyzer": Accuracy is not defined for NaN or infinity elements

-----
ArgumentException: Accuracy is not defined for NaN or infinity elements
   at HeuristicLab.Problems.DataAnalysis.Classification.OnlineAccuracyEvaluator.Add(Double original, Double estimated)
   at HeuristicLab.Problems.DataAnalysis.Classification.ValidationBestSymbolicClassificationSolutionAnalyzer.UpdateBestSolutionResults()
   at HeuristicLab.Problems.DataAnalysis.Classification.ValidationBestSymbolicClassificationSolutionAnalyzer.Apply()
   at HeuristicLab.Operators.Operator.Execute(IExecutionContext context)
   at HeuristicLab.SequentialEngine.SequentialEngine.ProcessNextOperation()

Could not reproduce this bug and also checked the source code if !NaN elements are filtered. Could you attach a sample or test againg of the bug still exists.

[mkommend]

Corrected SymbolicRegressionSolution to use UpperEstimationLimit instead double.NaN with r4797.

[gkronber]

The order of variables occurring in the target variable drop down box does not match the order of variables in the original dataset. This also has the effect that a random variable is initially selected as the target variable.

[gkronber]

For binary classification problems the AUC for the two classes, displayed in the legend tooltip in the ROC curve view does not match. Shouldn't the two AUC values be the same when only two classes are available?

[gkronber]

A demo for classification would be nice.

[gkronber]

Probably the attached file would be a nice demo.

[gkronber]

SGP symbolic classification - mammographic dataset (UCI)

[gkronber]

In the symbolic classification view, the class visibility is reset when the content is set. This is a bit annoying.

[gkronber]

The automatic calculation of the separator value for the discriminating function produces strange results (see attached screenshot).

[swagner]

Replying to gkronber:

Probably the attached file would be a nice demo.

Please feel free to add this sample to the samples in the optimizer.

[mkommend]

Replying to gkronber:

The order of variables occurring in the target variable drop down box does not match the order of variables in the original dataset. This also has the effect that a random variable is initially selected as the target variable.

Corrected ordering of valid target variables with r4836.

[mkommend]

Corrected ValidationBestSymbolicClassificationSolutionAnalyzer with r4837.

[mkommend]

Replying to gkronber:

In the symbolic classification view, the class visibility is reset when the content is set. This is a bit annoying.

Although this is annoying there is nothing we can do against it, because the view must be reset if a new content is set. There was also a bug in the solution analyzer that got fixed with r4837 and now the best solution is updated correctly and less often => the reset occurs less frequent.

[gkronber]

I really don't like the random order of variables in the target variable drop down. Also as soon as there are binary input variables the target variable selection heuristic fails.

However, this is only my subjective opinion and this issue does not necessarily have to be fixed for the 3.3.2 release. (fixed in r4836)

Everything else looks great, thanks for implementing this plugin!

Feel free to release the plugin and close this ticket.

[gkronber]

Updated symbolic classification with SGP demo with r4884.