Opened 5 years ago

Closed 4 years ago

#1924 closed feature request (done)

Evaluator for classification problems that considers classification penalties

Reported by: abeham Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.8
Component: Problems.DataAnalysis.Symbolic.Classification Version: 3.3.8
Keywords: Cc:

Description (last modified by mkommend)

According to mkommend, the penalties are currently only accounted for in the thresholds. An evaluator should be created that calculates the penalty score which is to be minimized by the algorithm.

Change History (14)

comment:1 Changed 5 years ago by abeham

  • Owner changed from mkommend to abeham
  • Status changed from new to accepted

comment:2 Changed 5 years ago by abeham

  • Owner changed from abeham to mkommend
  • Status changed from accepted to reviewing

r8548: Added evaluator

comment:3 Changed 5 years ago by abeham

  • Owner changed from mkommend to abeham
  • Status changed from reviewing to assigned

divide score by evaluated samples

comment:4 Changed 5 years ago by abeham

  • Owner changed from abeham to mkommend
  • Status changed from assigned to reviewing

r8549: Divided penalty by the number of evaluated fitness cases

comment:5 Changed 5 years ago by mkommend

r8551: Adapted evaluator to changes in r8550 (#1919).

Last edited 4 years ago by gkronber (previous) (diff)

comment:6 Changed 5 years ago by mkommend

  • Description modified (diff)

comment:7 Changed 5 years ago by mkommend

r8636:

  • Changed the accuracy threshold calculator to eliminate the necessity that the class values are ordered.
  • Adapted the symbolic classification simplifier to work with all ISymbolicClassificationModels.
  • Corrected ROCCurvesView to also work if the class values are not sorted.
Last edited 4 years ago by gkronber (previous) (diff)

comment:8 Changed 5 years ago by mkommend

  • Owner changed from mkommend to abeham

Please test the changes and forward the ticket to gkronber for review.

comment:9 Changed 5 years ago by mkommend

r8637: Added comment that explains the usage of the cloner in the classification simplifier view.

comment:10 Changed 5 years ago by mkommend

r8646: Corrected Pearson's R² evaluator for classification (estimated values were not limited in quality calculation)

comment:11 Changed 5 years ago by abeham

r8636:8637 would better fit in #1928 in my opinion, r8646 also seems somewhat of a rogue change here.

Anyway, I got an exception when opening the ROC Curves view, I assume as a consquence of r8636:

System.ArgumentException: A chart element with the name 'Class 1' already exists in the 'SeriesCollection'.
   at System.Windows.Forms.DataVisualization.Charting.ChartNamedElementCollection`1.InsertItem(Int32 index, T item)
   at HeuristicLab.Problems.DataAnalysis.Views.DiscriminantFunctionClassificationRocCurvesView.UpdateChart() in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Classification\DiscriminantFunctionClassificationRocCurvesView.cs:line 158
   at HeuristicLab.Problems.DataAnalysis.Views.DiscriminantFunctionClassificationRocCurvesView.OnContentChanged() in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Classification\DiscriminantFunctionClassificationRocCurvesView.cs:line 89
   at HeuristicLab.MainForm.WindowsForms.ContentView.set_Content(IContent value) in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Views\ContentView.cs:line 51
   at HeuristicLab.MainForm.WindowsForms.ViewHost.OnContentChanged() in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Controls\ViewHost.cs:line 146
   at HeuristicLab.MainForm.WindowsForms.ContentView.set_Content(IContent value) in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Views\ContentView.cs:line 51
   at HeuristicLab.Problems.DataAnalysis.Views.DataAnalysisSolutionView.itemsListView_SelectedIndexChanged(Object sender, EventArgs e) in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Solution Views\DataAnalysisSolutionView.cs:line 116
   at System.Windows.Forms.ListView.WmReflectNotify(Message& m)
   at System.Windows.Forms.ListView.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

comment:12 Changed 4 years ago by abeham

  • Owner changed from abeham to gkronber

I did test the changes and didn't find any more issues.

comment:13 Changed 4 years ago by gkronber

  • Status changed from reviewing to readytorelease

Reviewed r8646, the whole evaluation procedure for regression and classification models has been unified in ticket #1951.

Reviewed r8636:8637 which indeed does not fit into this ticket.

Reviewed r8548:8549 and r8551. The changes look good.

In the future, please make sure that correct revision numbers are stated in the ticket comments! I had to change several comments and search correct revision numbers from the time line.

comment:14 Changed 4 years ago by swagner

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.7 to 3.3.8
Note: See TracTickets for help on using tickets.