Free cookie consent management tool by TermsFeed Policy Generator

Opened 12 years ago

Closed 11 years ago

#1924 closed feature request (done)

Evaluator for classification problems that considers classification penalties

Reported by: abeham Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.8
Component: Problems.DataAnalysis.Symbolic.Classification Version: 3.3.8
Keywords: Cc:

Description (last modified by mkommend)

According to mkommend, the penalties are currently only accounted for in the thresholds. An evaluator should be created that calculates the penalty score which is to be minimized by the algorithm.

Change History (14)

comment:1 Changed 12 years ago by abeham

  • Owner changed from mkommend to abeham
  • Status changed from new to accepted

comment:2 Changed 12 years ago by abeham

  • Owner changed from abeham to mkommend
  • Status changed from accepted to reviewing

r8548: Added evaluator

comment:3 Changed 12 years ago by abeham

  • Owner changed from mkommend to abeham
  • Status changed from reviewing to assigned

divide score by evaluated samples

comment:4 Changed 12 years ago by abeham

  • Owner changed from abeham to mkommend
  • Status changed from assigned to reviewing

r8549: Divided penalty by the number of evaluated fitness cases

comment:5 Changed 12 years ago by mkommend

r8551: Adapted evaluator to changes in r8550 (#1919).

Last edited 11 years ago by gkronber (previous) (diff)

comment:6 Changed 12 years ago by mkommend

  • Description modified (diff)

comment:7 Changed 12 years ago by mkommend

r8636:

  • Changed the accuracy threshold calculator to eliminate the necessity that the class values are ordered.
  • Adapted the symbolic classification simplifier to work with all ISymbolicClassificationModels.
  • Corrected ROCCurvesView to also work if the class values are not sorted.
Last edited 11 years ago by gkronber (previous) (diff)

comment:8 Changed 12 years ago by mkommend

  • Owner changed from mkommend to abeham

Please test the changes and forward the ticket to gkronber for review.

comment:9 Changed 12 years ago by mkommend

r8637: Added comment that explains the usage of the cloner in the classification simplifier view.

comment:10 Changed 12 years ago by mkommend

r8646: Corrected Pearson's R² evaluator for classification (estimated values were not limited in quality calculation)

comment:11 Changed 12 years ago by abeham

r8636:8637 would better fit in #1928 in my opinion, r8646 also seems somewhat of a rogue change here.

Anyway, I got an exception when opening the ROC Curves view, I assume as a consquence of r8636:

System.ArgumentException: A chart element with the name 'Class 1' already exists in the 'SeriesCollection'.
   at System.Windows.Forms.DataVisualization.Charting.ChartNamedElementCollection`1.InsertItem(Int32 index, T item)
   at HeuristicLab.Problems.DataAnalysis.Views.DiscriminantFunctionClassificationRocCurvesView.UpdateChart() in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Classification\DiscriminantFunctionClassificationRocCurvesView.cs:line 158
   at HeuristicLab.Problems.DataAnalysis.Views.DiscriminantFunctionClassificationRocCurvesView.OnContentChanged() in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Classification\DiscriminantFunctionClassificationRocCurvesView.cs:line 89
   at HeuristicLab.MainForm.WindowsForms.ContentView.set_Content(IContent value) in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Views\ContentView.cs:line 51
   at HeuristicLab.MainForm.WindowsForms.ViewHost.OnContentChanged() in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Controls\ViewHost.cs:line 146
   at HeuristicLab.MainForm.WindowsForms.ContentView.set_Content(IContent value) in D:\HL3\trunk\sources\HeuristicLab.MainForm.WindowsForms\3.3\Views\ContentView.cs:line 51
   at HeuristicLab.Problems.DataAnalysis.Views.DataAnalysisSolutionView.itemsListView_SelectedIndexChanged(Object sender, EventArgs e) in D:\HL3\trunk\sources\HeuristicLab.Problems.DataAnalysis.Views\3.4\Solution Views\DataAnalysisSolutionView.cs:line 116
   at System.Windows.Forms.ListView.WmReflectNotify(Message& m)
   at System.Windows.Forms.ListView.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

comment:12 Changed 11 years ago by abeham

  • Owner changed from abeham to gkronber

I did test the changes and didn't find any more issues.

comment:13 Changed 11 years ago by gkronber

  • Status changed from reviewing to readytorelease

Reviewed r8646, the whole evaluation procedure for regression and classification models has been unified in ticket #1951.

Reviewed r8636:8637 which indeed does not fit into this ticket.

Reviewed r8548:8549 and r8551. The changes look good.

In the future, please make sure that correct revision numbers are stated in the ticket comments! I had to change several comments and search correct revision numbers from the time line.

comment:14 Changed 11 years ago by swagner

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.7 to 3.3.8
Note: See TracTickets for help on using tickets.