Opened 7 years ago

Closed 12 months ago

#1087 closed feature request (done)

Implement multiobjective test functions

Reported by: abeham Owned by: mkommend
Priority: high Milestone: HeuristicLab 3.3.14
Component: Problems.TestFunctions Version: 3.3.14
Keywords: Cc: b.werth@gmx.at

Description

There are a number of popular test functions for multiobjective optimization that would fit in the scope of Problems.TestFunctions.

These functions are, e.g. ZDT1-6 (except ZDT5 which uses binary representation), Kursawe, Golinski, Schaffer, Fonseca, DTLZ1-7, ... with some of them having constraints as well.

Change History (61)

comment:1 Changed 7 years ago by swagner

  • Summary changed from Implement Problems: Multiobjective Testfunctions to Implement multiobjective test functions

comment:2 Changed 7 years ago by swagner

  • Owner changed from abeham to gkronber
  • Status changed from new to assigned

comment:3 Changed 4 years ago by gkronber

  • Owner changed from gkronber to swagner
  • Priority changed from medium to high

comment:4 Changed 21 months ago by mkommend

  • Owner changed from swagner to bwerth

comment:5 Changed 20 months ago by mkommend

  • Version changed from 3.3 to branch

comment:6 Changed 20 months ago by mkommend

r13394: Created new plugin for multi-objective test functions.

comment:7 Changed 20 months ago by bwerth

r13421: Implemented skeleton structure and first testfunctions(Fonesca, SchafferN1 & SchafferN2)

comment:8 Changed 20 months ago by bwerth

r13448 Implemented more Testfunctions; enabled setting of SolutionSize

comment:9 Changed 20 months ago by bwerth

r13451 extensive Testing; fixed minor bugs

Last edited 20 months ago by bwerth (previous) (diff)

comment:10 Changed 19 months ago by bwerth

r13515 minor bugfixes and added unittests

comment:11 Changed 18 months ago by bwerth

r13562 added more unittests and comments

comment:12 Changed 18 months ago by mkommend

r13563: Added reference to System.Drawing in HL.Problems.MultiObjectiveTestFunctions.

comment:13 Changed 18 months ago by bwerth

r13620 regorganized testfunctions, added view for qualities

comment:14 Changed 18 months ago by bwerth

r13622 recommited project file, renamed IMOSolution to MOSolution

Last edited 18 months ago by bwerth (previous) (diff)

comment:15 Changed 18 months ago by mkommend

r13623: Added instance provider base class for MOTestFunctions.

comment:16 Changed 17 months ago by bwerth

r13672 added Analyzers, reworked PFStore, added licence information, cleaned code

comment:17 Changed 17 months ago by bwerth

r13673 fixed bug in DTLZ1, DTLZ3 and DTLZ7

comment:18 Changed 16 months ago by bwerth

r13725 minor bugfixes, added Parameters to Analyzers, convenience Tooltips for ScatterPlot

comment:19 Changed 16 months ago by bwerth

  • Owner changed from bwerth to mkommend
  • Status changed from assigned to reviewing

comment:20 Changed 16 months ago by mkommend

  • Milestone changed from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.14

comment:21 Changed 16 months ago by bwerth

r13726 fixed Naming in FastHyperVolumeTests.cs

comment:22 Changed 16 months ago by bwerth

r13729 annotated unit tests

comment:23 Changed 15 months ago by bwerth

r13771 bugfix + additional relative HV calculation added

comment:24 Changed 15 months ago by bwerth

r13776 added ConstrainedFunctions

Last edited 15 months ago by bwerth (previous) (diff)

comment:25 Changed 14 months ago by bwerth

r13894 set pareto-Series to lower Opacity so overlapping series can be identified

comment:26 Changed 13 months ago by bwerth

r13936 added more functions (IHR1-4, IHR6, CIGTAB, ELLI)

comment:27 Changed 13 months ago by bwerth

r13988 reformatted project, bugfixes

comment:28 Changed 13 months ago by bwerth

r13989 removed unused usings

comment:29 Changed 13 months ago by bwerth

r13991 removed unused function

comment:30 Changed 13 months ago by mkommend

r14018: Rewrote and adapted the multi-objective calculators.

comment:31 Changed 13 months ago by mkommend

Review comments

I only had time to review and work on the calculator classes. Review of remaining classes and functionality is still pending.

Calculators

  • Inconsistent usage of points and front for member names. Please check and unify.
  • GenerationalDistance.cs
    • Is the comment "TODO build a kd-tree, sort the array, do something intelligent here" still valid
      • Why has not been something intelligent done?
    • How should the parameter p be set? A comment (XMLDoc) on this would be helpful
      • The method is everywhere called with p == 1.0
    • IMHO the Calculate method calculates math(1/n (\sum d[i])^{1/p})? which is not math(1/n \sum d[i]^{1/p})? as described in the XMLDoc.
      • This is only true for p == 1.0.
  • HyperVolume.cs
    • Does Calculate2D work with differing maximization values (e.g., min dim0, max dim1)?
    • CalculateMD should reuse the domination functionality of NonDominatedSelect (line 97)
    • IsDominated must be removed. The method provides little to no benefits.
  • Crowding.cs
    • No unit tests
    • There should be a method that calculates the crowding distance between a point and a set of points / front. This method should then be reused for calculating the crowding distance of set of points / front. The crowding distance calculation should yield the same results as the CrowdingDistanceAssignment operator of the NSGA-II!
    • Why have inifinity values to be excluded while calculating the average? Is it normal for the individual calculation to result in infinity?
    • Line 58 - 67 can be rewritten as: return pointsums.Where(d => !double.IsPositiveInfinity(d)).DefaultIfEmpty(double.PositiveInfinity).Average();
  • What is the difference between HyperVolume.CalculateMD and MultiDimensionalHypervolume?
    • The code looks the same to me.

comment:32 Changed 13 months ago by mkommend

  • Cc b.werth@gmx.at added

comment:33 Changed 13 months ago by bwerth

r14030 several fixes according to the review comments in comment 31

Last edited 13 months ago by mkommend (previous) (diff)

comment:34 Changed 13 months ago by mkommend

r14044: Refactored and improved analyzers for multi-objective test functions.

comment:35 Changed 13 months ago by mkommend

Review of Analyzers

  • Why is the BestKnownFrontParameter defined in the MOTFAnalyzer, although it is only used by the hyper volume analyzers?
  • HypervolumeAnalyzer
    • Why is the best known hyper volume saved in a parameter of the Analyzer? Should be in the TestFunction / Problem.
    • The ReferencePoint should be taken from the TestFunction (no value parameter).
    • Reuse DoubleValues by setting their values instead of creating new ones (results). Similar to the changes in r14044.
    • Could NonDominatedSelect.selectNonDominatedVectors (line 93) return an empty enumberable? If not the check in line 95 is obsolete.
  • NormalizedHypervolumeAnalyzer
    • Most of the comments for the hypervolume analyzers apply to this class as well.
    • BestKnownFront Zitzler ?!?
    • Always use FixedValueParameters for ValueParameters<T> where T in HeuristicLab.Data and T can be modified.

comment:36 Changed 13 months ago by mkommend

r14065: Refactored instance providers and data descriptors for MOTF. r14066: Forgot to commit the project file.

Last edited 13 months ago by mkommend (previous) (diff)

comment:37 Changed 13 months ago by mkommend

r14067: Refactored multi-objective test functions. r14068: Added checks for min and max objectives to test functions.

Last edited 13 months ago by mkommend (previous) (diff)

comment:38 Changed 13 months ago by mkommend

Review of TestFunctions

  • IHR
    • What is the purpose of the Method Y protected RealVector Y(RealVector x) { return x; }
    • Why is the method G defined as abstract and implemented in derived classes, but never called?
    • F2 never uses its first parameter r
    • Evaluate calls F2(r,y) where y = Y(r) and Y just returns its argument. Therefore y == r. Why does F2 need two arguments at all?
    • IHR5 ?
  • ELLI
    • Line 70 x is never used.
  • SchafferN2
    • Optimal ParetoFront throws NotImplementedException!
  • ZDT5 ?
Last edited 12 months ago by mkommend (previous) (diff)

comment:39 Changed 13 months ago by mkommend

r14069: Moved Pareto fronts to subfolders in the according test functions folder and changed PFStore.

comment:40 Changed 13 months ago by mkommend

r14073: Worked on Multi-objective test function problem and adapted plugin dependencies.

comment:41 Changed 13 months ago by mkommend

r14085: Further refactoring of testfunction problem and analyzers.

comment:42 Changed 13 months ago by bwerth

r14090 several fixes according to the review comments 35 and 38

comment:43 Changed 12 months ago by bwerth

r14092 removed NormalizedHypervolumeAnalyzer and IMOFrontModel.cs, refactored ScatterPlotAnalyzer ,fixed bug in HypervolumeAnalyzer

comment:44 Changed 12 months ago by bwerth

r14093 altered Cloning in ScatterPlotContent

Last edited 12 months ago by bwerth (previous) (diff)

comment:45 Changed 12 months ago by bwerth

r14097 refactored Analyzers to use ResultParamters

comment:46 Changed 12 months ago by mkommend

r14108: Fixed bugs in Spacing and Hypervolume calculation, corrected plugin dependencies and minor changes to the scatter plot view.

comment:47 Changed 12 months ago by mkommend

r14111: Change plugin and folder name from HeuristicLab.Problems.MultiObjectiveTestFunctions to HeuristicLab.Problems.Testfunctions.MultiObjective and adapted namespaces and projects accordingly.

Persistence break in the branch for those already using the multi-objective test functions!

comment:48 Changed 12 months ago by mkommend

r14120: Integrated multi-objective test functions in the trunk.

comment:49 Changed 12 months ago by mkommend

r14121: Copied unit tests from branch and adapted path of multi-objective test function project file.

comment:50 Changed 12 months ago by mkommend

  • Version changed from branch to 3.3.14

comment:51 Changed 12 months ago by mkommend

r14122: Renamed Testfunctions folder to TestFunctions to comply to name reference in the solution and that the compile works also with case-sensitive file systems (e.g., Linux).

comment:52 Changed 12 months ago by bburlacu

r14124, r14125: Accidental commit, reverted. r14126: Remove duplicate PreBuildEvent in the project file.

Last edited 12 months ago by bburlacu (previous) (diff)

comment:53 Changed 12 months ago by mkommend

r14127: Readded accidentally deleted TestFunctions.

comment:54 Changed 12 months ago by bburlacu

r14130: Remove HintPath for DataVisualization dll and unnecessary source control xml tags from project file.

comment:55 Changed 12 months ago by pfleck

r14132 Corrected namespace of the MultiObjectiveTestFunctionParetoFrontScatterPlotView.

comment:56 Changed 12 months ago by mkommend

This ticket depends on #2281 and #2594.

The following changesets must be merged to stable: 14120, 14121, 14122, 14124, 14125, 14126, 14127, 14130, 14132

Last edited 12 months ago by mkommend (previous) (diff)

comment:57 Changed 12 months ago by mkommend

  • Status changed from reviewing to readytorelease

comment:58 Changed 12 months ago by pfleck

r14150 Changed BestKnownFront parameter to OptionalValueParameter to allow null if no best front is known (DTLZ8 and SchafferN2 and DTLZs with objectives > 2).

comment:59 Changed 12 months ago by mkommend

r14164: Reverse merged changes in r14125 to hive solution and settings.

comment:60 Changed 12 months ago by mkommend

comment:61 Changed 12 months ago by mkommend

  • Resolution set to done
  • Status changed from readytorelease to closed
Note: See TracTickets for help on using tickets.