Opened 2 years ago

Last modified 8 weeks ago

#2431 accepted feature request

Create algorithm iterator optimizer

Reported by: abeham Owned by: abeham
Priority: medium Milestone: HeuristicLab 3.3.16
Component: Analysis Version: branch
Keywords: Cc:

Description (last modified by abeham)

The algorithm iterator should execute an algorithm instance over and over until a certain stopping criterion is met. The stopping criteria are:

  • Maximum execution time reached
  • Maximum number of evaluated solutions
  • A certain target fitness was achieved

Design issues:

  • Algorithm Execution time is hard to obtain (similar problem with the introduction of terminators)
    • The precision of execution time from the algorithm is not good enough (~200ms)
  • Adding EvaluatedMoves and EvaluatedSolutions is complicated by different cost factors
    • Some algorithms/operators count evaluated solution-equivalents instead
    • Integer is sometimes too small to hold the amount of evaluated moves for long runs / big problems
  • Analyzers depend on BestAverageWorstQualityAnalyzer being run before them
    • Arguably, best-so-far quality and execution time are results that the algorithm itself should provide and not an analyzer
    • Quality progress chart should be calculated in a separate analyzer
  • ISingleObjectiveHeuristicOptimizationProblem specifies MaximizationParameter as an IParameter instead of an IValueParameter<BoolValue>
  • Which parameters in a run characterize the algorithm instance?
    • E.g. Seed does not characterize the instance
    • Analyzers do not characterize the instance, but some "analyzers" may modify penalties
  • ResultCollection contains Result objects, but RunCollection.Results contains only the items
    • The Result objects are always in the way, they should be removed

Change History (44)

comment:1 Changed 2 years ago by abeham

  • Status changed from new to accepted
  • Version changed from 3.3.12 to branch

r12764:

  • branched relevant plugins
  • added analyzer to calculate quality vs evaluation first-hit graph
  • (also added results parameter)

comment:2 Changed 2 years ago by abeham

r12766: forgot to commit

comment:3 Changed 2 years ago by abeham

r12771:

  • Added run collection view
  • Changed name of analyzers
  • Modified algorithms to include Execution Time as a result

Logscaling is still an issue

comment:4 Changed 2 years ago by abeham

  • Description modified (diff)

comment:5 Changed 2 years ago by abeham

  • Description modified (diff)

r12774: minor improvements

comment:6 Changed 2 years ago by abeham

  • Description modified (diff)
  • Summary changed from Algorithm performance comparison by ECDF to Run-length distribution (RLD) Analysis

comment:7 Changed 2 years ago by abeham

r12803: worked on RLD analysis

  • started implementation of IRRRun
  • renamed view from ECDF to RLD
  • reverted algorithms (execution time)
  • changed per clock analyzer

comment:8 Changed 2 years ago by abeham

r12804:

  • worked on IRRestarter (early abort still troublesome)
  • Updated RLD view to allow defining targets
  • Attempting to handle maximization/minimization

Fun detail:

  • HeuristicLab.Analysis references HeuristicLab.Optimization
  • HeuristicLab.Optimization.Views references HeuristicLab.Analysis.Views

IRRestarter is currently placed in HeuristicLab.Analysis, its view in HeuristicLab.Optimization.Views.

comment:9 Changed 2 years ago by abeham

  • Description modified (diff)

comment:10 Changed 2 years ago by abeham

r12805: fixed error due to removed resx file

comment:11 Changed 2 years ago by abeham

r12806: added options to add results based on targets or budgets

comment:12 Changed 2 years ago by abeham

r12808:

  • added ability to plot curves for multiple targets at once
  • fixed bug in IndexedDataTableView regarding log-scaling
  • added result that counts restarts
  • fixed bugs in IRRestarter
  • Set LineWidth = 2 in analyzers for chart
  • fixed bugs in RLD view

comment:13 Changed 2 years ago by abeham

  • Description modified (diff)

comment:14 Changed 2 years ago by abeham

r12813:

  • Added calculation of RTs, RTus, FEs, FEus and prune runs in alg before every restart
  • merged changes from trunk

comment:15 Changed 2 years ago by abeham

r12822: if multiple targets are plotted, the same color is used for each group

comment:16 Changed 2 years ago by abeham

r12825:

  • removed output of RTs, RTus, ...
  • added 2nd row to charts that state restarts until a target was achieved
  • added option to include or exclude solution results

comment:17 Changed 2 years ago by abeham

r12826: omitting empty series

comment:18 Changed 2 years ago by abeham

r12831: updated to trunk

comment:19 Changed 2 years ago by abeham

r12834: Changed analyzer to always output last value (even if quality is equal)

comment:20 Changed 2 years ago by abeham

r12838: updated RLD analysis view

  • Added analysis by cost
  • Prepared tab for output of ERT tables

comment:21 Changed 2 years ago by abeham

r12841:

  • Added calculation of ERT results tables
  • Added grouping by problem to avoid having different problems in one graph
  • Some restructuring to avoid duplicate code

comment:22 Changed 2 years ago by abeham

r12856: Added AlgorithmIterator instead of IRRestarter

  • AlgorithmIterator implements IAlgorithm whereas IRRestarter implemented only IOptimizer
  • Deriving from Algorithm leads to better design

comment:23 Changed 2 years ago by abeham

r12859: Renamed AlgorithmIterator to IteratedAlgorithm

comment:24 Changed 2 years ago by abeham

r12862: modified EngineAlgorithm to clear log on prepare

comment:25 Changed 2 years ago by abeham

  • Description modified (diff)

comment:26 Changed 2 years ago by abeham

r12864:

  • fixed bug in iterated algorithm
  • changed RLD view to use % deviation as targets
  • RLD view now always groups by problem

comment:27 Changed 2 years ago by abeham

r12865:

  • Added label on last data point
  • Changed target in cost view to display as ratio to best-known or best found

comment:28 Changed 2 years ago by abeham

r12888:

  • Some bug fixes

comment:29 Changed 2 years ago by abeham

r12890: fixed a bug in the iterated algorithm

comment:30 Changed 2 years ago by abeham

r12939: fixed calculation of ERT table when a certain group doesn't contain any runs of a certain problem

comment:31 Changed 2 years ago by abeham

r12956: Extracted class that calculates ERT values from runs

comment:32 Changed 21 months ago by ascheibe

  • Milestone changed from HeuristicLab 3.3.13 to HeuristicLab 4.0

comment:33 Changed 21 months ago by abeham

  • Milestone changed from HeuristicLab 4.0 to HeuristicLab 3.3.14

comment:34 Changed 20 months ago by abeham

r13475: updated from trunk, worked on okb connection for downloading knowledge base

comment:35 Changed 17 months ago by abeham

r13736: fixed analysis for runs that terminated earlier than others

comment:36 Changed 17 months ago by abeham

r13737:

  • removed IRRestarter (replaced by IteratedAlgorithm)
  • removed cut-off estimation view (didn't work satisfactorily)

comment:37 Changed 17 months ago by abeham

r13738: fixed bug in IteratedAlgorithm (exception when problem is loaded, but no algorithm set)

comment:38 Changed 13 months ago by abeham

r14035:

  • Fixed bug that occurs when runs of unequal length are aggregated over multiple problem instances

comment:39 Changed 13 months ago by abeham

  • Description modified (diff)

r14058: Refactored ResultsParameter

comment:40 Changed 13 months ago by abeham

r14059: Removed NonDiscoverableType attribute on ResultParameter

comment:41 Changed 13 months ago by abeham

r14070: fixed some bugs in the result parameter view (thx to mkommend)

comment:42 Changed 13 months ago by abeham

  • Description modified (diff)
  • Summary changed from Run-length distribution (RLD) Analysis to Create algorithm iterator optimizer

comment:43 Changed 13 months ago by abeham

  • Milestone changed from HeuristicLab 3.3.14 to HeuristicLab 3.3.15

comment:44 Changed 8 weeks ago by abeham

  • Milestone changed from HeuristicLab 3.3.15 to HeuristicLab 3.3.16
Note: See TracTickets for help on using tickets.