Run-length distribution (RLD) Analysis

[abeham]

Run-length distributions give insight into the required algorithmic effort (in terms of function evaluations or wall-clock time) for reaching a certain target. These distributions can be

The empirical cumulative distribution function can be calculated from several runs of an optimization algorithm. From a first-hit graph several targets will be defined in the objective function and it will be measured when a target is hit by an algorithm. The proportion of all these hits forms the empirical cumulative distribution function (ECDF).

Necessary tasks: 1. Create analyzers that calculate the quality progress with respect to function evaluations and also with respect to wall-clock time 2. Create a run collection view that will calculate and compare the ECDF in one chart

TODO:

• ECDF comparison view:
  • Minimization is assumed when comparing against levels -> extract Maximization parameter from run
  • Logscaling sometimes fails breaking the chart
  • Test with view open and new runs arriving
  • Add feature to calculate fixed-cost and fixed-target values and add them to the runs
• IRRestarter:
  • Count number of restarts
  • Difficult to abort Algorithm earlier as the Analyzers may not have run

Design issues:

• Algorithm Execution time is hard to obtain (similar problem with the introduction of terminators)
  • The precision of execution time from the algorithm was not good enough (~200ms), so the analyzer counts execution time by itself
• ResultParameter's ActualValue property is confusing, I had to lookup usage of this parameter myself
• Move-based algorithms often output EvaluatedMoves, we should probably have all algorithms output EvaluatedSolutions
  • Instead, I added a cost factor for moves
• Analyzers depend on BestAverageWorstQualityAnalyzer, in my opinion best-so-far quality and execution time are results that the algorithm itself must provide and not an analyzer

[abeham]

r12764:

• branched relevant plugins
• added analyzer to calculate quality vs evaluation first-hit graph
• (also added results parameter)

[abeham]

r12766: forgot to commit

[abeham]

r12771:

• Added run collection view
• Changed name of analyzers
• Modified algorithms to include Execution Time as a result

Logscaling is still an issue

[abeham]

r12774: minor improvements

[abeham]

r12803: worked on RLD analysis

• started implementation of IRRRun
• renamed view from ECDF to RLD
• reverted algorithms (execution time)
• changed per clock analyzer

[abeham]

r12804:

• worked on IRRestarter (early abort still troublesome)
• Updated RLD view to allow defining targets
• Attempting to handle maximization/minimization

Fun detail:

• HeuristicLab.Analysis references HeuristicLab.Optimization
• HeuristicLab.Optimization.Views references HeuristicLab.Analysis.Views

IRRestarter is currently placed in HeuristicLab.Analysis, its view in HeuristicLab.Optimization.Views.

[abeham]

r12805: fixed error due to removed resx file

[abeham]

r12806: added options to add results based on targets or budgets

[abeham]

r12808:

• added ability to plot curves for multiple targets at once
• fixed bug in IndexedDataTableView regarding log-scaling
• added result that counts restarts
• fixed bugs in IRRestarter
• Set LineWidth = 2 in analyzers for chart
• fixed bugs in RLD view