Opened 4 years ago
Last modified 20 months ago
#2431 accepted feature request
Create algorithm iterator optimizer
Reported by: | abeham | Owned by: | abeham |
---|---|---|---|
Priority: | medium | Milestone: | HeuristicLab 3.3.16 |
Component: | Analysis | Version: | branch |
Keywords: | Cc: |
Description (last modified by abeham)
The algorithm iterator should execute an algorithm instance over and over until a certain stopping criterion is met. The stopping criteria are:
- Maximum execution time reached
- Maximum number of evaluated solutions
- A certain target fitness was achieved
Design issues:
- Algorithm Execution time is hard to obtain (similar problem with the introduction of terminators)
- The precision of execution time from the algorithm is not good enough (~200ms)
- Adding EvaluatedMoves and EvaluatedSolutions is complicated by different cost factors
- Some algorithms/operators count evaluated solution-equivalents instead
- Integer is sometimes too small to hold the amount of evaluated moves for long runs / big problems
- Analyzers depend on BestAverageWorstQualityAnalyzer being run before them
- Arguably, best-so-far quality and execution time are results that the algorithm itself should provide and not an analyzer
- Quality progress chart should be calculated in a separate analyzer
- ISingleObjectiveHeuristicOptimizationProblem specifies MaximizationParameter as an IParameter instead of an IValueParameter<BoolValue>
- Which parameters in a run characterize the algorithm instance?
- E.g. Seed does not characterize the instance
- Analyzers do not characterize the instance, but some "analyzers" may modify penalties
- ResultCollection contains Result objects, but RunCollection.Results contains only the items
- The Result objects are always in the way, they should be removed
Change History (44)
comment:1 Changed 4 years ago by abeham
- Status changed from new to accepted
- Version changed from 3.3.12 to branch
comment:2 Changed 4 years ago by abeham
r12766: forgot to commit
comment:3 Changed 4 years ago by abeham
- Added run collection view
- Changed name of analyzers
- Modified algorithms to include Execution Time as a result
Logscaling is still an issue
comment:4 Changed 4 years ago by abeham
- Description modified (diff)
comment:6 Changed 4 years ago by abeham
- Description modified (diff)
- Summary changed from Algorithm performance comparison by ECDF to Run-length distribution (RLD) Analysis
comment:7 Changed 4 years ago by abeham
r12803: worked on RLD analysis
- started implementation of IRRRun
- renamed view from ECDF to RLD
- reverted algorithms (execution time)
- changed per clock analyzer
comment:8 Changed 4 years ago by abeham
- worked on IRRestarter (early abort still troublesome)
- Updated RLD view to allow defining targets
- Attempting to handle maximization/minimization
Fun detail:
- HeuristicLab.Analysis references HeuristicLab.Optimization
- HeuristicLab.Optimization.Views references HeuristicLab.Analysis.Views
IRRestarter is currently placed in HeuristicLab.Analysis, its view in HeuristicLab.Optimization.Views.
comment:9 Changed 4 years ago by abeham
- Description modified (diff)
comment:10 Changed 4 years ago by abeham
r12805: fixed error due to removed resx file
comment:11 Changed 4 years ago by abeham
r12806: added options to add results based on targets or budgets
comment:12 Changed 4 years ago by abeham
- added ability to plot curves for multiple targets at once
- fixed bug in IndexedDataTableView regarding log-scaling
- added result that counts restarts
- fixed bugs in IRRestarter
- Set LineWidth = 2 in analyzers for chart
- fixed bugs in RLD view
comment:13 Changed 4 years ago by abeham
- Description modified (diff)
comment:14 Changed 4 years ago by abeham
- Added calculation of RTs, RTus, FEs, FEus and prune runs in alg before every restart
- merged changes from trunk
comment:15 Changed 4 years ago by abeham
r12822: if multiple targets are plotted, the same color is used for each group
comment:16 Changed 4 years ago by abeham
- removed output of RTs, RTus, ...
- added 2nd row to charts that state restarts until a target was achieved
- added option to include or exclude solution results
comment:17 Changed 4 years ago by abeham
r12826: omitting empty series
comment:18 Changed 4 years ago by abeham
r12831: updated to trunk
comment:19 Changed 4 years ago by abeham
r12834: Changed analyzer to always output last value (even if quality is equal)
comment:20 Changed 4 years ago by abeham
r12838: updated RLD analysis view
- Added analysis by cost
- Prepared tab for output of ERT tables
comment:21 Changed 4 years ago by abeham
- Added calculation of ERT results tables
- Added grouping by problem to avoid having different problems in one graph
- Some restructuring to avoid duplicate code
comment:22 Changed 4 years ago by abeham
r12856: Added AlgorithmIterator instead of IRRestarter
- AlgorithmIterator implements IAlgorithm whereas IRRestarter implemented only IOptimizer
- Deriving from Algorithm leads to better design
comment:23 Changed 4 years ago by abeham
r12859: Renamed AlgorithmIterator to IteratedAlgorithm
comment:24 Changed 4 years ago by abeham
r12862: modified EngineAlgorithm to clear log on prepare
comment:25 Changed 4 years ago by abeham
- Description modified (diff)
comment:26 Changed 4 years ago by abeham
- fixed bug in iterated algorithm
- changed RLD view to use % deviation as targets
- RLD view now always groups by problem
comment:27 Changed 4 years ago by abeham
- Added label on last data point
- Changed target in cost view to display as ratio to best-known or best found
comment:28 Changed 4 years ago by abeham
- Some bug fixes
comment:29 Changed 4 years ago by abeham
r12890: fixed a bug in the iterated algorithm
comment:30 Changed 3 years ago by abeham
r12939: fixed calculation of ERT table when a certain group doesn't contain any runs of a certain problem
comment:31 Changed 3 years ago by abeham
r12956: Extracted class that calculates ERT values from runs
comment:32 Changed 3 years ago by ascheibe
- Milestone changed from HeuristicLab 3.3.13 to HeuristicLab 4.0
comment:33 Changed 3 years ago by abeham
- Milestone changed from HeuristicLab 4.0 to HeuristicLab 3.3.14
comment:34 Changed 3 years ago by abeham
r13475: updated from trunk, worked on okb connection for downloading knowledge base
comment:35 Changed 3 years ago by abeham
r13736: fixed analysis for runs that terminated earlier than others
comment:36 Changed 3 years ago by abeham
- removed IRRestarter (replaced by IteratedAlgorithm)
- removed cut-off estimation view (didn't work satisfactorily)
comment:37 Changed 3 years ago by abeham
r13738: fixed bug in IteratedAlgorithm (exception when problem is loaded, but no algorithm set)
comment:38 Changed 3 years ago by abeham
- Fixed bug that occurs when runs of unequal length are aggregated over multiple problem instances
comment:39 Changed 3 years ago by abeham
- Description modified (diff)
r14058: Refactored ResultsParameter
comment:40 Changed 3 years ago by abeham
r14059: Removed NonDiscoverableType attribute on ResultParameter
comment:41 Changed 3 years ago by abeham
r14070: fixed some bugs in the result parameter view (thx to mkommend)
comment:42 Changed 3 years ago by abeham
- Description modified (diff)
- Summary changed from Run-length distribution (RLD) Analysis to Create algorithm iterator optimizer
comment:43 Changed 3 years ago by abeham
- Milestone changed from HeuristicLab 3.3.14 to HeuristicLab 3.3.15
comment:44 Changed 20 months ago by abeham
- Milestone changed from HeuristicLab 3.3.15 to HeuristicLab 3.3.16
r12764: