Opened 6 years ago

Closed 5 years ago

#1659 closed feature request (done)

Add algorithm to execute performance benchmarks

Reported by: ascheibe Owned by: swagner
Priority: medium Milestone: HeuristicLab 3.3.6
Component: Algorithms.Benchmarks Version: 3.3.6
Keywords: Cc:

Description (last modified by swagner)

Add an algorithm to execute performance benchmarks (Linpack, Dhrystone, Whetstone, etc.).

Change History (24)

comment:1 Changed 6 years ago by ascheibe

  • Owner changed from ascheibe to spimming
  • Status changed from new to assigned

comment:2 Changed 6 years ago by ascheibe

r6917 create branch for benchmarking algorithms

comment:3 Changed 6 years ago by ascheibe

r6920 added benchmarking algorithms

comment:4 Changed 6 years ago by ascheibe

mkommend suggested that we pull the common parts out of FixedDataAnalysisAlgorithm and put them in a base class in HeuristicLab.Optimization (OptimizerBase and ExecutableBase) because there is now a lot of duplicate code in our benchmarking algorithms.

comment:5 Changed 6 years ago by ascheibe

on behalf of spimming: r6934

  • implemented abstract base class for benchmarking algorithms
  • added License information
  • corrected plugin dependencies
  • corrected descriptions

comment:6 Changed 6 years ago by spimming

r6948:

  • restructuring of the benchmarking algorithms
  • common interface for benchmarks
  • 'benchmark' class discovers benchmarking algorithms
  • added TimeLimit and ChunkSize parameters to benchmark

comment:7 Changed 6 years ago by spimming

  • Owner changed from spimming to ascheibe
  • Status changed from assigned to reviewing

comment:8 Changed 6 years ago by ascheibe

General:

  • Getters/Setters should be placed in the code directly after the corresponding private data members.

Benchmark.cs:

  • In DiscoverBenchmarks i wouldn't throw an exception if no benchmarks are found. If there are no benchmarks then no benchmarks can be selected and executed, i think that's no problem.
  • There is some uncommented code that should be removed.

GUI:

  • Mention in the description of the TimeLimit parameter that 0 means a fixed number of iterations.
  • In the parameters tab where you can choose the Benchmarking algorithm there is below the drop down menu the text "No view available". Can we get rid of this?
  • Input validation would be nice so that no negative values can be entered.
  • In the Run Collection View if a run is selected, both "Benchmark Name" and "Benchmark Type" display "Benchmark" as value. Can we hide them? This point isn't really important though.

comment:9 Changed 6 years ago by spimming

r6987:

  • removed uncommented code
  • removed ‘Benchmark Name’ and ‘Benchmark Type’ from result collection
  • placed Getters/Setters after data members
  • added input validation
  • updated parameter description

comment:10 Changed 6 years ago by ascheibe

r7000 updated branch from trunk

comment:11 Changed 6 years ago by ascheibe

r7002

  • fixed errors found by unit tests
  • set references to copylocal=false
  • added missing platforms to project files

comment:12 Changed 6 years ago by ascheibe

r7004

  • renamed benchmarking algs to Benchmark where it was missing
  • merged from trunk

comment:13 Changed 6 years ago by ascheibe

r7006 merged benchmarking algorithms branch back to trunk

comment:14 Changed 6 years ago by ascheibe

r7007 fixed small typo

comment:15 Changed 6 years ago by ascheibe

  • Owner changed from ascheibe to swagner

r7008 added an empty view for IBenchmark so that "No view available" won't be displayed in HL

comment:16 Changed 6 years ago by ascheibe

swagner: Just in case you have some spare time you can have a look at it ;-)

comment:17 Changed 6 years ago by ascheibe

r7015 removed .resx files and moved Benchmark to Algorithms category

comment:18 Changed 6 years ago by ascheibe

r7016 deleted benchmarking branch

comment:19 Changed 6 years ago by abeham

  • Version changed from branch to 3.3.5

comment:20 Changed 5 years ago by swagner

  • Component changed from Hive.General to Algorithms.Benchmarks
  • Description modified (diff)
  • Summary changed from Add benchmarking algorithms to Add algorithm to execute performance benchmarks
  • Type changed from enhancement to feature request

comment:21 Changed 5 years ago by swagner

Thanks for implementing this feature. I reviewed the code and made some additional changes:

r7246:

  • renamed Benchmark to BenchmarkAlgorithm and BenchmarkView to BenchmarkAlgorithmView
  • removed unnecessary assembly references
  • removed IBenchmarkView as the user should be informed that there is no specific view for benchmarks
  • adapted descriptions of plugins and assemblies
  • created new and unique assembly Guids
  • sealed BenchmarkAlgorithm
  • adapted item names and item descriptions
  • adapted code formatting

r7247:

  • added base class Benchmark for benchmark implementations
  • implemented ToString in Benchmark base class
  • removed suffix Benchmark from class names of benchmark implementations
  • sealed benchmark implementations Linpack, Dhrystone and Whetstone

r7248:

  • removed unnecessary problem-specific parts in BenchmarkAlgorithm
  • enabled name and description changes in BenchmarkAlgorithm
  • adapted parameter descriptions
  • implemented ToString in BenchmarkAlgorithm
  • renamed CreateDataChuck to CreateChunkData
  • adapted code formatting

comment:22 Changed 5 years ago by swagner

  • Owner changed from swagner to spimming

Please review my changes together with ascheibe and also test in the GUI once more. If you do not find any other issues, please set this ticket to ready to release. Thanks.

comment:23 Changed 5 years ago by spimming

  • Owner changed from spimming to swagner
  • Status changed from reviewing to readytorelease

comment:24 Changed 5 years ago by swagner

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.5 to 3.3.6
Note: See TracTickets for help on using tickets.