Opened 7 years ago

Closed 5 years ago

#1886 closed enhancement (obsolete)

Implement measures for analyzing algorithm behavior

Reported by: ascheibe Owned by: ascheibe
Priority: medium Milestone: HeuristicLab 3.3.x Backlog
Component: Analysis Version: branch
Keywords: Cc:

Description


Change History (185)

comment:1 Changed 7 years ago by ascheibe

  • Status changed from new to accepted

comment:2 Changed 7 years ago by jkarder

r8161: initial commit

comment:3 Changed 7 years ago by jkarder

r8162: fixed PreBuildEvent.cmd

comment:4 Changed 7 years ago by jkarder

r8191: added BuildingBlockAnalyzer

comment:5 Changed 7 years ago by jkarder

r8216:

  • added SchemaAnalyzer
  • added BrokenInteritanceSchemaAnalyzer

comment:6 Changed 7 years ago by ascheibe

r8217 added more analyzers

comment:7 Changed 7 years ago by ascheibe

r8218 added an analyzer for calculating all subtours of the best solution

comment:8 Changed 7 years ago by jkarder

r8227:

  • added BrokenInheritanceWildcardAnalyzer
  • separated analyzers and data structure creators
  • fixed CalculateBrokenInheritanceOccurrences method

comment:9 Changed 7 years ago by ascheibe

r8229 worked on an analyzer for the best found solution

comment:10 Changed 7 years ago by ascheibe

r8230 fixed a bug

comment:11 Changed 7 years ago by ascheibe

r8234 updated graph from evolutionary tracking branch

comment:12 Changed 7 years ago by ascheibe

r8237 added ranks list to nodes in graph

comment:13 Changed 7 years ago by jkarder

r8239: added BeforeMutatorOperator to update the trace map accordingly before mutation

comment:14 Changed 7 years ago by jkarder

r8247: removed unnecessary pattern matching in CalculateBrokenInheritanceOccurrences method

comment:15 Changed 7 years ago by jkarder

r8250: added BestIndividualSchemataAnalyzer

comment:16 Changed 7 years ago by ascheibe

r8267 added wild card analyzer for the best individual

comment:17 Changed 7 years ago by ascheibe

r8268 added an analyzer for the edges of the best found solution

comment:18 Changed 7 years ago by ascheibe

r8269 added convergence analyzer

comment:19 Changed 7 years ago by ascheibe

r8277 added distinct solution analyzer

comment:20 Changed 7 years ago by ascheibe

r8278 fixed a bug in the distinct solution analyzer

comment:21 Changed 7 years ago by ascheibe

r8279 save distinct solutions in a data table

comment:22 Changed 7 years ago by ascheibe

r8281 added datatable to convergence speed analyzer

comment:23 Changed 7 years ago by ascheibe

r8287 worked on analyzers

comment:24 Changed 7 years ago by ascheibe

r8336 worked on crossover performance analyzer

comment:25 Changed 7 years ago by ascheibe

r8337 worked on crossover performance analyzer

comment:26 Changed 7 years ago by ascheibe

r8352 added mutation performance analyzer

comment:27 Changed 7 years ago by ascheibe

r8382 added more plots for crossover performance analysis

comment:28 Changed 7 years ago by ascheibe

r8410 added an operator to clone variables

comment:29 Changed 7 years ago by ascheibe

r8411 worked on mutation performance analyzer

comment:30 Changed 7 years ago by ascheibe

r8415 added an analyzer for tracking duplicate solutions

comment:31 Changed 7 years ago by ascheibe

r8447 removed external evaluation from HL solution

comment:32 Changed 7 years ago by ascheibe

r8496 removed deprecated code and cleaned up plugin dependencies

comment:33 Changed 7 years ago by ascheibe

r8498 added helper classes for generating charts

comment:34 Changed 7 years ago by ascheibe

r8499 improved crossover analyzer and added duplicate parent counting

comment:35 Changed 7 years ago by ascheibe

r8501 fixed cloning and made helper classes items

comment:36 Changed 7 years ago by ascheibe

r8502 added a run collection view that shows statistical information for data table rows

comment:37 Changed 7 years ago by ascheibe

r8503 switched rows and columns of statistical tabular view

comment:38 Changed 7 years ago by ascheibe

r8505 fixed cloning and storing

comment:39 Changed 7 years ago by ascheibe

r8511 added an option so that the scatter plot helper can now also produce data tables

comment:40 Changed 7 years ago by ascheibe

r8515 fixed cloning and resetting of the scatter plot helper

comment:41 Changed 7 years ago by ascheibe

r8521 added gradient to statistical tabular view

comment:42 Changed 7 years ago by ascheibe

r8522 added relative error and possibility to add fitted line to charts

comment:43 Changed 7 years ago by ascheibe

r8558 runs can now be opened in the statistical tabular view

comment:44 Changed 7 years ago by ascheibe

r8569 added functionality so that calculated values can be stored in the results of the runs

comment:45 Changed 7 years ago by ascheibe

r8570 fixed a small bug

comment:46 Changed 7 years ago by ascheibe

r8661 adapted operators so that they can be used with Scatter Search

comment:47 Changed 7 years ago by ascheibe

r8663 added an performance analyzer for Path Relinkers

comment:48 Changed 7 years ago by ascheibe

r8709 added an operator that combines all operators that need to be executed after crossover

comment:49 Changed 7 years ago by ascheibe

r8711 added a combined operator for mutation

comment:50 Changed 7 years ago by ascheibe

r8722 improved the DuplicateSolutionsAnalyzer

comment:51 Changed 7 years ago by ascheibe

r8739 removed static method calls to similarity calculators

comment:52 Changed 7 years ago by ascheibe

r8797 added a comparison factor parameter to the crossover analyzer so that it can be defined what a successful crossover is

comment:53 Changed 7 years ago by ascheibe

r8864 added more analyzers for crossover performance

comment:54 Changed 7 years ago by ascheibe

r8865 fixed crossover performance operator initialization

comment:55 Changed 7 years ago by ascheibe

r8866 further improvements for the crossover performance analyzer

comment:56 Changed 7 years ago by ascheibe

r8867 improved mutation analyzer

comment:57 Changed 7 years ago by ascheibe

r9020 added an analyzer for unwanted mutation for the TSP

comment:58 Changed 7 years ago by ascheibe

r9032 added a crossover analyzer for the QAP

comment:59 Changed 7 years ago by ascheibe

r9042 added mutation performance analyzer for the QAP

comment:60 Changed 7 years ago by ascheibe

r9044

  • improved unwanted mutation analyzer
  • added test functions crossover performance analyzer

comment:61 Changed 7 years ago by ascheibe

r9045 added mutation performance analyzer for test functions

comment:62 Changed 7 years ago by ascheibe

r9050 added performance analyzers for the VRP

comment:63 Changed 7 years ago by ascheibe

r9053 added a dialog for adding operators as successors to other operators

comment:64 Changed 7 years ago by ascheibe

r9054 added an analyzer for comparing the current solution to the rest of the population

comment:65 Changed 7 years ago by ascheibe

r9058 also track quality of solutions

comment:66 Changed 7 years ago by ascheibe

r9059 track min and max quality values for scaling

comment:67 Changed 7 years ago by ascheibe

r9060 cleaned up plugin dependencies

comment:68 Changed 7 years ago by ascheibe

r9061 added average of first, last, upper and lower 25% to statistical tabular view

comment:69 Changed 7 years ago by ascheibe

r9087 added coloring of the rows in the statistical view

comment:70 Changed 6 years ago by ascheibe

r9095 added a project for views

comment:71 Changed 6 years ago by ascheibe

r9150 fixed successor configuration and added successful mutation chart

comment:72 Changed 6 years ago by ascheibe

r9185 fixed scaling of qualities

comment:73 Changed 6 years ago by ascheibe

r9189 fixed unwanted mutations analyzer

comment:74 Changed 6 years ago by ascheibe

r9192 RunCollectionStatisticalTabularView now respects RunCollectionFilters

comment:75 Changed 6 years ago by ascheibe

r9196

  • fixed a cloning bug
  • fixed forgotten event handler deregistration
  • added a run collection view for analyzing parameter influences

comment:76 Changed 6 years ago by ascheibe

r9197 don't do testing if there aren't enough groups

comment:77 Changed 6 years ago by ascheibe

r9198 added calculation of the overall parameter influence

comment:78 Changed 6 years ago by ascheibe

r9210 added view for displaying result correlations

comment:79 Changed 6 years ago by ascheibe

r9252 automatically calculate statistical values for each run

comment:80 Changed 6 years ago by ascheibe

r9316 implemented Kruskal-Wallis Test

comment:81 Changed 6 years ago by ascheibe

r9317 started working on a statistical testing run collection view

comment:82 Changed 6 years ago by ascheibe

r9318 added test for normality and pairwise statistical testing

comment:83 Changed 6 years ago by ascheibe

r9319 improved view and added documentation

comment:84 Changed 6 years ago by ascheibe

r9320 some more improvements for the statistical testing view

comment:85 Changed 6 years ago by ascheibe

r9328

  • fixed a bug in the ResultCorrelationView
  • added sample size determination by estimating means, Cohen's D and Hedges' G

comment:86 Changed 6 years ago by ascheibe

r9329 calculate Cohen's D and Hedges' G for pairwise comparison and updated documentation

comment:87 Changed 6 years ago by ascheibe

r9330 added a view for sample size determination

comment:88 Changed 6 years ago by ascheibe

r9331 added analyzer for selection pressure

comment:89 Changed 6 years ago by ascheibe

r9333 improved run collection views

comment:90 Changed 6 years ago by ascheibe

r9336 added ttest and sample size determination for the ttest

comment:91 Changed 6 years ago by ascheibe

r9337 fixed bugs in TTest

comment:92 Changed 6 years ago by ascheibe

r9341 some minor AB fixes

comment:93 Changed 6 years ago by ascheibe

r9354 removed views that went into the statistical hypothesis testing branch (#2031)

comment:94 Changed 6 years ago by ascheibe

r9361 adapted create experiment dialog so that it can generate random configurations

comment:95 Changed 6 years ago by ascheibe

r9364 added repetitions for random parameter configurations

comment:96 Changed 6 years ago by ascheibe

r9384 added calculation of selection intensity

comment:97 Changed 6 years ago by ascheibe

r9412 added GA with evaluated solutions as a stopping criteria

comment:98 follow-up: Changed 6 years ago by gkronber

r9412: I'm not sure it is necessary to add a GA variant with evaluated solutions as stopping criterion (this is almost the same as pop-size * generations)

comment:99 in reply to: ↑ 98 Changed 6 years ago by ascheibe

Replying to gkronber:

r9412: I'm not sure it is necessary to add a GA variant with evaluated solutions as stopping criterion (this is almost the same as pop-size * generations)

Yes, that's true, but it makes creating experiments simpler as I can fix the evaluated solutions and just vary the population size. Generations are not varied and are then simply a result. Because I want to define ranges for parameters and then draw random samples it would be more complicated to check if a drawn configuration has the correct amount of evaluated solutions (or only calculated configurations where this is the case).

comment:100 Changed 6 years ago by ascheibe

r9418

  • changed duplicate solutions analyzer to also track avg. diversity as the standard diversity analyzer needs too much memory
  • fixed project references of eslGA

comment:101 Changed 6 years ago by ascheibe

r9431 added configuration of frequency with which the ab measures are calculated

comment:102 Changed 6 years ago by ascheibe

r9711 started working on a solution cache

comment:103 Changed 6 years ago by ascheibe

r9723 worked on solution caching

comment:104 Changed 6 years ago by ascheibe

r9728 worked on solution caching

comment:105 Changed 6 years ago by ascheibe

r9729 fixed serialization bug in solution caching

comment:106 Changed 6 years ago by ascheibe

r9730 added a library that calculates convex hulls

comment:107 Changed 6 years ago by ascheibe

r9731 cleaned up references

comment:108 Changed 6 years ago by ascheibe

r9754 added a caching analyzer for crossover

comment:109 Changed 6 years ago by ascheibe

r9757 fixed a bug and improved PermutationInformation

comment:110 Changed 6 years ago by ascheibe

r9789 fixed alglib reference and updated year and version information

comment:111 Changed 6 years ago by ascheibe

r9791 removed useless resource file

comment:112 Changed 6 years ago by ascheibe

r9945

  • added new convex hull algorithm based on SMO/SVM
  • added unit test and a visualization
  • updated MIConvexHull

comment:113 Changed 6 years ago by ascheibe

r10007 started working on volume calculation

comment:114 Changed 6 years ago by ascheibe

r10008 added volume calculation for convex d-polytopes

comment:115 Changed 6 years ago by ascheibe

r10023

  • moved solution caching to an own plugin
  • added initial support for real vector encoding

comment:116 Changed 6 years ago by ascheibe

r10024 added views project for solution caching

comment:117 Changed 6 years ago by ascheibe

r10025 added support for permutation encoding for solution caching

comment:118 Changed 6 years ago by ascheibe

r10026

  • removed old caching code
  • adapted convex hull view and some other minor improvements

comment:119 Changed 6 years ago by ascheibe

r10027 adapted convex hull view to work with real vectors and permutations

comment:120 Changed 6 years ago by ascheibe

r10060

  • added point calculation from distance matrices
  • added convex hull algorithm based on LP (work in progress)
  • added unit tests

comment:121 Changed 6 years ago by ascheibe

r10077 fixed a bug in the point calculation from distance matrices

comment:122 Changed 6 years ago by ascheibe

r10078 another bugfix in the point calculation from distance matrices

comment:123 Changed 6 years ago by ascheibe

r10081 fixed LPHull, seems to work now

comment:124 Changed 6 years ago by ascheibe

r10082 correction of LPHull Func

comment:125 Changed 6 years ago by ascheibe

r10090 added an algorithm that executes analyzers on a solution cache

comment:126 Changed 6 years ago by ascheibe

r10091 renamed analyzers to collectors to avoid confusion

comment:127 Changed 6 years ago by ascheibe

r10092 fixed volume calculation of polytopes

comment:128 Changed 6 years ago by ascheibe

r10096 started working on crossover wrappers that can execute analyzers for crossovers

comment:129 Changed 6 years ago by ascheibe

r10097 added metric mds for testing purposes

comment:130 Changed 6 years ago by ascheibe

r10098 added method for converting similarity-based measures to euclidean distances

comment:131 Changed 6 years ago by ascheibe

r10101 added PermutationPointcut and refactored pointcut code

comment:132 Changed 6 years ago by ascheibe

r10102 added mutation pointcuts for realvector and permutation encodings

comment:133 Changed 6 years ago by ascheibe

r10104 fixed RealVectorEqualityComparer and added unit tests for caches

comment:134 Changed 6 years ago by ascheibe

r10105 renamed some properties in SolutionInformation as suggested by swagner

comment:135 Changed 6 years ago by ascheibe

r10108 fixed metric MDS and updated unit test

comment:136 Changed 6 years ago by ascheibe

r10109 fixed other MDS and added more unit tests

comment:137 Changed 6 years ago by ascheibe

r10111 created a new Hive task that runs RunCollectionModifiers in Hive

comment:138 Changed 6 years ago by ascheibe

r10112 added constant adding for metric mds

comment:139 Changed 6 years ago by ascheibe

r10113 added basic dialog for configuring a RunCollectionModifierTask

comment:140 Changed 6 years ago by ascheibe

r10114 added tasks and hive tasks for RunCollectionModifiers

comment:141 Changed 6 years ago by ascheibe

r10115 some improvements in the RunCollectionModifierExecutable

comment:142 Changed 6 years ago by ascheibe

r10116 added convex hull RunCollectionModifiers

comment:143 Changed 6 years ago by ascheibe

r10117 made RunCollectionModifierHiveTask distributable

comment:144 Changed 6 years ago by ascheibe

r10118 added another unit test for permutations

comment:145 Changed 6 years ago by ascheibe

r10119 improved RealVectorConvexHullModifier

comment:146 Changed 6 years ago by ascheibe

r10120 fixed a bug in RunCollectionModifierExecutable

comment:147 Changed 6 years ago by ascheibe

r10121 improved RunCollectionModifierExecutable and added a new view for it

comment:148 Changed 6 years ago by ascheibe

r10127 fixed a bug in metric mds and added another unit test

comment:149 Changed 6 years ago by ascheibe

r10128 cleaned up Hive RunCollection Modifier Task classes

comment:150 Changed 6 years ago by ascheibe

r10129 added a view for the RunCollectionModifierHiveTask

comment:151 Changed 6 years ago by ascheibe

r10131 adapted the RunCollectionModifierTask to the changes of r10130

comment:152 Changed 6 years ago by ascheibe

r10132 updated unit test to take account for rounding errors

comment:153 Changed 6 years ago by ascheibe

r10133 updated the PermutationConvexHullModifier to collect the same results as the RealVectorConvexHullModifier

comment:154 Changed 6 years ago by ascheibe

r10139 fixed a bug in the PermutationConvexHullModifier

comment:155 Changed 6 years ago by ascheibe

r10141

  • renamed MenuItem for the HiveRunCollectionModifier
  • deactivated calculation of complete convex hull for permutations

comment:156 Changed 6 years ago by ascheibe

r10151 just a small change to go along with the changes of r10150

comment:157 Changed 6 years ago by ascheibe

r10198

  • added liblrs and c# wrapper for vertex enumeration/volume calculation
  • use MIConvexHull for triangulation and volume calculation
  • fixed vertex conversion code for MIConvexHull lib
  • added a unit test for measuring performance of convex hull/volume calculation

comment:158 Changed 6 years ago by ascheibe

r10200

  • moved lrs code into analyzer project
  • added lrs performance test
  • moved performance tests into an own project
  • don't use 64 bit number format of lrs as it does not work correctly

comment:159 Changed 6 years ago by ascheibe

r10201 fixed a small bug in volume calculation

comment:160 Changed 6 years ago by ascheibe

r10207 added a unit test for volume calculation and the qhull library

comment:161 Changed 6 years ago by ascheibe

r10208

  • added x64 configurations for qhull
  • added a wrapper for qhull volume calculation
  • removed old volume calculation code

comment:162 Changed 6 years ago by ascheibe

r10209 missed a file in last commit (r10208)

comment:163 Changed 6 years ago by ascheibe

r10210 removed unnecessary qhull projects

comment:164 Changed 6 years ago by ascheibe

r10211 added a wrapper for calculating the convex hull using qhull

comment:165 Changed 6 years ago by ascheibe

r10222

  • improved qhull wrapper
  • cleaned up unit tests

comment:166 Changed 6 years ago by ascheibe

r10223 removed lrs code from the main solution as this did not work out as expected

comment:167 Changed 6 years ago by ascheibe

r10224

  • added more performance tests
  • tried to mute qhull

comment:168 Changed 6 years ago by ascheibe

r10262 also allow unsafe code for release configurations

comment:169 Changed 5 years ago by ascheibe

r10272 improved convex hull modifier for test functions

comment:170 Changed 5 years ago by ascheibe

r10274 added calculation of the centroid of a convex hull and measures for motion characteristics

comment:171 Changed 5 years ago by ascheibe

r10275 added calculation of diameter and updated convex hull modifier to calculate the new measures

comment:172 Changed 5 years ago by ascheibe

r10279 added calculation of the ratio between the convex hull volume and the max. hypercube volume

comment:173 Changed 5 years ago by ascheibe

r10283 added volume ratios for incremental convex hull building over generations

comment:174 Changed 5 years ago by ascheibe

r10284 added missing plugin dependencies

comment:175 Changed 5 years ago by ascheibe

r10286 added support for calculating big volumes

comment:176 Changed 5 years ago by ascheibe

r10287 adapted convex hull modifier to use fractions for big volumes

comment:177 Changed 5 years ago by ascheibe

r10288 fixed ratio calculation

comment:178 Changed 5 years ago by ascheibe

r10289 fixed another bug

comment:179 Changed 5 years ago by ascheibe

r10301 fixed hypercube volume calculation

comment:180 Changed 5 years ago by ascheibe

r10416 adapted branch to changes of #2119

comment:181 Changed 5 years ago by ascheibe

r11180 added unit test

comment:182 Changed 5 years ago by ascheibe

r11181 small change in view

comment:183 Changed 5 years ago by ascheibe

r11182 added another unit test

comment:184 Changed 5 years ago by ascheibe

r11613 deleted algorithm behavior branch as this has been moved to another repository

comment:185 Changed 5 years ago by ascheibe

  • Resolution set to obsolete
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.