Opened 8 years ago
Closed 6 years ago
#1886 closed enhancement (obsolete)
Implement measures for analyzing algorithm behavior
Reported by: | ascheibe | Owned by: | ascheibe |
---|---|---|---|
Priority: | medium | Milestone: | HeuristicLab 3.3.x Backlog |
Component: | Analysis | Version: | branch |
Keywords: | Cc: |
Description
Change History (185)
comment:1 Changed 8 years ago by ascheibe
- Status changed from new to accepted
comment:2 Changed 8 years ago by jkarder
comment:3 Changed 8 years ago by jkarder
r8162: fixed PreBuildEvent.cmd
comment:4 Changed 8 years ago by jkarder
r8191: added BuildingBlockAnalyzer
comment:5 Changed 8 years ago by jkarder
- added SchemaAnalyzer
- added BrokenInteritanceSchemaAnalyzer
comment:6 Changed 8 years ago by ascheibe
r8217 added more analyzers
comment:7 Changed 8 years ago by ascheibe
r8218 added an analyzer for calculating all subtours of the best solution
comment:8 Changed 8 years ago by jkarder
- added BrokenInheritanceWildcardAnalyzer
- separated analyzers and data structure creators
- fixed CalculateBrokenInheritanceOccurrences method
comment:9 Changed 8 years ago by ascheibe
r8229 worked on an analyzer for the best found solution
comment:10 Changed 8 years ago by ascheibe
r8230 fixed a bug
comment:11 Changed 8 years ago by ascheibe
r8234 updated graph from evolutionary tracking branch
comment:12 Changed 8 years ago by ascheibe
r8237 added ranks list to nodes in graph
comment:13 Changed 8 years ago by jkarder
r8239: added BeforeMutatorOperator to update the trace map accordingly before mutation
comment:14 Changed 8 years ago by jkarder
r8247: removed unnecessary pattern matching in CalculateBrokenInheritanceOccurrences method
comment:15 Changed 8 years ago by jkarder
r8250: added BestIndividualSchemataAnalyzer
comment:16 Changed 8 years ago by ascheibe
r8267 added wild card analyzer for the best individual
comment:17 Changed 8 years ago by ascheibe
r8268 added an analyzer for the edges of the best found solution
comment:18 Changed 8 years ago by ascheibe
r8269 added convergence analyzer
comment:19 Changed 8 years ago by ascheibe
r8277 added distinct solution analyzer
comment:20 Changed 8 years ago by ascheibe
r8278 fixed a bug in the distinct solution analyzer
comment:21 Changed 8 years ago by ascheibe
r8279 save distinct solutions in a data table
comment:22 Changed 8 years ago by ascheibe
r8281 added datatable to convergence speed analyzer
comment:23 Changed 8 years ago by ascheibe
r8287 worked on analyzers
comment:24 Changed 8 years ago by ascheibe
r8336 worked on crossover performance analyzer
comment:25 Changed 8 years ago by ascheibe
r8337 worked on crossover performance analyzer
comment:26 Changed 8 years ago by ascheibe
r8352 added mutation performance analyzer
comment:27 Changed 8 years ago by ascheibe
r8382 added more plots for crossover performance analysis
comment:28 Changed 8 years ago by ascheibe
r8410 added an operator to clone variables
comment:29 Changed 8 years ago by ascheibe
r8411 worked on mutation performance analyzer
comment:30 Changed 8 years ago by ascheibe
r8415 added an analyzer for tracking duplicate solutions
comment:31 Changed 8 years ago by ascheibe
r8447 removed external evaluation from HL solution
comment:32 Changed 8 years ago by ascheibe
r8496 removed deprecated code and cleaned up plugin dependencies
comment:33 Changed 8 years ago by ascheibe
r8498 added helper classes for generating charts
comment:34 Changed 8 years ago by ascheibe
r8499 improved crossover analyzer and added duplicate parent counting
comment:35 Changed 8 years ago by ascheibe
r8501 fixed cloning and made helper classes items
comment:36 Changed 8 years ago by ascheibe
r8502 added a run collection view that shows statistical information for data table rows
comment:37 Changed 8 years ago by ascheibe
r8503 switched rows and columns of statistical tabular view
comment:38 Changed 8 years ago by ascheibe
r8505 fixed cloning and storing
comment:39 Changed 8 years ago by ascheibe
r8511 added an option so that the scatter plot helper can now also produce data tables
comment:40 Changed 8 years ago by ascheibe
r8515 fixed cloning and resetting of the scatter plot helper
comment:41 Changed 8 years ago by ascheibe
r8521 added gradient to statistical tabular view
comment:42 Changed 8 years ago by ascheibe
r8522 added relative error and possibility to add fitted line to charts
comment:43 Changed 8 years ago by ascheibe
r8558 runs can now be opened in the statistical tabular view
comment:44 Changed 8 years ago by ascheibe
r8569 added functionality so that calculated values can be stored in the results of the runs
comment:45 Changed 8 years ago by ascheibe
r8570 fixed a small bug
comment:46 Changed 8 years ago by ascheibe
r8661 adapted operators so that they can be used with Scatter Search
comment:47 Changed 8 years ago by ascheibe
r8663 added an performance analyzer for Path Relinkers
comment:48 Changed 8 years ago by ascheibe
r8709 added an operator that combines all operators that need to be executed after crossover
comment:49 Changed 8 years ago by ascheibe
r8711 added a combined operator for mutation
comment:50 Changed 8 years ago by ascheibe
r8722 improved the DuplicateSolutionsAnalyzer
comment:51 Changed 8 years ago by ascheibe
r8739 removed static method calls to similarity calculators
comment:52 Changed 8 years ago by ascheibe
r8797 added a comparison factor parameter to the crossover analyzer so that it can be defined what a successful crossover is
comment:53 Changed 8 years ago by ascheibe
r8864 added more analyzers for crossover performance
comment:54 Changed 8 years ago by ascheibe
r8865 fixed crossover performance operator initialization
comment:55 Changed 8 years ago by ascheibe
r8866 further improvements for the crossover performance analyzer
comment:56 Changed 8 years ago by ascheibe
r8867 improved mutation analyzer
comment:57 Changed 8 years ago by ascheibe
r9020 added an analyzer for unwanted mutation for the TSP
comment:58 Changed 8 years ago by ascheibe
r9032 added a crossover analyzer for the QAP
comment:59 Changed 8 years ago by ascheibe
r9042 added mutation performance analyzer for the QAP
comment:60 Changed 8 years ago by ascheibe
- improved unwanted mutation analyzer
- added test functions crossover performance analyzer
comment:61 Changed 8 years ago by ascheibe
r9045 added mutation performance analyzer for test functions
comment:62 Changed 8 years ago by ascheibe
r9050 added performance analyzers for the VRP
comment:63 Changed 8 years ago by ascheibe
r9053 added a dialog for adding operators as successors to other operators
comment:64 Changed 8 years ago by ascheibe
r9054 added an analyzer for comparing the current solution to the rest of the population
comment:65 Changed 8 years ago by ascheibe
r9058 also track quality of solutions
comment:66 Changed 8 years ago by ascheibe
r9059 track min and max quality values for scaling
comment:67 Changed 8 years ago by ascheibe
r9060 cleaned up plugin dependencies
comment:68 Changed 8 years ago by ascheibe
r9061 added average of first, last, upper and lower 25% to statistical tabular view
comment:69 Changed 8 years ago by ascheibe
r9087 added coloring of the rows in the statistical view
comment:70 Changed 8 years ago by ascheibe
r9095 added a project for views
comment:71 Changed 8 years ago by ascheibe
r9150 fixed successor configuration and added successful mutation chart
comment:72 Changed 8 years ago by ascheibe
r9185 fixed scaling of qualities
comment:73 Changed 8 years ago by ascheibe
r9189 fixed unwanted mutations analyzer
comment:74 Changed 8 years ago by ascheibe
r9192 RunCollectionStatisticalTabularView now respects RunCollectionFilters
comment:75 Changed 8 years ago by ascheibe
- fixed a cloning bug
- fixed forgotten event handler deregistration
- added a run collection view for analyzing parameter influences
comment:76 Changed 8 years ago by ascheibe
r9197 don't do testing if there aren't enough groups
comment:77 Changed 8 years ago by ascheibe
r9198 added calculation of the overall parameter influence
comment:78 Changed 8 years ago by ascheibe
r9210 added view for displaying result correlations
comment:79 Changed 7 years ago by ascheibe
r9252 automatically calculate statistical values for each run
comment:80 Changed 7 years ago by ascheibe
r9316 implemented Kruskal-Wallis Test
comment:81 Changed 7 years ago by ascheibe
r9317 started working on a statistical testing run collection view
comment:82 Changed 7 years ago by ascheibe
r9318 added test for normality and pairwise statistical testing
comment:83 Changed 7 years ago by ascheibe
r9319 improved view and added documentation
comment:84 Changed 7 years ago by ascheibe
r9320 some more improvements for the statistical testing view
comment:85 Changed 7 years ago by ascheibe
- fixed a bug in the ResultCorrelationView
- added sample size determination by estimating means, Cohen's D and Hedges' G
comment:86 Changed 7 years ago by ascheibe
r9329 calculate Cohen's D and Hedges' G for pairwise comparison and updated documentation
comment:87 Changed 7 years ago by ascheibe
r9330 added a view for sample size determination
comment:88 Changed 7 years ago by ascheibe
r9331 added analyzer for selection pressure
comment:89 Changed 7 years ago by ascheibe
r9333 improved run collection views
comment:90 Changed 7 years ago by ascheibe
r9336 added ttest and sample size determination for the ttest
comment:91 Changed 7 years ago by ascheibe
r9337 fixed bugs in TTest
comment:92 Changed 7 years ago by ascheibe
r9341 some minor AB fixes
comment:93 Changed 7 years ago by ascheibe
comment:94 Changed 7 years ago by ascheibe
r9361 adapted create experiment dialog so that it can generate random configurations
comment:95 Changed 7 years ago by ascheibe
r9364 added repetitions for random parameter configurations
comment:96 Changed 7 years ago by ascheibe
r9384 added calculation of selection intensity
comment:97 Changed 7 years ago by ascheibe
r9412 added GA with evaluated solutions as a stopping criteria
comment:98 follow-up: ↓ 99 Changed 7 years ago by gkronber
r9412: I'm not sure it is necessary to add a GA variant with evaluated solutions as stopping criterion (this is almost the same as pop-size * generations)
comment:99 in reply to: ↑ 98 Changed 7 years ago by ascheibe
Replying to gkronber:
r9412: I'm not sure it is necessary to add a GA variant with evaluated solutions as stopping criterion (this is almost the same as pop-size * generations)
Yes, that's true, but it makes creating experiments simpler as I can fix the evaluated solutions and just vary the population size. Generations are not varied and are then simply a result. Because I want to define ranges for parameters and then draw random samples it would be more complicated to check if a drawn configuration has the correct amount of evaluated solutions (or only calculated configurations where this is the case).
comment:100 Changed 7 years ago by ascheibe
- changed duplicate solutions analyzer to also track avg. diversity as the standard diversity analyzer needs too much memory
- fixed project references of eslGA
comment:101 Changed 7 years ago by ascheibe
r9431 added configuration of frequency with which the ab measures are calculated
comment:102 Changed 7 years ago by ascheibe
r9711 started working on a solution cache
comment:103 Changed 7 years ago by ascheibe
r9723 worked on solution caching
comment:104 Changed 7 years ago by ascheibe
r9728 worked on solution caching
comment:105 Changed 7 years ago by ascheibe
r9729 fixed serialization bug in solution caching
comment:106 Changed 7 years ago by ascheibe
r9730 added a library that calculates convex hulls
comment:107 Changed 7 years ago by ascheibe
r9731 cleaned up references
comment:108 Changed 7 years ago by ascheibe
r9754 added a caching analyzer for crossover
comment:109 Changed 7 years ago by ascheibe
r9757 fixed a bug and improved PermutationInformation
comment:110 Changed 7 years ago by ascheibe
r9789 fixed alglib reference and updated year and version information
comment:111 Changed 7 years ago by ascheibe
r9791 removed useless resource file
comment:112 Changed 7 years ago by ascheibe
- added new convex hull algorithm based on SMO/SVM
- added unit test and a visualization
- updated MIConvexHull
comment:113 Changed 7 years ago by ascheibe
r10007 started working on volume calculation
comment:114 Changed 7 years ago by ascheibe
r10008 added volume calculation for convex d-polytopes
comment:115 Changed 7 years ago by ascheibe
- moved solution caching to an own plugin
- added initial support for real vector encoding
comment:116 Changed 7 years ago by ascheibe
r10024 added views project for solution caching
comment:117 Changed 7 years ago by ascheibe
r10025 added support for permutation encoding for solution caching
comment:118 Changed 7 years ago by ascheibe
- removed old caching code
- adapted convex hull view and some other minor improvements
comment:119 Changed 7 years ago by ascheibe
r10027 adapted convex hull view to work with real vectors and permutations
comment:120 Changed 7 years ago by ascheibe
- added point calculation from distance matrices
- added convex hull algorithm based on LP (work in progress)
- added unit tests
comment:121 Changed 7 years ago by ascheibe
r10077 fixed a bug in the point calculation from distance matrices
comment:122 Changed 7 years ago by ascheibe
r10078 another bugfix in the point calculation from distance matrices
comment:123 Changed 7 years ago by ascheibe
r10081 fixed LPHull, seems to work now
comment:124 Changed 7 years ago by ascheibe
r10082 correction of LPHull Func
comment:125 Changed 7 years ago by ascheibe
r10090 added an algorithm that executes analyzers on a solution cache
comment:126 Changed 7 years ago by ascheibe
r10091 renamed analyzers to collectors to avoid confusion
comment:127 Changed 7 years ago by ascheibe
r10092 fixed volume calculation of polytopes
comment:128 Changed 7 years ago by ascheibe
r10096 started working on crossover wrappers that can execute analyzers for crossovers
comment:129 Changed 7 years ago by ascheibe
r10097 added metric mds for testing purposes
comment:130 Changed 7 years ago by ascheibe
r10098 added method for converting similarity-based measures to euclidean distances
comment:131 Changed 7 years ago by ascheibe
r10101 added PermutationPointcut and refactored pointcut code
comment:132 Changed 7 years ago by ascheibe
r10102 added mutation pointcuts for realvector and permutation encodings
comment:133 Changed 7 years ago by ascheibe
r10104 fixed RealVectorEqualityComparer and added unit tests for caches
comment:134 Changed 7 years ago by ascheibe
r10105 renamed some properties in SolutionInformation as suggested by swagner
comment:135 Changed 7 years ago by ascheibe
r10108 fixed metric MDS and updated unit test
comment:136 Changed 7 years ago by ascheibe
r10109 fixed other MDS and added more unit tests
comment:137 Changed 7 years ago by ascheibe
r10111 created a new Hive task that runs RunCollectionModifiers in Hive
comment:138 Changed 7 years ago by ascheibe
r10112 added constant adding for metric mds
comment:139 Changed 7 years ago by ascheibe
r10113 added basic dialog for configuring a RunCollectionModifierTask
comment:140 Changed 7 years ago by ascheibe
r10114 added tasks and hive tasks for RunCollectionModifiers
comment:141 Changed 7 years ago by ascheibe
r10115 some improvements in the RunCollectionModifierExecutable
comment:142 Changed 7 years ago by ascheibe
r10116 added convex hull RunCollectionModifiers
comment:143 Changed 7 years ago by ascheibe
r10117 made RunCollectionModifierHiveTask distributable
comment:144 Changed 7 years ago by ascheibe
r10118 added another unit test for permutations
comment:145 Changed 7 years ago by ascheibe
r10119 improved RealVectorConvexHullModifier
comment:146 Changed 7 years ago by ascheibe
r10120 fixed a bug in RunCollectionModifierExecutable
comment:147 Changed 7 years ago by ascheibe
r10121 improved RunCollectionModifierExecutable and added a new view for it
comment:148 Changed 7 years ago by ascheibe
r10127 fixed a bug in metric mds and added another unit test
comment:149 Changed 7 years ago by ascheibe
r10128 cleaned up Hive RunCollection Modifier Task classes
comment:150 Changed 7 years ago by ascheibe
r10129 added a view for the RunCollectionModifierHiveTask
comment:151 Changed 7 years ago by ascheibe
comment:152 Changed 7 years ago by ascheibe
r10132 updated unit test to take account for rounding errors
comment:153 Changed 7 years ago by ascheibe
r10133 updated the PermutationConvexHullModifier to collect the same results as the RealVectorConvexHullModifier
comment:154 Changed 7 years ago by ascheibe
r10139 fixed a bug in the PermutationConvexHullModifier
comment:155 Changed 7 years ago by ascheibe
- renamed MenuItem for the HiveRunCollectionModifier
- deactivated calculation of complete convex hull for permutations
comment:156 Changed 7 years ago by ascheibe
comment:157 Changed 7 years ago by ascheibe
- added liblrs and c# wrapper for vertex enumeration/volume calculation
- use MIConvexHull for triangulation and volume calculation
- fixed vertex conversion code for MIConvexHull lib
- added a unit test for measuring performance of convex hull/volume calculation
comment:158 Changed 7 years ago by ascheibe
- moved lrs code into analyzer project
- added lrs performance test
- moved performance tests into an own project
- don't use 64 bit number format of lrs as it does not work correctly
comment:159 Changed 7 years ago by ascheibe
r10201 fixed a small bug in volume calculation
comment:160 Changed 7 years ago by ascheibe
r10207 added a unit test for volume calculation and the qhull library
comment:161 Changed 7 years ago by ascheibe
- added x64 configurations for qhull
- added a wrapper for qhull volume calculation
- removed old volume calculation code
comment:162 Changed 7 years ago by ascheibe
comment:163 Changed 7 years ago by ascheibe
r10210 removed unnecessary qhull projects
comment:164 Changed 7 years ago by ascheibe
r10211 added a wrapper for calculating the convex hull using qhull
comment:165 Changed 7 years ago by ascheibe
- improved qhull wrapper
- cleaned up unit tests
comment:166 Changed 7 years ago by ascheibe
r10223 removed lrs code from the main solution as this did not work out as expected
comment:167 Changed 7 years ago by ascheibe
- added more performance tests
- tried to mute qhull
comment:168 Changed 7 years ago by ascheibe
r10262 also allow unsafe code for release configurations
comment:169 Changed 7 years ago by ascheibe
r10272 improved convex hull modifier for test functions
comment:170 Changed 7 years ago by ascheibe
r10274 added calculation of the centroid of a convex hull and measures for motion characteristics
comment:171 Changed 7 years ago by ascheibe
r10275 added calculation of diameter and updated convex hull modifier to calculate the new measures
comment:172 Changed 7 years ago by ascheibe
r10279 added calculation of the ratio between the convex hull volume and the max. hypercube volume
comment:173 Changed 7 years ago by ascheibe
r10283 added volume ratios for incremental convex hull building over generations
comment:174 Changed 7 years ago by ascheibe
r10284 added missing plugin dependencies
comment:175 Changed 7 years ago by ascheibe
r10286 added support for calculating big volumes
comment:176 Changed 7 years ago by ascheibe
r10287 adapted convex hull modifier to use fractions for big volumes
comment:177 Changed 7 years ago by ascheibe
r10288 fixed ratio calculation
comment:178 Changed 7 years ago by ascheibe
r10289 fixed another bug
comment:179 Changed 7 years ago by ascheibe
r10301 fixed hypercube volume calculation
comment:180 Changed 7 years ago by ascheibe
comment:181 Changed 6 years ago by ascheibe
r11180 added unit test
comment:182 Changed 6 years ago by ascheibe
r11181 small change in view
comment:183 Changed 6 years ago by ascheibe
r11182 added another unit test
comment:184 Changed 6 years ago by ascheibe
r11613 deleted algorithm behavior branch as this has been moved to another repository
comment:185 Changed 6 years ago by ascheibe
- Resolution set to obsolete
- Status changed from accepted to closed
r8161: initial commit