Free cookie consent management tool by TermsFeed Policy Generator
wiki:ReviewHeuristicLab3.3.0ApplicationDone

Reviews - HeuristicLab 3.3.0 Application Review - Implemented Review Comments

Reviewer: abeham

Priority: HIGHEST

  • In some way we need to make the best solution available to view. When you're optimizing something other than benchmark problems you'll probably want to make use of the best solution.
    • abeham: done with the change to use analyzers starting from r3616 (#999)
  • The relative / absolute difference to best known quality in the multipop algorithms is doing strange things.
    • abeham: fixed with the change to analyzers starting from r3616 (#999)
  • SinglePointCrossover for RealVector fails when vector has less than 3 dimensions
  • ES: Setting the population size to a value smaller than parents per child does not update the parents per child parameter.
  • Some operators are overwritten to their defaults when an algorithm is loaded, these include: Selection and Analyzers parameters.
    • swagner: Fixed for GA in r3680.
    • abeham: Fixed for the rest in r3689.
  • NormalAllPositionsManipulator fails for the genetic algorithm (and others), because the StrategyVector is not set by default
  • TSP Tour view flickers when using a trajectory-based algorithm.
  • Regarding Drag/Drop operations: The Ctrl modifier key should not link. In Windows Explorer linking is Alt. Sometimes out of habit I press Ctrl while dragging, this should not change the default action of copy.
    • swagner: Changed in r3694.
  • Sometimes the splash screen does not display when starting the Optimizer.
    • mkommend: I assume that the splash screen was shown, but was placed behind the starter form cause of switching between different windows. This was corrected by setting that starter form as the owner of the splash screen in r3697.
  • ComparisonFactor does not update in OSGA and is missing in Island OSGA
  • The population in OSGA grows each generation by the number of elites
  • Island OSGA should have SelectedParents parameter and not reduce selected parents by twice the number of elites
  • Remove Parallel parameter in multipopulation algorithms (for now)
  • In the OSGA the elites replace the best individual currently
    • abeham: changed in r3713
  • When a run collection tabular view is opened to analyze some runs, it cannot be closed again after all runs are cleared. It says there is an invalid number of rows (0).
    • mkommend: corrected in r3714
  • The operator graph view of a user defined algorithm is uneditable after the user defined algorithm has been executed.
    • abeham: fixed in r3725, thx for review mkommend
  • In the operator graph view: Deleting an operator that is connected from e.g. a SubScopesProcessor throws an exception "Value cannot be null". I think that's an issue with MultiOperator in general. It also occurs when you create a new operator graph and drag a MultiPermutationManipulator in. If you delete one of the manipulators an exception is thrown.
    • swagner: Fixed in r3729 after discussing some possible solutions with mkommend.
    • mkommend: corrected CheckedItemCollectionViews in r3738.
  • ComparisonFactor remains 0, if lower and upper bound are set to 1 and no modifier is specified
  • Generations and MaximumGenerations parameters in MichalewiczNonUniform(One|All)PositionsManipulator are not wired
    • abeham: set them to default to "Generations" in r3677. This is not a complete fix, but will work for now.
    • abeham: fixed in r3750
  • Test functions don't correctly set the learning parameters of their strategy parameter operators
  • Deserialization fails for LS/TS in some cases (my assumption is that it doesn't correctly parse NaN values)
    • epitzer fixed this in r3811
  • Some predefined samples contain runs.
    • swagner: Fixed in r3814.
  • Adding a ValueParameter<T> in a VariableCreator and selecting a generic type e.g. ItemList<T> fails because no choice can be made on the generic type of the ItemList (ticket #42).
    • swagner: Fixed as described in ticket #42 and ticket #1015.

Priority: HIGH

  • Creating a new TSP problem and viewing the parameter "Coordinates": The view can be switched to an "OptionalValueParameterView" which adds a button to delete the value. However when clicked an exception is thrown that the value cannot be null. The OptionalValueParameterView should not be selectable (Coordinates is not an optional parameter). This also holds for other parameters like the SolutionCreator.
    • swagner: Changed in r2948.
  • When one creates a new user defined algorithm, the user finds a button called "Create User Defined Algorithm" button on the form. This feels strange.
    • swagner: Set the button to be invisible in r2947.
  • In the user defined algorithm:
    • On the parameter tab: I press "+" to add a new Parameter. I select a ValueParameter, term it "MutationRate" and select DoubleData as type I get a MissingMethodException (Constructor on type ValueParameter not found).
      • swagner: Fixed in r2948.
  • The operator graph view is a bit confusing:
    • Selecting an operator in "Operator Graph" does not bring up the details in the details view.
      • swagner: Fixed in r2949.
    • Should there really be a view host in the operator graph group box? It's confusing being able to switch views there (also because these are always just views on the initial operator which can be viewed with view host in the details group box as well).
      • swagner: Removed view host in r2947.
  • Creating a user defined algorithm from the SGA doesn't reconfigure the SGA operator with the parameters set in the SGA interface.
    • swagner: Fixed as described in ticket #898.
  • The successor parameter reads "Successor: ResultsCollector", "Successor: SubScopesSorter", ... It's more readable if "Successor" was not present all the time (in the operator graph group box).
    • swagner: "Successor" is required to identify an operator parameter. Of course, in the case of ResultsCollector, SubScopesSorter, etc. there is only one operator parameter, but there are other operators which have multiple operator parameters (e.g. branches). If we skip the parameter name, the user will not be able to tell which operator parameter is shown in the tree view.
    • abeham: After some while I got used to it.
  • All Parameters have the same icon, can we have different ones for values and operators?
    • abeham: Implemented this in r3106, fixed a related bug in r3105.
  • There is a bug in the implementation of SimulatedBinaryCrossover.
  • EvolutionStrategy:
    • Prevent self adaptive mutation with non-selfadaptive crossover.
    • BoundsChecking after Crossover and Mutation with optional attribute.
    • Implement SelfAdaptiveCrossover on the algorithm side with two ICrossover parameters.
      • abeham: Implemented in r3183 (the interface thing got killed, since it's ES specific).
  • Disable start button in prewired algorithm (LocalSearch, ...) that expect a problem, when no problem has been loaded. Also prevent start in trajectory based algorithms when e.g. MoveGenerator, etc. is null.
    • swagner: Implemented in SGA in r3188.
    • abeham: Done in r3189.
  • Tabu search should abort when no feasible move found in the neighborhood.
  • Hide choosing of different views when there is only one type of view to select.
  • When double clicking in a multi-line field (such as description) the text represents the old text and not any new text that the user has entered in the meantime.
    • swagner: Fixed in r3396.
  • ComparisonFactorModifier in OffspringSelectionGeneticAlgorithm should default to LinearDiscreteDoubleValueModifier
  • PermutationTypes is missing a view
  • The TypeSelector locks up the application when text is deleted in the search box (#904).
    • swagner: Fixed in r3516.
  • The "Copy To Clipboard" option in the Edit menu always seems to be disabled.
    • swagner: Fixed in r3518.
  • Using "Save all" with multiple open "documents" doesn't indicate which document is saved in the save file dialog.
    • swagner: Removed "Save All" in r3498.
  • The engine tab in an algorithm doesn't fill the whole space.
    • swagner: Fixed in r3505.
  • Closing the start page through the "x" still shows a check in the views menu. Also if it was closed that way and then "Close all" is chosen the start page cannot be shown again.
    • swagner: Fixed in r3519.
  • Evaluators of test functions allow name to be changed
    • abeham: Fixed in r3520 (as well as a few other operators too)
  • Some algorithms (e.g. TabuSearch) don't plot best solution and best known solution.
    • abeham: Fixed this for the tabu search in r3521
    • abeham: Fixed this for simulated annealing in r3522
  • Operator graph view
    • The "Zoom area tool" in the operator graph view does not really perform the action expected by the icon
    • Saving a screenshot from an operator graph and specifying the filename doesn't automatically add the extension
    • The icon for the screenshot button doesn't really indicate it's taking a screenshot, some kind of camera icon would be nicer
    • The icon for redraw the graph is okay, but usually there's one with 3 boxes connected by lines
    • UserDefinedAlgorithms cannot be edited (connections, operators adding or removing) through the graph view (either cloned or newly created)
    • "Break Point" (should be spelled "Breakpoint") and "Initial Operator" are selectable in the context menu although the graph view is not editable (they do nothing when clicked).
    • The context menu should display "Open View" in bold letters
    • Connecting the input port to another input port throws an exception
    • Deleting an operator from the chart still leaves the details there
    • Showing an operator graph in the chart and the tree view at the same time, selecting an operator in the chart, but deleting that operator from the graph in the tree view still shows a frame and the details in the chart
    • mkommend: implemented all errors above in r3514
    • Dragging an item from the clipboard does not really create a copy as the mouse cursor indicates. It will link them.
    • The ConditionalBranch gets a strange third false out-port in the lower middle if its false out-port is connected to another operator and the conditional branch is "breakpointed". To reproduce: Create a new user defined algorithm, switch to the operator graph view, drag in an EmptyOperator and a ConditionalBranch, connect the FalseBranch of the ConditionalBranch to the in-port of the EmptyOperator and then activate the breakpoint in the ConditionalBranch.
    • mkommend: implemented in r3570
  • Closing the clipboard and reopening through the views menu leads to an exception
    • mkommend: corrected in r3417
  • Bubble chart for RunCollection
    • The combobox listing the parameters for bubble size is too small to see the whole parameter name.
    • In a test with an ES and some runs, selecting bubble size to show whether plus or comma selection was used resulted in some runs to be hidden. The minimum bubble size seems to be such that it doesn't display the bubble at all.
    • I got a NullReferenceException when hovering over a bubble (when it tried to create the tooltip).
    • The tooltip of a bubble doesn't display the value for bubble size.
    • It would be good to have an "Index" parameter to be able to plot all the runs one after another.
    • mkommend: implemented in r3524
  • A KeyNotFoundException is thrown in Values_ItemsAdded when a DataTableView is closed while it is updated.
    • swagner: Fixed in r3561.
  • Using "Close all", also closes the operators list, clipboard and the start page. The start page okay, but operators list or clipboard I wouldn't have exptected to be closed.
    • mkommend: Fixed in r3571.
  • TabuSearch visualizes the current solution, but not the best
    • abeham: was decided with swagner to leave it that way for now
    • abeham: or not.. fixed with change to analyzers r3616 (#999)
  • Execution Time should be fetched as a parameter of a run.
  • When one is looking at the run collection bubble chart view and has opened the combobox of bubble size, hovering over a certain parameter with the mouse, this parameter is applied to the chart when an update to the run collection occurs (e.g. a new run is added). However the previously selected parameter is still displayed for bubble size in the combobox.
    • mkommend: I investigated a little deeper in to this and found out that when an additional run was added, the combobox shows the correct value again. IMHO the problem described above is a minor point.
    • abeham: I'm fine with that, moving this to lower priority.
    • mkommend: fixed in r3726.
  • When the TSP importer is used after pausing the execution, the engine should be reset.
    • swagner: Changed in r3739.
  • Opening a file that is not an HL file should display a nicer dialog than the PersistenceException that complains about a missing typecache.xml.
    • swagner: Changed in r3758.
  • Creating a UserDefinedAlgorithm from any existing wired algorithm before adding a new problem is not the best idea. There should be some kind of warning to the unexperienced user that he's entering dangerous territory if he clicks the prominent, big and inviting button "Create User Defined Algorithm".
    • swagner: Removed button and added menu item for creating user-defined algorithms in r3759.
  • Operator graph view: A breakpoint can be set in the details view, but not through the right-click context menu.
    • mkommend: Fixed in r3765.
  • Remember the optimizer to start maximized if it was maximized when last closed.
    • swagner: Implemented in r3768.
  • Would be good to be able to discriminate runs based on the problem type and instance when analyzing in a run collection.
    • swagner: Implemented in r3785.
  • Test functions do not feature a BestKnownSolution

Priority: MEDIUM

  • The TSP problem view:
    • How do I set the (allowed) evaluation operator?
      • swagner: Enabled changing values of value parameters in r2948.
    • Why do I want to change e.g. "TSPRoundedEuclideanPathEvaluator" in the textbox "Name" when viewing Parameter "Evaluator"?
      • swagner: In a discussion with abeham it was decided to allow name changes for all operators except for selection, crossover and manipulation operators (or any other operators which can be selected in a standard algorithm or problem view). This was implemented in r3034.
    • BestKnownTour is missing as parameter.
      • swagner: Added import of optimal TSP solutions in r3151.
    • Would be nice to be able to view a "scatter plot" of the coordinates.
      • swagner: Added visualization of the TSP instance in r3153.
    • The TSP path visualization seems to be flipped upside down.
      • swagner: Fixed in r3213.
  • When clicking "start" to start the engine disabling of the user interface elements has similar problems to what we fixed a few revisions ago (ticket #887).
  • The ViewHost menu contains a lot of views which are quite similar. ItemView doesn't display anything, should it be selectable? ParameterView and NamedItemView are very similar. Sometimes there are only very fine differences between the views (e.g. a ParameterView and a NamedItemView).
    • mkommend: Changed in r2992 as described in ticket #902.
  • In my user defined algorithm:
    • After selecting CurrentScope, all further parameters will be displayed as ParameterView and appear as if they cannot be changed. I had to change the view manually in the view host. Sometimes it seems to select the "wrong" default views. It reverts back to ParameterView as default as soon as CurrentScope is clicked again.
      • swagner: Fixed in r2949.
    • InversionManipulator has the "correct" actual name set already (Permutation), however in OrderCrossover the default names are "Parents" and "Child" which have to be changed to Permutation. This isn't obvious right away when the string description of the parameter says: "Child: Child (Permutation)".
      • swagner: Set the actual name of the parents and child parameter to "Permutation" by default in r2947.
    • My initial expectations probably weren't correct regarding the user defined algorithm. The SGA operator is not a complete SGA, but just the main loop (yes we discussed this).
      • swagner: Renamed SGAOperator to SGAMainLoop in r3020.
  • LeftSelector and RightSelector should not be displayed as selection operators in an SGA (there is a related ticket #673).
    • swagner: Replaced LeftSelector and RightSelector by BestSelector and WorstSelector as described in ticket #926.
  • I'd wish there was just a value view as well. Sometimes I want to hide name, description, type, set, remove, ... elements and just display the value (e.g. when viewing the quality chart).
    • swagner: Added VariableValueView in r2989.
  • In the operator graph chart view:
    • Long operator names require higher shapes as the operator name breaks into the next line.
      • mkommend: Implemented with r3169.
    • Scrolling using the mouse wheel looks very odd, if this can't be fixed it better should be disabled.
      • mkommend: Disabled the scrolling using the mouse wheel with r3167.
    • OpenView should be default double click behavior in operator graph chart.
      • mkommend: Implemented with r3184.
    • Sometimes arrows cannot be selected (e.g. when they're pointing directly east).
      • mkommend: Corrected selection of connections with r3175.
    • Just single clicking the successor port with the connector tool connects the box with itself, I don't think this is a desired behavior.
      • mkommend: Fixed with r3181.
    • I would be nice if: As long the space keyboard is pressed the selection mode switches to panning.
      • mkommend: Implemented with r3173.
    • The vertical scroll bar allows scrolling way past the graph.
      • abeham: On second review, this isn't actually an issue.
    • Sometimes one of the ports (in port) remains selected even if clicked somewhere in the white space.
      • mkommend: This is only a minor repainting issue and won't be fixed right now.
    • You can select a connector by drawing a selection rectangle that doesn't cover the connector, but covers the bounding rectangle of the connector.
      • mkommend: This is the default behavior. DiagramEntities are selected if the bounding boxes intersect the selection rectangle.
    • A selected box should change the mouse cursor to the move cursor when hovering over it.
      • mkommend: Implemented with r3186.
  • Viewing different result variables (switching between them) is impossible, because the results view is disabled when running the engine.
    • swagner: Changed in r3275.
  • The GeneticAlgorithm should use ProportionalSelector as default selection operator instead of BestSelector.
    • swagner: Implemented in r3304.
  • Saving a problem doesn't remember the filename when saving again.
    • swagner: Changed as described in ticket #990.
  • The default action for drag & drop actions should be copy (e.g. to/from the clipboard). Linking is more advanced, assumes programming knowledge, and can lead to all sorts of unwanted situations.
    • swagner: Changed in r3526.
  • The description in the start page should show the respective icon or menu entry as crop out of a screenshot in the explanation of what to do.
  • A problem should inject a variable that indicates its dimensionality (sometimes you want to set values depending on the dimension of the problem).
    • swagner: In a discussion with abeham it was decided that a problem dimension parameter is not really required.
  • The TSP problem view:
    • Would be nice to have paste support (e.g. from Excel) in the datagrid of the string convertible matrix view.
      • abeham: mkommend fixed this in r3643
  • In the operator graph chart view:
    • Some boxes are sometimes placed above the top end of the chart.
      • mkommend: This behavior could not be reproduced in r3176.
      • abeham: neither could I, about 400 revisions later
  • TypeSelector view: When only one single type is possible (e.g. DoubleData), select that type and auto-close with OK.
    • In a discussion with maffenze it was decided to show the type selector dialog in every case as we fear that it might be irritating for a user, if the dialog is sometimes shown and sometimes not shown.
  • Photoshop cannot parse the bitmap file format generated by the screenshot tool of the operator graph view. Is this a standard format?
    • gkronber: The file extension didn't match the actual file format (which is PNG by default). Fixed with r3753.

Priority: LOW

  • Sequential engine should probably not be createable. At least currently there's nothing one could do with just an engine.
    • swagner: Removed Creatable attribute in r2924.
  • When creating e.g. a new TSP in the parameter collection view the tool tip doesn't really display the description of the parameter but just a very generic one.
    • swagner: Changed to more specific tool tips in r2947.
  • There is a typo in the description of the parameters (tooltips), it says "it" instead of "in".
    • swagner: Fixed in r2947.
  • "Breakpoint" doesn't explain if it breaks before or after executing the operator.
    • swagner: Added tool tip in r2949.
  • The available parameters view lists just one namespace which is collapsed by default (should be expanded if it's just this one).
    • swagner: Changed in r2953.
  • When you remove the value of the results (such as "BestQuality") the text in the box does not update.
    • swagner: Fixed updating in r2989.
  • The quality chart looks great!
    • "BestKnownQuality" is not displayed currently.
      • swagner: Implemented tracking of best and best known quality values as described in ticket #920.
  • Does it make sense to select no engine in an user defined algorithm?
    • swagner: Disabled that the engine can be set to null in r3214.
  • Does it make sense to change the name of result variables or to remove their values?
    • swagner: Disabled in r3226.
  • The button for changing the evaluation in the TSP view features "+" which is used for adding, not setting in other views.
    • swagner: In a discussion with maffenze it was decided to use the "+" icon for adding and for setting values as long as there is no better idea which icon to use for setting values.
    • abeham: I'd suggest EditTable for change operations. Also in the results collection, I'd like to see different icons for DataTable and the solution visualization so that they pop out. I'd suggest Object, or if it hasn't been used already Properties.
    • swagner: Changed in r3306, r3327 and r3341.
  • VariableCreator throws an exception when injecting a variable whose value is null.
    • swagner: Fixed in r3503.
  • Going through the scopes with the arrow keys doesn't update the details view of the scope.
    • swagner: Fixed in r3528.
  • Resetting the engine doesn't update the global scope details view (user defined algorithm).
    • swagner: Fixed in r3528.
  • There should be a nicer way of saying the chart controls are not installed instead of an exception.
    • swagner: The HeuristicLab setup will check if the Microsoft Chart Controls are installed and will show a meaningful error message.
  • The tooltips should not exceed a certain width. With a long description you've got one line running over the whole screen.
    • swagner: It is not possible to define a maximum text width and to enforce word wrapping for tool tips. Therefore line breaks have to be set for each tool tip text individually. Please specify which tool tips are too long so that appropriate line breaks can be added.
    • abeham: has not been a problem for a long time
  • We do not currently count the number of evaluated solutions.
    • abeham: added evaluated solutions for offspring selection where this cannot be computed a priori in r3426
    • abeham: added evaluated solutions also for TS, SA, LS where it's possible, but uncomfortable to calculate this with the exhaustive move generators
  • Would be good to be able to hide certain lines in the quality line chart.
    • gkronber: fixed with r3703
  • Name in docking tab is not synchronized with the item name it displays.
    • mkommend: fixed in r3764.
  • The DataTableValuesCollector says it can only display double values, why not ints?
    • abeham: In r3489 the collector was expanded to display IEnumerable<DoubleValue>, this could further be extended to ints or probably solved through the unified INumber.
    • swagner: Enabled DataTableValuesCollector to collect ints and enumerables of ints in r3818. A unified INumber interface will not be implemented in HeuristicLab 3.3.

Reviewer: gkronber

Priority: HIGHEST

  • Symbolic regression problem & island offspring selection GA -> selection pressures in all islands are the same over the whole run (dropping from 2 initially to 1)?!
    • This seems to be an issue with overlapping variable names in selection pressure analyzer and best scaled validation solution analyzer.
    • swinkler: This problem is also seen in OS-GA with symbReg problem.
    • abeham: fixed in r3699
  • CheckedItemList (and probably CheckedItemCollection) are not working correctly if the same item is added multiple times to the collection. This leads to exception for instance if operators are reordered in a MultiAnalyzer. A fix is to check ifthe item is still referenced in the collection before removing it from the dictionary that holds the checked-states for each item.
  • Problem view is not shown after dragging a pre-configured algorithm onto a batch-run.
    • mkommend: fixed in r3805.

Priority: HIGH

  • "CTRL-X" closes the application. Even when I actually want to cut a text in a textbox!
    • swagner: Removed shortcut in r3517.
  • In the global scope view: If the selected item in the tree view is changed via keyboard (cursor keys) the Variables view is not updated.
    • swagner: Fixed in r3528.
  • The name of data analysis solutions is not set or displayed correctly.
    • swagner: Fixed in r3687.
  • Exception in BestQualityMemorizer because the actual name of the quality parameter is not updated. (To reproduce open a new user defined GA and then load a problem that changes the actual name of the quality parameter and then start the engine.)
    • swagner: Setting parameters automatically does not work in user defined algorithms. In a user defined algorithm all parameter settings (values, actual names, etc.) have to be set manually by the user.
  • Closing the Optimizer while the global scope view is shown (user defined algorithm) and expanded throws ArgumentException in HeuristicLab.Core.Views.ScopeView.Dispose().
    • swagner: Could not reproduce this exception in r3502.
  • Breakpoints should be visually emphasized.
    • swagner: At the moment, breakpoints are marked red in the OperatorTreeView and will be visually emphasized in the OperatorGraph visualization in the next version. Where else should they be emphasized?
  • BestKnownQuality tracking doesn't work correctly.
    1. The best average and worst quality are tracked by an analyzer configured by the algorithm.
    2. The best known solution is tracked by an analyzer configured by the problem.
    3. Problem specific analyzers are executed after the algorithm specific analyzers
    4. The best known solution value shown for generation 'i' in the qualities line chart is the value calculated by the problem-specific analyzer in the generation 'i-1'.
    5. The best known solution value displayed in the qualities line chart can be better than the best current best quality value from generations 0 to i
      • gkronber: Addressed with r3799.
      • abeham: Changes in r3799 look good. Fixed in r3809:3810 for tabu search and local search.

Priority: MEDIUM

  • HeuristicLab.Algorithms.TS and HeuristicLab.Algorithms.LS should be renamed to HeuristicLab.Algorithms.TabuSearch and HeuristicLab.Algorithms.LocalSearch.
    • abeham: Resolved.
  • HeuristicLab.Algorithms.SGA should be renamed to HeuristicLab.Algorithms.StandardGeneticAlgorithm.
    • swagner: Renamed HeuristicLab.Algorithms.SGA to HeuristicLab.Algorithms.GeneticAlgorithm as described in #945.
  • Loading the "New Item" dialog takes a long time the first time the "New" action is activated. Probably an issue in the plugin infrastructure.
    • swagner: When opening the new dialog for the first time, all creatable items are instantiated which might take some time (although I do not really notice a long loading time of the new dialog on my machine). At the moment many items are shown in the new dialog for testing purposes (category "Test"). After these items have been removed, the loading time of the new dialog will be much faster. I will remove all [Creatable("Test")] attributes on items in the next version.
    • swagner: Removed all [Creatable("Test")] attributes in r3160.
  • The execution state of an engine (e.g. Execution Time, Log...) is not persisted or restored correctly.
    • swagner: Fixed storing and loading of an engine's execution time and state in r2933. Implemented storing and loading of an engine's log in r3289 (#963).
  • Dropdown-boxes for crossover and mutation operator selection are empty.
    • swagner: They are empty as long as no problem has been selected as an algorithm cannot know a priori which crossover and mutation operators are allowed.
  • Importing new data into a symbolic regression problem for which a solution already exists throws an exception.
    • gkronber: fixed this issue temporarily. Import button is disabled after the data has been imported. (r3723)
  • Using an architecture manipulation operator when the number of function defining branches or function arguments is set to 0 leads to an exception.
    • gkronber: fixed with r3722

Priority: LOW

  • Are keyboard shortcuts for groupboxes necessary? E.g. Details groupbox and Description shortcuts are conflicting. Value groupbox and Value textbox are conflicting.
    • swagner: Removed keyboard shortcuts for all groupboxes in r2924.
  • Description of RandomCreator should contain a hint about the type of PRNG that is created.
  • ProgrammableOperator: 'Compile' is a normal button but 'Show Generated Code' looks like a label. I think the style should be unified.
    • epitzer: Fixed in r3008.
  • R², MSE, and relative Error of the best solution (for test and training set) should be collected into the results.
    • gkronber: implemented with r3710
  • Min selection granularity for the x-axis should be adapted to the range of displayed values in all line charts and scatter plots.
    • gkronber: implemented with r3707.
  • Only the best R², best MSE and best relative error should be collected into the results.
    • gkronber: implemented with r3710
  • Max, min and avg. tree size should not be collected into the results.
    • gkronber: implemented with r3710
  • All quality values tracked for symbolic regression problems should have nicely formatted names.
    • gkronber: implemented with r3710
  • Min selection granularity for zooming should be 1 for the x-axis in SymbolicRegressionSolutionLineChartView
    • gkronber: implemented with r3703
  • Function icon for the best symbolic regression solution.
    • gkronber: implemented with r3710
  • In the splash-screen: The display of plugin infrastructure messages looked better when the messages were displayed left-aligned.
    • swagner: Changed in r3813.

Reviewer: swinkler

Priority: HIGHEST

  • After running the SA/Rastrigin demo example I chose to display the results as bubble chart. Selecting Qualities as y-values and the relative difference of best known quality leads to an exception; I cannot even continue, the program has to be quit.
    • mkommend: Corrected in r3543.
  • Plugin Manager:
    • After deleting a plugin, how do I get it back?
    • In the list of available plugins I see the product "HeuristicLab", and after clicking "Install Selected Items" nothing happens - it is still there in the list of available plugins.
    • After deleting the plugin "Data Analysis" and starting the Optimizer I get the following exception: System.IO.FileNotFoundException: Could not load file or assembly 'HeuristicLab.Problems.DataAnalysis-3.3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=ba48961d6f65dcec' or one of its dependencies. The system cannot find the file specified.
    • swinkler: PluginManager has been improved meanwhile.
  • I just got an InvalidOperationException: "Drag Drop registration did not succeed". After clicking OK the program does not respond anymore. At the moment I cannot reproduce the error; what I did is the following: I created a new TSP instance, dragged it into the clipboard, opened a new island GA, dragged the TSP instance into it, started the algorithm, paused it, and clicked through the island results. Then suddenly this exception came.
    • Follow-up: I just got this exception again while clicking through results of a SASEGASA, whithout having dragged / dropped anything. Obviously I moved the mouse a little bit while clicking through the village results.
    • swagner: Fixed in r3552.
  • After starting an algorithm with a Knapsack problem the following InvalidOperationException is thrown: "Type mismatch. Variable 'KnapsackSolution' does not contain a 'ItemArray<BinaryVector>'".
    • Error not seen anymore.
  • After starting a SASEGASA with a TSP problem, an InvalidOperationException is thrown.
    • Error not seen anymore.
  • I have started a new ES (solving a TSP instance) and set breakpoint flag in the RandomCreater. After running and stopping the algorithm I cannot disable this breakpoint because the respective check box is disabled.
    • mkommend: corrected breakpoints behavior in r3696.
  • After defining a batch run (GA, TSP, 5 repetitions) and dragging it into an experiment, the "play" button of the experiment is still disabled.
    • Follow-up: The play-button is enabled after adding a new algorithm (and adding a problem to this algorithm).
    • swagner: Fixed in r3716.
  • HeuristicLab crashes fatally when I do the following: I run a GA solving the TSP ch130 problem several times, filter the results and let the results be displayed in a bubble chart (showing the best qualities). Then I start the GA with another problem (a Knapsack problem) without clearing the results, and then HL crashes.
    • gkronber: This might be caused by my changes to the bubble chart today (r3707).
    • mkommend: This is related to the implementation of RunCollectionConstraints and was corrected in r3717. Feel free to do some more testing.

Priority: HIGH

  • Using a population with 0 individuals leads to an overflow exception, no matter which selector is chosen.
    • swinkler: The exception message is understandable now.
  • After starting an SGA the user is able to add a new problem; the OK button in the following dialog is initially enabled, clicking it causes no (?) action.
    • swagner: Fixed in r2924.
  • Adding a new parameter, for example a ValueParameter<IntData>, leads to an exception.
    • swagner: Fixed in r2948.
  • After starting a SGA with 130 cities TSP and population size 100 the memory consumption is 1.2 GB after approx. 6500 generations.
    • swagner: Memory and runtime performance has been drastically improved in r2932.
  • Adding a paramter to a ProgrammableOperator, for example a ValueParameter<DoubleData>, leads to a null-pointer-exception.
  • (...) As a follow-up, after aborting adding a parameter to a ProgrammableOperator, a PersistenceException is thrown.
  • After loading a TSP instance into a TS and letting the TS, saving the TSP instance takes very long time (~ ten seconds)
    • abeham: It seems that the execution context and scope tree are persisted together with the problem. This is not specific to the TS, but with the exhaustive move generator the scope tree is noticeably larger and thus takes more time.
    • swagner: Fixed in r3501.
    • As a follow-up, loading this saved TSP instance fails (PersistenceException).
  • Setting the number of rows in the TSP coordinates view to 1 leads to an exception.
    • swagner: Fixed in r3502.
  • After creating a new TSP instance the user has to define the coordinates matrix; setting the number of columns to 2 and the rows to 4, e.g., leads to an overflow error.
    • swagner: Fixed in r3502.
  • After opening a new algorithm and creating a new knapsack problem, starting the algorithm causes an null-pointer-exception.
    • abeham: That's related to the MultiCrossover issue described by svonolfe below.
    • abeham: Fixed in r3527.
  • After opening a new algorithm and creating a new OneMax problem, starting the algorithm causes an null-pointer-exception.
    • abeham: That's related to the MultiCrossover issue described by svonolfe below.
    • abeham: Fixed in r3527.
  • SASEGASA does not show the best result found by the algorithm.
    • mkommend: As far as I know no multi population algorithm shows the best result.
    • swinkler: I'm not an expert on multi population algorithms, but I think there should be a display of the best solution. E.g., this is also what is published in papers.
    • abeham: This will be solved by the tracing / analysis operators that swagner is about to implement. Meanwhile you can see the best result by checking the result collections of the individual islands / villages.
    • abeham: Fixed as of r3673
  • Tabu Search: Adding a new symb reg problem disables the play button; that's ok, I suppose. But after adding a Knapsack problem and then a symb reg problem the play button is enabled, and that leads to an exception.
    • abeham: fixed in r3715 (the algorithm now just silently terminates)

Priority: MEDIUM

  • Algorithm testing: I have tested an OS-GA solving the TSP130 in HL2.5 (legacy plugins) and HL 3.3; pop. size: 100, success rate: 0.8, comparison factor: 0.0, max. selection pressure: 100, insertion mutation, CX crossover in HL2.5, CyclicCrossover2 in HL3.3. The resullts differ significantly: In HL 2.5 the produced solutions' quality is in the range 13,000 - 18,000, in HL3.3 it is in the range 20,000 - 24,000. Maybe I have overseen something?
    • swinkler: This problem is probably related to differences regarding the number of selected parents.
  • There is no way to see the optimal solution that is found by the algorithm, for example the best tour or the best knapsack selection. Even though not all users might be interested to see these best results, it might help to increase the confidence of users.
    • swinkler: Solution displays have been implemented meanwhile.
  • After starting the SGA the current generation index is not displayed; I have to switch to the qualities chart to see the current generation.
    • swagner: Added generations counter to results in r2924.
  • During the run of an algorithm the values displayed for the variables BestQuality, AverageQuality, and WorstQuality flicker.
    • swagner: Fixed in r2924.
  • After starting a new SGA and loading a problem the "play" button is enabled; clicking it leads to a error report that might be not interpretable by each potential user. A rather complicated check of all parameters would be needed here (as we had it for example in HL 2) - this might be not essentially necessary for the initial release (?), but maybe we could keep it in mind.
    • swagner: Changed in r2924 that a selection and a crossover operator are automatically selected if available to avoid the cryptic error message. In general I would prefer to set reasonable parameters by default instead of implementing some (probably quite sophisticated) parameter validation procedure.
  • I would suggest displaying the name of an item in the tab head; e.g., "Evolution Strategy" instead of "Item3.hl".
    • swagner: Changed in r3483. View captions still have to be checked, if they are meaningful.
  • After closing the start page it is still checked (as if it was opened) in the View menu.
    • swagner: Fixed in r3519.
  • After starting the Optimizer the cursor is still in "waiting" mode when hovering over the HL starter.
    • gkronber: Changed in r3573.
  • The lines in the variables impact graph can be a bit unclear, especially when many lines are shown. E.g., tooltips could help, maybe also displaying the impact / frequency values in a table.
    • swinkler: Implemented meanwhile
  • The progress of the R² value during the algorithm's execution is not shown.
    • swinkler: Implemented meanwhile
  • It would be nice to be able to save the whole clipboard to a file.
    • swagner: When the Save button is clicked, the whole content of the clipboard is saved and restored the next time when the optimizer is started. Should there be any additional saving (exporting/importing) functionality? I am not so sure about that ... What would be a typical use case for this feature?
    • swinkler: The idea here was that it could happen that I work on something (using several different problems or algorithms, e.g.), and that I would want to send my current configuration to a friend, so that s/he could use the same problems and algorithms and see what I am doing.
    • swagner: For this use case I would recommend to use an experiment. Additionally, we discussed that it might be nice to have "projects" in HeuristicLab which represent a collection of files/items (similar to a solution in VS). However, we will not be able to implement this in the HL 3.3.0 release. In any case, I am not really happy with adding import/export functionality to the clipboard. Please use experiments for saving and sharing multiple algorithms and problems for now as long as projects are still missing.
  • An exception is thrown if a variable is declared as input variable twice.
    • This error can not be produced anymore beacuse lists with check boxes are used now.
  • An exception is thrown if an input variable is misspelled.
    • This error can not be produced anymore beacuse lists with check boxes are used now.
  • If a string is shown twice in a results or parameters list (s:s; e.g., "R-squared Values: R-squared Values" or "Qualities:Qualities"), I prefer showing this string only once (if the name is not changable for the user).
    • swagner: Changed in r3688.
  • Quality chart of SA:
    • The legend "Quality 0" is suboptimal.
      • swagner: Changed in r3690.
  • I am not completely happy about being asked if I want to clear all runs after each click on the reset button. I would prefer to have a button for this in the runs section.
    • swagner: Changed in r3716.
  • As this just happened to me again: When closing the Optimizer I am not asked if I really want to close it. I know that this can be annoying, but how about a dialog asking if the user really wants to close HL if at least one tab opened that is not the start page.
    • swagner: Added dialog in r3741.
  • There is no Help menu, and also no About dialog.
    • gkronber: Implemented with r3736
  • When analyzing the results of a batch run in a table, the name of the tab page is "StringConvertibleMatrix (RunC..." - could this be replaced by "RunCollection Results Analysis", e.g.?
    • mkommend: Displayed ViewName in r3764.
  • Regarding the line charts (displaying quality values, R² values, variables frequencies, ...):
    • A possibility to view the values in tabular form would be nice (for copying them to Excel, e.g.).
      • mkommend: Implemented in r3767.
    • The tooltips are great; beside the display of the name of the parameter and the current value it would be nice to see in which generation this value was protocolled.
      • gkronber: Implemented in r3773.
  • Some analyzers are called "MinAvgMax*", other "MinAverageMax*". A unification of the namig of these analyzers would be nice.
    • swagner: Unified analyzer names in r3817.

Priority: LOW

  • Regarding the description of operators:
    • As the references to articles and books are incorporated in grammatically meaningful sentences (which I like a lot!) I would not use full stops (.) within references; I would use semicola (;) instead.
      • abeham: The point should be understood as being part of the reference than part of the sentence.
    • I would prefer to not to capitalize names of algorithms, i.e., for example I would prefer "... the standard genetic algorithm ..." over "... the Standard Genetic Algorithm ...".
      • swagner: Changed capitalization of algorithm names in r3026.
  • The name of an imported TSP instance is not stored / displayed.
    • swagner: Stored name of TSP instances imported from TSPLIB in r3147.
  • After changing the number of weights in a Knapsack problem there is an exception explaining that the number of weights is not equal to the number of values. This is quite understandable, but it could be better to give a parameter that defines the number of items and let the numbers of values and weights be set automatically.
    • svonolfe: Changed in r3463.
  • Charts (results display, quality progress, ...) can slow down the execution very much; it would be nice to have a parameter that sets the update interval.
    • swagner: Added MultiAnalyzer which supports update intervals in r3616.
  • The description of the "selected parents" parameter should be improved - for new users the meaning of this parameter could be unclear.
    • abeham: changed in r3735

Reviewer: maffenze

Priority: HIGHEST

Priority: HIGH

  • When no crossover is selected an error message is thrown which gives no hint about the reason.
    • swagner: Changed in r2924 that a selection and a crossover operator are automatically selected if available to avoid the cryptic error message.
  • Something like a starter package of problems (which do not have to be imported before) should also be included in the release version.
    • swagner: We can include several problems in the setup as it was the case in HL 1.1 or we can offer a set of problems on the same web page where the HL setup can be downloaded. I would prefer the second solution, as it is simpler to realize and the user can easily choose where he wants to save the sample problems. Furthermore, he can download the sample problems again if necessary without having to reinstall HL. However, the first solution has the benefit that the user does not have to download two files. Is the second solution acceptable?
    • swagner: In a discussion with maffenze it was decided to take the second solution (i.e. to offer a set of problems on the HL download page).

Priority: MEDIUM

  • After starting HeuristicLab it is not very obvious that the user has to choose the small new item button in order to have access to the standard methods.
    • swagner: Would it be an acceptable solution to show the new dialog automatically each time the HL 3.3 Optimizer is started? Another solution would be to show a wizard that guides the user through the first steps. If we want to have a wizard, should it show anything else than the available items that can be created?
    • gkronber: I think this is an important point because potential new users can be scared off within one minute if HeuristicLab doesn't have a few simple entry points to begin experimenting. One idea that comes to mind is to open a 'starter document' that has just 3 or 4 large buttons that activate a few basic entry points (e.g. "Genetic Algorithm: Optimize Tours", "Evolution Strategy: Find Minimum of Real-Valued Functions", "Show List of all Demos", "Load Algorithm From File"). Maybe we add an opt-out button for this starter screen. Sure a wizard of the kind "Solve your problem in 4 steps" would be nice but is probably not really necessary.
    • swagner: In a discussion with maffenze it was decided to show a "Start Page" when the optimizer is started that contains initial information how to use HeuristicLab and how to create new algorithms and problems.
    • swagner: A start page has been implemented in r3202. Is this enough to guide new users? Should there be any additional or other information shown on the start page?

Priority: LOW

  • It is not obvious what is meant with the breakpoint option in an operator.
    • swagner: Added tool tip in r2949.

Reviewer: vdorfer

Priority: HIGHEST

  • Got two different exceptions (made screenshots) when crazy clicking the results tab of a SASEGASA with TSP (when clicking on VillagesSelectionPressures):
    • InvalidOperationException: Invoke oder BeginInvoke kann für ein Steuerelement erst aufgerufen werden, wenn das Fensterhandle erstellt wurde
    • ObjectDisposedException: Auf das verworfene Objekt kann nicht zugegriffen werden, Objektname: "DataTableView"
    • swagner: Fixed in r3552.
  • InvalidOperationException: Selecting a MultiPermutationManipulator without selecting Mutators and starting the run leads to an InvalidOperationException ("Please add at least one permutation manipulator to choose from"); Exception persists if another Mutator is chosen
    • abeham: This is by (current) design.
    • swagner: If an exception is thrown during the execution of an algorithm, the algorithm is not reset automatically in order to be able to reproduce the error. If the mutator is changed, the algorithm has to be reset manually to remove the old mutator from the engine's execution stack. However, it is possible to add a mutator to the multi-operator after the exception has occurred and to continue the algorithm.
    • abeham: mkommend fixed this in r3674

Priority: HIGH

  • First of all: I would regard a user manual with first steps (shall I first create a new TSP or create a new SGA and how can they be combined...) as very important, for me at the first sight it was not really clear how to start.
    • swagner: A start page was added in r3202 which is shown when the optimizer is started. What do you think about it? Is it helpful when starting to work with HeuristicLab? Which additional information should be shown?
    • vdorfer: I like the start page, it gives a good first introduction. Maybe the second and the third example could run a bit longer...
    • swagner: At the moment the samples shown in the start page are just dummies. Meaningful samples will be added right before the release.
  • PersistenceException when saving a TabuSearch with a problem (no matter if TSP, Knapsack,..).
  • PersistenceException when saving an EvolutionStrategy (with and without a problem).
  • It is possible to start an algorithm with a TSP without having loaded the TSPLib file --> leads to IndexOutOfRangeException.
    • swagner: Fixed in r3504.
  • System.Runtime.InteropServices.ExternalException: "Allgemeiner Fehler in GDI+"; the following settings lead to this exception: new Experiment with EvolutionStrategy and GeneticAlgorithm with TSP 150 -> Run -> Analyze Runs -> Run Collection Bubble Chart -> set one axis to sth you like, the other one to "Coordinates", scroll down in the "Coordinates"-Drop-Down-Box with the scroll wheel, scroll back up to "Coordinates"
    • mkommend: fixed in r3536.

Priority: MEDIUM

  • I would change the stop sign to a pause sign, as it actually pauses the run (maybe also change the tooltip of the start button to "Start/Resume Algorithm"). A stop sign signalizes for me: better not click it, or else I have to start all over again.
    • mkommend: I would also recommend a pause icon instead of the stop icon.
    • abeham: Count me in on that too.
    • swinkler: Me too!
    • swagner: Ok, ok, I got it. Changed in r3225.
  • Add != Set Value; In all forms with a simple value (IntData, DoubleData, BoolData,..) a plus icon signalizes the possible change/edit. I would change the plus sign for example to a pencil (as you do not add a new value but change it).
  • A TabuSearch with a ch150 TSP is not running fluently, it jerks.
    • swagner: Which parameter settings of the tabu search were used? If an exhaustive move generator is used, the tabu search jerks as computing the whole set of possible moves is quite time-consuming.
    • vdorfer: I used the default parameter settings, which have an ExhaustiveTwoOptMoveGenerator as move generator.
    • abeham: Seems to run "fluently" in r3527, at least with a ch130.
    • vdorfer: runs fluently, problem caused by infrequent GUI updates, see comment of mkommend.
  • SASEGASA - Results Tab: The generation counter does not increase by 1 but jumps e.g. from 0 to 60 then to 71 ....
    • abeham: That's because each village is run until it terminates. If all villages are terminated the generations are updated displaying the maximum of the generation that you see jumping and the maximum of the generations of the villages. The name "Generations" is probably misleading. I am open for suggestions.
    • swinkler: I think "generations" is ok; we're not used to nonlinear progresses of the generations counter, but in this case it's ok.
    • abeham: fixed in r3615

Priority: LOW

Reviewer: mkommend

Priority: HIGHEST

  • ItemCollectionListView: reproducible InvalidOperationException occurs, if a saved experiment is loaded, started and then an algorithm is displayed by selecting an IOptimizer contained in the experiment.
    • swagner: Fixed in r3552.
  • The start page does not resize correctly. The list box with the samples always has the same height.
    • swagner: Changed in r3711.
  • Experiment: If rerunning an experiment without clearing the contained runs, the included batchruns are only execute once instead of 10 times.
    • swagner: Fixed in r3716.
  • MS Chart Controls: The folder Microsoft Chart Controls should not be included in HeuristicLab.Analysis.
    • swagner: Removed in r3815.

Priority: HIGH

  • Moving a running engine to another dock point (e.g. floating, right) makes the engine unresponsive (no repaint or reaction on mouse clicks).
    • mkommend: This issue was described and fixed in ticket #953.
  • The execution time textbox is only updated after an operation completed. If a long running operation is executed (e.g. SGA + TSP and population size 10000), the textbox is updated infrequently and it looks like as if the application hangs. This should be handled in the GUI by updating the execution time more frequent (every X milliseconds) in a separate thread.
    • mkommend: Changed in r3262.
  • Experiment - BatchRun: When stopping a BatchRun explicitly during the execution of an experiment, it often results in an error message stating that the BatchRun cannot be stopped while in stopped stated. I guess this is somehow related to the asynchronous GUI updating / executing of engines.
    • swagner: Fixed in r3716.

Priority: MEDIUM

  • In my opinion a reordering of the tabs in the SGAView should be considered. The current order is Parameters, Problem, Engine, Results. It would be better if the order reflects the workflow to create an engine, e.g. Problem, Parameters, Results, Engine.
    • swagner: Changed tab order in r3505.

Priority: LOW

  • The new dialog should also react on double clicks on the description column, if this is possible.
    • swagner: Implemented in r3202.

Reviewer: epitzer

Priority: HIGHEST

Priority: HIGH

  • Empty TSP problem always throws an exception.
    • swagner: Fixed in r3504.
  • Searching for "prog" in the operator pane and using backspace twice => application hangs.
    • abeham: See also ticket #904.
    • swagner: Fixed in r3516.

Priority: MEDIUM

  • Remove "-" entry from list of available engines if an engine is available or maybe rename to e.g. "no engine".
    • swagner: Changed in r3214.
  • Clicking or double clicking on one of the operators in the operator pane does not do anything this is the first thing useres see and they can't interact with it until rather late in the "development" of a user, when he is ready to create a UserDefinedAlgorithm.
    • Maybe the operators pane should be hidden by default, or only shown once an algorihtm graph is loaded or the click action could give an explanation why nothing happens and that you should drag them over into an operator graph.
    • Another idea, double clicking an operator could add it to the currently visible operator graph. (Probably not so easy to implement)
    • swagner: Changed in r3290 that the operators sidebar is not visible by default.
  • The [+] Button in the ValueParameterView is slightly misleading once a value has been added, then it should mean change the value.
  • Some algorithm parameters cannot be set until problem is selected: display reason why parameters cannot be changed yet or maybe make the Problem tab the default one for Algorithms.
    • swagner: Changed tab order in r3505.
  • A Help menu should be available, maybe also an about dialog.
    • gkronber: implemented with r3736

Priority: LOW

  • Value of Items from the Results pane can be added/changed and deleted.
    • swagner: Disabled in r3226.
  • Engine has to be stopped to be able to click and view results.
    • swagner: Changed in r3275.
  • Import from TSPLIB button is easily overlooked (TSP Problem).
    • swagner: Changed position of import button in r3505.
  • The Solution Visualization icon of an Artificial Ant Problem is not the same as in other problems
    • gkronber: fixed with r3710

Reviewer: svonolfe

Priority: HIGHEST

  • In the class EngineAlgorithm the method CreateUserDefinedAlgorithm should be made virtual and the return value of CreateUserDefinedAlgorithm should be IAlgorithm.
    • swagner: Changed in r3551.
  • Local search crashes if a TSP instance with less than 4 cities is solved
    • swinkler: It seems to me that this problem occurs a TSP instance with less than 3 cities is solved
    • abeham: I would argue that there does not exist a Traveling Salesman *Problem* with 3 or less cities.
    • swinkler: Yes, on the one hand you're right here, but on the other hand one could also argue that that's an optimization problem that is really easy to solve. That's why I think that the algorithm should not crash whensolving a TSP with 3, 2, or even one city.
    • abeham: done in r3706. I'm throwing an exception if there's just one city, because there's no way you can e.g. invert a "tour" of just one city.
  • When trying to resize the coordinates matrix in the TSP view an IndexOutOfRangeException occurs in UpdateRowHeaders

Priority: HIGH

Priority: MEDIUM

  • When choosing the MultiXCrossover (e.g. MultiBinaryCrossover) and no Operators are selected an exception should be thrown, otherwise e.g. the GA crashes at a later point (e.g. evaluation)

Priority: LOW

Other Reviewers

Priority: HIGHEST

  • Sometimes child controls are not correctly resized when the Optimizer is maximized.
    • mkommend: Fixed in r3572.

Priority: HIGH

  • The name and description of a run should be editable.
    • swagner: Changed in r3766.
  • When switching between the result collections in a multi-population algorithm, the shown result collection view should memorize the selected result and should show its details automatically.
    • swagner: Implemented in r3775.

Priority: MEDIUM

Priority: LOW

Last modified 15 years ago Last modified on 05/17/10 23:29:51