Reviews - HeuristicLab 3.3.0 Application Review

Review comments which have been implemented are listed here.

Reviewer: abeham

Priority: HIGHEST

• Adding a ValueParameter<T> in a VariableCreator and selecting a generic type e.g. ItemList<T> fails because no choice can be made on the generic type of the ItemList (ticket #42).
  • abeham: swagner fixed this, but there was still an issue with an ItemList<IItemList<T>> which cannot be assigned a value

Priority: HIGH

• When there is an error in one of the item's constructors, the TypeSelector dialog (when pressing the "new" button) will open the first time, but stop listing items after encountering the item with the error. The second time the TypeSelector is shown through the "new" button, the optimizer crashes with an exception.
• Operator graph view:
  • The boxes grow in size when the text gets larger, but they do not shrink again when the text gets smaller.
  • Sometimes the arrow is not drawn when connecting two operators, or it's drawn wrong.
  • The red square surrounding a port still remains selected sometimes after creating a connection.
• Operator graph tree-view:
  • The breakpoint icon looks a bit big.
  • swagner: This is the default breakpoint icon which was taken from Visual Studio.
• Would be good to be able to discriminate runs based on the problem type and instance when analyzing in a run collection
• Could be interesting to plot PercentTabu over time
• Test functions do not feature a BestKnownSolution

Priority: MEDIUM

• Every IDeepCloneable item needs to have a public default constructor. If Activator.CreateInstance in DeepCloneable.Clone is given a second parameter "true", non-public constructors would also be used.
  • swagner: See ticket #922.
• Adding a new parameter through the GUI and choosing a ConstrainedValueParameter does not make sense as the set of valid values cannot be defined anywhere in the GUI. The parameter will never be able to take a value. Additionally, switching the view for this ConstrainedValueParameter to a ValueParameterView and setting a value there leads to an exception. The ValueParameterView should not be a view for the ConstrainedValueParameter.
• Adding a new ValueParameter<T> could create a default value if T is instantiable. For most ValueParameters you will want to have a value upon creation.
• The tabs don't change when you drag something over them (e.g. you have parameters tab open and drag a problem in: So you have to cancel drag, switch to problem tab and redo the drag).
• In the operator graph view it's possible to add operators to the selection by (ctrl|shift)-clicking, but not removing them again through another (ctrl|shift)-click.
  • mkommend: This is the default behavior of the Netron library. As far as I understand the source it is related to selection of grouped / stacked items.
    • abeham: If that means it's impossible to change please delete the whole issue. I'm moving it down in the priority list.
• Placing two operators in the operator graph view above each other and left-clicking selects the operator on top, but right-clicking opens the context menu on the bottom one
• When removing the initial operator in a user defined algorithm the solve button is still enabled (there is no exception)
• I create a batch run, produce some runs, clear the runs and save the batch run. When I load it again and produce some more runs they'll have IDs starting from the last run produced.

Priority: LOW

• The quality chart looks great!
  • We should probably use the same colors we had in 1.1 for best/average/worst qualities.
  • At some point we'll certainly need export to png, eps, etc. functionality (would be great to be able to specify resolution and width/height).
• The DataTableValuesCollector says it can only display double values, why not ints?
  • abeham: In r3489 the collector was expanded to display IEnumerable<DoubleValue>, this could further be extended to ints or probably solved through the unified INumber
• I got a very cryptic "Operation is not valid due to the current state of the object" exception at one point building the TS operator (as discussed Parameters should have an optional attribute that says they should throw an error in case ActualValue is null after retrieving).
• Improve description of SearchIntervalFactor in BreederGeneticAlgorithmManipulator.
• EvolutionStrategy:
  • Add parameter to ES to indicate if 1/5 success rule or sigma-self adapation should be used.

Open Discussions

• The operator graph view is a bit confusing:
  • The flow chart is very nice, but sometimes arrows are a bit wild and going backwards which creates a not so nice strikethrough appearance. Would be good to have non-direct arrows as well.
    • swagner: In a discussion with maffenze it was decided to address this issue after the release of HeuristicLab 3.3.0.
• The OperatorGraphVisualization doesn't respect operator collections (e.g. SequentialSubScopesProcessor).
  • mkommend: This feature will rarely be used by a standard user and it will not be fixed right now. Although special shapes for MultipleCallOperators are planned.
• In TypeSelector filter types that don't have a constructor (like currently PathTSPTour).
  • swagner: Generally all items should provide a parameterless constructor. I implemented one for PathTSPTour in r3139.
• Run collection: By drag-drop runs can be duplicated (also reported by vdorfer).
  • swagner: Dropping runs into a run collection is allowed to enable users to copy/move interesting runs into a run collection and to analyze these runs later on. If we disable drop operations on run collections to avoid the duplication of runs, this will not be possible anymore. Do you think that this feature is not required?
    • abeham: In my opinion the algorithm view should not be used as analyzer for runs other than the ones it created. It would be better to allow creating a new run collection and drop interesting runs there for further analysis.
• In the operator graph chart view:
  • Top-down layouting would be better instead of left-right for two reasons:
    1. The boxes are usually much wider than high and thus a left-right layout makes the graph even longer.
    2. A top-down graph better fits the paper orientation in a publication.
    • mkommend: This will be implemented with the other changes regarding the layout of the operator graph.
• Focus is not removed from a textbox if one clicks anywhere in the "gray" area.
  • swagner: This is the behavior of Windows Forms.
  • abeham: I mentioned this because I want to validate the control by clicking out of the text box and thus losing the focus. Maybe this can be achieved by allowing focus on the control or form. I think by default the "gray area" doesn't accept focus.
• The view host icon should act like a button that when single-clicked opens the context-menu, when hovering over it, there should be a frame around it like a button so that it's clear that it can be clicked.
  • swagner: When hovering over the view host icon a tool tip is displayed that explains how the different views can be selected.
  • abeham: I'm still not fully happy with the view host icon, but it's probably the least priority thing right now
• Some operators have a Parameter "CurrentScope" that cannot be changed in any way, is there a necessity to display this parameter?
  • swagner: The scope parameter "CurrentScope" indicates that the current scope (i.e. its sub-scopes) is manipulated by an operator. It is shown in order to give the user comprehensive information about what an operator does. Maybe this is not really helpful. Should I remove scope parameters in general? In the code of an operator they are not really required, as the current scope can always be accessed via the current execution context.
  • swagner: In a discussion with maffenze it was decided to keep scope parameters.
  • abeham: I think more parameter are just more confusing and description of the internal mechanisms of an operator are better displayed in the description text. I'll vote for removal of the current scope parameters, but I'll move this to low priority for now.

---

Reviewer: gkronber

Priority: HIGHEST

Priority: HIGH

• Read-only parameters would be nice (similar to the data items of HeuristicLab.Data)
• It should be possible to show/hide parameters on demand in order to create simplified GUIs for not so experienced users.

Priority: MEDIUM

• It seems strange now that operators have the property Breakpoint. It is more usual to set breakpoints on invocations/calls of operators. Also isn't the engine responsible for breakpoints?
  • swagner: As in HL 3.2 the Breakpoint property is implemented in operators and can be changed in the GUI of an OperatorGraph. Additionally, I decided to add a checkbox to each operator in order to be able to mark breakpoints there as well (this was not possible in HL 3.2). If an operator is marked as breakpoint, an engine stops its execution after this operator has been processed. I agree that it would be more intuitive to store it in some other location, which operators are currently breakpoints. However, I do not immediately know where. Any suggestions?

Priority: LOW

• Relative errors should be displayed as percentage values and rounded to ~4 digits after the decimal point.

---

Reviewer: swinkler

Priority: HIGHEST

Priority: HIGH

Priority: MEDIUM

• IMHO the meaning of the result "CurrentSuccessRatio" is not completely clear, especially because there is no description given. This value rises steadily during a generation of an OS-GA; from the publications one could expect this parameter to be either constant (because it is a parameter of the OS) or fluctuating (because it represents something like the "success rate").
  • abeham: It's an indication of how much of the currently generated offspring population in the OS step is successful compared to the total population. It's not a result as such, but rather a live status feedback. If it's causing confusion I'll remove it, but I'll have to talk to maffenze about this first.
• I like the graphical display of the twodimensional test function solutions - I would suggest adding the display of min and max displayed values.
• The details of the GP grammar currently used are not displayed.
• Regarding the line charts (displaying quality values, R² values, variables frequencies, ...):
  • A possibility to view the values in tabular form would be nice (for copying them to Excel, e.g.).
    • mkommend: Implemented in r3767.
  • The tooltips are great; beside the display of the name of the parameter and the current value it would be nice to see in which generation this value was protocolled.
  • Setting the number of generations to n leads to the display of all tracked values for generations 1 to (n+1) - this looks a bit strange.

Priority: LOW

• In the symbReg problem parameter "SymbolicExpressionTreeInterpreter:" I would remove the ":"

Open Discussions

• Could it be reasonable to display a special icon for parameters that are not yet set properly (e.g., with a red exclamation mark etc.)?
  • swagner: This will be quite hard as it is not easy to decide programmatically, if a parameter is properly set or not. But we can think about it. Maybe we can introduce constrained parameters? Although it will be a lot of work to enable users to define constraints of parameters in the GUI (for example when adding a new parameter to a CombinedOperator).
• On a screen with resolution 1280x800 I can for example not change the group size of tournament selection because the text box is not visible (unless the OptionalValueParameter View is chosen).
  • swagner: Can you give me some hints where we can save space? Which information/controls might not be necessary?
  • swinkler: As names and data types cannot be edited in the details sections of parameters, I would suggest combining these two into one TextBox and showing data types in brackets (as it is done in the list box shown left, e.g.).
• IMHO the standard selection operators in gender specific selection should be flipped; this would more reflect behavior in nature. The reason for this is that these selectors do not specify how fe/male individuals select their mates, but rather how they are selected.

---

Reviewer: maffenze

Priority: HIGHEST

Priority: HIGH

Priority: MEDIUM

Priority: LOW

---

Reviewer: vdorfer

Priority: HIGHEST

• When clicking on the open file dialog before loading of the start page is finished the program crashes without exception.
  • swagner: Could not reproduce this exception in r3561. Does it always occur? May you give me some additional hints?
  • vdorfer: It is reproducible in the release candidate and in r3561 - if not started in debug mode of VS! The open command has to be made in the first half of the loading bar.
  • abeham: Could not reproduce this exception in r3695.
  • swagner: I still cannot reproduce this exception on my machine. However, I saw it on the workstation of vdorfer. Very strange. As this exception seems to occur only on the workstation of vdorfer (at least as far as we know by now), I do not think that we have to spend more time on this issue.

Priority: HIGH

Priority: MEDIUM

• Is it really intended that runs in the run tab can be copied (through drag&drop)?
  • swagner: Dropping runs into a run collection is allowed to enable users to copy/move interesting runs into a run collection and to analyze these runs later on. If we disable drop operations on run collections to avoid the duplication of runs, this will not be possible anymore. Do you think that this feature is not required?
• The "Show Algorithm" button in the run tab opens a new tab containing the algorithm but no runs and no results - shouldn't it rather show the problem tab in the current algorithm?
  • swagner: A new algorithm view is shown as the algorithm stored in a run is cloned when "Show Algorithm" is clicked to avoid any manipulation of the algorithm that created the run. The algorithm stored in a run does not contain any runs or results as this would require too much memory.
• In the results view I would show by default the quality data table (instead of first having to stop (pause) the algorithm, select the data table and resume the algorithm). When the tour visualization is done, this would be best to show by default.
  • swagner: Showing results by default is critical, as the continuous redraw of a result's value (e.g. quality chart, tour visualization) might consume a lot of runtime. Therefore the user should have to select a result manually, if he wants to look at a result.
  • abeham: I agree with swagner and I think the bigger issue (at that time) was that results were locked. Now you can click around them freely while the algorithm continues to run.
• A "Do you want to save your work" question is missing when clicking the "x" of an open SGA, TSP, ...
  • swagner: In a discussion with maffenze it was decided not to show such a message box. When closing a view the optimizer cannot know, if the content of this view has been changed or not. Therefore, it would have to ask the user every time which might be quite annoying. As an alternative a message box will be shown when the optimizer itself is closed (see review comment of swinkler).
• Data Analysis Problem: Trying to import a csv file which is opened in Excel leads to an IOException (cannot access file, used by another process)

Priority: LOW

• The comments in the generated code of a ProgrammableOperator are German.
  • epitzer: This is a feature of the .NET runtime and depends on the installed operating system language.

---

Reviewer: mkommend

Priority: HIGHEST

• MS Chart Controls: The folder Microsoft Chart Controls should not be included in HeuristicLab.Analysis.
• Results: Unify naming and spelling of results (e.g. with blank or without, ...). Further more the result Generations should be renamed, e.g. ActualGeneration, Generation, ...

Priority: HIGH

• Parameter checks should be added to every algorithm. E.g. it should not be possible to start an GA/OSGA with negative or zero population size. Neg. population size results in an ArithmeticOverflowException in the selection operator, because arrays of negative size could not be allocated.
• Negative mutation rates or maximum generations do not produce an error, but are meaningless and should IMHO also be checked.
• The start page does not respect the loading cursor.
  • swagner: This is the default behavior of Windows Forms. The cursor property is not automatically propagated to all child controls. Consequently, the app starting cursor is not shown for example when hovering over the first steps text.

Priority: MEDIUM

Priority: LOW

• GP Solution: The usage of views for sym reg GP solutions should be improved (e.g. by using tabs).

---

Reviewer: epitzer

Priority: HIGHEST

Priority: HIGH

Priority: MEDIUM

• Problem parameters that cannot be changed by the user could be hidden.
  • i.e. Maximation is always the first parameter but can rarely be changed.
  • swagner: In a discussion with maffenze it was decided not to hide parameters. The parameters collection of a problem should show all parameters of a problem so that a user sees and understands how the underlying problem object looks like. This should help users when they start to implement own problems.
• The [+] Button in the ValueParameterView should also be disabled if the type of the parameter cannot be changed.
  • swagner: Even if the type of a value parameter cannot be changed, it might be necessary to be able to exchange the whole value object. Therefore the [+] button should be enabled in every case.
• New Item Dialog: Algorithm and Problem items should be listed before the advanced items in "Algorithm Design".
  • swagner: In a discussion with maffenze it was decided to keep alphabetical sorting of creatable groups in the new item dialog. Alphabetical sorting is used in many places throughout the whole application and it might be irritating for a user if we change this in the new item dialog. Furthermore, it depends on a user which creatable groups are considered to be more important.
• Parameters of sub-operators that are not changeable by the user should be hidden (e.g. lookup parameter).
• Graph View
  • Icon for expanding/collapsing details is rather confusing (undo/redo), should be +/- imho
  • A zoom rectangle tool would be nice
  • panning while pressing the middle mouse button would be nice too
  • display updates while scrolling and panning are rather slow

Priority: LOW

• Show results tab when engine is started, set a default item that is selected too (e.g. quality chart).
  • swagner: Showing results by default is critical, as the continuous redraw of a result's value (e.g. a quality chart) might consume a lot of runtime. Therefore the user should have to select a result manually, if he wants to look at a result.
• Hierarchical grouping of operators by namespace in operator tab (instead of just by plugin) or maybe just remove the intial "HeuristicLab" as this is common for all of them and makes it harder to scan for the right plugin/namespace.
  • swagner: In a discussion with maffenze it was decided not to change the way how operators are grouped and displayed in the operators sidebar. Reasons are that grouping hierarchically by namespaces might lead to deep trees that have to be opened to get an operator and that the initial "HeuristicLab" in each plugin name might not be common for all plugins if an external user develops an own plugin.

---

Reviewer: svonolfen

Priority: HIGHEST

Priority: HIGH

Priority: MEDIUM

Priority: LOW

• The view flickers, because double buffering is not enabled.

---

Other Reviewers

Priority: HIGHEST

Priority: HIGH

• When switching between the result collections in a multi-population algorithm, the shown result collection view should memorize the selected result and should show its details automatically.

Priority: MEDIUM

• It would be better to have scrollbars instead of infinitely small controls. Maybe this could be achieved by setting the MinimumSize for each View/Control.
• It would be nice to be able to export the images of solution visualizations to a file.
• For some items it is possible to switch between different views. However, some of these views do not seem to make any sense. For example, why is it possible to switch between "EngineAlgorithm View" and "ParameterizedNamedItem View" in an algorithm or between "ValueParameter View", "LookupParameter View" and "ValueLookupParameter View" in a parameter?

Priority: LOW

• After zooming out in the operator graph visualization multiple times (approx. 50 times) an exception is thrown.

---