Enhance "Create Experiment" dialog

[abeham]

Using the new problem instance infrastructure we can automatically create experiments where a certain algorithm is applied to a range of problem instances. We could detect that the algorithm contains a problem for which there are instances available. The user can then select the instances that the algorithm should be applied on.

[abeham]

r7841: Updated CreateExperimentDialog to discover and display available problem instances...

[abeham]

I'm going to make experiment creation an async operation since it can take several seconds for larger experiments.

[abeham]

r7846:

• Moved common methods from problem instance management into a ProblemInstanceManager
• Made experiment creation asynchronous by using a BackgroundWorker
• Added a progress bar to display the progress as well as the instance that is currently processed
• Added "select all" and "select none" options which are updated to the state of the listview

[abeham]

r7885:

• Made instance discovery asynchronous and interruptible
• Added check if instance could be loaded

[abeham]

r7908: Added possibility to create simple parameter variation experiments

[abeham]

r7957:

• Used a TreeView instead of a ListView to display the instances
• Fixed the bug regarding algorithms without a defined problem
• Improved the UI speed
• Added a warning sign if the number of variations is greater than 1000
• Stated for every parameter if it was an int, double, or bool parameter
• The .resx contains the warning icon for the error provider and should not be removed

[abeham]

swinkler reported another issue (#1868): "Exception occors when creating an experiment and trying to vary a String parameter. To reproduce this open a new SVM, e.g., create experiment, and click SvmType."

[abeham]

r7974:

• fixed problem with choices that are not INamedItem
• fixed a bug in the enumeration of choice parameters
• improved UI feedback

[gkronber]

The dialog throws a cross-thread exception when the backgroundworker is finished.

[gkronber]

Isn't there an equivalent warning icon in HeuristicLab.Common.Resources?

[gkronber]

r7975: fixed cross-thread exception related to the backgroundworker for creating experiments.

[gkronber]

At least for regression problems the problem instance is not set correctly in the created experiments.

[abeham]

Okay, I'll look into it, but the catch and finally on a new line don't look good in my opinion. We don't have else on a new line either.

[gkronber]

r7976: fixed a bug which had the effect that the problem instance was not set correctly for each variant

[gkronber]

sorry for the formatting.

[abeham]

thx for fixing this

[abeham]

Since you had a look at this already, I have an open issue and wanted to get your input: When you define a range for an integer parameter e.g. Minimum = 1, Maximum = 1000, Step = 100. Would you expect to get the sequence: 1, 101, 201, ..., 901 or would you expect to get (1,) 100, 200, ..., 1000.

Currently you obtain the first sequence because enumeration starts at the minimum and adds the step value while its <= the maximumm, but I think of changing it to the second. To get the second sequence would be to start at the maximum and subtract step (and finally add the minimum). Also would you always expect that it includes the maximum as well as the minimum value in the sequence?

Edit: Or is it better to forget about defining the sequence as an arithmetic progression and just type all the values directly, e.g. through a ";"-separated list. Since sometimes you could want a geometric progression of the values. Direct specification may be inconvenient for more than say 10 values, but I think that one could always create larger sequences in Excel quickly and paste that into the box. Alternatively we could use an IntArray or DoubleArray and the according view (which already supports pasting from Excel).

[swagner]

May I join this discussion and throw in my opinion?

• probably we need both, sequences defined by a minimum, maximum and increment AND explicitly given values (although the second case is more important). For the first case I would recommend the following syntax min:max/step (e.g. 1:1000/100) and for the second case I think a semicolon-separated list is fine.

• for 1:1000/100 I would expect the sequence 1, 100, 200, 300, ..., 1000; minimum and maximum value should be included.

[abeham]

Hmmm, I'm not so much a fan of defining and parsing a syntax - it's something the user has to learn first and then remember later.

How about we remember and display only the actual sequence of values and allow the user to generate that sequence in the UI? So he can type in min, max, and step in respective text boxes then hit a button (or on textbox validation) and the values are generated and displayed in something similar to a StringConvertibleArrayView. This has the advantage that the values can also be modified after they're generated and it's also always clear what the actual values are going to be.

[swagner]

Yes, that is a good idea. By the way, I recommend to use a button and not the textbox validation events, so that it is clear for the user, when new values are created and maybe already existing values are overwritten in the StringConvertibleArrayView.

[abeham]

r7985:

• Allow to specify each value of numeric parameters explicitly as well as provide a dialog for generating the values
• Changed SetMode() call back to originate from DoWork, since that it is the right place to make the call, but added a check for InvokeRequired in SetMode()

[swinkler]

We have just found the following problem: When we create hundreds of SVMs by varying several algorithm parameters and several of these algorithm combinations lead to exceptions, then it can happen that the user is forced to react to hundreds of exceptions with the same type and exception message (e.g., "specified nu is infeasible").

[gkronber]

Replying to swinkler:

We have just found the following problem: When we create hundreds of SVMs by varying several algorithm parameters and several of these algorithm combinations lead to exceptions, then it can happen that the user is forced to react to hundreds of exceptions with the same type and exception message (e.g., "specified nu is infeasible").

Thanks for reporting this issue. After discussion with mkommend we came to the conclusion that I'd probably best if we handle this in the experiment. By adding a flag to indicate if an experiment should stop on the first exception or aggregate all exceptions to show them at the end. I'm opening a new ticket for this (#1872)

[gkronber]

Btw. "specified nu is infeasible" happens if you have unbalanced class distributions. You should limit the range of nu to a subrange of ]0..1] in this case.

[ascheibe]

I just played around with the new Create Experiment dialog and found that the DefineArithmeticProgressionDialog popped up behind the Create Experiment Dialog. Fixed in r8011.

[abeham]

r8012: fixed cancellation of experiment creation

[abeham]

r8013: Enabled to set Value to null for OptionalConstrainedValueParameters

[abeham]

r8014:

• made dialog a bit larger
• added clearing of the run collection of the cloned optimizer
• renamed variations tab

[gkronber]

An exception occurs when checking/unchecking a problem instance provider that does not have problem instances (CSV Problem Provider)

[gkronber]

• The dialog sets the 'all' checked box already when all checkboxes except for the CSV Problem Provider are checked.
• The dialog should not be displayed on top of all windows (click create experiment -> switch to another application (visual studio) -> dialog is still on top).

[abeham]

Good point, I will exclude all providers that do not have a list of instances from being shown. I don't know an easy way to make them useful in this feature.

[gkronber]

Good idea.

[abeham]

r8095:

• Changed to not display instance providers that do not contain data descriptors
• Added some default values when checking an int/double parameter (made some rough guesses on what reasonable variations could be like based on the given value)
• Set TopMost property to false

[gkronber]

reviewed r8011:8013

[gkronber]

Why do we need Application.DoEvents() in the event handlers for ProgressChanged (line 692 and 875)?

[gkronber]

r8179: minor code improvements in CreateExperimentDialog

[abeham]

At some point in the development I was firing a lot of events and they didn't got updated in the UI so I was thinking this would do the trick. But I then changed the number of events that are fired and forgot to remove it. Does it hurt?

[gkronber]

Please remove it since its not necessary

[abeham]

r8188: Removed calls to Application.DoEvents()