Opened 5 years ago

Closed 5 years ago

#1851 closed enhancement (done)

Enhance "Create Experiment" dialog

Reported by: abeham Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.7
Component: Optimizer Version: 3.3.7
Keywords: Cc:

Description

Using the new problem instance infrastructure we can automatically create experiments where a certain algorithm is applied to a range of problem instances. We could detect that the algorithm contains a problem for which there are instances available. The user can then select the instances that the algorithm should be applied on.

Change History (44)

comment:1 Changed 5 years ago by abeham

  • Owner changed from swagner to abeham
  • Status changed from new to accepted

comment:2 Changed 5 years ago by abeham

r7841: Updated CreateExperimentDialog to discover and display available problem instances...

comment:3 Changed 5 years ago by abeham

  • Owner changed from abeham to gkronber
  • Status changed from accepted to reviewing

comment:4 Changed 5 years ago by abeham

  • Owner changed from gkronber to abeham
  • Status changed from reviewing to assigned

I'm going to make experiment creation an async operation since it can take several seconds for larger experiments.

comment:5 Changed 5 years ago by abeham

r7846:

  • Moved common methods from problem instance management into a ProblemInstanceManager
  • Made experiment creation asynchronous by using a BackgroundWorker
  • Added a progress bar to display the progress as well as the instance that is currently processed
  • Added "select all" and "select none" options which are updated to the state of the listview

comment:6 Changed 5 years ago by abeham

  • Owner changed from abeham to gkronber
  • Status changed from assigned to reviewing

comment:7 Changed 5 years ago by abeham

r7885:

  • Made instance discovery asynchronous and interruptible
  • Added check if instance could be loaded

comment:8 Changed 5 years ago by abeham

r7908: Added possibility to create simple parameter variation experiments

comment:9 Changed 5 years ago by abeham

r7957:

  • Used a TreeView instead of a ListView to display the instances
  • Fixed the bug regarding algorithms without a defined problem
  • Improved the UI speed
  • Added a warning sign if the number of variations is greater than 1000
  • Stated for every parameter if it was an int, double, or bool parameter
  • The .resx contains the warning icon for the error provider and should not be removed

comment:10 Changed 5 years ago by abeham

  • Owner changed from gkronber to abeham
  • Status changed from reviewing to assigned

swinkler reported another issue (#1868): "Exception occors when creating an experiment and trying to vary a String parameter. To reproduce this open a new SVM, e.g., create experiment, and click SvmType."

comment:11 Changed 5 years ago by abeham

r7974:

  • fixed problem with choices that are not INamedItem
  • fixed a bug in the enumeration of choice parameters
  • improved UI feedback

comment:12 Changed 5 years ago by gkronber

The dialog throws a cross-thread exception when the backgroundworker is finished.

comment:13 Changed 5 years ago by gkronber

Isn't there an equivalent warning icon in HeuristicLab.Common.Resources?

comment:14 Changed 5 years ago by gkronber

r7975: fixed cross-thread exception related to the backgroundworker for creating experiments.

comment:15 Changed 5 years ago by gkronber

At least for regression problems the problem instance is not set correctly in the created experiments.

comment:16 Changed 5 years ago by abeham

Okay, I'll look into it, but the catch and finally on a new line don't look good in my opinion. We don't have else on a new line either.

comment:17 Changed 5 years ago by gkronber

r7976: fixed a bug which had the effect that the problem instance was not set correctly for each variant

comment:18 Changed 5 years ago by gkronber

sorry for the formatting.

comment:19 Changed 5 years ago by abeham

thx for fixing this

comment:20 follow-up: Changed 5 years ago by abeham

Since you had a look at this already, I have an open issue and wanted to get your input: When you define a range for an integer parameter e.g. Minimum = 1, Maximum = 1000, Step = 100. Would you expect to get the sequence: 1, 101, 201, ..., 901 or would you expect to get (1,) 100, 200, ..., 1000.

Currently you obtain the first sequence because enumeration starts at the minimum and adds the step value while its <= the maximumm, but I think of changing it to the second. To get the second sequence would be to start at the maximum and subtract step (and finally add the minimum). Also would you always expect that it includes the maximum as well as the minimum value in the sequence?

Edit: Or is it better to forget about defining the sequence as an arithmetic progression and just type all the values directly, e.g. through a ";"-separated list. Since sometimes you could want a geometric progression of the values. Direct specification may be inconvenient for more than say 10 values, but I think that one could always create larger sequences in Excel quickly and paste that into the box. Alternatively we could use an IntArray or DoubleArray and the according view (which already supports pasting from Excel).

Last edited 5 years ago by abeham (previous) (diff)

comment:21 in reply to: ↑ 20 Changed 5 years ago by swagner

May I join this discussion and throw in my opinion?

  • probably we need both, sequences defined by a minimum, maximum and increment AND explicitly given values (although the second case is more important). For the first case I would recommend the following syntax min:max/step (e.g. 1:1000/100) and for the second case I think a semicolon-separated list is fine.
  • for 1:1000/100 I would expect the sequence 1, 100, 200, 300, ..., 1000; minimum and maximum value should be included.

comment:22 Changed 5 years ago by abeham

Hmmm, I'm not so much a fan of defining and parsing a syntax - it's something the user has to learn first and then remember later.

How about we remember and display only the actual sequence of values and allow the user to generate that sequence in the UI? So he can type in min, max, and step in respective text boxes then hit a button (or on textbox validation) and the values are generated and displayed in something similar to a StringConvertibleArrayView. This has the advantage that the values can also be modified after they're generated and it's also always clear what the actual values are going to be.

Last edited 5 years ago by abeham (previous) (diff)

comment:23 Changed 5 years ago by swagner

Yes, that is a good idea. By the way, I recommend to use a button and not the textbox validation events, so that it is clear for the user, when new values are created and maybe already existing values are overwritten in the StringConvertibleArrayView.

comment:24 Changed 5 years ago by abeham

r7985:

  • Allow to specify each value of numeric parameters explicitly as well as provide a dialog for generating the values
  • Changed SetMode() call back to originate from DoWork, since that it is the right place to make the call, but added a check for InvokeRequired in SetMode()
Last edited 5 years ago by abeham (previous) (diff)

comment:25 follow-up: Changed 5 years ago by swinkler

We have just found the following problem: When we create hundreds of SVMs by varying several algorithm parameters and several of these algorithm combinations lead to exceptions, then it can happen that the user is forced to react to hundreds of exceptions with the same type and exception message (e.g., "specified nu is infeasible").

comment:26 in reply to: ↑ 25 Changed 5 years ago by gkronber

Replying to swinkler:

We have just found the following problem: When we create hundreds of SVMs by varying several algorithm parameters and several of these algorithm combinations lead to exceptions, then it can happen that the user is forced to react to hundreds of exceptions with the same type and exception message (e.g., "specified nu is infeasible").

Thanks for reporting this issue. After discussion with mkommend we came to the conclusion that I'd probably best if we handle this in the experiment. By adding a flag to indicate if an experiment should stop on the first exception or aggregate all exceptions to show them at the end. I'm opening a new ticket for this (#1872)

comment:27 Changed 5 years ago by gkronber

Btw. "specified nu is infeasible" happens if you have unbalanced class distributions. You should limit the range of nu to a subrange of ]0..1] in this case.

comment:28 Changed 5 years ago by ascheibe

I just played around with the new Create Experiment dialog and found that the DefineArithmeticProgressionDialog popped up behind the Create Experiment Dialog. Fixed in r8011.

comment:29 Changed 5 years ago by abeham

r8012: fixed cancellation of experiment creation

comment:30 Changed 5 years ago by abeham

r8013: Enabled to set Value to null for OptionalConstrainedValueParameters

comment:31 Changed 5 years ago by abeham

  • Owner changed from abeham to gkronber
  • Status changed from assigned to reviewing

r8014:

  • made dialog a bit larger
  • added clearing of the run collection of the cloned optimizer
  • renamed variations tab

comment:32 Changed 5 years ago by gkronber

An exception occurs when checking/unchecking a problem instance provider that does not have problem instances (CSV Problem Provider)

comment:33 Changed 5 years ago by gkronber

  • The dialog sets the 'all' checked box already when all checkboxes except for the CSV Problem Provider are checked.
  • The dialog should not be displayed on top of all windows (click create experiment -> switch to another application (visual studio) -> dialog is still on top).

comment:34 Changed 5 years ago by abeham

Good point, I will exclude all providers that do not have a list of instances from being shown. I don't know an easy way to make them useful in this feature.

comment:35 Changed 5 years ago by gkronber

Good idea.

comment:36 Changed 5 years ago by abeham

r8095:

  • Changed to not display instance providers that do not contain data descriptors
  • Added some default values when checking an int/double parameter (made some rough guesses on what reasonable variations could be like based on the given value)
  • Set TopMost property to false

comment:37 Changed 5 years ago by gkronber

reviewed r8011:8013

comment:38 Changed 5 years ago by gkronber

Why do we need Application.DoEvents() in the event handlers for ProgressChanged (line 692 and 875)?

Last edited 5 years ago by gkronber (previous) (diff)

comment:39 Changed 5 years ago by gkronber

r8179: minor code improvements in CreateExperimentDialog

comment:40 Changed 5 years ago by abeham

At some point in the development I was firing a lot of events and they didn't got updated in the UI so I was thinking this would do the trick. But I then changed the number of events that are fired and forgot to remove it. Does it hurt?

comment:41 Changed 5 years ago by gkronber

Please remove it since its not necessary

comment:42 Changed 5 years ago by abeham

r8188: Removed calls to Application.DoEvents()

comment:43 Changed 5 years ago by gkronber

  • Status changed from reviewing to readytorelease

comment:44 Changed 5 years ago by mkommend

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.6 to 3.3.7
Note: See TracTickets for help on using tickets.