Support Running Experiments in Parallel

[pfleck]

Before #2258, the Optimizers in an Experiment could be run in parallel by running the experiment and then starting some Optimizers manually. For each manually started Optimizer, a "Worker Thread" is spawned implicitly that (after the manually started Optimizer finished) assists executing the remaining Optimizers of the Experiment.

To support synchronous calls of Optimizer.Start (implemented in #2258), the behavior was changed and a "Worker Thread" does not run other Optimizers than the Optimizer it was spawned for.

To allow the user to run an experiment in parallel, we should introduce a more intuitive way of paralleling the experiment, similar as the "Workers" within a CrossValidation.

[pfleck]

r15328 Created branch and added plugins.

[pfleck]

r15337 Added parallel execution support for Experiment (similar to CrossValidation).

• Fixed race-condition in BasicAlgorithm (potential NullReference of CancellationTokenSource).
• Fixed race-condition in BatchRun (calling .Stop potentially fires both OnStopped and OnPaused).

[pfleck]

r15373:

• Merged recent trunk-changes into branch.
• Using task-continuations for semaphore-releasing to avoid manually started optimizers to mess with the semaphore.

[pfleck]

r15375

• Correctly triggers optimizer task exception within continuation.
• Using lazy evaluation to re-include "externally started but then paused" optimizers.

[jkarder]

I have tested the changes and noticed that when a user pauses an optimizer which has been started by the experiment, the experiment will stay in execution state Started after completion.

[pfleck]

r15389: When the user manually pauses a running optimizer, the experiment now correctly changes to Paused.

After this changes, the behavior of pausing matches the behavior of HL 3.3.14: As soon as an optimizer is paused (regardless of why and who) the experiment won't start any new optimizer (but still finishes the ones that are already running).

[jkarder]

I noticed that batch run and optimizer are not synchronized as before, because required code has been commented out (presumably for testing purposes) in r15337. When the batch run completes, preparing the optimizer should also prepare the batch run. I reverted this behavioral change. All in all, the code looks good, thanks!

r15394: fixed synchronization of batch run and optimizer

[pfleck]

r15407: Merged trunk changes to branch

[pfleck]

r15408 Merged branch to trunk

[pfleck]

r15409 removed C# 6 feature and fixed enabled-state.

[swagner]

Are the ManualResetEventSlim allOptimizerFinished and the two flags experimentStarted and experimentStopped still necessary?

[pfleck]

Replying to swagner:

Are the ManualResetEventSlim allOptimizerFinished and the two flags experimentStarted and experimentStopped still necessary?

The ManualResetEventSlim allOptimizerFinished can indee be dropped because Task.WaitAll already blocks until all started optimizers are finished. The two flags are still necessary to distinguish between "experiment started"and "an optimizer was started manually", and to indicate a "stop pending".

[pfleck]

r15452

• Removed ManualResetEventSlim allOptimizerFinished.
• Reworked execution state transitions into simple, documented rules
• Fixed potential NullReferenceException in ExperimentView

[pfleck]

r15538: deleted branch

[jkarder]

We (swagner, pfleck and I) had a look at the implemented rules. During my final tests I noticed a behavioral change in the following situation:

1. Create the following experiment:

• Experiment
  • Optimizer 1
  • Optimizer 2
  • Optimizer 3

2. Start Optimizer 2 manually. The experiment will fire Started. (✓)
3. Pause Optimizer 2. The experiment will fire Paused. (✓)
4. Prepare the experiment. The experiment will fire Paused once and Prepared twice. (✕)

Paused should not be fired. This is caused by rule #3. When the experiment gets prepared, Optimizer 1 gets prepared and other paused optimizers (i.e. Optimizer 2) are found, resulting in OnPaused being called.

The old implementation did fire the two Prepared events (which is also questionable), but no Paused event.

[pfleck]

r15560:

• Fixed bug that multiple of the same event was triggreed.
• Set experimentStarted to false if experiment is paused/prepared/stopped.

Note for the 2nd fix: Consider an experiment with 3 optimizer, start the experiment and start the 2nd optimizer manually before the first one finishesand let all 3 optimizer finish. As a result, the 2nd optimizer was never started by the experiment, thus is not present in the startedOptimizers-list. If you prepare the 2nd optimizer, the experiment does nothing because rule 1 apples (experimentRunning = true and we have a startable-optimizer (the one that was not started by the experiment, which is currently prepared because we did so above)).

After the fix, we set experimentStarted to false if the experiment is paused, prepared or stoped; therefore rule 1 does not apply anymore and the experiment is correctly set to prepared.

[jkarder]

r15573: merged r15408:15409, r15452 and r15560 into stable