Opened 8 months ago

Closed 4 months ago

#2822 closed feature request (done)

Support Running Experiments in Parallel

Reported by: pfleck Owned by: jkarder
Priority: medium Milestone: HeuristicLab 3.3.15
Component: Optimization Version: trunk
Keywords: Cc:

Description (last modified by pfleck)

Before #2258, the Optimizers in an Experiment could be run in parallel by running the experiment and then starting some Optimizers manually. For each manually started Optimizer, a "Worker Thread" is spawned implicitly that (after the manually started Optimizer finished) assists executing the remaining Optimizers of the Experiment.

To support synchronous calls of Optimizer.Start (implemented in #2258), the behavior was changed and a "Worker Thread" does not run other Optimizers than the Optimizer it was spawned for.

To allow the user to run an experiment in parallel, we should introduce a more intuitive way of paralleling the experiment, similar as the "Workers" within a CrossValidation.

Change History (30)

comment:1 Changed 8 months ago by pfleck

  • Status changed from new to accepted

comment:2 Changed 8 months ago by pfleck

  • Description modified (diff)
  • Summary changed from Support Running Experiment in Parallel to Support Running Experiments in Parallel
  • Version changed from 3.3.14 to branch

comment:3 Changed 8 months ago by pfleck

r15328 Created branch and added plugins.

comment:4 Changed 8 months ago by pfleck

r15337 Added parallel execution support for Experiment (similar to CrossValidation).

  • Fixed race-condition in BasicAlgorithm (potential NullReference of CancellationTokenSource).
  • Fixed race-condition in BatchRun (calling .Stop potentially fires both OnStopped and OnPaused).

comment:5 Changed 7 months ago by pfleck

r15373:

  • Merged recent trunk-changes into branch.
  • Using task-continuations for semaphore-releasing to avoid manually started optimizers to mess with the semaphore.

comment:6 Changed 7 months ago by pfleck

r15375

  • Correctly triggers optimizer task exception within continuation.
  • Using lazy evaluation to re-include "externally started but then paused" optimizers.

comment:7 Changed 7 months ago by pfleck

  • Owner changed from pfleck to jkarder
  • Status changed from accepted to reviewing

comment:8 Changed 7 months ago by jkarder

I have tested the changes and noticed that when a user pauses an optimizer which has been started by the experiment, the experiment will stay in execution state Started after completion.

comment:9 Changed 7 months ago by jkarder

  • Owner changed from jkarder to pfleck
  • Status changed from reviewing to assigned

comment:10 Changed 7 months ago by pfleck

  • Status changed from assigned to accepted

comment:11 Changed 7 months ago by pfleck

  • Owner changed from pfleck to jkarder
  • Status changed from accepted to reviewing

r15389: When the user manually pauses a running optimizer, the experiment now correctly changes to Paused.

After this changes, the behavior of pausing matches the behavior of HL 3.3.14: As soon as an optimizer is paused (regardless of why and who) the experiment won't start any new optimizer (but still finishes the ones that are already running).

Last edited 7 months ago by pfleck (previous) (diff)

comment:12 Changed 7 months ago by jkarder

  • Owner changed from jkarder to pfleck
  • Status changed from reviewing to assigned

I noticed that batch run and optimizer are not synchronized as before, because required code has been commented out (presumably for testing purposes) in r15337. When the batch run completes, preparing the optimizer should also prepare the batch run. I reverted this behavioral change. All in all, the code looks good, thanks!

r15394: fixed synchronization of batch run and optimizer

comment:13 Changed 7 months ago by pfleck

r15407: Merged trunk changes to branch

comment:14 Changed 7 months ago by pfleck

r15408 Merged branch to trunk

comment:15 Changed 7 months ago by pfleck

r15409 removed C# 6 feature and fixed enabled-state.

comment:16 Changed 7 months ago by pfleck

  • Version changed from branch to trunk

comment:17 Changed 6 months ago by pfleck

  • Owner changed from pfleck to swagner
  • Status changed from assigned to reviewing

comment:18 follow-up: Changed 6 months ago by swagner

Are the ManualResetEventSlim allOptimizerFinished and the two flags experimentStarted and experimentStopped still necessary?

comment:19 in reply to: ↑ 18 Changed 6 months ago by pfleck

  • Owner changed from swagner to pfleck
  • Status changed from reviewing to assigned

Replying to swagner:

Are the ManualResetEventSlim allOptimizerFinished and the two flags experimentStarted and experimentStopped still necessary?

The ManualResetEventSlim allOptimizerFinished can indee be dropped because Task.WaitAll already blocks until all started optimizers are finished. The two flags are still necessary to distinguish between "experiment started"and "an optimizer was started manually", and to indicate a "stop pending".

comment:20 Changed 6 months ago by pfleck

  • Status changed from assigned to accepted

comment:21 Changed 6 months ago by pfleck

r15452

  • Removed ManualResetEventSlim allOptimizerFinished.
  • Reworked execution state transitions into simple, documented rules
  • Fixed potential NullReferenceException in ExperimentView

comment:22 Changed 5 months ago by pfleck

  • Owner changed from pfleck to jkarder
  • Status changed from accepted to reviewing

comment:23 Changed 4 months ago by pfleck

r15538: deleted branch

comment:24 Changed 4 months ago by jkarder

We (swagner, pfleck and I) had a look at the implemented rules. During my final tests I noticed a behavioral change in the following situation:

  1. Create the following experiment:
  • Experiment
    • Optimizer 1
    • Optimizer 2
    • Optimizer 3
  1. Start Optimizer 2 manually. The experiment will fire Started. (✓)
  2. Pause Optimizer 2. The experiment will fire Paused. (✓)
  3. Prepare the experiment. The experiment will fire Paused once and Prepared twice. (✕)

Paused should not be fired. This is caused by rule #3. When the experiment gets prepared, Optimizer 1 gets prepared and other paused optimizers (i.e. Optimizer 2) are found, resulting in OnPaused being called.

The old implementation did fire the two Prepared events (which is also questionable), but no Paused event.

comment:25 Changed 4 months ago by jkarder

  • Owner changed from jkarder to pfleck
  • Status changed from reviewing to assigned

comment:26 Changed 4 months ago by pfleck

  • Status changed from assigned to accepted

comment:27 Changed 4 months ago by pfleck

r15560:

  • Fixed bug that multiple of the same event were triggered.
  • Set experimentStarted to false if experiment is paused/prepared/stopped.

Note for the 2nd fix: Consider an experiment with 3 optimizer, start the experiment and start the 2nd optimizer manually before the first one finishesand let all 3 optimizer finish. As a result, the 2nd optimizer was never started by the experiment, thus is not present in the startedOptimizers-list. If you prepare the 2nd optimizer, the experiment does nothing because rule 1 apples (experimentRunning = true and we have a startable-optimizer (the one that was not started by the experiment, which is currently prepared because we did so above)).

After the fix, we set experimentStarted to false if the experiment is paused, prepared or stoped; therefore rule 1 does not apply anymore and the experiment is correctly set to prepared.

Last edited 4 months ago by pfleck (previous) (diff)

comment:28 Changed 4 months ago by pfleck

  • Owner changed from pfleck to jkarder
  • Status changed from accepted to reviewing

comment:29 Changed 4 months ago by jkarder

  • Status changed from reviewing to readytorelease

comment:30 Changed 4 months ago by jkarder

  • Resolution set to done
  • Status changed from readytorelease to closed

r15573: merged r15408:15409, r15452 and r15560 into stable

Note: See TracTickets for help on using tickets.