IExecutable Methods should be thread-safe

[pfleck]

Some implementations of Prepare/Start/Pause/Stop-Methods of the IExecutable-interface are not thread-safe. Especially since the introduction of StartAsync(in #2258) we should make sure that no race-conditions occur when calling those methods.

Known Issues

General

• Pause/Stop are asynchronous:
  For starting an IExecutable we offer synchronous and asynchronous methods. However, Pause and Stop are usually asynchronous too. Only Prepare is (currently) always synchronous. We should either make all methods blocking or offer additional async versions such as the Start(Async).
• ExecutionStates are not changed immediately after calling Prepare/Start/Pause/Stop:
  For instance, after calling StartAsync the ExecutionState is almost certainly not Started but still Prepared. (Same for Prepare/Pause/Stop). As discussed with mkommenda, our current ExecutionStates only mark "finished" states. Maybe we should also introduce "transition"-states such as "Starting" vs "Started" or "PausePending" vs "Paused".
• Potential NullReferenceExeption from CancellationTokenSources:
  Calling Stop on some IExecutables (BasicAlgorithm, Engine) when the Executable is "almost" finished: The Executable is not Stopped yet, however the CancellationTokenSource is already disposed or set to null.

BatchRun

• Calling Stop while the (inner) optimizer is executing its Prepare method:
  • BatchRun.Stop thinks the (inner) optimizer is Stopped and calls BatchRun.OnStopped (to indicate the BR is finished).
  • After the (inner) optimizer finishes its Prepare method the BatchRun itself becomes Prepared.
  • In this case the BatchRun starts an additional (inner) optimizer Run and then pauses and fires OnPaused (the BR thinks the optimizer was started externally).

Experiment

• An Experiment can continue executing some optimizers after it was stopped when one optimizer is a BatchRun that was paused instead of stopped (see above).

Additional Thoughts

• Throwing Exceptions when calling invalid states (e.g. calling Start on an already Running Optimizer)
  • Synchronous calls should throw an exception
  • Asynchronous calls should return the same Task
• Consistent usage/passing of CancellationTokens (also for Pause/Stop?)
• Usage of IProgress<T> (​link)