Opened 6 years ago

Closed 5 years ago

#1783 closed defect (done)

Batchruns are not executed correctly

Reported by: sforsten Owned by: swagner
Priority: medium Milestone: HeuristicLab 3.3.7
Component: Optimization Version: 3.3.7
Keywords: Cc:

Description (last modified by mkommend)

Batchruns which contain optimizers that produce multiple runs are not executed correctly, because the batchrun stops when the runs counter is larger than the repetitions counter. The described behavior can be reproduced by executing the attached file.

E.g., A batchrun should be repeated 5 times and contains an experiment with 3 standard algorithms. The batchrun stops after 2 repetitions, as the embedded experiment produces 3 runs with every execution.

Additionally the execution time is not correct after removing all runs from the run collection (should be 00:00:00).

Attachments (2)

Batchrun - Experiment test.hl (195.7 KB) - added by mkommend 6 years ago.
test file that demonstrates the described behavior
Batchrun.hl (44.4 KB) - added by abeham 5 years ago.
This batch run can be used to view the behavior described in comment 6.

Download all attachments as: .zip

Change History (15)

Changed 6 years ago by mkommend

test file that demonstrates the described behavior

comment:1 Changed 6 years ago by mkommend

  • Component changed from ### Undefined ### to Optimization
  • Description modified (diff)
  • Milestone changed from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.7
  • Summary changed from Experiment pauses always after several runs to Batchruns are not executed correctly

comment:2 Changed 6 years ago by mkommend

  • Owner changed from ascheibe to mkommend
  • Status changed from new to accepted

comment:3 Changed 6 years ago by mkommend

Upon further investigation it is obvious that this behavior has nothing to to with the number of runs present in the BatchRun.

comment:4 Changed 6 years ago by mkommend

r7576: Removed questionable source lines in execution state handling in the BatchRun and the Experiment.

comment:5 Changed 6 years ago by mkommend

  • Owner changed from mkommend to ascheibe
  • Status changed from accepted to reviewing

Please review and test carefully r7576. It would be best if the execution state handling works correctly right now, but I am not quite convinced.

In addition I am not sure of the purpose of the lines, which were removed with r7576.

comment:6 Changed 5 years ago by abeham

I just found a bug that I think is related to this: When you create a new batch run, add an optimizer and click on the run button of that optimizer instead of that on the batch run, the optimizer will run once and then terminates. This is correct and expected, but the batch run also stops which is incorrect - it should be in paused state.

Maybe we should disable showing the run controls for the algorithm altogether when it's embedded into a batch run. I mean it does look complicated when you have two play, pause, stop, and prepare buttons.

Changed 5 years ago by abeham

This batch run can be used to view the behavior described in comment 6.

comment:7 Changed 5 years ago by ascheibe

  • Owner changed from ascheibe to mkommend
  • Status changed from reviewing to assigned

I have talked to Andreas today and we found another bug. When an inner optimizer is executed once and then the batch run is prepared and afterwards the inner optimizer is executed again, it is executed repetitions-times. I have fixed this bug and the one mentioned in comment 6 and 7 in r8189. Maybe you can have a look at it again if everything still works as expected. I have also tested your fix for the original problem and it still works.

comment:8 Changed 5 years ago by mkommend

  • Status changed from assigned to accepted

comment:9 Changed 5 years ago by mkommend

r8190: Corrected tool tips in optimizer views and last errors in batchrun execution state handling.

comment:10 Changed 5 years ago by abeham

  • Owner changed from mkommend to swagner
  • Status changed from accepted to reviewing

r8194:

  • Replaced in BatchRun the four boolean action flags with an enum
  • Caught InvalidOperationException when changing the state of the nested optimizer, which usually occurs in a race-condition

comment:11 Changed 5 years ago by swagner

Thank you for working on this issue. I reviewed the changes and I do not have any additional comments. However, to be honest I am not really happy with this code, as it is still quite complex, error-prone and hard to understand. We all know that execution state handling in optimizers, batchruns and experiments is complicated and cumbersome. Therefore we should think about a fundamental redesign at the next opportunity. I also agree with abeham that we probably should remove the steering buttons of embedded optimizers.

comment:12 Changed 5 years ago by swagner

  • Status changed from reviewing to readytorelease

comment:13 Changed 5 years ago by mkommend

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.6 to 3.3.7
Note: See TracTickets for help on using tickets.