Integrate irace for automatic algorithm configuration

[swagner]

irace (​http://iridia.ulb.ac.be/irace/) is an R package which implements the iterated racing procedure for automatic configuration of algorithm parameters.

[swagner]

Created branch irace in r11210.

[mkommend]

r13384: Deleted irace branch.

[abeham]

r15923: created branch

[abeham]

Okay, here's what I learnt so far:

irace needs to be extended to work with the HeuristicLab.Hive backend. I ​forked the official CRAN repository on GitHub and extended the (already) generic interface. There are three interaction points

1. target.runner - submits a job to Hive and outputs the JobId (GUID)
2. cluster.status - repeatedly checks if a given JobId is finished
3. target.evaluator - fetches a job and outputs the fitness for irace

The 2nd step was not yet supported generically, but hard-coded to several cluster systems.

For each step I created an executable

1. HiveIraceSubmit.exe - takes a .hl file, opens it and sets the parameters according to the values given by irace (Parameter=value string format). It considers parameters of operators (parameter paths) using dot-notation (e.g. Selector.Pressure=1.3) and can handle IValueParameter and IConstrainedValueParameter<T>. It creates a Hive job and uploads the algorithm. The job's name is a unique identifier. The JobId is printed to Console.Out.
2. HiveStatus.exe - takes a GUID and outputs either DONE or CALC to Console.Out.
3. HiveIraceResult.exe - currently doesn't have the JobId, but looks for jobs with an identifying name. irace doesn't support propagating the JobId from the first step. It downloads the job and fetches the the result name (stated in the config) and prints it to Console.Out

This setting is generic and works. However, I don't really like having three executables. I'd rather use the HL plugin system and provide one plugin with three applications, but this is currently hard to achieve because it's not possible to get any Console output back to the executing shell. I created #2917 that describes the problem.

I also learnt that R's system2 command is ​buggy.

My conclusion right now is that submission to and downloading from Hive is rather slow. The whole process (launching the executable, discovering all plugins, loading the file, setting parameter, uploading to and downloading from Hive) takes quite some time.

[abeham]

r16102: some pending changes exploring the topic