Opened 4 years ago

Last modified 3 months ago

#2213 accepted feature request

Integrate irace for automatic algorithm configuration

Reported by: swagner Owned by: abeham
Priority: medium Milestone: HeuristicLab 3.3.16
Component: ### Undefined ### Version: branch
Keywords: Cc:

Description

irace (http://iridia.ulb.ac.be/irace/) is an R package which implements the iterated racing procedure for automatic configuration of algorithm parameters.

Change History (7)

comment:1 Changed 4 years ago by swagner

Created branch irace in r11210.

comment:2 Changed 4 years ago by gkronber

  • Milestone changed from HeuristicLab 3.3.11 to HeuristicLab 3.3.x Backlog

comment:3 Changed 3 years ago by mkommend

r13384: Deleted irace branch.

comment:4 Changed 8 months ago by abeham

  • Milestone changed from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.16
  • Owner changed from swagner to abeham
  • Status changed from new to accepted

comment:5 Changed 8 months ago by abeham

r15923: created branch

comment:6 Changed 8 months ago by abeham

Okay, here's what I learnt so far:

irace needs to be extended to work with the HeuristicLab.Hive backend. I forked the official CRAN repository on GitHub and extended the (already) generic interface. There are three interaction points

  1. target.runner - submits a job to Hive and outputs the JobId (GUID)
  2. cluster.status - repeatedly checks if a given JobId is finished
  3. target.evaluator - fetches a job and outputs the fitness for irace

The 2nd step was not yet supported generically, but hard-coded to several cluster systems.

For each step I created an executable

  1. HiveIraceSubmit.exe - takes a .hl file, opens it and sets the parameters according to the values given by irace (Parameter=value string format). It considers parameters of operators (parameter paths) using dot-notation (e.g. Selector.Pressure=1.3) and can handle IValueParameter and IConstrainedValueParameter<T>. It creates a Hive job and uploads the algorithm. The job's name is a unique identifier. The JobId is printed to Console.Out.
  2. HiveStatus.exe - takes a GUID and outputs either DONE or CALC to Console.Out.
  3. HiveIraceResult.exe - currently doesn't have the JobId, but looks for jobs with an identifying name. irace doesn't support propagating the JobId from the first step. It downloads the job and fetches the the result name (stated in the config) and prints it to Console.Out

This setting is generic and works. However, I don't really like having three executables. I'd rather use the HL plugin system and provide one plugin with three applications, but this is currently hard to achieve because it's not possible to get any Console output back to the executing shell. I created #2917 that describes the problem.

I also learnt that R's system2 command is buggy.

My conclusion right now is that submission to and downloading from Hive is rather slow. The whole process (launching the executable, discovering all plugins, loading the file, setting parameter, uploading to and downloading from Hive) takes quite some time.

comment:7 Changed 3 months ago by abeham

r16102: some pending changes exploring the topic

Note: See TracTickets for help on using tickets.