How to use Hive-3.4 for optimization

This page gives you instructions how to use HeuristicLab.Hive-3.4 for optimization problems.

Hive gives you the possibility to execute independent algorithms in parallel. It distributes those algorithms as jobs to a network of slaves which execute them and post back the results. Hive performs best if there is a large number of jobs which have long executen-times. Also if limited memory is a issue on a local PC, Hive can be an advantage since it can decide to run the jobs only on machines with enough memory (like the blade server nodes).

Hive is currently at beta stage (18.04.2011) and can (and should) be tested by anyone who needs to execute a large number of algorithms. The following steps are required to run a Hive Experiment:

1. Download the most recent plugins for the Hive Experiment Manager Client (source:/branches/HeuristicLab.Hive-3.4/binaries/HeuristicLab.Hive.ExperimentManager-3.4.zip)
2. Copy plugins to HeuristicLab directory (usually "C:\Program Files\HeuristicLab 3.3\")
   • Notice that you also need to copy "HeuristicLab 3.3.exe.config" and overwrite the existing one. This is no problem as long as you haven't made any modifications in the config file. If you have your own version of the config file see Merging Configuration Files?.
3. Start HeuristicLab Optimizer
4. Go to Services > Set Username / Password and enter your credentials
   • You need a user and proper privileges as Hive User on the server (which is services.heuristiclab.com). These privileges can be requested from S. Wagner.
5. Go to Services > Hive > Experiment Manager (dev). Now you see a list of your Hive Experiments which are stored on the server.
6. Click the plus symbol on top of the list to create a new Hive Experiment
   
   
7. Now you need to specify which experiment should be executed on the Hive. You can either
   • ... create a new experiment by clicking "New Experiment"
   • ... open an existing experiment from a file.
8. After having created or loaded an experiment you can click "Show Experiment" to modify it.
9. Back at the Hive Experiment Manager you can go to Jobs where you can configure every single job and observe its status. Each algorithm (also experiments and batch-runs) represent one job.
   
   
10. Important configurations are:
    • CoresNeeded: (default: 1) Specifies how many cores will be reserved for this job on the executing machine. If you know the algorithm corresponding to this job will use multiple threads for computation, increase this number!
    • MemoryNeeded: (default: 0) Specifies how much memory will be reserved for this job on the executing machine. It will only deployed on machines where the specified amount of memory is available.
    • Priority: (default: 0) This number affects the scheduling of the jobs. Higher numbers will be executed earlier. It is recommended to rank jobs with long execution-times to be executed earlier to avoid waiting for them in the end.
    • ComputeInParallel: This flag is only available for experiments and batch-runs.
      • Experiment: (default: true) If true, a job will be created for each child-optimizer. If false the whole experiment will be executed as one job.
      • BatchRun: (default: false) If true, a number of jobs will be created corresponding to the number of repetitions specified. If false the batch-run will be executed as one job.
11. After configuring the jobs you can hit "Start Experiment" and the jobs will be uploaded to the Hive. Hive will then take care of the distribution and execution of each job.
12. While you keep the Hive Experiment Manager open, it will periodically fetch the status-updates of all jobs. When jobs are finished it will download the results automatically.
    
    
13. You can also close HeuristicLab after uploading an Hive Experiment. When you open the Hive Experiment Manager again, it will download you list of Hive Experiments and you can choose to download a specific one.
14. Notice that when results are downloaded they will be reassembled into the original experiment. So after it finished you can again open the original experiment ("Open Experiment") and see all the results in the RunCollection as if it would have been executed locally.
    • Unfortunately it is not possible to preserve the ResultsCollection of each algorithm when reassembling the experiment. However each ResultsCollection is stored in the RunsCollection anyway.

Plugins

Hive automatically uses your local plugin versions. If they are not yet available on the server (because you are the first one to use this version), they are uploaded. When the jobs are executed, exactly those plugins are used.

If you have local modifications of a plugin, which you have not checked in yet, the version of the plugin is not incremented. In this case you need to select useLocalPlugins before uploading an experiment. This ensures, that all of your local plugins will be uploaded and used at execution, no matter existing versions.

Merging configuration files

If you are building the trunk or your own HL-solution, the build-process will most likely overwrite "HeuristicLab 3.3.exe.config" in the target folder. This will overwrite all the Hive-specific configuration. To avoid this, you can use the ConfigMerger to merge the Hive-specific configurations in your own build-process:

1. Download config for HiveExperimentManager (source:/branches/HeuristicLab.Hive-3.4/sources/HeuristicLab.Clients.Hive/3.4/ExperimentManagerOnly.config?format=txt)
2. Copy the file to your own solution and rename it to "HeuristicLab.Hive.config"
3. Open properties of the file in Visual Studio and select "Copy if newer" under "Copy to Output Directory".
4. Add the following line to your "MergeConfigs.cmd" (if you have one, otherwise specify directly in post-build events):
   • ConfigMerger "%SolutionDir%\HeuristicLab\3.3\%Outdir%\HeuristicLab.Hive.config" "HeuristicLab 3.3.exe.config"