Changes between Version 11 and Version 12 of Documentation/Howto/UseHiveForOptimization


Ignore:
Timestamp:
10/12/11 13:06:03 (9 years ago)
Author:
ascheibe
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Documentation/Howto/UseHiveForOptimization

    v11 v12  
    44Hive gives you the possibility to execute independent algorithms in parallel. It distributes those algorithms as jobs to a network of slaves which execute them and post back the results. Hive performs best if there is a large number of jobs which have long executen-times. Also if limited memory is a issue on a local PC, Hive can be an advantage since it can decide to run the jobs only on machines with enough memory (like the blade server nodes).
    55
    6 The following steps are required to run a Hive Experiment:
     6The following steps are required to run a Hive Job with HeuristicLab 3.3.5:
    77
    8  1. Download the most recent plugins for the Hive Experiment Manager Client (source:/branches/HeuristicLab.Hive-3.4/binaries/HeuristicLab.Hive.ExperimentManager-3.4.zip)
     8 1. Download the plugins for the Hive Job Manager Client (source:/branches/HeuristicLab.Hive-3.4/binaries/Plugins_Hive_HL_3.3.5.zip)
    99 1. Copy plugins to HeuristicLab directory (usually "C:\Program Files\HeuristicLab 3.3\")
    1010  * Notice that you also need to copy ''"HeuristicLab 3.3.exe.config"'' and overwrite the existing one. This is no problem as long as you haven't made any modifications in the config file. If you have your own version of the config file see [wiki:HiveHowTosUseHiveForOptimization#Mergingconfigurationfiles Merging Configuration Files].
     
    1212 1. Go to ''Services > Set Username / Password'' and enter your credentials
    1313  * You need a user and proper privileges as ''Hive User'' on the server (which is services.heuristiclab.com). These privileges can be requested from S. Wagner.
    14  1. Go to ''Services > Hive > Experiment Manager''. Now the list of your Hive Experiments is updated. The experiments are grouped into the owners of the experiments.
    15  1. Click the plus symbol on top of the list to create a new ''Hive Experiment'' \\ [[Image(HiveExperiment.png, width=700)]] \\
    16  1. Now you need to specify which experiment should be executed on the Hive. You can either
    17   * ... create a new experiment by clicking the ''"plus symbol"''.
     14 1. Go to ''Services > Hive > Job Manager''. Now the list of your Hive Jobs is updated. The jobs are grouped into the owners of the experiments.
     15 1. Click the plus symbol on top of the list to create a new ''Hive Job'' \\ [[Image(HiveExperiment.png, width=700)]] \\
     16 1. Now you need to specify which job should be executed on the Hive. You can either
     17  * ... create a new job by clicking the ''"plus symbol"''.
    1818  * ... move an existing experiment onto the view via drag and drop.
    1919 1. After having created or loaded an experiment you can click ''"Modify Optimizer"'' in the details view to modify it.
     
    2323  * !MemoryNeeded: (default: 0) Specifies how much memory will be reserved for this job on the executing machine. It will only deployed on machines where the specified amount of memory is available.
    2424  * Priority: (default: 0) This number affects the scheduling of the jobs. Higher numbers will be executed earlier. It is recommended to rank jobs with long execution-times to be executed earlier to avoid waiting for them in the end.
    25   * !ComputeInParallel: This flag is only available for experiments and batch-runs.
     25  * !ComputeInParallel: This flag is only available for jobs and batch-runs.
    2626   * Experiment: (default: true) If true, a job will be created for each child-optimizer. If false the whole experiment will be executed as one job.
    2727   * !BatchRun: (default: false) If true, a number of jobs will be created corresponding to the number of repetitions specified. If false the batch-run will be executed as one job.
    28  1. There are also some configurations available for the whole experiment:
     28 1. There are also some configurations available for the whole job:
    2929   * !ResourceIds: You can specify on which machines your jobs should be executed. Those machines/resources are grouped into resource-groups. The top group is ''"HEAL"''. You can either enter the name of a group or the name of a specific machine.
    3030   * !IsPrivileged: Jobs on Hive are executed in a secure sandboxed appdomain. However some plugins might require elevated privileges. If !IsPrivileged is checked, the jobs will be executed in a unrestricted appdomain. This option is only enabled if the user is allowed to (there is a role called ''"Hive !IsAllowedPrivileged"''). If you need this permission, please contact S. Wagner.
    31  1. After configuring the experiment you can hit ''"Start Experiment"'' and the jobs will be uploaded to the Hive. Hive will then take care of the distribution and execution of each job.
    32  1. While you keep the Hive Experiment Manager open, it will periodically fetch the status-updates of all jobs. When jobs are finished it will download the results automatically.
    33  1. You can also close HeuristicLab after uploading an ''Hive Experiment''. When you open the ''Hive Experiment Manager'' again, it will download you list of ''Hive Experiments'' and you can choose to download a specific one.
    34  1. Notice that when results are downloaded they will be reassembled into the original experiment. So after it finished you can again open the original experiment (''"Open Experiment"'') and see all the results in the !RunCollection as if it would have been executed locally.
     31 1. After configuring the job you can hit ''"Start Job"'' and the tasks will be uploaded to the Hive. Hive will then take care of the distribution and execution of each task.
     32 1. While you keep the Hive Job Manager open, it will periodically fetch the status-updates of all jobs. When jobs are finished it will download the results automatically.
     33 1. You can also close HeuristicLab after uploading an ''Hive Job''. When you open the ''Hive Job Manager'' again, it will download you list of ''Hive Jobs'' and you can choose to download a specific one.
     34 1. Notice that when results are downloaded they will be reassembled into the original job. So after it finished you can again open the original experiment (''"Open Experiment"'') and see all the results in the !RunCollection as if it would have been executed locally.
    3535  * Unfortunately it is not possible to preserve the ''!ResultsCollection'' of each algorithm when reassembling the experiment. However each ''!ResultsCollection'' is stored in the ''!RunsCollection'' anyway.
    3636
     
    4545You can observe the states of all jobs in the ''"States"'' tab. It shows a gantt-chart of your jobs. This is very helpful to get a quick overview of the runtimes of all jobs: \\ [[Image(JobStates.png, width=700)]]
    4646
    47 == Experiment Sharing ==
    48 Experiments can be shared with Full or Read permissions. To do so, go to the ''Sharing'' tab and hit refresh. You will see the list of already granted permissions (empty at first -- the owner always has full permissions). Click add and enter the username of who you want to share with as well as the desired permission. After saving the permission, the other user will see that experiment in his experiment-list. Permissions can only be added to experiments which are already uploaded. Permission include the following actions:
     47== Job Sharing ==
     48Jobs can be shared with Full or Read permissions. To do so, go to the ''Sharing'' tab and hit refresh. You will see the list of already granted permissions (empty at first -- the owner always has full permissions). Click add and enter the username of who you want to share with as well as the desired permission. After saving the permission, the other user will see that job in his job-list. Permissions can only be added to jobs which are already uploaded. Permission include the following actions:
    4949
    50  * Read: Download experiment, view jobs
    51  * Full: Control experiment and jobs, delete experiment, grant permissions to other users
     50 * Read: Download job, view tasks
     51 * Full: Control job and tasks, delete job, grant permissions to other users
    5252
    5353\\ [[Image(Permissions.png, width=500)]]
    5454
    5555== Statistics ==
    56 To observe the current status and the workload there is a website which shows this information: http://services.heuristiclab.com/Hive-3.4/Status.aspx
     56To observe the current status and the workload there is a website which shows this information: http://services.heuristiclab.com/Hive-3.3/Status.aspx
    5757
    5858== Merging configuration files ==
    5959If you are building the trunk or your own HL-solution, the build-process will most likely overwrite ''"HeuristicLab 3.3.exe.config"'' in the target folder. This will overwrite all the Hive-specific configuration. To avoid this, you can use the !ConfigMerger to merge the Hive-specific configurations in your own build-process:
    6060
    61  1. Download config for !HiveExperimentManager (source:/branches/HeuristicLab.Hive-3.4/sources/HeuristicLab.Clients.Hive/3.4/ExperimentManagerOnly.config?format=txt)
     61 1. Download config for !HiveJobManager (source:/branches/HeuristicLab.Hive-3.4/sources/HeuristicLab.Clients.Hive/3.4/ExperimentManagerOnly.config?format=txt)
    6262 1. Copy the file to your own solution and rename it to ''"HeuristicLab.Hive.config"''
    6363 1. Open properties of the file in Visual Studio and select ''"Copy if newer"'' under ''"Copy to Output Directory"''.