External evaluation problems for Scilab and Matlab

[mkommend]

RealVector-based external evaluation problems should be provided by HL, where the evaluator simply calls a script of the chosen platform(either Scilab or Matlab) and retrieves the calculated results.

[mkommend]

• r9676: Created branch for new external evaluation problems.
• r9677: Branched HeuristicLab.Optimization.

[mkommend]

r9678: Adapted HL.Optimization to be built from the branch.

[mkommend]

r9679: Branched HL.Encodings.RealVectorEncoding.

[gkronber]

How does this ticket relate to our existing external evaluation problem. Do you plan to create a new plugin or extend and integrate into the existing functionality?

[mkommend]

Replying to gkronber:

How does this ticket relate to our existing external evaluation problem. Do you plan to create a new plugin or extend and integrate into the existing functionality?

There will be two additional plugins, one for Scilab and Matlab. The common base-classes will be implemented somewhere else (maybe in the encoding), because the existing external evaluation cannot be used as this would automatically add a dependency to protocol buffers.

[mkommend]

r9682: Added new plugins for parameter optimization and external evaluation in Scilab.

[mkommend]

r9683: Removed unused projects from the branch.

[mkommend]

r9684: Removed not used references in HeuristicLab.Problems.ParameterOptimization.

[mkommend]

r9687: Corrected plugin dependencies of parameter optimization plugins.

[mkommend]

r9688: Corrected typos in parameter descriptions.

[mkommend]

r9690: Added stub for matlab evaluation, updated plugin files to contain the license information and corrected project references.

[ascheibe]

r9691 fixed assembly name of parameter optimization problems

[mkommend]

r9698: Improved error handling and correct matlab license file.

[mkommend]

r9715: Updated external evaluation scientific branch to use the new HL data types.

[mkommend]

r9747: Changed path variable in the Scilab plugin Load method to avoid problems when MATLAB and Scilab are both installed.

[mkommend]

r9748: Corrected spelling of MATLAB in all names and descriptions.

[mkommend]

r10094: Improved error handling in Scilab evaluator and prepared the project files for MONO compilation.

[mkommend]

r10122: Added Scilab .NET coupling to external evaluation branch.

[mkommend]

r10123: Added dllmap config for linux compatibility.

[mkommend]

r10124: Added config file.

[mkommend]

r10125: Updated scilab mono connector with latest source and corrected marshalling of sci_err.

[mkommend]

r10126: Forgot to add Util.cs.

[mkommend]

r10135: Corrected marshalling and configuration of the Scilab .NET wrapper.

[mkommend]

r10136: Corrected dllmap config for scilab.

[mkommend]

r10137: Renamed dllmap config.

[mkommend]

r10138: Adapted project file to include renamed dllmap config file.

[mkommend]

r10140: Corrected build configuration of the external evaluation plugins for Scilab and MATLAB.

[mkommend]

r10190: Added BestSolutionsAnalyzer for ParameterOptimizationProblems.

[gkronber]

Review comments:

• I tested the functionality with MATLAB using a CMAES for the tuning of parameters for a Gaussian process using the GPML package. The integration works nicely and the performance seems to be good!
• The major drawback, that HL algorithms does not support problem specific initialization operators, is very problematic here and caused some confusions
  • the initialization script has to be set in the evaluator while it should be a parameter of the problem.
  • MATLAB is started when the algorithm is prepared (reset button)
  • The script is executed only once together when MATLAB is started. I had a bug in the script leading to an exception in the evaluation script. After I fixed the bug in the initialization script the same exception was thrown because the initialization script was not re-run after I changed it (even when resetting the algorithm).
  • Would it be possible to just check if MATLAB is running whenever the evaluator is applied and initializing it when necessary? I think this would be better than initializing it in the InitializeState() method. The MATLAB instance should be disposed (not only cleared) at the end of a run (do we clear IStatefulItems at the end of the run?)

  Yes it would be possible. However, it takes quite some time for the MATLAB instance to be started and therefore I decided to just clear the variables.

• I think it would be more convenient to have the full parameter vector available in MATLAB instead of each element individually (only set parameter name in problem instead of the name for each element).
• At one point I had an error that forced me to close the problem view because each time I pressed prepare I got an exception dialog. Maybe it would be better to dispose the MATLAB instance on reset and instantiating a new one whenever the problem should be initialized.
• I only tested the MATLAB problem. The same comments apply also to the SciLab problem.
• Use Environment.Newline instead of "\r\n"
• Update copyright year before merging into trunk
• The implementation of the parameter optimization problem seems to duplicate parts of the single objective test function problem, NCA algorithm, and the Gaussian Process algorithm, it would probably be reasonable to derive those implementations from the more general parameter optimization problem.

[mkommend]

r10594: Updated copyright and used Environment.NewLine instead of \r\n.

[mkommend]

r10595: Implemented reviewer comments for the Scilab parameter optimization problem & evaluator.

[mkommend]

r10600: Updated HL Scilab coupling (license, config, plugin, ...).

[mkommend]

r10608: Removed already integrated external evaluation for Scilab (ticket #2171).

[mkommend]

r11028 & r11029: Implemented reviewer comments for the MATLAB parameter optimization problem.

[mkommend]

r11033: Further bug fixes in the MATLAB evaluator.

[gkronber]

r11094: fixed a few strings while reviewing the source code of the MATLAB problem.

[gkronber]

I tested the functionality using CMAES and everything seems fine.

I also tried to run two algorithms using the same problem at the same time. This also works fine as in the background a separate MATLAB instance is started for each problem.

I did not find any other problems in testing. Please integrate the MATLAB problem into the trunk.

[mkommend]

r11099: Integrated HeuristicLab.ExternalEvaluation.Matlab in the trunk.

[mkommend]

r11100: Removed obsolete branch HeuristicLab.ExternalEvaluation Scientific.

[mkommend]

r11102: Forgot to set copyLocal to false for updated references in ExternalEvaluation.Matlab.

[mkommend]

The MATLAB COM component should be extracted in a separate ExtLib plugin.

[mkommend]

r11124: Added extlib for the COM matlab connector.

[mkommend]

r11125: Adapted ExternalEvaluation.Matlab to use new external library.

[mkommend]

r11126: Corrected plugin file for Problems.ExternalEvaluation.Matlab.

[mkommend]

r11127: Corrected typo in MatlabConnector plugin and adapted plugin dependency unit test.

[ascheibe]

r11128 updated new extlib project file to work with linux

[abeham]

r11131: fixed output path of project and path to interop dll

[gkronber]

r11153: fixed plugin name and a string in the assembly info.

[gkronber]

Reviewed r11124:11128 and r11131.

[mkommend]

r11155: Merged r11099, r11100, r11102, r11124, r11125, r11126, r11127, r11128, r11131, r11153 into the trunk.