Opened 4 years ago

Closed 3 years ago

#2082 closed feature request (done)

External evaluation problems for Scilab and Matlab

Reported by: mkommend Owned by: mkommend
Priority: high Milestone: HeuristicLab 3.3.10
Component: Problems.ExternalEvaluation Version: branch
Keywords: Cc:

Description

RealVector-based external evaluation problems should be provided by HL, where the evaluator simply calls a script of the chosen platform(either Scilab or Matlab) and retrieves the calculated results.

Change History (55)

comment:1 Changed 4 years ago by mkommend

  • Status changed from new to accepted
  • Version changed from 3.3.8 to branch

comment:2 Changed 4 years ago by mkommend

  • r9676: Created branch for new external evaluation problems.
  • r9677: Branched HeuristicLab.Optimization.

comment:3 Changed 4 years ago by mkommend

r9678: Adapted HL.Optimization to be built from the branch.

comment:4 Changed 4 years ago by mkommend

r9679: Branched HL.Encodings.RealVectorEncoding.

comment:5 follow-up: Changed 4 years ago by gkronber

How does this ticket relate to our existing external evaluation problem. Do you plan to create a new plugin or extend and integrate into the existing functionality?

comment:6 in reply to: ↑ 5 Changed 4 years ago by mkommend

Replying to gkronber:

How does this ticket relate to our existing external evaluation problem. Do you plan to create a new plugin or extend and integrate into the existing functionality?

There will be two additional plugins, one for Scilab and Matlab. The common base-classes will be implemented somewhere else (maybe in the encoding), because the existing external evaluation cannot be used as this would automatically add a dependency to protocol buffers.

comment:7 Changed 4 years ago by mkommend

r9682: Added new plugins for parameter optimization and external evaluation in Scilab.

comment:8 Changed 4 years ago by mkommend

r9683: Removed unused projects from the branch.

comment:9 Changed 4 years ago by mkommend

r9684: Removed not used references in HeuristicLab.Problems.ParameterOptimization.

comment:10 Changed 4 years ago by mkommend

r9687: Corrected plugin dependencies of parameter optimization plugins.

Last edited 4 years ago by mkommend (previous) (diff)

comment:11 Changed 4 years ago by mkommend

r9688: Corrected typos in parameter descriptions.

comment:12 Changed 4 years ago by mkommend

r9690: Added stub for matlab evaluation, updated plugin files to contain the license information and corrected project references.

comment:13 Changed 4 years ago by ascheibe

r9691 fixed assembly name of parameter optimization problems

comment:14 Changed 4 years ago by mkommend

r9698: Improved error handling and correct matlab license file.

comment:15 Changed 4 years ago by mkommend

r9715: Updated external evaluation scientific branch to use the new HL data types.

comment:16 Changed 4 years ago by mkommend

r9747: Changed path variable in the Scilab plugin Load method to avoid problems when MATLAB and Scilab are both installed.

comment:17 Changed 4 years ago by mkommend

r9748: Corrected spelling of MATLAB in all names and descriptions.

comment:18 Changed 4 years ago by mkommend

r10094: Improved error handling in Scilab evaluator and prepared the project files for MONO compilation.

comment:19 Changed 4 years ago by mkommend

r10122: Added Scilab .NET coupling to external evaluation branch.

comment:20 Changed 4 years ago by mkommend

r10123: Added dllmap config for linux compatibility.

comment:21 Changed 4 years ago by mkommend

r10124: Added config file.

comment:22 Changed 4 years ago by mkommend

r10125: Updated scilab mono connector with latest source and corrected marshalling of sci_err.

comment:23 Changed 4 years ago by mkommend

r10126: Forgot to add Util.cs.

comment:24 Changed 4 years ago by mkommend

r10135: Corrected marshalling and configuration of the Scilab .NET wrapper.

comment:25 Changed 4 years ago by mkommend

r10136: Corrected dllmap config for scilab.

comment:26 Changed 4 years ago by mkommend

r10137: Renamed dllmap config.

comment:27 Changed 4 years ago by mkommend

r10138: Adapted project file to include renamed dllmap config file.

comment:28 Changed 4 years ago by mkommend

r10140: Corrected build configuration of the external evaluation plugins for Scilab and MATLAB.

Last edited 4 years ago by mkommend (previous) (diff)

comment:29 Changed 4 years ago by mkommend

r10190: Added BestSolutionsAnalyzer for ParameterOptimizationProblems.

comment:30 Changed 4 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from accepted to reviewing

comment:31 Changed 4 years ago by mkommend

  • Milestone changed from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.10

comment:32 Changed 3 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from reviewing to assigned

Review comments:

  • I tested the functionality with MATLAB using a CMAES for the tuning of parameters for a Gaussian process using the GPML package. The integration works nicely and the performance seems to be good!
  • The major drawback, that HL algorithms does not support problem specific initialization operators, is very problematic here and caused some confusions
    • the initialization script has to be set in the evaluator while it should be a parameter of the problem.
    • MATLAB is started when the algorithm is prepared (reset button)
    • The script is executed only once together when MATLAB is started. I had a bug in the script leading to an exception in the evaluation script. After I fixed the bug in the initialization script the same exception was thrown because the initialization script was not re-run after I changed it (even when resetting the algorithm).
    • Would it be possible to just check if MATLAB is running whenever the evaluator is applied and initializing it when necessary? I think this would be better than initializing it in the InitializeState() method. The MATLAB instance should be disposed (not only cleared) at the end of a run (do we clear IStatefulItems at the end of the run?)

    Yes it would be possible. However, it takes quite some time for the MATLAB instance to be started and therefore I decided to just clear the variables.

  • I think it would be more convenient to have the full parameter vector available in MATLAB instead of each element individually (only set parameter name in problem instead of the name for each element).
  • At one point I had an error that forced me to close the problem view because each time I pressed prepare I got an exception dialog. Maybe it would be better to dispose the MATLAB instance on reset and instantiating a new one whenever the problem should be initialized.
  • I only tested the MATLAB problem. The same comments apply also to the SciLab problem.
  • Use Environment.Newline instead of "\r\n"
  • Update copyright year before merging into trunk
  • The implementation of the parameter optimization problem seems to duplicate parts of the single objective test function problem, NCA algorithm, and the Gaussian Process algorithm, it would probably be reasonable to derive those implementations from the more general parameter optimization problem.
Last edited 3 years ago by mkommend (previous) (diff)

comment:33 Changed 3 years ago by mkommend

r10594: Updated copyright and used Environment.NewLine instead of \r\n.

comment:34 Changed 3 years ago by mkommend

r10595: Implemented reviewer comments for the Scilab parameter optimization problem & evaluator.

comment:35 Changed 3 years ago by mkommend

r10600: Updated HL Scilab coupling (license, config, plugin, ...).

comment:36 Changed 3 years ago by mkommend

r10608: Removed already integrated external evaluation for Scilab (ticket #2171).

comment:37 Changed 3 years ago by mkommend

r11028 & r11029: Implemented reviewer comments for the MATLAB parameter optimization problem.

comment:38 Changed 3 years ago by mkommend

r11033: Further bug fixes in the MATLAB evaluator.

comment:39 Changed 3 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from assigned to reviewing

comment:40 Changed 3 years ago by gkronber

r11094: fixed a few strings while reviewing the source code of the MATLAB problem.

comment:41 Changed 3 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from reviewing to assigned

I tested the functionality using CMAES and everything seems fine.

I also tried to run two algorithms using the same problem at the same time. This also works fine as in the background a separate MATLAB instance is started for each problem.

I did not find any other problems in testing. Please integrate the MATLAB problem into the trunk.

comment:42 Changed 3 years ago by mkommend

  • Status changed from assigned to accepted

r11099: Integrated HeuristicLab.ExternalEvaluation.Matlab in the trunk.

comment:43 Changed 3 years ago by mkommend

  • Status changed from accepted to readytorelease

r11100: Removed obsolete branch HeuristicLab.ExternalEvaluation Scientific.

comment:44 Changed 3 years ago by mkommend

r11102: Forgot to set copyLocal to false for updated references in ExternalEvaluation.Matlab.

comment:45 Changed 3 years ago by mkommend

  • Status changed from readytorelease to assigned

The MATLAB COM component should be extracted in a separate ExtLib plugin.

comment:46 Changed 3 years ago by mkommend

  • Status changed from assigned to accepted

r11124: Added extlib for the COM matlab connector.

comment:47 Changed 3 years ago by mkommend

r11125: Adapted ExternalEvaluation.Matlab to use new external library.

comment:48 Changed 3 years ago by mkommend

r11126: Corrected plugin file for Problems.ExternalEvaluation.Matlab.

comment:49 Changed 3 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from accepted to reviewing

r11127: Corrected typo in MatlabConnector plugin and adapted plugin dependency unit test.

comment:50 Changed 3 years ago by ascheibe

r11128 updated new extlib project file to work with linux

comment:51 Changed 3 years ago by abeham

r11131: fixed output path of project and path to interop dll

comment:52 Changed 3 years ago by gkronber

r11153: fixed plugin name and a string in the assembly info.

comment:53 Changed 3 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from reviewing to readytorelease

Reviewed r11124:11128 and r11131.

comment:54 Changed 3 years ago by mkommend

r11155: Merged r11099, r11100, r11102, r11124, r11125, r11126, r11127, r11128, r11131, r11153 into the trunk.

comment:55 Changed 3 years ago by mkommend

  • Resolution set to done
  • Status changed from readytorelease to closed
Note: See TracTickets for help on using tickets.