Remove bugs in LM-BFGS

[mkommend]

The following bugs were encountered in the LM-BFGS algorithm:

• Bounds are ignored

  gkronber: BFGS doesn't support bounds. Use an algorithm that supports constraints instead (such as BLEIC in alglib). Also see #2217 for a similar issue in relation with CMAES.

• Maximization flag is ignored

  fixed with r12794

• Does not work with the programmable problem, because the operators are not wired to the new encoding
• Best solution is not saved in the global scope and thus cannot be received by the best scope solution analyzer

  gkronber: I checked this. The point stored in the global scope is already the best solution. This might have been an indirect result of the problem with the maximization flag?

See attached HL file that demonstrates all those bugs.

[mkommend]

Faulty LM-BFGS configured with a programmable problem

[gkronber]

r12794: handle maximization problems correctly in BFGS

[gkronber]

r12795: added wiring code to make BFGS work with programmable problem

[gkronber]

r12797 fixed problem in GaussianProcess algorithm after changes in BFGS

[abeham]

By our high standards of scientific rigor, I would encourage you to increase minor version number if the results are not reproducible anymore

[gkronber]

Replying to abeham:

By our high standards of scientific rigor, I would encourage you to increase minor version number if the results are not reproducible anymore

Results of old experiments are reproducible even after the changes in this ticket. The comment relates to ticket #2435

[mkommend]

Reviewed r12794, r12795, and r12797.

[gkronber]

r13053: merged r12794:12795 and r12797 from trunk to stable