Optimizing External Applications

Sometimes it is not possible to directly write a new problem for HeuristicLab and integrate it through the plugin system. Some people already have an application of which at least a part represents a NP hard problem that they'd like to solve. This guide explains how to use the ExternalEvaluationProblem that is available in HeuristicLab 3.3 to optimize problems written in a language other than C# or written using different frameworks, to name just a few possibilities. First the basic concepts are explained in more detail, and then a short tutorial should be given as well in which such a connection is configured and finally optimized.

The most important part in any optimization problem is the evaluation function. Without knowing about the quality of a certain solution configuration the algorithm is not able to come close to an optimal solution. In NP hard problems evaluating a solution luckily is usually a rather simple task, whereas finding the best solution is extremely difficult. Of course there can be complex problems which require high computational effort to calculate the quality of a solution, but in many cases there is a deterministic algorithm that can do this. So, if the problem is not a HeuristicLab plugin we assume that it is available in another kind of executable format, either in an application itself or as part of another framework for example. We thus have a situation where we need inter-process communication (IPC). There are several possibilities of how to do IPC, in the following we will explain the approach we offer in HeuristicLab 3.3.

Architecture

Among the many possibilities and technologies that have emerged to provide a base for performing "distributed computing" in a wide sense, one of the first technologies was Remote Procedure Call (RPC). The idea is very simple: Instead of calling a local procedure to do some kind of calculation, a procedure is called that is not defined within the same executable or one of its dependencies. The client thus is in some ways hands the parameters of the method to another program or server, waits for the computation and then reads back the return value. In such a broad sense this is how most web applications nowadays work and indeed it is not until the invention of Representational State Transfer (REST) that there was a revolutionary change from the early RPC paradigm.

In HeuristicLab the using remote procedure calls seems to fit very well with what we are trying to achieve in "exporting" the problem definition. A client is embedded into HeuristicLab and a framework exists with which to write a service that can be called for evaluating a solution. Using this service foreign language applications and problems can effectively communicate with HeuristicLab and have their parameters optimized by HeuristicLab's powerful optimization library. Please note that while we're talking about RPC, the provided frameworks are not compatible with the RPC standard, but are rather simplified to ease application. Still they're inspired by this paradigm.

There are a number of media over which to communicate and so far there are two possibilities:

1. Start the external program as a process from HeuristicLab and communicate via the process' stdin and stdout. This requires that the external program can be executed under Windows.
2. Start the external program independently from HeuristicLab and communicate over a TCP/IP network.

In the tutorial part, it will be shown how to configure HeuristicLab to make use of these possibilities, in this section we will concentrate on the server part and explain how to prepare an external application to work as an "external evaluation service". For writing services, generally two types of services are supported by our framework: Push Service and Poll Service.

Push Service

Poll Service