Implement grid search for LibSVM parameters

[bburlacu]

• The grid search should find the optimal set of parameters by minimizing the k-fold cross validation error.

• A user-specified number of folds should be generated from the training data without performing unnecessary cloning

• The method should accept any combination of SVM parameters together with their respective ranges

[bburlacu]

r11308: SupportVectorMachineUtil.cs

[bburlacu]

r11309: Added CartesianProduct extension method to HeuristicLab.Common/3.3/EnumerableExtensions.cs, used to generate all possible combinations of parameters for the grid search.

[bburlacu]

r11326: Refactored CrossValidate and GridSearch methods.

[bburlacu]

r11337: Refactored SVM grid search, added support for symbolic classification.

[mkommend]

r11340: Fixed bug in svm classification regarding the parameter extraction.

[mkommend]

r11339: Minor code changes in SVMUtil to perform cross validation (code reorganization, naming).

[mkommend]

Review comments:

• use range transform to scale the problem before learning the SVM model
• remove RF parameters from the scripts
• include a demo problem data in the script, so that it can be started directly
• the scripts should also work with problems and not only with problem data objects
• provide some kind of progress indication (e.g. printed dots, number of calculated combinations, ...)
• elapsed time has no unit displayed
• provide sensible default parameter ranges in the script (e.g., AFAIK nu_svc does not use the c parameter)
• attach the scripts to the start page

[mkommend]

r11342: Corrected locking in SVM Util class.

[bburlacu]

r11361: Added the option to shuffle the crossvalidation folds (this option is on by default since libsvm does it too). Implemented stratified fold generation for classification data (ensures similar label distribution in each fold). Fixed bug causing incorrect behavior in GenerateFolds<T> when the number of values is less than the requested number of folds.

[bburlacu]

Sample script for SVM grid search (regression)

[bburlacu]

Sample script for SVM grid search (classification)

[bburlacu]

r11427: Fixed thread synchronisation bug.

[bburlacu]

r11447: Added svm grid search scripts to the HeuristicLab start page.

[gkronber]

data for input variables should be scaled also in the SvmUtil.CrossValidate method (using RangeTransform).

[gkronber]

Using a static lock object for the following is not necessary. Instead lock object with method scope and life-time is sufficient.

      Parallel.ForEach(crossProduct, new ParallelOptions { MaxDegreeOfParallelism = maxDegreeOfParallelism },
      parameterCombination => {
        var parameters = DefaultParameters();
        var parameterValues = parameterCombination.ToList();
        for (int i = 0; i < parameterValues.Count; ++i)
          setters[i](parameters, parameterValues[i]);

        double testMse = CalculateCrossValidationPartitions(partitions, parameters);
        lock (locker) {
          if (testMse < mse.Value) {
            mse.Value = testMse;
            bestParam = (svm_parameter)parameters.Clone();
          }
        }
      });
      return bestParam;

[bburlacu]

r11464: Moved lock object inside the GridSearch method. Added scaling for the svm partitions.

[bburlacu]

r11542: Fixed bug in CrossValidate method where the OnlineCalculatorError was ignored. Updated GridSearch method to return the crossvalidation mse as an out parameter and to skip nan values.

[bburlacu]

r11544: Updated sample scripts.

[bburlacu]

r11545: Updated test resources.

[bburlacu]

r11547: Updated svm grid search unit tests.

[bburlacu]

r11548: Updated svm grid search scripts with the versions created by the unit tests.

[jkarder]

r11907: merged r11308, r11309, r11326, r11337, r11340, r11339, r11342, r11361, r11427, r11447, r11464, r11542, r11544, r11545, r11547, r11548 into stable