Generalized additive models (GAM)

[gkronber]

Generalized additive models would be a great addition to the set of data-based modeling algorithms.

Feature wishlist:

• Base-learner for the terms is configurable (default: smoothing spline or penalized regression spline). E.g. it would be great if we could use an efficient symbolic regression solver as base learner.
• Individually adjustable smoothing or regularization parameter for each term.
• Automatic selection of smoothing or regularization parameter for each term ideally based on generalized cross-validation (GCV).
• The variables allowed in each term are configureable.

Idea for a first prototype:

• Only uni-variate terms are allowed
• Use alglib penalized regression spline for each term
• The variables together with penalization parameters for each term are read from a list (algorithm parameter)

[gkronber]

r15774: created branch

[gkronber]

r15775: added simple implementation of GAM based on uni-variate penalized regression splines with the same penalization factor for each term

[gkronber]

r17812: copied implementation from branch to trunk.

[gkronber]

r17813: delete branch

[gkronber]

r17815: fix header

[mkommend]

r17839: Fixed base ctor call in Spline1dModel.

[mkommend]

Review comments

• What's the point of the ToArray call within GetEstimatedValues (Spline1dModel line 74)? (-> this was an artifact from older code, fixed in r17867)

[gkronber]

r17867: simplified code in Spline1dModel

[mkommend]

r17888: Corrected calculation of MSE and RMSE in GAMs by implementing helper methods for their calculation.

The previous implementation used the stddev or variance of the residuals. However, stddev(res) == RMSE and var(res) == MSE only holds iff mean(res) == 0.0. In practice this is not the case due to calculation differences for the training and almost never for the test data (e.g. data shifts), hence the RMSE and MSE have to be calculated the traditional way.

@gkronber I am unsure about the value in the RSS table (line 201). Previously it contained the var(res) and so I changed it to the MSE. However, the name RSS suggest residual sum of squares, thus it should contain MSE * n. Please, comment, correct, rename, on this. Maybe I miss something obvious.

[mkommend]

r17889: Minor changes in Spline1dModel (added field for inputVariable, and named model and solution more appropriately).

[gkronber]

Reviewed r17888:17889.

[gkronber]

r18014: merged r17812,r17815,r17839,r17867,r17888,r17889 from trunk to stable