Persistence of random forest solutions takes a long time and creates really big files

[gkronber]

ALGLIB also provides two proceedures for serialization of decision forests (dfserialize / dfunserialize) which produce similar output as our current serialization code (also same size).

The size of the double array that is persisted is O(nTrees * nSamples). It seems this is overly pessimistic and only a small fragment of this buffer is actually necessary.

[gkronber]

The space required for storing the trees of a random forest model is much larger than the space required for the original dataset. It is possible to recalculate the trees on demand when the random seed and the original data set is saved. Recalculation is only necessary for seldomly required functionality. Therefore, it should be possible to improve persistence and memory requirements.

[gkronber]

r10306: created a branch for the ticket.

[gkronber]

r10321: refactored RandomForestModel and changed persistence (store data and parameters instead of model)

[gkronber]

r10322: changed the RF model to store the original problem data directly and fixed bugs in backwards compatibility loading and saving.

Persistence has been improved. For example for the Friedman-II problem (10000 rows) a RF solution with 500 trees takes up 33MB zipped in HL 3.3.9 and only 1MB zipped after the changes in this ticket. Loading the old file takes about 5 seconds with no recalculation effort. Loading the new file is almost instantanous (<1s), however when e.g. the line chart is opened for the first time the recalculation takes around 7 sec.

[mkommend]

Extensively tested the implemented functionality and it works perfect (file size reduction from ~350 MB to ~ 1.5 MB). The evaluation time was also acceptable < 1s and the evaluation result stays constant. All in all a really good solution.

Source code review pending.

[mkommend]

Reviewed r10321:10322.

[gkronber]

r10963: merged r10321:10322 from feature branch to the trunk

[gkronber]

r10966: deleted feature branch

[gkronber]

r11006: merged improved random forest persistence from trunk to stable branch