Shuffle samples in the cross-validation wrapper for data analysis algorithms

[bburlacu]

The cross-validation wrapper should offer an option to shuffle the data samples.

[bburlacu]

r14864: Implement shuffling of crossvalidation samples.

[bburlacu]

r14865: Fix issue with resources in CrossValidationView.Designer.cs

[gkronber]

It seems that in the ensemble the information wether a point was used for training or test is not stored correctly. Reproduce:

1. Use cross-validation with shuffling and produce an overfit model on purpose.
2. Check line chart
3. Expected result: errors for training predictions (yellow) are very small, errors for test predictions (red) are significantly higher.
4. Actual result: some errors for training predictions are also high, some errors for test points are suspiciously small.

[bburlacu]

r14904: Reuse the shuffled data when creating the solution ensemble.

[gkronber]

Overlaps with changes in r14781 (#2756) must be merged together.

[mkommend]

Review comments:

• Backwards compatibility is not ensured
• Shuffling can be changed during execution yielding inconsistent results
• Clone shows wrong value of shuffle samples in view
• Shuffled problemData is neither cloned nor serialized

Why do we need the shuffledProblemData at all?

[bburlacu]

r15002: Got rid of the shuffledProblemData by using a shared seed for all the folds (so that the dataset for each fold is shuffled in exactly the same way). Backwards compatibility should be restored. Shuffling cannot be changed during algorithm execution, cloning also clones the checked value for the shuffled checkbox.

[mkommend]

This ticket broke the backwards compatibility for CrossValidation (probably due to the shuffle sample flag).

[bburlacu]

r15026: Ensure that the shuffleSamples flag is initialized after deserialization.

[mkommend]

r15077: Reordered backwards compatibility and event registration in after deserialization hook of CrossValidation.

[mkommend]

Review comments

CrossValidationView

• The ShuffleSamples checkbox should be checked / unchecked in OnContentChanged and not SetEnabledStateOfControls
• I would enable the ShuffleSamples checkbox only if the CrossValidation is prepared.

CrossValidation

• Why do extract and aggregate regression / classification solution work differently (clone of problemData)?

[bburlacu]

r15111: Set check state of the ShuffleSamples checkbox inthe OnContentChanged method. Enable the checkbox only when the CrossValidation is prepared.

Regarding the different way of cloning the classification solution: this is done differently to account for a special use case when the GBT algorithm with the logistic regression loss function returns a regression solution (from which a new classification solution is built).

[mkommend]

Reviewed r15111.

[gkronber]

r15150: merged r14864, r14865, r14904, r15002, r15026, r15077, r15111 from trunk to stable (all changesets merged)

[gkronber]

Depends on #2723.