Plugin for symbolic time series prognosis with GP

[gkronber]

Worked on symbolic regression classes to prepare for time series prognosis plugin with r4022.

[gkronber]

removed commented code with r4023.

[gkronber]

Moved code for calculation of covariance from the scaled MSE evaluator into a separate online evaluator with r4027.

[mkommend]

Removed resource file for LaggedVariableView r4029.

[gkronber]

Added plugin for time series prognosis with r4113.

[gkronber]

set svn:ignore properties with r4114.

[gkronber]

r6802: added classes (problem, evaluators, analyzers, solution, model, online-calculators, and views) for time series prognosis problems and added an algorithm implementation to generation linear AR (auto-regressive) time series prognosis solution.

[gkronber]

r6807: added views for time series prognosis

[gkronber]

r6808: fixed build

[gkronber]

r6811:

• added configuration of default grammar for time-series prognosis
• improved multiple update of variable impacts result

[gkronber]

r6974:

• cleared up definition of accuracy metrics for time series prognosis to make the distinction between one n-step forecast and n one-step forecasts clearer
• implemented calculators for time series accuracy metrics to support the calculation of the average accuracy over m n-step forecasts,
• and adapted the unit tests accordingly.

[gkronber]

Open tasks:

• implement unit tests for n-step forecasts (only one-step forecasts are tested right now)
• write blog post / wiki page to document the accuracy metrics
• add a benchmark test set for time series prognosis with financial data

[gkronber]

r7058: added unit tests for directional symmetry and Theil's U and multi-step forecasts

[gkronber]

A larger overhaul of the time series implementation is necessary.

The most important aspect of time series analysis is prognosis of future values. As of now we only support one-step forecasts from y(t) to y(t+1). However, often it is necessary to create a forecast for the next n time-steps. For uni-variate time series, it would be possible to calculate the forecast outside of HeuristicLab, by simple recurrence (using the first one-step prognosis y(t+1) as additional input for the next one-step prognosis y(t+2) and so on). It would be nice to support this also directly in HeuristicLab and show for instance the line chart of the forecast for the whole test partition.

The second problem when n-step forecasts are supported arises when additional variables are used as inputs for the time-series model. Because the values of the input variables are usually not known for the forecast period. This issue can be solved by learning a multi-variate time-series model instead.

[gkronber]

Because of the considerations mentioned above I decided to pull the time-series plugins from the 3.3.6 release as this would lead to difficulties when we have to maintain compatibility in future releases.

The time-series prognosis functionality will be released with 3.3.7.

[gkronber]

r7097: reverse merged r6802, r6807:6808, r6811, r6974, r7058 to prepare for the 3.3.6 release

[gkronber]

r7098: created a branch for the development of time series prognosis features.

[gkronber]

r7099: merged old changesets r6802, r6807:6808, r6811, r6974, r7058 from the trunk into the TimeSeries branch to bring it to version r7096.

[gkronber]

r7100: worked on multi-variate time series prognosis

[gkronber]

r7120: implemented multi-variate symbolic expression tree interpreter for time series prognosis.

[gkronber]

r7129: worked on multi-variate time series prognosis

[gkronber]

r7154: worked on multi-variate time series prognosis

[gkronber]

r7160: worked on multi-variate time series prognosis

[gkronber]

r7183:

• implemented remaining metrics for time series prognosis solutions
• added estimation limits
• fixed training and validation best solution analyzers
• implemented overfitting analyzer.

[gkronber]

r7184: removed resx files

[gkronber]

r7194: small changes in calculation of time series solution results

[gkronber]

r7213 merged r7103:7209 from trunk into time series branch

[gkronber]

r7268: merged r7214:7266 from trunk into time series branch.

[gkronber]

r7460: merged r7266:7459 from the trunk into the time series prognosis branch.

[gkronber]

r7461: fixed a few problems related to a previous incorrect merge and removed unrelated projects from the solution.

[gkronber]

In the simplified for time series solutions the correct horizon should be used to calculate impacts.

[gkronber]

r7463: improved formatter, line chart view, simplifier for time series prognosis solutions

[gkronber]

r7615: merged r7462:7609 from trunk into time series branch

[gkronber]

r7842: merged r7609:7840 from trunk into time series branch

[gkronber]

r7843: fixed compile errors after merging changes from the trunk

[mkommend]

r7886: Adapted timeseries branch to contain only modified plugins.

[mkommend]

r7889: Branched problem instances to enable CSV import for timeseries problems.

[mkommend]

r7890: Implemented CSV problems instances provider for time series prognosis problems.

[mkommend]

r7926: Added interpreter performance unit test for type coherent grammar.

[mkommend]

r7930: Refactored symbolic expression tree interpreter in preparation for autoregressive single variate prognosis.

[mkommend]

r7989: Improved performance of time series prognosis.

[mkommend]

r7991: Updated estimation limits correctly in TimeSeriesPrognosisProblem.

[mkommend]

r7998: Corrected time series interpreter and updated visualizations.

[mkommend]

r8010: Corrected time series solution results and implemented new models.

[mkommend]

r8114: Corrected scaling behavior of symbolic time series models.

[mkommend]

r8116: Branched HL.DataAnalysis to add AR models.

[mkommend]

r8428: Temporary removed files in branch to prepare for trunk update.

[mkommend]

r8430: Intermediate commit of trunk updates - interpreter changes must be redone.

[mkommend]

r8431: Moved interpreter specific classes to separate directory.

[mkommend]

r8432: Corrected path of previous move operation.

[mkommend]

r8436: Merged changes from trunk into timeseries branch and refactored GP interpreter.

[mkommend]

r8458: Derived time series classes from regression classes to avoid code duplication.

[mkommend]

r8459: Code clean up in time series prognosis model and problem data.

[mkommend]

r8460: Added horizon parameters to timeseries problem data.

[mkommend]

r8468: Adapated calculation of results for horizons larger than 1.

[mkommend]

r8477:

• Added autoregressive target variable Symbol
• Merged trunk changes into the branch.

[mkommend]

r8483: Corrected evaluators and time series models.

[mkommend]

r8487: Added ErrorCharacteristics View for time series models.

[mkommend]

r8490: Updated TimeSeriesPrognosisSolutionView.

[mkommend]

r8742: Merged trunk changes and fixed compilation errors due to the merge.

[mkommend]

r8743: Readded ErrorCharacteristicsView for time series prognosis solutions.

[mkommend]

r8750: Extracted prognosis results into separate class and added a view for them.

[mkommend]

r8762: Fixed serialization and corrected AR(n) modeling.

[mkommend]

r8764: Removed wrong MA models from timeseries prognosis branch.

[mkommend]

r8765: Fixed minor bugs in time series prognosis views.

[mkommend]

r8766: Reintegrated trunk updates in timeseries branch.

[mkommend]

r8767: Merged remaining changes from the trunk.

[mkommend]

r8768: Corrected plugin dependencies.

[mkommend]

r8770: Corrected plugin.cs.frame for symbolic time series prognosis views.

[mkommend]

r8789: Merged trunk changes into timeseries branch.

[mkommend]

r8798: Reintegrated time series modeling branch into trunk.

[mkommend]

r8799: Corrected time series projects (copy local, output path).

[mkommend]

r8801: Changed project reference to an assembly reference in Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis and updated build dependencies in the solution file.

This change was necessary to allow compilation on the build server.

[mkommend]

Only r8798, r8799 and r8801 must be reviewed, because r8798 is the trunk integration and contains all previous changes.

[abeham]

r8851: Changed assembly reference to a project reference

It always gave me build errors when starting from a fresh checkout. I don't know why it didn't fail on the build server, but it should have.

[gkronber]

r8852: reverse merge of r8851. Project reference doesn't work on the builder because of a strange path-length bug. see r8801

[abeham]

Ok, the problem that I was seeing seems to be related to this ​bug in msbuild. The article says it should be fixed in .Net 4.5.

[mkommend]

r8940: Added EstimationLimits to ISymbolicTimeSeriesPrognosisModel.

[gkronber]

Again had to correct revision numbers in previous comments. Please be careful.

[gkronber]

review comments:

• fields in TimeSeriesPrognosisResults are not cloned.
• Update of EvaluatedSolutions is not thread safe in the interpreters.
• Cannot activate the symbol LaggedVariable in the GUI.
• the default grammar does only allow real-valued symbols in the root (not time series symbols)
• node impact calculation should be based on the currently selected evaluator. In particular the currently selected training horizon value should be used. Right now impacts are calculated in the same way as for regression models.
• It would be nice to also provide a line-chart showing the n-step forecast.

[gkronber]

r9004:

• locking for update of EvaluatedSolutions in interpreter
• cloning of fields in TimeSeriesPrognosisResults

[gkronber]

I think the training partition for time-series instances should automatically start at 1 or 10 otherwise this has to be changed manually each time (otherwise an exception occurs)

[gkronber]

r9452: created an override for the Load() method in SymbolicTimeSeriesPrognosisSingleObjectiveProblem to set the first index of the start partition to 10 (prevents ArgumentException)

[gkronber]

Please fix the grammar/symbol problems mentioned in comment:95 for the 3.3.8 release. The other issues should be tracked in new tickets.

[mkommend]

r9459: Updated TypeCoherentGrammar to take time-series, power, and condition symbols into account.

[mkommend]

review comments:

• Cannot activate the symbol LaggedVariable in the GUI.

All variable symbols can only be activated if at least one input variable is allowed.

• the default grammar does only allow real-valued symbols in the root (not time series symbols)

This was corrected with r9459. However, I have not updated the special functions.

[mkommend]

r9466: Fixed SGP symbolic classification tests, whose results got updated, due to a change in the grammar in r9459.

[gkronber]

Reviewed r9459 and r9466. Thanks!