Opened 7 years ago

Closed 4 years ago

#1081 closed feature request (done)

Plugin for symbolic time series prognosis with GP

Reported by: gkronber Owned by: gkronber
Priority: medium Milestone: HeuristicLab 3.3.8
Component: Problems.DataAnalysis Version: 3.3.8
Keywords: Cc:

Description


Change History (105)

comment:1 Changed 7 years ago by gkronber

  • Status changed from new to assigned

comment:2 Changed 7 years ago by gkronber

Worked on symbolic regression classes to prepare for time series prognosis plugin with r4022.

comment:3 Changed 7 years ago by gkronber

removed commented code with r4023.

comment:4 Changed 7 years ago by gkronber

Moved code for calculation of covariance from the scaled MSE evaluator into a separate online evaluator with r4027.

comment:5 Changed 7 years ago by mkommend

Removed resource file for LaggedVariableView r4029.

comment:6 Changed 7 years ago by gkronber

  • Version changed from 3.3 to 3.3.1

comment:7 Changed 7 years ago by gkronber

Added plugin for time series prognosis with r4113.

comment:8 Changed 7 years ago by gkronber

set svn:ignore properties with r4114.

comment:9 Changed 7 years ago by gkronber

  • Version changed from 3.3.1 to branch

comment:10 Changed 7 years ago by gkronber

  • Milestone changed from HeuristicLab x.x.x to HeuristicLab 3.3.3

comment:11 Changed 6 years ago by gkronber

  • Milestone changed from HeuristicLab 3.3.3 to HeuristicLab x.x.x

comment:12 Changed 6 years ago by gkronber

  • Component changed from ZZZ OBSOLETE: Problems.DataAnalysis.MultiVariate.TimeSeriesPrognosis to Problems.DataAnalysis
  • Milestone changed from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.6
  • Status changed from assigned to accepted
  • Version changed from branch to 3.3.5

comment:13 Changed 6 years ago by gkronber

r6802: added classes (problem, evaluators, analyzers, solution, model, online-calculators, and views) for time series prognosis problems and added an algorithm implementation to generation linear AR (auto-regressive) time series prognosis solution.

comment:14 Changed 6 years ago by gkronber

r6807: added views for time series prognosis

comment:15 Changed 6 years ago by gkronber

r6808: fixed build

comment:16 Changed 6 years ago by gkronber

r6811:

  • added configuration of default grammar for time-series prognosis
  • improved multiple update of variable impacts result

comment:17 Changed 6 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from accepted to reviewing

comment:18 Changed 6 years ago by gkronber

  • Owner changed from mkommend to gkronber
  • Status changed from reviewing to assigned

comment:19 Changed 6 years ago by gkronber

  • Status changed from assigned to accepted

comment:20 Changed 6 years ago by gkronber

r6974:

  • cleared up definition of accuracy metrics for time series prognosis to make the distinction between one n-step forecast and n one-step forecasts clearer
  • implemented calculators for time series accuracy metrics to support the calculation of the average accuracy over m n-step forecasts,
  • and adapted the unit tests accordingly.

comment:21 Changed 6 years ago by gkronber

Open tasks:

  • implement unit tests for n-step forecasts (only one-step forecasts are tested right now)
  • write blog post / wiki page to document the accuracy metrics
  • add a benchmark test set for time series prognosis with financial data

comment:22 Changed 6 years ago by gkronber

r7058: added unit tests for directional symmetry and Theil's U and multi-step forecasts

comment:23 Changed 6 years ago by gkronber

A larger overhaul of the time series implementation is necessary.

The most important aspect of time series analysis is prognosis of future values. As of now we only support one-step forecasts from y(t) to y(t+1). However, often it is necessary to create a forecast for the next n time-steps. For uni-variate time series, it would be possible to calculate the forecast outside of HeuristicLab, by simple recurrence (using the first one-step prognosis y(t+1) as additional input for the next one-step prognosis y(t+2) and so on). It would be nice to support this also directly in HeuristicLab and show for instance the line chart of the forecast for the whole test partition.

The second problem when n-step forecasts are supported arises when additional variables are used as inputs for the time-series model. Because the values of the input variables are usually not known for the forecast period. This issue can be solved by learning a multi-variate time-series model instead.

comment:24 Changed 6 years ago by gkronber

Because of the considerations mentioned above I decided to pull the time-series plugins from the 3.3.6 release as this would lead to difficulties when we have to maintain compatibility in future releases.

The time-series prognosis functionality will be released with 3.3.7.

comment:25 Changed 6 years ago by gkronber

r7097: reverse merged r6802, r6807:6808, r6811, r6974, r7058 to prepare for the 3.3.6 release

comment:26 Changed 6 years ago by gkronber

r7098: created a branch for the development of time series prognosis features.

comment:27 Changed 6 years ago by gkronber

  • Milestone changed from HeuristicLab 3.3.6 to HeuristicLab 3.3.7

comment:28 Changed 6 years ago by gkronber

r7099: merged old changesets r6802, r6807:6808, r6811, r6974, r7058 from the trunk into the TimeSeries branch to bring it to version r7096.

comment:29 Changed 6 years ago by gkronber

r7100: worked on multi-variate time series prognosis

comment:30 Changed 6 years ago by gkronber

r7120: implemented multi-variate symbolic expression tree interpreter for time series prognosis.

comment:31 Changed 6 years ago by gkronber

r7129: worked on multi-variate time series prognosis

comment:32 Changed 6 years ago by gkronber

r7154: worked on multi-variate time series prognosis

comment:33 Changed 6 years ago by gkronber

r7160: worked on multi-variate time series prognosis

comment:34 Changed 6 years ago by gkronber

r7183:

  • implemented remaining metrics for time series prognosis solutions
  • added estimation limits
  • fixed training and validation best solution analyzers
  • implemented overfitting analyzer.

comment:35 Changed 6 years ago by gkronber

r7184: removed resx files

comment:36 Changed 6 years ago by gkronber

r7194: small changes in calculation of time series solution results

comment:37 Changed 6 years ago by gkronber

r7213 merged r7103:7209 from trunk into time series branch

comment:38 Changed 5 years ago by gkronber

r7268: merged r7214:7266 from trunk into time series branch.

comment:39 Changed 5 years ago by gkronber

r7460: merged r7266:7459 from the trunk into the time series prognosis branch.

comment:40 Changed 5 years ago by gkronber

r7461: fixed a few problems related to a previous incorrect merge and removed unrelated projects from the solution.

comment:41 Changed 5 years ago by gkronber

In the simplified for time series solutions the correct horizon should be used to calculate impacts.

comment:42 Changed 5 years ago by gkronber

r7463: improved formatter, line chart view, simplifier for time series prognosis solutions

comment:43 Changed 5 years ago by gkronber

  • Milestone changed from HeuristicLab 3.3.7 to HeuristicLab 3.3.x Backlog

comment:44 Changed 5 years ago by gkronber

r7615: merged r7462:7609 from trunk into time series branch

comment:45 Changed 5 years ago by gkronber

r7842: merged r7609:7840 from trunk into time series branch

comment:46 Changed 5 years ago by gkronber

r7843: fixed compile errors after merging changes from the trunk

comment:47 Changed 5 years ago by mkommend

r7886: Adapted timeseries branch to contain only modified plugins.

comment:48 Changed 5 years ago by mkommend

r7889: Branched problem instances to enable CSV import for timeseries problems.

comment:49 Changed 5 years ago by mkommend

  • Owner changed from gkronber to mkommend
  • Status changed from accepted to assigned

comment:50 Changed 5 years ago by mkommend

  • Status changed from assigned to accepted

comment:51 Changed 5 years ago by mkommend

r7890: Implemented CSV problems instances provider for time series prognosis problems.

comment:52 Changed 5 years ago by mkommend

r7926: Added interpreter performance unit test for type coherent grammar.

comment:53 Changed 5 years ago by mkommend

r7930: Refactored symbolic expression tree interpreter in preparation for autoregressive single variate prognosis.

comment:54 Changed 5 years ago by gkronber

  • Version changed from 3.3.5 to branch

comment:55 Changed 5 years ago by mkommend

r7989: Improved performance of time series prognosis.

comment:56 Changed 5 years ago by mkommend

r7991: Updated estimation limits correctly in TimeSeriesPrognosisProblem.

comment:57 Changed 5 years ago by mkommend

r7998: Corrected time series interpreter and updated visualizations.

comment:58 Changed 5 years ago by mkommend

r8010: Corrected time series solution results and implemented new models.

comment:59 Changed 5 years ago by mkommend

  • Milestone changed from HeuristicLab 3.3.x Backlog to HeuristicLab 3.3.8

r8114: Corrected scaling behavior of symbolic time series models.

comment:60 Changed 5 years ago by mkommend

r8116: Branched HL.DataAnalysis to add AR models.

comment:61 Changed 5 years ago by mkommend

r8428: Temporary removed files in branch to prepare for trunk update.

comment:62 Changed 5 years ago by mkommend

r8430: Intermediate commit of trunk updates - interpreter changes must be redone.

comment:63 Changed 5 years ago by mkommend

r8431: Moved interpreter specific classes to separate directory.

comment:64 Changed 5 years ago by mkommend

r8432: Corrected path of previous move operation.

comment:65 Changed 5 years ago by mkommend

r8436: Merged changes from trunk into timeseries branch and refactored GP interpreter.

comment:66 Changed 5 years ago by mkommend

r8458: Derived time series classes from regression classes to avoid code duplication.

comment:67 Changed 5 years ago by mkommend

r8459: Code clean up in time series prognosis model and problem data.

comment:68 Changed 5 years ago by mkommend

r8460: Added horizon parameters to timeseries problem data.

comment:69 Changed 5 years ago by mkommend

r8468: Adapated calculation of results for horizons larger than 1.

comment:70 Changed 5 years ago by mkommend

r8477:

  • Added autoregressive target variable Symbol
  • Merged trunk changes into the branch.

comment:71 Changed 5 years ago by mkommend

r8483: Corrected evaluators and time series models.

comment:72 Changed 5 years ago by mkommend

r8487: Added ErrorCharacteristics View for time series models.

comment:73 Changed 5 years ago by mkommend

r8490: Updated TimeSeriesPrognosisSolutionView.

comment:74 Changed 5 years ago by mkommend

r8742: Merged trunk changes and fixed compilation errors due to the merge.

comment:75 Changed 5 years ago by mkommend

r8743: Readded ErrorCharacteristicsView for time series prognosis solutions.

comment:76 Changed 5 years ago by mkommend

r8750: Extracted prognosis results into separate class and added a view for them.

comment:77 Changed 5 years ago by mkommend

r8762: Fixed serialization and corrected AR(n) modeling.

comment:78 Changed 5 years ago by mkommend

r8764: Removed wrong MA models from timeseries prognosis branch.

comment:79 Changed 5 years ago by mkommend

r8765: Fixed minor bugs in time series prognosis views.

comment:80 Changed 5 years ago by mkommend

r8766: Reintegrated trunk updates in timeseries branch.

comment:81 Changed 5 years ago by mkommend

r8767: Merged remaining changes from the trunk.

comment:82 Changed 5 years ago by mkommend

r8768: Corrected plugin dependencies.

comment:83 Changed 5 years ago by mkommend

r8770: Corrected plugin.cs.frame for symbolic time series prognosis views.

comment:84 Changed 5 years ago by mkommend

r8789: Merged trunk changes into timeseries branch.

comment:85 Changed 5 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from accepted to reviewing

r8798: Reintegrated time series modeling branch into trunk.

comment:86 Changed 5 years ago by mkommend

  • Owner changed from gkronber to mkommend
  • Status changed from reviewing to assigned

comment:87 Changed 5 years ago by mkommend

r8799: Corrected time series projects (copy local, output path).

comment:88 Changed 5 years ago by mkommend

r8801: Changed project reference to an assembly reference in Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis and updated build dependencies in the solution file.

This change was necessary to allow compilation on the build server.

comment:89 Changed 5 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from assigned to reviewing

Only r8798, r8799 and r8801 must be reviewed, because r8798 is the trunk integration and contains all previous changes.

Last edited 5 years ago by gkronber (previous) (diff)

comment:90 Changed 5 years ago by abeham

r8851: Changed assembly reference to a project reference

It always gave me build errors when starting from a fresh checkout. I don't know why it didn't fail on the build server, but it should have.

comment:91 Changed 5 years ago by gkronber

r8852: reverse merge of r8851. Project reference doesn't work on the builder because of a strange path-length bug. see r8801

comment:92 Changed 5 years ago by abeham

Ok, the problem that I was seeing seems to be related to this bug in msbuild. The article says it should be fixed in .Net 4.5.

comment:93 Changed 5 years ago by mkommend

r8940: Added EstimationLimits to ISymbolicTimeSeriesPrognosisModel.

comment:94 Changed 5 years ago by gkronber

Again had to correct revision numbers in previous comments. Please be careful.

comment:95 follow-ups: Changed 5 years ago by gkronber

review comments:

  • fields in TimeSeriesPrognosisResults are not cloned.
  • Update of EvaluatedSolutions is not thread safe in the interpreters.
  • Cannot activate the symbol LaggedVariable in the GUI.
  • the default grammar does only allow real-valued symbols in the root (not time series symbols)
  • node impact calculation should be based on the currently selected evaluator. In particular the currently selected training horizon value should be used. Right now impacts are calculated in the same way as for regression models.
  • It would be nice to also provide a line-chart showing the n-step forecast.
Last edited 4 years ago by gkronber (previous) (diff)

comment:96 Changed 5 years ago by gkronber

r9004:

  • locking for update of EvaluatedSolutions in interpreter
  • cloning of fields in TimeSeriesPrognosisResults
Last edited 5 years ago by gkronber (previous) (diff)

comment:97 Changed 4 years ago by gkronber

I think the training partition for time-series instances should automatically start at 1 or 10 otherwise this has to be changed manually each time (otherwise an exception occurs)

comment:98 Changed 4 years ago by gkronber

r9452: created an override for the Load() method in SymbolicTimeSeriesPrognosisSingleObjectiveProblem to set the first index of the start partition to 10 (prevents ArgumentException)

comment:99 Changed 4 years ago by gkronber

  • Owner changed from gkronber to mkommend
  • Status changed from reviewing to assigned
  • Version changed from branch to 3.3.7

comment:100 Changed 4 years ago by gkronber

Please fix the grammar/symbol problems mentioned in comment:95 for the 3.3.8 release. The other issues should be tracked in new tickets.

comment:101 in reply to: ↑ 95 Changed 4 years ago by mkommend

r9459: Updated TypeCoherentGrammar to take time-series, power, and condition symbols into account.

comment:102 in reply to: ↑ 95 Changed 4 years ago by mkommend

  • Owner changed from mkommend to gkronber
  • Status changed from assigned to reviewing

review comments:

  • Cannot activate the symbol LaggedVariable in the GUI.

All variable symbols can only be activated if at least one input variable is allowed.

  • the default grammar does only allow real-valued symbols in the root (not time series symbols)

This was corrected with r9459. However, I have not updated the special functions.

comment:103 Changed 4 years ago by mkommend

r9466: Fixed SGP symbolic classification tests, whose results got updated, due to a change in the grammar in r9459.

comment:104 Changed 4 years ago by gkronber

  • Status changed from reviewing to readytorelease

Reviewed r9459 and r9466. Thanks!

comment:105 Changed 4 years ago by swagner

  • Resolution set to done
  • Status changed from readytorelease to closed
  • Version changed from 3.3.7 to 3.3.8
Note: See TracTickets for help on using tickets.