Persistence Overhaul

[swagner]

The HeuristicLab persistence layer should be overhauled for HeuristicLab 4.0.

[swagner]

r13323: Created new branch for persistence overhaul

[swagner]

r13324: Removed new empty branch in order to create branch of full trunk

[swagner]

r13325: Created branch PersistenceOverhaul of trunk r13321.

[swagner]

r13326: Created plugin for new persistence implementation.

[swagner]

r13347: Worked on new persistence implementation.

[swagner]

r13358: Worked on new persistence implementation.

[swagner]

r13367: Worked on new persistence implementation.

[ascheibe]

r13368 added guids to storable classes

[ascheibe]

r13370 fixed unit tests

[swagner]

r13375: Worked on new persistence implementation.

[ascheibe]

r13386

• fixed duplicate guids
• adapted/added unit tests for new persistence

[swagner]

r13387: Worked on new persistence implementation.

[ascheibe]

r13407 worked on type and enum transformers

[ascheibe]

r13409

• added profiling to unit tests
• added more unit tests

[ascheibe]

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

[gkronber]

Replying to ascheibe:

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.

[ascheibe]

r13410 increased repetitions to get more realistic results

[ascheibe]

Replying to gkronber:

Replying to ascheibe:

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.

Yeah, I know that this is not significant, but we don't have composite transformers yet, so this is all we can do at the moment ;-)

[gkronber]

r14537: worked on persistence

[jkarder]

r14549: worked on persistence

[jkarder]

r14594: worked on persistence

[gkronber]

r14711

• renamed StorableClass -> StorableType
• changed persistence to use GUIDs instead of type names

[gkronber]

r14712: added GUIDs for (almost) all interface types (probably still too many) also added newlines at end of all files

[gkronber]

r14714 added missing StorableType attributes and registered more types. All samples can be deserialized and serialized now

[gkronber]

r14731: added tool program which adds StorableType attributes to interfaces and enums

[gkronber]

r14739: added a sketch for a unit test for conversion (and set svn:ignore properties)

[gkronber]

r14771: added versions to storable types and implemented conversion unit test

[gkronber]

r14919: created a branch for reintegration

[gkronber]

r14920: deleted branch of trunk

[gkronber]

r14921: created a new branch from stable for reintegration

[gkronber]

r14922: deleted branch again

[gkronber]

r14923: created branch from trunk (again)

[gkronber]

r14924: copied relevant files from persistence overhaul branch

[gkronber]

r14925: moved storable attributes (construction)

[gkronber]

r14927: changed all usages of StorableClass to use StorableType with an auto-generated GUID (did not add StorableType to other type definitions yet)

[gkronber]

r14928: added unit tests and fixed all test cases for old persistence

[gkronber]

r14929: fixed all unit tests for new persistence - loading & storing all samples

[gkronber]

r14930: changed optimizer to use new persistence per default

[gkronber]

r14931: do not store graph visualization info for the operatorgraph

[gkronber]

r14932: check if results are the same for the GA TSP sample test

[gkronber]

r14933: added a code fix for generating StorableTypeAttributes

[gkronber]

r14934: svn:ignore

[gkronber]

r14935: support persistence for OperatorGraphVisualizationInfo

[jkarder]

r14985: worked on persistence code fix

• split existing analyzer and code fix
• added analyzer and code fix for missing storable ctors

[jkarder]

r14989: worked on persistence code fix

• added SequentialFixAllProvider

[jkarder]

r14990: worked on persistence code fix

• added missing UnitTests.cs

[gkronber]

• r15018: introduced StorableConstructorFlag type for StorableConstructors
• r15020: fixed unit tests checking for exceptions

[jkarder]

r15022: removed ColorTransformer for now as it only handled the color's argb values (StructTransformer will handle more, e.g. IsEmpty)

[gkronber]

r15034: improved conversions

[gkronber]

r15035: made some changes related to renaming of storable members

[jkarder]

r15501: added protobuf version 3.5.0 and changed proto definition

[jkarder]

r15509: worked on new persistence

• changed message definitions
• updated transformers
• cleaned up

[jkarder]

r15529: worked on new persistence

• changed message definitions
• updated transformers
• worked on conversions
• cleaned up

[jkarder]

r15857: worked on conversions

[jkarder]

r15878: worked on new persistence

• updated tests

[jkarder]

r15986: worked on new persistence

• removed type version
• removed conversions
• execute AfterDeserialization hooks after all objects have been populated
• worked on transformers

[jkarder]

r16210: worked on new persistence

• added cache for different attributes
• refactored StorableTypeBoxTransformer
• implemented paths for Storable members
• implemented more unit tests

[jkarder]

r16396: worked on new persistence

• improved EnumerableBoxTransformer
• fixed output path

[jkarder]

Since we moved the implementation of the new persistence to GitLab, we will continue as follows:

• Create fresh branch from trunk and continue working from there
• Upgrade all projects to .NET 4.6.1
• Publish and reference NuGet package of new persistence

[jkarder]

r16450: deleted branch

[jkarder]

r16451: created branch for reintegration of new persistence

[jkarder]

r16452: upgraded projects to .NET 4.6.1

[jkarder]

r16453: updated year of copyrights

[jkarder]

r16454: added nuget references to HEAL.Fossil (which has not been published yet)

[jkarder]

r16462: worked on reintegration of new persistence

• added nuget references to HEAL.Fossil
• added StorableType attributes to many classes
• changed signature of StorableConstructors
• removed some classes in old persistence
• removed some unnecessary usings

[gkronber]

I'm currently not able to build the branch using VS2017 or the Build.ps script.

The error messages are:

...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error : Could not find a part of the path '...2520_PersistenceReintegration\ \packages\HEAL.Fossil.1.0.0'.

...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error : Could not find a part of the path '...2520_PersistenceReintegration\ \packages\Google.Protobuf.3.6.1'.

...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error MSB3073: The command ""...2520_PersistenceReintegration\.nuget\NuGet.exe" install "...\2520_PersistenceReintegration\HeuristicLab.Persistence\3.3\packages.config" -source "" -NonInteractive -RequireConsent -solutionDir "...\2520_PersistenceReintegration\ "" exited with code 1.

Two observations:

• there is a space in the path for the packages
• the package sources list is empty.

[gkronber]

r16465: fixed issue when building with Build.ps and VS 2017 when packages must be restored

Seemingly the "." is necessary so that nuget (or msbuild?) produces a correct path. I have no idea.

Btw. the docs on nuget are a complete mess.

[gkronber]

r16466: removed unit tests which are moved to the HEAL.Fossil repository

[abeham]

I want to point out that Fossil is a ​registered trademark (trademark search: ​TM View).

[gkronber]

r16467: added unit tests for converting all samples (open old format, store new format)

[gkronber]

r16468: added the necessary StorableType attributes in HeuristicLab.Algorithms.DataAnalysis

[gkronber]

Replying to abeham:

I want to point out that Fossil is a ​registered trademark (trademark search: ​TM View).

Any other suggestions? What about Gajost or Kawakr :)

[gkronber]

r16469: Adapted VS code fixes:

• using HeuristicLab.Persistence -> using HEAL.Fossil
• Made missing StorableCtor a Warning
• Ignore StorableCtor for structs and interfaces

[gkronber]

r16470: fixed usings

[abeham]

Replying to gkronber:

Replying to abeham:

I want to point out that Fossil is a ​registered trademark (trademark search: ​TM View).

Any other suggestions? What about Gajost or Kawakr :)

Kawakr doesn't sound too bad. Other suggestions:

• Parrot (repeats things)
• Dolphin (long memory)
• Beagle (Darwin's famous ship)
• Serio (or SerioSharp) - SERialize Input Output
• Rializer (seRIALIZER)
• Alizer (or just Alize)
• Revasio (reverse of saver + io)
• StoreOfTheFiles
• Inout
• Bittersweet
• Bytespot
• Bytepack

[gkronber]

Replying to abeham:

Kawakr doesn't sound too bad. Other suggestions:

• Parrot (repeats things)
• Dolphin (long memory)
• Beagle (Darwin's famous ship)
• Serio (or SerioSharp) - SERialize Input Output
• Rializer (seRIALIZER)
• Alizer (or just Alize)
• Revasio (reverse of saver + io)
• StoreOfTheFiles
• Inout
• Bittersweet
• Bytespot
• Bytepack

Good stuff, I especially like: Alize or Alizer (HEAL.Alizer) and Rialize or Rializer.

[gkronber]

r16471:16472: checked all projects from top to HeuristicLab.Common and added StorableType attributes.

[gkronber]

Here is a list of all folders and their status concerning the StorableType attribute check:

• HeuristicLab (not necessary)
• HeuristicLab.Algorithms.ALPS (done)
• HeuristicLab.Algorithms.Benchmarks (done)
• HeuristicLab.Algorithms.Benchmarks.Views (done)
• HeuristicLab.Algorithms.CMAEvolutionStrategy (done)
• HeuristicLab.Algorithms.DataAnalysis (done)
• HeuristicLab.Algorithms.DataAnalysis.Glmnet (done)
• HeuristicLab.Algorithms.DataAnalysis.Views (done)
• HeuristicLab.Algorithms.EvolutionStrategy (done)
• HeuristicLab.Algorithms.GeneticAlgorithm (done)
• HeuristicLab.Algorithms.GradientDescent (done)
• HeuristicLab.Algorithms.LocalSearch (done)
• HeuristicLab.Algorithms.MOCMAEvolutionStrategy (done)
• HeuristicLab.Algorithms.NSGA2 (done)
• HeuristicLab.Algorithms.OffspringSelectionEvolutionStrategy (done)
• HeuristicLab.Algorithms.OffspringSelectionGeneticAlgorithm (done)
• HeuristicLab.Algorithms.ParameterlessPopulationPyramid (done)
• HeuristicLab.Algorithms.ParticleSwarmOptimization (done)
• HeuristicLab.Algorithms.RandomSearch (done)
• HeuristicLab.Algorithms.RAPGA (done)
• HeuristicLab.Algorithms.ScatterSearch (done)
• HeuristicLab.Algorithms.SimulatedAnnealing (done)
• HeuristicLab.Algorithms.TabuSearch (done)
• HeuristicLab.Algorithms.VariableNeighborhoodSearch (done)
• HeuristicLab.Analysis (done)
• HeuristicLab.Analysis.Statistics.Views (done)
• HeuristicLab.Analysis.Views (done)
• HeuristicLab.Clients.Access (skipped)
• HeuristicLab.Clients.Access.Administration (skipped)
• HeuristicLab.Clients.Access.Views (skipped)
• HeuristicLab.Clients.Common (skipped)
• HeuristicLab.Clients.Hive (skipped)
• HeuristicLab.Clients.Hive.Administrator (skipped)
• HeuristicLab.Clients.Hive.JobManager (skipped)
• HeuristicLab.Clients.Hive.Slave
• HeuristicLab.Clients.Hive.Slave.App
• HeuristicLab.Clients.Hive.Slave.ConsoleClient
• HeuristicLab.Clients.Hive.Slave.TrayIcon
• HeuristicLab.Clients.Hive.Slave.Views
• HeuristicLab.Clients.Hive.Slave.WindowsService
• HeuristicLab.Clients.Hive.Views (skipped)
• HeuristicLab.Clients.OKB (done)
• HeuristicLab.Clients.OKB.Views (done)
• HeuristicLab.CodeEditor (skipped)
• HeuristicLab.Collections (done)
• HeuristicLab.Common (done)
• HeuristicLab.Common.Resources (done)
• HeuristicLab.Core (done)
• HeuristicLab.Core.Views (done)
• HeuristicLab.Data (done)
• HeuristicLab.Data.Views (done)
• HeuristicLab.DataPreprocessing (done)
• HeuristicLab.DataPreprocessing.Views (done)
• HeuristicLab.DebugEngine (done)
• HeuristicLab.DebugEngine.Views (done)
• HeuristicLab.Encodings.BinaryVectorEncoding (done)
• HeuristicLab.Encodings.IntegerVectorEncoding (done)
• HeuristicLab.Encodings.LinearLinkageEncoding (done)
• HeuristicLab.Encodings.PermutationEncoding (done)
• HeuristicLab.Encodings.PermutationEncoding.Views (done)
• HeuristicLab.Encodings.RealVectorEncoding (done)
• HeuristicLab.Encodings.ScheduleEncoding (done)
• HeuristicLab.Encodings.ScheduleEncoding.Views (done)
• HeuristicLab.Encodings.SymbolicExpressionTreeEncoding (done)
• HeuristicLab.Encodings.SymbolicExpressionTreeEncoding.Views (done)
• HeuristicLab.ExtLibs (not necessary)
• HeuristicLab.Hive (skipped)
• HeuristicLab.MainForm (done)
• HeuristicLab.MainForm.WindowsForms (done)
• HeuristicLab.Operators (done)
• HeuristicLab.Operators.Programmable (done)
• HeuristicLab.Operators.Programmable.Views (done)
• HeuristicLab.Operators.Views (done)
• HeuristicLab.Operators.Views.GraphVisualization (done)
• HeuristicLab.Operators.Views.GraphVisualization.Views (done)
• HeuristicLab.Optimization (done)
• HeuristicLab.Optimization.Operators (done)
• HeuristicLab.Optimization.Views (done)
• HeuristicLab.Optimizer (done)
• HeuristicLab.ParallelEngine (done)
• HeuristicLab.ParallelEngine.Views (done)
• HeuristicLab.Parameters (done)
• HeuristicLab.Parameters.Views (done)
• HeuristicLab.Persistence (skipped)
• HeuristicLab.Persistence.GUI (not necessary)
• HeuristicLab.PluginInfrastructure (not necessary)
• HeuristicLab.Problems.Binary (done)
• HeuristicLab.Problems.BinPacking (done)
• HeuristicLab.Problems.BinPacking.Views (done)
• HeuristicLab.Problems.DataAnalysis (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic.Classification (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic.Classification.Views (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic.Regression (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic.Regression.Views (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis.Views (done)
• HeuristicLab.Problems.DataAnalysis.Symbolic.Views (done)
• HeuristicLab.Problems.DataAnalysis.Trading (done)
• HeuristicLab.Problems.DataAnalysis.Trading.Views (done)
• HeuristicLab.Problems.DataAnalysis.Views (done)
• HeuristicLab.Problems.ExternalEvaluation (done)
• HeuristicLab.Problems.ExternalEvaluation.GP (done)
• HeuristicLab.Problems.ExternalEvaluation.Matlab (done)
• HeuristicLab.Problems.ExternalEvaluation.Scilab (done)
• HeuristicLab.Problems.ExternalEvaluation.Views (done)
• HeuristicLab.Problems.GeneticProgramming (done)
• HeuristicLab.Problems.GeneticProgramming.Views (done)
• HeuristicLab.Problems.GrammaticalEvolution (done)
• HeuristicLab.Problems.GraphColoring (done)
• HeuristicLab.Problems.Instances (not necessary)
• HeuristicLab.Problems.Instances.CordeauGQAP (not necessary)
• HeuristicLab.Problems.Instances.DataAnalysis (done)
• HeuristicLab.Problems.Instances.DataAnalysis.Views (not necessary)
• HeuristicLab.Problems.Instances.DIMACS (not necessary)
• HeuristicLab.Problems.Instances.ElloumiCTAP (not necessary)
• HeuristicLab.Problems.Instances.Orienteering (not necessary)
• HeuristicLab.Problems.Instances.QAPLIB (not necessary)
• HeuristicLab.Problems.Instances.Scheduling (not necessary)
• HeuristicLab.Problems.Instances.TSPLIB (not necessary)
• HeuristicLab.Problems.Instances.TSPLIB.Views (not necessary)
• HeuristicLab.Problems.Instances.VehicleRouting (not necessary)
• HeuristicLab.Problems.Instances.VehicleRouting.Views (not necessary)
• HeuristicLab.Problems.Instances.Views (not necessary)
• HeuristicLab.Problems.Knapsack (done)
• HeuristicLab.Problems.Knapsack.Views (done)
• HeuristicLab.Problems.LinearAssignment (done)
• HeuristicLab.Problems.LinearAssignment.Views (done)
• HeuristicLab.Problems.NK (done)
• HeuristicLab.Problems.Orienteering (done)
• HeuristicLab.Problems.Orienteering.Views (done)
• HeuristicLab.Problems.ParameterOptimization (done)
• HeuristicLab.Problems.Programmable (done)
• HeuristicLab.Problems.Programmable.Views (done)
• HeuristicLab.Problems.PTSP (done)
• HeuristicLab.Problems.PTSP.Views (done)
• HeuristicLab.Problems.QuadraticAssignment (done)
• HeuristicLab.Problems.QuadraticAssignment.Algorithms (done)
• HeuristicLab.Problems.QuadraticAssignment.Views (done)
• HeuristicLab.Problems.Scheduling (done)
• HeuristicLab.Problems.Scheduling.Views (done)
• HeuristicLab.Problems.TestFunctions (done)
• HeuristicLab.Problems.TestFunctions.MultiObjective (done)
• HeuristicLab.Problems.TestFunctions.Views (done)
• HeuristicLab.Problems.TravelingSalesman (done)
• HeuristicLab.Problems.TravelingSalesman.Views (done)
• HeuristicLab.Problems.VehicleRouting (done)
• HeuristicLab.Problems.VehicleRouting.Views (done)
• HeuristicLab.Random (done)
• HeuristicLab.Scripting (done) TODO: Test
• HeuristicLab.Scripting.Views (done)
• HeuristicLab.Selection (done)
• HeuristicLab.SequentialEngine (done)
• HeuristicLab.Services.Access
• HeuristicLab.Services.Access.DataAccess
• HeuristicLab.Services.Deployment
• HeuristicLab.Services.Deployment.DataAccess
• HeuristicLab.Services.Hive
• HeuristicLab.Services.Hive.DataAccess
• HeuristicLab.Services.Hive.JanitorService
• HeuristicLab.Services.Hive.Web
• HeuristicLab.Services.OKB
• HeuristicLab.Services.OKB.DataAccess
• HeuristicLab.Services.WebApp
• HeuristicLab.Services.WebApp.Maintenance
• HeuristicLab.Services.WebApp.Statistics
• HeuristicLab.Services.WebApp.Status
• HeuristicLab.Tests (not necessary)
• HeuristicLab.Tracing
• HeuristicLab.Visualization.ChartControlsExtensions

[abeham]

Need help?

[gkronber]

Not working on it right now. Waiting for Johannes to commit the fixed code fix to make it easier.

[swagner]

Replying to abeham:

Replying to gkronber:

Replying to abeham:

I want to point out that Fossil is a ​registered trademark (trademark search: ​TM View).

Any other suggestions? What about Gajost or Kawakr :)

Kawakr doesn't sound too bad. Other suggestions:

• Parrot (repeats things)
• Dolphin (long memory)
• Beagle (Darwin's famous ship)
• Serio (or SerioSharp) - SERialize Input Output
• Rializer (seRIALIZER)
• Alizer (or just Alize)
• Revasio (reverse of saver + io)
• StoreOfTheFiles
• Inout
• Bittersweet
• Bytespot
• Bytepack

Some additional suggestions:

• Hippocampus (part of the brain responsible for creating memory)
• Freezer
• Cryo
• Iceberg
• O2D (objects to disk)
• Mnemonic
• Memorizer
• Recall
• Stash
• ObjectStore
• ObjectVault
• Boxing
• ObjectBox

By the way, I also like Dolphin, Serio and Alizer ...

gkronber and jkarder, please choose a name you like.

[gkronber]

r16473: changed name and type of parameter for StorableConstructor

[gkronber]

r16474: checked and added StorableType attribute in projects up to HeuristicLab.MainForm

[jkarder]

r16475: added analyzer and fix for adapting storable ctor signature

[gkronber]

r16476: checked and added StorableType attribute in projects up to HeuristicLab.Problems.GraphColoring

[gkronber]

r16477: checked and added StorableType attribute in projects up to HeuristicLab.Problems.VRP.Views

[gkronber]

r16479: checked and added StorableType attribute projects included in HeuristicLab 3.3.sln

[gkronber]

OKB uses XmlGenerator / XmlParser in several places. How do we handle persisted data in OKB?

[gkronber]

ExternalEvaluation.GP uses XmlGenerator to serialize trees to byte[] for transmission to external module for evaluation. How do we proceed?

[gkronber]

r16480: use new persistence in ContentManager (with fallback), ClipBoard (with fallback), as well as for deep-cloning in Scripting and for Hive Persistence (without fallback & incomplete)

[abeham]

Good questions. I would prefer starting over with a fresh OKB database and re-uploading any data that is still available in files. I would be fine with that and would be less hassle than trying to convert it. But we should ask others if they've got any data there worthy of being preserved. I also think something like a convergence graph shouldn't be binary data, but that's a design problem. The other binary data is, I think, algorithms and problems.

Regarding ExternalEvaluation, it should be used to transmit trees and not meant to persist them. Thus, we should get away with changing that to the new persistence. However, there's also the EvaluationCache that may store solution data. Erik implemented this, we should check with him.

[gkronber]

r16481: fixed StorableSerializer of old persistence to make the old persistence unit test succeed

[gkronber]

• r16482: added two custom transformers (for Fonts and Bitmaps) and registered types that are available in .NET Framework but not in .NET Standard 2.0 and added.
• r16483: added unit test for IndexedDataTable and Point2D and made necessary fixes
• r16482: minor cleanup in PersistenceContentManager

[gkronber]

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

[gkronber]

I still have issues with building in VS. VS complains e.g. that IEnumerable<> is defined in .netstandard and I need to add a reference, while building with Build.ps works. Sometimes restarting VS works. No idea what is happening here.

[gkronber]

Replying to gkronber:

I still have issues with building in VS.

Notably, VS shows errors (in the error window and using intellisense) but building still succeeds.

[gkronber]

r16458: replaced static type registration in HL.Persistence plugin with the KnownStorableTypesMap which is discovered and used by HEAL.Fossil.

As a consequence we do not need to explicitly call type registration for unit tests. This change has been necessary because on the first access of StaticCache all StorableTypes are registered. However, when the HL.Persistence plugin is loaded we have not yet loaded all plugins and as a consequence we would miss StorableTypes.

[gkronber]

r16486: added license headers

[gkronber]

Current state:

• all persistence unit tests (old and new) succeed. This includes a unit test that loads all samples in old format and stores them in the new format.
• GA - TSP sample can be loaded and executed in HL. Pause / store / load / continue using the new format works and produces the same results.
• Clipboard store / load works with new new format.
• load, start, pause & store works for all samples
• Complete implementation of HEAL.Attic, publish to nuget, update the package source for HEAL.Attic
• Change ExternalEvaluation and EvaluationCache to use new serialization format (existing users must update their evaluation code). added a new serializer and marked the old serializer as obsolete.
• Adapt Hive to use new persistence format.
• Final decision on the name and license.
• Re-enable cancellation of serialization for new persistence.

TODO:

• unit test for for pause, store, load and continue both and compare results of both algorithms
• test C# scripting and programmable problem (regarding cloning using persistence in VariableStore)
• re-deploy all HiveWorkers.
• Implement a converter for all serialized data in OKB and convert the data. Change OKB to use new persistence.
• Apply code fixes to all Addons and test.
• Performance comparison old vs. new
• Change unit tests for samples to use new persistence

[gkronber]

r16487: merged r16478 (HEAD) from trunk to branch

[mkommend]

Replying to gkronber:

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

Is there a possibility that we could solve this?

It is a major drawback to use a software on one PC and then not be able to open created files on another machine, even more so as this is not obvious to the user.

What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?

It would still be preferable if the persistence is .NET version / implementation agnostic.

[gkronber]

Replying to mkommend:

Replying to gkronber:

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

Is there a possibility that we could solve this?

Yes.

What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?

There would be different release bundles one for .NET Core and one for .NET Framework. The user would be aware which version of HL he or she is running.

It would still be preferable if the persistence is .NET version / implementation agnostic.

Yes. This means we need to restrict ourselves to types and members which are available in both .NET Core and .NET Framework (which .NET Standard types).

Currently, we have the following additional types registered for HL which are not part of .NET Standard:

• System.Drawing.Font, (used in DataTableVisualProperties, ScatterplotVisualProperties. Maybe we can remove persistence of fonts?
• System.Drawing.FontStyle, (for serialization of Font)
• System.Drawing.GraphicsUnit, (for serialization of Font)
• System.Drawing.Bitmap, (necessary for ItemImage)
• System.Collections.StructuralEqualityComparer (not sure if this is actually necessary)

[abeham]

Good work, Gabriel!

Maybe we can escape the System.Drawing problems by replacing all references to the System.Drawing assembly in the GAC with the respective ​nuget package.

I can't compile the branch either, but due to missing HEAL.Fossil.

[gkronber]

As discussed with swagner:

• OKB: swagner will try to implement a converter for existing data
• Hive: The HiveClient should use the new format, HiveWorkers should try to deserialize using the new format and if it fails fallback to the old format.

[gkronber]

jkarder suggests: heal.attic (attic = dachbodenspeicher)

[pfleck]

The types HeuristicLab.Analysis.IDataRow and HeuristicLab.Analysis.IDataTable<TDataRow> were added to the trunk in r16527. Please also add a StorableTypeAttribute to them after merging the Persistence back to the trunk.

[gkronber]

r16529: merged r16491:16528 from trunk to persistence branch

[gkronber]

r16530: made the necessary changes for the new Persistence for types which have been added in the trunk

[gkronber]

r16539: use nuget package System.Drawing.Common to allow cross-platform persistence for types from System.Drawing.

[gkronber]

r16551: merged r16542:16543 from trunk to branch

[gkronber]

r16552: added a new serializer for trees in the ExternalEvaluation.GP plugin and marked the serializer using the old XML format as obsolete

[gkronber]

r16553:16554 for Hive serialization: added collection of serialized types and fall-back to old persistence format on deserialization. ITask must be a StorableType

[gkronber]

r16555: added a unit test to try to serialize samples which have been running for 20secs.

[jkarder]

r16558: updated Fossil version and PersistenceUtil for hive

[jkarder]

r16559: renamed Fossil to Attic and set version to 1.0.0-pre01

[gkronber]

r16562: deleted unnecessary app.config file

[gkronber]

r16563: re-added support for cancellation of save operation

[gkronber]

r16564: added HEAL.Attic.dll, Google.Protobuf.dll and System.Drawing.Common.dll as PluginFiles provided by HeuristicLab.Persistence

[gkronber]

r16565: merged changes from PersistenceOverhaul branch (r16451:16564) into trunk

[gkronber]

r16566: updated HeuristicLab.DataImporter addon for new Persistence (introduced in 3.3.16)

[gkronber]

r16567: changed StorableConstructors and added StorableType attributes in HeuristicLab.DataImporter addon

[gkronber]

r16568: updated HEAL.Attic version (and reference the net461 specific build

[gkronber]

r16569: updated version of HEAL.Attic for the HeuristicLab.Tests solution

[gkronber]

r16570: updated HEAL.Attic version and use the net461 version in HeuristicLab.DataImporter addon

[gkronber]

r16571: several fixes based on failing unit tests

• adapted unit tests that check StorableConstructors
• missing translations of StorableClass -> StorableType
• missing StorableConstructors
• missing or unecessary PluginDependencies / ProjectReferences

[gkronber]

• r16572: removed unnecessary references / plugin dependencies in Hive.Slave projects
• r16573: changed HeuristicLab.FLA addon to compile with new HL.Persistence

[gkronber]

r16574: changed HeuristicLab.MetaOptimization addon to compile with new HL.Persistence

[gkronber]

r16575: changed HeuristicLab.BioBoost addon to compile with new HL.Persistence

[gkronber]

r16576: changed Fossil -> Attic in VS Code Fix

[gkronber]

r16577: added missing StorableType attributes for interfaces in HeuristicLab.FitnessLandspaceAnalysis

[gkronber]

r16578: changed unit tests which load old samples and try storing and loading using the new format to recreate all sample files first

[gkronber]

r16579: try fall-back to old persistence for all exceptions that might occur in persistence

[gkronber]

r16584: added missing StorableType attributes

[gkronber]

r16585: Update HEAL.Attic to pre03

[gkronber]

r16594: added unit tests for profiling old vs new persistence with samples

[gkronber]

r16595: changed a comment

[jkarder]

r16625: switched projects in the following solutions from packages.config format to PackageReference format (where possible)

• HeuristicLab 3.3
• HeuristicLab 3.3 Tests
• HeuristicLab 3.3 Services

[gkronber]

TODO:

• update format of samples on StartPage as well as serializer used for these samples.
• check setters of storable properties whether they assume that the recieved value is already initialized. This is not the case for e.g. StorableOperators in Encoding.cs line 44.

[gkronber]

r16658: created an extlibs wrapper plugin for the HEAL.Attic nuget package (and updated to 1.0.0-pre4)

[gkronber]

r16659: svn:ignore

[gkronber]

r16672: removed nuget package reference missed in r16658

[jkarder]

r16704 (merged): fixed RunGaussianProcessRegressionSample unit test which failed since we migrated to .NET 4.6.1 in r16452

[mkommend]

r16706 (merged): Adapted sample unit test GPSymRegWithOS in x64 bit mode due to the occuring quality differences when compiling with .Net 4.6.1.

Careful: trunk commit!

[gkronber]

r16707: Update HEAL.Attic to release version 1.0.

[abeham]

In r16565 usings were removed from the C# Script template and the programmable problem templates in the trunk. You have to disable remove and sort usings when saving those files, because the usings are put there only for convenience. I reverted those changes in these files:

r16729: Readded missing usings in code template files

[mkommend]

Could you please remove the deprecated warnings introduced by the new persistence in the trunk solution?

Warning CS0618 'StorableAttribute.AllowOneWay' is obsolete: 'Use OldName instead.'

This occurs multiple times throughout the whole solution.

[gkronber]

We should probably fix all occurrences in HL trunk.

[abeham]

r16792: Extracted Google.Protobuf into own extlib plugin and added both plugin dependency and assembly reference (build order) in HeuristicLab.Attic

I also tried to get rid of the other protobuf implementation that we currently use in external evaluation, but this proved to be a more complex and time consuming task (cf. #3003).

[gkronber]

r16796: removed usages of AllowOneWay

[gkronber]

r16797: fixed comparison logic bug introduced in previous commit

[gkronber]

r16799: made another fix concerning handling of OldName instead of AllowOneWay in old persistence (fix unit tests).

[abeham]

I think we should add a unit test to check for duplicate GUIDs in StorableType. I just copy and pasted a class and forgot to change the GUID. I think this may be a common error people will make.

Additionally, Michael and I discussed that we should ensure, also by test, that all types have a StorableType attribute. Potentially, generate a list once of all current types, then add exceptions and require that all types not covered in the exceptions are StorableTypes.

[abeham]

r16819: changed references to PluginInfrastructure from file reference to project reference

[gkronber]

r16906: disable compiler warning for old persistence unit tests

[gkronber]

r16907: Show more info in the exception dialog when persistence fails

[gkronber]

r16908: added unit test to check for duplicate GUIDs

[gkronber]

r16933: changed exception handling / retry with old persistence format in PersistenceContentManager

[jkarder]

r16934 (stable commit): merged r16704 and r16706 into stable

[jkarder]

r16945: changed deserialization in PersistenceContentManager

• added method to check if old persistence can open a file
• used this method to determine which persistence to use

[abeham]

Is there any more work necessary? Otherwise, I think we should move this to review. I would propose that mkommend and myself schedule some time to do a review within the next two weeks together.

[gkronber]

r16992: Use ContentManager in Clipboard and StartPage. Show info when objects cannot be restored.

[gkronber]

r16994: Update plugin dependencies and references for HL.DataImporter for new persistence

[gkronber]

r16995: Update plugin dependencies and references for HL.FLA for new persistence

[gkronber]

r16996: Update plugin dependencies and references for HL.MetaOptimization for new persistence

[gkronber]

r16997: Update plugin dependencies and references for Core.Views and Optimizer for new persistence

[jkarder]

r17014: Update HEAL.Attic to version 1.1.0

[abeham]

r17015: fixed bug in the clipboard when it contains objects not of type IStorableContent

[abeham]

r17017: removed branch

[abeham]

r17019: fixed unit test to check for duplicate GUIDs (includes NonDiscoverable types)

[mkommend]

r17020: Adapted storable properties from SymbolicExpressionGrammarBase to use new temporary fields and populate the real data fields in the after deserialization hook.

[mkommend]

r17021: Adapted all unit tests to use attic instead of the xml persistence. This affects all sample unit tests, the test resources, script unit tests and some general unit tests.

[mkommend]

r17022: Changed samples to attic format and adapted startpage to load the samples with attic without the creation of temporary files.

[mkommend]

r17023: Added forgotten plugin dependency to HEAL.Attic in HL.Optimizer plugin file.

[mkommend]

r17024: Adapted unit test for loading samples.

[mkommend]

r17029: Updated Attic Version of FLA, MetaOpt, and BioBoost addon.

[abeham]

Patch that alters retry with old persistence, to be applied when HEAL.Attic 1.2 is available.

[abeham]

TODO:
5. ExtLibs / SimSharp (folder)
7. Persistence.GUI (folder)
8. Persistence
  8.1 Default / Xml / XmlGenerator.cs
9. DataAnalysis.Symbolic.Regression
  9.1 MultiObjective / PearsonRSquaredAverageSimilarityEvaluator.cs
10. DataAnalysis.Symbolic:
  10.1. Analyzers / SymbolicDataAnalysisBuildingBlockAnalyzer.cs
  10.2 Converters / DerivativeCalculator.cs
  10.3 Crossovers / SymbolicDataAnalysisExpressionDiversityPreservingCrossover.cs
  10.4 Hashing (folder TC)
  10.5 csproj file
  10.6 Interpreter / SymbolicDataAnalysisExpressionTreeBatchInterpreter.cs
  10.7 Interpreter / SymbolicDataAnalysisExpressionTreeNativeInterpreter.cs
  10.8 packages.config
  10.9 Symbols / Absolute.cs
  10.10 Symbols / AnalyticQuotient.cs
  10.11 Symbols / Cube.cs
  10.12 Symbols / CubeRoot.cs
11. Tests
  11.1. DataAnalysis.Symbolic / DeriveTest.cs

DONE:
1. Algorithms.DataAnalysis(.Views)
  1.1 LinearRegressionModelView.cs -> #2892 / 16449
  1.2 LinearRegressionModelView.Designer.cs -> #2892 / 16449
  1.3 Linear / LinearRegressionModel.cs -> #2892 / 16389, 16415, 16448, 16449
2. Core
  2.1 PersistenceContentManager.cs
3. Data
  3.1 TriangularMatrix.cs -> #2919 / 15931, 15932, 15934
4. ExtLibs / NativeInterpreter (folder)
6. MainForm.WindowsForms
  6.1 Progress.cs

[abeham]

r17086: updated nuget package of Attic to 1.2

[abeham]

r17087:

• fixed loading of samples, removed TODO (due to upgrade of Attic 1.2)
• corrected check when old persistence should be used to retry download from hive

[abeham]

r17088: tested and simplified check

[abeham]

r17089: Removed reference to Protobuf in Clients.Hive since the exception is no longer referenced

[mkommend]

Reviewed the last four changesets by abeham.

This ticket blocks the release of several other tickets. Therefore, I am moving this to ready to be release. Any subsequent development should be tracked in another ticket.

[mkommend]

r17097: Merged r16565:16579 into stable.

[mkommend]

r17105: Merged r16584, r16585,r16594,r16595, r16625, r16658, r16659, r16672, r16707, r16729, r16792, r16796, r16797, r16799, r16819, r16906, r16907, r16908, r16933, r16945, r16992, r16994, r16995, r16996, r16997, r17014, r17015, r17017, r17020, r17021, r17022, r17023, r17024, r17029, r17086, r17087, r17088, r17089 into stable.

[gkronber]

r17165: merged r17019 from trunk to stable