Opened 4 years ago

Last modified 5 days ago

#2520 accepted enhancement

Persistence Overhaul

Reported by: swagner Owned by: gkronber
Priority: high Milestone: HeuristicLab 3.3.16
Component: Persistence Version: trunk
Keywords: Cc:

Description

The HeuristicLab persistence layer should be overhauled for HeuristicLab 4.0.

Change History (177)

comment:1 Changed 4 years ago by swagner

r13323: Created new branch for persistence overhaul

comment:2 Changed 4 years ago by swagner

r13324: Removed new empty branch in order to create branch of full trunk

comment:3 Changed 4 years ago by swagner

r13325: Created branch PersistenceOverhaul of trunk r13321.

comment:4 Changed 4 years ago by swagner

r13326: Created plugin for new persistence implementation.

comment:5 Changed 4 years ago by swagner

r13347: Worked on new persistence implementation.

comment:6 Changed 4 years ago by swagner

r13358: Worked on new persistence implementation.

comment:7 Changed 4 years ago by swagner

r13367: Worked on new persistence implementation.

comment:8 Changed 4 years ago by ascheibe

r13368 added guids to storable classes

comment:9 Changed 4 years ago by ascheibe

r13370 fixed unit tests

comment:10 Changed 4 years ago by swagner

r13375: Worked on new persistence implementation.

comment:11 Changed 4 years ago by ascheibe

r13386

  • fixed duplicate guids
  • adapted/added unit tests for new persistence

comment:12 Changed 4 years ago by swagner

r13387: Worked on new persistence implementation.

comment:13 Changed 4 years ago by ascheibe

r13407 worked on type and enum transformers

comment:14 Changed 4 years ago by ascheibe

r13409

  • added profiling to unit tests
  • added more unit tests

comment:15 follow-up: Changed 4 years ago by ascheibe

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

comment:16 in reply to: ↑ 15 ; follow-up: Changed 4 years ago by gkronber

Replying to ascheibe:

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.

comment:17 Changed 4 years ago by ascheibe

r13410 increased repetitions to get more realistic results

comment:18 in reply to: ↑ 16 Changed 4 years ago by ascheibe

Replying to gkronber:

Replying to ascheibe:

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.

Yeah, I know that this is not significant, but we don't have composite transformers yet, so this is all we can do at the moment ;-)

comment:19 Changed 2 years ago by gkronber

r14537: worked on persistence

comment:20 Changed 2 years ago by jkarder

r14549: worked on persistence

comment:21 Changed 2 years ago by jkarder

r14594: worked on persistence

comment:22 Changed 2 years ago by gkronber

r14711

  • renamed StorableClass -> StorableType
  • changed persistence to use GUIDs instead of type names

comment:23 Changed 2 years ago by gkronber

r14712: added GUIDs for (almost) all interface types (probably still too many) also added newlines at end of all files

comment:24 Changed 2 years ago by gkronber

r14714 added missing StorableType attributes and registered more types. All samples can be deserialized and serialized now

comment:25 Changed 2 years ago by gkronber

r14731: added tool program which adds StorableType attributes to interfaces and enums

comment:26 Changed 2 years ago by gkronber

r14739: added a sketch for a unit test for conversion (and set svn:ignore properties)

comment:27 Changed 2 years ago by gkronber

r14771: added versions to storable types and implemented conversion unit test

comment:28 Changed 2 years ago by gkronber

r14919: created a branch for reintegration

comment:29 Changed 2 years ago by gkronber

r14920: deleted branch of trunk

comment:30 Changed 2 years ago by gkronber

r14921: created a new branch from stable for reintegration

comment:31 Changed 2 years ago by gkronber

r14922: deleted branch again

comment:32 Changed 2 years ago by gkronber

r14923: created branch from trunk (again)

comment:33 Changed 2 years ago by gkronber

r14924: copied relevant files from persistence overhaul branch

comment:34 Changed 2 years ago by gkronber

r14925: moved storable attributes (construction)

comment:35 Changed 2 years ago by gkronber

r14927: changed all usages of StorableClass to use StorableType with an auto-generated GUID (did not add StorableType to other type definitions yet)

comment:36 Changed 2 years ago by gkronber

r14928: added unit tests and fixed all test cases for old persistence

comment:37 Changed 2 years ago by gkronber

r14929: fixed all unit tests for new persistence - loading & storing all samples

comment:38 Changed 2 years ago by gkronber

r14930: changed optimizer to use new persistence per default

comment:39 Changed 2 years ago by gkronber

r14931: do not store graph visualization info for the operatorgraph

comment:40 Changed 2 years ago by gkronber

r14932: check if results are the same for the GA TSP sample test

comment:41 Changed 2 years ago by gkronber

r14933: added a code fix for generating StorableTypeAttributes

comment:42 Changed 2 years ago by gkronber

r14934: svn:ignore

comment:43 Changed 2 years ago by gkronber

r14935: support persistence for OperatorGraphVisualizationInfo

comment:44 Changed 2 years ago by gkronber

  • Owner changed from swagner to gkronber
  • Status changed from new to accepted

comment:45 Changed 2 years ago by jkarder

r14985: worked on persistence code fix

  • split existing analyzer and code fix
  • added analyzer and code fix for missing storable ctors

comment:46 Changed 2 years ago by jkarder

r14989: worked on persistence code fix

  • added SequentialFixAllProvider

comment:47 Changed 2 years ago by jkarder

r14990: worked on persistence code fix

  • added missing UnitTests.cs

comment:48 Changed 2 years ago by gkronber

  • r15018: introduced StorableConstructorFlag type for StorableConstructors
  • r15020: fixed unit tests checking for exceptions

comment:49 Changed 2 years ago by jkarder

r15022: removed ColorTransformer for now as it only handled the color's argb values (StructTransformer will handle more, e.g. IsEmpty)

comment:50 Changed 2 years ago by gkronber

r15034: improved conversions

comment:51 Changed 2 years ago by gkronber

r15035: made some changes related to renaming of storable members

comment:52 Changed 19 months ago by jkarder

r15501: added protobuf version 3.5.0 and changed proto definition

comment:53 Changed 19 months ago by jkarder

r15509: worked on new persistence

  • changed message definitions
  • updated transformers
  • cleaned up

comment:54 Changed 19 months ago by jkarder

r15529: worked on new persistence

  • changed message definitions
  • updated transformers
  • worked on conversions
  • cleaned up

comment:55 Changed 17 months ago by jkarder

  • Milestone changed from HeuristicLab 4.0 to HeuristicLab 3.3.16

comment:56 Changed 15 months ago by jkarder

r15857: worked on conversions

comment:57 Changed 15 months ago by jkarder

r15878: worked on new persistence

  • updated tests

comment:58 Changed 12 months ago by jkarder

r15986: worked on new persistence

  • removed type version
  • removed conversions
  • execute AfterDeserialization hooks after all objects have been populated
  • worked on transformers

comment:59 Changed 9 months ago by jkarder

r16210: worked on new persistence

  • added cache for different attributes
  • refactored StorableTypeBoxTransformer
  • implemented paths for Storable members
  • implemented more unit tests

comment:60 Changed 6 months ago by jkarder

r16396: worked on new persistence

  • improved EnumerableBoxTransformer
  • fixed output path

comment:61 Changed 6 months ago by jkarder

Since we moved the implementation of the new persistence to GitLab, we will continue as follows:

  • Create fresh branch from trunk and continue working from there
  • Upgrade all projects to .NET 4.6.1
  • Publish and reference NuGet package of new persistence

comment:62 Changed 6 months ago by jkarder

r16450: deleted branch

comment:63 Changed 6 months ago by jkarder

r16451: created branch for reintegration of new persistence

comment:64 Changed 6 months ago by jkarder

r16452: upgraded projects to .NET 4.6.1

comment:65 Changed 6 months ago by jkarder

r16453: updated year of copyrights

comment:66 Changed 6 months ago by jkarder

r16454: added nuget references to HEAL.Fossil (which has not been published yet)

comment:67 Changed 6 months ago by jkarder

r16462: worked on reintegration of new persistence

  • added nuget references to HEAL.Fossil
  • added StorableType attributes to many classes
  • changed signature of StorableConstructors
  • removed some classes in old persistence
  • removed some unnecessary usings

comment:68 Changed 6 months ago by gkronber

I'm currently not able to build the branch using VS2017 but it works with the Build.ps script.

Version 0, edited 6 months ago by gkronber (next)

comment:69 Changed 6 months ago by gkronber

r16465: fixed issue when building with Build.ps and VS 2017 when packages must be restored

Seemingly the "." is necessary so that nuget (or msbuild?) produces a correct path. I have no idea.

Btw. the docs on nuget are a complete mess.

comment:70 Changed 6 months ago by gkronber

r16466: removed unit tests which are moved to the HEAL.Fossil repository

comment:71 follow-up: Changed 6 months ago by abeham

I want to point out that Fossil is a registered trademark (trademark search: TM View).

comment:72 Changed 6 months ago by gkronber

r16467: added unit tests for converting all samples (open old format, store new format)

comment:73 Changed 6 months ago by gkronber

r16468: added the necessary StorableType attributes in HeuristicLab.Algorithms.DataAnalysis

comment:74 in reply to: ↑ 71 ; follow-up: Changed 6 months ago by gkronber

Replying to abeham:

I want to point out that Fossil is a registered trademark (trademark search: TM View).

Any other suggestions? What about Gajost or Kawakr :)

Last edited 6 months ago by gkronber (previous) (diff)

comment:75 Changed 6 months ago by gkronber

r16469: Adapted VS code fixes:

  • using HeuristicLab.Persistence -> using HEAL.Fossil
  • Made missing StorableCtor a Warning
  • Ignore StorableCtor for structs and interfaces

comment:76 Changed 6 months ago by gkronber

r16470: fixed usings

comment:77 in reply to: ↑ 74 ; follow-ups: Changed 6 months ago by abeham

Replying to gkronber:

Replying to abeham:

I want to point out that Fossil is a registered trademark (trademark search: TM View).

Any other suggestions? What about Gajost or Kawakr :)

Kawakr doesn't sound too bad. Other suggestions:

  • Parrot (repeats things)
  • Dolphin (long memory)
  • Beagle (Darwin's famous ship)
  • Serio (or SerioSharp) - SERialize Input Output
  • Rializer (seRIALIZER)
  • Alizer (or just Alize)
  • Revasio (reverse of saver + io)
  • StoreOfTheFiles
  • Inout
  • Bittersweet
  • Bytespot
  • Bytepack

comment:78 in reply to: ↑ 77 Changed 6 months ago by gkronber

Replying to abeham:

Kawakr doesn't sound too bad. Other suggestions:

  • Parrot (repeats things)
  • Dolphin (long memory)
  • Beagle (Darwin's famous ship)
  • Serio (or SerioSharp) - SERialize Input Output
  • Rializer (seRIALIZER)
  • Alizer (or just Alize)
  • Revasio (reverse of saver + io)
  • StoreOfTheFiles
  • Inout
  • Bittersweet
  • Bytespot
  • Bytepack

Good stuff, I especially like: Alize or Alizer (HEAL.Alizer) and Rialize or Rializer.

comment:79 Changed 6 months ago by gkronber

r16471:16472: checked all projects from top to HeuristicLab.Common and added StorableType attributes.

Last edited 6 months ago by gkronber (previous) (diff)

comment:80 Changed 6 months ago by gkronber

Here is a list of all folders and their status concerning the StorableType attribute check:

  • HeuristicLab (not necessary)
  • HeuristicLab.Algorithms.ALPS (done)
  • HeuristicLab.Algorithms.Benchmarks (done)
  • HeuristicLab.Algorithms.Benchmarks.Views (done)
  • HeuristicLab.Algorithms.CMAEvolutionStrategy (done)
  • HeuristicLab.Algorithms.DataAnalysis (done)
  • HeuristicLab.Algorithms.DataAnalysis.Glmnet (done)
  • HeuristicLab.Algorithms.DataAnalysis.Views (done)
  • HeuristicLab.Algorithms.EvolutionStrategy (done)
  • HeuristicLab.Algorithms.GeneticAlgorithm (done)
  • HeuristicLab.Algorithms.GradientDescent (done)
  • HeuristicLab.Algorithms.LocalSearch (done)
  • HeuristicLab.Algorithms.MOCMAEvolutionStrategy (done)
  • HeuristicLab.Algorithms.NSGA2 (done)
  • HeuristicLab.Algorithms.OffspringSelectionEvolutionStrategy (done)
  • HeuristicLab.Algorithms.OffspringSelectionGeneticAlgorithm (done)
  • HeuristicLab.Algorithms.ParameterlessPopulationPyramid (done)
  • HeuristicLab.Algorithms.ParticleSwarmOptimization (done)
  • HeuristicLab.Algorithms.RandomSearch (done)
  • HeuristicLab.Algorithms.RAPGA (done)
  • HeuristicLab.Algorithms.ScatterSearch (done)
  • HeuristicLab.Algorithms.SimulatedAnnealing (done)
  • HeuristicLab.Algorithms.TabuSearch (done)
  • HeuristicLab.Algorithms.VariableNeighborhoodSearch (done)
  • HeuristicLab.Analysis (done)
  • HeuristicLab.Analysis.Statistics.Views (done)
  • HeuristicLab.Analysis.Views (done)
  • HeuristicLab.Clients.Access (skipped)
  • HeuristicLab.Clients.Access.Administration (skipped)
  • HeuristicLab.Clients.Access.Views (skipped)
  • HeuristicLab.Clients.Common (skipped)
  • HeuristicLab.Clients.Hive (skipped)
  • HeuristicLab.Clients.Hive.Administrator (skipped)
  • HeuristicLab.Clients.Hive.JobManager (skipped)
  • HeuristicLab.Clients.Hive.Slave
  • HeuristicLab.Clients.Hive.Slave.App
  • HeuristicLab.Clients.Hive.Slave.ConsoleClient
  • HeuristicLab.Clients.Hive.Slave.TrayIcon
  • HeuristicLab.Clients.Hive.Slave.Views
  • HeuristicLab.Clients.Hive.Slave.WindowsService
  • HeuristicLab.Clients.Hive.Views (skipped)
  • HeuristicLab.Clients.OKB (done)
  • HeuristicLab.Clients.OKB.Views (done)
  • HeuristicLab.CodeEditor (skipped)
  • HeuristicLab.Collections (done)
  • HeuristicLab.Common (done)
  • HeuristicLab.Common.Resources (done)
  • HeuristicLab.Core (done)
  • HeuristicLab.Core.Views (done)
  • HeuristicLab.Data (done)
  • HeuristicLab.Data.Views (done)
  • HeuristicLab.DataPreprocessing (done)
  • HeuristicLab.DataPreprocessing.Views (done)
  • HeuristicLab.DebugEngine (done)
  • HeuristicLab.DebugEngine.Views (done)
  • HeuristicLab.Encodings.BinaryVectorEncoding (done)
  • HeuristicLab.Encodings.IntegerVectorEncoding (done)
  • HeuristicLab.Encodings.LinearLinkageEncoding (done)
  • HeuristicLab.Encodings.PermutationEncoding (done)
  • HeuristicLab.Encodings.PermutationEncoding.Views (done)
  • HeuristicLab.Encodings.RealVectorEncoding (done)
  • HeuristicLab.Encodings.ScheduleEncoding (done)
  • HeuristicLab.Encodings.ScheduleEncoding.Views (done)
  • HeuristicLab.Encodings.SymbolicExpressionTreeEncoding (done)
  • HeuristicLab.Encodings.SymbolicExpressionTreeEncoding.Views (done)
  • HeuristicLab.ExtLibs (not necessary)
  • HeuristicLab.Hive (skipped)
  • HeuristicLab.MainForm (done)
  • HeuristicLab.MainForm.WindowsForms (done)
  • HeuristicLab.Operators (done)
  • HeuristicLab.Operators.Programmable (done)
  • HeuristicLab.Operators.Programmable.Views (done)
  • HeuristicLab.Operators.Views (done)
  • HeuristicLab.Operators.Views.GraphVisualization (done)
  • HeuristicLab.Operators.Views.GraphVisualization.Views (done)
  • HeuristicLab.Optimization (done)
  • HeuristicLab.Optimization.Operators (done)
  • HeuristicLab.Optimization.Views (done)
  • HeuristicLab.Optimizer (done)
  • HeuristicLab.ParallelEngine (done)
  • HeuristicLab.ParallelEngine.Views (done)
  • HeuristicLab.Parameters (done)
  • HeuristicLab.Parameters.Views (done)
  • HeuristicLab.Persistence (skipped)
  • HeuristicLab.Persistence.GUI (not necessary)
  • HeuristicLab.PluginInfrastructure (not necessary)
  • HeuristicLab.Problems.Binary (done)
  • HeuristicLab.Problems.BinPacking (done)
  • HeuristicLab.Problems.BinPacking.Views (done)
  • HeuristicLab.Problems.DataAnalysis (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Classification (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Classification.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Regression (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Regression.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Trading (done)
  • HeuristicLab.Problems.DataAnalysis.Trading.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Views (done)
  • HeuristicLab.Problems.ExternalEvaluation (done)
  • HeuristicLab.Problems.ExternalEvaluation.GP (done)
  • HeuristicLab.Problems.ExternalEvaluation.Matlab (done)
  • HeuristicLab.Problems.ExternalEvaluation.Scilab (done)
  • HeuristicLab.Problems.ExternalEvaluation.Views (done)
  • HeuristicLab.Problems.GeneticProgramming (done)
  • HeuristicLab.Problems.GeneticProgramming.Views (done)
  • HeuristicLab.Problems.GrammaticalEvolution (done)
  • HeuristicLab.Problems.GraphColoring (done)
  • HeuristicLab.Problems.Instances (not necessary)
  • HeuristicLab.Problems.Instances.CordeauGQAP (not necessary)
  • HeuristicLab.Problems.Instances.DataAnalysis (done)
  • HeuristicLab.Problems.Instances.DataAnalysis.Views (not necessary)
  • HeuristicLab.Problems.Instances.DIMACS (not necessary)
  • HeuristicLab.Problems.Instances.ElloumiCTAP (not necessary)
  • HeuristicLab.Problems.Instances.Orienteering (not necessary)
  • HeuristicLab.Problems.Instances.QAPLIB (not necessary)
  • HeuristicLab.Problems.Instances.Scheduling (not necessary)
  • HeuristicLab.Problems.Instances.TSPLIB (not necessary)
  • HeuristicLab.Problems.Instances.TSPLIB.Views (not necessary)
  • HeuristicLab.Problems.Instances.VehicleRouting (not necessary)
  • HeuristicLab.Problems.Instances.VehicleRouting.Views (not necessary)
  • HeuristicLab.Problems.Instances.Views (not necessary)
  • HeuristicLab.Problems.Knapsack (done)
  • HeuristicLab.Problems.Knapsack.Views (done)
  • HeuristicLab.Problems.LinearAssignment (done)
  • HeuristicLab.Problems.LinearAssignment.Views (done)
  • HeuristicLab.Problems.NK (done)
  • HeuristicLab.Problems.Orienteering (done)
  • HeuristicLab.Problems.Orienteering.Views (done)
  • HeuristicLab.Problems.ParameterOptimization (done)
  • HeuristicLab.Problems.Programmable (done)
  • HeuristicLab.Problems.Programmable.Views (done)
  • HeuristicLab.Problems.PTSP (done)
  • HeuristicLab.Problems.PTSP.Views (done)
  • HeuristicLab.Problems.QuadraticAssignment (done)
  • HeuristicLab.Problems.QuadraticAssignment.Algorithms (done)
  • HeuristicLab.Problems.QuadraticAssignment.Views (done)
  • HeuristicLab.Problems.Scheduling (done)
  • HeuristicLab.Problems.Scheduling.Views (done)
  • HeuristicLab.Problems.TestFunctions (done)
  • HeuristicLab.Problems.TestFunctions.MultiObjective (done)
  • HeuristicLab.Problems.TestFunctions.Views (done)
  • HeuristicLab.Problems.TravelingSalesman (done)
  • HeuristicLab.Problems.TravelingSalesman.Views (done)
  • HeuristicLab.Problems.VehicleRouting (done)
  • HeuristicLab.Problems.VehicleRouting.Views (done)
  • HeuristicLab.Random (done)
  • HeuristicLab.Scripting (done) TODO: Test
  • HeuristicLab.Scripting.Views (done)
  • HeuristicLab.Selection (done)
  • HeuristicLab.SequentialEngine (done)
  • HeuristicLab.Services.Access
  • HeuristicLab.Services.Access.DataAccess
  • HeuristicLab.Services.Deployment
  • HeuristicLab.Services.Deployment.DataAccess
  • HeuristicLab.Services.Hive
  • HeuristicLab.Services.Hive.DataAccess
  • HeuristicLab.Services.Hive.JanitorService
  • HeuristicLab.Services.Hive.Web
  • HeuristicLab.Services.OKB
  • HeuristicLab.Services.OKB.DataAccess
  • HeuristicLab.Services.WebApp
  • HeuristicLab.Services.WebApp.Maintenance
  • HeuristicLab.Services.WebApp.Statistics
  • HeuristicLab.Services.WebApp.Status
  • HeuristicLab.Tests (not necessary)
  • HeuristicLab.Tracing
  • HeuristicLab.Visualization.ChartControlsExtensions
Last edited 6 months ago by gkronber (previous) (diff)

comment:81 Changed 6 months ago by abeham

Need help?

comment:82 Changed 6 months ago by gkronber

Not working on it right now. Waiting for Johannes to commit the fixed code fix to make it easier.

comment:83 in reply to: ↑ 77 Changed 6 months ago by swagner

Replying to abeham:

Replying to gkronber:

Replying to abeham:

I want to point out that Fossil is a registered trademark (trademark search: TM View).

Any other suggestions? What about Gajost or Kawakr :)

Kawakr doesn't sound too bad. Other suggestions:

  • Parrot (repeats things)
  • Dolphin (long memory)
  • Beagle (Darwin's famous ship)
  • Serio (or SerioSharp) - SERialize Input Output
  • Rializer (seRIALIZER)
  • Alizer (or just Alize)
  • Revasio (reverse of saver + io)
  • StoreOfTheFiles
  • Inout
  • Bittersweet
  • Bytespot
  • Bytepack

Some additional suggestions:

  • Hippocampus (part of the brain responsible for creating memory)
  • Freezer
  • Cryo
  • Iceberg
  • O2D (objects to disk)
  • Mnemonic
  • Memorizer
  • Recall
  • Stash
  • ObjectStore
  • ObjectVault
  • Boxing
  • ObjectBox

By the way, I also like Dolphin, Serio and Alizer ...

gkronber and jkarder, please choose a name you like.

comment:84 Changed 6 months ago by gkronber

r16473: changed name and type of parameter for StorableConstructor

comment:85 Changed 6 months ago by gkronber

r16474: checked and added StorableType attribute in projects up to HeuristicLab.MainForm

comment:86 Changed 6 months ago by jkarder

r16475: added analyzer and fix for adapting storable ctor signature

comment:87 Changed 6 months ago by gkronber

r16476: checked and added StorableType attribute in projects up to HeuristicLab.Problems.GraphColoring

comment:88 Changed 6 months ago by gkronber

r16477: checked and added StorableType attribute in projects up to HeuristicLab.Problems.VRP.Views

comment:89 Changed 6 months ago by gkronber

r16479: checked and added StorableType attribute projects included in HeuristicLab 3.3.sln

comment:90 Changed 6 months ago by gkronber

OKB uses XmlGenerator / XmlParser in several places. How do we handle persisted data in OKB?

comment:91 Changed 6 months ago by gkronber

ExternalEvaluation.GP uses XmlGenerator to serialize trees to byte[] for transmission to external module for evaluation. How do we proceed?

comment:92 Changed 6 months ago by gkronber

r16480: use new persistence in ContentManager (with fallback), ClipBoard (with fallback), as well as for deep-cloning in Scripting and for Hive Persistence (without fallback & incomplete)

comment:93 Changed 6 months ago by abeham

Good questions. I would prefer starting over with a fresh OKB database and re-uploading any data that is still available in files. I would be fine with that and would be less hassle than trying to convert it. But we should ask others if they've got any data there worthy of being preserved. I also think something like a convergence graph shouldn't be binary data, but that's a design problem. The other binary data is, I think, algorithms and problems.

Regarding ExternalEvaluation, it should be used to transmit trees and not meant to persist them. Thus, we should get away with changing that to the new persistence. However, there's also the EvaluationCache that may store solution data. Erik implemented this, we should check with him.

comment:94 Changed 6 months ago by gkronber

r16481: fixed StorableSerializer of old persistence to make the old persistence unit test succeed

comment:95 Changed 6 months ago by gkronber

  • r16482: added two custom transformers (for Fonts and Bitmaps) and registered types that are available in .NET Framework but not in .NET Standard 2.0 and added.
  • r16483: added unit test for IndexedDataTable and Point2D and made necessary fixes
  • r16482: minor cleanup in PersistenceContentManager

comment:96 follow-up: Changed 6 months ago by gkronber

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

comment:97 follow-up: Changed 6 months ago by gkronber

I still have issues with building in VS. VS complains e.g. that IEnumerable<> is defined in .netstandard and I need to add a reference, while building with Build.ps works. Sometimes restarting VS works. No idea what is happening here.

comment:98 in reply to: ↑ 97 Changed 6 months ago by gkronber

Replying to gkronber:

I still have issues with building in VS.

Notably, VS shows errors (in the error window and using intellisense) but building still succeeds.

comment:99 Changed 6 months ago by gkronber

r16458: replaced static type registration in HL.Persistence plugin with the KnownStorableTypesMap which is discovered and used by HEAL.Fossil.

As a consequence we do not need to explicitly call type registration for unit tests. This change has been necessary because on the first access of StaticCache all StorableTypes are registered. However, when the HL.Persistence plugin is loaded we have not yet loaded all plugins and as a consequence we would miss StorableTypes.

Last edited 6 months ago by gkronber (previous) (diff)

comment:100 Changed 6 months ago by gkronber

r16486: added license headers

comment:101 Changed 6 months ago by gkronber

Current state:

  • all persistence unit tests (old and new) succeed. This includes a unit test that loads all samples in old format and stores them in the new format.
  • GA - TSP sample can be loaded and executed in HL. Pause / store / load / continue using the new format works and produces the same results.
  • Clipboard store / load works with new new format.
  • load, start, pause & store works for all samples
  • Complete implementation of HEAL.Attic, publish to nuget, update the package source for HEAL.Attic
  • Change ExternalEvaluation and EvaluationCache to use new serialization format (existing users must update their evaluation code). added a new serializer and marked the old serializer as obsolete.
  • Adapt Hive to use new persistence format.
  • Final decision on the name and license.
  • Re-enable cancellation of serialization for new persistence.

TODO:

  • unit test for for pause, store, load and continue both and compare results of both algorithms
  • test C# scripting and programmable problem (regarding cloning using persistence in VariableStore)
  • re-deploy all HiveWorkers.
  • Implement a converter for all serialized data in OKB and convert the data. Change OKB to use new persistence.
  • Apply code fixes to all Addons and test.
  • Performance comparison old vs. new
  • Change unit tests for samples to use new persistence
Last edited 4 months ago by gkronber (previous) (diff)

comment:102 Changed 6 months ago by gkronber

r16487: merged r16478 (HEAD) from trunk to branch

comment:103 in reply to: ↑ 96 ; follow-up: Changed 6 months ago by mkommend

Replying to gkronber:

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

Is there a possibility that we could solve this?

It is a major drawback to use a software on one PC and then not be able to open created files on another machine, even more so as this is not obvious to the user.

What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?

It would still be preferable if the persistence is .NET version / implementation agnostic.

comment:104 in reply to: ↑ 103 Changed 6 months ago by gkronber

Replying to mkommend:

Replying to gkronber:

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

Is there a possibility that we could solve this?

Yes.

What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?

There would be different release bundles one for .NET Core and one for .NET Framework. The user would be aware which version of HL he or she is running.

It would still be preferable if the persistence is .NET version / implementation agnostic.

Yes. This means we need to restrict ourselves to types and members which are available in both .NET Core and .NET Framework (which .NET Standard types).

Currently, we have the following additional types registered for HL which are not part of .NET Standard:

  • System.Drawing.Font, (used in DataTableVisualProperties, ScatterplotVisualProperties. Maybe we can remove persistence of fonts?
  • System.Drawing.FontStyle, (for serialization of Font)
  • System.Drawing.GraphicsUnit, (for serialization of Font)
  • System.Drawing.Bitmap, (necessary for ItemImage)
  • System.Collections.StructuralEqualityComparer (not sure if this is actually necessary)

comment:105 Changed 6 months ago by abeham

Good work, Gabriel!

Maybe we can escape the System.Drawing problems by replacing all references to the System.Drawing assembly in the GAC with the respective nuget package.

I can't compile the branch either, but due to missing HEAL.Fossil.

comment:106 Changed 6 months ago by gkronber

As discussed with swagner:

  • OKB: swagner will try to implement a converter for existing data
  • Hive: The HiveClient should use the new format, HiveWorkers should try to deserialize using the new format and if it fails fallback to the old format.

comment:107 Changed 6 months ago by gkronber

jkarder suggests: heal.attic (attic = dachbodenspeicher)

comment:108 Changed 6 months ago by pfleck

The types HeuristicLab.Analysis.IDataRow and HeuristicLab.Analysis.IDataTable<TDataRow> were added to the trunk in r16527. Please also add a StorableTypeAttribute to them after merging the Persistence back to the trunk.

comment:109 Changed 5 months ago by gkronber

r16529: merged r16491:16528 from trunk to persistence branch

comment:110 Changed 5 months ago by gkronber

r16530: made the necessary changes for the new Persistence for types which have been added in the trunk

comment:111 Changed 5 months ago by gkronber

r16539: use nuget package System.Drawing.Common to allow cross-platform persistence for types from System.Drawing.

comment:112 Changed 5 months ago by gkronber

r16551: merged r16542:16543 from trunk to branch

comment:113 Changed 5 months ago by gkronber

r16552: added a new serializer for trees in the ExternalEvaluation.GP plugin and marked the serializer using the old XML format as obsolete

comment:114 Changed 5 months ago by gkronber

r16553:16554 for Hive serialization: added collection of serialized types and fall-back to old persistence format on deserialization. ITask must be a StorableType

Last edited 5 months ago by gkronber (previous) (diff)

comment:115 Changed 5 months ago by gkronber

r16555: added a unit test to try to serialize samples which have been running for 20secs.

comment:116 Changed 5 months ago by jkarder

r16558: updated Fossil version and PersistenceUtil for hive

comment:117 Changed 5 months ago by jkarder

r16559: renamed Fossil to Attic and set version to 1.0.0-pre01

comment:118 Changed 5 months ago by gkronber

r16562: deleted unnecessary app.config file

comment:119 Changed 5 months ago by gkronber

r16563: re-added support for cancellation of save operation

comment:120 Changed 5 months ago by gkronber

r16564: added HEAL.Attic.dll, Google.Protobuf.dll and System.Drawing.Common.dll as PluginFiles provided by HeuristicLab.Persistence

comment:121 Changed 5 months ago by gkronber

r16565: merged changes from PersistenceOverhaul branch (r16451:16564) into trunk

comment:122 Changed 5 months ago by gkronber

r16566: updated HeuristicLab.DataImporter addon for new Persistence (introduced in 3.3.16)

comment:123 Changed 5 months ago by gkronber

r16567: changed StorableConstructors and added StorableType attributes in HeuristicLab.DataImporter addon

comment:124 Changed 5 months ago by gkronber

r16568: updated HEAL.Attic version (and reference the net461 specific build

comment:125 Changed 5 months ago by gkronber

r16569: updated version of HEAL.Attic for the HeuristicLab.Tests solution

comment:126 Changed 5 months ago by gkronber

r16570: updated HEAL.Attic version and use the net461 version in HeuristicLab.DataImporter addon

Last edited 5 months ago by gkronber (previous) (diff)

comment:127 Changed 5 months ago by gkronber

r16571: several fixes based on failing unit tests

  • adapted unit tests that check StorableConstructors
  • missing translations of StorableClass -> StorableType
  • missing StorableConstructors
  • missing or unecessary PluginDependencies / ProjectReferences

comment:128 Changed 5 months ago by gkronber

  • r16572: removed unnecessary references / plugin dependencies in Hive.Slave projects
  • r16573: changed HeuristicLab.FLA addon to compile with new HL.Persistence

comment:129 Changed 5 months ago by gkronber

r16574: changed HeuristicLab.MetaOptimization addon to compile with new HL.Persistence

comment:130 Changed 5 months ago by gkronber

r16575: changed HeuristicLab.BioBoost addon to compile with new HL.Persistence

comment:131 Changed 5 months ago by gkronber

r16576: changed Fossil -> Attic in VS Code Fix

Last edited 5 months ago by gkronber (previous) (diff)

comment:132 Changed 5 months ago by gkronber

r16577: added missing StorableType attributes for interfaces in HeuristicLab.FitnessLandspaceAnalysis

comment:133 Changed 5 months ago by gkronber

r16578: changed unit tests which load old samples and try storing and loading using the new format to recreate all sample files first

comment:134 Changed 5 months ago by gkronber

r16579: try fall-back to old persistence for all exceptions that might occur in persistence

comment:135 Changed 5 months ago by gkronber

r16584: added missing StorableType attributes

comment:136 Changed 5 months ago by gkronber

r16585: Update HEAL.Attic to pre03

comment:137 Changed 5 months ago by gkronber

r16594: added unit tests for profiling old vs new persistence with samples

comment:138 Changed 5 months ago by gkronber

r16595: changed a comment

comment:139 Changed 4 months ago by jkarder

r16625: switched projects in the following solutions from packages.config format to PackageReference format (where possible)

  • HeuristicLab 3.3
  • HeuristicLab 3.3 Tests
  • HeuristicLab 3.3 Services

comment:140 Changed 4 months ago by gkronber

  • Version changed from branch to trunk

comment:141 Changed 4 months ago by gkronber

TODO:

  • update format of samples on StartPage as well as serializer used for these samples.
  • check setters of storable properties whether they assume that the recieved value is already initialized. This is not the case for e.g. StorableOperators in Encoding.cs line 44.
Last edited 4 months ago by gkronber (previous) (diff)

comment:142 Changed 4 months ago by gkronber

r16658: created an extlibs wrapper plugin for the HEAL.Attic nuget package (and updated to 1.0.0-pre4)

comment:143 Changed 4 months ago by gkronber

r16659: svn:ignore

comment:144 Changed 3 months ago by gkronber

r16672: removed nuget package reference missed in r16658

comment:145 Changed 3 months ago by jkarder

r16704 (merged): fixed RunGaussianProcessRegressionSample unit test which failed since we migrated to .NET 4.6.1 in r16452

Last edited 7 weeks ago by jkarder (previous) (diff)

comment:146 Changed 3 months ago by mkommend

r16706 (merged): Adapted sample unit test GPSymRegWithOS in x64 bit mode due to the occuring quality differences when compiling with .Net 4.6.1.

Careful: trunk commit!

Last edited 7 weeks ago by jkarder (previous) (diff)

comment:147 Changed 3 months ago by gkronber

r16707: Update HEAL.Attic to release version 1.0.

comment:148 Changed 3 months ago by abeham

In r16565 usings were removed from the C# Script template and the programmable problem templates in the trunk. You have to disable remove and sort usings when saving those files, because the usings are put there only for convenience. I reverted those changes in these files:

r16729: Readded missing usings in code template files

comment:149 Changed 2 months ago by mkommend

Could you please remove the deprecated warnings introduced by the new persistence in the trunk solution?

Warning CS0618 'StorableAttribute.AllowOneWay' is obsolete: 'Use OldName instead.'

This occurs multiple times throughout the whole solution.

comment:150 Changed 2 months ago by gkronber

We should probably fix all occurrences in HL trunk.

comment:151 Changed 2 months ago by abeham

r16792: Extracted Google.Protobuf into own extlib plugin and added both plugin dependency and assembly reference (build order) in HeuristicLab.Attic

I also tried to get rid of the other protobuf implementation that we currently use in external evaluation, but this proved to be a more complex and time consuming task (cf. #3003).

comment:152 Changed 2 months ago by gkronber

r16796: removed usages of AllowOneWay

Last edited 2 months ago by gkronber (previous) (diff)

comment:153 Changed 2 months ago by gkronber

r16797: fixed comparison logic bug introduced in previous commit

comment:154 Changed 2 months ago by gkronber

r16799: made another fix concerning handling of OldName instead of AllowOneWay in old persistence (fix unit tests).

comment:155 Changed 2 months ago by abeham

I think we should add a unit test to check for duplicate GUIDs in StorableType. I just copy and pasted a class and forgot to change the GUID. I think this may be a common error people will make.

Additionally, Michael and I discussed that we should ensure, also by test, that all types have a StorableType attribute. Potentially, generate a list once of all current types, then add exceptions and require that all types not covered in the exceptions are StorableTypes.

comment:156 Changed 2 months ago by abeham

r16819: changed references to PluginInfrastructure from file reference to project reference

comment:157 Changed 7 weeks ago by gkronber

r16906: disable compiler warning for old persistence unit tests

comment:158 Changed 7 weeks ago by gkronber

r16907: Show more info in the exception dialog when persistence fails

comment:159 Changed 7 weeks ago by gkronber

r16908: added unit test to check for duplicate GUIDs

comment:160 Changed 7 weeks ago by gkronber

r16933: changed exception handling / retry with old persistence format in PersistenceContentManager

comment:161 Changed 7 weeks ago by jkarder

r16934 (stable commit): merged r16704 and r16706 into stable

Last edited 7 weeks ago by jkarder (previous) (diff)

comment:162 Changed 6 weeks ago by jkarder

r16945: changed deserialization in PersistenceContentManager

  • added method to check if old persistence can open a file
  • used this method to determine which persistence to use

comment:163 Changed 5 weeks ago by abeham

Is there any more work necessary? Otherwise, I think we should move this to review. I would propose that mkommend and myself schedule some time to do a review within the next two weeks together.

comment:164 Changed 4 weeks ago by gkronber

r16992: Use ContentManager in Clipboard and StartPage. Show info when objects cannot be restored.

comment:165 Changed 4 weeks ago by gkronber

r16994: Update plugin dependencies and references for HL.DataImporter for new persistence

comment:166 Changed 4 weeks ago by gkronber

r16995: Update plugin dependencies and references for HL.FLA for new persistence

comment:167 Changed 4 weeks ago by gkronber

r16996: Update plugin dependencies and references for HL.MetaOptimization for new persistence

comment:168 Changed 4 weeks ago by gkronber

r16997: Update plugin dependencies and references for Core.Views and Optimizer for new persistence

comment:169 Changed 6 days ago by jkarder

r17014: Update HEAL.Attic to version 1.1.0

comment:170 Changed 6 days ago by abeham

r17015: fixed bug in the clipboard when it contains objects not of type IStorableContent

comment:171 Changed 6 days ago by abeham

r17017: removed branch

comment:172 Changed 6 days ago by abeham

r17019: fixed unit test to check for duplicate GUIDs (includes NonDiscoverable types)

comment:173 Changed 6 days ago by mkommend

r17020: Adapted storable properties from SymbolicExpressionGrammarBase to use new temporary fields and populate the real data fields in the after deserialization hook.

comment:174 Changed 6 days ago by mkommend

r17021: Adapted all unit tests to use attic instead of the xml persistence. This affects all sample unit tests, the test resources, script unit tests and some general unit tests.

comment:175 Changed 6 days ago by mkommend

r17022: Changed samples to attic format and adapted startpage to load the samples with attic without the creation of temporary files.

comment:176 Changed 6 days ago by mkommend

r17023: Added forgotten plugin dependency to HEAL.Attic in HL.Optimizer plugin file.

comment:177 Changed 5 days ago by mkommend

r17024: Adapted unit test for loading samples.

Note: See TracTickets for help on using tickets.