Opened 3 years ago

Last modified 3 days ago

#2520 accepted enhancement

Persistence Overhaul

Reported by: swagner Owned by: gkronber
Priority: high Milestone: HeuristicLab 3.3.16
Component: Persistence Version: branch
Keywords: Cc:

Description

The HeuristicLab persistence layer should be overhauled for HeuristicLab 4.0.

Change History (111)

comment:1 Changed 3 years ago by swagner

r13323: Created new branch for persistence overhaul

comment:2 Changed 3 years ago by swagner

r13324: Removed new empty branch in order to create branch of full trunk

comment:3 Changed 3 years ago by swagner

r13325: Created branch PersistenceOverhaul of trunk r13321.

comment:4 Changed 3 years ago by swagner

r13326: Created plugin for new persistence implementation.

comment:5 Changed 3 years ago by swagner

r13347: Worked on new persistence implementation.

comment:6 Changed 3 years ago by swagner

r13358: Worked on new persistence implementation.

comment:7 Changed 3 years ago by swagner

r13367: Worked on new persistence implementation.

comment:8 Changed 3 years ago by ascheibe

r13368 added guids to storable classes

comment:9 Changed 3 years ago by ascheibe

r13370 fixed unit tests

comment:10 Changed 3 years ago by swagner

r13375: Worked on new persistence implementation.

comment:11 Changed 3 years ago by ascheibe

r13386

  • fixed duplicate guids
  • adapted/added unit tests for new persistence

comment:12 Changed 3 years ago by swagner

r13387: Worked on new persistence implementation.

comment:13 Changed 3 years ago by ascheibe

r13407 worked on type and enum transformers

comment:14 Changed 3 years ago by ascheibe

r13409

  • added profiling to unit tests
  • added more unit tests

comment:15 follow-up: Changed 3 years ago by ascheibe

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

comment:16 in reply to: ↑ 15 ; follow-up: Changed 3 years ago by gkronber

Replying to ascheibe:

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.

comment:17 Changed 3 years ago by ascheibe

r13410 increased repetitions to get more realistic results

comment:18 in reply to: ↑ 16 Changed 3 years ago by ascheibe

Replying to gkronber:

Replying to ascheibe:

The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:

Performance Report for <TestBoolArray>b__28_0: 

Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407
Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271
Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8
Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569
Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851

Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.

Yeah, I know that this is not significant, but we don't have composite transformers yet, so this is all we can do at the moment ;-)

comment:19 Changed 2 years ago by gkronber

r14537: worked on persistence

comment:20 Changed 2 years ago by jkarder

r14549: worked on persistence

comment:21 Changed 2 years ago by jkarder

r14594: worked on persistence

comment:22 Changed 23 months ago by gkronber

r14711

  • renamed StorableClass -> StorableType
  • changed persistence to use GUIDs instead of type names

comment:23 Changed 23 months ago by gkronber

r14712: added GUIDs for (almost) all interface types (probably still too many) also added newlines at end of all files

comment:24 Changed 23 months ago by gkronber

r14714 added missing StorableType attributes and registered more types. All samples can be deserialized and serialized now

comment:25 Changed 23 months ago by gkronber

r14731: added tool program which adds StorableType attributes to interfaces and enums

comment:26 Changed 23 months ago by gkronber

r14739: added a sketch for a unit test for conversion (and set svn:ignore properties)

comment:27 Changed 22 months ago by gkronber

r14771: added versions to storable types and implemented conversion unit test

comment:28 Changed 21 months ago by gkronber

r14919: created a branch for reintegration

comment:29 Changed 21 months ago by gkronber

r14920: deleted branch of trunk

comment:30 Changed 21 months ago by gkronber

r14921: created a new branch from stable for reintegration

comment:31 Changed 21 months ago by gkronber

r14922: deleted branch again

comment:32 Changed 21 months ago by gkronber

r14923: created branch from trunk (again)

comment:33 Changed 21 months ago by gkronber

r14924: copied relevant files from persistence overhaul branch

comment:34 Changed 21 months ago by gkronber

r14925: moved storable attributes (construction)

comment:35 Changed 21 months ago by gkronber

r14927: changed all usages of StorableClass to use StorableType with an auto-generated GUID (did not add StorableType to other type definitions yet)

comment:36 Changed 21 months ago by gkronber

r14928: added unit tests and fixed all test cases for old persistence

comment:37 Changed 21 months ago by gkronber

r14929: fixed all unit tests for new persistence - loading & storing all samples

comment:38 Changed 21 months ago by gkronber

r14930: changed optimizer to use new persistence per default

comment:39 Changed 21 months ago by gkronber

r14931: do not store graph visualization info for the operatorgraph

comment:40 Changed 21 months ago by gkronber

r14932: check if results are the same for the GA TSP sample test

comment:41 Changed 21 months ago by gkronber

r14933: added a code fix for generating StorableTypeAttributes

comment:42 Changed 21 months ago by gkronber

r14934: svn:ignore

comment:43 Changed 21 months ago by gkronber

r14935: support persistence for OperatorGraphVisualizationInfo

comment:44 Changed 21 months ago by gkronber

  • Owner changed from swagner to gkronber
  • Status changed from new to accepted

comment:45 Changed 21 months ago by jkarder

r14985: worked on persistence code fix

  • split existing analyzer and code fix
  • added analyzer and code fix for missing storable ctors

comment:46 Changed 21 months ago by jkarder

r14989: worked on persistence code fix

  • added SequentialFixAllProvider

comment:47 Changed 21 months ago by jkarder

r14990: worked on persistence code fix

  • added missing UnitTests.cs

comment:48 Changed 20 months ago by gkronber

  • r15018: introduced StorableConstructorFlag type for StorableConstructors
  • r15020: fixed unit tests checking for exceptions

comment:49 Changed 20 months ago by jkarder

r15022: removed ColorTransformer for now as it only handled the color's argb values (StructTransformer will handle more, e.g. IsEmpty)

comment:50 Changed 19 months ago by gkronber

r15034: improved conversions

comment:51 Changed 19 months ago by gkronber

r15035: made some changes related to renaming of storable members

comment:52 Changed 14 months ago by jkarder

r15501: added protobuf version 3.5.0 and changed proto definition

comment:53 Changed 14 months ago by jkarder

r15509: worked on new persistence

  • changed message definitions
  • updated transformers
  • cleaned up

comment:54 Changed 13 months ago by jkarder

r15529: worked on new persistence

  • changed message definitions
  • updated transformers
  • worked on conversions
  • cleaned up

comment:55 Changed 12 months ago by jkarder

  • Milestone changed from HeuristicLab 4.0 to HeuristicLab 3.3.16

comment:56 Changed 10 months ago by jkarder

r15857: worked on conversions

comment:57 Changed 10 months ago by jkarder

r15878: worked on new persistence

  • updated tests

comment:58 Changed 7 months ago by jkarder

r15986: worked on new persistence

  • removed type version
  • removed conversions
  • execute AfterDeserialization hooks after all objects have been populated
  • worked on transformers

comment:59 Changed 4 months ago by jkarder

r16210: worked on new persistence

  • added cache for different attributes
  • refactored StorableTypeBoxTransformer
  • implemented paths for Storable members
  • implemented more unit tests

comment:60 Changed 5 weeks ago by jkarder

r16396: worked on new persistence

  • improved EnumerableBoxTransformer
  • fixed output path

comment:61 Changed 4 weeks ago by jkarder

Since we moved the implementation of the new persistence to GitLab, we will continue as follows:

  • Create fresh branch from trunk and continue working from there
  • Upgrade all projects to .NET 4.6.1
  • Publish and reference NuGet package of new persistence

comment:62 Changed 4 weeks ago by jkarder

r16450: deleted branch

comment:63 Changed 4 weeks ago by jkarder

r16451: created branch for reintegration of new persistence

comment:64 Changed 4 weeks ago by jkarder

r16452: upgraded projects to .NET 4.6.1

comment:65 Changed 4 weeks ago by jkarder

r16453: updated year of copyrights

comment:66 Changed 4 weeks ago by jkarder

r16454: added nuget references to HEAL.Fossil (which has not been published yet)

comment:67 Changed 3 weeks ago by jkarder

r16462: worked on reintegration of new persistence

  • added nuget references to HEAL.Fossil
  • added StorableType attributes to many classes
  • changed signature of StorableConstructors
  • removed some classes in old persistence
  • removed some unnecessary usings

comment:68 Changed 3 weeks ago by gkronber

I'm currently not able to build the branch using VS2017 or the Build.ps script.

The error messages are:

...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error : Could not find a part of the path '...2520_PersistenceReintegration\ \packages\HEAL.Fossil.1.0.0'.

...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error : Could not find a part of the path '...2520_PersistenceReintegration\ \packages\Google.Protobuf.3.6.1'.

...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error MSB3073: The command ""...2520_PersistenceReintegration\.nuget\NuGet.exe" install "...\2520_PersistenceReintegration\HeuristicLab.Persistence\3.3\packages.config" -source "" -NonInteractive -RequireConsent -solutionDir "...\2520_PersistenceReintegration\ "" exited with code 1.

Two observations:

  • there is a space in the path for the packages
  • the package sources list is empty.
Last edited 3 weeks ago by gkronber (previous) (diff)

comment:69 Changed 3 weeks ago by gkronber

r16465: fixed issue when building with Build.ps and VS 2017 when packages must be restored

Seemingly the "." is necessary so that nuget (or msbuild?) produces a correct path. I have no idea.

Btw. the docs on nuget are a complete mess.

comment:70 Changed 3 weeks ago by gkronber

r16466: removed unit tests which are moved to the HEAL.Fossil repository

comment:71 follow-up: Changed 3 weeks ago by abeham

I want to point out that Fossil is a registered trademark (trademark search: TM View).

comment:72 Changed 3 weeks ago by gkronber

r16467: added unit tests for converting all samples (open old format, store new format)

comment:73 Changed 3 weeks ago by gkronber

r16468: added the necessary StorableType attributes in HeuristicLab.Algorithms.DataAnalysis

comment:74 in reply to: ↑ 71 ; follow-up: Changed 3 weeks ago by gkronber

Replying to abeham:

I want to point out that Fossil is a registered trademark (trademark search: TM View).

Any other suggestions? What about Gajost or Kawakr :)

Last edited 3 weeks ago by gkronber (previous) (diff)

comment:75 Changed 3 weeks ago by gkronber

r16469: Adapted VS code fixes:

  • using HeuristicLab.Persistence -> using HEAL.Fossil
  • Made missing StorableCtor a Warning
  • Ignore StorableCtor for structs and interfaces

comment:76 Changed 3 weeks ago by gkronber

r16470: fixed usings

comment:77 in reply to: ↑ 74 ; follow-ups: Changed 3 weeks ago by abeham

Replying to gkronber:

Replying to abeham:

I want to point out that Fossil is a registered trademark (trademark search: TM View).

Any other suggestions? What about Gajost or Kawakr :)

Kawakr doesn't sound too bad. Other suggestions:

  • Parrot (repeats things)
  • Dolphin (long memory)
  • Beagle (Darwin's famous ship)
  • Serio (or SerioSharp) - SERialize Input Output
  • Rializer (seRIALIZER)
  • Alizer (or just Alize)
  • Revasio (reverse of saver + io)
  • StoreOfTheFiles
  • Inout
  • Bittersweet
  • Bytespot
  • Bytepack

comment:78 in reply to: ↑ 77 Changed 3 weeks ago by gkronber

Replying to abeham:

Kawakr doesn't sound too bad. Other suggestions:

  • Parrot (repeats things)
  • Dolphin (long memory)
  • Beagle (Darwin's famous ship)
  • Serio (or SerioSharp) - SERialize Input Output
  • Rializer (seRIALIZER)
  • Alizer (or just Alize)
  • Revasio (reverse of saver + io)
  • StoreOfTheFiles
  • Inout
  • Bittersweet
  • Bytespot
  • Bytepack

Good stuff, I especially like: Alize or Alizer (HEAL.Alizer) and Rialize or Rializer.

comment:79 Changed 3 weeks ago by gkronber

r16471: checked all projects from top to HeuristicLab.Common and added StorableType attributes.

Version 0, edited 3 weeks ago by gkronber (next)

comment:80 Changed 3 weeks ago by gkronber

Here is a list of all folders and their status concerning the StorableType attribute check:

  • HeuristicLab (not necessary)
  • HeuristicLab.Algorithms.ALPS (done)
  • HeuristicLab.Algorithms.Benchmarks (done)
  • HeuristicLab.Algorithms.Benchmarks.Views (done)
  • HeuristicLab.Algorithms.CMAEvolutionStrategy (done)
  • HeuristicLab.Algorithms.DataAnalysis (done)
  • HeuristicLab.Algorithms.DataAnalysis.Glmnet (done)
  • HeuristicLab.Algorithms.DataAnalysis.Views (done)
  • HeuristicLab.Algorithms.EvolutionStrategy (done)
  • HeuristicLab.Algorithms.GeneticAlgorithm (done)
  • HeuristicLab.Algorithms.GradientDescent (done)
  • HeuristicLab.Algorithms.LocalSearch (done)
  • HeuristicLab.Algorithms.MOCMAEvolutionStrategy (done)
  • HeuristicLab.Algorithms.NSGA2 (done)
  • HeuristicLab.Algorithms.OffspringSelectionEvolutionStrategy (done)
  • HeuristicLab.Algorithms.OffspringSelectionGeneticAlgorithm (done)
  • HeuristicLab.Algorithms.ParameterlessPopulationPyramid (done)
  • HeuristicLab.Algorithms.ParticleSwarmOptimization (done)
  • HeuristicLab.Algorithms.RandomSearch (done)
  • HeuristicLab.Algorithms.RAPGA (done)
  • HeuristicLab.Algorithms.ScatterSearch (done)
  • HeuristicLab.Algorithms.SimulatedAnnealing (done)
  • HeuristicLab.Algorithms.TabuSearch (done)
  • HeuristicLab.Algorithms.VariableNeighborhoodSearch (done)
  • HeuristicLab.Analysis (done)
  • HeuristicLab.Analysis.Statistics.Views (done)
  • HeuristicLab.Analysis.Views (done)
  • HeuristicLab.Clients.Access (skipped)
  • HeuristicLab.Clients.Access.Administration (skipped)
  • HeuristicLab.Clients.Access.Views (skipped)
  • HeuristicLab.Clients.Common (skipped)
  • HeuristicLab.Clients.Hive (skipped)
  • HeuristicLab.Clients.Hive.Administrator (skipped)
  • HeuristicLab.Clients.Hive.JobManager (skipped)
  • HeuristicLab.Clients.Hive.Slave
  • HeuristicLab.Clients.Hive.Slave.App
  • HeuristicLab.Clients.Hive.Slave.ConsoleClient
  • HeuristicLab.Clients.Hive.Slave.TrayIcon
  • HeuristicLab.Clients.Hive.Slave.Views
  • HeuristicLab.Clients.Hive.Slave.WindowsService
  • HeuristicLab.Clients.Hive.Views (skipped)
  • HeuristicLab.Clients.OKB (done)
  • HeuristicLab.Clients.OKB.Views (done)
  • HeuristicLab.CodeEditor (skipped)
  • HeuristicLab.Collections (done)
  • HeuristicLab.Common (done)
  • HeuristicLab.Common.Resources (done)
  • HeuristicLab.Core (done)
  • HeuristicLab.Core.Views (done)
  • HeuristicLab.Data (done)
  • HeuristicLab.Data.Views (done)
  • HeuristicLab.DataPreprocessing (done)
  • HeuristicLab.DataPreprocessing.Views (done)
  • HeuristicLab.DebugEngine (done)
  • HeuristicLab.DebugEngine.Views (done)
  • HeuristicLab.Encodings.BinaryVectorEncoding (done)
  • HeuristicLab.Encodings.IntegerVectorEncoding (done)
  • HeuristicLab.Encodings.LinearLinkageEncoding (done)
  • HeuristicLab.Encodings.PermutationEncoding (done)
  • HeuristicLab.Encodings.PermutationEncoding.Views (done)
  • HeuristicLab.Encodings.RealVectorEncoding (done)
  • HeuristicLab.Encodings.ScheduleEncoding (done)
  • HeuristicLab.Encodings.ScheduleEncoding.Views (done)
  • HeuristicLab.Encodings.SymbolicExpressionTreeEncoding (done)
  • HeuristicLab.Encodings.SymbolicExpressionTreeEncoding.Views (done)
  • HeuristicLab.ExtLibs (not necessary)
  • HeuristicLab.Hive (skipped)
  • HeuristicLab.MainForm (done)
  • HeuristicLab.MainForm.WindowsForms (done)
  • HeuristicLab.Operators (done)
  • HeuristicLab.Operators.Programmable (done)
  • HeuristicLab.Operators.Programmable.Views (done)
  • HeuristicLab.Operators.Views (done)
  • HeuristicLab.Operators.Views.GraphVisualization (done)
  • HeuristicLab.Operators.Views.GraphVisualization.Views (done)
  • HeuristicLab.Optimization (done)
  • HeuristicLab.Optimization.Operators (done)
  • HeuristicLab.Optimization.Views (done)
  • HeuristicLab.Optimizer (done)
  • HeuristicLab.ParallelEngine (done)
  • HeuristicLab.ParallelEngine.Views (done)
  • HeuristicLab.Parameters (done)
  • HeuristicLab.Parameters.Views (done)
  • HeuristicLab.Persistence (skipped)
  • HeuristicLab.Persistence.GUI (not necessary)
  • HeuristicLab.PluginInfrastructure (not necessary)
  • HeuristicLab.Problems.Binary (done)
  • HeuristicLab.Problems.BinPacking (done)
  • HeuristicLab.Problems.BinPacking.Views (done)
  • HeuristicLab.Problems.DataAnalysis (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Classification (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Classification.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Regression (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Regression.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Symbolic.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Trading (done)
  • HeuristicLab.Problems.DataAnalysis.Trading.Views (done)
  • HeuristicLab.Problems.DataAnalysis.Views (done)
  • HeuristicLab.Problems.ExternalEvaluation (done)
  • HeuristicLab.Problems.ExternalEvaluation.GP (done)
  • HeuristicLab.Problems.ExternalEvaluation.Matlab (done)
  • HeuristicLab.Problems.ExternalEvaluation.Scilab (done)
  • HeuristicLab.Problems.ExternalEvaluation.Views (done)
  • HeuristicLab.Problems.GeneticProgramming (done)
  • HeuristicLab.Problems.GeneticProgramming.Views (done)
  • HeuristicLab.Problems.GrammaticalEvolution (done)
  • HeuristicLab.Problems.GraphColoring (done)
  • HeuristicLab.Problems.Instances (not necessary)
  • HeuristicLab.Problems.Instances.CordeauGQAP (not necessary)
  • HeuristicLab.Problems.Instances.DataAnalysis (done)
  • HeuristicLab.Problems.Instances.DataAnalysis.Views (not necessary)
  • HeuristicLab.Problems.Instances.DIMACS (not necessary)
  • HeuristicLab.Problems.Instances.ElloumiCTAP (not necessary)
  • HeuristicLab.Problems.Instances.Orienteering (not necessary)
  • HeuristicLab.Problems.Instances.QAPLIB (not necessary)
  • HeuristicLab.Problems.Instances.Scheduling (not necessary)
  • HeuristicLab.Problems.Instances.TSPLIB (not necessary)
  • HeuristicLab.Problems.Instances.TSPLIB.Views (not necessary)
  • HeuristicLab.Problems.Instances.VehicleRouting (not necessary)
  • HeuristicLab.Problems.Instances.VehicleRouting.Views (not necessary)
  • HeuristicLab.Problems.Instances.Views (not necessary)
  • HeuristicLab.Problems.Knapsack (done)
  • HeuristicLab.Problems.Knapsack.Views (done)
  • HeuristicLab.Problems.LinearAssignment (done)
  • HeuristicLab.Problems.LinearAssignment.Views (done)
  • HeuristicLab.Problems.NK (done)
  • HeuristicLab.Problems.Orienteering (done)
  • HeuristicLab.Problems.Orienteering.Views (done)
  • HeuristicLab.Problems.ParameterOptimization (done)
  • HeuristicLab.Problems.Programmable (done)
  • HeuristicLab.Problems.Programmable.Views (done)
  • HeuristicLab.Problems.PTSP (done)
  • HeuristicLab.Problems.PTSP.Views (done)
  • HeuristicLab.Problems.QuadraticAssignment (done)
  • HeuristicLab.Problems.QuadraticAssignment.Algorithms (done)
  • HeuristicLab.Problems.QuadraticAssignment.Views (done)
  • HeuristicLab.Problems.Scheduling (done)
  • HeuristicLab.Problems.Scheduling.Views (done)
  • HeuristicLab.Problems.TestFunctions (done)
  • HeuristicLab.Problems.TestFunctions.MultiObjective (done)
  • HeuristicLab.Problems.TestFunctions.Views (done)
  • HeuristicLab.Problems.TravelingSalesman (done)
  • HeuristicLab.Problems.TravelingSalesman.Views (done)
  • HeuristicLab.Problems.VehicleRouting (done)
  • HeuristicLab.Problems.VehicleRouting.Views (done)
  • HeuristicLab.Random (done)
  • HeuristicLab.Scripting (done) TODO: Test
  • HeuristicLab.Scripting.Views (done)
  • HeuristicLab.Selection (done)
  • HeuristicLab.SequentialEngine (done)
  • HeuristicLab.Services.Access
  • HeuristicLab.Services.Access.DataAccess
  • HeuristicLab.Services.Deployment
  • HeuristicLab.Services.Deployment.DataAccess
  • HeuristicLab.Services.Hive
  • HeuristicLab.Services.Hive.DataAccess
  • HeuristicLab.Services.Hive.JanitorService
  • HeuristicLab.Services.Hive.Web
  • HeuristicLab.Services.OKB
  • HeuristicLab.Services.OKB.DataAccess
  • HeuristicLab.Services.WebApp
  • HeuristicLab.Services.WebApp.Maintenance
  • HeuristicLab.Services.WebApp.Statistics
  • HeuristicLab.Services.WebApp.Status
  • HeuristicLab.Tests (not necessary)
  • HeuristicLab.Tracing
  • HeuristicLab.Visualization.ChartControlsExtensions
Last edited 3 weeks ago by gkronber (previous) (diff)

comment:81 Changed 3 weeks ago by abeham

Need help?

comment:82 Changed 3 weeks ago by gkronber

Not working on it right now. Waiting for Johannes to commit the fixed code fix to make it easier.

comment:83 in reply to: ↑ 77 Changed 3 weeks ago by swagner

Replying to abeham:

Replying to gkronber:

Replying to abeham:

I want to point out that Fossil is a registered trademark (trademark search: TM View).

Any other suggestions? What about Gajost or Kawakr :)

Kawakr doesn't sound too bad. Other suggestions:

  • Parrot (repeats things)
  • Dolphin (long memory)
  • Beagle (Darwin's famous ship)
  • Serio (or SerioSharp) - SERialize Input Output
  • Rializer (seRIALIZER)
  • Alizer (or just Alize)
  • Revasio (reverse of saver + io)
  • StoreOfTheFiles
  • Inout
  • Bittersweet
  • Bytespot
  • Bytepack

Some additional suggestions:

  • Hippocampus (part of the brain responsible for creating memory)
  • Freezer
  • Cryo
  • Iceberg
  • O2D (objects to disk)
  • Mnemonic
  • Memorizer
  • Recall
  • Stash
  • ObjectStore
  • ObjectVault
  • Boxing
  • ObjectBox

By the way, I also like Dolphin, Serio and Alizer ...

gkronber and jkarder, please choose a name you like.

comment:84 Changed 3 weeks ago by gkronber

r16473: changed name and type of parameter for StorableConstructor

comment:85 Changed 3 weeks ago by gkronber

r16474: checked and added StorableType attribute in projects up to HeuristicLab.MainForm

comment:86 Changed 3 weeks ago by jkarder

r16475: added analyzer and fix for adapting storable ctor signature

comment:87 Changed 3 weeks ago by gkronber

r16476: checked and added StorableType attribute in projects up to HeuristicLab.Problems.GraphColoring

comment:88 Changed 3 weeks ago by gkronber

r16477: checked and added StorableType attribute in projects up to HeuristicLab.Problems.VRP.Views

comment:89 Changed 3 weeks ago by gkronber

r16479: checked and added StorableType attribute projects included in HeuristicLab 3.3.sln

comment:90 Changed 3 weeks ago by gkronber

OKB uses XmlGenerator / XmlParser in several places. How do we handle persisted data in OKB?

comment:91 Changed 3 weeks ago by gkronber

ExternalEvaluation.GP uses XmlGenerator to serialize trees to byte[] for transmission to external module for evaluation. How do we proceed?

comment:92 Changed 3 weeks ago by gkronber

r16480: use new persistence in ContentManager (with fallback), ClipBoard (with fallback), as well as for deep-cloning in Scripting and for Hive Persistence (without fallback & incomplete)

comment:93 Changed 3 weeks ago by abeham

Good questions. I would prefer starting over with a fresh OKB database and re-uploading any data that is still available in files. I would be fine with that and would be less hassle than trying to convert it. But we should ask others if they've got any data there worthy of being preserved. I also think something like a convergence graph shouldn't be binary data, but that's a design problem. The other binary data is, I think, algorithms and problems.

Regarding ExternalEvaluation, it should be used to transmit trees and not meant to persist them. Thus, we should get away with changing that to the new persistence. However, there's also the EvaluationCache that may store solution data. Erik implemented this, we should check with him.

comment:94 Changed 3 weeks ago by gkronber

r16481: fixed StorableSerializer of old persistence to make the old persistence unit test succeed

comment:95 Changed 3 weeks ago by gkronber

  • r16482: added two custom transformers (for Fonts and Bitmaps) and registered types that are available in .NET Framework but not in .NET Standard 2.0 and added.
  • r16483: added unit test for IndexedDataTable and Point2D and made necessary fixes
  • r16482: minor cleanup in PersistenceContentManager

comment:96 follow-up: Changed 3 weeks ago by gkronber

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

comment:97 follow-up: Changed 3 weeks ago by gkronber

I still have issues with building in VS. VS complains e.g. that IEnumerable<> is defined in .netstandard and I need to add a reference, while building with Build.ps works. Sometimes restarting VS works. No idea what is happening here.

comment:98 in reply to: ↑ 97 Changed 3 weeks ago by gkronber

Replying to gkronber:

I still have issues with building in VS.

Notably, VS shows errors (in the error window and using intellisense) but building still succeeds.

comment:99 Changed 3 weeks ago by gkronber

r16458: replaced static type registration in HL.Persistence plugin with the KnownStorableTypesMap which is discovered and used by HEAL.Fossil.

As a consequence we do not need to explicitly call type registration for unit tests. This change has been necessary because on the first access of StaticCache all StorableTypes are registered. However, when the HL.Persistence plugin is loaded we have not yet loaded all plugins and as a consequence we would miss StorableTypes.

Last edited 3 weeks ago by gkronber (previous) (diff)

comment:100 Changed 3 weeks ago by gkronber

r16486: added license headers

comment:101 Changed 3 weeks ago by gkronber

Current state:

  • all persistence unit tests (old and new) succeed. This includes a unit test that loads all samples in old format and stores them in the new format.
  • GA - TSP sample can be loaded and executed in HL. Pause / store / load / continue using the new format works and produces the same results.
  • Clipboard store / load works with new new format.

TODO:

  • test pause / store / load / continue for all samples
  • test C# scripting and programmable problem (regarding cloning using persistence in VariableStore)
  • Complete implementation of HEAL.Fossil, publish to nuget, update the package source for HEAL.Fossil
  • Fix issues with package restore & building in VS
  • Change ExternalEvaluation and EvaluationCache to use new serialization format (existing users must update their evaluation code).
  • Adapt Hive to use new persistence format. This also means we need to re-deploy all HiveWorkers.
  • Implement a converter for all serialized data in OKB and convert the data. Change OKB to use new persistence.
  • Apply code fixes to all Addons and test.
  • Final decision on the name and license.
  • Performance comparison old vs. new

comment:102 Changed 3 weeks ago by gkronber

r16487: merged r16478 (HEAD) from trunk to branch

comment:103 in reply to: ↑ 96 ; follow-up: Changed 3 weeks ago by mkommend

Replying to gkronber:

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

Is there a possibility that we could solve this?

It is a major drawback to use a software on one PC and then not be able to open created files on another machine, even more so as this is not obvious to the user.

What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?

It would still be preferable if the persistence is .NET version / implementation agnostic.

comment:104 in reply to: ↑ 103 Changed 3 weeks ago by gkronber

Replying to mkommend:

Replying to gkronber:

Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).

Is there a possibility that we could solve this?

Yes.

What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?

There would be different release bundles one for .NET Core and one for .NET Framework. The user would be aware which version of HL he or she is running.

It would still be preferable if the persistence is .NET version / implementation agnostic.

Yes. This means we need to restrict ourselves to types and members which are available in both .NET Core and .NET Framework (which .NET Standard types).

Currently, we have the following additional types registered for HL which are not part of .NET Standard:

  • System.Drawing.Font, (used in DataTableVisualProperties, ScatterplotVisualProperties. Maybe we can remove persistence of fonts?
  • System.Drawing.FontStyle, (for serialization of Font)
  • System.Drawing.GraphicsUnit, (for serialization of Font)
  • System.Drawing.Bitmap, (necessary for ItemImage)
  • System.Collections.StructuralEqualityComparer (not sure if this is actually necessary)

comment:105 Changed 3 weeks ago by abeham

Good work, Gabriel!

Maybe we can escape the System.Drawing problems by replacing all references to the System.Drawing assembly in the GAC with the respective nuget package.

I can't compile the branch either, but due to missing HEAL.Fossil.

comment:106 Changed 2 weeks ago by gkronber

As discussed with swagner:

  • OKB: swagner will try to implement a converter for existing data
  • Hive: The HiveClient should use the new format, HiveWorkers should try to deserialize using the new format and if it fails fallback to the old format.

comment:107 Changed 2 weeks ago by gkronber

jkarder suggests: heal.attic (attic = dachbodenspeicher)

comment:108 Changed 11 days ago by pfleck

The types HeuristicLab.Analysis.IDataRow and HeuristicLab.Analysis.IDataTable<TDataRow> were added to the trunk in r16527. Please also add a StorableTypeAttribute to them after merging the Persistence back to the trunk.

comment:109 Changed 11 days ago by gkronber

r16529: merged r16491:16528 from trunk to persistence branch

comment:110 Changed 11 days ago by gkronber

r16530: made the necessary changes for the new Persistence for types which have been added in the trunk

comment:111 Changed 3 days ago by gkronber

r16539: use nuget package System.Drawing.Common to allow cross-platform persistence for types from System.Drawing.

Note: See TracTickets for help on using tickets.