#2520 closed enhancement (done)
Persistence Overhaul
Reported by: | swagner | Owned by: | mkommend |
---|---|---|---|
Priority: | high | Milestone: | HeuristicLab 3.3.16 |
Component: | Persistence | Version: | trunk |
Keywords: | merged | Cc: |
Description
The HeuristicLab persistence layer should be overhauled for HeuristicLab 4.0.
Attachments (1)
Change History (189)
comment:1 Changed 8 years ago by swagner
comment:2 Changed 8 years ago by swagner
r13324: Removed new empty branch in order to create branch of full trunk
comment:3 Changed 8 years ago by swagner
comment:4 Changed 8 years ago by swagner
r13326: Created plugin for new persistence implementation.
comment:5 Changed 8 years ago by swagner
r13347: Worked on new persistence implementation.
comment:6 Changed 8 years ago by swagner
r13358: Worked on new persistence implementation.
comment:7 Changed 8 years ago by swagner
r13367: Worked on new persistence implementation.
comment:8 Changed 8 years ago by ascheibe
r13368 added guids to storable classes
comment:9 Changed 8 years ago by ascheibe
r13370 fixed unit tests
comment:10 Changed 8 years ago by swagner
r13375: Worked on new persistence implementation.
comment:11 Changed 8 years ago by ascheibe
- fixed duplicate guids
- adapted/added unit tests for new persistence
comment:12 Changed 8 years ago by swagner
r13387: Worked on new persistence implementation.
comment:13 Changed 8 years ago by ascheibe
r13407 worked on type and enum transformers
comment:14 Changed 8 years ago by ascheibe
- added profiling to unit tests
- added more unit tests
comment:15 follow-up: ↓ 16 Changed 8 years ago by ascheibe
The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:
Performance Report for <TestBoolArray>b__28_0: Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407 Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271 Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8 Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569 Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851
comment:16 in reply to: ↑ 15 ; follow-up: ↓ 18 Changed 8 years ago by gkronber
Replying to ascheibe:
The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:
Performance Report for <TestBoolArray>b__28_0: Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407 Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271 Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8 Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569 Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851
Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.
comment:17 Changed 8 years ago by ascheibe
r13410 increased repetitions to get more realistic results
comment:18 in reply to: ↑ 16 Changed 8 years ago by ascheibe
Replying to gkronber:
Replying to ascheibe:
The Persistence4 unit tests now generate an output where you can see different performance measures, e.g.:
Performance Report for <TestBoolArray>b__28_0: Avg. old vs. new time for serializing a file: 00:00:00.0060189 / 00:00:00.0008129; Factor: 7,40452962958407 Avg. old vs. new time for deserializing a file: 00:00:00.0048589 / 00:00:00.0001852; Factor: 26,2386866832271 Avg. old vs. new file size (in bytes): 594 / 55; Factor: 10,8 Avg. old vs. new memory consumption for serializing a file (in bytes): 59849,6 / 24600; Factor: 2,43291056910569 Avg. old vs. new memory consumption for deserializing a file (in bytes): 178737,6 / 57392; Factor: 3,11432952327851Micro-benchmark alarm. I'd be more interested to see how long it takes to store and load a full experiment with results.
Yeah, I know that this is not significant, but we don't have composite transformers yet, so this is all we can do at the moment ;-)
comment:19 Changed 7 years ago by gkronber
r14537: worked on persistence
comment:20 Changed 7 years ago by jkarder
r14549: worked on persistence
comment:21 Changed 7 years ago by jkarder
r14594: worked on persistence
comment:22 Changed 7 years ago by gkronber
- renamed StorableClass -> StorableType
- changed persistence to use GUIDs instead of type names
comment:23 Changed 7 years ago by gkronber
r14712: added GUIDs for (almost) all interface types (probably still too many) also added newlines at end of all files
comment:24 Changed 7 years ago by gkronber
r14714 added missing StorableType attributes and registered more types. All samples can be deserialized and serialized now
comment:25 Changed 7 years ago by gkronber
r14731: added tool program which adds StorableType attributes to interfaces and enums
comment:26 Changed 7 years ago by gkronber
r14739: added a sketch for a unit test for conversion (and set svn:ignore properties)
comment:27 Changed 7 years ago by gkronber
r14771: added versions to storable types and implemented conversion unit test
comment:28 Changed 7 years ago by gkronber
r14919: created a branch for reintegration
comment:29 Changed 7 years ago by gkronber
r14920: deleted branch of trunk
comment:30 Changed 7 years ago by gkronber
r14921: created a new branch from stable for reintegration
comment:31 Changed 7 years ago by gkronber
r14922: deleted branch again
comment:32 Changed 7 years ago by gkronber
r14923: created branch from trunk (again)
comment:33 Changed 7 years ago by gkronber
r14924: copied relevant files from persistence overhaul branch
comment:34 Changed 7 years ago by gkronber
r14925: moved storable attributes (construction)
comment:35 Changed 7 years ago by gkronber
r14927: changed all usages of StorableClass to use StorableType with an auto-generated GUID (did not add StorableType to other type definitions yet)
comment:36 Changed 7 years ago by gkronber
r14928: added unit tests and fixed all test cases for old persistence
comment:37 Changed 7 years ago by gkronber
r14929: fixed all unit tests for new persistence - loading & storing all samples
comment:38 Changed 7 years ago by gkronber
r14930: changed optimizer to use new persistence per default
comment:39 Changed 7 years ago by gkronber
r14931: do not store graph visualization info for the operatorgraph
comment:40 Changed 7 years ago by gkronber
r14932: check if results are the same for the GA TSP sample test
comment:41 Changed 7 years ago by gkronber
r14933: added a code fix for generating StorableTypeAttributes
comment:42 Changed 7 years ago by gkronber
r14934: svn:ignore
comment:43 Changed 7 years ago by gkronber
r14935: support persistence for OperatorGraphVisualizationInfo
comment:44 Changed 7 years ago by gkronber
- Owner changed from swagner to gkronber
- Status changed from new to accepted
comment:45 Changed 7 years ago by jkarder
r14985: worked on persistence code fix
- split existing analyzer and code fix
- added analyzer and code fix for missing storable ctors
comment:46 Changed 7 years ago by jkarder
r14989: worked on persistence code fix
- added SequentialFixAllProvider
comment:47 Changed 7 years ago by jkarder
r14990: worked on persistence code fix
- added missing UnitTests.cs
comment:48 Changed 7 years ago by gkronber
comment:49 Changed 7 years ago by jkarder
r15022: removed ColorTransformer for now as it only handled the color's argb values (StructTransformer will handle more, e.g. IsEmpty)
comment:50 Changed 7 years ago by gkronber
r15034: improved conversions
comment:51 Changed 7 years ago by gkronber
r15035: made some changes related to renaming of storable members
comment:52 Changed 6 years ago by jkarder
r15501: added protobuf version 3.5.0 and changed proto definition
comment:53 Changed 6 years ago by jkarder
r15509: worked on new persistence
- changed message definitions
- updated transformers
- cleaned up
comment:54 Changed 6 years ago by jkarder
r15529: worked on new persistence
- changed message definitions
- updated transformers
- worked on conversions
- cleaned up
comment:55 Changed 6 years ago by jkarder
- Milestone changed from HeuristicLab 4.0 to HeuristicLab 3.3.16
comment:56 Changed 6 years ago by jkarder
r15857: worked on conversions
comment:57 Changed 6 years ago by jkarder
r15878: worked on new persistence
- updated tests
comment:58 Changed 6 years ago by jkarder
r15986: worked on new persistence
- removed type version
- removed conversions
- execute AfterDeserialization hooks after all objects have been populated
- worked on transformers
comment:59 Changed 5 years ago by jkarder
r16210: worked on new persistence
- added cache for different attributes
- refactored StorableTypeBoxTransformer
- implemented paths for Storable members
- implemented more unit tests
comment:60 Changed 5 years ago by jkarder
r16396: worked on new persistence
- improved EnumerableBoxTransformer
- fixed output path
comment:61 Changed 5 years ago by jkarder
Since we moved the implementation of the new persistence to GitLab, we will continue as follows:
- Create fresh branch from trunk and continue working from there
- Upgrade all projects to .NET 4.6.1
- Publish and reference NuGet package of new persistence
comment:62 Changed 5 years ago by jkarder
r16450: deleted branch
comment:63 Changed 5 years ago by jkarder
r16451: created branch for reintegration of new persistence
comment:64 Changed 5 years ago by jkarder
r16452: upgraded projects to .NET 4.6.1
comment:65 Changed 5 years ago by jkarder
r16453: updated year of copyrights
comment:66 Changed 5 years ago by jkarder
r16454: added nuget references to HEAL.Fossil (which has not been published yet)
comment:67 Changed 5 years ago by jkarder
r16462: worked on reintegration of new persistence
- added nuget references to HEAL.Fossil
- added StorableType attributes to many classes
- changed signature of StorableConstructors
- removed some classes in old persistence
- removed some unnecessary usings
comment:68 Changed 5 years ago by gkronber
I'm currently not able to build the branch using VS2017 or the Build.ps script.
The error messages are:
...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error : Could not find a part of the path '...2520_PersistenceReintegration\ \packages\HEAL.Fossil.1.0.0'.
...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error : Could not find a part of the path '...2520_PersistenceReintegration\ \packages\Google.Protobuf.3.6.1'.
...2520_PersistenceReintegration\.nuget\NuGet.targets(100,9): error MSB3073: The command ""...2520_PersistenceReintegration\.nuget\NuGet.exe" install "...\2520_PersistenceReintegration\HeuristicLab.Persistence\3.3\packages.config" -source "" -NonInteractive -RequireConsent -solutionDir "...\2520_PersistenceReintegration\ "" exited with code 1.
Two observations:
- there is a space in the path for the packages
- the package sources list is empty.
comment:69 Changed 5 years ago by gkronber
r16465: fixed issue when building with Build.ps and VS 2017 when packages must be restored
Seemingly the "." is necessary so that nuget (or msbuild?) produces a correct path. I have no idea.
Btw. the docs on nuget are a complete mess.
comment:70 Changed 5 years ago by gkronber
r16466: removed unit tests which are moved to the HEAL.Fossil repository
comment:71 follow-up: ↓ 74 Changed 5 years ago by abeham
I want to point out that Fossil is a registered trademark (trademark search: TM View).
comment:72 Changed 5 years ago by gkronber
r16467: added unit tests for converting all samples (open old format, store new format)
comment:73 Changed 5 years ago by gkronber
r16468: added the necessary StorableType attributes in HeuristicLab.Algorithms.DataAnalysis
comment:74 in reply to: ↑ 71 ; follow-up: ↓ 77 Changed 5 years ago by gkronber
Replying to abeham:
I want to point out that Fossil is a registered trademark (trademark search: TM View).
Any other suggestions? What about Gajost or Kawakr :)
comment:75 Changed 5 years ago by gkronber
r16469: Adapted VS code fixes:
- using HeuristicLab.Persistence -> using HEAL.Fossil
- Made missing StorableCtor a Warning
- Ignore StorableCtor for structs and interfaces
comment:76 Changed 5 years ago by gkronber
r16470: fixed usings
comment:77 in reply to: ↑ 74 ; follow-ups: ↓ 78 ↓ 83 Changed 5 years ago by abeham
Replying to gkronber:
Replying to abeham:
I want to point out that Fossil is a registered trademark (trademark search: TM View).
Any other suggestions? What about Gajost or Kawakr :)
Kawakr doesn't sound too bad. Other suggestions:
- Parrot (repeats things)
- Dolphin (long memory)
- Beagle (Darwin's famous ship)
- Serio (or SerioSharp) - SERialize Input Output
- Rializer (seRIALIZER)
- Alizer (or just Alize)
- Revasio (reverse of saver + io)
- StoreOfTheFiles
- Inout
- Bittersweet
- Bytespot
- Bytepack
comment:78 in reply to: ↑ 77 Changed 5 years ago by gkronber
Replying to abeham:
Kawakr doesn't sound too bad. Other suggestions:
- Parrot (repeats things)
- Dolphin (long memory)
- Beagle (Darwin's famous ship)
- Serio (or SerioSharp) - SERialize Input Output
- Rializer (seRIALIZER)
- Alizer (or just Alize)
- Revasio (reverse of saver + io)
- StoreOfTheFiles
- Inout
- Bittersweet
- Bytespot
- Bytepack
Good stuff, I especially like: Alize or Alizer (HEAL.Alizer) and Rialize or Rializer.
comment:79 Changed 5 years ago by gkronber
r16471:16472: checked all projects from top to HeuristicLab.Common and added StorableType attributes.
comment:80 Changed 5 years ago by gkronber
Here is a list of all folders and their status concerning the StorableType attribute check:
HeuristicLab(not necessary)HeuristicLab.Algorithms.ALPS(done)HeuristicLab.Algorithms.Benchmarks(done)HeuristicLab.Algorithms.Benchmarks.Views(done)HeuristicLab.Algorithms.CMAEvolutionStrategy(done)HeuristicLab.Algorithms.DataAnalysis(done)HeuristicLab.Algorithms.DataAnalysis.Glmnet(done)HeuristicLab.Algorithms.DataAnalysis.Views(done)HeuristicLab.Algorithms.EvolutionStrategy(done)HeuristicLab.Algorithms.GeneticAlgorithm(done)HeuristicLab.Algorithms.GradientDescent(done)HeuristicLab.Algorithms.LocalSearch(done)HeuristicLab.Algorithms.MOCMAEvolutionStrategy(done)HeuristicLab.Algorithms.NSGA2(done)HeuristicLab.Algorithms.OffspringSelectionEvolutionStrategy(done)HeuristicLab.Algorithms.OffspringSelectionGeneticAlgorithm(done)HeuristicLab.Algorithms.ParameterlessPopulationPyramid(done)HeuristicLab.Algorithms.ParticleSwarmOptimization(done)HeuristicLab.Algorithms.RandomSearch(done)HeuristicLab.Algorithms.RAPGA(done)HeuristicLab.Algorithms.ScatterSearch(done)HeuristicLab.Algorithms.SimulatedAnnealing(done)HeuristicLab.Algorithms.TabuSearch(done)HeuristicLab.Algorithms.VariableNeighborhoodSearch(done)HeuristicLab.Analysis(done)HeuristicLab.Analysis.Statistics.Views(done)HeuristicLab.Analysis.Views(done)- HeuristicLab.Clients.Access (skipped)
- HeuristicLab.Clients.Access.Administration (skipped)
- HeuristicLab.Clients.Access.Views (skipped)
- HeuristicLab.Clients.Common (skipped)
- HeuristicLab.Clients.Hive (skipped)
- HeuristicLab.Clients.Hive.Administrator (skipped)
- HeuristicLab.Clients.Hive.JobManager (skipped)
- HeuristicLab.Clients.Hive.Slave
- HeuristicLab.Clients.Hive.Slave.App
- HeuristicLab.Clients.Hive.Slave.ConsoleClient
- HeuristicLab.Clients.Hive.Slave.TrayIcon
- HeuristicLab.Clients.Hive.Slave.Views
- HeuristicLab.Clients.Hive.Slave.WindowsService
- HeuristicLab.Clients.Hive.Views (skipped)
HeuristicLab.Clients.OKB(done)HeuristicLab.Clients.OKB.Views(done)- HeuristicLab.CodeEditor (skipped)
HeuristicLab.Collections(done)HeuristicLab.Common(done)HeuristicLab.Common.Resources(done)HeuristicLab.Core(done)HeuristicLab.Core.Views(done)HeuristicLab.Data(done)HeuristicLab.Data.Views(done)HeuristicLab.DataPreprocessing(done)HeuristicLab.DataPreprocessing.Views(done)HeuristicLab.DebugEngine(done)HeuristicLab.DebugEngine.Views(done)HeuristicLab.Encodings.BinaryVectorEncoding(done)HeuristicLab.Encodings.IntegerVectorEncoding(done)HeuristicLab.Encodings.LinearLinkageEncoding(done)HeuristicLab.Encodings.PermutationEncoding(done)HeuristicLab.Encodings.PermutationEncoding.Views(done)HeuristicLab.Encodings.RealVectorEncoding(done)HeuristicLab.Encodings.ScheduleEncoding(done)HeuristicLab.Encodings.ScheduleEncoding.Views(done)HeuristicLab.Encodings.SymbolicExpressionTreeEncoding(done)HeuristicLab.Encodings.SymbolicExpressionTreeEncoding.Views(done)HeuristicLab.ExtLibs(not necessary)- HeuristicLab.Hive (skipped)
HeuristicLab.MainForm(done)HeuristicLab.MainForm.WindowsForms(done)HeuristicLab.Operators(done)HeuristicLab.Operators.Programmable(done)HeuristicLab.Operators.Programmable.Views(done)HeuristicLab.Operators.Views(done)HeuristicLab.Operators.Views.GraphVisualization(done)HeuristicLab.Operators.Views.GraphVisualization.Views(done)HeuristicLab.Optimization(done)HeuristicLab.Optimization.Operators(done)HeuristicLab.Optimization.Views(done)HeuristicLab.Optimizer(done)HeuristicLab.ParallelEngine(done)HeuristicLab.ParallelEngine.Views(done)HeuristicLab.Parameters(done)HeuristicLab.Parameters.Views(done)- HeuristicLab.Persistence (skipped)
HeuristicLab.Persistence.GUI(not necessary)HeuristicLab.PluginInfrastructure(not necessary)HeuristicLab.Problems.Binary(done)HeuristicLab.Problems.BinPacking(done)HeuristicLab.Problems.BinPacking.Views(done)HeuristicLab.Problems.DataAnalysis(done)HeuristicLab.Problems.DataAnalysis.Symbolic(done)HeuristicLab.Problems.DataAnalysis.Symbolic.Classification(done)HeuristicLab.Problems.DataAnalysis.Symbolic.Classification.Views(done)HeuristicLab.Problems.DataAnalysis.Symbolic.Regression(done)HeuristicLab.Problems.DataAnalysis.Symbolic.Regression.Views(done)HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis(done)HeuristicLab.Problems.DataAnalysis.Symbolic.TimeSeriesPrognosis.Views(done)HeuristicLab.Problems.DataAnalysis.Symbolic.Views(done)HeuristicLab.Problems.DataAnalysis.Trading(done)HeuristicLab.Problems.DataAnalysis.Trading.Views(done)HeuristicLab.Problems.DataAnalysis.Views(done)HeuristicLab.Problems.ExternalEvaluation(done)HeuristicLab.Problems.ExternalEvaluation.GP(done)HeuristicLab.Problems.ExternalEvaluation.Matlab(done)HeuristicLab.Problems.ExternalEvaluation.Scilab(done)HeuristicLab.Problems.ExternalEvaluation.Views(done)HeuristicLab.Problems.GeneticProgramming(done)HeuristicLab.Problems.GeneticProgramming.Views(done)HeuristicLab.Problems.GrammaticalEvolution(done)HeuristicLab.Problems.GraphColoring(done)HeuristicLab.Problems.Instances(not necessary)HeuristicLab.Problems.Instances.CordeauGQAP(not necessary)HeuristicLab.Problems.Instances.DataAnalysis(done)HeuristicLab.Problems.Instances.DataAnalysis.Views(not necessary)HeuristicLab.Problems.Instances.DIMACS(not necessary)HeuristicLab.Problems.Instances.ElloumiCTAP(not necessary)HeuristicLab.Problems.Instances.Orienteering(not necessary)HeuristicLab.Problems.Instances.QAPLIB(not necessary)HeuristicLab.Problems.Instances.Scheduling(not necessary)HeuristicLab.Problems.Instances.TSPLIB(not necessary)HeuristicLab.Problems.Instances.TSPLIB.Views(not necessary)HeuristicLab.Problems.Instances.VehicleRouting(not necessary)HeuristicLab.Problems.Instances.VehicleRouting.Views(not necessary)HeuristicLab.Problems.Instances.Views(not necessary)HeuristicLab.Problems.Knapsack(done)HeuristicLab.Problems.Knapsack.Views(done)HeuristicLab.Problems.LinearAssignment(done)HeuristicLab.Problems.LinearAssignment.Views(done)HeuristicLab.Problems.NK(done)HeuristicLab.Problems.Orienteering(done)HeuristicLab.Problems.Orienteering.Views(done)HeuristicLab.Problems.ParameterOptimization(done)HeuristicLab.Problems.Programmable(done)HeuristicLab.Problems.Programmable.Views(done)HeuristicLab.Problems.PTSP(done)HeuristicLab.Problems.PTSP.Views(done)HeuristicLab.Problems.QuadraticAssignment(done)HeuristicLab.Problems.QuadraticAssignment.Algorithms(done)HeuristicLab.Problems.QuadraticAssignment.Views(done)HeuristicLab.Problems.Scheduling(done)HeuristicLab.Problems.Scheduling.Views(done)HeuristicLab.Problems.TestFunctions(done)HeuristicLab.Problems.TestFunctions.MultiObjective(done)HeuristicLab.Problems.TestFunctions.Views(done)HeuristicLab.Problems.TravelingSalesman(done)HeuristicLab.Problems.TravelingSalesman.Views(done)HeuristicLab.Problems.VehicleRouting(done)HeuristicLab.Problems.VehicleRouting.Views(done)HeuristicLab.Random(done)HeuristicLab.Scripting(done) TODO: TestHeuristicLab.Scripting.Views(done)HeuristicLab.Selection(done)HeuristicLab.SequentialEngine(done)- HeuristicLab.Services.Access
- HeuristicLab.Services.Access.DataAccess
- HeuristicLab.Services.Deployment
- HeuristicLab.Services.Deployment.DataAccess
- HeuristicLab.Services.Hive
- HeuristicLab.Services.Hive.DataAccess
- HeuristicLab.Services.Hive.JanitorService
- HeuristicLab.Services.Hive.Web
- HeuristicLab.Services.OKB
- HeuristicLab.Services.OKB.DataAccess
- HeuristicLab.Services.WebApp
- HeuristicLab.Services.WebApp.Maintenance
- HeuristicLab.Services.WebApp.Statistics
- HeuristicLab.Services.WebApp.Status
HeuristicLab.Tests(not necessary)HeuristicLab.TracingHeuristicLab.Visualization.ChartControlsExtensions
comment:81 Changed 5 years ago by abeham
Need help?
comment:82 Changed 5 years ago by gkronber
Not working on it right now. Waiting for Johannes to commit the fixed code fix to make it easier.
comment:83 in reply to: ↑ 77 Changed 5 years ago by swagner
Replying to abeham:
Replying to gkronber:
Replying to abeham:
I want to point out that Fossil is a registered trademark (trademark search: TM View).
Any other suggestions? What about Gajost or Kawakr :)
Kawakr doesn't sound too bad. Other suggestions:
- Parrot (repeats things)
- Dolphin (long memory)
- Beagle (Darwin's famous ship)
- Serio (or SerioSharp) - SERialize Input Output
- Rializer (seRIALIZER)
- Alizer (or just Alize)
- Revasio (reverse of saver + io)
- StoreOfTheFiles
- Inout
- Bittersweet
- Bytespot
- Bytepack
Some additional suggestions:
- Hippocampus (part of the brain responsible for creating memory)
- Freezer
- Cryo
- Iceberg
- O2D (objects to disk)
- Mnemonic
- Memorizer
- Recall
- Stash
- ObjectStore
- ObjectVault
- Boxing
- ObjectBox
By the way, I also like Dolphin, Serio and Alizer ...
gkronber and jkarder, please choose a name you like.
comment:84 Changed 5 years ago by gkronber
r16473: changed name and type of parameter for StorableConstructor
comment:85 Changed 5 years ago by gkronber
r16474: checked and added StorableType attribute in projects up to HeuristicLab.MainForm
comment:86 Changed 5 years ago by jkarder
r16475: added analyzer and fix for adapting storable ctor signature
comment:87 Changed 5 years ago by gkronber
r16476: checked and added StorableType attribute in projects up to HeuristicLab.Problems.GraphColoring
comment:88 Changed 5 years ago by gkronber
r16477: checked and added StorableType attribute in projects up to HeuristicLab.Problems.VRP.Views
comment:89 Changed 5 years ago by gkronber
r16479: checked and added StorableType attribute projects included in HeuristicLab 3.3.sln
comment:90 Changed 5 years ago by gkronber
OKB uses XmlGenerator / XmlParser in several places. How do we handle persisted data in OKB?
comment:91 Changed 5 years ago by gkronber
ExternalEvaluation.GP uses XmlGenerator to serialize trees to byte[] for transmission to external module for evaluation. How do we proceed?
comment:92 Changed 5 years ago by gkronber
r16480: use new persistence in ContentManager (with fallback), ClipBoard (with fallback), as well as for deep-cloning in Scripting and for Hive Persistence (without fallback & incomplete)
comment:93 Changed 5 years ago by abeham
Good questions. I would prefer starting over with a fresh OKB database and re-uploading any data that is still available in files. I would be fine with that and would be less hassle than trying to convert it. But we should ask others if they've got any data there worthy of being preserved. I also think something like a convergence graph shouldn't be binary data, but that's a design problem. The other binary data is, I think, algorithms and problems.
Regarding ExternalEvaluation, it should be used to transmit trees and not meant to persist them. Thus, we should get away with changing that to the new persistence. However, there's also the EvaluationCache that may store solution data. Erik implemented this, we should check with him.
comment:94 Changed 5 years ago by gkronber
r16481: fixed StorableSerializer of old persistence to make the old persistence unit test succeed
comment:95 Changed 5 years ago by gkronber
comment:96 follow-up: ↓ 103 Changed 5 years ago by gkronber
Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).
comment:97 follow-up: ↓ 98 Changed 5 years ago by gkronber
I still have issues with building in VS. VS complains e.g. that IEnumerable<> is defined in .netstandard and I need to add a reference, while building with Build.ps works. Sometimes restarting VS works. No idea what is happening here.
comment:98 in reply to: ↑ 97 Changed 5 years ago by gkronber
Replying to gkronber:
I still have issues with building in VS.
Notably, VS shows errors (in the error window and using intellisense) but building still succeeds.
comment:99 Changed 5 years ago by gkronber
r16458: replaced static type registration in HL.Persistence plugin with the KnownStorableTypesMap which is discovered and used by HEAL.Fossil.
As a consequence we do not need to explicitly call type registration for unit tests. This change has been necessary because on the first access of StaticCache all StorableTypes are registered. However, when the HL.Persistence plugin is loaded we have not yet loaded all plugins and as a consequence we would miss StorableTypes.
comment:100 Changed 5 years ago by gkronber
r16486: added license headers
comment:101 Changed 5 years ago by gkronber
Current state:
- all persistence unit tests (old and new) succeed. This includes a unit test that loads all samples in old format and stores them in the new format.
- GA - TSP sample can be loaded and executed in HL. Pause / store / load / continue using the new format works and produces the same results.
- Clipboard store / load works with new new format.
- load, start, pause & store works for all samples
Complete implementation of HEAL.Attic, publish to nuget, update the package source for HEAL.AtticChange ExternalEvaluation and EvaluationCache to use new serialization format (existing users must update their evaluation code).added a new serializer and marked the old serializer as obsolete.Adapt Hive to use new persistence format.Final decision on the name and license.Re-enable cancellation of serialization for new persistence.
TODO:
- unit test for for pause, store, load and continue both and compare results of both algorithms
- test C# scripting and programmable problem (regarding cloning using persistence in VariableStore)
- re-deploy all HiveWorkers.
- Implement a converter for all serialized data in OKB and convert the data. Change OKB to use new persistence.
- Apply code fixes to all Addons and test.
Performance comparison old vs. new- Change unit tests for samples to use new persistence
comment:102 Changed 5 years ago by gkronber
comment:103 in reply to: ↑ 96 ; follow-up: ↓ 104 Changed 5 years ago by mkommend
Replying to gkronber:
Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).
Is there a possibility that we could solve this?
It is a major drawback to use a software on one PC and then not be able to open created files on another machine, even more so as this is not obvious to the user.
What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?
It would still be preferable if the persistence is .NET version / implementation agnostic.
comment:104 in reply to: ↑ 103 Changed 5 years ago by gkronber
Replying to mkommend:
Replying to gkronber:
Observation: files stored with .net core version of HL cannot be opened with the .net framework version and vice versa (because framework types differ and we have different GUIDs for them).
Is there a possibility that we could solve this?
Yes.
What happens if both or event more .NET versions are installed on one machine? Which one is used for executing HL? Or is it so that HL is executed with the .NET version it is compiled against?
There would be different release bundles one for .NET Core and one for .NET Framework. The user would be aware which version of HL he or she is running.
It would still be preferable if the persistence is .NET version / implementation agnostic.
Yes. This means we need to restrict ourselves to types and members which are available in both .NET Core and .NET Framework (which .NET Standard types).
Currently, we have the following additional types registered for HL which are not part of .NET Standard:
- System.Drawing.Font, (used in DataTableVisualProperties, ScatterplotVisualProperties. Maybe we can remove persistence of fonts?
- System.Drawing.FontStyle, (for serialization of Font)
- System.Drawing.GraphicsUnit, (for serialization of Font)
- System.Drawing.Bitmap, (necessary for ItemImage)
- System.Collections.StructuralEqualityComparer (not sure if this is actually necessary)
comment:105 Changed 5 years ago by abeham
Good work, Gabriel!
Maybe we can escape the System.Drawing problems by replacing all references to the System.Drawing assembly in the GAC with the respective nuget package.
I can't compile the branch either, but due to missing HEAL.Fossil.
comment:106 Changed 5 years ago by gkronber
As discussed with swagner:
- OKB: swagner will try to implement a converter for existing data
- Hive: The HiveClient should use the new format, HiveWorkers should try to deserialize using the new format and if it fails fallback to the old format.
comment:107 Changed 5 years ago by gkronber
jkarder suggests: heal.attic (attic = dachbodenspeicher)
comment:108 Changed 5 years ago by pfleck
The types HeuristicLab.Analysis.IDataRow and HeuristicLab.Analysis.IDataTable<TDataRow> were added to the trunk in r16527. Please also add a StorableTypeAttribute to them after merging the Persistence back to the trunk.
comment:109 Changed 5 years ago by gkronber
r16529: merged r16491:16528 from trunk to persistence branch
comment:110 Changed 5 years ago by gkronber
r16530: made the necessary changes for the new Persistence for types which have been added in the trunk
comment:111 Changed 5 years ago by gkronber
r16539: use nuget package System.Drawing.Common to allow cross-platform persistence for types from System.Drawing.
comment:112 Changed 5 years ago by gkronber
r16551: merged r16542:16543 from trunk to branch
comment:113 Changed 5 years ago by gkronber
r16552: added a new serializer for trees in the ExternalEvaluation.GP plugin and marked the serializer using the old XML format as obsolete
comment:114 Changed 5 years ago by gkronber
r16553:16554 for Hive serialization: added collection of serialized types and fall-back to old persistence format on deserialization. ITask must be a StorableType
comment:115 Changed 5 years ago by gkronber
r16555: added a unit test to try to serialize samples which have been running for 20secs.
comment:116 Changed 5 years ago by jkarder
r16558: updated Fossil version and PersistenceUtil for hive
comment:117 Changed 5 years ago by jkarder
r16559: renamed Fossil to Attic and set version to 1.0.0-pre01
comment:118 Changed 5 years ago by gkronber
r16562: deleted unnecessary app.config file
comment:119 Changed 5 years ago by gkronber
r16563: re-added support for cancellation of save operation
comment:120 Changed 5 years ago by gkronber
r16564: added HEAL.Attic.dll, Google.Protobuf.dll and System.Drawing.Common.dll as PluginFiles provided by HeuristicLab.Persistence
comment:121 Changed 5 years ago by gkronber
r16565: merged changes from PersistenceOverhaul branch (r16451:16564) into trunk
comment:122 Changed 5 years ago by gkronber
r16566: updated HeuristicLab.DataImporter addon for new Persistence (introduced in 3.3.16)
comment:123 Changed 5 years ago by gkronber
r16567: changed StorableConstructors and added StorableType attributes in HeuristicLab.DataImporter addon
comment:124 Changed 5 years ago by gkronber
r16568: updated HEAL.Attic version (and reference the net461 specific build
comment:125 Changed 5 years ago by gkronber
r16569: updated version of HEAL.Attic for the HeuristicLab.Tests solution
comment:126 Changed 5 years ago by gkronber
r16570: updated HEAL.Attic version and use the net461 version in HeuristicLab.DataImporter addon
comment:127 Changed 5 years ago by gkronber
r16571: several fixes based on failing unit tests
- adapted unit tests that check StorableConstructors
- missing translations of StorableClass -> StorableType
- missing StorableConstructors
- missing or unecessary PluginDependencies / ProjectReferences
comment:128 Changed 5 years ago by gkronber
comment:129 Changed 5 years ago by gkronber
r16574: changed HeuristicLab.MetaOptimization addon to compile with new HL.Persistence
comment:130 Changed 5 years ago by gkronber
comment:131 Changed 5 years ago by gkronber
r16576: changed Fossil -> Attic in VS Code Fix
comment:132 Changed 5 years ago by gkronber
r16577: added missing StorableType attributes for interfaces in HeuristicLab.FitnessLandspaceAnalysis
comment:133 Changed 5 years ago by gkronber
r16578: changed unit tests which load old samples and try storing and loading using the new format to recreate all sample files first
comment:134 Changed 5 years ago by gkronber
r16579: try fall-back to old persistence for all exceptions that might occur in persistence
comment:135 Changed 5 years ago by gkronber
r16584: added missing StorableType attributes
comment:136 Changed 5 years ago by gkronber
r16585: Update HEAL.Attic to pre03
comment:137 Changed 5 years ago by gkronber
r16594: added unit tests for profiling old vs new persistence with samples
comment:138 Changed 5 years ago by gkronber
r16595: changed a comment
comment:139 Changed 5 years ago by jkarder
r16625: switched projects in the following solutions from packages.config format to PackageReference format (where possible)
- HeuristicLab 3.3
- HeuristicLab 3.3 Tests
- HeuristicLab 3.3 Services
comment:140 Changed 5 years ago by gkronber
- Version changed from branch to trunk
comment:141 Changed 5 years ago by gkronber
TODO:
- update format of samples on StartPage as well as serializer used for these samples.
- check setters of storable properties whether they assume that the recieved value is already initialized. This is not the case for e.g. StorableOperators in Encoding.cs line 44.
comment:142 Changed 5 years ago by gkronber
r16658: created an extlibs wrapper plugin for the HEAL.Attic nuget package (and updated to 1.0.0-pre4)
comment:143 Changed 5 years ago by gkronber
r16659: svn:ignore
comment:144 Changed 5 years ago by gkronber
comment:145 Changed 5 years ago by jkarder
comment:146 Changed 5 years ago by mkommend
r16706 (merged): Adapted sample unit test GPSymRegWithOS in x64 bit mode due to the occuring quality differences when compiling with .Net 4.6.1.
Careful: trunk commit!
comment:147 Changed 5 years ago by gkronber
r16707: Update HEAL.Attic to release version 1.0.
comment:148 Changed 5 years ago by abeham
In r16565 usings were removed from the C# Script template and the programmable problem templates in the trunk. You have to disable remove and sort usings when saving those files, because the usings are put there only for convenience. I reverted those changes in these files:
r16729: Readded missing usings in code template files
comment:149 Changed 5 years ago by mkommend
Could you please remove the deprecated warnings introduced by the new persistence in the trunk solution?
Warning CS0618 'StorableAttribute.AllowOneWay' is obsolete: 'Use OldName instead.'
This occurs multiple times throughout the whole solution.
comment:150 Changed 5 years ago by gkronber
We should probably fix all occurrences in HL trunk.
comment:151 Changed 5 years ago by abeham
r16792: Extracted Google.Protobuf into own extlib plugin and added both plugin dependency and assembly reference (build order) in HeuristicLab.Attic
I also tried to get rid of the other protobuf implementation that we currently use in external evaluation, but this proved to be a more complex and time consuming task (cf. #3003).
comment:152 Changed 5 years ago by gkronber
r16796: removed usages of AllowOneWay
comment:153 Changed 5 years ago by gkronber
r16797: fixed comparison logic bug introduced in previous commit
comment:154 Changed 5 years ago by gkronber
r16799: made another fix concerning handling of OldName instead of AllowOneWay in old persistence (fix unit tests).
comment:155 Changed 5 years ago by abeham
I think we should add a unit test to check for duplicate GUIDs in StorableType. I just copy and pasted a class and forgot to change the GUID. I think this may be a common error people will make.
Additionally, Michael and I discussed that we should ensure, also by test, that all types have a StorableType attribute. Potentially, generate a list once of all current types, then add exceptions and require that all types not covered in the exceptions are StorableTypes.
comment:156 Changed 5 years ago by abeham
r16819: changed references to PluginInfrastructure from file reference to project reference
comment:157 Changed 5 years ago by gkronber
r16906: disable compiler warning for old persistence unit tests
comment:158 Changed 5 years ago by gkronber
r16907: Show more info in the exception dialog when persistence fails
comment:159 Changed 5 years ago by gkronber
r16908: added unit test to check for duplicate GUIDs
comment:160 Changed 5 years ago by gkronber
r16933: changed exception handling / retry with old persistence format in PersistenceContentManager
comment:161 Changed 5 years ago by jkarder
comment:162 Changed 5 years ago by jkarder
r16945: changed deserialization in PersistenceContentManager
- added method to check if old persistence can open a file
- used this method to determine which persistence to use
comment:163 Changed 5 years ago by abeham
Is there any more work necessary? Otherwise, I think we should move this to review. I would propose that mkommend and myself schedule some time to do a review within the next two weeks together.
comment:164 Changed 5 years ago by gkronber
r16992: Use ContentManager in Clipboard and StartPage. Show info when objects cannot be restored.
comment:165 Changed 5 years ago by gkronber
r16994: Update plugin dependencies and references for HL.DataImporter for new persistence
comment:166 Changed 5 years ago by gkronber
r16995: Update plugin dependencies and references for HL.FLA for new persistence
comment:167 Changed 5 years ago by gkronber
r16996: Update plugin dependencies and references for HL.MetaOptimization for new persistence
comment:168 Changed 5 years ago by gkronber
r16997: Update plugin dependencies and references for Core.Views and Optimizer for new persistence
comment:169 Changed 5 years ago by jkarder
r17014: Update HEAL.Attic to version 1.1.0
comment:170 Changed 5 years ago by abeham
r17015: fixed bug in the clipboard when it contains objects not of type IStorableContent
comment:171 Changed 5 years ago by abeham
r17017: removed branch
comment:172 Changed 5 years ago by abeham
r17019: fixed unit test to check for duplicate GUIDs (includes NonDiscoverable types)
comment:173 Changed 5 years ago by mkommend
r17020: Adapted storable properties from SymbolicExpressionGrammarBase to use new temporary fields and populate the real data fields in the after deserialization hook.
comment:174 Changed 5 years ago by mkommend
r17021: Adapted all unit tests to use attic instead of the xml persistence. This affects all sample unit tests, the test resources, script unit tests and some general unit tests.
comment:175 Changed 5 years ago by mkommend
r17022: Changed samples to attic format and adapted startpage to load the samples with attic without the creation of temporary files.
comment:176 Changed 5 years ago by mkommend
r17023: Added forgotten plugin dependency to HEAL.Attic in HL.Optimizer plugin file.
comment:177 Changed 5 years ago by mkommend
r17024: Adapted unit test for loading samples.
comment:178 Changed 5 years ago by mkommend
Changed 5 years ago by abeham
Patch that alters retry with old persistence, to be applied when HEAL.Attic 1.2 is available.
comment:179 Changed 5 years ago by abeham
TODO: 5. ExtLibs / SimSharp (folder) 7. Persistence.GUI (folder) 8. Persistence 8.1 Default / Xml / XmlGenerator.cs 9. DataAnalysis.Symbolic.Regression 9.1 MultiObjective / PearsonRSquaredAverageSimilarityEvaluator.cs 10. DataAnalysis.Symbolic: 10.1. Analyzers / SymbolicDataAnalysisBuildingBlockAnalyzer.cs 10.2 Converters / DerivativeCalculator.cs 10.3 Crossovers / SymbolicDataAnalysisExpressionDiversityPreservingCrossover.cs 10.4 Hashing (folder TC) 10.5 csproj file 10.6 Interpreter / SymbolicDataAnalysisExpressionTreeBatchInterpreter.cs 10.7 Interpreter / SymbolicDataAnalysisExpressionTreeNativeInterpreter.cs 10.8 packages.config 10.9 Symbols / Absolute.cs 10.10 Symbols / AnalyticQuotient.cs 10.11 Symbols / Cube.cs 10.12 Symbols / CubeRoot.cs 11. Tests 11.1. DataAnalysis.Symbolic / DeriveTest.cs DONE: 1. Algorithms.DataAnalysis(.Views) 1.1 LinearRegressionModelView.cs -> #2892 / 16449 1.2 LinearRegressionModelView.Designer.cs -> #2892 / 16449 1.3 Linear / LinearRegressionModel.cs -> #2892 / 16389, 16415, 16448, 16449 2. Core 2.1 PersistenceContentManager.cs 3. Data 3.1 TriangularMatrix.cs -> #2919 / 15931, 15932, 15934 4. ExtLibs / NativeInterpreter (folder) 6. MainForm.WindowsForms 6.1 Progress.cs
comment:180 Changed 5 years ago by abeham
r17086: updated nuget package of Attic to 1.2
comment:181 Changed 5 years ago by abeham
- fixed loading of samples, removed TODO (due to upgrade of Attic 1.2)
- corrected check when old persistence should be used to retry download from hive
comment:182 Changed 5 years ago by abeham
r17088: tested and simplified check
comment:183 Changed 5 years ago by abeham
r17089: Removed reference to Protobuf in Clients.Hive since the exception is no longer referenced
comment:184 Changed 5 years ago by mkommend
- Owner changed from gkronber to mkommend
- Status changed from accepted to readytorelease
Reviewed the last four changesets by abeham.
This ticket blocks the release of several other tickets. Therefore, I am moving this to ready to be release. Any subsequent development should be tracked in another ticket.
comment:185 Changed 5 years ago by mkommend
r17097: Merged r16565:16579 into stable.
comment:186 Changed 5 years ago by mkommend
- Keywords merged added
r17105: Merged r16584, r16585,r16594,r16595, r16625, r16658, r16659, r16672, r16707, r16729, r16792, r16796, r16797, r16799, r16819, r16906, r16907, r16908, r16933, r16945, r16992, r16994, r16995, r16996, r16997, r17014, r17015, r17017, r17020, r17021, r17022, r17023, r17024, r17029, r17086, r17087, r17088, r17089 into stable.
comment:187 Changed 5 years ago by abeham
- Resolution set to done
- Status changed from readytorelease to closed
r13323: Created new branch for persistence overhaul