#2916 closed enhancement (done)
Improve serialization of IndexedDataTable
Reported by: | abeham | Owned by: | abeham |
---|---|---|---|
Priority: | medium | Milestone: | HeuristicLab 3.3.16 |
Component: | Analysis | Version: | trunk |
Keywords: | Cc: |
Description
IndexedDataTables are stored extremely inefficiently in the serialized xml and take a huge amount of space.
A simple graph of 3 points (6 double values) requires more than 700 characters
<COMPOSITE name="values" typeId="30" id="79"> <COMPOSITE name="list" typeId="31" id="80"> <COMPOSITE typeId="32" id="81"> <METAINFO> <PRIMITIVE typeId="13" name="Item1">4.7700000000000001e-05</PRIMITIVE> <PRIMITIVE typeId="13" name="Item2">943176159</PRIMITIVE> </METAINFO> </COMPOSITE> <COMPOSITE typeId="32" id="82"> <METAINFO> <PRIMITIVE typeId="13" name="Item1">7.7999999999999999e-05</PRIMITIVE> <PRIMITIVE typeId="13" name="Item2">943175954</PRIMITIVE> </METAINFO> </COMPOSITE> <COMPOSITE typeId="32" id="83"> <METAINFO> <PRIMITIVE typeId="13" name="Item1">9.3899999999999992e-05</PRIMITIVE> <PRIMITIVE typeId="13" name="Item2">943175905</PRIMITIVE> </METAINFO> </COMPOSITE> ... </COMPOSITE> </COMPOSITE>
Change History (12)
comment:1 Changed 7 years ago by abeham
comment:2 Changed 7 years ago by abeham
- Owner set to abeham
- Status changed from new to accepted
comment:3 Changed 7 years ago by abeham
- Owner changed from abeham to mkommend
- Status changed from accepted to reviewing
After these changes I tested with a large experiment containing more than 50,000 runs. Each containing an indexed data table. Compressed storage space was reduced from 87,605 kb to 49,710 kb. Uncompressed storage space was reduced from 1538 Mb to 831 Mb.
Deserialization is "quite" fast, but serialization is very slow.
comment:4 Changed 6 years ago by mkommend
comment:5 Changed 6 years ago by mkommend
- Owner changed from mkommend to abeham
- Status changed from reviewing to assigned
comment:6 Changed 6 years ago by abeham
r16177: merged to trunk
comment:7 Changed 6 years ago by abeham
- Status changed from assigned to reviewing
- Version set to trunk
r16178: removed branch
comment:8 Changed 6 years ago by abeham
- Status changed from reviewing to readytorelease
comment:9 Changed 6 years ago by abeham
- Component changed from ### Undefined ### to Analysis
comment:10 Changed 6 years ago by abeham
r16282: merged to stable
comment:11 Changed 6 years ago by abeham
- Resolution set to done
- Status changed from readytorelease to closed
comment:12 Changed 6 years ago by abeham
- Type changed from defect to enhancement
Note: See
TracTickets for help on using
tickets.
r15917:15918: implemented more efficient storage format