Opened 8 months ago

Closed 6 weeks ago

Last modified 6 weeks ago

#2916 closed enhancement (done)

Improve serialization of IndexedDataTable

Reported by: abeham Owned by: abeham
Priority: medium Milestone: HeuristicLab 3.3.16
Component: Analysis Version: trunk
Keywords: Cc:

Description

IndexedDataTables are stored extremely inefficiently in the serialized xml and take a huge amount of space.

A simple graph of 3 points (6 double values) requires more than 700 characters

<COMPOSITE name="values" typeId="30" id="79">
  <COMPOSITE name="list" typeId="31" id="80">
    <COMPOSITE typeId="32" id="81">
      <METAINFO>
        <PRIMITIVE typeId="13" name="Item1">4.7700000000000001e-05</PRIMITIVE>
        <PRIMITIVE typeId="13" name="Item2">943176159</PRIMITIVE>
      </METAINFO>
    </COMPOSITE>
    <COMPOSITE typeId="32" id="82">
      <METAINFO>
        <PRIMITIVE typeId="13" name="Item1">7.7999999999999999e-05</PRIMITIVE>
        <PRIMITIVE typeId="13" name="Item2">943175954</PRIMITIVE>
      </METAINFO>
    </COMPOSITE>
    <COMPOSITE typeId="32" id="83">
      <METAINFO>
        <PRIMITIVE typeId="13" name="Item1">9.3899999999999992e-05</PRIMITIVE>
        <PRIMITIVE typeId="13" name="Item2">943175905</PRIMITIVE>
      </METAINFO>
    </COMPOSITE>
    ...
  </COMPOSITE>
</COMPOSITE>

Change History (12)

comment:1 Changed 8 months ago by abeham

r15917:15918: implemented more efficient storage format

comment:2 Changed 8 months ago by abeham

  • Owner set to abeham
  • Status changed from new to accepted

comment:3 Changed 8 months ago by abeham

  • Owner changed from abeham to mkommend
  • Status changed from accepted to reviewing

After these changes I tested with a large experiment containing more than 50,000 runs. Each containing an indexed data table. Compressed storage space was reduced from 87,605 kb to 49,710 kb. Uncompressed storage space was reduced from 1538 Mb to 831 Mb.

Deserialization is "quite" fast, but serialization is very slow.

comment:4 Changed 5 months ago by mkommend

Reviewed r15918 (r15917 is only the branch creation).

Changes look good and are ready for trunk integration and subsequent release.

comment:5 Changed 5 months ago by mkommend

  • Owner changed from mkommend to abeham
  • Status changed from reviewing to assigned

comment:6 Changed 3 months ago by abeham

r16177: merged to trunk

comment:7 Changed 3 months ago by abeham

  • Status changed from assigned to reviewing
  • Version set to trunk

r16178: removed branch

comment:8 Changed 3 months ago by abeham

  • Status changed from reviewing to readytorelease

comment:9 Changed 3 months ago by abeham

  • Component changed from ### Undefined ### to Analysis

comment:10 Changed 6 weeks ago by abeham

r16282: merged to stable

comment:11 Changed 6 weeks ago by abeham

  • Resolution set to done
  • Status changed from readytorelease to closed

comment:12 Changed 6 weeks ago by abeham

  • Type changed from defect to enhancement
Note: See TracTickets for help on using tickets.