Opened 11 years ago

Closed 9 years ago

#173 closed enhancement (obsolete)

Programmable operators are recompiled after deserialization

Reported by: swagner Owned by: epitzer
Priority: low Milestone: HeuristicLab 3.3.0
Component: Operators.Programmable Version: 3.2
Keywords: Cc:

Description

As already stated in ticket #171, the compiled code of programmable operators is not persisted. Programmable operators have to be compiled again after deserialization. Therefore, using programmable operators in a distributed environment is quite slow.

Change History (10)

comment:1 Changed 11 years ago by gkronber

In my opinion this is probably not an issue. The overhead incurred by recompilation of programmable operators is likely very small when compared with the overhead of serialization, transmission and execution of jobs. Even if we serialize the generated assembly it's still necessary to JIT-compile the function on the first evaluation.

comment:2 Changed 11 years ago by swagner

Yop, that's a good point. I will not fix this issue for now. We will see, if it turnes out to be worthwhile, when we use distributed engines more frequently.

comment:3 Changed 11 years ago by swinkler

Maybe we have already worked on something similar in HL2. In ProgrammableOptimizationElementBase, e.g., we (de)serialize the binary assembly, from which the specific delegate is then extracted. Could be that something similar could be used here, too.

comment:4 Changed 11 years ago by gkronber

It turns out this is an issue after all however not because of computation time but because of memory consumption. CodeDom creates a new assembly each time the programmable operator is recompiled. Assemblies are loaded into the AppDomain and can't be unloaded until the AppDomain is disposed. In a distributed environment the number of loaded assemblies grows into the order of 20.000. It seems at least 64k are reserved for each assembly even if the actual code is only a few bytes. This means >1GB RAM is reserved for unused assemblies which can't be freed by the GC which leads to dreaded OutOfMemoryExceptions.

comment:5 Changed 11 years ago by gkronber

  • Owner changed from swagner to gkronber
  • Status changed from new to assigned

comment:6 Changed 11 years ago by gkronber

  • Owner changed from gkronber to swagner
  • Status changed from assigned to new

After some thinking there is actually no good workaround for the memory consumption problem because even if we persist the compiled assembly we still have to load the code with Assembly.Load() into the AppDomain (creating a new assembly).

The only viable solution coming to mind is to use a form of caching on the client side. Switching to Reflection.Emit which allows dynamic code generation without creating new assemblies is not really an option because we would basically have to write our own C#->IL compiler.

comment:7 Changed 11 years ago by swagner

  • Milestone changed from 3.1 to 3.2
  • Version changed from 3.1 to 3.2

comment:8 Changed 11 years ago by swagner

  • Milestone changed from 3.2 to Iteration 0

Milestone 3.2 deleted

comment:9 Changed 10 years ago by swagner

  • Owner changed from swagner to epitzer

comment:10 Changed 9 years ago by swagner

  • Milestone changed from HeuristicLab x.x.x to HeuristicLab 3.3.0
  • Resolution set to obsolete
  • Status changed from new to closed

This issue has not occurred in HeuristicLab 3.3 so far. If it turns up again, a new ticket should be written.

Note: See TracTickets for help on using tickets.