Programmable operators are recompiled after deserialization

[swagner]

As already stated in ticket #171, the compiled code of programmable operators is not persisted. Programmable operators have to be compiled again after deserialization. Therefore, using programmable operators in a distributed environment is quite slow.

[gkronber]

In my opinion this is probably not an issue. The overhead incurred by recompilation of programmable operators is likely very small when compared with the overhead of serialization, transmission and execution of jobs. Even if we serialize the generated assembly it's still necessary to JIT-compile the function on the first evaluation.

[swagner]

Yop, that's a good point. I will not fix this issue for now. We will see, if it turnes out to be worthwhile, when we use distributed engines more frequently.

[swinkler]

Maybe we have already worked on something similar in HL2. In ProgrammableOptimizationElementBase, e.g., we (de)serialize the binary assembly, from which the specific delegate is then extracted. Could be that something similar could be used here, too.

[gkronber]

It turns out this is an issue after all however not because of computation time but because of memory consumption. CodeDom creates a new assembly each time the programmable operator is recompiled. Assemblies are loaded into the AppDomain and can't be unloaded until the AppDomain is disposed. In a distributed environment the number of loaded assemblies grows into the order of 20.000. It seems at least 64k are reserved for each assembly even if the actual code is only a few bytes. This means >1GB RAM is reserved for unused assemblies which can't be freed by the GC which leads to dreaded OutOfMemoryExceptions.

[gkronber]

After some thinking there is actually no good workaround for the memory consumption problem because even if we persist the compiled assembly we still have to load the code with Assembly.Load() into the AppDomain (creating a new assembly).

The only viable solution coming to mind is to use a form of caching on the client side. Switching to Reflection.Emit which allows dynamic code generation without creating new assemblies is not really an option because we would basically have to write our own C#->IL compiler.

[swagner]

Milestone 3.2 deleted

[swagner]

This issue has not occurred in HeuristicLab 3.3 so far. If it turns up again, a new ticket should be written.