Hive-3.4 development

[cneumuel]

General notes

Server

• Refactor domain objects and db-schema
  • Split info-objects and data-objects (like Job and JobData)
• Data Access Layer (more consistent method names, more compact code, inspired by OKB)
• Split transaction and db-context handling
• Allow uploading of plugins for a job (or hiveexperiment)
• Make WCF service completely stateless. Put all remaining state-information into the database (latestHeartbeats, latestConsistencyCheck, newlyAssignedJobs (remove completely and solve by adding a heartbeat))
• StateLog: Log state transitions of jobs.
• Statistics
  • Measure core capacity and utilization every minute
  • Measure CPU and memory capacity and utilization every minute
  • Reliably measure the execution time spent on hive per user / in total. Also measure speedup values (maybe also per minute). Keep jobs deleted jobs in database (flag them) - only delete JobData, plugins ect.
  • Number of experiments / jobs (per user). Job per slave
  • Calculate overall productivity per job (waiting time vs. computation time)
• Scheduler
  • Consider waiting time to avoid starvation
  • Users should have priorities
  • A user should be able to manage priorities only in the scope of his own experiments
  • Childjobs should automatically have the priorities of their parent jobs
  • Precomputed job-queue
• Fix wrong timestamps in statelog on services.heuristiclab.com

Slave

• Adapt Slave for new Server
• Refactor Slave (easier communication between core and executor)
• Tests
• Console Client
• Windows Service Client
• Installer for Slave
• Windows Tray Icon for Slave
• HL App Client
• Sort out problem with uploaded, modified assemblies which aren't downloaded to the slave; Add GUIDs to PluginCache
• Heartbeat interval should be controllable by the server
• Creation of a unique Id for a machine which does not change if the config is deleted
• Correct total physical memory available for a slave (ConfigManager)
• Test sandboxing and security of appdomains. If any assemblies can be uploaded by users, becomes very important.
• React on SayHello action (call Hello service method)
• Send cpu utilisation with every heartbeat
• Log exceptions to Windows Event Log
• FreeCores needs to be decremented right after a CalculateJob message has been received. Otherwise a slave reports free cores which are already reserved for new jobs.
• PluginTemp directory should be cleaned up from time to time (or on startup)
• SlaveCommListener in Slave.Tests should not be used in ConsoleClient
• Heartbeats are massively delayed, because the heartbeat-method locks on engines (in GetExecutionTimeOfAllJobs) and the same lock is made at StartJobInAppDomain. This causes the a slave-heartbeat-timeout (1 minute), thus a reset and reassignment of all jobs.

Experiment Manager

• Show jobs in treeview. Would greatly save screen space and navigation-clicks
  • to be enhanced (event wiring)
• Sort HiveExperiments alphabetically
• Plugin-Upload (optional)
• Experiment Sharing
• Appropriate numbering of Runs
• Use Service-Call pattern from OKB (or PPOV-Cockpit)
• Show StateLog - use Gantt Chart like view
• Pause and stop single jobs
• Paused jobs should not be integrated into experiment, so results are not lost. Parameters of paused jobs should be changable (and used when resumed).
• Deleting jobs after adding them (neither the remove button, nor the del key, nor the context menu entry succeeds in deleting a job (experiment) that has just been dragged in)

Hive Engine

• HiveEngine jobs should have a HiveExperiment, which is marked, so a user cannot see it in HiveExperimentManager. However it should be visible in Administration GUI. If a Hive Engine crashes and cannot delete the experiment, this should be detected by the server and it should be automatically deleted.
• Improve HiveEngine View (list of jobs, with status ect.)
• Stabilize

Administration

Missing WebService Methods:

• GetAllHiveExperiments
• GetUsers
• GetUserStatistics
• GetJobsBySlave -> GetJobsByResourceId
• GetGlobalStatistics (for Statistics TabPage)
• GetScheduleForResource (+ Add/Update/Delete)

TODOS:

• convert HeuristicLab.Calender to a plugin
• use svcutil
• write partial classes for dtos and implement IContent
• build Observable Collections for Users/Slaves/Groups
• add ContentViews for Users and SlaveGroups
• show some fancy statistics
• add Save Button
• integrate HeuristicLab.Services.Hive.Common-3.4 in Server
• get rid of HiveItem etc. on Server

Meeting protocols

Architects meeting ^(16.06.2011)

DataAccess:

• TransactionManager with interface again
• remove AssignedResourcesId in AssignedResources, use JobId+ResourceId as primary keys
• remove CreateHiveDatabaseApplication. the db schema should not be developed dbml first, since dbml does not support most sql-server features. instead the sql-server schema should be designed first and the dbml should be generated.
• UptimeCalendar should be named DowntimeCalendar
• DataAccess layer and Dao classes should be removed, access to linq to sql should happen directly in server-implementation.

Server

• Lifecycle should be named differently. maybe EventHandler, EventManager.
• put magic numbers into config
  • timeout in Lifecycle
  • ApplicationConstants
  • look for magic numbers in hive client
• GetWaitingJobs should be implemented as a stored procedure and should also assign a job to a slave. it should make sure no race conditions occur if it is called concurrently.

HiveExperiment

• rename: HiveExperiment -> Job, Job -> Task
• HiveExperimentPermissions
  • the GrantedUserId could be removed
    • ascheibe: GrantedUserId is part of the PK and can't be removed. GrantedByUserId is not necessary and could be removed, but it still could be interesting information?
  • only Full and Read permissions are necessary (Read: just read!, Full: control, delete, grant permissions)
• remove LastAccessed and IsHiveEngine. there should be a category field instead.

Remarks for the future (cneumuel) ^(28.06.2011)

Security

• GetPlugins currently returns all plugins from the server. This exposes all uploaded assemblies. When confidentiality for plugins is relevant this method should be removed and only GetPlugin(s)ById and GetPlugin(s)ByHash should be available.
• Slave-user: Each hive slave uses the same username and password. A slave is allowed to download jobs. When a slave downloads a job it should be checked if the job is assigned to this slave or a parent-slave-group (not implemented yet). However it is still possible for an attacker to fake the ID of another slave (if it is known) and get access to jobs.

Statistics

• Further measures to include (as total sums, also keep deleted jobs in DeletedJobStatistics):
  • Globally: FinishedJobs, WaitingJobs, FailedJobs, AbortedJobs, TransferringJobs, PausedJobs
  • Per user: total jobdata-size (MB)

Server performance

• Increasing number of slaves puts pressure on the server with increasing response times and some deadlock-situations. Ideas to resolve:
  • Increase heartbeat-interval (maybe dynamic when the number of slaves gets higher). Remember to increase the SlaveHeartbeatTimeout in the web.config too.
  • Make GetWaitingJobs faster by using stored procedure or use a job-queue instead of querying the whole job-table.
• Large jobs (>15MB) are sometimes result in database-timeouts, especially if multiple of them are uploaded concurrently. Ideas to resolve:
  • Use Filestream as db-type instead of Varbinary as it is supposed to be faster for large data-blobs.
  • As streaming is not an option (no security, encryption), using a chunking channel could work (​http://msdn.microsoft.com/en-us/library/aa717050.aspx).

Scheduling
Some ideas for a scheduler:

• 3 levels of priorities:
  • Job priority (fixed at upload)
  • User priority (fixed)
  • Time (dynamic: f(Now-Uploaded))
• Those 3 priority values are aggregated (average, (weighted-)sum) represent the final priority by which the jobs are ordered.
• Fast-slaves-first: Faster slaves get the jobs first, slow slaves later. This would require:
  • Performance-index: Let each slave calculate a benchmark-job before it is used.
  • Job-queues per slaves: Right now every slave who sends a heartbeat gets a job (if one is available). One queue per slave would allow the server to actively assign jobs to slaves. Such a queue could also ease performance issues and race conditions.
• Re-scheduling: Sometimes fast slaves finish their jobs and slow slaves are still calculating. In those cases it might be reasonable to pause the jobs and have them calculated on the faster slaves.

[ascheibe]

r5633 added Appointment/Schedule ws and dao methods

[cneumuel]

r5636

• updated jobstates documentation
• enhanced ganttChart
• fixed setting of jobstates
• added option to force lifecycle-trigger (mainly for testing purposes)

[cneumuel]

r5637 added treeview for hive jobs in experiment manager

[ascheibe]

r5638 worked on Administration UI

[cneumuel]

r5675 improved treeview for hive jobs

[ascheibe]

r5676 worked on Administration UI

[ascheibe]

r5677 some minor ui fixes for slave

[cneumuel]

r5708 changed the way transactions are handled

[ascheibe]

r5711

• use SlaveComm Endpoint from app.config
• various further slave bugfixes/cleanups
• added preliminary icon for hive slave ui and some slave ui improvements
• added resource deletion to admin ui
• fix service exception thrown if there is no EventLog

[cneumuel]

r5718

• fixed statelog when time on server differs from slave or client
• fixed wrong creation of childjobs in experiment manager
• made ganttchardview the default view for statelogs

[ascheibe]

r5721 worked on slave and slave service installer

[ascheibe]

r5778

• log uncaught exceptions to an eventlog if available
• fixed job pause bug

[cneumuel]

r5779

• implemented pause, stop for single jobs
• introduced Command property for jobs (to distinguish between state and command (abort vs. aborted))
• improved behaviour of ItemTreeView (double click opens new window, selected item stays marked)
• fixed bugs in StateLogGanttChartListView and HiveJobView
• fixed cloning of client-side dtos

[ascheibe]

r5780 various improvments on the service installer and slave tray icon

[ascheibe]

r5782

• fixed job pause bug... again
• general Executor improvements

[cneumuel]

r5786

• implemented correct numbering of BatchRuns
• improvements in ExperimentManager
• fixed bug in server (jobs were scheduled multiple times)
• added exception handling for task in slave
• improved timeout handling of jobs (LifecycleManager)

[cneumuel]

r5787 made deleting and creating directories for PluginTemp more robust

[ascheibe]

r5789

• added autostart for tray icon to installer
• machine unique id now includes the machine name
• core: check if job already exists on slave
• already finished jobs now fail and are sent back

[ascheibe]

r5790 don't save the unique machine id

[cneumuel]

r5793

• implemented correct downloading of paused jobs. its now also possible to change parameters and resume a algorithm
• removed Prepare() calls in ExperimentManager and in slave, as it prevents corrent resuming of paused jobs
• made events in ItemTreeView be invoked in the correct thread
• reduced log output in ExperimentManager

[ascheibe]

r5795 various slave and slave tray icon improvements

[cneumuel]

r5797

• ItemTreeView robustifications
• compactified the layout in HiveJobView

[ascheibe]

r5826 slave ui now receives status information and displays it in doughnut chart

[cneumuel]

r5955

• seperated ExperimentMangerClient (OKB-Style, contains business logic) and HiveExperiment (mainly only contains information)
• fixed redundant cloning methods in dtos
• added simple statistics in HiveExperiment which the user can see before downloading an experiment
• added db-delete cascade for slaves and statelogs - now slaves can be safely deleted

[cneumuel]

r5958 initial port of HiveEngine

[cneumuel]

r6000 :)

• added GetPlugin service method
• fixed minor issues with double plugins in database
• worked on HiveEngine
• fixed wrong role name for Hive User
• fixed bug in group assignment of slaves

[ascheibe]

r6004

• fix pause/stop bug when serializing big experiments
• use proper newlines
• use GetPlugin(..) instead of GetPlugins()

[cneumuel]

r6006

• changed relationship between Job and HiveExperiment. There is no more HiveExperiment.RootJobId, instead there is Job.HiveExperimentId.
• one HiveExperiment can now have multiple Experiments.
• TreeView supports multiple root nodes
• HiveEngine creates a HiveExperiment for each set of jobs, so jobs cannot be without an parent experiment anymore (no more loose jobs)
• updated ExperimentManager binaries

[ascheibe]

r6008

• increase timeout when sending (for sending large jobs/lot's of plugins)
• handle failed GetPluginDatas() properly

[gkronber]

GetPluginDatas() is a strange identifier. Plural of data is data.

[cneumuel]

r6033

• created baseclass for jobs (ItemJob) which derives OperatorJobs and EngineJobs
• created special view for OptimizerJobs which derives from a more general view
• removed logic from domain class HiveExperiment and moved it into RefreshableHiveExperiment
• improved ItemTreeView
• corrected plugin dependencies
• fixed bug in database trigger when deleting HiveExperiments
• added delete cascade for Plugin and PluginData
• lots of fixes

[ascheibe]

r6100

• Executor now sends all exceptions to the ExperimentManager as NetNamedPipe communication won't be possible in a Sandbox due to security constraints
• count stopped and aborted jobs correctly
• send correct status when a job is stopped by the ExperimentManager
• try to log unhandled exceptions to gui if no EventLog is available
• don't crash if job is sent more than once by server

[ascheibe]

r6101

• don't lock engines for so long in StartJobInAppDomain
• move SlaveCommListener to ConsoleClient
• delete orphaned job folders at startup

[ascheibe]

r6107

• simplify PreparePlugins
• send more exceptions to ExperimentManager

[cneumuel]

r6110

• renamed engines to executors
• changed locking in StartJobInAppDomain
• avoid destruction of proxy object after 5 minutes for Slave.Core
• added JobStarted event and fixed ExecutionStateChanged and ExecutionTimeChanged
• slaves which are moved to another slavegroup will pause their jobs now, if they must not calculate them

[cneumuel]

r6111 improved the way jobs are downloaded by ExperimentManager and HiveEngine

[ascheibe]

r6112

• HeartbeatManager: don't sleep while starting jobs
• Executor: make Start() blocking
• shutdown properly if an uncaught exception is thrown

[ascheibe]

r6116

• SlaveTrayIcon: don't try to kill TrayIcons from other users
• split installer to fix config installer bug for users who did not run the installer

[ascheibe]

r6166 forgot to check in HL icon for installers

[ascheibe]

r6167

• increased send/receive timeout
• renamed hive binding name

[cneumuel]

r6168

• removed Job-dto objects from slave core (since it stores outdated objects)
• added command textbox to HiveJobView
• improved the way the control buttons behave in HiveJobView
• improved job control (pause and stop is also possible when job is not currently calculating)
• improved gantt chart view (last state log entry is also displayed)
• unified code for downloading jobs between experiment manager and hive engine

[ascheibe]

r6175 temporary switch to privileged sandboxing until communication between core and executor works with sandboxing

[cneumuel]

r6178

• added semaphores to ensure an appdomain is never unloaded when the start method has not finished
• HiveEngine uploading and downloading of jobs works and is displayed in the view

[ascheibe]

r6203

• dropped dependency of Core from Executor
• enabled sandboxing
• moved most parts of Job handling from Core to SlaveJob to simplify locking
• optimized how UsedCores is handled
• SlaveStatusInfo is now thread-save and counts jobs more correct

[ascheibe]

r6204 don't crash on shutdown

[cneumuel]

r6212 created HiveEngine.Views plugin

[ascheibe]

r6216

• make UsedCores more reliable
• some cosmetic fixes

[cneumuel]

r6219 improved exception handling for hive experiments

[ascheibe]

r6225

• Slave UI now uses tab pages
• balloon tips are displayed on receiving new jobs

[cneumuel]

r6229

• added basic statistics recording (once per minute) for
  • executiontime per user
  • usedcores, usedmemory per slave

[ascheibe]

r6230

• don't set every view as default in slave ui
• fixed bug in PluginCache where files got accessed by multiple threads

[ascheibe]

r6248

• don't set job failed if JobNotFoundException is thrown
• disable AboutView for all items
• avoid NullRefException in SendFinishedJob

[ascheibe]

r6257

• added UAC self elevation for start/stop of windows service
• added slave states and simplified ui commands

[ascheibe]

r6263

• added view for displaying jobs
• improved slave ui

[cneumuel]

r6267

• extended statistics recording:
  • execution times of users are captured
  • execution times and start-to-finish time of finished jobs is captured (to computer hive overhead)
  • data of deleted jobs is automatically captured in DeletedJobStatistics
• changed ExecutionTime type in database from string to float (milliseconds are stored instead of TimeSpan.ToString())
• added IsPrivileged field to job to indicate if it should be executed in a privileged sandbox
• added CpuUtilization field to slave to be able to report cpu utilization
• added GetJobsByResourceId to retrieve all jobs which are currently beeing calculated in a slave(-group)
• TransactionManager now allows to use serializable tranactions (used for lifecycle trigger)

[cneumuel]

r6269 added CpuUtilization to heartbeats

[cneumuel]

r6357

• refactoring of slave core
• created JobManager, which is responsible for managing jobs without knowing anything about the service. this class is easier testable than slave core
• lots of cleanup
• created console test project for slave

[cneumuel]

r6362 changed roles authentication to use an AuthenticationManager instead of method attributes. this makes unit tests easier.

[cneumuel]

r6369

• added consideration of appointments in heartbeats
• code cleanup

[ascheibe]

r6371

• code cleanups for slave review
• added switch between privileged and unprivileged sandbox
• removed childjob management because it's not used

[ascheibe]

r6372 changed year to 2011

[cneumuel]

r6373

• moved ExperimentManager into separate plugin
• moved Administration into separate plugin

[cneumuel]

r6381

• locking for childHiveJobs in OptimizerHiveJob avoid multi threaded access issues
• added IsPrivileged to gui
• minor changes

[ascheibe]

r6407

• implemented usage of checksums for comparing assemblies
• re-added CreateHiveDatabaseApplication.cs to project

[abeham]

r6418

• fixed references to absolute path references

[cneumuel]

r6419, r6420

• created events when statelog changed
• fixed memory leak in hiveengine
• extended timeout for long running transactions and database contexts (when jobdata is stored)
• replaced random guids in database with sequential guids for performance reasons
• minor fixes and cleanups
• updated hive binaries
• updated statistics

[abeham]

r6422

• synchronized config file with that from trunk

[abeham]

Added TODO point regarding the deletion of jobs in the experiment manager

[ascheibe]

r6426 removed useLocalPlugins

[cneumuel]

r6431 - applied some review comments

General:

• changed Log to ThreadSafeLog
• added license information to all files
• added assembly descriptions
• using blocks before namespace

HeuristicLab.Services.Hive.DataAccess:

• made TransactionManager static
• removed DaoException
• removed TimeSpanExtensions
• renamed prepareHiveDatabase.sql should be renamed to Prepare Hive Database.sql
• created Initialize Hive Database.sql

[cneumuel]

r6435

• some cleanup in HiveEngine
• using ThreadSafeLog instead of synchronized methods

[ascheibe]

r6437 Admin UI:

• some bugfixes
• removed dummy stuff

[cneumuel]

r6444

• stability improvements for HiveExperiment and HiveEngine
• parallelized upload of jobs
• enabled cancellation of job upload
• reduced the amount of double-assignment of jobs by an additional check in HeartbeatManager
• tried to tackle the amount of deadlocks by automatically rerunning transactions
• some fixes

[ascheibe]

r6451 Admin UI:

• added subgroups
• groups can now have calendars
• calendar bugfixes

[cneumuel]

r6452

• renamed UptimeCalendar and Appointment to Downtime
• added service methods to delete plugins and get plugin by hash
• made reverted TransactionManager change, made it non-static and added interface
• moved magic numbers to application settings

[ascheibe]

r6456

• fixed Admin Views plugin dependencies
• use settings instead of magic numbers and strings

[cneumuel]

r6457

• added methods for granting and revoking hive experiment permissions
• added unit tests for hive experiment permissions
• added a status webpage to see how some statistics and current status

[cneumuel]

r6458 visualization of statistics on status page (requires MS charting controls) (​http://services.heuristiclab.com/Hive-3.4/Status.aspx)

[cneumuel]

r6463

• created user interface for experiment sharing
• created UserManager which provides access to the users
• inserted a lot of security and authorization checks serverside
• minor fixes in experiment manager

[ascheibe]

r6464

• some Admin UI bugfixes

Slave:

• fixed bug when Pause is called immediately after Calculate
• send exceptions when something goes wrong in Pause or Stop

[cneumuel]

r6465 show owner of experimens in listview

[cneumuel]

r6479

• finished experiment sharing
• added role for executing privileged jobs
• refreshing experiments in experimentManager does not delete already downloaded jobs
• moved some properties from HiveExperiment into RefreshableHiveExperiment

[cneumuel]

r6481

• added web.config from services
• added some help-sql scripts

[ascheibe]

r6492 Admin UI:

• don't completely rebuild treeview on drag and drop
• some bugfixes

[ascheibe]

r6521 catch exception when querying execution times

[ascheibe]

r6546: Don't call Clear() on ThreadSafeLog in log_MessageAdded. This doesn't work with the changes made in r6536 (LockRecursionPolicy.SupportsRecursion). Instead use maxLogCount of Core.Log to limit the number of log messages kept in memory.

[cneumuel]

Solved problems with large jobs (>15MB) by adding the following node to the <requestFiltering section in %windir%\System32\inetsrv\config\applicationHost.config on the server:

<requestLimits maxAllowedContentLength="2147483647" />

Limit should theoretically be 2GB, tested with 150MB.

source: ​http://cutesoft.net/forums/thread/42292.aspx

[ascheibe]

r6595 removed unused config section

[ascheibe]

r6683 replaced the CreateHiveDatabaseApplication HL App with a Windows Forms application

[ascheibe]

r6688 some renaming to be more consistent with OKB

[ascheibe]

r6689 some more renaming to be more consistent with OKB

[ascheibe]

r6696

• some cleanups
• removed unused code

[ascheibe]

r6698

• implemented review comments
• more cleanups

[ascheibe]

r6700 changed version in project files, AssemblyInfo and plugin files back to 3.3

[ascheibe]

r6701 renamed folders from 3.4 to 3.3

[ascheibe]

r6703

• really removed 3.4 folders
• added skeleton for a Slave HL App
• added missing license headers and AssemblyInfo frames
• fixed merging of config files

[ascheibe]

r6704 added plugin frame file to Slave HL App

[ascheibe]

r6712

• got db scripts up-to-date
• renamed db related stuff back to 3.3
• fixed a bug in the Status page that occured when the db is empty

[ascheibe]

r6717

• moved DTO's to Services.Hive project
• removed Services.Hive.Common project
• some cleanups
• added DTO's for enums