Hive-3.4 development

[cneumuel]

Server

• Refactor domain objects and db-schema
  • Split info-objects and data-objects (like Job and JobData)
• Data Access Layer (more consistent method names, more compact code, inspired by OKB)
• Split transaction and db-context handling
• Allow uploading of plugins for a job (or hiveexperiment)
• Make WCF service completely stateless. Put all remaining state-information into the database (latestHeartbeats, latestConsistencyCheck, newlyAssignedJobs (remove completely and solve by adding a heartbeat))
• StateLog: Log state transitions of jobs.
• Statistics
  • Measure core capacity and utilization every minute
  • Measure CPU and memory capacity and utilization every minute
  • Reliably measure the execution time spent on hive per user / per slave / in total. Also measure speedup values (maybe also per minute). Keep jobs deleted jobs in database (flag them) - only delete JobData, plugins ect.
  • Number of experiments / jobs (per user). Job per slave
  • Calculate overall productivity per job (waiting time vs. computation time)
• Scheduler
  • Consider waiting time to avoid starvation
  • Users should have priorities
  • A user should be able to manage priorities only in the scope of his own experiments
  • Childjobs should automatically have the priorities of their parent jobs
  • Precomputed job-queue
• Fix wrong timestamps in statelog on services.heuristiclab.com

Slave

• Adapt Slave for new Server
• Refactor Slave (easier communication between core and executor)
• Tests
• Console Client
• Windows Service Client
• Installer for Slave
• Windows Tray Icon for Slave
• HL App Client
• Sort out problem with uploaded, modified assemblies which aren't downloaded to the slave; Add GUIDs to PluginCache
• Heartbeat interval should be controllable by the server
• Creation of a unique Id for a machine which does not change if the config is deleted
• Correct total physical memory available for a slave (ConfigManager)
• Test sandboxing and security of appdomains. If any assemblies can be uploaded by users, becomes very important.
• React on SayHello action (call Hello service method)
• Send cpu utilisation with every heartbeat
• Log exceptions to Windows Event Log
• FreeCores needs to be decremented right after a CalculateJob message has been received. Otherwise a slave reports free cores which are already reserved for new jobs.
• PluginTemp directory should be cleaned up from time to time (or on startup)

Experiment Manager

• Show jobs in treeview. Would greatly save screen space and navigation-clicks
  • to be enhanced (event wiring)
• Sort HiveExperiments alphabetically
• Plugin-Upload (optional)
• Experiment Sharing
• Appropriate numbering of Runs
• Use Service-Call pattern from OKB (or PPOV-Cockpit)
• Show StateLog - use Gantt Chart like view
• Pause and stop single jobs
• Paused jobs should not be integrated into experiment, so results are not lost. Parameters of paused jobs should be changable (and used when resumed).

Hive Engine

• HiveEngine jobs should have a HiveExperiment, which is marked, so a user cannot see it in HiveExperimentManager. However it should be visible in Administration GUI. If a Hive Engine crashes and cannot delete the experiment, this should be detected by the server and it should be automatically deleted.
• Improve HiveEngine View (list of jobs, with status ect.)

Administration

Missing WebService Methods:

• GetAllHiveExperiments
• GetUsers
• GetUserStatistics
• GetGlobalStatistics (for Statistics TabPage)
• GetScheduleForResource (+ Add/Update/Delete)

TODOS:

• convert HeuristicLab.Calender to a plugin
• use svcutil
• write partial classes for dtos and implement IContent
• build Observable Collections for Users/Slaves/Groups
• add ContentViews for Users and SlaveGroups
• show some fancy statistics
• add Save Button
• integrate HeuristicLab.Services.Hive.Common-3.4 in Server
• get rid of HiveItem etc. on Server

Client Management Integration

User Management Integration

[ascheibe]

r5633 added Appointment/Schedule ws and dao methods

[cneumuel]

r5636

• updated jobstates documentation
• enhanced ganttChart
• fixed setting of jobstates
• added option to force lifecycle-trigger (mainly for testing purposes)

[cneumuel]

r5637 added treeview for hive jobs in experiment manager

[ascheibe]

r5638 worked on Administration UI

[cneumuel]

r5675 improved treeview for hive jobs

[ascheibe]

r5676 worked on Administration UI

[ascheibe]

r5677 some minor ui fixes for slave

[cneumuel]

r5708 changed the way transactions are handled

[ascheibe]

r5711

• use SlaveComm Endpoint from app.config
• various further slave bugfixes/cleanups
• added preliminary icon for hive slave ui and some slave ui improvements
• added resource deletion to admin ui
• fix service exception thrown if there is no EventLog

[cneumuel]

r5718

• fixed statelog when time on server differs from slave or client
• fixed wrong creation of childjobs in experiment manager
• made ganttchardview the default view for statelogs

[ascheibe]

r5721 worked on slave and slave service installer

[ascheibe]

r5778

• log uncaught exceptions to an eventlog if available
• fixed job pause bug

[cneumuel]

r5779

• implemented pause, stop for single jobs
• introduced Command property for jobs (to distinguish between state and command (abort vs. aborted))
• improved behaviour of ItemTreeView (double click opens new window, selected item stays marked)
• fixed bugs in StateLogGanttChartListView and HiveJobView
• fixed cloning of client-side dtos

[ascheibe]

r5780 various improvments on the service installer and slave tray icon

[ascheibe]

r5782

• fixed job pause bug... again
• general Executor improvements

[cneumuel]

r5786

• implemented correct numbering of BatchRuns
• improvements in ExperimentManager
• fixed bug in server (jobs were scheduled multiple times)
• added exception handling for task in slave
• improved timeout handling of jobs (LifecycleManager)