Implement project management for Hive

[jkarder]

Hive jobs should be assigned to projects instead of resources. We could better track resource usage per project and introduce some kind of credit system. Furthermore, this would allow us to assign one resource to different projects, which is currently not possible using resource groups.

Has to be merged with #2845.

fHolzing: In the "Select Project"-View, changing the selection of HiveSlaves via up/down arrow won't update the "Computing Resources"-Details.
By checking the Checkbox besides the Project (in my case HEAL), the slaves cannot be unchecked independently.
By Right-clicking on any Slave, the "Computing Resources" are updated, just like the selection (for a fraction of a second). The selection gets reverted und jumps back to the previously selected item, but the "Computing Resources" still displays the information about the now unselected item.

[jkarder]

r15377: created branch for hive project management

[jkarder]

r15378: worked on database model

[jkarder]

r15379:

• added Project and ProjectPermission DAOs and service methods
• made Project.EndDate nullable

[jkarder]

r15380: worked on Project and ProjectPermission service methods

[jkarder]

r15399: worked on database model

[jkarder]

r15401:

• worked on hive administrator view
• updated service clients

[jkarder]

r15411: worked on database model

[jkarder]

r15412:

• worked on resources and projects views
• changed resource selector to be able to select projects and assigned resources
• updated service clients

[jkarder]

r15422: worked on resources and projects views

[jzenisek]

r15496: Added Dao for ResourcePermission entity

[jzenisek]

r15497: worked on permission check at AddTask

[jzenisek]

r15500: worked on resource permission checks (v2 = accelerated version)

[jzenisek]

r15503: added ResourcePermission handling (still in progress)

[jzenisek]

r15508: finalized permission checks in AddTask and revised implementation of ResourcePermission methods

[jzenisek]

r15523: worked on db model and updated dbml

[jzenisek]

r15526: reverted changes from last revision (r15523)

[jzenisek]

r15527:

• updated dbml (removed ResourcePermission and ResourceIds in Job-Table)
• updated Resource and Project Daos

[jzenisek]

r15528: added AssignedJobResource to dbml and Daos

[jzenisek]

r15530:

• worked on Job operations add&update
• worked on ProjectPermission handling
• worked on Project-Resource assignment

[jzenisek]

r15540: added checks for the administration of project-resource assignments

[jzenisek]

r15546: worked on ProjectResource assignment: (cascading) assigning & unassigning

[jzenisek]

r15547: worked on ProjectResource assignment: (cascading) assigning & unassigning (reverted changes of config files from r15546)

[jzenisek]

r15552: worked on permission checks in listing methods

[jzenisek]

r15557: fixed tree building routine in ResourcesView

[jzenisek]

r15559: worked on views for Projects and ProjectResources

[jzenisek]

r15567: worked on views for project and resource administration

[jzenisek]

r15576: worked on client side mgmt of project-resource assignments and project-user permissions

[jzenisek]

r15577: worked on service side mgmt of project-resource assignments and project-user permissions

[jzenisek]

r15580:

• minor bugfix in HiveService.cs
• corrected behavior of ProjectPermissionsView

[jzenisek]

r15627: worked on Hive Job Manager: improved project/resource selector to show existent assignments and preserve selections

[jzenisek]

r15628:

• updated TaskDao towards independency of the formerly used Task-Resource assignment entity
• updated several permission/assignment handling service methods for client
• added AssignedJobResource DTO

[jzenisek]

r15630:

• updated Heartbeat processing (regarding: checking against AssignedJobResources and handling of the updated Job deletion routine)
• updated Job deletion routine (still in progress at GenerateStatistics)

[jzenisek]

r15641: finished implementation of job deletion routine

[jzenisek]

r15642:

• adapted handling of cascading revocations of project-user permissions and project-resource assignments: from deleting all AssignedJobResource (formerly) to deleting only entries which are effected by the previously mentioned revocations (now)
• implemented job update (client side)
• corrected & improved stateful project/resource selection dialog
• corrected resource statistic calculation in project/resource selection dialog

[jzenisek]

r15643:

• worked on UpdateJob(..)
• adapted permission and assignment handling methods
• adpated dbml of HiveDataContext using the designer (added delete rules and onNull-handling)

[jzenisek]

r15644: removed AssignedTaskResource entity

[jzenisek]

r15658:

• worked on (restricted) accessibility of hive's administration area for non-admin roles
• adapted HiveClient & HiveAdminClient entity loading (client- & service-side)

[jzenisek]

r15659:

• added DimProject and FactProjectInfo entities to statistics generation
• implemented tracking for projects

[jzenisek]

r15666: implemented project facts-logging in HiveStatisticsGenerator

[jzenisek]

r15671:

• adapted computation of DimClient and FactClientInfo stats
• adapted web app according to new stats computation

[jzenisek]

r15715: improved permission checking of HiveService methods

[jzenisek]

r15716: worked on stats generation

[jzenisek]

r15737:

• updated sql scripts (necessary foreign key option alterations & introduction of statistic tables)
• updated HiveService according to changed client side (deletion-routine, permission checking,...)

[jzenisek]

r15742: worked on HiveAdministrator:

• corrected and modified CRUD operations
• improved usability by providing detailed state information, adding dialogs etc.

[jzenisek]

r15760:

• minor changes regarding project- & resource selection
• adapted DeleteOnNull rules in dbml

[jzenisek]

r15761: renamed branch according to guidelines

[jzenisek]

r15767: fixed cyclic-relation bug in projects- and resources-view by preventing parents from being dragged onto descendants

[jzenisek]

r15768: fixed cyclic-relation bug (part 2: HiveAdminClient)

[jzenisek]

r15777: improved project- & resource selection handling

[jzenisek]

r15792: preliminary integration of several HiveAdministrator related improvements from #2877 (full merge pending)

• slaves and slave groups are sorted by name
• slave state is indicated by different colors
• number of contained slaves is shown for each group
• CPU utilization is shown for each calculating slave
• last heartbeat date is shown for each offline slave

[jzenisek]

r15813: improved HiveAdmin interactions:

• collaps groups with slaves only
• update scheduleView only if currently visible
• made schedule scrollable for non-admins
• changed selection backcolor
• fixed add/delete project permission bug for non-admins

[jzenisek]

r15819: implemented refreshing list of available (i.e. for non-admins assignable) resources depending on currently selected project

[pfleck]

I would also like to have some additional features (mainly GUI wise) for the "Nr. of needed cores" for the Hive tasks.

• Currently, the combobox to set the nr of cores is not working (no event registered), although, the rest of Hive already propagates the nr of needed cores correctly.

• For a scenario where a batch run of algorithms should be distributed, setting the nr of cores needed for all tasks is quite tedious. Maybe, setting the nr of cores needed on the batch run propagates the demand to its children.

• An option for Nr of needed cores that represents "all cores" could be useful (also with respect to executing ParallelEngines in Hive). Such a tasks should only be scheduled on a slave where no other tasks are currently running and the task will use up all available cores.

[jkarder]

r15908: worked on hive project management

• fixed some bugs
• cleaned up

[jzenisek]

r15913: updated genealogy computation for hive job administrator

[jzenisek]

r15914: fixed several bugs in resource selector

[jzenisek]

r15920: fixed several bugs in JobAdmin:

• disabled play button if no project or resource is selected
• extract stats for selected or alternatively for checked resources
• collapse slaves-only groups if no slave is assigned explicitly
• prevent unnecessary resource tree rebuilds (reset and style instead)
• suppress double clicks in resource tree (i.e. prevent win forms check-error)

...

[jzenisek]

r15922:

• replaced use of TreeView with NoDoubleClickTreeView everywhere allegedly reasonable for Hive
• updated genealogy computation for HiveAdminClient and corresponding views (ResourcesView & ProjectResourcesView)

[jzenisek]

r15925: fixed bug in HeartbeatManager (more specific: corrected check if a task via its parenting job is still granted for a certain resource)

[jzenisek]

r15933: several fixes:

• rebuilt run in hive
• adapted numeric selector (added FixedNumericUpDown.cs)

[jzenisek]

r15953: Several changes and fixes:

• added button tooltips
• changed position of buttons
• added refresh functionality to view of lightweight users ...

[jzenisek]

r15954: updated behavior of permission assignment for project owners (client and server side)

[jzenisek]

r15955:

• fixed refresh in resource view
• fixed assignment/include behavior in permission view

[jzenisek]

r15956:

• fixed project/resource selector in HiveJobManager
• added error provider to project/resource selector
• changed datetime pickers in HiveAdmin...

[jzenisek]

r15966: Fixed several bugs and added project duration handling

• fixed project duration handling (client- & service-side)
• fixed tagging in HiveJobAdmin
• added ProjectJobs view (under construction)
• added necessary service methods

[jzenisek]

r15969: finalized ProjectJobsView

[jzenisek]

r15978: applied several fixes:

• show full project-path in project/resource selector
• handle lost of project-ownership by not withdrawing permissions
• update automatically after hand-down save
• lock jobs for which statistics/deletion is pending
• lock the disabled checkbox in ProjectResourcesView...

[jzenisek]

r15992: fixed couple of minor issues:

• changed tags in resource selector
• added project information in job list and adapted sortation
• fixed hand-down save by withdrawing additional offset-rights (permissions, resources),...

[jzenisek]

r15995: adapted illustration of project and resource ancestry in HiveAdministrator and HiveJobAdministrator

[jzenisek]

r16040: fixed handling of project related updates for project owners, who are no admins and do not own parent projects

[jzenisek]

r16043: adapted handling of project start/end boundaries for non-admins

[jzenisek]

r16044:

• adapted icons: replaced ".PublishToWeb" with ".Save"
• pre-select project with all resources for new jobs, if it is the only available

[jkarder]

r16050: fixed connection string

[jkarder]

r16057:

• merged [15377-15681/trunk/sources] into branch
• merged [15683-16056/trunk] into branch

[jkarder]

r16060: worked on hive project management

• added .ToList() calls within some service methods to prevent TransactionExceptions
• added check for InvokeRequired
• improved RunInHiveMenuItem
• removed .resx files from projects

[jkarder]

r16062: worked on hive project management

• removed .resx file from project

[jkarder]

r16064: worked on hive project management

• fixed bug in StyleTreeNode

[jkarder]

r16066: worked on hive project management

• improved RunInHiveMenuItem

[jkarder]

r16068: worked on hive project management

• derived UserListView, UserGroupListView and RoleListView from ItemCollectionView<T>

[jzenisek]

r16072: worked on HiveProjectMgmt:

• adapted enabled state of buttons

[jzenisek]

r16089: updated setting enabled state of controls in project view

[jkarder]

r16091: worked on hive project management

• updated app/web configs and settings

[jkarder]

r16092: merged [16057-16091/trunk] into branch

[jkarder]

r16093: worked on hive project management

• removed usage of null-conditional operator

[jkarder]

r16094: worked on hive project management

• removed usage inline out variables and string interpolation

[jkarder]

r16095: worked on hive project management

• removed usage of null-conditional operator

[jkarder]

r16116: worked on hive project management

• fixed plugin dependencies
• fixed NullReferenceException in HiveTaskView
• added cloning ctor for AssignedJobResource and AssignedProjectResource

[jkarder]

r16117: merged [15377-16116/branches/2839_HiveProjectManagement] into trunk

[jkarder]

r16122: worked on hive project management

• fixed handling of enabled states in ResourcesView

[abeham]

Exceptions

Create a project and then delete it before it is saved:

System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
   at System.ThrowHelper.ThrowKeyNotFoundException()
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at HeuristicLab.Clients.Hive.Administrator.Views.ProjectsView.<removeButton_Click>d__13.MoveNext() in c:\...\HeuristicLab.Clients.Hive.Administrator\3.3\Views\ProjectsView.cs:line 268
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

Click on a user in the Permissions tab:

System.ArgumentException: View "HeuristicLab.Clients.Access.Views.RefreshableLightweightUserInformationView" cannot display content "HeuristicLab.Clients.Access.LightweightUser".
   at HeuristicLab.MainForm.WindowsForms.ViewHost.set_ViewType(Type value) in c:\...\HeuristicLab.MainForm.WindowsForms\3.3\Controls\ViewHost.cs:line 126
   at HeuristicLab.Clients.Hive.Administrator.Views.ProjectPermissionsView.treeView_AfterSelect(Object sender, TreeViewEventArgs e) in c:\...\HeuristicLab.Clients.Hive.Administrator\3.3\Views\ProjectPermissionsView.cs:line 120
   at System.Windows.Forms.TreeView.OnAfterSelect(TreeViewEventArgs e)
   at System.Windows.Forms.TreeView.TvnSelected(NMTREEVIEW* nmtv)
   at System.Windows.Forms.TreeView.WmNotify(Message& m)
   at System.Windows.Forms.TreeView.WndProc(Message& m)
   at HeuristicLab.Clients.Hive.Views.TreeView.NoDoubleClickTreeView.WndProc(Message& m) in c:\...\HeuristicLab.Clients.Hive.Views\3.3\TreeView\NoDoubleClickTreeView.cs:line 31
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

There is probably some dangling event handler that does not get deregistered. Opening Services > Access > User Information and clicking on the "Refresh" button results in the following exception after playing around with the projects:

ObjectDisposedException: Cannot access a disposed object.
Object name: 'ProjectView'.
   at System.Windows.Forms.Control.CreateHandle()
   at System.Windows.Forms.Control.get_Handle()
   at HeuristicLab.MainForm.WindowsForms.ControlExtensions.SuspendRepaint(Control control) in c:\...\HeuristicLab.MainForm.WindowsForms\3.3\Controls\ControlExtensions.cs:line 38
   at HeuristicLab.MainForm.WindowsForms.View.SuspendRepaint() in c:\...\HeuristicLab.MainForm.WindowsForms\3.3\Views\View.cs:line 255
   at HeuristicLab.MainForm.WindowsForms.View.set_Enabled(Boolean value) in c:\...\HeuristicLab.MainForm.WindowsForms\3.3\Views\View.cs:line 82
   at HeuristicLab.Clients.Hive.Administrator.Views.ProjectsView.SetEnabledStateOfControls() in c:\...\HeuristicLab.Clients.Hive.Administrator\3.3\Views\ProjectsView.cs:line 126
   at HeuristicLab.Clients.Hive.Administrator.Views.ProjectsView.AccessClient_Instance_Refreshing(Object sender, EventArgs e) in c:\...\HeuristicLab.Clients.Hive.Administrator\3.3\Views\ProjectsView.cs:line 183
   at System.EventHandler.Invoke(Object sender, EventArgs e)
   at HeuristicLab.Clients.Access.AccessClient.OnRefreshing() in c:\...\HeuristicLab.Clients.Access\3.3\AccessClient.cs:line 96
   at HeuristicLab.Clients.Access.AccessClient.<>c__DisplayClass8.<ExecuteActionAsync>b__6() in c:\...\HeuristicLab.Clients.Access\3.3\AccessClient.cs:line 74

Issues

• Creating a new project, then changing its name -> treeview is not updated until saved
• It's a bit odd that the save button for details is above the treeview. What does that button save?
• The "Slaves" tab should be renamed "Resources"
• I would use a fixed (approx.) 300px width of panel1 in the split control and keep panel1 size fixed during resize
• Trying to add HEAL resource group to a project that is not a sub-project of the HEAL project is not possible. The resources can be selected and save can be clicked. There is no exception or message that the save was not successful.
  • Btw, first creating that project under group HEAL, then dragging this to root level maintains the assigned resources. I can then create sub-projects where I can assign the same resources.

Questions

• Project Details: Why a refresh after the Owner combobox?
• What does "Permissions" of projects actually mean? What kind of permissions are there? Read, Execute?
• How can I create a new project that is not a sub-project of some other?
  • jkarder showed that this can be done using drag & drop. I think there should be a more obvious way.
• What kind of privilege is required to open the "Services > Access > User Administration". Being "Hive Administrator" doesn't seem to qualify: "Access Service Administrator - You do not seem to have the permissions to use the Access Service Administrator. If that's not the case or you have any questions please write an email to support..."
  • Access Service Administrator

[gkronber]

r16117 broke the HeuristicLab.MetaOptimization addon

HiveEngine.cs(298, 24): error CS1061: 'Job' does not contain a definition for 'ResourceNames' and no extension method 'ResourceNames' accepting a first argument of type 'Job' could be found (are you missing a using directive or an assembly reference?)

HiveEngine.cs(321, 60): error CS1061: 'IHiveService' does not contain a definition for 'DeleteJob' and no extension method 'DeleteJob' accepting a first argument of type 'IHiveService' could be found (are you missing a using directive or an assembly reference?)

[jkarder]

r16173:

• fixed compilation errors in HiveEngine
• minor changes

[jzenisek]

#r16184: fixed bug appeared when clicking on user

[jzenisek]

r16185:

• updated enabled state of controls for projects
• updated enabled state of controls for resources

[jzenisek]

r16186: fixed lightweightUserInformationView

[jzenisek]

r16187:

• fixed enabled state of controls bug for initial project selection regarding permission detail views
• fixed problem at resource assignment for root projects

[jzenisek]

r16202: fixed several bugs:

• renamed tab "Slaves" to "Resources"
• fixed bugs in job manager > resource selector reported by fholzinger
• adapted client-side handling of disabled resource ancestors
• adapted parentship check in HiveAdminClient
• adapted seting of enabled state of controls for project details, project resources
• introduced "inactive" tag in ProjectsView

[jzenisek]

r16203: fixed job listing method for hive administrators

[jzenisek]

r16204: added assembly reference

[jzenisek]

r16205: adapting project job view

[jzenisek]

r16208: updated job execution implementation in ProjectJobsView

[jzenisek]

r16209:

• adapted job execution implementation at ProjectJobsView
• prohibited resource checking for non-admins

[jzenisek]

r16211:

• disabled „Start Time“, „End Time“, „All Day Event“ checkbox for non admins
• reload ProjectResources on change of resources
• fixed bug in ProjectResources regarding ungroupedGroupNode

[jzenisek]

r16219: fixed several bugs:

• made additional slaves checkable
• changed default collapsing of additional slave group
• added info in project name text box

[jkarder]

r16257: worked on hive project management

• fixed GetAvailabilityStatsPerProjectQueryString

[jzenisek]

r16427:

• removed update button and functionality
• fixed bug in hive resource selector (resource update on click)
• performed some refactoring

[jkarder]

r16446: worked on hive project management

• improved event handler (de-)registration
• refactored content updates

[gkronber]

On my machine I can produce a deadlock from the Hive 'Select Project' dialog.

To reproduce:

1. Open GA - TSP Sample
2. Select 'Create Hive Job' in the menu
3. Click on the 'Magnifier Glass' icon to select a project
4. The dialog is opened but disabled
5. HL deadlocks and must be killed via task manager

HiveProjects Dialog.png​ src_line.png​ thread_1.png​ thread_2.png​

[pfleck]

Fixed the freezing "Select Project" Dialog bug in r16511:16512

[jkarder]

r16513: worked on hive project management

• fixed ArgumentNullException

[jkarder]

r16622: worked on hive project management

• Core.cs
  • locked checks for amount of available resources (cores/memory)
  • task state of aborted tasks is now updated accordingly
• JobDao.cs
  • improved readability of GetJobsReadyForDeletionQuery
• HiveStatisticsGenerator.cs
  • DimJobs, FactTasks and Jobs (DeletionPending) are now updated in one transaction

[gkronber]

Updating the resources tree view takes up to a minute from my connection at home.

[abeham]

Replying to gkronber:

Updating the resources tree view takes up to a minute from my connection at home.

Same for me. In the resource selector dialog (e.g. RefreshableJobView) it works reasonably fast, but in the Hive Administrator it takes quite some time.

[jkarder]

r16878: worked on hive project management

• improved loading times of hive administrator view

[jkarder]

Replying to abeham:

Replying to gkronber:

Updating the resources tree view takes up to a minute from my connection at home.

Same for me. In the resource selector dialog (e.g. RefreshableJobView) it works reasonably fast, but in the Hive Administrator it takes quite some time.

Fixed in r16878.

[mkommend]

r17059: Merged 16117, 16122, 16173, 16184, 16185, 16186, 16187, 16202, 16203, 16204, 16205, 16208, 16211, 16209, 16211, 16219, 16257, 16247 into stable.
r17060: Merged 16427 into stable.
r17063: Merged 16446 into stable.
r17065: Merged 16513 into stable.
r17067: Merged 16622, 16878 into stable.

[gkronber]

Change sets should be added correctly to the mergeinfo for stable branch (see r17059 ... )