How to ... setup Hive statistics

Overview

This page gives you information and instructions about how to setup and run Hive statistics. It is assumed that you have already done all the previous steps described in SetupHiveServer and have read SetupHiveSlave. Up until now you have a running Hive server where you can schedule tasks, create users and groups, have slave clients and administer their timetables when they are allowed to calculate. But you don't see what's going on in the system (e.g. how many jobs are waiting, calculating, who does what, is everything running smoothly, and so on). That's what Hive statistics is for: It is a web application which provides an overview of what is going on in the system. While normal users can see some general information and information concerning their own jobs, administrators can view more comprehensive information and also see what single users are doing. This information is also persistent, meaning that deleting a job does not delete it's execution history.

The Hive statistics sources can be found in the HeuristicLab 3.3. Services solution in the HeuristicLab.Services.WebApp* projects:

• HeuristicLab.Services.WebApp-3.3.: This is a general frame for HL and Hive web applications. It is plugin-based and all other parts of the web application are just plugins that get discovered by the WebApp and then get displayed.
• HeuristicLab.Services.WebApp.Status-3.3: Shows an overview of what's going on in Hive:

• HeuristicLab.Services.WebApp.Statistics-3.3: Shows the information collected by the Hive janitor service. This is detailed historic data about jobs and slaves.
• HeuristicLab.Services.WebApp-.Maintenance3.3.: Just a small app for viewing how much size the database consumes and some methods for compacting data in the db.

Installation

• Compile the WebApp projects in the HeuristicLab 3.3 Services solution
• Create a new application in IIS (like the Hive-3.3 application) and call it WebApp (please take care, like with Hive-3.3, to have the same app pool, file and database permissions and connection strings)
• Copy the HeuristicLab.Services.WebApp\3.3\bin folder from the sources to your server into the newly created web application (WebApp) folder and delete from bin the WebApp folder.
• Copy the folder HeuristicLab.Services.WebApp\3.3\WebApp into the WebApp folder on the server.
• Copy HeuristicLab.Services.WebApp\3.3\Web.config and HeuristicLab.Services.WebApp\3.3\Global.asax into the WebApp folder on the server.
• Review and adapt the Web.config file to your configuration.
• If you now browse to ​http://servername/WebApp you should see the skeleton web app with a login, the list of loaded modules and the about page.
• You can now also install the other plugins (status, statistics, maintenance) by creating on the server new directories in WebApp\WebApp\plugins\ and copying the contents of the projects bin directory to this location, e.g. for status the path would be WebApp\WebApp\plugins\status\ and the contents of this directory would be content of e.g. HeuristicLab.Services.WebApp.Status\3.3\bin\Debug.
• If you reload the WebApp in the browser, you should now see new entries in the left pane. Please note that this may depend on your user and rights.
• If you have installed all 3 plugins, the final folder structure on the server should look like this:

For more information or if you want to create your own plugin, have a look at the WebApp Howto