How to ... setup Hive statistics

Overview

This page contains information and instructions on how to setup and run Hive statistics. It is assumed that you have already done all the previous steps described in SetupHiveServer and read SetupHiveSlave. Up until now you have a running Hive server where you can schedule tasks, create users and groups, have slaves and administer their timetables. To see what's going on in the system (e.g. how many jobs are waiting, calculating, who does what, is everything running smoothly, and so on), Hive statistics provides this information in the form of a web application. While normal users can see some general information and information concerning their own jobs, administrators can view more comprehensive information and see what users are doing. This information is also persistent, meaning that deleting a job does not delete it's execution history.

The Hive statistics sources can be found in the HeuristicLab 3.3. Services solution in the HeuristicLab.Services.WebApp* projects:

• HeuristicLab.Services.WebApp-3.3.: This is a general frame for HL and Hive web applications. It is plugin-based and all other parts of the web application are just plugins that get discovered and displayed by the WebApp.
• HeuristicLab.Services.WebApp.Status-3.3: Shows an overview of what's going on in Hive:

• HeuristicLab.Services.WebApp.Statistics-3.3: Shows the information collected by the Hive janitor service. This is detailed historic data about jobs and slaves.
• HeuristicLab.Services.WebApp-.Maintenance3.3.: An app for viewing how much size the database consumes and some methods for compacting data in the db.

Installation

• Compile the WebApp projects in the HeuristicLab 3.3 Services solution
• Create a new application in IIS (like the Hive-3.3 application) and call it WebApp (please take care, like with Hive-3.3, to have the same app pool, file and database permissions and connection strings)
• Copy the HeuristicLab.Services.WebApp\3.3\bin folder from the sources to your server into the newly created web application (WebApp) folder and delete the WebApp folder from bin.
• Copy the folder HeuristicLab.Services.WebApp\3.3\WebApp into the WebApp folder on the server.
• Copy HeuristicLab.Services.WebApp\3.3\Web.config and HeuristicLab.Services.WebApp\3.3\Global.asax into the WebApp folder on the server
• Review and adapt the Web.config file to your configuration
• If you now browse to ​http://servername/WebApp you should see the skeleton web app with a login, the list of loaded plugins and the about page
• You can now also install the other plugins (status, statistics, maintenance) by creating new directories in WebApp\WebApp\plugins\ on the server and copying the contents of the projects bin directory to this location. For e.g. status the path would be WebApp\WebApp\plugins\status\ and the contents of this directory would be the contents of e.g. HeuristicLab.Services.WebApp.Status\3.3\bin\Debug
• If you reload the WebApp in the browser, you should now see new entries in the left pane. Please note that this may depend on your user and rights.
• If you have installed all 3 plugins, the final folder structure on the server should look like this:

If you want to create your own plugin, have a look at the WebApp Howto