Opened 6 years ago

Closed 4 years ago

#2063 closed enhancement (obsolete)

Implement Hive Statistics

Reported by: pfleck Owned by: ascheibe
Priority: medium Milestone: HeuristicLab 3.3.x Backlog
Component: Hive.Server Version: branch
Keywords: Cc:

Description


Change History (28)

comment:1 Changed 6 years ago by pfleck

  • Status changed from new to accepted

comment:2 Changed 6 years ago by pfleck

r9522

Created Branch for Hive-Statistics.

comment:3 Changed 6 years ago by pfleck

r9524

Added statistic tables in Hive database.

Updated InitializeHiveDatabase.sql script with new create statements.

Added Linq2Sql entity types for new tables.

comment:4 Changed 6 years ago by pfleck

r9526

Added StatisticsGenerator for generating statistic data for Hive.

ServiceLocator returns the StatisticsGenerator.

Integrated StatisticsGenerator in Janitor for periodic calls.

comment:5 Changed 6 years ago by pfleck

r9527

Provides a GUI for SelfHosting Hive-Service to start and stop the service.

Enables start and stop Janitor with adjustable frequency.

Run Cleanup and GenerateStatistics separately on click.

comment:6 Changed 6 years ago by pfleck

r9533

StatisticsGenerator now updates Time, Job and User dimension-tables.

comment:7 Changed 6 years ago by pfleck

r9538

StatisticsGenerator fills and updates client dimension table.

Removed week from time dimension table.

comment:8 follow-up: Changed 6 years ago by pfleck

r9541

UserId in ClientInfo fact table is nullable. UserId is removed from primary key therefore.

StatisticsGenerator insert new ClientInfo into fact table.

comment:9 in reply to: ↑ 8 Changed 6 years ago by gkronber

Replying to pfleck:

r9541

UserId in ClientInfo fact table is nullable. UserId is removed from primary key therefore.

Not a good idea. Instead, use a surrogate key for the user id and keep an entry for the 'NULL' user in the DimUser table. This entry has non-null surrogate key value and NULL UserId value.

comment:10 Changed 6 years ago by pfleck

r9545

ClientInfo facts are now generated correctly when resources are changed and new client dimensions are created.

Query username for statistics tables from user id.

comment:11 Changed 6 years ago by pfleck

r9546

As suggested by gkronber, added user to primary key again in ClientFact table and added a 'NULL'-user in the DimUser table instead.

comment:12 Changed 6 years ago by pfleck

r9547

Changed asp.net membership provider for querying usernames.

comment:13 Changed 6 years ago by pfleck

r9557

Added DataSize column in TaskData (calculated column).

Created queries for fetching required data for generating task facts.

comment:14 Changed 6 years ago by pfleck

r9560

Query last client for task facts and calculate task fact data. (TotalRuntime/Wait/TransferRime pending)

comment:15 Changed 6 years ago by pfleck

r9561

Simplified and optimized query for task fact statistics.

comment:16 Changed 6 years ago by pfleck

r9571

Calculating total waiting-, transfer- and calculating time for task statistics.

comment:17 Changed 6 years ago by pfleck

r9578

Task statistics contains only current state and no change history.

Statistics calculation is no longer based on increment updates.

TaskId is primary key in task statistics and EndTime is nullable.

Added TaskState to Task statistics.

comment:18 Changed 6 years ago by pfleck

r9589

Added Asp.net MVC project and all required libraries.

Added publish profile to web services and web pages.

comment:19 Changed 6 years ago by pfleck

r9604

Set up basic webpage layout based on WebApplication branch.

Added Asp.Net MVC packages and some helper packages.

Implemented login.

comment:20 Changed 6 years ago by pfleck

r9617

Started integrating Hive statistics into statistics web project.

Added jqPlot library for charting.

comment:21 Changed 6 years ago by pfleck

r9625

Insert charts on index page for CPU utilization, cores and memory usage.

Added html helper for generating chart javascript.

Added ChartDataController for generating JSON data for the charts.

comment:22 Changed 6 years ago by pfleck

r9643

Added time selection for charts.

Implemented redrawing of charts using Ajax.

comment:23 Changed 6 years ago by pfleck

r9644

Fixed timezone bug in chart data.

Fixed zooming bug in jqplot dateAxisRenderer. (https://bitbucket.org/cleonello/jqplot/pull-request/29/removed-unbalanced-addition-of-utc-offset/diff)

Reformatting

comment:24 Changed 6 years ago by pfleck

r9646

Changed year in headers from 2012 to 2013.

Minor reformattings in chart helpers.

Refactored and simplified some statistics generator methods.

comment:25 Changed 6 years ago by pfleck

r9661

Updated wrong endpoint address.

Re-enabled performance counters for slaves.

comment:26 Changed 6 years ago by pfleck

r9662

Rearranged axis options in jqplot charts.

Renamed selectors into seriesSelectors in CreateSeriesData.

Changed default date range on index page.

comment:27 Changed 6 years ago by ascheibe

  • Owner changed from pfleck to ascheibe
  • Status changed from accepted to reviewing

comment:28 Changed 4 years ago by ascheibe

  • Resolution set to obsolete
  • Status changed from reviewing to closed

The work from this ticket has already been released with #2388

Note: See TracTickets for help on using tickets.