Optimization-as-a-Service

[spimming]

r8228 initial version for optimization as a service branch

[spimming]

r8232 scripts for sql azure db adapted

[spimming]

r8235 Hive server web role initial commit

[spimming]

r8242

• slave worker role initial commit
• start client communication service optionally
• make slave client communication interchangeable ('PipeCom', 'TraceCom')

[spimming]

r8243 added pre-build event to project properties

[spimming]

r8244

• added signing certificates
• added status pages

[spimming]

r8251

• Change hive server from service configuration file
• Created service configuration settings for server address and certificate
• Added HeuristicLab.Clients.Hive-3.3 project to solution

[spimming]

r8265

• calling roles caused an exception
• fix: call default provider instance directly

[spimming]

r8266

• disabled some transactions since dtc is not supported in sql azure
• settings in dataaccess updated

[spimming]

r8270 Caution: disabled transactions in the TransactionManager

[fschoepp]

r8326

• Added IConnectionProvider interface + implementation based on jheinzelreiters transaction management
• Fixed transaction being promoted to dtc; transactions enabled again
• Changed HiveDao: transactions are managed properly

[fschoepp]

r8384

• Created a project which contains the back-end controller for the Optimization WebSite
• Added a WCF-back-end-controller which generates all available optimization problems (currently in-memory solution: PlaceholderControllerService.cs)
• Created a WebRole using ASP.NET MVC 3 for the Optimization Web Site
• WebSite authenticates users with the HeuristicLab.Authentication membership provider and database
• WebSite crawls and displays all available optimization scenarios by using the WCF-back-end controller (test with: ​http://localhost:.../optimization)

[fschoepp]

r8506

• Web project now creates custom html for each type we want to enter / display
• Added endpointConfigurationName to HiveServiceLocator (because Web Project contains more than one endpoint configuration)
• Removed logging statement from ConfigurationService to prevent exception during failure of loading ConfigurationSettings
• ApplicationManager: Changed default implementation to WebApplicationManager (instead of LightWeight) for testing purposes within Web Project
• WebApplicationManager: The application manager which returns plugin descriptors from the currently loaded assemblies (instead of LightweightAppManager)
• HiveService: Fixed a transaction bug
• IControllerService: Created a method to dispatch Scenarios to certain IScenarioManager (in this case HiveScenarioManager)
• Added more mappable types to ControllerModel
• PlaceholderControllerService dispatches all Scenarios to the HiveScenarioManager
• Web project now dispatches the scenario to the controller after pressing "Run Job"

[fschoepp]

r8545

• Controller is now capable of gathering Hive Jobs
• Hive Jobs will be mapped to independent Job-Class (shared between Controller and Frontend)
• HiveScenarioManager is capable of gathering Hive Jobs + their results
• Job Results will be mapped to string properties
• Frontend renders all results after opening the job details
• Misc: Frontend now passes User-object to Controller so that it is able to connect to the Hive Service (hardcoded values removed)

[fschoepp]

r8817

• Added a parser for independent scenarios (using the model of the optimization backend)
• Optimization scenario sample can be found in mappings folder of the web project.
• Added IScenarioMapper interface which provides functionality to map from the optimization data model to a backend model (e.g. Heuristic Lab data model)
• Implementations of IScenarioMapper have to be provided as C# code (strings) which will be compiled by using a CSharpCodeProvider. Once compiled, the implementations of the IScenarioMapper are being cached within the platform for further usage.
• Fixed a bug in web template DecimalMatrix (using i instead of j)
• Added missing thumprint of localhost certificate to the optimization web project (ServiceConfiguration.Local.cscfg / ServiceConfiguration.Cloud.cscfg)
• Test project now provides following test cases: Mapping types using IronPython and mapping types using Otis

[fschoepp]

r8958

• Added a administrator web interface for job management
• Fixed Hive Client (PluginValidator) to find the assemblies within the right directories
• Reorganized controller classes (Folders HL, Interfaces, Azure)
• You may now successfully schedule and run jobs with the web ui.

[fschoepp]

r9062

Backend changes:

• Simplified job state detection (only one hive call will be made to detect all states now, instead of one additional call per job)
• Reorganized classes (moved model classes into Model folder)

Website changes:

• Website now heavily uses JavaScript to achieve better user experience
• JavaScript degrades gracefully, except for plots
• Tables: Added jquery-datatable-plugin to extend tables (pagination + search functionality)
• OaaS-Website now uses the design of the HL websites (found in WebApplication branch)
• Added jqplot to render zoomable line plots for HL-Datatables
• Styling.js: Plots will be generated by using an ajax call; additional jquery-styling occurs within this file.
• Added jquery-ui-1.9.2 which is capable of handling/rendering tabs, accordions and resizers.

[fschoepp]

r9166

• Model: OptimizationScenario may be a tree of algorithms (and problems)
• Model: Renamed InputParameters to ProblemParameters (as they are the parameters of a problem)
• Model: Added JobExecutionDetails which contain Repetitions + Group (resource to use)
• ScenarioParser parses the new XML scenario files
• Website + Model: You are now able to add/remove rows from a table (no JavaScript involved yet)
• Website + Controller: Added repetitions (enables batch jobs) and group (resource to use) to OaaS which will be used by the controller to schedule the job
• Website: Updated templates to use new model structure
• Website + Scenarios: Added the new algorithm Benchmark Algorithm
• Controller: Added a singleton to make the (Azure/Mockup)-DAL exchangeable
• Controller: Added mockup classes for DAL + IScenarioManager
• Website/Result Page: Line Diagrams will be added via JavaScript, crawling their data using AJAX
• Website: Most configuration parameters can be set in the ServiceDefinition directly
• Added a mockup for the Membership classes: These can be used if no network connection is available or if other parts of the app shall be tested
• Scenarios: Updated TSP mappings to new xsd

[fschoepp]

r9215

• Janitor is now working as expected in Windows Azure
• Added basic support for experiments (draggable experiments)
• Added methods to save/read experiments from TableStorage
• The job status can now be retrieved by using the GetTasks/GetTaskData methods
• Added a class to convert JSON-objects to Algorithm instances
• Web page: Added experiment button to navigation

[fschoepp]

r9227:

• Experiments will be saved as JSON elements within the blob store.
• Added simple model and JSON converters.
• Backend stores and runs experiments.
• Updated interfaces to save/read experiments.
• Added a binding to automatically map incoming JSON ajax requests to experiment models.
• Added the javascript DatatypeMapper to map parameter inputs to the right html elements and vice versa.
• Added smartwizard to generate Wizards for creating new experiments (New.cshtml).

[fschoepp]

r9253, r9254, r9255, r9256:

• Fixed build of OaaS-HL (removed an unnecessary reference to a .resx file).
• Build order is as follows: HeuristicLab.ExtLibs, HeuristicLab 3.3, HeuristicLab 3.3 Services (Administrator)
• Added missing respositories.config for nuget (Heuristiclab/branches/OaaS/packages).
• Added missing nuget references (Json).

[fschoepp]

r9305:

• Added Update / GetExperiment... methods to the controller for updating and querying experiments.
• The AlgorithmConverter class now properly converts from/to JSON format.
• Integrated backbone js as MVC provider for JavaScript + jquery.
• Added experiment.model.js + experiment.view.js + experiment.controller.js containing the MVC impl. for the Experiment pages.
• Added new methods to the ExperimentController usable by the backbone js model implementation.
• Added the experiment dialog from HL 3.3.7 (variate experiment parameters). It's capable of variating the algorithm parameters.
• JavaScript is a must have for the "Edit Experiment" / "Build Experiment" pages.

[fschoepp]

r9324:

• DAL: Added a Delete method which deletes by experiment id.
• HL DataTables will now be transposed and mapped as double[ROWS][COLUMNS] (transposed)
• JS: Moved all classes into "modules" to prevent namespace pollution (using OAAS_MODEL for model classes, OAAS_VIEW for views and OAAS_CONTROLLER for controllers)
• JS: Moved DatatypeMapper classes into Backbone views
• JS: Models now correctly send DELETE requests
• Added a new job overview page (which also renders run details) using AJAX
• Using moment.min.js to format DateTime as string
• Controllers now inherit from BaseController which provides a RedirectToLoginIfNecessary-method
• Added loading animations to several AJAX bound places (loading experiments / scenarios)
• Added a section to _Layout.cshtml which allows page-specific JavaScript includes (<script> only for a certain page)
• Fixed Build/Edit of experiment menu redirecting to the wrong page
• The Experiment Variation Dialog disables input fields, if the property has not been activated before

[fschoepp]

r9335:

• The Edit-Experiment-Page now correctly displays the experiments again.
• Disabled key navigation for all wizards.
• Added a RunCollectionView to render different Plots for the run results of a job.
• Added a Boxplot-Chart and a Bubble-Chart for rendering the results.
• Added underscoreAddon.js which provides methods to calculate mean/sum of an JavaScript array/list.

[fschoepp]

r9350: #1888:

• Added input parameters to the run class. They will be populated by the back-end and returned to the web pages which renders them.
• Added a ParameterMapper class which converts HL specific model classes to OaaS independent classes. The conversion gets delegated to IParameterHandler which have to be implemented manually and registered for a certain Type before. All parameters which can be converted to IStringConvertible, IStringConvertibleMatrix, IString* will be converted into a OaaS-StringValue instances.
• Added IParameterHandlers for PathTSPTour and PermutationType (required for TSP).
• AlgorithmConverter now makes sure that the id of a run is unique. (All runs of a RunList will be shown now.)
• Web pages are capable of rendering both the results of a run and their input parameters (added a accordion to wrap their content).
• Renamed "Traveling Salesman Problem" to "Genetic Algorithm - TSP".
• Changed js-files to render both input and result parameters of a Run.

[fschoepp]

r9362:

• Added IVisualExtensionDao to the backend representing js extension that create new viewable elements of an algorithm.
• Started to upgrade the frontend to render those new javascript UI extensions.

[spimming]

r9363:

• Merged revisions from trunk

[fschoepp]

r9365:

• WorkerRole of Slave now logs all exceptions during startup to the blobstore (slavelog).
• The PluginManager throws an exception during CheckWorkingDirectories() (Save method) which will be caught now, preventing crashes in Windows Azure.
• "db.DeferredLocal = false" has been removed to prevent loading bugs.
• HiveScenarioManager doesn't crash anymore, if he can't find an algorithm within a job during retrieval of the run results.

[fschoepp]

r9366, r9367, r9370, r9371, r9375:

• Removed unnecessary files from Optimization.Web project
• Added missing HeuristicLab 3.3.exe.config to optimization project
• Added necessary HL exe file
• Marked necessary configuration steps with ADJUST_ME
• Getting connection information from configuration file

[fschoepp]

r9395:

• Added visual extensions (dynamic JavaScript) which will be used to render additional result parameters specific to scenarios (e. g. create a graphical representation of a TSP).
• Added relationship between jobs and experiments (otherwise, it's not possible to get the job's experiment).
• Updated Admin page to allow removal/addition of visual extensions.
• Added back-end logic to store/retrieve/delete visual extensions.
• Added visual extension functionality to the JavaScript views/controllers (job.*.js).
• Added tsp.js which is a visual extension for the "Genetic Algorithm - TSP" scenario. It adds a graphical representation of the TSP (just like the C# version) to the results.

[fschoepp]

r9508: HL:

• Web projects requires different users to interact with hive. The singleton HiveServiceLocator.Instance doesn't allow different users at the same time, resulting in serialization during access of HiveClient methods.

The following changes have been introduced in favor of a parallel use of the HL libs:

• HiveClient, TaskDownloader and ConcurrentTaskDownloader may now use a different IHiveServiceLocator than HiveServiceLocator.Instance (all methods have appropriate overloads now).
• The default instance is still HiveServiceLocator.Instance.

Automated Scaling of Instances:

• Added Scaler project to solution which represents a WorkerRole that scales the slave instances based on the global cpu utilization of all slaves.
• Scaler is based on WASABi, rules can be adjusted in rulesstore.xml. Basic rule is: if < 45% global cpu utilization => remove an instance; if > 65% cpu => add an instance. Minimum boundary is 1 and maximum boundary is 8 slave instances.
• Adjusted Slave project to automatically register itself to a SlaveGroup during WebRole startup (can be adjusted in service configuration).

Web-Frontend:

• Added basic error messages to the dialogs when an ajax call fails.
• Removed Styling.js from scripts.

[fschoepp]

r9549:

• Removed unnecessary files from web project.
• Cleaned up slave.

[spimming]

r9576

• Added billing test client
• Use entity framework for DAL
• Modified service interface and model classes

[spimming]

r9577

• Added dal and service interfaces and implementations
• Added dbcontext for billing

[spimming]

r9579

• Revised model classes
• Modified relationship definitions according to model types

[fschoepp]

r9582

• Added an overview for users to inspect their orders
• Order Administrators may now suspend or reactivate orders
• When creating an order, its necessary to enter payment information (paypal, credit card,...) before
• Also, the billing period and type must be entered during the creation of an order.

[fschoepp]

r9586

• Billing Component may now be deactivated by setting the BillingEnabled flag within the OaaS service configuration.
• Added views for invoices and usage data.
• Changed appearance of the SaveOrder view.

[fschoepp]

r9588

• Removed readme from scaler
• Added BillingEnabled flag to configuration files

[spimming]

r9602

• Workaround for enums in model classes
• Implemented dao methods for model
• Added test data in db seed method
• changed EF configuration
• Implemented some BL methods

[spimming]

r9619

• Added new BillingService methods
• Disabled proxy generation and lazy loading!
• Extended see method with additional test data
• Added properties to order and invoice model
• Initial commit of BillingEngine module

[spimming]

r9641

• Users now contain ContactInformation
• Fixed formatting in invoice template
• Fixed calculation for next invoice date
• Worked on indentation for invoice fields

[spimming]

r9645

• Model classes implement new interface to track current state of entity
• Mark EntityState property as 'not mapped'
• Set hook to initialize entity as unchanged
• Extension method for DbContext to apply changes only on modified entity

[spimming]

r9653

• Set flag to indicate a modified model object
• Add dao query method to specify includes as parameter
• cleaned up the using of namespaces to qualify types

[spimming]

r9654

• Added model and view for contact information
• Insert contact information on SaveOrder page
• Set correct entity state

[gkronber]

Please accept this ticket.

[spimming]

r10013

• enabled transactions
• enabled tracing output
• set correct invoice data
• saving invoices

[jkarder]

HL architects: swagner will rename this branch

[abeham]

r16115: renamed branch