Opened 5 years ago

Last modified 9 months ago

#1888 assigned task

Optimization-as-a-Service

Reported by: spimming Owned by: swagner
Priority: medium Milestone: HeuristicLab 3.3.x Backlog
Component: General Version: branch
Keywords: Cc:

Description


Change History (52)

comment:1 Changed 5 years ago by spimming

  • Status changed from new to assigned

comment:2 Changed 5 years ago by spimming

r8228 initial version for optimization as a service branch

comment:3 Changed 5 years ago by spimming

r8232 scripts for sql azure db adapted

comment:4 Changed 5 years ago by spimming

r8235 Hive server web role initial commit

comment:5 Changed 5 years ago by spimming

r8242

  • slave worker role initial commit
  • start client communication service optionally
  • make slave client communication interchangeable ('PipeCom', 'TraceCom')

comment:6 Changed 5 years ago by spimming

r8243 added pre-build event to project properties

comment:7 Changed 5 years ago by spimming

r8244

  • added signing certificates
  • added status pages

comment:8 Changed 5 years ago by spimming

r8251

  • Change hive server from service configuration file
  • Created service configuration settings for server address and certificate
  • Added HeuristicLab.Clients.Hive-3.3 project to solution

comment:9 Changed 5 years ago by spimming

r8265

  • calling roles caused an exception
  • fix: call default provider instance directly

comment:10 Changed 5 years ago by spimming

r8266

  • disabled some transactions since dtc is not supported in sql azure
  • settings in dataaccess updated

comment:11 Changed 5 years ago by spimming

r8270 Caution: disabled transactions in the TransactionManager

comment:12 Changed 5 years ago by fschoepp

r8326

  • Added IConnectionProvider interface + implementation based on jheinzelreiters transaction management
  • Fixed transaction being promoted to dtc; transactions enabled again
  • Changed HiveDao: transactions are managed properly

comment:13 Changed 5 years ago by fschoepp

  • Owner changed from spimming to fschoepp

comment:14 Changed 5 years ago by fschoepp

r8384

  • Created a project which contains the back-end controller for the Optimization WebSite
  • Added a WCF-back-end-controller which generates all available optimization problems (currently in-memory solution: PlaceholderControllerService.cs)
  • Created a WebRole using ASP.NET MVC 3 for the Optimization Web Site
  • WebSite authenticates users with the HeuristicLab.Authentication membership provider and database
  • WebSite crawls and displays all available optimization scenarios by using the WCF-back-end controller (test with: http://localhost:.../optimization)

comment:15 Changed 5 years ago by fschoepp

r8506

  • Web project now creates custom html for each type we want to enter / display
  • Added endpointConfigurationName to HiveServiceLocator (because Web Project contains more than one endpoint configuration)
  • Removed logging statement from ConfigurationService to prevent exception during failure of loading ConfigurationSettings
  • ApplicationManager: Changed default implementation to WebApplicationManager (instead of LightWeight) for testing purposes within Web Project
  • WebApplicationManager: The application manager which returns plugin descriptors from the currently loaded assemblies (instead of LightweightAppManager)
  • HiveService: Fixed a transaction bug
  • IControllerService: Created a method to dispatch Scenarios to certain IScenarioManager (in this case HiveScenarioManager)
  • Added more mappable types to ControllerModel
  • PlaceholderControllerService dispatches all Scenarios to the HiveScenarioManager
  • Web project now dispatches the scenario to the controller after pressing "Run Job"

comment:16 Changed 5 years ago by fschoepp

r8545

  • Controller is now capable of gathering Hive Jobs
  • Hive Jobs will be mapped to independent Job-Class (shared between Controller and Frontend)
  • HiveScenarioManager is capable of gathering Hive Jobs + their results
  • Job Results will be mapped to string properties
  • Frontend renders all results after opening the job details
  • Misc: Frontend now passes User-object to Controller so that it is able to connect to the Hive Service (hardcoded values removed)

comment:17 Changed 4 years ago by fschoepp

r8817

  • Added a parser for independent scenarios (using the model of the optimization backend)
  • Optimization scenario sample can be found in mappings folder of the web project.
  • Added IScenarioMapper interface which provides functionality to map from the optimization data model to a backend model (e.g. Heuristic Lab data model)
  • Implementations of IScenarioMapper have to be provided as C# code (strings) which will be compiled by using a CSharpCodeProvider. Once compiled, the implementations of the IScenarioMapper are being cached within the platform for further usage.
  • Fixed a bug in web template DecimalMatrix (using i instead of j)
  • Added missing thumprint of localhost certificate to the optimization web project (ServiceConfiguration.Local.cscfg / ServiceConfiguration.Cloud.cscfg)
  • Test project now provides following test cases: Mapping types using IronPython and mapping types using Otis

comment:18 Changed 4 years ago by fschoepp

r8958

  • Added a administrator web interface for job management
  • Fixed Hive Client (PluginValidator) to find the assemblies within the right directories
  • Reorganized controller classes (Folders HL, Interfaces, Azure)
  • You may now successfully schedule and run jobs with the web ui.

comment:19 Changed 4 years ago by fschoepp

r9062

Backend changes:

  • Simplified job state detection (only one hive call will be made to detect all states now, instead of one additional call per job)
  • Reorganized classes (moved model classes into Model folder)

Website changes:

  • Website now heavily uses JavaScript to achieve better user experience
  • JavaScript degrades gracefully, except for plots
  • Tables: Added jquery-datatable-plugin to extend tables (pagination + search functionality)
  • OaaS-Website now uses the design of the HL websites (found in WebApplication branch)
  • Added jqplot to render zoomable line plots for HL-Datatables
  • Styling.js: Plots will be generated by using an ajax call; additional jquery-styling occurs within this file.
  • Added jquery-ui-1.9.2 which is capable of handling/rendering tabs, accordions and resizers.

comment:20 Changed 4 years ago by fschoepp

r9166

  • Model: OptimizationScenario may be a tree of algorithms (and problems)
  • Model: Renamed InputParameters to ProblemParameters (as they are the parameters of a problem)
  • Model: Added JobExecutionDetails which contain Repetitions + Group (resource to use)
  • ScenarioParser parses the new XML scenario files
  • Website + Model: You are now able to add/remove rows from a table (no JavaScript involved yet)
  • Website + Controller: Added repetitions (enables batch jobs) and group (resource to use) to OaaS which will be used by the controller to schedule the job
  • Website: Updated templates to use new model structure
  • Website + Scenarios: Added the new algorithm Benchmark Algorithm
  • Controller: Added a singleton to make the (Azure/Mockup)-DAL exchangeable
  • Controller: Added mockup classes for DAL + IScenarioManager
  • Website/Result Page: Line Diagrams will be added via JavaScript, crawling their data using AJAX
  • Website: Most configuration parameters can be set in the ServiceDefinition directly
  • Added a mockup for the Membership classes: These can be used if no network connection is available or if other parts of the app shall be tested
  • Scenarios: Updated TSP mappings to new xsd

comment:21 Changed 4 years ago by fschoepp

r9215

  • Janitor is now working as expected in Windows Azure
  • Added basic support for experiments (draggable experiments)
  • Added methods to save/read experiments from TableStorage
  • The job status can now be retrieved by using the GetTasks/GetTaskData methods
  • Added a class to convert JSON-objects to Algorithm instances
  • Web page: Added experiment button to navigation

comment:22 Changed 4 years ago by fschoepp

r9227:

  • Experiments will be saved as JSON elements within the blob store.
  • Added simple model and JSON converters.
  • Backend stores and runs experiments.
  • Updated interfaces to save/read experiments.
  • Added a binding to automatically map incoming JSON ajax requests to experiment models.
  • Added the javascript DatatypeMapper to map parameter inputs to the right html elements and vice versa.
  • Added smartwizard to generate Wizards for creating new experiments (New.cshtml).

comment:23 Changed 4 years ago by fschoepp

r9253, r9254, r9255, r9256:

  • Fixed build of OaaS-HL (removed an unnecessary reference to a .resx file).
  • Build order is as follows: HeuristicLab.ExtLibs, HeuristicLab 3.3, HeuristicLab 3.3 Services (Administrator)
  • Added missing respositories.config for nuget (Heuristiclab/branches/OaaS/packages).
  • Added missing nuget references (Json).

comment:24 Changed 4 years ago by fschoepp

r9305:

  • Added Update / GetExperiment... methods to the controller for updating and querying experiments.
  • The AlgorithmConverter class now properly converts from/to JSON format.
  • Integrated backbone js as MVC provider for JavaScript + jquery.
  • Added experiment.model.js + experiment.view.js + experiment.controller.js containing the MVC impl. for the Experiment pages.
  • Added new methods to the ExperimentController usable by the backbone js model implementation.
  • Added the experiment dialog from HL 3.3.7 (variate experiment parameters). It's capable of variating the algorithm parameters.
  • JavaScript is a must have for the "Edit Experiment" / "Build Experiment" pages.

comment:25 Changed 4 years ago by fschoepp

r9324:

  • DAL: Added a Delete method which deletes by experiment id.
  • HL DataTables will now be transposed and mapped as double[ROWS][COLUMNS] (transposed)
  • JS: Moved all classes into "modules" to prevent namespace pollution (using OAAS_MODEL for model classes, OAAS_VIEW for views and OAAS_CONTROLLER for controllers)
  • JS: Moved DatatypeMapper classes into Backbone views
  • JS: Models now correctly send DELETE requests
  • Added a new job overview page (which also renders run details) using AJAX
  • Using moment.min.js to format DateTime as string
  • Controllers now inherit from BaseController which provides a RedirectToLoginIfNecessary-method
  • Added loading animations to several AJAX bound places (loading experiments / scenarios)
  • Added a section to _Layout.cshtml which allows page-specific JavaScript includes (<script> only for a certain page)
  • Fixed Build/Edit of experiment menu redirecting to the wrong page
  • The Experiment Variation Dialog disables input fields, if the property has not been activated before

comment:26 Changed 4 years ago by fschoepp

r9335:

  • The Edit-Experiment-Page now correctly displays the experiments again.
  • Disabled key navigation for all wizards.
  • Added a RunCollectionView to render different Plots for the run results of a job.
  • Added a Boxplot-Chart and a Bubble-Chart for rendering the results.
  • Added underscoreAddon.js which provides methods to calculate mean/sum of an JavaScript array/list.

comment:27 Changed 4 years ago by fschoepp

r9350: #1888:

  • Added input parameters to the run class. They will be populated by the back-end and returned to the web pages which renders them.
  • Added a ParameterMapper class which converts HL specific model classes to OaaS independent classes. The conversion gets delegated to IParameterHandler which have to be implemented manually and registered for a certain Type before. All parameters which can be converted to IStringConvertible, IStringConvertibleMatrix, IString* will be converted into a OaaS-StringValue instances.
  • Added IParameterHandlers for PathTSPTour and PermutationType (required for TSP).
  • AlgorithmConverter now makes sure that the id of a run is unique. (All runs of a RunList will be shown now.)
  • Web pages are capable of rendering both the results of a run and their input parameters (added a accordion to wrap their content).
  • Renamed "Traveling Salesman Problem" to "Genetic Algorithm - TSP".
  • Changed js-files to render both input and result parameters of a Run.

comment:28 Changed 4 years ago by fschoepp

r9362:

  • Added IVisualExtensionDao to the backend representing js extension that create new viewable elements of an algorithm.
  • Started to upgrade the frontend to render those new javascript UI extensions.

comment:29 Changed 4 years ago by spimming

r9363:

  • Merged revisions from trunk

comment:30 Changed 4 years ago by fschoepp

r9365:

  • WorkerRole of Slave now logs all exceptions during startup to the blobstore (slavelog).
  • The PluginManager throws an exception during CheckWorkingDirectories() (Save method) which will be caught now, preventing crashes in Windows Azure.
  • "db.DeferredLocal = false" has been removed to prevent loading bugs.
  • HiveScenarioManager doesn't crash anymore, if he can't find an algorithm within a job during retrieval of the run results.

comment:31 Changed 4 years ago by fschoepp

r9366, r9367, r9370, r9371, r9375:

  • Removed unnecessary files from Optimization.Web project
  • Added missing HeuristicLab 3.3.exe.config to optimization project
  • Added necessary HL exe file
  • Marked necessary configuration steps with ADJUST_ME
  • Getting connection information from configuration file

comment:32 Changed 4 years ago by fschoepp

r9395:

  • Added visual extensions (dynamic JavaScript) which will be used to render additional result parameters specific to scenarios (e. g. create a graphical representation of a TSP).
  • Added relationship between jobs and experiments (otherwise, it's not possible to get the job's experiment).
  • Updated Admin page to allow removal/addition of visual extensions.
  • Added back-end logic to store/retrieve/delete visual extensions.
  • Added visual extension functionality to the JavaScript views/controllers (job.*.js).
  • Added tsp.js which is a visual extension for the "Genetic Algorithm - TSP" scenario. It adds a graphical representation of the TSP (just like the C# version) to the results.

comment:33 Changed 4 years ago by fschoepp

r9508: HL:

  • Web projects requires different users to interact with hive. The singleton HiveServiceLocator.Instance doesn't allow different users at the same time, resulting in serialization during access of HiveClient methods.

The following changes have been introduced in favor of a parallel use of the HL libs:

  • HiveClient, TaskDownloader and ConcurrentTaskDownloader may now use a different IHiveServiceLocator than HiveServiceLocator.Instance (all methods have appropriate overloads now).
  • The default instance is still HiveServiceLocator.Instance.

Automated Scaling of Instances:

  • Added Scaler project to solution which represents a WorkerRole that scales the slave instances based on the global cpu utilization of all slaves.
  • Scaler is based on WASABi, rules can be adjusted in rulesstore.xml. Basic rule is: if < 45% global cpu utilization => remove an instance; if > 65% cpu => add an instance. Minimum boundary is 1 and maximum boundary is 8 slave instances.
  • Adjusted Slave project to automatically register itself to a SlaveGroup during WebRole startup (can be adjusted in service configuration).

Web-Frontend:

  • Added basic error messages to the dialogs when an ajax call fails.
  • Removed Styling.js from scripts.

comment:34 Changed 4 years ago by fschoepp

r9549:

  • Removed unnecessary files from web project.
  • Cleaned up slave.

comment:35 Changed 4 years ago by spimming

r9576

  • Added billing test client
  • Use entity framework for DAL
  • Modified service interface and model classes

comment:36 Changed 4 years ago by spimming

r9577

  • Added dal and service interfaces and implementations
  • Added dbcontext for billing

comment:37 Changed 4 years ago by spimming

r9579

  • Revised model classes
  • Modified relationship definitions according to model types

comment:38 Changed 4 years ago by fschoepp

r9582

  • Added an overview for users to inspect their orders
  • Order Administrators may now suspend or reactivate orders
  • When creating an order, its necessary to enter payment information (paypal, credit card,...) before
  • Also, the billing period and type must be entered during the creation of an order.

comment:39 Changed 4 years ago by fschoepp

r9586

  • Billing Component may now be deactivated by setting the BillingEnabled flag within the OaaS service configuration.
  • Added views for invoices and usage data.
  • Changed appearance of the SaveOrder view.

comment:40 Changed 4 years ago by fschoepp

r9588

  • Removed readme from scaler
  • Added BillingEnabled flag to configuration files

comment:41 Changed 4 years ago by spimming

r9602

  • Workaround for enums in model classes
  • Implemented dao methods for model
  • Added test data in db seed method
  • changed EF configuration
  • Implemented some BL methods

comment:42 Changed 4 years ago by spimming

r9619

  • Added new BillingService methods
  • Disabled proxy generation and lazy loading!
  • Extended see method with additional test data
  • Added properties to order and invoice model
  • Initial commit of BillingEngine module

comment:43 Changed 4 years ago by spimming

  • Owner changed from fschoepp to spimming

comment:44 Changed 4 years ago by spimming

r9641

  • Users now contain ContactInformation
  • Fixed formatting in invoice template
  • Fixed calculation for next invoice date
  • Worked on indentation for invoice fields

comment:45 Changed 4 years ago by spimming

r9645

  • Model classes implement new interface to track current state of entity
  • Mark EntityState property as 'not mapped'
  • Set hook to initialize entity as unchanged
  • Extension method for DbContext to apply changes only on modified entity

comment:46 Changed 4 years ago by spimming

r9653

  • Set flag to indicate a modified model object
  • Add dao query method to specify includes as parameter
  • cleaned up the using of namespaces to qualify types

comment:47 Changed 4 years ago by spimming

r9654

  • Added model and view for contact information
  • Insert contact information on SaveOrder page
  • Set correct entity state

comment:48 Changed 4 years ago by gkronber

Please accept this ticket.

comment:49 Changed 4 years ago by spimming

  • Status changed from assigned to accepted

comment:50 Changed 3 years ago by spimming

r10013

  • enabled transactions
  • enabled tracing output
  • set correct invoice data
  • saving invoices

comment:51 Changed 3 years ago by ascheibe

  • Owner changed from spimming to ascheibe
  • Status changed from accepted to assigned

comment:52 Changed 9 months ago by ascheibe

  • Owner changed from ascheibe to swagner
Note: See TracTickets for help on using tickets.