Vestpocket Software








Web Based Parallel Data Flow Project Development Framework

This page describes the Web Data Flow Development Framework we have developed for creating libraries of coarse grained SOA objects, executing data flow directed graphs constructed of these SOA objects while passing data between them, and displaying the results of their calculations. The SOA objects in the libraries are somewhat like objects in libraries used in object oriented programming languages. These objects can be combined in any way the user desires as long as the result is an acyclic directed graph. The system for creating collections, referred to as projects, of SOA objects can be thought of as being something like a graphical programming language. Ordinary programs, perhaps previously written in C, C++, VB., etc., are turned into 'SOA objects' by 'wrapping' them in a predefined input output structure.  The web version of the development framework closely mimics the desktop version in functionality and form. There are a few things that cannot easily be done in a browser such as dragging and connecting graphical objects. This functionality is replicated in the web version using tables, drop down list and the like. It also serves as an interface to a master server which is part of an collection of networked computers. Each computer on the network has a subordinate server and a worker server which are interconnected to the master server that is connected to the parallel console. The console is used to define a computational project made up of individual jobs that will be run in parallel on the parallel computer network as shown in the PERT Chart view below. The individual jobs may also be projects. The console directs the master server to run a project. The console can also submit a batch run of projects to the master server. The system requires a database which supports ODBC. The master server breaks down each project into a form that can be run in a parallel fashion as shown in the GANTT Chart view below. It runs a brief simulation of the project taking into account the estimated complexity of the jobs in the project and the power of its subordinate master server computers to help it work out a distribution strategy. Using its distribution strategy it then assigns parallel parts of the project to subordinate servers which schedule the jobs in a serial fashion to subordinate worker servers. The worker servers put the jobs into a queue and run them in a serial fashion. If a job is a project, the process is repeated recursively on the subordinate master server. A project is a set of SOA objects or 'transformations', and a job is a single SOA object or 'transformation'. Each SOA object may be connected to one or more SOA objects by data flow 'relationships' as shown in the Connections Lists view below. No loops are allowed, therefore these SOA objects form an acyclic directed graph. A job can be thought of as a computer program that completes a single part of a project. A project is also considered a transformation and can appear as a job within a project thus allowing recursion. After the project is defined, it can be assigned to be run on the parallel network. A project can also be referred to as an executable directed graph that allows for data flow between the SOA objects in the graph. The user can view the progress of the project as the jobs within the project are run in parallel. As the jobs are run their colors change from red, indicating that they are running, to cyan indicating they are finished. After the overall project is finished, the user can view the results of any of the jobs within the project and also view the results at the end points of the overall project. A report writer can produce a hard copy of the results of the execution of the project. A list of projects can also be created and submitted to the master server as a batch run. There is no theoretical limit to the number of computers in the parallel network and no theoretical limit to the number of jobs in the project.

The Web Control Console allows a user to run projects using a browser. When the page is first displayed, the application is not connected to the Master Server. A 'Connect' button is visible and enabled.. When the user presses the 'Connect' button, the ASP Web Server connects to the Master Server and they begin exchanging messages. When the connection is complete, the Web Control Console page, shown below, appears. The 'Connect' button is replaced by a 'Disconnect' button. The projects that are available to be run are shown in the list on the left. When the user clicks a project in the list, the selected project becomes the current project. The 'Run Project' button appears and the buttons at the top of the page become active. Pressing the 'Projects' causes the web projects page to appear.  Pressing the 'Properties' button at the top of the page causes the properties page to appear. Pressing the 'Inputs' button causes the input selection page to appear. Pressing the 'Outputs' button causes the output selection page to appear. Pressing the 'Runs' button causes the run selection page to appear. Pressing the 'New' or the 'Edit' button causes the project editor page the appear. Pressing the 'Show Graph' button causes the pert chart page to appear.

Web Projects



The Web Control Console, shown above, has five views. The 'Projects' view allows the user to select a project to be the current project. Once the project is selected, it can be run by clicking the 'Run Project' button. When the 'Run Project' button is clicked, a 'Stop' button replaces the 'Run Project' button and a 'Pause' button replaces the 'Disconnect' button. A run project message is then sent to the Master Server. As the project runs, messages from the Master Server and Workers Servers appear in the list at the right. The messages in the list show the jobs in the project running on the Worker Servers starting and completing . Any error message from a Worker Server causes a flurry of stop run messages to be sent and the project stops running. When the project is finished running, the 'Run Project' button replaces the 'Stop' button and the 'Disconnect' button replaces the 'Pause' button.  If the 'Stop' button is clicked during the run,. the server sends a stop run message to the Master Server. The Master server sends stop run messages to the Worker Servers, and the run is stopped. The 'Run Project' and the 'Disconnect' buttons then reappear in place of the 'Stop' and 'Pause' buttons. If the 'Pause' button is clicked during the run, a pause message is sent to the Master Server and a 'Resume' button replaces the 'Pause' button.  The Master Server then sends pause messages to the Worker Servers. When the 'Resume' button is clicked a resume message is sent to the Master Server.  The Master Server sends resume messages to the Worker Servers and the run is resumed.  All of this functionality is much like running a project using the desktop version of the console and the messages to and from the Master Server and Workers Servers are identical.

Web Project Properties



If the user clicks the 'Properties' button at the top of the page, a table appears showing the properties of the projects that are available to be run.  If the user selects a project in this view, it becomes the current project. Pressing the 'Project Details' button will bring up the project details page shown below.

Project Details



The project details page, shown above, shows the details of the current project. It also allows the user to delete the current project, delete all oe fhr jobs in the current project, and auto map of the fields in all of the jobs in the current project.

Web Input Job Selection



If the user clicks the 'Inputs' button at the top of the page, a table appears showing the SOA Job Objects in the project.  If the user selects  a  job, a view appears showing the input values the job will have when the job is executed.  The user can change the input values and rerun the project if they wish to do so.

Web Project Graph



If the user clicks the 'Show Graph' button in the form, a graph of the current project is shown. The graph shows the jobs in the project and the data flow connections between the jobs. If the users presses the 'Connections' button the job connections list page will appear. If the user clicks the 'Gantt Chart' button the gantt chart page will appear.

Web Connections Lists



The Connections Lists view of the project shows the connections between jobs in a list. Clicking on a job in the lower right hand list will show the jobs that are downstream of the selected job in the list to the right.

Web Gantt Chart



The Gantt Chart view of the jobs in the project shows the dependencies between the jobs in the project and the run time of each phase of the project along with the total tun time.

Web Job Input Details



When the user selects a job in the input selection table, a form appears showing the details of the inputs for a job.

Web Job Input Change



When the user clicks the 'Change Values' button in the Job Input Details form, a form appears that allows the user to change the inputs for the next run of the project.

Web Output Job Selection



If the user clicks the 'Outputs' button at the top of the page, a table appears showing the SOA Job Objects in the project.  If the user selects  a  job, a view appears showing the output values that were created when the job was run. Naturally the user cannot change these values as they are computed.  The user must change the input values of the job and rerun the project to change the outputs of a job.

Web Job Output Details



When the user selects a job in the output selection table, a form appears showing the details of the outputs for a job. The output values are read only.

Web Runs



If the user clicks the 'Runs' button at the top of the page, a table appears showing the previous runs of the project.  If the user selects a run, that run becomes the current run of the project. If the user clicks 'Delete' for a run, the run and it's output data is deleted from the data base. The user can then click the 'Inputs' button or the 'Outputs' button, and proceed the view the inputs and outputs for the current  run of the project.

Create And Edit Project Properties



The Create And Edit Project Properties view, shown above, allows the used to edit the properties of a project. The user opens the project edit in the new project creation mode, a similar screen is displayed that allows the user to insert and edit a new project rather than editing the properties of an existing project. The screen looks the same as the one above but the 'edit' link is replaced by an 'insert' link. Pressing the 'Project Details' button will bring up the project details page shown below. If the user clicks the 'Jobs' button the jobs list page will appear. If the user clicks the 'Connections' button the connections list page will appear.

Project Details



The Projects Details page, shown above, displays the details of the project. It also allows the user to specify image to be displayed in the background of the graph associated with the project.

Create And Edit Jobs



The Job Creation And Edit view, shown above, allows the user to insert new jobs into the project. It also allows the user to edit and delete jobs that are already in the project. Pressing the 'Details' button for a job brings up the Job Details page shown below.

Job Details



The Job Details page displays the details of the job. It also allows the user to specify the default estimated run time  for a job,  and allows the user to specify the long term archival characteristics for a job 

Editing a Job



When editing a job The Executable Name is chosen from a drop down list..

Create And Edit Connections Between Jobs



The Create And Edit Connections view, shown above, allows the user to insert new connections between jobs into the project. It also allows the user to edit and delete connections between jobs that are already in the project.

Editing a Connection



When editing a connection, the From Job Name, To Job Name and Connection Type are chosen from drop down lists.