How Task Automation/Navigation Should Be...

Any .Net programmer of late has heard of the Enterprise Library from Microsoft. It's a collection of code blocks that can automate and harden your code in many different ways. I've used it for over a year, and most of it is great. Most of these are abstractions of the GoF (Gang of Four) patterns that can be used in contemporary programming models.

One of these is the UIP (User Interface process) block provides automation of task and navigation methodologies. It also works in step and flow with the other parts of the EL to provide a common fluid way to automate the user to software interaction and navigation. I can see where in complex business applications this can be handy. I have two problems with this though:

1. It is extremely complicated to use. It involves several "moving" parts that have to be coded in conjunction in order to work.
2. Because of its complexity it takes a long time to implement.

I have written about a piece of software here before that I wrote for my current employer. Its called the Web Scripting System. The nice thing about it is that it completely separates the code from the UI (I'll explain a little later on). The other nice thing is once it is fully mature (right now it is too simplistic to realize its full potential, but it could be sooo powerful) it can be used in so many ways.

The architecture works like this: There is a web service (the Web Scripting Processor). The WSP uses xml files from a database that are called by a simple code. The code can be anything as long as it is unique. The XML file is a "script configuration" of sorts. It contains information about what to do in a normal flow, a disconnected flow and an exception flow. It also contains information about what to display to the end user UI. The latter part is what is simplistic right now.

The flow steps are where the power lies. The Scripting Engine (what the WSP runs in) is a state machine. That means that it is really good at processing states until it hits a state that is has to display something to the user, then it stops processing and sends XML to the caller. The caller then can do with the XML what he needs to in order to display the content. The reason I built it this way is to allow any UI use the system. Primary we had clients that would use both HTML and traditional green screens.

There are 5 main types of states:

1. Display Page: as it describes, this state simply sends parsed XML back to the caller. More about parsing later on.
2. SetVariables: This state performs an evaluation and if it returns true, then it sets a variable.
3. Switch: This state performs both SetVariables and Case Statements. The case statements only go to another state if the evaluation returned true.
4. Command: This performs a remoting call to a script server, passing variables in and then the script executes and returns variables back.
5. Code: This checks to see if the assembly is loaded, and if not it loads the assembly into the app domain and processes a blind method call passing parameters in and getting a result back.
6. Control: Performs a system level command control.

The Command and Code states are the powerful parts. The Command that calls scripts actually uses an "Object Engine" ( a home grown script processor that uses WSH to process COM based JScripts dynamically loaded from a database. It's extremely powerful because it can do just about anything: Make SQL calls, make HTTP Request, Send Mail, File System and so on). The Code state can load a .Net assembly and essentially keep it cached. You can then have access to any .Net component you want. So, if you have some funky logic that you want to execute when a user presses that little button, then you write a small stub and let reflection handle the rest.

So, when you look at the UIP and it talks about tasks and navigation management and switching between UI's, just remember that you have to write, test and deploy compiled code, no matter what you want to do. The Scripting Engine works off a state machine that is for the most part controlled by an XML configuration file for that "task" or set of tasks. What can it be used for?

Examples:
1. A call center needs a script to get "popped" when a call comes in. The pop can be a call to a web server or terminal screen server. The script can be processed and depending on the actions that the agent selects go down certain paths and can provide the agent with content to repeat back to the caller and fields to collect data.

2. A post web chat survey. This would provide everything from a simple flat path set of questions to a complex branching logic set of questions.

3. An IVR engine(almost). An IVR can make request to the web service and process commands and the xml coming back can tell the IVR what prompt to play next.

4. A corporate web site could use the engine to provide either basic navigation and page display to specific task embedded in the script (such as registration or checkout and credit card processing).


What happens to the data? That the ever-so-nice part. You see, every new request creates a session XML file on a centralized file server (NAS in fact). Each response sent back out carries with it your session id. On the return trip the Engine picks up your session data which contains all the variable data you have during that session. The UI's I've written also take advantage of a Heartbeat method in the web service that let's us know the user is still there, but has not click on the page yet.

Once the user hits the last possible state the Engine closes out the session and all the users session data gets logged to a vertical table. That data can be flattened at any time and exported or reported on.

If the heartbeat misses after 10 minutes, we consider the user has aborted and process his session data.

By writhing this article I am not bashing what the EL team at MS has done. The block is impressive, but I think the Scripting Engine can provide something they can't. Once the system is in place, and you have a UI in place then anyone who understands the flow logic they are trying to implement can do so in an exaggerated pace. We had a new developer come on staff and in a few hours was able to start writing a script. It was an extremely complex script that included over 100 "pages" of display, almost all of which contained forms to collect information, and several hundred states of logic flow. He completed the script in less than a week. He admitted that if he had to write asp.net pages and logic to do the same thing it would have taken him over a month.

I don't know if my employer will ever want to sell the system as a stand alone, but I would :).

Reflect

I've discussed my DataXplorer here before. It's a configurable, searchable data page with a result set and a grid. You can configure a context based menu per item to perform a redirect to a page and set either session variables or query string variables from the row selected.

I've recently updated it to include the ability to configure a menu that is not context based, but goes with our paradigm of each row having a check box to signify a selection. Therefore you can now perform some action against more than one row at a time. But wait, there's more.

I've also added a top row (not on the grid, but the page) button based menu system. Each menu corresponds to a configuration to which you can load an assembly, build parameters (both static and items on that particular row) and execute a method. You also get a feedback object which tells you of the success or failure of your action. I learned a lot about reflection by doing this.

One thing I learned is that if you don't do some concurrency check to validate that you don't already have the assembly loaded into your Application Domain you can get bloated fast. Speaking of Application Domains, in asp.net it's a bear. Your application runs under it's own app domain, which is how asp.net manages to isolate the memory of app from each other. All of your assemblies are loaded into the app domain, which is why it's so much faster. However, once loaded always loaded (I really wish there was a way to recycle certain assemblies on a sliding contention timeout), so to unload an assembly you have to unload the app domain. Bad idea.

What I ended up with is a method that checks all the assemblies that are loaded in the current app domain for the one I am dealing with at runtime. If I can't find it, then I load the assembly. I have read all sorts of blogs and news postings that really push you towards building another app domain and loading everything you need there, then tearing it down when your execution is done. It's a bad, bad idea.

Think about it. In this case the same assembly may be loaded and unloaded 15 times in 5 minutes by one user. It doesn't scale well. Plus there are all sorts of issues about assembly caching and so on. If you do it using the methodology I did, then asp.net manages it all. Your app runs a little heavy, but for this purpose it's actually better. I can now make extremely small assemblies (my first one was 16k compiled in debug mode) that target specific functionality. Plus you can reuse functionality that the app already knows about. It's a win win. yay yay.

Now here comes the cool part: I am loading an assembly, defining what the method is I want to call and then dynamically building its parameters and executing the method. I have this rolled out to about 100 users and it's working perfectly so far.

I'll add some designs when I have them polished a bit.

Race Condition

Q. When does the need for rapid development take precedence over a good QA process?

A. It shouldn't ever, but it does all the time.

I work for a small company. We are in a constant state of flux. We don't have enough Developers, QA testers, Program Managers and so on. We are trying to make a break out product, and in the mean time beat everyone else to the market.

So, when the PM's are screaming that they need a fix, and you provide a fix and it breaks something else (something that is innocuous enough that you wouldn't see it without true regression testing with client data) which path do you take? You're already working 80 hours a week, so is everyone else. You don't want to not see your kid to bed so you can test this one more time all the way through before you release.