The deployment should have been pretty straightforward.  It was just a few services, none of which were impacted by the larger project that we are working on in parallel, so we didn’t have to do any by-hand merging or anything.  However, when our deployment manager tried to import the changed services into the production environment, he got the following error message:

The excluded service "Map Documents" does not have service parameter documentMappingInfoList defined in the DB. You must either overlay the service or to create the service parameter. The Id of the service parameter in the XML file is 10525.

He asked me about this, and although I had not really worked on the Map Documents or anything related to this minor release, it seemed clear that there must have been some change to the Map Documents service, so he would need to import that as well. 

The next thing we knew, all hell broke loose.

During troubleshooting, we discovered that there was a new version of the Map Documents service under development.  The old version had been renamed to Map Documents (deprecated).  There was one service mistakenly included in the import which was using the new version.  But, all of the other services were still using the old version, so why did this cause a system-wide problem?

Services (and other items) are stored in the process server database with an auto-incrementing ID, and you can’t count on the ID for a service being the same in production as it is in development.  So, when resolving dependent items during import, Teamworks must use some other mechanism.  No problem, that’s why all these things have GUIDs, right?

Well I guess that’s not what the GUIDs are for, because Teamworks uses the name for resolution instead.

Here’s what happened: (deep breath)

There was one service which used the new Map Documents service, and when Teamworks tried to import it, it looked at the existing Map Documents service, even though the GUIDs did not match.  It should have recognized that there was no service with the requested GUID, but instead it complained that the input parameters did not match.  This prompted us to include the new Map Documents service.  This overwrote the existing Map Documents service, again, even though the GUIDs did not match.  It should have recognized that this was not actually the same service, but it silently ignored that.  That was what did us in, because now all the services that were expecting the old Map Documents are now getting the new one.  Obviously this caused errors, since the new service had different parameters and slightly different behavior.

The moral of the story is that you must carefully track your renames.  Anything that you rename in development must be manually renamed in production before you perform your normal deployment.

 

When you start the authoring environment with a new workspace you are presented with a number of prompts as you start working. For example, the first time you make a change to an item you are prompted if you want to check this item out. These prompts are numerous and annoying enough that you are trained to check the box that says to always perform this action. Are there any cases where this is bad though? Well…

After you have been using the authoring environment for some time you will eventually want to delete an instance. You will get the same prompt that you have seen for numerous other actions and will absentmindedly click through to always perform this action. What you have just done is get rid of the "Did you really mean to delete this?" prompt. This is bad enough already, but it is compounded by the following fact.

Yes, that is a commonly used refresh button right next to your now irreversible delete button. Now, for example, if you are debugging a process in production and accidentally hit delete instead of refresh you are screwed. Somebody needs to take a remedial class in UI design. This problem exists in both Teamworks 6 and 7.

This is a pretty common error to get and it usually means exactly what it says: You have no line coming out of your service so Teamworks can’t route you anywhere. But what do you do when there is a line leading to the next service?

In this case there is a good change you have the wonderful phantom line bug. Even though the UI shows that you have a line to the next service and you can select it, as far as the process server knows it doesn’t exist. You can sometimes see if this if you do an export and look at the XML. The line will not be defined. Other times it will show and you need to delete the nested service and insert it again. More so than most other bugs this one angers me, because it leads to start debugging down the wrong path.

Another shortcoming of this is the lack of information with the error message. This is a common problem throughout Teamworks. It would be incredibly helpful for the error message to give you the name of the last service that completed.

Teamworks has a nice feature that allows you to create new section and control templates that will appear in your palette when creating coaches. The full potential of this feature is not being realized though.

One central tenet when writing maintainable software is DRY (Don’t Repeat Yourself). This means you only have to change a feature in one location in your application and it will apply everywhere. Teamworks ignores this principle when applying custom templates. Everytime you use your custom section or control in a coach a new copy of it is created in that coach instead of a reference pointing to the shared template. Six months later when you wish to change how that control works you have to update it separately in every coach instead of in one location.

DRY is an important principle of software development for a reason. While Teamworks allows analysts to do some work that used to be done by developers, it is still primarily programming with pictures and proven principles of software development should still be followed.

Cory asked how we do revision control and Brian has asked me to respond since I’ve been looking at this a lot harder than he has.

If you are hoping to do any kind of revision control with Teamworks such as branch, tag, merge, dif, or commit, then you are going to be disappointed. Teamworks revision control consists of:

• develop in your environment
• export the components (and their dependencies) that you want to change
• import into your test environment
• test your changes
• import into production
• hope nothing unexpected changed

Important points:

• You can’t "branch" unless you have multiple develepment server/databases setup.
• Instead of "merging" you "replace"
• I don’t think that "tagging" can be done in any meaningful way.
• All commits (saves) are directly to the main trunk.
• When you deploy to production, all the existing (in-flight) processes will immediately begin using the new version.
• You can’t just export "changes", you have to export the entire component.
• When you export / import, you can’t see what changes are being made (since there is no good DIF tool).
• When you do an export, all the dependant components hitch allong for the ride.  If you want to make sure that changes in the dependant components don’t make it into production, you have to carefully hand pick your changes during the import step.  

So what have we done?  We are still relatively new and inexperienced with Teamworks and have only done one (major) deployment.  Conceptually we have a Production branch and a Development branch which bounce around between 4 different servers.  Anytime we want to do a deployment, we mirror Production onto the Test server and practice our deployment there.  It works ok so far but when we start a new development effort, we’re going to start stepping on our own toes.

 Although not being able to branch and merge is a pain in rear, the true gremlin in all of this is our inability to reliably see what we are changing.  To address this, I’ve built an application I call ‘Referee’ that allows us to:

• DIF components
• view a change history

Referee takes each of the first class entities (bpd, calendar, connector, epv, historicalScenario, integrationComponent, layout, metric, participant, priority, process, report, resourceBundleGroup, scoreboard, sla, timingInterval, trackingGroup, twClass, underCoverAgent, userAttributeDefinition, webScript, webService) in Teamworks and outputs an XML file for it which can be committed to your source control repository such as SVN.  

With each entity split into its own XML file, it is now possible to use Subversion’s DIF tool to see exactly what has changed.  (I also highly recommend a tool called BeyondCompare.)  So just before we are about to do a deployment to production, we generate all of the entity-XML files from production and all the entity-XML files from what we are going to import and compare them.  Doing this, it is easy to see what we are changing, This SIGNIFICANTLY reduces the risk of unwanted changes creaping in.

How does Referee build these first-class entity-XML files?  Referee has 2 options.  

The easy-to-implement option is to parse the ProcessDefinitions.xml file that is part of the Teamworks export archive file.  I basically wrote this parser in an evening while watching Heros and Medium. And you could too.

The hard-to-implement option is to connect directly to the Teamworks Server database and build all the XML files from scratch.  This is much-much-much-much harder and I would discourage you from doing it unless you have no hobbies, family, girlfriend, or social activities planned for the next 4 months.  That must have applied to me because I’ve managed to reverse-engineer the XML for bpd, process, and twClass entities so that I now have an identical match to the XML generated from the export file.

Having the direct-to-db option has allows us to have a separate ‘watcher’ application that monitors the DateModified column of each of the entities tables and automatically generate the XML file and commit it to SVN anytime someone makes a change.  This has proven very effective for seeing what is being changed.

Unfortunately this process is one-way.  I am not yet able to use the XML files to reconstruct the ProcessDefinitions.xml file so changes made to the XML files can’t be ‘merged’ back into the database.  Doing so is going to be very difficult because the XML file contains many db thingys (db ids, guids, etc) and I don’t fully understand the nature of the problem.  For now, Referee is just tool for improving visibility.

Teamworks 7 promises some kind of revision control.  I’ve spent a lot of time looking at the database and I think this is a pretty tall order.  The database schema doesn’t encourage anything like the branch-edit-merge that we’d like to see.  Whatever they come up with will probably be something else.

After doing a major deployment and thinking about how to manage later deployments of bug fixes it really hit me just how poor change tracking is in Teamworks. Specifically, there is basically none. Change tracking consists entirely of an audit trail that tells you who changed a file and when, but not what they changed or why they changed it.

The Teamworks change control model is a simple Check Out-Check In one. When you make a change to a file the authoring environment checks it out. This puts an exclusive lock on the file until you check it back in. If any other user wants to make edits to the file they must wait until you check it back in. This may have been state-of-the-art in the 1980s with RCS (Revision Control System), but not today.

There is also no concept of meta-data associated with these audits, meaning you can’t even associate specific changes with specific fixes. For example, say you are trying to figure out which three services you edited to fix a bug from six months ago similar to one that just cropped up. There is no way for you to find those changes. This makes the audit trail essentially useless for auditing. A change that tidies up the layout looks exactly the same as a change that rewrites the whole service. Its only seeming use is seeing who last edited a file when I need it so I can tell them to check it back in if they are done with it.

Supposedly Teamworks 7 will have real version control, but I am skeptical. First, this is a pretty essential feature to get to version 6 and still not have. To me it shows that they are not dogfooding their product in any meaningful manner. Second, having spoken to a couple of consultants about the new version control coming to version 7 I have real fears that they are implementing their own solution instead of integrating something like Subversion. If this is true it is pretty much guaranteed to be buggy.

On of the first things you will notice when you start to use Teamworks Authoring environment is that every window pane is too small.  You can maximize the too-small pane but then it takes over your entire window at the expense of the other panes.  So everytime you want to look at something, the process basically goes like this:

• Restore current pane to normal size
• Go to new pane of interest
• Maximize new pane

 

This means you can only look at 1 window at time.  And since there are no hot keys (that I can figure out), it is very slow to switch between these views. 

Probably the worse offender is the Data Mapping tab in the Property Pane.  The image below shows how you map data into and out of an embeded component.

 

 

To reference local variable in a service you specify the variable name with a "tw.local." prefix.  So to reference equity-in-property, you must type in tw.local.equityInProperty.  Due to the design of the Data Mapping pane, all you see is the "tw.".

So essentially, in the normal view,  you have no idea what is mapped to the component.  This makes the Data Mapping view nearly worthless in its normal size.  The only way to get anything done is to maximize the pane and sometimes you still need to click into the field and press END to see what’s in the edit box.

A better solution would be for the Data Mapping pane to have 1 column and we can scroll down to see the output mappings.  At least that way, we wouldn’t have to maximize the window just to see what’s mapped to what.

Trying to work with the Data Mappings tab/pane is like trying to tie your shoe with only one hand.  It can be done but its a lot harder than it needs to be.

 

 

 

 

This is been on my mind for a while…

I’ve worked with several consultants from Lombardi and I feel for the most part, that they deserve high marks.  Somewhere in the A to B+ range.  If someone didn’t know the answer to our problem, they would do their best to figure it out.  They were very agreable and pleasant to work with.  Also, they were very informative and tried their best to educate our team as much as possible.  And when we ran into an issue, they were empathetic and felt our pain.

Lombardi also has an online Support site.  Whenever I have submitted an issue, they have been very prompt, very helpful, and very easy to work with.

Basically, everyone from Lombardi that I have come into contact with has been a pleasure to work with.

Grade: A

 

 

I have no clue what a narrative is but don’t you dare ever go and delete one.

Today we got an error message that basically said: "No more error info available."  So I started trying a bunch of different things trying to figure out what was wrong. After working on this for over  4 HOURS, I come across a line in the debug log that says "Premature end of binary stream after reading 0 bytes".

Searching their website for that text lead me to exactly 1 hit.  The result said that the problem was that a process item had a "narrative" and then someone probably removed it.

Normally, I would have dismissed this error as unrelated.  However, earlier I ran a dif tool that I wrote (why Lombardi doesn’t know how to write this kind of tool is beyond me) and saw that someone had changed a description and removed a "narative".

So - I stuck a narrative back in and now everything works.  Another 4 hours of my life I won’t get back because of Teamworks.

I’m just lucky I’d written my own dif tool or I’d still be looking.  I pity anyone that doesn’t have the ability to do a DIF on Teamworks stuff.

 

 

I’ve been monumentally dissappointed with Teamworks Authoring environment for a long long time but this one really takes the cake.

You can embed items inside BPDs and services.  When you embed something there is a tab that allows you to map data from the higher level item to the lower level item.   

For Services, there is a check box for every parameter that allows you to "Use Default".  This is good and natural behavior.

But for BPDs, there is no visual indication that there are any defaults or not.  To use the default value, you must first know it exists, then leave the data mapping blank.

Why is this so incredibly WRONG? 

1) Because we have 2 different Data-Mapping interfaces that behave differently.

2) You can’t tell if a mapping will generate an error or not without openning the embedded item.

3) If an item is set up to use defaults and the default is later moved, there is no indication from the outside that it will crash.

 

Lombardi Tech support finally explained this to me.  I pity anyone trying to develop a process that doesn’t understand how this works.

 

 

Teamworks services define input and output variables that, much like a function takes arguments and returns values. One feature of both inputs and outputs is the ability to set default values. However, it can be tricky to determine when the default output value is first initialized. There are three logical choices:

1. When the service starts
2. When the service completes
3. Some nebulous time in between

The problem for output variables seems to be that the answer is none of the above. Although you can set a default value just as you can on an input, it has no effect that we have been able to find. This is not an isolated problem when using Teamworks either, as there appear to be many features that have been implemented differently in different places or that appear to have never been implemented at all. In addition, there are other features that should never be used as they make long-term maintenance and debugging very difficult. I will touch on some of these soon.

There are some features of modern software that are just accepted as required and you never even consider to ask about them when evaluating software. One of these features is the ability to undo an action, preferably an unlimited number of them. However, the javascript editor in the Teamworks authoring environment does not have an undo. I repeat, you cannot undo an action in the javascript editor. How this happens I don’t know, but this obviously seriously effects the usability of the editor. You are left with the choice of never making an error or using the editor of your choice and pasting in your code when done. If you do use the editor of your choice you lose the autocompletion the authoring environment offers, but the autocompletion really isn’t that useful.