How do you improve Quality?

There typically are 3 types of projects, when it comes to quality that I have seen.

1. The focus is getting the code out the door and then you pick up the pieces.
   
2. The focus is on getting quality code out the door until deadline nears then you revert to number 1.
3. The focus is on getting quality code out the door and as deadline nears features may be dropped and the deadline may slip.

When the project starts everyone seems to be on board with the idea of #3 but typically I have seen it slip all the way to #1 quite quickly.

At my current employer we have an internal application that is used by 300+ end users.

We used to have:

• ZERO people dedicated to QA. (4-5 Delphi Developers, 1 Mainframe Programmer, no analysts)
  
• Testing was done by end users testing as they had time.
  
• We had no automated or unit tests.
  
• The build process was compile from your own machine and copy it the network, with any developer free to release any code they wanted. Which would sometimes step over the other developers uncommitted code.
  
• We had two environment's
• Development used by both developers and testing.
  
• Production
• Exceptions had no details or context to help track down problems.
• We had a very small and painful bug tracking system.
  
• We would commonly be related to #1 type model above.
  

Over the few years that I have been here we have made many changes.
We now have:

• We now have three people dedicated to QA. ( 7 Delphi Developers, 1 Database PL/SQL developer, and 1 Analyst)
  
• We have end users still testing with the time that they have. But we now have better management support to get additional time needed.
  
• We have a few automated tests, and a few more unit tests, but we really could use many more.
  
• Our build process runs every hour through Final Builder, alerting us with email when someone checks in bad code.
• We now have 4 environment's
• Development (Dedicated to Developers only)
• TEST where users get first look
• QA a final look before production usually with data refreshed from production
• Production.
• We now use the JCL Debug to help use find those Hard to Reproduce Errors.
  
• We have a better bug tracking system, but still not nearly as nice as some of the home grown solutions I have used at past employers.
• We are now some where between the #2 and #3 model's above.
  

Every release we ask what can we do better? We try to learn from our mistakes and implement solutions to prevent problems. We release a new version of our software nearly every month, so we are continually improving, and quality has improved in many ways in the past few years.

However, today I feel awful!

We put out a release on Friday morning last week. I don't think I have ever had a release go bad as this one has. Today we are still working on fixing major problems that were caused by the release.

• Some of which would have been caught by better testing.
• Some of which would have been caught by better management of the information coming in about problems. (i.e. it was reported but not acted on!) Since, I usually manage this information, it's the reason I feel awful.
• Of course like all systems some of them would have taken an unreasonable amount of testing to find.

So I have been thinking, it's time to go back to and look at quality in new and different ways.

I have been making a list for a while that specific to our product on ways to improve quality. Several items on the list are general enough to share with others.

• Build a health check system detects common/possible problems that can be run daily in each environment's, which is very database specific. We have scripts for all of these, but not an automated way to run them.
  
• Tables without primary keys
  
• Missing Synonyms
  
• Invalid Synonyms
  
• Tables without indexes
  
• Missing Grants
  
• Disabled Constraints in Oracle
  
• Tables that have not been analyzed recently.
  
• FK Constraints without index on child table (table level lock will be placed on the parent table if these are found)
  
• Invalid Database Objects
  
• Plus several that are specific to our schema, i.e. constraints that can't be described in meta data.
  
• Error Messages (Exceptions)
  
  • Ability to Send error messages to a web service
    
  • Ability to capture a screen shot with the error message.
    
  • Ability to ask users what they are doing when the exception occurred.
  • Ability to attach reported errors to items in bug tracking system.
    
• Code Review to help identify and build additional unit tests around the code.
  
• Automated Performance Testing Benchmark system. (I think DUnit has something that will work for us)
• Get current Delphi Unit tests into an daily build and smoke test instead of just being run by hand by developers.
  

Ok...

I have shared some of my ideas on how to improve quality, as well as some of the things we have done.

How have you improved Quality in the software you write?

I am open to any suggestions, this is just a small attempt to think out side of the box.

Since I just started this blog, I really hope that someone is reading this and has some good ideas they are willing to share.

Enhanced Screen Capture Code

I had this crazy idea the other day. I want to see what my users screen looks like at the time an exception occurred. A quick Google search led me to this snippet of code on about.com.

After a bit of testing I found out that it only captures the main desktop or active window. Well since many of our users run with multiple monitors, it's possible our application is running on the second monitor. This means that if I capture the main desktop would potentially reveal nothing valuable.

So I rewrote the code to handle multiple monitors better.

So here is the interface section.

uses
  Windows, SysUtils, Graphics, MultiMon;

type
  EMonitorCaptureException = class(Exception);

  TCaptureContext = (ccActiveWindow,ccDesktopMonitor,ccActiveMonitor,
                     ccSpecificMonitor,ccAllMonitors);


function MonitorCount : Integer;

procedure CaptureScreen(aCaptureContext : TCaptureContext; destBitmap : TBitmap;aMonitorNum : Integer = 1);

procedure CaptureRect(aCaptureRect : TRect;destBitMap : TBitmap); inline;

procedure CaptureDeviceContext(SrcDC: HDC;aCaptureRect : TRect;destBitMap : TBitmap); inline;

The enumerated type TCaptureContext allows few different options.

• ccActiveWindow: Capture an image of the ActiveWindow only.
  
• ccDesktopMonitor: Capture an image of the primary/first monitor.
  
• ccActiveMonitor: Capture an image of the monitor with the active window on it. If the active window is on more than one monitor it will pick the monitor that has the larger portion of the window on it. If the active window is not in the visible space of any monitor, it will pick the closest one.
  
• ccSpecificMonitor: Capture an Image of a Specific Monitor specified by the aMonitorNum parameter. Note: aMonitorNum the first monitor is 1. You can use the MonitorCount function to determine the valid range of monitors. An EMonitorCaptureException will occur if you select a monitor number outside the valid range.
  
• ccAllMonitors: Capture a single image of all the monitors.
  

The entire code for the unit can be downloaded from my SVN Repository.

To see it in Action...

1. File | New VCL Form Application.
   
2. Add SCapture.pas to the project, or place it in your library path
   
3. Drop down an Button and an Image on the Form
   
4. Place the following code in your button.
   

var
 b : TBitMap;
begin
 b := TBitmap.Create;
 try
    CaptureScreen(ccAllMonitors,B,1);
    Image1.Picture.Assign(b);
 finally
   b.FreeImage;
   b.Free;
 end;

There is also another routine that allows you to capture any Rect from the display.

var
 b : TBitMap;
 r : TRect;
begin
 b := TBitmap.Create;
 try
    r.top := 20;
    r.Left := 20;
    r.Bottom := 200;
    r.Right := 200;
    CaptureRect(R,B);
    Image1.Picture.Assign(b);
 finally
   b.FreeImage;
   b.Free;
 end;

Going back to the crazy idea of getting a screenshot on an exception. I have not implemented it yet, but something things to think about if you going to try.

• EOutofMemory and EOutofResource Exceptions will most likely cause this code to fail, I would check the Exception class and if it's one of these don't take the screen shot.
  
• Unhandled Exception's raised in the Exception Handler mask the original error. So if taking the screen shot fails, don't annoy the user.
  
• Privacy & Security: The images of the desktop may contain information that may violate users privacy or be a security risk of image is not secured.
  
• You want to capture the screen before the exception dialog appears otherwise the information you want will be masked by the dialog.
  

Regardless of my concerns, I will be coming up with a solution the positive outcome is greater than any potential negatives.

Finding the Hard to Reproduce Errors

I used to do a ton of contract work. One of the most common reasons I was called out to a company was that their users were experiencing an exception, usually an access violation, which they could not reproduce in the development environment.

Since I have been fairly busy, and I really don't have time to do side work. I had a request to share what I do in this case, so they could go about fixing it with out me. Instead of just responding via email with my methods of madness, I decided to share the way I do this with everyone.

First off the only way your going to get information to help is with good logging. There are a variety of tools out there to help. But I have found a good old text file is typically the easiest method, although products like SmartInspect and CodeSite offer quite a bit more and may be worth investing in, I will let you evaluate and decide.

The problem with logging is that the information is only as good as what you supply. You don't know where the error is going to occur and you really don't need a log file endlessly filling up with useless information. So how do you know where to log? Well my favorite places to log information on Screen changes. That way I know the order of the screens visited, and sometimes key values that are needed for that screen. I also add verbose logging on a temporary basis for hard to find problems.

But hey! The odds are I have told you nothing that you did not know already.

I have found many people are not familiar with call stack logging when an exception occurs. In .NET getting the exception call stack is easy, you just need to reference the .StackTrace property on the exception. Once you have that write it out to your log file, using your method of choice.

However, with native code it requires quite a bit more work. This is because information about method names and line numbers are not part of the compiled code.
In Delphi the information that contains this information is in the MAP file. The key is how do you get this information at the time of the exception and log the details you need.

There are two commercial products out there that do just that and more.

• EurekaLog
  
• MadExcept
  

I however use an open source alternative that works well. It's was also great when doing consulting as I did not have to make them buy a commercial product.

The Jedi Class Library (JCL) has the libraries needed to do this. It's used by Embarcadero inside of Delphi to report errors to Quality Central.

So take a moment if you have not already do it and download it now.

After installing the JCL, you will want to open and look at the following Demo applications found in the "jcl\examples\jclDebugExamples.bdsgroup" project group.
It offers a fairly good look at the possibilities.

Ultimately the question is: How do I plug the JCL Debug into my application and make it useful.

Here are simple steps to do this.

1. Download and Install JCL
   
2. Open "jcl\experts\debug\tools\MakeJclDbg.dpr" and Compile. It may have already been compiled during install.
   
3. Open: "jcl\experts\debug\dialog\CreateStdDialogs.dpr"
   
4. Look at "Params." around line 80 and set the params you want.
   
5. Run the application.
   ExceptDlg.pas/dfm and ExceptDlgMail.pas/dfm will be generated
   
6. Copy these to your application source directory.
   
7. Add ExceptDlg to the uses of your project.
   
8. Turn on .MAP file generation in your project options. (Found under linker section)
   
9. Build your project.
   
10. At the command line run MakeJclDbg -E YourProjectName.MAP
    

You are good to go! Unhandled exceptions in your application will be replaced by this new dialog. Since the code to the dialog is in your source directory you can change it to meet your specific needs and/or design layout.

In the last few steps, I showed you my preferred method of dealing with information for symbols, but the JCL supports many other methods.

For example there is an expert that you can install that will do the same thing without dropping to the command prompt. Since I never deploy an application that is compiled in the IDE, I use FinalBuilder, calling MakeJclDbg seems second nature.

One of the key things is that symbols can be stored in a variety of different ways, and JCL will search for them in the following order.

1. JCL Debug data in the executable file (What MakeJclDbg and the Expert will do)
   
2. JDBG file
   
3. Borland TD32 symbols
   
4. MAP file
   
5. Library or Borland package exports
   

There is a good document on the options found in "\jcl\experts\debug\HowTo.txt"

Now the most surprising question that I have heard over and over after doing this is: What do I do with this information?

Well you now have a call stack when the exception occurred. You can see also see the line the exception occurred on. Often looking at the line of code where the error occurred shows that the given method was called incorrectly. With the call stack you can see the line of code that called that method. This typically this should give you enough of a clue to resolve the problem, or at the very least you will know where to add more logging to figure it out.

The Dark Side (of the moon) or Not?

For the past couple of months we have been doing a fairly in depth review of Delphi Prism. We have a 6-8 month project that we need a fairly complex web service. Our past experience with Delphi for .NET and C# we knew that the API's in .NET would be a big advantage to our project.

Since we already own RAD Studio, it was time to review and see if Prism was the right technology or if we should just move back to Delphi Native only. The flip-flopping on the .NET side of Delphi has driven me crazy.

We decided to start with something simple but complex enough to test the needs of our project. If the technology did not prove it's self then we would not be out a ton of time.

The language changes took a few moments to get used to. Then after awhile you start wishing that Delphi Native had the features that are present in the Prism Compiler. I suspect that is partially why they are completely rewriting the Delphi Native Compiler.

The IDE in Prism is Visual Studio but I found the transition to the different IDE fairly easy, of course I have done quite a bit of C# work in the past.

Overall I have to say I am impressed. We have been able to build a solid web service in short time. The language being Pascal based is a huge win as it requires far less training that switching to entirely different language. I also liked several of the language features that C# does not have. Specifically Sets and the colon operator.

The one drawback I noticed was that there was not direct support for LINQ to SQL. If you have the full VS.NET product you can build a LINQ to SQL assembly and use it fine in Prism. However, Microsoft does not support Oracle which is the Database we use. Microsoft only supports SQL Server. If the DBX team could conquer LINQ to SQL for DBX it could be a huge win. But then again I think the designers only ship with the VS product and not with the Shell, which has been major reason for the flip-flopping of Delphi in the .NET world in the past.

Will we be using Prism for anything other than ASP.NET technologies? Well at this point I doubt it. I still am a huge fan of native code development. It does work well in the ASP.NET environment and we will be using it for our 6-8 Month project we were reviewing the technology for.

As for the title of the post? Look in the About box!

Blogging Again -or- Opening Mouth to Insert Foot

Open Mouth Insert Foot. I feel like I do that enough... So why not start blogging again.

It's been several years since I last blogged. This time I am not going to try to develop and maintain a site for it. I just don't have the time to play around with duplicating technology that is already free and easy to use. Also, I don't have to worry about server costs and maintenance.

Well what has happened in the past few years since I quit blogging.

I started working at the State of Utah full-time, I used to do contract work there. It was the right move for my family and in total I have spent 6+ years there. The State of Utah currently pays for tuition so I am enrolled in school. The school is WGU, it's an online only school. I feel I might as well get that degree sooner or later, and I enjoy research and learning.

I am still doing the following:

• Develop in Delphi although now I have to manage a team of nine.
• President of the Salt Lake City Delphi Users Group.
• Finding other ways to open my mouth and insert my foot.
• CodeRage
• DataRage
• DelphiLive
• Cooking outdoors as much as possible.