A new problem, a new stack

Time to Rewrite

In the past year, my employer decided to rewrite our Internal Application.     It's not something we took lightly, the current Delphi Application is over 5 Million lines of code.     So as we rewrite portions of that existing application we will need to maintain our existing Delphi application so I will continue to use Delphi for at least 4-5 more years.   Our system was released in 1999 as a Y2K project.   Most of the original code was had the business rules right in button events, and a majority of that still exists.     With my employer, it is unheard of to have an application survive 20 years, and it will most likely be 25 years old when we turn it off.      It says a lot about our staff and Delphi.     

User Devices are always changing

However, I have always believed we should use the right tools for the job at hand.    Our infrastructure is greatly changing, we now have a few telecommuters, with a push to support more.     This is not friendly to our existing VCL Client Server Application.    We have a mixed environment of Desktop PC, Macs, Tablets and have to be prepared to support the ever-changing device form factors.   Our user's desktops are locked down with no administrative rights.    Implementing Continuous Deployment with a desktop application has been increasing more difficult to manage.

We decided it was best to create a browser-based application.    That is really not something Delphi is a market leader in so a change was in order.  Having now spent several months of development using our new stack I am very pleased with this decision.   

New Development Stack

Front end

We are using Angular 7+ with strict compiler settings on the TypeScript compiler.    This Angular CLI provided a huge selling point, it provided much of the front end stack without having to build it up.  Angular provided a component model and TypeScript syntax that reminds me of Delphi helped with the learning curve on the transition.   

We looked at the ever changing component lanscape with Angular.  We spent a few weeks and decided to write own library that provides a facade around the PrimeNg Controls.   This allows a majority of our front end to not care about the underlying library, if control is deprecated or problematic we can swap it out without impacting the rest of the application.

The browser world is always changing,    It's a major moving target, things like web assembly may really change the landscape in this space.     As such it's critical to avoid tightly coupling the various parts of our application.   We are using Visual Studio Code as the editor of all our front end work and I highly recommend it.

Back end

The backend is a series of REST endpoints written using C# and ASP.NET Web API.   We have no user-facing ASP.NET pages.  We now also provide a real-time client to client communication channel implement SignalR, which has increased our ability for our users to collaborate in our application.  

Our data layers are abstracted and loosely coupled to our business rules which has greatly enhanced our ability to unit test, and we chose XUnit to help us with that.  Which has in turn increase our quality.   

We are using Entity Framework to managed our database access, which came with a bit of a learning curve as someone who has used SQL for some many years.    But now after using LINQ, I have found it really easy to use and so far as produced performant access to our data.   

We are using Visual Studio 2017 Professional for our back end code, and we will eventually be upgrading to Visual Studio 2019 which was released today.

The Value of Code Reviews

I have recently seen code that that looked like this.

 
try
  ... Lines of Code here ...
except
end;

or

 
procedure TCustomClass.MethodDoThis()
begin
  ... 20 Lines of code ...
end; 

procedure TCustomClass.MethodDoThis2()
begin
  ... Same 20 Lines of code with slight modification ...
end;

Both produced code that may have passed end user testing. But both cause long-term problems.

• The first example was just hiding all exceptions, a bad practice. After the code review, it was determined that the try | except block was not even needed.
• The second example was refactored to reduce the need for duplicate code. The duplicate code causes problems when someone has to make changes that would apply to both methods but does not notice the 2nd method and therefore neglects to change it.

My point is simple, code reviews have value.

Sidenote: Looking forward to reviewing Tokyo release of Delphi soon. With 500+ Bug Fixes with many in areas that impact our system is definitely worth investigating. Security and Quality go hand in hand, and I appreciate work to improve security. Typically, security issues are related to bugs in code that are exploited by hackers.

Ideara / Embaracdero is flushing away user trust in their ability to do secure computing.

In today's computing society you have to think of security first in the design of everything you do.

It's costly to retrofit security. In my job I have think about security every day. The costs of sensitive data escaping is too big to ignore. Trust is a huge word in security.    I have to trust the vendors we use, that they will act quickly to resolve security issues.   So we are not left vulnerable.    A good vendor realizes there will be security problems and has a way to publicly communicate that has does so in an open and transparent method.

Over 10 months ago I sent Embarcadero management several problems with Embarcadero's websites. I finally saw a bit of action after the main website was hacked quickly after it went live.   I even talked with Atanas Popov the General Manager of Embarcadero.   But after that they went dark and have not communicated with me regarding security since.      It's clear they have no plans to create a place on where we can go learn about security issues and the fixes that may have occurred in the product.  

I also called for them to hire a CSIO and empower them, which clearly has not happened.  Even if they did not have a CSIO, they need security to be monitored and prioritized.

Today I received a plaintext password in email for an Idera Community website.   Have they not heard about hashing passwords?   A hashed password can not be sent in clear text.    In email they claimed it was stored securely, and they did it this way to prevent spam.    But what they did was give passwords out  insecurely which could enable more spam instead of reduce it.   They defended this action and clearly thought the practice was ok.     They offered to delete my account.   I did not request the account so I took them up on the offer.   Regardless maybe they should watch the password portion of my session I did for them at CodeRageX.    

Idera and Embarcadero are clueless when it comes to website security.    Maybe just websites in general since Embarcadero's website is now serving content to me in Portuguese even when I select English.

I have NO TRUST in SECURITY from Idera or Embarcadero at this point.

Moving too slowly on a security vulnerability is like using and flushing a clogged toilet, bad things will happen.    Being vulgar and swearing is not in my character, however, it was very very hard not to be that way in this post.   I have invested decades in a company and product that I can see clogged in their own toilet water.

Code in Application or the database?

My employer has a fairly large Delphi application with ~5 Million lines of pascal code.

They have a large database and use Oracle 12c to manage it.

I have seen many methodologies on how to manage your database access code.
On the edges I see two extremes are:

• Do everything in Stored Procedures 
• Do everything in Code outside the database, (Typical ORM)

I believe you need to use the best tool for the job.   So sometimes we are doing massive amounts of data processing that never leaves the database.     For that I use Packages/Stored Procedures in Oracle.     

Years ago we had a process that was 100% in code, and it took 20+ hours in code to execute, due to the round trips to the database.   When we looked at optimization, it was clear that network time was a huge part of the equation.    This Delphi process was converted to Oracle PL/SQL and was further optimized and our time was was reduced to <30 minutes

A simple example using the best tool for the job.    I have been a advocate of unit testing in Delphi.    Several months ago we had do some major PL/SQL work that resulted in about 50k lines of PL/SQL code.     Way too many lines of code to write and trust without sometime of unit testing framework.    We had been testing PL/SQL code from Delphi using DUnitX, but it was one too many steps away from the code under test.    I started working with utPLSQL a user testing framework for PL/SQL.   

In the past few weeks: I finished the move of utPLSQL from SourceForge to GitHub, and have released version 2.3.1 of utPLSQL.

Still love and use DUnitX but I use it from my Delphi Code not my PL/SQL code.

Regardless of the technology you find yourself using for development you need to unit test.

Rad Studio Berlin 10.1 - First Look via Twitter

Today I installed Rad Studio 10.1 Berlin.    This was my first chance to look at it.  
Here are some the tweets are shared earlier today.

Just installed Rad Studio 10.1 Berlin. Loved the new install experience.

— Robert Love (@DelphiDev) April 21, 2016

Rad Studio 10.1 Berlin has a new welcome page. It's very clean, easy to use. Thank you RSP-11183 can now be closed pic.twitter.com/3gzSLXKw3F

— Robert Love (@DelphiDev) April 21, 2016

Rad Studio 10.1 Berlin - Cool New Feature. Press Ctrl-W to expand syntax selection. pic.twitter.com/FsgkCWnO3c

— Robert Love (@DelphiDev) April 21, 2016

Rad Studio 10.1 Berlin, supports a great list of Z-Wave and Bluetooth IOT devices. pic.twitter.com/KNULTKfzLB

— Robert Love (@DelphiDev) April 21, 2016

SQL Injection - Stored Procedures

Security needs to be multiple levels deep. At one point in time several years ago there was a trend to have all websites code use stored procedures. I heard developers say it prevented SQL Injection.

If the database you use allows you to build SQL statements with a string dynamically at run time in a stored procedure then you can have SQL injection in a Stored Procedure.  

Each Database slightly different syntax and guidelines:

• Oracle  (See Page 12)
• PostgesSQL 
• SQL Server

In short: If your doing dynamic SQL inside your Stored Procedures you will have validate the input otherwise you can be impacted.

SQL Injection

I just wrote a blog posts on security assumptions. I realized there was a huge missing assumption. (This has since been updated.)

•  Assume that all input can be malicious 

Today I want to show the most common mistake, and exploit that I have ever seen. It impacts all languages and all SQL Based databases. It's called SQL Injection. I have heard developers make the incorrect assumption that this only impacts web based applications. I have seen this problem in all types of applications that touch a database.

Today's example is VCL Desktop application uses FireDAC to access a SQLite database.

The example code used here can be found in the security-demo GitHub repository.

SQL Injection occurs when you don't use Parameters, which allows input to modify your SQL statement.

Here is the unsafe way to do it:

   
  FDQuery1.SQL.Add('select * from tbl1 where name like ''' + edtSearchTerm.Text +'%''');
  FDQuery1.Open;

With this example if I entered the following into edtSearchTerm.text

a ' union select name, sql from sqlite_master -- 

I will get a list of tables in the database, with that it's simple to see that there another table called salary in the database. Where I can then enter the following to get all the salary information.

a ' union select name, value from salary --

Here is the safe way to do it:

  FDQuery1.SQL.Add('select * from tbl1 where name like :name');
  FDQuery1.ParamByName('name').asString := edtSearchTerm.Text + '%';
  FDQuery1.Open;

Now I can enter both strings that attempt SQL Injection and they will fail. This is because they end up inside the parameter and thereby can't modify the sql. Simple to avoid, but still a common mistake that that is made when it comes to security. Testing for SQL Injection is pretty simple you enter try terminating the input with single and double quote. If you get an unexpected error odds are you have a possibility of SQL injection.

We tend to focus on user input, but API's like SOAP or REST end points are also common targets for SQL Injection attacks. That is why I have been very careful to say all input, and not all just user input.

Security assumptions

When it comes to application security there is lots of discussion on the Internet about web applications and network security.   I however found far less information when it comes to desktop applications and other applications that typically reside behind a firewall.

For example the initial version of App Tethering communication was clear text.    It has a password but it was sent in the clear.    The reasoning I was given that the original was sent in clear text was that it was intended to be used behind the firewall.    This design flaw has since been resolved.  
By default your data in the communication is still clear text, although your passwords now use a HMAC method for authentication with each other.    You now have optional hooks where you can encrypt the data.     Not as nice as communicating over TLS, but is far more secure that the original version.

I tend to take a different approach; I make the following assumptions when it comes to security when I code something.

• Assume there is no firewall and hackers can hit any server you publish.
• Assume a server, network switch or router on your network will be compromised and will be collecting your data.
• Assume the machine your code is on will be compromised and/or stolen.
• Assume that all input can be malicious 

Every day computers around the world are infected with virus, malware, etc...  

These are sometimes these might have been caught by a scanner and sometimes they can be slip past the scanner.  So although a virus scanner is a good idea, it's not perfect. Given that you have to assume that someone on your network is infected, and a hacker now has access to a machine behind the firewall.   

If an application takes these assumptions into account during its design, it will help prevent a data breach from occurring. 

Data Breaches are costly just ask Target and Home Depot.  So spending some time upfront can go a long way in reducing potential unexpected downstream costs.

New Rad Studio Coming - Security?

On April 21st & 22nd there will be a Webinar that goes over the Highlights of RAD Studio.

 Embarcadero presenters will discuss among other things the following topics:

•  A new installation tool with GetIt technology and the choice of what you want to install 
•  Extended support for Bluetooth LE on Windows 10 and a framework for IoT components 
•  FireUI App Preview - Preview your forms on any target device (desktop or mobile)

It looks interesting,  I have had many complaints with installer over the years for taking up too much disk space.   I am also working on IoT devices (usually building them) every week now.   So I am excited to any improvements in this area. 

I also wonder how much of it will have improvements in the security areas that I have concerns with.  Granted most of my concerns deal with the websites more than the product.   I suspect I won't get the information I want in this webinar, but will only come through a review of the product.  

I started contacting Embarcadero regarding several specific security issues on  Aug 30,  2015 right before the Idera purchase.    Some of these were addressed: for example the community toolbar in Rad Studio no longer uses an unencrypted session when you log in.    AppAnalytics uses HTTPS instead of HTTP.

But nothing appeared to change on the websites then on March 12th Embacardero website was hacked,  After a couple of "I told you so" emails, problems were escalated.    I produced a multi page security report detailing issues with every Embarcadero website.   This finally generated some action.    I had a very good call with Atanas Popov the General Manager of Embarcdero's Developer Tools we discussed how they can improve security on both the websites and products.      I now know they are listening, I am now watching to see if the listening turns to action.     

I love Delphi it's a great tool I want to see it succeed and has had a very positive impact on my career.  Given that it's been really difficult to be positive for the past several months, so I have chosen instead to be quiet, but I believe it's time to be publicly vocal.    In my job, I have to deal with security concerns all the time.   My employer is constantly under attack.   So harding systems and software is always a concern.    So I am required to demand that from my from the vendors we use.  If a vendor fails to take security seriously it has a huge negative impact my ability to use that vendor.

So here is to hoping we get some greater transparency and action on security issues.     

Raspberry Pi 3 - Arrived and setup.

After a bit of waiting my new Raspberry Pi 3 Model B just arrived.  

I wanted to share a few of my steps I used in getting started and paired up with my Mac for easy development.

Having played with both Raspbian and Microsoft IOT Core with my Pi 2.  I currently
prefer to use Raspbian, so this article will focus on that.

1.  Download & Install the Raspbian on an SD Card.
    If you're not familiar with dd then you should consider using the NOOBS install method as it's painless.

    Raspbian is based on Debian Jessie or Jessie Lite (Depending on the choice you made)
    I used the one based on Jessie.
 
2.  Insert SD Card, Hook up Monitor, Mouse, Keyboard, and a network cable.

3.  By default the Raspbian installation ships with pi as the default user and raspberry as the default password.   Unless you plan on keeping your Pi off of any network you need change this.   It's just a matter of time until a worm will be developed looking to infect a Pi with a default password.  So change the default password and if desired create and use a different user all together.

4.  As with all new OS installations you should perform updates right after installation.
    This is very good security practice.   It's also the good to do on regular basis.  
   
    To update Raspbian you use apt-get
    The following two commands will get you updated.
 
    sudo apt-get update
    sudo apt-get dist-upgrade
 
    The first updates the list of packages.
    The second performs all of the upgrades and manages the dependencies that they require.
 
    You will want to do this on a regular basis to keep your machine updated.
    If you want you can setup cron to automatically do this.    After reading the article check out the comments for a good bash script, that will email you when updates are available.
 
5.  Change the hostname if you want.   Since I have multiple pi's I rename each image to be unique.
     If you have the graphical interface you can select the following menu to rename the hostname.

• Menu (With the raspberry image on it)
• Preferences
• Raspberry Pi Configuration

    If you don't have the gui you can edit two files to change the hostname.
       
6.  If you have Pi 3 or an earlier Pi with a USB Wifi Adapter you may want to setup wireless.

7.  Now I set up Fuse for OSX on my Mac so I can make my PI a mount point on mac.   This article I found covers how to set this up.    This allows me to use editors on my Mac to edit files on my machine.   I then use SSH to execute that code on the machine.    I then can disconnect the keyboard, monitor, and mouse from the Pi as it's no longer required as I can use my main machine to easily control it.
 
8.  If you need access to the graphical interface from your mac you can do that as well using VNC, due the nature of what I want to do next I am not installing this at this time.

9.   Beware although this setup works.   You are storing your code directly on the Pi.   So when coding you do run this risk of losing your code if something bad happens.   So since git is preinstalled on Raspbian, I tend to initialize a repository in my code directory, setup remote origin and push the changes to a remote repository.     That way you don't lose your work that is on the pi.  I love BitBucket for this as you can as many free private repositories as you want.

CodeRageX - Security Session Files

For those that were able to attend my CodeRage X session.   I had a few files I wanted to share.

I intend to update them, so they may be better than the one shown in the presentation.    However, they are in  GIT so you can see the history if you want.   When recordings are posted I will link to them here.

Windows command script that allows you to generated self signed keys using OpenSSL.
I intend to add a few other scripts and resolve some of the hard coding in the config files.

https://github.com/rlove/OpenSSL-WinCmd-Snippets


I also created a single project that demonstrates how to setup SSL/TLS with Indy and OpenSSL.
That is also stored in a separate GIT Repository.   I wanted to flush out the examples some more.

The repository contains C++ and Object Pascal examples, even though the video was just in Object Pascal.

https://github.com/rlove/Indy-SSL-Examples

Although it may take me a couple of weeks to do updates, due to a vacation that will keep me offline.   I will update both the repository and this post when that occurs.

XE8 Update 1 - Update Subscription

Update XE8 Update 1 was released and there is two different versions of the update one for Update Subscription and one without.

For several years now the company where I work as been on an Software & Maintenance for Rad Studio.  The key benefits that I really like are:

• Support Incidents for when we run into a bug that we can not find a work around.   They work with us to find a solution the problem at hand.   Sometimes it's been a quick and easy problem for the support team and some time more difficult, ultimately we end up with something that works.     One time we ended up with special version of the compiler to solve our problems so we could stay working.    To be honest we don't have to contact support very often at all but it is nice to know it is there.
• New version just arrive without having to deal with a purchasing nightmare, the budget stays relatively fixed so it's easy to plan the budget around the cost.
• My entire team can participate in beta tests.

The Support and Maintenance program was recently was renamed to Update Subscription, and some of the terms have changed. Ultimately I believe these changes are good thing, but has some hurdles to get over, that some customers may not like.   Specifically that customer need to be on Update Subscription to the full set of bug fixes.  But some of the new terms allow for the following:

• Ongoing maintenance for up to 2 yrs for major version and up to 3 concurrent major versions. This really means that our XE6 and XE8 should be getting some of the bug fixes from XE8 coming to it. This is really helpful as we can't keep our entire code base current.
• Webinars and web content that is just for Update Subscription customers.   

We chose not to upgrade to XE8 as we have projects going on right now.  These projects can't afford a delay that would come with an major upgrade the development tools.   But under Update Subscription have value coming for XE6 and XE7 which we do use.    I look forward to those releases but do wish we had an idea when they may be coming.  

So in short I like the Update Subscription.   

PPL - TTask Exception Management

Recently I wrote a blog post titled: "PPL - TTask an example in how not to use." The goal of that post was to help introduce some of the new thought processes that are required around multi-threaded programming.

To keep GUI code responsive, threads can be used to keep time consuming code out of the Main thread where the GUI code runs. For example a good usage for this is database access, and calling web services. But, what happens when the database access or web service call fails?   Using the same methodology as the prior blog post of doing it wrong first, this blog post now exists.

I have modified the code from the prior blog post, where we dropped a listbox and button on a form. The new code now raises an exception during the execution.

 
procedure TForm5.Button1Click(Sender: TObject);
begin
  Button1.Enabled := False;
  SlowProc;
end;

procedure TForm5.FormDestroy(Sender: TObject);
begin
  Task.Cancel;
end;

procedure TForm5.SlowProc;
begin
 Task := TTask.Create( procedure
                var
                   I : Integer;
                begin
                  for I := 0 to 9 do
                  begin
                     if TTask.CurrentTask.Status = TTaskStatus.Canceled then
                        exit;
                     Sleep(1000);
                     if I = 2 then
                        raise EProgrammerNotFound.Create('Something bad just happened');
                  end;
                  if TTask.CurrentTask.Status <> TTaskStatus.Canceled then
                  begin
                    TThread.Queue(TThread.CurrentThread,
                    procedure
                    begin
                      if Assigned(ListBox1) then
                      begin
                        Listbox1.Items.Add('10 Seconds');
                        Button1.Enabled := True;
                      end;
                    end);
                 end;
              end);
 Task.Start;
end;

When we run this code and press the button on the form the button is disabled and then nothing happens. The user gets no notification of the error. That is because the TTask has no way to notify the GUI of the exception. That is up to the developer. Never fail I know how exceptions work just wrap the code with a TRY EXCEPT block and raise it in the main thread.

 
procedure TForm5.SlowProc;
begin
 Task := TTask.Create( procedure
                var
                   I : Integer;
                begin
                  try
                    for I := 0 to 9 do
                    begin
                       if TTask.CurrentTask.Status = TTaskStatus.Canceled then
                          exit;
                       Sleep(1000);
                       if I = 2 then
                          raise EProgrammerNotFound.Create('Something bad just happened');
                    end;
                    if TTask.CurrentTask.Status <> TTaskStatus.Canceled then
                    begin
                      TThread.Queue(TThread.CurrentThread,
                      procedure
                      begin
                        if Assigned(ListBox1) then
                        begin
                          Listbox1.Items.Add('10 Seconds');
                          Button1.Enabled := True;
                        end;
                      end);
                   end;
                 except
                  on E : Exception do
                  begin
                      TThread.Queue(TThread.CurrentThread,
                      procedure
                      begin
                        raise E;
                      end);
                  end;
                 end;
              end);
 Task.Start;
end;

The application is run the application and get some ugly error like this one. "Exception TForm5.SlowProc$2$ActRec.$0$Body$3$ActRec in module Project4.exe at 00208756."

The reason we don't get the correct errors is that the variable of E that is created during the during the TRY EXCEPT block is freed by the time the main thread gets around to raising the exception.

So we try changing this segment of the code from this:

 
TThread.Queue(TThread.CurrentThread,
    procedure
    begin
       raise E;
    end);

to

     
TThread.Synchronize(TThread.CurrentThread,
                      procedure
                      begin
                        raise E;
                      end);

Because the Synchronize will halt the current thread and wait for the main thread to execute the the synchronized code. But we run the code and we are back to nothing happening again, but why?

This is because Synchronize captures the exception and re-raises the exception in the originating thread.

AcquireExceptionObject function to the rescue.

Calling AcquireExceptionObject allows you increment the Exception Object reference count so that it's not destroyed at the end of the TRY EXCEPT Block.   Then we can call TThread.Queue and raise the exception in the main thread.

 
procedure TForm5.SlowProc;
begin
 Task := TTask.Create( procedure
                var
                   I : Integer;
                   CapturedException : Exception;
                begin
                  try
                    for I := 0 to 9 do
                    begin
                       if TTask.CurrentTask.Status = TTaskStatus.Canceled then
                          exit;
                       Sleep(1000);
                       if I = 2 then
                          raise EProgrammerNotFound.Create('Something bad just happened');
                    end;
                    if TTask.CurrentTask.Status <> TTaskStatus.Canceled then
                    begin
                      TThread.Queue(TThread.CurrentThread,
                      procedure
                      begin
                        if Assigned(ListBox1) then
                        begin
                          Listbox1.Items.Add('10 Seconds');
                          Button1.Enabled := True;
                        end;
                      end);
                   end;
                 except
                     CapturedException := AcquireExceptionObject;
                     TThread.Queue(TThread.CurrentThread,
                     procedure
                     begin
                       if Assigned(Button1) then 
                          Button1.Enabled := true;
                       raise CapturedException;
                     end);
                  end;
              end);
 Task.Start;
end;

Now when something bad happens in our task the GUI is notified.  Problem solved! But it's not the whole story, there are other ways to manage exceptions with TTasks, and depending on the nature of your code you this option may be better.

You can remove the TRY EXCEPT Block. When a TTask is executed your user code is already wrapped in a TRY EXCEPT block, and it captures the exception for you already.

If I have a reference to the Task I can call Task.Wait(TimeoutValue), which will wait for the time out for the task to complete and return true if it completed.  If it has stopped executing due to an exception an EAggregateException will be raised in the thread that called Task.Wait() if that is the main thread then the user would be notified of the problem.

TTask has the ability to have N number of child tasks. Because of this exceptions that are raised in a TTask are aggregated together in an EAggregateException object. The EAggregateException is defined with the following public interface.

 
  EAggregateException = class(Exception)
  public type
    TExceptionEnumerator = class
    public
      function MoveNext: Boolean; inline;
      property Current: Exception read GetCurrent;
    end;
  public
    constructor Create(const AExceptionArray: array of Exception); overload;
    constructor Create(const AMessage: string; const AExceptionArray: array of Exception); overload;
    destructor Destroy; override;

    function GetEnumerator: TExceptionEnumerator; inline;
    procedure Handle(AExceptionHandlerEvent: TExceptionHandlerEvent); overload;
    procedure Handle(const AExceptionHandlerProc: TExceptionHandlerProc); overload;
    function ToString: string; override;
    property Count: Integer read GetCount;
    property InnerExceptions[Index: Integer]: Exception read GetInnerException; default;
  end;

With this interface a developer can loop through each individual exceptions, or call .ToString which places all the exception messages into a single string.

Hopefully this give a few more bits of insight into exception management with threads and TTask.

Legacy Code: What goes into a unit

I found this blog post in my drafts folder from August of 2013.  I think I was going to add code examples but never had the time.    I guess it's better late than never, even if I don't have code examples.

As a child I watched Sesame Street, and remember the "One of These Things" song.

"One of these things is not like the others,
One of these things just doesn't belong,
Can you tell which thing is not like the others
By the time I finish my song?"

When spending time in legacy and recent code you are going to find code that mimics that song.

When developing code it is best separate out distinct functionality into different units.

Our system contains 80+ console applications.  Each runs on a unique schedule.  Such as Nightly, Monthly, Quarterly, and Annually.

In our GUI application we have a factory that creates some of our screens.   The factory then knows about those screens.   Using this unit in one of our console applications is not needed.

One day someone on my team noticed several of our console applications had dramatically increased in size.    Looking at the code it was because several method were added to an existing unit that required access to the factory, although none of the console applications actually needed that code.  We had to move that new method to where it belonged.    

When working with an existing piece of code, and you are adding functionality you need to ask does it belong.    One clue to help is if you need to change the uses clause you must be aware of what you are linking not only to your unit but all the other units that use your unit.   When designing new code care should be put into keeping the visual interface out of the underlying business rules.     Functionality should be grouped it a way that when using unit X you are not linking code for units A, B, and C that will never be used.

But now comes the problem with legacy code.  You are not adding new code, your modifying the spaghetti mess that was left for you by someone else.    What should you do?

1. Attempt to understand the existing code... Sometimes this is the most difficult part.
2. If the code can be separated into two different units without changing the actual implementation details then it's far less risky to separate the units.   Then all your doing is adding the new unit to the uses clause of the units that used that code where needed.   Beware if your code uses RTTI this can still break things, depending on how the RTTI was used, specifically since the unit name could have been used as text either in code or an external file.
3. If separation of code can not occur without changing implementation details greater well then you need decide if it's worth it.    Unlike the Matrix movie you get to find out how deep the rabbit hole goes before you take the red or blue pill.  The deeper hole, the bigger the problem. Key factors I use in determining if it's worth changing the design to separate the implementation.
• How many places is the unit used...
• How many places is the unit used, directly. (i.e. in the uses clause of another unit)
• How many places is the unit used, indirectly. (i.e. In another class you inherit from a class that was in the prior unit, then you need to find out how many places the new unit is used. This search will continue on recursively until the unit is no longer indirectly used.
• This allows you to answer some of these questions...
• How critical is this piece of code to my application?
• How much of the application will have to be retested if I change this code?
• What benefits do I get from changing this code?
• Sometimes going through this exercise I have determined what I thought was a  small problem has turned into a very large problem.    

Clear separation of concerns is critical to long term maintainability of a program.

PPL - TTask an example in how not to use.

Delphi XE7 contains a new Parallel Programming Library, which is really powerful and easy to start using.   But it can be something that can be done wrong, and not realize it until it's much later.

Lets take the following fictional example.

There is form with a single button and a list box.   When the button is pressed a long process occurs, when the process is complete it needs to add an item to the list box.

This could be done without Multi-threading.

 
procedure TForm5.Button1Click(Sender: TObject);
begin
  Button1.Enabled := False;
  SlowProc;
  Button1.Enabled := True;
end;

procedure TForm5.SlowProc;
begin
  Sleep(10000); // simulate long process
  Listbox1.Items.Add('10 Seconds');
end;

During testing it is determined that the user needs to be able to re-size the application while the long process is running.

But never fail XE7 has been released with TPL and TTask to the rescue.

The first iteration SlowProc is changed to use a task.

 
procedure TForm5.SlowProc;
var
 Task : ITask;
begin
 Task := TTask.Create( procedure
                begin
                  Sleep(10000); // simulate long process
                  Listbox1.Items.Add('10 Seconds');
                end);
 Task.Start;
end;

Run the application and it appears to work.   Then further testing reveals a couple of problems.

The first being that button can now be pressed multiple times.   The second is that if the form is closed form right after pressing the button an few seconds later and access violation occurs.   

• The reason the button can be pressed multiple times is that the enabled is set back to true after the task is started and not 
• The cause of the access violation is that the code is still executing after the form has been freed.

The second iteration the code is now changed:

 
procedure TForm5.Button1Click(Sender: TObject);
begin
  Button1.Enabled := False;
  SlowProc;
end;

procedure TForm5.SlowProc;
var
 Task : ITask;
begin
 Task := TTask.Create( procedure
                begin
                  Sleep(10000);
                  if Assigned(ListBox1) then
                  begin
                    Listbox1.Items.Add('10 Seconds');
                    Button1.Enabled := True;
                  end;
                end);
 Task.Start;
end;

This now appears to work.  But, now there can be an up to 10 second delay before the application stops running after the form closes.  Now this is fictional example that has just a single sleep() call.   This can occur with real world items as well, but often there are several steps in the method, so I am going to simulate multiple steps, with a loop 0..9 with a call to sleep(1000);

 
procedure TForm5.SlowProc;
var
 Task : ITask;
begin
 Task := TTask.Create( procedure
                var
                   I : Integer;
                begin
                  for I := 0 to 9 do
                     Sleep(1000);
                  if Assigned(ListBox1) then
                  begin
                    Listbox1.Items.Add('10 Seconds');
                    Button1.Enabled := True;
                  end;
                end);
 Task.Start;
end;

Now the fictional example show multiple steps.    But it does not solve the problem with the application running for up to 10 seconds after the main form is closed.     When the form is begin closed the Task needs to be notified so it can stop running. This can be be done with the ITask.Cancel method.

To resolve this a third iteration is produced.

Task : ITask;  has been moved from SlowProc, and is now a member of the form.

 
procedure TForm5.Button1Click(Sender: TObject);
begin
  Button1.Enabled := False;
  SlowProc;
end;

procedure TForm5.FormDestroy(Sender: TObject);
begin
  Task.Cancel;
end;

procedure TForm5.SlowProc;
begin
 Task := TTask.Create( procedure
                var
                   I : Integer;
                begin
                  for I := 0 to 9 do
                  begin
                     if TTask.CurrentTask.Status = TTaskStatus.Canceled then
                        exit;
                     Sleep(1000);
                  end;
                  if Assigned(ListBox1) then
                  begin
                    Listbox1.Items.Add('10 Seconds');
                    Button1.Enabled := True;
                  end;
                end);
 Task.Start;
end;

This all appears to work and is released.   Sometime later in real world strange behaviors and errors are reported on this screen.    After research it is learned that the GUI is not thread safe, so we use a TThread.Queue, to the GUI code to run in the main thread.

Now onto the forth iterations of the code

 
procedure TForm5.SlowProc;
begin
 Task := TTask.Create( procedure
                var
                   I : Integer;
                begin
                  for I := 0 to 9 do
                  begin
                     if TTask.CurrentTask.Status = TTaskStatus.Canceled then
                        exit;
                     Sleep(1000);
                  end;
                  if TTask.CurrentTask.Status <> TTaskStatus.Canceled then
                  begin
                    TThread.Queue(TThread.CurrentThread,
                    procedure
                    begin
                      if Assigned(ListBox1) then
                      begin
                        Listbox1.Items.Add('10 Seconds');
                        Button1.Enabled := True;
                      end;
                    end);
                 end;
              end);
 Task.Start;
end;

Now we have an finally application that should work without error.     Granted this a fictional example, but it shows just some of the pitfalls that can come with multi-threading.  Each is relatively easy to deal with.   

Introduction to GIT

This week at CodeRage 9 I will be presenting.  "Introduction to Distributed Version Control"

My session covers the basics of how a Distributed Version Control System works.   I focus is on GIT but it applies to Mercurial - Hg.

Here is a link to the slides I used for the first 10 minutes of the presentation.

I used 3 clients in the Demo.

• Command Line
• SourceTree GIT/Hg Client in the Demo. 
• Rad Studio XE7

I highly recommend learning the command line tools,   there are two great demos for GIT available that I reference in the presentation.

• Command Line Demo - https://try.github.io
• Command Line Demo with Branching - https://pcottle.github.io/learnGitBranching/

What I don't cover heavily in the introduction is branching.  Understanding this allows you to really leverage the power of Distributed Version Control.  Atlassian has a great tutorial on GIT but the best part of this document is how it compares the various workflows you could use with GIT.

If your coming from SVN and moving to GIT I created this cheat sheet to remember the common commands.

Another thing I wanted to cover but did not have time was options when it comes to hosting your git repositories that I have used.

Public and Private Repositories outside your firewall:

• GitHub
  Great visibility for open source projects.
  Priced per Private Repository, with unlimited users.
  Public Repositories are free.
• BitBucket
  Good Visibility for open source projects
  Priced per user, with unlimited repositories.  5 and under users are Free.
  Public Repository with unlimited users are free.

I have found that I create several small repositories,  and the pricing for BitBucket works well for that.      So I tend to put my private repositories on BitBucket and my Public Repositories on GitHub.

There are several products that are designed for behind the firewall.  I have only used the free Bonobo Git Server.    Really want to switch to Stash, but I have to wait right now due to budget limits.

VCL or FireMonkey - XE7

In a previous blog post I mentioned we are starting a rewrite of an existing application using Delphi XE7 .     One of the comments was raised if we are going to use VCL or Firemonkey.

It was decision we took quite seriously.    Using Delphi  we spent a few days developing Firemonkey sample applications and compared them to what we could do in VCL.    Firemonkey is quite nice.   It has some really nice way of presenting data.  However, ultimately we have decided on writing our application using the VCL.  

Primary reasons we to considered using VCL

• Existing libraries that can be reused.
• We have written 30+ Custom Controls, that we can reuse.
• Still maintained and enhanced.

Reasons we considered Firemonkey

• Can create some really pretty applications, animations, etc... that are more time consuming to create under VCL.
• Gets most of the new features in the Delphi libraries.
• Gives a potential to go cross platform   However, we have no requirements for this.

Reasons we choose the VCL over Firemonkey.

• We have no need to go cross platform with this application
• VCL applications look like the belong on windows, Firemonkey applications look like that are made to work on Windows.
• Firemonkey is a fast moving target, and Delphi upgrades may be more difficult if we skip versions.   
• VCL is still getting improvements (But since it's mature it's not as difficult to adapt)  but it's easier to upgrade between Delphi versions.
• Existing code libraries can be reused.
• Cost we don't have to buy or develop controls we already have.
• Specific VCL controls - (Where we could not find a equivalent in Firemonkey)
• TImageScrollBox in the Envision Library  (Really nice way to view images)
• TRichViewEdit from the TRichView Library (Any RTF editor may work)
• TTrayIcon support 
• TWebBrowser - HTML Viewing Component for desktop, we see it there for mobile.

The good news is that XE7 supports both and I have a decision to make.    I know some have said get rid of one or the other.    I personally like having the choice, they each meet a different set of needs.   

That being said part of my hope is that some of the reasons we chose VCL or Firemonkey are removed are removed in future releases.  I also really want supported interactions that allow us to mix Firemonkey and VCL Code, such as a TFrame in VCL or Firemonkey being able to be embedded into VCL or Firemonkey form.      

Regardless of the the framework choice,  XE7 appears to be a good choice for development and has compelling reasons to use over XE6.

Product/Component Vendors and XE7 Support

Here are the most prominent 3rd Party components(That don't ship in the Rad Studio/Delphi products) that we use.   We may or may not use these in the application we will be rewriting in Delphi XE7 as we have make lots of decisions first such as VCL or Firemonkey.

2 Days after XE7 was released and some people are are on top of there game.
THANK YOU!

The following already support XE7

• JCL 
• JVCL 
• DUnitX
• Delphi.Mocks
• Spring4D
• TMS Component Pack
• FinalBuilder

Patiently waiting on Support for XE7 

• Fast Reports 5
• TMS VCL Subscription 
• Specifically waiting on
• TMS Advanced Charts
• TMS FlexCel component Suite
• TMS Diagram Studio
• TMS Business Subscription (Waiting to purchase )
• SmartInspect 
• Envision Imaging Library

I really like what TMS did for upgrading to XE6 and now they have done the same for XE7.

XE6 Upgrade Story, Now on to XE7...

We started the Delphi XE6 so early in the process we had to convert all the 3rd party components we used to XE6 without help from the vendors.  Our plan was to upgrade Delphi XE6, and move to ContinuaCI.  We also rewrote our FinalBuilder scripts from scratch, keeping in mind we would be upgrading more often.    This project was scheduled to go from April and release at the end of September.    We actually finished development in mid June and released after regression testing on Aug 7th.    The release went really well we had ONE new bug reported in Production    We also had less than 15 reported bugs during the development cycle.    Comparing it to our last major upgrade from Delphi 2007 to XE we were over schedule,  we lost count on the number development bugs, and we had many more problems in production.  
Testing our application suite takes apx 1.5 months to complete.    Due to the time it takes to completely retest the application we tend to have to isolate areas and focus on them for release.    This allows us to have faster testing time as we don't have to as thoroughly test all areas.   A luxury we don't get when upgrading Delphi as all area of the code need to be retested.

Because testing time is such a major part of development, we have been slowly but surely increasing the number of of unit tests.    We also have some research going on in GUI Automation Tools, but we are not pleased with our current options here.     We are also researching adding GUI testing to DUnitX.  Each has benefits and drawbacks that are obvious and others that we will only find out by trying.

That being said our team will not be able to upgrade our whole product suite to XE7 as 3 months of QA time a year to do Delphi Upgrades is too costly.     So we will have to wait for XE8  and maybe even XE9 for our whole product suite to be upgraded.  

That being said XE7 still appears to is a very compelling upgrade.

Our very first Delphi application that was developed here (long before I arrived) is showing it's age, it was originally developed in Delphi 3.    It came complete with every bad programming practice I have ever seen. Global variables, events containing all the business logic, the worst being it's desire to use hidden TEdit as variables.  Granted it was the first Delphi application written by that team, and they were learning the language on this project.... But it's just not worth trying to fix.  Not to mention that the business has evolved and wants something different that what it did.

The last version of Delphi that will touch this application is XE6.  

Today we are starting a project that will rewrite this application, were we will be using XE7.  

Support and how it should work.

During our XE to XE6 upgrade project, we felt it was time to redo our build process.

We are using FinalBuilder and ContinuaCI produced by VSoft Technologies in Canberra, Australia.

During the process we ran into some minor bugs, in the product.    We also had some major and minor feature requests.

Anytime I ran into one of these, I would shoot off an email to the product support email address.  

Sending an email to them would result in a automatic Ticket Number response email from the FogBugz software they are using.   I don't know much about FogBugz, but as customer I love the interaction and being able to see the status of the emailed ticket at any time.

Then after sending the email, I would then leave for the day,  the next morning I would come in and have a response to my ticket.

What made me happy about these responses was most of the time they contained a link to a build with a fix to the problem I was experiencing.     If I had made a minor feature request, that commonly was also implemented in the same time period.     For major feature requests, I was informed that the request was placed on the list to implement, some times with follow up questions to ensure they knew what I was asking for.

Typical response time for a bug fix: < 24 hours.
Longest I remember waiting on a bug fix < 2 weeks.

As a customer I have never felt my opinion or bug was under valued or should be discounted because of some other mitigating factor.    A behavior I commonly get from other companies.

I could be getting lucky with my requests, but I highly doubt it as we have seen this behavior several times over the years we have been using their products.   I am sure sooner or later something nasty will come up that will take longer, but I know it won't just be sidelined, it will be worked on, and I will be kept informed of what is going on.

My point is, VSoft Technologies really has support covered.   I wish other companies, I work with had that level of support.