Sunday, April 16, 2023

Properties of Resilient Enduring Systems

With minimal IT knowledge, the System must be Easy-to-Use and Easy-to-Maintain.

Assume that you, the builder, won't be available.

Our Solution *

We packaged our system as a virtual appliance (a virtual machine) and made it easy to perform the following:
  1. Backup the System to a file
  2. Restore the System from a file
  3. Backup the System's data to a file
  4. Restore the System's data from a file
  5. Perform System admin functions (i.e. add user, update ref table, etc.)
  6. Access the System's daily backup files
  7. View the states of the System's services
No knowledge of the technologies used internally by the System is needed.

* Solution was put into production in 2012.

Tuesday, September 26, 2017

Search and Rescue

Think "Search and Rescue" when a large project failure hits.

It's a big project failure. Front page news. Tens of millions down the drain. Angry executives. Lawsuits. Etc..

The natural response is to move into Monday morning quarterback mode and demand answers. Save that thought. There's a more urgent task at hand: Stop the damage from cascading.

People are hurting now. Wrecked careers and worse are real possibilities. This is especially true for a long running project where team members have invested years of their lives. A project failure is a heart-wrenching event that usually inflicts trauma long before its ugly termination. After termination, that trauma's a 100 times worse. It's an ongoing accident scene.

The ones especially vulnerable are the young engineers. They are not equipped for this; they just don't have the life experience.

You, the senior engineer, PM, manager, (your title here) that's already suffered through one of these nightmares, need to step in immediately and stop the damage from cascading.

How to help:

  1. Search and Rescue - Find them. Listen to them. Let them know they're not alone; that all of us have a Titanic in our past; it's part of growing-up. Let them know that they probably won't be able to make any sense out of this experience for months. It's natural; part of the grieving process. Advise them not to lash out. Be patient. Don't judge. Remember how stupid you were at their age.
  2. Leverage Growth Opportunity - Make sure their next assignment is with a great team that will mentor them up. This is the perfect time, when young engineers are the most thirsty for guidance. They just want to succeed. They'll work especially hard, learn all they can, and grow into great senior engineers.

Example: My boss at IBM back in the day. After my Titanic went down, he did both and saved my career. He found a proven team that would take me under their wing. They mentored me up and the rest is history. Several successful projects. No more Titanics.

Lesson Learned - After a large project failure, find those that are hurting right away. Console them. Then leverage the experience as a major growth opportunity.

Tuesday, August 30, 2016

Release Strategy

My specialty is custom application development. With hard work, some great teams, and a little luck, I've had a long run of success since my embarrassing disaster long ago. One key part of this success is release strategy: (1) breaking releases into right-sized increments, and (2) wisely selecting the content of those releases.


Digging through some project notes, I found a great table that succinctly describes a release strategy.

Release  New Technology New Business Functionality
r3 new SSL network stack  company registration
r4 e-forms w/markdown e-filing
r5 accounting integration e-receipting

Release Guidelines

Planning a great release is hard work. You have to make tough choices. How large should the next release be? What business functionality should be next? How can we implement that business functionality in valuable increments? How can we minimize the technical risk? How can we do all of this in such a fashion that we set the stage for the next release?

Here are some guidelines I like to use:

  • Limit release length to (3) months;
  • Limit release scope to (6) user level use cases (see example);
  • Select use cases with high business value;
  • Focus release on one functional area;
  • Introduce (at most) one major new technology; and
  • Set the stage for the next release.

This is a large release. The number of use cases will double when you work through their variations and failure handling. So don't add more.

And finally, remember that the standard rules apply for that new technology you're excited about: Use Proven Products and No Plumbing.

Lesson Learned: Strategically roll-out large applications as a series of right-sized high business value releases.

Monday, August 10, 2015

Life Matters

All-nighters, (80) hour work weeks, crunch-time, etc.. Are these the hallmarks of great technical leads? No. These align more closely with that of a marathon runner who limps across the finish line hours after the others. Did he finish race? Yes. Did he fare well in the race? Absolutely not.

Don't be silly. You want to win.

Intense arguments, no smiles, stressing over disagreements, explosive outbursts, etc.. Are these the hallmarks of great technical leads? No. These align more closely with my two year old on a grumpy day.

Stop it. You're better than that.

The greatest performers I've worked with don't have these problems and are a joy to be around. They have this characteristic in common: they are great family men and women. Key behaviors I noticed:

  1. They put in great work weeks, then go home. No wacky hours, no coming in to work on the weekends, no email off-hours*, etc.. Sure, they do overtime here and there when truly needed, but this is not the norm.
  2. They lead healthy lives outside of work. Examples: Romance their spouses. Play with their kids. Workout. Fellowship. Volunteer. Read. Cook. Etc..

I remember marveling at these performers back-in-the-day. How do they do it? How do they find the time? More puzzling, there were young stars on our teams that seemed beyond their years they were so good. So what did they all have in common? You guessed it, the same thing.

About ten years ago, I made the decision to join them; to put at least as much energy into my personal life as my professional one. Incredibly, this made me a better performer, teammate, and leader. This simple realignment of priorities did far more for me than those legendary all-nighters ever did.

Lesson Learned: Life matters. Put at least as much energy into your personal life as your professional one.

* Added 'no email off-hours' on 25 Aug 2015. Forgot all about this (because I haven't done it for years now). This takes great discipline, but is well worth it. While it may seem counter-intuitive, this 'break from work' gives you huge energy to attack problems. Don't underestimate what you can do (and how fast you can do it) with a well rested mind.

Wednesday, June 10, 2015

Deep Dive with Docker

For me, the best way to learn new technology is to do something real with it.

I want to learn Docker. We also need to rebuild our VM appliances with Debian 8 (Jessie). Hmm... How about leveraging Docker in our new VM appliances? Maybe we can lower the risk of clobbering app's when applying security updates on the VM's and get some great deployment benefits.

Excellent! We'll get some work done and learn some new stuff too.

Some interesting notes about this experience:
  • Docker has a friendly learning curve. I was delving into the advanced stuff almost right-away without getting lost.
  • The hardest part for me was installing Docker. Doing so on Debian 8 isn't well documented. The only reliable method I found was using Docker's magic script (here).
  • Resources I kept turning back to:
  • A Docker image build is usually created from (2) simple files:
    • `Dockerfile` - Describes how to build a new app container
    • `` - Shell script to bootstrap your app container on first run
  • You can use and share Docker images on Docker Hub.
  • You can easily create automated builds of these images from your GitHub repositories.
  • My images are based on some great examples on Docker Hub:
  • My open source images:
  • Current approach to keep my containers small, generic, and ephemeral:
    • Use official Debian image as a base (debian)
    • Use volumes on the host for persistent file storage
    • Bootstrap the container settings via an `` script
  • Creating great images for containers is harder than it looks. Getting the configuration 'just right' takes time, But at least you only have to do this work once!
  • Once you have your images built, it's a breeze to spin-up containers and deploy your app's.
  • Be very security conscious when using Docker. Make sure you understand each build's `Dockerfile` / `` combo line-by-line (it's only a page or two, you can do it).
  • Given the youth of Docker, expect significant changes in the near term to address security.
Go ahead, dive in, the water's nice.

Monday, April 20, 2015

Why FrontAccounting?

Once again, we found ourselves going with an open source product for a key component of our client's system. Here's the solution fit statement. A generic 'Company' is used in-place of our client's name.

Why FrontAccounting
The Accounts section of the Company was tasked in late 2014 to switch to a formal electronic accounting system. As such, a large number of commercial and open source products were evaluated. Ultimately we settled on FrontAccounting (FA) for the following reasons:

  • Traditional Accounting - FA is a classic double entry accounting system. Many products we evaluated did not operate this way or hid this functionality behind a checkbook style user interface.
  • Security and Privacy - FA runs in-house on the Company's own infrastructure using strict access controls and modern cryptography [1]. Many products were available online only and used outdated cryptography. More, most of these systems were hosted in the United States where the privacy of the Company's data is not guaranteed.
  • Right Sizing - FA is designed for medium sized organizations like the Company. Some products we evaluated were too simplistic to handle multiple currencies and user access security. While other products were too large and complicated, having far more functions than the Company would ever need or want to support. FA was one of the few products to sit between these two crowded product spaces.
  • Ease of Customization - As an open source product, FA is easy to customize and taylor to the Company's needs. With FA's development documentation and source code readily available online, we can easily extend FA to integrate with the Company's other systems (i.e. the Company's Payroll and Registration Systems).
  • Ease of Deployment - FA is easy to deploy, backup, and restore; especially when compared to large ERP systems. Ease of deployment is a critical need for us given (1) the high failure rate of computing infrastructure in this environment, and (2) our limited technical resources. 
  • Ideal Licensing - FA is free software, released under the GNU General Public License. This eliminates having to deal with frustrating licensing issues (i.e. software license expiration, number of licensed users, access to help resources, access to bug fixes, maintenance contracts, ownership changes, etc.).
  • Low Cost - No licensing fees.

FrontAccounting Resources

  • Home - Title page
  • Wiki - Resource for installation and configuration tips
  • Forum - Feedback from developers and users
  • Download - sourceforge

Quick Start

[1] Strong cryptography does not come 'out-of-the-box.' We configured the FA web application to use modern cryptography using TLS 1.2, encrypted using AES_28_GCM with an ECDHE_RSA key exchange mechanism. We did so the traditional way using OpenSSL and Apache HTTP Server on Linux.

Tuesday, January 20, 2015

Team-Up with a Great Project Manager

Great project management is a critical success factor (CSF). The project's management lead is as critical as the technical lead. Young architects, downplay this. Seasoned architects who have been burned by large project failures, know better.

For a project of any significant size, you simply don't have time to do project management. So don't try. Examples of duties you don't have time for include: managing the client relationship, managing the contract, change orders, proposal writing, costing, HR management, time tracking, billing, etc..

These days, I team-up with proven PM's who've taken multiple projects the distance successfully. I sit next to them and do everything in my power to assist so that they can help us get to the finish line. It's a team thing, a close partnership, a great one-two punch. If it's tech, you got it; if it's management, your PM's got it.

More, I structure the work in such a manner, that the PM has great understanding and visibility - beyond the project plan's task list. I think of it as providing management with a great project dashboard (gauges and controls). With this management can leverage their skills to help us tackle the project and drive it to a successful finish. Some dashboard examples:
  • Use case list with statuses (what's being delivered)
  • Test case list with status by release (how ready is it)
  • Issue list with statuses (what's standing in the way)
  • Technical decisions with statuses (what decisions were / need to be made)
  • Etc.

Lesson Learned: Team-up with a great project manager (PM). Provide your PM great visibility to the technical work and processes. And lead together to drive the project to a successful finish.