Why Git?

OK you architect types, here's a classic solution fit statement... enjoy.

Why Git
Here at the State and in past lives, we used a number of SCM systems. Ultimately, our Department settled on Git for the following reasons:

• Standard - Git is the long overdue successor to CVS and has quickly become the defacto standard for SCM. As such, great documentation and extensive tooling are widely available.
• Best Practice - Old school limitations like check-out locking are long gone. High value features from years of SCM system evolution are built-in.
• Cross Platform - Our systems, code, and tools span multiple platforms. As such, we require an SCM that plays well across platforms.
• Low Admin Overhead - Git is easy to setup and maintain. Some SCM systems had great capabilities but extraordinary administrative requirements (i.e. configure / maintain multiple large products across multiple servers) - this is not workable for small teams.
• Ideal Licensing - An open source SCM eliminated hairy licensing issues. For example, proprietary SCM's had strict remote access licensing requirements; making working with outsourced resources difficult.
• Low Cost - No licensing fees.

Git Resources
git - Git Home
msysgit - Git for Windows (core)
TortoiseGit - Git Windows Explorer Integration

Recommended Doc's

Pro Git - The book

Indexing Documents with Solr

Apache Solr is great at indexing thousands and thousands of office documents (Word, Excel, PDF, etc.). But where's the handy tool you use to upload all those documents to Solr for indexing? You have to write it friend. Ahh... the joys of open source! As such, we wrote one. It's working great here at the State. It's been in production running hourly over the last couple of months, and we just completed a substantial upgrade last week.

Here it is: sinject v1.3.

Like Solr, Sinject is under the Apache license. Enjoy. If you find it useful or can offer some advice, drop us a note.

Core of Sinject - Submitting Files to Solr

1. Get the target file's unique id (inode) and other info:
   @file_info = stat($filepath);
   $inode = $file_info[1];
2. Stage the file to be indexed (eliminate issues with bad filenames):
   # deal with possible quotes in filepath
   $filestring = $filepath;
   $filestring =~ s/'/'\\''/g;
   # copy the file to a temp file w/o weird ch's
   # (retain the file's mod date)
   $cmdstring = "cp -p '$filestring' " . "tempfile";
   # do it!
   $cmdoutput = `$cmdstring`;
3. Submit file to Solr indexing engine (remember to escape troublesome literals):
   # Solr upload and index file command
   $cmdstring = "curl \"$SOLR_URL/update/extract?" .
       "literal.id=$inode" .
       "&literal.filename=" . uri_escape($filename) .
       "&literal.filelink=" . uri_escape($filelink) .
       "&literal.filetype=$filetype" .
       "&literal.filedate=$filedate" . uri_escape($timesuff) .
       "&commit=false\" -F \"myfile=\@tempfile\"";
   # do it!
   $cmdoutput = `$cmdstring`;
   # check for successful result
   if ($cmdoutput =~ m/<lst name="responseHeader"><int name="status">0<\/int>/) {
       # success - move on to next file
   } else {
       # fail - log failure, then move on to next file
   }
   
4. Don't forget to commit the updates (i.e. every 100 or so):
   # Solr commit command
   $cmdstring = "curl '$SOLR_URL/update' -H " .
       "\"Content-Type: text/xml\" --data-binary " .
       "'<commit/>'";
   # do it!
   $cmdoutput = `$cmdstring`;
   # use same successful result check as above...

Check-out the code for further detail.

(see earlier post for info about our initial experience with Apache Solr)

Low tech search tags: How to index office documents using Solr, how to upload office documents using Solr, how to index rich text documents using Solr, how to upload PDF documents to Solr, how to upload documents using Solr Cell, Solr document uploader, Solr PDF uploader, Solr rich document crawler, Solr cell sample code, example of using Solr cell.

Going Solr

Over the last couple of weeks, we whipped-up an enterprise search site using Apache Solr. It was a fantastic experience. Solr is impressive. My first proper exposure to full text search (FTS) technology was via its inclusion in a recent version of PostgreSQL (see earlier post). I quickly learned that incorporating FTS does wonders for user satisfaction. So upon discovering the large number of public websites that are Solr Powered, we had to take Solr for a spin ourselves. We were not disappointed.

Some interesting notes about our experience:

• Solr is not a turn-key solution; it's a search engine for a solution you create.
• Doing the tutorial was critical to understanding Solr.
• XSLT worked great for rendering Solr's XML results.
• Some helpful documentation sources we found:
  o Apache Solr (tutorial and presentations);
  o Lucid Imagination (lucidworks ref guide); and
  o w3schools.com (xslt tutorial).
• Solr is lightning fast.
• Solr is packed with enterprise features. However, like most open source tools, you must have an adventurous spirit to realize them.
• Jetty makes deploying web / app servers simple. Launching the Solr example app was literally a one-liner. Incredible.

Tip: If you're implementing an enterprise search engine, consider using Apache Solr.

20 Jan 2012 Update - Posted the detailed 'how-to' for indexing office documents with Solr (source code and all). Enjoy.

Fastest Way to Learn

Here's the fastest way I've found to learn the core skills of an IT architect / specialist. Unfortunately and with much embarrassment, I didn't figure this out on my own. It took that huge project failure mentioned earlier, a great boss, and caring mentors to set me on the right path:

1. Find an experienced team with a history of delivering projects into production, and do whatever it takes to get on that team.
2. Stay with the project through delivery to production. Even better, stay with the project through the first service release.
3. Be a major contributor. Shift rolls as necessary to maintain this status (i.e. requirements lead to design lead to testing lead).
4. Keep your eyes and ears open. Roll up your sleeves and learn everything you can from the 'stars' on that team.

Repeat this as often as you can throughout your career to continue your growth and keep your skills current.

Lesson learned: To accelerate learning and keep your skills current, serve as a major contributor on an accomplished delivery team throughout a project's life-cycle. Repeat.

16 May 2012 Update: Be sure to read the Fastest Way to Learn Part 2.

AQT Saves the Day

AQT has several time saving features like Export

Relational databases are pervasive in the enterprise. More, they come in all sorts of shapes and sizes in the same shop - from modest desktop DB's like Access / Excel to open source workhorses like PostgreSQL / MySQL to top-shelf like Oracle / MS-SQL / DB2 to old powerhouses like Informix.

Wouldn't it be nice if you could have just one tool to interact with all of these databases: construct queries, view data, import / export data, perform administration, compare / move data between databases, etc.? Advanced Query Tool (AQT) is that tool. Like the SQL Cookbook, AQT comes highly recommended by those same 'scary good' DB guys.

I've found AQT to be especially valuable when working with: (1) old beasts that don't have nice interactive query tools, and (2) newer beasts that require access to the DB's installation media and heavyweight installs. Our team here uses AQT everyday for developing, testing, and supporting the State's production systems.

Note: The inspiration for this post was from one of my colleagues. He used AQT to verify the recovery of image files into blob fields on an Informix database. You can imagine the command line wizardry this would require had he used the database's built-in tools.

5 Oct 2011 Update - AQT has come to our rescue again twice in the last couple of days. We used the data loader to recover about 14,000 records, and used the data compare to identify duplicate records between production and archive databases.

TestLink is a Great Test Case Management Tool

There are dozens and dozens of testing tools out there in the wild. For large custom application development projects, I've found that using a tool to store and manage test cases is very valuable. It's just too resource intensive to do this work by hand. In the old days, we'd roll-our-own solution by using the document management features of a tool like Notes or Exchange. We'd customize them so that we could create test suites, enter test results, add build numbers, assign testers, etc..

This brings us to today. Surely, someone has built a great open source tool for test case management. Thankfully, the answer is a resounding YES. Of the choices available, we selected TestLink. It's open source. It's simple. It's proven (used by test shops at Intel, Lucent, Philips, Samsung, Symantec, Toshiba, Yahoo, etc.).

After using TestLink during the system testing of Wyoming's new excise tax system (went live July 2011), I can confidently recommend TestLink. Management likes it, the users like it, and their crazy architect likes it.

Discovery: If you plan on performing formal testing for a large project, consider using TestLink.

Use Cases Example

Continuing the example from the previous post, here's a selection of use cases from that project (click here or on the pic to see the detailed example). These were written using a modified form of the fully-dressed use case template from Writing Effective Use Cases. Some notes regarding the example:

o Version History - Evidence that use cases are living documents that evolve with your project;
o Use Case List - The table form of the use case diagram (often a PM's best friend); and
o (5) Use Cases - Implemented use cases. Note their brevity. Note how the complexity of multiple flows are accounted for in their extensions. Note how the main flow focuses on the frequently traveled path (the happy path).

How to Read Extensions: The extension number indicates the step in the main success scenario for which a different path may be taken. They are usually used to describe significant options you must provide or special failure handling that you must address (rather than just give an error message and end).

Low tech search tags: Real world use case example, actual use case example, use case list example, example using fully-dressed use case template, use cases from a real project, use cases from a successful project, great use case example, detailed use case example, professional use case example, recommended use case example, use case model example, use case model from a successful project, use case work product.