Going Solr

Over the last couple of weeks, we whipped-up an enterprise search site using Apache Solr. It was a fantastic experience. Solr is impressive. My first proper exposure to full text search (FTS) technology was via its inclusion in a recent version of PostgreSQL (see earlier post). I quickly learned that incorporating FTS does wonders for user satisfaction. So upon discovering the large number of public websites that are Solr Powered, we had to take Solr for a spin ourselves. We were not disappointed.

Some interesting notes about our experience:

• Solr is not a turn-key solution; it's a search engine for a solution you create.
• Doing the tutorial was critical to understanding Solr.
• XSLT worked great for rendering Solr's XML results.
• Some helpful documentation sources we found:
  o Apache Solr (tutorial and presentations);
  o Lucid Imagination (lucidworks ref guide); and
  o w3schools.com (xslt tutorial).
• Solr is lightning fast.
• Solr is packed with enterprise features. However, like most open source tools, you must have an adventurous spirit to realize them.
• Jetty makes deploying web / app servers simple. Launching the Solr example app was literally a one-liner. Incredible.

Tip: If you're implementing an enterprise search engine, consider using Apache Solr.

20 Jan 2012 Update - Posted the detailed 'how-to' for indexing office documents with Solr (source code and all). Enjoy.

Fastest Way to Learn

Here's the fastest way I've found to learn the core skills of an IT architect / specialist. Unfortunately and with much embarrassment, I didn't figure this out on my own. It took that huge project failure mentioned earlier, a great boss, and caring mentors to set me on the right path:

1. Find an experienced team with a history of delivering projects into production, and do whatever it takes to get on that team.
2. Stay with the project through delivery to production. Even better, stay with the project through the first service release.
3. Be a major contributor. Shift rolls as necessary to maintain this status (i.e. requirements lead to design lead to testing lead).
4. Keep your eyes and ears open. Roll up your sleeves and learn everything you can from the 'stars' on that team.

Repeat this as often as you can throughout your career to continue your growth and keep your skills current.

Lesson learned: To accelerate learning and keep your skills current, serve as a major contributor on an accomplished delivery team throughout a project's life-cycle. Repeat.

16 May 2012 Update: Be sure to read the Fastest Way to Learn Part 2.

AQT Saves the Day

AQT has several time saving features like Export

Relational databases are pervasive in the enterprise. More, they come in all sorts of shapes and sizes in the same shop - from modest desktop DB's like Access / Excel to open source workhorses like PostgreSQL / MySQL to top-shelf like Oracle / MS-SQL / DB2 to old powerhouses like Informix.

Wouldn't it be nice if you could have just one tool to interact with all of these databases: construct queries, view data, import / export data, perform administration, compare / move data between databases, etc.? Advanced Query Tool (AQT) is that tool. Like the SQL Cookbook, AQT comes highly recommended by those same 'scary good' DB guys.

I've found AQT to be especially valuable when working with: (1) old beasts that don't have nice interactive query tools, and (2) newer beasts that require access to the DB's installation media and heavyweight installs. Our team here uses AQT everyday for developing, testing, and supporting the State's production systems.

Note: The inspiration for this post was from one of my colleagues. He used AQT to verify the recovery of image files into blob fields on an Informix database. You can imagine the command line wizardry this would require had he used the database's built-in tools.

5 Oct 2011 Update - AQT has come to our rescue again twice in the last couple of days. We used the data loader to recover about 14,000 records, and used the data compare to identify duplicate records between production and archive databases.

TestLink is a Great Test Case Management Tool

There are dozens and dozens of testing tools out there in the wild. For large custom application development projects, I've found that using a tool to store and manage test cases is very valuable. It's just too resource intensive to do this work by hand. In the old days, we'd roll-our-own solution by using the document management features of a tool like Notes or Exchange. We'd customize them so that we could create test suites, enter test results, add build numbers, assign testers, etc..

This brings us to today. Surely, someone has built a great open source tool for test case management. Thankfully, the answer is a resounding YES. Of the choices available, we selected TestLink. It's open source. It's simple. It's proven (used by test shops at Intel, Lucent, Philips, Samsung, Symantec, Toshiba, Yahoo, etc.).

After using TestLink during the system testing of Wyoming's new excise tax system (went live July 2011), I can confidently recommend TestLink. Management likes it, the users like it, and their crazy architect likes it.

Discovery: If you plan on performing formal testing for a large project, consider using TestLink.

Use Cases Example

Continuing the example from the previous post, here's a selection of use cases from that project (click here or on the pic to see the detailed example). These were written using a modified form of the fully-dressed use case template from Writing Effective Use Cases. Some notes regarding the example:

o Version History - Evidence that use cases are living documents that evolve with your project;
o Use Case List - The table form of the use case diagram (often a PM's best friend); and
o (5) Use Cases - Implemented use cases. Note their brevity. Note how the complexity of multiple flows are accounted for in their extensions. Note how the main flow focuses on the frequently traveled path (the happy path).

How to Read Extensions: The extension number indicates the step in the main success scenario for which a different path may be taken. They are usually used to describe significant options you must provide or special failure handling that you must address (rather than just give an error message and end).

Low tech search tags: Real world use case example, actual use case example, use case list example, example using fully-dressed use case template, use cases from a real project, use cases from a successful project, great use case example, detailed use case example, professional use case example, recommended use case example, use case model example, use case model from a successful project, use case work product.

Use Cases Evolve

R1 Planned

R1 Implemented + R2 Planned

R2 Implemented + R3 Planned

R3 Implemented + Rx Planned

The use case version history, use case list, and use case text for the above can be found here in a follow-up post.

Healthy systems evolve. With each release, you add the knowledge gained from the previous release and improve the next. So naturally, your use cases will reflect that. To illustrate this, take a look at the use case diagrams for a business registration system built for the Government of Samoa. Note how they progress from release to release. Note how the implemented use cases evolve from the planned.

This registration system continues to evolve. It has 21 releases so far, with 5 major releases (include several new/updated use cases). To insure the right business functionality was built, 'worked closely with the client business experts on the use cases as each major release progressed. New use cases were discovered, some existing ones updated, and others retired. This is normal for a healthy development cycle.

Note: The meat of the use cases is the text. To me, the diagrams are a visual form of the use case list: they do a nice job of listing the use cases, their actors, and functional groupings. I'm not a big fan of / nor recommend using the fancy UML include/extends/generalization constructs; it's just too easy to venture into design and forget to concentrate on the use case text.

SQL Cookbook to the Rescue

This week was rough. We were auditing complex financial records in one of our new production databases. To do so, we had to wield some advanced SQL. I had to study-up on joins and a few other advanced techniques. How did I do it? I had a great coach: DB2 SQL Cookbook by Graeme Birchall. I found an old original PDF (here), a modern HTML fork (here), and more via Google (here).

On a previous project, some 'scary good' data mining experts and DBA's recommended this book. I've been using it ever since. Those guys were so right; this is an excellent resource for the practitioner. More, the content applies regardless of the database you use. Just adjust the syntax slightly for your target database engine and you'll be slaying queries like a pro in no time. Now here's the kicker: Graeme’s book is free to download… so no excuses, go get that data!

Favorite recipe: Multiple Counts in One Pass

13 Aug 2015 - Updated links (old ones were decommissioned).
31 Jan 2022 - Updated links again (same reason).