Move over design patterns, here come crash patterns! I'll be talking about this concept in my "Server Down" CFUnited 2008 presentation in June. As many of you know, we here at Webapper have gotten loads of "server down" calls and emails over the years, so the purpose of my talk is to organize all of what we've seen into best practices for preventing performance and stability problems, and also for what to do when problems do arise. And one way we've started to organize this information is with the concept of "crash patterns". It's modeled after the familiar idea of design patterns, but applies instead to common/recurrent causes of performance and/or stability problems (the two usually go together). And, as with design patterns, once you come to understand crash patterns (both in general, and those that are specific to your applications), you can use them to make your life a whole lot easier. And when it comes to performance and stability, that means using the knowledge of crash patterns to avoid these problems in the first place, and it also means having optimal steps to take when/if problems do arise on your production systems.

I hope to see tens of thousands of you packed into the room in D.C! In the meantime, if you have any thoughts/comments/questions, please send them along, as I'd love to incorporate even more community/attendee input into my presentation.

SeeFusion 4.0.7 is now available for download at http://www.seefusion.com.

Primarily, this update removes the need for the JDI debug settings that were required to produce stack traces with version of ColdFusion before 8, which ran on Java 1.4. (There is no change for ColdFusion 6/7, which still require JDI to gather stack traces under Java 1.4.)

The update also squelches the reporting of spurious "Auto-generated keys were not requested" exceptions being caused by changes in the way ColdFusion 8 interacts with database drivers. (ColdFusion 8 silently ignores those exceptions, so SeeFusion now does the same.)

ColdFusion 8 added the ability to specify a "validation query" that will be run every time a connection is reused in a connection pool. This is useful in cases where database servers or firewalls sometimes drop connections without warning, as it causes a new connection to be made to the database server for the page, so that no user sees an error caused by a stale database connection. A good validation query would be something like "SELECT 1" for SQL Server or Sybase, or "SELECT 1 FROM DUAL" for Oracle.

We've seen a few customers lately that have had the value "0" sneak in as their Validation Query for datasources in the ColdFusion Administrator. I've tried and failed to reproduce this, but zeros don't magically appear in that field for me; suffice to say that unrelated people were finding "0" as their validation query, and seeing SeeFusion report errors such as these in their coldfusion-out.log files:

...or...

...or...

Note that while SeeFusion is reporting the error, it's not causing the error.

n.b. If you read the stack from the bottom up, it's telling its story: QueryTag.executeQuery() calls DataSrcImpl.getConnection(), which tries to getCachedConnection(), which hits a JDBCPool object (connection pool) that does a .checkout(), followed by a .validate(). The validate() method calls (after passing through SeeFusion's driver wrapper) the db driver's Statement.execute() method, and that database driver throws the exception.

The deeper, more pernicious problem, is that there is no way to test validation queries in the ColdFusion administrator. And when an invalid validation query is used, connection pooling is basically disabled, forcing new connections to be made to the database for every page view.

Interestingly, when ColdFusion validates the datasource, it runs the validation query, but silently ignores the exception thrown. (We used SeeFusion to see which JDBC calls the ColdFusion Administrator is making.)

Feature request for the next ColdFusion 8 service pack: add a warning message when you save a datasource and the validation query fails!

-=-=-=-=-=-=-=-=- Update -=-=-=-=-=-=-=-=-

As it turns out, there's a bug in the ColdFusion 8 Admin API for datasources. Any datasource you create with the Admin API will have the validationQuery set to "0", no matter what value is actually set (and even if that value is intentionally set blank). This ColdFusion snippet demonstrates:

If you're using ColdFusion to access BLOBs stored in a Sybase server, you may have a problem.  Apparently, Sybase severely limits the timeframe wherein you're allowed to retrieve the actual binary data, which is supposed to be available until the end of the current transaction.  ColdFusion doesn't pull the backing binary data immediately, and when it does, Sybase's driver returns a 0-byte result.

We've implemented a workaround in SeeFusion's driver for this.  If you append ";convertBlobToBytes=true" to the end of the JDBC URL, SeeFusion (starting with version 4.0.7) will quietly convert any BLOB responses to byte array objects before returning the data to ColdFusion's JDBC getObject() call.  I'm not aware of any other drivers having this issue, but it should work for any other driver as well.

Hey all. Just wanted to let everyone know that we're nominated for SysCon's Readers' Choice awards (CFDJ and WebDDJ) for our consulting services and for our SeeFusion tool. Our consulting services are in the Best Consulting Service category (duh), and SeeFusion is in the Best Web Development Tool and Most Innovative Application categories (you can vote in whichever category you think is most fitting). If you've benefited from either SeeFusion, and/or from our tuning/troubleshooting consulting, we'd be honored to have your vote!

Or, if you know someone who has, or if you know someone who knows someone, or...sorry, I'll stop now. I'm from Chicago, where the theme for every election is "Vote Early, Vote Often." ;)

Seriously, we'd love your vote. Please help get the word out if you can. Here's the link to vote, so it's handy:

SysCon's Readers' Choice Awards.

Best,
Patrick

If you're here at MAX, just a quick reminder that we'll be doing our SeeFusion demo at MAXUP this morning right after the keynote address. We'll be giving away lots of swag--tshirts and free software. See you there!

Just a friendly reminder about our earlier post this week. We're giving away 5 3-server copies of SeeFusion throughout the week at MAX. Just track one of us down in our white Webapper shirts (logo on front), and we'll give you a password to play our slot machine game in order to try to win.

Here's the game site:

http://www.seefusion.com/max2006/

Also, I've linked to the original post below (click on the title if you don't see the related entry). See you there!

We're giving away up to 5 3-server copies of SeeFusion 4 at MAX, which, with our special 2-for-1 licensing plan, will allow you to monitor up to 6 ColdFusion servers. Each 3-server license has a retail value of $897!

Here's how the giveaway will work. We're sponsoring CFUnderground 8 on Sunday, attending MiniMax on Sunday evening, presenting at the MAXUP area, and of course we'll be at MAX all week, too. In the spirit of Sin City, we're launching a slot machine application, and each player will be given a preset spending limit. You'll play the slots until you win a copy of SeeFusion, or run out of money. In order to play, all you need to do is track one of us down, and we'll hand you a business card with the slot machine URL, and a special entry code, and you'll be able to play wherever/whenever you like. We'll all be wearing white t-shirts with our Webapper logo on the front (SEE BELOW).

Looking forward to seeing you all there, and to giving away SeeFusion!

Best, Patrick

Greetings, all. We''re excited to announce that we formally released SeeFusion 4 on Monday! Here''s a quick summary of new/enhanced features:

• Flex 2 rich interface! From real-time charts to spreadsheet export, this is the crown jewel of SeeFusion 4. Never again can it be said that SeeFusion is great engineering with a dull UI!
• Active monitoring rules. Create user-defined server events that SeeFusion will react to automatically by killing requests, logging events to the SeeFusion database, or sending email alerts.
• Enhanced database logging, including logging of user-defined server events.
• Stack trace filtering. Quickly parse out only the parts of a stack trace that you want to see, filtering out all the rest.
• Debug output by IP address. Manually enter an IP address to view SeeFusion''s debug output for only that user, in order to facilitate addressing user-reported problems
• Streaming running requests. Use a convenient slider control to set a request duration threshold, in order to see all running requests beyond that limit quickly and easily, without having to change the overall slow request threshold.
• Improved trace() method. The SeeFusion.trace() method now gets called automatically for any unhandled exceptions. This can be useful in debugging errors with little or no information available, such as with some CFC errors.
• New 2-for-1 pricing model for individual servers. For any single physical server, every 2 ColdFusion instances that you want to monitor now require just 1 SeeFusion license.

Extended evaluation keys are available to anyone who''d like one. Just ask!

As always, send along any and all questions/comments to our support address--support (at) seefusion.com. Or, call us toll-free (US only) at 866.816.4700 (International customers dial {IDD prefix} + 1.970.223.2278).

Best,
Patrick Quinn
Co-Founder, President & CTO
Webapper Services, LLC

We made an interesting discovery in a recent consulting engagement, and we wanted to share what we found with the community. The gist of what we found is that cfswitch, with a string expression, and especially under load, runs dramatically slower than the equivalent cfif-cfelseif-cfelse block. Here''s what happened...

A customer purchased one of our one day remote consulting engagements. They reported an intermittent performance problem that they were unable to pin down. They were already a SeeFusion customer, so we used SeeFusion to generate stack traces on the running requests. This line showed up quite often in the traces we took:

  "jrpp-27" prio=5 tid=0x09572cb0 nid=0x12d4 runnable [5da2f000..5da2fdb8]
    at java.lang.FloatingDecimal.readJavaFormatString(Unknown Source)
    at java.lang.Double.parseDouble(Unknown Source)  

Elsewhere in the stack traces we saw evidence of cfswitch usage. So we did some further digging, and sure enough, the customer was using cfswitch tags inside query loops--not an uncommon practice at all. But the switch expressions were all strings. So here''s the kicker. Under the covers, ColdFusion attempts to convert the switch expression to a floating point number, using Java''s parseDouble method. If the conversion works, then ColdFusion uses the expression as a number. If the conversion fails, however, then an exception is thrown, and an exception stack is generated, but then ColdFusion handles the exception and uses the expression as a string. The problem is, that exception throwing and stack generation gets very, very expensive under load.

We cooked up some tests to verify, and sure enough, on average the performance difference between cfswitch with string expressions, and the equivalent cfif-cfelseif-cfelse blocks, was a 10-fold difference. Under load, that will multiply dramatically.

We had the customer convert their cfswitch blocks, and they had an immediate improvement in performance and stability. There were other things left to tune, but this was a primary culprit.

We also found evidence that the floating point conversion might be single-threaded, which would make matters even worse. But the evidence for this wasn''t clear, and there also appeared to be some variation across different VM versions, but we had to move on to other things and couldn''t pursue this other point.

Moral of the story--only use numeric expressions for cfswitch! Otherwise, use equivalent cfif-cfelseif-cfelse blocks. And this is especially true if your code will be running under heavy load.

Test files