There's an article in the latest eWeek magazine, entitled "Scaling Ruby on Rails", that's instructive for those of us in the ColdFusion community. RoR is going through that time-honored rite of passage experienced by any programming language that gains wide adoption. Namely, so-called critics are questioning its scalability and its suitability for mission-critical applications. Sound familiar?

One supporter of Rails notes the following:

"The critiques we hear about Rails is it's not scalable, that it's not well-suited for mission-critical applications. I think those critiques are similar in nature to what we heard about Java in the mid-90s."

And my favorite retort to the scalability questions comes from Rails creator David Heinemeier:

"This is the known as the 'last stance' defense. When you have nothing left of substance to argue with, you draw the 'but does it scale?' card."

Awesome. Amen, my brother. I couldn't have said it better myself. And the exact same response applies to ColdFusion, just as it did to Java and many others over the years. Once you've tuned a few hundred queries in your career from 3000ms to 30ms, you come to know that the scalability question is almost always a red herring. To paraphrase a quip from politics, "It's the code, stupid!"

Our co-founder and former CTO, Mike Brunt, tells a great story about when he was onsite doing some of our famed tuning work some years ago. He had been applying our tuning efforts on a system well into the night during an engagement, and around 6am the next morning, when user traffic ramped up dramatically every day, he got a frantic call from the CEO, breathlessly saying that the servers were all down. And why did he think this? Because the traffic monitoring graphs had all dropped to sub-1-second response times, such that they looked so different than "normal" that that he concluded there must not be any traffic on the servers. In true British form, Mike said something along the lines of "Bollox to you and your bad code--the servers are fine!" Again, "It's the code, stupid!"

So, to all ColdFusion compatriots--don't fall for this nonsense. Ever. If you hear it, smile. It's just a reflection of the prominence of your platform. And if you're stuck on performance problems, contact us--we've never NOT tuned a system we've been engaged to tune.

To our brethren in the Rails community--welcome to the party. Come on in, the water's warm. Now that you're hearing all the same "last stance" questions that ColdFusion has been dispatching for years, we wish you all the growth that ColdFusion has experienced!

Move over design patterns, here come crash patterns! I'll be talking about this concept in my "Server Down" CFUnited 2008 presentation in June. As many of you know, we here at Webapper have gotten loads of "server down" calls and emails over the years, so the purpose of my talk is to organize all of what we've seen into best practices for preventing performance and stability problems, and also for what to do when problems do arise. And one way we've started to organize this information is with the concept of "crash patterns". It's modeled after the familiar idea of design patterns, but applies instead to common/recurrent causes of performance and/or stability problems (the two usually go together). And, as with design patterns, once you come to understand crash patterns (both in general, and those that are specific to your applications), you can use them to make your life a whole lot easier. And when it comes to performance and stability, that means using the knowledge of crash patterns to avoid these problems in the first place, and it also means having optimal steps to take when/if problems do arise on your production systems.

I hope to see tens of thousands of you packed into the room in D.C! In the meantime, if you have any thoughts/comments/questions, please send them along, as I'd love to incorporate even more community/attendee input into my presentation.

At Webapper, we've always described our tuning & troubleshooting consulting as a "wire-to-wire" service. This means that we find and fix performance and/or stability problems wherever they are, even if they're in the network layer (e.g., the TCP "silly window" problem). And beyond troubleshooting production systems, we've always found that HTTP sniffers are useful during development, too. We just heard about a free tool from Microsoft, named Fiddler, so I thought I'd blog it here along with some others that we use regularly:

• Fiddler: It's a free tool written by Eric Lawrence at Microsoft. It's really technically an HTTP proxy, and thus is a specialized tool for sniffing Web requests.
• Wireshark (formerly Ethereal): This is one we've used for a long time. It's a general purpose packet sniffer, and sees all packets, not just HTTP.
• ServiceCapture: Written by Kevin Langdon, we've got copies of this laying all over the virtual office. The killer feature in this tool is its decoding of service requests. It'll decode AMF/remoting packets, for example, and show you their full data structure. Very, very useful.

At Webapper, we're big fans of monitoring tools. You absolutely MUST be able to see what's going on inside your software in real time. We use MySQL on some of our production servers, and we've been using SQLyog as an administration GUI for many years. It's a great way to run MySQL databases--it's one of the fastest, most bug-free pieces of software I've ever seen--but we've always lamented the lack of tools for real-time monitoring, which are common with other databases like SQL Server. Enter, MONyog.

The same company that created SQLyog has released a tool called MONyog for monitoring and troubleshooting MySQL query activity. Here's a link to the screenshots:

MONyog for MySQL Monitoring

We'll be checking that out soon.

I wrote a post not too long ago about using lighttpd (lighty) to ease your server load. My setup consisted of one high-end server box (2x Dual-core Xeon's, RAID 5, 6GB RAM) running Windows Server 2003 x64 and Virtual Server 2005. Virtual Server had three VM's running... my CF/IIS server, my database server, and my lighttpd server... All running on Windows 2003 Server Standard.

Now, the lighttpd box always had a cpu load of 30-50%... Not a big deal, considering it was always pushing 6-10mbps of files at any given moment, day or night. I eventually hit up the friendly folks in #lightpd on FreeNode, asking if I could somehow tweak my config file even more, to get performance as such that I could run a single lighttpd thread to handle all the traffic. The general response was "Don't use Windows."

Now, I know you're all thinking that the response was typical for OSS people, but I decided to give it a try. I whipped up another VM, installed SUSE 10.2 on it, and set up lighttpd. I redirected my traffic from the Windows VM to the Linux one (which was easy and instantaneous thanks to my firewall, IPCop, which will be covered in a future post), and low and behold, a single lighttpd thread was handling all the traffic.

Not only was a single thread handling it all, but CPU usage on the linux box was peaking at 2%, with little disk usage and plenty of free memory. Also, as I watched the bandwidth usage over the course of the day, I noticed that the bursts were able to actually burst higher (i.e. push more traffic and/or handle more simultaneous requests) than they were on Windows. This enabled me to take advantage of low-cost rented hardware to do some Round-Robin DNS load balancing.

So, in summary, if you're going to use lighttpd, use it on Linux.

So it was brought to my attention during my CFUNITED presentation on Tuesday morning that a few slides of high interest were not viewable on-screen or in the conference book. Specifically, there's a great Viso diagram of the template request workflow as it applies to ColdFusion and the settings for Trusted Cached and Saving Class files. I've placed a copy of my CFUNITED 2007 presentation online so that individuals who are interested in viewing those more detailed slides can download the information directly.

If you have any questions at all about the material or the presentation, you're more than welcome to contact me either via email or directly here at the conference over by the Webapper / SeeFusion booth in the sponsor's area.

To those of you who attended my presentation on Tuesday morning and shared all the positive feedback with me, I appreciate your opinions and your attendance. If you missed the session, I'll be re-presenting again on Saturday afternoon at 2:45pm in Ballroom GH.

To download the PowerPoint presentation directly, just click the "Download" at the end of this blog entry.

I had a website with an issue. This site is successful on several levels... It gets a good number of visits (500,000/month), each visitor hits nearly 10 pages per visit (so around 5,000,000 page views/month), and ranks well with search engines. While this might sound like a slightly above-average site, the problem lies with it's key feature... It is almost exclusively devoted to letting any visitor hotlink images and other files from this site onto their personal sites (blogs, social sites, etc). This means that on top of a good amount of what most people call legitimate traffic, there are also thousands upon thousands of pages all around the web that make requests to our server.

These files and images are each mostly between 8-20k in size. So, let's separate the main issues:

Server Load
IIS doesn't offer too much in the way of performance tuning, so I switched to Apache. Apache offered more of what I wanted, but similarly crashed and burned on a daily basis. I did try, out of desperation, hosting these files on a couple different shared hosting plans, but with both services I tried, I brought their servers down mere seconds after switching DNS over to them.

Page Load Time
Part of how the site works is offering visitors gallery-style pages to preview images they want to use on their sites. The average page has on it around 60 images, not including the look and feel images (logo, navigation, etc). IE and Firefox, by default, limit the number of simultaneous connections they will make to a server (I believe that with IE it's 4, and with Firefox it's 8), which was making the relatively small images appear to load quite slowly.

Fixing the page load time was actually the easiest task. I needed a way to trick the browser into making more connections. There were two key steps involved:

1. Setting up DNS. I set up four sub-domains to all point to the same unique IP address (server1.mysite.com, server2, server3, server4). I then had to adjust my website code to not just display a relative path to the image in question, I had to randomly display a fully qualified path to one of the four sub-domains I had set up. 60 images divided among 4 servers, with 4 simultaneous connections at the browser meant that the browser only has to make 3-4 queued sets of requests. Yes, this means that I would potentially be circumventing the browser's built-in cache system, but most users have the cache set to default, which means the HTML will get cached, which in turn means the same images are called and retrieved from cache. After this, I would get around 16-100 requests/second just for images, bursting even higher, occasionally.

2. Now the server is well loaded. I was restarting the server nightly to combat the crashes, and even wrote a monitoring script that would automatically restart apache when it would go down 1-2 times each day. What I needed was an extremely lightweight and high-performance image server. Doing some research, I found my answer from the Ruby on Rails crowd.... Lighttpd (aka Lighty). It runs on multiple platforms, is well-documented, and is quite easy to set up. In fact, my config file is quite short:

I won't go over the entire configuration, but the key settings are server.max-worker (number of threads to spawn), server.max-fds (number of file descriptors), and server.stat-cache-engine (the "fam" setting enables the File Alteration Monitor, a higher performance way of checking to see if a file has been modified since last read). I was able to set Lighty to the same root directory as IIS, and then excluded the ColdFusion and other non-asset files using the static-file.exclude-extensions setting. Of course, everyone will need to tune their settings to their site, but my settings are based on the performance section of the Lighty documentation.

Since switching to this IIS (for ColdFusion) and Lighttpd (for images and other assets) setup, I haven't had any crash-related downtime. Page views are up year-over-year, and complaints are now few and far between. In fact, all these services are running on the same physical hardware, thanks to the magic of virtualization. But that's for another blog post.