Move over design patterns, here come crash patterns! I'll be talking about this concept in my "Server Down" CFUnited 2008 presentation in June. As many of you know, we here at Webapper have gotten loads of "server down" calls and emails over the years, so the purpose of my talk is to organize all of what we've seen into best practices for preventing performance and stability problems, and also for what to do when problems do arise. And one way we've started to organize this information is with the concept of "crash patterns". It's modeled after the familiar idea of design patterns, but applies instead to common/recurrent causes of performance and/or stability problems (the two usually go together). And, as with design patterns, once you come to understand crash patterns (both in general, and those that are specific to your applications), you can use them to make your life a whole lot easier. And when it comes to performance and stability, that means using the knowledge of crash patterns to avoid these problems in the first place, and it also means having optimal steps to take when/if problems do arise on your production systems.

I hope to see tens of thousands of you packed into the room in D.C! In the meantime, if you have any thoughts/comments/questions, please send them along, as I'd love to incorporate even more community/attendee input into my presentation.

I was reading with some amusement Doug Boude's retort to a Java purist that was dismissing ColdFusion for being too... simple? to be considered a "real" language, and I was reminded of a topic I commonly rant on. It's what I call the Three Phases of Programmer Development.

Phase One

A novice programmer is happy to get anything to work. Eventually, the novice programmer becomes a journeyman programmer, and is able to not just get things to work, but to start to create some truly complicated programs.

Phase Two

Here, a journeyman programmer starts to develop their chops. They can take a complicated problem and build applications with a level of complexity equal to (or greater than) the problem at hand.

Phase Three

Programmers usually find Phase Three when they've been asked to go back and maintain some Phase Two code they wrote six months or two years ago... and all they can think was "WTF was I doing /here/?" "What's /this/, now?" "Geez, did /I/ write this? Wow."

The epiphany that leads a programmer to Phase Three is the realization that the hallmark of the professional programmer is not in the complexity of the code they produce, but rather in the simplicity. When you can take a complex problem, and solve it with simple code, you are a professional programmer. A lot of people that are die-hard ColdFusion fans are Phase Three programmers, who use it because it allows them to produce simpler, more maintainable code. And a lot of ColdFusion's detractors are Phase Two programmers, who think that anything /that/ simple must be for novices/children.

Note that frameworks of all types can often be categorized into Phase Two or Phase Three frameworks. The question to ask is, "in the long run, does this framework (or design pattern) make this application simpler, or more complex?" Or, "Will this framework make troubleshooting errors or performance problems nightmarish?" Or, "do I have to use this whole bloated framework for this problem, or can I just use a few of tastier bits to address this issue at hand?" If the use of a framework or design pattern makes the solution simpler, or at least more maintainable, great. If not, it may be good to evaluate whether you're using it for Phase Two or for Phase Three reasons. I suppose it bears mentioning that "resume-building" is never a valid reason for the use of a pattern or framework. That being said, having at least a passing familiarity with many patterns and frameworks is good, so that when you start designing something that does fit an existing solution well, you know to use it.

[n.b. I often recommend that people new to ColdFusion or Java (or any other language) read the description of *every* function or JDK library method. Not for the sake of memorization, but so when you run into a problem that's already solved, the voice in back of your head can say, "Hey, I think I saw something that does that..."]

Code complexity is like an infestation; it tends to spread to all areas that touch it. When you find code for a problem growing overly complex, STOP. Take a moment to understand where the complexity is coming from, and refactor it to something more simple; don't let complexity sprawl and infest the rest of the system. I tend to dislike ORM implementations, because they tend to reduce object cohesion and often require you to write more code in support of the ORM than writing simple loadFromDB(id)/loadFromResultSet() methods would require, and I've found performance (for other than basic CRUD applications) to be anywhere from unpredictable to abysmal. I know this makes me something of a heretic, and there are bound to be people that insist I was doing something wrong or "just don't GET it", but when I recently ripped Hibernate out of a Java application, the code size shrunk by about 30%, and the performance improved by an order of magnitude. (For the record, I was the one who put it in there in the first place, because I bought into the hype. Now I understand better what it's good for, and what it's less good for.)

It bears repeating: the hallmark of the professional programmer is not the complexity of the code they produce, but rather the simplicity. Whenever you see code getting complex, stop and ask yourself if there's a way to do it simpler/better.

I was updating an app to take advantage of CF8's new EXIF reading abilities, and came across an error. When I tried to call imageGetEXIFMetaData() on a certain image, I got a 500 java.lang.StackOverflowError (you can read the complete error in the attached HTML file).

I took a peek at the stack, and found that Adobe is using Metadata Extractor by Drew Noakes. Reading the change log, I found that the bug I hit was fixed with version 2.3.0.

So, I stopped my CF8 service, and replaced %CFROOT%/lib/metadata-extractor-2.2.2.jar with my newly downloaded metadata-extractor-2.3.1.jar. Started CF back up, and my error went away.

Ah, the beauty of Java!

I recently set up a virtual server with hostmysite.com and we installed our own copy of ColdFusion Enterprise. In the process, I wanted to follow hms's guidelines about putting all log files on the E: drive as opposed to the default C: drive where ColdFusion gets installed. Now I'm not entirely certain why they're so adamant about the segregation (beyond the obvious security considerations of growing log files impeding the smooth operation of the OS), but I am personally concerned about the lack of hard drive space on the VPS. There's only 2 gigs allocated to the C: drive, and after loading it up with ColdFusion (two instances), MSSQL 2005, Subversion (repos on a different drive), and some other tidbits, I'm down to only 750 MB free. Another production server I manage has almost that much taken up in the JRun log files alone!

It's easy enough to use the ColdFusion administrator to move the ColdFusion log files (like application.log and server.log, etc), but I really need to move the JRun-specific log files, namely default-err.log, since that's the one that grows bit-by-bit whenever an error is thrown in CF (including those pesky "java.net.SocketException: Connection reset").

You can't use the CF administrator to modify the location of those log files, but you can use regedit to do so. The key is in

If you have multiple CF instances, you'll have to make the change in each key, where "Macromedia CFMX AS cfusion" above is the Windows service name. For a single-instance, standalone ColdFusion installation, the key would be in

Change the "SystemErr" and "SystemOut" values from

to

and bounce the service. Voila! You don't need to shuttle the log files by hand. When the service starts, new log files will be created in the new location. You can do this for the -err and -out logs.

If you're on hostmysite's VPS plan that includes CF, you may want to make this change yourself. The default image for the CF VPS plans don't put these log files on the E: drive, which means you could run out of room on the C: drive in a relatively short amount of time, depending on traffic and your application's profile.

Concepts

People often think of SSL as a form of encryption. Strictly speaking, it's not; SSL is a protocol or "cryptosystem" that uses existing encryption algorithms in a way that ensures privacy. To that end, SSL is not just about encryption; it's also about authentication, ensuring that you're connecting to the party you think you're connecting to. This is done using a chain of trust; effectively, you trust "certificate authorities" (CAs) to use due diligence in ensuring that SSL public keys belong to the party you're connecting to.

The combination of a public key, the trust information about that key, and the digital signature of the CA, is known as a "SSL Certificate", and that certificate is sent by an SSL server to a client at the beginning of every conversation.

Embedded in every SSL certificate is the "common name" of the resource that the certificate is for, which, in SSL, is the name of the server as you type it into a browser. For example, one of the "common names" for the SSL certificate at paypal.com is "www.paypal.com". If you try to make an SSL connection to paypal.com based on IP address, your browser will pop up a warning that the name you typed doesn't match the name in the SSL certificate.

It can be argued that the "real" chain of trust extends back to the browser, however, since the list of trusted CAs (and their public keys) are distributed with the browser.

For Java, this list of trusted CAs is (usually) stored in a file called "cacerts". Unlike with a browser, however, if either (a) the common name doesn't match the name in the URL, or (b) the CA that signed the SSL certificate isn't in the list of trusted CAs, then the SSL communication fails with an exception similar to this:

There is no "workaround" for this to disable the authentication step; the CA for the SSL cert must be trusted, and the common name must match, or the SSL connection will fail.

Common Name Mismatch

Fixing a common name mismatch is simple; just ensure that the host name in the URL you're using for communication matches one of the common names in the SSL certificate. If you double-click on the "key" in a browser after connecting to any page on the SSL server, you can view the list of common names that the certificate applies to.

Unknown Certificate Authority

The default certificate store that's shipped with the JVM is called "cacerts", and is typically stored in the JRE's ./lib/security folder, unless you're using the "-Djavax.net.ssl.trustStore=" JVM parameter to explicitly load a different list of CA certificates.

If a trusted CA is not found in the SSL certificate, you get an exception similar to this:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found

This can have one of two root causes:

File not found

If the trusted CA certificate list isn't in the default location, nor at the location specified using the "-Djavax.net.ssl.trustStore=" JVM parameter, a "file not found error" is not produced; the symptom is that all SSL communication fails. For this reason, if using JVM args to specify an alternate trusted CA list file, only absolute paths should be used, as the use of relative paths is rather unreliable.

The CA isn't in the trusted CA list

If the party you're connecting to has recently changed server hardware or updated their SSL certificate, it's possible that they've started using a CA that isn't in your trusted list. Before continuing, verify that one of these things has occurred, and that you're not seeing the effects of some sort of man-in-the-middle attack, where someone is trying to decrypt and reencrypt communications using a fake SSL certificate. (The odds of actually experiencing such an attack are vanishingly small, but cannot be discounted altogether.)

If the CA isn't in the trusted CA list, but should be, you can import it into the list. Step one is saving the CA's public key into a file; step two is importing that file into our trusted CA list.

You SHOULD get the CA's new public key directly from the CA's website; however, if this is not an option (and you're really sure you're not seeing the side effect of a man-in-the-middle attack) you can use Internet Explorer to extract the CA public key into a file that can be imported into Java's cacerts file.

Saving the CA public key into a file:

(These directions are from http://www.webmilhouse.com/7b/?p=85 )

Steps in IE 6.0 on Windows 2000 to get the cert:

1. Connect to https://my.domain.com
2. Go to Tools > Internet Options > Content > Certificates > Intermediate Certification Authorities [or "Trusted Root Certification Authorities"]
3. Choose "Thawte SGC CA" [or whichever is needed]
4. Click "Export...", then "Next>"
5. Select "DER encoded binary X.509 (.CER)"
6. Name the file thawtesgcca.cer [change the name as applicable]
7. Select "Finish"

Importing the CA public key into the trust list (file and alias names will likely vary; the alias name isn't important):

Note that you must have a full JDK installed (not just the JRE) to have access to keytool and javac (used below).

Testing your New Cacerts

Here is a simple Java class that can be used to test SSL connections:

import java.io.BufferedReader;
 import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.net.URL;
 import java.net.URLConnection;
 
 /**
  * @author Daryl Banttari
  *
  */
 public class TestSSL {
 
    public static void main(String[] args) {
       // default url: 
       String urlString = "https://www.paypal.com/";
 
       // if any url specified, use that instead: 
       if(args.length > 0) {
          urlString = args[0];
       }
       System.out.println("Connecting to " + urlString + "...");
       
       try {
          // convert user string to URL object 
          URL url = new URL(urlString);
 
          // connect! 
          URLConnection cnx = url.openConnection();
          cnx.connect();
 
          // read the page returned 
          InputStream ins = cnx.getInputStream();
          BufferedReader in = new BufferedReader(new InputStreamReader(ins));
          String curline;
          while( (curline = in.readLine()) != null ) {
             System.out.println(curline);
          }
 
          // close the connection 
          ins.close();
       }
       catch(Throwable t) {
          t.printStackTrace();
       }
 
    }
 }

Save this code into a file named TestSSL.java (case sensitive!) and compile:

Now run it to attempt to load the target URL:

Either the contents of the page will be displayed, or an exception will be shown indicating why the page could not be retrieved.

SSL: Not Just About Encryption!

Remember, it's not enough to encrypt communication, you also have to be sure you're communicating with the right person. SSL, and all good public key cryptosystems, ensure not only encryption but also authentication. Since you can't manually authenticate every SSL certificate to every company that runs an SSL server, you trust other certificate authorities to ensure that the server you're connecting to is the server you think you're connecting to. If Java doesn't trust the certificate authority used by the SSL server, then Java will refuse to talk to that server.

Browsers will pop up a warning dialog indicating that there's a problem if the CA isn't trusted, but Java can't assume anyone is watching for that, so it throws an exception and fails. Updating the cacerts file used by Java, so that it includes the new certificate authority's public key, will solve that issue.

Hey all. Just wanted to let everyone know that we're nominated for SysCon's Readers' Choice awards (CFDJ and WebDDJ) for our consulting services and for our SeeFusion tool. Our consulting services are in the Best Consulting Service category (duh), and SeeFusion is in the Best Web Development Tool and Most Innovative Application categories (you can vote in whichever category you think is most fitting). If you've benefited from either SeeFusion, and/or from our tuning/troubleshooting consulting, we'd be honored to have your vote!

Or, if you know someone who has, or if you know someone who knows someone, or...sorry, I'll stop now. I'm from Chicago, where the theme for every election is "Vote Early, Vote Often." ;)

Seriously, we'd love your vote. Please help get the word out if you can. Here's the link to vote, so it's handy:

SysCon's Readers' Choice Awards.

Best,
Patrick

Or, "Installing Flex Data Services 2 Express as a Windows Service"

This guide is for those who know nothing of Tomcat, yet are sick of having a command prompt in the background when you are trying to work with FDS. I'll assume that you're running Windows, and that you're doing this on your local machine, so you don't have to worry about firewall settings.

Step 1: Download and install Tomcat
Go to http://tomcat.apache.org/download-55.cgi and download the latest Binary Distribution for Windows.
Run the installer, and choose all the default options (unless you really want to change them, and understand the implications)

Step 2: Download and install FDS2
Go to http://www.adobe.com/cfusion/tdrc/index.cfm?product=flex and download Flex Data Services 2 Express for Windows.
Run the installer, except when you get to the "JRUN or J2EE" option, pick J2EE

Step 3: Download and install JOTM
Go to http://forge.objectweb.org/projects/jotm/ and download the latest version of JOTM (I downloaded jotm-2.0.10.tgz).
Decompress that file, and there'll be a new directory called "jotm-2.0.10"
Inside that directory, there's a lib directory.
Copy everything that's inside jotm-2.0.10\lib to C:\Program Files\Apache Software Foundation\Tomcat 5.5\common\lib

Step 4: Copy FDS files to Tomcat
Copy flex.war, flex-admin.war, and samples.war from C:\fds2\ to C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\

Step 5: Restart the Apache Tomcat service
I hope you remember where the services console is :)

Step 6: Test it out
If you did the default options, you should be able to go to http://localhost:8080/samples/ and run the sample applications.

We made an interesting discovery in a recent consulting engagement, and we wanted to share what we found with the community. The gist of what we found is that cfswitch, with a string expression, and especially under load, runs dramatically slower than the equivalent cfif-cfelseif-cfelse block. Here''s what happened...

A customer purchased one of our one day remote consulting engagements. They reported an intermittent performance problem that they were unable to pin down. They were already a SeeFusion customer, so we used SeeFusion to generate stack traces on the running requests. This line showed up quite often in the traces we took:

  "jrpp-27" prio=5 tid=0x09572cb0 nid=0x12d4 runnable [5da2f000..5da2fdb8]
    at java.lang.FloatingDecimal.readJavaFormatString(Unknown Source)
    at java.lang.Double.parseDouble(Unknown Source)  

Elsewhere in the stack traces we saw evidence of cfswitch usage. So we did some further digging, and sure enough, the customer was using cfswitch tags inside query loops--not an uncommon practice at all. But the switch expressions were all strings. So here''s the kicker. Under the covers, ColdFusion attempts to convert the switch expression to a floating point number, using Java''s parseDouble method. If the conversion works, then ColdFusion uses the expression as a number. If the conversion fails, however, then an exception is thrown, and an exception stack is generated, but then ColdFusion handles the exception and uses the expression as a string. The problem is, that exception throwing and stack generation gets very, very expensive under load.

We cooked up some tests to verify, and sure enough, on average the performance difference between cfswitch with string expressions, and the equivalent cfif-cfelseif-cfelse blocks, was a 10-fold difference. Under load, that will multiply dramatically.

We had the customer convert their cfswitch blocks, and they had an immediate improvement in performance and stability. There were other things left to tune, but this was a primary culprit.

We also found evidence that the floating point conversion might be single-threaded, which would make matters even worse. But the evidence for this wasn''t clear, and there also appeared to be some variation across different VM versions, but we had to move on to other things and couldn''t pursue this other point.

Moral of the story--only use numeric expressions for cfswitch! Otherwise, use equivalent cfif-cfelseif-cfelse blocks. And this is especially true if your code will be running under heavy load.

Test files

Abstract

Use of the ConcMarkSweep garbage collector for the "tenured" generation is undesirable when RMI (Remote Method Invocation) garbage collection is occurring.

Introduction

Java doesn''t require that you explicitly release memory allocations in your program, which greatly reduces the time to build useful applications and eliminates most memory leaks. However, a significant part of any application is spent doing memory management, and in the case of Java, all application threads must be suspended for some period of time during any "garbage collection", or "GC". Garbage collection is the operation performed automatically by the JVM that recycles memory that had been allocated for an object, but now is not referenced by any part of the application, and therefore can be reclaimed and reused. The application pauses caused by GC are known as "stop-the-world pauses", because all application threads must be suspended during those pauses.

The challenge with user-facing applications (like Web application servers) is to make those pauses as brief as possible, with the goal that no user will notice them.

Since most memory allocations are released almost immediately after they''re allocated, the Sun JVM uses a "generational" memory allocation and collection scheme to improve efficiency by focusing collection efforts on recently allocated memory. The main memory allocation heap is split into two parts: the "young" (aka "new") generation, and the "old" (aka "tenured") generation. The young generation is collected every time it fills up; objects that survive a few collections in the young gen are promoted to the old gen. The old generation is collected more infrequently.

There are three main types of collections:

1. Young generation collections (using the currently selected young gen collector)
2. Old generation collections (using the currently selected old gen collector)
3. Full collections, which stop the JVM and clean and compact all memory.

Full collections are expensive and tend to produce long pauses, so it''d be nice to be able to avoid them.

You can change the selection of garbage collector by editing the "jvm.args" line of your "jvm.config" file. (Note that "-server" must always be the first argument on that line.)

Young Generation Collection

The default collector for the young gen is either the "Default New" or "Parallel New" collector, depending on the platform and number of processors in the machine. ColdFusion explicitly selects the old "Parallel" collector using the JVM argument "-XX:+UseParallelGC", as opposed to the "new" parallel collector selected using "-XX:+UseParNewGC". Unfortunately, the "old" parallel collector is incompatible with both JVM debugging and the "concurrent" tenured generation collector.

A larger young generation means that collections will occur less often, but take more time. Also, it''s fairly important to ensure that the young generation isn''t too large to fit in the available memory in the tenured generation, or full collections may be forced. This is a requirement called the "young generation guarantee".

Young generation collections, even with a large young generation size selected, tend to be very brief, so long as the young generation guarantee is met. If the young generation guarantee isn''t met, then a Full collection is forced.

Tenured Generation Collection

The default collector for the tenured gen uses a "stop-the-world" pause for the duration of its collection. If you have many objects to be collected in the tenured generation, this can cause excessive pauses.

An alternative, the Concurrent Mark Sweep (CMS) collector, uses a few brief stop-the-world pauses during collection, but the bulk of collection occurs while application threads are running. While less efficient than the default tenured gen collector, the reduced pauses are often desirable for quasi-realtime application like Web applications. On the face, it appears that the CMS collector is ideal for Web applications.

The CMS collector is enabled using the JVM arg "-XX:+UseConcMarkSweepGC". But read on before rushing out to enable it.

Enter RMI

When you''re using RMI, you may be holding references to memory on non-local machines, so RMI will schedule Full collections at some interval. By default for Java 1.5 and below, that interval is every minute, which can cause significant overhead. However, there is no way to eliminate RMI scheduled collections; even "-XX:+DisableExplicitGC" will not prevent these collections. You can change the interval of collections, but setting too long of an interval may create very long Full GC pauses when it finally does run. (Pauses in excess of 60 seconds are not rare with a one hour interval, if the old generation is large and close to full.)

RMI collections and ConcMarkSweep

Normally, the Young generation space is split into a few areas: Eden, where new allocations are made, and two Survivor spaces, called "from space" and "to space". Measurements of these spaces are visible in the stdout log when you have "-XX:+PrintHeapAtGC" set.

When the Young gen is collected, the objects in Eden that are still allocated are copied to a Survivor space. Those objects will be bounced back and forth between the two Survivor spaces for a few collections, and eventually promoted to the tenured gen if they live long enough. Most allocations should not live long enough to make it to tenured.

It appears that (at least on Windows) enabling ConcMarkSweepGC actually disables the Survivor spaces, and so when Full GCs are finally called (which are unavoidably called by RMI), the Full GCs wind up throwing out all of the objects that were promoted early to Tenured, which adds significant time to those stop-the-world collections.

So, the goal is to give the Full collector as little work to do as possible, and since ConcMarkSweep constantly bleeds memory into Tenured, the Full GC pauses are end-user noticeable, often taking more than four seconds from every minute.

What seems to work really well is this set of options (split into separate lines for clarity):

-Dsun.rmi.dgc.client.gcInterval=600000
-Dsun.rmi.dgc.server.gcInterval=600000
-XX:SurvivorRatio=4
-XX:NewSize=64m
-XX:+UseParNewGC
[no ConcMarkSweepGC]

The RMI collection intervals control how often RMI triggers a Full GC. The default is one minute (60,000ms) which is too frequent, but the most commonly suggested replacement is one hour (3,600,000ms), which can cause the "[Unloading class....]" phase of a Full GC (collection of the "perm" gen, where classes are stored) to take a noticeably long time.

The survivor ratio sets the Survivor spaces a bit larger than normal (1/4 the size of Eden), so that there''s no overflow into Tenured. The NewSize parameter actually sets the Eden size, so the adjusted size of the whole Young gen (Eden + two survivor spaces) is 64mb + the two survivor spaces (16mb each) = 96mb.

We''re finding that the Full GCs are taking about 1.2-1.5s when called every 10 minutes by RMI, and the Tenured collector never needs to run, because the size of the tenured gen (after startup) is almost totally static, since practically all new objects die somewhere in the Young gen.

Young gen collections are occurring about every 10-20sec, and usually take about 0.15s, with an occasional 0.6s.

The only problem we''re having now, is that it appears that sometimes the RMI collections don''t start until after the first tenured generation collection, which in our case means 19+ hours of slow memory creep after startup before the first major collection, which takes about a minute (stop-the-world) to complete. If we notice that occurring, we use the SeeFusion interface to force a Full GC once, shortly after startup.

I have been observing the great work that Doug Hughes has been doing on Reactor. "Reactor is an ORM framework (Object Relational Mapper), meaning it transcodes your relational database into objects for you to use in your CF programming". Because of pressure of work I had not been able to be very useful to the project at all.

Recently, Doug had expressed concerns about what appeared to be a memory leak. As we have been working on such issues for Allaire and then Macromedia clients for many years, I volunteered to help.

This is the first blog of what we did and achieved so far and this was already circulated to the Reactor email list.

Here is what I have observed since we made the tweaks to the jvm.config file, shown below, on Doug''s server. Doug restarted a little while ago but I don''t think it was due to low memory in the instance. Here is the memory situation just before the restart:

Total Memory=512768 Free=215700

At this point the instance had been up and running from 05/25 21:47:47 to 05/28 14:16:57, around 2 days 17 hours and the free memory never got dangerously low during that time, in the logs I saw. Disabling those Explicit Full GC''s had a beneficial impact. I feel there are still improvements to be made but I know Doug is also looking at code-level issue at present so will hold off further recommendations at present.

In essence, we did two different things, the first group were related to monitoring or logging. We enabled Verbose Garbage Collection logging using these commands in the jvm.config file:

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:reactorgc.log (the last bit -Xloggc:reactorgc.log gives the log a name). This information goes into the jrun4 or runtime bin directory, by default. As a note point, this is not too verbose so the log does not get too big, however it should eventually be disabled. This logging helped us identify that too many Full GC''s were occurring.

We also enabled JRun Metrics Logging which shows us the memory status at a point in time along with thread usage details. We set this to report every 60 seconds.

The most useful piece that I used was SeeFusion which enables us to observe memory and thread behavior in real-time by looking at requests as they literally run. We also wrapped the JDBC driver with SeeFusion which enabled us to monitor database calls, SQL passed and query times.

Moving on the second group of things we did. As a result of looking at this monitor and log data, we observed to many Full GC''s taking place and an difficulty for JRun-CFMX to get up and running "comfortably". As a result, we made these changes to the jvm.config file:

Added these arguments (I have these on separate lines here for clarity but they should be added to the arguments string with a single space between each):

-Xms512m (this sets the memory available to the jvm at start at 512MB which saves the effort of the jvm acquiring it)

-XX:PermSize=128m (this does the same thing as above for the permanent memory space which is what the jvm uses for its own operation).

Note the start settings here must always be either equal to or less that the maximum setting, they cannot be set to higher. Also note, if you only have a single jvm.config file in a single jvm whatever you set in that file will apply to all instances. So if you have 5 JRun/CFMX instances they will each take the start amount of memory when they start up.

-XX:+DisableExplicitGC (this stops all explicitly called Full Garbage Collections) this does not stop all Full GC''s only those called explicitly by something.