Performance Tuning Apache Tomcat Steve Heckler, President Accelebrate

Performance Tuning Apache Tomcat Steve Heckler, President Accelebrate http://www.accelebrate.com [email protected]

What We’ll Cover Removing unneeded applications Tuning and monitoring the JVM – This section “borrows” some from Sun’s official documentation Tuning and monitoring connectors Compiling native connectors Tuning and monitoring database connection pools Turning off JSP development mode Reducing logging Precompiling JSPs and caching output 2

Tomcat Version in Use We’ll use Tomcat 6, but the majority of strategies shown are applicable to earlier versions 3

Tomcat Versions (see http://tomcat.apache.org for details) General Version Latest Version JDK Servlet API JSP API 3.3.x 3.3.2 1.2 2.2 1.1 4.1.x 4.1.36 1.2 2.3 1.2 5.0.x 5.0.28 1.4 (1.3) 2.4 2.0 5.5.x 5.5.25 1.5 (1.4) 2.4 2.0 6.0.x 6.0.14 1.5 2.5 2.1 4

Word of Advice #1: Use a Recent Java SE version Java 1.5 sizes heap generations much more efficiently than 1.4 and earlier Java 1.6 has an option to perform garbage collections in parallel (more about this later) 5

Remove Unneeded Applications At your discretion, you can remove every installed application that ships “out of the box” in Tomcat This will save you startup time, as well as memory used by preloaded servlets ROOT should definitely be replaced Only keep manager (and in 5.5 and later, hostmanager) if you need them 6

Tomcat Monitoring Tomcat is difficult to monitor prior to Java 1.5 and later builds of Tomcat 4.1.x Java 1.5 and later support monitoring the JVM using JConsole and the jstat command line tool (both included with the JDK) Tomcat 5.0 and later have especially good JMX MBeans support 7

What are JMX and MBeans JMX: Java Management Extensions, a standard way of managing Java applications MBeans: Management beans. An application can provide management beans that enable you to interact with and configure the applications. MBeans have: – Attributes that you can get or set – Operations that you can invoke MBeans can be interacted with programmatically or via JConsole 8

Enabling JMX monitoring (via setenv.sh) #!/bin/sh if [ " 1" "start" ] ; then CATALINA OPTS " CATALINA OPTS \ -Dcom.sun.management.jmxremote" # -Dcom.sun.management.jmxremote.port 9086 \ # -Dcom.sun.management.jmxremote.ssl false \ # -Dcom.sun.management.jmxremote.authenticate false" echo 0: CATALINA OPTS " CATALINA OPTS" fi 9

Enabling JMX Monitoring (Windows) In a production environment, you can set these in CATALINA OPTS by editing the service.bat before registering the service Alternatively, edit the registry keys directly after the service is registered 10

JMX Monitoring: Key Decisions Only specify a port if you want to allow remote access (potentially a security hole) Consider configuring authentication and/or SSL if you do open a port – http://java.sun.com/j2se/1.5.0/docs/guide/ management/agent.html has details on this 11

Once JMX is enabled You can run jconsole from the command line of any computer with JDK 1.5 or later installed Accessing your JVM remotely will require that a port be opened jstat can be run locally to monitor your JVM (dumps output to STDOUT) 12

Introduction to Java Garbage Collection Java memory utilization and garbage collection are two of the most critical issues in Tomcat performance tuning. Garbage collection is the process whereby memory is reclaimed from the application Java tends to shield developers from control of garbage collection, but not from the consequences 13

Introduction to Java Garbage Collection In Java, objects become eligible for garbage collection when they no longer have any references pointing to them The story of how this occurs is rather complicated 14

GC Scalability (graphic shown at http://java.sun.com/javase/technologies/hotspot/gc/gc tuning 6.html) 15

GC Scalability Demonstrates that an application that spends 10% of its time in garbage collection can lose 75% of its throughput when scaled out to 32 processors 16

Heap versus Non-Heap Memory JVM manages 2 kinds of memory: heap and non-heap: – Heap memory is the runtime data area from which the JVM allocates memory for all class instances and arrays. – The heap may be of a fixed or variable size. – The garbage collector is an automatic memory management system that reclaims heap memory for objects. 17

Heap versus Non-Heap Memory – Non-heap memory includes a method area shared among all threads and memory required for the internal processing or optimization for the JVM. – It stores per-class structures such as a runtime constant pool, field and method data, and the code for methods and constructors. 18

Heap versus Non-Heap Memory – A JVM implementation may require memory for internal processing or optimization which also belongs to non-heap memory. – For example, the JIT compiler requires memory for storing the native machine code translated from the JVM code for high performance. 19

Garbage Collection Garbage collection (GC) is how the JVM frees memory occupied by objects that are no longer referenced. It is common to think of objects that have active references as being "alive" and un-referenced (or unreachable) objects as "dead.“ Garbage collection is the process of releasing memory used by the dead objects. The algorithms and parameters used by GC can have dramatic effects on performance 20

Generational Garbage Collection In practice, most programs create: – many objects that have short lives (for example, iterators and local variables). – some objects that have very long lifetimes (for example, high level persistent objects) 21

Generational Garbage Collection GC divides memory into several generations, and assigns each a memory pool. When a generation uses up its allotted memory, the VM performs a partial garbage collection (also called a minor collection) on that memory pool to reclaim memory used by dead objects. This partial GC is usually much faster than a full GC. 22

Generational Garbage Collection Generations: – young generation (“the nursery”) eden space – Most objects initially assigned here (and die here) two survivor spaces – Objects that survive a minor GC in eden space are moved here – old generation tenured space – Objects that survive long enough in the survivor spaces – When tenured space fills up, full GC occurs (often slow and involves all live objects) – permanent generation holds all the reflective data of the virtual machine itself, such as class and method objects 23

Generational Garbage Collection If the garbage collector has become a bottleneck, you may be able to improve performance by customizing the generation sizes. http://java.sun.com/docs/hotspot/gc/index.html details how to customize these sizes. – Java 6 version at http://java.sun.com/javase/technologies/hotspot/gc/gc tuning 6.ht ml Customizing the sizes is less necessary in Java 1.5 and later 24

Tuning the Total Heap Total available memory is the most important knob affecting GC performance By default, the JVM grows or shrinks the heap at each collection to try to keep the proportion of free space to living objects at each collection within a specific range This target range is set as a percentage by the parameters XX:MinHeapFreeRatio minimum and XX:MaxHeapFreeRatio maximum , and the total size is bounded below by -Xms and above by -Xmx 25

Tuning the Total Heap Some notes: – Unless you have problems with pauses, try granting as much memory as possible to the JVM. The default size (64MB on a 32-bit OS) is often too small. – Setting -Xms and -Xmx to the same value increases predictability by removing the most important sizing decision from the JVM. – Be sure to increase heap size as you add cores or processors – 32-bit OSes cap the heap size at between 1.5 and 2.5GB 26

Tuning and Monitoring Connectors Tomcat supports connectors for http, https, and ajp Suggestions: – Set enableLookups to false – Make sure maxThreads and acceptCount are set sufficiently high (but not so high that you’re accepting more traffic than your Tomcat instance can handle) – compression trades off bandwidth and processing time 27

Alternatives to the Classic Java Blocking Connector NIO – New input/output – Supported starting in Tomcat 6 – Provides access to low-level I/O operations of modern operating systems, including multiplexed, non-blocking I/O and polling APR connector (Apache Portable Runtime) – Tomcat 5.5.15 and later (reliably) – Uses OpenSSL – Native code and thus faster Bottom of http://tomcat.apache.org/tomcat-6.0-doc/config/http.html has a good comparison 28

Building the Native Connectors Dependencies – OpenSSL (you need the source) - 0.9.8a or later – APR (on Red Hat and variants, apr-devel RPM is sufficient) - 2.2 or later Steps: – Locate or download APR – Download OpenSSL (you do need the source) – Unpack the native connector 29

Building the Native Connectors Steps (continued) – ./configure --with-ssl [path to extracted SSL] -with-apr [path to apr-1-config, possibly /usr/bin/apr-1-config] – make – make install – Update setenv.sh to use the built library CATALINA OPTS " CATALINA OPTS Djava.library.path /usr/local/apr/lib" 30

Building the Native Connectors Steps (continued) – configure the APR connector(s) in server.xml – Update protocol attributes: HTTP: org.apache.coyote.http11.Http11AprProtocol AJP: org.apache.coyote.ajp.AjpAprProtocol – Optionally, configure SSL and set up the APR https connector 31

Building the Native Connectors Sample APR SSL connector Connector protocol "org.apache.coyote.http11.Http11AprProtocol" port "8443" minSpareThreads "5" maxSpareThreads "75" enableLookups "true" disableUploadTimeout "true" acceptCount "100" maxThreads "200" scheme "https" secure "true" SSLEnabled "true" SSLCertificateFile "/usr/local/ssl/server.crt" SSLCertificateKeyFile "/usr/local/ssl/server.pem" clientAuth "false" sslProtocol "TLS"/ 32

Monitoring Database Connection Pools Can be done via /Catalina/DataSource in the MBeans tree within JConsole Parameters for your pool are shown at http://commons.apache.org/dbcp/configuration.html Be sure to make your pool large enough for the traffic you anticipate Be sure to time out requests for connections Consider removing abandoned connections 33

Other Suggestions Turn off JSP development mode in production by setting the development parameter of the jsp servlet to false – Keeps JSPs from being checked for modification Scale back access and error logging to just what’s needed Consider precompiling your JSPs – see http://tomcat.apache.org/tomcat-6.0-doc/jasperhowto.html#Web%20Application%20Compilation 34

In Closing Thank you for joining me! Please email me at [email protected] with questions Good luck with tuning Tomcat! 35