Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Troubleshooting

Memory Leaks, Thread Dumps and Deadlocks in Tomcat

Find Tomcat classloader and ThreadLocal memory leaks after redeploys, take and read thread dumps, spot stuck and deadlocked threads, and capture heap dumps.

Two kinds of production problems are hard to see in logs: a server that slowly eats memory until it falls over, and a server that is up but stops answering. Both have well-known diagnostic tools. For leaks you use a heap dump and the garbage collector's behavior. For hangs you use thread dumps, snapshots of what every thread is doing right now. This lesson explains both, including the leak patterns specific to web applications on Tomcat, with runnable Java stand-ins that create the same situations in small.

The commands for a real Tomcat are read-only here, since Tomcat cannot run on this site.

How leaks happen in a web container

Java collects objects nothing references. A leak is an object you are finished with but something still points to. In a long-running server, small leaks add up. The most common sources in Tomcat applications:

  • A static collection that only grows: a static Map used as a cache with no eviction, or a list of listeners that are never removed.
  • ThreadLocal values on pooled threads. Tomcat's worker threads live as long as the server, so a ThreadLocal set in a request and never remove()d keeps its value alive. If the value's class came from your web application, it keeps the application's whole classloader alive.
  • Classloader leaks on redeploy: each redeploy creates a new WebappClassLoader. If anything outside the app (a JDBC driver registered with DriverManager, a thread you started, a shutdown hook, a JMX bean) still references a class from the old loader, the old loader and all its classes stay in memory. After a few redeploys you see OutOfMemoryError: Metaspace.
  • Unclosed resources: connections, statements, streams, sessions that are never invalidated.
  • Threads started by the application that do not stop when the context stops.

Tomcat helps: its JreMemoryLeakPreventionListener and ThreadLocalLeakPreventionListener (enabled by default) clean up several known leaks, and at undeploy it logs warnings like The web application [shop] created a ThreadLocal with key of type [...] but failed to remove it when the web application was stopped. Treat those warnings as a to-do list. To avoid leaks, stop your own threads and executors and deregister drivers in a ServletContextListener.contextDestroyed.

Stand-in: a ThreadLocal leak on a pooled thread

This stand-in uses a one-thread pool, as a worker thread would be. A request handler sets a ThreadLocal holding a large object and forgets to remove it. A weak reference shows whether the object was collected after the "request" ends.

Java Stand-in: ThreadLocal leak

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The first object stays alive as long as the worker thread lives, which on a server is until shutdown. (When the second request reuses the thread, set replaces the old value, so in this tiny demo the first value may eventually be freed; the point is the lifetime tied to the thread.) Wrap request-scoped ThreadLocal use in try/finally with remove().

Finding a leak with a heap dump

  1. Confirm the pattern: after each full GC, the heap floor rises (GC logs).
  2. Capture a heap dump from the live JVM: jcmd <pid> GC.heap_dump /tmp/heap.hprof (or jmap -dump:live,format=b,file=heap.hprof <pid>), or rely on -XX:+HeapDumpOnOutOfMemoryError.
  3. Open it in Eclipse MAT, VisualVM or IntelliJ. Run the leak suspects report: it lists the biggest retained objects and the path to GC roots, which shows what is holding them: a static field, a thread, a classloader.
  4. Fix the holder, not the symptom. Raising -Xmx only delays the crash.

For classloader leaks, compare dumps before and after a redeploy and look for several WebappClassLoader instances of the same app. Tomcat's manager has a Find leaks button that lists contexts which were stopped but are still in memory (/manager/text/findleaks, in versions that include it).

Thread dumps: what are the threads doing?

When Tomcat stops answering, the cause is often that all worker threads are stuck on something: a database lock, a remote call without timeout, or a deadlock. A thread dump lists every thread with its state and stack.

Take one with jcmd <pid> Thread.print or jstack <pid>, on Linux also kill -3 <pid> (the dump goes to catalina.out), or from the manager at /manager/text/threaddump. Take three dumps a few seconds apart: a thread at the same line in all three is stuck, not just busy.

Thread states you will see:

State Meaning
RUNNABLE executing, or blocked in native I/O (a socket read shows as RUNNABLE)
BLOCKED waiting to enter a synchronized block held by another thread
WAITING waiting indefinitely (wait(), park()), common for idle pool threads
TIMED_WAITING sleeping or waiting with a timeout

Tomcat worker threads are named http-nio-8080-exec-N. In a healthy idle server they sit in WAITING on the pool's queue. If you see 200 of them BLOCKED or in the same JdbcConnection or SocketInputStream.read frame, that is your bottleneck: look at the top application frame in the stack to find the line. StuckThreadDetectionValve can log such threads automatically after a threshold.

Stand-in: a thread dump and deadlock detection

The JDK can describe its own threads. This stand-in creates two threads that lock two objects in opposite order, a classic deadlock, then uses ThreadMXBean to detect it and print each thread's state and the lock it waits for.

Java Stand-in: detect a deadlock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Both threads are BLOCKED, each waiting for the lock the other holds, and getLockOwnerName() closes the circle. A real thread dump prints the same thing under Found one Java-level deadlock:. The fix is to always acquire locks in the same order (or use timeouts with tryLock). Because the threads are daemons, the program can end even though they never finish.

Stand-in: spotting a stuck thread from repeated samples

Taking several dumps and comparing is exactly what an operator does by hand. This stand-in samples a worker's stack three times and reports whether it is still on the same frame.

Java Stand-in: is the thread stuck?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The busy thread is always RUNNABLE in the lambda of Main, and the stuck one is TIMED_WAITING inside slowRemoteCall in every sample. In a real dump you would see the same method on every sample and then ask "why is a request waiting there, and why is there no timeout?"

A checklist for a hung Tomcat

  1. Is the process alive and the port open? Is the CPU idle (stuck on locks or I/O) or at 100% (a loop or GC thrash)?
  2. Take three thread dumps; check worker thread states.
  3. Check heap and GC (a server in constant full GC looks hung).
  4. Check the database and downstream services for blocking queries or connection-pool exhaustion.
  5. Capture evidence before restarting: dumps and logs vanish with the process.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is an unremoved ThreadLocal a leak risk on Tomcat?

    Pooled threads outlive requests and even redeploys, so values stored in them are not collected.
  2. 2. How many thread dumps should you compare to tell a stuck thread from a busy one?

    A thread at the same stack line across several snapshots is stuck rather than just momentarily busy.
  3. 3. What does a path to GC roots in a heap dump tell you?

    It shows the static field, thread or classloader that holds the object, which is what you must fix.
  4. 4. How do you prevent a deadlock between two locks?

    Consistent lock ordering removes the circular wait; tryLock with a timeout avoids waiting forever.

Summary

  • Leaks come from static collections, unremoved ThreadLocals, unstopped threads and classloaders kept alive after redeploys.
  • Use heap dumps and path-to-GC-roots (Eclipse MAT, VisualVM) to find what holds memory.
  • For hangs, take several thread dumps and read worker thread states and top frames.
  • Deadlocks show as BLOCKED threads waiting on each other; fix lock order and add timeouts.
  • Capture evidence before restarting.

Next: Project: reviewing a production server.xml