Java - Java Performance Benchmarking with JMH (Java Microbenchmark Harness)

Introduction

Performance is an important aspect of software development. As Java applications become larger and more complex, developers often need to determine which parts of their code execute efficiently and which parts require optimization. Measuring performance accurately is not as simple as recording execution time using System.currentTimeMillis() or System.nanoTime(). The Java Virtual Machine (JVM) performs various optimizations such as Just-In-Time (JIT) compilation, garbage collection, and dead code elimination that can significantly influence benchmark results.

To solve this problem, Java provides the Java Microbenchmark Harness (JMH). JMH is a benchmarking framework developed by the OpenJDK team specifically for measuring the performance of small pieces of Java code, known as microbenchmarks. It automatically handles JVM warm-up, multiple iterations, thread management, and statistical analysis to produce reliable and repeatable benchmark results.

What is JMH?

Java Microbenchmark Harness (JMH) is a tool designed to accurately measure the performance of Java methods and algorithms. Instead of relying on manual timing techniques, JMH provides a structured environment that eliminates common benchmarking mistakes.

JMH helps developers answer questions such as:

  • Which algorithm is faster?

  • Does a new implementation improve performance?

  • How much memory does a method allocate?

  • How does performance change with multiple threads?

  • Does compiler optimization affect execution time?

It is widely used by Java developers, library authors, and JVM engineers.

Why Normal Benchmarking is Inaccurate

Many beginners write benchmarks like this:

long start = System.nanoTime();

for(int i = 0; i < 1000000; i++) {
    calculate();
}

long end = System.nanoTime();

System.out.println(end - start);

Although this appears correct, the result is often misleading because:

  • JVM may compile code during execution.

  • Garbage collection may occur.

  • CPU scheduling changes over time.

  • Dead code elimination removes unused computations.

  • CPU cache effects influence execution.

  • The first execution is usually slower.

These factors make manual benchmarks unreliable.

Why JMH is Preferred

JMH automatically manages several performance-related tasks:

  • JVM warm-up

  • Multiple benchmark iterations

  • Statistical averaging

  • Thread synchronization

  • Preventing compiler optimizations

  • Measuring execution throughput

  • Memory allocation measurement

  • Forking multiple JVM instances

Because of these features, benchmark results become much more reliable.

Features of JMH

Automatic Warm-up

When Java code runs for the first time, the JVM has not yet optimized it.

JMH executes several warm-up iterations before actual measurement begins.

Example:

Warm-up

Iteration 1
Iteration 2
Iteration 3

↓

JIT Compiler Optimizes Code

↓

Actual Benchmark Starts

This ensures optimized code is measured instead of startup performance.

Multiple Measurement Iterations

Instead of executing a benchmark once, JMH runs it repeatedly.

Example:

Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5

The average execution time provides more consistent results.

JVM Forking

JMH can launch completely new JVM processes.

Example:

Fork 1

Run Benchmark

Fork 2

Run Benchmark

Fork 3

Run Benchmark

Each benchmark runs in a clean environment, reducing interference from previous executions.

Statistical Analysis

JMH calculates:

  • Average execution time

  • Minimum time

  • Maximum time

  • Standard deviation

  • Confidence intervals

This helps developers understand whether performance differences are significant or caused by random variation.

Adding JMH to a Maven Project

To use JMH, add the required dependencies.

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.37</version>
</dependency>

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.37</version>
</dependency>

These dependencies provide the benchmarking framework and annotation processing.

Structure of a JMH Benchmark

A typical benchmark contains:

public class MyBenchmark {

    @Benchmark
    public void testMethod() {

    }

}

The @Benchmark annotation tells JMH that the method should be measured.

Example Benchmark

import org.openjdk.jmh.annotations.Benchmark;

public class StringBenchmark {

    @Benchmark
    public String concatenate() {

        String s = "";

        for(int i = 0; i < 100; i++) {
            s += i;
        }

        return s;
    }

}

JMH repeatedly executes the concatenate() method and records its performance.

Running a Benchmark

JMH generates benchmark reports similar to:

Benchmark                     Mode   Score   Error   Units

concatenate                  avgt    5.12    0.11    ms/op

Explanation:

  • Benchmark = Method tested

  • Mode = Average Time

  • Score = Average execution time

  • Error = Statistical error

  • Units = Milliseconds per operation

Benchmark Modes

JMH supports several benchmark modes.

Average Time

Measures average execution time.

Mode.AverageTime

Output:

Average = 2.5 ms

Useful when comparing algorithms.

Throughput

Measures how many operations execute every second.

Mode.Throughput

Output:

12000 operations/second

Useful for servers and APIs.

Sample Time

Randomly samples execution time.

Mode.SampleTime

Useful for long-running operations.

Single Shot Time

Measures only one execution.

Mode.SingleShotTime

Useful for startup benchmarks.

All Modes

Mode.All

Runs every benchmark mode.

Warm-up Configuration

Developers can customize warm-up.

@Warmup(iterations = 5)

Meaning:

Iteration 1

Iteration 2

Iteration 3

Iteration 4

Iteration 5

↓

Start Measurement

More warm-up iterations generally produce more stable results.

Measurement Configuration

Specify the number of measurement iterations.

@Measurement(iterations = 10)

JMH performs:

Warm-up

↓

10 Benchmark Iterations

↓

Average Result

Fork Configuration

Specify JVM forks.

@Fork(3)

JMH launches:

JVM 1

↓

JVM 2

↓

JVM 3

The final result is averaged across all JVMs.

Thread Benchmarking

JMH measures multi-threaded performance.

@Threads(4)

Benchmark executes using four concurrent threads.

Useful for:

  • Concurrent collections

  • Synchronization

  • Parallel algorithms

  • Thread-safe classes

Measuring Memory Allocation

JMH can report memory allocation.

Example output:

Allocation Rate

150 MB/sec

This helps identify methods that create excessive objects.

Comparing Two Algorithms

Example:

Algorithm A

Collections.sort(list);

Algorithm B

list.stream().sorted().toList();

JMH measures:

  • Execution time

  • Throughput

  • Memory allocation

Developers can determine which approach is more efficient for a given workload.

Avoiding Dead Code Elimination

If the result of a benchmark is never used, the JVM may optimize away the entire computation.

Example:

int sum = a + b;

If sum is ignored, the JVM might remove the calculation.

JMH prevents this by encouraging the use of returned values or helper classes such as Blackhole, ensuring the code being benchmarked is actually executed.

Common Benchmark Mistakes

Developers often make mistakes that lead to inaccurate results:

  • Using System.currentTimeMillis() for benchmarking.

  • Running only a single test iteration.

  • Ignoring JVM warm-up.

  • Comparing code without multiple benchmark runs.

  • Benchmarking code that performs file or network I/O.

  • Forgetting that garbage collection can influence timing.

  • Ignoring memory allocation during performance analysis.

JMH helps avoid these problems by providing a standardized benchmarking environment.

Best Practices for Using JMH

  • Always allow sufficient warm-up before measuring performance.

  • Run benchmarks multiple times and compare averages rather than a single execution.

  • Use multiple JVM forks to reduce the impact of previous benchmark runs.

  • Benchmark only the code you want to measure, keeping setup work separate.

  • Compare implementations under the same hardware and JVM configuration.

  • Consider both execution speed and memory allocation when evaluating performance.

  • Interpret results in the context of real-world usage instead of relying only on synthetic benchmarks.

Advantages of JMH

  • Produces accurate and repeatable performance measurements.

  • Handles JVM optimizations automatically.

  • Supports multiple benchmarking modes.

  • Measures both execution time and throughput.

  • Supports multi-threaded benchmarking.

  • Reduces errors caused by JVM warm-up and compiler optimizations.

  • Provides statistical reports for informed decision-making.

  • Widely adopted by the Java community and OpenJDK developers.

Limitations of JMH

  • It measures small units of code (microbenchmarks) and may not represent the performance of an entire application.

  • Writing meaningful benchmarks requires understanding what code should be measured.

  • Results can still vary across different hardware, operating systems, and JVM versions.

  • It cannot replace full application profiling tools when diagnosing production performance issues.

Applications of JMH

JMH is commonly used in:

  • Comparing different algorithms and data structures.

  • Optimizing utility libraries and frameworks.

  • Evaluating the impact of code changes on performance.

  • Measuring the efficiency of concurrent and parallel code.

  • Testing memory allocation patterns.

  • Benchmarking APIs, collections, and string manipulation techniques.

  • Validating performance improvements before deploying applications to production.

Conclusion

Java Microbenchmark Harness (JMH) is the standard framework for accurately measuring the performance of Java code. Unlike manual timing methods, it accounts for JVM warm-up, compiler optimizations, garbage collection, and other factors that can distort results. By providing configurable benchmark modes, statistical analysis, and support for multi-threaded testing, JMH enables developers to make informed decisions about performance optimization. It is an essential tool for anyone developing high-performance Java applications or evaluating the efficiency of different coding approaches.