New Relic Billing Questions

andrew

I’m extremely frustrated with New Relic and their handling of a recent billing situation. I’m sharing my experience here to warn others who might be in a similar situation.

We’ve been using New Relic with our Kubernetes cluster, following their basic setup instructions using their Helm chart. Our monthly bill had been consistently around $900. However, in November, it suddenly jumped to $1,400, and then in December, it skyrocketed to over $8,000.

I reached out to New Relic for an explanation, and after a few days, they responded, attributing the increase to a significant rise in log ingestion. They asked if we had made any changes, but we hadn’t, so I suspected it might be an issue with their integration.

A few days later, they told me that one of our services was responsible for the spike and provided a query to verify it: SELECT count(*) FROM Log WHERE newrelic.source = 'api.logs' FACET service SINCE 4 weeks ago TIMESERIES LIMIT MAX. The query showed that one of our services had the highest number, but it didn’t make sense since it was only generating a few MB per day.

Suspecting something was off, I ran the query again, this time using the namespace as the facet. To my surprise, the newrelic namespace was generating almost 2GB of logs daily.

New Relic itself was sending a massive amount of logs to their own system and charging us for it. This feels incredibly unethical, as we have no control over these log messages. While I can set up a filter to drop them, this should be the default setting. Moreover, the logs were tagged with a NULL service, making it difficult to identify the issue based on service name alone, which makes me wonder if this was intentional.

In our case, the charges were so high that I was forced to investigate, but I suspect that many other accounts using New Relic’s Helm chart are unknowingly paying for ingesting newrelic logs.

olivia

Even if all of this is accurate, how could 60GB of data per month possibly cost $7,000?

Sam Richards

It’s still not entirely clear, right? The log volume from the New Relic agent likely depends on various factors, such as the events it’s monitoring. It could be considered unethical depending on the circumstances, but based on what you’ve described, I’m not certain. For instance, if an engineer enabled verbosity for those logs, I wouldn’t blame New Relic. However, if the agent is crashing and generating verbose crash dumps, then maybe New Relic would be at fault. What caused the increase in log volume?

Rynn Mancuso

So one thing I would suggest is making sure you’re instrumenting your observability pipeline so you catch issues like this sooner. If you don’t have a pipeline and it’s literally just the New Relic agent doing everything, that’s something to think about changing - one of the useful things about multiple tools is that one of your other tools will usually alert on something like this.

I agree with everyone else that there isn’t enough information here for me to offer a really good answer. The first thing I would do is check the helm chart very carefully. Maybe “newrelic” was in there as a sample and someone forgot to replace it. Or maybe someone set the helm chart to automatically deploy on all new containers, the agent died and the process restarted in a new container, and boom.

Having said that I do think your success engineer at New Relic should be way more interested in helping you fix this situation, because obviously you’ll jump ship if your bill keeps on like this, and if it does turn out to be a configuration error I would still tell New Relic that they should drop the newrelic namespace by default so it’s harder for other users to make the same error.