Tesla Blocked Teslemetry

It's been a very hectic month for me, both personally and with Teslemetry. I moved out of my house, temporarily moved to my in-laws over an hour away, and started renovating our new house. If you have ever renovated I'm sure you understand all the stress and challenges that come with that, but it also meant I wouldn't have access to a Tesla Powerwall for testing until my new system could be installed at the new house a few weeks later. It's only a few weeks though right, what could go wrong in that time?

Forbidden

A few days later, I woke up and started responding to support request notifications, and after doing some basic troubleshooting I saw an error I had never seen before: Powergate (the Tesla Energy backend) was blocking the request with a 403 Forbidden. I had no energy site of my own to confirm, but that's when I saw the support inbox with 60-70 new tickets, all reporting the same issue.

I initially assumed this was another issue with Tesla's backend, as the Energy APIs are notoriously unreliable and suffer small outages every few weeks, but after asking around it was clear this issue was only impacting Teslemetry.

Tesla had blocked the Teslemetry Client ID.

Support

I had been contacted by Tesla Energy in the past about an issue the Teslemetry service was causing, and rightly so as a bug in error handling was causing a storm of command retries that would never succeed. The Tesla staff I dealt with there were very helpful and friendly, and we resolved that issue within a few hours by implementing a limited exponential backoff.

So when Teslemetry got cut off with zero contact, it came as a shock. I reached out to Tesla Fleet API support by email and through their official Jira service desk, to those friendly Tesla Energy staff from the last issue, and other Tesla employees I had spoken to in the past. Unfortunately, over two weeks later, I still haven't received a single response from any of them.

Teslemetry pays Tesla thousands of dollars a month to use the Fleet API, admittedly for the Vehicle side, but it's infuriating their support is basically non-existent.

I eventually reached out to a friendly competitor who I knew was more well connected at Tesla and they were able to confirm my Client ID was blocked as I was "overloading" their system. The block was lifted a few hours after that.

Overload?

Had Teslemetry's energy API usage spiked right before the block, I could have understood we may have triggered some automated rate limit system that cut off access, but I have an extensive OpenTelemetry implementation to help troubleshoot and understand what's happening in Teslemetry, so I know that we made approximately 10 million API calls every day leading up to the block. There was no spike in total requests or increase in specific endpoints, as you can see in the graph below, which shows the last 30 days in 12 hour blocks.

You may notice a small bump on the day of the incident, but if we zoom in you can see this is actually after the 403 responses started (light-green in color) and was a result of a change pushed out to the API, and an expected response to Home Assistant and other clients retrying their failed requests.

Rerouting

After 24 hours with no response from Tesla I decided to implement a workaround, so Teslemetry started routing all energy API requests through another application, but this required new systems to ensure users were correctly authenticated to the new application and seamless routing of requests. After a few iterations, we thankfully had a working solution that allowed all customers to continue using the Tesla Fleet API Energy APIs.

To compensate customers for the ~36 hours of downtime, we are offering an optional 5 day subscription extension to all customers. I greatly appreciate those of you who reached out to me asking not to receive a subscription extension, in support of Teslemetry. As stressful as this incident was, knowing I have such a passionate and supportive community has been amazing.

Where to from here?

At this moment, Teslemetry is now operating normally, with all energy API requests being routed as normal. I was told Tesla will reach out to me, which I am hopeful will occur early next week, and I will update this blog if they do. A lot of businesses were built off the back of reverse engineering the Tesla Owner API, whereas Teslemetry has only ever operated using the official Tesla Fleet API. This was meant to represent a more stable and supported path to building products and services on top of customers' Tesla product data, but this incident has shattered my thinking in this regard.

Now that I have my Powerwall 3 with 2 expansions, I am going to invest time in supplementing the Fleet API with higher frequency real-time local data in Home Assistant using techniques recently discovered by pypowerwall contributors and PowerSync. This should both provide a better user experience and reduce our reliance on Powergate systems.

I am hopeful that I will be able to have a fruitful discussion with Tesla regarding this incident and come up with a reliable path forward, but that would require them to return my emails.

Update

Tesla has reached out to me as a result of this blog post, and seemly their email system blocks my emails. More to come...