It's not just cloud costs that are out of control: burning 450 million yuan a year for observability

Tech 2023-05-30 09:35:43 Source: Network

InfoQ Architecture HeadlinesMay 2023-2915:00 Published in HeilongjiangOrganize | Chu Xingjuan, Nuclear ColaWhich company spent $65 million on Datadog in 2022? "Datadog recently revealed at a financial meeting that a customer made a one-time prepayment of up to $65 million, which instantly sparked industry attention: which company is so generous? Can you observe how suppliers make money?Datadog is the main force of observability, with a financing listing in 2019 and a current market value of $28 billion. The company's revenue in 2022 was $1

InfoQ Architecture HeadlinesMay 2023-2915:00 Published in Heilongjiang

Organize | Chu Xingjuan, Nuclear Cola

Which company spent $65 million on Datadog in 2022? "Datadog recently revealed at a financial meeting that a customer made a one-time prepayment of up to $65 million, which instantly sparked industry attention: which company is so generous? Can you observe how suppliers make money?

Datadog is the main force of observability, with a financing listing in 2019 and a current market value of $28 billion. The company's revenue in 2022 was $1.67 billion, with a monthly average of around $140 million. In a small investigation on cost reduction, "AWS" and "Datadog" were the two most mentioned suppliers. This clearly proves that the cost of infrastructure and observability is already quite high, with AWS being the leader in the field of infrastructure.

On May 4th during the financial report conference call, Datadog CFO David Obstler mentioned a "non recurring expense" (also known as a one-time expense) and stated:

This quarter's revenue was $511 million, an increase of 15% compared to the same period last year. In the first quarter of 2022, a customer made a huge advance payment, but there were no similar large expenses in the first quarter of 2023. According to estimates for this customer, their bill growth rate will be less than 30% year-on-year

This detail was captured by Mark Ronald Murphy, Executive Director and Financial Analyst of JPMorgan Chase's research department. After conducting calculations, Murphy revealed that the prepayment was approximately $65 million (approximately RMB 450 million), and Datadog also acknowledged the accuracy of the figure. Obstler stated that the company has changed the billing frequency and amount, so over time, the customer's bills will be shared more.

OBSTLER revealed, "This is a cryptocurrency company that is still our client. They are an early optimization vendor in the industry that we often discuss with the greatest impact and the widest optimization space

Olivier Pomel, co founder and CEO of Datadog, stated that in the past year, the client's vertical industry has been almost destroyed. Their own business revenue has decreased by three to four times. In this situation, we will work with the client to restructure their contract with us. We want to be part of their solution, not part of the problem

At this point, there are various speculations online about which encryption company spent $65 million on Datadog in 2022.

Investor TurnerNovak speculates that it is Coinbase, but it is not very certain. There have even been people impersonating Coinbase employees online. For example, an anonymous commentator on HackerNews claimed that $65 million is actually a prepayment for the next three years, but the content of the message cannot be verified. Later, FarmleND full stack engineer Gergely Orosz wrote that he confirmed that the company was Coinbase, and that payment was their due bill that year. Let's take a look at Orosz's detailed explanation below.

No one cares about infrastructure costs anymore

Coinbase went public in June of that year, with a valuation of $85.7 billion on its first day of listing. By comparison, nearly two years later, the company's valuation was approximately $14 billion, a decrease of approximately 75%. During the boom period, transaction volume surged and hit new highs, while Coinbase infrastructure was almost unable to keep up. Brian Amstrong, CEO of Coinbase, once stated:

2021 is an incredible year for Coinbase, not to mention a rare trend in a person's life, it is also one of the few in the entire history of business development. Our monthly transaction users reached a record high of 11.4 million, a four fold increase compared to the same period last year. A growth rate of 400% is simply unbelievable

After the initial public offering in 2021, no one at Coinbase cares about infrastructure costs anymore, and the only goal is to continue growing. The company has paid significant fees to manufacturers such as AWS, Snowflake, and Datadog. So, in 2021, this $65 million was indeed spent on Datadog, and Coinbase settled this expenditure in the first quarter of 2022.

At the beginning of 2022, the situation in Coinbase has sharply declined, requiring immediate cuts in infrastructure spending. This is because the encryption industry suddenly experienced a downturn, which naturally affected Coinbase's business. As revenue dried up, the company began to shift its focus to reducing costs and increasing efficiency.

In terms of observability, Coinbase has established a dedicated team with the goal of transferring this part of functionality from Datadog to internal
On the Grafana/Prometheus/Slickhouse technology stack.Below is a brief introduction to these technologies:

Prometheus: A temporal database. As a highly popular open-source solution for system and service monitoring, Prometheus collects metrics from configured targets (services) at given time intervals and combines evaluation rules to trigger alerts.

Prometheus is mainly written by Go and also useful for Java, Python, and Ruby code. Prometheus can store temporal data in an efficient and customizable format in memory and on persistent storage media (HDD or SSD), and supports partitioning and federated deployment.

Prometheus is a project under the jurisdiction of cloud native base, so building business on top of it is relatively secure. This project will have stable maintenance and support for both the current and foreseeable future.

Prometheus can be self hosted, and some cloud service providers are also offering Prometheus hosting services: both GoogleCloud and AWS offer production level service options, while Azure's services currently only offer preview versions.

Grafana: Visualization indicator front-end. Grafana is a popular source analysis and monitoring visualization solution. If you need to view or delve deeper into metrics or alerts, Grafana is the ideal tool of choice and widely used in technology enterprises. Example of Grafana dashboard:

Clickhouse: Logging management tool. This is a fast, open source, and column oriented database management system, and is currently a popular log management option. Clickhouse is mainly written in C++and is widely used throughout the industry. For example, Cloudflare uses Clickhouse to store all its DNS and HTTP logs - over 10 million rows per second! Clickhouse is also Uber's central logging platform.

The initial reason why Coinbase chose to do it on its own was not to save costs, but to achieve complete control and observability. Observability and reliability are also Coinbase's biggest trumps when competing with competitors in the market.

But as the cryptocurrency market cools down,Cost has become a core concern, and the Grafana/Prometheus solution for internal operations is indeed much cheaper. The Coinbase team has been repeatedly debugging the new technology stack for several months, ultimately resolving all issues and confirming that everything is running normally.

In this way, Coinbase originally planned to say goodbye to Datadog about this, but Datadog saved the partnership at the last moment by offering Coinbase a favorable clause that it could not refuse. Simply put, the subsequent bills issued by Datadog will be much lower than the $65 million in 2021. After all, as Brian Armstrong said when talking about the cryptocurrency market in 2021,A bill of $65 million is also uncommon throughout business history.

In order to retain customers, Datadog suffered a "fracture"

Orosz Coinbase Datadog Datadog . Datadog .

Coinbase has finally been able to design a similar experience internally, but achieving a seamless developer experience similar to Datadog may require additional costsDecades.

. Coinbase Datadog 1000 . 1000

Datadog .

2022 Coinbase 18 12 . 2022 900 . Datadog Datadog 24 .

Datadog Coinbase 2 Datadog Datadog 36 .

By doing a simple mathematical calculation like this, Coinbase chose Datadog and saved $9 million in downtime alone, so now the annual expenditure of $10 million can be said to be worth it!

A considerable bill worth millions of dollars is not uncommon

DataDog . SaaS .

6500 1000 . Groundcover Shahar Azulay .

Coinbase 1000 .Azulay SplunkDynatraceDataDog 1000 .

Azulay .: ()..

It is full of great unpredictability and heavily relies on the amount of data pushed to the log.Azulay .

log lines 100 100 .Azulay log lines .:: 50%.

Azulay believes that,The problem may fall on the developers as they push too much information into the observability stack..

. eBPF Groundcover .

Who is the 'big enemy'

Datadog Coinbase. Coinbase More reflecting the overall trend of the market.

Datadog CEO Olivier Pomel confirms that similar cost optimization measures are occurring among all customers:

...

Datadog .Orosz Shopify Datadog .

Orosz Grafana/Prometheus .

200 500 ..Orosz .

. AWSGoogle Cloud Platform Azure..

Infrastructure cost+platform team cost< Existing supplier costs

100 200 . 45 15 40 .

200 300 ..

Orosz Coinbase : 6500

6500 Coinbase 10 / 500 . 100 .Orosz .

https://investors.datadoghq.com/static-files/18234a4f-04f9-4a9f-9679-668cd672fb7b

https://blog.pragmaticengineer.com/datadog-65m-year-customer-mystery/

https://thenewstack.io/datadogs-65m-bill-and-why-developers-should-care/

Disclaimer: The content of this article is sourced from the internet. The copyright of the text, images, and other materials belongs to the original author. The platform reprints the materials for the purpose of conveying more information. The content of the article is for reference and learning only, and should not be used for commercial purposes. If it infringes on your legitimate rights and interests, please contact us promptly and we will handle it as soon as possible! We respect copyright and are committed to protecting it. Thank you for sharing.(Email:[email protected])

Mobile advertising space rental

Tag: It not just cloud costs that are out of