Incident Post Mortem: November 19, 2021

Original author: Coinbase

Go to source

Summary

On November 19, 2021, Coinbase learned that it had erroneously credited some customers transacting in GYEN and POWR either 100x or 1/100th the amount they purchased. Coinbase promptly disabled trading in POWR and GYEN, worked around the clock to resolve the underlying technical issue, and then made adjustments in customer accounts to reflect the amount of GYEN and POWR that customers actually purchased. This incident affected approximately 0.0072% of Coinbase’s total verified users.

What happened?

On November 19 at approximately 4:00 p.m. EST, Coinbase updated an internal data source related to POWR and GYEN precision. The update was tested through our standard automated testing and deployment monitoring procedures. However, the testing didn’t detect that the update would propagate at various speeds through a number of internal systems and would result in customers being credited either 100x or 1/100th the amount of GYEN or POWR they purchased.

The data rollout error was identified through our position risk monitoring systems shortly after the November 19 4:00 p.m. EST update. At 5:35 p.m. EST, we disabled transacting in GYEN and POWR pending resolution of the underlying issue. At 7:26 p.m. EST, we identified accounts that transacted in GYEN or POWR during the data rollout, and temporarily restricted these accounts pending further investigation. By November 21, restrictions were removed for 98.8% of these accounts and, by December 13, Coinbase restored full trading for GYEN and POWR.

What did Coinbase do to correct the problem?

Coinbase immediately devoted substantial engineering resources to quickly correct the problem, ensuring our customers received the correct amount of GYEN and POWR that they purchased. For customers who were erroneously over-credited 100x the GYEN and POWR they purchased, we ensured that they received the correct amount of assets that they paid for. For those who still had GYEN and POWR in their accounts, this was relatively straightforward — we notified customers of the error and simply debited those customers’ accounts, removing the extra GYEN or POWR that was erroneously credited.

Some customers had already converted their GYEN and POWR to other digital assets, such as Bitcoin. Other customers sent their GYEN and POWR to wallets off the Coinbase platform, but kept other digital assets on the Coinbase platform. For these customers, we notified them of the error and, in accordance with the Coinbase User Agreement, withdrew other assets from these customers’ Coinbase accounts equal to the amount of GYEN or POWR they had been over-credited.

When determining how much to debit from these customers’ accounts, we used the most favorable exchange rate for our customers. Specifically, we calculated the USD value of the GYEN or POWR owed to Coinbase by using the lowest exchange rate on the Coinbase Exchange from the time this incident began until trading was halted ($0.00825/GYEN, $0.4742/POWR). This minimized the amount owed to Coinbase by these customers. We then debited funds from user accounts up to this USD value, starting with their fiat balances, then USDC and other stablecoin balances, followed by other digital asset balances ranked by descending market cap. The value of these digital assets was calculated using the market rate at the time user accounts were debited.

A small group of customers who were erroneously over-credited GYEN or POWR sent these digital assets off-platform and left no other assets on the Coinbase platform. Coinbase has been reaching out to those customers individually and appreciates our customers’ cooperation returning the erroneously credited GYEN and POWR. Repayment of the over-credited funds is required under the Coinbase User Agreement.

For customers who were undercredited GYEN or POWR, receiving a lower amount than they purchased, Coinbase first determined the amount of GYEN or POWR owed to these customers. Coinbase then calculated the USD value of the GYEN or POWR owed to customers by using the highest exchange rate from the start of the incident until the remediation process was completed ($0.009799/GYEN, $0.9617/POWR), which was the most favorable exchange rate for our customers. That means that regardless of the price customers purchased at, we assumed that the customers would have sold these assets at the highest price while trading was disabled. After calculating this USD value, we credited customers an equivalent amount of Bitcoin. We credited these customers in Bitcoin because GYEN and POWR trading was still suspended, and Bitcoin is used in every country where customers were affected.

To further benefit our customers we used an exchange rate of $55,000/BTC, which was lower than the market rate of BTC at the time these BTC payments were made. This exchange rate ensured our customers received more Bitcoin than they would have received had we used the actual BTC-USD exchange rate at the time.

Additionally, for all customers whose accounts were restricted, Coinbase provided a customer experience credit of up to $100 in BTC.

What happens next?

Many customers still have questions about how their accounts were credited or debited because of this incident. For questions specific to your account, please feel free to reach out to Coinbase Support.

Coinbase is also revising the information in our customers’ account statements and tax forms to correctly reflect our customers’ GYEN and POWR transactions. If your statements or tax forms appear incorrect, please reach out to Coinbase Support, but know that we are working to correct that information as well.

GYEN Values Before the Data Rollout Error.

In the days leading up to the data rollout error, between November 16 and November 19, Coinbase Exchange observed GYEN-USD break parity when compared to JPY/USD. We have seen customers speculate on social media that this incident was somehow related to this break in parity. We have also seen customers speculate that this break in parity was somehow caused by Coinbase. These allegations are false and reflect a misunderstanding about what GYEN is and how Coinbase works.

The Break in Parity Occurred Before and Was Not Related to the Incident. This break in parity occurred days before the incident. At the peak of this break in parity, on November 17, 1 GYEN traded for approximately ¥7.48.

By the time the data rollout error occurred, on November 19, GYEN’s price stability had recovered and GYEN was trading at approximately ¥0.96–0.98. When Coinbase unrestricted impacted customers’ accounts, GYEN was trading at approximately ¥0.98. In other words, the break in parity occurred before the data rollout error and the two issues had nothing to do with each other.

How the Break in Parity Occurred. When Coinbase listed GYEN, there was significant demand for GYEN that could not be matched by supply. The surge in buyer demand for GYEN, coupled with the insufficient supply of GYEN across all markets (not just Coinbase), ultimately caused the break in parity. From November 17 through November 19, Coinbase implemented an alert, informing its customers who were buying, selling and trading GYEN of “Unusual Market Activity — Due to unusual market activity for GYEN, you may have trouble trading GYEN on Coinbase.com. We apologize for any inconvenience caused by this.” The break in parity occurred because of these market conditions specific to the GYEN digital asset unrelated to Coinbase operations.

---

Incident Post Mortem: November 19, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.