ETH 2.0 Medalla Testnet Crashes to Cloudflare-Related Time Bug on Prysm Client

Earlier this month, the ETH2 development team rolled out the Medalla testnet, marking another vital step towards the final release of the long-awaited improved version of the Ethereum network.

Launching testnets allow for developers to find bugs in the software, and engineer fixes before rolling out advanced testnet versions and eventually shipping the final product.

On August 14, ETH2 developers noted an otherwise critical bug that caused the network to not reach finality for approximately 90 minutes. Although the issues have been mitigated at press time, it resulted in the number of Medalla testnet validators and their testnet ETH holdings plummeting from near 75% to 5% within the said period.

Prsym clients suffer from Cloudflare time bug

The Medalla testnet downtime was subsequently linked to an erroneous roughtime responses on Prysm, an ETH2 client that relies on third-party web infrastructure provider, Cloudflare.

While Cloudflare roughtime servers are expected to report an offset of 0.1s or less, the time bug resulted in all Prsym nodes exhibiting a clock skew up to 4 hours into the future. Because of this clock skew, validators incorrectly proposed and rejected blocks and attestations for future slots. 

A post mortem report subsequently published by Prysmatic Labs, the Prysm client development team shows that the clock skew lasted approximately 90 minutes between 17:30 to 19:00 UTC.

The issue with Prsym nodes had a significant impact on the Medalla testnet because over 70% of ETH2 validators were using the client. Although four other clients, namely Teku, Nimbus, Lodestar, and Lighthouse can also be used on the testnet, the heavy reliance of Prysm meant the network could not withstand the sudden inactivity by a majority of validators.

As noted at the onset, the discovery of bugs such as the latest involving Prysm nodes is one of the reasons why testnets exist. A fairer distribution of client usage and increased validator participation may easily mitigate similar occurrences in the future.

On the other hand, it is worth recalling that the genesis launch of Ethereum 2.0 expected later this year depends on the progress of the Medalla testnet, and whether all major bugs are fixed in this phase of development.

Either way, the Ethereum Foundation is reportedly developing an internal security team for the long-anticipated release. According to Ethereum co-founder Joseph Lubin, prime-time Ethereum 2.0 will increase the network scalability by over 1000x.