Layer-1 blockchain network Solana is currently witnessing a degraded performance following a bug that caused block production to drop significantly. The number of transactions processed per second on the network has dropped from the usual highs of close to 5000 TPS to less than 100 in the past few hours.
(Source: Solana Explorer)
Solana Labs engineers and validators are still investigating the root cause of the chain fork and degraded network performance. However, early speculations are that the issue relates to a recent Solana software upgrade from version 1.13.6 to version 1.1.14. The recent degraded performance began around the same time the majority of Solana validators (a supermajority) switched to the newer version.
UPDATE: Although root cause is not identified yet, some potential fixes are being considered:
– Incremental downgrade (v1.14 -> v1.13), issue may be buried in major v1.14 upgrade that was recently rolled out
– Network restart attempt (last resort)NOTHING HAS BEEN CONFIRMED
— SolBlaze.org | Stake with us! ???????? (@solblaze_org) February 25, 2023
Therefore, one of the solutions being explored by Solana validators on Discord is a voluntary downgrade to the previous stable version. A supermajority of validators would need to downgrade to the older version, hence the continued poor performance.
Another alternative being reviewed is a network restart which could take the network offline for around 4-5 hours, barring any technical challenges. Notably, the downgrade process failed with validators electing to restart the network from the last workable version, 1.13.
The cluster restart was completed successfully after a second attempt, with Solana producing blocks again.
UPDATE: This report was updated to include information that Solana validators successfully restarted the network.
