- Over a week ago, Polygon zkEVM experienced a lengthy outage.
- Developers have released a report explaining the events that sparked the network halt.
- Developers also detailed steps taken to prevent future occurrences.
Just over a week ago, Polygon Labs’ Ethereum Layer 2 solution, Polygon zkEVM, experienced an issue that saw block production on the network halt for hours. While the issue has since been resolved, the details of the problem remained unclear until now. As promised, developers have released a detailed post-mortem explaining the root causes of the event and how developers intend to avoid future occurrences.
What Caused the Polygon zkEVM Outage?
In earlier statements, Polygon Labs had disclosed that the Polygon zkEVM outage had been triggered by an Ethereum mainnet “reorg.” A reorg, short for reorganization, arises when a disagreement over transaction ordering or validity forces a network to converge on a single version of the truth, considering the chain of transactions backed by the most computational work or stake.
According to developers, following this reorg, the Polygon zkEVM sequencer, responsible for ordering and batching transactions to Ethereum, encountered an issue leading to the outage.
In the recently released post-mortem, developers detail that the Ethereum reorg had omitted a Polygon zkEVM deposit transaction—a change that the Layer 2 chain’s synchronizer, responsible for communicating Layer 1 changes, had failed to realize on time.
As explained by developers, this lapse allowed Polygon zkEVM’s sequencer to order transactions with timestamps that had been deprecated on the mainnet. The result: transactions batched to the mainnet were rejected because they did not carry the expected timestamps.
"Valid proofs for these transactions were generated, but the result of these transactions were no-operation, as if they were not there," developers reported.
Impacts and Fixes
In all, about 4,000 transactions were affected. Developers attribute Polygon zkEVM’s lengthy 14-hour outage to efforts to reconcile these transactions.
Developers disclosed that they released new node and prover versions as part of fixes targeting the synchronizer. They also included an extra feature in the sequencer to help double-check timestamps with the mainnet.
Meanwhile, Polygon zkEVM’s TVL does not appear to have been affected by the outage. The metric is up about 2% since the outage, from about $150 million to $153 million, per L2Beat data at the time of writing.
On the Flipside
- Over a week later, Polygon Labs noted that some of the affected transactions may still not have been processed, calling on DApps to contact them in these instances.
- Polygon zkEVM is still in mainnet beta.
- The Polygon zkEVM outage did not impact the Polygon PoS chain or any CDK-powered chain.
- Network outages are becoming increasingly frequent in crypto, and the Ethereum Layer 2 space is no exception. In December 2023, Arbitrum, the largest Ethereum Layer 2 network by TVL, experienced an over five-hour outage due to a surge in inscriptions. In the same month, zkSync Era, the largest ZK-rollup by TVL, also experienced a four-hour outage.
Why This Matters
The post-mortem report helps address lingering questions over the lengthy 14-hour Polygon zkEVM outage.
Read this for more on the Polygon zkEVM outage:
Polygon zkEVM Outage Resolved: What Led to the Network Halt?
Learn about last week’s crypto fund flows:
Crypto Funds Net Inflows Resume on Bitcoin’s $70,000 Rebound
