Network Performance Report: October 2022

A review of the Solana network's performance, and upgrades being made to strengthen the network.

Network Performance Report: October 2022

Solana is a public, open-source blockchain that anyone in the world can build on and run.

The mandate of the Solana Foundation is to support the decentralization, security, resilience, and adoption of the Solana blockchain. The Foundation supports this mission through various programs and initiatives, ranging from creating documentation for validators to issuing grants for projects that further network decentralization.

As the Solana network matures and broadens its user base, it’s important that the Solana Foundation and greater web3 community understand the state of the network, where it’s going, and key metrics to track in monitoring its progress. The goal has always been to create a low-cost, accessible, and efficient global state machine. Anyone on the network should be able to access the latest information — its “state”— at the same time as everyone else, and the only barrier to entry should be an internet connection.

This is an ambitious goal, and while the network and community of builders have come a long way, there is still work to do. This report, the first of a regular series, is a deep dive into the network’s performance. It’s an opportunity to reflect on how the network has progressed over the past six months, examine where it’s failed, and identify key metrics that provide a holistic perspective on the network’s current performance.

This report is part of a broader series of research on the Solana network that includes the validator health report and energy use report. The Solana Foundation welcomes input from the Solana community on this report. Fill out this form with questions, feedback, and metrics that you’d like to see tracked.

Network highlights since March 2022

Announcement of Firedancer, an alternative open source validator client: In August, Jump Crypto announced their intent to build a new open-source validator implementation for Solana. This would be the second open-source validator client for the Solana blockchain, a major step towards furthering the reliability and throughput of the network.

Testnet reaches 24,000 transactions per second: Testnet reached its highest transaction throughput to date on September 22, 2022,  eclipsing 24,000 transactions per second. While testnet load is not an exact copy of Mainnet-beta’s load, it is a leading indicator for Mainnet-beta’s progress. This is a significant step for the performance of the network.

QUIC rollout to Mainnet-beta: Solana’s transaction ingestion protocol is being reimplemented on top of QUIC, a protocol built by Google, designed for fast asynchronous communication. This is an improvement on top of the existing UDP-based protocol, which does not have sessions and flow control built in. QUIC is currently live on testnet and mainnet-beta. However, Mainnet-beta still supports both UDP and QUIC. QUIC is planned to be the default transaction ingest and forwarding protocol on Mainnet-beta in the 1.13.4 release.  Once QUIC is adopted, there will be additional options available to optimize data ingestion and discourage abusive behavior. (Learn more in this tech talk by Anatoly Yakovenko on QUIC).

Surpassing 2,000 block-producing validators: In August 2022, the network surpassed 2,000 active block-producing validators on mainnet-beta, a notable milestone in the health of the validator network and the evolution of the blockchain. This is one of the largest number of validators for a Proof of Stake network and represents 4x growth in validator count from August 30 2021, when 495 block-producing validators were online. See below for comparison:

Network

Block-producing nodes

Solana 1

2,124

Ethereum 2

7,139

Avalanche 3

1,234

NEAR 4

127

Polygon 5

79

Binance Smart Chain 6

51


Network challenges since March 2022

The Solana network experienced three outages since March 2022. No user funds were lost in any of these incidents.

Below, we review the outages and what is being done to address them.

September 2022 Outage: On September 30, Mainnet-beta stopped producing blocks for 6 hours and 19 minutes when the network was unable to recover from a fork caused by a bug in the consensus algorithm implementation. This led to a validator producing duplicate blocks for its leader slots.  The bug was removed and patched. Read the full outage report.

June 2022 Outage: On June 1, the Mainnet-beta cluster ceased producing blocks for approximately 4 and a half hours. This was a result of stalled consensus, caused by a bug in the durable nonce transactions feature. Releases v1.9.28/v1.10.23 included an update designed to prevent the network from halting if the same situation were to arise again. Read the full outage report.

April 2022 outage: On April 30, Solana’s mainnet beta cluster suffered a 7 hour outage caused by stalled consensus after an enormous amount of inbound transactions (6 million per second) flooded the network. Evidence suggested that bots programmatically tried to win a new NFT being minted. Read the full outage report.

Several proposed network upgrades have been implemented or are being rolled out to address the causes of this outage, specifically the implementation of QUIC, fee markets, and stake-weighted Quality of Service (QoS).

Upcoming initiatives

Solana core developers have been working on a number of new network upgrades aimed at strengthening the network in the face of massive user growth and adoption. Follow the status of these upgrades.

QUIC: QUIC is live on most validators and RPC nodes in Mainnet-beta. However, Mainnet-beta still supports both UDP and QUIC for transaction ingest. QUIC is planned to be the default transaction ingest and forwarding protocol on Mainnet-beta starting with the 1.13.4 validator client release. Follow its progress on Github.

The Solana network previously used a UDP-based protocol to pass transactions between RPC nodes and the current block leader. UDP lacks flow control and receipt acknowledgments, so it doesn’t provide a meaningful way to discourage or mitigate abusive behavior. To address this, Solana’s transaction ingestion protocol is being implemented on top of QUIC, a protocol that includes sessions and flow control.

Stake-weighted Quality-of-Service (QoS): This was built in parallel with QUIC and enabled before QUIC was adopted on Mainet-beta. Follow its progress on Github.

Currently, block producers accept transactions without regard for the transaction’s source. Solana is a Proof of Stake network, and stake-weighted QoS extends the utility of stake-weighting to transaction quality. Under this model, a node with .5% stake will have the right to transmit at least .5% of packets to the leader. Transactions submitted by validators that have stake in the network will always be received by the block producer regardless of the amount of traffic coming in from nodes without stake.

Fee Markets: Fee markets have been partially introduced and are expected to be further implemented after QUIC is fully adopted. Follow its progress on Github.

The first piece of Fee Market implementation is support for Priority Fees. Previously, the network processed transactions on a first-come-first-served basis, which didn’t leave users with any way to express the urgency of a transaction’s execution. Priority Fees change this by offering users the ability to specify an additional fee at their discretion to be collected upon the execution of a transaction and its inclusion in a block. Priority Fees are calculated based on the amount of computing resources that a transaction is expected to require. For example, a simple token transfer would require a lower total Priority Fee than an NFT mint that expresses the same level of urgency. Priority Fees have been live on Mainnet-bteta since July 2022.

Future releases will add additional functionality to Fee Markets. These include new RPC methods that will help users and apps determine the minimum additional fee required to be likely included in the next block, higher fees for highly contested accounts, and improvements to block scheduling. These pending improvements are expected to create a more efficient Fee Market for all Solana participants.

Overview of network performance

In order for a billion people to use and take advantage of the benefits of the Solana network, users need to feel confident that they can use the network any time, it will give them accurate information, it will be fast, and that their funds will be safe.

Below are a few of the metrics that the Foundation tracks to see how well the network is measuring up to this goal. While a variety of metrics may help illustrate the network’s performance, we highlight four for the sake of simplicity and to allow users to easily track these metrics and how they progress over time. We’ve included these metrics, as well a few others, on a Dune dashboard that makes it easy to dig a layer deeper into the data or run your own analyses.

Uptime

The Solana and broader web3 community have raised important questions about the reliability of the network, given outages in the second half of 2021 and first half of 2022. A constantly reliable network is foundational to the trust and continued growth of the network.

One of the most important measures of reliability is network uptime.

Below, we snapshot the network’s monthly uptime, measured as the percentage of uptime in a given month over the past 6 months.

Average monthly uptime

04/2022

05/2022

06/2022

07/2022

08/2022

09/2022

10/20227

99.6%

99.3%

99.4%

100%

100%

99.81%

99.12%


Source

Ratio of voting to non-voting transactions

Voting transactions are necessary for the functioning of the network; a vote transaction happens when a validator votes to confirm one or more proposed blocks. Non-voting transactions are the ones triggered by user behavior on the blockchain. For example, a non-voting transaction might represent an NFT mint or a user transferring tokens from one wallet to another. A certain number of voting transactions are required for the network to reach consensus, but over time, we would expect to see the ratio of voting to non-voting transactions go down because the overall percentage of voting transactions should drop as the network gets more efficient.

Below is a look at voting to non-voting transactions over time:

Block time (time to produce a block)

Block time measures the speed of a single transaction, since it measures how quickly the network adds more “blocks” to the blockchain. In this chart, we see the average time to produce a block, how consistent this metric is, and how it changes over time.

Average and maximum transactions per second

Total transactions per second are the most accurate reflection of the current throughput of the network and demonstrate its potential and growth over time. The benchmark throughput is 65,000 transactions per second, based on simple transactions like sending money from one place to another. Actual transactions per second will differ due to the varied mix of complex transactions on the network, and based on demand at any given moment. An NFT purchase, for example, is much more complex than a simple movement of tokens across wallets.

The chart below is the best snapshot we have of how well the network is performing in real time. It’s segmented into average transactions per second on a given day, along with maximum transactions per second that the network has ever hit at that point in time. Average transactions per second have been increasing since June.

It’s important to note that actual network transactions per second is not a reflection of network capacity, but of demand for transaction throughput. In almost all cases, Mainnet-beta is operating below capacity.  

Average transactions per second

Max transactions per second

As major proposed upgrades are rolled out, the Solana Foundation will update the community on its progress on the upgrades page. The Solana Foundation will continue to monitor the performance of the Solana network and use reports like this one to help the community track its development over time.

Footnotes (Click to expand)
1. Source: https://solanabeach.io/validators as of 10.28.22
2. Source: https://ethernodes.org/sync as of 10.25.22
3. Source: https://explorer-xp.avax.network/validators as of 10.28.22
4. Source: https://explorer.near.org/ as of 10.28.22 5. Source: https://polygonscan.com/stat/miner?range=7&blocktype=blocks as of 10.28.22
6. Source: https://www.bscscan.com/validators as of 10.28.22
7. This reflects uptime as of October 30, 2022. This report will be released before we have data on uptime for the full month of October.