Superchain testnets' blob fee bug

At the time of writing, all OP Stack testnets are currently sequencing a "wrong" chain. The L1 Block's blob base fee is calculated with pre-Prague blob schedule parameters. This is not a critical security issue on any OP Sepolia (or Holesky) chain. However, when L1 blob base fees rise to 100 Mwei or higher on the L1 testnet, there is a temporary liveness risk, because we'd be overcharge L1 fees.

This was fixed on the develop branch with this PR (opens in a new tab) and there will be no impact on any mainnet chains. Mainnet chains will just need to update their binaries to the latest release prior to the activation of Pectra on Ethereum, so the bug isn't triggered when Pectra activates.

The following sections describe the impact of this bug and how to fix it on your testnet.

Impact

Testnet L1 cost overcharging

All OP Stack testnet chains are currently sequencing an unintended chain. The L1 Block's blob base fee is calculated with pre-Prague blob schedule parameters. The L1 Block's blob base fee is calculated with pre-Prague schedule parameters. This has little impact as long as blob base fees on Sepolia are low, but as they rise, we exponentially overcharge. More concretely, our blob base fee calculation is off by an exponential of 1.5:

BBF_Cancun = BBF_Prague^1.5

This has a huge impact as fees grow:

BBF Prague (actual)	BBF Cancun (we charge)
1 Mwei	1 Gwei
10 Mwei	31.6 Gwei
100 Mwei	1 Twei
1 Gwei	31.6 Twei (31.6e12)
10 Gwei	1e15 Wei = 1 PetaWei
100 Gwei	31.6e15 PetaWei
1 Twei	1e18 Wei = 1 ExaWei

For chain and node operators

All chain operators will need to coordinate with their node operators to schedule a hardfork activation time to utilize the correct blob base fee calculation. The following is a list of chains who have opted into the Superchain hardfork activation inheritance behavior (opens in a new tab) are OP Sepolia, Soneium Minato, Zora Sepolia, Unichain Sepolia, Base Sepolia, Mode Sepolia, Metal Sepolia, Creator Chain Sepolia, Ink Sepolia, and Ethernity Sepolia.

The node operators for these chains should use the network flags (--network and op-geth with the --op-network) to inherit the Thu Mar 20 16:00:00 UTC 2025 (1742486400) hardfork activation time.

The following chains are in the Superchain Registry and are not opted into the hardfork activation inheritance behavior: Arena-Z Sepolia, Cyber Sepolia, Funki Sepolia, Lisk Sepolia, Pivotal Sepolia, Race Sepolia, Shape Sepolia, TBN Sepolia, and Worldchain Sepolia. These chains and any other OP Stack testnet that is not included in the lists above will need to manually set the hardfork activation time for their network to activate on. These following steps are necessary for every node operator:

Update to the latest release

⚠️

You must configure your op-node to utilize the activation timestamp outlined in step 2 at the same time as upgrading your node binary. This is to ensure that the hardfork is activated uniformly across the network. If the Pectra Blob Schedule flag is not set, your node will either not start or automatically apply the hardfork at startup causing the node to fork from the rest of the network.

The following op-node/v1.12.2 adds a kill-switch to op-node to print an error at startup if the Pectra Blob Schedule Fix time is not set for a Sepolia or Holesky chain. The check only happens if the chain's genesis is before the Holesky/Sepolia Pectra activation time. The check can be disabled with a hidden flag.

The purpose of the kill-switch is to make sure that node operators don't accidentally upgrade their nodes without scheduling the fix because most Holesky and Sepolia chains were running a sequencer that ran into the Pectra Blob Schedule bug. So it's better to not start op-node in such cases at all rather than accidentally forking the chain.

The op-node/v1.12.1 and op-node/v1.12.0 binaries do not have this kill-switch and will automatically apply the hardfork at startup if there is no Pectra Blob Schedule Fix time set.

op-node at v1.12.2 (opens in a new tab)
op-geth at v1.101503.1 (opens in a new tab)

Update the hardfork activation time

⚠️

If you are operating a node for an OP Chain that has opted into the hardfork activation inheritance behavior (opens in a new tab), the Pectra Blob Schedule Fix activation date is part of the op-node node. So, no action is needed for the sequencer after upgrading to the latest release, assuming you're using the network flags.

That is: OP Sepolia, Soneium Minato, Zora Sepolia, Unichain Sepolia, Base Sepolia, Mode Sepolia, Metal Sepolia, Creator Chain Sepolia, Ink Sepolia, and Ethernity Sepolia.

For chains that are not opted into the hardfork activation inheritance behavior, you will need to manually set the hardfork activation time. This can be done one of two ways:

Option 1: Set the activation time in the rollup.json for op-node. You do not need to set any configurations in op-geth.
Option 2: Set the activation time via overrides (CLI) in the op-node. These will need to be set on op-node for the sequencer and all other nodes. The override flag looks like this: --override.pectrablobschedule=1742486400.

Update the Superchain Registry

If your Pectra Blob Schedule fix activation time is not in the Superchain Registry, decide on a hardfork activation time, and open a PR update the Superchain Registry with the activation time. See this PR (opens in a new tab) as an example.

Fixing diverged nodes

If you upgraded your node binary to op-node/v1.12.1 or op-node/v1.12.0 and did not set the Pectra Blob Schedule Fix activation time your node will have forked from the rest of the network. This could have happened if you set the incorrect activation time e.g. set the wrong timestamp in your rollup.json, set the wrong timestamp in the op-node CLI, or did not set the timestamp at all.

To fix this, you can either restore your node from a snapshot and configure the Pectra Blob Schedule Fix activation time. Or you can use op-wheel (opens in a new tab) to roll back your node to a block before you diverged from the network.

Stop the node

Stop op-node: This component is stateless, so you can simply terminate the process. Stop op-geth: It's crucial to use Ctrl+C to stop op-geth gracefully to avoid database corruption. An unclean shutdown can lead to various problems when restarting.

Query `op-geth` for a block to reorg to

cast block --rpc-url=http://localhost:8545 6185373
#hash: 0x8b39f479198a5e658bbb28b1ce80790ca863ac39206c36152ba5c38db68989b2
#timestamp: 1735565082
# 6185373 in hex is 0x5E619D

Reorg the chain

cast rpc --rpc-url=http://localhost:8545 debug_setHead 0x5E619D

Explicitly set the forkchoice to the new head

For example:

go run ./op-wheel/cmd engine set-forkchoice --engine=http://localhost:8551 --engine.jwt-secret=.jwt_secret.txt --finalized=6185373 --safe=6185373 --unsafe=6185373