The Vasil hard fork finally has a fixed date of September 22. This milestone will introduce a lot of new functionality to Cardano. One thing that critics notice is the planned date is 3 months after it was expected to go live in June. As of this writing, more than 80% of Stake Pool Operators are already running on the latest Cardano version which supports the hard fork (v1.35.3). These numbers had gone up steadily in the last two weeks.
You might be wondering why did a lot of operators take a long time to upgrade their setup. Should this not take just a day or two like in previous upgrades?
Actually, compared to past upgrades, there are more factors which made this hard fork a lot more time consuming and resource intensive. Just what are these factors that caused delay? Please read on to find out
1. The Bug
It is normal for developed software to have bugs that need to be fixed am Cardano is no exception. The issues that popped up in the past were mostly minor. However, going into Vasil, one of the recent versions had a “catastrophic” bug that slipped through the rigorous testing and caused the Cardano Testnet to be “broken”. Fixes need to be made and new test environments setup before the hard fork can proceed.
2. Hardware Purchase/Setup
The new minimum requirement for running a Cardano node has been increased from 12Gb to 16GB. This along with the fact that SPO’s now need to spin up a Preprod and a Preview environment meant that a lot of SPO will need to replace or upgrade their existing hardware.
Purchasing hardware is not easy as SPO’s first need to save up to be able to afford the purchase and when they finally order, there may be a delay in delivery if there is no stock.
3. Need to migrate production
SPO’s running on cloud infrastructure may have the option to upgrade their memory and hard disk seamlessly if their cloud provider allows it. Those running on bare metal however are not so lucky as upgrading existing servers means there will be downtime. This can be mitigated if new replacement servers are bought and setup so that installation and testing can be done without affecting the current production systems.
When the new servers are ready, a quick switch in configuration followed by a service restart can bring the downtime down to minutes. Nevertheless, the amount of research and preparation needed in order to prepare for this short outage is huge.
4. New Environments
Two new environments need to be spun up to replace the old Testnet environment which is now broken. These are:
- Pre-Production Environment which is maintained at the same version as the production environment
- Preview Environment which is for testing new functionality
Each of these environments will require at least 2 servers. As these are very new environments, previously the documentation on how to set this up has been very sparse. Thankfully, it has improved a lot since then. Nevertheless, setting up these environments requires a lot of time and effort. Servers need to be setup and software installed. Scripts need to be written and rewritten for the new environments. Note that these environments are created from scratch and before you can even do any testing, these should be in place and working correctly.
Other things that need time to complete include:
- Upgrade third party tools like cncli
- Delegation request and waiting 2 epochs for delegation to be active
5. Proper Testing
A reckless SPO can skip all the environment setup and testing and just immediately upgrade Mainnet. They may not even need to upgrade any hardware if their current Mainnet specs is already high. There may be reasons for them to do so (usually economic reasons) but this is not recommended.
The correct way is to setup the new environments and do rigorous testing to ensure that the changes will not affect your current environment. Some things to check include:
- P2P topology – this allows peer discovery
- Leaderlogs – to see your block producing schedule
- Block production – ensure that servers are able to produce blocks
- Airgap machine – make sure this can work with new environments
- Special Capabilities – some SPO make use of extra functionalities like smart contracts and NFT minting. These should be tested
In summary, in the words of IOG itself, the Vasil hard fork is the most ambitious Cardano upgrade to date. Your stake pool operators are all hard at work to prepare for it and make it happen. Therefore, if your SPO is giving you regular updates about their preparations for Vasil, please be patient and stay loyal to them. I personally like SPO’s who takes the trust given to them seriously and does proper (albeit slow) testing. I’d take that SPO any day over an SPO that upgrades immediately without proper testing.
In other news, we have just decided to select a new educational charity to support. The name of the charity is Children’s Wishing Well (not related to WISH Pool in any way. They are a charity established in 2002 supporting the educational and emotional needs of disadvantaged children. We have been looking into this charity for some time now given that their name is very similar to ours and they also support disadvantaged children. After having met the students that they support recently, we decided to support them. We like that the funds we give will directly help disadvantaged children and not get mixed with funds for children who don’t need financial assistance.
WISH Pool is the creator of Cardano SPOT Check, an online resource for Cardano Stake Pool Operators for maintaining their nodes. We are part of the Ardana ISPO and were a recipient of the IOG delegation in 2021 and the Cardano Foundation delegation in 2022. If you like this article, we greatly appreciate if you will delegate your ADA to WISH Pool. You will be earning interest rewards while helping the lives of disadvantaged children. This is because we pledge to give at least 10% of our own profit to educational charities so that successful students can help get their families out of poverty. Let’s help grow the community together!