Technical Game of Zones Update from Interchain GmbH

There are several goals of Game of Zones: to educate the public about IBC; to stress test the system in an adversarial and competitive setting; and to shine a light on any potential vulnerabilities in the IBC protocol and implementation. Earlier this week, participants in Game of Zones successfully helped us identify a problem. It wasn’t exactly a bug in IBC, but it was a problem nonetheless. We’re excited to fix it.

I’ll share more on that in a moment, but first, some context:

On Monday, the Iqlusion team, which sometimes also goes by “Cosmos Devs,” posted an update to the Game of Zones challenge. In short, the timeline has been adjusted to accommodate the aforementioned problem, which popped up at the end of Phase 1. (For more details on Game of Zones logistics, please see Iqlusion’s post.)

Today, I’m following up on that issue on behalf of the Interchain GmbH team, which includes the core development team for IBC. (We’re helping Iqlusion out as they administer Game of Zones.) I’ll share a deeper analysis of the bug and root cause, as well as the cross-entity next steps that we’ll be taking together, to ensure a smooth conclusion to Game of Zones, and, more importantly, a robust Cosmos Network.

The Bug

At the conclusion of Phase 1 of Game of Zones, participants were asked to upgrade their software to a new commit of Gaia and the Cosmos SDK, and restart their nodes. After doing so, a number of participants reported that their chains were halting, and other participants reported seeing “invalid port” errors.

The Berlin-based Interchain GmbH team started investigating on Monday morning CEST, and was able to identify the problem that afternoon and merge a patch. The team then worked with community members to verify the fix across several nodes. That night, the Iqlusion team updated the Game of Zones timeline.

Root Cause

The bug came from the way that the capabilities module was being initialized inside Gaia. It was erroneously being initialized with a cached context, which meant that the changes made during application initialization were not correctly persisted to the store.

We were able to identify this problem quickly, because it was, in fact, a known bug. It had actually been fixed in a patch for the Cosmos SDK, a week beforehand!

So how did this bug persist in Gaia, even after it was identified and patched in the Cosmos SDK?

Root-er Cause

We were able to pinpoint the buggy code, but we’ve also been able to find the bigger problem: As core development teams have decentralized across the Cosmos ecosystem, responsibility for maintaining Gaia hasn’t been clear. Although Gaia plays a key role as the daemon that runs the Cosmos Hub, it has a different software lifecycle from the Cosmos SDK and Tendermint Core. More concretely: Gaia is not updated as frequently, and changes to its dependencies may not get incorporated immediately.

In this case, this mismatch created trouble.

Next Steps

In the immediate term, Iqlusion will be responsible for porting SDK updates to Gaia. This arrangement will ensure that key fixes will be applied to Gaia, and that Gaia will be proactively maintained moving forward; not just during Game of Zones, but beyond, too. The Interchain GmbH team will help out here, too, by reviewing all Gaia releases and providing a second pair of eyes on this very important piece of software.

The Interchain GmbH team will also start helping out with relayer development, testing, and support during Game of Zones. The truth is that Game of Zones has ended up being much more popular than any of us expected! Our hands-on community is overwhelmingly a blessing, but y’all have made Game of Zones a bigger project than what one team can handle alone.

As I wrote in the beginning of this post: Game of Zones is an opportunity to unearth vulnerabilities in the IBC protocol and software; today we also found a gap in our ecosystem-wide organizational coverage and communication. And we’re patching these things, just as we’d patch our code. These changes will eventually help us continue Game of Zones in a robust and intentional manner, and it will ultimately help us launch IBC in a secure and stable way on the Cosmos Hub.

The Interchain GmbH team wants to thank everyone for their ongoing participation, and especially for their feedback, during this process. We’re looking forward to the next phase — not only of Game of Zones, but of the Cosmos Ecosystem. Thanks for working with us to make it as ✨celestial✨ as possible.