Hi, I would like to share some feedback on the governance process surrounding SIMD-123 and SIMD-228. I focus on our engineering operations at Chorus One, and I don’t follow every community proposal closely, but because the governance process required an exception to our key access policies, I would like to share my experience from that point of view.
I would like to thank the people who put a lot of effort into organizing the SIMD-123 and SIMD-228 votes, especially Laine. I hope that sharing my thoughts here will help to take the process for future votes to the next level.
The status of this process and the tooling around it was unclear to me
There is a spectrum of how ‘official’ a voting process is.
- An example of a very official and mature process is governance on Cosmos SDK-based chains. Governance is built into the chain as a first-party feature. It’s integrated with block explorers and wallets. The governance module was enabled by the chain authors, so it’s clear that they want it to be there, they endorse it, and they intend to respect the outcome of the process. In some cases, the outcome can even be directly enforced on-chain.
- An example of a very unofficial process, is a poll on Twitter or Discord, started by a prominent person in the community. This is very useful to gauge sentiment, it does carry some weight, and it can be used to inform implementation decisions for node software authors or a foundation, or as a stepping stone to a more official proposal, but nobody expects such a poll to be binding.
The governance process for SIMD-128 and SIMD-228 fell somewhere in between, which left me confused. I saw conflicting information about whether this was an official process set by the Solana Foundation and node software implementers, or whether it was fully a community initiative. It was also not clear to me whether this was an advisory vote, or whether node software implementers are committed to respecting the outcome of the vote.
Of course, not every chain can have a mature and streamlined process from the start. Especially in the beginning, having an official governance process is not a priority, and improving the process is a journey and learning process. I think Solana is now at a point where it could benefit from more clarity around the process, and that’s why I am sharing this feedback here.
In particular, the things that left me confused are:
- I could not find any documentation in an official place that explains how the governance process works, or that one even exists in the first place. I would expect to see a section about that in a place like https://solana.com/docs. I later learned about the existing
spl-feature-proposaltool, and documentation at https://spl.solana.com/feature-proposal (which is not very discoverable, e.g. search on solana.com does not find it when you search for ‘governance’), however the process for SIMD-123 and SIMD-228 is different from the one documented there. - The proposals were announced here on the community forum, by community members. They were not announced by the Solana Foundation in an official place, like a blog post on solana.com.
- The voting process depends critically on a third-party tool developed and hosted by a community member, which is a fork of a tool developed originally by Jito. If this were an official process, I would expect the tooling for it to be at least hosted in an official place, such as https://github.com/solana-program — or ideally — have it be part of the
solanacommand-line program. - These third-party tools were endorsed on Twitter by several individuals, e.g. by Anatoly who prominently pinned the tweet, and by Dan Albert.
- However, the
@solanaTwitter account did not tweet about it at all. (If it did, I couldn’t find it among the many unrelated tweets.) - The
@SolanaFndnaccount did not tweet about it, however it did repost the tweet by Dan Albert without additional comment. It is not clear to me whether that repost means “this is the official process” or “this is a cool community initiative that we want to highlight”. - The
@anza_xyzaccount did not tweet about it, nor repost or mention anything related to the SIMDs. - There was an announcement on Discord in
#validator-announcementspointing to the third-party tool and dashboard. - All links I saw on Twitter and Discord point to readmes in the master branch of https://github.com/laine-sa/solgov-distributor. The contents of those pages, and the tool itself, are mutable, and can be changed by Laine unilaterally. Now of course Laine is a reputable community member who I trust to not do that, but an official process should not have critical dependencies on external third parties like this, and when I first saw it, it made me question the legitimacy of the process.
- Although an earlier forum post from 2023 discusses a voting process that is clearly labelled “advisory”, in the forum post for SIMD-288, the word ‘advisory’ does not occur, and the instructions around the voting process look rather tightly organized.
The impression I got from all of this, is that these votes are an unofficial community initiative, that at best serve to inform Solana Foundation and node software authors, but without any commitment about whether the outcome will be respected. I thank the organizers of the process for all the hard work they put in to build and coordinate this, and I have a lot of respect for Laine always being present to help out in #mb-validators on Discord. This vote is clearly not some random Twitter poll, the process definitely carries some weight. However, it does not look like an official process to me, and the lack of documentation and communication from official accounts and websites left me wondering how seriously we should take this vote, and how much time we should invest in ensuring that we can vote without putting our private keys at risk.
I suggest the following clarifications for future governance proposals:
- Ensure that all the required tooling is hosted under an official organization, such as https://github.com/solana-program, or better yet, built into the
solanacommand-line program. - Add a section about governance on https://solana.com, to explain that a process exists, and how the process works.
- Although the SIMD itself can of course be written by community members, when it is time to vote on it, the voting process and instructions on how to participate should be announced in a way that is clearly coming from Solana Foundation. For example, through a post on https://solana.com/news, or through a repository on GitHub under the
solana-foundationorganization, with Discord announcements, and posts by@solanaand@SolanaFnDnlinking to it.
Security considerations of the voting tool
Voting requires a signature from our validator identity account. This makes sense, it is what authenticates us as a validator. The identity account is the most sensitive key we have as a validator, keeping it secure is paramount. Obviously, when a third-party tool requires access to it, we don’t blindly run that — we stop to think, “Hmm, what is this tool? Can we trust it? Is there a risk of compromising our private keys?”.
If such a tool would be part of the regular Solana Program Library repository, or included in the Agave repository, then that gives some assurance that it’s an official tool that went through the same review process as the other software that we already trust with our private keys. In this case though, the tool was hosted by a third-party community member, which was a surprise to me. Again, Laine is a reputable community member that I’ve seen making valuable contributions for many years; I don’t think they would on purpose try to steal or leak other validator’s private keys. However, we treat third-party tools with even more caution than we would treat official tools, especially when they require simultaneous access to our most sensitive private key, as well as the Internet.
Because all links were to this third-party repository, the owner of that repository in principle has the power to unilaterally change the instructions, or change the tool itself. Again, this is something that I trust Laine to not do, but if this is an official process, then it would be better for the process to be guarded by mechanical access controls (like a repository where only members of the Solana Foundation have push access), rather than having to rely on trust.
We did audit the source code in https://github.com/laine-sa/solgov-distributor and found no trace of it doing unexpected things with the private key. However, this repository also comes with a 4531-line Cargo.lock file that pulls in 429 external libraries. Any of those could include a build.rs that targets the solgov-distributor to mess with it, or include malicious code in a different way. The recent liblzma backdoor attempt shows that such supply chain attacks are no longer hypothetical. One might think, well, is this small community tool going to be the target of an attack like that? But this vote is happening at a time when there is renewed attention to blockchains; interest in Solana was at an all-time high, and token prices were booming. Getting your hands on some of these identity accounts would certainly be very lucrative.
There are a few things that alleviate this particular concern about supply chain attacks: many of the dependencies are shared by Solana/Agave itself (though different versions), and the lockfile has not been touched for two years (according to Git metadata, which can easily be spoofed, although that would be malicious behavior from the repository author which I think is unlikely here). However, security is a domain where failing to find issues does not prove their absence, and I would prefer a solution that categorically eliminates risk.
What we ended up doing, is splitting out the part of the tool that generates the transaction, and writing a small utility ourselves to sign and submit the transaction. We could then run the transaction generation without access to our private keys, and we trust our own signing utility. Also, it could not have been the target of a targeted supply chain attack, as it didn’t exist before.
Let’s take a step back though. Why is the solgov-distributor tool needed to vote at all? Couldn’t the initiator of the process just distribute SPL tokens to all validators? The validators could use the regular spl-token CLI to vote, and no additional tools would be needed for validators. From what I understand about https://spl.solana.com/feature-proposal, that is also how the spl-feature-proposal tool is designed to work in the first place.
User experience of the voting process
The experience of running command-line tools and verifying and pasting addresses is a bit rough. Nothing that should be difficult for a professional validator, but in practice, as @Umberto on our end discovered, a significant amount of tokens for SIMD-123 were sent to the ABSTAIN address for SIMD-228. I don’t blame anybody for this, UX is always rough initially, but clearly there is room for improvement here.
Use of vanity addresses
Vanity addresses are a controversial topic, and I am aware that our main Chorus One validator is using a vanity address, but lately I’ve come to realize that vanity addresses are harmful. Vanity addresses can be generated by anybody; a particular prefix or suffix provides zero authentication or legitimacy. Humans should not rely on them. But humans are lazy, and so they will.
In the context of this governance vote, consider e.g.
spl-token accounts --owner ABSTA1Nsimd22811111111111111111111111111111 --url https://api.mainnet-beta.solana.com
At the time of writing, it outputs:
Token Balance
--------------------------------------------------------------
s1233un3oMnNWo4EuKsEjtV7QtXfimhybHMQP7GLPwM 279728857811751
s228VmFcuiEfroSCQTvEp1pYUownL7JRZMTd7FqHJVK 8415997563262940
It is possible for anybody with a bit of SOL to start their own token with an address that starts with s228Vm, and send a similar amount of tokens as currently listed to the ABSTAIN address, to create confusion about what the real token address is. In fact, you could generate dozens or hundreds of tokens, and completely drown the true address among the fake ones, in an address poisoning attack.
This does not affect the outcome of the vote, and real vote tallying tools like the one developed by Staking Facilities should work with full addresses. But if working with full addresses (and only copying them from official announcements!) is needed anyway, then let’s not tempt humans into taking shortcuts. If we want better UX, the only secure way is to build that into the voting tools itself, and not by abusing vanity addresses.
Conclusion
I think it’s great that governance on Solana is evolving, and I would like to thank the people who organized the SIMD-123 and SIMD-228 votes for taking the lead on this. For future votes, on the communication side I think we can create more clarity about what the status and goal of the process is, and on the implementation side, I think there is room to make the tooling more secure and more streamlined.