This is a complex topic but I would like to start the discussion on how can we better handle security vulnerability disclosure/advisory within the substrate and polkadot ecosystem (relaychain/parachain).
Currently, as I’m writing this post, there are some security vulnerabilities in Substrate that could be abused on some Parachains but have not been disclosed by Parity. I suppose they want to have it fixed/deployed in the relaychain before revealing them but I think it is a bad strategy for the ecosystem.
We should consider having a process in place to disclose those to impacted blockchains giving them the possibility to prepare a fix before it gets spread to the public (It is often easy for the public to find them once the fix is proposed on substrate repo)
In the case of Frontier, we have a private “security advisories” channel, including “trusted” (I know this part is very hard) people who play an important role in Frontier. So far it has been used for 2 critical vulnerabilities and allowed each parachain to get fixed before a public disclosure.
Maybe we could do something similar for substrate. Specially once we start working on the new FRAME repository that “should” not be maintained by Parity anymore.
I think we can take the first move by establishing a common channel that exclusively handles urgent security matters with Substrate or custom parachain implementations (which could affect other chains via XCMP). But since Substrate is an open framework, this method will not be scalable in the long run. I think we could perhaps make something like a CVE for Substrate as a long term solution.
Yeah, I’m in favor of improving the process. Currently it is too focused on the relay chain. I already brought this up internally that we need to improve on this. As you said, maybe we should start with some sort of internal channel where we more openly communicate these issues.
AFAIK there is nothing serious, that would open up some doors to steal funds or similar. Most of the stuff is mainly around dos vectors etc. However, even that should be shared with a selected group to make everyone aware.
Who should be in charge to create/maintain this channel ?
Concerning the vulnerabilities, I’m aware of one currently known by Parity that could be considered critical (not allowing to steal funds, but allowing to cheat some pallet to easil scam other people)
I don’t see how it would be possible to alert all teams without a public discussion - but I do understand the sensitivity of publishing alleged security vulnerability via the issues mechanism.
I would counter that this public visibility process is one of the fundamental reasons for open source code repositories existing in the first place (versus closed source of course), and as risky as making that knowledge public is - it is outweighed by the benefit of validating the issue and solving it early.
I think this conversation, where the issue is not described, highlights the fact that eyes on the code are reduced to a select opinionated few who may or may not be mistaken, and therein lies the rub. Of course I would like to know what this is because it may affect all our chains… or not.
I also agree that there should be some kind of group where each polkadot (&kusama) parachain is represented. In that group vulnerabilities could be disclosed before they are made public.
IMHO it’s also important to not announce them before enough time has passed to release the fix.
First of all parachains should get some time to mitigate the issue and depending on the governance model that might take some time. Second a big enough security issue could mean the parachain could get exploited. Together with XCM it becomes very hard to revert the damage that the exploit did, if not impossible.
Yes the proposed approach should be balanced having a dedicated forum room/channel in order these vulnerabilities to be discussed with representatives (to be identified) from each parachain .
Objective is to share as soon as possible in order to
maximise partnership on remediation as an ecosystem
and/or mitigation at parachain level.
Wider disclosure will require adequate time for the fix to be released upfront
My point was rather focussed on a future where there aren’t teams managing parachains but where full decentralised governance exists. Having a forum with special membership won’t work for that decentralised circumstance, because literally no single person would be in charge. It would therefore require some public disclosure.
It might be that certain developers may take on the responsibility within those communities, but event then they may just keep the information for themselves and make exploits, which even a closed forum cannot necessarily prevent from happening. That’s why I would say that a public forum is probably the only place where we can be sure that enough people would be aware.
I would also add that as a founder/lead on one of the parachains that may be affected by this, I have not received any responsible disclosure on this alleged vulnerability. That also seems very odd to me.
I don’t think a public forum would be a good idea (exploit would get abused before any governance could prevent it).
I also don’t agree that in the future parachains will be fully operated by the community. There will always be teams in charge of the implementing/maintaining ideas/features desired by the community. What is important is to ensure the incentive of the people receiving the disclosure is to fix the vulnerability instead of exploiting it (For exemple, having them to deposit a huge number of tokens that would motivate them to ensure they would prefer to keep the network safe, or even adding a slash mechanism if they exploit a vulnerability, but I doubt the last part is a valid)
It seems both reasonable and easy enough to establish a channel where the audit reports from Polkadot and parachains are forwarded to, and representative of each parachain team is present there to make sure action is taken.
That does raise the question of what to do with parathreads though. Or parachains that are now offboarded, but intend to come back. Should they still be kept in the loop?
Of course, this is all about the period of time before a vulnerability has been fixed. Once fixed on substrate (or whatever repo it belongs to) and x month have passed (assuming all parachains have updated during x months), then the issue and the patch should be publicly disclosed.
We can actually start this at Parity an implement the aforementioned process with the audit process that we have in place. We already have a backlog of all open and closed audit issues and disclosing the closed ones that are old enough is both great for educational purposes, and is aligned with the spirit of having an open community.
You are advocating for a closed source model which we know historically doesn’t work when it comes to trying to limit the impact of a security vulnerability. Just because you have a “trusted” private group, doesn’t mean your vulnerability cannot be found or exploited by third parties or indeed any member of the “trusted” group.
In my opinion the way Acala handled the recent bug is a model for how things should be handled. It was declared publicly even after the fact, and governance decisions were made - again after the fact - to fix and restore confidence. In fact the solution required that the problem be public in order to get buy-in from the community.
That is a bit utopian
(In the case of Acala aUSD issue, the vulnerability was discovered publicly, so it was already too late to hide it).
You can look at what happened with Nomad. As soon as the vulnerability became public, everything got drained instantly.
We’ve gone through multiple vulnerabilities related to the Ethereum/EVM (Frontier) recently and I believe the way all parachains (including Acala) kept it secret (through a dedicated channel for Frontier security) until the fixes were provided helped to prevent a disaster.
It is obviously not a bullet-proof mechanism, but it is still better than making it public or trying to deal with it alone.
Yeah, I’m also not convinced that everything should be opened directly. When there is some security vulnerability found it should be reported. Parachain teams then for example could pause all kind of transactions to not let someone abuse whatever kind of bug is there.
When there is something found publicly like with Acala, you can not hide it anymore.
Sounds good. But things appear to have stalled. I’ve encountered something I think may be considered a vulnerability. Something like the below would be useful right now.
Any chance of this being setup?
If so, maybe a forum administrator can set the following up and post a question and self-answer on StackExchange and I’ll exercise it shortly.
Vulnerability Reporting
Send a Discourse message to the @vulnerable group by clicking on a link.
This link can be added to a pinned thread in this forum. But, it can also be added in the StackExchange group as the answer to the question “How do I report and track Substrate vulnerabilities?”. Something like:
creates a thread labeled “YYYYMMDDHHmm” in the Category “Vulnerabilities”. The thread is read-write restricted to the reporter and @vulnerable members.
Adds other groups or individuals as the topic suggests or discussion evolves. Example: impacted blockchains.
The reporter is sent a message with any requirements or guidelines.
Maybe the link above can be pre-populated, saving an exchange delay.
Depending on the report:
If there is no vulnerability: the thread “YYYYMMDDHHmm” is renamed “NA-YYYYMMDDHHmm”, locked, and made public.
If the vulnerability is resolved: the thread “YYYYMMDDHHmm” is renamed “Resolved-YYYYMMDDHHmm”, locked, and made public.
Until resolved the thread is restricted to the reporter and @vulnerable members.
Hi all thanks for inputs, before talking about using Discourse or other tools which may be useful, happy to share the approach Parity has deployed and shared with the parachains.
Context
When a vulnerability is identified within Polkadot Ecosystem, what is the approach to follow about its disclosure within Parity, with the parachains, more widely to the Ecosystem
Situation
Vulnerability can be discovered by multiple different channels
Internally
A employee/contractor
By a security provider mandated to perform a review
Externally
A hacker directly via BugBounty
Via Parachains: directly or via their BugBounty scheme or auditors/pentesters
Via ecosystem
The fix to a vulnerability can take a certain amount of effort and time to be identified
Deployment of a fix can take a certain time to be rollout
From a risk perspective, longer a vulnerability exist and more people are aware more the likelihood of exploitation exists
Approach
In case Parity is made aware of a vulnerability which may impact the ecosystem, it is communicated with the following approach. Same reciprocal approach has been discussed with the parachains to be consistent.
When
If it is critical and Parity is not able to have it address within 5 days
If it is high and Parity is not able to have it address within 20 days
To whom
To the parachains ONLY to balance opportunity to have partnership to fix it and for parachains to deploy mitigation and in parallel to limit number of people informed who may potentially leverage the vulnerability to perform bad activities
When a Security audit firm is initially involved they may be part of the remedial discussion
The dedicated Vulnerability Disclosure Matrix channels (Polkadot and Kusama) are used to sync/inform with the different parties
NO larger communication to the community or more widely will be done, outside of exceptional situations which will be reviewed on an ad-hoc basis.
When the fix/mitigation is deployed by the Parachains and Parity, a global communication via a Post in the Forum
Reference
A guidance Vulnerability Disclosure template has been create to help capture key informations
A specific public index to track the history and full list of vulnerabilities is being created to facilitate access to everyone including new joiners to the community
What is the best channel to discuss disclosures? It is very likely the parachain team will need to clarify something or seek for additional help to confirm the remediation.
There is an ongoing effort to enable more feature for vulnerability disclosure process including Q & A channel. I expect improvements to be implemented before end of Q2 in 2024.
