Security Audit Log and Tracking

1

AppSec and engineering teams have been discussing and working on what pieces of code have been audited (and vice versa) to make it clear for the benefit of the ecosystem. Please find the AppSec team’s proposal below and kindly provide your input.

In the Rust ecosystem, there are some advantages of using cargo-vet and cargo-audit, so the AppSec team has been keeping an eye on it. The idea of having a repository that Rust installations can consume to detect (un)audited packages while installing different pieces of Polkadot, Substrate and more; The logic behind it:

  1. We have often observed a redundancy in company audits, where multiple firms repeatedly scrutinize the same piece of code or software package to ensure its reasonable safety for use. For the sake of transparency, it’s crucial to distinguish between internal and external findings in the tracking and reporting of vulnerabilities. This distinction should be clearly communicated in a public statement from Parity, Ecosystem, or the auditing company, detailing whether the findings were identified by security experts or were part of a quality and performance review. This clarity is essential as both types of audits are important but serve different purposes. In the same vein, some of the findings are under Parity’s disclosure process. On the other hand, other projects such as ink!, have run public audits with OpenZeppelin with public results.

  2. With the monorepo and the introduction of semver for small feature packages, it is possible to implement following two ideas:
    a. Using cargo-vet to keep a track of specific crates of polkadot-sdk code being audited, as audits do not audit (not that we know!) the whole codebase, and probably will not be the case with the monorepo now.
    b. Using GitHub Security Advisories to publicly disclose known vulnerabilities (specially for legacy versions). This can help developers be aware of common issues in the development of polkadot-sdk. An example of this can be seen in the frontier repository. By sharing the vulnerabilities in GitHub, they will be assigned automatically a CVE ID and considering the vulnerabilities are constraints to specific versions of the respectives packages affecting, a developer will be able to use cargo-audit in their runtimes to check if any of the dependencies contains known vulnerabilities.

  3. This initiative might mark the beginning of a wider effort, not limited to Parity, aimed at maintaining and incorporating audit sources for the community, including Parachains and new projects. The goal is to establish a more robust and transparent audit verification process. The concept is to launch an audit repository, which would then be sustained by developers within the ecosystem.

9 Likes
2

Following the previous announcement, I’m pleased to share the AppSec proposal for tracking and logging Audits in the Polkadot ecosystem:

Security Audit Log and Tracking

Implementation Plan

The purpose of this document is to provide a detailed technical roadmap for implementing a robust security audit log and tracking system within the Polkadot ecosystem. It focuses on the organisation and diffusion of audit results and important findings in a manner that is seamlessly integrable with automated scanning tools such as Cargo Vet and Cargo Audit. This plan aims to establish a structured and efficient framework for monitoring and ensuring the security integrity of dependencies in Polkadot-related projects.

Audit Tracking

In this plan, Cargo Vet is selected as the primary tool for tracking audit results of Rust crates within the Polkadot ecosystem. This choice is informed by the successful adoption of Cargo Vet by leading institutions such as Mozilla, Google, ZCash, and Bytecode Alliance, which are recognized as trusted sources in the Cargo Vet registry. To effectively implement Cargo Vet in the ecosystem, a series of methodical stages will be outlined, ensuring thorough integration and optimal utilisation of its capabilities for maintaining the security and integrity of the software supply chain.

Stage 1: Tracking audits per repository

Organisations like Parity Tech, which offer widely-used crates, are to initialise Cargo Vet within their repositories. This process involves creating a ‘supply chain’ folder in each repository, used for audit tracking and composed of the following files:

  • audits.toml: This file documents all audited crate versions. It categorises audits into three types:

    • Version Audits: Complete assessments of a crate at a specific version.
    • Delta Audits: Assessments of changes between versions, crucial for maintaining a clear audit trail and useful for Security Fix Pull Requests assessments.
    • Violation Entries: Noting crates that fail to meet audit standards.

  • config.toml: Contains two main elements:

    • Audit Imports: Listing audits records collections from trusted sources like Mozilla, Google, and eventually Parity Tech and others within the Polkadot ecosystem.
    • Crate Exemptions: Listing unaudited but deemed safe crates, initially populated with all unaudited dependencies for smoother integration, but eventually reduce to contain only crates with low impact in the main crates.

  • imports.lock: An automatically generated file maintaining a record of fetched audit data, crucial for upcoming commits or PRs, as Cargo Vet will flag any newly added unaudited code.

There is mainly two types of audits, Full Version Audits and Pull Request Audits, the first ones are required for new crates or MAJOR version of a crate (see semver) and are tracked with as Version Audits. While the second one are required only for Pull Request that affect critical components and are recorded as Delta Audits or Violation Entries (if missing). In order to track Pull Request Audits, the following rules need to be enforced for Pull Request requiring an audit:

  1. If the Pull Request (requiring an audit) needs to be merged before audit finish, all the crates modified need to be recorded with violation in order to alert other developers that part of the code in the new version of the crate is not audited in case it continues being unaudited on version release. The violation record can be removed once the audit is complete.

  2. A Pull Request Audit is recorded as delta audit from the current version to the same with git (i.e. “1.0.0 → 1.0.0@82573…”). If another PR audit was recorded for the same version, the new one is added exactly after the latest one.

  3. In a new release, a crate can record a delta audit with the criteria safe-to-deploy only if there are no violation records of the crate in the current version. The latest PR audit record is cloned with some modifications:

  4. The ‘to’ version changes to the version of the release.

  5. The notes are removed.

For example, a Polkadot SDK Pull Request violation record will look as following:


[[audits.staging-xcm-executor]]

violation = "1.0.0 -> 1.1.0"

notes = "PR #1672"

While a record of a Pull Request could look like:


[[audits.staging-xcm-executor]]

who = "SRLabs <srlabs@srlabs.de>"

criteria = "safe-to-deploy"

delta = "1.0.0 -> 1.0.0@8257373b313fcfad4f7a2ab2cb02cb8bfc5b8f1"

notes = "PR #1672"

And a release audit would look like:


[[audits.staging-xcm-executor]]

who = "SRLabs <srlabs@srlabs.de>"

criteria = "safe-to-deploy"

delta = "1.0.0 -> 1.1.0"

notes = "PR #1672"

Designated teams within each organisation, such as the Application Security team at Parity Tech, will assume responsibility for managing Cargo Vet files. Their role includes adding new audits using ‘cargo vet certify’ and actively working to reduce the list of exempted crates by conducting thorough audits. This continuous effort ensures the security and integrity of the crates, contributing to the overall robustness of the Polkadot ecosystem’s software supply chain.

Stage 2: Tracking audits per organisation

For organisations like Parity Tech, with multiple repositories, a central ‘supply-chain’ aggregator repository is essential (Mozilla example). This repository, using ‘cargo vet aggregate’, will routinely (through cron jobs) collate all audit data from various repositories, storing trusted audits.toml files records from projects like ‘polkadot-sdk’ and ‘wasmi’. This centralization simplifies access for developers, allowing them to easily import a comprehensive set of audits into their projects, thereby streamlining the process of ensuring codebase security within the Polkadot ecosystem using Cargo Vet.

Stage 3: Vetting

Developers across the Polkadot ecosystem will leverage the audit records provided by the aggregator repositories. This enables them to verify the security of their systems by using the ‘cargo vet’ command in their repositories after initialized and added the ‘supply-chain’ aggregator sources. This is designed to ensure that the dependencies they use meet the rigorous standards established in the audits.

Vulnerability Tracking

Integrating Cargo Audit with GitHub’s Advisories vulnerability disclosure system enhances the security of all the projects in the ecosystem as it sets an standard way to communicate vulnerabilities. This approach involves two key stages: manually reporting vulnerabilities with Github Advisories and continuous monitoring of vulnerabilities with Cargo Audit.

Manual Vulnerability Disclousures

If vulnerabilities are detected and they require a disclosure according to the policy, they should be reported responsibly using GitHub’s advisory features. This process involves creating a confidential security advisory within the repository, collaborating privately with the security team or project maintainers, and finally disclosing the vulnerability publicly once a patch or update is ready.

The steps to vulnerability disclosure using GitHub’s Advisory system include:

  1. Creating a security advisory in the GitHub repository.
  2. Documenting the vulnerability details and potential impact privately within the advisory.
  3. Collaborating with GitHub’s security team or project maintainers to verify and address the vulnerability.
  4. Preparing a fix or patch for the issue.
  5. Publishing the advisory publicly once the solution is ready and the issues is patched, following GitHub’s coordinated disclosure guidelines. This approach ensures that vulnerabilities are handled discreetly and responsibly, maintaining project integrity and user trust.

Automated Vulnerability Detection

Integrating Cargo Audit with GitHub’s Advisory vulnerability disclosure system, as demonstrated by Polkadot projects like Frontier, enhances software security and manages vulnerabilities effectively. To do this, developers should incorporate Cargo Audit into their GitHub workflows using GitHub Actions. These actions, triggered on new commits and Pull Requests, automatically run Cargo Audit to identify vulnerabilities in dependencies.

Actions

A summary of all the actions to be performed in order to execute this plan:

Public good crates (i.e. Polkadot SDK)

  • Setup cargo vet in their repositories and put effort in keeping up-to-date records of all the audits executed.

  • Disclose vulnerabilities according to their organisation policy using Github’s Advisories.

Developers in the ecosystem

  • Use cargo vet in order to identify which dependencies of their systems (parachains, pallets, primitives, etc.) are audited. This is a preventive security measure, the goal is to help builders determine when they shouldn’t deploy something to production (i.e. critical components unaudited).
  • Use cargo audit in order to continuously identify if their systems are affected by a known vulnerability. This is a reactive security measure, employed by developers to determine quickly if they need to patch their systems in case a new vulnerability is disclosed.

Please feel free to leave any feedback or comments!

3 Likes
3

This idea makes sense to have code audit trace directly within the relevant codebase locations. Some ideas, as a start, could we not 1/ make this a requirement for those teams awarded an audit from the Polkadot Assurance Legion bounty (Approve Polkadot Assurance Legion Bounty | Polkassembly) and 2/ for audit vendors to raise a PR to include this record trace when they finish their audit?