Emerging threat: ci/cd vulns

Ecosystem
1

EMERGING THREAT: CI/CD VULNS

TL;DR

To enhance the security of your CI/CD environment, please adhere to the following guidelines:

  1. Branch Protection and Auto-Merge Controls: Enforce branch protection rules for sensitive code and limit auto-merge features to prevent unauthorized changes from being deployed without review.

  2. Deployment and Artifact Management: Require additional approvals for production deployments and ensure that only pre-approved artifacts proceed through the pipeline.

  3. Identity and Access Management: Conduct continuous identity analysis, optimize permissions, manage stale accounts, unified identity management, regularly review external access, enforce organizational email use, control system registrations, selectively assign permissions, and use dedicated accounts for specific roles.

  4. Dependency Management Security: Use internal proxies for fetching third-party packages, disallow direct external fetches, enforce checksum and signature verifications, control package versions, manage scope for private packages, isolate installation scripts, include secure package manager configurations, and avoid publicly listing internal projects.

  5. Pipeline Execution Security: Isolate execution environments, especially for unreviewed code; enforce controls over external contributions; secure CI configuration files; restrict SCM repository permissions; apply the principle of least privilege; maintain updated and patched execution nodes; implement network segmentation; and isolate execution contexts for installation scripts.

Introduction:

Continuous Integration/Continuous Deployment (CI/CD) has become a fundamental practice in the dynamic field of software development, allowing teams to automate code change deployment and testing. This methodology makes sure that code updates are constantly merged, tested, and prepared for deployment at any time, which not only speeds up the development process but also improves the overall quality of the software. But CI/CD is not without its difficulties, as every technology development is, especially when it comes to security flaws.

Organizations are exposed to a distinct set of security risks when CI/CD pipelines are integrated into software development processes. These risks can result in serious breaches, compromised systems, and data loss if they are not adequately controlled. Organizations must implement strong security measures since CI/CD pipelines are automated and networked, which means that a single vulnerability can spread through the system at an alarming rate and scale.

All skill levels of adversaries are turning their focus to CI/CD because they understand that these services offer a quick and easy way to get to an organization’s most valuable assets. The number, frequency, and severity of incidents and attack vectors targeting vulnerabilities in the CI/CD ecosystem are significantly increasing, as observed by the industry.

The purpose of this forum article is to highlight excellent practices for risk mitigation while shedding light on the vulnerabilities present in CI/CD procedures and their possible effects. Organizations may improve the security of their software solutions and the safety of their development processes by being aware of the vulnerabilities that can jeopardize CI/CD pipelines. This forum post will be in 2 parts because the topic is huge and it will be beneficial for readers to understand.

Vulnerability Types:

According to OWASP, top ten vulnerabilities are like below:

  • CICD-SEC-1: Insufficient Flow Control Mechanisms
  • CICD-SEC-2: Inadequate Identity and Access Management
  • CICD-SEC-3: Dependency Chain Abuse
  • CICD-SEC-4: Poisoned Pipeline Execution
  • CICD-SEC-5: Insufficient PBAC
  • CICD-SEC-6: Insufficient Credential Hygiene
  • CICD-SEC-7: Insecure System Configuration
  • CICD-SEC-8: Ungoverned Usage of 3rd Party Services
  • CICD-SEC-9: Improper Artifact Integrity Validation
  • CICD-SEC-10: Insufficient Logging and Visibility

CICD-SEC-1: Insufficient Flow Control Mechanisms

According to OWASP, Insufficient flow control mechanisms refer to the ability of an attacker that has obtained permissions to a system within the CI/CD process (SCM, CI, Artifact repository, etc.) to single handedly push malicious code or artifacts down the pipeline, due to a lack in mechanisms that enforce additional approval or review.

In simpler terms, insufficient control mechanisms for code or artifact deployment can result in the dissemination of malicious code or unauthorized access within the CI system. A notable example of this vulnerability was observed in the PHP git repository, where attackers managed to inject unreviewed, malicious code directly into the PHP main branch.

A malicious user pushed changes on PHP source code which allows malicious users to run commands on affected build (PHP-8.1.0-dev) by using zerodium as User-Agent.

Based on the incident report, the root cause of the git.php.net security incident was the unauthorized use of HTTPS and password-based authentication to push malicious commits, circumventing the gitolite infrastructure.

To ensure robust pipeline flow control and prevent unauthorized shipment of sensitive code and artifacts, the following measures should be implemented:

  1. Branch Protection Rules: Configure branch protection rules for branches containing production and sensitive system code. Minimize exceptions for user accounts or branches from these rules. Ensure that accounts permitted to push unreviewed code do not have the ability to trigger connected deployment pipelines.
  2. Auto-Merge Rules Limitation: Restrict the use of auto-merge rules to the least possible contexts. Conduct a thorough review of auto-merge rule code to prevent bypassing, and abstain from incorporating third-party code in the auto-merge process.
  3. Deployment Pipeline Controls: Prohibit accounts from initiating production build and deployment pipelines without additional approval or review.
  4. Artifact Flow Control: Only allow artifacts created by pre-approved CI service accounts to progress through the pipeline. Require secondary review and approval for artifacts uploaded by other accounts before they can proceed. Also ensure that binaries/artifacts are not changed - check integrity of binaries and use some framework for entire process like “in-toto”
  5. Drift and Inconsistency Prevention: Actively detect and mitigate discrepancies between the code in production and its CI/CD source. Adjust any resource exhibiting drift to maintain consistency.

CICD-SEC-2: Inadequate Identity and Access Management

According to OWASP, Inadequate Identity and Access Management risks stem from the difficulties in managing the vast amount of identities spread across the different systems in the engineering ecosystem, from source control to deployment. The existence of poorly managed identities - both human and programmatic accounts - increases the potential and the extent of damage of their compromise.

Because there are so many identities in the CI/CD ecosystem—both human and programmatic—along with weak identity and access management procedures and a propensity to use accounts with too many permissions, the result is a situation in which virtually any user account on any system can grant substantial privileges within the environment.

A good example of this issue is the situation encountered by Mercedes-Benz, where the organization’s publicly accessible GitLab infrastructure permitted the registration of unauthorized users. Consequently, because there were not enough checks on email confirmation, this failure allowed for the unauthorized registration of an account on the company’s official GitLab server using a non-existent Daimler corporate email address.

Following countermeasures should be implemented in order to secure systems:

  1. Continuous Identity Analysis: Conduct ongoing analysis and mapping of all identities across every system within the engineering ecosystem, documenting the identity provider, assigned permissions, and actual permissions utilization for each identity. Ensure the inclusion of all programmatic access methods in this analysis.
  2. Permission Optimization: Systematically remove unnecessary permissions for each identity across various systems to align with their current operational requirements.
  3. Stale Account Management: Establish a defined period of inactivity after which accounts are considered stale. Proactively disable or remove any identity that exceeds this period of inactivity to maintain security hygiene.
  4. Centralized Identity Management: Avoid the creation of local user accounts; instead, leverage a centralized Identity Provider (IdP) for creating and managing identities. For any necessary local user accounts, ensure that those no longer needing access are promptly disabled or removed, and security protocols are consistent with organizational standards.
  5. External Collaborator Access: Regularly assess all external collaborators, ensuring their access adheres to the principle of least privilege. Where feasible, assign permissions with a specific expiration date for both human and programmatic accounts, revoking access upon completion of their tasks.
  6. Domain-Based Email Requirement: Prohibit the use of personal or external domain email addresses for registration on SCM, CI, or any CI/CD platforms, enforcing the use of domain addresses owned by the organization. Actively monitor and remove users not complying with this policy.
  7. Controlled System Registration: Eliminate the ability for users to self-register on systems, allocating access strictly based on necessity.
  8. Selective Permission Assignment: Refrain from granting universal base permissions across systems to all users or to large automatically assigned groups, favoring a more targeted approach to permission allocation.
  9. Dedicated Account Usage: Eschew the use of shared accounts in favor of creating dedicated accounts for individual contexts, assigning only the specific permissions necessary for those contexts.

CICD-SEC-3: Dependency Chain Abuse

According to OWASP, Dependency chain abuse risks refer to an attacker’s ability to abuse flaws relating to how engineering workstations and build environments fetch code dependencies. Dependency chain abuse results in a malicious package inadvertently being fetched and executed locally when pulled.

The management of dependencies and external packages, essential to internally developed code, has grown considerably in complexity due to the multitude of systems engaged across an organization’s development environments. This diverse ecosystem necessitates robust management strategies to ensure compatibility, security, and efficiency in the use of external packages.

In the context of dependency management, the primary security threats include:

  1. Dependency Confusion: This involves the malicious act of publishing packages in public repositories under the same name as internal packages. The objective is to deceive the dependency management clients into downloading these malicious packages instead of the intended, private ones.
  2. Dependency Hijacking: This vector targets the accounts of package maintainers on public repositories. Attackers aim to seize control of these accounts to upload malicious versions of widely utilized packages, thereby endangering clients who automatically update to the latest version of these packages.
  3. Typosquatting: Attackers publish malicious packages that bear a striking resemblance in name to popular packages. This strategy banks on the possibility of a developer mistyping a package name, leading to the accidental download of the malicious, similarly named package.
  4. Brandjacking: This tactic involves the publication of malicious packages that mimic the naming conventions or other distinctive features associated with a particular brand’s packages. The goal is to mislead developers into downloading these packages under the false impression that they are associated with a trusted brand.

A learning example of this type of vulnerability is the Ledger NPM hijack. December 14th, 2023, Ledger experienced an exploit on Ledger Connect Kit, a Javascript library to connect Web sites to wallets.Malicious actors have changed connect-kit javascript library, resulting in total loss of over $850,000. Malicious actors were successful because applications were not using versioning and hash control on these packages.

The following measures should be implemented, in order to not to affected by any dependency chain abuse :

  1. Internal Proxy Usage: Ensure that all third-party packages are fetched through an internal proxy instead of directly from the internet. This facilitates the implementation of additional security layers at the proxy and provides capabilities for investigating package-related security incidents.

  2. Disallow Direct External Fetches: Prohibit the direct pulling of packages from external repositories. Configure clients to exclusively fetch packages from internal repositories that host pre-vetted packages, and establish mechanisms to verify and enforce this configuration.

  3. Checksum and Signature Verification: Activate checksum and signature verification processes for all pulled packages to validate their integrity and authenticity.

  4. Version Control: Refrain from automatically pulling the latest version of a package. Instead, opt for pre-vetted specific versions or version ranges. Employ framework-specific methods to lock package versions to those that are stable and secure within your organizational context.

  5. Scope Management:

  6. Register all private packages under the organization’s designated scope.

  7. Ensure code referencing private packages includes the package’s scope.

  8. Mandate that package fetching under the organization’s scope is performed solely from the internal registry.

  9. Installation Script Isolation: When executing installation scripts as part of package installation, ensure they run in a separate context that lacks access to secrets and other sensitive resources available during other stages of the build process.

  10. Configuration File Inclusion: Include package manager configuration files (e.g., .npmrc for NPM) within the project’s code repository to prevent the use of potentially insecure configurations on clients fetching the package.

  11. Avoid Public Listing of Internal Projects: Do not publish names or details of internal projects in public repositories to prevent exposure to potential threats.

CICD-SEC-4: Poisoned Pipeline Execution

According to OWASP, Poisoned Pipeline Execution (PPE) risks refer to the ability of an attacker with access to source control systems - and without access to the build environment, to manipulate the build process by injecting malicious code/commands into the build pipeline configuration, essentially ‘poisoning’ the pipeline and running malicious code as part of the build process.

The “poisoned pipeline” attack vector necessitates that a threat actor acquires Source Code Management (SCM) permissions, such as user credentials or access tokens. These permissions enable the manipulation of Continuous Integration (CI) configuration files or similar content, facilitating unauthorized pipeline activities.

To successfully execute such an attack, perpetrators must also be capable of altering these files without initiating a review process.

PPE attacks are categorized into distinct types based on the method of execution:

  • Direct (D-PPE): In this scenario, attackers directly modify CI configuration files that are co-located with the targeted projects. This direct approach allows for immediate and specific manipulation of the pipeline’s operations.
  • Indirect (I-PPE): This method involves the injection of malicious code into files that are indirectly invoked by the pipeline configuration files. Such an approach can obscure the malicious activity from direct observation, making it harder to detect.
  • Public (P-PPE/3PE): This category requires attackers to gain access to repositories that host pipeline configuration files. Access can be obtained through credentials acquisition or by being granted permissions. Additionally, public projects may be compromised via pull requests if they incorporate and execute unreviewed code, providing another avenue for attack.

The following measures should be implemented, in order to not to affected by any poisoned pipeline execution :

  1. Isolation of Execution Environments: Execute pipelines running unreviewed code on isolated nodes. These nodes should be segregated from the production environment and should not have access to sensitive data or secrets. This minimizes the risk of attackers exploiting unreviewed code to access or steal sensitive information.

  2. Control Over External Contributions:

  3. Carefully evaluate the necessity of automatically triggering pipelines for code from public repositories, especially those made by external contributors. It’s advisable to avoid running CI/CD pipelines on code from forked repositories without thorough scrutiny.

  4. Implement controls like manual approval for pipelines triggered by external contributions. This adds a layer of human review that can catch malicious changes before they are executed.

  5. Branch Protection and Pipeline Triggers:

  6. For branches that trigger sensitive pipelines (i.e., those with access to secrets), ensure there are corresponding branch protection rules. This means only approved changes can trigger the pipeline, reducing the risk of malicious code execution.

  7. Branch protection rules can include requirements for pull request reviews, status checks, and restrictions on who can push to the branch.

  8. CI Configuration File Security:

  9. The CI configuration file, which dictates how the pipeline runs, is a critical security point. Ensure it is reviewed by a trusted party before any pipeline runs, to prevent unauthorized modifications.

  10. Alternatively, store the CI configuration in a separate, protected branch from the code it builds. This way, changes to the pipeline configuration can be controlled and monitored separately from code changes.

  11. Restriction of SCM Repository Permissions: Regularly audit and restrict access to the Source Code Management (SCM) repository. Only users who need to contribute to the repository should have access, and their permissions should be limited to what is necessary for their role.

  12. Principle of Least Privilege for Credentials: Each pipeline should only access the specific credentials it needs to operate. These credentials should be limited to the least privileges necessary to perform their function. This reduces the potential impact of credential compromise.

CICD-SEC-5: Insufficient PBAC

According to the OWASP, Pipeline execution nodes have access to numerous resources and systems within and outside the execution environment. When running malicious code within a pipeline, adversaries leverage insufficient PBAC (Pipeline-Based Access Controls) risks to abuse the permission granted to the pipeline for moving laterally within or outside the CI/CD system.

CI/CD relies heavily on pipelines. Nodes running pipelines execute the commands given in the pipeline configuration, carrying out a variety of delicate tasks in the process:

  • Get the source code, compile it, and test it.
  • Acquire secrets from a variety of sources, including environment variables, vaults, and specialized cloud-based identification services (like the AWS metadata service).
  • Produce, alter, and implement artifacts.

The word PBAC describes the environment in which every pipeline, including its individual steps, operates. It is essential to restrict each pipeline to the precise set of data and resources it requires access to, given how sensitive and vital each pipeline is.

  • Access within the pipeline execution environment: to code, secrets, environment variables, and other pipelines.
  • Permissions to the underlying host and other pipeline nodes.
  • Ingress and egress filters to the internet.

The following measures should be implemented, in order to not to affected by insufficient PBAC :

  1. Node Isolation for Sensitive Pipelines: Utilize dedicated nodes for pipelines with varying levels of sensitivity or differing resource requirements. Only pipelines with identical security and confidentiality requirements should share execution nodes. This precaution prevents the potential cross-contamination of sensitive data.
  2. Secrets Management: Scope secrets precisely to ensure that each pipeline and its constituent steps have access exclusively to the secrets necessary for their operation. This targeted access control prevents unauthorized access to sensitive information across different stages of the CI/CD process.
  3. Node Reversion Post-Execution: After the completion of a pipeline execution, reset the execution node to its original, pristine state. This measure eliminates any residual data or configurations that could compromise future pipeline executions.
  4. Principle of Least Privilege: The operating system user responsible for running the pipeline jobs should be assigned permissions strictly necessary for the tasks at hand. This principle minimizes the risk of unauthorized actions or access in the event of a security breach.
  5. Restricted Permissions on Controller Nodes: Limit the permissions of CI and CD pipeline jobs on the controller node. Consider executing pipeline jobs on a separate, dedicated node to further isolate and protect critical system components.
  6. Regular Patching of Execution Nodes: Ensure that execution nodes are regularly updated and patched to protect against vulnerabilities. Timely application of security updates is crucial for defending against exploits targeting outdated software components.
  7. Network Segmentation: Implement network segmentation to restrict the execution node’s network access solely to the resources necessary for the job. Avoid providing build nodes with unrestricted internet access, minimizing exposure to external threats and reducing the attack surface.
  8. Isolated Execution Context for Installation Scripts: When executing installation scripts as part of package installations, create a distinct execution context for these scripts. This context should not have access to secrets or sensitive resources available to other stages of the build process, thereby safeguarding critical data throughout the pipeline.

Sources:

3 Likes