This case study talks about how Google leverages StepSecurity’s GitHub Actions security platform to harden their GitHub-hosted runners and automate various GitHub Actions security best practices in several of their open-source projects.
Industry: Technology
Runners: GitHub-Hosted
Google has greatly contributed to the developer community over the years with its open-source projects on GitHub. These projects include machine learning frameworks, cloud computing tools, and more. With its projects, Google has not only empowered thousands of developers worldwide but has also proved its dedication to knowledge-sharing with the community.
CI/CD runs untrusted third-party software in a highly privileged environment that has access to source code, software builds, and elevated credentials. This can lead to supply chain attacks causing massive losses to enterprises.
This happens when compromised workflows, dependencies, and build tools exfiltrate CI/CD credentials, or tamper source code, dependencies, or artifacts during the build. Google needed to harden the GitHub-hosted runners on which their pipelines run to detect compromised dependencies and poisoned workflows.
Another major challenge faced by Google was compliance with GitHub Actions’ security best practices. There are various security misconfigurations in GitHub Actions and addressing each one of them for so many projects was turning out to be quite a laborious task for Google's developers. Google needed a solution that would ease this process and enable their developers to save time and be more productive.
Google needed to add OpenSSF Scorecard, Static application security testing (SAST) and Dependency review (SCA) workflows for their projects. With so many projects, it was a challenge for them to consistently add the same manually in each.
Google leverages StepSecurity Harden-Runner to harden the GitHub-hosted runners on which their workflows run. Harden-Runner provides network and runtime security for GitHub-hosted and self-hosted runners. This purpose-built platform is based on the learnings from past CI/CD security attacks (e.g., the SolarWinds and Codecov incidents).
Harden-Runner provides contextualized runtime security insights for Google’s projects’ workflows, prevents exfiltration of code and CI/CD credentials by blocking egress traffic, detects if there is source code tampering, and detects deviations from the usual runtime behavior in workflows.
Google leverages StepSecurity Harden-Runner to harden the GitHub-hosted runners on which their workflows run. Harden-Runner provides network and runtime security for GitHub-hosted and self-hosted runners. This purpose-built platform is based on the learnings from past CI/CD security attacks (e.g., the SolarWinds and Codecov incidents).
The insights from Harden-Runner are available on the StepSecurity dashboard, which is public for open-source projects and private for private repositories.
The insights are aggregated at the GitHub organization level making it easy to spot suspicious outbound calls across workflows that are using Harden-Runner. The “All Observed Endpoints” report shows all the unique destinations to which outbound network calls were made. Each destination has links to some of the workflows that made that call. This enables the security team to review the list periodically and if a destination is suspicious, they can view the workflow runs that made the outbound call to investigate further.
https://app.stepsecurity.io/github/GoogleCloudPlatform/actions/all-endpoints
Google implements GitHub Actions best practices with the help of StepSecurity’s orchestration solution. The solution not only recommends security fixes but also creates pull requests automatically with the recommended fixes.
Some GitHub Actions security best practices adhered to by Google withthe help of StepSecurity were limiting GitHub token permissions, hardeningrunners (with StepSecurity Harden-Runner), pinning actions to full-lengthcommit SHA, and configuring Dependabot version updates.
StepSecurity is also recommended by Google’s security team to other developers as seen in this issue:
With the orchestration solution, Google could adopt standardized workflows across various repositories, and this further helped them to save hundreds of developer hours and enabled increased productivity.
This included using security tools like OpenSSF Scorecard, CodeQL (for SAST), and Dependency Review (SCA) in their workflows across different repositories to streamline their processes efficiently and uniformly release their software.
Here are some examples of Google’s standardized workflows enabled by StepSecurity:
StepSecurity’s solutions have not only been leveraged but have also been recommended by various developers at Google. Here's one of these developers recommending the use of StepSecurity for hash pinning in an issue:
https://github.com/google/double-conversion/issues/197#issuecomment-1557835283
Google's use of Harden-Runner and the StepSecurity orchestration solution enabled them to secure their workflows from possible CI/CD supply chain attacks and helped them save hundreds of developer hours in implementing GitHub Actions security best practices. This empowered developers to enhance the security of their workflows with network monitoring, detecting tampering of source code, adopting standardized workflows, and efficiently utilizing their time on innovative tasks rather than spending hours on the laborious task of abiding by security best practices. Moreover, StepSecurity Harden-Runner and orchestration solution enabled several Google developers working on different projects resolve their security problems seamlessly without any training.
CISA Enforces Network Egress Control and CI/CD Infrastructure Security to Harden their GitHub-hosted Runners
CISA’s case study talks about how it leverages StepSecurity Harden-Runner 's network egress control and runtime security in over 175 GitHub repositories to prevent Codecov and SolarWinds-style attacks.
