Case studiesDevSecOpsRegulated sector, Hong Kong

DevSecOps case study: Secure DevSecOps with GitOps for software supply chain to On-Premise Kubernetes

We helped a regulated organisation in Hong Kong check every open-source package against policy before it reaches a build, and gave its applications one standard path to on-premise Kubernetes: Jenkins to build, approval, then FluxCD to deploy from Git.

From commit to cluster

  1. 01Source repo
  2. 02Jenkins CI
  3. 03Scans
  4. 04Nexus
  5. 05Jenkins CD
  6. 06GitOps repo
  7. 07FluxCD
  8. 08Kubernetes
Client
Regulated organisation, Hong Kong
Environment
On-premise Kubernetes, designed to run air-gapped
Duration
9 months
Service
DevSecOps

Where they started

Each team built and released software its own way.

The organisation builds software in highly varied technologies, such as Java, Node.js, Python, Go, C# and native mobile, and runs it as containers on on-premise Kubernetes.

Before

  1. Each team copied and changed its own pipeline script.
  2. Teams started builds by hand, with no automated tests or quality gates.
  3. Teams uploaded open-source packages by hand, with no security check.
  4. Security review came late, just before user acceptance testing.
  5. Teams deployed to production by hand with cluster commands.
  6. Image tags showed a branch and a timestamp, not a version number.

Every team kept its own copy of the pipeline script, so nobody could fix or improve the build in one place. Security review close to user acceptance testing made release dates hard to predict. Image tags carried a branch name and a timestamp instead of a version, so tracing a production issue back to the code meant digging through build logs.

Starting from a DevSecOps gap analysis

The work started from a DevSecOps gap analysis of the whole delivery lifecycle, based on workshops with the teams, process documents and a hands-on review of the pipeline configuration. The analysis mapped each step of the current flow to a proposed one and kept what already worked, such as the teams' familiar branching model. For each tool category it compared three or four products on features, resource needs and the skills needed to run them.

Two goals

A secure software supply chain and one standard CI/CD pipeline.

Goal 1

Secure the software supply chain

Fetch, cache and check every open-source component in one place.

Goal 2

One standard CI/CD pipeline

Give every Kubernetes application one pipeline that the organisation's DevSecOps governance can build on.

The organisation chose BSOL because our vast experience let us get up to speed quickly.

The platform must:

  • run on-premise with high availability and no single point of failure in the application and storage tiers;
  • keep the source control, CI and GitOps tools the teams already use;
  • fit the organisation's change-management process, with strict approvals and a full audit trail before anything reaches production.

High availability

High availability on-premise, ready for an air-gapped network.

Our engineers built every tier as a pair of nodes behind a virtual IP, so the platform keeps running when a node fails.

REPOSITORY SERVERS VIP node A node B POLICY SERVERS VIP node A node B DB CONNECTION POOLERS VIP node A node B SHARED REPLICATED STORAGE REPLICATED DATABASE
Fig. 01 Repository servers, policy servers and database connection poolers: two nodes each, behind a virtual IP.

We tested failover for every tier before acceptance and wrote the results into a recovery manual.

We started with the infrastructure, working alongside the organisation's platform operations team. Each component runs as a pair of nodes behind a virtual IP: the repository servers, the policy servers and the connection poolers in front of a replicated database, with shared replicated storage for the artifacts. Database, application and storage tiers run separately, and the whole design works in an air-gapped network.

The operations and recovery manual tells the in-house team what to do when a node goes down.

Supply chain

How we secured open-source packages with Sonatype Repository Firewall.

package ecosystems, one address

  • Maven
  • Gradle
  • npm
  • PyPI
  • Go
  • NuGet
  • CocoaPods
  • Docker

Our engineers put one Nexus proxy between the organisation's teams and every public package source. Repository Firewall checks each package against policy as it arrives and quarantines any package that breaks it before it can reach a build.

PUBLIC SOURCES MAVENGRADLENPMPYPIGONUGETCOCOAPODSDOCKER CHECKED ON ARRIVAL Nexus proxy+ FirewallPOLICY CHECK QUARANTINE GroupONE ADDRESS DevelopersBuild agentsClusters
Fig. 02 One Nexus proxy with Repository Firewall. A package that breaks policy goes to quarantine.

Sonatype Nexus Repository Pro is now the single place for every artifact. Hosted repositories hold the organisation's own images and packages. Proxy repositories fetch and cache approved public sources for eight package ecosystems. A group repository puts both behind one address, so developers, build agents and clusters all pull from the same place.

The assessment suggested several open-source scanners, but the organisation preferred fewer tools with commercial support. So we integrated Checkmarx One for application security testing and kept SonarQube for code quality.

Jenkins pipeline

Each new application joins the Jenkins shared-library pipeline with one YAML file.

Our engineers moved all pipeline logic into one Jenkins shared library that the DevOps team maintains.

# apps/example-service.yamlname: example-servicebranch: release/1.4steps: - security-scan - image-buildSEED JOBGENERATEDexample-service · CIGENERATEDexample-service · CDIMAGE TAG1.4.2VERSION-release-1.4BRANCH-87BUILD
Fig. 03 A seed job turns the YAML file into Jenkins CI and CD jobs. Illustrative example.

Application teams now onboard a new application with one YAML file, without writing pipeline code.

The DevOps team keeps all pipeline logic in one Jenkins shared library, and application repositories carry no Jenkinsfile. A short YAML file describes each application: its name, source branch and the steps it needs, such as security scanning and image build. Seed jobs generate the Jenkins jobs from these files.

Every application uses three repositories: source code, Helm configuration and deployment state. Services share one Helm library chart, so each service chart is a thin wrapper that switches on what it needs, such as scheduled jobs, autoscaling or disruption budgets. Image tags combine a semantic version with the release branch and build number. The pipeline also writes that version back to source control as a Git tag, so developers can see what each build contains without opening the repository manager.

Promotion and rollback

GitOps promotion and rollback with FluxCD.

One commit promotes a release. Reverting it rolls the release back.

The same image moves from test to production unchanged, and only FluxCD changes the clusters. Every change has an approver, an author and a history in Git, so the release team can see what changed, when, and who approved it.

System testAUTOMATIC ON NEW IMAGECOMMITUATTEAM LEAD APPROVESCOMMITProductionRELEASE TEAM APPROVESCOMMIT v1.4.2 DEPLOYMENT-STATE REPO · FLUXCD APPLIES ROLLBACK = REVERT ONE COMMIT
Fig. 04 The same image, promoted by commits that FluxCD applies.

When CI pushes a new image to Nexus, a webhook starts the CD job, which deploys to the system-test environment automatically. Promotion to UAT and production waits for approval. The application team lead approves UAT, and the release team approves production. The CI pipeline also writes a release manifest listing every service in the release, so promoting a multi-service application takes one job and one commit.

The CD pipeline commits to the deployment-state repository. To roll back, the team reverts one commit or re-promotes the previous manifest; the earlier images are already in Nexus, so nothing needs rebuilding. Every change to every environment has an author and a history in Git.

Hand-over

Hand-over: the organisation's teams run the platform themselves.

Our engineers trained the organisation's administrators and developers hands-on and documented the platform. Their teams now run it and onboard new applications themselves.

  • Installation, configuration and operations guides
  • Recovery manual
  • Developer guide
  • Build and release guide
  • Deployment configuration guide
  • Step-by-step onboarding guide
  • Hands-on training for administrators and developers
  • Support after go-live

Results

Results

Fig. 05 · Before, after and benefits: CI/CD pipeline, new applications, open-source packages, security review, production deploys and issue tracing
AreaBeforeAfterBenefits
CI/CD pipelineDifferent in every team, hard to maintainOne standard pipeline, with managed variationsOne standard that is much easier to maintain
New applicationCopy and paste pipeline configAdd one YAML fileAdopts the organisation's best practice from day one
Open-source packagesMaintained by handOne proxy that scans every packageAutomated, with security assurance
Security reviewLate, close to UATRuns during coding and every buildIssues surface early, so release dates stay predictable
Production deployDeployed by handGitOps deploymentAutomated after approval, with a full audit trail
Issue tracingSearch build logs by handRead the version from the image and Git tagsTrace a production issue straight back to its code

Every new application now has a secure, standard path to production.

Securing your software supply chain or delivery path?

Twenty minutes, no preparation needed.

Book a discovery call