I will talk about my experience for GitOpsCon Europe 2022 in this blog. I didn't attend it in person or virtually, but I watched uploaded videos for the same on CNCF YouTube channel. It happened in Valencia, Spain on 17th May 2022. It was a one-day event. It was co-located event with KubeCon + CloudNativeCon Europe 2022 - Valencia, Spain. It was purely based on GitOps. It is hosted by CNCF (Cloud Native Computing Foundation).

GitOpsCon EU 2022 Videos:

GitOps As a Journey:

Dan Garfield from Codefresh, Scott Rigby from Weaveworks, and Chris Short from AWS presented it. They talked about general instructions first. They discussed GitOps principles. There are four principles:

1. Declarative: A system managed by GitOps must have its desired state expressed declaratively
2. Versioned and Immutable: The desired state is stored in a way that enforces immutability, versioning, and retains a complete version history
3. Pulled Automatically: Software agents automatically pull the desired state declarations from the source
4. Continuously Reconciled: Software agents continuously observe the actual system state and attempt to apply the desired state

They referred to the above principles as GitOps principles 1.0. They mapped modern practices with these GitOps principles:

1. Declarative: Infrastructure as Code (IaC), Configuration as Code (CaC), DevOps & DevSecOps, and GitOps
2. Versioned and Immutable: Configuration as Code (CaC), DevOps & DevSecOps, and GitOps
3. Pulled Automatically: DevOps & DevSecOps, and GitOps
4. Continuously Reconciled: DevOps & DevSecOps, and GitOps

They also discussed the OpenGitOps project. It is a set of open-source standards, best practices, and community-focused education to help organizations adopt a structured, standardized approach to implementing GitOps. It is a CNCF sandbox project. You can refer to its GitHub repository.

Keynote: Everything as Code: Declarative Application Delivery with GitOps Workflows:

Christian Hernandez gave this lightning talk. He is Senior Principal Technical Marketing Manager at Red Hat. They gave information about the t-shirt swag of OpenGitOps for this event initially. It discussed everything as code using open source projects.

He talked about how Red Hat contributes to open-source projects. So other enterprise-level companies can use it. Red Hat is a big contributor to Tekton project. It is a cloud-native CI/CD (Continuous Integration/Continuous Deployment) tool for K8S (Kubernetes).

He gave information about Argo CD. Red Hat uses it for reconciliation (4th GitOps principle). It is based on ideas like GitHub Actions.

They also use Stackrox. It is used to take care of compliances. It is Kubernetes native security platform for cloud-native applications, containers, serverless, and Kubernetes. Red Hat acquired it. It is in the DevSecOps field.

He is briefly idea about Open Cluster Management project. It is a sandbox project in CNCF. It manages the life cycle of Kubernetes.

100,000 Different Ways to Manage Secrets in GitOps:

Andrew Block presented it. He is a Distinguished Architect at Red Hat. He published two books:

1. Learn Helm
2. Securing Kubernetes Secrets

He is also a Helm maintainer. GitOps principles apply to a variety of contexts like infrastructure, applications, cloud, and non-cloud assets.

Local machines, CI/CD, and OS are places where we managed sensitive GitOps resources. GitOps commonly involves the use of sensitive assets in the given below areas:

1. Source Code Repositories
2. Infrastructure Management
3. Application Configurations
4. Access to Platform Resources

He gave details about sensitive information like passwords. It should not be in plain text in the above areas. He discussed related understanding of tool limitations and capabilities.

He gave information about the impacts of GitOps secrets that were handled improperly. It can have substantial consequences as given below:

1. Time and Effort
2. Financial Implications
3. Career Repercussions
4. Brand Casualty

He gave an idea about how we can handle secrets in GitOps properly. There are three areas as shown below:

1. Storing Secrets - Encrypted content and external references
2. Accessing Secrets
3. Consuming Secrets

There are many tools to store secrets like Azure Key Vault, GitHub, etc. Some are self-hosted and cloud-managed such as GitHub, GitLab, etc. Some are only cloud-managed such as Azure Key Vault, Google Cloud KMS (Key Management System), etc.

How can we access stored values?

1. Tools Involved
2. Access Model
3. Consumption Considerations

He discussed various utilities that are available to retrieve store sensitive assets as given below:

1. CLI (Command Line Interface)
2. CI/CD: Jenkins, GitHub, etc.
3. Plugins: kubectl, helm, etc.

He gave details about Kubernetes-based solutions. There are three methods:

1. Controllers: Configurations can apply against any resource
2. Webhooks: Configurations can apply through annotations on resources
3. Operators: It is a declarative and automated approach. We can implement using CRDs (Custom Resource Definitions)

There are three popular tools to support the secret management in K8S:

1. Sealed Secrets: It is an easy option. Controller containing private keys to encrypt/decrypt sensitive values
2. External Secrets: Retrieve values from external secret management. e.g. HashiCorp Vault
3. Vault Agent Sidecar Injector: It injects sidecars to applications

There are two popular GitOps engine integration projects in K8S as given below:

1. Argo CD
2. Flux

He discussed areas where we can research and choose the right tool:

1. Considerations: Evaluate options supported by the tools, identify security boundaries
2. Recommendations: Create/use dedicated principal, short-lived credentials, emphasize the least privilege principle
3. Common Access Methods: Basic authentication, bearer token, assumed identity/OIDC (OpenID Connect). They use this method in their sigstore project at Red Hat.

There are two standard methods to consume secret values from within applications:

1. Environment Variables
2. File System

He talked about K8S CSI (Container Storage Interface) driver. It is included in K8S. It is pluggable and supports multiple secrets management backends like AWS, Azure, etc. It avoids sensitive assets such as K8S Secrets.

He discussed secret detection tools like GitHub secret scanning and other GitHub repositories that help us to detect secrets in our source code. Security is continuous. We should investigate vulnerable code if we found in our source code of the organization.

I learned a lot from this event. Please check out other videos as well in that playlist.

References:

1. GitOpsCon EU 2022 playlist on CNCF YouTube channel: youtube.com/playlist?list=PLj6h78yzYM2PTHsP..
2. GitOps As a Journey: youtu.be/LQgsxT3SlN8
3. Keynote: Everything as Code: Declarative Application Delivery with GitOps Workflows: youtu.be/I7u-v53BW-o
4. 100,000 Different Ways to Manage Secrets in GitOps: youtu.be/FVaaqP7_AJg