ArgoCD Multi-tenancy strategy

Introduction

When you decide to adopt ArgoCD in your organization, you usually start by letting all users do what they want until you go into production and notice that someone, by deleting their application, has deleted a namespace or CRDs :D Ok, I’m being a bit dramatic but this use case is possible. That’s why you need to have multi-tenant strategy with access control. I’m going to show you how you can implement such a strategy by taking advantage of ArgoCD’s native features.

As a bonus, I’ll offer you a Chart + ApplicationSet for implementing it in a large organization using an enterprise Agile framework (example SAfe).

ArgoCD multi-tenancy: the basics

AppProject

ArgoCD AppProject is your first resource for implementing multitenancy. First, you need to ask yourself the following questions

You need first to ask you the following questions

• Who ? Who can deploy
• What ? What can be deployed ?
• Where ? Where can my code be deployed ?(namespace, cluster)
• From Where ? From which repository can the code be deployed, and also from which namespace (we’ll look at this feature later).

The AppProject Spec looks like this:

spec:
  # Where ?
  destinations:
  - namespace: '*'
    server: '*'
  # From Where ?
  sourceRepos: 
  - '*'
  # What ?
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  # Who
  roles:
    - name: admins
      # ...

AppProject supports globs. As you can see from the example above, this AppProject lets you deploy everything anywhere and everywhere. This is the default project and, except for experimentation, should never be used!!!! Even for administrators, I strongly recommend creating an AppProject with dedicated permissions. Let’s look at a concrete example with a teamone:

• Who ? The team one (Azure Active Directory ID is 00000000–0000–0000–0000–000000000000) will be readonly on this project
• What ? Only namespace resources (no cluster resources)
• Where ? A good practice that I recommend is to use the team name as namespace prefix. So here any namespace starting by teamone
• From Where ? Team one is hosting repositories in https://github.com/organization/department/team-one/

# Source: argocd-team-configuration/templates/AppProject.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: teamone
spec:
  description: Teamone components
  sourceRepos:
    - https://github.com/organization/department/team-one/*
  destinations:
    - namespace: teamone*
      server: https://kubernetes.default.svc
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'
  roles:
    - name: readonly
      description:  Teamone readonly role 
      policies:
        - p, proj:teamone:readonly, applications, get, teamone/*, allow
      groups:
        - 00000000-0000-0000-0000-000000000000

ArgoCD RBACs

ArgoCD RBACs allows you to define roles with permissions. You can then attach group of users to theses roles.

RBACs can be set globally and per-project.

For global RBACs I recommend you to define only Platform admin role and default policy.

rbac:
    policy.default: "role:cluster-reader"
    policy.csv: |
      g, ffffffff-ffff-ffff-ffff-ffffffffffff, role:readonly
      # Allows all users to get clusters
      p, role:cluster-reader, clusters, get, *, allow

In the above example our admins (Azure ID: ffffffff-ffff-ffff-ffff-ffffffffffff) will have the default readonly role. All other users will be able to list / get the clusters.

Projects RBACs allows you to define granular roles. For example you could imagine having:

User administrators, who can perform all actions on the applications belonging to your project. Limited edition users who will be able to perform all actions on the applications belonging to your project, with the exception of creating, updating and deleting applications. Read-only users, who can only obtain application content and read logs. You can as well enforce Application globs to restrict on which Application a role can perform actions

Here are some examples:

# Readonly can get any application belonging to teamone project
p, proj:teamone:readonly, applications, get, teamone/*, allow

# RedSquad can perform any action on Application 
# resources whose name starts with redsquad 
# and which belong to the teamone project
p, proj:teamone:redsquad, applications, action/*, teamone/redsquad*, allow

Application in any namespace

Remember:

From Where ? From which repository code can be deployed and also from which namespace (we will see this feature later)

ArgoCD allows you to deploy Application (since 2.5) in different namespaces (other than the control plane namespace). This allows admin to restrict AppProject usage to certain namespaces.

spec:
  sourceNamespaces:
  - team-one-cd
In the above example, only Application deployed in team-one-cd will be able to use this AppProject

Here is an explicative schema:

Applications in any namespace - Argo CD - Declarative GitOps CD for Kubernetes

Current feature state: Beta Warning Please read this documentation carefully before you enable this feature… argo-cd.readthedocs.io

Implementation

Prerequisites

As prerequisite you need to enable Application in any namespace.

We will use argocd-* as namespace prefix. You need as well to define global rbacs.

If you are using helm you can do it like this:

configs:

  # Configure oidc authentication
  oidc.config: |
    name: Azure
    issuer: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0
    clientID: 00000000-0000-0000-0000-000000000000
    clientSecret: $argo-azure-secrets:AZURE_AD_SECRET
    requestedIDTokenClaims:
      groups:
        essential: true
    requestedScopes:
     - openid
     - profile
     - email

  # Configure Application and ApplicationSet in any namespace
  params:
    application.namespaces: argocd-*

  # Global rbac
  # Admin is readonly
  # Default user is cluster reader (adjust if needed)
  rbac:
    policy.default: "role:cluster-reader"
    policy.csv: |
      g, ffffffff-ffff-ffff-ffff-ffffffffffff, role:readonly
      # Allows all users to get clusters
      p, role:cluster-reader, clusters, get, *, allow

AppProject per team

Let’s imagine we’re in an organization with an agile framework (like Safe) with entities (i.e. trains) divided into scrum teams (or squads). We have the following roles:

A system team that will be the AppProject administrator 3 agile teams (reqsquad, bluesquad, greensquad) who will be read-only on the whole project, but will have editing permissions on their respective applications. A coordination team, with the system architect, product manager and production release manager (or RTE in Safe), who will be read-only on the entire project so that they can control the application. A front-line incident management team, which will be read-only throughout the project. Each project will use repositories in this format:

https://github.com/organization/${trainName}/*

Each squad will have Applications prefixed by the squad name:

redsquad-data
redsquad-customerone
...

The Application / ApplicationSet will be deployed in argocd-train-one namespace.

Let’s now create the AppProject

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: train-one
spec:
  description: Train-One components
  # From where ?
  sourceRepos:
    - https://github.com/organization/train-one/*
  # Where ?
  # Train one will be able to deploy to both argocd-train-one
  # And any namespace prefixed by train-one
  destinations:
    - namespace: argocd-train-one
      server: https://kubernetes.default.svc
    - namespace: train-one*
      server: https://kubernetes.default.svc
  # What ?
  # Only namespace resources are allowed
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'
  # From where ? 
  # train-one will be usable only from Application and ApplicationSet
  # deployed in namespace argocd-train-one
  sourceNamespaces:
    - argocd-train-one

We need then to apply project RBACs:

For admins we want them to be able to create/delete/get/override/sync/update & perform any actions on Application deployed as part of project train-one. To make it simple we can take inspiration from the admin built-in policy

    - name: admins
      description:  train-one Admin role 
      policies:
        - p, proj:train-one:admins, applications, create, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, delete, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, override, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, sync, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, update, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, action/*, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, create, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, update, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, delete, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, exec, create, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, projects, get, train-one, allow
        - p, proj:train-one:admins, repositories, get, train-one/*, allow
        - p, proj:train-one:admins, repositories, create, train-one/*, allow
        - p, proj:train-one:admins, repositories, update, train-one/*, allow
        - p, proj:train-one:admins, repositories, delete, train-one/*, allow
      groups:
        - 00000000-0000-0000-0000-000000000000

Please note that we force usage of namespace argocd-train-one when using train-one/argocd-train-one/*

- For readonly we want users to be able to get applications, repositories and logs in applications:

    - name: frontline-readonly
      description: train-one Readonly role for frontline 
      policies:
        - p, proj:train-one:frontline-readonly, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:frontline-readonly, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:frontline-readonly, repositories, get, train-one/*, allow
      groups:
        - 22222222-2222-2222-2222-222222222222

For edit we want to be able to get/sync and perform any actions on Application that are prefixed by the squad name (example: red)

    - name: redsquad-edit
      description: train-one Readonly role for redsquad 
      policies:
        - p, proj:train-one:redsquad-edit, applications, get, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, applications, sync, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, applications, actions/*, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, logs, get, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, exec, create, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, repositories, get, train-one/*, allow
      groups:
        - 33333333-3333-3333-3333-333333333333

Make this configurable

Now we have in mind what we want to put in place let’s think about a chart.

apiVersion: v2
name: argocd-team-configuration
description: A chart to deploy team configuration

type: application

version: 0.1.0

The values:

id: project-id

# Defines the source Repos from which an applications can be deployed. It can use globs
sourceRepos: []
#  - repo1
#  - repo2

# Defines the list of Admins. It is a list of Azure AD security group UUIDs
# the key is an arbitrary name
admins: null
#  admin-id-1: uuid1
#  admin-id-2: uuid2

# Defines the additional namespaces on which a team can deploy. It can use globs
additionnalNamespaces: []
#  - namespace1
#  - namespace2

# Defines the list of readonly users with the associated Azure AD security group UUID and application glob (ie: customerone-* for all applications prefixed by customerone in team project)
# the key is an arbitrary name
readonly: null
#  readonly-id-1:
#    glob: application-glob*
#    azureAdId: uuid
#  readonly-id-2:
#    glob: application-glob*
#    azureAdId: uuid

# (Default null) Defines the list of edit users with the associated Azure AD security group UUID and application glob (ie: customerone-* for all applications prefixed by customerone in team project)
# the key is an arbitrary name
edit: null
#  readonly-id-1:
#    glob: application-glob*
#    azureAdId: uuid
#  readonly-id-2:
#    glob: application-glob*
#    azureAdId: uuid

The AppProject template:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: 
spec:
  description:  components
  sourceRepos:

  destinations:
    - namespace: 
      server: https://kubernetes.default.svc
    - namespace: 
      server: https://kubernetes.default.svc
    - namespace: 
      server: https://kubernetes.default.svc
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'
  sourceNamespaces:
    - 
  roles:
    - name: admins
      description:   
      policies:
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
        - 
      groups:

    - name: 
      description:  
      policies:
        - 
        - 
        - 
      groups:
        - 
    - name: 
      description:  
      policies:
        - 
        - 
        - 
        - 
        - 
        - 
      groups:
        - 

An example of config with above example:

id: train-one

sourceRepos:
  - https://github.com/organization/train-one/*

admins: 
  system-team: 00000000-0000-0000-0000-000000000000

readonly:
  coordination:
    glob: '*'
    azureAdId: 11111111-1111-1111-1111-111111111111
  frontline:
    glob: '*'
    azureAdId: 22222222-2222-2222-2222-222222222222
  redsquad:
    glob: '*'
    azureAdId: 33333333-3333-3333-3333-333333333333
  bluesquad:
    glob: '*'
    azureAdId: 44444444-4444-4444-4444-444444444444
  greensquad:
    glob: '*'
    azureAdId: 55555555-5555-5555-5555-555555555555

edit:
  redsquad:
    glob: 'red-*'
    azureAdId: 33333333-3333-3333-3333-333333333333
  bluesquad:
    glob: 'blue-*'
    azureAdId: 44444444-4444-4444-4444-444444444444
  greensquad:
    glob: 'green-*'
    azureAdId: 55555555-5555-5555-5555-555555555555

Another example with a simple scrum team

id: scrum-one

sourceRepos:
  - https://github.com/organization/department/team-one/*

admins: 
  system-team: 00000000-0000-0000-0000-000000000000

An ApplicationSet to deploy it

If you want to put in place a self service so that teams can manage their own permission. You can create a git repository hosting project configurations where team can submit pull requests for their values that will be approved and merged by the platform team.

You can then create an ApplicationSet which will automate the deployment.

Example:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-configurations
  namespace: argocd
  labels:
    owner: platform-team
spec:
  goTemplate: true
  generators:
  - git:
      repoURL: https://github.com/speedfl/argocd-reallife-samples.git
      revision: HEAD
      files:
        - path: 'organization-multitenancy/configs/*.yaml'
  template:
    metadata:
      name: '-team-configuration'
      labels:
        owner: platform-team
    spec:
      project: platform
      sources:
        - repoURL: https://github.com/speedfl/argocd-reallife-samples.git
          targetRevision: HEAD
          ref: values
        - repoURL: https://github.com/speedfl/argocd-reallife-samples.git
          targetRevision: HEAD
          path: organization-multitenancy/chart
          helm:
            valueFiles:
              - $values//
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

The result:

And the content:

---
# Source: argocd-team-configuration/templates/AppProject.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: train-one
spec:
  description: Train-One components
  sourceRepos:
    - https://github.com/organization/train-one/*
  destinations:
    - namespace: argocd-train-one
      server: https://kubernetes.default.svc
    - namespace: train-one*
      server: https://kubernetes.default.svc
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'
  sourceNamespaces:
    - argocd-train-one
  roles:
    - name: admins
      description:  train-one Admin role 
      policies:
        - p, proj:train-one:admins, applications, create, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, delete, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, override, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, sync, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, update, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applications, action/*, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, create, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, update, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, applicationsets, delete, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, exec, create, train-one/argocd-train-one/*, allow
        - p, proj:train-one:admins, projects, get, train-one, allow
        - p, proj:train-one:admins, repositories, get, train-one/*, allow
        - p, proj:train-one:admins, repositories, create, train-one/*, allow
        - p, proj:train-one:admins, repositories, update, train-one/*, allow
        - p, proj:train-one:admins, repositories, delete, train-one/*, allow
      groups:
        - 00000000-0000-0000-0000-000000000000
    - name: bluesquad-readonly
      description: train-one Readonly role for bluesquad 
      policies:
        - p, proj:train-one:bluesquad-readonly, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:bluesquad-readonly, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:bluesquad-readonly, repositories, get, train-one/*, allow
      groups:
        - 44444444-4444-4444-4444-444444444444
    - name: coordination-readonly
      description: train-one Readonly role for coordination 
      policies:
        - p, proj:train-one:coordination-readonly, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:coordination-readonly, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:coordination-readonly, repositories, get, train-one/*, allow
      groups:
        - 11111111-1111-1111-1111-111111111111
    - name: frontline-readonly
      description: train-one Readonly role for frontline 
      policies:
        - p, proj:train-one:frontline-readonly, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:frontline-readonly, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:frontline-readonly, repositories, get, train-one/*, allow
      groups:
        - 22222222-2222-2222-2222-222222222222
    - name: greensquad-readonly
      description: train-one Readonly role for greensquad 
      policies:
        - p, proj:train-one:greensquad-readonly, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:greensquad-readonly, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:greensquad-readonly, repositories, get, train-one/*, allow
      groups:
        - 55555555-5555-5555-5555-555555555555
    - name: redsquad-readonly
      description: train-one Readonly role for redsquad 
      policies:
        - p, proj:train-one:redsquad-readonly, applications, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:redsquad-readonly, logs, get, train-one/argocd-train-one/*, allow
        - p, proj:train-one:redsquad-readonly, repositories, get, train-one/*, allow
      groups:
        - 33333333-3333-3333-3333-333333333333
    - name: bluesquad-edit
      description: train-one Edit role for bluesquad 
      policies:
        - p, proj:train-one:bluesquad-edit, applications, get, train-one/argocd-train-one/blue-*, allow
        - p, proj:train-one:bluesquad-edit, applications, sync, train-one/argocd-train-one/blue-*, allow
        - p, proj:train-one:bluesquad-edit, applications, actions/*, train-one/argocd-train-one/blue-*, allow
        - p, proj:train-one:bluesquad-edit, logs, get, train-one/argocd-train-one/blue-*, allow
        - p, proj:train-one:bluesquad-edit, exec, create, train-one/argocd-train-one/blue-*, allow
        - p, proj:train-one:bluesquad-edit, repositories, get, train-one/*, allow
      groups:
        - 44444444-4444-4444-4444-444444444444
    - name: greensquad-edit
      description: train-one Edit role for greensquad 
      policies:
        - p, proj:train-one:greensquad-edit, applications, get, train-one/argocd-train-one/green-*, allow
        - p, proj:train-one:greensquad-edit, applications, sync, train-one/argocd-train-one/green-*, allow
        - p, proj:train-one:greensquad-edit, applications, actions/*, train-one/argocd-train-one/green-*, allow
        - p, proj:train-one:greensquad-edit, logs, get, train-one/argocd-train-one/green-*, allow
        - p, proj:train-one:greensquad-edit, exec, create, train-one/argocd-train-one/green-*, allow
        - p, proj:train-one:greensquad-edit, repositories, get, train-one/*, allow
      groups:
        - 55555555-5555-5555-5555-555555555555
    - name: redsquad-edit
      description: train-one Edit role for redsquad 
      policies:
        - p, proj:train-one:redsquad-edit, applications, get, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, applications, sync, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, applications, actions/*, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, logs, get, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, exec, create, train-one/argocd-train-one/red-*, allow
        - p, proj:train-one:redsquad-edit, repositories, get, train-one/*, allow
      groups:
        - 33333333-3333-3333-3333-333333333333

The full example is here: https://github.com/speedfl/argocd-reallife-samples/tree/main/organization-multitenancy

Disclaimer Please note that this is only a sample implementation. You must use these features according to your organization’s needs and restrictions (ie: Per env / per cluster / single argocd).

To implement multi-tenancy you mus as well to use:

Kubernetes RBAC Workload / Storage / Network isolation Quotas Policies engine