Gitops-based kubernetes phoenix infrastructure using argocd

A low level strategy …

• deploy a cluster, we’ll call it ‘root’
• deploy argocd using helm, use namespace ‘argocd-seed’
• create a git repo named ‘k-argocd-root’, and set the paths as:
  • /appofapps/base/templates/(symlink to each app)
  • /apps/argocd-clusters
  • /apps/argocd-global
  • /apps/argocd
• now apply the appofapps application with
  • cd /appofapps
  • kubectl apply -f applicationset.yaml
• we now have 4 argocd instances running, the first ‘argocd-seed’ manages the three
• the three have the following goals:
  • argocd-clusters – used to deploy vclusters (or use clusterapi, or tanzu, etc…)
  • argocd-global – used to deploy common addons to all registered clusters
  • argocd – an argocd instance used to deploy all remaining apps to all clusters
• create a git repo named ‘k-argocd-root-global’, and set the paths as:
  • /appofapps/base/templates/(symlink to each app)
  • /apps/cert-manager
  • /apps/external-dns
  • …
• now apply the appofapps application with
  • cd /appofapps
  • kubectl apply -f applicationset.yaml
• create a git repo named ‘k-argocd-root-clusters’, and set the paths as:
  • /appofapps/base/templates/(symlink to each cluster)
  • /apps/<cluster>/base/templates/<cluster>.yaml
• now apply the appofapps application with
  • cd /appofapps
  • kubectl apply -f applicationset.yaml
• now, whenever you wish to create a new cluster (use a pipeline to do this):
  • create yaml representing new cluster and add to:
    • /apps/<cluster>/base/templates/<cluster>.yaml
  • add the cluster to the ‘argocd-global’ argocd instance
• a note regarding the 4th ‘argocd’ instance:
  • whenever using argocd ‘appofapps’ & ‘apps’ exist at the root of the git repo, name the repo by the clustername to keep things organized, e.g. ‘k-argocd-<clustername>’, such as:
    • k-argocd-core/appofapps/base/templates/(symlink to each app)
    • k-argocd-core/appofapps/applicationset.yaml
    • k-argocd-core/apps/harbor/base/templates
    • k-argocd-core/apps/harbor/applicationset.yaml
    • k-argocd-core/apps/jenkins/base/templates
    • k-argocd-core/apps/jenkins/applicationset.yaml

Conclusions

This setup allows a gitops based infrastructure with a phoenix style deployment.  In theory, you can delete everything and start over, and with only four ‘kubectl apply -f ./applicationset.yaml’ commands the entire infrastructure will come back up… including the additional clusters and apps deployed to them. Perhaps a script, phoenix-rise.sh …

… except for one thing, though the clusters will come back up, a step is still required to register all the clusters with the ‘argocd-global’ instance so that it can target all the addons to all the clusters.  We need a way to watch for new clusters to come up and automatically register them with ‘argocd-global’. Additionally, it might be nice to create groups to work with RBAC. Something to look into …

Also, applicationsets, where appropriate, need to target a cluster by name (registered via dns, use external-dns for this) rather than ip address, at least … if you want to test out deleting it all and having it all come back up.

Thank you

I hope you have found this discussion useful!