What is Azimuth?
Azimuth leverages infrastructure managed by Openstack to provide self-service applications such as Jupyter notebooks and Kubernetes clusters.
Traditional HPC/Big Data approaches were to have individual infrastructure for specific use cases, managed by different tools and teams. Azimuth brings all of these tools together into a single platform to provide use cases for HPC, Big Data and more.
How it works
Setup
Azimuth is setup of a management cluster which runs the Azimuth UI and API responsible for provisioning workload apps and resources, and workload clusters which run in a separate OpenStack project which hosts the self-service clusters and applications.
Authentication
Azimuth relies on Keycloak as an identity provider, where each Keycloak realm is an Azimuth tenancy.
Keycloak is deployed alongside Azimuth within the management cluster.
When logging into Azimuth, it is actually authenticating with OpenStack. So the user authenticating with Azimuth will have the same tenancies/projects as the OpenStack user.
Networking
The workload project requires one external network. If there is more than one, then the Neutron tag is used to select which network to use.
Azimuth OpenTofu auto-creates networks within the workload project which defaults to 192.168.3.0/24.
Once provisioned the Ansible will spit out a URL to connect to Azimuth. By default, it will make use of sslip.io domains for automatic DNS handling to be able to route to the Azimuth UI via the nginx ingress controller in the management cluster.
Zenith Application Proxy
A Zenith client generates a key pair which connects out to a Zenith server. An SSH tunnel is configured between the client and the server for each proxied application. Before this tunnel is established, the client will need to be authenticated through OIDC.
A k8s service is generated by the Zenith server which is exposed by the ingress controller. This ensures that IPs are not exposed as the client goes through NAT before it goes out to the Zenith server.
The Zenith server will also handle the TLS termination for all proxied services.
Deployment Stack
Infrastructure
Azimuth is deployed using Ansible as an interface, but under the hood it will run OpenTofu to provision the Openstack infrastructure and deploy Kubernetes resources.
The OpenTofu state is stored in Kubernetes etcd.
Under the hood, when a user creates a tenancy, a namespace is configured on the underlying cluster which will be used to manage the TF state for the platforms that the user will provision.
Kubernetes
The K8s infrastructure is deployed via ClusterAPI, which allows for the granular behavior of clusters from the node groups to the auto-scaling behavior. Again, the interface for this is via Ansible which templates the Ansible variables into Helm chart values, where Helm charts are used to provision all types of resources from new workload clusters, workstations and applications like Jupyter.
Management Cluster Deployment
Azimuth deployment configuration is managed through the azimuth-config repository.
It requires OS_CLOUD and OS_CLIENT_CONFIG_FILE environment variables to be exported to be able to authenticate to the OpenStack backend.
Deployment steps:
- Upload images to Glance (K8s slurm etc.)
- Use OpenTofu to deploy a Seed node
- Seed is configured as a K3s cluster with ClusterAPI deployed (Management Cluster)
- Deploy a HA K8s cluster thorugh CAPI (K8s API only exposed on the internal network)
- Octavia Loadbalancers used for Nginx ingress
- Deploy applications to HA cluster
- Keycloak
- Zenith Server
- CaaS components
- Azimuth API
- Harbor (Used as a pull-through cache for images)
These above steps deploy the HA cluster which hosts Azimuth and CAPI components which will allow for workload projects and platforms to be provisioned.
Platform Deployment
When a user selects a new tenancy in the Azimuth UI, the Azimuth API will create Kubernetes CRDs within the cluster which provisions all of the OpenStack infrastructure, sharing the internal network into the OS project and deploying a Zenith client inside the workload project in OpenStack.
When a cluster CRD object is created, it triggers a K8s job which executes Ansible to configure the cluster logic in Openstack. These jobs and their progress can be viewed using the K8s API from the HA cluster deployed previously.
CaaS Operator
Azimuth will create a Lease CRD which creates and deletes Blazar reservations. As a cluster is deployed, it will add a finalizer on the Lease which blocks the lease being deleted until the cluster is ready.
When Zenith is deployed, metadata is passed to it as part of the cluster CRD. The cluster node image is configured to look for this metadata.
Cluster API
Cluster API allows for provisioning of K8s clusters using CRDs. OpenStack is one of many supported infrastructure APIs/clouds which CAPI supports. This allows for completely immutable infrastructure that can be easily destroyed and reproduced.
Under the hood of the Azimuth CRDs, Azimuth creates resources to talk to ClusterAPI. The cluster configuration set by the user in the UI is templated into Helm charts which configure the Cluster resources to be deployed onto the management cluster, then telling OpenStack how to provision the infrastructure.
Cluster Addons
StackHPC provide a cluster addon operator which manages deployments of tools to be deployed ontop of the workload clusters. These tools include tools like the CNI, ingress controller, device drivers and monitoring. The charts which deploy resources for this operator can be found here
One of the useful tools deployed, is the node-feature-discovery which will identify PCIe devices on the host and appropriately label the host with its features. This provides the information for addons for device drivers to be added to the nodes for things like GPU accessibility.
Addons are deployed using Helm charts where each chart deployment is its own CRD/resource on the management cluster which has the context of the remote workload cluster for the chart resources to be installed on to.
Cluster Templates
Cluster Templates are another StackHPC CRD which defines a flavour of cluster such as the available K8s versions and the CNI types available.
App Templates and Applications
App templates pre-define which applications are available in a platform/project.
StackHPC provide helm charts to deploy particular applications from Azimuth. Under the hood, the Azimuth API is installing these application charts, using the HelmRelase StackHPC CRD, onto the workload cluster to deploy the application resources.
Azimuth Config Repo
The Azimuth configuration repo allows you to define different Azimuth deployment environments. It comes with pre-defined Ansible configurations for deploying HA environments just by pointing an environment's ansible.cfg inventory at ../ha/inventory.