Adrian Cowan
b53bfe4075
Some changes were required to set 3 VMs as the cluster since the NUC failed and we are waiting for new hardware to arrive. The ingest routing from the internet was changed to use cloudflared tunnel to traefik instead of via a specific host.
Terraform State
Mount the state on the fileshare to 2-nomad-config/.tfstate/
sudo mount -t cifs //192.168.1.192/appdata/terraform /home/othrayte/Code/infra/2-nomad-config/.tfstate/ -o rw,username=othrayte,password=<pw>,uid=$(id -u),gid=$(id -g)
Tailscale Oauth Client
We use a Tailscale oauth client secret to allow our containers to connect to tailscale. We created an oauth client called nomad with the auth_keys (write) scope for the tag nomad and stored the secret in our secrets file.
Secrets
The secrets file is encrypted using sops and will be automatically decrypted in the terraform provider.
Put the age keys in /home//.config/sops/age/keys.txt
Adding Secrets
Edit the secrets using sops secrets.enc.json
Bootstrapping (starting without PostgreSQL running)
terraform apply -target=nomad_job.postgres
Restoring PostgreSQL DBs
psql -h jaglan-beta-m21 -p 5432 -U postgres -f ~/Downloads/all_databases.sql postgres
Deploying and testing changes
Sometimes the nomad job fails but the solution is to fix another job and so we need to tell nomad to retry the unchanged job.
nomad job eval -force-reschedule glance