Airflow docker
hariselva | Feb. 21, 2023, 8:24 a.m.

## Docker build

  1. Setup .env file with env-values.yaml (key=value format)
  2. Set AWS access key (Dev) to environment variable
    1. export AWS_ACCESS_KEY_ID=
    2. export AWS_SECRET_ACCESS_KEY=
    3. export AWS_DEFAULT_REGION=us-east-1
    4. export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain finops-commons --query authorizationToken --output text`
    5. pip config set global.extra-index-url https://aws:$CODEARTIFACT_AUTH_TOKEN@finops-commons-733509834600.d.codeartifact.us-east-1.amazonaws.com/pypi/finops-commons/simple/
  3. Setting Kubernetes cluster
  1. To build new config, remove the old ~/.kube/config and rebuild it so that it won't append the new config details.
  2. Option 1: Using Docker Desktop (https://docs.docker.com/desktop/kubernetes/)
    1. Follow step on https://docs.docker.com/desktop/kubernetes/#enable-kubernetes
    2. Use "kubectl get svc" to make sure cluster is up and running
  3. Option 2: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
    1. aws eks update-kubeconfig \ --region us-east-1 \ --name airflow \ --role-arn arn:aws:iam::733509834600:role/eks-cluster-eksclusterMastersRole4E0FE7C8-158ZTL8CC46YM
    2. Use "kubectl get svc" to make sure cluster is up and running
  4. Container Build: 1. docker build . -t finops-airflow --no-cache \ --build-arg CODEARTIFACT_AUTH_TOKEN=$CODEARTIFACT_AUTH_TOKEN \ --build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \ --build-arg AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
  5. Container Mgmnt:
    1. Init airflow: docker-compose up -d --build postgres airflow-init
    2. Up: docker-compose up -d --build postgres airflow-webserver airflow-scheduler
    3. Down: docker-compose down -v