hariselva | Feb. 21, 2023, 8:24 a.m.
## Docker build
- Setup .env file with env-values.yaml (key=value format)
- Set AWS access key (Dev) to environment variable
- export AWS_ACCESS_KEY_ID=
- export AWS_SECRET_ACCESS_KEY=
- export AWS_DEFAULT_REGION=us-east-1
- export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain finops-commons --query authorizationToken --output text`
- pip config set global.extra-index-url https://aws:$CODEARTIFACT_AUTH_TOKEN@finops-commons-733509834600.d.codeartifact.us-east-1.amazonaws.com/pypi/finops-commons/simple/
- Setting Kubernetes cluster
- To build new config, remove the old ~/.kube/config and rebuild it so that it won't append the new config details.
- Option 1: Using Docker Desktop (https://docs.docker.com/desktop/kubernetes/)
- Follow step on https://docs.docker.com/desktop/kubernetes/#enable-kubernetes
- Use "kubectl get svc" to make sure cluster is up and running
- Option 2: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
- aws eks update-kubeconfig \ --region us-east-1 \ --name airflow \ --role-arn arn:aws:iam::733509834600:role/eks-cluster-eksclusterMastersRole4E0FE7C8-158ZTL8CC46YM
- Use "kubectl get svc" to make sure cluster is up and running
- Container Build: 1. docker build . -t finops-airflow --no-cache \ --build-arg CODEARTIFACT_AUTH_TOKEN=$CODEARTIFACT_AUTH_TOKEN \ --build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \ --build-arg AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
- Container Mgmnt:
- Init airflow: docker-compose up -d --build postgres airflow-init
- Up: docker-compose up -d --build postgres airflow-webserver airflow-scheduler
- Down: docker-compose down -v