AWS: プライベートサブネットの EKS でプロキシを使う

投稿日 by

 
なかなか苦戦したのでメモ。

private subnet の中にある EC2 で、外部との通信は public subnet にあるプロキシを経由してる環境。
で、VPC CIDR の 10.0.0.0/16 とは自由に通信出来るけど 0.0.0.0/0 への outbound 経路はない。
そんな環境で EKS も使いたくて方法を探した次第。

というわけで VPC endpoint を作成するのと eksctl で EKS cluster + node group を作る時にプロキシ設定を追加した。

VPC endpoint

まずは下記 9 個のエンドポイントを作成

com.amazonaws.ap-northeast-1.s3 (gateway)
com.amazonaws.ap-northeast-1.ec2
com.amazonaws.ap-northeast-1.sts
com.amazonaws.ap-northeast-1.ecr.api
com.amazonaws.ap-northeast-1.ecr.dkr
com.amazonaws.ap-northeast-1.eks
com.amazonaws.ap-northeast-1.eks-auth
com.amazonaws.ap-northeast-1.cloudformation
com.amazonaws.ap-northeast-1.autoscaling

s3 だけは gateway type にする。

AWS コンソールで作業するとメンドクサイので awscli を shell script で書いて EC2 の中でサクっと実行。

#!/bin/bash

# Variables
VPC_ID="vpc-xxxxxxxx"
SUBNET_IDS="subnet-xxxxxxxx,subnet-yyyyyyyy"
ROUTE_TABLE_ID="rtb-xxxxxxxx"
SECURITY_GROUP_ID="sg-xxxxxxxx"

# Create Gateway Endpoint for S3
aws ec2 create-vpc-endpoint \
    --vpc-id $VPC_ID \
    --service-name com.amazonaws.ap-northeast-1.s3 \
    --route-table-ids $ROUTE_TABLE_ID \
    --vpc-endpoint-type Gateway

# Create Interface Endpoints
ENDPOINT_SERVICES=(
    "com.amazonaws.ap-northeast-1.ec2"
    "com.amazonaws.ap-northeast-1.sts"
    "com.amazonaws.ap-northeast-1.ecr.api"
    "com.amazonaws.ap-northeast-1.ecr.dkr"
    "com.amazonaws.ap-northeast-1.eks"
    "com.amazonaws.ap-northeast-1.eks-auth"
    "com.amazonaws.ap-northeast-1.cloudformation"
    "com.amazonaws.ap-northeast-1.autoscaling"
)

for SERVICE in "${ENDPOINT_SERVICES[@]}"; do
    aws ec2 create-vpc-endpoint \
        --vpc-id $VPC_ID \
        --service-name $SERVICE \
        --vpc-endpoint-type Interface \
        --subnet-ids $SUBNET_IDS \
        --security-group-ids $SECURITY_GROUP_ID
done

echo "VPC Endpoints creation completed."

 

EKS

これもサクっと config.yaml 書いて eksctl で一発で実行。

プロキシ (10.0.0.8) は public subnet にあって、そこの squid proxy を経由して外部と通信する。
しかし各 AWS サービスはさきほど VPC endpoint を作ったのでプロキシ経由させる必要はなく NO_PROXY にする。

あと注意点は privateCluster を有効にする事で EKS エンドポイントにプライベート IP が振られる。
そうしないとパブリック IP になってしまって EC2 と通信出来ない。

config.yaml はこれ

accessConfig:
  authenticationMode: API_AND_CONFIG_MAP
apiVersion: eksctl.io/v1alpha5
iam:
  vpcResourceControllerPolicy: true
  withOIDC: false
kind: ClusterConfig
kubernetesNetworkConfig:
  ipFamily: IPv4
managedNodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 1
  disableIMDSv1: true
  disablePodIMDS: false
  iam:
    withAddonPolicies:
      albIngress: false
      appMesh: false
      appMeshPreview: false
      autoScaler: false
      awsLoadBalancerController: false
      certManager: false
      cloudWatch: false
      ebs: false
      efs: false
      externalDNS: false
      fsx: false
      imageBuilder: true
      xRay: false
  instanceType: t3.large
  labels:
    alpha.eksctl.io/cluster-name: oreno-eks
    alpha.eksctl.io/nodegroup-name: oreno-eks-ng
  maxSize: 1
  minSize: 1
  name: oreno-eks-ng
  preBootstrapCommands:
  - echo "export http_proxy=http://10.0.0.8:3128" | sudo tee -a /etc/environment
  - echo "export https_proxy=http://10.0.0.8:3128" | sudo tee -a /etc/environment
  - echo "export HTTP_PROXY=http://10.0.0.8:3128" | sudo tee -a /etc/environment
  - echo "export HTTPS_PROXY=10.0.0.8:3128" | sudo tee -a /etc/environment
  - echo "export no_proxy=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,localhost,127.0.0.1,169.254.169.254,.internal,sts.ap-northeast-1.amazonaws.com,ec2.ap-northeast-1.amazonaws.com,s3.ap-northeast-1.amazonaws.com,dkr.ecr.ap-northeast-1.amazonaws.com,api.ecr.ap-northeast-1.amazonaws.com,.ap-northeast-1.eks.amazonaws.com,eks.ap-northeast-1.amazonaws.com,*.s3.ap-northeast-1.amazonaws.com,cloudformation.ap-northeast-1.amazonaws.com,autoscaling.ap-northeast-1.amazonaws.com" | sudo tee -a /etc/environment
  - echo "export NO_PROXY=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,localhost,127.0.0.1,169.254.169.254,.internal,sts.ap-northeast-1.amazonaws.com,ec2.ap-northeast-1.amazonaws.com,s3.ap-northeast-1.amazonaws.com,dkr.ecr.ap-northeast-1.amazonaws.com,api.ecr.ap-northeast-1.amazonaws.com,.ap-northeast-1.eks.amazonaws.com,eks.ap-northeast-1.amazonaws.com,*.s3.ap-northeast-1.amazonaws.com,cloudformation.ap-northeast-1.amazonaws.com,autoscaling.ap-northeast-1.amazonaws.com" | sudo tee -a /etc/environment
  - echo "export AWS_DEFAULT_REGION=ap-northeast-1" | sudo tee -a /etc/environment
  - echo "export AWS_STS_REGIONAL_ENDPOINTS=regional" | sudo tee -a /etc/environment
  - sudo mkdir -p /etc/systemd/system/containerd.service.d
  - echo "[Service]" | sudo tee -a /etc/systemd/system/containerd.service.d/proxy.conf
  - echo 'Environment="HTTP_PROXY=http://10.0.0.8:3128"' | sudo tee -a /etc/systemd/system/containerd.service.d/proxy.conf
  - echo 'Environment="HTTPS_PROXY=http://10.0.0.8:3128"' | sudo tee -a /etc/systemd/system/containerd.service.d/proxy.conf
  - echo 'Environment="NO_PROXY=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,localhost,127.0.0.1,169.254.169.254,.internal,sts.ap-northeast-1.amazonaws.com,ec2.ap-northeast-1.amazonaws.com,s3.ap-northeast-1.amazonaws.com,dkr.ecr.ap-northeast-1.amazonaws.com,api.ecr.ap-northeast-1.amazonaws.com,.ap-northeast-1.eks.amazonaws.com,eks.ap-northeast-1.amazonaws.com,*.s3.ap-northeast-1.amazonaws.com,cloudformation.ap-northeast-1.amazonaws.com,autoscaling.ap-northeast-1.amazonaws.com"' | sudo tee -a /etc/systemd/system/containerd.service.d/proxy.conf
  - sudo mkdir -p /etc/systemd/system/sandbox-image.service.d
  - cat /etc/systemd/system/containerd.service.d/proxy.conf | sudo tee -a  /etc/systemd/system/sandbox-image.service.d/proxy.conf
  - sudo mkdir -p /etc/systemd/system/kubelet.service.d
  - cat /etc/systemd/system/containerd.service.d/proxy.conf | sudo tee -a /etc/systemd/system/kubelet.service.d/proxy.conf
  - source /etc/environment
  - sudo systemctl daemon-reload
  - sudo systemctl restart containerd
  - sudo systemctl restart sandbox-image
  - sudo systemctl restart kubelet
  privateNetworking: true
  securityGroups:
    attachIDs:
    - sg-xxxxxxx
    withLocal: null
    withShared: null
  ssh:
    allow: false
  tags:
    alpha.eksctl.io/nodegroup-name: oreno-eks-ng
    alpha.eksctl.io/nodegroup-type: managed
  volumeIOPS: 3000
  volumeSize: 50
  volumeThroughput: 125
  volumeType: gp3
metadata:
  name: oreno-eks
  region: ap-northeast-1
  version: '1.30'
privateCluster:
  enabled: true
  skipEndpointCreation: true
vpc:
  autoAllocateIPv6: false
  cidr: 10.0.0.0/16
  clusterEndpoints: null
  id: vpc-xxxxxxx
  manageSharedNodeSecurityGroupRules: true
  nat:
    gateway: Disable
  securityGroup: sg-xxxxxxx
  subnets:
    private:
      ap-northeast-1a:
        az: ap-northeast-1a
        cidr: 10.0.128.0/20
        id: subnet-xxxxxxx
      ap-northeast-1c:
        az: ap-northeast-1c
        cidr: 10.0.144.0/20
        id: subnet-yyyyyyy

これを EC2 インスタンスの中で eksctl に食わせる。

eksctl create cluster -f config.yaml

CloudFormation のコンソールに EKS cluster と EKS node group のスタックが出来上がったら完了。

これで EKS からもプロキシ経由の通信が可能になる。

参考 URL
https://repost.aws/knowledge-center/eks-http-proxy-containerd-automation