aws(學習筆記第三十一課)

• aws cdk深入學習

學習內容：

• 深入練習aws cdk下部署batch-arm64-instance-type

---

1. 深入練習aws cdk下部署batch-arm64-instance-type

1. 代碼鏈接

   • 代碼鏈接
     代碼鏈接 -> batch-arm64-instance-type
   • 之前代碼學習
     之前學習代碼鏈接 -> aws(學習筆記第十八課) 使用aws cdk(python)進行部署

2. batch-arm64-instance-type代碼測試

   • setup環境
     • setup virtual environment

       python -m venv .venv
       

     • 激活python virtual environment

       source .venv/Scripts/activate #這里使用的是git bash
       

     • 安裝python的依賴包

        pip install -r requirements.txt
       

     • 代碼中cdk.json的python3進行替換
       這里的phthon3沒有配置，所以不能執行python3，講這里改成了python。
       

3. 代碼執行

   • 預先執行cdk bootstrap
cdk提供了bootstrap命令來創建cdk執行需要的前提條件。

     CDKToolkit: creating CloudFormation changeset...
     CDKToolkit |  0/12 | 21:22:46 | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit User Initiated
     CDKToolkit |  0/12 | 21:22:52 | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit User Initiated
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | FilePublishingRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | CloudFormationExecutionRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::ECR::Repository    | ContainerAssetsRepository
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | StagingBucket
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | ImagePublishingRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | LookupRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::SSM::Parameter     | CdkBootstrapVersion
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::SSM::Parameter     | CdkBootstrapVersion Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | FilePublishingRole Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::ECR::Repository    | ContainerAssetsRepository Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | CloudFormationExecutionRole Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | ImagePublishingRole Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | StagingBucket Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | LookupRole Resource creation Initiated
     CDKToolkit |  1/12 | 21:22:57 | CREATE_COMPLETE      | AWS::SSM::Parameter     | CdkBootstrapVersion
     CDKToolkit |  2/12 | 21:22:58 | CREATE_COMPLETE      | AWS::ECR::Repository    | ContainerAssetsRepository
     CDKToolkit |  3/12 | 21:23:12 | CREATE_COMPLETE      | AWS::S3::Bucket         | StagingBucket
     CDKToolkit |  3/12 | 21:23:14 | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy   | StagingBucketPolicy
     CDKToolkit |  3/12 | 21:23:15 | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy   | StagingBucketPolicy Resource creation Initiated
     CDKToolkit |  4/12 | 21:23:15 | CREATE_COMPLETE      | AWS::S3::BucketPolicy   | StagingBucketPolicy
     CDKToolkit |  5/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | ImagePublishingRole
     CDKToolkit |  6/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | FilePublishingRole
     CDKToolkit |  7/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | CloudFormationExecutionRole
     CDKToolkit |  7/12 | 21:23:16 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy
     CDKToolkit |  8/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | LookupRole
     CDKToolkit |  8/12 | 21:23:17 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy
     CDKToolkit |  8/12 | 21:23:17 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | DeploymentActionRole
     CDKToolkit |  8/12 | 21:23:18 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy Resource creation Initiated
     CDKToolkit |  8/12 | 21:23:18 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy Resource creation Initiated
     CDKToolkit |  8/12 | 21:23:19 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | DeploymentActionRole Resource creation Initiated
     CDKToolkit |  9/12 | 21:23:34 | CREATE_COMPLETE      | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy
     CDKToolkit | 10/12 | 21:23:34 | CREATE_COMPLETE      | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy
     CDKToolkit | 11/12 | 21:23:38 | CREATE_COMPLETE      | AWS::IAM::Role          | DeploymentActionRole
     CDKToolkit | 12/12 | 21:23:39 | CREATE_COMPLETE      | AWS::CloudFormation::Stack | CDKToolkit?  Environment aws://081353481087/ap-northeast-1 bootstrapped.
     

     這里，cdk bootstrap命令執行完之后，會生成一個CDKToolkit名字的cloudformation。
     
     如果由于平時的誤刪除之類的操作，會使CDKToolkit的創建的cdk deploy需要的前提條件遭到破壞。這時如果重新執行cdk bootstrap會導致沒有差分執行出來。這時候需要將CDKToolkit這個cloudformation刪除掉，之后執行cdk bootstrap命令，這樣就會重新生成cdk deploy執行的所有條件。

   • 執行cdk --require-approval never deploy
     如果執行不加上--require-approval never，會導致錯誤，所以這里加上該選項。

   • Trouble Shouting
     如果存在默認的vpc，這里會出現如下錯誤。
     原因在于:
     Fn:GetAZs這個函數調用的話，會只返回default VPC的Available Zone的數量，從而影響cdk的執行。
     對策：
     所以要刪除default VPC，不過不用擔心，default VPC可以重新創建，大可放心。

4. 代碼解析

   • 全體架構
     

   • 作成VPC

      # This resource alone will create a private/public subnet in each AZ as well as nat/internet gateway(s)
     vpc = ec2.Vpc(self, "VPC")
     

     創建一下service組件：

     • 一個VPC
     • 一個public subnet
     • 一個private subnet
     • 一個Internet Gateway
     • 一個NAT Gateway Compute Environment通過這個NAT Gateway訪問internet
       

   • 創建Job Queue

     # Create AWS Batch Job Queue
     self.batch_queue = batch.JobQueue(self, "JobQueueArm64")
     

     

   • 創建Compute Environment，并將每個Compute Environment分配給JobQueue

             # For loop to create Batch Compute Environmentsfor i in range(count):name = "MyBatchARM64Env" + str(i)batch_environment = batch.ManagedEc2EcsComputeEnvironment(self, name,spot=True,spot_bid_percentage=75,instance_types=[ec2.InstanceType("c7g.medium"),ec2.InstanceType("c7g.large")],use_optimal_instance_classes=False,vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PRIVATE_WITH_NAT),vpc=vpc)self.batch_queue.add_compute_environment(batch_environment, i)
     

     

   • 創建Job Defintion

            # Create ECS Job Definition to submit job in batch job queue.batch_jobDef = batch.EcsJobDefinition(self, "MyJobDefArm64",container=batch.EcsEc2ContainerDefinition(self, "CDKJobDefArm64",image=ecs.ContainerImage.from_registry("public.ecr.aws/amazonlinux/amazonlinux:latest"),command=["sleep", "60"],memory=Size.mebibytes(512),cpu=1))
     

     

5. 執行job

   • 投入新的Job
     
   • Job執行成功