Create AWS Resource for Covalent
1.create AWS root user and IAM user
open AWS web portal and register root user account. https://portal.aws.amazon.com/billing/signup
Sign in AWS web portal with root user
Open AWS IAM web console, click "create user"
enter username,choose "Custom password", then input your password over here,click "next" button.
On Set permissions window,choose AdministratorAccess and IAMUserChangePassword permissions for user, then click "next" button.
In the end, check the user and permissions, if all things are correct, click "Create user" button to create IAM user.
2. create access key and secret key for IAM user
Sign in AWS web portal with IAM user
Create Security credentials for IAM user
Enter tag and click "Create access key" button to make access key and secret access key.
NOTE:
Please keep the access key and secret access key, it will be used in below steps.
2. Install AWS CLI and configuration
Refer this link to install AWS CLI tool https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
$ aws configure
AWS Access Key ID [None]: xxxxxxxxxxxxxxxxxxx
AWS Secret Access Key [None]: xxxxxxxxxxxxxxxxxxx
Default region name [None]: us-east-1
Default output format [None]: json
The credential file is saved in following file path.
$ cat ~/.aws/credentials3. Install Terraform
Refer this link to install terraform tool https://developer.hashicorp.com/terraform/downloads?product_intent=terraform
4. Install docker-ce
#Step 1. Enable docker-ce repo:
dnf install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#Step 2. Install docker-ce:
dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
#Step 3. Enable docker services
systemctl enable docker --now5. Download terraform scripts for AWS Lambda
Download terraform scripts.tar to HOME folder:
Extract the files from scripts.tar
tar -xvf scripts.tar
mv scripts/ awslambda_terraform_scripts/6. Configuration
cd awslambda_terraform_scripts/
vim variables.tf
variable "name" {
default = "covalent-lambda"
description = "Prefix to use for all provisioned resources"
}
variable "executor_base_image_tag_name" {
default = "stable"
description = "Image tag for image in provisioned ecr repo to be used for lambda invocations"
}
variable "aws_region" {
default = "us-east-1"
description = "The aws region"
}
variable "timeout" {
default = 900
description = "The amount of time your Lambda Function has to run in seconds"
}
variable "memory_size" {
default = 1024
description = "The amount of memory in MB your Lambda Function can use at runtime"
}
variable "ephemeral_storage" {
default = 1024
description = "Size of the ephemeral storage in MB"
}
Note:
name: default value is covalent-lambda, if running terraform command "terraform apply" occur "BucketAlreadyExists" error, please change a new name .
7. Executing terraform script files to provision required resource for AWS Lambda
Initial terraform environment
# terraform init
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.17.0
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Generate the execution plan
# terraform plan
data.aws_caller_identity.current: Reading...
data.aws_caller_identity.current: Read complete after 0s [id=280167912684]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_ecr_repository.ecr_repository will be created
+ resource "aws_ecr_repository" "ecr_repository" {
+ arn = (known after apply)
+ force_delete = true
+ id = (known after apply)
+ image_tag_mutability = "MUTABLE"
+ name = "covalent-lambda-lambda-executor-base-ecr-repo"
+ registry_id = (known after apply)
+ repository_url = (known after apply)
+ tags_all = (known after apply)
+ image_scanning_configuration {
+ scan_on_push = false
}
}
# aws_iam_role.lambda_iam_role will be created
+ resource "aws_iam_role" "lambda_iam_role" {
+ arn = (known after apply)
+ assume_role_policy = jsonencode(
{
+ Statement = [
+ {
+ Action = "sts:AssumeRole"
+ Effect = "Allow"
+ Principal = {
+ Service = "lambda.amazonaws.com"
}
+ Sid = ""
},
]
+ Version = "2012-10-17"
}
)
+ create_date = (known after apply)
+ force_detach_policies = false
+ id = (known after apply)
+ managed_policy_arns = [
+ "arn:aws:iam::aws:policy/AWSLambdaExecute",
]
+ max_session_duration = 3600
+ name = "covalent-lambda-lambda-iam-role"
+ name_prefix = (known after apply)
+ path = "/"
+ tags = {
+ "Terraform" = "true"
}
+ tags_all = {
+ "Terraform" = "true"
}
+ unique_id = (known after apply)
}
# aws_lambda_function.lambda will be created
+ resource "aws_lambda_function" "lambda" {
+ architectures = (known after apply)
+ arn = (known after apply)
+ function_name = "covalent-lambda-lambda-fn"
+ id = (known after apply)
+ image_uri = (known after apply)
+ invoke_arn = (known after apply)
+ last_modified = (known after apply)
+ memory_size = 1024
+ package_type = "Image"
+ publish = false
+ qualified_arn = (known after apply)
+ qualified_invoke_arn = (known after apply)
+ reserved_concurrent_executions = -1
+ role = (known after apply)
+ signing_job_arn = (known after apply)
+ signing_profile_version_arn = (known after apply)
+ skip_destroy = false
+ source_code_hash = (known after apply)
+ source_code_size = (known after apply)
+ tags_all = (known after apply)
+ timeout = 900
+ version = (known after apply)
+ ephemeral_storage {
+ size = 1024
}
}
# aws_s3_bucket.s3_bucket will be created
+ resource "aws_s3_bucket" "s3_bucket" {
+ acceleration_status = (known after apply)
+ acl = (known after apply)
+ arn = (known after apply)
+ bucket = "covalent-lambda-covalent-artifact-bucket"
+ bucket_domain_name = (known after apply)
+ bucket_prefix = (known after apply)
+ bucket_regional_domain_name = (known after apply)
+ force_destroy = true
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ object_lock_enabled = (known after apply)
+ policy = (known after apply)
+ region = (known after apply)
+ request_payer = (known after apply)
+ tags_all = (known after apply)
+ website_domain = (known after apply)
+ website_endpoint = (known after apply)
}
Plan: 4 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ function_name = "covalent-lambda-lambda-fn"
+ s3_bucket_name = (known after apply)
Create the AWS Lambda resources
# terraform apply -auto-approve
data.aws_caller_identity.current: Reading...
data.aws_caller_identity.current: Read complete after 0s [id=280167912684]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_ecr_repository.ecr_repository will be created
+ resource "aws_ecr_repository" "ecr_repository" {
+ arn = (known after apply)
+ force_delete = true
+ id = (known after apply)
+ image_tag_mutability = "MUTABLE"
+ name = "covalent-lambda-lambda-executor-base-ecr-repo"
+ registry_id = (known after apply)
+ repository_url = (known after apply)
+ tags_all = (known after apply)
+ image_scanning_configuration {
+ scan_on_push = false
}
}
# aws_iam_role.lambda_iam_role will be created
+ resource "aws_iam_role" "lambda_iam_role" {
+ arn = (known after apply)
+ assume_role_policy = jsonencode(
{
+ Statement = [
+ {
+ Action = "sts:AssumeRole"
+ Effect = "Allow"
+ Principal = {
+ Service = "lambda.amazonaws.com"
}
+ Sid = ""
},
]
+ Version = "2012-10-17"
}
)
+ create_date = (known after apply)
+ force_detach_policies = false
+ id = (known after apply)
+ managed_policy_arns = [
+ "arn:aws:iam::aws:policy/AWSLambdaExecute",
]
+ max_session_duration = 3600
+ name = "covalent-lambda-lambda-iam-role"
+ name_prefix = (known after apply)
+ path = "/"
+ tags = {
+ "Terraform" = "true"
}
+ tags_all = {
+ "Terraform" = "true"
}
+ unique_id = (known after apply)
}
# aws_lambda_function.lambda will be created
+ resource "aws_lambda_function" "lambda" {
+ architectures = (known after apply)
+ arn = (known after apply)
+ function_name = "covalent-lambda-lambda-fn"
+ id = (known after apply)
+ image_uri = (known after apply)
+ invoke_arn = (known after apply)
+ last_modified = (known after apply)
+ memory_size = 1024
+ package_type = "Image"
+ publish = false
+ qualified_arn = (known after apply)
+ qualified_invoke_arn = (known after apply)
+ reserved_concurrent_executions = -1
+ role = (known after apply)
+ signing_job_arn = (known after apply)
+ signing_profile_version_arn = (known after apply)
+ skip_destroy = false
+ source_code_hash = (known after apply)
+ source_code_size = (known after apply)
+ tags_all = (known after apply)
+ timeout = 900
+ version = (known after apply)
+ ephemeral_storage {
+ size = 1024
}
}
# aws_s3_bucket.s3_bucket will be created
+ resource "aws_s3_bucket" "s3_bucket" {
+ acceleration_status = (known after apply)
+ acl = (known after apply)
+ arn = (known after apply)
+ bucket = "covalent-lambda-covalent-artifact-bucket"
+ bucket_domain_name = (known after apply)
+ bucket_prefix = (known after apply)
+ bucket_regional_domain_name = (known after apply)
+ force_destroy = true
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ object_lock_enabled = (known after apply)
+ policy = (known after apply)
+ region = (known after apply)
+ request_payer = (known after apply)
+ tags_all = (known after apply)
+ website_domain = (known after apply)
+ website_endpoint = (known after apply)
}
Plan: 4 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ function_name = "covalent-lambda-lambda-fn"
+ s3_bucket_name = (known after apply)
aws_ecr_repository.ecr_repository: Creating...
aws_iam_role.lambda_iam_role: Creating...
aws_s3_bucket.s3_bucket: Creating...
aws_ecr_repository.ecr_repository: Provisioning with 'local-exec'...
aws_ecr_repository.ecr_repository (local-exec): Executing: ["/bin/sh" "-c" "docker pull public.ecr.aws/covalent/covalent-lambda-executor:stable && aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 280167912684.dkr.ecr.us-east-1.amazonaws.com && docker tag public.ecr.aws/covalent/covalent-lambda-executor:stable 280167912684.dkr.ecr.us-east-1.amazonaws.com/covalent-lambda-lambda-executor-base-ecr-repo:stable && docker push 280167912684.dkr.ecr.us-east-1.amazonaws.com/covalent-lambda-lambda-executor-base-ecr-repo:stable"]
aws_ecr_repository.ecr_repository (local-exec): Trying to pull public.ecr.aws/covalent/covalent-lambda-executor:stable...
aws_iam_role.lambda_iam_role: Creation complete after 1s [id=covalent-lambda-lambda-iam-role]
aws_ecr_repository.ecr_repository (local-exec): Getting image source signatures
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:34b7c205d4feb46c0f56041c2a1eb5aa2f99af04a3b29a418bc2847a49074f01
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:eaa89787764f0d8c8485166dfb4ff7f5ff2909fd336cb4391a9d6c064308ebe5
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:b1b45f1ab1972d8e4c70ac3e2f57e632c83a1a7f6038dc8dfb356ee5ad540aeb
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:9377b4a0d06d83adc4f1747f1d2381f039a386d284afb8e6e5ab025e2983191d
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:f0d77123acbc898092f3c405bd66cb88e9fa9d6b0a0aca53e8793234eb909ee1
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:4500a762c54620411ae491a547c66b61d577c1369ecbf5a7e91b4e153181854b
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:d0e5195f1f50024aa35a21de39155d502e7c55e4b20dd68cd308075f0ad5bcc3
aws_ecr_repository.ecr_repository (local-exec): Copying config sha256:5e290d5b208e549ef13c989b22159ab4cb44dbd44d97a8e5e49a834cfe2eef10
aws_ecr_repository.ecr_repository (local-exec): Writing manifest to image destination
aws_ecr_repository.ecr_repository (local-exec): Storing signatures
aws_ecr_repository.ecr_repository (local-exec): 5e290d5b208e549ef13c989b22159ab4cb44dbd44d97a8e5e49a834cfe2eef10
aws_ecr_repository.ecr_repository (local-exec): Login Succeeded!
aws_ecr_repository.ecr_repository (local-exec): Getting image source signatures
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:9938cb82df4c8bb1de233b15e2dfc9cd1e365ab13c6d8c2114d878c96fdbe2a7
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:a8f9e3c1138b250cabb1c0ad30307eb2ffc50334303074196fad464260086738
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:51e998f9a3d78e2ac32603742620e049270b301fb8e2eb8cefd4ad9f955f69f5
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:570d66224c69354daeb8110be8f4759b2c3c8937c39dd7c2b5202c1076258a8c
aws_ecr_repository.ecr_repository: Still creating... [10s elapsed]
aws_s3_bucket.s3_bucket: Still creating... [10s elapsed]
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:7c769e6a9a6c7e19d745878cd9447a12021fcc306407f5ea070009db4f01f248
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:9d3db3fe69d8386cf4fa4b0b923ea3d07e7a64301c8d6efd6444062a4f82fba3
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
aws_ecr_repository.ecr_repository: Still creating... [20s elapsed]
aws_s3_bucket.s3_bucket: Still creating... [20s elapsed]
aws_s3_bucket.s3_bucket: Creation complete after 20s [id=covalent-lambda-covalent-artifact-bucket]
aws_ecr_repository.ecr_repository (local-exec): Copying blob sha256:74f8a626d86ff49780a3ed3618155eb37a91a357fb9a5f99735a334810ecc00f
aws_ecr_repository.ecr_repository: Still creating... [30s elapsed]
aws_ecr_repository.ecr_repository (local-exec): Copying config sha256:5e290d5b208e549ef13c989b22159ab4cb44dbd44d97a8e5e49a834cfe2eef10
aws_ecr_repository.ecr_repository (local-exec): Writing manifest to image destination
aws_ecr_repository.ecr_repository (local-exec): Storing signatures
aws_ecr_repository.ecr_repository: Creation complete after 34s [id=covalent-lambda-lambda-executor-base-ecr-repo]
aws_lambda_function.lambda: Creating...
aws_lambda_function.lambda: Still creating... [10s elapsed]
aws_lambda_function.lambda: Still creating... [20s elapsed]
aws_lambda_function.lambda: Creation complete after 20s [id=covalent-lambda-lambda-fn]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
Outputs:
function_name = "covalent-lambda-lambda-fn"
s3_bucket_name = "covalent-lambda-covalent-artifact-bucket"
Note:
Please keep the function_name and s3-bucket_name, it will be used in code of covalent aws lambda executor.
Add covalent executor for AWS Lambda in LiCO web portal
In the AWS Lambda Executor form, filled with data which Terraform apply outputs, then Covalent template can use AWS Lambda Executor.
Delete AWS Lambda resource
If you don't need to use AWS Lambda resource, you can use following command to delete resources.
# terraform destroy -auto-approve
data.aws_caller_identity.current: Reading...
aws_iam_role.lambda_iam_role: Refreshing state... [id=covalent-lambda-lambda-iam-role]
aws_s3_bucket.s3_bucket: Refreshing state... [id=covalent-lambda-covalent-artifact-bucket]
data.aws_caller_identity.current: Read complete after 1s [id=280167912684]
aws_ecr_repository.ecr_repository: Refreshing state... [id=covalent-lambda-lambda-executor-base-ecr-repo]
aws_lambda_function.lambda: Refreshing state... [id=covalent-lambda-lambda-fn]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
# aws_ecr_repository.ecr_repository will be destroyed
- resource "aws_ecr_repository" "ecr_repository" {
- arn = "arn:aws:ecr:us-east-1:280167912684:repository/covalent-lambda-lambda-executor-base-ecr-repo" -> null
- force_delete = true -> null
- id = "covalent-lambda-lambda-executor-base-ecr-repo" -> null
- image_tag_mutability = "MUTABLE" -> null
- name = "covalent-lambda-lambda-executor-base-ecr-repo" -> null
- registry_id = "280167912684" -> null
- repository_url = "280167912684.dkr.ecr.us-east-1.amazonaws.com/covalent-lambda-lambda-executor-base-ecr-repo" -> null
- tags = {} -> null
- tags_all = {} -> null
- encryption_configuration {
- encryption_type = "AES256" -> null
}
- image_scanning_configuration {
- scan_on_push = false -> null
}
}
# aws_iam_role.lambda_iam_role will be destroyed
- resource "aws_iam_role" "lambda_iam_role" {
- arn = "arn:aws:iam::280167912684:role/covalent-lambda-lambda-iam-role" -> null
- assume_role_policy = jsonencode(
{
- Statement = [
- {
- Action = "sts:AssumeRole"
- Effect = "Allow"
- Principal = {
- Service = "lambda.amazonaws.com"
}
- Sid = ""
},
]
- Version = "2012-10-17"
}
) -> null
- create_date = "2023-10-24T08:43:46Z" -> null
- force_detach_policies = false -> null
- id = "covalent-lambda-lambda-iam-role" -> null
- managed_policy_arns = [
- "arn:aws:iam::aws:policy/AWSLambdaExecute",
] -> null
- max_session_duration = 3600 -> null
- name = "covalent-lambda-lambda-iam-role" -> null
- path = "/" -> null
- tags = {
- "Terraform" = "true"
} -> null
- tags_all = {
- "Terraform" = "true"
} -> null
- unique_id = "AROAUCO2PCTWD7E5B3VWO" -> null
}
# aws_lambda_function.lambda will be destroyed
- resource "aws_lambda_function" "lambda" {
- architectures = [
- "x86_64",
] -> null
- arn = "arn:aws:lambda:us-east-1:280167912684:function:covalent-lambda-lambda-fn" -> null
- function_name = "covalent-lambda-lambda-fn" -> null
- id = "covalent-lambda-lambda-fn" -> null
- image_uri = "280167912684.dkr.ecr.us-east-1.amazonaws.com/covalent-lambda-lambda-executor-base-ecr-repo:stable" -> null
- invoke_arn = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:280167912684:function:covalent-lambda-lambda-fn/invocations" -> null
- last_modified = "2023-10-24T08:44:20.143+0000" -> null
- layers = [] -> null
- memory_size = 1024 -> null
- package_type = "Image" -> null
- publish = false -> null
- qualified_arn = "arn:aws:lambda:us-east-1:280167912684:function:covalent-lambda-lambda-fn:$LATEST" -> null
- qualified_invoke_arn = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:280167912684:function:covalent-lambda-lambda-fn:$LATEST/invocations" -> null
- reserved_concurrent_executions = -1 -> null
- role = "arn:aws:iam::280167912684:role/covalent-lambda-lambda-iam-role" -> null
- skip_destroy = false -> null
- source_code_hash = "306b0a64e72b1efeb9e75ffa587a269828b6bb61d287fd5c1926f1086946c064" -> null
- source_code_size = 0 -> null
- tags = {} -> null
- tags_all = {} -> null
- timeout = 900 -> null
- version = "$LATEST" -> null
- ephemeral_storage {
- size = 1024 -> null
}
- tracing_config {
- mode = "PassThrough" -> null
}
}
# aws_s3_bucket.s3_bucket will be destroyed
- resource "aws_s3_bucket" "s3_bucket" {
- arn = "arn:aws:s3:::covalent-lambda-covalent-artifact-bucket" -> null
- bucket = "covalent-lambda-covalent-artifact-bucket" -> null
- bucket_domain_name = "covalent-lambda-covalent-artifact-bucket.s3.amazonaws.com" -> null
- bucket_regional_domain_name = "covalent-lambda-covalent-artifact-bucket.s3.us-east-1.amazonaws.com" -> null
- force_destroy = true -> null
- hosted_zone_id = "Z3AQBSTGFYJSTF" -> null
- id = "covalent-lambda-covalent-artifact-bucket" -> null
- object_lock_enabled = false -> null
- region = "us-east-1" -> null
- request_payer = "BucketOwner" -> null
- tags = {} -> null
- tags_all = {} -> null
- grant {
- id = "6ffc8229dfe6d3e57a26dc29b3f2e0f1b518c0464958fd495cec6ba1b88cc02d" -> null
- permissions = [
- "FULL_CONTROL",
] -> null
- type = "CanonicalUser" -> null
}
- server_side_encryption_configuration {
- rule {
- bucket_key_enabled = false -> null
- apply_server_side_encryption_by_default {
- sse_algorithm = "AES256" -> null
}
}
}
- versioning {
- enabled = false -> null
- mfa_delete = false -> null
}
}
Plan: 0 to add, 0 to change, 4 to destroy.
Changes to Outputs:
- function_name = "covalent-lambda-lambda-fn" -> null
- s3_bucket_name = "covalent-lambda-covalent-artifact-bucket" -> null
aws_lambda_function.lambda: Destroying... [id=covalent-lambda-lambda-fn]
aws_s3_bucket.s3_bucket: Destroying... [id=covalent-lambda-covalent-artifact-bucket]
aws_lambda_function.lambda: Destruction complete after 0s
aws_ecr_repository.ecr_repository: Destroying... [id=covalent-lambda-lambda-executor-base-ecr-repo]
aws_iam_role.lambda_iam_role: Destroying... [id=covalent-lambda-lambda-iam-role]
aws_s3_bucket.s3_bucket: Destruction complete after 1s
aws_ecr_repository.ecr_repository: Destruction complete after 1s
aws_iam_role.lambda_iam_role: Destruction complete after 1s
Destroy complete! Resources: 4 destroyed.