Securely Access S3 Buckets Using IAM Roles
Create IAM Role and Policy to Access S3 Bucket
-
Login to AWS console and go to IAM service.
-
Click the Roles from left menu and then click the Create role.
-
Under Select type of trusted entity, select the AWS service.
-
Under Choose a use case (that will use this role), select the EC2.
-
Click the Next: Permissions.
-
Click the Next: Tags.
-
Click the Next: Review.
-
Enter the Role name.
-
Click the Create role. The Role is created successfully. You will be nagivated to create role page with newly created role.
-
-
In the role list, click on the newly created role. Now, let’s add an inline policy to the role. This policy grants access to the S3 bucket.
-
Under Permissions tab, click the + Add inline policy.
-
Click the JSON tab
-
Copy the below policy and set ${s3_bucket_name} to the name of your bucket. Note: The policy in the JSON can be changed as per your requirement.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::${s3_bucket_name}" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::${s3_bucket_name}/*" ] } ] }
-
-
Click the Review Policy.
-
Enter the Policy name.
-
Click the Create policy.
-
In the role summary, copy the Instance Profile ARNs.
Add S3 IAM Role to EC2 IAM Role
-
Login to AWS console and go to IAM service.
-
Click the Policies from left menu and then click the Create policy.
-
Click the JSON tab.
- Copy the below policy and update ${iam_role_for_s3_access} with the role which you have created in above section (Create IAM Role and Policy to Access S3 Bucket).
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": "arn:aws:iam::xxxxxxxxxxxx:role/${iam_role_for_s3_access}" } ] }
-
Click the Review policy.
-
Enter the Policy name.
-
Click the Create policy. Now attach this policy to IAM role which was used to create the Databricks instance.
Add S3 IAM Role to Databricks
-
Login to Databricks and click on top-right menu.
-
Click the Admin Console.
-
Click the IAM Roles tab.
-
Click the +Add IAM Role.
-
Enter the Instance Profile ARN which you have created in step 1 Create IAM Role and Policy to Access S3 Bucket
Databricks validates that this Instance Profile ARN is both syntactically and semantically correct. To validate semantic correctness, Databricks does a dry run by launching a cluster with this IAM role. Any failure in this dry run produces a validation error in the UI.
-
Click Add.
-
You can specify the users who can launch clusters with the IAM role. (Optional)