Securely Access S3 Buckets Using IAM Roles

Create IAM Role and Policy to Access S3 Bucket

1. Login to AWS console and go to IAM service.

2. Click the Roles from left menu and then click the Create role.

   • Under Select type of trusted entity, select the AWS service.

   • Under Choose a use case (that will use this role), select the EC2.

   

   • Click the Next: Permissions.

   • Click the Next: Tags.

   • Click the Next: Review.

   • Enter the Role name.

   • Click the Create role. The Role is created successfully. You will be nagivated to create role page with newly created role.

3. In the role list, click on the newly created role. Now, let’s add an inline policy to the role. This policy grants access to the S3 bucket.

   • Under Permissions tab, click the + Add inline policy.

   • Click the JSON tab

   • Copy the below policy and set ${s3_bucket_name} to the name of your bucket. Note: The policy in the JSON can be changed as per your requirement.

     {
       "Version": "2012-10-17", "Statement": [
         {
           "Effect": "Allow", "Action": [
             "s3:ListBucket"
           ], "Resource": [
             "arn:aws:s3:::${s3_bucket_name}"
           ]
         }, {
           "Effect": "Allow", "Action": [
             "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:PutObjectAcl"
           ], "Resource": [
             "arn:aws:s3:::${s3_bucket_name}/*"
           ]
         }
       ]
     }
   

4. Click the Review Policy.

5. Enter the Policy name.

6. Click the Create policy.

7. In the role summary, copy the Instance Profile ARNs.

Add S3 IAM Role to EC2 IAM Role

1. Login to AWS console and go to IAM service.

2. Click the Policies from left menu and then click the Create policy.

3. Click the JSON tab.

   • Copy the below policy and update ${iam_role_for_s3_access} with the role which you have created in above section (Create IAM Role and Policy to Access S3 Bucket).

   {
       "Version": "2012-10-17", "Statement": [
           {
               "Effect": "Allow", "Action": [
                   "iam:PassRole"
               ], "Resource": "arn:aws:iam::xxxxxxxxxxxx:role/${iam_role_for_s3_access}"
           }
       ]
   }
   

4. Click the Review policy.

5. Enter the Policy name.

6. Click the Create policy. Now attach this policy to IAM role which was used to create the Databricks instance.

Add S3 IAM Role to Databricks

1. Login to Databricks and click on top-right menu.

2. Click the Admin Console.

3. Click the IAM Roles tab.

4. Click the +Add IAM Role.

5. Enter the Instance Profile ARN which you have created in step 1 Create IAM Role and Policy to Access S3 Bucket

   

   Databricks validates that this Instance Profile ARN is both syntactically and semantically correct. To validate semantic correctness, Databricks does a dry run by launching a cluster with this IAM role. Any failure in this dry run produces a validation error in the UI.

6. Click Add.

7. You can specify the users who can launch clusters with the IAM role. (Optional)

   

Launch Cluster with S3 IAM Role

1. Login to Databricks and click the Clusters from left menu.

2. Select or create a cluster.

3. Expand the Advanced Options section, under Instances tab, select the IAM role from the IAM Role drop-down list. This drop-down includes all of the IAM roles that are available for the cluster.