[aws] Mounting through s3 - s3fs

2023.08 Update: Official Release of AWS S3 Mount Method as "Mountpoint for Amazon S3"

For detailed information about this solution, please refer to the AWS Blog link. We plan to test it and share our findings.

https://aws.amazon.com/ko/s3/features/mountpoint/

Summary

We want to use s3fs to mount an S3 Bucket on an EC2 Instance. How can we proceed?

Solution

*Before I provide an answer, please note that s3fs is not an official solution supported by AWS; it is a third-party solution.

• s3fs may have intermittent connection issues. Therefore, it is not recommended for production use.
• If you want to mount an S3 Bucket to an EC2 Instance for production use, it's recommended to use Amazon EFS.

Here's how you can use s3fs to mount S3 on an EC2 instance.

[Prerequisites]

Services: EC2 Instance, S3 Bucket

Below are the steps for your reference.

1. Define a policy granting access to the S3 bucket for the EC2 server.

The IAM policy granted is as follows:

• S3 Bucket: sohyun-bucket-test
• IAM Role: sohyun-s3-mount-test-role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::sohyun-bucket-test"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::sohyun-bucket-test/*"
        }
    ]
}

This role grants permissions to perform actions such as listing, uploading, downloading, deleting, and managing ACLs for a specific S3 bucket. You can modify permissions as needed.

[Testing Process]

1. Install s3fs

2. # yum update
   # yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
   # git clone https://github.com/s3fs-fuse/s3fs-fuse.git
   # cd s3fs-fuse
   # ./autogen.sh
   # ./configure — prefix=/usr — with-openssl
   # make 
   # sudo make install

   


2. Create a directory for mounting.

3. # mkdir /backup_data

   


3. Mount S3.

4. # s3fs -o iam_role="sohyun-s3-mount-test-role" -o url=https://s3-us-west-2.amazonaws.com -o endpoint=us-west-2 -o allow_other -o use_cache=/tmp sohyun-bucket-test /backup_data

   


Explanation of options used:

• -o iam_role="sohyun-s3-mount-test-role": Specifies that s3fs should access the S3 bucket using the IAM role "sohyun-s3-mount-test-role," allowing access to S3 without authentication, using the permissions assigned to the role.
• -o url=https://s3-us-west-2.amazonaws.com: Specifies the endpoint URL of S3. In this case, it uses S3 in the "us-west-2" region.
• -o endpoint=us-west-2: Specifies the AWS region where the S3 bucket is located, instructing s3fs to use this region.
• -o allow_other: Allows other users to access the mounted file system, enabling other users to access the mounted S3 bucket.
• -o use_cache=/tmp: Specifies the directory where s3fs caches data. In this case, it maintains the cache in the "/tmp" directory.

Verification

Once you create a test.txt file on the server, you will see the same file uploaded directly to the S3 bucket console.

Remember to adjust permissions and configurations according to your specific use case.

 test.txt 파일을 생성하면, s3 

Reference

▶ GitHub Repository for s3fs