Mounting Amazon S3 Cloud Storage in Linux Virtual Machine.

Diagram

Amazon Simple Storage Service is storage for the Internet. It is designed to make web-scale computing easier for developers.

Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

AWS provides an API to work with Amazon S3 buckets using third-party applications. You can even write your own application that can interact with S3 buckets by using the Amazon API. You can create an application that uses the same path for uploading files to Amazon S3 cloud storage and provide the same path on each computer by mounting the S3 bucket to the same directory with S3FS.

S3FS, a special solution based on FUSE (file system in user space), was developed to mount S3 buckets to directories of Linux operating systems similarly to the way you mount CIFS or NFS share as a network drive. S3FS is a free and open source solution. After mounting Amazon S3 cloud storage with S3FS to your Linux machine, you can use cp, mv, rm, and other commands in the Linux console to operate with files as you do when working with mounted local or network drives. S3FS is written on Python and you can familiarize yourself with the source code on GitHub.

Let’s find out how to mount an Amazon S3 bucket to a Linux directory with Ubuntu 18.04. I have made one bash script through which it would be very easy to mount S3 Storage in Linux.

1. Firstly you need to generate the Access Key ID and Secret Access Key in the AWS web interface for your account (IAM user). The IAM user must have S3 full access. You can click here to create IAM user.
2. Create S3 Bucket and add some data in bucket. (link to create S3 Bucket)
3. Launch Virtual Machine with Ubuntu Linux. (link to launch VM)
4. Create a bash script name s3mount.sh with below content (replace mountpath, bucketname, dirname, AWS — Access_Key_ID and Secret_Acces_Key)

#!/bin/bash
#Which location you have to mount with s3:
mountpath=/home/ubuntu
#Give your S3 bucketname:
bucketname=mountdata
#Give your VM Directory name:
dirname=s3data
#Install s3fs package
sudo apt-get update
sudo apt-get remove fuse -y
sudo apt-get install s3fs -y
#AWS S3 access key and secretkey with S3full access
sudo echo  "Access_Key_ID:Secret_Access_Key"  > ${mountpath}/.passwd-s3fs
cat ${mountpath}/.passwd-s3fs
sudo chmod 600 ${mountpath}/.passwd-s3fs
sudo chmod 666 /etc/fuse.conf
# Skip Duplicacy in File:
if  grep -qe "^user_allow_other" /etc/fuse.conf
  then echo "FOUND"
  else sudo echo "user_allow_other" >> /etc/fuse.conf
fi
#Creating Directory:
sudo mkdir -p ${mountpath}/${dirname}
#Mount S3 with VM Directory:
sudo s3fs ${bucketname} ${mountpath}/${dirname} -o passwd_file=${mountpath}/.passwd-s3fs -o allow_other
# Skip Duplicacy in File:
if  grep -q "s3fs ${bucketname}  ${mountpath}/${dirname} -o passwd_file=${mountpath}/.passwd-s3fs -o allow_other" ~/.bashrc
  then echo "FOUND"
  else sudo echo "s3fs ${bucketname}  ${mountpath}/${dirname} -o passwd_file=${mountpath}/.passwd-s3fs -o allow_other"  >>  ~/.bashrc
fi
#Execute the .bashrc file
source ~/.bashrc
#Check s3 data is mounted with directory ot not:
sudo ls ${mountpath}/${dirname}

Now you can access S3 data in VM directory where all S3 data is being mounted.

Thank-you for reading.

If you find this article helpful, it would be appreciable if you could give 1 clap for it.