This is a drive through the requirement to install an AWS S3 Bucket on a Virtual Machine, on your PC and provisioned by VirtualBox, with an FTP service that can make use of the AWS S3 Bucket for persistent distributed storage. The reason to have a S3 bucket is to allow the use of a single persistent storage for several services (for instance multiple FTP servers) running for different clients and exposed by a dedicated virtual machine.
Disclaimer: Please note this is just a development exercise and this tutorial should not be used to expose a production FTP Service. Use at your own risk
After running the steps on this article you will obtain:
- a VM on VirtualBox or EC2 straight from a Vagrant File
- Your VM will have a S3 bucket mounted at /s3mnt
- the FTP is exposed but notice that it is running with its default configuration and thus if logging in with the root user then you can browse to location /s3mnt to read or write the files to and from S3
- for simplicity the solution uses bash scripting provisioning so it is not the most elegant way though it does the job
Requirements:
And now lets start by
- Clone the project from github at https://github.com/bizmate/s3_mount
git clone https://github.com/bizmate/s3_mount.git
- Configure the environment variables required to mount a S3 bucket
You can also read what environment variables are required directly on the README file on the github repository
AWS_S3_ACCESS_KEY=myverylongkey
AWS_S3_ACCESS_SECRET=myverylongsecret
AWS_S3_BUCKET=myAwsS3BucketName
export AWS_S3_ACCESS_KEY
export AWS_S3_ACCESS_SECRET
export AWS_S3_BUCKET
- Provision the VM on virtualbox using vagrant by running
cd s3_mount
vagrant up
Vagrant will first of all download a ubuntu VM, start it and run the bash script for provisioning.
If you have not configured the environment variable the provisioning will not run and exit.
Also Both the vagrant file and the bash script will print on the command the environment variables values you set up in step 2, thus you can verify there.
For instance if they are emtpy you will see no the first few lines
Running S3 mount install with provider: virtualbox KEY SECRET on Bucket
#and in the last two lines
==> default: S3 mount provisioning running in : /vagrant/provision with KEY SECRET on Bucket
==> default: Please set up the access environment variables as described in README
- Verify the FTP service is working and upload a file
On your vagrant/virtualbox machine you should be able to ftp on the machine with user and password of both ‘vagrant’ on IP ‘192.168.53.70‘. Notice this is the fixed IP used in the Vagrantfile and it is an internal ip on the virtual network provided by VirtualBox on your local PC. The FTP user should load the files from location /home/vagrant when you login. From there you can still browse to location /s3mnt. Once here try to upload a file, a simple text file will do it.
- Check that the files loaded on the S3 Bucket are saved correctly.
Use a separate client to check the S3 bucket is receiving the files. I have used clients on OSX like Cyberduck but also you should be able to use another client available for you OS.
- Optional – If you want to provision an instance on AWS EC2 you need to set up all the environment variables required by the Vagrant provisioner. The variables are visible here – https://github.com/bizmate/s3_mount/blob/master/Vagrantfile#L25-L36 .
I will not describe this in details in this article but in short you will need to set up the key access and access group for your EC2 to run the provisioning. Notice the Vagrant file uses by default a ‘eu-west-1‘, ‘t1.micro‘ as zone and instance type.
- Run Vagrant with the AWS provider. This is the last step and just requires to run vagrant by specifing the aws provider rather than the default virtualbox one.
vagrant up –provider=aws
I hope you enjoyed the article and for any feedback, comments or help please contact me through this site
