Nowadays ‘Cloud’ is a magical word which attracts most of the Small, Medium and even Large Businesses due to its Cost effective, scalable and secure infrastructure. Having ushered into this cloud era, we built our Vembu cloud backup and DR service by using the secure and reliable Amazon’s cloud computing infrastructure.
In Vembu Cloud, we use Amazon’s Elastic Compute Cloud (EC2 instances) as our server machine and EBS volumes/S3 storage as our primary storage. Our cloud infrastructure is something highly scalable and elastic, where we can launch any number of instances on demand to handle large number of simultaneous client connections whenever needed. To scale up better we have split our Vembu cloud deployment into modular pieces – WebsiteServer, MasterNode, Clustered SubNodes and Clustered DatabaseNodes (both RDBMS and NoSQL databases). Each one has its own responsibility in handling client’s requests and scalability which we will share in detail here.
Website Server:
It will handle all the signups, registrations and login requests across all the AWS regions. Once you successfully register a Vembu cloud account for a particular AWS region, you will become as a Vembu cloud customer and the customer information will be captured in our website server as well as in our Master node specific to the particular region.
Master Node:
In the current Vembu Cloud deployment, the Master node is a main node which receives the initial requests from client and give the cluster subnode url to the client for further requests. It will monitor the health of cluster subnodes periodically to check the availability of the particular system. This server will have all the entries of cloud accounts, clients, backups and their detailed information. In addition, this node will process all the UI requests and do the necessary redirects. We also have web service APIs for some of the functionalities and the API requests are handled by the master node only.
Clustered SubNodes:
We have a set of EC2 instances which act as a clustered sub nodes in our Vembu Cloud. These instances will handle the backup/restore/delete requests from the clients and the entire data processing will be handled by these nodes.
Clustered Database instances:
We are using both RDBMS and NoSQL Databases in our Vembu cloud deployment. We store all the account information and backup file metadata information in the MySQL server and we store all our chunk file meta data into the MongoDB. Both MySQL and Mongo are sharded horizontally at backup level across EC2 instances to scale a large number of clients in our Cloud server.
EBS & S3:
Vembu Cloud uses both Amazon EBS storage and S3 as a storage destination for our customer data. cloud MySQL and Mongo databases reside in the EBS volume and the EBS acts as a temporary local cache for the backup data. We store the backed up data in our own chunk file format (VembuHive File System – VHFS chunk format) where all the file content will be consolidated and written into a single file to improve I/O performance. We have a max size limit for Chunk files and once it reaches its max size or at the end of backup, we will mark the chunk files as filled and upload them to the Amazon S3 storage by using the S3 REST APIs. To improve large file upload performance, we are using ‘Multipart Upload‘ API to upload the larger chunk files. Amazon S3 provides unlimited storage on cloud and you pay only for the storage actually used. At the same time EBS storage has a size limitation of 1TB and you have to pay for the disk size of the EBS irrespective of how much you use. Once the chunk files are successfully uploaded to the S3 storage, we will delete the files from the EBS volume to avoid it overfill. During download process, instead of downloading the entire file to EBS, we directly read the specific ‘Range of bytes‘ from the S3 file object. We are also taking snapshots for the EBS volumes in a scheduled interval to make it a more reliable storage.
We are also working on improving our cloud infrastructure to make it even more stable. Lot more changes will be announced in the upcoming days. You can try our Cloud online backup services by going to the:
