The Ins and Outs of Bulk Data Transfer to the Cloud: How Long, How Much, and Who Does it?

Nov 4 2015 | by Yoav Mor

23 Flares Twitter 0 Facebook 0 LinkedIn 23 Email -- 23 Flares ×

Enterprises are quickly jumping bulk data transfer on the cloud migration band wagon, and bringing the need to transfer bulk data (e.g., databases and on-prem storage) with them. In this article, we will share various methods that leading cloud providers offer to transfer bulk data to the cloud. Whether you’re a media site, big data use case or enterprise, if you’re interested in transferring bulk data to the cloud, see which of the four options listed below is best for you.

We’ve included the pros and cons of each method as well as the estimated time it would take and cost of transferring 1TB of data to the cloud.

1. The Naive Method

With your on-premises data center connected to the internet, you can transfer data to the cloud provider of your choice. Just install an FTP server on an instance and start transferring the data.

Pros: By using the public internet, this is the simplest and most straightforward way of sending data to the cloud. Additionally, big cloud providers generally do not charge for bulk data transfer into the cloud, so it’s free.

Cons: While this method sounds ideal, the downfall is that due to the fact that it depends on the public internet, congestion and unstable latency can slow down transfer speeds. Additionally, it is not secure and is prone to tampering and eavesdropping.

Time calculation: At an upload rate of 20Mbps, it would take approximately five days to upload 1TB of data to the cloud (that is if all goes as planned).

2. VPN Tunnels

Connect your on-premises data center to cloud providers via their partnering network service providers. AWS Direct Connect is a great example of how VPN tunnels work. AWS is partnered with telcos and data center providers such as Equinix, which has locations scattered across the U.S. By directly connecting to Equinix’s site, enterprises can enjoy a secure, reliable private connection straight to their AWS accounts (which is a service that can also be enjoyed by Azure customers).

Pros: This method is fast, relatively secure, and reliable due to the fact that you use a private connection rather than the public internet.

Cons: While data transfer is free with cloud providers, the VPN tunnel service isn’t. For example:

  • AWS may charge anywhere from $0.03 per port per hour for a transfer speed of 50Mbps to $2.25 per port per hour for a transfer speed of 10Gbps.
  • Azure comes out to be a bit more expensive than AWS, ranging from $55 per port per month for 50Mbps to $5,000 per port per month for a 10Gbps connection.

In terms of pricing, it would be wise to choose a cloud provider according to how frequently you plan on transferring bulk data. As mentioned above, Azure charges per month, regardless of how long your transfers actually take to complete, while Amazon charges by the hour.

Time calculation: It will take you about 48 hours to transfer 1TB of data to the cloud via VPN tunnels using the lowest transfer rate of 50Mbps; 5 hours at a rate of 500Mbps; and 15 minutes at the highest rate of 10Gbps.

3. Snail Mail

While it may seem like an oxymoron to transfer your data to the cloud via snail mail, it’s a bonafide method offered by leading providers. Simply transfer your data to a hard drive, pack it and send it off to the cloud provider of your choice. The provider then unpacks the hard drive, connects it to a storage unit, and downloads the data locally in the data center to your account. After all of the data is transferred, the cloud provider strips the hard drive and sends it back.

Pros: Contrary to what you may think, this method is relatively fast and reliable. While the overhead of sending 1TB via snail mail will cause the transfer to be slower than via VPN tunnels, if you send a larger amount of data, say 10TB or 100TB, the overhead is worthwhile. Price-wise, AWS and Azure both charge about $80 per volume (1TB).

Cons: AWS has an additional downloading charge of $2.50 per hour, so if you have a regular USB connection on your volume, the transfer will be longer, but if you have an advanced eSATA interface, the transfer will be shorter (and the price lower).

Time calculation: With AWS, you can calculate the amount of time your transfer will take by entering the amount of volumes you want to send along with each volume’s interface connection speed into the AWS price calculator. For example, sending 1TB of data using this method will cost $104.9, as shown below:


4. AWS Import/Export Snowball

Amazon recently announced this method at re:Invent 2015, as a more advanced version of the disk import/export option. Simply start a snowball job on the portal and AWS will send a rugged appliance by mail within a few days with a 50TB capacity and a 10Gbps interface. Connect the appliance to your data center network, transfer your data and send the appliance back to AWS.

Pros: The appliance is encrypted and rugged in a tamper-resistant enclosure. AWS refers to this as a petabyte-scale data transfer solution, which would require a number of appliances, but you can get it running if you have large scales of data, especially in big data projects and the like, you can scale your capacity to extremely large storage volumes. The cost of this type of transfer is $200 per transaction, and since you can transfer as much data as you like, the time you lose by sending via snail mail is virtually insignificant.

Cons: Now that doesn’t mean you should take your time sending the appliances back to AWS, especially since they charge late fees if appliances are not returned within 10 days after delivery.

Time calculation: It will take you about 15 minutes to transfer 1TB of data to one of AWS’ import/export snowball appliances plus a few days of shipping on either side.

*It’s important to note that each of these options also allows data to be transferred out of the cloud, at your provider’s regular rates, which you can find here.

Final Thoughts

Virtual network speeds are increasing, but they have a lot of catching up to do with today’s rapidly growing storage capacity. While it’s not intuitive that snail mail can actually be faster than the modern day internet, if you look at the numbers, it can definitely be worthwhile. By being aware of how much each option costs, we hope we’ve helped you choose the best method for your use case.

Start Optimizing Your Cloud Deployment

Start your 20-day free trial today to easily identify greater cloud efficiency opportunities and view your entire AWS deployment in one place. Gain full transparency, accountability, and control.
start free trial


Forgot Password?

No account yet? Register.

23 Flares Twitter 0 Facebook 0 LinkedIn 23 Email -- 23 Flares ×