We have touched on Hadoop before in our blog, first announcing our partnership with Hortonworks, back in February of 2014, and more recently on our Big Data Solutions Page. Hadoop is a powerful open source framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models, making it an excellent Big Data tool. Hadoop can benefit from the operational efficiencies of Cloud-A’s elastic, API driven infrastructure-as-a-service – and it is a growing use case among our users.
There are many aspects of our compute that lend themselves to successful Hadoop deployments on Cloud-A, like our Big Data High Memory® and Big Data High Compute® instance flavours that provide the scale and power for the biggest jobs, but there are other elements of Cloud-A infrastructure-a-service that make it an ideal host for Hadoop Big Data solutions.
Let’s take a look at how you can leverage Cloud-A Bulk Storage, our object storage product, as a reliable, scalable and cost-effective Big Data repository for your Hadoop platform.
Cloudberry Lab is a company that makes backup and file management software for hybrid cloud environments, allowing users to backup or sync files from their local systems to the public cloud. While Cloudberry has paid products for backing up Windows servers and applications, they offer a piece of freeware called Cloudberry Explorer, which is a file manager that allows you to sync files from your Windows system to a number of public cloud options including OpenStack.
Create Cloud-A Bulk Volume Container
CloudBerry Explorer for OpenStack is built on OpenStack Swift technology, which means that users can use it with Cloud-A’s Bulk Storage ($0.075 per GB per month). You will need to create at least one Bulk Storage container by navigating to the storage tab in the Cloud-A dashboard and selecting “New Container.” Appropriately name your container and you are ready to download Cloudberry Explorer.
Tip: To keep your cloud-synced files organized, we recommend creating multiple Bulk Storage containers and treat them as if they were a folder directory on your local system.
Simply follow the steps to completed the installation wizard program.
Authenticate to your Bulk Storage Container
Once CloudBerry Explorer has launched you will notice that the left side of the screen represents your local systems folder directory and the right represents cloud storage. On the cloud storage side click the source drop down menu and select: <New Storage Account>
Now Select “Test Connection” to ensure that the system has accepted your credentials.
If Test Connection fails, ensure that you have entered your credentials correctly. If you have entered your credentials correctly but are still receiving a “Connection Failed” error message, ensure that you have the correct ports open for Bulk Storage. Those ports are: 80, 443, 8443 and 8444.
If your credentials were entered correctly, the Bulk Storage container you created in the first step will appear in the file directory on the right side of the screen. To test the connection, select a test file from your local system, and select “Copy.” A transfer status message will appear briefly at the bottom of the screen and the file will copy from the left side of the screen and appear in your cloud storage container on the right.
To prove this concept, log into your Cloud-A dashboard and navigate to your new Bulk Storage container. You should see your test file.
Functional Use Cases:
Upload very large files, like 4K HD videos, disk images, or backup archives, in multiple pieces efficiently and have them downloaded / served as a single file using an Object Manifest to glue the data back together.
Archive data from old projects taking up unnecessary space on your production storage (CAD files, BIM files, PSD files.)
Use with Cloud-A Windows instances and move infrequently used, non-mission critical data of off high performing SSD volume storage.
CloudBerry Explorer is a great way to manually sync files to Cloud-A, and a great introduction into hybrid cloud solutions. Check out some of CloudBerry Lab’s other products for more advanced features like scheduled backups and encrypting files.
If you have been following our blog you will know that we have recently published two posts on ownCloud. The first, “Deploying ownCloud on Cloud-A” was a tutorial on how to install and configure ownCloud on a Windows 2008 R2 instance on Cloud-A and the second titled: “ownCloud: Infinite Expandability with Cloud-A’s Bulk Storage” was how to expand your ownCloud deployment with our bulk storage powered by Swift. Today we are going to show you how to install the ownCloud desktop client for OSX and Windows Server 2008 R2 (instructions will be the same for Windows 7.)
Download and Install Desktop Client
You will need to download the appropriate ownCloud desktop client from https://owncloud.org/install/. Once your download has completed, run the installer for the ownCloud desktop client.
Authenticate to your ownCloud Server
Upon completion of the installation you will need to authenticate to your ownCloud server with the correct IP address.
Next, you will need to authenticate with your ownCloud credentials.
At this point you can choose your folder syncing preferences. Depending on your preference, you can choose from syncing everything from your ownCloud server or just specific files and folders.
Much like Dropbox, ownCloud will create a cloud-syncing local drive on your desktop. An ownCloud folder shortcut will appear in the top menu bar as well as your Favorites under in Finder. In Windows, an ownCloud folder shortcut will appear in the tray as well as your Favorites under in My Computer.
At this point in our ownCloud blog series you have learned how to create an ownCloud server on a Cloud-A Windows instance, expand the storage space with Bulk Storage and configure desktop clients. To take it one step further and enable your users for mobility you can download and configure mobile apps for iOS and Android.
We previously published a blog post on creating an ownCloud server on Cloud-A’s public cloud, but we would like to build upon that and show just how expandable and agile a Cloud-A hosted ownCloud deployment can be by introducing bulk storage.
By leveraging our Bulk Storage powered by Swift, users can expand the size of their ownCloud deployment very quickly and inexpensively to facilitate growth. Unlike a hardware deployment, where you would purchase drive space up front to account for future growth, a Cloud-A deployment will allow an organization to scale their storage as needed on a pay-as-you-go utility model.
We will begin with the assumption that you already have an ownCloud deployment running on Cloud-A with administrator access to the program.
Create an Object Storage Container
From your Cloud-A dashboard, select “Storage” and then “Containers.” Select “New Container,” and name the new container.
Configure External Storage in ownCloud
ownCloud comes prepackaged with external storage support, but the functionality must be enabled in the “apps” dashboard of your ownCloud instance. In the “apps” dashboard select “External storage support” on the left-hand side bar and enable it.
This will populate an External Storage section in your ownCloud Admin panel. Select “OpenStack Object Storage” from the “External Storage” dropdown menu and fill enter the following credentials:
Timeout: Timeout of HTTP requests in seconds (optional)
If you have correctly input the information above and ownCloud accepts that information, a green dot will appear to the left of the folder name.
Validate External Storage
To further validate the access to the new external storage, go back to the main ownCloud screen by clicking the ownCloud logo in the top left corner, and select external storage. You should see your newly created ownCloud folder which points to your Cloud-A object storage powered by Swift.
Adding additional external object storage to your Cloud-A hosted ownCloud instance sets you free from the traditional limitations of hardware, allowing you to scale on demand. This is an ideal solution for any growing company looking to have control of their own data, but also have that data stored securely in Canada.
Stay tuned for the next post in our ownCloud series.