We have been seeing increasing demand for our object storage product, Bulk Storage over the past few months, which is exciting for us because we think that object storage is a phenomenal storage solution for many specific applications. We wanted to provide our followers with an overview of object storage technology and highlight some effective use cases. The biggest hurdle we see to adoption of our Bulk Storage product is an understanding of how exactly the technology can be used. As a result of this, a big portion of our job at Cloud-A is to play matchmaker and point users in the direction of existing third party software solutions that integrate with object storage.
What is Object Storage?
Object Storage is an extremely simple yet scalable and reliable file storage system that has been made popular by Amazon’s S3 product, the world’s most recognized object storage deployment. In addition to S3, OpenStack Swift is an open source project that allows organizations to build out their own object storage solutions – Swift is what Cloud-A Bulk Storage is built with. For organizations who prefer to keep their data within their own walls, and do not have in-house development talent or experience to build their own Swift deployment, SwiftStack provides a vendor supported, optimized and hardened solution for on-prem object storage solutions.
Rather than organizing files in a directory, object storage stores files in a flat organization called containers, and unique identifiers or “keys” are used to retrieve them. Object storage requires less metadata than traditional file systems do to store and access files by grouping metadata with each object. Object storage offers basic functionality, allowing users to store, retrieve, copy, and delete files and most object storage systems provide a REST API that allows users to programmatically integrate with containers and objects.
Benefits
Scalability
The simplicity of object storage makes it incredibly scalable. Scaling an object storage cluster is as simple as adding additional nodes.
Reliability
Object storage achieves reliability by replicating objects across multiple hard disks, nodes, and geographical locations.
Cost Effective
Since object storage is “software defined,” clusters can be built on commodity hardware, avoiding the cost of specialised appliances. This paired with the economies of scale that a cloud provider like Cloud-A achieves enables the price point of object storage to be much lower than traditional network attached storage (NAS) solutions.
Object Storage Limitations
Object storage is built for unstructured data within cloud applications. The technology was not built to replace block storage for operating systems and databases, which are both structured data. Object storage in a public cloud setting generally has a lower throughput than traditional storage systems.
Use Cases
Content Repository
Object storage makes for an excellent solution to house content repositories for web & mobile applications. A single, web-addressable namespace is used rather than putting infrastructure knowledge in the application.
These applications access data via native HTTP RESTful APIs and developers leverage broad API support for programming languages including Java, C, C#, Python, PHP, Ruby, and others.
Recently at Cloud-A we have seen more and more users serving static files from Bulk Storage as an alternative storage backend for content management systems like Drupal or WordPress.
Large-File Workflows
Several industries that store and archive large files like video production, life sciences and research have found object storage to be an extremely useful storage solution. The scalability of object storage, both in public or private deployments, allows organizations to distribute their data across multiple regions for essentially unlimited amounts of capacity.
Most of these large-file use cases involve industry specific software that has been modernized to support cloud storage APIs like the OpenStack Swift API.
Document Sync & Sharing
One of the largest consumers of Cloud-A Bulk Storage is third party sync and share solutions. Organizations are champing at the bit for DropBox-like solutions that are under their own control.
Object Storage provides an excellent storage backend for sync and share as it allows organizations to store unlimited documents and unstructured data, all accessed with no compatibility issues or data lock-in. Cloud-A Bulk Storage, powered by OpenStack Swift supports a wide range of third-party file sync services and document management systems such as:
CTERA Enterprise File Sync & Share
CTERA provides an enterprise-grade alternative to consumer cloud services, utilizing cloud infrastructure that is completely under ITs control, and providing end-to-end security. The solution allows enterprises and service providers to set up file share and sync services for their users based on a variety of object storage platforms and cloud IaaS providers (on public or ‘virtual private’ environments).
Storage Made Easy
SME’s File Sync and Share solution provides seamless integration with OpenStack Swift. SME’s Enterprise File Share and Sync solution uniquely allows IT to regain control of “cloud Sprawl” and unifies private and public file sharing into a single, converged storage infrastructure.
AeroFS
Like Dropbox, but behind your firewall. AeroFS is file sync and share for the enterprise.
ownCloud
Available as an open source download or a paid enterprise edition, ownCloud provides an open, extendable and modifiable file sync and share solution, allowing users to access their data from any device.
Cyberduck
Cyberduck is an open-source client for browsing data in a variety of storage systems — including OpenStack Swift. Combining Cyberduck with object storage enables users to quickly store and retrieve files in a cloud storage system with a familiar drag-and-drop interface.
CloudBerry Explorer
CloudBerry Explorer for OpenStack is a freeware file manager for cloud storage build on OpenStack Swift technology. CloudBerry Explorer provides a user interface to cloud storage accounts allowing managing and moving data across your local and cloud storage. CloudBerry Explorer for OpenStack comes in two versions – Freeware and PRO.
Big Data Repository
Object storage can provide a repository for incoming data, as well as the results of analysis and modeling in big data use cases.
Object storage provides simple storage management, allows organizations to avoid silos and sharding, and fast access to any data either as files or as native objects from Hadoop and other analytic tools.
Utilizing the Hadoop Swift Filesystem client, organizations can leverage OpenStack Swift deployments like Cloud-A Bulk Storage for their Hadoop workflows. The Hadoop Swift filesystem library defines which Swift object hosts a container – and how to connect to it.
Hortonworks
Architected, developed, and built completely in the open, Hortonworks Data Platform (HDP) provides an enterprise ready data platform that enables organizations to adopt a Modern Data Architecture.
With YARN as its architectural center it provides a data platform for multi-workload data processing across an array of processing methods – from batch through interactive to real-time, supported by key capabilities required of an enterprise data platform — spanning Governance, Security and Operations.
Backup and DR and Archival Storage
Another leading use case for Cloud-A Bulk Storage is backup, disaster recovery and archival storage. Object storage provides a resilient and easily scaled storage target for backups of VMs, files, and databases.
As a backup target, object storage can provide a much lower-cost solution than other on-premise storage options. Native data replication in object storage technology like OpenStack Swift ensures that backups remain available even if a server, site or network fails.
CTERA Hybrid Backup & Recovery
CTERA’s cloud storage gateway appliances provide both clientless and built-in agent-based backup, allowing you to protect servers, desktops and laptops. The appliances de-duplicate, compress and encrypt all backups before they are sent to the cloud and offer bandwidth throttling options, making this a highly optimized and secure backup solution.
For roaming users, CTERA’s Cloud Agent provides direct-to-cloud end-point backup and recovery, including de-duplication and bandwidth control.
Commvault Simpana Backup & Recovery
Recognized as an industry leader, CommVault® has redefined enterprise backup and recovery to help you speed recovery, reduce costs and simplify operations. As the foundation for a modern approach to data protection,Simpana software transcends the limitations of legacy approaches and point products by integrating application awareness with hardware snapshots, indexing, global deduplication, replication, search, and reporting, all within a single platform.
Duplicity
Duplicity is a backup software suite that provides encrypted, digitally signed, versioned, remote backup of files requiring little of the remote server. Released under the terms of the GNU General Public License (GPL), Duplicity is free, open source software. Duplicity supports OpenStack Swift as a backup end-point.
Quaddra Software Storage Insight
Quaddra’s Storage Insight object management software quickly analyses the status of billions of objects. System administrators use it to identify object categories and details that enable fast and safe data management. For example, files containing certain strings, such as “confidential” could be exempted from being copied to the cloud.
Storage Insight is a virtual appliance that scans data without needing to migrate it first. Automated policies effectively manage growth and monitor status and changes in high-volume data stores.
CloudBerry Lab Cloud Backup for Windows Server
CloudBerry will backup file shares and Windows Server files data directly to your cloud storage account and provide easy restore with a few clicks.
The backup client compresses and encrypts data before it leaves your server and then securely transfers it to the cloud of your choice, serving as a transport between Windows Server and cloud storage. You can connect your storage account with CloudBerry software and configure all backup options using a step-by-step Backup Wizard.
Whats next?
We will continue to help guide our users on the best way to leverage our Bulk Storage offering as we approach our goal of managing the largest public object storage cloud offering in Canada. Stay tuned for a series of announcements over the next several months on how Cloud-A is working towards making object storage more consumable for our Canadian users.