A View From Egnyte Engineering

Scale and ScalabilityBy Krishna SankarThe other day we were observing that scale is a tough master but a delightful slave. If scale rides you, there is no place to hide. As Joe Louis puts it, "You can run but you can't hide." But once you master scale, things start coming together.The whole philosophical musing started as we embarked on fitting in a new Object Store Layer. It is not that our current infrastructure is bursting at the seams, but we wanted to get ahead of the scale that we will face in the very near future.Conceptually an object layer is concise:

  • It accepts objects from multiple applications and stores them.
  • It has a metadata service that manages the distribution as well as location and access.
  • Given an object ID, it returns the object to the application.

Life would be much easier if the feature set ended here. Fortunately or unfortunately, life is never that simple.

An object store also needs to guarantee integrity of the objects it holds against coordinated and random failures, even byzantine failures, which adds replication, object auditor, rebalance, storage failure recovery and so forth. One interesting characteristic about scale is that one has to expect and embrace failure and work in spite of it.

So an innocent excursion into object store suddenly turns into a universe of databases, multiple sharding, consistent hashing, monitoring, coordinated consensus, caching, Fisher-Yates shuffle, soft deletes and smart object keys.

We will visit these and other interesting challenges from the land of scale, coming to your favorite blog this summer.

Get started with Egnyte today

Explore the best secure platform for business-critical content across clouds, apps, and devices.

Share this article
Author
Emily Ganz

View All Posts
Tags
No items found.
Don’t miss an update

Subscribe today to our newsletter to get all the updates right in your inbox.

By submitting this form, you are acknowledging that you have read and understand Egnyte's Privacy Policy

Thank you for your subscription!

Welcome to
Egnyte Blog

Product and Company Updates
Customer and Partner Updates
Life at Egnyte
Industry Insights
Contact Support