Data Motivates You to Do More

I’ve always believed in trusting  data over the human gut, but lately I’ve observed a simple fact that being data driven has a side effect; it motivates you and keeps you on track.What do I mean by this? Here are some recent examples, both personal and professional:Fitbit:Last year, I started an afternoon walk “ritual” because by 2:00 PM after conducting calls and replying to a lot of emails, my brain would be fried, and I wouldn’t have energy left to code or think. Taking these 30-40 minute walks daily recharges the brain.I had an Omron pedometer sitting around for almost a year, and I seldom took it with me on walks. The problem with it was that it used to store only the past 30 days of data about my steps and other things, and it didn’t have a good way to graphically see the data.

When it comes to data, "less is more," but also one more important aspect is that you need to present your data in graphs so it doesn’t take a huge amount of cognitive effort to make sense of it. Recently, Egnyte gave a Fitbit to everyone who participated in our summer fitness challenge, and while it isn’t a better pedometer than my old Omron, I immediately saw that it could easily sync data stored on the pedometer via bluetooth to my Fitbit app. Also, with a quick sync and glance, I can see when I am lagging and need to catch up. It motivates me — comparing my own steps week to week, and versus my colleagues.

Large scale migration to Elasticsearch: Now, it wouldn’t be a post from me, if I didn’t share how this relates to our technology…We recently migrated billions of files from old an lucene-based system to Elasticsearch to improve search performance, and the migration took months with the data keeping us on our toes, telling us if the “compass” was pointing toward North or not. We built various dashboards to monitor migration rates, and as migration was running day and night, I would start my day with checking how many more files we migrated and if we needed to add more servers, jvms, memory or CPU to meet the goal.For example, here is a graph in one data center:

This data kept us on track, and we were able to spot many issues well before they became major challenges.Exception Analysis:We do exceptions and 5xx status analysis on incoming requests daily, and in a two-week sprint, we strive to fix as many as we can. However, after the release, either new issues pop up or customers use the flow in a different way, causing minor temporary spikes in some components. And as I mentioned before, through all of this, the one thing that has kept us on our toes was the data.By looking at the one screen report, we can tell how a data center did yesterday and which areas require immediate attention vs. areas that require attention in 1-2 days. This leads to a lesser number of customer escalations, since we are able to spot many of the issues before they do. Little things add up and  having this report daily enables us to optimize more, and we can spend quality time doing the things we love - like writing code for scalable systems.In short, data points out that there is a problem, and once a bug has been implanted in your brain that a problem exists, you then try to fix it. This allows us to get back to the normal routine of doing what you enjoy the most. Whether it’s getting more steps on your Fitbit or fixing a potential technology issue, data helps drive each person to make quick changes for the better.