As the cost per byte of storage has declined, it has become a habit to simply store data “just in case.” At a time when the overwhelming majority of data was generated by human beings, nobody thought much of it. Data was summarized, information extracted from it, and the raw data points were still kept should they be needed later. Later seldom came.

Cisco tells us that as of 2008, there were more things connected to the internet than people, so we can use that as the point in time when the amount of data being generated and stored had its hockey stick moment. Now we have more sensors in more places monitoring more and more activity and generating more and more data points. By 2010, then Google CEO Eric Schmidt explained how we generated and stored as much data every two days as we did from the dawn of civilization up to 2003.

That’s a lot of data.

Running Out Of Room, Or …

The natural reaction is to instinctively feel that, at some point, we’re going to run out of storage capacity. If Moore’s Law holds, that won’t happen. We’ll just keep inventing new, more compressed storage technologies.

But what we are running out of is time.

Long ago, the last thing anyone in the data center did was to make sure the daily backups were running. They would run into the night all by themselves. Then they would run through the night. Then they were still running when everyone came into the office in the morning.

Fortunately, we’re clever and adaptable, so we came up with incremental backups. Instead of recopying and recopying data we had already copied, we only copied data that had changed since the last backup. Then we moved to faster backup media. Now we’re backing up the data as we’re saving it in primary storage. Ultimately, the restore time objective becomes impossible to achieve in the time available to us.

Making Tough Choices

Now we have to make a difficult choice. Once we’ve processed the data and created valuable information, do we or do we not keep the original raw data as it was collected? Or do we decide to discard it?

Or do we have to choose to save some of the raw data and not other parts of it? What are the criteria upon which that choice can be made? How do we anticipate in our planning which data points need to be stored and which will be discarded?

Now Add Machine Learning

This problem becomes exacerbated by the introduction of machine learning and artificial intelligence technologies to data analytics. When a machine is performing much of the data collation, selection, and processing, how are we to know which data points the machine will want to retrieve to complete its analysis? What if we choose incorrectly?

Other Possible Strategies

Being more pragmatic about this challenge, we need to think about data reduction. First of all, when and where does it occur?

Many of us take a physical relocation from one place to another as an opportunity to discard belongings that we no longer need. Some perform this discarding as they are packing to move. Others, often in a rush to make the move, simply pack everything and promise to do the discarding when they arrive at the new location. Many of us have boxes upon boxes that have yet to be unpacked since we moved in many years ago.

In the classic framework, we can choose to perform data reduction at the core of the network, in the server processors that will perform all the analytics. Or we can choose to perform data reduction at the edge where the data is being collected so the load on the servers and storage are reduced.

The ultimate solution will likely be a combination of both, depending on the workload and the processing required.

Begin With The End In Mind

There has been much discussion about data science — how it’s the art of extrapolating useful information from data and turning it into knowledge that facilitates superior decision-making.

As we continue to see the internet of things produce Schmidt’s estimate of five exabytes per day, data science must expand its scope to include the development of an end-to-end data strategy. This must begin with careful planning and consideration surrounding the collection of data, layers of summarization and reduction, preprocessing, and, finally, deciding which data points get stored and which are discarded.

As always is the case with data storage issues, this will be a volume-velocity-value process based on the business use case involved and at what point data gains value. The science is nascent, but the opportunity is immense.

A case can be made that every company is now a data company. But, it is the effective use of data and not amassing stockpiles of archived data that counts. The advent of big data inspired many businesses to ride the data intelligence wave. While businesses across verticals bought into the idea of big data, what they were doing was hoarding massive sets of raw data. The hype had their attention, but they were not fully prepared. And their real challenge today is – how to unlock meaningful insights from this data haystack.

It’s about time businesses move from gathering massive amounts of data to using this data to arrive at better business decisions. It starts with fixing data hoarding and then being able to leverage data intelligence wisely.

Cross-industry surveys from the likes of McKinsey prove that only a few companies have right out of the gate managed to achieve significant business impact from their investments in big data analytics. What really distinguishes one competitive business from the next is how it uses its own data.

YOU ALREADY OWN A GOLDMINE. HERE’S HOW YOU UNLOCK IT.

If your business struggles with data collation and isn’t getting the desired ROI from your investment and efforts, there is no need to panic and take short-sighted decisions, unless you want your competition to benefit from your impatience.

Now is the time to take advantage of inexpensive data storage, IoT data streams, and the availability of affordable and simple-to-deploy artificial intelligence (AI), Machine Learning, and cloud analytics tools. These technologies make it easier for your business to capture the right data at the right time with more meaningful insights, to get the right business outcomes.

There is strong merit in leveraging these new opportunities as this clearly adds to your competitive advantage. Tangible evidence suggests that data-driven businesses clearly outperform the competition and they are twice or thrice as likely to better succeed in achieving greater sales performance, deeper customer engagement, higher employee productivity, better marketing effectiveness, and other desired organizational performance metrics.

ADOPT ORGANIZED DATA MANAGEMENT AND HARNESS DISRUPTIVE TECH TO MAKE BETTER BUSINESS DECISIONS

A recently published article in The Economist rightly says, “the world’s most valuable resource is no longer oil, but data.” But, only when data is organized and easy to access. Turning data into corporate value requires having a solid data Management Strategy. Today’s modern business deals with plenty of distributed data that resides and emanates from multiple silos. These include, but are not limited to, data centers, global branch offices, CRMs, public clouds, private clouds, IoT sensors, and SaaS platforms.

With some initial expert help and cloud-based AI or analytics services, your business can organize and process such siloed data from disparate sources. Deploying affordable and flexible data analytics tools helps figure out what’s valuable, also making it more accessible. Possibilities go beyond making the data usable. Companies and entrepreneurs can also benefit from powerful (yet affordable) predictive analytics tools, which let you build frameworks to extract greater ROI.

For example, a leading retail store was in the news for creatively targeting pregnant women even before they became mothers! By mining nuggets of useful data from their shopping patterns, this retail store managed to strengthen engagement and increase value capture with expectant mothers. Such Data mining and predictive analytics tools let you identify and tap into many such opportunities that may otherwise go unnoticed.

Building data intelligence into your organization starts with data organization and assessment. Organized data can provide predictive insights, a deeper understanding of your customers, risks, and opportunities, and also helps arrive at better business decisions.

DThe 5 pillars of cloud data management

As the lifeblood of your business, data must be easily available in the cloud to boost your agility and ability to innovate, but easy accessibility must be balanced with protection to ensure maximum business value.

As more and more businesses adopt cloud services, seizing on the latest software tools and development methodologies, the lines between them are blurring. What really distinguishes one business from the next is its data.

Much of the intrinsic value of a business resides in its data, but we’re not just talking about customer and product data, there’s also supply chain data, competitor data, and many other types of information that might fall under the big data umbrella. Beyond that there are a multitude of smaller pieces of data, from employee records to HVAC system logins, that are rarely considered, but are necessary for the smooth running of any organization. And don’t forget about source code. Your developers are using cloud-based repositories for version control of application code. It also needs to be protected.

In the past, companies would typically try to centralize their data and lock it safely away in an impenetrable vault, but hoarding data doesn’t allow you to extract value from it. Data gains business value when it’s transported from place to place as needed and available to be leveraged, not locked away in some dark place. People need swift, easy access to data and real-time analysis to make innovative leaps, achieve operational excellence and gain that all-important competitive edge.

Cloud Data Management

As the importance of data has grown clearer many businesses have been stockpiling as much of it as they can get their hands on with the idea that the value will come along later. Businesses grow organically, so new systems and software are adopted, mergers and acquisitions prompt integrations and migrations, and new devices and endpoints are added to networks all the time. Even the most organized of businesses inevitably end up with a complex structure and data that are distributed globally.

Another layer that exacerbates this problem is people. Sometimes your employees will show poor judgment. They may unexpectedly wipe out critical data or accidentally delete configuration files. Disgruntled employees may even do these things deliberately. Then you must consider all the employees and contractors working for your partners and vendors, who often have access to your business-critical data.

To effectively manage your data without shuttering it and blocking legitimate requests for access, you need a solid cloud data management strategy and that begins with five important considerations.

1. Resting data.

Most of the time data sits in storage. It’s often behind firewalls and other layers of security, which it should be, but it’s also vital to ensure that your data is encrypted. It should be encrypted all the time, even when you think it’s safely tucked up in your vault.

If you properly protect your data at rest by encrypting it, then anyone stealing it will end up with lines of garbled junk that they can’t decipher. You may think it’s unlikely a cybercriminal will breach your defenses, but what about a motivated insider with malicious intent or even a careless intern? Hackers most common point of penetration is actually your employees’ devices, whereby they gain a foothold that can be leveraged to go deeper into your networks. Encrypt everything and take proper precautions to restrict access to the decryption key.

2. Accessing data

It’s very important that your employees can access the data they need to do their jobs whenever and wherever they want, but access must also be controlled. Start by analyzing which people need access to what data and create tailored access rights and controls that restrict unnecessary access. Any person requesting access to data must be authenticated and every data transaction should be recorded so you can audit later if necessary. Active Directory is the most common place to manage and control such access today.

Access control should also scan the requesting device to ensure it’s secure and doesn’t harbor any malware or viruses. Analyzing behavior to see if the user or device requesting access falls into normal patterns of use can also be a great way of highlighting nefarious activity.

3. Data in transit

It’s crucial to create a secure, authenticated and encrypted tunnel between the authenticated user and device and the data they’re requesting. You want to make the data transfer as swift and painless as possible for the end user, but without comprising security. Make sure data remains encrypted in transit, so no interceptor can read it. Choosing the right firewalls and virtual private network (VPN) services is vital. You may also want to compartmentalize endpoints to keep data safely siloed or employ virtualization to ensure it doesn’t reside on insecure devices. ‘

There’s no doubt that most companies focus their data protection efforts here and it is important, but don’t focus on data in transit to the detriment of other areas.

4. Arriving data

When the data arrives at its destination you want to be certain that it is authentic and hasn’t been tampered with. Can you prove data integrity? Do you have a clear audit trail? This is key to effectively managing data and reducing the risk of any breach or infection. Phishing attacks often show up in the inbox as genuine data to fool people into clicking somewhere they shouldn’t and downloading malware that bypasses your carefully constructed defenses.

5. Defensible backup and recovery

Even with the first four pillars solidly implemented, things can and do go sideways from time to time when least expected. Most companies recognize the importance of proper backup hygiene today and have implemented backup and recovery processes. Be sure to actually test and validate your ability to restore the backups and recover periodically.

In the cloud, there’s another critical area to carefully consider. Be careful not to put all your data eggs in one basket. Do not store your backups in the same cloud account where your production data resides. That’s a formula for disaster you may not recover from should a hacker somehow gain access to your network and delete everything.

That is, leverage multiple cloud accounts to segregate your backup data from your production data. Be certain to back up your cloud infrastructure configuration information as well, in case you ever need to rebuild it for any reason.

In the unlikely event your production environment should somehow become compromised, it’s critical a copy of all backups and cloud configuration are stored separately and secured from tampering and deletion. One way to do this is to create a separate backup account (on the same cloud or different cloud) with a “write only” policy that allows backup and archival data to be written and read, but not deleted. This protects your business by ensuring your DR systems and backups will always be available should you need them to recover.

By crafting a plan to cover data storage, data access, data in transit, data arrival, and defensible data backup/recovery, you’ve erected five pillars that will be strong enough to bear the load of your company data and withstand the forces which are trying to break in. But there are still many cloud data management pitfalls to avoid. Ensure that you can quickly recover from the most common issues that arise from operating in cloud environments.

You can have the best products and employees in the world, but without data they are powerless, so take steps to ensure it flows freely and safely. Smart data management will empower your staff to leverage the latest cloud technologies, innovate new products and services and differentiate your organization from the competition.

Check Also

Data Management Software
Cloud Data Management Success
Cloud data management strategy
Cloud File Data Management Pitfalls

It’s important to weigh up the costs and limitations of traditional data centers and consider transitioning your business to the cloud. By modernizing your infrastructure, you can focus on gaining a competitive edge in your core business.

Agility and speed are of paramount importance for most organizations as they try to innovate and differentiate themselves from the competition. The need for flexibility and rapid scalability is driving more and more companies into the cloud, as traditional data centers are no longer proving to be competitive, agile or robust enough.

It should come as no surprise that Cisco predicts 94 percent of workloads and compute instances will be processed by cloud data centers by 2021. But deciding when to take the leap, weighing the costs and risks, and developing a successful strategy is easier said than done.

Why companies are ditching those data centers and how they can make the transition as smooth as possible.

The push of traditional data center costs

Traditional data centers are enormously expensive to maintain. To set one up you need to find a suitable space and then fit it out with everything from uninterruptible power systems (UPS) to cooling HVAC units that keep servers from overheating, not to mention extensive investments in storage and networking equipment.

All of that comes before you consider the cost of hiring data center personnel with the right expertise to keep things running. These are employees outside your core competency, required just to keep your infrastructure working. Then there’s the ongoing energy costs of maintaining the data center and dealing with maintenance.

Economies of scale are important across the board in data centers, but they make a huge difference when it comes to operating and energy costs. A Ponemon Institute report found that the average annual cost per kW for a data center that’s 50,000 square feet or larger was $5,467, compared to an annual cost of $26,495 per kW for data centers between 500 and 5,000 square feet in size.

It’s not easy to scale up and down quickly, so when you’re not using full capacity, cash is being burned. When you push beyond capacity you’re faced with the prospect of expensive expansion or outsourcing. Even outsourcing the physical data center to a colo facility leaves one effectively in the data center hardware and IT infrastructure business.

The pull of the cloud

As many as 81 percent of enterprises now operate multi-cloud strategies, according to Right Scale’s State of the Cloud Report. Business units want to be free to adopt the very latest tools and technologies. They want to be able to pivot and pounce where they see an opportunity, innovating through machine learning and AI, the automation of software development pipelines, and greater depth in agile data analysis. And they want to do all this unencumbered by an internal IT department and bureaucracy.

Shadow IT is a reality, whether you accept it or not, and that genie is not going back into the bottle. While Gartner studies have found that shadow IT is 30 to 40 percent of IT spending in large enterprises, the Everest Group suggests it’s closer to 50 percent or more. It’s time to embrace the cloud. Maintaining a data center and a layer of IT infrastructure to support it is fast-becoming untenable.

No wonder Gartner names cloud system infrastructure services as the fastest growing segment in a fast-growing market. It’s possible to get better service and greater value by leveraging this competitive space.

While cost is perceived as a big barrier, you can offset the savings you’ll make by shutting down the data center or reducing capacity. Security is evolving from a big concern into a driver for cloud adoption. Major cloud providers are protecting data by encrypting and segmenting it across several locations and they hire the finest talent available because it’s their core business.

The shift to the cloud won’t happen overnight, but make no mistake, it is happening. The big question then becomes – how do you plan for and make the transition successfully?

Creating a roadmap

Most organizations have huge data centers and colos with lots of entanglements based on decades of legacy and acquisitions. There’s no way to unwind all of this quickly. For the health of the business, disruption must be managed carefully, which means assessing existing infrastructure and business assets and developing a plan to move them incrementally. Look at how to start developing new apps in the cloud and divest yourself of legacy infrastructure step-by-step.

What is required to mitigate the risk and enable organizations to transition is some sort of connective tissue that bridges the gaps between your data centers, applications and legacy systems and your chosen cloud environments. Make sure you formulate a strategy that allows you to integrate and access your data without sacrificing control.

There are lots of data management pitfalls to watch out for as you migrate to the cloud, but with the right kind of cloud fabric, you can move things along at your own pace based on your available resources, budget, and business priorities.

Ultimately, leveraging the cloud isn’t about one data center hosting strategy vs. another. It’s about modernizing the infrastructure required to run the business, focusing on your core business instead of IT infrastructure and hardware management, and connecting your business with cloud services to differentiate and compete more effectively.

Check Also:

How Can Cloud Infrastructure Help You Stay Competitive?
How to Maintain Control of Your Core in the Cloud