Enterprise Hadoop may be less than a decade old, but analysts at Forrester recently estimated that 100% of all large enterprises will adopt it and related technologies such as Spark for big data analytics within the next two years. It’s clear that we are getting beyond the need for businesses to be educated about Hadoop. Time and again, Hadoop has gained significant interest across various industries globally, and is considered to be one of the core platforms available for managing Big Data (structured and unstructured).
We are past the age of talking about what Hadoop IS and even outlining best practices for implementation of this phenomenal technology. In most large data-driven enterprises, you are likely to find at least one senior executive who has been through the process. The US is currently leading in this adoption curve, but Europe is estimated to account for the second largest market share - according to Transparency Market Research - followed by Asia Pacific. Specifically, Transparency Market Research estimates that the European Hadoop market will be worth $8.9 million by 2023. Asia Pacific is expected to undergo the fastest growth in the global Hadoop market owing to having rapidly developing economies coupled a rising IT sector.
The benefits that Hadoop can bring organisations - from cost-effective data storage to marketing and sales optimisation - are increasingly well understood. Yet, Hadoop is a complex technology and there is a lot for any business to get their head around when they first deploy it. Thus, as we approach this year’s Hadoop summit in Dublin, I believe that many attendees will want to hear about lesson learned from people who have deployed Hadoop and their advice on how to produce a data-driven culture within an organization. For organisations in the financial services sector and healthcare, advice on how to address data security and data governance—two of their top concerns—will be critical as they look to build on Hadoop’s open source platform.
Additionally, one of the major factors inhibiting even more rapid growth is the availability of qualified and experienced working professionals who can effectively handle the Hadoop architecture. Companies across almost all industrial sectors are seeking qualified individuals to handle this architecture. Thus, Hadoop Summit Dublin will also likely be a hot spot for networking and recruiting this year. So if you’re a Hadoop developer, get ready to show off your best skills!
Why Secure Systems Matter
In today’s day and age, data security, which also encompasses encryption, data privacy and protection, is critical. After all, it’s clear that if businesses are buying into the dream of a data lake that everyone within the organisation can draw on, they actually create a huge potential problem of people within the organisation accessing data that they shouldn’t. If you simply think about it at a departmental level, if an individual has a database that only they can control and nobody knows that they have it, the chances of that database being infiltrated are far lower than if there is a shared data repository in place across the entire company. Hadoop adopters struggle to integrate user roles and permissions that work well within their existing infrastructure, as well as within Hadoop. Couple this with the fact that each Hadoop vendor can have their own add-on solution, and it becomes a real challenge for users to find a cohesive solution for their organization.
Good Governance is Key
Certainly, any business that is looking to implement Hadoop needs to have the best possible data governance in place. Indeed, not having it is often a complete barrier to adoption, even if the organisation in question is just testing data. So it’s critical that enterprises act quickly to secure their data - but good data security alone is not enough. Organisations also need good data lineage, often seen as the first step to good data governance, and typically involving an understanding of the data lifecycle - what happens to the data as it goes through various processes over time. Good data governance takes this a step further, outlining a full set of processes to ensure that important data assets are formally managed across the entire enterprise.
Without data governance, security, risk, and compliance officers within organisations will not allow business workers to use data processed in Hadoop for any of their financial reports. And that would be a complete loss. So unless data governance tools step up to the mark very quickly, we are likely to see a trough of disillusionment setting in amongst businesses about Hadoop.
Both data security and data governance are critically important factors that all enterprises will need to get right as they roll out Hadoop today – and these issues are becoming even more important to businesses in the new age of self-service access and data preparation tools which help drive productivity and competitive advantage by enabling any decision-maker in the enterprise to quickly prepare data so they can spend more time using it.
Indeed, having the right systems and solutions in place in these areas is paramount to furthering the adoption of Hadoop over the next 2-3 years—not just in delivering a fully secure, compliant and well-managed data environment within businesses, but also by laying the technology and organizational foundation needed to create a truly data-driven enterprise.