What is Big data ? Importance of Big Data and Challenges of Big Data

What is Big data ? Importance of Big Data and Challenges of Big Data

Big data

Big data is a term that refers to data sets or combinations of data sets whose size (volume), complexity (variability), and rate of growth (velocity) make them difficult to be captured, managed, processed or analyzed by conventional technologies and tools, such as relational databases and desktop statistics or visualization packages, within the time necessary to make them useful.

While the size used to determine whether a particular data set is considered big data is not firmly defined and continues to change over time, most analysts and practitioners currently refer to data sets from 30-50 terabytes(10 12 or 1000 gigabytes per terabyte) to multiple petabytes (1015 or 1000 terabytes per petabyte) as big data.

The complex nature of big data is primarily driven by the unstructured nature of much of the data that is generated by modern technologies, such as that from web logs, radio frequency Id (RFID), sensors embedded in devices, machinery, vehicles, Internet searches, social networks such as Facebook, portable computers, smart phones and other cell phones, GPS devices, and call center records.

In most cases, in order to effectively utilize big data, it must be combined with structured data (typically from a relational database) from a more conventional business application, such as Enterprise Resource Planning (ERP) or Customer Relationship Management (CRM). Similar to the complexity, or variability, aspect of big data, its rate of growth, or velocity aspect, is largely due to the ubiquitous nature of modern on-line, real-time data capture devices, systems, and networks. It is expected that the rate of growth of big data will continue to increase for the foreseeable future.

Specific new big data technologies and tools have been and continue to be developed. Much of the new big data technology relies heavily on massively parallel processing (MPP) databases, which can concurrently distribute the processing of very large sets of data across many servers.

As another example, specific database query tools have been developed for working with the massive amounts of unstructured data that are being generated in big data environments.

Why is Big Data Important?

When big data is effectively and efficiently captured, processed, and analyzed, companies are able to gain a more complete understanding of their business, customers, products, competitors, etc. which can lead to efficiency improvements, increased sales, lower costs, better customer service, and/or improved products and services.

For example: –

• Manufacturing companies deploy sensors in their products to return a stream of telemetry. Sometimes this is used to deliver services like OnStar, that delivers communications, security and navigation services. Perhaps more importantly, this telemetry also reveals usage patterns, failure rates and other opportunities for product improvement that can reduce development and assembly costs.(**Oracle)

• The proliferation of smart phones and other GPS devices offers advertisers an opportunity to target consumers when they are in close proximity to a store, a coffee shop or a restaurant. This opens up new revenue for service providers and offers many businesses a chance to target new customers.(**)

• Retailers usually know who buys their products. Use of social media and web log files from their ecommerce sites can help them understand who didn’t buy and why they chose not to, information not available to them today. This can enable much more effective micro customer segmentation and targeted marketing campaigns, as well as improve supply chain efficiencies.(**)

•  Other widely-cited examples of the effective use of big data exist in the following areas:  

o   Using information technology (IT) logs to improve IT troubleshooting and security breach detection, speed, effectiveness, and future occurrence prevention.

o   Use of voluminous historical call center information more quickly, in order to improve customer interaction and satisfaction.

o   Use of social media content in order to better and more quickly understand customer sentiment about you/your customers, and improve products, services, and customer interaction.

o   Fraud detection and prevention in any industry that processes financial transactions online, such as shopping, banking, investing, insurance and health care claims.

o   Use of financial market transaction information to more quickly assess risk and take corrective action.

Big Data Challenges

Understanding and Utilizing Big Data – It is a daunting task in most industries and companies that deal with big data just to understand the data that is available to be used, determining the best use of that data based on the companies’ industry, strategy, and tactics.

• Also, these types of analyses need to be performed on an ongoing basis as the data landscape changes at an ever increasing rate, and as executives develop more and more of an appetite for analytics based on all available information.

New, Complex, and Continuously Emerging Technologies – Since much of the technology that is required in order to utilize big data is new to most organizations, it will be necessary for these organizations to learn about these new technologies at an ever-accelerating pace, and potentially engage with different technology providers and partners than they have used in the past.

·         Like with all technology, firms entering into the world of big data will need to balance the business needs associated with big data with the associated costs of entering into and remaining engaged in big data capture, storage, processing, and analysis.

Cloud Based Solutions – A new class of business software applications has emerged whereby company data is managed and stored in data centers around the globe. While these solutions range from ERP, CRM, Document Management, Data Warehouses and Business Intelligence to many others, the common issue remains the safe keeping and management of confidential company data.

·         These solutions often offer companies tremendous flexibility and cost savings opportunities compared to more traditional on premise solutions but it raises a new dimension related to data security and the overall management of an enterprise’s Big Data paradigm.

Privacy, Security, and Regulatory Considerations - Given the volume and complexity of big data, it is challenging for most firms to obtain a reliable grasp on the content of all of their data and to capture and secure it adequately, so that confidential and/or private business and customer data are not accessed by and/or disclosed to unauthorized parties.

·         The costs of a data privacy breach can be enormous. For instance, in the health care field, class action lawsuits have been filed, where the plaintiff has sought $1000 per patient record that has been inappropriately accessed or lost. In the regulatory area, for instance, the proper storage and transmission of personally identifiable information (PII), including that contained in unstructured data such as emails can be problematic and necessitate new and improved security measures and technologies.

·         For companies doing business globally there are significant differences in privacy laws between the U.S. and other countries. Lastly, it will be very important for most forms to tightly integrate their big data, data security/privacy, and regulatory functions.

Archiving and Disposal of Big Data – Since big data will lose its value to current decision making over time, and since it is voluminous and varied in content and structure, it is necessary to utilize new tools, technologies, and methods to archive and delete big data, without sacrificing the effectiveness of using your big data for current business needs.

The Need for IT, Data Analyst, and Management Resources – It is estimated that there is a need for approximately 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, either retrained or hired.