Big data Hadoop - Deepstash
A Job Seeker's Guide

Learn more about career with this collection

How to write an effective resume

How to network and make connections

How to prepare for a job interview

Big data Hadoop

Big data Hadoop

  • Ability to store and process huge amounts of any kind of data, quickly. With data volumes and varieties constantly increasing, especially from social media and the Internet of Things (IoT) , that's a key consideration.
  • Computing power. Hadoop's distributed computing model processes big data fast. The more computing nodes you use, the more processing power you have.
  • Fault tolerance. Data and application processing are protected against hardware failure. If a node goes down, jobs are automatically redirected to other nodes to make sure the distributed computing does not fail. Multiple copies of all data are stored automatically.
  • Flexibility. Unlike traditional relational databases, you don’t have to preprocess data before storing it. You can store as much data as you want and decide how to use it later. That includes unstructured data like text, images and videos.
  • Low cost. The open-source framework is free and uses commodity hardware to store large quantities of data.
  • Scalability. You can easily grow your system to handle more data simply by adding nodes. Little administration is required.

MapReduce programming is not a good match for all problems. It’s good for simple information requests and problems that can be divided into independent units, but it's not efficient for iterative and interactive analytic tasks. MapReduce is file-intensive. Because the nodes don’t intercommunicate except through sorts and shuffles, iterative algorithms require multiple map-shuffle/sort-reduce phases to complete. This creates multiple files between MapReduce phases and is inefficient for advanced analytic computing.

There’s a widely acknowledged talent gap. It can be difficult to find entry-level programmers who have sufficient Java skills to be productive with MapReduce. That's one reason distribution providers are racing to put relational (SQL) technology on top of Hadoop. It is much easier to find programmers with SQL skills than MapReduce skills. And, Hadoop administration seems part art and part science, requiring low-level knowledge of operating systems, hardware and Hadoop kernel settings.

56

248 reads

MORE IDEAS ON THIS

CURATED FROM

CURATED BY

samuelbancroft

Keep reading, keep studying, the more you learn the more you change. If you are doing the Python lessons please join this discord channel https://discord.gg/kugXx9KY but please follow the rules

Related collections

More like this

Big Data

Big Data

  • There is an explosion of valuable digital data generated by consumers as we live our daily lives.
  • Advances in technology such as AI, Machine Learning and Cloud Computing provide numerous ways to leverage the data.
  • Apart from having Big Data, companies need to have an analytic...

Python data processing with pandas

Pandas is a Python language package, which is used for data processing. This is a very common basic programming library when we use Python language for machine learning programming. This article is an introductory tutorial to it. Pandas provide fast, flexible and expressive data structures with t...

Edge Computing

As the quantity of data organizations are dealing with continues to increase, they have realized the shortcomings of cloud computing in some situations. 

Edge computing is designed to help solve some of those problems as a way to bypass the latency caused by cloud computin...

Read & Learn

20x Faster

without
deepstash

with
deepstash

with

deepstash

Access to 200,000+ ideas

Access to the mobile app

Unlimited idea saving & library

Unlimited history

Unlimited listening to ideas

Downloading & offline access

Personalized recommendations

Supercharge your mind with one idea per day

Enter your email and spend 1 minute every day to learn something new.

Email

I agree to receive email updates