Explore
For Enterprise
Join for Free
Log In

Гуманитарные науки и искусства
Бизнес
Компьютерные науки
Наука о данных
Информационные технологии
здоровье
Математика и логика
Личное развитие
Естественные и технические науки
Социальные науки
Изучение языков
Степени
Сертификаты

Полный обзор Coursera

Обзор
Поиск
Для организаций
Войти
Присоединиться бесплатно

RDDs, Benefits

Loading...

Data Manipulation at Scale: Systems and Algorithms

Вашингтонский университет

4.3 (оценок: 688) | Зарегистрировано учащихся: 49K

Курс 1 из 4 — Наука о больших данных Специализация

Data analysis has replaced data acquisition as the bottleneck to evidence-based decision making --- we are drowning in it. Extracting knowledge from large, heterogeneous, and noisy datasets requires not only powerful computing resources, but the programming abstractions to use them effectively. The abstractions that emerged in the last decade blend ideas from parallel databases, distributed systems, and programming languages to create a new class of scalable data analytics platforms that form the foundation for data science at realistic scales. In this course, you will learn the landscape of relevant systems, the principles on which they rely, their tradeoffs, and how to evaluate their utility against your requirements. You will learn how practical systems were derived from the frontier of research in computer science and what systems are coming on the horizon. Cloud computing, SQL and NoSQL databases, MapReduce and the ecosystem it spawned, Spark and its contemporaries, and specialized systems for graphs and arrays will be covered. You will also learn the history and context of data science, the skills, challenges, and methodologies the term implies, and how to structure a data science project. At the end of this course, you will be able to: Learning Goals: 1. Describe common patterns, challenges, and approaches associated with data science projects, and what makes them different from projects in related fields. 2. Identify and use the programming models associated with scalable data manipulation, including relational algebra, mapreduce, and other data flow models. 3. Use database technology adapted for large-scale analytics, including the concepts driving parallel databases, parallel query processing, and in-database analytics 4. Evaluate key-value stores and NoSQL systems, describe their tradeoffs with comparable systems, the details of important examples in the space, and future trends. 5. “Think” in MapReduce to effectively write algorithms for systems including Hadoop and Spark. You will understand their limitations, design details, their relationship to databases, and their associated ecosystem of algorithms, extensions, and languages. write programs in Spark 6. Describe the landscape of specialized Big Data systems for graphs, arrays, and streams

Просмотреть программу курса

Получаемые навыки

Relational Algebra, Python Programming, Mapreduce, SQL

Рецензии

4.3 (оценок: 688)

5 stars
387 ratings
4 stars
183 ratings
3 stars
67 ratings
2 stars
33 ratings
1 star
18 ratings

DK

Jan 24, 2016

Good! I like the final (optional) project on running on a large dataset through EC2. The lectures aren't as polished and compact as they could be but certainly a very valuable course.

LG

Jun 20, 2017

Comprehensive and clear explanation of theory and interlinks of the up-to-date tools, languages, tendencies. Kudos and thanks to Bill Howe.\n\nHighly recommended.

Из урока

NoSQL: Systems and Concepts

NoSQL systems are purely about scale rather than analytics, and are arguably less relevant for the practicing data scientist. However, they occupy an important place in many practical big data platform architectures, and data scientists need to understand their limitations and strengths to use them effectively.

Spark Examples5:03

RDDs, Benefits6:11

Преподаватели

Bill Howe
Director of Research

Текст видео

[MUSIC]

So the concepts used in Spark are exactly those concepts you've been studying in

the context of other systems and you can apply all that knowledge you learned

in order to understand what's going on inside of Spark.

But there is some terminology that you should be familiar with that is

specifically associated with Spark.

One such term is the Resilient Distributed Dataset, the RDD.

This is a distributed collection of key value pairs, which should sound familiar,

because it's exactly the same data model that you used in Map Reduce and

Hadoop, and is very similar to the data model used even in a relational database.

The difference between a key value pair and

a record, potentially with a primary key defined, is not terribly significant.

The other key distinction about an RDD is that it can be persistent in memory across

tasks.

So unlike Map Reduce where we wrote everything out to disk after every task

for full tolerance reasons, these can hang out in memory and

you can apply multiple tasks to them across the durations, even across jobs.

And that in memory caching is really the key feature.

And we'll talk about that again in just a minute or two.

So working with these RDDs, there's really two kinds of actions you can take and

this is Spark terminology again.

There's transformations and then there's actions.

Transformations are the operators that we've been discussing, so this is the map,

and the reduce, and the filter, and the join, and the coat group and so on.

And they take in an RDD or more than one RDD and produce another RDD.

Actions on the other hand take RDD and produce a value,

extracting a value from the RDD into the host language, Python or Scholar and

Java, and they can be manipulated using that programming language.

So let's look at an example.

The text file is read from some path and

saved in a text file variable that creates an RDD.

We apply a transformation to it,

in this case the filter transformation that takes in another function.

This lambda, if you're not familiar with that term,

that's how you spell an anonymous function in Python.

Important thing is that this function takes one argument, the line and

then if that line contains the word spark, if spark is in line,

then this expression turns the value to true, otherwise it evaluates to false.

Only those records for which this whole function evaluates to true are saved into

the resulting data with lines sparks, and that's why it's named lines with spark.

And then finally, the action taken is perhaps to count

the number of records in linesWithSpark, which in this case, I guess, is 74,

or to take the first record out from this data set.

So these things actually return values, and one evaluation principle that

Spark uses is lazy evaluation, which means that transformations

aren't actually executed as soon as that line of code is encountered.

Rather it waits until an actual value must be returned to the host language before it

stacks up all the transformation required to compute that value,

potentially optimizes them and then runs them through and extracts the value.

Okay.

So there's a bunch of RDD operations.

We've seen that you can program directly in Map Reduce.

You have Map and you have Reduce calls that behave in the same ways that they do

in Map Reduce and Hadoop.

But you also have these other operators you can do that look a little bit more

like what you'd find in a relational algebra based system.

Including Pig or even more generally,

I shouldn't say more generally, or a database system.

Okay so you can just write a program in a variety of different ways,

all again Spark but all of these concepts you've already learned.

So let's remind ourselves of some of the important weaknesses of MapReduce/Hadoop.

The first of which is that Map and Reduce functions have to be written by hand,

so this could become tedious and error prone.

And this problem was recognized by people very early on, and

systems like HIVE and Pig were designed to address it, in part, so

you could program at a higher level model.

In the case of HIVE, it was SQL.

In the case of Pig,

it was this relational algebra based language that we explored and those

expressions would generate the distributed computation in MapReduce itself.

The second problem, or

a second problem, was that there are no indexes available for efficient lookup.

So if you only wanted to find a particular set of records in a very, very large

MapReduce data set, you had no choice but to scan the entire data set into memory.

Apply some Map function that identified the record you were interested in and

then passed those on to the next task.

And this problem was also recognized pretty early on and

addressed by attaching a key value store into the Hadoop ecosystem and

H/base which is based on this.

System called BigTable from Google is perhaps one of the most well

known examples of this.

So the third problem, though, is that every step in a MapReduce work flow

writes its entire output to disk for fault tolerance reasons.

And so this is good for fault tolerance, but

it's absolutely abysmal for performance.

And so this third problem is really what Spark was designed to fix.

The first problem and the second problem were addressed by other systems.

But the raw performance of Hadoop, primarily because of

the writing everything out to disk, perhaps not exclusively because of that,

was really seen as fatal and so bringing everything into memory and

operating on large data sets in memory was the key design goal of Spark itself.

So you can ask, does this optimization matter?

The answer is absolutely.

Right? So this first iteration of this task,

the times are approximately the same.

One is 80 seconds.

Spark is 80 seconds and Hadoop is 110 seconds.

But then as the computation proceeds every other iteration of Hadoop has to reread

the same data set over and over and over again, so the time just goes linearly up.

Spark only has to read it that first time.

And so all the other computation is able to, all the other iterations are able

to operate on it directly in memory and not take any additional time, and

so the savings can be enormous, especially for these repeated computations.

[MUSIC]

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2019 Все права защищены.

Загрузить из App Store

Загрузить в Google Play

Coursera
О проекте
Руководство
Карьера
Каталог
Сертификаты
Сертификаты MasterTrack™
Степени
Для организаций
Для правительственных организаций

Сообщество
Learners
Partners
Developers
Beta Testers
Translators

На связи
Блог
Facebook
LinkedIn
Twitter
Instagram
YouTube
Технический блог

Еще
Условия
Конфиденциальность
Помощь
Доступность
Пресса
Контакты
Справочник
Филиалы