Об этом курсе: This course is for novice programmers or business people who would like to understand the core tools used to wrangle and analyze big data. With no prior experience, you will have the opportunity to walk through hands-on examples with Hadoop and Spark frameworks, two of the most common in the industry. You will be comfortable explaining the specific components and basic processes of the Hadoop architecture, software stack, and execution environment. In the assignments you will be guided in how data scientists apply the important concepts and techniques such as Map-Reduce that are used to solve fundamental problems in big data. You'll feel empowered to have conversations about big data and the data analysis process.
Автор: Калифорнийский университет в Сан-Диего
Преподаватели: Natasha Balac, Director, Predictive Analytics Center of Excellence (PACE)
San Diego Supercomputer Center
Преподаватели: Paul Rodriguez, Research Programmer
San Diego Supercomputer Center (SDSC)
Преподаватели: Andrea Zonca, HPC Applications Specialist
San Diego Supercomputer Center (SDSC)
| Выполнение | 5 weeks of study, 1-2 hours/week |
| Язык | English |
| Как пройти курс | Чтобы пройти курс, выполните все оцениваемые задания. |
Программа курса
НЕДЕЛЯ 1
Hadoop Basics
Welcome to the first module of the Big Data Platform course. This first module will provide insight into Big Data Hype, its technologies opportunities and challenges. We will take a deeper look into the Hadoop stack and tool and technologies associated with Big Data solutions.
7 видео, 4 материалов для самостоятельного изучения
- Видео: Hadoop Stack Basics
- Видео: The Apache Framework: Basic Modules
- Видео: Hadoop Distributed File System (HDFS)
- Видео: The Hadoop "Zoo"
- Видео: Hadoop Ecosystem Major Components
- Reading: Apache Hadoop Ecosystem
- Reading: Lesson 1 Slides (PDF)
- Reading: Hardware & Software Requirements
- Видео: Exploring the Cloudera VM: Hands-On Part 1
- Видео: Exploring the Cloudera VM: Hands-On Part 2
- Reading: Lesson 2 Slides - Cloudera VM Tour
Оцениваемый: Basic Hadoop Stack
НЕДЕЛЯ 2
Introduction to the Hadoop Stack
In this module we will take a detailed look at the Hadoop stack ranging from the basic HDFS components, to application execution frameworks, and languages, services.
10 видео, 6 материалов для самостоятельного изучения
- Видео: Overview of the Hadoop Stack
- Видео: The Hadoop Distributed File System (HDFS) and HDFS2
- Видео: MapReduce Framework and YARN
- Reading: Hadoop Basics - Lesson 1 Slides
- Видео: The Hadoop Execution Environment
- Видео: YARN, Tez, and Spark
- Видео: Hadoop Resource Scheduling
- Reading: Lesson 2: Hadoop Execution Environment - Slides
- Видео: Hadoop-Based Applications
- Видео: Introduction to Apache Pig
- Видео: Introduction to Apache HIVE
- Видео: Introduction to Apache HBASE
- Reading: Lesson 3: Hadoop-Based Applications Overview - All Slides
- Reading: Command list for Applications Slides
- Reading: Tips to handle service connection errors
- Reading: References for Applications
Оцениваемый: Overview of Hadoop Stack
Оцениваемый: Hadoop Execution Environment
Оцениваемый: Hadoop Applications
НЕДЕЛЯ 3
Introduction to Hadoop Distributed File System (HDFS)
In this module we will take a detailed look at the Hadoop Distributed File System (HDFS). We will cover the main design goals of HDFS, understand the read/write process to HDFS, the main configuration parameters that can be tuned to control HDFS performance and robustness, and get an overview of the different ways you can access data on HDFS.
9 видео, 5 материалов для самостоятельного изучения
- Видео: Overview of HDFS Architecture
- Видео: The HDFS Performance Envelope
- Видео: Read/Write Processes in HDFS
- Reading: Lesson 1: Introduction to HDFS - Slides
- Reading: HDFS references
- Видео: HDFS Tuning Parameters
- Видео: HDFS Performance and Robustness
- Reading: Lesson 2: HDFS Performance and Tuning - Slides
- Видео: Overview of HDFS Access, APIs, and Applications
- Видео: HDFS Commands
- Видео: Native Java API for HDFS
- Видео: REST API for HDFS
- Reading: HDFS Access, APIs
- Reading: Lesson 3: HDFS Access, APIs, Applications - Slides
Оцениваемый: HDFS Architecture
Оцениваемый: HDFS performance,tuning, and robustness
Оцениваемый: Accessing HDFS
НЕДЕЛЯ 4
Introduction to Map/Reduce
This module will introduce Map/Reduce concepts and practice. You will learn about the big idea of Map/Reduce and you will learn how to design, implement, and execute tasks in the map/reduce framework. You will also learn the trade-offs in map/reduce and how that motivates other tools.
9 видео, 3 материалов для самостоятельного изучения
- Видео: Introduction to Map/Reduce
- Видео: The Map/Reduce Framework
- Видео: A MapReduce Example: Wordcount in detail
- Reading: Lesson 1: Introduction to MapReduce - Slides
- Reading: A note on debugging map/reduce programs.
- Видео: MapReduce: Intro to Examples and Principles
- Видео: MapReduce Example: Trending Wordcount
- Видео: MapReduce Example: Joining Data
- Видео: MapReduce Example: Vector Multiplication
- Видео: Computational Costs of Vector Multiplication
- Видео: MapReduce Summary
- Reading: Lesson 2: MapReduce Examples and Principles - Slides
Оцениваемый: Running Wordcount with Hadoop streaming, using Python code
Оцениваемый: Lesson 1 Review
Оцениваемый: Joining Data
НЕДЕЛЯ 5
Spark
Welcome to module 5, Introduction to Spark, this week we will focus on the Apache Spark cluster computing framework, an important contender of Hadoop MapReduce in the Big Data Arena.
Spark provides great performance advantages over Hadoop MapReduce,especially for iterative algorithms, thanks to in-memory caching. Also, gives Data Scientists an easier way to write their analysis pipeline in Python and Scala,even providing interactive shells to play live with data.
10 видео, 4 материалов для самостоятельного изучения
- Видео: Introduction to Apache Spark
- Видео: Architecture of Spark
- Reading: Setup PySpark on the Cloudera VM
- Reading: Lesson 1: Intro to Apache Spark - Slides
- Видео: Resilient Distributed Datasets
- Видео: Spark Transformations
- Видео: Wide Transformations
- Reading: Lesson 2: RDD and Transformations - Slides
- Видео: Directed Acyclic Graph (DAG) Scheduler
- Видео: Actions in Spark
- Видео: Memory Caching in Spark
- Видео: Broadcast Variables
- Видео: Accumulators
- Reading: Lesson 3: Scheduling, Actions, Caching - Slides
Оцениваемый: Spark Lesson 1
Оцениваемый: Spark Lesson 2
Оцениваемый: Simple Join in Spark
Оцениваемый: Spark Lesson 3
Оцениваемый: Advanced Join in Spark
Часто задаваемые вопросы
Как это работает
Coursework
Each course is like an interactive textbook, featuring pre-recorded videos, quizzes and projects.
Help from Your Peers
Connect with thousands of other learners and debate ideas, discuss course material, and get help mastering concepts.
Certificates
Earn official recognition for your work, and share your success with friends, colleagues, and employers.
Авторы
Калифорнийский университет в Сан-Диего
UC San Diego is an academic powerhouse and economic engine, recognized as one of the top 10 public universities by U.S. News and World Report. Innovation is central to who we are and what we do. Here, students learn that knowledge isn't just acquired in the classroom—life is their laboratory.
Стоимость
| Приобрести курс | |
|---|---|
| Получить доступ к материалам курса | Доступен |
| Получить доступ к оцениваемым материалам курса | Доступен |
| Получить итоговую оценку | Доступен |
| Earn a shareable Course Certificate | Доступен |
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
