Explore
For Enterprise
Join for Free
Log In

Гуманитарные науки и искусства
Бизнес
Компьютерные науки
Наука о данных
Информационные технологии
здоровье
Математика и логика
Личное развитие
Естественные и технические науки
Социальные науки
Изучение языков
Степени
Сертификаты

Полный обзор Coursera

Обзор
Поиск
Для организаций
Войти
Присоединиться бесплатно

12.5 - Parsing Web Pages

Loading...

Using Python to Access Web Data

Мичиганский университет

4.8 (оценок: 20,861) | Зарегистрировано учащихся: 230K

Курс 3 из 5 — Python для всех Специализация

This course will show how one can treat the Internet as a source of data. We will scrape, parse, and read web data as well as access data using web APIs. We will work with HTML, XML, and JSON data formats in Python. This course will cover Chapters 11-13 of the textbook “Python for Everybody”. To succeed in this course, you should be familiar with the material covered in Chapters 1-10 of the textbook and the first two courses in this specialization. These topics include variables and expressions, conditional execution (loops, branching, and try/except), functions, Python data structures (strings, lists, dictionaries, and tuples), and manipulating files. This course covers Python 3.

Просмотреть программу курса

Получаемые навыки

Json, Xml, Python Programming, Web Scraping

Рецензии

4.8 (оценок: 20,861)

5 stars
17,304 ratings
4 stars
2,984 ratings
3 stars
459 ratings
2 stars
65 ratings
1 star
49 ratings

AG

Sep 13, 2017

The course teaches the basics of how to interact with the internet using Python. The course is simple, well explained, has great interviews with people who were part of the creation of the internet.

RG

May 31, 2018

This course really helped me in learning how the network and api work. Many things happen in the background which we are not aware of. To understand those things one should really try this course.

Из урока

Programs that Surf the Web (Chapter 12)

In this section we learn to use Python to retrieve data from web sites and APIs over the Internet.

12.3 - Unicode Characters and Strings17:17

12.4 - Retrieving Web Pages6:36

Worked Example: Using Urllib (Chapter 12)3:39

12.5 - Parsing Web Pages9:28

Worked Example: BeautifulSoup (Chapter 12)9:20

Преподаватели

Charles Russell Severance
Professor

Текст видео

So now we're just going to talk a little bit about web scraping.

I want you to come and visit this again later in the book.

But the basic idea is just now we have this protocol that we can send

GET requests and get data back.

That's how browsers talk to servers.

What scraping really is, is when we write a Python program that fakes or

pretends to be a browser.

It retrieves the web pages, it looks at them, and maybe extracts some information,

maybe for searching or for more links and then goes and

perhaps gets some more pages.

And we call this either spidering the web because it kind of works its way out like

a spider's web.

Or we call it crawling the web, so it's kind of crawling and crawling

and crawling from one place to another, and eventually it's going to crawl

one link at a time, and it finds all or nearly all the web.

Now, why might you scrape?

Well, you might scrape data that you can't get any other way.

Now, you've got to be careful, because it might not be legal to do this.

You want to scrape the inventory of a competitor.

Maybe you're not supposed to do that or maybe you posted a bunch

of things to a blog and you lost the blog is about to get shut down.

I actually used scraping recently for

a Wiki that I typed a bunch of stuff into someone else's Wiki.

And they were going to shut the Wiki down and I'm like, I want my pages back.

And I could have pulled them all out by hand, but I just wrote a scraper.

And I scraped all my pages back.

You might write something that runs every night to

check to see if something changed.

Or you might build yourself a simple equivalent of a search engine.

Now, there's lots of controversy about it.

People who build web sites often copyright their material and say you can't have it.

You can't just like make a online store by

borrowing Amazon's inventory from their pages.

They will be very unhappy about that and they'll catch you.

They have terms of service and they can block your network.

They can block your computer.

They can shut your account off.

And so be very careful about the things that you spider.

Spider things like Wikipedia or

dr-chuck.com, somewhere that really doesn't care if you look at their stuff and

actually is happy to have their stuff looked at.

So the biggest problem in spidering is the parsing of the HTML that comes back.

And like I said before HTML is a nasty little representation and

it turns out that when your browser retrieves HTML it goes through

a whole bunch of things that effectively forgives syntax errors in HTML.

Because they figured ah, we'll just show it.

Why tell people about each syntax error?

So it turns out that there is

lots of HTML on the web that has syntax errors in it but you don't even notice it.

And so you could do regular expressions, you could do finds, you could do splits,

you could do all kinds of things and you would find quickly it'll work for

the first two pages but the third through the thousandth page blows up because.

I didn't realize that's what the href,

I didn't realize you could put a newline there.

Who would have put a newline there, why would they do that, and then you fix that.

And you realize after you try to fix this and you try to parse all the links that

there is just so many variations and someone has already done that.

And it's a library called BeautifulSoup from a place called crummy.com

I think the naming of this is all sort of this tongue in cheek of what

a mess HTML is.

And so instead of calling it HTML super parser, they just call it something silly

because it's kind of a silly problem because HTML on the web is just so bad.

And so it's kind of fun.

But once you use this

you've taken the easy way out.

Now, the first thing that you've got to do is you've got to install BeautifulSoup and

there's two ways to do it.

I put in the code, the code folders, and all the zip files.

So you can either go and follow the instructions at their site and

install it, and that installs it for all Python programs.

That's fine if you can figure that out but sometimes you're running on

a campus computer and you can't actually reinstall software on that, maybe you're

bringing your python programs on a USB stick or on a shared drive or something.

And so you actually kind of have to locally install it and so if you make this

file, you actually can download this file bs4.zip and then unzip it.

And so if you put it in a folder with the URL links, this one.

.py

And there's a folder bs4 and then this is actually a folder and

then all the stuff underneath it.

Then in that one folder, BeautifulSoup will work, okay?

So you can either have it installed for all Pythons or

you can put it in any folder that you're going to use BeautifulSoup

you make this subfolder called bs4.

And it's got some .py files in it and etc, etc.

So you unzip.

You download this file and unzip it in to that same folder.

And then, this import will work.

Now until you get BeautifulSoup properly installed,

this import is going to blow up.

And so you just keep running it until the import works.

And then the import works and you can focus on the rest of the program.

So, you got to install it one way or the other.

Look at the instructions.

I'll probably have a little screen capture or

something that shows you how I make this work.

Okay, so assuming that you can import BeautifulSoup,

we're going to read some data from the net.

And we're going to import the BeautifulSoup library.

We're going to ask for a URL.

Then we are going to open it, and then do a .read, okay? We're going to do a .read.

Remember what .read is?

That says read it all.

It's not a for loop that reads it.

It reads the whole blob with newlines at the end, and

it all comes in to one big string.

And then we're going to tell BeautifulSoup to parse it.

Okay, now BeautifulSoup is smart enough to convert all this stuff so

it just takes the whole file and uses this HTML parser.

There's some other options that you could have here,

but that's the one that I recommend in this case.

And then you get back an object.

And what happens is, in this soup variable,

it's somehow taken all the nasty bits that come from the web page

and cleaned them up, and made a little pretty tree of things.

You don't have to worry about the internal structure, it is a soup object.

I named the variable soup just because it's cute to

name it soup. That could be x for all we care.

And you can ask soup some questions at that point.

So it takes the nasty HTML and cleans it up.

That's what this parsing is doing.

It's like, that's the ad hoc heuristic challenging difficult part of it.

Thank you BeautifulSoup for writing such a wonderful library.

And then we can ask questions of it and this first question that we can ask

and BeautifulSoup is a little tricky to use and so you'll end up reading some

documentation if you want to build really sophisticated programs.

But this basically is, we are going to go through all of the anchor tags in

a document, give me a list of all the anchor tags in a document.

So these tags are sitting there in this document, you know,

blah blah blah anchor tag, blah blah blah anchor tag, blah blah blah anchor tag.

And so what this basically says is give me a list of those anchor tags.

Anchor tag, anchor tag, anchor tag, and comma, comma, comma.

Now these tags are themselves objects, and it has the data in the anchor tag and

you have to kind of query those.

But given that this is a list, then the tag is going to go through

each of the anchor tags in the document and iterate through them.

And it's going to pull out the href.

Href equals quote ... quote, right?

It's pulling this text out and that's what we get using tag.get.

So this is like a dictionary.

Get the href key or None.

And so then this will loop through.

It retrieves dr-chuck.com/page1.htm.

And then it prints out trivially with no regular expressions, no nothing.

It just prints the links out.

Now if you do this to a page that has more links you will get many links here, okay?

So that is a simple use of the BeautifulSoup library

to retrieve and parse HTML and

pull out anchor tags, which is really sort of the beginning of a browser.

I mean, a beginning of a web crawler.

So if we go all the way back to the beginning of this chapter,

we started with the notion of computers making phone calls using sockets.

They call to a host, and a port within that host, and

then they have a two-way socket that they can send and retrieve data.

We have to consult the application protocols like HTTP.

HyperText Transfer Protocol.

To figure out what we send and how we send it and when we send it.

Who talks first, what the responses are, and what the data is, and

how to interpret that data.

Like the headers, followed by a blank line, followed by the data.

That's all part of the protocol, right?

And then we made it even simpler with urllib, so that's just like one line, go get it.

If you do a .read, you get the data as well.

And then with BeautifulSoup, if you're dealing with HTML,

BeautifulSoup really simplifies it.

So in a surprisingly few lines of code you can do the hard part of a web crawler.

And that's one of the things that people really like about Python.

You think, I want to do this.

And like 30 minutes later, you've got a basic web crawler working.

And so the support for HTML, HTTP, and sockets in Python

is one of the very, very charming things that people really like about Python.

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2019 Все права защищены.

Загрузить из App Store

Загрузить в Google Play

Coursera
О проекте
Руководство
Карьера
Каталог
Сертификаты
Сертификаты MasterTrack™
Степени
Для организаций
Для правительственных организаций

Сообщество
Learners
Partners
Developers
Beta Testers
Translators

На связи
Блог
Facebook
LinkedIn
Twitter
Instagram
YouTube
Технический блог

Еще
Условия
Конфиденциальность
Помощь
Доступность
Пресса
Контакты
Справочник
Филиалы