Computation Graph, CUDA

Loading...

Из курса от партнера IBM

Applied AI with DeepLearning

151 оценки

IBM

Applied AI with DeepLearning

151 оценки

Курс 3 из 4 — Specialization Advanced Data Science with IBM

>>> By enrolling in this course you agree to the End User License Agreement as set out in the FAQ. Once enrolled you can access the license in the Resources area <<< This course, Applied Artificial Intelligence with DeepLearning, is part of the IBM Advanced Data Science Certificate which IBM is currently creating and gives you easy access to the invaluable insights into Deep Learning models used by experts in Natural Language Processing, Computer Vision, Time Series Analysis, and many other disciplines. We’ll learn about the fundamentals of Linear Algebra and Neural Networks. Then we introduce the most popular DeepLearning Frameworks like Keras, TensorFlow, PyTorch, DeepLearning4J and Apache SystemML. Keras and TensorFlow are making up the greatest portion of this course. We learn about Anomaly Detection, Time Series Forecasting, Image Recognition and Natural Language Processing by building up models using Keras one real-life examples from IoT (Internet of Things), Financial Marked Data, Literature or Image Databases. Finally, we learn how to scale those artificial brains using Kubernetes, Apache Spark and GPUs. IMPORTANT: THIS COURSE ALONE IS NOT SUFFICIENT TO OBTAIN THE "IBM Watson IoT Certified Data Scientist certificate". You need to take three other courses where two of them are currently built. The Specialization will be ready late spring, early summer 2018 Using these approaches, no matter what your skill levels in topics you would like to master, you can change your thinking and change your life. If you’re already an expert, this peep under the mental hood will give your ideas for turbocharging successful creation and deployment of DeepLearning models. If you’re struggling, you’ll see a structured treasure trove of practical techniques that walk you through what you need to do to get on track. If you’ve ever wanted to become better at anything, this course will help serve as your guide. Prerequisites: Some coding skills are necessary. Preferably python, but any other programming language will do fine. Also some basic understanding of math (linear algebra) is a plus, but we will cover that part in the first week as well. If you choose to take this course and earn the Coursera course certificate, you will also earn an IBM digital badge. To find out more about IBM digital badges follow the link ibm.biz/badging.

Из урока

deep learning frameworks

PyTorch Installation2:21

PyTorch Packages2:45

Tensor Creation and Visualization of Higher Dimensional Tensors6:08

Math Computation and Reshape7:08

Computation Graph, CUDA17:10

Linear Model17:44

Познакомьтесь с преподавателями

Romeo Kienzler
Chief Data Scientist, Course Lead
IBM Watson IoT
Niketan Pansare
Senior Software Engineer
IBM Research
Tom Hanlon
Training Director
Skymind
Max Pumperla
Deep Learning Engineer
Ilja Rasin
Data Scientist
IBM Watson Health

Welcome back. In the last session,

we have seen how to reshape a torch Tensor.

We saw that we are calling view method and this method would reshape our torch Tensor.

Today, we're going to discuss a very important topic.

This is the topic of Computational Graphs in PyTorch.

As we know in other deep learning frameworks,

we also deal with computational graphs like in Keras or in TensorFlow.

But, in these frameworks,

the computational graph are fixed,

so you create a model,

and as in Keras for example you define, for example,

if it is a neural network you define different layers, optimizer,

you define loss function,

number of epochs, the batch size,

and then you call compile.

If you call compile,

the run time will create a computational graph of

this model and this computational graph is fixed,

so you cannot change,

if you call model.fit it will execute the computational graph,

and during run time you cannot change it anymore, it's fixed.

And PyTorch has created something completely different.

The creators of PyTorch have decided that they

need a flexibility to change the computational graph at the runtime.

How did they accomplish this?

They have created a component which is called autograd.Variable.

This is the main building block of computational graph in PyTorch. How it works?

Let us see, let us execute this next cell.

So, first of all, we are creating autograd.Variable,

we are passing two parameters.

One parameter is actually data of this autograd.Variable.

This is a torch Tensor of size three,

and then we can pass the second argument which is requires_grad,

which is the meaning requires gradient, true or false.

It means following, it will say requires_grad true,

we are saying that this autograd.Variable should track how it was created.

This is very important.

So, we can print out here the data,

which is inside of this autograd.Variable.

This is simply the tensor of size three; one, two, three.

Then we are creating the next autograd.Variable,

which is Y. autograd.Variable, y the same.

We are passing this argument, data,

this is the first parameter,

then we are passing the second parameter,

requires gradient true or false, we're passing true.

And then we are adding up x and y,

and here we are printing the data in the component z,

which is the sum of x and y.

So, this is the sum,

so we just have summed up element wise two vectors.

But here, what is very interesting here is that we can also see how z was created.

If we write z.grad, function grad_fn,

and print out this operation,

we see that z was created by add operation.

It's very interesting. Let us see next example.

Here we are summing up the components of the z vector.

So, z vector we have created a sum of x and y,

this is five, seven, nine.

If we add up all those elements we are getting 21.

Nothing very interesting and special. But if we call in the next step,

if we call s grad function,

grad fn, we see that it was created as a sum.

So, the PyTorch knows exactly for every variable how it was created.

Here we are with the next example,

in here we have a function s. This function s is

the sum of two variable vectors of size three,

so we are summing up this vectors element wise.

So, we are summing the first element of vector x,

with the first element of vector y.

The second element with the second and third with the third.

So, now if we create partial differentiation,

so if we differentiate s

with respect to x_0, for example.

Due to the rules of partial differentiation,

we would deal with other variables other than x,

y as with the constants.

So, the partial differentiation for s,

for the function s, with respect to variable x_0 is one.

Let us see how it works with PyTorch.

So, first of all, let us recap that s was the sum.

This is actually the sum element wise.

And here we are calling backward,

and backward means, start back propagation from this point backwards.

So, it's starting back propagation,

so we have the function s which is the sum,

and then we say backwards,

and here we can pass the argument retain_graph true.

I will not go into details,

but it's optional actually.

So, here we round this in Excel what we see here is following.

So, first of all, we are printing vector x,

we see he has nothing special with just printing out this vector.

But, now if we print out the gradients,

not the gradient function as in a previous time,

but now we are we can print out the gradient.

Meaning gradient is the value of,

if we create differentiation,

if we differentiate the function at a special point,

we're getting also the value of this partial differentiation.

And here where it differentiates this function.

With respect to variable x,

x is consisting of three variables: x zero,

x one, x three.

So if we differentiate partially with respect to x zero,

we're getting one and then,

again with respect to x one,

we're getting one and with respect to x two, we're getting one.

We saw that this is very simple function.

If we create partial differentiation,

it would be one and all other variables would get zeros.

The level over here, we have one, one, one.

The same thing if we create differentiation,

partial differentiation with respect to y.

Because y is also, as x,

a simple plain variable here without any coefficient or exponentiations.

It's just one, one, one.

But what will happen if we call again <Backward>?

And then we call again: (x.grad) and (y.grad).

You see, it was updated.

So now, we have here two, two, two and

that was for x and for y the same, two, two, two.

What has happened? Let us call again <Backward> and again print out.

Now it's again changed.

It's three. The reason is,

that every time you call <Backward>,

the gradient property is accumulated.

This is a technology of PyTorch which is used for special,

there are some models where it's very convenient to use.

In this introductory session,

we will not use it but it's important to know that: every time if you call <Backward>,

the gradient will be accumulated.

Next important point is how to preserve the computational graph.

As you know, as you already know,

the autograd of variables are consisting of two components.

One is the data.

Another one is the grad fn, gradient function.

So with the data,

you can get out of the gradient.

Of the autograd variable,

you can get out the data.

And with grad fn,

you can get out of the function how this variable was created.

Let us execute next cell.

In this cell, we are creating two Tensors of size two.

And then we are summing up these Tensors.

So very simple.

So we're getting actually Tensor of size two by two, sorry.

So now let us execute next cell.

Here, we're creating autograd variable x, out of x,

and autograd variable y, out of y,

and saying both times (requires_gradient=True).

So now I can sum up these variables:

var_x + var_y and then we're printing out the gradient function,

how it was created.

Let us do this and we see z was created with add operation, <AddBackward>.

But now, what we do is the Following: we are extracting

the data out of this sum. Just the data.

And passing it to a new variable which is called var_z_data.

And then, we are creating actually new variable, new autograd.Variable,

and passing this new data variable in it and

then we are printing out and then we are trying to print out how it was created.

So we're actually printing out new_variable_z.gradient_function.

And then you see, none.

It's lost, because here,

if we are extracting the data.

We have extracted only data but not the gradient function.

And, yes, then the computational graph at this point is already broken so

the grad_fn was not passed and we

have retained only data but not the gradient function.

Now, if we tried to call <Backward> on this new autograd variable,

new_var_z which was created again with the data out of var_z,

we will get exception because there's nothing there.

Here we have a runtime error

that the element of variable does not require grad and does not have a gradient function.

This is the important point.

Does not have a gradient function.

Gradient function was lost and last but not least,

in this session, I would like to

briefly mention the CUDA functionality of the Torch Tensor.

As you probably remember,

the keras in Tensor flow,

automatically detect whether you have GPU acceleration on your machine or not.

And they will execute everything on the GPU,

if your GPU is available,

or on CPU if GPU is not available,

for PyTorch, you can very granually decide what to execute on GPU

and which Tensors you want to execute where, on GPU or on CPU.

Here you have a check.

And you can check torch.CUDA.is available().

And this check you can run every time if you want to decide where to execute the Tensor.

If you want to execute the tensor on CUDA,

and CUDA is available,

you just add CUDA function,.cuda().

And then it will run this Tensor on GPU.

If you don't want to run the Tensor on GPU,

you don't add this CUDA function at the end and it will run on CPU.

This is very, very flexible.

So in my opinion,

the main advantage of PyTorch is its huge flexibility.

Okay. I hope you enjoyed the session and next time we're going

to try something really practical.

We're going to build up a linear model with PyTorch.

See you then. Enjoy our sessions. Bye bye.

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2018 Все права защищены.

Загрузить из App Store

Загрузить в Google Play