>>> By enrolling in this course you agree to the End User License Agreement as set out in the FAQ. Once enrolled you can access the license in the Resources area <<< This course, Applied Artificial Intelligence with DeepLearning, is part of the IBM Advanced Data Science Certificate which IBM is currently creating and gives you easy access to the invaluable insights into Deep Learning models used by experts in Natural Language Processing, Computer Vision, Time Series Analysis, and many other disciplines. We’ll learn about the fundamentals of Linear Algebra and Neural Networks. Then we introduce the most popular DeepLearning Frameworks like Keras, TensorFlow, PyTorch, DeepLearning4J and Apache SystemML. Keras and TensorFlow are making up the greatest portion of this course. We learn about Anomaly Detection, Time Series Forecasting, Image Recognition and Natural Language Processing by building up models using Keras one real-life examples from IoT (Internet of Things), Financial Marked Data, Literature or Image Databases. Finally, we learn how to scale those artificial brains using Kubernetes, Apache Spark and GPUs. IMPORTANT: THIS COURSE ALONE IS NOT SUFFICIENT TO OBTAIN THE "IBM Watson IoT Certified Data Scientist certificate". You need to take three other courses where two of them are currently built. The Specialization will be ready late spring, early summer 2018 Using these approaches, no matter what your skill levels in topics you would like to master, you can change your thinking and change your life. If you’re already an expert, this peep under the mental hood will give your ideas for turbocharging successful creation and deployment of DeepLearning models. If you’re struggling, you’ll see a structured treasure trove of practical techniques that walk you through what you need to do to get on track. If you’ve ever wanted to become better at anything, this course will help serve as your guide. Prerequisites: Some coding skills are necessary. Preferably python, but any other programming language will do fine. Also some basic understanding of math (linear algebra) is a plus, but we will cover that part in the first week as well. If you choose to take this course and earn the Coursera course certificate, you will also earn an IBM digital badge. To find out more about IBM digital badges follow the link ibm.biz/badging.
Methods for neural network training
Loading...
From the course by IBM
Applied AI with DeepLearning
99 оценки
From the lesson
Introduction to deep learning
Meet the Instructors
Romeo KienzlerChief Data Scientist, Course Lead
IBM Watson IoT
Niketan PansareSenior Software Engineer
IBM Research
Tom HanlonTraining Director
Skymind
Max PumperlaDeep Learning Engineer
Ilja RasinData Scientist
IBM Watson Health
So in order to get an idea how neural networks are trained,
we'll use a very simple neural network in Python.
First of all, let's recap
how to so-called feedforward path for feedforward network is computed.
Remember, that you have an input vector X and the label vector Y.
Especially in the case of classification, Y is a vector.
For regression, Y is often a scalar but for the math it doesn't matter,
since we are working with vectors and matrices anyway.
So, the task of a neural network now is to compute Y_hat
from an input vector X using the weight matrices
W1 and W2 such that Y_hat and
Y are as close as possible for all vectors in the training set X.
Let's actually have a look at the map.
So to compute from left to right,
the first thing you do is multiply X by W1 to obtain C2.
Since we are squashing intermediate layer calculations with activation functions,
we have to apply the activation function to C2 to obtain A2.
Then we compute the output layer.
So note that we are multiplying A2 by W2.
This is some sort of stacking of the computations on top of each other.
And finally, we compute Y_hat by applying the activation function to C3 and we are done.
So our weight matrices W1 and W2 are still randomly initialized.
Therefore, Y_hat and Y are different.
So now, we have to train this neural network to do
a better job and training basically means updating
the weights in order to obtain the best Y_hat on
a given X with respect to Y for all data points in X.
But how do we find the optimal values for the weights?
This is the plot,
our two weight values and the cost.
So the cost basically tells you how could or better your weights are chosen.
The lower the cost, the better neural network performance.
The cost function is called tree.
Note that it depends on W and T. So W is the weight matrix of the neural network.
This is the one you want to optimize over.
And T stands for a training dataset,
which stays always the same.
Therefore, tree is dependent on W. This is a two-dimensional example,
but in reality,
this so-called optimization space is very high-dimensional and impossible to plot.
So, how can we now find good values for W?
Of course, we can check every possible combination of values for W or can't we?
So one way of doing this is called Grid Search.
So, in this example, we choose a discrete set of values within W1 and W2
and just apply the cost function tree to
all of those combinations and the one combination
with the lowest cost is an optimal value set for W. Now,
imagine that each individual scalar weight can
be assigned to a value between minus 1 and 1.
Now, we create a grid over this continuous range with the step size of 0.1.
So this basically forms a grid, therefore,
the method is called Grid Search.
And now imagine, we have only a single scalar weight value to optimize over.
So, we have to test 21 individual values.
If we add one more weight,
we already have to test 411,
in three dimensions, we are on 9,261 combinations.
In four dimensions, we already have to test and compute 194,481 values.
And as we see,
this number keeps growing exponentially.
With only 100 dimensions,
you have to test more combinations than the atoms in the universe.
So, this obviously is not a good way for neural network training.
So, what if we not strictly follow a grid
and just randomly select value combinations for the weights.
So it turns out that this method,
which is called Monte Carlo, outperforms Grid Search.
But still in order to get a good approximation of the cost function,
we need to test a lot of values which is not feasible for neural network training.
So what if we had a possibility when evaluating
a single value to know in
which direction we should go in order to test for the next value.
We then could climb down the ladder one step by another until we reach the optimal value.
This method is called Gradient Descent and with all its variants,
it's [inaudible] for neural network training. So how does this work?
Let's have a look at our cost function tree.
In this case, the so-called quadratic cost,
which basically sums the squares of the differences between
the prediction of the neural network and the actual real value.
This difference is called error.
Therefore, this is also called a sum of squared errors.
The axis of a cost function as well.
The cross entropy, exponential cost,
hellinger distance, and many more.
Choosing an appropriate cost function depends on your data and neural network topology
and is part of the so-called hyperparameter space.
An idea selection is often considered as black magic,
in other words, trial and error.
So if we now have a look, how Y_hat is calculated,
we notice that this is dependent on the activation function F and on C3.
C3 is calculated by the activation of
the previous layer multiplied by the weights of the actual layer.
If you replace the two by its definition,
you obtain the following form and if you replace C2,
it's X multiplied by the weight of layer 1 and we obtain this form.
Finally, you put this form into tree and we obtain
the cost function tree based on
all individual computations of the forward pass of the neural network.
So, now, we will use the so-called backpropagation method.
This calculates the neural network computations backwards and
it does so by back propagating the error Y minus Y_hat.
Intuitively, this means that on the reverse computation of the neural network,
big weight values end up in back
propagating big amounts of the error to the upstream layers,
where a small weight value only let small amounts of the error being back propagated.
So let's assume, on a given data row,
we end up with a certain amount of error indicated by red circle.
So, if we now backpropagate this error,
we use the weight to determine how much of this error should be back propagated.
Since the weight is high,
we also increase the error,
which then gets back propagated.
Now, if the weight is small,
the error also gets reduced and the amount of error for
that particular back propagation to the upstream layer neuron is less.
This is the desired behavior since big weights
are contributing more to the overall error than small weights.
So, this is nice but what we actually want is not the error or cost on each neuron,
but the gradient of each error.
If we know the gradient,
we know in which direction we have to update
each weight in order to go down here to cross the hypersurface.
Therefore, we compute the partition derivative with respect to W2 first.
I skipped the linear algebra and the calculus here,
but it turns out that derivative is A2 transpose times delta 3,
which is based under reverse computation,
which is also called backpropagation.
So note that F_prime is the first derivative of the activation function F. Now,
we continue in back propagating the error and taking
the first derivative of that function in parallel to obtain delta 2.
And finally, we get the partial derivative with respect to W1.
And again, the mathematical reason why this is computed using the transposed of
X times delta 2 is beyond the scope of this class and due to the Chain rule in calculus.
So as you have seen, training the neural network can be quite complicated.
But it's okay if you didn't understand all the details,
just note the gradient descent does not make any guarantees that
it converges just to the global optimum unless the cost function is convex,
which is not the case in neural networks.
So it might get stuck in local optimas and even has not any guarantees to convert it all.
Therefore, a lot of hyperparameter tuning might be necessary.
In the next module, we will cover TensorFlow,
one of the hottest deep learning frameworks out now.
Ознакомьтесь с нашим каталогом
Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
