Deep Learning Video Lecture | What is Data Analytics? - Data & Analytics

so in the previous module we saw how we
can use that propagation in order to
find the weights of her neural network
three layers of liquid so one of the
only success stories of backpropagation
was learning to recognize handwritten
digits in an address right so this was
work done by an takuna back in 89 and
essentially they trained a slightly more
complex network architecture called the
convolutional neural network in order to
recognize handwritten digits you can see
the the variation in the in the digits
that their network managed to handle and
the listed fairly well so another early
success story of our neural networks was
the backgammon player built by Jerry
trasero from IBM Kathy J Watson Labs in
1992 so he built two versions of this
backgammon play one called the neuro
gamma which came in 89 which was
essentially a neural network cut line
using back propagation in a supervised
learning manner now in order to play a
game of backgammon so it's like you know
if people know about it so you throw a
dice you throw a dice and depending on
the the die roll you move your coins
around so there is a white and a black
site so the idea is to move all your
coins off the board by repeatedly
rolling the die so the the the TD gammon
player essentially used a three layer
standard three layer neural architecture
and that was trained using a specific
form of what is called reinforcement
learning you look at reinforcement
learning in a later module but the he
was reinforcement learning in order to
generate the error signals and but then
use back propagation to train the
weights of the hidden unit and he
managed to build backgammon player which
was able to beat human backgammon
champions in gameplay so this was some
of the early successes and a lot of
interest was being spent on
looking at neural networks for solving a
variety of different problems Hey but
then again people discovered a second
drawback with neural networks so the one
thing was that it was incredibly hard to
claim a neural network because he had so
many prior because you had to clean them
I had to treat them appropriately so
that they decide results were obtained
further there was this problem called
the vanishing gradient problem so people
quickly figured out that if you have
many layers in the neural network right
now it was easier for the network to
represent more complex functions even
though a two layer network or a 3 layer
Network and depending on how we counted
and the standard three layer network was
a universal approximator having more
layers allowed more complex
representations to be built but past the
number of layers in the neural network
increases the the gradient that we
compute by doing back propagation
becomes vanishingly small right and
since there are not enough feedback for
the early layers in the network the
weights in the lower layer remains
random Maryborough initialized them
right and so you are not able to learn
any deep networks right so the network's
the neural networks that you learn this
is something had to be shallow because
only a few three layers could be three
or four layers could be trained
meaningfully using that propagation so
people have to come up with different
tricks for trainings deeper networks so
on this discovery of these tricks for
training deeper networks is essentially
what has revived the interest in the
field and now we can build networks that
have eight layers ten layers and so on
so forth and this produced some
fantastic performances in problems that
were thought to be very hard to solve
for AI and machine learning and so
there's been this revived interest in
looking at neural networks so I will
talk about one specific mechanism for
training multiple layers in a network
and then you can if you're interested in
this you can follow it up with other
between so this was proposed in
mid-2000s and by hinton in 2006 and
independently by Arab India and others
in 2007 it's more like a greedy
unsupervised layer by spree training so
what we mean by this so you start off
with a very simple Network architecture
called an autoencoder
so you have an input layer right and you
want to produce the same input at the
output so the input layer and output
layer need to be identical but in
between the connections will go through
a the smaller hidden layer of neurons so
the idea here is that the smaller layer
is going to learn some kind of a
encoding or some kind of a reduced
representation of the input that is
sufficient for you to produce the output
that you are looking for so you could
train this using back rock or other
slightly more advanced gradient
techniques so if you think about it
there is no real supervision that is
required here because as soon as I have
the input and the output is just the
same input and I'm going to send it
through a smaller hidden layer so that I
am learning a reduced representation of
the input so once I have this first
level of reduced representation I am
going to iteratively deepen the network
so what I do so once I have the first
hidden layer of representation the first
hidden layer of a presentation I am
going to add another autoencoder of
network on top of it right so this
network takes the hidden layer
representation for the input and tries
to produce the same hidden layer the
presentation at the output but in the
middle layer is going to have fewer
neurons so in the effect I am taking
this larger input and reducing it to a
smaller you don't the presentation here
and I can't repeat this right so once I
have trained this I remove the outer
layer okay and then tack on an another
layer of auto-encoders so now we can see
that my training has gone several layers
deep so instead of just looking at
one layer deep autoencoder which I was
able to try using back for efficiently
so I'm essentially using the same
construction again in the game so at any
point the training happens only or now
one layer whatever recorder right but
then because of this iterative deepening
now I have actually taken this input
representation and progressively reduced
it to a much smaller representation at
the hidden layer so this kind of an
observation that you can use this layer
wise pre-training of the data led to a
resurgence in a lot of deep
architectures so why do we call this
free training so far I have not talked
about any kind of classification tasks
that you are performing you're just
trying to find features in the input
space so once I have found these
features in the input space then I can
take this and I can tack on top of it
right on top of these features that they
have learned
I can tack on another butyl network and
now I have my full-fledged deep
autoencoder and then this hidden layer
representation can now be used as an
input to any learning task right so so
this part is called the the fine-tuning
part and the the layer wise training
part is called the pre training part
where I find the hidden representation
and after that I can add it to any
complex neural network that can do my
classification task or my regression
task whatever it is that I'm looking at
so this kind of layer by spray trading
allowed allowed people to drive more
complex compressed representations that
were useful in a variety of
problem-solving scenarios right so this
again has the wave a lot of hype in
neural networks or in deep networks as
they are called right so you can see a
very human similarities to the news
items that I had showed you from the
1959 thing so a simulated brain right
and can teach context to computers
machines that learn without humans
etcetera etcetera so so take what you
read in these papers with a pinch of
salt but again there is significant
renewed interest in
in deep networks in neural networks as
there was in the 50s so but the overall
feeling in the community is that deep
learning is come to stay because there
are a lot lot of nice properties about
this generation of neural networks as
compared to the earlier generations so
things are more stable and things are
reproducible and even though significant
computing power is needed so most of
these successful applications of deep
learning that you see I would have had
significant amounts of computational
power but then computational power is
also cheap right and you're able to
solve really complex problems using
neural networks right and so deep
learning records have altered state of
thought performance in a variety of
domains like image classification object
production language modeling machine
translation speech recognition image
description these are the problems that
were traditionally considered very hard
for machine learning algorithms and deep
learning seems to have significant
impact in these areas some of the images
are used here taken by taken from the
neural networks book where all Rajas so
I'd like to show you a few demos of
people Network Zin action now
so here is one algorithm from come from
the company called clarifying so which
given an image right tags it with
different kinds of textual labels so it
says that okay here's given this image
it's a that is coffee that is cross song
tonight that is a beverage in it and
this is probably breakfast and it is had
in the morning and overall hey this
looks like food so it's able to derive
all of these tags just by looking at
this image right these are all similar
images it has managed to retrieve images
that are similar to this you can see
that all of these are images of coffee
breakfast or of a continental breakfast
and all of these have a cup of coffee in
there and so even though the
the variety with this is able to cover
is truly astounding so likewise look at
this picture here it's able to figure
out that there's a suspension bridge
here and there is a river here with oh
the river is not explicitly visible and
that it is night because it's like era
and this looks like British in the
middle of the city right and likewise it
has found similar images which are all
suspension bridges in cities on top of
reverse and so likewise it is able to
work on a variety of outdoor scenes and
as well as indoor seats and it's able to
perform really well and this is this is
the state of the art in terms of image
understanding and labeling correctly and
likewise you can look at how well it
works in machine translation right so
both the the image tagging and the
machine translation tasks is use a much
more complex neural network architecture
than we have seen so far but then so
here is a simple example from here so I
typed in the sentence what a wonderful
idea and English and then it gives me
the the equivalent in French and not
only does it do that it tells me which
words correspond to which word in the
translated language which words in the
original English language correspond to
which word with eight hours later
language
- I don't know French so I'm not sure if
it's a for classless let's not do this
okay so let's stop with the previous
yeah so it's it and this thing actually
can learn translate can do translations
between multiple languages not just
English and French
it was trained on the data of dominated
Nations and European Parliament cert can
do translations between all the European
European languages so that brings us to
the end of this module on deep learning