Introduction to Machine Learning Video Lecture | What is Data Analytics? - Data & Analytics

welcome to this module on machine
learning so till now you have mostly
looked at data analysis it's all the
tools and techniques that we have looked
at have to do with analyzing data and
trying to understand the data better
right with possibly the exception of
regression from now on for the rest of
the course we will be looking at how you
can infer models about the process that
generated the data by looking at the
data alone this is essentially the idea
behind machine learning so what we call
even though it's a kind of a fancy name
and well even is you have visions of
robots and and terminators floating
around in your head but machine learning
is essentially trying to learn models
about the process that generated the
data from the data itself right so you
could look at examples where you learn
what how the rainfall pattern varies
over the season right so it could say
that rainfall pattern is very
intermittent over the season or you
could look at how the temperature of a
certain equipment is varying with time
of operation you can say that the
temperature is showing a linear growth
it is possible to do this kind of
learning from the data by a machine and
this kind of machine learning algorithms
becomes essential when we move away from
data analysis into either predictive
descriptive or prescriptive analytics
all right for example I can ask what
will be the temperature in 15 minutes of
a particular equipment now and if I know
that the pattern is that of linear
growth I should be able to tell you what
the temperature would be in 15 minutes
and likewise I can ask you what are the
areas that are likely to receive rain
right in the next season then if I know
what the variation has been seasonally
with the data then I should be able to
tell you what the what are the areas
that'll likely to the see rain the next
year right and so when they talk about
prescriptive analytics so the first two
questions I was asking you or more about
questions on the system and prescriptive
analytics we would be asking questions
what what I should do in response to the
patterns that you are discovering for
example I can find out patterns in stock
data and I could ask a question should I
invest in this stock or not right for
all of these kinds of analytics right so
we need to have AI technique tools and
techniques from machine learning so what
exactly is machine learning so I will
fall back on this old definition from
Tom Mitchell the Tom Mitchell said an
agent is said to learn from experience
with respect to some class of tasks and
the performance measure P if the
learners performance at tasks as
measured by P improves with experience
there's a lot of qualifications here in
the first case he didn't say a machine
he just said an agent so in Tom
Mitchell's opinion this applies to all
learning agents it could be humans
animals or machines right so an agent is
set to learn from experience so you have
to have experience in trying to solve
something and now you measure this
experience with respect to some class of
tasks I mean so it's not that you can
learn everything's you have to very we
have to be very specific about what is
it that you are trying to learn under
performance measure P right so you need
to know how well you are doing in that
particular task that you are learning
about right and then if you are set to
learn if your performance as measured by
P keeps improving with experience the
very very inclusive definition of
learning so you have to be very careful
when you use this because you could even
apply it to bizarre scenarios right for
example you could think of a slipper a
new slipper that becomes more
comfortable as you keep wearing it so
you cannot really say that the slipper
is learning to fit your feet with
experience right so you have to be
careful about how we apply this
definition but it's a fairly serviceable
definition as we will see as we go along
so the rest of the codes we'll be
looking at three different machine
learning paradigms it's the first one is
called supervised alone
right where you are expected to learn a
mapping from certain input variables to
output variables you already looked at
one example of such a supervised task
when you looked at regression right so
where the output variables was a
continuous valued variable right and
then the input was described by a set of
attributes and if the output is a
categorical output where it could be one
of many classes all right so I see is
the patient sick or is it see not sick
well will a customer default on the
payment or will they not default on the
pavement so these are like categorical
attributes right so both these examples
or the output could be either 0 or 1 you
could think of outputs with multiple
such levels in such cases the learning
problem is called a classification
problem so but what distinguishes
supervised learning from the other forms
of learning is that in the form of
experience that you will get so whenever
I give you an input a sample input I'll
always have an expected output for this
input right so I am going to I will give
you a set of samples which consists of
an input vector and an expected output
for that ok and that is what makes this
supervised learning in unsupervised
learning the goal here is to discover
patterns in the data that need not
necessarily be any output that I am
trying to produce right so there are
many different unsupervised learning
problems we will be particularly looking
at - in this course one of them is
called clustering where the idea is to
find cohesive grouping among the data
points that are given to you so in order
to find any patterns that are correct so
we will see how this works in a little
while so but you can readily think of
the the rainfall tasks that I was
talking about earlier so you can group
regions that somehow are similar in
their in their rainfall behavior right
and the and the second unsupervised
learning task
is known as association mining or
Association rule mining sometimes okay
and the goal here is primarily to find
the data points that occur together or
Co occur frequently so in Association
rule mining you essentially are trying
to figure out which data is associated
with which other data so how much which
Co occur frequently so in both of these
cases as you can see there is no real
output that you are expected to produce
except to find the patterns in the data
and the third class of machine learning
problems which are called reinforcement
learning essentially has to do with
learning how we would control a system
like we talk more about it towards the
end of the course but roughly you can
think of the following problem right so
how did you learn to cycle it is not
just discovering patterns right it's
knows there is nobody giving you an
input-output pairs that tell you how to
cycle right in if somebody actually does
give you an input/output pair will
probably not learn to cycle if your
cycle is tilting by 30 degrees to the
horizontal then you should push down
with your right foot with so many new
terms of pressure if somebody gives you
directions like that you will never go
into cycle right so you have to do some
kind of trial and error learning so
that's essentially what reinforcement
learning talks about so we do a little
bit of this towards the end of the
course
so the different machine learning tasks
that we are talking about all right so
classification regression which are
essentially supervised learning problems
right and remember I told you that you
really need a measure by which you are
going to decide whether the algorithm is
performing well or not right and the
measure in the case of classification
and regression is just going to be error
right in the case of classification it
will be the classification error that is
the how many mistakes you make in
predicting the the categorical outputs
and in the case of regression it's going
to be the prediction error which is how
far away you are from the actual value
that you need to predict right in the
case of unsupervised learning problems
tricky as to what the measures should be
and there are many look at clustering
and Association rules we'll talk about
many such measures in detail but roughly
in clustering one of the measures is how
tight your clusters are or how scattered
they are I'm talking very roughly here
but we will formalize this as we go
along and in the case of associations
it's more on how confident you are that
these two items are associated right and
what is the the fraction of the
population in which these associations
appear so that is what we mean by
support and confidence and again as I
said this just to give you an idea that
for every task you are going to have an
Associated measure right and we will
elaborate on the actual measures as we
go along
so having said this there are many
challenges that we need to address right
so one the first challenge in any
machine learning problem is to figuring
out how good is your model if somebody
gives you a machine learning algorithm
and say that okay here is a model that's
been learnt by the algorithm how do you
decide how good that model is right so
you could use the measures that I showed
you on the previous slide but that could
be other ways of deciding how good the
model is ok so I will of RIT on this as
we go along
for example you can build a very very
very very detailed model that gives you
almost zero error on all the data that
is given to you at the beginning but
then this model might essentially be
useless when it starts looking at unseen
data and whenever actually wanted to
make predictions on data that I haven't
seen before
this model might not perform well at all
right so how do I look at that kind of a
trade-off so there are different ways of
measuring how good a model is right and
then given the data and given the class
of models that you are going to look at
how do I choose the right model right so
that's that's the second challenge and
many different machine learning
algorithms are all about answering this
the choice of the model question right
but then it's not all about modeling so
you have to be
very very cognizant of the data that you
are operating with right so the first
question that plagues all of us is do I
have enough data right that might
surprise some of you who have heard of
terms like big data and having excess of
data data deluge and so on so forth but
then getting us in in a supervised
learning problem getting label data this
incredibly hard right so you have to
have an expert that is looking at all
the data and then labeling them for you
so getting such label data is incredibly
hard and so do have sufficient label
data or do it can I make use of
unlabeled data in in a Trever way so
these are kinds of questions that we'll
have to think about right it's the data
of sufficient quality there could be
errors in the data for example age could
be recorded as 225 or there could be
noise in very low resolution images that
you are feeding your algorithm and sure
so the algorithm is not able to make out
what is that in the image right or it
could be that some values are missing or
have not been recorded in your data and
then your algorithm has to deal with
that so it's a very important question
whether the data is of sufficient
quality and in many almost in every
large-scale real-life machine learning
installation a lot of effort has to go
into cleaning the data so making sure
the data is of sufficient quality to
feed into your system right and so how
confident can you be of the results at
the end of it it's both a factor of the
data quality the data volume and as well
as the machine learning exact machine
learning algorithm that you end up using
right so all of these factors together
influence how confident you can be of
the results right and at the end of it
there is one very important question
right which typically you know is not
address that carefully is am i
describing the data correctly
so for example there are two classes of
questions I could ask here so the first
thing is like do I have enough
information about what I am trying to
classify right or age and income enough
to describe all my customers or should I
look at gender also how would I answer
such a question a lot of it actually
comes from the data analysis that we
have been doing so far right our agent
income alone sufficient to explain all
the variance that I see in the data or
should I include another variable in
order to make my model more accurate so
such questions will tie back into all of
the analysis that we have seen so far
and the next question which is more of
an it's partly an engineering issue and
partly a theoretical issue is how I
should represent my variables you know
how do I represent age at age can be a
number well age can sometimes be merely
a number but sometimes you can classify
a age into different levels as an
middle-aged and old right so what kind
of an encoding would you choose for
representing age and again it goes back
to the data analysis if we have talked
about and so these are things that you
have to pay a lot of attention to and
more often than not when you are looking
at simple exercises that you can do on
the web you end up looking at the data
that is already where these questions
have already been answered and they are
given to you and you are able to do it
very easily but when you go out and
you're trying to solve your first real
problem using machine learning you will
find that these are very important
questions that you have to answer so in
the next module we will look at more
detail at the different learning
paradigms that we talked over today