Short Introduction to Regression Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to a class today on
regression today's class is going to be
a very short introductory level
treatment of the topic regression
analysis and the idea here was to kind
of position this towards the end of your
modules and descriptive and inferential
statistics so that you're at a point
where you know enough to appreciate how
regression uses these concepts but we
will also be revisiting regression down
the road where we will actually talk
about the mechanics about how you
implement it and that is something we
will do right after you get an
introduction to machine learning and
more specifically to supervised learning
okay so jumping into what a regression
is essentially a regression analysis and
I say regression analysis here because
the word regression itself in different
fields is used differently and in fact
even in statistics there's a concept
called regression towards the mean which
has a lot to do with regression analysis
but it's also quite different from how
we understand how we use regression
analysis today so again the idea is to
you know use the word regression
analysis so regression analysis
essentially is the study of
relationships between variables most
specifically it looks at understanding
the relationship between one or more
input variables and their relationship
to an output variable and the exact
normal creature is sometimes different
we use the words I'm using the words
input and output
you might also come across the terms
response to describe the output or
dependent variable to describe the
output and for the input variable you
might come across the terms explanatory
variable or independent variable for the
input okay so the idea is to study this
relationship between some set of inputs
or one input variable to this output
variable that you have in mind
and so right from the get-go at minimum
you are looking at two variables
right and the whole idea is have you
come across this kind of studying
relationships between variables so
what'swhat's this regression doing newly
right and the idea is yes you you have
you've studied you when we started
looking for instance at two sample tests
to sample t-tests for instance right
there you for the first time you
encountered scenarios where there were
two variables involved to jog your
memory an example of the two sample
t-test that we looked at was for
instance blood pressure and the idea of
whether the blood the average blood
pressure was lower when given calcium
supplements versus when given placebos
placebos are just sugar pills
that kind of look like the original
medicine so the idea was there were two
variables involved there was your notion
of an output variable or response
variable which was your blood pressure
it's a quantitative variable and you had
another variable and this other variable
was you can call it calcium and this
other variable calcium it's a variable
because it can take on two states you
can take on the state yes meaning you
were given the actual calcium supplement
or it can take on the state no which I'm
using as a proxy to mean that you were
given instead the placebo so you hadn't
you you now have two variables right and
you're and you can think of your this
this variable calcium is often called
the treatment but you can think of it as
the input variable which takes on two
possible states and you're looking at
essentially the to see if there is any
difference in blood pressure for people
who were given calcium versus people who
were given placebo right there you're
looking at a relationship between two
variables if you're looking at the
relationship between the variable
calcium and the variable blood pressure
now another example that we saw in the
two sample case was boys and girls in
10th standard in public schools to see
if the average height of boys was equal
to the average height
of course again here your output
variable is the height and your input
variable is gender whether you're a boy
or a girl you're looking to see if there
is a relationship between gender and
height yes the answer is obviously there
is but maybe in 10th standard is one
higher than the other in public schools
okay so you went even beyond that right
for instance with something like an
ANOVA analysis of variance that we
discussed in the last class you go
beyond just having two categories of
your input variable you can have
multiple categories we saw the example
where we looked at States as the input
variable you looked at height of 10
standard students and to make it simple
let's just say boys but you looked at
the heights and Tamil Nadu Karnataka
Maharashtra there you're looking at the
relationship between state as your input
variable and height as the output
variable is there any relationship is
there any difference between boys of
different states and their heights and
finally we also looked at something like
the test of Independence where here you
were looking at two categorical
variables and you were looking to
looking to see if there was a
relationship between them but so so the
reason for going into all of this and
kind of revising some of the topics is
to say that you have studied
relationship between variables but for
the first time you're going to go beyond
dealing only with categorical variables
regression analysis is especially
powerful because it creates a
relationship between two variables but
this relationship can both these
variables can be continuous and
quantitative and therefore the scope of
application for regression really really
becomes much greater so to give you an
extension for instance of some of the
previous examples we've looked at from
the top of my head one thing I can think
of is we were looking at the
relationship between blood pressure and
you were given calcium pills or placebo
pills right so there were just two
possible states now what would happen if
I gave if I had 20 different types of
pills
which started on one end from the
placebo which had zero calcium and went
all the way to the other end which has a
hundred percent calcium and in between
20 different types of pills I'm not
saying there are 20 pills there are 20
different types of pills the different
types of pills had varying degrees of
varying percentages of calcium right so
now you have at least 20 states and
those number of states can if especially
if you are randomly sampling that can be
infinite number of states but you just
have a range so it's a continuous
quantitative variable between 0 to 100
percent of calcium and your output
variable still remains your blood
pressure so you can actually create a
relationship between these two variables
it can be a mathematical relationship it
can be a graphical relationship but it
goes beyond just two states or three
states or n states it has the potential
to go to infinite number of states of
the input variable of variables and so
it can describe you know richer things
the idea again behind regression
analysis is to not just unlike most of
what we've studied with inferential
statistics it's the goal is not just to
establish that there is a relationship
but it is to quantify that relationship
as well so a lot of emphasis in a
regression is given to the model itself
that mathematical equation that
describes this relationship between the
input and the output variable and that
can be either linear or non-linear
relationship that the relationship
between amount the calcium that is given
and the drop in blood pressure if there
is one can be a linear or non-linear
relationship and you can extend this to
all the other examples okay now as with
most supervised learning techniques and
that's that's something that we briefly
discussed in the course overview but
it's also something that you will share
a lot more often
lectures the goal of a regression can be
twofold
it can be towards prediction the idea
that you now create this linear
relationship and soon I can now use it
to predict what the drop in blood
pressure would be for a given amount of
calcium supplement and that particular
amount of calcium supplement might not
have even been something that was given
in the original data set from which I
built the line but the fact that I used
this data to create this relationship in
the form of a mathematical equation
which represents a line which represents
the relationship between calcium
supplements the amount of calcium given
to the drop in blood pressure I loves me
this the fact that I've created this
allows me to use this tomorrow to
predict how much the drop and blood
pressure would be if someone came and
told me oh I'm thinking of giving this
amount of calcium supplement it allows
me to make predictions the other thing
that it also does is given that you are
now able to create a mathematical
formulation it gives you an
understanding of the world it gives you
an understanding of how for how much
extra calcium that I give how much of a
blood pressure drop will I see so it's
an understanding of how these two
variables are actually related and and
that's what we essentially call is one
prediction which is one goal which is to
kind of predict and the other is
interpretation which is to kind of
interpret the world that we live in to
through through the lens of this linear
relationship okay so let's just talk
about just to give you some feel for
where a regression analysis is useful in
what kinds of contexts here are some
examples that we've listed for instance
so how do wages how do salaries of
employees get affected with variables
such as experience education promotions
etcetera so the idea here is that salary
is the dependent variable it is the
output
variable it is the response variable and
variables such as experience which might
be measured a number of years education
again measured in terms of number of
years after high school or something and
promotions again a numerical value how
do how do these variables affect the
output variable another example is how
does the current price of a stock or a
share depend on its past values and
perhaps also on values of the market
indices or other stocks alright a subtle
difference here that you might notice
and we will circle back on this is in
the case of the stock for instance
you'll notice that the output variable
is the price of the stock but the input
variable is also the price of the same
stock on previous days right so how do
how does the current price depend on its
past value so here the past values
become the input variable and the
current value becomes the output
variable so we call that time series and
we will briefly revisit that concept as
well but regressions are also used in
that context okay so the next example is
how the sales revenue get affected as a
function of advertising expenses and
competitors advertisements so the more
money you throw into advertising you
might have a belief that the sales goes
up but what is that exact relationship
can I look at past data on different
advertisements that have different
campaigns advertising campaigns that
have made for different products and
what my competitors did and how that
affected my sales driving and create
this relationship now something else
from the mechanical engineering house
would be relationship between speed of a
car and fuel efficiency within a certain
range when you're driving on a certain
gear what is the relationship between
the speed at which you drive on the fuel
efficiency and you could take a sample
of 10 cars make them drive a different
speed or you know take the same car make
it drive at a certain speed
and get data and try to fit a line in
this data to understand the relationship
and finally another example could be how
does the price of a house get affected
by the number of bedrooms the square
footage of the house you know its
distance from the center of the city and
so on and so forth
again some of these input variables need
not neatly fit into being continuous
quantitative variables but the idea is
that a regression is not constrained by
that a typical application of regression
would use continuous variables but
though the whole selling point
associated with the regression is that
it's a technique that's also capable of
dealing with continuous qualitative
variables whereas a typical ANOVA or a
t-test or any of the tests that we've
seen so far are built around the idea of
using a categorical input variable and
that's what they're meant to do okay
okay so let's talk about some more
concepts associated with the regression
just for reference I've shown you a
typical graph graphical representation
of the regression where the x-axis is
your explanatory variable your y-axis is
your response or output variable and
each square is a data point by square I
mean each of these small points is a
data point and the idea is loosely the
idea is to kind of fit a line through
this data point and that's what we would
call linear regression if you're trying
to fit a line through the data point we
would call it linear regression the word
actually linear regression could also be
used when you're doing a linear
combination of variables but some of
these variables themselves are represent
a nonlinear transformation I think this
the simple way to put this is whether
you're looking to have a linear
relationship between the input and
output variables or a nonlinear
relationship you could still use the
core concept of regression but you
should be a little careful in terms of
what gets called linear and what gets
called nonlinear because often the camp
of linear regression could
include variables the input variables
themselves which are nonlinear
transformations and so it would still be
called a linear regression but it won't
exactly look like the graph that you're
looking at okay you also have the
categorization of simple versus multiple
regression the idea but behind simple
regression just means that there is one
input variable there is in multiple
regression you have more than one input
variables so obviously in the graph that
you see in this slide you're looking at
a simple regression because there's one
explanatory variable which is the x axis
if you have more than one that becomes
hard to represent on a two-dimensional
slide you would have to use a 3d model
for two variables and then after that it
becomes much harder but the idea is that
if you have one input variable it's a
simple regression if you have multiple
input variables more than one input
variable it's multiple regression we
then come to the next concept that we
briefly discussed which is
cross-sectional versus time series the
idea behind cross-sectional is that it
is not a function of time your data is
collected across the board and the
temporal the time component of it
basically means that it's not either a
function of time or it's not a function
of its previous values so you can put
all the variables in the basket and
think that they were all created at the
same time or at least that you don't
care whether they were created at
different points of time you're still
going to treat them as just different
data points and look at a relationship
and that would be cross-sectional
whereas time series is the idea that
previous values in time affect
subsequent values so your modeling
techniques themselves become a little
different and a classic example of the
time series was the stock example that
we spoke in the previous slide where
previous price of the share influences
tomorrow's price whereas the equivalent
of a cross-sectional analysis of that
would be to completely ignore the stock
price of this to completely ignore the
previous days or previous time period
stock price and just say can I predict
what this talk is this
value is going to be based on other
stocks or other indices so can I use the
Nifty index can I use another stock to
kind of say based on these stocks this
is what the stock should be this is what
the output the the response variable
stocks value should be okay next we just
come to the idea that this is notion of
a response variable or it's called
dependent variable and your input
variable is called the explanatory
variable or independent this is
terminology that you keep hearing and so
it's kind of important okay so the next
important point is that we learnt about
something called scatter plots during
the descriptive statistics
phase and a regression analysis in many
ways essentially captures mathematically
what the scatter plot tries to do
graphically so scatter plot tries to
graphically show you that relationship
between two quantitative variables there
is a regression analysis tries to go
beyond that and try to actually fit a
line to this fit a line of fit fit of
fit a model to be more general fit a
functional form to this data and
therefore that that's the improvement on
the scatter plots that the regression
analysis does but often even before
getting into a regression analysis a
scatter plot could be a very useful
graphical window into what you should
expect to even see if you should try to
fit a linear equation or a non-linear
equation because just visually you might
be able to say you know this doesn't
make sense to fit a linear equation
because that's not the core relationship
another big advantage of scatter plots
is that if there are these outliers and
by outliers we mean essentially there
are data points which look like they are
essentially just errors or something so
you let's say you had this data point
that was in somewhere completely
unrelated and that could come from an
error and that could completely ruin
your regression analysis
but if you see it in a scatterplot you
might choose to say I want to ignore
this data point that's the idea behind
scatter plots and outliers the next
concept is one of unequal variance we're
going to talk about this a little bit
but the idea could be that they could be
a very strong linear relationship or a
nonlinear relationship but the
variability at different points of X or
your input could be different so you
could have something that looked like
this in terms of the data okay so this
is the line that you're fitting so
there's this little so this central line
is still a line that you're fitting in
it's a linear line but the data points
in the lower end so in in this side of X
the data points are closer to the line
whereas once you go to this side of X
once you go to the right side of X the
data points of Farm was spread out and
that influences how the line gets
created and there are certain techniques
that are available to kind of counter
this problem it's a problem because
again visually you might just see this
speakerphone kind of shape and say still
I'm just going to fit a line in the
center but this higher variability on
one side lower variability kind of mix
does not always result in the correct
line being fit and you need some kind of
a transformation to understand the
relationship better so the last concept
associated with this slide and what I
wanted to talk to you about is with
correlations so correlations you can
think of as a more advanced form of
summary or descriptive statistics we
didn't speak about it in detail or in
the descriptive statistics part but the
idea is that when you have two variables
just like a scatter plot can be used to
graphically describe these two variables
a correlation is a single number that
describes that linear relationship and
it's essentially a quantitative
indicator of that linear relationship
what you will discover is that in a
regression
you will also be coming up with certain
numbers that quantify this linear
relationship and if you have not already
heard of this you will come across
something called R square and that has a
direct a quantitative mapping to
correlation so it's not a new concept
essentially R square is nothing but
correlation square but but you will come
across it under the term R square
whereas you might have heard of the word
correlation more colloquially used or
used more in the concept context of
descriptive statistics okay now let's
let's just briefly talk about the exact
role of descriptive and inferential
statistics right so if we start first
with descriptive statistics the whole
idea there was either graphically or to
quantitatively summarize the data that
you see and one way of summarizing that
one essential summarizing that you do of
the data in a regression is to capture a
linear relationship which is there in
the form of the actual data into a line
and so you essentially summarize this
relationship in the form of an equation
and many of you might know that if there
is for a simple line in two dimensions
you have this formula y is equal to MX
plus C you might have come across that
in high school in some form or the other
where C essentially represents the
intercept of the line on the y axis and
M represents the slope so you can think
of M s that angle M represents that
slope and you can in in regression often
we use the terms beta and here I'm just
using B naught and B 1 because I'm
representing the sample based off of the
sample that we have we have some
estimate of B naught and B 1 but the
idea is that this this is some form of
summarizing the data because you are
taking the data and your summer
raising it to this single equation so
what is the core concept between behind
how you do it and the idea is that it's
some form of optimization it's a form of
optimization because you have a set of
y's and x's so you are given a series of
y's and x's and that's your data and for
each Y and X you essentially plot it on
this graph right you then go ahead and
you take a line and this is that line
that we are going to play with and you
choose some criteria to fit the line
through the data points so you might say
that
look my goal is to fit the line through
the data points and you know I don't
want to draw a line out here that has
nothing to do with this data right so
what kind of a line has something to do
with the data you might say well I want
the line to split the number of data
points equally above and below it
all right so there's a little bit of
overlap there but you get the point
which is that you might want you might
want that line to split the data point
another idea could be to say I want the
line to go through the two extreme
points so if there are two extreme
points I want the two extreme points to
be connected by a line and you can have
various different other criteria another
very common one could be to say I more
advanced one could be to say I want to
minimize the distance between each point
to the line and that incidentally
happens to be what we call as the
ordinary least squares but we'll get
into that later but that could be one
criteria another criteria could very
well be that I want to minimize the
perpendicular distance of each point to
the line so you could have various
criteria and you know you can choose
some criteria and you can say I want to
achieve that criteria and that's how I
want to fit a line and so at the route
of doing that is the series of
techniques and optimization where you
say I want to do some form of
optimization so you might ask the
question okay what are you really
optimizing and the answer is you're
really optimizing
this objective of minimizing some metric
and let's stick to the metric of
minimizing this distance thus the sum of
this distance so how are you optimizing
that body what can you change what you
can change is this line alright and what
I'm going to do is I'm going to change
the two parameters associated with this
line and the two parameters are M and C
okay so C is where this line intersects
on the y axis so I'm going to try and
move this line above and below keeping
that slope the same and try to see where
should this line be such that I minimize
the sum of these distances right
those those red lines that you see my
goal is to minimize the sum of these
distances and that's an optimization
procedure that's my objective function
my objective is to minimize the sum of
these distances and I'm going to do that
by changing two variables I'm going to
change the variable C okay by moving
this line above and below and I'm going
to change the variable m by rotating
this line right so by rotating this line
this way and this way so simultaneously
I'm trying to change two variables and
thereby find that line by changing my m
and C I find that line which minimizes
this deviation the deviation of each
data point to that line right and there
are different ways of doing that I'm
just giving you one one objective that I
might choose to optimize but at the root
of it you need to realize that
ultimately process of a regression
fitting a regression line is a process
of optimizing and you might have
different criteria ordinary
least-squares which is one type of
regression uses the criteria of taking
this distance of each point to the line
squaring it and trying to minimize the
sum of those squares because each point
will have some distance to the line and
then you can take that distance and
square it and then you can take go to
the next point and do the same and then
you can sum all those squares and least
squares tries to minimize that objective
function but that need not be the only
objective function there can be many
other objective functions but at the
root of it
the idea behind putting a line through
many data points is to say I am
interested in maximizing or minimizing
some goal associated with the process of
fitting the line and I'm going to do
that by changing the two variables that
I can which is M and C or in other words
your B naught and B 1 okay so that's
that's essentially the summary
statistics the the process of describing
the data through a line and that's how
that line gets created now where is the
concept of inference right the concept
of inference again comes from this core
idea that ultimately these data points
are samples they don't represent the
population so any B naught and B 1 that
you calculate are coming from the sample
so they're essentially the sample
statistics so but just like x-bar in
some sense is the sample mean which is a
point estimate of MU which is a
population mean these does B naught and
B 1 essentially represent a beta naught
and a beta 1 so you can you you
essentially have the population
parameters beta naught and beta 1 okay
so these are the population parameters
let's just wait a naught and beta 1 now
the idea is that with statistical
inference what you're doing is you're
trying to see if either of these terms
beta naught or beta 1 are actually equal
to 0 because if they are right then they
don't have any business in being a part
of this equation right this equation
that you have out here
now the this equation that you have out
here right that's an equation that you
started with before you even fit
anything now you're just in the process
of finding out your B naught and B 1
through some optimization procedure now
imagine a situation where X has nothing
to do with Y right X has nothing to do
with Y now you calculate a sample of X's
in a sample of Y's
if you had an infinite number of samples
you might arrive at the conclusion that
B 1 is truly equal to 0 and therefore
this term itself becomes 0 and should
not be there in this equation but you
still only have a sample and the sample
is five data points ten data points 20
data points so it's perfectly possible
that even though true beta 1 is equal to
0 that's the null hypothesis even if
that is true you it's possible that you
just take a sample of Y comma X's and
you get some beta b1 which is not equal
to 0 so b1 winds up not being equal to 0
even though beta 1 is equal to 0 right
so you might be erroneously thinking
that X has this relationship to Y
whereas in reality beta 1 is 0 okay so
the whole idea is to do a statistical
test to see to test the hypothesis that
beta 1 is 0 so the null hypothesis would
be that beta naught is 0 and you can
think of it as also beta 1 is 0 and it
turns out that under certain assumptions
of normality and even when those
assumptions are violated to some extent
central limit theorem kicks in it turns
out that the distribution of the betas
of these coefficients winds up being T
distributed so like a like a like a T
distribution and so you can use
essentially a t-test to test the
hypothesis that each of these
coefficients are actually equal to 0 or
not and the idea is that if you wind up
you need to reject the null hypothesis
and reject the hypothesis that beta 1 is
zero and only then can you actually have
that term in the model and this becomes
really useful when you have you know
multiple input variables so you might
have something that says beta naught
plus beta 1 X 1 plus beta 2 X 2 and so
on and it can go on right you can have
many variables so which of these terms
actually get to stay in the model and
which don't is not just a function of
this magnitude of beta 1 and beta 2
because that's the magnitude becomes
really a function of it becomes also a
function of the magnitudes of X 1 X 2 in
the units and so on but what really
determines whether these things stay in
the equation or not is the inferential
statistic test that you do for each of
their population parameters which is for
beta 1 for beta 2 and so on and insofar
as you can reject the null hypothesis
that these are equal to 0 then you can
leave them in the model but if you
cannot if you if you cannot reject the
null hypothesis then these don't these
terms don't have any business being in
the model ok ok so that's the idea
behind doing statistical inference on
the individual parameters you will also
notice that in a typical regression
analysis or a regression analysis output
independent of the software that you use
there is a you will find an inference
for the overall model so in aside from
the inference that you see for beta 1
beta 2 separately for a given model you
will have an inference and that
inference again uses the concept of one
ANOVA and the way it does that is by
looking at if you remember the ANOVA in
the past you have the concept of mean
squares between and mean squares error
where the mean square between was trying
to quantify the effect of each
treatment or each state of the input
variable and mean square error was
trying to quantify the inherent noise
that was there even within each state
here however you don't have a finite
number of mean square between because
they could be infinite states of the
input variable X right so you have a
single term that tries to capture how
much of the variability is being
captured by this linear or nonlinear
model that you built how much of
variation in X okay is being captured by
this equation how much of the variation
in I'm sorry how much of the variation
and Y is being captured by this model
versus how much of the variation in the
data points is not captured by the model
so one becomes the mean square of the
model essentially and the other becomes
the mean square error which is just the
gender which is some notion of general
noise that is there in the system and
essentially the anova out here again
will be we will be using the f-test the
same way as we use the F test when we
were describing the ANOVA in the
previous class for categorical variables
and thereby rejecting that null
hypothesis would mean that this model
explains variability in Y whereas if you
fail to reject it it the idea is that
you cannot say that this model
essentially you you cannot say that this
model is any different than if you were
to just randomly choose y's completely
unaware of what the model was so no
variability in Y is being explained by
different points in X so and essentially
when you think of it that that's nothing
but the concept of a straight line it's
it's essentially a concept of
something where you're not describing
the variability in Y at different points
of X so at various points of X you're
looking at the same same same position
same position Y okay I hope that made
that gave you an introduction to the
concept of regression and like I
mentioned after we introduced the idea
of machine learning and supervised
learning we will be revisiting
regression to look at a more in-depth
treatment of how you actually do it and
give you an understanding for how the
core derivation czar made thank you