Confidence Intervals Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to our next lecture in
inferential statistics today we are
going to be talking about confidence
intervals so statistical inference you
know even in terms of classification
mainly is discussed in terms of
hypothesis testing and estimation so
these are the two broad categories in
statistical inference and hypothesis
testing is something that we've
discussed in good detail in this class
we've talked about single and two sample
tests various tests today we're going to
be primarily talking about estimation
and you might notice that almost any
text that talks about statistical
inference talks about these two topics
and some of them might be introducing
estimation or confidence intervals
before hypothesis testing but that
doesn't really matter essentially these
are two sides of the same coin if you
may and so today we are going to be
talking about estimation now the idea
behind estimation is that estimation can
be off in terms of point or interval
what we mean by that is it's the same
core concept as what we introduced with
inferential statistics during hypothesis
testing which is that we are interested
in some population parameter okay what
we mean by that is that there is this
concept so the examples that we've used
in this class at things like amount of
phosphate in our blood the height of 10
standard students in public schools in
India in all these cases you define some
population and you're interested in some
parameter more often than not we've
discussed the parameter being the mean
right so what is the average amount of
phosphated blood what is the average
height but it does not that's just one
of the parameters with that that's the
most common one but it can be other
parameters so stepping back we're
interested in some population parameter
you but you don't know this parameter
it's like it's some truth that you don't
already know if you did know that there
would be no need for any of this or any
of the statistics but what you do have
is a sample
so you have five data points ten data
points 20 data points 38 upon some
sample from this population and what
you're most interested is you're most
interested about this population
parameter so in hypothesis testing you
would hypothesize that this population
parameter is equal to 4.8 or 2.3 or it's
less than 4.2 and then you would go
about and look at this sample and see if
that's true or not with estimation
you're not having any hypothesis you're
not having any hypotheses in mind in
that sense what you're trying to do is
you're trying to take the sample and
with point estimation you're trying to
come up with a single point estimate of
the population parameter and that might
seem fairly straightforward right so for
instance let's say you were interested
in the population parameter which is the
average average amount of phosphate in
blood and you took a datas you took some
sample and that sample was about 20 data
points a simple point estimate of the
population mean could be the sample mean
so you take the sample of 20 data points
take their average and that's your best
that could be one of your best point
estimates of the population mean so
point estimate just means you're making
as good a guess on the population
parameter based on the data that you
have but today we are going to be
talking and and more interestingly this
is what gets this is this is what people
are more interested in it is interval
estimation in the form of an interval so
this goes back to again the core concept
with hypothesis testing which is fine
you don't know the population mean you
have a sample mean and you acknowledge
that if you go take another sample of
another 20 points you might not get the
exact same value and both these values
the first time around and the second
time around might not be exactly equal
to the population mean
okay if it's not exactly equal to the
population mean then can I come up with
some range around my point estimate
let's I have a point estimate which is
actually my sample mean my sample mean
is my best bet let's say at my
population mean but I acknowledge that I
might not have exactly hit target right
the population mean might be a little
higher or a little lower than my sample
mean in which case I ask the question
can I come up with a range around this
sample mean by which means it's
essentially like I'm giving myself a
margin of error by which I'm fairly
certain that I've covered the population
mean so that's the goal that we are
going to embark upon and in many ways
it's the same map because we've
introduced hypothesis testing it becomes
a little easier so that we can reason by
the same logic in math that we've
already discussed so let's do that so
the core idea is that let's take an
example that we've looked at many times
which is that we might be hypothesizing
that the amount of phosphate in blood is
equal to exactly 4.8 now we discussed
that in confidence intervals you don't
have this number you don't come up with
the hypothesis you're only interested in
coming up with some bounds around your
point estimate so what you essentially
do one way of thinking about confidence
intervals given that we have already
introduced hypothesis tests is well for
different values of MU naught so let's
say you have some data set and you
calculate some x-bar right
essentially 4.8 gets plugged in out here
to calculate your Z statistic if you're
doing a Z test you've given a sigma if
you're doing a t-test you take an S but
in either case that gets plugged in n is
again the number of data points so you
get some Z value some Z value that's out
here and based off of that Z value you
calculate some probability you calculate
a p-value now the core idea with
hypothesis test
was that if this p-value is really small
right you reject the null hypothesis
okay so the question with confidence
intervals one way of thinking about it
is given that I don't have some
hypothesis I ask myself the question
within water range can my muse be such
that I will calculate a Z such that I
will get a p-value which I will not
reject okay
I'm just going to repeat that given that
you don't have a Mew not you can think
of a confidence interval as what is the
range of values that you could
potentially be such that given that for
a given data set you get some X bar some
Sigma some root n such that you will
calculate a value Z such that you will
get a p-value which you will not reject
so if you won't reject that that means
you need to have some bounds I suppose
you start off by saying well I'm going
to reject any p-value less than 0.05 so
that's something you started off with
now given that you started off with that
then is there some rate for a given data
set is there some range of Mews okay
such that you would not be rejecting
this hypothesis test and essentially to
compute that all you do is just
rearrange the terms out here so you keep
the Mew not on the MU of the Mew not I'm
using those two terms here
interchangeably but you keep that on one
side and you essentially move the terms
to the other side to get this formula
for confidence interval so typically if
you know the formula for the hypothesis
test for the test statistic you can just
essentially rearrange it but the core
idea is that this is your point estimate
which is your X bar so for your best
estimate of MU is your X bar but you
create a margin of error or you create a
range around it as plus or minus the Z
associated with this alpha and alpha
here is that point zero five that you
said essentially you said within a
certain brain
right so within that range and Sigma n
square root of n are the same okay
another way a more formal definition
associated with confidence intervals is
that if we were to repeatedly take
identical samples of the same size and
build similar confidence interval bounds
for each sample then you're building a
bound such that 95% of such confidence
interval bounds will cover the true mean
or in other words we are 95% confident /
certain that the true mean mu is within
our confidence so for single sample
tests the process of creating confidence
interval is fairly straightforward I
explained to you how the in the Z test
it essentially just becomes a
rearrangement of terms and this Sigma
buys this concept of Sigma by square
root of n essentially goes towards the Z
and then the x-bar goes here such that
it's a plus or minus and and and that's
how you get the formula for that
interval okay the same core idea for the
T distribution the formula is no
different except that the T distribution
gets also defined by the number of
degrees of freedom so not just the Alpha
but the number of degrees of freedom and
in all these cases mind you because I'm
putting a plus or minus this is the
equivalent of the two-tailed Z test or
the two-tailed t-test you could also
create a one-sided bound if you were
interested and that would be again the
formulaic equivalent of you know having
a one sample test again it'd be the same
Sigma divided by square root of the
formula itself won't be different but
the way this alpha gets used up will be
different okay again again it goes back
to the core concept of how you would
shade that region under the probability
distribution okay with the chi-square
distribution that a rearrangement is not
obvious to some extent it is but the
plus or minus is not because you don't
have a plus or minus term it's
essentially like this Sigma naught goes
out here this Chi square distribution
comes down here but the way we
differentiate between the lower bound
and the upper bound is by changing the
alpha in the bottom of the chi-square
distribution so it's the same
rearrangement it's the same core concept
of taking the hypothesis test and
rearranging that is in this case it
would be to put the Sigma naught out
here and bring the chi-square down here
and you would have the same formula that
you see here but you get an upper bound
in a lower bound by looking at the 1
minus alpha by 2 and alpha by 2 and by
the way this notion of alpha by 2
depends on how you define it now if you
say I want a 95% confidence bound that
means you're left with 5% and if it's a
two-tailed test that 5% gets divided
into 2.5% times 2 that's how you get the
alpha by 2 so it really depends if
someone starts by stating alpha and and
you know it's a two-tailed test then
technically the correct way to do this
would not be to just have an alpha out
here but it would be to have an alpha by
2 so that you being you're being
technically correct and the same thing
goes for here as well okay again the
same core idea with respect to the Z
test if this this alpha is more a
generic term that I've used out here so
if somebody comes and says I need a
two-tailed test so there is a plus or
minus and the alpha gets divided by 2
but if it's a one-sided test that alpha
can stay as alpha so it depends on how
it gets formally defined typically if
it's a two-tailed test you'll represent
it as alpha by 2 but it's again if you
look at it it's the same rearrangement
from your test statistic to create the
confidence interval so the same idea
goes towards two-tailed tests just to
give you an idea of how it works for a
two-tailed test I've given a single
sample for the Z test for the T tests
and the proportion Z test we've
consciously left it out that might be a
part of your assignments that you could
work on but the idea is the same we we
are interested in this term we're
interested in the term x1 bar
- x2 bar okay and so we want to create a
confidence mount around X 1 bar minus X
2 bar if you remember the simple way of
thinking about X 1 bar minus X 2 bar was
to say to test the null hypothesis that
X 1 bar is equal to X 2 bar but that's
logically the same as asking the
question is what is the confidence
bounds around X 1 bar minus X 2 bar and
seeing if that essentially covers 0 or
not of course this D naught is an extra
term suppose you were interested in s in
a hypothesis that looked more like this
if this was your null hypothesis then
you could you would have this D naught
being something nonzero otherwise D
naught is just equal to 0 okay but this
is the idea right so you are interested
in some bounds around the term X 1 bar
minus X 2 bar and again you would do the
same logical rearrangement this goes
here and the bounds go around X 1 bar
minus X 2 bar and you're essentially
creating bounds around this value okay
sex there's X 1 bar minus X 2 bar and
what what is your range around that okay
great again similar to the chi-square
test the F test is not so
straightforward
so I'm including the formula associated
with that out here you do the same thing
that you did with the chi-square test
which is you have the same s 1 square by
s 2 square that's your core upper bound
and lower bound but you divide by this f
this F kind of comes down and you divide
by that but the lower bound is 1 minus
alpha by 2 and the upper bound is alpha
by 2 of course also remember that when
this is a two-tailed test the Alpha that
you see here becomes 1/2 divided by 2 if
it's a one tailed test meaning you just
had a plus or a minus you can keep that
as alpha okay I hope that clarified this
concept of confidence intervals thank
you