Introduction to Experimentation and Active Learning(contd) Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to our second lecture
in the series where we talk about
experimentation and active learning in
the first lecture we briefly spoke about
be motivated the idea of using active
learning experimentation or
reinforcement learning in a broader data
analytics setting where we said these
are some approaches that are fairly
relevant when you don't have data when
data does not exist or the data exists
and it's not enough or you have only
partial data in the first lecture we
also went into design of experiments and
spoke about how perhaps an approach
where you sequentially change one
variable at a time
need not be the best way to conduct an
experiment and we also spoke about
something called orthogonal arrays and
specifically other we spoke about a form
of experimentation called the full
factorial experiment full factorial
design where it's essentially a complete
enumeration of a discrete input space
where even if you have continuous input
variables you break them into one two or
maybe sometimes three discrete points
and you essentially do a complete
enumeration which means that every
variable is set to every possible value
it can be set to with relation to every
other variable being set to all their
possibilities so all the possible points
in the input space are essentially
looked at and that was essentially
looking at the design meaning what
points in the input space do you choose
to experiment in today's lecture we'll
briefly take just the full factorial
which is a very basic design and talk
about some approaches that are used in
analyzing such an approach as such a
design so jumping into the subject in in
this slide what we have is a full
factorial design associated with three
input variables a B and C
and the the idea here is that what we
have is a full factorial design because
a can take on three values minus one
zero plus one and we can take on two
values and C can take on two values just
minus one plus one and that gives you
twelve combinations totally and we have
taken the output variable is y now I've
called them y1 y2 y3 y4 only because
they are y1 y2 y3 and y4 are replicates
so it's the same core variable but
essentially you conduct the experiment
at the settings for instance minus one
minus one minus one you conduct that
experiment four times again it can be a
parallel effort or a sequential effort
either way what we mean by that is when
we say we conduct experiment four times
you might have one experimental unit and
you separately conduct the experiment
four times on it or you could have four
experimental units and you might you
might use choose two parallely
try the same setting of minus one minus
one minus one on those four different
experimental units but essentially this
is almost a setting where you conduct 48
separate experiments and in design of
experiments language that's just called
48 trials or 48 runs and these are
essentially your results these are your
outputs okay and the convenience of this
is sometimes you can look at this raw
data or you can look at the average Y
and this is nothing but the average for
instance for the average eighty point
seven five comes from taking the average
of these four numbers so each row is
averaged and it's represented and that
that's that those are the results that
we have and this this is essentially
what we look to analyze okay so what is
one way that you can take these results
and arrive at a conclusion of what
values a B and C should be set to
because that's kind of the goal the goal
is to figure out what to set a B and C
to mean to be accurate one of the goals
can be
on to what values to set a B and C such
that you get the best y and here we're
going to treat best as the highest value
okay so how should I set a B and C so I
get the best y1 approach is called the
classical analysis
the idea behind classical analysis is to
take each variable individually each
input variable individually and ask the
question at what setting am i getting
the best results so a is set to minus 1
a is set to 0 and a is set to plus 1 so
if I ask the question what is my AB what
is my Y on average when a is minus 1
what is my Y on average when a is equal
to 0 what is my Y on average when a is
equal to plus 1 and I look at these 3
numbers and I'll choose the value of a
where my average Y is the best
I will similarly do that for B and C and
I'll come up with a recommendation on
that basis so what does that look like
for this data it's a fairly
straightforward calculation when a is
set to minus 1 us instigate a t 1.81 so
that just means you can think of it in
many ways you can think of it as the
average of these data points right here
what I've circled and the average of
these data points or you can think of it
as the average of these data points
because ultimately each row comes from
an average from from that perspective
row and the sizes are equal but you can
think of it either way either way this
is the average Y when a is set to minus
1 that's 81 point 8 and similarly you
get an average for 0 you get an average
for plus 1 and so you would basically
say ah I like a at minus 1 it's giving
me the best results similarly you do the
do it for be at minus 1 and B at plus 1
now when you do it for be at minus 1 and
B at plus 1 how do you take the average
it's the same principle
alright so you would for instance at B
at minus 1 you
would be interested in taking the
average of these points right B is minus
one at these points so it would
ultimately be the average of these
points so so essentially that average
would be B at minus one hundred seventy
seven point eight three so you do the
same process and it's clear that eight
minus one is the best be eight minus one
is good because be at minus one is
greater than B at plus one and C it plus
one is good because that's better than C
at minus one okay so that's essentially
classical analysis and it's it's it's a
fairly heuristic approach and it's like
a first cut approach now that's one way
of going about the analysis another
approach is to take the best the take
the best essentially says I'm going to
look at that treatment combination which
gave me the have highest average Y so
what would that be Archer so it looks
like eighty five point seven five is the
highest average Y right and that setting
is I believe a one one one and and so we
would essentially go with that
recommendation so take the best would
have selected a is equal to 1 B is equal
to 1 C is equal to plus so here are two
fairly contrasting approaches and the
question you need to ask yourself is
where would you want to use classical
analysis and where would you want to use
take the best and to answer that
question we need to answer that question
in terms of if you were to take a one
factor at a time approach right what are
two reasons that experiments that that
particular approach fails or for that
matter what are what are the what is the
two major difficulties with design of
experiments the two major difficulty is
are the following one is that you could
have some interactive effects between
the input variables which means the
effect that an input variable will have
on the output variable it really depends
on how some other input variable is set
okay so that's called an interactive
effect the other reason
that sometimes you get fooled because
yes why is some function of a BMC but on
top of that the right there is also some
noise so that noise would could have
given you results which you are
erroneously interpreting and you're
essentially overfitting and you're
getting fooled now the the question is
under what circumstances would a
classical analysis work under what
circumstances would I take the best work
and the quick answer is a classical
analysis essentially works really well
in an environment of high error or noise
so when this that when that when when
the noise which is so we said Y is equal
to f of a B and C right but it's also
got this noise component and when this
noise component although this there
could be no bias to this noise so this
noise could be something like it's
normally distributed with mean 0 and
standard deviation some standard
deviation right then if Sigma E is very
high and so it's a very noisy
environment then classical analysis
would work quite well because it's
averaging a lot of data points now where
it won't work well is when there is lots
of interaction so you need low
interaction effects for classical
analysis to work because it's just
taking the average at a and average it
be in contrast take the best would work
only when there's very low error or
noise it would not work well when Sigma
Y square is high but because it just
takes the best combination it's almost
like it doesn't care about the
interaction so high interactions work
very well in its favor and so you would
use take the best there now the reason
we talked about these two heuristics is
to really motivate the statistical way
of doing things which is this dichotomy
that you see between high noise versus
low noise and and fitting to a
shape you want versus not being able to
fit any shape you want has a lot to do
with something you've seen before which
is the bias-variance dichotomy right and
this bias variance I cata me is what
you're seeing in these two extreme
approaches the statistical way can
sometimes balance that out and so the
truth is almost any supervised learning
algorithm could be a fair game the only
context is that the data set is fairly
small but any data set any supervised
learning algorithm that does not require
a very large data set that requires a
very small data set but can still give
you meaningful results could be a fair
game and in that regard stepwise
regression is usually popular in
analyzing designed experiments you start
with the basic model that y is equal to
a linear function of all the - all the
inputs a B and C and then you also try
to incorporate two-way interactions in
the form of saying a times B a times C B
times C and you could even have a
three-way interaction a times B times C
and because your variables are coded to
minus 1 and plus 1 just multiplying a
and B as an input can can have a
meaningful interpretation ok now one
thing that can be said about this
process of design of experiments is the
way we've done it with full factorials
and the way we've explained it makes it
essentially a one-shot approach it
basically means you've decided even
before you see any results the entire
set of points that you want to look at
at the input space so if you look at
this you've already decided so each each
treatment combination is a point in the
input space you've already decided all
the points and how many times you want
to look query each of these points even
before you look at even one result so
it's ideal in some sense for a parallel
deployment but if you could do the
sequential e is there a better way of
doing it which is can you react to the
data you're seeing to say I want to
query this point
more because I'm less certain about this
point or variable versus saying oh I'm
very confident about something else so I
don't want to waste my resources in
conducting experiments in a particular
place and that's primarily the
motivation of sequential experimentation
this idea that you can you can conduct
experiments sequentially and that gives
you an opportunity to react to the data
now in many ways while sequential
experimentation has been an effort from
the statistics community active learning
essentially is the same core idea and
it's been motivated more from the
machine learning community and the
context in which the problems are
applied do sometimes differ as a result
active learning is often seen as a semi
semi supervised learning approach and
understandably it's also called as
optimal experimental design because
active learning is this process where
the system sequential II chooses to
query the design space and and and
therefore be able to build knowledge on
what what we see and how it can be
better understood the the key difference
especially in the you know in the
typical context of application is that
while design of experiments often
focuses a lot on this whole idea of
starting with zero data often active
learning we'll start with this notion
that there's some amount of data and
it's not enough and you might want to
sequentially query the system to improve
upon your understanding of the system
itself and so in many ways it's kind of
seen as semi-supervised learning because
there is an abundance of unsupervised
learning data just means there's an
abundance of states of the input space
and
you sequentially get you choose points
in the input state for which you really
want to get answers and therefore get
outputs for and by doing that the whole
idea is that you can get away with much
less data on the output space and you
can still come up with predictions that
are meaningful so what are some
prominent strategies in the whole act of
learning framework is that in a sense
all of these strategies that we are
going to talk about now learn one thing
right it relies on you know evaluating
the how informative different points on
the input space can be if you queried
them and you got an answer what we mean
by query it is you choose a particular
point in the input space and say okay
can you please give me an output for
that and that now goes into your data
set where you can apply some kind of
supervised learning approach to make
sense of what you have and the
strategies that are involved with active
learning can be broadly classified in
these four and you know this this is an
area of you know where there is active
research and and so there might be some
strategies that also follow a fall that
are new that might fall out of it but
these have been historically the more
popular ones so let's take a look at the
first one which is uncertainty sampling
now this is perhaps the most simplest
and also therefore a fairly common
framework and in this framework
essentially the active learner chooses
to query instances which it is least
certain about how to classify or how to
predict so if it's a classification
problem it says I'm going to ask
questions about
I'm gonna choose points on which I want
answers on which I want you to give me
an output and I'm gonna choose points
where I am least certain currently given
the information I already have I am
least certain about if it's a
classification problem of zero and one
I'm least certain about whether to
classify it as zero or one in those
instances I really want you to I want to
conduct my experiment in that point
where I'm least certain now if you
cannot you can also think of this as a
regression problem where at a certain
point in the input space you might have
a prediction but that prediction is not
cast in stone right you have some kind
of confidence bounds you have some kind
of bounds around that prediction there's
some amount of uncertainty associated
with a particular predicted value and
you might choose to pick the point where
your uncertainty is the highest another
approach also a fairly popular one is
this idea of query by committee so let
me just mark where we are we finish this
and the second is query by committee
query by committee approach essentially
involves having a committee of different
models which are all trained on the
current current data set and when we see
the data set here we are talking about
the data set for which we have the
outputs already right you have a dataset
with the outputs and those so you have
some kind of supervised learner which is
capable of interpreting the data but you
might have a committee of models which
are all trained on the current data set
but they might represent competing
hypotheses now each of these members is
now allowed to vote right and you
basically choose to take you choose to
get a data point from
the input space where the committee
members have the most disagreement right
so think of it you know one wait you
know one example that I've always kind
of liked about this is it really maps
onto a lot of ensemble methods something
that we saw earlier in this course so if
you had a set of methods a set of
supervised learning approaches all
trying to predict the same thing so
think of it as a random forests where
you have multiple trees trying to
predict the same thing so for a given
input vector each tree it is not going
to come up with the exact same class
prediction or it might not come up with
the exact same prediction even if it's
just the output is a continuous variable
so this method simply says let's go with
let's go get an answer for a data point
where the tree is disagree most about
what to classify it or what where
there's the highest variance in the
predicted value of the models yeah the
next approach is expected model change
the idea behind expected model change is
to see if we knew the label right of a
particular data point then which label
of the input space if we knew would
contribute to the greatest change in the
current model we have based on the
current data we have of the inputs and
outputs so you have some some data on
the inputs and outputs and you basically
extrapolate this to this broader
question of asking the question saying
you could get data you could get an
output for any point in the input space
which point would essentially lead to
you making the largest change in the
model and the idea is to kind of query
that that point very specifically
okay the last two sets which is the
expected error reduction and various
reduction use the following approaches
the approach is to basically say with
error reduction right the idea is
there's some deviation of the prediction
versus actual so you can think of it as
essentially the residuals in a
regression case and you know in other in
every other case like whether it's bias
or variance was for whatever reason you
are unable to predict you are unable to
predict the exact value at a particular
location and the question we need to ask
ourselves is an answer to which point in
the input design space could lead to the
largest expected error reduction okay
and in that sense it's it's fairly close
to uncertainty sampling but it's not
just uncertainty sampling
it's the the where it really differs
from uncertainty sampling is uncertainty
sampling ask this question about each
data point in the input design space
with respect to that data point so I'll
go to one data point in the design space
and ask the question saying how much
uncertainty is there in that point right
in terms of my prediction at that point
where as expected error reduction does
not talk about a single point it talks
about how if I got an answer to a
question at a particular point so I go
to a particular point in the input
design space and I query the Oracle and
I get an output now that output if I now
refit my supervised learning with this
extra data point there's going to be an
overall reduction in my residuals across
the board because this new data point
could change the entire regression line
fit right now this new regression line
fit will create new residuals and so
looking at the overall reduction in
error between the data points and the
fitted model and so I choose to query
that data point in input space and get
an output for that data point which will
lead to an overall reduction in error
now the variance reduction approach is a
deviation from that it's a deviation in
that it's you don't look at the error
between the fitted model and the data
points but you just look to see reduce
the overall variance in the output space
okay
now that is done probably because it's
it's much easier to do this but you what
the core idea here is that you have some
variance associated with the outputs and
you can still continues to reduce
generalization error indirectly by
minimizing output variance and and and
that also can sometimes you do that
because it's it's mathematically easier
and you can kind of get a closed form
solution and and so that's that's
essentially the core idea okay so I hope
this gives you some feel for
experimentation and the whole idea of
active learning and at least some
strategies that we could use in
experimentation and active learning
thank you