Summary+ Insights into the Final Exam Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to our last lecture
for the course introduction to data
analytics this is Professor nonlin and
in this lecture I will just briefly give
you a summary of all the topics that we
have covered during this course
perhaps tie some topics together and in
the last few minutes I will try to give
you some insights into what we will be
looking at during the final exam so
first off thank you for joining us I
speak for both professor Raveendran and
myself in saying that this has been a
very valuable and enriching experience
for us as teachers you know one thing to
understand is that this whole online
learning teaching model is relatively
new in the Indian context and so for us
as teachers as well as you as learners
but the hope is that you know we've
we've been able to reach wider and
hopefully more people have been able to
kind of get access to our content and
are able to apply some of the things
that they've learnt in this course
either in their academic lives or in
their jobs their organisations or maybe
even their personal lives but in any
case we have be happy to have undertaken
this endeavor and you know from the
bottom of our hearts thank you for
joining us what this course in many ways
try to do is is so this course was it
was not a course strictly in machine
learning it was not a course strictly in
inferential statistics it was not a
course in probability 101 alone you know
so in many ways we try to give you the
the broad brushstroke associated with
the entire landscape of data analytics
so in some sense hopefully in you're now
familiar with a lot more terms and lot
more techniques and concepts then if you
take in a far narrower course so the
hope is that we've covered the
breath of data analytics and it's never
possible of course to come with the full
breadth but to a reasonable extent the
fundamentals and the basics of most
concepts we've also tried to go deep in
certain areas and it's primarily been
more in the conceptual front to give you
insight into the house and the whys and
the watts of the different techniques
why does why do some techniques work in
some places why don't they work what are
things to watch out for and so on and so
forth and and so we we also hope that
it's given you a well-rounded enough
knowledge to build upon further so you
know with this highly changing landscape
of different software packages that come
on different days we've tried to make
this a little less tutorial like if
we've given you some derivations but
this has not been fully about the
theoretical math of the course because
the idea is that that kind of depth is
something that depending on your place
of application and depending on your
more specific area of use of data
analytics you should hopefully be able
to build upon given the knowledge that
you've gained in this course in this
lecture I'm going to go week by week and
just talk about you know what we hope to
cover there but this is not as you will
see shortly not just a list of each
lecture and talking about it so to some
extent the topics themselves are
beautifully aligned with the list of
lectures but please do not think that
each bullet point as I go from one week
to the next is a comprehensive list of
lectures it's more the topics in the
concepts so I'll just talk to you a
little bit about the broad topics and
how they tie together and in the last
few minutes as I mentioned I'll give you
some insight into the exam so in week
one
we we really started off with the course
overview and the hope was both professor
Raveendran and I were able to give these
you know long overview lectures we had
an intro lecture and then I gave these
fairly I would say long lectures that
just try to cover the entire landscape
in one to two hours and so hopefully
that gave you some insight into what the
overall course was about um the rest of
the week was spent in descriptive
statistics right we talked about
graphical approaches so different ways
of representing data and in this
particular case with graphical
approaches you're really talking about
different ways of visualizing data an
important concept that was also covered
in this this lecture was that data comes
in different types and we spoke about
continuous data categorical data we
spoke about actually we broke it down a
small today to versus qualitative and
then quantitative broke down as
continuous versus discrete and then
categorical broke down as ordinal and
nominal but that was that's an important
concept that we kept on seeing at
different points of the course we moved
on from visualization and we moved on to
kind of individual statistics right
statistics that represent or metrics
that represent you know central tendency
so those were the mean median mode and
then there were these measures of
dispersion so things like standard
deviation mean absolute deviation and a
lot of the time was not I mean so some
amount of time was explained is spent on
explaining how they work but some amount
on where they work and how they work and
why some are suited in some situations
and others and other situations we then
moved on to probability distributions
and inferential statistics so we spent a
good amount of time you know introducing
the concept of random variables as
different from you know regular
variables or constants that we are used
to so what does it mean when we talk
about random variable and how do you
represent random variables through you
know probability density functions
cumulative density functions probability
mass functions and about how probability
distributions can again be discrete or
continuous and and you know how they get
treated differently we also spend some
time talking about
set of specific probability
distributions namely we spoke about the
uniform the continuous version the
discrete version we spoke about the
binomial the Poisson the geometric the
exponential as well as the normal
distribution finally in the latter half
of this week we moved on to inferential
statistics we spoke about this core idea
where sometimes we have a phenomena
that's generating data and and what you
have is essentially a sample of that
what you have therefore in terms of data
is a sample the data is a sample partly
perhaps because you we were only able to
access a part of the data the overall
data or sometimes it's a sample because
this a broader phenomena that's creating
this data and we've just taken a subset
of that and the core of this inferential
statistics has been about how you don't
just want to make statements about the
data that you have at hand but more
about the underlying process that's
creating this data and that was our
departure from descriptive statistics
and that's what we try to motivate in
those lectures and we also spoke about
this idea of single sample tests so
tests where you have a sample and the
sample is coming from this broader
universe and how do you compare this to
specific numbers or test hypotheses for
parameters associated with the
population we then got deep into
inferential statistics so we went in to
sample tests the idea that you might
have two sets of data that represent two
different populations or two different
phenomena and how can you compare them
you can compare their means you can
compare their standard deviations you
can compare their proportions and so on
and then we got into a lot of other
concepts so we spoke about type 1 and
type 2 errors the errors associated with
saying that the null hypothesis is
incorrect or you're rejecting the null
hypothesis when actually
then our hypothesis was not incorrect
and the other error the type 2 error
which is the idea that you fail to
reject the null hypothesis when you
really should have we also extended that
to confidence intervals another
important concept in inferential
statistics and we spoke about ANOVA
analysis of variance and the test of
Independence so with the ANOVA we spoke
about how you are able to frame more
complex hypotheses like the mean of
population a is equal to the mean of
population B is equal to the mean of
population C and so on so with a normal
test of independence really the for the
first time we were able to look at more
than two states more than essentially
two samples if you can think of it that
way more than two states of a particular
variable so up on to the one sample and
two sample tests at most a two sample
test you have you had this variable that
could take on two possible States and we
would compare them but for now now you
have you still are looking at
categorical variables with the ANOVA
especially in terms of the the input
variable but it's taking on more than
two states and with the test of
independence both the the two variables
that you are interested in whether you
want to think of them as input and
output variables or just two independent
variables to to two variables that they
are taking on could take on two or more
states and you're interested in seeing
if these two variables are independent
of each other or not we also use this
week to come up with a short
introduction to regression right what is
regression all about what is it trying
to achieve and what kinds of problem
statements do we address in regression a
part of the focus has been about talking
about how typically the input and output
variables associated with the regression
are continuous quantitative variables
and how this process of regression gets
used for from a predictor for prediction
as well as interpretation of the
relationship between
the input and output variables we then
choose to introduce machine learning and
and the the the whole idea of mixing up
the regression process and the machine
learning process was very carefully
thought through because machine learning
being a relatively a relatively new
field and regression but also one where
a huge subset of it which is supervised
learning really focuses on many problems
that regression at one point right to do
and perhaps does it much better but to
kind of not think of them as isolated
topics but to think of regression very
much as one of the tools in the toolbox
of machine learning irrespective of you
know which when they came into
prominence was was important so many of
the concepts in machine learning just is
equally applies to regression so we
introduced the idea of supervised and
unsupervised learning where with
supervised learning we said there is
this notion of an explicit output
variable that you are looking to predict
often the way it's formally defined is
that it's labeled data whereas with
unsupervised learning you have unlabeled
data or basically there is no one single
output variable that you are trying to
you know predict or to classify we then
we're able to go back to the regression
and talk about using the ordinary
least-squares regression we saw a
derivation associated with how we get to
that and we also spoke about how we can
do regression in Excel and MATLAB what
are the various concepts associated with
it we spoke about the idea of using
backwards and forwards stepwise
regression best subsets regression and
so on we then went on to a concept
that's very important with regression
called regularization and coefficient
string shrinkage but it's it's in some
sense extendable to other methods but we
focused it more on the regression front
and the idea with the regularization was
how do you focus
on models that don't wind up overfitting
the data because you can always
introduce enough complexity into the
model make the model so complex that it
winds up doing a very good job of
representing the data at hand but it
doesn't really do a good job of it
really doesn't do a very good job of
predicting new data because all the
complexity and all the use of many
variables more than those required they
wound up getting used to kind of just
over fit the data to just do an exact
replica of the data highly related to
that is also the concept of coefficients
shrinkage especially when you have a
couple of variables that are highly
correlated to each other a couple of
input variables that are highly
correlated to each other and we call
this the problem of multicollinearity
and how there are methods for
coefficients shrinkage like a ridge
regression or lasso regression which
attempts to not allow coefficients to
balloon in typically opposite directions
in an effort to kind of better represent
the data we also gave you some insight
into the whole data modelling versus
algorithmic modeling ideas and the the
core concept that we were trying to
instill there is we introduced a new
approach called the K nearest neighbors
and contrasting the K nearest neighbors
with the regression based approaches
really spoke about how with some
approaches which we're going to call the
algorithmic modeling approaches you
didn't have to have an explicit
mathematical model and in fact it was
better represented by a set of
algorithmic instructions rather than a
mathematical model that connects the
input to the output or at least the
mathematical model is not constricting
the you know potential relationship that
the input and output variables could
have
because an algorithmic representation
could be more flexible in terms of
representing this relationship and the
KNN versus regression hopefully captured
that dichotomy in our next week we
focused on supervised learning which is
as we just previously discussed this
area where you have an explicit output
variable and you really trying to
predict that output variable and as you
can see in this set and this is a
commonly used normal clay Chur of having
regression and classification techniques
hear the word regression
unfortunately when it's being used
really means problems where the output
variable is continuous and quantitative
rather than discrete or categorical and
for those where the output variable is
discrete a categorical you had these
classification techniques and in that
light we we spoke about logistic
regression and support vector machines
and and decision trees now all of these
techniques are definitely work as
classification techniques while
classification and regression trees can
do both regression as well as
classification with logistic regression
we introduce the subject we also taught
you how to train a logistic regression
classifier
similarly with classification regression
trees also known as cart or more broadly
cut becomes algorithm the decision trees
are just a way of talking about such
trees support vector machines was also
introduced along with some derivations
and how the classification through the
support vector machine happens and the
core concept there being that you are
really creating this hyperplane and
you're trying to maximize the distance
between different classes to the
hyperplane which is actually splitting
the input space thereby enabling you to
build a classifier and we also spoke
about the case where you have some kind
of violations where some data points of
one class over
spill to the other side of the
hyperplane and the case where there are
non linearly separable data and the use
of kernel transformations now we also
spoke about this concept of the
bias-variance dichotomy and the
bias-variance dichotomy talks about the
concept of overfitting versus
underfitting where the idea is that
there is a sweet spot in terms of the
model complexity for a given data set
for a given underlying relationship
between input and output variables there
is a sweet spot in terms of the model
complexity and when you have too little
model complexity you're essentially
building a regresar or classifier that
is bound to have a high amount of bias
which means that irrespective of how
many times you repeat the process with
different data sets you're always going
to be chronically under representing or
over representing such certain regions
in the input space in your process of
prediction now when you move to the
other side which is you have a model
that's too complex you over shopped your
sweet spot for this given data set I
mean for this given underlying
relationship you you are essentially
likely to create a model with high
amount of variance and a low amount of
bias which just means that each time you
do this task you're going to get a
completely different model on average if
you were to do this millions of times
the average of all your models might not
have much bias so you're not chronically
under representing or over representing
sometimes you just don't know what
you're going to get each time you do it
and so at a given exercise where you
have a given data set and you're
choosing to build a model it could still
be a really bad model and so this is
known as the bias variance dichotomy and
it that really segwayed into the idea of
how do you
therefore evaluate models and therefore
select models in that like we spoke
about some of the metrics some of the
metrics that penalize model complexity
beyond a point because you can use that
penalization in conjunction with trying
to do the best job you can of fitting to
the data to come up with that balance
but we also spoke about some more
computationally intensive but but more
effective I would say techniques like
cross-validation which don't just do a
blanket penalization based on complexity
but instead allows you to hide some data
we build the model on other data and
then try to actually do the job of
predicting on this new data set and
essentially do this kind of
out-of-sample validation and then
thereby make decisions on both the
complexity and the type of model a small
note on the use of linear discriminant
analysis and the quadratic discriminant
analysis you can also think of them as
methods those were essentially that this
is the last bullet point that I'm
talking about we we made a careful
decision not to actually include those
lectures if you look at this week it's a
very this was a very packed week with a
lot of content and way more hours and
lectures and so while this is included
in the syllabus we made a conscious
decision not to give out lectures in
this topic and that's partly because we
actually we had discussed a lot of very
important classification techniques
relatively I would also say more popular
ones but to just give you some insight
the way linear and quadratic
discriminant analysis works is that it
tries to create a distribution a
multivariate distribution essentially in
the input space associated with each
class and come up with the decision
boundary based on the probabilities
associated with this multivariate
distribution of each class for a given
input point so we don't wait to actually
ask the question of each multivariate
distribution means
to create these multivariate
distributions for each class because
this is primarily used in classification
it's a multivariate distribution in the
input space and and we basically come up
with decision boundaries and this
decision boundary is a linear or
quadratic but essentially you come up
with these decision boundaries based off
of the distributions you've built so
right off the bat
tomorrow when someone comes and says
here is this input can you classify this
for me you can see on which side of the
boundary
this falls and more often than not the
typical approach is to use the
multivariate Gaussian or the normal
distribution and an important
distinction between linear the linear
case and the quadratic case has to do
with the assumption of equal variance so
if you make the assumption of equal
variance we essentially using linear
discriminant analysis if you do not
assume equality of variance you using
quadratic discriminant analysis okay so
in the next week we really took on some
more advanced topics in supervised
learning we spoke about the use of
ensemble methods and more specifically
we Illustrated that with random forests
the idea behind ensemble methods in
random forests were really to do with
the idea that you can sometimes have a
deterministic algorithm on a given data
set but sometimes it really helps to
inject some form of stochastic city
either in the algorithm itself or in the
data and thereby create multiple
versions of this algorithm more often
than not is in the form of versions of
the same algorithm but sometimes you can
also have completely different
algorithms that are trying to do the
same job but the idea is that you create
multiple versions and then you aggregate
their predictions whether the
predictions are regressions or
classifications you try to kind of
aggregate them to come up with one final
prediction and this sometimes helps
especially this helps in these methods
where a fairly like greedy algorithm has
gotten stuck into some form of
suboptimal right
because as you would have realized by
now a lot of the machine learning
techniques rely on optimizing some
function or the other whether it's
support vector machines whether it's you
know ordinary least-squares regression
or whether it's classification and
regression trees yeah at some point
class you know trying to optimize some
function and thereby improve your
ability to predict but as with any
optimization the most common thing that
gets discussed is getting stuck and
sometimes this kind of ensemble approach
where you just forcefully introduce some
noise into the whole process helps you
break out of that and overall prediction
kind of improves we spoke about how
random forests this is one way of doing
that with classification and regression
trees we then address the topic of
artificial neural networks and extremely
popular and becoming even more popular
approach to machine learning and
predictive analytics we introduce the
topic in terms of how artificial neural
networks are constructed what is their
motivation or inspiration we spoke about
how we can use the backpropagation
algorithm to construct the artificial
neural networks and we finally took it
to towards deep learning in a reality
it's essentially just this neural
network with many many many layers and
these really new and innovative
techniques of actually being able to do
the the computation and the math to
create such elaborate networks which was
previously not really feasible more than
anything we move on to week seven where
we talk about Association rule mining
and big data with Association rule
mining we spoke about this in the first
context of it being there were two
topics in unsupervised learning and one
of them was Association rule mining the
other being clustering analysis we
introduced the idea of Association
mining and spoke about it's fairly you
know most common use case which is the
market basket analysis the idea that you
can create this table where the rows
represent various instances or
transactions that have happened and the
columns really represent features and
what you have is essentially a binary
table of how frequently these attributes
or columns have been seen with each
instance and with Association rule
mining you're really trying to make a
statement not about the rows themselves
using the rules you're using the
instances but to make statements of the
form that typically when column a occurs
column B also occurs in the market
basket or context which envisions a
supermarket you are really saying things
like the rows here are people making
actual transactions of purchases the
columns are the different products that
they choose so the end result being that
you want to make statements like people
who bought coffee also bought milk or
more complex things like people who
bought coffee and milk also bought sugar
so Association rule mining associative
rule mining is useful there and but it
really extends it it extends to any
context where you can create such rules
and it extends to any context where you
have instances as rows features as
columns and you really have a binary
data set in that sense we briefly talk
of spoke about different ways of
measuring how good a rule is and these
are called interestingness measures and
some of the other challenges that are
associated with scaling Association rule
mining finally we spoke about a buzzword
that we've all been hearing about and
this really spilt a little over week 7
and week 8 right the whole idea being
that we've heard this word big data how
is that different from the broader topic
of just data analytics or data science
in general
what this means and why is it that we're
seeing what do we mean when we say big
data you know does it just mean volume
we've heard other v's associated with it
velocity variety so on we've also spoke
about house some of the challenges with
the regular techniques take on a
different form with big data when you
have a much larger data set what are
some of the computational challenges
that are associated with it are some
methods more amenable than other methods
and how do you scale up such that you
are able to actually process a huge
amount of data and and still get
insights from it finally in week 8
we presented clustering analysis one of
the other unsupervised learning
techniques and we spoke about how this
is used primarily to segment data to
take it to take unique instances and
group them together so here in some
sense you have this data set which is
represented by these instances or
transactions or whatever it is that you
want to think of as rows and these rows
share a common set of features now based
on the common set of features that they
share and the different values that they
take up with respect to these common set
of features you want to create clumps or
groups or segments or clusters of your
data points of your rows and the
different ways of going about doing that
we spoke about some core concepts
associated with clustering but we also
introduced two concrete methods the
k-means and the hierarchical forms of
clustering and how they work and why
they work and so on towards the end we
spoke about you know some amount of
prescriptive analytics and the idea here
was that sometimes you don't have data
and this lack of data you
could mean that you do different things
are different forms of analytics or I
should rephrase that you sometimes you
don't have enough data and actually a
small correction this should not be
predictive analytics there should be
prescriptive analytics but going back to
my point you know you you have you
either don't have data at all or you
have very limited data and in some sense
you have some control over what data it
is that you're trying to get a hold of
so in the case of experimentation much
of the focus is on figuring out where in
the input space do you want to generate
data or another way of rephrasing that
or another completely different way of
looking at it is can you go about
systematically and purposefully making
changes to your input space and my
changes we just mean setting it to some
value and then getting a data point
getting an output associated with that
now that has a lot of advantages right
because you can get rid of problems of
multicollinearity and because you very
carefully choosing where you're getting
outputs you're able to analyze the data
much better or with far more limited
data and that's the same whole premise
of active learning as well active
learning starts more from a space of
saying I might have some results already
but can I go about sequentially querying
the system and and choosing one one at a
time you know sequentially rather than a
parallel deployment of 10 20 30 data
points but can I can you give me based
on the results based on the outputs that
I see I'm going to come back to you and
say can you please give me a data point
at this point and and get an output
specifically for a given region and
update your model or update your system
based off of that and again the whole
premise is that you should now
technically need a lot fewer data points
to learn about the system or to model
the system than a traditional
where you only have observational data
you had no choice or right in saying
where you got your labels where you got
your output variables which points in
the input space for which points in the
input space did you get your outputs
this also links a lot with reinforcement
learning where again you have a say in
which points of the input space get you
get an output but here the output
carries some value the output is
essentially also seen as a reward you
know seen as some form of online
experimentation where online here does
not refer to the internet but as much as
online refers to this this idea that
whatever it is the output that you're
getting the sole purpose of you doing
this querying of the system is not to
just learn about the system but it is
also to maximize some reward from the
system so you are essentially live you
can think of it in many contexts you can
think of it in a manufacturing process
you can think of it as a website or
anyone who's choosing to experiment but
they're experimenting with a real system
the system that is actually creating
products and services and tasks that are
actually going to the end user so how do
you go about experimenting on such a
system where at the same time as you
wanting to learn about the system you
also want to provide your end user with
the best service or the best product so
how do you go about experimenting to
learn but at the same time also to do as
well as you can while your experiment
hope that gives you an overall picture
of the course itself in terms of the
final exam we're really looking at an
exam where you're given about 30 to 40
questions and they're all going to be
multiple choice and the we've we've had
some concern from students in terms of
the types of questions and the one thing
that I wanted to kind of emphasize is
that the questions themselves will not
need you to do a derivation
you know partly because their
multiple-choice we're not going to say
here the first five steps of the
derivation what's the next step kind of
questions the you know derivations are
important but they're important in terms
of from this courses perspective that
you understand how they're done and why
they're done what is it that you're
deriving and so in some sense a lot of
the questions themselves would be
conceptual you're not going to get like
a fill in the blanks that says what is
the next step of the derivation and
we're going to focus more on the
conceptual part of the courses content
and you know in some sense the why is
the wats where do some methods work
where don't they work what will be the
likely outcome if I do a particular
method on a particular data set you know
so they're very very conceptual and then
therefore they you know any amount of
working out is going to be more
back-of-the-envelope so you can pretty
much take pen on paper and you should be
able to work out some stuff and you know
calculators are fine but nothing you you
really don't need a computer for much of
this and there I would say fairly a
limited number of questions that require
you to actually put even pen on paper
most of them have to do with how deeply
you've understood the concept and how
deeply you've understood how the
methodology works or how a particular
concept manifests itself when you carry
out a methodology and so so you know I
don't think you should have too many
worries about doing complex derivations
or doing you know complex math to figure
out the right number focus I would say
more on understanding the concept
associated with be it a methodology be
it a machine learning concept be it
inferential statistics versus
descriptive statistics or be it a
derivation focus on the concept than the
some sense the details and and finally
good luck good luck for your exams and
we look forward to having this kind of
engagement in interactions with with
students going forward
thank you