Big Data; A small introduction Video Lecture | What is Data Analytics? - Data & Analytics

hello and welcome to this module on big
data analytics so what I would do in the
next two modules has tried to introduce
you to the problems of big data
analytics so we start off by looking at
what is Big Data of course I cannot hope
to cover all the aspects of big data
analytics in a couple of modules in this
course the whole idea is to give you a
very brief or as I say a small
introduction to big data analytics so
when people started talking about data
organizing data and managing data they
initially started with database
management systems might have heard of
relational databases and other forms of
data management systems so the goal here
was ease of access right and to be able
to answer simple aggregation select
queries essentially trying to do some
form of analytics of the data but by
looking at very specific queries that
you can run on the data so the volumes
of data were not as large as we talked
about nowadays and then from database
management systems when the data became
much larger that it couldn't make sense
out of the raw data itself then people
started talking about data mining or
data analytic systems where the goal was
to detect patterns in the data and
people used a lot of ideas from machine
learning and applied statistics in order
to do this kind of data mining to
understand the data better so what has
happened nowadays so this whole system
has evolved to a point where you cannot
just hope to run your machine learning
algorithms directly on the raw data but
you have to worry about the data
management aspects of it as well as the
analytics aspects of it so that is
essentially the crux of what people call
Big Data nowadays the fact that the data
is so large that you cannot look at
analytics devoid of data management so
to give you a feel of what this Big Data
it is a quote from a Richmond who is the
executive chair
at Google so from the dawn of
civilization until 2003 humankind
generated Phi X a bytes of data now we
produce five exabytes of data every two
days
the pace is accelerating so if you can
just think about it right what we have
produced for thousands of years since
till 2003 were able to produce that kind
of data every two days that's the volume
of data that humans are producing and
obviously we cannot make sense out of
this data unless some kind of
organization and analytics is goes hand
in hand with it so what are the main
challenges that come about when we are
talking about data of the scale right so
this is a somewhat of a becoming a
cliched the view of big data analytics
but one that is popular so I thought I
should present it as part of this model
so the challenges are essentially
encapsulated under the four V's of Big
Data
the first volume velocity variety
veracity and I would like to add a fifth
V to it we should call value which is
essentially the goal of all of data
analytics in some sense so I'll briefly
introduce you to each of these ways and
that'll be the end of the first model
right so why is it that we are able to
generate such huge volumes of data so
one of the reasons is that all aspects
of modern life has been digitized all
aspects of modern life has been
digitized and nowhere is it more
apparent than in your social activity so
we have Facebook a lot of people spend a
significant portion of the earth waking
hours on Facebook and possibly a
significant portion of the sleeping I
was dreaming about Facebook and then
they have such volumes of data that is
uploaded on YouTube and whatsapp and
Twitter and Flickr and all the web
sharing portals so if you think about it
almost every aspect of your life is now
being recorded somewhere or better
right and the next thing it's just not
the social activities that you perform
only in social media any kind of
interactions you are having this very
high probability is being recorded for
example phone conversations right you
call customer support calls they are
being recorded any feedbacks and sad
ways that you have that you fill out a
form anywhere that's being with
preserved for posterity any kind of
online conversation chats or sub
transcripts and other things they are
also recorded right so this is like
everything that they're working at is
actually being recorded at some point or
whether right another aspect is the
increasing availability of cheap storage
and cheaper bandwidth that allows us to
share multimedia content at rates and
volumes there was never imagined
possible even five years back I wrote a
few examples that I took from the
internet so the Instagram is growing at
more than 1 billion photographs
I mean Instagram has more than 1 billion
photographs YouTube has 100 hours of new
videos uploaded to the site every minute
and over 1 billion unique users visit
YouTube each month at Facebook ads 30
billion pieces of content every day and
likewise you could take any of these
online portals and you can come up with
these mind-boggling numbers that tell
you how how much data is being shared on
these portals so apart from these
aspects of everyday life and that is
generating this data there has been
significant advance in technology in
other areas for example in biological
data so 3 Giga bases of the human genome
can now be sequence in a few days if
this is like 3 times 10 power 9 base
pairs which is there is a significant
volume of data but then this is for one
human genome right and you can
essentially get your genome sequence for
a few thousand dollars in the US and so
it should be made accessible at a much
lower price range and so the data is
being generated at every enormous
unbelievable mind-boggling space I
protein data banks have tens of
thousands of structures remoting to
several terabytes of data and again
people are continuing to add to this
experimentation so to put this in
perspective so do not note one of the my
father's of computing algorithms and so
on so forth has this to say about
computational biology I think the most
exciting computer research now is partly
in robotics and partly in applications
to biochemistry biology is so digital
incredibly complicated but incredibly
useful biology easily has 500 years of
exciting problems to work on it's at
that level and not only is it in the
data analytics sphere which is of most
relevant to us but in a variety of other
areas as well including data storage and
also in computing hardware and access
technologies biology is challenging
computer science at levels never seen
before so this is not about volume but
what about the rate at which this data
is being accumulated so data is
generated tremendous volumes but a
tremendous rates at the Instagram in May
2012 58 photographs were being uploaded
and the new user was being gained each
second and Facebook ads half a petabyte
of data every 24 hours right and then
it's not even talked about sensor
networks which are generating data
almost at the continuous rate for
example at IIT Madras we get GPS
readings from about 300 buses every 30
seconds and if you think about per city
city by transportation this is really a
small scale and if we are able to
instrument all the buses that run in
Chennai city we can get data at a much
much higher velocity just hold on a
minute
right why were you worried about all
this big data so we are not really I
mean not everyone
offices kind of work for Facebook or
Instagram or Google right so tremendous
amounts of data is already available in
the public domain right either through
Facebook API or Twitter API is all
through government data that is being
made available on open forums and we can
obtain significant insights from
analyzing this public data right and you
could think about sentiments you could
think about trends and a variety of
things see the kind of insights you can
derive is limited by your imagination
and access to data and there is a
significant amount of public data
available that you don't really have to
work for Google or Facebook to think
about big data the second thing is that
sensors are becoming ubiquitous right so
a lot of everyday I think that we see or
being instrumented and then there are
sensors in buildings there are sensors
on bridges and other kinds of
infrastructure out there and people are
talking about Internet of Things this
essentially the whole idea is to connect
different kinds of equipment through
sensors on low-power wireless networks
so once you have these kinds of data
being generated by sensors then you
don't have any drugs of volume or
velocity of data and so you would really
need to develop techniques like this
that allow us to work with data can I
say already mentioned a lot large
volumes of public data is already
available government is pushing for more
access to public information so due to
all of these factors it really makes
sense for us to look at big data and we
don't have to work for such large
companies even though the examples I
gave you to make you appreciate the
volumes that they have to look at we are
drawn from these kinds of online social
media controls so the next to be I want
to talk about a little bit its variety
so traditionally talked about relational
databases then we talked about
structured data with very well defined
fields may be of a few types you know we
could have strings you could have
you could have categorical variables and
things like that but now with big data
data coming from different kinds of
sources right you have many many
variations of this so the first and very
widely available source of data is
unstructured text so we can get things
from the web we can get scientific
articles newspaper clippings a variety
of the sources give you unstructured
text these are not really marked into
saying this is the heading this is the
most important keyword here and these
are all the attributes of this keyword I
mean so you don't get this kind of
organized data essentially you have to
just read freeform text from that form
some kind of a representation which you
can then use for your data analytics
right second the source of information
nowadays it's multimedia as I've been
mentioning you get pictures you get
videos and you get variety of different
sources of videos and pictures and as
well as audio sound clippings and so on
so forth on the on the internet knob and
so you have to come up with technology
for analyzing all of this especially at
the skin and maybe I'd mentioned a poor
sensor data and again this is going to
look like unstructured data that's going
to be continuous time varying signals
that you would be measuring from these
sensors and how would you even record
these how do you make sense out of this
data that's another main challenge and
then you have scientific data and
scientific data comes in a variety of
different forms if you are talking about
medical data it could come in terms of
reading some instruments it could be
black-and-white images it could be
thermal images on hill whatever seeing
here is a snapshot from microarray data
just in computational biology a lot and
so making sense of this kind of data is
requires not only new computing
techniques but also significant
understanding of the domain in which you
are operating and therefore we have to
work more closely in collaboration with
the the scientists who are handling this
kind of data
data analytics cannot be performed in
isolation here so lastly I would like to
talk about linked data like this is
essentially data that comes with some
kind of structure but not the kind of
structure that we are normally used to
this is data and that comes with a
network structure so we are talking you
can talk about entities let's look at a
small close up of this so each of these
nodes here this is a very complex graph
and each of this node here is
essentially an entity and the links tell
you how are these entities connected all
right so this in large volumes of linked
data which give you a relationship
between entities is already available in
the public domain now apart from that
many sources of data that you generate
nowadays have this kind of link
structure associated with it so now the
question is how would you mind such
large volumes of big data so variety
there is tremendous amount of writing
and each of this come with its own
challenge and quite often significant
requirement for domain knowledge so in
the next week we look at this veracity I
had one of the biggest problems in
social media data is figuring out if the
data that is given to you is true or not
right so the falsification of radar I'm
pretty sure that many of you have
actually given false information to
create email ids and other things online
and there's been cases of people
maintaining multiple profiles on
facebook and so that selectively share
information etcetera so it's hard to
discern big weight when you are being
false and when you are giving true
information so a significant amount of
resources are spent by many of these
companies in order to verify the
information that is provided to them
right or if you take sensor data right
the data could be noisy like that could
be some missing values so how do you
account for all of this and so how do
you how do you trust the data that you
are getting from the sensor is it are we
sure that the sensor is not
malfunctioning so how do you account for
that right maybe if they're not a few
sensors
you can do that manually but suppose
you're talking about millions of sensors
deployed over a high-rise building then
the question of manual verification is
moot you just come to that right and I
talked about how biological data is one
of the biggest sources of future
problems for us but even though there
are huge volumes of data available more
often than not these are estimated
interactions and estimated data and
again you are not sure about the
veracity of the data you're not sure
with that let's see if you say protein
interacts with another protein you're
not 100% sure the interaction happens we
can say like 80% confidence I'm sure
this interaction happens so in such
cases how would you handle this kind of
unsure data so this is just examples if
we take any source of data coming from
any kind of domain you'll always find
that there are issues of noise and
trustworthiness in the data so the
verification and people still work with
this kind of data already so some of you
if you've already worked working in some
kind of a data analytics company know
that you have to work with this kind of
data already right but then verification
becomes hard due to the scale that we
are talking about so current technology
people typically end up doing some kind
of internal consistency checks and some
amount of manual verification but we
need something more scalable to account
this big data scales we're talking about
and in the last me I wanted to mention
very briefly it's this whole idea of
value so how all this big data how do i
monetize the big data so we have to
think about problems where some sense
the returns are higher than the
investment and now the investment is no
longer trivial because we really have to
put in a lot of resources in order to
just manage this data at this scale and
then run analytics on top of it so all
the four V's play a role here and so it
basically have to figure out what is the
right balance in terms of the effort
that you put in and the kind of
monetization that we can do so some
examples is people actually
try out our on recommendations trying to
build better recommender systems for
different domains looking at sentiment
analysis trying to get a sense of what
people feel about new products maybe
movies or what do people feel about a
particular candidate election so looking
at that and then other examples include
business process optimization
surveillance healthcare spawn cities
there are many many domains in which
people are trying to find out
monetization for big data so the next
module you look at some of the
challenges in running data analytics at
the scale and we have talked about today
that brings us to the end of this moment