Is Big Data secure enough?

Most people have heard the concept of Big Data, but few understand how it impacts our lives on a daily basis. The news outlets will sensationalize the government (particularly the NSA) for using Big Data techniques on public data in order to spy on people. For example, selfies are now used by the NSA for facial recognition, but should any of this surprise people? The data we put on the web is viewable by almost anyone and you must realize that companies like Facebook, twitter, LinkedIn are in the business to make money. So let’s face it their data is priceless to the right audience, this includes governments, and don’t think for a second that the US is the only government doing this, however we are the only ones where it makes an interesting news story.To level set, here are some interesting data points from Wikipedia

Government

• In 2012, the Obama administration announced the Big Data Research and Development Initiative, which explored how big data could be used to address important problems faced by the government. The initiative was composed of 84 different big data programs spread across six departments.
• Big data analysis played a large role in Barack Obama's successful 2012 re-election campaign.
• The United States Federal Government owns six of the ten most powerful supercomputers in the world.
• The Utah Data Center is a data center currently being constructed by the United States National Security Agency. When finished, the facility will be able to handle a large amount of information collected by the NSA over the Internet. The exact amount of storage space is unknown, but more recent sources claim it will be on the order of a few Exabytes.

Private sector

• eBay.com uses two data warehouses at 7.5 petabytes and 40PB as well as a 40PB Hadoop cluster for search, consumer recommendations, and merchandising. Inside eBay’s 90PB data warehouse
• Amazon.com handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and as of 2005 they had the world’s three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.
• Walmart handles more than 1 million customer transactions every hour, which are imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.
• Facebook handles 50 billion photos from its user base.[36]
• FICO Falcon Credit Card Fraud Detection System protects 2.1 billion active accounts world-wide.
• The volume of business data worldwide, across all companies, doubles every 1.2 years, according to estimates.
• Windermere Real Estate uses anonymous GPS signals from nearly 100 million drivers to help new home buyers determine their typical drive times to and from work throughout various times of the day.

The amount of data we create daily is mind boggling and the amount of data we put on the web unsecured should be of great concern, but we often do not think twice of posting photos of our children on Facebook to share with friends, or writing on our timeline that we are on vacation (this is invaluable information if you are a thief), then we get upset because the NSA is collecting and using this information to “spy” on American citizens.

The truth of the matter is that the US Government is not spying on anyone; they, like all private sector companies, use the data to form patterns and then target high profile individuals. Is the US Government really doing anything different than Google? Google looks at your browsing history to serve up targeted ads that you are more likely to click on to generate revenue. Amazon serves up potential items for you to impulse purchase based on your previous buying history and items you have looked at, all of this is an invasion of privacy, but some we accept as helpful, even cool, and others we deem as violations of our civil rights. Let’s face it, if you don’t want someone spying on you, don’t put it out on public display!

Big Data security is finally becoming an important topic, many companies are bringing in large amounts of data purchased from Twitter, Facebook and other public sites to develop buying profiles to increase the target rates of their marketing spend, however at what point does a bunch of public data start to become private information. For example, let’s say on Facebook, you always put down that you are at the local Starbucks getting coffee, you go to work and then Tweet about your favorite sports team acquiring a new star player, after work you use Google maps to find a restaurant and you run into an old friend and post a selfie on SnapChat. You go home and then browse a few of your favorite sites. By themselves they mean very little, but combined a company now has a very strong profile of your behavior, what you like, what you may buy, political views, how to increase the chance you will buy from them and so on.
So what happens when these profiles are hacked and the wrong person acquires a bunch of data that seems worthless until it is combined with all the other data they have collected? Is this any better than Target losing 40 Million credit card numbers?

Interestingly both Hortonworks and Cloudera (both Big Data software companies) acquired security companies to enhance their offerings. From my perspective, it is about time. But protecting the data in storage is not the only risk, if you are going to move large subsets of data that do not have strong relevancy on its own, but when combined build very sensitive profiles, then you must secure the transport of the data at all times. Many companies feel that public data does not need to be moved securely since it is already public, but the real risk is the selection of data these companies are collecting and how they eventually assemble the profiles, both components must be secured.

Big Data is a part of our lives now and will not go away, it will evolve and become even more invasive, it is time that companies collecting data secure all transmissions, even public information!