In an interesting article in Sunday's New York Times (Algorithms Get A Human Hand In Steering Web), Steve Lohr reports that companies like Google, Twitter, and I.B.M. are discovering it's not enough to turn algorithms loose to figure things out, because computers are just too literal, that is to say they can't decipher ambiguity of language & mystery of reasoning. So if "for all their brilliance, computers are thick as a brick," according to Carnegie-Mellon University computer scientist Tom M. Mitchell, human presence regarding Big Data remains an important issue for pricing and retail intelligence among the myriad Big Data pursuits.
Several weeks ago, Deloitte raised a similar debate on their website: Does Big Data Still Need The Human Touch? Saying, on the one hand, data volumes have become so large that oftentimes only advanced analytics and other data analysis technologies can make sense of them; on the other hand, humans – armed with powerful data visualization tools – can glean valuable insights from Big Data that escape even sophisticated analytics software.
So, while automation delivers essential processing speed and efficiency, can it ever fully replace human judgment and insight?
#1. Argument Against Humans: Why overwhelm people with minutiae if computers can do the work for them? Automation lightens everyone’s work load. We have the technology to track, cleanse, transform and summarize data to make decision-making easier.
#1. Argument For Humans: Human analysis is still important. Raw data often contain outliers, and summarizing it might obscure patterns that a computer may not recognize as important.
#2. Argument Against Humans: Big Data’s scale often overwhelms visualization capabilities. Big Data can often involve billions of records, yet even effective visualization tools can’t plot more than a few million data points on a single display.
#2. Argument For Humans: Visualizations can highlight what’s important. Just as a database with a multitude of rows can be queried and summarized, good visualizations make it possible for users to apply filters to data and drill down on items of interest, and then communicate their findings more efficiently at the appropriate level of abstraction.
#3. Argument Against Humans: Humans only introduce the potential for error. Among other cognitive biases, people may often be overconfident in their initial impressions and only see what they want to see. Better to let a computer make the decisions objectively and consistently.
#3. Argument For Humans: Humans may catch what a computer can’t see. Human eyes, with their ability to detect symmetry and adjacency, can see patterns in data that even sophisticated automated recognizers can overlook.
#4. Argument Against Humans: The wrong choice of variables, ranges and scales found in Interactive, colorful 3-D models are pretty and stylish, but they can be a dangerous distraction; and might create an overly simplified and misleading picture.
#4. Argument For Humans: Excessive dependence on computer models can create blind spots. Interacting with and visualizing Big Data is integral to understanding – and trusting – analysis results. Humans should check the assumptions of a model and verify that the right questions are being asked in the first place.
#5. Argument Against Humans: Many naturally curious individuals will likely disengage or question the validity of cleanly scrubbed and presented data packaged from a black box.
#5. Argument For Humans: Some inveterate data crunchers are able to recognize valuable symmetries, relationships and adjacencies in data that analytics cannot recognize. There are people who want to get into the data and dig deeper; they want to understand how it is being processed and to look for patterns and insights…
Of course there are some circumstances in which automation alone could be sufficient for the analytical task at hand. For example, algorithms quickly and effectively determine if a credit card should be accepted or declined in a retail transaction. Human intervention in such a routine time-sensitive process would likely bring the retail industry to a halt. For operational uses, humans get in the way of the algorithm, comments Greg ... call center queues, claims routing, next best offer ... all good opportunities to let the math rule.
Leaving the Big Data alone is ok if it's a part of operational workflow (like a business rule) or it's to inform a strategic decision.
In other circumstances, such as medical diagnoses, the consequences of making incorrect decisions can be severe, so it may well be beneficial to “get under the hood” and question how and why data analysis technologies arrived at their conclusions.
Data volumes are too big and the data changes too rapidly for manual analysis to add value, say some, who posit: How many Wall Street traders still base buy-sell decisions on data they crunch in spreadsheets any more?
“You need judgment, and to be able to intuitively recognize the smaller sets of data that are most important. To do that, you need some level of human involvement,” say others.
Bottom Line For Your Bottom Line:
Some say the sheer volume of data being generated makes most modes of traditional manual analysis unrealistic at best. That doesn’t mean that we should simply automate the entire process and walk away, says David Steier, Director, Deloitte Analytics.
In cases of data visualization and strategy, it's ALL ABOUT allowing the humans to get the insight, form new hypotheses, and make decisions. (And, this is when the math and charts are servants to the business decision-maker), comments Greg on the NYT article.
Upstream Commerce has humans in its pricing and assortment intelligence equations and spoke to this subject in point #3 in our blog post: 7 Dynamic Considerations For Choosing The Best Pricing Intelligence Solution Vendor: When it comes to pricing data, accuracy is paramount. Noise or source of the data directly affects access and accuracy, and, most important, it affects bottom line. So, it's not enough to count on automated systems alone to verify themselves. Human-based, quality assurance checks, performed on a regular basis are imperative to ensure that your data is 100% accurate.
What do you think?
Yes. Is human analysis still important in Big Data?
No. Big Data analytics and other automated data management tools can analyze more data more efficiently, so leave them alone.
Credit for Thumbnail and for more information about The Human Face of Big Data: http://www.humanfaceofbigdata.com/