Will access to Big Data further enable fact-based decision-making or analysis paralysis? Will analytics, as well as the supply of analytics-savvy managers, so badly lag ‘big data’ that it will only lead to confusion and misguided decisions? An exclusive HBS Working Knowledge article.
Ideas and trends converge from time to time in a way that suggests the possible shape of the future. Sometimes I think I can comprehend what they may mean. But other times I know I need help. This is one of those times.
Just two decades ago, we didn’t have Google and other information sources; storage constraints would not have permitted Google to provide everyday access to the ‘world’s information’. If we had had the information, we couldn’t have accessed it effectively anyway. Email systems were not widely available, let alone mobile devices with capacity to access the data. Now the capacity to store and access information through cloud computing is so great that we are entering a post-Google era in which new organisations like Factual (founded by a former Google employee) have set as their goal that of providing access to all of the world’s facts. Presumably this means data such as the location of every factory in the world, data that has not already been massaged and spun. Some facts have to be acquired and organised. Other facts are generated by so-called digital sensors operating worldwide in industrial equipment, autos, and the like. By linking the sensors, an ‘industrial Internet’ can be created. These trends appear to have ‘opportunity’ written all over them, particularly for those who are training now for jobs in data analytics. In addition to less wasteful marketing efforts (we should be able to know, for example, ‘which half’ of advertising is effective, thereby making an old marketing saw obsolete), they should produce more effective business strategies and inject added certainty into the appraisal of opportunities for new business startups. Furthermore, analytics (not the data) should be a source of continuing competitive advantage. In his new book, Charles Duhigg describes how the retailer Target uses data on consumption patterns to discern and address promotions to pregnant customers, perhaps even before they’ve announced their pregnancy to friends (and Target competitors). This is particularly important because pregnancy is one of those life events associated with significant shifts in consumption habits.
A problem is that the shortage of experts in data analytics (some call them ‘data whisperers’) is so acute that it may be years before a sufficient supply can be trained. The McKinsey Global Institute estimates that up to 190,000 are needed now in US, along with 1.5 million managers capable of using their work. The shortage appears to be growing along with the potential for competitive advantage associated with data analytics.
This all raises many questions. Will the age of big data eliminate most or all uncertainty from business decisions for those most able to make effective use of ‘all the facts in the world?’ Will it fuel the next ‘gold rush’ for talent in a quest for competitive advantage? Will analytics, as well as the supply of analytics-savvy managers, so badly lag ‘big data’ that it will only lead to confusion and misguided decisions? Or is this just the latest management fad? How, if at all, should this affect education for management? What do you think?
“To T. S. Eliot’s prescient words ‘Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?’ we might now add, ‘Where is the information we have lost in data?’”
That’s Paul Nicholas’ (Director, Soul-Chaplain Consultancy) reaction after reading most of the responses to this month’s column. It’s not a bad “sense of the meeting,” in which many contributors offered suggestions to managers wishing to get the most out of so-called big data and avoid paying the price of paralysis in the process. The challenges Big Data pose for managers include “identifying which data are relevant” (Subrata Chakraborty, a retired professor of management) and “seeing through the woods to know what to use and what not” (Pieter J. de Beer, Senior Partner, De Beer & Keulder Legal Practitioners). Scott Waller (VP, M/A/R/C Research) expressed the fear that “the age of Big Data seems to be arriving at the time of death for the Big Thinker.” Gerald Nanninga (Founder, Planninga from Nanninga) cautioned us that “the big risk is that it gives executives a false sense of comfort.” Clifford Francis Baker (Chair, Neffel Corporation) added, “My concern is primarily focused on the possibility of complacency…data derived from data analytics must…be handled with care.” Mok Tuck Sung (Owner, Profit Tools) commented that “technology and knowledge advancement have again developed faster than the managers’ and business owners’ capabilities to leverage its usefulness in their decision making process.” Philippe Gouamba (Vice President – Human Resources, Skyline Windows, LLC) reminded us that “perfection is but an illusion… A successful outcome is almost never the result of perfect information.”
Among the suggestions for managers were these: 1. Help “data analysers consume and translate the data” (Scott Kemme, Senior Business Analyst, ClickFox, who also suggested an alternative title for the column, which I instead used above);
2. Know “what not to look at” (Phillip Clark of Clark & Associates);
3. Work to reduce turnover among business analysts who “tend to be lower level employees and have a high turnover,” creating a “losing battle” through the loss of “institutional data knowledge” (Kim Kraemer, Data Architect, USG Corporation);
4. Avoid the belief that “whatever is new will solve their problems,” concentrating on the appropriate application of Big Data (Seena Sharp, Executive Director, Sharp Market Intelligence);
5. Avoid allowing Big Data to remain the “purview of the select few” only for use for one-off and one-time decisions” (Jonathan Spier, Entrepreneur in Residence, Altos Ventures);
6. Maintain the attitude that “fast is better than perfect” (Mike Flanagan a Purchasing Manager by profession);
7. Avoid the temptation of harvesting just the “’low hanging fruit’ rather than waiting for the analysts to gain understanding (presumably of the decisions to be made)” (Sean O’Riordain of Trinity College, Dublin);
8. Rather than concentrating on the data, focus on “being able to formulate the right questions to ask at precisely the right moment” (Edward Hare, Retired Director – Strategy and Planning, Fortune 250 Manufacturer).
Tom Dolembo (Founder, New North Institute) drew a picture of direct process control in which decisions have to be made “in the flow” in situations where there isn’t time for conventional analysis (at least by humans). Is this a glimpse into the future of decision-making (without analysis paralysis) based on big data for a widening range of decisions or is it confined to a special set of conditions? Will access to Big Data further enable fact-based decision making or analysis paralysis?
