Donald Farmer, Principal, TreeHive Strategy
What was the most important thing that happened within BI/Analytics in 2016?
- The election of Donald Trump, and to a lesser extent the UK Brexit vote! In the simple, popular story, the pollsters, with all their statistics and Big Data and analytics got the results badly wrong. However, I believe we should draw a different, and more important, lesson …
- Let’s say that the analysts got it right. There was indeed a 30% chance that Donald Trump would win. In fact he did (the 30% case happened) but that does not mean that analysis was wrong. The difficulty lies in communicating the complexity and ambiguity of the analysis. Just as a weather forecast may say there’s a 30% chance of rain tomorrow: if it does rain, that does not mean the forecast was not the best available. However, as the political analyst Berwood Yost has said, “The incentives now favor offering a single number … Certainty is rewarded, it seems.»
- Despite these public “failures,” predictive analytics and artificial intelligence are not going away. They will simply be a fact of life for business and public decision making. Think of a sales pipeline forecast. In order to plan effectively, the analyst needs to communicate, and sales management should understand, not only a single number, but the range and variability of the possible outcomes.
- As data scientists, analysts and storytellers, we must find better ways to communicate uncertainty, whether in the boardroom, the newsroom or in social media.
What will be the most important development in 2017
- The continuing and rather rapid decline of Hadoop. It’s not just a question of living through the “hype cycle.” Customers struggle with security and management. And when customers seek to save costs, the severe performance hit when virtualizing Hadoop, limits the savings that can be made. On top of this, there’s a skills shortage, and yet the Spark community is growing more rapidly than we have seen with the Hadoop community.
What new concepts will emerge in 2017?
- User experiences for analytics are developing quite rapidly. In particular, natural language engines and bots are becoming more common and more proficient. When I look at the development of intelligent agents, more-or-less autonomous bots and automation, I think there is one great change coming to us all in the world of BI and Analytics. We will no longer be «users of» decision-support applications but “participants in” decision-making structures with intelligence distributed throughout the system.