The Global State of Data and Insights: Emerging Technologies Maturing

An ESOMAR Industry Report extracted from Global Market Research 2019

The data and insights industry can seem almost invisible. It’s tucked behind marketing and business decisions, providing proof points for new products and services, for governments and businesses alike. But this behind-the-scenes industry—valued at $80 billion globally—would be the 67th richest country by global domestic product and is worth almost twice as much as the film industry’s global box office value.

In many ways, it’s unsurprising that an industry built on consumers—their perceptions and behaviors—reflects the characteristics and influences of the consumer world. The same innovations transforming the way organizations provide better and more efficient services to the public are also driving change within the insights industry—and the shift has been rapid.

For the first time, in this year’s ESOMAR Global Market Research report, those innovative methodologies (big data analytics, automation, artificial intelligence, do-it-yourself services, text and advanced analytics, etc.) are valued at the same amount as the originally defined market research sector (online and offline qualitative and quantitative, brand trackers, focus groups, etc.). In fact, 2019 is forecast to be the first year that these new methodologies will exceed the value of the traditional market research sector.

This certainly feels like a watershed moment for the data and insights industry. Heightened trade sensitivity, commercial tariff brinkmanship, currency volatility and unpredictable geopolitical conditions have slowed global growth in many sectors, including conventional market research. An initially healthy-seeming 2.1% growth in the sector reveals a more stagnant picture once inflation is factored in, reversing the trend to a slightly negative 0.3% decline. However, the maturing data analytics and new methodology counterpart bucked these trends with a remarkable increase of 10% based on the previous year.

It’s astonishing to witness the hardships of one half of the industry in contrast with the buoyancy of the other. Yet both are devoted to the same basic activity: analyzing data in order to provide contextualized insights and facilitate better decision-making. If the goal is comparable, then what are the reasons for the conventional sector reporting a net decrease of 0.3%, while the data analytics and insights counterpart reports an increase of more than 10%?

There are numerous reasons for this; the growth of tech-enabled systems, automation and differing definitions of what is and isn’t included in those sectors are but a few contributing factors. What is clear is that the industry is evolving and transforming, and that investment in market and consumer intelligence is on the rise.

Implications for the U.S.

The U.S. is the largest market of data and insights in the world. For the conventionally defined market research sector, North America accounts for 45% of turnover, followed by all of Europe at 35%. We also suspect that the U.S. holds a larger share of the research performed in data and analytics. Throughout the years, the U.S. has been one of the most dynamic and resilient global markets. When looking at the expanded top 10 global companies (both traditional market research firms and newer data and analytics companies) in terms of revenue, five of the 10 are headquartered in the U.S.

But there is an underlying story to the top 10 traditional market research organizations. Four out of the top 10 companies signaled a reverse in growth between 2017 and 2018. There can be many reasons for that, but it does fit with the growing industry trend of specialization. As brands bring more insight functions in-house, there is a greater demand for specialization in insight and analytics companies, whether in methodology, vertical sectors or data sets (think big data sets or passive measurement). These companies are often smaller and more agile than their larger counterparts. Smaller boutique agencies, and those exploding out of the tech sector, are driving innovation in data and insight.

For marketers and insights teams in the U.S., it can be beneficial to explore the new perspectives these companies provide to solve business problems, while balancing the new with the reliability and tried-and-tested work of the larger generalist agencies. Although North America is a vital hub for innovation in data and insight, incredible tech-driven transformations in data are happening all over the world. Insight is being disrupted by companies across Europe, Asia Pacific and Latin America. Because we exist in an always-on, tech-enabled global economy, it’s as easy for a New York-based brand to work with an agency in Portugal as it is with one in Chicago.

Two Sides of the Same Coin

With the traditional market research sector facing a decline for the first time in some years, and with an ascending tech-driven data and insights sector, does that mean we’re looking at the start of the slow death of traditional market research? We don’t think so.

A huge part of this new sector has been created by our ever-increasing digital fingerprint. Our behaviors—what we think, feel, say and do—are being recorded digitally. From this, a hosepipe of data companies emerged to analyze the increasing volumes of data to reveal key patterns, trends and associations. This has undoubtedly been key to the expansion of the industry and the delivery of new actionable insights at scale over the past few years. But there’s still a vital role for traditional market research to play.

Big data, social media measurement and other forms of passive analytics only tell one side of the story. Just because something is measurable doesn’t mean that it can be applied to key business questions. The value of insight is getting behind the data and looking at perceptions and motivations. Big data can sometimes seem adrift, floating all around us, untethered by context. The problem is that while big data is sizeable, it can also be thin. It hasn’t been the silver bullet that some commentators expected. When combined with other methods, qualitative research in particular—or “thick” data, if you will—then you can really start building valuable and strategic insights and informing better decisions. Providing context is where the traditional market research sector excels.

That’s not to say that newer methodologies are all based on the analysis of big data. Technological advancements such as automation and AI are providing opportunities to revolutionize how we conduct and structure traditional market research methodologies, but they’re based in the theory and practice of decades of statistical and modeling work—the very epitome of conventional research.

Another consideration is around data ethics and transparency. In Europe, the General Data Protection Regulation has been introduced, while in the U.S. we see the introduction of the California Consumer Privacy Act and the beginning of a lengthy discussion around a federal privacy bill. With the growing uncertainty of third-party cookies, it’s become increasingly apparent that success is not defined by the amount of data gathered, but what organizations can do around compliance. In the traditional research sphere, a core pillar in the collection and treatment of data has been data ethics and transparency. This pillar has created a set of checks and balances that’s allowed us to stay self-regulated and offers protection to both consumers and companies. Whatever happens with passive data collection, the traditional research industry will be asking questions and providing insights for better business decisions.

For marketers, the expanded research universe is a dynamic and valuable sector. Insights can be generated with speed and efficiency never seen before, and the U.S. is a vital part of that ecosystem. The Global Market Research report shows an industry that progresses, transforms and impacts businesses across sectors like no other. Whatever the future holds, data and insight will continue to be the keystone for marketing decisions.