The Market Research Industry Is Being Squeezed by Consultancies

Faced with myriad pressures in a dynamic marketplace, traditional market researchers are seeing themselves upended by technology and consultancies. Can they diversify before being subsumed by larger players, or is their reach finite?

As marketing evolves, so do practitioners’ expectations of market researchers. 

Much has been written about the Madison Avenue set coping with cuts and revisions to their core mission. (Look no further than the world’s biggest ad spender, Procter & Gamble, which has scaled back its advertising budget by an eye-popping $1.15 billion in recent years while pursuing an “open sourcing of creative talent and production capability.”) 

At the same time, agents of change are also shaking up agencies’ left-brained cousins in the world of market research. This sanctum of hard science finds itself in the midst of a years-long transformation that will conclude in widespread adoption of artificial intelligence (AI) in place of many traditional research functions performed by advanced stats nerds, who will see themselves repurposed as boardroom warriors.  

That human researchers concern themselves with the impact of technology is not in question. The 2017 Q3-Q4 GreenBook Research Industry Trends (GRIT) Report identifies technology and innovation as the biggest challenges facing the market research industry. Market research suppliers report feeling threatened by the pervasiveness of fraudulent survey respondents, disillusionment with the actionable value of Big Data, skills shortages in advanced statistical modeling and the looming threat of AI eliminating research jobs.  

That the industry has floundered amid these emerging trends also seems a given, as evinced by the 70% customer satisfaction rate noted in the GRIT report. Customers here are any client paying for research products, who often include other people within market research, as opposed to pure marketers. In a way, researchers dumping on their own industry’s work product is an even more damning indictment than that of downstream marketers.

“If you were doing a CX study of your own users, and they were only 70% satisfied, [that’d be] a big problem,” says GreenBook executive editor and producer Leonard Murphy. 

Another indicator of the field’s stagnation can be found on Wall Street. “Check the stock price for market research companies, the big publicly traded ones, whether they be Ipsos, GfK, Kantar,” says Jeff Reynolds, president and chief operating officer of SWHoldings, a company that includes market research giant Lieberman Research Worldwide (LRW). “These are not companies that are on fire. Most of them are generally flat over five years, indexing way below the S&P [500].”

Market research companies have not ignored these problems, but they have struggled to respond to them. Converting practices to incorporate and complement new tech capabilities is a tall order, and for far too many players, gaining traction is proving elusive. Still, it’s early enough for this signal to serve as a wake-up call, rather than taps. 

Market research is still valued by corporations. That hasn’t changed. But market research consumers now place greater value on insights beyond what traditional market research has measured—and they don’t care where those new insights come from, which has opened the door to tech upstarts and legacy consultancies with new research arms. Industry insiders must act swiftly to keep pace with technological advances, ward off corporate interlopers and refashion themselves as psychologists and market-forecasting tycoons, lest they be overtaken. 

What’s Wrong With Market Research?

Fixing market research starts with understanding what’s wrong with it in the first place. There are a few ways to parse the dearth of confidence in and poor stock performance of market research suppliers. Murphy interprets the low customer satisfaction as a reflection of market research’s narrow product suite, which fails to meet marketers’ expectation in 2018: advice on what they should do next to grow their business. 

“That consultant component is something that market research struggles to achieve,” Murphy says. “A researcher’s recommendations will generally be something like, ‘Utilize ad A versus ad B,’ or ‘Add these features into this product.’ It’s not strategic.”

The concentration on tactical execution to the exclusion of strategic advice often relegates research companies to also-ran status if and when CMOs decide to seek help for answering the big-picture questions about their brands.

“A CMO pays McKinsey millions of dollars a year,” Murphy says, “but a CMO is not necessarily going to pay a GfK millions of dollars a year for consulting. They may pay millions of dollars a year for data that helps support their internal KPIs and helps drive their decision-making.”  

This might not appear to be a problem for market research folks, provided they maintain their position as number-crunchers. However, there are signs that role is collapsing, as outsider firms that have spent much of the last decade adding market research arms to their roster of services displace research companies. Consulting stalwarts McKinsey and Bain & Co. have beefed up analytics offerings to tackle data synthesis: The former launched McKinsey Analytics in 2011, and the latter hired market researcher Barbara Bilodeau to lead its advanced analytics group. Tech giants with plenty of capital have joined contention as well. Google’s 2016 purchase of UX platform Dialogflow was followed up by Snap Inc.’s acquisition of intelligence research platform Placed, as well as Amazon’s purchase of data aggregation and visualization brand Graphiq a year later.

These businesses all want to collect the millions of dollars going to market research outfits. And for many complicated reasons related to how companies are valued, Murphy believes it’s easier for the consulting and tech firms to acquire a market research company than vice-versa. 

Market research firms are answering these new demands with their own enhancements. It’s a race in which many major market research companies are playing catch-up, but not all. Reynold’s LRW, for instance, carved out an early niche in the consultancy space and is well-positioned to market itself to companies that need strategic guidance. 

“The job of a CMO is more difficult now than it ever was in the history of marketing,” Reynolds says. “The complexity of what they’re dealing with is huge, so they need partners that can deal with that complexity.”

In Reynolds’ conversations with high-placed marketers, he says it becomes clear that there’s too much data and not enough insight. “Sadly, the market research business pretty much sat still from 1980 to 2000,” Reynolds says. “Data-[mining companies and techniques are] growing exponentially; [market research firms are] growing at inflation or less. There’s a sign of some of the dissatisfaction.” The way marketers express their dissatisfaction is not by bemoaning market researchers, but rather by looking elsewhere for partners to help them grow. 

Much of traditional market research isn’t formulated to provide instant feedback. Though no researcher wants to push out unvetted information, the length of time needed to digest market data is longer than the shelf life of much of its relevance. This conflicts with how much of modern marketing operates, which is responsive in real time. 

LRW’s consultancy roots date back to the ‘90s, when it became one of three firms chosen to partner with McKinsey to conduct broad-based engagement surveys to better understand the structure of a marketplace. Nowadays, LRW is looking more like a McKinsey competitor than a partner, however. Reynolds says LRW offers 13 different flavors of analytics to perform segmentation, as well as many methodologies to assess client and competitor brands for untapped opportunities. 

LRW is one of the largest research companies in the world. Its annual revenue of $200 million places it among the top five private-owned research firms. A hefty amount of investor-backed capital allowed the company to embark on a string of acquisitions in 2016. That year, LRW gobbled up a trio of niche research providers: Sentient Services, maker of the community insight platform icanmakeitbetter; social media insights company MotiveQuest; and online community specialist Tonic Insight. The buying spree was put on hiatus in 2017 but returned with alacrity this year. In February, LRW acquired customer experience and culture design firm Strativity Group and struck again two months later, announcing the purchase of consumer insights specialist Kelton Global.

Reynolds says he sold investors on the ambitious growth plan by presenting a strategy that seizes on the disruption roiling the research industry by piecing together several different components, including consulting services.

“What we’re really trying to be is a data-agnostic consultant that has a full suite of skills to be able to go to clients, hear their problems and give them answers that are going to be driven by data,” Reynolds says. 

The pivot to consultancy services is also happening at fellow research giant Kantar. The company, which comprises 12 specialty brands, announced in January it would fold four of them into a specialist sales and marketing arm called Kantar Consulting. Questions abound over the future of the research outfit, as its parent company, WPP, copes with the departure of founder Martin Sorrell. London investment bank Liberum Capital Limited has gone so far as to recommend selling Kantar as a way to pay WPP’s debt and jettison a unit that is out of step with the rest of the organization. In an April statement, the bank wrote, “One outcome of the investigation may be to bring closer a sale of WPP’s data management (i.e., market research) unit, which Sir Martin has been a keen supporter of keeping, but where there has been questioning of why WPP retains the asset, given the issues in the market research area.”

J. Walker Smith, head of Kantar Futures, one of the brands housed under the new consultancy, views the demand for actionable strategic advice as a symptom of shifting client needs, not necessarily a response to overly narrow data sets compiled by researchers. Marketplace priorities have shifted over the past several decades to worship growth above all other metrics, he says. A consequence of this new objective is pressure on advisory services such as market research to deliver end-to-end solutions focused not just on identifying growth opportunities but developing action plans. Not the traditional wheelhouse of math jockeys lacking MBAs.

Kantar Consulting is WPP’s response to client demands to drive growth. “Our single-minded focus is around growth,” Smith says. “Research companies of all sorts are being asked to deliver marketplace solutions for clients. Not just research results, but actual marketplace solutions.”

The new multidimensional intelligence offerings are on display in some of the work Kantar Consulting is performing. Its website crows about developing Land Rover’s global master-brand positioning and business strategy prior to the automaker’s launch of the new Range Rover and Range Rover Sport. The “BrandZ Top 100 Most Valuable Global Brands” study released by Kantar Millward Brown reflects some results of the efforts. Land Rover’s brand value surged 17% between 2016 and 2017, more than any other top 10 auto brand besides Tesla. Other case studies include helping a global scent and flavor manufacturer identify factors that will dominate food and beverage trends over the next decade and a national homebuilder that wanted to know how to develop powerful hooks to place in customized messages pushed out to highly targeted consumers.

These forecasts are far afield from focus groups and survey distribution, market research’s traditional products. They point toward an elevated form of market research. But these vaguely successful case studies trumpeted by Kantar are not the consistent, overwhelming success needed to push back the encroachment of established consulting firms. 

As Murphy says, “There is certainly an emerging category of research companies that look more like consultancies, they just have an uphill climb to fight a big consultancy.” Even Smith is quick to throw cold water on how far the broadened focus can get brands. Yes, he says, the advice might be more strategic, but at the end of the day, it’s no more than problem-solving, limited to a specific set of circumstances. 

“I do think we get too enamored of the notion that if we could just figure out how to analyze this bit of data, we would unlock the key to the meaning of life,” Smith says. “The challenge in marketing is that as soon as you figure out how to solve a problem, everybody knows how to solve it. …  We’re never going to find a kind of data or research methodology that is going to create a permanent competitive advantage in the marketplace.”

The Role of AI and Automation

As market forces debate and refine the upper levels of what market researchers should provide, the lower and mid-levels are refashioned by advances in technology. AI and automation, with the same processing power that is remaking every facet of business from manufacturing to sales, can also be applied to quantitative data collection and analysis.

Sixty percent of respondents in the GRIT Report say that their company is adopting automation. However, even as the industry hands off increasing amounts of work to AI, the true revolution is on hold until AI’s next advancement.

“A lot of people are saying AI is just rules-based automation, but the true AI is starting to take effect as well, and I anticipate it being a major trend over the next few years,” Murphy says. 

“If you play the idea of artificial intelligence out to its nth degree, you can say, ‘I don’t really need to ask people questions anymore because I can just take all this Big Data and apply the tools of artificial intelligence, which gives us predictive analytics to determine patterns in people’s behavior,” says Michael Brereton, a professor of marketing at Michigan State University and former CEO of Maritz Research. 

The change is equally troubling to many in market research, for parallel reasons: It’s seen as a threat to job security. “There’s a resistance to that idea that if we keep giving up the tasks to automation AI, eventually those tasks will continue to move upstream, and at some point, what’s left for the human?” Murphy says. 

But like consultant competitors, AI is a challenge to market research that can no longer be ignored. And of the two issues, it should be the one market research organizations address first. AI wouldn’t be hurtling toward critical mass in a host of industries if it weren’t beneficial to companies across the business intelligence spectrum. And viewed through a different lens, the tech issues that scare marketers can be spun as net-positives for the market research industry.

One market research firm leaning into bots and dashboards instead of consultancies is Zappi. Formerly known as ZappiStore, Zappi was founded on the idea of providing quick, cheap and easy access to market research automation. It employs a staff of 220, grossed roughly $30 million in revenue last year and is looking to grow 70% by the end of 2018. Zappi is the brainchild of Stephen Phillips, a lifelong qualitative researcher and serial entrepreneur. Another of Phillips’ companies is the aforementioned Tonic Insight, which became part of LRW in 2016.

“The business models in market research are broken,” says Ryan Barry​, chief revenue officer at Zappi. “That’s the real problem. There are tried and true ways of getting consumer feedback that are statistically validated to performance. There’re a lot of boutique-y innovations in the research space, but the macro-level issue is: Marketing is programmatic and insights are project-based. The two haven’t been copacetic for a very long time.”

Zappi’s marketing reflects this disparaging view of the market research industry as a whole. Its cheerfully framed animated video introducing Zappi to the world nevertheless goes negative when discussing traditional market research. 

“Until now, market research has failed to keep pace, spending valuable time and money on writing questionnaires, recruiting samples and building reports,” its British-accented narrator intones. “Our process is simple: You choose your tool from our extensive selection, define your sample by selecting your audience demographic, upload your ideas and we deliver you results in hours, not weeks.”

The video highlights the business propositions behind Zappi: to become the App Store of market research, where all the functions of research can be selected and manipulated from a single dashboard. Zappi has brought to market automated tools for data collection, statistical modeling, analysis, visualization and reporting. It’s a blueprint of what AI-assisted market research looks like now, but Zappi may be the first to show what phase two of AI-assisted market research looks like as well. 

“The irony is that we only found our true product market fit a year ago,” Barry says. “The first step of that process, when we were in that App-Store mindset, was to automate the process. … As we’ve moved forward, we figured out there’re bigger problems we can solve.”

Zappi proposes that research tends to be very siloed due to heavy levels of manual input required to crunch numbers. Linking standalone data points to build an all-encompassing marketing model is the dream of every CMO, but the reality falls short as spreadsheets and PowerPoint slides recede deeper into the inboxes of individual employees like isolated puzzle pieces. 

Zappi’s focus now is on managing that research, and harnessing the collective knowledge revealed by those legacy learnings. Powered by an intelligence platform Zappi acquired last year in a merger, the new AI product reads reports with a semantic understanding and a computation engine sophisticated enough to overlay opinion on top of modeling. It’s computer consulting, or artificial intelligence.

“It allows the researcher to say, for example, ‘OK if the brand score is X, and the purchase intent score is Y, as a new brand, we should spend more money on organic sales promotion,’” Barry says. 

This new series of tools allows clients to answer all the business questions they encounter when bringing new products to market. Already, it has garnered interest from PepsiCo, which has partnered with Zappi to create an even more powerful set of digitized, automated tools that promises smarter automation informed by broad institutional knowledge.

“Every time [Pepsi] learns something, they teach it back to the system, and the system just gets smarter and smarter and smarter, and they get to a place where they’re starting to teach muscle memory to the business to look at the platform first and then test and learn,” Barry says.

The Human Element

The market research industry finds itself at a critical juncture, with the acute understanding that progress requires adaptation. As research leaders engage in unprecedented broadsides with alien business ventures, the statisticians find their very humanity a hindrance against new machines and software. Whether market research companies can act on the changes to save their place in marketing’s hierarchy remains to be seen. But as market research moves forward, its operatives should proceed with a willingness to shed the aspects of their profession that have become millstones. Along with retrofitting their offerings to include higher-level business decisions, they must also refocus on their core charge, what Murphy calls being the “keepers of the why.” Automation of analysis can give easy answers to who, what, where and how, but often misses the creative context or the emotion that answers the why of human behavior, which remains the key question for most marketers.