• January 31, 2017

It’s Quality, Not Quantity: The Data Scientist Shortage

More with less might be your best strategy.

The sprint of mathematics and computer science graduates toward data science as a future career may bring sighs of relief from those attempting to fill empty data scientist chairs in an industry suffering from a shortage of candidates.

But for Hewlett Packard Enterprise Fellow Joe Hill, a data scientist himself, the influx of candidates is a cause for concern. “I’m less worried about the shortage of talent and more concerned with the quality of the talent that’s flooding in,” says Hill, who earned his Ph.D. in mathematics with an emphasis on statistics from The University of Texas at Austin.

“I remember 35 years ago when people said computer science skills were so scarce that everyone with any kind of technical background was going to end up coding,” he explains. “That proved true, and the same thing is happening with data science. Everybody and their brother wants to get in. But that doesn’t mean that everyone that gets in will be great at it.”

However, Hill says there are specific steps the enterprise can take to not only hire but develop the skillful talent they will need to move into the future.

Professional Sports-esque

The best data scientists can be likened to elite athletes, says Hill. Whether it’s the NFL or the U.S. Women’s National Soccer Team, only the best of the best rise to the top—and only after years of development. There’s a natural filtering process that ensures the most talented people with the widest range and diversity of skills are the ones leading their teams.

According to Hill, that same process isn’t being applied to data science right now. That’s largely due to the complexity of data science, the training that’s required to be great at it, the creative thinking that’s needed to advance the discipline, and the incredible demand for this talent. Those ingredients combine to create an environment in which many organizations are settling for talent that’s good enough.

Unfortunately, Hill says that approach presents more risk than the value it delivers. “If you think about what it takes to automate a car or build complex financial models that dictate our economy, the skill set that’s required is incredible,” Hill says. “Now think about if you had average data scientists working on those problems. This is true in many other disciplines, too. It’s not unique to data science. Do you want people who barely graduated with degrees in engineering to be the ones who are building the bridges that you’re crossing?”

Closing the Gap

There’s no denying that there’s a sizable gap between the demand for data science talent and the current supply of it. A 2015 survey by the MIT Sloan Management Review found that 43 percent of organizations say they lack appropriate analytical skills.

Hill doesn’t believe the quality of the current data science talent supply is good enough to meet all of that demand. Which leaves organizations with one of two options: Hire whomever you can and hope for the best, or focus on the few hires that will really make an impact in your organization.

“There’s a lot of training and development that’s required to become a great data scientist,” Hill explains. “I’d rather have 50 outstanding data scientists than 5,000 marginal ones. Those 50 outstanding hires will have a bigger short- and long-term impact, and they’ll lay the foundation that helps you train and develop the right skills internally.”

Hill says there are some basic competencies that he thinks organizations should be targeting as they hunt for that talent:

  1.      A deep understanding of the various methodologies that can be used for data analysis—and the creativity to develop new theories and methodologies.
  2.      Computational capabilities, including coding experience and, because of the nature of business today, DevOps.
  3.      Experience parsing large data sets to determine what’s really relevant—a skill that’s most often learned on the job at a large organization.
  4.      Domain expertise, because context and relevance are critical to applying the right analyses and methodologies.
  5.      The ability to tell a story when an analysis is complete. It’s one thing to discover something new, but it’s quite another to be able to translate that insight so it can be used by the people who are making decisions.

“I don’t want to make it sound like our discipline is full of imposters,” Hill says. “But the onus is on the organization to be diligent about who it hires. Every company could use more data science talent today. But is the goal to just fill seats or is it to hire and develop people that will drive meaningful change and deliver transformational intelligence?”

For Hill, the risks of hiring the wrong data scientist could cost a lot more than not having enough, and the savvy enterprise will keep that in mind as it recruits from the latest crop of potential employees.

For more on the future of enterprise talent, read “The Millennials Are Coming for Your Workforce.”