Is there a data scientist shortage?

A surge in AI deployments among businesses is seeing a boom in demand for data scientists.
18 February 2019

U.S. Chief Data Scientist DJ Patil. Source: AFP

The rapid growth of artificial intelligence (AI), across its many guises and use cases, is seeing a boom in demand for data scientists, according to a recent report by recruitment site Indeed. But supply might not be meeting demand.

According to the author of the report, Andrew Flowers, an economist at Indeed, the data scientist’s role has “only grown sexier”, and more employers than ever are looking to hire personnel with the skills to crunch big numbers.

The report last month found that across the leading jobs sites, there was a 29 percent increase in demand for the role year on year and a 344 percent since 2013. This uptick, however, hasn’t been met searches by job seekers with the right skillets. These have grown by just 14 percent— or 50 percent slower— throughout the year.

Employers now use data scientists to solve all sorts of problems, said Flowers, with these experts generally tasked with taking raw data and using programming, visualization, and statistical modeling to extract insights, according to the Bureau of Labor Statistics (BLS).

So, why in today’s age of a growing digitally-native workforce do these specialists remain somewhat elusive? Well, it’s likely a result of just how much training and coding experience is required.

The report notes that data scientists are typically fluent in one or more programming languages used for statistical analysis, with languages such as Python and R ‘data scientist favorites’. But knowledge is also needed in tools such as Hive, BigQuery, AWS, Spark, and Hadoop.

Besides a formal education in computer science, statistics, or a quantitative social sciences qualification, data scientists will frequently have training in statistical modeling and machine learning.

In essence, the data scientist job mixes rigorous theory and the software craft,” says Flowers.

That hefty, if rare, combination can see a data scientists demand a high salary. In the highest-paying US cities of Houston and San Francisco, these roles can earn employees an average of US$ 138,000 and US$ 167,000 respectively.

Interestingly, Indeed found that searches for data science jobs showed a seasonal pattern. Across 2017 and 2018, searches spiked in February and March, thought to reflect an influx of computer science students searching for internships or graduates looking for their first job, with the role having seen building hype over the last few years.

While employers might face a lack of skilled candidates for the role of the data scientist in the organization, however, they shouldn’t necessarily write off similarly skilled applicants for related positions. Some may be better suited as a data analyst or quantitative analyst and could be trained or develop the experience over time to take a more senior role.

Businesses should also not overlook the importance of communication skills; owing to the need for data scientists to work across functions, purely technical skills won’t put them in best stead.

In actuality, data scientists need to be able to communicate the problems they’re dealing with in clear language; present their analysis to a team who may not understand the subject, and explain results to frequently non-technical staff at the top of the ladder.

If you are one of the 15 percent of enterprises using AI already or part of the third of groups that plan to implement the technology this year, then picking the right people will be vital to your success. According to KPMG’s partner, data & analytics lead, Brad Fisher, however, AI deployment across your business requires “more than just data scientists”.

“It takes a robust team with a range of specialized AI and business skills who can help with every stage of the process,” adds Fisher. “And right now few companies seem to have the right mix needed for AI to bring strategic value to an organization.”