Dynamics of skills demand and job transition opportunities: A machine learning approach

Patricia Prüfer, Pradeep Kumar, and Marcia den Uijl2

15 October 2019

What are consequences of the ongoing digitalization and automatization for the labor market? We analyze to which extent the skill requirements for different types of occupations change over time and how these insights can be used to substantiate demand for reskilling of several groups of employees. To answer these questions, we make use of a novel approach in which we combine unstructured data from the Internet with structured data from labor market forecasts. Based on a dataset of 95% of all job vacancies in the Netherlands 2012-17 with 7.7 million data points, we show which skills are particularly important for which type of profession. Besides, we provide job transition opportunities for employees from shrinking sectors or occupations to those not affected negatively by technological change. Our results suggest that the labor market is undergoing a transition from degree-based to skill-based demand. This has consequences for both the participants and the institutions connected to the labor market. In this paper, we derive insights on the consequences of the ongoing digitalization and automatization for the (Dutch) labor market labor market and for various sectors. What are the dynamics in skill demand on the labor market? How do they affect certain types of professions such as managers, ICT professionals, and employees in non-IT/technical jobs? Will these developments eventually lead to new labor market structures, for example by offsetting clearly separated sectors? To answer these questions, we make use of a novel approach of ‘labor market analytics’ in which information from online vacancies, thus from unstructured (big) Internet data, is combined with information from labor market forecasts, that is, with structured data from We use data from the vacancy database Jobfeed, administered by TextKernel, a tech company. We received data for a period of 6 years, from January 2012 until December 2017, in total about 7.7 million vacancies. In addition, we use information gathered from multiple sources, including the Occupational Information Network (O*NET), an online database with information about the 1 This research received funding of the Dutch Ministry of Social Affairs and Employment under the DWSRA grant and of a consortium consisting of CA-ICT, NL Digital, and CIO Platform Netherlands, representatives of the ICT sector as well as four of the so-called Dutch ‘top sectors’: Chemical Industry, Energy sector, High-tech Industries, and Life Sciences and Health. 2 All authors: CentERdata, P.O. Box 90153, 5000 LE Tilburg, The Netherlands. Corresponding author: Patricia Prüfer (p.prufer@uvt.nl). We are grateful to conference participants at The Netherlands Economists Day 2018, the ROA Workshop on Dynamics of Skills Supply and Demand, and at SIOE 2019 for valuable comments. All errors are our own. knowledge, skills, tasks, training and experience required for a large number of occupations. Another data source is ISCO (International Standard Classification of Occupations; version ISCO- 2008), a classification of 436 professions supplied by the International Labor Organization (ILO). Other sources for skills data we used include the EU skills framework, Stackoverflow, and Dbpedia, Wikipedia’s skills database. Additionally, we use data on labor market projections on labor market developments by economic sector, occupational group, and type of education from the Research Center for Education and the Labor Market (ROA). Methods As the collected vacancies from the Internet consist of unstructured text, we apply Natural Language Processing (NLP) techniques, more specifically the bag-of-words algorithm. We calculate an index of similarity, or ‘similarity scores’ between any job pair based on the information about the required skills, experience, training, knowledge and education, the profile, for a specific profession. These similarity scores are used to objectively measure the similarity between each pair of our 371 unique job types and, thus, to identify the job-fit between all jobs in our dataset. Figure 1 (below) depicts the job-fit matrix for the Netherlands. Where a zone is highlighted in dark blue, the corresponding row and column define two occupations with a combined profile that suggests a high degree of job-fit. By themselves, similarity scores provide a useful tool for a systematic and comprehensive comparison of job-fit and for identifying viable job transition options. However, as with any composite index, the scores provide a highly aggregated summary view of the theoretical viability of any given job transition. Additional filter criteria are needed to ensure that the job-fit indicated by the aggregate similarity score stays realistic. Therefore, we apply three constraints to our linear optimization. Figure 1: Job transition matrix between 371 jobs in the Netherlands Findings Our analyses show that digital and technical skills are becoming increasingly important: the overall demand in vacancies in which these types of skills are requested increases. Moreover, we see an increase in the total demand for these skills, as more and more digital and technical skills are required per profession. Professions are, thus, become more and more technical. Regarding occupations dynamics and how certain types of professions are affected by technological change, we find suitable transition possibilities for a large majority of shrinking occupations. In the scenario where we loosen this assumption of the same salary range, we find for 91% of all the employees a viable transition possibility. From an individual perspective of a single employee affected negatively by digitalization and automation, we observe on average 24 options to switch from one profession to another one, whereby 11 of these possibilities come with and 13 transition with equal or less salary. These positive findings apply to both women and men. Based on these results, we can provide job transition opportunities for employees from shrinking sector or occupations to sectors and professions not affected negatively by technological change. Figure 2 shows an example of pathways for secretaries also indicating whether the transition is accompanied by a salary increase or decrease. Figure 2: Example of pathways for secretaries Secretaries (general) Hourly wage: € 18 17 professional transitions with salary increase Debt-collectors Hourly wage: €17 Score: 0,89 Legal and notarial assistants and bailiffs Hourly wage: €19 Score: 0,88 Legal and notarial assistants and bailiffs Hourly wage: €19 Score: 0,88 Legal and notarial assistants and bailiffs Hourly wage: €19 Score: 0,88 12 professional transitions with salary decrease ICT user support technicians and call center employees Hourly wage: €21 Score: 0,88 General office clerks Hourly wage: € 17 Score: 0,92 Clearing and forwarding agents Hourly wage: €27 Score: 0,87 In summary, this research leads to positive outcomes with regard to the impact of digitization and automation demonstrating necessary opportunities for maintaining sustainable employability and economic growth options. By combining unstructured ‘big data’ from online vacancies with structured information from labor market forecasts, we are able to distinguish not only the effects of general trends on the labor market, but we also show the impact on career prospects of employees in many different sectors and professions. Thereby, we can provide clear-cut suggestions on viable and preferable job transitions for employees in shrinking sectors and/or professions.