A Hole in the ML Talent Market

We are Physics Majors

True story: My partner Thomas and I are both Physicist-turned marketers, full-stack engineers, storytellers, and designers. A Physics background brings with it a penchant for solving difficult problems, and enough math and science to delve into any specialty. It is the ultimate generalist degree, which lends itself well to the “many hats” requirement of most jobs in the start-up world.

Our company, Standard Data, has both a product side and a machine learning consulting side. On both halves, we’ve seen a great hole in the current labor market; Namely, data science and software engineering should be disciplines that go together, and they almost never are.

Where are all the Polymaths? A Hole in the Labor Market

What we have found is there’s usually a great divide between data scientists and software engineers, they are usually very different people, very rare to find the unified skill set in 1 engineer; When both of these disciplines come in 1, magic happens.

Machine Learning is Software, as Software is Machine Learning. Data has to be processed, models have to be trained on supercomputers and deployed to production environments…they are inextricably linked: 2 sides to the same coin.

However, as mentioned, few possesses this unified skill set, and the industry has paid dearly for it in my opinion, in a few different ways:

1. Extremely fragmented data pipelines.
2. Confusing repository structure for ML projects.
3. Difficult-to-use training and deployment workflows.
4. Little to no industry standards.
5. No really good, widely adopted versioning tools for ML projects
6. Communication between data scientists and software engineers is lackluster.

Machine learning, as a piece of software, deserves the quality of tools and community that coders have: That is why we need more people that understand both, for both product synthesis and technology development.

Our company is more product-focused, so let me tell you why polymathic (data science + software) builders are the best at creating AI products. I’ll first tell you the difference between building tech and building product.

The Difference Between Product Companies and Tech Companies

‍
Product companies are integrators: They take lots of pieces of tech that other people have made and synthesize new solutions for end customers; Moreover, they usually don’t invent new tech, they invent new and innovative solutions (i.e. Apple is arguably a product company rather than a tech company).

Tech companies are more focused on advancing technology itself, not usually with the end-customer in mind. They are more upstream from product companies, since they are creating parts to be used by others (downstream) in order for new goods and services to be delivered to end users.

For product companies, the tech is a means to an end. For tech companies, the tech is an end in and of itself.

You think you need a PhD?

‍
Not if the goal is to create products. If anything, it’s a disadvantage.

For building products, it is the synthesis that’s important: It isn’t as feasible to create full product solutions when you are only really good at one thing; Rather, it’s important to be versed in both software and data science disciplines at the same time, only going as deep as necessary.

The most creative and fun part of our business is taking a problem and going: “Is this possible to solve? Where do we get the data? Should we make a synthetic dataset since real data isn’t available? How should the end-user interact with the product? What safety protections do we need in-place to prevent misuse of the AI?”

That is where the generalist capabilities really shine through. We are artists in this way, taking disparate pieces of data, tech, product design, and business, and crystallizing it all into highly performant AI software products that provide value for others.

Our Unsolicited Advice to Industry: We need more Polymaths.

People with any kind of engineering/data science background, add software engineering to your skill-set. Brush up on your front-end and back-end skills, learn how to synthesize full products and use your powers to make extraordinary new AI products: The industry needs you.

‍

Dillon Peterson