What I Learned in My First Year as a Director of Data Science | by CJ Sullivan | May, 2023

The continued and constant learnings on managing data science teams outside of tech

Six months ago I noticed that there were not many posts out there about how to manage a data science team (versus data science projects, a topic that has considerably more content). In an attempt to address that gap, I wrote my original post: “What I Learned in my First Six Months as a Director of Data Science.” The response was largely positive to that post, so I decided that, now that I have been in the role of Director of Data Science for 1 year, I would update the original post based on the next 6 months of my experience. Some of the learnings are unchanged, although I have observed new things about them or have learned more details about them. Others have changed in the evolving layoff situations in the tech world. For more background on myself and why I started writing about these things, please see my original article.

When I wrote the original post in November of 2022, the tech layoffs had just started picking up steam. I started to feel very optimistic about there being a change in the hiring market. My management started feeling optimistic that there would be a change in the salary expectations and offers for data scientists as a result.

Simply put, that didn’t pan out.

I joined several online forums established to help those laid off in tech find new jobs. However, I quickly noticed that the data scientists and machine learning engineers were not really there in great numbers. I also noted that my friends working in the field at these companies were not being laid off. My (anecdotal) conclusion to this is that data scientists still have a fair bit of job security in the tech world, including with the FAANG companies. Those that I have found that have been laid off have been quickly scooped up by other tech companies. So in this regard, my hiring situation has not changed in the past 6 months even though the tech world has.

That being said, I have gained 6 more months of experience and observations on hiring data scientists. And there are some of these that I think would be valuable to share.

The interview design: to code or not to code?

This is a long-debated question within the software industry. We are hiring people to write code, so don’t we want to see their code? Would we like to give them assignment similar to what our day-to-day work looks like at our company? Would we like to see them code live?

This is something I have wrangled with in my current role. I do value seeing how people code. I look at the code and its overall organization. Does the candidate use well-crafted classes? Are the functions clear and include docstrings, type hinting, and other helpful debugging things? Is there some sort of testing implemented? What does the overall documentation look like? Did they just copy and paste from the scikit-learn docs or did they put some thought into hyperparameters and going beyond the doc’s example problems?

Yes, this is absolutely something that I could look at in either a take-home coding challenge or a real-time coding exercise. However, are these fair evaluations of someone’s abilities? I was subjected to both as an IC and there are pros and cons to each. These have been discussed in a variety of settings, but I will just briefly summarize them.

Take-home coding challenges or homework assignments assume that the candidate has time available to do them. It does not consider that they might still be working in their “day job” from 9–5 and that they might have family or other obligations that do not allow for the time it would take to adequately address the challenge. So you could be biasing against, say, working parents who have child care duties in their off hours.

Sure, you could do a live-coding session such as pair programming or the dreaded “we will watch you code” or whiteboarding exercises. After all, most people, including those who are interviewing while still working at another job, will typically take an hour off here and there to do interviews. But what about that person whose activities are monitored on their company-owned computer who does not own another computer? They will likely not want their boss seeing them working on a coding challenge on the company’s dime and resources. Further, many people do not do their best work in such a high-pressure situation. And really, is that the type of environment they would be working in on a daily basis? (Side note: if these high-pressure situations are how your team normally operates, you should maybe examine the culture that you have created for your team.)

At the end of the day, we do not do either on my team. We still do technical interviews, but these are discussions rather than coding.

The importance of a portfolio

Despite the above, I would still like to see some code or analysis written by the candidate. And it might be a contradiction to the above. I am not needing a personal webpage with glorious formatting and things deployed to Heroku. A GitHub account with a couple of well thought-out projects is sufficient. Your GitHub account with this code should be provided on your resume. These should not be clones of someone else’s repository. (I am always amused when I get applications from candidates that have cloned my repos.)

Again, this is hard to assemble when you are a working parent with limited spare time or when you don’t have access to a computer outside of your work computer. This system is not perfect. However, one (albeit minor) benefit of looking at a candidate’s code this way is that there is no time pressure for the candidate to put it together. (Although it is a good idea to maintain it with the latest packages, because you never know when I might try to run your code!)

Reading the job ad is important!

I talked about this subject in my previous post, but I have had several learnings based on the volume of applicants and interviews since then.

On my team we have two different types of people: data scientists and machine learning engineers. These job titles were created before I joined the company and, for the sake of consistency, I have endeavored to maintain them. However, if you work in the field you know that these can be very overloaded terms.

To attempt to distinguish between the two, I have been very clear in the ads (if I do say so myself!) on the responsibilities of each. Again, in keeping with the historical use of them on this team, what the data scientists on the team do is experimentation and model creation, typically in notebooks and maybe some basic Python scripts. The machine learning engineers take those models and put them into production code and pipelines. In some companies, these might be called “data engineers” or “MLOps.” In other companies, the machine learning engineers are the ones writing the models.

I have learned over the past year that it is very important when attempting to hire people while using these ambiguous titles to really specify the day-to-day responsibilities. The more specific I have been on the job ads, the better the matches have been.

And yet it is very far from a perfect solution. I am constantly amazed by the number of (what I would call) data scientists applying for roles as (what I would call) machine learning engineers. They have not read the job ads. In the data science ads I have specified experience in statistics, modeling (traditional ML, deep learning, etc.), exploratory data analysis, etc. For the machine learning ads I have called out things like CI/CD, devops, Terraform, etc. Maybe I have still written the wrong job ads. Based on the numbers of data scientists applying for the machine learning roles (I would estimate it is greater than two thirds), the problem must be with me. It must be, right?

It goes beyond just the job descriptions though. Certain states including my own have laws in place requiring hiring managers to disclose the pay range for the role. As we have discussed, tech companies typically can pay better than non-tech companies, and I work for the later. That is not to say that our problems are not exciting or worthwhile! It is just the nature of the business. And yet the number of applicants I get who state that their desired minimum salary is completely beyond what I have advertised is significant.

As a hiring manager, given the above I can only conclude that the candidate has not read the job ad. That is unfortunate for both the candidate and the hiring manager.

Let’s face it. We all have to sometimes sit at multiple roles at a company. However, if you have worked both at a tech and a non-tech company in your career, you know that this takes on much more meaning outside of the tech world.

In the tech world, you frequently have the opportunity to really specialize in a particular discipline. For example, when I worked for one employer as a machine learning engineer (which, for what it is worth, meant “data scientist” in my current company), I started getting very involved in graph data science. It both interested me on a technical level and helped solve some challenges for the company. If you have followed my other Medium posts, you have probably noticed that they revolve around graphs to the point that I went to work for a time at a graph database company.

That model of specializing is much more difficult to do as a data scientist outside of tech. The difference stems from the fact that outside of tech your job is to solve business problems, not create the technology to solve those problems. As such, you need to be a generalist, not a specialist.

This goes beyond just knowing which Python packages you will need and employing a diverse set to solve said problems. For example, my team is very well versed in cloud infrastructure. This is out of necessity. We don’t have an army of infrastructure folks ready to help us do this. Frequently we are advising the company on what that infrastructure looks like. We have had to teach ourselves infosec. Is that great or ideal? No. But it is how the human resourcing at non-tech companies works.

In many ways that is actually a lot of fun! If you are working in a generalist role, you must be constantly learning new things. I don’t expect everyone on my team will understand absolutely everything about how to create a production pipeline for our models, for example. But everyone is learning. I love learning new things. I think this is a requirement for a data scientist.

Managing non-infinite budgets

I have a story here. When I worked in a tech company that shall remain nameless, we were using a certain very popular cloud provider. A colleague on my team decided to test out whether they were allowed to spin up the absolute fanciest GPU instance the provider offered, just for the sake of seeing if there was a limit for us. They left that instance running for more than a month without using it to calculate a thing. The total bill was tens of thousands of dollars. Nobody cared.

One of my responsibilities in my current role is to monitor budgets, particularly for our cloud infrastructure as well as our CI/CD pipelines, which I do on a weekly basis. We have a monthly budget (one that is significantly below what was spent on my previous story). When the budget was set it was quite realistic so we don’t usually have to worry about going over it, but it is there and we monitor for it. Sometimes we have people who don’t quite know how to manage CI/CD runner minutes, for example. When I see usage spike and put us in danger of going over, I ask the more senior members of the team to work with those working on the offending projects to make them more resource efficient. Like I said above, everyone is constantly learning.

While this goes for our compute, it also goes for salaries. I mentioned before that the salaries in tech tend to be higher than in non-tech. It might seem like I should just be able to go to my management and get more money for salaries. Unfortunately, this is not the way businesses work. One of my jobs is to fight for my team and the best salaries I can get for them. One of their jobs is to keep salaries across the company fairly consistent and within the larger budget picture. Frequently (always?) one of those it at odds with another. When you consider that data scientists are frequently the highest-paid ICs in a company, it is a hard sell for me to go to senior management and ask for even more money for salaries. This is the nature of the beast.

Communicating with non-technical teams

This is one of the biggest challenges at a non-tech company. My team must generate solutions for the business. How is the quality of those solutions measured? One way is return on investment (ROI). Businesses exist to make money. That is obvious. Everyone must contribute to that mission. Models that do not contribute to that are models that never should have been developed.

But who is on the receiving end of those models? In my case, it is usually marketing. I will be the first to admit that I am not a marketer. (Although following my statements above about continuous learning, I am going back to school to get an MBA so I can learn about how marketing works.) Marketers are highly technical in a discipline that is not data science. So it is very important in my current role that I learn how to communicate with them.

That might seem obvious, but what does it really mean? For starters, it means that I cannot just throw a bunch of math and plots over the fence, and I especially cannot do that without explanations of what it means. I have presented at many data science conferences in my career and there are things that I can say there that I wouldn’t have to think twice about explaining. A great example of this are the variety of performance metrics used to evaluate models. People in marketing roles typically understand a term like “accuracy.” But mentioning “precision/recall,” “F1 scores,” or any other metrics that we consider standard will result in glossed over stares from them. Frankly, that is just fine and we shouldn’t expect them to go and get a degree in statistics to understand. We might think they should care, but at the end of the day they are looking to us to deliver a good model.

This importance of trust

A real hard part of my job though is to deliver to our marketing friends the news on when the model is not good. Hopefully it is something that the team has discovered on their own. It is a bad day when the team missed it and they found it.

The thing is that they are not in the weeds on a daily basis looking at what data we have (or, more often, don’t have). Watching models drift is not something they should worry about. Ultimately, they have to trust that the data science team to do it.

But how is that trust gained?

Communication is key! In the ideal case, each model the team develops was based off of a business problem that had been extensively discussed before data science hands ever touch a keyboard. I work very hard to get the stakeholders together and really spell out the business need. This typically cannot be achieved in a single meeting. This is done over many meetings. Sometimes the same meeting has to be repeated multiple times to catch the stakeholders whose calendars didn’t allow for them to attend the original meeting. And then many more meetings should happen to continue to define the problem and what the solution should look like.

If communication is key then documentation is the keychain. I try to end every meeting with a summary of what was decided. This gets written up and put in a common location accessible by all. That location is shared out every meeting. I ask for feedback on the documentation after every meeting. Did I get it right? Are there any edits, comments, or questions? (Pro tip: email is not a great way to document these discussions. It needs to be some place more easily discoverable.)

And yet it still happens that there might be a model that is not performant. Maybe it has a bug that wasn’t caught. Maybe the data doesn’t exist to create a model. Maybe it is discovered by the team or maybe it is discovered by the marketing folks. Regardless, trust is established through honesty. This seems like it should be obvious, but I have seen too many instances in my career where a data scientist has quietly swept something under the rug because it doesn’t put the team or the model in the best light.

Marketers are typically not statisticians. When something breaks or they don’t understand it, it is very easy to blame the math (or those who created it). And let’s face it. Machine learning math can easily be seen as a black box and very hard to accept or believe. When there is an honest, open, trusting relationship, it is harder for a lack of trust in the team or the models to bring things down.

If my first 6 months in this role were about adapting to being outside of tech and learning the state of hiring in data science, my second 6 months have been about my attempt to combat the challenges associated with each. There are still challenges and I am still learning. How do I get more quality candidates through the door? How do I work within the boundaries of a non-tech company to get the best for my team, the resources we need to do our jobs, and identify the best places where to spend the team’s time?

I continue to learn every day.