The most common mistake I encounter in practical data science work is a certain type of unwarranted complexity in model choice and deployment. All organizations, but especially those that are not technology companies, should avoid this if they can. As Dijkstra said in the 70s: “The art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible.“

Let’s admit it: when it comes to data science, many organizations have underwhelming track records. Startups and FAANG-like companies might be different, but in the typical organization, initiatives tend to become messy. Data science is not unique in this regard: the average corporate codebase, data warehouse, internal documentation, etc. is often not that great either.

Still, data science is especially plagued by complexity because it sits on top of, and depends on, other systems. Bad design, conceptual degradation, backwards-incompatible changes, and lack of documentation tends to accumulate and cause problems. The paper Machine Learning: The High Interest Credit Card of Technical Debt lists some concrete reasons why data science is hard in practice.

So how should an organization approach its first data science initiatives?

Avoid complexity

Organizations often trip up because they overcomplicate matters; they focus on the wrong things at the wrong time. Some inexperienced organizations immediately go the “best practice” route: multiple deployment environments, elaborate monitoring, automatic re-training, API-endpoints, clusters, etc. Often this is not needed.

The same faulty thinking tends to afflict the models too. On regression tasks, boosting models with thousands of base learners is the default choice. Hyperparameters (which are often in conflict with each other or are slight variations of the same basic idea) are all at once tuned uncritically by fancy Bayesian optimization algorithms that take ages to run. Dozens or hundreds of input features are used, whether they are meaningfully predictive or not.

The resulting model is slow to train, hard to explain and difficult to maintain. This leads to mountains of busywork for data scientists.

Aim for simplicity

The average organization should choose a ridiculously simple model as the starting point. On regression tasks, this means a linear model or a decision tree. Then it should be “deployed” in the dumbest way possible. If a batch run on a weekly schedule is good enough, then just run it manually from a laptop. If real-time or a nightly run is needed, convert the model to an SQL statement. Follow “good enough” practices, but do nothing more. Then give it a few months and see if it delivers any value to the organization. This is commandment 5 (Keep it simple) in my ten commandments of data science.

Let’s illustrate this with the California housing dataset. The goal is to use eight features to predict house prices. For reference, a dummy model that predicts the mean achieves an out-of-sample Root Mean Squared Error (RMSE) of 1.17, while a boosting model (widely considered the best choice on tabular data) achieves 0.64. These models give realistic upper and lower bounds on the RMSE.

Linear regression

Linear regression achieves an RMSE of 0.83, which is two thirds of the way between the dummy and the boosting model. The Lasso variant selects four out of eight features. The resulting model can be deployed anywhere, for instance as an SQL expression in a database:

SELECT
  ((0.393414 * MedInc) +
   (0.016651 * HouseAge) +
   (-0.003222 * AveOccup) +
   (-0.021127 * Latitude) +
   0.797172) AS predicted_price
FROM california_housing_data;

This model is super interpretable, so the business side of the organization will be delighted when they inevitably wish to understand the model. Predictive power can be improved by adding non-linear transformations of the features and interaction terms, but we won’t pursue that here.

Decision trees

Do you need interactions and/or non-linearities? But you still want an interpretable model that can be deployed anywhere? Use a simple decision tree! The tree below achieves an RMSE of 0.84, which is very similar to the linear model’s RMSE, and it only uses two features. Here is the decision tree written in SQL:

SELECT
CASE WHEN MedInc <= 5.04 THEN
  CASE WHEN MedInc <= 3.07 THEN
    1.36
  ELSE
    CASE WHEN AveOccup <= 2.37 THEN
      2.79
    ELSE
      1.88
    END
  END
ELSE
  CASE WHEN MedInc <= 6.82 THEN
    CASE WHEN AveOccup <= 2.74 THEN
      3.39
    ELSE
      2.56
    END
  ELSE
    4.22
  END
END
AS predicted_price
FROM california_housing_data;

Like the regression model, the tree is extremely interpretable. Below it is shown as a figure.

Two approaches to improvement

Broadly speaking there are two approaches to improving any situation. One is to start from an abstract (and perhaps unattainable) ideal and work backwards, e.g. ask yourself “How does Google do data science?” and then try to emulate it. The other approach is to start where you are, assess your resources and skills honestly, and ask “What is a small step I can take to improve?” Both methods have their place, but the second one is often more concrete and less risky. Practical project experience helps a lot in balancing the two. Many data science projects fail because complexity is “spent” on the wrong things.

Sometimes a simple model with a dumb deployment is not only a good first step, but the best place to end up too. It’s fast to build, requires little upkeep, is interpretable and has few data dependencies. On many problems performance will match a complex approach. It might be unglamorous and unfashionable, but it will quietly keep working—and boy can it save you money and time.

We’ll end with another Dijkstra quote, my personal favorite: “Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.“