Better Features for a Tree-Based Model

When you understand how a model works, it becomes much easier to create successful features. It is because you can reason about the model's strong and weak sides and prepare features accordingly. Let's take a look together at what features can be understood by a tree-based model and what features are harder for it to use (and how can we help the model in such cases).

How a tree-based model uses features

We'll start by taking a closer look at what's inside the tree-based model.

The main building block of tree-based models is a binary decision. It takes the value of a specific feature and make a split depending on it. Many such decisions form a decision tree. And many trees predictions averaged give model predictions. Of course, this is a quite simplified explanation of tree-based models, but it fits very well for understanding what properties are required from successful features.

The number of required splits (or required binary decisions) is an important property of the feature

Given the binary nature, tree-based models are good at handling features where the main signal can be extracted by a few splits. Let's take a look at some features with such property.

In order to better reason about required splits, we'll use "feature vs target" graphs. On the X axis, we have feature values sorted from smallest to largest. And on the Y axis, we have corresponding target values for each feature value. Additionally, we'll use a split line to separate between which target values we would like to consider low and high.

Here are some examples of "feature vs target" graphs for features requiring one or a few splits, which are handled well by tree-based models.

As you can see, tree-based models are good at handling features, where the graph doesn't change the direction often from upward to downward and vice-versa. Features don't need to be linear, the main requirement is that they don't require many split points to separate low and high target values.

Additionally, what is the strong side of tree-based models, is the ability to handle feature interactions. This is when feature A behaved in one way for small feature B values but changes its behavior for big feature B values. Not all models can capture such interaction, but tree-based models handle it very naturally.

Now let's take a look at some features, which are harder for tree-based models to handle.

The more splits are needed for a feature to capture the full signal, the deeper trees are required. But deeper trees with many leaves means a higher risk of over-fitting. Allowing the model to grow deeper trees, doesn't mean it will make splits only on the features we want. It can use splits to unnecessarily divide some other features resulting in capturing noise.

Feature engineering examples

Now, having some intuition on what type of features we want for our tree-based model, let's look at some examples of how to actually transform features to make them better.

Capture repeating pattern

Here we can see a feature with some repeating pattern. One real example of a feature with a similar pattern is a date. In many datasets, there are one or several date features. They can specify the date of registration, date of birth, date of measurement, and so on. One typical property of date features is that sometimes they have a seasonality component. E.g. the target variable can depend on weekday - in weekends target variable behaves differently than in working days.

It will be hard for our model to extract such information. As it will need two splits for each weekend - to separate it from both sides. An additional pitfall is that making splits on exact date values won't help for predicting on unseen data in the future as dates are not repeating.

The best way how to help our model is to extract the pattern to much simpler binary feature "is_weekend", which will have value "1" for weekends and "0" for working days. Or maybe it will be more effective to use a "weekday" feature, having values from 1 to 7 for each weekday. When we see what this means visually for our previous example, we are reducing necessary splits to fully capture the signal in data, thus making the feature more friendly for tree-based models.

Basically, we push all repetitive intervals (e.g. weeks) together and let the model use an average value for making decisions on splits. In the example above this allows us to reduce the number of required splits from 8 to 2. If you wonder, how this is done in code - we simply took x mod 25 here as the new feature value, because the original feature had repeating intervals of length 25.

We can improve features by reducing the number of required splits for the model to capture the signal

Remove a noise

Another situation where a tree-based model might need some help is very noisy data. Consider some feature where the values are very noisy, but in reality, only one point is meaningful. If this point is known to us, we can help the model by making the necessary split ourselves. This can be achieved by transforming raw feature to the binary one.

For example, we are given a number of purchases at a store in different dates and there is a known date when this store was moved to a different location. If we would check the average value before and after the move, the corresponding means would be 25 and 30. But given a high variance with values changing from 0 to 60, it is hard for the model to determine the correct split point.

In this particular example, given that the mean value changed, most likely the model will manage to make a split somewhere near the correct point. However, it may choose some point to one or other side, resulting in worse predictions around this moving point. We can help the model by introducing a new feature specifying this point explicitly - for example, this could be a "new_location" feature, having value 1 for all rows after the move and value 0 before it.

Of course, such transformation requires some domain knowledge and "expert decision" to choose the correct split point. But when this is possible, such a feature can greatly help the model to avoid making unnecessary wrong splits, which could cause over-fitting to noise.

More on what a tree-based model can/can't do

So far we explored how tree-based models are using features and looked at some examples of how to use that knowledge to fix or engineer better features.

Now let's talk about other considerations that can help us when working with tree-based models.

Beware of feature values outside the known interval

One weak spot of tree-based models is an inability to extrapolate. Simply speaking, a tree-based model will typically fail to predict a value that is smaller than the smallest value in training data, or a value that is bigger than the biggest value in the training set.

Consider the feature X values and corresponding target Y values as follows:

• X=1, Y=2

• X=2, Y=3

• X=3, Y=4

• X=4, Y=?

Any simple linear model will capture the linear relationship between X and Y and will guess the last Y value correctly. But tree-based models most likely will predict a value around 4. And not 5, which is obviously the correct value.

To understand the reason, we should recall the way how tree-based models are using features - by making splits on them. And if the model has never seen feature X value bigger than 3, there is no way for it to make splits for bigger values. As a result, all X values bigger than 3 will be treated exactly the same by the tree-based model.

Now, can we help a model to overcome this? In general - no, we can't in any way force a model to make splits outside the known feature interval.

A tree-based model can't make splits for feature values outside the interval seen in training data, therefore it can't distinguish between them.

There actually exist some approaches around this limitation, but those typically involve a transformation of target value - therefore I won't consider them here as in this article I want to focus only on transforming/creating features. I can just give you a clue regarding the direction of those approaches. For example, sometimes it is possible to predict not the actual target value, but the difference from some previous value and at the end reconstruct the real target values by incrementally adding predicted difference from the previous value one by one.

Don't need to scale or normalize

When working with tree-based models, it is not needed to scale or normalize data. Why? Because making split at 0.8 in the interval from 0 to 1 is equally hard (or easy) as making split at e.g. 700 in an interval from -100 to 900.

To see it visually, let's plot the "feature vs target" graph for some random feature with values ranging from 30 to 130. And let's plot the same graph for the same feature, but after normalization to the interval (0,1).

Notice, the only thing changed is the scale of the X-axis. The model will need exactly the same splits as before normalization to separate low and high target values.

Similarly, tree-based models are not sensitive to outliers in features (on the contrary to e.g. linear models).

Conclusions

Despite undeniable advancements of deep learning lately, tree-based models are still very competitive. And if we talk about tabular data, in many (if not most) of the cases, tree-based models with accurate feature engineering can still outperform the deep learning approach, which is proven by many Kaggle competition results. In case of tree-based models, feature engineering is the key to success. But the understanding of models underlying structure and operation is the key to successful feature engineering.

Hope you found something interesting, and useful in this article, and thanks for reading! Follow me not to miss further articles about machine learning.