AI glossary
Feature Engineering
Feature engineering is the process of using domain knowledge to create, select, and transform the input variables (features) fed into a machine learning model, so the model can more easily capture patterns relevant to the prediction task.
Raw data rarely arrives in a format that algorithms can use immediately. A timestamp is just a string until you extract the day of the week. A category like “customer_type” is meaningless to a linear regression until you convert it into a numerical format. This gap between raw observation and model-ready input is where the work happens.
Common techniques
Practitioners use a variety of methods to turn messy inputs into predictive signals. These methods generally fall into a few distinct categories.
Creating derived features involves generating new variables from existing data. For example, you might extract the hour of the day from a timestamp to capture time-based patterns, or compute a ratio between two numeric columns to highlight relative performance.
Encoding categorical variables converts non-numeric categories into a form models can process. One-hot encoding creates a separate binary column for each category, while label encoding assigns a specific integer to each class. The choice between them often depends on the model type and the number of unique categories.
Scaling and normalization adjust numeric features to a common range. Standardization subtracts the mean and divides by the standard deviation, while min-max scaling rescales values to a fixed range, typically between 0 and 1. This ensures that large values do not dominate the learning process.
Handling missing values is critical because most algorithms cannot process null entries. Imputation techniques fill these gaps using the column’s mean, median, or a specific placeholder value, allowing the model to process incomplete records without discarding them.
Binning converts continuous variables into discrete buckets or ranges. This can help models capture non-linear relationships by grouping similar values together, though it does result in some loss of precision.
Interaction and polynomial features create new variables by combining existing ones. Multiplying two features or raising one to a power can expose complex relationships that a model would not otherwise capture directly.
Dimensionality reduction reduces the number of features while preserving useful information. Techniques like Principal component analysis transform correlated variables into a smaller set of uncorrelated ones, making training faster and reducing the risk of overfitting.
A worked example
Consider raw e-commerce order data containing a purchase timestamp and an order total. A naive approach might feed these values directly into a model. However, engineered features reveal deeper insights.
You could extract the day of the week and the hour of the purchase to capture shopping-pattern effects. You might calculate the number of days since the customer’s previous order to measure purchase frequency. Additionally, computing the ratio of the current order total to the customer’s average order total highlights whether this specific transaction is unusually large or small compared to their history.
These transformations turn simple numbers into rich signals that reflect customer behavior.
Feature engineering vs feature learning
It is important to distinguish between manual design and automated discovery. Feature engineering is done by a person applying domain knowledge to manually design or transform input variables before training begins. You decide what matters.
In contrast, feature learning (also called representation learning) occurs when the model itself learns useful features directly from raw data during training. This happens without a person hand-designing them. This is a defining strength of deep learning, where layers of a neural network learn increasingly abstract representations automatically.
Deep learning has reduced, but not eliminated, the need for manual work. It is especially effective on raw, unstructured data such as images or text, where hand-designed features are hard to specify. However, manual feature engineering still commonly improves performance on structured, tabular data used with models such as XGBoost.
Why it still matters
Even with powerful algorithms, well-engineered features frequently produce a bigger performance improvement than switching to a more complex model. This is because they directly encode domain knowledge that the model would otherwise have to infer from limited training data.
Good features can make a simpler, faster, and more interpretable model perform as well as or better than a more complex one. This matters significantly for cost, latency, and explainability in production systems.
Furthermore, feature engineering plays a key role in reducing overfitting. Fewer, more relevant, and well-constructed features can generalize better than many raw or redundant ones. This often leads to more stable models that perform consistently on unseen data.
FAQ
What is the difference between feature engineering and feature selection?
Feature engineering creates new features from raw data, while feature selection chooses the most relevant existing features from a larger set. Both aim to improve model performance but approach the problem from different angles.
Do I need feature engineering for deep learning models?
Not always. Deep learning models can learn features automatically from raw data, especially in unstructured domains like images or text. However, engineering features can still boost performance on structured data.
What is the most common feature engineering technique?
Encoding categorical variables and handling missing values are among the most common steps. They are foundational because most algorithms require numerical input and cannot process nulls natively.
How does preprocessing relate to feature engineering?
Preprocessing is often considered a subset of feature engineering. It involves cleaning and transforming data, such as scaling or normalization, to prepare it for the modeling phase.