This Post is included in Section 3: Choosing the Right Features.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

#1: Features Are Not “Data,” but “Investment Hypotheses”

(1) Introduction: We Finally Enter the Feature Section

Up to this point in this series, we have examined the characteristics of models in financial machine learning, primarily through a comparison of LightGBM and ExtraTrees.

The first major theme was to understand how these two tree-based models, LightGBM and ExtraTrees, behave when applied to financial data.

LightGBM is a very sharp model.

Through its gradient boosting mechanism, in which each new tree corrects the errors left by the previous trees, it can strongly capture complex nonlinear patterns in historical data as well as interactions among features.

At the same time, that sharpness is also a risk.

Financial data contains a great deal of noise, including many coincidental patterns that existed only in the past.

For that reason, with LightGBM, the important question was:

How carefully can we use a powerful model?

ExtraTrees has a very different character from LightGBM.

ExtraTrees creates a large number of randomized trees and averages them.

Rather than making each individual tree excessively intelligent, it creates many trees with randomness and produces stable predictions by averaging them.

Therefore, with ExtraTrees, the important question was:

How diversely can we create randomized trees, and how stably can we average them?

To understand this difference, we have built up several series of posts so far.

First, as a “personality assessment” of LightGBM and ExtraTrees, we organized the fundamental differences between the two.

Resistance to overfitting.
Ease of hyperparameter tuning.
Training efficiency.
Inference speed.
Robustness to outliers.
Robustness to noisy targets.
Ability to handle interactions among features.
Ability to handle multicollinearity.
Suitability for small-cap stocks and the S&P 500.
Suitability for short-term and long-term returns.
Behavior as the number of features increases.
Ability to distinguish between improvement and deterioration.
Response to analyst estimate revisions and surprises.

From these perspectives, we examined the differences between LightGBM and ExtraTrees.

Next, in the section on interpreting results, we organized how model results should be read.

RMSE, Pearson, Spearman.
Avg%(H-L).
Feature Importance.
SD.
Turn%.
Return curves.
What it means for a model to withstand the test of time.

In financial ML, a model is not necessarily good simply because it has a small prediction error.

Particularly in investment models, how well the model can rank stocks may be more important than the absolute values of its predictions.

The difference between the top and bottom groups, the stability of the return curve, Turn%, variation across folds, and the stability of Feature Importance are also important.

Therefore, in the section on interpreting results, we established the following perspective:

Evaluating model performance does not mean looking at a single score. It means comprehensively examining prediction, ranking, portfolio performance, and stability over time.

Furthermore, in the section on visualizing the learning process, we looked more structurally at how LightGBM and ExtraTrees learn.

With LightGBM, trees are added sequentially, with each new tree correcting errors left by the preceding trees.

Therefore, the progression of training, early stopping, best_iteration, and the optimal number of trees for each fold become important.

With ExtraTrees, many randomized trees are built independently, and the prediction is determined by averaging them.

Therefore, rather than focusing on what a single tree learned, what matters is how stable the aggregate prediction becomes when many trees are averaged.

In the validation section, we also organized the role of CV, or cross-validation.

Validation is not where the model is updated.

Validation is:

A place to check how resistant the model is to breaking down when confronted with unseen data.

Particularly in financial ML, there are time structures, regime changes, target noise, changes in the cross-section, differences in the universe, and other complications, so simple validation may not be sufficient.

Validation is necessary not only for measuring model performance, but also for understanding model flexibility, overfitting, parameter selection, early stopping, and the meaning of the number of folds.

Then, in the immediately preceding prior part, we covered hyperparameters in detail for LightGBM and ExtraTrees.

For LightGBM, we examined:

• learning_rate

• n_estimators

• early stopping

• num_leaves

• max_depth

• min_child_samples

• feature_fraction

• bagging_fraction

• bagging_freq

• lambda_l1

• lambda_l2

• min_gain_to_split

For ExtraTrees, we examined:

• n_estimators

• max_features

• max_depth

• min_samples_leaf

• min_samples_split

• bootstrap

• max_samples

• random_state

The central message of the prior part was clear.

Hyperparameter tuning is not the task of finding the settings that produce the highest score on historical data.
In financial ML, it is the task of controlling the model’s learning flexibility so that it does not overreact to market noise.

With LightGBM, the key was how much to restrain a powerful model.

With ExtraTrees, the key was how stably to average randomized trees.

And at the end of prior part, we arrived at the following important perspective:

Features are the information given to the model.
Hyperparameters are the rules that determine how the model handles that information.

Even with the same features, LightGBM and ExtraTrees use them differently.

Even within LightGBM, if num_leaves, min_child_samples, and feature_fraction differ, the meaning of those features changes.

Even within ExtraTrees, if max_features, min_samples_leaf, and bootstrap differ, the way those features are used changes.

Therefore, when thinking about features, it is not enough to ask:

Is this feature good or bad?

What we really need to examine is:

Which model is using this feature, with what degree of flexibility, and under what market environment?

Now, we finally enter the “feature” section.

(2) The Role of This part

This part covers feature design in financial ML.

This is one of the most important parts of this series.

That is because no matter how good the model is, there is a limit to what it can achieve if the features are weak.

Conversely, even if you have excellent features, if the model has too much flexibility, it will overfit historical data.

Models and features cannot be considered separately.

Features are information passed to the model.

The model uses that information to rank stocks and attempt to predict future returns.

But the model does not create alpha out of nothing.

Based on the information supplied as features, the model judges which stocks may perform well in the future and which may perform poorly.

Therefore, features are not merely columns of data.

A feature is:

An investment hypothesis passed to the model.

Including ROE as a feature means giving the model the hypothesis:

Companies with high profitability may have some relevance to future returns.

Including P/E as a feature means giving the model the hypothesis:

Stocks that appear inexpensive may have some relevance to future returns.

Including sales growth means giving the model the hypothesis:

Growing companies may have some relevance to future returns.

Including analyst estimate revisions means giving the model the hypothesis:

Changes in market participants’ expectations may affect future returns.

Including momentum means giving the model the hypothesis:

Price trends and market reactions may persist into the future.

Including volatility means giving the model the hypothesis:

The magnitude of a stock’s risk and the instability of its price movements may be related to the reliability of predictions.

Including liquidity means giving the model the hypothesis:

How easily a stock can be traded, and its supply-demand risk, may be related to investability and the reproducibility of returns.

In this way, every feature is an investment hypothesis.

The model does not automatically validate those hypotheses for us.

The model learns patterns among the supplied features that appear useful for prediction in historical data.

The problem is whether those patterns will continue to have meaning in the future.

This is where the difficulty of financial ML lies.

(3) Overall Structure of This part

This part is planned to proceed through five major chapters.

Feature Design in Financial ML

Chapter 1: Alpha Factors

Chapter 2: Risk Factors

Chapter 3: Feature Engineering

Chapter 4: Feature Usage by Model

Chapter 5: Feature Importance / Feature Selection

The number of posts will not be strictly limited from the outset.

Rather, because feature design is one of the climaxes of this series, I intend to cover it in as much detail as necessary.

At present, I expect it to consist of more than 50 posts.

That may sound long.

However, if we are serious about feature design in financial ML, that much material is necessary.

That is because feature design is not simply a question of “which factors should we include?”

Feature design includes at least the following elements:

• Investment hypotheses

• Factor classification

• Correlations among factors

• Compatibility among factors

• Redundant features

• Complementary features

• Levels, differences, and rates of change

• Ranking

• Standardization

• Winsorization

• Sector neutralization

• Missing-value handling

• Outlier handling

• Differences between small-cap and large-cap stocks

• Differences between short-term and long-term targets

• How features are used by different models

• LightGBM conditional splits

• Averaging of randomized trees in ExtraTrees

• How to interpret Feature Importance

• Feature selection

• Feature removal

• Evaluation by feature group

• Liquidity, Turn%, and transaction costs in actual implementation

To cover all of these together requires a certain amount of length.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction: We Finally Enter the Feature Section
(2) The Role of This part
(3) Overall Structure of This part

（Paid Section)
(4) Chapter 1: Alpha Factors
(5) Chapter 2: Risk Factors
(6) Chapter 3: Feature Engineering
(7) Chapter 4: Feature Usage by Model
(8) Chapter 5: Feature Importance / Feature Selection
(9) Features Are Not “Data”
(10) A Model Cannot Create Alpha Without Features
(11) What Is a Good Feature?
(12) Features That Work Individually and Features That Work in Combination
(13) Alpha Factors and Risk Factors
(14) The Meaning of a Feature Changes Depending on the Model
(15) The Same Feature Changes Meaning Depending on the Universe
(16) The Same Feature Changes Meaning Depending on the Target Horizon
(17) The Risk of Increasing the Number of Features
(18) Feature Design Is the Intersection of ML and Domain Knowledge
(19) Summary of This post

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

#10: Practical Tuning Order — What Should We Adjust First in Financial ML?

(1) Introduction

So far in Part 5, we have looked at the hyperparameters of LightGBM and ExtraTrees in sequence.

In post 1, we organized hyperparameter tuning not simply as “performance improvement,” but as:

Controlling the model’s degree of freedom so that it does not overreact to market noise.

For LightGBM, the important question was:

How cautiously should we make a strong model learn?

For ExtraTrees, the important question was:

How diversely should we create random trees, and how stably should we average them?

From post 2 to post 7, we looked at LightGBM’s major parameters.

• learning_rate

• n_estimators

• early stopping

• num_leaves

• max_depth

• min_child_samples

• feature_fraction

• bagging_fraction

• bagging_freq

• lambda_l1

• lambda_l2

• min_gain_to_split

In post 3, post 8, and post 9, we looked at ExtraTrees’ major parameters.

• n_estimators

• max_features

• max_depth

• min_samples_leaf

• min_samples_split

• bootstrap

• max_samples

• random_state

This post is the summary of Part 5.

The theme is:

When actually tuning a financial ML model, what should we adjust first?

Not all hyperparameters have the same importance.

Parameters that should be examined first.
Parameters that should be examined later.
Parameters that greatly change the model’s personality.
Parameters that are closer to fine-tuning.
Parameters that matter when the data is noisy.
Parameters that matter when there are many features.
Parameters that require caution in small caps.
Parameters that require caution in the S&P 500.

If these are not organized, tuning quickly becomes a maze.

In this post, we will organize a practical tuning order for both LightGBM and ExtraTrees.

(2) First Premise: Hyperparameters Are Not About “Searching for the Highest Point”

First, let’s confirm an important premise once again.

Hyperparameter tuning in financial ML is not:

The task of finding the setting that produces the highest score on past data.

Of course, scores are important.

We look at RMSE, Pearson, Spearman, Avg%(H-L), top-group returns, return curves, Turn%, and Feature Importance stability.

However, what truly matters in financial ML is not:

A model that is slightly good in the past,

but:

A model that is hard to break in the future.

A setting that produces the highest CV score on past data does not necessarily remain strong in the future.

Rather, in financial data, settings that look too good in the past can often be weak in the future.

That is because past data contains a lot of noise.

Short-term returns.
Small caps.
Outliers.
Missing values.
Regime changes.
Temporary theme markets.
Macro factors.
Features that worked only during a specific period.

When a model fits too sharply to data containing these things, it learns historical coincidences.

Therefore, in tuning, the important mindset is not:

Search for the highest value,

but:

Search for a stable usable range.

This is the central theme running through all of this Part 5.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction
(2) First Premise: Hyperparameters Are Not About “Searching for the Highest Point”

（Paid Section)
(3) LightGBM’s Tuning Philosophy
(4) What to Adjust First in LightGBM
(5) LightGBM Stage 2: Decide Tree Complexity
(6) LightGBM Stage 3: Examine Minimum Leaf Size
(7) LightGBM Stage 4: Examine Sampling
(8) LightGBM Stage 5: Finish With Regularization
(9) Practical Tuning Order for LightGBM
(10) ExtraTrees’ Tuning Philosophy
(11) What to Adjust First in ExtraTrees
(12) ExtraTrees Stage 2: Examine Minimum Leaf Size
(13) ExtraTrees Stage 3: Examine Tree Depth
(14) ExtraTrees Stage 4: Examine min_samples_split
(15) ExtraTrees Stage 5: Examine bootstrap and max_samples
(16) Practical Tuning Order for ExtraTrees
(17) Priorities in Small-Cap Universes
(18) Priorities in the S&P 500
(19) When the Number of Features Is Large
(20) When the Target Is Noisy
(21) For Medium- and Long-Term Targets
(22) Metrics to Check During Tuning
(23) What Is Good Tuning?
(24) The Final Difference Between LightGBM and ExtraTrees
(25) Initial Comparison Sets in Practice
(26) Cautions When Moving Parameters
(27) Summary of the Hyperparameter Section
(28) Next Step: Toward the Feature Section

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

ExtraTrees Sampling and Randomness — bootstrap / max_samples / random_state

(1) Introduction

In the previous post, we discussed max_depth, min_samples_leaf, and min_samples_split as parameters for controlling ExtraTrees complexity.

ExtraTrees is a model that creates many random trees and averages them.

Therefore, unlike LightGBM, it does not correct the errors of previous trees with the next tree.

However, that does not mean tree complexity can be ignored.

Trees that are too deep.
Leaves that are too small.
Too few trees.
Feature sampling that is too strong.
And results that change significantly just by changing random_state.

These things can make ExtraTrees unstable as well.

The key point of the previous post was:

Use max_features to create diversity,
use n_estimators to strengthen averaging,
and use min_samples_leaf to prevent the trees from running too wild.

This time, we will discuss parameters related to ExtraTrees sampling and randomness.

Specifically:

• bootstrap

• max_samples

• random_state

ExtraTrees is already a model with strong randomness.

Therefore, the theme of this post is:

How much additional randomness should we add to ExtraTrees?

The especially important parameter is bootstrap.

In Random Forest, bootstrap is commonly used.
However, in ExtraTrees, the situation is slightly different.

ExtraTrees already has strong randomness in how splits are created.

Therefore, even without bootstrap, tree diversity is already created to some extent.

So what does it mean to use bootstrap in ExtraTrees?
What does it mean to limit the number of samples shown to each tree with max_samples?
Is random_state merely a setting for reproducibility?

In this post, we will organize these points.

(2) ExtraTrees Is Already a Random Model

First, let’s review the basic idea of ExtraTrees.

ExtraTrees is short for Extremely Randomized Trees.

As the name suggests, it creates many highly randomized trees and averages them.

An ordinary decision tree searches for the best threshold at each split.
Random Forest also searches for relatively good splits while considering only part of the features.

On the other hand, ExtraTrees introduces stronger randomness into how split candidates and thresholds are chosen.

Therefore, each individual tree does not necessarily create optimal splits.

Rather, by deliberately increasing randomness, the model creates diverse trees.

Then, by averaging those diverse trees, it creates stability.

This is the important personality of ExtraTrees.

Instead of making one tree too smart,
create many random trees and average them to stabilize the model.

For this reason, in ExtraTrees, randomness is not a weakness.

Rather, it is a central design principle of the model.

However, randomness is not something where more is always better.

If randomness is too weak, the trees become too similar and the effect of averaging weakens.
If randomness is too strong, each individual tree becomes too weak, and the overall model also becomes unstable.

The bootstrap and max_samples parameters that we will discuss this time are parameters for extending this randomness further in the sample direction.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction
(2) ExtraTrees Is Already a Random Model

（Paid Section)
(3) What Is bootstrap?
(4) When bootstrap Is Not Used
(5) When bootstrap Is Used
(6) The Meaning of bootstrap Is Different in Random Forest and ExtraTrees
(7) What Is max_samples?
(8) When max_samples Is Large
(9) When max_samples Is Small
(10) bootstrap and max_samples Should Be Considered Together
(11) What Is random_state?
(12) random_state Is Not Only About Reproducibility
(13) What to Do When Dependence on random_state Is Strong
(14) Does bootstrap Suppress Overfitting?
(15) About OOB Score
(16) Thinking About Small-Cap Universes
(17) Thinking About the S&P 500
(18) Relationship With Target Horizon
(19) When the Number of Features Is Large
(20) Concrete Setting Examples
(21) What to Check When Using bootstrap
(22) What to Check When Changing max_samples
(23) What to Check When Changing random_state
(24) Common Ways to Read the Results
(25) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Controlling ExtraTrees Complexity — max_depth / min_samples_leaf / min_samples_split

(1) Introduction

In the previous post, we discussed LightGBM’s regularization-related parameters: lambda_l1, lambda_l2, and min_gain_to_split.

LightGBM is a very sharp model.

It has a strong ability to find complex patterns in past data, but at the same time, it also carries the risk of capturing detailed noise in past data.

Therefore, in the previous post, we examined regularization from the perspective of:

Preventing leaf prediction values from becoming too extreme
Preventing the model from creating splits for improvements that are too small

This time, we return to ExtraTrees.

In post 3, we discussed n_estimators and max_features as the first step in ExtraTrees.

ExtraTrees is a model that creates many random trees and averages them.

Therefore, the first important questions were:

How many trees should we create?
How many different features should each tree be allowed to see?

n_estimators was the parameter that determines the amount of averaging.
max_features was the parameter that determines tree diversity.

This time, we will discuss parameters that control the complexity of the ExtraTrees trees themselves.

Specifically:

• max_depth

• min_samples_leaf

• min_samples_split

In LightGBM, we controlled tree complexity and minimum leaf size using num_leaves, max_depth, and min_child_samples.

ExtraTrees has similar ideas as well.

However, ExtraTrees has a different learning mechanism from LightGBM.

LightGBM is a sequential learning model where the next tree corrects the errors of the previous trees.
ExtraTrees is a model that creates many random trees and averages them.

Therefore, controlling complexity in ExtraTrees is less about:

Where to stop a strong model

and more about:

How far to let random trees run wild
and how far to stabilize that wildness through averaging.

(2) What Does “Complexity” Mean in ExtraTrees?

ExtraTrees is also a model that uses decision trees.

Decision trees split data using features.

For example, they can divide stocks using conditions such as:

Is ROE high or low?
Is PER low or high?
Is momentum strong or weak?
Have analyst estimates been revised upward?
Is market capitalization large or small?
Is volatility high or low?

By layering these splits, stocks are divided into more detailed groups.

The deeper a tree becomes, the longer the chain of conditions it can create.

The smaller the leaves become, the more the prediction is based on a small number of stocks.

In this respect, ExtraTrees is similar to LightGBM.

However, ExtraTrees has one major characteristic.

Its splitting process contains strong randomness.

In ExtraTrees, splits are created in an even more randomized way than in ordinary decision trees or Random Forests.

Because of this randomness, each individual tree becomes quite unstable.

However, by creating many trees and averaging them, the instability of individual trees is smoothed out.

In other words, when thinking about ExtraTrees complexity, we should not look at only one tree.

What matters is:

How freely each individual tree is allowed to grow
and how many of those free trees are averaged.

The n_estimators and max_features that we discussed in post 3 were mainly related to the latter: averaging and diversity.

The max_depth, min_samples_leaf, and min_samples_split that we will discuss this time are mainly parameters that control the former: how wildly each individual tree is allowed to behave.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction
(2) What Does “Complexity” Mean in ExtraTrees?

（Paid Section)
(3) What Is max_depth?
(4) Benefits of Making max_depth Deeper
(5) Risks of Making max_depth Too Deep
(6) Benefits and Risks of Making max_depth Shallow
(7) What Is min_samples_leaf?
(8) When min_samples_leaf Is Small
(9) When min_samples_leaf Is Large
(10) min_samples_leaf Is Especially Important in ExtraTrees
(11) What Is min_samples_split?
(12) The Difference Between min_samples_leaf and min_samples_split
(13) Relationship Between max_depth and min_samples_leaf
(14) Relationship With max_features
(15) Relationship With n_estimators
(16) Thinking About Small-Cap Universes
(17) Thinking About the S&P 500
(18) Relationship With Target Horizon
(19) When the Number of Features Is Large
(20) Concrete Setting Examples
(21) What to Check When Increasing min_samples_leaf
(22) What to Check When Limiting max_depth
(23) What to Check When Adjusting min_samples_split
(24) Impact on Feature Importance
(25) Relationship With random_state
(26) Common Ways to Read the Results
(27) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

LightGBM Regularization — lambda_l1 / lambda_l2 / min_gain_to_split

(1) Introduction

In the previous post, we discussed LightGBM’s sampling-related parameters: feature_fraction, bagging_fraction, and bagging_freq.

feature_fraction was sampling in the feature direction.

In other words, it was the parameter that determines:

How much of the full feature set each tree is allowed to see.

On the other hand, bagging_fraction was sampling in the sample direction.

In other words, it was the parameter that determines:

How much of the full sample set each tree is allowed to see.

The key point of the previous post was that showing LightGBM all information every time is not necessarily good.

In financial ML, showing all features, all stocks, and all periods every time can cause the model to fit too strongly to coincidences in past data.

Therefore, we introduced a little randomness through sampling and organized the idea of suppressing excessive dependence on:

Specific features
Specific groups of stocks
Specific periods
Specific market regimes

This time, we will discuss LightGBM’s regularization-related parameters.

Specifically:

• lambda_l1

• lambda_l2

• reg_alpha

• reg_lambda

• min_gain_to_split

lambda_l1 and reg_alpha are basically parameters in the same direction.
lambda_l2 and reg_lambda are also basically parameters in the same direction.

The theme of this post can be summarized as:

Directly applying brakes so that the model does not create overly detailed splits or extreme prediction values.

Compared with the parameters we have discussed so far, regularization is a little harder to see intuitively.

Unlike learning_rate or num_leaves, it is not immediately obvious as “learning speed” or “tree complexity.”

However, in financial ML, regularization is also extremely important.

That is because LightGBM is a model that can fit very sharply to past data.

A strong ability to find detailed patterns in past data also means a strong ability to find detailed noise in past data.

Regularization is a mechanism for asking this sharp model:

Is it really worth reacting that finely?

(2) What Is Regularization?

First, let’s briefly organize what regularization means.

In machine learning, regularization means adding constraints so that the model does not become too complex.

A model tries to make predictions that are as good as possible on the training data.

However, training data contains noise.

Therefore, if we give the model too much freedom, it will learn not only the general patterns it should learn, but also the noise that happened to exist in the training data.

This is overfitting.

Regularization is used to suppress this overfitting.

Intuitively, it is like saying to the model:

You may do something complex.
But if you do, you must pay a penalty.

It does not completely prohibit the model from making complex predictions, producing extreme values, or creating detailed splits.

However, it asks:

Is there enough reason to do that?

In financial ML, this attitude is extremely important.

That is because a split or prediction value that slightly improves the score on past data does not necessarily have meaning in the future.

Regularization is a brake against overly detailed fitting to past data.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction
(2) What Is Regularization?

（Paid Section)
(3) The Role of Regularization in LightGBM
(4) What Is lambda_l1?
(5) What Is lambda_l2?
(6) The Difference Between lambda_l1 and lambda_l2
(7) What Is min_gain_to_split?
(8) min_gain_to_split Asks “Is This Split Worth It?”
(9) The Difference Between Lambda Regularization and min_gain_to_split
(10) Typical Situations Where Regularization Becomes Necessary
(11) The Risk of Making Regularization Too Strong
(12) Why Regularization Is Important in Financial ML
(13) Thinking About Small-Cap Universes
(14) Thinking About the S&P 500
(15) Relationship With Target Horizon
(16) When the Number of Features Is Large
(17) Concrete Setting Examples
(18) What to Check When Increasing lambda_l2
(19) What to Check When Increasing lambda_l1
(20) What to Check When Increasing min_gain_to_split
(21) Impact on best_iteration
(22) Practical Tuning Order
(23) Common Ways to Read the Results
(24) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

LightGBM Sampling — feature_fraction and bagging_fraction

(1) Introduction

In the previous post, we discussed min_child_samples, also known as min_data_in_leaf, as an extremely important parameter for controlling overfitting in LightGBM.

min_child_samples was the parameter that determines the minimum number of samples required in one leaf.

If num_leaves and max_depth, which we discussed two posts ago, determine:

How finely the model is allowed to split,

then min_child_samples determines:

How much evidence is required at minimum in each detailed group.

This time, we will discuss LightGBM’s sampling-related parameters.

Specifically:

• feature_fraction

• colsample_bytree

• bagging_fraction

• subsample

• bagging_freq

Among these, feature_fraction and colsample_bytree mainly refer to sampling in the feature direction.

On the other hand, bagging_fraction and subsample mainly refer to sampling in the data direction, meaning sample-level sampling.
And in LightGBM, when using bagging_fraction, bagging_freq also becomes important.

The theme of this post can be summarized as:

Should we show the model all information every time?
Or should we deliberately show only part of the information?

Intuitively, it may seem better to show all information.

Using all features and all stocks and periods for training appears to give the model more information.

However, in financial ML, that is not always the case.

By showing everything, the model may fit too strongly to coincidences in past data.

Conversely, by training the model with only part of the features or part of the samples, we may be able to prevent it from depending too heavily on specific features or specific periods.

Sampling is not about throwing data away.

In financial ML, sampling is better understood as:

A mechanism for preventing the model from trusting past data too much.

(2) What Is Sampling in LightGBM?

LightGBM is a very powerful model.

If there is a strong feature, it finds and uses it.
If there is a complex interaction, it tries to capture it.
If there is a split that improves performance on past data, it actively uses it.

This is a strength.

However, in financial ML, this strength also becomes a risk.

That is because past data contains a huge amount of noise.

For example, suppose a certain feature worked very strongly during one specific period.

That feature may truly be an investment signal that remains effective in the future.
However, it may simply have matched the market regime of that period.

If we show LightGBM all features every time, the model may become heavily dependent on that strong feature.

Also, suppose that a specific group of stocks rose significantly in one period.

If that is a reproducible pattern, that is fine.
But it may have been caused by individual news, liquidity, themes, the macro environment, or simple coincidence.

If we show LightGBM all samples every time, the model may react strongly to that period or that group of stocks.

This is where sampling comes in.

Sampling means limiting the features or samples used during training to only a subset.

In feature-direction sampling, instead of using all features every time, the model uses only some features.

In sample-direction sampling, instead of using all data rows every time, the model uses only some rows.

This introduces a little randomness into the model.

The purpose is:

To prevent the model from depending too heavily on specific features or specific samples.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction
(2) What Is Sampling in LightGBM?

（Paid Section)
(3) What Is feature_fraction?
(4) When feature_fraction Is Large
(5) When feature_fraction Is Small
(6) feature_fraction Is a Bridge to the Feature Section
(7) Multicollinearity and feature_fraction
(8) What Is bagging_fraction?
(9) Why bagging_freq Is Important
(10) When bagging_fraction Is Large
(11) When bagging_fraction Is Small
(12) The Difference Between feature_fraction and bagging_fraction
(13) What Happens When We Use Both?
(14) Why Sampling Matters in Financial ML
(15) Thinking About Small-Cap Universes
(16) Thinking About the S&P 500
(17) Relationship With Target Horizon
(18) Concrete Setting Examples
(19) What to Check When Lowering feature_fraction
(20) What to Check When Lowering bagging_fraction
(21) Sampling and How to Read Feature Importance
(22) Sampling and Turn%
(23) Common Ways to Read the Results
(24) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Minimum Leaf Size in LightGBM — min_child_samples / min_data_in_leaf

(1) Introduction

In the previous post, we discussed num_leaves and max_depth as important parameters that determine the complexity of LightGBM.

num_leaves was the parameter that determines how many leaves one tree can have.
In other words, it determines how finely the model is allowed to divide stocks into groups.

max_depth was the parameter that determines the depth of the tree.
In other words, it determines how many stages of conditional splits are allowed.

These two parameters control the “shape complexity” of LightGBM.

This time, we will discuss the next extremely important parameter.

That is:

• min_child_samples

• min_data_in_leaf

These two are often used with essentially the same meaning. They determine the minimum number of samples required in one leaf.

Using the language from the previous post, num_leaves and max_depth determine:

How finely the model is allowed to split.

In contrast, min_child_samples determines:

How many samples are required at minimum in each of those detailed groups.

This is extremely important in financial ML.

That is because patterns observed only in a small number of stocks or a small number of periods may not continue into the future.

If the model learns:

This small group of stocks performed well in the past,
so it should also perform well in the future,

that can be quite dangerous.

min_child_samples is an important brake that prevents LightGBM from creating too many groups that are too small.

(2) What Is a Leaf?

First, let’s review what a leaf is.

A decision tree splits data based on conditions.

For example, it may use splits such as:

Is ROE high or low?
Is PER low or high?
Is momentum strong or weak?
Have analyst estimates been revised upward?
Is market capitalization large or small?

By layering these conditions over several stages, stocks are divided into multiple groups.

The final groups at the end of the tree are called leaves.

For example, one leaf may contain stocks such as:

Stocks with high ROE, low PER, strong momentum, and upward analyst estimate revisions.

Another leaf may contain stocks such as:

Stocks with low ROE, high PER, weak momentum, and downward analyst estimate revisions.

LightGBM assigns a prediction value to each leaf.

In other words, for the group of stocks that fall into each leaf, the model makes judgments such as:

This group is expected to have high average returns.
This group is expected to have low average returns.

What matters here is when a leaf is too small.

If only a very small number of stocks fall into a certain leaf, that leaf’s prediction value is strongly influenced by a small number of samples.

For example, suppose only 5 stocks fall into a leaf.
If those 5 stocks happened to produce very good returns in the past, the model may judge that leaf to be a “good group.”

But did those 5 stocks truly share a meaningful common pattern?
Or did they simply rise by chance during that period?

In financial ML, distinguishing between these two is extremely difficult.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction
(2) What Is a Leaf?

（Paid Section)
(3) What Is min_child_samples?
(4) Relationship With min_data_in_leaf
(5)When min_child_samples Is Small
(6) When min_child_samples Is Large
(7) min_child_samples Determines the “Amount of Evidence”
(8) Relationship With num_leaves
(9) Relationship With max_depth
(10) Relationship With learning_rate
(11) Why min_child_samples Is Especially Important in Financial ML
(12) Thinking About Small-Cap Universes
(13) Thinking About the S&P 500
(14) Relationship With Target Horizon
(15) When the Number of Features Is Large
(16) Relationship With Missing Values and Outliers
(17) Concrete Setting Examples
(18) What to Check When Increasing min_child_samples
(19) What to Check When Decreasing min_child_samples
(20) Impact on Feature Importance
(21) Impact on best_iteration
(22) Practical Tuning Order
(23) Common Ways to Read the Results
(24) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Controlling LightGBM Complexity — num_leaves and max_depth

(1) Introduction

In the previous post, we discussed n_estimators and max_features as the first hyperparameters to examine in ExtraTrees.

ExtraTrees builds many random trees and averages them.
Therefore, the first things to consider were:

How many trees should we build?
How diverse should each tree be?

n_estimators determines the amount of averaging.
max_features determines the diversity of the trees being averaged.

In this post, we return to LightGBM.

In Post 2, we discussed learning_rate and n_estimators as the first step in LightGBM.

Those two parameters determined:

How cautiously the model learns
How far the learning process is allowed to continue

This time, we will discuss the next important parameters.

They are:

• num_leaves

• max_depth

If learning_rate and n_estimators determine “how much the model is allowed to learn,” then num_leaves and max_depth determine:

How complexly the model is allowed to learn.

LightGBM is a very powerful model.
A major reason for that strength is its ability to capture complex nonlinear patterns and interactions between features.

However, in financial ML, this strength also becomes a risk.

If a model can capture detailed patterns in past data, it can also capture detailed noise in past data.

Therefore, in LightGBM, we need to carefully decide:

How complex the conditional splits are allowed to become.

At the center of this issue are num_leaves and max_depth.

(2) What Does “Complexity” Mean in LightGBM?

First, let’s organize what complexity means in LightGBM.

LightGBM is a model that accumulates decision trees.

A decision tree splits data using features.

To simplify greatly, possible splits might look like this:

Is ROE high or low?
Is PER low or high?
Is recent momentum strong or weak?
Have analyst estimates been revised upward?
Is sales growth high?
Is market capitalization large or small?

By combining these conditions, the model divides stocks into several groups.

For example, it can create a group such as:

Stocks with high ROE, low PER, and strong recent momentum.

Or it can create a group such as:

Stocks with high sales growth, low profitability, and downward analyst estimate revisions.

In this way, decision trees can create quite complex conditions by combining features.

As the tree becomes deeper and the number of leaves increases, it can create more detailed groups.

This is the expressive power of LightGBM.

However, there is a problem.

Being able to create detailed groups also means being able to react to accidental combinations in past data.

For example, suppose that in one period, the following group happened to produce good returns:

Stocks with high ROE, low PER, medium 6-month momentum, slightly positive analyst estimate revisions, and mid-cap market capitalization.

If this pattern is truly effective in the future, that is excellent.

However, it may simply have occurred by chance in that market environment.

If the tree is too complex, it can learn these accidental combinations of conditions.

This is extremely important in financial ML.

The ability to capture complex patterns
and the danger of capturing accidental noise

are two sides of the same coin.

What the Paid Section Deeply Explores:

（Free Section)
(1) Introduction
(2) What Does “Complexity” Mean in LightGBM?

（Paid Section)
(3) What Is num_leaves?
(4) When num_leaves Is Large
(5) When num_leaves Is Small
(6) num_leaves Strongly Changes LightGBM’s Personality
(7) What Is max_depth?
(8) The Difference Between num_leaves and max_depth
(9) When max_depth Is Deep
(10) When max_depth Is Shallow
(11) LightGBM’s Leaf-Wise Growth and max_depth
(12) The Relationship Between num_leaves and max_depth
(13) Concrete Setting Examples
(14) What to Check When Increasing num_leaves
(15) What to Check When Limiting max_depth
(16) Relationship With best_iteration
(17) Relationship With min_child_samples
(18) Thinking About Small-Cap Universes
(19) Thinking About the S&P 500
(20) Short-Term Targets and Long-Term Targets
(21) When the Number of Features Is Large
(22) Practical Tuning Order
(23) Common Ways to Read the Results
(24) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

The First Tuning Step in ExtraTrees — n_estimators and max_features

Introduction

In the previous episode, we discussed learning_rate and n_estimators as the first hyperparameters to examine in LightGBM.

In LightGBM, trees are added one by one, and each new tree corrects the errors of the previous trees.
Therefore, learning_rate and n_estimators were the most upstream parameters that determine the model’s overall learning speed and overfitting risk.

In LightGBM, the important questions were:

How much should each tree learn?
How many trees should be accumulated?

In this episode, we will discuss ExtraTrees.

ExtraTrees also has important parameters that we should examine first.

They are:

• n_estimators

• max_features

However, there is one important point to keep in mind.

LightGBM also has a parameter called n_estimators.
ExtraTrees also has n_estimators.

But even though the name is the same, the meaning is quite different.

In LightGBM, n_estimators was the number of times the model continues sequential error correction.
In other words, increasing the number of trees meant making the model fit the past data further.

In ExtraTrees, on the other hand, n_estimators is the number of many randomized trees that are created and then averaged.
Increasing the number of trees mainly means strengthening the averaging effect and stabilizing predictions.

This difference is extremely important.

In ExtraTrees, the first things we should consider are:

How many trees should we build?
How many different features should each tree be allowed to see?

In other words, the first step in ExtraTrees is:

Create stability through the number of trees,
and create diversity through max_features.

(1) ExtraTrees Is a Model That Averages Random Trees

First, let’s recall the basic personality of ExtraTrees.

ExtraTrees is short for Extremely Randomized Trees.
As the name suggests, it is a tree ensemble model that uses a fairly strong degree of randomness.

If we use only a single decision tree, the model tends to react strongly to small movements in the data.
Especially when the data contains a lot of noise, as financial data does, a single tree is quite unstable.

It may split well in one period but fail completely in another.
A certain feature may appear to work by chance, but that does not mean it will continue to work in the future.

Therefore, ExtraTrees builds many trees.

Moreover, it builds each tree with a considerable amount of randomness.
Then it averages their predictions.

Each individual tree is not necessarily smart.
However, when many trees are averaged, the randomness of individual trees is smoothed out.

This idea is very important in financial ML.

In financial data, individual signals are weak and unstable.
Fundamentals, momentum, and analyst estimates may work in one period and stop working in another.

For this reason, relying on one strong rule is often less stable than averaging many weak judgments.

This is exactly the basic idea behind ExtraTrees.

Instead of building one strong tree,
build many random trees and stabilize the model by averaging them.

For this reason, the first parameters to examine in ExtraTrees naturally become the following two:

• How many trees should be built?

• How many different features should each tree be allowed to see?

These are n_estimators and max_features.

What the Paid Section Deeply Explores:

（Free Section)
(1) ExtraTrees Is a Model That Averages Random Trees

（Paid Section)
(2) What Is n_estimators?
(3) Benefits of Increasing n_estimators
(4) Risks of Increasing n_estimators
(5) Does n_estimators Increase Overfitting?
(6) What Is max_features?
(7) When max_features Is Large
(8) When max_features Is Small
(9) max_features Also Helps Deal With Multicollinearity
(10) When the Number of Features Increases to 120, 200, or 300
(11) n_estimators and max_features Should Be Considered Together
(12) Concrete Setting Examples
(13) Points to Watch When Reading max_features Result
(14) Check the Stability of Feature Importance
(15) What to Check When Increasing n_estimators
(16) random_state and Reproducibility
(17) Organizing the Difference From LightGBM Again
(18) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

The First Tuning Step in LightGBM — learning_rate and n_estimators

Introduction

In the previous post, we organized the overall picture of hyperparameter tuning.

The especially important point was that we should not think of hyperparameter tuning simply as “performance improvement.”

In financial machine learning, the setting that produces the best score on past data is not necessarily the strongest setting for the future.
In fact, a model that fits the past too well may collapse more easily in future market environments.

Therefore, the purpose of hyperparameter tuning is not to create the best historical backtest.

The purpose is:

To control the model’s degree of freedom so that it does not overreact to market noise.

From this post, we will begin discussing specific parameters.

The first parameters we will cover are the most upstream parameters in LightGBM.

They are:

• learning_rate

• n_estimators

In LightGBM, these two should not be considered separately.

learning_rate determines how much each individual tree learns.
n_estimators determines how many trees are accumulated.

In other words, together, these two parameters determine:

How cautiously the model learns
How far the learning process is allowed to continue

In LightGBM tuning, it is natural to look at these first.

This is because these two parameters have a major impact on the model’s overall learning speed, overfitting risk, cross-validation behavior, and early stopping results.

(1) LightGBM Is a Model That Learns by Adding Trees

First, let’s recall the basic learning mechanism of LightGBM.

LightGBM is a gradient boosting model.

It is not a model like ExtraTrees, where many trees are built independently and averaged.
In LightGBM, trees are added one by one in sequence.

The first tree makes a prediction.
That prediction contains errors.
The next tree learns in a way that corrects those errors.
The following tree then corrects the remaining errors.

In this way, LightGBM is:

A model in which each new tree gradually corrects the failures of the previous trees.

This is a major difference from ExtraTrees.

In ExtraTrees, increasing the number of trees mainly means strengthening the averaging effect.
By building and averaging many random trees, the model stabilizes its predictions.

In LightGBM, on the other hand, increasing the number of trees means pushing the learning process further.
That is, the model continues to fit the past data more deeply.

For this reason, we need to be cautious about increasing the number of trees in LightGBM.

If we add more trees, performance will certainly improve at first.
However, beyond a certain point, the model may begin learning small pieces of noise that will not be reproduced in the future.

When thinking about “how far to let the model learn,” learning_rate and n_estimators become crucial.

What the Paid Section Deeply Explores:

（Free Section)
(1) LightGBM Is a Model That Learns by Adding Trees

（Paid Section)
(2) What Is learning_rate?
(3) What Is n_estimators?
(4) learning_rate and n_estimators Should Be Considered Together
(5) Benefits and Risks of a High learning_rate
(6) Benefits and Risks of a Low learning_rate
(7) Set n_estimators Large and Let Early Stopping Decide
(8) Early Stopping Is a Tool for Deciding the Endpoint of Learning
(9) CV Fold Count and the Optimal Number of Trees
(10) In Financial ML, a Lower learning_rate + Early Stopping Often Becomes the Baseline
(11) Concrete Setting Examples
(12) Look at best_iteration Rather Than the Number of Trees Itself
(13) When best_iteration Differs Greatly Across Folds
(14) What to Check When Lowering learning_rate
(15) What to Check When Increasing n_estimators
(16) “A Low learning_rate Is Safe” Is Not Always True
(17) The First Thing to Do When Tuning LightGBM
(18) Typical Ways to Read learning_rate and n_estimators
(19) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Hyperparameters Are Not About “Improving Performance,” but About “Controlling Runaway Behavior”

Introduction

So far in several series, we have compared LightGBM and Extra Trees from several different angles.

In Part 1, we conducted a “personality diagnosis” of the two models.
We examined the basic characteristics of each model, including resistance to overfitting, robustness to outliers, robustness to noisy targets, handling of interactions between factors, linear and nonlinear patterns, multicollinearity, and suitability for small-cap universes and the S&P 500.

In Part 2, we discussed how to interpret model results.
We looked at RMSE, Pearson correlation, Spearman correlation, Avg%(H-L), Feature Importance, SD, Turn%, and return curves.
In particular, we confirmed that in financial machine learning, it is not enough for prediction error to be small. We also need to examine the spread between the top-ranked and bottom-ranked stocks, the stability of the return curve, turnover, standard deviation, and other metrics in a comprehensive way.

In Part 3, we visualized the learning steps of LightGBM and Extra Trees.
We saw that LightGBM is a sequential model that corrects the errors of previous trees with subsequent trees, while Extra Trees is a model that builds many highly randomized trees independently and gains stability by averaging them.

In Part 4, we discussed the true role of validation, or cross-validation.
Validation data is not where the model is directly updated. Rather, it is where we examine whether the model can withstand future data.
We also covered early stopping, the number of folds, the strictness of validation, and why ensembles of weak models can be powerful.

At this point, we have a fairly clear picture of the basic personalities of LightGBM and Extra Trees, how to read their results, how their learning processes work, and what role validation plays.

The next major topic should be “features”.

In other words, we need to discuss what kinds of fundamental indicators, price momentum indicators, analyst estimate indicators, quality indicators, value indicators, growth indicators, and risk indicators should be fed into the model.

However, before moving on to that major topic, there is one important preliminary step.

That is the topic we will begin covering now: “hyperparameters”.

Before increasing or modifying features, we first need to understand the “control knobs” on the model side.
This is because even if we use the same features, the model’s behavior can change significantly depending on the hyperparameter settings.

Even when using the same data, the same target, and the same features, the following can happen:

• LightGBM may try to predict too sharply.

• Extra Trees may average too much and become too dull.

• The scores of top-ranked stocks may become unstable.

• Results may look good in cross-validation but collapse in backtesting.

• RMSE may improve while Avg% (H-L) does not improve.

Many of these differences are deeply related to hyperparameter settings.

Therefore, in Part 5, we will organize the major hyperparameters of LightGBM and Extra Trees in an order that is close to the actual order in which we would examine them during tuning.

In this first post, before getting into individual parameters, we will first review the overall picture.

(1) What Are Hyperparameters?

In machine learning models, there are broadly two types of “settings.”

The first type is what the model automatically learns from the data.
For example, which feature to use at each split, what threshold to split on, and what prediction value to place in each leaf.

These are determined by the model itself from the training data.

The second type is what humans decide in advance.
For example, how many trees to build, how deep the trees are allowed to grow, how many samples are required in a leaf, how much randomness to apply to feature selection, and what learning rate to use.

These are hyperparameters.

Put simply, hyperparameters are:

Control knobs that humans set in advance to determine how the model learns.

For LightGBM, examples include:

• learning_rate

• n_estimators

• num_leaves

• max_depth

• min_child_samples

• feature_fraction

• bagging_fraction

• lambda_l1

• lambda_l2

• min_gain_to_split

For Extra Trees, examples include:

• n_estimators

• max_features

• max_depth

• min_samples_leaf

• min_samples_split

• bootstrap

• max_samples

These are not just minor settings.

They determine how complex a pattern the model is allowed to learn.
How much the model reacts to noise.
How stable its predictions are.
How much it depends on specific features.
How much randomness is introduced.

In other words, they are important elements that determine the model’s “personality.”

What the Paid Section Deeply Explores:

（Free Section)
(1) What Are Hyperparameters?

（Paid Section)
(2) Is Hyperparameter Tuning a Process for Improving Performance?
(3) In Financial ML, “Robustness” Matters More Than “Predictive Power”
(4) LightGBM and Extra Trees Require Different Tuning Philosophies
(5) LightGBM Is “Sharp, but Can Become Unstable”
(6) Extra Trees Is “Duller, but Easier to Stabilize”
(7) Even With the Same Features, Hyperparameters Can Change the Results
(8) RMSE Optimization and Investment Performance Optimization Are Not the Same
(9) Hyperparameters Are Not Tools for Finding the “Best Historical Performance”
(10) What to Look at First in LightGBM
(11) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Why Ensembles of Weak Models Are So Powerful in Financial ML

Introduction

Why LightGBM and ExtraTrees Work So Well with Financial Data

Throughout this series, we have examined the relationship between Validation and model training.

The most important takeaways were:

1. Validation is not where the model gets updated

2. CV is a mechanism for evaluating models

3. Early Stopping is determined by the average CV score

4. Stricter CV leads to shallower models

5. In financial ML, making CV too strict can sometimes reduce performance

Once you understand these concepts, a natural question arises:

What kind of models perform best in financial ML?

The answer is:

In financial machine learning, ensembles of many weak models often outperform a single strong model.

And two algorithms that implement this idea exceptionally well are:

• LightGBM

• ExtraTrees

In this final article, we will examine why.

(1) Stock Market Data Contains Overwhelming Amounts of Noise

Let’s begin with the fundamental reality of financial ML.

Stock market data has the following structure:

signal << noise

As a rough illustration, imagine a world where:

• True predictive power (IC) ≈ 0.02

• Noise ≈ ±0.10

In other words:

Model prediction = signal + noise

Because of this,

the influence of noise on any single model is extremely large.

The Problem with a Single Model

Suppose we train a single model.

Its prediction can be expressed as:

Prediction = signal + noise

The problem is that:

noise > signal

In other words,

most of the model’s prediction may be driven by randomness.

This is what makes financial machine learning so difficult.

What the Paid Section Deeply Explores:

（Free Section)
(1) Stock Market Data Contains Overwhelming Amounts of Noise

（Paid Section)
(2) The Basic Principle of Ensembles
(3) Noise Declines at a Rate of √N
(4) Why Weak Models Become Important
(5) Weak Models Are Less Likely to Learn Noise
(6) LightGBM Builds Signal by Adding Weak Trees
(7) ExtraTrees Averages Many Random Trees
(8) The Difference Between LightGBM and ExtraTrees
(9) Why These Two Methods Work So Well in Financial ML
(10) Series Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Why Validation Becomes More Stringent as the Number of Folds Increases

Introduction

In the previous article, we discussed the following phenomenon:

• Increase the number of CV folds
  ↓

• Validation becomes more stringent
  ↓

• Earlier Early Stopping
  ↓

• Fewer trees

This is a natural consequence of how LightGBM works.

Once you understand this, a new question naturally arises:

Is more CV always better?

In conventional machine learning, the answer is almost always yes.

• More CV folds
  ↓

• More stable evaluation
  ↓

• Better model selection

However, financial machine learning introduces a more complicated problem.

In practice, the following phenomenon is not uncommon:

• Increase CV folds
  ↓

• CV scores become more stable
  ↓

• But live trading performance declines

In other words,

increasing the amount of CV can sometimes reduce real-world performance.

In this article, we will examine why.

(1) Stock Prediction Has Extremely Weak Signals

First, there is a fundamental characteristic of financial machine learning.

Stock market data has the following structure:

signal << noise

As a rough illustration, imagine a world where:

• True predictive power (IC) ≈ 0.02

• Noise ≈ ±0.10

In other words:

Model score = signal + noise

As a result,

CV scores tend to exhibit very large fluctuations.

What the Paid Section Deeply Explores:

（Free Section)
(1) Stock Prediction Has Extremely Weak Signals

（Paid Section)
(2) Strong but Unstable Signals
(3) Weak but Stable Signals
(4) What Happens When CV Becomes More Stringent?
(5) But Live Trading Is Not a World of Averages
(6) What Happens as a Result?
(7) Winner’s Curse (Model Selection Bias)
(8) What Do Practitioners Actually Do in Financial ML?
(9) Summary
(10) Next Up

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Why Increasing the Number of CV Folds Can Reduce the Number of Trees

Introduction

In the previous article, we explained how Early Stopping is determined within Cross-Validation (CV).

The process looked like this:

iteration
↓
Validation scores from each fold
↓
Average score
↓
Improvement stops
↓
Early Stopping
↓
best_iteration determined

In other words,

Early Stopping is determined by the overall CV score.

Once you understand this, a natural question arises:

What happens when we increase the number of CV folds?

Intuitively, many people think:

More folds
↓
More accurate evaluation
↓
Better model

However, in real LightGBM models, the following phenomenon often occurs:

More folds
↓
Earlier Early Stopping
↓
Smaller best_iteration
↓
Fewer trees

This is not a bug or abnormal behavior.

In fact, once you understand how CV and Early Stopping work, it is a very natural outcome.

In this article, we will examine why.

(1) Early Stopping Adds Trees Only While Improvement Continues

Let’s start with the basic idea.

LightGBM trains by adding decision trees one by one:

Tree1
Tree2
Tree3
Tree4
...

After each tree is added, it evaluates the:

Validation metric

The process is:

• Score improves
  ↓
  Continue training

• Score stops improving
  ↓
  Early Stopping

Therefore,

best_iteration is determined by how long the Validation score continues to improve.

What the Paid Section Deeply Explores:

（Free Section)
(1) Early Stopping Adds Trees Only While Improvement Continues

（Paid Section)
(2) With Fewer Folds, Luck Has More Influence
(3) More Folds Make Evaluation Stricter
(4) More Folds Reduce the Impact of Lucky Results
(5) More Folds Mean a Stricter Overfitting Check
(6) A Common Real-World Result
(7) This Happens Especially Often with Stock Market Data
(8) However, More Folds Are Not Always Better
(9) This Leads to an Important Financial ML Question
(10) Next Up

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

How Early Stopping (LightGBM) Is Determined in Cross-Validation

Introduction

In the previous posts, we discussed the role of Validation (CV).

The three key takeaways were:

• Validation is not where the model gets updated

• Validation is where the model gets evaluated

• Validation is where the best model configuration gets selected

An important point is:

the model weights themselves are not updated using validation data.

However, at the same time:

validation scores heavily influence the final model structure through mechanisms such as Early Stopping and hyperparameter selection.

One of the most important things controlled by validation is:

Early Stopping (LightGBM)

If you use LightGBM, you have probably seen a value called:

best_iteration

Simply put, this represents:

“How many trees should the model build?”

But how exactly is this value determined inside cross-validation?

In this article, we will break down the mechanism step by step.

(1) Trees Keep Getting Added in LightGBM

Before diving into the main topic of this post, it’s important to review the differences in the “Tree” concepts between LightGBM and Extra Trees.

Note: The concept of “Number of Trees” differs between LightGBM and Extra Trees

• Extra Trees builds trees independently and in parallel — each tree is a complete, standalone predictor. More trees = more stability, like a larger jury.

• LightGBM builds trees sequentially — each tree corrects the residual errors left by the previous one. No single tree makes sense on its own.

• In LightGBM, the final prediction is the sum of all trees weighted by the learning rate — the “number of trees” is closer to the number of correction steps than to independent voters.

• Adding more trees in ExtraTrees is generally safe; in LightGBM it increases the risk of overfitting, which is why Early Stopping is needed to find the right stopping point.

• The best_iteration in LightGBM represents the optimal number of sequential correction steps, not the number of independent models.

Let’s start with the basic idea.

During training, LightGBM builds trees sequentially:

• Tree1

• Tree2

• Tree3

• Tree4

...

Each new tree is added on top of the previous ones.

As this happens:

• More trees
  ↓

• Higher model complexity
  ↓

• Better fit to the training data

This is one of the core properties of boosting models.

However, at the same time:

• Too much complexity
  ↓

• The model starts learning noise
  ↓

• Overfitting

also occurs.

This is where Early Stopping becomes important.

What the Paid Section Deeply Explores:

（Free Section)
(1) Trees Keep Getting Added in LightGBM

（Paid Section)
(2) The Basic Idea of Early Stopping in LightGBM
(3) Training and Validation Behave Differently
(4) In CV, Each Fold Gets Evaluated Separately
(5) Early Stopping Uses the Overall CV Score
(6) How Early Stopping Gets Determined
(7) After CV, the Final Model Gets Built
(8) A New Question Naturally Appears
(9) Next up

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Mastering the Four Cross-Validation Methods

Introduction

Portfolio123’s AI Factor feature provides four cross-validation (CV) methods:

• Basic Holdout

• Time Series CV

• Rolling Time Series CV

• K-fold CV (Blocked)

At first glance, they all appear to evaluate the same thing: model generalization performance.
However, after several years of developing AI factors, a recurring pattern becomes hard to ignore:

K-fold CV (Blocked) often produces dramatically higher scores,
while Time Series CV and Rolling Time Series CV remain mediocre.

What does this actually mean?

Does it imply that K-fold is simply the “better” validation method?
Or is it revealing something problematic about the model itself?

In this article, we will carefully examine the design philosophy behind all four CV methods and explain the practical implications of the “K-fold only looks great” phenomenon.

(1) The Design Philosophy Behind the Four CV Methods

1) Basic Holdout

This is the simplest validation method: the dataset is split once into training and validation periods.

• [Training ───────────────────] [Gap] [Holdout]

Characteristics

• Simple implementation

• Low computational cost

• Useful for rapid prototyping

However:

• Results heavily depend on the split point

• Only one validation test is performed

• Weak at detecting overfitting

This method is best suited for the initial “does this idea even work?” stage.

2) Time Series CV (Expanding Window)

The start date remains fixed while the training period gradually expands over multiple folds.

• Fold 1: [Training ───] [Gap] [Holdout]

• Fold 2: [Training ─────────] [Gap] [Holdout]

• Fold 3: [Training ───────────────] [Gap] [Holdout]

• Fold 4: [Training ─────────────────────] [Gap] [Holdout]

Characteristics

• Strictly preserves the “past → future” direction

• Closely resembles real-world deployment

• More robust against overfitting through repeated validation

On the other hand:

• Older data accumulates in every fold

• Sensitivity to regime changes is somewhat reduced

For evaluating long-term generalization, this is one of the most trustworthy CV approaches.

3) Rolling Time Series CV

The training window size is fixed while the entire window rolls forward through time.

• Fold 1: [Training ─────────────] [Gap] [Holdout]

• Fold 2: 　[Training ─────────────] [Gap] [Holdout]

• Fold 3: 　　 [Training ─────────────] [Gap] [Holdout]

• Fold 4: 　　　 [Training ─────────────] [Gap] [Holdout]

Characteristics

• Emphasizes recent market environments

• Better suited for testing regime adaptability

• Particularly useful around structural transitions such as:

  • interest rate regime shifts

  • post-COVID market transitions

  • volatility regime changes

The tradeoff:

• Smaller training datasets

• Reduced model stability

Practically speaking, this CV answers the question:

“Does the model still work in the current market environment?”

4) K-fold CV (Blocked)

This is the most misunderstood method.

Portfolio123 does not use standard randomized K-fold CV.
Instead, it uses a time-aware variation called Blocked K-fold CV.

Conceptually, it looks like this:

• Fold 1: [Holdout ] [Gap] [Training ─────────────────────]

• Fold 2: [Training ──────] [Gap] [Holdout] [Gap] [Training ────────]

• Fold 3: [Training ─────────────] [Gap] [Holdout] [Gap] [Training ─]

• Fold 4: [Training ───────────────────── ] [Gap] [Holdout]

The critical point is Fold 1 and Fold2:

some folds train on future data and validate on the past.

This is fundamentally different from Time Series CV.

What the Paid Section Deeply Explores:

（Free Section)
(1) The Design Philosophy Behind the Four CV Methods

（Paid Section)
(2) What “Blocked” and “Gap” Really Mean
(3) The Real Meaning of “Only K-fold Looks Great”
(4) Meta-Overfitting: The Hidden Danger of Repeated Development
(5) Why Hyperparameter Tuning Often Fails to Fix It
(6) Practical Approaches for Narrowing the Gap
(7) Practical Conclusions
(8) Summary
(9) Next up

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Validation Is Not “Where the Model Gets Updated”

Introduction

In the previous article, we covered two key points:

• Why predictions change

• When the model changes (or doesn’t) in weekly operations

Now we get to the heart of this series:

The relationship between Validation (CV) and training.

When looking at the P123 screen, many people get the following impression:

Is the model continuously improving while Validation is running?

There is actually a common misconception here.

To cut straight to the conclusion:

Validation is not “where the model gets updated.”
Validation is where the model gets scored.

Understanding this distinction makes the behavior of CV immediately clear.

(1) The Role of Validation Is “Scoring”

The basic structure of machine learning consists of three stages:

1. Training

2. Validation

3. Prediction

Of these, the role of Validation is to measure model performance.

In other words, the validation data is data used for scoring the model.

If you were to use the validation data to:

• modify splits, or

• update leaf values,

you would effectively be training on future data.

This is data leakage.

For this reason, the basic rule is simple:

Do not train on validation data.

What the Paid Section Deeply Explores:

（Free Section)
(1) The Role of Validation Is “Scoring”

（Paid Section)
(2) So Why Does “Training” Run During Validation?
(3) What Validation Is Actually Deciding
(4) And the Model Used for Prediction Is Built Separately
(5) Summary
(6) Next Up

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Why Validation Is Difficult to Understand

Introduction

In the previous series, we explained the learning logic of LightGBM and Extra Trees, along with diagrams.

Starting from this article, we begin a new series that comprehensively covers validation, titled “The True Role of Validation (CV)”
We plan for this series to consist of approximately 6 articles in total.

The fundamentals of AI consist of Training and Inference.

However, in the AI Factor menu screen of Portfolio123 (P123), the interface is divided into “Validation” and “Prediction,” and there is no menu labeled “Training.”

Why is it structured this way?

One of the topics that readers most often find confusing is the relationship between Validation and Training.

Understanding this relationship is one of the key hurdles that beginners of P123 AI factors need to overcome.

By reading this series, that confusion should become much clearer.

When using P123, the following two ideas are often mixed together:

1. When the number of Validation splits (CV: Cross-Validation) increases, what exactly increases?

2. When predictions change every week, is it because the model itself changes every week?

Let us first present the conclusion.

The more CV splits there are,

the more times models are created for evaluation.

When Weekly predictions change every week,

it does not necessarily mean the model itself has changed.

Simply separating these two ideas already helps clarify the relationship among

• Training

• Validation

• Prediction

considerably.

(1) Two Terms We Should First Distinguish

In this series, we first separate the following two concepts:

• The model changes

• The prediction changes

These two ideas sound similar, but in reality they are completely different phenomena.

Much of the confusion begins here.

“The Model Changes” and “The Prediction Changes” Are Different Things

— Even with a fixed model, predictions move: stocks move inside the trees

What the Paid Section Deeply Explores:

（Free Section)
(1) Two Terms We Should First Distinguish

（Paid Section)
(2) What Does It Mean When the Model Changes?
(3) What Does It Mean When the Prediction Changes?
(4) Stocks Move Inside the Trees
(5) Two Patterns of Weekly Operation
(6) An Important Point About the 3MRel Target
(7) Next Up

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Introduction

In this Substack “AI-Driven Quant Investment Strategies”, this series of posts is planned to be structured as follows, and this post is the second one.

Visualizing the Learning Logic — ExtraTrees & LightGBM

Most ML comparisons focus on “which is more accurate” or “how results differ.”
This series instead focuses on the learning logic itself: what the algorithm is doing under the hood.

Planned total: ~11 posts (subject to change)

• LightGBM learning logic: 4 posts

• ExtraTrees learning logic: 3 posts

• Comparing LightGBM vs. ExtraTrees learning logic: 2 posts (← Now we are here)

• Validation and training: 2 posts

Assumptions for this series

• Target: 3MRel (3-month relative return)

• Universe: S&P 500

• Features and hyperparameters: We won’t deep-dive here (planned for later series)

Comparing the Learning Logic of LightGBM and ExtraTrees (2/2)

Up to 8th post in this series, we have visualized the learning logic of LightGBM and Extra Trees separately. Starting with prior (9th) post, we summarize their learning logic in a comparative format.

Introduction: When does it swing back to “LightGBM tends to win”?

In the previous post, we dug into LightGBM’s weakness: when features are few and noise is large, LightGBM’s strength of “smartly chasing residuals” can turn into a weakness by chasing noise as well.

This time, we explain the opposite: the cases where LightGBM’s strengths show up.
So—when does LightGBM tend to win?

(1) When the number of features increases and “good split candidates” increase

When conditions align, it often swings back to the side where “LightGBM tends to win.” The key point is that LightGBM’s strength—its ability to target residuals and improve step-by-step—is most powerful when it is applied to reproducible signal, not noise.

When features are few, LightGBM tends to dig into the same variables repeatedly and create “past-only thresholds.”
But when the number of features increases:

• you can explain the data from many different angles,

• you don’t rely too heavily on a single feature, and you can build more natural splits,

so boosting’s “smart correction” capability starts working more cleanly.

(In stock terms: it tends to be strong when you have a full set of factors—fundamentals, value, quality, momentum, positioning/flow, etc.)

What the Paid Section Deeply Explores:

（Free Section)
(1) When the number of features increases and “good split candidates” increase

（Paid Section)
(2) When noise decreases (= the target and features become more stable)
(3) When regularization, constraints, and validation design are working properly
(4) When you want to “hit the top bucket hard” (a regime with clear signal)
(5) When you have enough data and it does not collapse in walk-forward testing
(6) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more

This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:

AI-Driven Quant Investment Strategies

Introduction

In this Substack “AI-Driven Quant Investment Strategies”, this series of posts is planned to be structured as follows, and this post is the second one.

Visualizing the Learning Logic — ExtraTrees & LightGBM

Most ML comparisons focus on “which is more accurate” or “how results differ.”
This series instead focuses on the learning logic itself: what the algorithm is doing under the hood.

Planned total: ~11 posts (subject to change)

• LightGBM learning logic: 4 posts

• ExtraTrees learning logic: 3 posts

• Comparing LightGBM vs. ExtraTrees learning logic: 2 posts (← Now we are here)

• Validation and training: 2 posts

Assumptions for this series

• Target: 3MRel (3-month relative return)

• Universe: S&P 500

• Features and hyperparameters: We won’t deep-dive here (planned for later series)

Comparing the Learning Logic of LightGBM and ExtraTrees (1/2)

Up to this point, we have visualized the learning logic of LightGBM and Extra Trees separately. Starting with this post, we will summarize their learning logic in a comparative format.

Introduction: Why Extra Trees often performs better than LightGBM when features are few and the data is noisy

In one sentence, the reason Extra Trees tends to outperform LightGBM in regimes with few features and/or high noise is this:

Extra Trees has an advantage on the “average away the wobble” side in situations where LightGBM’s “ability to aggressively optimize and hit the target (sharp optimization)” is more likely to backfire.

Let’s unpack this in an investment-strategy context.

(1) When there are few features, LightGBM can “over-dig the same small set of features”

In a world with few features, the model has limited “tools” available.

LightGBM (boosting) tends to reuse features that seem effective and create finer and finer conditional splits in order to reduce residuals.

As a result, it may start assigning meaning to tiny fluctuations in those few features (thresholds that happened to work), making it easier to over-optimize to the past.

Stocks are especially noisy, so the more you try to precisely “hit” using a small number of features, the more likely you are to pick up accidental boundaries—meaning the model is more likely to collapse in the future.

On the other hand, Extra Trees:

• randomizes both features and thresholds,

• builds many trees and averages them,

so even with only a few features, it tends to rely less excessively on any single tree (i.e., it tends to be less biased toward one fragile structure).

What the Paid Section Deeply Explores:

（Free Section)
(1) When there are few features, LightGBM can “over-dig the same small set of features”

（Paid Section)
(2) When noise is large, LightGBM is more likely to “chase noise as residual”
(3) The “power of averaging” works: variance goes down (= predictions become more stable)
(4) With few features × high noise, differences show up more clearly in ranking-based operation
(5) Summary

Pricing Plans

To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.

• Monthly Plan: $8 / Month, Flexible starter option

• Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)

Subscribe now and advance to the next level of AI Quant Strategy!

Subscribe now

The following section is available to paid subscribers only.

Read more