Guide

Strategy optimization

Last reviewed: 26 September 2026·Tradelyze

Strategy optimization is backtesting the same trading strategy many times with different input settings, then keeping the settings whose results look best. Each backtest is called a trial. The more trials a search runs, the more likely its best result fits luck in past prices rather than an edge that will repeat.

New to this? Start with in-sample vs out-of-sample.

What is trading strategy optimization?

Trading strategy optimization is a search for the best settings of a strategy whose rules stay the same. In Pine Script those settings are the strategy's inputs, such as an RSI length or a stop-loss percentage. In TradingView they appear on the Inputs tab of the strategy's settings. Commission, slippage and order size are different. They are strategy properties, set in the script's strategy() declaration or on the Properties tab, and TradingView strategy properties explains them.

One backtest with one combination of inputs is a trial. The number used to judge each trial is the objective, such as net profit or maximum drawdown. Maximum drawdown is the largest fall from an account peak to a later low. Every combination the optimizer is allowed to try makes up the search space.

A constructed example shows how quickly a search space grows. Suppose three inputs may each take a handful of values:

• RSI length from 10 to 30 in steps of 2, which is 11 values
• Stop loss from 1.0% to 3.0% in steps of 0.5%, which is 5 values
• Take profit from 2% to 6% in steps of 1%, which is 5 values

Testing every combination takes:

Adding a fourth input with 10 values turns 275 trials into 2,750. These figures are arithmetic on invented ranges, not measured data.

Optimization matters for your money because the best trial is chosen on the same price history it is scored on. Its backtest is an in-sample result, measured on the same data used to pick the settings. That makes it the most flattering number the whole search produced. It says little about future trading until the settings are tested on out-of-sample data, meaning prices the search never used.

Why does optimizing a trading strategy lead to overfitting?

Optimizing a trading strategy leads to overfitting because the optimizer keeps whichever settings scored best on one stretch of past prices. Part of every backtest score is luck. Overfitting, also called curve fitting, means settings that describe the random noise in that history rather than behavior that repeats. A setting can look excellent simply because its entries happened to line up with a few price swings that will not come again.

The more settings a search tries, the more chances some get to line up with noise. The search always reports the luckiest one. As a constructed example, suppose 100 settings of a strategy have no real edge and each has an even chance of a profitable backtest. On average about 50 of them would still show a profit, and the best of the 100 could look impressive. That arithmetic is invented to show the effect, not measured data. For a prop firm challenge, it means paying the fee on the strength of a number that was partly luck.

Three signs point to an overfit winner:

• Few trades behind the result. See how many trades a backtest needs.
• A best value surrounded by much worse neighboring values. See plateau versus spike.
• A sharp drop on data the search never used. Tradelyze's held-out test measures it, and so does walk-forward analysis when it re-tunes the settings in every window.

How many trials should an optimizer run, and why do more trials raise the bar?

No published source gives a correct trial count for a trading strategy; the right number balances two opposite risks. With too few trials, a search that learns from earlier trials never gets past its first random guesses. With too many, the best trial becomes more likely to be a lucky fit to past prices.

Traders tend to underrate the second risk. Bailey, Borwein, López de Prado and Zhu studied it in a May 2014 paper in Notices of the American Mathematical Society. The abstract posted with that paper states that the higher the number of configurations tried, the greater is the probability that the backtest is overfit.

Each extra trial is another chance for a setting to match noise. The same Sharpe ratio is therefore weaker evidence as the best of 300 trials than as the best of 30. Overfitting and sample size explains why the trial count, more than the number of inputs, sets that risk.

In Tradelyze, the Optimization trials setting sits in the Walk-Forward & Robustness step of the Run Optimization window. The same panel appears when you submit a new strategy. It can be Automatic or Custom. Automatic sizes the budget from how many inputs your strategy searches, and never runs fewer than 60 trials. Custom takes a number within the range printed under the box. A separate Walk-forward trials setting sets how many trials run inside each walk-forward window. It applies only when walk-forward re-tunes the settings in every window. Tradelyze, not you, picks the walk-forward method: when a strategy qualifies, it scores the search's overall best settings, normally the recommended ones, on periods cut from one run, which costs no extra trials.

Two account conditions change what that setting does. If walk-forward and robustness settings are not enabled for your account, the form is read-only and runs use the platform defaults. If you have never saved the settings, runs use the server's configured values until you do. That server trial count can be smaller than the Automatic budget. Either way, read the trial count in the results card header rather than assuming the form's value ran.

Two things about a small Custom budget quietly weaken a run:

• Every search spends its first 10 trials on random settings before its model proposes any. A small budget leaves few trials for the model to guide, and a budget of 10 trials or fewer would be random search only.
• Any budget under 50 trials skips the parameter-sensitivity check, and the form warns about it. That check re-tests settings close to the winning ones.

A check that does not run shows NOT RUN on the robustness card. The robustness verdict can then be ACCEPTABLE at best; see parameter sensitivity.

A practical order of work helps. Search fewer inputs before adding trials, and fix the inputs you would never change. Write down how many trials produced the result you are about to trade.

What should an optimizer maximize?

An objective is the number an optimizer uses to judge each trial, such as net profit. No primary source names one correct objective for a trading optimizer, but single objectives fail in predictable ways. An optimizer rewards exactly what it is told to reward, so each objective's blind spot becomes the winning trial's weakness.

Scoring every trial on several objectives at once shows the trade-off instead of hiding it. This is called a multi-objective search. A minimum trade count also stops the search from crowning a two-trade fluke. Tradelyze's optimizer searches on profit, Sharpe ratio and maximum drawdown together; profit factor and win rate appear only as results.

For a prop firm challenge, give drawdown the most attention. The firm ends the account at a fixed loss, however profitable the strategy was. Prop firm rules and backtest metrics covers which numbers predict a breach.

What is multi-objective optimization and a Pareto front?

Multi-objective optimization judges each trial on two or more objectives at once instead of one. A trader usually wants higher profit and a smaller maximum drawdown, and those two goals pull against each other. Rather than forcing them into a single number, a multi-objective optimizer keeps every trade-off that is not clearly worse than another.

One trial dominates another when it is at least as good on every objective and better on at least one. The Pareto front is the set of trials that no other trial dominates.

Take three constructed trials. Trial A made 18% profit with a 12% drawdown. Trial B made 11% with a 5% drawdown, and trial C made 10% with a 9% drawdown. B dominates C, because B made more profit with a smaller drawdown. A and B do not dominate each other, because A made more money and B risked less. Both sit on the Pareto front. Choosing between them depends on how much drawdown you can accept.

A Pareto front is useful to a prop firm trader because the firm closes the account at a fixed loss. A trial with less profit and a much smaller drawdown can be the better choice for a challenge. A ranking on profit alone would still put it lower.

How does Tradelyze's optimizer search?

Tradelyze searches with TPE, short for Tree-structured Parzen Estimator. TPE is a Bayesian optimization method: it learns from the trials already run which settings are worth trying next. Tradelyze uses the version built into Optuna, an open-source optimization library. Every trial is a full backtest of your Pine Script, and TPE scores each one on three separate objectives rather than on a single blended number:

• Maximize profit, as a percentage of starting capital
• Maximize the Sharpe ratio, which is average return divided by how much returns swing. The figure used is the bar-by-bar one the report labels Sharpe (Bar) — not the Sharpe (Daily) tile beside it, and not TradingView's monthly-return Sharpe ratio
• Minimize maximum drawdown

Before a trial's profit and Sharpe reach those objectives, Tradelyze scales both down when the trial took more than one trade per ten bars, so a setting that trades far more often than the data can support cannot win on volume alone. Drawdown is passed through unchanged.

TPE has no generations, populations or breeding. It proposes one trial at a time:

1. The first 10 trials are chosen at random from your ranges.
2. From the 11th trial on, TPE splits the finished trials into a better group and a worse group. With three objectives, it sorts them by Pareto front rather than by one blended score.
3. It builds a probability model of the settings in each group and proposes settings that are likely under the better model and unlikely under the worse one. Tradelyze's version models inputs jointly rather than one at a time. Inputs used together on the same trials are modeled as one group.
4. That trial is backtested, and its result informs the next proposal.

The Automatic trial budget depends on how many inputs are searched. Every budget starts with 10 random trials; the rest are proposed by the model.

Tradelyze's Automatic budget is never fewer than 60 trials. A budget of 10 trials or fewer never reaches the model, so its result is the best of a batch of random draws. How grid search, random search, genetic algorithms and TPE compare is covered under Going deeper.

Tradelyze switched from NSGA-II, a genetic algorithm, to TPE on 24 September 2026, so results produced before that date came from NSGA-II. The switch followed an internal comparison on three strategies at the same trial budget, in which TPE reached better settings in 12 of 15 paired runs. That is Tradelyze's own measurement and has not been independently verified. Tradelyze implementation. No primary source.

Tradelyze's recommended settings are chosen from every completed trial, not only from the Pareto front. After the search has finished, Tradelyze orders the completed trials with one combined ranking of the same three numbers: profit, the Sharpe (Bar) figure and maximum drawdown. That combined ranking is not a fourth objective and the search never optimizes it — TPE searches on the three separately, and the ranking exists only to put the finished trials in order, pick the recommendation and fill the Top Trials table. In the ranking, profit and Sharpe count for less when few trades support them. Trials with fewer than 5 trades are left out unless no trial reached 5.

For each prop firm you select, the recommendation is the highest-ranked trial that passes all of that firm's checked rules. When none passes, the highest-ranked remaining trial is shown as the closest attempt rather than a pass. When no prop firm rule set applies, the recommendation is simply the highest-ranked trial.

The combined ranking only puts trials in order. Tradelyze does not display it, and its size is not a quality score. A profitable strategy tested on a handful of trades can rank first with a score close to zero. Judge a trial by its own metrics and its trade count, and judge the recommended settings by the checks that follow the search. Of those, only the held-out test runs these exact settings on data the search never saw; walk-forward analysis answers a different question.

The trial count of a Tradelyze run depends on how many inputs are searched and on the Optimization trials setting. The number that actually ran is printed in the header of the optimization results card, in the form N trials (M feasible).

How should I set Fixed, Min, Max and Step values and ordering constraints?

Set each range to values you would genuinely trade. Use steps large enough that neighboring values behave differently. Fix every input you do not need to search. In Tradelyze these controls are in Step 1: Configure Parameters of the Run Optimization window.

That editor appears when the strategy has no optimization trials yet. It lists every input the search can change.

For true-or-false and list inputs, click the values the search may use instead of typing a range. A constructed example: an RSI length with a script default of 14, set to Min 8, Max 30 and Step 2, gives 12 values. The same range with Step 1 gives 23 values, nearly doubling that input's share of the search with lengths only one bar apart.

An ordering constraint keeps one input smaller than another, such as a fast moving-average length smaller than a slow one. In Step 2: Ordering Constraints, a rule reads left < right or left ≤ right. Tradelyze suggests a rule when two input names form a known pair. The pairs are fast and slow, short and long, min and max, low and high, entry and exit, and oversold and overbought.

A suggested rule can be switched off but not deleted, and a rule you add yourself can be removed. The form rejects a rule that compares an input with itself, and two rules that contradict each other.

When a proposed trial breaks an ordering constraint, the optimizer moves one of the two values. It usually raises the right-hand value, so the pair is back in order inside its ranges. An input you fixed is never moved to satisfy a rule. If the Min and Max values make the order impossible, the trial runs as proposed and the rule is not enforced. Check that the two ranges can actually be put in order.

How do I read the Top Trials table?

The Top Trials table lists up to five of the best-ranked trials from the search. It lets you compare the recommended settings with their closest alternatives. Tradelyze ranks them the same way it makes its recommendation, combining profit, Sharpe ratio and drawdown. Profit and Sharpe are weighted by how many trades support them. A losing result is never discounted that way.

Trials with fewer than 5 trades are left out unless no trial in the search reached 5, and identical settings appear only once. If the recommended settings are not among the five, Tradelyze adds them as an extra row at the bottom. The table appears even when no trial passed your firm's rules, so you can see the closest attempts. Clicking a column heading re-sorts the rows; the # column keeps each trial's rank.

In the Top Trials table, a PF of 0.00 can mean no winning trades or no losing trades. A trial with no losses has no defined profit factor. Read the Win Rate % beside it, which will normally be 0% or 100% in those two cases.

Clicking a row opens that trial's full settings and four tools. Copy Pine Defaults copies the settings as Pine Script input lines. Download Trades saves the trial's trades as a CSV file. Verify Trial prepares a TradingView re-run. The TradingView Verification panel compares an uploaded TradingView trade list with the trial's backtest trades. Download Trades and the comparison panel need a trade list that was kept for that trial.

Both rows are invented to show the ranking logic and do not come from a real run. For a prop firm trader, trial 41's 6.5% drawdown also leaves more room than trial 88's 18.0% under a firm's loss limit.

What does Feasible YES or NO mean in Top Trials?

Feasible YES means every rule of the selected prop firm's rule set was checked against that trial's trades and passed. Feasible NO means at least one rule failed, or at least one rule could not be checked. A rule normally cannot be checked when that trial's trade list was not kept. The table shows both cases as NO.

Every row normally reads NO when the run had no prop firm rule set to check. A column of NO values is therefore not proof that every trial broke a rule. When you select several firms, the results page still shows one Top Trials table. Its Feasible column follows one firm's rule set, normally the first you selected. Each firm's own card and its Rule Results table show which limit failed; what Qualifies and Not Feasible mean covers that card.

Feasible answers the firm's rules and nothing else. The feasible count in the card header also leaves out trials with fewer than 5 trades. A YES row can therefore be missing from that count. Neither covers evaluation time limits, news-trading restrictions or weekend-holding rules, which Tradelyze does not check. Both are also measured on the same in-sample data the search tuned on.

What does Verify Trial do?

Verify Trial checks whether a trial's result reproduces in TradingView before you rely on it. Tradelyze makes a copy of your Pine Script with that trial's settings written in as the input defaults. It then opens a new strategy draft, named after the trial, with the modified script attached.

You run that script in TradingView, export its trade list, and upload it with matching market data. Submitting the draft costs 1 credit and starts a normal run, beginning with Baseline Matching. Baseline Matching compares Tradelyze's backtest with your TradingView trades.

Before exporting, check that every input on the TradingView Inputs tab shows the value listed in the Verify Trial window. An input written in an unusual form in the script may keep its old default. For a quick comparison without a new submission, expand the trial's row and use Upload TV CSV in the TradingView Verification panel.

A trial that reproduces in TradingView has shown that the backtest is faithful, not that the settings will keep working. The settings are still in-sample. The one check of exact settings on data the search did not use is the held-out test, and it runs each firm's recommended settings, not other trials. Read it and the walk-forward result before trading. If a later check fails, what to do when a strategy fails validation covers what to change and what to leave alone.

How do grid search, random search, genetic algorithms and TPE differ?

Grid search, random search, genetic algorithms and TPE differ in how they pick the next combination of settings to test. None of them changes the fact that the winner is chosen on past data, so none of them protects against overfitting on its own.

Bergstra and Bengio's paper, Random Search for Hyper-Parameter Optimization, reports that randomly chosen trials were more efficient than trials on a grid. They tested machine-learning models, not trading strategies, so read the result as evidence about search efficiency rather than about finding a trading edge.

A genetic algorithm borrows the idea of natural selection. Each round of candidate settings is a generation, and each new generation is bred from the better members of the one before it, with small random changes mixed in.

NSGA-II, short for Nondominated Sorting Genetic Algorithm II, is a genetic algorithm built around the Pareto front. Deb, Pratap, Agarwal and Meyarivan introduced it in IEEE Transactions on Evolutionary Computation in April 2002. It sorts each generation into fronts: first the trials no other trial dominates, then the trials dominated only by those, and so on. It keeps the trials on the best fronts, and favors trials spread out along a front so the search does not crowd into a single trade-off. Tradelyze searched with NSGA-II, through Optuna, until 24 September 2026.

TPE, short for Tree-structured Parzen Estimator, is a Bayesian optimization method. Bergstra, Bardenet, Bengio and Kégl introduced it in 2011, in Algorithms for Hyper-Parameter Optimization. Instead of breeding, it splits the finished trials at a threshold into a better group and a worse group, and fits a probability model to the settings in each. The next trial is a setting with a high probability under the better model and a low probability under the worse one. The 2011 paper judged each trial on a single number. Optuna's version also handles several objectives: with Tradelyze's three, there is no single threshold, so it sorts the finished trials into the better and worse groups by Pareto front instead. It draws random trials until a set number have finished; Tradelyze sets that number to 10. Like Bergstra and Bengio's random-search study, the 2011 paper tested machine-learning models, not trading strategies.

Stage 3 · step 10 of 18. Next in the learning path: Overfitting and sample size

Frequently asked questions about strategy optimization

Questions about Tradelyze files, credits and accounts are answered in the Learn FAQ.

Sources

• K. Deb, A. Pratap, S. Agarwal and T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Transactions on Evolutionary Computation 6(2), April 2002, pages 182–197, DOI 10.1109/4235.996017.
• Optuna documentation, optuna.samplers.NSGAIISampler, which describes the class as a multi-objective sampler using the NSGA-II algorithm and cites Deb et al. (2002); retrieved 15 September 2026.
• James Bergstra and Yoshua Bengio, Random Search for Hyper-Parameter Optimization, Journal of Machine Learning Research 13, 2012, pages 281–305. The study concerns machine-learning models, not trading strategies.
• James Bergstra, Rémi Bardenet, Yoshua Bengio and Balázs Kégl, Algorithms for Hyper-Parameter Optimization, Advances in Neural Information Processing Systems 24 (NIPS 2011), pages 2546–2554. Section 4 introduces the Tree-structured Parzen Estimator (TPE), which models the settings of better and worse trials with two densities and favors settings likely under the first and unlikely under the second. The study concerns machine-learning models, not trading strategies.
• Optuna documentation, optuna.samplers.TPESampler, which describes the class as a sampler using the TPE (Tree-structured Parzen Estimator) algorithm, states that random sampling is used instead of TPE until a given number of startup trials finish, and cites Bergstra et al. (2011); for its multi-objective version it cites Multiobjective Tree-Structured Parzen Estimator for Computationally Expensive Optimization Problems (DOI 10.1145/3377930.3389817) and Multiobjective Tree-Structured Parzen Estimator (DOI 10.1613/jair.1.13188); retrieved 26 September 2026.
• David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu, Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance, Notices of the American Mathematical Society 61(5), May 2014, pages 458–471, DOI 10.1090/noti1105. The printed Notices article carries no abstract; the sentence quoted on this page comes from the abstract posted with the paper in Western Michigan University's ScholarWorks repository, math_pubs/40, retrieved 15 September 2026.
• Tradelyze optimization implementation, reviewed 26 September 2026: since 24 September 2026, an Optuna TPE (Tree-structured Parzen Estimator) study, which replaced NSGA-II, created with directions maximize, maximize, minimize over the objective tuple (profit percent, Sharpe ratio, maximum drawdown), where the Sharpe is the bar-sampled, horizon-capped figure the UI labels Sharpe (Bar), not the daily or raw annualized figures carried beside it; the first 10 trials drawn at random before the model proposes any; the inputs modeled jointly rather than one at a time, in groups of inputs used together on the same trials (multivariate and group TPE); trials sorted into better and worse groups by Pareto rank; no generations or populations; profit and Sharpe scaled down before scoring when a trial takes more than one trade per ten bars; the combined ranking applied only after the search, never as an objective; an Automatic Optimization trials budget of 60, 90, 120 or 150 trials for 1 to 5, 6 to 15, 16 to 30 or more than 30 searched inputs, never fewer than 60; runs with no saved settings use the server's configured trial count; the switch from NSGA-II followed an internal comparison on three strategies in which TPE reached better settings in 12 of 15 paired runs at the same trial budget, an internal measurement that has not been independently verified and has no primary source; the recommendation chosen from every completed trial, not only the Pareto front; recommendation and Top Trials ranked on a trade-weighted combination of profit, Sharpe ratio and drawdown with a 5-trade floor; per prop firm, the highest-ranked trial passing that firm's checked rules; Feasible shown as YES or NO; the parameter-sensitivity check skipped below 50 trials; the per-window Walk-forward trials setting used only when walk-forward re-tunes the settings in every window, and the walk-forward method chosen by Tradelyze rather than by the user; the held-out test run on each firm's recommended settings; the Configure Parameters and Ordering Constraints editors; Verify Trial.
• The search-space arithmetic, the 100-setting luck example, the Pareto-front trials and the Top Trials rows on this page are constructed illustrations, not measured data.