How to Test a Trading Tool Before You Recommend It

To test a trading tool properly you write down a falsifiable hypothesis about it before you open the software, fix the exact procedure and the metrics you’ll record, run it for a pre-committed length of time, and log observations with dates as you go. If the tool’s output can be independently recomputed against data you already hold, the test is stronger still — because then the trial can fail, and a test that cannot fail is marketing.

Counted from the trial logs as they stand now: three trials opened, none completed, none abandoned, across 42 dated log rows — 18 in the TradingView file, 15 in the Alpaca file, 9 in the Kraken file, counted to the pass dated 14 September 2026. The count is of rows in each file’s ## Log table, not of distinct dates; those two numbers differ, and the table below says where. That tally is the point of this page — a protocol is only worth something if its empty cells are as visible as its full ones, and “opened” versus “completed” is exactly the distinction a tool review is most tempted to blur.

Trial	Opened	Pre-committed duration	Dated log rows	Status
TradingView	2026-08-23	30 calendar days, ends 2026-09-22	18 (16 distinct dates)	Open — counted to the 2026-09-14 pass, its day 23 of 30
Alpaca paper trading API	2026-08-25	30 calendar days, ends 2026-09-24	15 (15 distinct dates)	Open — counted to the 2026-09-14 pass, its day 21 of 30
Kraken $5,000 prop eval	2026-09-04	Open-ended; publishable at 10+ logged sessions	9 (7 distinct dates)	Open — counted to the 2026-09-14 pass, its day 11
Completed	—	—	—	0
Abandoned	—	—	—	0

All three logs carry a row dated 14 September 2026, and that is the pass every figure above is counted to — not a claim that no later pass exists by the time you read this, which is the whole point of stamping it. The row counts were recounted from those files this session rather than carried over. The Kraken file is the reason the row count and the date count come apart: three of its nine rows are dated 2026-09-04, the day the eval was bought, because terms mining, an API check and the purchase itself were logged separately. The TradingView file has two such dates of its own — 25 August appears twice (a pre-open pass and a post-close pass), and 4 September appears twice (a pre-open pass, plus the session itself logged retrospectively on the 7th). Eighteen rows against sixteen dates is exactly those two duplicates. Those two figures drift apart constantly, and a tally that hides the gap is a tally you cannot check.

That gap is not a defect in the tally — it is the tally doing its job. The previous version of this page put the same three trials at 17, 14 and 8 rows and their last pass at 11 September. Every one of those numbers was correct when it was written and wrong within a working day, because each file gained a 14 September row. That is the failure mode this page exists to make visible: a tally is a measurement with a timestamp, not a fact about the project, and a page that carries one forward without recounting it is reporting history as if it were the present.

The counts will change, and the public record of them changing is the evidence. A log that only ever appears already full is not a log, it’s a brochure.

Why the protocol comes before the review

Almost every “we tested it” claim in the trading-tools space is unfalsifiable. It describes features, quotes a pricing page, and reports a favorable impression. The viral Claude trading-bot playbook is the same shape in automated form — a loop that can only ever terminate on a good-looking result. There is no hypothesis, so there is no way the trial could have gone badly, and therefore no information in the fact that it went well.

The fix is not more rigor in the write-up; it is four commitments made in advance — state what would make the tool fail, fix the method, pre-commit the duration, log with dates. The template below is their operational form.

The trial record: four header fields plus a log

Every trial on this desk is opened as a file in pipeline/desk-log/trials/ from a fixed template. Four header fields — hypothesis, method, start-and-duration, metrics — and beneath them one running dated table:

Field	What it captures	Why it is fixed in advance
Hypothesis	One sentence phrased so it can be wrong — a specific claim about what the tool will do, not “evaluate the screener”	If you cannot describe today the observation that would refute it, the trial has no failure state
Method	The exact repeatable procedure: which preset or settings, which universe, what is recorded each pass	Stops results being produced by quietly changing the procedure until the numbers cooperate
Start and duration	A start date and a fixed run length	Otherwise the trial ends on the day the results look good
Metrics	What is measured — candidates per day, false-positive rate, agreement with an independently computed benchmark	Metrics chosen afterward are chosen to flatter
Log (the table, not a header field)	A dated table of date, observation, supporting data, appended as the trial runs	A dated trail is what separates a test from a recollection

The template is deliberately short. A heavy protocol produces no completed trials, and a trial that doesn’t finish teaches nothing.

What makes a hypothesis testable

The strongest hypotheses compare the tool’s output to something you can compute yourself. That turns a subjective impression into an arithmetic check.

A worked example, drawn from a candidate hypothesis our week-34 log drafted for TrendSpider — a tool no trial has been opened on: “TrendSpider’s automated trendline and moving-average detection on daily SPY/QQQ charts reproduces the 200-day-SMA levels our own computation produces.”

That works because this desk already computes 200-day averages on those two tickers from its own data pull, so the comparison has a right answer existing independently of the tool. Either the levels reconcile or they don’t.

The three open trials are built the same way. The TradingView one tests reliability, screener output against our existing watchlist process, and alert-delivery latency; the Alpaca one tests REST latency and order-lifecycle timing against a scripted daily probe; the Kraken eval tests whether a prop product’s risk-rule arithmetic behaves as its own documents describe. Each is recorded daily against pre-committed metrics rather than summarised at the end.

Nothing about any of them is asserted here. A hypothesis is a question, not a finding.

The validation standards a credible trial has to meet

Where a trial involves a tool’s backtesting or signal-generation output, four standards apply — not product-specific, just what separates a result from a curve-fit.

Test out of sample

Any rule tuned on a stretch of history will describe that stretch well; the only meaningful question is how it behaves on data it was not fitted to. Hold back a period, or step forward re-fitting as you go, and report that — not the in-sample figures the tool shows by default.

Insist on a real sample size

A handful of trades tells you nothing about a rule, because the per-trade variance swamps the per-trade edge. Treat 100 trades as a floor — the point below which the exercise is not worth doing — and several hundred as where a mean return starts to be distinguishable from noise at all.

That is not a round number picked for feel. The standard error of a mean falls with the square root of the sample, so the trades you need to reach a given t-statistic is fixed by one ratio: the per-trade standard deviation divided by the per-trade edge.

# Illustrative: trades needed to reach t = 2 on the mean trade.
# noise_ratio = per-trade standard deviation / per-trade edge.
def trades_needed(noise_ratio, t=2):
    return round((t * noise_ratio) ** 2)

for r in (5, 8, 10, 15, 20):
    print(r, trades_needed(r))      # -> 100, 256, 400, 900, 1600

So 100 trades clears t = 2 only in the friendly case where a trade’s standard deviation is five times its edge; at a more ordinary ten-to-one it takes 400. That is the same floor our trading journal guide states — one number, phrased the same way in both places. It is arithmetic about sample means, not a claim about any strategy’s returns, and fewer trades over a longer period is not a substitute: the count is what constrains the uncertainty.

Model the costs that actually apply

A backtest with no commissions, no spread, no slippage and fills at the exact close simulates a market that does not exist. Whatever the tool assumes by default, find the setting, make it realistic for your instrument and size, and re-run. Strategies that survive the first run and die on the second are common, and that is the test working — step two of the seven checks we run before a strategy trades a dollar. Our mid-versus-taker comparison on real options quotes is the worked example: the fill assumption alone flipped a credit spread from profitable to unprofitable.

Forward test before capital

The last step runs the rule on live data, in real time, without money on it, for a pre-committed period. It is the only stage that catches what a historical test structurally cannot: data that arrives late or revised, signals that fire when you can’t act, and the gap between what you said you would do and what you do.

What we will and won’t say about a tool

Three rules govern how a trial becomes a page.

No first-hand claim without a log. “We ran it” is publishable only when a dated trial file supports it. Where no completed trial exists, the tool gets written up third-person from the vendor’s own published material, and the page says that no finished trial stands behind it. The trial logs are what make that checkable rather than merely stated.

Vendor specifics get verified at the time of writing. Prices, fees and feature availability change without notice, so any such figure is checked against the vendor’s live page in the session the page is written — or described without a number. Our look at the 2026 prop-firm crackdown shows why: the most-repeated third-party figure for one firm’s challenge rules disagreed with the firm’s own terms page on the day we read both.

A negative result gets published. A protocol that only surfaces trials confirming the recommendation is a filter, not a method. The log is the deliverable, and it’s public whichever way it comes out.

Frequently asked questions

How long should a trading tool trial run?

Long enough to cover varied conditions, and fixed before you start. The number matters less than the pre-commitment — a duration chosen while you can already see the results is not a duration, it’s an exit.

What sample size does a backtest need to be credible?

One hundred trades is the floor, and several hundred is where a mean return separates from noise — the arithmetic is above. Below the floor, ordinary variance produces impressive-looking equity curves from rules with no edge at all.

Can you test a trading tool on a free trial?

Often yes — a free tier is enough for a reconciliation-style hypothesis, where you check whether the tool’s output matches something you compute independently. It is usually not enough for a forward test, which needs the pre-committed duration to run uninterrupted.

What’s the difference between a tool trial and a strategy backtest?

A trial evaluates the instrument: does the software do what it claims, accurately and repeatably. A backtest evaluates a rule. Conflating them is how a good tool gets blamed for a bad strategy, and how a bad tool escapes scrutiny because the strategy happened to work.

Why publish a trial log with no results in it yet?

Because the alternative is publishing the protocol only once it has produced flattering results, which tells a reader nothing about how often it produces unflattering ones. Three trials are open and none has finished — no completions and no abandonments across 42 dated log rows — which is exactly what the tally at the top of this page says.

---

Educational content only. Nothing here is a recommendation to buy or sell any security or to purchase any product. No tool named above has a completed trial behind it, and no finding about any of them is reported here.