Did you ever meet a smart, charming new friend and bond immediately, only to find out later that friend was untrustworthy? Backtesting systems can be like that new friend. You'll enjoy spending time with that person, but you aren't going to turn over all your investment decisions to that friend.
For several days the first week in February, I watched my orders to buy back credit spreads and lock in profits sit unfilled day after day. Those orders were at times set $0.10 to $0.15 above the mark or mid-price on my SPX credit spreads. That mark or mid-price was showing up as anything from negative values--irrationally suggesting that I could buy back the spread for a credit--to a nickel or dime.
I wasn't surprised the orders weren't filling. The absolute values on the deltas of the sold strikes were in the 4.00-5.50 range during that entire time. Through long experience, I've found that SPX market makers aren't going to take my $0.20 offers for those 10-point SPX spreads until the absolute values on the sold strikes drop into about the 3.00-3.50 range. It doesn't matter what the mark or midprice of those spreads might be: they're just not likely to fill those orders.
For a number of days, the orders remained unfilled and the risk remained open. A big move after the jobs numbers could have forced an adjustment or turned a theoretically profitable position into one that hit maximum loss. Fortunately, I was able to close out those spreads, locking in profit and avoiding any risk if the markets should suddenly roll into a swift descent.
What if I were backtesting a system that automatically exited and locked in profit as soon as the spreads narrowed to $0.20? Maybe I even accounted for the likelihood that the SPX spreads weren't likely to fill at mark or midprice at all times, although there are certainly times when they do. For the sake of honesty and to err on the conservative side, I could watch until the spread narrowed to $0.05, perhaps, but still charge the trade the $0.20, accounting for the kind of slippage we see all the time as options traders. However, if I didn't know from long experience that quirk that I talked about in the SPX spreads, my results still wouldn't be all that trustworthy. Some of the times my backtest showed me safely closing out those spreads and locking in profits might have experienced much different real-life results. I might not have been able to close the spreads, and the SPX could have barreled toward the sold strikes, eventually forcing me to close the position at a max loss if my efforts to adjust the trade weren't successful.
If I were backtesting a system using the more liquid SPY options, I'd probably be able to trust the results better. The RUT? Depends on the year and, sometimes, the moment. I've traded iron condors on the RUT for years, some of which you pretty well had to pay the offer and sell at the bid, or not far off. Some years, the fills prove to be excellent. I used to trade SOX options, but about the time the PHLX suddenly started charging for feed without adequate provisions for the changeovers, and we all got stuck in SOX options trades, temporarily without any way to price those spreads, those options became less liquid. I wouldn't have trusted any backtesting system's results on the SOX's options. Perhaps they're again more liquid. I don't trade them any longer and can't be sure.
I used to trade Enron options. Those options had some funky values about the time inventories numbers would come out every week.
That's the point. Unless you trade a vehicle long enough to know its peculiarities, you can't trust backtested results, no matter what you do to build in some slippage in that system.
Charming but untrustworthy friends are still charming, unless you bestow too much trust on them. Backtesting systems are still interesting and helpful, unless you bestow too much trust on them, too. Backtesting a trade can absolutely reveal flaws in a way of trading or adjusting a trade. For example, it could show you that your intention to hedge deltas at the end of every single day is going to cost so much in commissions that it will weigh down your trade and sink it. It can show you that your intention to carry a certain trade into option expiration week will result in a trade that is no longer adjustable if it goes wrong. It can show you that the trade that worked beautifully on the RUT or the SPX fails abysmally on the SPY and IWM.
What it can't do is show you reliable profits. Too many times, I've heard from traders who have backtested a particular way of trading, even through at least 30 months, as they should do to approach any kind of statistical reliability. The trade performed so beautifully on the backtests that they plowed a lot of money into the trade. You've guessed the rest. Their refrain is almost always something along the line of, "Why didn't it work?"
If you've backtested a way of trading or adjusting that's produced spectacular results, do keep live trades small for the first few months trading it. Try to weather at least one bad trading month--the kind of month that provides many a trader with a "teachable" moment--before you scale up in size.