What you will find in this post
Why customize your backtesting tool?
As I tool you in the post about backtesting framework, you should test a few tools before making a decision on which one you will backtest your trading strategies. In the end, to suit your needs, you will eventually have to customize them for you needs. My choice was QStrader, an it was not prepared to deal with intraday backtests, so I had to make some changes to make sure my tests are as real as possible.
The other reason it is good to customize your backtesting tool, is the learning experience. By coding your need into the backtest framework of your choice, you will gain knowledge in how to tweek your tool. If you desire a job in the industry of finance and quantitative trading, this skill will be valuable.
SimulatedExchange Class
This class was a part of the backtesting engine, but I decided to “take it out”, cause it would give me more control on the hours the exchange opens and closes. My intention is to develop strategies for Brazilian futures markets and US ETFs, which trade on different hours. Since the Exchange open and closing times where hardcoded into the framework as London time, I decided to create my own class for this task.
class SimulatedExchange(Exchange):
"""
The SimulatedExchange class is used to model a live
trading venue.
Parameters
----------
start_dt : `pd.Timestamp`
The starting time of the simulated exchange.
xchg_open_time : 'datetime.time'
Time exchanges opens
xchg_close_time : 'datetime.time'
Time exchanges closes
"""
def __init__(self, start_dt,xchg_open_time,xchg_close_time):
self.start_dt = start_dt
# TODO: Make these timezone-aware
self.xchg_open_time = xchg_open_time
self.xchg_close_time = xchg_close_time
def get_open_time(self):
return self.xchg_open_time
def get_close_time(self):
return self.xchg_close_time
Note: the above code is not the entire code. Just the changes made, for illustration purposes
Asset Class
The Asset Class exists in QStrader but it was not implemented at this time, so I decided to implement and also because I needed it… I have an intraday database of IBOVESPA FUTURES CONTRACTS and MINI-IBOVESPA FUTURES CONTRACTS. The Mini contract has a point value of $0.20 BRL and this is a problem for the backtest engine. Before this change, every point was equal to $1.00 BRL which provided me with unreal results. Also when dealing with cash avaliable for trading, the orders where being executed incorrectly.
I also need to deal with the margin and leverage factors. Both contracts have different margins and this affects the avaliable cash as well. This will have a greater impact in the backtesting engine and it is not implemented at this time.
class Future(Asset):
"""
Stores meta data about an future contract.
Parameters
----------
name : `str`
The asset's name (e.g. the company name and/or
share class).
symbol : `str`
The asset's original ticker symbol.
TODO: This will require modification to handle proper
ticker mapping.
point_value: `float`
Is the point value used for indexes and futures to convert to cash.
tier_size: `int`, optional
Assets minimum traded size
For equities usually 1
"""
def __init__(
self,
name,
symbol,
point_value,
min_tier_size = 1
):
self.cash_like = False
self.name = name
self.symbol = symbol
self.point_value = point_value
self.min_tier_size = min_tier_size
def __str__(self):
return self.symbol.upper()
def __repr__(self):
"""
String representation of the Equity Asset.
"""
return (
"Future(symbol='%s', point_value=%s, min_tier_size=%s)" % (
self.symbol, self.point_value,self.min_tier_size
)
)
def __lt__(self, other):
if isinstance(other, Future):
return self.symbol < other.symbol
else:
raise TypeError("Cannot compare 'Future' with non-Future object")
def get_point_value(self):
return self.point_value
def get_min_tier_size(self):
return self.min_tier_size
Intraday
As briefly explained above, the simulation engine is not prepared to deal with intraday data. This was by far the biggest change I had to make. The highlighted classes below are the ones that had to be created or changed for this customization to work, but be aware that some internal code (not listed here) had to be adapted in order for the backtest to run properly.
Frequency Class
This class was created to deal with intraday data and allow flexibility in dealing with intraday data. With it a I backtest in different intraday periods as I wish with just small changes.
At this point I did not make any changes into the TearSheet Reports, but since I am dealing with intraday data, some (if not all) the ratios need to be ajusted and calculed based on this information as well. On daily data the 252 day is the default value for the calculation, but for intraday data this must be adjusted.
The Frequency Class will also help dealing with the reporting adjustments.
class Frequency():
"""
Frequency object.
Parameters:
frequency: 'pd.Frequency'
Frequency as used in pandas Ex. 5T equals 5minutes
candle_type: 'str'
Options: 'time' and 'non-time' candles
xchg_open_time : 'datetime.time'
Time exchanges opens
xchg_close_time : 'datetime.time'
Time exchanges closes
"""
def __init__(self, frequency, candle_type, xchg_open_time, xchg_close_time):
self.candle_type = candle_type
self.frequency = frequency
self.xchg_open_time = xchg_open_time
self.xchg_close_time = xchg_close_time
dict_freq ={
'1T':1,
'3T':3,
'5T':5,
'15T':15,
'30T':30,
'60T':60,
'D':None
}
try:
self.freq_int = dict_freq[self.frequency]
except:
raise ValueError('''Frequency value not found. Update "dict_freq" in frequency.py file
Only minute (T) values are accepted.
''')
#calculate the number of trading hours in the trading session
open_dt = datetime.datetime.combine(datetime.date.today(), self.xchg_open_time)
close_dt = datetime.datetime.combine(datetime.date.today(), self.xchg_close_time)
delta_sec = close_dt - open_dt
nro_trading_hours = delta_sec.total_seconds() / 3600
#using pandas offset timeseries where 'T' corresponds minutes
if (self.candle_type == 'time') and (self.frequency[-1] == 'T'):
self.occurrences_in_year = 252 * nro_trading_hours * 60 / self.freq_int
#event frequency in seconds
self.event_freq = self.freq_int * 60
elif (self.candle_type == 'time') and (self.frequency == 'D'):
self.occurrences_in_year = 252
# For daily, it is set to None
self.event_freq = self.freq_int
# for non-time candles, the events must be generated every second
# but keep the aproximated frequency for statistics
elif (self.candle_type == 'non-time') and (self.frequency[-1] == 'T'):
self.occurrences_in_year = 252 * nro_trading_hours * 60 / self.freq_int
#event frequency in seconds
self.event_freq = 1
else:
raise ValueError(
"This trading frequency {frequency} is not supported. For this type of data {candle_type}"
)
def get_event_frequency(self):
return self.event_freq
def get_frequency(self):
return self.frequency
def get_candle_type(self):
return self.candle_type
def get_xchg_open(self):
return self.xchg_open_time
def get_xchg_close(self):
return self.xchg_close_time
BusinessDaySimulationEngine Class
This class is responsible for creating the date events for the simulation engine. It only generated daily data and had to be changed to generate intraday data, considering the time the Exchange opens and closes and the desired Frequency.
def _generate_business_days(self):
"""
Generate the list of business days using midnight UTC as
the timestamp.
Returns
-------
business_days
`list[pd.Timestamp]`
The business day range list.
intraday_intervals = `list[pd.Timestamp]`
The business day range list.
"""
intraday_intervals = []
business_days = pd.date_range(
self.starting_day, self.ending_day, freq=BDay()
)
intraday_frequency_sec = self.frequency.get_event_frequency()
if intraday_frequency_sec != None:
interval_start_seconds = self.xchg_open_time.hour * 60 * 60 + self.xchg_open_time.minute * 60
interval_end_seconds = self.xchg_close_time.hour * 60 * 60 + self.xchg_close_time.minute * 60
for bday in business_days:
interval_start = bday + pd.Timedelta(seconds=interval_start_seconds)
interval_end = bday + pd.Timedelta(seconds=interval_end_seconds)
intraday_intervals.extend(
pd.to_datetime(
pd.date_range(start=interval_start,
end=interval_end,
freq=str(intraday_frequency_sec)+'S',
tz='UTC').values)
)
return business_days, intraday_intervals
And __iter___ when it is called
def __iter__(self):
"""
Generate the daily timestamps and event information
for pre-market, market open, market close and post-market.
Yields
------
`SimulationEvent`
Market time simulation event to yield
"""
if self.intraday_intervals == []:
# Generate Events for daily frequency
for index, bday in enumerate(self.business_days):
# code for daily frequency
else:
#Generate events for intraday data
checksum_day_change = None
previous_day = None
for index, bday in enumerate(self.intraday_intervals):
year = bday.year
month = bday.month
day = bday.day
hour = bday.hour
minute = bday.minute
second = bday.second
if (checksum_day_change != str(year)+str(month)+str(day)):
if (self.post_market) and (checksum_day_change != None):
yield SimulationEvent(
previous_day,
event_type="post_market"
)
# Create a pre_market event for the new day one hour before
if self.pre_market :
yield SimulationEvent(
pd.Timestamp(
datetime.datetime(year, month, day, hour-1, minute,second),
tz='UTC'
), event_type="pre_market"
)
#update checksum
checksum_day_change = str(year)+str(month)+str(day)
# Market Open Event
if (hour == self.xchg_open_time.hour) and \
(minute == self.xchg_open_time.minute) and \
(second == self.xchg_open_time.second):
yield SimulationEvent(
pd.Timestamp(
datetime.datetime(year, month, day,hour,minute,second),
tz='UTC'
), event_type="market_open"
)
# Market Close Event
elif (hour == self.xchg_close_time.hour) and \
(minute == self.xchg_close_time.minute) and \
(second == self.xchg_close_time.second):
yield SimulationEvent(
pd.Timestamp(
datetime.datetime(year, month, day,hour,minute,second),
tz='UTC'
), event_type="market_close"
)
else:
yield SimulationEvent(
pd.Timestamp(
datetime.datetime(year, month, day,hour,minute,second),
tz='UTC'
), event_type="intraday_bar"
)
# Used to create a post_market event
previous_day = pd.Timestamp(
datetime.datetime(year, month, day, 23, 59, 00),
tz='UTC'
)
IntradayRebalance Class
QStrader works based on this Rebalance Class. It calls the Quantitative strategy to generate it´s signals. By working on these module I realized the other ‘events’ could be added into the simulation engine, for example a training step for machine learning.
By default the tool has the following rebalances: buy_and_hold, end_of_month, weekly + rebalance_weekday and daily. I need it to also have a ‘intraday’ one so the strategy is called at an intraday frequency as well. This also create the possibility for “quarterly” rebalances as well.
def _generate_rebalances(self):
"""
Generate the list of business days timestamp.
Returns
-------
business_days
`list[pd.Timestamp]`
The business day range list.
intraday_intervals = `list[pd.Timestamp]`
The business day range list.
"""
rebalance_times = []
business_days = pd.date_range(
self.start_date, self.end_date , freq=BDay()
)
intraday_frequency_sec = self.frequency.get_event_frequency()
if intraday_frequency_sec != None:
interval_start_seconds = self.xchg_open_time.hour * 60 * 60 + self.xchg_open_time.minute * 60
interval_end_seconds = self.xchg_close_time.hour * 60 * 60 + self.xchg_close_time.minute * 60
for bday in business_days:
interval_start = bday + pd.Timedelta(seconds=interval_start_seconds)
interval_end = bday + pd.Timedelta(seconds=interval_end_seconds)
rebalance_times.extend(
pd.to_datetime(
pd.date_range(start=interval_start,
end=interval_end,
freq=str(intraday_frequency_sec)+'S',
tz='UTC').values).tz_localize('UTC')
)
return rebalance_times
IntradayBarDataSource
Finally the data source to read my database files with price intraday data and feed the simulation engine with information. This is a very specific need and I don´t actually see a reason to share the code. The main change that was made, was to consider the Open price of the next bar (since we are talking about OHLC info) as the buying / selling price the order was executed.