Test environment.
- Dell R210II platform with Intel i3-2100 CPU (HT off) and 16GB DDR3-1333 memory running FreeBSD 9.1-STABLE
- WDC WD1003FBYX hard drive as a system drive
- Corsair Neutron SSD as a file system for a MySQL database
- ExaBGP process (the latest source code from ExaBGP repository) run by pypy 2.0.2
- collector.py run by python 2.7.5
- MySQL 5.5.32 database (with skip-concurrent-insert and skip-innodb configuration options)
- Juniper and Cisco routers as BGP peers
The processing time was measured as a time difference between the first ExaBGP log entry “Connected to peer neighbor” and to the time when processed routes counters reached 462k prefixes for each full IPv4 peer.
Test 1 – ExaBGP without collector.py – 1 full IPv4 peer.
- 1st run – 148s / 2m28s
- 2nd run – 168s / 2m48s
- 3rd run – 146s / 2m26s
Test 2 – ExaBGP without collector.py – 2 full IPv4 peers.
- 1st run – 79s / 1m19s
- 2nd run – 94s / 1m34s
- 3rd run – 91s / 1m31s
Test 3 – ExaBGP without collector.py – 6 full IPv4 and 6 full IPv6 peers.
- 1st run – 184s / 3m4s
- 2nd run – 190s / 3m10s
Test 4 – ExaBGP with collector.py, but without SQL operations – 1 full IPv4 peer.
- 1st run – 69s
- 2nd run – 46s
- 3rd run – 48s
Test 5 – ExaBGP with collector.py, but without SQL operations – 2 full IPv4 peers.
- 1st run – 62s
- 2nd run – 64s
- 3rd run – 61s
Test 6 – ExaBGP with collector.py, but without SQL operations – 6 full IPv4 and 6 full IPv6 peers.
- 1st run – 195s / 3m15s
- 2nd run – 198s / 3m18s
As above, but with exabgp.cache.attributes=false and exabgp.cache.nexthops=false.
- 1st run – 204s / 3m24s
- 2nd run – 213s / 3m33s
Memory usage after processing all prefixes with attributes and nexthops caching enabled.
7814 nobody 1 20 0 223M 163M select 1 1:58 0.00% pypy2.0 < - ExaBGP 8624 nobody 1 20 0 41240K 7948K piperd 0 2:24 0.00% python2.7 <- collector.py
Memory usage with attributes and nexthops caching turned off.
7761 nobody 1 20 0 190M 130M select 1 2:04 0.05% pypy2.0 < - ExaBGP 8234 nobody 1 20 0 41240K 7948K piperd 0 2:24 0.00% python2.7 <- collector.py
Test 7 – ExaBGP with collector.py + SQL operations – 1 full IPv4 peer.
- 1st run – 374s / 6m14s
- 2nd run – 368s / 6m8s
- 3rd run – 368s / 6m8s
Test 8 – ExaBGP with collector.py + SQL operations – 2 full IPv4 peers.
- 1st run – 760s / 12m40s
- 2nd run – 761s / 12m41s
- 3rd run – 755s / 12m35s
Test 9 – ExaBGP with collector.py + SQL operations – 6 full IPv4 and 6 full IPv6 peers.
Number of prefixes processed by each full IPv4 peer:
- after 10 minutes – between 93k and 108k prefixes
- after 20 minutes – between 211k and 215k prefixes
- after 30 minutes – between 325k and 333k prefixes
- after 40 minutes – between 448k and 452k prefixes
Total processing time – 42m15s
For comparison an Atom N2800 based platform with 2GB of memory and 2 SATA drives in RAID1 needed more than 6 hours to process prefixes from 5 full IPv4 peers.
Some advice how to increase the overall performance of ExaBGP and collector.py:
- split full BGP peers to separate ExaBGP processes, especially on multi-core systems
- disable unnecessary logging in ExaBGP, collector.py and MySQL database
- MySQL database and file system tuning