summaryrefslogtreecommitdiffstats
path: root/docs/developer/corefeatures/fib/scale.rst
diff options
context:
space:
mode:
authorNathan Skrzypczak <nathan.skrzypczak@gmail.com>2021-08-19 11:38:06 +0200
committerDave Wallace <dwallacelf@gmail.com>2021-10-13 23:22:32 +0000
commit9ad39c026c8a3c945a7003c4aa4f5cb1d4c80160 (patch)
tree3cca19635417e28ae381d67ae31c75df2925032d /docs/developer/corefeatures/fib/scale.rst
parentf47122e07e1ecd0151902a3cabe46c60a99bee8e (diff)
docs: better docs, mv doxygen to sphinx
This patch refactors the VPP sphinx docs in order to make it easier to consume for external readers as well as VPP developers. It also makes sphinx the single source of documentation, which simplifies maintenance and operation. Most important updates are: - reformat the existing documentation as rst - split RELEASE.md and move it into separate rst files - remove section 'events' - remove section 'archive' - remove section 'related projects' - remove section 'feature by release' - remove section 'Various links' - make (Configuration reference, CLI docs, developer docs) top level items in the list - move 'Use Cases' as part of 'About VPP' - move 'Troubleshooting' as part of 'Getting Started' - move test framework docs into 'Developer Documentation' - add a 'Contributing' section for gerrit, docs and other contributer related infos - deprecate doxygen and test-docs targets - redirect the "make doxygen" target to "make docs" Type: refactor Change-Id: I552a5645d5b7964d547f99b1336e2ac24e7c209f Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com> Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Diffstat (limited to 'docs/developer/corefeatures/fib/scale.rst')
-rw-r--r--docs/developer/corefeatures/fib/scale.rst247
1 files changed, 247 insertions, 0 deletions
diff --git a/docs/developer/corefeatures/fib/scale.rst b/docs/developer/corefeatures/fib/scale.rst
new file mode 100644
index 00000000000..2ec8c6a85ec
--- /dev/null
+++ b/docs/developer/corefeatures/fib/scale.rst
@@ -0,0 +1,247 @@
+.. _scale:
+
+Scale
+-----
+
+The only limiting factor on FIB scale is the amount of memory
+allocated to each heap the FIB uses, and there are 2:
+
+* The main heap
+* The stats heap
+
+
+Main Heap
+^^^^^^^^^
+
+The main heap is used to allocate all memory needed for the FIB
+data-structures. Each table, created by the user, i.e. with;
+
+.. code-block:: console
+
+ $ ip table add 1
+
+or the default table, comprises 2 *ip4_fib_t* objects.
+The 'non-forwarding' *ip4_fib_t* contains all the entries in the table
+and, the 'forwarding' contains the entries that are matched against in
+the data-plane. The difference between the two sets are the entries
+that should not be matched in the data-plane.
+Each *ip4_fib_t* comprises an mtrie (for fast lookup in the data-plane)
+and a hash table per-prefix length (for lookup in the control plane).
+
+To see the amount of memory consumed by the IPv4 tables use:
+
+.. code-block:: console
+
+ vpp# sh ip fib mem
+ ipv4-VRF:0 mtrie:335744 hash:4663
+ ipv4-VRF:1 mtrie:333056 hash:3499
+ totals: mtrie:668800 hash:8162 all:676962
+
+this output shows two 'empty' (i.e. no added routes) tables. Each
+mtrie uses about 150k of memory, so each table about 300k.
+
+
+Below the output having added 1M, 2M and 4M routes respectively:
+
+.. code-block:: console
+
+ vpp# sh ip fib mem
+ ipv4-VRF:0 mtrie:335744 hash:4695
+ totals: mtrie:335744 hash:4695 all:340439
+
+.. code-block:: console
+
+ vpp# sh ip fib mem
+ ipv4-VRF:0 mtrie:5414720 hash:41177579
+ totals: mtrie:5414720 hash:41177579 all:46592299
+
+.. code-block:: console
+
+ vpp# sh ip fib mem
+ ipv4-VRF:0 mtrie:22452608 hash:168544508
+ totals: mtrie:22452608 hash:168544508 all:190997116
+
+
+IPv6 also has the concept of forwarding and non-forwarding entries,
+however for IPv6 all the forwarding entries are stored in a single
+hash table (same goes for the non-forwarding). The key to the hash
+table includes the IPv6 table-id.
+
+To see the amount of memory consumed by the IPv4 tables use:
+
+.. code-block:: console
+
+ vpp# sh ip6 fib mem
+ IPv6 Non-Forwarding Hash Table:
+ Hash table ip6 FIB non-fwding table
+ 7 active elements 7 active buckets
+ 1 free lists
+ 0 linear search buckets
+ arena: base 7f2fe28bf000, next 803c0
+ used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)
+
+ IPv6 Forwarding Hash Table:
+ Hash table ip6 FIB fwding table
+ 7 active elements 7 active buckets
+ 1 free lists
+ 0 linear search buckets
+ arena: base 7f2fe48bf000, next 803c0
+ used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)
+
+as we scale to 128k IPv6 entries:
+
+.. code-block:: console
+
+ vpp# sh ip6 fib mem
+ IPv6 Non-Forwarding Hash Table:
+ Hash table ip6 FIB non-fwding table
+ 131079 active elements 32773 active buckets
+ 2 free lists
+ [len 1] 2 free elts
+ 0 linear search buckets
+ arena: base 7fed7a514000, next 4805c0
+ used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)
+
+ IPv6 Forwarding Hash Table:
+ Hash table ip6 FIB fwding table
+ 131079 active elements 32773 active buckets
+ 2 free lists
+ [len 1] 2 free elts
+ 0 linear search buckets
+ arena: base 7fedba514000, next 4805c0
+ used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)
+
+and 256k:
+
+.. code-block:: console
+
+ vpp# sh ip6 fib mem
+ IPv6 Non-Forwarding Hash Table:
+ Hash table ip6 FIB non-fwding table
+ 262151 active elements 65536 active buckets
+ 2 free lists
+ [len 1] 6 free elts
+ 0 linear search buckets
+ arena: base 7fed7a514000, next 880840
+ used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)
+
+ IPv6 Forwarding Hash Table:
+ Hash table ip6 FIB fwding table
+ 262151 active elements 65536 active buckets
+ 2 free lists
+ [len 1] 6 free elts
+ 0 linear search buckets
+ arena: base 7fedba514000, next 880840
+ used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)
+
+and 1M:
+
+.. code-block:: console
+
+ vpp# sh ip6 fib mem
+ IPv6 Non-Forwarding Hash Table:
+ Hash table ip6 FIB non-fwding table
+ 1048583 active elements 65536 active buckets
+ 4 free lists
+ [len 1] 65533 free elts
+ [len 2] 65531 free elts
+ [len 4] 9 free elts
+ 0 linear search buckets
+ arena: base 7fed7a514000, next 3882740
+ used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)
+
+ IPv6 Forwarding Hash Table:
+ Hash table ip6 FIB fwding table
+ 1048583 active elements 65536 active buckets
+ 4 free lists
+ [len 1] 65533 free elts
+ [len 2] 65531 free elts
+ [len 4] 9 free elts
+ 0 linear search buckets
+ arena: base 7fedba514000, next 3882740
+ used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)
+
+as can be seen from the output the IPv6 hash-table in this case was scaled
+to 1GB and 1million prefixes has used 56MB of it.
+
+The main heap is also used to allocate objects that represent the FIB
+entries in the control and data plane (see :ref:`controlplane` and
+:ref:`dataplane`) such as *fib_entry_t* and *load_balance_t*. These come
+from the main heap because they are not protocol specific
+(i.e. they are used to represent either IPv4, IPv6 or MPLS
+entries).
+
+With 1M prefixes allocated the memory usage is:
+
+.. code-block:: console
+
+ vpp# sh fib mem
+ FIB memory
+ Tables:
+ SAFI Number Bytes
+ IPv4 unicast 1 33619968
+ IPv6 unicast 2 118502784
+ MPLS 0 0
+ IPv4 multicast 1 1175
+ IPv6 multicast 1 525312
+ Nodes:
+ Name Size in-use /allocated totals
+ Entry 72 1048589/ 1048589 75498408/75498408
+ Entry Source 40 1048589/ 1048589 41943560/41943560
+ Entry Path-Extensions 76 0 / 0 0/0
+ multicast-Entry 192 6 / 6 1152/1152
+ Path-list 40 18 / 18 720/720
+ uRPF-list 16 14 / 14 224/224
+ Path 72 22 / 22 1584/1584
+ Node-list elements 20 1048602/ 1048602 20972040/20972040
+ Node-list heads 8 24 / 24 192/192
+
+and with 2M
+
+.. code-block:: console
+
+ vpp# sh fib mem
+ FIB memory
+ Tables:
+ SAFI Number Bytes
+ IPv4 unicast 1 33619968
+ IPv6 unicast 2 252743040
+ MPLS 0 0
+ IPv4 multicast 1 1175
+ IPv6 multicast 1 525312
+ Nodes:
+ Name Size in-use /allocated totals
+ Entry 72 2097165/ 2097165 150995880/150995880
+ Entry Source 40 2097165/ 2097165 83886600/83886600
+ Entry Path-Extensions 76 0 / 0 0/0
+ multicast-Entry 192 6 / 6 1152/1152
+ Path-list 40 18 / 19 720/760
+ uRPF-list 16 18 / 18 288/288
+ Path 72 22 / 23 1584/1656
+ Node-list elements 20 2097178/ 2097178 41943560/41943560
+ Node-list heads 8 24 / 24 192/192
+
+However, the situation is not a simple as that. All of the 1M prefixes
+added above were reachable via the same next-hop, so the path-list
+(and path) they use is shared. As prefixes are added that use
+different (sets of) next-hops, the number of path-lists and paths
+requires will increase.
+
+
+Stats Heap
+^^^^^^^^^^
+
+VPP collects statistics for each route. For each route VPP collects
+byte and packet counters for packets sent to the prefix (i.e. the
+route was matched in the data-plane) and packets sent via the prefix (i.e. the
+matching prefix is reachable through it - like a BGP peer). This
+requires 4 counters per route in the stats segment.
+
+Below shows the size of the stats segment with 1M, 2M and 4M routes.
+
+.. code-block:: console
+
+ total: 1023.99M, used: 127.89M, free: 896.10M, trimmable: 830.94M
+ total: 1023.99M, used: 234.14M, free: 789.85M, trimmable: 668.15M
+ total: 1023.99M, used: 456.83M, free: 567.17M, trimmable: 388.91M
+