diff options
56 files changed, 3706 insertions, 3184 deletions
diff --git a/src/plugins/acl/acl_hash_lookup_doc.md b/src/plugins/acl/acl_hash_lookup_doc.md deleted file mode 100644 index 6b08e1bc953..00000000000 --- a/src/plugins/acl/acl_hash_lookup_doc.md +++ /dev/null @@ -1,240 +0,0 @@ -ACL plugin constant-time lookup design {#acl_hash_lookup} -====================================== - -The initial implementation of ACL plugin performs a trivial for() cycle, -going through the assigned ACLs on a per-packet basis. This is not very -efficient, even if for very short ACLs due to its simplicity it can beat -more advanced methods. - -However, to cover the case of longer ACLs with acceptable performance, -we need to have a better way of matching. This write-up proposes -a mechanism to make a lookup from O(M) where M is number of entries -to O(N) where N is number of different mask combinations. - -Preparation of ACL(s) ---------------------- - -The ACL plugin will maintain a global list of "mask types", i.e. the specific -configurations of "do not care" bits within the ACEs. -Upon the creation of a new ACL, a pass will be made through all the -ACEs, to assign and possibly allocate the "mask type number". - -Each ACL has a structure *hash_acl_info_t* representing the "hash-based" -parts of information related to that ACL, primarily the array of -*hash_ace_info_t* structures - each of the members of that array -corresponding to one of the rules (ACEs) in the original ACL, -for this they have a pair of *(acl_index, ace_index)* to keep track, -predominantly for debugging. - -Why do we need a whole separate structure, and are not adding new fields -to the existing rule structure? First, encapsulation, to minimize -the pollution of the main ACL code with the hash-based lookup artifacts. -Second, one rule may correspond to more than one "hash-based" ACE. -In fact, most of the rules do correspond to two of those. Why ? - -Consider that the current ACL lookup logic is that if a packet -is not the initial fragment, and there is an L4 entry acting on the packet, -the comparison will be made only on the L4 protocol field value rather -than on the protocol and port values. This behavior is governed by -*l4_match_nonfirst_fragment* flag in the *acl_main*, and is needed to -maintain the compatibility with the existing software switch implementation. - -While for the sequential check in *single_acl_match_5tuple()* -it is very easy to implement by just breaking out at the right moment, -in case of hash-based matching this cost us two checks: -one on full 5-tuple and the flag *pkt.is_nonfirst_fragment* being zero, -the second on 3-tuple and the flag *pkt.is_nonfirst_fragment* being one, -with the second check triggered by the *acl_main.l4_match_nonfirst_fragment* -setting being the default 1. This dictates the necessity of having a "match" -field in a given *hash_ace_info_t* element, which would reflect the value -we are supposed to match after applying the mask. - -There can be other circumstances when it might be beneficial to expand -the given rule in the original ACL into multiple - for example, as an -optimization within the port range handling for small port ranges -(this is not done as of the time of writing). - -Assigning ACLs to an interface ------------------------------- - -Once the ACL list is assigned to an interface, or, rather, a new ACL -is added to the list of the existing ACLs applied to the interface, -we need to update the bihash accelerating the lookup. - -All the entries for the lookups are stored within a single *48_8* bihash, -which captures the 5-tuple from the packet as well as the miscellaneous -per-packet information flags, e.g. *l4_valid*, *is_non_first_fragment*, -and so on. To facilitate the use of the single bihash by all the interfaces, -the *is_ip6*, *is_input*, *sw_if_index* are part of the key, -as well as *mask_type_index* - the latter being necessary because -there can be entries with the same value but different masks, e.g.: -`permit ::/0, permit::/128`. - -At the moment of an ACL being applied to an interface, we need to -walk the list of *hash_ace_info_t* entries corresponding to that ACL, -and update the bihash with the keys corresponding to the match -values in these entries. - -The value of the hash match contains the index into a per-*sw_if_index* vector -of *applied_ace_hash_entry_t* elements, as well as a couple of flags: -*shadowed* (optimization: if this flag on a matched entry is zero, means -we can stop the lookup early and declare a match - see below), -and *need_portrange_check* - meaning that what matched was a superset -of the actual match, and we need to perform an extra check. - -Also, upon insertion, we must keep in mind there can be -multiple *applied_ace_hash_entry_t* for the same key and must keep -a list of those. This is necessary to incrementally apply/unapply -the ACLs as part of the ACL vector: say, two ACLs have -"permit 2001:db8::1/128 any" - we should be able to retain the entry -for the second ACL even if we have deleted the first one. -Also, in case there are two entries with the same key but -different port ranges, say 0..42 and 142..65535 - we need -to be able to sequentially match on those if we decide not -to expand them into individual port-specific entries. - -Per-packet lookup ------------------ - -The simple single-packet lookup is defined in -*multi_acl_match_get_applied_ace_index*, which returns the index -of the applied hash ACE if there was a match, or ~0 if there wasn't. - -The future optimized per-packet lookup may be batched in three phases: - -1. Prepare the keys in the per-worker vector by doing logical AND of - original 5-tuple record with the elements of the mask vector. -2. Lookup the keys in the bihash in a batch manner, collecting the - result with lowest u64 (acl index within vector, ACE index) from - the hash lookup value, and performing the list walk if necessary - (for portranges). -3. Take the action from the ACL record as defined by (ACL#, ACE#) from the - resulting lookup winner, or, if no match found, then perform default deny. - -Shadowed/independent/redundant ACEs ------------------------------------- - -During the phase of combining multiple ACLs into one rulebase, when they -are applied to interface, we also can perform several optimizations. - -If a given ACE is a strict subset of another ACE located up in the linear -search order, we can ignore this ACE completely - because by definition -it will never match. We will call such an ACE *redundant*. Here is an example: - -``` -permit 2001:db8:1::/48 2001:db8:2::/48 (B) -deny 2001:d8b:1:1::/64 2001:db8:2:1::/64 (A) -``` - -A bit more formally, we can define this relationship of an ACE A to ACE B as: - -``` -redundant(aceA, aceB) := (contains(protoB, protoA) && contains(srcB, srcA) - && contains(dstB, dstA) && is_after(A, B)) -``` - -Here as "contains" we define an operation operating on the sets defined by -the protocol, (srcIP, srcPortDefinition) and (dstIP, dstPortDefinition) -respectively, and returning true if all the elements represented by -the second argument are represented by the first argument. The "is_after" -is true if A is located below B in the ruleset. - -If a given ACE does not intersect at all with any other ACE -in front of it, we can mark it as such. - -Then during the sequence of the lookups the successful hit on this ACE means -we do not need to look up other mask combinations - thus potentially -significantly speeding up the match process. Here is an example, -assuming we have the following ACL: - -``` -permit 2001:db8:1::/48 2001:db8:2::/48 (B) -deny 2001:db8:3::/48 2001:db8:2:1::/64 (A) -``` - -In this case if we match the second entry, we do not need to check whether -we have matched the first one - the source addresses are completely -different. We call such an ACE *independent* from another. - -We can define this as - -``` -independent(aceA, aceB) := (!intersect(protoA, protoB) || - !intersect(srcA, srcB) || - !intersect(dstA, dstB)) -``` - -where intersect is defined as operation returning true if there are -elements belonging to the sets of both arguments. - -If the entry A is neither redundant nor independent from B, and is below -B in the ruleset, we call such an entry *shadowed* by B, here is an example: - -``` -deny tcp 2001:db8:1::/48 2001:db8:2::/48 (B) -permit 2001:d8b:1:1::/64 2001:db8:2:1::/64 (A) -``` - -This means the earlier rule "carves out" a subset of A, thus leaving -a "shadow". (Evidently, the action needs to be different for the shadow -to have an effect, but for for the terminology sake we do not care). - -The more formal definition: - -``` -shadowed(aceA, aceB) := !redundant(aceA, aceB) && - !independent(aceA, aceB) && - is_after(aceA, aceB) -``` - -Using this terminology, any ruleset can be represented as -a DAG (Directed Acyclic Graph), with the bottom being the implicit -"deny any", pointing to the set of rules shadowing it or the ones -it is redundant for. - -These rules may in turn be shadowing each other. There is no cycles in -this graph because of the natural order of the rules - the rule located -closer to the end of the ruleset can never shadow or make redundant a rule -higher up. - -The optimization that enables can allow for is to skip matching certain -masks on a per-lookup basis - if a given rule has matched, -the only adjustments that can happen is the match with one of -the shadowing rules. - -Also, another avenue for the optimization can be starting the lookup process -with the mask type that maximizes the chances of the independent ACE match, -thus resulting in an ACE lookup being a single hash table hit. - - -Plumbing --------- - -All the new routines are located in a separate file, -so we can cleanly experiment with a different approach if this -does not fit all of the use cases. - -The constant-time lookup within the data path has the API with -the same signature as: - -``` -u8 -multi_acl_match_5tuple (u32 sw_if_index, fa_5tuple_t * pkt_5tuple, int is_l2, - int is_ip6, int is_input, u32 * acl_match_p, - u32 * rule_match_p, u32 * trace_bitmap) -``` - -There should be a new upper-level function with the same signature, which -will make a decision whether to use a linear lookup, or to use the -constant-time lookup implemented by this work, or to add some other -optimizations (e.g. by keeping the cache of the last N lookups). - -The calls to the routine doing preparatory work should happen -in `acl_add_list()` after creating the linear-lookup structures, -and the routine doing the preparatory work populating the hashtable -should be called from `acl_interface_add_del_inout_acl()` or its callees. - -The initial implementation will be geared towards looking up a single -match at a time, with the subsequent optimizations possible to make -the lookup for more than one packet. - diff --git a/src/plugins/acl/acl_hash_lookup_doc.rst b/src/plugins/acl/acl_hash_lookup_doc.rst new file mode 100644 index 00000000000..72842af423d --- /dev/null +++ b/src/plugins/acl/acl_hash_lookup_doc.rst @@ -0,0 +1,243 @@ +ACL plugin constant-time lookup +=============================== + +The initial implementation of ACL plugin performs a trivial for() cycle, +going through the assigned ACLs on a per-packet basis. This is not very +efficient, even if for very short ACLs due to its simplicity it can beat +more advanced methods. + +However, to cover the case of longer ACLs with acceptable performance, +we need to have a better way of matching. This write-up proposes a +mechanism to make a lookup from O(M) where M is number of entries to +O(N) where N is number of different mask combinations. + +Preparation of ACL(s) +--------------------- + +The ACL plugin will maintain a global list of “mask types”, i.e. the +specific configurations of “do not care” bits within the ACEs. Upon the +creation of a new ACL, a pass will be made through all the ACEs, to +assign and possibly allocate the “mask type number”. + +Each ACL has a structure *hash_acl_info_t* representing the “hash-based” +parts of information related to that ACL, primarily the array of +*hash_ace_info_t* structures - each of the members of that array +corresponding to one of the rules (ACEs) in the original ACL, for this +they have a pair of *(acl_index, ace_index)* to keep track, +predominantly for debugging. + +Why do we need a whole separate structure, and are not adding new fields +to the existing rule structure? First, encapsulation, to minimize the +pollution of the main ACL code with the hash-based lookup artifacts. +Second, one rule may correspond to more than one “hash-based” ACE. In +fact, most of the rules do correspond to two of those. Why ? + +Consider that the current ACL lookup logic is that if a packet is not +the initial fragment, and there is an L4 entry acting on the packet, the +comparison will be made only on the L4 protocol field value rather than +on the protocol and port values. This behavior is governed by +*l4_match_nonfirst_fragment* flag in the *acl_main*, and is needed to +maintain the compatibility with the existing software switch +implementation. + +While for the sequential check in *single_acl_match_5tuple()* it is very +easy to implement by just breaking out at the right moment, in case of +hash-based matching this cost us two checks: one on full 5-tuple and the +flag *pkt.is_nonfirst_fragment* being zero, the second on 3-tuple and +the flag *pkt.is_nonfirst_fragment* being one, with the second check +triggered by the *acl_main.l4_match_nonfirst_fragment* setting being the +default 1. This dictates the necessity of having a “match” field in a +given *hash_ace_info_t* element, which would reflect the value we are +supposed to match after applying the mask. + +There can be other circumstances when it might be beneficial to expand +the given rule in the original ACL into multiple - for example, as an +optimization within the port range handling for small port ranges (this +is not done as of the time of writing). + +Assigning ACLs to an interface +------------------------------ + +Once the ACL list is assigned to an interface, or, rather, a new ACL is +added to the list of the existing ACLs applied to the interface, we need +to update the bihash accelerating the lookup. + +All the entries for the lookups are stored within a single *48_8* +bihash, which captures the 5-tuple from the packet as well as the +miscellaneous per-packet information flags, e.g. *l4_valid*, +*is_non_first_fragment*, and so on. To facilitate the use of the single +bihash by all the interfaces, the *is_ip6*, *is_input*, *sw_if_index* +are part of the key, as well as *mask_type_index* - the latter being +necessary because there can be entries with the same value but different +masks, e.g.: ``permit ::/0, permit::/128``. + +At the moment of an ACL being applied to an interface, we need to walk +the list of *hash_ace_info_t* entries corresponding to that ACL, and +update the bihash with the keys corresponding to the match values in +these entries. + +The value of the hash match contains the index into a per-*sw_if_index* +vector of *applied_ace_hash_entry_t* elements, as well as a couple of +flags: *shadowed* (optimization: if this flag on a matched entry is +zero, means we can stop the lookup early and declare a match - see +below), and *need_portrange_check* - meaning that what matched was a +superset of the actual match, and we need to perform an extra check. + +Also, upon insertion, we must keep in mind there can be multiple +*applied_ace_hash_entry_t* for the same key and must keep a list of +those. This is necessary to incrementally apply/unapply the ACLs as part +of the ACL vector: say, two ACLs have “permit 2001:db8::1/128 any” - we +should be able to retain the entry for the second ACL even if we have +deleted the first one. Also, in case there are two entries with the same +key but different port ranges, say 0..42 and 142..65535 - we need to be +able to sequentially match on those if we decide not to expand them into +individual port-specific entries. + +Per-packet lookup +----------------- + +The simple single-packet lookup is defined in +*multi_acl_match_get_applied_ace_index*, which returns the index of the +applied hash ACE if there was a match, or ~0 if there wasn’t. + +The future optimized per-packet lookup may be batched in three phases: + +1. Prepare the keys in the per-worker vector by doing logical AND of + original 5-tuple record with the elements of the mask vector. +2. Lookup the keys in the bihash in a batch manner, collecting the + result with lowest u64 (acl index within vector, ACE index) from the + hash lookup value, and performing the list walk if necessary (for + portranges). +3. Take the action from the ACL record as defined by (ACL#, ACE#) from + the resulting lookup winner, or, if no match found, then perform + default deny. + +Shadowed/independent/redundant ACEs +----------------------------------- + +During the phase of combining multiple ACLs into one rulebase, when they +are applied to interface, we also can perform several optimizations. + +If a given ACE is a strict subset of another ACE located up in the +linear search order, we can ignore this ACE completely - because by +definition it will never match. We will call such an ACE *redundant*. +Here is an example: + +:: + + permit 2001:db8:1::/48 2001:db8:2::/48 (B) + deny 2001:d8b:1:1::/64 2001:db8:2:1::/64 (A) + +A bit more formally, we can define this relationship of an ACE A to ACE +B as: + +:: + + redundant(aceA, aceB) := (contains(protoB, protoA) && contains(srcB, srcA) + && contains(dstB, dstA) && is_after(A, B)) + +Here as “contains” we define an operation operating on the sets defined +by the protocol, (srcIP, srcPortDefinition) and (dstIP, +dstPortDefinition) respectively, and returning true if all the elements +represented by the second argument are represented by the first +argument. The “is_after” is true if A is located below B in the ruleset. + +If a given ACE does not intersect at all with any other ACE in front of +it, we can mark it as such. + +Then during the sequence of the lookups the successful hit on this ACE +means we do not need to look up other mask combinations - thus +potentially significantly speeding up the match process. Here is an +example, assuming we have the following ACL: + +:: + + permit 2001:db8:1::/48 2001:db8:2::/48 (B) + deny 2001:db8:3::/48 2001:db8:2:1::/64 (A) + +In this case if we match the second entry, we do not need to check +whether we have matched the first one - the source addresses are +completely different. We call such an ACE *independent* from another. + +We can define this as + +:: + + independent(aceA, aceB) := (!intersect(protoA, protoB) || + !intersect(srcA, srcB) || + !intersect(dstA, dstB)) + +where intersect is defined as operation returning true if there are +elements belonging to the sets of both arguments. + +If the entry A is neither redundant nor independent from B, and is below +B in the ruleset, we call such an entry *shadowed* by B, here is an +example: + +:: + + deny tcp 2001:db8:1::/48 2001:db8:2::/48 (B) + permit 2001:d8b:1:1::/64 2001:db8:2:1::/64 (A) + +This means the earlier rule “carves out” a subset of A, thus leaving a +“shadow”. (Evidently, the action needs to be different for the shadow to +have an effect, but for for the terminology sake we do not care). + +The more formal definition: + +:: + + shadowed(aceA, aceB) := !redundant(aceA, aceB) && + !independent(aceA, aceB) && + is_after(aceA, aceB) + +Using this terminology, any ruleset can be represented as a DAG +(Directed Acyclic Graph), with the bottom being the implicit “deny any”, +pointing to the set of rules shadowing it or the ones it is redundant +for. + +These rules may in turn be shadowing each other. There is no cycles in +this graph because of the natural order of the rules - the rule located +closer to the end of the ruleset can never shadow or make redundant a +rule higher up. + +The optimization that enables can allow for is to skip matching certain +masks on a per-lookup basis - if a given rule has matched, the only +adjustments that can happen is the match with one of the shadowing +rules. + +Also, another avenue for the optimization can be starting the lookup +process with the mask type that maximizes the chances of the independent +ACE match, thus resulting in an ACE lookup being a single hash table +hit. + +Plumbing +-------- + +All the new routines are located in a separate file, so we can cleanly +experiment with a different approach if this does not fit all of the use +cases. + +The constant-time lookup within the data path has the API with the same +signature as: + +:: + + u8 + multi_acl_match_5tuple (u32 sw_if_index, fa_5tuple_t * pkt_5tuple, int is_l2, + int is_ip6, int is_input, u32 * acl_match_p, + u32 * rule_match_p, u32 * trace_bitmap) + +There should be a new upper-level function with the same signature, +which will make a decision whether to use a linear lookup, or to use the +constant-time lookup implemented by this work, or to add some other +optimizations (e.g. by keeping the cache of the last N lookups). + +The calls to the routine doing preparatory work should happen in +``acl_add_list()`` after creating the linear-lookup structures, and the +routine doing the preparatory work populating the hashtable should be +called from ``acl_interface_add_del_inout_acl()`` or its callees. + +The initial implementation will be geared towards looking up a single +match at a time, with the subsequent optimizations possible to make the +lookup for more than one packet. diff --git a/src/plugins/acl/acl_lookup_context.md b/src/plugins/acl/acl_lookup_context.md deleted file mode 100644 index e95f82043f9..00000000000 --- a/src/plugins/acl/acl_lookup_context.md +++ /dev/null @@ -1,125 +0,0 @@ -Lookup contexts aka "ACL as a service" {#acl_lookup_context} -====================================== - -The initial implementation of the ACL plugin had tightly tied the policy (L3-L4) ACLs -to ingress/egress processing on an interface. - -However, some uses outside of pure traffic control have appeared, for example, -ACL-based forwarding, etc. Also, improved algorithms of the ACL lookup -could benefit of the more abstract representation, not coupled to the interfaces. - -This describes a way to accommodate these use cases by generalizing the ACL -lookups into "ACL lookup contexts", not tied to specific interfaces, usable -by other portions of the code by utilizing the exports.h header file, -which provides the necessary interface. - - -Why "lookup contexts" and not "match me an ACL#" ? -================================================ - -The first reason is the logical grouping of multiple ACLs. - -The interface matching code currently allows for matching multiple ACLs -in a 'first-match' fashion. Some other use cases also fall into a similar -pattern: they attempt to match a sequence of ACLs, and the first matched ACL -determines what the outcome is, e.g. where to forward traffic. Thus, -a match never happens on an ACL in isolation, but always on a group of -ACLs. - -The second reason is potential optimizations in matching. - -A naive match on series of ACLs each represented as a vector of ACEs -does not care about the API level - it could be "match one ACL", or -"match the set of ACLs" - there will be just a simple loop iterating over -the ACLs to match, returning the first match. Be it in the ACL code or -in the user code. - -However, for more involved lookup methods, providing a more high-level -interface of matching over the entire group of ACLs allows for future -improvements in the algorithms, delivered at once to all the users -of the API. - -What is a "lookup context" ? -============================ - -An ACL lookup context is an entity that groups the set of ACL#s -together for the purposes of a first-match lookup, and may store -additional internal information needed to optimize the lookups -for that particular vector of ACLs. - -Using ACL contexts in your code -=============================== - -In order to use the ACL lookup contexts, you need to include -plugins/acl/exports.h into your code. This header includes -all the necessary dependencies required. - -As you probably will invoke this code from another plugin, -the non-inline function calls are implemented via function pointers, -which you need to initialize by calling acl_plugin_exports_init(&acl_plugin), which, -if everything succeeds, returns 0 and fills in the acl_plugin structure -with pointers to the exported methods - else it will return clib_error_t with -more information about what went wrong. - -When you have initialized the symbols, you also need to register yourself -as a user of the ACL lookups - this allows to track the ACL lookup context -ownership, as well as make the debug show outputs more user friendly. - -To do that, call acl_plugin.register_user_module(caller_module_string, val1_label, val2_label) - -and record the returned value. This will bethe first parameter that you pass to create a new -lookup context. The passed strings must be static, and are used as descriptions for the ACL -contexts themselves, as well as labels for up to two user-supplied u32 labels, used to -differentiate the lookup contexts for the debugging purposes. - -Creating a new context is done by calling acl_plugin.get_lookup_context_index(user_id, val1, val2). -The first argument is your "user" ID obtained in a registration call earlier, the other two -arguments are u32s with semantics that you designate. They are used purely for debugging purposes -in the "show acl lookup context" command. - -To set the vector of ACL numbers to be looked up within the context, use the function -acl_plugin.set_acl_vec_for_context(lc_index, acl_list). The first parameter specifies the context -that you have created, the second parameter is a vector of u32s, each u32 being the index of the ACL -which we should be looking up within this context. The command is idempotent, i.e. -it unapplies the previously applied list of ACLs, and then sets the new list of ACLs. - -Subsequent ACL updates for the already applied ACLs will cause the re-application -on an as-needed basis. Note, that the ACL application is potentially a relatively costly operation, -so it is only expected that these changes will be done in the control plane, NOT in the datapath. - -The matching within the context is done using two functions - acl_plugin.fill_5tuple() and -acl_plugin.match_5tuple() and their corresponding inline versions, named acl_plugin_fill_5tuple_inline() -and acl_plugin_match_5tuple_inline(). The inline and non-inline versions have the equivalent functionality, -in that the non-inline version calls the inline version. These two variants are provided -for debugging/maintenance reasons. - -When you no longer need a particular context, you can return the allocated resources by calling -acl_plugin.put_lookup_context_index() to mark it as free. The lookup structured associated with -the vector of ACLs set for the lookup are cleaned up automatically. However, the ACLs themselves -are not deleted and are available for subsequent reuse by other lookup contexts if needed. - -There is one delicate detail that you might want to be aware of. -When the non-inline functions reference the inline functions, -they are compiled as part of ACL plugin; whereas when you refer to the inline -functions from your code, they are compiled as part of your code. -This makes referring to a single acl_main structure a little trickier. - -It is done by having a static p_acl_main within the .h file, -which points to acl_main of the ACL plugin, and is initialized by a static constructor -function. - -This way the multiple includes and inlines will "just work" as one would expect. - - -Debug CLIs -========== - -To see the state of the ACL lookup contexts, you can issue "show acl-plugin lookup user" to see -all of the users which registered for the usage of the ACL plugin lookup contexts, -and "show acl-plugin lookup context" to show the actual contexts created. You will notice -that the latter command uses the values supplied during the module registration in order to -make the output more friendly. - -The "show acl-plugin acl" and "show acl-plugin interface" commands have also acquired the -notion of lookup context, but there it is used from the client perspective, since -with this change the interface ACL lookup itself is a user of ACL lookup contexts. - diff --git a/src/plugins/acl/acl_lookup_context.rst b/src/plugins/acl/acl_lookup_context.rst new file mode 100644 index 00000000000..278e87381f3 --- /dev/null +++ b/src/plugins/acl/acl_lookup_context.rst @@ -0,0 +1,138 @@ +ACL Lookup contexts +=================== + +The initial implementation of the ACL plugin had tightly tied the policy +(L3-L4) ACLs to ingress/egress processing on an interface. + +However, some uses outside of pure traffic control have appeared, for +example, ACL-based forwarding, etc. Also, improved algorithms of the ACL +lookup could benefit of the more abstract representation, not coupled to +the interfaces. + +This describes a way to accommodate these use cases by generalizing the +ACL lookups into “ACL lookup contexts”, not tied to specific interfaces, +usable by other portions of the code by utilizing the exports.h header +file, which provides the necessary interface. + +Why “lookup contexts” and not “match me an ACL” ? +------------------------------------------------- + +The first reason is the logical grouping of multiple ACLs. + +The interface matching code currently allows for matching multiple ACLs +in a ‘first-match’ fashion. Some other use cases also fall into a +similar pattern: they attempt to match a sequence of ACLs, and the first +matched ACL determines what the outcome is, e.g. where to forward +traffic. Thus, a match never happens on an ACL in isolation, but always +on a group of ACLs. + +The second reason is potential optimizations in matching. + +A naive match on series of ACLs each represented as a vector of ACEs +does not care about the API level - it could be “match one ACL”, or +“match the set of ACLs” - there will be just a simple loop iterating +over the ACLs to match, returning the first match. Be it in the ACL code +or in the user code. + +However, for more involved lookup methods, providing a more high-level +interface of matching over the entire group of ACLs allows for future +improvements in the algorithms, delivered at once to all the users of +the API. + +What is a “lookup context” ? +---------------------------- + +An ACL lookup context is an entity that groups the set of ACL#s together +for the purposes of a first-match lookup, and may store additional +internal information needed to optimize the lookups for that particular +vector of ACLs. + +Using ACL contexts in your code +------------------------------- + +In order to use the ACL lookup contexts, you need to include +plugins/acl/exports.h into your code. This header includes all the +necessary dependencies required. + +As you probably will invoke this code from another plugin, the +non-inline function calls are implemented via function pointers, which +you need to initialize by calling acl_plugin_exports_init(&acl_plugin), +which, if everything succeeds, returns 0 and fills in the acl_plugin +structure with pointers to the exported methods - else it will return +clib_error_t with more information about what went wrong. + +When you have initialized the symbols, you also need to register +yourself as a user of the ACL lookups - this allows to track the ACL +lookup context ownership, as well as make the debug show outputs more +user friendly. + +To do that, call acl_plugin.register_user_module(caller_module_string, +val1_label, val2_label) - and record the returned value. This will be the +first parameter that you pass to create a new lookup context. The passed +strings must be static, and are used as descriptions for the ACL +contexts themselves, as well as labels for up to two user-supplied u32 +labels, used to differentiate the lookup contexts for the debugging +purposes. + +Creating a new context is done by calling +acl_plugin.get_lookup_context_index(user_id, val1, val2). The first +argument is your “user” ID obtained in a registration call earlier, the +other two arguments are u32s with semantics that you designate. They are +used purely for debugging purposes in the “show acl lookup context” +command. + +To set the vector of ACL numbers to be looked up within the context, use +the function acl_plugin.set_acl_vec_for_context(lc_index, acl_list). The +first parameter specifies the context that you have created, the second +parameter is a vector of u32s, each u32 being the index of the ACL which +we should be looking up within this context. The command is idempotent, +i.e. it unapplies the previously applied list of ACLs, and then sets the +new list of ACLs. + +Subsequent ACL updates for the already applied ACLs will cause the +re-application on an as-needed basis. Note, that the ACL application is +potentially a relatively costly operation, so it is only expected that +these changes will be done in the control plane, NOT in the datapath. + +The matching within the context is done using two functions - +acl_plugin.fill_5tuple() and acl_plugin.match_5tuple() and their +corresponding inline versions, named acl_plugin_fill_5tuple_inline() and +acl_plugin_match_5tuple_inline(). The inline and non-inline versions +have the equivalent functionality, in that the non-inline version calls +the inline version. These two variants are provided for +debugging/maintenance reasons. + +When you no longer need a particular context, you can return the +allocated resources by calling acl_plugin.put_lookup_context_index() to +mark it as free. The lookup structured associated with the vector of +ACLs set for the lookup are cleaned up automatically. However, the ACLs +themselves are not deleted and are available for subsequent reuse by +other lookup contexts if needed. + +There is one delicate detail that you might want to be aware of. When +the non-inline functions reference the inline functions, they are +compiled as part of ACL plugin; whereas when you refer to the inline +functions from your code, they are compiled as part of your code. This +makes referring to a single acl_main structure a little trickier. + +It is done by having a static p_acl_main within the .h file, which +points to acl_main of the ACL plugin, and is initialized by a static +constructor function. + +This way the multiple includes and inlines will “just work” as one would +expect. + +Debug CLIs +---------- + +To see the state of the ACL lookup contexts, you can issue “show +acl-plugin lookup user” to see all of the users which registered for the +usage of the ACL plugin lookup contexts, and “show acl-plugin lookup +context” to show the actual contexts created. You will notice that the +latter command uses the values supplied during the module registration +in order to make the output more friendly. + +The “show acl-plugin acl” and “show acl-plugin interface” commands have +also acquired the notion of lookup context, but there it is used from +the client perspective, since with this change the interface ACL lookup +itself is a user of ACL lookup contexts. diff --git a/src/plugins/acl/acl_multicore_doc.md b/src/plugins/acl/acl_multicore_doc.md deleted file mode 100644 index deec5e9d566..00000000000 --- a/src/plugins/acl/acl_multicore_doc.md +++ /dev/null @@ -1,349 +0,0 @@ -Multicore support for ACL plugin {#acl_multicore} -================================ - -This captures some considerations and design decisions that I have made, -both for my own memory later on ("what the hell was I thinking?!?"), -and for anyone interested to criticize/improve/hack on this code. - -One of the factors taken into account while making these decisions, -was the relative emphasis on the multi-thread vs. single-thread -use cases: the latter is the vastly more prevalent. But, -one can not optimize the single-thread performance without -having a functioning code for multi-thread. - -stateless ACLs -============== - -The stateless trivially parallelizes, and the only potential for the -race between the different threads is during the reconfiguration, -at the time of replacing the old ACL being checked, with -the new ACL. - -In case an acl_add_replace is being used to replace the rules -within the existing entry, a reallocation of `am->acls[X].rules` -vector will happen and potentially a change in count. - -acl_match_5tuple() has the following code: - -```{.c} - a = am->acls + acl_index; - for (i = 0; i < a->count; i++) - { - r = a->rules + i; - . . . -``` - -Ideally we should be immune from a->rules changing, -but the problem arises if the count changes in flight, -and the new ruleset is smaller - then we will attempt -to "match" against the free memory. - -This can(?) be solved by replacing the for() with while(), -so the comparison happens at each iteration. - -full_acl_match_5tuple(), which iterates over the list -of ACLs, is a bit less immune, since it takes the pointer -to the vector to iterate and keeps a local copy of -that pointer. - -This race can be solved by checking the -current pointer to the vector with the source pointer, -and seeing if there is an (unlikely) change, and if -there is, return the "deny" action, or, better, -restart the check. - -Since the check reloads the ACL list on a per-packet basis, -there is only a window of opportunity of one packet to -"match" packet against an incorrect rule set. -The workers also do not change anything, only read. -Therefore, it looks like building special structures -to ensure that it does not happen at all might be not -worth it. - -At least not until we have a unit-test able to -reliably catch this condition and test that -the measures applied are effective. Adding the code -which is not possible to exercise is worse than -not adding any code at all. - -So, I opt for "do-nothing" here for the moment. - -reflexive ACLs: single-thread -============================= - -Before we talk multi-thread, is worth revisiting the -design of the reflexive ACLs in the plugin, and -the history of their evolution. - -The very first version of the ACL plugin, shipped in -1701, mostly did the job using the existing components -and gluing them together. Because it needed to work -in bridged forwarding path only, using L2 classifier -as an insertion point appeared natural, also L2 classifier, -being a table with sessions, seemed like a good place -to hold the sessions. - -So, the original design had two conceptual nodes: -one, pointed by the next_miss from the L2 classifier table, -was checking the actual ACL, and inserting session into -the L2 classifier table, and the other one, pointed -to by the next_match within the specific session rule, -was checking the existing session. The timing out -of the existing connections was done in the datapath, -by periodically calling the aging function. - -This decision to use the existing components, -with its attractiveness, did bring a few limitations as well: - -* L2 classifier is a simple mask-and-value match, with -a fixed mask across the table. So, sanely supporting IPv6 -packets with extension headers in that framework was impossible. - -* There is no way to get a backpressure from L2 classifier -depending on memory usage. When it runs out of memory, -it simply crashes the box. When it runs out of memory ? -We don't really know. Depends on how it allocates it. - -* Since we need to match the *reflected* traffic, -we had to create *two* full session entries -in two different directions, which is quite wasteful memory-wise. - -* (showstopper): the L2 classifier runs only in -the bridged data path, so supporting routed data path -would require creating something else entirely different, -which would mean much more headaches support-wise going forward. - -Because of that, I have moved to a different model of -creating a session-5-tuple from the packet data - once, -and then doing all the matching just on that 5-tuple. - -This has allowed to add support for skipping IPv6 extension headers. - -Also, this new version started to store the sessions in a dedicated -bihash-per-interface, with the session key data being -aligned for the ingress packets, and being mirrored for the -egress packets. This allows of significant savings in memory, -because now we need to keep only one copy of the session table per -interface instead of two, and also to only have ONE node for all the lookups, -(L2/L3 path, in/out, IPv4/IPv6) - significantly reducing the code complexity. - -Unfortunately, bihash still has the "lack of backpressure" problem, -in a sense that if you try to insert too many entries and run out -of memory in the heap you supplied, you get a crash. - -To somewhat workaround against that, there is a "maximum tested number of sessions" -value, which tracks the currently inserted sessions in the bihash, -and if this number is being approached, a more aggressive cleanup -can happen. If this number is reached, two behaviors are possible: - -* attempt to do the stateless ACL matching and permit the packet - if it succeeds - -* deny the packet - -Currently I have opted for a second one, since it allows for -a better defined behavior, and if you have to permit -the traffic in both directions, why using stateful anyway ? - -In order to be able to do the cleanup, we need to discriminate between -the session types, with each session type having its own idle timeout. -In order to do that, we keep three lists, defined in enum acl_timeout_e: -ACL_TIMEOUT_UDP_IDLE, ACL_TIMEOUT_TCP_IDLE, ACL_TIMEOUT_TCP_TRANSIENT. - -The first one is hopefully obvious - it is just all UDP connections. -They have an idle timeout of 600 seconds. - -The second and third is a bit more subtle. TCP is a complicated protocol, -and we need to tread the fine line between doing too little and doing -too much, and triggering the potential compatibility issues because of -being a "middlebox". - -I decided to split the TCP connections into two classes: -established, and everything else. "Established", means we have seen -the SYN and ACK from both sides (with PUSH obviously masked out). -This is the "active" state of any TCP connection and we would like -to ensure we do not screw it up. So, the connections in this state -have the default idle timer of 24 hours. - -All the rest of the connections have the idle timeout of 2 minutes, -(inspired by an old value of MSL) and based on the observation -that the states this class represent are usually very short lived. - -Once we have these three baskets of connections, it is trivial to -imagine a simple cleanup mechanism to deal with this: take a -TCP transient connection that has been hanging around. - -It is debatable whether we want to do discrimination between the -different TCP transient connections. Assuming we do FIFO (and -the lists allow us to do just that), it means a given connection -on the head of the list has been hanging around for longest. -Thus, if we are short on resources, we might just go ahead and -reuse it within the datapath. - -This is where we are slowly approaching the question -"Why in the world have not you used timer wheel or such ?" - -The answer is simple: within the above constraints, it does -not buy me much. - -Also, timer wheel creates a leaky abstraction with a difficult -to manage corner case. Which corner case ? - -We have a set of objects (sessions) with an event that may -or may not happen (idle timeout timer firing), and a -necessity to reset the idle timeout when there is -activity on the session. - -In the worst case, where we had a 10000 of one-packet -UDP sessions just created 10 minutes ago, we would need -to deal with a spike of 10000 expired timers. - -Of course, if we have the active traffic on all -of these 10000 connections, then we will not have -to deal with that ? Right, but we will still have to deal -with canceling and requeueing the timers. - -In the best possible case, requeueing a timer is -going to be something along the lines of a linked-list -removal and reinsertion. - -However, keep in mind we already need to classify the -connections for reuse, so therefore we already have -the linked lists! - -And if we just check these linked lists periodically in -a FIFO fashion, we can get away with a very simple per-packet operation: -writing back the timestamp of "now" into the connection structure. - -Then rather than requeueing the list on a per-packet or per-frame -basis, we can defer this action until the time this session -appears on the head of the FIFO list, and the cleaning -routine makes the decision about whether to discard -the session (because the interval since last activity is bigger -than the idle timeout), or to requeue the session back to -the end of the list (because the last activity was less -than idle timeout ago). - -So, rather than using the timers, we can simply reuse our classification -FIFOs, with the following heuristic: do not look at the session that was -enqueued at time X until X+session_timeout. If we enqueue the sessions -in the order of their initial activity, then we can simply use enqueue -timestamp of the head session as a decision criterion for when we need -to get back at looking at it for the timeout purposes. - -Since the number of FIFOs is small, we get a slightly worse check -performance than with timers, but still O(1). - -We seemingly do quite a few "useless" operations of requeueing the items -back to the tail of the list - but, these are the operations we do not -have to do in the active data path, so overall it is a win. - -(Diversion: I believe this problem is congruent to poll vs. epoll or -events vs. threads, some reading on this subject: -http://web.archive.org/web/20120225022154/http://sheddingbikes.com/posts/1280829388.html) - -We can also can run a TCP-like scheme for adaptively changing -the wait period in the routine that deals with the connection timeouts: -we can attempt to check the connections a couple of times per second -(same as we would advance the timer wheel), and then if we have requeued -close to a max-per-quantum number of connections, we can half the waiting -interval, and if we did not requeue any, we can slowly increment the waiting -interval - which at a steady state should stabilize similar to what the TCP rate -does. - -reflexive ACLs: multi-thread -============================= - -The single-threaded implementation in 1704 used a separate "cleaner" process -to deal with the timing out of the connections. -It is all good and great when you know that there is only a single core -to run everything on, but the existence of the lists proves to be -a massive difficulty when it comes to operating from multiple threads. - -Initial study shows that with a few assumptions (e.g. that the cleaner running in main thread -and the worker have a demarcation point in time where either one or the other one touches -the session in the list) it might be possible to make it work, but the resulting -trickiness of doing it neatly with all the corner cases is quite large. - -So, for the multi-threaded scenario, we need to move the connection -aging back to the same CPU as its creation. - -Luckily we can do this with the help of the interrupts. - -So, the design is as follows: the aging thread (acl_fa_session_cleaner_process) -periodically fires the interrupts to the workers interrupt nodes (acl_fa_worker_session_cleaner_process_node.index), -using vlib_node_set_interrupt_pending(), and -the interrupt node acl_fa_worker_conn_cleaner_process() calls acl_fa_check_idle_sessions() -which does the actual job of advancing the lists. And within the actual datapath the only thing we will be -doing is putting the items onto FIFO, and updating the last active time on the existing connection. - -The one "delicate" part is that the worker for one leg of the connection might be different from -the worker of another leg of the connection - but, even if the "owner" tries to free the connection, -nothing terrible can happen - worst case the element of the pool (which is nominally free for a short period) -will get the timestamp updated - same thing about the TCP flags seen. - -A slightly trickier issue arises when the packet initially seen by one worker (thus owned by that worker), -and the return packet processed by another worker, and as a result changes the -the class of the connection (e.g. becomes TCP_ESTABLISHED from TCP_TRANSIENT or vice versa). -If the class changes from one with the shorter idle time to the one with the longer idle time, -then unless we are in the starvation mode where the transient connections are recycled, -we can simply do nothing and let the normal requeue mechanism kick in. If the class changes from the longer idle -timer to the shorter idle timer, then we risk keeping the connection around for longer than needed, which -will affect the resource usage. - -One solution to that is to have NxN ring buffers (where N is the number of workers), such that the non-owner -can signal to the owner the connection# that needs to be requeued out of order. - -A simpler solution though, is to ensure that each FIFO's period is equal to that of a shortest timer. -This way the resource starvation problem is taken care of, at an expense of some additional work. - -This all looks sufficiently nice and simple until a skeleton falls out of the closet: -sometimes we want to clean the connections en masse before they expire. - -There few potential scenarios: -1) removal of an ACL from the interface -2) removal of an interface -3) manual action of an operator (in the future). - -In order to tackle this, we need to modify the logic which decides whether to requeue the -connection on the end of the list, or to delete it due to idle timeout: - -We define a point in time, and have each worker thread fast-forward through its FIFO, -in the process looking for sessions that satisfy the criteria, and either keeping them or requeueing them. - -To keep the ease of appearance to the outside world, we still process this as an event -within the connection cleaner thread, but this event handler does as follows: -1) it creates the bitmap of the sw_if_index values requested to be cleared -2) for each worker, it waits to ensure there is no cleanup operation in progress (and if there is one, -it waits), and then makes a copy of the bitmap, sets the per-worker flag of a cleanup operation, and sends an interrupt. -3) wait until all cleanup operations have completed. - -Within the worker interrupt node, we check if the "cleanup in progress" is set, -and if it is, we check the "fast forward time" value. If unset, we initialize it to value now, and compare the -requested bitmap of sw_if_index values (pending_clear_sw_if_index_bitmap) with the bitmap of sw_if_index that this worker deals with. - -(we set the bit in the bitmap every time we enqueue the packet onto a FIFO - serviced_sw_if_index_bitmap in acl_fa_conn_list_add_session). - -If the result of this AND operation is zero - then we can clear the flag of cleanup in progress and return. -Else we kick off the quantum of cleanup, and make sure we get another interrupt ASAP if that cleanup operation returns non-zero, -meaning there is more work to do. -When that operation returns zero, everything has been processed, we can clear the "cleanup-in-progress" flag, and -zeroize the bitmap of sw_if_index-es requested to be cleaned. - -The interrupt node signals its wish to receive an interrupt ASAP by setting interrupt_is_needed -flag within the per-worker structure. The main thread, while waiting for the -cleanup operation to complete, checks if there is a request for interrupt, -and if there is - it sends one. - -This approach gives us a way to mass-clean the connections which is reusing the code of the regular idle -connection cleanup. - -One potential inefficiency is the bitmap values set by the session insertion -in the data path - there is nothing to clear them. - -So, if one rearranges the interface placement with the workers, then the cleanups will cause some unnecessary work. -For now, we consider it an acceptable limitation. It can be resolved by having another per-worker bitmap, which, when set, -would trigger the cleanup of the bits in the serviced_sw_if_index_bitmap). - -=== the end === - diff --git a/src/plugins/acl/acl_multicore_doc.rst b/src/plugins/acl/acl_multicore_doc.rst new file mode 100644 index 00000000000..142b6b216d2 --- /dev/null +++ b/src/plugins/acl/acl_multicore_doc.rst @@ -0,0 +1,354 @@ +Multicore support for ACL plugin +================================ + +This captures some considerations and design decisions that I have made, +both for my own memory later on (“what the hell was I thinking?!?”), and +for anyone interested to criticize/improve/hack on this code. + +One of the factors taken into account while making these decisions, was +the relative emphasis on the multi-thread vs. single-thread use cases: +the latter is the vastly more prevalent. But, one can not optimize the +single-thread performance without having a functioning code for +multi-thread. + +stateless ACLs +-------------- + +The stateless trivially parallelizes, and the only potential for the +race between the different threads is during the reconfiguration, at the +time of replacing the old ACL being checked, with the new ACL. + +In case an acl_add_replace is being used to replace the rules within the +existing entry, a reallocation of ``am->acls[X].rules`` vector will +happen and potentially a change in count. + +acl_match_5tuple() has the following code: + +.. code:: c + + a = am->acls + acl_index; + for (i = 0; i < a->count; i++) + { + r = a->rules + i; + . . . + +Ideally we should be immune from a->rules changing, but the problem +arises if the count changes in flight, and the new ruleset is smaller - +then we will attempt to “match” against the free memory. + +This can(?) be solved by replacing the for() with while(), so the +comparison happens at each iteration. + +full_acl_match_5tuple(), which iterates over the list of ACLs, is a bit +less immune, since it takes the pointer to the vector to iterate and +keeps a local copy of that pointer. + +This race can be solved by checking the current pointer to the vector +with the source pointer, and seeing if there is an (unlikely) change, +and if there is, return the “deny” action, or, better, restart the +check. + +Since the check reloads the ACL list on a per-packet basis, there is +only a window of opportunity of one packet to “match” packet against an +incorrect rule set. The workers also do not change anything, only read. +Therefore, it looks like building special structures to ensure that it +does not happen at all might be not worth it. + +At least not until we have a unit-test able to reliably catch this +condition and test that the measures applied are effective. Adding the +code which is not possible to exercise is worse than not adding any code +at all. + +So, I opt for “do-nothing” here for the moment. + +reflexive ACLs: single-thread +----------------------------- + +Before we talk multi-thread, is worth revisiting the design of the +reflexive ACLs in the plugin, and the history of their evolution. + +The very first version of the ACL plugin, shipped in 1701, mostly did +the job using the existing components and gluing them together. Because +it needed to work in bridged forwarding path only, using L2 classifier +as an insertion point appeared natural, also L2 classifier, being a +table with sessions, seemed like a good place to hold the sessions. + +So, the original design had two conceptual nodes: one, pointed by the +next_miss from the L2 classifier table, was checking the actual ACL, and +inserting session into the L2 classifier table, and the other one, +pointed to by the next_match within the specific session rule, was +checking the existing session. The timing out of the existing +connections was done in the datapath, by periodically calling the aging +function. + +This decision to use the existing components, with its attractiveness, +did bring a few limitations as well: + +- L2 classifier is a simple mask-and-value match, with a fixed mask + across the table. So, sanely supporting IPv6 packets with extension + headers in that framework was impossible. + +- There is no way to get a backpressure from L2 classifier depending on + memory usage. When it runs out of memory, it simply crashes the box. + When it runs out of memory ? We don’t really know. Depends on how it + allocates it. + +- Since we need to match the *reflected* traffic, we had to create + *two* full session entries in two different directions, which is + quite wasteful memory-wise. + +- (showstopper): the L2 classifier runs only in the bridged data path, + so supporting routed data path would require creating something else + entirely different, which would mean much more headaches support-wise + going forward. + +Because of that, I have moved to a different model of creating a +session-5-tuple from the packet data - once, and then doing all the +matching just on that 5-tuple. + +This has allowed to add support for skipping IPv6 extension headers. + +Also, this new version started to store the sessions in a dedicated +bihash-per-interface, with the session key data being aligned for the +ingress packets, and being mirrored for the egress packets. This allows +of significant savings in memory, because now we need to keep only one +copy of the session table per interface instead of two, and also to only +have ONE node for all the lookups, (L2/L3 path, in/out, IPv4/IPv6) - +significantly reducing the code complexity. + +Unfortunately, bihash still has the “lack of backpressure” problem, in a +sense that if you try to insert too many entries and run out of memory +in the heap you supplied, you get a crash. + +To somewhat workaround against that, there is a “maximum tested number +of sessions” value, which tracks the currently inserted sessions in the +bihash, and if this number is being approached, a more aggressive +cleanup can happen. If this number is reached, two behaviors are +possible: + +- attempt to do the stateless ACL matching and permit the packet if it + succeeds + +- deny the packet + +Currently I have opted for a second one, since it allows for a better +defined behavior, and if you have to permit the traffic in both +directions, why using stateful anyway ? + +In order to be able to do the cleanup, we need to discriminate between +the session types, with each session type having its own idle timeout. +In order to do that, we keep three lists, defined in enum acl_timeout_e: +ACL_TIMEOUT_UDP_IDLE, ACL_TIMEOUT_TCP_IDLE, ACL_TIMEOUT_TCP_TRANSIENT. + +The first one is hopefully obvious - it is just all UDP connections. +They have an idle timeout of 600 seconds. + +The second and third is a bit more subtle. TCP is a complicated +protocol, and we need to tread the fine line between doing too little +and doing too much, and triggering the potential compatibility issues +because of being a “middlebox”. + +I decided to split the TCP connections into two classes: established, +and everything else. “Established”, means we have seen the SYN and ACK +from both sides (with PUSH obviously masked out). This is the “active” +state of any TCP connection and we would like to ensure we do not screw +it up. So, the connections in this state have the default idle timer of +24 hours. + +All the rest of the connections have the idle timeout of 2 minutes, +(inspired by an old value of MSL) and based on the observation that the +states this class represent are usually very short lived. + +Once we have these three baskets of connections, it is trivial to +imagine a simple cleanup mechanism to deal with this: take a TCP +transient connection that has been hanging around. + +It is debatable whether we want to do discrimination between the +different TCP transient connections. Assuming we do FIFO (and the lists +allow us to do just that), it means a given connection on the head of +the list has been hanging around for longest. Thus, if we are short on +resources, we might just go ahead and reuse it within the datapath. + +This is where we are slowly approaching the question “Why in the world +have not you used timer wheel or such ?” + +The answer is simple: within the above constraints, it does not buy me +much. + +Also, timer wheel creates a leaky abstraction with a difficult to manage +corner case. Which corner case ? + +We have a set of objects (sessions) with an event that may or may not +happen (idle timeout timer firing), and a necessity to reset the idle +timeout when there is activity on the session. + +In the worst case, where we had a 10000 of one-packet UDP sessions just +created 10 minutes ago, we would need to deal with a spike of 10000 +expired timers. + +Of course, if we have the active traffic on all of these 10000 +connections, then we will not have to deal with that ? Right, but we +will still have to deal with canceling and requeueing the timers. + +In the best possible case, requeueing a timer is going to be something +along the lines of a linked-list removal and reinsertion. + +However, keep in mind we already need to classify the connections for +reuse, so therefore we already have the linked lists! + +And if we just check these linked lists periodically in a FIFO fashion, +we can get away with a very simple per-packet operation: writing back +the timestamp of “now” into the connection structure. + +Then rather than requeueing the list on a per-packet or per-frame basis, +we can defer this action until the time this session appears on the head +of the FIFO list, and the cleaning routine makes the decision about +whether to discard the session (because the interval since last activity +is bigger than the idle timeout), or to requeue the session back to the +end of the list (because the last activity was less than idle timeout +ago). + +So, rather than using the timers, we can simply reuse our classification +FIFOs, with the following heuristic: do not look at the session that was +enqueued at time X until X+session_timeout. If we enqueue the sessions +in the order of their initial activity, then we can simply use enqueue +timestamp of the head session as a decision criterion for when we need +to get back at looking at it for the timeout purposes. + +Since the number of FIFOs is small, we get a slightly worse check +performance than with timers, but still O(1). + +We seemingly do quite a few “useless” operations of requeueing the items +back to the tail of the list - but, these are the operations we do not +have to do in the active data path, so overall it is a win. + +(Diversion: I believe this problem is congruent to poll vs. epoll or +events vs. threads, some reading on this subject: +http://web.archive.org/web/20120225022154/http://sheddingbikes.com/posts/1280829388.html) + +We can also can run a TCP-like scheme for adaptively changing the wait +period in the routine that deals with the connection timeouts: we can +attempt to check the connections a couple of times per second (same as +we would advance the timer wheel), and then if we have requeued close to +a max-per-quantum number of connections, we can half the waiting +interval, and if we did not requeue any, we can slowly increment the +waiting interval - which at a steady state should stabilize similar to +what the TCP rate does. + +reflexive ACLs: multi-thread +---------------------------- + +The single-threaded implementation in 1704 used a separate “cleaner” +process to deal with the timing out of the connections. It is all good +and great when you know that there is only a single core to run +everything on, but the existence of the lists proves to be a massive +difficulty when it comes to operating from multiple threads. + +Initial study shows that with a few assumptions (e.g. that the cleaner +running in main thread and the worker have a demarcation point in time +where either one or the other one touches the session in the list) it +might be possible to make it work, but the resulting trickiness of doing +it neatly with all the corner cases is quite large. + +So, for the multi-threaded scenario, we need to move the connection +aging back to the same CPU as its creation. + +Luckily we can do this with the help of the interrupts. + +So, the design is as follows: the aging thread +(acl_fa_session_cleaner_process) periodically fires the interrupts to +the workers interrupt nodes +(acl_fa_worker_session_cleaner_process_node.index), using +vlib_node_set_interrupt_pending(), and the interrupt node +acl_fa_worker_conn_cleaner_process() calls acl_fa_check_idle_sessions() +which does the actual job of advancing the lists. And within the actual +datapath the only thing we will be doing is putting the items onto FIFO, +and updating the last active time on the existing connection. + +The one “delicate” part is that the worker for one leg of the connection +might be different from the worker of another leg of the connection - +but, even if the “owner” tries to free the connection, nothing terrible +can happen - worst case the element of the pool (which is nominally free +for a short period) will get the timestamp updated - same thing about +the TCP flags seen. + +A slightly trickier issue arises when the packet initially seen by one +worker (thus owned by that worker), and the return packet processed by +another worker, and as a result changes the the class of the connection +(e.g. becomes TCP_ESTABLISHED from TCP_TRANSIENT or vice versa). If the +class changes from one with the shorter idle time to the one with the +longer idle time, then unless we are in the starvation mode where the +transient connections are recycled, we can simply do nothing and let the +normal requeue mechanism kick in. If the class changes from the longer +idle timer to the shorter idle timer, then we risk keeping the +connection around for longer than needed, which will affect the resource +usage. + +One solution to that is to have NxN ring buffers (where N is the number +of workers), such that the non-owner can signal to the owner the +connection# that needs to be requeued out of order. + +A simpler solution though, is to ensure that each FIFO’s period is equal +to that of a shortest timer. This way the resource starvation problem is +taken care of, at an expense of some additional work. + +This all looks sufficiently nice and simple until a skeleton falls out +of the closet: sometimes we want to clean the connections en masse +before they expire. + +There few potential scenarios: 1) removal of an ACL from the interface +2) removal of an interface 3) manual action of an operator (in the +future). + +In order to tackle this, we need to modify the logic which decides +whether to requeue the connection on the end of the list, or to delete +it due to idle timeout: + +We define a point in time, and have each worker thread fast-forward +through its FIFO, in the process looking for sessions that satisfy the +criteria, and either keeping them or requeueing them. + +To keep the ease of appearance to the outside world, we still process +this as an event within the connection cleaner thread, but this event +handler does as follows: 1) it creates the bitmap of the sw_if_index +values requested to be cleared 2) for each worker, it waits to ensure +there is no cleanup operation in progress (and if there is one, it +waits), and then makes a copy of the bitmap, sets the per-worker flag of +a cleanup operation, and sends an interrupt. 3) wait until all cleanup +operations have completed. + +Within the worker interrupt node, we check if the “cleanup in progress” +is set, and if it is, we check the “fast forward time” value. If unset, +we initialize it to value now, and compare the requested bitmap of +sw_if_index values (pending_clear_sw_if_index_bitmap) with the bitmap of +sw_if_index that this worker deals with. + +(we set the bit in the bitmap every time we enqueue the packet onto a +FIFO - serviced_sw_if_index_bitmap in acl_fa_conn_list_add_session). + +If the result of this AND operation is zero - then we can clear the flag +of cleanup in progress and return. Else we kick off the quantum of +cleanup, and make sure we get another interrupt ASAP if that cleanup +operation returns non-zero, meaning there is more work to do. When that +operation returns zero, everything has been processed, we can clear the +“cleanup-in-progress” flag, and zeroize the bitmap of sw_if_index-es +requested to be cleaned. + +The interrupt node signals its wish to receive an interrupt ASAP by +setting interrupt_is_needed flag within the per-worker structure. The +main thread, while waiting for the cleanup operation to complete, checks +if there is a request for interrupt, and if there is - it sends one. + +This approach gives us a way to mass-clean the connections which is +reusing the code of the regular idle connection cleanup. + +One potential inefficiency is the bitmap values set by the session +insertion in the data path - there is nothing to clear them. + +So, if one rearranges the interface placement with the workers, then the +cleanups will cause some unnecessary work. For now, we consider it an +acceptable limitation. It can be resolved by having another per-worker +bitmap, which, when set, would trigger the cleanup of the bits in the +serviced_sw_if_index_bitmap). + +=== the end === diff --git a/src/plugins/af_xdp/af_xdp_doc.md b/src/plugins/af_xdp/af_xdp_doc.md deleted file mode 100644 index f5859dbb901..00000000000 --- a/src/plugins/af_xdp/af_xdp_doc.md +++ /dev/null @@ -1,129 +0,0 @@ -# AF_XDP Ethernet driver {#af_xdp_doc} - -This driver relies on Linux AF_XDP socket to rx/tx Ethernet packets. - -## Maturity level -Under development: it should work, but has not been thoroughly tested. - -## Features - - copy and zero-copy mode - - multiqueue - - API - - custom eBPF program - - polling, interrupt and adaptive mode - -## Known limitations - -### MTU -Because of AF_XDP restrictions, the MTU is limited to below PAGE_SIZE -(4096-bytes on most systems) minus 256-bytes, and they are additional -limitations depending upon specific Linux device drivers. -As a rule of thumb, a MTU of 3000-bytes or less should be safe. - -### Number of buffers -Furthermore, upon UMEM creation, the kernel allocates a -physically-contiguous structure, whose size is proportional to the number -of 4KB pages contained in the UMEM. That allocation might fail when -the number of buffers allocated by VPP is too high. That number can be -controlled with the `buffers { buffers-per-numa }` configuration option. -Finally, note that because of this limitation, this plugin is unlikely -to be compatible with the use of 1GB hugepages. - -### Interrupt mode -Interrupt and adaptive mode are supported but is limited by default to single -threaded (no worker) configurations because of a kernel limitation prior to -5.6. You can bypass the limitation at interface creation time by adding the -`no-syscall-lock` parameter, but you must be sure that your kernel can -support it, otherwise you will experience double-frees. -See -https://lore.kernel.org/bpf/BYAPR11MB365382C5DB1E5FCC53242609C1549@BYAPR11MB3653.namprd11.prod.outlook.com/ -for more details. - -### Mellanox -When setting the number of queues on Mellanox NIC with `ethtool -L`, you must -use twice the amount of configured queues: it looks like the Linux driver will -create separate RX queues and TX queues (but all queues can be used for both -RX and TX, the NIC will just not sent any packet on "pure" TX queues. -Confused? So I am.). For example if you set `combined 2` you will effectively -have to create 4 rx queues in AF_XDP if you want to be sure to receive all -packets. - -## Requirements -This drivers supports Linux kernel 5.4 and later. Kernels older than 5.4 are -missing unaligned buffers support. - -The Linux kernel interface must be up and have enough queues before -creating the VPP AF_XDP interface, otherwise Linux will deny creating -the AF_XDP socket. -The AF_XDP interface will claim NIC RX queue starting from 0, up to the -requested number of RX queues (only 1 by default). It means all packets -destined to NIC RX queue `[0, num_rx_queues[` will be received by the -AF_XDP interface, and only them. Depending on your configuration, there -will usually be several RX queues (typically 1 per core) and packets are -spread accross queues by RSS. In order to receive consistent traffic, -you **must** program the NIC dispatching accordingly. The simplest way -to get all the packets is to specify `num-rx-queues all` to grab all -available queues or to reconfigure the Linux kernel driver to use only -`num_rx_queues` RX queues (ie all NIC queues will be associated with -the AF_XDP socket): -``` -~# ethtool -L <iface> combined <num_rx_queues> -``` -Additionally, the VPP AF_XDP interface will use a MAC address generated at -creation time instead of the Linux kernel interface MAC. As Linux kernel -interface are not in promiscuous mode by default (see below) this will -results in a useless configuration where the VPP AF_XDP interface only -receives packets destined to the Linux kernel interface MAC just to drop -them because the destination MAC does not match VPP AF_XDP interface MAC. -If you want to use the Linux interface MAC for the VPP AF_XDP interface, -you can change it afterwards in VPP: -``` -~# vppctl set int mac address <iface> <mac> -``` -Finally, if you wish to receive all packets and not only the packets -destined to the Linux kernel interface MAC you need to set the Linux -kernel interface in promiscuous mode: -``` -~# ip link set dev <iface> promisc on -``` - -## Security considerations -When creating an AF_XDP interface, it will receive all packets arriving -to the NIC RX queue `[0, num_rx_queues[`. You need to configure the Linux -kernel NIC driver properly to ensure that only intented packets will -arrive in this queue. There is no way to filter the packets after-the-fact -using eg. netfilter or eBPF. - -## Quickstart -1. Put the Linux kernel interface up and in promiscuous mode: -``` -~# ip l set dev enp216s0f0 promisc on up -``` -2. Create the AF_XDP interface: -``` -~# vppctl create int af_xdp host-if enp216s0f0 num-rx-queues all -``` -3. Use the interface as usual, eg.: -``` -~# vppctl set int ip addr enp216s0f0/0 1.1.1.1/24 -~# vppctl set int st enp216s0f0/0 up -~# vppctl ping 1.1.1.100` -``` - -## Custom eBPF XDP program -This driver relies on libbpf and as such relies on the `xsks_map` eBPF -map. The default behavior is to use the XDP program already attached -to the interface if any, otherwise load the default one. -You can request to load a custom XDP program with the `prog` option when -creating the interface in VPP: -``` -~# vppctl create int af_xdp host-if enp216s0f0 num-rx-queues 4 prog extras/bpf/af_xdp.bpf.o -``` -In that case it will replace any previously attached program. A custom -XDP program example is provided in `extras/bpf/`. - -## Performance consideration -AF_XDP relies on the Linux kernel NIC driver to rx/tx packets. To reach -high-performance (10's MPPS), the Linux kernel NIC driver must support -zero-copy mode and its RX path must run on a dedicated core in the NUMA -where the NIC is physically connected. diff --git a/src/plugins/af_xdp/af_xdp_doc.rst b/src/plugins/af_xdp/af_xdp_doc.rst new file mode 100644 index 00000000000..de951340a2d --- /dev/null +++ b/src/plugins/af_xdp/af_xdp_doc.rst @@ -0,0 +1,164 @@ +AF_XDP device driver +==================== + +This driver relies on Linux AF_XDP socket to rx/tx Ethernet packets. + +Maturity level +-------------- + +Under development: it should work, but has not been thoroughly tested. + +Features +-------- + +- copy and zero-copy mode +- multiqueue +- API +- custom eBPF program +- polling, interrupt and adaptive mode + +Known limitations +----------------- + +MTU +~~~ + +Because of AF_XDP restrictions, the MTU is limited to below PAGE_SIZE +(4096-bytes on most systems) minus 256-bytes, and they are additional +limitations depending upon specific Linux device drivers. As a rule of +thumb, a MTU of 3000-bytes or less should be safe. + +Number of buffers +~~~~~~~~~~~~~~~~~ + +Furthermore, upon UMEM creation, the kernel allocates a +physically-contiguous structure, whose size is proportional to the +number of 4KB pages contained in the UMEM. That allocation might fail +when the number of buffers allocated by VPP is too high. That number can +be controlled with the ``buffers { buffers-per-numa }`` configuration +option. Finally, note that because of this limitation, this plugin is +unlikely to be compatible with the use of 1GB hugepages. + +Interrupt mode +~~~~~~~~~~~~~~ + +Interrupt and adaptive mode are supported but is limited by default to +single threaded (no worker) configurations because of a kernel +limitation prior to 5.6. You can bypass the limitation at interface +creation time by adding the ``no-syscall-lock`` parameter, but you must +be sure that your kernel can support it, otherwise you will experience +double-frees. See +https://lore.kernel.org/bpf/BYAPR11MB365382C5DB1E5FCC53242609C1549@BYAPR11MB3653.namprd11.prod.outlook.com/ +for more details. + +Mellanox +~~~~~~~~ + +When setting the number of queues on Mellanox NIC with ``ethtool -L``, +you must use twice the amount of configured queues: it looks like the +Linux driver will create separate RX queues and TX queues (but all +queues can be used for both RX and TX, the NIC will just not sent any +packet on “pure” TX queues. Confused? So I am.). For example if you set +``combined 2`` you will effectively have to create 4 rx queues in AF_XDP +if you want to be sure to receive all packets. + +Requirements +------------ + +This drivers supports Linux kernel 5.4 and later. Kernels older than 5.4 +are missing unaligned buffers support. + +The Linux kernel interface must be up and have enough queues before +creating the VPP AF_XDP interface, otherwise Linux will deny creating +the AF_XDP socket. The AF_XDP interface will claim NIC RX queue starting +from 0, up to the requested number of RX queues (only 1 by default). It +means all packets destined to NIC RX queue ``[0, num_rx_queues[`` will +be received by the AF_XDP interface, and only them. Depending on your +configuration, there will usually be several RX queues (typically 1 per +core) and packets are spread across queues by RSS. In order to receive +consistent traffic, you **must** program the NIC dispatching +accordingly. The simplest way to get all the packets is to specify +``num-rx-queues all`` to grab all available queues or to reconfigure the +Linux kernel driver to use only ``num_rx_queues`` RX queues (i.e. all NIC +queues will be associated with the AF_XDP socket): + +:: + + ~# ethtool -L <iface> combined <num_rx_queues> + +Additionally, the VPP AF_XDP interface will use a MAC address generated +at creation time instead of the Linux kernel interface MAC. As Linux +kernel interface are not in promiscuous mode by default (see below) this +will results in a useless configuration where the VPP AF_XDP interface +only receives packets destined to the Linux kernel interface MAC just to +drop them because the destination MAC does not match VPP AF_XDP +interface MAC. If you want to use the Linux interface MAC for the VPP +AF_XDP interface, you can change it afterwards in VPP: + +:: + + ~# vppctl set int mac address <iface> <mac> + +Finally, if you wish to receive all packets and not only the packets +destined to the Linux kernel interface MAC you need to set the Linux +kernel interface in promiscuous mode: + +:: + + ~# ip link set dev <iface> promisc on + +Security considerations +----------------------- + +When creating an AF_XDP interface, it will receive all packets arriving +to the NIC RX queue ``[0, num_rx_queues[``. You need to configure the +Linux kernel NIC driver properly to ensure that only intended packets +will arrive in this queue. There is no way to filter the packets +after-the-fact using e.g. netfilter or eBPF. + +Quickstart +---------- + +1. Put the Linux kernel interface up and in promiscuous mode: + +:: + + ~# ip l set dev enp216s0f0 promisc on up + +2. Create the AF_XDP interface: + +:: + + ~# vppctl create int af_xdp host-if enp216s0f0 num-rx-queues all + +3. Use the interface as usual, e.g.: + +:: + + ~# vppctl set int ip addr enp216s0f0/0 1.1.1.1/24 + ~# vppctl set int st enp216s0f0/0 up + ~# vppctl ping 1.1.1.100` + +Custom eBPF XDP program +----------------------- + +This driver relies on libbpf and as such relies on the ``xsks_map`` eBPF +map. The default behavior is to use the XDP program already attached to +the interface if any, otherwise load the default one. You can request to +load a custom XDP program with the ``prog`` option when creating the +interface in VPP: + +:: + + ~# vppctl create int af_xdp host-if enp216s0f0 num-rx-queues 4 prog extras/bpf/af_xdp.bpf.o + +In that case it will replace any previously attached program. A custom +XDP program example is provided in ``extras/bpf/``. + +Performance consideration +------------------------- + +AF_XDP relies on the Linux kernel NIC driver to rx/tx packets. To reach +high-performance (10’s MPPS), the Linux kernel NIC driver must support +zero-copy mode and its RX path must run on a dedicated core in the NUMA +where the NIC is physically connected. diff --git a/src/plugins/avf/README.md b/src/plugins/avf/README.md deleted file mode 100644 index 7aa2661fbba..00000000000 --- a/src/plugins/avf/README.md +++ /dev/null @@ -1,107 +0,0 @@ -# Intel AVF device plugin for VPP {#avf_plugin_doc} - -##Overview -This plugins provides native device support for intel Adaptive Virtual -Function (AVF). AVF is driver specification for current and future -Intel Virtual Function devices. AVF defines communication channel between -Physical Functions (PF) and VF. -In essence, today this driver can be used only with -Intel XL710 / X710 / XXV710 adapters. - -##Prerequisites - * Driver requires newer i40e PF linux driver to be installed on the system, -which supports virtualchnl interface. This code is tested with i40e driver -version 2.4.6. - -* Driver requires MSI-X interrupt support, which is not supported by -uio_pci_generic driver, so vfio-pci needs to be used. On systems without IOMMU -vfio driver can still be used with recent kernels which support no-iommu mode. - -##Known issues -This driver is still in experimental phase, however it shows very good -performance numbers. - -## Usage -### System setup - -1. load VFIO driver -``` -sudo modprobe vfio-pci -``` - -2. (systems without IOMMU only) enable unsafe NOIOMMU mode -``` -echo Y | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode -``` - -3. Create and bind SR-IOV virtual function(s) - -Following script creates VF, assigns MAC address and binds VF to vfio-pci -```bash -#!/bin/bash - -if [ $USER != "root" ] ; then - echo "Restarting script with sudo..." - sudo $0 ${*} - exit -fi - -setup () { - cd /sys/bus/pci/devices/${1} - driver=$(basename $(readlink driver)) - if [ "${driver}" != "i40e" ]; then - echo ${1} | tee driver/unbind - echo ${1} | tee /sys/bus/pci/drivers/i40e/bind - fi - ifname=$(basename net/*) - echo 0 | tee sriov_numvfs > /dev/null - echo 1 | tee sriov_numvfs > /dev/null - ip link set dev ${ifname} vf 0 mac ${2} - ip link show dev ${ifname} - vf=$(basename $(readlink virtfn0)) - echo ${vf} | tee virtfn0/driver/unbind - echo vfio-pci | tee virtfn0/driver_override - echo ${vf} | sudo tee /sys/bus/pci/drivers/vfio-pci/bind - echo | tee virtfn0/driver_override -} - -# Setup one VF on PF 0000:3b:00.0 and assign MAC address -setup 0000:3b:00.0 00:11:22:33:44:00 -# Setup one VF on PF 0000:3b:00.1 and assign MAC address -setup 0000:3b:00.1 00:11:22:33:44:01 -``` - -### Promisc mode -In cases when interface is used in the L2 mode or promisc mode is needed for some other reason, -trust needs to be set to "on" using the linux "ip link" utility. -``` -ip link set dev <PF inteface name> vf <VF id> trust on -``` - -### L2 spoofing check -By default Virtual Function is not allowed to send ethernet frames which -have source MAC address different than address assigned to the VF. -In some cases it is expected that VPP will send such frames (e.g. L2 bridging, -bonding, l2 cross-connect) and in such cases spoof chack needs to be turned -off by issuing following command: -``` -ip link set dev <PF inteface name> vf <VF id> spoofchk off -``` - -### Interface Creation -Interfaces can be dynamically created by using following CLI: -``` -create interface avf 0000:3b:02.0 -set int state avf-0/3b/2/0 up -``` - -### Interface Deletion -Interface can be deleted with following CLI: -``` -delete interface avf <interface name> -``` - -### Interface Statistics -Interface statistics can be displayed with `sh hardware-interface <if-name>` -command. - diff --git a/src/plugins/avf/README.rst b/src/plugins/avf/README.rst new file mode 100644 index 00000000000..339f5f13c3e --- /dev/null +++ b/src/plugins/avf/README.rst @@ -0,0 +1,135 @@ +Intel AVF device driver +======================= + +Overview +-------- + +This plugins provides native device support for intel Adaptive Virtual +Function (AVF). AVF is driver specification for current and future Intel +Virtual Function devices. AVF defines communication channel between +Physical Functions (PF) and VF. In essence, today this driver can be +used only with Intel XL710 / X710 / XXV710 adapters. + +Prerequisites +------------- + +- Driver requires newer i40e PF linux driver to be installed on the + system, which supports virtualchnl interface. This code is tested + with i40e driver version 2.4.6. + +- Driver requires MSI-X interrupt support, which is not supported by + uio_pci_generic driver, so vfio-pci needs to be used. On systems + without IOMMU vfio driver can still be used with recent kernels which + support no-iommu mode. + +Known issues +------------ + +This driver is still in experimental phase, however it shows very good +performance numbers. + +Usage +----- + +System setup +~~~~~~~~~~~~ + +1. load VFIO driver + +:: + + sudo modprobe vfio-pci + +2. (systems without IOMMU only) enable unsafe NOIOMMU mode + +:: + + echo Y | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + +3. Create and bind SR-IOV virtual function(s) + +Following script creates VF, assigns MAC address and binds VF to +vfio-pci + +.. code:: bash + + #!/bin/bash + + if [ $USER != "root" ] ; then + echo "Restarting script with sudo..." + sudo $0 ${*} + exit + fi + + setup () { + cd /sys/bus/pci/devices/${1} + driver=$(basename $(readlink driver)) + if [ "${driver}" != "i40e" ]; then + echo ${1} | tee driver/unbind + echo ${1} | tee /sys/bus/pci/drivers/i40e/bind + fi + ifname=$(basename net/*) + echo 0 | tee sriov_numvfs > /dev/null + echo 1 | tee sriov_numvfs > /dev/null + ip link set dev ${ifname} vf 0 mac ${2} + ip link show dev ${ifname} + vf=$(basename $(readlink virtfn0)) + echo ${vf} | tee virtfn0/driver/unbind + echo vfio-pci | tee virtfn0/driver_override + echo ${vf} | sudo tee /sys/bus/pci/drivers/vfio-pci/bind + echo | tee virtfn0/driver_override + } + + # Setup one VF on PF 0000:3b:00.0 and assign MAC address + setup 0000:3b:00.0 00:11:22:33:44:00 + # Setup one VF on PF 0000:3b:00.1 and assign MAC address + setup 0000:3b:00.1 00:11:22:33:44:01 + +Promisc mode +~~~~~~~~~~~~ + +In cases when interface is used in the L2 mode or promisc mode is needed +for some other reason, trust needs to be set to “on” using the linux “ip +link” utility. + +:: + + ip link set dev <PF inteface name> vf <VF id> trust on + +L2 spoofing check +~~~~~~~~~~~~~~~~~ + +By default Virtual Function is not allowed to send ethernet frames which +have source MAC address different than address assigned to the VF. In +some cases it is expected that VPP will send such frames (e.g. L2 +bridging, bonding, l2 cross-connect) and in such cases spoof check needs +to be turned off by issuing following command: + +:: + + ip link set dev <PF inteface name> vf <VF id> spoofchk off + +Interface Creation +~~~~~~~~~~~~~~~~~~ + +Interfaces can be dynamically created by using following CLI: + +:: + + create interface avf 0000:3b:02.0 + set int state avf-0/3b/2/0 up + +Interface Deletion +~~~~~~~~~~~~~~~~~~ + +Interface can be deleted with following CLI: + +:: + + delete interface avf <interface name> + +Interface Statistics +~~~~~~~~~~~~~~~~~~~~ + +Interface statistics can be displayed with +``sh hardware-interface <if-name>`` command. diff --git a/src/plugins/bufmon/bufmon_doc.md b/src/plugins/bufmon/bufmon_doc.md deleted file mode 100644 index bfa88f40a7d..00000000000 --- a/src/plugins/bufmon/bufmon_doc.md +++ /dev/null @@ -1,24 +0,0 @@ -# Buffers monitoring plugin {#bufmon_doc} - -This plugin enables to track buffer utilization in the VPP graph nodes. The -main use is to detect buffer leakage. -It works by keeping track of number of buffer allocations and free in graph -nodes and also of number of buffers received in input frames and in output -frames. -The formula to compute the number of "buffered" buffers in a node is simply: - #buffered = #alloc + #input - #free - #output -Note: monitoring will impact performances. - -## Basic usage -1. Turn buffer traces on: -``` -~# vppctl set buffer traces on -``` -2. Monitor buffer usage: -``` -~# vppctl show buffer traces verbose -``` -3. Turn buffer traces off: -``` -~# vppctl set buffer traces off -``` diff --git a/src/plugins/bufmon/bufmon_doc.rst b/src/plugins/bufmon/bufmon_doc.rst new file mode 100644 index 00000000000..34d5bd35474 --- /dev/null +++ b/src/plugins/bufmon/bufmon_doc.rst @@ -0,0 +1,33 @@ +.. _bufmon_doc: + +Buffers monitoring plugin +========================= + +This plugin enables to track buffer utilization in the VPP graph nodes. +The main use is to detect buffer leakage. It works by keeping track of +number of buffer allocations and free in graph nodes and also of number +of buffers received in input frames and in output frames. The formula to +compute the number of “buffered” buffers in a node is simply: #buffered += #alloc + #input - #free - #output Note: monitoring will impact +performances. + +Basic usage +----------- + +1. Turn buffer traces on: + +:: + + ~# vppctl set buffer traces on + +2. Monitor buffer usage: + +:: + + ~# vppctl show buffer traces verbose + +3. Turn buffer traces off: + +:: + + ~# vppctl set buffer traces off diff --git a/src/plugins/dhcp/dhcp6_pd_doc.md b/src/plugins/dhcp/dhcp6_pd_doc.md deleted file mode 100644 index 0d0e0865f1b..00000000000 --- a/src/plugins/dhcp/dhcp6_pd_doc.md +++ /dev/null @@ -1,86 +0,0 @@ -# DHCPv6 prefix delegation {#dhcp6_pd_doc} - -DHCPv6 prefix delegation client implementation is split between Control Plane and Data Plane. -Data Plane can also be used alone by external application (external Control Plane) using Data Plane Binary API. - -Number of different IA\_PDs managed by VPP is currently limited to 1 (and corresponding IAID has value 1). -Client ID is of type DUID-LLT (Link Layer address plus time) and is created on VPP startup from avaliable interfaces (or chosen at random for debugging purposes). -Server ID is only visible to Data Plane. Control Plane identifies servers by a 32-bit handle (server\_index) mapped to Server ID by Data Plane. - -## Control Plane - -DHCPv6 PD clients are configured per interface. -When configuring a PD client we have to choose a name of a prefix group for that client. -Each prefix obtained through this client will be flagged as belonging to specified prefix group. -The prefix groups are used as a filter by prefix consumers. - -To enable client on particular interface call Binary API function dhcp6\_pd\_client\_enable\_disable with param 'sw\_if\_index' set to that interface, -'prefix\_group' set to prefix group name and 'enable' set to true. -Format of corresponding Debug CLI command is: "dhcp6 pd client <interface> [disable]" - -To add/delete IPv6 address potentially using available prefix from specified prefix group call Binary API command ip6\_add\_del\_address\_using\_prefix with parameters: -> sw\_if\_index - software interface index of interface to add/delete address to/from -> prefix\_group - name of prefix group, prefix\_group[0] == '\0' means no prefix should be used -> address - address or suffix to be used with a prefix from selected group -> prefix\_length - subnet prefix for the address -> is\_add - 1 for add, 0 for remove -or Debug CLI command with format: "set ip6 addresses <interface> [prefix group <n>] <address> [del]" - -When no prefix is avaliable, no address is physically added, but is added once a prefix becomes avaliable. -Address is removed when all available prefixes are removed. -When a used prefix is removed and there is other available prefix, the address that used the prefix is reconfigured using the available prefix. - -There are three debug CLI commands (with no parameters) used to show the state of clients, prefixes and addresses: - show ip6 pd clients - show ip6 prefixes - show ip6 addresses - -### Example configuration - -set int state GigabitEthernet0/8/0 up -dhcp6 pd client GigabitEthernet0/8/0 prefix group my-dhcp6-pd-group -set ip6 address GigabitEthernet0/8/0 prefix group my-dhcp6-pd-group ::7/64 - -## Data Plane - -First API message to be called is dhcp6\_clients\_enable\_disable with enable parameter set to 1. -It enables DHCPv6 client subsystem to receive UDP messages containing DHCPv6 client port (sets the router to DHCPv6 client mode). -This is to ensure client subsystem gets the messages instead of DHCPv6 proxy subsystem. - -There is one common Binary API call for sending DHCPv6 client messages (dhcp6\_pd\_send\_client\_message) with these fields: -> msg\_type - message type (e.g. Solicit) -> sw\_if\_index - index of TX interface -> server\_index - used to dentify DHCPv6 server, - unique for each DHCPv6 server on the link, - value obrtained from dhcp6\_pd\_reply\_event API message, - use ~0 to send message to all DHCPv6 servers -> param irt - initial retransmission time -> param mrt - maximum retransmission time -> param mrc - maximum retransmission count -> param mrd - maximum retransmission duration for sending the message -> stop - if non-zero then stop resending the message, otherwise start sending the message -> T1 - value of T1 in IA\_PD option -> T2 - value of T2 in IA\_PD option -> prefixes - list of prefixes in IA\_PD option - -The message is automatically resent by Data Plane based on parameters 'irt', 'mrt', 'mrc' and 'mrd'. -To stop the resending call the same function (same msg\_type is sufficient) with 'stop' set to 1. - -To subscribe for notifications of DHCPv6 messages from server call Binary API function -want\_dhcp6\_pd\_reply\_events with enable\_disable set to 1 -Notification (dhcp6\_pd\_reply\_event) fileds are: -> sw\_if\_index - index of RX interface -> server\_index - used to dentify DHCPv6 server, unique for each DHCPv6 server on the link -> msg\_type - message type -> T1 - value of T1 in IA\_PD option -> T2 - value of T2 in IA\_PD option -> inner\_status\_code - value of status code inside IA\_PD option -> status\_code - value of status code -> preference - value of preference option in reply message -> prefixes - list of prefixes in IA\_PD option - -Prefix is a struct with with these fields: -> prefix - prefix bytes -> prefix\_length - prefix length -> valid\_time - valid lifetime -> preferred\_time - preferred lifetime diff --git a/src/plugins/dhcp/dhcp6_pd_doc.rst b/src/plugins/dhcp/dhcp6_pd_doc.rst new file mode 100644 index 00000000000..349abe215e1 --- /dev/null +++ b/src/plugins/dhcp/dhcp6_pd_doc.rst @@ -0,0 +1,113 @@ +DHCPv6 prefix delegation +======================== + +| DHCPv6 prefix delegation client implementation is split between + Control Plane and Data Plane. +| Data Plane can also be used alone by external application (external + Control Plane) using Data Plane Binary API. + +| Number of different IA_PDs managed by VPP is currently limited to 1 + (and corresponding IAID has value 1). +| Client ID is of type DUID-LLT (Link Layer address plus time) and is + created on VPP startup from avaliable interfaces (or chosen at random + for debugging purposes). +| Server ID is only visible to Data Plane. Control Plane identifies + servers by a 32-bit handle (server_index) mapped to Server ID by Data + Plane. + +Control Plane +------------- + +| DHCPv6 PD clients are configured per interface. +| When configuring a PD client we have to choose a name of a prefix + group for that client. +| Each prefix obtained through this client will be flagged as belonging + to specified prefix group. +| The prefix groups are used as a filter by prefix consumers. + +| To enable client on particular interface call Binary API function + dhcp6_pd_client_enable_disable with param ‘sw_if_index’ set to that + interface, ‘prefix_group’ set to prefix group name and ‘enable’ set to + true. +| Format of corresponding Debug CLI command is: “dhcp6 pd client + [disable]” + +To add/delete IPv6 address potentially using available prefix from +specified prefix group call Binary API command +ip6_add_del_address_using_prefix with parameters: +> sw_if_index - software interface index of interface to add/delete +address to/from > prefix_group - name of prefix group, prefix_group[0] +== ‘\\0’ means no prefix should be used > address - address or suffix to +be used with a prefix from selected group > prefix_length - subnet +prefix for the address > is_add - 1 for add, 0 for remove or Debug CLI +command with format: “set ip6 addresses [prefix group ] + +.. raw:: html + + <address> + +[del]” + +| When no prefix is avaliable, no address is physically added, but is + added once a prefix becomes avaliable. +| Address is removed when all available prefixes are removed. +| When a used prefix is removed and there is other available prefix, the + address that used the prefix is reconfigured using the available + prefix. + +| There are three debug CLI commands (with no parameters) used to show + the state of clients, prefixes and addresses: +| show ip6 pd clients +| show ip6 prefixes +| show ip6 addresses + +Example configuration +~~~~~~~~~~~~~~~~~~~~~ + +set int state GigabitEthernet0/8/0 up dhcp6 pd client +GigabitEthernet0/8/0 prefix group my-dhcp6-pd-group set ip6 address +GigabitEthernet0/8/0 prefix group my-dhcp6-pd-group ::7/64 + +Data Plane +---------- + +| First API message to be called is dhcp6_clients_enable_disable with + enable parameter set to 1. +| It enables DHCPv6 client subsystem to receive UDP messages containing + DHCPv6 client port (sets the router to DHCPv6 client mode). +| This is to ensure client subsystem gets the messages instead of DHCPv6 + proxy subsystem. + +| There is one common Binary API call for sending DHCPv6 client messages + (dhcp6_pd_send_client_message) with these fields: +| > msg_type - message type (e.g. Solicit) > sw_if_index - index of TX + interface > server_index - used to dentify DHCPv6 server, unique for + each DHCPv6 server on the link, value obrtained from + dhcp6_pd_reply_event API message, use ~0 to send message to all DHCPv6 + servers > param irt - initial retransmission time > param mrt - + maximum retransmission time > param mrc - maximum retransmission count + > param mrd - maximum retransmission duration for sending the message + > stop - if non-zero then stop resending the message, otherwise start + sending the message > T1 - value of T1 in IA_PD option > T2 - value of + T2 in IA_PD option > prefixes - list of prefixes in IA_PD option + +| The message is automatically resent by Data Plane based on parameters + ‘irt’, ‘mrt’, ‘mrc’ and ‘mrd’. +| To stop the resending call the same function (same msg_type is + sufficient) with ‘stop’ set to 1. + +| To subscribe for notifications of DHCPv6 messages from server call + Binary API function +| want_dhcp6_pd_reply_events with enable_disable set to 1 +| Notification (dhcp6_pd_reply_event) fileds are: +| > sw_if_index - index of RX interface > server_index - used to dentify + DHCPv6 server, unique for each DHCPv6 server on the link > msg_type - + message type > T1 - value of T1 in IA_PD option > T2 - value of T2 in + IA_PD option > inner_status_code - value of status code inside IA_PD + option > status_code - value of status code > preference - value of + preference option in reply message > prefixes - list of prefixes in + IA_PD option + +| Prefix is a struct with with these fields: +| > prefix - prefix bytes > prefix_length - prefix length > valid_time - + valid lifetime > preferred_time - preferred lifetime diff --git a/src/plugins/flowprobe/flowprobe_plugin_doc.md b/src/plugins/flowprobe/flowprobe_plugin_doc.md deleted file mode 100644 index 4c9b2342a83..00000000000 --- a/src/plugins/flowprobe/flowprobe_plugin_doc.md +++ /dev/null @@ -1,13 +0,0 @@ -IPFIX flow record plugin {#flowprobe_plugin_doc} -======================== - -## Introduction - -This plugin generates ipfix flow records on interfaces which have the feature enabled - -## Sample configuration - -set ipfix exporter collector 192.168.6.2 src 192.168.6.1 template-interval 20 port 4739 path-mtu 1500 - -flowprobe params record l3 active 20 passive 120 -flowprobe feature add-del GigabitEthernet2/3/0 l2
\ No newline at end of file diff --git a/src/plugins/flowprobe/flowprobe_plugin_doc.rst b/src/plugins/flowprobe/flowprobe_plugin_doc.rst new file mode 100644 index 00000000000..8ad9c88ffbf --- /dev/null +++ b/src/plugins/flowprobe/flowprobe_plugin_doc.rst @@ -0,0 +1,17 @@ +IPFIX flow record plugin +======================== + +Introduction +------------ + +This plugin generates ipfix flow records on interfaces which have the +feature enabled + +Sample configuration +-------------------- + +set ipfix exporter collector 192.168.6.2 src 192.168.6.1 +template-interval 20 port 4739 path-mtu 1500 + +flowprobe params record l3 active 20 passive 120 flowprobe feature +add-del GigabitEthernet2/3/0 l2 diff --git a/src/plugins/ioam/ioam_plugin_doc.md b/src/plugins/ioam/ioam_plugin_doc.md deleted file mode 100644 index 343abcf73d8..00000000000 --- a/src/plugins/ioam/ioam_plugin_doc.md +++ /dev/null @@ -1,464 +0,0 @@ -## VPP Inband OAM (iOAM) {#ioam_plugin_doc} - -In-band OAM (iOAM) is an implementation study to record operational -information in the packet while the packet traverses a path between -two points in the network. - -Overview of iOAM can be found in [iOAM-Devnet] page. -The following IETF drafts detail the motivation and mechanism for -recording operational information: - - [iOAM-ietf-requirements] - Describes motivation and usecases for iOAM - - [iOAM-ietf-data] - Describes data records that can be collected using iOAM - - [iOAM-ietf-transport] - Lists out the transport protocols - and mechanism to carry iOAM data records - - [iOAM-ietf-proof-of-transit] - Describes the idea of Proof of Transit (POT) - and mechanisms to operationalize the idea - -## Terminology -In-band OAM is expected to be deployed in a specific domain rather -than on the overall Internet. The part of the network which employs in-band OAM -is referred to as **"in-band OAM-domain"**. - -In-band OAM data is added to a packet on entering the in-band OAM-domain -and is removed from the packet when exiting the domain. -Within the in-band OAM-domain, network nodes that the packet traverses -may update the in-band OAM data records. - -- The node which adds in-band OAM data to the packet is called the -**"in-band OAM encapsulating node"**. - -- The node which removes the in-band OAM data is referred to as the -**"in-band OAM decapsulating node"**. - -- Nodes within the domain which are aware of in-band OAM data and read -and/or write or process the in-band OAM data are called -**"in-band OAM transit nodes"**. - -## Features supported in the current release -VPP can function as in-band OAM encapsulating, transit and decapsulating node. -In this version of VPP in-band OAM data is transported as options in an -IPv6 hop-by-hop extension header. Hence in-band OAM can be enabled -for IPv6 traffic. - -The following iOAM features are supported: - -- **In-band OAM Tracing** : In-band OAM supports multiple data records to be -recorded in the packet as the packet traverses the network. -These data records offer insights into the operational behavior of the network. -The following information can be collected in the tracing -data from the nodes a packet traverses: - - Node ID - - Ingress interface ID - - Egress interface ID - - Timestamp - - Pre-configured application data - -- **In-band OAM Proof of Transit (POT)**: Proof of transit iOAM data is -added to every packet for verifying that a packet traverses a specific -set of nodes. -In-band OAM data is updated at every node that is enabled with iOAM -proof of transit and is used to verify whether a packet traversed -all the specified nodes. When the verifier receives each packet, -it can validate whether the packet traversed the specified nodes. - - -## Configuration -Configuring iOAM involves: -- Selecting the packets for which iOAM data must be inserted, updated or removed - - Selection of packets for iOAM data insertion on iOAM encapsulating node. - Selection of packets is done by 5-tuple based classification - - Selection of packets for updating iOAM data is implicitly done on the - presence of iOAM options in the packet - - Selection of packets for removing the iOAM data is done on 5-tuple - based classification -- The kind of data to be collected - - Tracing data - - Proof of transit -- Additional details for processing iOAM data to be collected - - For trace data - trace type, number of nodes to be recorded in the trace, - time stamp precision, etc. - - For POT data - configuration of POT profile required to process the POT data - -The CLI for configuring iOAM is explained here followed by detailed steps -and examples to deploy iOAM on VPP as an encapsulating, transit or -decapsulating iOAM node in the subsequent sub-sections. - -VPP iOAM configuration for enabling trace and POT is as follows: - - set ioam rewrite trace-type <0x1f|0x7|0x9|0x11|0x19> - trace-elts <number of trace elements> trace-tsp <0|1|2|3> - node-id <node ID in hex> app-data <application data in hex> [pot] - -A description of each of the options of the CLI follows: -- trace-type : An entry in the "Node data List" array of the trace option -can have different formats, following the needs of the a deployment. -For example: Some deployments might only be interested -in recording the node identifiers, whereas others might be interested -in recording node identifier and timestamp. -The following types are currently supported: - - 0x1f : Node data to include hop limit (8 bits), node ID (24 bits), - ingress and egress interface IDs (16 bits each), timestamp (32 bits), - application data (32 bits) - - 0x7 : Node data to include hop limit (8 bits), node ID (24 bits), - ingress and egress interface IDs (16 bits each) - - 0x9 : Node data to include hop limit (8 bits), node ID (24 bits), - timestamp (32 bits) - - 0x11: Node data to include hop limit (8 bits), node ID (24 bits), - application data (32 bits) - - 0x19: Node data to include hop limit (8 bits), node ID (24 bits), - timestamp (32 bits), application data (32 bits) -- trace-elts : Defines the length of the node data array in the trace option. -- trace-tsp : Defines the timestamp precision to use with the enumerated value - for precision as follows: - - 0 : 32bits timestamp in seconds - - 1 : 32bits timestamp in milliseconds - - 2 : 32bits timestamp in microseconds - - 3 : 32bits timestamp in nanoseconds -- node-id : Unique identifier for the node, included in the node ID - field of the node data in trace option. -- app-data : The value configured here is included as is in -application data field of node data in trace option. -- pot : Enables POT option to be included in the iOAM options. - -### Trace configuration - -#### On in-band OAM encapsulating node - - **Configure classifier and apply ACL** to select packets for - iOAM data insertion - - Example to enable iOAM data insertion for all the packets - towards IPv6 address db06::06: - - vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst - - vpp# classify session acl-hit-next node ip6-add-hop-by-hop - table-index 0 match l3 ip6 dst db06::06 - - vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 - - - **Enable tracing** : Specify node ID, maximum number of nodes for which - trace data should be recorded, type of data to be included for recording, - optionally application data to be included - - Example to enable tracing with a maximum of 4 nodes recorded - and the data to be recorded to include - hop limit, node id, - ingress and egress interface IDs, timestamp (millisecond precision), - application data (0x1234): - - - vpp# set ioam rewrite trace-type 0x1f trace-elts 4 trace-tsp 1 - node-id 0x1 app-data 0x1234 - - - -#### On in-band OAM transit node -- The transit node requires trace type, timestamp precision, node ID and -optionally application data to be configured, -to update its node data in the trace option. - -Example: - - vpp# set ioam rewrite trace-type 0x1f trace-elts 4 trace-tsp 1 - node-id 0x2 app-data 0x1234 - -#### On the In-band OAM decapsulating node -- The decapsulating node similar to encapsulating node requires -**classification** of the packets to remove iOAM data from. - - Example to decapsulate iOAM data for packets towards - db06::06, configure classifier and enable it as an ACL as follows: - - - vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst - - vpp# classify session acl-hit-next node ip6-lookup table-index 0 - match l3 ip6 dst db06::06 opaque-index 100 - - vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 - - -- Decapsulating node requires trace type, timestamp precision, -node ID and optionally application data to be configured, -to update its node data in the trace option before it is decapsulated. - -Example: - - vpp# set ioam rewrite trace-type 0x1f trace-elts 4 - trace-tsp 1 node-id 0x3 app-data 0x1234 - - -### Proof of Transit configuration - -For details on proof-of-transit, -see the IETF draft [iOAM-ietf-proof-of-transit]. -To enable Proof of Transit all the nodes that participate -and hence are verified for transit need a proof of transit profile. -A script to generate a proof of transit profile as per the mechanism -described in [iOAM-ietf-proof-of-transit] will be available at [iOAM-Devnet]. - -The Proof of transit mechanism implemented here is based on -Shamir's Secret Sharing algorithm. -The overall algorithm uses two polynomials -POLY-1 and POLY-2. The degree of polynomials depends on number of nodes -to be verified for transit. -POLY-1 is secret and constant. Each node gets a point on POLY-1 -at setup-time and keeps it secret. -POLY-2 is public, random and per packet. -Each node is assigned a point on POLY-1 and POLY-2 with the same x index. -Each node derives its point on POLY-2 each time a packet arrives at it. -A node then contributes its points on POLY-1 and POLY-2 to construct -POLY-3 (POLY-3 = POLY-1 + POLY-2) using lagrange extrapolation and -forwards it towards the verifier by updating POT data in the packet. -The verifier constructs POLY-3 from the accumulated value from all the nodes -and its own points on POLY-1 and POLY-2 and verifies whether -POLY-3 = POLY-1 + POLY-2. Only the verifier knows POLY-1. -The solution leverages finite field arithmetic in a field of size "prime number" -for reasons explained in description of Shamir's secret sharing algorithm. - -Here is an explanation of POT profile list and profile configuration CLI to -realize the above mechanism. -It is best to use the script provided at [iOAM-Devnet] to generate -this configuration. -- **Create POT profile** : set pot profile name <string> id [0-1] -[validator-key 0xu64] prime-number 0xu64 secret_share 0xu64 -lpc 0xu64 polynomial2 0xu64 bits-in-random [0-64] - - name : Profile list name. - - id : Profile id, it can be 0 or 1. - A maximum of two profiles can be configured per profile list. - - validator-key : Secret key configured only on the - verifier/decapsulating node used to compare and verify proof of transit. - - prime-number : Prime number for finite field arithmetic as required by the - proof of transit mechanism. - - secret_share : Unique point for each node on the secret polynomial POLY-1. - - lpc : Lagrange Polynomial Constant(LPC) calculated per node based on - its point (x value used for evaluating the points on the polynomial) - on the polynomial used in lagrange extrapolation - for reconstructing polynomial (POLY-3). - - polynomial2 : Is the pre-evaluated value of the point on - 2nd polynomial(POLY-2). This is unique for each node. - It is pre-evaluated for all the coefficients of POLY-2 except - for the constant part of the polynomial that changes per packet - and is received as part of the POT data in the packet. - - bits-in-random : To control the size of the random number to be - generated. This number has to match the other numbers generated and used - in the profile as per the algorithm. - -- **Set a configured profile as active/in-use** : -set pot profile-active name <string> ID [0-1] - - name : Name of the profile list to be used for computing - POT data per packet. - - ID : Identifier of the profile within the list to be used. - -#### On In-band OAM encapsulating node - - Configure the classifier and apply ACL to select packets for iOAM data insertion. - - Example to enable iOAM data insertion for all the packet towards - IPv6 address db06::06 - - - - vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst - - vpp# classify session acl-hit-next node - ip6-add-hop-by-hop table-index 0 match l3 ip6 dst db06::06 - - vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 - - - - Configure the proof of transit profile list with profiles. -Each profile list referred to by a name can contain 2 profiles, -only one is in use for updating proof of transit data at any time. - - Example profile list example with a profile generated from the - script to verify transit through 3 nodes is: - - - vpp# set pot profile name example id 0 prime-number 0x7fff0000fa884685 - secret_share 0x6c22eff0f45ec56d lpc 0x7fff0000fa884682 - polynomial2 0xffb543d4a9c bits-in-random 63 - - - Enable one of the profiles from the configured profile list as active - so that is will be used for calculating proof of transit - -Example enable profile ID 0 from profile list example configured above: - - - vpp# set pot profile-active name example ID 0 - - - - Enable POT option to be inserted - - - vpp# set ioam rewrite pot - - -#### On in-band OAM transit node - - Configure the proof of transit profile list with profiles for transit node. -Example: - - - vpp# set pot profile name example id 0 prime-number 0x7fff0000fa884685 - secret_share 0x564cdbdec4eb625d lpc 0x1 - polynomial2 0x23f3a227186a bits-in-random 63 - -#### On in-band OAM decapsulating node / verifier -- The decapsulating node, similar to the encapsulating node requires -classification of the packets to remove iOAM data from. - - Example to decapsulate iOAM data for packets towards db06::06 - configure classifier and enable it as an ACL as follows: - - - vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst - - vpp# classify session acl-hit-next node ip6-lookup table-index 0 - match l3 ip6 dst db06::06 opaque-index 100 - - vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 - -- To update and verify the proof of transit, POT profile list should be configured. - - Example POT profile list configured as follows: - - vpp# set pot profile name example id 0 validate-key 0x7fff0000fa88465d - prime-number 0x7fff0000fa884685 secret_share 0x7a08fbfc5b93116d lpc 0x3 - polynomial2 0x3ff738597ce bits-in-random 63 - -## Operational data - -Following CLIs are available to check iOAM operation: -- To check iOAM configuration that are effective use "show ioam summary" - -Example: - - vpp# show ioam summary - REWRITE FLOW CONFIGS - Not configured - HOP BY HOP OPTIONS - TRACE CONFIG - - Trace Type : 0x1f (31) - Trace timestamp precision : 1 (Milliseconds) - Num of trace nodes : 4 - Node-id : 0x2 (2) - App Data : 0x1234 (4660) - POT OPTION - 1 (Enabled) - Try 'show ioam pot and show pot profile' for more information - -- To find statistics about packets for which iOAM options were -added (encapsulating node) and removed (decapsulating node) execute -*show errors* - -Example on encapsulating node: - - - vpp# show error - Count Node Reason - 1208804706 ip6-inacl input ACL hits - 1208804706 ip6-add-hop-by-hop Pkts w/ added ip6 hop-by-hop options - -Example on decapsulating node: - - vpp# show error - Count Node Reason - 69508569 ip6-inacl input ACL hits - 69508569 ip6-pop-hop-by-hop Pkts w/ removed ip6 hop-by-hop options - -- To check the POT profiles use "show pot profile" - -Example: - - vpp# show pot profile - Profile list in use : example - POT Profile at index: 0 - ID : 0 - Validator : False (0) - Secret share : 0x564cdbdec4eb625d (6218586935324795485) - Prime number : 0x7fff0000fa884685 (9223090566081300101) - 2nd polynomial(eval) : 0x23f3a227186a (39529304496234) - LPC : 0x1 (1) - Bit mask : 0x7fffffffffffffff (9223372036854775807) - Profile index in use: 0 - Pkts passed : 0x36 (54) - -- To get statistics of POT for packets use "show ioam pot" - -Example at encapsulating or transit node: - - vpp# show ioam pot - Pkts with ip6 hop-by-hop POT options - 54 - Pkts with ip6 hop-by-hop POT options but no profile set - 0 - Pkts with POT in Policy - 0 - Pkts with POT out of Policy - 0 - - -Example at decapsulating/verification node: - - - vpp# show ioam pot - Pkts with ip6 hop-by-hop POT options - 54 - Pkts with ip6 hop-by-hop POT options but no profile set - 0 - Pkts with POT in Policy - 54 - Pkts with POT out of Policy - 0 - -- Tracing - enable trace of IPv6 packets to view the data inserted and -collected. - -Example when the nodes are receiving data over a DPDK interface: -Enable tracing using "trace add dpdk-input 20" and -execute "show trace" to view the iOAM data collected: - - - vpp# trace add dpdk-input 20 - - vpp# show trace - - ------------------- Start of thread 0 vpp_main ------------------- - - Packet 1 - - 00:00:19:294697: dpdk-input - GigabitEthernetb/0/0 rx queue 0 - buffer 0x10e6b: current data 0, length 214, free-list 0, totlen-nifb 0, trace 0x0 - PKT MBUF: port 0, nb_segs 1, pkt_len 214 - buf_len 2176, data_len 214, ol_flags 0x0, data_off 128, phys_addr 0xe9a35a00 - packet_type 0x0 - IP6: 00:50:56:9c:df:72 -> 00:50:56:9c:be:55 - IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 - tos 0x00, flow label 0x0, hop limit 63, payload length 160 - 00:00:19:294737: ethernet-input - IP6: 00:50:56:9c:df:72 -> 00:50:56:9c:be:55 - 00:00:19:294753: ip6-input - IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 - tos 0x00, flow label 0x0, hop limit 63, payload length 160 - 00:00:19:294757: ip6-lookup - fib 0 adj-idx 15 : indirect via db05::2 flow hash: 0x00000000 - IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 - tos 0x00, flow label 0x0, hop limit 63, payload length 160 - 00:00:19:294802: ip6-hop-by-hop - IP6_HOP_BY_HOP: next index 5 len 96 traced 96 Trace Type 0x1f , 1 elts left - [0] ttl 0x0 node ID 0x0 ingress 0x0 egress 0x0 ts 0x0 - app 0x0 - [1] ttl 0x3e node ID 0x3 ingress 0x1 egress 0x2 ts 0xb68c2213 - app 0x1234 - [2] ttl 0x3f node ID 0x2 ingress 0x1 egress 0x2 ts 0xb68c2204 - app 0x1234 - [3] ttl 0x40 node ID 0x1 ingress 0x5 egress 0x6 ts 0xb68c2200 - app 0x1234 - POT opt present - random = 0x577a916946071950, Cumulative = 0x10b46e78a35a392d, Index = 0x0 - 00:00:19:294810: ip6-rewrite - tx_sw_if_index 1 adj-idx 14 : GigabitEthernetb/0/0 - IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 flow hash: 0x00000000 - IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 - IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 - tos 0x00, flow label 0x0, hop limit 62, payload length 160 - 00:00:19:294814: GigabitEthernetb/0/0-output - GigabitEthernetb/0/0 - IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 - IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 - tos 0x00, flow label 0x0, hop limit 62, payload length 160 - 00:00:19:294820: GigabitEthernetb/0/0-tx - GigabitEthernetb/0/0 tx queue 0 - buffer 0x10e6b: current data 0, length 214, free-list 0, totlen-nifb 0, trace 0x0 - IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 - - IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 - - tos 0x00, flow label 0x0, hop limit 62, payload length 160 - - -[iOAM-Devnet]: <https://github.com/ciscodevnet/iOAM> -[iOAM-ietf-requirements]:<https://tools.ietf.org/html/draft-brockners-inband-oam-requirements-01> -[iOAM-ietf-transport]:<https://tools.ietf.org/html/draft-brockners-inband-oam-transport-01> -[iOAM-ietf-data]:<https://tools.ietf.org/html/draft-brockners-inband-oam-data-01> -[iOAM-ietf-proof-of-transit]:<https://tools.ietf.org/html/draft-brockners-proof-of-transit-01> diff --git a/src/plugins/ioam/ioam_plugin_doc.rst b/src/plugins/ioam/ioam_plugin_doc.rst new file mode 100644 index 00000000000..0f84d5f7a36 --- /dev/null +++ b/src/plugins/ioam/ioam_plugin_doc.rst @@ -0,0 +1,490 @@ +Inband OAM (iOAM) +================= + +In-band OAM (iOAM) is an implementation study to record operational +information in the packet while the packet traverses a path between two +points in the network. + +Overview of iOAM can be found in +`iOAM-Devnet <https://github.com/ciscodevnet/iOAM>`__ page. The +following IETF drafts detail the motivation and mechanism for recording +operational information: - +`iOAM-ietf-requirements <https://tools.ietf.org/html/draft-brockners-inband-oam-requirements-01>`__ +- Describes motivation and usecases for iOAM - +`iOAM-ietf-data <https://tools.ietf.org/html/draft-brockners-inband-oam-data-01>`__ +- Describes data records that can be collected using iOAM - +`iOAM-ietf-transport <https://tools.ietf.org/html/draft-brockners-inband-oam-transport-01>`__ +- Lists out the transport protocols and mechanism to carry iOAM data +records - +`iOAM-ietf-proof-of-transit <https://tools.ietf.org/html/draft-brockners-proof-of-transit-01>`__ +- Describes the idea of Proof of Transit (POT) and mechanisms to +operationalize the idea + +Terminology +----------- + +In-band OAM is expected to be deployed in a specific domain rather than +on the overall Internet. The part of the network which employs in-band +OAM is referred to as **“in-band OAM-domain”**. + +In-band OAM data is added to a packet on entering the in-band OAM-domain +and is removed from the packet when exiting the domain. Within the +in-band OAM-domain, network nodes that the packet traverses may update +the in-band OAM data records. + +- The node which adds in-band OAM data to the packet is called the + **“in-band OAM encapsulating node”**. + +- The node which removes the in-band OAM data is referred to as the + **“in-band OAM decapsulating node”**. + +- Nodes within the domain which are aware of in-band OAM data and read + and/or write or process the in-band OAM data are called **“in-band + OAM transit nodes”**. + +Features supported in the current release +----------------------------------------- + +VPP can function as in-band OAM encapsulating, transit and decapsulating +node. In this version of VPP in-band OAM data is transported as options +in an IPv6 hop-by-hop extension header. Hence in-band OAM can be enabled +for IPv6 traffic. + +The following iOAM features are supported: + +- **In-band OAM Tracing** : In-band OAM supports multiple data records + to be recorded in the packet as the packet traverses the network. + These data records offer insights into the operational behavior of + the network. The following information can be collected in the + tracing data from the nodes a packet traverses: + + - Node ID + - Ingress interface ID + - Egress interface ID + - Timestamp + - Pre-configured application data + +- **In-band OAM Proof of Transit (POT)**: Proof of transit iOAM data is + added to every packet for verifying that a packet traverses a + specific set of nodes. In-band OAM data is updated at every node that + is enabled with iOAM proof of transit and is used to verify whether a + packet traversed all the specified nodes. When the verifier receives + each packet, it can validate whether the packet traversed the + specified nodes. + +Configuration +------------- + +Configuring iOAM involves: - Selecting the packets for which iOAM data +must be inserted, updated or removed - Selection of packets for iOAM +data insertion on iOAM encapsulating node. Selection of packets is done +by 5-tuple based classification - Selection of packets for updating iOAM +data is implicitly done on the presence of iOAM options in the packet - +Selection of packets for removing the iOAM data is done on 5-tuple based +classification - The kind of data to be collected - Tracing data - Proof +of transit - Additional details for processing iOAM data to be collected +- For trace data - trace type, number of nodes to be recorded in the +trace, time stamp precision, etc. - For POT data - configuration of POT +profile required to process the POT data + +The CLI for configuring iOAM is explained here followed by detailed +steps and examples to deploy iOAM on VPP as an encapsulating, transit or +decapsulating iOAM node in the subsequent sub-sections. + +VPP iOAM configuration for enabling trace and POT is as follows: + +:: + + set ioam rewrite trace-type <0x1f|0x7|0x9|0x11|0x19> + trace-elts <number of trace elements> trace-tsp <0|1|2|3> + node-id <node ID in hex> app-data <application data in hex> [pot] + +A description of each of the options of the CLI follows: - trace-type : +An entry in the “Node data List” array of the trace option can have +different formats, following the needs of the a deployment. For example: +Some deployments might only be interested in recording the node +identifiers, whereas others might be interested in recording node +identifier and timestamp. The following types are currently supported: - +0x1f : Node data to include hop limit (8 bits), node ID (24 bits), +ingress and egress interface IDs (16 bits each), timestamp (32 bits), +application data (32 bits) - 0x7 : Node data to include hop limit (8 +bits), node ID (24 bits), ingress and egress interface IDs (16 bits +each) - 0x9 : Node data to include hop limit (8 bits), node ID (24 +bits), timestamp (32 bits) - 0x11: Node data to include hop limit (8 +bits), node ID (24 bits), application data (32 bits) - 0x19: Node data +to include hop limit (8 bits), node ID (24 bits), timestamp (32 bits), +application data (32 bits) - trace-elts : Defines the length of the node +data array in the trace option. - trace-tsp : Defines the timestamp +precision to use with the enumerated value for precision as follows: - 0 +: 32bits timestamp in seconds - 1 : 32bits timestamp in milliseconds - 2 +: 32bits timestamp in microseconds - 3 : 32bits timestamp in nanoseconds +- node-id : Unique identifier for the node, included in the node ID +field of the node data in trace option. - app-data : The value +configured here is included as is in application data field of node data +in trace option. - pot : Enables POT option to be included in the iOAM +options. + +Trace configuration +~~~~~~~~~~~~~~~~~~~ + +On in-band OAM encapsulating node +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- **Configure classifier and apply ACL** to select packets for iOAM + data insertion + + - Example to enable iOAM data insertion for all the packets towards + IPv6 address db06::06: + + vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst + + vpp# classify session acl-hit-next node ip6-add-hop-by-hop + table-index 0 match l3 ip6 dst db06::06 + + vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 + +- **Enable tracing** : Specify node ID, maximum number of nodes for + which trace data should be recorded, type of data to be included for + recording, optionally application data to be included + + - Example to enable tracing with a maximum of 4 nodes recorded and + the data to be recorded to include - hop limit, node id, ingress + and egress interface IDs, timestamp (millisecond precision), + application data (0x1234): + + vpp# set ioam rewrite trace-type 0x1f trace-elts 4 trace-tsp 1 + node-id 0x1 app-data 0x1234 + +On in-band OAM transit node +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- The transit node requires trace type, timestamp precision, node ID + and optionally application data to be configured, to update its node + data in the trace option. + +Example: + +:: + + vpp# set ioam rewrite trace-type 0x1f trace-elts 4 trace-tsp 1 + node-id 0x2 app-data 0x1234 + +On the In-band OAM decapsulating node +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- The decapsulating node similar to encapsulating node requires + **classification** of the packets to remove iOAM data from. + + - Example to decapsulate iOAM data for packets towards db06::06, + configure classifier and enable it as an ACL as follows: + + vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst + + vpp# classify session acl-hit-next node ip6-lookup table-index 0 + match l3 ip6 dst db06::06 opaque-index 100 + + vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 + +- Decapsulating node requires trace type, timestamp precision, node ID + and optionally application data to be configured, to update its node + data in the trace option before it is decapsulated. + +Example: + +:: + + vpp# set ioam rewrite trace-type 0x1f trace-elts 4 + trace-tsp 1 node-id 0x3 app-data 0x1234 + +Proof of Transit configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For details on proof-of-transit, see the IETF draft +`iOAM-ietf-proof-of-transit <https://tools.ietf.org/html/draft-brockners-proof-of-transit-01>`__. +To enable Proof of Transit all the nodes that participate and hence are +verified for transit need a proof of transit profile. A script to +generate a proof of transit profile as per the mechanism described in +`iOAM-ietf-proof-of-transit <https://tools.ietf.org/html/draft-brockners-proof-of-transit-01>`__ +will be available at +`iOAM-Devnet <https://github.com/ciscodevnet/iOAM>`__. + +The Proof of transit mechanism implemented here is based on Shamir’s +Secret Sharing algorithm. The overall algorithm uses two polynomials +POLY-1 and POLY-2. The degree of polynomials depends on number of nodes +to be verified for transit. POLY-1 is secret and constant. Each node +gets a point on POLY-1 at setup-time and keeps it secret. POLY-2 is +public, random and per packet. Each node is assigned a point on POLY-1 +and POLY-2 with the same x index. Each node derives its point on POLY-2 +each time a packet arrives at it. A node then contributes its points on +POLY-1 and POLY-2 to construct POLY-3 (POLY-3 = POLY-1 + POLY-2) using +lagrange extrapolation and forwards it towards the verifier by updating +POT data in the packet. The verifier constructs POLY-3 from the +accumulated value from all the nodes and its own points on POLY-1 and +POLY-2 and verifies whether POLY-3 = POLY-1 + POLY-2. Only the verifier +knows POLY-1. The solution leverages finite field arithmetic in a field +of size “prime number” for reasons explained in description of Shamir’s +secret sharing algorithm. + +| Here is an explanation of POT profile list and profile configuration + CLI to realize the above mechanism. It is best to use the script + provided at `iOAM-Devnet <https://github.com/ciscodevnet/iOAM>`__ to + generate this configuration. - **Create POT profile** : set pot + profile name id [0-1] +| [validator-key 0xu64] prime-number 0xu64 secret_share 0xu64 +| lpc 0xu64 polynomial2 0xu64 bits-in-random [0-64] +| - name : Profile list name. - id : Profile id, it can be 0 or 1. A + maximum of two profiles can be configured per profile list. - + validator-key : Secret key configured only on the + verifier/decapsulating node used to compare and verify proof of + transit. - prime-number : Prime number for finite field arithmetic as + required by the proof of transit mechanism. - secret_share : Unique + point for each node on the secret polynomial POLY-1. - lpc : Lagrange + Polynomial Constant(LPC) calculated per node based on its point (x + value used for evaluating the points on the polynomial) on the + polynomial used in lagrange extrapolation for reconstructing + polynomial (POLY-3). - polynomial2 : Is the pre-evaluated value of the + point on 2nd polynomial(POLY-2). This is unique for each node. It is + pre-evaluated for all the coefficients of POLY-2 except for the + constant part of the polynomial that changes per packet and is + received as part of the POT data in the packet. - bits-in-random : To + control the size of the random number to be generated. This number has + to match the other numbers generated and used in the profile as per + the algorithm. + +- **Set a configured profile as active/in-use** : + set pot profile-active name ID [0-1] + + - name : Name of the profile list to be used for computing POT data + per packet. + - ID : Identifier of the profile within the list to be used. + +.. _on-in-band-oam-encapsulating-node-1: + +On In-band OAM encapsulating node +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Configure the classifier and apply ACL to select packets for iOAM + data insertion. + + - Example to enable iOAM data insertion for all the packet towards + IPv6 address db06::06 - + + vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst + + vpp# classify session acl-hit-next node ip6-add-hop-by-hop + table-index 0 match l3 ip6 dst db06::06 + + vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 + +- Configure the proof of transit profile list with profiles. Each + profile list referred to by a name can contain 2 profiles, only one + is in use for updating proof of transit data at any time. + + - Example profile list example with a profile generated from the + script to verify transit through 3 nodes is: + + vpp# set pot profile name example id 0 prime-number + 0x7fff0000fa884685 secret_share 0x6c22eff0f45ec56d lpc + 0x7fff0000fa884682 polynomial2 0xffb543d4a9c bits-in-random 63 + +- Enable one of the profiles from the configured profile list as active + so that is will be used for calculating proof of transit + +Example enable profile ID 0 from profile list example configured above: + +:: + + vpp# set pot profile-active name example ID 0 + +- Enable POT option to be inserted + + vpp# set ioam rewrite pot + +.. _on-in-band-oam-transit-node-1: + +On in-band OAM transit node +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Configure the proof of transit profile list with profiles for transit + node. Example: + + vpp# set pot profile name example id 0 prime-number + 0x7fff0000fa884685 secret_share 0x564cdbdec4eb625d lpc 0x1 + polynomial2 0x23f3a227186a bits-in-random 63 + +On in-band OAM decapsulating node / verifier +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- The decapsulating node, similar to the encapsulating node requires + classification of the packets to remove iOAM data from. + + - Example to decapsulate iOAM data for packets towards db06::06 + configure classifier and enable it as an ACL as follows: + + vpp# classify table miss-next node ip6-lookup mask l3 ip6 dst + + vpp# classify session acl-hit-next node ip6-lookup table-index 0 + match l3 ip6 dst db06::06 opaque-index 100 + + vpp# set int input acl intfc GigabitEthernet0/0/0 ip6-table 0 + +- To update and verify the proof of transit, POT profile list should be + configured. + + - Example POT profile list configured as follows: + + vpp# set pot profile name example id 0 validate-key + 0x7fff0000fa88465d prime-number 0x7fff0000fa884685 secret_share + 0x7a08fbfc5b93116d lpc 0x3 polynomial2 0x3ff738597ce bits-in-random + 63 + +Operational data +---------------- + +Following CLIs are available to check iOAM operation: - To check iOAM +configuration that are effective use “show ioam summary” + +Example: + +:: + + vpp# show ioam summary + REWRITE FLOW CONFIGS - Not configured + HOP BY HOP OPTIONS - TRACE CONFIG - + Trace Type : 0x1f (31) + Trace timestamp precision : 1 (Milliseconds) + Num of trace nodes : 4 + Node-id : 0x2 (2) + App Data : 0x1234 (4660) + POT OPTION - 1 (Enabled) + Try 'show ioam pot and show pot profile' for more information + +- To find statistics about packets for which iOAM options were added + (encapsulating node) and removed (decapsulating node) execute *show + errors* + +Example on encapsulating node: + +:: + + vpp# show error + Count Node Reason + 1208804706 ip6-inacl input ACL hits + 1208804706 ip6-add-hop-by-hop Pkts w/ added ip6 hop-by-hop options + +Example on decapsulating node: + +:: + + vpp# show error + Count Node Reason + 69508569 ip6-inacl input ACL hits + 69508569 ip6-pop-hop-by-hop Pkts w/ removed ip6 hop-by-hop options + +- To check the POT profiles use “show pot profile” + +Example: + +:: + + vpp# show pot profile + Profile list in use : example + POT Profile at index: 0 + ID : 0 + Validator : False (0) + Secret share : 0x564cdbdec4eb625d (6218586935324795485) + Prime number : 0x7fff0000fa884685 (9223090566081300101) + 2nd polynomial(eval) : 0x23f3a227186a (39529304496234) + LPC : 0x1 (1) + Bit mask : 0x7fffffffffffffff (9223372036854775807) + Profile index in use: 0 + Pkts passed : 0x36 (54) + +- To get statistics of POT for packets use “show ioam pot” + +Example at encapsulating or transit node: + +:: + + vpp# show ioam pot + Pkts with ip6 hop-by-hop POT options - 54 + Pkts with ip6 hop-by-hop POT options but no profile set - 0 + Pkts with POT in Policy - 0 + Pkts with POT out of Policy - 0 + +Example at decapsulating/verification node: + +:: + + vpp# show ioam pot + Pkts with ip6 hop-by-hop POT options - 54 + Pkts with ip6 hop-by-hop POT options but no profile set - 0 + Pkts with POT in Policy - 54 + Pkts with POT out of Policy - 0 + +- Tracing - enable trace of IPv6 packets to view the data inserted and + collected. + +Example when the nodes are receiving data over a DPDK interface: Enable +tracing using “trace add dpdk-input 20” and execute “show trace” to view +the iOAM data collected: + +:: + + vpp# trace add dpdk-input 20 + + vpp# show trace + + ------------------- Start of thread 0 vpp_main ------------------- + + Packet 1 + + 00:00:19:294697: dpdk-input + GigabitEthernetb/0/0 rx queue 0 + buffer 0x10e6b: current data 0, length 214, free-list 0, totlen-nifb 0, trace 0x0 + PKT MBUF: port 0, nb_segs 1, pkt_len 214 + buf_len 2176, data_len 214, ol_flags 0x0, data_off 128, phys_addr 0xe9a35a00 + packet_type 0x0 + IP6: 00:50:56:9c:df:72 -> 00:50:56:9c:be:55 + IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 + tos 0x00, flow label 0x0, hop limit 63, payload length 160 + 00:00:19:294737: ethernet-input + IP6: 00:50:56:9c:df:72 -> 00:50:56:9c:be:55 + 00:00:19:294753: ip6-input + IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 + tos 0x00, flow label 0x0, hop limit 63, payload length 160 + 00:00:19:294757: ip6-lookup + fib 0 adj-idx 15 : indirect via db05::2 flow hash: 0x00000000 + IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 + tos 0x00, flow label 0x0, hop limit 63, payload length 160 + 00:00:19:294802: ip6-hop-by-hop + IP6_HOP_BY_HOP: next index 5 len 96 traced 96 Trace Type 0x1f , 1 elts left + [0] ttl 0x0 node ID 0x0 ingress 0x0 egress 0x0 ts 0x0 + app 0x0 + [1] ttl 0x3e node ID 0x3 ingress 0x1 egress 0x2 ts 0xb68c2213 + app 0x1234 + [2] ttl 0x3f node ID 0x2 ingress 0x1 egress 0x2 ts 0xb68c2204 + app 0x1234 + [3] ttl 0x40 node ID 0x1 ingress 0x5 egress 0x6 ts 0xb68c2200 + app 0x1234 + POT opt present + random = 0x577a916946071950, Cumulative = 0x10b46e78a35a392d, Index = 0x0 + 00:00:19:294810: ip6-rewrite + tx_sw_if_index 1 adj-idx 14 : GigabitEthernetb/0/0 + IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 flow hash: 0x00000000 + IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 + IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 + tos 0x00, flow label 0x0, hop limit 62, payload length 160 + 00:00:19:294814: GigabitEthernetb/0/0-output + GigabitEthernetb/0/0 + IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 + IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 + tos 0x00, flow label 0x0, hop limit 62, payload length 160 + 00:00:19:294820: GigabitEthernetb/0/0-tx + GigabitEthernetb/0/0 tx queue 0 + buffer 0x10e6b: current data 0, length 214, free-list 0, totlen-nifb 0, trace 0x0 + IP6: 00:50:56:9c:be:55 -> 00:50:56:9c:df:72 + + IP6_HOP_BY_HOP_OPTIONS: db05::2 -> db06::6 + + tos 0x00, flow label 0x0, hop limit 62, payload length 160 diff --git a/src/plugins/lacp/lacp_doc.md b/src/plugins/lacp/lacp_doc.md deleted file mode 100644 index 7df82b5689a..00000000000 --- a/src/plugins/lacp/lacp_doc.md +++ /dev/null @@ -1,104 +0,0 @@ -# VPP Link Aggregation Control Protocol (LACP) implementation {#lacp_plugin_doc} - -This document is to describe the usage of VPP LACP implementation. - -## LACP - -The Link Aggregation Control Protocol (LACP) is an 802.3ad standard which -provides a protocol for exchanging information between Partner Systems on a -link to allow their protocol instances to reach agreement on the Link Aggregation -Group to which the link belongs and enable transmission and reception for the -higher layer. Multiple links may be bundled to the same Aggregation Group to form -a high bandwidth transmission medium and create a fault-tolerant link. - - -### Configuration - -1. Create the bond interface -create bond mode lacp [hw-addr <mac-address>] [load-balance { l2 | l23 | l34 } [numa-only]] - -2. Enslave the physical interface to the bond -bond add <bond-interface-name> <slave-interface> [passive] [long-timeout]" - -3. Delete the bond interface -delete bond {<interface> | sw_if_index <sw_idx>} - -4. Detach the slave interface from the bond -bond del <slave-interface> - -### Configuration example - -``` -create bond mode lacp -set interface state BondEthernet0 up -bond add BondEthernet0 TenGigabitEthernet7/0/0 -bond add BondEthernet0 TenGigabitEthernet7/0/1 -bond add BondEthernet0 TenGigabitEthernet5/0/0 -bond add BondEthernet0 TenGigabitEthernet5/0/1 -``` - -``` -bond del TenGigabitEthernet5/0/1 -``` - -``` -delete bond BondEthernet0 -``` - -### Operational data - -``` -show lacp [<interface>] [details] -``` - -Example: - -``` -DBGvpp# show lacp - actor state partner state -interface name sw_if_index bond interface exp/def/dis/col/syn/agg/tim/act exp/def/dis/col/syn/agg/tim/act -GigabitEthernet2/0/1 1 BondEthernet0 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 - LAG ID: [(ffff,e4-c7-22-f3-26-71,0000,00ff,0001), (ffff,fc-99-47-4a-0c-8b,0009,00ff,0001)] - RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX -TenGigabitEthernet4/0/0 2 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 - LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0001), (8000,00-2a-6a-e5-50-c1,0140,8000,011d)] - RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX -TenGigabitEthernet4/0/1 3 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 - LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0002), (8000,00-2a-6a-e5-50-c1,0140,8000,011e)] - RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX -TenGigabitEthernet8/0/1 7 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 - LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0003), (8000,00-2a-6a-e5-50-01,007a,8000,0114)] - RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX -TenGigabitEthernet8/0/0 6 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 - LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0004), (8000,00-2a-6a-e5-50-01,007a,8000,0115)] - RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX -TenGigabitEthernet6/0/1 5 BondEthernet2 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 - LAG ID: [(ffff,90-e2-ba-36-31-21,0002,00ff,0001), (ffff,90-e2-ba-29-f5-31,000f,00ff,0002)] - RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX -TenGigabitEthernet6/0/0 4 BondEthernet2 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 - LAG ID: [(ffff,90-e2-ba-36-31-21,0002,00ff,0002), (ffff,90-e2-ba-29-f5-31,000f,00ff,0001)] - RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX -DBGvpp# -``` - -``` -show bond [details] -```` - -Example: - -``` -DBGvpp# show bond -sh bond -interface name sw_if_index mode load balance active slaves slaves -BondEthernet0 10 lacp l2 1 1 -BondEthernet1 11 lacp l34 4 4 -BondEthernet2 12 lacp l23 2 2 -DBGvpp# -``` - -### Debugging - -``` -debug lacp [<interface>] <on | off> -``` diff --git a/src/plugins/lacp/lacp_doc.rst b/src/plugins/lacp/lacp_doc.rst new file mode 100644 index 00000000000..04b51ba22f8 --- /dev/null +++ b/src/plugins/lacp/lacp_doc.rst @@ -0,0 +1,109 @@ +LACP Protocol +============= + +This document is to describe the usage of VPP Link Aggregation Control +Protocol (LACP) implementation. + +LACP +---- + +The Link Aggregation Control Protocol (LACP) is an 802.3ad standard +which provides a protocol for exchanging information between Partner +Systems on a link to allow their protocol instances to reach agreement +on the Link Aggregation Group to which the link belongs and enable +transmission and reception for the higher layer. Multiple links may be +bundled to the same Aggregation Group to form a high bandwidth +transmission medium and create a fault-tolerant link. + +Configuration +~~~~~~~~~~~~~ + +1. Create the bond interface create bond mode lacp [hw-addr ] + [load-balance { l2 \| l23 \| l34 } [numa-only]] + +2. Enslave the physical interface to the bond bond add [passive] + [long-timeout]” + +3. Delete the bond interface delete bond { \| sw_if_index } + +4. Detach the slave interface from the bond bond del + +Configuration example +~~~~~~~~~~~~~~~~~~~~~ + +:: + + create bond mode lacp + set interface state BondEthernet0 up + bond add BondEthernet0 TenGigabitEthernet7/0/0 + bond add BondEthernet0 TenGigabitEthernet7/0/1 + bond add BondEthernet0 TenGigabitEthernet5/0/0 + bond add BondEthernet0 TenGigabitEthernet5/0/1 + +:: + + bond del TenGigabitEthernet5/0/1 + +:: + + delete bond BondEthernet0 + +Operational data +~~~~~~~~~~~~~~~~ + +:: + + show lacp [<interface>] [details] + +Example: + +:: + + DBGvpp# show lacp + actor state partner state + interface name sw_if_index bond interface exp/def/dis/col/syn/agg/tim/act exp/def/dis/col/syn/agg/tim/act + GigabitEthernet2/0/1 1 BondEthernet0 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 + LAG ID: [(ffff,e4-c7-22-f3-26-71,0000,00ff,0001), (ffff,fc-99-47-4a-0c-8b,0009,00ff,0001)] + RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX + TenGigabitEthernet4/0/0 2 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 + LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0001), (8000,00-2a-6a-e5-50-c1,0140,8000,011d)] + RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX + TenGigabitEthernet4/0/1 3 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 + LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0002), (8000,00-2a-6a-e5-50-c1,0140,8000,011e)] + RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX + TenGigabitEthernet8/0/1 7 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 + LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0003), (8000,00-2a-6a-e5-50-01,007a,8000,0114)] + RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX + TenGigabitEthernet8/0/0 6 BondEthernet1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 1 + LAG ID: [(ffff,90-e2-ba-76-cf-2d,0001,00ff,0004), (8000,00-2a-6a-e5-50-01,007a,8000,0115)] + RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX + TenGigabitEthernet6/0/1 5 BondEthernet2 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 + LAG ID: [(ffff,90-e2-ba-36-31-21,0002,00ff,0001), (ffff,90-e2-ba-29-f5-31,000f,00ff,0002)] + RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX + TenGigabitEthernet6/0/0 4 BondEthernet2 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 + LAG ID: [(ffff,90-e2-ba-36-31-21,0002,00ff,0002), (ffff,90-e2-ba-29-f5-31,000f,00ff,0001)] + RX-state: CURRENT, TX-state: TRANSMIT, MUX-state: COLLECTING_DISTRIBUTING, PTX-state: PERIODIC_TX + DBGvpp# + +:: + + show bond [details] + +Example: + +:: + + DBGvpp# show bond + sh bond + interface name sw_if_index mode load balance active slaves slaves + BondEthernet0 10 lacp l2 1 1 + BondEthernet1 11 lacp l34 4 4 + BondEthernet2 12 lacp l23 2 2 + DBGvpp# + +Debugging +~~~~~~~~~ + +:: + + debug lacp [<interface>] <on | off> diff --git a/src/plugins/lb/lb_plugin_doc.md b/src/plugins/lb/lb_plugin_doc.md deleted file mode 100644 index 5f6538974e9..00000000000 --- a/src/plugins/lb/lb_plugin_doc.md +++ /dev/null @@ -1,192 +0,0 @@ -# Load Balancer plugin for VPP {#lb_plugin_doc} - -## Version - -The load balancer plugin is currently in *beta* version. -Both CLIs and APIs are subject to *heavy* changes, -which also means feedback is really welcome regarding features, apis, etc... - -## Overview - -This plugin provides load balancing for VPP in a way that is largely inspired -from Google's MagLev: http://research.google.com/pubs/pub44824.html - -The load balancer is configured with a set of Virtual IPs (VIP, which can be -prefixes), and for each VIP, with a set of Application Server addresses (ASs). - -There are four encap types to steer traffic to different ASs: -1). IPv4+GRE ad IPv6+GRE encap types: -Traffic received for a given VIP (or VIP prefix) is tunneled using GRE towards -the different ASs in a way that (tries to) ensure that a given session will -always be tunneled to the same AS. - -2). IPv4+L3DSR encap types: -L3DSR is used to overcome Layer 2 limitations of Direct Server Return Load Balancing. -It maps VIP to DSCP bits, and reuse TOS bits to transfer DSCP bits -to server, and then server will get VIP from DSCP-to-VIP mapping. - -Both VIPs or ASs can be IPv4 or IPv6, but for a given VIP, all ASs must be using -the same encap. type (i.e. IPv4+GRE or IPv6+GRE or IPv4+L3DSR). -Meaning that for a given VIP, all AS addresses must be of the same family. - -3). IPv4/IPv6 + NAT4/NAT6 encap types: -This type provides kube-proxy data plane on user space, -which is used to replace linux kernel's kube-proxy based on iptables. - -Currently, load balancer plugin supports three service types: -a) Cluster IP plus Port: support any protocols, including TCP, UDP. -b) Node IP plus Node Port: currently only support UDP. -c) External Load Balancer. - -For Cluster IP plus Port case: -kube-proxy is configured with a set of Virtual IPs (VIP, which can be -prefixes), and for each VIP, with a set of AS addresses (ASs). - -For a specific session received for a given VIP (or VIP prefix), -first packet selects a AS according to internal load balancing algorithm, -then does DNAT operation and sent to chosen AS. -At the same time, will create a session entry to store AS chosen result. -Following packets for that session will look up session table first, -which ensures that a given session will always be routed to the same AS. - -For returned packet from AS, it will do SNAT operation and sent out. - -Please refer to below for details: -https://schd.ws/hosted_files/ossna2017/1e/VPP_K8S_GTPU_OSSNA.pdf - - -## Performance - -The load balancer has been tested up to 1 millions flows and still forwards more -than 3Mpps per core in such circumstances. -Although 3Mpps seems already good, it is likely that performance will be improved -in next versions. - -## Configuration - -### Global LB parameters - -The load balancer needs to be configured with some parameters: - - lb conf [ip4-src-address <addr>] [ip6-src-address <addr>] - [buckets <n>] [timeout <s>] - -ip4-src-address: the source address used to send encap. packets using IPv4 for GRE4 mode. - or Node IP4 address for NAT4 mode. - -ip6-src-address: the source address used to send encap. packets using IPv6 for GRE6 mode. - or Node IP6 address for NAT6 mode. - -buckets: the *per-thread* established-connections-table number of buckets. - -timeout: the number of seconds a connection will remain in the - established-connections-table while no packet for this flow - is received. - -### Configure the VIPs - - lb vip <prefix> [encap (gre6|gre4|l3dsr|nat4|nat6)] \ - [dscp <n>] [port <n> target_port <n> node_port <n>] [new_len <n>] [del] - -new_len is the size of the new-connection-table. It should be 1 or 2 orders of -magnitude bigger than the number of ASs for the VIP in order to ensure a good -load balancing. -Encap l3dsr and dscp is used to map VIP to dscp bit and rewrite DSCP bit in packets. -So the selected server could get VIP from DSCP bit in this packet and perform DSR. -Encap nat4/nat6 and port/target_port/node_port is used to do kube-proxy data plane. - -Examples: - - lb vip 2002::/16 encap gre6 new_len 1024 - lb vip 2003::/16 encap gre4 new_len 2048 - lb vip 80.0.0.0/8 encap gre6 new_len 16 - lb vip 90.0.0.0/8 encap gre4 new_len 1024 - lb vip 100.0.0.0/8 encap l3dsr dscp 2 new_len 32 - lb vip 90.1.2.1/32 encap nat4 port 3306 target_port 3307 node_port 30964 new_len 1024 - lb vip 2004::/16 encap nat6 port 6306 target_port 6307 node_port 30966 new_len 1024 - -### Configure the ASs (for each VIP) - - lb as <vip-prefix> [<address> [<address> [...]]] [del] - -You can add (or delete) as many ASs at a time (for a single VIP). -Note that the AS address family must correspond to the VIP encap. IP family. - -Examples: - - lb as 2002::/16 2001::2 2001::3 2001::4 - lb as 2003::/16 10.0.0.1 10.0.0.2 - lb as 80.0.0.0/8 2001::2 - lb as 90.0.0.0/8 10.0.0.1 - -### Configure SNAT - - lb set interface nat4 in <intfc> [del] - -Set SNAT feature in a specific interface. -(applicable in NAT4 mode only) - - lb set interface nat6 in <intfc> [del] - -Set SNAT feature in a specific interface. -(applicable in NAT6 mode only) - -## Monitoring - -The plugin provides quite a bunch of counters and information. -These are still subject to quite significant changes. - - show lb - show lb vip - show lb vip verbose - - show node counters - - -## Design notes - -### Multi-Threading - -MagLev is a distributed system which pseudo-randomly generates a -new-connections-table based on AS names such that each server configured with -the same set of ASs ends up with the same table. Connection stickyness is then -ensured with an established-connections-table. Using ECMP, it is assumed (but -not relied on) that servers will mostly receive traffic for different flows. - -This implementation pushes the parallelism a little bit further by using -one established-connections table per thread. This is equivalent to assuming -that RSS will make a job similar to ECMP, and is pretty useful as threads don't -need to get a lock in order to write in the table. - -### Hash Table - -A load balancer requires an efficient read and write hash table. The hash table -used by ip6-forward is very read-efficient, but not so much for writing. In -addition, it is not a big deal if writing into the hash table fails (again, -MagLev uses a flow table but does not heaviliy relies on it). - -The plugin therefore uses a very specific (and stupid) hash table. - - Fixed (and power of 2) number of buckets (configured at runtime) - - Fixed (and power of 2) elements per buckets (configured at compilation time) - -### Reference counting - -When an AS is removed, there is two possible ways to react. - - Keep using the AS for established connections - - Change AS for established connections (likely to cause error for TCP) - -In the first case, although an AS is removed from the configuration, its -associated state needs to stay around as long as it is used by at least one -thread. - -In order to avoid locks, a specific reference counter is used. The design is quite -similar to clib counters but: - - It is possible to decrease the value - - Summing will not zero the per-thread counters - - Only the thread can reallocate its own counters vector (to avoid concurrency issues) - -This reference counter is lock free, but reading a count of 0 does not mean -the value can be freed unless it is ensured by *other* means that no other thread -is concurrently referencing the object. In the case of this plugin, it is assumed -that no concurrent event will take place after a few seconds. - diff --git a/src/plugins/lb/lb_plugin_doc.rst b/src/plugins/lb/lb_plugin_doc.rst new file mode 100644 index 00000000000..603453e7848 --- /dev/null +++ b/src/plugins/lb/lb_plugin_doc.rst @@ -0,0 +1,223 @@ +Load Balancer plugin +==================== + +Version +------- + +The load balancer plugin is currently in *beta* version. Both CLIs and +APIs are subject to *heavy* changes, which also means feedback is really +welcome regarding features, apis, etc… + +Overview +-------- + +This plugin provides load balancing for VPP in a way that is largely +inspired from Google’s MagLev: +http://research.google.com/pubs/pub44824.html + +The load balancer is configured with a set of Virtual IPs (VIP, which +can be prefixes), and for each VIP, with a set of Application Server +addresses (ASs). + +There are four encap types to steer traffic to different ASs: 1). +IPv4+GRE ad IPv6+GRE encap types: Traffic received for a given VIP (or +VIP prefix) is tunneled using GRE towards the different ASs in a way +that (tries to) ensure that a given session will always be tunneled to +the same AS. + +2). IPv4+L3DSR encap types: L3DSR is used to overcome Layer 2 +limitations of Direct Server Return Load Balancing. It maps VIP to DSCP +bits, and reuse TOS bits to transfer DSCP bits to server, and then +server will get VIP from DSCP-to-VIP mapping. + +Both VIPs or ASs can be IPv4 or IPv6, but for a given VIP, all ASs must +be using the same encap. type (i.e. IPv4+GRE or IPv6+GRE or IPv4+L3DSR). +Meaning that for a given VIP, all AS addresses must be of the same +family. + +3). IPv4/IPv6 + NAT4/NAT6 encap types: This type provides kube-proxy +data plane on user space, which is used to replace linux kernel’s +kube-proxy based on iptables. + +Currently, load balancer plugin supports three service types: a) Cluster +IP plus Port: support any protocols, including TCP, UDP. b) Node IP plus +Node Port: currently only support UDP. c) External Load Balancer. + +For Cluster IP plus Port case: kube-proxy is configured with a set of +Virtual IPs (VIP, which can be prefixes), and for each VIP, with a set +of AS addresses (ASs). + +For a specific session received for a given VIP (or VIP prefix), first +packet selects a AS according to internal load balancing algorithm, then +does DNAT operation and sent to chosen AS. At the same time, will create +a session entry to store AS chosen result. Following packets for that +session will look up session table first, which ensures that a given +session will always be routed to the same AS. + +For returned packet from AS, it will do SNAT operation and sent out. + +Please refer to below for details: +https://schd.ws/hosted_files/ossna2017/1e/VPP_K8S_GTPU_OSSNA.pdf + +Performance +----------- + +The load balancer has been tested up to 1 millions flows and still +forwards more than 3Mpps per core in such circumstances. Although 3Mpps +seems already good, it is likely that performance will be improved in +next versions. + +Configuration +------------- + +Global LB parameters +~~~~~~~~~~~~~~~~~~~~ + +The load balancer needs to be configured with some parameters: + +:: + + lb conf [ip4-src-address <addr>] [ip6-src-address <addr>] + [buckets <n>] [timeout <s>] + +ip4-src-address: the source address used to send encap. packets using +IPv4 for GRE4 mode. or Node IP4 address for NAT4 mode. + +ip6-src-address: the source address used to send encap. packets using +IPv6 for GRE6 mode. or Node IP6 address for NAT6 mode. + +buckets: the *per-thread* established-connections-table number of +buckets. + +timeout: the number of seconds a connection will remain in the +established-connections-table while no packet for this flow is received. + +Configure the VIPs +~~~~~~~~~~~~~~~~~~ + +:: + + lb vip <prefix> [encap (gre6|gre4|l3dsr|nat4|nat6)] \ + [dscp <n>] [port <n> target_port <n> node_port <n>] [new_len <n>] [del] + +new_len is the size of the new-connection-table. It should be 1 or 2 +orders of magnitude bigger than the number of ASs for the VIP in order +to ensure a good load balancing. Encap l3dsr and dscp is used to map VIP +to dscp bit and rewrite DSCP bit in packets. So the selected server +could get VIP from DSCP bit in this packet and perform DSR. Encap +nat4/nat6 and port/target_port/node_port is used to do kube-proxy data +plane. + +Examples: + +:: + + lb vip 2002::/16 encap gre6 new_len 1024 + lb vip 2003::/16 encap gre4 new_len 2048 + lb vip 80.0.0.0/8 encap gre6 new_len 16 + lb vip 90.0.0.0/8 encap gre4 new_len 1024 + lb vip 100.0.0.0/8 encap l3dsr dscp 2 new_len 32 + lb vip 90.1.2.1/32 encap nat4 port 3306 target_port 3307 node_port 30964 new_len 1024 + lb vip 2004::/16 encap nat6 port 6306 target_port 6307 node_port 30966 new_len 1024 + +Configure the ASs (for each VIP) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + lb as <vip-prefix> [<address> [<address> [...]]] [del] + +You can add (or delete) as many ASs at a time (for a single VIP). Note +that the AS address family must correspond to the VIP encap. IP family. + +Examples: + +:: + + lb as 2002::/16 2001::2 2001::3 2001::4 + lb as 2003::/16 10.0.0.1 10.0.0.2 + lb as 80.0.0.0/8 2001::2 + lb as 90.0.0.0/8 10.0.0.1 + +Configure SNAT +~~~~~~~~~~~~~~ + +:: + + lb set interface nat4 in <intfc> [del] + +Set SNAT feature in a specific interface. (applicable in NAT4 mode only) + +:: + + lb set interface nat6 in <intfc> [del] + +Set SNAT feature in a specific interface. (applicable in NAT6 mode only) + +Monitoring +---------- + +The plugin provides quite a bunch of counters and information. These are +still subject to quite significant changes. + +:: + + show lb + show lb vip + show lb vip verbose + + show node counters + +Design notes +------------ + +Multi-Threading +~~~~~~~~~~~~~~~ + +MagLev is a distributed system which pseudo-randomly generates a +new-connections-table based on AS names such that each server configured +with the same set of ASs ends up with the same table. Connection +stickiness is then ensured with an established-connections-table. Using +ECMP, it is assumed (but not relied on) that servers will mostly receive +traffic for different flows. + +This implementation pushes the parallelism a little bit further by using +one established-connections table per thread. This is equivalent to +assuming that RSS will make a job similar to ECMP, and is pretty useful +as threads don’t need to get a lock in order to write in the table. + +Hash Table +~~~~~~~~~~ + +A load balancer requires an efficient read and write hash table. The +hash table used by ip6-forward is very read-efficient, but not so much +for writing. In addition, it is not a big deal if writing into the hash +table fails (again, MagLev uses a flow table but does not heavily +relies on it). + +The plugin therefore uses a very specific (and stupid) hash table. - +Fixed (and power of 2) number of buckets (configured at runtime) - Fixed +(and power of 2) elements per buckets (configured at compilation time) + +Reference counting +~~~~~~~~~~~~~~~~~~ + +When an AS is removed, there is two possible ways to react. - Keep using +the AS for established connections - Change AS for established +connections (likely to cause error for TCP) + +In the first case, although an AS is removed from the configuration, its +associated state needs to stay around as long as it is used by at least +one thread. + +In order to avoid locks, a specific reference counter is used. The +design is quite similar to clib counters but: - It is possible to +decrease the value - Summing will not zero the per-thread counters - +Only the thread can reallocate its own counters vector (to avoid +concurrency issues) + +This reference counter is lock free, but reading a count of 0 does not +mean the value can be freed unless it is ensured by *other* means that +no other thread is concurrently referencing the object. In the case of +this plugin, it is assumed that no concurrent event will take place +after a few seconds. diff --git a/src/plugins/lldp/lldp_doc.md b/src/plugins/lldp/lldp_doc.md deleted file mode 100644 index 717de898c4e..00000000000 --- a/src/plugins/lldp/lldp_doc.md +++ /dev/null @@ -1,86 +0,0 @@ -# VPP Link Layer Discovery Protocol (LLDP) implementation {#lldp_doc} - -This is a memo intended to contain documentation of the VPP LLDP implementation -Everything that is not directly obvious should come here. - - -## LLDP -LLDP is a link layer protocol to advertise the capabilities and current status of the system. - -There are 2 nodes handling LLDP - -1.) input-node which processes incoming packets and updates the local database -2.) process-node which is responsible for sending out LLDP packets from VPP side - - -### Configuration - -LLDP has a global configuration and a per-interface enable setting. - -Global configuration is modified using the "set lldp" command - -set lldp [system-name <string>] [tx-hold <value>] [tx-interval <value>] - -system-name: the name of the VPP system sent to peers in the system-name TLV -tx-hold: multiplier for tx-interval when setting time-to-live (TTL) value in the LLDP packets (TTL = tx-hold * tx-interval + 1, if TTL > 65535, then TTL = 65535) -tx-interval: time interval between sending out LLDP packets - -Per interface setting is done using the "set interface lldp" command - -set interface lldp <interface> | if_index <idx> [port-desc <string>] [disable] - -interface: the name of the interface for which to enable/disable LLDP -if_index: sw interface index can be used if interface name is not used. -port-desc: port description -disable: LLDP feature can be enabled or disabled per interface. - -### Configuration example - -Configure system-name as "VPP" and transmit interval to 10 seconds: - -set lldp system-name VPP tx-interval 10 - -Enable LLDP on interface TenGigabitEthernet5/0/1 with port description - -set interface lldp TenGigabitEthernet5/0/1 port-desc vtf:eth0 - - -### Operational data - -The list of LLDP-enabled interfaces which are up can be shown using "show lldp" command - -Example: -DBGvpp# show lldp -Local interface Peer chassis ID Remote port ID Last heard Last sent Status -GigabitEthernet2/0/1 never 27.0s ago inactive -TenGigabitEthernet5/0/1 8c:60:4f:dd:ca:52 Eth1/3/3 20.1s ago 18.3s ago active - -All LLDP configuration data with all LLDP-enabled interfaces can be shown using "show lldp detail" command - -Example: -DBGvpp# show lldp detail -LLDP configuration: -Configured system name: vpp -Configured tx-hold: 4 -Configured tx-interval: 30 - -LLDP-enabled interface table: - -Interface name: GigabitEthernet2/0/1 -Interface/peer state: inactive(timeout) -Last known peer chassis ID: -Last known peer port ID: -Last packet sent: 12.4s ago -Last packet received: never - -Interface name: GigabitEthernet2/0/2 -Interface/peer state: interface down -Last packet sent: never - -Interface name: TenGigabitEthernet5/0/1 -Interface/peer state: active -Peer chassis ID: 8c:60:4f:dd:ca:52(MAC address) -Remote port ID: Eth1/3/3(Locally assigned) -Last packet sent: 3.6s ago -Last packet received: 5.5s ago - diff --git a/src/plugins/lldp/lldp_doc.rst b/src/plugins/lldp/lldp_doc.rst new file mode 100644 index 00000000000..a6737985aab --- /dev/null +++ b/src/plugins/lldp/lldp_doc.rst @@ -0,0 +1,84 @@ +LLDP Protocol +============= + +This is a memo intended to contain documentation of the VPP LLDP (Link +Layer Discovery Protocol) implementation Everything that is not directly +obvious should come here. + +LLDP +---- + +LLDP is a link layer protocol to advertise the capabilities and current +status of the system. + +There are 2 nodes handling LLDP + +1.) input-node which processes incoming packets and updates the local +database 2.) process-node which is responsible for sending out LLDP +packets from VPP side + +Configuration +~~~~~~~~~~~~~ + +LLDP has a global configuration and a per-interface enable setting. + +Global configuration is modified using the “set lldp” command + +set lldp [system-name ] [tx-hold ] [tx-interval ] + +system-name: the name of the VPP system sent to peers in the system-name +TLV tx-hold: multiplier for tx-interval when setting time-to-live (TTL) +value in the LLDP packets (TTL = tx-hold \* tx-interval + 1, if TTL > +65535, then TTL = 65535) tx-interval: time interval between sending out +LLDP packets + +Per interface setting is done using the “set interface lldp” command + +set interface lldp \| if_index [port-desc ] [disable] + +interface: the name of the interface for which to enable/disable LLDP +if_index: sw interface index can be used if interface name is not used. +port-desc: port description disable: LLDP feature can be enabled or +disabled per interface. + +Configuration example +~~~~~~~~~~~~~~~~~~~~~ + +Configure system-name as “VPP” and transmit interval to 10 seconds: + +set lldp system-name VPP tx-interval 10 + +Enable LLDP on interface TenGigabitEthernet5/0/1 with port description + +set interface lldp TenGigabitEthernet5/0/1 port-desc vtf:eth0 + +Operational data +~~~~~~~~~~~~~~~~ + +The list of LLDP-enabled interfaces which are up can be shown using +“show lldp” command + +Example: DBGvpp# show lldp Local interface Peer chassis ID Remote port +ID Last heard Last sent Status GigabitEthernet2/0/1 never 27.0s ago +inactive TenGigabitEthernet5/0/1 8c:60:4f:dd:ca:52 Eth1/3/3 20.1s ago +18.3s ago active + +All LLDP configuration data with all LLDP-enabled interfaces can be +shown using “show lldp detail” command + +Example: DBGvpp# show lldp detail LLDP configuration: Configured system +name: vpp Configured tx-hold: 4 Configured tx-interval: 30 + +LLDP-enabled interface table: + +Interface name: GigabitEthernet2/0/1 Interface/peer state: +inactive(timeout) Last known peer chassis ID: Last known peer port ID: +Last packet sent: 12.4s ago Last packet received: never + +Interface name: GigabitEthernet2/0/2 Interface/peer state: interface +down Last packet sent: never + +Interface name: TenGigabitEthernet5/0/1 Interface/peer state: active +Peer chassis ID: 8c:60:4f:dd:ca:52(MAC address) Remote port ID: +Eth1/3/3(Locally assigned) Last packet sent: 3.6s ago Last packet +received: 5.5s ago diff --git a/src/plugins/map/map_doc.md b/src/plugins/map/map_doc.md deleted file mode 100644 index f3e2a56706d..00000000000 --- a/src/plugins/map/map_doc.md +++ /dev/null @@ -1,69 +0,0 @@ -# VPP MAP and Lw4o6 implementation {#map_doc} - -This is a memo intended to contain documentation of the VPP MAP and Lw4o6 implementations. -Everything that is not directly obvious should come here. - - - -## MAP-E Virtual Reassembly - -The MAP-E implementation supports handling of IPv4 fragments as well as IPv4-in-IPv6 inner and outer fragments. This is called virtual reassembly because the fragments are not actually reassembled. Instead, some meta-data are kept about the first fragment and reused for subsequent fragments. - -Fragment caching and handling is not always necessary. It is performed when: -* An IPv4 fragment is received and the destination IPv4 address is shared. -* An IPv6 packet is received with an inner IPv4 fragment, the IPv4 source address is shared, and 'security-check fragments' is on. -* An IPv6 fragment is received. - -There are 3 dedicated nodes: -* ip4-map-reass -* ip6-map-ip4-reass -* ip6-map-ip6-reass - -ip4-map sends all fragments to ip4-map-reass. -ip6-map sends all inner-fragments to ip6-map-ip4-reass. -ip6-map sends all outer-fragments to ip6-map-ip6-reass. - -IPv4 (resp. IPv6) virtual reassembly makes use of a hash table in order to store IPv4 (resp. IPv6) reassembly structures. The hash-key is based on the IPv4-src:IPv4-dst:Frag-ID:Protocol tuple (resp. IPv6-src:IPv6-dst:Frag-ID tuple, as the protocol is IPv4-in-IPv6). Therefore, each packet reassembly makes use of exactly one reassembly structure. When such a structure is allocated, it is timestamped with the current time. Finally, those structures are capable of storing a limited number of buffer indexes. - -An IPv4 (resp. IPv6) reassembly structure can cache up to MAP_IP4_REASS_MAX_FRAGMENTS_PER_REASSEMBLY (resp. MAP_IP6_REASS_MAX_FRAGMENTS_PER_REASSEMBLY) buffers. Buffers are cached until the first fragment is received. - -#### Virtual Reassembly configuration - -IPv4 and IPv6 virtual reassembly support the following configuration: - map params reassembly [ip4 | ip6] [lifetime <lifetime-ms>] [pool-size <pool-size>] [buffers <buffers>] [ht-ratio <ht-ratio>] - -lifetime: - The time in milliseconds a reassembly structure is considered valid. The longer, the more reliable is reassembly, but the more likely it is to exhaust the pool of reassembly structures. IPv4 standard suggests a lifetime of 15 seconds. IPv6 specifies a lifetime of 60 seconds. Those values are not realistic for high-throughput cases. - -buffers: - The upper limit of buffers that are allowed to be cached. It can be used to protect against fragmentation attacks which would aim to exhaust the global buffers pool. - -pool-size: - The number of reassembly structures that can be allocated. As each structure can store a small fixed number of fragments, it also sets an upper-bound of 'pool-size * MAP_IPX_REASS_MAX_FRAGMENTS_PER_REASSEMBLY' buffers that can be cached in total. - -ht-ratio: - The amount of buckets in the hash-table is pool-size * ht-ratio. - - -Any time pool-size and ht-ratio is modified, the hash-table is destroyed and created again, which means all current state is lost. - - -##### Additional considerations - -Reassembly at high rate is expensive in terms of buffers. There is a trade-off between the lifetime and number of allocated buffers. Reducing the lifetime helps, but at the cost of loosing state for fragments that are wide appart. - -Let: -R be the packet rate at which fragments are received. -F be the number of fragments per packet. - -Assuming the first fragment is always received last. We should have: -buffers > lifetime * R / F * (F - 1) -pool-size > lifetime * R/F - -This is a worst case. Receiving the first fragment earlier helps reducing the number of required buffers. Also, an optimization is implemented (MAP_IP6_REASS_COUNT_BYTES and MAP_IP4_REASS_COUNT_BYTES) which counts the number of transmitted bytes and remembers the total number of bytes which should be transmitted based on the last fragment, and therefore helps reducing 'pool-size'. - -But the formula shows that it is challenging to forward a significant amount of fragmented packets at high rates. For instance, with a lifetime of 1 second, 5Mpps packet rate would require buffering up to 2.5 millions fragments. - -If you want to do that, be prepared to configure a lot of fragments. - - diff --git a/src/plugins/map/map_doc.rst b/src/plugins/map/map_doc.rst new file mode 100644 index 00000000000..663e815d545 --- /dev/null +++ b/src/plugins/map/map_doc.rst @@ -0,0 +1,99 @@ +MAP and Lw4o6 +============= + +This is a memo intended to contain documentation of the VPP MAP and +Lw4o6 implementations. Everything that is not directly obvious should +come here. + +MAP-E Virtual Reassembly +------------------------ + +The MAP-E implementation supports handling of IPv4 fragments as well as +IPv4-in-IPv6 inner and outer fragments. This is called virtual +reassembly because the fragments are not actually reassembled. Instead, +some meta-data are kept about the first fragment and reused for +subsequent fragments. + +Fragment caching and handling is not always necessary. It is performed +when: \* An IPv4 fragment is received and the destination IPv4 address +is shared. \* An IPv6 packet is received with an inner IPv4 fragment, +the IPv4 source address is shared, and ‘security-check fragments’ is on. +\* An IPv6 fragment is received. + +There are 3 dedicated nodes: \* ip4-map-reass \* ip6-map-ip4-reass \* +ip6-map-ip6-reass + +ip4-map sends all fragments to ip4-map-reass. ip6-map sends all +inner-fragments to ip6-map-ip4-reass. ip6-map sends all outer-fragments +to ip6-map-ip6-reass. + +IPv4 (resp. IPv6) virtual reassembly makes use of a hash table in order +to store IPv4 (resp. IPv6) reassembly structures. The hash-key is based +on the IPv4-src:IPv4-dst:Frag-ID:Protocol tuple (resp. +IPv6-src:IPv6-dst:Frag-ID tuple, as the protocol is IPv4-in-IPv6). +Therefore, each packet reassembly makes use of exactly one reassembly +structure. When such a structure is allocated, it is timestamped with +the current time. Finally, those structures are capable of storing a +limited number of buffer indexes. + +An IPv4 (resp. IPv6) reassembly structure can cache up to +MAP_IP4_REASS_MAX_FRAGMENTS_PER_REASSEMBLY (resp. +MAP_IP6_REASS_MAX_FRAGMENTS_PER_REASSEMBLY) buffers. Buffers are cached +until the first fragment is received. + +Virtual Reassembly configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +IPv4 and IPv6 virtual reassembly support the following configuration: +map params reassembly [ip4 \| ip6] [lifetime ] [pool-size ] [buffers ] +[ht-ratio ] + +lifetime: The time in milliseconds a reassembly structure is considered +valid. The longer, the more reliable is reassembly, but the more likely +it is to exhaust the pool of reassembly structures. IPv4 standard +suggests a lifetime of 15 seconds. IPv6 specifies a lifetime of 60 +seconds. Those values are not realistic for high-throughput cases. + +buffers: The upper limit of buffers that are allowed to be cached. It +can be used to protect against fragmentation attacks which would aim to +exhaust the global buffers pool. + +pool-size: The number of reassembly structures that can be allocated. As +each structure can store a small fixed number of fragments, it also sets +an upper-bound of ‘pool-size \* +MAP_IPX_REASS_MAX_FRAGMENTS_PER_REASSEMBLY’ buffers that can be cached +in total. + +ht-ratio: The amount of buckets in the hash-table is pool-size \* +ht-ratio. + +Any time pool-size and ht-ratio is modified, the hash-table is destroyed +and created again, which means all current state is lost. + +Additional considerations +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Reassembly at high rate is expensive in terms of buffers. There is a +trade-off between the lifetime and number of allocated buffers. Reducing +the lifetime helps, but at the cost of loosing state for fragments that +are wide apart. + +Let: R be the packet rate at which fragments are received. F be the +number of fragments per packet. + +Assuming the first fragment is always received last. We should have: +buffers > lifetime \* R / F \* (F - 1) pool-size > lifetime \* R/F + +This is a worst case. Receiving the first fragment earlier helps +reducing the number of required buffers. Also, an optimization is +implemented (MAP_IP6_REASS_COUNT_BYTES and MAP_IP4_REASS_COUNT_BYTES) +which counts the number of transmitted bytes and remembers the total +number of bytes which should be transmitted based on the last fragment, +and therefore helps reducing ‘pool-size’. + +But the formula shows that it is challenging to forward a significant +amount of fragmented packets at high rates. For instance, with a +lifetime of 1 second, 5Mpps packet rate would require buffering up to +2.5 millions fragments. + +If you want to do that, be prepared to configure a lot of fragments. diff --git a/src/plugins/marvell/README.md b/src/plugins/marvell/README.md deleted file mode 100644 index 3f3c27e3618..00000000000 --- a/src/plugins/marvell/README.md +++ /dev/null @@ -1,65 +0,0 @@ -# Marvell device plugin for VPP {#marvell_plugin_doc} - -##Overview -This plugins provides native device support for Marvell PP2 network device, by use of Marvell Usermode SDK ([MUSDK][1]). -Code is developed and tested on [MACCHIATObin][2] board. - -##Prerequisites -Plugins depends on installed MUSDK and Marvell provided linux [kernel][3] with MUSDK provided kernel patches (see `patches/linux` in musdk repo and relevant documentation. -Kernel version used: **4.14.22 armada-18.09.3** -MUSDK version used: **armada-18.09.3** -Following kernel modules from MUSDK must be loaded for plugin to work: -* `musdk_cma.ko` -* `mv_pp_uio.ko` - -##Musdk 18.09.3 compilation steps - -``` -./bootstrap -./configure --prefix=/opt/vpp/external/aarch64/ CFLAGS="-Wno-error=unused-result -g -fPIC" --enable-shared=no -sed -i -e 's/marvell,mv-pp-uio/generic-uio/' modules/pp2/mv_pp_uio.c -sed -i -e 's/O_CREAT/O_CREAT, S_IRUSR | S_IWUSR/' src/lib/file_utils.c -make -sudo make install -``` - -## Usage -### Interface Cration -Interfaces are dynamically created with following CLI: -``` -create interface marvell pp2 name eth0 -set interface state mv-ppio-0/0 up -``` - -Where `eth0` is linux interface name and `mv-ppio-X/Y` is VPP interface name where X is PP2 device ID and Y is PPIO ID -Interface needs to be assigned to MUSDK in FDT configuration and linux interface state must be up. - -### Interface Deletion -Interface can be deleted with following CLI: -``` -delete interface marvell pp2 <interface name> -``` - - -### Interface Statistics -Interface statistics can be displayed with `sh hardware-interface mv-ppio0/0` -command. - -### Interaction with DPDK plugin -This plugin doesn't have any dependency on DPDK or DPDK plugin but it can -work with DPDK plugin enabled or disabled. It is observed that performace is -better around 30% when DPDK plugin is disabled, as DPDK plugin registers -own buffer manager, which needs to deal with additional metadata in each packet. - -DPKD plugin can be disabled by adding following config to the startup.conf. - -``` -plugins { - dpdk_plugin.so { disable } -} -``` - - -[1]: https://github.com/MarvellEmbeddedProcessors/musdk-marvell -[2]: http://macchiatobin.net -[3]: https://github.com/MarvellEmbeddedProcessors/linux-marvell diff --git a/src/plugins/marvell/README.rst b/src/plugins/marvell/README.rst new file mode 100644 index 00000000000..19cf1c49d0e --- /dev/null +++ b/src/plugins/marvell/README.rst @@ -0,0 +1,85 @@ +Marvell device plugin +===================== + +Overview +-------- + +This plugins provides native device support for Marvell PP2 network +device, by use of Marvell Usermode SDK +(`MUSDK <https://github.com/MarvellEmbeddedProcessors/musdk-marvell>`__). +Code is developed and tested on +`MACCHIATObin <http://macchiatobin.net>`__ board. + +Prerequisites +------------- + +Plugins depends on installed MUSDK and Marvell provided linux +`kernel <https://github.com/MarvellEmbeddedProcessors/linux-marvell>`__ +with MUSDK provided kernel patches (see ``patches/linux`` in musdk repo +and relevant documentation. Kernel version used: **4.14.22 +armada-18.09.3** MUSDK version used: **armada-18.09.3** Following kernel +modules from MUSDK must be loaded for plugin to work: \* +``musdk_cma.ko`` \* ``mv_pp_uio.ko`` + +Musdk 18.09.3 compilation steps +------------------------------- + +:: + + ./bootstrap + ./configure --prefix=/opt/vpp/external/aarch64/ CFLAGS="-Wno-error=unused-result -g -fPIC" --enable-shared=no + sed -i -e 's/marvell,mv-pp-uio/generic-uio/' modules/pp2/mv_pp_uio.c + sed -i -e 's/O_CREAT/O_CREAT, S_IRUSR | S_IWUSR/' src/lib/file_utils.c + make + sudo make install + +Usage +----- + +Interface Creation +~~~~~~~~~~~~~~~~~~ + +Interfaces are dynamically created with following CLI: + +:: + + create interface marvell pp2 name eth0 + set interface state mv-ppio-0/0 up + +Where ``eth0`` is linux interface name and ``mv-ppio-X/Y`` is VPP +interface name where X is PP2 device ID and Y is PPIO ID Interface needs +to be assigned to MUSDK in FDT configuration and linux interface state +must be up. + +Interface Deletion +~~~~~~~~~~~~~~~~~~ + +Interface can be deleted with following CLI: + +:: + + delete interface marvell pp2 <interface name> + +Interface Statistics +~~~~~~~~~~~~~~~~~~~~ + +Interface statistics can be displayed with +``sh hardware-interface mv-ppio0/0`` command. + +Interaction with DPDK plugin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This plugin doesn’t have any dependency on DPDK or DPDK plugin but it +can work with DPDK plugin enabled or disabled. It is observed that +performance is better around 30% when DPDK plugin is disabled, as DPDK +plugin registers own buffer manager, which needs to deal with additional +metadata in each packet. + +DPKD plugin can be disabled by adding following config to the +startup.conf. + +:: + + plugins { + dpdk_plugin.so { disable } + } diff --git a/src/plugins/mdata/mdata_doc.md b/src/plugins/mdata/mdata_doc.md deleted file mode 100644 index cbbfb012183..00000000000 --- a/src/plugins/mdata/mdata_doc.md +++ /dev/null @@ -1,24 +0,0 @@ -# Buffer metadata change tracker {#mdata_doc} - -## Introduction - -The mdata plugin uses the vlib main loop "before" performance counter -hook to snapshoot buffer metadata before calling the node dispatch -function. Similarly, the plugin uses the main loop "after" hook to -compare a vectors' worth of buffer metadata after the fact. - -The comparison function is a simple octet-by-octet A != B check. We -accumulate changed octets per-node across the entire run, using a -single spinlock-protected accumulator. - -The "show buffer metadata" command produces a report of all fields -whose values are changed by nodes visited during a given run. - -Since many fields in the vnet_buffer_opaque_t are union members, -it may appear that a certain node changes numerous fields. The entire -point of the exercise is to warn developers that if a packet visits -node N, data placed into opaque union field F *will* be affected. - -One should never assume much about buffer metadata immutability across -arbitrary subgraphs. This tool generates accurate reports, to the -extent that one exercises the required subgraph trajectories. diff --git a/src/plugins/mdata/mdata_doc.rst b/src/plugins/mdata/mdata_doc.rst new file mode 100644 index 00000000000..95746bd3d0e --- /dev/null +++ b/src/plugins/mdata/mdata_doc.rst @@ -0,0 +1,26 @@ +Buffer metadata change tracker +============================== + +Introduction +------------ + +The mdata plugin uses the vlib main loop “before” performance counter +hook to snapshoot buffer metadata before calling the node dispatch +function. Similarly, the plugin uses the main loop “after” hook to +compare a vectors’ worth of buffer metadata after the fact. + +The comparison function is a simple octet-by-octet A != B check. We +accumulate changed octets per-node across the entire run, using a single +spinlock-protected accumulator. + +The “show buffer metadata” command produces a report of all fields whose +values are changed by nodes visited during a given run. + +Since many fields in the vnet_buffer_opaque_t are union members, it may +appear that a certain node changes numerous fields. The entire point of +the exercise is to warn developers that if a packet visits node N, data +placed into opaque union field F *will* be affected. + +One should never assume much about buffer metadata immutability across +arbitrary subgraphs. This tool generates accurate reports, to the extent +that one exercises the required subgraph trajectories. diff --git a/src/plugins/nat/nat44-ei/nat44_ei_ha_doc.md b/src/plugins/nat/nat44-ei/nat44_ei_ha_doc.md deleted file mode 100644 index f0ea209e250..00000000000 --- a/src/plugins/nat/nat44-ei/nat44_ei_ha_doc.md +++ /dev/null @@ -1,70 +0,0 @@ -# Active-Passive NAT HA {#nat_ha_doc} - -## Introduction - -One NAT node actively manages traffic while the other is synchronized and ready to transition to the active state and takes over seamlessly and enforces the same NAT sessions when failure occur. Both nodes share the same configuration settings. - -## Configuration - -### NAT HA protocol -Session synchronization traffic is distributed through an IPv4 UDP connection. The active node sends NAT HA protocol events to passive node. To achieve reliable transfer NAT HA protocol uses acknowledgement with re-transmission. This require the passive node to respond with an acknowledgement message as it receives the data. The active node keeps a record of each packet it sends and maintains a timer from when the packet was sent. The active node re-transmits a packet if the timer expires before receiving the acknowledgement. - -### Topology - -The two NAT nodes have a dedicated link (interface GE0/0/3 on both) to synchronize NAT sessions using NAT HA protocol. - -``` - +-----------------------+ - | outside network | - +-----------------------+ - / \ - / \ - / \ - / \ - / \ -+---------+ +---------+ -| GE0/0/1 | Active Passive | GE0/0/1 | -| | | | -| GE0/0/3|-------------------|GE0/0/3 | -| | sync network | | -| GE0/0/0 | | GE0/0/0 | -+---------+ +---------+ - \ / - \ / - \ / - \ / - \ / - +-----------------------+ - | inside network | - +-----------------------+ -``` - -### Active node configuration - -``` -set interface ip address GigabitEthernet0/0/1 10.15.7.101/24 -set interface ip address GigabitEthernet0/0/0 172.16.10.101/24 -set interface ip address GigabitEthernet0/0/3 10.0.0.1/24 -set interface state GigabitEthernet0/0/0 up -set interface state GigabitEthernet0/0/1 up -set interface state GigabitEthernet0/0/3 up -set interface nat44 in GigabitEthernet0/0/0 out GigabitEthernet0/0/1 -nat44 add address 10.15.7.100 -nat ha listener 10.0.0.1:1234 -nat ha failover 10.0.0.2:2345 -``` - -### Passive node configuration - -``` -set interface ip address GigabitEthernet0/0/1 10.15.7.102/24 -set interface ip address GigabitEthernet0/0/0 172.16.10.102/24 -set interface ip address GigabitEthernet0/0/3 10.0.0.2/24 -set interface state GigabitEthernet0/0/0 up -set interface state GigabitEthernet0/0/1 up -set interface state GigabitEthernet0/0/3 up -set interface nat44 in GigabitEthernet0/0/0 out GigabitEthernet0/0/1 -nat44 add address 10.15.7.100 -nat ha listener 10.0.0.2:2345 -``` - diff --git a/src/plugins/nat/nat44-ei/nat44_ei_ha_doc.rst b/src/plugins/nat/nat44-ei/nat44_ei_ha_doc.rst new file mode 100644 index 00000000000..46befc52351 --- /dev/null +++ b/src/plugins/nat/nat44-ei/nat44_ei_ha_doc.rst @@ -0,0 +1,88 @@ +Active-Passive NAT HA +===================== + +Introduction +------------ + +One NAT node actively manages traffic while the other is synchronized +and ready to transition to the active state and takes over seamlessly +and enforces the same NAT sessions when failure occur. Both nodes share +the same configuration settings. + +Configuration +------------- + +NAT HA protocol +~~~~~~~~~~~~~~~ + +Session synchronization traffic is distributed through an IPv4 UDP +connection. The active node sends NAT HA protocol events to passive +node. To achieve reliable transfer NAT HA protocol uses acknowledgment +with re-transmission. This require the passive node to respond with an +acknowledgment message as it receives the data. The active node keeps a +record of each packet it sends and maintains a timer from when the +packet was sent. The active node re-transmits a packet if the timer +expires before receiving the acknowledgment. + +Topology +~~~~~~~~ + +The two NAT nodes have a dedicated link (interface GE0/0/3 on both) to +synchronize NAT sessions using NAT HA protocol. + +:: + + +-----------------------+ + | outside network | + +-----------------------+ + / \ + / \ + / \ + / \ + / \ + +---------+ +---------+ + | GE0/0/1 | Active Passive | GE0/0/1 | + | | | | + | GE0/0/3|-------------------|GE0/0/3 | + | | sync network | | + | GE0/0/0 | | GE0/0/0 | + +---------+ +---------+ + \ / + \ / + \ / + \ / + \ / + +-----------------------+ + | inside network | + +-----------------------+ + +Active node configuration +~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + set interface ip address GigabitEthernet0/0/1 10.15.7.101/24 + set interface ip address GigabitEthernet0/0/0 172.16.10.101/24 + set interface ip address GigabitEthernet0/0/3 10.0.0.1/24 + set interface state GigabitEthernet0/0/0 up + set interface state GigabitEthernet0/0/1 up + set interface state GigabitEthernet0/0/3 up + set interface nat44 in GigabitEthernet0/0/0 out GigabitEthernet0/0/1 + nat44 add address 10.15.7.100 + nat ha listener 10.0.0.1:1234 + nat ha failover 10.0.0.2:2345 + +Passive node configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + set interface ip address GigabitEthernet0/0/1 10.15.7.102/24 + set interface ip address GigabitEthernet0/0/0 172.16.10.102/24 + set interface ip address GigabitEthernet0/0/3 10.0.0.2/24 + set interface state GigabitEthernet0/0/0 up + set interface state GigabitEthernet0/0/1 up + set interface state GigabitEthernet0/0/3 up + set interface nat44 in GigabitEthernet0/0/0 out GigabitEthernet0/0/1 + nat44 add address 10.15.7.100 + nat ha listener 10.0.0.2:2345 diff --git a/src/plugins/nat/nat64/nat64_doc.md b/src/plugins/nat/nat64/nat64_doc.md deleted file mode 100644 index f65b46338b0..00000000000 --- a/src/plugins/nat/nat64/nat64_doc.md +++ /dev/null @@ -1,73 +0,0 @@ -# Stateful NAT64: Network Address and Protocol Translation from IPv6 Clients to IPv4 Servers {#nat64_doc} - -## Introduction - -Stateful NAT64 in VPP allows IPv6-only clients to contact IPv4 servers using unicast UDP, TCP, or ICMP based on RFC 6146. - -## Configuration - -### Enable/disable NAT64 feature on the interface - -> set interface nat64 in|out <intfc> [del] - -in: inside/local/IPv6 network -out: outside/external/IPv4 network -intfc: interface name - -### Add/delete NAT64 pool address - -One or more public IPv4 addresses assigned to a NAT64 are shared among several IPv6-only clients. - -> nat64 add pool address <ip4-range-start> [- <ip4-range-end>] [tenant-vrf <tenant-vrf-id>] [del] - -ip4-range-start: First IPv4 address of the range -ip4-range-end: Last IPv4 address of the range (optional, not used for single address) -tenant-vrf-id: VRF id of the tenant associated with the pool address (optional, if not set pool address is global) - -### Add/delete static BIB entry - -Stateful NAT64 also supports IPv4-initiated communications to a subset of the IPv6 hosts through staticaly configured bindings. - -> nat64 add static bib <ip6-addr> <in-port> <ip4-addr> <out-port> tcp|udp|icmp [vfr <table-id>] [del] - -ip6-addr: inside IPv6 address of the host -in-port: inside port or ICMPv6 identifier -ip4-addr: outside IPv4 address of the host -out-port: outside port or ICMPv4 identifier -table-id: VRF id of the tenant associated with the BIB entry (optional, default use global VRF) - -### Set NAT64 session timeouts - -Session is deleted when timer expires. If all sessions corresponding to a dynamically create BIB entry are deleted, then the BIB entry is also deleted. When packets are flowing sessiom timer is refreshed to keep the session alive. - -> set nat64 timeouts udp <sec> icmp <sec> tcp-trans <sec> tcp-est <sec> tcp-incoming-syn <sec> | reset - -udp: UDP session timeout value (default 300sec) -icmp: ICMP session timeout value (default 60sec) -tcp-trans: transitory TCP session timeout value (default 240sec) -tcp-est: established TCP session timeout value (default 7440sec) -tcp-incoming-syn: incoming SYN TCP session timeout value (default 6sec) -reset: reset timers to default values - -### Set NAT64 prefix - -Stateful NAT64 support the algorithm for generating IPv6 representations of IPv4 addresses defined in RFC 6052. If no prefix is configured, Well-Known Prefix (64:ff9b::/96) is used. - -> nat64 add prefix <ip6-prefix>/<plen> [tenant-vrf <vrf-id>] [del] - -ip6-prefix: IPv6 prefix -plen: prefix length (valid values: 32, 40, 48, 56, 64, or 96) -tenant-vrf: VRF id of the tenant associated with the prefix - -### Show commands - -> show nat64 pool -> show nat64 interfaces -> show nat64 bib tcp|udp|icmp -> show nat64 session table tcp|udp|icmp -> show nat64 timeouts -> show nat64 prefix - -## Notes - -Multi thread is not supported yet (CLI/API commands are disabled when VPP runs with multiple threads). diff --git a/src/plugins/nat/nat64/nat64_doc.rst b/src/plugins/nat/nat64/nat64_doc.rst new file mode 100644 index 00000000000..f375fba68bd --- /dev/null +++ b/src/plugins/nat/nat64/nat64_doc.rst @@ -0,0 +1,91 @@ +Stateful NAT64 +============== + +This document describes stateful NAT64 Network Address and Protocol +Translation + +Introduction +------------ + +Stateful NAT64 in VPP allows IPv6-only clients to contact IPv4 servers +using unicast UDP, TCP, or ICMP based on RFC 6146. + +Configuration +------------- + +Enable/disable NAT64 feature on the interface +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + set interface nat64 in|out [del] + +in: inside/local/IPv6 network out: outside/external/IPv4 network intfc: +interface name + +Add/delete NAT64 pool address +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One or more public IPv4 addresses assigned to a NAT64 are shared among +several IPv6-only clients. + + nat64 add pool address [- ] [tenant-vrf ] [del] + +ip4-range-start: First IPv4 address of the range ip4-range-end: Last +IPv4 address of the range (optional, not used for single address) +tenant-vrf-id: VRF id of the tenant associated with the pool address +(optional, if not set pool address is global) + +Add/delete static BIB entry +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Stateful NAT64 also supports IPv4-initiated communications to a subset +of the IPv6 hosts through statically configured bindings. + + nat64 add static bib tcp|udp|icmp [vfr ] [del] + +ip6-addr: inside IPv6 address of the host in-port: inside port or ICMPv6 +identifier ip4-addr: outside IPv4 address of the host out-port: outside +port or ICMPv4 identifier table-id: VRF id of the tenant associated with +the BIB entry (optional, default use global VRF) + +Set NAT64 session timeouts +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Session is deleted when timer expires. If all sessions corresponding to +a dynamically create BIB entry are deleted, then the BIB entry is also +deleted. When packets are flowing session timer is refreshed to keep the +session alive. + + set nat64 timeouts udp icmp tcp-trans tcp-est tcp-incoming-syn \| + reset + +udp: UDP session timeout value (default 300sec) icmp: ICMP session +timeout value (default 60sec) tcp-trans: transitory TCP session timeout +value (default 240sec) tcp-est: established TCP session timeout value +(default 7440sec) tcp-incoming-syn: incoming SYN TCP session timeout +value (default 6sec) reset: reset timers to default values + +Set NAT64 prefix +~~~~~~~~~~~~~~~~ + +Stateful NAT64 support the algorithm for generating IPv6 representations +of IPv4 addresses defined in RFC 6052. If no prefix is configured, +Well-Known Prefix (64:ff9b::/96) is used. + + nat64 add prefix / [tenant-vrf ] [del] + +ip6-prefix: IPv6 prefix plen: prefix length (valid values: 32, 40, 48, +56, 64, or 96) tenant-vrf: VRF id of the tenant associated with the +prefix + +Show commands +~~~~~~~~~~~~~ + + show nat64 pool show nat64 interfaces show nat64 bib tcp|udp|icmp + show nat64 session table tcp|udp|icmp show nat64 timeouts show nat64 + prefix + +Notes +----- + +Multi thread is not supported yet (CLI/API commands are disabled when +VPP runs with multiple threads). diff --git a/src/plugins/nat/pnat/pnat.md b/src/plugins/nat/pnat/pnat.md deleted file mode 100644 index 1e6bc130848..00000000000 --- a/src/plugins/nat/pnat/pnat.md +++ /dev/null @@ -1,37 +0,0 @@ -# PNAT: 1:1 match and rewrite programmable NAT {#pnat_doc} - -PNAT is a stateless statically configured, match and rewrite plugin. -It uses a set of match and rewrite rules that are applied on the IP -input and output feature paths. A PNAT rule is unidirectional. - -The match is done using up to a 6-tuple; IP source and destination address, -IP protocol, transport layer source and destination ports, and FIB table / interface index. - -While multiple match/rewrite rules can be applied to an interface (per direction), the match -pattern must be the same across all rules on that interface/direction. - -If required in the future, matching could be done using the general classifier, allowing matching -on any protocol field, as well having an ordered set of match patterns. - -If the packet does not match, it will by default be passed to the next graph node in the feature chain. -If desired a different miss behaviour could be implemented, e.g. similarly to dynamic NAT, the packet punted to a slow path. - -## Rewrite instructions - -``` c -typedef enum { - PNAT_INSTR_NONE = 1 << 0, - PNAT_INSTR_SOURCE_ADDRESS = 1 << 1, - PNAT_INSTR_SOURCE_PORT = 1 << 2, - PNAT_INSTR_DESTINATION_ADDRESS = 1 << 3, - PNAT_INSTR_DESTINATION_PORT = 1 << 4, -} pnat_instructions_t; -``` - -These are the supported rewrite instructions. -The IP checksum and the TCP/UDP checksum are incrementally updated as required. - -There are only a few "sanity checks" on the rewrites. For example, the rewrite in the outbound direction -is applied on the ip-output feature chain. If one were to rewrite the IP destination address, the routing -decision and determination of the next-hop has already been done, and the packet would still be forwarded -to the original next-hop. diff --git a/src/plugins/nat/pnat/pnat.rst b/src/plugins/nat/pnat/pnat.rst new file mode 100644 index 00000000000..5cac047a236 --- /dev/null +++ b/src/plugins/nat/pnat/pnat.rst @@ -0,0 +1,45 @@ +PNAT 1:1 match & rewrite NAT +============================ + +PNAT is a stateless statically configured, match and rewrite plugin. It +uses a set of match and rewrite rules that are applied on the IP input +and output feature paths. A PNAT rule is unidirectional. + +The match is done using up to a 6-tuple; IP source and destination +address, IP protocol, transport layer source and destination ports, and +FIB table / interface index. + +While multiple match/rewrite rules can be applied to an interface (per +direction), the match pattern must be the same across all rules on that +interface/direction. + +If required in the future, matching could be done using the general +classifier, allowing matching on any protocol field, as well having an +ordered set of match patterns. + +If the packet does not match, it will by default be passed to the next +graph node in the feature chain. If desired a different miss behaviour +could be implemented, e.g. similarly to dynamic NAT, the packet punted +to a slow path. + +Rewrite instructions +-------------------- + +.. code:: c + + typedef enum { + PNAT_INSTR_NONE = 1 << 0, + PNAT_INSTR_SOURCE_ADDRESS = 1 << 1, + PNAT_INSTR_SOURCE_PORT = 1 << 2, + PNAT_INSTR_DESTINATION_ADDRESS = 1 << 3, + PNAT_INSTR_DESTINATION_PORT = 1 << 4, + } pnat_instructions_t; + +These are the supported rewrite instructions. The IP checksum and the +TCP/UDP checksum are incrementally updated as required. + +There are only a few “sanity checks” on the rewrites. For example, the +rewrite in the outbound direction is applied on the ip-output feature +chain. If one were to rewrite the IP destination address, the routing +decision and determination of the next-hop has already been done, and +the packet would still be forwarded to the original next-hop. diff --git a/src/plugins/rdma/rdma_doc.md b/src/plugins/rdma/rdma_doc.md deleted file mode 100644 index 3fed5b6fc49..00000000000 --- a/src/plugins/rdma/rdma_doc.md +++ /dev/null @@ -1,75 +0,0 @@ -# RDMA (ibverb) Ethernet driver {#rdma_doc} - -This driver relies on Linux rdma-core (libibverb) userspace poll-mode driver -to rx/tx Ethernet packets. Despite using the RDMA APIs, this is **not** about -RDMA (no Infiniband, no RoCE, no iWARP), just pure traditional Ethernet -packets. - -## Maturity level -Under development: it should work, but has not been thoroughly tested. - -## Supported Hardware - - Mellanox ConnectX-4 - - Mellanox ConnectX-5 - -## Features - - bifurcation: MAC based flow steering for transparent sharing of a single -physical port between multiple virtual interfaces including Linux netdev - - multiqueue - -## Security considerations -When creating a rdma interface, it will receive all packets to the MAC address -attributed to the interface plus a copy of all broadcast and multicast -traffic. -The MAC address is under the control of VPP: **the user controlling VPP can -divert all traffic of any MAC address to the VPP process, including the Linux -netdev MAC address as long as it can create a rdma interface**. -The rights to create a rdma interface are controlled by the access rights of -the `/dev/infiniband/uverbs[0-9]+`device nodes. - -## Quickstart -1. Make sure the `ib_uverbs` module is loaded: -``` -~# modprobe ib_uverbs -``` -2. In VPP, create a new rdma virtual interface tied to the Linux netdev of the -physical port you want to use (`enp94s0f0` in this example): -``` -vpp# create int rdma host-if enp94s0f0 name rdma-0 -``` -3. Use the interface as usual, eg.: -``` -vpp# set int ip addr rdma-0 1.1.1.1/24 -vpp# set int st rdma-0 up -vpp# ping 1.1.1.100` -``` - -## Containers support -It should work in containers as long as: - - the `ib_uverbs` module is loaded - - the device nodes `/dev/infiniband/uverbs[0-9]+` are usable from the - container (but see [security considerations](#Security considerations)) - -## SR-IOV VFs support -It should work on SR-IOV VFs the same way it does with PFs. Because of VFs -security containment features, make sure the MAC address of the rdma VPP -interface matches the MAC address assigned to the underlying VF. -For example: -``` -host# echo 1 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs -host# ip l set dev enp94s0f0 vf 0 mac 92:5d:f5:df:b1:6f spoof on trust off -host# ip l set dev enp94s0f2 up -vpp# create int rdma host-if enp94s0f2 name rdma-0 -vpp# set int mac address rdma-0 92:5d:f5:df:b1:6f -``` -If you plan to use L2 features such as switching, make sure the underlying -VF is configured in trusted mode and spoof-checking is disabled (of course, be -aware of the [security considerations](#Security considerations)): -``` -host# ip l set dev enp94s0f0 vf 0 spoof off trust on -``` - -## Direct Verb mode -Direct Verb allows the driver to access the NIC HW RX/TX rings directly -instead of having to go through libibverb and suffering associated overhead. -It will be automatically selected if the adapter supports it. diff --git a/src/plugins/rdma/rdma_doc.rst b/src/plugins/rdma/rdma_doc.rst new file mode 100644 index 00000000000..c22ea550a75 --- /dev/null +++ b/src/plugins/rdma/rdma_doc.rst @@ -0,0 +1,102 @@ +RDMA (ibverb) device driver +=========================== + +This driver relies on Linux rdma-core (libibverb) userspace poll-mode +driver to rx/tx Ethernet packets. Despite using the RDMA APIs, this is +**not** about RDMA (no Infiniband, no RoCE, no iWARP), just pure +traditional Ethernet packets. + +Maturity level +-------------- + +Under development: it should work, but has not been thoroughly tested. + +Supported Hardware +------------------ + +- Mellanox ConnectX-4 +- Mellanox ConnectX-5 + +Features +-------- + +- bifurcation: MAC based flow steering for transparent sharing of a + single physical port between multiple virtual interfaces including + Linux netdev +- multiqueue + +Security considerations +----------------------- + +When creating a rdma interface, it will receive all packets to the MAC +address attributed to the interface plus a copy of all broadcast and +multicast traffic. The MAC address is under the control of VPP: **the +user controlling VPP can divert all traffic of any MAC address to the +VPP process, including the Linux netdev MAC address as long as it can +create a rdma interface**. The rights to create a rdma interface are +controlled by the access rights of the +``/dev/infiniband/uverbs[0-9]+``\ device nodes. + +Quickstart +---------- + +1. Make sure the ``ib_uverbs`` module is loaded: + +:: + + ~# modprobe ib_uverbs + +2. In VPP, create a new rdma virtual interface tied to the Linux netdev + of the physical port you want to use (``enp94s0f0`` in this example): + +:: + + vpp# create int rdma host-if enp94s0f0 name rdma-0 + +3. Use the interface as usual, e.g.: + +:: + + vpp# set int ip addr rdma-0 1.1.1.1/24 + vpp# set int st rdma-0 up + vpp# ping 1.1.1.100` + +Containers support +------------------ + +It should work in containers as long as: - the ``ib_uverbs`` module is +loaded - the device nodes ``/dev/infiniband/uverbs[0-9]+`` are usable +from the container (but see `security +considerations <#Security%20considerations>`__) + +SR-IOV VFs support +------------------ + +It should work on SR-IOV VFs the same way it does with PFs. Because of +VFs security containment features, make sure the MAC address of the rdma +VPP interface matches the MAC address assigned to the underlying VF. For +example: + +:: + + host# echo 1 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs + host# ip l set dev enp94s0f0 vf 0 mac 92:5d:f5:df:b1:6f spoof on trust off + host# ip l set dev enp94s0f2 up + vpp# create int rdma host-if enp94s0f2 name rdma-0 + vpp# set int mac address rdma-0 92:5d:f5:df:b1:6f + +If you plan to use L2 features such as switching, make sure the +underlying VF is configured in trusted mode and spoof-checking is +disabled (of course, be aware of the `security +considerations <#Security%20considerations>`__): + +:: + + host# ip l set dev enp94s0f0 vf 0 spoof off trust on + +Direct Verb mode +---------------- + +Direct Verb allows the driver to access the NIC HW RX/TX rings directly +instead of having to go through libibverb and suffering associated +overhead. It will be automatically selected if the adapter supports it. diff --git a/src/plugins/srtp/srtp_plugin.md b/src/plugins/srtp/srtp_plugin.md deleted file mode 100644 index 81185864dbe..00000000000 --- a/src/plugins/srtp/srtp_plugin.md +++ /dev/null @@ -1,72 +0,0 @@ -# SRTP (Secure Real-time Transport Protocol) {#srtp_doc} - -libsrtp2 based SRTP transport protocol implementation. - -## Maturity level -Experimental - -## Quickstart - -1. Install libsrtp2-dev. On debian based OS: - -``` -sudo apt get install libsrtp2-dev -``` - -2. Build vpp - -``` -make build -``` - -3. Test protocol using vcl test server and client. On server side, start vpp and server app: - -``` -export VT_PATH=$WS/build-root/build-vpp_debug-native/vpp/bin -$VT_PATH/vcl_test_server 1234 -p srtp -``` - -On client side: - -``` -export VT_PATH=$WS/build-root/build-vpp_debug-native/vpp/bin -$VT_PATH/vcl_test_client <server-ip> 1234 -U -X -S -N 10000 -T 128 -p srtp -``` - -## Custom libsrtp2 build - -1. Create `build/external/packages/srtp.mk` with following example contents: - -``` -srtp_version := 2.3.0 -srtp_tarball := srtp_$(srtp_version).tar.gz -srtp_tarball_md5sum := da38ee5d9c31be212a12964c22d7f795 -srtp_tarball_strip_dirs := 1 -srtp_url := https://github.com/cisco/libsrtp/archive/v$(srtp_version).tar.gz - -define srtp_build_cmds - @cd $(srtp_build_dir) && \ - $(CMAKE) -DCMAKE_INSTALL_PREFIX:PATH=$(srtp_install_dir) \ - -DCMAKE_C_FLAGS='-fPIC -fvisibility=hidden' $(srtp_src_dir) > $(srtp_build_log) - @$(MAKE) $(MAKE_ARGS) -C $(srtp_build_dir) > $(srtp_build_log) -endef - -define srtp_config_cmds - @true -endef - -define srtp_install_cmds - @$(MAKE) $(MAKE_ARGS) -C $(srtp_build_dir) install > $(srtp_install_log) -endef - - -$(eval $(call package,srtp)) -``` - -2. Include `srtp.mk` in `build/external/Makefile` and add to install target. - -3. Rebuild external dependencies: - -``` -make install-ext-deps -``` diff --git a/src/plugins/srtp/srtp_plugin.rst b/src/plugins/srtp/srtp_plugin.rst new file mode 100644 index 00000000000..568ebb66f01 --- /dev/null +++ b/src/plugins/srtp/srtp_plugin.rst @@ -0,0 +1,82 @@ +SRTP Protocol +============= + +This document describe the VPP SRTP (Secure Real-time Transport +Protocol) implementation libsrtp2 based SRTP transport protocol +implementation. + +Maturity level +-------------- + +Experimental + +Quickstart +---------- + +1. Install libsrtp2-dev. On debian based OS: + +:: + + sudo apt get install libsrtp2-dev + +2. Build vpp + +:: + + make build + +3. Test protocol using vcl test server and client. On server side, start + vpp and server app: + +:: + + export VT_PATH=$WS/build-root/build-vpp_debug-native/vpp/bin + $VT_PATH/vcl_test_server 1234 -p srtp + +On client side: + +:: + + export VT_PATH=$WS/build-root/build-vpp_debug-native/vpp/bin + $VT_PATH/vcl_test_client <server-ip> 1234 -U -X -S -N 10000 -T 128 -p srtp + +Custom libsrtp2 build +--------------------- + +1. Create ``build/external/packages/srtp.mk`` with following example + contents: + +:: + + srtp_version := 2.3.0 + srtp_tarball := srtp_$(srtp_version).tar.gz + srtp_tarball_md5sum := da38ee5d9c31be212a12964c22d7f795 + srtp_tarball_strip_dirs := 1 + srtp_url := https://github.com/cisco/libsrtp/archive/v$(srtp_version).tar.gz + + define srtp_build_cmds + @cd $(srtp_build_dir) && \ + $(CMAKE) -DCMAKE_INSTALL_PREFIX:PATH=$(srtp_install_dir) \ + -DCMAKE_C_FLAGS='-fPIC -fvisibility=hidden' $(srtp_src_dir) > $(srtp_build_log) + @$(MAKE) $(MAKE_ARGS) -C $(srtp_build_dir) > $(srtp_build_log) + endef + + define srtp_config_cmds + @true + endef + + define srtp_install_cmds + @$(MAKE) $(MAKE_ARGS) -C $(srtp_build_dir) install > $(srtp_install_log) + endef + + + $(eval $(call package,srtp)) + +2. Include ``srtp.mk`` in ``build/external/Makefile`` and add to install + target. + +3. Rebuild external dependencies: + +:: + + make install-ext-deps diff --git a/src/plugins/srv6-ad-flow/ad_flow_plugin_doc.md b/src/plugins/srv6-ad-flow/ad_flow_plugin_doc.md deleted file mode 100644 index 1f58fc2b663..00000000000 --- a/src/plugins/srv6-ad-flow/ad_flow_plugin_doc.md +++ /dev/null @@ -1,25 +0,0 @@ -# SRv6 endpoint to SR-unaware appliance via per-flow dynamic proxy {#srv6_ad_flow_plugin_doc} - -## Overview - -TBD - -## CLI configuration - -The following command instantiates a new End.AD.Flow segment that sends the inner -packets on interface `IFACE-OUT` towards an appliance at address `S-ADDR` and -restores the encapsulation headers of the packets coming back on interface -`IFACE-IN`. - -``` -sr localsid address SID behavior end.ad.flow nh S-ADDR oif IFACE-OUT iif IFACE-IN -``` - -For example, the below command configures the SID `1::A1` with an End.AD.Flow -function for sending traffic on interface `GigabitEthernet0/8/0` to the -appliance at address `A1::`, and receiving it back on interface -`GigabitEthernet0/9/0`. - -``` -sr localsid address 1::A1 behavior end.ad.flow nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 -``` diff --git a/src/plugins/srv6-ad-flow/ad_flow_plugin_doc.rst b/src/plugins/srv6-ad-flow/ad_flow_plugin_doc.rst new file mode 100644 index 00000000000..7e628742f84 --- /dev/null +++ b/src/plugins/srv6-ad-flow/ad_flow_plugin_doc.rst @@ -0,0 +1,31 @@ +SRv6 per-flow dynamic proxy +=========================== + +This document describes SRv6 endpoint to SR-unaware appliance via +per-flow dynamic proxy + +Overview +-------- + +TBD + +CLI configuration +----------------- + +The following command instantiates a new End.AD.Flow segment that sends +the inner packets on interface ``IFACE-OUT`` towards an appliance at +address ``S-ADDR`` and restores the encapsulation headers of the packets +coming back on interface ``IFACE-IN``. + +:: + + sr localsid address SID behavior end.ad.flow nh S-ADDR oif IFACE-OUT iif IFACE-IN + +For example, the below command configures the SID ``1::A1`` with an +End.AD.Flow function for sending traffic on interface +``GigabitEthernet0/8/0`` to the appliance at address ``A1::``, and +receiving it back on interface ``GigabitEthernet0/9/0``. + +:: + + sr localsid address 1::A1 behavior end.ad.flow nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 diff --git a/src/plugins/srv6-ad/ad_plugin_doc.md b/src/plugins/srv6-ad/ad_plugin_doc.md deleted file mode 100644 index 993eeb63589..00000000000 --- a/src/plugins/srv6-ad/ad_plugin_doc.md +++ /dev/null @@ -1,73 +0,0 @@ -# SRv6 endpoint to SR-unaware appliance via dynamic proxy (End.AD) {#srv6_ad_plugin_doc} - -## Overview - -The dynamic proxy is an improvement over the static proxy (@ref srv6_as_plugin_doc) -that dynamically learns the SR information before removing it from the incoming -traffic. The same information can then be re-attached to the traffic returning -from the SF. As opposed to the static SR proxy, no CACHE information needs to be -configured. Instead, the dynamic SR proxy relies on a local caching mechanism on -the node instantiating this segment. Therefore, a dynamic proxy segment cannot -be the last segment in an SR SC policy. A different SR behavior should thus be -used if the SF is meant to be the final destination of an SR SC policy. - -Upon receiving a packet whose active segment matches a dynamic SR proxy -function, the proxy node pops the top MPLS label or applies the SRv6 End -behavior, then compares the updated SR information with the cache entry for the -current segment. If the cache is empty or different, it is updated with the new -SR information. The SR information is then removed and the inner packet is sent -towards the SF. - -The cache entry is not mapped to any particular packet, but instead to an SR SC -policy identified by the receiving interface (IFACE-IN). Any non-link-local IP -packet or non-local Ethernet frame received on that interface will be -re-encapsulated with the cached headers as described in @ref srv6_as_plugin_doc. The -SF may thus drop, modify or generate new packets without affecting the proxy. - -For more information, please see -[draft-xuclad-spring-sr-service-chaining](https://datatracker.ietf.org/doc/draft-xuclad-spring-sr-service-chaining/). - -## CLI configuration - -The following command instantiates a new End.AD segment that sends the inner -packets on interface `IFACE-OUT` towards an appliance at address `S-ADDR` and -restores the encapsulation headers of the packets coming back on interface -`IFACE-IN`. - -``` -sr localsid address SID behavior end.ad nh S-ADDR oif IFACE-OUT iif IFACE-IN -``` - -For example, the below command configures the SID `1::A1` with an End.AD -function for sending traffic on interface `GigabitEthernet0/8/0` to the -appliance at address `A1::`, and receiving it back on interface -`GigabitEthernet0/9/0`. - -``` -sr localsid address 1::A1 behavior end.ad nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 -``` - -## Pseudocode - -The dynamic proxy SRv6 pseudocode is obtained by inserting the following -instructions between lines 1 and 2 of the static proxy SRv6 pseudocode. - -``` -IF NH=SRH & SL > 0 THEN - Decrement SL and update the IPv6 DA with SRH[SL] - IF C(IFACE-IN) different from IPv6 encaps THEN ;; Ref1 - Copy the IPv6 encaps into C(IFACE-IN) ;; Ref2 -ELSE - Drop the packet -``` - -**Ref1:** "IPv6 encaps" represents the IPv6 header and any attached extension -header. - -**Ref2:** C(IFACE-IN) represents the cache entry associated to the dynamic SR proxy -segment. It is identified with IFACE-IN in order to efficiently retrieve the -right SR information when a packet arrives on this interface. - -In addition, the inbound policy should check that C(IFACE-IN) has been defined -before attempting to restore the IPv6 encapsulation, and drop the packet -otherwise. diff --git a/src/plugins/srv6-ad/ad_plugin_doc.rst b/src/plugins/srv6-ad/ad_plugin_doc.rst new file mode 100644 index 00000000000..cfb6cea7a15 --- /dev/null +++ b/src/plugins/srv6-ad/ad_plugin_doc.rst @@ -0,0 +1,86 @@ +.. _srv6_ad_plugin_doc: + +SRv6 dynamic proxy +================== + +SRv6 endpoint to SR-unaware appliance via dynamic proxy (End.AD) +---------------------------------------------------------------- + +Overview +~~~~~~~~ + +The dynamic proxy is an improvement over the static proxy (@ref +srv6_as_plugin_doc) that dynamically learns the SR information before +removing it from the incoming traffic. The same information can then be +re-attached to the traffic returning from the SF. As opposed to the +static SR proxy, no CACHE information needs to be configured. Instead, +the dynamic SR proxy relies on a local caching mechanism on the node +instantiating this segment. Therefore, a dynamic proxy segment cannot be +the last segment in an SR SC policy. A different SR behavior should thus +be used if the SF is meant to be the final destination of an SR SC +policy. + +Upon receiving a packet whose active segment matches a dynamic SR proxy +function, the proxy node pops the top MPLS label or applies the SRv6 End +behavior, then compares the updated SR information with the cache entry +for the current segment. If the cache is empty or different, it is +updated with the new SR information. The SR information is then removed +and the inner packet is sent towards the SF. + +The cache entry is not mapped to any particular packet, but instead to +an SR SC policy identified by the receiving interface (IFACE-IN). Any +non-link-local IP packet or non-local Ethernet frame received on that +interface will be re-encapsulated with the cached headers as described +in @ref srv6_as_plugin_doc. The SF may thus drop, modify or generate new +packets without affecting the proxy. + +For more information, please see +`draft-xuclad-spring-sr-service-chaining <https://datatracker.ietf.org/doc/draft-xuclad-spring-sr-service-chaining/>`__. + +CLI configuration +~~~~~~~~~~~~~~~~~ + +The following command instantiates a new End.AD segment that sends the +inner packets on interface ``IFACE-OUT`` towards an appliance at address +``S-ADDR`` and restores the encapsulation headers of the packets coming +back on interface ``IFACE-IN``. + +:: + + sr localsid address SID behavior end.ad nh S-ADDR oif IFACE-OUT iif IFACE-IN + +For example, the below command configures the SID ``1::A1`` with an +End.AD function for sending traffic on interface +``GigabitEthernet0/8/0`` to the appliance at address ``A1::``, and +receiving it back on interface ``GigabitEthernet0/9/0``. + +:: + + sr localsid address 1::A1 behavior end.ad nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 + +Pseudocode +~~~~~~~~~~ + +The dynamic proxy SRv6 pseudocode is obtained by inserting the following +instructions between lines 1 and 2 of the static proxy SRv6 pseudocode. + +:: + + IF NH=SRH & SL > 0 THEN + Decrement SL and update the IPv6 DA with SRH[SL] + IF C(IFACE-IN) different from IPv6 encaps THEN ;; Ref1 + Copy the IPv6 encaps into C(IFACE-IN) ;; Ref2 + ELSE + Drop the packet + +**Ref1:** “IPv6 encaps” represents the IPv6 header and any attached +extension header. + +**Ref2:** C(IFACE-IN) represents the cache entry associated to the +dynamic SR proxy segment. It is identified with IFACE-IN in order to +efficiently retrieve the right SR information when a packet arrives on +this interface. + +In addition, the inbound policy should check that C(IFACE-IN) has been +defined before attempting to restore the IPv6 encapsulation, and drop +the packet otherwise. diff --git a/src/plugins/srv6-am/am_plugin_doc.md b/src/plugins/srv6-am/am_plugin_doc.md deleted file mode 100644 index 11aad855408..00000000000 --- a/src/plugins/srv6-am/am_plugin_doc.md +++ /dev/null @@ -1,100 +0,0 @@ -# SRv6 endpoint to SR-unaware appliance via masquerading (End.AM) {#srv6_am_plugin_doc} - -The masquerading proxy is an SR endpoint behavior for processing SRv6 traffic on -behalf of an SR-unaware SF. This proxy thus receives SR traffic that is formed -of an IPv6 header and an SRH on top of an inner payload. The masquerading -behavior is independent from the inner payload type. Hence, the inner payload -can be of any type but it is usually expected to be a transport layer packet, -such as TCP or UDP. - -A masquerading SR proxy segment is associated with the following mandatory -parameters: - -- S-ADDR: Ethernet or IPv6 address of the SF -- IFACE-OUT: Local interface for sending traffic towards the SF -- IFACE-IN: Local interface receiving the traffic coming back from the SF - -A masquerading SR proxy segment is thus defined for a specific SF and bound to a -pair of directed interfaces or sub-interfaces on the proxy. As opposed to the -static and dynamic SR proxies, a masquerading segment can be present at the same -time in any number of SR SC policies and the same interfaces can be bound to -multiple masquerading proxy segments. The only restriction is that a -masquerading proxy segment cannot be the last segment in an SR SC policy. - -The first part of the masquerading behavior is triggered when the proxy node -receives an IPv6 packet whose Destination Address matches a masquerading proxy -segment. The proxy inspects the IPv6 extension headers and substitutes the -Destination Address with the last segment in the SRH attached to the IPv6 -header, which represents the final destination of the IPv6 packet. The packet is -then sent out towards the SF. - -The SF receives an IPv6 packet whose source and destination addresses are -respectively the original source and final destination. It does not attempt to -inspect the SRH, as RFC8200 specifies that routing extension headers are not -examined or processed by transit nodes. Instead, the SF simply forwards the -packet based on its current Destination Address. In this scenario, we assume -that the SF can only inspect, drop or perform limited changes to the packets. -For example, Intrusion Detection Systems, Deep Packet Inspectors and non-NAT -Firewalls are among the SFs that can be supported by a masquerading SR proxy. - -The second part of the masquerading behavior, also called de- masquerading, is -an inbound policy attached to the proxy interface receiving the traffic -returning from the SF, IFACE-IN. This policy inspects the incoming traffic and -triggers a regular SRv6 endpoint processing (End) on any IPv6 packet that -contains an SRH. This processing occurs before any lookup on the packet -Destination Address is performed and it is sufficient to restore the right -active segment as the Destination Address of the IPv6 packet. - -For more information, please see -[draft-xuclad-spring-sr-service-chaining](https://datatracker.ietf.org/doc/draft-xuclad-spring-sr-service-chaining/). - -## CLI configuration - -The following command instantiates a new End.AM segment that sends masqueraded -traffic on interface `IFACE-OUT` towards an appliance at address `S-ADDR` and -restores the active segment in the IPv6 header of the packets coming back on -interface `IFACE-IN`. - -``` -sr localsid address SID behavior end.am nh S-ADDR oif IFACE-OUT iif IFACE-IN -``` - -For example, the below command configures the SID `1::A1` with an End.AM -function for sending traffic on interface `GigabitEthernet0/8/0` to the -appliance at address `A1::`, and receiving it back on interface -`GigabitEthernet0/9/0`. - -``` -sr localsid address 1::A1 behavior end.am nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 -``` - -## Pseudocode - -### Masquerading - -Upon receiving a packet destined for S, where S is an IPv6 masquerading proxy -segment, a node N processes it as follows. - -``` -IF NH=SRH & SL > 0 THEN - Update the IPv6 DA with SRH[0] - Forward the packet on IFACE-OUT -ELSE - Drop the packet -``` - -### De-masquerading - -Upon receiving a non-link-local IPv6 packet on IFACE-IN, a node N processes it -as follows. - -``` -IF NH=SRH & SL > 0 THEN - Decrement SL - Update the IPv6 DA with SRH[SL] ;; Ref1 - Lookup DA in appropriate table and proceed accordingly -``` - -**Ref1:** This pseudocode can be augmented to support the Penultimate Segment -Popping (PSP) endpoint flavor. The exact pseudocode modification are provided in -[draft-filsfils-spring-srv6-network-programming](https://datatracker.ietf.org/doc/draft-filsfils-spring-srv6-network-programming/). diff --git a/src/plugins/srv6-am/am_plugin_doc.rst b/src/plugins/srv6-am/am_plugin_doc.rst new file mode 100644 index 00000000000..576379868fd --- /dev/null +++ b/src/plugins/srv6-am/am_plugin_doc.rst @@ -0,0 +1,116 @@ +.. _srv6_am_plugin_doc: + +SRv6 masquerading +================= + +SRv6 endpoint to SR-unaware appliance via masquerading (End.AM) +--------------------------------------------------------------- + +The masquerading proxy is an SR endpoint behavior for processing SRv6 +traffic on behalf of an SR-unaware SF. This proxy thus receives SR +traffic that is formed of an IPv6 header and an SRH on top of an inner +payload. The masquerading behavior is independent from the inner payload +type. Hence, the inner payload can be of any type but it is usually +expected to be a transport layer packet, such as TCP or UDP. + +A masquerading SR proxy segment is associated with the following +mandatory parameters: + +- S-ADDR: Ethernet or IPv6 address of the SF +- IFACE-OUT: Local interface for sending traffic towards the SF +- IFACE-IN: Local interface receiving the traffic coming back from the + SF + +A masquerading SR proxy segment is thus defined for a specific SF and +bound to a pair of directed interfaces or sub-interfaces on the proxy. +As opposed to the static and dynamic SR proxies, a masquerading segment +can be present at the same time in any number of SR SC policies and the +same interfaces can be bound to multiple masquerading proxy segments. +The only restriction is that a masquerading proxy segment cannot be the +last segment in an SR SC policy. + +The first part of the masquerading behavior is triggered when the proxy +node receives an IPv6 packet whose Destination Address matches a +masquerading proxy segment. The proxy inspects the IPv6 extension +headers and substitutes the Destination Address with the last segment in +the SRH attached to the IPv6 header, which represents the final +destination of the IPv6 packet. The packet is then sent out towards the +SF. + +The SF receives an IPv6 packet whose source and destination addresses +are respectively the original source and final destination. It does not +attempt to inspect the SRH, as RFC8200 specifies that routing extension +headers are not examined or processed by transit nodes. Instead, the SF +simply forwards the packet based on its current Destination Address. In +this scenario, we assume that the SF can only inspect, drop or perform +limited changes to the packets. For example, Intrusion Detection +Systems, Deep Packet Inspectors and non-NAT Firewalls are among the SFs +that can be supported by a masquerading SR proxy. + +The second part of the masquerading behavior, also called de- +masquerading, is an inbound policy attached to the proxy interface +receiving the traffic returning from the SF, IFACE-IN. This policy +inspects the incoming traffic and triggers a regular SRv6 endpoint +processing (End) on any IPv6 packet that contains an SRH. This +processing occurs before any lookup on the packet Destination Address is +performed and it is sufficient to restore the right active segment as +the Destination Address of the IPv6 packet. + +For more information, please see +`draft-xuclad-spring-sr-service-chaining <https://datatracker.ietf.org/doc/draft-xuclad-spring-sr-service-chaining/>`__. + +CLI configuration +~~~~~~~~~~~~~~~~~ + +The following command instantiates a new End.AM segment that sends +masqueraded traffic on interface ``IFACE-OUT`` towards an appliance at +address ``S-ADDR`` and restores the active segment in the IPv6 header of +the packets coming back on interface ``IFACE-IN``. + +:: + + sr localsid address SID behavior end.am nh S-ADDR oif IFACE-OUT iif IFACE-IN + +For example, the below command configures the SID ``1::A1`` with an +End.AM function for sending traffic on interface +``GigabitEthernet0/8/0`` to the appliance at address ``A1::``, and +receiving it back on interface ``GigabitEthernet0/9/0``. + +:: + + sr localsid address 1::A1 behavior end.am nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 + +Pseudocode +~~~~~~~~~~ + +Masquerading +^^^^^^^^^^^^ + +Upon receiving a packet destined for S, where S is an IPv6 masquerading +proxy segment, a node N processes it as follows. + +:: + + IF NH=SRH & SL > 0 THEN + Update the IPv6 DA with SRH[0] + Forward the packet on IFACE-OUT + ELSE + Drop the packet + +De-masquerading +^^^^^^^^^^^^^^^ + +Upon receiving a non-link-local IPv6 packet on IFACE-IN, a node N +processes it as follows. + +:: + + IF NH=SRH & SL > 0 THEN + Decrement SL + Update the IPv6 DA with SRH[SL] ;; Ref1 + Lookup DA in appropriate table and proceed accordingly + +**Ref1:** This pseudocode can be augmented to support the Penultimate +Segment Popping (PSP) endpoint flavor. The exact pseudocode modification +are provided in +`draft-filsfils-spring-srv6-network-programming <https://datatracker.ietf.org/doc/draft-filsfils-spring-srv6-network-programming/>`__. diff --git a/src/plugins/srv6-as/as_plugin_doc.md b/src/plugins/srv6-as/as_plugin_doc.md deleted file mode 100644 index 6ef29274318..00000000000 --- a/src/plugins/srv6-as/as_plugin_doc.md +++ /dev/null @@ -1,152 +0,0 @@ -# SRv6 endpoint to SR-unaware appliance via static proxy (End.AS) {#srv6_as_plugin_doc} - -## Overview - -The static proxy is an SR endpoint behavior for processing SR-MPLS or SRv6 -encapsulated traffic on behalf of an SR-unaware SF. This proxy thus receives SR -traffic that is formed of an MPLS label stack or an IPv6 header on top of an -inner packet, which can be Ethernet, IPv4 or IPv6. - -A static SR proxy segment is associated with the following mandatory parameters: - -- INNER-TYPE: Inner packet type -- S-ADDR: Ethernet or IP address of the SF (only for inner type IPv4 and IPv6) -- IFACE-OUT: Local interface for sending traffic towards the SF -- IFACE-IN: Local interface receiving the traffic coming back from the SF -- CACHE: SR information to be attached on the traffic coming back from the SF, -including at least - * CACHE.SA: IPv6 source address (SRv6 only) - * CACHE.LIST: Segment list expressed as MPLS labels or IPv6 address - -A static SR proxy segment is thus defined for a specific SF, inner packet type -and cached SR information. It is also bound to a pair of directed interfaces on -the proxy. These may be both directions of a single interface, or opposite -directions of two different interfaces. The latter is recommended in case the SF -is to be used as part of a bi-directional SR SC policy. If the proxy and the SF -both support 802.1Q, IFACE-OUT and IFACE-IN can also represent sub-interfaces. - -The first part of this behavior is triggered when the proxy node receives a -packet whose active segment matches a segment associated with the static proxy -behavior. It removes the SR information from the packet then sends it on a -specific interface towards the associated SF. This SR information corresponds to -the full label stack for SR-MPLS or to the encapsulation IPv6 header with any -attached extension header in the case of SRv6. - -The second part is an inbound policy attached to the proxy interface receiving -the traffic returning from the SF, IFACE-IN. This policy attaches to the -incoming traffic the cached SR information associated with the SR proxy segment. -If the proxy segment uses the SR-MPLS data plane, CACHE contains a stack of -labels to be pushed on top the packets. With the SRv6 data plane, CACHE is -defined as a source address, an active segment and an optional SRH (tag, -segments left, segment list and metadata). The proxy encapsulates the packets -with an IPv6 header that has the source address, the active segment as -destination address and the SRH as a routing extension header. After the SR -information has been attached, the packets are forwarded according to the active -segment, which is represented by the top MPLS label or the IPv6 Destination -Address. - -In this scenario, there are no restrictions on the operations that can be -performed by the SF on the stream of packets. It may operate at all protocol -layers, terminate transport layer connections, generate new packets and initiate -transport layer connections. This behavior may also be used to integrate an -IPv4-only SF into an SRv6 policy. However, a static SR proxy segment can be used -in only one service chain at a time. As opposed to most other segment types, a -static SR proxy segment is bound to a unique list of segments, which represents -a directed SR SC policy. This is due to the cached SR information being defined -in the segment configuration. This limitation only prevents multiple segment -lists from using the same static SR proxy segment at the same time, but a single -segment list can be shared by any number of traffic flows. Besides, since the -returning traffic from the SF is re-classified based on the incoming interface, -an interface can be used as receiving interface (IFACE-IN) only for a single SR -proxy segment at a time. In the case of a bi-directional SR SC policy, a -different SR proxy segment and receiving interface are required for the return -direction. - -For more information, please see -[draft-xuclad-spring-sr-service-chaining](https://datatracker.ietf.org/doc/draft-xuclad-spring-sr-service-chaining/). - -## CLI configuration - -The following command instantiates a new End.AS segment that sends the inner -packets on interface `IFACE-OUT` towards an appliance at address `S-ADDR` and -restores the segment list ``<S1, S2, S3>`` with a source address `SRC-ADDR` on -the packets coming back on interface `IFACE-IN`. - -``` -sr localsid address SID behavior end.as nh S-ADDR oif IFACE-OUT iif IFACE-IN src SRC-ADDR next S1 next S2 next S3 -``` - -For example, the below command configures the SID `1::A1` with an End.AS -function for sending traffic on interface `GigabitEthernet0/8/0` to the -appliance at address `A1::`, and receiving it back on interface -`GigabitEthernet0/9/0`. - -``` -sr localsid address 1::A1 behavior end.as nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 src 1:: next 2::20 next 3::30 next 4::40 -``` - -## Pseudocode - -### Static proxy for inner type IPv4 - -Upon receiving an IPv6 packet destined for S, where S is an IPv6 static proxy -segment for IPv4 traffic, a node N does: - -``` -IF ENH == 4 THEN ;; Ref1 - Remove the (outer) IPv6 header and its extension headers - Forward the exposed packet on IFACE-OUT towards S-ADDR -ELSE - Drop the packet -``` - -**Ref1:** 4 refers to IPv4 encapsulation as defined by IANA allocation for Internet -Protocol Numbers. - -Upon receiving a non link-local IPv4 packet on IFACE-IN, a node N does: - -``` -Decrement TTL and update checksum -IF CACHE.SRH THEN ;; Ref2 - Push CACHE.SRH on top of the existing IPv4 header - Set NH value of the pushed SRH to 4 -Push outer IPv6 header with SA, DA and traffic class from CACHE -Set outer payload length and flow label -Set NH value to 43 if an SRH was added, or 4 otherwise -Lookup outer DA in appropriate table and proceed accordingly -``` - -**Ref2:** CACHE.SRH represents the SRH defined in CACHE, if any, for the static SR -proxy segment associated with IFACE-IN. - -### Static proxy for inner type IPv6 - -Upon receiving an IPv6 packet destined for S, where S is an IPv6 static proxy -segment for IPv6 traffic, a node N does: - -``` -IF ENH == 41 THEN ;; Ref1 - Remove the (outer) IPv6 header and its extension headers - Forward the exposed packet on IFACE-OUT towards S-ADDR -ELSE - Drop the packet -``` - -**Ref1:** 41 refers to IPv6 encapsulation as defined by IANA allocation for Internet -Protocol Numbers. - -Upon receiving a non-link-local IPv6 packet on IFACE-IN, a node N does: - -``` -Decrement Hop Limit -IF CACHE.SRH THEN ;; Ref2 - Push CACHE.SRH on top of the existing IPv6 header - Set NH value of the pushed SRH to 41 -Push outer IPv6 header with SA, DA and traffic class from CACHE -Set outer payload length and flow label -Set NH value to 43 if an SRH was added, or 41 otherwise -Lookup outer DA in appropriate table and proceed accordingly -``` - -**Ref2:** CACHE.SRH represents the SRH defined in CACHE, if any, for the static SR -proxy segment associated with IFACE-IN. diff --git a/src/plugins/srv6-as/as_plugin_doc.rst b/src/plugins/srv6-as/as_plugin_doc.rst new file mode 100644 index 00000000000..9fa7f8fc19e --- /dev/null +++ b/src/plugins/srv6-as/as_plugin_doc.rst @@ -0,0 +1,172 @@ +.. _srv6_as_plugin_doc: + +SRv6 static proxy +================= + +The document describes SRv6 endpoint to SR-unaware appliance via static +proxy (End.AS) + +Overview +-------- + +The static proxy is an SR endpoint behavior for processing SR-MPLS or +SRv6 encapsulated traffic on behalf of an SR-unaware SF. This proxy thus +receives SR traffic that is formed of an MPLS label stack or an IPv6 +header on top of an inner packet, which can be Ethernet, IPv4 or IPv6. + +A static SR proxy segment is associated with the following mandatory +parameters: + +- INNER-TYPE: Inner packet type +- S-ADDR: Ethernet or IP address of the SF (only for inner type IPv4 + and IPv6) +- IFACE-OUT: Local interface for sending traffic towards the SF +- IFACE-IN: Local interface receiving the traffic coming back from the + SF +- CACHE: SR information to be attached on the traffic coming back from + the SF, including at least + + - CACHE.SA: IPv6 source address (SRv6 only) + - CACHE.LIST: Segment list expressed as MPLS labels or IPv6 address + +A static SR proxy segment is thus defined for a specific SF, inner +packet type and cached SR information. It is also bound to a pair of +directed interfaces on the proxy. These may be both directions of a +single interface, or opposite directions of two different interfaces. +The latter is recommended in case the SF is to be used as part of a +bi-directional SR SC policy. If the proxy and the SF both support +802.1Q, IFACE-OUT and IFACE-IN can also represent sub-interfaces. + +The first part of this behavior is triggered when the proxy node +receives a packet whose active segment matches a segment associated with +the static proxy behavior. It removes the SR information from the packet +then sends it on a specific interface towards the associated SF. This SR +information corresponds to the full label stack for SR-MPLS or to the +encapsulation IPv6 header with any attached extension header in the case +of SRv6. + +The second part is an inbound policy attached to the proxy interface +receiving the traffic returning from the SF, IFACE-IN. This policy +attaches to the incoming traffic the cached SR information associated +with the SR proxy segment. If the proxy segment uses the SR-MPLS data +plane, CACHE contains a stack of labels to be pushed on top the packets. +With the SRv6 data plane, CACHE is defined as a source address, an +active segment and an optional SRH (tag, segments left, segment list and +metadata). The proxy encapsulates the packets with an IPv6 header that +has the source address, the active segment as destination address and +the SRH as a routing extension header. After the SR information has been +attached, the packets are forwarded according to the active segment, +which is represented by the top MPLS label or the IPv6 Destination +Address. + +In this scenario, there are no restrictions on the operations that can +be performed by the SF on the stream of packets. It may operate at all +protocol layers, terminate transport layer connections, generate new +packets and initiate transport layer connections. This behavior may also +be used to integrate an IPv4-only SF into an SRv6 policy. However, a +static SR proxy segment can be used in only one service chain at a time. +As opposed to most other segment types, a static SR proxy segment is +bound to a unique list of segments, which represents a directed SR SC +policy. This is due to the cached SR information being defined in the +segment configuration. This limitation only prevents multiple segment +lists from using the same static SR proxy segment at the same time, but +a single segment list can be shared by any number of traffic flows. +Besides, since the returning traffic from the SF is re-classified based +on the incoming interface, an interface can be used as receiving +interface (IFACE-IN) only for a single SR proxy segment at a time. In +the case of a bi-directional SR SC policy, a different SR proxy segment +and receiving interface are required for the return direction. + +For more information, please see +`draft-xuclad-spring-sr-service-chaining <https://datatracker.ietf.org/doc/draft-xuclad-spring-sr-service-chaining/>`__. + +CLI configuration +----------------- + +The following command instantiates a new End.AS segment that sends the +inner packets on interface ``IFACE-OUT`` towards an appliance at address +``S-ADDR`` and restores the segment list ``<S1, S2, S3>`` with a source +address ``SRC-ADDR`` on the packets coming back on interface +``IFACE-IN``. + +:: + + sr localsid address SID behavior end.as nh S-ADDR oif IFACE-OUT iif IFACE-IN src SRC-ADDR next S1 next S2 next S3 + +For example, the below command configures the SID ``1::A1`` with an +End.AS function for sending traffic on interface +``GigabitEthernet0/8/0`` to the appliance at address ``A1::``, and +receiving it back on interface ``GigabitEthernet0/9/0``. + +:: + + sr localsid address 1::A1 behavior end.as nh A1:: oif GigabitEthernet0/8/0 iif GigabitEthernet0/9/0 src 1:: next 2::20 next 3::30 next 4::40 + +Pseudocode +---------- + +Static proxy for inner type IPv4 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Upon receiving an IPv6 packet destined for S, where S is an IPv6 static +proxy segment for IPv4 traffic, a node N does: + +:: + + IF ENH == 4 THEN ;; Ref1 + Remove the (outer) IPv6 header and its extension headers + Forward the exposed packet on IFACE-OUT towards S-ADDR + ELSE + Drop the packet + +**Ref1:** 4 refers to IPv4 encapsulation as defined by IANA allocation +for Internet Protocol Numbers. + +Upon receiving a non link-local IPv4 packet on IFACE-IN, a node N does: + +:: + + Decrement TTL and update checksum + IF CACHE.SRH THEN ;; Ref2 + Push CACHE.SRH on top of the existing IPv4 header + Set NH value of the pushed SRH to 4 + Push outer IPv6 header with SA, DA and traffic class from CACHE + Set outer payload length and flow label + Set NH value to 43 if an SRH was added, or 4 otherwise + Lookup outer DA in appropriate table and proceed accordingly + +**Ref2:** CACHE.SRH represents the SRH defined in CACHE, if any, for the +static SR proxy segment associated with IFACE-IN. + +Static proxy for inner type IPv6 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Upon receiving an IPv6 packet destined for S, where S is an IPv6 static +proxy segment for IPv6 traffic, a node N does: + +:: + + IF ENH == 41 THEN ;; Ref1 + Remove the (outer) IPv6 header and its extension headers + Forward the exposed packet on IFACE-OUT towards S-ADDR + ELSE + Drop the packet + +**Ref1:** 41 refers to IPv6 encapsulation as defined by IANA allocation +for Internet Protocol Numbers. + +Upon receiving a non-link-local IPv6 packet on IFACE-IN, a node N does: + +:: + + Decrement Hop Limit + IF CACHE.SRH THEN ;; Ref2 + Push CACHE.SRH on top of the existing IPv6 header + Set NH value of the pushed SRH to 41 + Push outer IPv6 header with SA, DA and traffic class from CACHE + Set outer payload length and flow label + Set NH value to 43 if an SRH was added, or 41 otherwise + Lookup outer DA in appropriate table and proceed accordingly + +**Ref2:** CACHE.SRH represents the SRH defined in CACHE, if any, for the +static SR proxy segment associated with IFACE-IN. diff --git a/src/plugins/srv6-mobile/extra/runner_doc.md b/src/plugins/srv6-mobile/extra/runner_doc.md deleted file mode 100644 index 64f06d77299..00000000000 --- a/src/plugins/srv6-mobile/extra/runner_doc.md +++ /dev/null @@ -1,105 +0,0 @@ -# What's `runner.py` doing? {#srv6_mobile_runner_doc} - -## Common configurations - -### VPP1 -``` -create host-interface name eth1 -set int ip addr host-eth1 A1::1/120 -set int state host-eth1 up -ip route add ::/0 via host-eth1 A1::2 -``` - - -### VPP2 - -``` -create host-interface name eth1 -set int ip addr host-eth1 A1::2/120 -create host-interface name eth2 -set int ip addr host-eth2 A2::1/120 -set int state host-eth1 up -set int state host-eth2 up -ip route add ::/0 via host-eth2 A2::2 -``` - - -### VPP3 - -``` -create host-interface name eth1 -set int ip addr host-eth1 A2::2/120 -create host-interface name eth2 -set int ip addr host-eth2 A3::1/120 -set int state host-eth1 up -set int state host-eth2 up -ip route add ::/0 via host-eth1 A2::1 -``` - -### VPP4 - -``` -create host-interface name eth1 -set int ip addr host-eth1 A3::2/120 -set int state host-eth1 up -ip route add ::/0 via host-eth1 A3::1 -``` - - -## Drop-in for GTP-U over IPv4 - -Drop-in mode is handy to test both GTP-U-to-SRv6 and SRv6-to-GTP-U functions at same time. Let's see what's happened when you run `test gtp4`: - - $ ./runner.py test gtp4 - - -Setting up a virtual interface of packet generator: - -#### VPP1 - -``` -create packet-generator interface pg0 -set int mac address pg0 aa:bb:cc:dd:ee:01 -set int ip addr pg0 172.16.0.1/30 -set ip arp pg0 172.16.0.2/30 aa:bb:cc:dd:ee:02 -``` - -#### VPP4 - -``` -create packet-generator interface pg0 -set int mac address pg0 aa:bb:cc:dd:ee:11 -set int ip addr pg0 1.0.0.2/30 -set ip arp pg0 1.0.0.1 aa:bb:cc:dd:ee:22 -``` - -SRv6 and IP routing settings: - -#### VPP1 - -``` -sr policy add bsid D4:: next D2:: next D3:: -sr policy add bsid D5:: behavior t.m.gtp4.d D4::/32 v6src_prefix C1::/64 nhtype ipv4 -sr steer l3 172.20.0.1/32 via bsid D5:: -``` - -#### VPP2 - -``` -sr localsid address D2:: behavior end -ip route add D3::/128 via host-eth2 A2::2 -``` - -#### VPP3 - -``` -sr localsid address D3:: behavior end -ip route add D4::/32 via host-eth2 A3::2 -``` - -#### VPP4 - -``` -sr localsid prefix D4::/32 behavior end.m.gtp4.e v4src_position 64 -ip route add 172.20.0.1/32 via pg0 1.0.0.1 -``` diff --git a/src/plugins/srv6-mobile/extra/runner_doc.rst b/src/plugins/srv6-mobile/extra/runner_doc.rst new file mode 100644 index 00000000000..b5be91cbfc8 --- /dev/null +++ b/src/plugins/srv6-mobile/extra/runner_doc.rst @@ -0,0 +1,135 @@ +.. _srv6_mobile_runner_doc: + +SRv6 Mobile Runner +================== + +What’s ``runner.py`` doing? + +Common configurations +--------------------- + +VPP1 +~~~~ + +:: + + create host-interface name eth1 + set int ip addr host-eth1 A1::1/120 + set int state host-eth1 up + ip route add ::/0 via host-eth1 A1::2 + +VPP2 +~~~~ + +:: + + create host-interface name eth1 + set int ip addr host-eth1 A1::2/120 + create host-interface name eth2 + set int ip addr host-eth2 A2::1/120 + set int state host-eth1 up + set int state host-eth2 up + ip route add ::/0 via host-eth2 A2::2 + +VPP3 +~~~~ + +:: + + create host-interface name eth1 + set int ip addr host-eth1 A2::2/120 + create host-interface name eth2 + set int ip addr host-eth2 A3::1/120 + set int state host-eth1 up + set int state host-eth2 up + ip route add ::/0 via host-eth1 A2::1 + +VPP4 +~~~~ + +:: + + create host-interface name eth1 + set int ip addr host-eth1 A3::2/120 + set int state host-eth1 up + ip route add ::/0 via host-eth1 A3::1 + +Drop-in for GTP-U over IPv4 +--------------------------- + +Drop-in mode is handy to test both GTP-U-to-SRv6 and SRv6-to-GTP-U +functions at same time. Let’s see what’s happened when you run +``test gtp4``: + +:: + + $ ./runner.py test gtp4 + +Setting up a virtual interface of packet generator: + +.. _vpp1-1: + +VPP1 +~~~~ + +:: + + create packet-generator interface pg0 + set int mac address pg0 aa:bb:cc:dd:ee:01 + set int ip addr pg0 172.16.0.1/30 + set ip arp pg0 172.16.0.2/30 aa:bb:cc:dd:ee:02 + +.. _vpp4-1: + +VPP4 +~~~~ + +:: + + create packet-generator interface pg0 + set int mac address pg0 aa:bb:cc:dd:ee:11 + set int ip addr pg0 1.0.0.2/30 + set ip arp pg0 1.0.0.1 aa:bb:cc:dd:ee:22 + +SRv6 and IP routing settings: + +.. _vpp1-2: + +VPP1 +~~~~ + +:: + + sr policy add bsid D4:: next D2:: next D3:: + sr policy add bsid D5:: behavior t.m.gtp4.d D4::/32 v6src_prefix C1::/64 nhtype ipv4 + sr steer l3 172.20.0.1/32 via bsid D5:: + +.. _vpp2-1: + +VPP2 +~~~~ + +:: + + sr localsid address D2:: behavior end + ip route add D3::/128 via host-eth2 A2::2 + +.. _vpp3-1: + +VPP3 +~~~~ + +:: + + sr localsid address D3:: behavior end + ip route add D4::/32 via host-eth2 A3::2 + +.. _vpp4-2: + +VPP4 +~~~~ + +:: + + sr localsid prefix D4::/32 behavior end.m.gtp4.e v4src_position 64 + ip route add 172.20.0.1/32 via pg0 1.0.0.1 diff --git a/src/plugins/srv6-mobile/mobile_plugin_doc.md b/src/plugins/srv6-mobile/mobile_plugin_doc.md deleted file mode 100644 index 3a44e795838..00000000000 --- a/src/plugins/srv6-mobile/mobile_plugin_doc.md +++ /dev/null @@ -1,201 +0,0 @@ -SRv6 Mobile User Plane Plugins {#srv6_mobile_plugin_doc} -======================== - -# Introduction - -This plugin module can provide the stateless mobile user plane protocols translation between GTP-U and SRv6. The plugin also provides FIB table lookup for an IPv4/IPv6 packet encapsulated in GTP-U. These plugin functions take advantage of SRv6 network programmability. - -[SRv6 Mobile User Plane](https://tools.ietf.org/html/draft-ietf-dmm-srv6-mobile-uplane) defines the user plane protocol using SRv6 -including following stateless translation functions: - -- **T.M.GTP4.D:** - GTP-U over UDP/IPv4 -> SRv6 -- **End.M.GTP4.E:** - SRv6 -> GTP-U over UDP/IPv4 -- **End.M.GTP6.D:** - GTP-U over UDP/IPv6 -> SRv6 -- **End.M.GTP6.E:** - SRv6 -> GTP-U over UDP/IPv6 - -These functions benefit user plane(overlay) to be able to utilize data plane(underlay) networks properly. And also it benefits data plane to be able to handle user plane in routing paradigm. - -In addition to the above functions, the plugin supports following functions: - -- **T.M.GTP4.DT{4|6|46}:** - FIB table lookup for IPv4/IP6 encapsulated in GTP-U over UDP/IPv4 -- **End.M.GTP6.DT{4|6|46}:** - FIB table lookup for IPv4/IP6 encapsulated in GTP-U over UDP/IPv6 - -Noted that the prefix of function names follow naming convention of SRv6 network programming. "T" means transit function, "End" means end function, "M" means Mobility specific function. The suffix "D" and "E" mean that "decapsulation" and "encapsulation" respectively. - - -# Implementation - -All SRv6 mobile functions are implemented as VPP plugin modules. The plugin modules leverage the sr_policy and sr_localsid mechanisms. - -# Configurations - -## GTP-U to SRv6 - -The GTP-U tunnel and flow identifiers of a receiving packet are mapped to a Segment Identifier(SID) of sending SRv6 packets. - -### IPv4 infrastructure case - -In case that **IPv4** networks are the infrastructure of GTP-U, T.M.GTP4.D function translates the receiving GTP-U packets to SRv6 packets. - -A T.M.GTP4.D function is associated with the following mandatory parameters: - -- SID: A SRv6 SID to represents the function -- DST-PREFIX: Prefix of remote SRv6 segment. The destination address or last SID of out packets consists of the prefix followed by dst IPv4 address, QFI and TEID of the receiving packets. -- SRC-PREFIX: Prefix for src address of sending packets. The src IPv6 address consists of the prefix followed by the src IPv4 address of the receiving packets. - -The following command instantiates a new T.M.GTP4.D function. - -``` -sr policy add bsid SID behavior t.m.gtp4.d DST-PREFIX v6src_prefix SRC-PREFIX [nhtype {ipv4|ipv6|non-ip}] -``` - -For example, the below command configures the SID 2001:db8::1 with `t.m.gtp4.d` behavior for translating receiving GTP-U over IPv4 packets to SRv6 packets with next-header type is IPv4. - -``` -sr policy add bsid 2001:db8::1 behavior t.m.gtp4.d D1::/32 v6src_prefix A1::/64 nhtype ipv4 -``` - -It should be interesting how a SRv6 BSID works to decapsulate the receiving GTP-U packets over IPv4 header. To utilize ```t.m.gtp4.d``` function, you need to configure some SR steering policy like: - -``` -sr steer l3 172.20.0.1/32 via bsid 2001:db8::1 -``` - -The above steering policy with the BSID of `t.m.gtp4.d` would work properly for the GTP-U packets destined to 172.20.0.1. - -If you have a SID(s) list of SR policy which the configured gtp4.d function to be applied, the SR Policy can be configured as following: - -``` -sr policy add bsid D1:: next A1:: next B1:: next C1:: -``` - -### IPv6 infrastructure case - -In case that GTP-U is deployed over **IPv6** infrastructure, you don't need to configure T.M.GTP4.D function and associated SR steering policy. Instead of that, you just need to configure a localsid of End.M.GTP6.D segment. - -An End.M.GTP6.D segment is associated with the following mandatory parameters: - -- SID-PREFIX: SRv6 SID prefix to represent the function. In this function, it should be the dst address of receiving GTP-U packets. -- DST-PREFIX: Prefix of remote SRv6 Segment. The destination address or last SID of output packets consists of the prefix followed by QFI and TEID of the receiving packets. - -The following command instantiates a new End.M.GTP6.D function. - -``` -sr localsid prefix SID-PREFIX behavior end.m.gtp6.d DST-PREFIX [nhtype {ipv4|ipv6|non-ip}] -``` -For example, the below command configures the SID prefix 2001:db8::/64 with `end.m.gtp6.d` behavior for translating receiving GTP-U over IPv6 packets which have IPv6 destination addresses within 2001:db8::/64 to SRv6 packets. The dst IPv6 address of the outgoing packets consists of D4::/64 followed by QFI and TEID. - -``` -sr localsid prefix 2001:db8::/64 behavior end.m.gtp6.d D4::/64 -``` - -In another case, the translated packets from GTP-U over IPv6 to SRv6 will be re-translated back to GTP-U, which is so called 'Drop-In' mode. - -In Drop-In mode, an additional IPv6 specific end segment is required, named End.M.GTP6.D.Di. It is because that unlike `end.m.gtp6.d`, it needs to preserve original IPv6 dst address as the last SID in the SRH. - -Regardless of that difference exists, the required configuration parameters are same as `end.m.gtp6.d`. - -The following command instantiates a new End.M.GTP6.D.Di function. - -``` -sr localsid prefix 2001:db8::/64 behavior end.m.gtp6.d.di D4::/64 -``` - - -## SRv6 to GTP-U - -The SRv6 Mobile functions on SRv6 to GTP-U direction are End.M.GTP4.E and End.M.GTP6.D. - -In this direction with GTP-U over IPv4 infrastructure, an End.M.GTP4.E segment is associated with the following mandatory parameters: - -- SID-PREFIX: SRv6 SID prefix to represent the function. -- V4SRC-ADDR-POSITION: Integer number indicates bit position where IPv4 src address embedded. - -The following command instantiates a new End.M.GTP4.E function. - -``` -sr localsid prefix SID-PREFIX behavior end.m.gtp4.e v4src_position V4SRC-ADDR-POSITION -``` - -For example, the below command configures the SID prefix 2001:db8::/32 with `end.m.gtp4.e` behavior for translating the receiving SRv6 packets to GTP-U packets encapsulated with UDP/IPv4 header. All the GTP-U tunnel and flow identifiers are extracted from the active SID in the receiving packets. The src IPv4 address of sending GTP-U packets is extracted from the configured bit position in the src IPv6 address. - -``` -sr localsid prefix 2001:db8::/32 behavior end.m.gtp4.e v4src_position 64 -``` - -In IPv6 infrastructure case, an End.M.GTP6.E segment is associated with the following mandatory parameters: - -- SID-PREFIX: SRv6 SID prefix to represent the function. - -The following command instantiates a new End.M.GTP6.E function. - -``` -sr localsid prefix SID-PREFIX behavior end.m.gtp6.e -``` - -For example, the below command configures the SID prefix 2001:db8::/64 with `end.m.gtp6.e` behavior for translating the receiving SRv6 packets to GTP-U packets encapsulated with UDP/IPv6 header. While the last SID indicates GTP-U dst IPv6 address, 32-bits GTP-U TEID and 6-bits QFI are extracted from the active SID in the receiving packets. - -``` -sr localsid prefix 2001:db8::/64 behavior end.m.gtp6.e -``` - -## FIB Table Lookup for Inner IPv4/IPv6 packet - -SRv6 Mobile functions of `t.m.gtp4.dt*` and `end.m.gtp6.dt*` support decapsulating outer IP/UDP/GTP-U headers and forwarding inner IP packet based on specific fib table. - -In case of the both outer and inner IP address families are IPv4, `t.m.gtp4.dt4` function supports GTP-U decapsulation and fib lookup for inner IPv4 with an associated steering policy and the following parameters: - -- SID: A SRv6 SID to represents the function -- FIB: fib-table number for inner IPv4 packet lookup and forwarding - -The following command instantiates a new T.M.GTP4.DT4 function. - -``` -sr policy add bsid SID behavior t.m.gtp4.dt4 fib-table FIB -``` - -For example, the below commands configure D5:: as the SID instantiates `t.m.gtp4.dt4` function. A steering policy for packets destine to 172.20.0.1 binds to the SID. - -``` -sr steer l3 172.20.0.1/32 via bsid D5:: -sr policy add bsid D5:: behavior t.m.gtp4.dt4 fib-table 0 -``` - -In addition, inner IPv6, or mix of IPv4 and IPv6 inner packet cases require the function to be configured with local-fib table. - -- LOCAL-FIB: fib-table number for lookup and forward GTP-U packet based on outer IP destination address - -This is inner IPv6 case specific. The reason is that GTP-U encapsulates link local IPv6 packet for NDP (Neighber Discovery Protocol). Outer GTP-U header should be kept until the packets reach to the node responsible for NDP handling. It is typically UPF(User Plane Function) node. - -The following command instantiate a new T.M.GTP4.DT6 function. - -``` -sr policy add bsid D5:: behavior t.m.gtp4.dt6 fib-table 0 local-fib-table LOCAL-FIB -``` - -Following example configures fib 0 for inner packet and fib 1 for outer GTP-U packet forwarding: - -``` -sr policy add bsid D5:: behavior t.m.gtp4.dt6 fib-table 0 local-fib-table 1 -``` - -If you need to suport both IPv4 and IPv6 inner packet lookup with just one SID, you can configure `t.m.gtp4.dt46` function: - -``` -sr policy add bsid D5:: behavior t.m.gtp4.dt46 fib-table 0 local-fib-table 1 -``` - -In case of GTP-U over IPv6 case, `end.m.gtp6.dt4`, `end.m.gtp6.dt6` and `end.m.gtp6.dt46` functions support inner IPv4, IPv6 and IPv4/IPv6 lookup and forwarding respectively. Specifiyng fib table for inner IP packet forwarding is required as same as GTP-U over IPv4 case, and local-fib table for inner IPv6 and IPv4/IPv6 cases as well. - -``` -sr localsid prefix D::/64 behavior end.m.gtp6.dt46 fib-table 0 local-fib-table 0 -``` - -To run some demo setup please refer to: @subpage srv6_mobile_runner_doc - diff --git a/src/plugins/srv6-mobile/mobile_plugin_doc.rst b/src/plugins/srv6-mobile/mobile_plugin_doc.rst new file mode 100644 index 00000000000..1aca3aaf229 --- /dev/null +++ b/src/plugins/srv6-mobile/mobile_plugin_doc.rst @@ -0,0 +1,278 @@ +.. _srv6_mobile_plugin_doc: + +SRv6 Mobile User Plane +====================== + +Introduction +------------ + +This plugin module can provide the stateless mobile user plane protocols +translation between GTP-U and SRv6. The plugin also provides FIB table +lookup for an IPv4/IPv6 packet encapsulated in GTP-U. These plugin +functions take advantage of SRv6 network programmability. + +`SRv6 Mobile User +Plane <https://tools.ietf.org/html/draft-ietf-dmm-srv6-mobile-uplane>`__ +defines the user plane protocol using SRv6 including following stateless +translation functions: + +- **T.M.GTP4.D:** GTP-U over UDP/IPv4 -> SRv6 +- **End.M.GTP4.E:** SRv6 -> GTP-U over UDP/IPv4 +- **End.M.GTP6.D:** GTP-U over UDP/IPv6 -> SRv6 +- **End.M.GTP6.E:** SRv6 -> GTP-U over UDP/IPv6 + +These functions benefit user plane(overlay) to be able to utilize data +plane(underlay) networks properly. And also it benefits data plane to be +able to handle user plane in routing paradigm. + +In addition to the above functions, the plugin supports following +functions: + +- **T.M.GTP4.DT{4|6|46}:** FIB table lookup for IPv4/IP6 encapsulated + in GTP-U over UDP/IPv4 +- **End.M.GTP6.DT{4|6|46}:** FIB table lookup for IPv4/IP6 encapsulated + in GTP-U over UDP/IPv6 + +Noted that the prefix of function names follow naming convention of SRv6 +network programming. “T” means transit function, “End” means end +function, “M” means Mobility specific function. The suffix “D” and “E” +mean that “decapsulation” and “encapsulation” respectively. + +Implementation +-------------- + +All SRv6 mobile functions are implemented as VPP plugin modules. The +plugin modules leverage the sr_policy and sr_localsid mechanisms. + +Configurations +-------------- + +GTP-U to SRv6 +~~~~~~~~~~~~~ + +The GTP-U tunnel and flow identifiers of a receiving packet are mapped +to a Segment Identifier(SID) of sending SRv6 packets. + +IPv4 infrastructure case +^^^^^^^^^^^^^^^^^^^^^^^^ + +In case that **IPv4** networks are the infrastructure of GTP-U, +T.M.GTP4.D function translates the receiving GTP-U packets to SRv6 +packets. + +A T.M.GTP4.D function is associated with the following mandatory +parameters: + +- SID: A SRv6 SID to represents the function +- DST-PREFIX: Prefix of remote SRv6 segment. The destination address or + last SID of out packets consists of the prefix followed by dst IPv4 + address, QFI and TEID of the receiving packets. +- SRC-PREFIX: Prefix for src address of sending packets. The src IPv6 + address consists of the prefix followed by the src IPv4 address of + the receiving packets. + +The following command instantiates a new T.M.GTP4.D function. + +:: + + sr policy add bsid SID behavior t.m.gtp4.d DST-PREFIX v6src_prefix SRC-PREFIX [nhtype {ipv4|ipv6|non-ip}] + +For example, the below command configures the SID 2001:db8::1 with +``t.m.gtp4.d`` behavior for translating receiving GTP-U over IPv4 +packets to SRv6 packets with next-header type is IPv4. + +:: + + sr policy add bsid 2001:db8::1 behavior t.m.gtp4.d D1::/32 v6src_prefix A1::/64 nhtype ipv4 + +It should be interesting how a SRv6 BSID works to decapsulate the +receiving GTP-U packets over IPv4 header. To utilize ``t.m.gtp4.d`` +function, you need to configure some SR steering policy like: + +:: + + sr steer l3 172.20.0.1/32 via bsid 2001:db8::1 + +The above steering policy with the BSID of ``t.m.gtp4.d`` would work +properly for the GTP-U packets destined to 172.20.0.1. + +If you have a SID(s) list of SR policy which the configured gtp4.d +function to be applied, the SR Policy can be configured as following: + +:: + + sr policy add bsid D1:: next A1:: next B1:: next C1:: + +IPv6 infrastructure case +^^^^^^^^^^^^^^^^^^^^^^^^ + +In case that GTP-U is deployed over **IPv6** infrastructure, you don’t +need to configure T.M.GTP4.D function and associated SR steering policy. +Instead of that, you just need to configure a localsid of End.M.GTP6.D +segment. + +An End.M.GTP6.D segment is associated with the following mandatory +parameters: + +- SID-PREFIX: SRv6 SID prefix to represent the function. In this + function, it should be the dst address of receiving GTP-U packets. +- DST-PREFIX: Prefix of remote SRv6 Segment. The destination address or + last SID of output packets consists of the prefix followed by QFI and + TEID of the receiving packets. + +The following command instantiates a new End.M.GTP6.D function. + +:: + + sr localsid prefix SID-PREFIX behavior end.m.gtp6.d DST-PREFIX [nhtype {ipv4|ipv6|non-ip}] + +For example, the below command configures the SID prefix 2001:db8::/64 +with ``end.m.gtp6.d`` behavior for translating receiving GTP-U over IPv6 +packets which have IPv6 destination addresses within 2001:db8::/64 to +SRv6 packets. The dst IPv6 address of the outgoing packets consists of +D4::/64 followed by QFI and TEID. + +:: + + sr localsid prefix 2001:db8::/64 behavior end.m.gtp6.d D4::/64 + +In another case, the translated packets from GTP-U over IPv6 to SRv6 +will be re-translated back to GTP-U, which is so called ‘Drop-In’ mode. + +In Drop-In mode, an additional IPv6 specific end segment is required, +named End.M.GTP6.D.Di. It is because that unlike ``end.m.gtp6.d``, it +needs to preserve original IPv6 dst address as the last SID in the SRH. + +Regardless of that difference exists, the required configuration +parameters are same as ``end.m.gtp6.d``. + +The following command instantiates a new End.M.GTP6.D.Di function. + +:: + + sr localsid prefix 2001:db8::/64 behavior end.m.gtp6.d.di D4::/64 + +SRv6 to GTP-U +~~~~~~~~~~~~~ + +The SRv6 Mobile functions on SRv6 to GTP-U direction are End.M.GTP4.E +and End.M.GTP6.D. + +In this direction with GTP-U over IPv4 infrastructure, an End.M.GTP4.E +segment is associated with the following mandatory parameters: + +- SID-PREFIX: SRv6 SID prefix to represent the function. +- V4SRC-ADDR-POSITION: Integer number indicates bit position where IPv4 + src address embedded. + +The following command instantiates a new End.M.GTP4.E function. + +:: + + sr localsid prefix SID-PREFIX behavior end.m.gtp4.e v4src_position V4SRC-ADDR-POSITION + +For example, the below command configures the SID prefix 2001:db8::/32 +with ``end.m.gtp4.e`` behavior for translating the receiving SRv6 +packets to GTP-U packets encapsulated with UDP/IPv4 header. All the +GTP-U tunnel and flow identifiers are extracted from the active SID in +the receiving packets. The src IPv4 address of sending GTP-U packets is +extracted from the configured bit position in the src IPv6 address. + +:: + + sr localsid prefix 2001:db8::/32 behavior end.m.gtp4.e v4src_position 64 + +In IPv6 infrastructure case, an End.M.GTP6.E segment is associated with +the following mandatory parameters: + +- SID-PREFIX: SRv6 SID prefix to represent the function. + +The following command instantiates a new End.M.GTP6.E function. + +:: + + sr localsid prefix SID-PREFIX behavior end.m.gtp6.e + +For example, the below command configures the SID prefix 2001:db8::/64 +with ``end.m.gtp6.e`` behavior for translating the receiving SRv6 +packets to GTP-U packets encapsulated with UDP/IPv6 header. While the +last SID indicates GTP-U dst IPv6 address, 32-bits GTP-U TEID and 6-bits +QFI are extracted from the active SID in the receiving packets. + +:: + + sr localsid prefix 2001:db8::/64 behavior end.m.gtp6.e + +FIB Table Lookup for Inner IPv4/IPv6 packet +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +SRv6 Mobile functions of ``t.m.gtp4.dt*`` and ``end.m.gtp6.dt*`` support +decapsulating outer IP/UDP/GTP-U headers and forwarding inner IP packet +based on specific fib table. + +In case of the both outer and inner IP address families are IPv4, +``t.m.gtp4.dt4`` function supports GTP-U decapsulation and fib lookup +for inner IPv4 with an associated steering policy and the following +parameters: + +- SID: A SRv6 SID to represents the function +- FIB: fib-table number for inner IPv4 packet lookup and forwarding + +The following command instantiates a new T.M.GTP4.DT4 function. + +:: + + sr policy add bsid SID behavior t.m.gtp4.dt4 fib-table FIB + +For example, the below commands configure D5:: as the SID instantiates +``t.m.gtp4.dt4`` function. A steering policy for packets destine to +172.20.0.1 binds to the SID. + +:: + + sr steer l3 172.20.0.1/32 via bsid D5:: + sr policy add bsid D5:: behavior t.m.gtp4.dt4 fib-table 0 + +In addition, inner IPv6, or mix of IPv4 and IPv6 inner packet cases +require the function to be configured with local-fib table. + +- LOCAL-FIB: fib-table number for lookup and forward GTP-U packet based + on outer IP destination address + +This is inner IPv6 case specific. The reason is that GTP-U encapsulates +link local IPv6 packet for NDP (Neighbor Discovery Protocol). Outer +GTP-U header should be kept until the packets reach to the node +responsible for NDP handling. It is typically UPF(User Plane Function) +node. + +The following command instantiate a new T.M.GTP4.DT6 function. + +:: + + sr policy add bsid D5:: behavior t.m.gtp4.dt6 fib-table 0 local-fib-table LOCAL-FIB + +Following example configures fib 0 for inner packet and fib 1 for outer +GTP-U packet forwarding: + +:: + + sr policy add bsid D5:: behavior t.m.gtp4.dt6 fib-table 0 local-fib-table 1 + +If you need to support both IPv4 and IPv6 inner packet lookup with just +one SID, you can configure ``t.m.gtp4.dt46`` function: + +:: + + sr policy add bsid D5:: behavior t.m.gtp4.dt46 fib-table 0 local-fib-table 1 + +In case of GTP-U over IPv6 case, ``end.m.gtp6.dt4``, ``end.m.gtp6.dt6`` +and ``end.m.gtp6.dt46`` functions support inner IPv4, IPv6 and IPv4/IPv6 +lookup and forwarding respectively. Specifying fib table for inner IP +packet forwarding is required as same as GTP-U over IPv4 case, and +local-fib table for inner IPv6 and IPv4/IPv6 cases as well. + +:: + + sr localsid prefix D::/64 behavior end.m.gtp6.dt46 fib-table 0 local-fib-table 0 + +To run some demo setup please refer to: :ref:`srv6_mobile_runner_doc` diff --git a/src/plugins/vmxnet3/README.md b/src/plugins/vmxnet3/README.md deleted file mode 100644 index 6e9fb194c94..00000000000 --- a/src/plugins/vmxnet3/README.md +++ /dev/null @@ -1,64 +0,0 @@ -# VMWARE vmxnet3 device driver plugin {#vmxnet3_doc} - -##Overview -This plugin provides native PCI driver support for VMWare vmxnet3. - -##Prerequisites - * This code is tested with vfio-pci driver installed with Ubuntu 18.04 which -has kernel version 4.15.0-33-generic. - - * This driver is tested with ESXi vSwitch version 6.5/6.7 for LRO/TSO support, VMware Workstation 15 Pro (no LRO/TSO), and VMware Fusion 11 Pro (no LRO/TSO) - - * Driver requires MSI-X interrupt support, which is not supported by -uio_pci_generic driver. So vfio-pci must be used. On systems without IOMMU, -vfio driver can still be used with 4.15.0-33-generic kernel (Ubuntu 18.04) which supports no-iommu mode. - -##Known issues - -* VLAN filter - -## Usage -### System setup - -1. load VFIO driver -``` -sudo modprobe vfio-pci -``` - -2. Make sure the interface is down -``` -sudo ifconfig <if-name> down -``` - -Steps 3 and 4 are optional. They can be accomplished by specifying the optional keyword "bind" when creating the vmxnet3 interface. - -3. (systems without IOMMU only) enable unsafe NOIOMMU mode -``` -echo Y | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode -``` - -4. Bind interface to vfio-pci -``` -sudo dpdk-devbind.py --bind vfio-pci 0b:00.0 -``` - -### Interface Creation -Interface can be dynamically created with following CLI, with or without the bind option. If step 3 and 4 were executed, bind can be omitted. -``` -create interface vmxnet3 0000:0b:00.0 bind -set int state vmxnet3-0/b/0/0 up -``` - -### Interface Deletion -Interface can be deleted with following CLI: -``` -delete interface vmxnet3 <if-name> -``` - -### Interface Statistics -Interface statistics can be displayed with `show hardware-interface <if-name>` -command. - -### Show Interface CLI -Interface and ring information can be obtained with -`show vmxnet3 [if-name] [desc]` diff --git a/src/plugins/vmxnet3/README.rst b/src/plugins/vmxnet3/README.rst new file mode 100644 index 00000000000..14430433c17 --- /dev/null +++ b/src/plugins/vmxnet3/README.rst @@ -0,0 +1,86 @@ +VMWARE vmxnet3 device driver +============================ + +##Overview This plugin provides native PCI driver support for VMWare +vmxnet3. + +##Prerequisites \* This code is tested with vfio-pci driver installed +with Ubuntu 18.04 which has kernel version 4.15.0-33-generic. + +- This driver is tested with ESXi vSwitch version 6.5/6.7 for LRO/TSO + support, VMware Workstation 15 Pro (no LRO/TSO), and VMware Fusion 11 + Pro (no LRO/TSO) + +- Driver requires MSI-X interrupt support, which is not supported by + uio_pci_generic driver. So vfio-pci must be used. On systems without + IOMMU, vfio driver can still be used with 4.15.0-33-generic kernel + (Ubuntu 18.04) which supports no-iommu mode. + +##Known issues + +- VLAN filter + +Usage +----- + +System setup +~~~~~~~~~~~~ + +1. load VFIO driver + +:: + + sudo modprobe vfio-pci + +2. Make sure the interface is down + +:: + + sudo ifconfig <if-name> down + +Steps 3 and 4 are optional. They can be accomplished by specifying the +optional keyword “bind” when creating the vmxnet3 interface. + +3. (systems without IOMMU only) enable unsafe NOIOMMU mode + +:: + + echo Y | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + +4. Bind interface to vfio-pci + +:: + + sudo dpdk-devbind.py --bind vfio-pci 0b:00.0 + +Interface Creation +~~~~~~~~~~~~~~~~~~ + +Interface can be dynamically created with following CLI, with or without +the bind option. If step 3 and 4 were executed, bind can be omitted. + +:: + + create interface vmxnet3 0000:0b:00.0 bind + set int state vmxnet3-0/b/0/0 up + +Interface Deletion +~~~~~~~~~~~~~~~~~~ + +Interface can be deleted with following CLI: + +:: + + delete interface vmxnet3 <if-name> + +Interface Statistics +~~~~~~~~~~~~~~~~~~~~ + +Interface statistics can be displayed with +``show hardware-interface <if-name>`` command. + +Show Interface CLI +~~~~~~~~~~~~~~~~~~ + +Interface and ring information can be obtained with +``show vmxnet3 [if-name] [desc]`` diff --git a/src/plugins/wireguard/README.md b/src/plugins/wireguard/README.md deleted file mode 100755 index 0b624b8104c..00000000000 --- a/src/plugins/wireguard/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# Wireguard vpp-plugin {#wireguard_plugin_doc} - -## Overview -This plugin is an implementation of [wireguard protocol](https://www.wireguard.com/) for VPP. It allows one to create secure VPN tunnels. -This implementation is based on [wireguard-openbsd](https://git.zx2c4.com/wireguard-openbsd/). - -## Crypto - -The crypto protocols: - -- blake2s [[Source]](https://github.com/BLAKE2/BLAKE2) - -OpenSSL: - -- curve25519 -- chachapoly1305 - -## Plugin usage example - -### Create wireguard interface - -``` -> vpp# wireguard create listen-port <port> private-key <priv_key> src <src_ip4> [generate-key] -> *wg_interface* -> vpp# set int state <wg_interface> up -> vpp# set int ip address <wg_interface> <wg_ip4> -``` - -### Add a peer configuration: -``` -> vpp# wireguard peer add <wg_interface> public-key <pub_key_other> endpoint <ip4_dst> allowed-ip <prefix> port <port_dst> persistent-keepalive [keepalive_interval] -> vpp# *peer_idx* -``` - -### Add routes for allowed-ip: -``` -> ip route add <prefix> via <wg_ip4> <wg_interface> -``` - -### Show config -``` -> vpp# show wireguard interface -> vpp# show wireguard peer -``` - -### Remove peer -``` -> vpp# wireguard peer remove <peer_idx> -``` - - -### Delete interface -``` -> vpp# wireguard delete <wg_interface> -``` - -## Main next steps for improving this implementation -1. Use all benefits of VPP-engine. -2. Add IPv6 support (currently only supports IPv4) -3. Add DoS protection as in original protocol (using cookie) diff --git a/src/plugins/wireguard/README.rst b/src/plugins/wireguard/README.rst new file mode 100644 index 00000000000..cb7a024fdf9 --- /dev/null +++ b/src/plugins/wireguard/README.rst @@ -0,0 +1,81 @@ +.. _wireguard_plugin_doc: + +Wireguard vpp-plugin +==================== + +Overview +-------- + +This plugin is an implementation of `wireguard +protocol <https://www.wireguard.com/>`__ for VPP. It allows one to +create secure VPN tunnels. This implementation is based on +`wireguard-openbsd <https://git.zx2c4.com/wireguard-openbsd/>`__. + +Crypto +------ + +The crypto protocols: + +- blake2s `[Source] <https://github.com/BLAKE2/BLAKE2>`__ + +OpenSSL: + +- curve25519 +- chachapoly1305 + +Plugin usage example +-------------------- + +Create wireguard interface +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + > vpp# wireguard create listen-port <port> private-key <priv_key> src <src_ip4> [generate-key] + > *wg_interface* + > vpp# set int state <wg_interface> up + > vpp# set int ip address <wg_interface> <wg_ip4> + +Add a peer configuration: +~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + > vpp# wireguard peer add <wg_interface> public-key <pub_key_other> endpoint <ip4_dst> allowed-ip <prefix> port <port_dst> persistent-keepalive [keepalive_interval] + > vpp# *peer_idx* + +Add routes for allowed-ip: +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + > ip route add <prefix> via <wg_ip4> <wg_interface> + +Show config +~~~~~~~~~~~ + +:: + + > vpp# show wireguard interface + > vpp# show wireguard peer + +Remove peer +~~~~~~~~~~~ + +:: + + > vpp# wireguard peer remove <peer_idx> + +Delete interface +~~~~~~~~~~~~~~~~ + +:: + + > vpp# wireguard delete <wg_interface> + +Main next steps for improving this implementation +------------------------------------------------- + +1. Use all benefits of VPP-engine. +2. Add IPv6 support (currently only supports IPv4) +3. Add DoS protection as in original protocol (using cookie) |