Add 'doc/' from commit '5925379b5fb30e548f1ba4e804af9865de8903ae'

git-subtree-dir: doc
git-subtree-mainline: bdd1dea5e21a00b6af1b0ded09e7d528485d5196
git-subtree-split: 5925379b5fb30e548f1ba4e804af9865de8903ae

1 files changed, 674 insertions, 0 deletions

diff --git a/doc/packet_builder_yaml.asciidoc b/doc/packet_builder_yaml.asciidoc
new file mode 100755
index 00000000..ed80358e
--- /dev/null
+++ b/doc/packet_builder_yaml.asciidoc
@@ -0,0 +1,674 @@
+Packet Builder Language
+=======================
+:author: hhaim
+:email: <hhaim@cisco.com>
+:revnumber: 0.04
+:quotes.++:
+:numbered:
+
+
+== change log
+
+include::trex_ga.asciidoc[]
+
+
+[options="header",cols="^1,^h,a"]
+|=================
+| Version |  name   |  meaning
+| 0.01    |  hhaim |
+- first version
+| 0.02    | hhaim
+|
+- change the bool fields to properties
+- add external/internal  property
+- add const property  ( instead cant_change)
+- change  TLV property - now learn the prev header
+- add choice of next protocol that is not base on a field ( TCP->IP->TCP)
+| 0.03    | ybrustin
+|
+- add MAC address regexp
+- add gui_representation class with data_type, form_type, combobox_values, data_type_regexp items to describe GUI view of field
+- rename choice attribute to value_based_next_header
+- fixed some typos
+| 0.04    | ybrustin
+|
+- change value_based_next_header, combobox_values (to be consistent with value_based_next_header) to dictionary
+- added value_based_next_class for options
+- move 'help' attribute to gui_representation
+- add link to headers.yaml (references at bottom of the page)
+
+|=================
+
+
+== A file format for GUI packet builder
+
+=== Introduction
+
+We would like a file that will be read by GUI and will give us the ability to build packets using GUI
+
+The format should be *YAML*
+
+
+=== High Level Requirement
+
+* Define a YAML object format for dynamic building of packets and a program that change various fields
+* Ability to *parse* back the same buffer that was created using this tool (reversibility)
+** Ability to load packet from a pcap file and parse it
+* Ability to save the packet to a pcap file
+* Ability to save the packet and program in JSON format (same JSON-RPC format)
+* Set a value for any field of any protocol
+* Vary packet fields across packets at run time e.g. changing IP/MAC addresses
+* Stack protocols in any arbitrary order define in YAML format
+
+=== Header that should be supported (first phase)
+
+==== L2
+
+* Ethernet
+* 802.3
+* LLC SNAP
+* VLAN (with QinQ) stack
+* MPLS stack
+
+==== L3
+
+* ARP
+* IPv4
+* IPv6 (4x header)
+* IP-in-IP a.k.a IP Tunnelling (6over4, 4over6, 4over4, 6over6)
+
+==== L4
+
+* TCP
+* UDP
+* ICMPv4
+* ICMPv6
+* IGMP
+
+==== L7  anchor:Payload[]
+
+* Any text based protocol (HTTP, SIP, RTSP, NNTP etc.)
+** random string
+** repeat string
+
+* Pattern Binary
+** repeat of value (e.g 0x55)
+** random
+** seq (1,2,3,3,4)
+** User Hex Dump editor
+
+
+=== YAML Format
+
+==== Header section
+
+.Default Types  anchor:Types[]
+[options="header",cols="1,2,3"]
+|=================
+| Field Name        |   meaning                                        | size in bits
+| bit               |  describe the header object e.g tcp              |  1
+| uint8             |  describe the header object e.g tcp              |  8
+| uint16            |  the name in the GUI                             |  16
+| uint32            |  sub fields of this header                       |  32
+| uint64            |  sub fields of this header                       |  64
+| other class type  | name of other class. for example, "c-mac-addr"; take fields from there, optionally overload them later  | The size taken from that class
+| Payload           |     xref:Payload[Payload]                        | total packet size - all header until now
+| vlen_t            | in case of varible size header this include the size to the end of varible size header see example xref:IpvOption[Ipv4Option] |total size of the object
+|=================
+
+
+.Default Data_Type anchor:Data_Type[]
+[options="header",cols="1,2"]
+|=================
+| Field Name               | meaning
+| none                     |  use Hex Editor as Types
+| ipv4_t                   |  4 decimals 0-255 each
+| mac_addr_t               |  ([0-9a-fA-F]\{2\}:)\{5\}[0-9a-fA-F]\{2\}
+| ipv4_mask_t              |  should match uint32 type
+| ipv6_t                   |  should have 16 bytes field size 8x16
+| ipv6_mask_t              |  should have 16 bytes field size 8x16
+| another header class     |  sub fields of this header
+| char_t                   | array of bytes , look into the array_size of cost string
+| var_char_t               | array based on a field value look into
+| regexp_t                 |  define a Java function that converts a reg exp string to a buffer see here  xref:GenRegExp[RegExp]
+|=================
+
+.Default Form_Type anchor:Form_Type[]
+[options="header",cols="1,3"]
+|=================
+| Field Name          |  meaning
+| none                |  simple editing field
+| combo_with_edit     |  combo box with predefined choices, can edit the field value manually
+| combo_without_edit  |  combo box with predefined choices, can [underline]#not# edit the field value manually
+| checkbox            |  toggle bits values, if item is array of bits, display several checkboxes per number of bits
+|=================
+
+
+.Default Gui_Representation anchor:Gui_Representation[]
+[options="header",cols="1,^1,5,^1,10"]
+|=================
+| Field Name         | value type      | meaning                                            | Link                      | Additional info
+| help               | string          | the name in the GUI                                |                           |
+| data_type          | string          | how to represent data                              | xref:Data_Type[Data_Type] | data_type could get data_type_regexp e.g data_type = "ipv4"; data_type = "regexp" data_type_regexp = "string that define regexp and Java function"
+| form_type          | string          | which editing form to use                          | xref:Form_Type[Form_Type] | for example for ip address use combobox with option to edit value manually or choose: key "localhost" value "127.0.0.1" etc.
+| combobox_values    | dictionary      | pairs of 'key - value' for combo_with/without_edit |                           |
+| data_type_regexp   | string          | in case it is reg_exp the name of the function     | xref:GenRegExp[GenRegExp] |
+|=================
+
+
+.Default Properties    anchor:Properties[]
+[options="header",cols="1,7"]
+|=================
+| Field Name             | meaning
+| ipv4_checksum          | auto calculates checksum on this header Ipv4 type
+| tcp_checksum           | calculate TCP checksum
+| udp_checksum           | calculate UDP checksum
+| ipv4_total_length      | calculate ipv4 total length this pkt_size = header  + reset of packet
+| tlv                    | TLV length of the header (inlcudes the prev field length) example ip-option, tcp-option
+| le                     | little endian. deault is big
+| const                  | const field for example the 4 version of ipv4 header - this GUI won't give option to change this field
+| external               | marks the header as an external header for the GUI. for example IPv4 is external header and mac-addr is internal header ( compose external header)
+|=================
+
+
+.Field_Type  anchor:Field_Type[]
+[options="header",cols="1,^1,30,^1,^1,30"]
+|=================
+| Field Name         | value type      |  meaning                                           | Default Value | Link | Example
+| class              | string          | describe the class type                            |   in case class is defined no need to have name and vise versa           |      | class : tcp
+| name               | string          | describe the instance name                         |   in case class is defined no need to have name and vise versa          |      | name : tcp
+| array_size         | integer         | how many objects of this type, default value is 1  |  1            |      | array_size : 6 in case of mac-addr
+| type               | string          | type, see  Types  define the size                  |   "uint8_t"   |  xref:Types[Types]   | type : "uint32_t" type : "mac_addr"
+| gui_representation | dictionary      | description of how to view/edit data in GUI        |           | xref:Gui_Representation[Gui_Representation] | xref:Gui_Representation_Example[Gui_Representation_Example]
+| default            | array/value     | default value in the packets , you can override value for subfields in parent see example
+| [0 ]x header size  |   |    xref:Overide_Subfields_Example[Overide_Subfields_Example]
+| properies          | array of string like masks
+| properies of this fields             |  []        | xref:Properties[Properties] | ["le","external"] , ["tlv","le","const"]
+| value_based_next_header              | dictionary |   define the next protocol based on a field value                    |  none         | xref:Value_Based_Next_Header[Value_Based_Next_Header]  |
+| value_based_next_class               | dictionary |   define the next class based on a field value (useful for options)  |  none         | xref:Value_Based_Next_Class[Value_Based_Next_Class]  |
+| next_headers       | string or type  |   a name of class that define the next or just an array                           |  "none"       | xref:Next_headers[Next_headers]  |
+| fields             | array           | array of Field_Type                                                               |   []          |        | fields : [  ]
+| offset             | integer/string  |   offset into the packet in bits, in case of auto add base of prev fields         |   "auto"      |        |
+| option             | string          |   a java code that define a way to calculate varible size                         |   "none"      |        |  |
+|=================
+
+
+.Field_Type  anchor:ConstHeadesClass[]
+[options="header",cols="^1,^10"]
+|=================
+| Field Name | value type
+| "root"     | the root pointer to the start of blocks L2/802.3 etc
+| "end"      | end TLV headers
+| "payload"   | the rest of the packets as buffer/string etc
+|=================
+
+
+.Next_headers anchor:Next_headers[]
+Example of Next_headers
+[source,python]
+----
+
+ - class : "next-example-t-1"
+   gui_representation:
+       help : "next-example-t-1"
+   next_headers : ["ipv4", "ipv6, "tcp"]
+
+# option 1 define in the header itself
+ - class : "tcp"
+   gui_representation:
+       help : "TCP header"
+   properies : ["external"]
+   next_headers : ["ipv4", "ipv6, "tcp"]
+   fields :
+        - name : "ver"
+
+# option 2 define throw a class
+ - class : "tcp"
+   gui_representation:
+       help : "TCP header"
+   properies : ["external"]
+   next_headers : "next-example-t-1" #
+   fields :
+        - name : "ver"
+----
+
+
+.Value_Based_Next_Header anchor:Value_Based_Next_Header[]
+Example of value_based_next_header
+[source,python]
+----
+        value_based_next_header:
+            0x0800: 'ipv4'# name of an external or internal class , the GUI should distinct betwean internal and external
+            0x0806: 'arp'
+            0x86DD: 'ipv6'
+            0x8100: 'vlan'
+            0x8847: 'mpls unicast'
+            default: 'payload' # if no match for any of above
+
+----
+
+
+.Generic RegExp Edit Field anchor:GenRegExp[]
+
+This will define a regexp that match for user input and how to converts it to buffer of bytes
+
+[source,python]
+----
+
+class MyClass : public RegExpBase {
+  public:
+
+
+  string get_reg_exp_string( ) {
+    return ((\d){1-3})[.]((\d){1-3})[.]((\d){1-3})[.]((\d){1-3}))
+  }
+
+  # in case of match
+  buffer get_buffer(){
+     g= [get_group()[1].to_int()*256,get_group()[1].to_int()]
+     # return list
+     return (g)
+  }
+
+}
+
+----
+
+
+
+==== Relations between object headers
+
+There would be a root object to point to possible starting headers
+
+
+[source,python]
+----
+
+-  class : "root"
+   gui_representation:
+       help  : "Root"
+   next_headers  : [ "ethernet", "llc", "_802-3"]
+----
+
+So in a way you could define a tree like this
+
+[source,python]
+----
+
+root -> L2 (  Ethernet   , 802.3 , LLC SNAP  )
+                |( by field )
+                |
+                  ------------------------------------- ( VLAN (with QinQ), MPLS , ipv4, ipv6,     ARP , ICMP )
+                                                                                  |   |               |     |
+                                                                                  |   ipv4/ipv6       -     -
+                                                                                  |     |
+                                                                                  |    |
+  [Possibility - Ethernet/802.3/LLC SNAP)                                          |    UDP/TCP/Pyload
+ Object                                                                            |     |
+  for each option there tree of all the option                                      --- -
+----
+
+
+==== Rules
+
+* The size of the header and offset is automatically defined in default by the order of the fields ( inc by type size multiply by array_size)
+* It can be overrided by offset field ( put offset in the object ) and then an more advanced field can be shown earlier in the GUI
+* The packet size is defined before the headers.  Header Should not be allowed to be added if the size + header size is bigger than packet size
+* "Payload" is predefined Fields that take the reset of the packet and user can edit it ( see xref:Payload[Payload] )
+* There would be a spare field in the Stream object so GUI could add more metadata for reconstructing the builder types
+  for example in this example Ethrenet/IP/TCP/IP/TCP  you can't extrac from buffer alone that Payload is IP/TCP only the builder known that in build time.
+* Ip total length need to keep the total_pkt_size - this ip header . this should work for internal header too.
+* When GUI add header ("external") the total size of this header should be calculated ( varible size should be given a default - ipv4)
+
+
+=== Examples
+
+
+==== TLV (Ip option) anchor:IpvOption[], value_based_next_class anchor:Value_Based_Next_Class[]
+
+
+IP-option see link:http://tools.ietf.org/html/rfc791[ip_option]
+
+0 : END
+
+1 : Length 1
+
+other : Byte : Length ( +first) |option
+
+
+
+[source,python]
+----
+
+ - class : "ip_option_131"
+   gui_representation:
+       help  : "ip_option"
+   fields :
+         - name : "length"    # tree with leaf of bits
+           gui_representation:
+               help : "length"
+           type : uint8
+           properties : ["tlv"] # the length include the prev field size (8 byte)
+
+         - name : "pointer"    # tree with leaf of bits
+           type : uint8
+
+         - name : "buffer"    # tree with leaf of bits
+           type : "tlv_reset"
+
+ - class : "default_ip4_option_tlv"
+   gui_representation:
+       help  : "ip_option"
+   fields :
+         - name : "length"    # tree with leaf of bits
+           gui_representation:
+               help : "length"
+           type : uint8
+           properties : "tlv" # the length include the prev field size (8 byte)
+
+         - name : "buffer"    # tree with leaf of bits
+           type : "vlen_t"
+
+
+ - class : "ip_option"
+   gui_representation:
+       help  : "ip_option"
+   type : uint8
+   default : [0x01]
+   value_based_next_class :
+       0x00  : "end"   # reserve name for ending the loop
+       0x01  : "ip_option"   # back to this header
+       0x131 : "ip_option_131"
+       0x0812: "gre"
+       default : "default_ip4_option_tlv"
+
+
+----
+
+* case of varible length field ip_option example
+
+
+
+==== Example TCP/IP
+
+
+[source,python]
+----
+
+ - class : "c-mac-addr"
+   type : "uint8"
+   array_size : 6
+   default : [0x00, 0x00, 0x01, 0x00, 0x00, 0x00]
+   gui_representation:
+      data_type  : "mac-addr_t" # format ([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}
+      help  : "Mac addrees"
+
+
+ - class : "ethernet"
+   gui_representation:
+       help : "Ethernet-L2"
+   properties: ['external']
+   fields :
+        - name : "Dst"
+          gui_representation:
+              help : "destination mac"
+          type : "c-mac-addr"
+
+        - name : "Src"
+          gui_representation:
+              help : "source mac"
+          type : "c-mac-addr"
+
+        - name: "Ethertype"
+          gui_representation:
+              help: "Ethertype"
+          type: "uint16"
+          default: [0x0800]
+          value_based_next_header :
+                0x0800  : "ipv4"
+                0x86DD  : "ipv6"
+                0x8100  : "vlan"
+                0x8847  : "mpls" #unicast
+                default : "payload"
+
+
+ - class : "ipv4"
+   gui_representation:
+       help : "Ipv4"
+   fields :
+        - name : "ver"
+          gui_representation:
+              help : "Version"
+          type : "bit"
+          array_size : 4
+          default : [0, 1, 0, 0]
+          properties : ["const"]
+
+         - name : "ihl"
+          type : "bit"
+          array_size : 4
+          default : [0, 1, 1, 1]
+          properties : ["ipv4_ihl"]
+          gui_representation:
+              help : "IHL"
+              form_type: "checkbox"
+
+          ..
+
+         - name : "hdr_chsum"
+          gui_representation:
+              help : "Header Checksum"
+          default : [0x00,0x00]
+          properties : ["ipv4_check_sum", "const"]
+
+         - name : "total_len"
+          gui_representation:
+              help : "Total Length"
+          default : [0x00,0x00]
+          properties : ["ipv4_total_len", "const"] # auto calculate total_size-offset_header
+
+         - name : "protocol"
+          type : uint8
+          default : 0x06
+          value_based_next_header : &ipv4_next_header
+               0x06  : "tcp"
+               0x11  : "udp"
+               0x29  : "ipv6"
+               0x2F  : "gre"
+               default : "payload"
+          gui_representation:
+              help : "IPv4 next Protocol"
+              form_type: "combo_without_edit"
+              combobox_values:
+                <<: *ipv4_next_header # take same choices as value_based_next_header
+
+         - name : "src_addr"
+          type : uint32
+          default : [16, 0, 0, 0]
+          gui_representation:
+              help : "Source Address"
+              data_type  : "ipv4" # reserve
+
+         - name : "dst_addr"
+          default : [48, 0, 0, 0]
+          type : uint32
+          gui_representation:
+              help : "Destination Address"
+              data_type  : "ipv4" # reserve
+              form_type : "combo_with_edit"
+              combobox_values:
+                    [127, 0, 0, 1]: 'localhost'
+                    [255, 255, 255, 255]: 'broadcast'
+
+
+ - class : "tcp"
+   gui_representation:
+       help : "TCP"
+   properties : ["external"]
+   fields :
+         - name : "src_port"
+          gui_representation:
+              help : "Source Port"
+          default : [0x30,0x00]
+          type : uint16
+
+         - name : "dest_port"
+          gui_representation:
+              help : "Source Port"
+          default : [0x30,0x00]
+          type : uint16
+
+         - name : "seq"
+          gui_representation:
+              help : "Seq Number"
+          type : uint32
+          default : [0x30,0x00,00,00]
+
+         - name : "ack"
+          gui_representation:
+              help : "Ack Number"
+          type : uint32
+          default : [0x30,0x00,00,00]
+
+        ...
+
+         - name : "flags"    # tree with leaf of bits
+          gui_representation:
+              help : "Ack Number"
+          type : uint8
+          default : [0x30]
+          fields :
+                 - name : "urg"
+                  help : "URG"
+                  type  : bit
+                  default : [0x0]
+
+                 - name : "ack"
+                  help : "ACK"
+                  type  : bit
+                  default : [0x1]
+                  ..
+
+         - name : "checksum"
+          gui_representation:
+              help : "TCP Checksum"
+          type : uint16
+          default : [0x00,0x00]
+          properties : ["tcp_checksum"] # auto calculate total_size-offset_header
+
+
+-  class : "root" # reserve
+   gui_representation:
+       help  : "Root"
+   next_headers : [ "ethrenet" ,"llc","_802-3"]
+---------------------------
+
+
+==== Overide subfields example anchor:Overide_Subfields_Example[]
+
+In this example parent class default value overrides default values of sub-fields ( 2 different mac-addr)
+
+[source,python]
+----
+
+ - class : "c-mac-addr"
+   type : "uint8"
+   array_size : 6
+   gui_representation:
+      help  : "Mac addrees"
+      data_type  : "mac-addr_t" # format ([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}
+   default : [0x00,0x00,0x01,0x00,0x00,0x00]
+
+
+ - class : "ethernet"
+   gui_representation:
+       help : "Ethernet-L2"
+   properties : ["external"]
+   default : [0x00,0x01,0x01,0x00,0x00,0x00, 0x00,0x02,0x02,0x00,0x00,0x00 ,0x08,00] # change the default of sub-fields . it is const size
+   fields :
+        - name : "Dst"
+          gui_representation:
+              help : "destination mac"
+          type : "c-mac-addr"
+
+        - name : "Src"
+          gui_representation:
+              help : "source mac"
+          type : "c-mac-addr"
+
+        - name : "ip_protocol"
+          type : "uint16_t"
+          default : [0x08,0x00]
+          value_based_next_header :
+                0x0800  : "ipv4"
+                0x86DD  : "ipv6"
+                0x8100  : "vlan"
+                0x8847  : "mpls unicast"
+                default : "payload"
+----
+
+==== Gui Representation example anchor:Gui_Representation_Example[]
+[underline]#In YAML:#
+[source,python]
+----
+  - name: 'Flags'
+    type: 'bit'
+    array_size: 3
+    gui_representation:
+        help: 'IPv4 Flags'
+        form_type: 'checkbox' # can check each bit
+
+
+  - name: 'dst_addr'
+    default: [48, 0, 0, 0]
+    type: uint32
+    gui_representation:
+        help: 'IPv4 Destination Address'
+        data_type: 'ipv4_t' # special representation case, show as 4 decimal numbers
+        form_type: 'combo_with_edit' # can choose from pre-defined values or edit manually
+        combobox_values:
+            [127, 0, 0, 1]: 'localhost'
+            [255, 255, 255, 255]: 'broadcast'
+
+
+  - name: 'protocol'
+    type: uint8
+    default: 0x06
+    value_based_next_header: &ipv4_next_header
+        0x06: 'tcp'
+        0x11: 'udp'
+        default : "payload"
+    gui_representation:
+        help: 'IPv4 Protocol Field'
+        form_type: 'combo_without_edit' # choose from supported protocols, no manual edit
+        combobox_values:
+            <<: *ipv4_next_header # take same choices as value_based_next_header
+----
+
+[underline]#In GUI:#
+
+checkbox for bits:
+
+image:images/checkbox.jpg[]
+
+editing in combo-box:
+
+image:images/combo_button_editing.jpg[]
+
+choosing from predefined values:
+
+image:images/combo_button_choosing.jpg[]
+
+==== Union base
+
+TBD
+
+
+
+=== Resource
+*  link:yaml/headers.yaml[headers.yaml]
+*  link:https://wireedit.com/[WireEdit]
+*  link:http://ostinato.org/[ostinato]
+*  link:http://www.slideshare.net/nlekh/ixiaexplorer[IxExplorer]
+
+