Packet Builder Language ======================= :author: hhaim :email: :revnumber: 0.04 :quotes.++: :numbered: == change log include::trex_ga.asciidoc[] [options="header",cols="^1,^h,a"] |================= | Version | name | meaning | 0.01 | hhaim | - first version | 0.02 | hhaim | - change the bool fields to properties - add external/internal property - add const property ( instead cant_change) - change TLV property - now learn the prev header - add choice of next protocol that is not base on a field ( TCP->IP->TCP) | 0.03 | ybrustin | - add MAC address regexp - add gui_representation class with data_type, form_type, combobox_values, data_type_regexp items to describe GUI view of field - rename choice attribute to value_based_next_header - fixed some typos | 0.04 | ybrustin | - change value_based_next_header, combobox_values (to be consistent with value_based_next_header) to dictionary - added value_based_next_class for options - move 'help' attribute to gui_representation - add link to headers.yaml (references at bottom of the page) |================= == A file format for GUI packet builder === Introduction We would like a file that will be read by GUI and will give us the ability to build packets using GUI The format should be *YAML* === High Level Requirement * Define a YAML object format for dynamic building of packets and a program that change various fields * Ability to *parse* back the same buffer that was created using this tool (reversibility) ** Ability to load packet from a pcap file and parse it * Ability to save the packet to a pcap file * Ability to save the packet and program in JSON format (same JSON-RPC format) * Set a value for any field of any protocol * Vary packet fields across packets at run time e.g. changing IP/MAC addresses * Stack protocols in any arbitrary order define in YAML format === Header that should be supported (first phase) ==== L2 * Ethernet * 802.3 * LLC SNAP * VLAN (with QinQ) stack * MPLS stack ==== L3 * ARP * IPv4 * IPv6 (4x header) * IP-in-IP a.k.a IP Tunnelling (6over4, 4over6, 4over4, 6over6) ==== L4 * TCP * UDP * ICMPv4 * ICMPv6 * IGMP ==== L7 anchor:Payload[] * Any text based protocol (HTTP, SIP, RTSP, NNTP etc.) ** random string ** repeat string * Pattern Binary ** repeat of value (e.g 0x55) ** random ** seq (1,2,3,3,4) ** User Hex Dump editor === YAML Format ==== Header section .Default Types anchor:Types[] [options="header",cols="1,2,3"] |================= | Field Name | meaning | size in bits | bit | describe the header object e.g tcp | 1 | uint8 | describe the header object e.g tcp | 8 | uint16 | the name in the GUI | 16 | uint32 | sub fields of this header | 32 | uint64 | sub fields of this header | 64 | other class type | name of other class. for example, "c-mac-addr"; take fields from there, optionally overload them later | The size taken from that class | Payload | xref:Payload[Payload] | total packet size - all header until now | vlen_t | in case of varible size header this include the size to the end of varible size header see example xref:IpvOption[Ipv4Option] |total size of the object |================= .Default Data_Type anchor:Data_Type[] [options="header",cols="1,2"] |================= | Field Name | meaning | none | use Hex Editor as Types | ipv4_t | 4 decimals 0-255 each | mac_addr_t | ([0-9a-fA-F]\{2\}:)\{5\}[0-9a-fA-F]\{2\} | ipv4_mask_t | should match uint32 type | ipv6_t | should have 16 bytes field size 8x16 | ipv6_mask_t | should have 16 bytes field size 8x16 | another header class | sub fields of this header | char_t | array of bytes , look into the array_size of cost string | var_char_t | array based on a field value look into | regexp_t | define a Java function that converts a reg exp string to a buffer see here xref:GenRegExp[RegExp] |================= .Default Form_Type anchor:Form_Type[] [options="header",cols="1,3"] |================= | Field Name | meaning | none | simple editing field | combo_with_edit | combo box with predefined choices, can edit the field value manually | combo_without_edit | combo box with predefined choices, can [underline]#not# edit the field value manually | checkbox | toggle bits values, if item is array of bits, display several checkboxes per number of bits |================= .Default Gui_Representation anchor:Gui_Representation[] [options="header",cols="1,^1,5,^1,10"] |================= | Field Name | value type | meaning | Link | Additional info | help | string | the name in the GUI | | | data_type | string | how to represent data | xref:Data_Type[Data_Type] | data_type could get data_type_regexp e.g data_type = "ipv4"; data_type = "regexp" data_type_regexp = "string that define regexp and Java function" | form_type | string | which editing form to use | xref:Form_Type[Form_Type] | for example for ip address use combobox with option to edit value manually or choose: key "localhost" value "127.0.0.1" etc. | combobox_values | dictionary | pairs of 'key - value' for combo_with/without_edit | | | data_type_regexp | string | in case it is reg_exp the name of the function | xref:GenRegExp[GenRegExp] | |================= .Default Properties anchor:Properties[] [options="header",cols="1,7"] |================= | Field Name | meaning | ipv4_checksum | auto calculates checksum on this header Ipv4 type | tcp_checksum | calculate TCP checksum | udp_checksum | calculate UDP checksum | ipv4_total_length | calculate ipv4 total length this pkt_size = header + reset of packet | tlv | TLV length of the header (inlcudes the prev field length) example ip-option, tcp-option | le | little endian. deault is big | const | const field for example the 4 version of ipv4 header - this GUI won't give option to change this field | external | marks the header as an external header for the GUI. for example IPv4 is external header and mac-addr is internal header ( compose external header) |================= .Field_Type anchor:Field_Type[] [options="header",cols="1,^1,30,^1,^1,30"] |================= | Field Name | value type | meaning | Default Value | Link | Example | class | string | describe the class type | in case class is defined no need to have name and vise versa | | class : tcp | name | string | describe the instance name | in case class is defined no need to have name and vise versa | | name : tcp | array_size | integer | how many objects of this type, default value is 1 | 1 | | array_size : 6 in case of mac-addr | type | string | type, see Types define the size | "uint8_t" | xref:Types[Types] | type : "uint32_t" type : "mac_addr" | gui_representation | dictionary | description of how to view/edit data in GUI | | xref:Gui_Representation[Gui_Representation] | xref:Gui_Representation_Example[Gui_Representation_Example] | default | array/value | default value in the packets , you can override value for subfields in parent see example | [0 ]x header size | | xref:Overide_Subfields_Example[Overide_Subfields_Example] | properies | array of string like masks | properies of this fields | [] | xref:Properties[Properties] | ["le","external"] , ["tlv","le","const"] | value_based_next_header | dictionary | define the next protocol based on a field value | none | xref:Value_Based_Next_Header[Value_Based_Next_Header] | | value_based_next_class | dictionary | define the next class based on a field value (useful for options) | none | xref:Value_Based_Next_Class[Value_Based_Next_Class] | | next_headers | string or type | a name of class that define the next or just an array | "none" | xref:Next_headers[Next_headers] | | fields | array | array of Field_Type | [] | | fields : [ ] | offset | integer/string | offset into the packet in bits, in case of auto add base of prev fields | "auto" | | | option | string | a java code that define a way to calculate varible size | "none" | | | |================= .Field_Type anchor:ConstHeadesClass[] [options="header",cols="^1,^10"] |================= | Field Name | value type | "root" | the root pointer to the start of blocks L2/802.3 etc | "end" | end TLV headers | "payload" | the rest of the packets as buffer/string etc |================= .Next_headers anchor:Next_headers[] Example of Next_headers [source,python] ---- - class : "next-example-t-1" gui_representation: help : "next-example-t-1" next_headers : ["ipv4", "ipv6, "tcp"] # option 1 define in the header itself - class : "tcp" gui_representation: help : "TCP header" properies : ["external"] next_headers : ["ipv4", "ipv6, "tcp"] fields : - name : "ver" # option 2 define throw a class - class : "tcp" gui_representation: help : "TCP header" properies : ["external"] next_headers : "next-example-t-1" # fields : - name : "ver" ---- .Value_Based_Next_Header anchor:Value_Based_Next_Header[] Example of value_based_next_header [source,python] ---- value_based_next_header: 0x0800: 'ipv4'# name of an external or internal class , the GUI should distinct betwean internal and external 0x0806: 'arp' 0x86DD: 'ipv6' 0x8100: 'vlan' 0x8847: 'mpls unicast' default: 'payload' # if no match for any of above ---- .Generic RegExp Edit Field anchor:GenRegExp[] This will define a regexp that match for user input and how to converts it to buffer of bytes [source,python] ---- class MyClass : public RegExpBase { public: string get_reg_exp_string( ) { return ((\d){1-3})[.]((\d){1-3})[.]((\d){1-3})[.]((\d){1-3})) } # in case of match buffer get_buffer(){ g= [get_group()[1].to_int()*256,get_group()[1].to_int()] # return list return (g) } } ---- ==== Relations between object headers There would be a root object to point to possible starting headers [source,python] ---- - class : "root" gui_representation: help : "Root" next_headers : [ "ethernet", "llc", "_802-3"] ---- So in a way you could define a tree like this [source,python] ---- root -> L2 ( Ethernet , 802.3 , LLC SNAP ) |( by field ) | ------------------------------------- ( VLAN (with QinQ), MPLS , ipv4, ipv6, ARP , ICMP ) | | | | | ipv4/ipv6 - - | | | | [Possibility - Ethernet/802.3/LLC SNAP) | UDP/TCP/Pyload Object | | for each option there tree of all the option --- - ---- ==== Rules * The size of the header and offset is automatically defined in default by the order of the fields ( inc by type size multiply by array_size) * It can be overrided by offset field ( put offset in the object ) and then an more advanced field can be shown earlier in the GUI * The packet size is defined before the headers. Header Should not be allowed to be added if the size + header size is bigger than packet size * "Payload" is predefined Fields that take the reset of the packet and user can edit it ( see xref:Payload[Payload] ) * There would be a spare field in the Stream object so GUI could add more metadata for reconstructing the builder types for example in this example Ethrenet/IP/TCP/IP/TCP you can't extrac from buffer alone that Payload is IP/TCP only the builder known that in build time. * Ip total length need to keep the total_pkt_size - this ip header . this should work for internal header too. * When GUI add header ("external") the total size of this header should be calculated ( varible size should be given a default - ipv4) === Examples ==== TLV (Ip option) anchor:IpvOption[], value_based_next_class anchor:Value_Based_Next_Class[] IP-option see link:http://tools.ietf.org/html/rfc791[ip_option] 0 : END 1 : Length 1 other : Byte : Length ( +first) |option [source,python] ---- - class : "ip_option_131" gui_representation: help : "ip_option" fields : - name : "length" # tree with leaf of bits gui_representation: help : "length" type : uint8 properties : ["tlv"] # the length include the prev field size (8 byte) - name : "pointer" # tree with leaf of bits type : uint8 - name : "buffer" # tree with leaf of bits type : "tlv_reset" - class : "default_ip4_option_tlv" gui_representation: help : "ip_option" fields : - name : "length" # tree with leaf of bits gui_representation: help : "length" type : uint8 properties : "tlv" # the length include the prev field size (8 byte) - name : "buffer" # tree with leaf of bits type : "vlen_t" - class : "ip_option" gui_representation: help : "ip_option" type : uint8 default : [0x01] value_based_next_class : 0x00 : "end" # reserve name for ending the loop 0x01 : "ip_option" # back to this header 0x131 : "ip_option_131" 0x0812: "gre" default : "default_ip4_option_tlv" ---- * case of varible length field ip_option example ==== Example TCP/IP [source,python] ---- - class : "c-mac-addr" type : "uint8" array_size : 6 default : [0x00, 0x00, 0x01, 0x00, 0x00, 0x00] gui_representation: data_type : "mac-addr_t" # format ([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2} help : "Mac addrees" - class : "ethernet" gui_representation: help : "Ethernet-L2" properties: ['external'] fields : - name : "Dst" gui_representation: help : "destination mac" type : "c-mac-addr" - name : "Src" gui_representation: help : "source mac" type : "c-mac-addr" - name: "Ethertype" gui_representation: help: "Ethertype" type: "uint16" default: [0x0800] value_based_next_header : 0x0800 : "ipv4" 0x86DD : "ipv6" 0x8100 : "vlan" 0x8847 : "mpls" #unicast default : "payload" - class : "ipv4" gui_representation: help : "Ipv4" fields : - name : "ver" gui_representation: help : "Version" type : "bit" array_size : 4 default : [0, 1, 0, 0] properties : ["const"] - name : "ihl" type : "bit" array_size : 4 default : [0, 1, 1, 1] properties : ["ipv4_ihl"] gui_representation: help : "IHL" form_type: "checkbox" .. - name : "hdr_chsum" gui_representation: help : "Header Checksum" default : [0x00,0x00] properties : ["ipv4_check_sum", "const"] - name : "total_len" gui_representation: help : "Total Length" default : [0x00,0x00] properties : ["ipv4_total_len", "const"] # auto calculate total_size-offset_header - name : "protocol" type : uint8 default : 0x06 value_based_next_header : &ipv4_next_header 0x06 : "tcp" 0x11 : "udp" 0x29 : "ipv6" 0x2F : "gre" default : "payload" gui_representation: help : "IPv4 next Protocol" form_type: "combo_without_edit" combobox_values: <<: *ipv4_next_header # take same choices as value_based_next_header - name : "src_addr" type : uint32 default : [16, 0, 0, 0] gui_representation: help : "Source Address" data_type : "ipv4" # reserve - name : "dst_addr" default : [48, 0, 0, 0] type : uint32 gui_representation: help : "Destination Address" data_type : "ipv4" # reserve form_type : "combo_with_edit" combobox_values: [127, 0, 0, 1]: 'localhost' [255, 255, 255, 255]: 'broadcast' - class : "tcp" gui_representation: help : "TCP" properties : ["external"] fields : - name : "src_port" gui_representation: help : "Source Port" default : [0x30,0x00] type : uint16 - name : "dest_port" gui_representation: help : "Source Port" default : [0x30,0x00] type : uint16 - name : "seq" gui_representation: help : "Seq Number" type : uint32 default : [0x30,0x00,00,00] - name : "ack" gui_representation: help : "Ack Number" type : uint32 default : [0x30,0x00,00,00] ... - name : "flags" # tree with leaf of bits gui_representation: help : "Ack Number" type : uint8 default : [0x30] fields : - name : "urg" help : "URG" type : bit default : [0x0] - name : "ack" help : "ACK" type : bit default : [0x1] .. - name : "checksum" gui_representation: help : "TCP Checksum" type : uint16 default : [0x00,0x00] properties : ["tcp_checksum"] # auto calculate total_size-offset_header - class : "root" # reserve gui_representation: help : "Root" next_headers : [ "ethrenet" ,"llc","_802-3"] --------------------------- ==== Overide subfields example anchor:Overide_Subfields_Example[] In this example parent class default value overrides default values of sub-fields ( 2 different mac-addr) [source,python] ---- - class : "c-mac-addr" type : "uint8" array_size : 6 gui_representation: help : "Mac addrees" data_type : "mac-addr_t" # format ([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2} default : [0x00,0x00,0x01,0x00,0x00,0x00] - class : "ethernet" gui_representation: help : "Ethernet-L2" properties : ["external"] default : [0x00,0x01,0x01,0x00,0x00,0x00, 0x00,0x02,0x02,0x00,0x00,0x00 ,0x08,00] # change the default of sub-fields . it is const size fields : - name : "Dst" gui_representation: help : "destination mac" type : "c-mac-addr" - name : "Src" gui_representation: help : "source mac" type : "c-mac-addr" - name : "ip_protocol" type : "uint16_t" default : [0x08,0x00] value_based_next_header : 0x0800 : "ipv4" 0x86DD : "ipv6" 0x8100 : "vlan" 0x8847 : "mpls unicast" default : "payload" ---- ==== Gui Representation example anchor:Gui_Representation_Example[] [underline]#In YAML:# [source,python] ---- - name: 'Flags' type: 'bit' array_size: 3 gui_representation: help: 'IPv4 Flags' form_type: 'checkbox' # can check each bit - name: 'dst_addr' default: [48, 0, 0, 0] type: uint32 gui_representation: help: 'IPv4 Destination Address' data_type: 'ipv4_t' # special representation case, show as 4 decimal numbers form_type: 'combo_with_edit' # can choose from pre-defined values or edit manually combobox_values: [127, 0, 0, 1]: 'localhost' [255, 255, 255, 255]: 'broadcast' - name: 'protocol' type: uint8 default: 0x06 value_based_next_header: &ipv4_next_header 0x06: 'tcp' 0x11: 'udp' default : "payload" gui_representation: help: 'IPv4 Protocol Field' form_type: 'combo_without_edit' # choose from supported protocols, no manual edit combobox_values: <<: *ipv4_next_header # take same choices as value_based_next_header ---- [underline]#In GUI:# checkbox for bits: image:images/checkbox.jpg[] editing in combo-box: image:images/combo_button_editing.jpg[] choosing from predefined values: image:images/combo_button_choosing.jpg[] ==== Union base TBD === Resource * link:yaml/headers.yaml[headers.yaml] * link:https://wireedit.com/[WireEdit] * link:http://ostinato.org/[ostinato] * link:http://www.slideshare.net/nlekh/ixiaexplorer[IxExplorer]