diff options
Diffstat (limited to 'doc/trex_scapy_rpc_server.asciidoc')
| -rwxr-xr-x | doc/trex_scapy_rpc_server.asciidoc | 867 |
1 files changed, 867 insertions, 0 deletions
diff --git a/doc/trex_scapy_rpc_server.asciidoc b/doc/trex_scapy_rpc_server.asciidoc new file mode 100755 index 00000000..65d9e7b2 --- /dev/null +++ b/doc/trex_scapy_rpc_server.asciidoc @@ -0,0 +1,867 @@ +The TRex Scapy RPC Server +========================= +:Author: Itamar Raviv +:email: trex-dev@cisco.com +:revnumber: 1.00 +:quotes.++: +:numbered: +:web_server_url: https://trex-tgn.cisco.com/trex +:local_web_server_url: csi-wiki-01:8181/trex +:toclevels: 4 + +include::trex_ga.asciidoc[] + +== Change log + +[options="header",cols="^1,^h,3a"] +|================= +| Version | name | meaning +| 1.00 | Itamar Raviv (itraviv) | +- first version +| 2.00 | Anton (XORED) | update by xored software +|================= + + +== Audience of this document + +TRex GUI guys + +== Scapy RPC Server - Overview +Scapy Server is implemented following the link:http://www.jsonrpc.org/specification[JSON-RPC 2.0 specification], + +Therefore, requests and replies follow the JSON-RPC 2.0 spec. + +The server operates on a Request-Response basis *over ZMQ*, and does not support batched commands handling. + + +Read more about ZMQ link:http://zguide.zeromq.org/page:all[here] + +image::images/scapy_json_rpc_server.png[title="Scapy JSON RPC Server",align="left",width=800, link="images/scapy_json_rpc_server.png"] + +=== Error Codes + +Error codes are given according to this table: [also follows the JSON-RPC spec, with added error codes] + +[options="header",cols="^1,^h,3a"] +|================= +| Error Code | Message | Meaning +| -32700 | Parse Error | Invalid JSON was received by the server. An error occurred on the server while parsing the JSON input. +| -32600 | Invalid Request | The JSON sent is not a valid Request object. +| -32601 | Method not found | The method does not exist / is not available +| -32603 | Invalid params | Invalid method parameter(s) +| -32097 | Syntax Error | Syntax Error in input +| -32098 | Scapy Server: message | Scapy Server had an error while executing your command, described in the message given +| -32096 | Scapy Server: Unknown Error | Scapy Server encountered an error that cannot be described + + +|================= + + + +== Data Bases and Data Structures used in Scapy Server +=== build_pkt, reconstruct_pkt packet model [[build_pkt_input]] + +Following JSON represents a Scapy structure, which can be used to build packet from scratch(build_pkt) or to modify particular fields in the prococol(reconstruct_pkt). Most fields can be omitted, in this case default or calculated values will be used. +For reconstruct_pkt default values will be taken from the original packet. + +Exaples of JSON payloads and their scapy expression alternatives + +[source,python] +---- +Ether(src="de:ad:be:ef:de:ad")/Dot1Q()/Dot1Q(vtype=1)/IP(src="127.0.0.1", chksum="0x312")/TCP(sport=443) +---- + +[source,python] +---- +[ + { "id": "Ether", "fields": [{"id": "src", "value": "de:ad:be:ef:de:ad"}] }, + { "id": "Dot1Q"}, + { "id": "Dot1Q", "fields": [{"id": "vtype", "value": "1"}] }, + { "id": "IP", "fields": [{"id": "src", "value": "127.0.0.1"}, {"id": "chksum", "value": "0x312"}] }, + { "id": "TCP", "fields": [{"id": "sport", "value": "443"}] } +] +---- + +=== Scapy server value types +Most values can be passed as strings(including decimal numbers, hex numbers, enums, values), +but for binary payload, value object should be used + +[source,python] +---- +- int/long/str - they can de specified directly as a value of a field +- {"vtype": "BYTES", "base64": "my_payload_base64"} - binary payload passed as base64 +- {"vtype": "EXPRESSION", "expr": "TCPOptions()"} - python expression(normally, should be avoided) +- {"vtype": "UNDEFINED"} - unset field value, and let it be assigned automatically +- {"vtype": "RANDOM"} - assign a random value to a field +---- + +Example of object value usage(to specify binary payload) +---- +Ether()/IP()/TCP()/Raw(load=my_payload) +---- + +[source,python] +---- +[ + { "id": "Ether"}, + { "id": "IP"}, + { "id": "TCP"}, + { "id": "Raw", "fields": [ + { + "id": "load", + "value": {"vtype": "BYTES", "base64": "my_payload_base64"} + } + ]} +] +---- + +=== Scapy packet result payload [[build_pkt_output]] +build_pkt and reconstruct pkt take packet model and produce result JSON, +with the binary payload and field values and offsets defined + +[source,python] +---- +{ + "binary": "AAAAAQAAAAAAAgAACABFAAAoAAEAAEAGOs4QAAABMAAAAQAUAFAAAAAAAAAAAFACIABPfQAA", // base64 encoded binary payload + "data": [ + { + "id": "Ether", # scapy class + "name": "Ethernet", # human-readable protocol name + "offset": 0, # global offset for all fields + "fields": [ + { + "id": "dst", # scapy field id + "hvalue": "00:00:00:01:00:00", # human readable value + "length": 6, # 6 bytes + "offset": 0, # 0 bytes offset from + "value": "00:00:00:01:00:00" # internal value, which for this type is the same as hvalue + }, + { + "id": "src", + ... # same as for dst + }, + { + "hvalue": "IPv4", # human-readable value + "id": "type", + "length": 2, + "offset": 12, # + "value": 2048 # integer value for IPv4(0x800) + } + ] + }, + { + "id": "IP", + "name": "IP", + "offset": 14, + "fields": [ + { + "hvalue": "4", + "id": "version", + "length": 0, # the length is 0, which means it is a bitfield. mask should be used to show location + "offset": 0, # offset from the IP.offset. it needs to be added to all fields of IP + "value": 4 + }, + { + "hvalue": "5", + "id": "ihl", + "length": 0, # again length is 0. that's other part of the first byte of IP + "offset": 0, + "value": 5 + }, + { + "hvalue": "0x0", + "id": "tos", + "length": 1, + "offset": 1, + "value": 0 + }, + { + "hvalue": "40", + "id": "len", + "length": 2, + "offset": 2, + "value": 40 + }, + { + "hvalue": "1", + "id": "id", + "length": 2, + "offset": 4, + "value": 1 + }, + { + "hvalue": "", # no flags are specified here. but this field can contain "US" for URG+SYN flags + "id": "flags", + "length": 0, + "offset": 6, + "value": 0 + }, + { + "hvalue": "0", + "id": "frag", + "length": 0, + "offset": 6, + "value": 0 + }, + { + "hvalue": "64", + "id": "ttl", + "length": 1, + "offset": 8, + "value": 64 + }, + { + "hvalue": "tcp", # this field is enum. enum dictionary can be obtained as a medatata for IP fields. + "id": "proto", + "length": 1, + "offset": 9, + "value": 6 + }, + { + "hvalue": "0x3ace", + "id": "chksum", + "length": 2, + "offset": 10, + "value": 15054 + }, + { + "hvalue": "[]", + "id": "options", + "length": 2, + "offset": 20, + "value": { # options can not be representted as a human string, so they are passed as an expression + "expr": "[]", + "vtype": "EXPRESSION" + } + } + ] + }, + { + "id": "TCP", + "name": "TCP", + "offset": 34 + "fields": [ + { + "hvalue": "20", + "id": "sport", + "length": 2, + "offset": 0, + "value": 20 + }, + # .. some more TCP fields here + { + "hvalue": "{}", + "id": "options", + "ignored": true, + "length": 2, + "offset": 20, + "value": { # TCPOptions are represented as a python expression with tuple and binary buffers + "expr": "[('MSS', 1460), ('NOP', None), ('NOP', None), ('SAckOK', b'')]", + "vtype": "EXPRESSION" + } + } + ] + } + ] +} + +---- + +=== Scapy server field definitions [[get_definitions_model]] +Scapy server can return metadata object, describing protocols and fields. +Most values, including field types are optional in the definition. +If field type is missing, it can be treated as a STRING. + +[source,python] +---- +"protocols": [ +{ + "id": "Ether", # scapy class + "name": "Ethernet", # name of the protocol + "fields": [ + { + "id": "dst", + "name": "Destination", # GUI will display Destination instead of dst + "type": "STRING", + "regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$" + }, + { + "id": "src", + "name": "Source", + "type": "STRING", + "regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$" + }, + { + "values_dict": { + "ATMMPOA": 34892, + "RAW_FR": 25945, + "DNA_DL": 24577, + "ATMFATE": 34948, + "ATALK": 32923, + "BPQ": 2303, + "X25": 2053, + "PPP_DISC": 34915, + "DEC": 24576, + "n_802_1Q": 33024, + "PPP_SES": 34916, + "TEB": 25944, + "SCA": 24583, + "PPP": 34827, + "FR_ARP": 2056, + "CUST": 24582, + "ARP": 2054, + "DNA_RC": 24578, + "NetBEUI": 33169, + "AARP": 33011, + "DIAG": 24581, + "IPv4": 2048, + "DNA_RT": 24579, + "IPv6": 34525, + "LAT": 24580, + "IPX": 33079, + "LOOP": 36864 + }, + "id": "type", + "name": "Type" + "type": "ENUM" + } + ] +}, +{ + "id": "TCP", + "name": "TCP", + "fields": [ + { + "id": "sport", + "name": "Source port", + "type": "NUMBER", + "min": 0, # optional min value + "max": 65535 # optional max value + + }, + { + "id": "dport", + "name": "Destination port", + "type": "NUMBER", + "min": 0, + "max": 65535 + }, + { + "id": "seq", + "name": "Sequence number", + "type": "NUMBER" + }, + { + "id": "ack", + "name": "Acknowledgment number", + "type": "NUMBER" + }, + { + "id": "dataofs", + "name": "Data offset", + "type": "NUMBER" + }, + { + "id": "reserved", + "name": "Reserved", + "type": "NUMBER" + }, + { + "id": "flags", + "name": "Flags", + "auto": false, + "type": "BITMASK", + "bits": [ # fields definition for the UI + {"name": "URG", "mask": 32, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 32}]}, + {"name": "ACK", "mask": 16, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 16}]}, + {"name": "PSH", "mask": 8, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 8}]}, + {"name": "RST", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]}, + {"name": "SYN", "mask": 2, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 2}]}, + {"name": "FIN", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]} + ] + }, + { + "id": "window", + "name": "Window size", + "type": "NUMBER" + }, + { + "id": "chksum", + "name": "Checksum", + "auto": true, + "type": "NUMBER" + }, + { + "id": "urgptr", + "name": "Urgent pointer", + "type": "NUMBER" + }, + { + "id": "options", + "name": "Options", + "type": "EXPRESSION" + } + ] +}, +{ + "id": "IP", + "name": "Internet Protocol Version 4", + "fields": [ + { + "id": "version", # only renaming + "name": "Version" + }, + { + "id": "ihl", + "name": "IHL", + "type": "NUMBER", + "auto": true # calculate IHL automatically + }, + { + "id": "tos", + "name": "TOS", + "type": "NUMBER" + }, + { + "id": "len", + "name": "Total Length", + "type": "NUMBER", + "auto": true + }, + { + "id": "id", + "name": "Identification", + "type": "NUMBER" + }, + { + "id": "flags", + "name": "Flags", + "type": "BITMASK", + "min": 0, + "max": 8, + "bits": [ # bitmask definition for IP.flags + {"name": "Reserved", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]}, + {"name": "Fragment", "mask": 2, "values":[{"name":"May fragment (0)", "value": 0}, {"name":"Don't fragment (1)", "value": 2}]}, + {"name": "More Fragments(MF)", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]} + ] + }, + { + "id": "frag", + "name": "Fragment offset", + "type": "NUMBER" + }, + { + "id": "ttl", + "name": "TTL", + "type": "NUMBER", + "min": 1, + "max": 255 + + }, + { + "id": "proto", + "name": "Protocol" + }, + { + "id": "chksum", + "name": "Checksum", + "type": "STRING", + "auto": true + }, + { + "id": "src", + "name": "Source address", + "type": "STRING", + "regexp": "regexp-to-check-this-field" + }, + { + "id": "dst", + "name": "Destination address", + "regexp": "regexp-to-check-this-field" + }, + { + "id": "options", + "name": "Options", + "type": "EXPRESSION" + } + ] +}, +{ + "id": "Dot1Q", + "name": "802.1Q", + "fields": [ + { + "id": "prio", + "name": "prio" + "type": "NUMBER", + }, + { + "id": "id", + "type": "NUMBER", + "name": "id" + }, + { + "id": "vlan", + "type": "NUMBER", + "name": "vlan" + }, + { + "values_dict": { + "ATMMPOA": 34892, + "RAW_FR": 25945, + "DNA_DL": 24577, + "ATMFATE": 34948, + "ATALK": 32923, + "BPQ": 2303, + "X25": 2053, + "PPP_DISC": 34915, + "DEC": 24576, + "n_802_1Q": 33024, + "PPP_SES": 34916, + "TEB": 25944, + "SCA": 24583, + "PPP": 34827, + "FR_ARP": 2056, + "CUST": 24582, + "ARP": 2054, + "DNA_RC": 24578, + "NetBEUI": 33169, + "AARP": 33011, + "DIAG": 24581, + "IPv4": 2048, + "DNA_RT": 24579, + "IPv6": 34525, + "LAT": 24580, + "IPX": 33079, + "LOOP": 36864 + }, + "id": "type", + "name": "type", + "type": "ENUM" + } + ] +}, +{ + "id": "Raw", + "name": "Raw", + "fields": [ + { + "id": "load", + "name": "Payload", + "type": "BYTES" + } + ] +} +] + +] +---- + + + +== RPC Commands +The following RPC commands are supported. Please refer to databases section for elaboration for each database. + +=== Supported Methods +* *Name* - supported_methods +* *Description* - returns the list of all supported methods by Scapy Server and their parameters +* *Parameters* - the parameter ('all') will return *ALL* supported methods. + + other string delivered as parameter will return True/False if the string matches a supported method name +* *Result* - according to input: 'all' string will return list of supported methods, otherwise will return True/False as mentioned. + + The returned dictionary describes for each method it's number of parameters followed by a list of their names. + +*Example:* + +[source,python] +---- +'Request': +{ + "jsonrpc": "2.0", + "id": "1", + "method": "supported_methods", + "params": ["all"] +} + +'Result': +{'id': '1', + 'jsonrpc': '2.0', + 'result': { . + . + . + . + 'build_pkt': [1, [u'pkt_descriptor']], + 'check_update': [2, [u'db_md5', u'field_md5']], + 'get_all': [0, []], + 'get_tree': [0, []], + 'get_version': [0, []], + 'supported_methods': [1, [u'method_name']] + } +} +---- + + + +=== GetAll +* *Name* - 'get_all' +* *Description* - Returns the supported protocols library (DB) and Field-to-RegEx mapping library, and their MD5 +* *Paramters* - None +* *Result* ['object'] - JSON format of dictionary. see table below + +.Object type 'return values for get_all' +[options="header",cols="1,1,3,3"] +|================= +| Key | Key Type | Value | Value Type +| db | string | supported protocols dictionary | protocol dictionary +| fields | string | Field-to-RegEx dictionary | Field-to-RegEx dictionary +| db_md5 | string | MD5 of DB | encoded in base64 +| fields_md5 | string | MD5 of fields | encoded in base64 +|================= + +*Example:* + +[source,python] +---- +'Request': +{ + "jsonrpc": "2.0", + "id": 1, + "method": "get_all", + "params": [] +} + +'Response': +{ + "jsonrpc" : "2.0", + "id" : 1, + "result" : {'db': {'ARP': [('hwtype', 'XShortField', '(1)'), + ('ptype', 'XShortEnumField', '(2048)'), + ('hwlen', 'ByteField', '(6)'), + ('plen', 'ByteField', '(4)'), + ('op', 'ShortEnumField', '(1)'), + ('hwsrc', 'ARPSourceMACField', '(None)'), + ('psrc', 'SourceIPField', '(None)'), + ('hwdst', 'MACField', "('00:00:00:00:00:00')"), + ('pdst', 'IPField', "('0.0.0.0')")], + . + . + . + 'db_md5': 'Z+gRt88y7SC0bDu496/DQg==\n', + 'fields': {'ARPSourceMACField': 'empty', + 'BCDFloatField': 'empty', + 'BitEnumField': 'empty', + . + . + . + +} + +---- + +=== Check if Database is updated +* *Name* - 'check_update' +* *Description* - checks if both protocol database and fields database are up to date according to md5 comparison +* *Parameters* - md5 of database, md5 of fields +* *Result* - upon failure: error code -32098 (see link:trex_scapy_rpc_server.html#_error_codes[RPC server error codes]) + + followed by a message: "Fields DB is not up to date" or "Protocol DB is not up to date" + + upon success: return 'true' as result (see below) + + + +*Example:* + +[source,python] +---- +'Request': + +{ + "jsonrpc": "2.0", + "id": "1", + "method": "check_update", + "params": ["md5_of_protocol_db", "md5_of_fields"] +} + +'Response': #on failure + +{ + "jsonrpc": "2.0", + "id": "1", + "error": { + "code": -32098, + "message:": "Scapy Server: Fields DB is not up to date" + } +} + +'Response': #on success + +{ + "jsonrpc": "2.0", + "id": "1", + "result": true +} +---- + + +=== Get Version +* *Name* - 'get_version' +* *Description* - Queries the server for version information +* *Paramters* - None +* *Result* ['object'] - See table below + +.Object type 'return values for get_version' +[options="header",cols="1,1,3"] +|================= +| Field | Type | Description +| version | string | Scapy Server version +| built_by | string | who built this version +|================= + + +*Example:* +[source,python] +---- + +'Request': + +{ + "jsonrpc": "2.0", + "id": "1", + "method": "get_version", + "params": [] +} + + + +'Response': + +{ + "jsonrpc": "2.0", + "id": "1", + "result": { + "version": "v1.0", + "built_by": "itraviv" + } +} + +---- + +=== Build Packet +* *Name* - 'build_pkt' +* *Description* - Builds a new packet from the definition and returns binary data and json structure + +* *Return Value* - Returns xref:build_pkt_output[Scapy packet result payload]. +* *Paramters* - JSON xref:build_pkt_input[packet definition model]. + +=== Create packet from binary data and modify fields +* *Name* - 'reconstruct_pkt' +* *Description* - Builds a new packet from the binary data and returns binary data and json structure + +* *Return Value* - Returns xref:build_pkt_output[Scapy packet result payload]. +* *Paramters* - base64-encoded packet bytes, optional JSON xref:build_pkt_input[packet definition model] with fields to override. + +=== Get protocol definitions +* *Name* - 'get_definitions' +* *Description* - Returns definitions for protocols and fields + +* *Return Value* - array of protocol definitions in a "result.protocols" json. xref:get_definitions_model[Output model] +* *Paramters* - array of protocol class names to define or null to fetch metadata for all protocols. ex. ["Ether", "TCP"] + +=== Get protocol tree hierarchy example +* *Name* - 'get_tree' +* *Description* - returns a *suggested* dictionary of protocols ordered in a hierarchy tree. + +User can still create non valid hierarchies. (such as Ether()/DNS()/IP()) +* *Parameters* - none +* *Result* [dictionary] - Example for packet layers that can be used to build a packet. Ordered in an hierarchy tree. + +*Example:* + +[source,python] +---- + +'Request': + +{ + "id": "1", + "jsonrpc": "2.0", + "method": "get_tree", + "params": [] +} + + +'Response': + +{'id': '1', + 'jsonrpc': '2.0', + 'result': {'ALL': { + 'Ether': {'ARP': {}, + 'IP': { 'TCP': {'RAW': 'payload'}, + 'UDP': {'RAW': 'payload'} + } + } + } + } +} +---- + + +== Usage of Scapy RPC Server +Notice the existance of the following files: + +* scapy_service.py +* scapy_zmq_server.py +* scapy_zmq_client.py + +=== Scapy_zmq_server.py +In this section we will see how to bring up the Scapy ZMQ server. +There are 2 ways to run this server: + +* Through command line +* Through Python interpreter + +==== Running Scapy ZMQ Server from command line +Run the file scapy_zmq_server.py with the argument -s to declare the port that the server will listen to. + +Running the file without the "-s" argument will use *port 4507 by default*. + + + +Notice: + +* The Server's IP will be the IP address of the local host. +* The Server will accept requests from *any* IP address on that port. + +[source,bash] +---- +user$ python scapy_zmq_server.py -s 5555 + +***Scapy Server Started*** +Listening on port: 5555 +Server IP address: 10.0.0.1 + +---- + +==== Running Scapy ZMQ Server from the Python interpreter +* Run the Python Interpreter (Scapy Server currently supports Python2) +* Import the scapy_zmq_server.py file +* Create a Scapy_server Object with argument as port number. default argument is port 4507 +* Invoke method activate(). (This method is blocking because the server is listening on the port). + +[source,bash] +---- +user$ python +>>> from scapy_zmq_server import * +>>> s = Scapy_server() // starts with port 4507 +>>> s = Scapy_server(5555) //starts with port 5555 +>>> s.activate() +***Scapy Server Started*** +Listening on port: 5555 +Server IP address: 10.0.0.1 + +---- + +==== Shutting down Scapy ZMQ Server +There are 2 ways to shut down the server: + +* The server can be shut down using the keyboard interrupt Ctrl+C +* The server can be shut down remotely with the method "shut_down" with no arguments + +[source,bash] +---- +//Sending Request: {"params": [], "jsonrpc": "2.0", "method": "shut_down", "id": "1"} +//Will result in this print by the server: +Server: Shut down by remote user +---- + + + + + + |