The TRex Scapy RPC Server
=========================
:Author: Itamar Raviv
:email: trex-dev@cisco.com
:revnumber: 1.00
:quotes.++:
:numbered:
:web_server_url: https://trex-tgn.cisco.com/trex
:local_web_server_url: csi-wiki-01:8181/trex
:toclevels: 4
include::trex_ga.asciidoc[]
== Change log
[options="header",cols="^1,^h,3a"]
|=================
| Version | name | meaning
| 1.00 | Itamar Raviv (itraviv) | first version
| 2.00 | Anton Kiselev (kisel) | Add description for updated compatibilities of packet creation and modification
| 3.00 | Vyacheslav Ogai (hedjuo) | Add description for supporting Filed Engine and predefined packet templates
|=================
== Audience of this document
TRex GUI guys
== Scapy RPC Server - Overview
Scapy Server is implemented following the link:http://www.jsonrpc.org/specification[JSON-RPC 2.0 specification], +
Therefore, requests and replies follow the JSON-RPC 2.0 spec. +
The server operates on a Request-Response basis *over ZMQ*, and does not support batched commands handling. +
Read more about ZMQ link:http://zguide.zeromq.org/page:all[here]
image::images/scapy_json_rpc_server.png[title="Scapy JSON RPC Server",align="left",width=800, link="images/scapy_json_rpc_server.png"]
=== Error Codes
Error codes are given according to this table: [also follows the JSON-RPC spec, with added error codes]
[options="header",cols="^1,^h,3a"]
|=================
| Error Code | Message | Meaning
| -32700 | Parse Error | Invalid JSON was received by the server. An error occurred on the server while parsing the JSON input.
| -32600 | Invalid Request | The JSON sent is not a valid Request object.
| -32601 | Method not found | The method does not exist / is not available
| -32603 | Invalid params | Invalid method parameter(s)
| -32097 | Syntax Error | Syntax Error in input
| -32098 | Scapy Server: message | Scapy Server had an error while executing your command, described in the message given
| -32096 | Scapy Server: Unknown Error | Scapy Server encountered an error that cannot be described
|=================
== Data Bases and Data Structures used in Scapy Server
=== build_pkt, build_pkt_ex, reconstruct_pkt packet model [[build_pkt_input]]
Following JSON represents a Scapy structure, which can be used to build packet from scratch(build_pkt) or to modify particular fields in the prococol(reconstruct_pkt). Most fields can be omitted, in this case default or calculated values will be used.
For reconstruct_pkt default values will be taken from the original packet.
Exaples of JSON payloads and their scapy expression alternatives
[source,python]
----
Ether(src="de:ad:be:ef:de:ad")/Dot1Q()/Dot1Q(vtype=1)/IP(src="127.0.0.1", chksum="0x312")/TCP(sport=443)
----
[source,python]
----
[
{ "id": "Ether", "fields": [{"id": "src", "value": "de:ad:be:ef:de:ad"}] },
{ "id": "Dot1Q"},
{ "id": "Dot1Q", "fields": [{"id": "vtype", "value": "1"}] },
{ "id": "IP", "fields": [{"id": "src", "value": "127.0.0.1"}, {"id": "chksum", "value": "0x312"}] },
{ "id": "TCP", "fields": [{"id": "sport", "value": "443"}] }
]
----
Field Engine VM instruction model +
For more reference see link:https://trex-tgn.cisco.com/trex/doc/cp_stl_docs/api/field_engine.html[Field Engine modules spec]
[source,python]
----
{
"field_engine":{
"instructions":[
{
"id":"STLVmFlowVar",
"parameters": {
"name": "Ether_dst",
"init_value": "00:00:00:01:00:00", #can be a string or an integer
"max_value": "0", #can be a string or an integer
"min_value": "0", #can be a string or an integer
"step": "1",
"size": "4",
"op": "inc"
}
},
...
],
"global_parameters":{
"cache_size":"1000"
}
}
}
----
=== Scapy server value types
Most values can be passed as strings(including decimal numbers, hex numbers, enums, values),
but for binary payload, value object should be used
[source,python]
----
- int/long/str - they can de specified directly as a value of a field
- {"vtype": "BYTES", "base64": "my_payload_base64"} - binary payload passed as base64
- {"vtype": "EXPRESSION", "expr": "TCPOptions()"} - python expression(normally, should be avoided)
- {"vtype": "UNDEFINED"} - unset field value, and let it be assigned automatically
- {"vtype": "RANDOM"} - assign a random value to a field
----
Example of object value usage(to specify binary payload)
----
Ether()/IP()/TCP()/Raw(load=my_payload)
----
[source,python]
----
[
{ "id": "Ether"},
{ "id": "IP"},
{ "id": "TCP"},
{ "id": "Raw", "fields": [
{
"id": "load",
"value": {"vtype": "BYTES", "base64": "my_payload_base64"}
}
]}
]
----
=== Scapy packet result payload [[build_pkt_output]]
build_pkt, build_pkt_ex and reconstruct pkt take packet model and produce result JSON,
with the binary payload and field values and offsets defined
[source,python]
----
{
"binary": "AAAAAQAAAAAAAgAACABFAAAoAAEAAEAGOs4QAAABMAAAAQAUAFAAAAAAAAAAAFACIABPfQAA", // base64 encoded binary payload
"data": [
{
"id": "Ether", # scapy class
"name": "Ethernet", # human-readable protocol name
"offset": 0, # global offset for all fields
"fields": [
{
"id": "dst", # scapy field id
"hvalue": "00:00:00:01:00:00", # human readable value
"length": 6, # 6 bytes
"offset": 0, # 0 bytes offset from
"value": "00:00:00:01:00:00" # internal value, which for this type is the same as hvalue
},
{
"id": "src",
... # same as for dst
},
{
"hvalue": "IPv4", # human-readable value
"id": "type",
"length": 2,
"offset": 12, #
"value": 2048 # integer value for IPv4(0x800)
}
]
},
{
"id": "IP",
"name": "IP",
"offset": 14,
"fields": [
{
"hvalue": "4",
"id": "version",
"length": 0, # the length is 0, which means it is a bitfield. mask should be used to show location
"offset": 0, # offset from the IP.offset. it needs to be added to all fields of IP
"value": 4
},
{
"hvalue": "5",
"id": "ihl",
"length": 0, # again length is 0. that's other part of the first byte of IP
"offset": 0,
"value": 5
},
{
"hvalue": "0x0",
"id": "tos",
"length": 1,
"offset": 1,
"value": 0
},
{
"hvalue": "40",
"id": "len",
"length": 2,
"offset": 2,
"value": 40
},
{
"hvalue": "1",
"id": "id",
"length": 2,
"offset": 4,
"value": 1
},
{
"hvalue": "", # no flags are specified here. but this field can contain "US" for URG+SYN flags
"id": "flags",
"length": 0,
"offset": 6,
"value": 0
},
{
"hvalue": "0",
"id": "frag",
"length": 0,
"offset": 6,
"value": 0
},
{
"hvalue": "64",
"id": "ttl",
"length": 1,
"offset": 8,
"value": 64
},
{
"hvalue": "tcp", # this field is enum. enum dictionary can be obtained as a medatata for IP fields.
"id": "proto",
"length": 1,
"offset": 9,
"value": 6
},
{
"hvalue": "0x3ace",
"id": "chksum",
"length": 2,
"offset": 10,
"value": 15054
},
{
"hvalue": "[]",
"id": "options",
"length": 2,
"offset": 20,
"value": { # options can not be representted as a human string, so they are passed as an expression
"expr": "[]",
"vtype": "EXPRESSION"
}
}
]
},
{
"id": "TCP",
"name": "TCP",
"offset": 34
"fields": [
{
"hvalue": "20",
"id": "sport",
"length": 2,
"offset": 0,
"value": 20
},
# .. some more TCP fields here
{
"hvalue": "{}",
"id": "options",
"ignored": true,
"length": 2,
"offset": 20,
"value": { # TCPOptions are represented as a python expression with tuple and binary buffers
"expr": "[('MSS', 1460), ('NOP', None), ('NOP', None), ('SAckOK', b'')]",
"vtype": "EXPRESSION"
}
}
]
}
]
}
----
=== Scapy packet result payload in extended mode [[build_pkt_ex_output]]
[source,python]
----
{
"binary": #same as in build_pkt
"data": #same as in build_pkt
"field_engine": {
"error": "",
"instructions": {
"cache": 1000,
"instructions": [
...
#FlowVar definition instruction
{
"name": "Ether_dst",
"max_value": 0,
"min_value": 0,
"init_value": 65536,
"step": 1,
"size": 4,
"type": "flow_var",
"op": "inc"
},
#FlowVar write instruction
{
"name": "Ether_dst",
"is_big_endian": true,
"pkt_offset": 6,
"type": "write_flow_var",
"add_value": 0
}
...
]
}
}
}
----
=== Scapy server field definitions [[get_definitions_model]]
Scapy server can return metadata object, describing protocols and fields.
Most values, including field types are optional in the definition.
If field type is missing, it can be treated as a STRING.
[source,python]
----
"protocols": [
{
"id": "Ether", # scapy class
"name": "Ethernet", # name of the protocol
"fields": [
{
"id": "dst",
"name": "Destination", # GUI will display Destination instead of dst
"type": "STRING",
"regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$"
},
{
"id": "src",
"name": "Source",
"type": "STRING",
"regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$"
},
{
"values_dict": {
"ATMMPOA": 34892,
"RAW_FR": 25945,
"DNA_DL": 24577,
"ATMFATE": 34948,
"ATALK": 32923,
"BPQ": 2303,
"X25": 2053,
"PPP_DISC": 34915,
"DEC": 24576,
"n_802_1Q": 33024,
"PPP_SES": 34916,
"TEB": 25944,
"SCA": 24583,
"PPP": 34827,
"FR_ARP": 2056,
"CUST": 24582,
"ARP": 2054,
"DNA_RC": 24578,
"NetBEUI": 33169,
"AARP": 33011,
"DIAG": 24581,
"IPv4": 2048,
"DNA_RT": 24579,
"IPv6": 34525,
"LAT": 24580,
"IPX": 33079,
"LOOP": 36864
},
"id": "type",
"name": "Type"
"type": "ENUM"
}
]
},
{
"id": "TCP",
"name": "TCP",
"fields": [
{
"id": "sport",
"name": "Source port",
"type": "NUMBER",
"min": 0, # optional min value
"max": 65535 # optional max value
},
{
"id": "dport",
"name": "Destination port",
"type": "NUMBER",
"min": 0,
"max": 65535
},
{
"id": "seq",
"name": "Sequence number",
"type": "NUMBER"
},
{
"id": "ack",
"name": "Acknowledgment number",
"type": "NUMBER"
},
{
"id": "dataofs",
"name": "Data offset",
"type": "NUMBER"
},
{
"id": "reserved",
"name": "Reserved",
"type": "NUMBER"
},
{
"id": "flags",
"name": "Flags",
"auto": false,
"type": "BITMASK",
"bits": [ # fields definition for the UI
{"name": "URG", "mask": 32, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 32}]},
{"name": "ACK", "mask": 16, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 16}]},
{"name": "PSH", "mask": 8, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 8}]},
{"name": "RST", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]},
{"name": "SYN", "mask": 2, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 2}]},
{"name": "FIN", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]}
]
},
{
"id": "window",
"name": "Window size",
"type": "NUMBER"
},
{
"id": "chksum",
"name": "Checksum",
"auto": true,
"type": "NUMBER"
},
{
"id": "urgptr",
"name": "Urgent pointer",
"type": "NUMBER"
},
{
"id": "options",
"name": "Options",
"type": "EXPRESSION"
}
]
},
{
"id": "IP",
"name": "Internet Protocol Version 4",
"fields": [
{
"id": "version", # only renaming
"name": "Version"
},
{
"id": "ihl",
"name": "IHL",
"type": "NUMBER",
"auto": true # calculate IHL automatically
},
{
"id": "tos",
"name": "TOS",
"type": "NUMBER"
},
{
"id": "len",
"name": "Total Length",
"type": "NUMBER",
"auto": true
},
{
"id": "id",
"name": "Identification",
"type": "NUMBER"
},
{
"id": "flags",
"name": "Flags",
"type": "BITMASK",
"min": 0,
"max": 8,
"bits": [ # bitmask definition for IP.flags
{"name": "Reserved", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]},
{"name": "Fragment", "mask": 2, "values":[{"name":"May fragment (0)", "value": 0}, {"name":"Don't fragment (1)", "value": 2}]},
{"name": "More Fragments(MF)", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]}
]
},
{
"id": "frag",
"name": "Fragment offset",
"type": "NUMBER"
},
{
"id": "ttl",
"name": "TTL",
"type": "NUMBER",
"min": 1,
"max": 255
},
{
"id": "proto",
"name": "Protocol"
},
{
"id": "chksum",
"name": "Checksum",
"type": "STRING",
"auto": true
},
{
"id": "src",
"name": "Source address",
"type": "STRING",
"regexp": "regexp-to-check-this-field"
},
{
"id": "dst",
"name": "Destination address",
"regexp": "regexp-to-check-this-field"
},
{
"id": "options",
"name": "Options",
"type": "EXPRESSION"
}
]
},
{
"id": "Dot1Q",
"name": "802.1Q",
"fields": [
{
"id": "prio",
"name": "prio"
"type": "NUMBER",
},
{
"id": "id",
"type": "NUMBER",
"name": "id"
},
{
"id": "vlan",
"type": "NUMBER",
"name": "vlan"
},
{
"values_dict": {
"ATMMPOA": 34892,
"RAW_FR": 25945,
"DNA_DL": 24577,
"ATMFATE": 34948,
"ATALK": 32923,
"BPQ": 2303,
"X25": 2053,
"PPP_DISC": 34915,
"DEC": 24576,
"n_802_1Q": 33024,
"PPP_SES": 34916,
"TEB": 25944,
"SCA": 24583,
"PPP": 34827,
"FR_ARP": 2056,
"CUST": 24582,
"ARP": 2054,
"DNA_RC": 24578,
"NetBEUI": 33169,
"AARP": 33011,
"DIAG": 24581,
"IPv4": 2048,
"DNA_RT": 24579,
"IPv6": 34525,
"LAT": 24580,
"IPX": 33079,
"LOOP": 36864
},
"id": "type",
"name": "type",
"type": "ENUM"
}
]
},
{
"id": "Raw",
"name": "Raw",
"fields": [
{
"id": "load",
"name": "Payload",
"type": "BYTES"
}
]
}
]
]
----
== RPC Commands
The following RPC commands are supported. Please refer to databases section for elaboration for each database.
=== Supported Methods
* *Name* - supported_methods
* *Description* - returns the list of all supported methods by Scapy Server and their parameters
* *Parameters* - the parameter ('all') will return *ALL* supported methods. +
other string delivered as parameter will return True/False if the string matches a supported method name
* *Result* - according to input: 'all' string will return list of supported methods, otherwise will return True/False as mentioned. +
The returned dictionary describes for each method it's number of parameters followed by a list of their names.
*Example:*
[source,python]
----
'Request':
{
"jsonrpc": "2.0",
"id": "1",
"method": "supported_methods",
"params": ["all"]
}
'Result':
{'id': '1',
'jsonrpc': '2.0',
'result': { .
.
.
.
'build_pkt': [1, [u'pkt_descriptor']],
'check_update': [2, [u'db_md5', u'field_md5']],
'get_all': [0, []],
'get_tree': [0, []],
'get_version': [0, []],
'supported_methods': [1, [u'method_name']]
}
}
----
=== GetAll
* *Name* - 'get_all'
* *Description* - Returns the supported protocols library (DB) and Field-to-RegEx mapping library, and their MD5
* *Paramters* - None
* *Result* ['object'] - JSON format of dictionary. see table below
.Object type 'return values for get_all'
[options="header",cols="1,1,3,3"]
|=================
| Key | Key Type | Value | Value Type
| db | string | supported protocols dictionary | protocol dictionary
| fields | string | Field-to-RegEx dictionary | Field-to-RegEx dictionary
| db_md5 | string | MD5 of DB | encoded in base64
| fields_md5 | string | MD5 of fields | encoded in base64
|=================
*Example:*
[source,python]
----
'Request':
{
"jsonrpc": "2.0",
"id": 1,
"method": "get_all",
"params": []
}
'Response':
{
"jsonrpc" : "2.0",
"id" : 1,
"result" : {'db': {'ARP': [('hwtype', 'XShortField', '(1)'),
('ptype', 'XShortEnumField', '(2048)'),
('hwlen', 'ByteField', '(6)'),
('plen', 'ByteField', '(4)'),
('op', 'ShortEnumField', '(1)'),
('hwsrc', 'ARPSourceMACField', '(None)'),
('psrc', 'SourceIPField', '(None)'),
('hwdst', 'MACField', "('00:00:00:00:00:00')"),
('pdst', 'IPField', "('0.0.0.0')")],
.
.
.
'db_md5': 'Z+gRt88y7SC0bDu496/DQg==\n',
'fields': {'ARPSourceMACField': 'empty',
'BCDFloatField': 'empty',
'BitEnumField': 'empty',
.
.
.
}
----
=== Check if Database is updated
* *Name* - 'check_update'
* *Description* - checks if both protocol database and fields database are up to date according to md5 comparison
* *Parameters* - md5 of database, md5 of fields
* *Result* - upon failure: error code -32098 (see link:trex_scapy_rpc_server.html#_error_codes[RPC server error codes]) +
followed by a message: "Fields DB is not up to date" or "Protocol DB is not up to date" +
upon success: return 'true' as result (see below) +
+
*Example:*
[source,python]
----
'Request':
{
"jsonrpc": "2.0",
"id": "1",
"method": "check_update",
"params": ["md5_of_protocol_db", "md5_of_fields"]
}
'Response': #on failure
{
"jsonrpc": "2.0",
"id": "1",
"error": {
"code": -32098,
"message:": "Scapy Server: Fields DB is not up to date"
}
}
'Response': #on success
{
"jsonrpc": "2.0",
"id": "1",
"result": true
}
----
=== Get Version
* *Name* - 'get_version'
* *Description* - Queries the server for version information
* *Paramters* - None
* *Result* ['object'] - See table below
.Object type 'return values for get_version'
[options="header",cols="1,1,3"]
|=================
| Field | Type | Description
| version | string | Scapy Server version
| built_by | string | who built this version
|=================
*Example:*
[source,python]
----
'Request':
{
"jsonrpc": "2.0",
"id": "1",
"method": "get_version",
"params": []
}
'Response':
{
"jsonrpc": "2.0",
"id": "1",
"result": {
"version": "v1.0",
"built_by": "itraviv"
}
}
----
=== Build Packet
* *Name* - 'build_pkt'
* *Description* - Builds a new packet from the definition and returns binary data and json structure +
* *Return Value* - Returns xref:build_pkt_output[Scapy packet result payload].
* *Paramters* - JSON xref:build_pkt_input[packet definition model].
=== Build Packet and VM instructions
* *Name* - 'build_pkt_ex'
* *Description* - Builds a new packet with VM instructions from the definition and returns binary data and json structure +
* *Return Value* - Returns xref:build_pkt_ex_output[Scapy packet result with VM instructions payload].
* *Paramters* - JSON xref:build_pkt_input[Packet definition model].
=== Create packet from binary data and modify fields
* *Name* - 'reconstruct_pkt'
* *Description* - Builds a new packet from the binary data and returns binary data and json structure +
* *Return Value* - Returns xref:build_pkt_output[Scapy packet result payload].
* *Paramters* - base64-encoded packet bytes, optional JSON xref:build_pkt_input[packet definition model] with fields to override.
=== Get protocol definitions
* *Name* - 'get_definitions'
* *Description* - Returns definitions for protocols and fields +
* *Return Value* - array of protocol definitions in a "result.protocols" json. xref:get_definitions_model[Output model]
* *Paramters* - array of protocol class names to define or null to fetch metadata for all protocols. ex. ["Ether", "TCP"]
=== Get protocol tree hierarchy example
* *Name* - 'get_tree'
* *Description* - returns a *suggested* dictionary of protocols ordered in a hierarchy tree. +
User can still create non valid hierarchies. (such as Ether()/DNS()/IP())
* *Parameters* - none
* *Result* [dictionary] - Example for packet layers that can be used to build a packet. Ordered in an hierarchy tree.
*Example:*
[source,python]
----
'Request':
{
"id": "1",
"jsonrpc": "2.0",
"method": "get_tree",
"params": []
}
'Response':
{'id': '1',
'jsonrpc': '2.0',
'result': {'ALL': {
'Ether': {'ARP': {},
'IP': { 'TCP': {'RAW': 'payload'},
'UDP': {'RAW': 'payload'}
}
}
}
}
}
----
=== Get a list of predefined packet templates
* *Name* - 'get_templates'
* *Description* - returns a list of predefined templates.
* *Parameters* - none
* *Result* [array] - A list of predefined templates.
*Example:*
[source,python]
----
[
{
"id": "TCP-SYN",
"meta": {
"name": "TCP-SYN",
"description": ""
},
}
]
----
=== Get predefined packet template
* *Name* - 'get_template'
* *Description* - returns a JSON template encoded in Base64.
* *Parameters* - none
* *Result* [string] - String contains Base64 encoded JSON.
== Usage of Scapy RPC Server
Notice the existence of the following files:
* scapy_service.py
* scapy_zmq_server.py
* scapy_zmq_client.py
=== Scapy_zmq_server.py
In this section we will see how to bring up the Scapy ZMQ server.
There are 2 ways to run this server:
* Through command line
* Through Python interpreter
==== Running Scapy ZMQ Server from command line
Run the file scapy_zmq_server.py with the argument -s to declare the port that the server will listen to. +
Running the file without the "-s" argument will use *port 4507 by default*. +
+
Notice:
* The Server's IP will be the IP address of the local host.
* The Server will accept requests from *any* IP address on that port.
[source,bash]
----
user$ python scapy_zmq_server.py -s 5555
***Scapy Server Started***
Listening on port: 5555
Server IP address: 10.0.0.1
----
==== Running Scapy ZMQ Server from the Python interpreter
* Run the Python Interpreter (Scapy Server currently supports Python2)
* Import the scapy_zmq_server.py file
* Create a Scapy_server Object with argument as port number. default argument is port 4507
* Invoke method activate(). (This method is blocking because the server is listening on the port).
[source,bash]
----
user$ python
>>> from scapy_zmq_server import *
>>> s = Scapy_server() // starts with port 4507
>>> s = Scapy_server(5555) //starts with port 5555
>>> s.activate()
***Scapy Server Started***
Listening on port: 5555
Server IP address: 10.0.0.1
----
==== Shutting down Scapy ZMQ Server
There are 2 ways to shut down the server:
* The server can be shut down using the keyboard interrupt Ctrl+C
* The server can be shut down remotely with the method "shut_down" with no arguments
[source,bash]
----
//Sending Request: {"params": [], "jsonrpc": "2.0", "method": "shut_down", "id": "1"}
//Will result in this print by the server:
Server: Shut down by remote user
----