1. Change log
| Version | name | meaning |
|---|---|---|
1.00 |
Itamar Raviv (itraviv) |
|
2.00 |
Anton Kiselev (kisel) |
|
3.00 |
Vyacheslav Ogai (hedjuo) |
|
2. Audience of this document
TRex GUI guys
3. Scapy RPC Server - Overview
Scapy Server is implemented following the JSON-RPC 2.0 specification,
Therefore, requests and replies follow the JSON-RPC 2.0 spec.
The server operates on a Request-Response basis over ZMQ, and does not support batched commands handling.
Read more about ZMQ here
3.1. Error Codes
Error codes are given according to this table: [also follows the JSON-RPC spec, with added error codes]
| Error Code | Message | Meaning |
|---|---|---|
-32700 |
Parse Error |
|
-32600 |
Invalid Request |
|
-32601 |
Method not found |
|
-32603 |
Invalid params |
|
-32097 |
Syntax Error |
|
-32098 |
Scapy Server: message |
|
-32096 |
Scapy Server: Unknown Error |
|
4. Data Bases and Data Structures used in Scapy Server
4.1. build_pkt, build_pkt_ex, reconstruct_pkt packet model
Following JSON represents a Scapy structure, which can be used to build packet from scratch(build_pkt) or to modify particular fields in the protocol(reconstruct_pkt). Most fields can be omitted, in this case default or calculated values will be used. For reconstruct_pkt default values will be taken from the original packet.
Examples of JSON payloads and their scapy expression alternatives
Ether(src="de:ad:be:ef:de:ad")/Dot1Q()/Dot1Q(vtype=1)/IP(src="127.0.0.1", chksum="0x312")/TCP(sport=443)
[ { "id": "Ether", "fields": [{"id": "src", "value": "de:ad:be:ef:de:ad"}] }, { "id": "Dot1Q"}, { "id": "Dot1Q", "fields": [{"id": "vtype", "value": "1"}] }, { "id": "IP", "fields": [{"id": "src", "value": "127.0.0.1"}, {"id": "chksum", "value": "0x312"}] }, { "id": "TCP", "fields": [{"id": "sport", "value": "443"}] } ]
Field Engine VM instruction model
For more reference see Field Engine modules spec
{ "field_engine":{ "instructions":[ { "id":"STLVmFlowVar", "parameters": { "name": "Ether_dst", "init_value": "00:00:00:01:00:00", #can be a string or an integer "max_value": "0", #can be a string or an integer "min_value": "0", #can be a string or an integer "step": "1", "size": "4", "op": "inc" } }, ... ], "global_parameters":{ "cache_size":"1000" } } }
4.2. Scapy server value types
Most values can be passed as strings(including decimal numbers, hex numbers, enums, values), but for binary payload, value object should be used
- int/long/str - they can de specified directly as a value of a field - {"vtype": "BYTES", "base64": "my_payload_base64"} - binary payload passed as base64 - {"vtype": "EXPRESSION", "expr": "TCPOptions()"} - python expression(normally, should be avoided) - {"vtype": "UNDEFINED"} - unset field value, and let it be assigned automatically - {"vtype": "RANDOM"} - assign a random value to a field
Example of object value usage(to specify binary payload)
Ether()/IP()/TCP()/Raw(load=my_payload)
[ { "id": "Ether"}, { "id": "IP"}, { "id": "TCP"}, { "id": "Raw", "fields": [ { "id": "load", "value": {"vtype": "BYTES", "base64": "my_payload_base64"} } ]} ]
4.3. Scapy packet result payload
build_pkt, build_pkt_ex and reconstruct pkt take packet model and produce result JSON, with the binary payload and field values and offsets defined
{ "binary": "AAAAAQAAAAAAAgAACABFAAAoAAEAAEAGOs4QAAABMAAAAQAUAFAAAAAAAAAAAFACIABPfQAA", // base64 encoded binary payload "data": [ { "id": "Ether", # scapy class "name": "Ethernet", # human-readable protocol name "offset": 0, # global offset for all fields "fields": [ { "id": "dst", # scapy field id "hvalue": "00:00:00:01:00:00", # human readable value "length": 6, # 6 bytes "offset": 0, # 0 bytes offset from "value": "00:00:00:01:00:00" # internal value, which for this type is the same as hvalue }, { "id": "src", ... # same as for dst }, { "hvalue": "IPv4", # human-readable value "id": "type", "length": 2, "offset": 12, # "value": 2048 # integer value for IPv4(0x800) } ] }, { "id": "IP", "name": "IP", "offset": 14, "fields": [ { "hvalue": "4", "id": "version", "length": 0, # the length is 0, which means it is a bitfield. mask should be used to show location "offset": 0, # offset from the IP.offset. it needs to be added to all fields of IP "value": 4 }, { "hvalue": "5", "id": "ihl", "length": 0, # again length is 0. that's other part of the first byte of IP "offset": 0, "value": 5 }, { "hvalue": "0x0", "id": "tos", "length": 1, "offset": 1, "value": 0 }, { "hvalue": "40", "id": "len", "length": 2, "offset": 2, "value": 40 }, { "hvalue": "1", "id": "id", "length": 2, "offset": 4, "value": 1 }, { "hvalue": "", # no flags are specified here. but this field can contain "US" for URG+SYN flags "id": "flags", "length": 0, "offset": 6, "value": 0 }, { "hvalue": "0", "id": "frag", "length": 0, "offset": 6, "value": 0 }, { "hvalue": "64", "id": "ttl", "length": 1, "offset": 8, "value": 64 }, { "hvalue": "tcp", # this field is enum. enum dictionary can be obtained as a medatata for IP fields. "id": "proto", "length": 1, "offset": 9, "value": 6 }, { "hvalue": "0x3ace", "id": "chksum", "length": 2, "offset": 10, "value": 15054 }, { "hvalue": "[]", "id": "options", "length": 2, "offset": 20, "value": { # options can not be represented as a human string, so they are passed as an expression "expr": "[]", "vtype": "EXPRESSION" } } ] }, { "id": "TCP", "name": "TCP", "offset": 34 "fields": [ { "hvalue": "20", "id": "sport", "length": 2, "offset": 0, "value": 20 }, # .. some more TCP fields here { "hvalue": "{}", "id": "options", "ignored": true, "length": 2, "offset": 20, "value": { # TCPOptions are represented as a python expression with tuple and binary buffers "expr": "[('MSS', 1460), ('NOP', None), ('NOP', None), ('SAckOK', b'')]", "vtype": "EXPRESSION" } } ] } ] }
4.4. Scapy packet result payload in extended mode
{ "binary": #same as in build_pkt "data": #same as in build_pkt "field_engine": { "error": "", "instructions": { "cache": 1000, "instructions": [ ... #FlowVar definition instruction { "name": "Ether_dst", "max_value": 0, "min_value": 0, "init_value": 65536, "step": 1, "size": 4, "type": "flow_var", "op": "inc" }, #FlowVar write instruction { "name": "Ether_dst", "is_big_endian": true, "pkt_offset": 6, "type": "write_flow_var", "add_value": 0 } ... ] } } }
4.5. Scapy server field definitions
Scapy server can return metadata object, describing protocols and fields. Most values, including field types are optional in the definition. If field type is missing, it can be treated as a STRING.
"protocols": [ { "id": "Ether", # scapy class "name": "Ethernet", # name of the protocol "fields": [ { "id": "dst", "name": "Destination", # GUI will display Destination instead of dst "type": "STRING", "regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$" }, { "id": "src", "name": "Source", "type": "STRING", "regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$" }, { "values_dict": { "ATMMPOA": 34892, "RAW_FR": 25945, "DNA_DL": 24577, "ATMFATE": 34948, "ATALK": 32923, "BPQ": 2303, "X25": 2053, "PPP_DISC": 34915, "DEC": 24576, "n_802_1Q": 33024, "PPP_SES": 34916, "TEB": 25944, "SCA": 24583, "PPP": 34827, "FR_ARP": 2056, "CUST": 24582, "ARP": 2054, "DNA_RC": 24578, "NetBEUI": 33169, "AARP": 33011, "DIAG": 24581, "IPv4": 2048, "DNA_RT": 24579, "IPv6": 34525, "LAT": 24580, "IPX": 33079, "LOOP": 36864 }, "id": "type", "name": "Type" "type": "ENUM" } ] }, { "id": "TCP", "name": "TCP", "fields": [ { "id": "sport", "name": "Source port", "type": "NUMBER", "min": 0, # optional min value "max": 65535 # optional max value }, { "id": "dport", "name": "Destination port", "type": "NUMBER", "min": 0, "max": 65535 }, { "id": "seq", "name": "Sequence number", "type": "NUMBER" }, { "id": "ack", "name": "Acknowledgment number", "type": "NUMBER" }, { "id": "dataofs", "name": "Data offset", "type": "NUMBER" }, { "id": "reserved", "name": "Reserved", "type": "NUMBER" }, { "id": "flags", "name": "Flags", "auto": false, "type": "BITMASK", "bits": [ # fields definition for the UI {"name": "URG", "mask": 32, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 32}]}, {"name": "ACK", "mask": 16, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 16}]}, {"name": "PSH", "mask": 8, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 8}]}, {"name": "RST", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]}, {"name": "SYN", "mask": 2, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 2}]}, {"name": "FIN", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]} ] }, { "id": "window", "name": "Window size", "type": "NUMBER" }, { "id": "chksum", "name": "Checksum", "auto": true, "type": "NUMBER" }, { "id": "urgptr", "name": "Urgent pointer", "type": "NUMBER" }, { "id": "options", "name": "Options", "type": "EXPRESSION" } ] }, { "id": "IP", "name": "Internet Protocol Version 4", "fields": [ { "id": "version", # only renaming "name": "Version" }, { "id": "ihl", "name": "IHL", "type": "NUMBER", "auto": true # calculate IHL automatically }, { "id": "tos", "name": "TOS", "type": "NUMBER" }, { "id": "len", "name": "Total Length", "type": "NUMBER", "auto": true }, { "id": "id", "name": "Identification", "type": "NUMBER" }, { "id": "flags", "name": "Flags", "type": "BITMASK", "min": 0, "max": 8, "bits": [ # bitmask definition for IP.flags {"name": "Reserved", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]}, {"name": "Fragment", "mask": 2, "values":[{"name":"May fragment (0)", "value": 0}, {"name":"Don't fragment (1)", "value": 2}]}, {"name": "More Fragments(MF)", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]} ] }, { "id": "frag", "name": "Fragment offset", "type": "NUMBER" }, { "id": "ttl", "name": "TTL", "type": "NUMBER", "min": 1, "max": 255 }, { "id": "proto", "name": "Protocol" }, { "id": "chksum", "name": "Checksum", "type": "STRING", "auto": true }, { "id": "src", "name": "Source address", "type": "STRING", "regexp": "regexp-to-check-this-field" }, { "id": "dst", "name": "Destination address", "regexp": "regexp-to-check-this-field" }, { "id": "options", "name": "Options", "type": "EXPRESSION" } ] }, { "id": "Dot1Q", "name": "802.1Q", "fields": [ { "id": "prio", "name": "prio" "type": "NUMBER", }, { "id": "id", "type": "NUMBER", "name": "id" }, { "id": "vlan", "type": "NUMBER", "name": "vlan" }, { "values_dict": { "ATMMPOA": 34892, "RAW_FR": 25945, "DNA_DL": 24577, "ATMFATE": 34948, "ATALK": 32923, "BPQ": 2303, "X25": 2053, "PPP_DISC": 34915, "DEC": 24576, "n_802_1Q": 33024, "PPP_SES": 34916, "TEB": 25944, "SCA": 24583, "PPP": 34827, "FR_ARP": 2056, "CUST": 24582, "ARP": 2054, "DNA_RC": 24578, "NetBEUI": 33169, "AARP": 33011, "DIAG": 24581, "IPv4": 2048, "DNA_RT": 24579, "IPv6": 34525, "LAT": 24580, "IPX": 33079, "LOOP": 36864 }, "id": "type", "name": "type", "type": "ENUM" } ] }, { "id": "Raw", "name": "Raw", "fields": [ { "id": "load", "name": "Payload", "type": "BYTES" } ] } ] ]
5. RPC Commands
The following RPC commands are supported. Please refer to databases section for elaboration for each database.
5.1. Supported Methods
-
Name - supported_methods
-
Description - returns the list of all supported methods by Scapy Server and their parameters
-
Parameters - the parameter (all) will return ALL supported methods.
other string delivered as parameter will return True/False if the string matches a supported method name -
Result - according to input: all string will return list of supported methods, otherwise will return True/False as mentioned.
The returned dictionary describes for each method it’s number of parameters followed by a list of their names.
Example:
'Request': { "jsonrpc": "2.0", "id": "1", "method": "supported_methods", "params": ["all"] } 'Result': {'id': '1', 'jsonrpc': '2.0', 'result': { . . . . 'build_pkt': [1, [u'pkt_descriptor']], 'check_update': [2, [u'db_md5', u'field_md5']], 'get_all': [0, []], 'get_tree': [0, []], 'get_version': [0, []], 'supported_methods': [1, [u'method_name']] } }
5.2. GetAll
-
Name - get_all
-
Description - Returns the supported protocols library (DB) and Field-to-RegEx mapping library, and their MD5
-
Parameters - None
-
Result [object] - JSON format of dictionary. see table below
| Key | Key Type | Value | Value Type |
|---|---|---|---|
db |
string |
supported protocols dictionary |
protocol dictionary |
fields |
string |
Field-to-RegEx dictionary |
Field-to-RegEx dictionary |
db_md5 |
string |
MD5 of DB |
encoded in base64 |
fields_md5 |
string |
MD5 of fields |
encoded in base64 |
Example:
'Request': { "jsonrpc": "2.0", "id": 1, "method": "get_all", "params": [] } 'Response': { "jsonrpc" : "2.0", "id" : 1, "result" : {'db': {'ARP': [('hwtype', 'XShortField', '(1)'), ('ptype', 'XShortEnumField', '(2048)'), ('hwlen', 'ByteField', '(6)'), ('plen', 'ByteField', '(4)'), ('op', 'ShortEnumField', '(1)'), ('hwsrc', 'ARPSourceMACField', '(None)'), ('psrc', 'SourceIPField', '(None)'), ('hwdst', 'MACField', "('00:00:00:00:00:00')"), ('pdst', 'IPField', "('0.0.0.0')")], . . . 'db_md5': 'Z+gRt88y7SC0bDu496/DQg==\n', 'fields': {'ARPSourceMACField': 'empty', 'BCDFloatField': 'empty', 'BitEnumField': 'empty', . . . }
5.3. Check if Database is updated
-
Name - check_update
-
Description - checks if both protocol database and fields database are up to date according to md5 comparison
-
Parameters - md5 of database, md5 of fields
-
Result - upon failure: error code -32098 (see RPC server error codes)
followed by a message: "Fields DB is not up to date" or "Protocol DB is not up to date"
upon success: return true as result (see below)
Example:
'Request': { "jsonrpc": "2.0", "id": "1", "method": "check_update", "params": ["md5_of_protocol_db", "md5_of_fields"] } 'Response': #on failure { "jsonrpc": "2.0", "id": "1", "error": { "code": -32098, "message:": "Scapy Server: Fields DB is not up to date" } } 'Response': #on success { "jsonrpc": "2.0", "id": "1", "result": true }
5.4. Get Version
-
Name - get_version
-
Description - Queries the server for version information
-
Parameters - None
-
Result [object] - See table below
| Field | Type | Description |
|---|---|---|
version |
string |
Scapy Server version |
built_by |
string |
who built this version |
Example:
'Request': { "jsonrpc": "2.0", "id": "1", "method": "get_version", "params": [] } 'Response': { "jsonrpc": "2.0", "id": "1", "result": { "version": "v1.0", "built_by": "itraviv" } }
5.5. Build Packet
-
Name - build_pkt
-
Description - Builds a new packet from the definition and returns binary data and json structure
-
Return Value - Returns Scapy packet result payload.
-
Parameters - JSON packet definition model.
5.6. Build Packet and VM instructions
-
Name - build_pkt_ex
-
Description - Builds a new packet with VM instructions from the definition and returns binary data and json structure
-
Return Value - Returns Scapy packet result with VM instructions payload.
-
Parameters - JSON Packet definition model.
5.7. Create packet from binary data and modify fields
-
Name - reconstruct_pkt
-
Description - Builds a new packet from the binary data and returns binary data and json structure
-
Return Value - Returns Scapy packet result payload.
-
Parameters - base64-encoded packet bytes, optional JSON packet definition model with fields to override.
5.8. Get protocol definitions
-
Name - get_definitions
-
Description - Returns definitions for protocols and fields
-
Return Value - array of protocol definitions in a "result.protocols" json. Output model
-
Parameters - array of protocol class names to define or null to fetch metadata for all protocols. ex. ["Ether", "TCP"]
5.9. Get protocol tree hierarchy example
-
Name - get_tree
-
Description - returns a suggested dictionary of protocols ordered in a hierarchy tree.
User can still create non valid hierarchies. (such as Ether()/DNS()/IP()) -
Parameters - none
-
Result [dictionary] - Example for packet layers that can be used to build a packet. Ordered in an hierarchy tree.
Example:
'Request': { "id": "1", "jsonrpc": "2.0", "method": "get_tree", "params": [] } 'Response': {'id': '1', 'jsonrpc': '2.0', 'result': {'ALL': { 'Ether': {'ARP': {}, 'IP': { 'TCP': {'RAW': 'payload'}, 'UDP': {'RAW': 'payload'} } } } } }
5.10. Get a list of predefined packet templates
-
Name - get_templates
-
Description - returns a list of predefined templates.
-
Parameters - none
-
Result [array] - A list of predefined templates.
Example:
[ { "id": "TCP-SYN", "meta": { "name": "TCP-SYN", "description": "" }, } ]
5.11. Get predefined packet template
-
Name - get_template
-
Description - returns a JSON template encoded in Base64.
-
Parameters - none
-
Result [string] - String contains Base64 encoded JSON.
6. Usage of Scapy RPC Server
Notice the existence of the following files:
-
scapy_service.py
-
scapy_zmq_server.py
-
scapy_zmq_client.py
6.1. Scapy_zmq_server.py
In this section we will see how to bring up the Scapy ZMQ server. There are 2 ways to run this server:
-
Through command line
-
Through Python interpreter
6.1.1. Running Scapy ZMQ Server from command line
Run the file scapy_zmq_server.py with the argument -s to declare the port that the server will listen to.
Running the file without the "-s" argument will use port 4507 by default.
Notice:
-
The Server’s IP will be the IP address of the local host.
-
The Server will accept requests from any IP address on that port.
user$ python scapy_zmq_server.py -s 5555 ***Scapy Server Started*** Listening on port: 5555 Server IP address: 10.0.0.1
6.1.2. Running Scapy ZMQ Server from the Python interpreter
-
Run the Python Interpreter
-
Import the scapy_zmq_server.py file
-
Create a Scapy_server Object with argument as port number. default argument is port 4507
-
Invoke method activate(). (This method is blocking because the server is listening on the port).
user$ python >>> from scapy_zmq_server import * >>> s = Scapy_server() // starts with port 4507 >>> s = Scapy_server(5555) //starts with port 5555 >>> s.activate() ***Scapy Server Started*** Listening on port: 5555 Server IP address: 10.0.0.1
6.1.3. Shutting down Scapy ZMQ Server
There are 2 ways to shut down the server:
-
The server can be shut down using the keyboard interrupt Ctrl+C
-
The server can be shut down remotely with the method "shut_down" with no arguments
//Sending Request: {"params": [], "jsonrpc": "2.0", "method": "shut_down", "id": "1"} //Will result in this print by the server: Server: Shut down by remote user
Table of Contents