Table of Contents

1. Change log

Version name meaning

1.00

Itamar Raviv (itraviv)

first version

2.00

Anton Kiselev (kisel)

Add description for updated compatibilities of packet creation and modification

3.00

Vyacheslav Ogai (hedjuo)

Add description for supporting Filed Engine and predefined packet templates

2. Audience of this document

TRex GUI guys

3. Scapy RPC Server - Overview

Scapy Server is implemented following the JSON-RPC 2.0 specification,
Therefore, requests and replies follow the JSON-RPC 2.0 spec.
The server operates on a Request-Response basis over ZMQ, and does not support batched commands handling.

Read more about ZMQ here

Figure 1. Scapy JSON RPC Server

3.1. Error Codes

Error codes are given according to this table: [also follows the JSON-RPC spec, with added error codes]

Error Code Message Meaning

-32700

Parse Error

Invalid JSON was received by the server. An error occurred on the server while parsing the JSON input.

-32600

Invalid Request

The JSON sent is not a valid Request object.

-32601

Method not found

The method does not exist / is not available

-32603

Invalid params

Invalid method parameter(s)

-32097

Syntax Error

Syntax Error in input

-32098

Scapy Server: message

Scapy Server had an error while executing your command, described in the message given

-32096

Scapy Server: Unknown Error

Scapy Server encountered an error that cannot be described

4. Data Bases and Data Structures used in Scapy Server

4.1. build_pkt, build_pkt_ex, reconstruct_pkt packet model

Following JSON represents a Scapy structure, which can be used to build packet from scratch(build_pkt) or to modify particular fields in the prococol(reconstruct_pkt). Most fields can be omitted, in this case default or calculated values will be used. For reconstruct_pkt default values will be taken from the original packet.

Exaples of JSON payloads and their scapy expression alternatives

Ether(src="de:ad:be:ef:de:ad")/Dot1Q()/Dot1Q(vtype=1)/IP(src="127.0.0.1", chksum="0x312")/TCP(sport=443)
[
        { "id": "Ether", "fields": [{"id": "src", "value": "de:ad:be:ef:de:ad"}] },
        { "id": "Dot1Q"},
        { "id": "Dot1Q", "fields": [{"id": "vtype", "value": "1"}] },
        { "id": "IP", "fields": [{"id": "src", "value": "127.0.0.1"}, {"id": "chksum", "value": "0x312"}] },
        { "id": "TCP", "fields": [{"id": "sport", "value": "443"}] }
]

Field Engine VM instruction model
For more reference see Field Engine modules spec

{
    "field_engine":{
        "instructions":[
            {
                "id":"STLVmFlowVar",
                "parameters": {
                    "name": "Ether_dst",
                    "init_value": "00:00:00:01:00:00", #can be a string or an integer
                    "max_value": "0",                  #can be a string or an integer
                    "min_value": "0",                  #can be a string or an integer
                    "step": "1",
                    "size": "4",
                    "op": "inc"
                }
            },
            ...
        ],
        "global_parameters":{
            "cache_size":"1000"
        }
    }
}

4.2. Scapy server value types

Most values can be passed as strings(including decimal numbers, hex numbers, enums, values), but for binary payload, value object should be used

- int/long/str - they can de specified directly as a value of a field
- {"vtype": "BYTES", "base64": "my_payload_base64"} - binary payload passed as base64
- {"vtype": "EXPRESSION", "expr": "TCPOptions()"} - python expression(normally, should be avoided)
- {"vtype": "UNDEFINED"} - unset field value, and let it be assigned automatically
- {"vtype": "RANDOM"} - assign a random value to a field

Example of object value usage(to specify binary payload)

Ether()/IP()/TCP()/Raw(load=my_payload)
[
        { "id": "Ether"},
        { "id": "IP"},
        { "id": "TCP"},
        { "id": "Raw", "fields": [
            {
              "id": "load",
              "value": {"vtype": "BYTES", "base64": "my_payload_base64"}
            }
        ]}
]

4.3. Scapy packet result payload

build_pkt, build_pkt_ex and reconstruct pkt take packet model and produce result JSON, with the binary payload and field values and offsets defined

{
    "binary": "AAAAAQAAAAAAAgAACABFAAAoAAEAAEAGOs4QAAABMAAAAQAUAFAAAAAAAAAAAFACIABPfQAA", // base64 encoded binary payload
    "data": [
        {
            "id": "Ether",      # scapy class
            "name": "Ethernet", # human-readable protocol name
            "offset": 0,        # global offset for all fields
            "fields": [
                {
                    "id": "dst", # scapy field id
                    "hvalue": "00:00:00:01:00:00", # human readable value
                    "length": 6, # 6 bytes
                    "offset": 0, # 0 bytes offset from
                    "value": "00:00:00:01:00:00" # internal value, which for this type is the same as hvalue
                },
                {
                    "id": "src",
                    ... # same as for dst
                },
                {
                    "hvalue": "IPv4", # human-readable value
                    "id": "type",
                    "length": 2,
                    "offset": 12, #
                    "value": 2048 # integer value for IPv4(0x800)
                }
            ]
        },
        {
            "id": "IP",
            "name": "IP",
            "offset": 14,
            "fields": [
                {
                    "hvalue": "4",
                    "id": "version",
                    "length": 0, # the length is 0, which means it is a bitfield. mask should be used to show location
                    "offset": 0, # offset from the IP.offset. it needs to be added to all fields of IP
                    "value": 4
                },
                {
                    "hvalue": "5",
                    "id": "ihl",
                    "length": 0, # again length is 0. that's other part of the first byte of IP
                    "offset": 0,
                    "value": 5
                },
                {
                    "hvalue": "0x0",
                    "id": "tos",
                    "length": 1,
                    "offset": 1,
                    "value": 0
                },
                {
                    "hvalue": "40",
                    "id": "len",
                    "length": 2,
                    "offset": 2,
                    "value": 40
                },
                {
                    "hvalue": "1",
                    "id": "id",
                    "length": 2,
                    "offset": 4,
                    "value": 1
                },
                {
                    "hvalue": "", # no flags are specified here. but this field can contain "US" for URG+SYN flags
                    "id": "flags",
                    "length": 0,
                    "offset": 6,
                    "value": 0
                },
                {
                    "hvalue": "0",
                    "id": "frag",
                    "length": 0,
                    "offset": 6,
                    "value": 0
                },
                {
                    "hvalue": "64",
                    "id": "ttl",
                    "length": 1,
                    "offset": 8,
                    "value": 64
                },
                {
                    "hvalue": "tcp", # this field is enum. enum dictionary can be obtained as a medatata for IP fields.
                    "id": "proto",
                    "length": 1,
                    "offset": 9,
                    "value": 6
                },
                {
                    "hvalue": "0x3ace",
                    "id": "chksum",
                    "length": 2,
                    "offset": 10,
                    "value": 15054
                },
                {
                    "hvalue": "[]",
                    "id": "options",
                    "length": 2,
                    "offset": 20,
                    "value": { # options can not be representted as a human string, so they are passed as an expression
                        "expr": "[]",
                        "vtype": "EXPRESSION"
                    }
                }
            ]
        },
        {
            "id": "TCP",
            "name": "TCP",
            "offset": 34
            "fields": [
                {
                    "hvalue": "20",
                    "id": "sport",
                    "length": 2,
                    "offset": 0,
                    "value": 20
                },
                # .. some more TCP fields here
                {
                    "hvalue": "{}",
                    "id": "options",
                    "ignored": true,
                    "length": 2,
                    "offset": 20,
                    "value": { # TCPOptions are represented as a python expression with tuple and binary buffers
                        "expr": "[('MSS', 1460), ('NOP', None), ('NOP', None), ('SAckOK', b'')]",
                        "vtype": "EXPRESSION"
                    }
                }
            ]
        }
    ]
}

4.4. Scapy packet result payload in extended mode

{
    "binary": #same as in build_pkt
    "data": #same as in build_pkt
    "field_engine": {
        "error": "",
        "instructions": {
            "cache": 1000,
            "instructions": [
                ...
                #FlowVar definition instruction
                {
                    "name": "Ether_dst",
                    "max_value": 0,
                    "min_value": 0,
                    "init_value": 65536,
                    "step": 1,
                    "size": 4,
                    "type": "flow_var",
                    "op": "inc"
                },
                #FlowVar write instruction
                {
                    "name": "Ether_dst",
                    "is_big_endian": true,
                    "pkt_offset": 6,
                    "type": "write_flow_var",
                    "add_value": 0
                }
                ...
            ]
        }
    }
}

4.5. Scapy server field definitions

Scapy server can return metadata object, describing protocols and fields. Most values, including field types are optional in the definition. If field type is missing, it can be treated as a STRING.

"protocols": [
{
    "id": "Ether", # scapy class
        "name": "Ethernet", # name of the protocol
        "fields": [
        {
            "id": "dst",
            "name": "Destination", # GUI will display Destination instead of dst
            "type": "STRING",
            "regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$"
        },
        {
            "id": "src",
            "name": "Source",
            "type": "STRING",
            "regex": "^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$"
        },
        {
            "values_dict": {
                "ATMMPOA": 34892,
                "RAW_FR": 25945,
                "DNA_DL": 24577,
                "ATMFATE": 34948,
                "ATALK": 32923,
                "BPQ": 2303,
                "X25": 2053,
                "PPP_DISC": 34915,
                "DEC": 24576,
                "n_802_1Q": 33024,
                "PPP_SES": 34916,
                "TEB": 25944,
                "SCA": 24583,
                "PPP": 34827,
                "FR_ARP": 2056,
                "CUST": 24582,
                "ARP": 2054,
                "DNA_RC": 24578,
                "NetBEUI": 33169,
                "AARP": 33011,
                "DIAG": 24581,
                "IPv4": 2048,
                "DNA_RT": 24579,
                "IPv6": 34525,
                "LAT": 24580,
                "IPX": 33079,
                "LOOP": 36864
            },
            "id": "type",
            "name": "Type"
                "type": "ENUM"
        }
    ]
},
{
    "id": "TCP",
    "name": "TCP",
    "fields": [
    {
        "id": "sport",
        "name": "Source port",
        "type": "NUMBER",
        "min": 0, # optional min value
        "max": 65535 # optional max value

    },
    {
        "id": "dport",
        "name": "Destination port",
        "type": "NUMBER",
        "min": 0,
        "max": 65535
    },
    {
        "id": "seq",
        "name": "Sequence number",
        "type": "NUMBER"
    },
    {
        "id": "ack",
        "name": "Acknowledgment number",
        "type": "NUMBER"
    },
    {
        "id": "dataofs",
        "name": "Data offset",
        "type": "NUMBER"
    },
    {
        "id": "reserved",
        "name": "Reserved",
        "type": "NUMBER"
    },
    {
        "id": "flags",
        "name": "Flags",
        "auto": false,
        "type": "BITMASK",
        "bits": [ # fields definition for the UI
        {"name": "URG", "mask": 32, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 32}]},
        {"name": "ACK", "mask": 16, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 16}]},
        {"name": "PSH", "mask": 8, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 8}]},
        {"name": "RST", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]},
        {"name": "SYN", "mask": 2, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 2}]},
        {"name": "FIN", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]}
        ]
    },
    {
        "id": "window",
        "name": "Window size",
        "type": "NUMBER"
    },
    {
        "id": "chksum",
        "name": "Checksum",
        "auto": true,
        "type": "NUMBER"
    },
    {
        "id": "urgptr",
        "name": "Urgent pointer",
        "type": "NUMBER"
    },
    {
        "id": "options",
        "name": "Options",
        "type": "EXPRESSION"
    }
    ]
},
{
    "id": "IP",
    "name": "Internet Protocol Version 4",
    "fields": [
    {
        "id": "version", # only renaming
        "name": "Version"
    },
    {
        "id": "ihl",
        "name": "IHL",
        "type": "NUMBER",
        "auto": true # calculate IHL automatically
    },
    {
        "id": "tos",
        "name": "TOS",
        "type": "NUMBER"
    },
    {
        "id": "len",
        "name": "Total Length",
        "type": "NUMBER",
        "auto": true
    },
    {
        "id": "id",
        "name": "Identification",
        "type": "NUMBER"
    },
    {
        "id": "flags",
        "name": "Flags",
        "type": "BITMASK",
        "min": 0,
        "max": 8,
        "bits": [ # bitmask definition for IP.flags
        {"name": "Reserved", "mask": 4, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 4}]},
        {"name": "Fragment", "mask": 2, "values":[{"name":"May fragment (0)", "value": 0}, {"name":"Don't fragment (1)", "value": 2}]},
        {"name": "More Fragments(MF)", "mask": 1, "values":[{"name":"Not Set", "value": 0}, {"name":"Set", "value": 1}]}
        ]
    },
    {
        "id": "frag",
        "name": "Fragment offset",
        "type": "NUMBER"
    },
    {
        "id": "ttl",
        "name": "TTL",
        "type": "NUMBER",
        "min": 1,
        "max": 255

    },
    {
        "id": "proto",
        "name": "Protocol"
    },
    {
        "id": "chksum",
        "name": "Checksum",
        "type": "STRING",
        "auto": true
    },
    {
        "id": "src",
        "name": "Source address",
        "type": "STRING",
        "regexp": "regexp-to-check-this-field"
    },
    {
        "id": "dst",
        "name": "Destination address",
        "regexp": "regexp-to-check-this-field"
    },
    {
        "id": "options",
        "name": "Options",
        "type": "EXPRESSION"
    }
    ]
},
{
    "id": "Dot1Q",
    "name": "802.1Q",
    "fields": [
    {
        "id": "prio",
        "name": "prio"
            "type": "NUMBER",
    },
    {
        "id": "id",
        "type": "NUMBER",
        "name": "id"
    },
    {
        "id": "vlan",
        "type": "NUMBER",
        "name": "vlan"
    },
    {
        "values_dict": {
            "ATMMPOA": 34892,
            "RAW_FR": 25945,
            "DNA_DL": 24577,
            "ATMFATE": 34948,
            "ATALK": 32923,
            "BPQ": 2303,
            "X25": 2053,
            "PPP_DISC": 34915,
            "DEC": 24576,
            "n_802_1Q": 33024,
            "PPP_SES": 34916,
            "TEB": 25944,
            "SCA": 24583,
            "PPP": 34827,
            "FR_ARP": 2056,
            "CUST": 24582,
            "ARP": 2054,
            "DNA_RC": 24578,
            "NetBEUI": 33169,
            "AARP": 33011,
            "DIAG": 24581,
            "IPv4": 2048,
            "DNA_RT": 24579,
            "IPv6": 34525,
            "LAT": 24580,
            "IPX": 33079,
            "LOOP": 36864
        },
        "id": "type",
        "name": "type",
        "type": "ENUM"
    }
    ]
},
{
    "id": "Raw",
    "name": "Raw",
    "fields": [
    {
        "id": "load",
        "name": "Payload",
        "type": "BYTES"
    }
    ]
}
]

]

5. RPC Commands

The following RPC commands are supported. Please refer to databases section for elaboration for each database.

5.1. Supported Methods

  • Name - supported_methods

  • Description - returns the list of all supported methods by Scapy Server and their parameters

  • Parameters - the parameter (all) will return ALL supported methods.
    other string delivered as parameter will return True/False if the string matches a supported method name

  • Result - according to input: all string will return list of supported methods, otherwise will return True/False as mentioned.
    The returned dictionary describes for each method it’s number of parameters followed by a list of their names.

Example:

'Request':
{
        "jsonrpc": "2.0",
        "id": "1",
        "method": "supported_methods",
        "params": ["all"]
}

'Result':
{'id': '1',
 'jsonrpc': '2.0',
 'result': { .
             .
             .
             .
             'build_pkt': [1, [u'pkt_descriptor']],
             'check_update': [2, [u'db_md5', u'field_md5']],
             'get_all': [0, []],
             'get_tree': [0, []],
             'get_version': [0, []],
             'supported_methods': [1, [u'method_name']]
           }
}

5.2. GetAll

  • Name - get_all

  • Description - Returns the supported protocols library (DB) and Field-to-RegEx mapping library, and their MD5

  • Paramters - None

  • Result [object] - JSON format of dictionary. see table below

Table 1. Object type return values for get_all
Key Key Type Value Value Type

db

string

supported protocols dictionary

protocol dictionary

fields

string

Field-to-RegEx dictionary

Field-to-RegEx dictionary

db_md5

string

MD5 of DB

encoded in base64

fields_md5

string

MD5 of fields

encoded in base64

Example:

'Request':
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "get_all",
    "params": []
}

'Response':
{
   "jsonrpc" : "2.0",
   "id" : 1,
   "result" : {'db': {'ARP': [('hwtype', 'XShortField', '(1)'),
                              ('ptype', 'XShortEnumField', '(2048)'),
                              ('hwlen', 'ByteField', '(6)'),
                              ('plen', 'ByteField', '(4)'),
                              ('op', 'ShortEnumField', '(1)'),
                              ('hwsrc', 'ARPSourceMACField', '(None)'),
                              ('psrc', 'SourceIPField', '(None)'),
                              ('hwdst', 'MACField', "('00:00:00:00:00:00')"),
                              ('pdst', 'IPField', "('0.0.0.0')")],
                              .
                              .
                              .
               'db_md5': 'Z+gRt88y7SC0bDu496/DQg==\n',
               'fields': {'ARPSourceMACField': 'empty',
                          'BCDFloatField': 'empty',
                          'BitEnumField': 'empty',
                          .
                          .
                          .

}

5.3. Check if Database is updated

  • Name - check_update

  • Description - checks if both protocol database and fields database are up to date according to md5 comparison

  • Parameters - md5 of database, md5 of fields

  • Result - upon failure: error code -32098 (see RPC server error codes)
    followed by a message: "Fields DB is not up to date" or "Protocol DB is not up to date"
    upon success: return true as result (see below)

    Example:

'Request':

{
        "jsonrpc": "2.0",
        "id": "1",
        "method": "check_update",
        "params": ["md5_of_protocol_db", "md5_of_fields"]
}

'Response': #on failure

{
        "jsonrpc": "2.0",
        "id": "1",
        "error": {
                  "code": -32098,
                  "message:": "Scapy Server: Fields DB is not up to date"
                 }
}

'Response': #on success

{
        "jsonrpc": "2.0",
        "id": "1",
        "result": true
}

5.4. Get Version

  • Name - get_version

  • Description - Queries the server for version information

  • Paramters - None

  • Result [object] - See table below

Table 2. Object type return values for get_version
Field Type Description

version

string

Scapy Server version

built_by

string

who built this version

Example:


'Request':

{
        "jsonrpc": "2.0",
        "id": "1",
        "method": "get_version",
        "params": []
}



'Response':

{
        "jsonrpc": "2.0",
        "id": "1",
        "result": {
                "version": "v1.0",
                "built_by": "itraviv"
        }
}

5.5. Build Packet

5.6. Build Packet and VM instructions

5.7. Create packet from binary data and modify fields

  • Name - reconstruct_pkt

  • Description - Builds a new packet from the binary data and returns binary data and json structure

  • Return Value - Returns Scapy packet result payload.

  • Paramters - base64-encoded packet bytes, optional JSON packet definition model with fields to override.

5.8. Get protocol definitions

  • Name - get_definitions

  • Description - Returns definitions for protocols and fields

  • Return Value - array of protocol definitions in a "result.protocols" json. Output model

  • Paramters - array of protocol class names to define or null to fetch metadata for all protocols. ex. ["Ether", "TCP"]

5.9. Get protocol tree hierarchy example

  • Name - get_tree

  • Description - returns a suggested dictionary of protocols ordered in a hierarchy tree.
    User can still create non valid hierarchies. (such as Ether()/DNS()/IP())

  • Parameters - none

  • Result [dictionary] - Example for packet layers that can be used to build a packet. Ordered in an hierarchy tree.

Example:


'Request':

{
        "id": "1",
        "jsonrpc": "2.0",
        "method": "get_tree",
        "params": []
}


'Response':

{'id': '1',
 'jsonrpc': '2.0',
 'result': {'ALL': {
                        'Ether': {'ARP': {},
                                  'IP': {   'TCP': {'RAW': 'payload'},
                                            'UDP': {'RAW': 'payload'}
                                        }
                                 }
                    }
           }
}

5.10. Get a list of predefined packet templates

  • Name - get_templates

  • Description - returns a list of predefined templates.

  • Parameters - none

  • Result [array] - A list of predefined templates.

Example:

[
    {
        "id": "TCP-SYN",
        "meta": {
            "name": "TCP-SYN",
            "description": ""
        },
    }
]

5.11. Get predefined packet template

  • Name - get_template

  • Description - returns a JSON template encoded in Base64.

  • Parameters - none

  • Result [string] - String contains Base64 encoded JSON.

6. Usage of Scapy RPC Server

Notice the existence of the following files:

  • scapy_service.py

  • scapy_zmq_server.py

  • scapy_zmq_client.py

6.1. Scapy_zmq_server.py

In this section we will see how to bring up the Scapy ZMQ server. There are 2 ways to run this server:

  • Through command line

  • Through Python interpreter

6.1.1. Running Scapy ZMQ Server from command line

Run the file scapy_zmq_server.py with the argument -s to declare the port that the server will listen to.
Running the file without the "-s" argument will use port 4507 by default.

Notice:

  • The Server’s IP will be the IP address of the local host.

  • The Server will accept requests from any IP address on that port.

user$ python scapy_zmq_server.py -s 5555

***Scapy Server Started***
Listening on port: 5555
Server IP address: 10.0.0.1

6.1.2. Running Scapy ZMQ Server from the Python interpreter

  • Run the Python Interpreter (Scapy Server currently supports Python2)

  • Import the scapy_zmq_server.py file

  • Create a Scapy_server Object with argument as port number. default argument is port 4507

  • Invoke method activate(). (This method is blocking because the server is listening on the port).

user$ python
>>> from scapy_zmq_server import *
>>> s = Scapy_server() // starts with port 4507
>>> s = Scapy_server(5555) //starts with port 5555
>>> s.activate()
***Scapy Server Started***
Listening on port: 5555
Server IP address: 10.0.0.1

6.1.3. Shutting down Scapy ZMQ Server

There are 2 ways to shut down the server:

  • The server can be shut down using the keyboard interrupt Ctrl+C

  • The server can be shut down remotely with the method "shut_down" with no arguments

//Sending Request: {"params": [], "jsonrpc": "2.0", "method": "shut_down", "id": "1"}
//Will result in this print by the server:
Server: Shut down by remote user