1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
|
.. PyZMQ serialization doc, by Min Ragan-Kelley, 2011
.. _serialization:
Serializing messages with PyZMQ
===============================
When sending messages over a network, you often need to marshall your data into bytes.
Builtin serialization
---------------------
PyZMQ is primarily bindings for libzmq, but we do provide three builtin serialization
methods for convenience, to help Python developers learn libzmq. Python has two primary
packages for serializing objects: :py:mod:`json` and :py:mod:`pickle`, so we provide
simple convenience methods for sending and receiving objects serialized with these
modules. A socket has the methods :meth:`~.Socket.send_json` and
:meth:`~.Socket.send_pyobj`, which correspond to sending an object over the wire after
serializing with json and pickle respectively, and any object sent via those
methods can be reconstructed with the :meth:`~.Socket.recv_json` and
:meth:`~.Socket.recv_pyobj` methods.
These methods designed for convenience, not for performance, so developers who do want
to emphasize performance should use their own serialized send/recv methods.
Using your own serialization
----------------------------
In general, you will want to provide your own serialization that is optimized for your
application or library availability. This may include using your own preferred
serialization ([msgpack]_, [protobuf]_), or adding compression via [zlib]_ in the standard
library, or the super fast [blosc]_ library.
There are two simple models for implementing your own serialization: write a function
that takes the socket as an argument, or subclass Socket for use in your own apps.
For instance, pickles can often be reduced substantially in size by compressing the data.
The following will send *compressed* pickles over the wire:
.. sourcecode:: python
import zlib, cPickle as pickle
def send_zipped_pickle(socket, obj, flags=0, protocol=-1):
"""pickle an object, and zip the pickle before sending it"""
p = pickle.dumps(obj, protocol)
z = zlib.compress(p)
return socket.send(z, flags=flags)
def recv_zipped_pickle(socket, flags=0, protocol=-1):
"""inverse of send_zipped_pickle"""
z = socket.recv(flags)
p = zlib.decompress(z)
return pickle.loads(p)
A common data structure in Python is the numpy array. PyZMQ supports sending
numpy arrays without copying any data, since they provide the Python buffer interface.
However just the buffer is not enough information to reconstruct the array on the
receiving side. Here is an example of a send/recv that allow non-copying
sends/recvs of numpy arrays including the dtype/shape data necessary for reconstructing
the array.
.. sourcecode:: python
import numpy
def send_array(socket, A, flags=0, copy=True, track=False):
"""send a numpy array with metadata"""
md = dict(
dtype = str(A.dtype),
shape = A.shape,
)
socket.send_json(md, flags|zmq.SNDMORE)
return socket.send(A, flags, copy=copy, track=track)
def recv_array(socket, flags=0, copy=True, track=False):
"""recv a numpy array"""
md = socket.recv_json(flags=flags)
msg = socket.recv(flags=flags, copy=copy, track=track)
buf = buffer(msg)
A = numpy.frombuffer(buf, dtype=md['dtype'])
return A.reshape(md['shape'])
.. [msgpack] Message Pack serialization library http://msgpack.org
.. [protobuf] Google Protocol Buffers http://code.google.com/p/protobuf
.. [zlib] Python stdlib module for zip compression: :py:mod:`zlib`
.. [blosc] Blosc: A blocking, shuffling and loss-less (and crazy-fast) compression library http://blosc.pytables.org/trac
|
