API in a nutshell¶
execnet ad-hoc instantiates local and remote Python interpreters. Each interpreter is accessible through a Gateway which manages code and data communication. Channels allow to exchange data between the local and the remote end. Groups help to manage creation and termination of sub interpreters.

Gateways: bootstrapping Python interpreters¶
All Gateways are instantiated via a call to makegateway()
passing it a gateway specification or URL.
Here is an example which instantiates a simple Python subprocess:
>>> gateway = execnet.makegateway()
gateways allow to remote execute code and exchange data bidirectionally.
examples for valid gateway specifications¶
ssh=wyvern//python=python3.3//chdir=mycache
specifies a Python3.3 interpreter on the hostwyvern
. The remote process will havemycache
as its current working directory.ssh=-p 5000 myhost
makes execnet pass “-p 5000 myhost” arguments to the underlying ssh client binary, effectively specifying a custom port.vagrant_ssh=default
makes execnet connect to a Vagrant VM nameddefault
via SSH through Vagrant’svagrant ssh
command. It supports the same additional parameters as regular SSH connections.popen//python=python2.6//nice=20
specification of a python subprocess using thepython2.6
executable which must be discoverable through the systemPATH
; running with the lowest CPU priority (“nice” level). By default current dir will be the current dir of the instantiator.popen//dont_write_bytecode
uses the same executable as the current Python, and also passes the-B
flag on startup, which tells Python not write.pyc
or.pyo
files. Note that this only works under CPython 2.6 and newer.popen//env:NAME=value
specifies a subprocess that uses the same interpreter as the one it is initiated from and additionally remotely sets an environment variableNAME
tovalue
.popen//execmodel=eventlet
specifies a subprocess that uses the same interpreter as the one it is initiated from but will run the other side using eventlet for handling IO and dispatching threads.socket=192.168.1.4:8888
specifies a Python Socket server process that listens on192.168.1.4:8888
New in version 1.5.
vagarant_ssh
opens a python interpreter via the vagarant ssh command
remote_exec: execute source code remotely¶
All gateways offer a simple method to execute source code in the instantiated subprocess-interpreter:
It is allowed to pass a module object as source code
in which case it’s source code will be obtained and
get sent for remote execution. remote_exec
returns
a channel object whose symmetric counterpart channel
is available to the remotely executing source.
-
Gateway.
reconfigure
([py2str_as_py3str=True, py3str_as_py2str=False])¶ reconfigures the string-coercion behaviour of the gateway
Channels: exchanging data with remote code¶
A channel object allows to send and receive data between two asynchronously running programs.
Grouped Gateways and robust termination¶
All created gateway instances are part of a group. If you
call execnet.makegateway
it actually is forwarded to
the execnet.default_group
. Group objects are container
objects (see group examples)
and manage the final termination procedure:
This method is implicitely called for each gateway group at
process-exit, using a small timeout. This is fine
for interactive sessions or random scripts which
you rather like to error out than hang. If you start many
processes then you often want to call group.terminate()
yourself and specify a larger or not timeout.
threading models: gevent, eventlet, thread¶
New in version 1.2: (status: experimental!)
execnet supports “thread”, “eventlet” and “gevent” as thread models on each of the two sides. You need to decide which model to use before you create any gateways:
# content of threadmodel.py
import execnet
# locally use "eventlet", remotely use "thread" model
execnet.set_execmodel("eventlet", "thread")
gw = execnet.makegateway()
print (gw)
print (gw.remote_status())
print (gw.remote_exec("channel.send(1)").receive())
You need to have eventlet installed in your environment and then you can execute this little test file:
$ python threadmodel.py
<Gateway id='gw0' receive-live, eventlet model, 0 active channels>
<RInfo 'numchannels=0, numexecuting=0, execmodel=thread'>
1
Note
With python3 you can (as of December 2013) only use the thread model because neither eventlet-0.14.0 nor gevent-1.0 support Python3. When they start to support Python3, execnet will probably just work because it is itself Python3 compatible.
How to execute in the main thread¶
When the remote side of a gateway uses the ‘thread’ model, execution will preferably run in the main thread. This allows GUI loops or other code to behave correctly. If you, however, start multiple executions concurrently, they will run in non-main threads.
remote_status: get low-level execution info¶
All gateways offer a simple method to obtain some status information from the remote side.
Calling this method tells you e.g. how many execution tasks are queued, how many are executing and how many channels are active.
rsync: synchronise filesystem with remote¶
execnet
implements a simple efficient rsyncing protocol.
Here is a basic example for using RSync:
rsync = execnet.RSync('/tmp/source')
gw = execnet.makegateway()
rsync.add_target(gw, '/tmp/dest')
rsync.send()
And here is API info about the RSync class.
Debugging execnet¶
By setting the environment variable EXECNET_DEBUG
you can
configure a tracing mechanism:
EXECNET_DEBUG=1: | |
---|---|
write per-process trace-files to execnet-debug-PID |
|
EXECNET_DEUBG=2: | |
perform tracing to stderr (popen-gateway slaves will send this to their instantiator) |
cross-interpreter serialization of python objects¶
New in version 1.1.
Execnet exposes a function pair which you can safely use to store and load values from different Python interpreters (e.g. Python2 and Python3, PyPy and Jython). Here is a basic example:
>>> import execnet
>>> dump = execnet.dumps([1,2,3])
>>> execnet.loads(dump)
[1,2,3]
For more examples see Dumping and loading values across interpreter versions.