Python: Separate Processes Logging To Same File?

Does Python's logging library provide serialised logging for two (or more) separate python processes logging to the same file? It doesn't seem clear from the docs (which I have rea

Solution 1:

No it is not supported. From python logging cookbook:

Although logging is thread-safe, and logging to a single file from multiple threads in a single process is supported, logging to a single file from multiple processes is not supported, because there is no standard way to serialize access to a single file across multiple processes in Python.

Afterwards the cookbook suggests to use a single socket-server process that handles the logs and the other processes sending log messages to it. There is a working example of this apporach in the section Sending and Receiving logging events across a network.

Solution 2:

One grotty solution to this problem is to create a logging process which listens on a socket, on a single thread, which just outputs whatever it receives

The point is to hijack the socket queue as an arbitration mechanism.

#! /usr/bin/env pythonimport sys
import socket
import argparse

p = argparse.ArgumentParser()
p.add_argument("-p", "--port", help="which port to listen on", type=int)
p.add_argument("-b", "--backlog", help="accept backlog size", type=int)
p.add_argument("-s", "--buffersize", help="recv buffer size", type=int)
args = p.parse_args()

port = args.port if args.port else1339
backlog = args.backlog if args.backlog else5
size = args.buffersize if args.buffersize else1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', port))
s.listen(backlog)
print"Listening on port ", port, 'backlog size', backlog, 'buffer size', size, '\n'while1:
    try:
        (client, address) = s.accept()
        data = client.recv(size)
        print data
    except:
        client.close()

And to test it:

#! /usr/bin/env pythonimport sys
import socket
import argparse

p = argparse.ArgumentParser()
p.add_argument("-p", "--port", help="send port", action='store', default=1339, type=int)
p.add_argument("text", help="text to send")
args = p.parse_args()

ifnot args.quit andnot args.text:
    p.print_help()
else:
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect(('', args.port))
        s.send(args.text)
    except:
        s.close()

Then use it like this:

stdbuf -o L ./logger.py -b 10 -s 4096 >>logger.log 2>&1 &

and monitor recent activity with:

tail -f logger.log

Each logging entry from any given process will be emitted atomically. Adding this into the standard logging system shouldn't be too hard. Using sockets means that multiple machines can also target a single log, hosted on a dedicated machine.

Solution 3:

The simplest way is to using custom handler for logging which will pass all logs with queue from child process to main, and there log it. in such way for example working logging on client application where u have main UI thread and worker threads.

Also on POSIX system, u can use logging in append mode. Up to 4kb it would be atomic.