Capture print output from multiple threads in python

I have a function which inside lauches multiple threads running in parallel. Threads print something and I want to capture this output from external function. I tried the next code to capture the output:

import sys, io
stdout = sys.stdout
sys.stdout = io.StringIO()
threads_conn(connect, devices) #- here many threads starts with many print inside
output = sys.stdout.getvalue()
sys.stdout = stdout

print(output)


This code works. But the problem is that output is printed only when all theads complete. So I have a freeze. Once all threads are done - whole output is printed. What I would like to have - have output once it is generated inside - in other words in real-time. Now I have the whole outout buffer printed at once.
How get all of the threads to output in real-time?

1 Answer
1

What you want to do is write your own TextIOBase class (or, if you want binary data, write your own RawIOBase and then wrap a stock TextIOWrapper around it), where you can put whatever behavior you want.

TextIOBase

RawIOBase

TextIOWrapper

As you can see from the docs. all you need to implement for TextIOBase is detach, read, readline, and write. And the first three aren't relevant for what you're doing.

TextIOBase

detach

read

readline

write

So, what should your write look like? Well, it depends on what you want to do.

write

it sounds like your goal is to tee everything to both real stdout and to a StringIO. If so, this is pretty trivial.

StringIO

The only question is what you want to do if one of the targets raises an exception, or writes fewer bytes than the other, etc. Since an IOString is never going to do either of those, we can write something really dumb that just assumes that whatever real stdout did was the right thing to do.

IOString

class TeeTextIO(io.TextIOBase):
def __init__(self, target):
self.target = target
self.stringio = io.StringIO()
def write(self, s):
writecount = self.target.write(s)
self.stringio.write(s[:writecount])
return writecount


And now:

stdout = sys.stdout
sys.stdout = TeeTextIO(sys.stdout)
threads_conn(connect, devices) #- here many threads starts with many print inside
output = sys.stdout.stringio.getvalue()
sys.stdout = stdout


Now, the output has gone to the real stdout as it came in, but it's also been stored in the StringIO for whatever you want to do with it later.

stdout

StringIO

(Notice that this class will work with any TextIOBase, like a file you open, not just with stdout. It didn't cost us anything to make it general, so why not?)

TextIOBase

open

stdout

What if you wanted to do something totally different, like spread each write randomly among 10 different files? It should be obvious:

write

class SpreadTextWriter(io.TextIOBase):
def __init__(self, *files):
self.files = files
def write(self, s):
return random.choice(self.files).write(s)


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.