Streaming Utils

process_read_stream(file_stream, processor, chunk_size, yield_processed=True) async staticmethod

Read a file/stream in chunks and optionally process each chunk before yielding it. Supports both synchronous and asynchronous .read() methods.

This helper is useful when streaming large files or socket data and applying on-the-fly transformations, such as:

• hashing

• compression / decompression

• encryption / decryption

• line parsing

• transcoding

Parameters

• file_stream (AsyncBufferedReader | BufferedReader) Object with a .read(chunk_size) method. If the method is async, it will be awaited automatically.

• processor (Callable[[bytes], T] | Callable[[bytes], Any]) Function applied to each chunk.

  • If yield_processed=True, the output of this function is yielded.

  • If yield_processed=False, the raw chunk is yielded instead.

• chunk_size (int) Maximum number of bytes to read per iteration.

• yield_processed (bool, default True) Controls what gets yielded:

  • True yield processor(chunk)

  • False yield the raw bytes chunk

Yields

• Union[T, bytes] - This generator yields values: either processed items of type T produced by processor() when yield_processed=True, or raw bytes chunks when yield_processed=False.