How to Accumulate Streamed Tokens into a Final String in Python

Accumulate a stream of tokens into a single final string by concatenating each token in sequence.

Easy Python 3.9+ Aug 9, 2026 AI & LLM integration patterns 16 views 0 copies

streaming tokens strings accumulate llm

Python code

12 lines
Python 3.9+
Run it 
def accumulate_tokens(tokens):
    """Accumulate a stream of tokens into a single final string."""
    result = ""
    for token in tokens:
        result += token
    return result


if __name__ == "__main__":
    token_stream = ["Hello", ", ", "world", "!", " This ", "is ", "accumulated."]
    final_string = accumulate_tokens(token_stream)
    print(final_string)

Output

stdout 
Hello, world! This is accumulated.

How it works

This function iterates over an iterable of tokens and appends each one to an accumulating result string. Because strings are immutable in Python, each += creates a new string and rebinds result, which is fine for small token counts. For very large streams, consider using ''.join(tokens) for better performance.

Common mistakes

  • Assuming tokens arrive as a list, but they might be a generator – the loop handles both.
  • Forgetting that strings are immutable, so repeated `+=` can be slow for huge token counts.
  • Not handling non-string tokens, which would raise a TypeError.

Variations

  1. Use `functools.reduce(lambda a, b: a + b, tokens, '')` for a functional approach.
  2. Use `''.join(token for token in tokens)` to avoid intermediate string creation.

Real-world use cases

  • Collecting streamed text chunks from an LLM API response into a complete answer.
  • Aggregating log lines emitted by a streaming process into a single report.
  • Combining partial web socket messages into a full payload before parsing.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from AI & LLM integration patterns

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes