Process-Striped Buffering with gen_stream

By Erlang Central | Published: March 21, 2011

A new OTP behaviour named ‘gen_stream’ is proposed for R15.  It features a simple interface for consuming serial streams which reside on slow media, are generated computationally or may be infinite in length.  Internally, the stream may be prefetched and/or generated concurrently, allowing the application implementor the ability to declaratively define the concurrency characteristics, improving the performance characteristics of the data source.

The inspiration for gen_stream came from Tim Bray’s Widefinder challenge and the ensuing discussions and submissions of the Erlang mailing list members in attempting to efficiently handle text file I/O and analysis.

Target audience: Anyone who has complained of slow serial I/O performance, was involved in or observed the WideFinder competition, or needs to efficiently generate or simulate infinite data streams or transform serial data on the fly.

Talk Objectives: Introduce the API and implementation of the gen_stream behaviour and its approach of prefetching data buffers by striping across processes to leverage overlapped I/O.

Download Presentation Download


  • Jay Nelson

    Concurrently Serial Erlangist
    DuoMark International, Inc

    Jay Nelson has been working with Erlang under the guise of DuoMark International, Inc. since 2002 on various projects involving web-based casual gaming, backend social servers for iPhone apps and data aggregation using CouchDB in the cloud.

    Jay's Website

    Twitter: @duomark

    Jay Nelson

Follow Erlang Central:

Have an Erlang Question?

Reach out to the Erlang community