Process-Striped Buffering with gen_stream
A new OTP behaviour named ‘gen_stream’ is proposed for R15. It features a simple interface for consuming serial streams which reside on slow media, are generated computationally or may be infinite in length. Internally, the stream may be prefetched and/or generated concurrently, allowing the application implementor the ability to declaratively define the concurrency characteristics, improving the performance characteristics of the data source.
The inspiration for gen_stream came from Tim Bray’s Widefinder challenge and the ensuing discussions and submissions of the Erlang mailing list members in attempting to efficiently handle text file I/O and analysis.
Target audience: Anyone who has complained of slow serial I/O performance, was involved in or observed the WideFinder competition, or needs to efficiently generate or simulate infinite data streams or transform serial data on the fly.
Talk Objectives: Introduce the API and implementation of the gen_stream behaviour and its approach of prefetching data buffers by striping across processes to leverage overlapped I/O.
Jay Nelson Concurrently Serial Erlangist
DuoMark International, Inc
Jay Nelson has been working with Erlang under the guise of DuoMark International, Inc. since 2002 on various projects involving web-based casual gaming, backend social servers for iPhone apps and data aggregation using CouchDB in the cloud.