Erlang Factory SF Bay 2015 – Mark Anderson – Push Jobs: A Scalable Remote Execution System for Chef

By Erlang Central | Published: March 30, 2015

When Chef needed a scalable system for remote job execution on thousands of systems simultaneously, we chose Erlang and ZeroMQ for the job. This talk will cover:

* How we applied the ‘let it crash’ philosophy across a system of thousands of nodes.

* The various impedance mismatches between the ZeroMQg and Erlang approaches to distributed systems.

* The various bottlenecks we hit as we scaled our system, and how we overcame them.

Talk objectives:

- To describe our experiences scaling an zeromq application written in erlang, along with our missteps and unexpected successes.

Target audience:

- Anyone who is interested in building systems with ZeroMQ and Erlang