toker – Pluggable parsers for the Erlang compiler

By brucify | Published: December 1, 2014

Pluggable parsers for the Erlang compiler

This project started as an experiment with runtime code injection using the parse_trans library.

(Note, ‘Toker’ is the Swedish name for Dopey the dwarf, but also obviously a word play refering to token transformation.)

This library installs itself into the Erlang compiler, allowing modules to switch parser modules as well as install token transformers.

Example: the toker_test module uses a custom syntax:


-export([double/1, i2l/1]).


double(L) ->
lists:map(`(X) -> X*2`, L).

i2l(L) ->
lists:map(`integer_to_list/1, L).

This module will not compile with the standard Erlang parser, but toker’s own build chain bootstraps itself and installs a hook in the erl_parse module, which then detects the instruction -toker_parser(toker_erl_parse).


Eshell V5.10.3  (abort with ^G)

1> toker_test:double([1,2,3]).

2> toker_test:i2l([1,2,3]).

Contributions are welcome, although the library is mostly for fun (it’s been gathering dust for a while, and I thought I’d at least publish it).

Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.

Follow Erlang Central:

Have an Erlang Question?

Reach out to the Erlang community