toker – Pluggable parsers for the Erlang compiler


By brucify | Published: December 1, 2014



https://github.com/uwiger/toker

Pluggable parsers for the Erlang compiler

This project started as an experiment with runtime code injection using the parse_trans library.

(Note, ‘Toker’ is the Swedish name for Dopey the dwarf, but also obviously a word play refering to token transformation.)

This library installs itself into the Erlang compiler, allowing modules to switch parser modules as well as install token transformers.

Example: the toker_test module uses a custom syntax:

-module(toker_test).

-export([double/1, i2l/1]).

-toker_parser(toker_erl_parse).

double(L) ->
lists:map(`(X) -> X*2`, L).

i2l(L) ->
lists:map(`integer_to_list/1, L).

This module will not compile with the standard Erlang parser, but toker’s own build chain bootstraps itself and installs a hook in the erl_parse module, which then detects the instruction -toker_parser(toker_erl_parse).

Demonstration:

Eshell V5.10.3  (abort with ^G)

1> toker_test:double([1,2,3]).
[2,4,6]

2> toker_test:i2l([1,2,3]).
[“1″,”2”,”3”]

Contributions are welcome, although the library is mostly for fun (it’s been gathering dust for a while, and I thought I’d at least publish it).

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com



Follow Erlang Central:


Have an Erlang Question?

Reach out to the Erlang community