Erlang Central

How to use ei to marshal binary terms in port programs

Revision as of 20:19, 21 March 2008 by Asergey (Talk | contribs)

Contents

Author

Serge Aleynikov <saleyn at gmail.com>

Overview

The purpose of this tutorial is to illustrate how to use ei interface library (C API) for dealing with the Erlang binary term format used in marshaling data between an Erlang node and external port programs.

The reader is also encouraged to read another how to guide by Pete Kazmer that focuses on using OTP framework for building Erlang systems that interface with external programs.

ei offers a simple and convenient interface that makes it possible to communicate with Erlang from virtually any language that can call native C functions.

At the time of this writing there are two foreign function interface (FFI) models available in Erlang:

  * In-process port driver.
  * Out-of-process port program.

The first one is obviously much faster, but is more complicated and has more dangerous impact on stability of the virtual machine - a sloppy implementation can easily crash the emulator. The second approach is slower but offers a guaranteed isolation of port program's crashes from bringing down the emulator. The decision of which interface to use is case-by-case specific, however, unless you are implementing high-performance low-latency components, you will find second approach being adequate in many cases.

In this tutorial we'll illustrate the simplicity of the second approach to build a port program in C.

Introduction

The earlier releases of Erlang/OTP came with erl_interface library that later evolved into ei (Erlang Interface). The difference between ei and erl_interface is that, in contrast to erl_interface, ei uses external binary term format directly when sending and receiving terms in communications with an Erlang node, and it can use a given memory buffer without involving dynamic memory. While ei seems to be the preferred method of encoding/decoding terms, most of the help documents provided with Erlang describing how to build port programs still use erl_interface function calls or raw byte-coding methods as examples.

This tutorial demonstrates the use of ei's encoding and decoding functions for marshaling data between an Erlang process and a C program.

In order to focus this tutorial on ei rather than on details of OTP, we'll use only basic Erlang functionality in building the interface process communicating with the C program. The example described in this tutorial will show how to implement functions add(X, Y), multiply(X, Y) and divide(X, Y) that can be called from Erlang and executed in C.

Marshaling Protocol

First, we need to define the data marshaling contract between Erlang and C that will request from C to execute functions and receive the result back.

The objective of the port program is to be able to implement the following API:

Erlang -> C            C -> Erlang
===========            ===========

{add,      X, Y}  ->   {ok, Result::integer()} | {error, Reason::atom()}
{multiply, X, Y}  ->   {ok, Result::integer()} | {error, Reason::atom()}
{divide,   A, B}  ->   {ok, Result::float()}   | {error, Reason::atom()}
  X = integer()
  Y = integer()
  A = integer() | float()
  B = integer() | float()

Where the add, multiply, and divide instructions perform the corresponding action on the C-side given two arguments.

Erlang side

Let's start with writing an Erlang module responsible for marshaling data to and from the C port program. The module will use erlang:port_command/2 function for asynchronously sending data to the port. Note that the data is encoded to the external binary term format using erlang:term_to_binary/1.

Erlang module listing

-module(port).

% API
-export([start/0, start/1, stop/0, add/2, multiply/2, divide/2]).
% Internal exports
-export([init/1]).

start() ->
    start("./cport").
start(ExtPrg) ->
    spawn_link(?MODULE, init, [ExtPrg]).
stop() ->
    ?MODULE ! stop.

add(X, Y) ->
    call_port({add, X, Y}).
multiply(X, Y) ->
    call_port({multiply, X, Y}).
divide(X, Y) ->
    call_port({divide, X, Y}).

call_port(Msg) ->
    ?MODULE ! {call, self(), Msg},
    receive
    Result ->
        Result
    end.

init(ExtPrg) ->
    register(?MODULE, self()),
    process_flag(trap_exit, true),
    Port = open_port({spawn, ExtPrg}, [{packet, 2}, binary, exit_status]),
    loop(Port).

loop(Port) ->
    receive
    {call, Caller, Msg} ->
        io:format("Calling port with ~p~n", [Msg]),
        erlang:port_command(Port, term_to_binary(Msg)),
        receive
        {Port, {data, Data}} ->
            Caller ! binary_to_term(Data);
        {Port, {exit_status, Status}} when Status > 128 ->
            io:format("Port terminated with signal: ~p~n", [Status-128]),
            exit({port_terminated, Status});
        {Port, {exit_status, Status}} ->
            io:format("Port terminated with status: ~p~n", [Status]),
            exit({port_terminated, Status});
        {'EXIT', Port, Reason} ->
            exit(Reason)
        end,
        loop(Port);
    stop ->
        erlang:port_close(Port),
        exit(normal)
    end.

Data Types

Though in this example we didn't use ei_get_type() function, this function is quite useful when there's a need to determine the type of the next encoded term in the buffer. A question that gets frequently asked is which include file erl_interface.h or ei.h lists the macros that can be used to represent types returned by the ei_get_type() call? These macros are defined in the ei.h header:

#define ERL_SMALL_INTEGER_EXT 'a'
#define ERL_INTEGER_EXT       'b'
#define ERL_FLOAT_EXT         'c'
#define ERL_ATOM_EXT          'd'
#define ERL_REFERENCE_EXT     'e'
#define ERL_NEW_REFERENCE_EXT 'r'
#define ERL_PORT_EXT          'f'
#define ERL_PID_EXT           'g'
#define ERL_SMALL_TUPLE_EXT   'h'
#define ERL_LARGE_TUPLE_EXT   'i'
#define ERL_NIL_EXT           'j'
#define ERL_STRING_EXT        'k'
#define ERL_LIST_EXT          'l'
#define ERL_BINARY_EXT        'm'
#define ERL_SMALL_BIG_EXT     'n'
#define ERL_LARGE_BIG_EXT     'o'
#define ERL_NEW_FUN_EXT       'p'
#define ERL_FUN_EXT           'u'

Makefile

The following makefile can be used to build the sample port program and the corresponding Erlang module:

Makefile listing

$ cat Makefile
ERL_LIB=/usr/local/lib/erlang/lib/erl_interface-3.5.5
CFLAGS=-Wall -I/usr/local/include -I$(ERL_LIB)/include
LDFLAGS=-L. -L$(ERL_LIB)/lib 

all: cport port.beam

cport: cport.o
        gcc $(LDFLAGS) $< -lerl_interface -lei -lpthread -o $@

%.o: %.c
        gcc -g $(CFLAGS) -o $@ -c $<

%.beam: %.erl
        erlc $<

clean:
        rm port.beam cport cport.o

Running example

Running make with the Makefile above will create two files:

cport
port.beam

Now, let's start the emulator, and verify that everything works:

Running the port program

$ erl        
Erlang (BEAM) emulator version 5.5 [source] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5  (abort with ^G)
1> port:start().
<0.35.0>
2> port:add(10,5).
Calling port with {add,10,5}
{ok,15}
3> port:multiply(3,6).
Calling port with {multiply,3,6}
{ok,18}
4> port:divide(10,5).
Calling port with {divide,10,5}
{ok,2.00000}
5> port:divide(10,0).
Calling port with {divide,10,0}
{error,division_by_zero}
6> port:stop().
stop
7> 

Conclusion

As was illustrated in this tutorial ei provides a set of functions for convenient encoding and decoding of the erlang binary term format that can be used in other languages for interfacing with Erlang.