Erlang Central

Difference between revisions of "Distributed erlang using ssl through firewalls"

From ErlangCentral Wiki

m (Added application link)
(33 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=== Author ===
+
== Author ==
 
Michael McDaniel
 
Michael McDaniel
  
=== Title ===
+
== Updates ==
 +
Added link to application edids.org
 +
[[User:MichaelMcDaniel|MichaelMcDaniel]] 17:55, 16 January 2011 (PST)
 +
 
 +
Added updates about R11B-5 SSL distribution problems and fixes [[User:BruceFitzsimons|BruceFitzsimons]] 10:12, 14 September 2007 (BST)
 +
 
 +
== Title ==
 
Distributed Erlang using SSL through firewalls
 
Distributed Erlang using SSL through firewalls
  
=== License ===
+
== License ==
 
   Copyright (c)  2006 Michael McDaniel.
 
   Copyright (c)  2006 Michael McDaniel.
 
   Permission is granted to copy, distribute and/or modify this document
 
   Permission is granted to copy, distribute and/or modify this document
Line 20: Line 26:
 
URL links in this paper were valid as of Mon Nov 20 22:03:25 PST 2006.
 
URL links in this paper were valid as of Mon Nov 20 22:03:25 PST 2006.
  
=== References ===
+
== References ==
 
[http://www.erlang.org/doc/doc-5.5.1/lib/ssl-3.0.12/doc/html/ssl_distribution.html]1) Erlang SSL documentation
 
[http://www.erlang.org/doc/doc-5.5.1/lib/ssl-3.0.12/doc/html/ssl_distribution.html]1) Erlang SSL documentation
 
[http://www.erlang.org/ml-archive/erlang-questions/200412/msg00110.html]2) message from mailing list
 
[http://www.erlang.org/ml-archive/erlang-questions/200412/msg00110.html]2) message from mailing list
 
[http://www.erlang.org/ml-archive/erlang-questions/200605/msg00336.html]3) message from mailing list
 
[http://www.erlang.org/ml-archive/erlang-questions/200605/msg00336.html]3) message from mailing list
  
 +
== Overview ==
  
=== Overview ===
+
This howto describes how I got distributed Erlang nodes (each running
 
+
<br/>This howto describes how I got distributed Erlang nodes (each running
+
 
on different physical servers) to communicate using the Secure
 
on different physical servers) to communicate using the Secure
 
Sockets Layer (SSL) protocol over
 
Sockets Layer (SSL) protocol over
Line 40: Line 45:
 
firewall ports.
 
firewall ports.
  
<p>
+
 
I do <font color="red">not</font> describe the commands
+
I do <font color="red">not</font> describe the commands
 
needed to configure firewalls.
 
needed to configure firewalls.
 
I do, however, give general guidelines which are required for any firewall
 
I do, however, give general guidelines which are required for any firewall
 
with regards to opening and forwarding ports to have the Erlang nodes
 
with regards to opening and forwarding ports to have the Erlang nodes
 
communicate.
 
communicate.
</p>
 
  
<p>
+
 
<note>
+
 
Once you configure a node to use SSL, it can only talk with other nodes using SSL.
 
Once you configure a node to use SSL, it can only talk with other nodes using SSL.
</note>
 
</p>
 
</abstract>
 
  
  
=== Introduction ===
+
== Introduction ==
 +
 
 +
=== Motivation ===
  
Motivation
 
  
<p>
 
 
There are applications where you may want Erlang nodes to communicate
 
There are applications where you may want Erlang nodes to communicate
 
over the Internet.  Perhaps some distributed processing or data gathering.
 
over the Internet.  Perhaps some distributed processing or data gathering.
Line 67: Line 67:
 
Perhaps a distributed Mnesia database.  Also useful would be if they
 
Perhaps a distributed Mnesia database.  Also useful would be if they
 
communicate over the Internet securely using the SSL protocol.
 
communicate over the Internet securely using the SSL protocol.
</p>
+
 
</section>
+
=== Application ===
<section>
+
 
<title>Application</title>
+
<p>
+
 
It happens that my application is a security
 
It happens that my application is a security
application.  It determines if certains hosts (IPs) are attempting to
+
application [http://edids.org].  It determines if certains hosts (IPs) are attempting to
 
compromise a server (what that consists of is a subject for a different
 
compromise a server (what that consists of is a subject for a different
 
howto).  Once the application decides that an IP is an attacker, it
 
howto).  Once the application decides that an IP is an attacker, it
Line 79: Line 77:
 
any server on that network (that is, any server with Internet visibility
 
any server on that network (that is, any server with Internet visibility
 
on that subnet).
 
on that subnet).
</p>
 
  
<p>
+
 
 
Attacking IPs typically attack other servers on other networks, as
 
Attacking IPs typically attack other servers on other networks, as
 
well.  The three networks I use as an example contain four servers of
 
well.  The three networks I use as an example contain four servers of
 
interest with Internet visibility.  I often see the same IP attacking
 
interest with Internet visibility.  I often see the same IP attacking
 
other servers on other networks.
 
other servers on other networks.
</p>
 
  
<p>
+
 
 
Because the application is run on all the servers, any single server
 
Because the application is run on all the servers, any single server
 
is fully capable of determining if an IP is an attacker and taking  
 
is fully capable of determining if an IP is an attacker and taking  
Line 94: Line 90:
 
application "warn" other servers of attacking IPs and save doing
 
application "warn" other servers of attacking IPs and save doing
 
the same analysis work repeatedly on each server.
 
the same analysis work repeatedly on each server.
</p>
+
 
</section>
+
== Main Text ==
<section>
+
 
<title>Communications Overview</title>
+
=== Communications Overview ===
<p>
+
 
  If multiple servers are on a single network,
+
If multiple servers are on a single network,
 
having Erlang talk amongst them is trivial.  Simply use
 
having Erlang talk amongst them is trivial.  Simply use
<pre caption="">
+
<code caption="passing messages">
 
Registered_Name ! {From, Message}.  
 
Registered_Name ! {From, Message}.  
 
OR
 
OR
 
global:send(Registered_Name, {From, Message}).
 
global:send(Registered_Name, {From, Message}).
</pre>
+
</code>
 
and it works (well, almost that simple).
 
and it works (well, almost that simple).
</p>
 
  
<p>
+
 
 +
 
 
It becomes more complex if you want to get through firewalls over the
 
It becomes more complex if you want to get through firewalls over the
 
Internet (and want to use SSL).  The servers I wish to communicate amongst each other are distributed over the Internet,
 
Internet (and want to use SSL).  The servers I wish to communicate amongst each other are distributed over the Internet,
 
and I want them talking privately using SSL.
 
and I want them talking privately using SSL.
</p>
 
  
<p>
 
  
  You will need control of the firewalls and servers
+
You will need control of the firewalls and servers
 
in question so that you can set the proper configurations.  The techniques
 
in question so that you can set the proper configurations.  The techniques
 
discussed map to any application that has inter-node communication using
 
discussed map to any application that has inter-node communication using
 
standard Erlang semantics.
 
standard Erlang semantics.
</p>
 
  
<p>
+
 
 
Erlang can already "talk SSL".  However, presently, I know of no standard
 
Erlang can already "talk SSL".  However, presently, I know of no standard
 
way to choose what port to use.  I have tried using the  
 
way to choose what port to use.  I have tried using the  
<pre caption="proxylsport">
+
<code>
 
-ssl proxylsport NNNN
 
-ssl proxylsport NNNN
</pre>
+
</code>
 +
 
 
command line flag and was unsuccessful in having it work as I expected (I thought
 
command line flag and was unsuccessful in having it work as I expected (I thought
 
it would limit SSL communications to port NNNN).  I
 
it would limit SSL communications to port NNNN).  I
Line 135: Line 129:
 
to inet_dist_listen_{min,max} for normal TCP communications, though this
 
to inet_dist_listen_{min,max} for normal TCP communications, though this
 
new variable for SSL enables setting only a single port rather than a range
 
new variable for SSL enables setting only a single port rather than a range
of ports).  I have submitted
+
of ports).  I have submitted the patch to the  
the patch to the  
+
[http://www.ericsson.com/technology/opensource/erlang/index.shtml]Ericsson Erlang developers,
<a href="http://www.ericsson.com/technology/opensource/erlang/index.shtml">
+
and I am hopeful that it or something like (perhaps a range of ports) it will get incorporated into the [http://www.erlang.org/download.html]standard distribution (or
Ericsson Erlang</a> developers,
+
and I am hopeful that it or something like (perhaps a range of ports) it will get incorporated into the
+
<a href="http://www.erlang.org/download.html">standard distribution</a> (or
+
 
someone explains to me how to specify a single SSL port without my patch!).
 
someone explains to me how to specify a single SSL port without my patch!).
</p>
 
  
<p>
+
 
 +
 
 
If you do not have control of your firewalls, most of the techniques  
 
If you do not have control of your firewalls, most of the techniques  
 
described here will not be useful to you.  You will probably have to use
 
described here will not be useful to you.  You will probably have to use
 
something such as  
 
something such as  
<![CDATA[
+
[http://www.google.com/search?hl=en&lr=&q=tunnels+over+ssl&btnG=Search=tunnels over SSL] (though certainly, being able to control your SSL port  
<a href="http://www.google.com/search?hl=en&lr=&q=tunnels+over+ssl&btnG=Search">
+
tunnels over SSL</a> (though certainly, being able to control your SSL port  
+
 
will be useful!).
 
will be useful!).
]]>
 
</p>
 
</section>
 
  
</chapter>
 
  
 +
=== Environment ===
  
  
<chapter>
 
<title>Environment</title>
 
 
</chapter>
 
 
<p>
 
 
I have four servers (one Erlang node on each server), all running
 
I have four servers (one Erlang node on each server), all running
various release 9.x versions of <a href="http://www.suse.com">SuSE</a>
+
various release 9.x versions of [http://www.suse.com]SuSE  
<a href="http://www.gnu.org">GNU</a>/<a href="http://www.linux.org">Linux</a>
+
[http://www.gnu.org]GNU
distributions.
+
[http://www.linux.org]Linux distributions.
</p>
+
  
<p>
+
'''NOTE:''' R11B-5 has some ''bugs'' in the inet_ssl_dist code that prevent SSL distribution from working. [http://www.erlang.org/pipermail/erlang-bugs/2007-July/000386.html]
Much testing and development was done on a laptop loaded with  
+
There are patches to fix it. [http://www.erlang.org/pipermail/erlang-patches/2007-July/000184.html] [[User:BruceFitzsimons|BruceFitzsimons]] 10:12, 14 September 2007 (BST)
release 6.10 of the <a href="http://www.ubuntu.com">Ubuntu desktop</a>
+
 
[derived from <a href="http://www.debian.org">Debian</a>]  
+
=== diagram ===
 +
Much testing and development was done on a laptop loaded with  
 +
release 6.10 of the [http://www.ubuntu.com]Ubuntu desktop  
 +
[derived from [http://www.debian.org]Debian]  
 
distribution.  All servers run Erlang
 
distribution.  All servers run Erlang
<a href="http://erlang.org/download/otp_src_R11B-2.tar.gz">R11B-2</a>,
+
[http://erlang.org/download/otp_src_R11B-2.tar.gz]R11B-2,
and are
+
and are [[image:dist_erlang_thru_firewalls.png]]are visible on
<a href="dist_erlang_thru_firewalls.png">are visible on</a>
+
three different (public) networks distributed across the Internet.
three different (public) networks distributed across
+
 
the Internet.
+
</p>
+
  
<p>
 
 
Each server has it's own hardware firewall (even the two on the same
 
Each server has it's own hardware firewall (even the two on the same
public IP block); two of the servers also run software firewalls.
+
public IP block); two of the servers also run software firewalls.
</p>
+
 
 +
 
  
  
<p>
 
 
The Erlang port mapper daemon epmd runs on the IANA  
 
The Erlang port mapper daemon epmd runs on the IANA  
(<a href="http://www.iana.org">Internet Assigned Number Authority</a>)
+
([http://www.iana.org]Internet Assigned Number Authority)
well-known port <a href="http://www.iana.org/assignments/port-numbers">4369</a>
+
well-known port [http://www.iana.org/assignments/port-numbers]4369
 
and is reserved for both TCP and UDP protocols.  It was registered by
 
and is reserved for both TCP and UDP protocols.  It was registered by
<a href="http://www.ericsson.com/technology/opensource/index.shtml">Ericsson
+
[http://www.ericsson.com/technology/opensource/index.shtml]Ericsson
Corporation</a> for the <a href="http://www.erlang.org">Erlang</a>
+
Corporation for the [http://www.erlang.org]Erlang language.  Using my patch,
language.  Using my patch, I was able to force the SSL communications
+
I was able to force the SSL communications to run on the same port for each node.  
to run on the same port for each node. This allowed me to open only
+
This allowed me to open only two extra ports on the firewalls, 4369 for epmd, and
two extra ports on the firewalls, 4369 for epmd, and a port for SSL traffic.
+
a port for SSL traffic.
</p>
+
  
  
 +
=== Details ===
  
<section>
 
<title>Details</title>
 
  
<p>
 
 
<code caption="For commands, the following will be used">
 
<code caption="For commands, the following will be used">
<br/>
+
 
<br/>$ this is a user command (note $ prompt)
+
$ this is a user command (note $ prompt)
<br/># this is a root command (note # prompt)
+
# this is a root command (note # prompt)
 
</code>
 
</code>
<br/>
+
 
</p>
+
  
 
System: Ubuntu rel 6.10 (aka Edgy; Debian derived)
 
System: Ubuntu rel 6.10 (aka Edgy; Debian derived)
  
<code caption="patch to inet_ssl_dist.erl">
 
<br/>/80c80,86
 
<br/><    case ssl_prim:listen(0, [{active, false}, {packet,4}] ++
 
<br/>---
 
<br/>>
 
<br/>>    case application:get_env(kernel, inet_ssl_port) of
 
<br/>>          {ok, SSL_port} -> SSL_port ;
 
<br/>> _undefined    -> SSL_port = 0
 
<br/>>    end ,
 
<br/>>
 
<br/>>    case ssl_prim:listen(SSL_port, [{active, false}, {packet,4}] ++
 
<br/>
 
</code>
 
  
 
<code caption="versions and Erlang configure command">
 
<code caption="versions and Erlang configure command">
<br/>$
+
$
<br/>$ uname -a
+
$ uname -a
<br/>Linux delora 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686
+
Linux delora 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686
<br/>GNU/Linux
+
GNU/Linux
<br/>$
+
$
<br/>$ erl --version -s init stop      (OR erl -s init stop)
+
$ erl --version -s init stop      (OR erl -s init stop)
<br/>Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]
+
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]
<br/>
+
 
<br/>Eshell V5.5.2  (abort with ^G)
+
Eshell V5.5.2  (abort with ^G)
<br/>1>  
+
1>  
<br/>$
+
$
<br/>$ gcc --version
+
$ gcc --version
<br/>gcc (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
+
gcc (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
<br/>Copyright (C) 2006 Free Software Foundation, Inc.
+
Copyright (C) 2006 Free Software Foundation, Inc.
<br/>This is free software; see the source for copying conditions.  There is NO
+
This is free software; see the source for copying conditions.  There is NO
<br/>warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
<br/>$
+
$
<br/>$ echo version | openssl
+
$ echo version | openssl
<br/>OpenSSL> OpenSSL 0.9.8b 04 May 2006
+
OpenSSL> OpenSSL 0.9.8b 04 May 2006
<br/>OpenSSL>  
+
OpenSSL>  
<br/>$
+
$
<br/>$ isql --version
+
$ isql --version
<br/>unixODBC 2.2.11
+
unixODBC 2.2.11
<br/>$
+
$
<br/>$ ./configure  --with-odbc  --x-libraries=/usr/lib/unixODBC/ \
+
$ ./configure  --with-odbc  --x-libraries=/usr/lib/unixODBC/ \
<br/>  --x-includes=/usr/include/ --with-ssl                      \
+
  --x-includes=/usr/include/ --with-ssl                      \
<br/>  --x-includes=/usr/local/ssl/include/openssl/              \
+
  --x-includes=/usr/local/ssl/include/openssl/              \
<br/>  --x-libraries=/usr/local/ssl/lib/
+
  --x-libraries=/usr/local/ssl/lib/
<br/>
+
 
<br/>  [output not shown]
+
  [output not shown]
<br/>
+
 
<br/>$ sudo make install
+
$ sudo make install
<br/>
+
 
<br/>  [output not shown]
+
  [output not shown]
<br/>
+
 
<br/>$ cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
+
$ cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
<br/>$ for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
+
$ for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
<br/>$ cd /to/my/application/directory
+
$ cd /to/my/application/directory
<br/>$ erl -s systools make_script "start_ssl_clean" -s init stop
+
$ erl -s systools make_script "start_ssl_clean" -s init stop
<br/>Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]
+
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]
<br/>
+
 
<br/>Eshell V5.5.2  (abort with ^G)
+
Eshell V5.5.2  (abort with ^G)
<br/>1>  
+
1>  
<br/>$
+
$
<br/>
+
 
 
</code>
 
</code>
 
Here is my SSL boot script.
 
Here is my SSL boot script.
<code caption="start_ssl_clean.rel">
+
<code caption="boot script start_ssl_clean.rel">
<br/>
+
 
<br/>%
+
%
<br/>% make sure all version numbers are correct
+
% make sure all version numbers are correct
<br/>% * some will be in /usr/local/lib/erlang/bin/start.script
+
% * some will be in /usr/local/lib/erlang/bin/start.script
<br/>% * erts vers shows when starting command line erl OR
+
% * erts vers shows when starting command line erl OR
<br/>% * erlang:system_info(system_version)
+
% * erlang:system_info(system_version)
<br/>% * ssl shows with 'ls /usr/local/lib/erlang/lib/ssl<TAB>'
+
% * ssl shows with 'ls /usr/local/lib/erlang/lib/ssl<TAB>'
<br/>%
+
%
<br/>%
+
%
<br/>{release, {"OTP  APN 181 01","R11B"}, {erts, "5.5.2"},
+
{release, {"OTP  APN 181 01","R11B"}, {erts, "5.5.2"},
<br/> [{kernel,"2.11.2"},
+
[{kernel,"2.11.2"},
<br/>  {stdlib,"1.14.2"},
+
  {stdlib,"1.14.2"},
<br/>  {sasl,"2.1.4"},
+
  {sasl,"2.1.4"},
<br/>  {os_mon,"2.1.1"},
+
  {os_mon,"2.1.1"},
<br/>  {ssl,"3.0.12"}]}.
+
  {ssl,"3.0.12"}]}.
<br/>% see
+
% see
<br/>% file:///usr/local/lib/erlang/lib/ssl-3.0.12/doc/html/ssl_distribution.html
+
% file:///usr/local/lib/erlang/lib/ssl-3.0.12/doc/html/ssl_distribution.html
<br/>% for ssl info
+
% for ssl info
<br/>%  
+
%  
<br/>% cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
+
% cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
<br/>% for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
+
% for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
<br/>%
+
%
<br/>% systools:make_script("start_ssl_clean",[]).
+
% systools:make_script("start_ssl_clean",[]).
<br/>
+
 
 
</code>
 
</code>
  
  
<p>
+
 
 
I use RCS for local source control.
 
I use RCS for local source control.
<code caption="patch to R11B-2 inet_ssl_dist.erl (which had no version number)b">
+
<code caption="patch to R11B-2 inet_ssl_dist.erl (which had no version number)">
<br/>$ rcsdiff inet_ssl_dist.erl
+
$ rcsdiff inet_ssl_dist.erl
<br/>80c80,86
+
80c80,86
<br/><    case ssl_prim:listen(0, [{active, false}, {packet,4}] ++  
+
<    case ssl_prim:listen(0, [{active, false}, {packet,4}] ++  
<br/>---
+
---
<br/>>  
+
>  
<br/>>    case application:get_env(kernel, inet_ssl_port) of
+
>    case application:get_env(kernel, inet_ssl_port) of
<br/>>          {ok, SSL_port} -> SSL_port ;
+
>          {ok, SSL_port} -> SSL_port ;
<br/>>          _undefined    -> SSL_port = 0
+
>          _undefined    -> SSL_port = 0
<br/>>    end ,
+
>    end ,
<br/>>  
+
>  
<br/>>    case ssl_prim:listen(SSL_port, [{active, false}, {packet,4}] ++  
+
>    case ssl_prim:listen(SSL_port, [{active, false}, {packet,4}] ++  
<br/>
+
</code>
+
</p>
+
  
 +
</code>
  
  
  
<p>
 
 
Communication between nodes is initiated over epmd's well-known port
 
Communication between nodes is initiated over epmd's well-known port
 
4369.  In default behaviour, a different port is (randomly) chosen for the
 
4369.  In default behaviour, a different port is (randomly) chosen for the
Line 336: Line 295:
 
to limit what ports are used to communicate between nodes over SSL.  That
 
to limit what ports are used to communicate between nodes over SSL.  That
 
is the purpose of the simple hack described above, so I can choose my port.
 
is the purpose of the simple hack described above, so I can choose my port.
</p>
 
  
<p>
+
 
 +
 
 
I start the application using the SSL boot file (-boot ${HOME}/start_ssl_clean).
 
I start the application using the SSL boot file (-boot ${HOME}/start_ssl_clean).
 
I limit the ports using the sys.config file (-config ${HOME}/sys.config).
 
I limit the ports using the sys.config file (-config ${HOME}/sys.config).
Line 344: Line 303:
 
</code>
 
</code>
  
<p>
+
 
 
merl is a 'safe' account created only to run certain programs, and with limited authority.
 
merl is a 'safe' account created only to run certain programs, and with limited authority.
</p>
+
 
  
 
<code caption="starting from /etc/init.d/my_script">
 
<code caption="starting from /etc/init.d/my_script">
<br/>#!/bin/sh
+
#!/bin/sh
<br/>
+
 
<br/># -setcookie is needed else I get 'No home for cookie file'
+
# -setcookie is needed else I get 'No home for cookie file'
<br/># the cookie a123 is a throwaway as the app creates a special
+
# the cookie a123 is a throwaway as the app creates a special
<br/># cookie anyway
+
# cookie anyway
<br/>#
+
#
<br/># copy *.pem to /home/merl
+
# copy *.pem to /home/merl
<br/># copy start_ssl_clean.* to /home/merl
+
# copy start_ssl_clean.* to /home/merl
<br/>#  
+
#  
<br/>
+
 
<br/>export HOME=/home/merl
+
export HOME=/home/merl
<br/>export HOST=`/bin/hostname`
+
export HOST=`/bin/hostname`
<br/>
+
 
<br/>
+
 
<br/>(
+
(
<br/>/bin/su merl -c                                      \
+
/bin/su merl -c                                      \
<br/>"
+
"
<br/>cd /home/merl ;
+
cd /home/merl ;
<br/>/usr/local/bin/erl                                  \
+
/usr/local/bin/erl                                  \
<br/>-boot ${HOME}/start_ssl_clean                        \
+
-boot ${HOME}/start_ssl_clean                        \
<br/>-proto_dist inet_ssl                                \
+
-proto_dist inet_ssl                                \
<br/>-ssl_dist_opt client_certfile ${HOME}/ssl_client.pem \
+
-ssl_dist_opt client_certfile ${HOME}/ssl_client.pem \
<br/>-ssl_dist_opt server_certfile ${HOME}/ssl_server.pem \
+
-ssl_dist_opt server_certfile ${HOME}/ssl_server.pem \
<br/>-ssl_dist_opt verify 1 depth 1 -name ${HOST}_myapp  \
+
-ssl_dist_opt verify 1 depth 1 -name ${HOST}_myapp  \
<br/>-detached -config ${HOME}/sys.config              \
+
-detached -config ${HOME}/sys.config              \
<br/>-s my_app start                                      \
+
-s my_app start                                      \
<br/>"
+
"
<br/>)
+
)
<br/>
+
 
<br/>#% end
+
#% end
<br/>
+
 
 
</code>
 
</code>
  
  
<br/>$ cat sys.config
+
$ cat sys.config
<p>
+
 
 
<code caption="sys.config">
 
<code caption="sys.config">
<br/>%% for example
+
%% for example
<br/>[ {kernel,  [ {inet_ssl_port, 4370} ] }
+
[ {kernel,  [ {inet_ssl_port, 4370} ] }
<br/>].
+
].
 
</code>
 
</code>
</p>
+
 
  
 
I hope that is enough to get you started using secure communications between
 
I hope that is enough to get you started using secure communications between
 
your Erlang nodes.
 
your Erlang nodes.
 
+
[[Category:HowTo]]
</p>
+
</section>
+
</guide>
+
<!--
+
$Id: dist_erlang_thru_firewalls.xml,v 1.7 2007/03/18 05:44:39 mmcdanie Exp mmcdanie $
+
-->
+

Revision as of 01:55, 17 January 2011

Contents

Author

Michael McDaniel

Updates

Added link to application edids.org MichaelMcDaniel 17:55, 16 January 2011 (PST)

Added updates about R11B-5 SSL distribution problems and fixes BruceFitzsimons 10:12, 14 September 2007 (BST)

Title

Distributed Erlang using SSL through firewalls

License

 Copyright (c)  2006 Michael McDaniel.
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.2
 or any later version published by the Free Software Foundation;
 with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
 Texts.  A copy of the license can be seen by clicking the link entitled "GNU
 Free Documentation License".

[1]GNU Free Documentation License XML

[2]GNU Free Documentation License HTML

URL links in this paper were valid as of Mon Nov 20 22:03:25 PST 2006.

References

[3]1) Erlang SSL documentation [4]2) message from mailing list [5]3) message from mailing list

Overview

This howto describes how I got distributed Erlang nodes (each running on different physical servers) to communicate using the Secure Sockets Layer (SSL) protocol over the Internet through firewalls. I had control of the firewalls, and you will need control of your firewalls, to do what I describe here. Firewall configuration is dependent upon what kinds of firewalls are being used, and the network architecture. I also show a patch to inet_ssl_dist.erl which creates a new kernel environment variable to force SSL communication over a pre-determined port. This (or something similar) is a necessity if you want to limit your open firewall ports.


I do not describe the commands needed to configure firewalls. I do, however, give general guidelines which are required for any firewall with regards to opening and forwarding ports to have the Erlang nodes communicate.


Once you configure a node to use SSL, it can only talk with other nodes using SSL.


Introduction

Motivation

There are applications where you may want Erlang nodes to communicate over the Internet. Perhaps some distributed processing or data gathering. Perhaps embedded controllers doing remote control or monitoring, or TiVO (tm) like devices "phoning home" or receiving programming updates. Perhaps a distributed Mnesia database. Also useful would be if they communicate over the Internet securely using the SSL protocol.

Application

It happens that my application is a security application [6]. It determines if certains hosts (IPs) are attempting to compromise a server (what that consists of is a subject for a different howto). Once the application decides that an IP is an attacker, it takes some action(s). Attacks from that IP generally will be made on any server on that network (that is, any server with Internet visibility on that subnet).


Attacking IPs typically attack other servers on other networks, as well. The three networks I use as an example contain four servers of interest with Internet visibility. I often see the same IP attacking other servers on other networks.


Because the application is run on all the servers, any single server is fully capable of determining if an IP is an attacker and taking appropriate action. However, I decided I would have my application "warn" other servers of attacking IPs and save doing the same analysis work repeatedly on each server.

Main Text

Communications Overview

If multiple servers are on a single network, having Erlang talk amongst them is trivial. Simply use

Registered_Name ! {From, Message}. 
OR
global:send(Registered_Name, {From, Message}).

and it works (well, almost that simple).


It becomes more complex if you want to get through firewalls over the Internet (and want to use SSL). The servers I wish to communicate amongst each other are distributed over the Internet, and I want them talking privately using SSL.


You will need control of the firewalls and servers in question so that you can set the proper configurations. The techniques discussed map to any application that has inter-node communication using standard Erlang semantics.


Erlang can already "talk SSL". However, presently, I know of no standard way to choose what port to use. I have tried using the

-ssl proxylsport NNNN

command line flag and was unsuccessful in having it work as I expected (I thought it would limit SSL communications to port NNNN). I describe a patch to Erlang which creates a new kernel variable (inet_ssl_port) enabling selection of a specific port over which to "talk SSL" (similar to inet_dist_listen_{min,max} for normal TCP communications, though this new variable for SSL enables setting only a single port rather than a range of ports). I have submitted the patch to the [7]Ericsson Erlang developers, and I am hopeful that it or something like (perhaps a range of ports) it will get incorporated into the [8]standard distribution (or someone explains to me how to specify a single SSL port without my patch!).


If you do not have control of your firewalls, most of the techniques described here will not be useful to you. You will probably have to use something such as over SSL (though certainly, being able to control your SSL port will be useful!).


Environment

I have four servers (one Erlang node on each server), all running various release 9.x versions of [9]SuSE [10]GNU [11]Linux distributions.

NOTE: R11B-5 has some bugs in the inet_ssl_dist code that prevent SSL distribution from working. [12] There are patches to fix it. [13] BruceFitzsimons 10:12, 14 September 2007 (BST)

diagram

Much testing and development was done on a laptop loaded with release 6.10 of the [14]Ubuntu desktop [derived from [15]Debian] distribution. All servers run Erlang [16]R11B-2, and are Dist erlang thru firewalls.pngare visible on three different (public) networks distributed across the Internet.


Each server has it's own hardware firewall (even the two on the same public IP block); two of the servers also run software firewalls.



The Erlang port mapper daemon epmd runs on the IANA ([17]Internet Assigned Number Authority) well-known port [18]4369 and is reserved for both TCP and UDP protocols. It was registered by [19]Ericsson Corporation for the [20]Erlang language. Using my patch, I was able to force the SSL communications to run on the same port for each node. This allowed me to open only two extra ports on the firewalls, 4369 for epmd, and a port for SSL traffic.


Details


$ this is a user command (note $ prompt)
# this is a root command (note # prompt)


System: Ubuntu rel 6.10 (aka Edgy; Debian derived)


$
$ uname -a
Linux delora 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686
GNU/Linux
$
$ erl --version -s init stop      (OR erl -s init stop)
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> 
$
$ gcc --version
gcc (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$
$ echo version | openssl
OpenSSL> OpenSSL 0.9.8b 04 May 2006
OpenSSL> 
$
$ isql --version
unixODBC 2.2.11
$
$ ./configure  --with-odbc  --x-libraries=/usr/lib/unixODBC/ \
  --x-includes=/usr/include/ --with-ssl                      \
  --x-includes=/usr/local/ssl/include/openssl/               \
  --x-libraries=/usr/local/ssl/lib/

  [output not shown]

$ sudo make install

  [output not shown]

$ cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
$ for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
$ cd /to/my/application/directory
$ erl -s systools make_script "start_ssl_clean" -s init stop
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> 
$

Here is my SSL boot script.


%
% make sure all version numbers are correct
% * some will be in /usr/local/lib/erlang/bin/start.script
% * erts vers shows when starting command line erl OR
% * erlang:system_info(system_version)
% * ssl shows with 'ls /usr/local/lib/erlang/lib/ssl<TAB>'
%
%
{release, {"OTP  APN 181 01","R11B"}, {erts, "5.5.2"},
 [{kernel,"2.11.2"},
  {stdlib,"1.14.2"},
  {sasl,"2.1.4"},
  {os_mon,"2.1.1"},
  {ssl,"3.0.12"}]}.
% see
% file:///usr/local/lib/erlang/lib/ssl-3.0.12/doc/html/ssl_distribution.html
% for ssl info
% 
% cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
% for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
%
% systools:make_script("start_ssl_clean",[]).


I use RCS for local source control.

$ rcsdiff inet_ssl_dist.erl
80c80,86
<     case ssl_prim:listen(0, [{active, false}, {packet,4}] ++ 
---
> 
>     case application:get_env(kernel, inet_ssl_port) of
>          {ok, SSL_port} -> SSL_port ;
>          _undefined     -> SSL_port = 0
>     end ,
> 
>     case ssl_prim:listen(SSL_port, [{active, false}, {packet,4}] ++ 


Communication between nodes is initiated over epmd's well-known port 4369. In default behaviour, a different port is (randomly) chosen for the nodes to exchange messages. Since I do not want to open all ports on my firewalls (how would I know which one might be active?), I need some way to limit what ports are used to communicate between nodes over SSL. That is the purpose of the simple hack described above, so I can choose my port.


I start the application using the SSL boot file (-boot ${HOME}/start_ssl_clean). I limit the ports using the sys.config file (-config ${HOME}/sys.config).



merl is a 'safe' account created only to run certain programs, and with limited authority.


#!/bin/sh

# -setcookie is needed else I get 'No home for cookie file'
# the cookie a123 is a throwaway as the app creates a special
# cookie anyway
#
# copy *.pem to /home/merl
# copy start_ssl_clean.* to /home/merl
# 

export HOME=/home/merl
export HOST=`/bin/hostname`


(
/bin/su merl -c                                       \
"
cd /home/merl ;
/usr/local/bin/erl                                   \
-boot ${HOME}/start_ssl_clean                        \
-proto_dist inet_ssl                                 \
-ssl_dist_opt client_certfile ${HOME}/ssl_client.pem \
-ssl_dist_opt server_certfile ${HOME}/ssl_server.pem \
-ssl_dist_opt verify 1 depth 1 -name ${HOST}_myapp   \
-detached -config ${HOME}/sys.config               \
-s my_app start                                      \
"
)

#% end


$ cat sys.config

%% for example
[ {kernel,  [ {inet_ssl_port, 4370} ] }
].


I hope that is enough to get you started using secure communications between your Erlang nodes.