Erlang Central

Difference between revisions of "Distributed erlang using ssl through firewalls"

From ErlangCentral Wiki

m (Added application link)
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=== Author ===
+
== Author ==
 
Michael McDaniel
 
Michael McDaniel
  
=== Title ===
+
== Updates ==
 +
Added link to application edids.org
 +
[[User:MichaelMcDaniel|MichaelMcDaniel]] 17:55, 16 January 2011 (PST)
 +
 
 +
Added updates about R11B-5 SSL distribution problems and fixes [[User:BruceFitzsimons|BruceFitzsimons]] 10:12, 14 September 2007 (BST)
 +
 
 +
== Title ==
 
Distributed Erlang using SSL through firewalls
 
Distributed Erlang using SSL through firewalls
  
=== License ===
+
== License ==
 
   Copyright (c)  2006 Michael McDaniel.
 
   Copyright (c)  2006 Michael McDaniel.
 
   Permission is granted to copy, distribute and/or modify this document
 
   Permission is granted to copy, distribute and/or modify this document
Line 20: Line 26:
 
URL links in this paper were valid as of Mon Nov 20 22:03:25 PST 2006.
 
URL links in this paper were valid as of Mon Nov 20 22:03:25 PST 2006.
  
=== References ===
+
== References ==
 
[http://www.erlang.org/doc/doc-5.5.1/lib/ssl-3.0.12/doc/html/ssl_distribution.html]1) Erlang SSL documentation
 
[http://www.erlang.org/doc/doc-5.5.1/lib/ssl-3.0.12/doc/html/ssl_distribution.html]1) Erlang SSL documentation
 
[http://www.erlang.org/ml-archive/erlang-questions/200412/msg00110.html]2) message from mailing list
 
[http://www.erlang.org/ml-archive/erlang-questions/200412/msg00110.html]2) message from mailing list
 
[http://www.erlang.org/ml-archive/erlang-questions/200605/msg00336.html]3) message from mailing list
 
[http://www.erlang.org/ml-archive/erlang-questions/200605/msg00336.html]3) message from mailing list
  
 
+
== Overview ==
=== Overview ===
+
  
 
This howto describes how I got distributed Erlang nodes (each running
 
This howto describes how I got distributed Erlang nodes (each running
Line 41: Line 46:
  
  
I do <font color="red">not</font> describe the commands
+
I do <font color="red">not</font> describe the commands
 
needed to configure firewalls.
 
needed to configure firewalls.
 
I do, however, give general guidelines which are required for any firewall
 
I do, however, give general guidelines which are required for any firewall
Line 51: Line 56:
  
  
=== Introduction ===
+
== Introduction ==
  
Motivation
+
=== Motivation ===
  
  
Line 63: Line 68:
 
communicate over the Internet securely using the SSL protocol.
 
communicate over the Internet securely using the SSL protocol.
  
  Application
+
=== Application ===
  
 
It happens that my application is a security
 
It happens that my application is a security
application.  It determines if certains hosts (IPs) are attempting to
+
application [http://edids.org].  It determines if certains hosts (IPs) are attempting to
 
compromise a server (what that consists of is a subject for a different
 
compromise a server (what that consists of is a subject for a different
 
howto).  Once the application decides that an IP is an attacker, it
 
howto).  Once the application decides that an IP is an attacker, it
Line 86: Line 91:
 
the same analysis work repeatedly on each server.
 
the same analysis work repeatedly on each server.
  
=== Main Text ===
+
== Main Text ==
  
Communications Overview
+
=== Communications Overview ===
  
  If multiple servers are on a single network,
+
If multiple servers are on a single network,
 
having Erlang talk amongst them is trivial.  Simply use
 
having Erlang talk amongst them is trivial.  Simply use
 
<code caption="passing messages">
 
<code caption="passing messages">
Line 104: Line 109:
 
Internet (and want to use SSL).  The servers I wish to communicate amongst each other are distributed over the Internet,
 
Internet (and want to use SSL).  The servers I wish to communicate amongst each other are distributed over the Internet,
 
and I want them talking privately using SSL.
 
and I want them talking privately using SSL.
 
  
  
Line 111: Line 115:
 
discussed map to any application that has inter-node communication using
 
discussed map to any application that has inter-node communication using
 
standard Erlang semantics.
 
standard Erlang semantics.
 
  
  
Line 128: Line 131:
 
of ports).  I have submitted the patch to the  
 
of ports).  I have submitted the patch to the  
 
[http://www.ericsson.com/technology/opensource/erlang/index.shtml]Ericsson Erlang developers,
 
[http://www.ericsson.com/technology/opensource/erlang/index.shtml]Ericsson Erlang developers,
and I am hopeful that it or something like (perhaps a range of ports) it will get incorporated into the
+
and I am hopeful that it or something like (perhaps a range of ports) it will get incorporated into the [http://www.erlang.org/download.html]standard distribution (or
[http://www.erlang.org/download.html]standard distribution (or
+
 
someone explains to me how to specify a single SSL port without my patch!).
 
someone explains to me how to specify a single SSL port without my patch!).
  
Line 141: Line 143:
  
  
Environment
+
=== Environment ===
  
  
Line 149: Line 151:
 
[http://www.linux.org]Linux distributions.
 
[http://www.linux.org]Linux distributions.
  
 +
'''NOTE:''' R11B-5 has some ''bugs'' in the inet_ssl_dist code that prevent SSL distribution from working. [http://www.erlang.org/pipermail/erlang-bugs/2007-July/000386.html]
 +
There are patches to fix it. [http://www.erlang.org/pipermail/erlang-patches/2007-July/000184.html] [[User:BruceFitzsimons|BruceFitzsimons]] 10:12, 14 September 2007 (BST)
  
 
+
=== diagram ===
Much testing and development was done on a laptop loaded with  
+
Much testing and development was done on a laptop loaded with  
 
release 6.10 of the [http://www.ubuntu.com]Ubuntu desktop  
 
release 6.10 of the [http://www.ubuntu.com]Ubuntu desktop  
 
[derived from [http://www.debian.org]Debian]  
 
[derived from [http://www.debian.org]Debian]  
 
distribution.  All servers run Erlang
 
distribution.  All servers run Erlang
[http://erlang.org/download/otp_src_R11B-2.tar.gz]R11B-2,
+
[http://erlang.org/download/otp_src_R11B-2.tar.gz]R11B-2,
and are
+
and are [[image:dist_erlang_thru_firewalls.png]]are visible on
[[image:dist_erlang_thru_firewalls.png]]are visible on</a>
+
three different (public) networks distributed across the Internet.
three different (public) networks distributed across
+
the Internet.
+
 
+
  
  
 
Each server has it's own hardware firewall (even the two on the same
 
Each server has it's own hardware firewall (even the two on the same
public IP block); two of the servers also run software firewalls.
+
public IP block); two of the servers also run software firewalls.
  
  
Line 180: Line 181:
  
  
  Details
+
=== Details ===
  
  
Line 269: Line 270:
  
  
<p>
+
 
 
I use RCS for local source control.
 
I use RCS for local source control.
 
<code caption="patch to R11B-2 inet_ssl_dist.erl (which had no version number)">
 
<code caption="patch to R11B-2 inet_ssl_dist.erl (which had no version number)">
Line 352: Line 353:
 
I hope that is enough to get you started using secure communications between
 
I hope that is enough to get you started using secure communications between
 
your Erlang nodes.
 
your Erlang nodes.
 +
[[Category:HowTo]]

Revision as of 01:55, 17 January 2011

Contents

Author

Michael McDaniel

Updates

Added link to application edids.org MichaelMcDaniel 17:55, 16 January 2011 (PST)

Added updates about R11B-5 SSL distribution problems and fixes BruceFitzsimons 10:12, 14 September 2007 (BST)

Title

Distributed Erlang using SSL through firewalls

License

 Copyright (c)  2006 Michael McDaniel.
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.2
 or any later version published by the Free Software Foundation;
 with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
 Texts.  A copy of the license can be seen by clicking the link entitled "GNU
 Free Documentation License".

[1]GNU Free Documentation License XML

[2]GNU Free Documentation License HTML

URL links in this paper were valid as of Mon Nov 20 22:03:25 PST 2006.

References

[3]1) Erlang SSL documentation [4]2) message from mailing list [5]3) message from mailing list

Overview

This howto describes how I got distributed Erlang nodes (each running on different physical servers) to communicate using the Secure Sockets Layer (SSL) protocol over the Internet through firewalls. I had control of the firewalls, and you will need control of your firewalls, to do what I describe here. Firewall configuration is dependent upon what kinds of firewalls are being used, and the network architecture. I also show a patch to inet_ssl_dist.erl which creates a new kernel environment variable to force SSL communication over a pre-determined port. This (or something similar) is a necessity if you want to limit your open firewall ports.


I do not describe the commands needed to configure firewalls. I do, however, give general guidelines which are required for any firewall with regards to opening and forwarding ports to have the Erlang nodes communicate.


Once you configure a node to use SSL, it can only talk with other nodes using SSL.


Introduction

Motivation

There are applications where you may want Erlang nodes to communicate over the Internet. Perhaps some distributed processing or data gathering. Perhaps embedded controllers doing remote control or monitoring, or TiVO (tm) like devices "phoning home" or receiving programming updates. Perhaps a distributed Mnesia database. Also useful would be if they communicate over the Internet securely using the SSL protocol.

Application

It happens that my application is a security application [6]. It determines if certains hosts (IPs) are attempting to compromise a server (what that consists of is a subject for a different howto). Once the application decides that an IP is an attacker, it takes some action(s). Attacks from that IP generally will be made on any server on that network (that is, any server with Internet visibility on that subnet).


Attacking IPs typically attack other servers on other networks, as well. The three networks I use as an example contain four servers of interest with Internet visibility. I often see the same IP attacking other servers on other networks.


Because the application is run on all the servers, any single server is fully capable of determining if an IP is an attacker and taking appropriate action. However, I decided I would have my application "warn" other servers of attacking IPs and save doing the same analysis work repeatedly on each server.

Main Text

Communications Overview

If multiple servers are on a single network, having Erlang talk amongst them is trivial. Simply use

Registered_Name ! {From, Message}. 
OR
global:send(Registered_Name, {From, Message}).

and it works (well, almost that simple).


It becomes more complex if you want to get through firewalls over the Internet (and want to use SSL). The servers I wish to communicate amongst each other are distributed over the Internet, and I want them talking privately using SSL.


You will need control of the firewalls and servers in question so that you can set the proper configurations. The techniques discussed map to any application that has inter-node communication using standard Erlang semantics.


Erlang can already "talk SSL". However, presently, I know of no standard way to choose what port to use. I have tried using the

-ssl proxylsport NNNN

command line flag and was unsuccessful in having it work as I expected (I thought it would limit SSL communications to port NNNN). I describe a patch to Erlang which creates a new kernel variable (inet_ssl_port) enabling selection of a specific port over which to "talk SSL" (similar to inet_dist_listen_{min,max} for normal TCP communications, though this new variable for SSL enables setting only a single port rather than a range of ports). I have submitted the patch to the [7]Ericsson Erlang developers, and I am hopeful that it or something like (perhaps a range of ports) it will get incorporated into the [8]standard distribution (or someone explains to me how to specify a single SSL port without my patch!).


If you do not have control of your firewalls, most of the techniques described here will not be useful to you. You will probably have to use something such as over SSL (though certainly, being able to control your SSL port will be useful!).


Environment

I have four servers (one Erlang node on each server), all running various release 9.x versions of [9]SuSE [10]GNU [11]Linux distributions.

NOTE: R11B-5 has some bugs in the inet_ssl_dist code that prevent SSL distribution from working. [12] There are patches to fix it. [13] BruceFitzsimons 10:12, 14 September 2007 (BST)

diagram

Much testing and development was done on a laptop loaded with release 6.10 of the [14]Ubuntu desktop [derived from [15]Debian] distribution. All servers run Erlang [16]R11B-2, and are Dist erlang thru firewalls.pngare visible on three different (public) networks distributed across the Internet.


Each server has it's own hardware firewall (even the two on the same public IP block); two of the servers also run software firewalls.



The Erlang port mapper daemon epmd runs on the IANA ([17]Internet Assigned Number Authority) well-known port [18]4369 and is reserved for both TCP and UDP protocols. It was registered by [19]Ericsson Corporation for the [20]Erlang language. Using my patch, I was able to force the SSL communications to run on the same port for each node. This allowed me to open only two extra ports on the firewalls, 4369 for epmd, and a port for SSL traffic.


Details


$ this is a user command (note $ prompt)
# this is a root command (note # prompt)


System: Ubuntu rel 6.10 (aka Edgy; Debian derived)


$
$ uname -a
Linux delora 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686
GNU/Linux
$
$ erl --version -s init stop      (OR erl -s init stop)
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> 
$
$ gcc --version
gcc (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$
$ echo version | openssl
OpenSSL> OpenSSL 0.9.8b 04 May 2006
OpenSSL> 
$
$ isql --version
unixODBC 2.2.11
$
$ ./configure  --with-odbc  --x-libraries=/usr/lib/unixODBC/ \
  --x-includes=/usr/include/ --with-ssl                      \
  --x-includes=/usr/local/ssl/include/openssl/               \
  --x-libraries=/usr/local/ssl/lib/

  [output not shown]

$ sudo make install

  [output not shown]

$ cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
$ for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
$ cd /to/my/application/directory
$ erl -s systools make_script "start_ssl_clean" -s init stop
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> 
$

Here is my SSL boot script.


%
% make sure all version numbers are correct
% * some will be in /usr/local/lib/erlang/bin/start.script
% * erts vers shows when starting command line erl OR
% * erlang:system_info(system_version)
% * ssl shows with 'ls /usr/local/lib/erlang/lib/ssl<TAB>'
%
%
{release, {"OTP  APN 181 01","R11B"}, {erts, "5.5.2"},
 [{kernel,"2.11.2"},
  {stdlib,"1.14.2"},
  {sasl,"2.1.4"},
  {os_mon,"2.1.1"},
  {ssl,"3.0.12"}]}.
% see
% file:///usr/local/lib/erlang/lib/ssl-3.0.12/doc/html/ssl_distribution.html
% for ssl info
% 
% cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
% for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
%
% systools:make_script("start_ssl_clean",[]).


I use RCS for local source control.

$ rcsdiff inet_ssl_dist.erl
80c80,86
<     case ssl_prim:listen(0, [{active, false}, {packet,4}] ++ 
---
> 
>     case application:get_env(kernel, inet_ssl_port) of
>          {ok, SSL_port} -> SSL_port ;
>          _undefined     -> SSL_port = 0
>     end ,
> 
>     case ssl_prim:listen(SSL_port, [{active, false}, {packet,4}] ++ 


Communication between nodes is initiated over epmd's well-known port 4369. In default behaviour, a different port is (randomly) chosen for the nodes to exchange messages. Since I do not want to open all ports on my firewalls (how would I know which one might be active?), I need some way to limit what ports are used to communicate between nodes over SSL. That is the purpose of the simple hack described above, so I can choose my port.


I start the application using the SSL boot file (-boot ${HOME}/start_ssl_clean). I limit the ports using the sys.config file (-config ${HOME}/sys.config).



merl is a 'safe' account created only to run certain programs, and with limited authority.


#!/bin/sh

# -setcookie is needed else I get 'No home for cookie file'
# the cookie a123 is a throwaway as the app creates a special
# cookie anyway
#
# copy *.pem to /home/merl
# copy start_ssl_clean.* to /home/merl
# 

export HOME=/home/merl
export HOST=`/bin/hostname`


(
/bin/su merl -c                                       \
"
cd /home/merl ;
/usr/local/bin/erl                                   \
-boot ${HOME}/start_ssl_clean                        \
-proto_dist inet_ssl                                 \
-ssl_dist_opt client_certfile ${HOME}/ssl_client.pem \
-ssl_dist_opt server_certfile ${HOME}/ssl_server.pem \
-ssl_dist_opt verify 1 depth 1 -name ${HOST}_myapp   \
-detached -config ${HOME}/sys.config               \
-s my_app start                                      \
"
)

#% end


$ cat sys.config

%% for example
[ {kernel,  [ {inet_ssl_port, 4370} ] }
].


I hope that is enough to get you started using secure communications between your Erlang nodes.