How to finish the process in case of error

Home Forums Basic Erlang How to finish the process in case of error

This topic contains 4 replies, has 3 voices, and was last updated by  Aklaim 1 year ago.

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #37013

    Aklaim
    Member

    How to finish the process in case of error.

    For example:
    scheme
    Worker manager (gen_server)- contains information about the processes.

    Worker supervisor (supervisor)- start worker processes.

    Worker (gen_server) – worker process. Opens a connection with rabbitmq at startup.

    The problem is, that after application starts, each worker process will open a connection with rabbitmq, and save process pid.

    If critical error occur and worker process terminate, Worker supervisor will restart it, but connection which worker process started would remain, because they dont belong to current supervisor tree, and their pids will be lost.

    What is the right way to terminate processes (Connection 1, Connection 2, …) in case of errors?

    #37014

    pzel
    Member

    Hi Alkaim,

    If you link Connection 1 to Worker 1 and so forth, the worker processes dying should take the connection processes with them.

    What exactly is the issue you’re having with the setup above? Does RabbitMQ keep Connection processes alive after their respective Workers have died?

    Simon

    • This reply was modified 1 year ago by  pzel.
    #37021

    Aklaim
    Member

    Hi Simon,

    Link will not work. If i create link between them and then try to terminate Worker, RabbitMq superviser will crash with error:

    “12:26:57.577 [error] Supervisor {<0.106.0>,amqp_channel_sup} had child channel started with amqp_channel:start_link(network, <0.97.0>, 1, <0.107.0>) at <0.108.0> exit with reason killed in context child_terminated
    12:26:57.578 [error] Supervisor {<0.106.0>,amqp_channel_sup} had child channel started with amqp_channel:start_link(network, <0.97.0>, 1, <0.107.0>) at <0.108.0> exit with reason reached_max_restart_intensity in context shutdown
    12:26:57.578 [error] Connection (<0.97.0>) closing: internal error in channel (<0.108.0>): killed”

    Yes, RabbitMQ keep Connection processes alive after their respective Workers have died. Actualy they are not connections, they are channels to single connection.

    As a solution, should i store all connections pid in ets table of Worker Manager? And when Worker restarts, give him pid of already opened connection. But this will not solve the problem of lost connections if Worker Manager crash.

    • This reply was modified 1 year ago by  Aklaim.
    #37184

    mpmiszczyk
    Member

    Hi;

    I think what you are looking for is another layer of abstraction (how OO people would call it) or separation of responsibilities (how Uncle Bob describes it in SOLID; and what I find to be much better rule for system design).

    In general what you would like to do is to have some worker with ability to fail (where this work is his responsibility) and at the same time you have connections you would like to keep (which is second responsibility). Easiest way to deal with it is to use two processes. One would start and keep track of connection (set up environment and start worker), second would be just doing it’s work. You could easily fit this approach to your application by adding new level to supervision tree, that way should have some more control over RabbitMQ connection failure (WorkerConnection can monitor, not link RabittMQ processes, and as parent will gracefully terminate all Workers on any kind of failure).

    Regarding the WorkerManager (which would still keep the Pid‘s of Workers, and not WorkerConnections) things might get little more complicated. In general processes keeping state should plain enough to never fail. Even if you keep your data in ETS, it is linked to some process, and it could disappear. You could read little more about it
    http://steve.vinoski.net/blog/2011/03/23/dont-lose-your-ets-tables/
    http://steve.vinoski.net/blog/2013/05/08/implementation-of-dont-lose-your-ets-tables/

    Easier way to do it would be putting WorkerManager in supervision tree (above WorkerConnection, or at the same level with rest_for_one strategy). That way when you somehow loose you state in manager, all connections will be closed (not lost) and than restarted. This seems more in the Erlang spirit, and is fully acceptable since connection data is not business data, and could be lost and recreated without any problems.

    best
    marcin

    • This reply was modified 1 year ago by  mpmiszczyk.
    • This reply was modified 1 year ago by  mpmiszczyk.
    #37378

    Aklaim
    Member

    Sorry I could not answer.

    Problem was solved by using an additional process.

    Worker opens a connection and creates a child process, passing the open connection pid to it.
    Child sets a flag (trap_exit = true).

    As soon as Worker completes (under any circumstances) trap_exit works and connection will be closed.

    Example code:

    init(ChannelPid) ->
      erlang:process_flag(trap_exit, true),
      loop(ChannelPid).
    
    loop(ChannelPid) ->
      receive
        {'EXIT', _From, _Reason} ->
          amqp_channel:close(ChannelPid);
    
        _SomeMsg ->
          loop(ChannelPid)
      end.

    Thank you for useful links.

    • This reply was modified 1 year ago by  Aklaim.
Viewing 5 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.