Using Erlang's Observer in Kubernetes for an Elixir Release
Sep 2, 2019
3 minute read

Erlang’s observer is awesome. There are a lot of anecdotes on the usefulness of it for debugging production systems, and one of my favorites is how the Phoenix team used it to find a process that was filling up it’s mailbox.

Most of the guides out there are for using it with vanilla ssh.

We’ll work through getting a local Observer talking with a Kubernetes pod in production.

Pre-requisites

  • A Phoenix App using Elixir 1.9 Releases (At some point, I plan to write a complete getting started with Kubernetes and Phoenix from scratch guide)
  • That is deployed in Kubernetes
  • All kubectl commands should be run in your application’s namespace

A shared secret (or cookie, as Erlang calls it) is required for connecting to Erlang nodes.

mix phx.gen.secret | base64 | tr -d "\n" > release_cookie
kubectl create secret generic release --from-file=release_cookie
rm release_cookie

Note: the base64 part of the above command is because the release docs state you should restrict the characters to that encoding.

          env:
            - name: RELEASE_COOKIE
              valueFrom:
                secretKeyRef:
                  name: release
                  key: release_cookie

Update your release files

Add or uncomment the following to your rel/env.sh.eex

export RELEASE_DISTRIBUTION=name
export RELEASE_NODE=<%= @release.name %>@127.0.0.1

case $RELEASE_COMMAND in
  start*|daemon*)
    ELIXIR_ERL_OPTIONS="-kernel inet_dist_listen_min $BEAM_PORT inet_dist_listen_max $BEAM_PORT"
    export ELIXIR_ERL_OPTIONS
    ;;
  *)
    ;;
esac

This does two important things:

  1. It exports the RELEASE_NODE variable. Without this, the node will not let you connect to it from outside of the pod. I received Could not contact remote node budgetsh@127.0.0.1, reason: :nodedown. Aborting... when I tried.
  2. It gives you a known port to forward.

Next, add that environment variable to your K8S file:

          env:
            - name: BEAM_PORT
              value: 9001 # this is arbitrary
            - name: RELEASE_COOKIE

Deploy the New Code

Ensure your code and kubernetes changes have been deployed.o

Use Observer

First, you are going to need to forward some ports. Specifically, BEAM_PORT, and 4369. 4369 is for epmd, the erlang port mapper daemon.

 kubectl port-forward POD_NAME 9001 4369

Next, spin up an IEX session with the correct cookie:

iex --name $(whoami)@127.0.0.1 --cookie $(kubectl get secret release -o "go-template={{index .data \"release_cookie\"}}" | base64 -D)

From there, run :observer.start() in the iex session, and select your pod’s erlang node from the menu bar.

Note: I tried getting this working in one command, ie:

iex --name dan@127.0.0.1 --cookie SECRET_COOKIE --remsh budgetsh@127.0.0.1

but, when I got into an IEX session, I received: function :observer.start/0 is undefined. This is because my Phoenix application does not specify :observer, and :wx as applications, and it should not. Production applications do not need this bloat; runtime-tools is likely enough.

Conclusion

  • At some point, I’d like to get this working with proper Erlang node clustering.
  • It’s probably easy to orchestrate all this better in a one line command. Especially since Kubernetes has an API that provides port forwarding.

    • Right now, I just have an alias for the IEX command:

      alias reliex='iex --name $(whoami)@127.0.0.1 --cookie $(kubectl get secret release -o "go-template={{index .data \"release_cookie\"}}" | base64 -D)'
      
  • Getting an application deployed and an observer open is much easier on gigalixir.

  • Please leave a comment below if these did not work for your, or if there’s a glaring typo somewhere. (or open an issue here

  • Feel free to take a look at my WIP toy app where I’m testing all this stuff out