From P..... to Erlang

What is Swelen ?

• Mobile advertising (iOS, Android, WebMobile)
• RTB

The adserver team (2 people)

• https://github.com/jeromer/
• https://github.com/teh-cmc/
• no clue about Erlang
• no clue about functional programming

Before the rewrite

Adserver was

• written in P..... (+ MySQL, + Redis, + Nginx, + uwsgi, + ...)
• by a math genius (initial author, left the company)
• with piles of "temporary fixes"
• without any test
• (what could possibly go wrong ?)

in the end

• a nigthmare to maintain
• costly to add new features (exponential)

We could not keep up with the business

New adserver requirements

• fast (< 50ms max to return an ad)
• scalable (10x traffic anytime)
• maintainable
• observable
• no HTTP caching
• synchronous reads
• asynchronous writes

We had to pick a language

• Python (legacy adserver)
• PHP (hell no !)
• C (too complex to do well)
• Go (lots of interesting concepts)
• Rust (no stable version available)
• Erlang

Why we chose Erlang

• designed for fault-tolerance
• built-in scalability
• built-in observability (observer, etop)
• no locks :) (huge win)
• competition used Erlang

With Erlang, you not only have a language.

You also get the entire platform for free !

Time to learn

• LYSE (thank you @mononcqc)
• ton of manual pages (erldocs)
• lots of Erlang Factory videos
• Cowboy manual & source code
• Trial and (lots of) errors

Erlang, a week later

• 1st real world feature ready
• sent the entire trafic on a single node
• worked well, scaled well

WIN ! 🍺

Time to write the rest

We ♥ ETS

• blazingly fast
• super simple to use
• manage 10 different cache layers
• no longer need Redis

We ♥ pattern matching

• extremely powerful
• code much safer, with much less error checks (let it crash)
• much easier to maintain

We ♥ releases

• trivial deployements
  • git push
  • make release
  • tar + scp
  • restart the adserver
  • done
• (we do no do hot upgrades)

We ♥ observability (aka WTF is going on ?)

• remote shell on any node (shell history support ?)
• direct access to our API
• ♥ erl -name jerome -setcookie foo -hidden -remsh x.com -s observer

Stupid mistakes learned the hard way 1/2

• store BIG terms in ETS -> 💣 ETS performance
• avoid ets:select if possible -> full table scan -> 💣
• ETS table is lost when process crashes -> create a dedicated supervisor

Stupid mistakes learned the hard way 2/2

• create an OTP app for anything -> 💣 message passing
• pass BIG terms between process -> 💣 message passing

Summary

We went from

• unmaitainable adserver (by 2 people)
• complex infrastructure
• high response time
• unable to keep up with the business

To

• single release (aka tarball)
• response time is ~25ms (p90)
• can do trafic x8 without even sweating
• simplified infrastructure
• one maintainer (me)
• adding a feature takes 2/3 days in avg (w/ tests)

Future plans

• switch to Mnesia
• use dialyzer

Thoughts about Erlang

• Erlang not hard to learn, real motivation is required though
• Thinking in Erlang challenging sometimes
• Documentation complete, could be more accessible
• Unicode support could be more "natural"

Thank you Ericsson & the Erlang community

Thank you

• jerome.renard@kwanko.com
• @jeromerenard
• http://github.com/jeromer/