When I need to configure something in a complicated way, I find myself reviewing the embedded language that provided the server to create a flexible configuration. In Redis, you can improve the performance of requests, in Nginx, you can improve the handling of incoming requests, FreeSwitch offers alternatives for performing the same tasks using different embedded languages. Even in a software like TheGimp, you can add your own code to edit images.
Among the embedded languages, JavaScript and Lua are the most commonly used languages. JavaScript is very well known to the Erlang community because it was integrated (as a port, it is not implemented on top of Erlang) in popular products such as CouchDB and Riak. But I think the more exciting option, raised by Erlang Co-Creator, Robert Virding, is to implement Lua on top of Erlang, which can be used as an embedded language.
Why? Let’s take a look.
Complex Configuration
Usually, when tasked with fitting the definition of a behaviour we would like to configure, we would create an algorithm in a simple language such as Lua. This saves us from performing activities like:
-
Defining the configuration to fit all of the cases.
-
Reading and transmitting that information to be prepared for use.
-
Writing specific code to handle that standardised information.
This kind of implementation is used frequently. It is easy to think of examples you’re likely to come across in day-to-day life. For example, supermarket offers which have multiple dependencies, commissions for salespeople which might feature variable ranges and percentages based on the type of sale, amount of sale or tax brackets, even SMS, emails or HTTP requests could be considered examples.
To demonstrate this, let’s look at an example of developing a load balancer. This is a simple project using cowboy, and luerl as dependencies.Dependingdependencies., and depending on the headers and other information from the HTTP request, we can send it to the different web servers we have available and configured.
Based on the above premise we can write the configuration as follows:
{load_balancer, [
{servers, [
{odin, "1.1.1.1", [
{in, method, [post]},
{'>', <<"content-length">>, 10000},
{in, <<"accept">>, [<<"json">>]}
]},
{thor, "1.1.1.2", [
{in, method, [get, post]},
{'==', http_version, <<"2">>}
]},
{balder, "1.1.1.3", [
{in, method, [get, post]}
]}
]}
]}.
As you can see, we have to define a 3-tuple system for the rules with the operation in the first element and the two operators as the following elements inside of the tuple. In addition, we are occasionally handling the second element as a header name (if it is a binary), but at other times it’s the method we use to perform the request (using the atom “method”) and other times still, the HTTP version is used to gather the information.
The problem is that we have no closed specifications. We could add more elements or even change the meaning of them. What if we want to use logical modifiers like “and” and “or” to join the checks instead of assuming they are always using “and”? This change will add more complexity to our configuration and more complexity means more possibilities for making mistakes.
At the moment, if the configuration is wrong or adds something that is not granted, it is up to us to trigger the corresponding error and point to where it is to make it easier to fix. As you can imagine, that is not an easy thing to do if you are handling it during runtime.
Lua saves the day!
It’s not unconventional to think about configuration in terms of a specific code. At this point, Lua code could be put in charge of the definition because it is based on Lua semantics.
We only need the information for the configuration and running of the snippet to give us the desired behaviour we want to plug into the correct place. For example, the previous configuration could be written as:
local odin = "1.1.1.1"
local thor = "1.1.1.2"
local balder = "1.1.1.3"
local method = http.method()
local size = tonumber(http.header("content-length")) or 0
local accept = config.split(http.header("accept") or "", ", ")
local httpver = http.version()
if method == "post" and size > 10000 and config.member("json", accept) then
return odin
elseif config.member(method, {"get", "post"}) and httpver == "2" then
return thor
elseif config.member(method, {"get", "post"}) then
return balder
end
As you can see, we are able to optimise and fix the code to suit our needs, it is shorter and clearer than the original configuration and, most importantly, we can now test and check to be sure it is compiling correctly.
The important thing to keep in mind is that the configuration code must include the functions which are going to be needed to handle the request. In the example above, we are using functions like
http_version()
,
http_header("...")
or even
split(...)
and
member(...)
. These functions should be provided to the interpreter.
Of course, the interpreter also has other functions available, we only need to provide the specific functions that are required for our business logic.
In addition to improving the performance, using these functions we are also improving the security because it has not been able to access functions which are not used. You can check the file luerl_sandbox.erl where it is removing the access to the functions which use the underlying operating system.
Where the code dwells?
Inserting the Lua code into the configuration can be a little tricky. To avoid this, I recommend putting these scripts into the priv directory as a normal Lua file (using the extension .lua) this could even be done inside of a database if we are handling the configuration in an automated way using a key/value storage configuration such as etcd.
The most important thing to keep in mind before running that code is to have a specific task which helps you to parse it and ensure the code is correct. One solution is to conduct a testing phase to ensure that the configuration is not breaking or negatively impacting other parts of the system.
For example, in the previous code, we would write a couple of libraries, one called utils for the functions needed for strings and tables and another called http needed for the HTTP functions. An example would be:
-module(luerl_lib_http).
-export([load/1, install/1, put_request/2]).
-include_lib("luerl/include/luerl.hrl").
-define(REQUEST, http_request).
load(St) ->
luerl:load_module([<<"http">>], luerl_lib_http, St).
install(St) ->
luerl_heap:alloc_table(table(), St).
put_request(Request, St) ->
luerl:put_private(?REQUEST, Request, St).
table() ->
[
{<<"method">>, #erl_func{code = fun method/2}},
{<<"version">>, #erl_func{code = fun version/2}},
{<<"header">>, #erl_func{code = fun header/2}}
].
method(_Args, St) ->
#{method := Method} = request(St),
{[Method], St}.
version(_Args, St) ->
#{version := Version} = request(St),
{[Version], St}.
header([Name|_], St) when is_binary(Name) ->
#{headers := Headers} = request(St),
{[maps:get(Name, Headers, nil)], St};
header(Args, St) ->
luerl_lib:badarg_error(<<"header">>, Args, St).
request(St) -> luerl:get_private(?REQUEST, St).
defmodule LuerlLib.Http do
require Record
Record.defrecord(:erl_func, Record.extract(:erl_func, from_lib: "luerl/include/luerl.hrl"))
@private_key :http_request
def load(state), do: :luerl.load_module(["http"], __MODULE__, state)
def install(state), do: :luerl_heap.alloc_table(exports(), state)
def put_request(state, request), do: :luerl.put_private(@private_key, request, state)
defp exports do
[
{"method", erl_func(code: &method/2)},
{"version", erl_func(code: &version/2)},
{"header", erl_func(code: &header/2)}
]
end
defp method(_args, state), do: {[request(state).method], state}
defp version(_args, state), do: {[request(state).version], state}
defp header([name | _], state) when is_binary(name),
do: {[Map.get(request(state).headers, name)], state}
defp header(args, state), do: :luerl_lib.badarg_error("header", args, state)
defp request(state), do: :luerl.get_private(@private_key, state)
end
-module(luerl_lib_config).
-export([load/1, install/1]).
-include_lib("luerl/include/luerl.hrl").
load(St) ->
luerl:load_module([<<"config">>], luerl_lib_config, St).
install(St) ->
luerl_heap:alloc_table(table(), St).
table() ->
[
{<<"split">>, #erl_func{code = fun split/2}},
{<<"member">>, #erl_func{code = fun member/2}}
].
member([Entry, #tref{}=Table], St) ->
#table{a = Array} = luerl_heap:get_table(Table, St),
Result = array:foldl(fun
(_, V, false) when V =:= Entry -> true;
(_, _, Acc) -> Acc
end, false, Array),
{[Result], St};
member(Args, St) ->
luerl_lib:badarg_error(<<"member">>, Args, St).
split([String, Sep], St) when is_binary(String), is_binary(Sep) ->
{Tref, St1} = luerl:encode(string:split(String, Sep, all), St),
{[Tref], St1};
split(Args, St) ->
luerl_lib:badarg_error(<<"split">>, Args, St).
defmodule LuerlLib.Config do
require Record
Record.defrecord(:erl_func, Record.extract(:erl_func, from_lib: "luerl/include/luerl.hrl"))
Record.defrecord(:table, Record.extract(:table, from_lib: "luerl/include/luerl.hrl"))
def load(state), do: :luerl.load_module(["config"], __MODULE__, state)
def install(state), do: :luerl_heap.alloc_table(exports(), state)
defp exports do
[
{"split", erl_func(code: &split/2)},
{"member", erl_func(code: &member/2)}
]
end
defp split([string, separator | _], state) when is_binary(string) and is_binary(separator) do
{table_ref, state} = :luerl.encode(String.split(string, separator), state)
{[table_ref], state}
end
defp split(args, state), do: :luerl_lib.badarg_error("split", args, state)
defp member([entry, table_ref | _], state) when Record.is_record(table_ref, :tref) do
table(a: array) = :luerl_heap.get_table(table_ref, state)
found = :array.sparse_foldl(fn _index, value, acc -> acc or value === entry end, false, array)
{[found], state}
end
defp member(args, state), do: :luerl_lib.badarg_error("member", args, state)
end
As you can see, we are implementing the functions we need and making them available to our Lua interface under the
config
and
http
packages. To load these functions, we have to run the load function which is exported in both modules. NOTE:
install
is used as a callback function in
load_module
A great benefit of doing things this way is the “compile once, and ready many” approach. The config does not have to be loaded for every request. If we put it all together in our application, it would something like like the following
Add a
build_request
function and its helpers to your
luerl_lib_http
and
LuerlLib.Http
modules.
-export([load/1, install/1, put_request/2, build_request/1]). % add export
build_request(CowboyReq) ->
#{method => string:lowercase(cowboy_req:method(CowboyReq)),
version => version_to_binary(cowboy_req:version(CowboyReq)),
headers => cowboy_req:headers(CowboyReq)}.
version_to_binary('HTTP/1.0') -> <<"1.0">>;
version_to_binary('HTTP/1.1') -> <<"1.1">>;
version_to_binary('HTTP/2') -> <<"2">>.
def build_request(cowboy_req) do
%{
method: cowboy_req |> :cowboy_req.method() |> String.downcase(),
version: version_to_binary(:cowboy_req.version(cowboy_req)),
headers: :cowboy_req.headers(cowboy_req)
}
end
defp version_to_binary(:"HTTP/1.0"), do: "1.0"
defp version_to_binary(:"HTTP/1.1"), do: "1.1"
defp version_to_binary(:"HTTP/2"), do: "2"
Add handler modules for handling the request
-module(luerl_lib_handler).
-export([init/2]).
init(CowboyReq, {Form, St}) ->
LuaReq = luerl_lib_http:build_request(CowboyReq),
St1 = luerl_lib_http:put_request(LuaReq, St),
Reply = case luerl:call_chunk(Form, St1) of
{ok, [], _St2} ->
cowboy_req:reply(404, #{}, <<"no matching route">>, CowboyReq);
{ok, Rets, St2} ->
[Backend] = luerl:decode_list(Rets, St2),
cowboy_req:reply(200, #{}, Backend, CowboyReq);
{lua_error, _Reason, _St2} ->
cowboy_req:reply(500, #{}, <<"routing error">>, CowboyReq)
end,
{ok, Reply, {Form, St}}.
defmodule LuerlLib.Handler do
def init(cowboy_req, {form, state}) do
lua_req = LuerlLib.Http.build_request(cowboy_req)
state1 = LuerlLib.Http.put_request(state, lua_req)
reply =
case :luerl.call_chunk(form, state1) do
{:ok, [], _state2} ->
:cowboy_req.reply(404, %{}, "no matching route", cowboy_req)
{:ok, returns, state2} ->
[backend] = :luerl.decode_list(returns, state2)
:cowboy_req.reply(200, %{}, backend, cowboy_req)
{:lua_error, _reason, _state2} ->
:cowboy_req.reply(500, %{}, "routing error", cowboy_req)
end
{:ok, reply, {form, state}}
end
end
With all of that defined, we can now call the relevant functions from our application startup:
-module(luerl_lib_app).
-behaviour(application).
-export([start/2, stop/1]).
start(_StartType, _StartArgs) ->
{ok, Form, St} = build_lua_state(),
Dispatch = cowboy_router:compile([{'_', [{'_', luerl_lib_handler, {Form, St}}]}]),
{ok, _} = cowboy:start_clear(luerl_lib_listener, [{port, 8080}], #{env => #{dispatch => Dispatch}}),
luerl_lib_sup:start_link().
stop(_State) ->
ok = cowboy:stop_listener(luerl_lib_listener).
build_lua_state() ->
St0 = luerl_lib_http:load(luerl_lib_config:load(luerl_sandbox:init())),
luerl:loadfile("priv/config.lua", St0).
Supervisor
-module(luerl_lib_sup).
-behaviour(supervisor).
-export([start_link/0, init/1]).
start_link() ->
supervisor:start_link({local, ?MODULE}, ?MODULE, []).
init([]) ->
{ok, {#{strategy => one_for_one, intensity => 1, period => 5}, []}}.
defmodule LuerlLib.Application do
use Application
def start(_type, _args) do
state =
:luerl_sandbox.init()
|> LuerlLib.Config.load()
|> LuerlLib.Http.load()
{:ok, form, state} = :luerl.loadfile(~c"priv/config.lua", state)
dispatch = :cowboy_router.compile([{:_, [{:_, LuerlLib.Handler, {form, state}}]}])
{:ok, _} =
:cowboy.start_clear(:luerl_lib_listener, [port: 8080], %{env: %{dispatch: dispatch}})
LuerlLib.Supervisor.start_link()
end
end
Notice that we load the file once (with the lines below) and pass it down to the handler, it helps us obtain the forms that we need and the handler will always execute based on that initially loaded configuration.
{ok, Form, St1} = luerl:loadfile("priv/config.lua", St0).
{:ok, form, state} = :luerl.loadfile(String.to_charlist(path), state)
Scaling up
Lua scales up because it is built on top of Erlang. This means Lua is using the same processes as Erlang does, it also means we are not using ports to communicate with the Lua interpreter, we have the Lua interpreter running on Erlang.
This makes a great difference in comparison to JavaScript because if you are handling millions of requests and all of them require a JavaScript snippet, this can cause a bottleneck very quickly if you have to limit the number of ports or requests.
On the other hand, Lua is using native Erlang functions when it calls the functions we want to provide for the interpreter. That makes a clear improvement, saving us from performing data serialisation or transformation.
Conclusion
Using a language for flexible configuration gives us the possibility to create an easy interface to provide configurations, reduce the amount of code we need to write, and improve the maintenance without jeopardising the performance of the system. At the moment, you can use Lua as we have explained during the article or you can jump into PHP if you need to process text or templates on top of Erlang or Elixir.
Alternatively you can join the community and provide other solutions which help us build better fit-for-purpose software.
Get in touch
if you need help building your system with one of these solutions.
The post
How to use Lua for flexible configurations in Erlang and Elixir
appeared first on
Erlang Solutions
.
