The largest single piece of that work is where you stopped reading, and how Fluux shares that position with your other clients. The rest is spread thin on purpose: badges that would not clear, a "New messages" line that kept coming back, a conversation that moved while you were reading it, failures that never said why.

Where you left off, on every client

Every XMPP client on your account should agree on where you stopped reading. Fluux publishes that position with XEP-0490, and this release closes the cases where the publication used to stall.

• Your read position now leaves the device in the cases where it used to get stuck : after you reply in a one-to-one chat, in a conversation with nothing currently loaded, and after a first attempt that did not go through. Your other clients stop showing a stale "New messages" line and an inflated badge.
• A position arriving from another client is no longer dropped when Fluux cannot yet place it in its own history. It is kept and applied the moment you open the conversation.
• A conversation whose oldest messages have aged out of the server archive no longer freezes your position. Fluux could stop publishing that conversation altogether, which left your other clients months behind.
• A group chat you read elsewhere now shows what your other clients show , instead of sitting at zero unread with no read marker while they display both.

One unread count, everywhere it appears

There used to be more than one way to answer "how many messages are unread here", and the answers could disagree. Now there is one: the messages that sit after your last read position. Every surface that shows a number reads that same number.

• The sidebar badge, the "New messages" divider, the floating pill and the scroll-to-bottom badge always agree. A conversation left in the background no longer under-counts, and opening a conversation no longer zeroes its badge before you have read anything.
• The divider stays where it is while you are looking at it. It no longer reappears every time you reopen a conversation you have already read to the end.
• A badge that reading could not clear is gone. Your read position also stops moving backwards when the last message you read is missing from the loaded history.
• Rooms say more at a glance : a room row now tells you how many messages are unread, and the unread dot and the mention badge moved after the timestamp so they line up down the list.

A conversation that stays where you put it

Scrolling is how you navigate a conversation. It should never happen on its own.

• Closing an image or a dialog with Escape keeps your place. A late-arriving older message no longer pushes a scrolled-up reader backwards in time either.
• Returning to a room lands where you left it. Opening a room whose read position predates the loaded history no longer strands you at the oldest message, and Home works immediately after opening a conversation.
• The view stays on the newest message when a late link preview, reaction or attachment makes the last message taller, and when the composer collapses back to one line after you send.
• The typing indicator no longer covers messages. Its label wraps to a second line instead of being cut off.

When something fails, Fluux tells you

Silence is the worst answer an app can give.

• Older messages that could not be loaded now show a marker, right above the button that tries again.
• A bookmarked room that could not be rejoined says why : the room wants a password, or your nickname is already taken by somebody else in it, or it only admits members. Joining a password-protected room asks for the password wherever you join from, the sidebar, an invitation or Browse Rooms, and remembers it for next time.
• A message that cannot be read explains which problem it hit : a key this device does not have, an invalid signature, or content that could not be parsed, instead of always blaming a missing key.
• Opening a conversation shows a loading indicator and lets you go back while history loads, and an update that has to upgrade your local history shows a progress bar while it does.

Faster catch-up

• Launching after a long absence and opening a conversation are both faster. Fluux writes far less to local storage while it catches up on history, and upgrading local history after an update is faster too, on large archives especially.
• Group chat history no longer arrives in two visible phases on mobile and in the browser. After a reconnect, a joined room no longer sits inert in the sidebar with no preview and no timestamp.
• A busy room you join no longer announces its recent history as new arrivals.

Encryption, desktop, and the rest

• OpenPGP interoperability : keys generated by Fluux can now be imported by Gajim, existing keys are repaired when you unlock them, and Fluux no longer removes your other clients&apos keys from your published key list. Saving an image from the lightbox no longer writes the raw encrypted file to disk, and re-publishing an unchanged key no longer locks encryption behind a warning.
• A system tray on Windows and Linux : a new setting decides whether closing the window quits Fluux or leaves it running in the tray, wherever the desktop provides one. Clicking a notification restores the window and opens the conversation and the message it came from, and on Linux the notification settings button now opens the right panel on Cinnamon, KDE, XFCE, Budgie and LXQt.
• Connections that used to be refused : Fluux now asks your server where to connect before falling back to its own list of known addresses, so an account on a server whose address changed is no longer locked out. A password with accented or non-Latin characters is sent exactly as you typed it, so accounts that were rejected with "invalid username or password" can sign in. If you were affected and typed an address into the advanced server field, clear it once and Fluux will find the server on its own.
• Korean, strikethrough, and the small things : Korean joins the interface languages, contributed by the community. Markdown strikethrough renders alongside the XEP-0393 form. A deleted message stays deleted everywhere Fluux summarises a conversation, relative dates update after midnight even when the app stays open overnight, and copying several messages at once no longer drops polls and attachments.

Get it

Fluux Messenger 0.17.3 is available for macOS, Windows and Linux, or directly in your browser, from the Fluux Messenger page . If you upgrade and something still feels off, tell us: bug reports and feature requests both go to GitHub Issues .

Distributed systems need async message delivery across nodes. Phoenix provides Phoenix PubSub for this, with pluggable adapters for different backends — officially PG2 and Redis.

This post walks through implementing a Phoenix PubSub adapter backed by EventStore , an Elixir event sourcing library that persists events to PostgreSQL as an append-only log.

Using EventStore as a PubSub backend has a few advantages over the default PG2 adapter:

• No Erlang distribution required : nodes communicate through the shared database rather than through the Erlang cluster, so you can run multiple nodes without configuring Erlang node connectivity.
• Persistence : every broadcast is stored and can be replayed or audited later.

The tradeoffs are the need for storage and the additional latency of a database round-trip per broadcast, making it best suited for lower-throughput messaging where persistence and cross-node decoupling matter more than raw speed. This implementation is a proof of concept — no load tests were performed.

A full implementation of the adapter can be found on Github .

Phoenix.PubSub.Adapter in a nutshell

A Phoenix PubSub adapter must implement a few callbacks specified in Phoenix.PubSub.Adapter :

node_name(adapter_name)

Returns the node name as an atom or binary. Used mainly by Phoenix.Tracker. In most cases:

def node_name(nil), do: node()
def node_name(configured_name), do: configured_name

child_spec(keyword)

Generates the child spec for the adapter. GenServer provides a default; this rarely needs overriding.

broadcast(adapter_name, topic, message, dispatcher)

Called when a message is broadcast through Phoenix.PubSub.broadcast . The adapter_name is the PubSub name with .Adapter appended (e.g. MyApp.PubSub → MyApp.PubSub.Adapter ). The dispatcher module handles local delivery via dispatch/3 .

direct_broadcast(adapter_name, node_name, topic, message, dispatcher)

Same as broadcast/4 with an additional node_name — the message should only reach subscribers on that node.

The EventStore adapter

This section walks through a possible implementation of a Phoenix PubSub adapter that uses EventStore to distribute messages between nodes. This gives a solution that does not depend on Erlang/Elixir distribution, and an event log is stored in case further analysis is needed.

How Phoenix.PubSub works

Phoenix.PubSub uses Elixir’s Registry for subscriptions — each subscribe call registers an entry under the topic key. When broadcast is called, the framework invokes the adapter callback to distribute the message, then handles local dispatch.

The adapter’s job is to get the message to other nodes. For direct_broadcast , only subscribers on the target node should receive it.

The implementation

The adapter is a GenServer that joins the PubSub supervision tree. An eventstore option selects which EventStore module to use (in case you have multiple):

{Phoenix.PubSub,
  [name: MyApp.PubSub,
   adapter: Phoenix.PubSub.EventStore,
   eventstore: MyApp.EventStore]
}

The GenServer stores the EventStore module and the PubSub name in state — both are needed later:

defmodule Phoenix.PubSub.EventStore do
  @behaviour Phoenix.PubSub.Adapter
  use GenServer

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: opts[:adapter_name])
  end

  def init(opts) do
    {:ok,
     %{
       eventstore: opts[:eventstore],
       pubsub_name: opts[:name]
     }}
  end
  #... implementation will come here ...#
end

Note the difference between opts[:name] and opts[:adapter_name] . The former is the name of the PubSub as a whole and is reserved for the Registry. Publishers use it when broadcasting messages. opts[:adapter_name] can be used as the name of the GenServer.

Distributing a message as an event

The GenServer appends a new event to the EventStore when broadcast is called:

def broadcast(server, topic, message, dispatcher, metadata \\ %{}) do
  metadata = Map.put(metadata, :dispatcher, dispatcher)
  GenServer.call(server, {:broadcast, topic, message, metadata})
end

def handle_call(
      {:broadcast, topic, message, metadata},
      _from_pid,
      %{id: id, eventstore: eventstore, serializer: serializer, pubsub_name: pubsub_name} = state
    ) do
  event = %EventStore.EventData{
    # ... constructed below
  }

  res = eventstore.append_to_stream(topic, :any_version, [event])

  # For direct_broadcast targeting the current node, the framework does not
  # call local dispatch, so the adapter must do it. For regular broadcast,
  # the framework handles local dispatch after adapter.broadcast returns :ok.
  current_node = to_string(node())
  destination_node = Map.get(metadata, :destination_node)

  if destination_node == current_node do
    dispatcher = Map.get(metadata, :dispatcher, Phoenix.PubSub)
    Phoenix.PubSub.local_broadcast(pubsub_name, topic, message, dispatcher)
  end

  {:reply, res, state}
end

direct_broadcast/5 is a thin wrapper that sets destination_node in the metadata before delegating to broadcast/5 :

def direct_broadcast(server, node_name, topic, message, dispatcher) do
  metadata = %{
    destination_node: to_string(node_name),
    source_node: to_string(node())
  }
  broadcast(server, topic, message, dispatcher, metadata)
end

source_node is stored in the event metadata for auditing. Routing is handled downstream by comparing destination_node against the current node.

The key decision is how to wrap the message inside %EventStore.EventData{} . Serialization is handled by a pluggable module (defaulting to Phoenix.PubSub.EventStore.Serializer.Base64 ) so the adapter is not tied to a specific encoding. The default serializer base64-encodes :erlang.term_to_binary/ 1 output — this is necessary because EventStore stores data as JSON and raw binaries would be invalid, and because JSON cannot distinguish atoms from strings so a round-trip through term serialization preserves type fidelity.

event = %EventStore.EventData{
  event_type: to_string(serializer),
  data: serializer.serialize(message)
}

A custom serializer can be provided via the serializer option as long as it implements serialize/1 and deserialize/1 .

Handling events, local distribution

Now that events are in the event store, any subscribed process will receive them. The GenServer must subscribe to all topics ( "$all" ). If the event store is also used for another purpose, it’s best to have a separate one for PubSub. The subscription is set up via handle_continue/2 , which runs immediately after init/1 completes, before any other messages can be processed.

def handle_continue(:subscribe, %{eventstore: eventstore} = state) do
  eventstore.subscribe("$all")

  {:noreply, state}
end

def handle_info({:subscribed, _subscription}, state), do: {:noreply, state}

A transient subscription is used since previous messages are not needed. The event store replies with a { :subscribed, subscription } message, which must also be handled. After this, the server will start receiving { :events, events } messages.

To avoid dispatching a local message twice (once from broadcast and once when the event arrives back from EventStore), a unique ID is added to the process state:

def init(opts) do
  {:ok,
   %{
     id: generate_unique_id(opts),
     eventstore: opts[:eventstore],
     pubsub_name: opts[:name],
     serializer: opts[:serializer] || Phoenix.PubSub.EventStore.Serializer.Base64
   }, {:continue, :subscribe}}
end

defp generate_unique_id(opts) do
  unique_id_fn = opts[:unique_id_fn] || fn _name -> UUID.uuid4() end
  unique_id_fn.(opts[:name])
end

A custom ID generator can be provided via unique_id_fn — a function that receives the PubSub name and returns a unique string. Useful when UUID is unavailable or when a deterministic ID is needed for testing.

The id is added to the event’s metadata field as source_id , keeping it separate from the message data. Serialization is delegated to the configurable serializer module. The handle_call for :broadcast becomes:

event = %EventStore.EventData{
  event_type: to_string(serializer),
  data: serializer.serialize(message),
  metadata: Map.put(metadata, :source_id, id)
}

Where the value of id and serializer come from the state, and metadata already contains dispatcher and any destination_node for direct broadcasts. When an event arrives back, source_id identifies the origin node so duplicates can be skipped:

def handle_info({:events, events}, state) do
  Enum.each(events, &local_broadcast_event(&1, state))

  {:noreply, state}
end

defp local_broadcast_event(
       %EventStore.RecordedEvent{
         data: data,
         metadata: metadata,
         stream_uuid: topic,
         eventbies_type: event_type
       },
       %{id: id, serializer: serializer, pubsub_name: pubsub_name} = _state
     ) do
  current_node = to_string(node())

  %{source_id: source_id, destination_node: destination_node, dispatcher: dispatcher} =
    convert_metadata_keys_to_atoms(metadata)

  is_destination? = is_nil(destination_node) or destination_node == current_node

  if not is_nil(dispatcher) and is_destination? and source_id != id and
       event_type == to_string(serializer) do
    Phoenix.PubSub.local_broadcast(
      pubsub_name,
      topic,
      serializer.deserialize(data),
      maybe_convert_to_existing_atom(dispatcher)
    )
  end
end

That’s it — a complete implementation of Phoenix PubSub using EventStore, including support for direct_broadcast via the destination_node metadata field and pluggable serialization.

The complete implementation can be found at esl/phoenix_pubsub_eventstore .
Need help building reliable distributed systems with Elixir? Get in touch with our team .

The post Implementing a Phoenix PubSub Adapter with EventStore appeared first on Erlang Solutions .

When I need to configure something in a complicated way, I find myself reviewing the embedded language that provided the server to create a flexible configuration. In Redis, you can improve the performance of requests, in Nginx, you can improve the handling of incoming requests, FreeSwitch offers alternatives for performing the same tasks using different embedded languages. Even in a software like TheGimp, you can add your own code to edit images.

Among the embedded languages, JavaScript and Lua are the most commonly used languages. JavaScript is very well known to the Erlang community because it was integrated (as a port, it is not implemented on top of Erlang) in popular products such as CouchDB and Riak. But I think the more exciting option, raised by Erlang Co-Creator, Robert Virding, is to implement Lua on top of Erlang, which can be used as an embedded language.

Why? Let’s take a look.

Complex Configuration

Usually, when tasked with fitting the definition of a behaviour we would like to configure, we would create an algorithm in a simple language such as Lua. This saves us from performing activities like:

1. Defining the configuration to fit all of the cases.
2. Reading and transmitting that information to be prepared for use.
3. Writing specific code to handle that standardised information.

This kind of implementation is used frequently. It is easy to think of examples you’re likely to come across in day-to-day life. For example, supermarket offers which have multiple dependencies, commissions for salespeople which might feature variable ranges and percentages based on the type of sale, amount of sale or tax brackets, even SMS, emails or HTTP requests could be considered examples.

To demonstrate this, let’s look at an example of developing a load balancer. This is a simple project using cowboy, and luerl as dependencies.Dependingdependencies., and  depending on the headers and other information from the HTTP request, we can send it to the different web servers we have available and configured.

Based on the above premise we can write the configuration as follows:

{load_balancer, [
    {servers, [
        {odin, "1.1.1.1", [
            {in, method, [post]},
            {'>', <<"content-length">>, 10000},
            {in, <<"accept">>, [<<"json">>]}
        ]},
        {thor, "1.1.1.2", [
            {in, method, [get, post]},
            {'==', http_version, <<"2">>}
        ]},
        {balder, "1.1.1.3", [
            {in, method, [get, post]}
        ]}
    ]}
]}.

As you can see, we have to define a 3-tuple system for the rules with the operation in the first element and the two operators as the following elements inside of the tuple. In addition, we are occasionally handling the second element as a header name (if it is a binary), but at other times it’s the method we use to perform the request (using the atom “method”) and other times still, the HTTP version is used to gather the information.

The problem is that we have no closed specifications. We could add more elements or even change the meaning of them. What if we want to use logical modifiers like “and” and “or” to join the checks instead of assuming they are always using “and”? This change will add more complexity to our configuration and more complexity means more possibilities for making mistakes.

At the moment, if the configuration is wrong or adds something that is not granted, it is up to us to trigger the corresponding error and point to where it is to make it easier to fix. As you can imagine, that is not an easy thing to do if you are handling it during runtime.

Lua saves the day!

It’s not unconventional to think about configuration in terms of a specific code. At this point, Lua code could be put in charge of the definition because it is based on Lua semantics.

We only need the information for the configuration and running of the snippet to give us the desired behaviour we want to plug into the correct place. For example, the previous configuration could be written as:

local odin = "1.1.1.1"
local thor = "1.1.1.2"
local balder = "1.1.1.3"

local method = http.method()
local size = tonumber(http.header("content-length")) or 0
local accept = config.split(http.header("accept") or "", ", ")
local httpver = http.version()

if method == "post" and size > 10000 and config.member("json", accept) then
    return odin
elseif config.member(method, {"get", "post"}) and httpver == "2" then
    return thor
elseif config.member(method, {"get", "post"}) then
    return balder
end

As you can see, we are able to optimise and fix the code to suit our needs, it is shorter and clearer than the original configuration and, most importantly, we can now test and check to be sure it is compiling correctly.

The important thing to keep in mind is that the configuration code must include the functions which are going to be needed to handle the request. In the example above, we are using functions like http_version() , http_header("...") or even split(...) and member(...) . These functions should be provided to the interpreter.

Of course, the interpreter also has other functions available, we only need to provide the specific functions that are required for our business logic.

In addition to improving the performance, using these functions we are also improving the security because it has not been able to access functions which are not used. You can check the file luerl_sandbox.erl where it is removing the access to the functions which use the underlying operating system.

Where the code dwells?

Inserting the Lua code into the configuration can be a little tricky. To avoid this, I recommend putting these scripts into the priv directory as a normal Lua file (using the extension .lua) this could even be done inside of a database if we are handling the configuration in an automated way using a key/value storage configuration such as etcd.

The most important thing to keep in mind before running that code is to have a specific task which helps you to parse it and ensure the code is correct. One solution is to conduct a testing phase to ensure that the configuration is not breaking or negatively impacting other parts of the system.

For example, in the previous code, we would write a couple of libraries, one called utils for the functions needed for strings and tables and another called http needed for the HTTP functions. An example would be:

-module(luerl_lib_http).
-export([load/1, install/1, put_request/2]).

-include_lib("luerl/include/luerl.hrl").

-define(REQUEST, http_request).

load(St) ->
    luerl:load_module([<<"http">>], luerl_lib_http, St).

install(St) ->
    luerl_heap:alloc_table(table(), St).

put_request(Request, St) ->
    luerl:put_private(?REQUEST, Request, St).

table() ->
  [
    {<<"method">>,  #erl_func{code = fun method/2}},
    {<<"version">>, #erl_func{code = fun version/2}},
    {<<"header">>,  #erl_func{code = fun header/2}}
  ].

method(_Args, St) ->
  #{method := Method} = request(St),
  {[Method], St}.

version(_Args, St) ->
  #{version := Version} = request(St),
  {[Version], St}.

header([Name|_], St) when is_binary(Name) ->
  #{headers := Headers} = request(St),
  {[maps:get(Name, Headers, nil)], St};
header(Args, St) ->
  luerl_lib:badarg_error(<<"header">>, Args, St).

request(St) -> luerl:get_private(?REQUEST, St).

defmodule LuerlLib.Http do
  require Record

  Record.defrecord(:erl_func, Record.extract(:erl_func, from_lib: "luerl/include/luerl.hrl"))

  @private_key :http_request

  def load(state), do: :luerl.load_module(["http"], __MODULE__, state)

  def install(state), do: :luerl_heap.alloc_table(exports(), state)

  def put_request(state, request), do: :luerl.put_private(@private_key, request, state)

  defp exports do
    [
      {"method", erl_func(code: &method/2)},
      {"version", erl_func(code: &version/2)},
      {"header", erl_func(code: &header/2)}
    ]
  end

  defp method(_args, state), do: {[request(state).method], state}

  defp version(_args, state), do: {[request(state).version], state}

  defp header([name | _], state) when is_binary(name),
    do: {[Map.get(request(state).headers, name)], state}

  defp header(args, state), do: :luerl_lib.badarg_error("header", args, state)

  defp request(state), do: :luerl.get_private(@private_key, state)
end

-module(luerl_lib_config).
-export([load/1, install/1]).

-include_lib("luerl/include/luerl.hrl").

load(St) ->
    luerl:load_module([<<"config">>], luerl_lib_config, St).

install(St) ->
    luerl_heap:alloc_table(table(), St).

table() ->
  [
    {<<"split">>, #erl_func{code = fun split/2}},
    {<<"member">>, #erl_func{code = fun member/2}}
  ].

member([Entry, #tref{}=Table], St) ->
  #table{a = Array} = luerl_heap:get_table(Table, St),
  Result = array:foldl(fun
    (_, V, false) when V =:= Entry -> true;
    (_, _, Acc) -> Acc
  end, false, Array),
  {[Result], St};
member(Args, St) ->
  luerl_lib:badarg_error(<<"member">>, Args, St).

split([String, Sep], St) when is_binary(String), is_binary(Sep) ->
  {Tref, St1} = luerl:encode(string:split(String, Sep, all), St),
  {[Tref], St1};
split(Args, St) ->
  luerl_lib:badarg_error(<<"split">>, Args, St).

defmodule LuerlLib.Config do
  require Record

  Record.defrecord(:erl_func, Record.extract(:erl_func, from_lib: "luerl/include/luerl.hrl"))
  Record.defrecord(:table, Record.extract(:table, from_lib: "luerl/include/luerl.hrl"))

  def load(state), do: :luerl.load_module(["config"], __MODULE__, state)

  def install(state), do: :luerl_heap.alloc_table(exports(), state)

  defp exports do
    [
      {"split", erl_func(code: &split/2)},
      {"member", erl_func(code: &member/2)}
    ]
  end

  defp split([string, separator | _], state) when is_binary(string) and is_binary(separator) do
    {table_ref, state} = :luerl.encode(String.split(string, separator), state)
    {[table_ref], state}
  end

  defp split(args, state), do: :luerl_lib.badarg_error("split", args, state)

  defp member([entry, table_ref | _], state) when Record.is_record(table_ref, :tref) do
    table(a: array) = :luerl_heap.get_table(table_ref, state)
    found = :array.sparse_foldl(fn _index, value, acc -> acc or value === entry end, false, array)
    {[found], state}
  end

  defp member(args, state), do: :luerl_lib.badarg_error("member", args, state)
end

As you can see, we are implementing the functions we need and making them available to our Lua interface under the config and http packages. To load these functions, we have to run the load function which is exported in both modules. NOTE: install is used as a callback function in load_module

A great benefit of doing things this way is the “compile once, and ready many” approach. The config does not have to be loaded for every request. If we put it all together in our application, it would something  like like the following

Add a build_request function and its helpers to your luerl_lib_http and LuerlLib.Http modules.

-export([load/1, install/1, put_request/2, build_request/1]). % add export

build_request(CowboyReq) ->
    #{method => string:lowercase(cowboy_req:method(CowboyReq)),
      version => version_to_binary(cowboy_req:version(CowboyReq)),
      headers => cowboy_req:headers(CowboyReq)}.

version_to_binary('HTTP/1.0') -> <<"1.0">>;
version_to_binary('HTTP/1.1') -> <<"1.1">>;
version_to_binary('HTTP/2') -> <<"2">>.

  def build_request(cowboy_req) do
    %{
      method: cowboy_req |> :cowboy_req.method() |> String.downcase(),
      version: version_to_binary(:cowboy_req.version(cowboy_req)),
      headers: :cowboy_req.headers(cowboy_req)
    }
  end

  defp version_to_binary(:"HTTP/1.0"), do: "1.0"
  defp version_to_binary(:"HTTP/1.1"), do: "1.1"
  defp version_to_binary(:"HTTP/2"), do: "2"

Add handler modules for handling the request

-module(luerl_lib_handler).
-export([init/2]).

init(CowboyReq, {Form, St}) ->
    LuaReq = luerl_lib_http:build_request(CowboyReq),
    St1 = luerl_lib_http:put_request(LuaReq, St),
    Reply = case luerl:call_chunk(Form, St1) of
        {ok, [], _St2} ->
            cowboy_req:reply(404, #{}, <<"no matching route">>, CowboyReq);
        {ok, Rets, St2} ->
            [Backend] = luerl:decode_list(Rets, St2),
            cowboy_req:reply(200, #{}, Backend, CowboyReq);
        {lua_error, _Reason, _St2} ->
            cowboy_req:reply(500, #{}, <<"routing error">>, CowboyReq)
    end,
    {ok, Reply, {Form, St}}.

defmodule LuerlLib.Handler do
  def init(cowboy_req, {form, state}) do
    lua_req = LuerlLib.Http.build_request(cowboy_req)
    state1 = LuerlLib.Http.put_request(state, lua_req)

    reply =
      case :luerl.call_chunk(form, state1) do
        {:ok, [], _state2} ->
          :cowboy_req.reply(404, %{}, "no matching route", cowboy_req)

        {:ok, returns, state2} ->
          [backend] = :luerl.decode_list(returns, state2)
          :cowboy_req.reply(200, %{}, backend, cowboy_req)

        {:lua_error, _reason, _state2} ->
          :cowboy_req.reply(500, %{}, "routing error", cowboy_req)
      end

    {:ok, reply, {form, state}}
  end
end

With all of that defined, we can now call the relevant functions from our application startup:

-module(luerl_lib_app).
-behaviour(application).
-export([start/2, stop/1]).

start(_StartType, _StartArgs) ->
    {ok, Form, St} = build_lua_state(),
    Dispatch = cowboy_router:compile([{'_', [{'_', luerl_lib_handler, {Form, St}}]}]),
    {ok, _} = cowboy:start_clear(luerl_lib_listener, [{port, 8080}], #{env => #{dispatch => Dispatch}}),
    luerl_lib_sup:start_link().

stop(_State) ->
    ok = cowboy:stop_listener(luerl_lib_listener).

build_lua_state() ->
    St0 = luerl_lib_http:load(luerl_lib_config:load(luerl_sandbox:init())),
    luerl:loadfile("priv/config.lua", St0).

Supervisor

-module(luerl_lib_sup).
-behaviour(supervisor).
-export([start_link/0, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    {ok, {#{strategy => one_for_one, intensity => 1, period => 5}, []}}.

defmodule LuerlLib.Application do
  use Application

  def start(_type, _args) do
    state =
      :luerl_sandbox.init()
      |> LuerlLib.Config.load()
      |> LuerlLib.Http.load()

    {:ok, form, state} = :luerl.loadfile(~c"priv/config.lua", state)
    dispatch = :cowboy_router.compile([{:_, [{:_, LuerlLib.Handler, {form, state}}]}])

    {:ok, _} =
      :cowboy.start_clear(:luerl_lib_listener, [port: 8080], %{env: %{dispatch: dispatch}})

    LuerlLib.Supervisor.start_link()
  end
end

Notice that we load the file once (with the lines below) and pass it down to the handler, it helps us obtain the forms that we need and the handler will always execute based on that initially loaded configuration.

{ok, Form, St1} = luerl:loadfile("priv/config.lua", St0).

{:ok, form, state} = :luerl.loadfile(String.to_charlist(path), state)

Scaling up

Lua scales up because it is built on top of Erlang. This means Lua is using the same processes as Erlang does, it also means we are not using ports to communicate with the Lua interpreter, we have the Lua interpreter running on Erlang.

This makes a great difference in comparison to JavaScript because if you are handling millions of requests and all of them require a JavaScript snippet, this can cause a bottleneck very quickly if you have to limit the number of ports or requests.

On the other hand, Lua is using native Erlang functions when it calls the functions we want to provide for the interpreter. That makes a clear improvement, saving us from performing data serialisation or transformation.

Conclusion

Using a language for flexible configuration gives us the possibility to create an easy interface to provide configurations, reduce the amount of code we need to write, and improve the maintenance without jeopardising the performance of the system. At the moment, you can use Lua as we have explained during the article or you can jump into PHP if you need to process text or templates on top of Erlang or Elixir.

Alternatively you can join the community and provide other solutions which help us build better fit-for-purpose software. Get in touch if you need help building your system with one of these solutions.

The post How to use Lua for flexible configurations in Erlang and Elixir appeared first on Erlang Solutions .

The Ignite Realtime community is pleased to announce the release of Openfire 5.1.2, a maintenance update to our open-source XMPP real-time communication server!

This release primarily updates third-party libraries following reports of vulnerabilities in those dependencies. There’s no indication that Openfire itself was vulnerable to any of the reported issues, but as a precaution and to keep our dependencies up to date, we’ve included the updated libraries in this release.

The full changelog has all the details, with 12 items resolved in total.

You can obtain Openfire 5.1.2 for your platform from its download page . The SHA-256 checksums for the release artifacts are:

bbe0e3bd7837aeb6a78a1878b3e23a53a61e99d827ec966ddc40500db416eb86  openfire-5.1.2-1.noarch.rpm
354e14ebf80c03eee05649a43d52338a77797bb41529a4f1486c49e784119f9d  openfire_5.1.2_all.deb
7da2582b4bc6c25640deecd9f22cdc1278c989cd39c0a8832fc15b3aeb208235  openfire_5_1_2.dmg
b32e33bc0e5305dcac308a82447f970b59a095f1edbb425d81ac93203db2f667  openfire_5_1_2.exe
0f45006e319fb36bf869ef9ca3a3bef930a1bd683a792883b0b0b4c9a3974680  openfire_5_1_2.tar.gz
5e52d57fb9e20235ed31a6809c3ecd139fb6583c0b24575cab3459ad4ff0b0a1  openfire_5_1_2_x64.exe
4f2bcdde684ed157f19df8a792db4f907a62864baa8a6c982c970dfaa43d0421  openfire_5_1_2.zip

We’d love to hear from you! Please join our community forum or group chat and let us know what you think!

For other release announcements and news follow us on Mastodon or X

1 post - 1 participant

Read full topic

Contents:

• Changes in SQL schemas
• Security fixes
• mod_invites: New pages to create invites and WebAdmin
• mod_conversejs: Support ConverseJS 14
• Erlang/OTP 27.0 as a soft minimum
• Rebar/Rebar3: Update binaries to work with Erlang/OTP 26-29
• ChangeLog
• Acknowledgments
• Improvements in ejabberd Business Edition
• ejabberd 26.07 download & feedback

Changes in SQL schema

If you upgrade ejabberd from a previous release to 26.07 , there are no changes in SQL schemas, but there is one for ejabberd Business Edition (see below ).

Security fixes

This release contains fixes for those security issues:

• It&aposs possible to craft PLAIN auth request and authenticate as one user, but then open session for different one.
• mod_caps persistent cache can be poisoned by using legacy version requests.This cache was only used to determine list of nodes that should trigger notifications in PubSub presence-based delivery.
• SQL injection in mod_pubsub handling of paging requests.
• Possible atom exhaustion that can be triggered by issuing REST requests to mod_http_api .
• It was possible to make ejabberd send redirect response for OAuth requests to unvetted url. This required enabling ejabberd to act as OAuth provider (by adding request handler for ejabberd_oauth in http listener). As part of this fix we changed oauth_client_id_check default value to db .
• using ejabberd as OAuth provider will be only allowed by clients
  that were previously registered with oauth_add_client_password or oauth_add_client_implicit commands.
• Tokens generated by mod_bosh , captcha , mod_auth_fast , mod_http_upload and mod_invites used not cryptographically strong random number generators.
• Files server by mod_http_upload didn&apost have XSS prevention headers.
• Issues in authentication of SIP requests.
• Request to web_admin were lacking CSRF protection.
• It was possible to skip captcha verification in mod_register_web.
• mod_conversejs allowed putting unescaped value from url in page content.

mod_invites: New pages to create invites and WebAdmin

mod_invites now includes a startpage where regular users can use their account credentials to generate new account creation invites.

This is useful for people using XMPP clients that do no support that feature. The URL of that page is the root of mod_invites; you can find a link to that page in the bottom of WebAdmin left menu.

There are also new WebAdmin pages to view the existing invites, generate new invites, expire, delete ... Go and take a look at WebAdmin > "Virtual Hosts" > one of your hosts > "Invites"

mod_conversejs: Support ConverseJS 14

ConverseJS published version 14.0.0 recently, and it requires some changes in the web server. In this sense, mod_conversejs is updated to support ConverseJS 14, and also got other minor cosmetic improvements.

Erlang/OTP 27.0 as a soft minimum

Are you compiling ejabberd with Erlang/OTP 25 or 26? Then please try to update to Erlang/OTP 27, 28, or 29. For example, the ejabberd installers are compiled with Erlang/OTP 28.5.0.4.

ejabberd supports compilation with Erlang/OTP 25 and 26, and those versions are still tested in runtime.yml and weekly.yml , but those Erlang/OTP versions are not actively maintained anymore by Erlang/OTP.

Following the erlang security recommendation to Use Actively Maintained Versions of Erlang/OTP , from now ejabberd softly rejects compilation with Erlang/OTP lower than 27.

What does softly mean? If you really want to compile ejabberd with Erlang/OTP lower than 27 at your own risk, you can bypass that soft requirement by defining this option (Erlang/OTP 25.0 included Erlang Run-Time System 13.0 , and that is the number to provide in that option):

./configure --with-min-erlang=13.0

Rebar/Rebar3: Update binaries to work with Erlang/OTP 26-29

ejabberd source code includes Rebar and Rebar3 binaries, in case you don&apost have installed in your system. But those programs only support four Erlang releases (26 up to 29).

If you want to compile ejabberd with Erlang 25, then you need to grab a compatible Rebar3 (or Rebar) binary: either install one from your operating system, or you can download the old binaries included with ejabberd 26.04 (those still supported Erlang 25):

https://github.com/processone/ejabberd/raw/26.04/rebar
https://github.com/processone/ejabberd/raw/26.04/rebar3

ChangeLog

Security fixes

This release contains fixes for those issues:

• It&aposs possible to craft PLAIN auth request and authenticate as one user, but then open session for different one.
• mod_caps persistent cache can be poisoned by using legacy version requests.
  This cache was only used to determine list of nodes that should trigger notifications in PubSub presence-based delivery.
• SQL injection in mod_pubsub handling of paging requests.
• Possible atom exhaustion that can be triggered by issuing REST requests to mod_http_api
• It was possible to make ejabberd send redirect response for OAuth requests to unvetted url. This required enabling ejabberd to act as OAuth provider (by adding request handler for ejabberd_oauth in http listener). As part of this fix we changed oauth_client_id_check default value to db
• using ejabberd as OAuth provider will be only allowed by clients
  that were previously registered with oauth_add_client_password or oauth_add_client_implicit commands.
• Tokens generated by mod_bosh, captcha, mod_auth_fast, mod_http_upload and mod_invites used not cryptographically strong random number generators.
• Files server by mod_http_upload didn&apost have XSS prevention headers.
• Issues in authentication of SIP requests.
• Request to web_admin were lacking CSRF protection.
• It was possible to skip captcha verification in mod_register_web.
• mod_conversejs allowed putting unescaped value from url in page content.

Core

• Fixes delete_old_messages_batch command when used on pgsql
• Adds export_db_ext which allows exporting db content to json files
• Use constant time functions when doing password checks
• Optimize room_unused_* commands when room hibernation is configured
• We no longer add flag requesting client certificate for tls connections
  where certificate authentication is not enabled

Modules

• mod_auth_fast : Fixes exception for session that didn&apost set user agent
• mod_invites : Add page for creating invites.
• mod_invites : Fix generation of CSRF tokens.
• mod_invites : Update to changes in latest XEP-0401
• mod_http_upload : Attach custom headers from config when serving files.

Full Changelog

https://github.com/processone/ejabberd/compare/26.04...26.07

Acknowledgments

We would like to thank for the security reports provided by:

• arthurscchan
• EkiXu
• kah-ja
• LautaroPetaccio
• Marius Schwarz
• tinyb0y
• X1AOxiang

the contributions to the source code by:

• Stefan Strigler for the new features, improvements and fixes in mod_invites
• Christian Dröge
• Holger Weiß
• Jonathan Davies
• Kirill A. Korinsky

and the translation by:

• Mr. EddX for updating the Bulgarian translation
• Arif Budiman for updating the Indonesian translation
• TamilNeram for updating the Tamil translation

And also to all the people contributing in the ejabberd chatroom, issue tracker...

Improvements in ejabberd Business Edition

Customers of the ejabberd Business Edition , in addition to all those bugfixes, also get the following changes:

Changes in SQL schema

This release modifies the push_customizations table in the SQL database schemas to support the new push notifications mute option (see below ). This task is performed automatically by ejabberd by default.

However, if your configuration file has disabled update_sql_schema toplevel option, you must perform the SQL schema update manually yourself. Those instructions are valid for MySQL and PostgreSQL, both default and new schemas:

MySQL

ALTER TABLE push_customizations MODIFY COLUMN mute smallint;

PostgreSQL

ALTER TABLE push_customizations ALTER COLUMN mute TYPE smallint USING mute::int;

Push notifications

• Add a push customization option to allow mentioned users (as described in XEP-0513 ) to be notified even in muted conversation.
• Make proxy connections in applepushv3 always use http connect
• Report push gate tokens in the results of the user_push_state command
• Implement push errors reporting in the results of the user_push_state command.
• Make mod_webpush reject tokens that don&apost have correct format.
• Fix mod_webpush key parser.
• Add ability to retrieve user nodes from push_gate.
• Handle wildcard certificates now used by Google FCM service.

Commands

Add a new local_health_status command to return information about local node state.

Client certificate authentication

• Add option require_xmpp_addr that forces client certs to have XmppAddr .
• New recognize_email_addr option for mod_crl / mod_ocsp to add emailAddress from cert subject to be used beside xmppAddr for matching against provided user id.

p1db backend

• Make mam_p1db serialization properly serialize MUC archives.
• Fix JID encoding in roster_p1db serialization.
• Fix decoding of fast_tokens in p1db backend

Clustering

Make join_cluster robust to race conditions on leave_cluster .

Docker

sqlite backend has been fixed in Docker image.

ejabberd 26.07 download & feedback

As usual, the release is tagged in the Git source code repository on GitHub .

The source package and installers are available in ejabberd Downloads page. To check the *.asc signature files, see How to verify ProcessOne downloads integrity .

For convenience, there are alternative download locations like the ejabberd DEB/RPM Packages Repository and the GitHub Release / Tags .

The ecs container image is available in docker.io/ejabberd/ecs and ghcr.io/processone/ecs . The alternative ejabberd container image is available in ghcr.io/processone/ejabberd .

If you consider that you&aposve found a bug, please search or fill a bug report on GitHub Issues .

History without gaps

The largest piece of work in this release is the kind you only notice when it is absent.

• Interrupted archive syncs now heal. If a sync of the message archive (XEP-0313) was cut short, a conversation could be left with a silent gap: messages that were never fetched, and nothing on screen to say anything was missing. Interruptions are now detected and repaired from both directions.
• History no longer depends on your read position. Opening a conversation you had already read elsewhere could show an empty history on a new or freshly cleared device. The archive now downloads independently of how far you had read.
• A conversation read on another device opens at its live edge , instead of restoring a saved scroll position that multi-device sync had already made obsolete.
• The new-messages divider and the unread badge on the scroll-to-bottom button follow your read position, consistent with the sidebar and with read-marker sync (XEP-0490).

These are the paths we found and could reproduce. Archive sync has a long tail, and how it behaves depends a good deal on which server you are talking to. If you still see a conversation with history missing, we would love to hear about it.

Faster file transfers

• Desktop uploads and downloads are handled by native code. Transfers of large files are much faster and more reliable.
• Encrypted attachments decrypt on download. An end-to-end-encrypted attachment used to be decrypted only for the inline preview, so saving it to disk gave you something unreadable. Files of every type now arrive readable.

Emoji autocomplete & polish

• Type a colon and a keyword to complete emoji inline. Arrow keys move through the matches, Enter or Tab inserts.
• The send button answers with a press and a glow pulse when a message goes out.

On the web

• An unread badge on the app icon for the installed PWA.
• Media is cached by the service worker , so images are not re-downloaded on every visit.
• Repeated messages from one sender coalesce into a single "N new messages" notification instead of a stack.
• An update that was downloaded but parked is now applied at launch, instead of the installed app trailing the deployed build indefinitely.

Reliability and platform fixes

Twenty-three fixes in this release. The ones most likely to have affected you:

• Notifications behave. Clicking a reaction notification always jumps to the reacted message, reopening a conversation no longer re-posts a notification for something you have already seen, and an encrypted message with no readable content stays silent until it is decrypted instead of posting a blank notification and playing a sound.
• The sidebar holds still. A delayed message, whether an offline replay or a catch-up copy older than what you already have, no longer drags the conversation preview back to older text. The sidebar also keeps its scroll position while conversations reorder during catch-up.
• Authentication. A wrong saved password no longer triggers an endless keychain retry loop, and connecting to a server whose domain contains non-ASCII characters now works.
• Key backup. The backup passphrase is used exactly as it is displayed, so backups restore in other XMPP clients. Older backups still open, and are healed to the portable format on restore.
• Group chats. The public room directory no longer lists duplicate rooms or reaches past your server&aposs own directory, and a room notification banner no longer reappears when the room is reopened.

Smaller ones round out the release: the quoted-message and reply cards stay visually distinct when a message is selected, your own message group re-fits its width once an image inside it finishes loading, the typing indicator is centered between the last message and the composer, and the macOS traffic-light buttons stay centered in the app bar.

Get it

Fluux Messenger 0.17.2 is available for macOS, Windows, and Linux, or directly in your browser.

If you upgrade and something feels off, tell us. Bug reports and feature requests both go to GitHub Issues .

So I was bored and I set up an XMPP server under 👆️.op-co.de , in addition to the one I already had under ツ.op-co.de . It worked with some clients and failed with others. But both ツ and 👆️ are valid Unicode 1.1 characters, so WTF? Buckle up (or better: put on your 🤿) for the 19-RFC deep dive...

...or skip right to the TL;DR .

Note: despite my generous use of Emojis throughout this post, none of it was generated by a text extruder machine. All words are the product of artisanal typing on my keyboard, and the Emojis were hand-selected from an Emoji-picker widget.

XMPP addresses

XMPP, the eXtensible Messaging and Presence Protocol , formerly known as Jabber®, defines its address format in RFC 7622 .

An XMPP address (formerly known as Jabber® ID, or JID, as used in the RFC) has three parts:

   jid = [ localpart "@" ] domainpart [ "/" resourcepart ]

The localpart is usually the username, but is not used when addressing a server.

The domainpart is the hostname or domain name, and can be a Unicode DNS identifier, an IPv6 address in square brackets, or a legacy IP address. This is the only mandatory part of a JID.

The resourcepart is the internal identifier of an individual client, allowing a user to have multiple clients connected at the same time; it is also used for the nickname in XEP-0045: Multi-User Chat .

Each of these three parts must be valid UTF-8 and can be up to 1023 bytes (not characters!) in length.

Furthermore, there are restrictions for each part, for example:

localpart    = 1*1023(userbyte)

a "userbyte" is a byte used to represent a UTF-8 encoded Unicode code point that can be contained in a string that conforms to the UsernameCaseMapped profile of the PRECIS IdentifierClass defined in RFC 7613 [...]

Come again, please? Okay, let's take this apart, slowly.

a "userbyte" is a byte used to represent a UTF-8 encoded Unicode code point

This is a convoluted way to say that we accept up to 1023 bytes (not characters!) of valid UTF-8.

that can be contained in a string that conforms to the UsernameCaseMapped profile of the PRECIS IdentifierClass defined in RFC 7613

In addition to being valid UTF-8, it must also conform to the IdentifierClass in PRECIS ( RFC 7613 ).

PRECIS: Preparation, Enforcement, and Comparison of Internationalized Strings

PRECIS is the successor to Stringprep ( RFC 3454 , which we can ignore for now).

However, the PRECIS definition in RFC 7613 is obsoleted by RFC 8265 , which we can't ignore and will have to take our character profiles and classes from.

So we need the IdentifierClass for the localpart , and furthermore the FreeformClass for the resourcepart .

PRECIS classes, profiles and categories

The earlier Stringprep approach explicitly defined its classes as valid ranges of Unicode code points (characters). However, given that Unicode is a living (versioned) standard, new characters (and new Emojis! 💡) get added every year. This left Stringprep in an uncomfortable place, forever hard-coded to the long-superseded 2002 Unicode 3.2 standard.

To allow for future compatibility, PRECIS took a different path. It describes an algorithm that can be applied to an individual Unicode character in order to determine whether it belongs to a certain PRECIS class.

The classes ( IdentifierClass and FreeformClass ) are defined in RFC 8264 , and the profiles ( UsernameCasePreserved , UsernameCaseMapped , OpaqueString , Stringprep ) are defined in RFC 8265 .

Furthermore, PRECIS attempts to retain backward compatibility with earlier standards like IDNA2008, as well as with itself. If a certain character is "valid" under an earlier version of Unicode, PRECIS tries to ensure that it stays "valid" under later versions. The only explicit exception from this is that code points that were "undefined" in earlier Unicode versions can later be assigned and move to "valid" or "disallowed".

Each of these rules is applied to individual characters, or to character categories, as defined in RFC 5892: IDNA code points .

Given this toolset, we can now get back to the individual XMPP address parts.

XMPP address elements

localpart - the user name

As stated in RFC 7622 above, the localpart must be...

a string that conforms to the UsernameCaseMapped profile of the PRECIS IdentifierClass

So we have the profile ( UsernameCaseMapped ) and the class ( IdentifierClass ) to look up.

The UsernameCaseMapped transformation

The UsernameCaseMapped profile in RFC 8265 performs some normalization steps: it requires decomposition of certain East Asian characters, lowercasing, Unicode Normalization Form C, and application of the Bidi rule .

Later, RFC 8265 § 3.3.2 says:

Ensure that the string consists only of Unicode code points that are explicitly allowed by the PRECIS IdentifierClass defined in Section 4.2 of [RFC8264] .

What's allowed by IdentifierClass ?

RFC 8264 §4.2.1 defines the valid and disallowed character properties, as well as certain groups that require special treatment.

Valid identifiers contain "Code points traditionally used as letters and numbers in writing systems", the ASCII 7-bit characters U+0021 through U+007E, and a few characters that are only allowed in a certain context, like U+00B7 MIDDLE DOT which is only allowed inside the Catalan ela geminada "ŀl" .

The ツ character (U+30C4 KATAKANA LETTER TU) belongs to the "Letter, other" (Lo) category of Unicode ) and thus is a valid letter character allowed in IdentifierClass . The 👆️ emoji (U+261D WHITE UP POINTING INDEX) belongs to the "Symbol, other" (So) category with all the other Emojis. The whole "Symbol" category is disallowed inside of IdentifierClass . Bummer. Sad trombone! 🎶🪊

On the other hand, the "Nonspacing Mark" (Mn) category is allowed, and so ḩ̸̡͇͉̬̓͝e̷͙̪̯̬̬͍͒̂̓̽̀̄ ̵̨̨̪̯̞̠͒̐͘͝c̷͍͆o̸̡̢̥͌̒̌̀͜͜m̶̬̙̙̓̌͘͠ë̷́̉́͜t̴̍̔͜͠h̷̖̭̫̥̖̥͐͂͊͒̓.

Putting localpart together

So essentially, PRECIS only allows the boring regular lowercase letters from any supported language, and none of the fun Emojis.

To add insult to injury, RFC 7622 §3.3.1 imposes further restrictions by disallowing some more fun characters:

" U+0022 (QUOTATION MARK)
& U+0026 (AMPERSAND)
' U+0027 (APOSTROPHE)
/ U+002F (SOLIDUS)
: U+003A (COLON)
< U+003C (LESS-THAN SIGN)
> U+003E (GREATER-THAN SIGN)
@ U+0040 (COMMERCIAL AT)

However, there are still a bunch of "funny" permitted characters left from the ASCII7 block:

!#$%()*+;=?[\]^`{|}

This leaves us with some valid old-school ASCII smiley user name options on the table:

;=)
B*}

And a bunch of Unicode letters that can be abused, with special thanks to Egyptian hieroglyphs :

The domain part

Back to RFC 7622 §3.1 :

domainpart   = IP-literal / IPv4address / ifqdn

the "IPv4address" and "IP-literal" rules are defined in RFCs 3986 and 6874 , respectively, and the first-match-wins (a.k.a. "greedy") algorithm described in Appendix B of RFC 3986 applies to the matching process

We will leave IP literals out... for now. Just a note that the format for IPv6 literals need to be enclosed in brackets and may contain a %zone postfix.

ifqdn        = 1*1023(domainbyte)

a "domainbyte" is a byte used to represent a UTF-8 encoded Unicode code point that can be contained in a string that conforms to RFC 5890

RFC 5890: IDNA Definitions and Document Framework is a new addition to our list. It is the "Definitions" part of the IDNA2008 ("Internationalized Domain Names for Applications", released in 2008) specification. The RFC 5892 we encountered earlier belongs to the same specification suite.

However, there is not a single "string" that "conforms" to RFC 5890. RFC 7622 §3.2.1 has a more precise requirement:

the string consists only of Unicode code points that are allowed in NR-LDH labels or U-labels as defined in RFC5890 .

An NR-LDH (non-reserved letter, digit, hyphen) label is an ASCII label (not containing "special" Unicode characters) according to the "hostname" syntax defined in RFC 952 back in 1982.

IDNA-valid U-labels

The U-label definition can be found in RFC 5890 §2.3.2.1 :

[A U-label] is also subject to the constraints about permitted characters that are specified in Section 4.2 of the Protocol document and the rules in the Sections 2 and 3 of the Tables document [...].

Rant about RFC rendering

The links in the quoted paragraph are pointing to the wrong RFC, so I disarmed them in the quote above.

The normative RFC format before RFC 8650 (late 2019) was fixed-width ASCII, 58 lines, 72 characters with manual page breaks, designed to be printed by a 1982 line printer on US Legal (even though the PDF renderings are using US Letter). The markup in the HTML versions linked from this post is auto-generated from a semantic analysis of the normative ASCII documents.

The fixed-width fixed-page format is unreadable on mobile devices, and effectively trips up reflow algorithms. There used to be an alternative ebook rendering of RFCs that was the only useful way for people with bad eyes to read RFCs. It stopped rendering new documents in 2019 and was abandoned in 2022. Nobody cared.

The links above point to sections of RFC 5890, because the string parser saw "Section x.y.z" and assumed it to be a reference to section x.y.z of the current RFC. It was not. The links should go to RFC 5891 §4.2 , RFC 5892 §2 and §3 .

U-label definition

Let's get back to the U-label definition from RFC 5890 §2.3.2.1 . It is a variant of the "IDNA-valid string":

For IDNA-aware applications, the three types of valid labels are "A-labels", "U-labels", and "NR-LDH labels" [...]

A string is "IDNA-valid" if it meets all of the requirements of these specifications for an IDNA label. [...]

[A U-label] is also subject to the constraints about permitted characters that are specified in Section 4.2 of the Protocol document and the rules in the Sections 2 and 3 of the Tables document [...].

So. Uhm. A U-label needs to be IDNA-valid, and an IDNA-valid string is either a U-label, an A-label or an NR-LDH label. This is not a recursive definition!

The referenced RFC 5891 §4.2 Permitted Character and Label Validation further clarifies:

The candidate Unicode string MUST NOT contain characters that appear in the "DISALLOWED" and "UNASSIGNED" lists specified in the Tables document.

Furthermore, it may not begin or end with a "-", and must not contain a "--" at the third position, in order to not be mixed up with A-labels.

An U-label can be up to 252 bytes ( not characters! ) long ( §4.2 ), but its ASCII-compatible encoding (ACE / A-label) form must not exceed 63 ASCII characters (equal to bytes!). In addition, DNS limits the full hostname to 255 characters.

Valid U-label characters

The set of valid characters is defined by RFC 5892 §2 . A minor detail that we omitted above, when talking about valid localpart characters, was that RFC 8264 in fact does not define the character categories, but instead contains references to the respective subsections of RFC 5892 §2 .

Despite of that, the valid character sets for localpart and domainpart are not equal. ß U+00DF LATIN SMALL LETTER SHARP S is explicitly included for U-labels (I haven't figured out why it would be disallowed though), as is 〇 U+3007 IDEOGRAPHIC NUMBER ZERO (Nl) (which is in the disallowed "Letter Number" (Nl) category) and there is a number of other exceptions .

Korean is restricted to modern Hangul syllable characters .

IDNA is using case folding to normalize the letter case. This matches the lowercase conversion of RFC 8265, except when it doesn't .

Furthermore, DNS Registries are allowed to restrict the valid characters for domain names, probably in order to limit homoglyph attacks .

Putting domainpart together

There is a significant overlap between localpart and domainpart . However, - U+002D HYPHEN-MINUS is the only special character from the ASCII set that's still allowed, and it may not appear in all positions.

Lowercase letters (or uppercase Cherokee) and numbers are allowed, ASCII smileys are not. Egyptian hieroglyphs and diacritics are still in the game for subdomains, or if your Registry allows them on the domain name.

To prove a point, this post is reachable via ḧ̴͖́e̷͚̿-̸̧͘c̴͖͌o̴̻̊m̷͕̂e̷͔͊t̷͚̊h̵̦̄.op-co.de and there is an XMPP server, too:

The resource part (a.k.a. chatroom nickname)

RFC 7622 §3.4 is where the resourcepart gets defined:

The resourcepart of a JID is an instance of the OpaqueString profile of the PRECIS FreeformClass , which is specified in RFC7613 .

This is actually the same mechanism as with localpart , just with a different profile and a different class.

Characters in FreeformClass

RFC 8264 §4.3.1 defines the valid FreeformClass code points. This includes all traditional letters and numbers, printable ASCII (U+0021 through U+007E), punctuation, spaces, and 🚨 symbols‼️ 🤯 Finally!

On top, OpaqueString will apply some normalization , including the conversion of all non-ASCII whitespace into U+0020. Character case will be retained.

Stripping nicknames

RFC 7622 §3.4.1 also has a note regarding the use of resourcepart for nicknames:

In some contexts, it might be appropriate to apply more restrictive rules to the preparation, enforcement, and comparison of XMPP resourceparts. For example, in XMPP Multi-User Chat [XEP-0045] it might be appropriate to apply the rules specified in [PRECIS-Nickname] .

"it might be appropriate" is not normative language, right? The Nickname profile is derived from FreeformClass and is a mapping that removes leading and trailing whitespace, and reduces consecutive whitespace into one U+0020. And it applies the lowercase transformation for nickname comparisons, to disallow multiple users to have the same case-normalized nickname. That's it.

So you can have all the Emojis as your nickname, right? RIGHT?

Hysterical raisins

Jabber was born in 1999 . The first formal XMPP specification was RFC 3920 in 2004. Over the decades, both the XMPP specification and the Unicode standard evolved, thus also changing what is considered a valid XMPP address. Implementations that we need to interoperate with might be running on some older version of the specification, and accept a different subset of "valid" Unicode characters.

Let's sort this out as well!

2004: The Original Specification

RFC 3920 §3 Addressing Scheme defines the JID syntax:

• A "domain identifier" (later renamed to domainpart ) is an IDNA string according to RFC 3490 (IDNA2003) and must match the Nameprep profile defined in RFC 3491 .
• A "node identifier" ( localpart ) must match the Nodeprep profile defined in Appendix A .
• A "resource identifier" ( resourcepart ) must match the Resourceprep profile from Appendix B .

Nameprep , Nodeprep and Resourceprep are profiles of Stringprep (from RFC 3454 , which contains tables with allowed and prohibited characters, as well as character mappings to perform, based on Unicode 3.2).

Each of the profiles defines the set of tables and steps to apply. For example, the Nameprep processing consists of three steps:

1. Mapping:
   • remove ("map to nothing") 27 different hyphenation characters
   • apply case mapping and folding to 1676 characters ("A" ➡️"a", "𝛬" ➡️"λ", ...)
2. Prohibited Output (based on tables in RFC 3454 Appendix C ):
   • disallow ASCII and non-ASCII space characters (but not control characters - those are disallowed by XML 1.0 , which is the mandatory foundation of XMPP )
   • disallow Private Use and "non-character" ranges from Unicode, as well as surrogate codes
   • disallow some inappropriate characters (like � U+FFFD REPLACEMENT CHARACTER ) and orientation markers
3. Only allow unassigned code points according to IDNA rules (allowed in queries, not in "stored strings")

The handling of IPv6 literals in RFC3920 assumes that they are inserted verbatim, with no surrounding [] and no %zone identifier.

Nodeprep is similar to Nameprep , but disallows control characters, as well as the forbidden characters we know from localpart , namely "&'/:<>@

Resourceprep is also similar to Nameprep but allows ASCII whitespace and doesn't perform case folding, allowing for uppercase characters.

But on the good side, neither IDNA2003 nor the Stringprep profiles disallow the use of Emojis (that are part of Unicode 3.2) in domain names! 🎉

2008: A New IDNA Hope

However, the experience of operating IDNA2003 in the wild for a few years led to the documentation of 37 pages (measured in 72-character ASCII on US Legal) of issues and shortcomings, documented in RFC 4690 and including this section:

5.1.1. Elimination of All Non-Language Characters

Unicode characters that are not needed to write words or numbers in any of the world's languages should be eliminated from the list of characters that are appropriate in DNS labels. In addition to such characters as those used for box-drawing and sentence punctuation, this should exclude punctuation for word structure and other delimiters. While DNS labels may conveniently be used to express words in many circumstances, the goal is not to express words (or sentences or phrases), but to permit the creation of unambiguous labels with good mnemonic value.

I guess that Emojis lack good mnemonic value. RIP. 🪦

The result of this analysis was the replacement of IDNA2003 with IDNA2008 in... you guessed it... 2010! To be fair, the IDNA2008 suite was "largely completed in 2008", and got submitted to the IETF in October 2008.

The IDNA2008 RFC collection obsoleted the previous RFCs, and thus the stricter domainpart requirements (no Emojis) were automatically turned into law in 2010, without having to change any of the XMPP specifications.

But we can still have Emojis in usernames and nicknames, right? 🥹

2010 Revenge of the PRECIS

The update to IDNA made Stringprep obsolete, and prompted the creation of the Preparation and Comparison of Internationalized Strings Working Group at the IETF.

While the WG was working on the PRECIS specifications, the XMPP core specifications got a major overhaul in 2011. As part of that, the address format was updated and separated into its own document, RFC 6122 :

Because all other aspects of revised documentation for XMPP have been incorporated into [XMPP] , the XMPP Working Group decided to temporarily split the XMPP address format into a separate document so as not to significantly delay publication of improved documentation for XMPP. It is expected that this document will be obsoleted as soon as work on a new approach to preparation and comparison of internationalized addresses has been completed.

The updated address format still relied on IDNA2003, but developers were encouraged to look at IDNA2008.

RFC 6122 furthermore introduced the localpart , domainpart and resourcepart names and changed IPv6 literals to use the bracketed IP-literal syntax from RFC 3986 .

2015 The Next Generation

As announced in the intro of RFC 6122 , it was soon replaced by RFC 7622 , which we might vaguely remember from the beginning of this post. It was published in 2015, based on the still fresh RFC 7613 PRECIS specification.

The PRECIS suite and the updated XMPP address format introduced case folding, replaced the Stringprep profiles with the PRECIS classes, profiles and categories explained above, and effectively disallowed Emojis in the localpart and domainpart of XMPP addresses (following the IDNA2008 insights).

As mentioned before, RFC 7613 was obsoleted by RFC 8265 , which corrected a few things and went from case folding to lowercase again . 🤷

This happened in 2017 and, together with RFC 8266 (Nicknames) is the end of the evolution of the RFCs needed to understand XMPP addresses.

So you just told me that Emojis in nicknames are still allowed, yes?

XMPP address validation in the wild

The IETF is about "rough consensus and running code". We've seen the consensus and how it changed over two decades, but in the end it's the running code that will say "no" when you try to butt dial an XMPP address.

Consensus in distributed systems

Something that you enter might go through up to five different hops (and different XMPP implementations; I'm omitting protocol bridges, but the point should be clear):

1. Your own client, which is responsible for sanitizing (or refusing) your input, through its user interface or config file.
2. Your server, receiving your input through a client-to-server connection.
3. Optionally, a XEP-0045 Multi-User Chat (MUC) room where you are an occupant, through a server-to-server connection.
4. The recipient's server, through a server-to-server connection shared with other users.
5. The recipient's client, through its own connection to its server.

Your client is the easiest part, as it can simply reject forwarding something it disagrees with. If the "Add contact" button is greyed out, you've arrived at a dead-end. ⛔

The following hops on the path can't grey out the button if they consider your XMPP address, coming through an XML stream, as invalid. According to RFC 6120, they have to treat it as a (recoverable) stanza-related error, and reject the respective XML stanza (and not terminate the XML stream):

8.3.3.8. jid-malformed The sending entity has provided (e.g., during resource binding) or communicated (e.g., in the 'to' address of a stanza) an XMPP address or aspect thereof that violates the rules defined in [XMPP‑ADDR] ; the associated error type SHOULD be "modify".

So if a recipient disagrees about the PRECIS / IDNA version with your client or your server, it will reject the respective stanza before it can be processed.

Robot Face vs. the MUC Occupants

When joining a MUC, you send a presence stanza to your occupant address, constructed by appending your nickname as the resourcepart to the room address. If you choose an evil Emoji nickname and the room rejects it, it will send an error response, and you won't be able to join the room.

Now if the room does accept your nickname, it will forward the presence, sending it from your occupant address, to all other occupants.

I first ran into this issue, not knowing much about IDNA, PRECIS or Stringprep, back in 2017 :

11:28:50 ---> 🤖 joined the room
11:28:50 <--- T....s has left the room (Kicked: jid malformed: The source address
                                    is invalid: prosody@conference.prosody.im/🤖)
11:28:51 <--- N..........s has left the room (Kicked: jid malformed)
11:28:51 <--- d......n has left the room (Kicked: jid malformed: The source address
                                      is invalid: prosody@conference.prosody.im/🤖)
11:28:51 <--- d.......o has left the room (Kicked: jid malformed: The source address
                                       is invalid: prosody@conference.prosody.im/🤖)
11:28:51 <--- a..v has left the room (Disconnected: not-well-formed)
11:29:08 ---> a..v joined the room
11:32:18 ---> T....s joined the room
11:32:18 <--- T....s has left the room (Kicked: jid malformed: The source address
                                    is invalid: prosody@conference.prosody.im/🤖)

Any downstream server or client that does not accept this occupant presence will send a stanza error back to the MUC. The MUC will treat that error as a non-recoverable session error and remove the respective occupants.

As long as you stay in the room, the other clients will repeatedly reconnect, receive your presence, and get kicked out. If you send a message to the room, it will get pushed to joining clients as part of the room history even after you leave.

Today, the situation is only slightly different:

00:14:39 ---> 🤖 joined the room
00:14:39 <--- H....r (....) has left the room due to an error
            (Kicked: bad request)
00:14:39 <--- c...........s (Monocles) has left the room due to an error
            (Kicked: bad request)
00:14:39 <--- p......d (Conversations) has left the room due to an error
            (Kicked: bad request)
00:14:39 <--- E..a (Cheogram) has left the room due to an error
            (Kicked: bad request)
00:14:39 <--- m.....x (Conversations) has left the room due to an error
            (Kicked: bad request)
00:14:39 <--- y..h (Conversations) has left the room due to an error
            (Kicked: bad request)
00:14:39 <--- b.....a (.../Conversations....) has left the room due to an error
            (Kicked: bad request)

Most of the affected users seem to be running Conversations or its forks Cheogram and Monocles, and the clients(?) responded to the presence with a "bad-request" error.

In addition to that issue, ejabberd sends the error response over the wrong half of the server-to-server stream, so:

Jul 12 01:19:36 s2sout5e57d6de9d50      debug   Received[s2sout]: <presence to='test@chat.yax.im/🤖' type='error' id='noLNT-110871' from='georg@conversations.im/Conversations.jxm9v159lx' xml:lang='de-DE'> 
Jul 12 01:19:36 stanzarouter    warn    Received a stanza claiming to be from conversations.im, over a stream authed for chat.yax.im!
Jul 12 01:19:36 s2sout5e57d6de9d50      debug   Disconnecting chat.yax.im->conversations.im[s2sout], <stream:error> is: <stream:error><not-authorized xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error> 

Looking at implementations

So it seems like there is a bit of inertia with implementations to follow a fifteen years old specification update. This warrants a look at the major implementations.

Scroll down for the summary table.

Server implementations

According to the s.j.n stats , the top 5 server implementations on the federated XMPP network are Prosody (60%), ejabberd (24%), Spectrum (6%), biboumi (4%) and "Multi User Chat" (1%). And while biboumi is the only one without its own .IM domain, we can still exclude it and Spectrum from the list, as they are bridges to other networks and need to adhere to the limitations of those networks. "Multi User Chat" is in fact the MUC component of the Tigase server .

A cross-match with the servers connected to yax.im also yields similar results, and adds Openfire as a candidate with 1.5% market share.

prosody

Lua isn't exactly friends with Unicode , so prosody went for a manual approach and implemented Stringprep in encodings.c using either ICU or libidn , based on a compile-time switch. The binary packages built by the prosody team use ICU, so we'll take that for the comparison.

libICU

ICU (International Components for Unicode) has a very turbulent history - initiated at a spin-off from Apple and IBM, and written in Java, the first version got integrated into the Java SDK in 1997, then developed in parallel and ported from Java to C++ and C. prosody is using the C version. ICU supports IDNA2008, which prosody started using in 2019. However, ICU only supports Stringprep, not PRECIS (probably due to the fact that Stringprep was a required part of IDNA2003).

libidn and libidn2

libidn on the other hand started out as libstringprep , and supports IDNA2003 and Stringprep. libidn2 was created to support IDNA2008, but it removed Stringprep support , so can't be used as a drop-in replacement.

You have the choice between IDNA2003 with Stringprep and IDNA2008 without PRECIS.

ejabberd

ejabberd is written in Erlang , a language that's as powerful as it is obscure. The ejabberd developers have implemented their own stringprep library and use erlang-idna which supports both IDNA2003 and IDNA2008, but haven't tackled PRECIS.

Tigase

Tigase is written in Java and seems to have forked and heavily reformatted the December 2004 libidn 0.5.12 release. There is no mention of IDNA2008, nor of PRECIS in the source, so I would assume IDNA2003 and Stringprep.

Openfire

Openfire uses Tinder for the XMPP stanzas, and that makes use of libidn 1.35 . As this is not libidn2, Openfire is at IDNA2003 and Stringprep. But there is an abandoned half-finished PR to implement PRECIS !

Client implementations

According to the JabberFR client stats , the top 5 client implementations are:

1. Conversations
2. Cheogram (a Conversations fork)
3. Monocles (a Conversations fork)
4. Gajim
5. Monal
6. Pidgin
7. Blabber.im (an abandoned Conversations fork)
8. Dino

Conversations

Conversations is a modern Android client that's making use of jxmpp-stringprep-libidn , which is using libidn 1.15 which gives us IDNA2003 and Stringprep.

Gajim

Gajim was in fact the client that told me that I'm holding it wrong and that made me write this blog post.

Gajim is using nbxmpp and nbxmpp is using precis-i18n , which implements the trifecta of 8264 , 8265 , and 8266 ! In addition, idna is used for full IDNA2008 support.

So far, Gajim is the only client that will allow Unicode >3.2 emoji in nicknames (and nowhere else)!

Monal

The authentication code is doing manual stringprep , but other than that there is no support for Stringprep or PRECIS. IDNA2008 is handled by the underlying iOS core library.

Pidgin

Pidgin. My nemesis. The formerly most-widely used XMPP client that made a generation of users believe that XMPP is stuck in 2004. Pidgin is using libpurple , which was famously called "a flock a zero days flying in formation" a decade ago.

A 2009 patch implemented IDNA2003 and Stringprep support based on libidn, and it seems to have survived in the 2.14 "stable" branch, which was last released in January 2025.

The 3.0 development branch does not contain any traces of IDNA, Stringprep or PRECIS.

Dino

Dino , a modern client written in Vala, uses a binding to libICU, but without the UIDNA_USE_STD3_RULES flag that would enable IDNA2008.

Implementation overview

The analysis of the client and server implementations shows that most implementations lag behind by a decade. There are two notable exceptions: Gajim implements the current state-of-the-art, and Monal allows everything and lets the server sort things out.

❌ = not allowed | 👆️ = legacy Unicode 3.2 | 🤖 = modern Unicode

Summary / TL;DR

The original XMPP specification (2004-2010; IDNA2003 + Stringprep) didn't forbid Emojis in any parts of an XMPP address, but was limited to Unicode 3.2, which only had around 150 Emojis. xmpp:👆️@♻️.❤️/⁉️

When IDNA2003 was replaced by IDNA2008 in 2010, hostnames were restricted to characters from actual human languages. The two most widely deployed server implementations enforce this limit, but might support pre-existing legacy hostnames. xmpp:☹️@𓀐.𓂸/☢️

When the XMPP specification implemented PRECIS in 2017, usernames were also limited to human languages, but the resource / nickname part was left permissive, and opened up to all existing and future Unicode specifications. xmpp:𓀬@ツ.ۃ/🤖

So after going through 22 years of development, 19 RFCs and 17 Unicode standards, I have to say: the internet was right and I was wrong. 👆️.op-co.de is not a valid JID, but it was unti 2010.

Read state that actually syncs

0.17.0 introduced synced read markers (XEP-0490), but we shipped a bug: the payload we published used the wrong shape, so other XMPP clients ignored our read state, and we ignored theirs. 0.17.1 sets this right:

• Spec-accurate XEP-0490 payloads. Fluux now publishes the shape other clients expect, and migrates the legacy markers 0.17.0 wrote.
• Notifications follow your reading. When you read a conversation on one device, its native notification banner is dismissed on your other devices. Read markers synced while a room was inactive are now applied too.
• Your place is kept. A room no longer discards your read position on launch. Fluux anchors on the last-read message, the way other modern chat apps do, and the marker advances correctly once you reach the live edge of a conversation.

A Pure theme for OLED and e-ink

New in this release: the Pure theme , in pure-black and pure-white variants. Flat, high-contrast chrome with no gradients or translucency, designed for OLED displays and e-ink screens.

There is also a new "Play notification sounds" toggle in Accessibility settings.

Group chats

• Slash commands in the composer , including /nick to change your nickname. The change is reflected in the occupant list and as a timeline notice.
• Typing indicator in the sidebar for joined rooms. It only appears when a caught-up room lights up; busy or unread rooms keep their badge.
• Impersonation hardening. Nicknames padded with whitespace or invisible characters can no longer masquerade as another occupant.

Aurora refinements

• Shields and locks now mean different things. A shield shows encryption status; a lock is reserved for content that cannot be read. The two metaphors are applied consistently across the chat header, message indicators, composer, and security panel.
• Cleaner outgoing bubbles. Consecutive messages you send hug their content and form clean rectangular groups.
• Calmer motion. Space is reserved for the typing indicator so it no longer overlays the last message or fights an upward scroll, and Aurora gradients now harmonize with your accent color in each theme.

Reliability and platform fixes

• Jumps always land. Jumping to a reacted, replied-to, or poll message works even when the target is outside the loaded history window, and search previews stay centered on the match.
• Linux. The system tray dependency is standardized on libayatana-appindicator so tray menu labels render, Flatpak installs auto-pull the GNOME runtime, and modals stay solid where WebKitGTK advertises backdrop blur it does not actually paint.
• Link previews are now attached in the interoperable OGP format, so other XMPP clients display them correctly.

The full list of changes is in the changelog on GitHub .

Get it

Fluux Messenger 0.17.1 is available for macOS, Windows, and Linux, or directly in your browser. As always, it works with any standards-compliant XMPP server, and it remains our day-to-day client at ProcessOne.

If you upgrade and something feels off, tell us. A lot of what shipped in this release started as a user report — bugs and ideas are welcome on GitHub Issues .

Here is a new version for slixmpp, the python XMPP library.

This release has one major deprecation, two bug fixes, several new features as well as plenty of improvements under the hood.

Thanks to everyone involved!