• chevron_right

      Erlang Solutions: Instant Scalability with MongooseIM and CETS

      news.movim.eu / PlanetJabber · 2 days ago - 10:22 · 6 minutes

    The main feature of the recently released MongooseIM 6.2.1 is the improved CETS in-memory storage backend which makes it much easier to scale up.

    It is difficult to predict how much traffic your XMPP server will need to handle. Are you going to have thousands or millions of connected users? Will you need to deliver hundreds of millions of messages per minute? Answering such questions is almost impossible if you are just starting up. This is why MongooseIM offers several means of scalability.

    Clustering

    Even one machine running MongooseIM can handle millions of connected users, provided that it is powerful enough. However, one machine is not recommended for fault tolerance reasons, because every time it needs to be shut down for maintenance, upgrade or because of any issues, your service would experience downtime. This is why we recommend using a cluster of connected MongooseIM nodes, which communicate efficiently over the Erlang Distribution protocol . Having at least three nodes in the cluster allows you to perform a rolling upgrade , where each node is stopped, upgraded, and then restarted before moving to the next node, maintaining fault tolerance and eliminating unnecessary downtime. During such an upgrade procedure, you can increase the hardware capabilities of each node, scaling the system vertically. Horizontal scaling is even easier because you only need to add new nodes to the already deployed cluster.

    Mnesia

    Mnesia is a built-in Erlang Database that allows sharing both persistent and in-memory data between clustered nodes. It is a great option at the start of your journey because it resides on the local disk of each cluster node and does not need to be started as a separate component. However, when you are heading towards a real application, a few issues with Mnesia become apparent:

    1. Consistency issues, tend to show up quite frequently when scaling or upgrading the cluster. Resolving them requires Erlang knowledge, which should not be the case for a general-purpose XMPP server.
    2. A persistent volume for schema and data is required on each node. Such volumes can be difficult and costly to maintain, seriously impacting the possibilities for automatic scaling.
    3. Unlike relational databases, Mnesia is not designed for storing huge amounts of data, which can lead to performance issues.

    After trying to mitigate such issues for a couple of years, we have concluded that it is best not to use Mnesia at all. First and foremost, it is highly recommended not to store any persistent data in Mnesia, and MongooseIM can be configured to store such data in a relational database instead. However, up to version 6.1.0, MongooseIM would still need Mnesia to store in-memory data. For example, a shared table of user sessions is necessary for message routing between users connected to different cluster nodes. The problem is that even without persistent tables, Mnesia still needs to keep its schema on disk, and the first two issues listed above would not be eliminated.

    Introducing CETS

    Introduced in version 6.2.0 and further refined in version 6.2.1, CETS (Cluster ETS) is a lightweight replication layer for in-memory ETS tables that requires no persistent data. Instead, it relies on a discovery mechanism to connect and synchronise with other cluster nodes. When starting up, each node registers itself in a relational database, which you should use anyway to store all your persistent data. Getting rid of Mnesia removes the last obstacle on your way to easy and simple management of MongooseIM. For example, if you are using Kubernetes , MongooseIM no longer requires any persistent volume claims (PVC’s), which could be costly, can get out of sync, and require additional management. Furthermore, with CETS you can easily set up automatic scaling of your installation.

    Installing with Helm

    As an example, let’s quickly set up a cluster of three MongooseIM nodes. You will need to have Helm and Kubernetes installed. The examples were tested with Docker Desktop , but they should work with any Kubernetes setup. As the first step, let’s install and initialise a PostgreSQL database with Helm:

    $ curl -O https://raw.githubusercontent.com/esl/MongooseIM/6.2.1/priv/pg.sql
    $ helm install db oci://registry-1.docker.io/bitnamicharts/postgresql \
       --set auth.database=mongooseim --set auth.username=mongooseim --set auth.password=mongooseim_secret \
       --set-file 'primary.initdb.scripts.pg\.sql'=pg.sql
    

    It is useful to monitor all Kubernetes resources in another shell window:

    $ watch kubectl get pod,sts,pvc,pv,svc,hpa

    As soon as pod/db-postgresql-0 is shown as ready, you can check that the DB is running:

    $ kubectl exec -it db-postgresql-0 -- \
      env PGPASSWORD=mongooseim_secret psql -U mongooseim -c 'SELECT * from users'

    As a result, you should get an empty list of MongooseIM users. Next, let’s create a three-node MongooseIM cluster using the Helm Chart :

    $ helm repo add mongoose https://esl.github.io/MongooseHelm/
    $ helm install mim mongoose/mongooseim --set replicaCount=3 --set volatileDatabase=cets \
       --set persistentDatabase=rdbms --set rdbms.tls.required=false --set rdbms.host=db-postgresql \
       --set resources.requests.cpu=200m

    By setting persistentDatabase to RDBMS and volatileDatabase to CETS, we are eliminating the need for Mnesia, so no PVC’s are created. To connect to PostgreSQL, we specify db-postgresql as the database host. The requested CPU resources are 0.2 of a core per pod, and they will be useful for autoscaling. You can monitor the shell window, where watch kubectl … is running, to make sure that all MongooseIM nodes are ready. It is useful to verify logs as well, e.g. kubectl logs mongooseim-0 should display logs from the first node. To see how easy it is to scale up horizontally, let’s increase the number of MongooseIM nodes (which correspond to Kubernetes pods) from 3 to 6:

    $ kubectl scale --replicas=6 sts/mongooseim

    You can use kubectl logs -f mongooseim-0 to see the log messages about each newly added node of the CETS cluster. With helm upgrade , you can do rolling upgrades and scaling as well. The main difference is that the changes done with helm are permanent.

    Autoscaling

    Should you need automatic scaling, you can set up the Horizontal Pod Autoscaler . Please ensure that you have the Metrics Server installed. There are separate instructions to install it in Docker Desktop. We have already set the requested CPU resources to 0.2 of a core per pod, so let’s start the autoscaler now:

    $ kubectl autoscale sts mongooseim --cpu-percent=50 --min=1 --max=8

    It is going to keep the CPU usage at 0.1 (which is 50% of 0.2) of a core per pod. The threshold is so low to be able to easily trigger scaling up, and in any real application, it should be much higher. You should see the cluster getting scaled down until it has just one node because there is no CPU load yet. See the reported targets in the window, where you have the watch kubectl … command running. To trigger scaling up, we need to put some load on the server. We could just fire up random HTTP requests, but let’s instead use the opportunity to explore MongooseIM CLI and GraphQL API. Firstly, create a new user on the first node with the CLI:

    $ kubectl exec -it mongooseim-0 -- \
      mongooseimctl account registerUser --domain localhost --username alice --password secret

    Next, you can send XMPP messages in a loop with the GraphQL Client API:

    $ LB_HOST=$(kubectl get svc mongooseim-lb \
      --output jsonpath='{.status.loadBalancer.ingress[0].hostname}')
    $ BASIC_AUTH=$(echo -n 'alice@localhost:secret' | base64)
    $ while true; \
      do curl --get -N -H "Authorization:Basic $BASIC_AUTH" \
        -H "Content-Type: application/json" --data-urlencode \
        'query=mutation {stanza {sendMessage(to: "alice@localhost", body: "Hi") {id}}}' \
        http://$LB_HOST:5561/api/graphql; \
      done

    You should observe new pods being launched as the load increases. If there is not enough load, run the snippet in a few separate shell windows. Stopping the script should bring the cluster size back down.

    Summary

    Thanks to CETS and the Helm Chart , MongooseIM 6.2.1 can be easily installed, maintained and scaled in a cloud environment. What we have shown here are the first steps, and there is much more to explore. To learn more, you can read the documentation for MongooseIM or check out the live demo at trymongoose.im . Should you have any questions, or if you would like us to customise, configure, deploy or maintain MongooseIM for you, feel free to contact us .

    The post Instant Scalability with MongooseIM and CETS appeared first on Erlang Solutions .

    • wifi_tethering open_in_new

      This post is public

      www.erlang-solutions.com /blog/instant-scalability-with-mongooseim-and-cets/

    • chevron_right

      Erlang Solutions: Comparing Elixir vs Java

      news.movim.eu / PlanetJabber · 3 days ago - 09:39 · 21 minutes

    After many years of active development using various languages, in the past months, I started learning Elixir. I got attracted to the language after I heard and read nice things about it and the BEAM VM, but – to support my decision about investing time to learn a new language – I tried to find a comparison between Elixir and various other languages I already knew.

    What I found was pretty disappointing. In most of these comparisons, Elixir performed much worse than Java, even worse than most of the mainstream languages. With these results in mind, it became a bit hard to justify my decision to learn a new language with such a subpar performance, however fancy its syntax and other features were. After delving into the details of these comparisons, I realised that all of them were based on simplistic test scenarios and specialised use cases, basically a series of microbenchmarks (i.e. small programs created to measure a narrow set of metrics, like execution time and memory usage). It is obvious that the results of these kinds of benchmarks are rarely representative of real-life applications.

    My immediate thought was that a more objective comparison would be useful not only for me but for others as well. But before discussing the details, I’d like to compare several aspects of Elixir and Java that are not easily quantifiable.

    Development

    Learning curve

    Before I started learning Elixir, I used various languages like Java, C, C++, Perl, and Python. Despite that, all of them are imperative languages and Elixir is a functional language, I found the language concepts clear and concise, and – to tell the truth – much less complex than Java. Similarly, Elixir syntax is less verbose and easier to read and see through.

    When comparing language complexities, there is an often forgotten, but critical thing: It’s hard to develop anything more complex than a Hello World application just by using the core language. To build enterprise-grade software, you should use at least the standard library, but in most cases, many other 3rd party libraries. They all contribute to the learning curve.

    In Java, the standard library is part of the JDK and provides basic support for almost every possible use, but lacked the most important thing, the component framework (like Spring Framework or OSGi), for about 20 years. During that time, several good component frameworks were developed and became widespread, but they all come with different design principles, configuration and runtime behaviour, so for a novice developer, the aggregated learning curve is pretty steep.On the other side, Elixir has the OTP from the beginning, a collection of libraries once called Open Telecom Platform . OTP provides its own component framework which shares the same concepts and design principles as the core language.

    Documentation

    I was a bit spoiled by the massive amount of tutorials, guides and forum threads of the Java ecosystem, not to mention the really nice Javadoc that comes with the JDK. It’s not that Elixir lacks the appropriate documentation, there are really nice tutorials and guides, and most of the libraries are comparably well documented as their Java counterparts, but it will take time for the ecosystem to reach the same level of quality. There are counterexamples, of course, the Getting Started Guide is a piece of cake, I didn’t need anything else to learn the language and start active development.

    IDE support

    For me as a novice Elixir developer, the most important roadblock was the immature IDE support. Although I understand that supporting a dynamically typed language is much harder than a statically typed one like Java, I’m missing the most basic refactoring support from both the IntelliJ IDEA and VSCode . I know that Emacs offers more features, but being a hardcore vi user, I kept some distance from it.

    Fortunately, these shortcomings can be improved easily, and I’m sure there are enough interested developers in the open-source world, but as usual, some coordination would be needed to facilitate the development.

    Programming model

    Comparing entire programming models of two very different languages is too much for a blog entry, so I’d like to focus on the language support for performance and reliability, more precisely several aspects of concurrency, memory management and error handling.

    Concurrency and memory management

    The Java Memory Model is based on POSIX Threads (pthreads). Heap memory is allocated from a global pool and shared between threads. Resource synchronisation is done using locks and monitors. A conventional Java thread (Platform Thread) is a simple wrapper around an OS thread. Since an OS thread comes with its own large stack and is scheduled by the OS, it is not lightweight in any way. Java 21 introduced a new thread type ( Virtual Thread ) which is more lightweight and scheduled by the JVM, so it can be suspended during a blocking operation, allowing the OS thread to mount and execute another Virtual Thread. Unfortunately, this is only an afterthought. While it can improve the performance of many applications, it makes the already complex concurrency model even more complicated. The same is true for Structured Concurrency . While it can improve reliability, it will also increase complexity, especially if it is mixed with the old model. This is also true for the 3rd party libraries, adopting the new features, and upgrading the old deployments will take time, typically years. Until that, a mixed model will be used which can introduce additional issues.

    There are several advantages of adopting POSIX Threads, however: it is familiar for developers of languages implementing similar models (e.g. C, C++ etc.), and keeps the VM internals fairly simple and performant. On the other hand, this model makes it hard to effectively schedule tasks and heavily constrains the design of reliable concurrent code. And most importantly, it introduces issues related to concurrent access to shared resources. These issues can materialise in performance bottlenecks and runtime errors that are hard to debug and fix.

    The concurrency model of Elixir is based on different concepts, introduced by Erlang in the 80s. Instead of scheduling tasks as OS threads, it uses a construct called “process”, which is different from an operating system process. These processes are very lightweight, operating on independently allocated/deallocated memory areas and are scheduled by the BEAM VM. Scheduling is done by multiple schedulers, one for each CPU core. There is no shared memory, synchronised resource access, or global garbage collection, inter-process communication is performed using asynchronous signalling. This model eliminates the conventional concurrency-related problems and makes it much easier to write massively concurrent, scalable applications. There is one drawback, however: due to these conceptual differences, the learning curve is a bit steeper for developers experienced only with pthreads-related models.

    Fault tolerance

    Error recovery and fault tolerance in general are underrated in the world of enterprise software. For some reason, we think that fault tolerance is for mission-critical applications like controlling nuclear power plants, running medical devices or managing aircraft avionics. In reality, almost every business has critical software assets and applications that should be highly available or data, money and consumer trust will be lost. Redundancy may prevent critical downtimes, but no amount of redundancy can mitigate the risk of data corruption or other similar errors, not to mention the cost of duplicated resources.

    Java and Elixir handle errors in very different ways. While Java follows decades-old conventions and treats errors as exceptional situations, Elixir inherited a far more powerful concept from Erlang, originally borrowed from the field of fault-tolerant systems. In Elixir, errors are part of the normal behaviour of the application and are treated as such. Since there are no shared resources between processes, an error during the execution of a process does not affect nor propagate to the others; their states remain consistent, so the application can safely recover from the error. In addition, supervision trees can make sure that the failed components will be replaced immediately.

    This way, the BEAM VM provides guarantees against data loss during error recovery. But this kind of error recovery is possible only if no errors can leave the system in an inconsistent state. Since Java relies on OS threads, and shared memory can’t be protected from incorrectly behaving threads, under the JVM, there are no such safeties. Although there are Java libraries that provide better fault tolerance by implementing different programming models (probably the most noteworthy is Akka , implementing the Actor Model), the number of 3rd party libraries supporting these programming models is very limited.

    Runtime

    Performance

    For CPU or memory-intensive tasks, Java is a good choice, due to several things, like a more mature Just In Time compiler and tons of runtime optimisations in the JVM, but most importantly, because of its memory model. Since memory allocation and thread handling are basically done on OS level, the management overhead is very low.

    On the other hand, this advantage vanishes when concurrent execution is paired with a mixed workload, like blocking operations and data exchange between concurrent tasks. This is the field where Elixir thrives since Erlang and the BEAM VM were originally designed for these kinds of tasks. Due to the well-designed concurrency model, memory and other resources are not shared, requiring no synchronisation. BEAM processes are more lightweight than Java threads, and their scheduling is done at VM level, leading to fewer context switches and better scheduling granularity.

    Concurrent operations also affect memory use. Since a Java thread is not lightweight, the more threads are waiting for execution, the more memory is used. In parallel with the memory allocations related to the increasing number of waiting threads, the overhead caused by garbage collection also grows.

    Today’s enterprise applications are usually network-intensive. We have separate databases, microservices, clients accessing our services via REST APIs etc. Compared to operations on in-memory data, network communication is many orders of magnitude slower, latency is not deterministic, and the probability of erroneous responses, timeouts or infrastructure-related errors is not negligible. In this environment, Elixir and the BEAM VM offer more flexibility and concurrent performance than Java.

    Scalability

    When we talk about scalability, we should mention both vertical and horizontal scalability. While vertical scalability is about making a single hardware bigger and stronger, horizontal scalability deals with multiple computing nodes.

    Java is a conventional language in a sense that is built for vertical scaling, but it was designed at a time when vertical scaling meant running on bigger hardware with better single-core performance. It performs reasonably well on multi-core architectures, but its scalability is limited by its concurrency model since massive concurrency comes with frequent cache invalidations and lock contention on shared resources. Horizontal scaling enlarges these issues due to the increased latency. Moreover, since the JVM was also designed for vertical scaling, there is no simple way to share or distribute workload between multiple nodes, it requires additional libraries/frameworks, and in many cases, different design principles and massive code changes.

    On the other hand, a well-designed Elixir application can scale up seamlessly, without code changes. There are no shared resources that require locking, and asynchronous messaging is perfect for both multi-core and multi-node applications. Of course, Elixir itself does not prevent the developers from introducing features that are hard to scale or require additional work, but the programming model and the OTP make horizontal scaling much easier.

    Energy efficiency

    It is a well-known fact that resource and energy usage are highly correlated metrics. However, there is another, often overlooked factor that contributes significantly to energy usage. The concurrency limit is the number of concurrent tasks an application can execute without having stability issues. Near the concurrency limit, applications begin to use the CPU excessively, therefore the overhead of context switches begins to matter a lot. Another consequence is the increased memory usage, caused by the growing number of tasks waiting for CPU time. Since frequent context switches are also memory intensive, we can safely say that applications become much less energy efficient near the concurrency limit.

    Maintenance

    Tackling concurrency issues is probably the hardest part of any maintenance task. We certainly collect metrics to see what is happening inside the application, but these metrics often fail to provide enough information to identify the root cause of concurrency problems. We have to trace the execution flow to get an idea of what’s going on inside. Profiling or debugging of such issues comes with a certain cost: using these tools may alter the performance behaviour of the system in a way that makes it hard to reproduce the issue or identify the root cause.

    Due to the message-passing concurrency model, the code base of a typical concurrent Elixir application is less complex and free from resource-sharing-related implementation mistakes often poisoning Java code, eliminating the need for this kind of maintenance. Also, the BEAM VM is designed with traceability in mind, leading to lower performance cost of tracing the execution flow.

    Dependencies

    Most of the enterprise applications heavily depend on 3rd party libraries. In the Java ecosystem, even the component framework comes from a 3rd party, with its own dependencies on other 3rd party libraries. This creates a ripple effect that makes it hard to upgrade just one component of such a system, not to mention the backward incompatible changes potentially introduced by newer 3rd party components. Anyone who has tried to upgrade a fairly large Maven project could tell stories about this dependency nightmare.

    The Elixir world is no different, but the number of required 3rd party libraries can be much smaller since the BEAM VM and the OTP provide a few useful things (like the component platform, asynchronous messaging, seamless horizontal scalability, supervision trees), functionality that is very often used and can only be found in 3rd party libraries for Java.

    Let’s get more technical

    As I mentioned before, I was not satisfied with other language comparisons as they are usually based on simplistic or artificial test cases, and wanted to create something that mimics a common, but easy-to-understand scenario, and then measure the performance and complexity of different implementations. Although real-world performance is rarely just a number, it is a composite of several metrics like CPU and memory usage, I/O and network throughput, I tried to quantify the performance using the processing time, and the time an application needs to finish a task. Another important aspect is code complexity since it contributes to development and maintenance costs. The size and complexity of the implementations also matter since these factors contribute to the development and maintenance costs.

    Test scenario

    Most real-world applications process data in a concurrent way. These data originate from a database or other kind of backends, microservice or from a 3rd party service. In any way, data is transferred via a network. In the enterprise world, the dominating way of network communication is via HTTP, often part of a REST workflow. That is the reason why I chose to measure how fast and reliable REST clients can be implemented in Elixir and Java, and in addition, how complex each implementation is.

    The workflow starts with reading a configuration from a disk and then gathering data according to the configuration using several REST API calls. There are dependencies in between workflow steps, so several of them can’t be done concurrently, while the others can be done in parallel. The final step is to process the received data.

    The actual scenario is to evaluate rules, where each rule contains information used to gather data from 3rd party services and predict utility stock prices based on historical weather, stock price and weather forecast data.

    Rule evaluation is done in a concurrent manner. Both the Elixir and Java implementation are configured to evaluate 2 rules concurrently.

    Implementation details

    Elixir

    The Elixir-based REST client is implemented as an OTP application. I tried to minimise the external dependencies since I’d like to focus on the performance of the language and the BEAM VM, and the more 3rd party libraries the application depends on, the more probable there’ll be some kind of a bottleneck.

    The dependencies I use:

    Each concurrent task is implemented as a process, and data aggregation is done using asynchronous messaging. The diagram below shows the rule evaluation workflow.

    NddGhWGaFfmN-_NNFr7iux_zaPBM1w4_HBGE9mV6gN3HB15fIbh8LUgvtx657XFizU_D8rEWI-23VElOWpdD1d1Rv15eBHlZu-B8Oeu4hDma5QuFyf0xJ46MheA0rqvMAOzA_r9O2KLqAjxyBUbrN18

    There are altogether 8 concurrent processes in each task, one process is spawned for each rule, and then 3 processes are started to retrieve stock, historical weather and weather prediction data.

    Java

    The Java-based REST client is implemented as a standalone application. Since the Elixir application uses OTP, the fair comparison would be to use some kind of a component framework, like Spring or OSGi, since both are very common in the enterprise world. However, I decided not to use them, as they both would contribute heavily to the complexity of the application, although they wouldn’t change the performance profile much.

    Dependencies:

    There are two implementations of concurrent task processing. The first one uses two platform thread pools, for rule processing and for retrieving weather data. This might seem a bit naive, as this workflow could be optimised better, but please keep in mind that

    1. I wanted to model a generic workflow, and it is quite common that several thread pools are used for concurrent processing of various tasks.
    2. My intent was to find the right balance between implementation complexity and performance.

    The other implementation uses Virtual Threads for rule processing and weather data retrieval.

    The diagram below shows the rule evaluation workflow.

    d9KHuVMI6A2T_5EstLJ5TV3R1y6SNBw_4w9GLPNXsxkWF3sc2pMrKWIz3K87CRFTpAsckUV9QV7aUoHixUyK2cguAo5bgT7vz4R_wj-0T651d3txfgawwDrG6pZyviB2WGKXGxnMqGiNoncRUzX3noo

    There are altogether 6 concurrent threads in each task, one thread is started for each rule, and then 2 threads are started to retrieve historical weather and weather prediction data.

    Results

    Hardware

    Google Compute Node

    • CPU Information: AMD EPYC 7B12
    • Number of Available Cores: 8
    • Available memory: 31.36 GB

    9yY8Lp7tcsmHXxvvrZkkXYeTMjBFwAKSr6P0sm5B26DmjwtGb2GHxFGUOnjCAXVRSSD9SOxvnTvIzmcLkCz7WUmt9Ng4AjBaz90RlIh-7RyC145ikrx3zlrTS8271VIsM3sDD0sWaw0BNYmXdFfCO94

    Tasks Elixir Java Platform Threads Java Virtual Threads
    320 2.52 s 2.52 s 2.52 s
    640 2.52 s 2.52 s 2.52 s
    1280 2.51 s 2.52 s, 11% error 2.52 s
    2560 5.01 s 2.52 s, 7 errors
    5120 5.01 s High error rate
    10240 5.02 s
    20480 7.06 s

    Detailed results

    Elixir

    • Elixir 1.16.2
    • Erlang 26.2.4
    • JIT enabled: true
    Concurrent tasks per minute Average Median 99th % Remarks
    5 2.5 s 2.38 s 3.82 s
    10 2.47 s 2.38 s 3.77 s
    20 2.47 s 2.41 s 3.77 s
    40 2.5 s 2.47 s 3.79 s
    80 2.52 s 2.47 s 3.82 s
    160 2.52 s 2.49 s 3.78 s
    320 2.52 s 2.49 s 3.77 s
    640 2.52 s 2.47 s 3.81 s
    1280 2.51 s 2.47 s 3.8 s
    2560 5.01 s 5.0 s 5.17 s
    3840 5.01 s 5.0 s 5.11 s
    5120 5.01 s 5.0 s 5.11 s
    10240 5.02 s 5.0 s 5.15 s
    15120 5.53 s 5.56 s 5.73 s
    20480 7.6 s 7.59 s 8.02 s

    Java 21, Platform Threads

    • OpenJDK 64-Bit Server VM, version 21
    Concurrent tasks per minute Average Median 99th % Remarks
    5 2.5 s 2.36 s 3.71 s
    10 2.54 s 2.48 s 3.69 s
    20 2.5 s 2.5 s 3.8 s
    40 2.56 s 2.45 s 3.84 s
    80 2.51 s 2.46 s 3.8 s
    160 2.5 s 2.5 s 3.79 s
    320 2.52 s 2.46 s 3.8 s
    640 2.52 s 2.48 s 3.8 s
    1280 2.52 s 2.47 s 3.8 s 11% HTTP timeouts

    Java 21, Virtual Threads

    • OpenJDK 64-Bit Server VM, version 21
    Concurrent tasks per minute Average Median 99th % Remarks
    5 2.46 s 2.49 s 3.8 s
    10 2.51 s 2.52 s 3.68 s
    20 2.56 s 2.44 s 3.79 s
    40 2.53 s 2.46 s 3.8 s
    80 2.52 s 2.48 s 3.79 s
    160 2.52 s 2.49 s 3.77 s
    320 2.52 s 2.48 s 3.8 s
    640 2.52 s 2.49 s 3.8 s
    1280 2.52 s 2.48 s 3.8 s
    2560 2.52 s 2.48 s 3.8 s Errors: 7 (HTTP client EofException)
    3840 N/A N/A N/A Large amount of HTTP timeouts

    Stability

    Under high load, strange things can happen. Concurrency (thread contentions, races), operating system or VM-related (resource limits) and hardware-specific (memory, I/O, network etc.) errors may occur anytime. Many of them cannot be handled by the application, but the runtime usually can (or should) deal with them to provide reliable operation even in the presence of faults.

    During the test runs, my impression was that the BEAM VM is superior in this task, in contrast to the JVM which entertained me with various cryptic error messages, like the following one:

    java.util.concurrent.ExecutionException: java.io.IOException
            at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
            at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
            at esl.tech_shootout.RuleProcessor.evaluate(RuleProcessor.java:38)
            at esl.tech_shootout.RuleProcessor.lambda$evaluateAll$0(RuleProcessor.java:29)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.base/java.lang.Thread.run(Thread.java:840)
    Caused by: java.io.IOException
            at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:586)
            at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:123)
            at esl.tech_shootout.RestUtils.callRestApi(RestUtils.java:21)
            at esl.tech_shootout.StockService.stockSymbol(StockService.java:23)
            at esl.tech_shootout.StockService.stockData(StockService.java:17)
            at esl.tech_shootout.RuleProcessor.lambda$evaluate$3(RuleProcessor.java:37)
            ... 4 more
    Caused by: java.nio.channels.ClosedChannelException
            at java.base/sun.nio.ch.SocketChannelImpl.ensureOpen(SocketChannelImpl.java:195)

    Although in this case, I know the cause of this error, the error message is not very informative. Compare the above stack trace with the error raised by Elixir and the BEAM VM:

    16:29:53.822 [error] Process #PID<0.2373.0> raised an exception
    ** (RuntimeError) Finch was unable to provide a connection within the timeout due to excess queuing for connections. Consider adjusting the pool size, count, timeout or reducing the rate of requests if it is possible that the downstream service is unable to keep up with the current rate.
    
        (nimble_pool 1.0.0) lib/nimble_pool.ex:402: NimblePool.exit!/3
        (finch 0.18.0) lib/finch/http1/pool.ex:52: Finch.HTTP1.Pool.request/6
        (finch 0.18.0) lib/finch.ex:472: anonymous fn/4 in Finch.request/3
        (telemetry 1.2.1) /home/sragli/git/tech_shootout/elixir_demo/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
        (elixir_demo 0.1.0) lib/elixir_demo/rule_processor.ex:56: ElixirDemo.RuleProcessor.retrieve_weather_data/3
    

    This exception shows what happens when we mix different concurrency models:

    Thread[#816,HttpClient@6e579b8-816,5,VirtualThreads]
     at java.base@21/jdk.internal.misc.Unsafe.park(Native Method)
     at java.base@21/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
     at java.base@21/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
     at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
     at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1139)
    
    
    

    The Jetty HTTP client is a nice piece of code and very performant, but uses platform threads in its internals, while our benchmark code relies on virtual threads.

    That’s why I had to switch from JDK HttpClient to Jetty:

    Caused by: java.io.IOException: /172.17.0.2:60876: GOAWAY received
           at java.net.http/jdk.internal.net.http.Http2Connection.handleGoAway(Http2Connection.java:1166)
           at java.net.http/jdk.internal.net.http.Http2Connection.handleConnectionFrame(Http2Connection.java:980)
           at java.net.http/jdk.internal.net.http.Http2Connection.processFrame(Http2Connection.java:813)
           at java.net.http/jdk.internal.net.http.frame.FramesDecoder.decode(FramesDecoder.java:155)
           at java.net.http/jdk.internal.net.http.Http2Connection$FramesController.processReceivedData(Http2Connection.java:272)
           at java.net.http/jdk.internal.net.http.Http2Connection.asyncReceive(Http2Connection.java:740)
           at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.processQueue(Http2Connection.java:1526)
           at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:182)
           at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)
           at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:207)
           at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
           at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
           at java.base/java.lang.Thread.run(Thread.java:1583)

    According to the HTTP 2.0 standard, an HTTP server can send a GOAWAY response at any time (typically under high load, in our case, after about 2000 requests/min) to indicate connection shutdown. It is the client’s responsibility to handle this situation. The HttpClient implemented in the JDK fails to do that internally and it does not provide enough information to make the proper error handling possible.

    Concluding remarks

    As I expected, both the Elixir and Java applications performed well in low concurrency settings, but the Java application became less stable as the number of concurrent tasks increased, while Elixir exhibited rock-solid performance with minimal slowdown.

    The BEAM VM was also superior in providing reliable operation under high load, even in the presence of faults. After about 2000 HTTP requests per second, timeouts were inevitable, but they didn’t impact the stability of the application. On the other hand, the JVM started to behave very erratically after about 1000 (Platform Threads-based implementation) or 3000 (Virtual Threads-based implementation) concurrent tasks.

    Code complexity

    There are a few widely accepted complexity metrics to quantify code complexity, but I think the most representative ones are the Lines of Code and Cyclomatic Complexity.

    Lines of Code, more precisely the Source Lines of Code (SLoC for short) quantifies the total number of lines in the source code of an application. Strictly speaking, it is not very useful as a complexity measure, but it is a good indicator of how much effort is needed to look through a particular codebase. Source Lines of Code is measured by calculating the total number of lines in all source files, not including the dependencies and configuration files.

    Cyclomatic Complexity (CC for short) is more technical as it measures the number of independent execution paths through the source code. CC measurement works in a different way for each language. Cyclomatic Complexity of the Elixir application is measured using Credo , and CC of the Java application is quantified using the CodeMetrics plugin of IntelliJ IDEA.

    These numbers show that there is a clear difference in complexity even between such small and simple applications. While 9 is not a particularly high score for Cyclomatic Complexity, it indicates that the logical flow is not simple. It might not be concerning, but what’s more problematic is that even the most basic error handling increases the complexity by 3.

    Conclusion

    These results might paint a black-and-white picture, but keep in mind that both Elixir and Java have their advantages and shortcomings. If we are talking about CPU or memory-intensive operations in low concurrency mode, Java is the clear winner thanks to its programming model and the huge amount of optimisation done in the JVM. On the other hand, Elixir is a better choice for highly concurrent, available and scalable applications, not to mention the field of fault-tolerant systems. Elixir also has the advantage of being a more modern language with less syntactic clutter and less need to write boilerplate code.

    The post Comparing Elixir vs Java appeared first on Erlang Solutions .

    • chevron_right

      Erlang Solutions: A Comprehsive Guide to Ruby v Elixir

      news.movim.eu / PlanetJabber · Thursday, 9 May - 10:21 · 8 minutes

    Deciding what programming language is best for your long-term business strategy is a difficult decision. If you’re tossing the coin between Ruby and Elixir, or considering making a shift from one to the other, you probably have a lot of questions about both languages.

    So let’s compare these widely popular and dynamic languages: Elixir and Ruby. We’ll explore the advantages and disadvantages of each language, as well as their optimal use cases and other key points, providing you with a clearer insight into both.

    Ruby and Elixir: A History

    To gain a better understanding of the frequent Ruby and Elixir comparisons, let’s take it back to the 90’s when Ruby was created by Yukihiro Matsumoto. He combined the best features of Small, Perl, Eiffel, Ada, Lip and Smalltalk languages to simplify the tasks of developers. But Ruby’s popularity surged with the release of the open-source framework Ruby on Rails.

    This launch proved to be revolutionary in the world of web development, making code tasks achievable in a matter of days instead of months. As one of the leading figures on the Rails Core Team, Jose Valim recognised the potential for evolution within the Ruby language.

    In 2012, Elixir was born- a functional programming language, built on the Erlang virtual machine (VM). The aim of Elixir was to create a language with the friendly syntax of Ruby while boasting fault tolerance, concurrency capabilities and a commitment to developer satisfaction.

    The Elixir community also has Phoenix, an open-source framework from Phoenix’s creator Chris McCord. Working with Jose Valim and implementing the core values from Ruby on Rails perfected a much more effective framework for the Elixir ecosystem.

    So what is Elixir?

    Elixir describes itself as a “dynamic, functional language for building scalable and maintainable applications.” It is a great choice for any situation where scalability, performance and productivity are priorities, particularly within IoT endeavours and web applications.

    Elixir runs on the BEAM virtual machine, originating from Erlang’s virtual machine (VM). It is well known for managing fault-tolerant, low-latency distributed systems. Created in 1986 by the Ericsson company, Erlang was designed to address the growing demands within the telecoms industry.

    It was later released as free and open-source software in 1998 and has since grown in popularity thanks to the demand for concurrent services.

    What is Ruby?

    Ruby stands out as a highly flexible programming language. Developers who code in Ruby are able to make changes to its functionality. Unlike compiled languages like C or C++, Ruby is an interpreted language, similar to Python.

    But unlike Python which focuses on a singular, definitive solution for every problem, Ruby projects try to take on multiple problem-solving approaches. Depending on your project,  this approach has pros and cons.

    One hallmark of Ruby is its user-friendly nature. It hides a lot of intricate details from the programmer, making it much easier to use compared to other popular languages. But it also means that finding bugs in code can be harder.

    There is a major convenience factor to coding in Ruby. Any code that you write will run on any major operating system such as macOR, Windows, and Linux, without having to be ported.

    The pros of Elixir

    If you’re considering migrating from Ruby to Elixir, you’ll undoubtedly be looking into its benefits and some key advantages it has over other languages. So let’s jump into some of its most prominent features.

    Built on the BEAM

    As mentioned, Elixir operates on the Erlang virtual machine (BEAM). It has a long history as one of the oldest VMs in IT history and remains widely used. The Erlang VM BEAM is ideal for managing and building systems with concurrent connections.

    Immutable Data

    A major advantage of Elixir is its support for immutable data, which simplifies code understanding. Elixir ensures that data is unchanged once it has been defined, enhancing code reliability by preventing unexpected changes to variables, and making for easier debugging.

    Top Performance

    Elixir offers amazing performance. Phoenix framework is the most popular web development framework in Elixir, boasting remarkable speed and response times (a matter of milliseconds). While Rails isn’t a slow system either, Elixir’s performance just edges it out, making it a superior choice.

    Parallelism

    Parallel systems often have latency and responsiveness challenges due to how much computer power is required for a single task. But Elixir addresses this with its very clever process scheduler, which proactively reallocates control to different processes.

    So even under heavy loads, a slow process isn’t able to significantly impact the overall performance of an Elixir application. This capability ensures low latency, a key requirement for modern web applications.

    Highly Fault-Tolerant

    In most programming languages, when a bug is identified in one process, it crashes the whole application. But Elixir handles this differently. It has unmatched fault tolerance.

    A fan favourite of the language, Elixir inherits Erlang’s “let it crash” philosophy, allowing processes to restart after a critical failure. This eliminates the need for complex recovery strategies.

    Distributed Concurrency

    Elixir supports code concurrency, allowing you to run concurrent connections on a single computer and multiple machines.

    Scalability

    Elixir gets the most out of a single machine perfect for systems or applications that need to scale or maintain traffic. Thanks to its architecture, there’s no need to add servers to accommodate demand continuously.

    The pros of Ruby

    Now let’s explore the benefits that Ruby has to offer. There are massive advantages for developers, from its expansive library ecosystem to its user-friendly syntax and supportive community.

    Huge ecosystem

    Not only is Ruby a very user-friendly programming language, but it also boasts a vast library ecosystem. Whatever feature you want to implement, there’s likely something available to help you develop swift applications.

    Easy to work with

    The founder of Ruby’s aim was to make development a breeze and pleasant for users. For this reason, Ruby is straightforward, clean and has an easily understandable syntax. This makes for very easy and productive development, which is why it remains such a popular choice with developers.

    Helpful, vibrant community

    The Ruby community is a vibrant one that thrives with the consistent publishing of readily available solutions that are open to the public. This environment is very advantageous for new developers, who can easily seek assistance and valuable solutions online.

    Commitment to standards

    Ruby offers strong support for web standards across all aspects of an application, from its user interface to data transfer.

    When building an application with Ruby, developers adhere to already established software design principles such as  “coding by convention,” “don’t repeat yourself,” and the “active record pattern.”

    So why are all of these points considered so advantageous to Ruby?

    Firstly, it simplifies the learning curve for beginners, designed to enhance the professional experience. It also lends itself to better code readability, which is great for collaboration and developers and finally, it reduces the amount of code needed to implement features.

    Key differences between Ruby and Elixir

    There are some significant differences between the two powerhouse languages. While Elixir and Ruby are both versatile, dynamic languages, unlike Ruby, Elixir code undergoes ahead-of-time compilation to Erlang VM (virtual machine) bytecode, which enhances its single-core performance substantially. Elixir’s focus is on code readability and expressiveness, while its robust macro system facilitates easy extensibility.

    Elixir and Ruby’s syntax also differ in several ways. For instance, Elixir uses pipes (marked by |> operator) to pass the outcome of one expression as the initial argument to another function, while Ruby employs “.” for method chaining.

    Also, Elixir provides explicit backing for immutable data structures, a feature not directly present in Ruby. It also offers first-rate support for typespecs, a capability lacking in Ruby.

    Best use for developers

    Elixir is a great option for developers who want the productivity of Ruby and the scalability of Elixir. It also performs just as well as Ruby for Minimum Viable Products (MVPs) and startups for larger applications, while demonstrating robust scalability for extensive applications.

    For companies who want swift delivery without sacrificing quality, Elixir works out as a great overall choice.

    Exploring the talent pool for Elixir and Ruby

    Elixir is a newer language than Ruby and therefore has a smaller pool of developers. But let’s not forget it’s also a functional language, and functional programming typically demands a different way of thinking compared to object-oriented programming.

    As a result, Elixir developers tend to have more experience and understanding of programming concepts. And the Elixir community is also rapidly growing. Those who are familiar with Ruby commonly make the switch to Elixir.

    Although Elixir developers might be more difficult to find, once you do, they are worth their weight.

    Who is using Ruby and Elixir?

    Let’s take a look at some highly successful companies that have used Ruby and Elixir:

    Ruby

    Airbnb: Uses Ruby on Rails for its web platform, including features like search, booking, and reviews.

    GitHub: Is built primarily using Ruby on Rails.

    Shopify: Relies on Ruby on Rails for its backend infrastructure.

    Basecamp: Built using Ruby on Rails.

    Kickstarter: Uses Ruby on Rails for its website and backend services.

    Elixir

    Discord: Uses Elixir for its real-time messaging infrastructure, benefiting from Elixir’s concurrency and fault tolerance.

    Pinterest: Takes advantage of Elixir’s scalability and fault-tolerance features.

    Bleacher Report: Bleacher Report, a sports news website, utilizes Elixir for its backend services, including real-time updates and notifications.

    Moz: Uses Elixir for its backend services, benefiting from its concurrency model and fault tolerance.

    Toyota Connected: Leverages Elixir for building scalable and fault-tolerant backend systems for connected car applications.

    So, what to choose?

    Migrating or simply deciding between programming languages presents an opportunity for enhancing performance, scalability, and robustness. But it is a journey. One that requires careful planning and execution and achieve the best long-term results for your business.

    While the Ruby community offers longevity, navigating outdated solutions can be a challenge. Nonetheless, the overlap of the Ruby and Elixir communities fosters a supportive environment for transitioning from one to the other. Elixir provides a learning curve that may deter some, but for developers seeking typed languages and parallel computing benefits, it is invaluable.

    If you’re already working with existing Ruby infrastructure, incorporating Elixir to address scaling and reliability issues is a viable option. The synergies between the two languages promote a seamless transition.

    Ultimately, while Ruby remains a solid choice, the advantages of Elixir make it a compelling option worth considering for future development and business growth.

    The post A Comprehsive Guide to Ruby v Elixir appeared first on Erlang Solutions .

    • chevron_right

      Erlang Solutions: Naming your Daemons

      news.movim.eu / PlanetJabber · Thursday, 2 May - 13:46 · 4 minutes

    Within Unix systems, a daemon is a long-running background process which does not directly interact with users. Many similar processes exist within a BEAM application. At times it makes sense to name them, allowing sending messages without requiring the knowledge of their process identifier (aka PID). There are several benefits to naming processes, these include:

    1. Organised processes: using a descriptive and meaningful name organises the processes in the system. It clarifies the purpose and responsibilities of the process.
    2. Fault tolerance: when a process is restarted due to a fault it has to share its new PID to all callees. A registered name is a workaround to this. Once the restarted process is re-registered there is no additional action required and messages to the registered process resume uninterrupted.
    3. Pattern implementation: a Singleton, Coordinator, Mediator or Facade design pattern commonly has one registered process acting as the entry point for the pattern.

    Naming your processes

    Naturally, both Elixir and Erlang support this behaviour by registering the process. One downside with registering is requiring an atom. As a result, there is an unnecessary mapping between atoms and other data structures, typically between strings and atoms.

    To get around this is it a common pattern to perform the registration as a two-step procedure and manage the associations manually, as shown in below:

    
    #+begin_src Elixir
    {:ok, pid} = GenServer.start_link(Worker, [], [])
    register_name(pid, "router-thuringia-weimar")
    
    pid = whereis_name("router-thuringia-weimar")
    GenServer.call(pid, msg)
    
    unregister_name("router-thuringia-weimar")
    #+end_src
    
    
    

    Figure 1

    Notice the example uses a composite name: built up from equipment type, e.g. router, state, e.g. Thuringia, and city, e.g. Weimar. Indeed, this pattern is typically used to address composite names and in particular dynamic composite names. This avoids the issue of the lack of atoms garbage collection in the BEAM.

    As a frequently observed pattern, both Elixir and Erlang offer a convenient method to accomplish this while ensuring a consistent process usage pattern. In typical Elixir and Erlang style, this is subtly suggested in the documentation through a concise, single-paragraph explanation.

    In this write- up, we will demonstrate using built-in generic server options to achieve similar behaviour.

    Alternative process registry

    According to the documentation, we can register a GenServer into an alternative process registry using the via directive.

    The registry must provide the following callbacks:

    register_name/2 , unregister_name/1 , whereis_name/1 , and send/2 .

    As it happens there are two commonly available applications which satisfy these requirements: gproc and Registry . gproc is an external Erlang library written by Ulf Wiger, while Registry is a built-in Elixir library.

    gproc is an application in its own right, simplifying using it. It only needs to be started as part of your system, whereas Registry requires adding the Registry GenServer to your supervision tree.

    We will be using gproc in the examples below to address the needs of both Erlang and Elixir applications.

    To use gproc we have to add it to the project dependency.

    Into Elixir’s mix.exs :

    #+begin_src Elixir
      defp deps do
        [
          {:gproc, git: "https://github.com/uwiger/gproc", tag: "0.9.1"}
        ]
      end
    #+end_src
    
    

    Figure 2

    Next, we change the arguments to start_link , call and cast to use the gproc alternative registry, as listed below:

    #+begin_src Elixir :noweb yes :tangle worker.ex
    defmodule Edproc.Worker do
      use GenServer
    
      def start_link(name) do
        GenServer.start_link(__MODULE__, [], name: {:via, :gproc, {:n, :l, name}})
      end
    
      def call(name, msg) do
        GenServer.call({:via, :gproc, {:n, :l, name}}, msg)
      end
    
      def cast(name, msg) do
        GenServer.cast({:via, :gproc, {:n, :l, name}}, msg)
      end
    
      <<worker-gen-server-callbacks>>
    end
    #+end_src
    

    Figure 3

    As you can see the only change is using {:via, :gproc, {:n, :l, name}} as part of the GenServer name. No additional changes are necessary. Naturally, the heavy lifting is performed inside gproc .

    The tuple {:n, :l, name} is specific for gproc and refers to setting up a “l:local n:name” registry. See the gproc for additional options.

    Finally, let us take a look at some examples.

    Example

    In an Elixir shell:

    #+begin_src Elixir
    iex(1)> Edproc.Worker.start_link("router-thuringia-weimar")
    {:ok, #PID<0.155.0>}
    iex(2)> Edproc.Worker.call("router-thuringia-weimar", "hello world")
    handle_call #PID<0.155.0> hello world
    :ok
    iex(4)> Edproc.Worker.start_link({:router, "thuringia", "weimar"})
    {:ok, #PID<0.156.0>}
    iex(5)> Edproc.Worker.call({:router, "thuringia", "weimar"}, "reset-counter")
    handle_call #PID<0.156.0> reset-counter
    :ok
    #+end_src
    

    Figure 4

    As shown above, it is also possible to use a tuple as a name. Indeed, it is a common pattern to categorise processes with a tuple reference instead of constructing a delimited string.

    Summary

    The GenServer behaviour offers a convenient way to register a process with an alternative registry such as gproc . This registry permits the use of any BEAM term instead of the usual non-garbage collected atom name enhancing the ability to manage process identifiers dynamically. For Elixir applications, using the built-in Registry module might be a more straightforward and native choice, providing a simple yet powerful means of process registration directly integrated into the Elixir ecosystem.

    Appendix

    #+NAME: worker-gen-server-callbacks
    #+BEGIN_SRC Elixir
      @impl true
      def init(_) do
        {:ok, []}
      end
    
      @impl true
      def handle_call(msg, _from, state) do
        IO.puts("handle_call #{inspect(self())} #{msg}")
        {:reply, :ok, state}
      end
    
      @impl true
      def handle_cast(msg, state) do
        IO.puts("handle_cast #{inspect(self())} #{msg}")
        {:noreply, state}
      end
    #+END_SRC
    

    Figure 5

    The post Naming your Daemons appeared first on Erlang Solutions .

    • chevron_right

      Erlang Solutions: Technical debt and HR – what do they have in common?

      news.movim.eu / PlanetJabber · Thursday, 25 April - 09:17 · 3 minutes

    At first glance, it may sound absurd. Here we have technical debt, a purely engineering problem, as technical as it can get, and another area, HR, dealing with psychology and emotions, put into one sentence. Is it possible that they are closely related? Let’s take it apart and see.

    Exploring technical debt

    What is technical debt, anyway? A tongue-in-cheek definition is that it is code written by someone else. But there is more to it – it is code written years ago, possibly by someone who has left the company. Also, every major piece of software is written incrementally over many years. Even if it started with a very detailed, well-thought-through design, there inevitably came plenty of modifications and additions which the original design could not easily accommodate.

    Your predecessors sometimes had to cut corners and bend over backwards to achieve desired results in an acceptable amount of time. Then they moved on, someone else took over and so on.

    What you now have is a tangled mess, mixing coding styles, techniques and design patterns, with an extra addition of ad-hoc solutions and hacks. You see a docstring like “temporary solution, TODO: clean up”, you run git blame and it is seven years old. It happens to everybody.

    The business behind technical debt

    Technical debt is a business issue. You can read about it in more detail in our previous post .

    technical debt

    Source: Medium

    The daily tasks of most developers are fixing bugs and implementing new features in existing code. The more messy and convoluted the code is, the more time it takes every time one has to read it and reason about it. And it is real money: according to McKinsey Report , this burden amounts to 20%-40% of an average business’ technology stack. Engineers are estimated to spend up to 50% of their time struggling with it.

    So what can businesses do to get their code in check? Here are some suggestions:

    • Taking a step back
    • Reassessing the architecture and techniques
    • Making more informed choices
    • Rewriting parts of the code to make it consistent and understandable, removing unused code and duplications

    Unfortunately, this is very rarely done, since it does not bring any visible improvements to the product – clients are not interested in code quality, they want software that does its job. Improving the code costs real money, while the increase in developer productivity is impossible to quantify.

    Technical debt also has another property – it is annoying. And this brings us nicely to the second topic.

    Happy HR, Happier devs

    What is HR about? In part, it is about the well-being of employees. Every employer wants good people to stay in the company. The most valuable employee is someone who likes their job and feels good about the place. HR departments go to great lengths to achieve this.

    But, you can buy new chairs and phones, decorate the office, buy pizza, organise board games evenings – all this is mostly wasted if the following morning your devs show up in their workplace only to say “Oh no, not this old cruft again”, embellishing that statement with a substantial amount of profanities.

    Now I tell you this: Nothing makes developers happier than allowing them to address their pain points. Ask them what they hate the most about the codebase and let them improve it, the way they choose to, at their own pace. They will be delighted.

    You may ask how I know. Firstly, I’m a dev myself. Secondly, I’m fortunate enough to be currently working for a company that took the steps and did exactly that:

    Step 1: Set up a small “tech debt” team

    Step 2: Collected improvement proposals from all developers

    Step 3: Documented them

    Step 4: Defined priorities

    Currently, the technical debt team or the proposers themselves are gradually putting these proposals into action, one by one. The code is getting better. We are becoming more productive. And if we’re happy, isn’t HR?

    Calling upon the compassionate and proactive HR professionals out there: talk to your CTOs, tell them you all are after the same thing – you want these frustrated, burned-out coders happy, enthusiastic and more productive, and that you have an idea of how to achieve this.

    Chances are they will be interested.

    The post Technical debt and HR – what do they have in common? appeared first on Erlang Solutions .

    • wifi_tethering open_in_new

      This post is public

      www.erlang-solutions.com /blog/technical-debt-and-hr-what-do-they-have-in-common/

    • chevron_right

      ProcessOne: ejabberd Docs now using MkDocs

      news.movim.eu / PlanetJabber · Wednesday, 24 April - 17:02 · 1 minute

    The ejabberd Docs website did just get a major rework: new content management system, reorganized navigation, improved markdown, and several improvements!

    Brief documentation timeline

    ejabberd started in November 2002 (see a timeline in the ejabberd turns 20 blog post). And the first documentation was published in January 2003, using LaTeX, see Ejabberd Installation and Operation Guide . That was one single file, hosted in the ejabberd CVS source code repository, and was available as a single HTML file and a PDF.

    As the project grew and got more content, in 2015 the documentation was converted from LaTeX to Markdown , moved from ejabberd repository to a dedicated docs.ejabberd.im git repository, and published using a Go HTTP server in docs.ejabberd.im , see an archived ejabberd Docs site .

    New ejabberd Docs site

    Now the ejabberd documentation has moved to MkDocs+Material, and this brings several changes and improvements:

    Site and Web Server:

    • Replaced Go site with MkDocs
    • Material theme for great features and visual appeal, including light/dark color schemes
    • Still written in Markdown, but now using several MkDocs, Material and Python-Markdown extensions
    • The online site is built by GitHub Actions and hosted in Pages, with smaller
      automatic deployment time
    • Offline reading: the ejabberd Docs site can be downloaded as a PDF or zipped HTML, see the links in home page

    Navigation

    • Major navigation reorganization, keeping URLs intact so old links still work (only Install got some relevant URL changes)
    • Install section is split into several sections: Containers, Binaries, Compile, …
    • Reorganized the Archive section, and now it includes the corresponding Upgrade notes
    • Several markdown files from the ejabberd and docker-ejabberd repositories are now incorporated here

    Content

    • Many markdown visual improvements, specially in code snippets
    • Options and commands that were modified in the last release will show a mark, see for example API Reference
    • Version annotations are shown after the corresponding title, see for example sql_flags
    • Modules can have version annotations, see for example mod_matrix_gw
    • Links to modules, options and API now use the real name with _ character instead of - (compare old #auth-opts with #auth_opts ). The old links are still supported, no broken links.
    • Listen Modules section is now better organized
    • New experimental ejabberd Developer Livebook

    So, please check the revamped ejabberd Docs site, and head to docs.ejabberd.im git repository to report problems and propose improvements.

    The post ejabberd Docs now using MkDocs first appeared on ProcessOne .
    • wifi_tethering open_in_new

      This post is public

      www.process-one.net /blog/ejabberd-docs-now-using-mkdocs/

    • chevron_right

      Ignite Realtime Blog: Smack 4.4.8 released

      news.movim.eu / PlanetJabber · Thursday, 4 April - 17:04

    We are happy to announce the release of Smack 4.4.8, our XMPP-client library for JVMs and Android. For a high-level overview of what’s changed in Smack 4.4.8, check out Smack’s changelog

    Smack 4.4.8 contains mostly small fixes. However, we fixed one nasty bug in Smack’s reactor causing an, potentially endless, busy loop. Smack’s new connection infrastrucutre makes heavy use of the reactor, that enables tausands of connections being served by only a handful of threads.

    As always, this Smack patchlevel release is API compatible within the same major-minor version series (4.4) and all Smack releases are available via Maven Central .

    We would like to use this occasion to point at that Smack now ships with a NOTICE file. Please note that this adds some requirements when using Smack as per the Apache License 2.0 . The content of Smack’s NOTICE file can conveniently be retrieved using Smack.getNoticeStream() .

    1 post - 1 participant

    Read full topic

    • chevron_right

      Erlang Solutions: A Guide to RabbitMQ

      news.movim.eu / PlanetJabber · Thursday, 4 April - 12:20 · 5 minutes

    Looking to learn more about the basics of RabbitMQ? This powerful message broker plays a key role in modern, distributed systems.

    This post will break down its fundamentals and highlight its importance in the world of modern, distributed systems.

    An introduction to RabbitMQ

    RabbitMQ emerged from the need to create a scalable, robust messaging system that was able to handle high volumes of communications between applications, all while maintaining both data and performance.

    It is now a popular open-source messaging broker, with queue software written in Erlang. One of its key strengths is its ability to support and adhere to Application Programming Interface (API) protocols, for example, AMQP, HTTP AND STOMP.

    What are APIs you ask?

    They define the rules and conventions that allow for the interaction and communication of different software. For developers, APIs are the go-between that allows them to access a software or services functionality, without the need for a full understanding of the ins and outs of that particular system.

    In turn, these protocols offer a standard method of transmitting commands and data. The result? Seamless integration and interoperability between different systems.

    Let’s circle back to one previously mentioned protocol, the Advanced Message Queuing Protocol or AMQP. This protocol was made to ensure that messages are reliably delivered between applications, no matter where the platform it is running on is located. AMQP has precise rules for the delivery, formatting and confirmation of messages. This ensures that every message sent through an AMQP-based system, like RabbitMQ, reaches its intended location.

    Here’s an illustration better explaining the AMQP system:

    WLNyVn79-YOLOKFPxdXPwWcUNMU1zDFfgD7XJC7wTvD5CP6wDwME9yhMQ4Ji0sIJzRnIQ6wVfvX1r_nLpkzu9KI1o4tqdlSgRZesJ97N5KDCudIx5O1lW_jmbc6Nw2CQt6cnVPY67JMw_5-DqWR3E00

    Source: The Heart of RabbitMQ

    What is RabbitMQ used for?

    Developers use RabbitMQ to efficiently process high-throughput and reliable background jobs and facilitate the integration and communication between applications. It is also great at managing complex routing to consumers by integrating various applications and services.

    RabbitMQ is also a great solution for web servers that require a rapid-request response. It also effectively distributes workloads between workers, handling over 20,000 messages per second. It can manage background jobs and longer-running tasks, for example, PDF conversion and file scanning.

    How does RabbitMQ work?

    Think of RabbitMQ as a middleman. It collects messages from a producer (publisher) and passes them on to receivers (consumers). Using a messaging queue, it then holds messages until the consumers can process them.

    Here’s a better overview of these core systems:

    Producer (publisher) It sends messages to a queue for processing by consumers.
    Queue Where messages are transferred and stored until they can be processed.
    Consumer (receiver) It receives messages from queues and uses them for other defined tasks.
    Exchange The entry point for the messaging broker. It uses routing rules to determine which queues should receive the message.
    Broker A messaging system that stores produced data. Another application can connect to it using specific details, like parameters or connection strings, to receive and use that data.
    Channel Channels offer a lightweight connection to a broker via a shared Transmission Control Protocol (TCP) connection.

    kp3v0PXLxkDsL_nKMPs8IR55TsPsb5fRgzFaIVNVFuzStdjWTEhGBTNOEs-4SXuFEKvhuV-BQGGuOE2E7Ndp-p9xqoe82naibtoT1TeXP8_y_-7dSVatVyQN_W6r30q36-mb7Lzh7jLJv6v_BdH2ve8

    Source: RabbitMQ tutorials

    Key features of RabbitMQ

    As one of the most powerful and flexible messaging systems, RabbitMQ offers several key features, including:

    Security: Various security features in RabbitMQ are designed to protect systems from unauthorised access and potential data breaches. With authentication and authorisation support, administrators can control which users or applications have access to certain queues or exchanges. It also supports SSL/TLS encryption , to ensure clear communication between brokers and clients.

    Reliability: Reliable message delivery by supporting features, such as message acknowledgement and persistent message storage.

    Scalable and fault-tolerant: RabbitMQ provides features for building scalable and fault-tolerant messaging systems. It also supports clustering, whereby adding more nodes to the cluster allows the system to handle higher message volumes. It’s then able to distribute the workload across multiple nodes, making for efficient utilisation of resources. In the case of a node failure, other nodes in the cluster can continue to handle messages without interruption.

    Extended features: RabbitMQ is not limited to the AMQP protocol, but is very versatile and can support a host of others, such as MQTT and STOMP.

    Enterprise and the Cloud : RabbitMQ is lightweight and easy to deploy on the public as well as private clouds using pluggable authentication authorisation.

    Tools and Plugins: RabbitMQ offers a host of tools and plugins , ideal for integration and wider support.

    Common use cases for RabbitMQ

    We’ve already highlighted the versatility of RabbitMQ in modern distributed systems. With its robust features and flexible architecture, here are some most common use cases:

    Legacy applications: RabbitMQ integrates with legacy systems by using available or custom plugins. You can connect consumer apps to legacy apps for example, connecting JMS apps using the Java Message Service (JMS) plug-in and JMS client library.

    Distributed systems: RabbitMQ serves as a messaging infrastructure in distributed systems. It fosters asynchronous communication between different components, facilitating the scalability and decoupling of the system.

    IoT applications: When used in Internet of Things (IoT) applications, RabbitMQ can handle the exchange of messages between devices and backend systems, allowing for reliable and efficient communication, control and real-time monitoring of IoT devices.

    Chat applications: For real-time communication in chat applications, RabbitMQ manages messaging exchanges between users, facilitating group chat and instant messaging functionalities.

    Task/job queues: RabbitMQ manages task queues and distributes work across multiple workers. This means that tasks are processed efficiently and at scale, reducing bottlenecks and utilising resources.

    Event-driven architectures: RabbitMQ is great for carrying out event-driven architectures.

    It allows various system components to respond to events and seamlessly interact with each other.
    Microservices communication: A common use of RabbitMQ is enabling asynchronous and reliable communication between microservices. Messages are delivered, even if some services are unavailable.

    To conclude

    As businesses seek to adopt distributed architectures and microservices-based applications, RabbitMQ remains a go-to choice for improved adaptability and seamless integration across systems. If you’d like to discuss how RabbitMQ can improve your applications, get in touch with the Erlang Solutions team.

    The post A Guide to RabbitMQ appeared first on Erlang Solutions .