call_end

    • chevron_right

      Andy Wingo: whippet lab notebook: guile, heuristics, and heap growth

      news.movim.eu / PlanetGnome • 22 May • 5 minutes

    Greets all! Another brief note today. I have gotten Guile working with one of the Nofl -based collectors, specifically the one that scans all edges conservatively ( heap-conservative-mmc / heap-conservative-parallel-mmc ). Hurrah!

    It was a pleasant surprise how easy it was to switch—from the user’s point of view, you just pass --with-gc=heap-conservative-parallel-mmc to Guile’s build (on the wip-whippet branch); when developing I also pass --with-gc-debug , and I had a couple bugs to fix—but, but, there are still some issues. Today’s note thinks through the ones related to heap sizing heuristics.

    growable heaps

    Whippet has three heap sizing strategies : fixed, growable, and adaptive ( MemBalancer ). The adaptive policy is the one I would like in the long term; it will grow the heap for processes with a high allocation rate, and shrink when they go idle. However I won’t really be able to test heap shrinking until I get precise tracing of heap edges, which will allow me to evacuate sparse blocks.

    So for now, Guile uses the growable policy, which attempts to size the heap so it is at least as large as the live data size, times some multiplier. The multiplier currently defaults to 1.75×, but can be set on the command line via the GUILE_GC_OPTIONS environment variable. For example to set an initial heap size of 10 megabytes and a 4× multiplier, you would set GUILE_GC_OPTIONS=heap-size-multiplier=4,heap-size=10M .

    Anyway, I have run into problems! The fundamental issue is fragmentation. Consider a 10MB growable heap with a 2× multiplier, consisting of a sequence of 16-byte objects followed by 16-byte holes. You go to allocate a 32-byte object. This is a small object (8192 bytes or less), and so it goes in the Nofl space. A Nofl mutator holds on to a block from the list of sweepable blocks, and will sequentially scan that block to find holes. However, each hole is only 16 bytes, so we can’t fit our 32-byte object: we finish with the current block, grab another one, repeat until no blocks are left and we cause GC. GC runs, and after collection we have an opportunity to grow the heap: but the heap size is already twice the live object size, so the heuristics say we’re all good, no resize needed, leading to the same sweep again, leading to a livelock.

    I actually ran into this case during Guile’s bootstrap, while allocating a 7072-byte vector. So it’s a thing that needs fixing!

    observations

    The root of the problem is fragmentation. One way to solve the problem is to remove fragmentation; using a semi-space collector comprehensively resolves the issue, modulo any block-level fragmentation .

    However, let’s say you have to live with fragmentation, for example because your heap has ambiguous edges that need to be traced conservatively. What can we do? Raising the heap multiplier is an effective mitigation, as it increases the average hole size, but for it to be a comprehensive solution in e.g. the case of 16-byte live objects equally interspersed with holes, you would need a multiplier of 512× to ensure that the largest 8192-byte “small” objects will find a hole. I could live with 2× or something, but 512× is too much.

    We could consider changing the heap organization entirely. For example, most mark-sweep collectors (BDW-GC included) partition the heap into blocks whose allocations are of the same size, so you might have some blocks that only hold 16-byte allocations. It is theoretically possible to run into the same issue, though, if each block only has one live object, and the necessary multiplier that would “allow” for more empty blocks to be allocated is of the same order (256× for 4096-byte blocks each with a single 16-byte allocation, or even 4096× if your blocks are page-sized and you have 64kB pages).

    My conclusion is that practically speaking, if you can’t deal with fragmentation, then it is impossible to just rely on a heap multiplier to size your heap. It is certainly an error to live-lock the process, hoping that some other thread mutates the graph in such a way to free up a suitable hole. At the same time, if you have configured your heap to be growable at run-time, it would be bad policy to fail an allocation, just because you calculated that the heap is big enough already.

    It’s a shame, because we lose a mooring on reality: “how big will my heap get” becomes an unanswerable question because the heap might grow in response to fragmentation, which is not deterministic if there are threads around, and so we can’t reliably compare performance between different configurations. Ah well. If reliability is a goal, I think one needs to allow for evacuation, one way or another.

    for nofl?

    In this concrete case, I am still working on a solution. It’s going to be heuristic, which is a bit of a disappointment, but here we are.

    My initial thought has two parts. Firstly, if the heap is growable but cannot defragment, then we need to reserve some empty blocks after each collection, even if reserving them would grow the heap beyond the configured heap size multiplier. In that way we will always be able to allocate into the Nofl space after a collection, because there will always be some empty blocks. How many empties? Who knows. Currently Nofl blocks are 64 kB, and the largest “small object” is 8kB. I’ll probably try some constant multiplier of the heap size.

    The second thought is that searching through the entire heap for a hole is a silly way for the mutator to spend its time. Immix will reserve a block for overflow allocation : if a medium-sized allocation (more than 256B and less than 8192B) fails because no hole in the current block is big enough—note that Immix’s holes have 128B granularity—then the allocation goes to a dedicated overflow block , which is taken from the empty block set. This reduces fragmentation (holes which were not used for allocation because they were too small).

    Nofl should probably do the same, but given its finer granularity, it might be better to sweep over a variable number of blocks, for example based on the logarithm of the allocation size; one could instead sweep over clz(min-size)–clz(size) blocks before taking from the empty block list, which would at least bound the sweeping work of any given allocation.

    fin

    Welp, just wanted to get this out of my head. So far, my experience with this Nofl-based heap configuration is mostly colored by live-locks, and otherwise its implementation of a growable heap sizing policy seems to be more tight-fisted regarding memory allocation than BDW-GC’s implementation. I am optimistic though that I will be able to get precise tracing sometime soon, as measured in development time; the problem as always is fragmentation, in that I don’t have a hole in my calendar at the moment. Until then, sweep on Wayne, cons on Garth, onwards and upwards!

    • wifi_tethering open_in_new

      This post is public

      wingolog.org /archives/2025/05/22/whippet-lab-notebook-guile-heuristics-and-heap-growth

    • chevron_right

      Michael Meeks: 2025-05-21 Wednesday

      news.movim.eu / PlanetGnome • 21 May

    • Mail chew, early call, sync with Dave.
    • Published the next strip around how private initiative can smooth roads:
    • More catch-up, sales team call.
    • wifi_tethering open_in_new

      This post is public

      meeksfamily.uk /~michael/blog/2025-05-21.html

    • chevron_right

      Peter Hutterer: libinput and Lua plugins

      news.movim.eu / PlanetGnome • 21 May • 4 minutes

    First of all, what's outlined here should be available in libinput 1.29 but I'm not 100% certain on all the details yet so any feedback (in the libinput issue tracker) would be appreciated. Right now this is all still sitting in the libinput!1192 merge request. I'd specifically like to see some feedback from people familiar with Lua APIs. With this out of the way:

    Come libinput 1.29, libinput will support plugins written in Lua. These plugins sit logically between the kernel and libinput and allow modifying the evdev device and its events before libinput gets to see them.

    The motivation for this are a few unfixable issues - issues we knew how to fix but we cannot actually implement and/or ship the fixes without breaking other devices. One example for this is the inverted Logitech MX Master 3S horizontal wheel. libinput ships quirks for the USB/Bluetooth connection but not for the Bolt receiver. Unlike the Unifying Receiver the Bolt receiver doesn't give the kernel sufficient information to know which device is currently connected. Which means our quirks could only apply to the Bolt receiver (and thus any mouse connected to it) - that's a rather bad idea though, we'd break every other mouse using the same receiver. Another example is an issue with worn out mouse buttons - on that device the behavior was predictable enough but any heuristics would catch a lot of legitimate buttons. That's fine when you know your mouse is slightly broken and at least it works again. But it's not something we can ship as a general solution. There are plenty more examples like that - custom pointer deceleration, different disable-while-typing, etc.

    libinput has quirks but they are internal API and subject to change without notice at any time. They're very definitely not for configuring a device and the local quirk file libinput parses is merely to bridge over the time until libinput ships the (hopefully upstreamed) quirk.

    So the obvious solution is: let the users fix it themselves. And this is where the plugins come in. They are not full access into libinput, they are closer to a udev-hid-bpf in userspace. Logically they sit between the kernel event devices and libinput: input events are read from the kernel device, passed to the plugins, then passed to libinput. A plugin can look at and modify devices (add/remove buttons for example) and look at and modify the event stream as it comes from the kernel device. For this libinput changed internally to now process something called an "evdev frame" which is a struct that contains all struct input_events up to the terminating SYN_REPORT . This is the logical grouping of events anyway but so far we didn't explicitly carry those around as such. Now we do and we can pass them through to the plugin(s) to be modified.

    The aforementioned Logitech MX master plugin would look like this: it registers itself with a version number, then sets a callback for the "new-evdev-device" notification and (where the device matches) we connect that device's "evdev-frame" notification to our actual code:

    libinput:register(1) -- register plugin version 1
    libinput:connect("new-evdev-device", function (_, device)
        if device:vid() == 0x046D and device:pid() == 0xC548 then
            device:connect("evdev-frame", function (_, frame)
                for _, event in ipairs(frame.events) do
                    if event.type == evdev.EV_REL and 
                       (event.code == evdev.REL_HWHEEL or 
                        event.code == evdev.REL_HWHEEL_HI_RES) then
                        event.value = -event.value
                    end
                end
                return frame
            end)
        end
    end)
    
    This file can be dropped into /etc/libinput/plugins/10-mx-master.lua and will be loaded on context creation. I'm hoping the approach using named signals (similar to e.g. GObject) makes it easy to add different calls in future versions. Plugins also have access to a timer so you can filter events and re-send them at a later point in time. This is useful for implementing something like disable-while-typing based on certain conditions.

    So why Lua? Because it's very easy to sandbox. I very explicitly did not want the plugins to be a side-channel to get into the internals of libinput - specifically no IO access to anything. This ruled out using C (or anything that's a .so file, really) because those would run a) in the address space of the compositor and b) be unrestricted in what they can do. Lua solves this easily. And, as a nice side-effect, it's also very easy to write plugins in.[1]

    Whether plugins are loaded or not will depend on the compositor: an explicit call to set up the paths to load from and to actually load the plugins is required. No run-time plugin changes at this point either, they're loaded on libinput context creation and that's it. Otherwise, all the usual implementation details apply: files are sorted and if there are files with identical names the one from the highest-precedence directory will be used. Plugins that are buggy will be unloaded immediately.

    If all this sounds interesting, please have a try and report back any APIs that are broken, or missing, or generally ideas of the good or bad persuation. Ideally before we ship it and the API is stable forever :)

    [1] Benjamin Tissoires actually had a go at WASM plugins (via rust). But ... a lot of effort for rather small gains over Lua

    • wifi_tethering open_in_new

      This post is public

      who-t.blogspot.com /2025/05/libinput-and-lua-plugins.html

    • chevron_right

      Steven Deobald: Join the GNOME Board!

      news.movim.eu / PlanetGnome • 21 May • 6 minutes

    The past two weeks have gone by too quickly. I’ve had so many wonderful conversations with so many of our contributors and community friends at this point that I couldn’t be more confident that I’m in the right place. The next year is going to be a lot of fun and I’m looking forward to working with many more of you.

    This next year will be fun. But it is also going to be a lot of work . I’d like to take a minute to talk about some of the work we need to do, and how you can help. Because the 2025 election cycle has begun .

    ## Governance

    Being on the GNOME Foundation Board is, first and foremost, about governance. If you join the Board, you will probably do your onboarding with Rob. He’s good at it. He’ll walk you through a slide deck and the very first slide will contain this quote in 480pt font:

    Governance is not management.

    The full quote, from the second page of Kenneth Dayton’s book is:

    Governance is not management. In my opinion, one of the worst sins of charitable organizations is that too often they do not distinguish between the two. Rather, they confuse the two responsibilities and in the process hamper the mission of the institution.

    This is an important distinction. Joining the Board does not mean you’ll be managing staff. The Board exists to perform the functions you’ll see in the source material for the remaining 30 slides:

    These include (but are not limited to) responsibilities like:

    • Oversight
    • Making high-level decisions
    • Long-term strategy
    • Fiduciary responsibility
    • Ensuring the right ED is in place
    • Reviewing budgets
    • Deciding policy
    • Understanding the structure and duties of a 501(c)3 nonprofit

    Riveting, I know. But! This isn’t everything. And if you’re still awake, I will tell you that we don’t just need policy wonks and 50-year strategists. There’s some hands-dirty work associated with the GNOME Foundation Board, too. Even though the Board is not designed to manage , it can help execute . Once upon a time, the Board member were the people who scrambled to arrange GUADEC every year. It helps to be a little scrappy and we could learn to reclaim a bit of that history.

    ## Finance

    We could really use help with our finances. Our bookkeepers are absolutely metal and we’re very lucky to have them. Our Ops staff are super hard-working. But the Board rotates continuously. Not every Board will be able to read a balance sheet. We need to present the Board with a continuous view of finances — not just quarterly reports, but monthly reports which clearly describe our positions, our trajectory, and our best case / likely case / worst case. Financial clarity is paramount for an effective Board… but it’s also easier said than done.

    If you love spelunking through spreadsheets, tying together multiple data sources, documentation, automation, or doing a wee bit of data science and visualization on Small Data, the Board might be for you! Know that this effort will take time and patience, though. You won’t find clarity overnight. Determination — to the point of stubbornness, perhaps — will be your best ally, here.

    ## Executive

    In addition to finances, there are a number of committees in the Foundation . Some of these are Board committees and some are delegated committees. Committees, whether Board committees or not, all have some executive powers.

    The most obvious of these is the aptly-name Executive Committee. A lot of work gets done here and it’s safe to say meaningful participation in the Exec Committee is almost a daily commitment in 2025. But there are other committees that could certainly use help (or even rebooting) and don’t require a daily commitment as a volunteer:

    • Governance Committee (see Governance )
    • Finance Committee (see Finance )
    • Travel Committee
    • Internship Committee
    • etc.

    Committee participation doesn’t specifically require that you join the Board. It’s also possible for the Board to appoint Officers … and we’ll do that! But if you really want to get stuck in, I strongly suggest standing for election. It doesn’t matter if you’re a lawyer, an accountant, a designer, or developer.

    But beyond roles , who should stand for election? Let’s dig into that for a second.

    ## Representation

    The GNOME community is massive. I still have so many people to meet. And there seem to be people from every walk of life, every community, every corner of the planet, every cultural background, every linguistic background. If you’re one of the unlucky people I’ve forced one of my excited 3-hour sermons upon, you probably know by now I’m a huge fan of Network Resilience . I gave a keynote once that talks about this (among many other things). The Foundation, and the Board in particular, is more resilient when it’s more diverse. If we all think the same way, all have the same upbringing, same background, same weaknesses in our thinking… this weakens the Foundation.

    Someone recently told me they’ve thought about standing for elections for years but feel some Imposter Syndrome. I was shocked. This person has been contributing to GNOME for decades. They’re clearly very smart, hardworking, and talented. If such a strong candidate is just waiting there in the wings, I can’t help but believe there are other strong potential Board members just waiting for the right moment.

    The right moment is now.

    We have 2SLGBTQIA+ folks on the Board. We do have folks with disabilities, different language backgrounds, different continents. But we have only one woman (who also happens to be our only lawyer). We have no one from east of Europe. We can do better. I’m not worried any of you are going to apply to the Board because of your background. I worry that you, like my new friend mentioned above, might fight the temptation to stand for election because you think you don’t belong here.

    You absolutely belong here. Bring your talent, your energy, your tenacity. Help us kick ass.

    ## No Nonsense

    I’d like to take a minute to discuss some of the reasons not to join the Board.

    First, we need Board members who will help paddle our wee canoe in the same direction. GNOME, both the Project and the Foundation, have experienced some real hardship in recent years. If it’s your intention to join the Board only so you can paddle sideways (or backwards), please put that thought on hold for a couple years. We just can’t support that level of disruption right now. We will have our disagreements but we all need to be on the same page.

    Second, if everything above bores you but you have a single issue that you’re really dreaming of hammering home… that’s also a recipe for ineffective Board participation. If elected, you will be on the Board for TWO YEARS. That’s a long time. Make sure you want to be there to work with us for the entire two years. It will be hard work. And it will be worth it.

    ## Hard Work

    While talking to Maria ( @marimaj ) today — one of my favourite contributor conversations so far — she said something that resonated with me: “Almost no one appreciates just how much work is required! The Foundation requires so, so much work to function properly.” She then proceeded to name a bunch of folks who contribute day and night. Her hard-won understanding is built upon experience. She’s been a part of the community for a long time, and she’s seen how hard staff and volunteers work to keep the Foundation moving.

    If you want to know how much work the Foundation requires, ask her. Ask me. Ask any Board member.

    If you want to put in the work, I want you to stand for election. I’m looking forward to seeing your nomination. 🙂

    To announce your candidacy, read Andrea’s post and follow the instructions . (Thanks Andrea!)

    • wifi_tethering open_in_new

      This post is public

      blogs.gnome.org /steven/2025/05/21/join-the-gnome-board/

    • chevron_right

      Christian Hergert: Simplifying LSP Selection

      news.movim.eu / PlanetGnome • 20 May

    With Foundry I want to make LSP management much easier than it currently is in Builder .

    We have the foundry lsp run python3 command where python3 can be replaced with any language for which there is an installed LSP plugin. This will start an LSP using all the abstractions (including cross-container execution) and provide it via stdin / stdout .

    But what happens when you have a half-dozen language servers for Python with new ones added every week? There is a simple builtin tool now.

    Keep in mind the language identifiers should match GtkSourceView language identifiers.

    # Make clangd the preferred LSP for C
    foundry lsp prefer clangd c
    
    # Make sourcekit-lsp preferred LSP for C++
    foundry lsp prefer sourcekit-lsp cpp
    
    # Make ruff the preferred LSP for Python3
    foundry lsp prefer ruff python3
    

    If there is a clear LSP that your project should be using by all contributors, add --project and it will update the value in the projects settings.

    • wifi_tethering open_in_new

      This post is public

      blogs.gnome.org /chergert/2025/05/20/simplifying-lsp-selection/