Language Selection

English French German Italian Portuguese Spanish

Gnu Planet

Syndicate content
Planet GNU - https://planet.gnu.org/
Updated: 13 hours 4 min ago

Parabola GNU/Linux-libre: [From Arch] libbloom>=1.6-2 update requires manual intervention

Saturday 13th of July 2019 10:00:00 PM

The libbloom package prior to version 1.6-2 was missing a soname link. This has been fixed in 1.6-2, so the upgrade will need to overwrite the untracked soname link created by ldconfig. If you get an error

libbloom: /usr/lib/libbloom.so.1 exists in filesystem

when updating, use

pacman -Suy --overwrite usr/lib/libbloom.so.1

to perform the upgrade.

rush @ Savannah: Version 2.1

Friday 12th of July 2019 07:45:17 PM

Version 2.1 is available for download from GNU and Puszcza archives.

This version fixes several minor bugs that appeared in previous release 2.0.

GNU Guix: Towards Guix for DevOps

Friday 12th of July 2019 07:00:00 PM

Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to Guix. Since May, I've been working on a DevOps automation tool for the Guix System, which we've been calling guix deploy.

The idea for a Guix DevOps tool has been making rounds on the mailing lists for some time now. Years, in fact; Dave Thompson and Chris Webber put together a proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze upon the existing tools for this sort of thing -- Ansible, NixOps -- and fantasize about a similar tool, albeit with the expressive power of Guile scheme and the wonderful system configuration facilities of Guix. And now, those fantasies are becoming a reality.

"DevOps" is a term that might be unfamiliar to a fair number of Guix users. I'll spare you the detour to Wikipedia and give a brief explanation of what guix deploy does.

Imagine that you've spent the afternoon playing around with Guile's (web) module, developing software for a web forum. Awesome! But a web forum with no users is pretty boring, so you decide to shell out a couple bucks for a virtual private server to run your web forum. You feel that Wildebeest admirers on the internet deserve a platform of their own for discussion, and decide to dedicate the forum to that.

As it turns out, C. gnou is a more popular topic than you ever would have imagined. Your web forum soon grows in size -- attracting hundreds of thousands of simultaneous users. Despite Guile's impressive performance characteristics, one lowly virtual machine is too feeble to support such a large population of Wildebeest fanatics. So you decide to use Apache as a load-balancer, and shell out a couple more bucks for a couple more virtual private servers. Now you've got a problem on your hands; you're the proud owner of five or so virtual machines, and you need to make sure they're all running the most recent version of either your web forum software or Apache.

This is where guix deploy comes into play. Just as you'd use an operating-system declaration to configure services and user accounts on a computer running the Guix System, you can now use that same operating-system declaration to remotely manage any number of machines. A "deployment" managing your Wildebeest fan site setup might look something like this:

... ;; Service for our hypothetical guile web forum application. (define guile-forum-service-type (service-type (name 'guile-forum) (extensions (list (service-extension shepherd-root-service-type guile-forum-shepherd-service) (service-extension account-service-type (const %guile-forum-accounts)))) (default-value (guile-forum-configuration)) (description "A web forum written in GNU Guile."))) ... (define %forum-server-count 4) (define (forum-server n) (operating-system (host-name (format #f "forum-server-~a" n)) ... (services (append (list (service guile-forum-service-type (guile-forum-configuration "GNU Fan Forum!"))) %base-services)))) (define load-balancer-server (operating-system (host-name "load-balancer-server" ... (services (append (list (service httpd-service-type (httpd-configuration ...))) %base-services))))) ;; One machine running our load balancer. (cons (machine (system load-balancer-server) (environment manged-host-environment-type) (configuration (machine-ssh-configuration ...))) ;; And a couple running our forum software! (let loop ((n 1) (servers '())) (if (> n %forum-server-count) servers (loop (1+ n) (cons (machine (system (forum-server n)) (environment manged-host-environment-type) (configuration (machine-ssh-configuration ...))) servers)))))

The take-away from that example is that there's a new machine type atop the good ol' operating-system type, specifying how the machine should be provisioned. The version of guix deploy that's currently on the master branch only supports managed-host-environment-type, which is used for machines that are already up and running the Guix System. Provisioning, in that sense, only really involves opening an SSH connection to the host. But I'm sure you can imagine a linode-environment-type which automatically sets up a virtual private server through Linode, or a libvirt-environment-type that spins up a virtual machine for running your services. Those types are what I'll be working on in the coming months, in addition to cleaning up the code that's there now.

And yes, you did read that right. guix deploy is on the Guix master branch right now! In fact, we've already done a successful deployment right here on ci.guix.gnu.org. So, if this sounds as though it'd be up your alley, run guix pull, crack open the manual, and let us know how it goes!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

GNU Guix: Towards Guix for DevOps

Friday 12th of July 2019 05:00:00 PM

Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to Guix. Since May, I've been working on a DevOps automation tool for the Guix System, which we've been calling guix deploy.

The idea for a Guix DevOps tool has been making rounds on the mailing lists for some time now. Years, in fact; Dave Thompson and Chris Webber put together a proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze upon the existing tools for this sort of thing -- Ansible, NixOps -- and fantasize about a similar tool, albeit with the expressive power of Guile scheme and the wonderful system configuration facilities of Guix. And now, those fantasies are becoming a reality.

"DevOps" is a term that might be unfamiliar to a fair number of Guix users. I'll spare you the detour to Wikipedia and give a brief explanation of what guix deploy does.

Imagine that you've spent the afternoon playing around with Guile's (web) module, developing software for a web forum. Awesome! But a web forum with no users is pretty boring, so you decide to shell out a couple bucks for a virtual private server to run your web forum. You feel that Wildebeest admirers on the internet deserve a platform of their own for discussion, and decide to dedicate the forum to that.

As it turns out, C. gnou is a more popular topic than you ever would have imagined. Your web forum soon grows in size -- attracting hundreds of thousands of simultaneous users. Despite Guile's impressive performance characteristics, one lowly virtual machine is too feeble to support such a large population of Wildebeest fanatics. So you decide to use Apache as a load-balancer, and shell out a couple more bucks for a couple more virtual private servers. Now you've got a problem on your hands; you're the proud owner of five or so virtual machines, and you need to make sure they're all running the most recent version of either your web forum software or Apache.

This is where guix deploy comes into play. Just as you'd use an operating-system declaration to configure services and user accounts on a computer running the Guix System, you can now use that same operating-system declaration to remotely manage any number of machines. A "deployment" managing your Wildebeest fan site setup might look something like this:

... ;; Service for our hypothetical guile web forum application. (define guile-forum-service-type (service-type (name 'guile-forum) (extensions (list (service-extension shepherd-root-service-type guile-forum-shepherd-service) (service-extension account-service-type (const %guile-forum-accounts)))) (default-value (guile-forum-configuration)) (description "A web forum written in GNU Guile."))) ... (define %forum-server-count 4) (define (forum-server n) (operating-system (host-name (format #f "forum-server-~a" n)) ... (services (append (list (service guile-forum-service-type (guile-forum-configuration "GNU Fan Forum!"))) %base-services)))) (define load-balancer-server (operating-system (host-name "load-balancer-server" ... (services (append (list (service httpd-service-type (httpd-configuration ...))) %base-services))))) ;; One machine running our load balancer. (cons (machine (system load-balancer-server) (environment manged-host-environment-type) (configuration (machine-ssh-configuration ...))) ;; And a couple running our forum software! (let loop ((n 1) (servers '())) (if (> n %forum-server-count) servers (loop (1+ n) (cons (machine (system (forum-server n)) (environment manged-host-environment-type) (configuration (machine-ssh-configuration ...))) servers)))))

The take-away from that example is that there's a new machine type atop the good ol' operating-system type, specifying how the machine should be provisioned. The version of guix deploy that's currently on the master branch only supports managed-host-environment-type, which is used for machines that are already up and running the Guix System. Provisioning, in that sense, only really involves opening an SSH connection to the host. But I'm sure you can imagine a linode-environment-type which automatically sets up a virtual private server through Linode, or a libvirt-environment-type that spins up a virtual machine for running your services. Those types are what I'll be working on in the coming months, in addition to cleaning up the code that's there now.

And yes, you did read that right. guix deploy is on the Guix master branch right now! In fact, we've already done a successful deployment right here on ci.guix.gnu.org. So, if this sounds as though it'd be up your alley, run guix pull, crack open the manual, and let us know how it goes!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

FSF Events: John Sullivan - " 'Just don't buy it': Consumer choices in free software activism" (Curitiba, Brazil)

Friday 12th of July 2019 09:55:45 AM

FSF executive director John Sullivan will be giving his speech “‘Just don't buy it’: Consumer choices in free software activism” at DebConf 19 (2019-07-21–28):

Movement activism often focuses on economic decisions. Buy this ethically made product; don't buy that one made by a company that funds terrible things. In free software, we encourage people to boycott (for example) Microsoft, and to instead support companies who sell machines with GNU/Linux.

It's an intuitive idea that, as individuals wanting to make the world better, we should use our willingness to spend or not spend money to reward those who do right and punish those who do wrong. Throughout history, this has sometimes been effective. But how effective? Can it be dangerous?

There is a danger of reducing activism and social change strategy to these decisions. We see this in the free software movement, when some activist campaigns aimed at persuading people to stop using proprietary software are met with responses like, “If you don't like Apple products, just don't buy them. Help make free products that are better than theirs. Why campaign against them?” or “How can you criticize proprietary software but still drive a car that has it?”

As an advocate, have you ever heard these responses, or felt like a hypocrite, or stumbled trying to explain to others why the situation is more complicated than “just don't buy it”?

How do we form a holistic movement strategy for advancing user freedom that takes consumer activism as far as possible, without overprioritizing it?

I hope those interested in effectively fighting for user freedom will join me as I share thoughts formed from 16 years of experience working on the Free Software Foundation's advocacy efforts, against the backdrop of some highlights from the history of other social movements.

Location: Auditรณrio, Av. Sete de Setembro, 3165 Rebouรงas, Curitiba, PR 80.230-901, Brazil

Please fill out our contact form, so that we can contact you about future events in and around Curitiba.

FSF Events: John Sullivan - "The Free Software Foundation and Debian" (Curitiba, Brazil)

Friday 12th of July 2019 09:40:00 AM

FSF executive director John Sullivan will be giving his speech “The Free Software Foundation and Debian” at DebConf 19 (2019-07-21–28):

Debian and the Free Software Foundation, along with its GNU Project, share many goals and ideals. They are two of the most mature and dedicated organizations working in the free software movement. This is an annual opportunity to talk in person about how they can better work together and inspire each other.

FSF members (and potential members) will meet to talk about the FSF’s work, the FSF’s relationship with Debian, and progress of the movement as a whole:

  • A very brief update by the FSF’s executive director on highlights of the FSF’s work since the last DebConf
  • Feedback (positive and negative) from members, and a little brainstorming about what the FSF should and shouldn’t be doing
  • A review of the current relationship between the FSF and Debian
  • A discussion of areas for collaboration, focused on projects that could be completed or significantly advanced before the next DebConf

Recognizing that we can’t cover everything in 45 minutes, at the beginning, we’ll choose some specific topics to focus on under this general umbrella.

Past sessions at DebConf have led to concrete positive results like collaboration on the hardware database at https://h-node.org/. Let’s have another productive one!

This session will not be recorded, in order to make participants feel more comfortable speaking informally.

Location: Sala de Videoconferencia, Av. Sete de Setembro, 3165 Rebouças, Curitiba, PR 80.230-901, Brazil

Please fill out our contact form, so that we can contact you about future events in and around Curitiba.

FSF Events: Meetup - with FSF staff Donald Robertson, III, and Craig Topham (Portland, OR)

Thursday 11th of July 2019 06:55:00 PM

Come join us on July 18 at a social gathering for free software enthusiasts! Free Software Foundation (FSF) licensing and compliance manager, Donald R. Robertson, and copyright and licensing associate, Craig Topham will be visiting Portland, OR on July 18, and they'll be hosting an informal meetup to show our appreciation for your support of the FSF's work. This event will give you an opportunity to meet other FSF associate members and supporters and find out what the FSF is currently working on. We are curious to hear your thoughts, and/or answer any questions you may have.

This is an informal gathering for anyone who is interested in participating in the free software community or wants to learn more about the FSF; you don't have to be a current member to attend.

The FSF will be providing appetizers, including vegan- and vegetarian-friendly options.

Location: Loyal Legion Oregon Beer Hall, 710 SE 6th Avenue Portland, OR 97214

RSVP: It would be great if you could let us know by Wednesday July 17, if you're coming to the meetup, but you're also welcome to just drop in. Contact ZoĂŤ Kooyman at campaigns@fsf.org, to RSVP, or if you have any questions.

Please fill out our contact form, so that we can contact you about future events in Portland area.

FSF Blogs: June 2019: Photos from Brno

Tuesday 9th of July 2019 03:39:21 PM

Free Software Foundation president Richard Stallman (RMS) was in Brno, Czech Republic on June 6, 2019, to give two speeches.

In the morning, he took part in the URBIS Smart City Fair, at the Brno Fair Grounds, giving his speech "Computing, freedom, and privacy."1

(Copyright © 2019 Veletrhy Brno, a. s. Photos licensed under CC BY 4.0.)

In the afternoon, at the University Cinema Scala, he gave his speech "The free software movement and the GNU/Linux operating system," to about three hundred people.

(Copyright © 2019 Pavel Loutocký. Photos licensed under CC BY 4.0.)

(Copyright © 2019 Pavel Loutocký. Photos licensed under CC BY 4.0.)

Thank you to everyone who made this visit possible!

If you're in the area, please fill out our contact form, so that we can inform you about future events in and around Brno.

Please see www.fsf.org/events for a full list of all of RMS's confirmed engagements,
and contact rms-assist@gnu.org if you'd like him to come speak.

1. The recording will soon be posted on our audio-video archive.

Christopher Allan Webber: Racket is an acceptable Python

Tuesday 9th of July 2019 02:27:00 PM

A little over a decade ago, there were some popular blogposts about whether Ruby was an acceptable Lisp or whether even Lisp was an acceptable Lisp. Peter Norvig was also writing at the time introducing Python to Lisp programmers. Lisp, those in the know knew, was the right thing to strive for, and yet seemed unattainable for anything aimed for production since the AI Winter shattered Lisp's popularity in the 80s/early 90s. If you can't get Lisp, what's closest thing you can get?

This was around the time I was starting to program; I had spent some time configuring my editor with Emacs Lisp and loved every moment I got to do it; I read some Lisp books and longed for more. And yet when I tried to "get things done" in the language, I just couldn't make as much headway as I could with my preferred language for practical projects at the time: Python.

Python was great... mostly. It was easy to read, it was easy to write, it was easy-ish to teach to newcomers. (Python's intro material is better than most, but my spouse has talked before about some major pitfalls that the Python documentation has which make getting started unnecessarily hard. You can hear her talk about that at this talk we co-presented on at last year's RacketCon.) I ran a large free software project on a Python codebase, and it was easy to get new contributors; the barrier to entry to becoming a programmer with Python was low. I consider that to be a feature, and it certainly helped me bootstrap my career.

Most importantly of all though, Python was easy to pick up and run with because no matter what you wanted to do, either the tools came built in or the Python ecosystem had enough of the pieces nearby that building what you wanted was usually fairly trivial.

But Python has its limitations, and I always longed for a lisp. For a brief time, I thought I could get there by contributing to the Hy project, which was a lisp that transformed itself into the Python AST. "Why write Python in a syntax that's easy to read when you could add a bunch of parentheses to it instead?" I would joke when I talked about it. Believe it or not though, I do consider lisps easier to read, once you are comfortable to understand their syntax. I certainly find them easier to write and modify. And I longed for the metaprogramming aspects of Lisp.

Alas, Hy didn't really reach my dream. That macro expansion made debugging a nightmare as Hy would lose track of where the line numbers are; it wasn't until that when I really realized that without line numbers, you're just lost in terms of debugging in Python-land. That and Python didn't really have the right primitives; immutable datastructures for whatever reason never became first class, meaning that functional programming was hard, "cons" didn't really exist (actually this doesn't matter as much as people might think), recursive programming isn't really as possible without tail call elimination, etc etc etc.

But I missed parentheses. I longed for parentheses. I dreamed in parentheses. I'm not kidding, the only dreams I've ever had in code were in lisp, and it's happened multiple times, programs unfolding before me. The structure of lisp makes the flow of code so clear, and there's simply nothing like the comfort of developing in front of a lisp REPL.

Yet to choose to use a lisp seemed to mean opening myself up to eternal yak-shaving of developing packages that were already available on the Python Package Index or limiting my development community an elite group of Emacs users. When I was in Python, I longed for the beauty of a Lisp; when I was in a Lisp, I longed for the ease of Python.

All this changed when I discovered Racket:

  • Racket comes with a full-featured editor named DrRacket built-in that's damn nice to use. It has all the features that make lisp hacking comfortable previously mostly only to Emacs users: parenthesis balancing, comfortable REPL integration, etc etc. But if you want to use Emacs, you can use racket-mode. Win-win.
  • Racket has intentionally been built as an educational language, not unlike Python. One of the core audiences of Racket is middle schoolers, and it even comes with a built-in game engine for kids. (The How to Design Programs prologue might give you an introductory taste, and Realm of Racket is a good book all about learning to program by building Racket games.)
  • My spouse and I even taught classes about how to learn to program for humanities academics using Racket. We found the age-old belief that "lisp syntax is just too hard" is simply false; the main thing that most people lack is decent lisp-friendly tooling with a low barrier to entry, and DrRacket provides that. The only people who were afraid of the parentheses turned out to be people who already knew how to program. Those who didn't even praised the syntax for its clarity and the way the editor could help show you when you made a syntax error (DrRacket is very good at that). "Lisp is too hard to learn" is a lie; if middle schoolers can learn it, so can more seasoned programmers.
  • Racket might even be more batteries included than Python. At least all the batteries that come included are generally nicer; Racket's GUI library is the only time I've ever had fun in my life writing GUI programs (and they're cross platform too). Constructing pictures with its pict library is a delight. Plotting graphs with plot is an incredible experience. Writing documentation with Scribble is the best non-org-mode experience I've ever had, but has the advantage over org-mode in that your document is just inverted code. I could go on. And these are just some packages bundled with Racket; the Package repository contains much more.
  • Racket's documentation is, in my experience, unparalleled. The Racket Guide walks you through all the key concepts, and the Racket Reference has everything else you need.
  • The tutorials are also wonderful; the introductory tutorial gets your feet wet not through composing numbers or strings but by building up pictures. Want to learn more? The next two tutorials show you how to build web applications and then build your own web server.
  • Like Python, even though Racket has its roots in education, it is more than ready for serious practical use. These days, when I want to build something and get it done quickly and efficiently, I reach for Racket first.

Racket is a great Lisp, but it's also an acceptable Python. Sometimes you really can have it all.

FSF Blogs: Thank you for advancing free software: Read FSF spring news in the latest Bulletin

Tuesday 9th of July 2019 04:55:00 AM

Our Bulletin highlights some important activities and issues in free software over the last six months, including:

It highlights some important activities and issues in free software over the last six months, including:

  • an educational program we launched, together with free software activist Devin Ulibarri, where we used the program Music Blocks to teach Boston area public school youth about coding and free software, and then proceeded to donate ten fully freed laptops to the schools we visited;

  • some ideas on how the free software community can do better to bring visibility to our movement, according to LibrePlanet 2019 conference speaker and free software activist Mary Kate Fain, pulled from her superb LibrePlanet talk “Sparking change: What free software can learn from social justice movements”; and

  • our licensing and compliance manager, Donald Robertson, III, reports on the progress of our Respects Your Freedom program, with further explanation on the parameters needed for the certification of hardware devices that meet FSF's criteria for protecting the rights of users.

Thirty-five volunteers joined FSF staff over the course of three days to get all the Bulletins stuffed in envelopes and mailed out. This was a great opportunity to catch up on free software issues with some of our most dedicated free software enthusiasts here in Boston. We are grateful to have such a strong core of supporters that keep the movement growing, and thanks to your generous contribution, we will be even stronger.

Please be vocal about your support for free software. Read and share the Bulletin articles online using the #ISupportFreeSoftware hashtag, use our fundraiser support images, and talk to your community about why you support the FSF. It makes a difference.

Throughout our spring fundraiser, we have been enjoying both the public posts from supporters using the hashtag on social media, as well as answers to the "What inspired you to join today?" question we ask new members. Here are some of our favorites.

  • We see many excited calls for user freedom and user control:
    • "For freedom!"
    • "Does the software you use grant you the freedom to redistribute copies so you can help others? #FreeSoftware does. Learn more by listening to Richard M. Stallman, on the Making Better podcast. #ISupportFreeSoftware" from @makingbetterpod
    • @mmaug shared: "There are alternatives to Facebook, Microsoft, Google, and Twitter. Take control of your privacy, and your computer! #ISupportFreeSoftware #FSF"
    • @globalspectator posted: "Software you don't control seizes control over you. Help @fsf break the chains of proprietary software for a freer future #ISupportFreeSoftware"
  • We receive humbling thank you's from people appreciating our work, naming the LibrePlanet conference, our licensing work, and the GNU Project:

    • "Wanted to for a long time. Also, LibrePlanet!"
    • "Gratitude. And a long lasting debt feeling. Thanks so much!"
    • "As a software developer and GNU/Linux user I want to set a statement and do my part in keeping free software popular"
    • "I use GNU tools, and have since the beginning"
  • And most importantly, we hear from people who have come across our activities and campaign images online, or who were informed about FSF through their on- and offline community and decided to take action -- convincing us that people inspiring each other to join the Free Software Supporter mailing list, or to become an associate member is by far the most powerful way to expand our reach and strengthen our message:

    • "I watched an interview with Richard Stallman"
    • "A friend"
    • "A post in the 'Victorhck in the free world' blog"
    • @opc_5 called for digital liberation through free software: "Software Libre para la liberación digital. #ISupportFreeSoftware"
    • @AugustinPMichel tagged his connections with: "#ISupportFreeSoftware, do you?"

Today, we have one week left in our spring fundraiser, and we are confident we will achieve our membership goal of 200 members in 28 days if we keep at it. With your help, we may engage enough people to also reach 400 donations before the 15th of July. Your support helped get us where we are, in position to succeed. Your generosity and outspokenness fuel our message, increase our reach, and will allow us to continue to advocate on your behalf.

Thank you for your contribution to free software.

Aleksander Morgado: DW5821e firmware update integration in ModemManager and fwupd

Wednesday 3rd of July 2019 01:27:10 PM

The Dell Wireless 5821e module is a Qualcomm SDX20 based LTE Cat16 device. This modem can work in either MBIM mode or QMI mode, and provides different USB layouts for each of the modes. In Linux kernel based and Windows based systems, the MBIM mode is the default one, because it provides easy integration with the OS (e.g. no additional drivers or connection managers required in Windows) and also provides all the features that QMI provides through QMI over MBIM operations.

The firmware update process of this DW5821e module is integrated in your GNU/Linux distribution, since ModemManager 1.10.0 and fwupd 1.2.6. There is no official firmware released in the LVFS (yet) but the setup is completely ready to be used, just waiting for Dell to publish an initial official firmware release.

The firmware update integration between ModemManager and fwupd involves different steps, which I’ll try to describe here so that it’s clear how to add support for more devices in the future.

1) ModemManager reports expected update methods, firmware version and device IDs

The Firmware interface in the modem object exposed in DBus contains, since MM 1.10, a new UpdateSettings property that provides a bitmask specifying which is the expected firmware update method (or methods) required for a given module, plus a dictionary of key-value entries specifying settings applicable to each of the update methods.

In the case of the DW5821e, two update methods are reported in the bitmask: “fastboot” and “qmi-pdc“, because both are required to have a complete firmware upgrade procedure. “fastboot” would be used to perform the system upgrade by using an OTA update file, and “qmi-pdc” would be used to install the per-carrier configuration files after the system upgrade has been done.

The list of settings provided in the dictionary contain the two mandatory fields required for all devices that support at least one firmware update method: “device-ids” and “version”. These two fields are designed so that fwupd can fully rely on them during its operation:

  • The “device-ids” field will include a list of strings providing the device IDs associated to the device, sorted from the most specific to the least specific. These device IDs are the ones that fwupd will use to build the GUIDs required to match a given device to a given firmware package. The DW5821e will expose four different device IDs:
    • “USB\VID_413C“: specifying this is a Dell-branded device.
    • “USB\VID_413C&PID_81D7“: specifying this is a DW5821e module.
    • “USB\VID_413C&PID_81D7&REV_0318“: specifying this is hardware revision 0x318 of the DW5821e module.
    • “USB\VID_413C&PID_81D7&REV_0318&CARRIER_VODAFONE“: specifying this is hardware revision 0x318 of the DW5821e module running with a Vodafone-specific carrier configuration.
  • The “version” field will include the firmware version string of the module, using the same format as used in the firmware package files used by fwupd. This requirement is obviously very important, because if the format used is different, the simple version string comparison used by fwupd (literally ASCII string comparison) would not work correctly. It is also worth noting that if the carrier configuration is also versioned, the version string should contain not only the version of the system, but also the version of the carrier configuration. The DW5821e will expose a firmware version including both, e.g. “T77W968.F1.1.1.1.1.VF.001” (system version being F1.1.1.1.1 and carrier config version being “VF.001”)
  • In addition to the mandatory fields, the dictionary exposed by the DW5821e will also contain a “fastboot-at” field specifying which AT command can be used to switch the module into fastboot download mode.
2) fwupd matches GUIDs and checks available firmware versions

Once fwupd detects a modem in ModemManager that is able to expose the correct UpdateSettings property in the Firmware interface, it will add the device as a known device that may be updated in its own records. The device exposed by fwupd will contain the GUIDs built from the “device-ids” list of strings exposed by ModemManager. E.g. for the “USB\VID_413C&PID_81D7&REV_0318&CARRIER_VODAFONE” device ID, fwupd will use GUID “b595e24b-bebb-531b-abeb-620fa2b44045”.

fwupd will then be able to look for firmware packages (CAB files) available in the LVFS that are associated to any of the GUIDs exposed for the DW5821e.

The CAB files packaged for the LVFS will contain one single firmware OTA file plus one carrier MCFG file for each supported carrier in the give firmware version. The CAB files will also contain one “metainfo.xml” file for each of the supported carriers in the released package, so that per-carrier firmware upgrade paths are available: only firmware updates for the currently used carrier should be considered. E.g. we don’t want users running with the Vodafone carrier config to get notified of upgrades to newer firmware versions that aren’t certified for the Vodafone carrier.

Each of the CAB files with multiple “metainfo.xml” files will therefore be associated to multiple GUID/version pairs. E.g. the same CAB file will be valid for the following GUIDs (using Device ID instead of GUID for a clearer explanation, but really the match is per GUID not per Device ID):

  • Device ID “USB\VID_413C&PID_81D7&REV_0318&CARRIER_VODAFONE” providing version “T77W968.F1.2.2.2.2.VF.002”
  • Device ID “USB\VID_413C&PID_81D7&REV_0318&CARRIER_TELEFONICA” providing version “T77W968.F1.2.2.2.2.TF.003”
  • Device ID “USB\VID_413C&PID_81D7&REV_0318&CARRIER_VERIZON” providing version “T77W968.F1.2.2.2.2.VZ.004”
  • … and so on.

Following our example, fwupd will detect our device exposing device ID “USB\VID_413C&PID_81D7&REV_0318&CARRIER_VODAFONE” and version “T77W968.F1.1.1.1.1.VF.001” in ModemManager and will be able to find a CAB file for the same device ID providing a newer version “T77W968.F1.2.2.2.2.VF.002” in the LVFS. The firmware update is possible!

3) fwupd requests device inhibition from ModemManager

In order to perform the firmware upgrade, fwupd requires full control of the modem. Therefore, when the firmware upgrade process starts, fwupd will use the new InhibitDevice(TRUE) method in the Manager DBus interface of ModemManager to request that a specific modem with a specific uid should be inhibited. Once the device is inhibited in ModemManager, it will be disabled and removed from the list of modems in DBus, and no longer used until the inhibition is removed.

The inhibition may be removed by calling InhibitDevice(FALSE) explicitly once the firmware upgrade is finished, and will also be automatically removed if the program that requested the inhibition disappears from the bus.

4) fwupd downloads CAB file from LVFS and performs firmware update

Once the modem is inhibited in ModemManager, fwupd can right away start the firmware update process. In the case of the DW5821e, the firmware update requires two different methods and two different upgrade cycles.

The first step would be to reboot the module into fastboot download mode using the AT command specified by ModemManager in the “at-fastboot” entry of the “UpdateSettings” property dictionary. After running the AT command, the module will reset itself and reboot with a completely different USB layout (and different vid:pid) that fwupd can detect as being the same device as before but in a different working mode. Once the device is in fastboot mode, fwupd will download and install the OTA file using the fastboot protocol, as defined in the “flashfile.xml” file provided in the CAB file:

<parts interface="AP"> <part operation="flash" partition="ota" filename="T77W968.F1.2.2.2.2.AP.123_ota.bin" MD5="f1adb38b5b0f489c327d71bfb9fdcd12"/> </parts>

Once the OTA file is completely downloaded and installed, fwupd will trigger a reset of the module also using the fastboot protocol, and the device will boot from scratch on the newly installed firmware version. During this initial boot, the module will report itself running in a “default” configuration not associated to any carrier, because the OTA file update process involves fully removing all installed carrier-specific MCFG files.

The second upgrade cycle performed by fwupd once the modem is detected again involves downloading all carrier-specific MCFG files one by one into the module using the QMI PDC protocol. Once all are downloaded, fwupd will activate the specific carrier configuration that was previously active before the download was started. E.g. if the module was running with the Vodafone-specific carrier configuration before the upgrade, fwupd will select the Vodafone-specific carrier configuration after the upgrade. The module would be reseted one last time using the QMI DMS protocol as a last step of the upgrade procedure.

5) fwupd removes device inhibition from ModemManager

The upgrade logic will finish by removing the device inhibition from ModemManager using InhibitDevice(FALSE) explicitly. At that point, ModemManager would re-detect and re-probe the modem from scratch, which should already be running in the newly installed firmware and with the newly selected carrier configuration.

rush @ Savannah: Version 2.0

Monday 1st of July 2019 08:15:26 AM

Version 2.0 is available for download from GNU and Puszcza archives.

This release features a complete rewrite of the configuration support. It introduces a new configuration file syntax that offers a large set of control structures and transformation instructions for handling arbitrary requests.  Please see the documentation for details.

Backward compatibility with prior releases is retained and old configuration syntax is still supported.  This ensures that existing installations will remain operational without any changes. Nevertheless, system administrators are encouraged to switch to the new syntax as soon as possible.

GNU Guile: GNU Guile 2.2.6 released

Sunday 30th of June 2019 09:55:00 PM

We are pleased to announce GNU Guile 2.2.6, the sixth bug-fix release in the new 2.2 stable release series. This release represents 11 commits by 4 people since version 2.2.5. First and foremost, it fixes a regression introduced in 2.2.5 that would break Guile’s built-in HTTP server.

See the release announcement for details.

trans-coord @ Savannah: Malayalam team re-established

Saturday 29th of June 2019 06:54:45 AM

After more than 8 years of being orphaned, Malayalam team is active again.  The new team leader, Aiswarya Kaitheri Kandoth, made a new translation of the Free Software Definition, so now we have 41 translations of that page!

Currently, Malayalam the only active translation team of official languages of India.  It is a Dravidian language spoken by about 40 million people worldwide, with the most speakers living in the Indian state of Kerala.  Like many Indian languages, it uses a syllabic script derived from Brahmi.

Links to up-to-date translations are shown on the automatically generated report page.

Christopher Allan Webber: How do Spritely's actor and storage layers tie together?

Thursday 27th of June 2019 06:15:00 PM

I've been hacking away at Spritely (see previously). Recently I've been making progress on both the actor model (goblins and its rewrite goblinoid) as well as the storage layers (currently called Magenc and Crystal, but we are talking about probably renaming the both of them into a suite called "datashards"... yeah, everything is moving and changing fast right now.)

In the #spritely channel on freenode a friend asked, what is the big picture idea here? Both the actor model layer and the storage layer describe themselves as using "capabilities" (or more precisely "object capabilities" or "ocaps") but they seem to be implemented differently. How does it all tie together?

A great question! I think the first point of confusion is that while both follow the ocap paradigm (which is to say, reference/possession-based authority... possessing the capability gives you access, and it does not matter what your identity is for the most part for access control), they are implemented very differently because they are solving different problems. The storage system is based on encrypted, persistent data, with its ideas drawn from Tahoe-LAFS and Freenet, and the way that capabilities work is based on possession of cryptographic keys (which are themselves embedded/referenced in the URIs). The actor model, on the other hand, is based on holding onto a reference to a unique, unguessable URL (well, that's a bit of an intentional oversimplification for the sake of this explaination but we'll run with it) where the actor at that URL is "live" and communicated with via message passing. (Most of the ideas from this come from E and Waterken.) Actors are connected to each other over secure channels to prevent eavesdropping or leakage of the capabilities.

So yeah, how do these two seemingly very different layers tie together? As usual, I find that I most easily explain things via narrative, so let's imagine the following game scenario: Alice is in a room with a goblin. First Alice sees the goblin, then Alice attacks the goblin, then the goblin and Alice realize that they are not so different and become best friends.

The goblin and Alice both manifest in this universe as live actors. When Alice walks into the room (itself an actor), the room gives Alice a reference to the goblin actor. To "see" the goblin, Alice sends a message to it asking for its description. It replies with its datashards storage URI with its 3d model and associated textures. Alice can now query the storage system to reconstruct these models and textures from the datashards storage systems she uses. (The datashards storage systems themselves can't actually see the contents if they don't have the capability itself; this is much safer for peers to help the network share data because they can help route things through the network without personally knowing or being responsible for what the contents of those messages are. It could also be possible for the goblin to provide Alice with a direct channel to a storage system to retrieve its assets from.) Horray, Alice got the 3d model and images! Now she can see the goblin.

Assuming that the goblin is an enemy, Alice attacks! Attacking is common in this game universe, and there is no reason necessarily to keep around attack messages, so sending a message to the goblin is just a one-off transient message... there's no need to persist it in the storage system.

The attack misses! The goblin shouts, "Wait!" and makes its case, that both of them are just adventurers in this room, and shouldn't they both be friends? Alice is touched and halts her attack. These messages are also sent transiently; while either party could log them, they are closer to an instant messenger or IRC conversation rather than something intended to be persisted long-term.

They exchange their mailbox addresses and begin sending each other letters. These, however, are intended to be persisted; when Alice receives a message from the goblin in her mailbox (or vice versa), the message received contains the datashards URI to the letter, which Alice can then retrieve from the appropriate store. She can then always refer to this message, and she can choose whether or not to persist it locally or elsewhere. Since the letter has its own storage URI, when Alice constructs a reply, she can clearly mark that it was in reference to the previous letter. Even if Alice or the goblin's servers go down, either can continue to refer to these letters. Alice and the goblin have the freedom to choose what storage systems they wish, whether targeted/direct/local or via a public peer to peer routing system, with reasonable assumptions (given the continued strength of the underlying cryptographic algorithms used) that the particular entities storing or forwarding their data cannot read its content.

And so it is: live references of actors are able to send live, transient messages, but can only be sent to other actors whose (unguessable/unforgeable) address you have. This allows for highly dynamic and expressive interactions while retaining security. Datashards URIs allow for the storage and retrieval of content which can continue to be persisted by interested parties, even if the originating host goes down.

There are some things I glossed over in this writeup. The particular ways that the actors' addresses and references work is one thing (unguessable http based capability URLs on their own have leakage problems due to the way various web technologies are implemented, and not even every actor reference needs to be a long-lived URI; see CapTP for more details), how to establish connections between actor processes/servers (we can reuse TLS, or even better, something like tor's onion services), so are how interactions such as fighting can be scoped to a room (this paper explains how), as well as how we can map human meaningful names onto unguessable identifiers (the answer there is petnames). But I have plans for this and increasing confidence that it will come together... I think we're already on track.

Hopefully this writeup brings some clarity on how some of the components will work together, though!

FSF Blogs: GNU Spotlight with Mike Gerwitz: 17 new GNU releases in June!

Thursday 27th of June 2019 04:08:17 PM

For announcements of most new GNU releases, subscribe to the info-gnu mailing list: https://lists.gnu.org/mailman/listinfo/info-gnu.

To download: nearly all GNU software is available from https://ftp.gnu.org/gnu/, or preferably one of its mirrors from https://www.gnu.org/prep/ftp.html. You can use the URL https://ftpmirror.gnu.org/ to be automatically redirected to a (hopefully) nearby and up-to-date mirror.

A number of GNU packages, as well as the GNU operating system as a whole, are looking for maintainers and other assistance: please see https://www.gnu.org/server/takeaction.html#unmaint if you'd like to help. The general page on how to help GNU is at https://www.gnu.org/help/help.html.

If you have a working or partly working program that you'd like to offer to the GNU project as a GNU package, see https://www.gnu.org/help/evaluation.html.

As always, please feel free to write to us at maintainers@gnu.org with any GNUish questions or suggestions for future installments.

Andy Wingo: fibs, lies, and benchmarks

Wednesday 26th of June 2019 10:34:11 AM

Friends, consider the recursive Fibonacci function, expressed most lovelily in Haskell:

fib 0 = 0 fib 1 = 1 fib n = fib (n-1) + fib (n-2)

Computing elements of the Fibonacci sequence ("Fibonacci numbers") is a common microbenchmark. Microbenchmarks are like a Suzuki exercises for learning violin: not written to be good tunes (good programs), but rather to help you improve a skill.

The fib microbenchmark teaches language implementors to improve recursive function call performance.

I'm writing this article because after adding native code generation to Guile, I wanted to check how Guile was doing relative to other language implementations. The results are mixed. We can start with the most favorable of the comparisons: Guile present versus Guile of the past.


I collected these numbers on my i7-7500U CPU @ 2.70GHz 2-core laptop, with no particular performance tuning, running each benchmark 10 times, waiting 2 seconds between measurements. The bar value indicates the median elapsed time, and above each bar is an overlayed histogram of all results for that scenario. Note that the y axis is on a log scale. The 2.9.3* version corresponds to unreleased Guile from git.

Good news: Guile has been getting significantly faster over time! Over decades, true, but I'm pleased.

where are we? static edition

How good are Guile's numbers on an absolute level? It's hard to say because there's no absolute performance oracle out there. However there are relative performance oracles, so we can try out perhaps some other language implementations.

First up would be the industrial C compilers, GCC and LLVM. We can throw in a few more "static" language implementations as well: compilers that completely translate to machine code ahead-of-time, with no type feedback, and a minimal run-time.


Here we see that GCC is doing best on this benchmark, completing in an impressive 0.304 seconds. It's interesting that the result differs so much from clang. I had a look at the disassembly for GCC and I see:

fib: push %r12 mov %rdi,%rax push %rbp mov %rdi,%rbp push %rbx cmp $0x1,%rdi jle finish mov %rdi,%rbx xor %r12d,%r12d again: lea -0x1(%rbx),%rdi sub $0x2,%rbx callq fib add %rax,%r12 cmp $0x1,%rbx jg again and $0x1,%ebp lea 0x0(%rbp,%r12,1),%rax finish: pop %rbx pop %rbp pop %r12 retq

It's not quite straightforward; what's the loop there for? It turns out that GCC inlines one of the recursive calls to fib. The microbenchmark is no longer measuring call performance, because GCC managed to reduce the number of calls. If I had to guess, I would say this optimization doesn't have a wide applicability and is just to game benchmarks. In that case, well played, GCC, well played.

LLVM's compiler (clang) looks more like what we'd expect:

fib: push %r14 push %rbx push %rax mov %rdi,%rbx cmp $0x2,%rdi jge recurse mov %rbx,%rax add $0x8,%rsp pop %rbx pop %r14 retq recurse: lea -0x1(%rbx),%rdi callq fib mov %rax,%r14 add $0xfffffffffffffffe,%rbx mov %rbx,%rdi callq fib add %r14,%rax add $0x8,%rsp pop %rbx pop %r14 retq

I bolded the two recursive calls.

Incidentally, the fib as implemented by GCC and LLVM isn't quite the same program as Guile's version. If the result gets too big, GCC and LLVM will overflow, whereas in Guile we overflow into a bignum. Also in C, it's possible to "smash the stack" if you recurse too much; compilers and run-times attempt to mitigate this danger but it's not completely gone. In Guile you can recurse however much you want. Finally in Guile you can interrupt the process if you like; the compiled code is instrumented with safe-points that can be used to run profiling hooks, debugging, and so on. Needless to say, this is not part of C's mission.

Some of these additional features can be implemented with no significant performance cost (e.g., via guard pages). But it's fair to expect that they have some amount of overhead. More on that later.

The other compilers are OCaml's ocamlopt, coming in with a very respectable result; Go, also doing well; and V8 WebAssembly via Node. As you know, you can compile C to WebAssembly, and then V8 will compile that to machine code. In practice it's just as static as any other compiler, but the generated assembly is a bit more involved:

fib_tramp: jmp fib fib: push %rbp mov %rsp,%rbp pushq $0xa push %rsi sub $0x10,%rsp mov %rsi,%rbx mov 0x2f(%rbx),%rdx mov %rax,-0x18(%rbp) cmp %rsp,(%rdx) jae stack_check post_stack_check: cmp $0x2,%eax jl return_n lea -0x2(%rax),%edx mov %rbx,%rsi mov %rax,%r10 mov %rdx,%rax mov %r10,%rdx callq fib_tramp mov -0x18(%rbp),%rbx sub $0x1,%ebx mov %rax,-0x20(%rbp) mov -0x10(%rbp),%rsi mov %rax,%r10 mov %rbx,%rax mov %r10,%rbx callq fib_tramp return: mov -0x20(%rbp),%rbx add %ebx,%eax mov %rbp,%rsp pop %rbp retq return_n: jmp return stack_check: callq WasmStackGuard mov -0x10(%rbp),%rbx mov -0x18(%rbp),%rax jmp post_stack_check

Apparently fib compiles to a function of two arguments, the first passed in rsi, and the second in rax. (V8 uses a custom calling convention for its compiled WebAssembly.) The first synthesized argument is a handle onto run-time data structures for the current thread or isolate, and in the function prelude there's a check to see that the function has enough stack. V8 uses these stack checks also to handle interrupts, for when a web page is stuck in JavaScript.

Otherwise, it's a more or less normal function, with a bit more register/stack traffic than would be strictly needed, but pretty good.

do optimizations matter?

You've heard of Moore's Law -- though it doesn't apply any more, it roughly translated into hardware doubling in speed every 18 months. (Yes, I know it wasn't precisely that.) There is a corresponding rule of thumb for compiler land, Proebsting's Law: compiler optimizations make software twice as fast every 18 years. Zow!

The previous results with GCC and LLVM were with optimizations enabled (-O3). One way to measure Proebsting's Law would be to compare the results with -O0. Obviously in this case the program is small and we aren't expecting much work out of the optimizer, but it's interesting to see anyway:


Answer: optimizations don't matter much for this benchark. This investigation does give a good baseline for compilers from high-level languages, like Guile: in the absence of clever trickery like the recursive inlining thing GCC does and in the absence of industrial-strength instruction selection, what's a good baseline target for a compiler? Here we see for this benchmark that it's somewhere between 420 and 620 milliseconds or so. Go gets there, and OCaml does even better.

how is time being spent, anyway?

Might we expect V8/WebAssembly to get there soon enough, or is the stack check that costly? How much time does one stack check take anyway? For that we'd have to determine the number of recursive calls for a given invocation.

Friends, it's not entirely clear to me why this is, but I instrumented a copy of fib, and I found that the number of calls in fib(n) was a more or less constant factor of the result of calling fib. That ratio converges to twice the golden ratio, which means that since fib(n+1) ~= φ * fib(n), then the number of calls in fib(n) is approximately 2 * fib(n+1). I scratched my head for a bit as to why this is and I gave up; the Lord works in mysterious ways.

Anyway for fib(40), that means that there are around 3.31e8 calls, absent GCC shenanigans. So that would indicate that each call for clang takes around 1.27 ns, which at turbo-boost speeds on this machine is 4.44 cycles. At maximum throughput (4 IPC), that would indicate 17.8 instructions per call, and indeed on the n > 2 path I count 17 instructions.

For WebAssembly I calculate 2.25 nanoseconds per call, or 7.9 cycles, or 31.5 (fused) instructions at max IPC. And indeed counting the extra jumps in the trampoline, I get 33 cycles on the recursive path. I count 4 instructions for the stack check itself, one to save the current isolate, and two to shuffle the current isolate into place for the recursive calls. But, compared to clang, V8 puts 6 words on the stack per call, as opposed to only 4 for LLVM. I think with better interprocedural register allocation for the isolate (i.e.: reserve a register for it), V8 could get a nice boost for call-heavy workloads.

where are we? dynamic edition

Guile doesn't aim to replace C; it's different. It has garbage collection, an integrated debugger, and a compiler that's available at run-time, it is dynamically typed. It's perhaps more fair to compare to languages that have some of these characteristics, so I ran these tests on versions of recursive fib written in a number of languages. Note that all of the numbers in this post include start-up time.


Here, the ocamlc line is the same as before, but using the bytecode compiler instead of the native compiler. It's a bit of an odd thing to include but it performs so well I just had to include it.

I think the real takeaway here is that Chez Scheme has fantastic performance. I have not been able to see the disassembly -- does it do the trick like GCC does? -- but the numbers are great, and I can see why Racket decided to rebase its implementation on top of it.

Interestingly, as far as I understand, Chez implements stack checks in the straightfoward way (an inline test-and-branch), not with a guard page, and instead of using the stack check as a generic ability to interrupt a computation in a timely manner as V8 does, Chez emits a separate interrupt check. I would like to be able to see Chez's disassembly but haven't gotten around to figuring out how yet.

Since I originally published this article, I added a LuaJIT entry as well. As you can see, LuaJIT performs as well as Chez in this benchmark.

Haskell's call performance is surprisingly bad here, beaten even by OCaml's bytecode compiler; is this the cost of laziness, or just a lacuna of the implementation? I do not know. I do know I have this mental image that Haskell is a good compiler but apparently if that's the standard, so is Guile :)

Finally, in this comparison section, I was not surprised by cpython's relatively poor performance; we know cpython is not fast. I think though that it just goes to show how little these microbenchmarks are worth when it comes to user experience; like many of you I use plenty of Python programs in my daily work and don't find them slow at all. Think of micro-benchmarks like x-ray diffraction; they can reveal the hidden substructure of DNA but they say nothing at all about the organism.

where to now?

Perhaps you noted that in the last graph, the Guile and Chez lines were labelled "(lexical)". That's because instead of running this program:

(define (fib n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2)))))

They were running this, instead:

(define (fib n) (define (fib* n) (if (< n 2) n (+ (fib* (- n 1)) (fib* (- n 2))))) (fib* n))

The thing is, historically, Scheme programs have treated top-level definitions as being mutable. This is because you don't know the extent of the top-level scope -- there could always be someone else who comes and adds a new definition of fib, effectively mutating the existing definition in place.

This practice has its uses. It's useful to be able to go in to a long-running system and change a definition to fix a bug or add a feature. It's also a useful way of developing programs, to incrementally build the program bit by bit.


But, I would say that as someone who as written and maintained a lot of Scheme code, it's not a normal occurence to mutate a top-level binding on purpose, and it has a significant performance impact. If the compiler knows the target to a call, that unlocks a number of important optimizations: type check elision on the callee, more optimal closure representation, smaller stack frames, possible contification (turning calls into jumps), argument and return value count elision, representation specialization, and so on.

This overhead is especially egregious for calls inside modules. Scheme-the-language only gained modules relatively recently -- relative to the history of scheme -- and one of the aspects of modules is precisely to allow reasoning about top-level module-level bindings. This is why running Chez Scheme with the --program option is generally faster than --script (which I used for all of these tests): it opts in to the "newer" specification of what a top-level binding is.

In Guile we would probably like to move towards a more static way of treating top-level bindings, at least those within a single compilation unit. But we haven't done so yet. It's probably the most important single optimization we can make over the near term, though.

As an aside, it seems that LuaJIT also shows a similar performance differential for local function fib(n) versus just plain function fib(n).

It's true though that even absent lexical optimizations, top-level calls can be made more efficient in Guile. I am not sure if we can reach Chez with the current setup of having a template JIT, because we need two return addresses: one virtual (for bytecode) and one "native" (for JIT code). Register allocation is also something to improve but it turns out to not be so important for fib, as there are few live values and they need to spill for the recursive call. But, we can avoid some of the indirection on the call, probably using an inline cache associated with the callee; Chez has had this optimization since 1984!

what guile learned from fib

This exercise has been useful to speed up Guile's procedure calls, as you can see for the difference between the latest Guile 2.9.2 release and what hasn't been released yet (2.9.3).

To decide what improvements to make, I extracted the assembly that Guile generated for fib to a standalone file, and tweaked it in a number of ways to determine what the potential impact of different scenarios was. Some of the detritus from this investigation is here.

There were three big performance improvements. One was to avoid eagerly initializing the slots in a function's stack frame; this took a surprising amount of run-time. Fortunately the rest of the toolchain like the local variable inspector was already ready for this change.

Another thing that became clear from this investigation was that our stack frames were too large; there was too much memory traffic. I was able to improve this in the lexical-call by adding an optimization to elide useless closure bindings. Usually in Guile when you call a procedure, you pass the callee as the 0th parameter, then the arguments. This is so the procedure has access to its closure. For some "well-known" procedures -- procedures whose callers can be enumerated -- we optimize to pass a specialized representation of the closure instead ("closure optimization"). But for well-known procedures with no free variables, there's no closure, so we were just passing a throwaway value (#f). An unhappy combination of Guile's current calling convention being stack-based and a strange outcome from the slot allocator meant that frames were a couple words too big. Changing to allow a custom calling convention in this case sped up fib considerably.

Finally, and also significantly, Guile's JIT code generation used to manually handle calls and returns via manual stack management and indirect jumps, instead of using the platform calling convention and the C stack. This is to allow unlimited stack growth. However, it turns out that the indirect jumps at return sites were stalling the pipeline. Instead we switched to use call/return but keep our manual stack management; this allows the CPU to use its return address stack to predict return targets, speeding up code.

et voilà

Well, long article! Thanks for reading. There's more to do but I need to hit the publish button and pop this off my stack. Until next time, happy hacking!

FSF Blogs: GNU Emacs T-shirts available now at the GNU Press Shop

Tuesday 25th of June 2019 07:20:00 PM

Have you been waiting with bated breath for the opportunity to show your love for GNU Emacs, the text editor that also does everything else, with a nifty T-shirt? Wait no longer. The GNU Press Shop now has GNU Emacs logo T-shirts in unisex sizes S through XXXL. Order one at https://shop.fsf.org/tshirts-hoodies/gnu-emacs-logo-t-shirt, and we'll ship it to you sooner than you can say "extensible, customizable, self-documenting, real-time display editor."

All GNU Press Shop purchases support the Free Software Foundation's efforts to free all software, and FSF associate members get a 20% discount off of all purchases.

Christopher Allan Webber: Let's Just Be Weird Together

Tuesday 25th of June 2019 06:10:00 PM

Approximately a month ago was Morgan and I's 10 year wedding anniversary. To commemorate that, and as a surprise gift, I made the above ascii art and animation.

Actually, it's not just an animation, it's a program, and one you can run. As a side note, I originally thought I'd write up how I made it, but I kept procrastinating on that and it lead me to putting off writing this post for about a month. Oh well, all I'll say for now is that it lead to a major rewrite of one of the main components of Spritely. But that's something to speak of for another time, I suppose.

Back to the imagery! Morgan was surprised to see the animation, and yet the image itself wasn't a surprise. That's because the design is actually built off of one we collaborated on together:

I did the sketch for it and Morgan embroidered it. The plan is to put this above the tea station we set up in the reading area of our house.

The imagery and phrasing captures the philosophy of Morgan and I's relationship. We're both weird and deeply imperfect people, maybe even in some ways broken. But that's okay. We don't expect each other to change or become something else... we just try to become the best weird pairing we can together. I think that strategy has worked out for us.

Thanks for all the happy times so far, Morgan. I look forward to many weird years ahead.

FSF Blogs: Drop the journalism charges against Julian Assange

Tuesday 25th of June 2019 05:50:06 PM

The US government has persecuted Julian Assange for a decade for Wikileaks' journalism, and now seeks to use his case to label the publishing of leaked secret information as spying.

The Free Software Foundation stands for freedom of publication and due process, because they are necessary to exercise and uphold the software freedom we campaign for. The attack on journalism threatens freedom of publication; the twisting of laws to achieve an unstated aim threatens due process of law. The FSF therefore calls on the United States to drop all present and future charges against Julian Assange relating to Wikileaks activities.

Accusations against Assange that are unrelated to journalism should be pursued or not pursued based on their merits, giving him neither better nor worse treatment on account of his journalism.

More in Tux Machines

Security Leftovers

  • Researchers Build App That Kills To Highlight Insulin Pump Exploit

    By now the half-baked security in most internet of things (IOT) devices has become a bit of a running joke, leading to amusing Twitter accounts like Internet of Shit that highlight the sordid depth of this particular apathy rabbit hole. And while refrigerators leaking your gmail credentials and tea kettles that expose your home networks are entertaining in their own way, it's easy to lose sight of the fact that the same half-assed security in the IOT space also exists on most home routers, your car, your pacemaker, and countless other essential devices and services your life may depend on. Case in point: just about two years ago, security researchers discovered some major vulnerabilities Medtronic's popular MiniMed and MiniMed Paradigm insulin pumps. At a talk last year, they highlighted how a hacker could trigger the pumps to either withhold insulin doses, or deliver a lethal dose of insulin remotely. But while Medtronic and the FDA warned customers about the vulnerability and issued a recall over time, security researchers Billy Rios and Jonathan Butts found that initially, nobody was doing much to actually fix or replace the existing devices. [...] And of course that's not just a problem in the medical sector, but most internet-connected tech sectors. As security researcher Bruce Schneier often points out, it's part of a cycle of dysfunction where the consumer and the manufacturer of a flawed product have already moved on to the next big purchase, often leaving compromised products, and users, in a lurch. And more often than not, when researchers are forced to get creative to highlight the importance of a particular flaw, the companies in question enjoy shooting the messenger.

  • Desktop Operating Systems: Which is the safest? [Ed: This shallow article does not discuss NSA back doors and blames on "Linux" devices with open ports and laughable passwords -- based on narrative often pushed by corporate media to give illusion of parity. Also pushes the lie of Linux having minuscule usage.]
  • How Open Source Data Can Protect Consumer Credit Card Information
  • Open Source Hacking Tool Grows Up

    An open source white-hat hacking tool that nation-state hacking teams out of China, Iran, and Russia have at times employed to avoid detection....

Games: Dota Underlords and Stadia

  • Dota Underlords has another update out, this one changes the game quite a lot

    Valve continue to tweak Dota Underlords in the hopes of keeping players happy, this mid-Season gameplay update flips quite a few things on their head. I like their sense of humour, with a note about them removing "code that caused crashes and kept code that doesn't cause crashes". There's a few smaller changes like the addition of Loot Round tips to the Season Info tab, the ability to change equipped items from the Battle Pass and some buffs to the amount XP awarded for your placement in matches and for doing the quests. Meaning you will level up the Battle Pass faster.

  • Interested in Google's Stadia game streaming service? We have a few more details now

    With Google's game streaming service Stadia inching closer, we have some more information to share about it. Part of this, is thanks to a recent AMA (Ask Me Anything) they did on Reddit. I've gone over what questions they answered, to give you a little overview. Firstly, a few points about the Stadia Pro subscription: The Pro subscription is not meant to be like a "Netflix for Games", something people seem to think Stadia will end up as. Google said to think of it more like Xbox Live Gold or Playstation Plus. They're aiming to give Pro subscribers one free game a month "give or take". If you cancel Stadia Pro, you will lose access to free games claimed. However, you will get the previously claimed games back when you re-subscribe but not any you missed while not subscribed. As for Stadia Base, as expected there will be no free games included. As already confirmed, both will let you buy games as normal.

LabPlot has got some beautifying and lots of datasets

Hello everyone! The second part of this year's GSoC is almost over, so I was due to let you know the progress made in the last 3 weeks. I can assure you we haven't lazed since then. I think I managed to make quite good progress, so everything is going as planned, or I could say that even better. If you haven't read about this year's project or you just want to go through what has already been accomplished you can check out my previous post. So let's just go through the new things step by step. I'll try to explain the respective feature, and also give examples using videos or screenshots. The first step was to improve the welcome screen and make it easily usable, dynamic, clean and intuitive for users. This step was very important since the welcome screen is what the users will first get in contact with when they start using LabPlot. Read more

Graphics: Weston 7.0 Reaches Alpha and RadeonSI Gallium3D Driver Adds Navi Wave32 Support

  • weston 6.0.91
    This is the alpha release for weston 7.0.  A lot of new features and
    fixes are shipped in this release, including:
    
    - New internal debug scopes and logging framework
    - Improved documentation
    - HDCP support
    - A new PipeWire plugin
    
    Thanks to all contributors!
    
    We've moved to Meson as our only build system, autotools support has
    been removed.  Package maintainers: please report any issues you have
    with Meson before the stable release.
    
    Full commit history below.
    
  • Weston 7.0 Reaches Alpha With PipeWire, HDCP, EGL Partial Updates & Mores

    Wayland release manager Simon Ser announced the alpha release of the Weston 7.0 reference compositor on Friday that also marks the feature freeze for this Wayland compositor update. Some of the major changes to Weston 7.0 include HDCP content protection support, better documentation, new debugging and logging framework support, and the just-added PipeWire plug-in for remote streaming. There are also less prominent additions like EGL partial update support, various DRM compositor back-end restructuring, build system updates, and a variety of libweston updates.

  • RadeonSI Gallium3D Driver Adds Navi Wave32 Support

    One of the new features to the RDNA architecture with Navi is support for single cycle issue Wave32 execution on SIMD32. Up to now the RadeonSI code was using just Wave64 but now there is support in this AMD open-source Linux OpenGL driver for Wave32. Well known AMD open-source developer Marek Olšák landed this Wave32 support on Friday for the RadeonSI driver. The Wave32 support landed over several commits to Mesa 19.2-devel and is enabled for vertex, geometry, and tessellation shaders. Wave32 isn't enabled for pixel shaders but rather Wave64. Additionally, Wave32 isn't yet enabled for compute shaders due to Piglit OpenGL test case failures.