Language Selection

English French German Italian Portuguese Spanish

Debian

Syndicate content
Planet Debian - https://planet.debian.org/
Updated: 9 hours 11 min ago

Jonathan Dowland: Lockdown music

Wednesday 15th of July 2020 12:11:16 PM

Last Christmas, to make room for a tree, I dis-assembled my hifi unit and (temporarily, I thought) lugged my hi-fi and records up to the study.

I had been thinking about expanding the amount of storage I had for hifi and vinyl, perhaps moving from 2x2 storage cubes to 2x3, although Ikea don't make a 2x3 version of their stalwart vinyl storage line, the Kallax. I had begun exploring other options, both at Ikea and other places. Meanwhile, I re-purposed my old Expedit unit as storage for my daughter's Sylvanian Families.

Under Lockdown, I've spent a lot more time in my study, and so I've set up the hifi there. It turns out I have a lot more opportunity to enjoy the records up here, during work, and I've begun to explore some things which I haven't listened to in a long time, or possibly ever. I thought I'd start keeping track of some of them.

Power, Corruption and Lies is not something rarely listened to. It's steadily become my favourite New Order album. When I came across this copy (Factory, 1983), it was in pristine condition, but it now bears witness to my (mostly careful) use. There's now a scratch somewhere towards the end of the first track Age of Consent which causes my turntable to loop. By some good fortune the looping point is perfectly aligned to a bar. I don't always notice it straight away. This record rarely makes it back down from the turntable to where it's supposed to live.

Antoine Beaupré: Goatcounter analytics in ikiwiki

Wednesday 15th of July 2020 02:07:52 AM

I have started using Goatcounter for analytics after reading this LWN article called "Lightweight alternatives to Google Analytics". Goatcounter has an interesting approach to privacy in that it:

tracks sessions using a hash of the browser's user agent and IP address to identify the client without storing any personal information. The salt used to generate these hashes is rotated every 4 hours with a sliding window.

There was no Debian package for the project, so I filed a request for package and instead made a fork of the project to add a Docker image.

This page documents how Goatcounter was setup from there...

Server configuration
  1. build the image from this fork

    docker build -t zgoat/goatcounter .
  2. create volume for db:

    docker volume create goatcounter
  3. start the server:

    exec docker run --restart=unless-stopped --volume="goatcounter:/home/user/db/" --publish 127.0.0.1:8081:8080 --detach zgoat/goatcounter serve -listen :8080 -tls none
  4. apache configuration:

    <VirtualHost *:80> ServerName analytics.anarc.at Redirect / https://analytics.anarc.at/ DocumentRoot /var/www/html/ </VirtualHost> <VirtualHost *:443> ServerName analytics.anarc.at Use common-letsencrypt-ssl analytics.anarc.at DocumentRoot /var/www/html/ ProxyPass /.well-known/ ! ProxyPass / http://localhost:8081/ ProxyPassReverse / http://localhost:8081/ ProxyPreserveHost on </VirtualHost>
  5. add analytics.anarc.at to DNS

  6. create a TLS cert with LE:

    certbot certonly --webroot -d analytics.anarc.at --webroot-path /var/www/html/

    note that goatcounter has code to do this on its own, but we avoid it to follow our existing policies and simplify things

  7. create site:

    docker run -it --rm --volume="goatcounter:/home/user/db/" zgoat/goatcounter create -domain analytics.anarc.at -email anarcat+rapports@anarc.at
  8. add to ikiwiki template

  9. rebuild wiki:

    ikiwiki --setup ikiwiki.setup --rebuild --verbose
Remaining issues
  • Docker image should be FROM scratch, this is statically built golang stuff after all...
  • move to Docker Compose or podman instead of just starting the thing by hand
  • this is all super janky and should be put in config management somehow
  • remove "anarc.at" test site (the site is the analytics site, not the tracked site), seems like this is not possible yet
  • do log parsing instead of Javascript or 1x1 images?
  • compare with goaccess logs, probably at the end of july, to have two full weeks to compare
Fixed issues
  • cache headers are wrong (120ms!) deployed workaround in apache, reported as a bug upstream
  • remove self-referer done, just a matter of configuring the URL in the settings. could this be automated too?
  • add pixel tracking for noscript users done, but required a patch to ikiwi (and I noticed another bug while doing it)
  • goatcounter monitor doesn't with sqlite (fixed upstream!)
  • the :8080 port leaks in some places, namely in the "Site config" documentation that is because i was using -port 8080 which was not necessary.

Ian Jackson: MessagePack vs CBOR (RFC7049)

Tuesday 14th of July 2020 09:53:27 PM

tl;dr: Use MessagePack, rather than CBOR.

Introduction

I recently wanted to choose a binary encoding. This was for a project using Rust serde, so I looked at the list of formats there. I ended up reading about CBOR and MessagePack.

Both of these are binary formats for a JSON-like data model. Both of them are "schemaless", meaning you can decode them without knowing the structure. (This also provides some forwards compatibility.) They are, in fact, quite similar (although they are totally incompatible). This is no accident: CBOR is, effectively, a fork of MessagePack.

Both formats continue to exist and both are being used in new programs. I needed to make a choice but lacked enough information. I thought I would try to examine the reasons and nature of the split, and to make some kind of judgement about the situation. So I did a lot of reading [11]. Here are my conclusions.

History and politics

Between about 2010 and 2013 there was only MessagePack. Unfortunately, MessagePack had some problems. The biggest of these was that it lacked a separate string type. Strings were to be encoded simply as byte blocks. This caused serious problems for many MessagePack library implementors: for example, when decoding a MessagePack file the Python library wouldn't know whether to produce a Python bytes object, or a string. Straightforward data structures wouldn't round trip through MessagePack. [1] [2]

It seems that in late 2012 this came to the attention to someone with an IETF background. According to them, after unsatisfactory conversations with MessagePack upstream, they decided they would have to fork. They submitted an Internet Draft for a partially-incompatible protocol [3] [4]. Little seemed to happen in the IETF until soon before the Orlando in-person IETF meeting in February 2013.[5]

These conversations sparked some discussion in the MessagePack issue tracker. There were long threads including about process [1,2,4 ibid]. But there was also a useful technical discussion, about proposed backward compatible improves to the MessagePack spec.[5] The prominent IETF contributor provided some helpful input in these discussions in the MessagePack community - but also pushed quite hard for a "tagging" system, which suggestion was not accepted (see my technical analysis, below).

An improved MessagePack spec resulted, with string support, developed largely by the MessagePack community. It seems to have been available in useable form since mid-2013 and was officially published as canonical in August 2013.

Meanwhile a parallel process was pursued in the IETF, based on the IETF contributor's fork, with 11 Internet-Drafts from February[7] to September[8]. This seems to have continued even though the original technical reason for the fork - lack of string vs binary distinction - no longer applied. The IETF proponent expressed unhappiness about MessagePack's stewardship and process as much as they did about the technical details [4, ibid]. The IETF process culminated in the CBOR RFC[9].

The discussion on process questions between the IETF proponent and MessagePack upstream, in the MessagePack issue tracker [4, ibid] should make uncomfortable reading for IETF members. The IETF acceptance of CBOR despite clear and fundamental objections from MessagePack upstream[13] and indeed other respected IETF members[14], does not reflect well on the IETF. The much vaunted openness of the IETF process seems to have been rather one-sided. The IETF proponent here was an IETF Chair. Certainly the CBOR author was very well-spoken and constantly talks about politeness and cooperation and process; but what they actually did was very hostile. They accused the MessagePack community of an "us and them" attitude while simultaneously pursuing a forked specification!

The CBOR RFC does mention MessagePack in Appendix E.2. But not to acknowledge that CBOR was inspired by MessagePack. Rather, it does so to make a set of tendentious criticisms of MessagePack. Perhaps these criticisms were true when they were first written in an I-D but they were certainly false by the time the RFC was actually published, which occurred after the MessagePack improvement process was completely concluded, with a formal spec issued.

Since then both formats have existed in parallel. Occasionally people discuss which one is better, and sometimes it is alleged that "yes CBOR is the successor to MessagePack", which is not really fair.[9][10]

Technical differences

The two formats have a similar arrangement: initial byte which can encode small integers, or type and length, or type and specify a longer length encoding. But there are important differences. Overall, MessagePack is very significantly simpler.

Floating point

CBOR supports five floating point formats! Not only three sizes of IEEE754, but also decimal floating point, and bigfloats. This seems astonishing for a supposedly-simple format. (Some of these are supported via the semi-optional tag mechanism - see below.)

Indefinite strings and arrays

Like MessagePack, CBOR mostly precedes items with their length. But CBOR also supports "indefinite" strings, arrays, and so on, where the length is not specified at the beginning. The object (array, string, whatever) is terminated by a special "break" item. This seems to me to be a mistake. If you wanted the kind of application where MessagePack or CBOR would be useful, streaming sub-objects of unknown length is not that important. This possibility considerably complicates decoders.

CBOR tagging system

CBOR has a second layer of sort-of-type which can be attached to each data item. The set of possible tags is open-ended and extensible, but the CBOR spec itself gives tag values for: two kinds of date format; positive and negative bignums; decimal floats (see above); binary but expected to be encoded if converted to JSON (in base64url, base64, or base16); nestedly encoded CBOR; URIs; base64 data (two formats); regexps; MIME messages; and a special tag to make file(1) work.

In practice it is not clear how many of these are used, but a decoder must be prepared to at least discard them. The amount of additional spec complexity here is quite astonishing. IMO binary formats like this will (just like JSON) be used by a next layer which always has an idea of what the data means, including (where the data is a binary blob) what encoding it is in etc. So these tags are not useful.

These tags might look like a middle way between (i) extending the binary protocol with a whole new type such as an extension type (incompatible with old readers) and encoding your new kind data in a existing type (leaving all readers who don't know the schema to print it as just integers or bytes or string). But I think they are more trouble than they are worth.

The tags are uncomfortably similar to the ASN.1 tag system, which is widely regarded as one of ASN.1's unfortunate complexities.

MessagePack extension mechanism

MessagePack explicitly reserves some encoding space for users and for future extensions: there is an "extension type". The payload is an extension type byte plus some more data bytes; the data bytes are in a format to be defined by the extension type byte. Half of the possible extension byte values are reserved for future specification, and half are designated for application use. This is pleasingly straightforward.

(There is also one unused primary initial byte value, but that would be rejected by existing decoders and doesn't seem like a likely direction for future expansion.)

Minor other differences in integer encoding

The encodings of integers differ.

In MessagePack, signed and unsigned integers have different typecodes. In CBOR, signed and unsigned positive integers have the same typecodes; negative integers have a different set of typecodes. This means that a CBOR reader which knows it is expecting a signed value will have to do a top-bit-set check on the actual data value! And a CBOR writer must check the value to choose a typecode.

MessagePack reserves fewer shortcodes for small negative integers, than for small positive integers.

Conclusions and lessons

MessagePack seems to have been prompted into fixing the missing string type problem, but only by the threat of a fork. However, this fork went ahead even after MessagePack clearly accepted the need for a string type. MessagePack had a fixed protocol spec before the IETF did.

The continued pursuit of the IETF fork was ostensibly been motivated by a disapproval of the development process and in particular a sense that the IETF process was superior. However, it seems to me that the IETF process was abused by CBOR's proponent, who just wanted things their own way. I have seen claims by IETF proponents that the open decisionmaking system inherently produces superior results. However, in this case the IETF process produced a bad specification. To the extent that other IETF contributors had influence over the ultimate CBOR RFC, I don't think they significantly improved it. CBOR has been described as MessagePack bikeshedded by the IETF. That would have been bad enough, but I think it's worse than that. To a large extent CBOR is one person's NIH-induced bad design rubber stamped by the IETF. CBOR's problems are not simply matters of taste: it's significantly overcomplicated.

One lesson for the rest of us is that although being the upstream and nominally in charge of a project seems to give us a lot of power, it's wise to listen carefully to one's users and downstreams. Once people are annoyed enough to fork, the fork will have a life of its own.

Another lesson is that many of us should be much warier of the supposed moral authority of the IETF. Many IETF standards are awful (Oauth 2 [12]; IKE; DNSSEC; the list goes on). Sometimes (especially when network adoption effects are weak, as with MessagePack vs CBOR) better results can be obtained from a smaller group, or even an individual, who simply need the thing for their own uses.

Finally, governance systems of public institutions like the IETF need to be robust in defending the interests of outsiders (and hence of society at large) against eloquent insiders who know how to work the process machinery. Any institution which nominally serves the public good faces a constant risk of devolving into self-servingness. This risk gets worse the more powerful and respected the institution becomes.

References

  1. #13: First-class string type in serialization specification (MessagePack issue tracker, June 2010 - August 2013)
  2. #121: Msgpack can't differentiate between raw binary data and text strings (MessagePack issue tracker, November 2012 - February 2013)
  3. draft-bormann-apparea-bpack-00: The binarypack JSON-like representation format (IETF Internet-Draft, October 2012)
  4. #129: MessagePack should be developed in an open process (MessagePack issue tracker, February 2013 - March 2013)
  5. Re: JSON mailing list and BoF (IETF apps-discuss mailing list message from Carsten Bormann, 18 February 2013)
  6. #128: Discussions on the upcoming MessagePack spec that adds the string type to the protocol (MessagePack issue tracker, February 2013 - August 2013)
  7. draft-bormann-apparea-bpack-01: The binarypack JSON-like representation format (IETF Internet-Draft, February 2013)
  8. draft-bormann-cbor: Concise Binary Object Representation (CBOR) (IETF Internet-Drafts, May 2013 - September 2013)
  9. RFC 7049: Concise Binary Object Representation (CBOR) (October 2013)
  10. "MessagePack should be replaced with [CBOR] everywhere ..." (floatboth on Hacker News, 8th April 2017)
  11. Discussion with very useful set of history links (camgunz on Hacker News, 9th April 2017)
  12. OAuth 2.0 and the Road to Hell (Eran Hammer, blog posting from 2012, via Wayback Machine)
  13. Re: [apps-discuss] [Json] msgpack/binarypack (Re: JSON mailing list and BoF) (IETF list message from Sadyuki Furuhashi, 4th March 2013)
  14. "no apologies for complaining about this farce" (IETF list message from Phillip Hallam-Baker, 15th August 2013) Edited 2020-07-14 18:55 to fix a minor formatting issue, and 2020-07-14 22:54 to fix two typos



comments

Markus Koschany: My Free Software Activities in June 2020

Tuesday 14th of July 2020 01:31:08 PM

Welcome to gambaru.de. Here is my monthly report (+ the first week in July) that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games Short news
  • The last month saw a new upstream release of Minetest (version 5.3.), a multi-player sandbox game similar to Minecraft. A backport to buster-backports will follow shortly.
  • Asher Gordon helped release a new version of Berusky 2, a sokoban like logic game but in 3D. The game received several improvements including bug fixes, code polishing and a new way to access the data files. Previously those files were all packed in a special container format but now they can be accessed directly without someone having to rely on some sort of unarchiver. I have uploaded this version as 0.12-1 to Debian unstable.
  • I tested an upstream patch for empire to address the build failure with GCC 10. This one is a better solution than the currently implemented workaround and I expect it to be included in the next upstream release.
  • I fixed two FTBFS in simutrans-pak64 and simutrans-pak128.britain, two addon packages for the simulation game simutrans.
Debian Java
  • New upstream versions this month: hsqldb, libpdfbox2-java, jackson-jr, jackson-dataformat-xml and jackson-databind. The latter upload addressed several security vulnerabilites which have become rather minor because upstream has enabled safe default typing by default now. Nevertheless I have prepared a buster-security update as well which is already available in buster-proposed-updates.
Misc
  • I packaged new versions of wabt, privacybadger and binaryen and applied another upstream patch for xarchiver to address the incomplete fix for Debian bug #959914, to better handle encrypted multi-volume 7zip archives.
  • By popular request I uploaded imlib2 version 1.6 to buster-backports because the image library supports the webp format now.
Debian LTS

This was my 52. month as a paid contributor and I have been paid to work 60 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • DLA-2278-1. Issued a security update for squid3 fixing 19 CVE.
  • DLA-2279-1. Issued a security update for tomcat8 fixing 2 CVE.
  • Prepared and uploaded a stretch-pu update for jackson-databind fixing 20 CVE. (#964727)
  • Synced the proftpd-dfsg version from Jessie with Stretch to address a memory leak which leads to a denial-of-service and correct the version number to make seemless updates work.
  • Prepared the security update for imagemagick triaging and/or fixing 76 CVE.
  • Worked on updating the database about embedded code copies to determine how packages are affected by security vulnerabilities in embedded code copies. This included a) compiling a list of important packages which are regular affected by CVE, b) investigating if embedded code copies are present, c) determining the possible impact of a security vulnerability in those embedded code copies, d) writing a script that automates printing those findings on demand.

Thanks for reading and see you next time.

Bits from Debian: Let's celebrate DebianDay 2020 around the world

Tuesday 14th of July 2020 12:00:00 PM

We encourage our community to celebrate around the world the 27th Debian anniversary with organized [DebianDay][1] events. This year due to the COVID-19 pandemic we cannot organize in-person events, so we ask instead that contributors, developers, teams, groups, maintainers, and users promote The Debian Project and Debian activities online on August 16th (and/or 15th).

Communities can organize a full schedule of online activities throughout the day. These activities can include talks, workshops, active participation with contributions such as translations assistance or editing, debates, BoFs, and all of this in your local language using tools such as [Jitsi][2] for capturing audio and video from presenters for later streaming to YouTube.

If you are not aware of any local community organizing a full event or you don't want to join one, you can solo design your own activity using [OBS][3] and stream it to YouTube. You can watch an OBS tutorial [here][4].

Don't forget to record your activity as it will be a nice idea to upload it to [Peertube][5] later.

Please add your event/activity on the [DebianDay wiki page][6] and let us know about and advertise it on [Debian micronews][7]. To share it, you have several options:

  • Follow the steps listed [here][8] for Debian Developers.
  • Contact us using IRC in channel #debian-publicity on the OFTC network, and ask us there.
  • Send a mail to debian-publicity@lists.debian.org and ask for your item to be included in micronews. This is a publicly archived list.

PS: DebConf20 online is coming! It will be held from August 23rd to 29th, 2020. [Registration is already open][9].

[1] https://wiki.debian.org/DebianDay [2] https://meet.jit.si [3] https://obsproject.com [4] https://peertube.debian.social/videos/watch/7f41c0e7-66cc-4234-b929-6b3219d95c14 [5] https://peertube.debian.social [6] https://wiki.debian.org/DebianDay/2020 [7] https://micronews.debian.org [8] https://micronews.debian.org/pages/contribute.html [9] https://debconf20.debconf.org/news/2020-07-12-registration-is-open/

Russell Coker: Debian PPC64EL Emulation

Tuesday 14th of July 2020 03:29:20 AM

In my post on Debian S390X Emulation [1] I mentioned having problems booting a Debian PPC64EL kernel under QEMU. Giovanni commented that they had PPC64EL working and gave a link to their site with Debian QEMU images for various architectures [2]. I tried their image which worked then tried mine again which also worked – it seemed that a recent update in Debian/Unstable fixed the bug that made QEMU not work with the PPC64EL kernel.

Here are the instructions on how to do it.

First you need to create a filesystem in an an image file with commands like the following:

truncate -s 4g /vmstore/ppc mkfs.ext4 /vmstore/ppc mount -o loop /vmstore/ppc /mnt/tmp

Then visit the Debian Netinst page [3] to download the PPC64EL net install ISO. Then loopback mount it somewhere convenient like /mnt/tmp2.

The package qemu-system-ppc has the program for emulating a PPC64LE system, the qemu-user-static package has the program for emulating PPC64LE for a single program (IE a statically linked program or a chroot environment), you need this to run debootstrap. The following commands should be most of what you need.

apt install qemu-system-ppc qemu-user-static update-binfmts --display # qemu ppc64 needs exec stack to solve "Could not allocate dynamic translator buffer" # so enable that on SE Linux systems setsebool -P allow_execstack 1 debootstrap --foreign --arch=ppc64el --no-check-gpg buster /mnt/tmp file:///mnt/tmp2 chroot /mnt/tmp /debootstrap/debootstrap --second-stage cat << END > /mnt/tmp/etc/apt/sources.list deb http://mirror.internode.on.net/pub/debian/ buster main deb http://security.debian.org/ buster/updates main END echo "APT::Install-Recommends False;" > /mnt/tmp/etc/apt/apt.conf echo ppc64 > /mnt/tmp/etc/hostname # /usr/bin/awk: error while loading shared libraries: cannot restore segment prot after reloc: Permission denied # only needed for chroot setsebool allow_execmod 1 chroot /mnt/tmp apt update # why aren't they in the default install? chroot /mnt/tmp apt install perl dialog chroot /mnt/tmp apt dist-upgrade chroot /mnt/tmp apt install bash-completion locales man-db openssh-server build-essential systemd-sysv ifupdown vim ca-certificates gnupg # install kernel last because systemd install rebuilds initrd chroot /mnt/tmp apt install linux-image-ppc64el chroot /mnt/tmp dpkg-reconfigure locales chroot /mnt/tmp passwd cat << END > /mnt/tmp/etc/fstab /dev/vda / ext4 noatime 0 0 #/dev/vdb none swap defaults 0 0 END mkdir /mnt/tmp/root/.ssh chmod 700 /mnt/tmp/root/.ssh cp ~/.ssh/id_rsa.pub /mnt/tmp/root/.ssh/authorized_keys chmod 600 /mnt/tmp/root/.ssh/authorized_keys rm /mnt/tmp/vmlinux* /mnt/tmp/initrd* mkdir /boot/ppc64 cp /mnt/tmp/boot/[vi]* /boot/ppc64 # clean up umount /mnt/tmp umount /mnt/tmp2 # setcap binary for starting bridged networking setcap cap_net_admin+ep /usr/lib/qemu/qemu-bridge-helper # afterwards set the access on /etc/qemu/bridge.conf so it can only # be read by the user/group permitted to start qemu/kvm echo "allow all" > /etc/qemu/bridge.conf

Here is an example script for starting kvm. It can be run by any user that can read /etc/qemu/bridge.conf.

#!/bin/bash set -e KERN="kernel /boot/ppc64/vmlinux-4.19.0-9-powerpc64le -initrd /boot/ppc64/initrd.img-4.19.0-9-powerpc64le" # single network device, can have multiple NET="-device e1000,netdev=net0,mac=02:02:00:00:01:04 -netdev tap,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper" # random number generator for fast start of sshd etc RNG="-object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0" # I have lockdown because it does no harm now and is good for future kernels # I enable SE Linux everywhere KERNCMD="net.ifnames=0 noresume security=selinux root=/dev/vda ro lockdown=confidentiality" kvm -drive format=raw,file=/vmstore/ppc64,if=virtio $RNG -nographic -m 1024 -smp 2 $KERN -curses -append "$KERNCMD" $NET

Related posts:

  1. Debian S390X Emulation I decided to setup some virtual machines for different architectures....
  2. installing Xen domU on Debian Etch I have just been installing a Xen domU on Debian...
  3. Installing a Red Hat based DomU on a Debian Dom0 The first step is to copy /images/xen/vmlinuz and /images/xen/initrd.img from...

Antoine Beaupré: Not recommending Purism

Monday 13th of July 2020 10:15:59 PM

This is just a quick note to mention that I have updated my hardware documentation on the Librem 13v4 laptop. It has unfortunately turned into a rather lengthy (and ranty) piece about Purism. Let's just say that waiting weeks for your replacement laptop (yes, it died again) does wonders for creativity. To quote the full review:

TL;DR: I recommend people avoid the Purism brand and products. I find they have questionable politics, operate in a "libre-washing" fashion, and produce unreliable hardware. Will not buy again.

People who have read the article might want to jump directly to the new sections:

I have also added the minor section of the missing mic jack.

I realize that some folks (particularly at Debian) might still work at Purism, and that this article might be demoralizing for their work. If that is the case, I am sorry this article triggered you in any way and I hope this can act as a disclaimer. But I feel it is my duty to document the issues I am going through, as a user, and to call bullshit when I see it (let's face it, the anti-interdiction stuff and the Purism 5 crowd-funding campaign were total bullshit).

I also understand that the pandemic makes life hard for everyone, and probably makes a bad situation at Purism worse. But those problems existed before the pandemic happened. They were issues I had identified in 2019 and that I simply never got around to document.

I wish that people wishing to support the free software movement would spend their energy towards organisations that actually do honest work in that direction, like System76 and Pine64. And if you're going to go crazy with an experimental free hardware design, why not go retro with the MNT Reform.

In the meantime, if you're looking for a phone, I recommend you give the Fairphone a fair chance. It really is a "fair" (as in, not the best, but okay) phone that you can moderately liberate, and it actually frigging works. See also my hardware review of the FP2.

Update: this kind of blew up, for my standards: 10k visitors in ~24h while I usually get about 1k visitors after a week on any regular blog post. There were more discussions on the subject here:

Trigger warning: some of those threads include personal insults and explicitly venture into the free speech discussion, with predictable (sad) consequences...

Bits from Debian: Debian Long Term Support (LTS) users and contributors survey

Monday 13th of July 2020 12:00:00 PM

On July 18th Stretch LTS starts, offering two more years of security support to the Debian Stretch release. Stretch LTS will be the fourth iteration of LTS, following Squeeze LTS which started in 2014, Wheezy LTS in 2016 and Jessie LTS in 2018.

However, for the first time, we have prepared a small survey about our users and contributors, who they are and why they are using LTS.

Filling out the survey should take less than 10 minutes. We would really appreciate if you could participate in the survey online!

In two weeks (July 27th 2020) we will close the survey, so please don't hesitate and participate now! After that, there will be a followup email with the results.

More information about Debian LTS is available at https://wiki.debian.org/LTS, including generic contact information.

Click here to fill out the survey now!

Antoine Beaupré: On contact tracing apps

Sunday 12th of July 2020 11:58:23 PM

I have strong doubts about the efficiency of any tracing app of the sort, and even less in the context where it is unlikely that a majority of the population will use it.

There's also the problem that this app would need to work on Apple phones, or be incompatible with them, and cause significant "fracture" between those who have access to technology, and those who haven't. See this text for more details.

Such an app would be a security and privacy liability at no benefit to public health. There are better options, see for this research on hardware tokens. But I doubt any contact tracing app or hardware will actually work anyways.

I am a computer engineer with more than 20 years of experience in the domain, and I have been following this question closely.

Please don't do this.

I wrote the above in a response to the Québec government's survey about a possible tracing app.

Update: a previous version of this article was titled plainly "on contact tracing". In case that was not obvious, I definitely do not object to contact tracing per se. I believe it's a fundamental, critical, and important part of fighting the epidemic and I think we should do it. I do not believe any engineer has found a proper way of doing it with "apps" so far, but I do not deny the utility and importance of "contact tracing" itself. Apologies for the confusion.

Pour une raison que je m'explique mal, le sondage m'été envoyé en anglais, et j'ai donc écrit ma réponse dans la langue de Shakespeare au lieu de celle de molière... Je serai heureux de fournir une traduction française à ceux ou celles qui en ont besoin...

Enrico Zini: Police brutality links

Sunday 12th of July 2020 10:00:00 PM
Confessions of a Former Bastard Cop police politics archive.org 2020-07-13 I was a police officer for nearly ten years and I was a bastard. We all were. I was a police chief stopped by my own officer. After Floyd, we need change at all levels. police politics archive.org 2020-07-13 Hi White People! … I want to tell you something about my experience with white progressive backlash police politics archive.org 2020-07-13 We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter? Police: Last Week Tonight with John Oliver police politics archive.org 2020-07-13 As nationwide protests over the deaths of George Floyd and Breonna Taylor are met with police brutality, John Oliver discusses how the histories of policing ... Morte di Stefano Cucchi - Wikipedia italy police politics archive.org 2020-07-13 La morte di Stefano Cucchi avvenne a Roma il 22 ottobre 2009 mentre il giovane era sottoposto a custodia cautelare. Le cause della morte e le responsabilità sono oggetto di procedimenti giudiziari che hanno coinvolto da un lato i medici dell'ospedale Pertini,[1][2][3][4] dall'altro continuano a coinvolgere, a vario titolo, più militari dell’Arma dei Carabinieri[5][6]. Il caso ha attirato l'attenzione dell'opinione pubblica a seguito della pubblicazione delle foto dell'autopsia, poi riprese da agenzie di stampa, giornali e telegiornali italiani[7]. La vicenda ha ispirato, altresì, documentari e lungometraggi cinematografici.[8][9][10] Morte di Giuseppe Uva - Wikipedia italy police politics archive.org 2020-07-13 La morte di Giuseppe Uva avvenne il 14 giugno 2008 dopo che, nella notte tra il 13 e il 14 giugno, era stato fermato ubriaco da due carabinieri che lo portarono in caserma, dalla quale venne poi trasferito, per un trattamento sanitario obbligatorio, nell'ospedale di Varese, dove morì la mattina successiva per arresto cardiaco. Secondo la tesi dell'accusa, la morte fu causata dalla costrizione fisica subita durante l'arresto e dalle successive violenze e torture che ha subito in caserma. Il processo contro i due carabinieri che eseguirono l'arresto e contro altri sei agenti di polizia ha assolto gli imputati dalle accuse di omicidio preterintenzionale e sequestro di persona[1][2][3][4]. Alla vicenda è dedicato il documentario Viva la sposa di Ascanio Celestini[1][5]. Caso Aldrovandi - Wikipedia italy police politics archive.org 2020-07-13 Il caso Aldrovandi è la vicenda giudiziaria causata dall'uccisione di Federico Aldrovandi, uno studente ferrarese, avvenuta il 25 settembre 2005 a seguito di un controllo di polizia.[1][2][3] I procedimenti giudiziari hanno condannato, il 6 luglio 2009, quattro poliziotti a 3 anni e 6 mesi di reclusione, per "eccesso colposo nell'uso legittimo delle armi";[1][4] il 21 giugno 2012 la Corte di cassazione ha confermato la condanna.[1] All'inchiesta per stabilire la cause della morte ne sono seguite altre per presunti depistaggi e per le querele fra le parti interessate.[1] Il caso è stato oggetto di grande attenzione mediatica e ha ispirato un documentario, È stato morto un ragazzo.[1][5] Federico Aldrovandi - Wikipedia italy police politics archive.org 2020-07-13 Federico Aldrovandi (17 July 1987 in Ferrara – 25 September 2005 in Ferrara) was an Italian student, who was killed by four policemen.[1] Aldrovandi, il film di Vendemmiati gratis on line – Articolo21 italy police politics archive.org 2020-07-13 24 Giugno 2020

Evgeni Golov: Using Ansible Molecule to test roles in monorepos

Sunday 12th of July 2020 08:03:17 AM

Ansible Molecule is a toolkit for testing Ansible roles. It allows for easy execution and verification of your roles and also manages the environment (container, VM, etc) in which those are executed.

In the Foreman project we have a collection of Ansible roles to setup Foreman instances called forklift. The roles vary from configuring Libvirt and Vagrant for our CI to deploying full fledged Foreman and Katello setups with Proxies and everything. The repository also contains a dynamic Vagrant file that can generate Foreman and Katello installations on all supported Debian, Ubuntu and CentOS platforms using the previously mentioned roles. This feature is super helpful when you need to debug something specific to an OS/version combination.

Up until recently, all those roles didn't have any tests. We would run ansible-lint on them, but that was it.

As I am planning to do some heavier work on some of the roles to enhance our upgrade testing, I decided to add some tests first. Using Molecule, of course.

Adding Molecule to an existing role is easy: molecule init scenario -r my-role-name will add all the necessary files/examples for you. It's left as an exercise to the reader how to actually test the role properly as this is not what this post is about.

Executing the tests with Molecule is also easy: molecule test. And there are also examples how to integrate the test execution with the common CI systems.

But what happens if you have more than one role in the repository? Molecule has support for monorepos, however that is rather limited: it will detect the role path correctly, so roles can depend on other roles from the same repository, but it won't find and execute tests for roles if you run it from the repository root. There is an undocumented way to set MOLECULE_GLOB so that Molecule would detect test scenarios in different paths, but I couldn't get it to work nicely for executing tests of multiple roles and upstream currently does not plan to implement this. Well, bash to the rescue!

for roledir in roles/*/molecule; do pushd $(dirname $roledir) molecule test popd done

Add that to your CI and be happy! The CI will execute all available tests and you can still execute those for the role you're hacking on by just calling molecule test as you're used to.

However, we can do even better.

When you initialize a role with Molecule or add Molecule to an existing role, there are quite a lot of files added in the molecule directory plus an yamllint configuration in the role root. If you have many roles, you will notice that especially the molecule.yml and .yamllint files look very similar for each role.

It would be much nicer if we could keep those in a shared place.

Molecule supports a "base config": a configuration file that gets merged with the molecule.yml of your project. By default, that's ~/.config/molecule/config.yml, but Molecule will actually look for a .config/molecule/config.yml in two places: the root of the VCS repository and your HOME. And guess what? The one in the repository wins (that's not yet well documented). So by adding a .config/molecule/config.yml to the repository, we can place all shared configuration there and don't have to duplicate it in every role.

And that .yamllint file? We can also move that to the repository root and add the following to Molecule's (now shared) configuration:

lint: yamllint --config-file ${MOLECULE_PROJECT_DIRECTORY}/../../.yamllint --format parsable .

This will define the lint action as calling yamllint with the configuration stored in the repository root instead of the project directory, assuming you store your roles as roles/<rolename>/ in the repository.

And that's it. We now have a central place for our Molecule and yamllint configurations and only need to place role-specific data into the role directory.

Dirk Eddelbuettel: drat 0.1.7: New functionality

Sunday 12th of July 2020 02:56:00 AM

A new version of drat arrived on CRAN yesterday. Once again, this release is mostly the work of Felix Ernst who extended some work from the previous release, and added support for repository updates (outside of package insertion) and more.

drat stands for drat R Archive Template, and helps with easy-to-create and easy-to-use repositories for R packages. Since its inception in early 2015 it has found reasonably widespread adoption among R users because repositories with marked releases is the better way to distribute code.

As your mother told you: Friends don’t let friends install random git commit snapshots. Rolled-up releases it is. drat is easy to use, documented by five vignettes and just works.

The NEWS file summarises the release as follows:

Changes in drat version 0.1.7 (2020-07-10)
  • Functions insertPackages, archivePackages and prunePackages are now vectorised (Patrick Schratz and Felix Ernst in #93, #100).

  • The new functionality is supported by unit tests (Felix Ernst in #93, and #102 fixing #101).

  • Added new function updateRepo (Felix Ernst in #95, #97).

Courtesy of CRANberries, there is a comparison to the previous release. More detailed information is on the drat page.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Simon Quigley: Adventures in Writing

Saturday 11th of July 2020 10:59:03 AM

The Linux community is a fascinating and powerful space.

When I joined the Ubuntu project approximately five years ago, I (vaguely at the time) understood that there was a profound sense of community and passion everywhere that is difficult to find in other spaces. My involvement has increased, and so has my understanding. I had thought of starting a blog as a means of conveying the information that I stumbled across, but my writing skills were very crude and regrettable, being in my early teenage years.

I have finally decided to take the leap. In this blog, I would like to occasionally provide updates on my work, either through focused deep dives on a particular topic, or broad updates on low hanging fruit that has been eliminated. While the articles may be somewhat spontaneous, I decided that an initial post was in order to explain my goals. Feel free to subscribe for more detailed posts in the future, as there are many more to come.

Enrico Zini: Wait until a command opened a file

Friday 10th of July 2020 05:27:29 PM

In my last post I wrote:

The sleep 0.3s is needed because xdg-open exits right after starting the program, and when invoked by mutt it means that mutt could delete the attachment before evince has a chance to open it. I had to use the same workaround for sensible-browser, since the same happens when a browser opens a document in an existing tab. I feel like writing some wrapper about all this that forks the viewer, then waits for an IN_OPEN event on its argument via inotify before exiting.

I wrote it: https://github.com/spanezz/waitused/

$ ./waitused --help usage: waitused [-h] path ... Run a command exiting only after it quits and a given file has been opened and closed positional arguments: path file to monitor command command to run optional arguments: -h, --help show this help message and exit

This works around situations like mutt deleting the temporary attachment file after run-mailcap is run, while run-mailcap runs a program that backgrounds before opening its input file.

Example waitused file.pdf xdg-open file.pdf waitused file.pdf run-mailcap file.pdf Example ~/.mailcap entry application/pdf; waitused -- %s xdg-open %s; test=test -n "$DISPLAY"

Update: Teddy Hogeborn pointed out that the initial mailcap entry would fail on files starting with a dash. I added -- for waitused, but unfortunately there seems to be no way at the moment to have xdg-open open files starting with a dash (see: #964949

Iain R. Learmonth: Light OpenStreetMapping with GPS

Friday 10th of July 2020 02:00:00 PM

Now that lockdown is lifting a bit in Scotland, I’ve been going a bit further for exercise. One location I’ve been to a few times is Tyrebagger Woods. In theory, I can walk here from my house via Brimmond Hill although I’m not yet fit enough to do that in one go.

Instead of following the main path, I took a detour along some route that looked like it wanted to be a path but it hadn’t been maintained for a while. When I decided I’d had enough of this, I looked for a way back to the main path but OpenStreetMap didn’t seem to have the footpaths mapped out here yet.

I’ve done some OpenStreetMap surveying before so I thought I’d take a look at improving this, and moving some of the tracks on the map closer to where they are in reality. In the past I’ve used OSMTracker which was great, but now I’m on iOS there doesn’t seem to be anything that matches up.

My new handheld radio, a Kenwood TH-D74 has the ability to record GPS logs so I thought I’d give this a go. It records the logs to the SD card with one file per session. It’s a very simple logger that records the NMEA strings as they are received. The only sentences I see in the file are GPGGA (Global Positioning System Fix Data) and GPRMC (Recommended Minimum Specific GPS/Transit Data).

I tried to import this directly with JOSM but it seemed to throw an error and crash. I’ve not investigated this, but I thought a way around could be to convert this to GPX format. This was easier than expected:

apt install gpsbabel gpsbabel -i nmea -f "/sdcard/KENWOOD/TH-D74/GPS_LOG/25062020_165017.nme" \ -o gpx,gpxver=1.1 -F "/tmp/tyrebagger.gpx"

This imported into JOSM just fine and I was able to adjust some of the tracks to better fit where they actually are.

I’ll take the radio with me when I go in future and explore some of the other paths, to see if I can get the whole woods mapped out nicely. It is fun to just dive into the trees sometimes, along the paths that looks a little forgotten and overgrown, but also it’s nice to be able to find your way out again when you get lost.

Reproducible Builds (diffoscope): diffoscope 151 released

Friday 10th of July 2020 12:00:00 AM

The diffoscope maintainers are pleased to announce the release of diffoscope version 151. This version includes the following changes:

[ Chris Lamb] * Improvements and bug fixes: - Pass the absolute path when extracting members from SquashFS images as we run the command with our working directory set to the temporary directory. (Closes: #964365, reproducible-builds/diffoscope#189) - Increase the minimum length of the output from strings(1) to 8 characters to avoid unnecessary diff noise. (Re. reproducible-builds/diffoscope#148) * Logging improvements: - Fix the compare_files message when the file does not have a literal name. - Reduce potential log noise by truncating the has_some_content messages. * Codebase changes: - Clarify use of a "null" diff in order to remember an exit code. - Don't alias a variable when don't end up it; use "_" instead. - Use a "NullChanges" file to represent missing data in the Debian package comparator. - Update some miscellaneous terms.

You find out more by visiting the project homepage.

Enrico Zini: Mime type associations

Thursday 9th of July 2020 08:20:34 AM

The last step of my laptop migration was to fix mime type associations, that seem to associate opening file depending on whatever application was installed last, phases of the moon, and what option is the most annoying.

The state of my system after a fresh install, is that, for application/pdf, xdg-open (used for example by pcmanfm) runs inkscape, and run-mailcap (used for example by neomutt) runs the calibre ebook viewer.

It looks like there are at least two systems to understand, debug and fix, instead of one.

xdg-open

This comes from package xdg-utils, and works using .desktop files:

# This runs inkscape $ xdg-open file.pdf

There is a tool called xdg-mime that queries what .desktop file is associated with a given mime type:

$ xdg-mime query default application/pdf inkscape.desktop

You can use xdg-mime default to change an association, and it works nicely:

$ xdg-mime default org.gnome.Evince.desktop application/pdf $ xdg-mime query default application/pdf org.gnome.Evince.desktop

However, if you accidentally mistype the name of the .desktop file, it won't complain and it will silently reset the association to the arbitrary default:

$ xdg-mime default org.gnome.Evince.desktop application/pdf $ xdg-mime query default application/pdf org.gnome.Evince.desktop $ xdg-mime default evince.desktop application/pdf $ echo $? 0 $ xdg-mime query default application/pdf inkscape.desktop

You can use a GUI like xfce4-mime-settings from the xfce4-settings package to perform the same kind of changes avoiding typing mistakes.

The associations seem to be saved in ~/.config/mimeapps.list

run-mailcap

This comes from the package mime-support

You can test things by running it using --norun:

$ run-mailcap --norun file.pdf ebook-viewer file.pdf

run-mailcap uses the ~/.mailcap and /etc/mailcap to map mime types to commands. This is what's in the system default:

$ grep application/pdf /etc/mailcap application/pdf; ebook-viewer %s; test=test -n "$DISPLAY" application/pdf; calibre %s; test=test -n "$DISPLAY" application/pdf; gimp-2.10 %s; test=test -n "$DISPLAY" application/pdf; evince %s; test=test -n "$DISPLAY"

To fix this, I copypasted the evince line into ~/.mailcap, and indeed it gets used:

$ run-mailcap --norun file.pdf evince file.pdf

There is a /etc/mailcap.order file providing a limited way to order entries in /etc/mailcap, but it can only be manipulated system-wide, and cannot be used for user preferences.

Sadly, this means that if a package changes its mailcap invocation because of, say, a security issue in the former one, the local override will never get fixed.

I am really not comfortable about that. As a workaround, I put this in my ~/.mailcap:

application/pdf; xdg-open %s && sleep 0.3s; test=test -n "$DISPLAY"

The sleep 0.3s is needed because xdg-open exits right after starting the program, and when invoked by mutt it means that mutt could delete the attachment before evince has a chance to open it. I had to use the same workaround for sensible-browser, since the same happens when a browser opens a document in an existing tab. I feel like writing some wrapper about all this that forks the viewer, then waits for an IN_OPEN event on its argument via inotify before exiting.

I wonder if there is any reason run-mailcap could not be implemented as a wrapper to xdg-open.

I reported #964723 elaborating on these thoughts.

Enrico Zini: Laptop migration

Thursday 9th of July 2020 07:34:10 AM

My laptop battery started to explode in slow motion. HP requires 10 business days to repair my laptop under warranty, and I cannot afford that length of downtime.

Alternatively, HP quoted me 375€ + VAT for on-site repairs, which I tought was very funny.

For 376.55€ + VAT, which is pretty much exactly the same amount, I bought instead a refurbished ThinkPad X240 with a dual-core I5, 8G of RAM, 250G SSD, and a 1920x1080 IPS display, to use as a spare while my laptop is being repaired. I'd like to thank HP for giving me the opportunity to own a ThinkPad.

Since I'm migrating all my system to the spare and then (hopefully) back, I'm documenting what I need to be fully productive on new hardware.

Install Debian

A basic Debian netinst with no tasks selected is good enough to get going.

Note that if wifi worked in Debian Installer, it doesn't mean that it will work in the minimal system it installed. See here for instructions on quickly bringing up wifi on a newly installed minimal system.

Copy /home

A simple tar of /home is all I needed to copy my data over.

A neat way to do it was connecting the two laptops with an ethernet cable, and using netcat:

# On the source tar -C / -zcf - home | nc -l -p 12345 -N # On the target nc 10.0.0.1 12345 | tar -C / -zxf -

Since the data travel unencrypted in this way, don't do it over wifi.

Install packages

I maintain a few simple local metapackages that depend on the packages I usually used.

I could just install those and let apt bring in their dependencies.

For the build dependencies of the programs I develop, I use mk-build-deps from the devscripts package to create metapackages that make sure they are installed.

Here's an extract from debian/control of the metapackage:

Source: enrico Section: admin Priority: optional Maintainer: Enrico Zini <enrico@debian.org> Build-Depends: debhelper (>= 11) Standards-Version: 3.7.2.1 Package: enrico Section: admin Architecture: all Depends: mc, mmv, moreutils, powertop, syncmaildir, notmuch, ncdu, vcsh, ddate, jq, git-annex, eatmydata, vdirsyncer, khal, etckeeper, moc, pwgen Description: Enrico's working environment Package: enrico-devel Section: devel Architecture: all Depends: git, python3-git, git-svn, gitk, ansible, fabric, valgrind, kcachegrind, zeal, meld, d-feet, flake8, mypy, ipython3, strace, ltrace Description: Enrico's development environment Package: enrico-gui Section: x11 Architecture: all Depends: xclip, gnome-terminal, qalculate-gtk, liferea, gajim, mumble, sm, syncthing, virt-manager Recommends: k3b Description: Enrico's GUI environment Package: enrico-sanity Section: admin Architecture: all Conflicts: libapache2-mod-php, libapache2-mod-php5, php5, php5-cgi, php5-fpm, libapache2-mod-php7.0, php7.0, libphp7.0-embed, libphp-embed, libphp5-embed Description: Enrico's sanity Metapackage with a list of packages that I do not want anywhere near my system. System-wide customizations

I tend to avoid changing system-wide configuration as much as possible, so copying over /home and installing packages takes care of 99% of my needs.

There are a few system-wide tweaks I cannot do without:

  • setup postfix to send mail using my mail server
  • copy Network Manager system connections files in /etc/NetworkManager/system-connections/
  • update-alternatives --config editor

For postfix, I have a little ansible playbook that takes care of it.

Network Manager system connections need to be copied manually: a plain copy and a systemctl restart network-manager are enough. Note that Network Manager will ignore the files unless their owner and permissions are what it expects.

Fine tuning

Comparing the output of dpkg --get-selections between the old and the new system might highlight packages manually installed in a hurry and not added to the metapackages.

Finally, what remains is fixing the sad state of mimetype associations, which seem to associate opening file depending on whatever application was installed last, phases of the moon, and what option is the most annoying.

Currently on my system, PDFs are opened in inkscape by xdg-open and in calibre by run-mailcap. Let's see how long it takes to figure this one out.

Noah Meyerhans: Setting environment variables for gnome-session

Tuesday 7th of July 2020 11:51:25 PM

Am I missing something obvious? When did this get so hard?

In the old days, you configured your desktop session on a Linux system by editing the .xsession file in your home directory. The display manager (login screen) would invoke the system-wide xsession script, which would either defer to your personal .xsession script or set up a standard desktop environment. You could put whatever you want in the .xsession script, and it would be executed. If you wanted a specific window manager, you’d run it from .xsession. Start emacs or a browser or an xterm or two? .xsession. It was pretty easy, and super flexible.

For the past 25 years or so, I’ve used X with an environment started via .xsession. Early on it was fvwm with some programs, then I replaced fvwm with Window Maker (before that was even its name!), then switched to KDE. More recently (OK, like 10 years ago) I gradually replaced KDE with awesome and various custom widgets. Pretty much everything was based on a .xsession script, and that was fine. One particularly nice thing about it was that I could keep .xsession and any related helper programs in a git repository and manage changes over time.

More recently I decided to give Wayland and GNOME an honest look. This has mostly been fine, but everything I’ve been doing in .xsession is suddenly useless. OK, fine, progress is good. I’ll just use whatever new mechanisms exist. How hard can it be?

OK, so here we go. I am running GNOME. This isn’t so bad. Alt+F2 brings up the “Run Command” dialog. It’s a different keystroke than what I’m used to, but I can adapt. (Obviously I can reconfigure the key binding, and maybe someday I will, but that’s not the point here.) I have some executables in ~/bin. Oops, the run command dialog can’t find them. No problem, I just need to update the PATH variable that it sees. Hmmm… So how does one do that, anyway? GNOME has a help system, but searching that doesn’t doesn’t reveal anything. But that’s fine, maybe it’s inherited from the parent process. But there’s no xsession script equivalent, since this isn’t X anymore at all. The familiar stuff in /etc/X11/Xsession is no longer used. What’s the equivalent in Wayland? Turns out, there isn’t a shell script at all anymore, at least not in how Wayland and GNOME interact in Debian’s configuration, which seems fairly similar to how anybody else would set this up. The GNOME session runs from a systemd-managed user session.

Digging in to some web search results suggests that systemd provides a mechanism for setting some environment variables for services started by the user instance of the system. OK, so let’s create some files in ~/.config/environment.d and we should be good. Except no, this isn’t working. I can set some variables, but something is overriding PATH. I can create this file:

$ cat ~/.config/environment.d/01_path.conf USER_INITIAL_PATH=${PATH} PATH=${HOME}/bin:${HOME}/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin USER_CUSTOM_PATH=${PATH}

After logging in, the “Run a command” dialog still doesn’t see my PATH. So I use Alt+F2 and sh -c "env > /tmp/env" to capture the environment, and this is what I see:

USER_INITIAL_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PATH=/usr/local/bin:/usr/bin:/bin:/usr/games USER_CUSTOM_PATH=/home/noahm/bin:/home/noahm/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

So, my environment.d file is there, and it’s getting looked at, but something else is clobbering my PATH later in the startup process. But what? Where? Why? The systemd docs don’t indicate that there’s anything special about PATH, and nothing in /lib/systemd/user-environment-generators/ seems to treat it specially. The string “PATH” doesn’t appear in /lib/systemd/user/ either. Looking for the specific value that’s getting assigned to PATH in /etc shows the only occurrence of it being in /etc/zsh/zshenv, so maybe that’s where it’s coming from? But that should only get set there if it’s otherwise unset or otherwise very minimally set. So I still have no idea where it’s coming from.

OK, so ignoring where my custom value is getting overridden, maybe what’s configured in /lib/systemd/user will point me in the right direction. systemd --user status suggests that the interesting part of my session is coming from gnome-shell-wayland.service. Can we use a standard systemd drop-in as documented in systemd.unit(5)? It turns out that we can. This file sets things up the way I want:

$ cat .config/systemd/user/gnome-shell-wayland.service.d/path.conf [Service] Environment=PATH=%h/bin:%h/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Is that right? It really doesn’t feel ideal to me. Systemd’s Environment directive can’t reference existing environment variables, and I can’t use conditionals to do things like add a directory to the PATH only if it exists, so it’s still a functional regression from what we had before. But at least it’s a text file, edited by hand, trackable in git, so that’s not too bad.

There are some people out there who hate systemd, and will cite this as an illustration of why. However, I’m not one of those people, and I very much like systemd as an init system. I’d be happy to throw away sysvinit scripts forever, but I’m not quite so happy with the state of .xsession’s replacements. Despite the similarities, I don’t think .xsession is entirely the same as SysV-style init scripts. The services running on a system are vastly more important than my personal .xsession, and systemd is far better at managing them than the pile of shell scripts used to set things up under sysvinit. Further, systemd the init system maintains compatibility with init scripts, so if you really want to keep using them, you can. As far as I can tell, though, systemd the user session manager does not seem to maintain compatibility with .xsession scripts, and that’s unfortunate.

I still haven’t figured out what was overriding the ~/.config/environment.d/ setting. Any ideas?

Dirk Eddelbuettel: RcppSimdJson 0.1.0: Now on Windows, With Parsers and Faster Still!

Tuesday 7th of July 2020 10:45:00 PM

A smashing new RcppSimdJson release 0.1.0 containing several small updates to upstream simdjson (now at 0.4.6) in part triggered by very excisting work by Brendan who added actual parser from file and string—and together with Daniel upstream worked really hard to make Windows builds as well as complete upstream tests on our beloved (ahem) MinGW platform possible. So this version will, once the builders have caught up, give everybody on Windows a binary—with a JSON parser running circles around the (arguably more feature-rich and possibly easier-to-use) alternatives. Dave just tweeted a benchmark snippet by Brendan, the full set is at the bottom our issue ticket for this release.

RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators, which in its upstream release 0.4.0 improved once more (also see the following point releases). Via very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is ‘faster than CPU speed’ as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle use per byte parsed; see the video of the recent talk by Daniel Lemire at QCon (which was also voted best talk).

As mentioned, this release expands the reach of the package to Windows, and adds new user-facing functions. A big thanks for most of this is owed to Brendan, so buy him a drink if you run across him. The full NEWS entry follows.

Changes in version 0.1.0 (2020-07-07)
  • Upgraded to simdjson 0.4.1 which adds upstream Windows support (Dirk in #27 closing #26 and #14, plus extensive work by Brendan helping upstream with mingw tests).

  • Upgraded to simdjson 0.4.6 with further upstream improvements (Dirk in #30).

  • Change Travis CI to build matrix over g++ 7, 8, 9, and 10 (Dirk in #31; and also Brendan in #32).

  • New JSON functions fparse and fload (Brendan in #32) closing #18 and #10).

Courtesy of CRANberries, there is also a diffstat report for this release.

For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

More in Tux Machines

Linux Devices and Open Hardware

  • Mini-PC and SBC build on Whiskey Lake

    Supermicro’s 3.5-inch “X11SWN-H-WOHS” SBC and “SYS-E100-9W-H” mini-PC based it feature an 8th Gen UE-series CPU, HDMI and DP, 4x USB 3.1 Gen2, 2x GbE, and 3x M.2. Supermicro has launched a fanless, 8th Gen Whiskey Lake SBC and mini-PC. The SYS-E100-9W-H mini-PC (or SuperServer E100-9W-H), which was reported on by Fanless Tech, is certified only to run Windows 10, but the 3.5-inch X11SWN-H-WOHS SBC supports Ubuntu. Applications include industrial automation, retail, smart medical expert systems, kiosks, interactive info systems, and digital signage.

  • Exor nanoSOM nS02 System-on-Module Features the 800MHz version of STM32MP1 Processor

    Exor provides a Linux RT board support package (BSP) or Android BSP for the module which also fully supports the company’s X Platform including Exor Embedded Open HMI software, Corvina Cloud IIoT platform, and IEC61131 CODESYS or Exor xPLC runtime.

  • Onyx Boox Poke2 Color eReader Launched for $299

    Manga and comics fans, rejoice! After years of getting black & white eReaders, the first commercial color eReaders are coming to market starting with Onyx Boox Poke2 Color eReader sold for $299 (but sadly sold out at the time of writing). The eReader comes with a 6-inch, 1448 x 1072 E-Ink display that supports up to 4096 colors, and runs Android 9.0 on an octa-core processor coupled with 2GB RAM and 32GB storage.

  • xDrill Smart Power Drill Supports Intelligent Speed/Torque, Laser Measuring, Digital Leveling (Crowdfunding)

    Many home appliances now have smart functions, and in my cases, I fail to see the added value, and I’m not sure why I’d want/need a connected refrigerator with a touchscreen display. So when I first saw somebody make a “smart” power drill with a small touchscreen display I laughed. But after having a closer look, Robbox xDrill smart power drill could actually be a very useful device saving you time and helping work better.

  • Raspberry Pi calls out your custom workout routine
  • Odyssey Blue: A powerful x86 and Arduino machine that supports Windows 10 and Linux

    It has been a few months since we reported on the Odyssey, a single-board computer (SBC) designed by Seeedstudio. Unlike many SBCs, the Odyssey, or ODYSSEY-X86J4105800 to give it its full name, supported the x86 instruction set. While the Odyssey can run Windows 10, it is also compatible with the Arduino ecosystem. Now, Seeedstudio has expanded on the design of the Odyssey with the Odyssey Blue.

  • Bring two analog meters out of retirement to display temperature and humidity

    Tom of Build Comics created a unique analog weather station that shows temperature and humidity on a pair of recycled gauges. An Arduino Nano reads the levels using a DHT22 sensor and outputs them in the proper format for each display. Both units have a new printed paper backing to indicate conditions, along with a trimmer pot for calibration. To set the build off nicely, the Nano and other electronics are housed inside a beautiful custom wooden box, to which the antique meters are also affixed.

Programming Leftovers

  • Engineer Your Own Electronics With PCB Design Software

    A lot of self-styled geeks out there tend to like to customize their own programs, devices, and electronics. And for the true purists, that can mean building from the ground up (you know, like Superman actor Henry Cavill building a gaming PC to the delight of the entire internet). Building electronics from the ground up can mean a lot of different things: acquiring parts, sometimes from strange sources; a bit of elbow grease on the mechanical side of things; and today, even taking advantage of the 3D printing revolution that’s finally enabling people to manufacture customized objects in their home. Beyond all of these things though, engineering your own devices can also mean designing the underlying electronics — beginning with printed circuit boards, also known as PCBs. [...] On the other hand, there are also plenty of just-for-fun options to consider. For example, consider our past buyer’s guide to the best Linux laptop, in which we noted that you can always further customize your hardware. With knowledge of PCB design, that ability to customize even a great computer or computer setup is further enhanced. You might, for instance, learn how to craft PCBs and devices amounting to your own mouse, gaming keyboard, or homemade speakers — all of which can make your hardware more uniquely your own. All in all, PCB design is a very handy skill to have in 2020. It’s not typically necessary, in that there’s usually a device or some light customization that can give you whatever you want or need out of your electronics. But for “geeks” and tech enthusiasts, knowledge of PCB design adds another layer to the potential to customize hardware.

  • Programming pioneer Fran Allen dies aged 88 after a career of immense contributions to compilers

    Frances Allen, one of the leading computer scientists of her generation and a pioneer of women in tech, died last Tuesday, her 88th birthday. Allen is best known for her work on compiler organisation and optimisation algorithms. Together with renowned computer scientist John Cocke, she published a series of landmark papers in the late '60s and '70s that helped to lay the groundwork for modern programming. In recognition of her efforts, in 2006 Allen became the first woman to be awarded the AM Turing Award, often called the Nobel Prize of computing.

  • Excellent Free Tutorials to Learn ECMAScript

    ECMAScript is an object‑oriented programming language for performing computations and manipulating computational objects within a host environment. The language was originally designed as a scripting language, but is now often used as a general purpose programming language. ECMAScript is best known as the language embedded in web browsers but has also been widely adopted for server and embedded applications.

  • Alexander Larsson: Compatibility in a sandboxed world

    Compatibility has always been a complex problems in the Linux world. With the advent of containers/sandboxing it has become even more complicated. Containers help solve compatibility problems, but there are still remaining issues. Especially on the Linux desktop where things are highly interconnected. In fact, containers even create some problems that we didn’t use to have. Today I’ll take a look at the issues in more details and give some ideas on how to best think of compatibility in this post-container world, focusing on desktop use with technologies like flatpak and snap. [...] Another type of compatibility is that of communication protocols. Two programs that talk to each other using a networking API (which could be on two different machines, or locally on the same machine) need to use a protocol to understand each other. Changes to this protocol need to be carefully considered to ensure they are compatible. In the remote case this is pretty obvious, as it is very hard to control what software two different machines use. However, even for local communication between processes care has to be taken. For example, a local service could be using a protocol that has several implementations and they all need to stay compatible. Sometimes local services are split into a service and a library and the compatibility guarantees are defined by the library rather than the service. Then we can achieve some level of compatibility by ensuring the library and the service are updated in lock-step. For example a distribution could ship them in the same package.

  • GXml-0.20 Released

    GXml is an Object Oriented implementation of DOM version 4, using GObject classes and written in Vala. Has a fast and robust serialization implementation from GObject to XML and back, with a high degree of control. After serialization, provides a set of collections where you can get access to child nodes, using lists or hash tables. New 0.20 release is the first step toward 1.0. It provides cleaner API and removes old unmaintained implementations. GXml is the base of other projects depending on DOM4, like GSVG an engine to read SVG documents based on its specificacion 1.0. GXml uses a method to set properties and fill declared containers for child nodes, accessing GObject internals directly, making it fast. A libxml-2.0 engine is used to read sequentially each node, but is prepared to implement new ones in the future.

  • Let Mom Help You With Object-Oriented Programming

    Mom is a shortcut for creating Moo classes (and roles). It allows you to define a Moo class with the brevity of Class::Tiny. (In fact, Mom is even briefer.) A simple example: Mom allows you to use Moo features beyond simply declaring Class::Tiny-like attributes though. You can choose whether attributes are read-only, read-write, or read-write-private, whether they're required or optional, specify type constraints, defaults, etc.

  • Perl Weekly Challenge 73: Min Sliding Window and Smallest Neighbor

    These are some answers to the Week 73 of the Perl Weekly Challenge organized by Mohammad S. Anwar. Spoiler Alert: This weekly challenge deadline is due in a few days from now (on Aug. 16, 2020). This blog post offers some solutions to this challenge, please don’t read on if you intend to complete the challenge on your own.

  • [rakulang] 2020.32 Survey, Please

    The TPF Marketing Committee wants to learn more about how you perceive “The Perl Foundation” itself, and asks you to fill in this survey (/r/rakulang, /r/perl comments). Thank you!

Hardware With Linux Support: NUVIA and AMD Wraith Prism

  • Performance Delivered a New Way

    The server CPU has evolved at an incredible pace over the last two decades. Gone are the days of discrete CPUs, northbridges, southbridges, memory controllers, other external I/O and security chips. In today’s modern data center, the SoC (System On A Chip) does it all. It is the central point of coordination for virtually all workloads and the main hub where all the fixed-function accelerators connect, such as AI accelerators, GPUs, network interface controllers, storage devices, etc.

  • NUVIA Published New Details On Their Phoenix CPU, Talks Up Big Performance/Perf-Per-Watt

    Since leaving stealth last year and hiring some prominent Linux/open-source veterans to complement their ARM processor design experts, we have been quite eager to hear more about this latest start-up aiming to deliver compelling ARM server products. Today they shared some early details on their initial "Phoenix" processor that is coming within their "Orion" SoC. The first-generation Phoenix CPU is said to have a "complete overhaul" of the CPU pipeline and is a custom core based on the ARM architecture. They believe that Phoenix+Orion will be able to take on Intel/AMD x86_64 CPUs not only in raw performance but also in performance-per-Watt.

  • Take control of your AMD Wraith Prism RGB on Linux with Wraith Master

    Where the official vendor doesn't bother with supporting Linux properly, once again the community steps in to provide. If you want to tweak your AMD Wraith Prism lighting on Linux, check out Wraith Master. It's a similar project to CM-RGB that we previously highlighted. With the Wraith Master project, they provide a "feature-complete" UI and command-line app for controlling the fancy LED system on AMD's Wraith Prism cooler with eventual plans to support more.

The Massive Privacy Loopholes in School Laptops

It’s back to school time and with so many school districts participating in distance learning, many if not most are relying on computers and technology more than ever before. Wealthier school districts are providing their students with laptops or tablets, but not all schools can afford to provide each student with a computer which means that this summer parents are scrambling to find a device for their child to use for school. Geoffery Fowler wrote a guide in the Washington Post recently to aid parents in sourcing a computer or tablet for school. Given how rough kids can be with their things, many people are unlikely to give their child an expensive, premium laptop. The guide mostly focuses on incredibly low-cost, almost-disposable computers, so you won’t find a computer in the list that has what I consider a critical feature for privacy in the age of video conferencing: hardware kill switches. Often a guide like this would center on Chromebooks as Google has invested a lot of resources to get low-cost Chromebooks into schools yet I found Mr. Fowler’s guide particularly interesting because of his opinion on Chromebooks in education... Read more Also: Enabling Dark Mode on a Chromebook (Do not try this at home)