Language Selection

English French German Italian Portuguese Spanish

Debian

Syndicate content
Planet Debian - https://planet.debian.org/
Updated: 7 hours 45 min ago

Dirk Eddelbuettel: x13binary 1.1.39-2

Wednesday 8th of May 2019 02:26:00 AM

An updated x13binary package 1.1.39-2 of the X-13ARIMA-SEATS program by the US Census Bureau (with upstream release 1.1.39) is now on CRAN, pretty much exactly two years after the previous release 1.1.39-1.

The x13binary package takes the pain out of installing X-13ARIMA-SEATS by making it a fully resolved CRAN dependency. For example, if you install the excellent seasonal package by Christoph, then X-13ARIMA-SEATS will get pulled in via the x13binary package and things just work: Depend on x13binary and on all relevant OSs supported by R, you should have an X-13ARIMA-SEATS binary installed which will be called seamlessly by the higher-level packages such as seasonal or gunsales. With this the full power of the what is likely the world’s most sophisticated deseasonalization and forecasting package is now at your fingertips and the R prompt, just like any other of the 14100+ CRAN packages. You can read more about this (and the seasonal package) in the recent Journal of Statistical Software paper by Christoph and myself.

There is almost no change in this release – apart from having to force StagedInstall: no following the R 3.6.0 release as the macOS build is otherwise broken now.

Courtesy of CRANberries, there is also a diffstat report for this release showing changes to the previous release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Shirish Agarwal: Debutsav Mumbai and itsfoss.com changes

Monday 6th of May 2019 06:29:22 PM

Before starting of today’s blog post I will be sharing a video of food which will definitely make you hungry. So make sure you are either eating something or have eaten fully before seeing the video below.

The video is from a place called Aaoji Khaaoji Restaurant which is on J.M. Road . It is a proper foodie place. They have two offers there, if you can eat their whole Thali by yourself they will either feed you your whole life or you can take double the money you paid. Don’t think anybody has been even able to finish half of that thali.

Debutsav Mumbai

While I and a few members of Debian India has been trying to get a debutsav Mumbai happening, now we have the dates for the event as it was announced today on the mailing list. While there are definitely lot of things that would need to be done in order for a successful Debtusav to happen, at least we have got the dates so other things can get start moving.

Itsfoss.com

Another thing that I am sharing is that itsfoss.com, where I write debian-related articles would be going through some interesting changes. While I’m not at liberty to share the changes, the changes are and would be interesting to say the least. The only hint or giveaway I can give is that the changes may be seen in the new few weeks or at the most a month. All the rest, till later

Olivier Berger: Running networking labs over Kubernetes with Antidote

Monday 6th of May 2019 02:42:30 PM

I’ve just come across Antidote a recent project that intends at running networking-oriented labs over Kubernetes. It is developped by members of the Network Reliability Engineering community (Juniper-related, AFAIU), to power the NRE Labs platform.

It looks very similar to other platforms that allow you to run labs remotely in consoles opened on cloud machines, alongside lab instructions.

I find it interesting as the code is published under FLOSS license (Apache), and seems to be runable over any Kubernetes installation: you can test it with Minikube through the selfmedicate repo.

Antidote demo running virtual labs in Kubernetes with selfmedicate/minikube, running locally from Olivier Berger on Vimeo.

Internally, it uses Guacamole to provide the Web consoles connected via SSH to the hosts (or emulated devices) running on the k8s cluster. Each learner will get her own k8s namespace running the corresponding PODs.

In principle, it’s rather easy to package any app that can be used from the CLI to run it over Antidote.

The main drawback I’ve found so far, wrt our experiments with virtual labs, is the limitation to SSH access for a CLI: the Guacamole setup doesn’t provide access to VNC, AFAICS (yet).

Quite interesting and promising project anyway.

Julien Danjou: An Introduction to Functional Programming with Python

Monday 6th of May 2019 08:58:00 AM

Many Python developers are unaware of the extent to which you can use functional programming in Python, which is a shame: with few exceptions, functional programming allows you to write more concise and efficient code. Moreover, Python’s support for functional programming is extensive.

Here I'd like to talk a bit about how you can actually have a functional approach to programming with our favorite language.

Pure Functions

When you write code using a functional style, your functions are designed to have no side effects: instead, they take an input and produce an output without keeping state or modifying anything not reflected in the return value. Functions that follow this ideal are referred to as purely functional.

Let’s start with an example of a regular, non-pure function that removes the last item in a list:

def remove_last_item(mylist): """Removes the last item from a list.""" mylist.pop(-1) # This modifies mylist

This function is not pure: it has a side effect as it modifies the argument it is given. Let's rewrite it as purely functional:

def butlast(mylist): """Like butlast in Lisp; returns the list without the last element.""" return mylist[:-1] # This returns a copy of mylist

We define a butlast() function (like butlast in Lisp) that returns the list without the last element without modifying the original list. Instead, it returns a copy of the list that has the modifications in place, allowing us to keep the original. The practical advantages of using functional programming include the following:

  • Modularity. Writing with a functional style forces a certain degree of
    separation in solving your individual problems and makes sections of code
    easier to reuse in other contexts. Since the function does not depend on any
    external variable or state, call it from a different piece of code is
    straightforward.
  • Brevity. Functional programming is often less verbose than other paradigms.
  • Concurrency. Purely functional functions are thread-safe and can run
    concurrently. Some functional languages do this automatically, which can be
    a big help if you ever need to scale your application, though this is not
    quite the case yet in Python.
  • Testability. Testing a functional program is incredibly easy: all you need
    is a set of inputs and an expected set of outputs. They are idempotent,
    meaning that calling the same function over and over with the same arguments
    will always return the same result.

Note that concepts such as list comprehension in Python are already functionals in their approach, as they are designed to avoid side effects. We'll see in the following that some of the functional functions Python provide can actually be expressed as list comprehension!

Python Functional Functions

You might repeatedly encounter the same set of problems when manipulating data using functional programming. To help you deal with this situation efficiently, Python includes a number of functions for functional programming. Here, we'll see with a quick overview some of these built-in functions that allows you to build fully functional programs. Once you have an idea of what’s available, I encourage you to research further and try out functions where they might apply in your own code.

Applying Functions to Items with map

The map() function takes the form map(function, iterable) and applies function to each item in iterable to return an iterable map object:

>>> map(lambda x: x + "bzz!", ["I think", "I'm good"]) <map object at 0x7fe7101abdd0> >>> list(map(lambda x: x + "bzz!", ["I think", "I'm good"])) ['I thinkbzz!', "I'm goodbzz!"]

You could also write an equivalent of map() using list comprehension, which
would look like this:

>>> (x + "bzz!" for x in ["I think", "I'm good"]) <generator object <genexpr> at 0x7f9a0d697dc0> >>> [x + "bzz!" for x in ["I think", "I'm good"]] ['I thinkbzz!', "I'm goodbzz!"] Filtering Lists with filter

The filter() function takes the form filter(function or None, iterable) and filters the items in iterable based on the result returned by function. This will return iterable filter object:

>>> filter(lambda x: x.startswith("I "), ["I think", "I'm good"]) <filter object at 0x7f9a0d636dd0> >>> list(filter(lambda x: x.startswith("I "), ["I think", "I'm good"])) ['I think']

You could also write an equivalent of filter() using list comprehension, like
so:

>>> (x for x in ["I think", "I'm good"] if x.startswith("I ")) <generator object <genexpr> at 0x7f9a0d697dc0> >>> [x for x in ["I think", "I'm good"] if x.startswith("I ")] ['I think'] Getting Indexes with enumerate

The enumerate() function takes the form enumerate(iterable[, start]) and returns an iterable object that provides a sequence of tuples, each consisting of an integer index (starting with start, if provided) and the corresponding item in iterable. This function is useful when you need to write code that refers to array indexes. For example, instead of writing this:

i = 0 while i < len(mylist): print("Item %d: %s" % (i, mylist[i])) i += 1

You could accomplish the same thing more efficiently with enumerate(), like so:

for i, item in enumerate(mylist): print("Item %d: %s" % (i, item)) Sorting a List with sorted

The sorted() function takes the form sorted(iterable, key=None, reverse=False) and returns a sorted version of iterable. The key argument allows you to provide a function that returns the value to sort on:

>>> sorted([("a", 2), ("c", 1), ("d", 4)]) [('a', 2), ('c', 1), ('d', 4)] >>> sorted([("a", 2), ("c", 1), ("d", 4)], key=lambda x: x[1]) [('c', 1), ('a', 2), ('d', 4)] Finding Items That Satisfy Conditions with any and all

The any(iterable) and all(iterable) functions both return a Boolean depending on the values returned by iterable. These simple functions are equivalent to the following full Python code:

def all(iterable): for x in iterable: if not x: return False return True def any(iterable): for x in iterable: if x: return True return False

These functions are useful for checking whether any or all of the values in an iterable satisfy a given condition. For example, the following checks a list for two conditions:

mylist = [0, 1, 3, -1] if all(map(lambda x: x > 0, mylist)): print("All items are greater than 0") if any(map(lambda x: x > 0, mylist)): print("At least one item is greater than 0")

The key difference here, as you can see, is that any() returns True when at least one element meets the condition, while all() returns True only if every element meets the condition. The all() function will also return True for an empty iterable, since none of the elements is False.

Combining Lists with zip

The zip() function takes the form zip(iter1 [,iter2 [...]]) and takes multiple sequences and combines them into tuples. This is useful when you need to combine a list of keys and a list of values into a dict. Like the other functions described here, zip() returns an iterable. Here we have a list of keys that we map to a list of values to create a dictionary:

>>> keys = ["foobar", "barzz", "ba!"] >>> map(len, keys) <map object at 0x7fc1686100d0> >>> zip(keys, map(len, keys)) <zip object at 0x7fc16860d440> >>> list(zip(keys, map(len, keys))) [('foobar', 6), ('barzz', 5), ('ba!', 3)] >>> dict(zip(keys, map(len, keys))) {'foobar': 6, 'barzz': 5, 'ba!': 3} What's Next?

While Python is often advertised as being object oriented, it can be used in a very functional manner. A lot of its built-in concepts, such as generators and list comprehension, are functionally oriented and don’t conflict with an object-oriented approach. Python provides a large set of builtin functions that can help you keeping your code with no side effects. That also limits the reliance on a program’s global state, for your own good.

In the next blog post, we'll see how you can leverage Python functools and itertools module to enhance your functional adventure. Stay tuned!

Sergio Durigan Junior: Improve gcore and support dumping ELF headers

Monday 6th of May 2019 03:45:38 AM

Back in 2016, when life was simpler, a Fedora GDB user reported a bug (or a feature request, depending on how you interpret it) saying that GDB's gcore command did not respect the COREFILTER_ELF_HEADERS flag, which instructs it to dump memory pages containing ELF headers. As you may or may not remember, I have already written about the broader topic of revamping GDB's internal corefile dump algorithm; it's an interesting read and I recommend it if you don't know how Linux (or GDB) decides which mappings to dump to a corefile.

Anyway, even though the bug was interesting and had to do with a work I'd done before, I couldn't really work on it at the time, so I decided to put it in the TODO list. Of course, the "TODO list" is actually a crack where most things fall through and are usually never seen again, so I was blissfully ignoring this request because I had other major priorities to deal with. That is, until a seemingly unrelated problem forced me to face this once and for all!

What? A regression? Since when?

As the Fedora GDB maintainer, I'm routinely preparing new releases for Fedora Rawhide distribution, and sometimes for the stable versions of the distro as well. And I try to be very careful when dealing with new releases, because a regression introduced now can come and bite us (i.e., the Red Hat GDB team) back many years in the future, when it's sometimes too late or too difficult to fix things. So, a mandatory part of every release preparation is to actually run a regression test against the previous release, and make sure that everything is working correctly.

One of these days, some weeks ago, I had finished running the regression check for the release I was preparing when I noticed something strange: a specific, Fedora-only corefile test was FAILing. That's a no-no, so I started investigating and found that the underlying reason was that, when the corefile was being generated, the build-id note from the executable was not being copied over. Fedora GDB has a local patch whose job is to, given a corefile with a build-id note, locate the corresponding binary that generated it. Without the build-id note, no binary was being located.

Coincidentally or not, at the same I started noticing some users reporting very similar build-id issues on the freenode's #gdb channel, and I thought that this bug had a potential to become a big headache for us if nothing was done to fix it right now.

I asked for some help from the team, and we managed to discover that the problem was also happening with upstream gcore, and that it was probably something that binutils was doing, and not GDB. Hmm...

Ah, so it's ld's fault. Or is it?

So there I went, trying to confirm that it was binutils's fault, and not GDB's. Of course, if I could confirm this, then I could also tell the binutils guys to fix it, which meant less work for us :-).

With a lot of help from Keith Seitz, I was able to bisect the problem and found that it started with the following commit:

commit f6aec96dce1ddbd8961a3aa8a2925db2021719bb Author: H.J. Lu <hjl.tools@gmail.com> Date: Tue Feb 27 11:34:20 2018 -0800 ld: Add --enable-separate-code

This is a commit that touches the linker, which is part of binutils. So that means this is not GDB's problem, right?!? Hmm. No, unfortunately not.

What the commit above does is to simply enable the use of --enable-separate-code (or -z separate-code) by default when linking an ELF program on x86_64 (more on that later). On a first glance, this change should not impact the corefile generation, and indeed, if you tell the Linux kernel to generate a corefile (for example, by doing sleep 60 & and then hitting C-\), you will notice that the build-id note is included into it! So GDB was still a suspect here. The investigation needed to continue.

What's with -z separate-code?

The -z separate-code option makes the code segment in the ELF file to put in a completely separated segment than data segment. This was done to increase the security of generated binaries. Before it, everything (code and data) was put together in the same memory region. What this means in practice is that, before, you would see something like this when you examined /proc/PID/smaps:

00400000-00401000 r-xp 00000000 fc:01 798593 /file Size: 4 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd ex mr mw me dw sd

And now, you will see two memory regions instead, like this:

00400000-00401000 r--p 00000000 fc:01 799548 /file Size: 4 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 4 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd mr mw me dw sd 00401000-00402000 r-xp 00001000 fc:01 799548 /file Size: 4 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd ex mr mw me dw sd

A few minor things have changed, but the most important of them is the fact that, before, the whole memory region had anonymous data in it, which means that it was considered an anonymous private mapping (anonymous because of the non-zero Anonymous amount of data; private because of the p in the r-xp permission bits). After -z separate-code was made default, the first memory mapping does not have Anonymous contents anymore, which means that it is now considered to be a file-backed private mapping instead.

GDB, corefile, and coredump_filter

It is important to mention that, unlike the Linux kernel, GDB doesn't have all of the necessary information readily available to decide the exact type of a memory mapping, so when I revamped this code back in 2015 I had to create some heuristics to try and determine this information. If you're curious, take a look at the linux-tdep.c file on GDB's source tree, specifically at the functions dump_mapping_p and linux_find_memory_regions_full.

When GDB is deciding which memory regions should be dumped into the corefile, it respects the value found at the /proc/PID/coredump_filter file. The default value for this file is 0x33, which, according to core(5), means:

Dump memory pages that are either anonymous private, anonymous shared, ELF headers or HugeTLB.

GDB had the support implemented to dump almost all of these pages, except for the ELF headers variety. And, as you can probably infer, this means that, before the -z separate-code change, the very first memory mapping of the executable was being dumped, because it was marked as anonymous private. However, after the change, the first mapping (which contains only data, no code) wasn't being dumped anymore, because it was now considered by GDB to be a file-backed private mapping!

Finally, that is the reason for the difference between corefiles generated by GDB and Linux, and also the reason why the build-id note was not being included in the corefile anymore! You see, the first memory mapping contains not only the program's data, but also its ELF headers, which in turn contain the build-id information.

gcore, meet ELF headers

The solution was "simple": I needed to improve the current heuristics and teach GDB how to determine if a mapping contains an ELF header or not. For that, I chose to follow the Linux kernel's algorithm, which basically checks the first 4 bytes of the mapping and compares them against \177ELF, which is ELF's magic number. If the comparison succeeds, then we just assume we're dealing with a mapping that contains an ELF header and dump it.

In all fairness, Linux just dumps the first page (4K) of the mapping, in order to save space. It would be possible to make GDB do the same, but I chose the faster way and just dumped the whole mapping, which, in most scenarios, shouldn't be a big problem.

It's also interesting to mention that GDB will just perform this check if:

  • The heuristic has decided not to dump the mapping so far, and;
  • The mapping is private, and;
  • The mapping's offset is zero, and;
  • There is a request to dump mappings with ELF headers (i.e., coredump_filter).

Linux also makes these checks, by the way.

The patch, finally

I submitted the patch to the mailing list, and it was approved fairly quickly (with a few minor nits).

The reason I'm writing this blog post is because I'm very happy and proud with the whole process. It wasn't an easy task to investigate the underlying reason for the build-id failures, and it was interesting to come up with a solution that extended the work I did a few years ago. I was also able to close a few bug reports upstream, as well as the one reported against Fedora GDB.

The patch has been pushed, and is also present at the latest version of Fedora GDB for Rawhide. It wasn't possible to write a self-contained testcase for this problem, so I had to resort to using an external tool (eu-unstrip) in order to guarantee that the build-id note is correctly present in the corefile. But that's a small detail, of course.

Anyway, I hope this was an interesting (albeit large) read!

Shirish Agarwal: A tale of broken promises

Sunday 5th of May 2019 08:50:23 PM

Before I venture out I would like to share with all one of my old classic favorite songs –

It’s a beautiful soft song and there are many such songs that were composed in the 1960’s. This specific song is from the movie ‘Woh Kaun Thi’ . The original song was sung by Indian Nigtingale Mrs. Lata Mangeshkar. The newer sound that you hear though is of Shreya Ghosal.

At this time the election scenario has been where all promises have been broken. While in the last blog post I had promised I will share some of the issues lot of events have taken in-between and there also had been lot of memories of broken promises by the ruling party which has lead not just me but probably lot of Puneties unhappy and disappointed. I will try and share some of them.

2014 Water pipeline promises

Pune has been facing water shortages since the last decade or so. In fact the situation has turned from bad to worse. The previous government promises they will make lot of pipelines and were voted to power. There was water pipeline scam and the Government changed in 2014. From 2014 to 2019 the water crisis has worsened. In my own home, we get water for only 2-3 hours daily. We have not been able to entertain any families for more than a day because of water issues. The only fortunate part is that I have to walk at the most 20 meters to take water while in some places people have to go miles or do dangerous things such as one I’m sharing below.

As can be seen they are going 60 feet down a well without any support structure or even a rope which means they can fall, injure or even die at any moment, a false step. And this they do everyday in order to get water for their families. This place is around 100 – 120 kms. from Pune. More than 1/3rd of the state has been declared as drought hit

Now we have the present ruling Government in Central, State as well as Municipalities but still the water projects are at the same place since 2014. In fact last year they raised 200 crore rupees saying we are short of funds for the project. Luckily I didn’t participate and I will share why I say luckily. Instead of INR 200 crore, people gave between INR 400-500 crores and this was raised in 2-3 days when the municipal bonds were going to be there for 2 weeks. It was made to imply that the investments would be under sovereign guarantee of Government of India . After the money was collected, it was told that neither the Center nor the State takes any responsibility for the money that was collected. The last I heard on the issue is that apparently the money has been kept in a Fixed Deposit Account (no reason given) .

Apart from the monetary issue, why did it take them 4 years to know that Pune has a water problem when that was their political plank which they had sold to Puneties. Of the several projects that are supposed to happen, the Bhama Akshad Dam project has been hanging fire since 2010. In fact it has its own page on wikipedia so it’s one of the notable water projects. The less I say of the other projects, the better.

Pune Municipal Transport or PMT

One of the other electoral promises that the current Government had made was that there will be a huge improvement in lot number of areas of PMT’s functioning. Accountability, number of buses, number of routes, less breakdowns and less corruption. Neither of these promises have been implemented. PMPML which also a webpage on wikipedia clearly shows that number of breakdowns, while unoffically the number is probably one and half times than what has been reported. As can be seen the last stats. are of 2016-2017, there are no stats. either of 2017-18 or 2018-19. This shows at the very least negligence and lack of transparency on the present Government. This is the same Government which has put spent close to 5,200 crores on advertisements (probably even more) as shared by Rajvardhan Rathore in a Lok Sabha reply in July 2018. And the amount that was spent within July 2018 – May 2019 would probably be about around 2000 crores or something as they spent a lot in ads this year, so the final tax payer and black money must be around 7,200 crores. It is possible that some estimates could be made by PRS and Election Commission, although the less I say about the EC, the better.

Pune Metro Project

This is and was one of the projects I am and was looking forwards to. The Pune Metro was going to make it easier just like Delhi Metro makes it easier to travel from point A to point B without waiting for a bus which when it will come is not known. To keep updated on the project I followed the Pune Metro project twitter handle. The twitter handle is and was useful as it used to keep me updated about how the Pune project is doing. I am one of those who used to check the twitter page daily just to know if they had posted something new just because I have fascination about transit systems, rails, planes, buses you name it. From 18th March 2019, the twitter handle stopped sharing any new news. The first week or two went by and I thought it is possible that due to Election Comission rules they might not be sharing. But even after Pune voted i.e 26th March, 2019 there were still no updates. Somehow I felt something is fishy and on a hunch I tried to see if the Nagpur Metro Rail handle was also showing similar.

To my surprise, the Nagpur Metro Rail were sharing, giving updates even though there is and was a huge section of Nagpur who didn’t feel the need of a metro. Pune does. So I dug a little deeper and found out that HCC has been terminated out of Pune Metro . Just to share a bit of history, this route is the most crucial part of the project as it is the first phase. Just to put a bit of context here HCC used to be a Navratna status public-private company which used to be at the bourses around INR 40-44/- which has now come to INR 13/- and will probably slide to something like INR 10/- . While I do not really want to get into how they have destructed public limited companies and it would probably need a blog post or two to share all the public limited companies they have systematically destroyed let’s leave that for another day.

As far as Pune Metro is concerned, I don’t see anything now till whatever the New Government comes into power anything will happen. This may happen in a month or two from today. Let’s say whatever Government does comes to power, there is also no guarantee that the team that is/was driving the Pune Metro will be managing then. If there is a new team it will again take its own sweet time. Even if the old team is given responsibility it will take close to 6 months to a year for replacements to come by as they would have to re-issue a global tender all over again giving an updated picture of the work done, the work yet to be completed and so on and so forth. We know the delay is going to be so severe that an ardent Pune metro video blogger whom I know, Yogi Logic shared just how things are without saying much as there is nothing left to say

Reproducible builds folks: Reproducible Builds in April 2019

Sunday 5th of May 2019 05:08:27 PM

Welcome to the April 2019 report from the Reproducible Builds project! In these now-monthly reports we will outline the most important things which have been up to in and around the world of reproducible builds & secure toolchains.

As a quick recap, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users pre-compiled. The motivation behind reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

In this months’s report, we will cover:

  • Media coverageCompromised toolchains, what makes a good digital product?, etc.
  • Upstream newsScala and Go working on reproducibility, etc.
  • Distribution workDistributing build certificates, an update from openSUSE, etc.
  • Software developmentNew features in diffoscope, yet more test framework development, etc
  • Misc newsFrom our mailing list, etc.
  • Getting in touchHow to contribute, etc
Media coverage
  • The SecureList website reported on Operation “ShadowHammer”, a high-profile supply chain attack involving the ASUS Live Update Utility. As their post describes in more detail tampering with binaries would usually break the digital signature, but in this case the digital signature itself appeared to have been compromised. (Read more)

Upstream news

The first non-trivial library written in the Scala programming language on the Java Virtual Machine was released with Arnout Engelen’s sbt-reproducible-builds plugin enabled during the build. This resulted in Akka 2.5.22 becoming reproducible, both for the artifacts built with version 2.12.8 and 2.13.0-RC1 of the Scala compiler. For 2.12.8, the original release was performed on a Mac and the validation was done on a Debian-based machine, so it appears the build is reproducible across diverse systems. (Mailing list thread)

Jeremiah “DTMB” Orians announced the 1.3.0 release of M2-Planet, a self-hosting C compiler written in a subset of the features it supports. It has been bootstrapped entirely from hexadecimal (!) with 100% reproducible output/binaries. This new release sports a self-hosting port for an additional architecture amongst other changes. Being “self-hosted” is an important property as it can provide a method of validating the legitimancy of the build toolchain.

The Go programming language has been making progress in making their builds reproducible. In 2016, Ximin Luo had created issue #16860 requesting that the compiler generates the same result regardless of the path in which the package is built. However, progress was recently made in Change #173344 (and adjacent) that will permit a -trimpath mode that will generate binaries that do not contain any local path names, similar to -fpath-prefix-map.

The fontconfig library for configuring and customising font access in a number of distributions announced they had merged patches to allow various cache files to be reproducible. This is after Chris Lamb posted a historical summary and a request for action to Fontconfig’s mailing list in January 2019

Distribution work

In Debian, Chris Lamb added 90 reviews of Debian packages, adding to our knowledge about identified issues and 14 issues were automatically removed. Chris also added two issue types: build_date_in_egg_info_directory_name & randomness_in_perl6_precompiled_libraries.

Holger Levsen started a discussion regarding the distribution of .buildinfo files. These files record the environment that was used as part of a particular build in order that — along with the source code — ensure that the aforementioned environment can be recreated at a later date to reproduce the exact binary. Distributing these files is important so that others can validate that a build is actually reproducible. In his post, Holger refers to two services that now exist, buildinfo.debian.net and buildinfos.debian.net.

In addition, Holger restarted a long-running discussion regarding the reproducibility status of Debian buster touching on questions of potentially performing mass rebuilds of all packages in order that they use updated toolchains.

There was yet more progress towards making the Debian Installer images reproducible. Following-on from last months, Chris Lamb performed some further testing of the generated images. Cyril Brulebois then made an upload of the debian-installer package to Debian that included a number of Chris’ patches and Vagrant Cascadian filed a patch to fix the reproducibility of “u-boot” images by using -n argument to gzip(1).

Bernhard M. Wiedemann posted his monthly Reproducible Builds status update for the openSUSE distribution. Bernhard also posted to our mailing list regarding enabling the normalisation of file modification times in Python .pyc files and opened issue #1133809 in the openSUSE bug tracker.

Software development Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

diffoscope

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages. It does not define reproducibility, but rather provides a helpful and human-readable guidance for packages that are not reproducible, rather than relying essentially-useless diffs.

This month, Chris Lamb did a lot of development of diffoscope, including:

  • Updating the certificate of the try.diffoscope.org web-based version of the tool.

  • Uploaded version 114 to the Debian experimental distribution and made the corresponding upload to the PyPI package repository.

  • Added support for semantic comparison of GnuPG “keybox” (.kbx) files. (#871244)

  • Add the ability to treat missing tools as failures if a “magic” environment variable is detected in order to facilitate interpreting required tools on the Debian autopkgtests as actual test failures, rather than skipping them. The behaviour of the existing testsuite remains unchanged. (#905885)

  • Filed a “request for packaging” for the annocheck tool which can be used to “analyse an application’s compilation”. This is as part of an outstanding wishlist issue. (#926470)

  • Consolidated on a single alias as the exception value across the entire codebase. []

In addition, Vibhu Agrawal ensured that diffoscope failed more gracefully when running out of diskspace to resolve Debian bug #874582 and Vagrant Cascadian updated to diffoscope 114 in GNU Guix. Thanks!

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. It is used automatically in most Debian package builds. This month, Chris Lamb made the following improvements:

  • Workaround Archive::Zip’s incorrect handling of the localExtraField class member field by monkey-patching the accessor methods to always return normalised values. This fixes the normalisation of Unix ownership metadata within .zip and .epub files. (#858431)

  • Actually check the return status from Archive::Zip when writing file to disk. []

  • Catch an edge-case where we can’t parse the length of a particular field within .zip files. []

Chris then uploaded version 1.1.3-1 to the Debian experimental distribution.

Project website

Chris Lamb made a number of improvements to our project website this month, including:

  • Using an explicit “draft” boolean flag for posts. Jekyll in Debian stable silently (!) does not support the where_exp filter. []

  • Moving more pages away from the old design with HTML to Markdown formatting and the new design template. []

  • Adding a simple Makefile to implicitly document how to build the site [] and add a simple .gitlab-ci.yml to test branches/builds [].

  • Adding as simple “lint” command so we can see how many pages are using the old style. []

  • Adding an explicit link to our “Who is involved?” page in the footer of the newer design [] and add a link to donation page [].

  • Moved various bits of infrastructure to support a monthly report structure. []

Test framework

We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. The following changes were done in the last month:

  • Holger Levsen (Debian-related changes):

    • Add new experimental buildinfos.debian.net service. [][][]
    • Allow pushing of .buildinfo files from coccia. []
    • Permit rsync to write into subdirectories. []
    • Include the meta “pool” job in the overall job health view. []
    • Add support for host-specific SSH authorized_keys files used on a particular build node. []
    • Show link to maintenance jobs for offline nodes. [][]
    • Increase the job timeout for some runners from 3 to 5 days. []
    • Don’t try to turn Jenkins or nodes offline too quickly. [][]
    • Fix pbuilder lock files if necessary. []
  • Mattia Rizzolo:

    • Special-case the debian-installer package when building to allow it access to the internet.. []
    • Force installing the debootstrap from stretch backports and remove cdebootstrap. []
    • Install the python3-yaml package on nodes as it is needed by the deploy script. []
    • Add/update the new reproducible-builds.org MX records. [][]
    • Fix typo in comment; thanks to ijc for reporting! []

Holger Levsen [][][], Mattia Rizzolo [] and Vagrant Cascadian [] all performed a large amount of build node maintenance, system & Jenkins administration and Chris Lamb provided a patch to avoid double spaces in IRC notifications [].

Misc news Getting in touch

If you are interested in contributing the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:


This month’s report was written by Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Mattia Rizzolo and Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Colin Watson: Buster upgrade

Sunday 5th of May 2019 12:10:48 AM

I upgraded my home server from Debian stretch to buster recently, which is something I normally do once we’re frozen: this is a system that was first installed in 1999 and has a lot of complicated stuff on it, and while I try to keep it as cleanly-maintained as I can it still often runs into some interesting problems. Things went largely OK this time round, although there were a few snags of various degrees of severity, some of which weren’t Debian’s fault.

As ever, etckeeper made it much more comfortable to make non-trivial configuration file changes without fearing that I was going to lose information.

  • The first apt full-upgrade failed part-way through with “dependency problems prevent processing triggers for desktop-file-utils” for what didn’t seem like a particularly good reason; dpkg --configure -a sorted it out and I was able to resume the upgrade from there. I think I’ve seen a report of this somewhere recently as it rang a bell, though I haven’t yet found it.

  • I had a number of truly annoying configuration file resolutions to perform. There’s not much to be done about that except try to gradually move things to .d directories where available, and other such strategies to minimise the local differences I’m maintaining.

  • I had an old backup disk that had failed some time ago but was still plugged in and occasionally generating ATA errors. These made some parts of the upgrade excruciatingly slow, so as soon as I got to a point where I had to reboot anyway I took the opportunity to open up the case and unplug it.

  • I hit #919621 “lvm2: Update unexpectedly activates system ID check, bypassing impossible”. Fortunately I noticed the problem before rebooting due to warning messages from various things, and I adjusted my LVM configuration to set a system ID matching the one in my volume group. Unfortunately I forgot to run update-initramfs -u after doing so, and so I ended up having to use break=premount on the kernel command line and fix things up in the same way in the initramfs until I could update it properly. I’m not sure what the right fix for this is, although it probably only affects some rather old VGs; I created mine in 2004.

  • I ran into #924881 “postgresql: buster upgrade breaks older postgresql (9.6) and newer postgresql (11) is also inoperative” (in fact a bug in ssl-cert). It was correct to reject the snakeoil certificate, but the upgrade failure mode was pretty graceless and it would have been helpful for something to notice the situation and prompt me to regenerate the certificate.

  • My networking wasn’t happy after the upgrade; I ended up with some missing addresses, which I’m prepared to believe was the fault of my very old and badly-organised /etc/network/interfaces file, so I rearranged it to follow what seems to be the modern best practice of handling multiple addresses on an interface by just having one iface stanza per address using the same interface name, rather than pre-up ip addr add lines or alias interfaces or anything like that. After that, the interface sometimes refused to come up at all with “ADDRCONF(NETDEV_UP): eth0: link is not ready” messages. Some web-searching and grepping of the kernel source led me to the idea that listing inet6 stanzas before inet stanzas for a given interface name was likely to be helpful, and so it proved: I now have an /etc/network/interfaces that both works and is much easier to read.

  • I had to do some manual steps to get Icinga Web 2 authentication working again: I followed the upstream directions to upgrade the database schema, and I had to run a2enmod php7.3 manually since the previous enablement of php7.0 wasn’t carried over. (I’m not completely sure if the first step was required, but the second certainly was.)

Other than that, everything seems to be working well now.

More in Tux Machines

Games: Smith and Winston, 7 Billion Humans Sale

Servers: Ampere Computing, SUSE and Red Hat

  • Ampere Computing Is Keeping Close Track Of The Linux Performance For Their ARM Servers

    Hardware vendor Ampere Computing with their impressive ARM servers is doing a great job on closely following their hardware's Linux performance as part of a rigorous continuous testing regiment or ensuring quality, compatibility, and stability while being fully-automated. Ampere Computing's Travis Lazar talked at this week's Linux Foundation events in San Diego over the importance of continuous regression testing for software and hardware development by talking about their internal workflow and software in place. Their internal system is the "Totally Automated Regression System" or TARS for short. TARS makes use of various open-source components including the Phoronix Test Suite and its vast collection of benchmarks for providing comprehensive test coverage plus Ampere's own "extensions" to the Phoronix Test Suite. TARS also incorporates the provisioning/configuration responsibilities as well as analysis of the data.

  • [SUSE] Learn how the Multimodal OS can benefit your organization.
  • From ProdOps to DevOps: Surviving and thriving

    For many of us in Production Operations (ProdOps), change is the enemy. If something changes, there is now an opportunity for things that were working just fine to experience problems. It is like a game of Jenga. When will the tower fall because a seemingly minor change unbalances the whole stack of pieces? ProdOps teams hate change so much, that countless frameworks have been invented to "manage" changes; in reality, these frameworks make the procedure for effecting a change so onerous that most people give up and accept the status quo. Actually, that statement is a bit unfair. These frameworks are an attempt to wrap planning and consensus around production changes, thus minimizing potential downtime caused by random or rogue changes (see Why the lone wolf mentality is a sysadmin mistake).

  • Meet Red Hat at VMworld

    As Red Hat’s Ashesh Badani said in his blog post about the reference architecture for OpenShift on VMware’s SDDC stack “… this is just the first step — Red Hat OpenShift 4 brings optimized installation capabilities to a variety of infrastructures and for this, the companies are working towards a VMware Validated Design. We are excited that VMware is working closely with Red Hat to deliver a simplified experience there in the coming months.”

Late Coverage of Confidential Computing Consortium

  • Microsoft Partners With Google, Intel, And Others To Form Data Protection Consortium

    The software maker joined Google Cloud, Intel, IBM, Alibaba, Arm, Baidu, Red Hat, Swisscom, and Tencent to establish the Confidential Computing Consortium, a group committed to providing better private data protection, promoting the use of confidential computing, and advancing open source standards among members of the technology community.

  • #OSSUMMIT: Confidential Computing Consortium Takes Shape to Enable Secure Collaboration

    At the Open Source Summit in San Diego, California on August 21, the Linux Foundation announced the formation of the Confidential Computing Consortium. Confidential computing is an approach using encrypted data that enables organizations to share and collaborate, while still maintaining privacy. Among the initial backers of the effort are Alibaba, Arm, Baidu, Google Cloud, IBM, Intel, Microsoft, Red Hat, Swisscom and Tencent. “The context of confidential computing is that we can actually use the data encrypted while programs are working on it,” John Gossman, distinguished engineer at Microsoft, said during a keynote presentation announcing the new effort. Initially there are three projects that are part of the Confidential Computing Consortium, with an expectation that more will be added over time. Microsoft has contributed its Open Enclave SDK, Red Hat is contributing the Enarx project for Trusted Execution Environments and Intel is contributing its Software Guard Extensions (SGX) software development kit. Lorie Wigle, general manager, platform security product management at Intel, explained that Intel has had a capability built into some of its processors called software guard which essentially provides a hardware-based capability for protecting an area of memory.

Graphics: Mesa Radeon Vulkan Driver and SPIR-V Support For OpenGL 4.6

  • Mesa Radeon Vulkan Driver Sees ~30% Performance Boost For APUs

    Mesa's RADV Radeon Vulkan driver just saw a big performance optimization land to benefit APUs like Raven Ridge and Picasso, simply systems with no dedicated video memory. The change by Feral's Alex Smith puts the uncached GTT type at a higher index than the visible vRAM type for these configurations without dedicated vRAM, namely APUs.

  • Intel Iris Gallium3D Is Close With SPIR-V Support For OpenGL 4.6

    This week saw OpenGL 4.6 support finally merged for Intel's i965 Mesa driver and will be part of the upcoming Mesa 19.2 release. Not landed yet but coming soon is the newer Intel "Iris" Gallium3D driver also seeing OpenGL 4.6 support. Iris Gallium3D has been at OpenGL 4.5 support and is quite near as well with its OpenGL 4.6 support thanks to the shared NIR support and more with the rest of the Intel open-source graphics stack. Though it's looking less likely that OpenGL 4.6 support would be back-ported to Mesa 19.2 for Iris, but we'll see.