Language Selection

English French German Italian Portuguese Spanish

Custom scripting gives users a safe-du

Filed under

My company has a Linux cluster with a terabyte of attached storage. Over time we noticed the head node was becoming more overloaded. Inspection of the system showed that users were starting dozens of copies of the du utility to determine disk space usage. This was a natural thing for them to do, because they had a need to know how much disk space was available. A lack of disk space would cause their software builds and tests to fail. The problem was that it takes five to seven hours for a du of the entire shared filesystem. Thus, when the filesystem was nearly full (as it of course usually was), the number of du processes would increase almost exponentially.

To address this problem, we first set up automated nightly disk space reports, so that users could check the status without running du. This still did not solve the problem, as the amount of used space could fluctuate dramatically over the course of 24 hours. Users still wanted and needed to run their own du processes throughout the workday.

While adding more disk space would have solved the problem, we are using a large disk array that is already filled to maximum capacity. In general, users tend to fill up all available disk space anyway, no matter how much you give them.

We then developed a policy: users could run du on any directory they owned. In addition, user du processes would be allowed to run for a maximum of one hour of wall time. Users in the wheel group would be exempt from these restrictions.

I was given the task of developing a tool to implement this policy. Some sort of wrapper around the existing du seemed like an obvious choice: the script could validate the input, abort if an invalid path was given, and terminate the du process if it ran too long.

I wrote a basic bash script in perhaps an hour's time. Then I thought about how to run it, and that is where I ran into trouble. I had thought that I would make the script set user id (setuid) or set group id (setgid) root, i.e. when run by any user it would actually run in the root group. Then, I could change the permissions on the real du so that only root could run it. The result would be that normal users could only access the real du through the wrapper script.

Of course that would make a pretty boring article, and in reality it didn't turn out to be that simple:

Full Story.

More in Tux Machines

today's howtos

A tour of Google's 2016 open source releases

Open source software enables Google to build things quickly and efficiently without reinventing the wheel, allowing us to focus on solving new problems. We stand on the shoulders of giants, and we know it. This is why we support open source and make it easy for Googlers to release the projects they're working on internally as open source. We've released more than 20-million lines of open source code to date, including projects such as Android, Angular, Chromium, Kubernetes, and TensorFlow. Our releases also include many projects you may not be familiar with, such as Cartographer, Omnitone, and Yeoman. Read more

Viewing Linux Logs from the Command Line

At some point in your career as a Linux administrator, you are going to have to view log files. After all, they are there for one very important help you troubleshoot an issue. In fact, every seasoned administrator will immediately tell you that the first thing to be done, when a problem arises, is to view the logs. And there are plenty of logs to be found: logs for the system, logs for the kernel, for package managers, for Xorg, for the boot process, for Apache, for MySQL… For nearly anything you can think of, there is a log file. Read more

At Long Last, Linux Gets Dynamic Tracing

When the Linux kernel version 4.9 will be released next week, it will come with the last pieces needed to offer to some long-awaited dynamic thread-tracing capabilities. As the keepers of monitoring and debugging software start using these new kernel calls, some of which have been added to the Linux kernel over the last two years, they will be able to offer much more nuanced, and easier to deploy, system performance tools, noted Brendan Gregg, a Netflix performance systems engineer and author of DTrace Tools, in a presentation at the USENIX LISA 2016 conference, taking place this week in Boston. Read more