Language Selection

English French German Italian Portuguese Spanish

Development

Programming: OpenBSD, FreddieMeter, Python and More

Filed under
Development
  • [Older] Linux Systems Performance

    Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud.

  • Martin Pieuchot: The Unknown Plan

    Since I attend OpenBSD hackathons, I hear stories about how crazy are the ports hackathons. So I try my best to look like a porter in order to experience this craziness. I must admit p2k19 was awesome but the craziness of port hackathons is still an enigma to me.

  • Google's AI-powered FreddieMeter can tell if you sing like Queen's frontman

    While Freddie may have sadly bitten the dust, his fame lives on, so much so that Google's Creative Lab has cooked up the FreddieMeter.

    The show must go on! It's an AI-powered thingy which uses its smarts to figure out if one's singing voice has a pitch, melody and timbre to match that of Mercury's champion vocals.

  • What is Python? Powerful, intuitive programming

    Why the Python programming language shines for data science, machine learning, systems automation, web and API development, and more.

  • Ian Ozsvald: Training Courses for 2020 Q1 – Successful Data Science Projects & Software Engineering for Data Scientists
  • The simplest explanation of Decorators in Python

    Before starting about decorators, first, understand that functions in python have below three properties.

  • Basic Data Types in Python 3: Booleans

    Welcome back to our ongoing series of blog posts on basic data types in Python 3! Last time, we explored the functionality of strings. Today, we dive in to another key data type - booleans. Booleans (and "boolean logic") are an important concept in programming, representing the concept of "true" and "false".

    If you're learning Python, you might also want to check out TwilioQuest 3. You'll learn about basic data types like the boolean, and much more about Python programming.

    Ready to learn how to use booleans in Python 3? Let's get started!

GCC: OpenMP / OpenACC and Static Analysis Framework

Filed under
Development
GNU
  • The GCC 10 Compiler Lands OpenMP / OpenACC Offloading To AMD Radeon GPUs

    A few days ago I wrote about the OpenMP / OpenACC offloading patches for Radeon "GCN" GPUs being posted and seeking inclusion in the GCC 10 compiler that will be released in a few months. Those patches were successfully merged meaning this next annual update to the GNU Compiler Collection will feature initial OpenMP/OpenACC code offloading support to supported AMD GPU targets.

    After GCC 9 only had the initial AMD Radeon GCN target in place, GCC 10 in early 2020 will feature the initial offloading support using the modern OpenMP and OpenACC APIs, thanks to the merges this week. The libgomp port and associated bits for the AMD GCN back-end have landed thanks to the work done by Code Sourcery under contract with AMD.

  • RFC: Add a static analysis framework to GCC
    This patch kit introduces a static analysis pass for GCC that can diagnose
    various kinds of problems in C code at compile-time (e.g. double-free,
    use-after-free, etc).
    
    The analyzer runs as an IPA pass on the gimple SSA representation.
    It associates state machines with data, with transitions at certain
    statements and edges.  It finds "interesting" interprocedural paths
    through the user's code, in which bogus state transitions happen.
    
    For example, given:
    
       free (ptr);
       free (ptr);
    
    at the first call, "ptr" transitions to the "freed" state, and
    at the second call the analyzer complains, since "ptr" is already in
    the "freed" state (unless "ptr" is NULL, in which case it stays in
    the NULL state for both calls).
    
    Specific state machines include:
    - a checker for malloc/free, for detecting double-free, resource leaks,
      use-after-free, etc (sm-malloc.cc), and
    - a checker for stdio's FILE stream API (sm-file.cc)
    
    There are also two state-machine-based checkers that are just
    proof-of-concept at this stage:
    - a checker for tracking exposure of sensitive data (e.g.
      writing passwords to log files aka CWE-532), and
    - a checker for tracking "taint", where data potentially under an
      attacker's control is used without sanitization for things like
      array indices (CWE-129).
    
    There's a separation between the state machines and the analysis
    engine, so it ought to be relatively easy to add new warnings.
    
    For any given diagnostic emitted by a state machine, the analysis engine
    generates the simplest feasible interprocedural path of control flow for
    triggering the diagnostic.
    
  • GCC Might Finally Have A Static Analysis Framework Thanks To Red Hat

    Clang's static analyzer has become quite popular with developers for C/C++ static analysis of code while now the GNU Compiler Collection (GCC) might finally see a mainline option thanks to Red Hat.

    Red Hat's David Malcolm has proposed a set of 49 patches that appear to be fairly robust and the most we have seen out of GCC static analysis capabilities to date.

Programming: GCC, RcppEigen and Python

Filed under
Development
  • Introduce a new GCC option, --record-gcc-command-line
    I would like to propose the following patches which introduce a compile option --record-gcc-command-line. When passed to gcc, it saves the command line option into the produced object file. The option makes it trivial to trace back how a file was compiled and by which version of the gcc. It helps with debugging, reproducing bugs and repeating the build process.
    
    This option is similar to -frecord-gcc-switches. However, they have three fundamental differences: Firstly, -frecord-gcc-switches saves the internal state after the argv is processed and passed by the driver. As opposed to that, --record-gcc-command-line saves the command-line as received by the driver. Secondly, -frecord-gcc-switches saves the switches as separate entries into a mergeable string section. Therefore, the entries belonging to different object files get mixed up after being linked. The new --record-gcc-command-line, on the other hand, creates one entry per invocation. By doing so, it makes it clear which options were used together in a single gcc invocation. Lastly, --record-gcc-command-line also adds the version of the gcc into this single entry to make it clear which version of gcc was called with any given command line. This is useful in cases where .comment section reports multiple versions.
    
    While there are also similarities between the implementations of these two options, they are completely independent. These commands can be used separately or together without issues. I used the same section that -frecord-gcc-switches uses on purpose. I could not use the name -frecord-gcc-command-line for this option; because of a {f*} in the specs, which forwards all options starting with -f to cc1/cc1plus as is. This is not we want for this option. We would like to append it a filename as well to pass the argv of the driver to child processes.
    
    This functionality operates as the following: It saves gcc's argv into a temporary file, and passes --record-gcc-command-line <tempfilename> to cc1 or cc1plus. The functionality of the backend is implemented via a hook. This patch includes an example implementation of the hook for elf targets: elf_record_gcc_command_line function. This function reads the given file and writes gcc's version and the command line into a mergeable string section, .GCC.command.line.
    
    
  • GCC Developers Discuss New Option For Recording Compiler Flags / Details In Binaries

    GCC developers recently have been discussing a new proposal over an option for preserving the command-line flags/options used when building a binary as well as the associated compiler version.

    The proposal sent out last week was over a --record-gcc-command-line option to save the compiler options into the produced object file. The proposal is in the name of helping debugging, reproducing bugs, and repeating build process. There is already a -frecord-gcc-switches option that is somewhat similar in behavior but with key differences as explained in the proposal.

  • RcppEigen 0.3.3.7.0

    A new minor release 0.3.3.7.0 of RcppEigen arrived on CRAN today (and just went to Debian too) bringing support for Eigen 3.3.7 to R.

    This release comes almost a year after the previous minor release 0.3.3.5.0. Besides the upgrade to the new upstream version, it brings a few accumulated polishes to the some helper and setup functions, and switches to the very nice tinytest package for unit tests; see below for the full list. As before, we carry a few required changes to Eigen in a diff.

  • “Higher Performance Python” at PyDataCambridge 2019

    I’ve had the pleasure of speaking at the first PyDataCambridge conference (2019), this is the second PyData conference in the UK after PyDataLondon (which colleagues and I co-founded 6 years back). I’m super proud to see PyData spread to 6 regional meetups and now 2 UK conferences.

today's howtos and programming

Filed under
Development
HowTos

today's howtos and programming bits

Filed under
Development
HowTos

Programming: Interview With Guido van Rossum, Python Picks and New Release of Picolibc From Keith Packard

Filed under
Development
  • Interview Guido van Rossum: “I'd rather write code than papers.”

    Guido van Rossum (1956) is the founding father of the Python programming language, one of the most popular development tools in the world. In 2019 CWI will award him the Dijkstra Fellowship.

    What led you to come up with a brand new programming language during your time at CWI?

    “I started at CWI as a junior programmer on a research team with Lambert Meertens, Leo Geurts and Steven Pemberton. They wanted to develop a language which would enable people without programming experience – such as scientists – to start writing computer programs fairly quickly.”

    “It was at the time that Basic was on the rise due to the arrival of the microcomputer. Meertens looked at this inadequate language with horror. ‘Stamp out Basic!’ Was his motto. In the end, ABC, as our language was called, would not work. The target group could not use it on their microcomputers, which were not powerful enough for it, while Unix users already had other tools. Those users thought ABC was an odd man out.”

    “Then I came across the so-called Amoeba project. That was a distributed operating system based on a microkernel, developed by Andrew Tanenbaum at the Vrije Universiteit Amsterdam and Sape Mullender at CWI. Not aiming at popularizing their operating system, their first and foremost goal was writing papers. Scientifically it was a breakthrough indeed: those papers are still being studied. I myself was not a researcher but a programmer on that project. I must say thought that there was an atmosphere at CWI in which programmers had a major input in the projects.”

  • Python Tears Through Mass Spectrometry Data

    At the November 2019 Python Frederick event, Conor Jenkins showed the group how mass spectrometry works and how Python saves huge amounts of time when processing the large amount of data produced by a mass spec analysis.

  • Wingware News: Wing Python IDE 7.1.3 - November 14, 2019

    Wing 7.1.3 adds improved and expanded documentation and support for matplotlib, improves the accuracy of code warnings, fixes automatically debugging child processes on Windows with Python 3.8, fixes installing the remote agent from .rpm or .deb installations, solves several issues with runtime type introinspection, allows Open from Project and similar navigation commands from non-Browse vi mode, improves debugger reliability, and fixes about 30 other minor usability issues.

  • Easily specifying colours from the default colour cycle in matplotlib

    Another quick matplotlib tip today: specifically, how easily specify colours from the standard matplotlib colour cycle.

    A while back, when matplotlib overhauled their themes and colour schemes, they changed the default cycle of colours used for lines in matplotlib. Previously the first line was pure blue (color='b' in matplotlib syntax), then red, then green etc. They, very sensibly, changed this to a far nicer selection of colours.

  • Typing Mercurial with pytype

    Following the recent introduction of Python type annotations (aka "type hints") in Mercurial (see, e.g. this changeset by Augie Fackler), I've been playing a bit with this and pytype.

    pytype is a static type analyzer for Python code. It compares with the more popular mypy but I don't have enough perspective to make a meaningful comparison at the moment. In this post, I'll illustrate how I worked with pytype to gradually add type hints in a Mercurial module and while doing so, fix bugs!

    The module I focused on is mercurial.mail, which contains mail utilities and that I know quite well. Other modules are also being worked on, this one is a good starting point because it has a limited number of "internal" dependencies, which both makes it faster to iterate with pytype and reduces side effects of other modules not being correctly typed already.

  • Two Books About the Kivy GUI Framework

    The Kivy Python GUI framework is intriguing.

    Not only it’s cross-platform but also supports Android. Java is too verbose and low level for me and Kivy is an opportunity for developing native Android apps without leaving Python.

    Outside of the Kivy project documentation, there are few third-party advanced tutorials that go in more depth than the official tutorials. So, before diving into the code of the Kivy demos, I wanted some books to explore more features and get a broader picture of the framework and what it can do.

    I found two potentially interesting books: Building Android Apps in Python Using Kivy with Android Studio: With Pyjnius, Plyer, and Buildozer by Ahmed Fawzy Mohamed Gad (Apress, 2019), and Kivy - Interactive Applications and Games in Python - Second Edition by Roberto Ulloa (Packt, 2015).

  • A Qt GUI for logging

    A question that comes up from time to time is about how to log to a GUI application. The Qt framework is a popular cross-platform UI framework with Python bindings using PySide2 or PyQt5 libraries.

    The following example shows how to log to a Qt GUI. This introduces a simple QtHandler class which takes a callable, which should be a slot in the main thread that does GUI updates. A worker thread is also created to show how you can log to the GUI from both the UI itself (via a button for manual logging) as well as a worker thread doing work in the background (here, just logging messages at random levels with random short delays in between).

  • Picolibc 1.1 Released With POSIX File I/O Support

    Longtime X11 developer Keith Packard has spent a lot of time in recent months while being employed by SiFive working on Picolibc as a new C library for embedded systems.

    Picolibc is designed solely for embedded use-cases at this point and was formerly developed by Keith under the name newlib-nano. Picolibc 1.1 is out now as the project's second stable release.

  • Picolibc Version 1.1

    Picolibc development is settling down at last. With the addition of a simple 'hello world' demo app, it seems like a good time to stamp the current code as 'version 1.1'.

Audiocasts/Shows: Ubuntu Podcast, Linux Headlines and 'Talk Python to Me'

Filed under
Development
GNU
Linux
Ubuntu
  • Ubuntu Podcast from the UK LoCo: S12E32 – Dungeon Keeper

    This week we’ve become addicted to Sedna SSD to PCIe controller cards. We discuss why distro hoppers are the worst, bring you some GUI love and round up our listener feedback.

    It’s Season 12 Episode 32 of the Ubuntu Podcast! Alan Pope, Mark Johnson and Martin Wimpress are connected and speaking to your brain.

  • I.T. Phone Home | TechSNAP 416

    Ubiquiti’s troublesome new telemetry, Jim’s take on the modern Microsoft, and why Project Silica just might be the future of long term storage.

  • 2019-11-14 | Linux Headlines

    Mirantis acquires Docker, WordPress brings a big new feature to Jetpack, GitHub has a plan for archiving the world's open source code, and a new developer hub is available for Go.

  • Talk Python to Me: #238 Collaborative data science with Gigantum

    Collaborative data science has a few challenges. First of all, those who you are collaborating with might not be savvy enough in the computer science techniques (for example, git and source control or docker and Linux). Second, seeing the work and changes others have made is a challenge too.

Programming: GNU Grep, DevOps Hype, and Python

Filed under
Development
  • Manipulating text with grep

    Imagine you have a file (or bunch of files) and you want to search for a specific string or configuration setting within these files. Opening each file individually and trying to find the specific string would be tiresome and probably isn’t the right approach. So what can we use, then?

  • Hiring a technical writer in the age of DevOps

    It's common for enterprises to leave the technical writer's role out of the DevOps discussion. Even the marketing department joins the discussion in some DevOps-first organizations—so why not the writers?

    Our industry doesn't ask enough of its technical writers. Documentation is an afterthought. Companies farm out technical writing to contractors at the end of the project lifecycle. Corners get cut. Likewise, technical writers don't ask enough of their industry. The expectations for the role vary from company to company. Both circumstances lead to technical writers being left out of the DevOps discussion.

    As your organization matures its DevOps practices, it's time to revisit the role of your technical writer.

  • How to port an awk script to Python

    Scripts are potent ways to solve a problem repeatedly, and awk is an excellent language for writing them. It excels at easy text processing in particular, and it can bring you through some complicated rewriting of config files or reformatting file names in a directory.

  • Navigating Python Code with Wing Pro 7 (part 1 of 3)

    Wing Python IDE includes a boatload of features aimed at making it easier to navigate and understand the structure of Python code. Some of these allow for quick navigation between the definition and uses of a symbol. Others provide a convenient index into source code. And still others quickly find and open files or navigate to symbols matching a name fragment.

    [...]

    This tool supports text matching, wildcard, and regular expression searching and automatically updates the search results as files change.

    Searching on Project Files assumes that you have used Add Existing Directory in the Project menu to add your source code to your project. Typically the project should contain the code you are actively working on. Packages that your code uses can be left out of the project, unless you anticipate often wanting to search them with Search in Files.

Development and Documentation at Mozilla, SUSE, R, Vim an More

Filed under
Development
  • Firefox Nightly: These Weeks in Firefox: Issue 68

    The “Omniscient” Browser Toolbox will be enabled by default in the coming days

  • The Brains Behind the Books – Part VII: Alexandra Settle

    The content of this article has been contributed by Alexandra Settle, Technical Writer at the SUSE Documentation Team.

    It is part of a series of articles focusing on SUSE Documentation and the great minds that create the manuals, guides, quick starts, and many more helpful documents.

  • The 20 Best R Machine Learning Packages in 2019

    Almost all novice data scientists and machine learning developers are being confused about picking a programming language. They always ask which programming language will be best for their machine learning and data science project. Either we will go for python, R, or MatLab. Well, the choice of a programming language depends on developers’ preference and system requirements. Among other programming languages, R is one of the most potential and splendid programming languages that have several R machine learning packages for both ML, AI, and data science projects.

    [...]

    R is an open-source language so people can contribute from anywhere in the world. You can use a Black Box in your code, which is written by someone else. In R, this Black Box is refereed to as a package. The package is nothing but a pre-written code that can be used repeatedly by anyone. Below, we are showcasing the top 20 best R machine learning packages.

  • 8 Excellent Free Books to Learn Vim

    Vim is an open source configurable and powerful text editor. It’s an improved version of the vi editor, with development dating back to 1976. This software can be used to write any kind of text.

    Vim sports a minimalistic interface to help the writer focus on the task at hand. It’s popular among developers given that it’s inherently modal (you go into command modes where you cannot edit), efficient, extensible, fast, and terminal friendly.

    When getting started with Vim, users face a steep learning curve. It’s true the software is simple. It’s simple in the sense that its minimal interface focuses the user on their main task. But Vim is very powerful.

  • RelStorage 3.0

    We're happy to announce the release of RelStorage 3.0, the relational storage engine for ZODB. Compared to RelStorage 2, highlights include a 30% reduction in memory usage, and up to 98% faster performance! (Ok, yes, that's from one specific benchmark and not everything is 98% faster, but improved performance was a major goal.)

    RelStorage 3.0 is a major release of RelStorage with a focus on performance and scalability. It's the result of a concentrated development effort spanning six months, with each pre-release being in production usage with large databases.

  • Introduction to ZODB Data Storage

    ZODB is a powerful native object database for Python, widely known for its use in the Zope web framework and the Plone content management system. By enabling transparent object graph persistence with no need to predefine schemas, ZODB enables extremely flexible application development. With pluggable storage engines such as FileStorage, ZEO, and RelStorage, it also provides flexible ways to store data.

    [...]

    In addition, ZODB provides a transactional view of these objects with snapshot isolation. Any given connection to the database sees a consistent view of all the objects in the database (whether it reads or writes to any particular object or not) as-of the moment it began. When adding or updating objects, no changes are published and made visible to other connections until the writing connection commits its transaction, at which point either all the changes are made visible or none of them are. Existing connections that continue reading (or even writing!) will still not see those changes; they're "stuck" at the snapshot view of the objects they started with. (The ability for readers to continue to be able to retrieve old data that's been replaced in newer transactions is known as multi-version concurrency control, or MVCC.)

    Many connections may be reading and writing to the database at once. ZODB uses optimistic concurrency control. Readers don't block other readers or writers, and writers are allowed to proceed as if they were the only one making changes right up until they commit. Writes are defined to occur in a strict order. If a writer discovers that an earlier transaction had modified objects that it too wants to modify, a conflict occurs. Instead of just rolling back the writing transaction and forcing it to start over, taking the modified object into account, ZODB gives the application the chance to resolve the conflict using a three-way merge between the object as it existed when the transaction began, the object that the connection wants to commit, and the object that was committed by the other writer. Only if it cannot do so is the transaction rolled back.

GCC 7.5 Released

Filed under
Development
GNU

The GNU Compiler Collection version 7.5 has been released.

GCC 7.5 is a bug-fix release from the GCC 7 branch
containing important fixes for regressions and serious bugs in
GCC 7.4 with more than 215 bugs fixed since the previous release.

This is also the last release from the GCC 7 branch which will receive
no further fixes from now on.  GCC continues to be maintained on
the GCC 8 and GCC 9 branches and the development trunk.

This release is available from the FTP servers listed at:

  http://www.gnu.org/order/ftp.html

Please do not contact me directly regarding questions or comments
about this release.  Instead, use the resources available from
http://gcc.gnu.org.

As always, a vast number of people contributed to this GCC release
-- far too many to thank them individually!

Read more

Also: GCC 7.5 Released With 215+ Bug Fixes As The Last Update To GCC7

Syndicate content

More in Tux Machines

Type Title Author Replies Last Postsort icon
Story Android Leftovers Rianne Schestowitz 17/11/2019 - 6:39pm
Story Debian GNU/Linux 10.2 "Buster" Live & Installable ISOs Now Available to Download Rianne Schestowitz 17/11/2019 - 6:30pm
Story Zink Benchmarks - Mesa OpenGL Running Over Vulkan Rianne Schestowitz 17/11/2019 - 6:27pm
Story Updated Debian 10: 10.2 released Roy Schestowitz 7 17/11/2019 - 6:24pm
Story Compact Ryzen V1000 system starts at $689 with pre-loaded Ubuntu Rianne Schestowitz 17/11/2019 - 6:22pm
Story today's leftovers Roy Schestowitz 17/11/2019 - 4:45pm
Story Programming: OpenBSD, FreddieMeter, Python and More Roy Schestowitz 17/11/2019 - 4:39pm
Story Arch Conf 2019 Report Roy Schestowitz 17/11/2019 - 4:32pm
Story Testing Slax 10.2 beta1 Roy Schestowitz 17/11/2019 - 4:24pm
Story GCC: OpenMP / OpenACC and Static Analysis Framework Roy Schestowitz 17/11/2019 - 4:19pm