Posts tagged 'ubuntu'

Web Hooks for the Janitor

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor.

As covered in my post from last week, the Janitor now regularly tries to import new upstream git snapshots or upstream releases into packages in Sid.

Moving parts

There are about 30,000 packages in sid, and it usually takes a couple of weeks for the janitor to cycle through all of them. Generally speaking, there are up to three moving targets for each package:

  • The packaging repository; vcswatch regularly scans this for changes, and notifies the janitor when a repository has changed. For salsa repositories it is instantly notified through a web hook
  • The upstream release tarballs; the QA watch service regularly polls these, and the janitor scans for changes in the UDD tables with watch data (used for fresh-releases)
  • The upstream repository; there is no service in Debian that watches this at the moment (used for fresh-snapshots)

When the janitor notices that one of these three targets has changed, it prioritizes processing of a package - this means that a push to a packaging repository on salsa usually leads to a build being kicked off within 10 minutes. New upstream releases are usually noticed by QA watch within a day or so and then lead to a build. Now commits in upstream repositories don’t get noticed today.

Note that there are no guarantees; the scheduler tries to be clever and not e.g. rebuild the same package over and over again if it’s constantly changing and takes a long time to build.

Packages without priority are processed with a scoring system that takes into account perceived value (based on e.g. popcon), cost (based on wall-time duration of previous builds) and likelihood of success (whether recent builds were successful, and how frequently the repositories involved change).

webhooks for upstream repositories

At the moment there is no service in Debian (yet - perhaps this is something that vcswatch or a sibling service could also do?) that scans upstream repositories for changes.

However, if you maintain an upstream package, you can use a webhook to notify the janitor that commits have been made to your repository, and it will create a new package in fresh-snapshots. Webhooks from the following hosting site software are currently supported:

You can simply use the URL https://janitor.debian.net/ as the target for hooks. There is no need to specify a secret, and the hook can either use a JSON or form encoding payload.

The endpoint should tell you whether it understood a webhook request, and whether it took any action. It’s fine to submit webhooks for repositories that the janitor does not (yet) know about.

GitHub

For GitHub, you can do so in the Webhooks section of the Settings tab. Fill the form as shown below and click on Add webhook:

GitLab

On GitLab instances, you can find the Webhooks tab under the Settings menu for each repository (under the gear symbol). Fill the form in as shown below and click Add Webhook:

Launchpad

For Launchpad, go to the repository (for Git) web view and click Manage Webhooks. From there, you can add a new webhook; fill the form in as shown below and click Add Webhook:

comments.

Thousands of Debian packages updated from their upstream Git repository

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor.

Linux distributions like Debian fulfill an important function in the FOSS ecosystem - they are system integrators that take existing free and open source software projects and adapt them where necessary to work well together. They also make it possible for users to install more software in an easy and consistent way and with some degree of quality control and review.

One of the consequences of this model is that the distribution package often lags behind upstream releases. This is especially true for distributions that have tighter integration and standardization (such as Debian), and often new upstream code is only imported irregularly because it is a manual process - both updating the package, but also making sure that it still works together well with the rest of the system.

The process of importing a new upstream used to be (well, back when I started working on Debian packages) fairly manual and something like this:

  • Go to the upstream’s homepage, find the tarball and signature and verify the tarball
  • Make modifications so the tarball matches Debian’s format
  • Diff the original and new upstream tarballs and figure out whether changes are reasonable and which require packaging changes
  • Update the packaging, changelog, build and manually test the package
  • Upload

Ecosystem Improvements

However, there have been developments over the last decade that make it easier to import new upstream releases into Debian packages.

Uscan and debian QA watch

Uscan and debian/watch have been around for a while and make it possible to find upstream tarballs.

A debian watch file usually looks something like this:

1
2
version=4
http://somesite.com/dir/filenamewithversion.tar.gz

The QA watch service regularly polls all watch locations in the archive and makes the information available, so it’s possible to know which packages have changed without downloading each one of them.

Git

Git is fairly ubiquitous nowadays, and most upstream projects and packages in Debian use it. There are still exceptions that do not use any version control system or that use a different control system, but they are becoming increasingly rare. [1]

debian/upstream/metadata

DEP-12 specifies a file format with metadata about the upstream project that a package was based on. In particular relevant for our case is the fact it has fields for the location of the upstream version control location.

debian/upstream/metadata files look something like this:

1
2
3
---
Repository: https://www.dulwich.io/code/dulwich/
Repository-Browse: https://www.dulwich.io/code/dulwich/

While DEP-12 is still a draft, it has already been widely adopted - there are about 10000 packages in Debian that ship a debian/upstream/metadata file with Repository information.

Autopkgtest

The Autopkgtest standard and associated tooling provide a way to run a defined set of tests against an installed package. This makes it possible to verify that a package is working correctly as part of the system as a whole. ci.debian.net regularly runs these tests against Debian packages to detect regressions.

Vcs-Git headers

The Vcs-Git headers in debian/control are the equivalent of the Repository field in debian/upstream/metadata, but for the packaging repositories (as opposed to the upstream ones).

They’ve been around for a while and are widely adopted, as can be seen from zack’s stats:

The vcswatch service that regularly polls packaging repositories to see whether they have changed makes it a lot easier to consume this information in usable way.

Debhelper adoption

Over the last couple of years, Debian has slowly been converging on a single build tool - debhelper’s dh interface.

Being able to rely on a single build tool makes it easier to write code to update packaging when upstream changes require it.

Debhelper DWIM

Debhelper (and its helpers) increasingly can figure out how to do the Right Thing in many cases without being explicitly configured. This makes packaging less effort, but also means that it’s less likely that importing a new upstream version will require updates to the packaging.

With all of these improvements in place, it actually becomes feasible in a lot of situations to update a Debian package to a new upstream version automatically. Of course, this requires that all of this information is available, so it won’t work for all packages. In some cases, the packaging for the older upstream version might not apply to the newer upstream version.

The Janitor has attempted to import a new upstream Git snapshot and a new upstream release for every package in the archive where a debian/watch file or debian/upstream/metadata file are present.

These are the steps it uses:

  • Find new upstream version
    • If release, use debian/watch - or maybe tagged in upstream repository
    • If snapshot, use debian/upstream/metadata’s Repository field
    • If neither is available, use guess-upstream-metadata from upstream-ontologist to guess the upstream Repository
  • Merge upstream version into packaging repository, possibly importing tarballs using pristine-tar
  • Update the changelog file to mention the new upstream version
  • Run some checks to ensure there are no unintentional changes, e.g.:
    • Scan diff between old and new for surprising license changes
      • Today, abort if there are any - in the future, maybe update debian/copyright
    • Check for obvious compatibility breaks - e.g. sonames changing
  • Attempt to update the packaging to reflect upstream changes
    • Refresh patches
  • Attempt to build the package with deb-fix-build, to deal with any missing dependencies
  • Run the autopkgtests with deb-fix-build to deal with missing dependencies, and abort if any tests fail

Results

When run over all packages in unstable (sid), this process works for a surprising number of them.

Fresh Releases

For fresh-releases (aka imports of upstream releases), processing all packages maintained in Git for which QA watch reports new releases (about 11,000):

That means about 2300 packages updated, and about 4000 unchanged.

Fresh Snapshots

For fresh-snapshots (aka imports of latest Git commit from upstream), processing all packages maintained in Git (about 26,000):

Or 5100 packages updated and 2100 for which there was nothing to do, i.e. no upstream commits since the last Debian upload.

As can be seen, this works for a surprising fraction of packages. It’s possible to get the numbers up even higher, by both improving the tooling, the autopkgtests and the metadata that is provided by packages.

Using these packages

All the packages that have been built can be accessed from the Janitor APT repository. More information can be found at https://janitor.debian.net/fresh, but in short - run:

1
2
3
4
5
6
echo deb "[arch=amd64 signed-by=/usr/share/keyrings/debian-janitor-archive-keyring.gpg]" \
    https://janitor.debian.net/ fresh-snapshots main | sudo tee /etc/apt/sources.list.d/fresh-snapshots.list
echo deb "[arch=amd64 signed-by=/usr/share/keyrings/debian-janitor-archive-keyring.gpg]" \
    https://janitor.debian.net/ fresh-releases main | sudo tee /etc/apt/sources.list.d/fresh-releases.list
sudo curl -o /usr/share/keyrings/debian-janitor-archive-keyring.gpg https://janitor.debian.net/pgp_keys
apt update

And then you can install packages from the fresh-snapshots (upstream git snapshots) or fresh-releases suites on a case-by-case basis by running something like:

1
apt install -t fresh-snapshots r-cran-roxygen2

Most packages are updated based on information provided by vcswatch and qa watch, but it’s also possible for upstream repositories to call a web hook to trigger a refresh of a package.

These packages were built against unstable, but should in almost all cases also work for testing.

Caveats

Of course, since these packages are built automatically without human supervision it’s likely that some of them will have bugs in them that would otherwise have been caught by the maintainer.

[1]I’m not saying that a monoculture is great here, but it does help distributions.

comments.

Migrating packaging from Bazaar to Git

A while ago I migrated most of my packages from Bazaar to Git. The rest of the world has decided to use Git for version control, and I don’t have enough reason to stubbornly stick with Bazaar and make it harder for myself to collaborate with others.

So I’m moving away from a workflow I know and have polished over the last few years - including the various bzr plugins and other tools involved. Trying to do the same thing using git is frustrating and time-consuming, but I’m sure that will improve with time. In particular, I haven’t found a good way to merge in a new upstream release (from a tarball) while referencing the relevant upstream commits, like bzr merge-upstream can. Is there a good way to do this? What helper tools can you recommend for maintaining a Debian package in git?

Having been upstream for bzr-git earlier, I used its git-remote-bzr implementation to do the conversions of the commits and tags:

% git clone bzr::/path/to/bzr/foo.bzr /path/to/git/foo.git

One of my last contributions to bzr-git was a bzr git-push-pristine-tar-deltas subcommand, which will export all bzr-builddeb-style pristine-tar metadata to a pristine-tar branch in a Git repository that can be used by pristine-tar directly or through something like git-buildpackage.

Once you have created a git clone of your bzr branch, it should be a matter of running bzr git-push-pristine-tar-deltas with the target git repository and the Debian package name:

% cd /path/to/bzr/foo.bzr
% bzr git-push-pristine-tar-deltas /path/to/git/foo.git foo
% cd /path/to/git/foo.git foo
% git branch
*  master
   pristine-tar

comments.

Bazaar: A retrospective

For the last 7 years I’ve been involved in the Bazaar project. Since I am slowly stepping down, I recently wrote a retrospective on the project as I experienced it for the last 7 years.

Thanks to a few kind people for proofreading earlier drafts; if you spot any errors, please let me know in the comments.

comments.

Samba 4.0.0, finally

This afternoon we released version 4.0.0 of Samba. This is a significant milestone, and I’m very proud of the result. Samba 4 is the first version that can be a domain controller in an Active Directory domain.

We embarked on this journey almost a decade ago - the first commit is from August 2003. It’s been a long and bumpy ride. I hardly recognize the people in this team photo from 2003 (I’m second from the left).

A lot has happened in that time. We wrote a few million lines of code. We migrated from CVS to Subversion to Git. We’ve drifted apart and grown back together as a team.

In my youthful naivity I predicted a release “within 1 or 2 years” during a talk at the NLUUG in 2004. But Active Directory was a lot harder than we thought, and there were quite a few other distractions as well. I’m glad this release, which is by far the biggest and longest running software project I have ever worked on, has finally happened.

Some older RCs of Samba 4 have already been packaged for Debian and Ubuntu, in the samba4 source package. For Debian jessie, these will be integrated into the main samba source package. Please use experimental if you do want to try the existing packages, as it is most up to date.

comments.

Last day at Canonical

This Friday will be my last day at Canonical.

It has been a lot of fun working here. There is so much I have learned in the last three years. I’m going to miss my colleagues.

Over the last couple of months I have slowly stepped down from my involvement in Bazaar and the Bazaar packaging in Debian and Ubuntu. I would like to stay involved in Ubuntu, but we will see how that goes.

I’m taking some time off until the end of the year to see the world and hack, before starting something new in February.

comments.

Summer of Code 2011

The Samba team is once again participating in the Summer of Code this year. This year we have 4 students working on various projects related to Samba.

This year I am mentoring Dhananjay Sathe, who is improving the GTK+ frontends for Samba. In particular, he is making it possible to manage shares and users of a remote Samba or Windows machine.

Dhananjay is also blogging about his progress.

comments.

libapache2-mod-bzr

During the last two days I hacked together a Bazaar module for Apache. This module makes it possible to easily enable the Bazaar smart server for Bazaar branches. It also can display a simple placeholder page for Bazaar branches without working tree. It’s surprisingly easy to write Apache modules.

The main advantage this has over a mod_wsgi / mod_python / mod_fcgi setup is that it doesn’t require any additional Python hacking on the users side or other configuration outside of Apache, and it doesn’t require configuration for each single branch in the Apache configuration. In the future I’d also like to support the settings “BazaarFrontend Wikkid” and “BazaarFrontend Loggerhead“.

The configuration is currently as simple as:

1
2
3
LoadModule bzr_module /usr/lib/apache2/modules/mod_bzr.so
BazaarSmart on
BazaarFrontend Basic

in your apache2.conf. The BazaarSmart and BazaarFrontend directives can appear in <Directory> or <Location> clauses as well, if you’d like to have different behaviour for different directories.

At the moment this project is a proof of concept, and probably not something you would want to run in production. For example, there is no way to limit the access to a branch to read only. I need to double-check there are no threading issues.

Testing and patches are welcome. The project is hosted here:

Currently Playing: Stream of Passion - Calliopeia

comments.

On the way to Samba 4: Part 2

It’s been more than a month since the last status update on my Samba 4 work - much more than the two weeks I promised.

During the holidays I finally managed to release the new alpha of Samba 4, as well as releases of some of our companion libraries (tdb, talloc, tevent and ldb). The release includes a significant amount of bug fixes and a lot of work towards a properly functioning Active Directory DC, too much to list here.

This release I’ve mainly been involved in improving our Python bindings and our handling of internal and external libraries. We now use symbol versioning for our copy of Heimdal Kerberos as well as some of our other libraries. Hopefully this will fix some of the issues users of the evolution-mapi plugin have been seeing where they end up with both MIT Kerberos and Heimdal Kerberos loaded into the same process (with all the consequences of overlapping symbol names). Samba 4 now also has the ability to work with the system Heimdal rather than using the bundled copy. I have packaged alpha14 for Debian and Ubuntu (fixing most of the open bugs against the Samba 4 package in the BTS), but am currently waiting for the new release of ldb to pass through NEW before I can upload.

The next release is scheduled for the first week of February.

Currently Playing: Stream of Passion - Haunted

comments.

On the way to Samba 4: Part 1

After Samba XP 2008 Andrew and I started keeping a wiki page with our bi-weekly goals and achievements for Samba 4. Because planning in a Free Software project is hard (time availability and priorities change over time, and other volunteers are equally unpredictable) we called this our “Fantasy Page”; it listed things we wanted to work on next (“fantasies”), but reality being what it is we would usually actually end up working on something entirely different. We discussed our progress and new plans in - what I would now call - a bi-weekly standup call.

There were several reasons for doing this. It gave us some sense of direction as well as a sense of accomplishment; a way to look back at the end of the year and realize how much we had actually achieved. Because Samba 4 is such a long term project (it is 7 years old at this point) it is easy to become disillusioned, to look back at a year of commits and to not see the gradual improvement, just the fact that there is no release yet.

We managed to keep this up for two years, much longer than I had anticipated, and eventually started to slip last year.

More recently Kai and Tridge have started to blog weekly about their efforts to make Samba 4.0 a reality and I’m going to join them by trying to blog regularly - every two weeks - about my contributions, even if there were none.

In the next two weeks I plan to work on finally getting alpha 14 of Samba 4 out and on fixing the daily builds of Samba 4 and OpenChange for Ubuntu on Launchpad after we did a massive reorganization of the private libraries in Samba 4.

Current Playing: Zero 7 - Somersault

comments.

Mumble and bluetooth

Mumble is an open source, low-latency, high quality voice chat application that we’re using at Canonical, and which Samba has recently also adopted.

After I busted the cable of my cabled headset and inspired by Matt’s post about Mumble and Bluetooth I bought a cheap Nokia Bluetooth headset that works with my laptop as well as my phone. By using BlueMan I even found that the headset worked out of the box on Maverick.

A nice feature in Mumble is that it is controllable using D-Bus. Blueman supports running an arbitrary command when the answer button is pressed. Combining these two features, it is possible to automatically toggle mute when the answer button is pressed:

#!/usr/bin/python
import dbus
bus = dbus.SessionBus()
mumble = bus.get_object("net.sourceforge.mumble.mumble", "/")
is_muted = mumble.isSelfMuted()
mumble.setSelfMuted(not is_muted)

To use this script, set its path in the configuration tab for the “Headset” plugin in blueman.

The only remaining problem with this setup is that I can’t keep the headset on during my work day, as I don’t have a way to put it in standby mode automatically. This means that my battery runs out pretty quickly, even when nothing is happening on Mumble.

Currently Playing: Red Sparowes - Finally As That Blazing Sun Shone

comments.

OpenChange server and SOGo

There’s more good news on the OpenChange front. Julien has been working together with Wolfgang and Ludovic from Inverse to leverage the server-side support in OpenChange to provide native Exchange server support in SOGo.

A couple of days ago we announced that there now is an initial version that allows the use of Outlook against a SOGo server through OpenChange.

There is a screencast up on youtube (there is also a .mov version of the screencast).

As far as I know, this is the first time it’s possible to actually use Outlook clients with a non-Microsoft Exchange-compatible server, without the need for plugins on the Outlook side. And it’s all Free Software. Of course, this is just a preview, and not something we’d recommend everybody to run in production just yet. But it’s exciting to finally see this come together.

We already have OpenChange packages in Debian and Ubuntu but I hope I can help get SOGo packaged for both distributions as well.

comments.

Samba 4 and OpenChange daily Ubuntu packages

Daily builds

As of a month ago there are Ubuntu archives with fresh packages of Samba 4 and OpenChange, built on a daily basis day from the latest upstream revision.

This means that it is now possible to run a version of Samba 4 that is less than 24 hours old, without having to know how to extract source code from the version control system that upstream is using, without having to know how to build and install an application from source, but perhaps most importantly: without having to go through the tedious process of manually updating the source code and rebuilding.

OpenChange is tightly coupled to Samba 4, so installing a new version of OpenChange usually involves installing a new version of Samba 4 as well. To make matters more confusing, the two projects use different version control systems (Samba 4 is in Git, while OpenChange is in Subversion) and different build systems (Samba 4 uses waf, OpenChange uses autoconf and make).

I have been involved in Samba 4 and OpenChange as an upstream developer and more recently also as a packager for both Debian and Ubuntu.

As an upstream developer for both these projects it is important for me that users can easily run the development versions. It makes it possible for interested users to confirm the fixes for issues they have reported and to test new features. The more users run the development version, the more confident I can be as a developer that doing a release will not cause any unexpected surprises.

As a packager it is useful to know when there are upstream changes that are going to break my package with the next release.

Recipes

The daily builds work using so-called recipes which describe how to build a Debian source package from a set of Bazaar branches. For example, the Samba 4 recipe looks like this:

1
2
3
4
# bzr-builder format 0.2 deb-version 4.0.0~alpha14~bzr{revno}~ppa{revno:packaging}+{revno:debian}
lp:samba
merge debian lp:~samba-team/samba/unstable
merge packaging lp:~samba-team/samba/4.0-ppa-maverick

This dictates that a source package should be built by taking the upstream Samba branch and merging the Debian packaging and some recipe-specific tweaking. The last bit on the first line indicates the version string to be used when generating a changelog entry for the daily build.

Every night Launchpad (through bzr-builder) merges these branches and attempts to build the resulting source package, e-mailing me in case of build problems. Generally I fix issues that come up by committing directly to upstream VCS or to the Debian packaging branch. There is no overhead in maintaining the daily build after I’ve set it up.

For more information on creating source package recipes, see getting started.

Toolchain

The entire toolchain that does the daily package builds for Ubuntu is Free Software, and I have contributed to various bits of that toolchain over the years. It’s exciting to see everything come together.

Soyuz

Launchpad consists of multiple pillars - one of those pillars is Soyuz, which I hack on as part of my day job at Canonical. Soyuz is responsible for the archive management and package building. Debian source packages (a combination of upstream source code and packaging metadata) get uploaded by users and then built for various architectures on our buildfarm and published to the Ubuntu archive or to users personal package archives.

Launchpad-code

Another pillar of Launchpad is Launchpad-code, which is responsible for the hosting and management of version control branches. Launchpad users can either host their branches on Launchpad directly or mirror branches (either native Bazaar branches or branches in a foreign format such as Subversion, Git or Mercurial). The mirrorring of native and foreign branches happens using standard Bazaar API’s. In the case of Samba and OpenChange we import the branches of the upstream projects (Samba is in Git, OpenChange is in Subversion) and the packaging for both projects is in Bazaar.

Launchad-code calls out to Bazaar to do the actual mirrorring. Over the last few years I have done a lot of work to improve Bazaars support for foreign branches, in particular on supporting Subversion, Git and Mercurial. As the code mirrorring in Launchpad is one of the biggest users of bzr-svn and bzr-git it has helped find some of the more obscure bugs in those plugins over the last few years, to the point where there are only a handful of issues with Git imports and Subversion imports left.

bzr-git and dulwich

bzr-git provides transparent access to Git repositories from within Bazaar and is built on top of Dulwich. Dulwich is a Python library that provides access to the Git file formats and protocols that is completely independent of Bazaar. James Westby originally started it and I adopted it for bzr-git and further extended it. There are now several other projects that use it as well, including hg-git, and rabbitvcs. Apart from James and myself, almost two dozen other people have contributed it so far.

bzr-svn and subvertpy

bzr-svn provides transparant access to Subversion repositories in Bazaar. When I grew frustrated with the existing Subversion Python bindings for various reasons, I decided to create independent Python bindings for Subversion from scratch. These bindings have since been split out into a separate project - subvertpy - and other projects have since also started using them, e.g. hgsubversion and basie.

Using the daily builds

To use the Samba 4 and OpenChange daily builds (Ubuntu Maverick only for now), run:

1
2
$ apt-add-repository ppa:samba-team/ppa
$ apt-add-repository ppa:openchange/daily-builds

Currently Playing: Karnivool - Themata

comments.

Proof of concept OpenChange server working

Seeing this makes me very happy. It’s taken us a couple of years to get to this point but we’ve finally made it, mostly thanks to the dedication and persistence of Julien and Brad.

comments.

CtrlProxy: Looking for a new maintainer

After over 7 years of working on it off and on, I’m looking for somebody to help maintain (and eventually take over) CtrlProxy. I started working on CtrlProxy somewhere in 2002, only a short while after Wilmer started hacking on BitlBee. If I remember correctly I started working on it because I didn’t want to run a separate dircproxy (the only real competitor at the time) instance (with configuration) for each IRC network that I connected to. It was also just a good excuse to play with the IRC protocol a bit.

Over the years, CtrlProxy has served as a playground for me to try out new and interesting things. It’s been rewritten or severely refactored several times in its early history, the latest time being the 3.0 release (from 2005). I’ve tried different build systems, I’ve tried different implementation languages, I’ve tried different configuration file formats, I’ve tried different support libraries, I’ve tried different version control systems, I’ve tried different documentation formats. So while it’s definitely been a very educational project for me personally, I haven’t really had the time or the interest to dedicate to the project that it deserved during the last few years. This was mostly because there were other more interesting FOSS projects I spent my spare cycles on.

These days there are plenty of other good IRC proxies out there, such as BIP, so I doubt CtrlProxy will be missed if it were to disappear. Despite that, if anybody is interested in taking over, please send me an email (jelmer@samba.org) or contact me on IRC (jelmer on the OFTC and Freenode networks).

Currently Playing: Anathema - Shroud of False

comments.