Jelmer Vernooijhttps://www.jelmer.uk/2023-09-10T20:00:00+00:00Transcontinental Race No 92023-09-10T20:00:00+00:002023-09-10T20:00:00+00:00Jelmer Vernooijtag:www.jelmer.uk,2023-09-10:tcrno9.html<p>After cycling the <a class="reference external" href="https://www.northcape4000.com/">Northcape 4000</a> (from Italy to northern Norway) last year,
I signed up for the <a class="reference external" href="https://www.transcontinental.cc">transcontinental race</a>
this year.</p>
<p>The Transcontinental is bikepacking race across Europe, self-routed (but with some mandatory checkpoints), unsupported and
with a distance of usually somewhere around 4000 km. The cut-off time is 15 days, with the winner usually taking 7-10 days.</p>
<p>This year, the route went from Belgium to Thessaloniki in Greece, with control points in northern Italy, Slovenia, Albania and
Meteora (Greece).</p>
<p>The event was great - it was well organised and communication was a lot better than at the Northcape. It did feel
very different from the Northcape, though, being a proper race. Participants
are not allowed to draft off each other or help each other, though a quick chat
here or there as you pass people is possible, or when you’re both stopped at a
shop or control point.</p>
<img alt="" src="/images/tcr-rules.jpg" style="width: 408px; height: 307px;" />
<div class="section" id="my-experience">
<h2>My experience</h2>
<p>The route was beautiful - the first bit through France was a bit monotonic, but
especially the views in the alps were amazing. Like with other long events,
the first day or two can be hard but once you get into the rhythm of things
it’s a lot easier.</p>
<p>From early on, I lost a lot of time. We started in the rain, and I ran
several flats in a row, just 4 hours in. In addition to that, the thread on my pump
had worn so it wouldn’t fit on some of my spare tubes, and my tubes were all
<span class="caps">TPU</span> - which are hard to patch. So at 3 <span class="caps">AM</span> I found myself by the side of an N-road in
France without any usable tubes to put in my rear wheel. I ended up walking 20km
to the nearest town with a bike shop, where they fortunately had good old butyl
tubes and a working pump. But overall, this cost me about 12 hours in total.</p>
<p>In addition to that, my time management wasn’t great. On previous rides, I’d usually
gotten about 8 hours of sleep per night while staying in hotels. On the
transcontinental I had meant to get less sleep but still stay in hotels most
night, but I found that not all hotels accomodated
well for that - especially with a bike. So I ended up getting more sleep than I
had intended, and spending more time off the bike than I had planned - close to
11 or 12 hours per day. I hadn’t scheduled much time off work after the finish either, so
arriving in Greece late wasn’t really an option.</p>
<p>And then, on an early morning in Croatia (about 2000km in) in heavy fog, I rode into a kerb at 35 km/h, bending
the rim of my front wheel (but fortunately not coming off my bike). While I probably would have been able to continue with
a replacement wheel (and mailing the broken one home), that would have taken
another day to sort out and I almost certainly wouldn’t have been
able to source a new dynamo wheel in Croatia - which would have made night time riding a lot harder.
So I decided to scratch and take the train home from Zagreb.</p>
<p>Overall, I really enjoyed the event and I think I’ve learned some useful
lessons. I’ll probably try again next year.</p>
</div>
Porting Python projects to Rust2023-06-02T17:00:00+00:002023-06-02T17:00:00+00:00Jelmer Vernooijtag:www.jelmer.uk,2023-06-02:port-py-to-rust.html<p>I’ve recently been working on porting some of my Python code to rust, both for performance reasons, and because of the strong typing in the language. As a fan of Haskell, I also just really enjoy using the language.</p>
<p>Porting any large project to a new language can be a challenge. There is a temptation to do a rewrite from the ground-up in idiomatic rust and using all new fancy features of the language.</p>
<div class="section" id="porting-in-one-go">
<h2>Porting in one go</h2>
<p>However, this is a bit of a trap:</p>
<ul class="simple">
<li>It blocks other work. It can take a long time to finish the rewrite, during which time there is no good place to make other bug fixes/feature changes. If you make the change in the python branch, then you may also have to patch the in-progress rust fork.</li>
<li>No immediate return on investment. While the rewrite is happening, all of the investment in it is sunk costs.</li>
<li>Throughout the process, you can only run the tests for subsystems that have already been ported. It’s common to find subtle bugs later in code ported early.</li>
<li>Understanding existing code, porting it and making it idiomatic rust all at the same time takes more time and post-facto debugging.</li>
</ul>
</div>
<div class="section" id="iterative-porting">
<h2>Iterative porting</h2>
<p>Instead, we’ve found that it works much better to take an iterative approach. One of the hidden gems of rust is the excellent PyO3 crate, which allows creating python bindings for rust code in a way that is several times less verbose and less painful than C or <span class="caps">SWIG</span>. Because of rust’s strong ownership model, it’s also really hard to muck up e.g. reference counts when creating Python bindings for rust code.</p>
<p>We port individual functions or classes to rust one at a time, starting with functionality that doesn’t have dependencies on other python code and gradually working our way up the call stack.</p>
<p>Each subsystem of the code is converted to two matching rust crates: one with a port of the code to pure rust, and one with python bindings for the rust code. Generally multiple python modules end up being a single pair of rust crates.</p>
<p>The signature for the pure Rust code follow rust conventions, but the business logic is mostly ported as-is (just in rust syntax) and the signatures of the python bindings match that of the original python code.</p>
<p>This then allows running the original python tests to verify that the code still behaves the same way. Changes can also immediately land on the main branch.</p>
<p>A subsequent step is usually to refactor the rust code to be more idiomatic - all the while keeping the tests passing. There is also the potential to e.g. switch to using external rust crates (with perhaps subtly different behaviour), or drop functionality altogether.</p>
<p>At some point, we will also port the tests from python to rust, and potentially drop the python bindings - once all the caller’s have been converted to rust.</p>
</div>
<div class="section" id="example">
<h2>Example</h2>
<p>For example, imagine I have a Python module <em>janitor/mail_filter.py</em> with this function:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">def</span> <span class="nf">parse_plain_text_body</span><span class="p">(</span><span class="n">text</span><span class="p">):</span>
<span class="n">lines</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">splitlines</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">line</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">lines</span><span class="p">):</span>
<span class="k">if</span> <span class="n">line</span> <span class="o">==</span> <span class="s1">'Reply to this email directly or view it on GitHub:'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">lines</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'#'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="p">(</span><span class="n">line</span> <span class="o">==</span> <span class="s1">'For more details, see:'</span>
<span class="ow">and</span> <span class="n">lines</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'https://code.launchpad.net/'</span><span class="p">)):</span>
<span class="k">return</span> <span class="n">lines</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">]</span>
<span class="k">try</span><span class="p">:</span>
<span class="p">(</span><span class="n">field</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">':'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">field</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'merge request url'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">value</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">return</span> <span class="kc">None</span>
</pre></div></td></tr></table></div>
<p>Porting this to rust naively (in a crate I’ve called “mailfilter”) it might look something like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">parse_plain_text_body</span><span class="p">(</span><span class="n">text</span>: <span class="kp">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">lines</span>: <span class="nb">Vec</span><span class="o"><&</span><span class="kt">str</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">text</span><span class="p">.</span><span class="n">lines</span><span class="p">().</span><span class="n">collect</span><span class="p">();</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">line</span><span class="p">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">lines</span><span class="p">.</span><span class="n">iter</span><span class="p">().</span><span class="n">enumerate</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">&</span><span class="s">"Reply to this email directly or view it on GitHub:"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">lines</span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">].</span><span class="n">split</span><span class="p">(</span><span class="sc">'#'</span><span class="p">).</span><span class="n">next</span><span class="p">().</span><span class="n">unwrap</span><span class="p">().</span><span class="n">to_string</span><span class="p">());</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">&</span><span class="s">"For more details, see:"</span>
<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">lines</span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">].</span><span class="n">starts_with</span><span class="p">(</span><span class="s">"https://code.launchpad.net/"</span><span class="p">)</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">lines</span><span class="p">[</span><span class="n">i</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">].</span><span class="n">to_string</span><span class="p">());</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">((</span><span class="n">field</span><span class="p">,</span><span class="w"> </span><span class="n">value</span><span class="p">))</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">line</span><span class="p">.</span><span class="n">split_once</span><span class="p">(</span><span class="sc">':'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">field</span><span class="p">.</span><span class="n">to_lowercase</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">"merge request url"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">value</span><span class="p">.</span><span class="n">trim</span><span class="p">().</span><span class="n">to_string</span><span class="p">());</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nb">None</span>
<span class="w"> </span><span class="p">}</span>
</pre></div></td></tr></table></div>
<p>Bindings are created in a crate called mailfilter-py, which looks like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">use</span><span class="w"> </span><span class="n">pyo3</span>::<span class="n">prelude</span>::<span class="o">*</span><span class="p">;</span>
<span class="w"> </span><span class="cp">#[pyfunction]</span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">parse_plain_text_body</span><span class="p">(</span><span class="n">text</span>: <span class="kp">&</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">janitor_mail_filter</span>::<span class="n">parse_plain_text_body</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="cp">#[pymodule]</span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">_mail_filter</span><span class="p">(</span><span class="n">py</span>: <span class="nc">Python</span><span class="p">,</span><span class="w"> </span><span class="n">m</span>: <span class="kp">&</span><span class="nc">PyModule</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">m</span><span class="p">.</span><span class="n">add_function</span><span class="p">(</span><span class="n">wrap_pyfunction</span><span class="o">!</span><span class="p">(</span><span class="n">parse_plain_text_body</span><span class="p">,</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="o">?</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span>
<span class="w"> </span><span class="p">}</span>
</pre></div></td></tr></table></div>
<p>The metadata for the crates is what you’d expect. mailfilter-py uses PyO3 and depends on mailfilter.</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">[package]</span>
<span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"mailfilter-py"</span>
<span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"0.0.0"</span>
<span class="w"> </span><span class="n">authors</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s2">"Jelmer Vernooij <jelmer@jelmer.uk>"</span><span class="p">]</span>
<span class="w"> </span><span class="n">edition</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"2018"</span>
<span class="w"> </span><span class="k">[lib]</span>
<span class="w"> </span><span class="n">crate-type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s2">"cdylib"</span><span class="p">]</span>
<span class="w"> </span><span class="k">[dependencies]</span>
<span class="w"> </span><span class="n">janitor-mail-filter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">path</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s2">"../mailfilter"</span><span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">pyo3</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s2">">=0.14"</span><span class="p">,</span><span class="w"> </span><span class="n">features</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">[</span><span class="s2">"extension-module"</span><span class="p">]}</span>
</pre></div></td></tr></table></div>
<p>I use python-setuptools-rust to get the python ecosystem to build the python bindings. Here is what setup.py looks like:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span>
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="ch">#!/usr/bin/python3</span>
<span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="n">setup</span>
<span class="kn">from</span> <span class="nn">setuptools_rust</span> <span class="kn">import</span> <span class="n">RustExtension</span><span class="p">,</span> <span class="n">Binding</span>
<span class="n">setup</span><span class="p">(</span>
<span class="n">rust_extensions</span><span class="o">=</span><span class="p">[</span><span class="n">RustExtension</span><span class="p">(</span>
<span class="s2">"janitor._mailfilter"</span><span class="p">,</span> <span class="s2">"crates/mailfilter-py/Cargo.toml"</span><span class="p">,</span>
<span class="n">binding</span><span class="o">=</span><span class="n">Binding</span><span class="o">.</span><span class="n">PyO3</span><span class="p">)],</span>
<span class="p">)</span>
</pre></div></td></tr></table></div>
<p>And of course, <a class="reference external" href="https://pypi.org/project/setuptools-rust/">setuptools-rust</a> needs to be listed as a setup requirement in <em>pyproject.toml</em> or <em>setup.cfg</em>.</p>
<p>After that, we can replace the original python code with a simple import and verify that the tests still run:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">from</span> <span class="nn">._mailfilter</span> <span class="kn">import</span> <span class="n">parse_plain_text_body</span>
</pre></div></td></tr></table></div>
<p>Of course, not all bindings are as simple as this. Iterators in particular are more complicated, as is code that has a loose idea of ownership in python. But I’ve found that the time investment is usually well worth the ability to land changes on the development head early and often.</p>
<p>I’d be curious to hear if people have had success with other approaches to porting Python code to Rust. If you do, please leave a comment.</p>
</div>
The Kali Janitor2023-03-08T22:25:00+01:002023-03-08T22:25:00+01:00Jelmer Vernooijtag:www.jelmer.uk,2023-03-08:kali-janitor.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p><a class="reference external" href="https://www.kali.org/">Kali Linux</a> have been running their own instance
of the <a class="reference external" href="https://github.com/jelmer/janitor/">Janitor</a> for the last year, under
the <a class="reference external" href="https://gitlab.com/kali-bot">kali-bot</a> user on GitLab.
Their web site has <a class="reference external" href="https://www.kali.org/docs/development/leveraging-the-kali-bot/">some excellent documentation</a> explaining how the bot works.</p>
<p>Both projects share some common components - the <a class="reference external" href="https://github.com/jelmer/janitor">core janitor codebase</a>, <a class="reference external" href="https://github.com/jelmer/silver-platter">Silver-Platter</a> and the various codemods
(<a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> and
<a class="reference external" href="https://launchpad.net/brz-debian">deb-new-upstream</a>). The site and some
of the review logic is different for Kali.</p>
<p>The Kali bot has several campaigns:</p>
<ul class="simple">
<li><a class="reference external" href="https://janitor.kali.org/lintian-fixes/">Lintian-Fixes</a></li>
<li><a class="reference external" href="https://janitor.kali.org/fresh-releases/">Fresh releases</a></li>
<li><a class="reference external" href="https://janitor.kali.org/fresh-snapshots/">Fresh snapshots</a></li>
<li><a class="reference external" href="https://janitor.kali.org/merge-from-debian/">Merge from Debian (and other distributions)</a></li>
</ul>
<p>The last campaign doesn’t exist in the Debian janitor, and pulls in new changes from packages that have been imported from other distributions.</p>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
Silver Platter Batch Mode2023-02-25T22:44:00+01:002023-02-25T22:44:00+01:00Jelmer Vernooijtag:www.jelmer.uk,2023-02-25:silver-platter-batch.html<div class="section" id="background">
<h2>Background</h2>
<p><a class="reference external" href="https://github.com/jelmer/silver-platter">Silver-Platter</a> makes it easier to
publish automated changes to repositories. However, in its default mode, the
only option for reviewing changes before publishing them is to run in dry-run mode.
This can be quite cumbersome if you have a lot of repositories.</p>
<p>A new “batch” mode now makes it possible to generate a large number of changes
against different repositories using a script, review and optionally alter the
diffs, and then all publish them (and potentially refresh them later if
conflicts appear).</p>
</div>
<div class="section" id="example-running-pyupgrade">
<h2>Example running pyupgrade</h2>
<p>I’m using the <a class="reference external" href="https://raw.githubusercontent.com/jelmer/silver-platter/master/examples/pyupgrade.yaml">pyupgrade example</a> recipe that comes with silver-platter.</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">---</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">name</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pyupgrade</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="s">'pyupgrade</span><span class="nv"> </span><span class="s">--exit-zero-even-if-changed</span><span class="nv"> </span><span class="s">$(find</span><span class="nv"> </span><span class="s">-name</span><span class="nv"> </span><span class="s">"test_*.py")'</span>
<span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">propose</span>
<span class="w"> </span><span class="nt">merge-request</span><span class="p">:</span>
<span class="w"> </span><span class="nt">commit-message</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Upgrade Python code to a modern version</span>
</pre></div>
<p>And a list of candidate repositories to process in <tt class="docutils literal">candidates.yaml</tt>.</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">---</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">- url</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/jelmer/dulwich</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/jelmer/xandikos</span>
</pre></div>
<p>With these in place, the updated repositories can be created:</p>
<div class="highlight"><pre><span></span><span class="gp"> $ </span>svp<span class="w"> </span>batch<span class="w"> </span>generate<span class="w"> </span>--recipe<span class="o">=</span>pyupgrade.yaml<span class="w"> </span>--candidates<span class="o">=</span>candidate.syml<span class="w"> </span>pyupgrade
</pre></div>
</div>
<div class="section" id="the-intermediate-results">
<h2>The intermediate results</h2>
<p>This will create a directory called <tt class="docutils literal">pyupgrade</tt>, with a clone of each of the repositories.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>ls<span class="w"> </span>pyupgrade
<span class="go">batch.yaml dulwich xandikos</span>
<span class="gp">$ </span><span class="nb">cd</span><span class="w"> </span>pyupgrade/dulwich
<span class="gp">$ </span>git<span class="w"> </span>log
<span class="go">commit 931f9ffb26e9143c56f20e0b85e6ddb0a8eee2eb (HEAD -> master)</span>
<span class="go">Author: Jelmer Vernooij <jelmer@jelmer.uk></span>
<span class="go">Date: Sat Feb 25 22:28:12 2023 +0000</span>
<span class="go">Run pyupgrade</span>
<span class="go">diff --git a/dulwich/tests/compat/test_client.py b/dulwich/tests/compat/test_client.py</span>
<span class="go">index 02ab6c0a..9b0661ed 100644</span>
<span class="go">--- a/dulwich/tests/compat/test_client.py</span>
<span class="go">+++ b/dulwich/tests/compat/test_client.py</span>
<span class="go">@@ -628,7 +628,7 @@ class HTTPGitServer(http.server.HTTPServer):</span>
<span class="go"> self.server_name = "localhost"</span>
<span class="go"> def get_url(self):</span>
<span class="go">- return "http://{}:{}/".format(self.server_name, self.server_port)</span>
<span class="go">+ return f"http://{self.server_name}:{self.server_port}/"</span>
<span class="go"> class DulwichHttpClientTest(CompatTestCase, DulwichClientTestBase):</span>
<span class="go">...</span>
</pre></div>
<p>There is also a file called <tt class="docutils literal">batch.yaml</tt> that describes the pending changes:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pyupgrade</span>
<span class="nt">work</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/dulwich/dulwich</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dulwich</span>
<span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Upgrade to modern Python statements</span>
<span class="w"> </span><span class="nt">commit-message</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Run pyupgrade</span>
<span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">propose</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/jelmer/xandikos</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">xandikos</span>
<span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Upgrade to modern Python statements</span>
<span class="w"> </span><span class="nt">commit-message</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Run pyupgrade</span>
<span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">propose</span>
<span class="nt">recipe</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">../pyupgrade.yaml</span>
</pre></div></td></tr></table></div>
<p>At this point the changes can be reviewed, and <tt class="docutils literal">batch.yaml</tt> edited as the user sees fit - they
can remove entries that don’t appear to be correct, edit the metadata for the merge
requests, etc. It’s also possible to make changes to the clones.</p>
<p>Once you’re happy, publish the results:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>svp<span class="w"> </span>batch<span class="w"> </span>publish<span class="w"> </span>pyupgrade
</pre></div>
<p>This will publish all the changes, using the mode and parameters specified in
<tt class="docutils literal">batch.yaml</tt>.</p>
<p><tt class="docutils literal">batch.yaml</tt> is automatically stripped of any entries in work that have fully
landed, i.e. where the pull request has been merged or where the changes were
pushed to the origin.</p>
<p>To check up on the status of your changes, run <tt class="docutils literal">svp batch status</tt>:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>svp<span class="w"> </span>batch<span class="w"> </span>status<span class="w"> </span>pyupgrade
</pre></div>
<p>To refresh any merge proposals that may have become out of date, simply run publish again:</p>
<div class="highlight"><pre><span></span><span class="go">svp batch publish pyupgrade</span>
</pre></div>
</div>
Detecting Package Transitions2022-11-29T00:04:00+01:002022-11-29T00:04:00+01:00Jelmer Vernooijtag:www.jelmer.uk,2022-11-29:detecting-transitions.html<p>Larger transitions in Debian are usually announced on e.g. debian-devel, but
it’s harder to track the current status of all transitions. Having done a lot
of <span class="caps">QA</span> uploads recently, I have on occasion uploaded packages involved in a transition.
This can be unhelpful for the people handling the transition, but there’s also
often not much point in uploading if your uploads are going to get stuck.</p>
<p>Talking to one of the release managers at a recent <span class="caps">BSP</span>, it was great to find out that
the <a class="reference external" href="https://release.debian.org">release team</a>
actually publish <a class="reference external" href="https://release.debian.org/transitions/export/packages.yaml">a data dump with which packages are
involved in which transitions</a>.</p>
<p>Here’s the script I use to find out about the transitions the package
in my current working directory is involved in:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span></pre></div></td><td class="code"><div><pre><span></span><span class="ch">#!/usr/bin/python3</span>
<span class="kn">from</span> <span class="nn">urllib.request</span> <span class="kn">import</span> <span class="n">urlopen</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">debian.deb822</span> <span class="kn">import</span> <span class="n">Deb822</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'debian/control'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">package</span> <span class="o">=</span> <span class="n">Deb822</span><span class="p">(</span><span class="n">f</span><span class="p">)[</span><span class="s1">'Source'</span><span class="p">]</span>
<span class="k">with</span> <span class="n">urlopen</span><span class="p">(</span><span class="s2">"https://release.debian.org/transitions/export/packages.yaml"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">safe_load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">find_transitions</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">package</span><span class="p">):</span>
<span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="k">if</span> <span class="n">entry</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="o">!=</span> <span class="n">package</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">return</span> <span class="nb">dict</span><span class="p">(</span><span class="n">entry</span><span class="p">[</span><span class="s1">'list'</span><span class="p">])</span>
<span class="k">return</span> <span class="p">{}</span>
<span class="n">transitions</span> <span class="o">=</span> <span class="n">find_transitions</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">package</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">transitions</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span> <span class="k">if</span> <span class="s1">'ongoing'</span> <span class="ow">in</span> <span class="n">transitions</span><span class="o">.</span><span class="n">values</span><span class="p">()</span> <span class="k">else</span> <span class="mi">0</span><span class="p">)</span>
</pre></div></td></tr></table></div>
<p>In practice, the output looks something like this:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>debcheckout<span class="w"> </span>bctoolbox
<span class="go">git clone https://salsa.debian.org/pkg-voip-team/linphone-stack/bctoolbox.git bctoolbox ...</span>
<span class="go">Cloning into 'bctoolbox'...</span>
<span class="go">...</span>
<span class="gp">$ </span><span class="nb">cd</span><span class="w"> </span>bctoolbox
<span class="gp">$ </span><span class="k">in</span>-transition.py
<span class="go">{'auto-upperlimit-libbctoolbox1': 'ongoing'}</span>
</pre></div>
Bye bye Batavus2022-08-27T11:11:00+02:002022-08-27T11:11:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2022-08-27:bye-bye-batavus.html<p>After almost 20 years of service and upwards of 50.000km traveled, it’s finally time to retire my good old Batavus.</p>
<p>I bought this bike in 2003 for less than 1000 euros, but it’s done surprisingly well. It’s been across the continent and back a couple of times.</p>
<p>The potholes in England have proved too much though and it’s developed a fairly large crack in the front tube.</p>
<img alt="Before disassembling in Linköping" src="/images/batavus-linkoping.jpg" style="width: 410px; height: 307px;" />
Personal Streaming Audio Server2022-01-04T19:00:00+01:002022-01-04T19:00:00+01:00Jelmer Vernooijtag:www.jelmer.uk,2022-01-04:navidrome.html<p>For a while now, I’ve been looking for a good way to stream music from my home
music collection on my phone.</p>
<p>There are quite a few options for music servers that support streaming. However,
Android apps that can stream music from one of those servers tend to be
unmaintained, clunky or slow (or more than one of those).</p>
<p>It is possible to use something that runs in a web server, but that means
no offline caching - which can be quite convenient in spots without
connectivity, such as the Underground or other random bits of London with poor
cell coverage.</p>
<div class="section" id="server">
<h2>Server</h2>
<p>Most music servers today support some form of the subsonic <span class="caps">API</span>.</p>
<p>I’ve tried a couple, with mixed results:</p>
<blockquote>
<ul class="simple">
<li>supysonic; Python. Slow. Ran into some issues with subsonic clients. No real web <span class="caps">UI</span>.</li>
<li>gonic; Go. Works well <span class="amp">&</span> fast enough. Minimal web <span class="caps">UI</span>, i.e. no ability to play music from a browser.</li>
<li>airsonic; Java. Last in a chain of (abandoned) forks. More effort to get to work, and resource intensive.</li>
</ul>
</blockquote>
<p>Eventually, I’ve settled on <a class="reference external" href="https://www.navidrome.org/">Navidrome</a>. It’s got a
couple of things going for it:</p>
<blockquote>
<ul class="simple">
<li>Good subsonic implementation that worked with all the Android apps I used it with.</li>
<li>Great Web <span class="caps">UI</span> for use in a browser</li>
</ul>
</blockquote>
<p>I run Navidrome in Kubernetes. It’s surprisingly easy to get going. Here’s the
deployment I’m using:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span>
<span class="normal">48</span>
<span class="normal">49</span>
<span class="normal">50</span>
<span class="normal">51</span>
<span class="normal">52</span>
<span class="normal">53</span>
<span class="normal">54</span>
<span class="normal">55</span>
<span class="normal">56</span>
<span class="normal">57</span>
<span class="normal">58</span>
<span class="normal">59</span>
<span class="normal">60</span>
<span class="normal">61</span>
<span class="normal">62</span>
<span class="normal">63</span>
<span class="normal">64</span>
<span class="normal">65</span>
<span class="normal">66</span>
<span class="normal">67</span>
<span class="normal">68</span>
<span class="normal">69</span></pre></div></td><td class="code"><div><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">apps/v1</span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Deployment</span>
<span class="nt">metadata</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w"> </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span>
<span class="w"> </span><span class="nt">matchLabels</span><span class="p">:</span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="w"> </span><span class="nt">template</span><span class="p">:</span>
<span class="w"> </span><span class="nt">metadata</span><span class="p">:</span>
<span class="w"> </span><span class="nt">labels</span><span class="p">:</span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="w"> </span><span class="nt">spec</span><span class="p">:</span>
<span class="w"> </span><span class="nt">containers</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">deluan/navidrome:latest</span>
<span class="w"> </span><span class="nt">imagePullPolicy</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Always</span>
<span class="w"> </span><span class="nt">resources</span><span class="p">:</span>
<span class="w"> </span><span class="nt">limits</span><span class="p">:</span>
<span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="s">".5"</span>
<span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="s">"2Gi"</span>
<span class="w"> </span><span class="nt">requests</span><span class="p">:</span>
<span class="w"> </span><span class="nt">cpu</span><span class="p">:</span><span class="w"> </span><span class="s">"0.1"</span>
<span class="w"> </span><span class="nt">memory</span><span class="p">:</span><span class="w"> </span><span class="s">"10M"</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">4533</span>
<span class="w"> </span><span class="nt">volumeMounts</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome-data-volume</span>
<span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/data</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome-music-volume</span>
<span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/music</span>
<span class="w"> </span><span class="nt">env</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ND_SCANSCHEDULE</span>
<span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1h</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ND_LOGLEVEL</span>
<span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">info</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ND_SESSIONTIMEOUT</span>
<span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">24h</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ND_BASEURL</span>
<span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/navidrome</span>
<span class="w"> </span><span class="nt">livenessProbe</span><span class="p">:</span>
<span class="w"> </span><span class="nt">httpGet</span><span class="p">:</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/navidrome/app</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">4533</span>
<span class="w"> </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">30</span>
<span class="w"> </span><span class="nt">periodSeconds</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3</span>
<span class="w"> </span><span class="nt">timeoutSeconds</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">90</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome-data-volume</span>
<span class="w"> </span><span class="nt">hostPath</span><span class="p">:</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/navidrome</span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Directory</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome-music-volume</span>
<span class="w"> </span><span class="nt">hostPath</span><span class="p">:</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/media/music</span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Directory</span>
<span class="nn">---</span>
<span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span>
<span class="nt">metadata</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">4533</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">web</span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ClusterIP</span>
</pre></div></td></tr></table></div>
<p>At the moment, this deployment is still tied to the machine with my music
on it since it relies on <tt class="docutils literal">hostPath</tt> volumes, but I’m planning to move that to
ceph in the future.</p>
<p>I then expose this service on <em>/navidrome</em> on my private domain (here replaced
with <em>example.com</em>) using an Ingress:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span></pre></div></td><td class="code"><div><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">networking.k8s.io/v1</span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Ingress</span>
<span class="nt">metadata</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w"> </span><span class="nt">ingressClassName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span>
<span class="w"> </span><span class="nt">rules</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">example.com</span>
<span class="w"> </span><span class="nt">http</span><span class="p">:</span>
<span class="w"> </span><span class="nt">paths</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">backend</span><span class="p">:</span>
<span class="w"> </span><span class="nt">service</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">navidrome</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">web</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/navidrome(/|$)(.*)</span>
<span class="w"> </span><span class="nt">pathType</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Prefix</span>
</pre></div></td></tr></table></div>
</div>
<div class="section" id="client">
<h2>Client</h2>
<p>On the desktop, I usually just use navidrome’s web interface. Clementine’s support
for subsonic is also okay. sublime-music is meant to be a music player
specifically for Subsonic, but I’ve not really found it stable enough for
day-to-day usage.</p>
<p>There are various Android clients for Subsonic, but I’ve only really considered the
Open Source ones that are hosted on F-Droid. Most of those are
abandoned, but D-Sub works pretty well - as does my preferred option, <a class="reference external" href="https://f-droid.org/en/packages/com.subtracks/">Subtracks</a>.</p>
</div>
Web Hooks for the Janitor2021-09-06T22:00:00+02:002021-09-06T22:00:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-09-06:janitor-webhooks.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>As covered in <a class="reference external" href="https://www.jelmer.uk/fresh-builds.html">my post from last week</a>, the Janitor now regularly tries to
import new upstream git snapshots or upstream releases into packages in <a class="reference external" href="https://wiki.debian.org/DebianUnstable">Sid</a>.</p>
<div class="section" id="moving-parts">
<h2>Moving parts</h2>
<p>There are about 30,000 packages in sid, and it usually takes a couple of weeks
for the janitor to cycle through all of them. Generally speaking, there are up
to three moving targets for each package:</p>
<ul class="simple">
<li>The packaging repository; <a class="reference external" href="https://qa.debian.org/cgi-bin/vcswatch">vcswatch</a> regularly scans this for changes,
and notifies the janitor when a repository has changed. For <a class="reference external" href="https://salsa.debian.org">salsa</a>
repositories it is instantly notified through a web hook</li>
<li>The upstream release tarballs; the <span class="caps">QA</span> watch service regularly polls these,
and the janitor scans for changes in the <span class="caps">UDD</span> tables with watch data (used for
<a class="reference external" href="https://janitor.debian.net/fresh-releases/">fresh-releases</a>)</li>
<li>The upstream repository; there is no service in Debian that watches this at
the moment (used for
<a class="reference external" href="https://janitor.debian.net/fresh-snapshots/">fresh-snapshots</a>)</li>
</ul>
<p>When the janitor notices that one of these three targets has changed, it
prioritizes processing of a package - this means that a push to a packaging
repository on salsa usually leads to a build being kicked off within 10
minutes. New upstream releases are usually noticed by <span class="caps">QA</span> watch within a day or
so and then lead to a build. Now commits in upstream repositories don’t get
noticed today.</p>
<p>Note that there are no guarantees; the scheduler tries to be clever and not
e.g. rebuild the same package over and over again if it’s constantly changing
and takes a long time to build.</p>
<p>Packages without priority are processed with a scoring system that takes into
account perceived value (based on e.g. popcon), cost (based on wall-time
duration of previous builds) and likelihood of success (whether recent builds
were successful, and how frequently the repositories involved change).</p>
</div>
<div class="section" id="webhooks-for-upstream-repositories">
<h2>webhooks for upstream repositories</h2>
<p>At the moment there is no service in Debian (yet - perhaps this is something
that vcswatch or a sibling service could also do?) that scans upstream
repositories for changes.</p>
<p>However, if you maintain an upstream package, you can use a webhook to notify
the janitor that commits have been made to your repository, and it will create
a new package in fresh-snapshots. Webhooks from the
following hosting site software are currently supported:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/">GitHub</a></li>
<li><a class="reference external" href="https://gitlab.com/">Gitlab</a></li>
<li><a class="reference external" href="https://launchpad.net/">Launchpad</a></li>
<li><a class="reference external" href="https://gitea.io/">Gitea</a></li>
<li><a class="reference external" href="https://gogs.io/">Gogs</a></li>
</ul>
<p>You can simply use the <span class="caps">URL</span> <a class="reference external" href="https://janitor.debian.net/">https://janitor.debian.net/</a> as the target for hooks. There is no need to specify a secret, and the hook can either use a <span class="caps">JSON</span> or form encoding payload.</p>
<p>The endpoint should tell you whether it understood a webhook request, and
whether it took any action. It’s fine to submit webhooks for repositories that
the janitor does not (yet) know about.</p>
<div class="section" id="github-1">
<h3>GitHub</h3>
<p>For GitHub, you can do so in the <tt class="docutils literal">Webhooks</tt> section of the <tt class="docutils literal">Settings</tt> tab. Fill the form as shown below and click on <tt class="docutils literal">Add webhook</tt>:</p>
<img alt="" src="/images/github-webhook.png" />
</div>
<div class="section" id="gitlab-1">
<h3>GitLab</h3>
<p>On GitLab instances, you can find the <tt class="docutils literal">Webhooks</tt> tab under the <tt class="docutils literal">Settings</tt> menu for each repository (under the gear symbol). Fill the form in as shown below and click <tt class="docutils literal">Add Webhook</tt>:</p>
<img alt="" src="/images/gitlab-webhook.png" />
</div>
<div class="section" id="launchpad-1">
<h3>Launchpad</h3>
<p>For Launchpad, go to the repository (for Git) web view and click <tt class="docutils literal">Manage Webhooks</tt>. From there, you can add a new webhook; fill the form in as shown below and click <tt class="docutils literal">Add Webhook</tt>:</p>
<img alt="" src="/images/launchpad-webhook-git.png" />
</div>
</div>
Thousands of Debian packages updated from their upstream Git repository2021-08-25T20:00:00+02:002021-08-25T20:00:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-08-25:fresh-builds.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>Linux distributions like Debian fulfill an important function in the <span class="caps">FOSS</span> ecosystem - they are system integrators that take existing free and open source software projects and adapt them where necessary to work well together. They also make it possible for users to install more software in an easy and consistent way and with some degree of quality control and review.</p>
<p>One of the consequences of this model is that the distribution package often lags behind upstream releases. This is especially true for distributions that have tighter integration and standardization (such as Debian), and often new upstream code is only imported irregularly because it is a manual process - both updating the package, but also making sure that it still works together well with the rest of the system.</p>
<p>The process of importing a new upstream used to be (well, back when I started working on
Debian packages) fairly manual and something like this:</p>
<ul class="simple">
<li>Go to the upstream’s homepage, find the tarball and signature and verify the tarball</li>
<li>Make modifications so the tarball matches Debian’s format</li>
<li>Diff the original and new upstream tarballs and figure out whether changes
are reasonable and which require packaging changes</li>
<li>Update the packaging, changelog, build and manually test the package</li>
<li>Upload</li>
</ul>
<div class="section" id="ecosystem-improvements">
<h2>Ecosystem Improvements</h2>
<p>However, there have been developments over the last decade that make it easier to import new upstream releases into Debian packages.</p>
<div class="section" id="uscan-and-debian-qa-watch">
<h3>Uscan and debian <span class="caps">QA</span> watch</h3>
<p><a class="reference external" href="https://manpages.debian.org/buster/devscripts/uscan.1.en.html">Uscan</a> and
<a class="reference external" href="https://wiki.debian.org/debian/watch">debian/watch</a> have been around for a
while and make it possible to find upstream tarballs.</p>
<p>A debian watch file usually looks something like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="go">version=4</span>
<span class="go">http://somesite.com/dir/filenamewithversion.tar.gz</span>
</pre></div></td></tr></table></div>
<p>The <a class="reference external" href="https://qa.debian.org/cgi-bin/watch"><span class="caps">QA</span> watch service</a> regularly polls
all watch locations in the archive and makes the information available, so it’s
possible to know which packages have changed without downloading each one of them.</p>
</div>
<div class="section" id="git">
<h3>Git</h3>
<p>Git is fairly ubiquitous nowadays, and most upstream projects and packages in Debian use it. There are still exceptions that do not use any version control system or that use a different control system, but they are becoming increasingly rare. <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a></p>
<a class="reference external image-reference" href="https://trends.debian.net/#version-control-system"><img alt="" src="https://trends.debian.net/vcs_testing-stacked.png" /></a>
</div>
<div class="section" id="debian-upstream-metadata">
<h3>debian/upstream/metadata</h3>
<p><a class="reference external" href="https://dep-team.pages.debian.net/deps/dep12/"><span class="caps">DEP</span>-12</a> specifies a file format with metadata about the upstream project that a package was based on. In particular relevant for our case is the fact it has fields for the location of the upstream version control location.</p>
<p>debian/upstream/metadata files look something like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="nn">---</span>
<span class="nt">Repository</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://www.dulwich.io/code/dulwich/</span>
<span class="nt">Repository-Browse</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://www.dulwich.io/code/dulwich/</span>
</pre></div></td></tr></table></div>
<p>While <span class="caps">DEP</span>-12 is still a draft, it has already been widely adopted - there are <a class="reference external" href="https://codesearch.debian.net/search?q=path%3Adebian%2Fupstream%2Fmetadata+Repository%3A&literal=1">about 10000 packages</a> in Debian that ship a debian/upstream/metadata file with Repository information.</p>
</div>
<div class="section" id="autopkgtest">
<h3>Autopkgtest</h3>
<p>The <a class="reference external" href="https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst">Autopkgtest</a>
standard and associated tooling provide a way to run a defined set of tests
against an installed package. This makes it possible to verify that a package
is working correctly as part of the system as a whole. <a class="reference external" href="https://ci.debian.net">ci.debian.net</a> regularly runs these tests against Debian packages to
detect regressions.</p>
</div>
<div class="section" id="vcs-git-headers">
<h3>Vcs-Git headers</h3>
<p>The Vcs-Git headers in debian/control are the equivalent of the Repository field in debian/upstream/metadata, but for the packaging repositories (as opposed to the upstream ones).</p>
<p>They’ve been around for a while and are widely adopted, as can be seen from <a class="reference external" href="https://upsilon.cc/~zack/stuff/vcs-usage/">zack’s stats</a>:</p>
<img alt="" src="/images/fresh-builds-vcs.png" />
<p>The <a class="reference external" href="https://qa.debian.org/cgi-bin/vcswatch">vcswatch service</a> that regularly
polls packaging repositories to see whether they have changed makes it a lot
easier to consume this information in usable way.</p>
</div>
<div class="section" id="debhelper-adoption">
<h3>Debhelper adoption</h3>
<p>Over the last couple of years, Debian has slowly been converging on a single
build tool - debhelper’s dh interface.</p>
<p>Being able to rely on a single build tool makes it easier to write code to
update packaging when upstream changes require it.</p>
<a class="reference external image-reference" href="https://trends.debian.net/#build-systems"><img alt="" src="https://trends.debian.net/build-system_testing-stacked.png" /></a>
</div>
<div class="section" id="debhelper-dwim">
<h3>Debhelper <span class="caps">DWIM</span></h3>
<p>Debhelper (and its helpers) increasingly can figure out how to do the Right
Thing in many cases without being explicitly configured. This makes packaging
less effort, but also means that it’s less likely that importing a new upstream
version will require updates to the packaging.</p>
<p>With all of these improvements in place, it actually becomes feasible in a lot
of situations to update a Debian package to a new upstream version
automatically. Of course, this requires that all of this information is
available, so it won’t work for all packages. In some cases, the packaging for
the older upstream version might not apply to the newer upstream version.</p>
<p>The Janitor has attempted to import a new upstream Git snapshot and a new
upstream release for every package in the archive where a debian/watch file or
debian/upstream/metadata file are present.</p>
<p>These are the steps it uses:</p>
<ul class="simple">
<li><dl class="first docutils">
<dt>Find new upstream version</dt>
<dd><ul class="first last">
<li>If release, use debian/watch - or maybe tagged in upstream repository</li>
<li>If snapshot, use debian/upstream/metadata’s Repository field</li>
<li><em>If neither is available, use guess-upstream-metadata from upstream-ontologist to guess the upstream Repository</em></li>
</ul>
</dd>
</dl>
</li>
<li>Merge upstream version into packaging repository, possibly importing tarballs using pristine-tar</li>
<li>Update the changelog file to mention the new upstream version</li>
<li><dl class="first docutils">
<dt>Run some checks to ensure there are no unintentional changes, e.g.:</dt>
<dd><ul class="first last">
<li><dl class="first docutils">
<dt>Scan diff between old and new for surprising license changes</dt>
<dd><ul class="first last">
<li>Today, abort if there are any - in the future, maybe update debian/copyright</li>
</ul>
</dd>
</dl>
</li>
<li>Check for obvious compatibility breaks - e.g. sonames changing</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>Attempt to update the packaging to reflect upstream changes</dt>
<dd><ul class="first last">
<li>Refresh patches</li>
</ul>
</dd>
</dl>
</li>
<li>Attempt to build the package with deb-fix-build, to deal with any missing dependencies</li>
<li>Run the autopkgtests with deb-fix-build to deal with missing dependencies, and abort if any tests fail</li>
</ul>
</div>
</div>
<div class="section" id="results">
<h2>Results</h2>
<p>When run over all packages in unstable (<a class="reference external" href="https://packages.debian.org/unstable">sid</a>), this process works for a surprising number of them.</p>
<div class="section" id="fresh-releases">
<h3>Fresh Releases</h3>
<p>For fresh-releases (aka imports of upstream releases), processing all packages maintained in Git for which <span class="caps">QA</span> watch reports new releases (about 11,000):</p>
<img alt="" src="/images/fresh-releases.png" />
<p>That means about 2300 packages updated, and about 4000 unchanged.</p>
</div>
<div class="section" id="fresh-snapshots">
<h3>Fresh Snapshots</h3>
<p>For fresh-snapshots (aka imports of latest Git commit from upstream), processing all packages maintained in Git (about 26,000):</p>
<img alt="" src="/images/fresh-snapshots.png" />
<p>Or 5100 packages updated and 2100 for which there was nothing to do, i.e. no upstream commits since the last Debian upload.</p>
<p>As can be seen, this works for a surprising fraction of packages. It’s possible to get the numbers up even higher, by both improving the tooling, the autopkgtests and the metadata that is provided by packages.</p>
</div>
</div>
<div class="section" id="using-these-packages">
<h2>Using these packages</h2>
<p>All the packages that have been built can be accessed from the Janitor <span class="caps">APT</span> repository. More information can be found at <a class="reference external" href="https://janitor.debian.net/fresh">https://janitor.debian.net/fresh</a>, but in short - run:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="go">echo deb "[arch=amd64 signed-by=/usr/share/keyrings/debian-janitor-archive-keyring.gpg]" \</span>
<span class="go"> https://janitor.debian.net/ fresh-snapshots main | sudo tee /etc/apt/sources.list.d/fresh-snapshots.list</span>
<span class="go">echo deb "[arch=amd64 signed-by=/usr/share/keyrings/debian-janitor-archive-keyring.gpg]" \</span>
<span class="go"> https://janitor.debian.net/ fresh-releases main | sudo tee /etc/apt/sources.list.d/fresh-releases.list</span>
<span class="go">sudo curl -o /usr/share/keyrings/debian-janitor-archive-keyring.gpg https://janitor.debian.net/pgp_keys</span>
<span class="go">apt update</span>
</pre></div></td></tr></table></div>
<p>And then you can install packages from the fresh-snapshots (upstream git snapshots) or fresh-releases suites on a case-by-case basis by running something like:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="go">apt install -t fresh-snapshots r-cran-roxygen2</span>
</pre></div></td></tr></table></div>
<p>Most packages are updated based on information provided by vcswatch and qa watch, but it’s also possible for upstream repositories to call a web hook to trigger <a class="reference external" href="https://janitor.debian.net/fresh-builds#requesting-new-packages">a refresh of a package</a>.</p>
<p>These packages were built against unstable, but should in almost all cases also work for testing.</p>
</div>
<div class="section" id="caveats">
<h2>Caveats</h2>
<p>Of course, since these packages are built automatically without human supervision it’s likely that some of them will have bugs in them that would otherwise have been caught by the maintainer.</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>I’m not saying that a monoculture is great here, but it does help distributions.</td></tr>
</tbody>
</table>
</div>
Ognibuild2021-05-14T20:00:00+02:002021-05-14T20:00:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-05-14:ognibuild.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>The <span class="caps">FOSS</span> world uses a wide variety of different build tools; given a git
repository or tarball, it can be hard to figure out how to build and install a
piece of software.</p>
<p>Humans will generally know what build tool a project is using when they check
out a project from git, or they can read the <span class="caps">README</span>. And even then, the answer
may not always be straightforward to everybody. For automation, there
is no obvious place to figure out how to build or install a project.</p>
<div class="section" id="debian">
<h2>Debian</h2>
<p>For Debian packages, Debian maintainers generally will have determined
that the appropriate tools to invoke are, and added appropriate invocations to
<em>debian/rules</em>. This is really nice when rebuilding all of Debian - one can just
invoke <em>debian/rules</em> - a consistent interface - and it will in turn invoke the
right tools to build the package, meeting <a class="reference external" href="https://www.debian.org/doc/debian-policy/">a long list of requirements</a>.</p>
<p>With newer versions of <a class="reference external" href="https://salsa.debian.org/debian/debhelper">debhelper</a>
and most common build systems, debhelper can figure a lot of this out
automatically - the maintainer just has to add the
appropriate build and run time dependencies.</p>
<p>However, debhelper needs to be consistent in its behaviour per compat level -
otherwise builds might start failing with different versions of debhelper, when
the autodetection logic is changed. debhelper can also only do the right thing
if all the necessary dependencies are present. debhelper also only functions
in the context of a Debian package.</p>
</div>
<div class="section" id="ognibuild-1">
<h2>Ognibuild</h2>
<p><a class="reference external" href="https://github.com/jelmer/ognibuild">Ognibuild</a> is a new tool that figures
out the build system in use by an upstream project, as well as the other
dependencies it needs. This information can then be used to invoke said build
system, or to e.g. add missing build dependencies to a Debian package.</p>
<p>Ognibuild uses a variety of techniques to work out what the dependencies for an
upstream package are:</p>
<ul class="simple">
<li>Extracting dependencies and other requirements declared in build system
metadata (e.g. setup.py)</li>
<li>Attempting builds and parsing build logs for missing dependencies (repeating
until the build succeeds), calling out to <a class="reference external" href="https://github.com/jelmer/buildlog-consultant">buildlog-consultant</a></li>
</ul>
<p>Once it is determined which dependencies are missing, they can be resolved in a
variety of ways. Apt can be invoked to install missing dependencies on Debian
systems (optionally in a chroot) or ecosystem-specific tools can be used to do
so (e.g. <a class="reference external" href="https://pypi.org/">pypi</a> or <a class="reference external" href="https://www.cpan.org/">cpan</a>).
Instead of installing packages, the tool can also simply
inform the user about the missing packages and commands to install them, or
update a Debian package appropriately (this is what <a class="reference external" href="https://jelmer.uk/deb-fix-build.html">deb-fix-build</a> does).</p>
<p>The target audience of ognibuild are people who need to (possibly from
automation) build a variety of projects from different ecosystems or users who
are looking to just install a project from source. Developers who are just
hacking on e.g. a Python project are better off directly invoking the
ecosystem-native tools rather than a wrapper like ognibuild.</p>
</div>
<div class="section" id="supported-ecosystems">
<h2>Supported ecosystems</h2>
<p>(Partially) supported ecosystems currently include:</p>
<ul class="simple">
<li>Combinations of make and autoconf, automake or CMake</li>
<li>Python, including fetching packages from pypi</li>
<li>Perl, including fetching packages from cpan</li>
<li>Haskell, including fetching from hackage</li>
<li>Ninja/Meson</li>
<li>Maven</li>
<li>Rust, including fetching packages from crates.io</li>
<li><span class="caps">PHP</span> Pear</li>
<li>R, including fetching packages from <span class="caps">CRAN</span> and Bioconductor</li>
</ul>
<p>For a full list, see <a class="reference external" href="https://github.com/jelmer/ognibuild/blob/main/README.md">the <span class="caps">README</span></a>.</p>
</div>
<div class="section" id="usage">
<h2>Usage</h2>
<p>Ognibuild provides a couple of top-level subcommands that will seem familiar to anybody who has used a couple of other build systems:</p>
<ul class="simple">
<li><em>ogni clean</em> - remove build artifacts</li>
<li><em>ogni dist</em> - create a dist tarball</li>
<li><em>ogni build</em> - build the project in the current directory</li>
<li><em>ogni test</em> - run the test suite</li>
<li><em>ogni install</em> - install the project somewhere</li>
<li><em>ogni info</em> - display project information including discovered build system and dependencies</li>
<li><em>ogni exec</em> - run an arbitrary command but attempt to resolve issues like missing dependencies</li>
</ul>
<p>These tools all take a couple of common options:</p>
<div class="section" id="resolve-apt-auto-native">
<h3>—resolve=apt|auto|native</h3>
<p>Specifies how to resolve any missing dependencies:</p>
<ul class="simple">
<li><strong>apt</strong>: install the appropriate dependency using apt</li>
<li><strong>native</strong>: install dependencies using native tools like pip or cpan</li>
<li><strong>auto</strong>: invoke either apt or native package install, depending on whether
the current user is allowed to invoke apt</li>
</ul>
</div>
<div class="section" id="schroot-name">
<h3>—schroot=name</h3>
<p>Run inside of a schroot.</p>
</div>
<div class="section" id="explain">
<h3>—explain</h3>
<p>do not make any changes but tell the user which native on apt packages they could install.</p>
<p>There are also subcommand-specific options, e.g. to install to a specific directory on restrict which tests are run.</p>
</div>
</div>
<div class="section" id="examples">
<h2>Examples</h2>
<div class="section" id="creating-a-dist-tarball">
<h3>Creating a dist tarball</h3>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span>
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="gp">% </span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/dulwich/dulwich
<span class="gp">% </span><span class="nb">cd</span><span class="w"> </span>dulwich
<span class="gp">% </span>ogni<span class="w"> </span>--schroot<span class="o">=</span>unstable-amd64-sbuild<span class="w"> </span>dist
<span class="go">…</span>
<span class="go">Writing dulwich-0.20.21/setup.cfg</span>
<span class="go">creating dist</span>
<span class="go">Creating tar archive</span>
<span class="go">removing 'dulwich-0.20.21' (and everything under it)</span>
<span class="go">Found new tarball dulwich-0.20.21.tar.gz in /var/run/schroot/mount/unstable-amd64-sbuild-974d32d7-6f10-4e77-8622-b6a091857e85/build/tmpucazj7j7/package/dist.</span>
</pre></div></td></tr></table></div>
</div>
<div class="section" id="installing-ldb-from-source-resolving-dependencies-using-apt">
<h3>Installing ldb from source, resolving dependencies using apt</h3>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span>
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="gp">% </span>wget<span class="w"> </span>https://download.samba.org/pub/ldb/ldb-2.3.0.tar.gz
<span class="gp">% </span>tar<span class="w"> </span>xvfz<span class="w"> </span>ldb-2.3.0.tar.gz
<span class="gp">% </span><span class="nb">cd</span><span class="w"> </span>ldb-2.3.0
<span class="gp">% </span>ogni<span class="w"> </span>install<span class="w"> </span>--prefix<span class="o">=</span>/tmp/ldb
<span class="go">…</span>
<span class="go">+ install /tmp/ldb/include/ldb.h (from include/ldb.h)</span>
<span class="go">…</span>
<span class="go">Waf: Leaving directory `/tmp/ldb-2.3.0/bin/default'</span>
<span class="go">'install' finished successfully (11.395s)</span>
</pre></div></td></tr></table></div>
</div>
<div class="section" id="running-all-tests-from-xml-libxml-lazybuilder">
<h3>Running all tests from <span class="caps">XML</span>::LibXML::LazyBuilder</h3>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="gp">% </span>wget<span class="w"> </span><span class="sb">``</span>https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz<span class="sb">`</span>_<span class="w"> </span><https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz><span class="sb">`</span>_
<span class="gp">% </span>tar<span class="w"> </span>xvfz<span class="w"> </span>XML-LibXML-LazyBuilder-0.08.tar.gz
<span class="go">Cd XML-LibXML-LazyBuilder-0.08</span>
<span class="gp">% </span>ogni<span class="w"> </span><span class="nb">test</span>
<span class="go">…</span>
</pre></div></td></tr></table></div>
</div>
<div class="section" id="current-status">
<h3>Current Status</h3>
<p>ognibuild is still in its early stages, but works well enough that it can
detect and invoke the build system for most of the upstream projects packaged
in Debian. If there are buildsystems that it currently lacks support for or
other issues, then I’d welcome any <a class="reference external" href="https://github.com/jelmer/ognibuild/issues/new/choose">bug reports</a>.</p>
</div>
</div>
The upstream ontologist2021-04-12T00:40:00+02:002021-04-12T00:40:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-04-12:upstream-ontologist.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>The <a class="reference external" href="https://github.com/jelmer/upstream-ontologist">upstream ontologist</a> is a project that extracts metadata about upstream projects in a consistent format. It does this with a combination of heuristics and reading ecosystem-specific metadata files, such as Python’s <em>setup.py</em>, rust’s <em>Cargo.toml</em> as well as e.g. scanning <span class="caps">README</span> files.</p>
<div class="section" id="supported-data-sources">
<h2>Supported Data Sources</h2>
<p>It will extract information from a wide variety of sources, including:</p>
<ul class="simple">
<li>Python package metadata (<span class="caps">PKG</span>-<span class="caps">INFO</span>, setup.py, setup.cfg, pyproject.toml)</li>
<li><a class="reference external" href="https://docs.npmjs.com/cli/v7/configuring-npm/package-json">package.json</a></li>
<li><a class="reference external" href="https://getcomposer.org/doc/04-schema.md">composer.json</a></li>
<li><a class="reference external" href="https://pear.php.net/manual/en/guide.developers.package2.dependencies.php">package.xml</a></li>
<li>Perl package metadata (dist.ini, <span class="caps">META</span>.json, <span class="caps">META</span>.yml, Makefile.<span class="caps">PL</span>)</li>
<li><a class="reference external" href="https://perldoc.perl.org/perlpod">Perl <span class="caps">POD</span> files</a></li>
<li><span class="caps">GNU</span> configure files</li>
<li><a class="reference external" href="https://r-pkgs.org/description.html">R <span class="caps">DESCRIPTION</span> files</a></li>
<li><a class="reference external" href="https://doc.rust-lang.org/cargo/reference/manifest.html">Rust Cargo.toml</a></li>
<li><a class="reference external" href="https://maven.apache.org/pom.html">maven pom.xml</a></li>
<li><a class="reference external" href="https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html">metainfo.xml</a></li>
<li><a class="reference external" href="https://git-scm.com/docs/git-config">.git/config</a></li>
<li><span class="caps">SECURITY</span>.md</li>
<li><a class="reference external" href="https://github.com/ewilderj/doap"><span class="caps">DOAP</span></a></li>
<li><a class="reference external" href="https://cabal.readthedocs.io/en/3.4/cabal-package.html">Haskell cabal files</a></li>
<li><a class="reference external" href="https://guides.rubygems.org/specification-reference/">Ruby gemspec files</a></li>
<li><a class="reference external" href="https://golang.org/doc/modules/gomod-ref">go.mod</a></li>
<li><span class="caps">README</span>{,.rst,.md} files</li>
<li>Debian packaging metadata (debian/watch, debian/control, debian/rules, debian/get-orig-source.sh, debian/copyright, debian/patches)</li>
</ul>
</div>
<div class="section" id="supported-fields">
<h2>Supported Fields</h2>
<p>Fields that it currently provides include:</p>
<ul class="simple">
<li><strong>Homepage</strong>: homepage <span class="caps">URL</span></li>
<li><strong>Name</strong>: name of the upstream project</li>
<li><strong>Contact</strong>: contact address of some sort of the upstream (e-mail, mailing list <span class="caps">URL</span>)</li>
<li><strong>Repository</strong>: <span class="caps">VCS</span> <span class="caps">URL</span></li>
<li><strong>Repository-Browse</strong>: Web <span class="caps">URL</span> for viewing the <span class="caps">VCS</span></li>
<li><strong>Bug-Database</strong>: Bug database <span class="caps">URL</span> (for web viewing, generally)</li>
<li><strong>Bug-Submit</strong>: <span class="caps">URL</span> to use to submit new bugs (either on the web or an e-mail address)</li>
<li><strong>Screenshots</strong>: List of URLs with screenshots</li>
<li><strong>Archive</strong>: Archive used - e.g. SourceForge</li>
<li><strong>Security-Contact</strong>: e-mail or <span class="caps">URL</span> with instructions for reporting security issues</li>
<li><strong>Documentation</strong>: Link to documentation on the web:</li>
<li><strong>Wiki</strong>: Wiki <span class="caps">URL</span></li>
<li><strong>Summary</strong>: one-line description of the project</li>
<li><strong>Description</strong>: longer description of the project</li>
<li><strong>License</strong>: Single line license description (e.g. “<span class="caps">GPL</span> 2.0”) as declared in the metadata<a class="footnote-reference" href="#f1" id="footnote-reference-1">[1]</a></li>
<li><strong>Copyright</strong>: List of copyright holders</li>
<li><strong>Version</strong>: Current upstream version</li>
<li><strong>Security-<span class="caps">MD</span></strong>: <span class="caps">URL</span> to markdown file with security policy</li>
</ul>
<p>All data fields have a “certainty” associated with them (“certain”, “confident”, “likely” or “possible”), which gets set depending on how the data was derived or where it was found. If multiple possible values were found for a specific field, then the value with the highest certainty is taken.</p>
</div>
<div class="section" id="interface">
<h2>Interface</h2>
<p>The ontologist provides a high-level Python <span class="caps">API</span> as well as two command-line tools that can write output in two different formats:</p>
<ul class="simple">
<li><a class="reference external" href="https://manpages.debian.org/testing/python3-upstream-ontologist/guess-upstream-metadata.1.en.html">guess-upstream-metadata</a> writes <a class="reference external" href="https://wiki.debian.org/UpstreamMetadata"><span class="caps">DEP</span>-12-like</a> <span class="caps">YAML</span> output</li>
<li><a class="reference external" href="https://manpages.debian.org/testing/python3-upstream-ontologist/autodoap.1.en.html">autodoap</a> writes <a class="reference external" href="https://github.com/ewilderj/doap"><span class="caps">DOAP</span></a> files</li>
</ul>
<p>For example, running <em>guess-upstream-metadata</em> on <a class="reference external" href="https://github.com/dulwich/dulwich">dulwich</a>:</p>
<div class="highlight"><pre><span></span><span class="gp"> % </span>guess-upstream-metadata
<span class="go"> <string>:2: (INFO/1) Duplicate implicit target name: "contributing".</span>
<span class="go"> Name: dulwich</span>
<span class="go"> Repository: https://www.dulwich.io/code/</span>
<span class="go"> X-Security-MD: https://github.com/dulwich/dulwich/tree/HEAD/SECURITY.md</span>
<span class="go"> X-Version: 0.20.21</span>
<span class="go"> Bug-Database: https://github.com/dulwich/dulwich/issues</span>
<span class="go"> X-Summary: Python Git Library</span>
<span class="go"> X-Description: |</span>
<span class="go"> This is the Dulwich project.</span>
<span class="go"> It aims to provide an interface to git repos (both local and remote) that</span>
<span class="go"> doesn't call out to git directly but instead uses pure Python.</span>
<span class="go"> X-License: Apache License, version 2 or GNU General Public License, version 2 or later.</span>
<span class="go"> Bug-Submit: https://github.com/dulwich/dulwich/issues/new</span>
</pre></div>
</div>
<div class="section" id="lintian-brush-1">
<h2>Lintian-Brush</h2>
<p><a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> can update <a class="reference external" href="https://dep-team.pages.debian.net/deps/dep12/"><span class="caps">DEP</span>-12-style debian/upstream/metadata files</a> that hold information about the upstream project that is packaged as well as the <em>Homepage</em> in the debian/control file based on information provided by the upstream ontologist. By default, it only imports data with the highest certainty - you can override this by specifying the —uncertain command-line flag.</p>
<table class="docutils footnote" frame="void" id="f1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Obviously this won’t be able to describe the full licensing situation for many projects. Projects like <a class="reference external" href="https://github.com/nexB/scancode-toolkit">scancode-toolkit</a> are more appropriate for that.</td></tr>
</tbody>
</table>
</div>
Automatic Fixing of Debian Build Dependencies2021-04-06T23:46:16+02:002021-04-06T23:46:16+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-04-06:deb-fix-build.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>In my <a class="reference external" href="/buildlog-consultant.html">last blogpost</a>, I introduced the <a class="reference external" href="https://github.com/jelmer/buildlog-consultant">buildlog
consultant</a> - a tool that can
identify many reasons why a Debian build failed.</p>
<p>For example, here’s a fragment of a build log where the Build-Depends lack
<em>python3-setuptools</em>:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">849</span>
<span class="normal">850</span>
<span class="normal">851</span>
<span class="normal">852</span>
<span class="normal">853</span>
<span class="normal">854</span>
<span class="normal">855</span>
<span class="normal">856</span>
<span class="normal">857</span>
<span class="normal">858</span></pre></div></td><td class="code"><div><pre><span></span><span class="go"> dpkg-buildpackage: info: host architecture amd64</span>
<span class="go"> fakeroot debian/rules clean</span>
<span class="go"> dh clean --with python3,sphinxdoc --buildsystem=pybuild</span>
<span class="go"> dh_auto_clean -O--buildsystem=pybuild</span>
<span class="go"> I: pybuild base:232: python3.9 setup.py clean</span>
<span class="go"> Traceback (most recent call last):</span>
<span class="go"> File "/<<PKGBUILDDIR>>/setup.py", line 2, in <module></span>
<span class="go"> from setuptools import setup</span>
<span class="hll"><span class="go"> ModuleNotFoundError: No module named 'setuptools'</span>
</span><span class="go"> E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean</span>
</pre></div></td></tr></table></div>
<p>The buildlog consultant can identify the line in bold as being key, and interprets it:</p>
<div class="highlight"><pre><span></span><span class="gp"> % </span>analyse-sbuild-log<span class="w"> </span>--json<span class="w"> </span>~/build.log
<span class="go"> {</span>
<span class="go"> "stage": "build",</span>
<span class="go"> "section": "Build",</span>
<span class="go"> "lineno": 857,</span>
<span class="go"> "kind": "missing-python-module",</span>
<span class="go"> "details": {"module": "setuptools", "python_version": 3, "minimum_version": null}</span>
<span class="go"> }</span>
</pre></div>
<div class="section" id="automatically-acting-on-buildlog-problems">
<h2>Automatically acting on buildlog problems</h2>
<p>A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends.</p>
<p>Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to <em>Build-Depends</em>. In the example given above, we can use apt-file to look for the package that contains the path <em>/usr/lib/python3/dist-packages/setuptools/__init__.py</em> - and voila, we find python3-setuptools:</p>
<div class="highlight"><pre><span></span><span class="gp"> % </span>apt-file<span class="w"> </span>search<span class="w"> </span>/usr/lib/python3/dist-packages/setuptools/__init__.py
<span class="go"> python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py</span>
</pre></div>
<p>The <em>deb-fix-build</em> command automates these steps:</p>
<ol class="arabic simple">
<li>It builds the package using sbuild; if the package successfully builds then it just exits successfully</li>
<li>It tries to identify the problem by looking through the build log; if it can’t or if it’s a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code</li>
<li>It tries to find a dependency that can address the problem</li>
<li>It updates <em>Build-Depends</em> in <em>debian/control</em> or <em>Depends</em> in <em>debian/tests/control</em></li>
<li>Go to step 1</li>
</ol>
<p>This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again.</p>
<p>For example, when I ran deb-fix-build while packaging <a class="reference external" href="https://github.com/nexb/saneyaml">saneyaml</a>, the output looks something like this:</p>
<div class="highlight"><pre><span></span><span class="gp"> % </span>deb-fix-build
<span class="go"> Using output directory /tmp/tmpyz0nkgqq</span>
<span class="go"> Using sbuild chroot unstable-amd64-sbuild</span>
<span class="go"> Using fixers: …</span>
<span class="go"> Building debian packages, running 'sbuild --no-clean-source -A -s -v'.</span>
<span class="go"> Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4')</span>
<span class="go"> Using apt-file to search apt contents</span>
<span class="go"> Adding build dependency: python3-setuptools-scm (>= 4)</span>
<span class="go"> Building debian packages, running 'sbuild --no-clean-source -A -s -v'.</span>
<span class="go"> Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None)</span>
<span class="go"> Adding build dependency: python3-toml</span>
<span class="go"> Building debian packages, running 'sbuild --no-clean-source -A -s -v'.</span>
<span class="go"> Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’].</span>
</pre></div>
<p>And in our Git repository, we see these changes as well:</p>
<div class="highlight"><pre><span></span><span class="gp">% </span>git<span class="w"> </span>log<span class="w"> </span>-p
<span class="go"> commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master)</span>
<span class="go"> Author: Jelmer Vernooij <jelmer@jelmer.uk></span>
<span class="go"> Date: Sun Apr 4 02:35:56 2021 +0100</span>
<span class="go"> Add missing build dependency on python3-toml.</span>
<span class="go"> diff --git a/debian/control b/debian/control</span>
<span class="go"> index 5b854dc..3b27b73 100644</span>
<span class="go"> --- a/debian/control</span>
<span class="go"> +++ b/debian/control</span>
<span class="go"> @@ -1,6 +1,6 @@</span>
<span class="go"> Rules-Requires-Root: no</span>
<span class="go"> Standards-Version: 4.5.1</span>
<span class="go"> -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)</span>
<span class="go"> +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml</span>
<span class="go"> Testsuite: autopkgtest-pkg-python</span>
<span class="go"> Source: python-saneyaml</span>
<span class="go"> Priority: optional</span>
<span class="go"> commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1</span>
<span class="go"> Author: Jelmer Vernooij <jelmer@jelmer.uk></span>
<span class="go"> Date: Sun Apr 4 02:35:34 2021 +0100</span>
<span class="go"> Add missing build dependency on python3-setuptools-scm (>= 4).</span>
<span class="go"> diff --git a/debian/control b/debian/control</span>
<span class="go"> index a476cc2..5b854dc 100644</span>
<span class="go"> --- a/debian/control</span>
<span class="go"> +++ b/debian/control</span>
<span class="go"> @@ -1,6 +1,6 @@</span>
<span class="go"> Rules-Requires-Root: no</span>
<span class="go"> Standards-Version: 4.5.1</span>
<span class="go"> -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel</span>
<span class="go"> +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)</span>
<span class="go"> Testsuite: autopkgtest-pkg-python</span>
<span class="go"> Source: python-saneyaml</span>
<span class="go"> Priority: optional</span>
</pre></div>
</div>
<div class="section" id="using-deb-fix-build">
<h2>Using deb-fix-build</h2>
<p>You can run <em>deb-fix-build</em> by installing the <a class="reference external" href="https://packages.debian.org/ognibuild">ognibuild package</a> from unstable. The only requirements
for using it are that:</p>
<ul class="simple">
<li>The package is maintained in Git</li>
<li>A sbuild schroot is available for use</li>
</ul>
</div>
<div class="section" id="caveats">
<h2>Caveats</h2>
<p>deb-fix-build is fairly easy to understand, and if it doesn’t work then you’re no worse off than you were without it - you’ll have to add your own Build-Depends.</p>
<p>That said, there are a couple of things to keep in mind:</p>
<ul class="simple">
<li>At the moment, it doesn’t distinguish between general, Arch or Indep Build-Depends.</li>
<li>It can only add dependencies for things that are actually in the archive</li>
<li>Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn’t always get it right</li>
</ul>
</div>
The Buildlog Consultant2021-04-05T16:00:00+02:002021-04-05T16:00:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2021-04-05:buildlog-consultant.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<div class="section" id="reading-build-logs">
<h2>Reading build logs</h2>
<p>Build logs for Debian packages can be quite long and difficult
for a human to read. Anybody who has looked at these logs
trying to figure out why a build failed will have spent
time scrolling through them and skimming for certain phrases
(lines starting with “error:” for example). In many cases,
you can spot the problem in the last 10 or 20 lines of
output – but it’s also quite common that the error is somewhere
at the beginning of many pages of error output.</p>
</div>
<div class="section" id="the-buildlog-consultant-1">
<h2>The buildlog consultant</h2>
<p>The <a class="reference external" href="https://github.com/jelmer/buildlog-consultant">buildlog consultant project</a> attempts to aid in this
process by parsing sbuild and non-Debian (e.g. the output
of “make”) build logs and trying to identify the key line
that explains why a build failed. It can then either display
this specific line, or a fragment of the log around surrounding
the key line.</p>
</div>
<div class="section" id="classification">
<h2>Classification</h2>
<p>In addition to finding the key line explaining the failure,
it can also classify and parse the error in many cases
and return a result code and some metadata.</p>
<p>For example, in a <a class="reference external" href="https://janitor.debian.net/cupboard/pkg/gnss-sdr-upstream/5980a87d-b1a4-4d1d-9660-acd385ee5186/">failed build</a>
of <a class="reference external" href="https://gnss-sdr.org/">gnss-sdr</a> that has produced <a class="reference external" href="https://janitor.debian.net/cupboard/pkg/gnss-sdr-upstream/5980a87d-b1a4-4d1d-9660-acd385ee5186/build.log">2119 lines of output</a>,
the reason for the failure is that
log4cpp is missing – which is on line 641:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">634</span>
<span class="normal">635</span>
<span class="normal">636</span>
<span class="normal">637</span>
<span class="normal">638</span>
<span class="normal">639</span>
<span class="normal">640</span>
<span class="normal">641</span>
<span class="normal">642</span>
<span class="normal">643</span>
<span class="normal">644</span>
<span class="normal">645</span>
<span class="normal">646</span>
<span class="normal">647</span></pre></div></td><td class="code"><div><pre><span></span><span class="go"> -- Required GNU Radio Component: ANALOG missing!</span>
<span class="go"> -- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND)</span>
<span class="go"> -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)</span>
<span class="go"> -- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS</span>
<span class="go"> LOG4CPP_LIBRARIES)</span>
<span class="go"> CMake Error at CMakeLists.txt:593 (message):</span>
<span class="go"> *** Log4cpp is required to build gnss-sdr</span>
<span class="go"> -- Configuring incomplete, errors occurred!</span>
<span class="go"> See also "/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/CMakeFiles/</span>
<span class="go"> CMakeOutput.log".</span>
<span class="go"> See also "/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/CMakeFiles/</span>
<span class="go"> CMakeError.log".</span>
</pre></div></td></tr></table></div>
<p>In this case, the buildlog consultant can both figure out
line was problematic and what the problem was:</p>
<div class="highlight"><pre><span></span><span class="gp"> % </span>analyse-sbuild-log<span class="w"> </span>build.log
<span class="go"> Failed stage: build</span>
<span class="go"> Section: build</span>
<span class="go"> Failed line: 641:</span>
<span class="go"> *** Log4cpp is required to build gnss-sdr</span>
<span class="go"> Error: Missing dependency: Log4cpp</span>
</pre></div>
<p>Or, if you’d like to do something else with the output, use <span class="caps">JSON</span> output:</p>
<div class="highlight"><pre><span></span><span class="gp"> % </span>analyse-sbuild-log<span class="w"> </span>--json<span class="w"> </span>build.log
<span class="go"> {"stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details": {"name": "Log4cpp""}}</span>
</pre></div>
</div>
<div class="section" id="how-it-works">
<h2>How it works</h2>
<p>The consultant does some structured parsing (most notably
it can parse the sections from a sbuild log), but otherwise
is a large set of carefully crafted regular expressions
and heuristics. It doesn’t always find the problem, but
has proven to be fairly accurate. It is constantly improved
as part of the <a class="reference external" href="https://janitor.debian.net/">Debian Janitor project</a>, and
that exposes it to a wide variety of different errors.</p>
<p>You can see the classification and error detection in action
on the <a class="reference external" href="https://janitor.debian.net/cupboard/result-codes/">result codes</a>
page of the Janitor.</p>
</div>
<div class="section" id="using-the-buildlog-consultant">
<h2>Using the buildlog consultant</h2>
<p>You can get the buildlog consultant from either pip or Debian
unstable (package: <a class="reference external" href="https://packages.debian.org/python3-buildlog-consultant">python3-buildlog-consultant</a> ).</p>
<p>The buildlog consultant comes with two scripts – <em>analyse-build-log</em> and <em>analyse-sbuild-log</em>, for analysing
build logs and sbuild logs respectively.</p>
</div>
Debian Janitor: Hosters used by Debian packages2020-10-24T08:00:00+02:002020-10-24T08:00:00+02:00Jelmer Vernooij, Perry Lorriertag:www.jelmer.uk,2020-10-24:janitor-update-8.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>The Janitor knows how to talk to different hosting platforms.
For each hosting platform, it needs to support the platform-
specific <span class="caps">API</span> for creating and managing merge proposals.
For each hoster it also needs to have <a class="reference external" href="https://janitor.debian.net/credentials">credentials</a>.</p>
<p>At the moment, it supports the <a class="reference external" href="https://developer.github.com/v3/">GitHub <span class="caps">API</span></a>,
<a class="reference external" href="https://launchpad.net/+apidoc/devel.html">Launchpad <span class="caps">API</span></a> and <a class="reference external" href="https://docs.gitlab.com/ee/api/">GitLab <span class="caps">API</span></a>. Both GitHub and Launchpad have only a
single instance; the GitLab instances it supports are <a class="reference external" href="https://gitlab.com/">gitlab.com</a> and <a class="reference external" href="https://salsa.debian.org/">salsa.debian.org</a>.</p>
<p>This provides coverage for the vast majority of Debian packages
that can be accessed using Git. More than 75% of all packages
are available on salsa - although in some cases, the Vcs-Git
header has not yet been updated.</p>
<p>Of the other 25%, the majority either does not declare where
it is hosted using a Vcs-* header (10.5%), or have not
yet migrated from alioth to another hosting platform
(9.7%). A further 2.3% are hosted somewhere on
<a class="reference external" href="https://www.github.com/">GitHub</a> (2%),
<a class="reference external" href="https://launchpad.net/">Launchpad</a> (0.18%) or
<a class="reference external" href="https://gitlab.com/">GitLab.com</a> (0.15%), in many cases
in the same repository as the upstream code.</p>
<p>The remaining 1.6% are hosted on many other hosts, primarily
people’s personal servers (which usually don’t have an
<span class="caps">API</span> for creating pull requests).</p>
<img alt="Packages per hoster" src="/images/janitor-packages-per-hoster.png" />
<div class="section" id="outdated-vcs-headers">
<h2>Outdated Vcs-* headers</h2>
<p>It is possible that the 20% of packages that do not have
a Vcs-* header or have a Vcs header that say there on
alioth are actually hosted elsewhere. However, it is hard
to know where they are until a version with an updated
Vcs-Git header is uploaded.</p>
<p>The Janitor primarily relies on
<a class="reference external" href="https://qa.debian.org/cgi-bin/vcswatch">vcswatch</a> to find the correct
locations of repositories. vcswatch looks at Vcs-* headers
but has its own heuristics as well. For about 2,000 packages
(6%) that still have Vcs-* headers that point to alioth,
vcswatch successfully finds their new home on salsa.</p>
</div>
<div class="section" id="merge-proposals-by-hoster">
<h2>Merge Proposals by Hoster</h2>
<p>These proportions are also visible in the number of pull
requests created by the Janitor on various hosters. The
vast majority so far has been created on Salsa.</p>
<table class="docutils" border="1">
<colgroup>
<col width="26%" />
<col width="18%" />
<col width="35%" />
<col width="21%" />
</colgroup>
<thead>
<tr>
<td class="head"><b>Hoster</b></td>
<td class="head"><b>Open</b></td>
<td class="head"><b>Merged <span class="amp">&</span> Applied</b></td>
<td class="head"><b>Closed</b></td>
</tr>
</thead>
<tbody>
<tr><td>github.com</td><td align="right">92</td><td align="right">168</td><td align="right">5</td></tr></tr>
<tr><td>gitlab.com</td><td align="right">12</td><td align="right">3</td><td align="right">0</td></tr>
<tr><td>code.launchpad.net</td><td align="right">24</td><td align="right">51</td><td align="right">1</td></tr>
<tr><td>salsa.debian.org</td><td align="right">1,360</td><td align="right">5,657</td><td align="right">126</td></tr>
</tbody>
</table><img alt="Merge Proposal statistics" src="/images/janitor-merge-proposal-stats.png" />
<p>In this graph, “Open” means that the pull request has been
created but likely nobody has looked at it yet. Merged
means that the pull request has been marked as merged on
the hoster, and applied means that the changes have ended
up in the packaging branch but via a different route (e.g. cherry-picked or
manually applied). Closed means that the pull request was closed without the
changes being incorporated.</p>
<p>Note that this excludes ~5,600 direct pushes, all of which were to salsa-hosted repositories.</p>
<p>See also:</p>
<ul class="simple">
<li>Historical graphs on <a class="reference external" href="http://trends.debian.net">trends.debian.net</a> with number of packages <a class="reference external" href="https://trends.debian.net/#version-control-system">per <span class="caps">VCS</span></a> and <a class="reference external" href="https://trends.debian.net/#vcs-hosting">per hoster</a> (purely based on Vcs-* headers in the archive, with no heuristics applied)</li>
<li>Zack’s table of <a class="reference external" href="https://upsilon.cc/~zack/stuff/vcs-usage/">number of Vcs-* header by system</a> (Git, Svn, etc)</li>
<li>current <a class="reference external" href="https://janitor.debian.net/cupboard/stats">Janitor merge proposal statistics</a></li>
</ul>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
</div>
Debian Janitor: How to Contribute Lintian-Brush Fixers2020-10-15T20:00:00+02:002020-10-15T20:00:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2020-10-15:janitor-update-7.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>lintian-brush can currently fix about 150 different issues that lintian can
report, but that’s still a small fraction of the more than thousand different
types of issue that lintian can detect.</p>
<p>If you’re interested in contributing a fixer script to <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a>, there is now <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush/-/blob/master/doc/fixer-writing-guide.rst">a guide</a>
that describes all steps of the process:</p>
<ol class="arabic simple">
<li>how to identify lintian tags that are good candidates for automated fixing</li>
<li>creating test cases</li>
<li>writing the actual fixer</li>
</ol>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
Debian Janitor: Expanding Into Improving Multi-Arch2020-09-19T08:00:00+02:002020-09-19T08:00:00+02:00Jelmer Vernooij, Perry Lorriertag:www.jelmer.uk,2020-09-19:janitor-update-6.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>As of dpkg 1.16.2 and apt 0.8.13, Debian has full support
for <a class="reference external" href="https://wiki.debian.org/Multiarch">multi-arch</a>. To quote from the multi-arch implementation page:</p>
<blockquote>
Multiarch lets you install library packages from multiple architectures on
the same machine. This is useful in various ways, but the most common is
installing both 64 and 32- bit software on the same machine and having
dependencies correctly resolved automatically. In general you can have
libraries of more than one architecture installed together and applications
from one architecture or another installed as alternatives.</blockquote>
<p>The <a class="reference external" href="https://wiki.ubuntu.com/MultiarchSpec">Multi-Arch specification</a> describes a new Multi-Arch header which can be used
to indicate how to resolve cross-architecture dependencies.</p>
<p>The existing <a class="reference external" href="https://wiki.debian.org/MultiArch/Hints">Debian Multi-Arch hinter</a> is a version of <a class="reference external" href="dedup.debian.net">dedup.debian.net</a> that compares binary
packages between architectures and suggests fixes to resolve
multi-arch problems. It provides hints as to what Multi-
Arch fields can be set, allowing the packages to be safely
installed in a Multi-Arch world. The full list of almost
10,000 hints generated by the hinter is available at <a class="reference external" href="https://dedup.debian.net/static/multiarch-hints.yaml">https://dedup.debian.net/static/multiarch-hints.yaml</a>.</p>
<p>Recent versions of lintian-brush now include a command called <a class="reference external" href="https://manpages.debian.org/testing/lintian-brush/apply-multiarch-hints.1.en.html">apply-multiarch-hints</a>
that downloads and locally caches the hints and can apply
them to a package maintained in Git. For example, to apply
multi-arch hints to <a class="reference external" href="https://salsa.debian.org/js-team/autosize.js">autosize.js</a>:</p>
<div class="highlight"><pre><span></span><span class="gp"> $ </span>debcheckout<span class="w"> </span>autosize.js
<span class="go"> declared git repository at https://salsa.debian.org/js-team/autosize.js.git</span>
<span class="go"> git clone https://salsa.debian.org/js-team/autosize.js.git autosize.js ...</span>
<span class="go"> Cloning into 'autosize.js'...</span>
<span class="go"> [...]</span>
<span class="gp"> $ </span><span class="nb">cd</span><span class="w"> </span>autosize.js
<span class="gp"> $ </span>apply-multiarch-hints
<span class="go"> Downloading new version of multi-arch hints.</span>
<span class="go"> libjs-autosize: Add Multi-Arch: foreign.</span>
<span class="go"> node-autosize: Add Multi-Arch: foreign.</span>
<span class="gp"> $ </span>git<span class="w"> </span>log<span class="w"> </span>-p
<span class="go"> commit 3f8d1db5af4a87e6ebb08f46ddf79f6adf4e95ae (HEAD -> master)</span>
<span class="go"> Author: Jelmer Vernooij <jelmer@debian.org></span>
<span class="go"> Date: Fri Sep 18 23:37:14 2020 +0000</span>
<span class="go"> Apply multi-arch hints.</span>
<span class="go"> + libjs-autosize, node-autosize: Add Multi-Arch: foreign.</span>
<span class="go"> Changes-By: apply-multiarch-hints</span>
<span class="go"> diff --git a/debian/changelog b/debian/changelog</span>
<span class="go"> index e7fa120..09af4a7 100644</span>
<span class="go"> --- a/debian/changelog</span>
<span class="go"> +++ b/debian/changelog</span>
<span class="go"> @@ -1,3 +1,10 @@</span>
<span class="go"> +autosize.js (4.0.2~dfsg1-5) UNRELEASED; urgency=medium</span>
<span class="go"> +</span>
<span class="go"> + * Apply multi-arch hints.</span>
<span class="go"> + + libjs-autosize, node-autosize: Add Multi-Arch: foreign.</span>
<span class="go"> +</span>
<span class="go"> + -- Jelmer Vernooij <jelmer@debian.org> Fri, 18 Sep 2020 23:37:14 -0000</span>
<span class="go"> +</span>
<span class="go"> autosize.js (4.0.2~dfsg1-4) unstable; urgency=medium</span>
<span class="go"> * Team upload</span>
<span class="go"> diff --git a/debian/control b/debian/control</span>
<span class="go"> index 01ca968..fbba1ae 100644</span>
<span class="go"> --- a/debian/control</span>
<span class="go"> +++ b/debian/control</span>
<span class="go"> @@ -20,6 +20,7 @@ Architecture: all</span>
<span class="go"> Depends: ${misc:Depends}</span>
<span class="go"> Recommends: javascript-common</span>
<span class="go"> Breaks: ruby-rails-assets-autosize (<< 4.0)</span>
<span class="go"> +Multi-Arch: foreign</span>
<span class="go"> Description: script to automatically adjust textarea height to fit text - NodeJS</span>
<span class="go"> Autosize is a small, stand-alone script to automatically adjust textarea</span>
<span class="go"> height to fit text. The autosize function accepts a single textarea element,</span>
<span class="go"> @@ -32,6 +33,7 @@ Package: node-autosize</span>
<span class="go"> Architecture: all</span>
<span class="go"> Depends: ${misc:Depends}</span>
<span class="go"> , nodejs</span>
<span class="go"> +Multi-Arch: foreign</span>
<span class="go"> Description: script to automatically adjust textarea height to fit text - Javascript</span>
<span class="go"> Autosize is a small, stand-alone script to automatically adjust textarea</span>
<span class="go"> height to fit text. The autosize function accepts a single textarea element,</span>
</pre></div>
<p>The Debian Janitor also has a new <a class="reference external" href="https://janitor.debian.net/multiarch-fixes/">multiarch-fixes</a> suite that runs <em>apply-multiarch-hints</em> across packages in the archive
and proposes merge requests. For example, you can see the
merge request against autosize.js <a class="reference external" href="https://salsa.debian.org/js-team/autosize.js/-/merge_requests/1">here</a>.</p>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
Debian Janitor: All Packages Processed with Lintian-Brush2020-09-12T08:00:00+02:002020-09-12T08:00:00+02:00Jelmer Vernooij, Perry Lorriertag:www.jelmer.uk,2020-09-12:janitor-update-5.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>On 12 July 2019, the Janitor started fixing lintian issues in packages in the
Debian archive. Now, a year and a half later, it has processed every one of the
almost 28,000 packages at least once.</p>
<a class="reference external image-reference" href="https://janitor.debian.net/lintian-fixes/stats#burndown"><img alt="Graph with Lintian Fixes Burndown" src="/images/janitor-lintian-fixes-burndown.png" /></a>
<p>As discussed two weeks ago, this has resulted in roughly
65,000 total changes. These 65,000 changes were made to
a total of almost 17,000 packages. Of the remaining packages,
for about 4,500 lintian-brush could not make any improvements.
The rest (about 6,500) failed to be processed for one
of many reasons – they are e.g. not yet migrated off
<a class="reference external" href="https://wiki.debian.org/Alioth">alioth</a>, use uncommon formatting
that can’t be preserved or failed to build for one reason or another.</p>
<img alt="Graph with runs by status (success, failed, nothing-to-do)" src="/images/janitor-lintian-fixes-status.png" />
<p>Now that the entire archive has been processed, packages
are prioritized based on the likelihood of a change being
made to them successfully.</p>
<p>Over the course of its existence, the Janitor has slowly
gained support for a wider variety of packaging methods.
For example, it can now edit the templates for some of
the generated control files. Many of the packages that
the janitor was unable to propose changes for the first
time around are expected to be correctly handled when they
are reprocessed.</p>
<p>If you’re a Debian developer, you can find the list of improvements
made by the janitor in your packages by going to
<a class="reference external" href="https://janitor.debian.net/m/">https://janitor.debian.net/m/</a>.</p>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
Debian Janitor: The Slow Trickle from Git Repositories to the Debian Archive2020-08-29T08:00:00+02:002020-08-29T08:00:00+02:00Jelmer Vernooij, Perry Lorriertag:www.jelmer.uk,2020-08-29:janitor-update-4.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p><a class="reference external" href="https://jelmer.uk/janitor-update-3.html">Last week’s blog post</a>
documented how there are now over 30,000 lintian issues
that have been fixed in git packaging repositories by the Janitor.</p>
<p>It’s important to note that any fixes from the Janitor that
make it into a Git packaging repository will also need
to be uploaded to the Debian archive. This currently requires
that a Debian packager clones the repository and builds
and uploads the package.</p>
<p>Until a change makes it into the archive, users of Debian
will unfortunately not see the benefits of improvements
made by the Janitor.</p>
<p>82% of the 30,000 changes from the Janitor that have made
it into a Git repository have not yet been uploaded, although
changes do slowly trickle in as maintainers make other
changes to packages and upload them along with the lintian
fixes from the Janitor. This is not just true for changes
from the Janitor, but for all sorts of other smaller improvements
as well.</p>
<p>However, the process of cloning and building git repositories
and uploading the resulting packages to the Debian archive
is fairly time-consuming – and it’s probably not worth
the time of developers to follow up every change from the
Janitor with a labour-intensive upload to the archive.</p>
<p>It would be great if it was easier to trigger uploads from
git commits. Projects like <a class="reference external" href="https://spwhitton.name/blog/entry/tag2upload/">tag2upload</a> will hopefully help, and
make it more likely that changes end up in the Debian archive.</p>
<p>The majority packages do get at least one new source version
upload per release, so most changes will eventually make
it into the archive.</p>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
Debian Janitor: > 60,000 Lintian Issues Automatically Fixed2020-08-22T08:00:00+02:002020-08-22T08:00:00+02:00Jelmer Vernooij, Perry Lorriertag:www.jelmer.uk,2020-08-22:janitor-update-3.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<div class="section" id="scheduling-lintian-fixes">
<h2>Scheduling Lintian Fixes</h2>
<p>To determine which packages to process, the <a class="reference external" href="https://janitor.debian.net/">Janitor</a> looks at the import of <a class="reference external" href="https://lintian.debian.org/">lintian</a> output across the archive that is available
in <a class="reference external" href="https://wiki.debian.org/UltimateDebianDatabase/"><span class="caps">UDD</span></a> <a class="footnote-reference" href="#f1" id="footnote-reference-1">[1]</a>. It
will prioritize those packages with the most and more severe issues that it has
fixers for.</p>
<p>Once a package is selected, it will clone the packaging repository and run
<a class="reference external" href="https://manpages.debian.org/testing/lintian-brush/lintian-brush.1.en.html">lintian-brush</a>
on it. Lintian-brush provides a framework for applying a set of “fixers” to a
package. It will run each of a set of “fixers” in a pristine version of the
repository, and handles most of the heavy lifting.</p>
</div>
<div class="section" id="the-inner-workings-of-a-fixer">
<h2>The Inner Workings of a Fixer</h2>
<p>Each fixer is just an executable which gets run in a clean
checkout of the package, and can make changes there. Most
of the fixers are written in Python or shell, but they
can be in any language.</p>
<p>The contract for fixers is pretty simple:</p>
<ul class="simple">
<li>If the fixer exits with non-zero, the changes are reverted and fixer is
considered to have failed</li>
<li>If it exits with zero and made changes, then it should write a summary of its
changes to standard out</li>
</ul>
<p>If a fixer is uncertain about the changes it has made, it should report so on
standard output using a pseudo-header. By default, lintian-brush will discard
any changes with uncertainty but if you are running it locally you can still
apply them by specifying <tt class="docutils literal"><span class="pre">--uncertain</span></tt>.</p>
<p>The summary message on standard out will be used for the commit message and
(possibly) the changelog message, if the package doesn’t use gbp dch.</p>
</div>
<div class="section" id="example-fixer">
<h2>Example Fixer</h2>
<p>Let’s look at an example. The package priority “extra” is deprecated since
Debian Policy 4.0.1 (released August 2 017) – see
<a class="reference external" href="https://www.debian.org/doc/debian-policy/ch-archive.html#priorities">Policy 2.5 “Priorities”</a>.
Instead, most packages should use the “optional” priority.</p>
<p>Lintian will warn when a package uses the deprecated “extra” value for the
“Priority” - the associated tag is
<a class="reference external" href="https://lintian.debian.org/tags/priority-extra-is-replaced-by-priority-optional.html">priority-extra-is-replaced-by-priority-optional</a>.
Lintian-brush has a fixer script that can automatically replace “extra” with “optional”.</p>
<p>On systems that have lintian-brush installed, the source for the fixer lives in
<a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush/-/blob/master/fixers/priority-extra-is-replaced-by-priority-optional.py">/usr/share/lintian-brush/fixers/priority-extra-is-replaced-by-priority-optional.py</a>,
but here is a copy of it for reference:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="ch">#!/usr/bin/python3</span>
<span class="kn">from</span> <span class="nn">debmutate.control</span> <span class="kn">import</span> <span class="n">ControlEditor</span>
<span class="kn">from</span> <span class="nn">lintian_brush.fixer</span> <span class="kn">import</span> <span class="n">report_result</span><span class="p">,</span> <span class="n">fixed_lintian_tag</span>
<span class="k">with</span> <span class="n">ControlEditor</span><span class="p">()</span> <span class="k">as</span> <span class="n">updater</span><span class="p">:</span>
<span class="k">for</span> <span class="n">para</span> <span class="ow">in</span> <span class="n">updater</span><span class="o">.</span><span class="n">paragraphs</span><span class="p">:</span>
<span class="k">if</span> <span class="n">para</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"Priority"</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"extra"</span><span class="p">:</span>
<span class="n">para</span><span class="p">[</span><span class="s2">"Priority"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"optional"</span>
<span class="n">fixed_lintian_tag</span><span class="p">(</span>
<span class="n">para</span><span class="p">,</span> <span class="s1">'priority-extra-is-replaced-by-priority-optional'</span><span class="p">)</span>
<span class="n">report_result</span><span class="p">(</span><span class="s2">"Change priority extra to priority optional."</span><span class="p">)</span>
</pre></div></td></tr></table></div>
<p>This fixer is written in Python and uses the <a class="reference external" href="https://salsa.debian.org/jelmer/debmutate">debmutate</a> library to easily modify
control files while preserving formatting — or back out if it is not possible
to preserve formatting.</p>
<p>All the current fixers come with tests, e.g. for this particular fixer the
tests can be found here:
<a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush/-/tree/master/tests/priority-extra-is-replaced-by-priority-optional">https://salsa.debian.org/jelmer/lintian-brush/-/tree/master/tests/priority-extra-is-replaced-by-priority-optional</a>.</p>
<p>For more details on writing new fixers, see the <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush#writing-new-fixers"><span class="caps">README</span></a> for lintian-brush.</p>
<p>For more details on debugging them, see the <a class="reference external" href="https://manpages.debian.org/unstable/lintian-brush/lintian-brush.1.en.html">manual page</a>.</p>
</div>
<div class="section" id="successes-by-fixer">
<h2>Successes by fixer</h2>
<p>Here is a list of the fixers currently available, with the number of successful
merges/pushes per fixer:</p>
<table>
<thead>
<tr class="row-odd">
<th class="head">Lintian Tag</th>
<th class="head">Previously merged/pushed</th>
<th class="head">Ready but not yet merged/pushed</th>
</tr>
</thead>
<tbody valign="top" name="tag-table">
<tr>
<td><a href="https://lintian.debian.org/tags/uses-debhelper-compat-file.html">uses-debhelper-compat-file</a></td>
<td>4906</td>
<td>4161</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/upstream-metadata-file-is-missing.html">upstream-metadata-file-is-missing</a></td>
<td>4281</td>
<td>3841</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/package-uses-old-debhelper-compat-version.html">package-uses-old-debhelper-compat-version</a></td>
<td>4256</td>
<td>3617</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/upstream-metadata-missing-bug-tracking.html">upstream-metadata-missing-bug-tracking</a></td>
<td>2438</td>
<td>2995</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/out-of-date-standards-version.html">out-of-date-standards-version</a></td>
<td>2062</td>
<td>2936</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/upstream-metadata-missing-repository.html">upstream-metadata-missing-repository</a></td>
<td>1936</td>
<td>2987</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/trailing-whitespace.html">trailing-whitespace</a></td>
<td>1720</td>
<td>2295</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/insecure-copyright-format-uri.html">insecure-copyright-format-uri</a></td>
<td>1791</td>
<td>1093</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/package-uses-deprecated-debhelper-compat-version.html">package-uses-deprecated-debhelper-compat-version</a></td>
<td>1391</td>
<td>1287</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/vcs-obsolete-in-debian-infrastructure.html">vcs-obsolete-in-debian-infrastructure</a></td>
<td>872</td>
<td>782</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/homepage-field-uses-insecure-uri.html">homepage-field-uses-insecure-uri</a></td>
<td>527</td>
<td>1111</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/vcs-field-not-canonical.html">vcs-field-not-canonical</a></td>
<td>850</td>
<td>655</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-changelog-has-wrong-day-of-week.html">debian-changelog-has-wrong-day-of-week</a></td>
<td>224</td>
<td>376</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-watch-uses-insecure-uri.html">debian-watch-uses-insecure-uri</a></td>
<td>314</td>
<td>242</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/useless-autoreconf-build-depends.html">useless-autoreconf-build-depends</a></td>
<td>112</td>
<td>428</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/priority-extra-is-replaced-by-priority-optional.html">priority-extra-is-replaced-by-priority-optional</a></td>
<td>315</td>
<td>194</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-rules-contains-unnecessary-get-orig-source-target.html">debian-rules-contains-unnecessary-get-orig-source-target</a></td>
<td>35</td>
<td>428</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/tab-in-license-text.html">tab-in-license-text</a></td>
<td>125</td>
<td>320</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-changelog-line-too-long.html">debian-changelog-line-too-long</a></td>
<td>186</td>
<td>190</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-rules-sets-dpkg-architecture-variable.html">debian-rules-sets-dpkg-architecture-variable</a></td>
<td>69</td>
<td>166</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-rules-uses-unnecessary-dh-argument.html">debian-rules-uses-unnecessary-dh-argument</a></td>
<td>42</td>
<td>182</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/package-lacks-versioned-build-depends-on-debhelper.html">package-lacks-versioned-build-depends-on-debhelper</a></td>
<td>125</td>
<td>95</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/unversioned-copyright-format-uri.html">unversioned-copyright-format-uri</a></td>
<td>43</td>
<td>136</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/package-needs-versioned-debhelper-build-depends.html">package-needs-versioned-debhelper-build-depends</a></td>
<td>127</td>
<td>50</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/binary-control-field-duplicates-source.html">binary-control-field-duplicates-source</a></td>
<td>34</td>
<td>134</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/renamed-tag.html">renamed-tag</a></td>
<td>73</td>
<td>69</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/vcs-field-uses-insecure-uri.html">vcs-field-uses-insecure-uri</a></td>
<td>14</td>
<td>109</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/uses-deprecated-adttmp.html">uses-deprecated-adttmp</a></td>
<td>13</td>
<td>91</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debug-symbol-migration-possibly-complete.html">debug-symbol-migration-possibly-complete</a></td>
<td>12</td>
<td>88</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/copyright-refers-to-symlink-license.html">copyright-refers-to-symlink-license</a></td>
<td>51</td>
<td>48</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-control-has-unusual-field-spacing.html">debian-control-has-unusual-field-spacing</a></td>
<td>33</td>
<td>66</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/old-source-override-location.html">old-source-override-location</a></td>
<td>32</td>
<td>62</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/out-of-date-copyright-format.html">out-of-date-copyright-format</a></td>
<td>20</td>
<td>62</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/public-upstream-key-not-minimal.html">public-upstream-key-not-minimal</a></td>
<td>43</td>
<td>30</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/older-source-format.html">older-source-format</a></td>
<td>17</td>
<td>54</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/custom-compression-in-debian-source-options.html">custom-compression-in-debian-source-options</a></td>
<td>12</td>
<td>57</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/copyright-refers-to-versionless-license-file.html">copyright-refers-to-versionless-license-file</a></td>
<td>29</td>
<td>39</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/tab-in-licence-text.html">tab-in-licence-text</a></td>
<td>33</td>
<td>31</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/global-files-wildcard-not-first-paragraph-in-dep5-copyright.html">global-files-wildcard-not-first-paragraph-in-dep5-copyright</a></td>
<td>28</td>
<td>33</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/out-of-date-copyright-format-uri.html">out-of-date-copyright-format-uri</a></td>
<td>9</td>
<td>50</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/field-name-typo-dep5-copyright.html">field-name-typo-dep5-copyright</a></td>
<td>29</td>
<td>29</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/copyright-does-not-refer-to-common-license-file.html">copyright-does-not-refer-to-common-license-file</a></td>
<td>13</td>
<td>42</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debhelper-but-no-misc-depends.html">debhelper-but-no-misc-depends</a></td>
<td>9</td>
<td>45</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-watch-file-is-missing.html">debian-watch-file-is-missing</a></td>
<td>11</td>
<td>41</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-control-has-obsolete-dbg-package.html">debian-control-has-obsolete-dbg-package</a></td>
<td>8</td>
<td>40</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/possible-missing-colon-in-closes.html">possible-missing-colon-in-closes</a></td>
<td>31</td>
<td>13</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/unnecessary-testsuite-autopkgtest-field.html">unnecessary-testsuite-autopkgtest-field</a></td>
<td>32</td>
<td>9</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/missing-debian-source-format.html">missing-debian-source-format</a></td>
<td>7</td>
<td>33</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debhelper-tools-from-autotools-dev-are-deprecated.html">debhelper-tools-from-autotools-dev-are-deprecated</a></td>
<td>9</td>
<td>29</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/vcs-field-mismatch.html">vcs-field-mismatch</a></td>
<td>8</td>
<td>29</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-changelog-file-contains-obsolete-user-emacs-setting.html">debian-changelog-file-contains-obsolete-user-emacs-setting</a></td>
<td>33</td>
<td>0</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/patch-file-present-but-not-mentioned-in-series.html">patch-file-present-but-not-mentioned-in-series</a></td>
<td>24</td>
<td>9</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/ copyright-refers-to-versionless-license-file.html"> copyright-refers-to-versionless-license-file</a></td>
<td>22</td>
<td>9</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-control-has-empty-field.html">debian-control-has-empty-field</a></td>
<td>25</td>
<td>6</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/missing-build-dependency-for-dh-addon.html">missing-build-dependency-for-dh-addon</a></td>
<td>10</td>
<td>20</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/obsolete-field-in-dep5-copyright.html">obsolete-field-in-dep5-copyright</a></td>
<td>15</td>
<td>13</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/xs-testsuite-field-in-debian-control.html">xs-testsuite-field-in-debian-control</a></td>
<td>20</td>
<td>7</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/ancient-python-version-field.html">ancient-python-version-field</a></td>
<td>13</td>
<td>12</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/unnecessary-team-upload.html">unnecessary-team-upload</a></td>
<td>19</td>
<td>5</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/misspelled-closes-bug.html">misspelled-closes-bug</a></td>
<td>6</td>
<td>16</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/field-name-typo-in-dep5-copyright.html">field-name-typo-in-dep5-copyright</a></td>
<td>1</td>
<td>20</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/transitional-package-not-oldlibs-optional.html">transitional-package-not-oldlibs-optional</a></td>
<td>4</td>
<td>17</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/maintainer-script-without-set-e.html">maintainer-script-without-set-e</a></td>
<td>9</td>
<td>11</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/dh-clean-k-is-deprecated.html">dh-clean-k-is-deprecated</a></td>
<td>4</td>
<td>14</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/no-dh-sequencer.html">no-dh-sequencer</a></td>
<td>14</td>
<td>4</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/missing-vcs-browser-field.html">missing-vcs-browser-field</a></td>
<td>5</td>
<td>12</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/space-in-std-shortname-in-dep5-copyright.html">space-in-std-shortname-in-dep5-copyright</a></td>
<td>6</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/xc-package-type-in-debian-control.html">xc-package-type-in-debian-control</a></td>
<td>4</td>
<td>11</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-rules-missing-recommended-target.html">debian-rules-missing-recommended-target</a></td>
<td>4</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/desktop-entry-contains-encoding-key.html">desktop-entry-contains-encoding-key</a></td>
<td>1</td>
<td>13</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/build-depends-on-obsolete-package.html">build-depends-on-obsolete-package</a></td>
<td>4</td>
<td>9</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/license-file-listed-in-debian-copyright.html">license-file-listed-in-debian-copyright</a></td>
<td>1</td>
<td>12</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/missing-built-using-field-for-golang-package.html">missing-built-using-field-for-golang-package</a></td>
<td>9</td>
<td>4</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/unused-license-paragraph-in-dep5-copyright.html">unused-license-paragraph-in-dep5-copyright</a></td>
<td>4</td>
<td>7</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/missing-build-dependency-for-dh_command.html">missing-build-dependency-for-dh_command</a></td>
<td>6</td>
<td>4</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/comma-separated-files-in-dep5-copyright.html">comma-separated-files-in-dep5-copyright</a></td>
<td>3</td>
<td>6</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/systemd-service-file-refers-to-var-run.html">systemd-service-file-refers-to-var-run</a></td>
<td>4</td>
<td>5</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/copyright-not-using-common-license-for-apache2.html">copyright-not-using-common-license-for-apache2</a></td>
<td>3</td>
<td>5</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-tests-control-autodep8-is-obsolete.html">debian-tests-control-autodep8-is-obsolete</a></td>
<td>2</td>
<td>6</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/dh-quilt-addon-but-quilt-source-format.html">dh-quilt-addon-but-quilt-source-format</a></td>
<td>2</td>
<td>6</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/no-homepage-field.html">no-homepage-field</a></td>
<td>3</td>
<td>5</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/font-package-not-multi-arch-foreign.html">font-packge-not-multi-arch-foreign</a></td>
<td>1</td>
<td>6</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/homepage-in-binary-package.html">homepage-in-binary-package</a></td>
<td>1</td>
<td>4</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/vcs-field-bitrotted.html">vcs-field-bitrotted</a></td>
<td>1</td>
<td>3</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/built-using-field-on-arch-all-package.html">built-using-field-on-arch-all-package</a></td>
<td>2</td>
<td>1</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/ copyright-should-refer-to-common-license-file-for-apache-2.html"> copyright-should-refer-to-common-license-file-for-apache-2</a></td>
<td>1</td>
<td>2</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-pyversions-is-obsolete.html">debian-pyversions-is-obsolete</a></td>
<td>3</td>
<td>0</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/debian-watch-file-uses-deprecated-githubredir.html">debian-watch-file-uses-deprecated-githubredir</a></td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/executable-desktop-file.html">executable-desktop-file</a></td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/skip-systemd-native-flag-missing-pre-depends.html">skip-systemd-native-flag-missing-pre-depends</a></td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/vcs-field-uses-not-recommended-uri-format.html">vcs-field-uses-not-recommended-uri-format</a></td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/init.d-script-needs-depends-on-lsb-base.html">init.d-script-needs-depends-on-lsb-base</a></td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/maintainer-also-in-uploaders.html">maintainer-also-in-uploaders</a></td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/public-upstream-keys-in-multiple-locations.html">public-upstream-keys-in-multiple-locations</a></td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<td><a href="https://lintian.debian.org/tags/wrong-debian-qa-group-name.html">wrong-debian-qa-group-name</a></td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<td><b>Total</b></td>
<td><b>29656</b></td>
<td><b>32209</b></td>
</tr>
</table><p class="rubric">Footnotes</p>
<table class="docutils footnote" frame="void" id="f1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>temporarily unavailable due to <a class="reference external" href="https://bugs.debian.org/960156">Debian bug #960156</a> – but the Janitor is relying on historical data</td></tr>
</tbody>
</table>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
</div>
Debian Janitor: 8,200 landed changes landed so far2020-08-15T08:00:00+02:002020-08-15T08:00:00+02:00Jelmer Vernooij, Perry Lorriertag:www.jelmer.uk,2020-08-15:janitor-update-2.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>The bot has been submitting merge requests for about seven
months now. The rollout has happened gradually across the
Debian archive, and the bot is now enabled for all packages
maintained on <a class="reference external" href="https://salsa.debian.org/">Salsa</a>, <a class="reference external" href="https://gitlab.com/">GitLab</a>, <a class="reference external" href="https://github.com/">GitHub</a> and <a class="reference external" href="https://launchpad.net/">Launchpad</a>.</p>
<p>There are currently over 1,000 open merge requests, and
close to 3,400 merge requests have been merged so far.
Direct pushes are enabled for a number of large Debian
teams, with about 5,000 direct pushes to date. That covers
<a class="reference external" href="https://janitor.debian.net/lintian-fixes/stats">about 11,000 lintian tags of varying severities</a> (about 75 different varieties) fixed across Debian.</p>
<img alt="Janitor pushes over time" src="/images/janitor-pushes-over-time.png" />
<img alt="Janitor merges over time" src="/images/janitor-merges-over-time.png" />
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
Improvements to Merge Proposals by the Janitor2020-08-08T08:00:00+02:002020-08-08T08:00:00+02:00Jelmer Vernooij, Perry Lorriertag:www.jelmer.uk,2020-08-08:janitor-update-1.html<p class="italic">The <a class="reference external" href="https://jelmer.uk/debian-janitor.html">Debian Janitor</a> is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on sid packages maintained in
Git. This post is part of <a class="reference external" href="https://jelmer.uk/tag/janitor-update.html">a series</a> about the progress of the Janitor.</p>
<p>Since the original post, merge proposals created by the
janitor now include the
<a class="reference external" href="https://manpages.debian.org/testing/devscripts/debdiff.1.en.html">debdiff</a>
between a build with and without the changes (showing the impact to the binary
packages), in addition to the merge proposal diff (which shows the impact to
the source package).</p>
<p>New merge proposals also include a link to the <a class="reference external" href="https://www.diffoscope.org/">diffoscope</a> diff between a vanilla build and the build
with changes. Unfortunately these can be a bit noisy for packages that are not
<a class="reference external" href="https://reproducible-builds.org/">reproducible</a>
yet, due to the difference in build environment between
the two builds.</p>
<p>This is part of the effort to keep the changes from the
janitor <a class="reference external" href="https://www.jelmer.uk/debian-janitor.html#ensuring-quality">high-quality</a>.</p>
<p>The rollout surfaced <a class="reference external" href="https://bugs.debian.org/lintian-brush">some bugs in lintian-brush</a>; these have been either fixed or
mitigated (e.g. by disabling specified fixers).</p>
<p class="italic">For more information about the Janitor’s lintian-fixes efforts, see <a class="reference external" href="https://janitor.debian.net/lintian-fixes/">the landing page</a>.</p>
The Debian Janitor2019-12-03T22:00:12+01:002019-12-03T22:00:12+01:00Jelmer Vernooijtag:www.jelmer.uk,2019-12-03:debian-janitor.html<!-- TL;DR
=====
The `Janitor <https://janitor.debian.net/>` is a new bot that automatically
proposes improvements for Debian packages maintained in Git. -->
<!-- Problem statement -->
<p>There are a lot of small changes that can be made to the Debian archive to
increase the overall quality. Many of these changes are small and have just
minor benefits if they are applied to just a single package. Lintian encourages
maintainers to fix these problems by pointing out the common ones.</p>
<p>Most of these issues are often trivially fixable; they are in general an
inefficient use of human time, and it takes a lot of effort to keep up with.
This is something that can clearly be automated.</p>
<p>Several tools (e.g. onovy’s <a class="reference external" href="https://github.com/onovy/onovy-mass">mass tool</a>,
and the <a class="reference external" href="https://www.jelmer.uk/lintian-brush-intro.html">lintian-brush</a> tool
that I’ve been working on) go a step further and (for a subset of the issues
reported by lintian) fix the problems for you, where they can. Lintian-brush
can currently fix most instances of <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush#supported-tags">close to 100 lintian tags</a>.</p>
<p>Thanks to the Vcs-* fields set by many packages and <a class="reference external" href="https://docs.gitlab.com/ee/api/">the APIs provided by hosting
platforms like Salsa</a>, it is now possible to
proactively attempt to fix these issues.</p>
<p>The Debian Janitor is a tool that will run lintian-brush across the entire
archive, and propose fixes to lintian issues via pull request.</p>
<div class="section" id="objectives">
<h2>Objectives</h2>
<p>The aim of Debian Janitor is to take some drudge work away from Debian
maintainers where possible, so they can spend their time on more important
packaging work. Its purpose is to make automated changes quick and easy to
apply, with minimal overhead for package maintainers. It is essentially a bit
of infrastructure to run lintian-brush across all of the archive.</p>
<p>The actions of the bot are restricted to a limited set of problems for which
obviously correct actions can be taken. It is not meant to automate all
packaging, or even to cover automating all instances of the issues it knows about.</p>
<p>The bot is designed to be conservative and delight with consistently correct
fixes instead of proposing possibly incorrect fixes and hoping for the best.
Considerable effort has been made to avoid the janitor creating pull requests
with incorrect changes, as these take valuable time away from maintainers, the
package doesn’t actually improve (since the merge request is rejected) and it
makes it likelier that future pull requests from the Debian Janitor bot are
ignored or rejected.</p>
<p>In short: The janitor is meant to propose correct changes if it can, and back
off otherwise.</p>
</div>
<div class="section" id="design">
<h2>Design</h2>
<p>The Janitor finds package sources in version control systems from the Vcs*-
control field in Debian source packages. If the packaging branch is hosted on a
hosting platform that the Janitor has a presence on, it will attempt to run
<a class="reference external" href="https://www.jelmer.uk/lintian-brush-intro.html">lintian-brush</a> on the packaging branch and (if there are any changes made)
build the package and
propose a merge. It is based on
<a class="reference external" href="https://github.com/jelmer/silver-platter/">silver-platter</a> and currently has
support for:</p>
<ul class="simple">
<li><a class="reference external" href="https://salsa.debian.org/">Salsa</a></li>
<li><a class="reference external" href="https://github.com/">GitHub</a></li>
<li><a class="reference external" href="https://launchpad.net">Launchpad</a></li>
<li><a class="reference external" href="https://gitlab.com/">GitLab</a></li>
</ul>
<p>The Janitor is driven from the lintian and vcswatch tables in
<a class="reference external" href="https://wiki.debian.org/UltimateDebianDatabase/"><span class="caps">UDD</span></a>. It queries for packages
that are affected by any of the <a class="reference external" href="https://lintian.debian.org/">lintian</a> tags
that <a class="reference external" href="https://www.jelmer.uk/lintian-brush-intro.html">lintian-brush</a> has a
fixer script for. This way it can limit the number of repositories it has to process.</p>
</div>
<div class="section" id="ensuring-quality">
<h2>Ensuring quality</h2>
<p>There are a couple of things I am doing to make sure that the Debian
Janitor delights rather than annoys.</p>
<div class="section" id="high-quality-changes">
<h3>High quality changes</h3>
<p>Lintian-brush has end-to-end tests for its fixers.</p>
<p>In order to make sure that merge requests are useful and high-value, the bot
will only propose changes from lintian-brush that:</p>
<ul class="simple">
<li>successfully build in a chroot and pass <a class="reference external" href="https://wiki.debian.org/ContinuousIntegration/autopkgtest">autopkgtest</a> and <a class="reference external" href="https://wiki.debian.org/piuparts">piuparts</a>;</li>
<li>are not completely trivial - e.g. only stripping whitespace</li>
</ul>
<p>Changes for a package will also be reviewed by a human before they make it into
a pull request.</p>
</div>
<div class="section" id="one-open-pull-request-per-package">
<h3>One open pull request per package</h3>
<p>If the bot created a pull request previously, it will attempt to update the
current request by adding new commits (and updating the pull request
description). It will remove and fix the branch when the pull request conflicts
because of new upstream changes.</p>
<p>In other words, it will only create a single pull request per package and will
attempt to keep that pull request up to date.</p>
</div>
<div class="section" id="gradual-rollout">
<h3>Gradual rollout</h3>
<p>I’m slowly adding interested maintainers to receiving pull requests, before
opening it up to the entire archive. This should help catch any widespread
issues early.</p>
</div>
<div class="section" id="providing-control">
<h3>Providing control</h3>
<p>The bot will be upfront about its pull requests and try to avoid overwhelming
maintainers with pull requests by:</p>
<ul class="simple">
<li>Clearly identifying any merge requests it creates as being made by a bot.
This should allow maintainers to prioritize contributions from humans.</li>
<li>Limiting the number of open proposals per maintainer. It starts by opening a
single merge request and won’t open additional merge requests until the first
proposal has a response</li>
<li>Providing a way to opt out of future merge requests; just a reply on the
merge request is sufficient.</li>
</ul>
<p>Any comments on merge requests will also still be reviewed by a human.</p>
</div>
</div>
<div class="section" id="current-state">
<h2>Current state</h2>
<p>Debian janitor is running, generating changes and already creating merge
requests (albeit under close review). Some examples of merge requests it has created:</p>
<ul class="simple">
<li><a class="reference external" href="https://salsa.debian.org/i3-team/i3-wm/merge_requests/1">i3 (on Salsa)</a></li>
<li><a class="reference external" href="https://salsa.debian.org/wouter/nbd/merge_requests/2">nbd (on Salsa)</a></li>
<li><a class="reference external" href="https://code.launchpad.net/~debian-janitor/friendly-recovery/lintian-fixes/+merge/375943">friendly recovery (on Launchpad)</a></li>
<li><a class="reference external" href="https://github.com/algernon/dh-exec/pull/3">dh-exec (on GitHub)</a></li>
</ul>
<div class="section" id="using-the-janitor">
<h3>Using the janitor</h3>
<p>The janitor can process any package that’s maintained in Git and has its
Vcs-Git header set correctly (you can use
<a class="reference external" href="https://qa.debian.org/cgi-bin/vcswatch">vcswatch</a> to check this).</p>
<p>If you’re interested in receiving pull requests early, leave a comment below.
Eventually, the janitor should get to all packages, though it may take a while
with the current number of source packages in the archive.</p>
<p>By default, salsa does not send notifications when a new merge request for one
of the repositories you’re a maintainer for is created. Make sure you have
notifications enabled in
<a class="reference external" href="https://salsa.debian.org/profile/notifications">your Salsa profile</a>,
by ticking “New Merge Requests” for the packages you care about.</p>
<p>You can also see the number of open merge requests for a package repository on
<a class="reference external" href="https://qa.debian.org/developer.php"><span class="caps">QA</span></a> - it’s the ! followed by a number in the
pull request column.</p>
<p>It is also possible to download the diff for a particular package (if it’s been
generated) ahead of the janitor publishing it:</p>
<div class="highlight"><pre><span></span><span class="gp"> $ </span>curl<span class="w"> </span>https://janitor.debian.net/api/lintian-fixes/pkg/PACKAGE/diff
</pre></div>
<p>E.g. for i3-wm, look at
<a class="reference external" href="https://janitor.debian.net/api/lintian-fixes/pkg/i3-wm/diff">https://janitor.debian.net/api/lintian-fixes/pkg/i3-wm/diff</a>.</p>
</div>
</div>
<div class="section" id="future-plans">
<h2>Future Plans</h2>
<p>The current set of supported hosting platforms covers the bulk of packages in
Debian that is maintained in a <span class="caps">VCS</span>. The only other 100+ package platform
that’s unsupported is <a class="reference external" href="https://salsa.debian.org/dgit-team/dgit">dgit</a>. If you
have suggestions on how best to submit git changes to dgit repositories (<span class="caps">BTS</span>
bugs with patches? or would that be too much overhead?), let me know.</p>
<p>The next platform that is currently missing is
<a class="reference external" href="https://bitbucket.org/">bitbucket</a>, but there are only about 15 packages in
unstable hosted there.</p>
<p>At the moment, lintian-brush can fix close to 100 lintian tags. It would be
great to add fixers for more common issues.</p>
<p>The janitor should probably be more tightly integrated with other pieces of
Debian infrastructure, e.g. Jenkins for running jobs or linked to from the
<a class="reference external" href="https://tracker.debian.org/">tracker</a> or
<a class="reference external" href="https://lintian.debian.org/">lintian.debian.org</a>.</p>
</div>
<div class="section" id="more-information">
<h2>More information</h2>
<p>See <a class="reference external" href="https://janitor.debian.net/lintian-fixes/#faq">the <span class="caps">FAQ</span> on the homepage</a>.</p>
<p>If you have any concerns about these roll-out plans, have other ideas or
questions, please let me know in the comments.</p>
</div>
Silver Platter2019-04-11T03:04:00+02:002019-04-11T03:04:00+02:00Jelmer Vernooijtag:www.jelmer.uk,2019-04-11:silver-platter-intro.html<p>Making changes across the open source ecosystem is very hard; software is hosted
on different platforms and in many different version control repositories. Not
being able to make bulk changes slows down the rate of progress. For example, instead
of being able to actively run a a script that strips out an obsolete header file
(say “<span class="caps">DM</span>-Upload-Allowed”) across all Debian packages, we make the linter warn about
the deprecated header and wait as all developers manually remove the deprecated header.</p>
<div class="section" id="silver-platter-1">
<h2>Silver Platter</h2>
<p>Silver-platter is a new tool that aids in making automated changes across
different version control repositories. It provides a common command-line
interface and <span class="caps">API</span> that is not specific to a single version control system or hosting
platform, so that it’s easy to propose changes based on a single script across
a large set of repositories.</p>
<p>The tool will check out a repository, run a user-specified script that makes changes
to the repository, and then either push those changes to the upstream repository or
propose them for merging.</p>
<p>It’s specifically built so that it can be run in a shell loop over many
different repository URLs.</p>
<div class="section" id="example">
<h3>Example</h3>
<p>As an example, you could use the following script (<tt class="docutils literal"><span class="pre">fix-fsf-address.sh</span></tt>) to
update the <span class="caps">FSF</span> address in copyright headers:</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="c1">#!/bin/sh</span>
<span class="w"> </span>perl<span class="w"> </span>-i<span class="w"> </span>-pe<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'BEGIN{undef $/;} s/Free Software</span>
<span class="s1"> ([# ]+)Foundation, Inc\., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA/Free Software</span>
<span class="s1"> \1Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA/smg'</span><span class="w"> </span>*
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Update FSF postal address."</span>
</pre></div>
<p>Say you a wanted to create a merge proposal with these changes against
<tt class="docutils literal">offlineimap</tt>. First, log into GitHub (this needs to be done once per hosting site):</p>
<div class="highlight"><pre><span></span><span class="gp"> $ </span>svp<span class="w"> </span>login<span class="w"> </span>https://github.com
</pre></div>
<p>To see what the changes would be without actually creating the pull request, do a dry-run:</p>
<div class="highlight"><pre><span></span><span class="gp"> $ </span>svp<span class="w"> </span>run<span class="w"> </span>--dry-run<span class="w"> </span>--diff<span class="w"> </span>./fix-fsf-address.sh<span class="w"> </span>https://github.com/offlineimap/offlineimap
<span class="go"> Merge proposal created.</span>
<span class="go"> Description: Update FSF postal address.</span>
<span class="go"> === modified file 'offlineimap.py'</span>
<span class="go"> --- upstream/offlineimap.py 2018-03-04 03:28:30 +0000</span>
<span class="go"> +++ proposed/offlineimap.py 2019-04-06 21:07:25 +0000</span>
<span class="go"> @@ -14,7 +14,7 @@</span>
<span class="gp"> #</span>
<span class="gp"> # </span>You<span class="w"> </span>should<span class="w"> </span>have<span class="w"> </span>received<span class="w"> </span>a<span class="w"> </span>copy<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>GNU<span class="w"> </span>General<span class="w"> </span>Public<span class="w"> </span>License
<span class="gp"> # </span>along<span class="w"> </span>with<span class="w"> </span>this<span class="w"> </span>program<span class="p">;</span><span class="w"> </span><span class="k">if</span><span class="w"> </span>not,<span class="w"> </span>write<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>Free<span class="w"> </span>Software
<span class="go"> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA</span>
<span class="go"> +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA</span>
<span class="go"> import os</span>
<span class="go"> import sys</span>
<span class="go"> === modified file 'setup.py'</span>
<span class="go"> --- upstream/setup.py 2018-05-01 01:48:26 +0000</span>
<span class="go"> +++ proposed/setup.py 2019-04-06 21:07:25 +0000</span>
<span class="go"> @@ -19,7 +19,7 @@</span>
<span class="gp"> #</span>
<span class="gp"> # </span>You<span class="w"> </span>should<span class="w"> </span>have<span class="w"> </span>received<span class="w"> </span>a<span class="w"> </span>copy<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>GNU<span class="w"> </span>General<span class="w"> </span>Public<span class="w"> </span>License
<span class="gp"> # </span>along<span class="w"> </span>with<span class="w"> </span>this<span class="w"> </span>program<span class="p">;</span><span class="w"> </span><span class="k">if</span><span class="w"> </span>not,<span class="w"> </span>write<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>Free<span class="w"> </span>Software
<span class="go"> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA</span>
<span class="go"> +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA</span>
<span class="go"> import os</span>
<span class="go"> from distutils.core import setup, Command</span>
</pre></div>
<p>Then, create the actual pull request by running:</p>
<div class="highlight"><pre><span></span><span class="gp"> $ </span>svp<span class="w"> </span>run<span class="w"> </span>./fix-fsf-address.sh<span class="w"> </span>https://github.com/offlineimap/offlineimap
<span class="go"> ...</span>
<span class="go"> Reusing existing repository https://github.com/jelmer/offlineimap</span>
<span class="go"> Merge proposal created.</span>
<span class="go"> URL: https://github.com/OfflineIMAP/offlineimap/pull/609</span>
<span class="go"> Description: Update FSF postal address.</span>
</pre></div>
<p>This would create a new commit with the updated postal address (if any files
were changed) and the commit message <tt class="docutils literal">Update <span class="caps">FSF</span> postal address</tt>. You can see the resulting
pull request <a class="reference external" href="https://github.com/OfflineIMAP/offlineimap/pull/609">here</a>.</p>
</div>
<div class="section" id="debian-specific-operations">
<h3>Debian-specific operations</h3>
<p>To make working with Debian packaging repositories easier, Silver Platter comes
with a wrapper (<tt class="docutils literal"><span class="pre">debian-svp</span></tt>) specifically for Debian packages.</p>
<p>This wrapper allows specifying package names to refer to packaging branches; packaging
URLs are retrieved from the <tt class="docutils literal"><span class="pre">Vcs-Git</span></tt> header in a package. For example:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span>$ debian-svp run ~/fix-fsf-address.sh offlineimap
</pre></div></td></tr></table></div>
<p>to fix the same issue in the offlineimap package.</p>
<p>(Of course, you wouldn’t normally fix upstream issues like this in the Debian
package but forward them upstream instead)</p>
<p>There is also a <tt class="docutils literal"><span class="pre">debian-svp</span> <span class="pre">lintian-brush</span></tt> subcommand that will invoke
<a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">lintian-brush</a> on a packaging branch.</p>
</div>
<div class="section" id="supported-technologies">
<h3>Supported technologies</h3>
<p>Silver-Platter currently supports the following hosting platforms:</p>
<blockquote>
<ul class="simple">
<li><a class="reference external" href="https://github.com/">GitHub</a></li>
<li>GitLab instances (for example <a class="reference external" href="https://salsa.debian.org/">Salsa</a> or <a class="reference external" href="https://gitlab.gnome.org/"><span class="caps">GNOME</span> gitlab</a>)</li>
<li><a class="reference external" href="https://https://launchpad.net/">Launchpad</a> (both Git and Bazaar repositories through <a class="reference external" href="https://www.breezy-vcs.org/">Breezy</a>)</li>
</ul>
</blockquote>
<p>It works in one of three modes:</p>
<blockquote>
<ul class="simple">
<li><em>propose</em>: Always create a pull request with the changes</li>
<li><em>push</em>: Directly push changes back to the original branch</li>
<li><em>attempt-push</em>: Attempt <em>push</em>, and fall back to <em>propose</em> if the current
users doesn’t have permissions to push to the repository or the branch.</li>
</ul>
</blockquote>
</div>
<div class="section" id="installation">
<h3>Installation</h3>
<p>There is a Silver Platter repository <a class="reference external" href="https://github.com/jelmer/silver-platter">on GitHub</a>. Silver Platter is also available
as a Debian package <a class="reference external" href="https://packages.debian.org/silver-platter">in unstable</a> (not buster).</p>
</div>
<div class="section" id="more-information">
<h3>More information</h3>
<p>For a full list of <tt class="docutils literal">svp</tt> subcommands, see <a class="reference external" href="https://github.com/jelmer/silver-platter/blob/master/man/svp.1">svp(1)</a>.</p>
</div>
</div>
Breezy evolves2019-03-24T16:00:00+00:002019-03-24T16:00:00+00:00Jelmer Vernooijtag:www.jelmer.uk,2019-03-24:breezy-evolution.html<p>Last month Martin, Vincent and I finally released version 3.0.0 of
<a class="reference external" href="https://www.breezy-vcs.org/">Breezy</a>, a little over a year after we
originally <a class="reference external" href="https://www.jelmer.uk/bazaar/breezy-intro.html">forked
Bazaar</a>.</p>
<p>When we started working on Breezy, it was mostly as a way to keep Bazaar
working going forward - in a world where Python 2 has mostly disappeared
in favour of Python 3).</p>
<div class="section" id="improvements">
<h2>Improvements</h2>
<p>Since then, we have also made other improvements. In addition to Python 3
support, Breezy comes with the following other bigger changes:</p>
<div class="section" id="batteries-included">
<h3>Batteries Included</h3>
<p>Breezy bundles most of the common plugins. This makes the installation of
Breezy much simpler (<tt class="docutils literal">pip install brz</tt>), and prevents possible issues with
<span class="caps">API</span> incompatibility that plagued Bazaar.</p>
<p>Bundled plugins include: <tt class="docutils literal">grep</tt>, <tt class="docutils literal">git</tt>, <tt class="docutils literal">fastimport</tt>, <tt class="docutils literal">propose</tt>,
<tt class="docutils literal">upload</tt>, <tt class="docutils literal">stats</tt> and parts of <tt class="docutils literal">bzrtools</tt>.</p>
</div>
<div class="section" id="fixed-bugs">
<h3>>120 fixed bugs</h3>
<p>Since Bazaar 2.7, lots of bugs in the Bazaar code base have been fixed
(over 120 as of March 2019). We’ve also started an effort to go through
all bugs in the Bazaar bug tracker to see whether they also apply to Breezy.</p>
</div>
<div class="section" id="native-git-support">
<h3>Native Git Support</h3>
<p>Breezy now supports the Git file formats as a first class citizen; Git
support is included in Breezy itself, and should work just as well
as regular Bazaar format repositories.</p>
</div>
<div class="section" id="improved-abstractions">
<h3>Improved abstractions</h3>
<p>Bazaar has always had a higher level <span class="caps">API</span> that could be used for version control
operations, and which was implemented for both Bazaar, Git and Subversion formats.</p>
<p>As part of the work to support the Git format natively, we have changed the
<span class="caps">API</span> to remove Bazaar-specific artefacts, like the use of file ids. Inventories
(a Bazaar concept) are now also an implementation detail of the bzr formats,
and not a concept that is visible in the <span class="caps">API</span> or <span class="caps">UI</span>.</p>
<p>In the future, I hope the <span class="caps">API</span> will be useful for tools that want to make
automated changes to any version controlled resource, whether that be
Git, Bazaar, Subversion or Mercurial repositories.</p>
</div>
</div>
Lintian Brush2018-12-09T00:04:00+01:002018-12-09T00:04:00+01:00Jelmer Vernooijtag:www.jelmer.uk,2018-12-09:lintian-brush-intro.html<p>With Debian packages now widely being maintained in Git repositories, there has
been an uptick in the number of bulk changes made to Debian packages. Several
maintainers are running commands over many packages (e.g. all packages owned by a
specific team) to fix common issues in packages.</p>
<p>Examples of changes being made include:</p>
<blockquote>
<ul class="simple">
<li>Updating the <tt class="docutils literal"><span class="pre">Vcs-Git</span></tt> and <tt class="docutils literal"><span class="pre">Vcs-Browser</span></tt> URLs after migrating from alioth to salsa</li>
<li>Stripping trailing whitespace in various control files</li>
<li>Updating e.g. homepage URLs to use https rather than http</li>
</ul>
</blockquote>
<p>Most of these can be fixed with simple sed or perl one-liners.</p>
<p>Some of these scripts are publically available, for example:</p>
<blockquote>
<ul class="simple">
<li>The R packaging team’s <a class="reference external" href="https://salsa.debian.org/r-pkg-team/maintenance-utilities/blob/master/routine-update">routine-update</a>,</li>
<li>Ondřej Nový’s <a class="reference external" href="https://github.com/onovy/onovy-mass/tree/master/checks">onovy-mass</a> repository.</li>
</ul>
</blockquote>
<div class="section" id="lintian-brush-1">
<h2>Lintian-Brush</h2>
<p>Lintian-Brush is both a simple wrapper around a set of these kinds of scripts
and a repository for these scripts, with the goal of making it easy for any Debian
maintainer to run them.</p>
<p>The lintian-brush command-line tool is a simple wrapper that runs a set of
“fixer scripts”, and for each:</p>
<blockquote>
<ul class="simple">
<li>Reverts the changes made by the script if it failed with an error</li>
<li>Commits the changes to the <span class="caps">VCS</span> with an appropriate commit message</li>
<li>Adds a changelog entry (if desired)</li>
</ul>
</blockquote>
<p>The tool also provides some basic infrastructure for testing that these scripts
do what they should, and e.g. don’t have unintended side-effects.</p>
<p>The idea is that it should be safe, quick and unobtrusive to run <tt class="docutils literal"><span class="pre">lintian-brush</span></tt>,
and get it to opportunistically fix lintian issues and to leave the source tree
alone when it can’t.</p>
<div class="section" id="example">
<h3>Example</h3>
<p>For example, running lintian-brush on the package <tt class="docutils literal">talloc</tt> fixes two minor lintian issues:</p>
<div class="highlight"><pre><span></span><span class="gp"> % </span>debcheckout<span class="w"> </span>talloc
<span class="go"> declared git repository at https://salsa.debian.org/samba-team/talloc.git</span>
<span class="go"> git clone https://salsa.debian.org/samba-team/talloc.git talloc ...</span>
<span class="go"> Cloning into 'talloc'...</span>
<span class="go"> remote: Enumerating objects: 2702, done.</span>
<span class="go"> remote: Counting objects: 100% (2702/2702), done.</span>
<span class="go"> remote: Compressing objects: 100% (996/996), done.</span>
<span class="go"> remote: Total 2702 (delta 1627), reused 2601 (delta 1550)</span>
<span class="go"> Receiving objects: 100% (2702/2702), 1.70 MiB | 565.00 KiB/s, done.</span>
<span class="go"> Resolving deltas: 100% (1627/1627), done.</span>
<span class="gp"> % </span><span class="nb">cd</span><span class="w"> </span>talloc
<span class="go"> talloc% lintian-brush</span>
<span class="go"> Lintian tags fixed: {'insecure-copyright-format-uri', 'public-upstream-key-not-minimal'}</span>
<span class="gp"> % </span>git<span class="w"> </span>log
<span class="go"> commit 0ea35f4bb76f6bca3132a9506189ef7531e5c680 (HEAD -> master)</span>
<span class="go"> Author: Jelmer Vernooij <jelmer@debian.org></span>
<span class="go"> Date: Tue Dec 4 16:42:35 2018 +0000</span>
<span class="go"> Re-export upstream signing key without extra signatures.</span>
<span class="go"> Fixes lintian: public-upstream-key-not-minimal</span>
<span class="go"> See https://lintian.debian.org/tags/public-upstream-key-not-minimal.html for more details.</span>
<span class="go"> debian/changelog | 1 +</span>
<span class="go"> debian/upstream/signing-key.asc | 102 +++++++++++++++---------------------------------------------------------------------------------------</span>
<span class="go"> 2 files changed, 16 insertions(+), 87 deletions(-)</span>
<span class="go"> commit feebce3147df561aa51a385c53d8759b4520c67f</span>
<span class="go"> Author: Jelmer Vernooij <jelmer@debian.org></span>
<span class="go"> Date: Tue Dec 4 16:42:28 2018 +0000</span>
<span class="go"> Use secure copyright file specification URI.</span>
<span class="go"> Fixes lintian: insecure-copyright-format-uri</span>
<span class="go"> See https://lintian.debian.org/tags/insecure-copyright-format-uri.html for more details.</span>
<span class="go"> debian/changelog | 3 +++</span>
<span class="go"> debian/copyright | 2 +-</span>
<span class="go"> 2 files changed, 4 insertions(+), 1 deletion(-)</span>
</pre></div>
</div>
<div class="section" id="script-interface">
<h3>Script Interface</h3>
<p>A fixer script is run in the root directory of a package, where it can make
changes it deems necessary, and write a summary of what it’s done for the changelog
(and commit message) to standard out.</p>
<p>If a fixer can not provide any improvements, it can simply leave the working
tree untouched - lintian-brush will not create any commits for it or update the
changelog. If it exits with a non-zero exit code, then it is assumed that it
failed to run and it will be listed as such and its changes reset rather than committed.</p>
<p>In addition, tests can be added for fixers by providing various <tt class="docutils literal">before</tt> and
<tt class="docutils literal">after</tt> source package trees, to verify that a fixer script makes the expected changes.</p>
<p>For more details, see the <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush#writing-fixers">documentation on writing new fixers</a>.</p>
</div>
<div class="section" id="availability">
<h3>Availability</h3>
<p><tt class="docutils literal"><span class="pre">lintian-brush</span></tt> is currently available in <em>unstable</em> and <em>testing</em>. See man
<tt class="docutils literal"><span class="pre">lintian-brush(1)</span></tt> for an explanation of the command-line options.</p>
<p>Fixer scripts are included that can fix (some of the instances of) <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush#supported-tags">34 lintian
tags</a>.</p>
<p>Feedback would be great if you try <em>lintian-brush</em> - please file bugs in the <span class="caps">BTS</span>, or propose
pull requests with new fixers <a class="reference external" href="https://salsa.debian.org/jelmer/lintian-brush">on salsa</a>.</p>
</div>
</div>
Breezy: Forking Bazaar2018-01-08T17:00:00+01:002018-01-08T17:00:00+01:00Jelmer Vernoojtag:www.jelmer.uk,2018-01-08:breezy-intro.html<p>A couple of months ago, Martin and I announced a friendly fork of Bazaar, named <a class="reference external" href="https://www.breezy-vcs.org/">Breezy</a>.</p>
<p>It’s been 5 years since I wrote a <a class="reference external" href="/pages/bzr-a-retrospective.html">Bazaar retrospective</a> and around
6 since I seriously contributed to the Bazaar codebase.</p>
<div class="section" id="goals">
<h2>Goals</h2>
<p>We don’t have any grand ambitions for Breezy; the main goal is to keep Bazaar
usable going forward. Your open source projects should <a class="reference external" href="/pages/bzr-a-retrospective.html#conclusion">still be using Git</a>.</p>
<p>The main changes we have made so far come down to fixing a number of bugs and
to bundling useful plugins. Bundling plugins makes setting up an environment
simpler and to eliminate the <span class="caps">API</span> compatibility issues that plagued external
plugins in the Bazaar world.</p>
<p>Perhaps the biggest effort in Breezy is porting the codebase to Python 3, allowing
it to be used once Python 2 goes <span class="caps">EOL</span> in 2020.</p>
<!-- One of my hopes is also to see if we can get `bzr-git`_ into a state where it -->
<!-- functions well enough that it can be merged into Breezy and provide native, -->
<!-- reasonably performant Git support. -->
</div>
<div class="section" id="a-fork">
<h2>A fork</h2>
<p>Breezy is a fork of Bazaar and not just a new release series.</p>
<p>Bazaar upstream has been dormant for the last couple of years anyway - we don’t
lose anything by forking.</p>
<p>We’re forking because gives us the independence to make some of
the changes we deemed necessary and that are otherwise hard to make for
an established project, For example, we’re now bundling plugins, taking an axe
to a large number of APIs and dropping support for older platforms.</p>
<p>A fork also means independence from Canonical; there is no <span class="caps">CLA</span> for
Breezy (a hindrance for Bazaar) and we can set up our own infrastructure
without having to chase down Canonical staff for web site updates or the
installation of new packages on the <span class="caps">CI</span> system.</p>
</div>
<div class="section" id="more-information">
<h2>More information</h2>
<p>Martin gave <a class="reference external" href="https://www.youtube.com/watch?v=9vtToynsNSs">a talk</a> about Breezy at PyCon <span class="caps">UK</span> this year.</p>
<p>Breezy bugs can be filed <a class="reference external" href="https://launchpad.net/brz">on Launchpad</a>. For the moment, we are using the
<a class="reference external" href="https://lists.canonical.com/archives/bazaar/">Bazaar mailing list</a> and the <tt class="docutils literal">#bzr</tt>
<span class="caps">IRC</span> channel for any discussions and status updates around Breezy.</p>
</div>
Xandikos, a lightweight Git-backed CalDAV/CardDAV server2017-05-06T03:06:23+02:002017-05-06T03:06:23+02:00Jelmer Vernooijtag:www.jelmer.uk,2017-05-06:xandikos-intro.html<p>For the last couple of years, I have self-hosted my calendar and address book
data. Originally I just kept local calendars and address books in Evolution,
but later I moved to a self-hosted CalDAV/CardDAV server and a plethora of clients.</p>
<div class="section" id="caldav-carddav">
<h2>CalDAV/CardDAV</h2>
<p>CalDAV and CardDAV are standards for accessing, managing, and sharing
calendaring and addressbook information based on the iCalendar format that are
built atop the WebDAV standard, and WebDAV itself is a set of extensions to
<span class="caps">HTTP</span>.</p>
<p>CalDAV and CardDAV essentially store iCalendar (.ics) and vCard (.vcf) files
using WebDAV, but they provide some extra guarantees (e.g. files must be
well-formed) and some additional methods for querying the data. For example,
it is possible to retrieve all events between two dates with a single <span class="caps">HTTP</span>
query, rather than the client having to check all the calendar files in a directory.</p>
<p>CalDAV and CardDAV are (unnecessarily) complex, in large part because they are
built on top of WebDAV. Being able to use regular <span class="caps">HTTP</span> and WebDAV clients
is quite neat, but results in extra complexity. In addition, because the
standards are so large, clients and servers end up only implementing parts of it.</p>
<p>However, CalDAV and CardDAV have one big redeeming quality: they are the
dominant standards for synchronising calendar events and addressbooks, and are
supported by a wide variety of free and non-free applications. They’re the
status quo, until something better comes along. (and hey, at least there is a
standard to begin with)</p>
</div>
<div class="section" id="calypso">
<h2>Calypso</h2>
<p>I have tried a number of servers over the years. In the end, I settled for
<a class="reference external" href="https://keithp.com/blogs/calypso/">calypso</a>.</p>
<p>Calypso started out as friendly fork of <a class="reference external" href="http://radicale.org/">Radicale</a>, with some additional
improvements. I like Calypso because it is:</p>
<ul class="simple">
<li>quite simple, understandable, and small (sloccount reports 1700 <span class="caps">LOC</span>)</li>
<li>it stores plain .vcf and .ics files</li>
<li>stores history in git</li>
<li>easy to set up, e.g. no database dependencies</li>
<li>written in Python</li>
</ul>
<p>Its use of regular files and keeping history in Git is useful, because this
means that whenever it breaks it is much easier to see what is happening. If
something were to go wrong (i.e. a client decides to remove all
server-side entries) it’s easy to recover by rolling back history using git.</p>
<p>However, there are some downsides to Calypso as well.</p>
<p>It doesn’t have good test coverage, making it harder to change (especially
in a way that doesn’t break some clients), though there are some recent efforts
to make e.g. external spec compliance tests like <a class="reference external" href="https://www.calendarserver.org/CalDAVTester.html">caldavtester</a> work with it.</p>
<p>Calypso’s CalDAV/CardDAV/WebDAV implementation is a bit ad-hoc. The only WebDAV
REPORTs it implements are calendar-multiget and addressbook-multiget. Support
for properties has been added as new clients request them. The logic for replying
to <span class="caps">DAV</span> requests is mixed with the actual data store implementation.</p>
<p>Because of this, it can be hard to get going with some clients and sometimes
tricky to debug.</p>
</div>
<div class="section" id="xandikos">
<h2>Xandikos</h2>
<p>After attempting to fix a number of issues in Calypso, I kept running into
issues with the way its code is structured. This is only fixable by rewriting
sigifnicant chunks of it, so I opted to instead write a new server.</p>
<p>The goals of <a class="reference external" href="https://www.jelmer.uk/xandikos/">Xandikos</a> are along the same lines as those of Calypso, to be a
simple CalDAV/CardDAV server for personal use:</p>
<ul class="simple">
<li>easy to set up; at the moment, just running
<cite>xandikos -d $<span class="caps">HOME</span>/dav —defaults</cite> is enough to start a new server</li>
<li>use of plain .ics/.ivf files for storage</li>
<li>history stored in Git</li>
</ul>
<p>But additionally:</p>
<ul class="simple">
<li>clear separation between protocol implementation and storage</li>
<li>be well tested</li>
<li>standards complete</li>
<li>standards compliant</li>
</ul>
</div>
<div class="section" id="current-status">
<h2>Current status</h2>
<p>The CalDAV/CardDAV implementation of Xandikos is mostly complete, but there
still a number of <a class="reference external" href="https://github.com/jelmer/xandikos/issues">outstanding issues</a>.</p>
<p>In particular:</p>
<ul class="simple">
<li>lack of authentication support; setting up authentication support in uwsgi or a reverse proxy is one way of working around this</li>
<li>there is no useful <span class="caps">UI</span> for users accessing the <span class="caps">DAV</span> resources via a web browser</li>
<li>test coverage</li>
</ul>
<p>Xandikos has been tested with the following clients:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/pimutils/vdirsyncer">Vdirsyncer</a></li>
<li><a class="reference external" href="https://www.inf-it.com/open-source/clients/caldavzap/">caldavzap</a>/<a class="reference external" href="https://www.inf-it.com/open-source/clients/carddavmate/">carddavmate</a></li>
<li><a class="reference external" href="https://wiki.gnome.org/Apps/Evolution">evolution</a></li>
<li><a class="reference external" href="https://davdroid.bitfire.at/">DAVdroid</a></li>
<li><a class="reference external" href="http://v2.sogo.nu/english/downloads/frontends.html">sogo connector for Icedove/Thunderbird</a></li>
<li><a class="reference external" href="https://play.google.com/store/apps/details?id=de.we.acaldav&hl=en">aCALdav syncer for Android</a></li>
<li><a class="reference external" href="https://github.com/geier/pycarddav">pycardsyncer</a></li>
<li><a class="reference external" href="https://community.kde.org/KDE_PIM/Akonadi">akonadi</a></li>
<li><a class="reference external" href="https://dmfs.org/caldav/">CalDAV-Sync</a></li>
<li><a class="reference external" href="https://dmfs.org/carddav/">CardDAV-Sync</a></li>
</ul>
</div>
<div class="section" id="trying-it">
<h2>Trying it</h2>
<p>To run Xandikos, follow the instructions on the <a class="reference external" href="https://www.jelmer.uk/projects/xandikos/">homepage</a>:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span>./bin/xandikos --defaults -d $HOME/dav
</pre></div></td></tr></table></div>
<p>A server should now be listening on <a class="reference external" href="http://localhost:8080/">localhost:8080</a>
that you can access with your favorite client.</p>
</div>
The Samba Buildfarm2015-02-08T01:06:23+01:002015-02-08T01:06:23+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2015-02-08:samba-buildfarm-history.html<p>Portability has always been very important to Samba.
Nowadays Samba is mostly used on top of Linux, but Tridge developed
the <a class="reference external" href="http://www.rxn.com/services/faq/smb/samba.history.txt">early versions</a> of his <span class="caps">SMB</span> implementation on a Sun workstation.</p>
<p>A few years later, when the project was being picked up, it was ported to Linux
and eventually to a large number of other free and non-free Unix-like operating systems.</p>
<p>Initially regression testing on different platforms was done manually and ad-hoc.</p>
<p>Once Samba had support for a larger number of platforms, including numerous
variations and optional dependencies, making sure that it would still build
and run on all of these became a non-trivial process.</p>
<p>To make it easier to find regressions in the Samba codebase that were
platform-specific, tridge put together a system to automatically build
Samba regularly on as many platforms as possible. So, in Spring 2001, the build
farm was born - this was a couple of years before other tools like buildbot came around.</p>
<div class="section" id="the-build-farm">
<h2>The Build Farm</h2>
<p>The build farm is a collection of machines around the world that are
connected to the internet, with as wide a variety of platforms as possible. In 2001,
it wasn’t feasible to just have a single beefy machine or a cloud account on which
we could run virtual machines with <span class="caps">AIX</span>, <span class="caps">HPUX</span>, Tru64, Solaris and Linux so we needed
access to physical hardware.</p>
<p>The build farm runs as a single non-privileged user, which has a cron job set up that
runs the build farm worker script regularly. Originally the frequency was every
couple of hours, but soon we asked machine owners to run it as often as
possible. The <a class="reference external" href="https://github.com/jelmer/build-farm/blob/master/build_test">worker script</a> is as short as it is simple. It retrieves a
shell script from the main build farm repository with instructions to run and
after it has done so, it uploads a log file of the terminal output to
<tt class="docutils literal">samba.org</tt> using rsync and a secret per-machine password.</p>
<p>Some build farm machines are dedicated, but there have also been a large number of the
years that would just run as a separate user account on a machine that was tasked with
something else. Most build farm machines are hosted by Samba developers (or their employers)
but we’ve also had a number of community volunteers over the years that were happy to add
an extra user with an extra cron job on their machine and for a while companies like
SourceForge and <span class="caps">HP</span> provided dedicated porter boxes that ran the build farm.</p>
<p>Of course, there are some security usses with this way of running things.
Arbitrary shell code is downloaded from a host claiming to be samba.org and
run. If the machine is shared with other (sensitive) processes, some of the
information about those processes might leak into logs.</p>
<p>Our web page has a section about <a class="reference external" href="https://build.samba.org/build.cgi/instructions">adding machines</a> for new volunteers, with a
long list of warnings.</p>
<p>Since then, various other people have been involved in the build farm. Andrew
Bartlett started contributing to the build farm in July 2001, working on
adding tests. He gradually took over as the maintainer in 2002, and various
others (Vance, Martin, Mathieu) have contributed patches and helped out with
general admin.</p>
<p>In 2005, tridge added a script to automatically send out an e-mail to the
committer of the last revision before a failed build. This meant it
was no longer necessary to bisect through build farm logs on the web to find
out who had broken a specific platform when; you’d just be notified as soon as it happened.</p>
</div>
<div class="section" id="the-web-site">
<h2>The web site</h2>
<p>Once the logs are generated and uploaded to samba.org using <tt class="docutils literal">rsync</tt>, the
web site at <a class="reference external" href="http://build.samba.org/">http://build.samba.org/</a> is responsible for making
them accessible to the world. Initially there was a single perl file that would take
care of listing and displaying log files, but over the years the functionality has been
extended to do much more than that.</p>
<p>Initial extensions to the build farm added support for viewing per-compiler and per-host
builds, to allow spotting trends. Another addition was searching logs for common
indicators of running out of disk space.</p>
<p>Over time, we also added more samba.org-projects to the build farm. At the moment
there are about a dozen projects.</p>
<p>In a sprint in 2009, Andrew Bartlett and I changed the build farm to store
machine and build metadata in a SQLite database rather than parsing all recent
build log files every time their results were needed.</p>
<p>In a follow-up sprint a year later, we converted most of the code to Python.
We also added a number of extensions; most notably, linking the build result
information with version control information so we could automatically email
the exact people that had caused the build breakage, and automatically notifying
build farm owners when their machines were not functioning.</p>
</div>
<div class="section" id="autobuild">
<h2>autobuild</h2>
<p>Sometime in 2011 all committers started using the <tt class="docutils literal">autobuild</tt> script to push changes
to the master Samba branch. This script enforces a full build and testsuite run for
each commit that is pushed. If the build or any part of the testsuite fails,
the push is aborted. This alone massively reduced the number of problematic changes
that was pushed, making it less necessary for us to be made aware of issues by
the build farm.</p>
<p>The rewrite also introduced some time bombs into the code. The way we called out to
our <a class="reference external" href="http://storm.canonical.com/"><span class="caps">ORM</span></a> caused the code to fetch all build summary data from the database every
time the summary page was generated. Initially this was not a problem, but as
the table grew to 100,000 rows, the build farm became so slow that it was
frustrating to use.</p>
</div>
<div class="section" id="analysis-tools">
<h2>Analysis tools</h2>
<p>Over the years, various special build farm machines have also been used to run
extra code analysis tools, like static code analysis, lcov, valgrind or
various code quality scanners.</p>
</div>
<div class="section" id="summer-of-code">
<h2>Summer of Code</h2>
<p>Of the last couple of years the build farm has been running happily, and hasn’t
changed much.</p>
<p>This summer one of our summer of code students, Krishna Teja Perannagari,
worked on improving the look of the build farm - updating it to the current
Samba house style - as well as various performance improvements in the Python code.</p>
</div>
<div class="section" id="jenkins">
<h2>Jenkins?</h2>
<p>The build farm still works reasonably well, though it is clear that
various other tools that have had more developer attention have caught up with
it. If we would have to reinvent the build farm today, we would probably end
up using an off-the-shelve tool like Jenkins that wasn’t around 14 years ago.
We would also be able to get away with using virtual machines for most of our workers.</p>
<p>Non-Linux platforms have become less relevant in the last couple of years,
though we still care about them.</p>
<p>The build farm in its current form works well enough for us, and I
think porting to Jenkins - with the same level of platform coverage - would
take quite a lot of work and have only limited benefits.</p>
<p>(Thanks to <a class="reference external" href="https://samba.org/~abartlet/">Andrew Bartlett</a> for proofreading the draft of this post.)</p>
</div>
Autonomous Shard Distributed Databases2014-08-22T20:00:00+02:002014-08-22T20:00:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2014-08-22:autonomous-shard-distributed-databases.html<p>Distributed databases are hard. Distributed databases where you don’t have
full control over what shards run which version of your software are even
harder, because it becomes near impossible to deal with fallout when things go
wrong. For lack of a better term (is there one?), I’ll refer to these databases
as Autonomous Shard Distributed Databases.</p>
<p>Distributed version control systems are an excellent example of such databases.
They store file revisions and commit metadata in shards (“repositories”)
controlled by different people.</p>
<p>Because of the nature of these systems, it is hard to weed out corrupt data
if all shards ignorantly propagate broken data. There will be different people
on different platforms running the database software that manages the
individual shards.</p>
<p>This makes it hard - if not impossible - to deploy software updates to all
shards of a database in a reasonable amount of time (though a Chrome-like
update mechanism might help here, if that was acceptable). This has
consequences for the way in which you have to deal with every change to the
database format and model.</p>
<p>(e.g. imagine introducing a modification to the Linux kernel Git repository that
required everybody to install a new version of Git).</p>
<p>Defensive programming and a good format design from the start are essential.</p>
<p>Git and its database format do really well in all of these regards. As I wrote <a class="reference external" href="https://www.jelmer.uk/pages/bzr-a-restrospective.html">in my
retrospective</a>, Bazaar has made a number of mistakes in this area, and that
was a major source of user frustration.</p>
<p>I propose that every autonomous shard distributed databases should aim for the following:</p>
<blockquote>
<ul>
<li><p class="first">For the “base” format, keep it as simple as you possibly can. (<span class="caps">KISS</span>)</p>
<p>The simpler the format, the smaller the chance of mistakes in the design
that have to be corrected later. Similarly, it reduces the chances
of mistakes in any implementation(s).</p>
<p>In particular, there is no need for every piece of metadata to be
a part of the core database format.</p>
<p>(in the case of Git, I would argue that e.g. “author” might as well be a “meta-header”)</p>
</li>
<li><p class="first">Corruption should be detected early and not propagated. This means
there should be good tools to sanity check a database, and ideally
some of these checks should be run automatically during everyday operations
- e.g. when pushing changes to others or receiving them.</p>
</li>
<li><p class="first">If corruption does occur, there should be a way for as much of the database
as possible to be recovered.</p>
<p>A couple of corrupt objects should not render the entire database unusable.</p>
<p>There should be tools for low-level access of the database, but
the format and structure should be also documented well enough for power users to
understand it, examine and extract data.</p>
</li>
<li><p class="first">No “hard” format changes (where clients /have/ to upgrade to access the new format).</p>
<p>Not all users will instantly update to the latest and greatest version of the
software. The lifecycle of enterprise Linux distributions is long enough
that it might take three or four years for the majority of users to upgrade.</p>
</li>
<li><p class="first">Keep performance data like indexes in separate files. This makes it possible for
older software to still read the data, albeit at a slower pace, and/or generate
older format index files.</p>
</li>
<li><p class="first">New shards of the database should replicate the entire database if at all possible;
having more copies of the data can’t hurt if other shards go away or
get corrupted.</p>
<p>Having the data locally available also means users get quicker access to
more data.</p>
</li>
<li><p class="first">Extensions to the database format that require hard format changes (think
e.g. submodules) should only impact databases that actually use those extensions.</p>
</li>
<li><p class="first">Leave some room for structured arbitrary metadata, which gets propagated but
that not all clients need to be able to understand and can safely ignore.</p>
<p>(think fields like “Signed-Off-By”, “Reviewed-By”, “Fixes-Bug”, etc) in
git commit metadata headers, or the revision metadata fields in Bazaar.</p>
</li>
</ul>
</blockquote>
Using Propellor for configuration management2014-08-18T23:15:00+02:002014-08-18T23:15:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2014-08-18:propellor.html<p>For a while, I’ve been wanting to set up configuration management for my home network.
With half a dozen servers, a <span class="caps">VPS</span> and a workstation it is not big, but large
enough to make it annoying to manually log into each machine for network-wide changes.</p>
<p>Most of the servers I have are low-end <span class="caps">ARM</span> machines, each responsible for a
couple of tasks. Most of my machines run Debian or something derived from Debian.
Oh, and I’m a member of the declarative school of configuration management.</p>
<div class="section" id="propellor">
<h2>Propellor</h2>
<p><a class="reference external" href="http://joeyh.name/blog/entry/propellor/">Propellor</a> caught my eye earlier this year. Unlike some other configuration
management tools, it doesn’t come with its own custom language but it is
written in Haskell, which I am already familiar with. It’s also fairly simple,
declarative, and seems to do most of the handful of things that I need.</p>
<p>Propellor is essentially a Haskell application that you customize for your site.
It works very similar to e.g. xmonad, where you write a bit of Haskell code
for configuration which uses the upstream library code. When you run the
application it takes your code and builds a binary from your code and
the upstream libraries.</p>
<p>Each host on which Propellor is used keeps a clone of the site-local Propellor git
repository in <tt class="docutils literal">/usr/local/propellor</tt>. Every time propellor runs
(either because of a manual “spin”, or from a cronjob it can set up for you), it
fetches updates from the main site-local git repository, compiles the Haskell
application and runs it.</p>
</div>
<div class="section" id="setup">
<h2>Setup</h2>
<p>Propellor was surprisingly easy to set up. Running <tt class="docutils literal">propellor</tt>
creates a clone of the upstream repository under <tt class="docutils literal"><span class="pre">~/.propellor</span></tt> with a <span class="caps">README</span>
file and some example configuration. I copied <tt class="docutils literal"><span class="pre">config-simple.hs</span></tt> to
<tt class="docutils literal">config.hs</tt>, updated it to reflect one of my hosts and within a few minutes I
had a basic working propellor setup.</p>
<p>You can use <tt class="docutils literal">./propellor <host></tt> to trigger a run on a remote host.</p>
<p>At the moment I have propellor working for some basic things -
having certain Debian packages installed, a specific network configuration, mail
setup, basic Kerberos configuration and certain <span class="caps">SSH</span> options set.
This took surprisingly little time to set up, and it’s been great being able
to take full advantage of Haskell.</p>
<p>Propellor comes with convenience functions for dealing with some commonly used
packages, such as Apt, <span class="caps">SSH</span> and Postfix. For a lot of the other packages, you’ll
have to roll your own for now. I’ve written some extra code to make Propellor deal
with Kerberos keytabs and Dovecot, which I hope to submit upstream.</p>
<p>I don’t have a lot of experience with other Free Software configuration
management tools such as Puppet and Chef, but for my use case Propellor works
very well.</p>
<p>The main disadvantage of propellor for me so far is that it needs to build
itself on each machine it runs on. This is fine for my workstation and
high-end servers, but it is somewhat more problematic on e.g. my Raspberry Pi’s.
Compilation takes a while, and the Haskell compiler and libraries it needs amount
to 500Mb worth of disk space on the tiny root partition.</p>
<p>In order to work with Propellor, some Haskell knowledge is required.
The Haskell in the configuration file is reasonably easy to understand if you
keep it simple, but once the compiler spits out error messages then I suspect
you’ll have a hard time without any Haskell knowledge.</p>
<p>Propellor relies on having a central repository with the configuration that it
can pull from as root. Unlike Joey, I am wary of publishing the configuration
of my home network and I don’t have a highly available local git server setup.</p>
</div>
The state of distributed bug trackers2013-11-10T19:23:00+01:002013-11-10T19:23:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-11-10:distributed-bug-trackers.html<p>A whopping 5 years ago, <span class="caps">LWN</span> ran <a class="reference external" href="https://lwn.net/Articles/281849/">a story</a> about distributed bug trackers. This
was during the early waves of distributed version control adoption, and so
everybody was looking for other things that could benefit from decentralization.</p>
<p><span class="caps">TL</span>;<span class="caps">DR</span>: Not much has changed since.</p>
<p>The potential benefits of a distributed bug tracker are similar to those of
a distributed version control system: ability to fork any arbitrary project,
easier collaboration between related projects and offline access to full project data.</p>
<p>The article discussed a number of systems, including <a class="reference external" href="http://www.bugseverywhere.org/">Bugs Everywhere</a>,
<a class="reference external" href="http://www.mkgnu.net/?q=scmbug">ScmBug</a>, <a class="reference external" href="http://www.distract.wellquite.org/">DisTract</a>, <a class="reference external" href="http://www.ditrack.org/">DITrack</a>, <a class="reference external" href="https://github.com/schacon/ticgit/wiki">ticgit</a> and <a class="reference external" href="http://ditz.rubyforge.org/">ditz</a>. The conclusion of
our favorite grumpy editor at the time was that all of the available
distributed bug trackers were still in their infancy.</p>
<p>All of these piggyback on a version control system somehow - either by reusing
the <span class="caps">VCS</span> database, by storing their data along with the source code in the tree,
or by adding custom hooks that communicate with a central server.</p>
<p>Only <a class="reference external" href="http://www.mkgnu.net/?q=scmbug">ScmBug</a> had been somewhat widely deployed at the time, but its homepage
gives me a blank page now. Of the trackers reviewed by <span class="caps">LWN</span>, <a class="reference external" href="http://www.bugseverywhere.org/">Bugs Everywhere</a>
is the only one that is still around and somewhat active today.</p>
<p>In the years since the article, a handful of new trackers have come along. Two new
version control systems - <a class="reference external" href="http://veracity-scm.com/compare/">Veracity</a> and <a class="reference external" href="http://www.fossil-scm.org/">Fossil</a> - come with the kitchen
sink included and so feature a built-in bug tracker and wiki.</p>
<p>There is an extension for Mercurial called <a class="reference external" href="http://mercurial.selenic.com/wiki/ArtemisExtension">Artemis</a> that stores issues in an
<tt class="docutils literal">.issues</tt> directory that is colocated with the Mercurial repository.</p>
<p>The other new tracker that I could find (though it has also not changed since
2009) is <a class="reference external" href="http://syncwith.us/sd/"><span class="caps">SD</span></a>. It uses its own distributed database technology for storing bug
data - called <em>Prophet</em>, and doesn’t rely on a <span class="caps">VCS</span>. One of the nice features is
that it supports importing bugs from foreign trackers.</p>
<p>Some of these provide the benefits you would expect of a distributed bug
tracker. Unfortunately, all those I’ve looked at fail to even provide the basic
functionality I would want in a bug tracker. Moreso than with a version control system,
regular users interact with a bug tracker. They report bugs, provide comments
and feedback on fixes. All of the systems I tried make these actions a lot
harder than with your average bugzilla or mantis instance - they provide a
limited web <span class="caps">UI</span> or no web interface at all.</p>
<p><em>Update</em>: <span class="caps">LWN</span> later also published articles <a class="reference external" href="https://lwn.net/Articles/434922/">on <span class="caps">SD</span></a> and <a class="reference external" href="https://lwn.net/Articles/433843/">on Fossil</a>. Other
interesting links are <a class="reference external" href="http://www.ericsink.com/entries/dbts_fossil.html">Eric Sink’s article on distributed bug tracking</a> (Erik
works at Sourcegear who develop Veracity) and the <a class="reference external" href="http://dist-bugs.branchable.com/">dist-bugs mailing list</a>.</p>
Quantified Self2013-11-10T02:05:00+01:002013-11-10T02:05:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-11-10:quantified-self.html<p>Dear lazyweb,</p>
<p>I’ve been reading about what the rest of the world seems to be calling
“quantified self”. In essence, it is tracking of personal data like activity,
usually with the goal of data-driven decision making. Or to take a less
abstract common example: counting the number of steps you take each day to
motivate yourself to take more. I wish it’d been given a less annoying woolly
name but this one seems to have stuck.</p>
<p>There are a couple of interesting devices available that track sleep, activity
and overall health. Probably best known are the FitBit and the
jazzed-up armband pedometers like the Jawbone <span class="caps">UP</span> and the Nike Fuelband.
Unfortunately all existing devices seem to integrate with cloud services
somehow, rather than giving the user direct access to their data. Apart from
the usual privacy concerns, this means that it is hard to do your own data
crunching or create a dashboard that contains data from multiple sources.</p>
<p>Has anybody found any devices that don’t integrate with the cloud and
just provide raw data access?</p>
Porcelain in Dulwich2013-10-04T00:00:00+02:002013-10-04T00:00:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-10-04:dulwich-porcelain.html<p><span class="dquo">“</span>porcelain” is the term that is usually used in the Git world to refer to
the user-facing parts. This is opposed to the lower layers: the plumbing.</p>
<p>For a long time, I have resisted the idea of including a porcelain layer in
Dulwich. The main reason for this is that I don’t consider Dulwich a
full reimplementation of Git in Python. Rather, it’s a library that Python
tools can use to interact with local or remote Git repositories, without
any extra dependencies.</p>
<p>dulwich has always shipped a ‘dulwich’ binary, but that’s never been more
than a basic test tool - never a proper tool for end users. It was a
mistake to install it by default.</p>
<p>I don’t think there’s a point in providing a dulwich command-line tool that has
the same behaviour as the C Git binary. It would just be slower and less
mature. I haven’t come across any situation where it didn’t make sense to
just directly use the plumbing.</p>
<p>However, Python programmers using Dulwich seem to think of Git operations in
terms of porcelain rather than plumbing. Several convenience wrappers for Dulwich
have sprung up, but none of them is very complete. So rather than relying on external
modules, I’ve added a “porcelain” module to Dulwich in the <a class="reference external" href="https://github.com/jelmer/dulwich/tree/porcelain">porcelain</a>
branch, which provides a porcelain-like Python <span class="caps">API</span> for Git.</p>
<p>At the moment, it just implements a handful of commands but that should improve
over the next few releases:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">dulwich</span> <span class="kn">import</span> <span class="n">porcelain</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">porcelain</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="s2">"/path/to/repo"</span><span class="p">)</span>
<span class="n">porcelain</span><span class="o">.</span><span class="n">commit</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="s2">"Create a commit"</span><span class="p">)</span>
<span class="n">porcelain</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">r</span><span class="p">)</span>
</pre></div>
Book Review: Bazaar Version Control2013-09-28T19:33:00+02:002013-09-28T19:33:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-09-28:bzr-book.html<p><a class="reference external" href="http://www.packtpub.com">Packt</a> recently published a book on <a class="reference external" href="http://www.packtpub.com/bazaar-version-control/book">Version Control using Bazaar</a> written by
<a class="reference external" href="http://www.janosgyerik.com/">Janos Gyerik</a>. I was curious what the book was like, and they kindly
provided me with a digital copy.</p>
<p>The book is split into roughly five sections: an introduction to version
control using Bazaar’s main commands, an overview of the available workflows,
some chapters on the available extensions and integration, some more advanced topics
and finally, a quick introduction to programming using bzrlib.</p>
<p>It is assumed the reader has no pre-existing knowledge about version control
systems. The first chapters introduce the reader to the concept of revision
history, branching and merging and finally collaboration. All concepts are
first discussed in theory, and then demonstrated using the Bazaar command-line
<span class="caps">UI</span> and the bzr-explorer tool. The book follows roughly the same
track as the official documentation, but it is more extensive
and has more fancy drawings of revision graphs.</p>
<p>The middle section of the book discusses the modes in which Bazaar
can be used - centralized or decentralized - as well as
the various ways in which code can be landed in the main branch (“workflows”).
The selection of workflows in the book is roughly the same as those in the
official Bazaar documentation. The author briefly touches on a number of other
software engineering topics such as code reviews, code formatting and automated
testing, though not sufficiently to make it useful for people who are
unfamiliar with these techniques. Both the official documentation and the book
complicate things unnecessarily by listing every possible option.</p>
<p>The next chapter is a basic howto on the use of Bazaar with various hosting
solutions, such as Launchpad, Redmine and Trac.</p>
<p>The <tt class="docutils literal">Advanced Features</tt> chapter covers a wide range of obscure and less obscure
features in Bazaar: <tt class="docutils literal">uncommit</tt>, shelves, re-using working trees, lightweight
checkouts, stacked branches, signing revisions and using e-mail hooks.</p>
<p>The chapter on foreign version control system integration is a more extensive
version of the public docs. It has some factual inaccuracies; in particular, it
recommends the installation of a 2 year old buggy version of bzr-git.</p>
<p>The last chapter provides quite a good introduction to the Bazaar APIs and
plugin writing. It is a fair bit better than what is available publically.</p>
<p>Overall, it’s not a bad book but also not a huge step forward from the official
documentation. I might recommend it to people who are interested in
learning Bazaar and who do not have any experience with version control yet.
Those who are already familiar with Bazaar or another version control system
will not find much new.</p>
<p>The book misses an opportunity by following the official documentation so
closely. It has the same omissions and the same overemphasis on describing
every possible feature. I had hoped to read more about Bazaar’s data model, its
file format and some of the common problems, such as parallel imports, format
hell and slowness.</p>
<!-- I have reported in the order of 50 minor errors in the book to the publisher. -->
<!-- issues:
page 24-27: Not sure if I would consider bzr-explorer mature enough to be featured this prominently in the first chapter
page 78: fails to mention how tags are propagated, that they are not versioned and what the problem is with that
page 86-97: confusing how numbers are used for revisions that are not related to the revision numbers bazaar assigns to revisions itself. it'd be better to use letters.
page 103: not sure why it's necessary to cover three different languages?
page 125: mentioning different merge types isn't particularly useful unless you explain how they are different; just trying all of them every time you hit conflicts is a waste of time (and a potential way to lose changes)
page 130: topics such as merging multiple branches seem inappropriate at this point
page 185: this bit of bzr+ssh configuration is not specific to centralized mode, and seems out of place for this chapter (I'd expect it in the previous chapter) -->
<!-- inaccuracies:
page 16: Bazaar is licensed under the GPL, version 2 or later - not just version 2.
page 16: OpenStack no longer uses bzr
page 16: there is nothing "official" about Launchpad, just that the majority of FOSS projects in bzr and bzr itself are hosted there.
page 18,21: pip is not a recommended way for installing bzr and bzr plugins so I'm surprised to see it suggested here; at least for bzr-gtk, bzr-git, bzr-svn, bzr-rewrite, bzr-search, etc I do not keep it up to date and never test it. I suspect that's true for most other plugins as well.
page 30: fails to explain why it's called "committing", as in "committing a change"
page 32: doesn't explain properly what it means for a revision to be associated with a working tree, how it's the base for the revision the working tree is used to prepare. the .bzr directory of a working tree contains the metadata for the working tree and either the associated branch itself or a reference to it. it doesn't contain the full contents of the base revision
page 44: un-versioned and unknown are not the same thing. a file can be unversioned and known to bazaar but e.g. ignored. I have never seen the term non-versioned.
page 63: incorrect typeface for "maps/ directory", probably should apply to just "maps/"
page 67: 'bzr mv - -auto' works for single renames too, and is almost always preferable to 'bzr mv - -after'.
page 82: it seems a bit odd to mention "bzr push" here but not any of the other propagation commands
page 83: the reference to PCs is strange, considering Bazaar also runs on Macs
page 86: the diagram has two 6'' revisions (seems like it one should be 6''')
page 128: the explanation of cherry-picking is confusing
page 132: "mirrorring" is an incorrect term. The push and pull command propagate revisions only; they're not mirrorring anything else about the branch, e.g. branch config or stacking setup. THey also won't result in a mirror the tip if the target is ahead of the source.
page 139: protocols are not servers; servers implement or provide protocols
page 146: the bzr smart server protocol *is* available over HTTP, but it needs to be explicitly enabled on the server side.
page 152: bzr doesn't create working trees for remote branches by default (and will in fact complain that it can't update them if they exist); - -no-tree only makes sense for branches you create locally
page 161: the fact that the history in the branches is different doesn't have anything to do with the revision numbering - the ancestry graph is different and the revision numbering is inferred from the ancestry graph
page 164: confusing comparison of "code reviews" to the term "peer review", which is otherwise undefined. I only know "peer review" as the review of a peer's performance in personnel performance evaluation - not in terms of code review.
page 164: "code review" doesn't require another team member to do the eventual push, it could still be the original author after (or even before) approval. This is the case with bzr itself and various bzr plugins.
page 166: there is no such thing as a "default mode" in bzr; the distributed workflow is significantly more commonly used, but there is nothing default about it
page 171: the centralized workflow may be widely used with version control systems in general (such as CVS or Subversion), but there are not a lot of bzr users who use it
page 172: bound branches aren't necessary for the centralized model; see e.g. lightweight checkouts (which are probably the most common way of working with centralized mode)
page 186: same goes for using bzr server - it's not specific to centralized mode
page 206: how is using a bzr hosting site better? I'm not convinced that it is, and there are no supporting arguments here.
page 206: are there actually any other hosting sites but launchpad that do merge proposals? I'm not aware of any
page 213: code review was explained earlier, and this conflates the tasks of the gatekeeper and code review
page 229: OpenStack is no longer using bzr, and zope never officially used bzr
page 282: the switch command *is* a core feature, but the bzr-loom plugin overrides the default implementation
page 288: whether or not the passphrase has to be reentered depends on the users' local setup
page 296: it's possible to create a local clone of a remote repository in the foreign format; the *default* is to use the latest bzr format for the local sprout
page 296: bazaar doesn't necessarily preserve all metadata; there are various situations in which it doesn't, e.g. when branching from svn branches
page 304: pip is not the recommended way of installing bzr-svn
page 304: bzr-svn does *not* preserve file properties like svn:mime-type and only preserves svn:mergeinfo if it is set on the branch root
page 304: svn has no concept of file ids that is exposed to the outside world; bzr-svn does not preserve revision IDs or file ids
page 305: the revision ID shown here isn't preserved - it's autogenerated deterministically; there is no concept of file IDs and revision IDs in SVN; file and revision IDs are preserved when pushing from bzr *into* svn with bzr-svn. The point here is that this revision ID is deterministic - it is predictable so two clones of the same SVN repo result in the exact same bzr repo.
page 306: the equivalent of 'bzr commit' is 'bzr commit - -lossy'; the equivalent of 'bzr push' is 'bzr dpush'
page 315: pip is not the recommended way of installing bzr-git
page 315: bzr-git-1480 is an outdated copy of bzr-git which contains bugs that have since been fixed in trunk; it also appears to be uninstallable now
page 316: ssh://user@server:path/to/repo.git is not a valid URL in git either; it should be ssh://user@server/path/to/repo.git
page 316: the relative path to the users' home directory does work in bzr-git versions from 2012 and later
page 317: the example uses the 'branches' command but doesn't explain it
page 322: "bzr dpush" also changes the local branch; possibly its most dangerous side-effect
page 328: the bzr-fastimport plugin no longer ships exporters for other version control tools (hasn't since mid 2012)
page 337: the repository instance can be anything as long as it's derived from the Repository class, it doesn't have to be a CHKInventoryRepository instance
page 337: control_url doesn't have to refer to a .bzr/branch directory though it will for most native local branches
page 357: the bzr_* variables mentioned here are not used by anything; they were never more than a recommendation and setting them won't give you anything. -->
Migrating packaging from Bazaar to Git2013-06-02T02:01:00+02:002013-06-02T02:01:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-06-02:pristine-tar-migrate-bzr-to-git.html<p>A while ago I migrated most of my packages from Bazaar to Git. The rest of the
world has decided to use Git for version control, and I don’t have enough
reason to stubbornly stick with Bazaar and make it harder for myself to
collaborate with others.</p>
<p>So I’m moving away from a workflow I know and have polished over the last few
years - including the various bzr plugins and other tools involved. Trying to
do the same thing using git is frustrating and time-consuming, but I’m sure
that will improve with time. In particular, I haven’t found a good way to merge
in a new upstream release (from a tarball) while referencing the relevant
upstream commits, like <tt class="docutils literal">bzr <span class="pre">merge-upstream</span></tt> can. Is there a good way to do this?
What helper tools can you recommend for maintaining a Debian package in git?</p>
<p>Having been upstream for <tt class="docutils literal"><span class="pre">bzr-git</span></tt> earlier, I used its <tt class="docutils literal"><span class="pre">git-remote-bzr</span></tt>
implementation to do the conversions of the commits and tags:</p>
<pre class="literal-block">
% git clone bzr::/path/to/bzr/foo.bzr /path/to/git/foo.git
</pre>
<p>One of my last contributions to bzr-git was a <tt class="docutils literal">bzr <span class="pre">git-push-pristine-tar-deltas</span></tt>
subcommand, which will export all bzr-builddeb-style pristine-tar metadata
to a pristine-tar branch in a Git repository that can be used by
<em>pristine-tar</em> directly or through something like <em>git-buildpackage</em>.</p>
<p>Once you have created a git clone of your bzr branch, it should be a matter of
running <tt class="docutils literal">bzr <span class="pre">git-push-pristine-tar-deltas</span></tt> with the target git repository
and the Debian package name:</p>
<pre class="literal-block">
% cd /path/to/bzr/foo.bzr
% bzr git-push-pristine-tar-deltas /path/to/git/foo.git foo
% cd /path/to/git/foo.git foo
% git branch
* master
pristine-tar
</pre>
Kiln using Dulwich2013-03-21T00:00:00+01:002013-03-21T00:00:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-03-21:kiln-dulwich.html<p><a class="reference external" href="http://blog.fogcreek.com/kiln-harmony-internals-the-basics/">http://blog.fogcreek.com/kiln-harmony-internals-the-basics/</a></p>
<p>Nice to see that #Kiln is also using #Dulwich for some of its Git support. Unfortunately I haven’t been able to spend as much time on it recently as it deserves (busy changing jobs and countries), but hopefully that will change soon.</p>
OpenChange 2.0 released2013-02-08T20:00:00+01:002013-02-08T20:00:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2013-02-08:openchange-20-released.html<p>Apparently ‘tis the season for major software releases.</p>
<p><a class="reference external" href="https://twitter.com/jkerihuel">Julien</a> has just announced the release of <a class="reference external" href="http://www.openchange.org/">OpenChange</a> 2.0, codenamed
<a class="reference external" href="http://en.memory-alpha.org/wiki/Quadrant">quadrant</a>. This release fixes a number of important bugs and enables
integration with <a class="reference external" href="http://www.sogo.nu/english.html">SOGo</a>.</p>
<p>With the SOGo backend, it is now possible to set up an Exchange-compatible
groupware server that can be accessed from Outlook without the need to
connect any connectors.</p>
<p>See the <a class="reference external" href="http://openchange.org/developers/relnotes/2.0-quadrant.html">release notes</a> for more details.</p>
Bazaar: A retrospective2012-12-19T22:36:43+01:002012-12-19T22:36:43+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-12-19:bzr-retrospective.html<p>For the last 7 years I’ve been involved in the <a class="reference external" href="http://bazaar-vcs.org/">Bazaar</a> project. Since
I am slowly stepping down, I recently wrote <a class="reference external" href="/pages/bzr-a-retrospective.html">a retrospective</a> on the
project as I experienced it for the last 7 years.</p>
<p>Thanks to a few kind people for proofreading earlier drafts; if you spot any
errors, please let me know in the comments.</p>
Samba 4.0.0, finally2012-12-11T18:00:00+01:002012-12-11T18:00:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-12-11:samba-4.0.0.html<p>This afternoon <a class="reference external" href="https://www.samba.org/samba/news/releases/4.0.0.html">we released version 4.0.0</a> of Samba. This is a significant
milestone, and I’m very proud of the result. Samba 4 is the first version
that can be a domain controller in an Active Directory domain.</p>
<p>We embarked on this journey almost a decade ago - <a class="reference external" href="https://git.samba.org/?p=samba.git;a=commit;h=b0510b5428b3461aeb9bbe3cc95f62fc73e2b97f">the first commit</a> is from
August 2003. It’s been a long and bumpy ride. I hardly recognize
the people in this <a class="reference external" href="https://www.samba.org/samba/images/team_l2003.jpg">team photo from 2003</a> (I’m second from the left).</p>
<p>A lot has happened in that time. We wrote <a class="reference external" href="https://www.ohloh.net/p/samba/contributors/summary">a few million lines of code</a>. We migrated from <span class="caps">CVS</span> to Subversion to Git. We’ve drifted apart and grown back together as a <a class="reference external" href="https://www.samba.org/samba/team/">team</a>.</p>
<p>In my youthful naivity I predicted a release “within 1 or 2
years” during <a class="reference external" href="http://www.samba.org/~jelmer/jelmer-nluug-vj04.pdf">a talk at the <span class="caps">NLUUG</span></a> in 2004. But Active Directory was a lot
harder than we thought, and there were quite a few other distractions as well.
I’m glad this release, which is by far the biggest and longest running software
project I have ever worked on, has finally happened.</p>
<p>Some older RCs of Samba 4 have already been packaged for Debian and Ubuntu,
in the <tt class="docutils literal">samba4</tt> source package. For Debian jessie, these will be integrated
into the main <tt class="docutils literal">samba</tt> source package. Please use <tt class="docutils literal">experimental</tt> if you do
want to try the existing packages, as it is most up to date.</p>
Documentation2012-12-01T00:00:00+01:002012-12-01T00:00:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-12-01:documentation.html<p>From <span class="caps">LWN</span>’s weekly edition:</p>
<blockquote>
<blockquote>
Documentation is the sort of thing that will never be great unless someone from outside contributes it (since the developers can never remember which parts are hard to understand).</blockquote>
<p class="attribution">—<a class="reference external" href="https://github.com/apenwarr/bup">Avery Pennarun</a></p>
</blockquote>
Back to blogging2012-11-26T03:17:00+01:002012-11-26T03:17:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-11-26:251-back-to-blogging.html<p>Hello Internet. After a long silence and several fights with <a class="reference external" href="http://www.s9y.org/">Serendipity</a>
I am back.</p>
<p>The contents from my old <cite>Serendipity</cite> install have been migrated to
restructuredText in <a class="reference external" href="http://www.getpelican.com/">pelican</a>. Among other things, this means I can
now get rid of the last <span class="caps">PHP</span> install I had left on my server.</p>
Last day at Canonical2012-10-17T00:00:00+02:002012-10-17T00:00:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2012-10-17:last-day-at-canonical.html<p>This Friday will be my last day at <a class="reference external" href="http://www.canonical.com/">Canonical</a>.</p>
<p>It has been a lot of fun working here. There is so much I have learned in
the last three years. I’m going to miss my colleagues.</p>
<p>Over the last couple of months I have slowly stepped down from my involvement
in Bazaar and the Bazaar packaging in Debian and Ubuntu.
I would like to stay involved in Ubuntu, but we will see how that goes.</p>
<p>I’m taking some time off until the end of the year to see the world and
hack, before starting something new in February.</p>
Summer of Code 20112011-07-18T08:53:05+02:002011-07-18T08:53:05+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2011-07-18:272-Summer-of-Code-2011.html<p>The Samba team is once again participating in the Summer of Code this year.
This year we have 4 students working on various projects related to Samba.</p>
<p>This year I am mentoring Dhananjay Sathe, who is improving the
<a class="reference external" href="http://wiki.samba.org/index.php/SambaGtk"><span class="caps">GTK</span>+ frontends for Samba</a>. In
particular, he is making it possible to manage shares and users of a remote
Samba or Windows machine.</p>
<p>Dhananjay is also <a class="reference external" href="http://dsathe.blogspot.com/2011/07/samba-with-soc-google-summer-of-code.html">blogging about his progress</a>.</p>
libapache2-mod-bzr2011-01-02T19:07:53+01:002011-01-02T19:07:53+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2011-01-02:269-libapache2-mod-bzr.html<p>During the last two days I hacked together a <a class="reference external" href="http://bazaar.canonical.com/">Bazaar</a> module for <a class="reference external" href="http://www.apache.org/">Apache</a>.
This module makes it possible to easily enable the Bazaar smart server for
Bazaar branches. It also can display a simple placeholder page for Bazaar
branches without working tree. It’s surprisingly easy to write Apache modules.</p>
<p>The main advantage this has over a mod_wsgi / mod_python / mod_fcgi setup is that it doesn’t require any additional Python hacking on the users side or other configuration outside of Apache, and it doesn’t require configuration for each single branch in the Apache configuration. In the future I’d also like to support the settings “BazaarFrontend <a class="reference external" href="https://launchpad.net/wikkid">Wikkid</a>” and “BazaarFrontend <a class="reference external" href="https://launchpad.net/loggerhead">Loggerhead</a>“.</p>
<p>The configuration is currently as simple as:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="nb">LoadModule</span><span class="w"> </span>bzr_module<span class="w"> </span><span class="sx">/usr/lib/apache2/modules/mod_bzr.so</span>
<span class="nb">BazaarSmart</span><span class="w"> </span><span class="k">on</span>
<span class="nb">BazaarFrontend</span><span class="w"> </span>Basic
</pre></div></td></tr></table></div>
<p>in your <strong>apache2.conf</strong>. The <strong>BazaarSmart</strong> and <strong>BazaarFrontend</strong> directives can appear in <Directory> or <Location> clauses as well, if you’d like to have different behaviour for different directories.</p>
<p>At the moment this project is a proof of concept, and probably not something
you would want to run in production. For example, there is no way to limit the
access to a branch to read only. I need to double-check there are no threading issues.</p>
<p>Testing and patches are welcome. The project is hosted here:</p>
<ul class="simple">
<li><a class="reference external" href="https://launchpad.net/apache-bzr">https://launchpad.net/apache-bzr</a></li>
</ul>
<p><em>Currently Playing: Stream of Passion - Calliopeia</em></p>
On the way to Samba 4: Part 22011-01-02T16:30:13+01:002011-01-02T16:30:13+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2011-01-02:268-On-the-way-to-Samba-4-Part-2.html<p>It’s been more than a month since the last status update on my Samba 4 work -
much more than the two weeks I promised.</p>
<p>During the holidays I finally managed to release the <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Releases/4.0.0alpha14">new alpha of Samba 4</a>,
as well as releases of some of our companion libraries (<a class="reference external" href="http://tdb.samba.org/">tdb</a>, <a class="reference external" href="http://talloc.samba.org/">talloc</a>,
tevent and <a class="reference external" href="http://ldb.samba.org/">ldb</a>). The release includes a significant amount of bug fixes and
a lot of work towards a properly functioning Active Directory <span class="caps">DC</span>, too much to
list here.</p>
<p>This release I’ve mainly been involved in improving our Python bindings and our
handling of internal and external libraries. We now use symbol versioning for
our copy of Heimdal Kerberos as well as some of our other libraries. Hopefully
this will fix some of the issues users of the evolution-mapi plugin have been
seeing where they end up with both <span class="caps">MIT</span> Kerberos and Heimdal Kerberos loaded
into the same process (with all the consequences of overlapping symbol names).
Samba 4 now also has the ability to work with the system Heimdal rather than
using the bundled copy. I have packaged alpha14 for Debian and Ubuntu (fixing
most of the open bugs against the Samba 4 package in the <span class="caps">BTS</span>), but am currently
waiting for the new release of ldb to pass through <span class="caps">NEW</span> before I can upload.</p>
<p>The next release is scheduled for the first week of February.</p>
<p><em>Currently Playing: Stream of Passion - Haunted</em></p>
On the way to Samba 4: Part 12010-11-24T20:44:00+01:002010-11-24T20:44:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-11-24:267-On-the-way-to-Samba-4.html<p>After <a class="reference external" href="http://www.sambaxp.org/">Samba <span class="caps">XP</span> 2008</a> <a class="reference external" href="http://samba.org/~abartlet/">Andrew</a> and I started keeping a wiki page with our bi-weekly goals and achievements for Samba 4. Because planning in a Free Software project is hard (time availability and priorities change over time, and other volunteers are equally unpredictable) we called this our <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Andrew_and_Jelmers_Fantasy_Page">“Fantasy Page”</a>; it listed things we wanted to work on next (“fantasies”), but reality being what it is we would usually actually end up working on something entirely different. We discussed our progress and new plans in - what I would now call - a bi-weekly standup call.</p>
<p>There were several reasons for doing this. It gave us some sense of direction as well as a sense of accomplishment; a way to look back at the end of the year and realize how much we had actually achieved. Because Samba 4 is such a long term project (it is 7 years old at this point) it is easy to become disillusioned, to look back at a year of commits and to not see the gradual improvement, just the fact that there is no release yet.</p>
<p>We managed to keep this up for <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Andrew_and_Jelmers_Fantasy_Page/2008">two</a> <a class="reference external" href="http://wiki.samba.org/index.php/Samba4/Andrew_and_Jelmers_Fantasy_Page/2009">years</a>, much longer than I had anticipated, and eventually started to slip last year.</p>
<p>More recently <a class="reference external" href="http://www.kblin.org">Kai</a> and <a class="reference external" href="http://blog.tridgell.net/">Tridge</a> have started to blog weekly about their efforts to make Samba 4.0 a reality and I’m going to join them by trying to blog regularly - every two weeks - about my contributions, even if there were none.</p>
<p>In the next two weeks I plan to work on finally getting alpha 14 of Samba 4 out and on fixing the daily builds of Samba 4 and OpenChange for Ubuntu on Launchpad after we did a massive reorganization of the private libraries in Samba 4.</p>
<p><em>Current Playing: Zero 7 - Somersault</em></p>
Mumble and bluetooth2010-11-07T21:38:35+01:002010-11-07T21:38:35+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-11-07:266-Mumble-and-bluetooth.html<p><a class="reference external" href="http://mumble.sourceforge.net/">Mumble</a> is an open source, low-latency, high quality voice chat application that we’re using at Canonical, and which Samba has recently also adopted.</p>
<p>After I busted the cable of my cabled headset and inspired by <a class="reference external" href="http://mdzlog.alcor.net/2010/05/26/using-mumble-with-a-bluetooth-headset/">Matt’s post about Mumble and Bluetooth</a> I bought a cheap Nokia Bluetooth headset that works with my laptop as well as my phone. By using <a class="reference external" href="http://blueman-project.org/">BlueMan</a> I even found that the headset worked out of the box on Maverick.</p>
<p>A nice feature in Mumble is that it is controllable using D-Bus. Blueman supports running an arbitrary command when the answer button is pressed. Combining these two features, it is possible to automatically toggle mute when the answer button is pressed:</p>
<pre class="code python literal-block">
<span class="ch">#!/usr/bin/python</span><span class="w">
</span><span class="kn">import</span> <span class="nn">dbus</span><span class="w">
</span><span class="n">bus</span> <span class="o">=</span> <span class="n">dbus</span><span class="o">.</span><span class="n">SessionBus</span><span class="p">()</span><span class="w">
</span><span class="n">mumble</span> <span class="o">=</span> <span class="n">bus</span><span class="o">.</span><span class="n">get_object</span><span class="p">(</span><span class="s2">"net.sourceforge.mumble.mumble"</span><span class="p">,</span> <span class="s2">"/"</span><span class="p">)</span><span class="w">
</span><span class="n">is_muted</span> <span class="o">=</span> <span class="n">mumble</span><span class="o">.</span><span class="n">isSelfMuted</span><span class="p">()</span><span class="w">
</span><span class="n">mumble</span><span class="o">.</span><span class="n">setSelfMuted</span><span class="p">(</span><span class="ow">not</span> <span class="n">is_muted</span><span class="p">)</span>
</pre>
<p>To use this script, set its path in the configuration tab for the “Headset” plugin in blueman.</p>
<p>The only remaining problem with this setup is that I can’t keep the headset on during my work day, as I don’t have a way to put it in standby mode automatically. This means that my battery runs out pretty quickly, even when nothing is happening on Mumble.</p>
<p><em>Currently Playing: Red Sparowes - Finally As That Blazing Sun Shone</em></p>
OpenChange server and SOGo2010-10-26T17:47:16+02:002010-10-26T17:47:16+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-10-26:264-OpenChange-server-and-SOGo.html<p>There’s more good news on the OpenChange front. Julien has been working
together with Wolfgang and Ludovic from <a class="reference external" href="http://www.inverse.ca/english.html">Inverse</a> to leverage the server-side
support in OpenChange to provide native Exchange server support in <a class="reference external" href="http://ww.sogo.nu/">SOGo</a>.</p>
<p>A couple of days ago we <a class="reference external" href="http://www.openchange.org/">announced</a> that there now is an initial version that
allows the use of Outlook against a SOGo server through OpenChange.</p>
<p>There is a screencast <a class="reference external" href="http://www.youtube.com/v/oSZJ95YeXYE">up on youtube</a> (there is also a <a class="reference external" href="http://www.openchange.org/openchange-2010-10-15.mov">.mov version of the
screencast</a>).</p>
<p>As far as I know, this is the first time it’s possible to actually use Outlook
clients with a non-Microsoft Exchange-compatible server, without the need for
plugins on the Outlook side. And it’s all Free Software. Of course, this is
just a preview, and not something we’d recommend everybody to run in production
just yet. But it’s exciting to finally see this come together.</p>
<p>We already have OpenChange packages in <a class="reference external" href="http://packages.debian.org/openchangeserver">Debian</a> and <a class="reference external" href="http://packages.ubuntu.com/openchangeserver">Ubuntu</a> but I hope I can
help get SOGo packaged for both distributions as well.</p>
Samba 4 and OpenChange daily Ubuntu packages2010-09-28T10:39:50+02:002010-09-28T10:39:50+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-09-28:263-Samba-4-and-OpenChange-daily-Ubuntu-packages.html<div class="section" id="daily-builds">
<h2>Daily builds</h2>
<p>As of a month ago there are Ubuntu archives with fresh packages of <a class="reference external" href="http://wiki.samba.org/index.php/Samba4">Samba 4</a> and <a class="reference external" href="http://www.openchange.org/">OpenChange</a>, built on a daily basis day from the latest upstream revision.</p>
<p>This means that it is now possible to run a version of Samba 4 that is less than 24 hours old, without having to know how to extract source code from the version control system that upstream is using, without having to know how to build and install an application from source, but perhaps most importantly: without having to go through the tedious process of manually updating the source code and rebuilding.</p>
<p><a class="reference external" href="http://www.openchange.org/">OpenChange</a> is tightly coupled to Samba 4, so installing a new version of OpenChange usually involves installing a new version of Samba 4 as well. To make matters more confusing, the two projects use different version control systems (Samba 4 is in <a class="reference external" href="http://git-scm.com/">Git</a>, while OpenChange is in <a class="reference external" href="http://subversion.tigris.org/">Subversion</a>) and different build systems (Samba 4 uses <a class="reference external" href="http://code.google.com/p/waf/">waf</a>, OpenChange uses <a class="reference external" href="http://www.gnu.org/software/autoconf/">autoconf</a> and make).</p>
<p>I have been involved in Samba 4 and OpenChange as an upstream developer and more recently also as a packager for both <a class="reference external" href="http://www.debian.org/">Debian</a> and <a class="reference external" href="http://www.ubuntu.com/">Ubuntu</a>.</p>
<p>As an upstream developer for both these projects it is important for me that users can easily run the development versions. It makes it possible for interested users to confirm the fixes for issues they have reported and to test new features. The more users run the development version, the more confident I can be as a developer that <a class="reference external" href="http://lists.samba.org/archive/samba-technical/2010-September/073481.html">doing a release will not cause any unexpected surprises</a>.</p>
<p>As a packager it is useful to know when there are upstream changes that are going to break my package with the next release.</p>
</div>
<div class="section" id="recipes">
<h2>Recipes</h2>
<p>The daily builds work using so-called recipes which describe how to build a Debian source package from a set of Bazaar branches. For example, the <a class="reference external" href="https://code.edge.launchpad.net/~samba-team/+recipe/4.0-ppa">Samba 4 recipe</a> looks like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span># bzr-builder format 0.2 deb-version 4.0.0~alpha14~bzr{revno}~ppa{revno:packaging}+{revno:debian}
lp:samba
merge debian lp:~samba-team/samba/unstable
merge packaging lp:~samba-team/samba/4.0-ppa-maverick
</pre></div></td></tr></table></div>
<p>This dictates that a source package should be built by taking the <a class="reference external" href="https://code.launchpad.net/~samba-team/samba/trunk">upstream Samba branch</a> and merging the <a class="reference external" href="https://code.launchpad.net/~samba-team/samba/unstable">Debian packaging</a> and <a class="reference external" href="https://code.launchpad.net/~samba-team/samba/4.0-ppa-maverick">some recipe-specific tweaking</a>. The last bit on the first line indicates the version string to be used when generating a changelog entry for the daily build.</p>
<p>Every night Launchpad (through <a class="reference external" href="https://launchpad.net/bzr-builder">bzr-builder</a>) merges these branches and attempts to build the resulting source package, e-mailing me in case of build problems. Generally I fix issues that come up by committing directly to upstream <span class="caps">VCS</span> or to the Debian packaging branch. There is no overhead in maintaining the daily build after I’ve set it up.</p>
<p>For more information on creating source package recipes, see <a class="reference external" href="https://help.launchpad.net/Packaging/SourceBuilds/GettingStarted">getting started</a>.</p>
<div class="section" id="toolchain">
<h3>Toolchain</h3>
<p>The entire toolchain that does the daily package builds for Ubuntu is Free Software, and I have contributed to various bits of that toolchain over the years. It’s exciting to see everything come together.</p>
<div class="section" id="soyuz">
<h4>Soyuz</h4>
<p>Launchpad consists of multiple pillars - one of those pillars is Soyuz, which I hack on as part of my day job at Canonical. Soyuz is responsible for the archive management and package building. Debian source packages (a combination of upstream source code and packaging metadata) get uploaded by users and then built for various architectures on our <a class="reference external" href="http://launchpad.net/builders">buildfarm</a> and published to the Ubuntu archive or to users personal package archives.</p>
</div>
<div class="section" id="launchpad-code">
<h4>Launchpad-code</h4>
<p>Another pillar of Launchpad is Launchpad-code, which is responsible for the hosting and management of version control branches. Launchpad users can either host their branches on Launchpad directly or mirror branches (either native Bazaar branches or branches in a foreign format such as Subversion, Git or Mercurial). The mirrorring of native and foreign branches happens using standard Bazaar <span class="caps">API</span>’s. In the case of Samba and OpenChange we import the branches of the upstream projects (Samba is in Git, OpenChange is in Subversion) and the packaging for both projects is in Bazaar.</p>
<p>Launchad-code calls out to Bazaar to do the actual mirrorring. Over the last few years I have done a lot of work to improve Bazaars support for foreign branches, in particular on supporting Subversion, Git and Mercurial. As the code mirrorring in Launchpad is one of the biggest users of bzr-svn and bzr-git it has helped find some of the more obscure bugs in those plugins over the last few years, to the point where there are only a handful of <a class="reference external" href="https://dev.launchpad.net/FailingBzrGitImports">issues with Git imports</a> and <a class="reference external" href="https://dev.launchpad.net/FailingBzrSvnImports">Subversion imports</a> left.</p>
</div>
<div class="section" id="bzr-git-and-dulwich">
<h4>bzr-git and dulwich</h4>
<p>bzr-git provides transparent access to Git repositories from within Bazaar and is built on top of <a class="reference external" href="http://samba.org/~jelmer/dulwich">Dulwich</a>. Dulwich is a Python library that provides access to the Git file formats and protocols that is completely independent of Bazaar. James Westby originally started it and I adopted it for bzr-git and further extended it. There are now several other projects that use it as well, including <a class="reference external" href="http://hg-git.github.com/">hg-git</a>, and <a class="reference external" href="http://www.rabbitvcs.org/">rabbitvcs</a>. Apart from James and myself, almost two dozen other people have contributed it so far.</p>
</div>
<div class="section" id="bzr-svn-and-subvertpy">
<h4>bzr-svn and subvertpy</h4>
<p>bzr-svn provides transparant access to Subversion repositories in Bazaar. When I grew frustrated with the existing Subversion Python bindings <a class="reference external" href="http://jelmer.vernstok.nl/blog/archives/218-bzr-svn-now-with-its-own-Subversion-Python-bindings.html">for various reasons</a>, I decided to create independent Python bindings for Subversion from scratch. These bindings have since been split out into a separate project - <a class="reference external" href="http://samba.org/~jelmer/subvertpy">subvertpy</a> - and other projects have since also started using them, e.g. <a class="reference external" href="http://mercurial.selenic.com/wiki/HgSubversion">hgsubversion</a> and <a class="reference external" href="http://basieproject.org/">basie</a>.</p>
</div>
</div>
<div class="section" id="using-the-daily-builds">
<h3>Using the daily builds</h3>
<p>To use the <a class="reference external" href="http://wiki.samba.org/index.php/Samba4">Samba 4</a> and <a class="reference external" href="http://www.openchange.org/">OpenChange</a> daily builds (Ubuntu Maverick only for now), run:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span>$<span class="w"> </span>apt-add-repository<span class="w"> </span>ppa:samba-team/ppa
$<span class="w"> </span>apt-add-repository<span class="w"> </span>ppa:openchange/daily-builds
</pre></div></td></tr></table></div>
<p><em>Currently Playing: Karnivool - Themata</em></p>
</div>
</div>
subunit usage in Samba2010-09-24T00:46:11+02:002010-09-24T00:46:11+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-09-24:262-subunit-usage-in-Samba.html<p>Both Samba 3 and Samba 4 are now using the “<a class="reference external" href="http://launchpad.net/subunit">subunit</a>” protocol inside their testsuite (aka “make test”). subunit is a streaming protocol used to report test results that is aimed at being simple to generate and parse as well as being human readable.</p>
<p>A very simple subunit stream might look like this:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span>
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span>test: samba4.tests.util.strlist.check_list_make
creating list...
list created!
success: samba4.tests.util.strlist.check_list_make
test: samba4.tests.util.strlist.check_list_make_shell
creating list...
xfail: samba4.tests.util.strlist.check_list_make_shell [
returned NT_STATUS_NOT_IMPLEMENTED
]
</pre></div></td></tr></table></div>
<p>For those that are familiar with the <span class="caps">TAP</span> protocol used by Perl, it is similar to that, although it has a couple of features that <span class="caps">TAP</span> does not have. For example, it can report timestamps (useful for determining test duration) and has more flexible progress reporting.</p>
<p>Subunit is particularly useful for projects that use multiple programming languages as it allows a single tool to be used for test visualization or analysis rather than one per language. All that’s required per-language is a test runner that can spit out subunit streams.</p>
<p>selftest.pl, the main engine behind Samba’s test suite, has been using subunit internally since its creation a couple of years ago. Most other test tools we use can also report subunit, in particular our Python tests, blackbox tests, Perl tests (using tap2subunit) and smbtorture.</p>
<p><span class="dquo">“</span>make test” never displays raw subunit results, it always formats them using our format-subunit script. Samba 4’s “make test” stores the raw subunit output in st/subunit.</p>
<p>I’m attending <a class="reference external" href="http://www.snia.org/events/storage-developer2010"><span class="caps">SNIA</span> <span class="caps">SDC</span></a> at the moment and a couple of people here have asked me about the tools I use to display and analyse test results. They are:</p>
<div class="section" id="subunit-1">
<h2><a class="reference external" href="http://launchpad.net/subunit">subunit</a></h2>
<p>The subunit project contains a bunch of convenience tools for working with subunit. Other than libraries for parsing/generating subunit for several languages it contains tools for manipulating and analysing subunit streams, including:</p>
<ul class="simple">
<li>subunit-ls: List all tests in a subunit stream, optionally including their run times (I used this for the <a class="reference external" href="http://samba.org/~jelmer/samba4-test-times.ods">test duration summary</a> I sent to the Samba mailing list earlier)</li>
<li>tap2subunit: convert a <span class="caps">TAP</span> stream to a Subunit stream</li>
<li>subunit-stats: Print statistics for a subunit stream (how many successful tests, failed tests, skipped tests, etc)</li>
<li>subunit-filter: E.g. remove test result or output from a stream</li>
<li>subunit-diff: Compare two subunit streams and see what tests have started failing or are no longer failing</li>
<li>subunit2pyunit: Format a subunit stream using Python’s standard unit test test result formatter</li>
</ul>
<p>We’re including the subunit tree in the Samba git tree at lib/subunit.</p>
</div>
<div class="section" id="tribunal">
<h2><a class="reference external" href="http://launchpad.net/tribunal">tribunal</a></h2>
<p>Tribunal is a <span class="caps">GTK</span>+ viewer for subunit streams. It allows for easy browsing of test results. Tribunal is still a bit rough around the edges, although it should already be useful.</p>
<p>Example usage:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span>$<span class="w"> </span>make<span class="w"> </span><span class="nb">test</span>
$<span class="w"> </span>tribunal-subunit<span class="w"> </span>st/subunit
</pre></div></td></tr></table></div>
</div>
<div class="section" id="testrepository">
<h2><a class="reference external" href="http://launchpad.net/testrepository">testrepository</a></h2>
<p>Test Repository provides a database of test results which fits into developers work flow and keeps track of useful information like what tests are failing, or which failures have the same backtrace.</p>
<p>In particular Test Repository can re-run only the tests that failed in the previous test run:</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span>$<span class="w"> </span>testr<span class="w"> </span>init
<span class="c1"># Run the full testsuite (1 hour goes by)</span>
$<span class="w"> </span>testr<span class="w"> </span>run
<span class="c1"># Run those tests from the testsuite that failed in the previous run</span>
<span class="c1"># (this would be a lot shorter usually, depending on how many tests were</span>
<span class="c1"># failing)</span>
$<span class="w"> </span>testr<span class="w"> </span>run<span class="w"> </span>--failing
</pre></div></td></tr></table></div>
<p>testrepository is also still in its early days, but can potentially be very useful, e.g. when comparing old test runs on the <a class="reference external" href="http://buildfarm.samba.org/">buildfarm</a>.</p>
</div>
Working from home2010-06-25T12:38:00+02:002010-06-25T12:38:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-06-25:259-Working-from-home.html<p>For about 6 months now I’ve been working for <a class="reference external" href="http://www.canonical.com/">Canonical</a> on the <a class="reference external" href="http://launchpad.net/soyuz">Soyuz</a> component of <a class="reference external" href="https://launchpad.net/">Launchpad</a>. Like most other engineers at Canonical I don’t work at the office but from a desk at home, as our nearest office is in London, not really a distance that is feasible for a commute. I do work at regular hours during work days and stay in touch with my colleagues using <span class="caps">IRC</span> and voice over <span class="caps">IP</span>.</p>
<p>I did have some experience working on contracts and study assignments from home previously, but working a fulltime regular job has turned out to be a bigger challenge. It seems easy enough. No travel time, every day is <a class="reference external" href="http://en.wikipedia.org/wiki/Casual_Friday">casual Friday</a>, being able to listen to obscure death metal all day without driving coworkers crazy. Awesome, right?</p>
<p>Well, not entirely. I can’t say I wasn’t warned beforehand (I was) but I still ran head-first into some of the common mistakes.</p>
<div class="section" id="solitude">
<h2>Solitude</h2>
<p>I can work well by myself and I appreciate the occasional solitude, but it does get kinda lonely when you’re physically sitting by yourself for 8 hours a day, five days a week.</p>
<p>Fortunately we regularly have sprints at different locations around the world and, apart from appealing to the travel junkie in me, that brings some essential face time with coworkers. Electronic communication mechanisms such as mailing lists, <span class="caps">IRC</span>, Skype and, more recently, <a class="reference external" href="http://mumble.sourceforge.net/">mumble</a> also help make the rest of the company feel closer, but it’s still very different from being able to talk to people at the water cooler (the point of which, btw, still escapes me. What’s wrong with proper cold tap water?).</p>
<p>What also seems to help is going into the city and meeting up with others for lunch, or even just to get groceries.</p>
</div>
<div class="section" id="concentration-work-times">
<h2>Concentration, work times</h2>
<p>One of the nice things about working at home is that you’re quite flexible in planning your days; it’s possible to interrupt work to run an errand if necessary. The downside of it is that it is also really easy to get distracted, and there’s something I do very well: procrastinating. I initially ended up getting distracted quite often and then would end up working into the evening to make up for that lost time. The result being that, while only spending 8 hours doing actual work, it felt like having been at work for 12 hours in the end and having lost all spare time. Or as a friend summarized it accurately: working at home is all about boundaries.</p>
<p>This is at least partially related to the fact that I am a compulsive multi-tasker; I always do several things at once and context-switch every minute or so (prompted by e.g. having to wait for code to compile), including checking email and responding to conversations on <span class="caps">IRC</span> and Google Talk. This, among other things, has the effect that I respond quite slowly in <span class="caps">IRC</span>/<span class="caps">IM</span> conversations; if you’ve ever chatted with me you’ve probably noticed it. Multi-tasking has always worked well for me - despite research suggesting otherwise - because software development always involves a lot of waiting (for vcses, compilers, testsuites, …).</p>
<p>Recently I’ve tried to eliminate some of the other distractions by signing out off Skype, Empathy (Google Talk, <span class="caps">MSN</span>, etc) and Google reader completely and only checking email a couple of times per day.</p>
</div>
<div class="section" id="feeling-productive">
<h2>Feeling productive</h2>
<p>What has perhaps surprised me most of all was how essential the satisfaction of getting something done is. After spending about a day staring at Python code it’s important for your mood to have accomplished <em>something</em>. This appears to be a vicious circle, as lack of progress kills the fun of work, which kills motivation, which causes a lack of progress.</p>
<p>I am hard core, so during my first months I used my lunch breaks and evenings to hack on other free software projects, triaging bug reports that had come in or reviewing patches. Despite the fact that this is indeed technically a break from Launchpad, it didn’t (surprise!) seem to work as well as stepping away from the computer completely. Also, it turns out that spending 14 hours a day programming doesn’t make you all that much more productive than working a couple of hours less.</p>
<p>What I’ve discovered recently is that getting at least one branch done by the end of each day, even if it’s just by fixing a trivial bug, helps tremendously in giving me some sense of accomplishment. Julian also wrote a blog post with some useful hints on <a class="reference external" href="http://bigjools.wordpress.com/2010/02/12/feeling-productive">feeling productive</a> a while ago.</p>
<p>What is your experience working from home? Any good tips?</p>
<p><em>Currently Playing: Sieges Even - Unbreakable</em></p>
</div>
Samba Summer of Code2010-06-10T14:56:15+02:002010-06-10T14:56:15+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-06-10:257-Samba-Summer-of-Code.html<p>As I have done in previous years, I am again participating in the
<a class="reference external" href="http://code.google.com/soc/">Google Summer of Code</a> as mentor for the Samba project.</p>
<p>Last year I <a class="reference external" href="http://samba.org/~abartlet/">Andrew</a> and I co-mentored three students with mixed results. In the end we had to drop one of our students but the other two did well. I’ve only taken on one student this year for various reasons.</p>
<p>The amount of time required to mentor a student varies wildly depending on the student and is hard to predict based on their application. Some students seem to require quite a lot of mentoring while others are self-motivated and self-learning. This has not just been my experience, I’ve heard similar stories from fellow mentors on other projects.</p>
<p>Last summer Ricardo worked on <a class="reference external" href="http://github.com/rvelhote/GSoC-SWAT"><span class="caps">SWAT</span> for Samba 4</a> and he is still actively working on the project, even after the Summer of Code has finished. I hope to find the time to package <span class="caps">SWAT</span> in time for <a class="reference external" href="http://www.debian.org/">Debian</a> <a class="reference external" href="http://www.debian.org/releases/testing/">Squeeze</a>. At the moment <span class="caps">SWAT</span> just supports managing shares but Ricardo is working on user management.</p>
<p>In 2009 Calin worked on the <a class="reference external" href="http://wiki.samba.org/index.php/SambaGtk"><span class="caps">GTK</span>+ frontends for Samba</a>, in particular changing them to be Python-based rather than C-based. This year his work is going to be continued by Sergio, hopefully with the some user-ready tools as the end result.</p>
<p><em>Currently Playing: Gazpacho - 117</em></p>
Proof of concept OpenChange server working2010-06-08T19:09:08+02:002010-06-08T19:09:08+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-06-08:256-Proof-of-concept-OpenChange-server-working.html<p>Seeing <a class="reference external" href="http://openchange.org/">this</a> makes me very happy. It’s taken <a class="reference external" href="http://www.openchange.org/index.php?option=com_content&view=article&id=24&Itemid=15">us</a> a couple of years to get to this point but we’ve finally made it, mostly thanks to the dedication and persistence of Julien and Brad.</p>
Thoughts on the Nokia N9002010-04-17T06:30:00+02:002010-04-17T06:30:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-04-17:253-Thoughts-on-the-Nokia-N900.html<p>Yesterday I started writing up a quick review of my new phone/tablet, a Nokia
N900. Unfortunately it has since broken down by what appears to be a hardware
issue. It does still work to a certain degree, as the leds flash and I
can access it as a <span class="caps">USB</span> mass-storage device, but the screen hasn’t shown
anything since I woke up this morning. The timing could not have been worse, as
I’m currently abroad and having a phone (as well as a <span class="caps">GPS</span>) would be useful,
especially considering the ash cloud situation in Iceland.</p>
<p><em>Update</em>: 4 weeks after dropping my phone off at the Nokia care center I have
received a new N900; they weren’t able to tell me what was wrong with the old one.</p>
<p>Anyway, my thoughts about the N900. Please note that I haven’t used a lot of
other smartphones, so I can’t really compare this with Android phones or iPhones.</p>
<div class="section" id="the-good">
<h2>The good</h2>
<p>The integration between the different components of the phone is really well
done. Nokia has invested heavily in Telepathy, which is used for all voice and
text messaging, and it shows. Skype and regular telephony are very nicely
integrated with telepathy, albeit through proprietary daemons. There is one
global status for all protocols. Skype and <span class="caps">SIP</span> conferencing work well.</p>
<p>The address book is another thing that is very nicely integrated with the rest
of the device. It combines <span class="caps">IM</span> addresses, phone numbers, email addresses and
other information for a contact and there are third party applications that can
help keep that information up to date.</p>
<p>Both the Ovi maps app and its data are free, and it’s possible to copy the maps
onto the device so you don’t need to stream them all from the internet when
you’re abroad. It’s pretty quick compared to e.g. a Garmin <span class="caps">GPS</span>, but lacks
features. It can’t load <span class="caps">GPX</span> files, can’t show points of interest, store tracks, etc.</p>
<p>The music player is neat and automatically indexes all audio and video files
that are copied onto the device. The camera and photo manager work well, and
are reasonably snappy.</p>
<p>The web browser displays all pages I’ve accessed so far without rendering
issues, including flash.</p>
<p>There are quite a lot of neat third-party free software applications available
- eCoach, Hermes, grr, mBarcode, Mauku, tubes, tuner, fahrplan.</p>
</div>
<div class="section" id="the-bad">
<h2>The bad</h2>
<p>Maemo is Linux-based and it has a lot of similarities to your average
Debian-derived distribution. Despite that, it contains a lot of proprietary
software. In particular, the Skype and <span class="caps">POT</span> plugins for telepathy, the address
book and things like Ovi Maps are proprietary. This means it’s impossible for
me to fix little annoyances (see below) in these packages, but more
importantly, it’s impossible for others to fix these little annoyances or port
these apps to the desktop and other devices (where there’s an even larger pool
of potential bug fixers).</p>
<p>The <span class="caps">GPS</span> doesn’t work very well. This might be due to the fact that the hardware
is substandard, but I suspect it has got more to do with the fact that if it
does not find a signal within 30 seconds it will switch itself off. I assume
this is to save energy, but this behaviour can not be switched off anywhere. A
workaround is to close and reopen Ovi Maps every 15 seconds or so, but that’s
pretty annoying. I’m sure somebody with access to the source (I don’t!) can fix it.</p>
<p>I’m sure the calendaring app is nice, but it lacks support for synchronization
with Google calendar, which makes it unusable for me.</p>
<p>The standard email client does work, but it doesn’t scale well. It will lock
up at times while trying to index mailboxes. I’ve tried using claws mail for a
while, but its interface is just too cumbersome - in general, but in particular
on such a small screen.</p>
<p>The battery lifetime sucks. If I’m lucky I can make it through a single day with
a fully charged battery, with only mild usage. The form factor is also still
a minor issue for me, but I doubt that bit can be fixed in software.</p>
<p>Overall I’m pretty happy with the N900, although I’m not sure if I would pick
it over an Android phone again.</p>
</div>
Nostalgia: 10 Years of Samba Hacking2010-02-11T04:32:00+01:002010-02-11T04:32:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-02-11:250-Nostalgia-10-Years-of-Samba-Hacking.html<p>While searching for something else I happened to come across <a class="reference external" href="http://lists.samba.org/archive/samba-ntdom/2000-November/016339.html">one of my first posts to the ntdom list</a> in November 2000.</p>
<p>My post is a simple question about a Samba crash that I myself no doubt had introduced. I’m sure I could have found a solution to it by using Google - excuse me, <a class="reference external" href="http://www.altavista.com/">AltaVista</a> - but I still received a friendly reply from <a class="reference external" href="http://plainjoe.org/">Jerry</a> explaining me to use <a class="reference external" href="http://www.gnu.org/software/gdb/"><span class="caps">GDB</span></a>. I’m not too embarrassed, at least I used proper punctuation and already wrote somewhat comprehensible English back then.</p>
<p>It’s also strange to realize it’s already been almost ten years since I started hacking on <a class="reference external" href="http://www.samba.org/">the Samba project</a>.</p>
Linux.Conf.Au 2010 - Day 3 - Wednesday2010-02-04T00:06:17+01:002010-02-04T00:06:17+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-02-04:248-Linux.Conf.Au-2010-Day-3-Wednesday.html<p>I went to Jonathan Corbet’s yearly update of the status of the Linux kernel. He
talked about the various big changes that went into the kernel over the last
year as well as the development processes. The Linux kernel is probably one of
the largest open source projects, and very healthy - there are a lot of
individuals and companies contributing to it. With this size comes a few
interesting challenges coping with the flow of changes into Linus’ tree. Their
current processes seem to deal with this quite well, and don’t seem to need a
lot of major changes at the moment.</p>
<p>His talk also included the obligatory list of features that landed in the last
year. The only one that really matters to me is the Nouveau driver, which I’m
looking forward to trying out.</p>
<p>The second talk I went to in the morning was Selena Deckelmann’s overview of
the Open Source database landscape. She mentioned there’s new projects started
daily, but it was still a bit disappointing not to see <span class="caps">TDB</span> up there.</p>
<p>After lunch Rob gave a talk about Subunit, introducing to the ideas behind the
Subunit protocol as well as presenting an overview of the tools that are
available for it and the projects that have Subunitized as of yet. It’s
exciting to see the Subunit universe slowly growing, I wasn’t aware of some of
the projects that are using it. The recently announced <a class="reference external" href="http://launchpad.net/testrepository">testrepository</a> also looks interesting, even though it is still very rudimentary at the moment.</p>
<p>In the evening <a class="reference external" href="http://en.wikipedia.org/wiki/Andrew_Tridgell">Tridge</a>, <a class="reference external" href="http://en.wikipedia.org/wiki/Rusty_Russell">Rusty</a>, <a class="reference external" href="http://samba.org/~abartlet">Andrew</a>, <a class="reference external" href="http://en.wikipedia.org/wiki/Jeremy_Allison">Jeremy</a>, <a class="reference external" href="http://en.wikipedia.org/wiki/Anthony_Towns"><span class="caps">AJ</span></a> and I participated in the <a class="reference external" href="http://hackoff.lca2010.org.nz/">hackoff</a> as the “Samba Team”.</p>
<p>The hackoff was a lot of fun, and consisted of 6 problems, each of which involved somehow decoding the data file for the problem and extracting a short token from it in one way or another, which was required to retrieve the next problem. We managed to solve 4 problems in the hour that the organizers had allocated, and ended first because we were a bit quicker in solving the 4th problem than the runner-ups. No doubt the fact that we were the largest team had something to do with this.</p>
<p>I hung out with some of the awesome <a class="reference external" href="http://www.git-scm.com/">Git</a> and <a class="reference external" href="http://www.github.com/">Github</a> developers in the Malthouse in the evening, and talked about <a class="reference external" href="http://samba.org/~jelmer/dulwich">Dulwich</a>, <a class="reference external" href="http://bazaar.canonical.com/">Bazaar</a> and <a class="reference external" href="http://launchpad.net/">Launchpad</a> (“No <em>really</em>, I am not aware of any plans to add Git support to Launchpad.”).</p>
Linux.Conf.Au 2010 - Day 2 - Tuesday2010-02-03T19:44:24+01:002010-02-03T19:44:24+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-02-03:247-Linux.Conf.Au-2010-Day-2-Tuesday.html<p>On Tuesday we had our “Launchpad” mini-conf, which featured talks from
Launchpad developers and users alike. It wasn’t necessarily about
hosting projects on Launchpad, but rather about how various projects could
benefit from Launchpad.</p>
<p>I popped out of the Launchpad track for a bit to attend <a class="reference external" href="http://sysadmin.miniconf.org/presentations10.html#04">Andrews talk about the
current status of Samba 4</a>. He did a nice job of summarizing the events in the
last year, the most of import one of course being the support for <span class="caps">DC</span>
synchronization. I’m proud we’ve finally managed to pull this off - and
hopefully we’ll actually have a beta out next year. We have been saying “maybe
next year” for almost 4 years now when people asked us for estimates of a
release date.</p>
<p>At the end of the day Aaron and I gave a quick introduction to
Launchpad’s code imports and code reviews.</p>
Linux.Conf.Au 2010 - Day 12010-02-03T18:15:45+01:002010-02-03T18:15:45+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-02-03:246-Linux.Conf.Au-2010-Day-1.html<p><a class="reference external" href="http://www.lca2010.org.nz/">Linux.Conf.Au</a> has a reputation for being one of the best <span class="caps">FLOSS</span> conferences
in the world, and it more than lived up to my high expectations. The <a class="reference external" href="http://www.linux.org.au/conf/2006/">last
one</a> I attended was also in New Zealand, but further south - in Dunedin.</p>
<div class="section" id="day-1-monday">
<h2>Day 1 - Monday</h2>
<p>As usual the actual conference was preceded by two days of miniconfs.
On the first day I attended some of the talks in the Open Languages track.</p>
<p>mwhudson gave a talk about <a class="reference external" href="http://codespeak.net/pypy/dist/pypy/doc/">pypy</a> - Python implemented in <a class="reference external" href="http://www.python.org/">Python</a>. He
discussed the reasons for doing what they do and the progress they’ve made so
far. Like so many of the custom Python implementations, one of the main things
that’s holding them back is the lack of support for the extensions written in C
for CPython.</p>
<p><a class="reference external" href="http://en.wikipedia.org/wiki/Rusty_Russell">Rusty</a> gave a quick tutorial to <a class="reference external" href="http://talloc.samba.org">talloc</a> after lunch (“it’s a shame K&R
didn’t think of this!”) and explained why it’s so great.</p>
<p>In the afternoon I caught some of the talks in the <a class="reference external" href="http://distrosummit.org/">distro summit</a> track. Both
of the sessions that I attended happened to be Ubuntu-related - first Dustin
gave a quick introduction to the components of Launchpad, followed by a talk
from Lucas about the relationship between <a class="reference external" href="http://www.ubuntu.com/">Ubuntu</a> and <a class="reference external" href="http://www.debian.org/">Debian</a>. There was a
discussion afterwards about interoperability between the various hosting sites
and bug trackers. Several audience members questioned the relevance of Debian
and suggested everything should just switch to Launchpad, but this seemed to be
founded in ignorance. (For the record, none were actually Launchpad developers).</p>
</div>
Build from branch2010-02-01T16:49:10+01:002010-02-01T16:49:10+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2010-02-01:245-Build-from-branch.html<p>At the moment I am returning home after three very productive and awesome weeks
in Wellington, Sydney and Strasbourg.</p>
<p>I spent the first week in the West Plaza in Wellington, working together with fellow Launchpad developers <a class="reference external" href="http://bigjools.wordpress.com/2010/01/22/success-in-wellington/">on getting the basics of building from branches working</a>. We eventually managed to get something working at the end of Friday afternoon. We split the work up at the beginning of the week and then worked on it in pairs for a couple of days before integrating all work on Friday. At the end of the week <a class="reference external" href="http://launchpad.net/~wgrant">William</a> managed to get a basic source package build from recipe through the queue.</p>
<p>Pair-programming with <a class="reference external" href="http://launchpad.net/~jml">Jono</a> and <a class="reference external" href="http://launchpad.net/~mwhudson">Michael</a> was very educational, I suspect I’ll be a fair bit quicker when I get back to hacking on Launchpad by myself. It’s scary to see how some people can make the changes that would take me a full day in a mere hour.</p>
<p><a class="reference external" href="http://launchpad.net/~thumper">Tim</a> picked up my initial work on support for Mercurial imports and completed
and landed it during the sprint. Since the rollout on Wednesday it is possible
to request Mercurial imports on Launchpad. Most imports (e.g. mutt, dovecot,
hg) seem to work fine, with the main exception being the really large Mercurial
repositories such as OpenOffice.org and OpenJDK. This is because of (known)
scaling issues that will be fixed in one of the next releases of <a class="reference external" href="http://launchpad.net/bzr-hg">bzr-hg</a>.</p>
<p>This was the first time I was back in Wellington since 2006, and the weather
this year was exactly as I remembered it; showers and wind, with the occasional
day of sunshine. For a capital the city centre is quite small, but it has its
charm and the view from the various hills around the bay is amazing.</p>
<p>On the weekend I met up with <a class="reference external" href="http://samba.org/~abartlet">Andrew</a> and Kirsty and we did some hiking around
Wellington (where the weather allowed it).</p>
My first week as a Launchpad developer: impressions2009-12-10T20:10:17+01:002009-12-10T20:10:17+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-12-10:244-My-first-week-as-a-Launchpad-developer-impressions.html<p>Roughly a week ago I joined Julian, Muharem and Michael, working on the <a class="reference external" href="https://launchpad.net/soyuz">Soyuz</a> component of <a class="reference external" href="https://launchpad.net/">Launchpad</a>. For now I’ve been working on easy Soyuz bugs, as a way of becoming more familiar with the internals. I’m working from home but I had the chance to hang out with some of the other Launchpad developers, including the full Soyuz team, at <a class="reference external" href="http://wiki.ubuntu.com/UDS-L"><span class="caps">UDS</span> Lucid</a> in Dallas.</p>
<p>Launchpad is different from most other <span class="caps">FOSS</span> projects I have worked on so far. Some things I noticed during my first week:</p>
<p>The codebase is big and well tied together. I don’t think I’ve ever used grep and ctags as often as I have in the last week. Fortunately, the directory structure makes it relatively easy to predict where to look for things.</p>
<p>Reviews are really quick - no long round-trips between author and reviewer trying to get a branch landed. This is a really <em>really</em> great thing.</p>
<p>It’s easy to find somebody familiar with a particular piece of code and it doesn’t take long to get an answer when you ask questions. I’m still getting used to this - I tend to ask questions sporadically because I have gotten used to having to wait a couple of days for an answer that’s actually useful.</p>
<p>Setting up the development environment takes some time. Or perhaps I’m spoiled by Bazaar where “bzr branch lp:bzr bzr && ./bzr/bzr selftest” is all you need to start hacking. And it seems like karmic is the only platform on which things work - I tried with Debian Sid and Lucid as well, but things broke in strange and unusual ways.</p>
<p>The test suite is heavy and takes long to start up, something that makes proper <span class="caps">TDD</span> too hard. I also managed to run into some unexplainable problems where the librarian wouldn’t shut down on one of my systems. Since there is only one instance of the database it is not really possible to run multiple instances of the testsuite at the same time unless you use chroots or something like that - this makes it hard to work on multiple branches at the same time, something which would especially be nice since the testsuite is slow (so you can run the testsuite in one branch, hack in another and alterate).</p>
<p>Doctests, while fast, a bit of a nuisance. Because of the setup/teardown overhead that is paid for every single test, doc tests are a lot faster than unit tests. On the other hand, pdb doesn’t play well with doc tests - it doesn’t show any context. Conceptually I also prefer small unit tests over doc tests, since they’re quicker to read, easier to understand and there’s less side-effects from previous instructions in the test that could affect the code that’s being tested.</p>
<p>And for those that know me well; yes, getting used to somewhat regular working hours was indeed a challenge, but I seem to have managed.</p>
US: Observations2009-10-30T16:54:00+01:002009-10-30T16:54:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-10-30:242-US-Observations.html<p>These past few days in the <span class="caps">US</span> were a bit of a rollercoaster. Some random observations:</p>
<ul class="simple">
<li>The mentor summit was very nice and well organized (or rather: well disorganized). Lots of awesome people around from a wide variety of projects and nationalities.</li>
<li><span class="dquo">“</span>Next Generation <span class="caps">VCS</span>” seems to be an alias for git these days in the minds of most people.</li>
<li>I didn’t write a single line of code in almost a week, something that is very rare.</li>
<li>Driving an automatic gives you two spare limbs to use for other things. What those other things are, I have yet to figure out.</li>
<li>Is the fact that your kid was student of the month or the fact that you own two cats and a dog really something that belongs on a bumper sticker?</li>
<li>Gas is cheap (compared to Europe). I drove 300 miles on a $30 tank.</li>
<li>The malls in the Bay Area are some of the biggest I’ve ever seen, but strangely enough they seem to lack both book- and cd-stores.</li>
<li>It is legal to turn right on a red traffic sign in California unless otherwise indicated. It took me a while to realize this until people repeatedly started honking behind me…</li>
<li>The waiver I had to sign to be able to skydive in California was scary. I can live with my operating system coming without implied warranty or fitness for a particular purpose, but my parachute?</li>
<li>I stopped pretending to have any regularity in my sleeping habits. 6 <span class="caps">AM</span> flights? It seemed like a good idea at the time.</li>
</ul>
CtrlProxy: Looking for a new maintainer2009-09-13T21:26:13+02:002009-09-13T21:26:13+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-09-13:241-ctrlproxy-looking-for-a-new-maintainer.html<p>After over 7 years of working on it off and on, I’m looking for somebody to
help maintain (and eventually take over) <a class="reference external" href="http://www.ctrlproxy.org/">CtrlProxy</a>. I started working on
CtrlProxy somewhere in 2002, only a short while after Wilmer started hacking on
<a class="reference external" href="http://www.bitlbee.org/">BitlBee</a>. If I remember correctly I started working on it because I didn’t
want to run a separate dircproxy (the only real competitor at the time)
instance (with configuration) for each <span class="caps">IRC</span> network that I connected to. It was
also just a good excuse to play with the <span class="caps">IRC</span> protocol a bit.</p>
<p>Over the years, CtrlProxy has served as a playground for me to try out new and
interesting things. It’s been rewritten or severely refactored several times in
its early history, the latest time being the 3.0 release (from 2005). I’ve
tried different build systems, I’ve tried different implementation languages,
I’ve tried different configuration file formats, I’ve tried different support
libraries, I’ve tried different version control systems, I’ve tried different
documentation formats. So while it’s definitely been a very educational project
for me personally, I haven’t really had the time or the interest to dedicate to
the project that it deserved during the last few years. This was mostly because
<a class="reference external" href="http://www.samba.org/">there</a> <a class="reference external" href="http://www.wireshark.org/">were</a> <a class="reference external" href="http://www.ubuntu.com/">other</a> <a class="reference external" href="http://www.debian.org/">more</a> <a class="reference external" href="http://www.bitlbee.org/">interesting</a> <a class="reference external" href="http://www.openchange.org/"><span class="caps">FOSS</span></a> <a class="reference external" href="http://bazaar-vcs.org/">projects</a> I spent my
spare cycles on.</p>
<p>These days there are plenty of other good <span class="caps">IRC</span> proxies out there, such as
<a class="reference external" href="http://bip.t1r.net/"><span class="caps">BIP</span></a>, so I doubt CtrlProxy will be missed if it were to disappear. Despite
that, if anybody is interested in taking over, please send me an email
(<a class="reference external" href="mailto:jelmer@samba.org">jelmer@samba.org</a>) or contact me on <span class="caps">IRC</span> (jelmer on the <span class="caps">OFTC</span> and Freenode networks).</p>
<p><em>Currently Playing: Anathema - Shroud of False</em></p>
Summer of Code 20092009-09-11T14:22:06+02:002009-09-11T14:22:06+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-09-11:239-Summer-of-Code-2009.html<p>For this years (the fifth?) <a class="reference external" href="http://code.google.com/soc/">Summer of Code</a>, I participated once again as a mentor for the Samba and OpenChange projects.</p>
<p>Samba was assigned four slots this year: one was a <span class="caps">CIFSFS</span> project mentored by
Steve French and the other three were Python projects related to Samba 4,
co-mentored by <a class="reference external" href="http://samba.org/~abartlet/">Andrew</a> and me. Our students did very well this year, although
we unfortunately had to drop one after the mid-term evaluations due to lack of
effort. Nonetheless, we’re very happy with the results of the other two projects:</p>
<p>Calin Crisan (France) converted the rest of the applications in <a class="reference external" href="http://wiki.samba.org/index.php/SambaGtk">SambaGtk</a> to Python, and worked on a <span class="caps">GTK</span>+ user manager for Samba and Windows. With his improvements, it is now possible to edit registries, manage users, inspect the endpoint mapper, plan tasks and manage services on a remote Windows machine using a <span class="caps">GTK</span>+ application on a Linux workstation.</p>
<p>Ricardo Velhote (Portugal) designed and implemented a new version of <span class="caps">SWAT</span> - the Samba Web Administration Tool. Unlike the old <span class="caps">SWAT</span>, his implementation is more than just a simple web-based editor for smb.conf. As we were expecting at the start of the Summer of Code, not all of the functionality could be implemented properly in a couple of months, not while getting the design and infrastructure right. With a basic version working, we now hope the remaining subsystems can be contributed with help from the community.</p>
<p>I’m planning to merge Calin’s improvements to Samba-Gtk into the mainline in the next month or so. <span class="caps">SWAT</span> is a standalone application and will continue to live as a separate project, while being a part of the Samba ecosystem. Congratulations, Calin and Ricardo!</p>
DebCamp / DebConf92009-07-22T23:38:01+02:002009-07-22T23:38:01+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-07-22:237-DebCamp-DebConf9.html<p>So far I’m very much enjoying my first <a class="reference external" href="http://debconf9.debconf.org/">DebCamp / DebConf</a>. It’s nice to finally meet a lot of people in person that I have worked together with or talked to on <span class="caps">IRC</span> in the last few years. Cáceres is a relatively small town with a nice old city center.</p>
<p>I arrived early for DebCamp and spent the first few days here working on fixing bugs in the Bazaar and Samba packages as well as discussing the integration between Samba 4 and Kerberos with <a class="reference external" href="http://www.painless-security.com/blog/">Sam</a> (both in general and on Debian specifically). In trying to set up a Samba 4 domain we found a number of bugs in the provisioning script, most of which seem to be fixed now.</p>
<p>In the last few days I’ve mostly worked on getting Samba 4 and OpenChange ready to go into Sid (they’re in experimental only at the moment) and have discussed bzr-builddeb and related Bazaar issues with <a class="reference external" href="http://jameswestby.net/weblog">James</a>.</p>
<p><em>Currently Playing: Pixies - Velouria</em></p>
DebConf2009-07-04T13:46:16+02:002009-07-04T13:46:16+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-07-04:234-DebConf.html<img alt="" class="align-left" src="http://media.debconf.org/dc9/images/debconf9-going-to.png" />
<p>I’m looking forward to going to my first DebCamp/DebConf. I won’t be giving a talk, but I hope to work together with others on integrating Samba 3 and 4 better with the rest of the system and <span class="caps">VCS</span> integration.</p>
“Franky” Talk at SambaXP2009-06-04T00:51:00+02:002009-06-04T00:51:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-06-04:233-Franky-Talk-at-SambaXP.html<p>I’ll be giving a talk at the next <a class="reference external" href="http://www.nllgg.nl/"><span class="caps">NLLGG</span></a> <a class="reference external" href="http://www.nllgg.nl/bijeenkomst_20090606">meeting</a> about <a class="reference external" href="http://wiki.samba.org/index.php/Franky">the Franky project</a>.</p>
<p><strong>Update</strong>: <a class="reference external" href="http://samba.org/~jelmer/samba-nllgg2009.pdf">Slides</a></p>
SambaXP 20092009-04-20T00:27:00+02:002009-04-20T00:27:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-04-20:228-SambaXP-2009.html<p>Last week most of the Samba team met again for our annual conference in Göttingen. It was nice seeing everybody again, specially the folks I hadn’t seen since the last one.</p>
<p>Together with Andrew and his wife Kirsty I took the train from Amsterdam into
Germany a couple of days early and we did some sightseeing together with
Anatoli and Nadezhda during the weekend. There’s still plenty of things to
discover in Göttingen for me, even though I’ve already been there about two
dozen times. We did a tour of the city walls, visited some of the churches and
climbed the tower.</p>
<p>Julien’s talk about OpenChange was interesting and humorous as always. Volkers’
tutorial on asynchronous programming in C. Even though I’ve spent quite some
time working with and looking at these <span class="caps">API</span>’s it was nice going through them
step by step once again. It’s a strange thing to wrap your head around.</p>
<p>Andrew and I also gave our yearly <a class="reference external" href="http://samba.org/~jelmer/samba4-status-xp09.pdf">“State of Samba 4”</a> talk again. As I’ve
mentioned in other places, I’m really excited about the social effects of the
<a class="reference external" href="http://wiki.samba.org/index.php/Franky">Franky</a> project. Once again I was
reminded that giving a talk the morning after the conference party (this year
in the “Oriental Lounge”) is a bad idea.</p>
<p>Several of my fellow Debian Samba maintainers made it to SambaXP, it was nice to see Christian, Luk, Michael and Noël there. We made some decisions about the direction of the Samba packages, and a plan to allow the Samba 3 and Samba 4 packages to be installed on the same system. Unfortunately I had to miss <a class="reference external" href="http://www.perrier.eu.org/weblog/2009/05/24#samba-news-200905224">Christian’s talk</a> because it was in the same timeslot as Jeff’s talk about the <span class="caps">CIFS</span> kernel module.</p>
Finally a DD2009-01-15T20:53:20+01:002009-01-15T20:53:20+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2009-01-15:227-Finally-a-DD.html<p>After spending a little bit more than three years in the Debian New Maintainers
queue, I have finally become a Debian Developer.</p>
<p>Many thanks to the various people who have helped me through <span class="caps">NM</span> and have
sponsored my uploads over the past few years.</p>
bzr-builddeb FTW2008-11-07T18:01:30+01:002008-11-07T18:01:30+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-11-07:226-bzr-builddeb-FTW.html<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span></pre></div></td><td class="code"><div><pre><span></span>%<span class="w"> </span>bzr<span class="w"> </span>branch<span class="w"> </span>deb:line6-usb-source<span class="w"> </span>debian
Retrieving<span class="w"> </span>Vcs<span class="w"> </span>locating<span class="w"> </span>from<span class="w"> </span>line6-usb-source<span class="w"> </span>Debian<span class="w"> </span>version<span class="w"> </span><span class="m">0</span>.7.4-1
Branched<span class="w"> </span><span class="m">354</span><span class="w"> </span>revision<span class="o">(</span>s<span class="o">)</span>.
%<span class="w"> </span>bzr<span class="w"> </span>merge-upstream<span class="w"> </span>https://line6linux.svn.sourceforge.net/svnroot/line6linux/driver/trunk
All<span class="w"> </span>changes<span class="w"> </span>applied<span class="w"> </span>successfully.
Using<span class="w"> </span>version<span class="w"> </span>string<span class="w"> </span><span class="m">0</span>.7.4+svn511<span class="w"> </span><span class="k">for</span><span class="w"> </span>upstream<span class="w"> </span>branch.
The<span class="w"> </span>new<span class="w"> </span>upstream<span class="w"> </span>version<span class="w"> </span>has<span class="w"> </span>been<span class="w"> </span>imported.<span class="w"> </span>You<span class="w"> </span>should<span class="w"> </span>now<span class="w"> </span>update<span class="w"> </span>the<span class="w"> </span>changelog<span class="w"> </span><span class="o">(</span>try<span class="w"> </span>dch<span class="w"> </span>-v<span class="w"> </span><span class="m">0</span>.7.4+svn511-1<span class="w"> </span><span class="s2">"New upstream snapshot."</span><span class="o">)</span>,<span class="w"> </span>resolve<span class="w"> </span>any<span class="w"> </span>conflicts,<span class="w"> </span>and<span class="w"> </span><span class="k">then</span><span class="w"> </span>commit.
%<span class="w"> </span>dch<span class="w"> </span>-v<span class="w"> </span><span class="m">0</span>.7.4+svn511-1<span class="w"> </span><span class="s2">"New upstream snapshot.</span>
<span class="s2">% bzr builddeb</span>
<span class="s2">Building using working tree</span>
<span class="s2">Preparing the build area: ../build-area</span>
<span class="s2">Purging the build dir: ../build-area/line6-usb-0.7.4+svn511</span>
<span class="s2">[...]</span>
<span class="s2">Placing result in /home/jelmer/bzr/line6-usb/result</span>
<span class="s2">% ls ../result</span>
<span class="s2">line6-usb_0.7.4+svn511-1_amd64.changes line6-usb_0.7.4+svn511-1.dsc</span>
<span class="s2">line6-usb-source_0.7.4+svn511-1_all.deb</span>
<span class="s2">line6-usb_0.7.4+svn511-1.diff.gz line6-usb_0.7.4+svn511.orig.tar.gz</span>
</pre></div></td></tr></table></div>
<p><em>Currently Playing: Phideaux - Microdeath Softstar</em></p>
Reconciling the Samba 3 and Samba 4 source code trees2008-09-15T22:05:14+02:002008-09-15T22:05:14+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-09-15:223-Reconciling-the-Samba-3-and-Samba-4-source-code-trees.html<p>While a few of us have been working very hard on Samba 4 to allow it to rock your socks off as an Active Directory Domain Controller, some of the other Samba developers have been working just as hard on improving the existing Samba 3 codebase and adding features to that. This situation has caused tension between developers as well as technical problems in the past - code with the same purpose is being developed in parallel, libraries diverge because features are only added in one branch and not in the other, one codebase is considered “obsolete” by some and the other is considered only a playground for experimental features by others.</p>
<p>As of yesterday, we now have the two codebases living in one and the same git branch. This should make it a lot easier for the two to use the same libraries. Better yet, it should allow us to reconcile the copies of various libraries that exist in both codebases, all of which have diverged to some degree in the last few years.</p>
<p>After <a class="reference external" href="http://thread.gmane.org/gmane.comp.version-control.git/94861">a few problems came up</a> merging the two branches the easy way (they both have a directory called “source” and git doesn’t deal well with renaming them to “source3” and “source4” respectively), we decided to replay the history of both branches . This has the disadvantage that all existing branches that are based on the Samba 3 and Samba 4 branches will have to be rebased against the new master branch, but it also means we keep the ability to run “git log” inside of our source directories and having it work right.</p>
<p>Other than the fact that this makes it possible to share more code between the two codebases, one of the ideas we have is also to see if it is possible to provide an Active Directory <span class="caps">DC</span> by glueing the best bits of Samba 3 and Samba 4 together (aka “<a class="reference external" href="http://wiki.samba.org/index.php/Franky">Franky</a>“) before they are eventually merged completely.</p>
<p><em>Currently Playing: Phideaux - Formaldehyde</em></p>
Oxegen ‘082008-07-16T13:28:07+02:002008-07-16T13:28:07+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-07-16:221-Oxegen-08.html<p><a class="reference external" href="http://www.oxegen.ie/">Oxegen</a> was a bit expensive but awesome. Irish people rock.</p>
<p><strong>update</strong>: <a class="reference external" href="http://wilmer.gaast.net/">Wilmer</a> has <a class="reference external" href="http://wilmer.gaast.net/blog/archives/40-Oxegen-2008.html">details</a>.</p>
Microblogging2008-07-06T00:32:00+02:002008-07-06T00:32:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-07-06:229-Microblogging.html<p>I’ve joined the microblogging hype; I’ll be giving status updates on my coffee drinking habits and other uninteresting facts at <a class="reference external" href="http://identi.ca/jelmer">http://identi.ca/jelmer</a>.</p>
bzr-svn push without file properties2008-06-30T23:30:40+02:002008-06-30T23:30:40+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-06-30:220-bzr-svn-push-without-file-properties.html<p>Ever since bzr-svn started supporting “true push”, people have been complaining about the extra file properties it sets.</p>
<p>The key thing about “true” push is that it preserves the exact revisions that were present in Subversion. This lets bzr behave on Subversion branches transparently using the same <span class="caps">UI</span> you also use for “native” Bazaar branches.</p>
<p>In other words, if I push to a Subversion branch from my machine, then that branch in Subversion contains enough information for somebody else to reconstruct the exact bzr branch I had.</p>
<p>Since some Bazaar metadata can not be represented in Subversion, it is stored in Bazaar-specific Subversion properties. Unfortunately, these file properties show up in email commit notifications and <a class="reference external" href="http://trac.edgewall.org/">trac</a> and so they tend to annoy people.</p>
<p>There are two ways around this:</p>
<div class="section" id="revision-properties">
<h2>Revision properties</h2>
<p>Bazaar-specific metadata can be stored in in custom Subversion revision properties (these don’t show up in commit notifications). Unfortunately, this requires Subversion 1.5 or newer to run on the server.</p>
<p>I hope to start setting revision properties instead of file properties when possible as of the next bzr-svn release.</p>
</div>
<div class="section" id="less-strict-push">
<h2>less strict push</h2>
<p>It’s also possible to throw away any data that can not be represented in Subversion. Since this means that the remote branch won’t end up an exact same copy of the local revisions, this isn’t true push. The two branches will have diverged (no matter how slightly) after such a push so it is necessary to rebase on the remote branch after pushing.</p>
<p>This is similar to the way git-svn pushes data into Subversion - it calls it “dcommit”.</p>
<p>Since this uses rebase it has the usual disadvantages of rebases, which I won’t get into right now.</p>
<p>As of a couple of days ago, bzr-svn now also supports this mode of pushing using the “dpush” command, by popular demand.</p>
<p><em>Currently Playing: Brandi Carlile - The Story</em></p>
</div>
bzr-svn: now with its own Subversion Python bindings2008-06-22T18:17:42+02:002008-06-22T18:17:42+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-06-22:218-bzr-svn-now-with-its-own-Subversion-Python-bindings.html<p><a class="reference external" href="http://bazaar-vcs.org/BzrSvn">bzr-svn</a> has always been using the standard Python bindings that were
provided with Subversion itself. Unfortunately, I had to fix some issues in
these bindings since they were incomplete or broken and thus bzr-svn has always
depended on a development snapshot of Subversion.</p>
<p>As of today, bzr-svn is using its own Python bindings for Subversion.</p>
<p>There were several reasons for switching to our own bindings:</p>
<ul class="simple">
<li>There are no requirements for backwards compatibility within bzr-svn. This
means the <span class="caps">API</span> can be made sane without worrying about the mess it was in the
past and users who still rely on that.</li>
<li>Deployment. It took 2 years for my fixes to the Subversion Python bindings to
be part of a release. It’ll be even longer before Subversion 1.5 makes it
into most available distributions. That makes it very hard to just download
and install bzr-svn.</li>
<li>They’re in plain C, not <span class="caps">SWIG</span>. <span class="caps">SWIG</span> has a big advantage for the Subversion
folks since it can generate python, ruby, java or tcl bindings all at once
without a lot of overhead per language. However, it has issues as well that
make it a bad choice for bzr-svn.<ul>
<li>It generates inefficient code - it generates proxy classes that add more
layers in the stack</li>
<li>Bindings tend to be very much like the C <span class="caps">API</span> rather than “Pythonic”. To make
them more Pythonic, you need tons of typemaps. For example, the Python
bindings in bzr-svn provide an iterator when browsing the revision history
rather than a callback as C and the <span class="caps">SWIG</span> bindings do.</li>
<li>Hard to write - personally at least, I write bindings in C faster than in
<span class="caps">SWIG</span></li>
<li>Adds an extra dependency to the build process. Several people had trouble
building Subversion on their Mac machines because they didn’t have the right
version of <span class="caps">SWIG</span> available.</li>
</ul>
</li>
</ul>
<p>Since all of the patches that bzr-svn depended on previously were in the Python
bindings for Subversion, it is now possible to use bzr-svn with any version of
Subversion newer than 1.4.0. Of course, you do need to have the development
headers installed as well.</p>
<p><em>Currently Playing: Kathleen Edwards - Independent Thief</em></p>
Bazaar in the GNOME world2008-06-19T15:13:20+02:002008-06-19T15:13:20+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-06-19:217-Bazaar-in-the-GNOME-world.html<p>I was happy to see that <a class="reference external" href="http://blogs.gnome.org/johncarr">John Carr</a> has set up a <a class="reference external" href="http://bzr-mirror.gnome.org/">Bazaar Mirror</a> of all
projects in <span class="caps">GNOME</span> Subversion, all created using bzr-svn. There’s also a <a class="reference external" href="http://live.gnome.org/Bazaar">quick
introduction to using Bazaar for <span class="caps">GNOME</span> developers</a> on the <a class="reference external" href="http://live.gnome.org/"><span class="caps">GNOME</span> Wiki</a>.</p>
<p><a class="reference external" href="http://uwstopia.nl/">Wouter</a>, long time Bazaar user and <span class="caps">GNOME</span> dude, recently blogged about <a class="reference external" href="http://uwstopia.nl/blog/2008/06/gnome-specimen-now-in-gnome-svn/">pushing Bazaar branches</a> into <span class="caps">GNOME</span> Subversion, working around the restrictions imposed by the pre-commit hooks in <span class="caps">GNOME</span> Subversion.</p>
<p>The problems John ran into with memory usage in the Python Subversion bindings
encouraged me to continue the work on bzr-svn’s own Python bindings, thus
avoiding any dependency on unreleased versions of Subversion and several other issues.</p>
Git cutting corners2008-04-27T13:13:20+02:002008-04-27T13:13:20+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-04-27:216-Git-cutting-corners.html<p>My relationship with git is still one of love and hate. It cuts corners to
increase performance in a couple of places and that can be really bloody annoying.</p>
<p>For example, jerry renamed one of the top-level directories in Samba 3
(revision 9f672c26d63955f613088489c6efbdc08b5b2d14). Git will skip rename
detection in this revision because of the number of files it affects, thus
causing the output of “git log <path>” of this particular directory to be useless.</p>
<p>I’m the first to admit “bzr log” on directories and files in large history
projects is painfully slow, but at least it gets the output right.</p>
<p><em>Currently Playing: Brandi Carlile - The Story</em></p>
SambaXP and other travel2008-04-26T10:03:04+02:002008-04-26T10:03:04+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-04-26:215-SambaXP-and-other-travel.html<p>It’s been a busy two weeks. Wilco and I drove up to Göttingen on Sunday two weeks ago to spend some days hacking and meeting up with the other developers before the start of <a class="reference external" href="http://www.sambaxp.org/">SambaXP</a>. It was really nice to see everybody again after more than 7 months.</p>
<p>SambaXP was a bit different this year. There were three tracks during the second part of the conference this year, one more than previously and of course, there were several engineers from Microsoft attending this time! Some of the interesting talks this year included Julien’s update on OpenChange, Tridge’s talk on <span class="caps">PFIF</span>, the talk from the likewise folks and of course the talk from Microsofts’ Wolfgang Grieskamp on <span class="caps">SMB2</span>. We also had some other informal discussions with the Microsoft folks about specific topics - very useful!</p>
<p>There are <a class="reference external" href="http://sambaxp.org/index.php?id=137">some photos</a> up on the SambaXP homepage. And just to be ahead of the comments: yes, I know I need a haircut.</p>
<p>I did some initial work on several bits and pieces of code that I hope to expand over the next few months. Volker has started working on ncacn_ip_tcp support and I have been working on making the Samba 3 <span class="caps">DCE</span>/<span class="caps">RPC</span> library compatible with Samba 4. This should allow OpenChange to use Samba 3 in the future.</p>
<p>Guenther, Wilco and I made some initial progress on the policy library, allowing client-side manipulation of (group) policies in Samba. I worked with Simo on trying to get rid of an evil hack in Samba4’s event subsystem.</p>
<p>David Holder blogged about some of the IPv6 development that we did during the conference: <a class="reference external" href="http://www.ipv6consultancy.com/ipv6blog/?p=34">http://www.ipv6consultancy.com/ipv6blog/?p=34</a></p>
<p>And lots of other things I can’t remember at the moment…</p>
<p>After the conference Andrew, Wilco and I drove back to the Netherlands and I played tour guide for a bit showing Andrew around the country during the afternoon and hacking Samba together in the morning. Later this week we took the train to Brussels, Eurostar to London and visited Sam’s company
in the <span class="caps">UK</span> Midlands for a couple of days.</p>
<p>And in the midst of all this, it seems Ubuntu Hardy was released. Congratulations to all those involved!</p>
<p><em>Currently Playing: Brandi Carlile - Turpentine</em></p>
Using bzr-builddeb as a svn-buildpackage replacement2008-03-26T15:21:20+01:002008-03-26T15:21:20+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-03-26:214-Using-bzr-builddeb-as-a-svn-buildpackage-replacement.html<p><a class="reference external" href="http://samba.org/~jelmer/bzr-svn-buildpackage.diffThisslightlyevilhack">This slightly evil hack</a> to bzr-svn allows using bzr-builddeb as a drop-in replacement for svn-buildpackage, making it recognize the “mergeWithUpstream” property svn-buildpackage uses.</p>
<p><em>Currently Playing: Jeff Healey - Mess O’ Blues</em></p>
Adaption blockers Bazaar sprint2008-03-09T17:06:00+01:002008-03-09T17:06:00+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-03-09:213-Adaption-blockers-Bazaar-sprint.html<p>The London Bazaar sprint is over again for this year. It was really good to meet
everybody in person again and also to meet some of the folks who hadn’t been to
a sprint before.</p>
<p>Last years sprint was mainly about improving performance; this year, we
discussed adoption blockers and how to remove them. A short summary of the
<a class="reference external" href="http://bazaar-vcs.org/SprintLondonMarch08/Brainstorms">brainstorming</a> is on the wiki.</p>
<p><a class="reference external" href="http://beuno.com.ar/">Martin’s Blog</a> has some pictures.</p>
<div class="section" id="tmv">
<h2><span class="caps">TMV</span></h2>
<p>The Mars Volta concert we went to last night in Tilburg was absolutely
brilliant. Very energetic and definitely one of the best acts I’ve ever seen
live. We were standing in the back of a completely packed venue for 3 hours,
but it was very much worth it.</p>
<p><em>Currently Playing: Soft Machine - Teeth</em></p>
</div>
OpenChange Evolution plugin preview and Debian packages2008-01-21T14:42:07+01:002008-01-21T14:42:07+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-01-21:210-OpenChange-Evolution-plugin-preview-and-Debian-packages.html<p>Srini writes that a preview of the <a class="reference external" href="http://blogs.gnome.org/sragavan/2008/01/21/evolution-mapi-exchange-2007-preview/">Evolution OpenChange plugin</a> has just been
published. This plugin is now developed in the Evolution Subversion repository,
but is based on the original plugin that was written by the <a class="reference external" href="http://www.epitech.eu/">Epitech</a> team
that was assigned to OpenChange earlier this year.</p>
<p>I’ve packaged new snapshots of Samba and OpenChange for Debian/Ubuntu.
They’re available from my <a class="reference external" href="http://samba.org/~jelmer/debian/">personal apt repository</a> and will hopefully soon
also be available from Debian experimental.</p>
<p><em>Update</em>: The packages are now in <a class="reference external" href="http://packages.debian.org/experimental/">Debian experimental</a> as well as the
upcoming Intrepid release of Ubuntu. I have removed them from my personal
repository because I was running out of disk quota.</p>
Samba’s tdbdump reimplemented in Python2008-01-10T20:24:29+01:002008-01-10T20:24:29+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2008-01-10:207-Sambas-tdbdump-reimplemented-in-Python.html<p>Less than 150 characters in Python, while the original implementation in C requires more than 2000 characters</p>
<div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="kn">import</span> <span class="nn">tdb</span><span class="o">,</span> <span class="nn">sys</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">tdb</span><span class="o">.</span><span class="n">Tdb</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">for</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="ow">in</span> <span class="n">db</span><span class="o">.</span><span class="n">iteritems</span><span class="p">():</span>
<span class="nb">print</span> <span class="s2">"{</span><span class="se">\n</span><span class="s2">key(</span><span class="si">%d</span><span class="s2">) = </span><span class="si">%r</span><span class="se">\n</span><span class="s2">data(</span><span class="si">%d</span><span class="s2">) = </span><span class="si">%r</span><span class="se">\n</span><span class="s2">}</span><span class="se">\n</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">k</span><span class="p">),</span> <span class="n">k</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">),</span> <span class="n">v</span><span class="p">)</span>
<span class="p">}</span>
</pre></div></td></tr></table></div>
GTK+ LDB Browser2007-12-16T20:49:05+01:002007-12-16T20:49:05+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-12-16:204-GTK+-LDB-Browser.html<p>As some may have noticed, a large portion of my Samba 4 work during the last
few months has been focussed on adding Python bindings for our various public
libraries and the refactoring necessary to make it possible to add Python
bindings. So far, we have bindings for <span class="caps">LDB</span> and <span class="caps">TDB</span> but I intend to add bindings
for most of our public <span class="caps">API</span> so it is possible to, for example, open Windows
registry files, join domains, etc. from Python.</p>
<p><a class="reference external" href="http://ldb.samba.org"><span class="caps">LDB</span></a> is our <span class="caps">LDAP</span>-like embedded database, and is for <span class="caps">LDAP</span> what sqlite is for
<span class="caps">SQL</span>. Last night I decided to see how hard it would be to write a graphical
browser for <span class="caps">LDB</span> using Python, and it turned out to be quite easy, thanks to
PyGTK. There is a screenshot of what it looks like <a class="reference external" href="http://samba.org/~jelmer/gtkldb.png">here</a>. Packages with the
Python bindings for <span class="caps">LDB</span> are <a class="reference external" href="http://packages.debian.org/python-ldb">already in Debian</a>.</p>
<p>The sources for gtkldb are available in the samba-gtk bzr branch at
<a class="reference external" href="http://people.samba.org/bzr/jelmer/samba-gtk/trunk">http://people.samba.org/bzr/jelmer/samba-gtk/trunk</a>, along with some of the <span class="caps">GTK</span>+
frontends for Samba 4 I wrote earlier (gregedit, gwcrontab, gwsvcctl, gepdump
and gwsam).</p>
Building shared and static libs on different platforms2007-11-14T00:48:58+01:002007-11-14T00:48:58+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-11-14:201-Building-shared-and-static-libs-on-different-platforms.html<p>One of the main portability problems across different <span class="caps">POSIX</span> implementations is the way shared and static libs should be built.</p>
<p>Just came across a nice overview of the flags to use on the different platforms: <a class="reference external" href="http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html">http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html</a></p>
<p>it would be nice if <span class="caps">POSIX</span> could standardise this at some point.</p>
Epitech/OpenChange meeting2007-11-12T20:10:05+01:002007-11-12T20:10:05+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-11-12:199-EpitechOpenChange-meeting.html<p>I’ve had a lot of time to practice my French skills this weekend while visiting
<a class="reference external" href="http://www.epitech.fr/">Epitech</a> and meeting up with Julien, Ali, Dan and the
other OpenChange folks in Paris.</p>
<p>There was a forum at Epitech where student teams present the projects that
they’ve been working on over the course of the last year. One of the Epitech
student teams has done an excellent job on an OpenChange plugin for Evolution
that now even appears to have been picked up by upstream. The other projects
were also quite interesting as well and varied from a fun to play 3D racing
game to a much improved <span class="caps">LGPL</span>’ed implementation of the ClamAv virus scanner or a
personal logging application for diabetics. Afterwards there was a little bit
of an Open Source conference, where Dan and I gave <a class="reference external" href="http://www.epitech.eu/v4/conferences-open-source-art404.html">talks</a> about Open Source
and Samba, respectively.</p>
OpenChange team member2007-10-22T22:53:12+02:002007-10-22T22:53:12+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-10-22:195-OpenChange-team-member.html<p>As of today, I’m officially a member of the <a class="reference external" href="http://www.openchange.org/">OpenChange</a> development team. I
already had access to the Subversion server and have contributed a number of
patches in the last couple of years, mainly related to the build system.</p>
Bazaar: Need for a “Product” object?2007-10-10T13:17:02+02:002007-10-10T13:17:02+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-10-10:192-Bazaar-Need-for-a-Product-object.html<p>This is something that’s been lingering in the back of my head for the last
year or so. I think I am missing something in the sequence of [Branch,
Repository, WorkingTree]. Here are some of the reasons why I think this is the case:</p>
<ul class="simple">
<li>Tags should ideally be shared amongst a set of related branches. This has
come up often during discussions about where tags belong.</li>
<li>Management of sets of bzr branches is hard<ul>
<li>It seems to make sense for the configuration of several plugins to be
project-specific:<ul>
<li>bzr-pqm-submit’s pqm address</li>
<li>bzr-email target address</li>
<li>bzr-cia’s project setting</li>
<li>…</li>
</ul>
</li>
<li>may be useful to override whoami</li>
<li>having a way to group branches allows mass-pushes/pulls</li>
</ul>
</li>
<li>I often find that the public_location I set is almost the same for related branches, with only the last part of the url differing and containing the branch nick</li>
<li>It would be nice if “bzr register-branch” could automatically determine what product to register as</li>
</ul>
<p>I’m not looking for repositories:
- repositories can contain data from multiple totally unrelated branches. a tag “1.0” could conflict because there are multiple unrelated projects that have it.
- repositories are a storage optimization and I like them that way</p>
<p>although other projects (mercurial, git) seem to be using repositories to allow talking about a group of related branches.</p>
<p>I’m not looking for “just” directories:
- There’s no place in a directory to store settings or tags
- Having a long list of settings in ~/bazaar/locations.conf doesn’t scale and the settings won’t be able to propagate</p>
<p>Having another semantic object (”Product”?) on which options/tags can be set would help. Perhaps based on the root id (where available) ?</p>
<p><em>Currently Playing: Symphony X - The Odyssey</em></p>
SerNet asks Microsoft for specs2007-10-07T12:38:42+02:002007-10-07T12:38:42+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-10-07:191-SerNet-asks-Microsoft-for-specs.html<p>Now that the European court has decided SerNet <a class="reference external" href="http://www.heise.de/newsticker/meldung/96788/">has asked</a> Microsoft to publish specifications on <span class="caps">SMB</span>/<span class="caps">CIFS</span>.</p>
<p>Martijn has <a class="reference external" href="http://martijn.van.steenbergen.nl/journal/2007/10/07/kamelot/">posted some photos</a> of the concert we went to a couple of days ago.</p>
Podcast with the Samba team2007-10-02T20:23:26+02:002007-10-02T20:23:26+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-10-02:190-samba-podcast.html<p>Last week was the annual Samba get-together in the <span class="caps">US</span>. Jeremy and Leslie from
Google organized a somewhat more informal event than the <a class="reference external" href="http://www.cifs2006.org/"><span class="caps">SNIA</span> <span class="caps">CIFS</span></a>
interop lab that we usually attend.</p>
<p>Several Samba team members, including myself, took part in a <a class="reference external" href="http://googlesummerofcode.blogspot.com/2007/10/cifs-samba-and-summer-of-code.html">podcast</a>
about our involvement in the Summer of Code afterwards.</p>
Converting CSV files to QIF2007-09-07T01:42:09+02:002007-09-07T01:42:09+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-09-07:188-Converting-CSV-files-to-QIF.html<p>Recently I’ve started using <a class="reference external" href="http://www.gnucash.org/">GnuCash</a> for my personal finances. Eventually I got it running well, after I recompiled it myself, because the version in Debian was segfaulting on start-up (it is also very outdated).</p>
<p>My bank, <a class="reference external" href="http://www.postbank.nl/">PostBank</a> only supports exporting transactions in it’s own custom <span class="caps">CSV</span> files which, of course, aren’t supported by Gnucash. I wrote a simple Python script that converts PostBank <span class="caps">CSV</span> files to <a class="reference external" href="http://en.wikipedia.org/wiki/QIF"><span class="caps">QIF</span></a>, which Gnucash can handle ok.</p>
<p>I’m publishing the script here for others interested in importing PostBank <span class="caps">CSV</span> files in their finance apps:</p>
<pre class="code python literal-block">
<span class="ch">#!/usr/bin/python</span><span class="w">
</span><span class="c1"># Very simple converter from PostBank mijn.postbank.nl .csv files to</span><span class="w">
</span><span class="c1"># QIF files for use by (among others) gnucash.</span><span class="w">
</span><span class="c1">#</span><span class="w">
</span><span class="c1"># Written by Jelmer Vernooij <jelmer@samba.org>, 2007</span><span class="w">
</span><span class="kn">import</span> <span class="nn">csv</span><span class="o">,</span> <span class="nn">sys</span><span class="w">
</span><span class="n">rows</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">reader</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="p">)</span><span class="w">
</span><span class="n">header</span> <span class="o">=</span> <span class="n">rows</span><span class="o">.</span><span class="n">next</span><span class="p">()</span><span class="w">
</span><span class="k">assert</span> <span class="n">header</span> <span class="o">==</span> <span class="p">([</span><span class="s1">'Datum'</span><span class="p">,</span> <span class="s1">'Naam / Omschrijving'</span><span class="p">,</span> <span class="s1">'Rekening'</span><span class="p">]</span><span class="w">
</span> <span class="o">+</span> <span class="p">[</span><span class="s1">'Tegenrekening'</span><span class="p">,</span> <span class="s1">'Code'</span><span class="p">,</span> <span class="s1">'Af Bij'</span><span class="p">,</span> <span class="s1">'Bedrag (EUR)'</span><span class="p">]</span><span class="w">
</span> <span class="o">+</span> <span class="p">[</span><span class="s1">'MutatieSoort'</span><span class="p">,</span> <span class="s1">'Mededelingen'</span><span class="p">])</span><span class="w">
</span><span class="nb">print</span> <span class="s1">'!Type:Bank</span><span class="se">\n</span><span class="s1">'</span><span class="w">
</span><span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span><span class="w">
</span> <span class="k">if</span> <span class="s2">"-"</span> <span class="ow">in</span> <span class="n">l</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span><span class="w">
</span> <span class="n">p</span> <span class="o">=</span> <span class="n">l</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"-"</span><span class="p">)</span><span class="w">
</span> <span class="nb">print</span> <span class="s2">"D</span><span class="si">%s</span><span class="s2">/</span><span class="si">%s</span><span class="s2">/</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">p</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">p</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">p</span><span class="p">[</span><span class="mi">2</span><span class="p">])</span><span class="w">
</span> <span class="k">else</span><span class="p">:</span><span class="w">
</span> <span class="nb">print</span> <span class="s1">'D</span><span class="si">%s</span><span class="s1">/</span><span class="si">%s</span><span class="s1">/</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">l</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">4</span><span class="p">:</span><span class="mi">6</span><span class="p">],</span> <span class="n">l</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">6</span><span class="p">:</span><span class="mi">8</span><span class="p">],</span> <span class="n">l</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">:</span><span class="mi">4</span><span class="p">])</span> <span class="c1"># date</span><span class="w">
</span> <span class="k">if</span> <span class="n">l</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'Bij'</span><span class="p">:</span><span class="w">
</span> <span class="nb">print</span> <span class="s1">'T</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">l</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span> <span class="c1"># amount</span><span class="w">
</span> <span class="k">elif</span> <span class="n">l</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'Af'</span><span class="p">:</span><span class="w">
</span> <span class="nb">print</span> <span class="s1">'T-</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">l</span><span class="p">[</span><span class="mi">6</span><span class="p">]</span> <span class="c1"># amount</span><span class="w">
</span> <span class="k">else</span><span class="p">:</span><span class="w">
</span> <span class="k">raise</span> <span class="n">Error</span><span class="p">(</span><span class="s2">"Unknown field value '</span><span class="si">%s</span><span class="s2">'"</span> <span class="o">%</span> <span class="n">l</span><span class="p">[</span><span class="mi">5</span><span class="p">])</span><span class="w">
</span> <span class="nb">print</span> <span class="s1">'P</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">l</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="c1"># payee / description</span><span class="w">
</span> <span class="nb">print</span> <span class="s1">'M</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">l</span><span class="p">[</span><span class="mi">8</span><span class="p">]</span> <span class="c1"># comment</span><span class="w">
</span> <span class="nb">print</span> <span class="s1">'L</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">l</span><span class="p">[</span><span class="mi">7</span><span class="p">]</span><span class="w">
</span> <span class="nb">print</span> <span class="s1">'^</span><span class="se">\n</span><span class="s1">'</span> <span class="c1"># end transaction</span>
</pre>
Bzr-Svn2007-08-03T19:55:36+02:002007-08-03T19:55:36+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-08-03:185-Bzr-Svn.html<p>The next major release of <a class="reference external" href="http://bazaar-vcs.org/BzrSvn">bzr-svn</a>, 0.4, has now been released. The main
change in this release is that the behaviour of push is now intuitive. The big
hack that allowed push to somewhat work in the previous release has been
replaced by proper push which behaves the same way as it would against a Bazaar branch.</p>
<p>It’s now also possible to branch from non-standard branch locations such as
/foo in a repository and not necessarily standard locations like /trunk or
/trunk/foo. See the <a class="reference external" href="https://lists.ubuntu.com/archives/bazaar/2007q3/029189.html">release announcement</a> for a list of other changes.</p>
<p>It’s interesting to see what other people are saying about bzr-svn on their blogs:</p>
<ul class="simple">
<li><a class="reference external" href="http://blog.venthur.de/2007/04/20/reportbug-ng-migrated-from-svn-to-bzr/">Migrating to bzr using bzr-svn</a></li>
<li><a class="reference external" href="http://raraavisresearch.com/memo/archives/2007/06/09/saturday-night-2/">Missing support for svn:externals</a></li>
<li><a class="reference external" href="http://matthew.loar.name/blog/archives/2007/06/18.html">Issues with branching schemes</a></li>
<li><a class="reference external" href="http://swik.net/Ubuntu/Planet+Ubuntu/James+Henstridge:+FM+Radio+in+Rhythmbox+%E2%80%93+The+Code/77rb">using bzr-svn for rhythmbox</a></li>
<li><a class="reference external" href="http://blogger.malept.com/2007/03/awn-bzr-branch-bazaar-overlay.html">bzr-svn on Gentoo</a></li>
<li><a class="reference external" href="http://slipperysnippets.blogspot.com/2007/07/wherefore-art-thou-not-working-under.html">issues with Windows support</a></li>
</ul>
<p>Note that bzr-svn 0.4 has been tested on Windows and that branching schemes are now more flexible.</p>
LUG Radio Live2007-07-09T12:58:09+02:002007-07-09T12:58:09+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-07-09:180-LUG-Radio-Live.html<p>Just got back to the hotel after <span class="caps">LUG</span> Radio Live and the afterparty event. I
guess the best way to describe <span class="caps">LUG</span> Radio live is as a your average geek
conference but with a bit of a rock concert flavor: lots of cheering on,
groupies, swearing, simple humor and metal music being played in between
between sessions.</p>
<p>The talks were very good; in particular I liked Nat Friedman’s talk about,
among other things, the way the Oxford English Dictionary was developed
(although he still doesn’t seem to understand what is so bad about the
<span class="caps">MS</span>-Novell deal) and Aaron Seigo’s talk on <span class="caps">KDE</span> 4 (made my fingers itchy and want
to contribute..). The short talk Szilveszter and I gave on bzr-gtk went ok, but
was probably not all that entertaining right after Malcolm’s talk on zombies
and Ubuntu.</p>
<p>The major thing I disliked is the fact that the wireless was very crappy. I got
it to work on two occassions, just long enough to read a few emails. It took a
while to get used to the weird accent with which they speak English around here
and I seem to’ve picked up a few swearwords that I now have to unlearn again. Doh.</p>
<p>All in all really worth the trip from Amsterdam and definitely something to
keep in mind for next year!</p>
<p>Erik and I decided to stay an extra day to see a bit of Wolverhampton and
Birmingham, but it turns out this isn’t your typical tourist spot.</p>
Ohloh - Statistics on Free Software projects2007-07-03T14:51:40+02:002007-07-03T14:51:40+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-07-03:178-Ohloh-Statistics-on-Free-Software-projects.html<p><a class="reference external" href="http://www.ohloh.net/">Ohloh</a> is a nice web 2.0 site that contains stats on various Free Software projects. At the moment, they only support Subversion, <span class="caps">CVS</span> and Git. They’re <a class="reference external" href="http://www.ohloh.net/suggestions/new">open to feature requests</a> though. If enough people ask for it, hopefully they’ll support Bazaar at some point.</p>
OpenChange Interview2007-06-04T11:24:04+02:002007-06-04T11:24:04+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-06-04:173-OpenChange-Interview.html<p>Linux Weekly News had a good interview with Julien Kerihuel, lead developer of
OpenChange, two weeks ago. It’s now <a class="reference external" href="http://lwn.net/Articles/234642/">also available</a> for those who are not subscribed.</p>
Using a pqm with Subversion2007-05-25T12:04:14+02:002007-05-25T12:04:14+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-05-25:172-Using-a-pqm-with-Subversion.html<p>One of the things that I’ve always missed in <span class="caps">DVCS</span> is the ability to refuse
commits in a branch that’s shared by multiple people based on a test suite run.
Sure, it’s possible to have a pre-commit hook - but that would mean that you’d
have to wait for the full test suite to run until the commit finishes.
With the time it takes to run the Samba testsuite, this is not really an option.</p>
<p>One of the things that would work is to have everybody work in a separate
branch and then have some sort of tool that merges those revisions from
everybody’s personal branches that worked ok. However, to my knowledge, there
is no such tool for Subversion.</p>
<p>Bazaar uses a tool called <span class="caps">PQM</span> (Patch Queue Manager). <span class="caps">PQM</span> usually controls the
main branch (for example for Bazaar, it controls bzr.dev), and waits for
<span class="caps">GPG</span>-signed requests to merge a specific revision into that main branch. Before
accepting such a revision, it will try to run the testsuite to make sure it
passes. This guarantees that the main branch never contains broken code (as far
as can be indicated by the testsuite).</p>
<p>Now that bzr-svn supports true push, it is possible to actually use a <span class="caps">PQM</span> with
a Subversion branch. I’ve tried it on a smaller branch last week, and am now
looking into using this for my Samba work.</p>
London2007-05-19T14:56:19+02:002007-05-19T14:56:19+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-05-19:171-London.html<p>I’m currently doing a bit of sightseeing in London, after attending the
Bazaar sprint at the Canonical office. It was a good sprint, and quite
different from the previous ones - in that there was only a limited amount of
actual coding involved. The view from the Canonical office is magnificent, so
we were even able to do some sightseeing while working…</p>
<p>Bazaars’ focus has previously mainly been on correctness and features. The
first has always been one of our strengths, and we’re in pretty good shape
regarding the second. Performance has been one of the main complaints from
users about Bazaar and so we have recently tried to improve in that area.</p>
<p>Since 0.12, we have already tried to optimise some of the common code paths and
some people have been working on a high performance smart server (to speed up
remote operations).</p>
<p>During the first two-and-a-half days of the sprint, we analysed 20
of the most common use cases with Bazaar and determined what complexity they
should ideally require to be able to work. After this analysis, we looked at
ways to change our data structures to reach these goals.</p>
<p>I have mainly been a spectator during the latter parts of these discussion, but
they were interesting to follow.</p>
<p>One of the things I worked on was support for true push in bzr-svn. This was
one of the bugs that has bitten a lot of users of bzr-svn. The upcoming bzr-svn
0.4 now supports true push as well as commits in heavyweight checkouts. I hope
to release 0.4 after adding nested tree and ignores support so that I don’t
have to change the internal mapping mechanisms again.</p>
<p>And now, it is time for some more sightseeing. After that I hope to get
back to the reason I’m doing all of this in the first place: Samba!</p>
SambaXP: a year with leaps of progress2007-04-30T11:35:00+02:002007-04-30T11:35:00+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-04-30:169-SambaXP-a-year-with-leaps-of-progress.html<p>Like most of the Samba team, I attended <a class="reference external" href="http://www.sambaxp.org/">SambaXP</a> in Goettingen again last week. This year we were able to announce a couple of big leaps forward:</p>
<div class="section" id="openchange">
<h2>OpenChange</h2>
<p>Julien’s talk about <a class="reference external" href="http://www.openchange.org/">OpenChange</a> was probably the most interesting and surprising one this year. After three years of hard work (I remember spending a lot of time in front of a beamer looking at network packets in 2004 when they started), the OpenChange team has figured out the enough bits of the <span class="caps">MAPI</span> (Exchange/Outlook) protocol that they can fetch and send emails using an Exchange server.</p>
<p>At the moment, there is a simple command-line client that can fetch into a mbox file and send mails in a way somewhat similar to the “mail” command. There also is a highly experimental plugin for evolution.</p>
<p>Most work will now have to focus on the most interesting bit - providing a Free Software implementation of a <span class="caps">MAPI</span> server.</p>
</div>
<div class="section" id="drsuapi">
<h2><span class="caps">DRSUAPI</span></h2>
<p>Metze announced that he’s been able to work out how the directory replication protocol that is used between <span class="caps">DC</span>’s in an Active Directory environment works. This is very important, because we need that protocol in order to be able to live as a <span class="caps">DC</span> in an Active Directory environment that has more than one <span class="caps">DC</span>.</p>
<p>Halfway through preparing our (now yearly) talk about the status of Samba 4, Andrew and I realized we’re actually not too far away from being able to do the first alpha of Samba 4. We hope we can release one in the next few months.</p>
</div>
UK Summer2007-04-16T12:22:57+02:002007-04-16T12:22:57+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-04-16:166-UK-Summer.html<p>Great Britain appears to be the place to be this summer; a lot of the annual
conferences are held in various cities around the <span class="caps">UK</span> this year: <a class="reference external" href="http://www.debconf.org">DebConf</a>
(Edinborough, 17-23 June), <a class="reference external" href="http://www.guadec.org/"><span class="caps">GUADEC</span></a> (Birmingham, 15-21 July), <a class="reference external" href="http://akademy.kde.org/">Akademy</a>
(Glasgow, 30 June-7 July) and last but not least the
next <a class="reference external" href="http://bazaar-vcs.org/SprintLondonMay07">Bazaar sprint</a> (London, 14-20 May).</p>
<p>I’ll be attending at least the Bazaar sprint and <a class="reference external" href="http://www.lugradio.org/live/2007/index.php/Schedule"><span class="caps">LUG</span> Radio Live</a>, and I
hope to be present at <span class="caps">GUADEC</span>, time permitting.</p>
Bazaar for dvcs users2007-03-31T22:36:12+02:002007-03-31T22:36:12+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-03-31:165-Bazaar-for-dvcs-users.html<p><a class="reference external" href="http://changelog.complete.org/">John Goerzen</a> has been evaluating various version control systems and his
blogs posts on the subject have been interesting. One of the most interesting
points he raises about Bazaar is that there is no “darcs send”-like support.
The <a class="reference external" href="http://launchpad.net/bzr-submit">“submit”</a> plugin gets close, but I think it misses some important
features - and I agree it should be part of the core.</p>
<p>I’m not sure that the merge support in Bazaar is worse than that in Mercurial
or Git. I’ve never had problems with it, and I use it <em>a lot</em>. Performance,
other than remote access, is also no longer a problem for medium size projects
as of 0.15. <em>Update</em>: it’s no longer as much of a problem as it used to be.
Remote access is still significantly slower than it can be, but people are
working hard on fixing that problem.</p>
<p>It’s also interesting to see that the Bazaar user interface and documentation
is apparently confusing to people who are mostly familiar with other
distributed version control systems rather than centralized systems. We
definitely need to work on that.</p>
<p>The fact that John doesn’t come from <span class="caps">CVS</span> or Subversion also explains that
checkouts don’t appeal to him.</p>
Bazaar and Subversion nested tree support2007-03-31T22:15:56+02:002007-03-31T22:15:56+02:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-03-31:164-Bazaar-and-Subversion-nested-tree-support.html<p>As of 0.15, Bazaar will have initial support for nested trees. Nested trees
basically allow you to add a ‘magic’ directory to a Bazaar branch that in
itself is another Bazaar branch. A very good use case of this is
if you have a project for which you need to include some library (e.g. popt).
Nested trees allow you to add a reference to the upstream popt branch, avoiding
the need to synchronize every time upstream fixes a bug.</p>
<p>Subversion has similar functionality, although the term used there is
‘externals’. Externals are quite easy to use - using them only requires setting
the svn:externals property.</p>
<p>I’ve done some early work on supporting externals in bzr-svn and mapping them
to nested trees. At the moment, bzr-svn is able to track nested trees that are
being added. Removes and changes to existing nested tree locations are still on
my todo list, but will definitely be supported in bzr-svn 0.4.0.</p>
Bazaar 0.152007-03-20T12:58:43+01:002007-03-20T12:58:43+01:00Jelmer Vernooijtag:www.stationary-traveller.eu,2007-03-20:162-unknown.html<p>Since I’ve upgraded serendipity to version 1.1, it seems to keep losing my draft blog entries (either that, or hiding them somewhere I can’t find them). For that reason, this blog post is a bit short, as I’m too lazy to rewrite the original one I had drafted up.</p>
<p>Bazaar 0.15 is promising to be a really great release. Several new features are going to land: dirstate (fast working tree access), tags and initial nested tree support.</p>
<p>I miss travelling.</p>
<p>cp: Smashing Pumpkins - Ava Adore</p>