Thank you very much.
Thanks everybody for coming,…
If you are packaging software and you want me to work on with you,
this is how you can do that.
It is a very self-centered talk:
I just want to explain some of the things that I like about,
some practice that I prefer about Debian packaging,
and I don't pretend this is any sort of…
official, permanent or final thing.
I just wanted to share some ideas that I have about
the way that I work with packages, in the hope that maybe…
For two hopes:
One is that I hope that I can show you something that you have not heard of,
or maybe you were doing differently,
or maybe you think it is the right think to do
and it is just nice to see somebody else doing it.
My second hope is that you can tell me what I am doing wrong,
and you can help me learn and improve on my own packaging techniques.
If you see something that I am proposing up here,
and you think there is a problem with it, I would like to hear about it too.
I just want to see more of the culture within Debian,
of people who are doing packaging, explaining what they are doing,
and so I thought I would just step up and explain:
"Here is some of the practice that I do",
In the hope that other people will do the same and explain what they are doing,
and maybe they can learn from me and I can learn from them.
Without much further ado I am just going to dive into it.
If you have questions, I am perfectly happy to be interrupted,
we have some folks with walking mics in the crowd:
you can just raise your hand.
If you have got a question or an interruption or whatever,
that is fine.
I doubt I will go the whole 15 minutes, I think there are 20 minutes,
I doubt I will go the whole time, so there will be also
time for questions at the end if you prefer.
But I do not mind being interrupted.
So, this is all on this web page here,
you could probably skip this talk and go read the web page,
but then you would not have the nice in-person interactions,
and it is easier to tell me that I am wrong in person,
so I would like to have that happen.
I put this up on the Debian wiki,
because I want anyone to be able to find it.
If you think you have got some good ideas, you should put it on the Debian Wiki too:
other people can take advantage of the ideas that you have got.
First baseline is: I really like revision control.
And I know that it makes me a certain flavor on nerd,
but when we are working with things that are as complicated as software packages,
hmmm…
I think a lot of people don't get that in Debian you are not
just working on one software package:
you are actually probably, if you are doing a responsibly work,
on at least two software packages, and maybe 5.
So you have got the version that is unstable,
and you have got the version that you try to maintain for stable as well.
And we are committing to doing maintenance work.
A lot of our work in the project is janitorial in nature:
we want to clean up the mess and we want us to stay out of the way
and make sure things work, functionally,
for people who are relying on the operating system to not get in their way.
So revision control I think is really helpful because it means you can
keep track of what changes you have done on different branches of the project
while you are maintaining both of them.
Basically, I'd require working with the revision system I am comfortable with,
I prefer Git, I am not going to have a religious war about it.
If upstream uses Git, I am even happier, and I try to make
my packaging depend on upstream's revision control.
I like to use 'git-buildpackage', and I like to use it with debhelper.
If you have not tried out 'git-buildpackage',
we are going to have a 'git-buildpackage' skill share session
later on today, and I welcome you to come and share your tricks with it,
or learn some tricks from other people.
It is a particular way that you can keep your Debian packaging in a Git repository,
and it helps you to keep track of all of the changes that have happened within
your packaging and within upstream to make sure you are not accidentally
making other changes.
So it is very easy to go back and review what you have done.
I find that really useful.
I definitely also like to keep upstream's source code
in the same revision control system.
I like to keep the tarballs in the revision control system
because it means that if someone is interested,
they can uses a tool called 'debcheckout'.
You can use 'debcheckout' with a name of a package:
you just say "I am really interested in package 'foo',
let me see the source code for that": 'debcheckout foo'
You get the source code, and you get the source code…
from a revision control system that you can now track
and you can just propose changes on.
You can also extract the tarball from that revision control system.
'debcheckout' actually works even if you do not have upstream stuff in there,
but I like to keep it all in one revision control system,
it is just easier to find everything when you want.
Some of these things that I prefer have to do with
what the upstream software developer has done,
so I am less inclined to try the package, an upstream software project,
if they just throw tarballs here over the wall
to an FTP side every now and then.
It makes it more difficult for me to know what they are doing,
and why they are doing it.
So i like it, I have already said, when upstream uses Git,
I also like it when upstream signs their releases,
and says "hey, this is specific release",
because that is a signal that I can use,
or somebody else that understands the project.
As said, "we think that this is something that other people can use",
or "this is a particular version that we would like other people to test".
There are a lot of other situations where maybe it is not so important.
And having that be cryptographically signed is really useful.
I care about cryptographic signature on software because I want to know that
what I am running is related to the code that somebody else out should be run.
And if you don't verify your software cryptographically,
anyone who can intercept the network connection
between you and that software, can modify the software before it gets to you.
And the cryptographic signature just says:
"look, this is a version that I am OK with.
I am putting it out there and it comes from me".
So I can have a trace back to that point.
Just let me talk about briefly about how you do cryptographic verification
of upstream One is you might know upstream:
you might know them personally, you know their key already, that is fine.
That is not the usual case: we work on the Internet.
In the situation where your upstream is signing their tarballs
and you have not met them, you do not have to sign their key,
you do not have to say "I announce this is their key".
But it is probably the same one that is signing every release,
so you should keep track of that.
Debian has a nice way to keep track of that:
you can tell Debian how to find the new version of the upstream tarball.
This is in the Debian 'watch' file.
If you type 'man uscan', you can learn more about Debian 'watch',
and Debian 'watch' now has a feature that lets you say
"that is not only this way you find the tarball,
but upstream publishes signatures and the signatures look like this".
You know, they got a '.sig' at the end.
So there is a particular arcane way to specify that, but if you specify that,
then 'uscan' can find not only the upstream tarball,
it can find the upstream signature.
And, if you drop upstream's signing key
- which of course I did not put on the wiki page,
someone should edit that and fix it -
you can put the upstream signing key in 'debian/upstream/signing-key.asc'.
And then if you do that, when you say 'uscan', you can tell…
Maybe some people here do not know how to use 'uscan':
'uscan' is a very simple tool,
you run it from a software package that has a 'debian' directory,
or even one level up if you keep all of your software packages in one folder.
You can go one level up and say 'uscan',
and it will look in all of the folders that are children of it,
and look for new versions
by trying to find a new upstream version in 'debian/watch'.
And if you have configured 'debian/watch' properly,
it can find the new upstream signatures,
and if you have got the 'upstream/signing-key.asc',
then it will actually verify the signatures for you
as part of fetching the new upstream tarball.
So you can get all of those things just by setting a pre-packaging that way.
There is a hand up down there, could we get the mic down to the hand ?
Thanks.
Or to the person who has that hand, it is not just a hand.
[public laugh]
[attendee] Publish a tarball, and a hash, '.sha1',
and sign that hash, '.sha1.asc'.
Can 'uscan' cope with this and check the signature on the hash
and that the hash belongs to that tarball ?
[Daniel] I do not believe that 'uscan' can do that currently.
So anybody out there who wants to make things better for the world
should go hack on 'uscan': that is a pretty straightforward thing
that we should fix because I agree that is common pattern.
[attendee] I have no answer to this question by I have another question:
how do you convince upstreams who do not release tarballs
or who do not set tags in Git ?
[Daniel] Who do not make tags in Git ?
[someone] Yes, if there is no tags you can not check out a tarball.
Is there any good way to convince upstream to do this ?
[Daniel] Git has this nice feature, which is that you can create a tag,
which is associated with a particular revision,
and you would like to have a tag
everywhere that a tarball has been released from.
I am tempted to pull up a Git view and show people some tags.
The question that you ask is a social one tho, not just a technical one,
and I actually find that my upstreams are pretty responsive.
Usually I frame my request as
"hey, it like you made this tarball from this particular commit 'id'.
If you could tag you releases, it would be really helpful to me,
and here is the command that I would use to tag the release".
And I say "git tag…"
and of course I can never remember so first I look it up,
but it is either 'tag name' 'commit id' or 'commit id' 'tag name'.
But I would look it up and I would write the email so that
all they have to do is they read it,
understand my argument,
and execute one command.
I mean, it doesn't get them in the habit but it start them towards it
And if you say 'tag -s',
then your tag will be cryptographically signed,
which I think is a really good thing to do too.
So, cryptographic verification of upstream.
As I said, I want to keep upstream's code in the revision control system.
I also like to keep…
In my ideal case upstream is using Git: I am using Git for packaging.
I actually like to keep upsteam's Git history fully in my repository,
so that I do not just have the tarballs, but I actually have all of their commits.
And that turns out to be really useful for two specific cases:
In one case, there is a common scenario where upstream will fix a bug,
but they have not made a release yet.
And that bug is really, really obviously problematic
for the folks who are using Debian,
so I want to fix it.
All I can do, because I have their full revision history,
I can use Git to "cherry pick" the upstream commit.
And then I "cherry pick" that upstream commit
and I can have it applied separately,
and release a Debian version that has the fix,
even before upstream has made a release with the fix.
So one nice thing about having upstream revision
is that I can pull fixes from upstream before they decided to release it.
The other advantage is the other way around.
Often when I am doing packaging, I discover a problem,
and maybe I can fix the problem.
And in fact maybe I am already shipping a Debian package that fixes the problem.
If my Debian fixes can be directly applied to upstream,
then I can use whatever their preferred upstream patch submission guidelines are,
whether it is a Github pull request, or a patch to a mailing list,
or a "hey can you pull this from my Git repository over here", e-mail…
The fact that I am using the same Git history that they are using
makes it much easier for me to push my changes back to them.
So, it sort of smooths the interaction if you can consolidate
and use the same revision control system as their.
Towards that aim,
I use a system now called 'patch queue',
which is part of 'git-buildpackage'.
So 'git buildpackage' is 'gbp',
'patch queue' is 'pq',
so to deal with 'patch queue' you say 'gbp pq'
and then you have some commands.
And what that does, is it takes…
How many of you are Debian packagers ?
How many of you package software for Debian ?
[most attendees raise their hand]
A very large percentage, but not everyone.
I hope some folks are considering starting packaging if you have not done it yet.
Of those of you who package software,
how many of you package software with modifications,
how many of you ship a modified version of upstream sources ?
[most attendees raise their hand]
Beyond the 'debian' directory, just Debian patches ?
So the common way to do that,
for the Debian 3.0 quilt packaging skill,
is that in your 'debian' directory you have a 'patches' sub-directory
that has a set of individual patches that apply certain changes,
and they are applied in order based on the file called 'debian/patches/series'.
So maintaining that is kind of a drag when upstream makes big changes:
then all of sudden you have got this set of patches and they do not quite apply…
It is a drag even you do not have it in the 'debian/patches/' directory.
But what Debian 'patch queue' does is it maps that directory of patches
into a little branch on your Git revision history.
So when you get a new upstream version, you can say 'patch queue rebase',
and it treats it just as Git: it takes the 'patch queue'…
You have already imported the new version, and it re-applies your patches,
and sometimes that means some minor adjustments.
Git is really good at figuring out what the right minor adjustments are to make,
and so all of the sudden the patch queue is re-based,
you refresh it in your revision control system…
and there you go.
So I like to use git-buildpackage patch queue,
tagging, as already brought up, thank you for that,
I like to to tag everything that I release,
I like to push that as soon as I can,
so that other people who are following my work
can know where my releases come from.
The reason that I like other people following my work is
they can fix my bugs easier.
I make mistakes, everybody makes mistakes,
and it is really important to me that if someone catches one of my mistakes,
I can accept their feedback, their criticism, their improvements,
as easily as possible.
I want a low barrier to entry for people to help me fix my problems:
it is selfishness.
So I try to patch it and publish these things for people can find it.
I'm going to rattle through some of these pretty fast because were are almost out of time.
I like to put my repo in some place where other people get to the them,
at the moment I like to put them in 'collab-maint',
it has some problems but it is better than not publishing your stuff,
and it is nice because it is sort of a public use.
I like to standardize how of my branches are named,
so if I am working on something that has got a stable version,
that is for Jessie, I will name the branch 'jessie',
because I will probably making changes, like I said, editing multiple of software
I try to push as frequently as I have made something that looks sensible.
I do not feel obliged to push my commits to a public repository
when I am still experimenting,
I actually really like to experiment,
and I also like to keep track of my experiments while I am doing them.
So I try to push when there is a sensible set of changes,
and I am trying to get myself
to a point where I can understand what I have done, even if it is wrong.
If I can get myself to a conceptual point where it is done,
I will push my changes so other people can see what I am working on,
and then work from there.
That is OK to push something that is wrong,
as long as you push something that people can understand.
When you make a 'git commit' (if you are working with Git),
one of the things that helps me to think for commit messages…
People often think that commit messages should say "what change you made".
I think that the 'git patch' shows what change what change you have made,
and I thin your commit messages should say "why you made the change".
That is what people really want to read.
If you need to explain technically why the thing that you did
maps to the conceptual thing that you wanted to do,
that is fine: do that in your commit message too.
But it is really important to say why you made the change.
It is not just like "initialize variable to 'no'":
OK, we can see that from the patch,
but what you are really saying is "there was a crash if someone did 'x',
and we are avoiding that crash by setting this to 'no'.
So I like to send patches via email,
so I try to configure Git email,
which make it really easy to just push patches back upstream.
If I am starting taking over a project that somebody else has past on,
and they did not use Git, I will try to restore all of the imports.
I would be happy to talk with people about how to do that,
if you have questions come find me.
I like to keep my files nice and simple:
there is a tool 'wrap-and-sort',
that just canonicalizes your files to make them look
in a simple and sensible way.
And it is nice because it means that everything is…
It does things like alphabetize your list of build depends,
and brake them out one per line.
The nice thing about that, since you are using revision control,
when you make a change to your build depends,
the changes become very easy to see:
"oh, they added one new package here, there is a single '+'".
One new dependency, they removed a build dependency
so you can see that kind of thing.
I like to use DEP-5 to format Debian copyright to be machine readable,
it is nice for people who are doing scans of the archive
and try reason about what the patterns are,
and licensing of free software.
And if I am doing something really crazy, that is going to make a big change,
I like to use a feature branch in revision control.
So we have got one minute left, I want to open it up for other questions.
it's kind of rambling
[attendee] You said you are using 'wrap-and-sort' which is nice,
I had learned that Config::Model editors - 'cme' - do the same job,
and somehow does a better job: it also enhances standard version
if it does not fit, or it makes VCS fields properly has it should be.
'cme fix dpkg-control' fixes your control file.
[Daniel] 'cme' ? And it is in what package ?
[attendee] The package 'cme', in unstable is cme
In Jessie it's libconfig-model-perl
[Daniel] You are developing in unstable, that is OK.
'cme' OK, thank you.
Other questions or suggestions, or complains ?
[attendee] If you change the original source code, and do some commits,
how do you convert that into a series of quilt patches ?
[Daniel] I use 'patch queue' for that as well, so what I do is I say
"I want to move over to my 'patch queue' view of the tree",
and then I make may changes, I make my commits,
and then I say 'gbp pq export',
so that 'patch queue export',
and it takes the 'patch queue' that I am on
and dumps it back into the Debian patches directory.
If you have not use 'gbp pq', I recommend looking into it.
It takes a little while to get used to, and I still screwed it up sometimes,
but it makes easy to fix your mistakes too.
[organizer] Last question ?
[attendee] Do you think it is possible
to make this 'patch queue' branch "pullable" by upstream ?
[Daniel] I do not actually think it is possible to make it directly
"pullable" by upstream: I think upstream can cherry pick patches from it,
but I do not see how to make it "pullable".
If someone else does, I would be happy to learn.
[organizer] This was "before last", so last.
[attendee] Do you have a recording of you using the tools that you mentioned,
a video recording would be great, just to show your workflow ?
[Daniel] I do not really know how to do that:
if somebody wants to help me do that I would be happy to do it.
I am going to give one last plug,
I know we are out of time here, sorry.
This tool is called 'gitk'.
This is an example…
- sorry we should leave -
but this the way that I visualize my revision control system.
We could do a whole other session about 'gitk'.
If you do not try to visualize your revision control system,
you are missing out:
I recommend try to find a way to visualize stuff,
find one that works for you.
Thanks for coming.
[organizer] Thank you.