Once upon a time, there was Deb and Ian. That was about exactly 18 years ago. We don’t talk about the 0.939000 format any more, but they eventually settled on:

	$ ar rc pkg_1.0_all.deb debian-binary control.tar.gz data.tar.gz
	$ hexdump -C pkg_1.0_all.deb | head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62  |!<arch>.debian-b|
	00000010  69 6e 61 72 79 20 20 20  31 33 31 33 36 38 33 35  |inary   13136835|
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20  |100644  4       |
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e  |  `.2.0.control.|
	00000050  74 61 72 2e 67 7a 20 20  31 33 31 33 36 38 33 35  |tar.gz  13136835|
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20  |100644  1391    |
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59  |  `............Y|
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5  |.o.6......:^.t..|

By then, systems were a.out(5), and everything was good. (Of course, if you look at the mtimes, you’ll notice I faked this. But it’s really equivalent to the real thing.

But oh horror! GNU binutils, not always everyone’s friend, switched from using BSD style “Unix Archiver” libraries in ar(1) to SYSV style libraries on elf(5) systems:

	$ ar rc on-elf debian-binary control.tar.gz data.tar.gz
	$ hexdump -C on-elf | head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62  |!<arch>.debian-b|
	00000010  69 6e 61 72 79 2f 20 20  31 33 31 33 36 38 33 35  |inary/  13136835|
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20  |100644  4       |
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e  |  `.2.0.control.|
	00000050  74 61 72 2e 67 7a 2f 20  31 33 31 33 36 38 33 35  |tar.gz/ 13136835|
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20  |100644  1391    |
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59  |  `............Y|
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5  |.o.6......:^.t..|

Can you spot the difference?

Of course, ELF is what you want™, so there is little choice. Unix Archiver libraries are system dependent, and no format has ever been normed, but DEB files use it as format… so what is one to do?

	$ GNUTARGET=a.out-i386-linux ar rc with-aout \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum pkg_1.0_all.deb with-aout on-elf
	248f78d42f8ca8f2a3560f9800b2bf01  pkg_1.0_all.deb
	248f78d42f8ca8f2a3560f9800b2bf01  with-aout
	09eca70c9b11b6b55bbadcab5c3201fb  on-elf

“OK, and what do I do on my Debian/m68k system?”

ar(1) uses bfd, and GNU binutils can not only forcibly set the target emulation but also show them:

Wow. While binutils share no single supported working target, they can be built multiarch, or (on MirBSD) with --enable-targets=all --enable-64-bit-bfd. Doesn’t help if you want to stay portable: GNUTARGET=srec is common on all Debian (sid) binutils versions (single or multiarch), but errors out on older binutils. The a.out-* targets are not common. Sure, you could hack around things, but… this is tedious. If you follow things or know me a little, you might already have guessed that I wouldn’t let that stand.

pax(1) to the rescue. On MirBSD, we use paxtar, which has cpio(1) and tar(1) front-ends and supports multiple formats (4 cpio and 2 tar variants) and has already been extended a lot and is lovingly called paxmirabilis (mirabilos’ peace in Latin)  — it has options to anonymise archives: set uid and gid to zero, set mtime to zero, (for ustar) only write the numeric uid and gid to the archive, (for cpio formats) serialise inodes and device information, write content of hardlinked files only once (breaks partial extraction but saves a lot of space, e.g. 2 MiB off the Grml initrd.gz). And, recently, the ability to append a trailing slash to pathnames of ustar members which are directories (GNU tar does it — and I thought some Debian utilities check for it). So why not… (the -M dist and fakeroot set the uid/gid to 0)

	$ find debian-binary control.tar.gz data.tar.gz | \
	> mircpio -oHar -Mdist >with-mircpio
	$ mirpax -w -M dist -f with-mirpax -x ar \
	> debian-binary control.tar.gz data.tar.gz
	$ mirtar -M dist -A -cf with-mirtar \
	> debian-binary control.tar.gz data.tar.gz
	$ GNUTARGET=a.out-i386-linux fakeroot ar rc with-aout-ar \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum with-*
	a466e2fd57cdee141fe585a43245548f  with-aout-ar
	a466e2fd57cdee141fe585a43245548f  with-mircpio
	a466e2fd57cdee141fe585a43245548f  with-mirpax
	a466e2fd57cdee141fe585a43245548f  with-mirtar

Voilà. I got it, and even appending is possible. It supports the BSD format with special focus on DEB files, and deals with long filenames, but not symbol or filename tables (used by ranlib(1) or strange formats, respectively, but since we don’t create *.a files to use with some native linker/binder/loader, we don’t need that anyway).

On extraction (oh, and listing!) it deals with SYSV style filenames as well.

	$ mirtar tvf on-elf
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz
	$ mirtar tvf with-aout
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz

One of the real benefits is that you can use the front-ends interchangably — for example, “mirtar tzf foo.cpio.gz” would work (which GNU tar can’t do), and mircpio’s ustar implementation, unlike GNU cpio’s, is not horribly broken.

Of course, there are some drawbacks: it’s not GNU tar or GNU cpio, so there are absolutely zero --long-options. Some of their features are missing (but tar’s -O is implemented now), so it’s no replacement (but very well usable alongside it). The format called pax, committee-designed to replace ustar, isn’t yet supported ironically, but that’s on the TODO.

So, what do you think?

	tg@frozenfish:~/Debs/dists/sid/wtf/Pkgs/mircpio $ ll *.deb
	-rw-r--r-- 2 tg freewrt 78140 Aug 17 11:04 mircpio_20110817-0wtf2_amd64.deb
	-rw-r--r-- 3 tg freewrt 72262 Aug 17 11:00 mircpio_20110817-0wtf2_i386.deb
	-rw-r--r-- 1 tg freewrt 67446 Aug 17 18:21 mircpio_20110817-0wtf2_m68k.deb

Should I upload this to Debian proper?

As for the licence: 3-clause UCB (and 2-clause BSD, which is a subset of it), so no problem. I’m asking because the other package which I had been using for a long time and not uploaded, jupp, got uploaded recently (during DebConf) on user input (people wondered why it did not yet exist in Debian proper). I guess the old saying “if it’s not in Debian, it doesn’t exist” holds true in many parts of the OSS world.

It’s up to date wrt. standards btw, and lintian-clean save for two pedantic-class warnings (no upstream changelog file, no homepage link) which aren’t fulfillable (could link this wlog entry as homepage).

If you know Alioth you’re familiar with the software formerly known as SourceForge, formerly known as GForge, currently known as FusionForge. My employer both uses it and contributes to it, we run an adapted (mostly themed, prototyping new functions that often end up in FusionForge itself, and backporting functions from FF to our “production codebase”) version.

I’ve backported the extratabs plugin to appease project managers and other non-technical people while we move our codebase to FF 5.1, and I did so on an installed version of the plugin rather than the source because the latter was tightly integrated with rather heavy packaging style changes.

	[…]
	dh_builddeb
	# create fusionforge-plugin-extratabs binary package
	toplev=$$(pwd); cd plugins/fusionforge-plugin-extratabs; \
	p=$$(print -r -- $$(sed -n '/^Package: /s///p' C/control | head -1)); \
	v=$$(print -r -- $$(sed -n '/^Version: /s///p' C/control | head -1)); \
	a=$$(print -r -- $$(sed -n '/^Architecture: /s///p' C/control | head -1)); \
	d=$${p}_$${v}_$${a}.deb; \
	rm -f $$toplev/../$$d control.tar.gz data.tar.gz; \
	(cd control; find . | fgrep -v /.svn | sort | \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash) | gzip -n9 >control.tar.gz; \
	(cd data; find . | fgrep -v /.svn | sort | \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash) | gzip -n9 >data.tar.gz; \
	mirtar -M dist -Acf $$toplev/../$$d debian-binary cont*gz dat*gz; \
	rm -f control.tar.gz data.tar.gz; \
	cd $$toplev; dpkg-distaddfile $$d non-free/devel optional

The hardest part of extending debian/rules with that was to get the autobuild and dpkg-distaddfile call right. This works, even though I’d call it a temporary kludge. (No need to tell me I should have used && — I know. And I only shell out to mksh(1) because the “inner” part was already there from before, when I still used ar(1). This was slightly edited for the wlog.)

In the meanwhile, apt-extracttemplates can deal with SYSV style filenames in DEB files — on Debian sid, but not on K?buntu hardy, which some people are using as Desktop OS still…

Update 03.03.2012 — Jonathan Nieder replied quickly with a suggestion to instead take over the “pax” package in Debian. Eventually, I uploaded pax (1:20120211-1) from the former “mircpio” package to Debian, after I managed to talk to its previous maintainer Bdale Garbee (thanks for handing over). It is now present in Debian wheezy and Zubunt! precise as /bin/pax with /bin/paxcpio and /bin/paxtar offering the other interfaces.