(Dieses Posting wurde ursprünglich im internen Blog veröffentlicht, ist jedoch hierhin umgezogen. Es sind keine sensitiven Informationen enthalten, allerdings können die Links nur im Firmennetz angesteuert werden.)

Neues aus der Adminstube (und vom Evolvis-Team) für unsere Entwickler, die wir, nachdem wir die Projektmanager schon bedient haben, auch nicht zu kurz kommen lassen wollen:

Wenn ihr einen Jenkins-Job habt, der Debian-Pakete (oder für Derivate) erstellt, könnt ihr ab sofort (mindestens) ein Repository pro Job haben, in dem ihr die Pakete „veröffentlichen“ könnt. Diese Repositories werden automatisch signiert, und durch tarent-keyring werden die Signaturen bereits seit Dienstag verteilt, sodaß auch SecureAPT funktioniert.

Im folgenden wird die Arbeitsweise am Beispiel von Saschas WIP-Portalinstaller erklärt.

Ausgangspunkt ist ein Jenkins-Job, in diesem Fall auf dem test-hudson, hier mit dem Namen „portal-setup“, der unter „Build“ den Punkt „Execute shell“ aktiv hat. Hier wird durch diverse Kommandos ein Debian-Paket (Source und Binary) gebaut, das wirklich wichtige hierbei ist folgender Ausschnitt:

	cd portal-setup-$pv
	dpkg-buildpackage -rfakeroot -us -uc
	cd ..
	#rm -rf portal-setup-$pv

Der Befehl dpkg-buildpackage übernimmt das Bauen und schmeißt die erstellten Pakete (Source *.dsc und Binaries *.deb) und, ganz wichtig, die *.changes-Datei, ins Elternverzeichnis (daher die cd-Aufrufe). Nun möchte Sascha, daß diese Pakete einfach von Kollegen getestet werden, und ändert daher den Codeschnipsel so ab:

	cd portal-setup-$pv
	dpkg-buildpackage -rfakeroot -us -uc
	cd ..
	#rm -rf portal-setup-$uv portal-setup-$pv

	/opt/mvn-debs/mvndput.sh portal-setup squeeze main *.changes

Im ersten Teil bleibt alles beim alten, aber ein neuer Befehl ist am Schluß hinzugekommen. Was macht der?

Nun, /opt/mvn-debs/mvndput.sh ist die Magie, die unterhalb des Pfades /opt/mvn-debs ein Debian-Repository mit dem Jobnamen portal-setup anlegt. Wir publizieren Pakete in die „dist“ squeeze, und darin in die „suite“ main. Das ist hier lediglich eine Konvention; normalerweise heißt die dist so wie die Debian-Version (sarge, dapper, etch, hardy, lenny, squeeze, …), für die das Paket gedacht ist, und die suite ist eine Unterkategorie — also zum Beispiel main/contrib/non-free oder main/restricted/universe/multiverse oder wtf/tarent/evolvis (in unserem Adminrepo). Man kann die aber auch frei Schnauze nennen (ich hab zum Beispiel im m68k-Repo eine dist namens „cross“, unterhalb derer eine Cross-Toolchain liegt).

Das letzte Argument ist *.changes, was die Shell für uns expandiert: alle Dateien, die auf „.changes“ enden, werden dort (ASCIIbetisch sortiert) der Reihe nach aufgelistet.

Das Skript prüft nun zunächst die Namen und Dateien auf Plausibilität (es sind schließlich immer nur gewisse Zeichen erlaubt) und schiebt dann vermittels dput ein Release (also eine *.changes + alle drin enthaltenen *.deb + dazugehörige *.dsc, *.diff.gz/*.debian.tgz, *.orig.tar.gz, oder *.tar.gz bei native packages) ins APT-Repository und läßt danach den Index regenerieren und PGP-signieren. dput ist ein schlaues Tool, es schreibt nämlich nach getaner Arbeit eine *.upload-Datei, in welcher dieser Fakt verzeichnet wird, sodaß Pakete auch wenn man den workspace nicht jedes Mal wegschmeißt nur einmalig hochgeladen werden. (Es existiert allerdings hier kein Schutz davor, den workspace zu leeren und ein Paket mit derselben Version aber anderem Inhalt ein zweites Mal hochzuladen. Nur die Systeme, die das installieren wollen, mögen einen dann ggf. nicht mehr so gerne.)

mvndput.sh ruft nun also mvndebri.sh auf, was allerdings mehr macht als nur das Äquivalent von Debians dak oder CentOS’ createrepo. Es erstellt nämlich auch einen Index.

http://test-hudson-debs.bonn.tarent.de/portal-setup/debidx.htm heißt der Gute, und ist eine Auflistung aller Pakete nach dist, suite, Source und Binary im Repository. Weiterhin steht oben nochmal, um welches Repository es sich handelt, und welche dists und suites verfügbar sind. Der Name bildet sich aus „http://“ + Jenkins-Systemname + „-debs.bonn.tarent.de/“ + Jenkins-Jobname + „/debidx.htm“ — wenn man den Dateinamen wegläßt kann man das auch direkt als Verzeichnisstruktur anschauen.

Der Clou: die passende sources.list-Zeile steht auch schon da! (Allerdings muß man die suiten ggf. auf die, die wirklich benötigt werden, reduzieren.)

deb http://test-hudson-debs.bonn.tarent.de/portal-setup squeeze main

Das funktioniert auf allen acht Jenkins-Systemen, allerdings natürlich nur aus dem tarent-Netz heraus — dafür ohne https oder aufwendige Authentifizierung. Einfach so.

Bitte achtet darauf, den Namen des Jenkins-Jobs als erstes Argument zu mvndput.sh zu verwenden, ggf. gefolgt von einer Kennzeichnung, falls ihr mehr als ein Repo pro Job braucht (warum, weiß ich nicht, aber es geht). Da alles als maven-User läuft findet keine Abgrenzung statt.

Wenn ihr Pakete aufräumen möchtet, so loggt euch (als User maven) auf dem entsprechenden Jenkins ein und schaut auf der Verzeichnisebene nach; einen direkten Link gibt’s auch, wenn man das [dir] im tabellarischen Index anklickt, zum Beispiel http://test-hudson-debs.bonn.tarent.de/portal-setup/dists/squeeze/main/Pkgs/portal-setup/, was im Dateisystem dem Verzeichnis /opt/mvn-debs/portal-setup/dists/squeeze/main/Pkgs/portal-setup/ entspricht.

Nachdem ihr händisch Änderungen dort vorgenommen habt, müßt ihr natürlich den Index neu generieren lassen, das geht dann so:

mksh /opt/mvn-debs/mvndebri.sh /opt/mvn-debs portal-setup squeeze

Wenn ihr nicht nur eine dist angefaßt habt, könnt ihr die auch auflisten:

mksh /opt/mvn-debs/mvndebri.sh /opt/mvn-debs portal-setup hardy squeeze

Oder einfach weglassen, dann macht er alle:

mksh /opt/mvn-debs/mvndebri.sh /opt/mvn-debs portal-setup

Das ist dann auch der Unterschied zwischen dists und suites: letztere teilen sich ein Release-File, erstere nicht, nur den XHTML-Index.

So, ich hoffe, ihr findet das so nützlich wie Sascha und so arbeits- und zeitersparend wie ich ☺

PS: Der Code ist mittlerweile auch publiziert. Wer solch ein Szenario noch woanders aufsetzen möchte ist uns herzlich willkommen.

The pictures are hypertext references to large versions. Of course, your photographer (me, although Samuel helped to set up the PocketPC’s camera application correctly, 10x) also had some Kruškovac ☺ (imported from Croatia into Bosnia)…

Of course we were not above closing Front Desk either ☻☺

jupp 3.1.17 uploaded today, mostly thanks to user input suggesting I improve things, especially the syntax highlighting. (Maybe more to come.) I like users who don’t complain but give helpful comments and send in patches even.

Since the Debian FTP masters complain that the NEW queue is empty for the first time in ages, I also uploaded jupp to Debian proper (got requests, several, from actual users — independent of each other). I originally thought I were the only user, it’s not worth it, maybe too close to joe (which segfaults a lot more and has some ugly things, so I cherry-picked the better features of it instead of rebasing jupp), but it’s had a package in mports (MidnightBSD ports) for ages, users submitted one to FreeBSD® last year and keep it updated, there’s even a WIP package in pkgsrc®, and who knows where else or how many people are using my OpenSuSE Buildservice package or have had installed the previous DEB package I uploaded to my play repo. So now I feel it worth to upload.

I even invested some major packaging rework, such as splitting the build-arch and build-indep parts from each other, and importing the upstream source into the packaging VCS, as I have learned in the “packaging with git” talk here at DebConf. (No guys, I will stick to CVS as git doesn’t give me anything.)

Been hot and dry today (although the sky is now back full of dark clouds), so I had a headache most of the morning until way past noon. Better now though, and I found a place where I could get Cevapi, which are really some sort of quick imbiss / fast food here (no Đuveč pirinač though, and she didn’t have any Ajvar nor did she speak any language other than the local, but that wasn’t a problem, only a bit dry because I didn’t give in and took the offered Ketchup). Bought a 1ℓ bottle of Kruškovac (from Hrvatska, though) and some small plastic glasses, then.

I wonder how many people would, now, be willing to give Bosna i Hercegovina a try as holiday region (which might have been the intent of having a Balkan DebConf). I’m sure I do.

To all attendees: the hotel will give you some kind of stamped hardpaper card which states where you stayed on the trip, and for how long — give that to the border guards when exiting Bosnia.

Sitting in Бања Лука, Република Српска, Босна и Херцеговина (Banja Luka, Republika Srpska, Bosna i Hercegovina) let’s just say the country is pretty nice. People are okay, the beer is not called “Nektar” by accident, and the Mark (subunit Fennig, funnily enough) is worth 1 DM. Price niveau is below Germany (even when we had the DM) in some things, below or at modern European in others. In short, very affordable. They don’t accept paper money though, it’s really hard to get coins in most places, and they only want those. The food is okay, and my hotel is very luxurious. It’s also got LAN.

The weather is not so nice at the moment though: raining a lot, and expecting 30°C too-hot sun in two days. And there are still no Geocaches in the area.

Anyway, DebConf is going on, I’m acclimating and trying to get people, faces, nicknames and realnames connected. And accents. (And pronunciation of names — for example, Ian differs totally from what I’d use.) We even have working wire network (LAN) most of the time ;-)

We’re indeed still working on resurrecting m68k, but that’s no news. More on that later, I’d say.

Evolvis 4.8.35 has been released this week. This is an amazing upgrade within the 4.8 series of development, to bridge the time gap until such time as the 5.1 series can be used.

As announced earlier, our APT Repository contains packages targetting Debian Lenny (amd64, i386), including side packages and backports needed for a standard EvolvisForge installation, add one of these lines to your /etc/apt/sources.list to use it. These packages might work on Debian squeeze, maybe even *buntu, but will probably have issues with multiarch on Debian unstable.

First the big caveat: support for old-style (Jutta Horstmann) external Wiki instances has been removed, since we migrated all Evolvis installations to use gforge-plugin-mediawiki a while ago already.

Now the big news: bug trackers (old and new) contain several predefined search queries useful for Software Engineering and Quality Assurance. There’s a new standard tracker type called “Funktionsreferenz” (functionality specification) on every newly created project. The Extratabs plugin, backported from 5.1 then enhanced, allows adding arbitrary tabs as link or IFRAME (embedding content) to any project. And the DatePicker component allows easily entering a date, or a date plus time-of-day, using an ECMAscript pop-up while also accepting simple strings, for instance from Lynx users. The format used for displaying dates can be configured in “My Account” between d.m.y, y-m-d and m/d/y; the software accepts all three formats, always, nevertheless. Furthermore, the ECMAscript DatePicker widget is available in a number of languages for all three formats — English, German (Deutsch), Spanish (Español), French (Français), Italian (Italiano), Dutch (Nederlands), Polish (Polsku), Portuguese (Portugues), Romanian (Românesc), Bulgarian (Български), Russian (Русский), Hungarian (Magyar), Norwegian (but I’m not sure if it’s Nynorsk or Bokmål).

Now the small improvements: a robots.txt file is present by default, allowing wget and asking all crawlers to slow down; system administrators may configure it (in gforge.conf) to switch to one disallowing search engine spiders. Newly created Tasks have a default duration of two weeks (standard Scrum timeframe), not one week. /usr/share/gforge/bin/scm-newsubrepo.php supports easily adding new git repositories to a project already using them.

And finally, the most important / visible bug fixes: Sorting tables in Tasks works again, and the search drop-down boxen in Tracker are now sorted. The modify task/tracker forms have more “Submit” buttons and have been slightly rearranged to improve User Experience. “Copy+Close” a task works again. Some links, labels and translations (German only) have been corrected. Project members are now added to the default mailing lists correctly, i.e. to the -discuss group always, to the -commits group if they have SCM Write permissions. Custom Tracker fields of type Checkbox no longer have the “100 None” option. Tasks do not show up on “My Page” several times now, once is enough ;-)

Last but not least, a look at the future: we’re working on our Jenkins Plugin, which we might provide for Evolvis 4.8 if it’s functioning quickly enough. We’re also working on getting Evolvis 5.1 into a usable shape, and porting all Evolvis 4.8 functionality to it. Much has been integrated in the recently released FusionForge 5.1 already. A new functionality to mark bugs as duplicate, followup or having a simple, nontyped relationship to another tracker item is being worked on, as are improvements (new items, better look’n’feel) to the “My Page”. And that’s just the big news.

A German language User’s Guide / Handbook is also being worked on. It’s a Wiki, so feel free to help ☺ (although the licence is CC-BY-NC-SA and thus non-free, sorry for that, but the software itself is GPLv2+). And as usual, there’s a full changelog and you can look at the svn revision log.

We wish you an enjoyable software development and lifecycle management, your tarent solutions GmbH Evolvis Project Team

pkgsrc-2011Q2 has been released. This is the first release which is really usable on MirOS. Bootstrapping works without applying patches before; I updated the pkgsrc instructions accordingly. During the freeze, there were two last-minute fixes to important base packages (gmake and glib2) which did not go in before the branch was cut off. One of them (gmake) has been pulled up on my request after Alistair G. Crooks encouraged to do so.

I finally managed to get a MirOS instance running on Xen with HVM using NetBSD-current as the Domain-0. I had to compile my own DOM0 kernel to include support for the alc ethernet driver—yes, the one where I did not manage to fix the driver under MirOS. The first impression: Compilation and other CPU-intensive tasks are very fast, while I/O is quite slow. The qemu-dm process, which provides hard disk and network drivers to the DomU, seems to get congested quite rapidly. Btw, emulating an Intel Gigabit network card works very well with our em(4) driver.

To profit from the VM, I set up a bulk build with a fresh pkgsrc-2011Q2 checkout, using the pbulk framework. Technically, it looks very nice and more sane overall than a simple recursive make: there is a scan phase at the start, where a dependency tree of all packages is constructed. Then, a master process decides which package to build next. It can optionally distribute the builds over several machines at once. However, I found the documentation to be severely lacking; what’s more, the pkgsrc guide and the doc/HOWTO-pbulk file have obviously been written by persons with different approaches w.r.t. suggested directory layout etc.

I created a NetBSD slice of 55 GiB, mounted in MirOS under /pbulk, for all data relative to the bulk build and added it as a physical device in the VM configuration. However, the I/O congestion becomes worse after some time building things. The ssh connection becomes more or less unresponsive, and qemu-dm takes 100% of the Dom0 cpu. Even after stopping the build with ^Z, the hard disk is thrashing for several minutes with qemu-dm at 100% cpu, before slowly going back to normal after a few minutes. WTF? For what it is worth, the VM has 1 GiB of RAM allocated and no swap. More tuning required …

mksh R40b (nowadays with filled in user’s caveats (for R40, too!) and packager’s upgrade hints) has just been released. This is a should-have upgrade, fixing a number of — admittedly some obscure — bugs, changing things begun in R40, improving upon others. Thanks to the PLD Linux guys for spotting all these errors; thanks to them and phpnet.org both for adopting mksh so well.

I have also fixed a bug in nroff(1) which will lead to an even nicer looking HTML manpage mksh(1) (after the next rebuild and upload of a MirBSD snapshot — scheduled RSN).

jupp 3.1.16 took on the task of merging Debian joe changes (aiming at an upload). I also split the jupprc file into three versions (2.8 generic/DOS, 3.1+jupp and 3.7/Unix) because of the differences in the baseline executables making rc files partially mutually incompatible (think Insert key), annoyingly warning (think syntax, hmsg), or less usable (joe’s new menu system).

jupp 2.8.2 is a companion to jupp 3.1.16 — mostly because of the new help window “character map” ☺

Binaries for jupp should be updated RSN too.

Considering Banja Luka is arriving quickly, the “r” in RSN should be taken with a few grains of salt. I’ve also scheduled working on the pcc Debian package for the next future; updating lynx and maybe others like OpenSSH in MirBSD is also due; cvs(1) will receive more of my time, but before the next Upload I’d like to fix LP#12230 once verified.

Builds for Debian/m68k are also still running. I note I did in fact not manage to make a new base image, yet (but 2.6.39 kernels miss a patch, anyway, so waiting for 3.0 is ok). It’s still using gcc-4.4 because nobody tests gcc-4.6 and gcj-4.6 FTBFS due to SIGSEGV, but that’s ok in my books. rsyslog is broken but sysklogd works.

The #ksh|Freenode page finally got a well-deserved link to Planet Commandline. Throw more my way!

Acronyms and translations, too. (Got Norwegian and Rumanian covered in the meantime. No idea whether any RTL languages will work in that beast. But I’m young and need the money)

Since I’m writing a wlog entry anyway… let me thank Gunnar for a nice summary on the current Free Culture discussion; my comments on Nina’s site seem to be eaten, but let me support it fully, although, of course, I normally use a copycenter style licence, which is specifically written for general works of authorship under copyright law, not limited to software. I did in fact have that in mind. Maybe some people will like it (it’s less than one Kibibyte long) either generally or just for their everyday random musings (they can then keep CC-BY-SA for the “big works” if they so desire).

Wouter, grass background makes green headlines illegible. I’ve never liked, and never installed manually, cups either. (Benny tells me that Apple’s new version refuses to talk with a non-Apple cups, kinda defeating the whole idea I think.) Port 9100 is JetDirect (probably with an HP in front and some subset of ©®™ trailing) and just nice. (Being able to talk ESC/P with your printer like print '\033K\x07\0\x3E\x81\x99\xA5\xA5\x81\x3E' >/dev/lpa too rocks though, IMHO. Yes, mine can, and I still can. /dev/lpa is BSD.)

Kai, thanks for your vimrc lines:

	:highlight TrailWhitespace ctermbg=red guibg=red
	:match TrailWhitespace /\s\+$\| \+\ze\t/

Automatic removal is harmful, though — I just fell into the trap since jupprc contains needed whitespace at EOL… but manual removal (bound to ^K] in jupp) rocks. And I like that your solution uses such strong a colour — vim users are the single most represented offender group for actually leaving the redundant whitespace at EOL there, and it should hurt their eyes. (Sadly there is some vehement disagreement preventing them from inclusion in grml-etc-core — but that’s why I re-post them here.) Ah, and jupp can of course display whitespace visibly (although it uses ‘·’/‘→’, replacing the arrow with ‘¬’ if no UTF-8, not ‘»’), accessible with ^Ov.

Steve, want to put up a checklist for sites? We can “crowdsource” the… testing… to maybe get some interesting results…

Some other people would get more comments if they were idling in IRC (Freenode) or allow comments on their blog, specifically without too high an entrance barrier — OpenID is ok, but many other things, and ECMAscript, are not; but I can’t really say that loud because our wlog is static HTML compiled from a flat plaintext data source so it doesn’t allow such either. I often forget what I wanted to add if I can’t get it out quickly enough (especially at work). Sowwy…

Me like the cat picture postings (Amayita, Tiago, ¡Gracias!).

You might have noticed the release of mksh R40 recently, after more than a year of development. Well, stay tuned for both R40b (with accumulated fixes) and R41 (intent to speed up array handling a lot and prepare for what we postponed to mksh R42 now — associative, multi-dimensional arrays).

You should also upgrade, if you have not yet done so, to kwalletcli 2.11.

Finally, jupp 3.1.15 was left out to the world, including Minix 3 users this time, by special request of one of these on our mailing list. In addition to the MidnightBSD mport — which has been there in like forever — and the MirPort and the FreeWRT package, in December 2011 a user submitted it to FreeBSD® ports, and Benny is going to add it to NetBSD® pkgsrc® soon, he said. (He also updated their mksh source package. Thanks!) I’ve been asked by two people, independent from each other, when I’ll upload it to Debian proper, instead of the private-repo packaging. Maybe I should indeed do that, comments?

• √ Agreement to pay from company
• √ Going to drive with some apparently speed-loving brits
• √ Registration accepted
• √ Dienstreiseantrag prepared
• √ Sent that beast to the office ticket queue

So yes, this means I’m to what used to be Yugoslawia when I was there the last time, although in the Poreč region of Istria, Hrvatska.

mksh R40 has been released Sunday 12nd June. So, what can you do with The MirBSD Korn Shell?

Have a look at the collection of shell snippets, which admittedly is only the beginning — most scripts don’t make use of the cool new features yet. But they eventually will. The Shell-Toolkit project is definitely worth a visit! (Admittedly not using my most ❦ dear SCM, but with a DVCS, every checkout is a clone, i.e. a backup, and none of the contributors must fear a central repo being taken down, which, for such a loose collection, is desirable.)

(First posting to Plänet Commandline! Tag: pcli)

Vutral asked in IRC how to synchronise two shells’ environment while they’re running. As you may know, POSIX systems cannot change a process’ environment vector after it has been started, only the process itself can. Well, the shell can, and we’ll use a variety of things for this.

This trick assumes you have $HISTFILE set to the same pathname in both shells (obviously, they run under the same user). It uses export -p to render the current list of exported variables, then transforms the list from newline-separated to a single big one-line export statement.
Then it transforms all remaining newlines (which will be part of a single-quoted string, since that’s mksh(1)’s export format) into the sequence '$'\n'' which means: terminate current single-quoted string, append $'\n' and open up a new single-quoted string immediately; concatenate these three.
Now, $'\n' is just a fancy way of saying newline, and part of mksh because David Korn (yes, the Korn in Korn Shell) strongly suggested to me that this functionality be included — but, as we can see here, it pays off.
Finally, the so transformed string is prepended by unset \$(export); which, when executed, will cause the shell to unset (and unexport) all currently exported variables. The shell parameters that are not exported, i.e. not in the environment, are not affected by this code (except for $x and $nl, but… whatever).
This string is then passed to read -s (plus -r and clearing IFS to enable raw mode), which means, read into the parameter $REPLY (which we conveniently don’t use — but it’s trashed too, thus) but store into history at the same time.

Ah hah! Now, the persistent history feature comes into effect! After running the below statement in the “source” shell, switch into the terminal running the “destination” shell, press Enter once on the empty line (Ctrl-U to empty it if it wasn’t), then Cursor-Up (↑) to recall… voilà, an insanely large line with the previously created string sorta expanded… and press Enter again to run it. Now your set of exported parameters is the exact same (minus if you exported IFS, nl, x or REPLY) as in the “source” shell.

I’ve added extra spaces and a linewrap below, this is really just one big line:

Of course, this makes a nice function, for your ~/.mkshrc or somesuch.