Developers’ Weblog

Sponsored by
HostEurope Logo

Developers’ Weblog

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

FrOSCon

16.08.2017 by tg@
Tags: event fun pcli personal

I’ll not respond, much, until next Monday. We have FrOSCon.

[PSA] Fixing CVE-2017-12836 (Debian #871810) in GNU cvs

11.08.2017 by tg@
Tags: archaeology bug debian pcli security

Considering I’ve become the de-facto upstream of cvs(GNU) even if not yet formally the de-iure upstream maintainer, fixing this bug obviously falls to me — not quite the way I had planned passing this evening after coming home from work and a decent and, worse, very filling meal at the local Croatian restaurant. But, so’s life.

The problem here is basically that CVS invokes ssh(1) (well, rsh originally…) but doesn’t add the argument separator “--” before the (user-provided) hostname, which when starting with a hyphen-minus will be interpreted by ssh as an argument. (Apparently the other VCSes also had additional vulnerabilities such as not properly escaping semicoloi or pipes from the shell or unescaping percent-escaped fun characters, but that doesn’t affect us.)

The obvious fix and the one I implemented first is to simply add the dashes. This will also be backported to Debian {,{,old}old}stable-security.

Then I looked at other VCSes out of which only one did this, but they all added extra paranoia hostname checks (some of them passing invalid hostnames, such as those with underscores in them). OK, I thought, then also let’s add extra checks to CVS’ repository reference handling. This will end up in Debian sid and MirBSD, pending passing the regression tests of course… hah, while writing this article I had to fixup because a test failed. Anyway, it’s not strictly necessary AFAICT to fix the issue.

Update, about 2⅕ hours past midnight (the testsuite runs for several hours): of course, the “sanity” testsuite (which itself is rather insane…) also needs adjustments, plus a bonus fix (for something that got broken when the recent allow-root-regex patch was merged and got fixed in the same go to…night).

tl;dr: a fix will end up in Debian *stable-security and can be taken out of my mail to the bugreport; another few changes for robustness are being tested and then added to both MirBSD and Debian sid. The impact is likely small, as it’s hard to get a user (if you find one, in the first place) to use a crafted CVSROOT string, which is easy to spot as well.

Update, Monday: apparently someone took care of the DSA and DLA yesterday after ACCEPTing the uploads — thanks, I was outside during the day.

Update 2017-08-25: It was noted that ssh(1) does not parse its command line correctly, and therefore the patch above might not be enough in the general case. However, I still think it’s good enough for CVS because it constructs its command line in a way that doesn’t let users exploit that bug.

mksh R56 was released with experimental fixes for the “history no longer persisted when HISTFILE near-full” and interactive shell cannot wait on coprocess by PID issues (I hope they do not introduce any regressioins) and otherwise as a bugfix release. You might wish to know the $EDITOR selection mechanism in dot.mkshrc changed. Some more alias characters are allowed again, and POSIX character classes (for ASCII, and EBCDIC, only) appeared by popular vote.

mksh now has a FAQ; enjoy. Do feel free to contribute (answers, too, of course).

The jupp text editor has also received a new release; asides from being much smaller, and updated (mksh too, btw) to Unicode 10, and some segfault fixes, it features falling back to using /dev/tty if stdin or stdout is not a terminal (for use on GNU with find | xargs jupp, since they don’t have our xargs(1) -o option yet), a new command to exit nonzero (sometimes, utilities invoking the generic visual editor need this), and “presentation mode”.

Presentation mode, crediting Natureshadow, is basically putting your slides as (UTF-8, with fancy stuff inside) plaintext files into one directory, with sorting names (so e.g. zero-padded slide numbers as filenames), presenting them with jupp * in a fullscreen xterm. You’d hit F6 to switch to one-file view first, then present by using F8 to go forward (F7 to go backward), and, for demonstrations, F9 to pipe the entire slide through an external command (could be just “sh”) offering the previous one as default. Simple yet powerful; I imagine Sven Guckes would love it, were he not such a vim user.

The new release is offered as source tarball (as usual) and in distribution packages, but also, again, a Win32 version as PKZIP archive (right-click on setup.inf and hit I̲nstall to install it). Note that this comes with its own (thankfully local) version of the Cygwin32 library (compatible down to Windows 95, apparently), so if you have Cygwin installed yourself you’re better off compiling it there and using your own version instead.

I’ve also released a new DOS version of 2.8 with no code patches but an updated jupprc; the binary (self-extracting LHarc archive) this time comes with all resource files, not just jupp’s.

Today, the jupprc drop-in file for JOE 3.7 got a matching update (and some fixes for bugs discovered during that) and I added a new one for JOE 4.4 (the former being in Debian wheezy, the latter in jessie, stretch and buster/sid). It’s a bit rudimentary (the new shell window functionality is absent) but, mostly, gives the desired jupp feeling, more so than just using stock jstar would.

CVS’ ability to commit to multiple branches of a file at the same time, therefore grouping the commit (by commitid at least, unsure if cvsps et al. can be persuaded to recognise it). If you don’t know what cvs(GNU) is: it is a proper (although not distributed) version control system and the best for centralised tasks. (For decentral tasks, abusing git as pseudo-VCS has won by popularity vote; take this as a comparison.)

If desired, I can make these new versions available in my “WTF” APT repository on request. (Debian buster/sid users: please change “https” to “http” there, the site is only available with TLSv1.0 as it doesn’t require bank-level security.)

I’d welcome it very much if people using an OS which does not yet carry either to package it there. Message me when one more is added, too ☺

In unrelated news I uploaded MuseScore 2.1 to Debian unstable, mostly because the maintainers are busy (though I could comaintain it if needed, I’d just need help with the C++ and CMake details). Bonus side effect is that I can now build 2.2~ test versions with patches of mine added I plan to produce to fix some issues (and submit upstream) ☻

In other news, I’m working on a new i386+sparc MirBSD snapshot more than ever. Mostly to get everything old out from under my feet before tackling the LibreSSL import (to get TLSv1.2 support, due to the aforementioned idio…decision). I’ve yet to see whether our G++ port works on sparc, and I’ve yet to create ports for libGLU and xlock which used to be in the base X system but had to go away for being written in an unmaintainable language (plus a system is only reliable if it has only one libstdc++), but it’ll be a good stepping stone (plus mfny asked for a sparc snapshot on IRC). I was considering distributing ISOs at FrOSCon but, with an installed user base in the single digits (likely), you can imagine how useful that’d be. (Fun side idea: distribute ISOs with a boot menu where you can choose not only MirBSD installer or live system but also “minimal Debian system directly booting into the MirBSD live system running under qemu-kvm”. But I’ve got not enough spare time right now.)

Midsummer

26.06.2017 by tg@
Tags: plan snapshot

Yes, well. Sorry. Didn’t manage Beltane, not this one either. But, progress.

Updates to the last two posts

16.03.2017 by tg@
Tags: bug debian grml news pcli rant snippet tip work

Someone from the FSF’s licencing department posted an official-looking thing saying they don’t believe GitHub’s new ToS to be problematic with copyleft. Well, my lawyer (not my personal one, nor for The MirOS Project, but related to another association, informally) does agree with my reading of the new ToS, and I can point out at least a clause in the GPLv1 (I really don’t have time right now) which says contrary (but does this mean the FSF generally waives the restrictions of the GPL for anything on GitHub?). I’ll eMail GitHub Legal directly and will try to continue getting this fixed (as soon as I have enough time for it) as I’ll otherwise be forced to force GitHub to remove stuff from me (but with someone else as original author) under GPL, such as… tinyirc and e3.

My dbconfig-common Debian packaging example got a rather hefty upgrade because dbconfig-common (unlike any other DB schema framework I know of) doesn’t apply the upgrades on a fresh install (and doesn’t automatically put the upgrades into a transaction either) but only upgrades between Debian package versions (which can be funny with backports, but AFAICT that part is handled correctly). I now append the upgrades to the initial-version-as-seen-in-the-source to generate the initial-version-as-shipped-in-the-binary-package (optionally, only if it’s named .in) removing all transaction stuff from the upgrade files and wrapping the whole shit in BEGIN; and COMMIT; after merging. (This should at least not break nōn-PostgreSQL databases and… well, database-like-ish things I cannot test for obvious (SQLite is illegal, at least in Germany, but potentially worldwide, and then PostgreSQL is the only remaining Open Source database left ;) reasons.)

Update: Yes, this does mean that maintainers of databases and webservers should send me patches to make this work with not-PostgreSQL (new install/name.in, upgrade files) and not-Apache-2.2/2.4 (new debian/*/*.conf snippets) to make this packaging example even more generally usable.

Natureshadow already forked this and made a Python/Flask package from it, so I’ll prod him to provide a similarily versatile hello-python-world example package.

Since I use this as base for other PHP packages like SimKolab, I’ve updated my packaging example with:

  • PHP 7 support (untested, as I need libapache2-mod-php5)
  • tons more utility code for you to use
  • a class autoloader, with example (build time, for now)
  • (at build time) running a PHPUnit testsuite (unless nocheck)

The old features (Apache 2.2 and 2.4 support, dbconfig-common, etc.) are, of course, still there. Support for other webservers could be contributed by you, and I could extend the autoloader to work at runtime (using dpkg triggers) to include dependencies as packaged in other Debian packages. See, nobody needs “composer”! ☻

Feel free to check it out, play around with it, install it, test it, send me improvement patches and feature requests, etc. — it’s here with a mirror at GitHub (since I wrote it myself and the licence is permissive enough anyway).

This posting and the code behind it are sponsored by my employer ⮡ tarent.

Please use the correct (perma)link to bookmark this article, not the page listing all wlog entries of the last decade. Thank you.</update>

Some updates inline and at the bottom.

The new Terms of Service of GitHub became effective today, which is quite problematic — there was a review phase, but my reviews pointing out the problems were not answered, and, while the language is somewhat changed from the draft, they became effective immediately.

Now, the new ToS are not so bad that one immediately must stop using their service for disagreement, but it’s important that certain content may no longer legally be pushed to GitHub. I’ll try to explain which is affected, and why.

I’m mostly working my way backwards through section D, as that’s where the problems I identified lie, and because this is from easier to harder.

Note that using a private repository does not help, as the same terms apply.

Anything requiring attribution (e.g. CC-BY, but also BSD, …)

Section D.7 requires the person uploading content to waive any and all attribution rights. Ostensibly “to allow basic functions like search to work”, which I can even believe, but, for a work the uploader did not create completely by themselves, they can’t grant this licence.

The CC licences are notably bad because they don’t permit sublicencing, but even so, anything requiring attribution can, in almost all cases, not “written or otherwise, created or uploaded by our Users”. This is fact, and the exceptions are few.

Anything putting conditions on the right to “use, display and perform” the work and, worse, “reproduce” (all Copyleft)

Section D.5 requires the uploader to grant all other GitHub users…

  • the right to “use, display and perform” the work (with no further restrictions attached to it) — while this (likely — I didn’t check) does not exclude the GPL, many others (I believe CC-*-SA) are affected, and…
  • the right to “reproduce your Content solely on GitHub as permitted through GitHub's functionality”, with no further restructions attached; this is a killer for, I believe, any and all licences falling into the “copyleft” category.

Note that section D.4 is similar, but granting the licence to GitHub (and their successors); while this is worded much more friendly than in the draft, this fact only makes it harder to see if it affects works in a similar way. But that doesn’t matter since D.5 is clear enough. (This doesn’t mean it’s not a problem, just that I don’t want to go there and analyse D.4 as D.5 points out the same problems but is easier.)

This means that any and all content under copyleft licences is also no longer welcome on GitHub.

Anything requiring integrity of the author’s source (e.g. LPPL)

Some licences are famous for requiring people to keep the original intact while permitting patches to be piled on top; this is actually permissible for Open Source, even though annoying, and the most common LaTeX licence is rather close to that. Section D.3 says any (partial) content can be removed — though keeping a PKZIP archive of the original is a likely workaround.

Affected licences

Anything copyleft (GPL, AGPL, LGPL, CC-*-SA) or requiring attribution (CC-BY-*, but also 4-clause BSD, Apache 2 with NOTICE text file, …) are affected. BSD-style licences without advertising clause (MIT/Expat, MirOS, etc.) are probably not affected… if GitHub doesn’t go too far and dissociates excerpts from their context and legal info, but then nobody would be able to distribute it, so that’d be useless.

But what if I just fork something under such a licence?

Only “continuing to use GitHub” constitutes accepting the new terms. This means that repositories from people who last used GitHub before March 2017 are excluded.

Even then, the new terms likely only apply to content uploaded in March 2017 or later (note that git commit dates are unreliable, you have to actually check whether the contribution dates March 2017 or later).

And then, most people are likely unaware of the new terms. If they upload content they themselves don’t have the appropriate rights (waivers to attribution and copyleft/share-alike clauses), it’s plain illegal and also makes your upload of them or a derivate thereof no more legal.

Granted, people who, in full knowledge of the new ToS, share any “User-Generated Content” with GitHub on or after 1ˢᵗ March, 2017, and actually have the appropriate rights to do that, can do that; and if you encounter such a repository, you can fork, modify and upload that iff you also waive attribution and copyleft/share-alike rights for your portion of the upload. But — especially in the beginning — these will be few and far between (even more so taking into account that GitHub is, legally spoken, a mess, and they don’t even care about hosting only OSS / Free works).

Conclusion (Fazit)

I’ll be starting to remove any such content of mine, such as the source code mirrors of jupp, which is under the GNU GPLv1, now and will be requesting people who forked such repositories on GitHub to also remove them. This is not something I like to do but something I am required to do in order to comply with the licence granted to me by my upstream. Anything you’ve found contributed by me in the meantime is up for review; ping me if I forgot something. (mksh is likely safe, even if I hereby remind you that the attribution requirement of the BSD-style licences still applies outside of GitHub.)

(Pet peeve: why can’t I “adopt a licence” with British spelling? They seem to require oversea barbarian spelling.)

The others

Atlassian Bitbucket has similar terms (even worse actually; I looked at them to see whether I could mirror mksh there, and turns out, I can’t if I don’t want to lose most of what few rights I retain when publishing under a permissive licence). Gitlab seems to not have such, but requires you to indemnify them… YMMV. I think I’ll self-host the removed content.

And now?

I’m in contact with someone from GitHub Legal (not explicitly in the official capacity though) and will try to explain the sheer magnitude of the problem and ways to solve this (leaving the technical issues to technical solutions and requiring legal solutions only where strictly necessary), but for now, the ToS are enacted (another point of my criticism of this move) and thus, the aforementioned works must go off GitHub right now.

That’s not to say they may not come back later once this all has been addressed, if it will be addressed to allow that. The new ToS do have some good; for example, the old ToS said “you allow every GitHub user to fork your repositories” without ever specifying what that means. It’s just that the people over at GitHub need to understand that, both legally and technically¹, any and all OSS licences² grant enough to run a hosting platform already³, and separate explicit grants are only needed if a repository contains content not under an OSI/OKFN/Copyfree/FSF/DFSG-free licence. I have been told that “these are important issues” and been thanked for my feedback; we’ll see what comes from this.

① maybe with a little more effort on the coders’ side³

② All licences on one of those lists or conformant to the DFSG, OSD or OKD should do⁴.

③ e.g. when displaying search results, add a note “this is an excerpt, click HERE to get to the original work in its context, with licence and attribution” where “HERE” is a backlink to the file in the repository

④ It is understood those organisations never un-approve any licence that rightfully conforms to those definitions (also in cases like a grant saying “just use any OSS² licence” which is occasionally used)

Update: In the meantime, joeyh has written not one but two insightful articles (although I disagree in some details; the new licence is only to GitHub users (D.5) and GitHub (D.4) and only within their system, so, while uploaders would violate the ToS (they cannot grant the licence) and (probably) the upstream-granted copyleft licence, this would not mean that everyone else wasn’t bound by the copyleft licence in, well, enough cases to count (yes it’s possible to construct situations in which this hurts the copyleft fraction, but no, they’re nowhere near 100%).

How to use the subtree git merge strategy

20.12.2016 by tg@
Tags: debian grml pcli tip work

This article might be perceived as a blatant ripoff of this Linux kernel document, but, on the contrary, it’s intended as add-on, showing how to do a subtree merge (the multi-project merge strategy that’s actually doable in a heterogenous group of developers, as opposed to subprojects, which many just can’t wrap their heads around) with contemporary git (“stupid content tracker”). Furthermore, the commands are reformatted to be easier to copy/paste.

To summarise: you’re on the top level of a checkout of the project into which the “other” project (Bproject) is to be merged. We wish to merge the top level of Bproject’s “master” branch as (newly created) subdirectory “dir-B” under the current project’s top level.

	$ git remote add --no-tags -f Bproject /path/to/B/.git
	$ git merge -s ours --allow-unrelated-histories --no-commit Bproject/master
	$ git read-tree -u --prefix=dir-B/ Bproject/master
	$ git commit -m 'Merge B project as our subdirectory dir-B'

	Later updates are easy:
	$ git pull -s subtree Bproject master
 

(mind the trailing slash after dir-B/ on the read-tree command!)

Besides reformatting, the use of --allow-unrelated-histories recently became necessary. --no-tags is also usually what you want, because tags are not namespaced like branches.

Another command you might find relevant is how to clean up orphaned remote branches:

	$ for x in $(git remote); do git remote prune "$x"; done
 

This command locally deletes all remote branches (those named “origin/foo”) that have been deleted on the remote side.

Update: Natureshadow wishes you to know that there is such a command as git subtree which can do similar things to the subtree merge strategy explained above, and several more related things. It does, however, need the præfix on every subsequent pull.

“I don’t like computers”

13.11.2016 by tg@
Tags: debian pcli personal rant tip

cnuke@ spotted something on the internet, and shared. Do read this, including the comments. It’s so true. (My car is 30 years old, I use computers mostly for sirc, lynx and ssh, and I especially do not buy any product that needs to be “online” to work.)

Nice parts of the internet, to offset this, though, do exist. IRC as a way of cheap (affordable), mostly reliant, communication that’s easy enough to do with TELNET.EXE if necessary. Fanfiction; easy proliferation of people’s art (literature, in this case). Fast access to documentation and source code; OpenBSD’s AnonCVS was a first, nowadays almost everything (not Tom Dickey’s projects (lynx, ncurses, xterm, cdk, …), nor GNU bash, though) is on a public version control system repository. (Now people need to learn to not rewrite history, just commit whatever shit they do, to record thought process, not produce the perfect-looking patch.) Livestreams too, I guess, but ever since live365.com went dead due to a USA law change on 2016-01-02, it got bad.

Please save GMane!

28.07.2016 by tg@
Tags: debian news pcli rant

GMane has been down for a day or two, and flakey for a day before that. MidnightBSD’s laffer1 just linked the reason, which made me cry out loud.

GMane is really great, and I rely on the NNTP interface a lot, both posting and especially reading — it gives me the ability to download messages from mailing lists I don’t receive in order to be able to compose replies with (mostly) correct References and In-Reply-To headers. Its web interface, especially the article permalinks, are also extremely helpful.

This is a request for a petition to save GMane. Please, someone, do something! Thanks in advance!

httpd CVE-2016-5387 “httpoxy” fixed

28.07.2016 by tg@
Tags: security

A small patch was applied to httpd(8) to not pass the HTTP Proxy header as HTTP_PROXY environment variable to CGI scripts, because those often call utilities such as ftp(1), lynx(1), GNU wget, etc. which may accept this as an alternative spelling of http_proxy which is used to set a proxy for outgoing connections — something e.g. the CGI scripts in MirKarte do.

I’ll have to add O_DIRECTORY support to open(2) for more security in cpio(1), pax(1), and tar(1). (Maybe I’ll also add O_CLOEXEC while there…) Today’s paxmirabilis will however pick this up as soon as it’s there and thus fail if it is not supported by the running kernel yet.

Morale: when upgrading to a snapshot take care of the kernel first (install and reboot), userspace second.

To be clear: this will affect the first -current snapshot to be published after today on /MirOS/current.

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

MirOS Logo