wisp: Whitespace to Lisp

New version: draketo.de/software/wisp [7]

display "Hello World!" ↦ (display "Hello World!")

define : factorial n     (define (factorial n)            
    if : zero? n       ↦     (if (zero? n)                
       . 1                      1                      
       * n : factorial {n - 1}  (* n (factorial {n - 1}))))

Table of Contents

display "Hello World!"      ↦      (display "Hello World!")

display "Hello World!"      ↦      (display "Hello World!")
display "Hello Again!"      ↦      (display "Hello Again!")

display                              ↦    (display
  string-append "Hello " "World!"    ↦      (string-append "Hello " "World!"))

string-append "Hello"        ↦    (string-append "Hello"
  string-append " " "World"  ↦      (string-append " " "World")
  . "!"                      ↦      "!")

= a        ↦    (= a
  . a      ↦      (. a))

. a b c    ↦    a b c

let                       ↦    (let
  :                       ↦      ((msg "Hello World!"))
    msg "Hello World!"    ↦      (display msg))
  display msg             ↦      

define : hello who                            ↦    (define (hello who)
  display : string-append "Hello " who "!"    ↦      (display (string-append "Hello " who "!")))

let                       ↦    (let
  : msg "Hello World!"    ↦      ((msg "Hello World!"))
  display msg             ↦      (display msg))

let : : msg "Hello World!"     ↦    (let ((msg "Hello World!"))
  display msg                  ↦      (display msg))

define : hello who                    ↦    (define (hello who)
_ display                             ↦      (display 
___ string-append "Hello " who "!"    ↦        (string-append "Hello " who "!")))

PS: Wisp is linked in the comparisions of SRFI-110 [118].

say Yes, I do!

Yes, I do!

wget https://bitbucket.org/ArneBab/wisp/downloads/wisp-0.8.6.tar.gz
tar xf wi*z; cd wi*/; ./c*e; make check; guile -L . --language=wisp

display "First Witch
    When shall we three meet again
    In thunder, lightning, or in rain?\n"

define-syntax-rule : First_Witch a ...
  format #t "~A\n" 
    string-join 
      map : lambda (x) (string-join (map symbol->string x))
            quote : a ...
      . "\n"

First_Witch
    When shall we three meet again
    In thunder, lightning, or in rain?

SCENE I. A desert place.

    Thunder and lightning.

    Enter : First Witch
            Second Witch
            Third Witch

First Witch
    When shall ,(emphasized we three) meet again
    In thunder, lightning, or in rain?

Second Witch
    When the hurlyburly's done,
    When the battle's lost and won.

; ...

ALL
    Fair is foul, and foul is fair:
    Hover through the fog and filthy air.

action
    Exeunt

Going from Python to Guile Scheme - a natural progression

[135]

Python is the first language I loved. I dreamt in Python, I planned in Python, I thought I would never need anything else.

 - Free: html [136] | pdf [137]
 - Softcover: 14.95 € [135]
   with pdf, epub, mobi
 - Source: download [138]
   free licensed under GPL

I will show you why I love Python

Python is a language where I can teach a handful of APIs and cause people to learn most of the language as a whole. — Raymond Hettinger (2011-06-20) [139]

• Pseudocode which runs
• One way to do it
• Hackable
• Batteries and Bindings
• Scales up

Where I hit its limits

Why, I feel all thin, sort of stretched if you know what I mean: like butter that has been scraped over too much bread. — Bilbo Baggins in “The Lord of the Rings”

• Dual Syntax: What we teach new users is no longer what we use
• Ceremony crops in
• Complexity is on the rise

And how I lost its shackles

You must unlearn what you have learned. — Yoda in “The Empire Strikes Back“

• Guile Scheme [25] compared to the strengths of Python
• Guile Scheme [25] shortcomings
• Guile [25] beyond Python

Guile Scheme [25] is the official GNU extension language, used for example in GNU Cash [140] and GNU Guix [83] and the awesome Lilypond [141].

Accompany me on a path beyond Python

Every sufficiently complex appli­ca­tion/langu­age/tool will either have to use Lisp or reinvent it the hard way. — Greenspuns 10th rule [142]

• Buy the softcover: 14.95 € [135] (includes the ebook)
• Read for free: html [136] | pdf [137]
• Download the source: from Sourcehut [138]

As free cultural work [143], py2guile is licensed under the GPLv3 or later [144]. You are free to share, change, remix and even to resell it as long as you say that it’s from me (attribution) and provide the whole corresponding source under the GPL (sharealike).

For instructions on building the ebook yourself, see the README in the source.

Happy Hacking!

— Arne Babenhauserheide

Gratis py2guile from Freenet

[145]

py2guile [145] is a book I wrote about Python and Guile Scheme. It’s selling at 14.95 €](https://www.epubli.de/shop/buch/47692) for the printed softcover.

To fight the new german data retention laws, you can get the ebook gratis: Just install Freenet [146], then the following links work:

• py2guile.epub [147]
• py2guile.mobi [148]
• py2guile.pdf [149]
• py2guile-0.9.0-fn.tar.gz [150]

Escape total surveillance and get an ebook about the official GNU extension language for free today!

Python chooses Github, therefore I’m releasing the py2guile PDF for free

[137]

Python is the first language I loved. I dreamt in Python, I planned in Python, I thought I would never need anything else.

  Download “Python to Guile” (pdf) [137]

You can read more about this on the Mercurial mailing list [151].

 - Free: html [136] | pdf [137]
   preview edition
   (complete)

Yes, this means that with Guile I will contribute to a language developed via Git, but it won’t be using a proprietary platform.

If you like py2guile, please consider buying the book:

 - Softcover: 14.95 € [152]
   with digital companion
 - Source: download [138]
   free licensed under GPL

More information: draketo.de/py2guile [153]

Commentary on Python and Github

Subjective popularity contest without robust data

I was curious why this happened so I read through PEP 0481. It's interesting that Git was chosen to replace Mercurial due to Git's greater popularity, yet a technical comparison was deemed as subjective. In fact, no actual comparison (of any kind) was discussed. What a shame. — Emmanuel Rosa on G+ [154]

yes. And the popularity contest wasn’t done in any robust way — they present values between 3x as popular and 18x as popular. That is a crazy margin of error — especially for a value on which to base a very disrupting decision. — my answer

No more dogfooding

Yesterday Python maintainers chose to move to GitHub and Git. Python is now developed using a C-based tool on a Ruby-based, unfree platform. And that changed my view on what’s happening in the community. Python no longer fosters its children and it even stopped dogfooding where its tools are as good as or better than other tools. I don’t think it will die. But I don’t bet on it for the future anymore. — EDIT to my answer on Quora [155] “is Python a dying language?” which originally stated “it’s not dying, it’s maturing”.

Github invades your workflows

The PEP for github hedges somewhat by using github for code but not bug tracker. Not ideal considering BitKeeper, but a full on coup for GitHub. — Martin Owens

that’s something like basic self-defense, but my experience with projects who moved to GitHub is that GitHub soon starts to invade your workflows, changing your cooperation habits. At some point people realize that they can’t work well without GitHub anymore.

Not becoming dependent on GitHub while using it requires constant vigilance. Seeing how Python already switched to Git and GitHub because existing infrastructure wasn’t maintained does not sound like they will be able or willing to put in the work to keep independent. — my answer on G+ [156]

Foreboding since 2014

I was already pretty disappointed when I heard that Python is moving to Git. Seeing it choose the proprietary platform is an even sadder choice from my perspective. Two indicators for a breakage in the culture of the project.

For me that’s a reason to leave Python. Though it’s not like I did not get a foreboding of that. It’s why I started learning Guile Scheme in 2013 — and wrote about the experience.

I will still use Python for many practical tasks — it acquired the momentum for that, especially in science (I was a champion for Python in the institute, which is now replacing Matlab and IDL for many people here, and I will be teaching Python starting two weeks from now). I think it will stay strong for many years; a good language to start and a robust base for existing programs. But with what I learned the past years, Python is no longer where I place my bets. — slightly adjusted version of my post on the Mercurial mailing list [151].

Popularity without robust data instead of quality

(this is taken from a message I wrote to Brett, so I don’t have to say later that I stayed silent while Python went down. I got a nice answer, and despite the disagreement we said a friendly good bye)

Back when I saw that Python might move to git, I silently resigned and stopped caring to some degree. I have seen a few projects move to Git in the past years (and in every project problems remained even years after the switch), so when it came to cPython, the quarrel with git-fans just didn’t feel worthwhile anymore.

Seeing Python choose GitHub with the notion of “git is 3x to 18x more popular than Mercurial and free solutions aren’t better than GitHub” makes me lose my trust in the core development community, though.

PEP 481 states, that it is about the quality of the tooling, but it names the popularity numbers quite prominently: python.org/dev/peps/pep-0481/ [157]

If they are not relevant, they shouldn’t be included, but they are included, so they seem to be relevant to the decision. And “the best tooling” is mostly subjective, too — which is shown in the PEP itself which mostly talks about popularity, not quality. It even goes into length about how to avoid many of the features of GitHub.

I’ve seen quite a few projects try to avoid lock-in to GitHub. None succeeded. Not even in one where two of about six active developers were deeply annoyed by GitHub. This is exactly what the scipy part of the PEP describes: lock-in due to group effects.

Finally, using hg-git is by far not seamless. I use it for several projects, and when the repositories become big (as cPython’s is), the overhead of the conversion becomes a major hassle. It works, but native Mercurial would be much more efficient. When pushing takes minutes, you start to think twice about whether you’ll just do the quick fix right now. Not to forget that at some point people start to demand signing of commits in git-style (not possible with hg-git, you can only sign commits mercurial-style) as well as other gitologisms (which have an analogue in Mercurial but aren’t converted by hg-git).

Despite my disappointment, I wish you all the best. Python is a really cool language. It’s the first one I loved and will always stay dear to me, so I’m happy that you work on it — and I hope you keep it up.

So, I think this is goodbye. A bit melancholic, but that’s how that goes.

Good luck to you in your endeavors,
Arne Babenhauserheide

Enough negativity

And that’s enough negativity from me.

Thank you, Brett, for reminding me that even though we might disagree, it’s important to remember that people in the project are hit by negativity much harder than it feels for the one who writes.

For my readers: If that also happened to you one time or the other, please read his article:

How I stay happy making open source software [158]

Thank you, Brett. Despite everything I wrote here, I still think that Python is a great project, and it got many things right — some of which are things which are at least as important as code but much less visible, like having a large, friendly community.

I’m happy that Python exists, and I hope that it keeps going. And where I use programming to make a living, I’m glad when I can do it in Python. Despite all my criticism, I consider Python as the best choice for many tasks, and this is also written in py2guile [145]: almost the the first half of the book talks about the strengths of Python. Essentially I could not criticize Python as strongly as I’m doing it here if I did not like it so much. Keep that in mind when you think about what you read.

Also Brett now published an article where he details his decision to move to GitHub. It is a good read: The history behind the decision to move Python to GitHub — Or, why it took over a year for me to make a decision [159]

For me, Gentoo is about *convenient* choice

It's often said, that Gentoo is all about choice, but that doesn't quite fit what it is for me.

After all, the highest ability to choose is Linux from scratch and I can have any amount of choice in every distribution by just going deep enough (and investing enough time).

What really distinguishes Gentoo for me is that it makes it convenient to choose.

Since we all have a limited time budget, many of us only have real freedom to choose, because we use Gentoo which makes it possible to choose with the distribution-tools. Therefore only calling it “choice” doesn't ring true in general - it misses the reason, why we can choose.

So what Gentoo gives me is not just choice, but convenient choice.

Some examples to illustrate the point:

KDE 4 without qt3

I recently rebuilt my system after deciding to switch my disk layout (away from reiserfs towards a simple ext3 with reiser4 for the portage tree). When doing so I decided to try to use a "pure" KDE 4 - that means, a KDE 4 without any remains from KDE3 or qt3.

To use kde without any qt3 applications, I just had to put "-qt3" and "-qt3support" into my useflags in /etc/make.conf and "emerge -uDN world" (and solve any arising conflicts).

Imagine doing the same with a (K)Ubuntu...

Emacs support

Similarly to enable emacs support on my GentooXO (for all programs which can have emacs support), I just had to add the "emacs" useflag and "emerge -uDN world".

Selecting which licenses to use

Just add

ACCEPT_LICENSE="-* @FSF-APPROVED @FSF-APPROVED-OTHER"

to your /etc/make.conf to make sure you only get software under licenses which are approved by the FSF.

For only free licenses (regardless of the approved state) you can use:

ACCEPT_LICENSE="-* @FREE"

All others get marked as masked by license. Default (no ACCEPT_LICENSE in /etc/make.conf) is “* -@EULA”: No unfree software. You can check your setting via emerge --info | grep ACCEPT_LICENSE. More information… [160]

One program (suite) in testing, but the main system rock stable

Another part where choosing is made convenient in Gentoo are testing and unstable programs.

I remember my pain with a Kubuntu, where I wanted to use the most recent version of Amarok. I either had to add a dedicated Amarok-only testing repository (which I'd need for every single testing program), or I had to switch my whole system into testing. I did the latter and my graphical package manager ceased to work. Just imagine how quickly I ran back to Gentoo.

And then have a look at the ease of deciding to take one package into testing in Gentoo:

• emerge --autounmask-write =cathegory/package-version
• etc-update
• emerge =cathegory/package-version

EDIT: Once I had a note here “It would be nice to be able to just add the missing dependencies with one call”. This is now possible with --autounmask-write.

And for some special parts (like KDE 4) I can easily say something like

• ln -s /usr/portage/local/layman/kde-testing/Documentation/package.keywords/kde-4.3.keywords /etc/portage/package.keywords/kde-4.3.keywords

(I don't have the kde-testing overlay on my GentooXO, where I write this post, so the exact command might vary slightly)

Closing remarks

So to finish this post: For me, Gentoo is not only about choice. It is about convenient choice.

And that means: Gentoo gives everybody the power to choose.

I hope you enjoy it as I do!

• more about Gentoo [161]

pmerge @glsa

\#! /bin/sh

\### Update the portage tree and the glsa packages via pkgcore

\# spew a status message
echo $(date) "start to update GLSA" >> /tmp/cron-update.log

\# Sync only portage
pmaint sync /usr/portage

\# security relevant programs
pmerge -uDN @glsa > /tmp/cron-update-pkgcore-last.log || cat \
    /tmp/cron-update-pkgcore-last.log >> /tmp/cron-update.log  

\# And keep everything working
revdep-rebuild

\# Finally update all configs which can be updated automatically
cfg-update -au

echo $(date) "finished updating GLSA" >> /tmp/cron-update.log

\#!/bin/sh                                                     

\### Update my computer using pgkcore, 
\### since that also works if some dependencies couldn't be resolved.

\# Sync all overlays
eix-sync

\## First use pkgcore
\# security relevant programs (with build-time dependencies (-B))
pmerge -BuD @glsa

\# system, world and all the rest
pmerge -BuD @system
pmerge -BuD @world
pmerge -BuD @installed

\# Then use portage for packages pkgcore misses (inlcuding overlays) 
\# and for *EMERGE_DEFAULT_OPTS="--keep-going"* in make.conf 
emerge -uD @security
emerge -uD @system
emerge -uD @world
emerge -uD @installed

\# And keep everything working
emerge @preserved-rebuild
revdep-rebuild

\# Finally update all configs which can be updated automatically
cfg-update -au

alias pix='pquery --raw -nv --attr=keywords'

$ time pmerge -p @glsa
 * Resolving...
Nothing to merge.

real    0m1.863s
user    0m1.463s
sys     0m0.100s

No, it ain’t “forever” (GNU Hurd code_swarm from 1991 to 2010)

If the video doesn’t show, you can also download it as Ogg Theora & Vorbis “.ogv” [166] or find it on youtube [167].

This video shows the activity of the Hurd coders and answers some common questions about the Hurd, including “How stagnated is Hurd compared to Duke Nukem Forever?”. It is created directly from commits to Hurd repositories, processed by community codeswarm [169].

Every shimmering dot is a change to a file. These dots align around the coder who did the change. The questions and answers are quotes from todays IRC discussions (2010-07-13) in #hurd at irc.freenode.net.

You can clearly see the influx of developers in 2003/2004 and then again a strenthening of the development in 2008 with less participants but higher activity than 2003 (though a part of that change likely comes from the switch to git with generally more but smaller commits).

I hope you enjoyed the high-level look on the activity of the Hurd project [170]!

PS: The last part is only the information title with music to honor Sean Wright [171] for allowing everyone to use and adapt his Album Enchanted [172].

Some technical advantages of the Hurd

→ An answer to just accept it, truth hurds [173], where Flameeyes told his reasons for not liking the Hurd and asked for technical advantages (and claimed, that the Hurd does not offer a concept which got incorporated into other free software, contributing to other projects). Note: These are the points I see. Very likely there are more technical advantages which I don’t see well enough to explain them.

The translator system in the Hurd is a simple concept that makes many tasks easy, which are complex with Linux (like init, network transparency, new filesystems, …). Additionally there are capabilities (give programs only the access they need - adjusted at runtime), subhurds and (academic) memory management.

Information for potential testers: The Hurd is already usable, but it is not yet in production state. It progressed a lot during the recent years, though. Have a look at the status report [174] if you want to see if it’s already interesting for you. See running the Hurd [175] for testing it yourself.

Table of Contents:

• Influence on other systems: FUSE and BSD [176]
• Advantages for users
  • Translator-based filesystem [177]
  • Network transparency on the filesystem level [178]
  • Unionmount as user [179]
  • Persistent services, activated when needed [180]
  • Add permissions at runtime [181]
• Advantages for administrators and developers
  • Lightweight virtualization [182]
  • More accessible lowlevel hacking [183]
  • Subdividing memory management [184]
• Summary [185]

Influence on other systems: FUSE in Linux and limited translators in NetBSD

Firstoff: FUSE [186] is essentially an implementation of parts of the translator system [187] (which is the main building block of the Hurd [170]) to Linux, and NetBSD recently got a port of the translators system of the Hurd [188]. That’s the main contribution to other projects that I see.

As an update in 2015: A pretty interesting development in the past few years is that the systemd developers have been bolting features onto Linux which the Hurd already provided 15 years ago. Examples: socket-activation provides on-demand startup like passive translators, but as crude hack piggybacked on dbus which can only be used by dbus-aware programs while passive translators can be used by any program which can access the filesystem, calling priviledged programs via systemd provides jailed priviledge escalation like adding capabilities at runtime, but as crude hack piggybacked on dbus and specialized services.

That means, there is a need for the features of the Hurd, but instead of just using the Hurd, where they are cleanly integrated, these features are bolted onto a system where they do not fit and suffer from bad performance due to requiring lots of unnecessary cruft to circumvent limitations of the base system. The clean solution would be to just set 2-3 full-time developers onto the task of resolving the last few blockers (mainly sound and USB) and then just using the Hurd.

translator-based filesystem

On the bare technical side, the translator-based filesystem stands out: The filesystem allows for making arbitrary programs responsible for displaying a given node (which can also be a directory tree) and to start these programs on demand. To make them persistent over reboots, you only need to add them to the filesystem node (for which you need the right to change that node). Also you can start translators on any node without having to change the node itself, but then they are not persistent and only affect your view of the filesystem without affecting other users. These translators are called active, and you don’t need write permissions on a node to add them.

network transparency on the filesystem level

The filesystem implements stuff like Gnome VFS (gvfs) and KDE network transparency on the filesystem level, so those are available for all programs. And you can add a new filesystem as simple user, just as if you’d write into a file “instead of this node, show the filesystem you get by interpreting file X with filesystem Y” (this is what you actually do when setting a translator but not yet starting it (passive translator)).

One practical advantage of this is that the following works:

settrans -a ftp\: /hurd/hostmux /hurd/ftpfs /
dpkg -i ftp://ftp.gnu.org/path/to/*.deb

This installs all deb-packages in the folder path/to on the FTP server. The shell sees normal directories (beginning with the directory “ftp:”), so shell expressions just work.

You could even define a Gentoo mirror translator (settrans mirror\: /hurd/gentoo-mirror), so every program could just access mirror://gentoo/portage-2.2.0_alpha31.tar.bz2 and get the data from a mirror automatically: wget mirror://gentoo/portage-2.2.0_alpha31.tar.bz2

unionmount as user

Or you could add a unionmount translator to root which makes writes happen at another place. Every user is able to make a readonly system readwrite by just specifying where the writes should go. But the writes only affect his view of the filesystem.

persistent translators, started when needed

Starting a network process is done by a translator, too: The first time something accesses the network card, the network translator starts up and actually provides the device. This replaces most initscripts in the Hurd: Just add a translator to a node, and the service will persist over restarts.

It’s a surprisingly simple concept, which reduces the complexity of many basic tasks needed for desktop systems.

And at its most basic level, Hurd is a set of protocols for messages which allow using the filesystem to coordinate and connect processes (along with helper libraries to make that easy).

add permissions at runtime (capabilities)

Also it adds POSIX compatibility to Mach while still providing access to the capabilities-based access rights underneath, if you need them: You can give a process permissions at runtime and take them away at will. For example you can start all programs without permission to use the network (or write to any file) and add the permissions when you need them.

Different from Linux, you do not need to start privileged and drop permissions you do not need (goverened by the program which is run), but you start as unprivileged process and add the permissions you need (governed by an external process):

groups # → root
addauth -p $(ps -L) -g mail
groups # → root mail 

lightweight virtualization

And then there are subhurds (essentially lightweight virtualization which allows cutting off processes from other processes without the overhead of creating a virtual machine for each process). But that’s an entire post of its own…

Easy to test lowlevel hacking

And the fact that a translator is just a simple standalone program means that these can be shared and tested much more easily, opening up completely new options for lowlevel hacking, because it massively lowers the barrier of entry.

For example the current Hurd can use the Linux network device drivers and run them in userspace (via DDE), so you can simply restart them and a crashing driver won’t bring down your system.

subdividing memory management

And then there is the possibility of subdividing memory management and using different microkernels (by porting the Hurd layer, as partly done in the NetBSD port), but that is purely academic right now (search for Viengoos to see what its about).

Summary

So in short:

The translator system in the Hurd is a simple concept that makes many tasks easy, which are complex with Linux (like init, network transparency, new filesystems, …). Additionally there are capabilities (give programs only the access they need - adjusted at runtime), subhurds and (academic) memory management.

Best wishes,
Arne

PS: I decided to read flameeyes’ post as “please give me technical reasons to dispell my emotional impression”.

PPS: If you liked this post, it would be cool if you’d flattr it: [189]

PPPS: Additional information can be found in Gaël Le Mignot’s talk notes [190], in niches for the Hurd [191] and the GNU Hurd documentation pages [192].

P4S: This post is also available in the Hurd Staging Wiki [193].

(A)GPL as hack on a Python-powered copyright system

AGPL [194] is a hack on copyright, so it has to use copyright, else it would not compile/run.

All the GPL [195] licenses are a hack on copyright. They insert a piece of legal code into copyright law to force it to turn around on itself.

You run that on the copyright system, and it gives you code which can’t be made unfree.

To be able to do that, it has to be written in copyright language (else it could not be interpreted).

my_code = "<your code>"

def AGPL ( code ): 
    """
    >>> is_free ( AGPL ( code ) )
    True
    """
    return eval (
        transform_to_free ( code ) )

copyright ( AGPL ( my_code ) )

You pass “AGPL ( code )” to the copyright system, and it ensures the freedom of the code.

The transformation means that I am allowed to change your code, as long as I keep the transformation, because copyright law sees only the version transformed by AGPL, and that stays valid.

Naturally both AGPL definition and the code transformed to free © must be ©-compatible. And that means: All rights reserved. Else I could go in and say: I just redefine AGPL and make your code unfree without ever touching the code itself (which is initially owned by you by the laws of ©):

def AGPL ( code ): 
    """ 
    >>> is_free ( AGPL ( code ) )
    False
    """
    return eval (
        transform_to_mine ( code ) )

In this Python-powered copyright-system, I could just define this after your definition but before your call to copyright(), and all calls to APGL ( code ) would suddenly return code owned by me.

Or you would have to include another way of defining which exact AGPL you mean. Something like “AGPL, but only the versions with the sha1 hashes AAAA BBBB and AABA”. cc tries to use links for that, but what do you do if someone changes the DNS resolution to point creativecommons.org to allmine.com? Whose DNS server is right, then - legally speaking?

In short: AGPL is a hack on copyright, so it has to use copyright, else it would not compile/run.

Are there 10x programmers?

→ An answer I wrote to this question on Quora [196].

Software Engineering: What is the truth of 10x programmers?
Do they really exist?…

Let’s answer the other way round: I once had to take heavy anti-histamines for three weeks. My mind was horribly hazy from that, and I felt awake only about two hours per day. However I spent every day working on a matrix multiplication problem.

It was three weeks of failure, because I just could not grasp the problem. I was unable to hold it in my mind.

Then I could finally drop the anti-histamine.

On the first day I solved the problem on my way to buy groceries. On the second day I planned the implementation while walking for two hours . On the third day I finished the program.

This taught me to accept it when people don’t manage to understand things I understand: I know that the brain can actually have different paces and that complexity which feels easy to me might feel infeasible for others. It sure did feel that way to me while I took the anti-histamines.

It also taught me to be humble: There might be people to whom my current state of mind feels like taking anti-histamines felt to me. I won’t be able to even grasp the patterns they see, because they can use another level of complexity.

To get a grasp of the impact, I ask myself a question: How would an alien solve problems who can easily keep 100 things in its mind — instead of the 4 to 7 which is the rough limit for humans [197]?

BY-SA and GPL: creativecommons closed the chasm in the sharealike/copyleft community

This is the biggest news item [198] for free culture and free software in the past 5 years: The creativecommons attribution sharealike license is now one-way compatible to the GPL — see the message from creativecommons [199] and from the Free Software Foundation [200].

• Blender [201] rigs under cc by-sa which use libraries under GPL are safe against suing, even if a Blender code contributor goes crazy,
• 3D models like the self replicating 3D printer RepRap [202] and libre hardware designs under cc by-sa and GPL can be combined,
• GPL-compatible Free Software can utilize all content from the Wikipedia [203] and
• On the personal side: we can merge art from Ryzom [204] and Battle for Wesnoth [205] and use both in the Pirate Party RPG [206].

Some license compatibility legalese might sound small, but the impact of this is hard to overestimate.

(I’ll now revise some of my texts about licensing — CC BY-SA got a major boost in utility because it not longer excludes usage in copyleft documents which need the source to have a defended sharealike clause)

Communicating your project: honest marketing for free software projects

You have an awesome project, but you see people reach for inferior tools? There are people using your project, but you can’t reach the ones you care about? Read on for a way to ensure that your communication doesn’t ruin your prospects but instead helps your project to shine.

Communicating your project is an essential step for getting the users you want. Here I summarize my experience from working on several different projects including KDE [207] (where I learned the basics of PR - yay, sebas!), the Hurd [170] (where I could really make a difference by improving the frontpage and writing the Month of the Hurd), Mercurial [208] (where I practiced minimally invasive PR) and 1d6 [206] (my own free RPG where I see how much harder it is to do PR, if the project to communicate is your own).

Since voicing the claim that marketing is important often leads to discussions with people who hate marketing of any kind, I added an appendix [209] with an example which illustrates nicely what happens when you don’t do any PR - and what happens if you do PR of the wrong kind.

If you’re pressed for time and want the really short form, just jump to the questionnaire [210].

• What is good marketing? [211]
• How to comunicate your project? [212]
  • Who are our Target Groups? [213]
  • What could they ask? [214]
  • Whose wishes can we fullfil? [215]
  • Provide those answers! [216]
  • Further points [217]
• bab-com Q: Short questionaire [210]
• Note: The mission statement [218]
• Summary [219]
• Appendix: Why communicating your project? [209]

What is good marketing?

Before we jump directly to the guide, there is an important term to define: Good marketing. That is the kind of marketing, we want to do.

The definition I use here is this:

Good marketing ensures that the people to whom a project would be useful learn about the project.

and

Good marketing starts with the existing strengths of a project and finds people to whom these strengths are useful.

Thus good marketing does not try to interfere with the greater plan of the project, though it might identify some points where a little effort can make the project much more interesting to users. Instead it finds users to whom the project as it is can be useful - and ensures that these know about the project.

Be fair to competitors, be honest to users, put the project goals before generic marketing considerations.

As such, good marketing is an interface between the project and its (potential) users.

How to communicate your project?

This guide depends on one condition: Your project already has at least one area in which it excels over other projects. If that isn’t the case, please start by making your project useful to at least some people.

The basic way for communicating your project to its potential users always follows the same steps.

To make this text easier to follow, I’ll intersperse it with examples from the latest project where I did this analysis: GNU Guile: The GNU Ubiquitous Intelligent Language for Extensions [220]. Guile provides a nice example, because its mission is clearly established in its name and it has lots of backing, but up until our discussion actually had a wikipedia-page which was unappealing to the point of being hostile against Guile itself.

To improve the communication of our project, we first identify our target groups.

Who are our Target Groups?

To do so, we begin by asking ourselves, who would profit from our project:

• What can we do well and how do we compare to others?
• To whom would we already be useful or interesting if people knew about our strengths?
• To whom are we already the best option?

Try to find about 3 groups of people and give them names which identify them. Those are the people we must reach to grow on the short term.

In the next step, we ask ourselves, whom we want or need as users to fullfill our mission (our long-term goal):

• Where do we want to get? What is our goal? (do we have a mission statement?)
• Whom do we need to get there?
• Whom do we want as users? Those shape us as they take part in the development - either as users or as fellow developers.

Again try to find about 3 groups of people and give them names which identify them. Those are the people we must reach to achieve our longterm goal. If while writing this down you find that one of the already identified groups which we could reach would actually detract us from our goal, mark them. If they aren’t direly needed, we would do best to avoid targeting them in our communication, because they will hinder us in our longterm progress: They could become a liability which we cannot get rid of again.

Now we have about 6 target groups: Those are the people who should know about our project, either because they would benefit from it for pursuing their goals, or because we need to reach them to achieve our own goals. We now need to find out, which kind of information they actually need or search.

Example: Target Groups for Guile

GNU Guile is called The GNU Ubiquitous Intelligent Language for Extensions [220]. So its mission is clear: Guile wants to become the de-facto standard language for extending programs - at least within the GNU project.

For whom are we already useful or interesting? Name them as Target-Groups.

1. Schemer: Wants to see what GNU Scheme can do.
2. Extender: GNU enthusiast wants to extend an existing program with a scripting language.
3. Learner: Free Software enthusiast thinks about using Guile to learn programming
4. Project-Starter: Experienced Programmer wants to start a new project.
5. 1337: Programmer wants the coolness-factor.
6. Emacser: Emacs users want to see what the potential future of Emacs would hold.

Whom do we want as users on the long run? Name them as Target-Groups.

1. GNU-folk: All GNU developers.

What could they ask?

This part just requires thinking ourselves into the role of each of the target groups. For each of the target groups, ask yourself:

What would you want to know, if you were to read about our project?

As result of this step, we have a set of answers. Judge them on their strengths: Would these answers make you want to invest time to test our project? If not, can we find a better answer?

Example: Questions for the Target-Groups of Guile

1. Schemer: What can guile do better than other Schemes?
2. Extender: What does Guile offer my program? Why Guile and not Python/Lua?
3. Learner: How easy and how powerful is Guile Scheme? Why Guile and not Python?
4. Starter: What’s the advantage of starting my advanced project with guile?
5. 1337: Why is guile cool?
6. Emacser: What does Guile offer for Emacs?
7. GNU-folk: What does Guile offer my program? (Being a GNU package is a distinct advantage, so there is less competition by non-GNU languages)

Whose wishes can we fullfill?

If our answers for a given group are not yet strong enough, we cannot yet communicate our project convincingly to them. In that case it is best to postpone reaching out to that group, otherwise they could get a lasting weak image of our project which would make it harder to reach them when we have stronger answers at some point in the future.

Remove all groups whose wishes we cannot yet fullfill, or for whom we do not see ourselves as the best choice.

Example: Chosen Target-Groups

1. Schemer: Guile is a solid implementation of Scheme. For a comparison, see An opinionated Guide to Scheme implementations [221].
2. Extender: The guile manual offers a nicely detailed guide for extending a program with Guile [222]. We’re a bit weak on the examples and existing extensions, though, especially on non-GNU-plattforms.
3. Learner: There aren’t yet tutorials for learning to program in Guile, though there are tutorials for learning to write scheme - and even one for understanding Scheme from the view of a Python-user [223]. But our project resources cannot yet support people who cannot program at all well enough, so we have to restrict ourselves to programmers who want to learn a new language.
4. Starter: Guile has solid support for many unix-specific things, but it is not yet a complete project-publishing solution. So we have to restrict ourselves to targeting people who want to start a project which is mainly intended to be used in environments with proper package management (mostly GNU/Linux).
5. 1337: Guile is explicitely named in the GNU Coding Standards [224]. It doesn’t get much cooler than that - at least for a certain idea of cool. We can’t get the Java-1337s, but we can get the Free Software-1337s.
6. Emacser: Guile provides foreign-function-call. If guile gets used as base for Emacs, Emacs users get direct access to all scheme functions, too - as well as real threading. And that’s pretty strong. Also Geiser [225] provides solid Guile Scheme support in Emacs.
7. GNU-folk: They are either extenders or project starters or learners, but additionally they want to know in which GNU projects they can use Guile.

Provide those answers!

Now we have answers for the target groups. When we now talk or write about our project, we should keep those target groups in mind.

You can make that arbitrarily complex, for example by trying to find out which of our target groups use which medium. But lets keep it simple:

Ensure that our website (and potentially existing wikipedia page) includes the information which matters to our target groups. Just take all the answers for all the target groups we can already reach and check whether the basic information contained in them is given on the front page of our website.

And if not, find ways to add it.

As next steps, we can make sure that the questions we found for the target groups not only get answered, but directly lead the target groups to actions: For example to start using our project.

Example: The new Wikipedia-Page of Guile

For Guile, we used this analysis to fix the Wikipedia-Page. The old-version [226] mainly talked about history and weaknesses (to the point of sounding hostile towards Guile), and aside from the latest release number, it was horribly outdated. And it did not provide the information our target groups required.

The current Wikipedia-Page of GNU Guile [227] works much better - for the project as well as for the readers of the page. Just compare them directly and you’ll see quite a difference. But aside from sounding nicer, the new site also addresses the questions of our target groups. To check that, we now ask: Did we include information for all the potential user-groups?

1. Schemers: Yepp (it’s scheme and there’s a section on Guile Scheme
2. Extenders: Yepp (libguile)
3. Learners: Not yet. We might need a syntax-section with some examples. But wikipedians do not like Howto-Like sections. Also the interpreter should get a notice.
4. Project-Starters: Partly in the “core idea”-part in the section Guile Scheme. It might need one more paragraph showing advantages of Guile which make it especially suited for that.
5. 1337s: It is the preferred extension system for the GNU Project. If you’re not that kind of 1337: The Macro-System is hygienic (no surprising side-effects).
6. Emacs users: They got their own section.
7. GNU-Folk: They have a section on Guile in make. We should add a list of GNU projects with Guile support.

So there you go: Not perfect, but most of the groups are covered. And this also ensures that the Wikipedia-page is more interesting to its readers: A clear win-win.

Further points

Additional points which we should keep in mind:

• On the website, do all of our target groups quickly find their way to advanced information about their questions? This is essential to keep the ones interested who aren’t completely taken by the short answers.
• What is a common negative misconception about our project? We need to ensure that we do not write anything which strengthens this misconception. Is there an existing strength, which we can show to counter the negative misconception?
• Where do we want to go? Do we have a mission statement?

bab-com q: Arne Babenhauserheide’s Project Communication Questionaire

• For whom are we already useful or interesting? Name them as Target-Groups.

  • (1)
  • (2)
  • (3)

• Whom do we want as users on the long run? Name them as Target-Groups.

  • (4)
  • (5)
  • (6)
• What could the Target-Groups ask? What are their needs? Formulate them as questions.
  • (1)
  • (2)
  • (3)
  • (4)
  • (5)
  • (6)
• Answer their questions.
  
  • (1)
  • (2)
  • (3)
  • (4)
  • (5)
  • (6)
• Whose needs can we already fulfill well? For whom do we see ourselves as the best choice?
  
  • (1)
  • (2)
  • (3)
  • (4)
• Ensure that our communication includes the answers to these questions (i.e. website, wikipedia page, talks, …), at least for the groups who are likely to use the medium on which we communicate!

Use bab-com to avoid bad-com ☺ - yes, I know this phrase is horrible, but it is catchy and that fits this article: you need catchy things

Note: The mission statement

The mission statement is a short paragraph in which a project defines its goal.

A good example is:

Our mission is to create a general-purpose kernel suitable for the GNU operating system, which is viable for everyday use, and gives users and programs as much control over their computing environment as possible. → GNU Hurd mission explained [228]

Another example again comes from Guile:

Guile was conceived by the GNU Project following the fantastic success of Emacs Lisp as an extension language within Emacs. Just as Emacs Lisp allowed complete and unanticipated applications to be written within the Emacs environment, the idea was that Guile should do the same for other GNU Project applications. This remains true today. → Guile and the GNU project [229]

Closely tied to the mission statement is the slogan: A catch-phrase which helps anchoring the gist of your project in your readers mind. Guile does not have that, yet, but judging from its strengths, the following could work quite well for Guile 2.0 - though it falls short of Guile in general:

GNU Guile scripting: Use Guile Scheme, reuse anything.

Summary

We saw why it is essential to communicate the project to the outside, and we discussed a simple structure to check whether our way of communication actually fits our projects strengths and goals.

Finding the communication strategy actually boils down to 3 steps:

• Target those who would profit from our project or whom we need.
• Check what they need to know.
• Answer that.

Also a clear mission statement, slogan and project description help to make the project more tangible for readers. In this context, good marketing means to ensure that the right people learn about the real strengths of the project.

With that I’ll conclude this guide. Have fun and happy hacking!
— Arne Babenhauserheide

Appendix: Why communicating your project?

In free software we often think that quality is a guarantee for success. But in just the 10 years I’ve been using free software nowadays, I saw my share of technically great projects succumbing to inferior projects which simply reached more people and used that to build a dynamic which greatly outpaced the technically better product.

One example for that are pkgcore and paludis. When portage, the package manager of Gentoo, grew too slow because it did ever more extensive tests, two teams set out to build a replacement.

One of the teams decided that the fault of the low performance lay in Python, the language used by portage. That team built a package manager in C++ and had --wonderfully-long-command-options without shortcuts (have fun typing), and you actually had to run it twice: Once to see what would get installed and then again to actually install it (while portage had had an --ask option for ages, with -a as shortcut). And it forgot all the work it had done in the previous run, so you could wait twice as long for the result. They also had wonderful latin names, and they managed the feat of being even slower than portage, despite being written in C++. So their claim that C++ would be magically faster than python was simply wrong (because they skipped analyzing the real performance bottlenecks). They called their program paludis.

Note: Nowadays paludis has a completely new commandline interface which actually supports short command options. That interface is called cave and looks sane.

The other team did a performance analysis and realized that the low performance actually lay with the filesystem: The portage tree, which holds the required information, contains about 30,000 ebuilds and almost 200,000 files in total, and portage accessed far more of those files than actually needed for resolving the dependencies needed to install the package. They picked python as their language - just like portage. They used almost the same commandline options as portage, except for the places where functionality differed. And they actually got orders of magnitude faster than portage - so fast that their search command often finished after less than a second while. portage took over 10 seconds. They called their program pkgcore.

Both had more exact resolution of packages and could break cyclic dependencies and so on.

So, judging from my account of the quality, which project would you expect to succeed?

I sure expected pkgcore to replace portage within a few months. But this is not what happened. And as I see it in hindsight, the difference lay purely in PR.

The paludis team with their slow and hard-to-use program went all over the Gentoo forums claiming that Python is a horrible language and that a C program will kick portage any time. On their website they repeated their attacks against python and claimed superiority at every step. And they gathered quite a few zealots. While actually being slower than portage. Eventually they rebranded paludis as just better and more correct, not faster. And they created their own distribution (exherbo) as direct rival of Gentoo. With a new, portage-incompatible package format. As if they had read the book, how not to be a friendly competitor.

The pkgcore team on the other hand focussed on good technology. They created the snakeoil library for high-performance python code, but they were friendly about it and actually contributed back to portage where code could be shared. But their website was out of date, often not noting the newest release and you actually had to run pmerge --help to see the most current commandline options (though you could simply guess them if you knew portage). And they got attacked by paludis zealots so much, that this year the main developer finally sacked the project: He told me on IRC that he had taken so much vitriol over the years that it simply wasn’t worth the cost anymore.

Update: About a year later someone else took over. Good code often survives the loss of its creator.

So, what can we learn from this? Technical superiority does not gain you anything, if you fail to convince people to actually use your project.

If you don't communicate your project, you don't get users. If you don’t get users, your chances of losing motivation are orders of magnitude higher than if you get users who support you.

And aggressive marketing works, even if you cannot actually deliver on your promises. Today they have a better user-interface and even short option-names. But even to date, exherbo has much fewer packages in its repositories than Gentoo. If the number of files is any measure, the 10,000 files in their special repositories are just about 5% of the almost 200,000 files portage holds. But they managed quite well to fraction the Gentoo users - at least for some time. And their repeated pushes for new standards in the portage tree (EAPIs) created a constant pressure on pkgcore to adapt, which had the effect that nowadays pkgcore cannot install from the portage tree anymore (the search still works, though, and I still use it - I will curse mightily on the day they manage to also break that).

Update: Someone else took over and now pkgcore can install again.

So aggressive marketing and doing everything in the book of unfriendly competition might have allowed the paludis devs to gather some users and destroy the momentum of pkgcore, but it did not allow them to actually become a replacement of portage within Gentoo. Their behaviour alienated far too many people for that. So aggressive and unfriendly marketing is better than no marketing, but it has severe drawbacks which you will likely want to avoid.

If you use overly aggressive, unfriendly or dishonest communication tactics, you get some users, but if your users know their stuff, you won’t win the mindshare you need to actually make a difference.

If on the other hand you want to see communication done right, just take a look at KDE and Gnome nowadays. They cooperate quite well, and they compete on features and by improving their project so users can take an informed choice about the project they choose.

And their number of contributors steadily keeps growing.

So what do they do? Besides being technically great, it boils down to good marketing.

Conveniently merge a NEWS file without conflicts

echo "
[merge-patterns]
# avoid bogus conflicts in NEWS files
NEWS = internal:union
" >> .hg/hgrc

echo "/NEWS merge=union" >> .gitattributes
git add .gitattributes
git commit -m "union-merge NEWS" .gitattributes

Download one page from a website with all its prerequisites

Often I want to simply backup a single page from a website. Until now I always had half-working solutions, but today I found one solution using wget [233] which works really well, and I decided to document it here. That way I won’t have to search it again, and you, dear readers, can benefit from it, too ☺

Update 2020: You can also use the copyweb-script [234] from pyFreenet: copyweb -d TARGET_FOLDER URL
Install via pip3 install --user pyFreenet3.

In short: This is the command:

wget --no-parent --timestamping --convert-links --page-requisites --no-directories --no-host-directories --span-hosts --adjust-extension --no-check-certificate -e robots=off -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' [URL]

Optionally add --directory-prefix=[target-folder-name]

(see the meaning of the options [235] and getting wget [236] for some explanation)

That’s it! Have fun copying single sites! (but before passing them on, ensure that you have the right to do it)

Does this really work?

As a test, how about running this:

wget -np -N -k -p -nd -nH -H -E --no-check-certificate -e robots=off -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' --directory-prefix=download-web-site http://draketo.de/english/download-web-page-with-all-prerequisites

(this command uses the short forms of the options)

Then test the downloaded page with firefox:

firefox download-web-site/download-web-page-all-prerequisites.html

Getting wget

If you run GNU/Linux, you likely already have it - and if not, then your package manager has it. GNU wget is one of the standard tools available everywhere.

Some information in the (sadly) typically terse style can be found on the wget website from the GNU project: gnu.org/s/wget [237].

In case you run Windows, have a look at Wget for Windows [238] from the gnuwin32 project or at GNU Wgetw for Windows [239] from eternallybored.

Alternatively you can get a graphical interface via WinWGet [240] from cybershade.

Or you can get serious about having good tools and install MSYS [241] or Cygwin [242] - the latter gets you some of the functionality of a unix working environment on windows, including wget.

If you run MacOSX, either get wget via fink [243], homebrew [244] or MacPorts [245] or follow the guide from osxdaily [246] or the german guide from dirk [247] (likely there are more guides - these two were just the first hits in google).

The meaning of the options (and why you need them):

• --no-parent: Only get this file, not other articles higher up in the filesystem hierarchy.
• --timestamping: Only get newer files (don’t redownload files).
• --page-requisites: Get all files needed to display this page.
• --convert-links: Change files to point to the local files you downloaded.
• --no-directories: Do not create directories: Put all files into one folder.
• --no-host-directories: Do not create separate directories per web host: Really put all files in one folder.
• --span-hosts: Get files from any host, not just the one with which you reached the website.
• --adjust-extension: Add a .html extension to the file.
• --no-check-certificate: Do not check SSL certificates. This is necessary if you’re missing one of the host certificates one of the hosts uses. Just use this. If people with enough power to snoop on your browsing would want to serve you a changed website, they could simply use one of the fake certifications authorities they control.
• -e robots=off: Ignore robots.txt files which tell you to not spider and save this website. You are no robot, but wget does not know that, so you have to tell it.
• -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4': Fake being an old Firefox to avoid blocking based on being wget.
• --directory-prefix=[target-folder-name]: Save the files into a subfolder to avoid having to create the folder first. Without that options, all files are created in the folder in which your shell is at the moment. Equivalent to mkdir [target-folder-name]; cd [target-folder-name]; [wget without --directory-prefix]

Conclusion

If you know the required options, mirroring single pages from websites with wget is fast and easy.

Note that if you want to get the whole website, you can just replace --no-parent with --mirror.

Happy Hacking!

Elegant commandline argument parsing on the shell

Parsing command line arguments on the shell is often done in an ad-hoc fashion, growing unwieldy as time goes by, but there are tools to make that elegant. Here’s a complete example.

I use this in the conf [248] project (easy setup of autotools projects). It builds on the great solution by Adam Katz [249].

# outer loop to allow processing option arguments at the end
while test ! $# -eq 0; do
    # getopts loop, here you define the short options: 
    # h for -h, l: for -l <lang>. -: provides support for long-options.
    while getopts -- hl:-: arg "$@"; do
        case $arg in
            h ) ARG_HELP=true ;;
            l ) ARG_LANG="$OPTARG" ;;
            - ) LONG_OPTARG="${OPTARG#*=}"
                case "$OPTARG" in
                    help    ) ARG_HELP=true;;
                    lang=?* ) ARG_LANG="$LONG_OPTARG" ;;
                    # FIXME: using the same option twice (either both
                    # after the argument or both before it) gives the
                    # first, not the second value
                    lang*   ) ARG_LANG="${@:$OPTIND:1}" ; OPTIND=$(($OPTIND + 1));;
                    vcs=?*  ) ARG_VCS="$LONG_OPTARG" ;;
                    vcs*    ) ARG_VCS="${@:$OPTIND:1}" ; OPTIND=$(($OPTIND + 1));;
                    '' )      break ;; # "--" terminates argument
                                       # processing to allow giving
                                       # options for autogen.sh after
                                       # --
                    * )       echo "Illegal option --$OPTARG" >&2; exit 2;;
                esac;;
            \? ) exit 2 ;;  # getopts already reported the illegal
                            # option
        esac
    done
    shift $((OPTIND-1)) # remove parsed options and args from $@ list
    # reinitialize OPTIND to allow parsing again
    OPTIND=1
    # provide help output.
    if test x"${ARG_HELP}" = x"true"; then
        echo "${PROG} new [-h | --help] [-l | --lang <LANGUAGE>] [--vcs <VCS>] PROJECT_NAME"
        exit 0
    fi
    # get the argument
    if test x"${1}" = x"--"; then
        if test x"${PROJ}" = x""; then
            echo "Missing project name." >&2; exit 2
        else
            # nothing more to parse.
            # Remove -- from the remaining arguments
            shift 1
            break
        fi
    fi
    if test ! x"${1}" = x""; then
        PROJ="${1%/}" # without trailing slash
    fi
    # remove the argument, then continue the loop to allow putting
    # the options after the argument
    shift 1
done

Additional explanation for this is available from Adam Katz (2015) [250]. I’m allowed to include it here, because every answer on Stackoverflow is licensed under creativecommons attribution sharealike (cc by-sa) [251] and because cc by-sa is upwards compatible [252] to GPLv3 [144].

# From Adam Katz, 2015: http://stackoverflow.com/users/519360/adam-katz
# Available at http://stackoverflow.com/a/28466267/7666
# License: cc by-sa: https://creativecommons.org/licenses/by-sa/3.0/
while getopts ab:c-: arg; do
  case $arg in
    a )  ARG_A=true ;;
    b )  ARG_B="$OPTARG" ;;
    c )  ARG_C=true ;;
    - )  LONG_OPTARG="${OPTARG#*=}"
         case $OPTARG in
           alpha    )  ARG_A=true ;;
           bravo=?* )  ARG_B="$LONG_OPTARG" ;;
           bravo*   )  echo "No arg for --$OPTARG option" >&2; exit 2 ;;
           charlie  )  ARG_C=true ;;
           alpha* | charlie* )
                       echo "No arg allowed for --$OPTARG option" >&2; exit 2 ;;
           '' )        break ;; # "--" terminates argument processing
           * )         echo "Illegal option --$OPTARG" >&2; exit 2 ;;
         esac ;;
    \? ) exit 2 ;;  # getopts already reported the illegal option
  esac
done
shift $((OPTIND-1)) # remove parsed options and args from $@ list

With this and with the practical usage at the top you should be able to implement clean commandline parsing with ease.

Happy Hacking!

GNU Guix in 5 minutes

So you get excited when you hear about surviving a power-outage during updates without a hitch [253] and you want to give Guix [83] a try — but woes, you only have 5 minutes of time?

Fear not, that’s enough to get it up and running — all the way to per-user environments and package install as non-priviledged user!

The instructions here are from the official docs [254], specialized for a GNU Linux host and cut to what I need in a working system.

as user:

$ cd /tmp
$ wget ftp://alpha.gnu.org/gnu/guix/guix-binary-0.8.3.x86_64-linux.tar.xz

become root

$ sudo screen

unpack install and setup Guix

# tar xf guix-binary-0.8.3.x86_64-linux.tar.xz 
# mv var/guix /var/ && mv gnu /
# ln -sf /var/guix/profiles/per-user/root/guix-profile ~root/.guix-profile

Create the build users as per Build-Environment-Setup [255]:

# groupadd --system guixbuild
# for i in `seq -w 1 10`;
   do
      useradd -g guixbuild -G guixbuild           \
              -d /var/empty -s `which nologin`    \
              -c "Guix build user $i" --system    \
              guixbuilder$i;
   done

Run the daemon:

# ~root/.guix-profile/bin/guix-daemon --build-users-group=guixbuild

Switch to a second root window with CTRL-a c to adjust the PATH, allow substitutes from the Hydra build server, and to install and set locales [256] (required since we’re installing an overlay, not a full distro).

# echo 'PATH="$HOME/.guix-profile/bin:$HOME/.guix-profile/sbin:${PATH}"' >> $HOME/.bashrc
# echo 'export LOCPATH=$HOME/.guix-profile/lib/locale'  >> $HOME/.bashrc
# source $HOME/.bashrc
# guix archive --authorize < ~root/.guix-profile/share/guix/hydra.gnu.org.pub
# guix package -i glibc-utf8-locales

Allow all users to use the guix command (as long as guix-daemon is running):

# mkdir -p /usr/local/bin
# cd /usr/local/bin
# ln -s /var/guix/profiles/per-user/root/guix-profile/bin/guix

Switch back to your regular user and provide the guix profile. Also install the locales (remember that the installation is really per-user, though the users share packages if they install them both). The per-user profile will be generated the first time you run guix package.

$ ln -sf /var/guix/profiles/per-user/$(whoami)/guix-profile ~/.guix-profile
$ echo 'export PATH="$HOME/.guix-profile/bin:$HOME/.guix-profile/sbin:${PATH}"' >> $HOME/.bashrc
$ echo 'export LOCPATH=$HOME/.guix-profile/lib/locale'  >> $HOME/.bashrc
$ source $HOME/.bashrc
$ guix package -i glibc-utf8-locales

And now:

$ guix package -i guile-emacs --fallback
$ ~/.guix-profile/bin/emacs -Q

So you believed that only a pipe-dream, just like power-loss-resistant updates and functional packaging using the official GNU extension language [25]? I was glad to be proven wrong, and I hope you’ll feel the same ☺ (though guile-emacs [257] is still experimental it already allows calling elisp functions directly from scheme)

Happy Hacking!

GnuPG/PGP signature, short explanation

»What is the .asc file?« This explanation is intended to be copied as-is into emails when someone asks about your signature.

The .asc file is a signature which can be used to verify that the email was really sent by me and wasn’t tampered with.[1] It can be verified with standard email security tools like Enigmail[2], Gpg4win[3] or MacGPG[4] - and others tools supporting OpenPGP[5].

Best wishes,
Arne

[1]: For further information on signatures see
    https://www.gnupg.org/gph/en/manual/x135.html [258]

[2]: Enigmail enables secure communication in Thunderbird:
    https://addons.mozilla.org/de/thunderbird/addon/enigmail/ [259]

[3]: GPG4win provides secure encryption for Windows:
    http://gpg4win.org/download.html [260]

[4]: MacGPG provides encryption for MacOSX:
    https://gpgtools.org/ [261]

[5]: Encryption for other systems is available from the GnuPG website:
    https://www.gnupg.org/download/ [262]

Going from a simple Makefile to Autotools

Table of Contents

Intro

I recently started looking into Autotools, to make it easier to run my code on multiple platforms.

Naturally you can use cmake or scons or waf or ninja or tup, all of which are interesting in there own respect. But none of them has seen the amount of testing which went into autotools, and none of them have the amount of tweaks needed to support about every system under the sun. And I recently found pyconfigure [267] which allows using autotools with python and offers detection of library features.

Warning 2016: Contains some cargo-cult-programming [268] — my current setup is cleaner thanks to using AC_CONFIG_LINKS in configure.ac.

I had already used Makefiles for easily storing the build information of anything from python projects (python setup.py build) to my PhD thesis with all the required graphs.

I also had used scons for those same tasks.

But I wanted to test, what autotools have to offer. And I found no simple guide which showed me how to migrate from a Makefile to autotools - and what I could gain through that.

So I decided to write one.

# comments start with #
thing : required source files # separated by spaces
    build command
    second build command
# ^ this is a TAB.

all: doktorarbeit.pdf sink.pdf

sink.pdf : sink.tex images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png images/bona-marble.png images/boas-marble.png
    pdflatex sink.tex
    rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter

comp-t3-s07-tem-boas.png comp-t3-s07-tem-bona.png : nee-comp.pyx nee-comp.txt
    pyxplot nee-comp.pyx

doktorarbeit.pdf : doktorarbeit.org
    emacs --batch --visit "doktorarbeit.org" --funcall org-export-as-pdf  

dnl run `autoreconf -i` to generate a configure script. 
dnl Then run ./configure to generate a Makefile.
dnl Finally run make to generate the project.

AC_INIT([Doktorarbeit Inverse GHG], [0.1], [arne.babenhauserheide@kit.edu])
dnl we use the build type foreign here instead of gnu because I do not have a NEWS file and similar, yet.
AM_INIT_AUTOMAKE([foreign])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT

pdf_DATA = sink.pdf doktorarbeit.pdf

sink = sink.tex
pkgdata_DATA = images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png
dist_pkgdata_DATA = images/bona-marble.png images/boas-marble.png

plotdir = .
dist_plot_DATA = nee-comp.pyx nee-comp.txt

doktorarbeit = doktorarbeit.org

EXTRA_DIST = ${sink} ${dist_pkgdata_DATA} ${doktorarbeit}

MOSTLYCLEANFILES = \#* *~ *.bak # kill editor backups
CLEANFILES = ${pdf_DATA}
DISTCLEANFILES = ${pkgdata_DATA}

sink.pdf : ${sink} ${pkgdata_DATA} ${dist_pkgdata_DATA}
    TEXINPUTS=${TEXINPUTS}:$(srcdir)/:$(srcdir)/images// pdflatex $<
    rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter

${pkgdata_DATA} : ${dist_plot_DATA}
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then cp -u "$(i)" "$(notdir $(i))"; fi;)
    ${MKDIR_P} images
    pyxplot $<
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then rm -f "$(notdir $(i))"; fi;)

doktorarbeit.pdf : ${doktorarbeit}
    if test "$<" != "$(notdir $<)"; then cp -u "$<" "$(notdir $<)"; fi
    emacs --batch --visit "$(notdir $<)" --funcall org-export-as-pdf
    if test "$<" != "$(notdir $<)"; then rm -f "$(notdir $<)"; rm -f $(basename $(notdir $<)).tex $(basename $(notdir $<)).tex~; else rm -f $(basename $<).tex $(basename $<).tex~; fi

dnl run `autoreconf -i` to generate a configure script. 
dnl Then run ./configure to generate a Makefile.
dnl Finally run make to generate the project.

AC_INIT([Doktorarbeit Inverse GHG], [0.1], [arne.babenhauserheide@kit.edu])
# Check for programs I need for my build
AC_CANONICAL_TARGET
AC_ARG_VAR([emacs], [How to call Emacs.])
AC_CHECK_TARGET_TOOL([emacs], [emacs], [no])
AC_ARG_VAR([pyxplot], [How to call the Pyxplot plotting tool.])
AC_CHECK_TARGET_TOOL([pyxplot], [pyxplot], [no])
AC_ARG_VAR([pdflatex], [How to call pdflatex.])
AC_CHECK_TARGET_TOOL([pdflatex], [pdflatex], [no])
AS_IF([test "x$pdflatex" = "xno"], [AC_MSG_ERROR([cannot find pdflatex.])])
AS_IF([test "x$emacs" = "xno"], [AC_MSG_ERROR([cannot find Emacs.])])
AS_IF([test "x$pyxplot" = "xno"], [AC_MSG_ERROR([cannot find pyxplot.])])
# Run automake
AM_INIT_AUTOMAKE([foreign])
AM_MAINTAINER_MODE([enable])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT

pdf_DATA = sink.pdf doktorarbeit.pdf

sink = sink.tex
pkgdata_DATA = images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png
dist_pkgdata_DATA = images/bona-marble.png images/boas-marble.png

plotdir = .
dist_plot_DATA = nee-comp.pyx nee-comp.txt

doktorarbeit = doktorarbeit.org

EXTRA_DIST = ${sink} ${dist_pkgdata_DATA} ${doktorarbeit}

MOSTLYCLEANFILES = \#* *~ *.bak # kill editor backups
CLEANFILES = ${pdf_DATA}
DISTCLEANFILES = ${pkgdata_DATA}

sink.pdf : ${sink} ${pkgdata_DATA} ${dist_pkgdata_DATA}
    TEXINPUTS=${TEXINPUTS}:$(srcdir)/:$(srcdir)/images// @pdflatex@ $<
    rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter

${pkgdata_DATA} : ${dist_plot_DATA}
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then cp -u "$(i)" "$(notdir $(i))"; fi;)
    ${MKDIR_P} images
    @pyxplot@ $<
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then rm -f "$(notdir $(i))"; fi;)

doktorarbeit.pdf : ${doktorarbeit}
    if test "$<" != "$(notdir $<)"; then cp -u "$<" "$(notdir $<)"; fi
    @emacs@ --batch --visit "$(notdir $<)" --funcall org-export-as-pdf
    if test "$<" != "$(notdir $<)"; then rm -f "$(notdir $<)"; rm -f $(basename $(notdir $<)).tex $(basename $(notdir $<)).tex~; else rm -f $(basename $<).tex $(basename $<).tex~; fi  

#!/usr/bin/env python
## I need a couple of special builders for my projects
# the $SOURCE replacement only uses the first source file. $SOURCES gives all.
# specifying all source files makes it possible to rerun the build if a single source file changed.
orgexportpdf = 'emacs --batch --visit "$SOURCE" --funcall org-export-as-pdf'
pyxplot = 'pyxplot $SOURCE'
# pdflatex is quite dirty. I directly clean up after it with rm.
pdflatex = 'pdflatex $SOURCE -o $TARGET; rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb'

# build the PhD thesis from emacs org-mode.
Command("doktorarbeit.pdf", "doktorarbeit.org",
        orgexportpdf)

# create plots
Command(["images/comp-t3-s07-tem-boas.png", 
         "images/comp-t3-s07-tem-bona.png"], 
        ["nee-comp.pyx", 
         "nee-comp.txt"],
        pyxplot)

# build my sink.pdf
Command("sink.pdf", 
        ["sink.tex", 
         "images/comp-t3-s07-tem-boas.png", 
         "images/comp-t3-s07-tem-bona.png", 
         "images/bona-marble.png", 
         "images/boas-marble.png"],
        pdflatex)

# My editors leave tempfiles around. I want them gone after a build clean. This is not yet supported!
tempfiles = Glob('*~') + Glob('#*#') + Glob('*.bak')
# using this here would run the cleaning on every run.
#Command("clean", [], Delete(tempfiles))

sconsrun : scons
    python scons/bootstrap.py -Q

scons : 
    hg clone https://bitbucket.org/ArneBab/scons

#+BEGIN_LaTeX
%%% Local Variables:
%%% TeX-master: t
%%% End:
#+END_LaTeX

How to fix a bug, using the example of Quod Libet empty panes on Gentoo GNU/Linux (bug solving process)

Traceback (most recent call last):
  File "/home/arne/.quodlibet/plugins/songsmenu/albumart.py", line 51, in <module>
    from gi.repository import Gtk, Pango, GLib, Gdk, GdkPixbuf
  File "/usr/lib64/python2.7/site-packages/gi/__init__.py", line 27, in <module>
    from ._gi import _API, Repository
ImportError: cannot import name _API

equery belongs /usr/lib64/python2.7/site-packages/gi/__init__.py

 * Searching for /usr/lib64/python2.7/site-packages/gi/__init__.py ... 
dev-python/pygobject-3.8.3 (/usr/lib64/python2.7/site-packages/gi/__init__.py)

qlop -l pygobject

...
Thu Dec  5 00:26:27 2013 >>> dev-python/pygobject-3.8.3

echo =dev-python/pygobject-3.8.3 >> /usr/portage/package.mask
emerge -u pygobject

LANG=C quodlibet

/usr/lib64/python2.7/site-packages/quodlibet/qltk/songlist.py:44: GtkWarning: Unable to locate theme engine in module_path: "clearlooks",
  _label = gtk.Label().create_pango_layout("")

cd /usr/portage/media-sound/
# create the category in my local portage overlay, defined as
# PORTAGE_OVERLAY=/usr/local/portage in /etc/make.conf
mkdir -p /usr/local/portage/media-sound
# copy over the quodlibet directory, keeping the permissiong with -p
cp -rp quodlibet /usr/local/portage/media-sound
# most times it is enough to simply rename the ebuild to the new version
cd /usr/local/portage/media-sound/quodlibet
mv quodlibet-2.5.ebuild quodlibet-2.5.1.ebuild
# now prepare all the metadata portage needs - this requires
# app-portage/gentoolkit
ebuild quodlibet-2.5.1.ebuild digest compile 
# now it's prepared for the package manager. Just update it as usual:
emerge -u quodlibet

How to run your own GNU Hurd (in 140 letters)

Don’t want to rely on other’s opinions about the Hurd? How to run your own GNU Hurd, in 140 letters:

wget http://people.debian.org/~sthibault/hurd-i386/debian-hurd.img.tar.gz; tar xf de*hu*gz; qemu-system-x86_64 -hda de*hu*g -m 1G

[291]

For additional convenience and performance, setup ssh access and enable kvm:

wget http://people.debian.org/~sthibault/hurd-i386/debian-hurd.img.tar.gz; tar xf de*hu*gz; qemu-system-x86_64 -enable-kvm -net user,hostfwd=tcp:127.0.0.1:2222-:22 -net nic -m 1G -drive cache=writeback,file=$(ls de*hu*g)

⇒ login: root, no pw needed. Set a password for user demo:

passwd demo

⇒ log into your Hurd via ssh:

ssh demo@localhost -p 2222

That’s it: You run the Hurd. You you would want to do that? See cat translator_intro — and much more [292].

Additional information:

• README [293]
• hurd/running/qemu [294]
• Technical Advantages of the Hurd [292]
• Instant Hurd Development Environment [295]
• QEMU [296]

[297]

Huge datafiles in free culture projects under GPL

4 ways how large raw artwork files are treated in free culture projects to provide the editable source.¹

In the discussion about license compatibility of the creativecommons sharealike license [251] towards the GPL [144], Anthony asked [300] how the source-requirement is solved for artwork which often has huge raw files. These are the 4 basic ways I described in my answer.

1. The Wesnoth Way

“The Source is what we have”

The project just asks artists for full resolution PNG image files (without all the layering information) - and only uses these to develop the art. This was spearheaded by the GPL-licensed strategy game Battle for Wesnoth [205].

This is a viable strategy and also allows developing art, though a bit less convenient than with the layered sources. For example the illustrator who created many of the images in the RPG I work on used our PNG instead of her photoshop file to extract a die [301] from the cover she created for us [302]. She took the chance to also touch up the colors a bit - she had learned some new tricks to improve her paintings.

This clearly complies with the GPL, because the GPL just requires providing the file used for editing published file. If the released file is what you actually use to change published files, then the published file is the source.

2. The External Storage

“Use the FTP, Luke”

Here, files which are too big to be versioned effectively or which most people don’t need when working with the project get version-numbers and are put into an external storage - like an FTP server.

I do that for gimp-files: I put these into our public release-listing [303] via FTP. For example I used that for a multi-layer cover [304] which gets baked into our PDF.

3. The Elegant Way

“Make it so!”

Here huge files are simply versioned alongside other files and the versions to be used are created directly from the multi-layered files. The usual way to do that is a Makefile in which scripts explicitly define how the derived file can be extracted.

This is most elegant, because it has no duplication of information, the source is always trivial to find, it’s always clear that the derived file really originated from the source and it is easy to avoid quality loss or even reduce it later.

The disadvantage is that it can be very cumbersome to force new developers to get all huge files and then create them before being able to really start developing.

The common way to do this is a Makefile - for example the one I use for building my PhD thesis [305].

4. Pragmatic Elegance

“Hybrids win”

All the ways above can be combined: Huge files are put in version control, but the derived files are included, too, to make it easier for new people to get in. Maybe the huge files are only included on request - for example they could be stubs with which the version control system can retrieve the full files when the user wants them. This can partially be done with the largefiles extension [306] in Mercurial by just not getting the large files.

Also you can just keep separate raw files and derived files. This is also used in Battle for Wesnoth [205]: Optimized files of the right size for the game are stored in one folder while the bigger full resolution files are stored separately.

If you want to include free art in a GPL-covered work, I hope this article gave you some inspiration!

Immutable function arguments and variables

1. Dev A: “Fortran is totally outdated.”
2. Dev B: “I wish we could declare objects in function arguments or variable values as immutable in Java and Javascript.”

Fortran developer silently weeps:

! immutable 2D array as argument in Fortran
  integer, intent(in) :: arg(:,:)
! constant value
  character(len=10), parameter :: numbers = "0123456789"

See parameter vs. intent(in) [307].

(yes, I’m currently reading a Javascript book)

If you now want to see more of Fortran:

• Fortran Surprises [308]
• Writing a commandline tool in Fortran [309]

Installing GNU Guix 0.6, easily

mkdir guix && cd guix
wget http://alpha.gnu.org/gnu/guix/guix-0.6.tar.gz
wget http://alpha.gnu.org/gnu/guix/guix-0.6.tar.gz.sig
gpg --verify guix-0.?.tar.gz.sig

tar xf guix-0.?.tar.gz
cd guix-0.?
./configure && make -j16
sudo make install

sudo screen
groupadd guix-builder
for i in `seq 1 10`;
  do
    useradd -g guix-builder -G guix-builder           \
            -d /var/empty -s `which nologin`          \
            -c "Guix build user $i" --system          \
            guix-builder$i;
  done
exit

sudo mkdir -p /usr/local/var/nix/profiles/per-user/$USER
sudo chown -R $USER:$USER /usr/local/var/nix/profiles/per-user/$USER

chgrp 1002 /nix/store; chmod 1775 /nix/store

sudo screen
echo "#\!/bin/sh" >> /etc/local.d/guix-daemon.start
echo "guix-daemon --build-users-group=guix-builder &" >> /etc/local.d/guix-daemon.start
echo "#\!/bin/sh" >> /etc/local.d/guix-daemon.stop
echo "pkill guix-daemon" >> /etc/local.d/guix-daemon.stop
chmod +x /etc/local.d/guix-daemon.start
chmod +x /etc/local.d/guix-daemon.stop
exit

sudo /etc/local.d/guix-daemon.start

echo "export PATH=$PATH:~/.guix-profile/bin" >> ~/.bashrc
. ~/.bashrc

$ guix --help
Usage: guix COMMAND ARGS...
Run COMMAND with ARGS.

COMMAND must be one of the sub-commands listed below:

   build
   download
   gc
   hash
   import
   package
   pull
   refresh
   substitute-binary

Installing Scipy and PyNIO on a Bare Cluster with the Intel Compiler

export CC='gcc-4.8.1 -Wall -Werror'                                                          
echo $CC | grep \ - | sed 's/.* -/-/'                                                                     
-Werror

#!/bin/bash

# Untar

for i in *.tar* *.tgz; do
  tar xvf $i || exit
done

# Install

PREFIX=/home/ws/babenhau/
PYPREFIX=/home/ws/babenhau/python/

# Blas

cd BLAS
cp ../blas-make.inc make.inc || exit
#make -j9 clean
F77=ifort make -j9 || exit
#make -j9 install --prefix=$PREFIX
# OR for Intel compiler:
ifort -fPIC -FI -w90 -w95 -cm -O3 -xHost -unroll -c *.f || exit
#Continue below irrespective of compiler:
ar r libfblas.a *.o || exit
ranlib libfblas.a || exit
cd ..
ln -s BLAS blas

## Lapack

cd lapack-3.3.1
ln -s ../blas
# this has a hardcoded absolute path to blas in it: replace is with the appropriate one for you.
cp ../lapack-make.inc make.inc || exit
make -j9 clean  || exit
make -j9
make -j9 || exit
cp lapack_LINUX.a libflapack.a || exit
#make -j9 install --prefix=$PREFIX
cd ..

# C interface

patch -p0 < lapacke-ifort.diff

cd lapacke
# patch for lapack 3.3.1 and blas
for i in gnu inc intel ; do 
    sed -i s/lapack-3\.2\.1\\/lapack\.a/lapack-3\.3\.1\\/lapack_LINUX.a/ make.$i; 
    sed -i s/lapack-3\.2\.1\\/blas\.a/blas\\/blas_LINUX.a/ make.$i; 
done

make -j9 clean || exit
#make -j9
LINKER=ifort LDFLAGS=-nofor-main make -j9 # || exit
#LINKER=ifort LDFLAGS=-nofor-main make -j9 install
cd ..

## ATLAS

cd ATLAS
cp ../Make.Linux_HC3 . || exit
echo "ATLAS needs manual intervention. Run make by hand first."
#echo "just say yes. It makes some stuff we need later."
#make
#mv bin/Linux_UNKNOWNSSE2_8 bin/Linux_HC3
#for i in bin/Linux_HC3/*; do sed -i s/UNKNOWNSSE2_8/HC3/ $i ; done
#rm bin/Linux_HC3/Make.inc
#cd bin/Linux_HC3/
#ln -s ../../Make.Linux_HC3 Make.inc
#cd -

make -j9 install arch=Linux_HC3 || exit
cd lib
for i in Linux_HC3/* ; do ln -s $i ; done
cd ../bin
for i in Linux_HC3/* ; do ln -s $i ; done
cd ../include
for i in Linux_HC3/* ; do ln -s $i ; done
cd ..
cd ..

# Numpy and SciPy with intel compilers

# Read this: http://marklodato.github.com/2009/08/30/numpy-scipy-and-intel.html

# patching

patch -p0 < SuiteSparse.diff  || exit
patch -p0 < SuiteSparse-umfpack.diff  || exit

rm numpy
ln -s numpy-*.*.*/ numpy
patch -p0 < numpy-icc.diff  || exit
patch -p0 < numpy-icpc.diff || exit
patch -p0 <<EOF
--- numpy/numpy/distutils/fcompiler/intel.py      2009-03-29 07:24:21.000000000 -0400
+++ numpy/numpy/distutils/fcompiler/intel.py  2009-08-06 23:08:59.000000000 -0400
@@ -47,6 +47,7 @@
     module_include_switch = '-I'

     def get_flags(self):
+        return ['-fPIC', '-cm']
         v = self.get_version()
         if v >= '10.0':
             # Use -fPIC instead of -KPIC.
@@ -63,6 +64,7 @@
         return ['-O3','-unroll']

     def get_flags_arch(self):
+        return ['-xHost']
         v = self.get_version()
         opt = []
         if cpu.has_fdiv_bug():
EOF
# include -fPIC in the fcompiler.
sed -i "s/w90/w90\", \"-fPIC/" numpy/numpy/distutils/fcompiler/intel.py
# and more of that
patch -p0 < numpy-ifort.diff

rm scipy
ln -s scipy-*.*.*/ scipy

patch -p0 < scipy-qhull-icc.diff || exit
patch -p0 < scipy-qhull-icc2.diff || exit

# # unnecessary!
# patch -p0 <<EOF
# --- scipy/scipy/special/cephes/const.c    2009-08-07 01:56:43.000000000 -0400
# +++ scipy/scipy/special/cephes/const.c        2009-08-07 01:57:08.000000000 -0400
# @@ -91,12 +91,12 @@
# double THPIO4 =  2.35619449019234492885;       /* 3*pi/4 */
# double TWOOPI =  6.36619772367581343075535E-1; /* 2/pi */
# #ifdef INFINITIES
# -double INFINITY = 1.0/0.0;  /* 99e999; */
# +double INFINITY = __builtin_inff();
# #else
# double INFINITY =  1.79769313486231570815E308;    /* 2**1024*(1-MACHEP) */
# #endif
# #ifdef NANS
# -double NAN = 1.0/0.0 - 1.0/0.0;
# +double NAN = __builtin_nanf("");
# #else
# double NAN = 0.0;
# #endif
# EOF


# building

# TODO: try again later

cd SuiteSparse

make -j9 -C AMD || exit
make -j9 -C UMFPACK || exit

cd ..

# TODO: build numpy again and make sure it has blas and lapack (and ATLAS?)

cd numpy
python setup.py -v build_src config --compiler=intel build_clib \
    --compiler=intel build_ext --compiler=intel || exit
python setup.py install --prefix=$PYPREFIX || exit
cd ..

# scons and numscons
cd scons-2.0.1
python setup.py -v install --prefix=/home/ws/babenhau/python/ || exit
cd ..

git clone git://github.com/cournape/numscons.git
cd numscons 
python setup.py -v install --prefix=/home/ws/babenhau/python/  || exit
cd ..

# adapt /home/ws/babenhau/python/lib/python2.7/site-packages/numpy/distutils/fcompiler/intel.py by hand to include fPIC for intelem

cd scipy

PYTHONPATH=/home/ws/babenhau/python//lib/scons-2.0.1/ ATLAS=../ATLAS/ \
    LAPACK=../lapack-3.3.1/libflapack.a LAPACK_SRC=../lapack-3.3.1 BLAS=../BLAS/libfblas.a \
    F77=ifort f77_opt=ifort python setup.py -v config --compiler=intel --fcompiler=intelem build_clib \
    --compiler=intel --fcompiler=intelem build_ext --compiler=intel --fcompiler=intelem \
    -I../SuiteSparse/UFconfig # no exit, because we do the linking by hand later on.

# one file is C++ :(
icpc -fPIC -I/home/ws/babenhau/python/include/python2.7 -I/home/ws/babenhau/python/lib/python2.7/site-packages/numpy/core/include -I/home/ws/babenhau/python/lib/python2.7/site-packages/numpy/core/include -c scipy/spatial/qhull/src/user.c -o build/temp.linux-x86_64-2.7/scipy/spatial/qhull/src/user.o || exit

# linking by hand

# for x in csr csc coo bsr dia; do
#    icpc -xHost -O3 -fPIC -shared \
#        build/temp.linux-x86_64-2.7/scipy/sparse/sparsetools/${x}_wrap.o \
#        -o build/lib.linux-x86_64-2.7/scipy/sparse/sparsetools/_${x}.so || exit
# done
#icpc -xHost -O3 -fPIC -openmp -shared \
#   build/temp.linux-x86_64-2.7/scipy/interpolate/src/_interpolate.o \
#   -o build/lib.linux-x86_64-2.7/scipy/interpolate/_interpolate.so || exit

# build again with the C++ file already compiled

PYTHONPATH=/home/ws/babenhau/python//lib/scons-2.0.1/ ATLAS=../ATLAS/ \
    LAPACK=../lapack-3.3.1/libflapack.a LAPACK_SRC=../lapack-3.3.1 BLAS=../BLAS/libfblas.a \
    F77=ifort f77_opt=ifort python setup.py config --compiler=intel --fcompiler=intelem build_clib \
    --compiler=intel --fcompiler=intelem build_ext --compiler=intel --fcompiler=intelem \
    -I../SuiteSparse/UFconfig || exit

# make sure we have cephes
cd scipy/special
PYTHONPATH=/home/ws/babenhau/python//lib/scons-2.0.1/ ATLAS=../../../ATLAS/ \
    LAPACK=../../../lapack-3.3.1/libflapack.a LAPACK_SRC=../lapack-3.3.1 BLAS=../../../BLAS/libfblas.a \
    F77=ifort f77_opt=ifort python setup.py -v config --compiler=intel --fcompiler=intelem build_clib \
    --compiler=intel --fcompiler=intelem build_ext --compiler=intel --fcompiler=intelem \
    -I../../../SuiteSparse/UFconfig
cd ../..

# install
PYTHONPATH=/home/ws/babenhau/python//lib/scons-2.0.1/ ATLAS=../ATLAS/ \
    LAPACK=../lapack-3.3.1/libflapack.a LAPACK_SRC=../lapack-3.3.1 BLAS=../BLAS/libfblas.a \
    F77=ifort f77_opt=ifort python setup.py config --compiler=intel --fcompiler=intelem build_clib \
    --compiler=intel --fcompiler=intelem install --prefix=$PYPREFIX || exit

cd ..

# PyNIO

# netcdf-4

patch -p0 < netcdf-patch1.diff || exit
patch -p0 < netcdf-patch2.diff || exit

cd netcdf-4.1.3

CPPFLAGS="-I/home/ws/babenhau/libbutz/hdf5-1.8.7/include -I/home/ws/babenhau/include" LDFLAGS="-L/home/ws/babenhau/libbutz/hdf5-1.8.7/lib/ -L/home/ws/babenhau/lib -lsz -L/home/ws/babenhau/libbutz/szip-2.1/lib -L/opt/intel/Compiler/11.1/080/lib/intel64/libifcore.a -lifcore" ./configure --prefix=/home/ws/babenhau/ --enable-netcdf-4 --enable-shared || exit

make -j9; make check install -j9 || exit

cd ..

# NetCDF4
cd netCDF4-0.9.7
HAS_SZIP=1 SZIP_PREFIX=/home/ws/babenhau/libbutz/szip-2.1/ HAS_HDF5=1 HDF5_DIR=/home/ws/babenhau/libbutz/hdf5-1.8.7 HDF5_PREFIX=/home/ws/babenhau/libbutz/hdf5-1.8.7 HDF5_includedir=/home/ws/babenhau/libbutz/hdf5-1.8.7/include HDF5_libdir=/home/ws/babenhau/libbutz/hdf5-1.8.7/lib HAS_NETCDF4=1 NETCDF4_PREFIX=/home/ws/babenhau/ python setup.py build_ext --compiler="intel" --fcompiler="intel -fPIC" install --prefix $PYPREFIX
cd ..

# parallel netcdf and hdf5: ~/libbutz/

patch -p0 < pynio-fix-no-grib.diff || exit

cd PyNIO-1.4.1
HAS_SZIP=1 SZIP_PREFIX=/home/ws/babenhau/libbutz/szip-2.1/ HAS_HDF5=1 HDF5_DIR=/home/ws/babenhau/libbutz/hdf5-1.8.7 HDF5_PREFIX=/home/ws/babenhau/libbutz/hdf5-1.8.7 HDF5_includedir=/home/ws/babenhau/libbutz/hdf5-1.8.7/include HDF5_libdir=/home/ws/babenhau/libbutz/hdf5-1.8.7/lib HAS_NETCDF4=1 NETCDF4_PREFIX=/home/ws/babenhau/ python setup.py install --prefix=$PYPREFIX || exit
# TODO: Make sure that the install goes to /home/ws/.., not home/ws/...
cd ..

# satexp_utils.so

f2py -c -m satexp_utils --f77exec=ifort --f90exec=ifort interpolate_levels.F90 || exit

## pyhdf

# recompile hdf with fPIC - grr!
cd hdf-4*/
# Fix configure for compilers with - in the name.
patch -p0 < ../hdf-fix-configure.ac.diff
autoconf
FFLAGS="-ip -O3 -xHost -fPIC -r8" CFLAGS="-ip -O3 -xHost -fPIC" CXXFLAGS="$CFLAGS -I/usr/include/rpc  -DBIG_LONGS -DSWAP" F77=ifort ./configure --prefix=/home/ws/babenhau/ --disable-netcdf --with-szlib=/home/ws/babenhau/libbutz/szip-2.1 # --with-zlib=/home/ws/babenhau/libbutz/zlib-1.2.5 --with-jpeg=/home/ws/babenhau/libbutz/jpeg-8c
# finds zlib and jpeg due to LD_LIBRARY_PATH (hack but works…)
make
make install
cd ..

# build pyhdf
cd pyhdf-0.8.3/
INCLUDE_DIRS="/home/ws/babenhau/include:/home/ws/babenhau/libbutz/szip-2.1/include" LIBRARY_DIRS="/home/ws/babenhau/lib:/home/ws/babenhau/libbutz/szip-2.1/lib" python setup.py build -c intel --fcompiler ifort install --prefix=/home/ws/babenhau/python 
cd ..

## matplotlib

cd matplotlib-1.1.0
patch -p0 < ../matplotlib-add-icc-support.diff
python setup.py build -c intel install --prefix=/home/ws/babenhau/python
cd ..

# GEOS → http://download.osgeo.org/geos/geos-3.3.2.tar.bz2

cd geos*/ 
./configure --prefix=/home/ws/babenhau/
make
make check
make install 
cd ..

# basemap

easy_install --prefix /home/ws/babenhau/python basemap
# fails but should now have all dependencies.

cd basemap-*/

python setup.py build -c intel install --prefix=/home/ws/babenhau/python

cd ..

--- SuiteSparse/UMFPACK/Lib/GNUmakefile 2009-11-11 21:09:54.000000000 +0100
+++ SuiteSparse/UMFPACK/Lib/GNUmakefile 2011-09-09 14:18:57.000000000 +0200
@@ -9,7 +9,7 @@
 C = $(CC) $(CFLAGS) $(UMFPACK_CONFIG) \
     -I../Include -I../Source -I../../AMD/Include -I../../UFconfig \
     -I../../CCOLAMD/Include -I../../CAMD/Include -I../../CHOLMOD/Include \
-    -I../../metis-4.0/Lib -I../../COLAMD/Include
+    -I../../COLAMD/Include

 #-------------------------------------------------------------------------------
 # source files

--- SuiteSparse/UFconfig/UFconfig.mk    2011-09-09 13:14:03.000000000 +0200
+++ SuiteSparse/UFconfig/UFconfig.mk    2011-09-09 13:15:03.000000000 +0200
@@ -33,11 +33,11 @@
 # C compiler and compiler flags:  These will normally not give you optimal
 # performance.  You should select the optimization parameters that are best
 # for your system.  On Linux, use "CFLAGS = -O3 -fexceptions" for example.
-CC = cc
-CFLAGS = -O3 -fexceptions
+CC = icc
+CFLAGS = -O3 -xHost -fPIC -openmp -vec_report=0

 # C++ compiler (also uses CFLAGS)
-CPLUSPLUS = g++
+CPLUSPLUS = icpc

 # ranlib, and ar, for generating libraries
 RANLIB = ranlib
@@ -49,8 +49,8 @@
 MV = mv -f

 # Fortran compiler (not normally required)
-F77 = f77
-F77FLAGS = -O
+F77 = ifort
+F77FLAGS = -O3 -xHost
 F77LIB =

 # C and Fortran libraries
@@ -132,13 +132,13 @@
 # The path is relative to where it is used, in CHOLMOD/Lib, CHOLMOD/MATLAB, etc.
 # You may wish to use an absolute path.  METIS is optional.  Compile
 # CHOLMOD with -DNPARTITION if you do not wish to use METIS.
-METIS_PATH = ../../metis-4.0
-METIS = ../../metis-4.0/libmetis.a
+# METIS_PATH = ../../metis-4.0
+# METIS = ../../metis-4.0/libmetis.a

 # If you use CHOLMOD_CONFIG = -DNPARTITION then you must use the following
 # options:
-# METIS_PATH =
-# METIS =
+METIS_PATH =
+METIS =

 #------------------------------------------------------------------------------
 # UMFPACK configuration:
@@ -194,7 +194,7 @@
 # -DNSUNPERF       for Solaris only.  If defined, do not use the Sun
 #          Performance Library

-CHOLMOD_CONFIG =
+CHOLMOD_CONFIG = -DNPARTITION

 #------------------------------------------------------------------------------
 # SuiteSparseQR configuration:

--- configure.ac    2012-03-01 15:00:28.000000000 +0100
+++ configure.ac    2012-03-01 15:00:40.000000000 +0100
@@ -815,7 +815,7 @@
 dnl Report anything stripped as a flag in CFLAGS and 
 dnl only the compiler in CC_VERSION.
 CC_NOFLAGS=`echo $CC | sed 's/ -.*//'`
-CFLAGS_TO_ADD=`echo $CC | grep - | sed 's/.* -/-/'`
+CFLAGS_TO_ADD=`echo $CC | grep \ - | sed 's/.* -/-/'`
 if test -n $CFLAGS_TO_ADD; then
   CFLAGS="$CFLAGS_TO_ADD$CFLAGS"
 fi

--- lapacke/make.intel.old  2011-10-05 13:24:14.000000000 +0200
+++ lapacke/make.intel  2011-10-05 16:17:00.000000000 +0200
@@ -56,7 +56,7 @@
 # Ensure that the libraries have the same data model (LP64/ILP64).
 #
 LAPACKE = lapacke.a
-LIBS = ../../../lapack-3.3.1/lapack_LINUX.a ../../../blas/blas_LINUX.a -lm
+LIBS = /opt/intel/Compiler/11.1/080/lib/intel64/libifcore.a ../../../lapack-3.2.1/lapack.a ../../../lapack-3.2.1/blas.a -lm -ifcore
 #
 #  The archiver and the flag(s) to use when building archive (library)
 #  If your system has no ranlib, set RANLIB = echo.

diff -r 38c2a32c56ae matplotlib-1.1.0/setup.py
--- a/matplotlib-1.1.0/setup.py Fri Mar 02 12:29:47 2012 +0100
+++ b/matplotlib-1.1.0/setup.py Fri Mar 02 12:30:39 2012 +0100
@@ -31,6 +31,13 @@
 if major==2 and minor1<4 or major<2:
     raise SystemExit("""matplotlib requires Python 2.4 or later.""")

+if "intel" in sys.argv or "icc" in sys.argv:
+    try: # make it compile with the intel compiler
+        from numpy.distutils import intelccompiler
+    except ImportError:
+        print "Compiling with the intel compiler requires numpy."
+        raise
+
 import glob
 from distutils.core import setup
 from setupext import build_agg, build_gtkagg, build_tkagg,\

--- netcdf-4.1.3/fortran/ncfortran.h    2011-07-01 01:22:22.000000000 +0200
+++ netcdf-4.1.3/fortran/ncfortran.h    2011-09-14 14:56:03.000000000 +0200
@@ -658,7 +658,7 @@
  * The following is for f2c-support only.
  */

-#if defined(f2cFortran) && !defined(pgiFortran) && !defined(gFortran)
+#if defined(f2cFortran) && !defined(pgiFortran) && !defined(gFortran) &&!defined(__INTEL_COMPILER)

 /*
  * The f2c(1) utility on BSD/OS and Linux systems adds an additional

--- netcdf-4.1.3/nf_test/fortlib.c  2011-09-14 14:58:47.000000000 +0200
+++ netcdf-4.1.3/nf_test/fortlib.c  2011-09-14 14:58:38.000000000 +0200
@@ -14,7 +14,7 @@
 #include "../fortran/ncfortran.h"


-#if defined(f2cFortran) && !defined(pgiFortran) && !defined(gFortran)
+#if defined(f2cFortran) && !defined(pgiFortran) && !defined(gFortran) &&!defined(__INTEL_COMPILER)
 /*
  * The f2c(1) utility on BSD/OS and Linux systems adds an additional
  * underscore suffix (besides the usual one) to global names that have

--- numpy/numpy/distutils/intelccompiler.py 2011-09-08 14:14:03.000000000 +0200
+++ numpy/numpy/distutils/intelccompiler.py 2011-09-08 14:20:37.000000000 +0200
@@ -30,11 +30,11 @@
     """ A modified Intel x86_64 compiler compatible with a 64bit gcc built Python.
     """
     compiler_type = 'intelem'
-    cc_exe = 'icc -m64 -fPIC'
+    cc_exe = 'icc -m64 -fPIC -xHost -O3'
     cc_args = "-fPIC"
     def __init__ (self, verbose=0, dry_run=0, force=0):
         UnixCCompiler.__init__ (self, verbose,dry_run, force)
-        self.cc_exe = 'icc -m64 -fPIC'
+        self.cc_exe = 'icc -m64 -fPIC -xHost -O3'
         compiler = self.cc_exe
         self.set_executables(compiler=compiler,
                              compiler_so=compiler,

--- numpy-1.6.1/numpy/distutils/intelccompiler.py   2011-10-06 16:55:12.000000000 +0200
+++ numpy-1.6.1/numpy/distutils/intelccompiler.py   2011-10-10 10:26:14.000000000 +0200
@@ -10,11 +10,13 @@
     def __init__ (self, verbose=0, dry_run=0, force=0):
         UnixCCompiler.__init__ (self, verbose,dry_run, force)
         self.cc_exe = 'icc -fPIC'
+   self.cxx_exe = 'icpc -fPIC'
         compiler = self.cc_exe
+   compiler_cxx = self.cxx_exe
         self.set_executables(compiler=compiler,
                              compiler_so=compiler,
-                             compiler_cxx=compiler,
-                             linker_exe=compiler,
+                             compiler_cxx=compiler_cxx,
+                             linker_exe=compiler_cxx,
                              linker_so=compiler + ' -shared')

 class IntelItaniumCCompiler(IntelCCompiler):

--- numpy-1.6.1/numpy/distutils/fcompiler/intel.py.old  2011-10-10 17:52:34.000000000 +0200
+++ numpy-1.6.1/numpy/distutils/fcompiler/intel.py  2011-10-10 17:53:51.000000000 +0200
@@ -32,7 +32,7 @@
     executables = {
         'version_cmd'  : None,          # set by update_executables
         'compiler_f77' : [None, "-72", "-w90", "-fPIC", "-w95"],
-        'compiler_f90' : [None],
+        'compiler_f90' : [None, "-fPIC"],
         'compiler_fix' : [None, "-FI"],
         'linker_so'    : ["<F90>", "-shared"],
         'archiver'     : ["ar", "-cr"],
@@ -129,7 +129,7 @@
         'version_cmd'  : None,
         'compiler_f77' : [None, "-FI", "-w90", "-fPIC", "-w95"],
         'compiler_fix' : [None, "-FI"],
-        'compiler_f90' : [None],
+        'compiler_f90' : [None, "-fPIC"],
         'linker_so'    : ['<F90>', "-shared"],
         'archiver'     : ["ar", "-cr"],
         'ranlib'       : ["ranlib"]
@@ -148,7 +148,7 @@
         'version_cmd'  : None,
         'compiler_f77' : [None, "-FI", "-w90", "-fPIC", "-w95"],
         'compiler_fix' : [None, "-FI"],
-        'compiler_f90' : [None],
+        'compiler_f90' : [None, "-fPIC"],
         'linker_so'    : ['<F90>', "-shared"],
         'archiver'     : ["ar", "-cr"],
         'ranlib'       : ["ranlib"]
@@ -180,7 +180,7 @@
         'version_cmd'  : None,
         'compiler_f77' : [None,"-FI","-w90", "-fPIC","-w95"],
         'compiler_fix' : [None,"-FI","-4L72","-w"],
-        'compiler_f90' : [None],
+        'compiler_f90' : [None, "-fPIC"],
         'linker_so'    : ['<F90>', "-shared"],
         'archiver'     : [ar_exe, "/verbose", "/OUT:"],
         'ranlib'       : None
@@ -232,7 +232,7 @@
         'version_cmd'  : None,
         'compiler_f77' : [None,"-FI","-w90", "-fPIC","-w95"],
         'compiler_fix' : [None,"-FI","-4L72","-w"],
-        'compiler_f90' : [None],
+        'compiler_f90' : [None, "-fPIC"],
         'linker_so'    : ['<F90>',"-shared"],
         'archiver'     : [ar_exe, "/verbose", "/OUT:"],
         'ranlib'       : None

--- PyNIO-1.4.1/Nio.py  2011-09-14 16:00:13.000000000 +0200
+++ PyNIO-1.4.1/Nio.py  2011-09-14 16:00:18.000000000 +0200
@@ -98,7 +98,7 @@
         if ncarg_dir == None or not os.path.exists(ncarg_dir) \
           or not os.path.exists(os.path.join(ncarg_dir,"lib","ncarg")):
             if not __formats__['grib2']:
-                return None
+                return "" # "", because an env variable has to be a string.
             else:
                 print "No path found to PyNIO/ncarg data directory and no usable NCARG installation found"
                 sys.exit()

--- scipy/scipy/spatial/qhull/src/qhull_a.h 2011-02-27 11:57:03.000000000 +0100
+++ scipy/scipy/spatial/qhull/src/qhull_a.h 2011-09-09 15:42:12.000000000 +0200
@@ -102,13 +102,13 @@
 #elif defined(__MWERKS__) && defined(__INTEL__)
 #   define QHULL_OS_WIN
 #endif
-#if defined(__INTEL_COMPILER) && !defined(QHULL_OS_WIN)
-template <typename T>
-inline void qhullUnused(T &x) { (void)x; }
-#  define QHULL_UNUSED(x) qhullUnused(x);
-#else
+/*#if defined(__INTEL_COMPILER) && !defined(QHULL_OS_WIN)*/
+/*template <typename T>*/
+/*inline void qhullUnused(T &x) { (void)x; }*/
+/*#  define QHULL_UNUSED(x) qhullUnused(x);*/
+/*#else*/
 #  define QHULL_UNUSED(x) (void)x;
-#endif
+*/#endif*/

 /***** -libqhull.c prototypes (alphabetical after qhull) ********************/

--- scipy/scipy/spatial/qhull/src/qhull_a.h 2011-09-09 15:43:54.000000000 +0200
+++ scipy/scipy/spatial/qhull/src/qhull_a.h 2011-09-09 15:45:17.000000000 +0200
@@ -102,13 +102,7 @@
 #elif defined(__MWERKS__) && defined(__INTEL__)
 #   define QHULL_OS_WIN
 #endif
-/*#if defined(__INTEL_COMPILER) && !defined(QHULL_OS_WIN)*/
-/*template <typename T>*/
-/*inline void qhullUnused(T &x) { (void)x; }*/
-/*#  define QHULL_UNUSED(x) qhullUnused(x);*/
-/*#else*/
 #  define QHULL_UNUSED(x) (void)x;
-*/#endif*/

 /***** -libqhull.c prototypes (alphabetical after qhull) ********************/

--- scipy-0.9.0/scipy/spatial/setup.py  2011-10-10 17:11:23.000000000 +0200
+++ scipy-0.9.0/scipy/spatial/setup.py  2011-10-10 17:11:09.000000000 +0200
@@ -22,6 +22,8 @@
                                      get_numpy_include_dirs()],
                        # XXX: GCC dependency!
                        #extra_compiler_args=['-fno-strict-aliasing'],
+                       # XXX intel compiler dependency
+                       extra_compiler_args=['-lifcore'],
                        )

     lapack = dict(get_info('lapack_opt'))

JSON will bite us badly

JSON, the javascript object notation format, is everywhere nowadays. But there are 3 facts which will challenge its dominance.

1. CPU cores are not getting much faster.
2. You can rent VMs per core, and you pay per core.
3. The network is still getting faster and cheaper, and HTTP/2 reduces the minimum cost per file.

Due to these changes, servers will become CPU bound again, and basic data structures on the web will become much more relevant. But the most efficient parsing of JSON requires guessing the final data structure while reading the data.

Therefore the changing costs will bring a comeback for binary data structures, and WebAssembly will provide efficient parsers and emitters in the clients.

Look at a typical website and count how much of the dynamic data it uses is structured data. Due to this I expect that 5 years from now, there will be celebrity talks with titles like

Scaling 10x higher with streams of structured data.

(And yes, that tech communication often works like this is a problem.)

If you have deep-rooted doubts, have a look at Towards a JavaScript Binary AST [369], which convinced me to finally publish this article.

(and parsing JSON is a minefield [370])

Memory requirement of Python datastructures: numpy array, list of floats and inner array

Easily answering the question: “How much space does this need?”

Intro

We just had the problem to find out whether a given dataset will be shareable without complex trickery. So we took the easiest road and checked the memory requirements of the datastructure.

If you have such a need, there’s always a first stop: Fire up the interpreter and try it out.

The test

We just created a three dimensional numpy array of floats and then looked at the memory requirement in the system monitor - conveniently bound to CTRL-ESC in KDE [207]. By making the array big enough we can ignore all constant costs and directly get the cost per stored value by dividing the total memory of the process by the number of values.

All our tests are done in Python3.

Numpy

For numpy we just create an array of random values cast to floats:

import numpy as np
a = np.array(np.random.random((100, 100, 10000)), dtype="float")

Also we tested what happens when we use "f4" and "f2" instead of "float" as dtype in numpy.

Native lists

For the native lists, we use the same array, but convert it to a list of lists of lists:

import numpy as np
a = [[[float(i) for i in j] for j in k] 
     for k in list(np.array(np.random.random((100, 100, 10000)), dtype="float"))]

Array module

Instead of using the full-blown numpy, we can also turn the inner list into an array.

import numpy as np
a = [[array.array("d", [float(i) for i in j]) for j in k] 
     for k in list(np.array(np.random.random((100, 100, 10000)), dtype="float"))]

The results

With a numpy array we need roughly 8 Byte per float. A linked list however requires roughly 32 Bytes per float. So switching from native Python to numpy reduces the required memory per floating point value by factor 4.

Using an inner array (via array module) instead of the innermost list provides roughly the same gains.

I would have expected factor 3: The value plus a pointer to the next and to the previous entry.

The details are in the following table.

This test was conducted on a 64 bit system, so floats are equivalent to doubles.

The scipy documentation provides a list of all the possible dtype definitions cast to C-types [371].

Summary

In Python large numpy arrays require 4 times less memory than a linked list structure with the same data. Using an inner array from the array module instead of the innermost list provides roughly the same gains.

Ogg Theora and h.264 - which video codec as standard for internet-video?

We had a kinda long discussion [373] on identi.ca [374] about Ogg Theora [375] and h.264, and since we lacked a simple comparision method, I hacked up a quick script to test them.

It uses frames from Big Buck Bunny [376] and outputs the files bbb.ogg and bbb.264 (license: cc by [377]).

The ogg file looks like this:

The h.264 file looks like this: download [380]

Results

What you can see by comparing both is that h.264 wins in terms of raw image quality at the same bitrate (single pass).

So why am I still strongly in favor of Ogg Theora?

The reason is simple:

Due to licensing costs of h.264 (a few millions per year, due from 2015 onwards) making h.264 the standard for internet video would have the effect that only big companies would be able to make a video enabled browser - or we would get a kind of video tax for free software: if you want to view internet video with free software, you have to pay for the right to use the x264 library (else the developers couldn't cough up the money to pay for the parent license). And noone but the main developers and huge corporations could distribute the x264 library, because they’d have to pay license fees for that.

And noone could hack on the browser or library and distribute the changed version, so the whole idea of free software would be led ad absurdum. It wouldn't matter that all code would be free licensed, since only those with a h.264 patent license could change it.

So this post boils down to a simple message:

“Support !theora against h.264 and #flash [as video codec for the web]. Otherwise only big companies will be able to write video browsers - or we get a h.264 tax on !fs” [381]

Theoras raw quality may still be worse, but the license costs and their implications provide very clear reasons for supporting Theora - which in my view are far more important than raw technical stuff.

The test-script

for k in {0..1}
     do for i in {0..9}
         do for j in {0..9}
             do
wget http://media.xiph.org/BBB/BBB-360-png/big_buck_bunny_00$k$i$j.png
         done
     done
done

mplayer -vo yuv4mpeg -ao null -nosound mf://*png -mf fps=50

theora_encoder_example -z 0 --soft-target -V 400 -o bbb.ogg stream.yuv

mencoder stream.yuv -ovc x264 -of rawvideo -o bbb.264 -x264encopts bitrate=400 -aspect 16:9 -nosound -vf scale=640:360,harddup

Phoronix conclusions distort their results, shown with the example of GCC vs. LLVM/Clang On AMD's FX-8350 Vishera

Phoronix recently did a benchmark of GCC vs. LLVM on AMD hardware [383]. Sadly their conclusion did not fit the data they showed. Actually it misrepresented the data so strongly, that I decided to speak up here instead of having my comments disappear in their forums [384]. This post was started on 2013-05-14 and got updates when things changed - first for the better, then for the worse.

Update 3 (the last straw, 2013-11-09): In the recent most blatant attack by Phoronix on copyleft programs - this time openly targeted at GNU [385] - Michael Larabel directly misrepresented a post from Josh Klint to badmouth GDB (Josh confirmed [386] this¹). Josh gave a report of his initial experience with GDB in a Kickstarter Update [387] in which he reported some shortcomings he saw in GDB (of which the major gripe is easily resolved with better documentation²) and concluded with “the limitations of GDB are annoying, but I can deal with it. It's very nice to be able to run and debug our editor on Linux”. Michael Larabel only quoted the conclusion up to “annoying” and abused that to support the claim that game developers (in general) call GDB “crap” and for further badmouthing of GDB. With this he provided the straw which I needed to stop reading Phoronix: Michael Larabel is hostile to copyleft and in particular to GNU and he goes as far as rigging test results³ and misrepresenting words of others to further his agenda. I even donated to Phoronix a few times in the past. I guess I won’t do that again, either. I should have learned from the error of the german pirates and should have avoided reading media which is controlled by people who want to destroy what I fight for (sustainable free software).
Update 2 (2013-07-06): But the next [388] went down the drain again… “Of course, LLVM/Clang 3.3 still lacks OpenMP support, so those tests are obviously in favor of GCC.” — I couldn’t find a better way to say that those tests are completely useless while at the same time devaluing OpenMP support as “ignore this result along with all others where GCC wins”…
Update (2013-06-21): The recent report of GCC 4.8 vs. LLVM 3.3 [389] looks much better. Not perfect, but much better.

Taking out the OpenMP benchmarks (where GCC naturally won, because LLVM only processes those tests single-threaded) and the build times (which are irrelevant to the speed of the produced binaries), their benchmark [383] had the following result:

LLVM is slower than GCC by:

• 10.2% (HMMer)
• 12.7% (MAFFT)
• 6.8% (BLAKE2)
• 9.1% (HIMENO)
• 42.2% (C-Ray)

With these results (which were clearly visible on their result summary on OpenBenchmarking [390], Michael Larabel from Phoronix concluded:

» The performance of LLVM/Clang 3.3 for most tests is at least comparable to GCC «

Nobu [391] from their Forums supplied a conclusion which represents the data much better:

» GCC is much faster in anything which uses OpenMP, and moderately faster or equal in anything (except compile times) which doesn't [use OpenMP] «

But Michael from Phoronix did not stop at just ignoring the performance difference between GCC and LLVM. He went on claiming, that

In a few benchmarks LLVM/Clang is faster, particularly when it comes to build times.

And this is blatant reality-distortion which I am very tempted to ascribe to favoritism. LLVM is not “particularly” faster when it comes to build times.

LLVM on AMD FX-8350 Vishera is faster ONLY when it comes to build times!

This was not the first time that I read data-distorting conclusions on Phoronix - and my complaints about that in their forum did not change their actions. So I hope that my post here can help making them aware that deliberately distorting test results is unacceptable.

For my work, compiler performance is actually quite important, because I use programs which run for days or weeks, so 10% runtime reduction can mean saving several days - not counting the cost of using up cluster time.

To fix their blunders, what they would have to do is:

• Avoiding Benchmarks which only one compiler supports properly (OpenMP).
• Marking the compile time tests explicitely, so they strongly stand out from the rest, because they measure a completely different parameter than the other tests: Compiler Runtime vs. Performance of the Compiled Binaries.
• Writing conclusions which actually fit their results.

Their current approach gives a distinct disadvantage to GCC (even for the OpenMP tests, because they convey the notion that if LLVM only had OpenMP, it would be better in everything - which as this test shows is simply false), so the compiler-tests from Phoronix work as covert propaganda against GCC, even in tests where GCC flat-out wins. And I already don’t like open propaganda, but when the propaganda gets masked as objective testing, I actually get angry.

I hope my post here can help move them towards doing proper testing again.

PS: I write so strongly here, because I actually like the tests from Phoronix a lot. I think we need rather more than less testing and their testsuite actually seems to do a good job - when given the right parameters - so seeing Phoronix distorting the tests to a point where they become almost useless (except as political tool against GCC) is a huge disappointment to me.

Python for beginning programmers

(written on ohloh [399] for Python)

Since we already have two good reviews from experienced programmers, I'll focus on the area I know about: Python as first language.

My experience:

• I began to get into coding only a short time ago. I already knew about processes in programs, but not how to get them into code.
• I wanted to learn C/C++ and failed at general structure. After a while I could do it, but it didn't feel right.
• I tried my luck with Java and didn't quite get going.
• Then I tried Python, and got in at once.

Advantages of Python:

• The structure of programs can be understood easily.
• The Python interpreter lets you experiment very quickly.
• You can realize complex programs, but Python also allows for quick and simple scripting.
• Code written by others is extremely readable.
• And coding just flows - almost like natural speaking/thinking.

How it looks:

def hello(user):
    print("Hello " + user + "!")
hello("Fan")
# prints Hello Fan! on screen

As a bonus, there is the great open book How to Think Like a Computer Scientist [400] which teaches Python and is being used for teaching Python and Programming at universities.

So I can wholeheartedly recommend Python to beginners in programming, and as the other reviews on Ohloh show, it is also a great language for experienced programmers and seems to be a good language to accompany you in your whole coding life.

PS: Yes, I know about the double meaning of "first language" :)

Recursion wins!

I recently read the little schemer [401] and that got me thinking about recursion and loops.

After starting my programming life with Python, I normally use for-loops to solve problems. But actually they are an inferior mechanism when compared to recursion, if (and only if) the language provides proper syntactic support for that. Since that claim pretty much damns Python on a theoretical level (even though it is still a very good tool in practice and I still love it!), I want to share a simplified version of the code which made me realize this.

Let’s begin with how I would write that code in Python.

res = ""
instring = False
for letter in text:
    if letter = "\"":
        # special conditions for string handling go here
        # lots of special conditions
        # and more special conditions
        # which cannot easily be moved out, 
        # because we cannot skip multiple letters
        # in one step
        instring = not instring
    if instring:
        res += letter
        continue
    # other cases

Did you spot the comment “special conditions go here”? That’s the point which damns for-loops: You cannot easily factor out these special conditions.¹ In this example all the complexity is in the variable instring. But depending on the usecase, this could require lots of different states being tracked within the loop and cluttering up the namespace as well as entangling complexity from different parts of the loop.

This is how the same could be done with proper let-recursion:

; first get SRFI-71: multi-value let for syntactic support for what I
; want to do
use-modules : srfi srfi-71

let process-text
    : res ""
      letter : string-take text 1
      unprocessed : string-drop text 1
    when : equal? letter "\""
           let-values 
               ; all the complexity of string-handling is neatly
               ; confined in the helper-function consume-string
               : (to-res next-letter still-unprocessed) : consume-string unprocessed
               process-text
                   string-append res to-res
                   . next-letter
                   . still-unprocessed
    ; other cases

The basic code for recursion is a bit longer, because the new values in the next step of the processing are given explicitly. But it is almost trivial to shell out parts of the loop to another function. It just needs to return the next state of the recursion.

And that’s what consume-string does:

define : consume-string text
    let
        : res ""
          next-letter : string-take text 1
          unprocessed : string-drop text 1
        ; lots of special handling here
        values res next-letter unprocessed

To recite from the Zen of Python [402]:

Explicit is better than implicit.

It’s funny to see how Guile Scheme [25] allows me to follow that principle more thoroughly than Python.

(I love Python, but this is a case where Scheme simply wins - and I’m not afraid to admit that)

PS: Actually I found this technique when thinking about use-cases for multiple return-values of functions.

PPS: This example uses wisp-syntax [403] for the scheme-examples to avoid killing Pythonistas with parens.

Reducing the Python startup time

The python startup time always nagged me (17-30ms) and I just searched again for a way to reduce it, when I found this:

The Python-Launcher [405] caches GTK imports and forks new processes to reduce the startup time of python GUI programs.

Python-launcher does not solve my problem directly, but it points into an interesting direction: If you create a small daemon which you can contact via the shell to fork a new instance, you might be able to get rid of your startup time.

To get an example of the possibilities, download the python-launcher [406] and socat [407] and do the following:

PYTHONPATH="../lib.linux-x86_64-2.7/" python python-launcher-daemon &
echo pass > 1
for i in {1..100}; do 
    echo 1 | socat STDIN UNIX-CONNECT:/tmp/python-launcher-daemon.socket & 
done

Todo: Adapt it to a given program and remove the GTK stuff. Note the & at the end: Closing the socket connection seems to be slow, so I just don’t wait for socat to finish. Breaks at somewhere over 200 simultaneous connections. Option: Use a datagram socket instead.

The essential trick is to just create a server which opens a socket. Then it reads all the data from the socket. Once it has the data, it forks like the following:

        pid = os.fork()
        if pid:
            return

        signal.signal(signal.SIGPIPE, signal.SIG_DFL)
        signal.signal(signal.SIGCHLD, signal.SIG_DFL)

        glob = dict(__name__="__main__")
        print 'launching', program
        execfile(program, glob, glob)

        raise SystemExit

Running a program that way 100-times took just 0.23 seconds for me so the Python startup time of 17ms got reduced to 2.3ms.

You might have to switch from forking to just executing the code instead of forking if you want to be even faster and the code snippets are small. For example when running the same test without the fork and the signals, 100 executions of the same code took just 0.09s, cutting down the startup time to an impressing 0.9ms - with the cost of no longer running in parallel.

(That’s what I also do with emacsclient… My emacs takes ~30s to start (due to excessive use of additional libraries I added), but emacsclient -c shows up almost instantly.)

I tested the speed by just sending a file with the following snippet to the server:

import time
with open("2", "a") as f:
    f.write(str(time.time()) + "\n")

Note: If your script only needs the included python libraries (batteries) and no custom-installed libs, you can also reduce the startuptime by avoiding site initialization:

python -S [script]

Without -S python -c '' takes 0.018s for me. With -S I am down to

time python -S -c '' → 0.004s. 

Note that you might miss some installed packages that way. This is slower than the daemon method by up to factor 4 (4ms instead of 0.9), but still faster than the default way. Note that cold disk buffers can make the difference much bigger on the first run which is not relevant in this case but very much relevant in general for the impression of startup speed.

PS: I attached the python-launcher 0.1.0 in case its website goes down. License: GPL and MIT; included. This message was originally written at stackoverflow [408].

Relicensing a project from GPLv2 or later to AGPLv3 or later

Switching from GPLv2 or later [410] to AGPL [411] is perfectly legal. But if it is not your own project, it is often considered rude.

This does not relicense the original code, it just sets the license of new code and of the project as a whole. The old code stays GPLv2+, but when it is combined with the new code under AGPLv3 (or later), the combined project will be under AGPLv3 (or later).

However switching from GPL2+ to AGPL3(+) without consensus of all other contributors is considered rude, because it could prevent some of the original authors from using future versions of the project. Their professional use of the project might depend on the loopholes in the copyleft of the GPL.

And the ones you will want most of all as users of your fork of a mostly discontinued project are the original authors, because that can mend the split between the two versions.

This question came up in a continuation of a widely used package whose development seemed to have stalled. The discussion was unfocussed, so I decided to write succinct information for all who might find themselves in a similar situation. I will not link to them, because I do not wish to re-ignite the discussion through an influx of rehashed arguments.

Replacing man with info

Update: If you prefer vi-keys, adjust the function below to call info --vi-keys instead of plain info. You could then call that function iv ☺

function i()
{
    INFOVERSIONLINE=$(info --version | head -n 1)
    INFOVERSION="${INFOVERSIONLINE##* }"
    INFOGT5=$(if test ${INFOVERSION%%.*} -gt 5; then echo true; else echo false; fi)
    # start with special cases which are quick to check for
    if test $# -lt 1; then
        # show info help notice
        info --help
    elif test $# -gt 1 && ! echo $1 | grep -q "[0-9]"; then
        # user sent complex request, but not with a section command. Just use info
        info "$@"
    elif test $# -gt 1 && echo $1 | grep -q "[0-9]"; then
        # user sent request for a section from the man pages, we must defer to man
        man "$@"
    elif test x"$1" = x"info"; then
        # for old versions of info, calling info --usage info fails to
        # provide info about calling info
        if test x"$INFOGT5" = x"true"; then
            info --usage info
        else
            info --usage -f info-stnd
        fi
    elif test x"$1" = x"man"; then
        # info --all -w ./man fails to find the man man page
        info man
    else
        # start with a fast but incomplete info lookup
        INFOPAGELOCATION="$(info --all -w ./"$@" | head -n 1)"
        INFOPAGELOCATION_PAGENAME="$(info --all -w "$1".info | head -n 1)"
        INFOPAGELOCATION_COREUTILS="$(info -w coreutils -n "$@")"
        # check for usage from fast info, if that fails check man and
        # if that also fails, just get the regular info page.
        if test x"${INFOPAGELOCATION}" = x"*manpages*" || test x"${INFOPAGELOCATION}" != x""; then
           info "$@"; # use info to read the known page, man or info
        elif test x"${INFOPAGELOCATION_COREUTILS}" != "x" && info -f "${INFOPAGELOCATION_COREUTILS}" -n "$@" | head -n 1 | grep -q -i "$@"; then
            # coreutils utility
            info -f "${INFOPAGELOCATION_COREUTILS}" -n "$@"
        elif test x"${INFOPAGELOCATION}" = x"" && test x"${INFOPAGELOCATION_PAGENAME}" = x""; then
           # unknown to quick search, try slow search or defer to man.
           # TODO: it would be nice if I could avoid this double search.
           if test x"$(info -w "$@")" = x"*manpages*"; then
               info "$@"
           else
               # defer to man, on error search for alternatives
               man "$@" || (echo nothing found, searching info ... && \
                            while echo $1 | grep -q '^[0-9]$'; do shift; done && \
                            info -k "$@" && false)
           fi
        elif test x"${INFOPAGELOCATION_PAGENAME}" != x""; then
             # search for alternatives (but avoid numbers)
           info --usage -f "${INFOPAGELOCATION_PAGENAME}" 2>/dev/null || man "$@" ||\
             (echo searching info &&\
              while echo $1 | grep -q '^[0-9]$'; do shift; done && \
              info -k "$@" && false)            
        else # try to get usage instructions, then try man, then
             # search for alternatives (but avoid numbers)
           info --usage -f "${INFOPAGELOCATION}" 2>/dev/null || man "$@" ||\
             (echo searching info &&\
              while echo $1 | grep -q '^[0-9]$'; do shift; done && \
              info -k "$@" && false)
        fi
        # ensure that unsuccessful requests report an error status
        INFORETURNVALUE=$?
        unset INFOPAGELOCATION
        unset INFOPAGELOCATION_COREUTILS
        if test ${INFORETURNVALUE} -eq 0; then
            unset INFORETURNVALUE
            return 0
        else
            unset INFORETURNVALUE
            return 1
        fi
    fi
}

{{{fun}}}
i info | head
echo ...

Next: Cursor Commands,  Prev: Stand-alone Info,  Up: Top

2 Invoking Info
***************

GNU Info accepts several options to control the initial node or nodes
being viewed, and to specify which directories to search for Info files.
Here is a template showing an invocation of GNU Info from the shell:

     info [OPTION...] [MANUAL] [MENU-OR-INDEX-ITEM...]
...

{{{fun}}}
i grep | head | sed 's/\[[0-9]*m//g' # stripping simple colors
echo ...

Next: Regular Expressions,  Prev: Introduction,  Up: Top

2 Invoking ‘grep’
*****************

The general synopsis of the ‘grep’ command line is

     grep OPTIONS PATTERN INPUT_FILE_NAMES

There can be zero or more OPTIONS.  PATTERN will only be seen as such
...

{{{fun}}}
i stat | head

Next: sync invocation,  Prev: du invocation,  Up: Disk usage

14.3 ‘stat’: Report file or file system status
==============================================

‘stat’ displays information about the specified file(s).  Synopsis:

     stat [OPTION]… [FILE]…

   With no option, ‘stat’ reports all information about the given files.

{{{fun}}}
i libc stat | head

Next: Testing File Type,  Prev: Attribute Meanings,  Up: File Attributes

14.9.2 Reading the Attributes of a File
---------------------------------------

To examine the attributes of files, use the functions 'stat', 'fstat'
and 'lstat'.  They return the attribute information in a 'struct stat'
object.  All three functions are declared in the header file
'sys/stat.h'.

{{{fun}}}
i man | head | sed "s,\x1B\[[0-9;]*[a-zA-Z],,g" # stripping colors

man(1)                      General Commands Manual                     man(1)



NAME
       man  -  Formatieren  und Anzeigen von Seiten des Online-Handbuches (man
       pages)
       manpath - Anzeigen  des  Benutzer-eigenen  Suchpfades  für  Seiten  des
       Online-Handbuches (man pages)

{{{fun}}}
i 2 stat | head | sed "s,\x1B\[[0-9;]*[a-zA-Z],,g" # stripping colors

STAT(2)                    Linux Programmer's Manual                   STAT(2)



NAME
       stat, fstat, lstat, fstatat - get file status

SYNOPSIS
       #include <sys/types.h>
       #include <sys/stat.h>

{{{fun}}}
i em | head
echo ...

nothing found, searching info ...
"(emacspeak)Speech System" -- speech system
"(cpio)Copy-pass mode" -- copy files between filesystems
"(tar)Basic tar" -- create, complementary notes
"(tar)problems with exclude" -- exclude, potential problems with
"(tar)Basic tar" -- extract, complementary notes
"(tar)Incremental Dumps" -- extract, using with --listed-incremental
"(tar)Option Summary" -- incremental, summary
"(tar)Incremental Dumps" -- incremental, using with --list
"(tar)Incremental Dumps" -- list, using with --incremental
...

Screencast: Tabbing of everything in KDE

I just discovered tabbing of everything in KDE:

(download [414])

Created with recordmydesktop [415], cut with kdenlive [416], encoded to ogg theora with ffmpeg2theora [417] (encoding command [418]).

Music: Beat into Submission [419] on Public Domain [420] by Tryad.

To embed the video on your own site you can simply use:

<video 
src="http://draketo.de/files/screencast-tabbing-everywhere-kde.ogv"
controls=controls>
</video>

If you do so, please provide a backlink here.

License: cc by-sa [421], because that’s the license of the song. If you omit the audio, you can also use one of my usual free licenses [422] (or all of them, including the GPL). Here’s the raw recording [423] (=video source).

¹: Feel free to upload the video to youtube or similar. I license my stuff under free licenses to make it easy for everyone to use, change and spread them.

²: Others have shown this before, but I don’t mind that. I just love the feature, so I want to show it :)

³: The command wheel I use for calling programs is the pyRad [424].

Simple daemon with start-stop-daemon and runit

echo '#!/bin/sh' > whiledaemon.sh
echo 'while true; do true; done' >> whiledaemon.sh
chmod +x whiledaemon.sh

start-stop-daemon --pidfile whiledaemon.pid \
--make-pidfile --background ./whiledaemon.sh

top | grep whiledaemon.sh

start-stop-daemon --pidfile whiledaemon.pid \
--stop ./whiledaemon.sh

echo '#!/usr/bin/env python\nfor i in range(100): a = i*i' >/tmp/foo.py
chmod +x /tmp/foo.py

mkdir -p ~/.local/run/runit_services/python
ln -sf /tmp/foo.py ~/.local/run/runit_services/python/run

runsvdir ~/.local/run/runit_services

# stop the running daemon
SVDIR=~/.local/run/runit_services/ sv stop python
# start the service (it shows as `run` in top)
SVDIR=~/.local/run/runit_services/ sv start python

echo '#!/usr/bin/env python\nfor i in range(100): a = i*i' >/tmp/foo.py &&
    chmod +x /tmp/foo.py &&
    mkdir -p /run/arne_service/python &&
    printf '#!/bin/sh\nexec /tmp/foo.py' >/run/arne_service/python/run &&
    chmod +x /run/arne_service/python/run &&
    chown -R arne /run/arne_service &&
    su - arne -c 'runsvdir /run/arne_service'

echo '#!/usr/bin/env python\nfor i in range(100): a = i*i' >/tmp/foo.py && 
    chmod +x /tmp/foo.py &&
    mkdir -p /run/arne_service/python &&
    ln -s /tmp/foo.py /run/arne_service/python/run &&
    chown -R arne /run/arne_service &&
    su - arne -c 'runsvdir /run/arne_service'

Simple positive trust scheme with threshholds

Update: I nowadays think that voting down is useful, but only for protection against spam and intentional disruption of communication. Essentially a distributed function to report spam.

I don’t see a reason for negative reputation schemes — voting down is in my view a flawed concept.

The rest of this article is written for freetalk inside freenet [429], and also posted there with my nonanonymous ID.

That just allows for community censorship, which I see as incompatible with the goals of freenet.

Would it be possible to change that to use only positive votes and a threshhold?

• If I like what some people write, I give them positive votes.
• If I get too much spam, I increase the threshhold for all people.
• Effective positive votes get added. It suffices that some people I trust also trust someone else and I’ll see the messages.
• Effective trust is my trust (0..1) · the trust of the next in the chain (0..1) · …

Usecase:

• Zwister trusts Alice and Bob.
• Alice trusts Lilith.
• Bob hates Lilith.

In the current scheme (as I understand it), zwister wouldn’t see posts from Lilith.

In a pure positive scheme, zwister would see the posts. If zwister wants to avoid seeing the posts from Lilith, he has to untrust Alice or ask Alice to untrust Lilith. Add to that a personal (and not propagating) blocking option which allows me to “never see anything from Lilith again”.

Bob should not be able to interfere with me seeing the messages from Lilith, when Alice trusts Lilith.

If zwisters trust for Alice (0..1) multiplied with Alices trust for Lilith (0..1) is lower than zwisters threshhold, zwister doesn’t see the messages.

PS: somehow adapted from Credence [430], which would have brought community spam control to Gnutella, if Limewire had adopted it.

PPS: And adaption for news voting: You give positive votes on news which show up. Negative votes assign a private threshhold to the author of the news, so you then only see news from that author which enough people vote for.

Simple steps to attach the GNU Public License (GPL) to your project

Here's the simple steps to attach a GPL license to your source files (written after requests by DiggClone [431] and Bandnet [432]):

For your own project, just add the following text-notice to the header/first section of each of your source-files, commented out in whatever way your language uses:

----------------following is the notice-----------------
/*
* Your Project Name - -you slogan-
* Copyright (C) 2007 - 2007 Your Name
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/
----------------------------------------------
the "2007 - 2007" needs to be adjusted to "year when you gave it the license in the first place" - "current year".

Then put the file gpl.txt into the source-folder or a docs folder: http://www.gnu.org/licenses/gpl.txt [433]

If you are developing together with other people, you need their permission to put the project under the GPL.

------

Just for additional Info, I found this license comparision paper by sun: http://mediacast.sun.com/share/webmink/SunLicensingWhitePaper042006.pdf [434]

And comments to it: http://blogs.sun.com/webmink/entry/open_source_licensing_paper#comments [435]

It does look nice, but it misses one point:

GPL is trust: Contributors can trust, that their contributions will keep helping the community, and that the software they contribute to will keep being accessible for the community.

(That's why I decided some years ago to only support GPL projects. My contributions to one semi-closed project got lost, because the project wasn't free and the developer just decided not to offer them anymore, and I could only watch hundreds of hours of work disappear, and that hurt.)

Best wishes,
Arne
PS: If anything's missing, please write a comment!

Some Python Programs of mine

heavily outdated page. See bitbucket.org/ArneBab [436] for many more projects…

Hi,

I created some projects with pyglet and some tools to facilitate 2D
game development (for me), and I though you might be interested.

• babglet: basic usage of pyglet for 2D games with optional collision
  detection and avoidance.
• blob_swarm: a swarm of blobs with emerging swarm behaviour through only pair relations.
• blob_battle: a duel-style battle between two blobs (basic graphics,
  control and movement done)
• fuzzy_collisions: 2 groups of blobs. One can be controlled. When two
  blobs collide, they move away a (random) bit to avoid the collision.

They are avaible from the rpg-1d6 project on sourceforge:
-> https://sf.net/projects/rpg-1d6/ [437]

The download can be found at the sf.net download page:
-> https://sourceforge.net/project/showfiles.php?group_id=199744 [438]

Strengths and weaknesses of Python

a reply I wrote on quora [439].

Python is easy to learn and low ceremony. Both are pretty hard targets to hit. It also has great libraries for scientific work, for system scripting and for web development — and for most everything else. And it is pragmatic in a sense: It gets stuff done. And in a way which others can typically understand easily. Which is an even harder target to hit, especially with low ceremony languages. If you look for reasons, import this aka PEP 20 -- The Zen of Python [440] is a good start.

Python has rightfully been called “Pseudocode which actually runs”. There’s often no need for pseudocode if you can show some Python.

However it has its weaknesses. Many here already talked about performance. I won’t go there, because you can fix most of that with cython, pypy and time (as the javascript engines in Browsers show which often get 50% of the speed of optimized C). What irks me are some limitations in its syntax which I begun to hit more and more about two years ago.

List comprehensions make actual code more complicated than simple examples, because you have kind of a dual syntax to it. And there is some ceremony in tools which were added later. For example this is the template I nowadays use to start a Python project: a minimal Python script [441] — this could be part of the language so that I would not even need to put it into the script. But this is not how history works: It cannot break backwards compatibility (a fate which hits all useful and widespread programming languages). Also things like having to spell out the underscore names feel more and more strange to me. Therefore I started into Guile Scheme to see how different programming could be if I shed the constraints of Python. You can read my journey in py2guile: Going from Python to Guile Scheme - a natural progression [145] (a free ebook).

Also see my other Python-articles [442] on this site.

Surprising behaviour of Fortran (90/95)

program base60
  ! first step: Base60 encode. 
  ! reference: http://faruk.akgul.org/blog/tantek-celiks-newbase60-in-python-and-java/
  ! 5000 should be 1PL
  implicit none
  <<declare-function-type-program>>
  <<function-test-calls>>
end program base60

<<declare-function-type-function>>
  implicit none
  !!! preparation
  <<unchanged-argument>>
  <<parameter>>
  ! work variables
  integer :: n = 0
  integer :: remainder = 0
  ! result
  <<variable-declare-init>>
  ! actual algorithm
  if (number == 0) then
     <<return>>
  end if
  ! calculate the base60 string
  <<variable-reset>>
  n = number ! the input argument: that should be safe to use.
  ! catch number = 0
  do while(n > 0)
     remainder = mod(n, 60)
     n = n/60
     <<indizes-start-at-1>>
     ! write(*,*) number, remainder, n
  end do
<<return-end>>

write(*,*) 0, trim(numtosxg(0))
write(*,*) 100000, trim(numtosxg(100000))
write(*,*) 1, trim(numtosxg(1))
write(*,*) 2, trim(numtosxg(2))
write(*,*) 60, trim(numtosxg(60))
write(*,*) 59, trim(numtosxg(59))

! I have to declare the return type of the function in the main program, too.
character(len=1000) :: numtosxg

character(len=1000) function numtosxg( number )

function numtosxg (number)
  character(len=1000) :: numtosxg
end function numtosxg

character(len=1000) :: res = ""

res = " " ! I have to explicitely set res to " ", otherwise it
          ! accumulates the prior results!

program accumulate
  implicit none
  integer :: acc
  write(*,*) acc(), acc(), acc() ! prints 1 2 3
end program accumulate

integer function acc()
  implicit none
  integer :: ac = 0
  ac = ac + 1
  acc = ac
end function acc

program accumulate
  implicit none
  integer :: acc
  write(*,*) acc(), acc(), acc() ! prints 1 1 1
end program accumulate

integer function acc()
  implicit none
  integer :: ac
  ac = 0
  ac = ac + 1
  acc = ac
end function acc

! constants: marked as parameter: not function parameters, but
! algorithm parameters!
character(len=61), parameter :: base60chars = "0123456789"&
     //"ABCDEFGHJKLMNPQRSTUVWXYZ_abcdefghijkmnopqrstuvwxyz"

! input: ensure that this is purely used as input.
! intent is only useful for function arguments.
integer, intent(in) :: number

numtosxg = "0"
return

  numtosxg = res
end function numtosxg

! note that fortran indizes start at 1, not at 0.
res = base60chars(remainder+1:remainder+1)//trim(res)

base60chars(n+1:n+1)

pythonarray[n:n+1]

program test
  character(len=5) :: res
  write(*,*) res ! undefined. In the last run it gave me null-bytes, but
                 ! that is not guaranteed.
  res = "0"
  write(*,*) res ! 0
  res = trim(res)//"a"
  write(*,*) res ! 0a
  res = res//"a"
  write(*,*) res ! 0a: trailing characters are silently removed.
  ! who else expected to see 0aa?
  write(res, '(a, "a")') trim(res) ! without trim, this gives an error!
                                   ! *happy*
  write(*,*) res
end program test

Tail Call Optimization (TCO), dependency, broken debug builds in C and C++ — and gcc 4.8

TCO: Reducing the algorithmic complexity of recursion.
Debug build: Add overhead to a program to trace errors.
Debug without TCO: Obliterate any possibility of fixing recursion bugs.

“Never develop with optimizations which the debug mode of the compiler of the future maintainer of your code does not use.”° [453]

UPDATE: GCC [454] 4.8 gives us -Og -foptimize-sibling-calls which generates nice-backtraces [455], and I had a few quite embarrassing errors in my C - thanks to AKF for the catch!

1 Intro

Tail Call Optimization (TCO) makes this

def foo(n):
    print(n)
    return foo(n+1)

foo(1)

behave like this

def foo(n):
    print(n)
    return n+1

n = 1
while True:
    n = foo(n)

2 The tests

(define (foo n)
  (display n)
  (newline)
  (foo (1+ n)))

(foo 1)

def foo(n):
    print n
    return foo(n+1)

foo(1)

#include <stdio.h>

int recurse(int n)
{
  printf("%i\n", n);
  return recurse(n+1);
}

int main()
{
  return recurse(1);
}

all : C2 Cg Cg2 C1

# optimized
C2 : tailcallc.c
    g++ -O2 tailcallc.c -o C2

# debug build
Cg : tailcallc.c
    g++ -g tailcallc.c -o Cg

# optimized debug build
Cg2 : tailcallc.c
    g++ -g -O2 tailcallc.c -o Cg2

# only slightly optimized
C1 : tailcallc.c
    g++ -O1 tailcallc.c -o C1

Traceback (most recent call last):
 File "tailcallpython.py", line 6, in <module>
   foo(1)
 File "tailcallpython.py", line 4, in foo
   return foo(n+1)
… repeat about 997 times …
RuntimeError: maximum recursion depth exceeded

g++ -O2 C.c -o C2
./C2

g++ -g C.c -o Cg
./Cg 

7228 KB   [stack]
56 KB [heap]
40 KB /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17
24 KB /lib64/libc-2.15.so
12 KB /home/arne/.emacs.d/private/journal/Cg

352 KB    /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17
252 KB    /lib64/libc-2.15.so
108 KB    /lib64/ld-2.15.so
60 KB /lib64/libm-2.15.so
16 KB /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libgcc_s.so.1

recurse (n=43) at C.c:5
5         printf("%i\n", n);
43
6         return recurse(n+1);

g++ -g -O2 C.c -o Cg2
./Cg2

5         printf("%i\n", n);
printf (__fmt=0x40069c "%i\n") at /usr/include/bits/stdio2.h:105
105       return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
5
6         return recurse(n+1);

g++ -g -O1 -foptimize-sibling-calls C.c -o Cgtco
./Cg 

g++ -g -Og -foptimize-sibling-calls C.c -o Cgtco
./Cgtco 

recurse (n=n@entry=1) at C.c:4
4       {
5         printf("%i\n", n);
1
6         return recurse(n+1);
5         printf("%i\n", n);
2
6         return recurse(n+1);

g++ -O1 C.c -o C1
./C1

Top 5 systemd troubles - a strategic view for distros

systemd [460] is a new way to start a Linux-system with the expressed goal of rethinking all of init. These are my top 5 gripes with it. (»skip the updates«) [461]

Update (2019): I now use GNU Guix [462] with shepherd [463]. That’s one more better option than systemd. In that it joins OpenRC [464] and many others [465].

Update (2016-09-28): Systemd is an exploit kit just waiting to be activated. And once it is active, only those who wrote it will be able to defuse it — and check whether it is defused. And it is starting: How to crash systemd in one tweet? [466] Alternatives? Use OpenRC [467] for system services. That’s simple and fast and full-featured with minimal fuss. Use runit [468] for process supervision of user-services and system-services alike.

Update (2014-12-11): One more deconstruction of the strategies around systemd: systemd: Assumptions, Bullying, Consent [469]. It shows that the attitude which forms the root of the dangers of systemd is even visible in its very source code.

Update (2014-11-19): The Debian General Resolution [470] resulted in “We do not need a general resolution to decide systemd”. The vote page provides detailed results and statistics [471]. Ian Jackson resigned from the Technical Committee [472]: “And, speaking personally, I am exhausted.”

Update (2014-10-16): There is now a vote on a General Resolution [473] in Debian for preserving the ability to switch init systems. It is linked under “Are there better solutions […]?” on the site Shall we fork Debian™? :^| [474].

Update (2014-10-07): Lennart hetzt [475] (german) describes the rhetoric tricks used by Lennart Poettering to make people forget that he is a major part of the communication problems we’re facing at times - and to hide valid technical, practical, pragmatical, political und strategical criticism [476] of Systemd.

Update (2014-09-24): boycott systemd calls for action with 12 reasons against systemd [477]: “We do recognize the need for a new init system in the 21st century, but systemd is not it.”

Update (2014-04-03): And now we have Julian Assange warning about NSA control over Debian [478], Theodore Ts’o, maintainer of ext4, complaining about incomprehensible systemd [479], and Linus Torvalds (you know him, right?) rant against disrupting behavior from systemd developers [480], going as far as refusing to merge anything from the developers in question into Linux. Should I say “I said so”? Maybe not. After all, I came pretty late. Others saw this trend 2 years before I even knew about systemd. Can we really assume that there won’t be intentional disruption [481]? Maybe I should look for solutions. It could be a good idea to start having community-paid developers [482].

Update (2014-02-18): An email to the mailing list of the technical committee of debian summarized the strategic implications of systemd-adoption for Debian and RedHat [483]. It was called conspiracy theory right away, but the gains for RedHat are obvious: RedHat would be dumb not to try this. And only a fool trusts a company. Even the best company has to put money before ethics [484].

Update (2013-11-20): Further [485] reading [486] shows [487] that [488] people [489] have been giving arguments from my list since 2011, and they got answers in the range of “anything short of systemd is dumb”, “this cannot work” (while OpenRC [464] clearly shows that it works well), requests for implementation details without justification [490] and insults [491] and further insults [492]; but the arguments stayed valid for the last 2 years. That does not look like systemd has a friendly community - or is healthy for distributions adopting it. Also an OpenRC developer wrote the best rebuttal of systemd propaganda I read so far: “Alternativlos”: Systemd propaganda [493] (note, though, that I am biased against systemd due to problems I had in the past with udev kernel-dependencies)

• Losing Control: systemd does so many crucial things itself that the developers of distributions lose their control over the init process: If systemd developers decide to change something, the distributions might actually have to fork systemd and keep the fork up-to-date, and this requires rare skills and lots of resources (due to the pace of systemd). See the Gentoo eudev-Project [494] for a case where this had to happen so the distribution could keep providing features its users rely on. Systemd nowadays incorporates udev. Go reason how systemd devs will act.¹ Why losing control is a bad idea: Strategy Letter V: Commodities [495]

• No scripts (as if you can know beforehand all the things the init system will need to do in each distribution). Nowadays any system should be user-extendable to avoid bottlenecks for development. This essentially boils down to providing a scripting language. Using the language which almost every system administrator knows is a very sane choice for that - and means making it possible to use Shell-Scripts to extend the init-system. Scripts mean that the distribution will never be in a position where it is blocked because it absolutely can’t provide a given fringe feature. And as the experiment with paludis in Gentoo shows [496], an implementation in C isn’t magically faster than one in a scripting language and can actually be much slower (just compare paludis to pkgcore), because the execution time of the language only very rarely is the real bottleneck - and you can easily shell out that part to a faster language with negligible time loss,² especially in shell-scripts (pun partially intended). While systemd can be told to run a shell script, this requires a mental context switch and the script cannot tie into all the machinery inside systemd. If there’s a bug in systemd, you need to fix systemd, if you need more than systemd provides out of the box, you need either a script or you have to patch systemd, and otherwise you write in a completely different language (so most people won’t have the skills to go beyond the fences of the ground defined by the systemd developers as proper for users). Why killing scripts is a bad idea: Bloatware and the 80/20 Myth [497]

• Linux-specific³ (are you serious??). This makes the distribution an add-on to the kernel instead of the distribution being a focus point of many different development efforts. This is a second point where distributions become commodities, and as for systemd itself, this is against the interest of the distributions. On the other hand, enabling the use of many different kernels strengthens the Distribution - even if currently only few people are using them. Why being Linux-only is a bad idea for distributions: Strategy Letter V: Commodities [495]

• Requiring an up-to-date kernel. This problem already gives me lots of headaches for my OLPC due to udev (from the same people as systemd… which is one of the reasons why I hope that Gentoo-devs will succeed with eudev [494]), since it is not always easy to go to a newer kernel when you’re on a fringe platform (I’m currently fighting with that). An init system should not require some special kernel version just to boot… Why those hard dependencies are a bad idea: Bloatware and the 80/20 Myth [497] AND Strategy Letter V: Commodities [495]

• Requiring D-Bus. D-Bus was already broken a few times for me, and losing not just some KDE functionality but instead making my system unbootable is unacceptable. It’s bad enough that so much stuff relies on udev.⁴

In my understanding, we need more services which can survive without the others, so the system gets resilient against failures in a given part. As the system gets more and more complex, this constantly gets more important: Less interdependencies, and the services which are crucial to get my system in a debuggable state should be small and simple - and should not require many changes to implement new features.

Having multiple tools to solve the same problem looks like wasted resources, but actually this extends the range of problems which can be solved with our systems and avoids bottlenecks and single points of failure (either tools or communities), so it makes us resilient. Also it encourages standard-formats to minimize the cost of maintaining several systems side-by-side.

You can see how systemd manages to violate all these principles…

This does not mean, that the features provided by systemd are useless. It says that the way they are embedded in systemd with its heavy dependencies is detrimental to a healthy distribution.

Note: I am neither a developer of systemd, nor of upstart, sysvinit or OpenRC. I am just a humble user of distributions, but I can recognize impending horrible fallout when I see it.

References:

• Comparison of init systems in the Gentoo wiki [498].

I’ll finish this with a quote from 30 myths about systemd [499], written by the systemd developers themselves:

We try to get rid of many of the more pointless differences of the various distributions in various areas of the core OS. As part of that we sometimes adopt schemes that were previously used by only one of the distributions and push it to a level where it's the default of systemd, trying to gently push everybody towards the same set of basic configuration.
— Lennart Poettering, main developer of systemd

I could not show much clearer why distributions should be very wary about systemd than Lennart Poettering does here in the post where he tries to refute myths about systemd.

PS: I’m definitely biased against systemd, after having some horrifying experiences with kernel-dependencies in udev. Resilience looks different. And I already modified some init scripts to adjust my systems behavior so it better fits my usecase. Now go and call me part of a fringe group which wants to add “pointless differences” to the system. If you force Gentoo devs to issue a warning in the style of “you MUST activate feature X in your kernel, else your system will become unbootable”, this should be a big red flag to you that you’re doing something wrong. If you do that twice, this is a big red flag to users not to trust your software. And regaining that trust requires reestablishing a long record of solid work. Which I do not see at the moment. Also do read Bloatware and the 80/20 Myth [497] (if you didn’t do that by now): It might be true that 80% of the users only use 20% of the features, but they do not use the same 20%.

Translating a lookup-dictionary to bash: Much simpler than I thought

I wanted to name Transcom Regions [504] in my plots by passing their names to the command-line tool, but I only had their region-number and a lookup dictionary in Python. To avoid tampering with the tool, I needed to translate the dictionary to a bash function, and thanks to the case statement [505] it was much simpler than I had expected.

This is the original dictionary:

#: Names of transcom regions
transcomregionnames = {
    1: "NAM Boreal",
    2: "NAM Temperate",
    3: "South American tropical",
    # and so forth
}

This is how lookup works in Python:

region = 2
name = transcomregionnames[2]

The solution in bash is a simple mechanic translation:

function regionname () {
    number="$1"
    case $number in
        1) echo "NAM Boreal";;
        2) echo "NAM Temperate";;
        3) echo "South American tropical";;
        # and so forth
    esac
}

And the lookup is easier than anything I hoped for:

region=2
name=$(regionname $region)

This is how it looks in my actual code:

for region in {1..22} ; do ./plotstation.py -c /home/arne/sun-work/ct-tccon/ct-tccon-2015-5x7-use-obspack-no-tccon-nc/ -C "GA: in-situ ground and aircraft"  -c /home/arne/sun-work/ct-tccon/ct-tccon-2015-5x7-use-obspack-use-tccon-noassimeu/ -C "TneGA: non-European TCCON and GA" -c /home/arne/sun-work/ct-tccon/ct-tccon-2015-5x7-use-obspack-no-tccon-no-aircraft-doesitbreaktoo/ -C "G: in-situ ground"  --regionfluxtimeseries $region --toaverage 5 --exclude-validation  --colorscheme paulforabp --linewidth 4 --font-size 36 --start 2009-12-03 --stop 2012-12-02  --title "Effect of assimilating non-EU TCCON, $(regionname ${region})"  -o ~/flux-GA-vs-TneGA-vs-G-region-${region}.pdf; done

For your convenience, here’s my entire transcom naming function:

function regionname () {
    number="$1"
    case $number in
        1) echo "NAM Boreal" ;;
        2) echo "NAM Temperate";;
        3) echo "South American tropical";;
        4) echo "South American temperate";;
        5) echo "Northern Africa";;
        6) echo "Southern Africa";;
        7) echo "Eurasian Boreal";;
        8) echo "Eurasian Temperate";;
        9) echo "Tropical Asia";;
        10) echo "Australia";;
        11) echo "Europe";;
        12) echo "North Pacific Temperate";;
        13) echo "West Pacific Tropics";;
        14) echo "East Pacific Tropics";;
        15) echo "South Pacific Temperate";;
        16) echo "Northern Ocean";;
        17) echo "North Atlantic Temperate";;
        18) echo "Atlantic Tropics";;
        19) echo "South Atlantic Temperate";;
        20) echo "Southern Ocean";;
        21) echo "Indian Tropical";;
        22) echo "South Indian Temperate";;
    esac
}

Happy Hacking!

Weltenwald-theme under AGPL (Drupal)

After the last round of polishing, I decided to publish my theme under AGPLv3 [506]. Reason: If you use AGPL code and people access it over a network, you have to offer them the code. Which I hereby do ;)
That’s the only way to make sure that website code stays free.

It’s still for Drupal 5, because I didn’t get around to port it, and it has some ugly hacks, but it should be fully functional.

Just untar it in any Drupal [507] 5 install.

tar xjf weltenwald-theme-2010-08-05_r1.tar.bz2

Maybe I’ll get around to properly package it in the future…

Until then, feel free to do so yourself :)

And should I change the theme without posting a new layout here, just drop me a line and I’ll upload a new version — as required by AGPL. And should you have some problem, or if something should be missing, please drop me a line, too.

No screenshot, because a live version kicks a screenshot any day ;)
(in case it isn’t clear: Weltenwald is the theme I use on this site)

Which language is best, C, C++, Python or Java?

My answer to the question about the best language on Quora [509]. If you continue reading from here, please stick with me to the end. Ready to read to the end? Enjoy the ride!

My current answer is: Scheme [25] ☺ It gives me a large degree of freedom to explore ways to program which were much harder hard to explore in Python, C++ and Java. That’s why I’m currently switching from Python to Scheme.¹

But depending on my current step on the road to improve my skills² and the development group and project, that answer might have been any other language — C, C++, Java, Python, Fortran, R, Ruby, Haskell, Go, Rust, Clojure, ….

Therefore this answer is as subjective as most other answers, because we have no context on your personal situation nor on the people with whom you’ll work and from whom you can learn or the requirements of the next project you want to tackle.

Put another way:

The only correct answer is “it depends”.

The other answers in this thread [510] should help you find the right answer for you.

Why Gnutella scales quite well

You might have read in some (almost ancient) papers, that a network like Gnutella can't scale. So I want to show you, why the current Version of Gnutella does scale, and does it well.

In earlier versions, up to v0.4, Gnutella was a a pure broadcast network. That means, that every search request did reach every participant, so the number of search requests hitting each node was for an optimal network exactly equal to the number of requests, made by nodes who were in the network. And you can see easily why that can't scale.
But that was only true for Gnutella 0.4.

In the current incarnation of Gnutella (Gnutella 0.6), Gnutella is no longer a pure Broadcast network. Instead, only the smallest percentage of the traffic is done via broadcast.

If you want to read about the methods used to realize this, please have a look at the GnuFU guide (english [512], german [513]).

Here I want to limit it to the statement, that the first two hops of a search request are governed via Dynamic Querying, which stops the request as soon as it has enough sources (this stops a search as soon as it gets about 250 results), and that the last two hops are governed via the Query Routing Protocol, which ensures, that a search request reaches only those hosts, which can actually have the file (which is only about 5% of the nodes).

So in todays reality, Gnutella is a quite structured and very flexible network.

To scale it, Ultrapeers can increase their number of connections from their current 32 upwards, which makes Dynamic Querying (DQ) and the Query Routing Protocol (QRP) even more effective.

In the case of DQ most queries for popular files will still provide enough results after the same number of clients have been contacted, so increasing the number of connections won't change the network traffic at all which is caused by the first two steps.

In the case of QRP, queries wil still only reach the hosts, which can have the file, and if Ultrapeers are connected to more nodes at the same time (by increasing the number of connections), it will provide more results for each connection, so DQ will stop even earlier than with fewer connections per Ultrapeer.

So Gnutella is now far from a broadcast model, and the act of increasing the size of the Gnutella Network can even increase its efficiency for popular files.

For rare files, QRP kicks in with full force, and even though DQ will likely check all other nodes for content, QRP will make sure that only those nodes are reached, which can have the content, which might be only 0.1% of the net or even far less.

Here, increasing the number of nodes per Ultrapeer means that nodes with rare files are in effect closer to you than before, so Gnutella also gets more efficient when you increase the network size, when rare file searches are your major concern.

So you can see, that Gnutella has become a network, which scales extremly well for keyword searches, and due to that it can also very efficiently be used to search for metadata and similar concepts.

The only thing which Gnutella can't do well are searches for strings which aren't seperate words (for example file-hashes), because that kills QRP, so they will likely not reach (m)any hosts. For these types of searches, the Gnutella developers work on a DHT (Distributed Hash Table), which will only be used, if the string can't be split into seperate words, and that DHT will most likely be Kademlia, which is also proven to work quite well.

And with that, the only problem which remains in need of fixing is spam, because that inhibits DQ when you do a rare search, but I am sure that the devs will also find a way to stop spamming, and even with spam, Gnutella is quite effective and consumes very little bandwidth, when you are acting as a leaf, and only moderate bandwidth when you are acting as ultrapeer.

Some figures as finishing touch:

• Leaf network traffic: About 1kB/s if you add outgoing and incoming traffic, which is about the seventh part of the speed of a 56k modem.
• Ultrapeer traffic: About 7kB/s, outgoing and incoming added together, which is about one full ISDN line of less than 1/8th of a DSLs outgoing speed.

Have fun with Gnutella!
- ArneBab 08:14, 15. Nov 2006 (CET)

PS: This guide ignores, that requests must travel through intermediate nodes. But since those nodes make up only about 3% of the network and only 3% of those nodes will be reached by a (QRP-routed) rare file request, it seems safe to ignore these 0.1% of the network in the calculations for the sake of making it easier to follow them mentally (QRP takes care of that).

Why Python 3?

At the Institute we use both Python 2 and Python 3. While researching the current differences (Python 3.5, compared to Python 2.7), I found two beautiful articles by Brett Cannon, the current manager of Python, and summarized them for my work group.

The articles:

1. Why Python 3: Why Python3 exists [514]
2. Why use 3: How to pitch Python 3 to Management [515]

The relevant points for us¹ are the following:

1. Why Python 3 was necessary:

   • Python2: string = byte-array.
     • Py3 avoids Encoding-Bugs in Unicode: all Strings are Unicode.
   • Python2: sources in ASCII. β in a comment needed # encoding: utf-8
     • Py3 uses utf-8 in source files by default.
   • Last chance: the cost of the chance increased every year.

2. Why use 3 (relevant for us, e.G. for new projects):

   • int/long -> int
   • Unicode in Code: σ = sqrt(var) # only letters, but i.e. not Σ
   • H.dot(β) -> H @ β
   • chained exceptions: Traceback ... during handling ... Traceback — simplifies debugging
   • print() facilitates structured output²

The effect of these points is much larger than this short text suggests: avoid surprises, avoid awkward workarounds, and easier debugging.

Write programs you can still hack when you feel dumb

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. — Brian Kernighan

In the article Hyperfocus and balance [516], Arc Riley from PySoy [517] talks about trying to get to the Hyperfocus state without endangering his health. Since I have similar needs, I am developing some strategies for that myself (though not for my health, but because my wife and children can’t be expected to let me work 8h without any interruptions in my free time).

Different from Arc, I try to change my programming habits instead of changing myself to fit to the requirements of my habits.¹

Easy times

Let’s begin with Programming while you feel great.

The guideline I learned from writing PnP roleplaying games [518] is to keep the number of things to know at 7 or less at each point (according to Miller, 1956 [519]; though the current best guess of the limitation for average humans is only 4 objects [520]!). For a function of code I would convert that as follows:

1. You need to keep in mind the function you work in (location), and
2. the task it should perform (purpose and effect), and
3. the resources it uses (arguments or global values/class attributes).

Only 4 things left for the code of your function. (three if you use both class attributes/global values and function arguments. Two, if you have complex custom data-structures with peculiar names or access-methods which you have to understand for doing anything. One if you also have to remember the commands of an unfamiliar² editor or VCS tool. See how fast this approaches zero even when starting with 7 things?)

Add an if-switch, for-loop or similar and you have only 3 things left.

You need those for what the function should actually do, so better put further complexities into subfunctions.

Also ensure that each of the things you work with is easy enough. If you get the things you use down to 7 by writing functions with 20 arguments, you don’t win anything. Just the resources you could use in the function will blow your mind when you try to change the function a few months later. This goes for every part of your program: The number of functions, the number of function arguments, the number of variables, the lines of code per function and even the number of hierarchy levels you use to reduce the other things you need to keep in mind at any given time.

Hard times

But if you want to be able to hack that code while you feel dumb (compared to those streaks of genius when you can actually hold the whole structure of your program in your head and forsee every effect of a given change before actually doing it), you need to make sure that you don’t have to take all 7 things into account.

Tune it down for the times when you feel dumb by starting with 5 things.³ After substracting one for the location, for the task and for the resources, you are left with only two things:

Two things for your function. Some Logic and calling stuff are 2 things.

If it is an if-switch, let it be just an if-switch calling other functions.⁴ Yes, it may feel much easier to do it directly here, when you are fully embedded in your code and feel great, but it will bite you when you are down. Which is exactly when you won’t want to be bitten by your own code.

Loose coupling and tight cohesion

Programming is a constant battle against complexity. Stumble from the sweet spot of your program into any direction, and complexity raises its ugly head. But finding the sweet spot requires constant vigilance, as it shifts with the size and structure of your program and your development group.

To find a practical way of achieving this, Django [521]’s concept of loose coupling and tight cohesion [522] (more detailed [523]) helped me most, because it reduces the interdependencies.

The effects of any given change should be contained in the part of the code you work in - and in one type of code.

As web framework, Django seperates the templates, the URI definitions, the program code and the database access from each other. (see how these are already 4 categories, hitting the limit of our mind again?)

For a game on the other hand, you might want to seperate story, game logic, presentation (what you see on the screen) and input/user actions. Also people who write a scenario or level should only have to work in one type of code, neatly confined in one file or a small set of files which reside in the same place.

And for a scientific program, data input, task definition, processing and data output might be separated.

Remember that this separation does not only mean that you put those parts of the code into different files, but that they are loosely coupled:⁵

They only use lean and clearly defined interfaces and don’t need to know much about each other.

Conclusions

This strategy does not only make your program easier to adapt (because the parts you need to change for implementing a given feature are smaller). If you apply it not only to the bigger structure, but to every part of the program, it’s main advantage is that any part of the code can be understood without having to understand other parts.

And you can still understand and hack your code, when your child is sick, your wife is overworked, you slept 3 hours the night before - and can only work for half an hour straight, because it’s evening and you don’t want to be a creep (but this change has to be finished nontheless).

Note that finding a design which accomplishes this is far more complex than it sounds. If people can read your code and say “oh, that’s easy. I can hack that” (and manage to do so), then you did it right.

Designing a simple structure to solve a complex task is far harder than designing a complex structure to solve that task.

And being able to hack your program while you feel dumb (and maybe even hold it in your head [524]) is worth investing some of your genius-time⁶ into your design (and repeating that whenever your code grows too hairy).

PS (7 years later): This only applies to the version of your code that stays in your codebase. During short-term experiments these rules do not apply, because there you still have the newly written code in your head. But take pains to clean it up before it takes on a life of its own. The last point for that is when you realize that you’re no longer sure how it works (then you know that you already missed the point of refactoring, but you can at least save your colleagues and your future self from stumbling even worse than you do at that moment). That way you also always have some leeway in short-term complexity that you can use during future experimentation. Also don’t make your code too simple: If you find that you’re bored while coding or that you spend more time fighting the structures you built than solving the actual problems, you took these principles too far, because you’re no longer getting full benefits from your brain. Well chosen local complexity reduces global complexity and the required work per change.

Writing a commandline tool in Fortran

Here I want to show you how to write a commandline tool in Fortran. Because Fortran is much better than its reputation — most of all in syntax. I needed a long time to understand that — to get over my predjudices — and I hope I can help you save some of that time.¹

This provides a quick-start into Fortran. After finishing it, I suggest having a look at Fortran surprises [308] to avoid stumbling over differences between Fortran and many other languages.

program hello
  use iso_fortran_env
  write (output_unit,*) "Hello World!"
  write (output_unit,*) 'Hello Single Quote!'
end program hello

gfortran -std=gnu -O3 fortran-hello.f90 -o fortran-hello
./fortran-hello

Hello World!
Hello Single Quote!

program cli
  implicit none ! no implicit declaration: all variables must be declared
  character(1000) :: arg

  call get_command_argument(1, arg) ! result is stored in arg, see 
  ! https://gcc.gnu.org/onlinedocs/gfortran/GET_005fCOMMAND_005fARGUMENT.html

  if (len_trim(arg) == 0) then ! no argument given
      write (*,*) "Call me --world!"
  else
      if (trim(arg) == "--world") then
          call get_command_argument(2, arg)
          if (len_trim(arg) == 0) then
              arg = "again!"
          end if
          write (*,*) "Hello ", trim(arg)
          ! trim reduces the fixed-size array to non-blank letters
      end if
  end if
end program

gfortran -std=gnu -O3 fortran-commandline.f90 -o fortran-helloworld
./fortran-helloworld
./fortran-helloworld --world World
./fortran-helloworld --world

Call me --world!
Hello World
Hello again!

module hello
  implicit none
  character(100),parameter :: prefix = "Hello" ! parameters are constants
  public :: parse_args, prefix
contains
  function parse_args() result ( res )
    implicit none
    character(1000) :: res

    call get_command_argument(1, res)  
    if (trim(res) == "--world") then
        call get_command_argument(2, res)
        if (len_trim(res) == 0) then
            res = "again!"
        end if
    end if
  end function parse_args
end module hello

program helloworld
  use hello, only : parse_args, prefix
  implicit none
  character(1000) :: world
  world = parse_args()
  write (*,*) trim(prefix), " ", trim(world)
end program helloworld

gfortran -std=gnu -O3 fortran-modules.f90 -o fortran-modules
./fortran-modules --world World

Hello World

! Thanks to http://infohost.nmt.edu/tcc/help/lang/fortran/time.html
program cheaplooptest
  integer :: i,j,k,s
  integer, parameter :: n=150 ! 50 breaks 32KB L1 cache, 150 breaks 256KB L2 cache
  integer,dimension(n,n,n) :: x, y
  real etime
  real elapsed(2)
  real total1, total2, total3, total4
  y(:,:,:) = 0
  x(:,:,:) = 1
  total1 = etime(elapsed)
  print *, "start time ", total1
  ! first index as outer loop
  do s=1,n
     do i=1,n
        do j=1,n
           y(i,j,:) = y(i,j,:) + x(i,j,:)
        end do
     end do
  end do
  total2 = etime(elapsed)
  print *, "time for outer loop", total2 - total1
  ! first index as inner loop is much cheaper (difference depends on n)
  do s=1,n
     do k=1,n
        do j=1,n
           y(:,j,k) = y(:,j,k) + x(:,j,k)
        end do
     end do
  end do
  total3 = etime(elapsed)
  print *, "time for inner loop", total3-total2
  ! plain copy is slightly faster still
  do s=1,n
     y = y + x
  end do
  total4 = etime(elapsed)
  print *, "time for simple loop", total4-total3

end program cheaplooptest

gfortran -std=gnu -O3 fortran-faster.f90 -o fortran-faster
./fortran-faster

start time    2.33319998E-02
time for outer loop   19.0533314    
time for inner loop  0.799999237    
time for simple loop  0.729999542    

program cleanloop
  integer, parameter :: n=150 ! 50 breaks 32KB L1 cache, 150 breaks 256KB L2 cache
  integer,dimension(n,n,n) :: x, y
  y(:,:,:) = 0
  x(:,:,:) = 1
  y = y + x
end program cleanloop

module base60conv
  implicit none ! if you use this here, the module must come before the program in gfortran
  ! constants: marked as parameter: not function parameters, but
  ! algorithm parameters!
  character(len=61), parameter :: base60chars = "0123456789"&
       //"ABCDEFGHJKLMNPQRSTUVWXYZ_abcdefghijkmnopqrstuvwxyz"
  integer, parameter :: longlong = selected_int_kind(32) ! length up to 32 in base10, int(16)
  integer(longlong), parameter :: sixty = 60
  public :: base60chars, numtosxg, sxgtonum, longlong
  private ! rest is private
contains
  function numtosxg( number ) result ( res )
    implicit none
    !!! preparation
    ! input: ensure that this is purely used as input.
    ! intent is only useful for function arguments.
    integer(longlong), intent(in) :: number
    ! work variables
    integer(longlong) :: n
    integer(longlong) :: remainder
    ! result
    character(len=1000) :: res ! do not initialize variables when
    ! declaring them: That only initializes
    ! at compile time not at every function
    ! call and thus invites nasty errors
    ! which are hard to find.  actual
    ! algorithm
    if (number == 0) then
       res = "0"
       return
    end if
    ! calculate the base60 string

    res = "" ! I have to explicitely set res to "", otherwise it
    ! accumulates the prior results!
    n = number ! the input argument: that should be safe to use.
    ! catch number = 0
    do while(n > 0)
       ! in the first loop, remainder is initialized here.
       remainder = mod(n, sixty)
       n = n/sixty
       ! note that fortran indizes start at 1, not at 0.
       res = base60chars(remainder+1:remainder+1)//trim(res)
       ! write(*,*) number, remainder, n
    end do
    ! numtosxg = res
  end function numtosxg

  function sxgtonum( base60string ) result ( number )
    implicit none
    ! Turn a base60 string into the equivalent integer (number)
    character(len=*), intent(in) :: base60string
    integer :: i ! running index
    integer :: idx, badchar ! found index of char in string
    integer(longlong) :: number
    ! integer,dimension(len_trim(base60string)) :: numbers ! for later openmp
    badchar = verify(base60string, base60chars)
    if (badchar /= 0) then ! one not
       write(*,"(a,i0,a,a)") "# bad char at position ", badchar, ": ", base60string(badchar:badchar)
       stop 1 ! with OS-dependent error code 1
    end if

    number = 0
    do i=1, len_trim(base60string)
       number = number * 60
       idx = index(base60chars, base60string(i:i), .FALSE.) ! not backwards
       number = number + (idx-1)
    end do
    ! sxgtonum = number
  end function sxgtonum

end module base60conv

program base60
  ! first step: Base60 encode. 
  ! reference: http://faruk.akgul.org/blog/tantek-celiks-newbase60-in-python-and-java/
  ! 5000 should be 1PL
  use base60conv
  implicit none

  integer(longlong) :: tests(14) = (/ 5000, 0, 100000, 1, 2, 60, &
       61, 59, 5, 100000000, 256, 65536, 215000, 16777216 /)
  integer :: i, badchar ! index for the for loop
  integer(longlong) :: n ! the current test to run
  integer(longlong) :: number
  ! program arguments
  character(1000) :: arg
  call get_command_argument(1, arg) ! modern fortran 2003!
  if (len_trim(arg) == 0) then ! run tests
     ! I have to declare the return type of the function in the main program, too.
     ! character(len=1000) :: numtosxg
     ! integer :: sxgtonum
     ! test the functions.
     do i=1,size(tests) 
        n = tests(i)
        write(*,"(i12,a,a,i12)") n, " ", trim(numtosxg(n)), sxgtonum(trim(numtosxg(n)))
     end do
  else
     if (trim(arg) == "-r") then
        call get_command_argument(2, arg)
        badchar = verify(arg, " 0123456789")
        if (badchar /= 0) then
           write(*,"(a,i0,a,a)") "# bad char at position ", badchar, ": ", arg(badchar:badchar)
           stop 1 ! with OS-dependent error code 1
        end if
        read (arg, *) number ! read from arg, write to number
        write (*,*) trim(numtosxg(number))
     else
        write (*,*) sxgtonum(arg)
     end if
  end if
end program base60

gfortran -std=gnu -O3 fortran-base60.f90 -o fortran-base60
./fortran-base60 P
./fortran-base60 h
./fortran-base60 D
./fortran-base60 PhD
factor $(./fortran-base60 PhD) # yes, it’s prime! :)
./fortran-base60 -r 85333
./fortran-base60 "!" || echo $?
echo "^ with error code on invalid input :)"

 23
 42
 13
 85333
85333: 85333
 PhD
# bad char at position 1: !
1
^ with error code on invalid input :)

Your browser history can be sniffed with just 64 lines of Python (tested with Firefox 3.5.3)

Update: The basic bug shown here is now fixed in Firefox [542]. Read on to see whether the fix works for you. Keep in mind that there are much stronger attacks than the one shown here. Use private mode to reduce the amount of data your Browser keeps. What’s not there cannot be claimed.

After the example of making-the-web, I was quite intrigued by the ease of sniffing the history via simple CSS tricks.

So I decided to test, how small I get a Python program which can sniff the history via CSS - without requiring any scripting ability on the browser-side.

I first produced fully commented code (see server.py [544]) and then stripped it down to just 64 lines (server-stripped.py [545]), to make it really crystal clear, that making your browser vulnerable to this exploit is a damn bad idea. I hope this will help get Firefox fixed quickly.

If you see http://blubber.blau as found, you're safe. If you don't see any links as found, you're likely to be safe. In any other case, everyone in the web can grab your history - if given enough time (a few minutes) or enough iframes (which check your history in parallel). This doesn't use Javascript.

It currently only checks for the 1000 or so most visited websites and doesn't keep any logs in files (all info is in memory and wiped on every restart), since I don't really want to create a full fledged history ripper but rather show how easy it would be to create one.

Besides: It does not need to be run in an iframe. Any Python-powered site could just run this test as regular part of the site while you browse it (and wonder why your browser has so much to do for a simple site, but since we’re already used to high load due to Javascript, who is going to care?). So don’t feel safe, just because there are no iframes. To feel and be safe, use one of the solutions from What the Internet knows about you [546].

Konqueror seems to be immune: It also (pre-)loads the "visited"-images from not visited links, so every page is seen as visited - which is the only way to avoid spreading my history around on the web and still providing “visited” image-hints in the browser!

Firefox 4.0.1 seems to be immune, too: It does not show any :visited-images, so the server does not get any requests.

So please don't let your browser load anything depending on the :visited state of a link tag! It shouldn't load anything based on internal information, because that always publicizes private information - and you don't know who will read it!

In short: Don't keep repeating Ennesbys Mistake:

• Mistake: [547]

• Effects: [548]

(comic strips not hosted here and not free licensed → copyright: Howard V. Tayler [549])

And to the Firefox developers: Please remove the optimization of only loading required css data based on the visited info! I already said so in a bug report, and since the bug isn't fixed, this is my way to put a bit of weight behind it. Please stop putting your users privacy at risk.

Usage:

• python server.py
  start the server at port 8000. You can now point your browser to http://127.0.0.1:8000 to get sniffed :)

To get more info, just use ./server.py --help.

adapt plainnat bibtex natbib style to only show the url if no doi is available

Since the URL in a bibtex [550] entry is typically just duplicate information when the entry has a DOI, I want to hide it.¹

Here’s how:

diff -r 5b78f551d0a0 plainnatnoturl.bst
--- a/plainnatnoturl.bst    Tue Apr 04 10:45:08 2017 +0200
+++ b/plainnatnoturl.bst    Tue Apr 04 10:52:25 2017 +0200
@@ -1,5 +1,7 @@
-%% File: `plainnat.bst'
-%% A modification of `plain.bst' for use with natbib package 
+%% File: `plainnatnoturl.bst'
+%% A modification of `plain.bst' and `plainnat.bst' for use with natbib package 
+%% 
+%% From /usr/share/texmf-dist/bibtex/bst/natbib/plainnat.bst
 %%
 %% Copyright 1993-2007 Patrick W Daly
 %% Max-Planck-Institut f\"ur Sonnensystemforschung
@@ -285,7 +288,11 @@
 FUNCTION {format.url}
 { url empty$
     { "" }
-    { new.block "URL \url{" url * "}" * }
+    { doi empty$
+      { new.block "URL \url{" url * "}" * }
+      { "" }
+      if$
+    }
   if$
 }