Email transcript of the interview with Jan Rähm of the
German Linux Magazine linuxmagazin.de.

13th August 2007

> ----------------------------------------------------------------------
>
> How long have you been working for the linux-kernel?

That's already a slight misconception I guess. I've never actually _worked_ 
for the linux kernel. I'm actually a medical doctor who specialised in 
Anaesthesia and that is my full time career. Linux kernel was, for a time, a 
passion and a hobby that I was mad about. I've only ever hacked on the linux 
kernel in my spare time for the fun of it, for approximately the last five 
years. I played with other people's code for about a year and was doing my 
own code for the 4 years following that.

> Why do you quit kernel-developement?

There were many reasons I involved myself in kernel development, but primarily 
it was as a hobby and for the fun of it. Initially the code I developed for 
the kernel was very small and low impact. As time passed and I understood the 
inner workings of the kernel and the various subsystems that I was interested 
in, the size of things I was working on became larger and the impact greater. 
Most of the code I developed I did for fun, and except for when I was 
obviously fixing real bugs noticeable in the kernel, the changes I made were 
minor tweaks to improve performance, mainly on the desktop. The very nature 
of desktop performance tweaks is that there is no metric, no benchmark, no 
measurable quantity that can quantify the changes. Qualitatively, and very 
subjectively, the changes are noticeable to a human user, but not in any 
quantitative fashion. As such, I noticed quite early on that it was 
impossible to convince the kernel community that my changes were worth 
pursuing. So, instead of trying to push my changes into the mainline kernel, 
I maintained the code as patches for the -ck patchset. 

As the magnitude of the changes became larger, though, the community that was 
using the -ck patchset, and even some of the kernel developers themselves, 
insisted I work on getting my patches into mainline instead of maintaining 
them as outside patches. So at that point I began trying to merge my more 
obvious performance change patches into mainline. From that moment on, trying 
to get kernel patches into mainline started becoming more like "work" and 
less like "play" to me. I continued to modify the code I had as requested by 
the kernel maitnainers to be suitable for mainline inclusion through 
modifications to the code, to be the appropriate quality and style for 
mainline and so on. However, very little of my unique code was ever included 
because, again, there was still nothing to quantify the improvements the code 
could provide, and the development community is (rightly) suspicious of 
dubious claims without evidence. I was at a loss for how to proceed forward, 
and started writing benchmarks that would try to quantify these changes, and 
these benchmarks were then accused of being too artificial and not 
representing real world workloads which I found both amusing and somewhat 
absurd. Most benchmarks are artificial, and also people had found my code 
useful for real world workloads already so I had no idea what else I could do 
to "prove" their worth. By this time, through a combination of frustration 
and loss of enjoyment, I was losing interest in continuing kernel 
development.

The one thing that closed the door on my interest level in kernel development, 
however, would have to be the response to my last attempt to improve the cpu 
scheduler. Having been dissatisfied with the behaviour of the cpu scheduler 
for some time, I developed an extremely fair, yet interactive behaving cpu 
scheduler. While other kernel developers were initially enthusiastic about 
the idea, there were some very small compromises that needed to be made in 
kernel behaviour, in order to fix some massive problems that were occurring 
elsewhere. The small compromises were truly miniscule and they were ones of 
perception, whereas the problems being fixed were showstoppers preventing the 
linux kernel even being suitable for many applications. Instead of accepting 
that compromise, the kernel maintainers insisted I fix the compromises. 
Fixing those compromises would reintroduce problems and I was adamant that 
they were still the correct way forward. At the same time I became very 
unwell making it difficult to work on the PC. Instead of agreeing with my 
approach and helping me with the code, the current cpu scheduler maintainer 
wrote his own implementation of a fair yet interactive cpu scheduler (CFS). 

Now I actually don't have anything against the alternative cpu scheduler, 
which is being merged into the 2.6.23 kernel. Similar compromises are to be 
expected in CFS by the way. I'm not advocating going with my code instead, or 
forking the kernel or any such idea. It is not the lack of merging my code 
that made me quit. 

People think this was a clash of egos that I was objecting to when I quit. 
Perhaps, but that's not quite how I and many others saw this. On many 
occasions before I have seen on linux kernel someone object to an idea, and 
then instead of changing their mind and supporting the idea, they write their 
own implementation of the same idea. I've even seen Andrew Morton (the 2nd in 
charge maintainer) call such behaviour "poor form". 

People also have claimed that I quit as a "stunt" in order to attract 
attention and that I'll be back in the future, hopefully with people on my 
side in future arguments. That is also wrong. I have no need whatsoever to 
involve myself in kernel development in any way. One nice thing, though, is 
that not long after my interview with another magazine, there has been far 
more useful discussion involving the desktop than ever before because of my 
comments, even if some of it has been misguided.

I involved myself in kernel development for the fun of it, and as a passionate 
hobby, and others found my involvement helped them. Once it stopped being 
fun, and degenerated into an unpleasant work like environment I decided to 
move on to more rewarding things. I am fortunate that in my real career I am 
very happy and enjoy what I do and am respected. The last thing I want to do 
is make my hobby less enjoyable than work. I've moved on from other hobbies 
before, and that's what I'm doing now.

> When and why you had started kernel-developing?

I've been using PCs for 25 years now and have watched the development and 
ultimate degeneration of the desktop PC over that time. The "desktop" as a 
whole (ie not just linux) has more or less failed the user in my eyes for 
many reasons. Since I'm ultimately a normal desktop user I was most 
interested in improving that desktop experience as much as possible. 
Initially I got involved in linux kernel by merging other people's patches 
that I thought would help the desktop. This was the -ck patchset in the 2.4.x 
series kernels.

What happened instead was that I found the new kernel development (in the lead 
up to the 2.6.0 kernel series) quite depressing. The reason was that new cpu 
scheduling (and to a lesser extent I/O scheduling and the virtual memory 
subsystem) would make the kernel perform quite poorly on the desktop. People 
have misquoted me on this fact thinking that the kernel behaviour is key to 
making the desktop perform well. That's not the case. The problem is that if 
the kernel does something _wrong_ for userspace, such as in its scheduling 
decisions, there is absolutely nothing that userspace can do about it. If the 
kernel does not give your audio playback software any cpu time for ages, 
there is no way that audio playback software can keep up and eventually skips 
on the sound, or the GUI environment can't draw your windows moving across 
the screen in time because X is waiting for cpu time, or because everything 
is waiting for a tiny bit of disk I/O once you write a large file to disk, 
the whole desktop dies waiting for the write to finish. These were the things 
happening in the pre 2.6.0 kernel development branch. Nothing you do in 
userspace can fix these problems. After pointing these things out to the 
developers and having seen various failed attempts to fix them, I decided to 
tinker with the code myself, even though I was quite new to C. I managed to 
do a half decent job at curbing many of these problems and so my code 
eventually got included into the pre 2.6.0 kernel. The "interactivity tuning" 
I put in largely improved the audio playback and X GUI smoothness. However 
for a long time after that I was never really happy with the way it tackled 
those and other issues so I started playing with them further to the point 
that I tried rewriting the areas responsible and the -ck patchset was reborn, 
this time with my own code.

> What was your main aim to got to kernel-developing?

There were recurring themes in aspects of the kernel behaviour that were 
obviously affecting desktop behaviour, and as I said, initially I tried to 
tweak the existing areas that were misbehaving there and eventually I tried 
rewriting them from scratch with the desktop clearly in mind, while still 
paying due respect to the fact the linux kernel is a multi-purpose tool and 
had to perform well everywhere.

> How long do you needed to learn "C"?

A difficult question to answer clearly. I don't think I ever really "learnt" 
C, apart from what I needed to know to implement the ideas I had in my head. 
I learnt C by looking at kernel code, and by referring to K&R and receiving 
help from other C experts along the way whenever I needed to. I'm sure that I 
still don't know how to do some very basic things in C that most computer 
science students would have no trouble with simply because I never had any 
need to write them.

> What was your main concern at kernel-development?

The desktop. There are recurring themes that, no matter how many times we 
tackle them, remain a problem on the desktop. As I've said before, the kernel 
is not the _key_ to the desktop, but a kernel misbehaving in the wrong places 
can cause great havoc to desktop behaviour. CPU process scheduling, disk I/O 
and virtual memory usage (which is pretty much most of the inner workings of 
a kernel I guess) can all be done in ways that completely destroy the 
desktop. My focus was always ways to minimise the harm of these areas, and 
changes in these areas in the future will continue to have the potential to 
destroy the desktop.

CPU scheduling is the obvious one, mainly because most aspects of cpu 
scheduling that improve throughput have the opposite effect on latency. While 
latency is important even to a server, the magnitude of what is important on 
a server is 10 to 100 times slower than what is noticeable by a human. Our 
desktop hardware has evolved into one where cpu scheduling decisions still 
affect video and audio playback, and they are not interdependant entities 
(which annoys me greatly by the way). This means that to pay deference to 
human perception we need to be swapping things around much faster than is 
optimal for maximum throughput. 

Disk I/O is a disaster for the desktop. Most other hardware gets faster 
exponentially, yet hard disks are still based on a 50 year old design and are 
only getting faster and fatter at a linear rate. This means they are becoming 
_relatively_ slower and slower. Anything we do that can avoid and minimise 
disk I/O will help. Write a very large file to (one) disk and see what else 
you can do on your desktop while this is going on.

The Virtual Memory subsystem can easily focus too much on storing all the 
wrong things and lead to more disk I/O for file backed data, or swapping. 
Copying an ISO image often may make your running application have a lot less 
memory at its disposal, or worse, swap out.

I spent most of my time tweaking and modifying these subsystems in the hope of 
improving the desktop or at least curtailing any harm the kernel can do in 
these areas. They will continue to be the kernel challenges on the desktop 
for as far as I can see into the future.

> How do you think about the future of linux?

Linux will continue to be one of the most important pieces of software in 
existence driven by, and driving development of, the very little competition 
out there that exists. In server space it has more or less won. In the 
desktop scene, GNU/Linux will continue to have a very important also-running 
position, but never first.

> What do you think, Linux needs to get more successful? Or, is Linux
> already successful these days?

It is already far more successful than it ever should have been. If you're 
asking what GNU/Linux needs to be first on the desktop, that's a different 
question entirely.

> ----------------------------------------------------------------------

> ---------------------------------------------------------------------------
>---
>
> Con Kolivas wrote:
> > [...] I've only ever hacked on the linux
> > kernel in my spare time for the fun of it, for approximately the last
> > five years. I played with other people's code for about a year and was
> > doing my own code for the 4 years following that.
>
> In my opinion kernel-development is a really time-intensive job, also if
> done as a hobby. So how much time do you invested in your former hobby?

Yes it's very time intensive. I probably dedicated about 4-5 hours a day to it 
while I was involved in it, and that doesn't take into account the constant 
checking of email every time I walked past a computer (any computer 
anywhere).
>
> > [...] Most benchmarks are artificial, and also people had found my code
> > useful for real world workloads already so I had no idea what else I
> > could do to "prove" their worth. By this time, through a combination of
> > frustration and loss of enjoyment, I was losing interest in continuing
> > kernel development.
>
> How can I imagine this process of losing the fun in your hobby? Did the
> others bother you? Was the tone rude? Please give us a little view into
> the way of communication and working on the Linux-kernel.

As for your actual question, I think I've given enough information for people 
to deduce why I wasn't enjoying it any more.

The linux kernel development process is so heavily linked to the internet and 
communication these days that most of the communication is done via email. 
Like all things in this world, there are good and bad things about that. 
Clearly email allows anyone from anywhere to be involved without a whole lot
of other restrictions that come with human contact (including having any 
social skills whatsoever). 

For those that do not directly maintain a subsystem and have direct 
repositories that feedback into the mainline kernel (and that's the majority 
of developers) the development process usually involves posting your code, 
comments, feedback etc to the linux kernel mailing list. Depending on the 
type of code, your recognition for past contributions and so on, the code 
either gets directly incorporated into the development arm of the kernel 
(the -mm tree) or you get feedback on the first posting of the patch by 
interested onlookers and then you resubmit the patch after attending to any 
suggested changes and so on. That's all assuming the code is definitely going 
to go in, and that only happens for clear bugfixes, new drivers and so on.

For new ideas the process is less clear. People interested in the particular 
subsystem involved are encouraged to comment on the idea and the code. Often 
this is where the system has the potential to fall down. Once all code issues 
are addressed, there still remains the requirement to prove that the code is 
worthwhile, the idea is worth implementing, and the solution is the best one 
available for that problem and so on. The ability of the person posting the 
code to debate the relative merits of their code and idea can be quite 
difficult, especially if they're not equipped with the skills necessary to 
debate. If a disagreement ensues, the parties involved in the dispute have to 
have the persistence and communication skills to convince the other party or 
the onlookers of the pros of their solution or code. Some people have 
incredible persistence and there are even certain techniques they implement 
without realising they're doing it that make these discussions impossible to 
win. The most frustrating argument is continually replying to any answer with 
further "what if" questions that are fuzzy in interpretation, have no real 
answer, and just lead to endless further discussion. Eventually the other 
party may not have the stamina to continue the discussion and the lack of 
further responses is taken as a "win" by the opposing party.

Email has an even bigger impediment to communication. All public discussion is 
done in English. While most people have a reasonable grasp of English to 
communicate, many do not. The ability to carry out a lengthy debate about 
subtleties when the language fails you can be quite a barrier. Furthermore, 
most people do not realise how much value language places on human 
interaction in the form of facial expression, tonality and stress when 
speaking, pauses, questions in rhetorical form only obvious when spoken and 
so on. Once these ideas are committed to text in email, all the emphasis is 
lost, and has the potential to be completely misunderstood. People do not see 
the emphasis when it's there and often find emotion in the response when it 
is not. It is not uncommon for responses to be misinterpreted as aggression 
and anger to enter the email discussions. The mailing lists are littered with 
examples of this. This happens even for native English speakers, so across 
continents, and across further language barriers it's safe to say that the 
intent will more often be misinterpreted than understood. It's easy to 
say "never read any emotion in email unless explicitly stated", but it's very 
hard to do.

> > [...] At the same time I became very
> > unwell making it difficult to work on the PC. Instead of agreeing with my
> > approach and helping me with the code, the current cpu scheduler
> > maintainer wrote his own implementation of a fair yet interactive cpu
> > scheduler (CFS).
>
> What were the reasons for not taking your code? What's the difference
> between the two schedulers?

Actually that's mostly my choice that my code would never even be considered 
in place of the other scheduler. As I abandoned linux kernel development, the 
only person who knew the code was myself and without a maintainer it would be 
the wrong choice to go with mine. What actually happened is that while I was 
still fixing bugs in my code, the cpu scheduler maintainer himself creates an 
alternative, but obviously very new and immature implementation of the same 
ideas. He very quickly attracted the help of other coders to help him with 
his project in stark contrast to what help I was receiving with my project. 
When you stand alone developing code as an outsider and the actual maintainer 
of the subsystem starts developing an alternative version of the code with 
others helping him, you need to ask yourself why you would even bother 
continuing with your own project, and that's what I did. I vowed to fix all 
the bugs in the existing code to allow the cpu scheduler to stand as a 
reference for comparison. 

As for the actual code difference, CFS is written from scratch without almost 
any code from my own cpu scheduler. The only thing they share in common is an 
effort to maintain strong notions of fairness between cpu proportioning and 
minimising latency of all tasks. Staircase Deadline (or SD, my cpu scheduler) 
works on a principal of never looking at past cpu behaviour of tasks and 
rigidly handing out fixed cpu proportions in a forward looking manner, 
staggered between processes in such a way that latency is minimised between 
processes of any nice levels. Process' sleep time is never considered when 
handing out cpu time as any "history" in the design leads to potential for 
misinterpreting the definition of fair depending on how far back you look 
into the process history, and it also creates the potential for exploits and 
starvation. SD does this with minimal overhead because of the lack of complex 
calculations. CFS takes the other approach, where it strictly accounts all 
cpu time and sleeping of each process and uses an RB tree lookup to determine 
who goes next and for how long; and it unlike SD it does account for sleep 
time. The overhead is greater, but probably only noticeable in 
microbenchmarks I would guess. As for their performance differences... I have 
no idea any more since I stopped developing. CFS continues to receive code 
attention whereas SD does not, so it's natural to assume that at some stage 
(past, present or future) it should surpass SD unless there is something 
fundamentally wrong with the CFS design.

> > People also have claimed that I quit as a "stunt" in order to attract
> > attention and that I'll be back in the future, hopefully with people on
> > my side in future arguments. That is also wrong. I have no need
> > whatsoever to involve myself in kernel development in any way. [...]
>
> Your decision is an absolute final decision? Not a decision in the
> meaning of "Say never no"?

There should be nothing open to interpretation in my answer.

> > [...] One nice thing, though, is
> > that not long after my interview with another magazine, there has been
> > far more useful discussion involving the desktop than ever before because
> > of my comments, even if some of it has been misguided.
>
> So your leaving lead into a new way of thinking about linux on the
> desktop? Is it you legacy?

Wow that makes it sound so much more important than it actually is. I'd love 
to believe that I created a legacy but that's overstating my importance 
greatly. I'll fade into obscurity like many other people so there is no 
special legacy in my opinion. Hopefully, though, I've inspired people close 
to and within the kernel community to care more about desktop issues. I did 
note with some amusement that Linus recently started recommending changes to 
default filesystem settings that I've been recommending for the desktop for 
ages (writeback journalling and noatime) so I guess Linus is being true to 
his word in that he says he cares about the desktop.

> > [...] I am fortunate that in my real career I am
> > very happy and enjoy what I do and am respected. The last thing I want to
> > do is make my hobby less enjoyable than work. I've moved on from other
> > hobbies before, and that's what I'm doing now.
>
> You mean, there was no respect for your work about the kernel?

Perhaps you're reading into the comment too much. I'll leave that open to 
interpretation.

> > [...] In the
> > desktop scene, GNU/Linux will continue to have a very important
> > also-running position, but never first.
>
> Why do think, that GNU/Linux will never be first one on the desktop?

see below...
>
> > It is already far more successful than it ever should have been. [...]
>
> Why? Or better, why should it not have been successful?

Now I'm not saying that Linux isn't good enough to be successful. It most 
definitely is.  My ideal view of the PC world is one where there is lots of 
competition from many manufacturers and software houses on many fronts 
driving innovative hardware and software solutions. The world we currently 
live in looks nothing like that. For whatever reasons, Microsoft has created 
a pc world standardised on one pc architecture, with one type of hardware, 
one standard set of hardware attachments, and all of it has to work with 
their software. Ironically, since the choice was Microsoft and Microsoft, it 
means that if users and developers were looking for a more flexible, more 
adaptable,freer (etc) alternative, they were driven to Linux being the 
defacto free standard. Linux was primarily developed for that same hardware 
platform and ported to everything else. So in essence, Microsoft is 
responsible for Linux's success. 

>
> > [...] If you're
> > asking what GNU/Linux needs to be first on the desktop, that's a
> > different question entirely.
>
> Can you answer this so quite different question? :-)

I usually try and avoid this question because it has become one of the most 
discussed points on the internet with every analyst, every news site, every 
journalist and every expert giving their opinion on it. Frankly, I got bored 
of this discussion years ago. GNU/Linux is a viable high quality alternative 
desktop operating system right now, and has been for a few years. I've used 
nothing but it on the desktop for the last 8 years (kde on debian if you must 
ask), and apart from some hiccups and limitations along the way, I've grown 
quite accustomed to it and find it quite difficult to use windows now. Which 
is exactly the opposite of what windows users feel.

So if we look at the discussion points on why Linux hasn't taken over the 
desktop, let's see what I can think of as the cited reasons off the top of my 
head.

There is  no standard distribution. Binaries from one Linux don't work on 
another. Package management varies from distribution to distribution. Package 
management breaks easily. Package installation is not easy for the average 
user. ABI isn't stable. API isn't stable. Development is too fast. 
Development is too slow. There is too much change. There isn't enough 
innovation. Releases come out too often. It tries too hard to be like 
windows. It doesn't try hard enough to be like windows. There are too many 
desktops to choose from, and not enough standardised effort on one. There 
aren't enough applications for it. There are too many varied applications 
doing the same thing. There are never stable releases, only repeated 
rewrites. Windows applications don't work on it. It doesn't blend well with 
Microsoft networks and exchange. MS office documents can't be read/write well 
by the Linux alternatives. All the applications have different names to the 
windows ones. They all look different. People are familiar with their current 
software and don't want to relearn everything. All this software I own won't 
work on Linux. There aren't any games for it. Hardware doesn't work well with 
it. The hardware vendors won't write drivers for it. Specs from hardware 
manufacturers are closed so we can't develop drivers for it.  All the big 
software houses don't release versions for linux. Your place of employment or 
schooling insists on using MS office, .doc format or windows or whatever. 
Your ISP provides no support for linux. There is no accountability. No 
guarantees. No software support, no vendor support. No expertise. Proprietary 
formats cant be read. Proprietary formats can't be played back. The legality 
of the software is questionable. Are software patents infringed upon. Will I 
get sued for doing that. How can free software be good, it is just made by 
hippies... 

Actually I can think of heaps more but I think that's enough for my point. 
Some of these are real barriers to adoption. Some are only minor barriers. 
Some are only perceived barriers. And some are just plain FUD (Fear, 
Uncertainty and Doubt). But with a list this long (it's much longer 
actually), why bother trying to find the one reason?

> ---------------------------------------------------------------------------