man7.org

TLPI third print run now available

2012-04-09T10:38:00.000+02:00

I'm happy to announce that the third print run of TLPI is now printed, and should be available for sale shortly. The third print run incorporates these 131 errata shown on the errata page. (If that seems like a lot of errata, take a look at this FAQ.)

Kernel capability usage statistics

2012-03-13T07:55:00.000+01:00

The idea of capabilities is to break the power of root (user ID 0) into independently assigned pieces governing specific privileged operations. Implicit in that model is that the set of privileged operations governed by each capability should be small (otherwise, why break the power of root into pieces at all?). However, that implication hasn't turned out to be true in practice.

Table 1 shows some statistics on the use of the 36 currently existing capabilities in the C files of the Linux 3.2 kernel source code. The "#uses" column is the number of uses of the capability across all source files; the "#files" column is the number of distinct source files where the capability is used.

Table 1: Breakdown of Linux capability uses in Linux 3.2

Capability	#uses	#files
CAP_AUDIT_CONTROL	2	2
CAP_AUDIT_WRITE	1	1
CAP_CHOWN	4	2
CAP_DAC_OVERRIDE	2	1
CAP_DAC_READ_SEARCH	4	2
CAP_FOWNER	9	8
CAP_FSETID	8	6
CAP_IPC_LOCK	13	8
CAP_IPC_OWNER	1	1
CAP_KILL	2	2
CAP_LEASE	1	1
CAP_LINUX_IMMUTABLE	13	13
CAP_MAC_ADMIN	25	5
CAP_MAC_OVERRIDE	6	2
CAP_MKNOD	3	3
CAP_NET_ADMIN	395	182
CAP_NET_BIND_SERVICE	13	10
CAP_NET_BROADCAST	0	0
CAP_NET_RAW	18	11
CAP_SETFCAP	3	2
CAP_SETGID	10	6
CAP_SETPCAP	2	2
CAP_SETUID	8	4
CAP_SYS_ADMIN	451	229
CAP_SYS_BOOT	2	2
CAP_SYS_CHROOT	1	1
CAP_SYSLOG	2	2
CAP_SYS_MODULE	4	3
CAP_SYS_NICE	14	8
CAP_SYS_PACCT	1	1
CAP_SYS_PTRACE	11	6
CAP_SYS_RAWIO	67	42
CAP_SYS_RESOURCE	36	24
CAP_SYS_TIME	22	13
CAP_SYS_TTY_CONFIG	11	4
CAP_WAKE_ALARM	2	1
Total	1167	610

What is of course notable is the extremely heavy use of two capabilities: CAP_SYS_ADMIN and CAP_NET_ADMIN. Together, these two capabilities account for more than 70% of all uses of capabilities! The uses of CAP_NET_ADMIN are limited to the drivers/ (mainly drivers/net/) and net/ directories. On the other hand, the uses of CAP_SYS_ADMIN, which accounts for nearly 39% of all capability uses, are spread widely across the kernel source tree.

One might wonder whether either of these two capabilities is overrepresented because of duplications in the drivers/ or the arch/ trees. (In particular, CAP_SYS_ADMIN is used for similar administrative functions on a lot of device drivers.) However, even if we strip drivers/ and architectures other than x86 from the measurements, the overall picture doesn't change greatly, as shown in Table 2.

Table 2: Breakdown of Linux capability uses in Linux 3.2, excluding drivers and architectures other than x86

Capability	#uses	#files
CAP_AUDIT_CONTROL	2	2
CAP_AUDIT_WRITE	1	1
CAP_CHOWN	4	2
CAP_DAC_OVERRIDE	2	1
CAP_DAC_READ_SEARCH	4	2
CAP_FOWNER	9	8
CAP_FSETID	8	6
CAP_IPC_LOCK	11	6
CAP_IPC_OWNER	1	1
CAP_KILL	1	1
CAP_LEASE	1	1
CAP_LINUX_IMMUTABLE	13	13
CAP_MAC_ADMIN	25	5
CAP_MAC_OVERRIDE	6	2
CAP_MKNOD	3	3
CAP_NET_ADMIN	167	73
CAP_NET_BIND_SERVICE	12	9
CAP_NET_BROADCAST	0	0
CAP_NET_RAW	18	11
CAP_SETFCAP	3	2
CAP_SETGID	9	5
CAP_SETPCAP	2	2
CAP_SETUID	8	4
CAP_SYS_ADMIN	167	80
CAP_SYS_BOOT	2	2
CAP_SYS_CHROOT	1	1
CAP_SYSLOG	2	2
CAP_SYS_MODULE	4	3
CAP_SYS_NICE	12	6
CAP_SYS_PACCT	1	1
CAP_SYS_PTRACE	10	5
CAP_SYS_RAWIO	10	9
CAP_SYS_RESOURCE	26	18
CAP_SYS_TIME	4	2
CAP_SYS_TTY_CONFIG	1	1
CAP_WAKE_ALARM	2	1
Total	552	291

CAP_SYS_ADMIN still accounts for 167 of 552 uses of capabilities--about 30%, and, by chance, usage of CAP_NET_ADMIN is the same.

It turns out that the overall picture hasn't changed that much since capabilities were first introduced with Linux 2.2 (Jan 1999). Then, there were 27 capabilities, broken down as shown in Table 3.

Table 3: Breakdown of Linux capability uses in Linux 2.2

Capability	#uses	#files
CAP_CHOWN	2	1
CAP_DAC_OVERRIDE	5	5
CAP_DAC_READ_SEARCH	4	4
CAP_FOWNER	7	5
CAP_FSETID	3	2
CAP_IPC_LOCK	4	2
CAP_IPC_OWNER	1	1
CAP_KILL	0	0
CAP_LINUX_IMMUTABLE	2	2
CAP_NET_ADMIN	75	32
CAP_NET_BIND_SERVICE	3	3
CAP_NET_BROADCAST	0	0
CAP_NET_RAW	8	6
CAP_SETGID	7	2
CAP_SETPCAP	2	2
CAP_SETUID	7	3
CAP_SYS_ADMIN	127	69
CAP_SYS_BOOT	1	1
CAP_SYS_CHROOT	1	1
CAP_SYS_MODULE	4	2
CAP_SYS_NICE	5	2
CAP_SYS_PACCT	1	1
CAP_SYS_PTRACE	9	9
CAP_SYS_RAWIO	2	1
CAP_SYS_RESOURCE	10	8
CAP_SYS_TIME	7	4
CAP_SYS_TTY_CONFIG	1	1
Total	298	169

In Linux 2.2, CAP_SYS_ADMIN accounted for 42% of the uses of capabilities, and CAP_NET_ADMIN accounted for 25%.

Table 4 repeats the earlier exercise of excluding drivers/ and architectures other than i386 (as the Intel arch/ directory was then named) from the Linux 2.2 data. In this case, an interesting difference emerges.

Table 4: Breakdown of Linux capability uses in Linux 2.2, excluding drivers and architectures other than i386

Capability	#uses	#files
CAP_CHOWN	2	1
CAP_DAC_OVERRIDE	5	5
CAP_DAC_READ_SEARCH	4	4
CAP_FOWNER	7	5
CAP_FSETID	3	2
CAP_IPC_LOCK	4	2
CAP_IPC_OWNER	1	1
CAP_KILL	0	0
CAP_LINUX_IMMUTABLE	2	2
CAP_NET_ADMIN	44	23
CAP_NET_BIND_SERVICE	3	3
CAP_NET_BROADCAST	0	0
CAP_NET_RAW	8	6
CAP_SETGID	7	2
CAP_SETPCAP	2	2
CAP_SETUID	7	3
CAP_SYS_ADMIN	23	17
CAP_SYS_BOOT	1	1
CAP_SYS_CHROOT	1	1
CAP_SYS_MODULE	4	2
CAP_SYS_NICE	5	2
CAP_SYS_PACCT	1	1
CAP_SYS_PTRACE	2	2
CAP_SYS_RAWIO	2	1
CAP_SYS_RESOURCE	5	4
CAP_SYS_TIME	3	1
CAP_SYS_TTY_CONFIG	1	1
Total	147	94

The overall picture differs in a way that I suspect is significant: just under 16% (23/147) of the uses of capabilities are CAP_SYS_ADMIN. (As we saw in Table 2, by Linux 3.2, this figure had grown to 30% (167/552).) This difference suggests to me that as a series of kernel developers was faced with the question: "What capability should I use to govern my new privileged kernel feature?", the answer was often something like "I don't know; maybe CAP_SYS_ADMIN?". (That certainly fits with a few anecdotal cases I've encountered while discussing things with kernel developers as I wrote man pages for new kernel features.)

The script (count_kernel_cap_uses.sh) used to generate the data for these statistics can be found here. The first and third tables above are based on analysis of the "p2" output files produced by the script. The second and fourth tables are based on analysis of the "p4" output files.

Look hard, and you can see TLPI

2012-02-22T03:07:00.003+01:00

This is fun... I'm fairly certain that the guess by this redditor about what you can see sitting behind the laptop in this photo (taken by John Snyder, and used in this 21 Feb 2012 Wired article) is correct. Probably, it's the copy I gave Linus last year. It's nice to know he held on to it, though I'm not so sure there's so much in the book that he doesn't already know.

Traditional Chinese translation of TLPI

2012-02-20T10:56:00.000+01:00

The Taiwanese publisher GOTOP has contracted the rights to do a Traditional Chinese publication of TLPI.

There are now four translations of TLPI in progress. It looks like the first of the translations that will appear will be the Korean translation, sometime around the middle of this year.

Web site updates

2012-02-04T20:29:00.002+01:00

I've rolled out a few updates to the man7.org web site, including a redesigned main page for TLPI, and the addition of several errata to the errata page. Special thanks for a long list of error reports to Junjiro Okajima, who's working on the Japanese translation of TLPI.

Third print run (and request for bug reports)

2012-01-16T09:09:00.001+01:00

Sales of The Linux Programming Interface have continued well enough that the publisher will soon start preparing the third print run. That print run will incorporate all of the outstanding errata.

If you've been reading TLPI and noticed any errors or typos, now would be a good time to report them, so that fixes can be included in the next printing.

Nice discount on No Starch books

2011-12-03T07:57:00.000+01:00

No Starch Press is doing a brief pre-Christmas sale -- 40% off all paper books, and the ebook is included for free. That's cheaper than the normal discount offered for TLPI, and it applies to all No Starch books. But I only found out about this sale a little late: according to http://nostarch.com/newsletters/2011_holiday.htm, the sale finishes at midnight on 3 Dec. I presume that means midnight in California (UTC-8), about 24 hours from now. So, you might want to take a quick look at the full catalog at No Starch.

PS The publisher's costs for international shipping from the US is pretty hefty, but with the free ebooks, and if you're ordering more than one book, the prices might still beat the online bookstores even if you're shipping outside the US.

Slides from OpenFest presentation

2011-11-09T19:04:00.001+01:00

The slides from my OpenFest presentation, Why kernel space sucks, can be found here.

Speaking tomorrow at OpenFest

2011-11-04T15:04:00.000+01:00

The organizers very kindly invited me to speak at OpenFest in Sofia, Bulgaria. My talk tomorrow is entitled Why Kernel Space Sucks.

That title is of course a reference to the highly amusing presentation of a few years ago by kernel hacker Dave Jones entitled Why Userspace Sucks (MagicPoint presentation). (For the PDF of Dave's complete paper presented at Linux Symposium 2006, look here; LWN.net has a nice tl;dr summary.)

Dave's presentation was all about the ways in which various userspace systems and applications kill performance by wasting system resources on pointless tasks. I'm not contradicting anything that Dave says, but it seems at least fair to point out that there are places where kernel space sucks too. My talk is about one of those places with special interest to me--the kernel-userspace programming interface--a place where kernel developers have inflicted a steady stream of small train wrecks (to borrow Dave's term) on userspace.

LinuxCon Europe

2011-10-20T08:56:00.003+02:00

I'll be at LinuxCon Europe, 26-28 October, in Prague, Czech Republic. (In fact, I arrive already on Monday, 24 October.) I'll have a few copies of The Linux Programming Interface with me for sale. If you want to buy a copy (signed even, if you want!), drop me a mail (mtk AT man7.org).

Linux Plumbers Conference

2011-08-14T08:54:00.004+02:00

I'm going to be at Linux Plumbers Conference, 7-9 September, in Santa Rosa, California. I'm looking forward to connect up with folk, and spend a couple of days in the Bay Area. I may have a limited number of copies of The Linux Programming Interface with me for sale. If you want to buy a copy directly from me (signed even, if you want!), please contact me by email by the end of this month (mtk AT man7.org).

Updated API Changes page

2011-05-27T07:53:00.003+02:00

With the release of Linux 2.6.39, and the opening of the merge window for Linux ~~2.8.0 (or 2.6.40, or 3.0, or whatever~~) 3.0, I've updated the API changes page to include recent and upcoming userspace API changes in the Linux kernel. (This page covers changes since Linux 2.6.36, which was the next kernel released after the book went to press.)

Updated 2011-05-30: fixed kernel version number

No Starch Press discount offer further extended

2011-05-18T18:42:00.000+02:00

Because TLPI was temporarily out of stock while the second print run was being completed, No Starch Press has further extended their earlier discount offer, allowing you to order the paper version of TLPI from them at a 30% discount (i.e., a total price of US$70 plus shipping), and receive the ebook free (look here for details on the ebook release). The offer is now extended to 30 June 2011.

To obtain the discount (and ebook), you must buy TLPI direct from the publisher's web site, and quote the coupon code Mamaku (what's that?). (No Starch Press is located in San Francisco, so international shipping rates will apply for readers outside North America.)

No Starch Press has the second print run in stock, so they can ship immediately.

Second print run now available

2011-05-17T21:33:00.000+02:00

The second print run of The Linux Programming Interface is now available. Sellers such as Amazon are now restocked.

Reworked errata page

2011-04-27T21:56:00.002+02:00

I've reworked the errata page to use color and fonts to classify the errata into three broad types and two levels of "severity".

One reason that I've done this is because there are by now quite a few errata. Another reason is because I earlier received a couple of inquiries along the lines "Because there are a lot of errata, should I wait for the second printing of the book before I buy a copy?" My response to this was "probably not", for reasons that I give below.

As of today, there are 104 errata posted on the errata page. One reason that there are a lot of errata is because there is a lot of book: 1500 pages. Another reason is that I'm very conscientious about fixing and documenting all errors that I find or that are reported to me (by contrast, a lot of books don't even maintain an errata list).

However, the observation that there are a lot of errata also needs to be tempered with some analysis. The current set of errata can be broken down (see the coding on the errata page) as follows:

5 fixes to significant code problems.
1 significant fix to an explanation in the text.
1 significant fix to a technical detail explained in the text.
10 minor code fixes. The corresponding code problem is minor and its solution is normally obvious to an attentive reader with a good understanding of C. (The most common culprit here was small code snippets that I wrote directly into the text--complete with syntax errors--rather than extracting directly from source files.)
24 minor technical fixes. These are typically obvious fixes to minor technical errors or imprecisions in the text. (In many cases, these issues probably wouldn't have impacted the reader's understanding of the topic.)
25 minor clarifications. These are improvements or fixes to minor details in the text. In many cases, the fix could simply have been omitted (the problem would have been unlikely to trouble most readers), but I took the opportunity anyway to improve the text because it was easily possible to do so.
38 typo fixes. Spelling and grammar corrections and obvious fixes to wording errors.

For most readers, probably only the first three categories of errata will matter--and possibly a few of those in the fourth category. In other words, the number of significant errata fixed between the two print runs is actually quite small, which is why I counseled those wondering whether to wait for the second print run that they probably didn't need to.

PS Meanwhile, we look to still be on target to have the second print run ready and in shops at the start of May.

Low stock

2011-04-17T08:00:00.000+02:00

While we wait for the second print run to complete, it looks like some of the online booksellers (such as Amazon.com) have temporarily run out of stock. The second print run should be completed at the start of next month, and sellers will be restocked within a couple of days after that.

man7.org down this weekend

2011-04-16T07:40:00.004+02:00

Currently, the company where I work and host my website is going through a move of offices. As a result, access to the man7.org website will see some interruptions. Everything should be right by Monday.

Update 2011-04-17: And we're back

Second print run going to press

2011-04-11T20:24:00.004+02:00

Last December, I wrote that we'd be preparing the second print run of the book soon. That time window changed a bit, mainly because it turned out that the lead time with the publisher became much shorter than we expected, so that it was possible to delay preparation of the print run until later. By now however, reserves from the first print run have run low enough that the files for the second print run have already gone to the publisher, and will emerge from the presses in a few weeks time.

The new print run will incorporate all of the errata reported up to the end of March, and I have now reorganized the main errata page so that it breaks the errata down by the print run in which they were applied (as well as a list of errata queued for the next print run).

Updated 2011-04-16 to fix a wording error.

Using TLPI in a university course? Let me know

2011-04-08T07:40:00.000+02:00

I didn't specifically target TLPI at the university market as I wrote it. But, by now I've had emails from a number of university teachers who are using TLPI as a required text or as recommended reading for courses on Linux or UNIX system programming.

If you are a teacher or a student using TLPI in a university course, I'd love to hear about it. In particular, I'm very interested to find out more details about how TLPI is being used as a course book, with the idea in mind of improving a future edition of the book for use in that market. If you're a teacher or a student using TLPI, take a look at the questions here, and drop me a mail.

Training course details posted on my web site

2011-04-06T06:30:00.000+02:00

I've posted the full training course outline for the Linux/UNIX system programming course that I teach on my web site. You can find it here.

Japanese translation of TLPI

2011-04-05T20:01:00.001+02:00

O'Reilly Japan have reached an agreement with No Starch Press to do a Japanese translation of TLPI. I'll add some more details on the web site when I have them.

Update, 2012-02-18: It's likely that this translation will be published toward the end of 2012. Ongoing information about this and other translations can be found on the TLPI translations page.

Portability versus innovation

2011-03-06T21:06:00.000+01:00

Choosing between portability and innovation, is an article on LWN.net by Koen Vervloesem that nicely summarizes the debate about portable application development versus (non-portable) innovation based on a single platform (i.e., Linux).

Although the benefits of portable programming are clear, I also have some sympathy for Lennart Poettering's position (which also gave a nice nod to TLPI):

In fact, the way I see things the Linux API has been taking the role of the POSIX API and Linux is the focal point of all Free Software development. Due to that I can only recommend developers to try to hack with only Linux in mind and experience the freedom and the opportunities this offers you. So, get yourself a copy of The Linux Programming Interface, ignore everything it says about POSIX compatibility and hack away your amazing Linux software. It's quite relieving!

I wrote TLPI in such a way that it could be used for programmers whose goal was portability (to the point that it should be useful even to programmers actually working only on UNIX platforms other than Linux). However, as Lennart suggests, TLPI can equally be used in the converse way: as a guide to fully exploiting the features of Linux, without worrying about portability.

Writing TLPI so that it could be used in these dual roles seemed (and still seems) appropriate for the current circumstances. But, by the time it becomes necessary to do a revision of TLPI, it'll be interesting to see if the operating-system landscape has changed to such an extent that Lennart's argument becomes even more compelling.

No Starch Press discount offer extended

2011-02-17T19:33:00.000+01:00

No Starch press has extended their earlier discount offer, allowing you to order the paper version of TLPI from them at a 30% discount (i.e., a total price of US$70 plus shipping), and receive the ebook free (look here for details on the ebook release). The offer is now extended to 30 April 2011 (this is a final deadline).

To obtain the discount (and ebook), you must buy TLPI direct from the publisher's web site, and quote the coupon code Mamaku (what's that?). (No Starch Press is located in San Francisco, so international shipping rates will apply for readers outside North America.)

All ebook formats now available

2011-02-15T22:14:00.000+01:00

The full range of ebook formats is now available: PDF, ePub, and mobi. These formats are currently all on sale on the No Starch and O'Reilly web sites, and will also soon be available through online booksellers as well.

ebook status

2011-01-20T19:33:00.006+01:00

Some days ago, No Starch Press started selling an ebook version of TLPI. The ebook is also provided for free when you buy a paper copy of TLPI from No Starch (look here for coupon—valid until end of February—that provides a publisher discount for the book+ebook package).

Currently, the ebook is available only in PDF form and only from No Starch. However, a full range of other ebook formats is currently in production, and when those formats are available they will be sold through the usual retail channels (and made available to No Starch customers who already have the ebook). I expect those other formats to be available around mid-February.