2012-04-09
TLPI third print run now available
I'm happy to announce that the third print run of TLPI is now printed, and should be available for sale shortly. The third print run incorporates these 131 errata shown on the errata page. (If that seems like a lot of errata, take a look at this FAQ.)
2012-03-13
Kernel capability usage statistics
(Update: 2012-07-16: I wrote a related article for this blog post, "CAP_SYS_ADMIN: the new root" that was published on LWN.net on 2012-03-14.)
The idea of capabilities is to break the power of root (user ID 0) into independently assigned pieces governing specific privileged operations. Implicit in that model is that the set of privileged operations governed by each capability should be small (otherwise, why break the power of root into pieces at all?). However, that implication hasn't turned out to be true in practice.
Table 1 shows some statistics on the use of the 36 currently existing capabilities in the C files of the Linux 3.2 kernel source code. The "#uses" column is the number of uses of the capability across all source files; the "#files" column is the number of distinct source files where the capability is used.
Table 1: Breakdown of Linux capability uses in Linux 3.2
What is of course notable is the extremely heavy use of two capabilities: CAP_SYS_ADMIN and CAP_NET_ADMIN. Together, these two capabilities account for more than 70% of all uses of capabilities! The uses of CAP_NET_ADMIN are limited to the drivers/ (mainly drivers/net/) and net/ directories. On the other hand, the uses of CAP_SYS_ADMIN, which accounts for nearly 39% of all capability uses, are spread widely across the kernel source tree.
One might wonder whether either of these two capabilities is overrepresented because of duplications in the drivers/ or the arch/ trees. (In particular, CAP_SYS_ADMIN is used for similar administrative functions on a lot of device drivers.) However, even if we strip drivers/ and architectures other than x86 from the measurements, the overall picture doesn't change greatly, as shown in Table 2.
Table 2: Breakdown of Linux capability uses in Linux 3.2, excluding drivers and architectures other than x86
CAP_SYS_ADMIN still accounts for 167 of 552 uses of capabilities--about 30%, and, by chance, usage of CAP_NET_ADMIN is the same.
It turns out that the overall picture hasn't changed that much since capabilities were first introduced with Linux 2.2 (Jan 1999). Then, there were 27 capabilities, broken down as shown in Table 3.
Table 3: Breakdown of Linux capability uses in Linux 2.2
In Linux 2.2, CAP_SYS_ADMIN accounted for 42% of the uses of capabilities, and CAP_NET_ADMIN accounted for 25%.
Table 4 repeats the earlier exercise of excluding drivers/ and architectures other than i386 (as the Intel arch/ directory was then named) from the Linux 2.2 data. In this case, an interesting difference emerges.
Table 4: Breakdown of Linux capability uses in Linux 2.2, excluding drivers and architectures other than i386
The overall picture differs in a way that I suspect is significant: just under 16% (23/147) of the uses of capabilities are CAP_SYS_ADMIN. (As we saw in Table 2, by Linux 3.2, this figure had grown to 30% (167/552).) This difference suggests to me that as a series of kernel developers was faced with the question: "What capability should I use to govern my new privileged kernel feature?", the answer was often something like "I don't know; maybe CAP_SYS_ADMIN?". (That certainly fits with a few anecdotal cases I've encountered while discussing things with kernel developers as I wrote man pages for new kernel features.)
The script (count_kernel_cap_uses.sh) used to generate the data for these statistics can be found here. The first and third tables above are based on analysis of the "p2" output files produced by the script. The second and fourth tables are based on analysis of the "p4" output files.
The idea of capabilities is to break the power of root (user ID 0) into independently assigned pieces governing specific privileged operations. Implicit in that model is that the set of privileged operations governed by each capability should be small (otherwise, why break the power of root into pieces at all?). However, that implication hasn't turned out to be true in practice.
Table 1 shows some statistics on the use of the 36 currently existing capabilities in the C files of the Linux 3.2 kernel source code. The "#uses" column is the number of uses of the capability across all source files; the "#files" column is the number of distinct source files where the capability is used.
Table 1: Breakdown of Linux capability uses in Linux 3.2
| Capability | #uses | #files |
| CAP_AUDIT_CONTROL | 2 | 2 |
| CAP_AUDIT_WRITE | 1 | 1 |
| CAP_CHOWN | 4 | 2 |
| CAP_DAC_OVERRIDE | 2 | 1 |
| CAP_DAC_READ_SEARCH | 4 | 2 |
| CAP_FOWNER | 9 | 8 |
| CAP_FSETID | 8 | 6 |
| CAP_IPC_LOCK | 13 | 8 |
| CAP_IPC_OWNER | 1 | 1 |
| CAP_KILL | 2 | 2 |
| CAP_LEASE | 1 | 1 |
| CAP_LINUX_IMMUTABLE | 13 | 13 |
| CAP_MAC_ADMIN | 25 | 5 |
| CAP_MAC_OVERRIDE | 6 | 2 |
| CAP_MKNOD | 3 | 3 |
| CAP_NET_ADMIN | 395 | 182 |
| CAP_NET_BIND_SERVICE | 13 | 10 |
| CAP_NET_BROADCAST | 0 | 0 |
| CAP_NET_RAW | 18 | 11 |
| CAP_SETFCAP | 3 | 2 |
| CAP_SETGID | 10 | 6 |
| CAP_SETPCAP | 2 | 2 |
| CAP_SETUID | 8 | 4 |
| CAP_SYS_ADMIN | 451 | 229 |
| CAP_SYS_BOOT | 2 | 2 |
| CAP_SYS_CHROOT | 1 | 1 |
| CAP_SYSLOG | 2 | 2 |
| CAP_SYS_MODULE | 4 | 3 |
| CAP_SYS_NICE | 14 | 8 |
| CAP_SYS_PACCT | 1 | 1 |
| CAP_SYS_PTRACE | 11 | 6 |
| CAP_SYS_RAWIO | 67 | 42 |
| CAP_SYS_RESOURCE | 36 | 24 |
| CAP_SYS_TIME | 22 | 13 |
| CAP_SYS_TTY_CONFIG | 11 | 4 |
| CAP_WAKE_ALARM | 2 | 1 |
| Total | 1167 | 610 |
What is of course notable is the extremely heavy use of two capabilities: CAP_SYS_ADMIN and CAP_NET_ADMIN. Together, these two capabilities account for more than 70% of all uses of capabilities! The uses of CAP_NET_ADMIN are limited to the drivers/ (mainly drivers/net/) and net/ directories. On the other hand, the uses of CAP_SYS_ADMIN, which accounts for nearly 39% of all capability uses, are spread widely across the kernel source tree.
One might wonder whether either of these two capabilities is overrepresented because of duplications in the drivers/ or the arch/ trees. (In particular, CAP_SYS_ADMIN is used for similar administrative functions on a lot of device drivers.) However, even if we strip drivers/ and architectures other than x86 from the measurements, the overall picture doesn't change greatly, as shown in Table 2.
Table 2: Breakdown of Linux capability uses in Linux 3.2, excluding drivers and architectures other than x86
| Capability | #uses | #files |
| CAP_AUDIT_CONTROL | 2 | 2 |
| CAP_AUDIT_WRITE | 1 | 1 |
| CAP_CHOWN | 4 | 2 |
| CAP_DAC_OVERRIDE | 2 | 1 |
| CAP_DAC_READ_SEARCH | 4 | 2 |
| CAP_FOWNER | 9 | 8 |
| CAP_FSETID | 8 | 6 |
| CAP_IPC_LOCK | 11 | 6 |
| CAP_IPC_OWNER | 1 | 1 |
| CAP_KILL | 1 | 1 |
| CAP_LEASE | 1 | 1 |
| CAP_LINUX_IMMUTABLE | 13 | 13 |
| CAP_MAC_ADMIN | 25 | 5 |
| CAP_MAC_OVERRIDE | 6 | 2 |
| CAP_MKNOD | 3 | 3 |
| CAP_NET_ADMIN | 167 | 73 |
| CAP_NET_BIND_SERVICE | 12 | 9 |
| CAP_NET_BROADCAST | 0 | 0 |
| CAP_NET_RAW | 18 | 11 |
| CAP_SETFCAP | 3 | 2 |
| CAP_SETGID | 9 | 5 |
| CAP_SETPCAP | 2 | 2 |
| CAP_SETUID | 8 | 4 |
| CAP_SYS_ADMIN | 167 | 80 |
| CAP_SYS_BOOT | 2 | 2 |
| CAP_SYS_CHROOT | 1 | 1 |
| CAP_SYSLOG | 2 | 2 |
| CAP_SYS_MODULE | 4 | 3 |
| CAP_SYS_NICE | 12 | 6 |
| CAP_SYS_PACCT | 1 | 1 |
| CAP_SYS_PTRACE | 10 | 5 |
| CAP_SYS_RAWIO | 10 | 9 |
| CAP_SYS_RESOURCE | 26 | 18 |
| CAP_SYS_TIME | 4 | 2 |
| CAP_SYS_TTY_CONFIG | 1 | 1 |
| CAP_WAKE_ALARM | 2 | 1 |
| Total | 552 | 291 |
CAP_SYS_ADMIN still accounts for 167 of 552 uses of capabilities--about 30%, and, by chance, usage of CAP_NET_ADMIN is the same.
It turns out that the overall picture hasn't changed that much since capabilities were first introduced with Linux 2.2 (Jan 1999). Then, there were 27 capabilities, broken down as shown in Table 3.
Table 3: Breakdown of Linux capability uses in Linux 2.2
| Capability | #uses | #files |
| CAP_CHOWN | 2 | 1 |
| CAP_DAC_OVERRIDE | 5 | 5 |
| CAP_DAC_READ_SEARCH | 4 | 4 |
| CAP_FOWNER | 7 | 5 |
| CAP_FSETID | 3 | 2 |
| CAP_IPC_LOCK | 4 | 2 |
| CAP_IPC_OWNER | 1 | 1 |
| CAP_KILL | 0 | 0 |
| CAP_LINUX_IMMUTABLE | 2 | 2 |
| CAP_NET_ADMIN | 75 | 32 |
| CAP_NET_BIND_SERVICE | 3 | 3 |
| CAP_NET_BROADCAST | 0 | 0 |
| CAP_NET_RAW | 8 | 6 |
| CAP_SETGID | 7 | 2 |
| CAP_SETPCAP | 2 | 2 |
| CAP_SETUID | 7 | 3 |
| CAP_SYS_ADMIN | 127 | 69 |
| CAP_SYS_BOOT | 1 | 1 |
| CAP_SYS_CHROOT | 1 | 1 |
| CAP_SYS_MODULE | 4 | 2 |
| CAP_SYS_NICE | 5 | 2 |
| CAP_SYS_PACCT | 1 | 1 |
| CAP_SYS_PTRACE | 9 | 9 |
| CAP_SYS_RAWIO | 2 | 1 |
| CAP_SYS_RESOURCE | 10 | 8 |
| CAP_SYS_TIME | 7 | 4 |
| CAP_SYS_TTY_CONFIG | 1 | 1 |
| Total | 298 | 169 |
In Linux 2.2, CAP_SYS_ADMIN accounted for 42% of the uses of capabilities, and CAP_NET_ADMIN accounted for 25%.
Table 4 repeats the earlier exercise of excluding drivers/ and architectures other than i386 (as the Intel arch/ directory was then named) from the Linux 2.2 data. In this case, an interesting difference emerges.
Table 4: Breakdown of Linux capability uses in Linux 2.2, excluding drivers and architectures other than i386
| Capability | #uses | #files |
| CAP_CHOWN | 2 | 1 |
| CAP_DAC_OVERRIDE | 5 | 5 |
| CAP_DAC_READ_SEARCH | 4 | 4 |
| CAP_FOWNER | 7 | 5 |
| CAP_FSETID | 3 | 2 |
| CAP_IPC_LOCK | 4 | 2 |
| CAP_IPC_OWNER | 1 | 1 |
| CAP_KILL | 0 | 0 |
| CAP_LINUX_IMMUTABLE | 2 | 2 |
| CAP_NET_ADMIN | 44 | 23 |
| CAP_NET_BIND_SERVICE | 3 | 3 |
| CAP_NET_BROADCAST | 0 | 0 |
| CAP_NET_RAW | 8 | 6 |
| CAP_SETGID | 7 | 2 |
| CAP_SETPCAP | 2 | 2 |
| CAP_SETUID | 7 | 3 |
| CAP_SYS_ADMIN | 23 | 17 |
| CAP_SYS_BOOT | 1 | 1 |
| CAP_SYS_CHROOT | 1 | 1 |
| CAP_SYS_MODULE | 4 | 2 |
| CAP_SYS_NICE | 5 | 2 |
| CAP_SYS_PACCT | 1 | 1 |
| CAP_SYS_PTRACE | 2 | 2 |
| CAP_SYS_RAWIO | 2 | 1 |
| CAP_SYS_RESOURCE | 5 | 4 |
| CAP_SYS_TIME | 3 | 1 |
| CAP_SYS_TTY_CONFIG | 1 | 1 |
| Total | 147 | 94 |
The overall picture differs in a way that I suspect is significant: just under 16% (23/147) of the uses of capabilities are CAP_SYS_ADMIN. (As we saw in Table 2, by Linux 3.2, this figure had grown to 30% (167/552).) This difference suggests to me that as a series of kernel developers was faced with the question: "What capability should I use to govern my new privileged kernel feature?", the answer was often something like "I don't know; maybe CAP_SYS_ADMIN?". (That certainly fits with a few anecdotal cases I've encountered while discussing things with kernel developers as I wrote man pages for new kernel features.)
The script (count_kernel_cap_uses.sh) used to generate the data for these statistics can be found here. The first and third tables above are based on analysis of the "p2" output files produced by the script. The second and fourth tables are based on analysis of the "p4" output files.
2012-02-22
Look hard, and you can see TLPI
This is fun... I'm fairly certain that the guess by this redditor about what you can see sitting behind the laptop in this photo (taken by John Snyder, and used in the 21 Feb 2012 Wired article, Lord of the Files: How GitHub Tamed Free Software (And More)) is correct. Probably, it's the copy I gave Linus last year. It's nice to know he held on to it, though I'm not so sure there's so much in the book that he doesn't already know.
Updated 2015-07-06: fix a broken URL, and add the title of Wired article.
Updates 2016-07-25: fix a broken URL (again... Wired.com doesn't seem to embrace the notion of stable URLs).
Updated 2015-07-06: fix a broken URL, and add the title of Wired article.
Updates 2016-07-25: fix a broken URL (again... Wired.com doesn't seem to embrace the notion of stable URLs).
2012-02-20
Traditional Chinese translation of TLPI
The Taiwanese publisher GOTOP has contracted the rights to do a Traditional Chinese publication of TLPI.
There are now four translations of TLPI in progress. It looks like the first of the translations that will appear will be the Korean translation, sometime around the middle of this year.
There are now four translations of TLPI in progress. It looks like the first of the translations that will appear will be the Korean translation, sometime around the middle of this year.
2012-02-04
Web site updates
I've rolled out a few updates to the man7.org web site, including a redesigned main page for TLPI, and the addition of several errata to the errata page. Special thanks for a long list of error reports to Junjiro Okajima, who's working on the Japanese translation of TLPI.
2012-01-16
Third print run (and request for bug reports)
Sales of The Linux Programming Interface have continued well enough that the publisher will soon start preparing the third print run. That print run will incorporate all of the outstanding errata.
If you've been reading TLPI and noticed any errors or typos, now would be a good time to report them, so that fixes can be included in the next printing.
If you've been reading TLPI and noticed any errors or typos, now would be a good time to report them, so that fixes can be included in the next printing.
2011-12-03
Nice discount on No Starch books
No Starch Press is doing a brief pre-Christmas sale -- 40% off all paper books, and the ebook is included for free. That's cheaper than the normal discount offered for TLPI, and it applies to all No Starch books. But I only found out about this sale a little late: according to http://nostarch.com/newsletters/2011_holiday.htm, the sale finishes at midnight on 3 Dec. I presume that means midnight in California (UTC-8), about 24 hours from now. So, you might want to take a quick look at the full catalog at No Starch.
PS The publisher's costs for international shipping from the US is pretty hefty, but with the free ebooks, and if you're ordering more than one book, the prices might still beat the online bookstores even if you're shipping outside the US.
PS The publisher's costs for international shipping from the US is pretty hefty, but with the free ebooks, and if you're ordering more than one book, the prices might still beat the online bookstores even if you're shipping outside the US.
2011-11-09
2011-11-04
Speaking tomorrow at OpenFest
The organizers very kindly invited me to speak at OpenFest in Sofia, Bulgaria. My talk tomorrow is entitled Why Kernel Space Sucks.
That title is of course a reference to the highly amusing presentation of a few years ago by kernel hacker Dave Jones entitled Why Userspace Sucks (MagicPoint presentation). (For the PDF of Dave's complete paper presented at Linux Symposium 2006, look here; LWN.net has a nice tl;dr summary.)
Dave's presentation was all about the ways in which various userspace systems and applications kill performance by wasting system resources on pointless tasks. I'm not contradicting anything that Dave says, but it seems at least fair to point out that there are places where kernel space sucks too. My talk is about one of those places with special interest to me--the kernel-userspace programming interface--a place where kernel developers have inflicted a steady stream of small train wrecks (to borrow Dave's term) on userspace.
That title is of course a reference to the highly amusing presentation of a few years ago by kernel hacker Dave Jones entitled Why Userspace Sucks (MagicPoint presentation). (For the PDF of Dave's complete paper presented at Linux Symposium 2006, look here; LWN.net has a nice tl;dr summary.)
Dave's presentation was all about the ways in which various userspace systems and applications kill performance by wasting system resources on pointless tasks. I'm not contradicting anything that Dave says, but it seems at least fair to point out that there are places where kernel space sucks too. My talk is about one of those places with special interest to me--the kernel-userspace programming interface--a place where kernel developers have inflicted a steady stream of small train wrecks (to borrow Dave's term) on userspace.
2011-10-20
LinuxCon Europe
I'll be at LinuxCon Europe, 26-28 October, in Prague, Czech Republic. (In fact, I arrive already on Monday, 24 October.) I'll have a few copies of The Linux Programming Interface with me for sale. If you want to buy a copy (signed even, if you want!), drop me a mail (mtk AT man7.org).
2011-08-14
Linux Plumbers Conference
I'm going to be at Linux Plumbers Conference, 7-9 September, in Santa Rosa, California. I'm looking forward to connect up with folk, and spend a couple of days in the Bay Area. I may have a limited number of copies of The Linux Programming Interface with me for sale. If you want to buy a copy directly from me (signed even, if you want!), please contact me by email by the end of this month (mtk AT man7.org).
2011-05-27
Updated API Changes page
With the release of Linux 2.6.39, and the opening of the merge window for Linux 2.8.0 (or 2.6.40, or 3.0, or whatever) 3.0, I've updated the API changes page to include recent and upcoming userspace API changes in the Linux kernel. (This page covers changes since Linux 2.6.36, which was the next kernel released after the book went to press.)
Updated 2011-05-30: fixed kernel version number
Updated 2011-05-30: fixed kernel version number
Subscribe to:
Posts (Atom)
