Chapters 29 to 33 are in copyedit

Chapters 22 to 25 are back from copyedit. Chapters 29 to 33 have gone to copyedit.

Chapter 13: File I/O Buffering

In the interests of speed and efficiency, both I/O system calls (i.e., the kernel) and the I/O functions of the standard C library (i.e., the stdio functions) buffer data when operating on disk files. In this chapter, we describe both types of buffering and consider how they affect application performance. We also look at various techniques for influencing and disabling both types of buffering, and look at a technique called direct I/O, which is useful for bypassing kernel buffering in certain circumstances.

13 File I/O Buffering
13.1 Kernel Buffering of File I/O: The Buffer Cache
13.2 Buffering in the stdio Library
13.3 Controlling Kernel Buffering of File I/O
13.4 Summary of I/O Buffering
13.5 Giving the Kernel Hints about I/O Patterns: posix_fadvise()
13.6 Bypassing the Buffer Cache: Direct I/O
13.7 Mixing Library Functions and System Calls for File I/O
13.8 Summary
13.9 Exercises

Chapters 1 to 12

Chapters 1 to 12 form the first two of eight logical parts of my book (see this earlier post). So, this is a good point to post the detailed table of contents so far. Here's the current table of contents for chapters 1 to 12.

1 History and Standards [~20 pages]
1.1 A Brief History of Unix and C
1.2 A Brief History of Linux
        1.2.1 The GNU Project
        1.2.2 The Linux Kernel
1.3 Standardization
        1.3.1 The C Programming Language
        1.3.2 The First POSIX Standards
        1.3.3 X/Open Company and The Open Group
        1.3.4 SUSv3 and POSIX.1-2001
        1.3.5 SUSv4 and POSIX.1-2008
        1.3.6 Unix Standards Timeline
        1.3.7 Implementation Standards
        1.3.8 Linux, Standards, and the Linux Standard Base
1.4 Summary

2 Fundamental Concepts [~20 pages]
2.1 The Core Operating System: The Kernel
2.2 The Shell
2.3 Users and Groups
2.4 Single Directory Hierarchy, Directories, Links, and Files
2.5 File I/O Model
2.6 Programs
2.7 Processes
2.8 Memory Mappings
2.9 Static and Shared Libraries
2.10 Interprocess Communication and Synchronization
2.11 Signals
2.12 Threads
2.13 Process Groups and Shell Job Control
2.14 Sessions, Controlling Terminals, and Controlling Processes
2.15 Pseudoterminals
2.16 Date and Time
2.17 Client-server Architecture
2.18 Realtime
2.19 The /proc File System
2.20 Summary

3 System Programming Concepts [~25 pages]
3.1 System Calls
3.2 Library Functions
3.3 The Standard C Library; The GNU C Library (glibc)
3.4 Handling Errors from System Calls and Library Functions
3.5 Notes on the Example Programs in This Book
        3.5.1 Command-line Options and Arguments
        3.5.2 Common Functions and Header Files
3.6 Portability Issues
        3.6.1 Feature Test Macros
        3.6.2 System Data Types
        3.6.3 Miscellaneous Portability Issues
3.7 Summary
3.8 Exercises

4 File I/O: The Universal I/O Model [~20 pages]
4.1 Overview
4.2 Universality of I/O
4.3 Opening a File: open()
4.4 Reading from a File: read()
4.5 Writing to a File: write()
4.6 Closing a File: close()
4.7 Changing the Current File Offset: lseek()
4.8 Operations Outside the Universal I/O Model: ioctl()
4.9 Summary
4.10 Exercises

5 File I/O: Further Details [~25 pages]
5.1 Atomicity and Race Conditions
5.2 File Control Operations: fcntl()
5.3 Open File Status Flags
5.4 Relationship Between File Descriptors and Open Files
5.5 Duplicating File Descriptors
5.6 File I/O at a Specified Offset: pread() and pwrite()
5.7 Scatter-gather I/O: readv() and writev()
5.8 Truncating a File: truncate() and ftruncate()
5.9 Nonblocking I/O
5.10 I/O on Large Files
5.11 The /dev/fd Directory
5.12 Creating Temporary Files
5.13 Summary
5.14 Exercises

6 Processes [~25 pages]
6.1 Processes and Programs
6.2 Process ID and Parent Process ID
6.3 Memory Layout of a Process
6.4 Virtual Memory Management
6.5 The Stack and Stack Frames
6.6 Command-line Arguments (argc, argv)
6.7 Environment List
6.8 Performing a Nonlocal Goto: setjmp() and longjmp()
6.9 Summary
6.10 Exercises

7 Memory Allocation [~15 pages]
7.1 Allocating Memory on the Heap
        7.1.1 Adjusting the Program Break: brk() and sbrk()
        7.1.2 Allocating Memory on the Heap: malloc() and free()
        7.1.3 Implementation of malloc() and free()
        7.1.4 Other Methods of Allocating Memory on the Heap
7.2 Allocating Memory on the Stack: alloca()
7.3 Summary
7.4 Exercises

8 Users and Groups [~15 pages]
8.1 The Password File: /etc/passwd
8.2 The Shadow Password File: /etc/shadow
8.3 The Group File: /etc/group
8.4 Retrieving User and Group Information
8.5 Password Encryption and User Authentication
8.6 Summary
8.7 Exercises

9 Process Credentials [~20 pages]
9.1 Real User ID and Real Group ID
9.2 Effective User ID and Effective Group ID
9.3 Set-user-ID and Set-group-ID Programs
9.4 Saved Set-user-ID and Saved Set-group-ID
9.5 File System User ID and File System Group ID
9.6 Supplementary Group IDs
9.7 Retrieving and Modifying Process Credentials
        9.7.1 Retrieving and Modifying Real, Effective, and Saved Set IDs
        9.7.2 Retrieving and Modifying File System IDs
        9.7.3 Retrieving and Modifying Supplementary Group IDs
        9.7.4 Summary of Calls for Modifying Process Credentials
        9.7.5 Example: Displaying Process Credentials
9.8 Summary
9.9 Exercises

10 Times and Dates [~25 pages]
10.1 Calendar Time
10.2 Time-Conversion Functions
        10.2.1 Converting time_t to Printable Form
        10.2.2 Converting Between time_t and Broken-down Time
        10.2.3 Converting Between Broken-down Time and Printable Form
10.3 Timezones
10.4 Locales
10.5 Updating the System Clock
10.6 The Software Clock (Jiffies)
10.7 Process Time
10.8 Summary
10.9 Exercises

11 System Limits and Options [~10 pages]
11.1 System Limits
11.2 Retrieving System Limits (and Options) at Run Time
11.3 Retrieving File-related Limits (and Options) at Run Time
11.4 Indeterminate Limits
11.5 System Options
11.6 Summary
11.7 Exercises

12 System and Process Information [~10 pages]
12.1 The /proc File System
        12.1.1 Obtaining Information about a Process: /proc/PID
        12.1.2 System Information under /proc
        12.1.3 Accessing /proc Files
12.2 System Identification: uname()
12.3 Summary
12.4 Exercises

Chapter 12: System and Process Information

In this chapter, we look at ways of retrieving and modifying a variety of system information. The primary focus of the chapter is a discussion of the /proc file system. We also describe the uname() system call, which is used to retrieve various system identifiers.

12 System and Process Information
12.1 The /proc File System
        12.1.1 Obtaining Information about a Process: /proc/PID
        12.1.2 System Information under /proc
        12.1.3 Accessing /proc Files
12.2 System Identification: uname()
12.3 Summary
12.4 Exercises

Chapter 11: System Limits and Options

Each Unix implementation sets limits on various system features and resources, and provides--or chooses not to provide--options defined in various standards. Examples include the following:

• How many files can a process hold open at one time?
• Does the system support realtime signals?
• What is the biggest value that can be stored in a variable of type int?
• How big an argument list can a program have?
• What is the maximum length of a pathname?

While we could hard-code assumed limits and options into an application, this reduces portability, since limits and options may vary:

• Across Unix implementations: Although limits and options may be fixed on an individual implementation, they can vary from one Unix implementation to another. The maximum value that can be stored in an int is an example of such a limit.
• At run time on a particular implementation: The kernel may have been reconfigured to change a limit, for example. Alternatively, the application may have been com-piled on one system, but run on another system with different limits and options.
• From one file system to another: For example, traditional System V file systems allow a filename to be up to 14 bytes, while traditional BSD file systems and most native Linux file systems allow filenames of up to 255 bytes.

Since these system limits affect what an application may do, a portable application needs ways of determining their values and whether various system options are supported. The C programming language standards and SUSv3 provide two principal avenues for an application to obtain such information:

• Some limits and options can be determined at compile time. For example, the maximum value of an int is determined by the hardware architecture and compiler design choices. Such limits can be recorded in header files.
• Other limits and options may vary at run time. For such cases, SUSv3 defines three functions--sysconf(), pathconf(), and fpathconf()--that an application can call to check these implementation limits and options.

SUSv3 specifies a range of limits that a conforming implementation may enforce, as well as a set of options, each of which may or may not be provided by a particular system. We provide an overview of these limits and options in this chapter. We also describe the various SUSv3-specified constants and functions that an application can use to obtain information about system limits and options. Using these techniques eases the task of writing applications that are portable to other Unix implementations.

11 System Limits and Options
11.1 System Limits
11.2 Retrieving System Limits (and Options) at Run Time
11.3 Retrieving File-related Limits (and Options) at Run Time
11.4 Indeterminate Limits
11.5 System Options
11.6 Summary
11.7 Exercises

Chapter 10: Times and Dates

Within a program, we may be interested in two kinds of time:

• Real time: This is the time as measured either from some standard point (calendar time) or from some fixed point (typically the start) in the life of a process (elapsed or wall clock time). Obtaining the calendar time is useful to programs that, for example, timestamp database records or files. Measuring elapsed time is useful for a program that takes periodic actions or makes regular measurements from some external input device.

• Process time: This is the amount of CPU time used by a process. Measuring process time is useful for checking or optimizing the performance of a program or algorithm.

Most computer architectures have a built-in hardware clock that enables the kernel to measure real and process time. In this chapter, we look at system calls for dealing with both sorts of time, and library functions for converting between human-readable and internal representations of time. Since human-readable representations of time are dependent on the geographical location and on linguistic and cultural conventions, discussion of these representations leads us into an investigation of timezones and locales.

10 Times and Dates
10.1 Calendar Time
10.2 Time-Conversion Functions
        10.2.1 Converting time_t to Printable Form
        10.2.2 Converting Between time_t and Broken-down Time
        10.2.3 Converting Between Broken-down Time and Printable Form
10.3 Timezones
10.4 Locales
10.5 Updating the System Clock
10.6 The Software Clock (Jiffies)
10.7 Process Time
10.8 Summary
10.9 Exercises

Chapter 9: Process Credentials

Every process has a number of associated numeric user identifiers (UIDs) and group identifiers (GIDs). Sometimes these are referred to as process credentials. These identifiers are as follows:

• real user ID and group ID;
• effective user ID and group ID;
• saved set-user-ID and saved set-group-ID;
• file system user ID and group ID (Linux-specific); and
• supplementary group IDs.

In the following pages, we look in detail at the purpose of these process identifiers and describe the system calls and library functions that can be used to retrieve and change these identifiers. We also discuss the notion of privileged and unprivileged processes, and the use of the set-user-ID and set-group-ID mechanisms, which allow the creation of programs that run with the privileges of a specified user or group.

9 Process Credentials
9.1 Real User ID and Real Group ID
9.2 Effective User ID and Effective Group ID
9.3 Set-user-ID and Set-group-ID Programs
9.4 Saved Set-user-ID and Saved Set-group-ID
9.5 File System User ID and File System Group ID
9.6 Supplementary Group IDs
9.7 Retrieving and Modifying Process Credentials
        9.7.1 Retrieving and Modifying Real, Effective, and Saved Set IDs
        9.7.2 Retrieving and Modifying File System IDs
        9.7.3 Retrieving and Modifying Supplementary Group IDs
        9.7.4 Summary of Calls for Modifying Process Credentials
        9.7.5 Example: Displaying Process Credentials
9.8 Summary
9.9 Exercises

Chapter 8: Users and Groups

Every user has a unique login name and an associated numeric user identifier (UID). Users can belong to one or more groups. Each group also has a unique name and a group identifier (GID).

The primary purpose of user and group IDs is to determine ownership of various system resources and to control the permissions granted to processes accessing those resources. For example, each file belongs to a particular user and group, and each process has a number of user and group IDs that determine who owns the process and what permissions it has when accessing a file (see Chapter 9 for details).

In this chapter, we look at the system files that are used to define the users and groups on the system, and then describe the library functions used to retrieve information from these files. We conclude with a discussion of the crypt() function, which is used to encrypt and authenticate login passwords.

8 Users and Groups
8.1 The Password File: /etc/passwd
8.2 The Shadow Password File: /etc/shadow
8.3 The Group File: /etc/group
8.4 Retrieving User and Group Information
8.5 Password Encryption and User Authentication
8.6 Summary
8.7 Exercises

Chapter 7: Memory Allocation

Many system programs need to be able to allocate extra memory for dynamic data structures (e.g., linked lists and binary trees), whose size depends on information that is available only at run time. In this chapter, we describe the functions that are used to allocate memory on the heap or the stack.

7 Memory Allocation
7.1 Allocating Memory on the Heap
        7.1.1 Adjusting the Program Break: brk() and sbrk()
        7.1.2 Allocating Memory on the Heap: malloc() and free()
        7.1.3 Implementation of malloc() and free()
        7.1.4 Other Methods of Allocating Memory on the Heap
7.2 Allocating Memory on the Stack: alloca()
7.3 Summary
7.4 Exercises

Chapter 6: Processes

In this chapter, we look at the structure of a process, paying particular attention to the layout and contents of a process’s virtual memory. We also examine some of the attributes of a process. In later chapters, we examine further process attributes (for example, process credentials in Chapter 9, and process priorities and scheduling in Chapter 35). In Chapters 24, 25, 26, and 27, we look at how processes are created, how they terminate, and how they can be made to execute new programs.

6 Processes
6.1 Processes and Programs
6.2 Process ID and Parent Process ID
6.3 Memory Layout of a Process
6.4 Virtual Memory Management
6.5 The Stack and Stack Frames
6.6 Command-line Arguments (argc, argv)
6.7 Environment List
6.8 Performing a Nonlocal Goto: setjmp() and longjmp()
6.9 Summary
6.10 Exercises

Chapters 24 to 28 are in copyedit

Chapters 20 and 21 came back copyedit. Chapters 24 to 28 have gone to copyedit.

Thanks, LWN

The kind folks at LWN.net picked up on this blog. Given the relatively small amount of publicity I've made so far for the blog, their article easily generated the best day of traffic so far. Thanks, LWN!