6.2.1. Queuing Architecture

The architecture of the network subsystem typically follows the architecture of the protocols used by the network subsystem. At the lowest level, the device drivers provide access to the network interfaces. At the highest level, the socket module implements the Berkeley socket interface. In between, the protocol modules implement the ARP, IP, UDP, TCP and other protocols. The modules typically communicate through queues of packets.

As described, the architecture has two pitfalls, both related to a potential loss of efficiency when a large number of modules processes packets.

The first pitfall is caused by excessive data copying. The individual modules that process packets may need to add headers or footers to the data, which may prompt a need for moving the data to make room for the headers or footers. With top desktop systems moving data in memory in hundreds to thousands of MB per second and top network systems moving data in wires in thousands of MB per second, even a small amount of data copying may be a problem.

The second pitfall is caused by excessive data dispatching. Many solutions exist, the traditional ones including hash tables, the wilder ones ranging from dispatcher shortcut caching to dispatcher code generation and dispatcher code upload.

Both pitfalls can be sidestepped by using smart hardware.

6.2.1.1. Example: Linux SK Buff Structure

To avoid data copying, the individual modules that process packets keep data in the sk_buff structure. The structure reserves space before and after data so that headers or footers can be added without data copying.

struct sk_buff *alloc_skb (unsigned int size, int priority);
void skb_reserve (struct sk_buff *skb, unsigned int len);
int skb_headroom (const struct sk_buff *skb);
int skb_tailroom (const struct sk_buff *skb);

unsigned char *skb_put (struct sk_buff *skb, unsigned int len);
unsigned char *skb_push (struct sk_buff *skb, unsigned int len);

unsigned char *skb_pull (struct sk_buff *skb, unsigned int len);
void skb_trim (struct sk_buff *skb, unsigned int len);

References. 

  1. Alan Cox: Network Buffers and Memory Management