thttpd, patches for thttpd-2.21b

A.D.F. -> httpd -> thttpd -> patches -> thttpd-2.21b

1. Features

The following patches apply to original thttpd-2.21b and enhance its performance and its stability (see below development status).  Latest patch includes.

• Keep-alive-patch, (p08) alias HTTP 1.1 persistent connections to increase speed of server reply and to decrease (a bit) network traffic ratio per served requests (see also HTTP performance overview and Network Performance Effects of HTTP 1.1 vs. HTTP 1.0);  since (p26) persistent connections can be disabled by option -nopc or by directive NoPersistentConns (-C configfile);  since (p28) persistent connections can be disabled also by option -noka or by directive NoKeepAliveConns (-C configfile).

• Pipelined-requests-patch, (p22) to properly handle pipelined requests when using HTTP 1.1 persistent connections; pipelined requests greatly reduce latency time effects between client request and server response (due to round time trip of TCP/IP packets plus other factors);
  
  WARNING: patched versions prior to thttpd-2.21b-p22 do not support pipelining and do not work well with HTTP/1.1 slow connections (i.e. bandwidth lower than 30 - 40 KB/sec. per connection);
  patched versions p22 - p27 keep persistent connections only for full GET requests (no GET if-modified-since, HEAD and POST); p28 and following versions keep alive persistent connections also for GET if-modified-since and HEAD requests.

• Max-conn-bytes-limit-patch, (p26) to always limit bandwith of every connection, independently from global throttling; this limit can be set in config.h or in -C configfile by directive MAX_CONN_BYTES_LIMIT or by command line option -mcbl.

• Write-response-patch, (p09) to avoid block-mode switch, when sending error responses, by adding a new thttpd state CNST_SENDING_RESP;  this allows a (very) small speed up under heavy load.

• Filenames-enc-patch, (p15) to allow ´?´ character in file names.

• Mmap-cache-file-size-patch, (p14) to avoid keeping medium and big files in mmap cache for long times; this feature along with other tricks should prevent 500 internal errors caused by mmap(2) failures when reaching max. total mmapped size system limit (usually 1 GB, 2 GB or 3 GB in 32 bit OS).

• Fdmap-cache-patch, (p20) to use a second cache to keep file descriptors of opened files; this cache is used when USE_SENDFILE option is enabled (config.h or -C configfile file) and size of served file is over MAX_FILE_SIZE_L3 or MAX_FILE_SIZE_L4 (p25);
  
  NOTE:  it is recommended to enable USE_SENDFILE if thttpd has to serve lots of big / huge files (i.e. > 1 - 200 MB, surely bigger than MAX_FILE_SIZE_L4), because sendfile() helps to avoid problems with limited mappable memory space;  anyway be aware of old / broken kernels when serving files lying on RAM disks (including TmpFS), on NTFS partitions or on NFS file systems;
  
  NOTE:  if sendfile is enabled by USE_SENDFILE then the max. number of concurrent connections is halved; if performances have to be taken in account, then it is suggested to verify that a native version (not emulated) of sendfile() is used (see also config.h for more details);
  
  NOTE:  there is no large file support (32 bit systems), thus, even when using sendfile(), only files whose size is less than 2 GB can be served (of course 64 bit systems DO NOT HAVE this limit).

• Malloc-cache-patch (p21), to use a third cache ( L0 ) to keep very small files (size between 1 and 4096 bytes) in "malloced RAM"; this often improves speed and memory utilization (in x86 systems mapped files are usually kept in memory pages of 4096 bytes or more); this cache is used only when MAX_FILE_SIZE_L0 is > 0; optimal value should be between 1024 bytes (for slow systems) and 3072 bytes (for very fast and busy systems), 2048 bytes is a good choice.

• Layout-patch (not recommended) (p20), to add a header and/or a footer file before and/or after each HTML file sent to clients.
  NOTE: unless you really need it, it is not recommended to enable this feature (by defining USE_LAYOUT in config.h).

• /dev/poll-patch (Sun OS) (p31), to improve performances when there are thousands of slow connections.

• epoll(4)-patch (Linux 2.6) (p32), to improve performances when there are thousands of slow connections.

• Ipthrottles-patch (p33), to throttle connections using IPv4 addresses, useful if you want to throttle HTTP clients also by IPv4 address classes.

• Enhanced-uri-throttling-patch (p33), to make throttling dynamically adaptive, under almost all conditions.

• Socket-receive-send-buffer-patch (p36f, p37), to change default size of socket buffer(s), in order to optimize memory usage when handling thousands of connections.

• Open-no-atime-patch (p36f, p40), to set O_NOATIME flag (Linux 2.6.8) in open file, in order to avoid updating access time; this can be useful to reduce the number of background disk updates when serving more than a million of files and for some reason file system is not mounted with noatime attribute.

• Max-age-patch (p33c, p36c, p43), to send HTTP header Cache-Control to HTTP/1.1 clients in order to tell them to not recheck for file changes before MAX_AGE seconds have elapsed (since last request).

• Expire-age-patch (p33c, p36c, p44), to send HTTP/1.0 header Expires: to HTTP/1.0 clients in order to tell them to not recheck for file changes before MAX_AGE seconds have elapsed (since last request).

• Sendfile-blk-size-patch (p36g, p44c), to set max. size of data block used by sendfile; MAX_SENDFILE_BLK_SIZE parameter (setable in config.h and in -C configfile) holds the value.

• Max-keepalive-file-size-patch, (p36h, p44d) to close a connection whenever a file bigger than a certain limit is requested and the HTTP response status is 200; this limit can be set in config.h or in -C configfile by directive MAX_KEEPALIVE_FILE_SIZE.

• Configuration-parameters-patch, to add these parameters:

  LOWM_KEEPALIVE_RQSLIMIT=nRequestsPerConnection
  HIWM_KEEPALIVE_RQSLIMIT=nRequestsPerConnection
  MAX_KEEPALIVE_FILE_SIZE=nBytes
  IDLE_SEND_RESP_TIMELIMIT=nSeconds
  IDLE_KEEPALIVE_TIMELIMIT=nSeconds
  OCCASIONAL_MMC_TIME=nSeconds
  DEFAULT_EXPIRE_AGE=nSeconds
  DESIRED_MAX_MALLOC_BYTES=nBytes
  DESIRED_MAX_MALLOC_FILES=nFiles
  DESIRED_MAX_MAPPED_BYTES=nBytes
  DESIRED_MAX_MAPPED_FILES=nFiles
  DESIRED_MAX_OPENED_FILES=nFiles
  MAX_FILE_SIZE_L0=nBytes
  MAX_FILE_SIZE_L1=nBytes
  MAX_FILE_SIZE_L2=nBytes
  MAX_FILE_SIZE_L3=nBytes
  MAX_FILE_SIZE_L4=nBytes
  MAX_SENDFILE_BLK_SIZE=nBytes
  USE_SENDFILE
  NO_SENDFILE
  USE_O_NOATIME
  NO_O_NOATIME
  CONN_SO_RCVBUF=nBytes
  CONN_SO_SNDBUF=nBytes
  MAX_CONN_BYTES_LIMIT=nBytesPerSecond
  MAX_AGE=nSeconds
  NoGenerateIndexes
  NoKeepAliveConns
  NoPersistentConns
  CgiCli=CgiCliFile
  IpThrottles=Ipv4ThrottlesFile
  SubNets=Ipv4SubNetsFile
  Layout_Header=FileName
  Layout_Footer=FileName

  to configuration file (see -C configfile  command line option), so that you don't have to modify file config.h and recompile thttpd to change their values;

• other minor features and bug fixes.

2. Patches

Please, read patch log file thttpd-2.21b-pNN-log.txt for details and bugs not fixed; see also thanks and caveats.

NEWS: 27-JUN-2005, the life-time cycle of these patches has almost reached its end (it has entered the freeze mode); p44 is the last major patch, a few minor bug-fix patches (p44b, p44c, etc.) will be released to make things stable again; the STABLE p36* patches will be maintained for a while (probably till the middle of 2006), so future p36c, p36d, p36e, p36f, etc. will be slowly released to fix or improve minor issues step by step.

NEWS: 15-JUL-2005, development of branch 1 (p33*) has been stopped.

NEWS: 28-FEB-2006, development of branch 2 (p36*) will be completely stopped within June 2006.

NEWS: 24-APR-2006, development of branch 3 (p44*) will be completely stopped within July 2006.

NEWS: 22-JUN-2006, development of branch 2 (p36*) has been stopped.

NEWS: 24-JUL-2006, development of branch 3 (p44*) has been stopped.

Q. Why did you start to release these patches ?
A. Because of many reasons but mainly because when thttpd-2.21b was downgraded to beta and left orphan in favour of thttpd-2.22-betas (followed by thttpd-2.23-betas), I decided to adopt thttpd-2.21b in order to apply quickly bug fixes and new features to it while waiting for next stable release of thttpd.

Q. Why have you maintained patches for thttpd-2.21b only ?
A. Because when thttpd-2.24 and thttpd-2.25 were released, I was too lazy to port all the new features along the fixes, etc. I had applied to thttpd-2.21b.

Q. Do you plan to maintain a web page with patches for thttpd-2.25b or later ?
A. No, I don't plan to maintain a web page with "unofficial" patches for thttpd-2.25b and later versions (I could do that, but it would not be appreciated); anyway I am willing to contribute patches and everything else is useful to new versions of thttpd by E-Mail.

Q. Will you release other major patches (pNN) for thttpd-2.21b ?
A. Hopefully NO, because it is time to ask for new official versions.

Q. Why did you split the patches into 3 branches ?
A. Well, at a certain point of the development, I started to think about the new features that had been added in some patches and when in the middle of 2005 I found a bug that affected all versions of thttpd I decided to split the patches into 3 branches (OLD, STABLE, DEVELOPEMENT) in order to maintain versions which seemed like to be stable enough to be used in Internet servers (i.e. p36*).

Q. Are there patches that belong to different branches and that have the same parameters ?
A. Yes, p36h, p44d and p44e have the same parameters, both in the command line and in the configuration file (see option -C).

Q. Which are the most stable patches ?
A. Assuming to serve static content only (by leaving EXECUTE_CGI disabled in config.h), version p36c and p36e are believed to be stable enough to be used in Internet servers (at least when USE_SENDFILE is disabled / commented), p36f, p36g, p36h and maybe p44d and p44e should be a lot better (specially with USE_SENDFILE enabled).

NOTE: patches that are believed to survive heavy Internet traffic (*), are emphasized by bold font;   this does NOT mean they are perfect !
(*)   (at least when using epoll(2) or poll(2) fdwatch pollers).

Development patches - branch 3 (optimized - development STOPPED)

Development patches - branch 2 (stable - development STOPPED)

Development patches - branch 1 (stale - development STOPPED)