------------------------------------------------ File (patch log): thttpd-2.21b-pNN-log.txt ------------------------------------------------ HINT ---- READ ALL this file CAREFULLY, some things / concepts are repeated more than once ("repetita iuvant"), so don't blame me if you miss some detail. IMPORTANT --------- These are development patches for thttpd-2.21b; they have to be considered as the result of a work done to optimize some aspects related to speed and resource utilization (file cache) (optimized for -- STATIC CONTENT -- only). NOTE: there are 3 branches: development (p37 - p44x) current stable (p34 - p36x) obsolete (p10 - p33x) ------------------------------------------------------------------------- NOTE: the most interesting features / enhancements added by these patches can be freely proposed for inclusion into future official versions. ------------------------------------------------------------------------- TODO (WON'T DO) ---- o Enhance HTTP/1.1 conformance against RFC 2616 (mostly done). o Auth file performances. TWEAKS TO ENHANCE PERFORMANCES ------------------------------ o for best output performances: - leave undefined EXECUTE_CGI, GENERATE_INDEXES, AUTH_FILE, etc. (see config.h, look for defines with the keyword: CPU); - tweak cache parameters (DEFAULT_EXPIRE_AGE, *_L1-L4, etc.); - disable logging (see below "logfile"); - consider using sendfile with zero copy TCP/IP stack (NOTE: not all network cards are zero copy ready), specially useful at gigabit speeds and/or with huge files; - be sure to prevent excessive fragmentation of files on your file systems (clustering files is a good idea); - consider using MAX_AGE=nSeconds if your files rarely change; if the server is very busy, then even a small value, i.e. 100-300 seconds can help a lot (HINT: if thttpd serves only images that never change then feel free to use a much longer time such as MAX_AGE=86400); - if possible put thttpd on a dedicated server (underpowered hardware is usually OK), i.e. don't let thttpd and Apache work on the same computer because Apache may kill thttpd performance by grabbing too much CPU time and disk I/O; under certain circumstances this may not be a real problem if small network latencies (50 - 2000 mlsec.) are tolerated. o max. number of files, if you need more file (socket) handles available to thttpd then tweak your OS limits and call ulimit -n (from shell, sh, bash, etc.) just before executing thttpd, i.e.: # big and busy server using kqueue or epoll ulimit -n 8192 ./thttpd -C thttpd.cfg NOTE: take note that ENABLING SENDFILE (USE_SENDFILE) HALVES THE MAX. NUMBER OF SOCKET CONNECTIONS, so in that case you may want to increase the limit, see the syslog file near "maxconnects" value. NOTE: if the limit of 1000-1500 simultaneous connections is frequently exceeded, then you may want to use/enable kqueue or epoll or /dev/poll network poll interfaces (look at periodic syslog statistics to discover which interface has been enabled at compile time); o socket send / receive buffer size, if you have thousands of concurrent connections you may want to tweak default values, see below CONN_SO_*, anyway it must be taken in account that nowadays most OSs are able to auto-tune the size of I/O buffers used by TCP/IP; o logfile, disabling it (logfile=/dev/null) can boost a bit performances (useful with several thousands of requests per second); o access time, see NO_ATIME (used only in Linux 2.6.8 and later). BUGS NOT FIXED -------------- o autoconf(1), newer versions (i.e. 2.57) don't work (2.13 worked), not a big deal, use existing ./configure. NOTE: the same problem affects all official versions, up to 2.25b. o ./configure, many Linux Red Hat systems (RHEL-3) were shipped with /usr/include/sys/epoll.h but without epoll(2) interface so that ./configure enables epoll without testing it with a run-time test and then at startup thttpd exits because of fdwatch initialization failure; in this case disable epoll(2) by undefining -DHAVE_EPOLL_CTL=1 in generated Makefile, then recompile thttpd: make clean; make. o CGI arguments, relative paths "../" cannot be used in CGI vars, i.e. this URL is forbidden: http://myhost/cgi-bin/mycgi?myFilePath=../../myfile&myFileExt=txt NOTE: this is fixed in official thttpd-2.25 and later versions. o CGI POST requests: - no chunked transfer encoding for HTTP/1.1 POST requests (only Content-Length header is understood); - possible starving POST requests with HTTP/1.1. o CGI no HTTP-100 continue response, well this is not implemented, thus it is not even a bug; after a few seconds client should timeout and then it should send POST contents anyway. o CGI zombie children: this is due to a buggy combination of OS version + libc libraries; if you are using Linux RH 7.x, 8.x, 9.x, then, please, update your system (apply all known patches, etc.) and if the problem persists complain with your software provider or (much better) use an official version of thttpd which catches SIGCHLD (thttpd-2.24 and following). NOTE: CGI handling is "believed" to work (more or less, at least nph-*. CGIs with GET requests and HTTP/1.0) but it has a few small problems that are better addressed only in thttpd-2.24 and following versions. --------------------------------------------------------------- HINT: if you plan to use CGI then I strongly RECOMMEND TO CHOOSE AN OFFICIAL VERSION of thttpd. As last resource try to use only nph-* CGIs and HTTP GET method. --------------------------------------------------------------- o other minor issues. BE AWARE OF ----------- o FILE CORRUPTION WHEN UPDATING AN EXISTING FILE; the ONLY SAFE METHOD TO UPDATE A FILE is the following one: 1) upload new file, using a temporary name, in the same filesystem where target file lies; 2) if upload is aborted then remove temporary file, otherwise if it succedes, rename temporary file to target file (this should be an atomic operation because target file should be unlinked / removed and a new fresh copy should appear); after update, thttpd uses the old copy to finish current downloads (if there are any in progress); when thttpd unmaps the old copy, OS will automatically remove that file as soon as noone else uses it anymore; the new copy is instead used for all new requests/downloads. NOTE: this method prevents unpleasant side effects in all OS, but specially in Linux 2.4 and later (no SIGSEGV, SIGBUS, etc.). NOTE: eventually consider tuning / tweaking / patching your FTP server to make it behave as suggested. o URL / path handling, it has been changed in p19 and in p29, thus be careful about checking thttpd behaviour. o size of single files (limited to 2 GB, no large file support). o file cache changes since p20, p25, p26, p36d and p44b. o sendfile support, native mode only for: - BSD (p20); - Linux (p20); - Solaris (p25); - Solaris (p31) - sendfilev(). NOTE: many kernels are broken and may have problems when using sendfile() for files lying on RAM disks, TmpFS, NFS or NTFS file systems. NOTE: see also below notes for p25 and p31. o keep-alive does not work well through slow connections in versions < p22. o pipelined requests are not supported in versions < p22. o in versions p19 - p23, association between file extension and corresponding MIME TYPE is CASE SENSITIVE (due to an optimization in search function), thus it works only if the two strings match exactly (i.e. index.html is found, index.HTML is not found); if upper case extensions are used (with above versions) then add them to mime_types.txt (duplicate each row and change only first column) and reconfigure thttpd: make distclean; ./configure; make this is fixed in p24 and later. o syslog, compilation / link warnings under Mac OSX fixed since p26 by undefining USE_MY_OLD_SYSLOG. o throttling calculation does not work well when using HTTP/1.1 (persistent connections): it should be fixed since p26. o kqueue interface (fdwatch.c), now it should work also under FreeBSD >= 5.0 (fixed in thttpd-2.23 and in thttpd-2.21b-p28). o keep-alive for HEAD and GET if-modified-since-304 requests, support added since p28. o CGI-CLI, command language interface, added in p29. o fdwatch enhancements in p30, p31, p32, p33 and p41: (p31) /dev/poll; (p32) epoll; (p33) kqueue fixes; (p41) fdwatch_mod_fd(). o BUG in tmr_run(), timers.c, in all official thttpd versions up to thttpd-2.25b and in the following patches for version thttpd-2.21b: p01 - p33 p34 - p36 p37 - p43 these patches have the fix instead: p33b, p33c p36b, p36c, p36d(BUG), p36e, ecc. p44, p44b, ecc. (BUG) p36d is buggy don't use it. o USE_SENDFILE vs. max. number of connections, fixed in: p36f, p44b ------------------------------------------------------------------ NOTE: config.h differs from original config.h, please check all settings. ------------------------------------------------------------------ --------------------------------------------------------- NOTE: see function read_config() in file thttpd.c for a list of options available in configuration file ( -C option ). --------------------------------------------------------- --------------------------------------------- NOTE: latest patch level includes all previous ones. --------------------------------------------- ==================== DEVELOPMENT BRANCH 3 - SHOULD WORK (since p44c) ==================== ----------------- thttpd-2.21b-p44e 24-Jul-2006 ----------------- Release Notes ------------- Minor fixes, added sendfile(v) tests to configure and configure.in. Configuration ------------- o configure.in, configure (A.D.F.) - added tests for native sendfile() (BSD, LINUX) and sendfilev() (SUN SOLARIS) in order to automatically add HAVE_*_SENDFILE or HAVE_*_SENDFILEV to generated Makefile. o config.h, thttpd.cfg (A.D.F.) - updated comments. o version.h (A.D.F.) - substituted SERVER_SOFTWARE_NAME for SERVER_NAME in order to let the command: make tar to work again as in official versions (it was broken in p28). Improvements ------------ o pipelining-detection (A.D.F.) - added function httpd_is_next_request() to improve the detection of pipelined requests. ----------------- thttpd-2.21b-p44d 02-Jun-2006 ----------------- Release Notes ------------- Minor tweaks plus added another parameter (-C configfile): - MAX_KEEPALIVE_FILE_SIZE. Added counter of connection overflows (too many connections). Configuration ------------- o config.h (A.D.F.) - added define: MAX_KEEPALIVE_FILE_SIZE to keep alive connections only if size of each requested file is equal or lower than MAX_KEEPALIVE_FILE_SIZE. o read_config (-C configuration-file) (A.D.F.) - added the following parameter: MAX_KEEPALIVE_FILE_SIZE=nBytes to close connection after an HTTP 200 OK response for a file whose size is bigger than MAX_KEEPALIVE_FILE_SIZE; - now only the first instance of -C option is executed. Improvements ------------ o thttpd-min-max-connections (A.D.F.) - tweaked lower limits for connections, now it should work smoothly even with a very low number of available file handles (i.e. 20-42). o thttpd_logstats (A.D.F.) - split longer lines to syslog info more nicely; - added stats_ovfconnects to count the number of connection overflows (too many connections). NOTE: connection overflows may be cured by increasing the number of files per process (see: ulimit -n). Fixes ----- o no-keepalive-with-cgi (A.D.F.) - if CGI is enabled then keepalive connections are not allowed. o close-on-exec-with-sendfile-and-cgi (A.D.F.) - now if sendfile is enabled along with cgi then the opened fds are closed on exec (execution of a CGI program). ----------------- thttpd-2.21b-p44c 28-Feb-2006 ----------------- Release Notes ------------- UNTESTED, anyway next patch will be tested in some way. Reformatted sample file thttpd.cfg and added more comments. Added new define / setting MAX_SENDFILE_BLK_SIZE. Improved / fixed a few things. Configuration ------------- o config.h (A.D.F.) - added define: MAX_SENDFILE_BLK_SIZE to allow the tuning of the max. block size of data sent by native sendfile() calls; this is useful because very small systems with slow disks, are very different from high end systems with ultra fast disks serving hundreds or thousands of parallel connections of big files whose total size is much bigger than RAM size. NOTE: take note that if files cannot be fully cached (i.e. they don't fit into RAM size) then thttpd performances will certainly suffer (in this case, if more than 2 GB of RAM are needed, then 64 bit machines are a very good solution). o read_config (-C configuration-file) (A.D.F.) - added the following parameter: MAX_SENDFILE_BLK_SIZE=nBytes to set max. size of sendfile blocks. o mmc.c (A.D.F.) - increased the expire age multiplier for L0 files from 100 to 120 times. Improvements ------------ o ignore-enotconn-errno (A.D.F.) - added ENOTCONN to the list of errno codes which are NOT syslogged after a write() or a sendfile() error (BSD work-around). o increased-max-http-request-length (A.D.F.) - this limit has been raised from 3 KB to 4.5 KB when EXECUTE_CGI is not defined (5.5 KB when it is defined), because "Referer:" HTTP field can be very large anyway. o tweaked-linger-time (A.D.F.) - increased dynamic linger time in order to avoid possible connection resets on very congested networks; - added a read_linger() call just before closing a keep-alive connection in order to handle very rare cases where read_linger() has not been called yet. Fixes ----- o tilde_map_1 (A.D.F.) - fixed the proper use of filename length. ----------------- thttpd-2.21b-p44b 27-Jan-2006 ----------------- Release Notes ------------- Completely UNTESTED, even if it should be better than previous version, there are still many details which are waiting to be checked, so, as usual, NO WARRANTY. Changes: - added a few more anti-crash tests; - now HTTP/1.0 "Expires:" header is sent only if request was HTTP/1.0; - now if sendfile is enabled (see USE_SENDFILE in -C thttpd.cfg) then the max. number of simultaneous connections is halved; - cgi POST method fix. Configuration ------------- o config.h (A.D.F.) - added define: SYSLOG_BEGEND_TOOMCONNS to write to syslog file both the BEGIN and the END message; by default it is undefined so that only the standard message "TOO MANY CONNECTIONS (num)" is written to syslog; this setting should help to spare space in syslog files. Performances ------------ o Expires-optimization (A.D.F.) - Expires: is sent only if request was HTTP/1.0 (feature enabled by USE_EXPIRES in config.h). Cleanups -------- o mmc-cleanup (A.D.F.) - sanitized usage of O_NOATIME and cfgOpenNoATime. - reset fd to -1 after close(), (not required but maybe useful to catch strange behaviours); - removed mmc_fdpurge(). o removed-NO_SMF (A.D.F.) - removed slow code sections. o add_responsefmt (A.D.F.) - removed calls to add_responsefmt() when NO_SMF is undefined. Improvements ------------ o early-close-on-high-load (A.D.F.) - moved the close logic before httpd_start_request() so that "Connection: close" HTTP header is sent with current response (not the next one). o mmc-up-limit-fd (A.D.F.) - added new parameter to set up-limit fd. Fixes ----- o httpd_parse_request (A.D.F.) - fixed the interpretation of HTTP protocol with more than one digit per major or minor version, i.e.: GET /index.html HTTP/1.12 o defang-check-upper-limit (A.D.F.) - fixed a very rare crash due to an incorrect test on upper limit of destination string. o handle-new-connect-fdmap (A.D.F.) - if USE_SENDFILE is set then the max. number of connections is halved in order to not run out of file descriptors. o mmc-fd-cache-shrink (A.D.F.) - now fd-cache should properly handle the up-limit of cached file descriptors. NOTE: DESIRED_MAX_OPENED_FILES is now ignored in favour of maxconnects (maxfiles / 2) in order to not consume too many fds. o cgi-child-cgi_interpose_input (Eric Malkowski) - now cgi_interpose_input() is always called when the method is POST, because a browser can first send only HTTP headers and then, after a small delay, the POST content. NOTE: indeed we should call cgi_interpose_input() only if Content-Length > 0, maybe this will be optimized in future versions of thttpd. ---------------- thttpd-2.21b-p44 27-Jun-2005 ---------------- Release Notes ------------- NOTE: this should be the last major patch based on thttpd-2.21b; a few minor subpatches (i.e. *p44b, *p44c, etc.) will follow to make things stable again. New features: - new HTTP/1.0 "Expires:" HTTP header (synchronization with official version 2.25b). Bug-Fixes, cleanups and micro-speed-ups: - timers-fix; - new mime types useful to modern browsers. NOTE: timers-fixes, if you use an older patch level (i.e. p33, p36, p43, etc.) you should at least upgrade timers.h and timers.c to this version and test whether thttpd is a bit more stable. A fix is in these patches: p33b, p36b, p44. Configuration ------------- o config.h (A.D.F.) - added define: USE_EXPIRES to send HTTP/1.0 Expires: header when max_age >= 0. - added define: SYSLOG_INDEXING_DISABLED to disable syslog messages about: "tried to index a directory but indexing is disabled". Mime-Types ---------- o mime-types-update (A.D.F.) - added 7 new mime types useful to modern browsers (Mozilla Suite, Mozilla FireFox, Opera, etc.). Log / Information ----------------- o log-max-age-value (A.D.F.) - added MAX_AGE to the parameters whose values are written to syslog at startup. o timers (A.D.F.) - added a few syslog(s) surrounded by TMR_DEBUG. HTTP-Headers ------------ o send_mime (Nick A.Smith + A.D.F.) - send "Expires:" header when USE_EXPIRES is defined and max_age >= 0 (now the HTTP cache headers should be almost the same as those found in thttpd-2.25b). HTTP-Connections ---------------- o read-linger-mode (A.D.F.) - set linger mode to ON by default because HTTP/1.1 is set to ON too; the new logic also fixes the case of "request too large" (before parsing the request); - added new function read_linger() in order to be used in handle_linger() and in handle_read() to empty input socket. HTTP Standard Compliance ------------------------ o HTTP-501-NO-POST (A.D.F.) - modified 501 handling in order to support METHOD_POST only if EXECUTE_CGI is defined. Bug-Fixes --------- o timers-fixes (A.D.F.) - tmr_run(), fixed a very rare and old bug (all thttpd-2.2x) which leaded to the improper use of timers placed in free list; - tmr_cancel(), tmr_reset(), added rough tests on hash value to prevent the use of removed timers (i.e. moved to free list), anyway these tests should not be needed in thttpd. Cleanups -------- o hash-mime (A.D.F.) - removed non standard macro _tolower(). o no-bgf-code (A.D.F.) - removed BGF code in function bufgets(). o occasional-idle (A.D.F.) - removed send-response in occasional_idle(), now when a send timeout happens then there is no attempt to complete the response. o mmc-functions (A.D.F.) - added a few more tests to enhance robustness. Performances ------------ o Micro-sparse-speedups (A.D.F.) - timers: - tmr_timeout, micro speedup; - tmr_mstimeout, micro speedup. ---------------- thttpd-2.21b-p43 17-Jan-2005 ---------------- Release Notes ------------- New features: - new HTTP/1.1 "max_age" option (partially back-ported from official version 2.25b). Stabilization efforts: - changes to pipelined requests handling; - sparse fixes aimed to fix very rare / random crashes; - micro strlen(s) changes and optimizations (speed-up), see below. Re-written sample configuration file (thttpd.cfg). Configuration ------------- o config.h (A.D.F.) - added define: DEFAULT_MAX_AGE to set a default max-age (HTTP/1.1 connections only) for all files served by thttpd. - added define: SYNC_FD_ON_CLOSE to enable fdwatch synchronization before every close of socket file descriptor (useful for Solaris /dev/poll fdwatch poller). - changed a few defaults: ERR_PAD_MSIE, ERR_HREF_SERVER_ADDRESS, OCCASIONAL_IDLE_TIME, etc. o command_line (parse_args) (Jef Poskanzer) - added option: -M max_age to set max_age seconds, used in HTTP/1.1 responses, see also: "Cache-Control:" HTTP header in RFC2616. o read_config (-C configuration-file) (Jef Poskanzer) - added the following parameter: MAX_AGE=nSeconds to set MAX_AGE seconds, used in HTTP/1.1 responses, see also: "Cache-Control:" HTTP header in RFC2616. Bug-Fixes --------- o sparse-anti-crash-fixes (A.D.F.) - httpd_initialize(), forced string termination; - httpd_realloc_str(), changed maxsize calculation; - defang(), substituted pointer subtraction for integer index; - expand_symlink(), forced string termination also in unusual / impossible case; - handle_read(), zero terminated read buffer at EOB (End Of Buffer) in order to allow proper use of fast string searches. HTTP Standard Compliance ------------------------ o HTTP-Connection-Token-Parsing (A.D.F.) - HTTP header "Connection:" could be followed (in theory) by more than one token, i.e.: Connection: Keep-Alive, Pipeline, etc. in this case server should be able to parse a list of tokens to keep itself compatible with future changes in HTTP specifications. NOTE: right now 99.99% of clients, proxies, etc. never send more than one token at a time (usually "Keep-Alive" or "Close") for each "Connection:" header. Mime-Types ---------- o mime-types-update (A.D.F.) - changed mime type for *.taz, *.tgz, etc. from x-compressed to octet-stream because usually they have to be downloaded, not showed by some decompressor / archiver. Performances ------------ o Micro-sparse-speedups (A.D.F.) - strdecode(): - added tests to use a fast copy of or to not copy at all source to target string when there are no special characters to decode. - tilde_map_1(): - removed strlen(s) by using and updating the length of modified strings. - tilde_map_2(): - removed strlen(s) by using and updating the length of modified strings. - vhost_map(): - removed strlen(s) by using and updating the length of modified strings. - httpd_parse_request(): - removed a few strlen(s) by keeping up-to-date the length of: - encodedurl; - decodedurl; - origfilename; - expnfilename. - lowered the number of string comparisons for HTTP headers for each HTTP request; - conditionally removed parsing of unused HTTP headers (in requests): - Accept; - Accept-Encoding; - Accept-Language. ---------------- thttpd-2.21b-p42 09-Dec-2004 ---------------- Release Notes ------------- Restored default padding of HTTP error messages for MSIE clients and added a new define in order to be able to disable this feature. Configuration ------------- o config.h (A.D.F.) - added define: ERR_PAD_MSIE to make built-in and custom HTTP error messages (i.e. 404 Not Found, etc.) conditionally bigger (if needed) than 512 bytes in order to force MSIE browsers to show server errors instead of their canned ones. This is accomplished by adding HTML comments to the body of messages. NOTE: till now this was the default behaviour, now you can choose to disable it (i.e. for performance reasons, specially when HTTP errors are very frequent). ---------------- thttpd-2.21b-p42rc1 23-Nov-2004 ---------------- Release Notes ------------- Maintenance enhancements: - sparse tweaks. Configuration ------------- o config.h (A.D.F.) - changed define: ALLOW_QRYSTR_IN_STFILES to DISALLOW_QRYSTR_IN_STFILES in order to be clearer about the fact that the default behaviour is to accept query strings also for static files because they can be used to implement trivial counters by tracking them in log file. i.e.: http://www.mydomain.com/mydir/myfile.jpg?param1=5 Bug-Fixes --------- o sparse-anti-crash-fixes (A.D.F.) - fixed a stupid hand-made bug added by last changes in httpd_realloc_str(); - fixed a possible compiler bug in httpd_realloc_str(); - removed a few const declarations as well; - added boundary checks to buffer operations; - increased size of a few temporary buffers. Error-Handling -------------- o numconnects (A.D.F.) - added more syslogs to check numfreeconnects and numconnects range values. o external-error-file-buffer (A.D.F.) - now only the first 2048 bytes are read from error files (errNNN.html). Improvements ------------ o handle_newconnect-trick (A.D.F.) - if at least one connection has been accepted in the accept loop and there are no more free connection slots, then return without triggering overflow code. ---------------- thttpd-2.21b-p41 12-Aug-2004 ---------------- Release Notes ------------- Maintenance enhancements: - fdwatch optimizations; - response optimizations. NOTE: kqueue is believed to work (changes not fully tested). Error-Handling -------------- o fdwatch (A.D.F.) - now a fatal error in fdwatch_sync() is remembered so that fdwatch() can return it (value -2 plus errno error code); NOTE: right know no caller checks for return codes of fdwatch_[add|mod|del]_fd()). NOTE: this change should force a thttpd exit on a fatal error that may happen when using kqueue, epoll or devpoll interfaces (anyway such a fatal error should never happen unless your system is seriously damaged). Performance ----------- o fdwatch (A.D.F.) - added fdwatch_mod_fd() to reduce the number of system calls (from 2 to 1) when using epoll interface; NOTE: fdwatch_mod_fd() is optimized to reset only client data if the new event is the same as the current (reduces FDW_READ / FDW_WRITE flip flop); WARNING: not fully tested (pure fun) ! - added fdwatch_get_fdw() to return the event state (FDW_NORW, FDW_READ, FDW_WRITE) of passed fd. o HTTP-Request-Response (A.D.F.) - small optimizations in date-time parser; - micro optimizations in HTTP response; - removed fdwatch flip/flop when response has no body (i.e. HEAD and GET 304) because we assume that a few hundred of bytes will be sent out fastly without delays (even in case of pipelined requests). ---------------- thttpd-2.21b-p40 26-Jul-2004 ---------------- Release Notes ------------- Maintenance enhancements: - new option USE_O_NOATIME (no update of access time) and NO_O_NOATIME options; - a few minor tweaks. Configuration ------------- o config.h (A.D.F.) - added new define USE_O_NOATIME to enable by default flag O_NOATIME (available since Linux 2.6.8.) in open file; this can be useful to not update access time (pico speed-up) when serving lots of different files accessed randomly (see also configuration file -C option). NOTE: probably it's much more efficient to mount web filesystem with option noatime because, in this case, also directories are not updated with access time. o read_config (-C configuration-file) (A.D.F.) - added the following parameters: USE_O_NOATIME NO_O_NOATIME to use or not use no access time flag in open file. Improvements ------------ o keep-alive-timeout (A.D.F.) - now keep alive timeout is assured to be reset only once per request, in any case. o disable-keep-alive-on-sigusr1 (A.D.F.) - disabled keep alive on SIGUSR1 in order to speed up graceful shutdown. ---------------- thttpd-2.21b-p39 21-Jun-2004 ---------------- Release Notes ------------- Maintenance enhancements: - timers workaround to buggy OS; - micro speed up in parsing HTTP requests. Workarounds ----------- o handle-system-time-changes (Ken Huffman) - try to handle an unexpected future time set in system clock and not filtered by OS (by monotonic increments); NOTE: this could happen on old / broken OSs; - timer activation is always set to a future time. Mime-Types ---------- o mime-types-update (A.D.F.) - added 2 new (image/x-sgi-*) mime types. Performances ------------ o tdate_parse (A.D.F.) - changed the order of date-time parsing to match first the most used Internet formats; - halved the number of sscanf(3) by interpreting more than one character separator at once, i.e.: ' ' and '-' instead of only ' ' or '-'; - added new time format "wdy mth DD HH:MM:SS YYYY" used only by very old (and maybe broken) browsers. ---------------- thttpd-2.21b-p38 24-May-2004 ---------------- Release Notes ------------- Maintenance enhancements: - new MIME types; - micro speed up in HTTP responses. Configuration ------------- o config.h (A.D.F.) - changed default values of IDLE_READ_LIMIT and MAX_KEEPALIVE_EXTRA_CRLFs. Mime-Types ---------- o mime-types-update (A.D.F.) - added 29 new mime types, as usual you can comment out those that are surely unused in mime_types.txt and recompile thttpd. NOTE: new compression extensions/types (i.e.: *.ace, etc.) should not be removed. NOTE: if you plan to never use a character set (i.e. thttpd -T ""), you can spare a few CPU cycles by removing "; charset=%s" (leave only first part, i.e.: "text/html") from entries in mime_types.txt and from config.h (then you have to recompile thttpd). Performances ------------ Miscellaneous micro / nano speedup(s): o init_mime (A.D.F.) - now mime type part "; charset=%s" is formatted only once, at startup, because "charset" value (option -T) does not change in main loop. o tmr_mstimeout (A.D.F.) - pico speed-up inside for-loop. o strlen-normalization (A.D.F.) - lowered number of strlen() calls. o HTTP-response-titles (A.D.F.) - removed a few global strings in favour of equivalent defines (constant length). o send_mime (A.D.F. + David Koblas) - optimized string usage; - moved format time string to fmt_rfc1123_time(). ---------------- thttpd-2.21b-p37 26-Apr-2004 ---------------- Release Notes ------------- Some more micro-speedups and new options: - new defines in config.h; - new options to change default receive / send size of socket buffer(s); - removed use of strcpy() with overlapping memory between source and destination (no problem up to now but some hacked / optimized C libraries could use some clever trick able to break things); - micro speedup(s). Configuration ------------- o config.h (A.D.F.) - added new define CONN_SO_RCVBUF to set max. receive size of socket buffer (see also configuration file -C option); - added new define CONN_SO_SNDBUF to set max. send size of socket buffer (see also configuration file -C option); - added new define ERR_HREF_SERVER_ADDRESS to conditionally add an hyperlink on SERVER_SOFTWARE to SERVER_ADDRESS (this has always been the default setting); - added new define LOG_PREPEND_VHOSTNAME to prepend (A.D.F. + vhostname to all other fields in log record Emmanuel Hocdet + instead of prepending it to encoded url. David Phillips) NOTE: this is useful if server does virtual hosting, in order to make log file analysis a bit easier. o read_config (-C configuration-file) (A.D.F.) - added the following parameters: CONN_SO_RCVBUF CONN_SO_SNDBUF to change default receive / send size of socket buffer(s), useful to tune buffer usage for Internet servers, i.e.: CONN_SO_RCVBUF=4096 CONN_SO_SNDBUF=16384 NOTE: the idea is that receive buffer can be much smaller than send buffer when serving static only contents (no CGI, etc.); NOTE: tuning is required to set optimal values, your mileage may vary; NOTE: we assume that accepted sockets inherit these values from listening socket. Tuning ------ o high-water-mark-4 (A.D.F.) - lowered max. difference between high water mark 4 and max. connection limit from 512 to 256. Fixes ----- o http-layout (Mihai Rusu) - restored layout functionality (broken since p25) (strcmp() -> strncmp()). o overlapping-strcpy (Adam Zell + A.D.F.) - substituted all occurrencies of strcpy(), using overlapping memory, for ovl_strcpy(dst, src) or memmove(dst, src, len), in order to work around undefined behaviour of standard strcpy() when source and destination memory overlap. NOTE: this should make "valgrind" happy. Error Handling -------------- o accept-aborted (Jef Poskanzer + - if ECONNABORTED is defined as an errno error code Nicholas Gimbrone) (*BSD systems), then an aborted connection in listen queue (accept) only increments a statistic counter (no synchronous syslog output). NOTE: this should be useful to not flood your syslog if you have enabled an accept filter and thousands of clients are making a DoS attack to your *BSD server. o HTTP-No-Err-On-CRLF2 (A.D.F.) - don't send 400 error on incomplete first HTTP line ending with a CRLF around first word. Performances ------------ Miscellaneous micro / nano speedup(s): o add_responselen (A.D.F.) - new function to not call strlen() when adding constant/literal strings to HTTP headers. o add_responsefmt (A.D.F. + Rob Ekl) - new function to avoid a copy from temporary buffer to response buffer when HAVE_VSNPRINTF is defined. o send_mime (A.D.F.) - substituted a few calls to add_response() for add_responsefmt() or add_responselen(); - moved allowed_methods to a separate function. o send_response_tail (A.D.F.) - removed a call to my_snprintf() because string concatenation does the same job (a C ANSI compiler is already required anyway). o make_log_entry (A.D.F. + David Koblas) - removed a couple of calls to my_snprintf(); - moved format time string to fmt_cern_time(). Cleanups -------- o CRLF_define (A.D.F.) - started using defines for HTTP CR-LF characters instead of hardcoded '\r', '\n' or "\r\n". o conditionally-declare-remoteuser-field (A.D.F.) - if AUTH_FILE is not defined, then "remoteuser" is never set with a non null value, thus we can safely #ifdef its declaration, its allocation and its initialization. ==================== DEVELOPMENT BRANCH 2 - STABLE ==================== ----------------- thttpd-2.21b-p36h 02-Jun-2006 ----------------- Release Notes ------------- Minor tweaks plus added another parameter (-C configfile): - MAX_KEEPALIVE_FILE_SIZE. Added counter of connection overflows (too many connections). Configuration ------------- o config.h (A.D.F.) - added define: MAX_KEEPALIVE_FILE_SIZE to keep alive connections only if size of each requested file is equal or lower than MAX_KEEPALIVE_FILE_SIZE. o read_config (-C configuration-file) (A.D.F.) - added the following parameter: MAX_KEEPALIVE_FILE_SIZE=nBytes to close connection after an HTTP 200 OK response for a file whose size is bigger than MAX_KEEPALIVE_FILE_SIZE; - now only the first instance of -C option is executed. Improvements ------------ o thttpd-min-max-connections (A.D.F.) - tweaked lower limits for connections, now it should work smoothly even with a very low number of available file handles (i.e. 20-42). o tdate-parse-test-optimization (A.D.F.) - added a test for empty date-time. o thttpd_logstats (A.D.F.) - split longer lines to syslog info more nicely; - added stats_ovfconnects to count the number of connection overflows (too many connections). NOTE: connection overflows may be cured by increasing the number of files per process (see: ulimit -n). Fixes ----- o no-keepalive-with-cgi (A.D.F.) - if CGI is enabled then keepalive connections are not allowed. ----------------- thttpd-2.21b-p36g 28-Feb-2006 ----------------- Release Notes ------------- Backported from p44c last missing setting (-C configfile): - MAX_SENDFILE_BLK_SIZE. NOTE: now settings in p44c and p36g should be the same. Micro speed-up (+5% - 7%) in parsing date fields in HTTP requests. Configuration ------------- o config.h (A.D.F.) - added define: SYNC_FD_ON_CLOSE to enable fdwatch synchronization before every close of socket file descriptor (useful for Solaris /dev/poll fdwatch poller). - added define: MAX_SENDFILE_BLK_SIZE to allow the tuning of the max. block size of data sent by native sendfile() calls; this is useful because very small systems with slow disks, are very different from high end systems with ultra fast disks serving hundreds or thousands of parallel connections of big files whose total size is much bigger than RAM size. NOTE: take note that if files cannot be fully cached (i.e. they don't fit into RAM size) then thttpd performances will certainly suffer (in this case, if more than 2 GB of RAM are needed, then 64 bit machines are a very good solution). o read_config (-C configuration-file) (A.D.F.) - added the following parameter: MAX_SENDFILE_BLK_SIZE=nBytes to set max. size of sendfile blocks. o mmc.c (A.D.F.) - increased the expire age multiplier for L0 files from 100 to 120 times. Performances ------------ o tdate_parse (A.D.F.) - changed the order of date-time parsing to match first the most used Internet formats; - halved the number of sscanf(3) by interpreting more than one character separator at once, i.e.: ' ' and '-' instead of only ' ' or '-'; - added new time format "wdy mth DD HH:MM:SS YYYY" used only by very old (and maybe broken) browsers. ----------------- thttpd-2.21b-p36f 20-Feb-2006 ----------------- Release Notes ------------- Applied fixes and optimizations regarding fd cache in order to not run out of fds. Backported from p44b a few settings (-C configfile): - USE_O_NOATIME; - NO_O_NOATIME; - CONN_SO_RCVBUF; - CONN_SO_SNDBUF. Configuration ------------- o config.h (A.D.F.) - added new define USE_O_NOATIME to enable by default flag O_NOATIME (available since Linux 2.6.8.) in open file; this can be useful to not update access time (pico speed-up) when serving lots of different files accessed randomly (see also configuration file -C option). NOTE: probably it's much more efficient to mount web filesystem with option noatime because, in this case, also directories are not updated with access time. - added new define CONN_SO_RCVBUF to set max. receive size of socket buffer (see also configuration file -C option); - added new define CONN_SO_SNDBUF to set max. send size of socket buffer (see also configuration file -C option); o read_config (-C configuration-file) (A.D.F.) - added the following parameters: USE_O_NOATIME NO_O_NOATIME to use or not use no access time flag in open file. - added the following parameters: CONN_SO_RCVBUF CONN_SO_SNDBUF to change default receive / send size of socket buffer(s), useful to tune buffer usage for Internet servers, i.e.: CONN_SO_RCVBUF=4096 CONN_SO_SNDBUF=16384 NOTE: the idea is that receive buffer can be much smaller than send buffer when serving static only contents (no CGI, etc.); NOTE: tuning is required to set optimal values, your mileage may vary; NOTE: we assume that accepted sockets inherit these values from listening socket. Cleanups -------- o mmc-cleanup (A.D.F.) - removed mmc_fdpurge(). Improvements ------------ o ignore-enotconn-errno (A.D.F.) - added ENOTCONN to the list of errno codes which are NOT syslogged after a write() or a sendfile() error (BSD work-around). o mmc-up-limit-fd (A.D.F.) - added new parameter to set up-limit fd. o tweaked-linger-time (A.D.F.) - increased dynamic linger time in order to avoid possible connection resets on very congested networks; - added a read_linger() call just before closing a keep-alive connection in order to handle very rare cases where read_linger() has not been called yet. Fixes ----- o mmc-fd-cache-shrink (A.D.F.) - now fd-cache should properly handle the up-limit of cached file descriptors. NOTE: DESIRED_MAX_OPENED_FILES is now ignored in favour of maxconnects (maxfiles / 2) in order to not consume too many fds. o cgi-child-cgi_interpose_input (Eric Malkowski) - now cgi_interpose_input() is always called when the method is POST, because a browser can first send only HTTP headers and then, after a small delay, the POST content. NOTE: indeed we should call cgi_interpose_input() only if Content-Length > 0, maybe this will be optimized in future versions of thttpd. ----------------- thttpd-2.21b-p36e 13-Jan-2006 ----------------- Release Notes ------------- Fixed a bug added in p36d plus some small improvements. Improvements ------------ o handle_newconnect-fd-check (A.D.F.) - added a conditional check before removal of listen fd. o read-linger-mode (A.D.F.) - added new function read_linger() in order to be used in handle_linger() and in handle_read() to empty input socket thus preventing the reset of connection when the request is too large. Fixes ----- o httpd_parse_request (A.D.F.) - fixed the interpretation of HTTP protocol with more than one digit per major or minor version, i.e.: GET /index.html HTTP/1.12 o defang-check-upper-limit (A.D.F.) - fixed a very rare crash due to a bug added in previous patch p36d. ----------------- thttpd-2.21b-p36d 12-Sep-2005 ----------------- Release Notes ------------- Small fixes to max. number of connections when USE_SENDFILE is enabled. Configuration ------------- o config.h (A.D.F.) - added define: SYSLOG_BEGEND_TOOMCONNS to write to syslog file both the BEGIN and the END message; by default it is undefined so that only the standard message "TOO MANY CONNECTIONS (num)" is written to syslog, thus helping to spare space in syslog files. - added define: HTTP_RETRY_AFTER_SEC to send a Retry-After HTTP header after a 503 error (too many connections). Fixes ----- o handle-new-connect-fdmap (A.D.F.) - if USE_SENDFILE is set then the max. number of connections is halved in order to not run out of file descriptors; also handle the case when there are many connections and spare fds have to be purged (closed). Improvements ------------ o early-close-on-high-load (A.D.F.) - moved the close logic before httpd_start_request() so that "Connection: close" HTTP header is sent with current response (not the next one). o handle_newconnect-trick (A.D.F.) - if at least one connection has been accepted in the accept loop and there are no more free connection slots, then return without triggering overflow code. o 503-retry-after (A.D.F.) - conditionally added a Retry-After HTTP header after a 503 HTTP error. ----------------- thttpd-2.21b-p36c 15-Jul-2005 ----------------- Release Notes ------------- New features: - new HTTP/1.1 "max_age" option (back-ported from official version 2.25b and from p44). Bug-Fixes: - exit fastly after SIGUSR1. Configuration ------------- o config.h (A.D.F.) - added define: DEFAULT_MAX_AGE to set a default max-age (HTTP/1.1 connections only) for all files served by thttpd. - added define: USE_EXPIRES to send HTTP/1.0 Expires: header when max_age >= 0. o command_line (parse_args) (Jef Poskanzer) - added option: -M max_age to set max_age seconds, used in HTTP/1.1 responses, see also: "Cache-Control:" HTTP header in RFC2616. o read_config (-C configuration-file) (Jef Poskanzer) - added the following parameter: MAX_AGE=nSeconds to set MAX_AGE seconds, used in HTTP/1.1 responses, see also: "Cache-Control:" HTTP header in RFC2616. Log / Information ----------------- o log-max-age-value (A.D.F.) - added MAX_AGE to the parameters whose values are written to syslog at startup. Bug-Fixes --------- o SIGUSR1-fast-exit (A.D.F.) - disable keep-alive when a SIGUSR1 is catched. HTTP Standard Compliance ------------------------ o HTTP-Connection-Token-Parsing (A.D.F.) - HTTP header "Connection:" could be followed (in theory) by more than one token, i.e.: Connection: Keep-Alive, Pipeline, etc. in this case server should be able to parse a list of tokens to keep itself compatible with future changes in HTTP specifications. NOTE: right now 99.99% of clients, proxies, etc. never send more than one token at a time (usually "Keep-Alive" or "Close") for each "Connection:" header. ----------------- thttpd-2.21b-p36b 20-Jun-2005 ----------------- Release Notes ------------- Bug-Fixes: - tmr_run(), fixed old standing bug. Bug-Fixes --------- o timers-fix (A.D.F.) - tmr_run(), fixed a very rare and old bug (all thttpd-2.2x) which leaded to the improper use of timers placed in free list. o http-layout (Mihai Rusu) - restored layout functionality (broken since p25) (strcmp() -> strncmp()). ---------------- thttpd-2.21b-p36 11-Mar-2004 ---------------- Release Notes ------------- Fixed an ugly bug, added since p34, in connection expiring and applied a few small cleanups. NOTE: if you are using p34 or p35 then it is RECOMMENDED to upgrade to this patch level ASAP. Configuration ------------- o config.h (A.D.F.) - added new define ALLOW_QRYSTR_IN_STFILES to allow CGI style URLs (query string + path info) also for static files, i.e.: http://www.mydomain.com/mydir/myfile.html?param1=5 - added new define SYSLOG_TOTCNT_CONNTMO to syslog total count of timed out connections independently from SYSLOG_EACH_CONNTMO define. Bug-Fixes --------- o occasional_idle-crash (Mihai Rusu) - fixed a bug (badly initialized variable) added to p34 by A.D.F. (it could crash the server after a few seconds after startup and/or on connection send timeout). Fixes ----- o http-error-titles (A.D.F.) - fixed association between http error 505 and its proper description in: - http_err_title(); - http_err_form(). NOTE: this micro-bug was harmless. ---------------- thttpd-2.21b-p35 01-Mar-2004 ---------------- Release Notes ------------- Micro speed-up(s) in connection handling: - sparse micro-tweaks; - new command line option -dd; - check for existence of log path after -d dir, -dd data_dir chdir options. In short, another step ahead, as usual use with care. Configuration ------------- o config.h (A.D.F.) - added new "experimental" define, INHERIT_FD_NONBLOCK_AA, to not set non-blocking I/O mode again after accept (on some OS I/O mode of new accepted socket connections is inherited from listening socket, thus, in such a case, setting it once in listening socket should suffice). NOTE: define (temporarily) -DTEST_INHERIT_FD_NONBLOCK in generated Makefile and look at syslog messages to find out if I/O mode inheritance is true or false. HINT: no need to try this if you don't love risky things. o command-line (Jef Poskanzer) - added option -dd data_dir (ex 2.25b). o read_config (-C configuration-file) (Jef Poskanzer) - added option data_dir, i.e.: data_dir=/usr/local/www/htdocs New Features ------------ o logfile-path-trim-and-test (Jef Poskanzer + A.D.F.) - if thttpd does chroot (-r option) and logfile is an absolute path then common initial path is elided in order to be able to reopen log file after a chroot, i.e.: cd /usr/local/www thttpd -l /usr/local/www/log/thttp.log -r "/usr/local/www/log/thttpd.log" (first open) "/log/thttpd.log" (trimmed path after chroot) - if thttpd is started as root, then logfile owner is changed to specified user; - at startup thttpd checks if logfile path exists after a chroot and/or changing directory (see: -d dir, -dd data_dir options); if logfile path is not found then a warning (telling that thttpd will not be able to reopen logfile) is printed. NOTE: the order of operations is: - open log file; - change directory to dir (-d option); - chroot (-r option); - change directory to data_dir (-dd option); - etc. o too-many-connections-control (A.D.F.) - now, if SYSLOG_EACH_TOOMCONNS is not defined, then listen socket is removed from watched set of socket descriptors when the maximum number of connections is reached; it is then re-added only when the number of connections is less than HIGH WATER MARK 4 (95%). o tcp_cork-sendfile (A.D.F.) - added set / unset of TCP_CORK mode (Linux 2.4) when sending files via httpd_sendfile() and there are no throttles; this setting should lower the number of partial TCP/IP segments sent when there are many (more than 40-50) big downloads (*) waiting for disk I/O; (*) whose total size is much bigger than available RAM. NOTE: *BSD TCP_NOPUSH is not used because this option was meant for T-TCP/IP and because it behaves differently between newer and older *BSD versions (i.e. older versions don't send last partial TCP segment when the option is cleared because they expect more data before doing this). Performances ------------ o main-loop (A.D.F.) - split main loop into two loops to lower the number of jumps done inside fast path (I know this is only an unuseful / dumb try); - removed a possibly redundant fdwatch_check_fd() call. o keep-alive-speed-up (A.D.F.) - don't wait for write socket buffer to become empty when sent bytes are less than 8KB. o linger-close (A.D.F.) - reordered system calls to get (in theory) a nano speed up under load; - tweaked linger_time to lower it when DYNAMIC_LINGER_TIME is defined and the connection is closed after first request (clients are supposed to not send pipelined requests until first server response is received). Cleanups -------- o really_start_request (A.D.F.) - cosmetic cleanups to regular-file check. o clear_connection (A.D.F.) - updated comments. o fcntl-FD_CLOEXEC (A.D.F.) - replaced 1 with FD_CLOEXEC in some fcntl(2) F_SETFD calls where it was missing (there was no problem anyway, because FD_CLOEXEC has always been defined as 1). Changes ------- o libhttpd-new-get-set-functions (A.D.F.) - added the following functions: - httpd_get_nonblock(); - httpd_set_nonblock(); - httpd_get_nagle(); - httpd_set_nagle(); - httpd_get_cork(); - httpd_set_cork(). - removed the following functions: - httpd_set_ndelay(); - httpd_clear_ndelay(); because suffix "ndelay" has been replaced with suffix "nonblock" which, IMHO, is more meaningful than no-delay mode, in order to indicate non-blocking I/O (i.e. a read may block forever waiting for new data to arrive, in this case no-delay is not a proper term to use). ---------------- thttpd-2.21b-p34 28-Jan-2004 ---------------- Release Notes ------------- Compile fix. Compile Fixes ------------- o fdwatch-kqueue (Yusuf Goolamabbas) - kqueue_sync() call was in the wrong place; this has been fixed by placing it inside HAVE_KQUEUE ifdefs. ---------------- thttpd-2.21b-p34 22-Jan-2004 ---------------- Release Notes ------------- Compile fix + fix for high CPU usage. Fixes ----- o idle-timer-callback (Gary Stanley + A.D.F.) - fixed periodic activation of idle timer, it was called too often (high CPU usage) because seconds were not multiplied by 1000 (sic). Compile Fixes ------------- o fdwatch-kqueue (Gary Stanley) - added a trailing semicolon at line 589 (fdwatch.c). ---------------- thttpd-2.21b-p34 16-Jan-2004 ---------------- Release Notes ------------- Small scalability improvements: - no unnecessary timers for connection timeouts; - no unnecessary syslogs. Configuration ------------- o config.h (A.D.F.) - inverted meaning of LIMIT_TOOMCONNS_SYSLOG by renaming it to SYSLOG_EACH_TOOMCONNS; - added new define, SYSLOG_EACH_CONNTMO, to call syslog(3) for each timed out connection instead of only a total count every OCCASIONAL_IDLE_TIME seconds. Scalability ----------- o periodic-connection-timer-expiring (Sascha Schumann + - removed usage of read / write timers, Jef Poskanzer) excepted for pause and linger time; now a periodic scan of connections set is performed in order to enhance performances / scalability when using thousands of active connections. ==================== DEVELOPMENT BRANCH 1 - OBSOLETE ==================== ----------------- thttpd-2.21b-p33c 15-Jul-2005 ----------------- Release Notes ------------- New features: - new HTTP/1.1 "max_age" option (back-ported from official version 2.25b and from p44). Bug-Fixes: - exit fastly after SIGUSR1. Configuration ------------- o config.h (A.D.F.) - added define: DEFAULT_MAX_AGE to set a default max-age (HTTP/1.1 connections only) for all files served by thttpd. - added define: USE_EXPIRES to send HTTP/1.0 Expires: header when max_age >= 0. o command_line (parse_args) (Jef Poskanzer) - added option: -M max_age to set max_age seconds, used in HTTP/1.1 responses, see also: "Cache-Control:" HTTP header in RFC2616. o read_config (-C configuration-file) (Jef Poskanzer) - added the following parameter: MAX_AGE=nSeconds to set MAX_AGE seconds, used in HTTP/1.1 responses, see also: "Cache-Control:" HTTP header in RFC2616. Log / Information ----------------- o log-max-age-value (A.D.F.) - added MAX_AGE to the parameters whose values are written to syslog at startup. Bug-Fixes --------- o SIGUSR1-fast-exit (A.D.F.) - disable keep-alive when a SIGUSR1 is catched. HTTP Standard Compliance ------------------------ o HTTP-Connection-Token-Parsing (A.D.F.) - HTTP header "Connection:" could be followed (in theory) by more than one token, i.e.: Connection: Keep-Alive, Pipeline, etc. in this case server should be able to parse a list of tokens to keep itself compatible with future changes in HTTP specifications. NOTE: right now 99.9% of clients, proxies, etc. never send more than one token at a time (usually "Keep-Alive" or "Close") for each "Connection:" header. ----------------- thttpd-2.21b-p33b 20-Jun-2005 ----------------- Release Notes ------------- Bug-Fixes: - tmr_run(), fixed old standing bug. Bug-Fixes --------- o timers-fix (A.D.F.) - tmr_run(), fixed a very rare and old bug (all thttpd-2.2x) which leaded to the improper use of timers placed in free list. o http-layout (Mihai Rusu) - restored layout functionality (broken since p25) (strcmp() -> strncmp()). o http-error-titles (A.D.F.) - fixed association between http error 505 and its proper description in: - http_err_title(); - http_err_form(). NOTE: this micro-bug was harmless. ---------------- thttpd-2.21b-p33 26-Jan-2004 ---------------- Release Notes ------------- Only compile fixes for fdwatch. ---------------- thttpd-2.21b-p33 09-Jan-2004 ---------------- Release Notes ------------- Only compile fixes for CGI code compiled when EXECUTE_CGI is defined. ---------------- thttpd-2.21b-p33 05-Jan-2004 ---------------- Release Notes ------------- New features and small bug fixes: - new IP throttles handling; - sparse micro tweaks and fixes. Configuration ------------- o config.h (A.D.F.) - added new define, USE_IPTHROTTLE, to enable IPv4 throttles (see below). o configuration-file (Emmanuel Hocdet) - added these two new parameters: ipthrottles=IpThrottlesFileName subnets=SubIpNetsFileName o mime_types.txt (A.D.F.) - added another extension to mime types: *.cab. New Features ------------ o ip-throttles (Emmanuel Hocdet) - added a new type of throttle based on IP interfaces and addresses; by now it works only with IPv4 addresses (IPv6 addresses are ignored, no IP throttling for them). Format of "ipthrottles" configuration file is: IpClassName upperThrottleLimitBytes i.e.: lan 6000000 peer 3000000 out 1000000 out is the default class used when others don't match. Format of "subnets" configuration file is: IPv4DotStartAddress/IPv4Last3DigitsMask IpClassName i.e.: 194.127.200.0/21 lan 195.136.216.0/23 peer 213.34.0.0/18 peer 214.44.160.0/19 lan 0.0.0.0/0 out # default rule if nothing else matches Above rules associate an IP class defined in IPv4ThrottlesFile with an IP subnet. NOTE: IPv6 addresses are not supported. NOTE: subnets overlap is not supported. NOTE: subnets file can be reloaded by raising USR2 signal. Performances ------------ o stat-call-reduction (David Koblas) - don't call stat() twice on the same path when chroot is enabled, there are no symbolic links and the request has no CGI path and/or parameters (+1% - 2% faster at max request rate). Fixes ----- o init-before-chroot (Jef Poskanzer) - initialize fdwatch package before chroot (useful if /dev/poll is used). o retry-interrupted-syscalls (A.D.F.) - now syscalls interrupted by a signal (errno == EINTR) are always restarted; NOTE: this should be useful to not close any connection when a user signal arises (i.e.: SIGUSR1, SIGUSR2, etc.). o fdwatch-select-syslog (A.D.F.) - added a missing argument to a syslog() call in function select_add_fd() (my bad). o fdwatch-kqueue-compile-fixes (Emmanuel Hocdet) - corrected a few define names and return type (void -> int) of a few functions. o limit-too-many-connections-syslogs (Christian Kagerhuber) - reset ovfconnects to 0 only *after* its value has been printed by syslog. o ls-no-truncated-defanged-names (A.D.F.) - don't print truncated defanged names because in this case html entities maybe incomplete. Changes ------- o throttle-send-rate-changes (E. Frank Ball + - partial resync throttle code with thttpd-2.25b; Jef Poskanzer + - added slow start for new throttled requests; A.D.F.) - added smart algorithm to lower send rate in one shot and raise it gradually in 1 - 16 times; - added start throttle time and sent bytes since previous throttle update in update_throttles(), in order to avoid, under corner conditions A) download stalls or B) download speed up over throttle limit. NOTE: now, after above changes, THROTTLE_TIME should not be set too low or too high, i.e.: recommended values are 4 or 8 seconds; NOTE: now send rate may suffer some "wow & flutter" because there is less sampling time to throttle downloads. Micro tweaks ------------ o new-upper-limit-for-tcp-port-number (A.D.F.) - set upper limit to 65535. o parse-request (David Leadbeater) - added a useful check for a leading dot character in order to never accept hostnames starting with a dot; NOTE: this is not a security fix, last security fixes were already applied in p27. o send_response (A.D.F.) - tweaked padding for MSIE in send_response(). o stats-body-bytes (A.D.F.) - now statistic counter for sent body contents, is updated after every handle_send() instead of only at the end of a download; this should give smoother statistics because total count does not increase suddenly at the end of big downloads (i.e. files 10 - 1000 MB). o const-args (A.D.F.) - added const qualifier to a few arrays and function parameters / return types (httpd_method_str(), httpd_send_err*(), need_defang(), defang(), etc.). Package Changes --------------- o fdwatch (A.D.F. + Adam Zell) - removed code used to collapse multiple add/del events in kqueue_add_fd(), kqueue_del_fd(), devpoll_add_fd(), devpoll_del_fd(); - added kqueue_sync() to flush array "change events" in order to properly handle corner cases, i.e. when all file descriptors are ready and there are also many pipelined requests; - changed devpoll_sync() to handle partial writes (I don't know if this can happen, but in case it might work); - changed select_watch() to make it slightly faster (Adam Zell) in some corner cases, i.e.: when there are only a few fds with high values (fd1 == 200, fd2 == 250, etc.); in practice in such a case there should be fewer loop iterations. ---------------- thttpd-2.21b-p32 25-Nov-2003 ---------------- Release Notes ------------- Some new features and some small work-arounds: - finer granularity for optimization level of C compiler; - improved compatibility with buggy HTTP/1.1 clients; - N. 12 new mime types; - new fdwatch interface "epoll" (not tested). Configuration ------------- o configure.in + configure (A.D.F.) - use gcc -dumpversion instead of gcc --version in order to get only dotted digits (without too many string details); - added gcc major version and gcc minor version in order to set proper optimization level ( -O, -O1, -O2 ), depending on the version of compiler. NOTE: most gcc 2.x and early 3.x versions may compile badly when using -O2; NOTE: make distclean; ./configure or even re-run autoconf(1) if it's not too recent. o config.h (A.D.F.) - added new define to ignore at most MAX_KEEPALIVE_EXTRA_CRLFs line terminator characters between two HTTP/1.1 requests, in order to be tolerant against buggy HTTP/1.1 clients (see following HTTP-Skip-Leading-CRLFs). - added new define to limit the number of syslogs when there are too many connections, LIMIT_TOOMCONNS_SYSLOGS. NOTE: this could be useful under a DoS attack. o mime_types.txt (A.D.F.) - added other 2 mime types for these extensions: rdf and xul; HINT: now it's a good time to start writing xul applets for Mozilla suite; SEE ALSO: http://www.oreillynet.com/mozilla/ - added other 10 mime types for these extensions: pls, mpeg, mpm, m75, moov, m15, m1s, m1v, vfw, vcd. HTTP Standard Compliance ------------------------ o HTTP-proper-version (A.D.F.) - if a request is malformed (400) then reply with highest supported version (HTTP/1.1) by thttpd. o HTTP-Skip-Leading-CRLFs (Nicholas Gimbrone + - for the sake of compatibility with Wojtek Meler + A.D.F) buggy HTTP/1.1 clients, now extra / superfluous CRLFs, preceding HTTP requests, are ignored. NOTE: Nicholas is named first because he warned us about this issue over a year ago. NOTE: no leading CRLFs are skipped for the first request of each connection. WARNING: MSIE CLIENTS older than MSIE 5 (i.e. MSIE 4.x) MUST disable HTTP/1.1 (use HTTP/1.0) protocol anyway because they are really too BUGGY! Overload Protection ------------------- o limit-too-many-connections-syslogs (Christian Kagerhuber) - defining LIMIT_TOOMCONNS_SYSLOGS enables a syslog protection to limit the number of syslogs called under extreme overload (i.e. a DoS attack). o run-timers-after-newconn-error (Alex Keahan) - run timers after an error occured in httpd_get_conn(). Fixes ----- o limit-sendfile-block-size (A.D.F.) - added a rough limit to the size of chunk data sent by native sendfile(); this should prevent semi-blocking calls when sending big files and network I/O is much faster than disk I/O (i.e. in a fast LAN >= 100 Mbit) and there are no other processes competing hard for system resources (CPU / IO). Log Changes ----------- o lengthy-user-agent-names (Jef Poskanzer) - allow lengthy user agent names in log file (upper limit has been increased from 80 to 200 characters). Package Changes --------------- o fdwatch (Christian Kagerhuber + - added "epoll" interface for Linux 2.6. A.D.F.) NOTE: slightly better performances with thousands of opened connections. NOTE: this feature should be considered EXPERIMENTAL (not tested). ---------------- thttpd-2.21b-p31 18-Sep-2003 ---------------- Release Notes ------------- Small performance improvements and new fdwatch interface "/dev/poll" + fixes. Configuration ------------- o configure.in + configure (A.D.F.) - fixed identification of some old compiler versions, i.e.: "egcs-2.91.66"; NOTE: "configure" script expected only dotted numbers, i.e.: "2.91.66", while now it can skip alphabetic chars before major version number; - changed gcc optimization level, now -O2 is activated only with a gcc major version >= 3; this is the right thing to do because most 2.x versions (excepted for gcc-2.95.[34] and some heavily patched gcc-2.96-xx RH sub-versions) have subtle bugs with -O2 (these bugs are really rare, but you can't bet they don't show up); NOTE: we also assume that nobody with a sane mind will use gcc versions between 3.0.x and 3.1.x (buggy). - added test to discover the presence of devpoll.h ("/dev/poll") header file. NOTE: SunOS users may want to run autoconf(1) after having applied this patch and after having backed up the original patched "configure" file. o config.h (A.D.F.) - added USE_ACCEPT_FILTER to enable or disable an accept filter (data ready) for incoming connections; this should be available in FreeBSD 4.x and Linux 2.4; NOTE: be sure to use a non broken (patched or really new) kernel before defining this symbol (i.e. Linux 2.4.18 does not behave correctly when using an accept filter with IPv6 sockets); - added DYNAMIC_LINGER_TIME to enable or disable automatic adjustment of LINGER_TIME value. Fixes ----- o SIGUSR1-shutdown (A.D.F.) - struct httpd_server is no more fred (by httpd_terminate) inside connection loop; NOTE: httpd_server must not be fred because it is used/referenced by active connections, i.e.: hc->hs. - server sockets are removed from the set of active file descriptors being watched before they are closed (by httpd_unlisten). o fdwatch.c (Emmanuel Hocdet) - removed unuseful syslog() in fdwatch_check_fd() to avoid syslog() floods. o thttpd.c (A.D.F.) - commented fdwatch_sync() in really_clear_connection() hopefully everything should work the same, only a bit faster. HTTP Standard Compliance ------------------------ o HTTP-Version (A.D.F.) - improved HTTP version parsing, now extraneous / invalid characters always lead to an error response (well, in some cases error response should be 400 Bad Request, not 505 HTTP Version not supported, but we are lazy and, by now, this is acceptable). Cleanups -------- o int-versus-size_t (Emmanuel Hocdet) - changed a few size_t types to int in libhttpd.c (send_redirect(), send_dirredirect()) to allow compilation in 64 bit environments. o make_log_entry (A.D.F.) - changed max url size in format string for syslog(), from %.200s to %.300s. Micro Optimizations ------------------- o hash-mime (Mihai Rusu) - added #ifdef to speed up computation when _tolower() or tolower() are defined as macros. Performances ------------ o httpd_server-nowtime (A.D.F.) - now current server time is assigned to field "nowtime" of httpd_server struct. o send_mime (A.D.F.) - small performance improvement (+3% - 5% at max. request rate); o timers-hash-size (A.D.F.) - increased HASH_SIZE from 67 to 101; NOTE: this is a temporary trick to slightly enhance performances when there are many connections (more than 400 - 500 connections). Portability ----------- o libhttpd-sendfile (Emmanuel Hocdet) - added native sendfilev() call for Solaris to httpd_sendfile() in order to emulate a missing sendfile() (it seems like Sun considers sendfile() to be obsoleted by sendfilev()); as usual you can enable BSD, Linux or Solaris support defining ONLY ONE of these definitions in Makefile: -DHAVE_BSD_SENDFILE=1 -DHAVE_LINUX_SENDFILE=1 -DHAVE_SOLARIS_SENDFILE=1 -DHAVE_SOLARIS_SENDFILEV=1 NOTE: if you do so, you have to recompile thttpd: make clean; make; NOTE: only Linux sendfile() has been tested, BSD and Solaris sendfile(v)() are believed to work; NOTE: on BSD and Solaris, please, read sendfile or sendfilev man page to find out library requirements; probably you need to add something like -lsendfile, i.e.: LIBS = -lcrypt -lsendfile in generated Makefile (after each ./configure); NOTE: using sendfile() with medium sized files is a win only if your system has zero-copy NIC(s), your mileage may vary. Package Changes --------------- o fdwatch (A.D.F.+Emmanuel Hocdet) - added "/dev/poll" interface for SunOS (Solaris); NOTE: in a chroot environment you may add a "/dev/poll" file by executing, as root, these commands: umask 111; mknod CHROOTPATH/dev/poll c 138 0 (substitute CHROOTPATH for absolute pathname of your chroot directory); NOTE: some small performance enhancements should be noted only when there are many (over 800 - 1000) active connections. NOTE: right now, there are a few bottlenecks left inside thttpd that limit performances when there are more than 1000 - 1500 connections (even when using "/dev/poll"); hopefully they will be partially removed in future releases. ---------------- thttpd-2.21b-p30 11-Aug-2003 ---------------- Release Notes ------------- Miscellaneous performance and robustness micro improvements, (nothing too exciting). Configuration ------------- o configure (Jef Poskanzer) - added a test for int64_t in files configure, configure.in and aclocal.m4 to define or not define HAVE_INT64T. o config.h (A.D.F.) - added CGICLI_WANTS_EXEC_BIT define to search for a matching CGICLI only when requested page has the executable bit set; - added LOG_FLUSH_TIME define to control flush policy of log file (HTTP requests). Cleanups -------- o HAVE_INT64T (A.D.F.) - added ifdefs to allow compilation also by (very old) compilers which don't have int64_t (or which have broken 64 bit arithmetics); o handler-loop (A.D.F.) - rewritten handler loop to use "switch" instead of many "else if" (no measureable speed-up). o HTTP-methods (A.D.F.) - added a new function httpd_method_id() to search for an HTTP method string and return its identifier (instead of using lots of if ... else if ...). o conversion-from-and-to-hexadecimal-representation (A.D.F.) - highered speed of conversion from hexadecimal to decimal by using a private table (O(1) algorithm); - removed unused function hexit() (substituted by local hex2dec_m() macro). Performances ------------ o TCP_DEFER_ACCEPT (Linux 2.4 and later) (A.D.F.+Sascha Schumann) - added a TCP/IP filter on first data packet (available in Linux 2.4 and following); *BSD users already had their SO_ACCEPTFILTER "data ready". NOTE: this kind of filters are able to reduce the number of open sockets when there many slow connections; in practice they are useful when there are high latencies, of at least 3/4 seconds, between accept time and the time of first data read. o fast-find-free-connect (Sascha Schumann) - added a fast search O(1) of a free connection slot by maintaining an array of pointers to free slots; this should be a win only when there are many active connections (much more than 100 - 150). NOTE: this algorithm is fast but it doesn't seem to take in account memory cache (L1, L2) needs, thus under some conditions, it could be a bit slower than previous linear search. o fast-make-log-entry (A.D.F.) - log date time is reformatted at most every second instead of at every log entry write; - log file can be flushed in three different modes (controlled by LOG_FLUSH_TIME define in config.h): - never flushed (LOG_FLUSH_TIME undefined); - periodically flushed (LOG_FLUSH_TIME defined and > 0); - flushed after every log write (LOG_FLUSH_TIME defined and == 0) (identical to original behaviour); NOTE: above speed-up(s) are effective only when an external log file is used (options -l or logfile=xxx) instead of default syslog(): NOTE: overall speedup (with LOG_FLUSH_TIME undefined), at max. request rate, is between +5% and +20% (your mileage may vary). Dynamic Requests ---------------- o CGICLI (A.D.F.) - removed the insensitive case handling of CGICLI extensions; now only match() is used and thus it is case sensitive (yeah, using strcasecmp() was a nice optimization, but it was not worthwhile in CGI context); - now interpreted pages should be executable because this speeds up things (no unneeded pattern matchings for static files) and secures access rights. Package Changes --------------- o fdwatch (A.D.F.) - added picky test conditions to make it more robust; - now error conditions are always checked, i.e.: allocated memory is always fred after a failure, etc.; - kqueue events are coalesced to reduce the number of used slots (useful for pipelined requests), i.e.: ADD_RD, DEL_RD, ADD_WR, DEL_WR, ADD_RD now above events are coalesced to: ADD_RD because now there are test conditions that prevent adding two consecutive (ADD or DEL) events for the same file descriptor (ADD + ADD or DEL + DEL are not allowed). o match (A.D.F.) - removed unused function match_pat2dot_ext() (previously added in p29). o mmc (A.D.F.) - added comments to fields of internal structures. o timers (A.D.F.) - added defines (TMR_ONE_SHOT, TMR_PERIODIC) for periodic flag argument in function tmr_create(). ---------------- thttpd-2.21b-p29 16-Jul-2003 ---------------- Release Notes ------------- Added CGI-CLI to support external interpreters (Command Language Interpreter) for scripts without having to add interpreter path name in the first line of every script. Configuration ------------- o configure, configure.in (Jef Poskanzer) - added check for existence of sigset() (HAVE_SIGSET). o mime_types.txt (A.D.F.) - added other 82 mime types, including: mht, mhtml, tbz2, etc.; NOTE: if you don't use some mime types, you can lower memory footprint by commenting unused entries in file mime_types.txt and then by recompiling thttpd (make clean; make). o config.h (A.D.F.) - added define EXECUTE_CGICLI to enable / disable CGI-CLI support; - added define MAX_CGICLI_ENTRIES to limit max. number of CLI patterns that can be accepted. o thttpd.c (CGI-CLI external file) (A.D.F.) - added support to load associations between pattern(s) and executable interpreter(s) from an external file into memory, see also: command line option: -cgicli filename and configfile directive: cgicli=filename NOTE: external CGI-CLI file has the following format: pattern interpreter-path i.e.: **.pl /usr/bin/perl **.php /usr/local/bin/php Dynamic Requests ---------------- o CGI-handling (David Phillips) - added the following CGI environment variables in function make_envp(): REQUEST_URI SCRIPT_FILENAME; - added a write loop, in cgi_interpose_input(), to handle partial writes; in fact, with keep-alive, there is the possibility that read buffer is greater than pipe size (4KB); - enhanced HTTP error reporting when HTTP headers are too long. o CGI-CLI-handling (A.D.F.+David Phillips) - added support for CGI-CLI (Command Language Interpreter) processes in order to run "interpreter command_file" as CGI process; this allows to not insert in first line of every script file, the absolute path of the executable interpreter, i.e. no more "# /usr/bin/interpreter". NOTE: you may want to set executable bits anyway in command files to prevent unwanted downloads of those files; NOTE: a "**.xxx" pattern is mapped to a file extension, thus comparisons are a bit faster than standard matching; NOTE: this new feature is a hack for people who are not willing to insert interpreter name (usually including path name) in the first line of every new script; NOTE: current implementation is a bit ugly / bloated. URL Handling ------------ o fixed handling of original name, now it should work properly again (it was broken in p15 by patch to allow '?' characters in filenames); now it should not be empty any more. o dir-redirect (A.D.F.) - fixed dirpath redirection, now it should work properly also for URLs with query string, i.e.: from: http://domain/dir?myvar=myval to: http://domain/dir/?myvar=myval and it also preserves query symbol '?' in empty query string: from: http://domain/dir? to: http://domain/dir/? NOTE: under dir/ there is a index.cgi or nph-index.cgi. o check-no-query-string-for-directory-listings (A.D.F.) - now, besides path-info, also query string is forbidden in any URL which resolves to a directory listing. o check-path-info-and-query-string (A.D.F.) - added a check to forbid query string for static files (even if Apache seems to accept it); this seems reasonable because pathinfo is already forbidden and this CGI stuff is totally unuseful if the file is not a CGI executable; these URL patterns are thereby forbidden (index.html is a file, not a directory): http://mydomain/index.html/mypathinfo http://mydomain/index.html/?querystring (NEW) http://mydomain/index.html/mypathinfo?querystring (NEW) o check-trailing-slashes (A.D.F.) - changed error status (from 404 to 403) and error message for filenames with trailing slash(es) ('/'); now this URL pattern is forbidden: http://mydomain/index.html/ use instead this one: http://mydomain/index.html Micro Optimizations ------------------- o figure_mime (micro-speed-up) (A.D.F.) - added check conditions on length of file extension (it must be > 0) prior to search for it. Cleanups -------- o httpd-errors (400, 408, 503) (A.D.F.) - removed these variables: httpd_errXXXtitle, httpd_errXXXform from libhttpd.h and thttpd.c; - added static variables errXXXtitle and errXXXform in libhttpd.c; - added these two functions: httpd_err_title( int status ); httpd_err_form ( int status ); to be used in thttpd.c, thus potentially every known HTTP error title and form can be used by external modules (not libhttpd.c) without exporting all variables (now they are all static). o httpd-error-padding (A.D.F.) - changed padding for MSIE in send_(error)response, now error message is padded up to a total of 512 bytes (there is no need to always add 480 bytes of padding comments). o send-mime (A.D.F.) - removed computation of mime type length. o cwd (current working directory) (A.D.F.) - added a new field (hs->cwd_len) to not call strlen(hs->cwd) (1 - 3 times) for each HTTP request (cwd never changes after server initialization); - reduced persistent stack utilization in function main(); now cwd string is dynamically allocated. ---------------- thttpd-2.21b-p28 25-Jun-2003 ---------------- Release Notes ------------- This is an highly experimental version, treat it with care: - now persistent connections are enabled also for HEAD and GET 304 methods; - a rough overload control on number of requests and read timeouts has been added to limit number of idle keep alive connections; - a few parameters and a command line option have been added; - now the definition of "keep alive connection" is used as an alias of "persistent connection" ("persistent connection" would be a better definition but everybody seems to use / understand only the obsolete term "keep alive"); - various upgrades, fixes and micro improvements. Configuration ------------- o config.h (A.D.F.) - joined DEFAULT_MIME_TYPE_EXT0 and DEFAULT_MIME_TYPE_EXT1 into DEFAULT_MIME_TYPE because it is not worthwhile having them split (current browsers are too dumb to handle well text contents without a text mime type); in fact also known file extensions such as source files (*.c, *.h) are misinterpreted by popular clients (MSIE, Netscape, etc.) because of HTML entities ('<', '>', etc.); - added DEFAULT_MIME_TYPE_BIN in order to be used when there are too many encodings or (in next future) when file contents are binary; - added MAX_MIME_ENCODINGS_LIMIT to limit the number of encodings for each file name; - added FNREG_FIX_TRAILING_SLASHES to trim trailing slashes from regular filenames (not CGIs and not directories) and then to redirect fixed URL; NOTE: that piece of code (trim and redirect) was added in p19, now it can be disabled to always return 404 error as Apache does; - added LOWM_KEEPALIVE_RQSLIMIT to limit number of "keep alive" requests through a connection when total number of connections is below "High Water Mark 1"; - added HIWM_KEEPALIVE_RQSLIMIT to limit number of "keep alive" requests through a connection when total number of connections is over "High Water Mark 1"; - added BAD_KEEPALIVE_UA_LIST to make a configurable list of bad user agents (browsers) which claim to support keep-alive but they really don't because they are buggy. NOTE: this list has always been hard-coded (inside code) but now those browsers should be considered really too old to be used thus you may want to undefine that list; - added INDEXES_SKIP_DOTCURDIR to skip the name of current directory ("./"); - added ERR_VHOST_DIR to enable / disable custom virtual host error pages separately from server wide custom error pages (under ERR_DIR directory). o command-line (A.D.F.) - added option -noka (no keep alive connections) which has the same meaning as option -nopc (no persistent connections). o read_config (-C configuration-file) (A.D.F.) - added a check for a too long line (missing newline character); - added support to skip empty lines (David Phillips) (reworked by me); - added the following parameters: LOWM_KEEPALIVE_RQSLIMIT HIWM_KEEPALIVE_RQSLIMIT to limit number of "keep alive" requests on a connection; if number of active connection is below "High Water Mark 1" then LOWM_KEEPALIVE_RQSLIMIT is used else HIWM_KEEPALIVE_RQSLIMIT is used; - added the following directive: NoKeepAliveConns which has the same meaning as NoPersistentConns (it disables keep alive connections). ------------------------------------------ NOTE: _only_ HTTP/1.1 connections are kept alive. ------------------------------------------ HTTP/1.0 are not kept alive because of possible old buggy proxy servers which may not handle well keep alive connections (and by now we don't want to deal with this level of details); BE AWARE of this issue also when using old test programs, i.e. ab(1) (Apache Bench). o mime_types.txt (A.D.F.) - added other 8 mime types, including: mp4, mng, jng, bz2, taz, tgz, etc.; NOTE: some of new entries are associated with mime type "application/octet-stream" because they are not registered by IANA. Version ------- o thttpd-name (David Phillips) - added #define SERVER_SOFTWARE_NAME in version.h and libhttpd.c. Compatibility ------------- o timezone (David Phillips) - compatibility definition of timezone for cygwin. o defang-html-entities (A.D.F.) - added new function need_defang() to improve efficiency (it allows to skip unnecessary copies); - added, in function defang(), other two characters '&' and '"' to those to be escaped into html entities. HTTP Standard Compliance ------------------------ o HTTP-Recognized-Methods (A.D.F.) - added the parsing (no support) of these missing HTTP/1.1 methods: OPTIONS, PUT, DELETE, TRACE, CONNECT, in order to log them in log file (their use leads to HTTP error 501 Not Implemented). NOTE: PATCH, LINK and UNLINK are not included because they have been removed from HTTP/1.1 specifications since RFC 2616. o HTTP-no-body-response (A.D.F.) - added checks in httpd_send_err() to not send error contents when replying to HEAD requests or to other methods with status equal to 100-199, 204, 304. o HTTP-errors (A.D.F.) - added HTTP error "416 Requested range not satisfiable" (triggered when init byte location is >= file length). HTTP Parsing ------------ o set-linger-mode-early (A.D.F.) - now keep alive and linger flags are setted as soon as possible (in httpd_parse_request) to be used properly also by some http error replies. URL Handling ------------ o check-trailing-slashes (A.D.F.) - added a check (really_start_request) on file names with trailing slash(es) ('/'), now if FNREG_FIX_TRAILING_SLASHES is defined and a file name ends with a slash but encoded URL does not, thttpd detects this and returns error 404 as Apache does. NOTE: thttpd is not willing to try the hard way to handle this strange case (bad URL). HTTP Tuning ----------- o keep-alive-requests (A.D.F.) - now number of keep alive requests can be limited depending on server load. o keep-alive-timeout (A.D.F.) - now keep alive timeout is dynamically incremented (+2-6 sec.) if there were pipelined requests or file content was sent through a connection and number of connections is less than "High Water Mark 1" and / or "High Water Mark 2". o linger-timeout (A.D.F.) - now linger timeout is slightly incremented (+1 sec.) when a closing connection had pipelined requests and number of connections is below "High Water Mark 2". o high-water-marks (A.D.F.) - added two internal connection limits: "High Water Mark 1" (60% of max. number of connections) "High Water Mark 2" (80% of max. number of connections) to shorten "keep alive" timeouts and number of keep alive requests when the total number of active connections gets high; NOTE: the aim is to lower the number of idle / unused connections when server is loaded. o overload-short-reply (A.D.F.) - added a check to shorten 503 error reply (service overloaded) when server is throttling and number of connections is getting high; in practice only HTTP headers (without any body content) are sent. File Indexes ------------ o ls-enhancements (A.D.F.+David Phillips) - reduced memory footprint of file names array by allocating only needed memory for each file name; this because MAXPATHLEN can be as high as 4095 bytes (indeed it should be NAME_MAX not MAXPATHLEN), thus for 1000 files, before this change, thttpd ls() needed up to 4 MB only for file names, instead now it needs only a mean of 30 * 1000 = 30 KB; - lowered CPU load (*) when listing large directories (with over 500 - 1000 files) along with long file names (less strlen(s)); NOTE(*): 7% of CPU instead of 11% on a PIII 733 128 MB RAM for a directory listing of 2000 files; - micro speedup in strencode() (used only in ls()) when encoding escaped characters (no slow sprintf); - linkprefix, changed "->" into html entity "->"; - added defang to escape html entities in file names; (David Phillips) - added support for INDEXES_SKIP_DOTCURDIR; - added an horizontal separator at the end of the listing. Q. Should we stat() files in readdir() order ? Are stats() done after sorting file names slower than those done prior of sorting file names ? File Cache ---------- o do-not-map-zero-files (A.D.F.) - zero sized files are no more stored in file cache, thus they are no more opened/closed (stat() and other checks should be enough to verify access rights). o new-oom-handler (A.D.F.) - rewritten the OOM (Out Of Map resources) handler in order to be smart in how to free cache levels; now levels lower than current, where OOM occured, are only partially fred (in a progressive way). o remove-unused-function-calls (A.D.F.) - mmc_map(), removed stat() and time() calls because since p20 mmc_unmap() needs the same statbuf passed to mmc_map(), thus statbuf MUST be handled by the caller of mmc_mmap() / mmc_unmap(); - mmc_map(), added checks for NULL ptr. to statbuf and time; - mmc_unmap(), removed time() call; - mmc_unmap(), added checks for NULL ptr. to statbuf and time. o slow-map-unmap-support (A.D.F.) - added mmc_slow_map() and mmc_slow_unmap() to be able to use less parameters (without statbuf and current time); NOTE: they are painfully slow, eventually enable them defining MMC_SLOW_MAP_UNMAP in mmc.h. NOTE: by default they are disabled because they are not used in thttpd. o cleanups (A.D.F.) - added more const qualifiers; - added function name as a const parameter to syslog() functions; - added MMC_DEBUG define to disable syslog() alerts on almost impossibile cases; - added MMC_UPD_REFMAP_TIME define to disable unuseful update of reference time for referenced maps (time has to be updated only for unreferenced maps). o update-comments (A.D.F.) - mmc.h, updated comments to reflect above changes. Other Fixes ----------- o figure_mime (A.D.F.) - fixed possible buffer overflow in hc->encodings[] (it was triggered by file names with more than 20-40 encodings); - added MAX_MIME_ENCODINGS_LIMIT to stop figuring encodings when they are over this limit (useful for malicious file names); - fixed encoding item separator, use ',' instead of ';'; - fixed encoding order, encodings must be listed in the order they were applied to file content (not the reverse); see also RFC 2616 14.11. Other Cleanups -------------- o thttpd-signals (A.D.F.) - moved setup signals stuff from main() to setup_signals(). o handle_send[_resp] (A.D.F.) - removed use of temporary variable to handle length of partial writes of response headers. o httpd_get_conn (A.D.F.) - tweaked initial allocation values, to match those found in common requests, in order to optimize memory usage. o httpd_parse_request (A.D.F.) - tweaked max. length limit for Accept: and "Accept-Encoding:" header fields. o httpd_close_conn (A.D.F.) - removed second (unneeded) close() call. o httpd_request_reset0 (A.D.F.) - added zeroing of USE_LAYOUT fields. Package Updates --------------- o fdwatch-resync (A.D.F.) - resynchronized code with official version 2.23, now kqueue should work also on FreeBSD >= 5.0. o match-resync (A.D.F.) - resynchronized code with official version 2.23 (only a comment). o tdate_parse (A.D.F.) - resynchronized code with official version 2.23 (cast to void). ---------------- thttpd-2.21b-p27 27-May-2003 ---------------- Security -------- o vhost-map-fix (David Leadbeater) - added another check to increase protection against unwanted file reads. Configuration ------------- o config.h (A.D.F.) - split DEFAULT_MIME_TYPE into DEFAULT_MIME_TYPE_EXT0 and DEFAULT_MIME_TYPE_EXT1 to better control sending of "Content-Type" header when mime type is unknown (not all unknown files are text files); - added DO_ALLOC_STATS to enable / disable statistics about strings allocations in libhttpd.c. o mime_types.txt (A.D.F.) - added more than 25 new mime types. HTTP Standard Compliance ------------------------ o HTTP-GET-if-modified (Eli White + - now 304 responses to conditional GET requests Jef Poskanzer + are handled properly: Ross Patterson + - no Content-Length; others) - log as GET not HEAD. o HTTP-Allowed-Methods (A.D.F.) - added error "405 Method not allowed" (for the resource); - added header "Allow: #Methods" in function send_mime(); it is required when server replies with 405 error and it is recommended when server replies with 501 error. File Cache ---------- o more-comments (A.D.F.) - added a few comments to source code; - changed some values inside macros; o restore-set-of-expire-age (A.D.F.) - restored the possibility to effectively change default value of expire age (mmc.c); o const-work-around (A.D.F.) - removed "const" qualifier from mmc_is_fdmap(), mmc_cleanup(), mmc_pmc_cleanup() and mmc_size_to_pmc() (just a try). o mmc_really_unmap-bug-fix (A.D.F.) - don't free / munmap zero sized files !!! (I'm sorry for this bug I introduced in p25, remember: don't leave incomplete code without a comment TODO; after several weeks you can make the mistake to release it as is). Other Fixes ----------- o micro-fixes (A.D.F.) - substituted a few sizes of link[] and path[] strings for [MAXPATHLEN + 1] constant; - added 1 to the size of MAXPATHLEN strings (link[], etc.) in functions: ls(), expand_symlinks() and main(); - passed sizeof(buf) - 1 to readlink() instead of sizeof(buf); - thttpd.c, enlarged size of stats_bytes from long to int64 to avoid overflows; - thttpd.c, fixed byte accounting, now only sent bytes are added to the total sum; - thttpd.c, split stats_bytes in two counters: - one for header and error responses; - one for file contents (only sent bytes to socket); - thttpd.c, added a counter for number of requests (with keep-alive there may be several requests per connection). Other Cleanups -------------- o micro-cleanups (A.D.F.) - thttpd.c, commented two calls to closelog() in signal handlers (less risky); - libhttpd.c: conditionally disabled string accounting; allocated string space is always incremented (at some point the total overflows and becomes negative) because it is never decremented when it is fred, thus it may be useful only when debugging new code; NOTE: eventually it should be rewritten properly; - libhttpd.h: added a prototype for function httpd_complete_request(). ---------------- thttpd-2.21b-p26 28-Apr-2003 ---------------- Configuration ------------- o no-persistent-connections (A.D.F.) - added command line flag -nopc (no persistent connections) to disable HTTP/1.1 persistent connections at run-time; - added the following parameter to external configuration file (option -C): NoPersistentConns to disable HTTP/1.1 persistent connections at run-time. NOTE: this can be useful if you want: - to slow down HTTP/1.1 requests rate for recent browsers (they usually use only a few parallel connections); - to emulate old thttpd behaviour. NOTE: this option/directive can waste network resources (i.e. more TCP/IP packets per served content, discarded pipelined requests, etc.) and slow down too much heavy loaded sites while not preventing smart users to use accelerators to make lots of parallel connections; I don't recommend it, try instead to use throttle. o max-conn-bytes-limit (A.D.F.) - added command line option -mcbl BytesPerSec to set an upper limit for every connection, see also MAX_CONN_BYTES_LIMIT. o config.h (A.D.F.) - changed USE_SENDFILE define, now you have to define or undefine it to enable / disable use of sendfile(); - added USE_MY_OLD_SYSLOG (undefined by default) to enable/disable old compatible syslog(3) functions in syslog.c; - added MAX_REAP_COUNT to set a limit to children launched without reaping; - added MIN_THROTTLE_LIMIT to set and grant the minimum limit (byte/sec.) for each request; - added MAX_CONN_BYTES_LIMIT to set a bandwidth upper limit (bytes/sec.) for every connection (independently from throttle); - added DEFAULT_MIME_TYPE to control sending of "Content-Type" header when mime type is unknown (not all unknown files are text files). Connection ---------- o handle_linger (thttpd.c) (A.D.F.) - increased local buffer size from 2048 to 4096 (and declared it to be static) in order to empty input socket using fewer read() calls and to avoid potential TCP/IP resets. NOTE: this is useful also because Mozilla 1.x and Netscape 7 allow to force pipelining even for the first HTTP request in a new connection (this violates RFCs but speeds up things); in the near future new aggressive settings in browsers could allow to send more than 3 pipelined requests per batch request before receiving a reply from server. o throttle-fix (A.D.F.) - fixed throttling when using HTTP/1.1 persistent connections (it was broken by keep-alive patch p08). HTTP Standard Compliance ------------------------ o HTTP-Version (A.D.F.) - added error "505 HTTP Version not supported"; - added check conditions to refuse HTTP versions not supported ( < 0.9 || >= 2.0 ); NOTE: HTTP Major and Minor versions are now properly interpreted and, in error responses, we use the highest HTTP version we support. CGI-Handling ------------ o reap-children (A.D.F.) - added global variable reap_count to count number of succesful fork() calls done in thttpd between two calls to do_reap(); - added function do_cond_reap() to call do_reap() at most every MAX_REAP_COUNT; - added calls to do_cond_reap() in libhttpd.c -> cgi() and libhttpd.c -> ls(); this is useful if hundreds of CGIs or directory listings are launched in a short time. Other Cleanups -------------- o thttpd-declarations (A.D.F.) - moved global static int terminate inside main function (there is no need to keep it global); ---------------- thttpd-2.21b-p25 10-Mar-2003 ---------------- Log --- o syslog-code-fixes (A.D.F.) - removed redundant connect() call in syslog.c::openlog(); this could prevent opening a log connection when USE_INET was defined (now by default it is undefined); - added a check on return code of fcntl(2), it should never fail but you cannot trust it blindly. o server-info-on-start (A.D.F.) - added a syslog() message (server-info) immediately after openlog() in thttpd.c. o mmc-stats (A.D.F.) - enhanced mmc statistics (file cache) "syslogged" by mmc_logstats(). Configuration ------------- o default-values-config.h (A.D.F.) - conditionally set names in INDEX_NAMES (depending on EXECUTE_CGI); - undefined by default EXECUTE_CGI; - undefined by default GENERATE_INDEXES. NOTE: if EXECUTE_CGI and GENERATE_INDEXES are undefined then thttpd can serve only static contents, thus a few small speed optimizations are applied (don't set close-on-exec on each new socket, etc.). o mmc-default-values-tuning (A.D.F.) - changed default value of DEFAULT_EXPIRE_AGE from 300 to 150 (due to file cache changes because now it is used for L3 files instead of L1 files); - added MAX_FILE_SIZE_L4 to config.h and thttpd.conf; - changed default expire age multipliers: L0 16 -> 64 L1 3 -> 8 L2 1.50 -> 4 L3 0.37 -> 1 o indexes-skip-dot-files (David Phillips+A.D.F.) - added new define INDEXES_SKIP_DOTFILES to skip files starting with dots (excepted for "..") when generating indexes; - added new define INDEXES_REMARK_DOTDIRS to add comments to "." and ".." (in order to clarify what they mean); - capitalized header strings, i.e. "mode" -> "Mode", etc. Fixes ----- o http-error-response (A.D.F.) - httpd_send_err(), forced a disable of keep alive because, after an error response, connection will be closed anyway. o cgi-empty-query-string (Suzuki Yoshio) - httpd_parse_request(), reverted a test, on empty query strings added in p20 (it prevented '?' to be removed from CGI file name) NOTE: using CGIs is still a bad idea with this version. Performances ------------ o mmc-cache-optimizations (A.D.F.) - generalized the handling of file cache zones, now there are 5 levels and 6 zones which are controlled by a dedicated "zone array": Level I/O method ----- ---------- L0 malloc L1 mmap L2 mmap L3 mmap L4 mmap L5 sendfile The extension of above zones can be changed by setting proper values in MAX_FILE_SIZE_L0 ... L4; L4 can be hidden (forcing it to be ignored) by setting MAX_FILE_SIZE_L4 <= MAX_FILE_SIZE_L3. o libhttpd-sendfile (A.D.F.) - added native sendfile() call for Solaris to httpd_sendfile(); you can enable BSD, Linux or Solaris support defining ONLY ONE of these definitions in Makefile: -DHAVE_BSD_SENDFILE=1 -DHAVE_LINUX_SENDFILE=1 -DHAVE_SOLARIS_SENDFILE=1 NOTE: if you do so, you have to recompile thttpd: make clean; make NOTE: only Linux sendfile() has been tested, BSD and Solaris are believed to work; NOTE: on BSD and Solaris, please, read sendfile man page for library requirements; probably you need to add something like -lsendfile, i.e.: LIBS = -lcrypt -lsendfile in generated Makefile (after each ./configure). o HTTP-Version (A.D.F.) - optimized parsing of http version to make common case a bit faster (libhttpd.c -> httpd_parse_request()). ---------------- thttpd-2.21b-p24 18-Jan-2003 ---------------- Standard Compliance ------------------- o hash-mime-case-insensitive (my bad) (A.D.F.) - hash value is no more case-sensitive, this restores the proper association between file extensions and mime types in a case-insensitive manner (as it was in p18); i.e. now index.HTML is served in the same way as index.html. ---------------- thttpd-2.21b-p23 27-Nov-2002 ---------------- Security -------- o rqs-headers - more checks for host name (malformed request) (Marcus Breiing+others) o url-handling (Mark Dunlap + - fixed de_dotdot(). Dana Dahlstrom) Authorization ------------- o ignore-trailing-fields (Jef Poskanzer) - ignore trailing fields in authorization files. o auth-variable (Alexandre Cherif) - fixed CGI variable AUTH_TYPE. Configuration ------------- o mmc-parameters (A.D.F.) - added the following parameter to external configuration file (option -C): NO_SENDFILE in order to disable use of sendfile() even if USE_SENDFILE has been set to 1 in config.h. o mmc-settings (A.D.F.) - lowered DESIRED_FREE_MAC_COUNT (free list for malloc cache) from 500 to 200; - changed expire_age_L3 calculation (new formula). o generate-indexes (A.D.F.) - added command line flag -nogi (no generate indexes) to disable directory listings at run-time; - added the following parameter to external configuration file (option -C): NoGenerateIndexes to disable directory listings at run-time. o cgi-execution (A.D.F.) - added #define EXECUTE_CGI to config.h CGI programs are executed only if also this symbol is defined otherwise all related code is not compiled (smaller executable, etc.). Optimizations ------------- o keep-alive-pipelining (A.D.F.) - changed default size of request buffer from 500 to 1536 (512 * 3) because now most browsers are HTTP/1.1 compliant and because some of them are going to enable pipelining by default. ---------------- thttpd-2.21b-p22 30-Aug-2002 ---------------- Configuration ------------- o default-values-tuning (A.D.F.) - changed default value of IDLE_KEEPALIVE_TIMELIMIT from 2 to 4. NOTE: it should be 10 seconds or more for very slow modem connections or congested networks, but the higher it is set the lower the number of served new TCP/IP connections per second is. o new-config-defines - added ALLOW_ACCESS_GRP to config.h in order to check for "group" rights instead of "other" rights when accessing files. Fixes ----- o keep-alive-pipelined-requests (A.D.F.) - added support for pipelined requests, see also handle_buf_read() and clear_connection(). - added a check to conditionally delay the start of read of next request; this should prevent a premature close of the connection because of slow send (low bandwidth). NOTE: this feature should work but it has not been tested with all popular browsers (only with Netscape and Mozilla). o parse-http-request (A.D.F.) - added a check to httpd_got_request(), now first word cannot start with spaces. ---------------- thttpd-2.21b-p21 26-Aug-2002 ---------------- Performance enhancements ------------------------ o mmc-file-cache (A.D.F.) - added / changed a few functions to handle new extended cache logic; - added a third independent cache for small files (strong impact against performances). NOTE: now file cache should be a bit faster when the number of cached files is over 5000 - 10000. NOTE: now there are three methods to store file contents in file cache: - malloc(), it is used (when MAX_FILE_SIZE_L0 is > 0, disabled by default), for files smaller than or equal to MAX_FILE_SIZE_L0; - mmap(), default method used for files between MAX_FILE_SIZE_L0 + 1 and MAX_FILE_SIZE_L3 (and over if USE_SENDFILE is not setted); - fdmap, caches files descriptors for file sizes bigger than MAX_FILE_SIZE_L3 when USE_SENDFILE is enabled (in config.h or in -C configfile) (disabled by default); o avoid-redundant-syscalls (A.D.F.) - httpd_get_conn(), added a test to set "close on exec" only if there is the possibility to fork() + exec() an external program (CGI, cgi_pattern != NULL). Configuration ------------- o mmc-parameters (A.D.F.) - added the following parameters to config.h and to external configuration file (option -C): DESIRED_MAX_MALLOC_BYTES (related to L0 cache) DESIRED_MAX_MALLOC_FILES (related to L0 cache) MAX_FILE_SIZE_L0 (upper limit for size of files stored in L0 cache) Cleanups -------- o thttpd (A.D.F.) - fixed a security check on DesiredMaxOpenedFiles; o libhttpd (A.D.F.) - added include . ---------------- thttpd-2.21b-p20 14-Aug-2002 ---------------- Changes ------- o layout-mapping (A.D.F.) - now layout files are no more mapped via mmc_map(), instead their contents are read into RAM using malloc() + read(); this allows to always use write() or writev() to send their contents (obvious fix now that we have a second file cache based on file descriptors). NOTE: now the size of each layout file has to be in the range 1 - 16384 bytes (otherwise it defaults to 0). o overload-check (A.D.F.) - added a check to reply with 503 error (service overloaded) when number of connections + opened files (fdmap) > max. number of connections; this rough check should prevent unpleasant conditions when reaching max. open file limit. NOTE: this check is not mandatory, if you don't like it you can comment it out in libhttpd.c->really_start_request(). New Features ------------ o mmc-fd-map (A.D.F.) - added a second file cache for file descriptors; now these files are kept opened so that thttpd can use read+write or sendfile() method (instead of mmap) to send file contents; - added internal option to set flag CLOSE_ON_EXEC on opened files (this is useful if thttpd spawns CGI subprocesses). Configuration ------------- o mmc-parameters (A.D.F.) - added the following parameters to config.h and to external configuration file (option -C): DESIRED_MAX_OPENED_FILES USE_SENDFILE DESIRED_MAX_OPENED_FILES sets a limit to desired opened files; if idle opened files are over that limit, then they are closed even before periodic cleanup; USE_SENDFILE enables use of sendfile(). o libhttpd-sendfile (A.D.F.) - added function httpd_sendfile(), default setting is to use "emulation" via read()+write(), otherwise you can enable Linux or BSD support defining one of these two definitions in Makefile: -DHAVE_LINUX_SENDFILE=1 -DHAVE_BSD_SENDFILE=1 NOTE: if you do so, you have to recompile thttpd: make clean; make NOTE: BSD sendfile() has not been tested so drop me an e-mail if it doesn't work (better, fix it yourself and send me a patch). o sendfile-method (A.D.F.) - added a call to httpd_sendfile() when fdmap cache is enabled (USE_SENDFILE + file size > MAX_FILE_SIZE_L3); NOTE: this should remove/reduce difficulties (error 500), when serving lots of big files, due to lack of mmap space on 32 bit machines (64 bit machines don't have this problem). Fixes ----- o mmc-enomem (Jef Poskanzer) - added a cleanup of all mapped files when a mmap() or malloc() fails because of lack of memory space. o url-redirect (A.D.F.) - added a redirection for wrong file paths, aimed to remove trailing '/', i.e.: "http://mydomain/file/" is redirected to: "http://mydomain/file" WARNING: this change might be critical under strange conditions, be aware of multiple redirections. o keep-alive (A.D.F.) - changed conditions under which keep-alive and do_keep_alive are used. Cleanups -------- o log-warning (A.D.F.+others) - downgraded severity of message logged when thttpd is started as root without chroot, from critical to warning (it's not a run-time error). o symbolic-link-expansion (A.D.F.) - added a few checks to expand_symlinks(). ---------------- thttpd-2.21b-p19 07-Aug-2002 ---------------- Fixes ----- o thttpd-signals (A.D.F.) - added #define my_signal to handle signal()/sigset(); - added new boolean variable "in_shut_down" to avoid calling shut_down() twice while shutting down connections; - added "volatile" and "sig_atomic_t" qualifiers to declaration of variable "gotSigMask"; - added new function handle_rsig() to avoid recursive calls to handle_term() for the same signal; - added a call to my_signal() to handle_hup(), handle_usr1() and handle_usr2() to reset signal handler if HAVE_SIGSET is not defined (rough hack for SYSV OS). o http-version (A.D.F.) - changed HTTP version parser; now thttpd downgrades or limits properly its replied HTTP version. o query-string-fix (A.D.F.) - added a check to disallow redirection when a query string follows immediately a directory name. o regular-file-security-fix (A.D.F.) - added checks to prevent retrieve of non regular files. Cleanups -------- o thttpd-declarations (A.D.F.) - added "static" keyword to declaration of: terminate; start_time; stats_time; stats_connections; stats_bytes; stats_simultaneous. Performance enhancements ------------------------ o figure-mime (A.D.F.) - substituted linear search for hash search, because it is: 8 - 30 times faster than linear search; 1 - 3 times faster than binary search. o config.h/ERR_DIR (A.D.F.) - put #define ERR_DIR inside #ifdef notdef ... #endif so that by default thttpd uses only its internal messages without trying to open ERR_DIR/error_file every time an HTTP error occurs. Configuration ------------- o CGI_MAX_HEADERS_LENGTH (A.D.F.) - moved #define CGI_MAX_HEADERS_LENGTH from libhttpd.c to config.h; a value of 0 tells thttpd to not check for length of CGI headers. ---------------- thttpd-2.21b-p18 06-Jun-2002 ---------------- File-Cache ---------- o mmap-cache-expire-age-calc3 (mmc.c + mmc.h) (A.D.F.) - added a new parameter (MMC_P_CLEANUP_TIME) to set a proper time increment for expire_age_L1; - improved granularity of expire_age calculation (mmc_cleanup(), mmc.c); now check points are 75%, 87%, 100% and 125% of current mmapped files (instead of 75% and 100%); in practice, number of mmapped files is kept nearer to DESIRED_MAX_MAPPED_FILES; - changed calculation of expire_age_L3 to decrease it less when there is a high mmc pressure (heavy load); now expire_age_L3 dynamically ranges between expire_age_L1 / 4 and expire_age_L1 / 8 (5 steps). Connection ---------- o handle_linger (thttpd.c) (A.D.F.) - increased local buffer size from 1024 to 2048 in order to empty input socket using fewer read() calls (NOTE: popular browsers are getting "very aggressive"). Configuration ------------- o configuration (config.h) (A.D.F.) - changed values of the following constants to fit common needs of a busy Internet web server (up to 50 - 100 new connections per second): OCCASIONAL_MMC_TIME 12 (was 16) MIN_REAP_TIME 5 (was 30) MAX_REAP_TIME 100 (was 900) MAXTHROTTLENUMS 2 (was 10: please, use 2, 10 or 18 to set size of connecttab elements to 64, 96 or 128 bytes) (NOTE: if you use more than 1-2 throttling pattern then restore default value of 10) LINGER_TIME 1 (was 2) (NOTE: restore it to 2 if thttpd serves only modem users and there aren't too many of them). o memory-footprint (thttpd.c) - reduced initial size of throttle tab from 100 to 16 (read_throttlefile()). Log-Information --------------- o log-notice (thttpd.c) - added a syslog() to log value of maxconnects (max number of connections) at start-up. ---------------- thttpd-2.21b-p17 29-Apr-2002 ---------------- Content-layout -------------- o memory-init (A.D.F.) - zeroed allocated memory for connection table (connects[]); o fdwatch-check (A.D.F.) - added a check to poll_check_fd on fd range values. o state-machine (A.D.F.) - added a missing FDW_WRITE event in resp_clear_connection (this could lead to strange effects - high CPU usage, etc. - under very rare error conditions); - setted cleared connections to CNST_FREE, in shut_down(); - added a test to return when conn_state == CNST_FREE, in really_clear_connection(); - added timer_cancel(linger_timer), in resp_clear_connection() and clear_connection(). o http-keep-alive (A.D.F.) - changed parsing of HTTP/x.x; now HTTP/1.1 and later versions have keep-alive enabled by default. o got-range-fixes (A.D.F.) - added a few checks on got_range + init / end byte location (httpd_parse_request() + send_mime()). o layout-header-footer (Mihai Rusu) - added support for sending a header and / or a footer before and / or after each html file (see USE_LAYOUT define in config.h). o layout-header-footer-fixes (A.D.F.) - don't enable layout if header or footer file size is zero; - protect header / footer file mmap() from multiple definitions in external configuration file (-C option); - added support for Content-range requests when layout is enabled. o configuration (config.h) (A.D.F.) - disabled by default AUTH_FILE putting it inside a notdef section: #ifdef notdef #define AUTH_FILE ".htpasswd" #endif NOTE: to reenable it you have to move #define AUTH_FILE ... outside #ifdef notdef ... #endif section and recompile thttpd. - changed values of the following parameters to fit common needs of an Internet web server (lots of slow connections + small files): DEFAULT_EXPIRE_AGE 300 MAX_FILE_SIZE_L1 (1024 * 16) MAX_FILE_SIZE_L2 (1024 * 96) MAX_FILE_SIZE_L3 (1024 * 256) ---------------- thttpd-2.21b-p16 06-Apr-2002 ---------------- Signals ------- o signal-handling (thttpd.c) (A.D.F.) - added a global variable to hold SIGHUP, SIGUSR1 and SIGUSR2 signal events; - removed got_usr1; - changed and moved code from inside: handle_hup(); handle_usr1(); handle_usr2(); to main loop; this seems to help avoiding SIGSEGV after a SIGHUP (log rotation) has been received under heavy load. Cleanup ------- o constant-string-declarations (libhttpd.c) (Rob Ekl) As proposed by Rob Ekl, constant strings should not be pointers because this may force compiler to handle two variables (string and pointer) instead of one (probably this matters only for old compilers): - substituted: char* myString = "This is a constant string"; for: char myString[] = "This is a constant string"; Correctness ----------- o cgi-error-messages (Jef Poskanzer + A.D.F.) Fixes for incorrect behaviour noticed by Neale Pickett: - added missing "writes" for error messages generated before running a CGI program; - partially merged Jef Poskanzer's CGI-patch (cgi_interpose_output()). Stability --------- o cgi-headers-length (A.D.F.) - added a check to limit length of httpd headers to CGI_MAX_HEADERS_LENGTH (128KB). o cgi-block-mode (A.D.F.) - added httpd_clear_ndelay() to: cgi_interpose_input(); cgi_interpose_output(); (just to be sure in case block-mode is lost after a fork()). ---------------- thttpd-2.21b-p15 25-Mar-2002 ---------------- Stability --------- o fdwatch-fixes (Jef Poskanzer + A.D.F.) - substituted malloc for calloc so that allocated memory is zeroed; - added a few checks in fdwatch.c (poll_watch(), etc.). o ls-fixes (A.D.F.) - removed non-blocking -> blocking mode switch in parent process (now it is allowed only in child process); - added closedir() calls to avoid fd directory leaks. o cgi-fixes (Jef Poskanzer + A.D.F.) - removed non-blocking -> blocking mode switch in parent process (now it is allowed only in child process); - added fd fixes. o default-setup-adjustment in config.h (A.D.F.) - lowered DEFAULT_EXPIRE_AGE from 300 to 100 seconds. - lowered MAX_FILE_SIZE_L1 from 256 KB to 128 KB. Filenames --------- o filenames-fixes (Cameron Gregory) - modified behaviour to allow '?' in filenames (non standard but probably useful when encoding them to escape strange characters) see also: libhttpd.c, httpd_parse_request(); Mime-Types ---------- o mime-types-update (Jef Poskanzer) - added a few MS mime types. Syslog ------ o syslog-default - commented line 44 /* USE_INET */ in file syslog.c so that, by default, syslog can work also under Linux (in a chrooted environment you have to play with syslogd options, i.e. -r under Linux). SEE ALSO: thttpd FAQ NOTE: syslog.c has gone in thttpd versions >= 2.23 so in those versions this problem is solved. ---------------- thttpd-2.21b-p14 06-Feb-2002 ---------------- File-Cache ---------- o config-parameters (A.D.F.) - added the following parameters to configuration file (see read_config() in thttpd.c): - MAX_FILE_SIZE_L1=NumBytes # NumBytes: (1024 - 1073741824) - MAX_FILE_SIZE_L2=NumBytes # NumBytes: (1024 - 1073741824) - MAX_FILE_SIZE_L3=NumBytes # NumBytes: (1024 - 1073741824) o mmap-cache-file-size (A.D.F.) - added expire_age by file size so that expire_age depends on file size: expire_age_L1 = expire_age; expire_age_L2 = expire_age / 2; expire_age_L3 = expire_age / 8; expire_age_L4 = 0; NOTE: this is a first (yet lazy) attempt to keep small files in mmc cache longer than bigger ones; obviously this doesn't solve cases where there are hundreds or thousands of unique big files (i.e. 4, 10, 100 MB), being downloaded at the same time, that get "500 internal errors" because of max total memory mapped limit (usually 1, 2, 3 GB under 32 bit OS). SEE ALSO: config.h ---------------- thttpd-2.21b-p13 14-Jan-2002 ---------------- File-Cache ---------- o mmap-cache-expire-age-calc2 (A.D.F.) - tuned granularity of expire_age calculation, mmc_cleanup(), mmc.c. Configuration ------------- o config-parameters (A.D.F.) - added the following parameters to configuration file (see read_config() in thttpd.c): - IDLE_SEND_RESP_TIMELIMIT=NumSecs # NumSecs: (2 - 300) - IDLE_KEEPALIVE_TIMELIMIT=NumSecs # NumSecs: (1 - 300) ---------------- thttpd-2.21b-p12 08-Jan-2002 ---------------- Correctness ----------- o send-resp-timelimit (A.D.F.) - increased N. of seconds of IDLE_SEND_RESP_TIMELIMIT from 5 to 15 seconds in config.h; this is the time thttpd waits while sending a chunk of data belonging to an error response; well, indeed, it is the time thttpd waits while data count in send socket buffer becomes less than LOW_MARK, usually 4 KB for a 16 KB sized socket send buffer; NOTE: if you want to be absolutely sure that big error responses are correctly sent, then set this value to, at least, 60 seconds; 120 - 300 seconds are better on very very slow (i.e. <= 9600 baud) and noisy links, but the higher is this value, the more likely thttpd is hurted by DoS (Denial of Service) attacks. - added a final write() in idle_send_resp_connection(), thttpd.c, before lingering and/or closing socket (maybe there is enough room in socket buffers to write all left data to be sent). File-Cache ---------- o occasional-mmc-cleanup (A.D.F.) improved control over "mmap storms" on servers with high connection rates and a large set of files (much larger than DESIRED_MAX_MAPPED_FILES): - config.h, replaced OCCASIONAL_TIME (300) with: - OCCASIONAL_MMC_TIME ( 32); - OCCASIONAL_TMR_TIME (300); - thttpd.c, add a second "occasional" timer: - occasional_mmc(); - occasional_tmr(). o mmap-cache-expire-age-calc (A.D.F.) - tuned granularity of expire_age calculation, mmc_cleanup(), mmc.c. NOTE: mmap() storms may lead thttpd to refuse serving new requests due to errors 500 that may be caused by new mmap() failures (system limit for max. number of mmap() has been reached); in this case you have to wait periodic (occasional_mmc()) memory map cache cleanup to evict / remove expired "mmaps" from file cache (in thttpd 2.21b, by default, a mapped file expire after 1 - 20 minutes); expire time is dynamically setted depending on number of active "mmapped" files in cache. NOTE: if you have a server with a lot of traffic, and a large set of small files you may want to: - increase DESIRED_MAX_MAPPED_FILES value (config.h) from 2000 to 3000, 4000 or more (this should lower CPU usage if you have enough RAM); - lower the value of OCCASIONAL_MMC_TIME (config.h) from 32 to 24 or 16 seconds (but higher than 8-9 seconds); this should prevent mmap() failures due to system limit. NOTE: if you have a server with a lot of traffic, and a large set of big files you may want to: - decrease DESIRED_MAX_MAPPED_FILES value (config.h) from 2000 to 1000, 500 or less (this should avoid mmap() failures due to the amount of memory mapped files in RAM, total amount should be less than 1, 2 or 3 GB under 32 bit OS); - lower the value of OCCASIONAL_MMC_TIME (config.h) from 32 to 24 or 16 seconds (keep it higher than 8-9 seconds); this should prevent mmap() failures due to system limit. Configuration ------------- o mmap-cache-parameters (A.D.F.) - added the following parameters to configuration file (see read_config() in thttpd.c), so that thttpd administrator can play more easily with mmap cache parameters (without having to recompile thttpd): - DEFAULT_EXPIRE_AGE=NumSecs # NumSecs: (10 - 30000) - DESIRED_MAX_MAPPED_FILES=NumMap # NumMap: ( 2 - 100000) - OCCASIONAL_MMC_TIME=NumSecs # NumSecs: ( 8 - 1000) Hint: do not add spaces around equal ('=') sign; - added two new functions: mmc_cfg_get_param(); mmc_cfg_set_param(); in mmc.c, to get / set file map cache parameter values. NOTE: now, above three defined values in config.h work even if config.h is not included in package mmc.c (this because values are re-setted at run-time). o expire-age-config (A.D.F.) - added #define DEFAULT_EXPIRE_AGE constant to file config.h; - re-increased DESIRED_MAX_MAPPED_FILES from 1000 to 2000 in config.h; this because thttpd is supposed to serve relatively small files (0 - 128 KB), if this is not your case then tune file map cache (see above 3 parameters). Log / Information ----------------- o log-parameter-values (A.D.F.) - added a LOG_NOTICE reporting values of file map parameters (executed at startup). o log-sigusr1 (A.D.F.) - added a LOG_NOTICE whenever thttpd catches a SIGUSR1 signal (shutdown all connections and exit). ---------------- thttpd-2.21b-p11 02-Jan-2002 ---------------- Cleanups -------- o shutdown-constant (Jef Poskanzer) - added constant SHUT_WR as second parameter for shutdown(). o remove-printf-comments (A.D.F.) - removed commented printf(), left by keep-alive patch, from thttpd.c and libhttpd.c o document-missing-option (Pavel Janik) - added [-s|-nos] to function usage() in thttpd.c ---------------- thttpd-2.21b-p10 17-Nov-2001 ---------------- Fixes ----- o restore-keep-alive (A.D.F.) - restored keep alive functionality that was broken in thttpd-p09 (this was only a regression to HTTP/1.0 in fact everything worked well). ---------------- thttpd-2.21b-p09 16-Nov-2001 ---------------- Fixes ----- o write-response (A.D.F.) - added a new state (CNST_SENDING_RESP) to write error response so that it can use non-blocking mode instead of blocking mode; this should: - make thttpd a bit faster (avoid no-block -> block -> no-block mode switch) under heavy load with other processes competing for CPU usage; - remove a possible extremely rare block condition with keep-alive, when serving lots of requests on a saturated connection that suddenly blocks; - added a few functions in thttpd.c: - resp_clear_connection(); - handle_send_resp(); - idle_send_resp_connection(); - etc. - added new parameter IDLE_SEND_RESP_TIMELIMIT in config.h (you may want to change default value of 5 seconds). Cleanups -------- o cleanups (A.D.F.) - changed function name: from httpd_close_outconn() to httpd_close_conn_wr() ---------------- thttpd-2.21b-p08 07-Nov-2001 ---------------- Fixes ----- o poll-handling (Sascha Schumann) - added missing POLL* error events to functions poll_watch() and poll_check_fd() in fdwatch.c. o libhttpd-close (George Schlossnagle) - added a missing close() to httpd_get_conn() in libhttpd.c. o off-by-one-patch in read()/write(). (David Koblas) o memory-copy-overlaps (A.D.F.) - substituted memcpy() for memmove() when copying http headers in handle_send(), thttpd.c (source and target area may overlap). o SIGUSR1-wait-active-connections (A.D.F.) - forced exit even if there are no active connections (main(), thttpd.c). Performance ----------- o keep-alive-patch (David Koblas) - added "keep alive" functionality so that more than one client request can be handled for each socket connection (in HTTP/1.1 this feature is called "persistent connections"). NOTE: to exploit this speed up and reduce (a bit) "nagle algorithm" latency (up to 200 mlsec.), you have to: - keep "round time trip" of TCP/IP packets + thttpd time needed to send first response less than 180 - 200 mlsec. (i.e. this is usually true in a LAN - Local Area Network); and / or - enable HTTP 1.1 "pipelining" in client browsers so that more than one HTTP request is sent before waiting for server reply. o keep-alive-defines (A.D.F.) - added *_KEEP_ALIVE defines in thttpd.c. o clear_connection-fast-close (A.D.F.) - added half connection close (shutdown()) in clear_connection(). o timers-shrinkage (A.D.F) - reduced size of fields periodic and hash from 4 to 2 bytes each (int to short) to force 32 byte alignment in timer struct. Tuning ------ o DESIRED_MAX_MAPPED_FILES (A.D.F.) - lowered from 2000 to 1000 (just a try). Test (help testing) ------------------- o all-signals (A.D.F.) - added signal() catches for all useful signals. o unique-exit-codes (A.D.F.) - set a unique number for each exit() call: 0 OK 1 - N_SIG when exiting because of a signal 60 - 99 thttpd.c exit 100 - 113 libhttpd.c exit #