5 November 2002: The SRE2003 API This technical document discusses features of SRE2003 that may be of interest to those interested in writing their own filters or scripts. You may also want to examine SRE2PRC.HTM (describes the SRE2003 procedure libarary) and SRE_DMN.TXT (describes how to use SRE2003 daemons, queues, and flags). Contents: I. Data structures supported by SRE2003 I.a. System Variables I.b. Global variables: I.c. Counter variables: I.d. Event Variables: I.e. Request specific variables. I.f. Request id variables. I.g. Caches 1.h. Queues 1.i Data series 1.j Repetitive tasks 2. Using the SRE2003 TEST Mode ------------------------------------------------- I. Data structures supported by SRE2003 SRE2003 supports the following data structures: System Variables: static variables set by SRE2003 at startup Global variables: dynamic variables that are available to all threads under SRE2003 Counter variables: counter variables are used to maintain running counts. Event variables: event variables change frequently, the most recent value (the event) is retained. Request specific variables: request specific variables are unique to a given request. There are two sets of request specific variables: pre-defined and custom. Request id variables: Request id variables are a small set of pre-defined request- specifc "identifiers". Caches: Caches are arrays of variables and their associated values. These variables can be randomly accessed. Caches have a finite size, when filled a LRU (least recently used) algorithim is used to free up space. Lists: Lists are lists of values. Values can be added or removed from the front or back of this hist. Lists have a finite size. When filled, a LIFO (or a FIFO) algorithim is used to free up space. Queues: Queues are arrays of values. Values can be added & removed from the front or back of queues, but only removed from the front. Queues have no size limit. I.a. System Variables Description: This is a set of about 50 variables that contain information on basic SRE2003 parameters. Essentially, these are the variables set in SRE2003.CFG. Although these parameters can not be changed on the fly, if SRE2003.CFG is changed these variables will be updated (it may take a few minutes). Reading & Writing: Use the SRE_VALUE function: aval=sre_value(varname,,'SRESYS') As noted in the description, these are semi-permanent; hence they can not be changed on the fly. To modify them, you have to edit SRE2003.CFG. List of system variables: Name Example ---------------------------------------------------------------------------- SRE2003_INI F:\test\temp\sre80.ini The SRE80_INI file is an OS/2 .INI file. System variables, and other information on flags and daemons, is stored there. This .INI file is not used by SRE2003, it is created as a convenience for other programs. LASTSTART 2001.12.10 12:57:53 The day and time that SRE2003 was started (local time). LISTEN_SOCK 699 The socket that SRE2003 is monitoring. PORT 80 The http port that SRE2003 is accepting requests on. LIMITBODY 500000 Max size of request body. LIMITCLIENTS 30 Max number of simultaneous clients. LIMITHEADER 10000 Max size of request header. LIMITTIMETOTAL 20 Max seconds per request LIMITTIMEWAIT 5 Max seconds to maintain a connection LIMITTIMEINACTIVE 10 Max seconds of connection inactivity (nothing sent or recieved) ENTITY_HEADERS_IGNORE A list of entity-headers that SRE2003 should ignore when automatically creating Etags. FILTER srelite2\srelite2.rxx The filter to use. FAST_FILTER 0 The fast-filter to use DEFAULT_HOSTS * List of hosts that will be assigned to the DEFAULT host-nickname. This complements the canonical host name. EXIT_PROC srelite2\srelexit.rxx List of procedures to run when exiting SRE2003. INIT_PROC bin\inithost.rxx,srelite2\SREL_INI.RXX List of procedures to run when starting SRE2003. DEFAULT_SOCKET_TIMEOUT 15 TIme out a socket after this many seconds of waiting for data SERVER_SOFTWARE SRE2003 for OS/2 ver 1.12d What SRE2003 reports as it's name. STATUS_ITEMS ERRORS LIMITS CLIENTS STAT CONNECTIONS PEAK Items to list on the status screen. ERRORFILE log\error.log Where to write errors. AUDITFILE log\audit.log Where to write audit information. AUDIT_ITEMS CLIENT RESET What types of event to audit. SERVER_SOFTWARE_SHORT SRE2003 1.12d Short descriptive name for SRE2003 UNALLOWED_FILE F:\test\cfg\BADIPS.IN File listing "unallowed IP addresses" NO_MODIFY_ONTHEFLY 0 Flag to suppress the ALT-M option for "on the fly" modification of SRE2003. AUDIT_DELAY 0.25 MInutes to delay writing to audit file (a "lazywrite" stall) MAXMD5S 500 Maximum entries in the MD5 cache. MAXCOUNTERS 200 Maximum number of counters. MAXCLIENTNAMES 120 Maximum entries on the clientnames cache MAXSTATVALS 100 Number of "most recent" requests to use in calculating performance statistics. REQ_CACHE_SIZE 500 Maximum number of request cache (the proxy-like cache) entries. REQ_CACHE_CALL_ANYWAYS 0 Call filter even on request cache hit. REQ_CACHE_ENABLE 1 Enable use of the request cache. REQ_CACHE_VERIFY 2 Check no-cache and pragma request headers before using the request cache. REQ_CACHE_RECORD 1 Record request cache hits in cachehit.log file. STATUS_MESSAGE Sre2 1.12 Short message written on side of status screen. TRACKING_ON 0 Enable SRE2003's request tracking mode. FILTER_VERSION sreLite2 ver 1.10a Descriptive name of the SRE2003 filter being used. SECURITY_LEVEL 1 Controls the ability to use on-line tools to modify SRE2003 parameters. GMTOFFSET 18000 GMT offset in seconds (add to local time to get GMT time) SERVERADDR_CANONICAL 113.12.55.36 The "canonical" IP address (as reported by TCP/IP). SRE2003 will report this value, unless overridden by a non-0 value of the SERVERADDR_USE parameter. SERVERNAME_CANONICAL www.foobar.net The "canonical" domain name (as reported by TCP/IP). SRE2003 will report this value, unless overridden by a non-0 value of the SERVERNAME_USE parameter. SERVDIR F:\test The directory containing SRE2003. SRV_DIR F:\test The directory containing SRE2003 (same as SERVDIR) SERVDRIVE F: The drive ontaining SRE2003. TEMPDIR F:\test\TEMP A temporary files directory. TMP_DIR F:\test\TEMP Same as TMP_DIR TEMP_DIR F:\test\TEMP Same as TMP_DIR SRE2003_CFG F:\test\cfg\SRE2003.CFG The SRE2003 configuration file. BIN_DIR F:\test\BIN Directory containing SRE2003 code. CFG_DIR F:\test\CFG Directory containing SRE2003 configuration files (SRE2003.CFG, BADIPS.IN, HOSTINFO.CFG). LOG_DIR F:\test\LOG Directory containing SRE201 log files. STORED_DATA_DIR F:\test\DATA Directory for storing miscellaneous data files (for use by filters). REQ_CACHE_HITFILE F:\test\log\CACHEHIT.LOG File containing request cache hit log. DATADIR F:\WWW The datadir -- requests for files are assumed to be relative to this directory. FILTER_DIR F:\test\srelite2 The directory containing the SRE2003 filter you are using. HOSTFILE F:\test\cfg\hostinfo.cfg The file containing host definitions. For more details on these "system" variables, see SRE2003.HTM I.b. SRE "internal", and "user", global variables: Description: Similar to system variables in that they are available to all threads, but they are set dynamically (that is, they can be changed on the fly). There are two kinds of "global" variables: SRE (internal) and GLOBAL (user). They are stored in seperate locations, which means that each can have variables with the same name (and different values). SRE2003 makes some use of the "internal" global variables -- for storing temporary counters, event variables, and small caches. In addition, several filters (such as sreLite2 and SREhttp/2) use global variables to store a number of configuration parameters. Reading & Writing: Use the SRE_VALUE function: avalue=sre_value(varname,newval,'SRE') -- for "internal globals" avalue=sre_value(varname,newval,'GLOBAL') -- for "user globals" or avalue=sre_value(varname,newval,'SRELARGE',[ownid]) The first and second are used for quick retrieval of small "textual" variables. In general, casual programmers (such as script authors) should use GLOBALS, while filter and IM-module authors can (carefully) use SRE. The third form (SRELARGE) is used for not-as-quick retrieval of any-sized, any-content variables. If newval is NOT specified, then read current value of varname. newval='' means "read current value" newval=' ' means "set varname equal to ' '" Ownid is strictly optional. It is the "id" of the request (say, as provided by the sre_dmn_own procedure). Including it will speed up throughput a bit. Notes: * Global variables are accessible from all threads (daemons) in the SRE2003 process. However, we recommend that the writing of global variables be limited to only one thread; although there are no limits on who can read global values. * They are not easily accessible from other processes. * Servicing (reading and writing) SRELARGE global variables is done via a dedicated daeamon (the SRE_VAR.RXX daemon). That means it's not very fast. * SRE2003 reserves several "internal" global variables for use as dynamically changing counters, and for transaction and request specific information. These are: TRAN_INFO_CT, REQ_INFO_CT, CLIENTNAME_CACHE, HOSTNAME_CACHE, several versions of REQ_nn_INFO where nn are integer "thread ids" several CT_varname variables, where varname is one of: BYTESREADTOTAL BYTESSENTTOTAL TRANSACTIONS REQUESTS LIMITS ERRORS BROKEN CACHEHITS) several EVENT_varname variables, where varname is one of: LASTACCEPT LASTIDLE LASTRESET LASTADDRESS PEAKCLIENTS I.c. Counter variables: Description: Counter variables are a special kind of global variable. They are used to maintain running counts. Morever, for a "quicklist", of several variable, a speed optimization is supported. Reading & Writing: Use the SRE_COUNTVARS function: a_count=SRE_COUNTVARS(varname,avalue,ownid) where: varname: the counter variable. If you begin varname with a "=" (without the quotes) then "force" an accurate read avalue: the value to add If avalue is not specified, return current value of varname ownid: optional, the clients "own_id". The Counter Quicklist: The "counter quicklist" contains the following variables: BYTESREADTOTAL BYTESSENTTOTAL TRANSACTIONS REQUESTS LIMITS ERRORS [see SRE2003.HTM for a decription of these counter variables] Variables that are in this quicklist are read using a speed optimization. However, use of this speed optimization may yield inaccurate results, if the count is changing rapidly. In particular, the value returned may not be up-to-date. If you require an accurate (up-to-date) value for a quicklist variable, use the "prepend varname with an = sign" (as noted above). For example: accurate_ct=SRE_COUNTVARS('=TRANSACTIONS') Notes: * For further details on SRE_COUNTVARS, see SRE2PRC.HTM * By "up-to-date", we mean "within the last few seconds" I.d. Event Variables: Description: Event variables are global variables that change frequently. They are similar to count variables (they change frequently), but instead of augmenting a counter, the most recent value (the event) is retained. As with counter variables, an "optimized for speed quicklist" of event variables is maintained. Reading & Writing: Use the SRE_EVENTVARS function: oldval=sre_eventvars(varname,newvalue,ownid) where: varname: the event variable. If you begin varname with a "=" (without the ") then "force" an up to date read. newvalue: the value to store If avalue is not specified, then return current value of varname ownid: optional, the clients "own_id". This is used ONLY if you are reading the value of a non "quick list" event. If not specified, an own_id will be generated automatically. For further details, see SRE2PRC.HTM The Event Quicklist: The "event quicklist" contains the following variables: LASTACCEPT LASTIDLE LASTRESET LASTADDRESS [see SRE2003.HTM for a decription of these event variables] Notes: * For further details on SRE_EVENTVARS, see SRE2PRC.HTM I.e. Request specific variables. Description Request specific variables are unique to a given request. There are two sets of request specific variables: pre-defined and custom. Reading & Writing: The pre-defined request specific variables can be read using the EXTRACT or SRE_COMMAND functions -- see SRE2003.HTM for the details. The custom variables can be read and written with the SRE_VALUE function. aval=sre_value(varname,newval,'REQ') As defined above, if newval is not specified, the value of varname is read. If newval is specified, varname will be given the value of newval. Each request has it's own custom variables. That is: custom variables are defined for requests, not for transactions. Hence, each request in a "maintained connection" will have it's own values (that is, values do not carry over to the next request read from a given connection). I.f. Request id variables. Request id variables are unique to a given request. They are a small set of pre-defined request-specifc "identifiers". In particular: CLIENT - client id of this request HOST - host name NTH - the transaction number (which will be the same for maintained-connection requests) OWN - "own id" of this reqeust (used to "ask" for info) PARENTID - the thread id of this requests "parent" (a transaction daemon) SEL - selector SOCK - socket of this request START - starttime (date & second) TRAN - return transaction id of this request (the transaction of this request) Read & Write aval=sre_request_info(avar,tid,ownid) where: avar - variable to lookup tid - optional. The request identifier. If not specified, lookup info for the current request. Or, the tid can be the "id_info" (provided as the 5th argument to the filter) -- using this can speed up processing a bit. ownid - optional. The caller's "own id". This can speed up some lookups a little bit. See SRE2003.HTM for the details. I.g. Caches Description: Caches are arrays of variables and their associated values. Each entry (array & variable) in a cache can be read independently (using the variable name). One can freely add new entries to a cache, up until the cache is full. When the cache is full, new entries will cause old entries (in the least recently used sense) to be removed (so as to make room for new variables). Reading & Writing: Use the SRE_CACHE function. val=sre_cache(cache_name,action,varname,varvalue,time_to_live,ownid) where cache_name: The cache to work with. Each cache is seperately maintained -- each cache has it's own size limit. action: What to do. Supported actions include: CREATE RESET WRITE ADD READ PRUNE REMOVE STATUS LIST varname: The variable to read, write, etc. varvalue: The value of the variable (used with WRITE and ADD) time_to_live: The lifespan of this cache entry. After this many minutes, the entry is removed ownid: Optional. The ID of the daemon that is calling sre_cache. For a complete description of SRE_CACHE, sre SRE2PRC.HTM. SRE2003 internal caches. SRE2003 creates and uses the following caches for internal purposes: REQUEST: the request cache (for quickly matching URIs to files) IPNAME: the IP name cache (for matching ip addresses to domain names) MD5: the MD5 cache (for matching files to their MD5 hashes) COUNTS: for counters (such as BYTESREADTOTAL and LIMITS) EVENTS: for retaining events (such as LASTACCEPT and PEAKCLIENTS) REQ_INFO: queue and semaphore information for request daemons TRAN_INFO: queue and semaphore information for transaction daemons Notes: * Caches are implemented through the SRE_CSH.RXX daemon. * If cachename is "!LIST" (without the quotes), a list of currently active caches is returned. 1.h. Lists Description Lists are arrays of values. Values can be added to the front or back of the list, and can also be read or removed from either end. Values can be added until the size of the list is reached, at which point when a new value is added, an old value is removed (just one value is remove). A list differs from queues and caches in several ways: i) Compare to a cache; there is no variable name associated with each entry in a list. There is a position (defined from the bottom of the list), though this position can change as new items are added ii) There is no least recently used, or time to live, mechanism for selecting which items to remove when more space is needed. Instead, the "oldest" element is dropped; where "old" simply means a value on the other end of the list! iii) A queue can only have it's top element read. if) The size of a queue is unlimited. Basically -- Caches are random access with a dynamic LRU algorithim used to remove old entries. List are sequential, with a simple overflow mechanism used to remove old entries. Queues are sequential with no internal readsd, and have no size limit. Reading & Writing: Use the SRE_LIST function: val=sre_list(list_name,action,avalue,ownid) where: list_name: The list to work with. Each list is seperately maintained -- each list has it's own size limit. action: What to do. Supported actions include: CREATE RESET PUSH QUEUE POP PEEK READ STATUS LIST avalue: A value to add to the queue ownid: Optional. The ID of the daemon that is calling sre_list SRE2003 internal queues: SRE2003 creates and uses the following lists for internal purposes -- they retain information for the last hundred or so requests: BYTES : bytes transmitted TOTAL_TIME : time required to complete request START_TRANSMIT_TIME : time required before first byte was sent CLIENTADDR : client's IP address REQUEST : the requst line For details on these "status" variables, see SRE2003.HTM. Notes: * Queues are implemented through the SRE_LIST.RXX daemon. * For further details on SRE_LIST, see SRE2003.HTM. 1.i. Data Series SRE2003's "data series" are simple databases that are designed to be quickly accessed. Compared to "caches", data-series are permanent -- they are stored in fixed-length text files. Data series do have some cache-like features; with recently read or modified values retained in memory. Features of data series: * You can have several data series open at the same time. * Each record in a data series is referenced by an arbitrarily long identifier string. * Random access, using a memory resident index, is used to lookup records. * Each record can have up to 15 unique variables assigned to it. * A few simple "increment" functions are easily implemented with data series. * SRE2003 data series are designed to be used with small to mid sized database. Large databases (say, greater the 10,000 records), or wide database (say, greater then 1000 bytes per record) are not efficiently supported as an SRE2003 data series. * SRE2003 data series are fairly robust, but they should not be used with crucial transaction data. That is, on occassion SRE2003 may fail to record a change (say, when the server prematurely closes). Reading and writing: Use the SRE_CACHE_DATA function to create, write, read, and delete SRE2003 data series. The basic syntax of SRE_CACHE_DATA is: aresult=sre_cache_date(series,action [options],identifier,val1,..,val10) where: series: a data series. action: what to do. The following actions are available: DEFINE : create a data series. OPEN : open a currently existing data series ADD : add a new record MODIFY : modify a record EXTRACT : Extract one or several records, by record number FIND : find a record, or records REMOVE : remove a record CLOSE : close the data series (save cached information) INFO : info about a data series SAVE : save any cached information CLEANUP : cleanup data REWRITE : rewrite the data file (fix odd records, remove old junk) DELETE : delete a data series [options] Space delimited list of optional action modifiers. identifier : an identifier val1 .. val10 : up to 10 values, whose used depends on action aresult: depends on action. For the details on working with SRE2003 data series, and on how to use the SRE_CACHE_DATA procedure, see SRE_DCSH.TXT. 1.j. Tasks The SRE2003 task manager is a facility that simplifies the execution of non-request related tasks. For example, the task manager can be used to schedule recurring housekeeping tasks (such as cleaning up temporary storage), or to produce output that a client will obtain at a later date. Running tasks is straightforward process of defining the task, and then launching instances of this task as needed (or scheduling instances to run a number of times). The main advantage of using the SRE2003 task manager, as opposed to a CRON like system utility, is that tasks can be easily launched by daemons, or by web based scripts. Using tasks: The SRE_TASK procedure is used to specify, invoke, and other work with tasks. Syntax: astat=sre_task(action,name,controls,data) where: ACTION The action to take. Four actions are recognized: * FAMILY = define a family (and set some parameters) * DEFINE = define a task (assign a task NAME to a rexx procedure), or remove an already defined task * LAUNCH = launch a task * STATS = compute some statistics NAME is a task name or a family name. The task name can also include a family name. CONTROLS is optional control information. DATA is used to pass ACTION specific parameters. ASTAT depends on action. For more details on tasks, and on how to use SRE_TASK, see SRE_TASK.HTM. ------------------------------------------------- 2. Using the SRE2003 TEST Mode To facilitate quick tests of a running server, SRE2003 supports a simple TEST mode. The test mode uses a special TEST verb (which is NOT a standard http/1.1 protocol). When a TEST request is recieved by SRE2003, it looks for a modifier (in the selector portion of the request), and sends back a simple 1 line response. That is, only a response line is returned -- no headers, and no body, are returned to the client. Several "modifiers" are supported: ADDRESS == Returns the server's dotted numeric IP address CLIENT == Return's the client's dotted numeric IP address NAME == Returns the server's IP name (domain name) PING == Returns a "1" PORT == Returns the server's port (typically, 80) TIME == Returns the time and date TRANSACTION == Returns the current transaction number Examples: Request: TEST NAME HTTP/1.1 Response: 200 foo.bar.net http/1.1 Request: TEST ADDRESS HTTP/1.1 Response: 200 121.151.2.5 http/1.1 Request: TEST PING HTTP/1.1 Response: 200 1 http/1.1 Request: TEST time HTTP/1.1 Response: 200 02:58:07 21 Jul 2001 http/1.1 Notes: * The DOTEST.CMD program demonstrates how the TEST mode is used. * The TEST mode is used by the SREWATCH program * SRE2003 will count TEST mode requests as transactions, but will i) NOT record information to any AUDIT file. ii) NOT call the filer