From 9401f816dd0d9d550fe98a8507224bde51c4b847 Mon Sep 17 00:00:00 2001 From: hongbotian Date: Mon, 30 Nov 2015 02:41:33 -0500 Subject: upload tomcat JIRA: BOTTLENECK-7 Change-Id: I875d474869efd76ca203c30b60ebc0c3ee606d0e Signed-off-by: hongbotian --- .../docs/generic_howto/timeouts.html | 374 +++++++++++++++++++++ 1 file changed, 374 insertions(+) create mode 100644 rubbos/app/tomcat-connectors-1.2.32-src/docs/generic_howto/timeouts.html (limited to 'rubbos/app/tomcat-connectors-1.2.32-src/docs/generic_howto/timeouts.html') diff --git a/rubbos/app/tomcat-connectors-1.2.32-src/docs/generic_howto/timeouts.html b/rubbos/app/tomcat-connectors-1.2.32-src/docs/generic_howto/timeouts.html new file mode 100644 index 00000000..67fcf90b --- /dev/null +++ b/rubbos/app/tomcat-connectors-1.2.32-src/docs/generic_howto/timeouts.html @@ -0,0 +1,374 @@ +The Apache Tomcat Connector - Generic HowTo - Timeouts HowTo

Links

Docs Home

Reference Guide

Generic HowTo

Webserver HowTo

AJP Protocol Reference

Miscellaneous Documentation

News

The Apache Tomcat Connector - Generic HowTo

Timeouts HowTo

print-friendly
version +

Introduction

+
+
Setting communication timeouts is very important to improve the +communication process. They help to detect problems and stabilise +a distributed system. JK can use several different timeout types, which +can be individually configured. For historical reasons, all of them are +disabled by default. This HowTo explains their use and gives +hints how to find appropriate values. +
+
All timeouts can be configured in the workers.properties file. +For a complete reference of all worker configuration +items, please consult the worker reference. +This page assumes, that you are using at least version 1.2.16 of JK. +Dependencies on newer versions will be mentioned where necessary. +
+
+Do not set timeouts to extreme values. Very small timeouts will likely +be counterproductive. +
+
+Long Garbage Collection pauses on the backend do not make a good +fit with some timeouts. Try to optimise your Java memory and GC settings. +
+

JK Timeout Attributes

+
+
CPing/CPong
+
+CPing/CPong is our notion for using small test packets to check the +status of backend connections. JK can use such test packets directly after establishing +a new backend connection (connect mode) and also directly before each request gets +send to a backend (prepost mode). +Starting with version 1.2.27 it can also be used when a connection was idle +for a long time (interval mode). +The maximum waiting time (timeout) for a CPong answer to a CPing and the idle +time in interval mode can be configured. +
+
+The test packets will be answered by the backend very fast with a minimal amount of +needed processing resources. A positive answer tells us, that the backend can be reached +and is actively processing requests. It does not detect, if some context is deployed +and working. The benefit of CPing/CPong is a fast detection of a communication +problem with the backend. The downside is a slightly increased latency. +
+
+The worker attribute ping_mode can be set to a combination of characters +to determine, in which situations test packets are used: +
+
C: connect mode, timeout ping_timeout overwritten by connect_timeout
+
P: prepost mode, timeout ping_timeout overwritten by prepost_timeout
+
I: interval mode, timeout ping_timeout, idle time connection_ping_interval
+
A: all modes
+
+
+
+Multiple values must be concatenated without any separator characters. +We recommend using all CPing tests. If your application is very latency sensitive, then +you should only use the combination of connect and interval mode. +
+
+Activating the CPing probing via ping_mode has been added in version 1.2.27. +For older versions only the connect and prepost modes exist and must be activated by +explicitely setting connect_timeout and prepost_timeout. +
+
+The worker attribute ping_timeout sets the default wait timeout +in milliseconds for CPong for all modes. By default the value is "10000" +milliseconds. The value only gets used, if you activate CPing/Cpong probes +via ping_mode. The default value should be fine, except if you experience +very long Java garbage collection pauses. +Depending on your network latency and stability, good custom values +often are between 5000 and 15000 milliseconds. +You can overwrite the timeout used for connect and prepost mode with +connect_timeout and prepost_timeout. +Remember: don't use extremely small values. +
+
+The worker attribute connect_timeout sets the wait timeout +in milliseconds for CPong during connection establishment. You can use it +if you want to overwrite the general timeout set with ping_timeout. +To use connect mode CPing, you need to enable it via ping_mode. +Since JK usually uses persistent connections, opening new connections is a +rare event. We therefore recommend activating connect mode. +Depending on your network latency and stability, good values often +are between 5000 and 15000 milliseconds. +Remember: don't use extremely small values. +
+
+The worker attribute prepost_timeout sets the wait timeout +in milliseconds for CPong before request forwarding. You can use it +if you want to overwrite the general timeout set with ping_timeout. +To use prepost mode CPing, you need to enable it via ping_mode. +Activating this type of CPing/CPong adds a small latency to each +request. Usually this is small enough and the benefit of CPing/CPong is more important. +So in general we also recommend using prepost_timeout. +Depending on your network latency and stability, good values often +are between 5000 and 10000 milliseconds. +Remember: don't use extremely small values. +
+
+Until version 1.2.27 ping_mode and ping_timeout did not +exist and to enable connect or prepost mode CPing you had to set connect_timeout +respectively prepost_timeout to some reasonable positive value. +
+
+ +
Low-Level TCP Timeouts
+
+Some platforms allow to set timeouts for all operations on TCP sockets. +This is available for Linux and Windows, other platforms do not support this, +e.g. Solaris. If your platform supports TCP send and receive timeouts, +you can set them using the worker attribute socket_timeout. +You can not set the two timeouts to different values. +
+
+JK will accept this attribute even if your platform does not support +socket timeouts. In this case setting the attribute will have no effect. +By default the value is "0" and the timeout is disabled. +You can set the attribute to some seconds value (not: milliseconds). +JK will then set the send and the receive timeouts of the backend +connections to this value. The timeout is low-level, it is +used for each read and write operation on the socket individually. +
+
+Using this attribute will make JK react faster to some types of network problems. +Unfortunately socket timeouts have negative side effects, because for most +platforms, there is no good way to recover from such a timeout, once it fired. +For JK there is no way to decide, if this timeout fired because of real network +problems, or only because it didn't receive an answer packet from a backend in time. +So remember: don't use extremely small values. +
+
+For the general case of connection establishment you can use +socket_connect_timeout. It takes a millisecond value and works +on most platforms, even if socket_timeout is not supported. +We recommend using socket_connect_timeout because in some network +failure situations failure detection during connection establishment +can take several minutes due to TCP retransmits. Depending on the quality +of your network a timeout somewhere between 1000 and 5000 milliseconds +should be fine. Note that socket_timeout is in seconds, and +socket_connect_timeout in milliseconds. +
+
+ +
Connection Pools and Idle Timeouts
+
+JK handles backend connections in a connection pool per web server process. +The connections are used in a persistent mode. After a request completed +successfully we keep the connection open and wait for the next +request to forward. The connection pool is able to grow according +to the number of threads that want to forward requests in parallel. +
+
+Most applications have a varying load depending on the hour of the day +or the day of the month. Other reasons for a growing connection pool +would be temporary slowness of backends, leading to an increasing +congestion of the frontends like web servers. Many backends use a dedicated +thread for each incoming connection they handle. So usually one wants the +connection pool to shrink, if the load diminishes. +
+
+JK allows connections in the pool to get closed after some idle time. +This maximum idle time can be configured with the attribute +connection_pool_timeout which is given in units of seconds. +The default value is "0", which disables closing idle connections. +
+
+We generally recommend values around 10 minutes, so setting +connection_pool_timeout to 600 (seconds). If you use this attribute, +please also set the attribute connectionTimeout in the AJP +Connector element of your Tomcat server.xml configuration file to +an analogous value. Caution: connectionTimeout is in milliseconds. +So if you set JK connection_pool_timeout to 600, you should set Tomcat +connectionTimeout to 600000. +
+
+JK connections do not get closed immediately after the timeout passed. +Instead there is an automatic internal maintenance task +running every 60 seconds, that checks the idle status of all connections. +The 60 seconds interval +can be adjusted with the global attribute worker.maintain. We do not +recommend to change this value, because it has a lot of side effects. +Until version 1.2.26, the maintenance task only runs, if requests get +processed. So if your web server has processes that do not receive any +requests for a long time, there is no way to close the idle connections +in its pool. Starting with version 1.2.27 you can configure an independent +watchdog thread when using Apache 2.x with threaded APR or IIS. +
+
+The maximum connection pool size can be configured with the +attribute connection_pool_size. We generally do not recommend +to use this attribute in combination with Apache httpd. For +Apache httpd we automatically detect the number of threads per +process and set the maximum pool size to this value. For IIS we use +a default value of 250 (before version 1.2.20: 10), +for the Sun Web Server the default is "1". +We strongly recommend adjusting this value for IIS and the Sun Web Server +to the number of requests one web server process should +be able to send to a backend in parallel. You should measure how many connections +you need during peak hours without performance problems, and then add some +percentage depending on your growth rate etc. Finally you should check, +whether your web server processes are able to use at least as many threads, +as you configured as the pool size. +
+
+The JK attribute connection_pool_minsize defines, +how many idle connections remain when the pool gets shrunken. +By default this is half of the maximum pool size. +
+
+ +
Firewall Connection Dropping
+
+One particular problem with idle connections comes from firewalls, that +are often deployed between the web server layer and the backend. +Depending on their configuration, they will silently drop +connections from their status table if they are idle for to long. +
+
+From the point of view of JK and of the web server, the other side +simply doesn't answer any traffic. Since TCP is a reliable protocol +it detects the missing TCP ACKs and tries to resend the packets for +a relatively long time, typically several minutes. +
+
+Many firewalls will allow connection closing, even if they dropped +the connection for normal traffic. Therefore you should always use +connection_pool_timeout and +connection_pool_minsize on the JK side +and connectionTimeout on the Tomcat side. +
+
+Furthermore using the boolean attribute socket_keepalive you can +set a standard socket option, that automatically sends TCP keepalive packets +after some idle time on each connection. By default this is set to "False". +If you suspect idle connection drops by firewalls you should set this to +"True". +
+
+Unfortunately the default intervals and algorithms for these packets +are platform specific. You might need to inspect TCP tuning options for +your platform on how to control TCP keepalive. +Often the default intervals are much longer than the firewall timeouts +for idle connections. Nevertheless we recommend talking to your firewall +administration and your platform administration in order to make them agree +on good configuration values for the firewall and the platform TCP tuning. +
+
+In case none of our recommendations help and you are definitively having +problems with idle connection drops, you can disable the use of persistent +connections when using JK together with Apache httpd. For this you set +"JkOptions +DisableReuse" in your Apache httpd configuration. +This will have a huge negative performance impact! +
+
+ +
Reply Timeout
+
+JK can also use a timeout on request replies. This timeout does not +measure the full processing time of the response. Instead it controls, +how much time between consecutive response packets is allowed. +
+
+In most cases, this is what one actually wants. Consider for example +long running downloads. You would not be able to set an effective global +reply timeout, because downloads could last for many minutes. +Most applications though have limited processing time before starting +to return the response. For those applications you could set an explicit +reply timeout. Applications that do not harmonise with reply timeouts +are batch type applications, data warehouse and reporting applications +which are expected to observe long processing times. +
+
+If JK aborts waiting for a response, because a reply timeout fired, +there is no way to stop processing on the backend. Although you free +processing resources in your web server, the request +will continue to run on the backend - without any way to send back a +result once the reply timeout fired. +
+
+JK uses the worker attribute reply_timeout to set reply timeouts. +The default value is "0" (timeout disabled) and you can set it to any +millisecond value. +
+
+In combination with Apache httpd, you can also set a more flexible reply_timeout +using an httpd environment variable. If you set the variable JK_REPLY_TIMEOUT +to some integer value, this value will be used instead of the value in +the worker configuration. This way you can set reply timeouts more flexible +with mod_setenvif and mod_rewrite depending on URI, query string etc. +If the environment variable JK_REPLY_TIMEOUT is not set, or is set to a +negative value, the default reply timeout of the worker will be used. If +JK_REPLY_TIMEOUT contains the value "0", then the reply timeout will be disabled +for the request. +
+
+In combination with a load balancing worker, JK will disable a member +worker of the load balancer if a reply timeout fires. The worker will then +no longer be used until it gets recovered during the next automatic +maintenance task. Starting with JK 1.2.24 you can improve this behaviour using +max_reply_timeouts. This +attribute will allow occasional long running requests without disabling the +worker. Only if those requests happen to often, the worker gets disabled by the +load balancer. +
+
+

Load Balancer Error Detection

+
+
Local and Global Error States
+
+A load balancer worker does not only have the ability to balance load. +It also handles stickyness and failover of requests in case of errors. +When a load balancer detects an error on one of its members, it needs to +decide, whether the error is serious, or only a temporary error or maybe +only related to the actual request that was processed. Temporary errors +are called local errors, serious errors will be called global errors. +
+
+If the load balancer decides that a backend should be put into the global error +state, then the web server will not send any more requests there. If no session +replication is used, this means that all user sessions located on the respective +backend are no longer available. The users will be send to another backend +and will have to login again. So the global error state is not transparent to the +users. The application is still available, but users might loose some work. +
+
+In some cases the decision between local error and global error is easy. +For instance if there is an error sending back the response to the client (browser), +then it is very unlikely that the backend is broken. +So this situation is a typical example of a local error. +
+
+Some situations are harder to decide though. If the load balancer can't establish +a new connection to a backend, it could be because of a temporary overload situation +(so no more free threads in the backend), or because the backend isn't alive any more. +Depending on the details, the right state could either be local error or global error. +
+
+
Error Escalation Time
+
+Until version 1.2.26 most errors were interpreted as global errors. +Starting with version 1.2.27 many errors which were previously interpreted as global +were switched to being local whenever the backend is still busy. Busy means, that +other concurrent requests are send to the same backend (successful or not). +
+
+In many cases there is no perfect way of making the decision +between local and global error. The load balancer simply doesn't have enough information. +In version 1.2.28 you can now tune, how fast the load balancer switches from local error to +global error. If a member of a load balancer stays in local error state for too long, +the load balancer will escalate it into global error state. +
+
+The time tolerated in local error state is controlled by the load balancer attribute +error_escalation_time (in seconds). The default value is half of recover_time, +so unless you changed recover_time the default is 30 seconds. +
+
+Using a smaller value for error_escalation_time will make the load balancer react +faster to serious errors, but also carries the risk of more often loosing sessions +in not so serious situations. You can lower error_escalation_time down to 0 seconds, +which means all local errors which are potentially serious are escalated to global errors +immediately. +
+
+Note that without good basic error detection the whole escalation procedure is useless. +So you should definitely use socket_connect_timeout and activate CPing/CPong +with ping_mode and ping_timeout before thinking about also tuning +error_escalation_time. +
+
+

\ No newline at end of file -- cgit 1.2.3-korg