Php文档 Php问答行业资讯 Php论坛 Php手册 Php博客

游戏榜单

软件榜单

关闭导航

热搜榜

热门下载

热门标签

关闭搜索

php爱好者> php文档>resin clustering

resin clustering

时间：2008-12-17 来源：admin126com

resin clustering

caching

database

As traffic increases beyond a single server, Resin's clustering lets you add new machines to handle the load and simultaneously improves uptime and reliability by failing over requests from a downed or maintenance server to a backup transparently.

Load Balancing
1. Two Server Configuration
2. Socket Pooling, Timeouts, and Failover
3. Dispatching
Persistent Sessions
1. Choosing a backend server
2. <persistent-store>
3. File Based
4. Distributed Sessions
  1. Symmetrical Sessions
  2. Sticky Sessions
  3. always-load-session
  4. always-save-session
5. Database Based
6. Cluster Sessions
7. Clustered Distributed Sessions
8. Configuration
  1. always-save-session
  2. Serialization
9. Protocol Examples
  1. Session Request
  2. Sticky Session Request
  3. Disk copy
  4. Failover
  5. Recovery
  6. No Distributed Locking
See Also

Load Balancing

When your traffic requires multiple servers, your site will naturally split into two clusters: an app-tier of identical servers to handle the content and a web-tier of HTTP servers talking to the browsers, caching the content, and distributing the load to the app-tier servers.

Since each app-tier servers produces the same content, Resin configures them identically: the app-tier servers have the same virtual hosts, web-applications and servlets, and use the same resin.xml. Adding a new machine just requires adding a new <server> tag to the cluster.

The new server has a unique name like "app-b" and a TCP cluster-port consisting of an <IP,port>, so the other servers can communicate with it. Although you can start multiple Resin servers on the same machine, TCP requires the <IP,port> must be unique, so you might need to assign unique ports like 6801, 6802 for servers on the same machine. On different machines, you'll use unique IP addresses. Because the cluster-port is for Resin servers to communicate with each other, they'll typically be private IP addresses like 192.168.1.10 and not public IP addresses. In particular, the load balancer on the web-tier uses the cluster-port of the app-tier to forward HTTP requests.

The load balancer on the web-tier forwards requests to the app-tier, distributing the load evenly and skipping any app-tier server that's down for maintenance or restarting due to a crash. This failover capability increases reliability and improves the customer's experiency by making your site look like every server is always up. The load balancer also steers traffic from a user session to the same app-tier server, improving caching and session performance, i.e. it supports sticky sessions.

Two Server Configuration

For example, a site running Drupal on Resin might now need two app-tier servers to handle additional load as it grows traffic. In the following configuration, the two app-tier servers "app-a" and "app-b" both serve the Drupal content, while the web-tier server "web-a" handles the HTTP, caches content, and balances the load to the app-tier cluster.

The web-tier is configured with a <cache> tag for the caching, and uses <rewrite-dispatch> with a <load-balance> tag for the dispatching. In this case, we send all content to the app-tier. <rewrite-dispatch> is Resin's equivalent of the Apache mod_rewrite module, providing a powerful and detailed URL matching and decoding, so more complicated sites might load-balance based on the virtual host or URL.

Example: resin.xml for load balancing

<resin xmlns="http://caucho.com/ns/resin"> <cluster id="app-tier"> <server id="app-a" address="192.168.0.10" port="6800"/> <server id="app-b" address="192.168.0.11" port="6800"/> <host id=""> <web-app id="" root-directory="/var/www/htdocs"/> </host> </cluster> <cluster id="web-tier"> <server-default> <http port="80"/> </server-default> <server id="web-a" address="192.168.0.1" port="6800"/> <cache disk-size="1024M" memory-size="256M"/> <host id=""> <web-app id="/"> <rewrite-dispatch> <load-balance regexp="" cluster="app-tier"/> </rewrite-dispatch> </web-app> </host> </cluster> </resin>

All three servers will use the same resin.xml, which makes managing multiple servers easier. The servers are name by the server id attribute, which must be unique. When you start Resin, you'll use the server-id as part of the command line

Example: starting servers

192.168.0.10> java -jar lib/resin.jar -server app-a start 192.168.0.11> java -jar lib/resin.jar -server app-b start 192.168.0.1> java -jar lib/resin.jar -server web-a start

Since Resin lets you start multiple servers on the same machine, a small site might start the web-tier server and one of the app-tier servers on one machine, and start the second server on a second machine. You can even start all three servers on the same machine, increasing reliability and easing maintenance, without addressing the additional load. If you do put multiple servers on the same machine, remember to change the port to something like 6801, so the TCP bind doesn't conflict.

In the /resin-admin management page, you can manage all three servers at once, gathering statistics and load and watching for any errors. When setting up /resin-admin on a web-tier server, you'll want to remember to add a separate <web-app> for resin-admin to make sure the <rewrite-dispatch> doesn't inadvertantly send the management request to the app-tier.

Socket Pooling, Timeouts, and Failover

For efficiency, Resin's load balancer manages a pool of sockets connecting to the app-tier servers. If Resin forwards a new request to an app-tier server and it has an idle socket available, it will reuse that socket, improving performance an minimizing network load. Resin uses a set of timeout values to manage those idle sockets and to handle any failures or freezes of the backend servers. The following diagram illustrates the main timeout values:

load-balance-connect-timeout: the load balancer timeout for the connect() system call to complete to the app-tier (5s).
load-balance-idle-time: load balancer timeout for an idle socket before closing it automatically (5s).
load-balance-recover-time: the load balancer connection failure wait time before trying a new connection (15s).
load-balance-socket-timeout: the load balancer timeout for a valid request to complete (665s).
keepalive-timeout: the app-tier timeout for a keepalive connection (15s)
socket-timeout: the app-tier timeout for a read or write (65s)

When an app-tier server is down due to maintenance or a crash, Resin will use the load-balance-recover-time as a delay before retrying the downed server. With the failover and recover timeout, the load balancer reduces the cost of a failed server to almost no time at all. Every recover-time, Resin will try a new connection and wait for load-balance-connect-timeout for the server to respond. At most, one request every 15 seconds might wait an extra 5 seconds to connect to the backend server. All other requests will automatically go to the other servers.

The socket-timeout values tell Resin when a socket connection is dead and should be dropped. The web-tier timeout load-balance-socket-timeout is much larger than the app-tier timeout socket-timeout because the web-tier needs to wait for the application to generate the response. If your application has some very slow pages, like a complicated nightly report, you may need to increase the load-balance-socket-timeout to avoid the web-tier disconnecting it.

Likewise, the load-balance-idle-time and keepalive-timeout are a matching pair for the socket idle pool. The idle-time tells the web-tier how long it can keep an idle socket before closing it. The keepalive-timeout tells the app-tier how long it should listen for new requests on the socket. The keepalive-timeout must be significantly larger than the load-balance-idle-time so the app-tier doesn't close its sockets too soon. The keepalive timeout can be large since the app-tier can use the keepalive-select manager to efficiently wait for many connections at once.

Dispatching

In most cases, the web-tier will dispatch everything to the app-tier servers. Because of Resin's proxy cache, the web-tier servers will serve static pages as fast as if they were local pages.

In some cases, though, it may be important to send different requests to different backend clusters. The <load-balance> tag can choose clusters based on URL patterns.

The following <rewrite-dispatch> keeps all *.png, *.gif, and *.jpg files on the web-tier, sends everything in /foo/* to the foo-tier cluster, everything in /bar/* to the bar-tier cluster, and keeps anything else on the web-tier.

split dispatching

<resin xmlns="http://caucho.com/ns/resin"> <cluster id="web-tier"> <server id="web-a"> <http port="80"/> </server> <cache memory-size="64m"/> <host id=""> <web-app id="/"> <rewrite-dispatch> <dispatch regexp="(\.png|\.gif|\.jpg)"/> <load-balance regexp="^/foo" cluster="foo-tier"/> <load-balance regexp="^/bar" cluster="bar-tier"/> </rewrite-dispatch> </web-app> </host> </cluster> <cluster id="foo-tier"> ... </cluster> <cluster id="bar-tier"> ... </cluster> </resin>

Persistent Sessions

A session needs to stay on the same JVM that started it. Otherwise, each JVM would only see every second or third request and get confused.

To make sure that sessions stay on the same JVM, Resin encodes the cookie with the host number. In the previous example, the hosts would generate cookies like:

INDEX	COOKIE PREFIX
1	axxx
2	bxxx
3	cxxx

On the web-tier, Resin will decode the cookie and send it to the appropriate host. So bacX8ZwooOz would go to app-b.

In the infrequent case that app-b fails, Resin will send the request to app-a. The user might lose the session but that's a minor problem compared to showing a connection failure error. To save sessions, you'll need to use distributed sessions. Also take a look at tcp sessions.

The following example is a typical configuration for a distributed server using an external hardware load-balancer, i.e. where each Resin is acting as the HTTP server. Each server will be started as -server a or -server b to grab its specific configuration.

In this example, sessions will only be stored when the server shuts down, either for maintenance or with a new version of the server. This is the most lightweight configuration, and doesn't affect performance significantly. If the hardware or the JVM crashes, however, the sessions will be lost. (If you want to save sessions for hardware or JVM crashes, remove the <save-only-on-shutdown/> flag.)

resin.xml

<resin xmlns="http://caucho.com/ns/resin"> <cluster id="app-tier"> <server-default> <http port='80'/> </server-default> <server id='app-a' address='192.168.0.1'/> <server id='app-b' address='192.168.0.2'/> <server id='app-c' address='192.168.0.3'/> <persistent-store type="cluster"> <init path="cluster"/> </persistent-store> <web-app-default> <!-- enable tcp-store for all hosts/web-apps --> <session-config> <use-persistent-store/> <save-only-on-shutdown/> </session-config> </web-app-default> ... </cluster> </resin>

Choosing a backend server

Requests can be made to specific servers in the app-tier. The web-tier uses the value of the jsessionid to maintain sticky sessions. You can include an explicit jsessionid to force the web-tier to use a particular server in the app-tier.

Resin uses the first character of the jsessionid to identify the backend server to use, starting with `a' as the first backend server. If wwww.example.com resolves to your web-tier, then you can use:

http://www.example.com/proxooladmin;jsessionid=abc
http://www.example.com/proxooladmin;jsessionid=bcd
http://www.example.com/proxooladmin;jsessionid=cde
http://www.example.com/proxooladmin;jsessionid=def
http://www.example.com/proxooladmin;jsessionid=efg
etc.

<persistent-store>

Configuration for persistent store uses the persistent-store tag.

File Based

For single-server configurations, the "cluster" store saves session data on disk, allowing for recovery after system restart or during development.

<resin xmlns="http://caucho.com/ns/resin"> <cluster id=""> <persistent-store type="cluster"/> <host id=""> </web-app id=""/> </host> </cluster> </resin>

Sessions are stored as files in the path directory. When the session changes, the updates will be written to the file. After Resin loads an Application, it will load the stored sessions.

Distributed Sessions

Distributed sessions are intrinsically more complicated than single-server sessions. Single-server session can be implemented as a simple memory-based Hashtable. Distributed sessions must communicate between machines to ensure the session state remains consistent.

Load balancing with multiple machines either uses sticky sessions or symmetrical sessions. Sticky sessions put more intelligence on the load balancer, and symmetrical sessions puts more intelligence on the JVMs. The choice of which to use depends on what kind of hardware you have, how many machines you're using and how you use sessions.

Distributed sessions can use a database as a backing store, or they can distribute the backup among all the servers using TCP.

Symmetrical Sessions

Symmetrical sessions happen with dumb load balancers like DNS round-robin. A single session may bounce from machine A to machine B and back to machine B. For JDBC sessions, the symmetrical session case needs the always-load-session attribute described below. Each request must load the most up-to-date version of the session.

Distributed sessions in a symmetrical environment are required to make sessions work at all. Otherwise the state will end up spread across the JVMs. However, because each request must update its session information, it is less efficient than sticky sessions.

Sticky Sessions

Sticky sessions require more intelligence on the load-balancer, but are easier for the JVM. Once a session starts, the load-balancer will always send it to the same JVM. Resin's load balancing, for example, encodes the session id as 'aaaXXX' and 'baaXXX'. The 'aaa' session will always go to JVM-a and 'baa' will always go to JVM-b.

Distributed sessions with a sticky session environment add reliability. If JVM-a goes down, JVM-b can pick up the session without the user noticing any change. In addition, distributed sticky sessions are more efficient. The distributor only needs to update sessions when they change. So if you update the session once when the user logs in, the distributed sessions can be very efficient.

always-load-session

Symmetrical sessions must use the 'always-load-session' flag to update each session data on each request. always-load-session is only needed for jdbc-store sessions. tcp-store sessions use a more-sophisticated protocol that eliminates the need for always-load-session, so tcp-store ignores the always-load-session flag.

The always-load-session attribute forces sessions to check the store for each request. By default, sessions are only loaded from persistent store when they are created. In a configuration with multiple symmetric web servers, sessions can be loaded on each request to ensure consistency.

always-save-session

By default, Resin only saves session data when you add new values to the session object, i.e. if the request calls setAttribute. This may be insufficient when storing large objects. For example, if you change an internal field of a large object, Resin will not automatically detect that change and will not save the session object.

With always-save-session Resin will always write the session to the store at the end of each request. Although this is less efficient, it guarantees that updates will get stored in the backup after each request.

Database Based

Database backed sessions are the easiest to understand. Session data gets serialized and stored in a database. The data is loaded on the next request.

For efficiency, the owning JVM keeps a cache of the session value, so it only needs to query the database when the session changes. If another JVM stores a new session value, it will notify the owner of the change so the owner can update its cache. Because of this notification, the database store is cluster-aware.

In some cases, the database can become a bottleneck. By adding load to an already-loaded system, you may harm performance. One way around that bottleneck is to use a small, quick database like MySQL for your session store and save the "Big Iron" database like Oracle for your core database needs.

The database must be specified using a <database>. The database store will automatically create a session table.

The JDBC store needs to know about the other servers in the cluster in order to efficiently update them when changes occur to the server.

JDBC store

<resin xmlns="http://caucho.com/ns/resin"> <cluster id="app-tier"> <server-default> <http port="80"/> </server-default> <server id="app-a" address="192.168.2.10" port="6800"/> <server id="app-b" address="192.168.2.11" port="6800"/> <database jndi-name="jdbc/session"> ... </database> <persistent-store type="jdbc"> <init> <data-source>jdbc/session<data-source> </init> </persistent-store> ... <web-app-default> <session-config> <use-persistent-store/> </session-config> </web-app-default> ... </cluster> </resin>

The persistent store is configured in the <server> with persistent-store. Each web-app which needs distributed sessions must enable the persistent store with a use-persistent-store tag in the session-config.

data-source	data source name for the table
table-name	database table for the session data
blob-type	database type for a blob
max-idle-time	cleanup time

CREATE TABLE persistent_session ( id VARCHAR(64) NOT NULL, data BLOB, access_time int(11), expire_interval int(11), PRIMARY KEY(id) )

The store is enabled with <use-persistent-store> in the session config.

<web-app xmlns="http://caucho.com/ns/resin"> <session-config> <use-persistent-store/> <always-save-session/> </session-config> </web-app>

Cluster Sessions

The distributed cluster stores the sessions across the cluster servers. In some configurations, the cluster store may be more efficient than the database store, in others the database store will be more efficient.

With cluster sessions, each session has an owning JVM and a backup JVM. The session is always stored in both the owning JVM and the backup JVM.

The cluster store is configured in the in the <cluster>. It uses the <server> hosts in the <cluster> to distribute the sessions. The session store is enabled in the <session-config> with the <use-persistent-store>.

<resin xmlns="http://caucho.com/ns/resin"> ... <cluster id="app-tier"> <server id="app-a" host="192.168.0.1" port="6802"/> <server id="app-b" host="192.168.0.2" port="6802"/> <persistent-store type="cluster"> <init path="cluster"/> </persistent-store> ... </cluster> </resin>

The configuration is enabled in the web-app.

<web-app xmlns="http://caucho.com/ns/resin"> <session-config> <use-persistent-store="true"/> </session-config> </web-app>

The <srun> and <srun-backup> hosts are treated as a cluster of hosts. Each host uses the other hosts as a backup. When the session changes, the updates will be sent to the backup host. When the host starts, it looks up old sessions in the other hosts to update its own version of the persistent store.

Symmetric load-balanced servers

<resin xmlns="http://caucho.com/ns/resin"> <cluster id="app-tier"> <server-default> <http port='80'/> </server-default> <server id="app-a" address="192.168.2.10" port="6802"/> <server id="app-b" address="192.168.2.11" port="6803"/> <persistent-store type="cluster"> <init path="cluster"/> </persistent-store> <host id=''> <web-app id=''> <session-config> <use-persistent-store="true"/> </session-config> </web-app> </host> </cluster> </resin>

Clustered Distributed Sessions

Resin's cluster protocol for distributed sessions can is an alternative to JDBC-based distributed sessions. In some configurations, the cluster-stored sessions will be more efficient than JDBC-based sessions. Because sessions are always duplicated on separate servers, cluster sessions do not have a single point of failure. As the number of servers increases, JDBC-based sessions can start overloading the backing database. With clustered sessions, each additional server shares the backup load, so the main scalability issue reduces to network bandwidth. Like the JDBC-based sessions, the cluster store sessions uses sticky-session caching to avoid unnecessary network traffic.

Configuration

The cluster configuration must tell each host the servers in the cluster and it must enable the persistent in the session configuration with use-persistent-store. Because session configuration is specific to a virtual host and a web-application, each web-app needs use-persistent-store enabled individually. The web-app-default tag can be used to enable distributed sessions across an entire site.

Most sites using Resin's load balancing will already have the cluster <srun> configured. Each <srun> block corresponds to a host, including the current host. Since cluster sessions uses Resin's srun protocol, each host must listen for srun requests.

resin.xml fragment

<resin xmlns="http://caucho.com/ns/resin"> <cluster id="app-tier"> <server id="app-a" host="192.168.0.1"/> <server id="app-b" host="192.168.0.2"/> <server id="app-c" host="192.168.0.3"/> <server id="app-d" host="192.168.0.4"/> <persistent-store type="cluster"> <init path="cluster"/> </persistent-store> ... <host id=""> <web-app id='myapp'> ... <session-config> <use-persistent-store/> </session-config> </web-app> </host> </cluster> </resin>

Usually, hosts will share the same resin.xml. Each host will be started with a different -server xx to select the correct block. On Unix, startup will look like:

Starting Host C on Unix

resin-3.2.x> bin/resin.sh -conf conf/resin.xml -server c start

On Windows, Resin will generally be configured as a service:

Starting Host C on Windows

resin-3.2.x> bin/resin -conf conf/resin.xml -server c -install-as ResinC

always-save-session

Resin's distributed sessions needs to know when a session has changed in order to save the new session value. Although Resin can detect when an application calls HttpSession.setAttribute, it can't tell if an internal session value has changed. The following Counter class shows the issue:

Counter.java

package test; public class Counter implements java.io.Serializable { private int _count; public int nextCount() { return _count++; } }

Assuming a copy of the Counter is saved as a session attribute, Resin doesn't know if the application has called nextCount. If it can't detect a change, Resin will not backup the new session, unless always-save-session is set. When always-save-session is true, Resin will back up the session on every request.

... <web-app id="/foo"> ... <session-config> <use-persistent-store/> <always-save-session/> </session-config> ... </web-app>

Like the JDBC-based sessions, Resin will ignore the always-load-session flag for cluster sessions. Because the cluster protocol notifies servers of changes, always-load-session is not needed.

Serialization

Resin's distributed sessions relies on Java serialization to save and restore sessions. Application object must implement java.io.Serializable for distributed sessions to work.

Protocol Examples

Session Request

To see how cluster sessions work, consider a case where the load balancer sends the request to a random host. Host C owns the session but the load balancer gives the request to Host A. In the following figure, the request modifies the session so it must be saved as well as loaded.

The session id encodes the owning host. The example session id, ca8MbyA, decodes to an srun-index of 3, mapping to Host C. Resin determines the backup host from the cookie as well. Host A must know the owning host for every cookie so it can communicate with the owning srun. The example configuration defines all the sruns Host A needs to know about. If Host C is unavailable, Host A can use its configuration knowledge to use Host D as a backup for ca8MbyA instead..

When the request first accesses the session, Host A asks Host C for the serialized session data (2:load). Since Host A doesn't cache the session data, it must ask Host C for an update on each request. For requests that only read the session, this TCP load is the only extra overhead, i.e. they can skip 3-5. The always-save-session flag, in contrast, will always force a write.

At the end of the request, Host A writes any session updates to Host C (3:store). If always-save-session is false and the session doesn't change, this step can be skipped. Host A sends the new serialized session contents to Host C. Host C saves the session on its local disk (4:save) and saves a backup to Host D (5:backup).

Sticky Session Request

Smart load balancers that implement sticky sessions can improve cluster performance. In the previous request, Resin's cluster sessions maintain consistency for dumb load balancers or twisted clients like the AOL browsers. The cost is the additional network traffic for 2:load and 3:store. Smart load-balancers can avoid the network traffic of 2 and 3.

Host C decodes the session id, caaMbyA. Since it owns the session, Host C gives the session to the servlet with no work and no network traffic. For a read-only request, there's zero overhead for cluster sessions. So even a semi-intelligent load balancer will gain a performance advantage. Normal browsers will have zero overhead, and bogus AOL browsers will have the non-sticky session overhead.

A session write saves the new serialized session to disk (2:save) and to Host D (3:backup). always-save-session will determine if Resin can take advantage of read-only sessions or must save the session on each request.

Disk copy

Resin stores a disk copy of the session information, in the location specified by the path. The disk copy serves two purposes. The first is that it allows Resin to keep session information for a large number of sessions. An efficient memory cache keeps the most active sessions in memory and the disk holds all of the sessions without requiring large amounts of memory. The second purpose of the disk copy is that the sessions are recovered from disk when the server is restarted.

Failover

Since the session always has a current copy on two servers, the load balancer can direct requests to the next server in the ring. The backup server is always ready to take control. The failover will succeed even for dumb load balancers, as in the non-sticky-session case, because the srun hosts will use the backup as the new owning server.

In the example, either Host C or Host D can stop and the sessions will use the backup. Of course, the failover will work for scheduled downtime as well as server crashes. A site could upgrade one server at a time with no observable downtime.

Recovery

When Host C restarts, possibly with an upgraded version of Resin, it needs to use the most up-to-date version of the session; its file-saved session will probably be obsolete. When a "new" session arrives, Host C loads the saved session from both the file and from Host D. It will use the newest session as the current value. Once it's loaded the "new" session, it will remain consistent as if the server had never stopped.

No Distributed Locking

Resin's cluster sessions does not lock sessions. For browser-based sessions, only one request will execute at a time. Since browser sessions have no concurrently, there's no need for distributed locking. However, it's a good idea to be aware of the lack of distributed locking.