Apache 负载均衡 tomcat websocket

Question

我目前正在开发一个部署在 Tomcat 服务器上的 websocket 应用程序。由于用户数量庞大，我想将工作负载分配给多个 Tomcat 实例。我决定使用 Apache 进行负载平衡。

现在我在为 websockets 请求实现 Apache 负载平衡和粘性 session 时遇到了问题。这是我的 Apache 配置:

ProxyRequests off
SSLProxyEngine on
RewriteEngine On

<Proxy balancer://http-localhost/>
    BalancerMember  https://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  https://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

<Proxy balancer://ws-localhost/>
    BalancerMember  wss://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  wss://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

RewriteCond       %{HTTP:Upgrade} =websocket
RewriteRule       /jddd/(.*)      balancer://ws-localhost/$1 [P,L]
ProxyPassReverse  /jddd/          balancer://ws-localhost/
RewriteCond       %{HTTP:Upgrade} !=websocket
RewriteRule       /jddd/(.*)      balancer://http-localhost/$1 [P,L]
ProxyPassReverse  /jddd/          balancer://http-localhost/

第一个https请求均衡到8443端口，升级后的wss请求也转发到8443

第二个https请求包含第一个请求的sessionID: https://...&sid=C28C13EEEC525D203F8CA4E827605E0B.jvm1

正如我在 Apache 日志文件中看到的，这个 sessionID 被评估为 stickySession:

...Found value C28C13EEEC525D203F8CA4E827605E0B.jvm1 for stickysession sid

...Found route jvm1

...balancer://http-localhost: worker (htttps://mcsgest1.desy.de:8443/Whiteboard/) rewritten to htttps://mcsgest1.desy.de:8443/Whiteboard//?file=octocenter.xml&address=///&sid=C28C13EEEC525D203F8CA4E827605E0B.jvm1

第二个https请求仍然在8443端口，但是在升级到websocket协议(protocol)后，ws-balancer不评估sessionID并重写为8444:

...balancer://ws-localhost: worker (wss://mcsgest1.desy.de:8444/Whiteboard/) rewritten to wss://mcsgest1.desy.de:8444/Whiteboard//whiteboardendpoint

我必须在 Apache 配置中更改什么才能为 wss 协议(protocol)启用 stickysession？我真的需要两个平衡器(http 和 ws)来平衡 websockets 吗？

Answer 1

你不需要为 websokets 单独的平衡器，因为初始的 http 请求已经有 http cookie 并且属于正确的实例。

您只需要检测连接升级并根据 cookie 的粘性部分手动路由请求

确保为 websokets 加载代理模块 - mod_proxy_wstunnel

例如

SSLProxyEngine on
RewriteEngine On

<Proxy balancer://http-localhost/>
    BalancerMember  https://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  https://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm1 [NC]
RewriteRule .* wss://mcsgest1.desy.de:8443%{REQUEST_URI} [P,L]

RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm2 [NC]
RewriteRule .* wss://mcsgest1.desy.de:8444%{REQUEST_URI} [P,L]

RewriteRule  /jddd/(.*)  balancer://http-localhost$1  [P,L]

ProxyPreserveHost On
ProxyRequests Off

ProxyPass /jddd/  balancer://http-localhost
ProxyPassReverse /jddd/ balancer://http-localhost

解释:

# if header upgrade = WebSocket
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]  
# and header connection contains Upgrade (header may be like this: connection=keep-alive, upgrade)
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] 
# and header cookie contains JSESSIONID or sid, ending with sticky part - .jvm1 in that case
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm1 [NC] 
#than we route request to application server via mod_proxy (P flag) and end rewrite rule check
RewriteRule .* wss://mcsgest1.desy.de:8443%{REQUEST_URI} [P]