Guacamole HTTP tunnel not Websockets

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Guacamole HTTP tunnel not Websockets

Tony Hooker

We are running into a curious issue with users in Guac experiencing extreme lag after opening 3+ connections in multiple tabs. We looked at the /var/log/tomcat8/catalina.out log and we found that our server is not passing the tunnel servlet via websockets.

 

16:14:22.737 [http-nio-8080-exec-8] INFO  o.a.g.t.h.RestrictedGuacamoleHTTPTunnelServlet - Using HTTP tunnel (not WebSocket). Performance may be sub-optimal.

 

We then checked our Nginx web service and made sure all the proper websocket configurations were in place, and it checked out.

 

My next thought was maybe tomcat8 isn’t setup properly for websockets. Has anyone had this issue and could reply with some insight?

 

 

Thanks,

Tony Hooker

Director of Information Technology Services

Mobile:  480-469-2609, Desk: 602-386-2951

Dynamic Worldwide Training Consultantshttp://m.dwwtc.com/images/email/twitter.pnghttp://m.dwwtc.com/images/email/facebook.pnghttp://m.dwwtc.com/images/email/linkedin.pnghttp://m.dwwtc.com/images/email/youtube.pnghttp://m.dwwtc.com/images/email/pinterest.png

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Guacamole HTTP tunnel not Websockets

Mike Jumper
On Wed, Jun 14, 2017 at 8:47 AM, Tony Hooker <[hidden email]> wrote:

We are running into a curious issue with users in Guac experiencing extreme lag after opening 3+ connections in multiple tabs. We looked at the /var/log/tomcat8/catalina.out log and we found that our server is not passing the tunnel servlet via websockets.

 

16:14:22.737 [http-nio-8080-exec-8] INFO  o.a.g.t.h.RestrictedGuacamoleHTTPTunnelServlet - Using HTTP tunnel (not WebSocket). Performance may be sub-optimal.


Browsers throttle the total number of HTTP connections established via JavaScript on a per-domain basis, even across tabs. Once the browser starts delaying the creation of outbound connections until existing connections close, there will be resource contention between each of the Guacamole tabs, and performance will suffer.

 

We then checked our Nginx web service and made sure all the proper websocket configurations were in place, and it checked out.



What version of Nginx are you using?

Can you post your configuration?

My next thought was maybe tomcat8 isn’t setup properly for websockets. Has anyone had this issue and could reply with some insight?

 


I believe Tomcat 8 is always setup for websocket.

- Mike

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Guacamole HTTP tunnel not Websockets

Erik Berndt
I'm seeing the same issue using Apache with mod_proxy enabled, so I'm
watching this thread with curiosity.
Erik Berndt / Systems Administrator
5551 Wellington Rd, Gainesville, VA 20155
703.631.0004 x520 (Phone) / 703.257.1725 (Fax)
http://www.superiorpaving.net

Need to open an IT support ticket?
http://FixIT.superiorpaving.net/portal or [hidden email]


On Wed, Jun 14, 2017 at 11:59 AM, Mike Jumper <[hidden email]> wrote:

> On Wed, Jun 14, 2017 at 8:47 AM, Tony Hooker <[hidden email]> wrote:
>>
>> We are running into a curious issue with users in Guac experiencing
>> extreme lag after opening 3+ connections in multiple tabs. We looked at the
>> /var/log/tomcat8/catalina.out log and we found that our server is not
>> passing the tunnel servlet via websockets.
>>
>>
>>
>> 16:14:22.737 [http-nio-8080-exec-8] INFO
>> o.a.g.t.h.RestrictedGuacamoleHTTPTunnelServlet - Using HTTP tunnel (not
>> WebSocket). Performance may be sub-optimal.
>
>
> Browsers throttle the total number of HTTP connections established via
> JavaScript on a per-domain basis, even across tabs. Once the browser starts
> delaying the creation of outbound connections until existing connections
> close, there will be resource contention between each of the Guacamole tabs,
> and performance will suffer.
>
>>
>>
>> We then checked our Nginx web service and made sure all the proper
>> websocket configurations were in place, and it checked out.
>>
>>
>
> What version of Nginx are you using?
>
> Can you post your configuration?
>
>> My next thought was maybe tomcat8 isn’t setup properly for websockets. Has
>> anyone had this issue and could reply with some insight?
>>
>>
>
>
> I believe Tomcat 8 is always setup for websocket.
>
> - Mike
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Guacamole HTTP tunnel not Websockets

Mike Jumper
On Wed, Jun 14, 2017 at 9:10 AM, Erik Berndt <[hidden email]> wrote:
I'm seeing the same issue using Apache with mod_proxy enabled, so I'm
watching this thread with curiosity.

If you're referring specifically to the issue that websocket is not used, can you confirm whether your mod_proxy has websocket support?

When proxying with Apache, the websocket tunnel must be explicitly declared in the configuration, or Apache will only ever use HTTP (even when JavaScript is trying to use websocket), and attempts to use websocket will fail:


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Guacamole HTTP tunnel not Websockets

Erik Berndt
Sorry, I should have been more clear. Websockets is not being used to
proxy connections
mod_proxy_ws_tunnel is enabled and the websocket proxy is explicitly
declared in the apache configuration file as seen below.

<VirtualHost _default_:443>
        ServerAdmin []
        ServerName []
        DocumentRoot /var/lib/tomcat8/webapps
        ProxyPreserveHost On

<Location />
        Order allow,deny
        Allow from all
        ProxyPass http://localhost:8080/guacamole/ flushpackets=on
        ProxyPassReverse http://localhost:8080/guacamole/
</Location>

<Location /guacamole/websocket-tunnel>
        Order allow,deny
        Allow from all
        ProxyPass ws://localhost:8080/guacamole/websocket-tunnel
        ProxyPassReverse ws://localhost:8080/guacamole/websocket-tunnel
</Location>
Erik Berndt / Systems Administrator
5551 Wellington Rd, Gainesville, VA 20155
703.631.0004 x520 (Phone) / 703.257.1725 (Fax)
http://www.superiorpaving.net

Need to open an IT support ticket?
http://FixIT.superiorpaving.net/portal or [hidden email]


On Wed, Jun 14, 2017 at 12:14 PM, Mike Jumper <[hidden email]> wrote:

> On Wed, Jun 14, 2017 at 9:10 AM, Erik Berndt <[hidden email]>
> wrote:
>>
>> I'm seeing the same issue using Apache with mod_proxy enabled, so I'm
>> watching this thread with curiosity.
>
>
> If you're referring specifically to the issue that websocket is not used,
> can you confirm whether your mod_proxy has websocket support?
>
> When proxying with Apache, the websocket tunnel must be explicitly declared
> in the configuration, or Apache will only ever use HTTP (even when
> JavaScript is trying to use websocket), and attempts to use websocket will
> fail:
>
> http://guacamole.incubator.apache.org/doc/gug/proxying-guacamole.html#websocket-and-apache
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Guacamole HTTP tunnel not Websockets

Tony Hooker
In reply to this post by Mike Jumper

We followed this individuals configuration strategy.

 

https://www.chasewright.com/lets-encrypt-and-nginx/

 

 

From: Mike Jumper [mailto:[hidden email]]
Sent: Wednesday, June 14, 2017 9:00 AM
To: [hidden email]
Subject: Re: Guacamole HTTP tunnel not Websockets

 

On Wed, Jun 14, 2017 at 8:47 AM, Tony Hooker <[hidden email]> wrote:

We are running into a curious issue with users in Guac experiencing extreme lag after opening 3+ connections in multiple tabs. We looked at the /var/log/tomcat8/catalina.out log and we found that our server is not passing the tunnel servlet via websockets.

 

16:14:22.737 [http-nio-8080-exec-8] INFO  o.a.g.t.h.RestrictedGuacamoleHTTPTunnelServlet - Using HTTP tunnel (not WebSocket). Performance may be sub-optimal.

 

Browsers throttle the total number of HTTP connections established via JavaScript on a per-domain basis, even across tabs. Once the browser starts delaying the creation of outbound connections until existing connections close, there will be resource contention between each of the Guacamole tabs, and performance will suffer.

 

 

We then checked our Nginx web service and made sure all the proper websocket configurations were in place, and it checked out.

 

 

What version of Nginx are you using?

 

Can you post your configuration?

 

My next thought was maybe tomcat8 isn’t setup properly for websockets. Has anyone had this issue and could reply with some insight?

 

 

I believe Tomcat 8 is always setup for websocket.

 

- Mike

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Guacamole HTTP tunnel not Websockets

Tony Hooker
In reply to this post by Mike Jumper

“Browsers throttle the total number of HTTP connections established via JavaScript on a per-domain basis, even across tabs. Once the browser starts delaying the creation of outbound connections until existing connections close, there will be resource contention between each of the Guacamole tabs, and performance will suffer.”

 

I just tested 4 tabs of SSH in Chrome, Opera, Mozilla, and Edge. Mozilla was by far the worst experience with crazy random SSH disconnects, Chrome and Opera was a poor experience as well but no disconnects, while Edge was experiencing almost no lag at all. We are discussing in the IT office that we feel it’s the Java script engine that browsers are designing around. Edge’s Java engine seems to be using the most compatible engine for Guac performance. Im currently recommending all remote users to switch to Edge until further testing or updates are released. I think this is the issue, but still doesn’t address the actual log websocket connection issue.

 

I would like to add, I’ve been using Guac since 0.9.9 and it wasn’t until I updated to 0.9.12 and added SSL, did I start getting reports of these issues from remote users.

 

 

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Guacamole HTTP tunnel not Websockets

Erik Berndt
what does your nginx config look like? I had the same issue on Apache,
but after reviewing my config files, noticed the websocket proxy
location was pointed to the wrong path, causing the web app to revert
to the http proxy.
Erik Berndt / Systems Administrator
5551 Wellington Rd, Gainesville, VA 20155
703.631.0004 x520 (Phone) / 703.257.1725 (Fax)
http://www.superiorpaving.net

Need to open an IT support ticket?
http://FixIT.superiorpaving.net/portal or [hidden email]


On Wed, Jun 14, 2017 at 1:13 PM, Tony Hooker <[hidden email]> wrote:

> “Browsers throttle the total number of HTTP connections established via
> JavaScript on a per-domain basis, even across tabs. Once the browser starts
> delaying the creation of outbound connections until existing connections
> close, there will be resource contention between each of the Guacamole tabs,
> and performance will suffer.”
>
>
>
> I just tested 4 tabs of SSH in Chrome, Opera, Mozilla, and Edge. Mozilla was
> by far the worst experience with crazy random SSH disconnects, Chrome and
> Opera was a poor experience as well but no disconnects, while Edge was
> experiencing almost no lag at all. We are discussing in the IT office that
> we feel it’s the Java script engine that browsers are designing around.
> Edge’s Java engine seems to be using the most compatible engine for Guac
> performance. Im currently recommending all remote users to switch to Edge
> until further testing or updates are released. I think this is the issue,
> but still doesn’t address the actual log websocket connection issue.
>
>
>
> I would like to add, I’ve been using Guac since 0.9.9 and it wasn’t until I
> updated to 0.9.12 and added SSL, did I start getting reports of these issues
> from remote users.
>
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Guacamole HTTP tunnel not Websockets

Tony Hooker
This is our /etc/nginx/nginx.conf

----------------------------------------------
user www-data;
worker_processes 4;
pid /run/nginx.pid;

events
{
        worker_connections 768;
}

http
{
        # My Certificates
        ssl_certificate /etc/nginx/ssl/lvc.dwwtc.com/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/lvc.dwwtc.com/privkey.pem;

        # SSL Performance Related
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;

        # SSL Protocols and Ciphers
        ssl_prefer_server_ciphers on;
        ssl_protocols TLSv1.2;
        ssl_ciphers "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:!AES128:!aNULL:!MD5:!eNULL:!EXPORT:!DES:!PSK:!RC4";
        # DHE Key-Exchange
        ssl_dhparam /etc/nginx/ssl/lvc.dwwtc.com/dhparam.pem;

        # Random Security Stuff
        server_tokens off;
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Strict-Transport-Security max-age=63072000;

        # Common Proxy Settings
        proxy_set_header Host      \$host;
        proxy_set_header X-Real-IP  \$remote_addr;
        proxy_set_header    X-Forwarded-For \$proxy_add_x_forwarded_for;

        ########################
        # Default Config Stuff #
        ########################
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 4096; #Default:2048
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        gzip on;
        gzip_disable "msie6";
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

        # REDIRECTS ALL PORT 80/HTTP to 443/HTTPS
        server
        {
                listen 80;
                listen [::]:80;
                server_name lvc.dwwtc.com;

                location ~ /.well-known/acme-challenge
                {
                    root /var/www/html/;
                }

                return 301 https://\$host\$request_uri;
        }

        # GUACAMOLE SERVER SETTINGS
        server
        {
                listen 443 ssl;
                listen [::]:443 ssl;
                server_name lvc.dwwtc.com;

                proxy_buffering off;
                proxy_redirect  off;
                proxy_cookie_path /guacamole/ /;
                proxy_http_version 1.1;
                proxy_set_header Upgrade \$http_upgrade;
                proxy_set_header Connection "upgrade";

                location ~ /.well-known/acme-challenge
                {
                    root /var/www/html/;
                }

                location /
                {
                        proxy_pass http://lvc.dwwtc.local:8080/;
                }
        }
}
----------------------------------------------
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Guacamole HTTP tunnel not Websockets

Mike Jumper
On Wed, Jun 14, 2017 at 2:56 PM, Tony Hooker <[hidden email]> wrote:

                proxy_set_header Upgrade \$http_upgrade;
                proxy_set_header Connection "upgrade";

Why have the "$" symbols for each variable been backslash-escaped?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Guacamole HTTP tunnel not Websockets

Mike Jumper
In reply to this post by Tony Hooker
On Wed, Jun 14, 2017 at 10:13 AM, Tony Hooker <[hidden email]> wrote:

“Browsers throttle the total number of HTTP connections established via JavaScript on a per-domain basis, even across tabs. Once the browser starts delaying the creation of outbound connections until existing connections close, there will be resource contention between each of the Guacamole tabs, and performance will suffer.”

 

I just tested 4 tabs of SSH in Chrome, Opera, Mozilla, and Edge. Mozilla was by far the worst experience with crazy random SSH disconnects, Chrome and Opera was a poor experience as well but no disconnects, while Edge was experiencing almost no lag at all. We are discussing in the IT office that we feel it’s the Java script engine that browsers are designing around. Edge’s Java engine seems to be using the most compatible engine for Guac performance. Im currently recommending all remote users to switch to Edge until further testing or updates are released. I think this is the issue, but still doesn’t address the actual log websocket connection issue.

 


To compare JavaScript performance between browsers, you should really use one tab only. With multiple tabs, you are not necessarily comparing the performance of JavaScript, but the different levels of connection throttling, the prioritization of JavaScript tasks in background tabs, etc. There are too many variables for that to be a good benchmark.

Edge performance is decent, but if you're seeing it being faster than Chrome and others, I'm suspicious. With respect to JavaScript, I would expect Chrome to be much faster.

I would like to add, I’ve been using Guac since 0.9.9 and it wasn’t until I updated to 0.9.12 and added SSL, did I start getting reports of these issues from remote users.



Not sure why you would only be seeing it now, but the issue with browsers throttling HTTP connections has definitely existed for quite some time. It was particularly annoying back in the days when Guacamole opened absolutely all connections in new tabs and lacked support for websocket (0.9.3 or older).

Once things moved to a single-tab style with 0.9.4, and websocket support was stabilized, the throttling encountered by Guacamole was much less aggressive, but HTTP connection throttling is still an unavoidable problem for cases where websocket is unavailable.

- Mike

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Guacamole HTTP tunnel not Websockets

Tony Hooker
In reply to this post by Mike Jumper

OMG! I removed the backslash and that fixed it! Now all connections are using websockets and all lag is gone! THANK YOU SO MUCH! Oh Linux, I love you and I hate you at the same time!

 

From: Mike Jumper [mailto:[hidden email]]
Sent: Thursday, June 15, 2017 11:02 AM
To: [hidden email]
Subject: Re: Guacamole HTTP tunnel not Websockets

 

On Wed, Jun 14, 2017 at 2:56 PM, Tony Hooker <[hidden email]> wrote:


                proxy_set_header Upgrade \$http_upgrade;
                proxy_set_header Connection "upgrade";

 

Why have the "$" symbols for each variable been backslash-escaped?

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Guacamole HTTP tunnel not Websockets

Tony Hooker
In reply to this post by Mike Jumper

On Wed, Jun 14, 2017 at 2:56 PM, Tony Hooker <[hidden email]> wrote:


                proxy_set_header Upgrade \$http_upgrade;
                proxy_set_header Connection "upgrade";

 

Why have the "$" symbols for each variable been backslash-escaped?

 

This is why…

 

https://www.chasewright.com/lets-encrypt-and-nginx/#comment-1664

 

We used Chase Wright’s configuration, and the configuration he displayed on his blog was used in scripting. He describes why the “\” is present in response to my comment about our findings.

Loading...