Server Out Of Memory

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Server Out Of Memory

James Fraser

Hi All

 

I recently upgraded to 0.9.13 and am experiencing an issue with my Production server.

This is potentially a Tomcat issue or JDBC driver issue.

 

If anyone can comment that would be great.

 

Looking at the Catalina.out file I found

 

WARNING: The web application [guacamole] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

java.lang.Object.wait(Native Method)

java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)

com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)

 

 

SEVERE: The web application [guacamole] created a ThreadLocal with key of type [com.google.inject.internal.InjectorImpl$1] (value [com.google.inject.internal.InjectorImpl$1@2c0acdd7]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@2ee3d179]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Aug 02, 2017 8:25:43 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks

SEVERE: The web application [guacamole] created a ThreadLocal with key of type [com.google.inject.internal.InjectorImpl$1] (value [com.google.inject.internal.InjectorImpl$1@1b5b9da0]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@b5aea9c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

 

Which leads too

 

Aug 03, 2017 10:04:16 PM org.apache.tomcat.util.net.NioEndpoint$Poller run

SEVERE:

java.lang.OutOfMemoryError: Java heap space

 

 

I am just running the standard apt-get configuration of Tomcat 8 on Ubuntu server 16.4

The server has 4GB of ram

 

Here is the output of Free

 

root@MGMT-GUAC-01:/var/log/tomcat8# free -h

              total        used        free      shared  buff/cache   available

Mem:           3.4G        939M        128M         22M        2.3G        2.1G

Swap:            0B          0B          0B

 

A restart of tomcat resolves the issue for a period of time, I have just written a cron job that restarts tomcat on appearance of this issue.

 


Any advice greatly appreciated.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Server Out Of Memory

Nick Couchman-2
Okay, let me try to take these one at a time...


On Thursday, August 3, 2017, 8:16:09 PM EDT, James Fraser <[hidden email]> wrote:


> I recently upgraded to 0.9.13 and am experiencing an issue with my Production server.

> This is potentially a Tomcat issue or JDBC driver issue.

What extensions do you have loaded?  Looks like MySQL JDBC - anything else?

> WARNING: The web application [guacamole] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
> ...

I use PostgreSQL and see these messages periodically, too, but they've never led to any adverse behavior.

> Which leads too

> Aug 03, 2017 10:04:16 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
> SEVERE:
> java.lang.OutOfMemoryError: Java heap space

Yeah, that's not good, but it doesn't mean your server is running out of memory, it means the JavaVM is running out of heap space.  Those are different things.  What parameters do you have set for memory in Java in your Tomcat startup?  Look for the -Xmx flag either in the ps output for the PID of Java associated with Tomcat or in the Tomcat startup.sh file.  If you don't see it, then the default is 1/4 of your total RAM, so 1GB.  You can add the -Xmx flag to the java runtime parameters for Tomcat and bump it up to 2GB or something like that and see if that helps.  If run out of RAM after bumping it up to 2 or 3GB, then you may have run into a memory leak, but I'd give that a shot, first.  When you set it, you can use abbreviations for various byte multiples - for example, -Xmx1024m is 1024MB or 1GB.  So, you might want to start with -Xmx2048m to bump up to 2GB and see if that helps.
 

> The server has 4GB of ram

I ran Guacamole 0.9.12 and the development versions of 0.9.13 on a system with 4GB of RAM for quite some time and never had any issues.  How many connections do you have?  How many users connecting concurrently?


> root@MGMT-GUAC-01:/var/log/tomcat8# free -h
>              total        used        free      shared  buff/cache   available
> Mem:           3.4G        939M        128M         22M        2.3G        2.1G
> Swap:            0B          0B          0B
 

> A restart of tomcat resolves the issue for a period of time, I have just written a cron job that restarts tomcat on appearance of this issue.

I've done Linux system admin/engineering for many years, and, from my point of view, those numbers from the output of free look just fine.  While it's easy to look at the "free" column and see 128M and think your system is short on RAM, the "available" column is what really counts.  Linux uses available RAM to cache and buffer things like disk and network I/O, and your system is consuming 2-ish GB for that.  Memory allocated for buffer/cache can be easily freed when applications need it, so that's why the available column shows 2.1GB.  So, whenever you ran the "free" command on your system, the system itself is fine on RAM (for the moment) - it's most likely a Java heap size issue (-Xmx flag needs to be set).

-Nick
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Server Out Of Memory

James Fraser

Hi Nick

 

Thanks for your response.

 

After sending off this message I did some digging.

 

I am using JDBC and LDAP auth together.

 

I was digging around the Server Heap error and think that you are on the right track with Xmx value.

It was out of the box (from apt-get) set to -Xmx128m, I have adjusted this to 1024m for now and will monitor

 

We concurrently have around 7 users, each user may be accessing 4-5 VM’s at once.

 

Thanks again for the quick response Nick

 

James Fraser • Microsoft Systems Engineer

 

From: Nick Couchman [mailto:[hidden email]]
Sent: Friday, 4 August 2017 10:52 AM
To: [hidden email]
Subject: Re: Server Out Of Memory

 

Okay, let me try to take these one at a time...


On Thursday, August 3, 2017, 8:16:09 PM EDT, James Fraser <[hidden email]> wrote:


> I recently upgraded to 0.9.13 and am experiencing an issue with my Production server.

> This is potentially a Tomcat issue or JDBC driver issue.

What extensions do you have loaded?  Looks like MySQL JDBC - anything else?

> WARNING: The web application [guacamole] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)

> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)

> ...

 

I use PostgreSQL and see these messages periodically, too, but they've never led to any adverse behavior.


> Which leads too

> Aug 03, 2017 10:04:16 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
> SEVERE:

> java.lang.OutOfMemoryError: Java heap space

 

Yeah, that's not good, but it doesn't mean your server is running out of memory, it means the JavaVM is running out of heap space.  Those are different things.  What parameters do you have set for memory in Java in your Tomcat startup?  Look for the -Xmx flag either in the ps output for the PID of Java associated with Tomcat or in the Tomcat startup.sh file.  If you don't see it, then the default is 1/4 of your total RAM, so 1GB.  You can add the -Xmx flag to the java runtime parameters for Tomcat and bump it up to 2GB or something like that and see if that helps.  If run out of RAM after bumping it up to 2 or 3GB, then you may have run into a memory leak, but I'd give that a shot, first.  When you set it, you can use abbreviations for various byte multiples - for example, -Xmx1024m is 1024MB or 1GB.  So, you might want to start with -Xmx2048m to bump up to 2GB and see if that helps.

 

> The server has 4GB of ram

I ran Guacamole 0.9.12 and the development versions of 0.9.13 on a system with 4GB of RAM for quite some time and never had any issues.  How many connections do you have?  How many users connecting concurrently?


> root@MGMT-GUAC-01:/var/log/tomcat8# free -h
>              total        used        free      shared  buff/cache   available
> Mem:           3.4G        939M        128M         22M        2.3G        2.1G
> Swap:            0B          0B          0B
 

> A restart of tomcat resolves the issue for a period of time, I have just written a cron job that restarts tomcat on appearance of this issue.

I've done Linux system admin/engineering for many years, and, from my point of view, those numbers from the output of free look just fine.  While it's easy to look at the "free" column and see 128M and think your system is short on RAM, the "available" column is what really counts.  Linux uses available RAM to cache and buffer things like disk and network I/O, and your system is consuming 2-ish GB for that.  Memory allocated for buffer/cache can be easily freed when applications need it, so that's why the available column shows 2.1GB.  So, whenever you ran the "free" command on your system, the system itself is fine on RAM (for the moment) - it's most likely a Java heap size issue (-Xmx flag needs to be set).

 

-Nick

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Server Out Of Memory

Mike Jumper
On Thu, Aug 3, 2017 at 6:50 PM, James Fraser
<[hidden email]> wrote:

> Hi Nick
>
> Thanks for your response.
>
> After sending off this message I did some digging.
>
> I am using JDBC and LDAP auth together.
>
> I was digging around the Server Heap error and think that you are on the
> right track with Xmx value.
>
> It was out of the box (from apt-get) set to -Xmx128m, I have adjusted this
> to 1024m for now and will monitor
>
> We concurrently have around 7 users, each user may be accessing 4-5 VM’s at
> once.
>

Would you be able to take a heap dump to see what is using up so much space?

7 users is relatively light, and having to manually increase the heap
shouldn't be necessary in practice. In past versions of Java, they can
cause more problems than they solve (lengthy GCs), and recent versions
of Java will ignore these options.

- Mike
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Server Out Of Memory

Mike Jumper
On Thu, Aug 3, 2017 at 6:54 PM, Mike Jumper <[hidden email]> wrote:

> On Thu, Aug 3, 2017 at 6:50 PM, James Fraser
> <[hidden email]> wrote:
>> Hi Nick
>>
>> Thanks for your response.
>>
>> After sending off this message I did some digging.
>>
>> I am using JDBC and LDAP auth together.
>>
>> I was digging around the Server Heap error and think that you are on the
>> right track with Xmx value.
>>
>> It was out of the box (from apt-get) set to -Xmx128m, I have adjusted this
>> to 1024m for now and will monitor
>>
>> We concurrently have around 7 users, each user may be accessing 4-5 VM’s at
>> once.
>>
>
> Would you be able to take a heap dump to see what is using up so much space?
>
> 7 users is relatively light, and having to manually increase the heap
> shouldn't be necessary in practice. In past versions of Java, they can
> cause more problems than they solve (lengthy GCs), and recent versions
> of Java will ignore these options.
>

Correction: it's permgen that vanished in recent versions of Java, not
heap limits.

My other points still stand though. ;)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Server Out Of Memory

James Fraser
Evening

So I setup a dump for the next time it happened however it failed with permissions on the folder I used.

First error

21:10:19.472 [http-nio-8080-exec-6] ERROR o.a.g.rest.RESTExceptionWrapper - Unexpected internal error:
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND user_id = 19
            AND permission = 'READ';

        SELECT primary_co' at line 15
### The error may exist in org/apache/guacamole/auth/jdbc/connection/ConnectionMapper.xml
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: SELECT             guacamole_connection.connection_id,             connection_name,             parent_id,             protocol,             max_connections,             max_connections_per_user,             proxy_hostname,             proxy_port,             proxy_encryption_method         FROM guacamole_connection         JOIN guacamole_connection_permission ON guacamole_connection_permission.connection_id = guacamole_connection.connection_id         WHERE guacamole_connection.connection_id IN                           AND user_id = ?             AND permission = 'READ';          SELECT primary_connection_id, guacamole_sharing_profile.sharing_profile_id         FROM guacamole_sharing_profile         JOIN guacamole_sharing_profile_permission ON guacamole_sharing_profile_permission.sharing_profile_id = guacamole_sharing_profile.sharing_profile_id         WHERE primary_connection_id IN                           AND user_id = ?             AND permission = 'READ';
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND user_id = 19
            AND permission = 'READ';

Second Error


java.lang.OutOfMemoryError: Java heap space
Dumping heap to /home/NOTREALUSER/java_pid15204.hprof ...
Unable to create /home/NOTREALUSER/java_pid15204.hprof: Permission denied
Aug 07, 2017 9:05:07 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor doRun
SEVERE:
java.lang.OutOfMemoryError: Java heap space
        at sun.util.resources.TimeZoneNames.getContents(TimeZoneNames.java:285)
        at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResourceBundle.java:137)
        at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(OpenListResourceBundle.java:128)
        at sun.util.resources.OpenListResourceBundle.handleKeySet(OpenListResourceBundle.java:96)
        at java.util.ResourceBundle.containsKey(ResourceBundle.java:1807)
        at sun.util.locale.provider.LocaleResources.getTimeZoneNames(LocaleResources.java:263)
        at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayNameArray(TimeZoneNameProviderImpl.java:124)
        at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayName(TimeZoneNameProviderImpl.java:99)
        at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getName(TimeZoneNameUtility.java:240)
        at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:198)
        at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:184)
        at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(LocaleServiceProviderPool.java:281)
        at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(LocaleServiceProviderPool.java:265)
        at sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayNamesImpl(TimeZoneNameUtility.java:166)
        at sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayName(TimeZoneNameUtility.java:137)
        at java.util.TimeZone.getDisplayName(TimeZone.java:400)
        at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1271)
        at java.text.SimpleDateFormat.format(SimpleDateFormat.java:966)
        at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936)
        at java.text.DateFormat.format(DateFormat.java:345)
        at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHttpDateFormat.java:117)
        at org.apache.coyote.http11.AbstractHttp11Processor.prepareResponse(AbstractHttp11Processor.java:1534)
        at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:739)
        at org.apache.coyote.Response.action(Response.java:179)
        at org.apache.coyote.Response.sendHeaders(Response.java:350)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:335)
        at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317)
        at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:110)
        at com.sun.jersey.spi.container.servlet.WebComponent$Writer.flush(WebComponent.java:315)
        at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.flush(ContainerResponse.java:145)
        at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1085)
        at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1606)

Followed by

SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.

I will adjust the dump for next time.

As noted the machine has 4GB of ram and around 7 users with 3-4 concurrent connections.
In 4 months on 0.9.12 I experienced an issue once which required blipping tomcat but did not delve into it so cannot confirm if the issue was related.

Looking at the glaring error it appears to be JDBC/MYSQL related.

To give some knowledge I did upgrade the database schema and also downloaded the 5.1.43 JDBC driver.


Anyway it is late here in Australia and I need to get some sleep.
Will update tomorrow with MYSQL version etc if requested.

Cheers




JAMES FRASER • MICROSOFT SYSTEMS ENGINEER

-----Original Message-----
From: Mike Jumper [mailto:[hidden email]]
Sent: Friday, 4 August 2017 12:05 PM
To: [hidden email]
Subject: Re: Server Out Of Memory

On Thu, Aug 3, 2017 at 6:54 PM, Mike Jumper <[hidden email]> wrote:

> On Thu, Aug 3, 2017 at 6:50 PM, James Fraser
> <[hidden email]> wrote:
>> Hi Nick
>>
>> Thanks for your response.
>>
>> After sending off this message I did some digging.
>>
>> I am using JDBC and LDAP auth together.
>>
>> I was digging around the Server Heap error and think that you are on
>> the right track with Xmx value.
>>
>> It was out of the box (from apt-get) set to -Xmx128m, I have adjusted
>> this to 1024m for now and will monitor
>>
>> We concurrently have around 7 users, each user may be accessing 4-5
>> VM’s at once.
>>
>
> Would you be able to take a heap dump to see what is using up so much space?
>
> 7 users is relatively light, and having to manually increase the heap
> shouldn't be necessary in practice. In past versions of Java, they can
> cause more problems than they solve (lengthy GCs), and recent versions
> of Java will ignore these options.
>

Correction: it's permgen that vanished in recent versions of Java, not heap limits.

My other points still stand though. ;)
Loading...