Start a new topic

No threads available in pool

Hi,

 

We are having an issue where a client is downloading several files in sequence and for some reason the worker threads are not completing. After a while the threads are all assigned and the server stops being able to handle new connections, at least that is what seems to be happening from looking in the log:



SERVER|No threads left! Busy:800 Available:0 Max:800 ... SERVER|java.io.IOException: No threads available in pool. ... SERVER|Server Memory Stats: Max=1.77 GB, Free=1.15 GB No more workers:800 #######################Start Thread Dump####################### ... abcd:(410077)-195.245.181.52 (idle_time) java.lang.Thread.sleep:-2 crushftp.handlers.IdlerKiller.run:49 com.crushftp.client.Worker.run:36 java.lang.Thread.run:-1 --------------------------------------------------------------------------------------------------- abcd:(410077)-195.245.181.52 (idle_time) java.lang.Thread.sleep:-2 crushftp.handlers.IdlerKiller.run:49 com.crushftp.client.Worker.run:36 java.lang.Thread.run:-1 --------------------------------------------------------------------------------------------------- abcd:(410077)-195.245.181.52 (idle_time) java.lang.Thread.sleep:-2 crushftp.handlers.IdlerKiller.run:49 com.crushftp.client.Worker.run:36 java.lang.Thread.run:-1 ...

 


We have disabled the user account and the situation seems to have normalized but it is worrisome that a single user can break the server.

In the meantime I've tried replicating the situation by cloning the user and the thread usage count remain stable. Is there something that we can do about this?

 

Below is a sample of the file download process being done by this user:

USER|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *CONNECT SSH-2.0-J2SSH_Maverick_1.4.47__Oracle_America,_Inc.
USER|01/22/2018 07:45:59 PM| CipherCS:aes128-cbc CipherSC:aes128-cbc*
MDTM|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *getFileAttributes /*
USER|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *CONNECT SSH-2.0-J2SSH_Maverick_1.4.47__Oracle_America,_Inc.
USER|01/22/2018 07:45:59 PM| CipherCS:aes128-cbc CipherSC:aes128-cbc*
MDTM|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *getFileAttributes /*
MDTM|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *fileExists /EDP/ETD_EDP_20141124.csv:true*
MDTM|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *getFileAttributes /EDP/ETD_EDP_20141124.csv*
MDTM|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *fileExists /EDP/ETD_EDP_20141124.csv:true*
RETR|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *RETR START /EDP/ETD_EDP_20141124.csv*
RETR|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *RETR /EDP/ETD_EDP_20141124.csv:Y7kpB*
RETR|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] WROTE: *150 Opening BINARY mode data connection for /EDP/ETD_EDP_20141124.csv (27009 bytes). ("/EDP/ETD_EDP_20141124.csv") R E T R*
RETR|01/22/2018 07:45:59 PM|226-Download File Size:27009 bytes @ 26K/sec.
RETR|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] WROTE: *226 Transfer complete. MD5=0b6147077f373057803e6a60f6488bbc ("/EDP/ETD_EDP_20141124.csv") RETR*
RETR|01/22/2018 07:45:59 PM|[406896:abcd:195.245.181.52] WROTE: *Adding ETD_EDP_20141124.csv to event tracking.*
RETR|01/22/2018 07:45:59 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *RETR END   /EDP/ETD_EDP_20141124.csv:Y7kpB*
USER|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *CONNECT SSH-2.0-J2SSH_Maverick_1.4.47__Oracle_America,_Inc.
USER|01/22/2018 07:46:00 PM| CipherCS:aes128-cbc CipherSC:aes128-cbc*
MDTM|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *getFileAttributes /*
USER|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *CONNECT SSH-2.0-J2SSH_Maverick_1.4.47__Oracle_America,_Inc.
USER|01/22/2018 07:46:00 PM| CipherCS:aes128-cbc CipherSC:aes128-cbc*
MDTM|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *getFileAttributes /*
MDTM|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *fileExists /EDP/ETD_EDP_20141125.csv:true*
MDTM|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *getFileAttributes /EDP/ETD_EDP_20141125.csv*
MDTM|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *fileExists /EDP/ETD_EDP_20141125.csv:true*
RETR|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *RETR START /EDP/ETD_EDP_20141125.csv*
RETR|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *RETR /EDP/ETD_EDP_20141125.csv:myBHS*
RETR|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] WROTE: *150 Opening BINARY mode data connection for /EDP/ETD_EDP_20141125.csv (27753 bytes). ("/EDP/ETD_EDP_20141125.csv") R E T R*
RETR|01/22/2018 07:46:00 PM|226-Download File Size:27753 bytes @ 27K/sec.
RETR|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] WROTE: *226 Transfer complete. MD5=421a666c0e694f0133457eb4785ad992 ("/EDP/ETD_EDP_20141125.csv") RETR*
RETR|01/22/2018 07:46:00 PM|[406896:abcd:195.245.181.52] WROTE: *Adding ETD_EDP_20141125.csv to event tracking.*
RETR|01/22/2018 07:46:00 PM|[SFTP:406896_0:abcd:195.245.181.52] READ: *RETR END   /EDP/ETD_EDP_20141125.csv:myBHS*

 


Latest v8 now protects this with a max of 5 channels.  In a few hours the UI will be done too allowing you to control it on the SFTP port if you need to.

Thank you very much Ben, we shall update the server during the weekend.

Login to post a comment