Start a new topic
Answered

S3 proxy performance issue

Nikolay @ Wed Feb 11 13:08:57 EET 2015
I've been testing CrushFTP to use the Amazon S3 service for backend storage.
I created 100 connections in 50 seconds for different users and transferred file 1M size for the each user.
All FTP users connect to one S3 bucket, however put the file in unique directory. I got 7 failed transfers (log data of the exception below), however the files were uploaded successfully in the S3 bucket. Also, according to the dashboard CrushFTP had 58 concurrent connections.

We going to have around 20k logins/hour.

Could you please help me to solve the issue ?

From CrushFTP.log...

 

USER|02/11/2015 04:26:18 PM|[FTP:223:ftp_test157:10.140.30.84] WROTE: *230 Password OK. Connected. logged in*
PASV|02/11/2015 04:26:18 PM|[FTP:223:ftp_test157:10.140.30.84] READ : *PASV *
PASV|02/11/2015 04:26:18 PM|[FTP:223:ftp_test157:10.140.30.84] WROTE: *227 Entering Passive Mode (10,102,178,45,131,41)*
STOR|02/11/2015 04:26:18 PM|[FTP:223:ftp_test157:10.140.30.84] READ : *STOR /test_157.xml*
STOR|02/11/2015 04:26:19 PM|[FTP:223:ftp_test157:10.140.30.84] WROTE: *150 Opening BINARY mode data connection. Ready to write file . "/test_157.xml") S T O R*
PROXY|S3Client:[FTP:223:ftp_test157:10.140.30.84] PROXY : ftp_test157/test_157.xml
PROXY|S3Client:[FTP:223:ftp_test157:10.140.30.84] PROXY : {url=s3://...@s3.amazonaws.com/dev-ftp-storage/ftp_test157/test_157.xml, name=test_157.xml, owner=owner, time_or_year=16:26, privs=-rwxrwxrwx, num_items=1, count=1, month=Feb, permissions=-rwxrwxrwx, type=FILE, day=11, modified=1423671980000, group=group, path=/ftp_test157/, size=1113985}
PROXY|S3Client:[FTP:223:ftp_test157:10.140.30.84] PROXY : 
NoSuchUploadThe specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.4n60ThV2wIZ6Iwtzgdq63QdEDFPCDGt1kocY3iPWHOgjFYRaH9xvZnx7_9DhQTt0kkC5vTTmcZ1j_aMKDT.pJDvW9OfhMsxm.z9VST0rvDbmQtMVZv7uPIz6nWZfvoMz5E683EF3202DDA8DBFJBa0MvHyEmULUe4lIOUa6OROr6V6Ntdxm9C1FckLlaN2Z7rWvvgt+cTL6X78hz
STOR|02/11/2015 04:26:21 PM|[223:ftp_test157:10.140.30.84] WROTE: Error with files (path):/
STOR|02/11/2015 04:26:21 PM|[223:ftp_test157:10.140.30.84] WROTE: Error with files (name):test_157.xml
PROXY|S3Client:[FTP:223:ftp_test157:10.140.30.84] PROXY : ftp_test157/test_157.xml
PROXY|S3Client:[FTP:223:ftp_test157:10.140.30.84] PROXY : {url=s3://...@s3.amazonaws.com/dev-ftp-storage/ftp_test157/test_157.xml, name=test_157.xml, owner=owner, time_or_year=16:26, privs=-rwxrwxrwx, num_items=1, count=1, month=Feb, permissions=-rwxrwxrwx, type=FILE, day=11, modified=1423671980000, group=group, path=/ftp_test157/, size=1113985}
STOR|02/11/2015 04:26:21 PM|[FTP:223:ftp_test157:10.140.30.84] WROTE: *550-
STOR|02/11/2015 04:26:21 PM|550 NoSuchUploadThe specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.4n60ThV2wIZ6Iwtzgdq63QdEDFPCDGt1kocY3iPWHOgjFYRaH9xvZnx7_9DhQTt0kkC5vTTmcZ1j_aMKDT.pJDvW9OfhMsxm.z9VST0rvDbmQtMVZv7uPIz6nWZfvoMz5E683EF3202DDA8DBFJBa0MvHyEmULUe4lIOUa6OROr6V6Ntdxm9C1FckLlaN2Z7rWvvgt+cTL6X78hz ("/test_157.xml") STOR*
QUIT|02/11/2015 04:26:21 PM|[FTP:223:ftp_test157:10.140.30.84] READ : *QUIT *
STOR|02/11/2015 04:26:21 PM|[223:ftp_test157:10.140.30.84] WROTE: Error:java.io.IOException: 
STOR|02/11/2015 04:26:21 PM|NoSuchUploadThe specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.4n60ThV2wIZ6Iwtzgdq63QdEDFPCDGt1kocY3iPWHOgjFYRaH9xvZnx7_9DhQTt0kkC5vTTmcZ1j_aMKDT.pJDvW9OfhMsxm.z9VST0rvDbmQtMVZv7uPIz6nWZfvoMz5E683EF3202DDA8DBFJBa0MvHyEmULUe4lIOUa6OROr6V6Ntdxm9C1FckLlaN2Z7rWvvgt+cTL6X78hz

 

Thanks,
Nikolay

Best Answer

The latest build should have this issue fixed now.  The speed has been improved too by using caching of dir listings during uploads.


Please let me know if there are further issues.


Thanks,

Ben


1 person has this question

Hope you visit this new forum, I emailed you on the old one letting you know the topic moved.  Just want to update you.  We've made progress and have replicated the errors you reported here on our side now.  So we retrying to understand what is going wrong here and will update you here when we do.


Thanks,

Ben

Sound good.
Thank you Ben!

Answer

The latest build should have this issue fixed now.  The speed has been improved too by using caching of dir listings during uploads.


Please let me know if there are further issues.


Thanks,

Ben

Hello Ben

The latest build has better performance, however this is not apply to our need 20k logins/hour.

I've tested new build in 200 connections in 200 second and got 31 connection fail (log data of the exception below). Also, the file was not uploaded to the S3 bucket for fail connections.

I tried to change setting of the server such as s3_threads/s3_buffer/s3_max_buffer_download, however no affect.

 

USER|02/19/2015 02:35:01 PM|[FTP:10410:ftp_5test129:10.140.30.84] WROTE: *230 Password OK.  Connected. logged in*
PASV|02/19/2015 02:35:01 PM|[FTP:10410:ftp_5test129:10.140.30.84] READ : *PASV *
PASV|02/19/2015 02:35:01 PM|[FTP:10410:ftp_5test129:10.140.30.84] WROTE: *227 Entering Passive Mode (10,150,92,170,155,197)*
STOR|02/19/2015 02:35:01 PM|[FTP:10410:ftp_5test129:10.140.30.84] READ : *STOR /test5_129.xml*
S3_CLIENT|Signing data:----------------
GET


Thu, 19 Feb 2015 14:35:01 GMT
/dev-ftp-storage/
----------------
PROXY|S3Client:[FTP:10410:ftp_5test129:10.140.30.84] PROXY : Looking for failed transfers, or in progress transfers.
S3_CLIENT|Signing data:----------------
GET


Thu, 19 Feb 2015 14:35:01 GMT
/dev-ftp-storage/?uploads
PROXY|S3Client:[FTP:10410:ftp_5test129:10.140.30.84] PROXY : <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchUpload</Code><Message>The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.</Message><UploadId>VuO2Mq3jOKiZCY4N58uo2Vo_LcFvdcpoTGRCCm3rY0GoTS_.zPYcnKocjvuxndgvV0ARoUebL0VKjqZDRiUx56EqoTLEHGhP7jAhXp8YevvExr.SU5sTAdcM5GwtfWH3</UploadId><RequestId>DB45C585D7F0C8FB</RequestId><HostId>Lcze9j1/CB9Opz4X63+BN8B8PAtUDo88fNCRzyzY3fsVpUDHy8BRsk9eozZqI3z9IByQn57GR50=</HostId></Error>
FTP_SERVER|02/19/2015 02:35:01 PM|ftp_5test129:(10410)-10.140.30.84 (control)
QUIT|02/19/2015 02:35:01 PM|[FTP:10410:ftp_5test129:10.140.30.84] *Disconnected.*

 

I've moved this into ticket 1716 so that we can better track this.  Working to replicate and debug what is going wrong.


--Ben

Login to post a comment