Start a new topic

S3 Service Integration Help

codewzrd @ Fri Feb 14 08:59:47 EET 2014
Hi,

I am evaluating CrushFTP for our business. What I am trying to accomplish is to setup CrushFTP with a S3 as our file storage.

So the first thing I did is to create a user (test_user). Still in the VFS tab, I clicked on the "Create Remote Item" button. I selected S3 as the FIle Type.

I entered a name for the new VFS Item (TestS3). I added "FTP-Service/" to the end of the URL.

I entered my Access Key and Security Key (which I have used via another client, so I know it works).

So the URL looks like this:
s3://removed:removed@s3.amazonaws.com/FTP-Service/

I clicked OK, then Save All Changes.

What I expect to happen (which doesn't) is when I double click on the TestS3 folder, I should see another folder (called US; previously setup via AWS Console). But I do not see the US folder. Within the US folder, I have 1 CSV file. But I don't get that far.

Any ideas on where to look?

Thanks,
-Svapan

spinkb @ Fri Feb 14 09:08:55 EET 2014
As a test, can you try a bucket name that is all lowercase?

No mixed case in your bucket name.

Let me know,
Ben
codewzrd @ Fri Feb 14 09:39:31 EET 2014
No luck. Still can't see a list of files. Is there some sort of log that might show errors connecting to S3?
spinkb @ Fri Feb 14 09:43:09 EET 2014
In the prefs, logging, set your debug level to 2.

Then repeat the test and see what shows in your main CrushFTP log.

Thanks,
Ben
codewzrd @ Fri Feb 14 10:54:00 EET 2014
From CrushFTP.log...

02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *POST /WebInterface/function/ HTTP/1.1*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *Host: localhost:9090*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *Content-Length: 2091*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *Origin: http://localhost:9090*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *X-Requested-With: XMLHttpRequest*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *Referer: http://localhost:9090/WebInterface/UserManager/index.html*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *Cookie: __WEBINTERFACE_PAGE_SIZE=100; mainServerInstance=; CrushAuth=1392396128461_XRcxFLIVIC80MC5QxIJqhpmBmkBqha; __WEBINTERFACE_HIDE_FILTER=false; __WEBINTERFACE_HIDE_CHECKBOX_COLUMN=undefined*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *command:getUserXMLListing*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *format:JSON*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *path:/TestS3/*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *random:0.8277206099592149*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *serverGroup:MainUsers*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *username:test_user*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *permissions:
02/14/2014 11:50:18 AM|
02/14/2014 11:50:18 AM|(read)(view)(resume)
02/14/2014 11:50:18 AM|(read)(view)(resume)
02/14/2014 11:50:18 AM|(read)(view)(resume)
02/14/2014 11:50:18 AM|*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] READ: *vfs_items:
02/14/2014 11:50:18 AM|
02/14/2014 11:50:18 AM|
02/14/2014 11:50:18 AM|TestS3
02/14/2014 11:50:18 AM|/
02/14/2014 11:50:18 AM|
02/14/2014 11:50:18 AM|
02/14/2014 11:50:18 AM|
02/14/2014 11:50:18 AM|s3:********
02/14/2014 11:50:18 AM|{vfs_item=[{haUpload=false, url=s3://removed:removed@s3.amazonaws.com/ftp-service/, secretKey=removed, ssh_two_factor=false, secretKeyID=removed, receive_compressed=false, ssh_private_key_pass=, haDownload=false, ssh_private_key=, encrypted=false, use_dmz=false, type=DIR, expires_on=, send_compressed=false}], name=TestS3, path=/}
02/14/2014 11:50:18 AM|{vfs_item=[{url=FILE://C:/Temp/a.sql, type=FILE}], name=a.sql, path=/}
02/14/2014 11:50:18 AM|Create new GenericClient to url:s3://removed:removed@s3.amazonaws.com/
Signing data:----------------
GET
Fri, 14 Feb 2014 11:50:18 EST
/
----------------
02/14/2014 11:50:18 AM|crushadmin:(566)-0.0.0.0 (control)
02/14/2014 11:50:18 AM[color=red]|java.io.IOException: java.lang.NullPointerException: Input string was null.[/color]
02/14/2014 11:50:18 AM|com.crushftp.client.URLConnection.connect:225
02/14/2014 11:50:18 AM|com.crushftp.client.URLConnection.buildResponseHeaders:331
02/14/2014 11:50:18 AM|com.crushftp.client.URLConnection.getResponseCode:421
02/14/2014 11:50:18 AM|com.crushftp.client.S3Client.login:117
02/14/2014 11:50:18 AM|crushftp.server.VFS.getClientSingle:230
02/14/2014 11:50:18 AM|crushftp.server.VFS.get_item:434
02/14/2014 11:50:18 AM|crushftp.server.VFS.getListing:525
02/14/2014 11:50:18 AM|crushftp.handlers.UserTools.get_virtual_list_fake:1372
02/14/2014 11:50:18 AM|crushftp.server.AdminControls.getUserXMLListing:977
02/14/2014 11:50:18 AM|crushftp.server.ServerSessionAJAX5_2.processItems:1406
02/14/2014 11:50:18 AM|crushftp.server.ServerSessionHTTP5_2.handle_http_requests:1453
02/14/2014 11:50:18 AM|crushftp.server.ServerSessionHTTP5_2.run:157
02/14/2014 11:50:18 AM|crushftp.server.QuickConnect.run:105
02/14/2014 11:50:18 AM|crushftp.server.Worker.run:32
02/14/2014 11:50:18 AM|java.lang.Thread.run:-1
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] WROTE: *HTTP/1.1 200 OK*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] WROTE: *Vary: Accept-Encoding*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] WROTE: *Content-Encoding: gzip*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] WROTE: *Transfer-Encoding: chunked*
02/14/2014 11:50:18 AM|[HTTP:566:crushadmin:0.0.0.0] WROTE: **
spinkb @ Fri Feb 14 14:26:19 EET 2014
Go to your about tab, click update now.

You will get a newer build that should fix this issue. If not, let me know again the debug level 2 log data like you just did.

7.0.2_14

Thanks,
Ben
codewzrd @ Fri Feb 14 15:34:46 EET 2014
So the fix helped me. It showed me the error I was getting.
[color=red]S3Client:PROXY:
AccessDeniedAccess Denied21EF9C23E391C0D2PbQ1FIjt/N/2JQh6xsNB6rSKcupoPGOWp3q+iVIlmtuEJSQr1P+kuHBMF/TemCG1
[/color]

I don't know why I was getting that error message against the FTP-Service bucket. So I deleted the bucket. I created a new bucket and tried again and it worked this time.

I am not sure why I could not access the bucket as I could from my custom .NET client. Anyway, S3 folder names are [b]case sensitive.[/b]
spinkb @ Fri Feb 14 15:46:02 EET 2014
There is a known, impossible to solve bug related to mixed case bucket names. The S3 documentation does not explain how to correct this, and we have spent hours trying to debug what it expected to no avail.

We will not use the S3 libraries as they are a *HUGE* bloat for the protocol that we need to do, and cannot do the advanced things we need as well.

So we are stuck with not being able to access a S3 bucket if it has mixed case.

Sorry,
Ben
Login to post a comment