S3 is very slow or not working - help!

dreamobjects

#1

I use DragonDisk and CrossFTP to access s3 buckets and it got very slow or not working at all.

Any suggestions?

CrossFTP shows error when connecting

“[R1] LIST Streaming Distributions
Secure random seed initialized.
[R1] CloudFront Error: Request failed with CloudFront Service error (CloudFrontServiceException: httpResponseCode=403, errorType=Sender, errorCode=InvalidClientTokenId, errorMessage=The security token included in the request is invalid, errorDetail=null, errorRequestId=9cd00126-f3bb-11e6-a8a3-b105e355e18d )
[R4] CloudFront Error: Request failed with CloudFront Service error (CloudFrontServiceException: httpResponseCode=403, errorType=Sender, errorCode=InvalidClientTokenId, errorMessage=The security token included in the request is invalid, errorDetail=null, errorRequestId=9cd0eb86-f3bb-11e6-b25a-c9dd96119e26 )
[R1] LIST Download Distributions
[R5] CloudFront Error: Request failed with CloudFront Service error (CloudFrontServiceException: httpResponseCode=403, errorType=Sender, errorCode=InvalidClientTokenId, errorMessage=The security token included in the request is invalid, errorDetail=null, errorRequestId=9cd13911-f3bb-11e6-838b-358e4e83b187 )
[R3] CloudFront Error: Request failed with CloudFront Service error (CloudFrontServiceException: httpResponseCode=403, errorType=Sender, errorCode=InvalidClientTokenId, errorMessage=The security token included in the request is invalid, errorDetail=null, errorRequestId=9cd0eaf0-f3bb-11e6-838b-358e4e83b187 )”


#2

Seeing similar issues here as well. It might have started a couple days ago as I just noticed some backups were not completing… I’m using duplicity to backup and it just started throwing SSL timeout errors. Running duplicity manually, I can see that uploads will work for a little bit then stall out. When they’re working, I’ll get normal upload speeds (3+MB/s).

Connecting via Cyberduck and I can browse around the archives but occasionally the Cyberduck will stall and complain about timeouts.


#3

Same here! Contact support always says that just added "new hardware"and ceph is rebalancing… Why I don’t believe anymore.
I already moving out to alternatives like:
backblaze, memset and rackspace.
Even moving my data out has been difficult, but I am almost free from this mess.


#4

I apologize for the errors and timeouts you’ve been seeing lately. We found a bad static route on some of the new hardware causing extra long delays in rebalancing data and serving traffic. The issue was diagnosed and resolved yesterday evening and has been running smoothly since.


#5

I noticed, well done.

Thanks @Justin Lund


#6

Yup, noticed my backups ran super smooth and fast this morning. Thanks!


#7

However I still see connection errors and my buckets won’t show up for some accounts.

“[R1] LIST Streaming Distributions
Secure random seed initialized.
[R1] CloudFront Error: Request failed with CloudFront Service error (CloudFrontServiceException: httpResponseCode=403, errorType=Sender, errorCode=InvalidClientTokenId, errorMessage=The security token included in the request is invalid, errorDetail=null, errorRequestId=0d8f1fe2-f56e-11e6-8f61-4ff71527bb56 )
[R1] LIST Download Distributions
[R1] CloudFront Error: Request failed with CloudFront Service error (CloudFrontServiceException: httpResponseCode=403, errorType=Sender, errorCode=InvalidClientTokenId, errorMessage=The security token included in the request is invalid, errorDetail=null, errorRequestId=0de81559-f56e-11e6-8f61-4ff71527bb56 )”


#8

@chudak I suspect that’s a different issue. Please open a ticket and have that investigated properly.


#9

The “httpResponseCode=403” makes me think it’s a credentials problem. I also see that there is a CloudFront service exception. Be sure you’re connecting to DreamObjects with the proper endpoint and that you don’t have CrossFTP configured to use CloudFront as that won’t work with DreamObjects.