Right now, I'm P*@$ed!

I am far from an “expert” in coding, I can manage basic stuff, a little bit of hard coding, but when it comes to in-depth server management, I really am clueless. My websites are very database heavy, and from previous problems, we switched both our hosting and MySQL to VPS.

and everything was working great, the VPS panel gave nice reporting so I knew where to set my RAM levels and everything was working smoothly.

THEN, for a reason that I have never been given other than “we’re still developing it”, they switched to the useless panel and no reporting. I don’t even know how much RAM I’m needing to plan for because the useless graph doesn’t even show how much cached memory i’m using.
So with some google assistance, I used sudo access to install PS Manager, and once again, everything was running smoothly.

NOW, with barely two weeks notice, that is gone and i’m back to crashing websites.

We also use a lot of storage space because of the file transferring features that we have for our customers. (I run the sites for 5 print/copy centers) because of this the DreamCompute was not a cost effective option. Oh, and they really don’t have a solution for what to do with my MySQL VPS hosting… at least, the support convo I had was less than helpful.

so, mostly a big rant, but open to any suggestions anyone might have…

Hi Alith,

I am sorry to hear about your problems, I can barely imagine what you’re going through. I did a project in the past working with print services and, boy, they can produce large files :slight_smile: We built a system to allow customers to assemble pages from an online catalogue and print it, plus we allowed photographers to upload hi-res images for the print-quality export… it was fun but required lots of horse-power. And this was many years ago, before modern things like object storage and replicating file systems. Fun times.

Anyway, to your case, I don’t know if you’ve already installed and setup a monitoring system on your servers: I think that besides RAM you many want to keep an eye on the application itself (is it up? how fast is it responding to users?), disk IO, network IO. There are a bunch of tools, from free/open source “do it yourself” to affordable/almost free online software as a service. From a recent survey I did, I suggest:

I suggest you to try one of these, if you’re not using one already.

On the storage space issue, I think I can help you but before saying something completely silly, maybe you can give me a link to your customer’s site so I can see what it does (if possible) or you can explain the architecture of your application. There may be things that you can change so that disk usage, backups, speed of file transfer etc can be mitigated. I’m available also in private message if you prefer not to share too many details in public.

I don’t mind sharing.
the two sites are http://geprinting.us and http://digitaledgecc.com.

the main part is the file uploader. we have had this service for our customers for a very long time, and it provides them with private little file manager folders so they can upload and manage their files and no one but us can see.

it used to run pretty smooth once I switched from shared to VPS and then even better when I added the PS Manager script through the sudo access. but now I’m back to square one with it stalling and seizing all the time. The biggest reason I liked the PS Manager was because our usage was primarily “business hours”. So M-F 8-5 was generally pretty heavy, but outside those times, the sites get very little use. PS Manager monitored what the usage was and adjusted the RAM accordingly based on what was cached to keep things flowing smoothly.

Hi, one of the DreamObjects developers here.

I don’t know what code you have already ensuring that the content is kept private between multiple customers, but I can comment on how your uploader should evolve from using FTP to using DreamObjects.

DreamObjects implements the S3 protocols, so you can use most S3 functionality on your websites.

  • Presigned URLs will allow your clients to upload directly to DreamObjects, simply by going to the upload functionality on your sites (and call back to your site when the upload is completed).
  • Presigned URLs also mean that you can allow downloads of content back again without the entire file having to be sent by your VPS (your site would simply give the customer a redirect to the presigned DreamObjects URL)
  • You are concerned about large files, so you should definitely look at the multipart uploads (which as a bonus also support resuming).
  • You WILL need to set a CORS policy on the DreamObjects bucket, to allow the clients to upload.

DreamObjects does NOT implement the Amazon IAM access control and bucket policies, so if you see a tutorial that mentions those, you’re going in the wrong direction.

Here is some of the DreamObjects-specific PHP documentation:
(the main change from any Amazon S3 documentation is remembering to specify to connect to DreamObjects and not the default of Amazon’s servers)

Here’s an example of direct upload, written in Node, and mentioning DreamObjects specifically (FYI CORS readme comment is out of date)

Here’s an example of direct upload with multipart, written in PHP

I haven’t used either of the preceeding examples myself, but I have used the same functionality in some other code I can share with you:
https://www.drupal.org/project/amazons3_cors (with modifications here https://github.com/BCLibCoop/nnels-drupal-modules-amazons3_cors)

I see, thanks for sharing… Indeed, you don’t need only monitoring but something that is more similar to autoscaling (get a larger machine based on demand on the host).

Without API access to the VPS (I don’t think that PS needs sudo), the only thing to do would be to make PSManager speak the DreamCompute API but… let’s first see if the problem with the RAM still exists on DreamCompute.

One thing to be aware of is that memory management on DreamCompute is a lot better than on VPS so you may not have any problem there after all.

From what you said, it looks like your needs are limited to file uploads: the upload.php scripts end up hogging the virtual server during peak times, requiring more memory to put files via FTP. Is that correct?

If that’s the case, probably a faster approach would be to use DreamObjects and the S3 instead of FTP. What Robin said above might work … if you need help please let me know.