DreamPress Site Broke Aug. 21 (tons of memory errors) but nothing changed on my end

wordpress

#1

Last Wednesday (August 20th) everything was working fine on my DreamPress hosted blog. When I logged in the next morning it was broken. I could neither access my site, nor the admin panel (which was throwing a memory error and white screen of death). This was odd, since nothing had changed on my end.

http://estrogenarmy.com

I’ve spent the past handful of days going through the logical steps to troubleshoot this. First, I disabled every single plug-in on the back end. This did, in fact, bring the site back up, so I assumed perhaps there had been a hotfix that might have created problems. I began to re-enable each of the plug-ins one by one, until it broke again. This happened pretty quickly, with only a small handful of things enabled, when I turned the NextGen Gallery plug-in back on. It seemed I had found my culprit, so I disabled it again, and tried to turn the next plug-in back on. This is where I realized it seemed unlikely to be a plug-in problem, as every single plug-in I tried to turn on after that did the same thing: broke the site, and threw a memory allocation error in the admin panel.

Next I went down and began physically deleting plug-ins through the admin panel, getting rid of at least half of what I had been using (completely without any problems until now) to try to ensure a lean load. Still, no joy.

This makes absolutely no sense, because it’s behaving the way it did when I hosted my blog on the regular DreamHost service. There, I expected memory errors once I started to build out a more robust blog using the BuddyPress community plug-ins. I migrated to the DreamPress service in March, and it’s been brilliant ever since. Something happened after midnight on the 21st that has basically thrown my site back into the stoneage. It feels like it’s back on the old side all over again.

In addition to the memory allocation errors it keeps throwing in the admin panel, I am encountering problems in almost every single function on the web site (from saving posts, to using functions in plug-ins). The errors are random, and often non-specific, which makes it incredibly difficult to see a pattern to help with troubleshooting.

I am wondering if my database has become corrupted, but don’t have enough working knowledge here to know how to check that, or what to do about it if it is.

I’ve also run it through the http://isvarnishworking.com tool to check on the slowness issues that kicked in the day this all went nuts, and it says it’s working “sort of”, suggesting I try it again later. I’ve done it three days in a row and get the same results, so I need to do a bit more digging here:

[quote]Varnish appears to be responding at that url, but the “Age” header is less than 1.

This means that either, 1) you checked right when Varnish cleared it’s cache for that url, or 2) for whatever reason Varnish is not actually serving the content for that url from cache.

If it is the former, just check again and see if you get a more solid confirmation.

If it is the latter, perhaps one of the following is happening:

That url is excluded from the cache on purpose in the Varnish vcl file (in which case, yay! It's working.)
The application is sending cache headers that are telling Varnish not to serve that content from cache. This means you'll have to fix the cache headers the application is sending to Varnish. A lot of the time those headers are Cache-Control and/or Expires.
The application is setting a session cookie, which can prevent Varnish from serving content from cache. This means you'll have to update the application and make it not send a session cookie for anonymous traffic.[/quote]

Speed issues aside (I can live with those at the moment), I’ve simply hit a wall with trying to figure out what is causing the sudden memory issues and the bumper crop of general functionality issues, and need to reach out for help. This is crippling, as I am about to go live next month, with a blog I can no longer update or manage.

I am still able to ftp in to upload files the same way, which suggests my DreamPress hosting is still intact, but the behaviour suggests otherwise.

What might I be missing?

Thanks a bunch for any help you might be able to provide me with in getting this corrected.

Sue.


#2

Is there somewhere else I should be asking for assistance with this? Calling the Tech Support line perhaps?

I am completely hobbled at the moment, and unable to move forward with a live launch looming imminently.

Thanks a bunch,
Sue.


#3

Opening a ticket or using live chat (which was off yesterday for a company meeting) would be faster than the forums in general. That’s just always going to be true :slight_smile:

But.

BuddyPress and DreamPress aren’t friends. There’s a logical reason for this. DreamPress gets it’s speed via Varnish, a front end cache that sits between your server and your visitors. And like pretty much every cache tool in the universe, if you’re a logged in user, you don’t get a cached page. Sadly, forums and membership driven sites pretty much live by having users be logged in. So right away you’re never going to get the same level of speed as other sites will on DP :frowning:

I took a quick look at your site and set up so Varnish will never run, because you have a cookie being set on every page load:

bp-activity-oldestpage=1

The problem with that is one of the ways Varnish detects if a user is logged in or not is if it has cookies Varnish doesn’t recognize. We can certainly tell it to ignore that cookie but it’s really not going to help a huge deal with BuddyPress, assuming you will have a lot of members (and looking at your plugins, I think that’s a safe bet).

That doesn’t mean I can’t do anything to help you out! I’ve changed you to PHP 5.4 (better, faster, makes julienne fries!) and added more memory to that. I also reactivated Memcached for you, since on PHP 5.4 it plays better with Advanced Custom Fields.

That will help out some, but without the varnish caching, it’s going to be a tough road :frowning:


#4

You. Rock. I’m sending cookies. Really … lol.

First of all, the changes you’ve made fixed the problem(s) immediately. Thank you!

Second, thank you for letting me know what I need to expect with the current BuddyPress plug-ins I’ve chosen, and for making changes that allow me to carry on in spite of this for now. I am very grateful. This will be helpful as I establish a budget for putting a small team behind properly developing this site over the next few months, and it sounds like perhaps I need to find a another solution. I had to bolt a prototype together to get things moving, and have actually been really happy with how well it functioned on DreamPress until it all went to hell in a hand-basket last week. (Still unclear about what suddenly broke it when it had been working fine the past 5 months …)

You not only dug into what I had configured there based on how specific your response was, but had a sense of humour about it.

I really appreciate your help! Thanks so much.

Sue.


#5

Poking my head back in the door because it seems perhaps the server has been reset to the old varnish configuration for my site.

@Ipstenu-DH do you know if this is perhaps the case? The site is crawling at a snail’s pace, and manifesting a number of the same issues again (inaccessible web panel, inability to post new content, horrendous lag) and throwing “varnish” errors now. As you had been kind enough to turn varnish off for my site back in August, I was a little surprised to see the latter.

Wondering if when the server migration occurred this might have been reset?

Thanks a bunch for any assistance you might be able to provide in checking into this. I’m really grateful for the support!

Sue.


#6

I didn’t actually turn Varnish off. I can’t. The way we locked down the DreamPress instance right now, it’s just on because that’s most of what makes DP awesome.

I out a word when I said this :

What I meant was “I took a quick look at your site and it’s set up…”

Basically that cookie tells Varnish “No, don’t run.”

I restarted apache on your box and that seems to have picked up speed again. It looked like a couple PHP processes got stuck. But there aren’t any major errors on your site right now, and the only thing in the logs is that it looks like you’re getting hit by a DDoS this past weekend (it happens, they’re pretty random). You have Extra Web Security on, so that will protect you.

Your background image is huge, by the way, and slow to load: http://estrogenarmy.com/wp-content/themes/cbox-theme/assets/images/bg-body.gif - You may want to try making that smaller than 1.4M


#7

Thank you so very much for taking the time to offer such detailed information, and for restarting the box. I really appreciate it!

The background is temporary, but you’re right: the size is too big to make it practical. I need to swap it out.

Regarding the DDoS: is there any way for me to know when this is happening? My guess is the logs you reviewed are not likely ones I have access to.

Thanks again for your support. Very grateful!

Sue.


#8

The log is available, it’s your error log, found in ~/logs/domain.com/http/

Reading it and understanding what’s happening is another story. I’m still learning new things :slight_smile:


#9

[quote=“Ipstenu-DH, post:6, topic:61893”]
I restarted apache on your box and that seems to have picked up speed again. It looked like a couple PHP processes got stuck. But there aren’t any major errors on your site right now, and the only thing in the logs is that it looks like you’re getting hit by a DDoS this past weekend (it happens, they’re pretty random). You have Extra Web Security on, so that will protect you.[/quote]

Seems like I’m back in the same boat again. (The error log shows a critical web attack today, deep directory recursion). 10-20 second response time when clicking a link right now, and even before today it’s been like that. Horrendous sluggishness on every single operation. I just opened a support ticket to ask for help.

I am wondering if there is ANY service DreamHost offers on which a BuddyPress installation of WordPress works? You were kind enough to tell me that DreamPress and BuddyPress do not get along because of Varnish. Is there any other option? Do I need to look at VPS? A dedicated server? Someone other than DreamHost? There seem to be a lot of people that do make it work, and scalable, and continuing to pay the extra money every month for DreamPress seems to be a waste.

Would be very grateful for any advice you might be able to offer so I can determine next steps.

Many thanks,
Sue.


#10

Thank you so much!