Kimler Sidebar Menu

Kimler Adventure Pages: Journal Entries

random top 40

Six Days in June

Filed in:Site News

Six Days in June

June 11th, 2007  · stk

For the past few days, Randsco.com has suffered through a series of server problems, even accused of overloading the server! Read about our love of shared hosting companies

Shared Hosting Problems Beleaguer Randsco

It may very well be that we have outgrown any sort of shared hosting plan. We have grown weary of suffering through server problems, poor loading speeds, high server loads and sharing a server with 543 "who-knows-what-they're-up-to" others.

For six days in June, we waited for SiteGround.com (our shared hosting provider) to fix a server that was on the fritz. During those six days, our site was up and down more times than even Paris Hilton has experienced (well, maybe I'm exaggerating a tad, but you get the picture).

We monitor server loads and we witnessed loads in excess of 150% (how is that even possible)?

In desperation (and by using MSDOS FTP) we posted a warning to our main page - "Watch out! Slow loading site because of crappy shared server. Stay tuned for details."

At the end of the day, after a new server was obtained and Randsco.com (along with the other 543 other sites) moved to this fancy new, high-speed, better-than-sliced-bread server, we received an email saying that our site was using too many resources. :-/ Gee, were we somehow responsible for the greater-than 150% server load on the old server? I don't think so.

To our host's credit, they worked with us to optimize our website and we're hoping that our server woes are now behind us.

To read all about the sordid affair ... press continue ...

Front Page Updates

We attempted to keep folks up-to-date with the progress (or lack thereof), by posting the following message at the top of all of our dynamically served pages:

** SERVER PROBLEMS **

Ongoing for: 6 days, 1 hour, 34 minutes

Update: June 11, 2007 - Siteground has moved Randsco to the new server. Unfortunately, they have informed us that we're "using too many server resources". If you see lots of Red X's, in place of images - or worse, the page styling is completly absent - this is what's happening (they've limited our account and some objects are being "blocked", to automatically lower our load). If it happens, try refreshing, which just puts on more load, but might get you a complete page.

Head here for timeline and more information.

June 8, 2007 - The ETA for a "fix" (move to a new server) will not happen until Monday, June 11th. I anticipate that site disruption will likely last past that point. I am very disappointed at the hosting service we've received (lack of communication, long disruption, no offer of reimbursement, slow response, etc.) Perhaps it is time to find a better host?

June 7, 2007 - Connections to Randsco.com are spotty, as our hosting provider reports they are working to resolve a server issue.

Thank you for your patience and understanding. (This notice will disappear when the problem has been sorted.)

- Scott & Rachel

 

Behind the Scenes

We also provided a link to a "details page", which attempted to follow the technical aspects of the outage:

This page updated on Mon, June 11th, 15:40 PST.

Jun 11th, 2007 - We received another email, this morning, from siteground.com. They assured us that our site was "working smoothly on the new machine".

Poppycock.

Server loads are still greater than their 'stated target' of 5%. (As I write this, sever load averages for BOTH the 5-minute and 15-minute periods exceed 10%).

In addition, we have the fun (new) problem that the server is serving page elements as 503 (Service Unavailable). The result is that these elements (image files, css, javascript, etc) are being BLOCKED from server agents (browsers) and don't show. Sometimes it's the main CSS file, which means "no page styling" or UGLY WEB PAGE!

It's enough to make one angry.

Later in the day, (after filing a support ticket and chasing my tail), I discovered that Siteground deem Randsco to be overloading the server. But I only find out about it after receiving an email. (Would have been nice to be informed BEFORE limitations were put into place). Grrrr.

After running DEBUG and checking mySQL load times, I basically could see that the new AstonishMe! plug-in "Search Cloud", was using up a lion's share of the load. Disabling this plug-in made a dramatic effect on loading speeds. I'm sad to see it go (and may work find a way to deploy it so that it's only called when it's clicked, rather than on each page load).

I'm hoping this will be an end to our recent server problems.

Here's the email exchange about the server overloading issue.

 

Dear Scott,

During the last 24 hours, we have noticed that your website was consuming a very high percentage of the server resources and endangered the performance of your site and all the sites on the server. Our Admin department started an immediate investigation to resolve the problem and decrease the high server load. Unfortunately, we had to limit the shared server resources you may use. This may result in occasional "Resource temporary unavailable" error messages.

A lot of people are accessing your website, using most of the httpd child processes. This endangers the accessibility of the other websites and causes severe load on the shared server.

We have seen that your website has a huge potential and will continue growing fast. That is why we also recommend you to consider a VPS (virtual private server) as a possible solution for the growing needs of your website. On the VPS, you will not have to share the server resources with other customers and running many simultaneous connections will not load the server.

If you wish to upgrade to a VPS, you may do that from the Extra Features section in your Customer Area. Please note that for the special needs of your website, we strongly recommend you to upgrade to the Elite or the Professional VPS.

Please contact us to inform us what solution you will take to optimize the performance of your website so that we can remove the connections limitation as quickly as possible.

Best regards,

Borislav
System Administrator
SiteGround.com

Borislav,

Please supply me with a copy of the server logs, showing me the time and percentage of resources being used, so that I may determine what is the cause of the problem.

Thank you.
-stk

Hello Scott,

The problem is caused by the fact that your web site is very popular and is accessed by many visitors:

Total visits for 48h: 17087 Average visits per hour: 355.979 Average visits per min. 5.93299

There are many links on the main page:

176. http://randsco.com/?blog=7&page=1&paged=39
177. http://randsco.com/#top
178. http://randsco.com/about.php?title=site_faqs
179. http://randsco.com/about.php?title=about_stk
180. http://randsco.com/about.php?title=copyright_policy
181. http://validator.w3.org/check?uri=referer
182. http://jigsaw.w3.org/css-validator/check/referer
183. http://b2evolution.net/
184. http://randsco.com/about.php?title=privacy_policy
185. http://randsco.com/index.php?disp=contact

real 0m2.571s
user 0m0.028s
sys 0m0.012s

When a visitor opens it, there are many SQL queries generated. Some of them need more time to be completed. The slow queries increase the load on the server and your web site uses more that the allocated server resources. It loads locally for 2.5 seconds, while the normal time is 1 second.

If you want I can allow full access to the web site for your IP and then you can optimize it.

Best Regards,

Svetlio
Senior Support Team
SiteGround.com

Svetlio,

Thank you for the information. You've mentioned three things and I'd like to have an understanding of each of them.

1) Visits - WOW!! That';s a LOT more visits than any other measurement methods: (17,087 for 2 days??)

Google Analytics -
408 Visits on Jun 9th,
475 Visits on Jun 10th
(From May 10th to Jun 10th - 17,163 UNIQUE Visits and 19,109 Non-unique visits).

ClustrMaps - 7394 Visits since 1-Jun-2007

b2evolution Statistics - (Includes XML/Robots/Browser/Unknown)
1,238 "Hits" so far today 11-Jun-2007
1,649 "Hits" on 10-Jun-2007

Webalizer Stats -
09-Jun-2007 - Hits: 34878; Pages: 7063; Visits: 1114
10-Jun-2007 - Hits: 966; Pages: 174; Visits: 56
11-Jun-2007 - Hits:11751; Pages: 2067; Visits 287

Maybe your "visits" are "hits"? (Each element on each page?)

2) Links. Yes, we have a lot of links on our main page. I don't understand why this would cause a server load? When a link is loaded, the server doesn't have to "do anything". A link is just text link.

Can you please explain why having 185 links on a page causes a server to overload?

3) Yes, our b2evolution blog is a PHP coded / mySQL database application. Many SQL queries ARE generated ... 53 to be exact and they appear to take approximately 1.934 seconds to execute.

(I have turned on the debug feature of b2evolution, so that you may see all of the database queries and the total time that it takes to load our page.)

I await your reply (but don't want to leave debug on for a very long time, as it is resource intensive).

Cheers,
-stk

PS - I did notice that a particular plugin that I was using on the site was using a lot of resources (seemed to be causing a big delay in the SQL query).

It also generates 40 links, which would help cut down the number of links (still don't understand why links put a load on the server).

Anyway ... have a look, as I'm noticing a much improved loading speed (0.69 total seconds for load) ... rather than the 2 seconds I first reported.

Does that help?

Hello Scott,

The provided statistics are for the hits on the web site.

Each link has corresponding records in the database and every time a visitors opens the web sites all the links are loaded and queries to the database are generated. If there are many users, who open the web site in the same time, this will increase the load on the server.

However, I just checked and since the web site loads for 1.2 seconds I restored the full access to it.

We will continue to monitor the web site and if there are issues in the future, we will notify you.

Should you have further questions, please refer to our HelpDesk at any time.

Best Regards,

Svetlio
Senior Support Team
SiteGround.com

Svetlio,

Thank you.

1) Hits ... okay ... that makes sense.

2) Only DB-generated links fall into that category - TRUE.

FYI - There are a great number of links (either included in the template or are put into a blog post, for example) that do NOT result in any PER-LINK DB queries. For example ... a blog post is one mySQL DB call (the post) , regardless of how many LINKS it might contain in the post code.

3) OKAY ... I'll leave OFF the "search cloud" feature of our website, as that appears to be the "problem". (And turn back off the debug).

Thank you for working with me and helping me to find the cause of the problem. I understand that we are on a shared server and share resources with over 500 other websites. Please accept my apologies for any problems that our site (specifically this apparent resource-intensive plug-in). We were unaware that it was a drain.

Cheers,
-stk

 

Here's the morning email that we recieved, assuring us that all was swell.

Subject: SiteGround Server Transfer Successfully Completed

Dear scott kimler,

The transfer of your host server was successfully completed. Your account(s) randsco.com is now working smoothly on the new machine.

Thank you for your understanding and cooperation during the transfer. We do our best to provide you with high quality hosting service!

Regards,

The Siteground Team

 


 

Jun 8th, 2007 - We received an email today, from our hosting provider, saying that they won't be moving Randsco.com to the new server until Monday, June 11th.

I suspect they're having some real problems. They had once told us "not to make any changes to our site, to avoid data loss" (2 days ago), but in the new email, they said not to make any changes until after Sunday. This makes me think they were going to move things sooner, rather than later, but ran into some type of issue.

I'm disappointed that they aren't more upfront about what is happening.

The full copy of the email is below:

This message was posted on Fri, June 8th, 11:15 PST.

Email - Sent June 8th

During the last few weeks, SiteGround system administrators have been carefully monitoring the performance of your host server siteground163.com. Recently, they concluded that the machine performs below our quality standards and needs to be replaced with a more stable and powerful one. Therefore, we will transfer your website(s) randsco.com to a more powerful machine on Monday, June 11, 2007.

Your new server is an Intel Woodcrest 5130 Dual Core with 4GB RAM. The new server will increase significantly the quality of our service in the following aspects:

** the new server will be more powerful and your site will load faster;
** the server configuration will allow future upgrades of space and traffic;
** the new hard drives are larger (more space) and faster (better performance for your site).

IMPORTANT INFORMATION ABOUT THE TRANSFER:

** The transfer of your account will be performed on Monday, June 11, 2007.
** There will be no information loss during the transfer, however, SiteGround strongly recommends that you do not perform any changes on your website starting from Sunday, June 10, 2007 until the transfer is completed.

This email is for your information only. We would like to kindly ask you not to post tickets in regards to the transfer. In your Customer Area --> Get Support --> Server Status there will be updated information about the progress of the transfer.

Thank you for your understanding and cooperation. We are looking forward to offering you a higher quality of service!

 


 

June 7th, 2007 - Our Hosting company system administrators will be transferring Randsco.com (and 542 other websites) off of the siteground163 server and onto a newer machine. Randsco.com will not be updated until after this transfer is complete. Unfortunately, Siteground has no provided us an ETA for completion of the transfer. Connectivity will be spotty during the next few hours / (couple of days?)

We will update this message once again as we get more information.

Thank you for your cooperation and understanding in this situation!

Scott and Rachel

This message was posted on Thu 7th, 7:00:03 PST.

Read on for a technical explanation of the hardware/software failure at our hosting company:

Original Server Status Notice - Posted June 7th

During the last few days the server that hosts your account experienced performance issues that cause response time below our quality standards.

The SiteGround system administrators have been trying to upgrade the server kernel to a newer one. The new kernel allows greater security options - applied by the Grsecurity patch - as well as 2 to 3 times better performance.

Despite all our attempts, though, we have still not succeeded in upgrading the kernel as it does not fully support the server CPU architecture and cannot effectively use the server resources. This leads to higher overall load on your hosting server during peak times and occasionally customers report slowness issues like the ones you are experiencing. We understand how inconvenient this might be for you and we assure you that we will concentrate all our efforts in finding the best resolution to these issues as soon as possible.

SiteGround is dedicated to continue security improvements and offering customers the best industry protection and server stability. That is why our security engineers and kernel developers are working on finding the best combination between supported hardware and kernel configuration.

For your account this will involve a server migration to new hardware within the next few weeks.

The new machine will be even more powerful than the current one, which will guarantee greater performance for your website, higher page loading speed and will solve all existing issues. The server will be a 64bit Single Intel Woodcrest 5130 Dual Core machine, with 4GB of RAM memory and will fully support the newer Grsecurity enabled kernel.

Once we confirm the server migration date and schedule all upgrades, we will inform you by email as well as with a notification placed in your SiteGround Customer Area -> Server Status page.

 

Post-Mortem on the Event

We're very happy to have our site back up in all of it's un-limited and "on-something-other-than-a-crap-server" glory. While we're very happy that SiteGround didn't peremptorily toss us onto another server reserved for "bad offenders" (like last time they said we were using too many resources), we're VERY unhappy about the server issues.

We don't think companies should host more than 500 websites - especially 500 diverse and unknown websites - on one server. We think that we should have more timely communication. We also think that a month's free service for a week's worth of server hardware problems would be a reasonable gesture of good-will.

Is SiteGround worse than other shared hosting providers? Probably. Are they better? Most definitely.

Conclusion: We've reached a level where we demand more than a shared hosting company can provide.

While we recognize this, there's precious little we can do about it commercially. The two solutions that come to mind - VPS and Dedicated Server - are both considerably more expensive than what we're paying now ($89 or more per month -vs- $5/month).

It's too bad that web hosts don't have much to offer in the hosting space in between. (Couldn't a host could offer a plan, say for $25 a month, that would guarantee less than 100 sites per server and that each site comprise of long-term, established clients)? That's our idea, anyway.

Fortunately for us, astonishme will likely be obtaining a dedicated Linux server in the near future. We may be moving soon. Off of a shared hosting company and onto a server which we control and has a select clientel. When we do, you'll be certain to notice fewer server-related problems!!

(Permalink)
Views: 14529 views
5 Comments · GuestBook
default pin-it button
Updated: 11-Jun-2007
Web View Count: 14529 viewsLast Web Update: 11-Jun-2007

Your Two Sense:

XHTML tags allowed. URLs & such will be converted to links.


Subscribe to Comments

Auto convert line breaks to <br />

1.flag Yabba Comment
06/12/07
*cough*enable the search cloud cache that was coded in to stop it consuming to many resources*cough* :|

¥
2.flag stk Comment
06/12/07
*cough* not too random then *cough*

-stk
3.flag Gary Comment
06/13/07
You two need to see a Doctor with those bad *coughs*. Glad to see ya back up and running smooth again Scott.

Gz.
4.flag danny Comment
06/14/07
Again with the 'too may links'! I love it. Are they brain dead?
5.flag stk Comment
06/14/07
Gary - LOL ... yep & thanks!

Danny - "Daim Bramaged", I think. :p