Site overload issues

[Malvineous]

@DOSGuy: Wow it sounds like you've had a pretty rough time recently. I hope you're doing a bit better now - it can't be easy to come to terms with all that, especially when it all happens so close together and it seems like it's just one thing after another.

As Litude said, the link is to the source control repository. It was created by adding each version on top of the previous one, which is how it would have happened if they had developed the game today. It makes it easier to see the changes between versions. It looks like the dates weren't set for each commit to match the release date which IMHO would have made the repository a bit more "correct", but being git, there's nothing stopping you from making your own copy and doing that yourself!

About the the site being overloaded, have you considered using something like CloudFare? They have a pretty decent free tier which will probably solve the bandwidth-related problems at no cost other than your time setting it up. Basically the service is a giant proxy server, so all requests to the site go to CloudFare, and they then contact your server once per page and cache the result, sending the page out over and over to anyone who asks without contacting your server again. If you suddenly get 100 times your normal traffic you might not even notice, unless you have set certain pages (like locally served ads) not to be cached at all. Plus on top of that you don't have to worry about your hard work disappearing completely, because if your server goes offline they will just keep serving up their cached copies until it comes back, so possibly nobody will notice the outage!

I haven't used their service yet (haven't had any bandwidth problems so far!) but they are quite widely used and I haven't heard anything bad said about them so it might be worth trialling, given that it's free and might save you both headaches and time in the long run.

[DOSGuy]

Sometimes I post references to the forums here in a Russian old gaming forum - could that be the cause? If you just give me the dates of recent peaks I could cross-check them with my posts in that forum which contain links here.

I just used Russia as an example. It's sometimes other countries, but there have been a few occasions when a huge number of people came from a Russian website, like on November 2 when 8500 people came from http://games.mail.ru/pc/articles/feat/5 ... _brauzere/. On September 10 and 11 when got double nailed when 17 000 people came from http://pikabu.ru/story/20_legendarnyikh ... re_2647815, and then 14 000 people came when vk.com picked up the story. We normally get about 4000 visitors a day, but it was 26 000 on September 10, 18 000 on September 11, 8000 on September 12 and 5500 on September 13 before things started to return to normal.

If you had anything to do with that, MrFlibble, keep up the good work! I mean, I don't know what else to say. We want people to come to the site, and I can't really ask them to space themselves out. These occasional spikes are a fact of life for any webmaster, and a good web host understands that.

About the the site being overloaded, have you considered using something like CloudFare?

I checked it out.

SECTION 10: LIMITATION ON NON-HTML CACHING

... Using an account primarily as an online storage space, including the storage or caching of a disproportionate percentage of pictures, movies, audio files, or other non-HTML content, is prohibited. ...

99.9% of the site is non-HTML content. All of the HTML on the site combined is less than 1 MB. The 7 GB of ZIP files (downloads) and JAR files (online games) would probably be considered disproportionate. I'm also not pleased with giving them permission to insert cookies and scripts into my code, per Section 7.

[Malvineous]

I have to admit, I looked into CloudFare as well after I posted about it and there were a few things I wasn't so keen about either. I'm not fussed about the scripts or cookies as they are non-intrusive, but you have to turn over your domain name to them to manage, which for me won't work because I run my own DNS server so I'd have to duplicate the configuration manually in their system, forcing me to maintain two independent copies of the same thing.

They do say that they only cache things for a relatively short period of time, so I don't think you'd run into problems with that non-HTML caching rule. I think they are mostly concerned about providing caching for a site like Dropbox. In your case most of the downloads are fairly small individually so I don't think it would be a problem. If it was an issue you could easily get around it by creating dl.classicdosgames.com, putting all the downloads there, and setting that subdomain so that it isn't cached. I did a similar thing with static.shikadi.net when I was using shared hosting as it meant those resources weren't shared with the main site.

However it does mean that 1) most of the requests would get passed through to the server anyway, unless the 4000 daily visitors are all looking at the same pages. It would probably help with the spikes, but if those visitors did a search (for example) after arriving, that would almost certainly hit the server anyway due to the dynamic nature of the search request, unless they all searched for the same thing. But then if they did, the search results page wouldn't be cached unless you made sure the search was using GET instead of POST, you set the expiry headers correctly, and generally did a lot of that rewrite you were talking about having to do anyway.

So upon reflection, CloudFare might help a little, but I don't think it would solve the issue quite as easily as I originally thought.

I am guessing however, that the problems are less to do with bandwidth and more to do with server CPU load from running e.g. PHP scripts and database queries?

[DOSGuy]

Exactly. I have unlimited bandwidth, but I'm only allowed to use so many CPU seconds before it's consider abusive. Too many people running too many database queries and PHP scripts pushes me over that limit. Maybe the non-Keen part of this thread could be its own thread?

[MrFlibble]

Maybe the non-Keen part of this thread could be its own thread?

Done.

I just used Russia as an example. It's sometimes other countries, but there have been a few occasions when a huge number of people came from a Russian website, like on November 2 when 8500 people came from http://games.mail.ru/pc/articles/feat/5 ... _brauzere/. On September 10 and 11 when got double nailed when 17 000 people came from http://pikabu.ru/story/20_legendarnyikh ... re_2647815, and then 14 000 people came when vk.com picked up the story. We normally get about 4000 visitors a day, but it was 26 000 on September 10, 18 000 on September 11, 8000 on September 12 and 5500 on September 13 before things started to return to normal.

If you had anything to do with that, MrFlibble, keep up the good work! I mean, I don't know what else to say. We want people to come to the site, and I can't really ask them to space themselves out. These occasional spikes are a fact of life for any webmaster, and a good web host understands that.

I certainly don't post anything either at games.mail.ru or at pikabu.ru. The only Russian forum I sometimes post links that direct to RGB Classic Games is old-games.ru.