Home of the BrewNosers

Last night I could not connect to the site. It has happed befor as well.

Sandy

I couldn't get on the site last night...it works now,

Rob and drunk and tripped over the power cord

Got* not and. Damn phone

Yeah couldn't connect last night at all. I put in a support request asking WTF was going on. No response. I rebooted the virtual machine this morning and it came back up. I've seen network blips before but nothing like this.

Anything in the server logs that might indicate why?

jeffsmith wrote:Anything in the server logs that might indicate why?

You don't figure that would have been the first thing he checked?

I have had one such outage on my CloudAtCost server and it was a week or two before this site moved over. It suddenly stopped responding one night, I tried rebooting and that appeared to hang, so I submitted a support ticket and it was back the next morning. It's a toy for me, so I didn't care that much. I will mention that since that happened I haven't even had a blip nor have I experienced nearly as much lag and packet loss. I will also say that there wasn't a thing in the logs afterwards; from inside the VM it looked pretty much like a sudden loss of power.

Rob, has the package been changed at all in the last two weeks? For instance did you upgrade from Dev2 to Dev3 or switch from monthly payments to the one-time fee? They recently instituted an automatic 10-day shut-off on all (or at least all Dev-level) servers. This was apparently to combat the scenario where someone signs up for a cheapo Dev box to do something, forgets about it, it gets hacked and turned into a botnet node. They claim this was happening a lot and contributing to all sorts of issues (network latency, disk latency, etc.). Anyway, I took advantage of their 50% off sale the other day to upgrade mine from a Dev2 to a Dev3 and the upgrade process re-engaged the 10-day auto shut-off and I had to go in and turn that off.

I've experienced nothing but increased performance, especially with Tapatalk, since the site move (which I should also say was impressively smooth, BTW). Hopefully we don't see this happen again or support can provide a reasonable explanation for why it happened.

mumblecrunch wrote:

jeffsmith wrote:Anything in the server logs that might indicate why?

You don't figure that would have been the first thing he checked?

Being a systems admin myself that question was curiosity, not ignorance.

Hah actually guys I didn't bother with the logs, I doubt they'd be helpful but I should just for due dilligence.

Aaron, I disabled the shot off thing when they enabled it. Also the dashboard said it was running. I did go from monthly to one-time earlier in the month.

At any rate, for what it costs us I don't expect a perfect host, nor is this a mission critical application. I do semi regular backups in case things go sideways. There's a few items on my todo list such as actually scheduling the backups and monitoring the system.

Rob,
Let me know if I can help. Maybe a few of us could get a support structure in place so it's not just on you.

RubberToe wrote:Hah actually guys I didn't bother with the logs, I doubt they'd be helpful but I should just for due dilligence.

Aaron, I disabled the shot off thing when they enabled it. Also the dashboard said it was running. I did go from monthly to one-time earlier in the month.

At any rate, for what it costs us I don't expect a perfect host, nor is this a mission critical application. I do semi regular backups in case things go sideways. There's a few items on my todo list such as actually scheduling the backups and monitoring the system.

I've got a pretty solid shell script that I use for doing daily backups from my servers to an Amazon S3 account. I'll send it along to you when I'm back at my desk for work tomorrow.

Already scripted db dumps and rsync to another host.

Just some minor tweaks and crontab entry.

Gotcha, perfect.

jeffsmith wrote:

mumblecrunch wrote:

jeffsmith wrote:Anything in the server logs that might indicate why?

You don't figure that would have been the first thing he checked?

Being a systems admin myself that question was curiosity, not ignorance.

And if I hadn't had pretty much exactly the same problem once before with this host it's the first question I would have asked too, hence the wink. Also, he didn't check the logs after all, so...uh...yeah

RubberToe wrote:Aaron, I disabled the shot off thing when they enabled it. Also the dashboard said it was running. I did go from monthly to one-time earlier in the month.

Just making sure you were aware they re-enable the 10-day thing when you change your package (not sure about switching from monthly to one-time because theoretically that's just billing and there's no reprovisioning there). Also that's exactly what happened to me: dashboard said "up," all real-life experience said, "not up."

RubberToe wrote:At any rate, for what it costs us I don't expect a perfect host, nor is this a mission critical application. I do semi regular backups in case things go sideways. There's a few items on my todo list such as actually scheduling the backups and monitoring the system.

For what it's worth I have a couple places where I could run a Nagios (or whatever) agent if you wanted redundant monitoring, but knowing you have another CaC box yourself and having seen the rack in your basement I suspect you have it covered It'd be cool if the monitoring could, say, send a tweet to the Brewnosers Twitter in the case of an outage like last night so that folks have a way of knowing it's not just them and the site is actually having some issues.

I use Uptime Robot for simple monitoring. It's free and gives you the ability to send a text or email to multiple people in the event of an outage.

It's definitely not a perfect solution (will sometimes give false positives) but works pretty well for the most part.