Was the Cellar hit again?
 

I had trouble getting on about noon today and noticed that there were no new posts since 1 am. At the same time, I could not navigate and I got a message that the Cellar was refusing connection.

Was it maintenance, an outage, or was the Cellar hit again?

I had the same thing, about the same time. Got the syntax error thingie. All is well now.

Just be thankful I'm here now. :lol2:

Yes, there was network congestion of the SOD variety between 4:30 am and about noon, and it's not clear whether the ISP or I have cleared it up 100% yet.

I was glib about that, on purpose. We are hit with some sort of DOS or similar attack on Apache. We continue to be hit by it. I expect to take the system offline for a few hours at some point to move it - a move many months in planning, long story.

If ever you can't reach the Cellar, just wait, do not panic, it will return.

Don't panic? Easier said than done.

I'd be less concerned about not being able to log onto any of the US Government sites than I am when The Cellar is unavailable. The Cellar is much more important to my day to day life.

And remember your towel.http://www.cellar.org/images/newsmilies/thumb.gif

Very nice.

Looks like a tag line to me..

OK! The system that the Cellar lives on should now be considered entirely compromised and rooted by a cracker far too clever to let me know exactly how he operates.

I *think* this is my fault, because I have not run the Cellar machine correctly wrt software levels and some of the software it runs is well out of date.

The system is continually cracked and used, I *think*, to generate DOS attacks. When it does this it completely saturates my connection and makes the system unreachable.

I have been planning for about 6 months now to move it to another, secure, co-located system but one last technical glitch does not permit me to move it immediately. I am waiting on another party to finish what they need to do in order to make it all work.

In the meantime, I have an alarm set to check the Cellar once every minute (!) and sound a loud alarm when it is unreachable. Last night it went off at 2am, 5am and 8am. Approx. The system was therefore "fixed" and DOS attack shut off before anyone could realize it.

But just now I had to drive home from the office where I'm spending 30 hrs a week, and that just cain't happen alla time, so the site may be down a few hours this afternoon if they strike again.

After today I should be home most of the time and able to respond to the alarm

And all I need is for this other person to come through with what they have to do...

The Cellar itself is thoroughly and securely backed up in two different ways, so your actual messages will not be lost, no matter what occurs.

The new system is on automatic update and so this sort of UT-based administration error cannot occur.

Does that mean you had to get up at 2AM and 5AM to fix things? Or was it automatic?

Either way, thanks for everything you do to keep things humming along. I hope this hobby continues to be fun for you, and doesn't turn into drudgery.

Worst-case - another system in the house is set to check the status, and because it's in another room, I have it set up to ring a VERY LOUD AND ANNOYING ALARM SOUND. This wakes me up, and I trundle downstairs and shut off the process that they're running, check to make sure everything's back on the air, and then go back to sleep.

Other than the Cellar there is precious little on this system, and the responsible thing to do would be to shut it down and fix it permanently, but because -- any hour now! -- it's all supposed to move off this system anyway, this is the path of least resistance.

I do that too, but the VERY LOUD AND ANNOYING SOUND is my cold 3 year old screaming for me from the other room so I'll come in and fix his blankets.

You are a parent too. :)

:lol: Me too! (cold 3yr old). I am trying to design some sheet suspenders for the bed that won't strangle the kid in the process, you can be on the test panel :)

I gotta remember to donate to the paypal link... Thanks UT, for all you do to keep this place up and running.

How did you get him to do that??? Ours just comes on in to our bedroom & climbs in.

Well, for me, admitting all this is like staceyv admitting she yelled at her puppy. Waking up in the middle of the night is the penance. I was dumb, I left the vulnerable machine up, I am punished.

Deadbolt on the bedroom door. Also, sleep naked or in just underwear, it gives you an added incentive not to give in and allow them in with you.

!!!

Ten attacks today. So far.

An lsof found that their open process was constantly connecting to 210.170.60.2. That address is now blocked at the firewall. I think.

It's in Japan. But I dunno if that was the target or the source, or whether it's just a bridge to somewhere else.

Can you notify the IP owner?

There's no reverse DNS on the address. But whois lookup says it belongs to

TELE PLANNING INTERNATIONAL INC.

in Japan. There's technical contact information:

http://whois.nic.ad.jp/cgi-bin/whois_gw?key=MJ018JP

Look, it's trying right now. netstat -an includes the lines:

tcp 0 1 207.245.113.66:43901 210.170.60.2:3982 SYN_SENT
tcp 0 1 207.245.113.66:43905 210.170.60.2:3982 SYN_SENT

I don't think this firewall works.

Naw, it just tried again. Damn.

lsof:

# lsof -p 6683
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
exe 6683 nobody cwd DIR 3,3 4096 2 /
exe 6683 nobody rtd DIR 3,3 4096 2 /
exe 6683 nobody txt REG 3,6 17828 30 /tmp/upxBQHBVKFAGQ0 (deleted)
exe 6683 nobody mem REG 3,3 90168 2371760 /lib/ld-2.3.2.so
exe 6683 nobody mem REG 3,3 1452984 49557 /lib/i686/libc-2.3.2.so
exe 6683 nobody 0r CHR 1,3 1701592 /dev/null
exe 6683 nobody 1r CHR 1,3 1701592 /dev/null
exe 6683 nobody 2r CHR 1,3 1701592 /dev/null
exe 6683 nobody 3u IPv4 511195499 TCP topaz:43909->210.170.60.2:3982 (SYN_SENT)

Gee whiz I wish I had some idea about this stuff. Keep fighting the good fight Bro.

I'd send an e-mail to the tech contact. If they've been infected, you would be doing them a favor by informing them.

Yeah, but I don't know if they're the target or the source.

I'm not sure the firewall was up, or maybe it was and it was preventing that attack. I do know the firewall blocked my DNS services for a bit. Damn I am supposed to know what I'm doing on this stuff.

I found it. In the crontab of the userid that runs the web server, was an entry that created a binary that would start its work, delete itself, and change its name to the same process name of the web server.

Before they loaded this, they trojaned every single utility used to do network administration. With VERY good trojans, exactly the same size as the originals. I only found that because I have safe copies of these utilities everywhere, along with the safe copies of the checksum programs that let you detect what's changed.

Things are better. Not great but better. The people who were supposed to do the network configuration at our new location, failed to do so again and so we can't move til Monday at the soonest.

I figured out that it was a cron entry because the parent process id of the DOSsing daemon was 0.

Good job UT !!!!!!

Trace down their address , and me and Louie will go have a TALK wit' em !

Aw bullshit. You're embarassed because you want to live some kind of normal life instead of holing up in a dark room like a L0pht member? Screw that, Sir. You are providing us, at considerable cost in time and money, with a great service. Greater than you can imagine at certain times.
We certainly have no right to complain one bit.
You, Sir, are our hero. :king:

I concur with Bruce. I really love the Cellar, but it's not worth screwing up your second relationship. When I was on call, I always felt guilty when the phone would wake up my wife. In your case this is a hobby, not your livelihood, so there is not even that justification.

BTW, as soon as the bills clear and I verify the status of the checking account I have tied to Paypal, I will drop something into the Cellar Defense Fund. I don't remember if my last donation was 2005 or 2004, so I figure that I'm due.

It won't be a lot, but at least enough for caffeine pills and aspirin.

Since these guys are technically cyber-terrorists, I would assume that under the current political climate, you have the authority to hire contractors to deal with them if you identify them.

Maybe the guys from BlackWater would like a warmup mission before going to Iraq. We could have a fundraiser.http://www.cellar.org/images/smilies/cool.gif

Y'know what it is - it's like an electrician who goes home and doesn't wire his own house to code, because he figures he pretty much knows what he's doing, and that, well that can be fixed later, as long as we don't plug something heavy into it, that's what it is.

Better but not fixed: down all Xmas day from about 11:30 am to 8:45 pm. Back home from mom's so I can fix it now.

Thanks dude !!!! I was getting worried !!

I hope you had a merry Xmas !!!

This have anything to do with that yogurt guy not liking being ridiculed?

Good job, UT :thumb: It was rather unnerving to wake up this morning and discover the Grinch had struck again.

And again, earlier this evening? I think we need to send homeland security after those Japs! :headshake

Yah, two hours down.

Time to send out the Yakuza.

Four hours tonight, and I swear they hit ten minutes after I left the goddamn house.

Maybe it's your neighbors. ;)

I had everything prepared for the move that will solve this problem, and stayed awake until way late last night to finish it. It was 5 am when I got to the last step. It takes an hour, and there's nothing else to do while it runs. And so I set my alarm for 6 and went to sleep.

Woke up at 10 to find I set the alarm for 6 PM. :lol:

So the move is not done yet! Soon. Soon.

yeah, except I can wire a house, (OK I'm a little weak on my three-way switches) but when I look at that code stuff you posted it might as well be written in xhosa.

I was totally impressed that you busted mage instantly.

So your awesomeness is inversely proportional to my knowledge of computers.

Right now you are a small galaxy.

btw, what is an appropriate amount for the tip jar?


Also, a couple of questions to show the extent of my knowledge of computers/internet:
1) when these attacks happen is that the same thing as being "hacked"
2) are PMs that we have saved readable by the "attackers"?
3) Is there any financial motivation for the attacks?

This is what happens when you let kids learn about computers "on the street"

I don't mean to tell y'all about how difficult this is just to get donations... I have to say

Otherwise, donations now have us on a pretty good server with a decent amount of bandwidth, and after the move we can think about expanding and trying to get more people and such, and maybe doing some projects and such in the new year. Things would be much more bleak without donations, so they do make a difference in the long run.

Isn't it nice not to have to worry about what to do with your spare time? :thankyou:

No xoB, :thankyou: let us start the thank-fest! :P

Appropriate amount: that's entirely up to you... I always think of this sort of thing as similar to public broadcasting as far as that sort of thing goes.

Despite my nice house, which is basically my only investment, I am basically living on half a salary while I play the entrepreneur game. Donations have seen that I have the luxury of never giving this up, always being able to keep it going and now working to make it better, etc. It is a hobby on my behalf but also a labor of love and of need. I love you folks, I need you folks.

UT loves what he is doing, but it does take a block of time out of every day to seek and find and post the IOTD's and to monitor the threads, etc, even when nothing is going wrong. As he says, it is a labor of love, but as an artist myself who would do my work without public support if I had to (and have done in the past), I understand the motivation. But the public also knows this and knows that they can get an artists work cheaper or even for trade because the artist is going to keep working anyway. For some of you, perhaps many of you, that is what is happening here.

Like public broadcasting, I am pushing the guilt button. But you all do need to think about what kind of benefit you get from having the Cellar available, and borrowing another argument used by public radio, waht would you do if it weren't here? UT has never wanted to make it a commercial site with ads, etc., paying for it.

Mother will shut up now.

Mother knows best. :queen:

I send UT the amount that I pay monthly for OUR server, because I know what it costs to maintain an intermediate-size website on a server without advertisements or cookies and The Cellar probably takes up twice the space Thalia's World does. Our server administrator is always whining that I take advantage of him because I was one of his first clients and got special consideration, but now I use more bandwidth than most of his commercial clients. I tell him he should feel honored to host the second-most-viewed Thalia site in the world, but he says he would rather have the money :right: