Around 2pm GMT on Wednesday 13th I began to get emails suggesting something abnormal was happening with the site. This was unusual because I had not been logged into the site or had been working on it for a day or so. I was at work and can’t readily get into the site so I waited until the evening.
On logging into the server I found all the site files were missing and a message had been uploaded basically telling me someone has access to the server that should not. I immediately changed all passwords including the database users which run on a different server. I removed the message and left one saying the site was down and I was looking into it. I began the process of restoring the previous night's backup.
On Thursday morning my message had been changed to “JANGAN GANNGU !!!". Indonesian for “don’t bother”. Face palm moment: while I had changed all passwords, I had not checked who was logged in. I immediately rebooted the server which would kill any other sessions running.
I had a look at who had logged in. There were two logins (167.99.4.184 United States 4 Dec-13 04:59:51 and 125.163.1.16 Indonesia 2 Dec-13 06:06:06) which were not me. Someone, possibly two people, had logged into the server. I could trust nothing.
I also found that aquaticrepublic.com, aquaticrepublicnetwork.com, clearriverpartnership.co.uk, dignall.com and zebrapleco.com had all suffered the same fate. Not good. I took all those sites out of service and began thinking about what to do. Thursday was a long day at work. Thursday night I started the process of checking everything. All files; everything.
Planet alone has around 93,000 files across about 53GB of disk. OK, a lot of those are standard forum files, but there’s still a lot of code! And I don’t know the attack vector.
Planet is backed up daily online and monthly offline. That’s all the files and the database too.
Friday I managed to get a good copy of the forum by checking every file against a clean copy from the phpBB site, I checked that all the database looked ok. I also checked all files against my local copy and against the most recent backup. By Saturday morning I had a clean version of the site. It took hours to upload everything.
On Sunday I couldn't get the forum to work. The forum software (phpBB) runs the forum but I also use it for all user authentication; it's well-maintained and secure but it's complicated and tightly integrated with the custom ARN/PlanetCatfish code. It's also modern - bear in mind I learned all this stuff 25 years ago. So I am fairly current but a lot of techniques have changed. So, I learned a few new tricks and set about fixing it all up again.
How did they get in? Possibly a brute force attack, possibly a file with the password in it was inadvertently copied somewhere. I don't know.
Did they steal anything? I don't know, it's possible they could have downloaded the site, and the database but there is no evidence of this and the database logs don't appear to show any activity. In case you're worried about your password, we don't store it. As per best practice, we only store a
hash of it.
Today is Monday. I've finally got the site up and running. Due to when the last online backup was taken, images uploaded after around 2023-12-05 1100 are not available and need to be reloaded.
I think this only affects @casscats, and some additions I made to the cat-elog (and in some cases wrong images will be against the wrong contributor). Going to make a cup of tea and tackle that.
Cheers,
Jools
