As I mentioned in the previous two posts, I had a hard time during the last 2 weeks dealing with different hacks on my hosting server... From the early 2000s, and the beginning of the Google Adsense era, I was "lured" into the hosting business by friends who needed powerful solutions to run their websites... In those days, I was running a forum about earning online opportunities, which was one of the most popular in my country... There was a small but strong community of webmasters who were running some heavy-traffic-load websites, and they needed a strong machine to be able to serve millions of visits per day...
One of them and I have decided to rent a powerful machine, run multiple websites on it, and split expenses with others... It was a good solution as people didn't have downtimes on their peak traffic, which was very common to have with "cheap hosting", where you were promised "unlimited bandwidth, space, and everything"! After the first machine, I rented the second, then the third, and at the peak, I was running around 10 machines!
Long story short, as time was passing, Google became greedy and instead of sharing 60% of revenue (yes, you see that right) like it was at the beginning, their "cut" became bigger and bigger... Running a website wasn't that profitable, and many quit... In the end, after 20-25 years, only a handful of people were running websites... As all of them were old-time friends, I was delaying my decision of shutting down my last hosting server... There was almost 0 profit, and it was more about doing a favor to others...
It was time to put a lock on it... Unfortunately (or luckily), a series of events "helped" me out to make my final decision... To shut down my hosting "business" (or better said, charity... π)... To remind myself WHY I made this decision, and maybe to help out others who want to do something similar, I decided to create this post with all the events that happened in the last 2 weeks or so... Let's go!
I like to call this day Day 0, as late in the evening, I got the first mail about suspicious activity around hosting...
The first email was from the Google Search Console Team, which notified me that another user has been added as an owner of one of my old websites... (btw. I have a couple of my own websites on the machine... most of them are just staying there without any updates for years... Btw. ALWAYS KEEP YOUR WEBSITES UPDATED... Not just with new posts, articles, but keep your scripts UPDATED! For example, WordPress, Joomla, etc, not updated scripts are magnets for hackers!)
Suspicious file alert
Around the same time, I got another email about the suspicious file running on the machine... I checked the details about it, deleted the file, and thought that the issue was solved... How naive...
This was Sunday... Early in the morning, I like to drive my wife to the seaside town, where she plays piano in a hotel... While she is playing, I have around 2 hours to enjoy the walk, sip a cup of coffee, and check out my online stuff... When I was in the furthest place in my walk, I got a message from a guy who hosts a very popular fishing forum on my server... "My site doesn't work. Can you check it out?"
I checked the machine from my mobile phone and saw that it is under a heavy DDoS attack, and it's almost not responsive at all! You can't do many things when that happens if you don't have an anti-DDoS service on your network...
The first idea that came to my mind is to try to set up CloudFlare on that website... To my surprise, it did help out with migrating spam traffic, and the machine was working again... Another thing patched/fixed!
Unfortunatelly, that wasn't all for Sunday! During the day, I got an email saying that someone was using phishing scripts on my machine!
It looks like another (completely different) website was hacked, too! It was redirecting to a phishing page... I deleted the script, restored a backup of the website, and told my client to change the cPanel password...
This was Monday... On that day, I noticed that more of the old Joomla and WordPress websites have suspicious code inside specific files... Hacker was adding the code which would show (or redirect) a specific page instead of the real website if the user landed on a homepage through some of the search engines... So, if you check out the website by entering the address directly in the browser, everything would look normal...
After checking most websites, I have deleted suspicious code from all... Also, I have decided to download ALL clients' backups to my local machine... Just in case... After a long day, I wrote my Workerbee Pool post, thinking that the worst had passed...
The message that you can see was occurring more and more on websites... As no DNS data was changed on domains, it meant that something is wrong with the DNS script in WHM/cPanel... I did poke around that, and in one moment, I almost made the complete machine unreachable! π Anyway, that problem disappeared after a while, and I still have no idea how and why... Truth to be told, sometimes the DNS issues can be outside of our reach, on our ISP, or some worldwide DNS router...
As of positive results from CloudFlare, I have set up more websites to go through it...
On this day, I received the 2nd "abuse email" from my service provider about phishing from my machine... This time, the hacker has created additional subdomains which has malicious code that would redirect them to some phishing website...
Also, I received the "second wave" of adding new owner emails from the Google Search Console Team!
It was obvious that hacks didn't stop, and I made my final decision... I will close down my hosting business, once and for ever! I have informed my clients about that in a short email, sending them links to their cPanel backups for downloading...
On this Thursday, I couldn't sleep, so I checked my mobile phone around 5 AM to see a bunch of warning emails about a suspicious number of emails sent in a very short time... Around 5-6K emails were sent in an hour, for around 4-5 hours...
Again, on one completely different account, a hacker has created 3 different email accounts and spammed people for hours... Luckily, I have reacted relatively fast, deleted those accounts, and prevented bigger damage and another abuse email...
On this day, something interesting happened which wasn't connected to this hack, but I would like to share with you... I got a personal email that looked like this:
You could say there is nothing special about this... Just a usual phishing email that tries to steal your data... Well, it is true, but...
The coincidence (or not) is that I do use a service from OVHCloud, and that is my email address used on their website! And to make it even weirder, my payment due was a few days before! I did pay that bill, but in one moment, I had doubts about it... Being suspicious, I didn't click on the link, but I went to my bookmark, loaded up my OVH member panel, and checked if everything was paid!
Was this just a lucky guess, or did they sell my email address from my VPS provider? Watch out for these types of things... Do not click on links inside emails, but go to your links from your browser/bookmarks...
Most of the clients moved their websites to other places!
My water heater died... π Completely irrelevant, but spending some sleepless nights, combining a lot of stress, makes you smell awful... Cold water, here I come...
At this stage, I was helping others to move their websites to other hosting providers... As some of them aren't tech savvy, I was setting up their accounts, copying files, importing databases, changing config files, setting up FTP access, SSL certificates, etc.
Ordered new water heater... and ironically, got the coldest shower ever!
ROOT LOGIN from another IP other than mine!
This is probably the biggest fear of any server admin... To see that someone else is logging on to YOUR machine with your ROOT password, which can lead to a complete disaster... As a ROOT user, a hacker can do anything he wants with it, delete everything, lock you OUT of your machine, etc.
As you can see, that happened at 23:41 and in 23:46 I checked my email... When I saw it, I immediately changed the root pass, rebooted the machine to log out the intruder, set up the 2FA login, and blocked the IP of the attacker...!
Also, I have deleted all hosting data from clients who moved out, to prevent DATA leaks...
In less than an hour after the root change, the machine got hit by another DDoS attack...
This time, I was notified about it from my provider, and they did the migration of traffic... It looks like the hacker got pissed off...
Changed the water heater... Took a hot water shower...
At the moment, it's time to reflect on all these events... Was there a way that I could prevent these events?
As I said previously, some of these things can happen to anyone who runs a website... When you don't update your scripts, the probabilities are much higher... If you use some lame, simple passwords that can contribute a lot to more hacks...
But, I don't want to avoid personal responsibilities... It was partly (or maybe mostly) my fault too... Firstly, I was delaying my decision to quit the hosting business for years, mostly because my friends didn't have other solutions, or didn't want to bother to find them out... Everything was working, they were happy, I was happy...
Secondly, there was a moment when cPanel announced that they will not support CentOS (Linux distribution) anymore for new updates... Because of that, my WHN/cPanel was outdated for more than a year, and that's a big disadvantage in fighting against hackers who are always up-to-date!
Thirdly, the company that was running the main firewall software announced that they will close their business earlier this year... They also stopped updating it for months, which was another security issue...
All in all, after so many stressful days, with a very few hours of sleep, I made a decision that I had to take a few years ago... Was this an unlucky event or a lucky event? Time will tell, but I wouldn't mind if I could skip all the stress that was taken... On a positive note, having this burden off my back will help me move my focus to more important things!
I would like to thank everyone who was checking on me through DMs and for sending support messages through the comment section here on HIVE! I appreciate it a lot! π As someone said in the comments, the most important thing is to avoid health issues, and luckily, this wasn't something like that... Hopefully, the stress levels are coming back to normal...
Thank you for your time.
--ph--
Don't forget to follow, reblog, and browse my Hivepage to stay connected with all the great stuff!
Huh, Vukovi sa Dunava π.
This is horrible what happens to you. But, that's expected if you're an administrator for multiple web sites. In early days when I have and managed my VPS hosting and something similar happen the provider just block my VPS until I find the bad script... Terrible...
Those abuse warning mails can be tricky... I had a similar experience in the past where I had to solve the issue in just a couple of hours, or the entire machine would go down... Later on, when you do business for a longer time, you get a bit more respect and confidence from others that you will solve the issue asap...
That's why I still have chills when I get some abuse warning email... π
Knock knock! Who is this? This sounds soooo recognizable! Same thing... Created a website, put adsense on it, earned well, created another website about how to earn with adsense, and put adsense on it, things started taking off very well, bigger hosting required, rented server, friends started... blah blah blah
100% same story, but I had the hackaton weekend about 2yrs ago :-) Then I made the same decision as you have. All: out!
Glad you to hear you got rid of the burden, and the cold water ;-)
!WEIRD
!DOOK
!MMB
!INEEDSLEEP
!BBH
!LOLZ
lolztoken.com
Hummus-cide
Credit: reddit
@ph1102, I sent you an $LOLZ on behalf of borniet
(6/10)
Delegate Hive Tokens to Farm $LOLZ and earn 110% Rewards. Learn more.
It is a bit comforting to hear that others had similar things happen in the past... I suppose I could continue doing it, improve my skills, but it just wasn't worth of money and stress...
One burden less, hot water ON! I was thinking of changing my nickname to "IT Plumber"! π
Brother, that reads like a very well-written cybercrime movie. Glad you even took something positive out of this!
When it was in for a couple of days, I thought about the exactly the same thing... It felt like being inside the movie... π΅
Man that sucks π. At least we have the water heater going!
As winter is coming, it would be hard without hot water... π
Oh man what a hassle, you should be getting well paid for this service you did... Indeed not worth it as of now to host and handle this for no profit
I suppose we all have certain things that we are doing for others, for free... But, I agree, this was maybe too much and not worth stress and money...
The hacker was actually very determined ππππ
That's if it's just one person though...
So sorry about all that
At this point, I'm not sure if there were more than 1 hacker, but the last one was determined, for sure... but probably the most experienced, too!
Man..What a story.
It will be alright, in this case it sorts of helped you make certain decisions, and as for what I can tell there was not some serious damage done to the other people. Lots of hassle but also a great experience and a reminder.
Hugs.
At this stage, it looks like youβve stayed in business and started working even harder. π I just hope you wonβt have to keep supporting all your former clients and friends for too long, even without your server now...
Yeah, the situation is, to put it mildly, pretty intense π₯. But I think it was something you really needed. Otherwise, youβd be carrying that weight for several more years. Itβs almost impossible to refuse helping those close to you by your own choice...
In any case, congratulations on the new shower!
Ouch... I am not trained on most of what you described to know what to do. Glad you knew all the ins and outs and reacted quickly so that it did not get worse. This hacker appears to be good/sly at what they do.
Ouch that sounds rough. It seems like that hacker wasn't very happy and wanted to make your life miserable. I guess all those security factors made it easy for him to target you, but I think you did a good job of responding, and helping others migrate to a different provider.
You really had a horrible period trying to fight off this sneaky intruder! At least now it's really over, if you shut down your server.
Well, the machine will be shut down in a week or so, but 99% of clients have been transferred to other places... Even if something very wrong happens, it will not harm others, which was my main concern...