When Vultr came out with their $2.50 a month VPS a while back, I jumped at the chance and put dokku on it to use it as a testing/staging ground for my pet projects. As of this morning, the only long-term tenant on it was a small Node.js app which forwarded texts to a Korean cell phone to an American one. (Long story short, I'm currently in an environment where I only have an access to a phone that can't make calls but can send texts.)
This morning, I noticed that my app wasn't responding quickly or sometimes at all to my texts. When I logged on and hit
docker ps, I saw an unfamiliar entry. (I failed to take a screenshot before I panicked and ran
docker kill, which I'm still bummed about.) When I looked at its logs, it was communicating with "xmr.crypto-pool.fr." The word "crypto" immediately raised red flags in my mind - is this a malware making my server mine some cryptocurrency for this attacker? I then looked at the CPU usage logs provided by vultr:
Yup, someone is using my server for mining.
Then, I looked at the SSH access logs (
sudo grep "Aug 4" /var/log/auth.log | less because I knew from the CPU logs above that this started happening on Auguest 4th):
which is, I guess, expected - bots try to brute-force your root password all the time. However, I was surprised to find these lines (there were more than one):
Somehow, my private key was compromised. This was a little scary to me, since I use this key-pair to do more than just push to dokku. (Thankfully, I only used it for pet projects - it wasn't associated with my Github account).
In situations like these, I was taught to "go nuclear," so to speak. I re-imaged the Vultr instance as soon as my curiosity of how this happened was satisfied. I deleted the SSH key-pair that seems to have been compromised and stopped it from being used anywhere. I plan on re-imaging even the machine which was storing the compromised key to be extra safe.
I am humbled by the dangers of being connected to the Internet and am reminded of the precautions we must take when setting up a public-facing server. These are the lessons I'm taking away from this scary (but honestly, quite fun) episode. I've found (this DigitalOcean article very educational.
- I'm always going to set
PasswordAuthentication nofrom now on. Bots are always going to try to bruteforce your root account - while I know mathematically they're incredibly unlikely to succeed, I'm not gonna give them even the satisfaction of trying. (This DO post has some good instructions)
- I haven't been setting a password for my private keys because... laziness. I really should do that from now on.
- Unrelated to this specific break-in, but I'm going to start configuring UFW on my servers, unimportant hobby instance or no.
It's a jungle out there... everyone be careful, and take care. :)