Sunday, 20 September 2009

Life, the Universe and Virtual Server Pwnage

A few weeks ago I urgently needed a server hosted in Asia to check whether a connectivity issue (using RSA secure tokens) from Belgium to one of my customers in Asia was caused by the severe internet latency I was experiencing.

So I did a quick internet search and ultimately decided on the Singaporean company Sparkstation. After all I do a lot of business in Singapore and the pricing seemed just right. A dedicated server was not necessary so I chose a Virtual Private Server (VPS).

I liked to the fact that I did not have to sign up for a long period (a year or more) of hosting but could go for only a month. I got unlimited data transfer, 20 GB disk, 512 MB memory and a 64 bit Windows server for less than € 30. And they even have cheaper plans! Connectivity to the outside world (and especially Asia) was also needed and this is also more than OK since they are located in the Qala datacenter in Singapore.

Using a hosting service provider (OK marketing guys, I will call this ‘cloud computing’ in the future) might give some security headaches. And I was going to be hit very hard pretty soon …

First of all, you need to trust the service provider and their staff and their abilities to provide a secure environment. This is however exactly the same as for your own internal staff but you might have less control (I hate that word, since it has been stolen by the governance, audit and security sellers) on the desirable state.

Note to auditors: if your company or client uses a hosting provider check how administrators connect to the panels that give administrative access to the servers. You will discover that these panels give access to the servers with a simple user-id and password combination. Of course, the operating system itself will be hardened and secured and administrators will proudly show that they use strong authentication to remotely administer those servers. PCI compliance no problem, sir! Not really a lie, but the administrative panels give the same access and more with weak authentication.  

I’m not going too deep into securing the operating system itself since this blog is supposed to be about application security. Suffice to tell that once connected to my container I immediately checked if the server had all security patches. Sadly, this was not the case, and I needed to download and install more than 100 patches. This took many hours and after the last install, the server did not want to boot anymore. Apparently one of the security patches was not compatible with the monitoring software used by the hosting provider. Ploink! (sound of security consultant throwing himself out of the window).

Using the administrative panels however I could revert to the initial set-up and started the painful process of taking a back-up, installing one patch, rebooting, if successful start this process again, …

This was taken too long and I did the short connectivity test to my client. This proofed that timing issues where not the cause of the connectivity problem. After the test I signed off with the intention to take a better look (and install the remaining patches) at the server when I had more time.

A few days later I decided to do some more experiments with my virtual server and I signed on again. I got the feeling that the server was a bit sluggish, which was strange, since no services where running, at least that was what I believed…

I checked the services running and there was the oddly named ‘SQL Server’ service (this is odd, because Microsoft uses MSSQL as service name). I got even more suspicious since

  • In the explanation for this service (telling that one should never stop this service since it was essential for the good working of the server) there was an obvious spelling mistake;
  • I never asked for a SQL Server installation or so I thought…

I followed up with a quick  a ‘netstat –na’ and found out that there were hundreds of active connections to my server.

My server statistics gave the following picture:


Note the nearly 16 Gig a day traffic. Party time, success at last! But wait a minute, oops … my web site was not even active, the host name not registered, so what where these people trafficking in?

There was no possible denial and no hiding anymore: I was pwned! I was surprised by the speed of this all: within 3 days after putting a new server online it was discovered, attacked and hacked to pieces. And this server was at least partially patched and had no public services such as HTTP and FTP running.

I had no choice but to re-install from scratch and investigate what went wrong. This took me a while, but here is the list of my mistakes …

  1. At the choice of the hosting plan I checked a box (hey, it was free) to install MSDE without thinking twice. MSDE is the SQL Server Desktop Engine that should never be put on a server. Since I forgot about this choice I never ran the configuration utility, so the administrative password was blank.
  2. I did not immediately install all patches (see above). Remote Desktop was open to the world, and vulnerabilities there might have offered an attacker the possibility to sign on.
  3. The firewall was active but had some ‘security holes’. The potential mistake there was that ‘local network’ machines had some connectivity rights (meaning all other virtual machines running on the same server). Since you have no control on those machines, they should have been blocked by default.

I recommend using the Microsoft Baseline Security Analyzer which gives a lot of security advice. The server is now finally fully patched and now I can start having some fun with it (until it is pwned again by a zero-day).

Read my following blogs on installing (and securing!) blog and wiki software. Ok, you’ll have to wait a bit longer, because again, I experienced more (security) problems than initially expected.

Have fun!