What should have been done
HoIn the case of previous mentioned hack it was clear that a couple of things should have been done.
Read more for:
What the admins should have done
A dom0 breach (THEORETICALLY!)
But what if this happens? What should my hoster have done and be doing now?
How do I check if the problem still exists?
What the admins should have done
- accesslists on the iscsi targets
- a firewall on the iscsi machine
- a firewall between the ‘complete setup’ and ‘the internet’
- passwords on the iscsi-target
- passwords on the iscsi-initiator
- no public IP on the storage (iscsi) server
- No listening for public IP’s on the iscsi server
- Enable logging. This way you could tell what could have been happened.
Yes this is a big list, and this list is big and seems to be double. That’s true.
But: If you forget 1 thing from the big list above here your still not hacked and you wouldn’t expose all your clients data to the whole world.
What the client could have done
If you don’t trust your admins (and there are good reasons in the world to not trust anyone else with sensitive data because if you share a network the admin can also be a bad guy with arp spoofing or so) you could:
- Use encrypted partitions
Probably not that usefull for /, but very usefull for any sensitive data. Still meaning that you have to reinstall your OS but you could still trust your data. Very usefull if you put a lot of effort in keeping the code running on your virtual machine secure. This definitly won’t protect against a breach of the dom0 (the VM hoster) but your hoster could follow the above list for dom0 too.
There are always smallers thing you can do in this world. For example ssh-agent forwarding. That way you don’t have to have any ssh private key (except the host key, the one I sha1′t) on the VM/VPS.
What a dom0 breach?
Don’t be that scared. dom0 is NOT breached as fas as I know and I intent to keep it that way!
But if it’s breached you’re complete screwed and encryption can’t save you anymore if the encryption is decryptable on a running VM (a mounted encrypted for example).
The more fun (no this is not a breach but just an example of the first list completely being ignored, BTW I got this information from 1 of the admins, they do really know about it.):
SNMPv2-MIB::sysDescr.0 = STRING: Linux dyon.soleus.nu 2.6.26-1-xen-686 #1 SMP Sat Jan 10 22:52:47 UTC 2009 i686
SNMPv2-MIB::sysDescr.0 = STRING: Linux geon.soleus.nu 2.6.26-1-xen-amd64 #1 SMP Mon Dec 15 20:07:26 UTC 2008 x86_64
SNMPv2-MIB::sysDescr.0 = STRING: Linux tachyon 2.6.18.8-xen #2 SMP Sun Aug 17 23:39:56 CEST 2008 i686
SNMPv2-MIB::sysDescr.0 = STRING: Linux axion.soleus.nu 2.6.18.8-xen #2 SMP Sun Aug 17 23:39:56 CEST 2008 i686
But what if this happens? What should my hoster have done and be doing now?
- Investigate the problem (done)
- Fix the problem (done)
- Acknowledge the impact of the problem (done, but not all public)
- Acknowledge the risk of the problem (done)
- Inform the clients (done)
- Help clients by giving them a new VM (not done?)
- Help clients with setting up encrypted partitions
- Help clients by giving them their old data read-only,noexec,nosuid,nodev in their new VM
- Help clients with checking their data. For example checksums from files from old backups (backups from before the problem started)
- Give clients time to convert (a month or 2)
How do I check if the problem still exists?
Telnet to the private IP of the storage machine on port 3260. If you get a connection you can read all the data. Since I’m not a member I can’t log in to the private lan (damn, it looks like security) and check it.
I really do hope this helps you all a little bit and please try to improve security. The more choices I have when I want to host something the happier I am. Really.
To acknowledge some people:
- Cinder for the SNMP hint
- Hepp for zoloos and thanks for all the fish
- Jnieuwen for helping me with some impact details
- burne for some advice on how to handle
- Zoloos for all the fish