I think they have a backup...
You bought One's and Zeros -- Don't Let it Become a Big Zero
Your entire investment is electronic when it's on the web. If you pay $30,000 for a spiffy, cool, fantastic web based program, you don't want to lose it just because someone accidentally wiped a hard drive.
Getting a copy of the site on a CD usually doesn't do you a lot of good. Don't forget, web applications aren't static. Much of what you see on a day to day basis comes out of a database. Beyond that, much of what makes your website work depends on other programs that may or may not be available when your web developer disappears off the face of the earth.
Your best bet is to have a live backup that synchronizes a couple times a day to a server you can check up on. It's even better if you can have that backup server sit in your office so that, in an absolute worst case scenario, you can pick up the box, go to a new ISP and be up and running again.
Get the passwords
I don't care if you have no idea what an SSH shell is, or an FTP server, for that matter, make sure you get all the passwords for all the services you're paying for. If something goes horribly wrong with your developer, you can hand the cryptic list to another developer and get out of trouble.
Remember: letting your web developer control everything means your web developer controls YOU as well.
Tad Benson: Re: I think they have a backup...
Might be a good way for web firms to market themselves....post a customer bill of rights which states they have access to any and all of their content, data, passwords, etc. even if the relationship goes sour. Or a good web business....the online web site escrow account. You, the third party, keeps all of the web site access info and does an auto-back-up of the site a few times a day.