Proxmox VM and CEPH Storage
Proxmox VM and CEPH Storage
Anyone used Proxmox to host VM's and/or used CEPH storage? I am currently exploring both to use for a High Availability setup, but wondered if anyone has experience with any pointers. My main objective is to have an HA system that will stay up even when a server (node) crashes due to the expected/unexpected (they all do eventually) HD crash, etc.
I have used Proxmox for testing a few years ago and it seemed very slick and includes live migration of VMs.
CEPH i have no experience with, but seems like the future of storage.
I have used Proxmox for testing a few years ago and it seemed very slick and includes live migration of VMs.
CEPH i have no experience with, but seems like the future of storage.
Re: Proxmox VM and CEPH Storage
I will take that as a no as far as experience from anyone :p
I am setting up my CEPH cluster now...
I am setting up my CEPH cluster now...
Re: Proxmox VM and CEPH Storage
HI David,
Please share your experiences. I am leaning towards a Esxi setup , but undecided on what is the best storage form. On bare metal, xpenology has worked reasonably well.
Everyone seems to rave about ZFS , but I am not entirely convinced.
Please share your experiences. I am leaning towards a Esxi setup , but undecided on what is the best storage form. On bare metal, xpenology has worked reasonably well.
Everyone seems to rave about ZFS , but I am not entirely convinced.
Re: Proxmox VM and CEPH Storage
Learning curve for sure, but I it all set up and have been testing it for many weeks now with good results. I have Proxmox VM setup with CEPH as the storage on three separate machines. It is slower than local storage for the mysql VM running long queries (8-10 seconds instead of 4-5 seconds running an 8 page aging report...this is because the Hard Drive for the VM machine is on another machine with shared CEPH storage) but for day to day use, don't see any performance decrease from a local storage setup. The advantage in theory, is that when one machine crashes or hard drives go down, all the data is still accessible and you keep on working in your practice. Fantastic if it works in a live environment...
Re: Proxmox VM and CEPH Storage
Dear David,
That sounds really cool. May I know why you chose CEPH as your storage with so many choices available ( eg Nexenta, FreeNAS, SANS)
is the CEPH acting as your file server for your radiographic images and OpenDentImages as well?
That sounds really cool. May I know why you chose CEPH as your storage with so many choices available ( eg Nexenta, FreeNAS, SANS)
is the CEPH acting as your file server for your radiographic images and OpenDentImages as well?
Re: Proxmox VM and CEPH Storage
Yes, it hosts everything indirectly. Ceph is not a direct file store by itself. It is just a storage platform for a virtual machine disk. I have multiple VMs hosting the SAMBA file store for my images, etc. The only reason I chose it over the other network attached storage solutions is the promise of better disaster live recovery. One disk goes out, even multiple disks, you still have your live virtual machines running and served by the same data with no downtime. Is this different than a RAID setup? Yes. In my experience RAID has always been a problem when a disk goes out. Either the array fails or fails on rebuild. CEPH is different, it is not striping info, it is storing it all in clusters. At least the little I know about it, it should recover easier. The other great thing is to add capacity, you just add a disk, and include it as an object for data storage. No RAID rebuilding, etc. The other great thing is that is is spread out over multiple machines, that way when one entire computer fails, the data is still there being served.
The real test will come when it actually does go out and we see how it performs. Always backing up to other places of course. No redundant system replaces separate backups in case of real disaster and system failure.
The real test will come when it actually does go out and we see how it performs. Always backing up to other places of course. No redundant system replaces separate backups in case of real disaster and system failure.
Re: Proxmox VM and CEPH Storage
Hi David,
Would you be able to detail (maybe via PM) how to setup up CEPH as you did ? It sounds exactly like what I am trying to achieve with an esxi virtualisation , file serving and storage functions for OD and my digital radiographic images
Would you be able to detail (maybe via PM) how to setup up CEPH as you did ? It sounds exactly like what I am trying to achieve with an esxi virtualisation , file serving and storage functions for OD and my digital radiographic images
- Justin Shafer
- Posts: 596
- Joined: Sat Jul 28, 2007 7:34 pm
- Location: Fort Worth, TX.
Re: Proxmox VM and CEPH Storage
I used ProxMox once about.. 5 years ago...
Any improvement? (I bet so!)
I stuck with DRBD and Heartbeat\Corosync back then.. Ran it at home with mysql and OD.
Failover was bad ass... for mysql.
Any improvement? (I bet so!)
I stuck with DRBD and Heartbeat\Corosync back then.. Ran it at home with mysql and OD.
Failover was bad ass... for mysql.
Justin Shafer
Onsite Dental Systems
817-909-4222
justin@onsitedentalsystems.com
http://www.onsitedentalsystems.com
http://iocsnapshot.com
http://justinshafer.blogspot.com
Onsite Dental Systems
817-909-4222
justin@onsitedentalsystems.com
http://www.onsitedentalsystems.com
http://iocsnapshot.com
http://justinshafer.blogspot.com
Re: Proxmox VM and CEPH Storage
Justin,
What was your experience with the failover?
What was your experience with the failover?
Re: Proxmox VM and CEPH Storage
I have been running on the proxmox/ceph config for a month or so now. Everything is great so far!
I have it setup with three proxmox VM servers and three proxmox/Ceph servers. So as long as one computer in each of those two clusters is online, everything hums along.
I have shutdown two of the three ceph servers and everything still hummed along and then was right back to normal when I brought them back up. The only noticeable issue is a delay of 10-15 seconds (everything hangs) when it goes down and then periodically when I assume it is trying to reconnect, but everything is still functional and online!
The VM's themselves restart on a failover node in their cluster if my primary VM server fails.
Everything is fenced with APC PDU's I bought on ebay for like $50 each when they list for $700 new....they work flawlessly. (One for each cluster)
I have it setup with three proxmox VM servers and three proxmox/Ceph servers. So as long as one computer in each of those two clusters is online, everything hums along.
I have shutdown two of the three ceph servers and everything still hummed along and then was right back to normal when I brought them back up. The only noticeable issue is a delay of 10-15 seconds (everything hangs) when it goes down and then periodically when I assume it is trying to reconnect, but everything is still functional and online!
The VM's themselves restart on a failover node in their cluster if my primary VM server fails.
Everything is fenced with APC PDU's I bought on ebay for like $50 each when they list for $700 new....they work flawlessly. (One for each cluster)
Re: Proxmox VM and CEPH Storage
Hi DAvid,
I am thinking of using VMware esxi for virtualisation instead, and a ZFS file system for the storage. Simply because i am a little more familiar with them.
Do u have a domain controller for your network, or a firewall VM eg Pfsense or Sophos?
I am thinking of using VMware esxi for virtualisation instead, and a ZFS file system for the storage. Simply because i am a little more familiar with them.
Do u have a domain controller for your network, or a firewall VM eg Pfsense or Sophos?
Re: Proxmox VM and CEPH Storage
I do not have a separate VM for domain control or firewall. My firewall is router based to the outside world. Do you think it is necessary to have something else? If so, what is it's role/main function?
I have each VM serving data handling security with encryption of the filesystem and then user level Linux access control, which seems to be working well.
I have each VM serving data handling security with encryption of the filesystem and then user level Linux access control, which seems to be working well.