Hi,
Original propolis architect here, and primary engineer for much of the effort porting bhyve to illumos.
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
eddythompson80 1 days ago [-]
Not really familiar with bhyve or even illumos in the slightest. But reading this comment makes me a bit sad in a way. These types of conversations and comments used to be common where I worked just few years ago. Now no one even cares how something is implemented let alone a "I felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware". The "good" ones today are looking exclusively at external properties rather than internal ones. Thank you for the comment.
pmooney 19 hours ago [-]
> Now no one even cares how something is implemented ...
> The "good" ones today are looking exclusively at external properties rather than internal ones.
To provide a little additional context here: This effectively was an external property of the system (just to the userspace VMM). One of the significant challenges for those implementing a feature live migration is how you'll deal with multiple version of the software (older and newer) interacting in the future. A primary use case for live migration is moving workloads around to permit software updates, making that multiple version issue even "worse".
Abstracting the implementation details of those in-kernel emulated devices was helpful in two ways. Firstly, it reduced (hopefully) the chances that a later change to the internal implementation would be visible in the exported/imported state. Secondly, that the act itself of doing that state abstraction required a much more intentional and thorough examination of the emulation logic itself, resulting in a cleaner and simpler artifact.
So while it may seem like it was driven by a desire to take pride in the craft (undeniably _a_ factor), the main motivation was avoiding a versioning mess in the future.
> Thank you for the comment.
My pleasure!
genxy 18 hours ago [-]
By your storing of a more abstract representation of the emulated device state, could you potentially live migrate to tape or object storage?
bigstrat2003 22 hours ago [-]
I feel you. Good programmers still care about those things, but it turns out there are a whole lot of people who never cared about the craft and were just in the profession because it was lucrative. It's a shame.
hnlmorg 1 days ago [-]
What’s FreeBSDs story like for Docker these days?
I love FreeBSD. But I reluctantly had to migrate one of my home servers to Ubuntu Server because I needed Docker support as well as ZFS. Ubuntu Server, compared to FreeBSD, is a complete clusterfuck of an OS. And I regret the effort I spent migrating.
warpflyght 1 days ago [-]
There's experimental support for Podman and OCI containers on FreeBSD 14.3 and later, with some additional system configuration required [0], [1]. I've been playing with it off and on with FreeBSD 15.0 and 15.1, but I'm not using it for anything I depend on.
Use Debian. Ubuntu is a clusterfuck but it's not all related to the software they use
frrlpp 12 hours ago [-]
Debian is in a downward spiral after Buster.
hnlmorg 8 hours ago [-]
Why’s that?
genxy 18 hours ago [-]
We run Linux in a bhyve VM for podman support but backed by FreeBSD ZFS. Migrate back!
johng 1 days ago [-]
I used to use Ubuntu a lot about 7 years ago. I saw the writing on the wall. Once they started forcing snap down people’s throats, especially. I’ve been on Debian ever since and have been very happy. I also have run many other OS’s and I feel that Debian is still a good blend of polished and still friendly to the hacker.
hnlmorg 1 days ago [-]
Yeah Debian is nice. But at the time of switching, only Ubuntu offered ZFS precompiled in their official repos.
I did, for a period, run some other distro with DKMS but that was a massive pain to keep up-to-date due to ZoL’s piss poor documentation and messy build tools.
This was quite a few years ago though. Long before even FreeBSD synced with ZoL for OpenZFS. So the problems I encountered have likely matured tenfold since.
But if I’m going to the hassle of installing a new OS then I might as well install FreeBSD again because I just prefer it to Linux (nothing against Linux in general, my laptops and PCs all run Arch and I’ve made a career from supporting a variety Linux platforms)
pacija 18 hours ago [-]
I have FreeBSD directly on metal, with just ZFS datastore and vm-bhyve [1] directly on it. All my services are in separate bhyve VMs. Most of those are also FreeBSD-based but some that require docker are Linux-based (my radio [2] is azuracast [3] in docker inside Ubuntu as a bhyve vm on FreeBSD). This way I can setup zrepl [4] on host to periodically snapshot whole VMs and send them to another box or to external USB disk, and I'm backed up.
If you need Linux compatibility layer AND ZFS without a hypervisor, unfortunately it's hard to get close to a Linux host. I've never used FreeBSD or SmartOS in production (I did use Solaris and zones in the past) and I know ZFS support is first class in those. But I've been using both zfs-dkms and kmod-zfs in RockyLinux and the experience has been pretty pleasant (ended up chosing kmod because we only follow OS releases anyway and install time is better there). I'm sure the story is similar for OSes like AlmaLinux (which also get upgrades sooner than Rocky, and I would prefer to use it instead of Rocky, but it is a requirement for us).
On my desktop I also use ZFS with kmod on cachy and updates have been seamless so far.
InsidiousWombat 16 hours ago [-]
was that like 5 years ago or more ? Debian "just have packages" for a while
hnlmorg 8 hours ago [-]
Now I think about it, it might have been 5 jobs ago!! So possibly 10 years.
Wow it’s scary how quickly time goes by!
wyager 22 hours ago [-]
Out of curiosity, what required docker that wasn't better suited by jails? I know such things exist (e.g. software projects that are only distributed as dockerfiles), but not sure how common they are.
hnlmorg 20 hours ago [-]
Laziness. I ran Jails for years and was largely happy with it. But I didn’t want to handle the jails upgrades manually. I just wanted to run “docker pull”.
TacticalCoder 1 days ago [-]
> What’s FreeBSDs story like for Docker these days?
My current setup for Docker (Podman really in my case):
Proxmox -> Linux VMs -> Podman containers inside Linux VMs
My plan:
FreeBSD's bhyve hypervisor -> Linux VMs -> Podman containers inside Linux VMs
So basically just ditching Proxmox for FreeBSD/bhyve. Then at long last I'll be, again, systemd-free.
hnlmorg 1 days ago [-]
I did consider the bhyve approach myself, before I made the switch. But it was still pretty experimental at the time (iirc) and wanted something reliable otherwise I’d have a wife and two kids moaning at me that their stuff isn’t working haha!
But maybe you’re right that it’s worth reconsidering now.
JohnDohnBohn 1 days ago [-]
Looks like this could really become a game changer for the FreeBSD folks using bhyve. The author often complained about this missing feature and he was also contributing to the Sylve project. Looks like he's crafting something.
e12e 1 days ago [-]
Anyone heard anything from dragonflyBSD lately? This sounds like something that might go well with HammerFS?
adrian_b 1 days ago [-]
I am using FreeBSD, not DragonflyBSD, but I happen to be on a DragonflyBSD developer mailing list, which is normally low traffic, except that I have noticed recently a lot of messages with various patches, mentioning both bug fixes and newly implemented features.
So the project is alive and it seems more active than previously.
eddythompson80 1 days ago [-]
There is some uptick in interest in various BSDs as a linux alternative given the rough few months it's been having. I know 2 companies that are trying to see if they can switch some edge nginx proxies to run on OpenBSD. A RCE in a TCP handshake is up there tbf
ksec 14 hours ago [-]
Interesting to hear / read this. I know OpenBSD has made a lot of improvement in the past 2 - 3 years on performance. But I would have thought for heavy traffic it is still not anywhere near FreeBSD or Linux.
But I guess we are at the point security is important and hardware ( CPUs ) is relatively cheap?
eddythompson80 12 hours ago [-]
It was a brief conversation in a social gathering for smallish engineer-led startups in my area. The one guy who went into details mentioned they are evaluating OpenBSD because of its presumed focus on security. I doubt they are super high throughput, but they might land on something else too. They run their own hardware at least, so the hardware compat issues of OpenBSD didn't seem to be a problem for them. FreeBSD was mentioned as well fwiw. I don't know how far along they got in performance testing or actual switch.
Just general angst some people were expressing for publicly exposed resources. The common wisdom just a few years ago was that throwing an internet facing nginx or haproxy on an up-to-date LTS linux with some firewall debouncer is as rock solid as you could get. And instead, you should worry about security in your own upstream app, not Linux's TCP sockets or nginx's regex path matching.
sigttou 1 days ago [-]
> So the project is alive and it seems more active than previously.
It indeed is! Looking forward to the next release, which should happen "soon", as 6.4 is already 4 years old.
twunde 1 days ago [-]
This brings back memories of watching someone do a live demo of bhyve at nycbug. I think that just have been almost a decade ago now, but the live demo was amazing.
yannick2k 12 hours ago [-]
If this is comming I can ditch proxmox
jbird99 1 days ago [-]
Nice! Can we have high availability / clustering for jails next?
vermaden 12 hours ago [-]
Even Solaris in Kernel Zones uses virtualization for the kernel part to have 'live migration' - regular Solaris Zones Containers does not have 'live migration' - you can just start/stop them.
But ... OpenVZ in the past had 'live migration' - so it would be nice to implement that feature into FreeBSD Jails to have 'high availability'.
MobiusHorizons 1 days ago [-]
That sounds interesting. Could you elaborate what that would mean?
1 days ago [-]
1 days ago [-]
d4gli 12 hours ago [-]
Nice!
InsidiousWombat 16 hours ago [-]
Welcome to 19 years ago I guess (KVM got it about that time)
jacobgorm 15 hours ago [-]
L4Linux on NomadBIOS got it in 2002, and Xen got it about a year later, after I had met with the Cambridge Xen team at SOSP in upstate New York.
bonzini 15 hours ago [-]
Yes, Xen is several years older than KVM. The NSDI paper on live migration from the Xen folks is from 2025.
jacobgorm 15 hours ago [-]
And I am one of the authors of that paper :-)
bonzini 10 hours ago [-]
I know, ;) this was more for context.
Also neither of us spotted that I wrote 2025 instead of 2005 but I can't edit anymore. Moral of the story, don't write comments at 7am.
vermaden 12 hours ago [-]
Same with Docker.
FreeBSD had Jails containers since 1999.
Linux joined the party starting with on 2014.
bashhq_r 1 days ago [-]
so bhyve getting live migration finally means sylve might actually be usable for production clusters, been waiting on this.
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
> The "good" ones today are looking exclusively at external properties rather than internal ones.
To provide a little additional context here: This effectively was an external property of the system (just to the userspace VMM). One of the significant challenges for those implementing a feature live migration is how you'll deal with multiple version of the software (older and newer) interacting in the future. A primary use case for live migration is moving workloads around to permit software updates, making that multiple version issue even "worse".
Abstracting the implementation details of those in-kernel emulated devices was helpful in two ways. Firstly, it reduced (hopefully) the chances that a later change to the internal implementation would be visible in the exported/imported state. Secondly, that the act itself of doing that state abstraction required a much more intentional and thorough examination of the emulation logic itself, resulting in a cleaner and simpler artifact.
So while it may seem like it was driven by a desire to take pride in the craft (undeniably _a_ factor), the main motivation was avoiding a versioning mess in the future.
> Thank you for the comment.
My pleasure!
I love FreeBSD. But I reluctantly had to migrate one of my home servers to Ubuntu Server because I needed Docker support as well as ZFS. Ubuntu Server, compared to FreeBSD, is a complete clusterfuck of an OS. And I regret the effort I spent migrating.
[0]: https://docs.freebsd.org/en/books/handbook/containers/
[1]: https://podman.io/docs/installation#installing-on-freebsd
https://daemonless.io/
I did, for a period, run some other distro with DKMS but that was a massive pain to keep up-to-date due to ZoL’s piss poor documentation and messy build tools.
This was quite a few years ago though. Long before even FreeBSD synced with ZoL for OpenZFS. So the problems I encountered have likely matured tenfold since.
But if I’m going to the hassle of installing a new OS then I might as well install FreeBSD again because I just prefer it to Linux (nothing against Linux in general, my laptops and PCs all run Arch and I’ve made a career from supporting a variety Linux platforms)
[1] https://github.com/freebsd/vm-bhyve
[2] https://www.dno.rs/radio
[3] https://github.com/AzuraCast/AzuraCast
[4] https://github.com/zrepl/zrepl
On my desktop I also use ZFS with kmod on cachy and updates have been seamless so far.
Wow it’s scary how quickly time goes by!
My current setup for Docker (Podman really in my case):
My plan: So basically just ditching Proxmox for FreeBSD/bhyve. Then at long last I'll be, again, systemd-free.But maybe you’re right that it’s worth reconsidering now.
So the project is alive and it seems more active than previously.
But I guess we are at the point security is important and hardware ( CPUs ) is relatively cheap?
Just general angst some people were expressing for publicly exposed resources. The common wisdom just a few years ago was that throwing an internet facing nginx or haproxy on an up-to-date LTS linux with some firewall debouncer is as rock solid as you could get. And instead, you should worry about security in your own upstream app, not Linux's TCP sockets or nginx's regex path matching.
It indeed is! Looking forward to the next release, which should happen "soon", as 6.4 is already 4 years old.
But ... OpenVZ in the past had 'live migration' - so it would be nice to implement that feature into FreeBSD Jails to have 'high availability'.
Also neither of us spotted that I wrote 2025 instead of 2005 but I can't edit anymore. Moral of the story, don't write comments at 7am.
FreeBSD had Jails containers since 1999.
Linux joined the party starting with on 2014.