I made this tweet that got reasonably popular:
We benchmarked how long it takes to run the Ruby test suite for Discourse across our various dev machines. I can not believe what a crazy tax I have paid over the years insisting on sticking with Windows, highlighted results mine.
https://twitter.com/samsaffron/status/1111511735851081728
This evoked a lot of extremely strong emotions from various people out there. Ranging from “Sam is a fool what kind of insane benchmark is this”, “the real story is MacOS has bad Ruby perf” to a general “Oh no”.
The core point I was trying to make was that I was paying a pretty high tax for deciding to “stick with with Windows”. There are a bunch of other points hiding here that are also worth discussing.
Why are you running sticking with Windows to run Linux in a VM?
What I did not know is the extent of the VM tax I was paying regularly. I never dual booted my computer so I had no proper anchoring point of reference.
I very strongly believe that many Ruby/Rust/Go/Elixir/Scala and even some Node developers who end up doing the WSL dance or run Linux in a VM for development, or use Linux Docker for dev on Windows are not aware of the full extent of the tax.
On my machine the price of admission for using WSL was 25% slowdown in my day to day running of specs. And a 38% slowdown for using a VMware based VM.
I am not alone here… other team members have experienced similar slowdowns. Other people out there also experience similar slowdowns.
What I thought was inside my wonderful wish hat was that the performance hit was minor:
Yes. But that is not the question. The difference is normally negligible (1% to 5%).
If you Google, well that is the general answer you get. VMs are more or less the same as no VM, 5-10% perf tax. My reality was different. Maybe I was missing magic bios turbo settings, maybe I needed to direct mount a volume instead of using a vmdk on my dedicated NVMe Samsung 960 pro, maybe there is some magic I could do to get to this magic 1-5% number. Maybe Hyper-V is better I am not sure. All I know is that I am not alone here.
WSL is not an option for me cause Ruby likes lots of small files and lots of stats calls, WSL has terrible lots of small file performance as documented by the WSL team. How terrible you ask? As a baseline just running a rails c console without bootsnap was taking me upwards of 30 seconds. Same operations takes 4-5 seconds on Linux without bootsnap. Even with all the workarounds we could place this bad IO perf was something that I just noticed too much. In fact I preferred the 38% slowdown cause at least stuff was consistent and not wildly off balance like WSL is. Being able to launch a console or web server quickly is critical during dev. Fuse does not appear to be happening any time soon so you can not work around this using ninja tricks of block mounting a device.
So, I stuck with a VM cause it was nice not to have to constantly reboot my computer and thought the price I was paying was not that high.
I like the Windows 10 font rendering, I like the HiDPI support, I like using Lightroom on Windows and playing Rocksmith on Windows. I like the out-of-the-box experience and minimal amount of tweaking needed. I like being able to launch Skype without it segfaulting cause I was LD_PRELOADing jemalloc. I feel Windows 10 as a window manager is on par (for my usage) to my Macbook Pro running MacOS.
Dual booting is a compromise for me, some stuff I have works best in Windows. I thought the compromise I was making performance wise was worth the comfort of living in a “known OS” that I like.
I felt that if I start booting to Linux I am going to have to fight with drivers, have stability issues, not have a complete toolset and so on.
I felt comfortable at home and moving is one of the most stressful life events.
Is 2019 the year of Linux on the Desktop?
The joke goes like this. Every year a bunch of people joke about how LOL this will be the year of Linux on the Desktop. It happens every year. It starts cause someone says “hi Linux is quite good these days, could this be the year of Linux on the Desktop?”. And then a bunch of happy and well meaning trolls, say ha ha … as always you are wrong… this is not the year of Linux on the Desktop.
And so it goes…
This banter is usually well meaning and grounded in reality. However it has a very big side effect, which impacts developers in a significant manner. Developers who do not use Linux on the desktop are scared of Linux. They are scared even if their production code only deploys on Linux (and not MacOS or Windows)
I felt super scared to go down the path of Linux cause I was terrified … about drivers … font rendering… HiDPI support… multi monitor support and the list goes on.
In fact, I was not wrong to be scared. It is fiddly to get Linux going. I almost gave up after my first 4-8 hours cause Firefox on Linux is still stuck on a very sad default there is no hardware acceleration out of the box, so scrolling is mega jerky. This very simply rectifiable behavior was a deal breaker for me. If I could not get scrolling a web page to be smooth, I am out of here, not going to use Linux. Luckily the issue was resolved after tweaking 1 value in about:config.
NVIDIA does not have a great story as well, the future of Desktop on Linux is Wayland. The windows manager I wanted to try, sway, only works properly if you use the open source community provided nouveau driver. Even getting NVIDIA to work nicely involves enabling hardware compositing and fiddling with X11 config.
My position is not that Linux is poised to take over the world in a storm this year. It is a far more humble position. If you want to get the best bang for buck and want to get the best possible performance developing Discourse, or any Ruby on Rails application Linux on the Desktop/Laptop with no VM is your best bet.
It is also important to note that I opted for medium hard mode when I moved to Linux. I am only 2 neck beards away from installing Linux from scratch.
My colleagues who went through similar exercises of shifting from Windows/Mac to Linux stuck with Ubuntu and Linux Mint, they tell me they had a very smooth ride.
Have you tried running Ruby on Windows?
Avdi triggered quite a discussion about this a few days ago:
The point he is trying to make is that a Ruby that works well on native Windows will help Ruby adoption a lot and eliminate drain to other frameworks. Installing a whole brand new OS is just too much of a barrier. Just install Linux is not a solution.
The reality is that running MRI Ruby native on Windows hits 2 big fundamental problems:
-
Filesystem performance characteristics of NTFS on Windows are badly suited to the current Ruby design. We love lots of small files, we love lots of stats calls.
-
It is a gigantic effort porting various dependencies to Windows native (and maintaining them), as it stands many of the Discourse dependencies simply do not work on Windows. The gems simply will not install. The fundamental issue is that if you are writing a c extension in a gem it is extra work to get it to work on Windows. Getting stuff to work on MacOS and Linux is no extra work vast majority of the time.
(2) is a tractable problem, but I wonder if it is worth any kind of effort given WSL has far wider compatibility and should offer reasonable performance once a workaround exists for the filesystem problem (which is fundamental and not going to change on Windows native). Discourse works fine on WSL (provided you skip using unicorn) Discourse does not work at all on Ruby on Windows native. The Apple tax is similar in cost to the Windows WSL tax (except for filesystem perf). I feel that once WSL gets a bit more polish and fixes it will be competitive with the current Mac experience.
The Apple performance tax
One pretty obvious thing from the chart I provided was showing there is a pretty severe Apple performance tax as well.
When looking at user benchmarks per: UserBenchmark: Intel Core i7-8559U vs i7-8750H. We expect an 8559U to have faster single core performance (thermal locking withstanding) than the 8750H. Yet this Linux 8750H laptop is clocking a spectacular 9m13s compared to the Macbook Pro 15m16s. We are seeing poor MacOS performance across the board. And we are not alone:
It appears that people insisting on the native MacOS experience are paying a significant tax for developing Ruby on Rails on a Mac.
I know that DHH loves his iMac Pro and recommends it enormously.
Yes, the hardware is real nice, the screen is beautiful, the machine is wonderfully put together. The Window manager is nice. Zero driver problems. However, sadly, there is a significant OS tax being paid sticking on MacOS for Ruby on Rails development.
I think the Ruby community should explore this problem, document the extent of this problem and see if anything can be done to bring Darwin closer to the numbers the same machine does with Linux. Is this problem rooted in the filesystem? The OS? The llvm compile of Ruby? Security features in MacOS? Something about how Spectre+Meltdown (which is already patched in my Linux)? It is very unclear.
As it stands I would not be surprised at all if you dual booted a Mac with Windows, installed WSL and got better performance running the Discourse test suite on Mac+Windows+WSL. In fact I am willing to take bets you would.
So, to all those people who say… oh there is an alternative … just hackintosh your way out of this mess. Not only are you stuck playing Russian roulette every MacOS update, you are also paying a tax which is similar to the tax you are paying on Windows already.
What about parallel testing?
Rails 6 is just around the corner. This is the first time Rails is going to ship with officially supported and sanctioned parallel testing. When I run the Discourse spec suite on my Linux system CPU barely scratches the 10% mark for the whole 8 minutes the test suite is running, IO is not saturated.
Here I am freaking out about a measly 38% perf hit when I could be running stuff concurrently and probably be able to run our entire test suite in 2 minutes on my current machine on Windows in a VM.
It may feel a bit odd to be making such a big deal prior to taking care of the obvious elephant in the room.
I completely agree, parallel testing is an amazing thing for Rails, this is going to make a lot of developers extremely happy.
Also, profiling our test suite, eliminating and improving slow tests is super important.
We are going to adopt parallel testing for our dev environments this year.
But I guess this was not my point here. My issue is that we I was driving with the hand break on. Even when our test suite gets faster, the hand break will remain on.
Where am I headed?
I am feeling significantly happier in my Arch Linux home. In a pretty surprising turn of events not only is stuff much faster for me, I also feel significantly more productive at work due to having a windows manager that works much better for me than my Mac or Windows setups ever did. Yes there are compromises, I need to get my hands far dirtier than I had to in the past. However the payoff has been huge.
I have been a long time I3wm user, however I never got the proper experience being straddled in the middle of 2 windows managers. Now that i3 is my only windows manager I am unlocking tremendous amount of value out of it.
Why, you ask? Well I plan to write a bit about my experience over the next few weeks. My plan is to try out a different tiling windows manager every month for the next few months to find the flavor that fits me best.
I stuck with Windows for 6 years developing an application that works best on Linux because I was comfortable in Windows. Habits are incredibly hard to break. I was not fully aware what price I was paying. I can also assure you many other developers are in the same boat as I was.
If I have one piece of advice here, it is … don’t be afraid to experiment. Linux on the desktop is getting better, it is reasonably straight forward to re-partition a drive and setup a dual booting system. If you are in the same boat as I was, living between 2 worlds, especially if you are on a desktop and not a laptop, take a break and experiment.
Please feel free to post any of your experiences or benchmarks here, I will try to answer every post on my blog carefully. I am curious to see more benchmarks from more people comparing MacOS to Linux on the same computer or Windows+WSL / VM and Linux.
And as always … enjoy.
