I work for a small company that produces a DAW and VST plugins. Supporting Linux is a huge amount of work compared to Windows and Mac.
The main issue is that 'Linux' is not a thing you can support. You have to pick the distros you want to support, and then once you've picked a distro, what versions you want to support.
And you need to use the C++ version that ships with that distro, so if you want to support old versions then the entire project is held back from using latest C++.
And distos aren't backwards compatible. ie when libcurl4 is released, they remove libcurl3. So you can't have one binary for Ubuntu 18.04 and 16.04.
So it means for every product, have a build for every distro / version combination you want to support.
So now the solution is AppImages where you bundle up your app with all dependencies like a container. Haven't investigated this yet, not sure it will work for plugins.
> And distos aren't backwards compatible. ie when libcurl4 is released, they remove libcurl3. So you can't have one binary for Ubuntu 18.04 and 16.04.
If you don't want to depend on the libcurl provided by the OS, ship your own.
If you don't want to depend on the glibc provided by the OS, ship your own, with your own dynamic linker.
readelf -n <your binary> will tell you the oldest kernel that will run your stuff. Put that in the requirements document. Write a bash script that sets LD_LIBRARY_PATH correctly and make sure that's the main entry point to your application during deployment.
This is not as easy as it sounds, because funny things happen if you ship your own libc - I remember last time I tried this there were problems with PAM and name services. But that was a number of years ago.
Can you ship with your own version of GNOME different from the currently running one?
> Can you ship with your own version of GNOME different from the currently running one?
I'm confused. GNOME is a desktop environment. It implements a task bar, a control panel, a way to set your desktop wallpaper etc. Why would you ship GNOME?
Maybe you meant shipping a proprietary application that uses a GTK version that is different than the OS-provided GNOME uses?
If so, of course you can. GTK is a bunch of shared objects that generate system calls, X messages, etc. It has no reason to interfere with the OS-provided GTK as long as it's got everything it needs in your bundle.
This doesn't work for things that are dynamically loaded into other applications (as VSTs are). In general, you need to be using the same version of the libraries as the rest of the application.
Your suggestion to ship glibc caused coffee to end up over my keyboard :) This is the road to hell, but you'd only know that if you had the slightest clue of the ABI interactions involved in swapping out a core library like that - starting with the reality that parts of the (probably binary-only) graphics stack must be linked and loaded in-process, and they naturally depend on at a minimum glibc. Your suggestion is to effectively ship the game in the form of its own Linux distro, which is of course complete nonsense.
Snap and Flatpack will hopefully help with the dependency aging problem, but they're brand new, and they both still suck one way or another.
To give you an idea of what swapping out glibc would involve, here is the list of shared libraries loaded by 'glxgears' on my machine, arguably the simplest possible OpenGL program:
Ignoring your personal attacks, you and I are on the same page.
I do agree that shipping your glibc is "the road to hell" and shipping with the glibc could be qualified as "shipping in the form of its own Linux distro", as amusing as that sounds :) But that's the price of freedom you pay if you want to draw your platform boundary at the kernel -- you need to ship the whole userland! Where's the surprise in that? That means you need to understand the interactions between different version of /usr/lib64/opengl/nvidia/lib/libGLX_nvidia.so.0 (that you need to ship) and nvidia.ko (that is nvidia's binary driver loaded by the kernel, that you do NOT ship)
That's what people mean when they say they "are targeting Linux", even when they don't really understand the kind of work that that statement entails :)
Linux is an OS kernel. I'd wager it's the most popular one by far, in terms of the number of platforms that uses it.
Saying that "Linux is fragmented" is the wrong way to look at the problem. One should rather realize that Linux is just and OS kernel that is used by countless platforms. It's up to the developer to draw platform boundaries and focus engineering effort on the chosen ones.
If you need to support GNU/Linux with glibc-2.25, then you need to set up your testing bench to accommodate that. If you need to have native look and feeling in ubuntu, kubuntu and lubuntu, you already have 3 platforms with at least 2 LTS versions that you need to test with.
Each platform has its own quirks. Linux distros are the ones with the least amount of quirks, but in exchange we get many many platforms, because it's so easy to create them.
glibc is pretty good at ABI backwards-compatibility. The right thing to do would be to statically link libstdc++, libgcc, and extras like libcurl, but dynamically link libc, libX11, and libGL. Luckily libX11 and libGL are pure-C so you can get away with static-linking libstdc++, which means you can use whatever C++ version you like.
It's definitely technically possible to find a hacky combination that works today, but only by shouldering the cost of understanding all the possible deps and symbols and the structure of their underlying objects used by all possible e.g. graphics drivers and X11 libraries now and into the future. You can hack around it today, but it's a fool's game and absolutely not something that can be relied on to continue working in any sound manner, without risking e.g. some memory corruption due to a struct layout difference long in the future, etc.
I wanted to check whether the official Nvidia driver uses C++, but it's not installed on this machine just now, and of course that I even have to check just highlights the problem with this approach.
> I wanted to check whether the official Nvidia driver uses C++,
I don't know for nvidia, but I had a problem two years ago on a machine with a radeon card: I was developing a GUI software which used LLVM at some point. Insta-crash at runtime on this computer whenever I'd oepn a window. The reason ? the radeon driver linked and initialized LLVM which didn't support being initialized twice...
There's no other option, though. If you aren't integrated into the distros' builds, then you have no way of getting your binary updated when a dependency makes an ABI-incompatible change -- much less shipping a single binary that works across all distros. The only reasonable option is to determine a subset of your dependencies which offer long-term stable ABIs, dynamically link those, statically link the rest, and hope for the best.
I'm not sure how per-distro (or distro-version) builds fix the problem, unless you're committing to continuously produce new builds for future distro versions to account for ABI changes. If you can do that, you can also fix issues with static linking if/when the come up.
Windows actually does require this, which is why applications tend to ship with (or otherwise require) some kind of C(++) runtime, like those provided by MSVC or MinGW. Some rely on these being installed separately, while others include them as part of the app's installation process. Same deal for .NET. I haven't done enough macOS development to confirm or deny, but I'd assume the situation is similar there, too.
The actual difference is that Visual C++ and XCode (presumably) automate most or all of this, since this is the standard way of compiling and distributing applications for those platforms. In contrast, Linux development tends to revolve around software that can be recompiled for each distribution, so the tooling is going to be optimized around a workflow of relying on system-wide libraries and binaries and other data managed by a package manager.
Great and now I have to also track security updates and ship those for all my dependencies. libcurl is a great example with a release cycle of 2(!) months and tons of vulnerabilities being fixed each time.
That is completely ridiculous. Pointing out that statically compiling dependencies or shipping them with a game can solve many so called compatibility problems is directly related to games working on linux.
As you can see, both comments were indeed flagged by one of those stackexchange types that is always marking useful questions as off-topic. Probably the same person who thinks using anything but rpm is bad.
The thread has then been detached for the crime of discussing voting, so only we can see it.
The first rule of HN voting is not to discuss voting.
It's a wonderful idea - on paper, but all that's needed is for one of the many mandatory deps for running a GL program to link against the system libcurl or any other replaced dep and you're back in Crashville again, population 1 - cloth-eared developer
Are there mandatory dependencies for running a GL program that can't work without using the system libcurl shared library? How does that make any sense?
If you distribute and link libcurl v69 dynamically, the dynamic linker symbol table for your process will contain references to v69 symbols. If you then load an e.g. GL driver that in turn links libcurl v42, the GL driver will end up with some combination of v69 and maybe v42 symbols, despite being compiled against v42 headers containing v42 struct sizes and offsets.
Now there is an opportunity for silent memory corruption in your program, one that even once detected, which may not happen for years, cannot easily be fixed without completely rearchitecting the build.
When using a wide variety of system APIs including at least NSS and OpenGL, there is little control over what dynamic libraries end up loaded, the exact configuration will vary across machines
Substitute libcurl for libexpat or libX11 or libstdc++ and the principle continues to apply. I can't think of a good reason something like libcurl would be used by the Nvidia driver or some bizarro embedded board's GL driver, but that doesn't mean the Nvidia driver doesn't now or in the future link it, and if/when that day arrives, the problem lies entirely with the program that mixed random system deps with random self-compiled deps.
> So now the solution is AppImages where you bundle up your app with all dependencies like a container.
But isn't this what you're doing with Windows? No version of Windows comes with libcurl (as far as I know) so you put the DLL in your application directory. It's no different.
On Windows, I use the Windows API for downloading files. I don't need to worry about curl.
On Linux, I could have statically linked curl, but didn't realize it was going to become an issue. Ubuntu 18.04 was released and curl3 was removed, so that means our software that was working was either automatically removed by the package manager or it started crashing.
Can a installer prompt user to download missing libraries if needed? For example - Dota2 on Windows downloads bunch of C++ runtime libraries on first launch. Sharpkeys and bunch of other software do that (certain version of .net runtime).
libcurl may or may not be a good example but surely all software has dependencies that isn't shipped with OS. In Linux case the special pain is, library that is shipped with OS might be outdated but bundling compiled lib/o/so files should work.
On Windows, I often get prompted to install C++ runtime dependencies when installing. One big thing Steam did was manage multiple installs of DirectX for each game. It's just a matter of Linux being cost/benefit to overcome these things, like they have on Windows.
Which is probably way at the back of the developer's minds: the main audience of Linux is happy to compile from source (either themselves or via a downloaded package).
Exactly. Quite a few of the "problems" with Linux development ultimately stem from trying to shoehorn Windows-oriented expectations into Linux development/deployment. Very few applications in the Linux world target specific distros themselves; instead they publish their source code and leave the actual distro-specific finagling to distro/package maintainers (sometimes those maintainers are also developers, but they're regardless treated as separate concerns).
Of course, quite a few game developers are curiously averse to publishing any kind of source code, which tends to thus require compatibility efforts to happen on the developer side instead of the package maintainer side. This can still be worked around by targeting a specific distro (say, Ubuntu or SteamOS) and letting package maintainers for other distros apply whatever workarounds they deem necessary to get the app running outside of a "supported" environment (see also: Steam, Spotify, Slack, etc.).
I'll grab source code if I'm doing development or something obscure, but I feel like packages have been pretty solid and widely available since the early to mid 2000s. When working with source code I'll almost always install to a custom PREFIX--but if it's just an app the source is the last resort.
All of the commercial packages I've used at work for the past 15 years target either one or a handful of distros. Even the open source stuff that's either too new or outside of the default repos I much prefer to install from a package; turbovnc, grafana, chrome, etc. The few games I've seen have worked the same way as other commercial user-oriented apps.
You don't have to statically link, you can use LD_PRELOAD to point to local versions of all libraries. It isn't optimal, from space/security perspectives, but it works. You can use the local libs first or last in the path.
You don't even have to do that. When you compile your binary you just need to set the rpath to your own directory of libs and they will be search automatically and fall back to the system if missing.
Some useful stack overflow answers on this topic. First, the trick to get GCC to pass the ORIGIN option to the linker: -Wl,-z,origin.
Secondly, how to pass it as the rpath option:
"-Wl,-rpath,'$ORIGIN' -- note that you need quotes around it to avoid having the shell interpret it as a variable, and if you try to do this in a Makefile, you need $$ to avoid having make interpret the $ as well."
The chrpath utility is also fascinating, as it allows you to rewrite the elf headers to change the rpath. Possibly better for installs than passing lots of env variables.
Various warnings on this topic to not use actual relative paths unless you really want that behaviour.
I work for VCV, which also ships for Linux. I don't find it an issue at all, since the build system is a Makefile supporting Mingw64/MSYS2 on Windows, Mac, and Linux.
We statically link everything except glibc, which we've decided to dynamically link to glibc 2.23 (meaning that Ubuntu 16.04 is the oldest we support). We've had no problems with this approach, although the disadvantage is that we have to use an old version of GCC to compile the software, since I haven't figured out a way to make a new GCC version produce binaries which link to old glibc.
No need for containers or anything, just make a mostly-static-except-for-glibc binary.
> the disadvantage is that we have to use an old version of GCC to compile the software, since I haven't figured out a way to make a new GCC version produce binaries which link to old glibc.
I personnally compile with latest GCC or LLVM on centos 7, this way I can use the very latest C++ standards with a venerable glibc
That sounds like more something to blame the Debian family over than the whole ecosystem. I might be running "bleeding edge" on Arch but I can still depend on and use libcurl3 with the libcurl-compat package.
I maintain about two dozen software packages in the AUR, including some really old stuff like the Heretic 2 Linux release from 1999 and RBDoom 3 BFG which has a boatload of dependencies. Breakages are extremely rare for the average package even with the rolling release and generally any breaking change in a common library will see the legacy version hang around since stuff will still depend on it.
I don't think it's Debian, because Debian tends to have compatibility packages for old versions too. For libcurl there's libcurl3. It depends on the package, of course.
I don't have a Debian machine to test it on but a casual search finds qt1-3 in the AUR on Arch. Of course building the whole GUI suite would be a bit annoying but its better than nothing if you have some really old software that depends on it.
Why not pick a "distribution" like org.freedesktop.Platform/18.08 (i.e. pick a flatpak runtime) and support that? It runs on all desktop distributions. Updating supported flatpak runtimes is then on your leisure, not on their release cadence.
The runtimes are not locked down, anyone can publish theirs (under their domain name, ofcourse), so it depends on who publishes it.
For the freedesktop one, they have following policy[1]:
- When a new stable release release is done, the changes on that branch will only be:
- security updates
- stable releases (tested carefully); no ABI breaks / API build breaks.
- we will try to keep updating that branch as much as we can
- We release a new major release every year, only if:
- There is a ABI break
- There is a "API build" break (apps might not compile because new major releases of important packages (GCC))
- Looking at the GCC and other major project cadences, this is likely going to happen annually.
- We only maintain the current stable release and the previous one, this means:
- Stable releases get 2 years of security updates
- We maintain maximum 3 releases at any given time:
- Development
- Stable
- Old stable
I am developing an open-source DAW with Qt (https://ossia.io), to solve the problem you mentioned I used AppImage with great success - of course you have to ship all your libs, but that's what you have to do on macOS and Windows too anyways.
I concluded ages ago that no substantial applications should be integrated into an operating system. Dynamic linking if you're building something portable is just an awful idea because of how much variety you'll inevitably have to support (or choose not to support).
Some people have written about AppImages, this also applies to FlatPak and similar techs too. Isolating anything more complicated than a command line tool is the way forward and the tech exists. Not using isolation this way is a recipe for pain, whether it be on the desktop or the server (or the phone).
It depends how the plugins are loaded - it'd be great if they could use sockets so that AppImages were viable. No idea myself, though.
Confusingly, you can statically link a dynamic library (i.e. the dynamic library includes all its dependencies statically instead of recursively depending on more dynamic libraries). You can even link it in such a way that the dynamic library gets its own version of each dependency regardless of what the main executable is linked to (otherwise the main executable's links could override yours).
I did this once with a ruby gem that had particular c++ dependencies that kept breaking when the build machine had different library versions than the production machines. If you aren't integrated directly with the development of a linux distro, it's best not to use their packages as runtime dependencies, since they really only consider their own use before changing things up.
Music software's dependance on Windows is such a mess. Tons of cool plugins are stuck as windows-only VSTs! I was really hoping that Propellerheads would use the fact that it can ship on any platform to ship Reason and all associated REs on linux or in-browser, but instead they gave in and added VST support.
A historical lack of low-latency audio is likely why you haven't seen many music software releases on Linux. For years, you had to have a custom kernel to get "decent" performance.
Can confirm. I fondly remember late nights hacking the Gentoo kernel so my college radio station could reliably live stream shows via icecast and darkice.
The DAW isn't Bitwig is it? (just because your username has the same amount of syllables ;) If so, then PLEASE keep up the great work supporting Linux. I was only able to ditch my Mac a year ago because I switched from Ableton Live to Bitwig!
it is a blessing to be able to work with a different paradigm than piano roll and with such modern tooling (renoise supports vsts, rewire integration and whatnot), being essentially a supercharged tracker.
biggest problem with plugins imo is that often they don't support the same things as the daw does so usually come without linux installers (even though the code runs just fine on linux... )
Audio software has strict buffer/latency demands which usually cannot be guaranteed on interpreted language platforms.
Doing audio synthesis with JS or any other interpreted language really is totally possible and has been done in a more or less serious way in several implementations and webtoys etc. But if you need extremely low latency and guarantees you cannot go that route, sadly.
the UI is the minority of the code. the main engine is gonna need to be implemented in some language which has strict guarantees about performance. also, not having a garbage collector that fires in a seemingly random way helps a lot too :)
There are few reasons:
- they most of the time run embedded in a DAW
- they are usually computationally intensive themselves
- they are meant to be instanced as far as memory/CPU can go
ad to the third point:
Music producers already require and use pretty powerful rigs: 32-128GB RAM is not uncommon, CPU as good as it gets.
There's great benefit when you can run 100 instrument synthesizer instances parallel vs 14 instances - it's a difference between a differentiated orchestra and a rock band.
> You have to pick the distros you want to support, and then once you've picked a distro, what versions you want to support.
What if game developers release the game's source code and let community developers help with porting to different distros and platforms? The game's assets can remain paid. For example, Doom has been ported to pretty much all platforms, and it's up to maintainers to ensure compatibility. I guess at this point it becomes a partly open source/free software game.
I'm aware this may not align with current business practices.
> And you need to use the C++ version that ships with that distro, so if you want to support old versions then the entire project is held back from using latest C++.
> And distos aren't backwards compatible. ie when libcurl4 is released, they remove libcurl3. So you can't have one binary for Ubuntu 18.04 and 16.04.
Release the source and the community will help you with many of these issues.
I don't know why people say that. You don't support Windows XP and Windows 10 either with the same binary.
In contrast to Windows you can provide or pay someone to provide libcurl3 if you want to continue to use it. MS will just say you can go F yourself.
In contrast to Windows you can study the source code and write a wrapper that provides a central API point that you can use in your single code base.
In contrast to Windows you can actually go there and provide patches. Even if the original authors won't merge it you can still use it via a fork etc.
And last but not least I bet there are actually still people supporting and providing libcurl3 binaries to this day and you just need to google their package server and add 2 lines to your installer script (one to add the public key for that package repo and one to add the package repo to your package manager).
PS: If you provide software for sizable amounts of people you need to provide 1-3 out of 3 reasonable Distros: Debian, RHEL, Suse. Even if you just provide one most people can deal with it thanks to VMs or docker.
> You don't support Windows XP and Windows 10 either with the same binary.
I think you picked a terrible example there, because quite often people do; certainly Vista upwards is quite normal (that was the transition point for lots of APIs).
Windows is much clearer about how you're supposed to solve "DLL hell". You have the OS libraries, which provide a stable API; COM, where interfacing is done at runtime dynamically; and for everything else you put it in your application's directory.
Theoretically if COM components aren't interchangeable - the API has expanded - they should have a different CLSID and therefore not clash.
This is a common misconception. Microsoft is a little insane about backward compatibility. You can target versions of Windows with a single binary from the latest all the way back to unsupported OS's like XP. There are lot's of companies that take advantage of this. It's one of the reasons why MS has so much trouble moving app developers to the new hotness, even if it's safer, faster, or whatever. Because of that the new hotness doesn't get enough traction to support continued development and they sunset it early which draws even more ire from people.
The main issue is that 'Linux' is not a thing you can support. You have to pick the distros you want to support, and then once you've picked a distro, what versions you want to support.
And you need to use the C++ version that ships with that distro, so if you want to support old versions then the entire project is held back from using latest C++.
And distos aren't backwards compatible. ie when libcurl4 is released, they remove libcurl3. So you can't have one binary for Ubuntu 18.04 and 16.04.
So it means for every product, have a build for every distro / version combination you want to support.
So now the solution is AppImages where you bundle up your app with all dependencies like a container. Haven't investigated this yet, not sure it will work for plugins.