What is the answer then?
Bottom line: this *&^? is hard, and anyone who pretends otherwise is either lying, deluded, or hasn't spent any serious time developing large-scale software. From the perspective of the developer of a small app with limited deps, the policies of the distro maintainers can look rather fascist; but they're looking at the bigger picture of trying to put together a whole system that works reliably and predictably according to a bunch of clear conventions that users can learn and follow. They have to worry about security and stability as well. What happens when libpng 1.4 turns out to have a horrific buffer overrun vuln that allows an OXP author to get root access on your box and turn it into a bitcoin mining bot for the Russian government? Then centralised dependency management starts to look like a much better idea!
One thing that would help a lot is if the whole OSS community learned to do versioning properly with SemVer. Then at least it would be possible to reason about dependencies automatically. I sympathise a lot with your complaint:
Its the ones who constantly change the structure of these dependencies with complete disregard for any backwards or forward comparability
Although there are often reasons why these changes make sense from the bigger perspective, I'm not going to deny that there are plenty of OSS projects that break compatibility wilfully and carelessly without providing warning or support to app authors that depend on them (*cough* GNOME *cough*) Proper versioning would help and might also give developers pause for thought before they break backwards compatibility because they feel stupid putting out version 254.3.1 a year after the first release
But in the end there's no getting away from the fact that the vast majority of software is not developed in isolation. It runs as part of bigger systems that change and develop over time. Oolite has a lot more dependencies than the obvious library deps, for example. It depends on a toolchain, the linux binary format, the OS itself. Those things change as well, albeit slowly. As the developer of an app it would be nice not to have to think about that stuff, but it's unavoidable. Software that isn't kept up-to-date slowly rots until it eventually becomes impossible to run without a lot of archaeological contortions (emulators, special sandbox environments etc). The only real alternative is controlling the whole stack from hardware upwards for your app (i.e. 80s nintendo games!)
What I will say is that since moving to Arch and it's rolling releases, I've had a wider range of software available, that's more up-to-date (usually within a day or less of developer release) and more stable than any other linux distro I've ever tried, bar none. The focus on a single, most-up-to-date version of everything really works wonders. Yes, there are old versions of libraries available for software that needs them. I can install libpng 1.4 from the AUR if I really need to; hell libpng 1.2 is in the community repo (I guess some really popular bit of software depends on it still); but that's the exception rather than the rule, and the overall result of that policy, for me, has been extremely positive!
It's probably worth thinking back to what started all this discussion in the first place - non-compliant PNG files in OXPs. The oolite.org packaged libpng doesn't spew crap into the logs, but that's because it's quietly accepting non-compliant PNGs. I think we can all agree that the *ideal* solution here would be for the PNG files to actually be correct. In this instance, moving to a more up-to-date library is actually helping to bring to light potential problems with bad colour profiles. Perhaps not the biggest issue in the world, granted, but not something that would make me feel like the library developers are being difficult or unhelpful.
As I said in my previous message, if I have a bit more time later this year I'll try and put my time where my mouth is and see what I can do to help. In the meantime I'm going to ignore my own advice and install the oolite.org packages like Getafix told me to (never disobey the druid!)