Having read a lot of these things from ex-Sun employees, I dearly want to read one with the topic my department is the one that fucked up Sun. I don't think I have seen a topic like that yet.
I actually haven't read a ton of these, so would love to be pointed to what you have found. But yes, everyone will be biased, and I am too. I think that the organization that I was in fucked up plenty (when I joined the company in 1996, SunSoft was led by truly one of the most incompetent executives I have ever encountered, a bumbling oaf that Sun kept putting in charge of different business units, letting him asphyxiate them one by one), but the fact that everyone also sees something else wrong probably tells you a few things: everyone at Sun earnestly wanted the company to succeed; Sun did a bunch of stuff right; Sun did a bunch of other stuff wrong.
Ultimately, the leadership of the company needs to bear the responsibility for the outcome -- and I think it's pretty telling that Sun's executive leadership basically went on to do more or less nothing after Sun. I loved Sun -- but it was a breeding ground for engineers, not executive leadership.
I'm reluctant to call anyone out. I liked Sun a lot, as well as the people I've encountered who worked there. There was a thread between a couple of ex-Sun guys here on hn years ago - I can't find it right now - which made me feel there was a humorous and very human pattern in Sun postmortem comments: "man, that other team really killed Sun, we were doomed after that thing they did." I was pretty curious what the people on that other team (the SPARC microprocessor designers, in the case of that comment thread) might have said in their own defense, where they might have spread some of the blame, and so on.
(perhaps it doesn't do your rather full blog post justice to compare it to something considerably less complete like a comment thread)
> Ultimately, the leadership of the company needs to bear the responsibility for the outcome
I emphatically agree. Maybe it goes without saying but with Sun, multiple generations of leadership, I think.
I seriously don't know what Sun could have done to survive - once Linux was stable, running Linux on commodity hardware was just a better solution than anything Sun could offer.
I've never spent much time with Sun equipment, but the hardware and OS had features I haven't seen on commodity hardware with Linux. IIRC, Solaris could deal with an unrecoverable ECC error by killing the process that had the bad memory. And I think they had hot swap memory on some systems.
ZFS was pretty unrivaled for long term storage for a long time. Solaris zones were containers before they were cool. Dtrace is pretty neat. Etc.
Maybe they could have survived with their OpenSolaris push driving low end adoption and high end customers buying their hardware. Pretty sure they would have had to migrate to x86, but they were dabling in that.
Maybe make some low end commodity x86 servers. Sell enterprise a fully engineered big ass database server and a bunch of commodity web servers. Doesn't have to be exceptionally priced, just good enough to let customers have single vendor server purchasing instead of getting the big stuff from sun and the little stuff from dell or hp.
More systems consulting? Sun had great engineers, but you don't see too many war stories about them fixing stuff for customers (maybe they keep quiet though).
> IIRC, Solaris could deal with an unrecoverable ECC error by killing the process that had the bad memory. And I think they had hot swap memory on some systems.
I saw references to Linux support for hotswappable ram a long time ago, but I've yet to run into a commodity server that supports it. And honestly, I've yet to run a system that needed it nor did I have systems where are hard stop from an unrecoverable ECC error was a problem.
I'm not terribly surprised that Linux grew support for poisoning pages eventually (and I appreciate the link!), but I believe Solaris had memory retirement in Solaris 10 (2005), and Linux got page poisoning in 2009. If you were running workloads where it would be important, then Solaris was better for you at that time, unlike the claim from the poster above that
> once Linux was stable, running Linux on commodity hardware was just a better solution than anything Sun could offer.
I think only a curmudgeon would claim Linux wasn't stable in 2005. Although certainly the userland could use less churn. :P
There was (and still is) a bunch of niche Linux kernels. I remember reading about a team that used Xen to do triple-redundancy with voting for Linux. They virtualized _all_ the non-deterministic IO paths (including rdtsc) and observed the outgoing network packets, ensuring that they are completely identical. I think this was around the mid-2000-s timeframe.
But yeah, it was all super-niche. The consensus now is that you should design your systems to be fault-tolerant and self-recovering, rather than depending on perfect hardware functionality. And for everything else IBM still exists.
Sun clearly suffered some from being a bit too early to what would become cloud computing —- didn’t help that their tagline was “the network is the computer”, which IMO was a bit too obtuse, because I didn’t really understand what it meant (sounded like a violation of the 7-layer OSI model lol) until I saw actual cloud computing come about.
But as some sibling comments indicate they already had a lot of the software infra for distributed computing the way it eventually came to be done today, with SOA/microservices, service discovery, or serverless lambdas etc. via frameworks like EJB or JXTA. They were admittedly pretty cumbersome, like the contemporary equivalents COM/DCOM, and made the fundamental mistake of trying to abstract away the fallacies of distributed computing, but also solved most of the common problems involved.
Potentially they could have pivoted to monetizing that software infra even if it got deployed on arbitrary hardware, which may have tided them over until they could spin up their own cloud service, even if it was a fast follow to AWS. However Sun seemed committed to making their expensive servers their primary business model and relegated the software to “loss leader” complements.
If they had just looked at how services were being developed on Linux + commodity servers, they’d have seen they already had offerings in that space that could be monetized.
They should have seen commodity hardware coming and built cheaper hardware themselves. They could have. Other companies were making clone Spark machines for much less so it would have been possible.
I'm willing to bet they couldn't have done that because of the mind-boggling array of 1990s UNIX standards Solaris seemed to support. By the mid 1990s it was clear that the (platform-independent) GNU command line tools were generally superior to what was on Solaris, but presumably they had to keep all the old fossils around to satisfy their long-standing supply / support contracts with industry, defence, and government.
That was my guess back then, and it's my guess now. Someone please correct me if I'm wrong and you know for certain why.
I'm not saying it would've been easy. They could've installed the GNU tools in a different path and left it up to the user to change their preference. They did something similar with putting the BSD tools in "/usr/ucb" (I may have that path wrong.)
They could've also just built a real package manager, left it up to the community to add GNU tools. I remember the Solaris package situation being annoying, having to download crap off FTP and install manually. Or often build from source.
Solaris was pretty bare bones, out of the box. I spent days customizing it. Even early Linux distros, like Slackware, were in better shape.
With the benefit of perfect hindsight, which is a bit silly?
Open source Solaris as early as possible.
Really compete with MIPS, ARM, Motorola, etc. in the embedded space. Don't even try to make it a huge financial win, just get the SPARC architecture and dev tools out there with good Linux support.
Consistently support Solaris x86 and frequently release new generations of Intel and AMD workstation-class hardware.
Partner with TSMC for SPARC production a lot earlier.
Since we're just having fun here: acquire Nvidia in 1999, at which time it was pretty obvious workstation graphics was going to be commodified, and Sun could have bought them with their pocket change.
That doesn't exist. No department is every completely at fault. No department has enough budget to sink a company alone. It is always a lot of departments that don't turn in expected results - often for reasons not in their control.
The fault is always upper management not doing their job. Even if one department really would be that bad (which again they are not), upper management allowed them to be that bad.
Preventing counterfeit boards with janky memory upgrades, or something, from getting out onto the market seems like a potentially worthwhile thing, but also probably a reaction to an imaginary problem. Preventing end users from doing what they want seems like a bad thing, but given how few users are actually capable of doing this upgrade, also probably an imaginary problem.
Given the state of the world I'm a little amazed anyone can muster any outrage about this.
Except it isn't - 8GB boards that started their lives as 1/2GB variants are absolutely on the market.
>but given how few users are actually capable of doing this upgrade, also probably an imaginary problem.
Just remember that in China there are places where you can get your iPhone storage upgraded by someone desoldering the original chip and soldering on a new one, they get the whole process done in 20 minutes. If labour is cheap and the skills high that's what you get.
I don't know where you can buy one now - but when the 8GB model first came out it was instantly out of stock everywhere except for aliexpress at suspicious prices...it's an anecdote so it's not like I can give you % numbers, but I know 2 people who bought them and they were both clearly re-soldered units(they worked fine though, afaik they are still using them).
> The "worst" RCP8.5 / SSP5-8.5-style futures (coal-dominated, ~4.4–4.9 °C) are no longer treated as plausible
That's a strange way of structuring such a comment. The temperature increase and concomitant catastrophes are still on the table, what was considered the plausible worst-case scenario with regard to how early that particular milestone might be reached has (apparently) been pushed back.
Lazily researched comment. Pretty much anyone who isn't in sales can have "engineer" in their title. I'm sure there were plenty of "support engineers" in the Shopify number you cited, and I'm sure there were zero in the GTA and Chrome numbers you cited.
> In retrospect it does seem like in the 1950s-70s the military was willing to fund some stuff that was actually not especially close to applications of killing people.
There's no "seem" about it, what you're saying is true, and it was because of changes in federal law. [0]
Thanks for pointing that out; I was not aware of it.
At the same time, when I helped with research in a cognitive psychology lab in school (much closer to the present than to that law), our PI said our funding came partly from a grant from the US Navy. We worked on Bayesian models of human learning and reasoning -- so "direct or apparent relationship to specific military function" has perhaps at times been interpreted somewhat loosely?
Consider that parts of yoga were stripped out and implemented for its effectiveness in reducing overuse injuries.
Teaching individuals quickly and at scale is a core necessity for militaries. Whether not or it gets used, the US Army usually has access to whatever is near the edge there (as in some unit or officer somewhere was promoting the method, and had the sign offs necessary to disseminate it, but culturally it was not fully put in place).
As a practical matter, I'm not sure the Mansfield amendment affected all agencies equally. My impression is that it affected ARPA (later DARPA) most profoundly. (pretty much everything I've read about this was in the context of ARPA and XEROX Parc so my perception might be a little skewed)
I'd speculate the Navy had perhaps more discretion over how it spent its research funding. For example, they funded polywell fusion research for years.
Having multiple maintained versions isn't a mystery. (Although figuring out which one you want might be; I'm.... 80%... sure you want the highest numbered "RELEASE", not "STABLE", but not 100%).
Nobody asked, but a quarter century ago we used FreeBSD stable with current ports in a small business setting. A mostly unchanging base system but always the most recent user facing software (KDE iirc).
Linux distros only recently started doing this with the rise of flatpaks on top of immutable distros (or Debian stable).
I cannot imagine any reason why somebody would want to use RELEASE for production.
A RELEASE is good for installing FreeBSD on a new computer, or for upgrading from a previous major version of FreeBSD, e.g. from 13 to 14 or from 14 to 15.
After installing a RELEASE, you normally update it to STABLE, before starting to use the computer.
STABLE versions correspond to the long-term-support versions of Linux, i.e. they include only essential back-ported patches, like security patches or bug fixes.
RELEASE are the initial versions, like a Linux x.x.0 version, which may have various problems that are discovered later and corrected in the STABLE versions.
I have been running FreeBSD continuously 24/7 on many servers for more than a quarter of century, and I have always used STABLE on them (after installing RELEASE first, especially when upgrading from an older major version, to minimize the risks of incompatibilities).
Yes, looking at the current handbook, today you are right and my posting was wrong.
However, this is because the policy of FreeBSD has changed. Decades ago, when I started using FreeBSD, STABLE was like I said, the recommended branch for production and frequently it was strictly necessary to update to STABLE because it had important patches missing in RELEASE.
It appears that they have changed this some years ago.
However, I was oblivious to this, because it did not affect me as I do not track automatically their STABLE versions, but I do only some audited updates.
The description of STABLE as a development branch to be followed with caveats, and that using it is an active process with risks you need to mitigate has been in the handbook since at least 1996 (https://freshbsd.org/freebsd/doc/commit/827b1b2b292dd02f9bc9...):
> FreeBSD-stable is our development branch for a more low-key and conservative set of changes intended for our next mainstream release.
..
> If you're a commercial user or someone who puts maximum stability of their FreeBSD system before all other concerns, you should consider tracking <em>stable</em>
..
> Please note that the <em>stable</em> tree endevors, above all, to be fully compilable and stable at all times, but we do occasionally make mistakes (these are still active sources with quickly-transmitted updates, after all)
Indeed, the first step to tracking stable is to join the mailing list:
> <heading>Using FreeBSD-stable</heading>
> <p><enum><item> Join the freebsd-stable mailing list.
So the nature of the branch hasn't really changed, it's more that there's now much less reason to track it - we have regular patch releases, and the ports system no longer only supports CURRENT and STABLE.
> I cannot imagine any reason why somebody would want to use RELEASE for production.
I used FreeBSD at Yahoo and WhatsApp, using RELEASE at both. I'm sure there were some times where some groups were running other than RELEASE (and they were running Yahoo builds anyway), but at both places when I was there, we didn't have habit of upgrading the OS. To my knowledge, none of the servers I ran at Yahoo had an OS update installed, I'm not sure there was a procedure; we would get a server it would have the then latest Y! FreeBSD build, we would install our stuff and go for 3-5 years until the server was sent to recycling. Most of our servers never rebooted.
That pattern doesn't really fly today, lots of kernel security fixes and what nots, so you've got to do updates and reboots. At the time, running a miminal kernel and minimal services meant most security updates were for things not on our machines or could be updated without rebooting or doing a full upgrade cycle.
Using RELEASE makes it easy to understand what host has what, rather than -STABLE from whatever day it was installed. This is pretty handy when you've got a mixed fleet of whatever was current when they were installed.
Towards the end of my time at WhatsApp, I did work on keeping our fleet more current, mostly because we had more servers where they didn't need hardware upgrades for a long time, so they didn't get OS refeshes. Running 4 different major versions is irritating in ways that are most easily addressed by doing the upgrade work.
The only time we ran outside of -RELEASE at WhatsApp was very ocassionally to confirm kernel patches we wanted to upstream, but we didn't have a lot of patches, not all of them were important to upstream, and many of them didn't need a report from CURRENT.
Some releases would have important upgrades that really helped some workloads so we'd push those, but at least I would find those out from reading release notes, not following development closely. Or sometimes we'd find out by accident... if a server lost its disk and we set up the replacement with a newer release and perf was significantly different, we'd try to figure out what changed and if perf was better, we might upgrade the other servers for that workload.
I've been using FreeBSD since 1996, both commercially and as a hobbyist.
Early on, back in the CVS days, I would do a you describe, building STABLE out of /usr/src.
These days, I always use RELEASE and apply patches with freebsd-update.
You're just wrong. FreeBSD stable branches are the development branches from which new minor releases (with a stable ABI) are forked. If you want to be pedantic you want to follow the releng branches for most production deployments (release + security and non-security errata patches). Unless you build from source yourself you the tools (pkgbase, freebsd-update) don't make that distinction visible to the user.
I know a ton of embedded shops that are running x86 (including some actual i386 still) in production, and will likely for years to come. That stuff was bullet proof, cheap and still runs today.
A CDN *cache* server is a special case. Netflix cache appliances can accept the risk of running FreeBSD -CURRENT and upgrading half their fleet to the latest snapshot every ~2 weeks and we thank them for battle testing FreeBSD's active development branch in production. It really helps to reveal regressions (both correctness and performance) early in the subsystems and drivers that matter to their usecase.
Most users will neither be willing nor able to accept the trade-offs that Netflix chose.
reply