Can you make a pitch to a Python/R user to give this a try?
What you’ve built looks very nice and heard nothing but good things about elixir elsewhere, but would take a lot to leave those much more robust ecosystems. Do you hope to grow into that over time? Is there enough in terms of viz, statistical models, and ml to survive?
José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.
My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).
In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:
* Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box
* Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.
Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.
Jose's reply suggests the basics have Elixir equivalents. I can't really speak to that side but I can say the usability story is much much better.
The last time I gave Jupyter notebooks a go it was a full session of installing and updating various Python tools: pip, conda, jupyter then struggling with Python versions. You end up piecing together your own bespoke setup based on other people's outdated bespoke setups you find while searching for your error messages. Maybe that's better now, this was a few years ago. For Livebook it's "download the app and run it." Other options exist and are well documented and straight forward. I set up a livebook server on our k8s dev cluster with a pretty simple Deployment I wrote just from looking at the livebook README notes on docker. We've made livebooks that connect to the elixir app running in a different namespace on the cluster. Very cool.
Once you have Livebook going the `.livemd` file is both version control friendly AND very readable markdown file rather than the big json objects used in `.ipynb`.
For Livebook rebuilding cells is a lot more repeatable. It also does a good job of determining if a cell re-execution is necessary or not if a previous cell is modified which can save you a lot of time. Likewise the dependencies installed are captured at the top so I've never had a problem when sharing a livebook. The other person always gets the same results that I had. I don't remember how it worked for Jupyter but it's really cool to collaborate with someone by both going to the same notebook session. It's like working on the same Google Doc but you are writing and executing code.
Now with the Publish functionality I can see using a livebook to throw together some functionality and share it with non-technical users in your org, while having it backed up to git for posterity.
I avoided Smart Cells for a while because I didn't like the "magic-ness" of the UI hiding what the code was doing, but as Jose has shown in the launch videos this week you can easily see the code they are backed with and replace the cell with the code if you want to take full control. Maybe it was always like that but I didn't realize it at first. They really make setting up stuff very easy without limiting you later on.
I'm pitching it to the data science department of the company I work for (huge insurance company in my Country) next week.
They do a lot of prototyping from CSV/parquet sources in Python and R.
I've waited to show them Livebook because Elixir syntax is somewhat alien to many, but now that the Livebook team has integrated ML models and dataframes (Explorer through polar.rs) as smart cells, I think they have a killer feature in their hands, much like Liveview was for Phoenix framework.
Let's see how it goes, I'm fairly optimistic about it.
That's half the story. The other half is that it's under the influence under an antagonistic authoritarian government. If tiktok were a japanese app, this would play out very differently.
If you take the thread's stated pros (low barrier to entry, experimental scripting workflow) and cons ([not] fast, compilable, no GIL) at face value, it's interesting evidence for what really matters compared to what needs to be merely acceptable for a ML language.
The thing neither of them list which is absolutely enormous is the breadth and quality of boring general purpose stuff in Python. It's a big deal that you can write a production grade web app or api in the same language you do modeling work in.
Personally I disagree that it feels empty. The period it examines has a rich set of defining issues (e.g. post-war trauma, feminism, rise of consumerism, generational shifts, automation) with several character arcs that I found compelling and realistic.
Agreed. It felt far more realistic to me too. The characters in Breaking Bad, Sopranos, etc... most people aren't that complex; for some reason, they fail to hold my focus. Mad Men and The Wire are more realistic to me. (Not surprising, as The Wire is of course based on real life accounts).
It took me giving it a second chance to see past this.
The first time I watched it I didn’t get it, a bunch of office drama, cheating and pettiness mixed in a timepiece. Great visuals but that’s it.
What I understood better the second time(and I suck at putting this into words) is that it’s a show where you have to appreciate the metadata more than the main story. Its about the changes in the times, the contrast between what life seemed to be and what life actually was, the psychological changes as time progressed, the “tough looking but house of cards psychology” if everything at the time is what got me to appreciate it more the second time.
I've always took the view that a show about the birth of consumer society from the point of view of ad executives is a brilliant take. We get to witness the machinery of post war capitalism taking shape. If the same concept was applied to investment bankers in the 80's it could be pretty interesting.
Don is kinda like the past, trying to adapt. Struggling with a new identity, escaping past trauma. I think the character embodies the old fashioned behavior (as himself and Betty refer to themselves often), while Peggy is the youth of the 60, stumbling into an old world and at the same time making it her own because many of the epochal changes resonate with her. Her walk into the 70s ready to shake things up at the end of the series is amazing.
I think the main point of the show was how fast American culture changed in the 1960s rather about the characters. The beginning seasons in particular set in the early 1960s focused a lot on how openly people said racist and sexist things and how all the advertising execs were white men, with women only at the secretary level, but as the decade goes on, women, and finally people of color begin to show up at the executive level and society begins to resemble much more our society. Despite cultural changes since, the 1960s probably were the decade of the most cultural change in the shortest time.
The show also followed several decades of demonization of the changes that happened in the 60s. I sometimes think of Mad Men as making the point that the 60s happened for (good) reasons.
Mad Men's popularity hinged on style over substance. There was a semi-interesting set of stories, but people mainly tuned in to experience an early 60's time capsule.
My mom watched the show because it reminded her of when she started work in an office, up to and including the casual sexual harassment. She wasn't so fond of that part but "that's the way it was back then". Similar to how cops claim The Shield was the most accurate depiction of policing.
> Mad Men's popularity hinged on style over substance
To the contrary, all of the fan base at the time hinged on character developments! Married Pete got Peggy pregnant in the first episode, was anything ever going to come of that? would Don ever stoop so low as to make a pass at Joan? would Roger and Joan's series-long tryst blow up their respective relationships? etc etc
A lot of it is left un-shown, as each season jumps ahead in time. You slowly figure out what happened during those jumps by inference. I thought it was great.
One attribute of the show I greatly appreciated was patience. We don’t get a “real” moment between Don and Joan—obliquely answering the “why not” question—until late in Season 5. It’s immensely more powerful and satisfying as a result. (Not to mention what happens later.)
The Don and Peggy relationship is also a fantastic slow burn.
I felt like none of this really paid off. Peggy has a child, that should be a life-changing event, but it's just ignored for the rest of the show because it's not convenient for where the writers wanted to take her character, I guess? I have to agree with other commenters that the show was stylistically impressive but dramatically hollow.
It is life-changing, and echoes in Peggy’s character and relationships through the entire series, as well as being called out explicitly at half a dozen key moments.
Peggy’s baby is a crystallization of the contrast between her and Don. Don tells her, “It will shock you how much this never happened.” He is always running, ignoring what life repeatedly tries to teach him. Peggy makes the decision to give up her baby the keystone of her character: She could have had Pete and the life of an adoring housewife, instead she embraced her choice and consciously took a different path.
What the writers resisted was the potential melodrama of reintroducing the baby to Peggy’s life in some way. But that would have undermined the moment of Peggy’s choice: It was gone.
That's all very tidy, but what you've described is barely in the show at all.
Until reading your comment my recollection was that she left it with her sister; apparently that's wrong and you can figure out what really happened by piecing together a few different short scenes across several seasons. As someone who watched the show as it came out over 8 years, it was like the baby fell off the face of the earth with no explanation.
The baby did fall off the face of (Peggy’s) earth. She gave it up shortly after it was born. That’s it.
As I said, it comes up again and again. Most directly in “Meditations in an Emergency”, wherein she tells Pete the whole story of his baby and why she gave it up.
So we're just going to pretend that Season 5 of The Wire never happened? Because that story sure stuck out like a sore thumb.
My point is, all shows have their ups and downs. Sopranos/BB/Wire were great and definitely kickstarted the golden age of television, but Mad Men just hits differently and I think I now know why: the visual richness and spectacular screenwriting/character development.
You saw one middle episode of a largely serialized show with no context as to the characters or overarching plot lines and you expected to be satisfied?
That's like flipping to a random chapter in the middle of the Harry Potter series and saying "I don't get what they're doing with the latin and these wands, and what's with the owl? Who's Ron? We never met him in this random chapter, whole book series is just unsatisfying."
The nice thing about bogleheads is it starts pretty simple but scales into the minutia of various tax efficiency techniques, fund selection, real estate, and so on when you need it.
The other thing I like about it is that the forums have lots of detailed personal scenario discussions that are helpful for getting real life examples of the application of the dogma.
Tangential to the discussion in the thread, this is a nice reminder of some of the nice things from the blog era:
- medium form essay that's not SEO optimized
- simple, fast loading websites
- real discussion in the comments
It's too bad distribution for this type of content was never really solved.
Most of the answers here are pointing to things like inefficiencies, short-sightedness, and over engineering, but I'm not seeing the positive side.
The web is an incredible distribution channel. With decent engineering, anyone can make an app that has access to an enormous marketplace with very little capital costs.
The past 20 years of software have been dominated by this distribution model so it's easy to take for granted, but compared to pre-web distribution that involved manufacturing and much larger differences in user platform, it makes sense that the world is trying to squeeze as much software as possible into this model.
While I don't think centralization of media is a good thing, the anti-competitive argument doesn't make sense given the amount of viable competition. Microsoft is probably in 3rd place in console sales, lower if you include PC and define Steam as a separate platform.
If the merger were completed, it would certainly change things, but it wouldn't be any guarantee of Microsoft's dominance, or even a first place finish. So to block the acquisition based on something that might happen seems like a clear case of regulatory overreach.
One of the things that always sort of annoys me about complaints that "management doesn't listen to data (science)" is the lack of awareness they consist of.
It turns out that data work is limited by all the same things every other part of the business is limited by: the need to make quick decisions, institutional imperative, the beliefs of decision makers, the ability to communicate well/influence, and so on.
Having better access or skill with data doesn't give you a pass on these things, despite the suggestions otherwise from laments such as this.
Many thought it was not only possible but inevitable: The Schlieffen plan, French revanchism, general military buildup across Europe, and of course Biskmark's 1888 comment:
"One day the great European War will come out of some damned foolish thing in the Balkans."
You're both wrong. Many people subscribed to each belief.
The economists, globalists and industry people believed that it could never happen. The military, nationalist and political folks figured it would be inevitable, or at very least if it happened they had to win.
Many people subscribed to the argument of the economists and globalists in more democratic countries because it was a comforting illusion the people in the more militaristic autocracies, believed it was inevitable, and incidentally were the nations that hold the most culpability for WW1, Germany, Russia, Austro-Hungarian empire.
Wait a minute why is all of my description starting to sound terribly and horribly familiar to what is going on now?
Seriously those that will continue to posist that war will not come are foolish and don't realizing that the first steps are already in progress with the information war being waged right now through cyberspace.
Note I don't want a war to happen and think it will be horrible and terrible, but all the elements are in place for it to happen. A shifting balance of power into a multipolar world, multiple nations either facing decline or ascendancy, realpolitik becoming the norm in international relations, it all looks very grim unless some very wise, peace loving and capable leaders emerge on the world stage soon.
Or, like the UK, the days of Russian empire are over and the new bi-polar world will be the NATO/India/AUNZ/Asia vs China/Africa.
So another Cold War for the 21st Century if we continue to rely on a mercantilist attitude in a world where networking is more important than some trade links.
No the cold war required a bipolar world for it's stability with MAD being the keystone that held the arch together. That was the only way for it to be stable and why we didn't have a massive war, with all actions being confined to small proxy wars.
We are in a multi-polar world now, Trump pointed out and many people are starting to agree, about whether or not the US really should be so closely aligned with Europe, NATO will probably stick around but it might not be enough. Meanwhile India has happily agreed to buy all the Russian oil that Europe isn't, which is done to spit in the face of the sanctions imposed on Russia. The mutli-polar world right now is US, India, China, EU, and Russia which is still a regional power, each of which have different interests.
The problem is China and the EU are facing huge demographic shortfalls in the next 30 years that will pose existential threats to their society, Russia is in the same boat. The EU is having this problem addressed to a certain extent through immigration but the nationalistic racist attitudes of the Chinese people make this a less palatable option for them. It is likely that the demographic cliff is going to continue to stress Chinese society to the breaking point until it snaps and begins an international incident that could quickly escalate to a global war. The best thing the US could do to preserve it's interests is do whatever we need to to schmooze up to India and cement an alliance with them, as they represent the best regional challenger to China and if they end up on the side of the CCP will cause huge problems as at that point a Bejing-Delhi alliance will be able to exert control over 1/3 > of the world's population. (This assumes they will be able to control all of Southeast Asia through soft and hard power)
it seems that only our instutions, or big chunks of the whole global trade system, want this war; but this this point, a lot of those systems are automated by rules and regulations and a overly complicated network that become inentelligible to the people 'running' it.
nobody that is alive and sentient (in the traditional sense) wants a war... and yet, we all see it looming.
Sounds a lot like how the assassination of the heir appearant of the Austro-Hungarian Empire led to the UK declaring war on Germany for marching through Belgium.
No one wanted the war, many knew it would be horrible, and yet it led to the most nightmarish collective human experience in human history.
What you’ve built looks very nice and heard nothing but good things about elixir elsewhere, but would take a lot to leave those much more robust ecosystems. Do you hope to grow into that over time? Is there enough in terms of viz, statistical models, and ml to survive?