All standards are ultimately controlled by private companies. Even non-profits require funding.
Open source always depended on a viable business model (of one or many companies) that can sustain not just the release, but also an ongoing maintenance of the standard.
The problem with corporate control isn't that they require funding or they are private, the problem is they are motivated first by profit. Sometimes exclusively. So when "what's best" is at odds with "what's profitable", they tend to make the wrong choice.
Take this project for instance. If one day their choice is to forgo all future profits, or to close the source to continue operating, it's very likely they will close the source to continue operating, rather than forgoing profits. We've seen it happen enough to be wary from the project structure alone.
But that has nothing to do with the development, maintenance, or enforcement of the standards, since the corporations have no involvement in any of the standards, and are probably opposed to their existence at all.
It's a great counterexample to "corporate money and influence are required to develop, maintain, and enforce standards", because it shows that it sprang up on its own in the absence of money and has persisted for decades.
There is plenty of AI extensions, but the experience matters. The depth of integration matters. When you execute queries against production warehouses and you make decisions based on the results of AI-generated code, accuracy matters. We had our first demo of an AI agent running in 2 days, it took us another 2 years to build the infrastructure to test it, monitor it, and integrate it into the existing data source.
You'd be surprised how many people collaborate together. Software engineering is solitary, collaboration happens in GitHub. But data analysis is collaborative. We frequently have 300+ people looking at the same notebook at the same time.
.py never worked for data exploration. You need to mix code, text, charts, interactive elements. And then you need to add metadata: comments, references to integrations, auth secrets. There are notebooks that are several pages long with 0 code. We are building a computational medium of the future and that goes beyond a plaintext file, no matter how much we love the simplicity of a plaintext file.
We seriously considered this, but decided against this. While elegant for demo projects, it doesn't scale for serious deployments. You still need to deal with secrets, metadata (lots of it), backwards-compatibility, and extensibility (we have 23 block types today, many more to come).
Didn't expect to see this trending here! We worked hard to execute on our vision of a data notebook and I'm glad we finally got a chance to open source it. We stand on the shoulders of giants. AMA!
why is your marketing team a bunch of bumbling morons who decided to not only shit on an open source project but also declare your product a "successor" of said project? as if you have any ties to the jupyter project that would allow you to say these things?
With no annotations of what was changed or acknowledgment of that things were even changed.
I read it originally when it came out, didnt have a chance to respond initially then came back to look through the comments. If there was anything was changed, it's not obviously apparent...
You’re saying vector DBs are the wrong abstraction, but companies keep throwing money at them. Why? Are they just slow to catch on, or are there legit cases where vectors actually make sense?
Founder of TopK here. There are legit use cases for vector-based retrieval (e.g. semantic search, recommendations, multi-modal search, etc.) but that only requires supporting vectors as a data type, not building the whole database around vectors as a first-class citizen (which is what vector DBs do). In practice, you also want to combine multiple vectors, text filters, and metadata alongside custom scoring functions to optimize relevance in your domain, which is not possible with a database built around a vector index.
Hey there, CEO of Deepnote here. It looks like we are thinking about this very similarly, as all 3 points are something that we are already doing or will be shipping in the coming weeks. Either way, good luck with Briefer and happy to chat about our learnings building all of these things.
$50 for 50GB seems like a fair offer for the US market, especially given the possibility of coverage in remote locations.
But keep in mind that US market is unique, as it's extremely overpriced with very poor quality/coverage. Looks like average price in the US is $6.00/GB. Compare this to countries like Israel ($0.02/GB) or Colombia ($0.20/GB). Whenever I travel, I usually have a better coverage and faster connection in a jungle than downtown SF.
I too often have a better connection in rural America than downtown.
Rural America is largely free of interference. Even if you have a weak signal, the tower and your phone can hear each other very clearly. In a downtown, it gets messy.
For mobile - I currently pay $330 NZD for 2 phones for 15GB/mo which turns out to be $8.4 USD per month. You do have to be look out for such deal tho (Kogan Mobile).
Deprioritised starlink is $80 nzd / month which is cheaper than fiber or vdsl and on par with "unlimited" 4G/5G.
Mini is quite good option for boating (campsites usually are well connected here) provided it can cope with swell like bigger dishes do.
Edit: Strange they haven't integrated WiFi HaLow yet since it's primary target is rural customers who would benefit a ton from 1km radius signal...
Germany has no newly educated technical personal, has no place to store the used fuel, and a majority of the public doesn‘t want it due to the risks in this densely populated country.
We have build a lot of solar and wind though. And more is coming after years of right leaning parties blocking wind.
As far as I know it‘s cheaper to build than nuclear, isn‘t it?
According to official statistics [1], in 2022 Germany produced 33% of its electricy from coal (up from 30% coal in 2021). Of this coal, one third is brown coal. Judging from these numbers, the situation in Germany is a catastrophy from an ecological standpoint. Any german green that acted agains nuclear in Germany should feel deeply ashamed for what was achieved in the country. A green catastrophy in a country that could have been a leader! A national shame, I say.
Oh come on. Germans have developed a completely irrational fear of nuclear energy, and everything is just a result of this.
Driving cost of nuclear power plants up as much as possible was a political decision. Not finding a permanent storage was a political decision.
Not having any personell was a poltical decision. What do you expect if you shut down modern and perfectly working power plants, and stop research in nuclear power?
Alternative energy is more expensive, as you can see on your energy bill. I am paying 45€ per khW. How come electricity is so much cheaper in France?
The modern Advanced CANDU reactor in fact does use moderately enriched uranium. The cost of enrichment has fallen greatly since the original designs, so the benefit of using natural uranium has mostly dissipated, and enrichment provides some operational benefits.
But the German government has stopped ALL research in nuclear power usage. Even if there was a safe and cheap solition that would not make us dependent from Russia, Germany would not find it because they decided to not even try. It is an absolute disgrace.
The German federal government is funding Wendelstein 7-X, a large, experimental fusion reactor of the stellerator type, to a high degree (9/10 of 80% = 72% as it seems?), so this statement cannot be true in general (maybe you are thinking only of nuclear fission research?).
Stellarators (like tokamaks) have terrible volumetric power density, so it's really doubtful they can ever be made economically competitive, even with fission as it currently exists.
Solar with battery storage seems to have passed nuclear on price 4 years ago, when it was 2-3x more expensive than it is today and perhaps 10x more expensive than it will be before a new nuclear plant can be finished.
If that's true (and I see no reason to doubt it) Germany will make the cheapest energy in the EU by a country mile in just a few years. Unfortunately, they are bound by treaties to cover the cost of nuclear in other countries, so it won't help them as much as it could.
The main problem for solar+storage is that centralized, government-controlled energy production is a huge cash cow for the state. In the EU, the price of electricity gives the governments extra income both when they sell electricity and when they tax the consumption. They can basically drain money from the population at will.
I see this declaration as a step to keep things that way, it just makes no sense otherwise.
I was told at one point that storage in Germany is inhibited because an entity wanting to operate grid-connected storage had to pay tax on the energy used to charge the storage (and then the output would be taxed again by whoever ultimately consumed it.)
Dunkelflauten (and seasonal leveling) are covered by use of an e-fuel like hydrogen.
You can see this effect in action at the optimization/simulation site https://model.energy/
Go there, solve for Germany (2011 weather data, 2030 cost assumptions), then disable hydrogen and try again. The optimum cost nearly doubles.
Interestingly, this simulation also suggests a 95% cost increase if Germany tried to do it without e-fuel/power-to-X (which it estimates would need to cover about 8% of demand)
As it stands now it will be a disastrous experiment which will (and already has) affect(ed) the rest of north-western Europe due to the interconnected nature of the electricity network. Electricity prices in the lower half of Sweden have risen dramatically due to this and also due to the fact that our own 'progressive/green' politicos took down half the nuclear generation capacity based on ideological reasoning.
This poses a number of questions:
- will these ideologically driven apparatchiks ever be held accountable in some way or will they just glide through the promotion circus and end up in cushy positions as ambassador in some warm country, in some UN organisation or as head of some NGO or (like Schroeder [1]) in the board of Gazprom or some similar organisation?
- if Germany continues to de-industrialise due to a shortage of affordable power it will only be harder to reach that pie in the sky called the H₂-based economy - can this downward spiral be halted in some way?
- how does a country's responsibility to help stabilise the European grid interact with another country's irresponsible experimentation with that stability?
While the climate-apostles have boarded their private jets for yet another posh gathering the temperature has steadily dropped, the land is white and frozen and electricity prices have risen up to tenfold. Gas prices are still low but that does not help for those who listened to the apostles and replaced their gas-burning central heating for an electric heat pump. Solar does not help either when snow covers the panels which look out over a steely-gray snow-laden sky (source: I just looked out of the window) nor does wind (source: same the before).
In a post-fossil fuel age, Germany is fucked energywise anyway compared to sun-drenched lower latitude countries. Unless nuclear in Germany can compete with solar in these places (competing with solar in Germany is not enough), German energy-intensive industries will operate at a great disadvantage.
That is not correct. Once the energy cost goes below a certain level, it simply is no longer important compared to other factors like human resources, tax, infrastructure,... .
Example: If the kWh/h costs 2 cent in Germany and only 1 cent in country X, that 100% difference would not matter, even for the most energy hungry industries.
Renewable energy has to get cheaper than it currently is before it becomes less of an economic input than fossil fuels currently are, for many industrial processes. For example, in production of iron renewable energy is competing against the raw chemical energy of coke.
> In a post-fossil fuel age, Germany is fucked energywise anyway compared to sun-drenched lower latitude countries
Assuming that the 'greens' either get the boot or a revelation which leads to new nuclear power stations being built in Germany I'd say they'll be more than fine at night while during the day they'll be able to use their installed and yet to be installed solar capacity in addition to their base load capacity.
If and when the long-term energy storage problem is solved in an actually useable way [1] this can change but until that time any country or region intent on going 'fully renewable' will need a backup base load capacity for those times the sun and wind are absent. The former happens predictably every night and less predictably when the skies are overcast, the latter is as predictable as the weather. That base load can - assuming that 'non-renewable/non-nuclear' sources are out - be nuclear or (possibly pumped) hydroelectric. If the geography allows for (pumped) hydro and if the local 'greens' do not get laws (re)written to block the establishment of such infrastructure that would offer a long-term solution. Realistically speaking most places which allow for large-scale hydroelectric facilities probably already have them. This leaves nuclear power... which is expensive to establish but cheap to run. This combination of facts makes nuclear a good base load provider since running a nuclear power plant at or near maximum capacity does not add much to the expenditures but means the investment in building the plant and the related processing/waste storage facilities is recuperated sooner. Adding solar and wind to the mix will lead to an excess in power during the times these sources produce which will necessitate lowering the output of the nuclear plant which in turn means the investment in building it will take longer before the investment is recuperated.
[1] something like 'e-fuels' using captured CO₂ and generated H₂ to create synthetic hydrocarbons - possible but extremely energy-inefficient - or some yet to be developed process
The proposal I've heard for using CO2 to make e-fuels for grid storage would work like this:
1) Electrolyze water to H2 and O2.
2) Store the O2 (either as compressed gas or LOX).
3) Use the H2 with CO2 to make hydrocarbons or other fuels (methanol?).
4) The fuel is eventually burned in Allam cycle turbines, using the stored oxygen to burn the fuel in compressed CO2 (oxyfuel combustion). The produced CO2 of combustion comes off with water and is easily separated for storage for use in step (3).
This still has storage (oxygen, CO2, and the produced fuel) but these may be easier than storing hydrogen in locations without suitable geology.
Green H2 is projected to get cheap enough that a combined cycle plant running as a base load source on green hydrogen would be economically superior to a new construction nuclear power plant. Of course, a 100% RE grid needn't operate any hydrogen plant in base load mode, only to fill in gaps not covered by direct use of solar, wind, and batteries. Very little H2 is actually needed to cover dunkelflauten; somewhat more might be useful for seasonal leveling.
Open source always depended on a viable business model (of one or many companies) that can sustain not just the release, but also an ongoing maintenance of the standard.