Hacker Newsnew | past | comments | ask | show | jobs | submit | more BiteCode_dev's commentslogin

Not only this will speed up comprehensions (which are already faster than map/filter), but it will likely solve this annoying weird edge case with the scoping of lambdas in comprehensions.

Last but not least, since it removes the additional frame that the comprensions used be wrapped in, it will fix the way you interact with them using pdb, which has always been rough thanks to this invisible indirection layer.

I assume JIT will also appreciate the change, so let's see if pypy moves on this.

Migrating to a PEG parser opened so many opportunities for Python. I never expected we would see so many good things come out of it, after years of defending LL1 to keep the reference implementation simple. E.G: the new error messages are awesome.


> Not only this will speed up comprehensions (which are already faster than map/filter)

As someone coming from Haskell I find it odd that comprehensions are prioritized over generic maps (fmaps) and traversals in Python. Not only they are more verbose syntax-wise and don't allow for partial application, they also discourage people from discovering generators, as most of the tutorials go with [] and {} brackets instead of the lazy (). At least with map() people could get acquainted with generators earlier.

> I assume JIT will also appreciate the change, so let's see if pypy moves on this.

I wonder if there's a real difference between map and comprehensions performance when it comes to JIT inlining.


Guido hates higher-order functions. Once upon a time there was map, filter and reduce as builtins. Then for Python 3, they tried to remove all three, but failed due to community pushback. reduce() did get banished however.

The argument is that making it hard to use higher-order functions discourages point-free style. "Explicit is better than implicit" is a design principle of the language.


"Explicit is better than implicit" seems vacuous in the context of Higher Order Functions and it's applied functions.

It's pretty explicit. Nothing is implied, it only can work one way.


Well you can see Guido in his own words at https://www.artima.com/weblogs/viewpost.jsp?thread=98196.

The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. As a result people have created arbitrary terminology for it like "lambda" and "reduce". When you've mastered the terminology it may seem like a good way of doing things. But there are easy alternatives for doing the same things that DON'T require mastering that terminology.

Python has a number of guiding principles, https://en.wikipedia.org/wiki/Zen_of_Python has a list, but one of them is, "There should be one– and preferably only one –obvious way to do it." And higher order functions do not win in Guido's mind.

Separately, while the operation of higher order functions is explicit, code that makes heavy use of them tends to make behavior implicit and dependent on setup elsewhere. I have heard Guido complain about that exact style of coding. I have grudging sympathy for his view. Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing.


> The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking.

and what is common sense thinking here? are you talking about python or just programming in general?

> Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing

Citations needed. I'm not sure who WE is in this context. Python is an imperative lang and doesn't even do OO that well, let alone FP. but, if you are talking about programming in general, I do find it to be much easier to hunt down bugs in languages where the design philosophy of the language favors functional programing ergonomics.

There's even been studies on this. https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-...

tl;dr: "For those with positive coefficients we can expect that the language is associated with a greater number of defect fixes. These languages include C, C++, Objective-C, Php, and Python. The languages Clojure, Haskell, Ruby, and Scala, all have negative coefficients implying that these languages are less likely than average to result in defect fixing commits."

> As a result people have created arbitrary terminology for it like "lambda" and "reduce".

But Python does exactly that (I'm not knocking it, terminology is important, but its all arbitrary)

In Python, we have "for" loops, we have "while" loops for lists and tuples(+ special syntax for "for" loops, when looping over a list of tuples!).

Any sufficient digging into "for loops" and their implementation, and we begin to realize it's all just iterators under the hood completely hidden from the developer, but I digress. We have a special syntax for list comprehensions, (special syntax for nested comprehensions) that extend to ranges. There's special syntax in the form of specific statements that exist only in the context of a loop; "continue", "break", "pass", etc.

There's all the implicit state passing into the bodies of these loops, and there's no referential transparency.

"map", "reduce", and "filter" are specific and have no special constructs, no magic, it does what it says on the box. I'd argue it has a very similar mental overhead to a for loop.

This is my experience, YMMV.


About common sense thinking.

The phrase "for x in y" has an immediate common sense meaning in English. What it means in Python works out to that. Sure, you find a lot of complexity under the hood when you dive into the implementation. But the whole point of programming languages is to allow the programmer to think in terms of human concepts rather than machine implementations. And the concept maps directly to one that beginning programmers already have.

By contrast I've yet to meet any non-programmer who has an intuition about what "map", "reduce", and "lambda" should mean. As great as they are as organizing concepts, they are something that programmers need to learn and gain experience with before it makes sense. For a programmer who has completely internalized the concepts, it may well be similar mental overhead to a for loop. But most programmers have NOT done so. And beginning programmers find for loops a heck of a lot easier to learn - because it corresponds to something that already makes sense to them.

And the implementation is less "completely hidden" than you claim. Heck, the official tutorial at https://docs.python.org/3/tutorial/ includes https://docs.python.org/3/tutorial/classes.html#iterators which shows you exactly how to poke under the hood and see how it works.

As for my comment about spending more time debugging than writing, that should be obvious. As https://www.scirp.org/journal/paperinformation.aspx?paperid=... notes, about 70% of total software costs come in the maintenance phase. Therefore as developers we spend over 2x the time maintaining software rather than developing it. The actual ratio varies by project size, industry, language, tooling, developer experience and so on. But that's a pretty good average. As your link notes, the contribution due to language is real, but modest.

My comment about functional programming being hard to maintain is based on anecdotal experience. I find my own functional code easy to maintain. But others have not. And my experience picking up other people's code has been at best mixed.

Oh finally, about Python, comments like "doesn't do OO very well" usually mean "doesn't do OO like I wish it did". I find them to be highly individual opinions, and not apples to apples when you compare people.


One more thing.

If you want to know why Python looks the way it does, you need to get into Guido's mind. You may disagree with him, but your disagreements will not help you understand why Python looks the way it looks.

And before arguing that he's wrong to think that, remember that for several years now, Python has added more developers in a year than Clojure, Haskell, Ruby, and Scala have *COMBINED*. So while he may be wrong-headed, he's wrong-headed in a way that works for a lot of people.

(Note, I think he's wrong-headed about a lot of things.)


You don't explicitly pass the arguments to the function, is the point. I am not sure I agree with it but it's a thing to consider.


AFAICT Guido has softened his stance on this somewhat. The main impediment to `map()` and friends at this point is the clunky `lambda` syntax. However here we see another benefit of syntactic comprehensions over higher-order functions: comprehensions can be inspected deeply and inlined or otherwise optimized as a special case, while a stack of higher-order functions might need a more advanced bytecode compiler or JIT to apply "deforesting" and inlining.


> don't allow for partial application

Unfortunately, partial application doesn't work in Python, the same way it works in Haskell. If a function has 3 args, calling `f(x)` is a guaranteed runtime `TypeError`. There is functools.partial to hack it into the language for cases where you absolutely need it but overall it's bad developer experience. In fact, when I do need to partially apply `f(x, y, z)` I much prefer doing `g = lambda y, z: f(fixed_x, y, z)` because it's clearer than using functools.partial.


I don't agree that the `lambda` is clearer than `partial`. This is a matter of personal style and preference, and IMO the top priority in cases like this is consistency within the codebase.

You can also refactor partial function applications into callable classes depending on your needs, which is basically what `functools.partial` does internally anyway.


Comprehensions vs. map/filter/etc. is a style thing.

If you think in terms of an expression language that composes functions, then map and filter are more natural.

If you think in terms of a statement language with special syntax for computation expressions, then comprehensions are more natural.

Even in Racket (a scheme), idiomatic code seems to prefer the `for/thing` family of macros, which have a variety of flavors and keyword arguments to support map, filter, reduce, mapcar, etc.

Whatever floats your boat.


So, out of curiosity: what does the PEG parser have to do with this particular PEP? It seems like this was just a somewhat inefficient (but simple) implementation choice from the start that is now being improved.


I don't know that it matters here specifically. But I think it was a kind of watershed moment in terms of how the development approach to CPython has changed.


Are you sure about the lambda scope bit? It seems to me there will still be only one loop variable, which gets re-used throughout the loop (unlike, for example, the 'do' loop in Scheme, where each turn through the loop explicitly gets new locations for all variables).


Given how terrible NMEA formats are, this is fantastic.


Also, abuse complaints are themselves abused so much that it's easily to want to tell the complainer to go to hell.

Running 0bin.net, we have complaints from outside of our jurisdiction, false claims, idiotic claims, DMCA used for censorship, and so on.

Once in a while we have a serious one, like CP or doxxing, so we read everything.

But honestly, our hosting is forcing us to delete so much content based of BS complaints it's tiring. And if we refuse, we have to make our case, which unlike complaints, cannot be automatized. 9 times out of 10, the hosting tells us we should delete the content anyway, and if we wait to much, proceed to cut the power cord until we comply.

I'm sure Meta does report some real website and phishing attempts, but I'm certain they also abuse their power like the GAFAM they are.


What's amazing about all it is that HP being hostile to its own customers is nothing new:

- In 2005, they had one printer that refused to print if the cartridge was past a certain date, even if full. Several also underestimated the volume of ink in cartridges and told you it was empty when not.

- In 2007, HP refused to honor hardware warranty if you install any other OS on their laptop.

- in 2010, HP refused to provide drivers for the new windows Vista for its old ScanJet scanners (linux worked fine though).

- in 2020 they hardcoded the path to the EFI of their laptop to be windows only.

That's just the examples I could easily google, I remember that HP was having one scandal every 6 months in 2000, and we didn't have twitter back then.

So if a brand has been very publicly misbehaving for 2 decades, how does it still have customers?

Well, Facebook has still a billion users. Microsoft is now considered the father of unicorns and rainbows. Oracle is still making banks, one of my biggest client just migrated to it.

So the so-called cancel culture seems very superficial to me: lots of noise, but very actual consumer behavioral change. The brands can get away with anything and thrive. Maybe they'll get a little bit of heat for a few months on social media, so what?

But the bottom line is: if you are a big company, just do whatever makes you money. You don't need reputation.


A client of yours migrated TO Oracle in 2023?

What are they smoking


I think you drastically underestimate Oracle and SAP in the enterprise world.

There’s not many companies that can provide all the mundane business management tools for a huge variety of business like an ATM manufacturer, an oil and gas company, a massive hotel chain, or a financial services company. The tech is not sexy by any means but it can run businesses from top to bottom.


Provided you hire an entire team of consultants for years until they change all your workflow to adapt to SAP/Oracle, then code specific routines for each role in your org because their tool cannot do most of what they say it does.

At this point, you have a completely custom stack that could have been developed entirely for half the price, that would run faster, without licensing cost. Money you could invest in hiring a company that will actually provide you support instead sending you interns they bill as experts.


> At this point, you have a completely custom stack that could have been developed entirely for half the price, that would run faster, without licensing cost. Money you could invest in hiring a company that will actually provide you support instead sending you interns they bill as experts.

This isn’t true for a majority of companies outside of FAANG otherwise the majority of companies would dump Oracle and SAP. A lot of executives hate them with passion and would love to dump Oracles horrible licensing models. They pay for CYA and the ability to point a finger at the vendor instead of themselves.

I went though a merger between two Fortune 500 financial services companies. One had dumped vendors 5 years prior and they were foaming at the mouth to get off all of their homegrown apps. The reality is a lot of companies are going to off shore this type of development and support to SE Asia for bottom dollar body shops.

The consultant thing is absolutely true for enterprise software, not just ERPs. I’ve seen it with Salesforce, Workday, ServiceNow, ERP/Oracle. Pretty much anything KPMG or Accenture are building.

However if you get top tier consultant lift companies or professionally services from the SAAS vendor it is much less likely to be greenhorn college grads.


> They pay for CYA and the ability to point a finger at the vendor instead of themselves.

That's a very good point. Nobody every been fired to buy IBM is still a stong motto as ever, except for the IBM part.


Not what. Whom with.


>in 2020 they hardcoded the path to the EFI of their laptop to be windows only.

So you can't install Linux at all on HPs post-2020?


cp /boot/efi/EFI/ubuntu/grubx64.efi /boot/efi/EFI/Microsoft/Boot/bootmgfw.efi 1>/dev/null 2>&1

cp /boot/efi/EFI/ubuntu/grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi 1>/dev/null 2>&1

:D


Unlike most of HN I am not a "computer person". I know not to click on suspicious links and that's about all that can be said of me. So I'm not sure if you're giving me a hint, or doing the equivalent of telling me to delete System32!


They explained how to trick HP laptops with hard-coded EFI paths into booting Ubuntu anyway. This will overwrite the Windows EFI, so only try this at home if you know what you are doing.


I probably wouldn't know but I appreciate having the resource!


Fresh fruit, vegetables, dried whole grain and beans, fresh meat and fish, fresh seashells, whole nuts, water.

It's not that it's hard to understand, it's just hard to stick to it.


+1. "Let me go back to you" and "i don't know" are sentences people should use more IMO.


Is ruff a good replacement for pylint yet ? It replaces flake8 well but since I use black and pylint, I don't need flake8. However, pylint does a lot so if it does replace it, even without plugins, that would be amazing.


ruff is exceptionally fast and capable, but doesn't handle all constructs, match/case are not handled correctly yet for example


Pylint value, unlike flake8, is not in checking formatting but rather naming conventions, code complexity, and basic code sanity.

Do you use pylint ? If yes what are the important pylint checks you missed in ruff?


Match is now supported (as of last week) I believe


wow nice


Pyenv compiles python. This excludes it for pretty much every solution by default, as compiling can fail in so many weird ways.

Nobody wants to support something that can turn even the simplest ticket into a game of cluedo.


After 10 years using all of them, in about 50 companies (as a freelancer) with various constraints, contexts, systems and skillsets involved, here are my definitive conclusions:

- stick to regular python installers from python.org if you can. Don't use homebrew, pyenv, conda, the windows store, asdf, etc. They have subtle and least subtle variations that will bite you.

- always use "-m" when you run a python tool, even in venv. This will let you be explicit about which python you use. On windows, use the "py" command to create the venv. No need for pyenv.

- if you are in linux, research how to install python seriously. If the command is simple, you probably got it wrong. E.g: Ubuntu has 3 packages to install, not one. People giving you a one true way are lying to you. Pyenv is a trap.

- learn to use basic venv and pip. Never do something outside of a venv, even for small scripts. You can have one big venv for all your small scripts. Don't even install black outside of a venv. You can get very far with just that. Forget about pipx.

- if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not flint. Not pdm. Not anything else. They will fail you. Pip-tools is just good enough and just reliable enough. Embrace manual venv handling. It's not a big deal. Automated tools will betray you.

- learn to be comfortable with the PATH and the PYTHONPATH. No matter what you do, something may need tinkering.

- tox is indeed the best tool to test several envs, but people that need it are rare. Don't bother for now.

- if you want to build a package, use hatch. But you probably don't.

- if you have to use conda, don't use pip or venv at all. Don't use piptool, poetry or anything. Stay on pure conda land. It sucks, but the alternative of mixing them is worse.


Couple of points/expansions

>if you are in linux, research how to install python seriously.

I.e, build from source, after installing necessary dependencies (https://devguide.python.org/getting-started/setup-building/#...). Use sudo make altinstall to install the specific version. Usually distro provided ones are pretty old anyways, so you want the faster 3.11 and on.

>Never do something outside of a venv. Not even install black. You can get very far with just that.

If you want to just write scripts to test things, working outside venv is fine as long as you manually install pip using get-pip.py, then invoke pip using python3.xx -m pip (make a bash alias for this command, and then delete the binaries in ~/.local/bin). This will install all the binaries to ~/.local/lib/python3.xx/site-packages, which keeps everything isolated.

>if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not anything else.

Also, stick to setup.py for module installation definition. It keeps everything in one place and allows you to run custom workflows. For example, if you wanna run flake8 on install to check your code, with pyproject.toml, it requires pyproject.toml and setup.cfg. With setup.py, you can do this all in one file. Not only that, it gives you extra options to run certain commands.

>if you want to build a package, use hatch.

setuptools is fine for building a wheel - all you are doing is moving files around and zipping them.

As far as OS goes, if you are on Windows, use WSL2 and run python there. Don't even bother with Windows python. If you are on mac, use `asdf` for python installation.


I would advice against all of those recommandations. It would be very long to explain point by point, but in short, my experience is that if someone is asking OP's question, they don't have the experience to deal with the modes of failure of what you are proposing

There is not a lot of things I'm very adament about.

Yet when it comes to python env setup, I have a strong opinion on the matter, built from seing hundreds of people failing at it in a thousand ways.

Looking for the best solution will not work.

You have to go to the minimal one that does the job, which have the least possible ways to fail. And still work in 5 years.

Most people like you, giving advices on HN, know too much about it to realize you solve lots of problems without even thinking when you perform your workflow.

But the average python user don't have this knowledge. They just want to get things done and have something that works.

I am very confident in these claims, although a HN comment is not the best format to demonstrate it.

I should write an article on it honestly at this point because so many devs are suffering for this.

A new team has been put in place to work on packaging officially in python. But the fruits will take years to mature.


Can you give me at least one of the points that you disagree with and reasons why?


The first one, build from sources, is incredibly easy to get wrong.

You may end up with an incomplete python without proper tkinter, ssl or zip support. Sometimes it will look like it works, only to fail later down the line when you use some part of openssl that requests don't exercice often or when you install numpy/scipy wheels. Not to mention someone will do "make install" instead of "altinstall" and break their machine with a 3 letters mistake since apt or rpm python will get erased.

If you wonder how to setup a python venv, you should not compile anything (so no oyenv neiter). In the same way that if you wonder what oil to put in your car, you should not attempt to fix your breaks.

The fact you suggested it underlines my point that you are too experienced with the stack to give advices that would be useful for most confused devs in python. A C# dev that starts python just wants, and needs, one reliable happy path.

In fact, it's part of the problem. Paradox of choice, with too many solutions that won't fit the average user context.

Funnily, now that I get close to 20 years of python, I find myself going back to the beginner stuff.


What's with the push to encourage people not to learn? Get em started, let em figure the rest out as they run into it. You aren't gonna be here forever.

The test of a tool is learnability.


Of they will learn.

But instead of having to learn about how to setup python, they will spend the same time learning on how to solve their problem with python.

Given that days have only 24 hours, the latter is more attractive to a lot of people.


>You may end up with an incomplete python without proper tkinter, ssl or zip support.

I specifically said after installing dependencies, which is mentioned in the documentation.


The documentation doesn't, and cannot, mention how to install said dependencies for each distro and version. It is non trivial, and hand waving that is like the "draw the rest of the owl" meme.

Even if the 10 most popular distros were documented, with all their supported versions, and if it covered windows and mac versions as well, and dealt with people in corporation not having admin rights on their machine, you'd still have so many steps a human error will be more likely to be inserted than just use what I suggested.

Again showing that you are so good at dealing with complexity you are not even seing it.

The only way I could convince you at this point would be to make you train 10 different teams, of 10 different people in 10 different contexts and give you one million of dollar if after one year you come back to see that their department has this stuff streamlined.

But I can't do that so that will have to be the end of this thread.


The really insidious one is:

> If you want to just write scripts to test things ...

So often "just scripts" evolve into long term projects, and, due to time pressure etc. never free themselves of bad foundations.


How do you handle multiple python versions? I have found pyenv the simplest cross system approach


On mac and linux, python executables are suffixed, on windows, use the py command (hence using the official installer).

Once you have created the venv, it doesn't matter.


Similar experience, but disagree with almost everything you have said.

- stick to regular python installers. Don't use homebrew, pyenv, conda, the windows store, etc.

For me pyenv has been a lifesaver, easier to manage multiple versions. Are you manually managing multiple python versions?

- always use "-m" when you run a python tool, even in venv. This will let you be explicit about which python you use. On windows, use the "py" command to create the venv.

Unsure how I feel about this, I do normally point direct to the binary being used rather than using "-m", Although "-m" still requires you to be in the correct venv. How does it allow you to be specific about the python you use though?

- if you are in linux, research how to install python seriously. If the command is simple, you probably got it wrong. E.g: Ubuntu has 3 packages to install, not one. People giving you a one true way are lying to you. Pyenv is a trap.

poetry has been another lifesaver that removes the boring writing of a setup.{py,cfg}. Pretty much everything can be managed through it, and it feels like a legit build tool for python. No more broken requirements because of a differing versions in transitive dependencies.

- learn to use basic venv and pip. Never do something outside of a venv. Not even install black. You can get very far with just that.

Can't disagree with this, although poetry will manage your venvs for you. Understanding venv and pip is useful, but not fundamental. Greatest success for devs has been getting them away from manually managing things, by and large perm staff don't care about learning lots of tools well, it's just a job.

- if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not flint. Not pdm. Not anything else. They will fail you. Pip-tools is just good enough and just reliable enough. Embrace manual venv handling. It's not a big deal. Automated tools will betray you.

lumping poetry in with pip-tools is weird. it's not a replacement, it's a tool with greater scope than pip-tools. pip-tools provides a subset of poetry functionality i.e. dependency management.

Manual handling is useful to learn, but is laborious after a while, hence tools such as poetry.

- learn to be comfortable with the PATH and the PYTHONPATH. No matter what you do, something may need tinkering.

This is a really interesting point, normally modifying PYTHONPATH is a smell in my experience, but yes is good to understand. Understanding the PATH in general is more useful as a dev than specific to python imo.

- tox is indeed the best tool to test several envs, but people that need it are rare. Don't bother for now.

Agreed, its a nice thing to do, but rarely do I see it unless you are building libs, and need to support multiple versions.

- if you want to build a package, use hatch. But you probably don't.

Why would you use hatch over a tool such as poetry, are they equivalent? I'm interested in exploring hatch, but the comparisons I have found seem to be fundamentally misinformed about poetry so its hard to take comparison seriously.

- if you have to use conda, don't use pip or venv at all. Don't use piptool, poetry or anything. Stay on pure conda land. It sucks, but the alternative of mixing them is worse.

This is a really good point, but is always a frustrating thing to try and solve time and time again, but still encounter problems.


If you recommend pyenv, it's game over, I will never been able to convince you otherwise.

Because it's a sure sign that you have not had to help a team of geographers in a school or quants in banks or small kids.

Your method will only work for a very small subset of python users, and you don't have met enough of them to even know what error they encounter.

It's ok, you are a skilled dev, you offer skilled devs advices.

But no, pyenv has too many ways to fail for the average python users. Espacially since the "py" command on windows and suffixed python on mac and linux works fine to deal with multiple versions.

If would be too long to debate each point so I won't. But I don't make those claims out of the will to be right or contradict people.

I worked with python since 2.4, in dozens in countries. In africa. In asia. In the silicon valley.

It gave me a huge sample to study.


Exactly. People keep telling how much ubuntu sucks and how they are now using x.

But eventually, my mother is still using it after 10 years.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: