Is there really such a thing as an abstract dependency?

There is such a thing as a tested system (ie two or more Independantly built packages that have been demonstrated to work together). This is the concrete - I have built these things and tested them - now take them both as they are and put them live. I now dislikes the idea that I will build packages (with pip literally from source) on production machines.

No I don't quite get this - dependencies are just that - we depend on the code they provide to run. A library author cares as much as a sysadmin that they avoid a version of dependency that breaks the package.

A better spec might be

    Requests=1.1,1.2,1.4  #oops 1.3 broke our tests

And yes, requirements.txt and setup.py conflict. We don't have a clean packaging system in Python and we should accept it and fix it.

None of this Precludes the useful ability to specify which location to retrieve which package from, but these are build instructions, not deployment instructions.

I call them abstract for lack of a better word. Referenced by name?

It's not about providing versions of code that does or doesn't pass tests. It's about where that code is fetched from. If you reference the dependency by name then you can easily switch it out for something that looks like the code the author originally developed against, but actually has some changes (patches, bugfix, whatever).

Where to retrieve packages from is both a build and a deployment instruction. After all you need to fetch the built packages from soemwhere yes? In general "reference by name and version/abstract" allows you to easily swap in a built package for a source package as well. Otherwise you'd be stuck with whatever formats the author published because the dependencies would be pointing to a specific url (likely outside your control).

The very fact you can build a binary package and host it yourself works because of what I call abstract dependencies (for lack of a better name). If they wern't abstract then you'd have to fork the dependent package in order to point it at your built package/repository instead.

OK, the concept of "abstract dependencies" makes sense, but boy is it a complicated way to fix an edge case. How many times have you had to swap out a concrete dependency vs how many times have you bashed your head against the wall about the proper way to setup a python library? I know my experience skews heavily toward the latter.

I wish python had an equivalent to npm. npm does a lot of things right: local packages by default, recursive dependency resolution, a simple declarative package specification (package.json), and the ability to run your own private npm registry. And it wraps up the logic of a bunch of tools: virtualenv + pip + (distutils / setuptools / distribute / distutils2).

> how many times have you bashed your head against the wall about the proper way to setup a python library

Hmm, almost never actually. The only example I can think of is PIL. I usually have to manually edit the installer to get libjpeg and zlib and fooimagelib to install properly. Other than that, pip seems to have all the magic I crave.

Use Pillow instead, then it works seamlessly with pip: https://pypi.python.org/pypi/Pillow/2.1.0

(assuming you have the relevant libjpeg-dev etc already installed)

What about creating a new library? It's been a giant PITA for me, especially figuring out which of the 4 major distribution tools to use.

The latest one. There was only the brief period when setuptools/disttools were confusing as far as I remember. Now, you can just use d2to1 and the new distribute config format.

So what's it called? "distribute"?

You should be using setuptools now.

There's distutils which was the original one and is part of the standard library, setuptools which extends distutils and adds some very useful things like dependencies, then distribute split off from setuptools because of a number of reasons. Just recently distribute merged back into mainline setuptools and the distribute project is now done and it's maintainers have taken over setuptools.

In the middle there was also distutils2/packaging which were attempts at replacing the entire thing but have since been abandoned for a number of reasons such as trying to replace the entire toolchain at once.

This is just plain wrong. There is no reason not to use setup.py for your private "application" as he calls it. Even if it's not distributed trough the public index you may still benefit from having version numbers and "abstract" dependencies for your app. That way you may choose where to look for the dependencies at deploy time and use a private cheeseshop for delivering your private packages, not to mention that requirements.txt is a pip only thing.

Actually you've misread it or I didn't explain it well enough :)

A private library/app can have abstract dependencies. As I said in the post it deals with the "deploy specific" side of a Python application. So each specific deployment would get it's own requirements.txt.

And yes requirements.txt is a pip only thing but the concept applies to the similar file in zc.buildout whose name escapes me at the moment and any other similar file.

The setup.py approach that loosely couple dependencies is definitely useful as the author describes.

However I think important advice to library developers that's missing in the article is to be explicit (somewhere, likely in documentation) about the version/source of each dependency that the library has been tested against.

Libraries are distributed, applications are deployed.

Sure that'd be good too :) This wasn't really aiming to be a comprehensive best practices guide to distributing things. Mostly trying to explain the differences between what a dependency in setup.py means vs a requirements.txt like file.

I have a hard time coming up with wording because sometimes applications are distributed as well and the "deployment" is just a config file + installing the dependencies. I hate using the term application because of that confusion but I haven't thought of a better way to word it yet :(

Yeah, I just can't agree with this.

I maintain a private bitbucket repository for my company. It contains two things: Django apps, and specific projects which implement those Django apps.

Each Django app is packaged using a standard setuptools/distribute package. All dependencies of each app is managed in setup.py.

I place the projects under version control, in order to maintain the specific templates, url.py, fixtures, and configuration for each site. I use a requirements.txt file to manage the dependencies of the project.

There is nothing abstract about either.

I have a lot of python code from my PhD which will be released when I pass my viva (with full version history, of course) - I'm not looking forward to working out how to deal with pypi after reading this!

I have always tried to make self-contained packages with a simple "just run setup.py" install procedure for my smaller public projects on github, but my larger codebase uses lots of libraries, and specifying which ones people should install is going to be a nightmare...

I think you're really overcomplicating this if your use case is to make something installable from PyPI.

Think of it like this:

* "install_requires" in setup.py = "I don't really care how you get this, or possibly even what version of it you get, just get this".

* requirements file = "I have a known-good, tested and functioning setup using these specific versions, from these specific URLs, of these specific packages; I cannot vouch for whether other versions or other distributions of those packages will work".

I've been working with Ivy which does dependency management for Java.

It's got a similar split: an ivy.xml file for specifying what it depends on, and an ivysettings.xml file for specifying what repositories to look in for those dependencies.

Are you suggesting using `pip freeze > requirements.txt` as some kind of Gemfile.lock compared to setup.py#install_requires as the standard Gemfile?

``setup.py#install_requires`` would be the gemspec file.

Ideally We'd have requirements.txt and requirements.txt.lock which would correspond to Gemfile and Gemfile.lock. However right now that doesn't really happen. So you're kind of stuck having requirements.txt play a double duty where for things where you need exact exact dependencies you'd check in a very exact requirements.txt and things where you have a supported set of deps (maybe more restricted ranges) and you expect people to Deploy by cloning/unzipping a tarball etc you'd use a less restrictive requirements.txt.

I would very much love to get a distinction between the two uses of requirements.txt so that we have a better correlation to Gemfile and Gemfile.lock

Because it makes you depend on urls instead of abstract name/version requirements. easy_install does provide some tooling to get around it but in general it shouldn't be used.

I guess this is where I'm confused. Isn't this the same as using the '-e' switch inside of a requirements.txt?