If you're downloading pre-built binaries like on Windows then unfortunately yes it does matter if you're using 32-bit or 64-bit libraries. Windows in general has always been a bit of a second class citizen with Python library support, although it will hopefully improve with easier access to compilers (there's a free version of visual studio tools for python library compilation apparently now). If you just want a lot of libraries that 'just work' you're probably best served using a pre-built distribution like Anaconda. If you're more adventurous this page has all kinds of pre-built libraries for Windows in both 32 and 64 bit: http://www.lfd.uci.edu/~gohlke/pythonlibs/
Alternatively it's insanely easy to spin up a Linux VM with vagrant and just compile and run python scripts from there. Everything will pip install without any issues.
Windows is a second class citizen for most language library support, that being said though Python does a great job of having the most recent versions available for Windows. The same can't be said of Ruby unfortunately. :(
Alternatively it's insanely easy to spin up a Linux VM with vagrant and just compile and run python scripts from there. Everything will pip install without any issues.