www.jdmz.net troy.jdmz.net

Multiple Python Installations
Why Would You?
As a system administrator, I like keeping the Python installation from a vendor pristine, whether it is from Sun or a Linux distribution, because they typically package tools that depend on the packaged Python and Python modules. They try to keep that Python, and the modules they choose to package with it, stable and secure for everyone who uses their OS. For most folks, the packaged Python is enough, but some people need newer (or older) versions of packaged modules, or they need to install modules that are not packaged. Upgrading packaged modules outside of the packaging scheme, or installing new and/or non-packaged modules, could have a destabilizing effect on your packaged Python installation.
Also, I like to change default Python installs quickly and easily. I have had to upgrade Python for a bunch of scripts that used the #!/usr/local/bin/python she-bang line. Modifying each script to test with the new Python version, then replacing the "Old Python" with the "Test Python" and modifying the scripts again was "too much fooling around".
It turns out that having your own custom Python, or even a few of them, isn't very costly. I'll use an example with four different Python versions.
Python 2.7.16
For Python 2.7.16 I did this:
sudo mkdir -p /usr/local/python/build
sudo chown $USER:$USER /usr/local/python/build
cd /usr/local/python/build/
wget https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tgz
tar xzvf Python-2.7.16.tgz
cd Python-2.7.16/
./configure --prefix=/usr/local/python/2.7.16
make && make test && sudo make install
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
/usr/local/python/2.7.16/bin/python get-pip.py
/usr/local/python/2.7.16/bin/pip install awscli boto boto3
But the pip install would not work today because 2.7.x is out of support, so get the latest 2.7.x below and use that pip URL.

Python 2.7.18
For Python 2.7.18 I did this:
sudo mkdir -p /usr/local/python/build
sudo chown $USER:$USER /usr/local/python/build
cd /usr/local/python/build/
wget https://www.python.org/ftp/python/2.7.18/Python-2.7.18.tgz
tar xzvf Python-2.7.18.tgz
cd Python-2.7.18/
./configure --prefix=/usr/local/python/2.7.18
make && make test && sudo make install
wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
/usr/local/python/2.7.18/bin/python get-pip.py
/usr/local/python/2.7.18/bin/pip install awscli boto boto3

Python 3.7.4
For Python 3.7.4 I did this:
sudo mkdir -p /usr/local/python/build
sudo chown $USER:$USER /usr/local/python/build
cd /usr/local/python/build/
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
tar xzvf Python-3.7.4.tgz
cd Python-3.7.4/
./configure --prefix=/usr/local/python/3.7.4
make && make test && sudo make install
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
/usr/local/python/3.7.4/bin/python get-pip.py
/usr/local/python/3.7.4/bin/pip install awscli awsebcli boto boto3
It looks like pip is now packaged with current python installs, so use the 'ensurepip' command method below instead.

Python 3.7.13
For Python 3.7.13 I did this:
sudo mkdir -p /usr/local/python/build
sudo chown $USER:$USER /usr/local/python/build
cd /usr/local/python/build/
wget https://www.python.org/ftp/python/3.7.13/Python-3.7.13.tgz
tar xzvf Python-3.7.13.tgz
cd Python-3.7.13/
./configure --prefix=/usr/local/python/3.7.13
make && make test && sudo make install
sudo chown $USER:$USER /usr/local/python/3.7.13
/usr/local/python/3.7.13/bin/python3 -m ensurepip --upgrade
/usr/local/python/3.7.13/bin/pip3 install awscli awsebcli boto boto3

Python 3.8.13
For Python 3.8.13 I did this:
sudo mkdir -p /usr/local/python/build
sudo chown $USER:$USER /usr/local/python/build
cd /usr/local/python/build/
wget https://www.python.org/ftp/python/3.8.13/Python-3.8.13.tgz
tar xzvf Python-3.8.13.tgz
cd Python-3.8.13/
./configure --prefix=/usr/local/python/3.8.13
make && make test && sudo make install
sudo chown $USER:$USER /usr/local/python/3.8.13
/usr/local/python/3.8.13/bin/python3 -m ensurepip --upgrade
/usr/local/python/3.8.13/bin/pip3 install awscli awsebcli boto boto3

Python 3.9.12
For Python 3.9.12 I did this:
sudo mkdir -p /usr/local/python/build
sudo chown $USER:$USER /usr/local/python/build
cd /usr/local/python/build/
wget https://www.python.org/ftp/python/3.9.12/Python-3.9.12.tgz
tar xzvf Python-3.9.12.tgz
cd Python-3.9.12/
./configure --prefix=/usr/local/python/3.9.12
make && make test && sudo make install
sudo chown $USER:$USER /usr/local/python/3.9.12
/usr/local/python/3.9.12/bin/python3 -m ensurepip --upgrade
/usr/local/python/3.9.12/bin/pip3 install awscli awsebcli boto boto3

Python 3.10.4
For Python 3.10.4 I did this:
sudo mkdir -p /usr/local/python/build
sudo chown $USER:$USER /usr/local/python/build
cd /usr/local/python/build/
wget https://www.python.org/ftp/python/3.10.4/Python-3.10.4.tgz
tar xzvf Python-3.10.14.tgz
cd Python-3.10.4/
./configure --prefix=/usr/local/python/3.10.4
make && make test && sudo make install
sudo chown $USER:$USER /usr/local/python/3.10.4
/usr/local/python/3.10.4/bin/python3 -m ensurepip --upgrade
/usr/local/python/3.10.4/bin/pip3 install awscli awsebcli boto boto3

Common Flexible Binary

To make the different versions of Python easy to use and change, the Python binaries have to be in the PATH. Typically /usr/local/bin/ is in the path, so that is where I want to put my Python binaries. Because I want to use my custom Python binaries by default, I will check my path and make sure /usr/local/bin/ comes before /usr/bin/ (or wherever the packaged Python binaries are located) [1].

I will then create a default Python symbolic link at /usr/local/python/current and point it at the Python version I want to use by default.

su -
cd /usr/local/python
ln -s 3.9.12 current
ln -s 3.9.12 current3
ln -s 3.10.4 current310
ln -s 3.9.12 current39
ln -s 3.8.13 current38
ln -s 3.7.13 current37
ln -s 2.7.18 current2
ln -s 2.7.18 current27
ln -s 3.10.4 test
I also want the other versions to be available without typing the full path, so I will do this:
cd /usr/local/bin/
for f in /usr/local/python/2.7.18/bin/* ; do echo $f ; b=`basename $f` ; ln -s $f ${b}2718 ; done
for f in /usr/local/python/3.8.13/bin/* ; do echo $f ; b=`basename $f` ; ln -s $f ${b}3813 ; done
for f in /usr/local/python/3.9.12/bin/* ; do echo $f ; b=`basename $f` ; ln -s $f ${b}3912 ; done

but, then again, typing the full path is not too hard (and that is a lot of links). You be the judge if that is a useful thing for command comparisons or not. I will then make the default binary links:

cd /usr/local/bin/
for f in /usr/local/python/current/bin/* ; do echo $f ; b=`basename $f` ; if [ ! -f $b ]; then ln -s $f $b ; fi ; done

The if avoids name collisions with version specific binaries. Keep in mind than you may want to limit what you put into /usr/local/bin/, or at least make it easy to clean up (like with a 'rm *2716' or something).

Keeping Python Up To Date

This process I layout is "uncareful" with regard to Python installation stability. Somethings may break when you install all the newest versions of every module installed. If this prospect bothers you, you may want to install a "Test Python" (the same version as the one you want to keep stable, maybe at /usr/local/python/3.7.4-test/) and try out the upgrade there first.

If you don't care, and just want the latest of everything, just do this:

/usr/local/python/2.7.16/bin/pip install -U $(/usr/local/python/2.7.16/bin/pip freeze | awk '{split($0, a, "=="); print a[1]}') 
/usr/local/python/3.7.13/bin/pip3 install -U $(/usr/local/python/3.7.13/bin/pip3 freeze | awk '{split($0, a, "=="); print a[1]}') 
To do them all, try this:
for d in /usr/local/python/2* ; do v=`basename $d` ; echo $v ; $d/bin/pip install -U $($d/bin/pip freeze | awk '{split($0, a, "=="); print a[1]}') ; done
for d in /usr/local/python/3* ; do v=`basename $d` ; echo $v ; $d/bin/pip3 install -U $($d/bin/pip3 freeze | awk '{split($0, a, "=="); print a[1]}') ; done

Each process could take a considerable amount of time. Again, I often use screen to make it easier on me.

Managing Ownership Of Python

I like having my Python installations owned by someone other than root. Running a ton of code as root can be avoided here, and so it should [2].

On my own personal servers, I don't mind that person being me (but I see that it could be a risk), but that arrangement may not work for groups larger than one.

For those situations, I think making a pythonadm user and group might serve:

sudo groupadd pythonadm
sudo useradd -m -g pythonadm -c "Python Install Administrator" pythonadm

Then apply the permissions to the installations:

cd /usr/local/
sudo chown -R pythonadm:pythonadm python

When you need to update or change an installation in some way, you will then want to become the 'pythonadm' user with su or sudo.

Notes:
[1] Different shells want the PATH set in different places. You'll probably be modifying a file within the /etc/ directory to change the default path for the entire system.
[2] The more limited the user, the less risk. I want my updates to work within the filesystem tree it owns and not outside it.
© 2022 Troy Johnson
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license should be available here and here.
The current copy of this document should be available here.