Generally, many Python packages are dependent on other packages but how do we know that on which packages is a module dependent?
pip freeze:
This is a python built-in module that can help us know the dependent packages but it shows all dependencies as a flat list, finding out which are the top-level packages and which packages do they depend on requires some effort. Let us see an example of how it works:
Type given below command on your command prompt:
$pip freeze
Output:
altair==4.1.0 attrs==19.3.0 docutils==0.15.2 entrypoints==0.3 Jinja2==2.11.2 jmespath==0.10.0 jsonschema==3.2.0 MarkupSafe==1.1.1 numpy==1.18.4 opencv-python==4.2.0.34 pandas==1.0.4 pyrsistent==0.16.0 python-dateutil==2.8.1 pytz==2020.1 six==1.15.0 toolz==0.10.0 urllib3==1.25.9
pipdeptree utility:
One easy way of doing so is to use the pipdeptree utility. The pipdeptree works on the command line and shows the installed python packages in the form of a dependency tree.
This module does not come built-in with Python. To install it type the below command in the terminal.
$pip install pipdeptree
This will install the latest version of pipdeptree which requires at least Python 2.7.
Now run this command on command prompt to get a dependency tree of all your Python modules.
Command:
$pipdeptree
Output:
$pipdeptree
altair==4.1.0
- entrypoints [required: Any, installed: 0.3]
- jinja2 [required: Any, installed: 2.11.2]
- MarkupSafe [required: >=0.23, installed: 1.1.1]
- jsonschema [required: Any, installed: 3.2.0]
- attrs [required: >=17.4.0, installed: 19.3.0]
- pyrsistent [required: >=0.14.0, installed: 0.16.0]
- six [required: Any, installed: 1.15.0]
- setuptools [required: Any, installed: 41.2.0]
- six [required: >=1.11.0, installed: 1.15.0]
- numpy [required: Any, installed: 1.18.4]
- pandas [required: >=0.18, installed: 1.0.4]
- numpy [required: >=1.13.3, installed: 1.18.4]
- python-dateutil [required: >=2.6.1, installed: 2.8.1]
- six [required: >=1.5, installed: 1.15.0]
- pytz [required: >=2017.2, installed: 2020.1]
- toolz [required: Any, installed: 0.10.0]
docutils==0.15.2
jmespath==0.10.0
opencv-python==4.2.0.34
- numpy [required: >=1.17.3, installed: 1.18.4]
pipdeptree==1.0.0
- pip [required: >=6.0.0, installed: 20.2.1]
urllib3==1.25.9
Combining pipdeptree and freeze:
Let’s see what happens when we use pipdeptree and freeze altogether,
Command:
$pipdeptree --freeze
Output:
altair==4.1.0
entrypoints==0.3
Jinja2==2.11.2
MarkupSafe==1.1.1
jsonschema==3.2.0
attrs==19.3.0
pyrsistent==0.16.0
six==1.15.0
setuptools==41.2.0
six==1.15.0
numpy==1.18.4
pandas==1.0.4
numpy==1.18.4
python-dateutil==2.8.1
six==1.15.0
pytz==2020.1
toolz==0.10.0
docutils==0.15.2
jmespath==0.10.0
Mako==1.1.3
MarkupSafe==1.1.1
opencv-python==4.2.0.34
numpy==1.18.4
pipdeptree==1.0.0
pip==20.2.1
scipy==1.5.2
numpy==1.18.4
urllib3==1.25.9
So, here we see that using pipdeptree along with freeze shows the output by combining the properties of both commands. So it looks like the output of pip freeze indicates which that package installed which another package, similar to pipdeptree but here indentation is used instead of a hyphen(-) to indicate tree.
Warnings in pipdeptree:
Commonly there occur two types of warnings while executing pipdeptree command, let us see them one by one.
1. Conflicting Dependencies: As the name suggests “conflicting dependency”, so is its relevance. Sometimes there is/are package(s) that are specified as a dependency of multiple packages with a different version, in this situation possible conflicting dependency warning arises. So, any package that’s specified as a dependency of multiple packages with a different version is considered as a possible conflicting dependency.
pipdeptree by default warns about possible conflicting dependencies.
Let us see one more example of pipdeptree:
Command:
$pipdeptree
Output:
Warning!!! Possibly conflicting dependencies found:
* impacket==0.9.20
- ldap3 [required: ==2.5.1, installed: ?]
- ldapdomaindump [required: >=0.9.0, installed: ?]
------------------------------------------------------------------------
alembic==1.0.11.dev0
attrs==18.2.0
dulwich==0.20.2
- certifi [required: Any, installed: 2018.11.29]
- urllib3 [required: >=1.24.1, installed: 1.24.1]
EditorConfig==0.12.1
Flask-Cors==3.0.8
- Flask [required: >=0.9, installed: 1.1.1]
- Six [required: Any, installed: 1.13.0]
Flask-Session==0.3.1
Flask-SocketIO==4.2.1
- Flask [required: >=0.9, installed: 1.1.1]
- python-socketio [required: >=4.3.0, installed: 4.5.1]
- python-engineio [required: >=3.9.0, installed: 3.12.1]
- six [required: >=1.9.0, installed: 1.13.0]
- six [required: >=1.9.0, installed: 1.13.0]
google==2.0.1
- beautifulsoup4 [required: Any, installed: 4.8.0]
html2text==2019.8.11
impacket==0.9.20
- ldap3 [required: ==2.5.1, installed: ?]
- ldapdomaindump [required: >=0.9.0, installed: ?]
ipython==5.8.0
- backports.shutil-get-terminal-size [required: Any, installed: 1.0.0]
- pathlib2 [required: Any, installed: 2.3.5]
- scandir [required: Any, installed: 1.10.0]
- pexpect [required: Any, installed: 4.6.0]
pip doesn’t have a true dependency resolution yet. The warning is printed to stderr (Standard error) instead of stdout (Standard Output). To completely silence this warning use the -w silence or –warn silence flag It can also be made mode strict with –warn fail in which case the command will not only print the warnings to stderr but also exit with a non-zero status code. This could be useful if you want to fit this tool into your CI pipeline.
Note: The –warn flag was added in version 0.6.0. For older version, use –nowarn flag.
2. Circular Dependencies: This dependency occurs when two packages depend on each other. Suppose, package A depends upon package B and package B depends upon package A.
For this let us see one more example:
Command:
$pipdeptree
Output:
Warning!!! Cyclic dependencies found: - CircularDependencyA => CircularDependencyB => CircularDependencyA - CircularDependencyB => CircularDependencyA => CircularDependencyB ------------------------------------------------------------------------ wsgiref==0.1.2 argparse==1.2.1
Note: They are also printed to stderr and can be controlled using the –warn flag.
To find why a particular package is installed:
Now, we may sometimes want to know why a particular package is installed. Then we can use –reverse (or simply -r) flag for this. To find out what all packages require a particular package(s), it can be combined with –packages
flag as shown in following example:
Command:
$pipdeptree --reverse --packages MarkupSafe,numpy
Output:
MarkupSafe==1.1.1
- Jinja2==2.11.2 [requires: MarkupSafe>=0.23]
- altair==4.1.0 [requires: jinja2]
- Mako==1.1.3 [requires: MarkupSafe>=0.9.2]
numpy==1.18.4
- altair==4.1.0 [requires: numpy]
- opencv-python==4.2.0.34 [requires: numpy>=1.17.3]
- pandas==1.0.4 [requires: numpy>=1.13.3]
- altair==4.1.0 [requires: pandas>=0.18]
- scipy==1.5.2 [requires: numpy>=1.14.5]
Using pipdeptree to write requirements.txt file:
If you wish to track only the top-level packages in your requirements.txt file, it’s possible to do so using pipdeptree by grep-ing only the top-level lines from the output,
Command:
$pipdeptree | grep -P '^\w+'
Output:
Lookupy==0.1 wsgiref==0.1.2 argparse==1.2.1 psycopg2==2.5.2 Flask-Script==0.6.6 alembic==0.6.2 ipython==2.0.0 slugify==0.0.1 redis==2.9.1
There is a problem here though. The output doesn’t mention anything about Lookupy being installed as an editable package (refer to the output of pip freeze above) and information about its source is lost. To fix this, pipdeptree must be run with a -f or –freeze flag
Command:
$pipdeptree -f --warn silence | grep -P '^[\w0-9\-=.]+'
Output:
-e git+git@github.com:naiquevin/lookupy.git@cdbe30c160e1c29802df75e145ea4ad903c05386#egg=Lookupy-master wsgiref==0.1.2 argparse==1.2.1 psycopg2==2.5.2 Flask-Script==0.6.6 alembic==0.6.2 ipython==2.0.0 slugify==0.0.1 redis==2.9.1
Command:
$ pipdeptree -f --warn silence | grep -P '^[\w0-9\-=.]+' > requirements.txt
The freeze flag will also not output the hyphens for child dependencies, so you could dump the complete output of pipdeptree -f to the requirements.txt file making the file human-friendly (due to indentations) as well as pip-friendly. (Take care of duplicate dependencies though)
Using pipdeptree with External tools:
pipdeptree uses flag –json to show output in JSON representation, as shown below:
Command:
$pipdeptree --json
Output:
[
{
"package": {
"key": "werkzeug",
"package_name": "Werkzeug",
"installed_version": "1.0.1"
},
"dependencies": []
},
{
"package": {
"key": "urllib3",
"package_name": "urllib3",
"installed_version": "1.25.9"
},
"dependencies": []
},
{
"package": {
"key": "pytz",
"package_name": "pytz",
"installed_version": "2020.1"
},
"dependencies": []
},
{
"package": {
"key": "python-dateutil",
"package_name": "python-dateutil",
"installed_version": "2.8.1"
},
"dependencies": [
{
"key": "six",
"package_name": "six",
"installed_version": "1.15.0",
"required_version": ">=1.5"
}
]
},
{
"package": {
"key": "pyrsistent",
"package_name": "pyrsistent",
"installed_version": "0.16.0"
},
"dependencies": [
{
"key": "six",
"package_name": "six",
"installed_version": "1.15.0",
"required_version": null
}
]
},
{
"package": {
"key": "pipdeptree",
"package_name": "pipdeptree",
"installed_version": "1.0.0"
},
"dependencies": [
{
"key": "pip",
"package_name": "pip",
"installed_version": "20.2.1",
"required_version": ">=6.0.0"
}
]
},
]
Note: –json will output a flat list of all packages with their immediate dependencies. To obtain nested JSON, use –-json-tree (added in version 0.11.0).
Command:
$pipdeptree --json-tree
Output:
[
{
"key": "altair",
"package_name": "altair",
"installed_version": "4.1.0",
"required_version": "4.1.0",
"dependencies": [
{
"key": "entrypoints",
"package_name": "entrypoints",
"installed_version": "0.3",
"required_version": "Any",
"dependencies": []
},
{
"key": "jinja2",
"package_name": "jinja2",
"installed_version": "2.11.2",
"required_version": "Any",
"dependencies": [
{
"key": "markupsafe",
"package_name": "MarkupSafe",
"installed_version": "1.1.1",
"required_version": ">=0.23",
"dependencies": []
}
]
},
{
"key": "urllib3",
"package_name": "urllib3",
"installed_version": "1.25.9",
"required_version": "1.25.9",
"dependencies": []
}
]
