Its been a while since I've had to deploy a complete python application. This it not a web app - just stand-alone python daemon. Its not even online, so no AWS Beanstalk, Heroku or other frameworks are relevant.
There is a good old
pip -r requirements.txt way. But it requires custom scriptology to roll out new code, run before/after update scripts and all that other tasks that we, programmers, like leave to "integration" stage of the project :). So I've googled to check whether is there any good news on this subject.
And there are! - dh-virtualenv. IMHO it is as good as it can be if you use Debian-based distro. In the nutshell, it wraps your whole
virtualenv together with your code into one single deb package. So the deployment boils down to:
- Copy resulting deb file to target machine
sudo dpkg -i mypackage.deb
sudo apt-get install -f- this one is to automatically install missing dependencies for your package
setup.pythen after installation you can run them just like this:
/usr/share/python/<YOU PACKAGE NAME>/bin/myscriptAnd it will automatically use your venv's python and all your packages! Convenient, isn't it?
Using dh-virtualenv is straight-forward, especially if you have some background on Debian packaging. I've just followed the tutorial. However there are some quirks you better be aware of:
Each time you'll run
dpkg-buildpackagedh-virtualenv will download all of the packages from your
requirements.txt. If you have a dozen packages that becomes really annoying and lengthens your build time significantly.
To make things worse,
pip gets random
Connection reset by peer errors. Its a lesser problem when installing packages interactively, because
pip uses cache
and install attempt usually succeeds from the second try. But if you hit this with dh-virtualenv, next time it will start from the beginning. I found myself running
6-7 times until download went successfully - definitely not an option.
The cure is to run your own, local and transparent PyPi mirror. I've used devpi - its sort of a transparent caching proxy daemon for PyPi. Just follow the setup steps in the above link and you'll have it running in no time. The next step is to make dh-virtualenv aware of it. Here are my
debian/rules to introduce dh-virtualenv to devpi:
#!/usr/bin/make -f %: dh $@ --with python-virtualenv override_dh_virtualenv: dh_virtualenv --pypi-url http://localhost:3141/root/pypi/
(Note that this a Makefile and those spaces in the beginning of the lines should be tabs!)
Build your package on a server with similar Debian/Ubuntu versionFor example, building your package on Ubuntu 13.10 and deploying it Ubuntu 12.04 LTS would not work if your code uses C extensions (even through imported packages). This issue is general to virtualenv and is not dh-virtualenv specific. For example if you create virtualenv and copy it to another, binary-incompatible server, then chances are that you'll be able to run
pythonfrom this copied venv, but will NOT be able to import, say,
ssl. This is because python executable is embedded into virtualenv, but standard libraries are not; and of course importing module compiled against other version of interpreter would not work.
The problem does not ends here - if you upgrade python on your production server (through apt) as part of security upgrade or similar, then it will break your virtualenv!
The bottom line - if you are going to deploy to a certain version of Debian/Ubuntu, then have another separate "builder" server with the same OS where you'll build your package and note:
- If you run
apt-get upgrade- run it on both servers
- If you upgrade python as part of the upgrade - better rebuild and redeploy your app
Hooking your own post-build commandsIf you custom operations performed before the deb file is being packed it may be not obvious where to hook them. Defining
override_dh_installtarget will probably be not that helpful, because it runs before
dh_virtualenv- your python files just will not be installed yet. So the solution I've found is to hook them into dh_virtualenv override:
#!/usr/bin/make -f %: dh $@ --with python-virtualenv override_dh_virtualenv: dh_virtualenv --pypi-url http://localhost:3141/root/pypi/ # !! your commands go here !!