This talk has been previously presented at:
The slides below are from the DjangoCon 2021 iteration.
 | Hi! I'm Katie, and this is "What is deployment, anyway?". This isn't the talk you're expecting. |
 | This isn't a talk about the "One Way" to do deployments. No talk can do that. This also isn't a talk about how to take your proof-of-concept VC-funded startup site and turning it into the next big dot com. If you want to talk about that |
 | Here's my work email. Let's chat. But for today, the scope of my discussions are going to be talking about the minimum number of things you need to consider in order to host your django website. This talk is more about taking your little cool, Django blog application that you've made on your local machine and putting it on the internet so you can share it with your friends. This talk is going to answer the question posed in the title: |
 | What is deployment, anyway? More specifically, what is django deployment, anyway? (because you know, this is a djangocon!) and we're going to focus on the "what" here. We're going to briefly mention some of the "where" and the "how", but this is going to focus on the "what" |
 | This talk and the code examples in this talk are specifically django 3.2 and Python 3.9, as these are the most current minor versions at time of recording. If you're joining me on YouTube from the year 2032 I'm sorry? This talk might be a little bit out of date. But hopefully you still learn something. |
 | So, let's start by looking at django, as it comes, out of the box. Just as is. If you haven't create a new django project in a while, it's nice to be reminded about how it all starts. |
 | So in my terminal, I want to install django so I type "pip install django" and pip goes and collects django and it's main dependencies. I then want to start a new project So for that I'll use the django-admin CLI that was just installed, and use it's startproject template function. and i'll call it myproject, and create it in the current folder. now I have no output from this command but if I check what files I now have in this folder I can see I have a manage.py file, and then a myproject folder with some different files in there. I can see an asgi.py, a settings.py, a urls.py, and a wsgi.py. .. Huh. |
 | so, clearing my terminal, I want to now start up django so I run python manage.py runserver which then runs, but I get a big red error message, I have unapplied by migrations but django has helpfully suggested that I run "python manage.py migrate" to apply them. Great! let's quit that process |
 | and run that migrate command. All the migrations are applied, and they all report OK great! |
 | so then let's see what's in my folder now Ah, I see there's a new file Looks like i have a dee-bee dot ess-qu-lite-three file. That's interesting. since I've applied my migrations, I should be able to run my application |
 | so if I runserver again I get no red warnings, and I'm told the development server is available so let's go there |
 | Oh hey, the installation worked successfully! Go me! I also know I should now have a django admin |
 | Buttt I can't login. I don't have an account yet. so let's go back to the terminal and do that |
 | Let's stop that process Clear our terminal |
 | And then run the createsuperuser command This command, will create a superuser for me, that will have access to the admin, I'll use my name For this I don't need an email I'll give it a super secure long password twice and now I have a superuser. I can then restart the process |
 | Then back in my login screen |
 | I can enter my username and passwrd |
 | And I have access to the admin! Hooray! |
 | Django has a great local development story we can get our app running locally really easily. and a lot of this boils down to |
 | runserver This command runs our application for us, and also does nice things like automatically restart itself when made an edit to our code. The django documentation describes what runserver does. Seriously, the django docs are great, and they are very good at explaining what django does. If you're newer to django, or any tech, some docs can be hard to parse and understand, but that's okay, I'm here. Let's look at the docs. |
 https://docs.djangoproject.com/en/3.2/ref/django-admin/#runserver | from the docs, the runserver command Starts a lightweight development web server on the local machine. But if you look down a few paragraphs, in all caps do 👏 not 👏 use 👏 this 👏 server 👏 in 👏 a production setting Why? it continues. It has not gone through security audits or performance tests and that's now it's gonna stay we're in the business of making web frameworks, not web servers. let me pull that last part out into it's own slide |
 | Django is an extremely stable, production ready, web framework. Django is very good at being a web framework. The fact that it provides a local web server is amazing for getting started but the fact that it calls out it's limitations is also very very good. So we're going to need to find another webserver. This part of the documentation also brings up some .. terms. |
 | what is "production" anyway? and also |
 | why is it called "production"? Historically, when computers were fed punch cards, you could say it was a "production" to process data. You had to physically move different things around. Much like a production line in manufacturing. You could also think of it nowdays as a stage production. It's the thing that paterons, attendees, or users see. Your production environment is the one you want people to go to. It's your "live" environment. It needs to be able to support a crowd. The size of your crowd is going to influence when you start thinking about things like load balancing and caching, but right now, it's just us, and our mate on the other side of the country who we want to show this website to. |
 | not only is runserver a development web server, runserver also serves the pretty, that being the images, css, javascript, etc that comes with our application. |
 | To be able to see the nice centrally formatted blue-on-grey login window, there needs to be css to make that happen. these comes with django itself. If you don't have this setup correctly |
 | your login screen looks very 1997 I mean, it's still usable -- web accessibility FTW -- but you can tell that something isn't setup correctly. Collectively, all these styling files are known as "static" |
 | But why is it called static? They're called "static" because they are unchanging. They came with Django itself, and they don't change. But they have to be 'served' so that the browser can make use of them. For instance |
 | if we take a look at the source of the login screen, we can see that we have some css and javascript being served from /static/ path -- The django documentation describes how runserver achieves this. |
 https://docs.djangoproject.com/en/3.2/howto/static-files/#serving-static-files-during-development | In the documentation for serving static files during development, there's another bit in here This is not suitable for production use! Also noted that runserver will only serve static files when you have DEBUG set to True. Well, we can see these files are being served, so we should have DEBUG as true, right? How can we check? This is where we start looking at some of the code that was automatically created when we run startproject |
 | if we remember from our file listing we have a settings.py sitting under our myproject folder. so let's check what's in that file. |
 | this file is quite long, and was automatically created for us. but one of the first things we see is quickstart development settings. unsuitable for production and these warnings just keep going |
 | going down the page, see security warning after security warning these settings all work for a local development environment, but it's completely understandable that all these warnings and stuff make it seem just so... inaccessible to a beginning developer who just wants to share their cool blog with their friends. this lack of knowledge is because they are a beginner developer, not a server operations person, or a system administrator. it's a completely different engineering specialisation. this is the entire reason for this talk to demystify deployment. oh look, there's that debug setting. it is true by default. this makes sense. while we're here, we also saw that db.sqlite3 file appear after running our migrations. |
 | If we scroll down the settings file we can see that that string is defined here as part of our DATABASES definition. so what is this sqlite thing? To the Docs! o/ |
 https://docs.djangoproject.com/en/3.2/ref/databases/#sqlite-notes | SQLite provides an excellent development alternative... for applications that are predominantly read-only or require a smaller installation footprint. -- Again, that word development. |
 | django has a great local development story you saw just how quickly we could get a django site working on our local machine. |
 | but to get our site to production is complex as we've discovered together, you need to provide you own production ready webserver, database, and static host. And if you aren't familiar with the production grade offerings out there, you'll end up confused or worse, using runserver in production which... you should not do. Good work! Oh dear, I enabled audience participation. Yes? What's that I hear from the back of the room, Do you have a question? ... Flask isn't this hard. ... not exactly a question, but let's talk about that. |
 | flask is simpler in production. a lot of deployment tutorials you'll see that say "How to Deploy Python in Production!" often use Flask as the target. "Just copy your code and run! Easy!" Well yes, that's true. You can get a out-the-box Hello World application running quite easily on many different platforms using flask. the issue that django has is one of |
 | state. django is a stateful application. Full of state. That is, it contains information and data that we really want to keep a hold of. the absolute bare bones django app we just ran comes with the django admin. the django admin requires a database. therefore, django requires a database. therefore django requires state. a Flask app that prints helloworld doesn't need a database. Yes you can absolutely deploy flask with a database, and django without a database, but any application that has state is going to be complex to deploy. Let me share another gem from the Django documentation. |
 https://docs.djangoproject.com/en/3.2/howto/static-files/#serving-static-files-during-development | Earlier I shared this section on serving static files. This Deploying static files link goes to |
 https://docs.djangoproject.com/en/3.2/howto/static-files/deployment/ | a page full of strategies for how to deploy static files. But it also has this line As with all deployment tasks, the devil’s in the details. Every production setup will be a bit different. |
 | _EVERY_ production setup will be a bit different. This line has been in the docs for over a decade now, but it still holds true. Every production setup will be a bit different. |
 | Most every deployment talk you'll see will include a line like "It depends" when telling you the ""one true way"" to do deployments Sidenote, there is a talk called "The one true way to do django deployments". It's quite good. But that's because it's true. It does depend. But I want to use a different line. |
 | I'm a sysadmin. I'm not your sysadmin. I'm not the one who will be making sure your django site stays online. So any recommendations I make in this talk are just that. Recommendations based on my experience. It's up to you and your team to understand what you're doing, because you're the ones that'll have to maintain it, and fix it when it's broken.. That being said: |
 | to be able to deploy to production, we need to upgrade the components we are using on our local machine to something more production ready we need a production grade webserver a production grade database and production grade static Django has support for a number of the most common of these, which will *really* help us here. That means we can move out of the scope of Django specific things, and find solutions that work for the generic use case, as that has more options, and more support. For instance, Flask |
 | I've said before that Flask is simpler in production. This is because while you may not need a database or static for a flask app, you will need a webserver. and the reason you can get flask up and running super easily is that hosting a site on a webserver is simpler than it's ever been Let's take a look at the Flask docs! |
 https://flask.palletsprojects.com/en/2.0.x/deploying/index.html | Yes, I'm showing you the Flask _docs_ at a DjangoCon. Fight me in the hallway. The Flask documentation don't need me to do the highlighting here, because it's already in bold: Flask's built-in server is not suitable for production. They also offer a list of hosting options. All these links go out to the hosting vendor's documentation, and show step by step ways to "just copy your code and host it". This isn't a complete list, of course. |
 | The host you use is going to be based on a few factors: what platform you already know, what platform your team already knows how to support, or just what platform you're interested in learning. It's your choice. There are also going to be different options for the same platform. Some of them provide a complete and ready to go python environment for you, and you just have to supply your code, others you have to configure and supply more. A lot of them cover basic load balancing for you, by "autoscaling" They'll provide HTTPS by default by handling SSL certification for you You can of course do this yourself, but if it's just you, your friend, and your blog, you don't have to. Hosting options are always evolving, and the best practice changes over time. Django doesn't list them because they are so in flux. This is why you need to understand where django let's go of your hand and says "Off you go, have fun storming the castle". Many hosting providers will have Django tutorials much like they do for Flask. I should know, I wrote a few of them. That'll get you to the how, as of now.We'll continue here with the "what" though. |
 | These Flask docs go onto mention this WSGI thing. Wait, we saw that earlier... waaay back in the beginning in the Django runserver docs |
 https://docs.djangoproject.com/en/3.2/ref/django-admin/#runserver | Sitting in there, there was the line "This server uses the WSGI application object specified by the WSGI_APPLICATION setting." Wait, that was in our settings file! |
 | Just above databases, we saw WSGI_APPLICATION = 'myproject.wsgi.application' wait, we saw a myproject.wsgi earlier! |
 | In our file listing we have a myproject/wsgi.py file. so let's open that up. |
 | This is the entire file. most of it is comments, but we can see that It exposes the WSGI callable as a module-level variable named ``application``. Ooh, a docs link! Let's check that |
 https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/ | These docs say that Django’s primary deployment platform is WSGI, the Python standard for web servers and applications. Well what's what then? TO THE PYTHON DOCS |
 https://www.python.org/dev/peps/pep-0333/ | This is Python Enhancement Proposal 333 (or PEP 3333) for the Python Web Server Gateway Interface The abstract is pretty self explanatory: "This document specifies a proposed standard interface between web servers and Python web applications or frameworks, to promote web application portability across a variety of web servers." it's this portability that we're taking advantage of. All those potential hosting options from earlier, they either directly provide, or allow you to easily provide, a WSGI server to host your web application. This is why hosting is so easy now days You could create a virtual machine and do this all manually, continuing your firewall and routing, but there are places that'll say "Give us your code, here's the URL for your site have fun". They may things so much easier, because this is a problem all web developers have. They wanna host a web. And so, they provide. But it's not just WSGI now. |
 https://asgi.readthedocs.io | There is also ASGI an Asynchronous server gateway interface You may have noticed an ASGI.py file in the project template from earlier. For right now, unless your blog does something fancy with ASGI, you don't have to worry about this. Support for ASGI is still increasing, but unless you're doing async stuff, just stay with WSGI. |
 | So we've established you to have a WSGI server on your platform host of choice. There are many, but the one I use is gunicorn. It's been listed on the Flask and Django pages we've seen so far. |
 https://docs.gunicorn.org/en/latest/run.html | I use guniron It is functionally very similar to runserver's main function -- it serves websites -- but importantly, it is a webserver that has been security reviewed and is suitable for production. You install it as a python package, then you can run the gunicorn command, specifying your WSGI app So we can do just that |
 | so in the terminal pip install gunicorn which pip does then we run gunicorn myproject.wsgi:application and run our web server and look, it's running just at the same place it was before for runserver |
 | Rocketship! But... |
 | our admin is 90's. And we know from our preivous pokering that's because of static issues. |
 | Indeed, looking back at our terminal, all our assets aren't found. this is because runserver handled static for us, which gunicorn doesn't do. |
 | so we can use gunicorn for our webserver, but it doesn'thandle the static, that's still another component. that's okay. We want to choose the best available options out there. as for hosting gunicorn, *How* you deploy is going to depend on your host of choice, check their documenation. This is a "what" talk, not a "how" talk. "How" changes over time. "what" is pretty consistent. But since you've now got our webserver, we now need to create and connect our stateful services. Our database, and our static. Your host will probably have these offerings. Starting with the database. |
 | Unless you have experience in another database that django supports, or a team that knows how to use another database I suggest using postgres. Django supports many databases, but PostgreSQL has the most support. So says me and the docs |
 https://docs.djangoproject.com/en/3.2/ref/contrib/postgres/ | From the reference for the postgres contrib module, in a high level note right at the top of the page. "Django is, and will continue to be, a database-agnostic web framework. We would encourage those writing reusable applications for the Django community to write database-agnostic code where practical. However, we recognize that real world projects written using Django need not be database-agnostic." "Django provides support for a number of data types which will only work with PostgreSQL. There is no fundamental reason why (for example) a contrib.mysql module does not exist, except that PostgreSQL has the richest feature set of the supported databases so its users have the most to gain." A lot of people put a lot of work into django, and supporting postgres in particular. If you'd like to get more support for your preferred database, go for it! We'd love to have more support for these things.But if you want an answer right now, use Postgres. |
 | But if you want an answer right now, use Postgres. |
 | Our original settings.py file, it's engine is the sqlite3 version. So what we need to do is replace that with postgres |
 | When we setup our postgres server on our host of choice, we'll be provided with information about to connect to it. |
 | This will include our host and port that the server can be accessed from, and there will be a database on that server that is ours. But we probably want to make sure that this data is secure, so we'll need to specify a username and password |
 | but i really don't ever want see these appear in plain text in your settings.py. what you should be doing is having these values stored securely else. There are choices for this. |
 | your host may allow you to set encrypted environment variables, you can then configure your database settings to pull these values from your environment. these are a lot of values to set. one thing I like to do is instead define an environment file. |
 https://django-environ.readthedocs.io | I use a package called Django-environ It's a package that allows you to use the Twleve Factor Methodoloy to configure our application with environment variables. What this code snippet does, it that it'll load your environment, then read in the file. So instead of using os.environ.get everywhere, we can just use "env". But Katie, what is this "twelve factor methodology"? |
 https://www.12factor.net/ | the twelve factors are a methodology to building web apps that make them easier to deploy, scale, and maintain. it was developed by people who made Heroku, and talks about the lessons they learnt there. This isn't a talk about 12 factor apps, but I will say that the method we're using to deploy our application is already covering a bunch of them, or we easily can. Dependencies: well, we if we throw our dependencies into a requirements file, then we're good there. Config: literally what we're doing now. Backing services: this is what our databases and static are. They're attaching to our process. And portbinding, that's our 8000 port right there. If you're using a hosted service, you're going to get processes, concurrency, disposability and logging for free. These are the concepts those providers work on. Instead of giving you an entire machine, they'll give you the ability to use part of one.These factors are what help them scale. Of the ones we haven't covered, you can think about using github to store your codebase, using your hosts build process to automatically update when you push code there, and replicate that setup if you want a testing environment. And then your admin process, that's your migrate command. There are other great talks that step into this methodology more, both in general and for django. But should get back to implementing django-envoron |
 | These factors are what help them scale. Of the ones we haven't covered, you can think about using github to store your codebase, using your hosts build process to automatically update when you push code there, and replicate that setup if you want a testing environment. And then your admin process, that's your migrate command. There are other great talks that step into this methodology more, both in general and for django. But should get back to implementing django-envoron |
 | Adding a little bit of code to the top of my file I can import and setup django-environ, |
 | then set all my configs in either my environment variables or environment file and my settings.py will pick them up. checkout the django-environ documentation for some more ways you can use package. the database url feature is quite neat. |
 | so, we need postgres for feature coverage, and django-environ for configuration portability. |
 | Now we have a database, we need to ensure we're actually applying the migrations to it. That's the admin process we mentioned serlier. Running manage.py migrate Later you're going to have to consider when you'll run these migrations, and if you'll automatically make them, or do any customisations. Markus Holtermann is a subject matter expert on this, check out some of his talks and blog posts. Back to the "what". |
 | so for our database, we've now postgres with a side of django-environ now for static. again, your host of choice is going to have a backing service solution for this. It'll be an "object store" of some sort. These are some of the oldest parts of cloud computing, and are exceptionally robust now. There's also a helper package for django for integrating these options into your application |
 https://django-storages.readthedocs.io/en/latest/ | Django-storages is a collection of custom storage backends for django. These hook in pretty deep into django, and allow you to do all sorts of things with whatever static host you want to use. The documentation will guide you through the config you need for each type. |
 | And you'll be adding a few things to the end of settings.py. Somewhere around here where your STATIC_URL setting is. |
 | Now we have a place to put ourstatic, how do we put in in there? How do we collect out static? We know we're missing that step as our admin looked web 1.0. But you see, that's another of those "HOW" words. The command you need is called... collectstatic. But the HOW for this is a completely separate talk in and of itself. A good example of this talk is Jacob Kaplan-Moss' talk "Assets in Django". He goes through different strategies for static, including a deeper dive into what collectstatic does. |
 | so here we have it. you use gunicorn as a webserver on your host of choice you use a hosted postgresql database and django-environ for settings management, applying said migrations with migrate and hosted static with django-storages, and applying with collectstatic. This is my recommendation of what deployment is in 2021 people in 2032, that disclaimer is also for you! Things may have changed! Giving this kind of talk is going to be an artefact of it's time! Please check the current django documentation for up to date recommendations! Try creating a django project from the template and see what comments are in the settings file! I've shown you how to do this! |
 | There are a lot of "how" and "when" things that I just didn't cover today There's a LOT to this space, and it's absolutely ok if you don't understand or even know about all of them. I don't. Each one of these are it's own talk, or at worst, it's own **conference**. |
 | The specialisation of operations engineering is complex and opinionated and ever evolving But what we're done in the last little bit is step through django gives you, helped you parse some of it's base documentation, understood some the concepts and made some suggestions, and given you helpers to get you deploying. |
 | Thank you so much for watching! I'm on glasnt on all the platforms, whatever those are in 2032, and if you want to chat, here's my email too. If you or your CTO wants to talk about serverless scalability and django hosting, Here's my work email. Have a brilliant rest of your day! |