5 | SettingsandRequirementsFiles

Django 1.8 has over 140 settings that can be controlled in the settings module, most of which come with default values. Settings are loaded when your server starts up, and experienced Django devel- opers stay away from trying to change settings in production since they require a server restart.

Some best practices we like to follow:

• All settings files need to be version-controlled. is is especially true in production environ- ments, where dates, times, and explanations for settings changes absolutely must be tracked.

• Don’t Repeat Yourself. You should inherit from a base settings le rather than cutting-and- pasting from one le to another.

• Keep secret keys safe. They should be kept out of version control.

5.1 Avoid Non-Versioned Local Settings

We used to advocate the non-versioned local settings anti-pattern. Now we know better.

As developers, we have our own necessary settings for development, such as settings for debug tools which should be disabled (and often not installed to) staging or production servers.

Furthermore, there are often good reasons to keep speci c settings out of public or private code repositories.The SECRET KEY setting is the first thing that comes to mind, but API key settings to services like Amazon, Stripe, and other password-type variables need to be protected.

A common solution is to create local settings.py modules that are created locally per server or development machine, and are purposefully kept out of version control. Developers now make development- specific settings changes, including the incorporation of business logic without the code being tracked in version control. Staging and deployment servers can have location speci c settings and logic without them being tracked in version control.

What could possibly go wrong?!?

Ahem...

• Every machine has untracked code.
• How much hair will you pull out, when after hours of failing to duplicate a production bug locally, you discover that the problem was custom logic in a production-only setting?
• How fast will you run from everyone when the ‘bug’ you discovered locally, xed and pushed to production was actually caused by customizations you made in your own local settings.py module and is now crashing the site?
• Everyone copy/pastes the same local settings.py module everywhere. Isn’t this a violation of Don’t Repeat Yourself but on a larger scale?

Let’s take a different approach. Let’s break up development, staging, test, and production settings into separate components that inherit from a common base object in a settings le tracked by version control. Plus, we’ll make sure we do it in such a way that server secrets will remain secret.

Read on and see how it’s done!

5.2 Using Multiple Settings Files

Instead of having one settings.py file, with this setup you have a settings/ directory containing your settings les. is directory will typically contain something like the following:

settings/
    __init__.py
    base.py
    local.py
    staging.py
    test.py
    production.py

Table 5.1: Settings files and their purpose

Let’s take a look at how to use the shell and runserver management commands with this setup. You’ll have to use the --settings command line option, so you’ll be entering the following at the command-line.

To start the Python interactive interpreter with Django, using your settings/local.py settings file:

To run the local development server with your settings/local.py settings file:

For the settings setup that we just described, here are the values to use with the --settings com- mand line option or the DJANGO SETTINGS MODULE environment variable:

Table 5.2: Setting DJANGO SETTINGS MODULE per location

5.2.1 A Development Settings Example

As mentioned earlier, we need settings con gured for development, such as selecting the console email backend, setting the project to run in DEBUG mode, and setting other con guration options that are used solely for development purposes. We place development settings like the following into settings/local.py :

# settings/local.py
from .base import *

DEBUG = True

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

DATABASES = {
    "default": {
        "ENGINE": "django.core.mail.backends.console.EmailBackend",
        "NAME": "twoscoops",
        "USER": "",
        "PASSWORD": "",
        "HOST": "localhost",
        "PORT": ""
    }
}

INSTALLED_APPS += ("debug_toolbar", )

Now try it out at the command line with:

python manage.py runserver --settings=twoscoops.settings.local

Open http://127.0.0.1:8000 and enjoy your development settings, ready to go into version control! You and other developers will be sharing the same development settings files, which for shared projects, is awesome.

Yet there’s another advantage: No more ‘if DEBUG’ or ‘if not DEBUG’ logic to copy/paste around between projects. Settings just got a whole lot simpler!

At this point we want to take a moment to note that Django settings files are the single, solitary place we advocate using import *. The reason is that for the singular case of Django setting modules we want to override all the namespace.

5.2.2 Multiple Development Settings

Sometimes we’re working on a large project where different developers need different settings, and sharing the same dev.py settings module with teammates won’t do.

Well, it’s still better tracking these settings in version control than relying on everyone customizing the same dev.py or local_settings.py module to their own tastes. A nice way to do this is with multiple dev settings files, e.g. dev_audrey.py and dev_pydanny.py :

# settings/dev_pydanny.py

from .local import *

# Set short cache timeout

CACHE_TIMEOUT = 30

Why? It’s not only good to keep all your own settings les in version control, but it’s also good to be able to see your teammates’ dev settings files. That way, you can tell if someone’s missing a vital or helpful setting in their local development setup, and you can make sure that everyone’s local settings files are synchronized. Here is what our projects frequently use for settings layout:

settings/
    __init__.py
    base.py
    dev_audreyr.py
    dev_pydanny.py
    local.py
    staging.py
    test.py
    production.py

5.3 Separate Configuration From Code

One of the causes of the local settings anti-pattern is that putting SECRET KEY, AWS keys, API keys, or server-specific values into settings files has problems:

• Config varies substantially across deploys, code does not.
• Secret keys are configuration values, not code.
• Secrets often should be just that: secret! Keeping them in version control means that everyone with repository access has access to them.
• Platforms-as-a-service usually don’t give you the ability to edit code on individual servers. Even if they allow it, it’s a terribly dangerous practice.

To resolve this, our answer is to use environment variables in a pattern we like to call, well, The Environment Variables Pattern.

Every operating system supported by Django (and Python) provides the easy capability to create environment variables.

Here are the benefits of using environment variables for secret keys:

• Keeping secrets out of settings allows you to store every settings file in version control without hesitation. All of your Python code really should be stored in version control, including your settings.
• Instead of each developer maintaining an easily-outdated, copy-and-pasted version of the local_settings.py.example file for their own development purposes, everyone shares the same version-controlled settings/local.py.
• System administrators can rapidly deploy the project without having to modify files containing Python code.
• Most platforms-as-a-service recommend the use of environment variables for configuration and have built-in features for setting and managing them.

5.3.1 A Caution Before Using Environment Variables for Secrets

Before you begin setting environment variables, you should have the following:

• A way to manage the secret information you are going to store.

• A good understanding of how bash works with environment variables on servers, or a willing- ness to have your project hosted by a platform-as-a-service.

For more information, see http://2scoops.co/wikipedia-env-variable.

5.3.2 How to Set Environment Variables Locally

On Mac and many Linux distributions that use bash for the shell, one can add lines like the following to the end of a .bashrc, .bash profile, or .profile. When dealing with multiple projects using the same API but with different keys, you can also place these at the end of your virtualenv’s bin/activate script:

On Windows systems, it’s a bit trickier. You can set them one-by-one at the command line (cmd.exe) in a persistent way with the setx command, but you’ll have to close and reopen your command prompt for them to go into effect. A better way is to place these commands at the end of the vir- tualenv’s bin/activate.bat script so they are available upon activation:

PowerShell is much more powerful than the default Windows shell and comes with Windows Vista and above. Setting environment variables while using PowerShell:

For the current Windows user only:

Machine-wide:

For more information on Powershell, see http://2scoops.co/powershell

5.3.3 How to Set Environment Variables in Production

If you’re using your own servers, your exact practices will differ depending on the tools you’re using and the complexity of your setup. For the simplest 1-server setup for test projects, you can set the environment variables manually. But if you’re using scripts or tools for automated server provisioning and deployment, your approach may be more complex. Check the documentation for your deployment tools for more information.

If your Django project is deployed via a platform-as-a-service, check the documentation for specific instructions. We’ve included Heroku instructions here so that you can see that it’s similar for platform-as-a-service options.

On Heroku, you set environment variables with the following command, executed from your development machine:

To see how you access environment variables from the Python side, open up a new Python prompt and type:

To access environment variables from one of your settings les, you can do something like this:

This snippet simply gets the value of the SOME_SECRET_KEY environment variable from the operating system and saves it to a Python variable called SOME_SECRET_KEY.

Following this pattern means all code can remain in version control, and all secrets remain safe.

5.3.4 Handling Missing Secret Key Exceptions

In the above implementation, if the SECRET_KEY isn’t available, it will throw a KeyError, making it impossible to start the project. That’s great, but a KeyError doesn’t tell you that much about what’s actually wrong. Without a more helpful error message, this can be hard to debug, especially under the pressure of deploying to servers while users are waiting and your ice cream is melting.

Here’s a useful code snippet that makes it easier to troubleshoot those missing environment variables. If you’re using our recommended environment variable secrets approach, you’ll want to add this to your settings/base.py file:

Then, in any of your settings files, you can load secret keys from envrionment variables as follows:

Now, if you don't have SOME_SECRET_KEY set as an environment variable, you get a traceback that ends with a useful error message like this:

5.4 When You Can't Use Environment Variables

The problem with using environment variables to store secrets is that is doesn't always work. The most common scenario for this is when using Apache for serving HTTP, but this also happens even in Nginx-based environments where operations wants to do things in a particular way. When this occurs, rather than going back to the local_settings anti-pattern, we advocate using non-executable files kept out of version control in a method we like to call the secrets file pattern.

To implement the secrets file pattern, follow these three steps:

1. Create a secrets file using the configuration format fo choice, be it JSON, Config, YAML, or even XML.
2. Add a secrets loader (JSON-powered example below) to manage the secrets in a cohesive, explicit manner.
3. Add the secrets file name to the .gitignore or .bgignre.

5.4.1 Using JSON Files

Our preference is to use shallow JSON file. The JSON format has the advantage of being the format of choise for various Python and non-Python tools. To use the JSON format, first create a secrets.json file:

To use the secrets.json file, add the following code to your base settings module.

Now we are loading secrets from non-executable JSON files instead of from unversioned executable code. Hooray!

5.4.2 Using Config, YAML, and XML File Formats

While we prefer the forced simplicity of shalow JSON, others might prefer other file formats. We'll leave it up to the reader to create additional get_secret() alternatives that work with these formats. Just remember to be familiar with things like yaml.safe.load() and XML bombs. See section 26.9, 'Defend Against Python Code Injection Attacks.'

5.5 Using Multiple Requirements Files

Finally, there’s one more thing you need to know about multiple settings files setup. It’s good practice for each settings file to have its own corresponding requirements file. This means we’re only installing what is required on each server.

To follow this pattern, recommended to us by Jeff Triplett, first create a requirements/ directory in the <repository_root>. Then create '.txt' files that match the contents of your settings directory. The results should look something like:

requirements/
    base.txt
    local.txt
    staging.txt
    prodution.txt

In the base.txt file, place the dependencies, used in all environments. For example, you might have something like the following in there:

Your local.txt file should have dependenceies used for local development, such as:

The needs of a continuous integration server might prompt the following for a ci.text file:

Prodution installations should be close to what is used in other locations, so prodution.txt commonly just calls base.txt:

5.5.1 Installing From Multiple Requirements Files

For local development:

For prodution:

5.5.2 Using Multiple Requirements Files With Platforms as a Service (PaaS)

See chapter 30, Deployment: Platforms as a Service.

For prodution:

5.5.2 Using Multiple Requirements Files With Platforms as a Service (PaaS)

See chapter 30, Deployment:Platforms as a Service.

5.6 Handling File Paths in Settings

If you switch to the multiple settins setup and get new file path errors to things like templates and media, don't be alarmed. This section will help you resolve these errors.

We humbly beseech the reader hardcode file paths in Django settings files. This is really bad:

The above code represents a common pitfall called hardcoding. The above code, called a fixed path, is bad because as far as you know, pydanny (Daniel Roy Greenfeld) is the only person who has set up their computer to match this path struture. Anyone else trying to use this example will see their project break, forcing to either change their directory structure(unlikely) or change the settings module to match their preference(causing problems for every else including pydanny).

Don't hardcode your paths!

To fix the path issue, we dynamically set a project root variable intuitively named BASE_DIR at the top of the base settigns module. Since BASE_DIR is determined in relation to the location of base.py, your project can be run from any location on any development computer or server.

We find the cleanest way to set a BASE_DIR-like setting is with Unipath (http://pypi.python.org/pypi/Unipath/)[http://pypi.python.org/pypi/Unipath/], a Python package that does elegant, clean path calculations:

If you really want to set your BASE_DIR with the Python standard library's os.path library, though, this is one way to do it in a way that will account for paths:

# At the top of setings/base.py

from os.path import join, abspath, dirname

here = lambda *dirs: join(abspath(dirname(__file__)), *dirs) BASE_DIR = here("..", "..")
root = lambda *dirs: join(abspath(BASE_DIR), *dirs)

# Configuring MEDIA_ROOT
MEDIA_ROOT = root("media")

# Configuring STATIC_ROOT
STATIC_ROOT = root("collected_static")

# Additional locations of static files
STATICFILES_DIRS = (
    root("assets"),
)

# Configuring TEMPLATE_DIRS
TEMPLATES = [
      {
          'BACKEND': 'django.template.backends.django.DjangoTemplates',
          DIRS = (root("templates"),)
}, ]

With your various path settings dependent on BASE_DIR, your file path settings should work, which means your templates and media should be loading without error.

5.7 Summary

Remember, everything except for critical security related values ought to be tracked in version control.

Any project that’s destined for a real live production server is bound to need multiple settings and requirements les. Even beginners to Django need this kind of settings/requirements le setup once their projects are ready to leave the original development machine. We provide our solution, as well as an Apache-friendly solution since it works well for both beginning and advanced developers.

Also, if you prefer a different shell than the ones provided, environment variables still work. You’ll just need to know the syntax for defining them.

The same thing applies to requirements files. Working with untracked dependency differences increases risk as much as untracked settings.