29 What About Those Random Ultities

29.1 Create a Core App for Your Utilities

Sometimes we end up writing shared classes or little general-purpose utilities that are useful everywhere. These bits and pieces don't belong tin any particular app. We don't just stick them into a sort-of-related random app, beacause we have a hard time finding them when we need them. We also don't like placing them as "random" modules in the root of the project.

Our way of handling our utilities is to place them into a Django app called core that contains modules which contains functions and objects for use across a project. (Other developers follow a similar pattern and call this sort of app commonm, generic, util, or utils.)

Foe example, perhaps our project has both a custom model manager and a custom view mixin used by serveral different apps. Our core app would therefore look like:

EXAMPLE 29.1

core/
    __init__.py
    managers.py # contains the custom model manager(s)
    models.py 
    views.py # Contains the custom view mixin(s)

TIP: Always make the core app a real Django app

We always make the core directory a Django app. At some point we inevitably end up doing at least one of the following:

    Have non-abstract models in core
    Have admin auto-discovery working in core
    Have template tags and filters in core

Now, if we want to import our custom model manager and/or view mixin, we import using the same pattern of imports we use for everything else:

EXAMPLE 29.2
from core.managers import PublishedManager
from core.views import IceCreamMixin

29.2 Optimize Apps with Ultility Modules

Synonymous with helpers, these are commonly called utils.py and sometimes helper.py. They are places where we place functions and classes which make common patterns shorter and easier. let's go into why this is helpful.

29.2.1 Storing Code Used in Many Places

There are times when we have functions or classes used in serveral places that doesn't quite fit in models.py, forms.py, or any other specifically named module. When this occurs, we put this logic in the utils.py module.

29.2.2 Trimming Models

This is best explained with an example.

We use the Flavor model frequently. We start attaching field after field, method after method, property after property, classmethod after classmethod. One day we notice that our fat model has reached brobdingnagian proprtions and is over a thousand lines of code. Debugging and maintenance have become hard. What do we do?

We start looking for methods (or properties or classmethods) whose logic can easily encapsulated in function in flavors/utils.py. The result is a more distuributed code base that encourages code reuse as well as easier testing.

29.2.3 Easier Testing

A side effert of moving logic from more complex constructs into functions placed in isolated modules is that it become easier to test. By isolation we mean it is usually imported within the app, rather than doing in-app/in-project imports. This cause less bussiness logic overhead, hence making it easier to write tests for what logic is present in the module.

TIP: Make Utility Code Constructs as Focused as Possible

Be it a function or a class, avoid allowing multiple behaviors or conditions. Each utility function should do once and only one thing well. Don't repeat yourself. Don't create utility function that are duplicates of model behaviors.

29.3 Django's Own Swiss Army Knife

The Swiss army knife is a multi-purpose tool that is compact and useful. Django has a number of useful helper fumctions that don't have better home than the Django.utils package. It's tempting to dig into the code in Django.utils and start using things, but don't. Most of those modules are designed for internal use and their behavior or inclusion can change between Django versions.

Instead, read https://docs/djangoproject.com/en/1.8/ref/utils/ to see which modules in there are stable.

TIP: Malcolm Tredinnick on Django's Utils Package

Django core developer Malcolm Tredinnick liked to think of django.utils as being in the same theme as Batman's utility belt: indispensable tools that are used everywhere internally.

There are some gems in there that have turned into best practice:

29.3.1 django.contrib.humanize

This is a set of localized template filterers designed to give use presented data a more 'human' touch. For example it include a filter called 'intcomma' that coverts integers to strings containing commas (or periods depending on locale) every three digits.

While django.contrib.humanize's filters are useful for marking template output more attractive, we can also import each filter individually as a function. This is quite handy when procssing any sort of text, especially when used in conjunction with REST APIs.

29.3.2 django.utils.decorators,method_decorator(decorator)

Django has some really great function decorators. Many of us have written decorators for Django projects, especially when we're working with Function-Based Views. However, there comes a time when we discover that our favorite function decorator would also make sense as a method decorator. Fortunately, Django provides the method_decorator

29.3.3 django.utils.decoraotrs.decorator_from_middleware(middlware)

Middleware is a wonderful tool, but is global in nature. This can generate extra queries or other complications. Fortunately, we can isolate the use of middleware on a per-vier basis by using this view decorator.

Also see the related decorator_from_moddileware_with_args decorator.

29.3.4 django.utils.encoding.force_text(value)

This forces Django to take anything and turn it into a plain str representation on Python 3 and unicode on Python 2. It avoids the display of a Django.utils.functional.proxyobject. For more details, see Appendix D.

29.3.5 django.utils.functional.cached_property

Reinout van Rees educated us about this incredibly useful method decorator introduced in Django 1.5. What it does is cache in memory the result of a method with a single self argument as a property. This has wonderful implications in regards to performance optimization of a project. We use it in every project, enjoying how it allows us ti cache the results of expensive computations trivially.

For a description on how to use the cached_property decorator, the offical Django documentation on the subject is excellent: http://2scoops.co/1.8-cached_property

In addition to the potential performance benefits, we've used this decorator to make sure that value fetched by methods remain static over the life APIs time og their object. This has proven very useful when dealing with third-party APIs or dealing with database transactions.

WARNING: Careful Using cached_property Outside of Django

It is tempting to copy/paste the source code for cached_property for use outside of Django. However, when used outside a web framework, we discovered the code for this function has problems in multithreaded environments. Therefore, if coding outside of Django, you might want to take a look at third-party cached_property library:
    https://github.com/pydanny/cached-property
    http://www.pydanny.com/cached-property.html

29.3.6 django.utils.html.format_html(format_str, args, **kwargs)

This is similar to Python's str.format() method, except designed for building up HTML fragments. All args and kwargs are escaped before being passed to str.format() which then combine the elements.

See http://2scoops.co/1.8-format_html

29.3.7 django.utils.html.remote_tags(value, tag)

Deprecated for security season per section 26.27, Stop Using django.utils.html.remove_tags

29.3.8 django.utils.html.strip_tags(value)

When we need to accept content from users and have to strip out anything that could be HTML, this function removes those tags for we while keeping all the existing text between tags.

WARNING: Security Advisory on strip_    tags Safety

When using the strip_tags function, or the striptags template tage, make absolutely certain that the outputted content is not marked as safe. This especially applies if you have disabled automatic escaping in your templates. Reference:
https://djangoproject.com/weblog/2014/mar/22/strip-tags-advisory/

29.3.9 django.utils.six

Six is a Python 2 and 3 compatibility library by Benjamin Peterson. It's bundled directly into Django (hence its inclusion in Django's utils library), but we can also find it as an independent package for other projects.

29.3.10 django.utils.text.slugify(value)

We recommend that whatever you do, don't write your own version of the slugify() function, as any inconsistency from what Django does with this function will cause subtle yet nasty problem in our data. Instead, we use the same function that Django uses and slugify() consistently.

It is possible to use django.templates.defaultfilters.slugify() in our Python code, as this calls the function described here. Nevertheless, we like to use the function directory from Django's utils directory, as it is a more appropriate import path.

However we decide to import this function, we try to keep it consistent across a project as there is a use case for when it has to be replaced, as described in the package tip on the next page.

PACKAGE TIP: slugify and language Besides English

Tomasz Paczkowski points out that we should note that slugify() can cause problems with localization:

EXAMPLE 29.3
>>> from django.utils.text import slugify
>>> slugify(u"straße") # German
u"strae"

Fortunately. unicode-slugify, is a Mozilla foundation-supported project that addresses the issue:

EXAMPLE 29.4
>>> from slugify import slugify
>>> slugify(u"straße") # Again with German
u"straße"

There is also awesome-slugify, which shares similar functionality to unicode-slugify, but does not have Django as dependency.

29.3.11 django.utils.timezone

It's good practice for us to have time zone support enabled. Chinese are that our users live in more than one zone.

When we use Django's time zone support, date and time information is stored in the database usiformly in UTC format and convert to local time zones as needed.

29.3.12 django.utils.translation

Much of the non-English speaking world world appreciates use of this tool, as it provides Django's i18n support. See Appendix D for a more in-depth reference.

29.4 Exceptions

Django comes with a lot of exception. Most of them are used internally, but a few of them stand out becausse the way they interact with Django can be leveraged in fun and creative ways. These, and other built-in Django exceptions, are documented at https://docs.djangoproject.com/en/dev/ref/exceptions.

29.4.1 django.core.exceptions.improperlyConfigured

The purpose of this module is to inform anyone attempting to run Django that there is a config issue. It servers as the signal Django code component considered acceptable to import into settings modules. We discuss it in both chapter 5 and Appendix E.

29.4.2 django.core.exceptions.ObjectDoesNotExist

This is the base Exception from which all DoesNotExist exceptions inherit from. We've found this is a really nice tool for working with utility functions that gerneric model instances and something with them. Here is a simple example:

EXAMPLE 29.5

# core/utils.py
from django.core.exceptions import ObjectDoesNotExist

class BorkedObject(object):
    loaded = False

def generic_load_tool(model, pk):
    try:
        instance = model.objects.get(pk=pk)
    except ObjectDoesNotExist:
        return BorkedObject()
    instance.loaded = True
    return instance

Also using this exception, we can create our own variant of DJango's django.shortcuts.get_objects_or_404 function, perhaps raising a HTTP 403 exception instead of a 404:

EXAMPLE 29.6

# core/utils.py
from django.core.exceptions import MultipleObjectsReturned
from django.core.exceptions import ObjectDoesNotExist
from django.core.exceptions import PermissionDenied

def get_object_or_403(model, **kwargs):
    try:
        return model.objects.get(**kwargs)
    except ObjectDoesNotExist:
        raise PermissionDenied
    except MultipleObjectsReturned:
        raise PermissionDenied

29.4.3 django.core.exceptions.PermissionDenied

This exception is usead when users, authenticated or not, attempt to get response from places they are not meant to be. Raising it in a view will trigger the view return a django.http.HttpResponseForbildden.

This exception can prove useful to use in functions that are touching the sensitive data and components of a high-security project. It means that if something bad happend, instead of just returning a 500 exception, which may right alarms users. we simply provide a "Permission Denied" screen.

EXAMPLE 29.7
# stores/calc.py

def finance_data_adjudication(store, sales, issues):

    if store.something_not_right:
        msg = "Something is not right. Please contact the support team."
        raise PermissionDenied(msg)

    # Continue on ot perform other logic

In this case, if this function were called by a view and something was 'not right', then the PermissionDenied exception would force the view to display the project's 403 error pages. Speaking of 403 error pages, we can set this to any view we want. In the root URLConf of a project, just ass:

EXAMPLE 29.8
# urls.py

# This demonstrates the use of a custom permission denied view. The default view is django.views,default.permission_denied
handler403 = 'core.views.permission_denied_view'

As always, which, with exception-handling viewsm beacuse they handle all HTTP methods equally, we prefer to use function-based views.

29.5 Serializers and Deserializers

Whether it's for creating dat files or generating one-off simple REST APIs, Django has some great tools for working with serialization and deserialization of data of JSON, Python, YAML, and XML data. They include the capability to turn model instances into serialzied data and then return it back to model instances.

Here is how we serialize data:

EXAMPLE 29.9
# serializer_example.py
from django.core.serializers import get_Serializer

from favorites.models import Favorite

# Get and instantiate the serializer class
# The 'json' can be replaced with 'python or xml'.
# If you have pyyaml installed, you can replace ti with 
#  'pyyaml'
JSONSerializer = get_serializer("json")
serializer = JSONSerializer()

favs = Favorite.objects.filter()[:5]

# Serialize model data
serialized_data = serializer.serialize(favs)

# save the serialzed data for use in the next example
with open("data.json", "w") as f:
    f.write(serialized_Data)

Here is how we deserialize data

EXAMPLE 29.10
# deserializer_eample.py
from django.core.serializers import get_serializer
from favorites.models improt Favorite

favs = Favorite.objects.filter()[:5]

# Get and instantiate the serializer class
# The 'json' can be replaced with 'python' or 'xml'.
# If you have pyyaml installed, you can replace it with
#    'pyyaml'
JSONSerializer = get_serializer("json")
serializer = JSONSerializer()

# open the serialized data file
with open("data.txt") as f:
    serialized_data = f.read()

# deserialize model data into a gererator object
#    we'll call 'python data'
python_data = serializer.deserialize(serialized_data)

# iterate through the python_data
for element in python_data:
    # Prints 'django.core.serializers.base.DeserializedObject'
    print(type(element))

    # Elements have an 'object' that are literally instantiated
    # model instance(in this case, favorites.models.Favorite)
    print (
        element.object.pk,
        element.object.created
    )

Django already provides a command-line tool for using these serializers and deserializers: the dump data and loaddajta management commands. While we can use them, they don't grant us the same amount of control that direct code access to the serializers provides.

This brings us to something that we always need to keep in mind when using Django's built-in serializers and deserializers: they can cause problems. From painful experience, we know that they don't handle complex data structures well.

Consider these guidelines that we follow in our project:

  • Serialize data at the simplest level.
  • Any database schema change may invalidate the serialized data.
  • Don't just import serialized data. Consider using Django's form libraries to validate incoming data before saving to the database.

Let's go over some of the features provided by Django when working with specific formats:

29.5.1 django.core.serializers.json.DjangoJSONEncoder

Out of the box, Python's built-in JSON module can't handle encoding of data/time or decimal type Anyone who has done DJango for a while has run into this problem. Fortunately for all of us, Django provides a very useful JSONEncoder class. See the code example below:

EXAMPLE 29.11
# json_encoding_example.py
import json

from django.core.serializers.json import DjangoJSONEncoder
from django.utils import timezone

data = {"data": timezone.now()}

# if you don't add the DjangoJSONEncoder class then
# the json library will throw a TypeError.
json_data = json.dumps(data, cls=DjangoJsonEncoder)

print(json_data)

29.5.2 django.core.serializers.pyyaml

While powered by the third-party library, pyyaml, Django's YAML serializer handles the time conversion from Python-to-YAML that pyyaml doesn't'.

For deserialization, it also uses the yaml.safe_load() function under the hood, which means that we don't have to worry about code injection. See subsection 26.9.3 Third-Party Libraries That Can Execute Code for more detail.

29.5.3 django.core.serializers.xml_serializer

By default Django's XML serializer uses Python's build-in XML handlers. It also incorporates elements of Christian Heime's defusedxml library, protecting usage of it from XML bomb attacks. For more information, please read section 26.22, 'Grand Against XML Bombing WIth defusedxml.'

29.6 Summary

We follow the practice of putting ofen reused files into utility packages. We enjoy being able to remember where we placed our often reused code. Projects that contain a mix of core, common, util, and utils directories are just that much harder to navigate.

Django's own 'utility belt' includes a plethora od useful tools, including useful function, exceptions, and serializers. Leveraging them is on of the ways experienced Django developers accelerate development and avoid some of the tangles that can be caused by some of the very features of Django.

Now that we've covered tools to make things work, in ther next chapter we'll begin to cover sharing a project with the world.

results matching ""

    No results matching ""