Populate your Django test database with pytest fixtures

I'm working on a side project that uses data from an external API. For performance reasons I store this data in a local database. But when running pytest, all my tests always start with a clean database. That's not good, as I need the data to run many of the tests, and adding it from the API is very time consuming.

Of course, Django has a solution for this, confusingly called fixtures, and pytest has a way to use Django fixtures in a custom pytest fixture to populate the database with initial test data.

Because it took me a while to find this, I document it here. It works like this:

Dump the data

Using Django's own dumpdata management command, you dump all or selected tables from your local database into a JSON file in a subfolder of the app named fixtures. My Django app is called potatoes, and I want the data for my two models Potato and SturdyPotato.

$ ./manage.py dumpdata potatoes.Potato potatoes.SturdyPotato -o potatoes/fixtures/potatoes_data.json

Load the data

The corresponding loaddata command can be used with pytest's django_db_setup fixture to load the data into the test database.

# tests/conftest.py

import pytest

from django.core.management import call_command

@pytest.fixture(scope='session')
def django_db_setup(django_db_setup, django_db_blocker):
    with django_db_blocker.unblock():
        call_command('loaddata', 'potatoes_data.json')

Use pytest fixture

Now, in every test that needs it, I use this session-scoped fixture, and the test data is available.

# tests/test_models.py

def test_my_potatoes(db, django_db_setup):
    # GIVEN a full database of potatoes, as provided by the django_db_setup fixture
    all_my_potatoes = Potato.objects.all()

Disable Django/Python logging with pytest fixture

Yesterday, I added Sentry error tracking to my Django app, and configured it to register every log entry with level INFO and above. Now, everytime I ran my test suite, there were events logged with Sentry that I didn't really care about. Naturally, I wanted to disable the default logging behavior for tests.

StackOverflow, naturally, provides part of the answer:

logging.disable(logging.CRITICAL)

will disable all logging calls with levels less severe than or equal to CRITICAL.

(http://stackoverflow.com/a/5255760)

But how to run this on every test? Pytest to the rescue! I use an autouse fixture:

  • if an autouse fixture is defined in a conftest.py file then all tests in all test modules below its directory will invoke the fixture.

And this is what I put into my conftest.py files:

@pytest.fixture(autouse=True)
def disable_logging():
    """Disable logging in all tests."""
    logging.disable(logging.INFO)

That's it. Love it!

How to install a Python virtual environment on macOS

This one is for my amazing designer Angélica. I should have written it before I failed to install a Python virtual environment on her machine this week.

First of all: trust me, when I tell you that you want to use a virtual environment for your Python work. Second: there are many ways to install and use virtual environments. This one works for me(TM).

Install homebrew

Homebrew is a package manager for macOS that allows us to install a current version of Python, e.g. Python 3.6 at the moment. This is what we want.

Start your terminal.app and copy and paste the installation command from the Homebrew website into it. Then hit enter.

Install Python3

When homebrew is installed, stay in the terminal.app and install Python3 using this command.

$ brew install python3

Now, the commands python3 and pip3 are available on the command line. You can check the installed Python version with

$ python3 --version
Python 3.6.0

Install virtualenvwrapper

Next, use pip3 to install the virtualenvwrapper tool (Official documentation) that makes working with virtual environments easy. I don't even know how much easier, because I only ever use virtualenvwrapper.

$ pip3 install virtualenvwrapper

Open the file /Users/<your_username>/.bashrc in your text editor (like SublimeText), and add the following lines at the bottom.

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

Quit and reload the terminal.app

Create a virtual environment

Finally, we can create a virtual environment. Go into our project directory (e.g. ~/code/secret_project/,

$ cd code/secret_project/

type into the terminal,

$ mkvirtualenv --python==python3.6 secret_project

and hit enter. This creates a new virtual environment in

~/.virtualenvs/secret_project

Activate virtual environment

You activate it with

$ workon secret_project

Using the pytest-mock plugin

After hearing about it from Brian Okken, I today tried out the pytest-mock plugin. It is surprisingly simple to use and useful, too.

The other day I wrote about mocks in Django views. The example test uses the with statement for patching the object.

from mock import patch

def test_detail_view(client):
    """Test the detail view for a Potato object with the Django test client."""
    potato = PotatoFactory.build()

    with patch.object(PotatoDetailView, 'get_object', return_value=potato):

        url = reverse('detail', kwargs={'pk': 1234})  # pk can be anything

        ...

This works fine when only one patch is applied, but probably gets tedious quickly with more than one.

Enter: the pytest-mock plugin and its mocker fixture. Using this fixture, the test looks much cleaner.

def test_detail_view_with_mocker(client, mocker):
    """Same test as above, but using the mocker fixture from pytest-mock."""
    potato = PotatoFactory.build()

    # This is new
    mocker.patch.object(PotatoDetailView, 'get_object', return_value=potato)

    url = reverse('detail', kwargs={'pk': 1234})

    [...]

pytest awesomeness!

django-crispy-forms and the 'cancel' button

I use crispy-forms to render my Django forms. With crispy-forms there is almost no need to write any HTML. The template for form views can look like this:

{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block content %}

{% crispy form %}

{% endblock content %}

By itself, this does not render a submit button. It has to be added to the form definition (cf. the crispy-forms documentation like this:

from potatoes.models import Potato

from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit

from django import forms


class PotatoForm(forms.ModelForm):
    """ModelForm for the Potato model."""

    class Meta:  # noqa
        model = Potato
        fields = (
            'weight',
            'variety',
        )

    def __init__(self, *args, **kwargs):
        """Initiate form with Crispy Form's FormHelper."""
        super(PotatoForm, self).__init__(*args, **kwargs)
        self.helper = FormHelper()

        # Add 'Submit' button
        self.helper.add_input(Submit('submit', 'Submit'))

A view that uses this form is my example project's PotatoCreateView:

class PotatoCreateView(CreateView):
    """Create view for the Potato model."""

    model = Potato
    form_class = PotatoForm
    template_name = 'potatoes/potato_form.html'

A submit button is nice, but I also want a cancel button. I add it to the crispy-forms helper, but that just shows a button that does the same as the submit button.

# Add 'Submit' button
self.helper.add_input(Submit('submit', 'Submit'))
self.helper.add_input(Submit('cancel', 'Cancel', css_class='btn-danger',)

I need to overwrite the view's post method to do what I want, when the cancel button is clicked. For this, I use a model mixin, because why not.

class FormActionMixin(object):

    def post(self, request, *args, **kwargs):
        """Add 'Cancel' button redirect."""
        if "cancel" in request.POST:
            url = reverse('index')     # or e.g. reverse(self.get_success_url())
            return HttpResponseRedirect(url)
        else:
            return super(FormActionMixin, self).post(request, *args, **kwargs)


class PotatoCreateView(FormActionMixin, CreateView):
    """Create view for the Potato model."""

    ...

When the cancel button is clicked, the resulting POST request includes the name attribute of the button. Overwriting the post method for this case let's me redirect the user to whatever page I want.

(note to self: research the Django way of redirecting to the previous page.)

Now I have a submit and a cancel button. But on my CreateView it complains about required form fields.

Screenshot

This has to do with HTML, and I found the solution on Coderwall: the HTML attribute formnovalidate.

self.helper.add_input(Submit('submit', 'Submit'))
self.helper.add_input(Submit(
    'cancel',
    'Cancel',
    css_class='btn-danger',
    formnovalidate='formnovalidate',
    )
)

That's it.

(note to self: I need a test for this.)

Mocking database calls in Django view tests

It took me a long time to understand the first thing about mocking in unit tests. The next few posts are intended to be a future reference to myself. Maybe you find them useful, or better, you can tell me how to do this better.

I created a simple Django project to document my solutions in working code: https://github.com/FlowFX/sturdy-potato. For the purpose of these posts I will use the models, views and tests from this project. All views are class-based views.

Why mocking? Because I want fast tests. Database calls are especially slow, and for many tests, it is not necessary to actually write to or load from the database. So I want to avoid these.

A simple view test

The Potato model has two attributes: weight and variety.

from django.db import models
from django.core.validators import MinValueValidator

class Potato(models.Model):
    """The Potato model."""

    slug = models.SlugField(unique=True)
    weight = models.IntegerField(validators=[MinValueValidator(1)])
    variety = models.CharField(max_length=255)

The URL for the detail page:

from django.conf.urls import url
from potatoes import views

urlpatterns = [
    [...]
    url(r'^potatoes/(?P<pk>[0-9]+)/$', views.PotatoDetailView.as_view(), name='detail'),
]

The view subclasses the DetailView:

from potatoes.models import Potato

from django.views.generic import DetailView

class PotatoDetailView(DetailView):
    """Detail view for the Potato object."""

    model = Potato

An simple way of testing this view is using the Django test client.

When using pytest, the test client is made available as a fixture by the pytest-django plugin. Because I don't use Django/unittest's TestCase, I need to make the test database available with the @pytest.mark.django_db decorator.

from django.urls import reverse

from potatoes.factories import PotatoFactory

import pytest


@pytest.mark.django_db
def test_detail_view(client):
    """Test the detail view for a Potato object with the Django test client."""

    # (1) GIVEN a Potato object in the database
    potato = PotatoFactory.create()  # saves to database

    # (2) WHEN calling the DetailView for this object
    url = reverse('detail', kwargs={'pk': potato.id})
    response = client.get(url)

    content = response.content.decode()
    # (3) THEN it shows the potato's ID and it's type
    assert response.status_code == 200
    assert str(potato.weight) in content
    assert potato.variety in content

What's happening here?

  1. Using Factory Boy's DjangoModelFactory, a test Potato is created and written to the database.
  2. The test client does a GET request to the URL of the details page of this Potato. This reads from the database.
  3. It is checked whether the Potato's attributes are displayed on the page.

This test hits the database twice, although I only want to test whether my view (and kind of my template) works or not. I'm pretty sure the Django ORM works fine.

View test with mock

In the test above, the object is only saved to the database so that the DetailView can read it from there. The method that reads from the database is the PotatoDetailView's get_object method.

In order to avoid the database request, I can use a so-called monkey patch that provides a return value for the method, without hitting the database.

from mock import patch

def test_detail_view(client):
    """Test the detail view for a Potato object with the Django test client."""

    # (1) GIVEN a Potato object
    potato = PotatoFactory.build()  # not saved to the database

    # (2) monkey-patching    
    with patch.object(PotatoDetailView, 'get_object', return_value=potato):

        # (3) WHEN calling the DetailView for this object
        url = reverse('detail', kwargs={'pk': 1234})  # pk can be anything
        response = client.get(url)
        content = response.content.decode()

        # THEN it shows the potato's ID and it's type
        assert response.status_code == 200
        assert str(potato.weight) in content
        assert potato.variety in content

This is the same test with just a few changes.

  1. The Potato instance is not saved to the database. (Check Factory Boy's build() vs. create() methods.)
  2. This is the fun part. The patch patch.object(PotatoDetailView, 'get_object', return_value=potato) takes the PotatoDetailView and, first of all, disables the get_object method. Second, it replaces the method by something that always returns the potato instance. Always.
  3. No matter what primary key we call the detail view with, it will always receive the test potato to work with. Which is really all we need to assert stuff.

There is no database call, no need for the django_db mark, just more speed.

For a ListView, the method that has to be replaced by the patch is get_queryset. Check out test_list_view in the example project.