How to install a Python virtual environment on macOS

This one is for my amazing designer Angélica. I should have written it before I failed to install a Python virtual environment on her machine this week.

First of all: trust me, when I tell you that you want to use a virtual environment for your Python work. Second: there are many ways to install and use virtual environments. This one works for me(TM).

Install homebrew

Homebrew is a package manager for macOS that allows us to install a current version of Python, e.g. Python 3.6 at the moment. This is what we want.

Start your terminal.app and copy and paste the installation command from the Homebrew website into it. Then hit enter.

Install Python3

When homebrew is installed, stay in the terminal.app and install Python3 using this command.

$ brew install python3

Now, the commands python3 and pip3 are available on the command line. You can check the installed Python version with

$ python3 --version
Python 3.6.0

Install virtualenvwrapper

Next, use pip3 to install the virtualenvwrapper tool (Official documentation) that makes working with virtual environments easy. I don't even know how much easier, because I only ever use virtualenvwrapper.

$ pip3 install virtualenvwrapper

Open the file /Users//.bashrc in your text editor (like SublimeText), and add the following lines at the bottom.

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

Quit and reload the terminal.app

Create a virtual environment

Finally, we can create a virtual environment. Go into our project directory (e.g. ~/code/secret_project/,

$ cd code/secret_project/

type into the terminal,

$ mkvirtualenv --python==python3.6 secret_project

and hit enter. This creates a new virtual environment in

~/.virtualenvs/secret_project

Activate virtual environment

You activate it with

$ workon secret_project

Using the pytest-mock plugin

After hearing about it from Brian Okken, I today tried out the pytest-mock plugin. It is surprisingly simple to use and useful, too.

The other day I wrote about mocks in Django views. The example test uses the with statement for patching the object.

from mock import patch

def test_detail_view(client):
    """Test the detail view for a Potato object with the Django test client."""
    potato = PotatoFactory.build()

    with patch.object(PotatoDetailView, 'get_object', return_value=potato):

        url = reverse('detail', kwargs={'pk': 1234})  # pk can be anything

        ...

This works fine when only one patch is applied, but probably gets tedious quickly with more than one.

Enter: the pytest-mock plugin and its mocker fixture. Using this fixture, the test looks much cleaner.

def test_detail_view_with_mocker(client, mocker):
    """Same test as above, but using the mocker fixture from pytest-mock."""
    potato = PotatoFactory.build()

    # This is new
    mocker.patch.object(PotatoDetailView, 'get_object', return_value=potato)

    url = reverse('detail', kwargs={'pk': 1234})

    [...]

pytest awesomeness!

django-crispy-forms and the 'cancel' button

I use crispy-forms to render my Django forms. With crispy-forms there is almost no need to write any HTML. The template for form views can look like this:

{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block content %}

{% crispy form %}

{% endblock content %}

By itself, this does not render a submit button. It has to be added to the form definition (cf. the crispy-forms documentation like this:

from potatoes.models import Potato

from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit

from django import forms


class PotatoForm(forms.ModelForm):
    """ModelForm for the Potato model."""

    class Meta:  # noqa
        model = Potato
        fields = (
            'weight',
            'variety',
        )

    def __init__(self, *args, **kwargs):
        """Initiate form with Crispy Form's FormHelper."""
        super(PotatoForm, self).__init__(*args, **kwargs)
        self.helper = FormHelper()

        # Add 'Submit' button
        self.helper.add_input(Submit('submit', 'Submit'))

A view that uses this form is my example project's PotatoCreateView:

class PotatoCreateView(CreateView):
    """Create view for the Potato model."""

    model = Potato
    form_class = PotatoForm
    template_name = 'potatoes/potato_form.html'

A submit button is nice, but I also want a cancel button. I add it to the crispy-forms helper, but that just shows a button that does the same as the submit button.

# Add 'Submit' button
self.helper.add_input(Submit('submit', 'Submit'))
self.helper.add_input(Submit('cancel', 'Cancel', css_class='btn-danger',)

I need to overwrite the view's post method to do what I want, when the cancel button is clicked. For this, I use a model mixin, because why not.

class FormActionMixin(object):

    def post(self, request, *args, **kwargs):
        """Add 'Cancel' button redirect."""
        if "cancel" in request.POST:
            url = reverse('index')     # or e.g. reverse(self.get_success_url())
            return HttpResponseRedirect(url)
        else:
            return super(FormActionMixin, self).post(request, *args, **kwargs)


class PotatoCreateView(FormActionMixin, CreateView):
    """Create view for the Potato model."""

    ...

When the cancel button is clicked, the resulting POST request includes the name attribute of the button. Overwriting the post method for this case let's me redirect the user to whatever page I want.

(note to self: research the Django way of redirecting to the previous page.)

Now I have a submit and a cancel button. But on my CreateView it complains about required form fields.

Screenshot

This has to do with HTML, and I found the solution on Coderwall: the HTML attribute formnovalidate.

self.helper.add_input(Submit('submit', 'Submit'))
self.helper.add_input(Submit(
    'cancel',
    'Cancel',
    css_class='btn-danger',
    formnovalidate='formnovalidate',
    )
)

That's it.

(note to self: I need a test for this.)

Mocking database calls in Django view tests

It took me a long time to understand the first thing about mocking in unit tests. The next few posts are intended to be a future reference to myself. Maybe you find them useful, or better, you can tell me how to do this better.

I created a simple Django project to document my solutions in working code: https://github.com/FlowFX/sturdy-potato. For the purpose of these posts I will use the models, views and tests from this project. All views are class-based views.

Why mocking? Because I want fast tests. Database calls are especially slow, and for many tests, it is not necessary to actually write to or load from the database. So I want to avoid these.

A simple view test

The Potato model has two attributes: weight and variety.

from django.db import models
from django.core.validators import MinValueValidator

class Potato(models.Model):
    """The Potato model."""

    slug = models.SlugField(unique=True)
    weight = models.IntegerField(validators=[MinValueValidator(1)])
    variety = models.CharField(max_length=255)

The URL for the detail page:

from django.conf.urls import url
from potatoes import views

urlpatterns = [
    [...]
    url(r'^potatoes/(?P<pk>[0-9]+)/$', views.PotatoDetailView.as_view(), name='detail'),
]

The view subclasses the DetailView:

from potatoes.models import Potato

from django.views.generic import DetailView

class PotatoDetailView(DetailView):
    """Detail view for the Potato object."""

    model = Potato

An simple way of testing this view is using the Django test client.

When using pytest, the test client is made available as a fixture by the pytest-django plugin. Because I don't use Django/unittest's TestCase, I need to make the test database available with the @pytest.mark.django_db decorator.

from django.urls import reverse

from potatoes.factories import PotatoFactory

import pytest


@pytest.mark.django_db
def test_detail_view(client):
    """Test the detail view for a Potato object with the Django test client."""

    # (1) GIVEN a Potato object in the database
    potato = PotatoFactory.create()  # saves to database

    # (2) WHEN calling the DetailView for this object
    url = reverse('detail', kwargs={'pk': potato.id})
    response = client.get(url)

    content = response.content.decode()
    # (3) THEN it shows the potato's ID and it's type
    assert response.status_code == 200
    assert str(potato.weight) in content
    assert potato.variety in content

What's happening here?

  1. Using Factory Boy's DjangoModelFactory, a test Potato is created and written to the database.
  2. The test client does a GET request to the URL of the details page of this Potato. This reads from the database.
  3. It is checked whether the Potato's attributes are displayed on the page.

This test hits the database twice, although I only want to test whether my view (and kind of my template) works or not. I'm pretty sure the Django ORM works fine.

View test with mock

In the test above, the object is only saved to the database so that the DetailView can read it from there. The method that reads from the database is the PotatoDetailView's get_object method.

In order to avoid the database request, I can use a so-called monkey patch that provides a return value for the method, without hitting the database.

from mock import patch

def test_detail_view(client):
    """Test the detail view for a Potato object with the Django test client."""

    # (1) GIVEN a Potato object
    potato = PotatoFactory.build()  # not saved to the database

    # (2) monkey-patching    
    with patch.object(PotatoDetailView, 'get_object', return_value=potato):

        # (3) WHEN calling the DetailView for this object
        url = reverse('detail', kwargs={'pk': 1234})  # pk can be anything
        response = client.get(url)
        content = response.content.decode()

        # THEN it shows the potato's ID and it's type
        assert response.status_code == 200
        assert str(potato.weight) in content
        assert potato.variety in content

This is the same test with just a few changes.

  1. The Potato instance is not saved to the database. (Check Factory Boy's build() vs. create() methods.)
  2. This is the fun part. The patch patch.object(PotatoDetailView, 'get_object', return_value=potato) takes the PotatoDetailView and, first of all, disables the get_object method. Second, it replaces the method by something that always returns the potato instance. Always.
  3. No matter what primary key we call the detail view with, it will always receive the test potato to work with. Which is really all we need to assert stuff.

There is no database call, no need for the django_db mark, just more speed.

For a ListView, the method that has to be replaced by the patch is get_queryset. Check out test_list_view in the example project.

Testing Django forms with pytest parameterization

Working on a largish Django project, I have to test a lot of web forms. My basic approach is to put data into the form and check if it validates. I started out using separate tests for valid and invalid input data, also thinking about for loops to handle different data sets. But you don't want to do that.

Pytest's Parametrizing offers a really neat and concise solution to this problem. Consider this simple example:

from django import forms

import pytest


class ExampleForm(forms.Form):
    name = forms.CharField(required=True)
    age = forms.IntegerField(min_value=18)


@pytest.mark.parametrize(
    'name, age, validity',
    [('Hugo', 18, True),
     ('Egon', 17, False),
     ('Balder', None, False),
     ('', 18, False),
     (None, 18, False),
     ])
def test_example_form(name, age, validity):
    form = ExampleForm(data={
        'name': name,
        'age': age,
    })

    assert form.is_valid() is validity

Here, three values are parameterized: the input for the two form fields and, given this data, whether or not the form should validate or not.

When working in a TDD-style, I start with one test case, code the form logic, then add the next test case by adding just one line of parameters. And repeat.

If you think there is a better way to test Django forms, please drop me a line on the Twitter!

DreamHost is awesome!

There are many crappy hosting providers. In fact, almost all of them are crappy.

Almost.

DreamHost is awesome!

In their standard shared-hosting plan they provide:

  • unlimited webspace
  • unlimited MySQL databases
  • unlimited email accounts
  • unlimited user accounts
    • with separate web spaces
    • with SSH access
  • free LetsEncrypt SSL certificates for every domain and subdomain
  • up-to-date PHP and MySQL versions
  • excellent and friendly customer support

(to be continued…)

They do offer domain registration as well, but I prefer using Hover for this. In cases where everything else is on DreamHost, I just switch the domain's name servers to the DreamHost name servers. And it just works.

By now, I recommend DreamHost to everyone who asks, especially to all my clients.

List comprehensions for physicists

A list comprehension provides a compact way of mapping a list to another list by applying a function to each of the elements of the list.

From "Dive Into Python 3"

Yeah. Well. Talk English to me! Please.

Until recently I had no clue what list comprehensions do or how to use them or what they're good for. Reading Dive Into Python 3 about two years ago, without much prior programming knowledge, maybe wasn't the best idea I ever had.

Today I know quite a bit more about programming in general and about Python in particular. Still, list comprehensions kept eluding me until I read From List Comprehensions to Generator Expressions by the inventor of Python himself, Guido van Rossum.

In this article, he starts by showing how lists are stated in mathematics. As a physicist, an expression like

$$ \{ x \in \mathbf{N}\ |\ x < 10 \} $$

is perfectly clear to me. It's the set of all natural numbers that are smaller than 10, i.e.:

$$ \{1, 2, 3, 4, 5, 6, 7, 8, 9\} $$

depending on your preference of zero being a natural number or not.

A python list of these integers would be:

[0,1,2,3,4,5,6,7,8,9]

Und suddenly, everything became clear to me.

[f(x) for x in S if P(x)]

is the list of the function values $f(x)$ of all $x$ in the set $\mathbf{S}$ that fulfill the condition $\mathbf{P}$.

Using a Python list comprehension, the first mathematical expression above reads:

[x for x in N if x < 10]

Of course, this list is not possible, because $\mathbf{N}$ is infinite.

The function value $f(x)$ is very handy as well. The set of the squares of all integers between 1 and 4:

$$ \{ y\ |\ y = x^{2}; x \in \mathbf{N}; 0 < x < 5 \} $$

expressed via a list comprehension is:

[x**2 for x in [1,2,3,4]]

Looking at list comprehensions in mathematical terms helped me understand them. It's a great example of one and the same idea expressed differently in different fields and thus more or less understandable by people in that field.

Run Selenium from PyCharm

In order to run the Selenium WebDriver from within PyCharm, you need to explicitly add the path of the browser driver executable.

It is in fact not necessary to add the PyCharm Selenium plugin.

My PyTest fixture for the Selenium browser now looks like this:

from selenium import webdriver
import pytest

@pytest.fixture(scope="session")
def browser():
    browser = webdriver.PhantomJS(executable_path="/usr/local/bin/phantomjs",
                                  desired_capabilities={
                                  'phantomjs.page.settings.loadImages': 'false',
                                  })
    browser.implicitly_wait(3)

    yield browser

    browser.quit()

I'm using PhantomJS as a headless browser because it is way faster than Firefox or Chrome. I wish I could disable all loading of CSS files.

Update 2017-01-18

Putting the explicit path to the browser driver executable into the test fixture causes problems when I run the same test on Snap CI. A better solution is to add the directory of the (PhantomJS) executable into PyCharm itself.

In my PyCharm Community Edition 2016.2 I go to Run >> Edit Configurations…, select the run configuration that runs my functional test and put

PATH=/usr/local/bin/

into Environment variabls. That's all. Now PyCharm finds the executable of PhantomJS in that directory and I can remove the executable_path line from the fixture.

I do not know why PyCharm doesn't just import the $PATH from my shell.

Disable CSRF validation when unit-testing a Flask app!

In my current Flask project, I wanted to test a view method that includes a form with a POST request. But it wouldn't work.

The view method looks something like this:

# views.py
@tennis.route('/tournaments/add', methods=['GET', 'POST'])
def add_tournament():
    form = AddTournamentForm()

    if form.validate_on_submit():
        t = Tournament(name=form.name.data)
        db.session.add(t)
        db.session.commit()

        flash('A tournament was added to the database.')
        return redirect(url_for('tennis.index'))

    return render_template('tennis/add_tournament.html',
                       form=form)

Flask-Testing provides a TestCase class with several useful assert methods. So the test would look like this:

# test_views.py  
from flask_testing import TestCase

class TestTennisViews(TestCase):    
    []

    def test_tennis_add_tournament(self):
        # WHEN submitting correct form data to the add_tournament view
        r = self.client.post('/tennis/tournaments/add',
                             data=dict(
                                 name='Chicharito Open',
                             ))

        # THEN it redirects to the tennis overview page
        self.assertRedirects(r,'/tennis/')
        # and it shows a message
        self.assertMessageFlashed('A tournament was added to the database.')
        # and there is a tournament in the database
        assert Tournament.query.count() > 0
        # and it has the correct tournament
        t = Tournament.query.first()
        assert t.name == 'Chicharito Open'

assertRedirects checks if the status code is 301 or 302. In my tests, it was always 200, and the test failed. Of course, nothing was added to the database, either.

I googled a lot, without success. I tried all combinations of the following keywords, among others:

flask
unit test
wtforms
POST request
testing

The solution hit me when taking a break (it works!). It lies in the form.validate_on_submit() part of the view function. validate_on_submit() does two things:

  1. It validates the form fields according to the validators specified in the form definition.
  2. It validates a CSRF token.

Guess what: there is no CSRF token generated when executing a POST request directly in the test.

The solution is easy, as FLASK-WTF provides a configuration option to disable CSRF:

# config.py

[]

class TestingConfig(Config):
    TESTING = True
    WTF_CSRF_ENABLED = False

    []

class FunctionalTestingConfig(TestingConfig)
    WTF_CSRF_ENABLED = False

Of course, only disable CSRF in your test configuration! I created a separate configuration for my functional tests, so that CSRF is enabled in those. When using the browser with Selenium, the CSRF token is provided by the WTForms form on the web page.