Update August 2018

Working a full-time job in an office is something different. Add 3 hours of commute each day, and my life is a lot different than it was a year ago. I firmly believe that I'm about as productive now than I was when I was working fewer hours but alone in my home office. But that's for another day.

More time to and at work means a lot less time for other things, which is something I am still struggling to get used to. Combine that with having moved across the Atlantic, finding and moving into a new apartment, and dealing with German bureaucracy! All things considered, I guess I'm dealing with it pretty well.

Still, I had to pause a lot of activities during the last four months, including this blog. As I'm learning a new programming language (Ruby) and a new web framework (Rails), there should be enough fun topics to write about (There are). Just like last year with Django and software testing. But who has the time? Not me at the moment. At least I played my first gig with my old band two weeks ago. That was good, and it won't be the last time.

Lastly, in case you're reading this and wonder why, after almost four months back in Germany, I haven't managed to come visit you: I'm sorry, I'm working on it. In the meantime, our guestroom is large, has an even larger bed and is open to you anytime!

Moving back to Germany

My search for a new job has been successful. I'm starting a new position as a Ruby-On-Rails developer with a company at the Frankfurt Airport on May 2nd (I will commute from Mannheim). In the end it took me 4 months, 7 applications, 5 interviews and 2 job offers. This doesn't sound too bad, but still, it's not an experience I want to repeat anytime soon.

Check my Now page to see where I'm currently at in the process of moving back to Germany.

Wait. Rails?

Yupp. Rails.

I'll need to update the about page.

To hear what it feels like to leave Mexico and start something completely new, you'll have to listen to one of the next episodes of Several Ways To Live. There Nick and I will discuss the coming-home part of being an expat.

Let me just quickly give you an incomplete and unordered list of things I am looking forward to in Germany:

  • Döner Kebap
  • my friends
  • my family
  • silence
  • good beer
  • bike tours
  • chaos events
  • my Reggae band
  • clean air
  • forests
  • good bread
  • cross-country skiing
  • drinking beer in public
  • Rindsrouladen
  • affordable good cheese
  • affordable good sausage
  • public swimming pools
  • lactose-free butter

Oh, yes. Döner!!

How to listen to your audio books using your favorite podcast app

This post talks about two things:

  1. how to save backup copies of your Audible audio books
  2. how to create an podcast feed of your audio books

Update 2018-05-27: Audible stopped offering to download the audio files in "Format 4". There are solutions out there, but I haven't gotten around to studying them, yet.

Update 2018-05-29: The solution is on GitHub and is called KrumpetPirate/AAXtoMP3. Thanks to @igami@social.tchncs.de for the tipp!

Why?

I buy all ebooks from the Amazon Kindle store, because I know I can save a backup copy of each book in a non-DRM format. This is not about sharing ebooks but rather about making sure that I can keep a copy of my books in a digital format that I will be able to read in the future, too. Also, I want to be able to use any ebook reading app or device I want to, not just Amazon's Kindle apps. Incidentally, I read most of my ebooks on a Kindle Paperwhite because it's just a great device. I sync the books via the Calibre app, though, and the Kindle device is not registered with my Amazon account.

Anyways.

I want the same for my for Audible books. I love audio books, but I hate that I have to rely on the Audible app, with no protection against someone closing down my Amazon account or what not. Turns out, there is a way to do that, without even removing any DRM, because, apparently, there is no DRM on these files.

How to backup your Audible books

Creating a backup mp3 of audible books is surprisingly simple. Thanks to @octotherp for this tipp!

  1. Go to your Audible library.
  2. In the upper right hand corner, select "Format 4" as the download audio format.
  3. Use ffmpeg to convert the downloaded files to mp3.

To convert a single audio book, the shell command is:

$  ffmpeg -i IAmABook.aa -codec copy IAmABook.mp3

To convert all downloaded audio files, use a shell for loop:

$ for file in *.aa; do ffmpeg -i "$file" -codec copy "${file/%aa/mp3}"; done

This converts all files with the extension .aa and renames them with the extension .mp3.

How to create a podcast feed of your audio books

To create a podcast (RSS) feed from a collection of mp3s is surprisingly easy, too. Because others have solved the problem for us.

I'm using the PHP script DirCaster and my shared hosting account at my favorite web host DreamHost.

Download and unzip DirCaster in a directory

    $ wget http://dircaster.org/DirCasterV09k.zip
    $ mkdir audiobooks
    $ cd audiobooks
    $ unzip ../DirCasterV09k.zip

Edit config_inc.php to set the title and description of the feed. I also add a feed image:

$rssImageUrlTAG   = "https://myaudiobookfeed.com/books.png";

To keep thinks neat, I put the mp3 files into a subfolder books and edit the config_inc.php accordingly:

$mediaDir = "./books"

The audiobooks directory now has to be served by a web server. A simple one that runs PHP is enough though. I put my feed at a random subdomain that no-one is going to find by accident.

At least my favorite podcast app Pocket Casts has no problem subscribing to it. I also tried adding password protection using HTTP auth, but that didn't work.

If you have questions about any of this, please send me a message.

The Xochimilco Trilogy

A few months ago I started a new podcast project Several Ways to Live in Mexico City with my friend Nick. We talk about Mexico and food and culture. And now I think we have created our first highlight of the series, a 3 episode expedition to the town of Xochimilco, where we talked to my friends Jahshua and Edgar.

Episode 8 is a short preview, recorded on the way back from Xochimilco. Episode 9 talks about the history and culture of Xochimilco. And episode 10 starts tackling some questions about -isms that we have observed but could not explain just yet.

  1. SW008 Segundo Piso
  2. SW009 Xochimilco I
  3. SW010 Xochimilco II: Chilling on the Chinampas

If you have comments or questions, please don't hesitate to talk to us on Twitter or Mastodon!

A short work history. Looking for more.

Update 04/04/2018: There's been a development.

My wife Kathrin will not like this post. She will find it self-deprecating and underselling my abilities. But this is my blog, so I don't care.

TL;DR: I am looking for a job as a Django/Python web developer or a test (automation) engineer, working remotely or in the Rhein-Neckar area in Germany. Starting now.

2017

2017 was a pretty good year for me workwise. I spent it working as a Django developer for the local LegalTech startup Lexa.mx. The basic idea of Lexa is that of a Mexican LegalZoom.

While being the sole developer, I also led our product development efforts. These included understanding the structures of Mexican legal agreements, translating the requirements into web forms and making those useful and into a good experience for the users. It was a very interesting inter-disciplinary project that included 3 Mexican lawyers and a designer.

At the same time, this was my first serious Django project. I had to learn a lot. And I did, thanks to mainly the Django documentation, Stack Overflow and the best of them all: ccbv.co.uk.

My favorite part of the Lexa app is how we use Jinja2 templates and LaTeX to dynamically generate the contract PDFs. I've wanted to do this for a long time, ever since a friend and I got the idea for Unkenmathe.

The frontend is ... subject to future improvements. I don't feel comfortable writing CSS because it is so much trial and error for me. For a little bit of dynamic behavior, I sprinkled some VueJS and jQuery on top of everything, mainly the forms, which works nicely. But before I do more JavaScript development I first have to learn how to test that stuff. Developing the backend following the TDD methodology and then testing all the frontend functions manually in the browser feels very wrong.

I also set up and maintained our continuous delivery pipeline using GitHub, CircleCI and PythonAnywhere.

During my time at Lexa I essentially worked remotely. Mexico City is so big that every in-person meeting requires a long commute for everyone involved. But what can I say? I loved it! Working from my quiet home on a schedule that I set myself suits me well, and I've never been more productive.

The main reason why I do not continue working for Lexa is that Kathrin and I will definitely move back to Germany this year, and Lexa is looking to open an office in the city. Remote work that goes beyond the "home office" is pretty much unheard of in Mexico and does not fit with the work culture, which is a shame.

For those not interested in my further work history and how I got to where I am today, here's what else I'm bringing to the table.

2009

A little bit of history.

I left University in early 2009 with a degree in physics, the good-old German "Diplom" (equivalent to a Master of Science). From time to time I am reminded of what I learned in University, and that is a particular way of thinking, ingrained into us students by endless exercises in higher Mathematics. One example is this story of how I finally understood list comprehensions.

But 2009 was not a good year for anybody looking for a job, let alone someone without any real world skills. My only coding skills back then consisted of a little bit of Matlab, numerically solving ordinary differential equations. Sounds amazing, but it's really not. At least I got to experience working as a bike messenger for a few months.

This was also the year when I remembered my 90's experimentations with HTML and started playing around with WordPress.

2011

2011 I got my first "real" job.

Looking for work. Am a physicist. Can do everything.

Yes, this tweet actually worked and landed me a job at a SAP consulting firm, solely based on my degree. It lasted about 18 months before both the company and I were happy to part ways.

Considering that my tasks involved business processes and code, two things I can be very passionate about, it's still a little bit of a mystery why I never got the hang of it, why the work never clicked for me. My best explanation is that I not only lacked mentorship but also there was virtually no relevant documentation that I could study on my own. So much of the company's knowledge was never explicitly written down that it made it too hard even for me to acquire enough of it to succeed in that particular industry.

2013

My favorite job during these years was working for the Ganter brewery where I learned a skill that I am very proud of: tapping the perfect beer. Seriously.

All the while I was developing WordPress sites for clients and administering web hosting accounts. The one service that I am still providing to a number of clients today is securing and backing up their WordPress installations.

2014

2014 I not only met Kathrin, but we also moved to Mexico City. Our first year in Mexico I worked at the German school where Kathrin is teaching. There I finally learned that teaching is not for me. And I didn't even teach. Also, I can't handle saying good morning to 30 people each and every day.

But this way I got to know my friend Angélica with whom I have since collaborated on various web projects using WordPress and Kirby CMS. She also later worked with me on the Lexa project.

2016

I had wanted to learn to program for a long time, but it had never clicked. Reading "programming" books never worked for me.

2016 it finally did click, and it took a book written for non-developers: Hello Web App by Tracy Osborn. It's an introduction to Django for designers and provided me with enough knowledge to later advance to the two other Django books that I cannot recommend enough: Obey the Testing Goat Test-Driven Development with Python by Harry Percival and Two Scoops of Django by Audrey Roy Greenfeld and Daniel Roy Greenfeld.

I've since started working on a couple personal side projects, most notably Unkenmathe and Reggae CDMX which are both still works in progress.

In December of 2016 Angélica and I then sat down with the 3 lawyers that had the idea for Lexa.mx, and that would be my primary occupation for the next 12 months.

What else I'm bringing to the table

I am looking for a job as a Django/Python web developer.

After working on many projects as the sole developer, it's way past time for me to join a team of developers and learn that part of software and web development.

  • I want to learn from others and teach them whenever I can.
  • I can take charge of a project when needed but prefer working among equals.
  • I love to collaborate with experts from non-technical fields – analyzing their problems, and developing and implementing technical solutions.
  • I'm pretty high on the TDD band wagon, and my testing framework of choice is pytest.

As I very much enjoy automated software testing, I am also interested in working as a test engineer. I am well aware that this includes much more than writing automated tests. But it's something that I believe I would enjoy doing, and I definitely know that I can learn it and be good at it.

Timetable

My current plan is to remain in Mexico while Kathrin finishes the school year and for us to move back to the Rhein-Neckar area in Germany in early July. But I am happy to relocate earlier as necessary.

If all this makes you want to think about maybe hiring me, or if you know a friend who might, then let's talk! I also have a detailed CV prepared if that's your sort of thing.

Until then I'll be happily working on Udacity's Full Stack Web Development Nanodegree program to fill some knowledge gaps.

Update 13/01/2018: minor edits.

Speeding up Django unit tests with SQLite, reuse-db and RAMDisk

Recently, Harry wrote a post about how to speed up Django unit-testing by using a persistent in-memory SQLite database.

While Harry uses the default Django testrunner and a Linux machine, I use pytest with pytest-django on a Mac. This changes a few things, and this post will show the differences. So it's very specific to:

  • Django
  • pytest
  • macOS

Create and mount the RAMDisk

/dev/shm is a Linux thing. But Harry already provided the link to a working solution on Stack Overflow to how to do this on macOS.

$ hdiutil attach -nomount ram://$((2 * 1024 * SIZE_IN_MB))
/dev/disk2

$ diskutil eraseVolume HFS+ RAMDisk /dev/disk2

This creates an in-memory Volume that can store the persistent SQLite database.

Use SQLite

Maybe I'm doing this all wrong, but because I use pytest, I configure the test database in a separate configuration file like this.

# config/settings/testing.py
from .common import *

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': ':memory:',
        'TEST': {}
    }
}

Use it in-memory

What --keepdb is for the default testrunner, --reuse-db is for pytest-django. I usually set this as the default for all test runs in pytest.ini.

# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE=config.settings.testing
addopts = --reuse-db

To not re-use the database, pytest-django's argument is --create-db. So Harry's

if 'keepdb' in sys.argv:

becomes

if not 'create-db' in sys.argv:

in my case. To use the RAMDisk:

# config/settings/testing.py

[...]

import sys
if not 'create-db' in sys.argv:
    # and this allows you to use --reuse-db to skip re-creating the db,
    # even faster!
    DATABASES['default']['TEST']['NAME'] = '/Volumes/RAMDisk/myfunnyproject.test.db.sqlite3'

When there's no RAMDisk

If there is no RAMDisk, for example after a reboot, then the test run fails as soon as it hits the database. Obviously.

>       conn = Database.connect(**conn_params)
E       django.db.utils.OperationalError: unable to open database file

I just throw another if statement in there to check for the existence of the RAMDisk.

if os.path.isdir('/Volumes/RAMDisk') and not 'create-db' in sys.argv:

Summary

You can check a full test configuration here:

https://github.com/FlowFX/unkenmathe.de/blob/master/src/config/settings/testing.py

Test-driven Python learning

Bugs are nice because I can learn from them. Recently I found a bug that taught me about floats in Python and how not to use them. I found it thanks to an automated test.

The test

In a legal contract I had to print a monetary amount in a format similar to

MXN $11,600.00 (eleven thousand, six hundred pesos 00/100 cents)

My tests check the LaTeX source of the resulting PDF, looking for the correctly formatted string 11,600.00, which represents an amount of 11,000 something plus a VAT of 16%, and also for the string eleven thousand, six hundred.

It essentially goes like this:

from app.models import LeaseAgreement

def test_contract_states_rent_plus_vat():
    """Test correct statement of rent amount with and without IVA (VAT (USt (MWSt)))."""

    contract = LeaseAgreement.build(    
        rent_amount=Decimal(10000.00)
    )

    tex = contract.render_tex()

    assert '10,000.00' in tex
    assert '11,600.00' in tex
    assert 'eleven thousand, six hundred' in tex

The first two asserts did not fail, the third one did. My code actually printed

MXN $11,600.00 (eleven thousand, five hundred and ninety-nine pesos 00/100 cents)

The bug

Turns out I had misused Python's Decimal class to calculate the amount plus VAT. Like this:

>>> x = Decimal(10000) * Decimal(1.16)

Printing the number in the desired format works fine.

>>> print('{:,.2f}'.format(x))
11,600.00

But rounding down for only printing the amount without decimal places does not.

>>> num2words(int(x))
'eleven thousand, five hundred and ninety-nine'

That is because floats are not exact.

>>> Decimal(1.16)
Decimal('1.1599999999999999200639422269887290894985198974609375')
>>> int(Decimal(1.16)*10000)
11599

The fix

Working with monetary amounts that round to 2 digits requires care. Thanks to Rami and others, I now know that I should have used e.g.

Decimal('1.16')

and

Decimal('1.16').quantize(Decimal('.01'), rounding=ROUND_HALF_UP)

when apropriate.

Test Django with Selenium, pytest and user authentication

When testing a Django app with Selenium, how do you authenticate the user and test pages that require to be logged in?

Of course: StackOverflow has the answer.

The following is the actual code that I use to make this work with pytest. It requires pytest-django.

pytest fixtures

In pytest everything is contained in neat test fixtures.

Browser

The first fixture provides a broser/webdriver instance with an anonymous user. The default way of using Selenium.

# system_tests/conftest.py

from selenium import webdriver

@pytest.fixture(scope='module')
def browser(request):
    """Provide a selenium webdriver instance."""
    # SetUp
    options = webdriver.ChromeOptions()
    options.add_argument('headless')

    browser_ = webdriver.Chrome(chrome_options=options)

    yield browser_

    # TearDown
    browser_.quit()

User

To be able to authenticate, I need a user in the database. Using Factory Boy:

# system_tests/conftest.py

from django.contrib.auth.hashers import make_password
from accounts.factories import UserFactory

TESTEMAIL = 'test-user@example.com'
TESTPASSWORD = 'a-super-secret-password'

@pytest.fixture()
def user(db):
    """Add a test user to the database."""
    user_ = UserFactory.create(
        name='I am a test user',
        email=TESTEMAIL,
        password=make_password(TESTPASSWORD),
    )

    return user_

Authenticated browser

To get the authenticated browser, the first two fixtures are required, plus the Django TestClient and LiveServer fixtures which, for pytest, are provided by pytest-django. Using the code from SO:

# system_tests/conftest.py

@pytest.fixture()
def authenticated_browser(browser, client, live_server, user):
    """Return a browser instance with logged-in user session."""
    client.login(email=TESTEMAIL, password=TESTPASSWORD)
    cookie = client.cookies['sessionid']

    browser.get(live_server.url)
    browser.add_cookie({'name': 'sessionid', 'value': cookie.value, 'secure': False, 'path': '/'})
    browser.refresh()

    return browser

The tests

Now the Selenium test can use the authenticated_browser fixture.

# system_tests/test_django.py

def test_django_with_authenticated_user(live_server, authenticated_browser):
    """A Selenium test."""
    browser = authenticated_browser

    # Open the home page
    browser.get(live_server.url)
    

To test logging in and out of the app, I use the unauthenticated browser instance plus the user fixture.

# system_tests/test_django.py

def test_login_of_anonymous_user(live_server, browser, user):
    # Open the home page
    browser.get(live_server.url)

    # Click the 'login' button
    browser.find_element_by_id('id_link_to_login')).click()
    

Summary

These are the pytest fixtures that I use to test a Django app with an authenticated user.

If there is a better way to do this, please tell me! I want to know.

pytest parameter matrices

A few months ago I explained how I efficiently test Django forms with pytest parameterization. Last week, I learned a new trick from Raphael Pierzina's post about ids for fixtures and parametrize, which is:

You you can add multiple parametrization markers to a test function which then create a test parameter matrix.

The list of test cases can thus be written much more clearly. Compare the example code from my previous post:

from django import forms

import pytest


class ExampleForm(forms.Form):
    name = forms.CharField(required=True)
    age = forms.IntegerField(min_value=18)


@pytest.mark.parametrize(
    'name, age, validity',
    [('Hugo', 18, True),
     ('Egon', 17, False),
     ('Balder', None, False),
     ('', 18, False),
     (None, 18, False),
     ])

def test_example_form(name, age, validity):
    form = ExampleForm(data={
        'name': name,
        'age': age,
    })

    assert form.is_valid() is validity

to the same test using multiple parameterization markers:

@pytest.mark.parametrize(
    'name, valid_name',
    [
        ('Hugo', True),
        ('', False),
        (None, False),
    ]
)
@pytest.mark.parametrize(
    'age, valid_age',
    [
        ('18', True),
        ('17', False),
        (None, False),
    ]
 )
def test_example_form(name, age, valid_name, valid_age):
    form = ExampleForm(data={
        'name': name,
        'age': age,
    })

    assert form.is_valid() is (valid_name and valid_age)

This tests all 9 possible combinations of the three test cases each for name and age. Maybe that's overkill, but on the other hand I can be sure that I touch all the relevant combinations.

Take note of the last line that checks that the form only validates when both input parameters are valid.

The main advantage I see is legibility. For each form field, I only have to understand the list of test parameters for exactly that field, and not any additional combinations with other fields.

I'm loving it!