Why do we care?

During web development project we use Django's management command named fixtures to backup and restore data. Command dumpdata to save data to json file and loaddata to restore/load from json file. We restore (or load) data into the application running on Kubernetes. Please note we do this way to backup/restore only for small data sets (this could be up to 10k records but it hardly depends on data model), because conversion from db to json and back takes long with bigger data sets.

So let me explain how we use Django management command to load data into application running on Kubernetes cluster.

Dump your data

First thing is to properly dump your data. While we develop on Windows this could be tricky sometimes especially when we talking about character encoding. I mean files created on Windows can have encoding not supported by loaddata management commnand executed on Linux based container within K8s cluster.

Cross-platform PowerShell Core

In the cross-platform PowerShell Core edition, UTF-8NoBOM is the default encoding. If you're using PowerShell Core you will have UTF-8 files by default. But if you need to be sure, set this encoding explicitly:

python.exe .\manage.py dumpdata | Out-File .\dunp.data.json -Encoding UTF8NoBOM`

Windows built-in PowerShell

python.exe .\manage.py dumpdata | Out-File .\dump.data.json -Encoding ascii`

Load data

Finally we have invoke kubectl to execute command in our pod. Note that this is one container pod so I don't need to point to specific container. It will be done using following command, where I pipe dump.data.json file into stdout, then execute manage.py loaddata reading from stdin Do not skip - character at the end or you will be 😲 suprised!

[vagrant@localhost]$ cat ./dump.data.json | kubectl exec --stdin <pod.name.here> -n career -- python ./manage.py loaddata --format=json -e contenttypes -e auth.permission -e admin.logentry -

After some time ⌚ when success something like folowing have to be shown:

Installed 1521 object(s) (of 1582) from 1 fixture(s)

Problem with encoding

If you found following error while loading data, this means your dump file have BOM encoding which is not supported by loader. If so please follow above instructions to dumpdata properly.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer
    objects = json.loads(stream_or_string)
  File "/usr/local/lib/python3.8/json/__init__.py", line 337, in loads
    raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

Intelligent Services Blog