Django fixtures on Kubernetes
Why do we care?
During web development project we use Django's management command named fixtures to backup and restore data. Command
dumpdata to save data to json file and
loaddata to restore/load from json file. We restore (or load) data into the application running on Kubernetes. Please note we do this way to backup/restore only for small data sets (this could be up to 10k records but it hardly depends on data model), because conversion from db to json and back takes long with bigger data sets.
So let me explain how we use Django management command to load data into application running on Kubernetes cluster.
Dump your data
First thing is to properly dump your data. While we develop on Windows this could be tricky sometimes especially when we talking about character encoding. I mean files created on Windows can have encoding not supported by
loaddata management commnand executed on Linux based container within K8s cluster.
Cross-platform PowerShell Core
In the cross-platform PowerShell Core edition, UTF-8NoBOM is the default encoding. If you're using PowerShell Core you will have UTF-8 files by default. But if you need to be sure, set this encoding explicitly:
python.exe .\manage.py dumpdata | Out-File .\dunp.data.json -Encoding UTF8NoBOM`
Windows built-in PowerShell
python.exe .\manage.py dumpdata | Out-File .\dump.data.json -Encoding ascii`
Finally we have invoke
kubectl to execute command in our pod. Note that this is one container pod so I don't need to point to specific container. It will be done using following command, where I pipe
dump.data.json file into
stdout, then execute
manage.py loaddata reading from
stdin Do not skip
- character at the end or you will be 😲 suprised!
[vagrant@localhost]$ cat ./dump.data.json | kubectl exec --stdin <pod.name.here> -n career -- python ./manage.py loaddata --format=json -e contenttypes -e auth.permission -e admin.logentry -
After some time ⌚ when success something like folowing have to be shown:
Installed 1521 object(s) (of 1582) from 1 fixture(s)
Problem with encoding
If you found following error while loading data, this means your dump file have BOM encoding which is not supported by loader. If so please follow above instructions to
Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 68, in Deserializer objects = json.loads(stream_or_string) File "/usr/local/lib/python3.8/json/__init__.py", line 337, in loads raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)", json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)