Stef Smeets

PyPalmSens - EIS measurements and Circuit Fitting

2026-11-04T00:00:00+00:00

PyPalmSens - How to get Started

2026-10-09T00:00:00+00:00

Why every device in my house was called 172.18.0.1

2026-09-20T00:00:00+00:00

I run AdGuard Home in Docker to filter DNS queries on my home network. It blocks ads and trackers for every device in my house. The AdGuard dashboard shows a nice query log, so you can see which device is asking for which domain and set per-client rules for the smart TV that really doesn't need to talk to seventeenthousand tracking servers (looking at you, LG).

This worked for a while, until it didn't. In my query log, every single client was listed as 172.18.0.1.

It shows up as one client, no matter how many devices are making requests. Although it did not make the DNS filtering any worse, it did make any sort of client analysis or statistics useless. Over a 2-day debugging session (using the GLM 5.3 model on the side) I finally looked into this.

Docker rewrites your IPs

My AdGuard container ran on a user-defined bridge network, which is shared with my reverse proxy. When a device on my network sends a DNS query to my server, Docker's NAT rewrites the packet before handing it to the container. The source address becomes the Docker bridge gateway (172.18.0.1). This is well-known docker networking behaviour. This means that AdGuard only ever sees this IP, and it cannot resolve the source.

This is a well-known issue, just use network_mode: host. I'm sure this work well, except that I run rootless Docker. With rootless, even host networking happens inside RootlessKit's own network namespace. The container never actually shares the host's real network stack, so this wasn't going to work.

RootlessKit V3

Then I found something interesting. Docker 29.5 bundles RootlessKit v3, which adds support for propagating the original source IP of incoming connections and I was on 29.7.2.

This did not work either:

The propagation is TCP only, whereas DNS is UDP.
Even for TCP, it doesn't work with the default userland-proxy enabled.

Pasta

The actual documented fix (on the same page) for source IPs in rootless Docker is switching the daemon to the pasta network driver with the implicit port driver. Pasta (from the passt project) was specifically designed to solve source IP preservation in rootless environments by forwarding traffic in a way that preserves real source addresses for both TCP and UDP.

Configuring it was pretty simple:

# ~/.config/systemd/user/docker.service.d/override.conf
[Service]
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_NET=pasta"
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=implicit"

Installing passt and restarting docker:

sudo apt install passt
systemctl --user daemon-reload
systemctl --user restart docker
docker compose down && docker compose up -d

Unfortunately, DNS stopped working entirely afterwards. The dashboard wasn't reachable either.

Privileged ports

To debug what was going on, GLM 5.3 kept suggesting ss commands. I had never heard of this, but apparently it's "another utility to investigate sockets".

sudo ss -tlnp | grep -E ':53|:3000'

This showed that port 3000 was bound by pasta.avx2 now, instead of rootlesskit. But it showed nothing for port 53? Nothing. Port 53 is a privileged port (< 1024). Therefore, I run this on start-up so that AdGuard (as a Docker container) gets forwarded DNS requests:

sudo setcap cap_net_bind_service=ep $(which rootlesskit)

This means that rootlesskit could bind port 53 as my unprivileged user. However, because Pasta replaces RootlessKit as the network stack handler, these capabilities don't apply to it. Since file capabilities don't propagate to child processes, Pasta tried to bind port 53 without the necessary permissions, whereas port 3000 doesn't require any special configuration.

Although I could change the command to give pasta access to privireleged ports, at this point I had hit two experimental edge cases, and I don't want to overconfigure my server.

The practical solution: Run it on the host

At the same time I realized that AdGuard home ships a single binary to deploy on my server. This is the recommended installation path. Running a UDP DNS server on a privileged port inside rootless Docker means fighting the kernel and port forwarding.

So I decided to ditch Docker and run AdGuard Home with systemd. It's a single static binary, and it installs itself as a service:

curl -LO https://github.com/AdguardTeam/AdGuardHome/releases/latest/download/AdGuardHome_linux_amd64.tar.gz
tar -xzf AdGuardHome_linux_amd64.tar.gz

I could re-use my existing config and query data by passing the work dir and config file:

sudo ./AdGuardHome -s install -c ~/docker/adguard/conf/adguardhome.yaml -w ~/docker/adguard/work

This worked pretty much out of the box. Running ss confirmed that this works:

$ sudo ss -ulpn | grep ':53 '
UNCONN 0 0 *:53 *:* users:(("AdGuardHome",pid=432113,fd=8))

AdGuard itself, on port 53, no extra configuration, no sysctls, no port forwarders. And the query log started filling with real client IPs 😅

A bad gateway

Caddy still runs in Docker and proxies the dashboard, so I updated the Caddyfile:

http://adguard.home.arpa {
    reverse_proxy host.docker.internal:3000
}

Firefox answered with a 502 Bad Gateway. It turned out that host.docker.internal is injected automatically in rootful Docker. In rootless mode, it often exists but points to the loopback of the user namespace rather than the host network, making it unreachable from the container. I could fix this by just using my server's LAN IP:

http://adguard.home.arpa {
    reverse_proxy 192.168.0.123:3000
}

I reloaded Caddy, hit refresh, and everything worked as expected.

Final setup

When I set up this server years ago, my goal was to learn Docker. So I did everything in Docker. This works well for complex systems such as home assistant. However, my focus on Docker got in the way here. AdGuard's setup and deployment with systemd is so simple. Static binaries are great.

My final setup:

AdGuard Home runs natively on the host via systemd, sees every client's real IP, per-client filtering works
Caddy (still in Docker) proxies the dashboard
DNS runs directly to the host on port 53.

TL;DR: To preserve source IPs in rootless Docker, use the pasta network driver. However, for simple binaries like AdGuard Home, running natively via systemd is often much simpler and avoids configuration and privilege management.

GEMDAT: a Python toolkit for site-resolved diffusion analysis in solid-state molecular dynamics

2026-05-16T00:00:00+00:00

How to mount a drive on Linux

2026-03-07T00:00:00+00:00

This is a small blog post with how to set up a drive on Debian.

Find the drive / device location:

lsblk

Make drive point (it can be whatever, on my system I use /mnt):

cd /mnt
sudo mkdir data2

Mount the drive:

sudo mount /dev/sdb1 /mnt/data2

That's it, the drive is now available under /mnt/data2.

Automounting

To automount the drive after reboot, add it to the fstab table:

First, get the GUID:

$ sudo blkid
/dev/sdb1: LABEL="Data" UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Note the UUID variable. Add the line to /etc/fstab.

$ nano /etc/fstab

Add:

/dev/disk/by-uuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx /mnt/data2 auto nosuid,nodev,n>

No.

2025-12-27T00:00:00+00:00

I don't want your newsletter.

I don't want your app.

I don't want your cookies.

I don't want to create an account.

I don't want to do a monthly donation to your charity (I will happily do a one-time donation though!)

I don't want to see your advertisements.

I don't want to have a 'no-obligation' conversation.

I don't want your loyalty plan.

I don't want your stamps.

I don't want your customer card.

I don't want your discount coupon.

I don't want your premium subscription.

I am not looking for a new challenge or opportunity.

I don't want to work with C++ for your innovative data & AI projects or your high-tech clients. I don't even know C++!

I still don't want to work with you if you send me a reminder; if I ignore your first message I'm also going to ignore the second one.

I don't want to decide now for a discount.

I don't want your exclusive offer.

I don't want give my credit card number for a free trial.

I don't want to give you a 'like or subscribe'.

I don't want to fill our your questionnaire.

I don't want to rate your app.

I don't want whatever you are calling me for.

I don't want to fill out your customer satisfaction survey.

I don't want to opt-in.

I don't want to share my usage data to 'help improve my experience'.

I don't want to SEO my website.

I don't want to try the 'new experience'.

I don't want to use your chatbot.

I don't want to share your article with all my friends.

I don't want to disable my adblocker.

I don't want to use co-pilot.

I don't want to have the 'best browsing experience'.

I don't want to change my default browser.

Yes, I'm really sure.

Thanks!

Pesticides 2024

2025-11-26T00:00:00+00:00

Ik was nieuwsgierig naar de hoeveelheid pesticides in de producten die wij gebruiken. Er zijn namelijk genoeg redenen om te letten op pesticide gebruik. Zo is er nog veel onbekend over het effect van pesticides, en met name de combinatie van 'pesticide cocktails'. Dit betekent niet dat pesticides per definitie schadelijk zijn, er zijn immers duidelijke wettelijke normen voor residuen, maar langdurige directe blootstelling door bv boeren wordt in verband gebracht met verhoogd risico op Parkinson. Daarnaast is overmatig pesticide gebruik (in de niet biologische teelt wordt preventief gespoten) desastreus voor de biodiversiteit, het bodemleven, en de waterkwaliteit, terwijl er prima biologisch geteeld kan worden. Een van de eisen voor biologische teelt is dat er niet preventief gespoten mag worden, waardoor het gebruikt van bestrijdingsmiddelen vele malen lager ligt.

PAN (Pesticide Action Network) publiceert de Pesticide eetwijzer met een ranglijst van groenten en fruit waar het meeste aantal pesticides bij gebruikt worden.

Ik wilde iets meer in de data duiken. Nu blijkt het dat de NVWA jaarlijks de individuele analyseresultaten van steekproeven naar residuen van bestrijdingsmiddelen op levensmiddelen publiceert: Individuele analyseresultaten van monsters van levensmiddelen 2024 (NVWA).

De data zijn alleen beschikbaar als pdf. Echter zijn er een aantal Python packages beschikbaar die beweren tabellen uit pdfs te kunnen lezen.

Ik kwam vrij snel bij pdfplumber uit, en deze gaf direct een bruikbaar resultaat. Min of meer kwam ik hier op uit:

import pdfplumber
import pandas as pd

pdf = pdfplumber.open(
   "individuele-analyseresultaten-van-monsters-levensmiddelen-2024.pdf"
)

for i, page in enumerate(pdf.pages):
    table = page.extract_table()

    for row in table:
        rows.append(row)

df = pd.DataFrame(rows)

df.to_csv("data.csv")S

Ik moest handmatig nog horizontale lijnen toevoegen, en soms versprongen de gevonden kolommen per pagina. Dit kwam omdat sommige producten over meerdere pagina's uitgerekt zijn en pdfplumber per pagina de tabel eruit trekt. Daarna heb ik de data nog wat opgeschoond om er wat makkelijker mee te kunnen werken.

De data staat hier: data2024.csv

Mijn code staat hier: github.com/stefsmeets/pesticides

En de bron data is hier te vinden: Individuele analyseresultaten van monsters van levensmiddelen 2024 (NVWA)

De tabellen hieronder laten per productgroep de meeste gebruikte bestrijdingsmiddelen zien. Hierbij geeft '% obs' aan in hoeveel steekproeven dit bestrijdingsmiddel was gevonden. 'mg/kg' is de gemiddelde hoeveelheid per analyse, en MRL is de wettelijk bepaalde maximale residulimit.

Productgroepen met minder dan 3 steekproeven heb ik niet meegenomen en analyses die minder dan 20% voorkomt ook niet.

aardappelen

analyse	type	% obs	mg/kg (gem.)	MRL
propamocarb	Fungicide	75%	0.01	0.3
fluopicolide	Fungicide	22%	0.01	0.03
fluxapyroxad	Fungicide	22%	0.01	0.3

Aantal metingen: 36
Gem. aantal pesticides: 1.7
Aantal analyses: 60 ( conform: 58, niet conform*: 2 )

aardbeien

analyse	type	% obs	mg/kg (gem.)	MRL
fluopyram	Fungicide	74%	0.12	2.0
trifloxystrobin	Fungicide	59%	0.06	1.0
boscalid	Fungicide	48%	0.29	6.0
cyprodinil	Fungicide	41%	0.22	5.0
fludioxonil	Fungicide	41%	0.19	4.0
mepanipyrim	-	37%	0.34	3.0
pyraclostrobin	Fungicide	33%	0.08	1.5
bupirimaat	-	30%	0.15	1.5
difenoconazool	Fungicide	22%	0.04	2.0
penconazool	Fungicide	22%	0.03	0.5
ethirimol	-	22%	0.02	0.3
fenhexamid	Fungicide	22%	0.30	10.0

Aantal metingen: 27
Gem. aantal pesticides: 6.3
Aantal analyses: 169 ( conform: 165, niet conform*: 4 )

abrikozen

analyse	type	% obs	mg/kg (gem.)	MRL
captan (som)	Fungicide	89%	0.63	30.0
dodine	-	89%	0.21	0.5
acetamiprid	Insecticide	78%	0.01	4.0
cypermethrin	Insecticide	67%	0.05	10.0
tebuconazool	Fungicide	67%	0.01	0.6
trifloxystrobin	Fungicide	44%	0.01	15.0
fenvaleraat (som)	-	44%	0.02	1.0
thiacloprid	-	22%	0.01	2.5
cyhalothrin‐lambda	-	22%	0.01	0.75

Aantal metingen: 9
Gem. aantal pesticides: 5.8
Aantal analyses: 52 ( conform: 52 )

ananassen

analyse	type	% obs	mg/kg (gem.)	MRL
fludioxonil	Fungicide	100%	0.53	7.0

Aantal metingen: 17
Gem. aantal pesticides: 1.2
Aantal analyses: 21 ( conform: 20, niet conform: 1 )

andijvie

analyse	type	% obs	mg/kg (gem.)	MRL
spirotetramat (som)	Insecticide	75%	0.22	7.0
pirimicarb	-	50%	0.06	1.0
boscalid	Fungicide	25%	4.40	50.0
azoxystrobin	Fungicide	25%	1.50	10.0
fenpropidin	-	25%	0.01	0.01
chloorantraniliprole	Insecticide	25%	0.01	20.0
fluopyram	Fungicide	25%	0.04	2.0
pyraclostrobin	Fungicide	25%	0.56	0.4
spinosad	Insecticide	25%	0.34	3.0
thiabendazool	Fungicide	25%	0.01	0.01

Aantal metingen: 4
Gem. aantal pesticides: 3.2
Aantal analyses: 13 ( conform: 12, niet conform*: 1 )

appelen

analyse	type	% obs	mg/kg (gem.)	MRL
captan (som)	Fungicide	95%	0.70	10.0
fludioxonil	Fungicide	71%	0.06	5.0
boscalid	Fungicide	44%	0.08	2.0
pyraclostrobin	Fungicide	35%	0.04	0.5
cyprodinil	Fungicide	24%	0.15	2.0
pyrimethanil	Fungicide	23%	0.65	15.0

Aantal metingen: 62
Gem. aantal pesticides: 3.6
Aantal analyses: 225 ( conform: 225 )

aubergines

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	75%	0.49	5.0
bromide	Fumigant/bodemsterilisator	69%	1.88	30.0
fluopyram	Fungicide	44%	0.03	0.4

Aantal metingen: 32
Gem. aantal pesticides: 3.7
Aantal analyses: 119 ( conform: 117, niet conform: 2 )

avocados

analyse	type	% obs	mg/kg (gem.)	MRL
thiabendazool	Fungicide	75%	0.48	20.0

Aantal metingen: 24
Gem. aantal pesticides: 1.2
Aantal analyses: 29 ( conform: 28, niet conform*: 1 )

baby‐en kleutervoeding

analyse	type	% obs	mg/kg (gem.)	MRL
BACs (som)	-	75%	0.00	0.01
chloraat	Herbicide/ontsmettingsmiddel	25%	0.01	0.01

Aantal metingen: 4
Gem. aantal pesticides: 1.0
Aantal analyses: 4 ( conform: 4 )

bakbananen

analyse	type	% obs	mg/kg (gem.)	MRL
azoxystrobin	Fungicide	100%	0.12	2.0
thiabendazool	Fungicide	67%	0.30	6.0
bifenthrin	Insecticide	33%	0.01	0.1
chloorpyrifos‐ethyl	Insecticide	33%	0.03	0.01
koper	Fungicide/bactericide	33%	0.74	20.0
pyriproxyfen	Insecticide	33%	0.07	0.7

Aantal metingen: 3
Gem. aantal pesticides: 3.0
Aantal analyses: 9 ( conform: 8, niet conform: 1 )

bananen

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	87%	0.81	20.0
azoxystrobin	Fungicide	83%	0.20	2.0
bromide	Fumigant/bodemsterilisator	53%	1.28	50.0
fenpropimorf	-	43%	0.02	0.6
myclobutanil	Fungicide	37%	0.11	3.0
bifenthrin	Insecticide	30%	0.03	0.1
pyriproxyfen	Insecticide	27%	0.13	0.7
thiabendazool	Fungicide	23%	0.23	6.0

Aantal metingen: 30
Gem. aantal pesticides: 4.3
Aantal analyses: 130 ( conform: 127, niet conform*: 3 )

basilicum

analyse	type	% obs	mg/kg (gem.)	MRL
dimethomorf	Fungicide	75%	1.29	50.0
metalaxyl	Fungicide	75%	0.26	15.0
azoxystrobin	Fungicide	50%	1.01	350.0
difenoconazool	Fungicide	50%	0.05	50.0
pyraclostrobin	Fungicide	50%	0.87	10.0
captan (som)	Fungicide	50%	0.16	0.3
chloorpyrifos‐ethyl	Insecticide	50%	0.02	0.05
thiofanaat‐methyl	-	50%	0.38	0.5
cyhalothrin‐lambda	-	25%	0.01	3.5
boscalid	Fungicide	25%	0.20	250.0
carbendazim	Fungicide	25%	0.23	0.5
haloxyfop (som)	-	25%	0.01	0.1
ethyleenoxide (som)	-	25%	0.15	0.25
propiconazool	-	25%	0.01	0.1
myclobutanil	Fungicide	25%	0.05	0.25

Aantal metingen: 4
Gem. aantal pesticides: 6.2
Aantal analyses: 25 ( conform: 24, niet conform*: 1 )

bataten (zoete aardappelen)

analyse	type	% obs	mg/kg (gem.)	MRL
fludioxonil	Fungicide	76%	0.86	10.0

Aantal metingen: 17
Gem. aantal pesticides: 1.4
Aantal analyses: 24 ( conform: 18, niet conform: 4, niet conform*: 2 )

blauwe bessen

analyse	type	% obs	mg/kg (gem.)	MRL
boscalid	Fungicide	59%	0.16	15.0
fludioxonil	Fungicide	47%	0.14	4.0
acetamiprid	Insecticide	35%	0.06	2.0
cyprodinil	Fungicide	29%	0.09	8.0
azoxystrobin	Fungicide	24%	0.09	5.0

Aantal metingen: 17
Gem. aantal pesticides: 3.0
Aantal analyses: 51 ( conform: 50, niet conform*: 1 )

bleekselderij

analyse	type	% obs	mg/kg (gem.)	MRL
difenoconazool	Fungicide	83%	0.04	7.0
azoxystrobin	Fungicide	67%	0.01	15.0
spirotetramat (som)	Insecticide	50%	0.01	4.0
fluxapyroxad	Fungicide	50%	0.01	9.0

Aantal metingen: 6
Gem. aantal pesticides: 2.8
Aantal analyses: 17 ( conform: 17 )

bloemkolen

analyse	type	% obs	mg/kg (gem.)	MRL
spirotetramat (som)	Insecticide	100%	0.09	1.0

Aantal metingen: 5
Gem. aantal pesticides: 1.0
Aantal analyses: 5 ( conform: 5 )

bonen (droog)

analyse	type	% obs	mg/kg (gem.)	MRL
acetamiprid	Insecticide	29%	0.01	0.15
carbendazim	Fungicide	29%	0.02	0.1
thiamethoxam	Insecticide	29%	0.07	0.04
haloxyfop (som)	-	29%	0.03	0.15
clothianidin	Insecticide	29%	0.01	0.02

Aantal metingen: 7
Gem. aantal pesticides: 2.6
Aantal analyses: 18 ( conform: 13, niet conform: 3, niet conform*: 2 )

bosuien/groene uien en stengeluien

analyse	type	% obs	mg/kg (gem.)	MRL
dimethomorf	Fungicide	75%	0.16	9.0
ametoctradin	Fungicide	50%	0.73	20.0
azoxystrobin	Fungicide	50%	0.11	10.0
cyprodinil	Fungicide	25%	0.20	0.8
difenoconazool	Fungicide	25%	0.01	9.0
fludioxonil	Fungicide	25%	0.06	5.0
fluopicolide	Fungicide	25%	1.30	10.0
fluopyram	Fungicide	25%	0.01	3.0
oxathiapiproline	-	25%	0.03	2.0
propamocarb	Fungicide	25%	6.60	30.0
pyraclostrobin	Fungicide	25%	0.02	1.5
spinosad	Insecticide	25%	0.01	4.0

Aantal metingen: 4
Gem. aantal pesticides: 4.0
Aantal analyses: 16 ( conform: 16 )

broccoli

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	59%	0.54	20.0
bromide	Fumigant/bodemsterilisator	56%	1.94	30.0
spirotetramat (som)	Insecticide	33%	0.04	1.0
difenoconazool	Fungicide	23%	0.02	1.0
propamocarb	Fungicide	21%	0.02	3.0

Aantal metingen: 39
Gem. aantal pesticides: 2.9
Aantal analyses: 112 ( conform: 110, niet conform*: 2 )

carambolas

analyse	type	% obs	mg/kg (gem.)	MRL
cypermethrin	Insecticide	100%	0.03	0.2
difenoconazool	Fungicide	75%	0.01	0.1

Aantal metingen: 4
Gem. aantal pesticides: 1.8
Aantal analyses: 7 ( conform: 7 )

cavendish‐bananen

analyse	type	% obs	mg/kg (gem.)	MRL
azoxystrobin	Fungicide	75%	0.47	2.0
bifenthrin	Insecticide	75%	0.01	0.1
koper	Fungicide/bactericide	75%	0.66	20.0
pyriproxyfen	Insecticide	75%	0.06	0.7
fenpropimorf	-	50%	0.01	0.6
captan (som)	Fungicide	25%	0.01	0.03
bromide	Fumigant/bodemsterilisator	25%	0.32	50.0
myclobutanil	Fungicide	25%	0.11	3.0
spinosad	Insecticide	25%	0.05	2.0
thiabendazool	Fungicide	25%	0.14	6.0

Aantal metingen: 4
Gem. aantal pesticides: 4.8
Aantal analyses: 19 ( conform: 19 )

champignons

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	100%	1.86	20.0
bromide	Fumigant/bodemsterilisator	22%	0.63	30.0

Aantal metingen: 27
Gem. aantal pesticides: 1.4
Aantal analyses: 39 ( conform: 37, niet conform: 1, niet conform*: 1 )

chayotes, vruchtgroente

analyse	type	% obs	mg/kg (gem.)	MRL
flupyradifurone	Insecticide	67%	0.01	0.6
fluopicolide	Fungicide	33%	0.01	0.5
fluopyram	Fungicide	33%	0.02	0.6

Aantal metingen: 3
Gem. aantal pesticides: 1.3
Aantal analyses: 4 ( conform: 4 )

chilipepers

analyse	type	% obs	mg/kg (gem.)	MRL
spirotetramat (som)	Insecticide	43%	0.05	1.0
cypermethrin	Insecticide	36%	0.04	0.5
difenoconazool	Fungicide	36%	0.12	0.9
azoxystrobin	Fungicide	29%	0.15	3.0
fluxapyroxad	Fungicide	21%	0.05	0.6
carbendazim	Fungicide	21%	0.27	0.1
thiabendazool	Fungicide	21%	0.01	0.01
pyraclostrobin	Fungicide	21%	0.06	0.5
imidacloprid	Insecticide	21%	0.03	4.5

Aantal metingen: 14
Gem. aantal pesticides: 5.4
Aantal analyses: 75 ( conform: 61, niet conform: 9, niet conform*: 5 )

chinese kool/petsai

analyse	type	% obs	mg/kg (gem.)	MRL
spirotetramat (som)	Insecticide	57%	0.15	7.0
thiabendazool	Fungicide	29%	0.01	0.01

Aantal metingen: 7
Gem. aantal pesticides: 1.6
Aantal analyses: 11 ( conform: 10, niet conform*: 1 )

citroenen

analyse	type	% obs	mg/kg (gem.)	MRL
imazalil	Fungicide	88%	1.70	5.0
hexythiazox	Acaricide	62%	0.03	0.5
spirotetramat (som)	Insecticide	62%	0.03	0.5
pyrimethanil	Fungicide	56%	1.59	8.0
pyriproxyfen	Insecticide	44%	0.04	0.6
fludioxonil	Fungicide	38%	1.52	10.0
thiabendazool	Fungicide	25%	0.66	7.0
fenpyroximaat	-	25%	0.03	0.5

Aantal metingen: 16
Gem. aantal pesticides: 5.3
Aantal analyses: 85 ( conform: 84, niet conform*: 1 )

clementines

analyse	type	% obs	mg/kg (gem.)	MRL
pyriproxyfen	Insecticide	75%	0.01	0.7
spirotetramat (som)	Insecticide	75%	0.02	0.5
acetamiprid	Insecticide	50%	0.02	0.9
hexythiazox	Acaricide	50%	0.02	0.5
pyrimethanil	Fungicide	50%	1.25	8.0
fenpyroximaat	-	25%	0.02	0.5
2,4‐D	Herbicide	25%	0.01	1.0
cyhalothrin‐lambda	-	25%	0.03	0.2
pyridaben	-	25%	0.01	0.3
pyraclostrobin	Fungicide	25%	0.03	2.0
imazalil	Fungicide	25%	3.40	5.0
thiabendazool	Fungicide	25%	1.10	7.0

Aantal metingen: 4
Gem. aantal pesticides: 4.8
Aantal analyses: 19 ( conform: 19 )

courgettes

analyse	type	% obs	mg/kg (gem.)	MRL
propamocarb	Fungicide	35%	0.14	5.0
flonicamid (som)	-	35%	0.04	0.5
acetamiprid	Insecticide	29%	0.02	0.3
fluopyram	Fungicide	24%	0.03	0.6

Aantal metingen: 17
Gem. aantal pesticides: 2.4
Aantal analyses: 40 ( conform: 40 )

erwten (met peul, vers)

analyse	type	% obs	mg/kg (gem.)	MRL
azoxystrobin	Fungicide	77%	0.06	3.0
tebuconazool	Fungicide	46%	0.03	2.0
deltamethrin	Insecticide	23%	0.01	0.2
difenoconazool	Fungicide	23%	0.01	1.0

Aantal metingen: 13
Gem. aantal pesticides: 2.8
Aantal analyses: 37 ( conform: 35, niet conform: 1, niet conform*: 1 )

frambozen (geel en rood)

analyse	type	% obs	mg/kg (gem.)	MRL
boscalid	Fungicide	75%	0.07	10.0
pyraclostrobin	Fungicide	50%	0.01	3.0
azoxystrobin	Fungicide	25%	0.05	5.0
clofentezin	-	25%	0.01	3.0
bifenazaat (som)	-	25%	0.06	7.0
cyprodinil	Fungicide	25%	0.20	3.0
fenhexamid	Fungicide	25%	0.48	15.0
fludioxonil	Fungicide	25%	0.20	5.0
pyrimethanil	Fungicide	25%	0.01	15.0

Aantal metingen: 4
Gem. aantal pesticides: 3.0
Aantal analyses: 12 ( conform: 12 )

gehakt

analyse	type	% obs	mg/kg (gem.)	MRL
chloraat	Herbicide/ontsmettingsmiddel	100%	0.05	0.05

Aantal metingen: 6
Gem. aantal pesticides: 1.0
Aantal analyses: 6 ( conform: 4, niet conform*: 2 )

gerst

analyse	type	% obs	mg/kg (gem.)	MRL
deltamethrin	Insecticide	100%	0.23	2.0
pirimifos‐methyl	-	80%	0.16	5.0

Aantal metingen: 5
Gem. aantal pesticides: 1.8
Aantal analyses: 9 ( conform: 9 )

granaatappels

analyse	type	% obs	mg/kg (gem.)	MRL
fludioxonil	Fungicide	38%	0.49	3.0

Aantal metingen: 16
Gem. aantal pesticides: 1.9
Aantal analyses: 31 ( conform: 24, niet conform*: 5, niet conform: 2 )

grapefruits

analyse	type	% obs	mg/kg (gem.)	MRL
imazalil	Fungicide	90%	1.29	4.0
thiabendazool	Fungicide	60%	1.10	7.0
pyrimethanil	Fungicide	58%	0.74	8.0
koper	Fungicide/bactericide	52%	0.41	20.0
methoxyfenozide	Insecticide	40%	0.04	2.0
pyraclostrobin	Fungicide	33%	0.03	2.0
bromide	Fumigant/bodemsterilisator	31%	1.65	30.0
pyriproxyfen	Insecticide	29%	0.01	0.7
fosetyl (som)	Fungicide	27%	1.52	75.0
azoxystrobin	Fungicide	25%	0.05	15.0
spirotetramat (som)	Insecticide	25%	0.01	0.5

Aantal metingen: 52
Gem. aantal pesticides: 5.8
Aantal analyses: 304 ( conform: 301, niet conform: 3 )

hop

analyse	type	% obs	mg/kg (gem.)	MRL
boscalid	Fungicide	100%	3.55	80.0
metrafenon	Fungicide	100%	2.81	80.0
azoxystrobin	Fungicide	83%	1.21	40.0
ametoctradin	Fungicide	83%	1.25	90.0
dimethomorf	Fungicide	83%	3.18	80.0
mandipropamid	Fungicide	83%	8.47	90.0
pyraclostrobin	Fungicide	83%	0.45	15.0
fluopyram	Fungicide	67%	0.41	60.0
spirotetramat (som)	Insecticide	67%	0.41	15.0
myclobutanil	Fungicide	50%	0.81	6.0

Aantal metingen: 6
Gem. aantal pesticides: 8.3
Aantal analyses: 50 ( conform: 50 )

ijsbergsla

analyse	type	% obs	mg/kg (gem.)	MRL
spirotetramat (som)	Insecticide	100%	0.02	7.0
boscalid	Fungicide	40%	0.02	50.0
acetamiprid	Insecticide	40%	0.01	1.5
fluopyram	Fungicide	40%	0.01	15.0
propamocarb	Fungicide	40%	0.02	40.0
metalaxyl	Fungicide	20%	0.04	3.0
trifloxystrobin	Fungicide	20%	0.01	15.0

Aantal metingen: 5
Gem. aantal pesticides: 3.0
Aantal analyses: 15 ( conform: 15 )

kaas hard

analyse	type	% obs	mg/kg (gem.)	MRL
chloraat	Herbicide/ontsmettingsmiddel	100%	0.05	0.1

Aantal metingen: 3
Gem. aantal pesticides: 1.0
Aantal analyses: 3 ( conform: 3 )

kakis/japanse persimoenen

analyse	type	% obs	mg/kg (gem.)	MRL
difenoconazool	Fungicide	75%	0.02	0.8
cyhalothrin‐lambda	-	25%	0.01	0.09
sulfoxaflor	Insecticide	25%	0.01	0.3

Aantal metingen: 4
Gem. aantal pesticides: 1.2
Aantal analyses: 5 ( conform: 5 )

kersen (zoet)

analyse	type	% obs	mg/kg (gem.)	MRL
acetamiprid	Insecticide	71%	0.03	1.5
fludioxonil	Fungicide	71%	0.44	5.0
tebuconazool	Fungicide	71%	0.21	1.0
fluopyram	Fungicide	57%	0.02	2.0
boscalid	Fungicide	43%	0.07	5.0
fosmet	-	29%	0.08	0.01

Aantal metingen: 7
Gem. aantal pesticides: 5.1
Aantal analyses: 36 ( conform: 35, niet conform: 1 )

kerstomaten

analyse	type	% obs	mg/kg (gem.)	MRL
mandipropamid	Fungicide	75%	0.02	3.0
difenoconazool	Fungicide	50%	0.02	2.0
fluopyram	Fungicide	50%	0.10	0.5
sulfoxaflor	Insecticide	50%	0.01	0.3
boscalid	Fungicide	25%	0.01	3.0
azoxystrobin	Fungicide	25%	0.01	3.0
fenpyrazamine	-	25%	0.01	3.0
cyprodinil	Fungicide	25%	0.12	1.5
chloorantraniliprole	Insecticide	25%	0.01	0.6
fludioxonil	Fungicide	25%	0.06	3.0
propamocarb	Fungicide	25%	0.14	4.0
spiromesifen	-	25%	0.01	1.0
spirotetramat (som)	Insecticide	25%	0.01	1.0
trifloxystrobin	Fungicide	25%	0.05	0.7

Aantal metingen: 4
Gem. aantal pesticides: 4.8
Aantal analyses: 19 ( conform: 19 )

kippeneieren

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	100%	0.61	2.0

Aantal metingen: 21
Gem. aantal pesticides: 1.0
Aantal analyses: 21 ( conform: 21 )

kiwis (geel, groen, rood)

analyse	type	% obs	mg/kg (gem.)	MRL
fludioxonil	Fungicide	100%	1.03	15.0
fenhexamid	Fungicide	33%	0.18	15.0

Aantal metingen: 6
Gem. aantal pesticides: 1.5
Aantal analyses: 9 ( conform: 9 )

knolselderij

analyse	type	% obs	mg/kg (gem.)	MRL
fluopyram	Fungicide	100%	0.02	0.4
difenoconazool	Fungicide	100%	0.02	2.0
prosulfocarb	-	75%	0.01	0.08
azoxystrobin	Fungicide	50%	0.01	1.0
cyhalothrin‐lambda	-	25%	0.01	0.07
fluxapyroxad	Fungicide	25%	0.01	0.9

Aantal metingen: 4
Gem. aantal pesticides: 3.8
Aantal analyses: 15 ( conform: 15 )

knolvenkel

analyse	type	% obs	mg/kg (gem.)	MRL
boscalid	Fungicide	75%	0.01	9.0
azoxystrobin	Fungicide	25%	0.01	10.0
cyprodinil	Fungicide	25%	0.09	4.0
difenoconazool	Fungicide	25%	0.02	5.0
fenylfenol‐2	Fungicide/bactericide	25%	0.01	0.01
fludioxonil	Fungicide	25%	0.01	1.5
penthiopyrad	-	25%	0.04	20.0
pyraclostrobin	Fungicide	25%	0.01	1.5

Aantal metingen: 4
Gem. aantal pesticides: 2.5
Aantal analyses: 10 ( conform: 10 )

koffiebonen

analyse	type	% obs	mg/kg (gem.)	MRL
clothianidin	Insecticide	57%	0.02	0.05
thiamethoxam	Insecticide	57%	0.03	0.2
imidacloprid	Insecticide	29%	0.02	1.0
chloorpyrifos‐ethyl	Insecticide	29%	0.04	0.01

Aantal metingen: 7
Gem. aantal pesticides: 2.3
Aantal analyses: 16 ( conform: 12, niet conform: 3, niet conform*: 1 )

kokosnoten

analyse	type	% obs	mg/kg (gem.)	MRL
anthrachinon	-	100%	0.15	0.1
fenylfenol‐2	Fungicide/bactericide	100%	0.24	0.05
bifenyl	-	33%	0.14	0.05

Aantal metingen: 3
Gem. aantal pesticides: 2.3
Aantal analyses: 7 ( niet conform: 4, niet conform*: 2, conform: 1 )

komkommers

analyse	type	% obs	mg/kg (gem.)	MRL
propamocarb	Fungicide	50%	0.35	5.0
cyprodinil	Fungicide	38%	0.06	0.5
cyazofamid	-	38%	0.03	0.2
flonicamid (som)	-	38%	0.04	0.5
spinosad	Insecticide	38%	0.07	0.3
fludioxonil	Fungicide	25%	0.04	0.4
fluopyram	Fungicide	25%	0.03	0.6
ametoctradin	Fungicide	25%	0.04	2.0

Aantal metingen: 8
Gem. aantal pesticides: 3.2
Aantal analyses: 26 ( conform: 26 )

krent

analyse	type	% obs	mg/kg (gem.)	MRL
boscalid	Fungicide	100%	0.13	25.0
fluxapyroxad	Fungicide	100%	0.15	15.0
myclobutanil	Fungicide	75%	0.01	7.5
trifloxystrobin	Fungicide	75%	0.03	15.0
fluopyram	Fungicide	75%	0.08	10.0
tebuconazool	Fungicide	75%	0.03	2.5
dimethomorf	Fungicide	50%	0.02	15.0
cypermethrin	Insecticide	50%	0.04	2.5
cyhalothrin‐lambda	-	25%	0.01	0.4
azoxystrobin	Fungicide	25%	0.01	15.0
ametoctradin	Fungicide	25%	0.01	30.0
cyflufenamid	Fungicide	25%	0.01	1.0
cyazofamid	-	25%	0.03	10.0
ethirimol	-	25%	0.01	2.0
mandipropamid	Fungicide	25%	0.03	10.0
fluopicolide	Fungicide	25%	0.01	10.0
deltamethrin	Insecticide	25%	0.03	1.0
mepanipyrim	-	25%	0.01	10.0
pyraclostrobin	Fungicide	25%	0.01	1.5
metalaxyl	Fungicide	25%	0.02	10.0
spiroxamine	-	25%	0.02	3.0
pyriofenone	-	25%	0.01	4.5

Aantal metingen: 4
Gem. aantal pesticides: 9.5
Aantal analyses: 38 ( conform: 38 )

kropsla

analyse	type	% obs	mg/kg (gem.)	MRL
fluopyram	Fungicide	100%	0.20	15.0
propamocarb	Fungicide	67%	0.98	40.0
spirotetramat (som)	Insecticide	67%	0.08	7.0
dimethomorf	Fungicide	33%	0.05	15.0
fludioxonil	Fungicide	33%	0.01	40.0
fluopicolide	Fungicide	33%	0.02	6.0

Aantal metingen: 3
Gem. aantal pesticides: 3.3
Aantal analyses: 10 ( conform: 10 )

kweeperen

analyse	type	% obs	mg/kg (gem.)	MRL
chloorantraniliprole	Insecticide	100%	0.02	0.4
acetamiprid	Insecticide	67%	0.01	0.8
methoxyfenozide	Insecticide	67%	0.01	2.0
boscalid	Fungicide	33%	0.05	1.5
cyhalothrin‐lambda	-	33%	0.03	0.2
deltamethrin	Insecticide	33%	0.01	0.1
pyraclostrobin	Fungicide	33%	0.01	0.5
pyriproxyfen	Insecticide	33%	0.01	0.01

Aantal metingen: 3
Gem. aantal pesticides: 4.0
Aantal analyses: 12 ( conform: 12 )

limoenen, lemmetjes

analyse	type	% obs	mg/kg (gem.)	MRL
imazalil	Fungicide	85%	1.07	5.0
azoxystrobin	Fungicide	81%	0.29	15.0
fludioxonil	Fungicide	62%	0.39	10.0
malathion (som)	Insecticide	46%	0.10	2.0
pyriproxyfen	Insecticide	42%	0.01	0.7
cyhalothrin‐lambda	-	23%	0.02	0.2

Aantal metingen: 26
Gem. aantal pesticides: 4.9
Aantal analyses: 128 ( conform: 125, niet conform: 2, niet conform*: 1 )

mandarijnen

analyse	type	% obs	mg/kg (gem.)	MRL
imazalil	Fungicide	89%	2.07	5.0
thiabendazool	Fungicide	87%	0.90	7.0
pyrimethanil	Fungicide	82%	1.78	8.0
2,4‐D	Herbicide	68%	0.16	1.0
azoxystrobin	Fungicide	47%	0.42	15.0
fludioxonil	Fungicide	34%	1.04	10.0
spirotetramat (som)	Insecticide	32%	0.02	0.5
pyriproxyfen	Insecticide	29%	0.01	0.7
pyraclostrobin	Fungicide	21%	0.02	2.0

Aantal metingen: 38
Gem. aantal pesticides: 5.7
Aantal analyses: 216 ( conform: 216 )

mangos

analyse	type	% obs	mg/kg (gem.)	MRL
fludioxonil	Fungicide	91%	0.28	2.0
azoxystrobin	Fungicide	68%	0.33	4.0

Aantal metingen: 47
Gem. aantal pesticides: 2.1
Aantal analyses: 99 ( conform: 99 )

meloenen

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	50%	0.42	5.0
azoxystrobin	Fungicide	36%	0.03	1.0
imazalil	Fungicide	30%	0.36	2.0
boscalid	Fungicide	23%	0.02	3.0

Aantal metingen: 44
Gem. aantal pesticides: 3.5
Aantal analyses: 152 ( conform: 151, niet conform: 1 )

nashis/japanse peren

analyse	type	% obs	mg/kg (gem.)	MRL
acetamiprid	Insecticide	83%	0.01	0.4
cyhalothrin‐lambda	-	33%	0.01	0.08

Aantal metingen: 6
Gem. aantal pesticides: 3.0
Aantal analyses: 18 ( conform: 15, niet conform: 2, niet conform*: 1 )

nectarines

analyse	type	% obs	mg/kg (gem.)	MRL
tebuconazool	Fungicide	70%	0.01	0.6
deltamethrin	Insecticide	60%	0.01	0.15
spinosad	Insecticide	40%	0.01	0.6
spirotetramat (som)	Insecticide	40%	0.01	3.0
fludioxonil	Fungicide	30%	0.16	10.0
acetamiprid	Insecticide	30%	0.01	0.2
difenoconazool	Fungicide	20%	0.01	0.5
cyprodinil	Fungicide	20%	0.04	2.0
captan (som)	Fungicide	20%	0.05	6.0
boscalid	Fungicide	20%	0.07	5.0
spinetoram	-	20%	0.02	0.3
fluopyram	Fungicide	20%	0.04	1.5

Aantal metingen: 10
Gem. aantal pesticides: 4.1
Aantal analyses: 41 ( conform: 41 )

niet‐scherpsmakende pepers/paprikas

analyse	type	% obs	mg/kg (gem.)	MRL
fluopyram	Fungicide	43%	0.03	2.0
koper	Fungicide/bactericide	37%	0.56	5.0
difenoconazool	Fungicide	32%	0.05	0.9
bromide	Fumigant/bodemsterilisator	32%	1.06	30.0
fluxapyroxad	Fungicide	26%	0.03	0.6
spirotetramat (som)	Insecticide	26%	0.05	1.0
azoxystrobin	Fungicide	20%	0.04	3.0

Aantal metingen: 65
Gem. aantal pesticides: 4.1
Aantal analyses: 269 ( conform: 264, niet conform*: 4, niet conform: 1 )

okras, okers

analyse	type	% obs	mg/kg (gem.)	MRL
imidacloprid	Insecticide	83%	0.03	0.5

Aantal metingen: 6
Gem. aantal pesticides: 2.8
Aantal analyses: 17 ( conform: 16, niet conform*: 1 )

olijven voor oliewinning

analyse	type	% obs	mg/kg (gem.)	MRL
deltamethrin	Insecticide	67%	0.02	0.6
difenoconazool	Fungicide	33%	0.01	2.0
trifloxystrobin	Fungicide	33%	0.01	0.3

Aantal metingen: 3
Gem. aantal pesticides: 1.3
Aantal analyses: 4 ( conform: 4 )

paksoi

analyse	type	% obs	mg/kg (gem.)	MRL
boscalid	Fungicide	40%	0.41	9.0
spirotetramat (som)	Insecticide	40%	0.04	7.0
pyraclostrobin	Fungicide	40%	0.08	1.5
spinosad	Insecticide	30%	0.30	2.0
fluopyram	Fungicide	20%	0.02	2.0
propyzamid	-	20%	0.01	0.01
propamocarb	Fungicide	20%	0.36	20.0
deltamethrin	Insecticide	20%	0.12	0.2
flupyradifurone	Insecticide	20%	0.01	0.01

Aantal metingen: 10
Gem. aantal pesticides: 2.9
Aantal analyses: 29 ( conform: 26, niet conform*: 3 )

pangasius

analyse	type	% obs	mg/kg (gem.)	MRL
chloraat	Herbicide/ontsmettingsmiddel	100%	1.06	0.7

Aantal metingen: 5
Gem. aantal pesticides: 1.0
Aantal analyses: 5 ( conform: 3, niet conform: 2 )

papajas

analyse	type	% obs	mg/kg (gem.)	MRL
thiabendazool	Fungicide	42%	0.26	10.0
difenoconazool	Fungicide	33%	0.01	0.2
azoxystrobin	Fungicide	25%	0.07	0.3

Aantal metingen: 12
Gem. aantal pesticides: 1.7
Aantal analyses: 20 ( conform: 20 )

paprikapoeder

analyse	type	% obs	mg/kg (gem.)	MRL
clothianidin	Insecticide	100%	0.04	0.2
lufenuron	-	100%	0.05	4.0
thiamethoxam	Insecticide	100%	0.03	3.5
acetamiprid	Insecticide	75%	0.01	1.5
chloorantraniliprole	Insecticide	75%	0.15	5.0
imidacloprid	Insecticide	75%	0.30	4.5
dimethomorf	Fungicide	50%	0.23	10.0
chloorfenapyr	Insecticide/miticide	50%	0.09	0.05
pyraclostrobin	Fungicide	50%	0.03	2.5
azoxystrobin	Fungicide	50%	0.04	15.0
spirotetramat (som)	Insecticide	50%	0.04	5.0
tebuconazool	Fungicide	50%	0.29	3.0
broompropylaat	-	25%	0.03	0.1
carbendazim	Fungicide	25%	0.06	1.0
2,4‐D	Herbicide	25%	0.10	0.25
MCPA en MCPB (som)	-	25%	0.09	0.25
difenoconazool	Fungicide	25%	0.02	4.5
deltamethrin	Insecticide	25%	0.07	1.0
cypermethrin	Insecticide	25%	1.00	5.0
cyhalothrin‐lambda	-	25%	0.09	0.5
chloorpyrifos‐ethyl	Insecticide	25%	0.02	0.05
chloorthalonil	-	25%	0.20	0.1
fluopicolide	Fungicide	25%	0.04	10.0
flubendiamide	-	25%	0.03	3.5
metalaxyl	Fungicide	25%	0.01	5.0
kresoxim‐methyl	-	25%	0.02	8.0
indoxacarb	-	25%	0.03	3.0
haloxyfop (som)	-	25%	0.03	0.1
pendimethalin	-	25%	0.05	0.25
penconazool	Fungicide	25%	0.02	2.0
methoxyfenozide	Insecticide	25%	0.05	10.0
trifloxystrobin	Fungicide	25%	0.17	4.0

Aantal metingen: 4
Gem. aantal pesticides: 13.2
Aantal analyses: 53 ( conform: 50, niet conform*: 3 )

passievruchten/maracujas

analyse	type	% obs	mg/kg (gem.)	MRL
azoxystrobin	Fungicide	100%	0.12	4.0
tebuconazool	Fungicide	67%	0.07	1.0
trifloxystrobin	Fungicide	50%	0.03	4.0
difenoconazool	Fungicide	25%	0.01	0.1

Aantal metingen: 12
Gem. aantal pesticides: 3.1
Aantal analyses: 37 ( conform: 36, niet conform*: 1 )

peren

analyse	type	% obs	mg/kg (gem.)	MRL
captan (som)	Fungicide	95%	0.56	10.0
fludioxonil	Fungicide	92%	0.26	5.0
cyprodinil	Fungicide	74%	0.14	2.0
pyrimethanil	Fungicide	51%	1.12	15.0
boscalid	Fungicide	44%	0.09	1.5
pyraclostrobin	Fungicide	41%	0.05	0.5
fluopyram	Fungicide	23%	0.02	0.8
difenoconazool	Fungicide	21%	0.01	0.8

Aantal metingen: 39
Gem. aantal pesticides: 4.8
Aantal analyses: 186 ( conform: 185, niet conform*: 1 )

perziken

analyse	type	% obs	mg/kg (gem.)	MRL
fludioxonil	Fungicide	45%	1.12	10.0
deltamethrin	Insecticide	45%	0.04	0.15
boscalid	Fungicide	27%	0.11	5.0
cyhalothrin‐lambda	-	27%	0.02	0.15
tebuconazool	Fungicide	27%	0.01	0.6
pyraclostrobin	Fungicide	27%	0.03	0.3

Aantal metingen: 11
Gem. aantal pesticides: 3.6
Aantal analyses: 40 ( conform: 40 )

peterselie

analyse	type	% obs	mg/kg (gem.)	MRL
pendimethalin	-	100%	0.11	10.0
chloorpyrifos‐ethyl	Insecticide	83%	0.09	0.05
anthrachinon	-	50%	0.07	0.1
difenoconazool	Fungicide	50%	1.90	50.0
azoxystrobin	Fungicide	50%	1.15	350.0
bifenyl	-	50%	0.05	0.5
tebuconazool	Fungicide	33%	0.35	10.0
thiofanaat‐methyl	-	33%	0.14	0.5
chloorfenapyr	Insecticide/miticide	33%	0.02	0.1
dimethomorf	Fungicide	33%	0.05	50.0
ethofumesaat	-	33%	0.02	7.5
profenofos	-	33%	0.01	0.15
propamocarb	Fungicide	33%	0.06	150.0
propiconazool	-	33%	3.40	0.1
penconazool	Fungicide	33%	0.06	0.1
mepanipyrim	-	33%	0.04	0.1
oxyfluorfen	-	33%	0.02	0.1
linuron	-	33%	0.02	0.1
carbendazim	Fungicide	33%	0.22	0.5
pyraclostrobin	Fungicide	33%	0.05	10.0

Aantal metingen: 6
Gem. aantal pesticides: 11.0
Aantal analyses: 66 ( conform: 63, niet conform: 3 )

pistaches

analyse	type	% obs	mg/kg (gem.)	MRL
acetamiprid	Insecticide	83%	0.03	0.07
methoxyfenozide	Insecticide	67%	0.03	0.1
clothianidin	Insecticide	33%	0.07	0.01
spirotetramat (som)	Insecticide	33%	0.02	0.5
thiamethoxam	Insecticide	33%	0.01	0.01

Aantal metingen: 6
Gem. aantal pesticides: 3.0
Aantal analyses: 18 ( conform: 13, niet conform*: 3, niet conform: 2 )

pitayas/drakenvruchten

analyse	type	% obs	mg/kg (gem.)	MRL
difenoconazool	Fungicide	50%	0.02	0.15
carbendazim	Fungicide	25%	0.06	0.1
acetamiprid	Insecticide	25%	0.04	0.01
chloorfenapyr	Insecticide/miticide	25%	0.08	0.01
cyflumetofen	-	25%	0.01	0.01
cyhalothrin‐lambda	-	25%	0.01	0.01
cypermethrin	Insecticide	25%	0.06	0.05
metalaxyl	Fungicide	25%	0.01	0.01
permethrin	-	25%	0.05	0.05
pyraclostrobin	Fungicide	25%	0.04	0.02
spirotetramat (som)	Insecticide	25%	0.04	0.02

Aantal metingen: 4
Gem. aantal pesticides: 3.0
Aantal analyses: 12 ( conform: 5, niet conform*: 4, niet conform: 3 )

plantaardige olie (ongeraffin)

analyse	type	% obs	mg/kg (gem.)	MRL
captan (som)	Fungicide	100%	0.05	0.07

Aantal metingen: 3
Gem. aantal pesticides: 1.0
Aantal analyses: 3 ( conform: 3 )

platte perziken/paraguayos

analyse	type	% obs	mg/kg (gem.)	MRL
deltamethrin	Insecticide	75%	0.02	0.15
trifloxystrobin	Fungicide	38%	0.01	3.0
tebuconazool	Fungicide	38%	0.04	0.6
acetamiprid	Insecticide	25%	0.03	0.2
fludioxonil	Fungicide	25%	1.28	10.0
fluopyram	Fungicide	25%	0.01	1.5

Aantal metingen: 8
Gem. aantal pesticides: 3.1
Aantal analyses: 25 ( conform: 25 )

pompoenen

analyse	type	% obs	mg/kg (gem.)	MRL
azoxystrobin	Fungicide	33%	0.02	1.0
boscalid	Fungicide	33%	0.01	3.0
cyhalothrin‐lambda	-	33%	0.01	0.06
fluopyram	Fungicide	33%	0.01	0.4

Aantal metingen: 3
Gem. aantal pesticides: 1.3
Aantal analyses: 4 ( conform: 4 )

preien

analyse	type	% obs	mg/kg (gem.)	MRL
fluopyram	Fungicide	78%	0.01	0.8
propamocarb	Fungicide	44%	0.16	20.0
fluopicolide	Fungicide	44%	0.02	1.5
ametoctradin	Fungicide	33%	0.04	5.0
spinosad	Insecticide	33%	0.03	0.2
tebuconazool	Fungicide	22%	0.01	0.6
difenoconazool	Fungicide	22%	0.01	0.6
trifloxystrobin	Fungicide	22%	0.04	0.7

Aantal metingen: 9
Gem. aantal pesticides: 3.8
Aantal analyses: 34 ( conform: 34 )

pruimen

analyse	type	% obs	mg/kg (gem.)	MRL
tebuconazool	Fungicide	60%	0.03	1.0
boscalid	Fungicide	40%	0.01	3.0
fludioxonil	Fungicide	30%	0.42	5.0
azoxystrobin	Fungicide	20%	0.01	7.0
cyprodinil	Fungicide	20%	0.01	2.0
chloorantraniliprole	Insecticide	20%	0.01	1.0
spirodiclofen	-	20%	0.01	7.0

Aantal metingen: 10
Gem. aantal pesticides: 2.4
Aantal analyses: 24 ( conform: 24 )

raketsla/rucola

analyse	type	% obs	mg/kg (gem.)	MRL
acetamiprid	Insecticide	67%	0.05	3.0
chloorantraniliprole	Insecticide	33%	0.49	20.0
deltamethrin	Insecticide	33%	0.10	2.0
fludioxonil	Fungicide	33%	0.01	20.0
mandipropamid	Fungicide	33%	0.34	25.0
metalaxyl	Fungicide	33%	0.40	3.0
spirotetramat (som)	Insecticide	33%	0.03	7.0

Aantal metingen: 3
Gem. aantal pesticides: 2.7
Aantal analyses: 8 ( conform: 8 )

rijst

analyse	type	% obs	mg/kg (gem.)	MRL
tebuconazool	Fungicide	52%	0.02	1.5
isoprothiolane	-	39%	0.39	6.0
tricyclazool	-	39%	0.03	0.01
azoxystrobin	Fungicide	35%	0.01	5.0
thiamethoxam	Insecticide	26%	0.02	0.01
dinotefuran	-	26%	0.04	8.0
difenoconazool	Fungicide	22%	0.01	3.0

Aantal metingen: 23
Gem. aantal pesticides: 3.3
Aantal analyses: 75 ( conform: 64, niet conform*: 6, niet conform: 5 )

rozijn

analyse	type	% obs	mg/kg (gem.)	MRL
boscalid	Fungicide	79%	0.13	25.0
cyhalothrin‐lambda	-	75%	0.03	0.4
azoxystrobin	Fungicide	71%	0.05	15.0
acetamiprid	Insecticide	64%	0.23	2.5
pyrimethanil	Fungicide	64%	0.40	25.0
cyprodinil	Fungicide	61%	0.42	15.0
methoxyfenozide	Insecticide	61%	0.11	5.0
dimethomorf	Fungicide	61%	0.03	15.0
fluopyram	Fungicide	57%	0.09	10.0
fluxapyroxad	Fungicide	57%	0.04	15.0
fenvaleraat (som)	-	57%	0.03	1.5
spirotetramat (som)	Insecticide	50%	0.04	10.0
metalaxyl	Fungicide	50%	0.26	10.0
penconazool	Fungicide	46%	0.02	2.5
ametoctradin	Fungicide	46%	0.03	30.0
proquinazid	-	43%	0.02	2.5
tebuconazool	Fungicide	43%	0.02	2.5
difenoconazool	Fungicide	39%	0.02	15.0
metrafenon	Fungicide	39%	0.01	35.0
cyflufenamid	Fungicide	36%	0.01	1.0
fluopicolide	Fungicide	36%	0.03	10.0
tetraconazool	-	29%	0.01	0.35
cypermethrin	Insecticide	29%	0.05	2.5
trifloxystrobin	Fungicide	29%	0.01	15.0
zoxamide	-	25%	0.02	25.0
sulfoxaflor	Insecticide	25%	0.02	10.0
indoxacarb	-	25%	0.02	10.0
fludioxonil	Fungicide	25%	0.02	25.0
mandipropamid	Fungicide	21%	0.01	10.0
iprovalicarb	-	21%	0.02	10.0

Aantal metingen: 28
Gem. aantal pesticides: 16.0
Aantal analyses: 447 ( conform: 444, niet conform*: 2, niet conform: 1 )

shaddocks/pomelos

analyse	type	% obs	mg/kg (gem.)	MRL
imazalil	Fungicide	92%	0.29	4.0
acetamiprid	Insecticide	69%	0.02	0.9
difenoconazool	Fungicide	62%	0.02	0.6
pyraclostrobin	Fungicide	62%	0.02	2.0
tebuconazool	Fungicide	46%	0.02	5.0
prochloraz (som)	-	38%	0.01	0.03
thiabendazool	Fungicide	38%	0.04	7.0
paclobutrazool	-	31%	0.01	0.01
pyriproxyfen	Insecticide	23%	0.01	0.6
clothianidin	Insecticide	23%	0.01	0.06
bifenthrin	Insecticide	23%	0.01	0.05

Aantal metingen: 13
Gem. aantal pesticides: 6.8
Aantal analyses: 88 ( conform: 80, niet conform*: 5, niet conform: 3 )

sinaasappelen

analyse	type	% obs	mg/kg (gem.)	MRL
thiabendazool	Fungicide	84%	0.84	7.0
imazalil	Fungicide	79%	1.86	4.0
pyrimethanil	Fungicide	67%	1.28	8.0
azoxystrobin	Fungicide	51%	1.00	15.0
fenylfenol‐2	Fungicide/bactericide	44%	1.23	10.0
pyriproxyfen	Insecticide	42%	0.02	0.6
2,4‐D	Herbicide	42%	0.10	1.0
fludioxonil	Fungicide	37%	1.63	10.0
pyraclostrobin	Fungicide	26%	0.02	2.0

Aantal metingen: 43
Gem. aantal pesticides: 5.8
Aantal analyses: 251 ( conform: 250, niet conform*: 1 )

sojabonen

analyse	type	% obs	mg/kg (gem.)	MRL
anthrachinon	-	60%	0.06	0.02
diquat	-	60%	0.02	0.3
glyfosaat	-	60%	2.10	20.0
cyproconazool	-	40%	0.03	0.07
fluazifop (som)	-	20%	0.03	15.0
haloxyfop (som)	-	20%	0.02	0.5
prothioconazool‐desthio	-	20%	0.03	1.0
tebuconazool	Fungicide	20%	0.01	0.75

Aantal metingen: 5
Gem. aantal pesticides: 3.0
Aantal analyses: 15 ( conform: 13, niet conform*: 2 )

sperziebonen/snijbonen

analyse	type	% obs	mg/kg (gem.)	MRL
chloorantraniliprole	Insecticide	46%	0.02	0.8
boscalid	Fungicide	38%	0.17	5.0
azoxystrobin	Fungicide	38%	0.03	3.0
fluopyram	Fungicide	38%	0.02	3.0
trifloxystrobin	Fungicide	27%	0.02	1.0

Aantal metingen: 26
Gem. aantal pesticides: 3.0
Aantal analyses: 77 ( conform: 74, niet conform*: 2, niet conform: 1 )

spinazie

analyse	type	% obs	mg/kg (gem.)	MRL
cyhalothrin‐lambda	-	71%	0.06	0.6
propamocarb	Fungicide	57%	4.26	40.0
fluopicolide	Fungicide	57%	0.24	6.0
flupyradifurone	Insecticide	43%	0.01	6.0
boscalid	Fungicide	29%	0.09	50.0
spirotetramat (som)	Insecticide	29%	0.02	7.0
mandipropamid	Fungicide	29%	0.23	25.0

Aantal metingen: 7
Gem. aantal pesticides: 4.0
Aantal analyses: 28 ( conform: 28 )

spruitjes

analyse	type	% obs	mg/kg (gem.)	MRL
spirotetramat (som)	Insecticide	86%	0.03	0.3
difenoconazool	Fungicide	71%	0.01	0.4
azoxystrobin	Fungicide	43%	0.01	5.0
prothioconazool‐desthio	-	29%	0.01	0.1
boscalid	Fungicide	29%	0.01	5.0

Aantal metingen: 7
Gem. aantal pesticides: 3.3
Aantal analyses: 23 ( conform: 23 )

suikerbiet

analyse	type	% obs	mg/kg (gem.)	MRL
fenpropidin	-	67%	0.01	0.07
fluopyram	Fungicide	33%	0.01	0.1

Aantal metingen: 3
Gem. aantal pesticides: 1.0
Aantal analyses: 3 ( conform: 3 )

tafeldruiven

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	56%	0.78	50.0
fluopyram	Fungicide	53%	0.20	2.0
boscalid	Fungicide	49%	0.18	5.0

Aantal metingen: 43
Gem. aantal pesticides: 4.8
Aantal analyses: 208 ( conform: 203, niet conform*: 4, niet conform: 1 )

tarwe

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	100%	2.41	10.0
chloormequat	Plantengroeiregulator	92%	0.20	7.0
deltamethrin	Insecticide	29%	0.05	1.0

Aantal metingen: 24
Gem. aantal pesticides: 2.9
Aantal analyses: 69 ( conform: 69 )

thee

analyse	type	% obs	mg/kg (gem.)	MRL
bifenthrin	Insecticide	25%	0.04	30.0
chloorfenapyr	Insecticide/miticide	25%	0.12	50.0
ddt (som)	-	25%	0.04	0.2
imazalil	Fungicide	25%	0.03	0.05
nicotine	-	25%	0.12	0.5
pyraclostrobin	Fungicide	25%	0.03	0.1

Aantal metingen: 4
Gem. aantal pesticides: 1.5
Aantal analyses: 6 ( conform: 6 )

tomaten

analyse	type	% obs	mg/kg (gem.)	MRL
difenoconazool	Fungicide	30%	0.04	2.0
fludioxonil	Fungicide	28%	0.03	3.0
boscalid	Fungicide	26%	0.06	3.0
fluopyram	Fungicide	25%	0.03	0.5
cyprodinil	Fungicide	21%	0.07	1.5

Aantal metingen: 57
Gem. aantal pesticides: 3.7
Aantal analyses: 211 ( conform: 209, niet conform: 1, niet conform*: 1 )

uien

analyse	type	% obs	mg/kg (gem.)	MRL
spirotetramat (som)	Insecticide	67%	0.03	0.4
azoxystrobin	Fungicide	40%	0.02	10.0
fluoxastrobin	-	33%	0.01	0.04
fluopicolide	Fungicide	20%	0.01	1.0
fluopyram	Fungicide	20%	0.01	0.07

Aantal metingen: 15
Gem. aantal pesticides: 2.6
Aantal analyses: 39 ( conform: 38, niet conform: 1 )

verse worst/saucijs/braadworst

analyse	type	% obs	mg/kg (gem.)	MRL
chloraat	Herbicide/ontsmettingsmiddel	100%	0.02	0.05

Aantal metingen: 3
Gem. aantal pesticides: 1.0
Aantal analyses: 3 ( conform: 3 )

watermeloenen

analyse	type	% obs	mg/kg (gem.)	MRL
koper	Fungicide/bactericide	50%	0.28	5.0
azoxystrobin	Fungicide	28%	0.02	1.0
fludioxonil	Fungicide	22%	0.01	0.3
flupyradifurone	Insecticide	22%	0.01	0.15

Aantal metingen: 18
Gem. aantal pesticides: 2.1
Aantal analyses: 37 ( conform: 35, niet conform*: 2 )

witlof/brussels lof

analyse	type	% obs	mg/kg (gem.)	MRL
fluopyram	Fungicide	90%	0.01	0.3

Aantal metingen: 10
Gem. aantal pesticides: 1.1
Aantal analyses: 11 ( conform: 11 )

wortels

analyse	type	% obs	mg/kg (gem.)	MRL
azoxystrobin	Fungicide	50%	0.01	1.0
boscalid	Fungicide	50%	0.02	2.0
difenoconazool	Fungicide	50%	0.01	0.4
prosulfocarb	-	50%	0.01	1.0
fluopyram	Fungicide	50%	0.02	0.4
cypermethrin	Insecticide	25%	0.01	0.05
fluopicolide	Fungicide	25%	0.01	0.2
fluxapyroxad	Fungicide	25%	0.09	0.9
pyraclostrobin	Fungicide	25%	0.01	0.5
quintozeen (som)	-	25%	0.01	0.02
tebuconazool	Fungicide	25%	0.01	0.4

Aantal metingen: 4
Gem. aantal pesticides: 4.0
Aantal analyses: 16 ( conform: 16 )

zonnebloempit

analyse	type	% obs	mg/kg (gem.)	MRL
pirimifos‐methyl	-	60%	0.13	0.5
chloorpyrifos‐ethyl	Insecticide	20%	0.01	0.01
deltamethrin	Insecticide	20%	0.15	0.05
mepiquat	-	20%	0.03	40.0

Aantal metingen: 5
Gem. aantal pesticides: 1.2
Aantal analyses: 6 ( conform: 4, niet conform: 1, niet conform*: 1 )

Advanced climate model evaluation with ESMValTool v2.11.0 using parallel, out-of-core, and distributed computing

2025-07-02T00:00:00+00:00

Custom toggle to hide the Reddit sidebar

2025-05-25T00:00:00+00:00

Because the sidebar is fixed size, it sometimes overlaps with the content on smaller screens. Using RES you can create a toggle for the sidebar.

First, create a toggle under: RES settings console > Core > Custom toggles > Toggle named: Sidebar toggle.

Add the following as a CSS snippet: RES settings console > Appearance > Stylesheet Loader > Snippets:

.side { display: none; }

Assign your toggle to the snippet.

A new toggle will appear under the RES menu (gear icon) in the top bar.

Convective shutdown in the atmospheres of lava worlds

2024-12-24T00:00:00+00:00

Duqtools: Dynamic uncertainty quantification for Tokamak reactor simulations modelling

2024-09-20T00:00:00+00:00

Optimizing ionic transport in argyrodites: a unified view on the role of sulfur/halide distribution and local environments

2024-09-09T00:00:00+00:00

Build log for ambilight on a rooted LG TV

2024-09-05T00:00:00+00:00

We recently bought a second-hand LG OLED TV to replace our old Phillips Ambilight TV. It's awesome, because OLED, but I missed the Ambilight feature of my old TV.

Although you can get these devices ready made, I thought it would be more fun to build it myself. These setups usually consist of the following:

Frame capture device (HDMI Splitter + USB frame grabber)
Image processing service + host (Raspberry Pi + HyperHDR)
LED strip + Controller (ESP32 + WLED)

The frame grabber is an hdmi video capture device video capture card that streams images to a Raspberry Pi. The Raspberry Pi runs HyperHDR, an image processing service that analyzes the video stream, and turns it into a signal for the LED strip. Note that HyperHDR does not control the LED strip itself, this is done using WLED.

Since many LG TVs running WebOS can be rooted, it can host PicCap (frame capture service) and HyperHDR directly. This avoids setting up a Rasperry Pi with video grabber, making the setup a whole lot cheaper and more efficient.

The Frame

I build a custum frame from solid wood, and bits and pieces I found at the DIY store and at home. The idea is that the frame acts as a support for the LED strip. It sits about 10 cm behind my TV, and I used diagonally cut wooden slats with a 45 degree angle, so that the LEDs project outward onto the wall. The frame is roughly 5 cm smaller on all sides that my TV.

For the LED strip, I used 2x WS2813 LED strips (1m / 30 LEDs). I cut the strip to size and soldered the corners.

Assembled frame

Corner detail, I used solid core wire to solder around the corner

The Controller

The second step was to set up the LED strip using WLED. I had an old ESP32 lying around, which I used to install WLED. I installed WLED through the very convenient web interface.

Parts list:

ESP32 (ESP-WROOM-32)
100 Ohm resistance
DC Jack Female 5.5mm to Terminal Block
5V/2A power adapter with 5.5mm DC Jack
Cable box from DIY store

Install WLED

List USB devices:

$ lsusb
...
Bus 001 Device 036: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
...

I used /dev/serial/ to find the right COM port:

$ ls -ltr /dev/serial/by-id/
total 0
lrwxrwxrwx. 1 root root 13 Sep  3 08:42 usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0 -> ../../ttyUSB0

To verify the installation was succesful, I could log in to the web interface and added it to Home Assistant for good measure 😅

`wled failed to open serial port`

I was struck with error: wled failed to open serial port. This means my browser does not have the correct rights to write to the USB device. This can be solved by setting read/write permissions. Do not forget to reset the permissions afterwards.

sudo chmod a+rw /dev/serial/by-id/usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0
Flash
sudo chmod 600 /dev/serial/by-id/usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0

Soldering

I build everything on a bread board first following the scheme in the quick-start guide. Later on I soldered everything onto a prototype board and put it in small cable box.

Some notes:

Make sure to connect data to GPIO16, which is labeled RX2 on my ESP: https://www.mathaelectronics.com/wp-content/uploads/2022/08/image-19.png
I used a 100 Ohm resistor, but apparently this is not necessary. When assembling the prototype board I had issues getting everything to work. It worked fine on the bread board, but no signal on the soldered prototype board. After a while I figured out that my resistor was faulty. To make things worse, I learned that is not necessary at all!! it turns out that the resistor only serves to clean up the data line. So I left it out in the final assembly.
Solder bridges were very difficult to make for me. I ended using jump wires for everything, which was much easier.

Bread board setup

Prototype board top

| |

Prototype board bottom

The final product in a box

The TV

I wanted to run everything directly from my TV, which means that I needed root on my LG C3.

I used this table to find a method root. I ended up using [dejavuln-autoroot](https://github.com/throwaway96/dejavuln-autoroot, which was very simple. Once succesful, this installs the homebrew channel.

Blocking updates

I added these addresses to my Adguard blocklist to block the TV from updating, which could break the root:

lgtvonline.lge.com
snu.lge.com
su.lge.com
su-ssl.lge.com
snu-dev.lge.com
su-dev.lge.com
nsu.lge.com

Software setup

Once rooted, I installed PicCap and HyperHDR. These are meant to be used in combination.

PicCap takes screenshots of the current image on the TV, and HyperHDR is an image processing that grabs the screenshots from Piccap and LED output, essentially rectreating ambilight frem Phillips TVs. s needed for WLED.

Note that HyperHDR does not have an interface for WebOS, you have to connect and configure it via the web interface. It took me a while to find out the default port, so here it is: http://<IP_OF_LG_TV>:8090.

Setting everything up in HyperHDR was a bit of fiddling, but it speaks for itself. To get the colors right, I had to tweak the Image processing settings to get the colors to pop on our green wall:

HyperHDR settings

The Satisfaction

Really happy with the end result. It's more reactive than what I'm used to from Ambilight, and the colours feel more vibrant. Bonus: I can tune the brightness via Home Assistant, and use it as a lamp when the TV is off.

It's working!

Notes on code quality

2024-08-20T00:00:00+00:00

As a software engineer we often talk about code quality as if it is something all encompassing that we can easily improve. For me, however, these discussions are often fraught with ambiguity and disagreements on what is actually high-quality code.

First, it's often not clear what we mean when we talk about code quality and we may not even agree on the same thing.

Second, as engineers we are easy to judge whether code is good or bad quality. And we all like to brag about the quality of our code. The fact is however that code quality is often difficult to measure objectively.

Not surprising then that many companies try to jump into this gap and sell you tools to measure and improve code quality. In my experience they mostly work at the superficial level, whereas true issues with code quality are often at a much deeper level.

Although linting tools, code formatters, type checks, and unit tests help, there implementation in the code is also subject to quality.

But what is good quality code and how can you measure it?

External

These measures are important for the users of the software. Note that these say nothing about how the programming language, data structure, which compiler etc.

Correctness
Usability
Efficiency
Reliability
Integrety
Adaptability
Accuracy
Robustness

Internal

These are measures that are important for the developers of the code. As programmers we often tend to focus on those, because they make the code more fun to work with.

Maintainability
Flexibility
Portability
Reusability
Readability
Testability
Understandability

You may notice that some of these measures are often at odds. For example, focusing on correctness and reliability may affect performance and efficiency.

The point is that what to focus on is a choice. Some quality measures are aimed at making life easier for the programmer, while some are aimed at making the code easier to use.

Having worked on many different code bases in recent years with varying degrees of 'code quality', I think for me it boils down to three things that define high quality software for me:

Easy to change - Code that is easy to change is often well structured, documented, and well encapsulated. This means we can focus on adding new features, and avoid getting stuck in difficult refactoring tasks.
Follows best practices - I'm a big fan of code that follow best practices or community standards. These code bases feel familiar, facilitating collaboration, and gives newcomers a head start when we need to contribute.
Does not surprise me - Code should work as expected. If I have to go down a rabbit hole to chase where that one variable is defined 12 levels deep, it's probably not a good sign (although we have all been there 😅).

Regression testing Plotly plots with Pytest

2024-05-29T00:00:00+00:00

I have long been using matplotlib to generate plots for my data. Matplotlib always serves me well for producing publication-style, static plots. For regression testing the plots produced by matplotlib, they provide this very useful image comparison decorator (see link for how it works). In short:

from matplotlib.testing.decorators import image_comparison

@image_comparison(baseline_images=['your_plot'])
def test_matplotlib_plot():
    your_plotting_code_here()

The decorator automatically picks up new figure objects, and compares these with the expected plots through direct image comparison. If the test fails, the difference between the plots is displayed, making it convenient to visualize the difference to help with debugging.

Regression testing in Plotly

For interactive plots that are better suited for online dashboards, we recently started using plotly a lot more. There I wanted something similar, but to my surprise, there is no documentation or infrastructure available for regression testing plots. This meant all of our plotly plots were untested. This gave us little confidence in the robustness of the plotting code, and meant that any bugs could go unnoticed for a while. To solve this, I ended up developing a function that mimicks the matplotlib way. It uses the same directories and image comparison algorithm.

actual

expected

difference

The visual comparison helps to identify where the problem lies.

Code

Below is the code that we use in one of my projects. See it in action here.

# helpers.py
from pathlib import Path
from matplotlib.testing.compare import compare_images
import inspect
from typing import Any


def assert_figures_similar(fig, *, name: str, ext: str = 'png', rms: float = 0.0):
    """Compare plotly figures and raise if different."""
    # Ensure same font is used on different machines (local/CI)
    fig.update_layout(
        font_family='Arial',
        title_font_family='Arial',
    )

    # Get path of caller
    frame = inspect.stack()[1]
    module = inspect.getmodule(frame[0])
    modulepath = Path(module.__file__)  # type: ignore

    results_dir = Path() / 'result_images' / modulepath.stem
    results_dir.mkdir(exist_ok=True, parents=True)

    filename = f'{name}.{ext}'

    actual = results_dir / filename
    fig.write_image(actual)

    expected_dir = modulepath.parent / 'baseline_images' / modulepath.stem
    expected = expected_dir / filename
    expected_link = results_dir / f'{name}-expected.{ext}'

    if expected_link.exists():
        expected_link.unlink()

    expected_link.symlink_to(expected)

    err: dict[str, Any] = compare_images(
        expected=str(expected_link), actual=str(actual), tol=rms, in_decorator=True
    )  # type: ignore

    if err:
        for key in ('actual', 'expected', 'diff'):
            err[key] = Path(err[key]).relative_to('.')
        raise AssertionError(
            (
                'images not close (RMS {rms:.3f}):'
                '\n\t{actual}\n\t{expected}\n\t{diff}'.format(**err)
            )
        )

There is no decorator, but you can use it like this:

from helpers import assert_figures_similar

def test_plot():
   fig = your_plotting_code_here()

   assert_figures_similar(fig, name='name_of_plot', rms=0.5)

My takeaways

You can save plotly images using fig.write_image('name_of_file.png') if you have the kaleido library installed.
The code uses the inspect module to get the filename of the caller.
Make sure to force plotly to use the same fonts. Arial is a safe choice, because of its ubiquity. It is available on most operating systems, and also on the Github CI. Initially all my tests failed because the font was completely different between my development environment and the CI.
There is a small deviation in rendering between my environment and the CI. These are invisible to the eye in the difference image, but enough to give up to 0.5 rms difference.
I rely on the compare_images function to do all the heavy lifting. My code only puts the images in the right place, compare_images does a file based compare and produces the difference.
The code for the matplotlib decorator is very clever. Essentially the contextmanager picks up any newly generated figures generated while it was active. There is a lot of code to make the decorator work seamlessly.

Non-utopian optical properties computed of a tomographically reconstructed real photonic band gap crystal

2024-02-14T00:00:00+00:00

Let there be word clouds: most common words per Bible book

2023-08-20T00:00:00+00:00

One of my goals for this year is to read the bible cover-to-cover. At the same time I'm learning about natural language processing for a project at work. So I got curious about the word usage per book. And particularly how the word usage evolves from book to book. In this blog post I share the word clouds and how I generated them.

All the code to generate the word clouds can be found in this repo. It consists of 2 parts:

Getting the word counts
Generating the word clouds

I'm reading the bible in Dutch (Bijbel in Gewone Taal), so the word clouds are also in Dutch ;-)

Get word counts

To get the word counts, I use the spacy library with the dutch pipeline. SpaCy is a natural language processing processing library that can tokenize and classify a text. At first I tried my own tokenization using regexes, but this is way easier and much better.

import spacy
nlp = spacy.load('nl_core_news_sm')

Tokenizing the text is then pretty straightforward. Just join all the lines together, pass it through the nlp engine, and get the tokens. For the word cloud, I'm only interested in the nouns, propositions, adjectives and verbs.

from collections import Counter

poss = ('NOUN', 'PROPN', 'ADJ', 'VERB')

def get_counts(book):
    text = ' '.join(df[df['book'] == book]['text'])
    doc = nlp(text)
    tokens = (token.text for token in doc if token.pos_ in poss)
    return Counter(tokens)

I use the handy collection.Counter class to count the tokens. This gives me a dictionary that looks something like this:

>>> c = get_counts('Genesis')
>>> c.most_common(10)
[('zei', 366),
 ('Jakob', 246),
 ('God', 223),
 ('Jozef', 189),
 ('Heer', 152),
 ('ging', 139),
 ('Abraham', 138),
 ('zoon', 131),
 ('land', 119),
 ('vader', 119)]

Generate word cloud

There really is a library for everything, for word clouds I used this one. It's pretty straightforward to use.

import wordcloud as wc

def get_wordcloud(counts, colormap='rainbow'):
    wordcloud = wc.WordCloud(
        width=800,
        height=500,
        colormap=colormap,
        background_color=None,
        mode='RGBA',
    ).generate_from_frequencies(counts)

    return wordcloud

What's left is to just loop over all the books and plot them:

for book in books:
    counts = get_counts(book)
    wordcloud = get_wordcloud(counts)

    plt.imshow(wordcloud)

Old Testament

Below are the word clouds for all books in the Old Testament.

Old Testament

Genesis

Exodus

Leviticus

| |

Numeri

Deuteronomium

Jozua

| |

Richteren

Ruth

1-Samuel

| |

2-Samuel

1-Koningen

2-Koningen

| |

1-Kronieken

2-Kronieken

Ezra

| |

Nehemia

Esther

Job

| |

Psalmen

Spreuken

Prediker

| |

Hooglied

Jesaja

Jeremia

| |

Klaagliederen

Ezechiël

Daniël

| |

Hosea

Joël

Amos

| |

Obadja

Jona

Micha

| |

Nahum

Habakuk

Sefanja

| |

Haggai

Zacharia

Maleachi

New Testament

Below are the word clouds for all books in the New Testament.

New Testament

Matteüs

Marcus

Lucas

| |

Johannes

Handelingen

Romeinen

| |

1-Korintiërs

2-Korintiërs

Galaten

| |

Efeziërs

Filippenzen

Kolossenzen

| |

1-Tessalonicenzen

2-Tessalonicenzen

1-Timoteüs

| |

2-Timoteüs

Titus

Filemon

| |

Hebreeën

Jakobus

1-Petrus

| |

2-Petrus

1-Johannes

2-Johannes

| |

3-Johannes

Judas

Openbaring

My 5 favourite IPython tricks

2023-08-03T00:00:00+00:00

These are my 5 favourite tricks I use daily when developing code using IPython, either in the shell or via a Jupyter notebook.

Auto reload

My all time favourite IPython magic is the auto reload function. This is an extension that automatically reloads modules before executing a cell. Whenever I'm working on a library, I add these 2 lines at the top of my notebook. This means that whenever I change the library I'm working, without having to restart the kernel. Super convenient for debugging.

%load_ext autoreload
%autoreload 2

Timing

For some quick performance testing of benchmarking, you can use the built-in time magics.

There are cell magics to time (%%time) or benchmark (%%timeit) the cell. Or you can time (%time) and (%timeit) a single line or function call.

Add virtual environment as Jupyter kernel

One of my habits is to create a custom virtual environment for every code that I'm working on. These are usually called .venv, and I want to enable these as a kernel in a Jupyter Notebook.

I do this so infrequently that I can never remember the command:

pip install ipykernel
python -m ipykernel install --user --name .venv

Breakpoint

I use the breakpoint() function added in Python 3.7 all the time when developing code. Before that I was already calling the IPython.embed module directly, but using a breakpoint() statement is much more convenient.

Especially, because you can customize the debugger by setting the PYTHONBREAKPOINT environment variable. The snippet below will open an IPython shell instead of the default pdb by:

PYTHONBREAKPOINT=IPython.embed python my_script.py

Or in your .bashrc:

export PYTHONBREAKPOINT="IPython.embed"

Sublime keybindings in Jupyter Notebook

As a daily Sublime Text user, the ctrl-D command to select the next word is burned into my fingers. One of my annoyances with the default install of Jupyter is that ctrl-D deletes the current line. There are different ways to achieve this ways to enable the Sublime keymap in a Jupyter environment. I found this way to be the most reliable.

Copy this snippet into ~/.config/jupyter/custom/custom.js

require(["codemirror/keymap/sublime", "notebook/js/cell", "base/js/namespace"],
    function(sublime_keymap, cell, IPython) {
        cell.Cell.options_default.cm_config.keyMap = 'sublime';
        cell.Cell.options_default.cm_config.extraKeys["Ctrl-Enter"] = function(cm) {}
        var cells = IPython.notebook.get_cells();
        for(var cl=0; cl< cells.length ; cl++){
            cells[cl].code_mirror.setOption('keyMap', 'sublime');
            cells[cl].code_mirror.setOption("extraKeys", {
                "Ctrl-Enter": function(cm) {}
            });
        }
    }
);

Reduce, reuse, recycle: Save the planet one Github action at the time

2023-06-15T00:00:00+00:00

Blog post I wrote with my colleague Sander van Rijn. Originally published at blog.esciencecenter.nl on June 15, 2023.

Ask any environmentalist what you can do to help combat climate change and protect the environment, and they will tell you to "Reduce, Reuse, Recycle." These actions are focused on maximizing practical benefit while minimizing the cost to the environment through conservation of resources and preventing the emission of greenhouse gases.

As software engineers, we are big fans of integration and continuous delivery (CI/CD) workflows as part of sustainable software development. GitHub actions is completely free for public repositories, so why wouldn't we? We use it for everything, from code analysis and linting, running tests and builds in isolated environments, generating documentation on readthedocs, and publishing packages on PyPI.

Until all of our power comes from nuclear fusion, and data centers can be made of trees, running any code will have an environmental impact: CO2 and other emissions from the power used and the production of the computers it runs on. Not writing code won't solve our problems either, so we need to find places in our workflow where we can reduce our impact.

Many CI/CD systems are configured to run automatically, often out of sight, so their impact to the planet adds up quickly. We have been asking ourselves some questions, like what we can do to make our use of CI/CD more sustainable.

Do we need to build the documentation for every commit we make to the code?
Do we need to test on every version of Python on all platforms to ensure code works?
Do we need to run all our machine learning notebooks on every push?
When we fix a typo in our documentation, do we need to run a complete suite of regression tests?
When we fix a bug, can we get away just re-running the failed tests?

In this blogpost we share what we have learned and give some ideas about what you can do to reduce your impact using GitHub Actions.

Why bother?

Data centers need large amounts of energy, from running servers, computing hardware, and refrigerating equipment. And, let's not forget that the production of servers, cables and supporting equipment, and mining of the required raw materials also contribute.

According to the IEA, all data centers and data transmission networks account for 2–3% of global electricity use. This corresponds to 300 Mt CO2-eq in 2020 (0.9% energy related greenhouse gas emissions). The costs for the environment are enormous. Although improvements in efficiency and hyperscale data centers help limit the growth in energy demands somewhat, the demand for data workloads and internet traffic still grows every year.

GitHub (this blog is about GitHub Actions after all!) claims to be committed to environmental sustainability. A noble effort, but its parent company Microsoft increased its CO2 emissions by nearly 15% from 2020 to 2022. Most companies who claim to achieve 'net zero' do so not by reducing their emissions, but by compensating elsewhere. Microsoft compensated only about 10% of their emissions in 2022. This is typically done through buying certified green energy, CO2 compensation, or emission trading. This allows companies to be green on paper by trading certificates, with little direct effect on their business operations. Such climate certificates are often sold on a promise, like planting trees or saving a forest from clearing. Better than nothing, but it’s even better to not have those emissions in the first place.

Less is more. Photo by Etienne Girardet on Unsplash.

Reduce

So, what can we do?

The best thing we can do is prevent resource usage by reducing the number of times we trigger the CI/CD workflows.

Restrict branch and event type

Many of us have configured our CI to run whenever a commit is pushed to a pull request (PR). While this gives great peace of mind to see that our tests pass, we can't help but wonder if this is really necessary.

One pattern we like to use when we are still working on a PR is to keep it in Draft mode. Once we are happy with our code and want some feedback, we mark the PR as as Ready for review. We can configure the CI to only run once a PR is Ready for review. This gives the reviewer peace of mind that the PR is bug free and is safe to be merged.

This also saves a whole bunch of unnecessary CI runs, and gives you more control when actually running the CI.

To set this up, add this to your github action:

# tests.yml
name: Tests

on:
  pull_request:
    branches:
      - main
    types:
      - opened
      - reopened
      - synchronize
      - ready_for_review
  push:
    branches:
      - main

jobs:
  test:
    if: github.event.pull_request.draft == false
    ...

    steps:
      ...

See an example of how we have implemented this for one of our projects here.

Filter by changed files

Another common pattern we use is to only run a job when the relevant files have changed. Let's say we have a simple project with the following folder structure:

project-to-save-the-planet/
├── docs/
│   └── the_plan.md
│   └── ...
├── src/
│   ├── save_the_planet.py
│   ├── supporting_documentation.md
│   └── ...
├── tests/
│   └── ...
├── ...
└── README.md

We don't have to re-run all tests if something only changed in the documentation, nor do we have to rebuild the documentation if we only make changes to some tests. And if we only fix a typo in README.md neither will be necessary. These restrictions can be enabled by adding filters in the paths section of the workflow triggers. These paths specifications can also be combined with the other filters such as branches shown earlier, to trigger only if it matches the paths and branch.

Here's an example configuration that triggers only on changes in the docs folders:

# docs.yml
on:
  push:
    ...
    paths:
      - 'docs/**'

jobs:
  ...

Certain paths or types of files can also be excluded using an exclamation mark (!). This can be useful if we want to trigger the tests to run on changes in src/ or tests/, except when we make changes to some supporting documentation or other data that does not affect the outcome of the tests. Here's an example of such a workflow:

# tests.yml
on:
  push:
    ...
    paths:
      - 'src/**'
      - '!src/supporting_documentation.md'
      - 'tests/**'

jobs:
  ...

Use CPU versions of tensorflow and pytorch

If you are working in machine learning, you will know that tensorflow and pytorch are enormous libraries. Together with their dependencies, they can be several gigabytes in size. On a CI, downloading these packages can already take up to a minute.

But, did you know that most of this space is taken up by GPU support? Try using tensorflow-cpu or pytorch-cpu instead of full-fat libraries. From our experience, this reduces the install size of our Python environments by approximately 1.5 GB on linux systems (this has no effect on Windows and Mac runners). The trick here is to install these prior to installing your package via pip. These will then be skipped by pip if these packages are also defined in your pyproject.toml or requirements.txt. Considering that all the common CI systems typically do not have GPUs in them, this makes the installs significantly lighter and therefore faster to run too!

To do so, add this step to your GitHub action prior to installing your package:

# tests.yml
jobs:
  test:
    steps:
    ...

    - name: Install tensorflow
      if: runner.os == 'Linux'
      run: |
        python3 -m pip install tensorflow-cpu

    - name: Install pytorch
      if: runner.os == 'Linux'
      run: |
        python3 -m pip install torch --index-url https://download.pytorch.org/whl/cpu

    - name: Install dependencies
      run: |
          pip install .

Why throw away when you can reuse? Photo by JulieN2212 via Wikimedia Commons.

Reuse

If you must run the CI, the best way to reduce its impact is limit the actual work that needs to be done.

With the cache action, you can cache dependencies and build outputs to make your workflows faster and thus more efficient. Maybe you need to compile some dependency, download and pre-compute some data, or set up your python environment via pip. These typically do not change much from run to run, so try to cache these where possible.

Caching your Python environment

Just the installation of the dependencies of some Python code via pip can be quite significant. Some libraries just seem to pull in an endless stream of dependencies. So, why don't we cache our entire Python environment?

Below is a snippet that we find effective in our workflows for Python code.

As the cache key, we use a combination of the Python directory name (this includes the version) in combination with the hash of the pyproject.toml, setup.cfg, or requirements.txt file. Whenever these get updated, the cache gets invalidated and regenerated.

This means we can also safely skip the pip install step if we hit the cache. Depending on the number of dependencies, this virtually eliminates the setup time of your workflow.

# tests.yaml
jobs:
  test:
    steps:
      ...

      - uses: actions/cache@v3
        id: cache-python-env
        with:
          path: ${{ env.pythonLocation }}
          key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}

      - name: Install dependencies
        if: steps.cache-python-env.outputs.cache-hit != 'true'
        run: |
          python -m pip install -e .[develop]

Caching any directory

To generalize the action below, the example below shows how any directory can be cached. We have used this to cache dependencies that need to be compiled. This can save minutes if not more from your workflow every single time it gets triggered.

We recommend adding a version to the cache key, so that you can easily invalidate the cache and force a re-trigger through the action itself.

Any subsequent workflows, will restore the directory from GitHub's cache.

# tests.yaml
jobs:
  test:
    steps:
      ...

      - uses: actions/cache@v3
        id: cache-important-dependency
        with:
          path: important-dependency
          key: cache-key-1

      - name: Compile dependency
        if: steps.cache-important-dependency.outputs.cache-hit != 'true'
        run: |
          mkdir important-dependency
          cd important-dependency
          echo "All I want is world peace... Or a piece of the world." > t.txt

Failing code into the red bin, please. Photo by Pawel Czerwinski on Unsplash.

Recycle

Finally, can we recycle anything that is left when our CI run is done? Although not exactly trash, what's typically leftover is the test results: which tests passed and failed. If any tests failed, those definitely have to pass before we want to continue. It makes sense then to re-run those tests first, and we can stop if any of those fail again.

Pytest last failed

Pytest has a very handy --last-failed flag for exactly this purpose, but that obviously depends on a local cache to remember the outcome of the previous run. Luckily for you, we've already made a GitHub action that takes care of setting this up for you! Simply use this action instead of your _run: pytest --my --pytest --args_ step:

# tests.yaml
jobs:
  test:
    steps:
      # create python environment and install pytest
      ...

    - name: Run pytest
      uses: sjvrijn/pytest-last-failed@v2
      with:
        pytest-args: '--my --pytest --args'

Conclusion

In this blog post, we introduced some options to make your sustainable software development more sustainable to the planet as well, whether by preventing double work or avoiding unnecessary CI runs in the first place. All it takes is a few extra lines in your workflows to help save the planet and save yourself some time in the process.

Of course, there are many other strategies to reduce your impact, such as using pre-built dependencies, more efficient libraries, or preparing a docker container for complex test environments.

It turns out that once you start looking, there are many technical options to reduce your impact. The more difficult challenge may be a social one, to convince your peers that change is necessary. In the end, we hope that this blog post inspired you, and that you will consider the environmental impact of your work.

Test your Streamlit dashboard using Playwright

2023-05-17T00:00:00+00:00

Over the last year, Streamlit has quickly become one of my favourite tools to showcase research data. I have been using it to create interactive dashboards and simple user interfaces for my data analysis tools.

As a software engineer, I want to make sure that whenever I make changes to my code, the dashboard that my users interact with still functions. Whether I did a refactoring, made some changes to the Python API of some functions, or a dependency got out-of-date. There are many reasons why a Streamlit dashboard may crash or refuse to work.

Testing a Streamlit dashboard manually can be quite time-consuming after every code change. Sometimes I simply forget. So I was looking for a way to automate this process.

Playwright

For my requirements, I was looking for some tool that I can call from Python, is fairly lightweight, and easy to get started with.

I had previously heard about Selenium as the goto tool for browser automation and testing of web apps, but in my search I quickly came across Playwright. There are many reasons to prefer one over the other, but it seems that Playwright is a more modern tool with a growing community.

The thing that won me over is that it has a plugin for pytest and is easy to install. It even has a code generator!

Getting started

Installing pytest into my virtual environment was straight-forward via the plugin:

pip install pytest-playwright

This will also install the playwright library, but you should install browsers seperately. I'm not interested in testing browser support, so I stick with the chromium default:

playwright install chromium

Writing your first Streamlit test

First, start your Streamlit app. For this example, I use the Streamlit demo page. You can replace this with your own Streamlit app. I open it in headless mode to prevent Streamlit from opening a browser.

streamlit hello --server.headless true

In a different tab, start the playwright code generator:

playwright codegen http://localhost:8501

Highlight an element and playwright will tell you how to locate it from code.

The code generator can be used to generate entire user workflows and automate testing procedures. You will get a little inspector window which will write out the code for you in Python, Java, Node.js, or C#.

I have found the code generate to be very useful to get locators. These can be used as targets to click on or to verify whether elements are present.

The playwright inspector automatically generates code for you.

Add the generated code to a file, i.e. test_hello.py and run pytest:

pytest test_hello.py

A cool feature, is that you can run it in headed mode, which will open up a Chromium window so you can follow along with the steps.

pytest test_hello.py --headed

Waiting for long running processes

If you're doing any sort of computation in your dashboard or need to wait for the page to load, you may have to tell playwright to pause at certain times.

One trick I found is to look for the Running... widget at the top of the page. While the widget is there, we cannot continue with the tests. You can add this line whever you need to wait for the next element to load:

page.get_by_text('Running...').wait_for(state='detached')

A complete script for testing

Below is an example of what a complete script may look like. This code should run 'as-is' if you have installed streamlit, pytest, and pytest-playwright. Save it as as test_hello.py and run pytest test_hello.py. Checkout the full documentation for the playwright python api here.

import time
from contextlib import contextmanager
import pytest
import subprocess as sp
from playwright.sync_api import Page, expect

PORT = '8502'
BASE_URL = f'localhost:{PORT}'


@pytest.fixture(scope='module', autouse=True)
def run_streamlit():
    """Run the app."""
    p = sp.Popen([
        *('streamlit', 'hello'),
        *('--server.port', PORT),
        *('--server.headless', 'true'),
    ])
    time.sleep(10)  # give Streamlit some time to start
    yield
    p.kill()


def test_page_load(page: Page):
    """Test performance of landing page."""
    page.goto(BASE_URL)

    page.get_by_text('Running...').wait_for(state='detached')

    expect(page).to_have_title("Hello")

    selector = page.get_by_text("Welcome to Streamlit! 👋")
    expect(selector).to_be_visible()


def test_plotting_demo(page: Page):
    """Test performance of plotting demo."""
    page.goto(f'{BASE_URL}/Plotting_Demo')

    page.get_by_text('Running...').wait_for(state='detached')

    expect(page).to_have_title('Plotting Demo')

    page.get_by_text('Running...').wait_for(state='detached')

    selector = page.locator("canvas")
    expect(selector).to_be_visible()

A workflow for making a release on PyPI

2023-05-05T00:00:00+00:00

Releasing a package on PyPI is not as difficult as it used to be. The tooling has evolved significantly since I uploaded my first package many years ago. Most of the steps can now be automated using github actions. In this blog I describe my typical workflow for releasing a package on PyPI.

Authentication on PyPI

After 2023-06-01, all uploads from user accounts with 2FA enabled must use an API token in place of their password.

So, first things first, we must set up authentication:

Go to https://pypi.org/manage/account/token/
Generate a new token, scope it to Entire account (all projects)
Copy the token and create a new file: $home/.pypirc

It looks something like this:

[pypi]
  username = __token__
  password = pypi-Fjk8z...39ZDiQ

Enjoy publishing to PyPI without every typing your password again

First release

Although I have automated releases of my packages, the first release has to be manual.

Decide on a package name on PyPI that is still available
Make a manual upload to PyPI. You can do this with build and twine:

pip install twine build
python -m build
twine upload dist/*

Celebrate first release on pypi 🥳

(I recommend using testpypi if this is the first time.)

Automating the next releases

The following releases can be completely automated using github actions.

Go to your PyPI settings and register an API token scoped to your package.
Add the token to this repo's secrets named PYPI_TOKEN: https://github.com/USER/PROJECT/settings/secrets/actions.

Afterwards, this github action will take care of the rest:

name: Publish on PyPI

on:
  release:
    types: [published]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: 3.10

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install setuptools wheel twine
        python setup.py sdist bdist_wheel

    - name: Publish package
      uses: pypa/gh-action-pypi-publish@master
      with:
        user: __token__
        password: ${{ secrets.PYPI_TOKEN }}

Place this action in .github/workflows/publish.yaml of your project.

New releases

The workflow then becomes as follows:

Bump the version (major/minor/patch as needed). I use bumpversion to help with this.

bumpversion minor

Make a new release on github. The upload to PyPI is triggered when a release is published. Make sure the tag matches the version.

If you use the standard github workflow (create branch->PR->merge->delete branch), it's really easy to make changelogs this way too.

eScience Center Showcases Modelling Tools for Next Generation Fusion Reactors

2023-05-01T00:00:00+00:00

High-throughput phase elucidation of polycrystalline materials using serial rotation electron diffraction

2023-04-01T00:00:00+00:00

Did you ever wonder what domain name to use for your home network?

2023-02-12T00:00:00+00:00

Neither did I. That is, until I wanted to have easy-to-remember urls that I can access from any device I own without having to type in a long ip-address. For example, instead of writing http://192.168.0.123:1234, I want to be able to type https://jellyfin.myhome to navigate to my jellyfin instance.

I run a local DNS server adguard, which means I can use use DNS rewrites to capture outgoing DNS requests. Adguard then points whoever sends the requests to a server or service in my network. Very convenient.

Firefox struggles

Except... this didn't work. It turns out my first roadblock was Firefox. All my queries would give me the following screen:

Hmm. We’re having trouble finding that site.

It turns out that firefox has an allowlist of accepted domains. These are all the registered global TLDs. Additional entries are defined in about:config under browser.fixup.domainsuffixwhitelist.*. These are the additional special TLDs that firefox allows as defined by IANA:

.develop
.example
.intranet
.local
.localhost
.test

While I could add a TLD of my own choice (i.e. browser.fixup.domainsuffixwhitelist.myhome)¹, I would have to do this on every device I own. The added risk is that I do not own the myhome TLD, so I have no control over if someone else registers this domain. This is not really an issue for a ~~hobby home network with a couple of devices~~ homelab, but still something to keep in mind.

Instead, I settled on .local. This worked well... for a while.

Multi-cast DNS?

I shortly realized that while I could access my services via firefox (🥳) and SSH in Windows, I could no longer SSH from linux anymore. Somehow the DNS would resolve correctly:

$ nslookup jellyfin.local
Server:		192.168.0.123
Address:	192.168.0.123#53

Non-authoritative answer:
Name:	jellyfin.local
Address: 192.168.0.123

But ping would fail:

$ ping jellyfin.local
ping: jellyfin.local: Name or service not known

What I didn't know is that .local is essentially reserved for multi-cast DNS (mDNS) as defined by RFC6762. I had never heard about this, but apparently mDNS means servers can define their own names and advertise them on the network. It is zero-configuration, as opposed to a conventional (unicast) DNS server.

My linux distro of choice (Debian) tries to resolve mDNS first, where Windows does not. On Linux this behaviour is defined in /etc/nsswitch.conf. Setting hosts to files dns works around the issue.

$ cat /etc/nsswitch.conf | grep hosts
hosts:          files mdns4_minimal [NOTFOUND=return] dns

This again leads me back to the problem of having to define this behaviour on all of my 3 linux machines.

tldr: use `.home.arpa`

All of this left me wondering, what is the right local domain to use?

Apparently there is a relatively new standard for this defined by RFC8375:

This document specifies the behavior that is expected from the Domain Name System with regard to DNS queries for names ending with .home.arpa. and designates this domain as a special-use domain name. home.arpa. is designated for non-unique use in residential home networks. The Home Networking Control Protocol (HNCP) is updated to use the home.arpa. domain instead of .home.

The idea is that using a TLD you do not control means that it might eventually get registered, leaking information onto the global web. .home.arpa is a subdomain of .arpa for exactly the purpose of home networks. .arpa itself is a special TLD used used for the management of technical network infrastructure of the internet.² This guarantees that this domain is safe to use.

Alternatively, you can register your own domain (i.e. https://stefsmeets.nl in my case), which holds the same guarantees.

I ended up using *.home.arpa for my local services. Everything, including http://jellyfin.home.arpa, seems to work as advertised. Which means zero-configuration on my part for any new devices I add to my network!

I tried and it totally works!

https://en.wikipedia.org/wiki/.arpa

TIL how to update docker compose images the right way

2023-01-14T00:00:00+00:00

Every time I want to update my docker images, I'm a bit confused about the best way to do it. I run all my images using docker compose. Sometimes I want to update a single image to try a new feature, and sometimes I update all of them at once.

The reason for this post is that the last time when I tried to update all my images, it did not go as planned. I updated all images by removing the running images, and then restarting them.

When you remove an image and then restart your container, docker will always pull the latest image version. While this usually works, it gave me issues this time because I run adguard as my DNS server. Removing adguard meant that the image got taken down before the new image was available. No DNS means no internet¹. No internet means no new docker images to pull. Not ideal. 😅

I find it works better to first pull the new image(s) and then restarting the containers:

docker compose pull
docker compose down
docker compose up -d

To pull the image for a specific service:

docker compose pull adguard

or by its image name directly:

docker pull adguard/adguardhome

Optionally, remove old unused images:

docker image prune

Well, mostly. At least firefox caches DNS requests, so my frequently visited sites are still accessible somehow.

5 Python tricks I learned from Advent of Code 2022

2022-12-28T00:00:00+00:00

As someone who loves to practice code optimization techniques, December has become one of my favourite times of the year. Like last year, I participated in Advent of Code. This is an annual set of Christmas-themed programming puzzles that follow an advent calendar. This is the second time I participated, because it is a great way to practice my coding skills on problems I do not normally encounter.

One of my favourite things about it is that everyone is sharing their solutions on reddit or other forums. I love to share my own tricks, and even more to read about how others approached the same problem. I often go back to improve my own solutions to implement what I learned.

This is a short list of some of the tricks that I picked up this year.

Count

Many puzzles require you to keep of some round or iteration number. While you could manually increment n in a while-loop, you can also iterate over count(). I used this in day 23 and day 24.

>>> from itertools import count
>>> for n in count():
>>>     if n == 3:
>>>         break
>>>     print(n)
1
2
3

This is especially useful to factor out the loop for some solutions. For day 23 I used this pattern to re-use the same code for both part 1 and 2:

def part1(s: str):
    return solve(s, rounds=range(10))

def part2(s: str):
    return solve(s, rounds=count(1))

Divmod

Divmod is a python builtin that does division and modulo in a single function. It literally returns the (x//y, x%y):

>>> divmod(7,2)
(3, 1)

Cool! I used this in day 17 and 25 to save a few lines of code.

Complex numbers are magic

Complex numbers can be used to store coordinates. Who knew!? I saw many solutions using this trick in combinations with sets hack to avoid having to set up a custom Pointer(x, y) class or (x, y) tuples. Where the real and imaginary components represent the x and y coordianates.

Complex numbers have 3 distinct advantages:

Complex numbers are immutable, so they can be added to sets (tuples or dataclasses cannot).

>>> coords = {1j, 1+2j, 2+3j}
>>> 1+2j in coords
True

They are a builtin type. This makes using them in list comprehensions a breeze.

>>> pos = 1+1j
>>> for new_pos in (pos + 1j, pos - 1 + 1j, pos + 1 + 1j):
>>>     ...

Complex math is really interesting for coordinate systems. Complex numbers are almost a coordinate system in itself. For example, a clockwise rotation is simply a multiplication by -1j:

>>> a = 2 + 1j
>>> for x in range(4):
>>>      print(a := a * -1j)
(1-2j)
(-2-1j)
(-1+2j)
(2+1j)

After seeing some solutions use this I tried it myself during day 14. While it made the code shorter, I thought it was a bit challenging to reason about it for myself. I tried again in some later solutions, but I prefer to pass row and column ((r, c)) tuples.

Sets are useful as sparse matrices

As someone with a background in image processing, I immediately reach for numpy whenever something can be parsed into an array. This turned out to be the wrong choice on day 14 and 23.

In both cases, the array could be infinitely expanded which made numpy the wrong choice. In fact, in both puzzles, you could simply keep track of the 'blocked' positions as a set of coordinates. Bonus: checking whether coordinates are in a set is really fast in Python.

For example, this snippet from my solution for day 24:

>>> for rnd in count(1):
>>>     gusts = its_gusty_here(gusts)
>>>     gust_map = sum(gusts)
>>>
>>>     new_safe = {start}
>>>
>>>     for pos in safe:
>>>         new_safe |= set(is_safe(gust_map, pos))
>>>
>>>     safe = new_safe
>>>
>>>     if stop in safe:
>>>         break

for/else

for/else and while/else are odd statements in Python. If it were up to Guido, he would not have included them at all. Their use is highly discouraged, because it's somewhat confusing what to do. In my head I read it as for/nobreak. However, they can be useful on some occasions. This advent of code I found two perfect opportunities to use them.

for x in range(5):
    if some_condition(x):
        break
else:
    # loop completed
    assert x == 4

For example, I used this in my solutions for day 14:

while hole not in cave:
    for new_pos in (grain + 1j, grain - 1 + 1j, grain + 1 + 1j):
        if part1 and (new_pos.imag == abyss):
            return n_grains
        elif new_pos.imag == abyss + 1:
            pass
        elif new_pos not in cave:
            grain = new_pos
            break
    else:
        cave.add(grain)
        n_grains += 1
        grain = hole

and day 23:

for pos in field:
    if not all_empty(field, pos):
        for func in moves:
            if new_pos := func(field, pos):
                break
        else:
            new_pos = pos
    else:
        new_pos = pos

Conclusion

There you have it, some of the tricks I picked up during advent of code. See you next year! 🎄

Nanomesh: A Python workflow tool for generating meshes from image data

2022-10-09T00:00:00+00:00

Streamlit + PySide = standalone data app

2022-10-07T00:00:00+00:00

For one of my projects I wanted to be able to open a streamlit app in its own window. Streamlit is an excellent way to turn scripts into web apps that I like to use to showcase research data. By default, it starts a web server and attempts to open this url in a webbrowser.

I was curious if it would be possible to redirect this to some sort of Python webview, so that I could turn my dashboard into a standalone app.

Qt webview

It turns out that Qt (using PySide2) has this capability via the QtWebEngineWidgets. The most straightforward way to install is via conda:

conda install -c conda-forge pyside2

With pyside installed, it is straightforward enough to make a little window pop up showing any url:

from PySide2 import QtCore, QtWebEngineWidgets, QtWidgets

URL = 'https://python.org'

app = QtWidgets.QApplication()
view = QtWebEngineWidgets.QWebEngineView()
view.load(QtCore.QUrl(URL))
view.show()
app.exec_()

This is what it looks like:

Python.org

Wrapping streamlit

To start a streamlit server (and closing it!) is a bit more involved. Normally you would run streamlit run my_script.py, and then pop the url into your webbrowser. But somehow we must wrap this behaviour. This is what I ended up with:

import atexit
import subprocess as sp
import os

from PySide2 import QtCore, QtWebEngineWidgets, QtWidgets

def kill_server(p):
    if os.name == 'nt':
        # p.kill is not adequate
        sp.call(['taskkill', '/F', '/T', '/PID', str(p.pid)])
    elif os.name == 'posix':
        p.kill()
    else:
        pass

if __name__ == '__main__':
    cmd = f'streamlit hello --server.headless=True'

    p = sp.Popen(cmd.split(), stdout=sp.DEVNULL)
    atexit.register(kill_server, p)

    hostname = 'localhost'
    port = 8501

    app = QtWidgets.QApplication()
    view = QtWebEngineWidgets.QWebEngineView()

    view.load(QtCore.QUrl(f'https://{hostname}:{port}'))
    view.show()
    app.exec_()

In this example I use streamlit hello. You can exchange this with the path to your script.
For the streamlit command, we need --server.headless=True. This will prevent streamlit from trying to open the URL in your default browser.
I run the streamlit command in subprocess. This is to isolate the event loop. I tried going with threads, but using a subprocess is so much easier.
Therefore, we must kill the streamlit server when we close the webview. To do so we can use the atexit handler. On Linux we can use p.kill(), but on Windows we must send the taskkill command. From experience, p.kill() is not adequate on Windows.

This is the end result:

Streamlit webview

No browser required :-)

Notes on moving Home Assistant from Raspberry Pi to Docker

2022-09-18T00:00:00+00:00

I started out running Home Assistant about 3 years ago, playing around with a few light bulbs. Installing it on an SD card was (and still is) the recommended way to go. However, SD cards can't handle small I/O very well, and tend to crash after a while. This has happened to me twice. Usually not a big deal if you keep your back-ups, but increasingly worrysome as the number of devices and thus my dependence on Home Assistant grows. This is always in the back of my mind.

Recently I installed a home server to serve as a NAS, media server, DNS, etc using Docker containers. And I figured I would also move my Home Assistant installation to it. This blog post contains some notes on the migration.

Restoring from back-up

I did a full backup of Home Assistant on my old installation and copied it over.

Unfortunately, the Docker install of Home Assistant does not support restoring from a back-up.

So in the end, I extracted the backup. I extracted homeassistant.tar.gz. Inside is a directory data, which I renamed to config and moved to $DOCKERDIR/homeassistant/ (see below). Anything else in the archive is not used. It seems that I simply could have copied over the config directory directly from my old install.

Docker compose

I use docker compose to load my Docker containers. This is the docker compose snippet I added to my docker-compose.yml:

version: "2.4"
services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - $DOCKERDIR/homeassistant/config:/config
      - /etc/localtime:/etc/localtime:ro
    devices:
      - /dev/ttyACM0:/dev/ttyACM0
      - /dev/ttyUSB0:/dev/ttyUSB0
    restart: always
    privileged: true
    ports:
      - 8123:8123

The snippet is slightly modified from the Home Assistant documentation. I did not want to use network_mode: host, so I had to forward the 8123 port.

$DOCKERDIR is an environment variable defined in a .env file in the same directory. It points to a path where my Docker config files are stored.

Uninstall system integrations

There are some integrations that are included for interfacing with the Home Assistant Operating System. With a Docker install these are no longer needed. Starting the container would greet me with a couple of errors of the likes: The following integrations could not be set up... for hassio and raspberrypi.

The Raspberry Pi integration could be removed from the integrations page.

Then I removed the hassio integration by hand (it cannot be removed via the GUI), by opening config/.storage/core.config_entries. Look for the entry that looks below this and remove it

      {
        "entry_id": "8809d872e74ca46913179fb6448b3621",
        "version": 1,
        "domain": "hassio",
        "title": "Supervisor",
        "data": {},
        "options": {},
        "pref_disable_new_entities": false,
        "pref_disable_polling": false,
        "source": "system",
        "unique_id": "hassio",
        "disabled_by": null
      },

USB permissions in Docker rootless

I run Docker in rootless mode. This gave some issues with accessing my Zigbee stick and other usb devices in Home Assistant. As a work-around, I had to update the permissions.

Get the IDs of the USB devices using: ls -l /dev/serial/by-id

Set the permissions:

sudo chmod 666 /dev/ttyACM0
sudo chmod 666 /dev/ttyUSB0

This has to be done on every reboot. This link provides a method to set the permissions automatically.

You can add shortcuts to he Home Assistant side bar to link to other services on the server, for example Jellyfin.

To do so, add this snippet to your configuration.yaml:

panel_iframe:
  portainer:
    title = "Jellyfin"
    url: "https://192.168.1.123:8096"
    icon: mdi:jellyfish
    require_admin: false

Migrating InfluxDB

I run InfluxDB on Home Assistant for long term data storage. The database also had to be moved. Make sure that you use InfluxDB version 1.8. To migrate the database I did the following.

On Home Assistant on the Raspberry Pi, I opend an SSH shell and entered the influxdb container:

docker exec -it addon_a0d7b954_influxdb /bin/bash

Then I backed up the database:

influxd backup -portable -database homeassistant /data/homeassistant.db

From the home assistant backup, I extracted a0d7b954_influxdb.tar.gz and moved homeassistant.db to my new server.

I entered my influxdb container:

docker exec -it influxdb /bin/bash

And imported restored the database:

influxd restore -portable /data/homeassistant.db

Afterwards I had to update the host/port to point to the new database location.

influxdb:
  host: ...
  port: ...

Conclusion

I cannot say it was a painless exercise, but fairly happy with the result and learned a lot in the process. My new installation runs noticably faster, and I sleep better at night not having to worry about SD crashes 😅

The eWaterCycle platform for Open and FAIR Hydrological collaboration

2022-07-14T00:00:00+00:00

Measure your water usage in Home Assistant

2022-06-20T00:00:00+00:00

In the Netherlands we have smart utility meters for gas and electricity meters. They have a digital port that sends an update every 10 seconds. I measure our usage through the DSMR integration in Home Assistant.

Most watermeters, however, are still completely anologue. We have a Sensus 620, and I was interested in reading it out to get some insight in our water usage. This blog details the steps of how I automated it using Home Assistant.

The Sensus 620 has a spinning disk that rotates for every liter.

The sensor I'm using is an inductive proximity sensor (NPN). The idea is that every cycle of the metal plate the sensor is activated and sends out a pulse. The specific model I'm using is the LJ18A3-8-Z/BX - 5V.

I used this together with an old nodeMCU v3 (based on esp8266 board) I still had lying around. See the connection scheme below.

Connection scheme

I connected it all together using a breadboard and some jumper wires.

The brown wire is connected to VU, which passes through th epower from the usb connection. There is no explicit 5V output on this board, but since the USB input runs on 5V, this works as well.
The black wire receives the pulses and is connected to any of the digital ports (I used D2).
The blue wire is connected to the ground port G.
There is a 10k Ohm resistor between the brown and black wire that acts as a pull-up, which helps with ghost pulses.

Installing esphome

I setup the using ESPhome. ESPHome is a system to control esp8266 and esp32 boards using simple configuration files. Once you flash the boards, you can control and update them remotely through different home automation systems.

I first tried to set it up using the ESPHome integration in Home Assistant, but I ran into a bunch of weird issues. So I used the command-line version (which worked the first try!).

conda create -n esphome python=3.10
pip install esphome

For nodeMCU v3 use board: nodemcuv2 (that's not a typo, apparently for v2+ all uses the same interface).

Below is the config I used.

# watermeter.yaml
esphome:
  name: watermeter
  comment: "Measure water cycles"
  platform: ESP8266
  esp8266_restore_from_flash: true
  board: nodemcuv2

wifi:
  ssid: "mywifi"
  password: "badpassword"
  fast_connect: true

logger:

api:

time:
  - platform: sntp
    id: my_time

sensor:
  - platform: pulse_meter
    pin: D2
    name: "Water usage"
    id: water_usage_per_min
    icon: "mdi:water"
    unit_of_measurement: 'L/min'
    accuracy_decimals: 0

    total:
      id: water_usage_total
      name: "Watermeter Total"
      icon: "mdi:cube-outline"
      state_class: "total_increasing"
      device_class: water
      unit_of_measurement: "L"
      accuracy_decimals: 0

This sets up the D2 pin as a pulse counter, that will be reported reported to Home assistant as water_usage_per_min (by default it reports the number of pulses per minute). It also keeps track of the total number of pulses under water_usage_total.

Save the config and compile it using:

esphome compile watermeter.yaml

The build file is in .esphome/build/watermeter/.pioenvs/watermeter/firmware.bin. I uploaded the binary using esphome-flasher from here.

You can test the sensor by holding a piece of metal near the blue part. The red LED will light up.

I built a little encasing using lego. To hold the sensor in place, I drilled out a 17 mm hole from a piece of multiplex, and two smaller holes to fit the holding pins on the sensor.

This is what it looks like in my utility closet.

Make sure to fix the IP of the water sensor, so it can be added to Home Assistant using the ESPHome integration. Afterwards, the setup was straightforward. The sensors were reported immediately. I added a utility meter helper that cycles every day to keep track of our daily use (water_per_day). And I set up a card in our dashboard.

Graphs in Home Assistant.

This is the code I used for the graphs:

type: vertical-stack
cards:
  - type: custom:mushroom-title-card
    title = "Watermeter"
  - type: horizontal-stack
    cards:
      - type: custom:mini-graph-card
        entities:
          - entity: sensor.water_per_day
        name: Vandaag
        hours_to_show: 168
        aggregate_func: max
        group_by: date
        show:
          graph: bar
        hour24: true
      - type: custom:mini-graph-card
        entities:
          - entity: sensor.water_usage_per_min
        name: Nu
        hours_to_show: 24
        hour24: true

That's it! I already learned that we use between 70 and 100 L for our showers, and 400 L to water our garden!

Update for Home Assistant 2022.11

Home Assistant Version 2022.11 adds support for water meters in the energy dashboard.

To make this work, you will need to add device_class: water to the sensor. I have updated the example above.

If you don't want to reflash your esphome, you can also customize your sensor:

homeassistant:
  customize:
    sensor.water_usage_total:
      device_class: water

Refresh your confg, and it should show up under the energy dashboard. 💦

Updating to LineageOS 19.1 on a Samsung Galaxy S10, S10+ or S10e

2022-05-15T00:00:00+00:00

Last week I got the prompt to update my phone (S10e) to LineageOS 19.1. Usually LineageOS can be update from within the OS, but version upgrades require a bit more work. It was obvious what I had to do, but the process was not entirely clear to me. So I logged my steps here. The steps are equivalent for the different S10 versions.

I use Ubuntu when I flash my phone, because I typically run into some driver issue of some sort on Windows.

Most of the information here I gathered from these links:

Updating the firmware

For this step you will need heimdall.

Download and unzip the firmware compatible with Android 12 for your model (S10e, S10, S10+).
Put your phone in download mode:
- Turn off the device
- Hold Volume Down + the Bixby button and plugin the usb cable connecting it to your pc
- In the menu, press 'Continue'
Once in download mode, flash the firmware:

heimdall flash --CM cm.bin --DQMDBG dqmdbg.img --KEYSTORAGE keystorage.bin --RADIO modem.bin --CP_DEBUG modem_debug.bin --PARAM param.bin --BOOTLOADER sboot.bin --UH uh.bin --UP_PARAM up_param.bin

(The phone will reboot normally afterwards)

Updating to LineageOS 19.1

This step needs adb. I installed it using apt: sudo apt install adb

Download the latest build of LineageOS version 19.1 (S10e, S10, S10+)
Download gapps for Lineage 19.1 (Android 12; MindTheGapps)
Put the phone in debugging mode
- enable USB debugging entry under Debugging in the developer tools
I ran into an error running adb: no permissions (user in plugdev group; are your udev rules wrong?
- To solve, I put the phone to File transfer mode as recommended here. This prompted the Allow USB debugging? prompt to add my computer's RSA fingerprint to the phone.
Put the phone in sideload mode:
- adb reboot sideload
Flash LineageOS 19.1
- adb sideload lineage-19.1-20220512-nightly-beyond0lte-signed.zip (replace with your version)
Flash MindTheGapps

adb sideload MindTheGapps-12.1.0-arm64-20220416_174313.zip

Reboot phone
🥳

Cool! So what's new? The most obvious to me is the updated UI with enormous clock on my home screen, giant volume and brightness sliders, and the enormous buttons in the pull-down menu.

And Android Auto is working again! 🏎️

Other than that, I haven't noticed much difference.

Forget about Jupyter Notebooks — showcase your research using Dashboards

2022-05-06T00:00:00+00:00

As a Python lover, I use Jupyter Notebooks for everything. Notebooks mix markdown, code, and inline plots, which makes them a fantastic tool for exploratory data analysis. I use them to develop and share code, prototype new ideas, explore libraries, play around with data, and create plots and visualizations. Notebooks can be rendered as static html and pdfs, so they are also excellent to write reports, documentation and tutorials… when the intent is to share the code alongside the data.

As a researcher, however, I find that the code sometimes gets in the way of the data I want to show. So when the intent is to share the data with a non-technical audience, what options do we have? Are there better alternatives?

The problem with notebooks

Before we continue, let’s take a step back and look at a few of the problems with Jupyter Notebooks.

There is no denying that Jupyter notebooks have become very popular over the last years to present research results. This means that the issues with Jupyter Notebooks are well known. My biggest gripes are:

The non-linear execution model. Notebooks often contain hidden state that is difficult to reason about. This can make them difficult to use for beginners.
Notebooks are difficult to share with a non-technical audience. They would need to know how to set up Python, install libraries, manage environments, and modify the code.

Notebooks can be hosted somewhere like binder (solving 2.), making the code immediately reproducible. Reproducible, only if one understands the code and how to run the notebook (including its quirks, see 1.). Here, the inline code can get in the way of itself. I have seen many notebooks spelling out what each variable does and how to run the code at the top. You would need some understanding of how the code works to use the notebooks.

Dashboards

That is where dashboards come into play.

Dashboards are a relatively new concept coming from the data science world making use of the modern web. In essence, dashboards are simple web-apps used to quickly glance at some data. Like a mini graphical interface for your data.

If you are using Python, these are the ones you should be looking at:

Dash (2017, 883k downloads/month)
Panel (2018, 387k downloads/month)
Streamlit (2019, 930k downloads/month)
voila (2019, 56k downloads/month)

All are excellent choices. For a full comparison, check out this blog post.

Streamlit

Out of these four, Streamlit stood out to me the most for its ease of use. For one of my projects, I have been playing with it to develop a simple data processing GUI. These are my initial impressions.

After having used it for a week, I found it extremely straightforward to get started with.
The linear execution model makes it easy to reason about the code (more on this later)
There is no need to know any web development, as one of the goals of the library is to look good out-of-the box (spoiler alert: it does).
The api is well-designed, easy-to-manage, and very Pythonic. You could get to grips with the entire API in a day. Some would say the API is limited, but in my opinion it has a very clear scope which fits my brain. It also helps that the documentation is well-structured, with clear explanations and examples.
The streamlit developers claim its the fastest way to build data apps in Python. This sounds like a salespitch, but it might be true. You can turn any Python script into an interactive dashboard in minutes.

From a normal plot…

Let’s have a look at an example. As a researcher, I had plenty of Python scripts or notebooks lying around which just did this:

Load or generate some data
Apply some operations to the data
Make some plots

I would endlessly tune the the parameters and re-run the script to get the right plot. No problem for me. But, when it came to sharing the scripts with my not-so-software-savvy colleagues, this meant taking on a support role. Think about setting up Python, managing environments, fixing bugs, feature requests, etc…

Sounds familiar?

The snippet below generates some data (a normal distribution), fits it, and creates a matplotlib plot out of it. It takes three parameters, mu_in, std_in, and size.

import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt

mu_in = 5
std_in = 5.0
size = 100

def norm_dist(mu, std, size=100):
    """Generate normal distribution."""
    return norm.rvs(mu, std, size=size)

data = norm_dist(mu_in, std_in, size=size)

# Fit the normal distribution
mu, std = norm.fit(data)

# Make some plots
x = np.linspace(-40, 40, 100)
y = norm.pdf(x, mu, std)

title = "f"Fit results: {mu=:.2f},  {std=:.2f}""

fig, ax = plt.subplots()
ax.hist(data, bins=50, density=True)
ax.plot(x, y, 'k', linewidth=2)
ax.set_title(title)

plt.show()

My cool python script 😎

A normal plot

…to a fancy dashboard

Let’s turn this into an interactive dashboard in four simple steps:

import streamlist as st 😅
Add a title using [st.title](https://docs.streamlit.io/library/api-reference/text)
Turn the input parameters into interactive sliders using [st.slider](https://docs.streamlit.io/library/api-reference/widgets)
Tell streamlit about our plot using [st.pyplot](https://docs.streamlit.io/library/api-reference/charts)

Note that we do not have to change any of the data generation, fitting, or plotting code!

import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
import streamlit as st

st.title('Normal distribution')

mu_in = st.slider('Mean', value=5, min_value=-10, max_value=10)
std_in = st.slider('Standard deviation', value=5.0, min_value=0.0, max_value=10.0)
size = st.slider('Number of samples', value=100, max_value=500)

def norm_dist(mu, std, size=100):
    """Generate normal distribution."""
    return norm.rvs(mu, std, size=size)

data = norm_dist(mu_in, std_in, size=size)

# Fit the normal distribution
mu, std = norm.fit(data)

# Make some plots
...

st.pyplot(fig)

Now as a dashboard 🐱‍💻

Then run the dashboard using:

streamlit run my_dashboard.py

This will start a server, and the dashboard can be accessed through the browser (much like a Jupyter Notebook).

A fancy dashboard

How does this work?

The way Streamlit works is quite interesting. Everytime a slider is moved, a box is checked, or a button is pressed, Streamlit triggers a re-run of the script. The input values are updated. The javascript back-end keeps track of the values.

This means that the code itself executes linearly. In my view, this simplicity is what sets it apart. There is no need for any callbacks or complex flow controls. Your python scripts runs from top-to-bottom. This makes it easy to reason about the code. And with minimal modifications to the python code, any script can be turned into a dashboard.

Are there any downsides? Yes. Because streamlit re-runs the entire script on every update, it can feel a bit slow. Especially when updating a large number of plots. It can also get stuck on long-running functions. For performance optimizations, streamlit has some options to cache the result.

Plotting libraries

The example above uses matplotlib for the plots. Matplotlib has been the go-to plotting library for Python for many for a long time. It has been around for nearly two decades, and it is tighly integrated in the scientific python stack.

If you are familiar with matplotlib, you will know that it is great for making making publication quality plots. You will also know that making interactive plots can be a hassle.

Streamlit supports these libraries:

Modern plotting libraries like plotly, bokeh, and altair render directly to javascript. This means they are built for the web, and interactivity is built-in. This makes them better suited for web-apps. If you are going to make a dashboard, I recommend checking out one of these alternatives.

Alright, so now that we have made a fancy looking dashboard, so that anyone can play with the data. How do we make it available?

Streamlit uses a host/server model, which means you can run it on your own server.

Easier is to use the streamlit cloud to host your dashboard (it’s free for students and open-source projects). I found this also quite straightforward to set up. All I had to do was to create a repository on github with the code and a requirements file.

Then I logged into streamlit cloud using the Github SSO, and started a new app pointing at my repo and code.

Click here for the result! 🥳

Final remarks

In this blog post, I introduced streamlit and showed how it can be used to turn a python script into a dashboard, and host it online. An excellent way to showcase your research to a non-technical audience, if you ask me. The linear execution model makes it straightforward to adapt existing scripts. The code does not get in the way, and the result look awesome.

So next time you want to present some data in a notebook, consider using a dashboard instead.

All the code in this blog post is available from Github.

PS. Don't actually forget about notebooks, they are awesome!

Resetting an unavailable Philips HUE bulb in Home Assistant

2022-04-10T00:00:00+00:00

After the most recent Home assistant update, some of my philips Hue lights became unresponsive. OK, no big deal, I have had this before, I will just reset them. Turns out this is not as easy as just flipping the switch a couple of times to put the bulb into pairing mode.

The manual is not very helpful either, and search the web sent me to install the HUE or Alexa apps (both were useless for this).

In the end I came accross a post somewhere about using the HUE remote. Luckily I have a couple around.

To reset a Philips HUE bulb, hold the on and off buttons on the remote for 10 seconds. The light will flash three times and the bulb will reset itself.

I'm sure I went through this before... in part this blog post serves as a reminder for myself.

This was not the end of it.

I managed to get one bulb to pair, using the find devices in the ZHA menu. But another only seemed to not complete the pairing process (stuck at 'starting interview'). It did not fully initialize, but it did register a device in Home Assistant. This blocked me from doing anything else.

In the end, I found that I first had to completely remove the device. In Developer tools, I used the zha.remove service and put the IEEE address of the bulb (you can find this in the device menu).

You can do it in the GUI, but the yaml looks like this:

service: zha.remove
data:
    ieee: 00:12:34:56:78:90:5a:b6

After this, the bulb paired on the second attempt. 2 hours well spent!

An interview with myself on my favourite meshing library

2022-03-05T00:00:00+00:00

Q: Hi self, for this blog series we are looking to highlight our developer's favourite libraries.

S: As a software engineer I always get excited by tools that exactly fit the job that we need it for. For one project, I was looking for a triangle mesh generator.

Q: Wait.. wait... Meshes?

S: Meshes simplify image data and describe complex geometries. A mesh is essentially a collection of vertices and triangles. They are used for computer graphics and physical simulations such as finite element analyses and fluid dynamics.

Q: Like in computer games?

S: Exactly!

Q: I have heard CGAL is the go-to library for Meshing.

S: CGAL is an excellent choice. It's written in C++, and provides easy access to efficient and reliable geometric algorithms. For my project, I was looking for a simple library that runs on all platforms, and can be easily interfaced with Python. Unfortunately, CGAL is a bit heavy-weight, difficult to install on Windows, and the python bindings are limited.

Q: So what did you end up using?

After having tried a few more meshing libraries, I came accross Triangle. Triangle is an open-source two-dimensional quality mesh generator that supports multiple domains. It's written in C++ and supports all three major operating systems. Although it can be used on its own, there is an awesome python library that wraps it.

Q: Cool, so how do you install it?

S: It's available on pypi. If you are already using Python, just do:

pip install triangle

Q: That's great! That will make it easy to integrate it into my existing project. How do you use it?

S: Let me give you an example. Here is the code to generate a simple square and triangulate it:

import matplotlib.pyplot as plt
import numpy as np

import triangle as tr

vertices = np.array([[0, 0], [1, 0], [1, 1], [0, 1]])
inp = {'vertices': vertices}
out = tr.triangulate(inp, opts='qa0.05')
tr.compare(plt, inp, out)
plt.show()

Example 1

Q: So I see triangle fills the box with little triangles. How does that work?

S: Alright, let me break it down for you.

First we import matplotlib, numpy and triangle.

Second, we generate the vertices representing a square with edges of length l and store them in a dictionary.

Next, we call triangulate. The options opts=qa0.05 tell triangle to perform a quality mesh, and that no triangles have an area larger than 0.05. A quality mesh is a mesh with all angles over 30 degrees.

Finally, triangle has a useful function to compare the input and the output.

Try playing around with the area to see what happens to the result!

Q: Cool, so if I give it a set of points, it will generate a mesh surface. Can I specify different regions?

S: Yes, that is one of the strenghts of triangle. It makes multi-region meshing easy. This means that it can label triangles according to the region it belongs to. Let me give you an example:

import matplotlib.pyplot as plt
import numpy as np

import triangle as tr

vertices = np.array([[0, 0], [1, 0], [1, 1], [0, 1]])
segments = np.array([[0,1], [1,2], [2,3], [3,0], [1,3]])
regions = np.array([[0.25, 0.25, 0, 0], [0.75, 0.75, 1, 0]])

inp ={'vertices': vertices, 'segments': segments, 'regions': regions}
out = tr.triangulate(inp, opts='pqa0.05')

tr.compare(plt, inp, out)
plt.show()

Example 2

Q: Can you explain how it works?

The difference here is that we specify the different segments. The segments describe the boundaries of a region inside the square. Triangles will be generated for each region independently. Each region thus contains its own set of triangles, and are labeled accordingly.

The format for defining the regions can seem a bit tricky at first. The first two numbers describe the coordinate, and the third number is the label of the region. The final number can be used to set the maximum area in that region. We don't use that here, so we set it to 0.

As before, use the compare function to check the result.

Q: Wow! That makes it useful for my finite element analyses.

S: Yup, now you can get started meshing your own data using triangle.

If you want to know more, the documentation for triangle is available here.

For one of my projects, we use triangle as the triangle mesh generator in nanomesh. Nanomesh is a python workflow tool to prepare meshes for finite element analysis from 2D (and 3D!) microscopy image data.

TIL you can refactor your documentation using decorators

2022-02-28T00:00:00+00:00

I don't mind writing documentation. I try to write and update docstrings as I'm working on the code. This is not always easy, and sometimes documentation is spread over different docstrings. Or if docstrings for similar functions are not consistent. This, I do mind. Especially if they are part of the public API.

Last week I found that you can copy docstrings using a decorator. This post is inspired by some code I found in matplotlib:

def copy(source):
    """Copy a docstring from another function."""
    def do_copy(target):
        if source.__doc__:
            target.__doc__ = source.__doc__
        return target
    return do_copy

It's a decorator for a function, that copies the docstring from the source function to the decorated function.

def roar():
    """Roar like a lion."""
    print('rawr')

@copy(roar)
def loud_roar():
    print('RAWR!')

help(roar)
help(loud_roar)

Help on function roar in module __main__:

roar()
    Roar like a lion.

Help on function loud_roar in module __main__:

loud_roar()
    Roar like a lion.

Cool! Not immediately useful for my own purposes. The good thing is that it can easily be extended using string formatting.

The decorator below takes a function, and tries to copy the docstring to the target function. Any keyword parameters are passed to string.formatting. A copy of the source docstring is kept on ._docstring.

from textwrap import dedent

def docstring(source=None, **params):
    """Copy a docstring from another function."""
    def do_copy(target):
        if source is None:
            docstring = dedent(target.__doc__)
        elif hasattr(source, '_docstring'):
            docstring = source._docstring
        elif source.__doc__:
            docstring = dedent(source.__doc__)

        target.__doc__ = docstring.format(**params)
        target._docstring = docstring

        return target

    return do_copy

Let's see how it works:

@docstring(cry='Roar', animal='cow')
def roar():
    """{cry} like a {animal}."""
    print('RAWR!')

@docstring(roar, cry='Moo', animal='cow')
def moo():
    print('MOO!')

help(roar)

help(moo)

Help on function roar in module __main__:

roar()
    Roar like a cow.

Help on function moo in module __main__:

moo()
    Moo like a cow.

It also works with class methods:

from statistics import median

class MyList(list):
    @docstring(op='minimum')
    def min(self):
        """Return {op} value from the list."""
        return min(self)

    @docstring(min, op='maximum')
    def max(self):
        return max(self)

    @docstring(min, op='median')
    def mean(self):
        return median(self)


help(MyList.min)
help(MyList.max)
help(MyList.mean)

Help on function min in module __main__:

min(self)
    Return minimum value from the list.

Help on function max in module __main__:

max(self)
    Return maximum value from the list.

Help on function mean in module __main__:

mean(self)
    Return median value from the list.

Essentially anything with a docstring (__doc__ attribute) works, such as classes:

@docstring(classname='A')
class A:
    """This is {classname}."""
    pass

@docstring(A, classname="B")
class B(A):
    pass

print(A.__doc__)
print(B.__doc__)

This is A.
This is B.

Nice! you can imagine that if you have large docstrings, that helps keeping the documentation consistent and concise.

As a bonus, this will also be picked up by Sphinx. If you are generating documentation for e.g. readthedocs, this will make it just a little bit nicer to read!

Type union operator in Python 3.7+

2022-01-21T00:00:00+00:00

I'm a big fan of Python type hints. I try to use them whenever I can. Not only for static type checking with mypy. They also help document functions (sphinx has nice support for them) and force me to think about how I structure my functions.

One thing that bothers me, is that type hints can get quite verbose. Especially if you have multiple possible input types or optional values. For example, take this function that takes an int or a float as input:

# type_union.py
from typing import Union

def square(a : Union[int, float]):
    return a**2

Python 3.10 introduces a new type union syntax to make this a bit nicer to read. The piping operator (|) can be used to join multiple types:

# type_union_310.py

def square(a : int | float):
    return a**2

If you try to run this code in older versions of Python, it will raise this error:

>>> type_union_future.py:5: error: X | Y syntax for unions requires Python 3.10

It is a little known fact that the new type union syntax is also available in older versions of Python (3.7+) using a futures import! __future__ is a Python module part of the standard library that makes newer features available to older versions of Python.

If you add from __future__ import annotations to the top of your code:

# type_union_future.py
from __future__ import annotations

def square(a : int | float):
    return a**2

You can also use the new type union syntax in Python 3.7+! 🥳

Extra

I was already aware that PEP-563 will introduce postponed evaluation of annotations. This will help deal with forward references. For example:

# forward_reference.py
from __future__ import annotations

class MyClass:
    def __constructor__(self) -> MyClass:
        pass

Without the futures import, this will be valid in mypy, but throws NameError: name 'MyClass' is not defined when trying to execute it. This has been available via the same from __future__ import annotations since Python 3.7.

HELLO WORLD

2022-01-10T00:00:00+00:00

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

In the age of electrons, do we still need powder diffraction?

2021-08-18T00:00:00+00:00

Serial rotation electron diffraction: Automated high-throughput data collection and data processing

2021-08-13T00:00:00+00:00

era5cli: the commandline tool to download ERA5 data

2021-04-28T00:00:00+00:00

Automated electron diffraction using Instamatic

2020-12-17T00:00:00+00:00

InsteaDMatic: towards cross-platform automated continuous rotation electron diffraction

2020-10-01T00:00:00+00:00

Towards automated serial electron diffraction for macromolecular crystallography

2019-12-09T00:00:00+00:00

Towards automated serial electron diffraction for macromolecular crystallography

2019-10-07T00:00:00+00:00

SSZ-27: A Small-Pore Zeolite with Large Heart-Shaped Cavities Determined by Using Multi-crystal Electron Diffraction

2019-09-09T00:00:00+00:00

The Role of Cluster Analysis in Automated Serial Electron Crystallography

2019-08-18T00:00:00+00:00

Glue it all together with Python: Automating electron diffraction data collection

2019-08-14T00:00:00+00:00

The Role of Cluster Analysis in Automated Serial Electron Crystallography

2019-08-05T00:00:00+00:00

The Role of Cluster Analysis in Automated Serial Electron Crystallography

2019-08-05T00:00:00+00:00

Automated serial rotation electron diffraction combined with cluster analysis: an efficient multi-crystal workflow for structure determination

2019-07-12T00:00:00+00:00

Towards automated serial electron diffraction for macromolecular crystallography

2019-07-02T00:00:00+00:00

Towards automated serial electron diffraction for macromolecular crystallography

2019-06-06T00:00:00+00:00

Preferential Siting of Aluminum Heteroatoms in the Zeolite Catalyst Al-SSZ-70

2019-05-06T00:00:00+00:00

Structure determination of nanocrystalline materials using electron diffraction

2019-03-18T00:00:00+00:00

Synthesis and Structure Determination of SCM-15: A 3D Large Pore Zeolite with Interconnected Straight 12x12x10-Ring Channels

2019-02-11T00:00:00+00:00

Quantitative Phase Analysis for Carbide Characterization in Steel Using Automated Electron Diffraction

2019-01-03T00:00:00+00:00

High-throughput continuous rotation electron diffraction data acquisition via software automation

2018-12-01T00:00:00+00:00

Site-Specific Adsorption of CO2 in Zeolite NaK-A

2018-11-05T00:00:00+00:00

Serial electron crystallography for structure determination and phase analysis of nanocrystalline materials

2018-10-01T00:00:00+00:00

High-Silica Zeolite SSZ-70: New Understanding of a Successful Catalytic Material

2018-08-22T00:00:00+00:00

CO2-Induced Displacement of Na+ and K+ in Zeolite |NaK|-A

2018-07-10T00:00:00+00:00

Structure determination of polycrystalline materials using X-rays and electrons

2018-07-01T00:00:00+00:00

Multidimensional Disorder in Zeolite IM-18 Revealed by Combining Transmission Electron Microscopy and X-ray Powder Diffraction Analyses

2018-02-27T00:00:00+00:00

Instamatic: a multi-purpose toolbox for collecting electron diffraction data

2017-12-09T00:00:00+00:00

Synthesis and Structure Determination of Large-Pore Zeolite SCM-14

2017-11-27T00:00:00+00:00

Well-Defined Silanols in the Structure of the Calcined High-Silica Zeolite SSZ-70: New Understanding of a Successful Catalytic Material

2017-11-22T00:00:00+00:00

Serial electron diffraction for structure determination and phase analysis of polycrystalline materials

2017-11-03T00:00:00+00:00

Structure analysis of polycrystalline materials with X-rays and electrons

2017-09-12T00:00:00+00:00

Serial electron diffraction for phase analysis and structure determination

2017-08-21T00:00:00+00:00

Location of Organic Structure-Directing Agents in Zeolites Using Diffraction Techniques

2017-08-17T00:00:00+00:00

Characterization of zeolites using powder diffraction: frameworks, heteroatoms, and SDAs

2017-07-03T00:00:00+00:00

The structure of zeolite SSZ-70 through combined HRTEM, XRPD, and 2D NMR

2017-07-03T00:00:00+00:00

The structure of zeolite SSZ-70 through combined HRTEM, XRPD, and 2D NMR

2017-06-13T00:00:00+00:00

Zeolite Structures

2017-06-07T00:00:00+00:00

Serial electron crystallography: merging diffraction data through rank aggregation

2017-06-01T00:00:00+00:00

Locating organic guests in inorganic host materials from XRPD

2016-09-22T00:00:00+00:00

Synthesis and Characterization of CIT-13, a Germanosilicate Molecular Sieve with Extra-Large Pore Openings

2016-09-13T00:00:00+00:00

Serial snapshot crystallography using electron diffraction

2016-08-28T00:00:00+00:00

Locating organic guests in inorganic host materials from XRPD

2016-06-12T00:00:00+00:00

Locating Organic Guests in Inorganic Host Materials from X-ray Powder Diffraction Data

2016-06-08T00:00:00+00:00

How do the positions of OSDAs determined from powder diffraction compare to molecular modelling

2015-09-14T00:00:00+00:00

Indexing of multi-crystal snapshots collected with a broad bandpass beam

2015-08-23T00:00:00+00:00

Ionothermal Synthesis and Structure of a New Layered Zirconium Phosphate

2015-08-17T00:00:00+00:00

How do the positions of OSDAs determined from powder diffraction compare to molecular modelling?

2015-07-26T00:00:00+00:00

Structure analysis of a BEC-type germanosilicate zeolite including the location of the flexible organic cations in the channels

2015-06-22T00:00:00+00:00

Serial snapshot crystallography for materials science with SwissFEL

2015-05-01T00:00:00+00:00

SSZ-87: A Borosilicate Zeolite with Unusually Flexible 10-Ring Pore Openings

2015-02-11T00:00:00+00:00

High-Silica Zeolite SSZ-61 with Dumbbell-Shaped Extra-Large-Pore Channels

2014-09-22T00:00:00+00:00

When two bad data sets are better than one

2014-08-05T00:00:00+00:00

SSZ-45: A High-Silica Zeolite with Small Pore Openings, Large Cavities, and Unusual Adsorption Properties

2014-07-08T00:00:00+00:00

Solvent-Mediated Reconstruction of the Metal-Organic Framework HKUST-1 (Cu3(BTC)2)

2014-07-01T00:00:00+00:00

Difficult structures come in a variety of flavors

2014-06-15T00:00:00+00:00

Synthesis, Structural Elucidation, and Catalytic Properties in Olefin Epoxidation of the Polymeric Hybrid Material [Mo3O9(2-[3(5)-Pyrazolyl]pyridine)]-n

2014-03-03T00:00:00+00:00

A two-dimensional zeolitic imidazolate framework with a cushion-shaped cavity for CO2 adsorption

2013-09-19T00:00:00+00:00

Using FOCUS and Superflip to solve structures from 3D electron and powder diffraction data

2013-08-25T00:00:00+00:00

Using FOCUS to solve zeolite structures from three-dimensional electron diffraction data

2013-08-01T00:00:00+00:00

Little myepisodes.com userscript (modified for Firefox)

2013-06-01T00:00:00+00:00

I have updated my myepisodes userscript to work with Firefox (link), which is apparently significantly different. It would be nice if at least all browsers had the same javascript standards... oh well.

To install, you need the Greasemonkey extension and then you can add this to it!

// ==UserScript==
// @name        myeps
// @include     http://www.myepisodes.com/*
// @version     1
// @grant       none
// ==/UserScript==

function myfunc()
 {
  // get table by class name as array
 arr = document.getElementsByClassName('showname')

  // loop over array
 for (var i = 0; i < arr.length; i++) {

    // get show name
   name = arr.item(i).textContent;

    // add torrentz.net search
   str = "http://torrentz.eu/search?f=" + name.replace(/ /g,"+")

    // generate link
   link = "<a href="+str+">"+name+"</a>"

    // generate 720p link
   link_720p = "<a href="+str+" +720p"+">"+"(720p)"+"</a>"

    // update cell
   arr.item(i).innerHTML = arr.item(i).innerHTML + " " + link_720p;
  }
 }

 // run after page has loaded
window.onload = myfunc;

Original post on web.archive.org.

Ab initio structure solution of a new zeolitic imidazolate framework

2012-10-28T00:00:00+00:00

Writing to NTFS on OS X

2012-08-17T00:00:00+00:00

Out of the box, OS X only supports reading from NTFS drives. The following steps explain how to install and where to find drivers to write to NTFS drives as well. From my experience, this works well on all recent versions of OS X, from Snow Leopard to Mountain Lion.

Step 1

Install osxfuse. For use with NTFS-3G below, make sure to tick the option "Install MacFuse compatibility layer". This is the general framework for extending OS X with new file system drivers.

Step 2

Install NTFS 3g. This is the NTFS driver for use with osxfuse. Make sure to reboot after installing. Writing to NTFS drives should now be possible. 2010.10.2 is the latest version available at the time of writing.

Step 3 (optional)

In Lion and newer you might experience a timeout bug every 15 seconds. This can be remedied with this patch: fuse_wait. To uninstall, simply reinstall the NTFS-3G driver.

More information is available on the osxfuse wiki page.

Original post on web.archive.org.

Git difftool

2012-08-07T00:00:00+00:00

To use your favourite visuall diff-tool with git:

git difftool -t xxdiff HEAD

Or 'HEAD' can be any particular file you want to diff.

git difftool

shows a list of programs it can use... if you happen to have them installed.

There is a thread on stackoverflow that details a few more advanced options, including setting up your own difftool, but I like this one the best, due to its simplicity.

Original post on web.archive.org.

Speed up Mountain Lion

2012-07-29T00:00:00+00:00

Some useful commands too speed up Mountain Lion I came across:

Disable smooth scrolling:

defaults write -g NSScrollAnimationEnabled -bool NO

Disable rubber banding (Sadly doesn't work everywhere, applications will need to restart after):

defaults write -g NSScrollViewRubberbanding -int 0

Reset rubber banding:

defaults delete -g NSScrollViewRubberbanding

Speed up the Dock animation:

defaults write com.apple.dock autohide-time-modifier -float 0.40;killall Dock

Reset Dock animation:

defaults delete com.apple.dock autohide-time-modifier;killall Dock

2D dock, because the indicators on the new 3D dock are absolutely impossible to see:

defaults write com.apple.dock no-glass -bool true

That said, last year I posted some Lion commands that still work too :-)

Speed up Mission Control, by reducing the length of animations and reduce the delay before switching spaces:

defaults write com.apple.dock expose-animation-duration -float 0.15
defaults write com.apple.dock workspaces-edge-delay -float 0.15
killall Dock

You can modify the 0.15 to whatever you want, but this is my preferred settings.

Remove mouse acceleration (needs a relog):

defaults write .GlobalPreferences com.apple.mouse.scaling -1

Original post on web.archive.org

OS X terminal alerts

2012-06-22T00:00:00+00:00

Sometimes when I'm running a long command I want to be alerted when it's done, so I don't have to check the other window all the time to see if it's finished.

To do so, simply append printf '\a' to your command, like so:

sleep 5; printf '\a'

This will add a 'bell' character to the message, which will bounce the dock icon and add a little badge if it is out of focus, very useful!

Original post on web.archive.org

Little myepisodes.com userscript

2012-05-01T00:00:00+00:00

As a little project for today to learn some javascript, I decided to modify one of my favourite websites myepisodes.com and make it a little bit more useful, by adding torrent links directly to the show list >:)

Down below is the userscript that will modify the contents of each cell and update the link from tvrage to a torrent search engine.

Installation for Opera:

Go to Preferences > content > javascript options > User javascript folder.

Set a folder to put your user javascripts if you haven't, otherwise add this script to that folder.

// ==UserScript==
// @include http://myepisodes.com/*
// ==/UserScript==

function myfunc()
 {
  // get table by class name as array
 arr = document.getElementsByClassName('showname')

  // loop over array
 for (var i = 0; i < arr.length; i++) {

    // get show name
   var name = arr.item(i).innerText;

    // add torrentz.net search
   var str = "http://torrentz.eu/search?f=" + name.replace(/ /g,"+")

    // generate link
   link = "<a href="+str+">"+name+"</a>"

    // generate 720p link
   link_720p = "<a href="+str+" +720p"+">"+"(720p)"+"</a>"

    // update cell
   arr.item(i).innerHTML = link + " " + link_720p;
  }
 }

 // run after page has loaded
window.onload = myfunc;

Original post on web.archive.org.

Co-sort multiple python lists

2012-03-08T00:00:00+00:00

I have written a small function to sort multiple lists. The values are sorted according to the values in the first list first, then the second etc. It can take any number of lists and returns the same number.

def cosort(*lsts):
    """Takes a any number of lists and sorts them based on the values of the first list."""
    tmp = zip(*lsts)
    tmp.sort()
    return zip(*tmp)

Little testing script:

from random import shuffle

def cosort(*lsts):
    """Takes a few lists and sorts them based on the values of the first list."""
    tmp = zip(*lsts)
    tmp.sort()
    return zip(*tmp)

a = range(10)
b = ['a','b','c','d','e','f','g','h','i','j']
c = ['j','i','h','g','f','e','d','c','b','a']

shuffle(a)

for i in range(len(a)):
    print a[i],b[i],c[i]

a,b,c = cosort(a,b,c)

print
for i in range(len(a)):
    print a[i],b[i],c[i]

Original post on web.archive.org.

Install fennec on Nokia N9

2012-01-31T00:00:00+00:00

Compiled deb packages have kindly been provided here: http://romaxa.bolshe.net/fennec/

Either download on your phone directly or dump it on there via USB. Next make sure to disconnect the USB cable and open up the terminal.

First we need root:

devel-su

Password: rootme

vi /etc/apt/sources.list.d/harmattan-dev.list

hit i for text entering mode and add the following repo:

deb http://harmattan-dev.nokia.com/ harmattan/sdk free non-free

hit esc to exit text-entering mode. The file is automatically saved. Doing so will add this source to the package manager apt. This particular repo gives us access to a library we need for installing Fennec, libpango, a library for laying out and rendering of text:

apt-get install libpango1.0-0

Now, it's time to install Fennec:

cd /home/user/MyDocs

or if you downloaded it on the phone:

cd /home/user/MyDocs/Download
dpkg -i fennec_12.0~a1_armel.deb

(or whatever version you downloaded, just hit f and tab :) )

And it should run with no errors and the Fennec icon will appear in your home screen!

Original post on web.archive.org.

How to fix "error trying to exec f951: execvp: No such file or directory"

2011-11-24T00:00:00+00:00

error trying to exec 'f951': execvp: No such file or directory
error trying to exec 'cc1': execvp: No such file or directory

I was trying to compile a program today on OS X Lion using gfortran, but to my disappointment I got these errors every time. I am not really familiar with this kind of stuff, so I wasn't sure where to look at first. Even when trying to compile a simple hello_world program.

The internet wasn't very useful either. Google gave me barely anything :-(

The solution

First I removed xcode, with the idea to reinstall it later:

sudo /Developer/Library/uninstall-devtools -mode=all

After that I dug into my /usr/local/lib and /usr/bin folders and removed any reference to gcc, gcov, g++, cpp and gfortran, which are all part of the gcc package.

I also removed all references to llvm-* (where * can be gcc, g++, etc). This comes with xcode, not really sure what this is used for. And finally I removed the libgcc/ folder and all libgfortran-related files in /usr/local/lib.

I rebooted for good measure.

Next I got Xcode 4 from the Appstore (for OS X 10.6 and lower, it's available on the Mac Dev Center and installed it. Again I went into my /usr/bin/ and removed the symlink xcode made to their (incomplete, it misses gfortran) gcc.

I downloaded and installed gcc-lion.tar.gz from HPC and installed using:

gunzip gcc-lion.tar.gz
sudo tar -xvf gcc-bin.tar -C /

After that, everything seemed to work! There is without doubt a faster way to fix it, but after trying to fix it for a few hours, I'm glad it now works! :-)

What else?

If gcc gives an error that looks something like this:

ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status,

an explanation can be found here. Basically, the lstdc++ parameter has to be given, so it uses the correct library. Alternativey, g++ or clang++ can be used directly.

Useful links:

http://www.webmo.net/support/fortran_osx.html http://hpc.sourceforge.net/

Original post on web.archive.org.

Hexakis(urea-kO)zinc(II) dinitrate at 110 and 250 K: uniaxial negative thermal expansion

2011-02-15T00:00:00+00:00

Temperature-dependent analysis of thermal motion, disorder and structures of tris(ethylenediamine)zinc(II) sulfate and tris(ethylenediamine)copper(II) sulfate

2011-02-01T00:00:00+00:00

catena-Poly[[bis-(ethylenediamine)copper(II)]-u-sulfato]

2010-06-01T00:00:00+00:00