Question about ever growing swap space on raspi 4 4 GB

I run buster on a raspberry pi4 with 4 GB RAM.
I notice swap space is getting exhausted slowly.

Uptime:
 07:37:54 up 12 days, 17:46, 17 users,  load average: 0.35, 0.62, 0.67

Top shows this now:
 MiB Swap:    100.0 total,     20.7 free,     79.3 used.   3375.6 avail Mem

Here I found a way to see swap space per running process:
 https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/

Running their script
 for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print
""}' $file; done | sort -k 2 -n -r | less

shows:
Xorg 9136 kB
rxvt 5648 kB
fvwm 4256 kB
rxvt 2548 kB
rxvt 2388 kB
rxvt 2196 kB
rxvt 2080 kB
rxvt 1796 kB
rxvt 1764 kB
rxvt 1764 kB
rxvt 1636 kB
(sd-pam) 1580 kB
zsh 1144 kB
zsh 1124 kB
bash 892 kB
(sd-pam) 752 kB
systemd-udevd 652 kB
systemd-journal 624 kB
systemd 584 kB
systemd 548 kB
su 508 kB
xterm 504 kB
xfm 504 kB
wpa_supplicant 496 kB
ssh 492 kB
rxvt 488 kB
rxvt 484 kB
rxvt 464 kB
rxvt 464 kB
rxvt 464 kB
etc etc sorted to ever less

So it seems the system is using more and more little bits of swap space for
each program!

The same on my lapop with even less memory shows zero swap for all programs!

The consequence would be that if I leave this raspi running (security camera
recordings)
it will soon run out of swap space.

It this an OS (buster) problem or something else?
All that runs is some terminals and ffmpeg instances.
The same on my laptop works without swap usage

Any ideas what causes this swapping and / or how to stop it?

--- SoupGate-Win32 v1.05