You need a VPN. It will cause inconveniences but the way currently have it setup is highly vulnerable to attack. I would look into something like Netbird or just a plain Wireguard VPN. If they are connecting primarily from home you could always setup a simple proxy device that forwards connections over a VPN from the local network. Security should absolutely come first. If you can’t provide basic security I wouldn’t recommend hosting anything especially for other people.