One way to solve this: do not load the “Backend” or “Editor” modules for the public domains (eg: www.mycustomdomain.tld) using environment variables. If you configure the webserver to set the LOAD_MODULES environment variable for the frontend domains:
# Only provide a frontend
LOAD_MODULES="System,Cms,Media"
With this module unloaded, the route will not be registered and have no available backend. Likewise you can do the reverse for the backend environment
# Only provide a backend
LOAD_MODULES="System,Backend,Editor"
I hope this helps. This works in theory but there may be some bugs involved, but we will be happy to fix them for you.