There are some services that I expose to the internet (using Apache reverse proxy) that really should be accessed by only a small set of devices. Requiring client certificates seems like a great way to reduce the attack surface and prevent brute force attacks (since the attacker doesn’t even get a chance to attempt a login).
I wonder about the difficulty on the client side as well as other practical implications. The clients are smartphones of various makes.
I use Tailscale for this, with my own Headscale server so that I am in control of everything. It’s a much easier approach that is more likely to pass the spousal acceptance factor.
This. Tailscale just works and has functioning VPN on demand for your networks.
I’m already running Headscale, and it works great. But to expose individual services to individual devices it feels like an overkill. I don’t actually need all these devices to connect to the tailnet all the time, and some of these devices I don’t even want to be able to access the entire tailnet.
I recommend taking a look at the new Tailscale access controls > policies (aka “grants”). Much easier to understand than their old ACLs. You can quickly draw up rules that only let specific devices access specific nodes and even only specific ports.
There’s one small potential point of confusion, in that you can’t use node names directly in the rules. You have to go to access controls > definitions > hosts and make up a name there assigned to the node IP address, and then you can use that name in a policy.
In other words, even if you already have a tailnode called “nas” with a fixed IP, you can’t just say “nas” in a policy. You have to go to hosts, define one called “nas” that points to that tailnode’s IP, and then you can use “nas” in the policy… 🤪
I understand the logic, which is that hosts and definitions in general are much more powerful and can define IP netmasks and IP groups and then you can use those groups in policies… but boy, the redundancy when you have to do this for single nodes that are already assigned a name and an IP is rubbing me wrong.
I use mTLS with Caddy to expose some of my services. It is quite manageable, and I only use if for myself and my wife.
We both use Android phones, and I configure access via apps for the services, that use the devices’ certificate store. It seems like the iOS way is to provide the client certificate and password to each app that’s gonna use it. I’m happy we can avoid that.
Big plus side for us is the simplicity of it all, there is no “always on VPN” requirement, and things “just work” with acceptable security.
I also sometimes access the services via Firefox, which is also able to use the device’s certificate store, although I have to keep selecting the same certificate each time.
It’s very easy to make a custom CA and issue certs. Here’s a good tutorial.
Unfortunately in practice It depends greatly on what’s on the other side (the client app). Some examples:
- DAVx5 on Android works perfectly fine and uses the client cert from the system store. 10/10, this is how all apps should work.
- Ntfy on Android works perfectly fine but wants the client cert file loaded in the app, it doesn’t use the one loaded in the system store. This sucks because instead of loading a cert into the system store once and then deleting it you have to keep the cert file around for this kind of apps, in Android shared storage, which is accessible to all apps.
- Same for Immich, wants the cert loaded in the app. Also, it will randomly lose it (on both iOS and Android). Yes, you heard that right. So it’s basically useless and I had to resort instead to a key in a custom HTTP header; which isn’t exactly the same as mTLS, but helps secure the service at reverse proxy level so it’s better than nothing.
- Firefox on Android will use the cert from the system store, and then it will crash. Again, useless.
Oh did I mention how you get a mTLS client cert to an app on an iOS device? You send it over email to an account that the device has access to through the Mail app, then share the attachment. Yep.
It’s also not exactly straightforward to use mTLS with reverse proxies.
Let’s take for example Caddy and say you want unconditional mTLS for all reverse proxy hosts. Easy enough:
tls /path/to/domain-cert/fullchain.pem /path/to/domain-cert/privkey.pem { client_auth { mode required trust_pool file /path/to/custom/ca.pem } }But suppose you don’t want unconditional mTLS, you’d like to let clients in if they have mTLS or a custom header, or do different things depending if the client has valid mTLS or not. Does Caddy offer a built-in conditional to act on mTLS status? Nope!
As a workaround I’m setting the client_auth mode to
verify_if_givenand then using a DIY conditional that checks if the variablehttp.request.tls.client.certificate_der_base64is empty or not. But it’s undocumented so who knows if it may break at any point.For reference, how you handle both custom headers and mTLS at once (after setting the mode as I’ve mentioned):
@immich host "whatever.example.com" handle @immich { @not_authorized { not header X-Custom-Pass "LONGRANDOMKEY01" # jim not header X-Custom-Pass "LONGRANDOMKEY02" # bob vars_regexp {http.request.tls.client.certificate_der_base64} ^$ } error @not_authorized 403 reverse_proxy http://immich.lan:port }The nested “not not” is required because Caddy can only do logical AND in group conditionals, so to do logical OR you basically have to do NOT (NOT a AND NOT b).
In caddy, isn’t an OR is just two separate blocks?
@authorized { header X-Custom-Pass “LONGRANDOMKEY01” # jim } @authorized { header X-Custom-Pass “LONGRANDOMKEY02” # Bob }
I could be wrong. Caddy config is a dark art.
Lol I don’t know, I’ll have to try. I would’ve never thought of it. If it works it’s definitely not intuitive. Dark art indeed.
I use it for Home Assistant and ntfy in combination with Caddy. Certificates are managed by Vaultls (https://github.com/7ritn/VaulTLS) which makes it quite easy to setup and use on new devices.
It can be a pain to manage but it works well once it is setup
Just make sure it isn’t your only line of defense
I am using mTLS implemented by nginx to access my services like Home Assistant, Paperless, Tandoor, Immich and so on. Clients are Windows, Linux and Android based so no iOS experience. Adding new clients can be tricky if you need to figure out how to provide the certificate first. Once it works, it’s rock solid
It’s a good option if you can’t do anything better, like a VPN. Because there is always the risk of an authentication bypass vulnerability. The less attack surface, the better.
Ditto VPN vulnerabilities. Belt and suspenders can help though if a remote is compromised likely all the things it uses will be, if not immediately then eventually.