Track the resolver setup that only existed on the box - #9
Merged
Conversation
moshpit-dns.service and the systemd-resolved drop-in were configured by hand on a live machine and lived in no repository, so the working DNS setup was one disk failure from being lost and impossible to reproduce. Both are here as deployed, with the reasoning that is easy to get wrong: - the bridge runs as the machine's primary resolver, with no list of Moshpit endings at all. A per-TLD list cannot work -- 5,661 routing domains blow systemd-resolved's ~1090 cap, and even a curated list fails because a routing domain selects a *scope*, and an upstream in that scope answers NXDOMAIN before the bridge is asked. - the drop-in is named 00- so it is read first; resolved uses the first server and rotates only on failure, never on NXDOMAIN, so pre-existing resolvers remain a real fallback instead of shadowing the bridge. - MOSHPIT_DNS_CATCHALL must stay unset: it parks every unresolved name, so github.com resolves to a parking page. Restart=always because it is the resolver now; it replaced an orphaned process that would have stayed dead.
vu1nz Security Review0 finding(s) in PR #? No security issues found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
moshpit-dns.serviceand the systemd-resolved drop-in were configured by hand on a live machine and lived in no repository. The working DNS setup was one disk failure from being lost, and impossible to reproduce on a second box.Both are committed here exactly as deployed (verified byte-identical to the running machine), along with the reasoning that is easy to get wrong.
The bridge runs as the primary resolver, with no TLD list at all
That is not a shortcut. A per-TLD list cannot be made to work, for two independent reasons:
1. It does not scale.
moshcode dns install --writeemits every claimed ending as a routing domain — 5,661 on one line, ~60KB. systemd-resolved caps search domains near 1090 and drops the rest alphabetically:8,199 of those in a single restart.
~hackerand~eggsfell past the cut — configured on disk, absent from the resolver.2. Curating the list does not fix it either. A routing domain selects a scope, and a scope tries its servers in order. With an upstream in the same scope,
.hackerwent to67.207.67.3, came back NXDOMAIN, and systemd-resolved treated that as final. The bridge was never asked. This is why the "just trim the list" fix kept appearing to work and didn't.Sending every query to the bridge removes both problems. It answers Moshpit names from the registry and forwards the rest upstream (
mode=clearnet).Two things that cost real debugging time
The
00-prefix is load-bearing. systemd-resolved appendsDNS=in filename order and uses the first server, rotating only on failure — never on NXDOMAIN. So pre-existing resolvers stay listed as a genuine fallback if the bridge stops, instead of shadowing it. Same servers, same list: ordering is the entire difference between a safety net and the bug.MOSHPIT_DNS_CATCHALLmust stay unset. It parks every unresolved name on the registry's parking address, so on a machine's own resolvergithub.comresolves to a parking page. Documented alongside the symptom to look for: if clearnet names start returning a69.46.46.xaddress, check for a straymoshcode dns startbound to127.0.0.1:5354— a loopback bind beats the service's0.0.0.0bind and silently wins every local query. That was the actual root cause of the outage.Restart=alwaysbecause it is the resolver now. It replaced an orphaned process (PPID 1, no unit) that would have stayed dead.Note on placement
moshpit-dnswould be the more natural home, but that directory is not a git repository on any machine I can see. This repo already carries the deployment-side config (nginx/moshpit-gateway.conf,nginx/moshpit-origin.conf), so it is the closest existing home. Worth moving ifmoshpit-dnsever gets version-controlled.🤖 Generated with Claude Code