Resolve the UN API token from the environment or a per-user file

[marcelolafleur]

One token per user instead of one per directory. Addresses #1205.

get_un_data looked for un_api_token.txt in the current working directory, and wrote one there when it didn't find one. So a copy built up in every folder anyone ran from.

It now looks in this order, first hit wins:

1. un_token= argument
2. UN_API_TOKEN environment variable
3. $XDG_CONFIG_HOME/og/un_api_token.txt (%APPDATA%\og\ on Windows)
4. un_api_token.txt in the working directory — still read, prints a deprecation notice

The prompt stays, since it's how people find out they need a token, but it now names both ways forward instead of leaving you to guess:

OG-Core can read population data directly from the UN Data Portal, which needs a free API token.
  To get one, open https://population.un.org/dataportalapi/index.html and click Generate Token.
  Or press return to use the archived copy of the same data at https://github.com/EAPD-DRB/Population-Data.

UN API token (input is hidden):

The token is read with getpass, so it is not echoed into the terminal or its scrollback. master used input() here, which showed the token as you typed it. Answering saves to the per-user file. Either way you are only asked once per machine, not once per folder. The prompt is skipped when stdin isn't interactive, so scheduled runs fall back to the archive instead of waiting on input.

People usually get a token later rather than at that moment, so a run that falls back to the archive says once how to register one:

No UN API token registered, so the archived data will be used.
  Get a free token at https://population.un.org/dataportalapi/index.html
  Then run: /Users/you/Projects/OG-PHL/.venv/bin/og-token set
  (or save the token to /Users/you/.config/og/un_api_token.txt)

The full path is there because og-token usually isn't on your PATH. The file is named too, in case that is easier. Printed once per run.

An expired token used to fail the same way as a missing one, so it now says so:

Your UN API token expired on 2026-08-11, so the archived data will be used.
  Get a new one at https://population.un.org/dataportalapi/index.html
  Then run: /Users/you/Projects/OG-PHL/.venv/bin/og-token set

The portal issues JSON Web Tokens, so the date is read from the token itself with no extra request and no signature check — the portal is still the authority on whether a token works. A token that isn't a readable JWT is used exactly as before, with no expiry shown. This matters more than it sounds: every token file on my machine had already expired, one of them 456 days ago, and nothing had ever said so.

There's also an og-token command so nobody has to hunt for the file:

og-token set    # asks once, saves to the config file
og-token show   # where it lives, which source wins, whether it has expired
og-token rm

It reads the token without echoing, and never prints it back.

Two things to flag rather than have you find them:

• This is OG-Core's first [project.scripts] entry. It's a separate commit, so it's easy to drop if you'd rather not take that on.
• No new dependency. The config path is eight lines instead of adding platformdirs, which is only an indirect dependency today.

Version bumped to 0.20.0 with a dated changelog section, following #1190. Happy to change the number.

Tested

• 35 new tests, none of them touch the network. One fails if the prompt ever goes back to input(), since that would echo the token; another checks the expiry parser returns None rather than raising on anything that isn't a readable JWT.
• Putting the old write-to-cwd behaviour back turns the main test red, so it is actually covering the change.
• One test checks the resolved token reaches the Authorization header, not just that the resolver returns the right string.
• Ran og-token end to end from an installed console script: saves, strips a Bearer prefix, writes the file 0600, prints only the last four characters, and gives a usage error on a bad argument.
• Live check against the portal: with UN_API_TOKEN set, get_un_data("47", country_id="608") returns 100 rows for ages 0-99 with no fallback to the archive, so the resolved token really does reach the request.
• Built the wheel and installed it into a clean venv on Python 3.13. og-token lands on PATH, and get_pop_objs with no token anywhere falls back to the archive and solves as before, without prompting or leaving a token file behind. An existing un_api_token.txt in the working directory still resolves, with the deprecation notice.
• ruff format --check . and ruff check . both clean. 48 demographics tests pass under CI's -m "not local and not benchmark", plus 485 across the rest of the suite.
• Full CI matrix green, including both Windows legs, which is where the %APPDATA% path, os.chmod, and og-token.exe in Scripts/ differ from what I could test locally.
• Not run: the docs build and the local-marked tests. The docs change adds the three new public functions to the existing :members: list, and I checked every name in that list resolves and has a docstring.
• og-token imports ogcore, so it pays about 1.2s of import before doing anything. Fine for a command run once, but I didn't try to avoid it.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.69048% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.07%. Comparing base (2b6a668) to head (c6f26b7).

Files with missing lines	Patch %	Lines
ogcore/demographics.py	88.62%	19 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1206      +/-   ##
==========================================
+ Coverage   72.74%   74.07%   +1.33%     
==========================================
  Files          22       22              
  Lines        5768     5922     +154     
==========================================
+ Hits         4196     4387     +191     
+ Misses       1572     1535      -37     

Flag	Coverage Δ
unittests	74.07% <88.69%> (+1.33%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
ogcore/__init__.py	100.00% <100.00%> (ø)
ogcore/demographics.py	72.83% <88.62%> (+14.33%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[arihantlodha-cmd]

Really like this, and I can speak to it from the user side: I hit exactly the problem it fixes. Running OG-Japan I ended up with un_api_token.txt copies in two different directories, and my non-interactive background runs kept tripping the old input() prompt (which then wrote an empty file). The env var, the per-user file, and skipping the prompt when stdin isn't a tty all land on precisely those pain points.

I also verified the expiry decode against a real UN-issued token rather than a synthetic one: my portal token decodes cleanly with un_token_expiry to its expiry date, with aud = data-portal-api. So reading the JWT exp locally works on what the portal actually hands out, and an expired token now gets a clear message instead of silently falling back to the archive, which is exactly how it fails today. Nice touch.

One small thing, non-blocking: unless I missed it, the per-user file is written with default permissions, but it holds a credential, so on a shared machine it's world-readable. Might be worth creating it 0600 (open with mode 0o600, or chmod after write), the way the aws/gh CLIs treat their token files.

getpass instead of input so the token isn't echoed is also a good call. Thanks for this, it's the fix I wanted while working with the token.

[marcelolafleur]

@arihantlodha-cmd
Thanks for checking this against a real token from the portal.

On permissions: the file is set to 600, with a chmod straight after it is created. If you would prefer it created as 600 directly, I can make that change.