Add the Insights page: usage chat and takeaways on hototel's own Modal model

Dockerfile

@@ -1,7 +1,11 @@
 # hototel server: ingest + admin dashboard, all state in hotdata.
 FROM python:3.12-slim
 
-RUN pip install --no-cache-dir hotdata duckdb && \
+# hotdata-langchain[agents] + langchain-anthropic back the Insights page; the
+# server boots without them (the page says it is off), but the image
+# carries them so setting ANTHROPIC_API_KEY is all that turns it on.
+RUN pip install --no-cache-dir hotdata duckdb \
+        "hotdata-langchain[agents]" langchain-anthropic langchain-openai && \
     useradd --create-home --uid 10001 appuser
 
 WORKDIR /app
@@ -13,7 +17,10 @@ USER 10001
 # Config via env: HOTDATA_API_KEY (required), HOTUSAGE_INGEST_TOKEN (required
 # outside dev; its value is not itself a credential unless
 # HOTUSAGE_ALLOW_SHARED_INGEST=1), HOTDATA_WORKSPACE / HOTDATA_API_HOST
-# optional overrides.
+# optional overrides. The Insights page needs a model: HOTOTEL_INSIGHTS_BASE_URL +
+# HOTOTEL_INSIGHTS_API_KEY for the Modal deployment (modal/hototel_llm.py), or
+# ANTHROPIC_API_KEY for the Anthropic API. HOTOTEL_INSIGHTS_MODEL overrides the
+# model name; HOTOTEL_INSIGHTS=0 forces the page off.
 
 # This image only ever runs behind App Runner, which rewrites X-Forwarded-For,
 # so the rate limiter may believe it here. A bare `python3 server.py` gets the

README.md

@@ -40,6 +40,27 @@ Use them to compare people, projects and trends; for actual spend see the spend
 report at `claude.ai/admin-settings/usage`, which exports per-user cost as a
 daily CSV.
 
+## Insights
+
+The **Insights** page, linked from the header, is the dashboard you can talk
+to. Ask a question in plain words — *who worked on the billing service
+this month?*, *which sessions burned context without producing anything?*
+— and a model answers it by running SQL against your organization's
+usage, listing every query it ran under the answer so a number can be checked
+rather than trusted.
+
+Above the chat, **Takeaways** is the same analyst's read of the last 30 days,
+sorted into three columns: **good work** (practices worth spreading),
+**valuable** (where the effort concentrates and appears to pay off) and
+**redundant** (duplication or waste worth a look). It regenerates on demand
+and otherwise every six hours.
+
+The model only ever sees your organization's database, and only members
+signed in to the dashboard can ask. On hototel.com the model is an open-weight
+one hototel runs itself, so your usage is not sent to a third-party AI
+provider. The page is off until the operator connects a model, and says so
+when it is.
+
 ## Running an organization
 
 Everything is on the **Organization** page, linked from the header.

docs/operating.md

@@ -119,6 +119,65 @@ server.py markverified jane@acme.com
 
 Links live 24 hours; resends are rate limited per IP.
 
+## Insights (the chat page)
+
+`/insights` is a LangChain agent built with
+[hotdata-langchain](https://github.com/hotdata-dev/hotdata-langchain), in
+`server/insights.py`. It gets two tools -- run SQL, describe tables -- both
+built with `database_id=` set to the viewer's org database, so the scope is
+fixed before the model sees a prompt and the SQL tool's only argument is the
+query. The instant-database tools (list, create, load) are not registered; the
+agent cannot write or reach another organization.
+
+It needs a model. The default deployment is hototel's **own model on Modal**:
+`modal/hototel_llm.py` serves Qwen3-30B-A3B-Instruct-2507 behind vLLM's
+OpenAI-compatible API on one H100, scaling to zero after ten idle minutes, so
+usage rows never leave the company's own accounts. Deploy it once and hand the
+server its URL and key:
+
+```bash
+modal secret create hototel-llm HOTOTEL_LLM_API_KEY=<random string>
+modal deploy modal/hototel_llm.py          # prints the endpoint URL
+export HOTOTEL_INSIGHTS_BASE_URL=https://<workspace>--hototel-server.<region>.modal.direct/v1
+export HOTOTEL_INSIGHTS_API_KEY=<the same random string>
+```
+
+`modal run modal/hototel_llm.py` waits for the server and sends one test
+request with a tool attached, which is the quickest way to confirm tool
+calling works after a model change. The first request after idle waits a
+minute or two for the cold start; the very first ever boot also downloads
+the weights (~60GB) into the `hototel-hf-cache` volume.
+
+The alternative is the **Anthropic API**: leave `HOTOTEL_INSIGHTS_BASE_URL`
+unset and set `ANTHROPIC_API_KEY`; the model defaults to `claude-opus-5-5`.
+Either way `HOTOTEL_INSIGHTS_MODEL` overrides the model name (on Modal the
+server answers to the alias `hototel`) and `HOTOTEL_INSIGHTS=0` forces the
+page off. `/api/insights/status` says which provider is active or what is
+missing. The packages (`hotdata-langchain[agents]`, `langchain-openai`,
+`langchain-anthropic`) are in the Docker image.
+
+Two endpoints, both cookie-session only -- deliberately not in
+`READ_TOKEN_PATHS`, since a read token was approved to pull numbers, not to
+start paid model runs:
+
+- `POST /api/insights/chat` takes the whole transcript (`messages`, user and
+  assistant turns only, at most 24 of them) and returns the answer plus the
+  queries the agent ran. The server keeps no conversation state.
+- `GET /api/insights/takeaways` returns the cached 30-day analysis for the
+  org, or `status: pending` while one generates in a background thread; the
+  page polls. `?fresh=1` starts a new run. The cache is in-process, so a
+  restart (or a second App Runner instance) regenerates on first view.
+
+Both count against one per-IP limit (`Handler.CHAT_RATE_LIMIT`, 60 runs an
+hour), and at most four agent runs execute at once per process so a burst of
+questions cannot take the thread pool away from `/ingest`. A chat holds its
+HTTP request open for the run, typically 20-90 seconds.
+
+On Modal the cost is GPU time while a container is up (an H100 is a few
+dollars an hour, and it stops ten minutes after the last request); on the
+Anthropic API a chat run is a few dollars of tokens at most and a takeaways
+run somewhat more. Nothing here prices anything or writes to hotdata.
+
 ## Platform operators (system admins)
 
 Creating or deleting an organization provisions or strands a hotdata database,

modal/hototel_llm.py

@@ -0,0 +1,159 @@
+"""hototel's own model endpoint on Modal: an open-weight model behind vLLM's
+OpenAI-compatible API, which the Insights page talks to through LangChain's
+ChatOpenAI instead of the Anthropic API.
+
+Why this exists: the Insights agent reads usage rows -- who worked on what,
+session titles, machine names -- and sends them to a model. Serving the model
+here keeps that inside the company's own Modal account rather than a third
+party's API, and bills by GPU-second while a container is up rather than by
+token.
+
+Deploy once (Modal CLI signed in):
+
+    modal secret create hototel-llm HOTOTEL_LLM_API_KEY=<random string>
+    modal deploy modal/hototel_llm.py
+
+then give the server the URL `modal deploy` prints (the "flash service endpoint")
+plus that key:
+
+    export HOTOTEL_INSIGHTS_BASE_URL=https://<workspace>--hototel-server.<region>.modal.direct/v1
+    export HOTOTEL_INSIGHTS_API_KEY=<the same random string>
+
+The container scales to zero after SCALEDOWN minutes idle. The first request
+after that waits for a cold start -- a minute or two with the weights already
+in the volume, longer the very first time while they download.
+
+Model: Qwen3-30B-A3B-Instruct-2507, a mixture-of-experts model with 3B active
+parameters per token. It is chosen for this workload over a dense 32B: the
+agent loop is many short model turns each followed by a SQL round trip, so
+tokens per second matters more than peak reasoning, and it is one of the
+stronger open models at tool calling. Apache 2.0, no Hugging Face gate.
+"""
+
+import os
+
+import modal
+
+MINUTES = 60
+VLLM_PORT = 8000
+N_GPU = 1
+SCALEDOWN = 10 * MINUTES
+
+MODEL_NAME = "Qwen/Qwen3-30B-A3B-Instruct-2507"
+# Pinned so a repo update upstream cannot change what answers questions here.
+MODEL_REVISION = "0d7cf23991f47feeb3a57ecb4c9cee8ea4a17bfe"
+# The short alias the server addresses it by, so the model can be swapped
+# here without touching HOTOTEL_INSIGHTS_MODEL on the hototel server.
+SERVED_NAME = "hototel"
+
+vllm_image = (
+    modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12")
+    .entrypoint([])
+    .uv_pip_install("vllm==0.21.0")
+    .env({
+        "HF_XET_HIGH_PERFORMANCE": "1",
+        # Stay on the prebuilt Triton MoE kernels. Left to itself vLLM picks
+        # FlashInfer's CUTLASS MoE for this model and JIT-compiles it with
+        # ninja on first run, which took longer than Modal's startup timeout
+        # and killed the container before it ever served a request.
+        "VLLM_USE_FLASHINFER_MOE_FP16": "0",
+    })
+)
+
+hf_cache_vol = modal.Volume.from_name("hototel-hf-cache", create_if_missing=True)
+vllm_cache_vol = modal.Volume.from_name("hototel-vllm-cache", create_if_missing=True)
+
+app = modal.App("hototel")
+
+
+@app.server(
+    image=vllm_image,
+    gpu=f"H100:{N_GPU}",
+    scaledown_window=SCALEDOWN,
+    startup_timeout=15 * MINUTES,     # the first boot downloads ~60GB of weights
+    volumes={
+        "/root/.cache/huggingface": hf_cache_vol,
+        "/root/.cache/vllm": vllm_cache_vol,
+    },
+    secrets=[modal.Secret.from_name("hototel-llm")],
+    port=VLLM_PORT,
+    target_concurrency=16,            # agent runs are few; 16 is plenty per replica
+    # Modal's own proxy auth is off because vLLM authenticates the request
+    # itself: VLLM_API_KEY (set in start) makes every call carry a Bearer
+    # token, the one shape every OpenAI-compatible client already sends.
+    unauthenticated=True,
+)
+class Server:
+    @modal.enter()
+    def start(self):
+        import subprocess
+
+        api_key = os.environ.get("HOTOTEL_LLM_API_KEY", "")
+        if not api_key:
+            raise RuntimeError("secret hototel-llm must define HOTOTEL_LLM_API_KEY")
+        cmd = [
+            "vllm", "serve", MODEL_NAME,
+            "--revision", MODEL_REVISION,
+            "--served-model-name", SERVED_NAME, MODEL_NAME,
+            "--host", "0.0.0.0", "--port", str(VLLM_PORT),
+            "--uvicorn-log-level=info",
+            "--tensor-parallel-size", str(N_GPU),
+            # the agent's whole transcript -- system prompt, tool schemas, a
+            # dozen query results -- fits comfortably; more only costs KV cache
+            "--max-model-len", "32768",
+            "--gpu-memory-utilization", "0.92",
+            # tool calling: Qwen3 writes Hermes-style <tool_call> blocks
+            "--enable-auto-tool-choice",
+            "--tool-call-parser", "hermes",
+        ]
+        print(*cmd)
+        # The key goes in through the environment, not --api-key: vLLM echoes
+        # its non-default arguments to the log at boot, and those lines land
+        # in Modal's log store, where a flag value would sit in plain text.
+        self.process = subprocess.Popen(cmd, env={**os.environ, "VLLM_API_KEY": api_key})
+
+    @modal.exit()
+    def stop(self):
+        self.process.terminate()
+
+
+@app.local_entrypoint()
+def test(content: str = "Reply with the single word: ready"):
+    """`modal run modal/hototel_llm.py` -- wait for the server and ask it one
+    thing, with a tool offered, to prove tool calling is wired."""
+    import json
+    import time
+    import urllib.error
+    import urllib.request
+
+    url = Server.get_url()
+    key = os.environ.get("HOTOTEL_INSIGHTS_API_KEY") or input("HOTOTEL_LLM_API_KEY: ")
+    deadline = time.time() + 15 * MINUTES
+    while time.time() < deadline:
+        try:
+            with urllib.request.urlopen(url + "/health", timeout=30) as r:
+                if r.status == 200:
+                    break
+        except (urllib.error.URLError, TimeoutError):
+            pass
+        time.sleep(5)
+    else:
+        raise SystemExit(f"{url} never became healthy")
+    print(f"healthy: {url}")
+    body = {
+        "model": SERVED_NAME,
+        "messages": [{"role": "user", "content": content}],
+        "tools": [{"type": "function", "function": {
+            "name": "run_sql", "description": "Run SQL",
+            "parameters": {"type": "object", "properties": {"sql": {"type": "string"}},
+                           "required": ["sql"]}}}],
+        "max_tokens": 200,
+    }
+    req = urllib.request.Request(url + "/v1/chat/completions", data=json.dumps(body).encode(),
+                                 headers={"Content-Type": "application/json",
+                                          "Authorization": f"Bearer {key}"})
+    with urllib.request.urlopen(req, timeout=300) as r:
+        out = json.load(r)
+    msg = out["choices"][0]["message"]
+    print("answer:", msg.get("content"))
+    print("tool_calls:", msg.get("tool_calls"))