diff --git a/README.md b/README.md index e124ffd..9b2a159 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,49 @@ The collector runs successfully with the documented read-only permission set. So If these optional permissions are not granted, OpenHound skips the affected resources and continues collecting the rest of the GitHub environment. +### Classic personal access token inventory + +With GitHub Enterprise Cloud credentials, the collector uses the enterprise +credential inventory export to collect classic personal access tokens. It +requests the full inventory, stores every CSV row and its original field values +in the `enterprise_credential_inventory` resource, and currently models only +classic PAT rows. The export contains credential metadata, including owners, +scopes, authorization details, and token hashes used for audit correlation. It +does not contain plaintext PAT values. Limit access to the raw output to people +who need the credential inventory. + +The enterprise GitHub App needs **Enterprise credentials: read** permission. +When collecting with a classic PAT instead of an app installation, that token +needs the `read:enterprise` scope. If an enterprise app also has +`credentials.pat_token` configured, the app remains the primary export +credential. The collector retries once with the PAT only when GitHub rejects +the app's export creation for authorization. It does not switch credentials +after an export starts or when GitHub reports a rate limit. + +Rate-limit retry warnings identify the affected method and API path. + +The CSV download is streamed, but the parsed export is held in memory as one +raw record; very large enterprises may need a chunked raw-resource design. + +GitHub limits the number of exports an enterprise can start per day. When the +last successful export was downloaded within the past 24 hours, the collector +downloads it again before trying to start a new export. If GitHub reports that +the saved export is gone, the collector starts a new one and polls until its +CSV is ready. GitHub may still reject a new export when its limit is reached. +The raw record's `as_of` value and each PAT node's `inventory_as_of` value +show when that snapshot was taken. An unavailable or denied export is logged +and does not stop other resources from collecting. + +Classic PAT nodes include the owner, scopes, lifecycle dates, credential +state, direct enterprise authorization, and total authorization count. +Organization edges record authorizations reported in the export. The +export does not enumerate repositories accessible to a classic PAT, so this +collection does not emit classic PAT-to-repository access edges. Potential +repository access can be derived from the token owner's repository roles and +the token's scopes, subject to organization policy and SSO authorization; it +is not a direct grant reported by the export. GitHub Enterprise Server and +organization-only configurations skip this resource. + [![Python Version](https://img.shields.io/badge/Python-3.13-brightgreen.svg)](#about) ## Getting Started diff --git a/descriptions/edges/GH_AuthorizedForOrganization.md b/descriptions/edges/GH_AuthorizedForOrganization.md new file mode 100644 index 0000000..797c5b2 --- /dev/null +++ b/descriptions/edges/GH_AuthorizedForOrganization.md @@ -0,0 +1,20 @@ +# GH_AuthorizedForOrganization + +## General Information + +Classic personal access token has a recorded authorization for this organization. + +## Edge Schema + +| Source | Destination | Traversable | +| --- | --- | --- | +| `GH_ClassicPersonalAccessToken` | `GH_Organization` | `false` | + +## Diagram + +```mermaid +graph LR + n0["GH_ClassicPersonalAccessToken"] + n1["GH_Organization"] + n0 -.->|GH_AuthorizedForOrganization| n1 +``` diff --git a/descriptions/edges/GH_Contains.md b/descriptions/edges/GH_Contains.md index 7950e5a..6bb309a 100644 --- a/descriptions/edges/GH_Contains.md +++ b/descriptions/edges/GH_Contains.md @@ -2,12 +2,13 @@ ## General Information -The non-traversable GH_Contains edge represents structural containment within the GitHub resource hierarchy. The enterprise contains enterprise teams, roles, managed users, runner groups, and enterprise runners through their groups. The organization serves as a top-level container for users, teams, repositories, roles, secrets, app installations, personal access tokens, and organization runner groups. Native organization runner groups contain organization runners. Repositories contain branches, workflows, branch protection rules, environments, repo-level secrets and variables, and repository-scoped runners. Environments contain environment branch policies, environment-scoped secrets, and environment-scoped variables. This edge is created by the collector to establish the resource hierarchy and is not traversable because containment alone does not imply privilege escalation. +The non-traversable GH_Contains edge represents structural containment within the GitHub resource hierarchy. The enterprise contains enterprise teams, roles, managed users, classic personal access tokens, runner groups, and enterprise runners through their groups. The organization serves as a top-level container for users, teams, repositories, roles, secrets, app installations, fine-grained personal access tokens, and organization runner groups. Native organization runner groups contain organization runners. Repositories contain branches, workflows, branch protection rules, environments, repo-level secrets and variables, and repository-scoped runners. Environments contain environment branch policies, environment-scoped secrets, and environment-scoped variables. This edge is created by the collector to establish the resource hierarchy and is not traversable because containment alone does not imply privilege escalation. ## Edge Schema | Source | Destination | Traversable | | --- | --- | --- | +| `GH_Enterprise` | `GH_ClassicPersonalAccessToken` | `false` | | `GH_Enterprise` | `GH_EnterpriseRole` | `false` | | `GH_Enterprise` | `GH_EnterpriseRunnerGroup` | `false` | | `GH_Enterprise` | `GH_EnterpriseTeam` | `false` | @@ -42,60 +43,62 @@ The non-traversable GH_Contains edge represents structural containment within th ```mermaid graph LR n0["GH_Enterprise"] - n1["GH_EnterpriseRole"] - n2["GH_EnterpriseRunnerGroup"] - n3["GH_EnterpriseTeam"] - n4["GH_Organization"] - n5["GH_EnterpriseRunner"] - n6["GH_Environment"] - n7["GH_EnvironmentBranchPolicy"] - n8["GH_EnvironmentSecret"] - n9["GH_EnvironmentVariable"] - n10["GH_OrgRunnerGroup"] - n11["GH_OrgRunner"] - n12["GH_AppInstallation"] - n13["GH_OrgRole"] - n14["GH_OrgSecret"] - n15["GH_OrgVariable"] - n16["GH_PersonalAccessToken"] - n17["GH_PersonalAccessTokenRequest"] - n18["GH_SecretScanningAlert"] - n19["GH_Repository"] - n20["GH_Branch"] - n21["GH_BranchProtectionRule"] - n22["GH_DeployKey"] - n23["GH_RepoRunner"] - n24["GH_RepoSecret"] - n25["GH_RepoVariable"] - n26["GH_Workflow"] - n27["GH_WorkflowJob"] - n28["GH_WorkflowStep"] + n1["GH_ClassicPersonalAccessToken"] + n2["GH_EnterpriseRole"] + n3["GH_EnterpriseRunnerGroup"] + n4["GH_EnterpriseTeam"] + n5["GH_Organization"] + n6["GH_EnterpriseRunner"] + n7["GH_Environment"] + n8["GH_EnvironmentBranchPolicy"] + n9["GH_EnvironmentSecret"] + n10["GH_EnvironmentVariable"] + n11["GH_OrgRunnerGroup"] + n12["GH_OrgRunner"] + n13["GH_AppInstallation"] + n14["GH_OrgRole"] + n15["GH_OrgSecret"] + n16["GH_OrgVariable"] + n17["GH_PersonalAccessToken"] + n18["GH_PersonalAccessTokenRequest"] + n19["GH_SecretScanningAlert"] + n20["GH_Repository"] + n21["GH_Branch"] + n22["GH_BranchProtectionRule"] + n23["GH_DeployKey"] + n24["GH_RepoRunner"] + n25["GH_RepoSecret"] + n26["GH_RepoVariable"] + n27["GH_Workflow"] + n28["GH_WorkflowJob"] + n29["GH_WorkflowStep"] n0 -.->|GH_Contains| n1 n0 -.->|GH_Contains| n2 n0 -.->|GH_Contains| n3 n0 -.->|GH_Contains| n4 - n2 -.->|GH_Contains| n5 - n6 -.->|GH_Contains| n7 - n6 -.->|GH_Contains| n8 - n6 -.->|GH_Contains| n9 - n10 -.->|GH_Contains| n11 - n4 -.->|GH_Contains| n12 - n4 -.->|GH_Contains| n13 - n4 -.->|GH_Contains| n10 - n4 -.->|GH_Contains| n14 - n4 -.->|GH_Contains| n15 - n4 -.->|GH_Contains| n16 - n4 -.->|GH_Contains| n17 - n4 -.->|GH_Contains| n18 - n19 -.->|GH_Contains| n20 - n19 -.->|GH_Contains| n21 - n19 -.->|GH_Contains| n22 - n19 -.->|GH_Contains| n6 - n19 -.->|GH_Contains| n23 - n19 -.->|GH_Contains| n24 - n19 -.->|GH_Contains| n25 - n19 -.->|GH_Contains| n18 - n19 -.->|GH_Contains| n26 - n26 -.->|GH_Contains| n27 + n0 -.->|GH_Contains| n5 + n3 -.->|GH_Contains| n6 + n7 -.->|GH_Contains| n8 + n7 -.->|GH_Contains| n9 + n7 -.->|GH_Contains| n10 + n11 -.->|GH_Contains| n12 + n5 -.->|GH_Contains| n13 + n5 -.->|GH_Contains| n14 + n5 -.->|GH_Contains| n11 + n5 -.->|GH_Contains| n15 + n5 -.->|GH_Contains| n16 + n5 -.->|GH_Contains| n17 + n5 -.->|GH_Contains| n18 + n5 -.->|GH_Contains| n19 + n20 -.->|GH_Contains| n21 + n20 -.->|GH_Contains| n22 + n20 -.->|GH_Contains| n23 + n20 -.->|GH_Contains| n7 + n20 -.->|GH_Contains| n24 + n20 -.->|GH_Contains| n25 + n20 -.->|GH_Contains| n26 + n20 -.->|GH_Contains| n19 + n20 -.->|GH_Contains| n27 n27 -.->|GH_Contains| n28 + n28 -.->|GH_Contains| n29 ``` diff --git a/descriptions/edges/GH_HasPersonalAccessToken.md b/descriptions/edges/GH_HasPersonalAccessToken.md index dbfe0a2..ab59f67 100644 --- a/descriptions/edges/GH_HasPersonalAccessToken.md +++ b/descriptions/edges/GH_HasPersonalAccessToken.md @@ -2,12 +2,13 @@ ## General Information -The non-traversable GH_HasPersonalAccessToken edge represents the relationship between a user and their fine-grained personal access tokens that have been granted access to the organization. This edge links each approved token back to the user who created it. Fine-grained personal access tokens are security-significant because they provide programmatic access to organization resources with specific scoped permissions. Tracking token ownership is essential for understanding which users have standing API access and for identifying tokens that may need revocation. +The non-traversable GH_HasPersonalAccessToken edge links a user to a personal access token they own. Fine-grained token ownership comes from organization approvals; classic token ownership comes from the enterprise credential inventory. The edge identifies the owner but does not by itself grant access to any organization or repository. ## Edge Schema | Source | Destination | Traversable | | --- | --- | --- | +| `GH_User` | `GH_ClassicPersonalAccessToken` | `false` | | `GH_User` | `GH_PersonalAccessToken` | `false` | ## Diagram @@ -15,6 +16,8 @@ The non-traversable GH_HasPersonalAccessToken edge represents the relationship b ```mermaid graph LR n0["GH_User"] - n1["GH_PersonalAccessToken"] + n1["GH_ClassicPersonalAccessToken"] + n2["GH_PersonalAccessToken"] n0 -.->|GH_HasPersonalAccessToken| n1 + n0 -.->|GH_HasPersonalAccessToken| n2 ``` diff --git a/descriptions/nodes/GH_ClassicPersonalAccessToken.md b/descriptions/nodes/GH_ClassicPersonalAccessToken.md new file mode 100644 index 0000000..1abf1a1 --- /dev/null +++ b/descriptions/nodes/GH_ClassicPersonalAccessToken.md @@ -0,0 +1,43 @@ +# GH_ClassicPersonalAccessToken + +## General Information + +A classic personal access token found in the enterprise credential inventory. + +## Properties + +| Property | Type | Description | +| --- | --- | --- | +| `name` | `string` | The node name used for matching and display. | +| `displayname` | `string` | The human-readable display name. | +| `environmentid` | `string` | The identifier of the GitHub environment where this node was collected. | +| `last_seen` | `datetime` | The timestamp when this node was last observed during collection. | +| `node_id` | `string` | The stable identifier used as the OpenGraph node ID; this is the native GitHub node ID where available. | +| `credential_id` | `integer` | GitHub's ID for this classic personal access token. | +| `owner_id` | `integer` | GitHub's numeric ID for the token owner. | +| `owner_login` | `string` | Login of the token owner. | +| `scopes` | `list[string]` | OAuth scopes granted to the token. | +| `credential_state` | `string` | Whether GitHub reports the credential as active, expired, revoked, or deleted. | +| `expiry_status` | `string` | Whether expiration is scheduled, never, or unknown. | +| `enterprise_authorized` | `boolean` | Whether the credential is authorized directly for the enterprise. | +| `authorization_count` | `integer` | Total organization authorizations plus enterprise authorization. | +| `inventory_as_of` | `string` | Timestamp of the export snapshot used to observe the token. | +| `created_at` | `datetime` | Token creation time reported by GitHub. | +| `last_used_at` | `datetime` | Last use time reported by GitHub. | +| `expires_at` | `datetime` | Token expiration time reported by GitHub. | +| `authorized_organizations` | `list[string]` | Organizations with a reported credential authorization. | +| `environment_name` | `string` | Enterprise environment name. | +| `enterprise_name` | `string` | Enterprise from which the inventory was exported. | + +## Diagram + +```mermaid +graph LR + n0["GH_ClassicPersonalAccessToken"] + n1["GH_Organization"] + n2["GH_Enterprise"] + n3["GH_User"] + n0 -.->|GH_AuthorizedForOrganization| n1 + n2 -.->|GH_Contains| n0 + n3 -.->|GH_HasPersonalAccessToken| n0 +``` diff --git a/descriptions/nodes/GH_Enterprise.md b/descriptions/nodes/GH_Enterprise.md index f41d4e8..86f0a9c 100644 --- a/descriptions/nodes/GH_Enterprise.md +++ b/descriptions/nodes/GH_Enterprise.md @@ -34,41 +34,43 @@ A GitHub Enterprise account that contains organizations, enterprise teams, roles ```mermaid graph LR n0["GH_Enterprise"] - n1["GH_EnterpriseManagedUser"] - n2["GH_EnterpriseRole"] - n3["GH_EnterpriseRunnerGroup"] - n4["GH_EnterpriseTeam"] - n5["GH_Organization"] - n6["GH_SamlIdentityProvider"] - n7["GH_User"] - n0 -.->|GH_HasMember| n1 - n0 -.->|GH_Contains| n2 + n1["GH_ClassicPersonalAccessToken"] + n2["GH_EnterpriseManagedUser"] + n3["GH_EnterpriseRole"] + n4["GH_EnterpriseRunnerGroup"] + n5["GH_EnterpriseTeam"] + n6["GH_Organization"] + n7["GH_SamlIdentityProvider"] + n8["GH_User"] + n0 -.->|GH_Contains| n1 + n0 -.->|GH_HasMember| n2 n0 -.->|GH_Contains| n3 n0 -.->|GH_Contains| n4 n0 -.->|GH_Contains| n5 - n0 -.->|GH_HasSamlIdentityProvider| n6 - n0 -.->|GH_HasMember| n7 - n2 -.->|GH_CreateEnterpriseOrganizations| n0 - n2 -.->|GH_EditEnterpriseCustomPropertiesForOrganizations| n0 - n2 -->|GH_ManageEnterpriseAdmins| n0 - n2 -.->|GH_ManageEnterpriseIdentityProvider| n0 - n2 -->|GH_ManageEnterpriseMembers| n0 - n2 -->|GH_ManageEnterpriseOrganizationAdmins| n0 - n2 -.->|GH_ManageEnterpriseOrganizations| n0 - n2 -.->|GH_ManageEnterpriseReferrals| n0 - n2 -.->|GH_ManageEnterpriseTeams| n0 - n2 -.->|GH_ReadEnterpriseAuditLog| n0 - n2 -.->|GH_ReadEnterpriseDomainVerification| n0 - n2 -.->|GH_ReadEnterpriseMembers| n0 - n2 -.->|GH_ReadEnterpriseOrgProjects| n0 - n2 -.->|GH_ReadEnterpriseOrganizationAdmin| n0 - n2 -.->|GH_SetEnterpriseInteractionLimits| n0 - n2 -.->|GH_ViewEnterpriseActionsUsageMetrics| n0 - n2 -.->|GH_ViewEnterpriseBilling| n0 - n2 -.->|GH_ViewEnterpriseSecretScanningAlerts| n0 - n2 -.->|GH_WriteEnterpriseActionsPolicies| n0 - n2 -.->|GH_WriteEnterpriseBilling| n0 - n2 -.->|GH_WriteEnterprisePersonalAccessTokenPolicies| n0 - n2 -.->|GH_WriteEnterpriseSso| n0 - n2 -.->|GH_WriteEnterpriseTeamMembers| n0 + n0 -.->|GH_Contains| n6 + n0 -.->|GH_HasSamlIdentityProvider| n7 + n0 -.->|GH_HasMember| n8 + n3 -.->|GH_CreateEnterpriseOrganizations| n0 + n3 -.->|GH_EditEnterpriseCustomPropertiesForOrganizations| n0 + n3 -->|GH_ManageEnterpriseAdmins| n0 + n3 -.->|GH_ManageEnterpriseIdentityProvider| n0 + n3 -->|GH_ManageEnterpriseMembers| n0 + n3 -->|GH_ManageEnterpriseOrganizationAdmins| n0 + n3 -.->|GH_ManageEnterpriseOrganizations| n0 + n3 -.->|GH_ManageEnterpriseReferrals| n0 + n3 -.->|GH_ManageEnterpriseTeams| n0 + n3 -.->|GH_ReadEnterpriseAuditLog| n0 + n3 -.->|GH_ReadEnterpriseDomainVerification| n0 + n3 -.->|GH_ReadEnterpriseMembers| n0 + n3 -.->|GH_ReadEnterpriseOrgProjects| n0 + n3 -.->|GH_ReadEnterpriseOrganizationAdmin| n0 + n3 -.->|GH_SetEnterpriseInteractionLimits| n0 + n3 -.->|GH_ViewEnterpriseActionsUsageMetrics| n0 + n3 -.->|GH_ViewEnterpriseBilling| n0 + n3 -.->|GH_ViewEnterpriseSecretScanningAlerts| n0 + n3 -.->|GH_WriteEnterpriseActionsPolicies| n0 + n3 -.->|GH_WriteEnterpriseBilling| n0 + n3 -.->|GH_WriteEnterprisePersonalAccessTokenPolicies| n0 + n3 -.->|GH_WriteEnterpriseSso| n0 + n3 -.->|GH_WriteEnterpriseTeamMembers| n0 ``` diff --git a/descriptions/nodes/GH_Organization.md b/descriptions/nodes/GH_Organization.md index c429333..372b9e3 100644 --- a/descriptions/nodes/GH_Organization.md +++ b/descriptions/nodes/GH_Organization.md @@ -90,40 +90,42 @@ Represents a GitHub organization. This is the root node of the graph and serves ```mermaid graph LR - n0["GH_Enterprise"] + n0["GH_ClassicPersonalAccessToken"] n1["GH_Organization"] - n2["GH_EnterpriseTeam"] - n3["GH_OrgRole"] - n4["GH_AppInstallation"] - n5["GH_OrgRunnerGroup"] - n6["GH_OrgSecret"] - n7["GH_OrgVariable"] - n8["GH_PersonalAccessToken"] - n9["GH_PersonalAccessTokenRequest"] - n10["GH_Repository"] - n11["GH_SamlIdentityProvider"] - n12["GH_SecretScanningAlert"] - n0 -.->|GH_Contains| n1 - n2 -.->|GH_AssignedTo| n1 - n3 -.->|GH_AddCollaborator| n1 - n3 -.->|GH_CanCreateInternalRepositories| n1 - n3 -.->|GH_CanCreatePrivateRepositories| n1 - n3 -.->|GH_CanCreatePublicRepositories| n1 - n3 -.->|GH_CanCreateRepositories| n1 - n3 -.->|GH_CreateTeam| n1 - n3 -.->|GH_InviteMember| n1 - n3 -.->|GH_ResolveSecretScanningAlerts| n1 - n3 -.->|GH_TransferRepository| n1 - n3 -.->|GH_ViewSecretScanningAlerts| n1 - n1 -.->|GH_Contains| n4 - n1 -.->|GH_Contains| n3 + n2["GH_Enterprise"] + n3["GH_EnterpriseTeam"] + n4["GH_OrgRole"] + n5["GH_AppInstallation"] + n6["GH_OrgRunnerGroup"] + n7["GH_OrgSecret"] + n8["GH_OrgVariable"] + n9["GH_PersonalAccessToken"] + n10["GH_PersonalAccessTokenRequest"] + n11["GH_Repository"] + n12["GH_SamlIdentityProvider"] + n13["GH_SecretScanningAlert"] + n0 -.->|GH_AuthorizedForOrganization| n1 + n2 -.->|GH_Contains| n1 + n3 -.->|GH_AssignedTo| n1 + n4 -.->|GH_AddCollaborator| n1 + n4 -.->|GH_CanCreateInternalRepositories| n1 + n4 -.->|GH_CanCreatePrivateRepositories| n1 + n4 -.->|GH_CanCreatePublicRepositories| n1 + n4 -.->|GH_CanCreateRepositories| n1 + n4 -.->|GH_CreateTeam| n1 + n4 -.->|GH_InviteMember| n1 + n4 -.->|GH_ResolveSecretScanningAlerts| n1 + n4 -.->|GH_TransferRepository| n1 + n4 -.->|GH_ViewSecretScanningAlerts| n1 n1 -.->|GH_Contains| n5 + n1 -.->|GH_Contains| n4 n1 -.->|GH_Contains| n6 n1 -.->|GH_Contains| n7 n1 -.->|GH_Contains| n8 n1 -.->|GH_Contains| n9 - n1 -->|GH_Owns| n10 - n1 -.->|GH_HasSamlIdentityProvider| n11 - n1 -.->|GH_Contains| n12 - n8 -.->|GH_CanAccess| n1 + n1 -.->|GH_Contains| n10 + n1 -->|GH_Owns| n11 + n1 -.->|GH_HasSamlIdentityProvider| n12 + n1 -.->|GH_Contains| n13 + n9 -.->|GH_CanAccess| n1 ``` diff --git a/descriptions/nodes/GH_User.md b/descriptions/nodes/GH_User.md index ba3002c..02de469 100644 --- a/descriptions/nodes/GH_User.md +++ b/descriptions/nodes/GH_User.md @@ -36,14 +36,15 @@ graph LR n4["GH_SecretScanningAlert"] n5["GH_Branch"] n6["GH_BranchProtectionRule"] - n7["GH_DeployKey"] - n8["GH_EnterpriseRole"] - n9["GH_Environment"] - n10["GH_OrgRole"] - n11["GH_PersonalAccessToken"] - n12["GH_PersonalAccessTokenRequest"] - n13["GH_RepoRole"] - n14["GH_TeamRole"] + n7["GH_ClassicPersonalAccessToken"] + n8["GH_DeployKey"] + n9["GH_EnterpriseRole"] + n10["GH_Environment"] + n11["GH_OrgRole"] + n12["GH_PersonalAccessToken"] + n13["GH_PersonalAccessTokenRequest"] + n14["GH_RepoRole"] + n15["GH_TeamRole"] n0 -.->|GH_HasMember| n1 n2 -.->|GH_MapsToUser| n1 n3 -.->|GH_MapsToUser| n1 @@ -51,13 +52,14 @@ graph LR n1 -.->|GH_CanWriteBranch| n5 n1 -.->|GH_BypassPullRequestAllowances| n6 n1 -.->|GH_RestrictionsCanPush| n6 - n1 -.->|GH_AddedDeployKey| n7 - n1 -->|GH_HasRole| n8 - n1 -.->|GH_ApprovesDeploymentTo| n9 - n1 -->|GH_CanDeployToEnvironment| n9 - n1 -->|GH_HasRole| n10 - n1 -.->|GH_HasPersonalAccessToken| n11 - n1 -.->|GH_HasPersonalAccessTokenRequest| n12 - n1 -->|GH_HasRole| n13 + n1 -.->|GH_HasPersonalAccessToken| n7 + n1 -.->|GH_AddedDeployKey| n8 + n1 -->|GH_HasRole| n9 + n1 -.->|GH_ApprovesDeploymentTo| n10 + n1 -->|GH_CanDeployToEnvironment| n10 + n1 -->|GH_HasRole| n11 + n1 -.->|GH_HasPersonalAccessToken| n12 + n1 -.->|GH_HasPersonalAccessTokenRequest| n13 n1 -->|GH_HasRole| n14 + n1 -->|GH_HasRole| n15 ``` diff --git a/extension/saved_searches/README.md b/extension/saved_searches/README.md index 27e887a..4e3d10b 100644 --- a/extension/saved_searches/README.md +++ b/extension/saved_searches/README.md @@ -40,7 +40,7 @@ Use the runner interception searches as a progression: | 1 | `active-leaked-secrets.json` | Active Leaked Secrets | Finds secret scanning alerts that are both unresolved and confirmed active. These are valid, usable credentials committed to source code and represent an immediate compromise risk. | | 2 | `secret-scanning-alerts.json` | Secret Scanning Alerts | Returns all repositories that have open secret scanning alerts. | | 3 | `pats-all-repo-access.json` | PATs with Access to All Repositories | Finds fine-grained personal access tokens scoped to all repositories. A single compromised token grants access to every repository in the organization. | -| 4 | `expired-pats.json` | Expired Personal Access Tokens | Finds expired personal access tokens that still exist. Expired tokens should be cleaned up to reduce credential inventory and audit noise. | +| 4 | `expired-pats.json` | Expired Personal Access Tokens | Finds expired fine-grained and classic personal access tokens that still exist. Expired tokens should be cleaned up to reduce credential inventory and audit noise. | | 5 | `pending-pat-requests.json` | Pending PAT Requests | Finds pending fine-grained personal access token requests awaiting approval. Review these to ensure requested permissions are appropriate before granting access. | ### :orange_circle: High — Organization Security Posture diff --git a/extension/saved_searches/expired-pats.json b/extension/saved_searches/expired-pats.json index e849144..518e793 100644 --- a/extension/saved_searches/expired-pats.json +++ b/extension/saved_searches/expired-pats.json @@ -1,5 +1,5 @@ { "name": "GitHub: Expired Personal Access Tokens", - "query": "MATCH p=(:GH_User)-[:GH_HasPersonalAccessToken]->(token:GH_PersonalAccessToken {token_expired: true})\nRETURN p\nLIMIT 1000", - "description": "Finds expired personal access tokens that still exist. Expired tokens should be cleaned up to reduce credential inventory and audit noise." + "query": "MATCH p=(:GH_User)-[:GH_HasPersonalAccessToken]->(token)\nWHERE (token:GH_PersonalAccessToken AND token.token_expired = true) OR (token:GH_ClassicPersonalAccessToken AND token.credential_state = 'expired')\nRETURN p\nLIMIT 1000", + "description": "Finds expired fine-grained and classic personal access tokens that still exist. Expired tokens should be cleaned up to reduce credential inventory and audit noise." } diff --git a/extension/schema.json b/extension/schema.json index 1569478..9c75357 100644 --- a/extension/schema.json +++ b/extension/schema.json @@ -76,6 +76,13 @@ "markdown": { "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (selected:GH_Enterprise {objectid: '%s'})-[:GH_HasSamlIdentityProvider]->(idp:GH_SamlIdentityProvider)\\nOPTIONAL MATCH p1 = (idp)-[:GH_HasExternalIdentity]->(:GH_ExternalIdentity)-[:GH_MapsToUser]->(:GH_User)\\nRETURN p, p1\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View the identity provider and mapped users for this enterprise in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" } + }, + "classic_personal_access_tokens": { + "title": "Classic Personal Access Tokens", + "position": 10, + "markdown": { + "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (selected:GH_Enterprise {objectid: '%s'})-[:GH_Contains]->(:GH_ClassicPersonalAccessToken)\\nRETURN p\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View classic personal access tokens in this enterprise in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" + } } } }, @@ -305,6 +312,13 @@ "markdown": { "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (selected:GH_Organization {objectid: '%s'})-[:GH_HasSamlIdentityProvider]->(idp:GH_SamlIdentityProvider)\\nOPTIONAL MATCH p1 = (idp)-[:GH_HasExternalIdentity]->(:GH_ExternalIdentity)-[:GH_MapsToUser]->(:GH_User)\\nRETURN p, p1\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View the identity provider and mapped users for this organization in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" } + }, + "classic_pat_authorizations": { + "title": "Classic PAT Authorizations", + "position": 9, + "markdown": { + "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (:GH_ClassicPersonalAccessToken)-[:GH_AuthorizedForOrganization]->(selected:GH_Organization {objectid: '%s'})\\nRETURN p\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View classic PATs authorized for this organization in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" + } } } }, @@ -1759,6 +1773,51 @@ } } }, + { + "name": "GH_ClassicPersonalAccessToken", + "display_name": "GitHub Classic Personal Access Token", + "description": "A classic personal access token found in the enterprise credential inventory", + "is_display_kind": true, + "icon": "key", + "color": "#F5A623", + "info": { + "description": { + "title": "Description", + "position": 1, + "markdown": { + "content": "## Overview\n\nRepresents a classic personal access token reported by the enterprise credential inventory. Organization edges record authorizations, while token scopes and owner permissions determine what the credential can do. Repository access is not enumerated by the inventory." + } + }, + "inbound_traversable_relationships": { + "title": "Inbound Traversable GH Relationships", + "position": 2, + "markdown": { + "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (selected:GH_ClassicPersonalAccessToken {objectid: '%s'})\\nWHERE false\\nRETURN p\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View direct inbound relationships in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" + } + }, + "outbound_traversable_relationships": { + "title": "Outbound Traversable GH Relationships", + "position": 3, + "markdown": { + "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (selected:GH_ClassicPersonalAccessToken {objectid: '%s'})\\nWHERE false\\nRETURN p\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View direct outbound relationships in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" + } + }, + "owner": { + "title": "Owner", + "position": 4, + "markdown": { + "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (:GH_User)-[:GH_HasPersonalAccessToken]->(selected:GH_ClassicPersonalAccessToken {objectid: '%s'})\\nRETURN p\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View the owner of this personal access token in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" + } + }, + "organization_scope": { + "title": "Organization Scope", + "position": 5, + "markdown": { + "content": "{{ $objectID := .Properties.objectid | default \"\" | trim }}\n{{ $escapedObjectID := $objectID | replace \"\\\\\" \"\\\\\\\\\" | replace \"'\" \"\\\\'\" }}\n{{ if ne $objectID \"\" }}\n{{ $query := printf \"MATCH p = (selected:GH_ClassicPersonalAccessToken {objectid: '%s'})-[:GH_AuthorizedForOrganization]->(:GH_Organization)\\nRETURN p\\nLIMIT 500\" $escapedObjectID }}\n{{ $encodedQuery := $query | b64enc | urlquery }}\n[View organizations authorizing this token in Explore](/ui/explore?exploreSearchTab=cypher&searchType=cypher&cypherSearch={{ $encodedQuery }})\n{{ else }}\nExplore navigation is unavailable because this object has no stable identifier.\n{{ end }}" + } + } + } + }, { "name": "GH_PersonalAccessToken", "display_name": "GitHub Personal Access Token", @@ -2490,7 +2549,7 @@ }, { "name": "GH_HasPersonalAccessToken", - "description": "User owns this personal access token that has been granted access to the organization", + "description": "User owns a personal access token reported by GitHub", "is_traversable": false }, { @@ -2503,6 +2562,11 @@ "description": "GitHub App is installed as this app installation on an organization", "is_traversable": true }, + { + "name": "GH_AuthorizedForOrganization", + "description": "Classic personal access token has a recorded authorization for this organization", + "is_traversable": false + }, { "name": "GH_CanAccess", "description": "Personal access token, app installation, or deploy key can access this repository or organization", diff --git a/src/openhound_github/helpers.py b/src/openhound_github/helpers.py index 2bc6a3d..919f621 100644 --- a/src/openhound_github/helpers.py +++ b/src/openhound_github/helpers.py @@ -390,6 +390,21 @@ def _is_graphql_response(response: requests.Response) -> bool: return urlparse(request.url).path.rstrip("/").endswith("/graphql") +def _response_request_label(response: requests.Response) -> str: + request = response.request + if request is None or not request.url: + return "unknown request" + return f"{request.method or 'UNKNOWN'} {urlparse(request.url).path}" + + +def _is_credential_export_creation_response(response: requests.Response) -> bool: + request = response.request + if request is None or request.method != "POST" or not request.url: + return False + path = urlparse(request.url).path.rstrip("/") + return "/enterprises/" in path and path.endswith("/credentials/exports") + + def github_retry_policy(auth: AuthConfigBase): def retry_policy( response: Optional[requests.Response], exception: Optional[BaseException] @@ -435,7 +450,9 @@ def retry_policy( delay = int(reset_at) - now if reset_at else 0 headers["Retry-After"] = str(delay) logger.warning( - "Primary rate limit reached, retrying in %s seconds", delay + "Primary rate limit reached for %s, retrying in %s seconds", + _response_request_label(response), + delay, ) return True return False @@ -443,15 +460,34 @@ def retry_policy( if response.status_code not in (403, 429): return False + if _is_credential_export_creation_response(response) and ( + response.status_code == 429 + or bool(headers.get("Retry-After")) + or is_primary_rate_limit_response(response) + or is_secondary_rate_limit_response(response) + ): + logger.warning( + "Rate limit reached for %s; skipping export creation retry", + _response_request_label(response), + ) + return False + if is_primary_rate_limit_response(response): reset_at = headers.get("x-ratelimit-reset") delay = int(reset_at) - now if reset_at else 0 headers["Retry-After"] = str(delay) - logger.warning("Primary rate limit reached, retrying in %s seconds", delay) + logger.warning( + "Primary rate limit reached for %s, retrying in %s seconds", + _response_request_label(response), + delay, + ) return True if is_secondary_rate_limit_response(response): - logger.warning("Secondary rate limit reached, retrying in 60 seconds") + logger.warning( + "Secondary rate limit reached for %s, retrying in 60 seconds", + _response_request_label(response), + ) headers["Retry-After"] = "60" return True diff --git a/src/openhound_github/kinds/edges.py b/src/openhound_github/kinds/edges.py index bcbff94..8baa260 100644 --- a/src/openhound_github/kinds/edges.py +++ b/src/openhound_github/kinds/edges.py @@ -17,6 +17,7 @@ # Access and capability edges CAN_ACCESS = "GH_CanAccess" +AUTHORIZED_FOR_ORGANIZATION = "GH_AuthorizedForOrganization" CAN_USE_RUNNER = "GH_CanUseRunner" RUNS_ON = "GH_RunsOn" CAN_INTERCEPT_JOB = "GH_CanInterceptJob" diff --git a/src/openhound_github/kinds/nodes.py b/src/openhound_github/kinds/nodes.py index beddeab..23e1820 100644 --- a/src/openhound_github/kinds/nodes.py +++ b/src/openhound_github/kinds/nodes.py @@ -24,6 +24,7 @@ ORG_VARIABLE = "GH_OrgVariable" # Personal access token nodes +CLASSIC_PERSONAL_ACCESS_TOKEN = "GH_ClassicPersonalAccessToken" PERSONAL_ACCESS_TOKEN = "GH_PersonalAccessToken" PERSONAL_ACCESS_TOKEN_REQUEST = "GH_PersonalAccessTokenRequest" diff --git a/src/openhound_github/lookup.py b/src/openhound_github/lookup.py index 6d8ec6b..c1a5129 100644 --- a/src/openhound_github/lookup.py +++ b/src/openhound_github/lookup.py @@ -95,6 +95,20 @@ def enterprise_id(self) -> str | None: res = self._find_single_object(f"""SELECT id FROM {self.schema}.enterprise""") return res + @lru_cache + def enterprise_user_id_for_database_id(self, database_id: int) -> str | None: + return self._find_single_object( + f"SELECT id FROM {self.schema}.enterprise_users WHERE database_id = ? LIMIT 1", + [database_id], + ) + + @lru_cache + def enterprise_organization_id_for_login(self, login: str) -> str | None: + return self._find_single_object( + f"SELECT id FROM {self.schema}.enterprise_organizations WHERE lower(login) = lower(?) LIMIT 1", + [login], + ) + @lru_cache def enterprise_organization_node_ids(self, enterprise_node_id: str): return self._find_all_objects( diff --git a/src/openhound_github/models/__init__.py b/src/openhound_github/models/__init__.py index 38c8b6e..81f2ba9 100644 --- a/src/openhound_github/models/__init__.py +++ b/src/openhound_github/models/__init__.py @@ -4,6 +4,7 @@ from .branch_pr_bypass_allowance import BranchPrBypassAllowance from .branch_protection_rule import BranchProtectionRule, BranchProtectionRuleActor from .branch_push_allowance import BranchPushAllowance +from .classic_personal_access_token import ClassicPersonalAccessToken from .enterprise import Enterprise from .enterprise_admin import EnterpriseAdmin from .enterprise_helpers import enterprise_role_node_id, enterprise_team_node_id @@ -113,6 +114,7 @@ "App", "AppInstallationRepoAccess", "PersonalAccessToken", + "ClassicPersonalAccessToken", "PatRepoAccess", "ProjectedEnterpriseTeam", "SelectedOrgSecret", diff --git a/src/openhound_github/models/classic_personal_access_token.py b/src/openhound_github/models/classic_personal_access_token.py new file mode 100644 index 0000000..1c758ee --- /dev/null +++ b/src/openhound_github/models/classic_personal_access_token.py @@ -0,0 +1,166 @@ +from dataclasses import dataclass +from datetime import datetime +from typing import ClassVar + +from dlt.common.libs.pydantic import DltConfig +from openhound.core.asset import BaseAsset, EdgeDef, NodeDef +from openhound.core.models.entries_dataclass import Edge, EdgePath, EdgeProperties + +from openhound_github.graph import GHNode, GHNodeProperties +from openhound_github.kinds import edges as ek +from openhound_github.kinds import nodes as nk +from openhound_github.main import app + + +@dataclass +class GHClassicPersonalAccessTokenProperties(GHNodeProperties): + """Metadata from the enterprise credential inventory export. + + Attributes: + credential_id: GitHub's ID for this classic personal access token. + owner_id: GitHub's numeric ID for the token owner. + owner_login: Login of the token owner. + scopes: OAuth scopes granted to the token. + credential_state: Whether GitHub reports the credential as active, expired, revoked, or deleted. + expiry_status: Whether expiration is scheduled, never, or unknown. + enterprise_authorized: Whether the credential is authorized directly for the enterprise. + authorization_count: Total organization authorizations plus enterprise authorization. + inventory_as_of: Timestamp of the export snapshot used to observe the token. + created_at: Token creation time reported by GitHub. + last_used_at: Last use time reported by GitHub. + expires_at: Token expiration time reported by GitHub. + authorized_organizations: Organizations with a reported credential authorization. + environment_name: Enterprise environment name. + enterprise_name: Enterprise from which the inventory was exported. + """ + + credential_id: int | None = None + owner_id: int | None = None + owner_login: str | None = None + scopes: list[str] | None = None + credential_state: str | None = None + expiry_status: str | None = None + enterprise_authorized: bool | None = None + authorization_count: int | None = None + inventory_as_of: str | None = None + created_at: datetime | None = None + last_used_at: datetime | None = None + expires_at: datetime | None = None + authorized_organizations: list[str] | None = None + environment_name: str | None = None + enterprise_name: str | None = None + + +@app.asset( + node=NodeDef( + kind=nk.CLASSIC_PERSONAL_ACCESS_TOKEN, + description="GitHub classic personal access token in an enterprise credential inventory", + icon="key", + properties=GHClassicPersonalAccessTokenProperties, + ), + edges=[ + EdgeDef( + start=nk.ENTERPRISE, + end=nk.CLASSIC_PERSONAL_ACCESS_TOKEN, + kind=ek.CONTAINS, + description="Enterprise inventory contains classic PAT", + traversable=False, + ), + EdgeDef( + start=nk.USER, + end=nk.CLASSIC_PERSONAL_ACCESS_TOKEN, + kind=ek.HAS_PERSONAL_ACCESS_TOKEN, + description="User owns classic PAT", + traversable=False, + ), + EdgeDef( + start=nk.CLASSIC_PERSONAL_ACCESS_TOKEN, + end=nk.ORGANIZATION, + kind=ek.AUTHORIZED_FOR_ORGANIZATION, + description="Classic PAT has a recorded organization authorization", + traversable=False, + ), + ], +) +class ClassicPersonalAccessToken(BaseAsset): + """One deduplicated classic PAT from an enterprise credential inventory export.""" + + dlt_config: ClassVar[DltConfig] = {"return_validated_models": True} + + credential_id: int + display_name: str | None = None + owner_id: int | None = None + owner_login: str | None = None + scopes: list[str] | None = None + credential_state: str | None = None + expiry_status: str | None = None + enterprise_authorized: bool | None = None + authorization_count: int | None = None + inventory_as_of: str | None = None + created_at: datetime | None = None + last_used_at: datetime | None = None + expires_at: datetime | None = None + authorized_organizations: list[str] + enterprise_node_id: str + enterprise_name: str + + @property + def node_id(self) -> str: + return f"GH_CLASSIC_PAT_{self.enterprise_node_id}_{self.credential_id}" + + @property + def as_node(self) -> GHNode: + name = self.display_name or f"Classic PAT {self.credential_id}" + return GHNode( + kinds=[nk.CLASSIC_PERSONAL_ACCESS_TOKEN], + properties=GHClassicPersonalAccessTokenProperties( + name=name, + displayname=name, + node_id=self.node_id, + environmentid=self.enterprise_node_id, + environment_name=self.enterprise_name, + credential_id=self.credential_id, + owner_id=self.owner_id, + owner_login=self.owner_login, + scopes=self.scopes, + credential_state=self.credential_state, + expiry_status=self.expiry_status, + enterprise_authorized=self.enterprise_authorized, + authorization_count=self.authorization_count, + inventory_as_of=self.inventory_as_of, + created_at=self.created_at, + last_used_at=self.last_used_at, + expires_at=self.expires_at, + authorized_organizations=self.authorized_organizations, + enterprise_name=self.enterprise_name, + ), + ) + + @property + def edges(self): + yield Edge( + kind=ek.CONTAINS, + start=EdgePath(value=self.enterprise_node_id, match_by="id"), + end=EdgePath(value=self.node_id, match_by="id"), + properties=EdgeProperties(traversable=False), + ) + if self.owner_id is not None: + owner_node_id = self._lookup.enterprise_user_id_for_database_id( + self.owner_id + ) + if owner_node_id: + yield Edge( + kind=ek.HAS_PERSONAL_ACCESS_TOKEN, + start=EdgePath(value=owner_node_id, match_by="id"), + end=EdgePath(value=self.node_id, match_by="id"), + properties=EdgeProperties(traversable=False), + ) + for login in self.authorized_organizations: + org_node_id = self._lookup.enterprise_organization_id_for_login(login) + if org_node_id: + yield Edge( + kind=ek.AUTHORIZED_FOR_ORGANIZATION, + start=EdgePath(value=self.node_id, match_by="id"), + end=EdgePath(value=org_node_id, match_by="id"), + properties=EdgeProperties(traversable=False), + ) diff --git a/src/openhound_github/models/enterprise_member.py b/src/openhound_github/models/enterprise_member.py index 08c2c18..d21254e 100644 --- a/src/openhound_github/models/enterprise_member.py +++ b/src/openhound_github/models/enterprise_member.py @@ -19,6 +19,7 @@ class BaseUser(BaseModel): typename: str = Field(alias="__typename") id: str + database_id: int | None = Field(alias="databaseId", default=None) login: str name: str | None = None url: str | None = None diff --git a/src/openhound_github/models/org.py b/src/openhound_github/models/org.py index 10039a3..f244d01 100644 --- a/src/openhound_github/models/org.py +++ b/src/openhound_github/models/org.py @@ -316,7 +316,12 @@ def as_node(self) -> GHNode: query_users=f"MATCH (n:GH_User {{environmentid:'{oid}'}}) RETURN n", query_teams=f"MATCH (n:GH_Team {{environmentid:'{oid}'}}) RETURN n", query_repositories=f"MATCH (n:GH_Repository {{environmentid:'{oid}'}}) RETURN n", - query_personal_access_tokens=f"MATCH p=(:GH_Organization {{node_id: '{oid}'}})-[:GH_Contains]->(token) WHERE token:GH_PersonalAccessToken OR token:GH_PersonalAccessTokenRequest RETURN p", + query_personal_access_tokens=( + f"MATCH p=(:GH_Organization {{node_id: '{oid}'}})-[:GH_Contains]->(token) " + "WHERE token:GH_PersonalAccessToken OR token:GH_PersonalAccessTokenRequest RETURN p " + f"UNION MATCH p=(:GH_ClassicPersonalAccessToken)-[:GH_AuthorizedForOrganization]->" + f"(:GH_Organization {{node_id: '{oid}'}}) RETURN p" + ), query_secret_scanning_alerts=f"MATCH p=(:GH_Organization {{node_id: '{oid}'}})-[:GH_Contains]->(alert:GH_SecretScanningAlert) RETURN p", query_identity_provider=f"MATCH p=(OIP:GH_SamlIdentityProvider)-[:GH_HasExternalIdentity]->(EI:GH_ExternalIdentity) MATCH p1=(OIP)<-[:GH_HasSamlIdentityProvider]-(:GH_Organization {{node_id:'{oid}'}}) MATCH p2=(EI)-[:GH_MapsToUser]->() RETURN p,p1,p2", query_app_installations=f"MATCH p=(:GH_Organization {{node_id:'{oid}'}})-[:GH_Contains]->(:GH_AppInstallation) RETURN p", diff --git a/src/openhound_github/models/user.py b/src/openhound_github/models/user.py index 76186eb..763061f 100644 --- a/src/openhound_github/models/user.py +++ b/src/openhound_github/models/user.py @@ -108,7 +108,8 @@ def as_node(self) -> GHNode: environmentid=org_node_id, query_personal_access_tokens=( f"MATCH p=(:GH_User {{node_id: '{uid}'}})-[]->(token) " - f"WHERE token:GH_PersonalAccessToken OR token:GH_PersonalAccessTokenRequest RETURN p" + f"WHERE token:GH_PersonalAccessToken OR token:GH_ClassicPersonalAccessToken " + f"OR token:GH_PersonalAccessTokenRequest RETURN p" ), query_roles=( f"MATCH p=(t:GH_User {{node_id:'{uid}'}})" diff --git a/src/openhound_github/resources/enterprise.py b/src/openhound_github/resources/enterprise.py index 8951967..44ef1d5 100644 --- a/src/openhound_github/resources/enterprise.py +++ b/src/openhound_github/resources/enterprise.py @@ -1,7 +1,16 @@ +import csv +import io import logging +import tempfile +import time from dataclasses import dataclass, field from typing import Any +from urllib.parse import urlparse +import dlt +import requests + +from dlt.common.exceptions import PipelineStateNotAvailable, SourceSectionNotAvailable from dlt.sources.helpers.rest_client.client import RESTClient from dlt.sources.helpers.rest_client.paginators import OffsetPaginator @@ -18,9 +27,14 @@ graphql_client_and_path, scim_skip_reason, ) +from openhound_github.github_retry import ( + is_primary_rate_limit_response, + is_secondary_rate_limit_response, +) from openhound_github.main import app from openhound_github.models import ( BaseUser, + ClassicPersonalAccessToken, Enterprise, EnterpriseAdmin, EnterpriseManagedUser, @@ -53,6 +67,11 @@ logger = logging.getLogger(__name__) +_CREDENTIAL_EXPORT_API_VERSION = "2026-03-10" +_CREDENTIAL_EXPORT_POLL_SECONDS = 5 +_CREDENTIAL_EXPORT_TIMEOUT_SECONDS = 300 +_CREDENTIAL_EXPORT_REUSE_SECONDS = 24 * 60 * 60 + @dataclass class SourceContext: @@ -106,7 +125,9 @@ def iter_enterprise_scim_resources( yield from page -def _log_enterprise_scim_failure(resource: str, enterprise_name: str, exception: BaseException): +def _log_enterprise_scim_failure( + resource: str, enterprise_name: str, exception: BaseException +): skip_reason = scim_skip_reason(exception) if skip_reason: logger.warning( @@ -124,6 +145,327 @@ def _log_enterprise_scim_failure(resource: str, enterprise_name: str, exception: ) +def _download_enterprise_credential_inventory( + client: RESTClient, + enterprise_slug: str, + export_id: str | None = None, + fallback_client: RESTClient | None = None, +): + """Export all credentials and return parsed CSV values.""" + path = f"/enterprises/{enterprise_slug}/credentials/exports" + headers = {"X-GitHub-Api-Version": _CREDENTIAL_EXPORT_API_VERSION} + export_metadata = {} + created_with_fallback = False + if export_id is None: + try: + created = client.post(path, headers=headers) + created.raise_for_status() + except requests.HTTPError as exc: + response = exc.response + status_code = response.status_code if response is not None else None + retry_after = ( + response.headers.get("Retry-After") if response is not None else None + ) + remaining = ( + response.headers.get("X-RateLimit-Remaining") + if response is not None + else None + ) + try: + message = ( + str(response.json().get("message", "")).lower() + if response is not None + else "" + ) + except ValueError: + message = "" + if ( + fallback_client is None + or status_code not in {401, 403} + or retry_after + or remaining == "0" + or "rate limit" in message + or "abuse" in message + ): + raise + logger.warning( + "Enterprise app could not create credential export for '%s' (HTTP %s); retrying with configured classic PAT", + enterprise_slug, + status_code, + ) + client = fallback_client + created = client.post(path, headers=headers) + created.raise_for_status() + created_with_fallback = True + if created.status_code != 202: + raise ValueError( + f"Unexpected credential export create status: {created.status_code}" + ) + export_metadata = created.json() + export_id = export_metadata.get("export_id") + if not export_id or not isinstance(export_id, str): + raise ValueError("Credential export response did not include an export_id") + + deadline = time.monotonic() + _CREDENTIAL_EXPORT_TIMEOUT_SECONDS + export_path = f"{path}/{export_id}" + while True: + response = client.get(export_path, headers=headers, allow_redirects=False) + response.raise_for_status() + if response.status_code == 302: + download_url = response.headers.get("Location") + parsed_url = urlparse(download_url or "") + if ( + parsed_url.scheme != "https" + or not parsed_url.hostname + or parsed_url.username + or parsed_url.password + ): + raise ValueError( + "Credential export returned an invalid HTTPS download URL" + ) + # The signed URL is fetched without the GitHub installation credential. + download = requests.get(download_url, timeout=120, stream=True) + try: + download.raise_for_status() + with tempfile.SpooledTemporaryFile(max_size=8 * 1024 * 1024) as buffer: + for chunk in download.iter_content(chunk_size=64 * 1024): + buffer.write(chunk) + buffer.seek(0) + with io.TextIOWrapper( + buffer, encoding="utf-8-sig", newline="" + ) as csv_stream: + reader = csv.DictReader(csv_stream) + rows = list(reader) + columns = reader.fieldnames or [] + finally: + download.close() + return { + "export_id": export_id, + "as_of": export_metadata.get("as_of"), + "created_with_fallback": created_with_fallback, + "columns": columns, + "rows": rows, + } + if response.status_code != 200: + raise ValueError( + f"Unexpected credential export status: {response.status_code}" + ) + polled_metadata = response.json() + export_metadata.update(polled_metadata) + status = polled_metadata.get("status") + if status not in {"queued", "started", "in_progress", "processing", "pending"}: + raise ValueError(f"Credential export did not complete: {status!r}") + if time.monotonic() >= deadline: + raise TimeoutError("Credential export did not complete within five minutes") + time.sleep(_CREDENTIAL_EXPORT_POLL_SECONDS) + + +@app.transformer( + name="enterprise_credential_inventory", + columns={"columns": {"data_type": "json"}, "rows": {"data_type": "json"}}, + parallelized=True, +) +def enterprise_credential_inventory(enterprise_data: Enterprise, ctx: SourceContext): + """Persist the full enterprise credential inventory for later parsing.""" + if ctx.deployment_type == "ghes" or not ctx.client or not ctx.enterprise_name: + return + + try: + state = dlt.current.resource_state("enterprise_credential_inventory") + except (PipelineStateNotAvailable, SourceSectionNotAvailable): + state = {} + + downloaded_at = state.get("last_export_downloaded_at") + previous_id = ( + state.get("last_export_id") + if state.get("last_export_enterprise") == ctx.enterprise_name + and isinstance(downloaded_at, (int, float)) + and 0 <= time.time() - downloaded_at <= _CREDENTIAL_EXPORT_REUSE_SECONDS + else None + ) + if previous_id: + previous_used_fallback = state.get("last_export_created_with_fallback", False) + previous_client = ctx.sso_client if previous_used_fallback else ctx.client + if previous_client: + try: + inventory = _download_enterprise_credential_inventory( + previous_client, ctx.enterprise_name, previous_id + ) + inventory["created_with_fallback"] = previous_used_fallback + inventory["as_of"] = inventory["as_of"] or state.get( + "last_export_as_of" + ) + logger.info( + "Using prior credential export %s for enterprise '%s' (as of %s)", + previous_id, + ctx.enterprise_name, + inventory["as_of"], + ) + yield { + **inventory, + "enterprise_node_id": enterprise_data.id, + "enterprise_name": ctx.enterprise_name, + } + return + except requests.HTTPError as exc: + if exc.response is None or exc.response.status_code != 404: + logger.warning( + "Skipping credential inventory for enterprise '%s': prior export %s unavailable (%s)", + ctx.enterprise_name, + previous_id, + exc, + ) + return + logger.info( + "Prior credential export %s is no longer available for enterprise '%s'; creating a new export", + previous_id, + ctx.enterprise_name, + ) + except ( + requests.RequestException, + ValueError, + TimeoutError, + UnicodeError, + csv.Error, + ) as exc: + logger.warning( + "Skipping credential inventory for enterprise '%s': prior export %s unavailable (%s)", + ctx.enterprise_name, + previous_id, + exc, + ) + return + else: + logger.info( + "No recent credential export recorded for enterprise '%s'; creating a new export", + ctx.enterprise_name, + ) + + try: + inventory = _download_enterprise_credential_inventory( + ctx.client, + ctx.enterprise_name, + fallback_client=( + ctx.sso_client if ctx.sso_client is not ctx.client else None + ), + ) + state["last_export_id"] = inventory["export_id"] + state["last_export_enterprise"] = ctx.enterprise_name + state["last_export_as_of"] = inventory["as_of"] + state["last_export_created_with_fallback"] = inventory.get( + "created_with_fallback", False + ) + state["last_export_downloaded_at"] = time.time() + yield { + **inventory, + "enterprise_node_id": enterprise_data.id, + "enterprise_name": ctx.enterprise_name, + } + except requests.HTTPError as exc: + response = exc.response + status_code = response.status_code if response is not None else None + if response is not None and is_secondary_rate_limit_response(response): + logger.warning( + "Skipping credential inventory for enterprise '%s': secondary rate limit reached", + ctx.enterprise_name, + ) + elif response is not None and is_primary_rate_limit_response(response): + logger.warning( + "Skipping credential inventory for enterprise '%s': primary rate limit reached", + ctx.enterprise_name, + ) + elif status_code == 429: + logger.warning( + "Skipping credential inventory for enterprise '%s': export creation rejected (HTTP 429)", + ctx.enterprise_name, + ) + else: + logger.warning( + "Skipping credential inventory for enterprise '%s': HTTP %s", + ctx.enterprise_name, + status_code, + ) + except ( + requests.RequestException, + ValueError, + TimeoutError, + UnicodeError, + csv.Error, + ) as exc: + logger.warning( + "Skipping credential inventory for enterprise '%s': %s", + ctx.enterprise_name, + exc, + ) + + +@app.transformer( + name="classic_personal_access_tokens", + columns=ClassicPersonalAccessToken, + parallelized=True, +) +def classic_personal_access_tokens(inventory: dict[str, Any]): + """Build classic PAT models from a stored enterprise credential inventory.""" + credentials: dict[int, dict[str, Any]] = {} + for row_number, row in enumerate(inventory["rows"], start=1): + if row.get("credential_type") != "classic_pat": + continue + credential_id = row.get("credential_id") + if not credential_id: + logger.warning( + "Skipping classic PAT inventory row %d without a credential_id", + row_number, + ) + continue + try: + token_id = int(credential_id) + owner_id = int(row["owner_id"]) if row.get("owner_id") else None + authorization_count = ( + int(row["authorization_count"]) + if row.get("authorization_count") + else None + ) + except (TypeError, ValueError, OverflowError): + logger.warning( + "Skipping classic PAT inventory row %d with an invalid credential_id, owner_id, or authorization_count", + row_number, + ) + continue + credential = credentials.get(token_id) + if credential is None: + credential = { + "credential_id": token_id, + "display_name": row.get("display_name") or None, + "owner_id": owner_id, + "owner_login": row.get("owner") or None, + "scopes": [ + scope.strip() + for scope in (row.get("scopes") or "").split(";") + if scope.strip() + ], + "credential_state": row.get("credential_state") or None, + "expiry_status": row.get("expiry_status") or None, + "enterprise_authorized": ( + row["enterprise_authorized"].lower() == "true" + if row.get("enterprise_authorized") + else None + ), + "authorization_count": authorization_count, + "inventory_as_of": inventory.get("as_of"), + "created_at": row.get("created_at") or None, + "last_used_at": row.get("last_used_at") or None, + "expires_at": row.get("expires_at") or None, + "authorized_organizations": [], + "enterprise_node_id": inventory["enterprise_node_id"], + "enterprise_name": inventory["enterprise_name"], + } + credentials[token_id] = credential + org_login = row.get("organization") + if org_login and org_login not in credential["authorized_organizations"]: + credential["authorized_organizations"].append(org_login) + yield from credentials.values() + + @app.resource(name="enterprise", columns=Enterprise, parallelized=True) def enterprise(ctx: SourceContext): data = { @@ -185,7 +527,10 @@ def enterprise_organizations(enterprise_data: Enterprise, ctx: SourceContext): except Exception as e: logger.error( f"Error in resource 'enterprise_organizations' processing enterprise '{ctx.enterprise_name}': {e}", - extra={"resource": "enterprise_organizations", "phase": "resource_iteration"}, + extra={ + "resource": "enterprise_organizations", + "phase": "resource_iteration", + }, ) return @@ -197,7 +542,9 @@ def enterprise_organizations(enterprise_data: Enterprise, ctx: SourceContext): ) def enterprise_scim_organizations(enterprise_data: Enterprise, ctx: SourceContext): if not ctx.client or not ctx.enterprise_name: - raise ValueError("Enterprise SCIM collection requires a client and enterprise slug") + raise ValueError( + "Enterprise SCIM collection requires a client and enterprise slug" + ) try: next( @@ -232,7 +579,9 @@ def enterprise_scim_users( scim_organization: EnterpriseScimOrganization, ctx: SourceContext ): if not ctx.client or not ctx.enterprise_name: - raise ValueError("Enterprise SCIM collection requires a client and enterprise slug") + raise ValueError( + "Enterprise SCIM collection requires a client and enterprise slug" + ) try: for user in iter_enterprise_scim_resources( ctx.client, @@ -259,7 +608,9 @@ def enterprise_scim_groups( scim_organization: EnterpriseScimOrganization, ctx: SourceContext ): if not ctx.client or not ctx.enterprise_name: - raise ValueError("Enterprise SCIM collection requires a client and enterprise slug") + raise ValueError( + "Enterprise SCIM collection requires a client and enterprise slug" + ) try: for group in iter_enterprise_scim_resources( ctx.client, @@ -389,7 +740,10 @@ def enterprise_runner_groups(enterprise_data: Enterprise, ctx: SourceContext): except Exception as e: logger.error( f"Error in resource 'enterprise_runner_groups' processing enterprise '{ctx.enterprise_name}': {e}", - extra={"resource": "enterprise_runner_groups", "phase": "resource_iteration"}, + extra={ + "resource": "enterprise_runner_groups", + "phase": "resource_iteration", + }, ) return @@ -727,7 +1081,9 @@ def _enterprise_admins_from_owner_info( es_data = enterprise_object.get("enterprise", {}) owner_info = es_data.get("ownerInfo") or {} for edge in (owner_info.get("admins") or {}).get("edges") or []: - row = _enterprise_admin_row(edge.get("node") or {}, enterprise_data, ctx) + row = _enterprise_admin_row( + edge.get("node") or {}, enterprise_data, ctx + ) if row: yield row @@ -787,7 +1143,7 @@ def _enterprise_admins_from_organizations( name="enterprise_saml_provider", table_name="saml_provider", columns=SamlProvider, - parallelized=True + parallelized=True, ) def enterprise_saml_provider(enterprise_data: Enterprise, ctx: SourceContext): client, graphql_path = _sso_graphql_client(ctx) @@ -808,7 +1164,10 @@ def enterprise_saml_provider(enterprise_data: Enterprise, ctx: SourceContext): except Exception as e: logger.error( f"Error in resource 'enterprise_saml_provider' processing enterprise '{ctx.enterprise_name}': {e}", - extra={"resource": "enterprise_saml_provider", "phase": "resource_iteration"}, + extra={ + "resource": "enterprise_saml_provider", + "phase": "resource_iteration", + }, ) return @@ -820,7 +1179,9 @@ def enterprise_saml_provider(enterprise_data: Enterprise, ctx: SourceContext): ) return - saml_provider = (enterprise_object.get("ownerInfo") or {}).get("samlIdentityProvider") + saml_provider = (enterprise_object.get("ownerInfo") or {}).get( + "samlIdentityProvider" + ) if not saml_provider: logger.warning( "No enterprise SAML provider returned for enterprise '%s'", @@ -838,11 +1199,12 @@ def enterprise_saml_provider(enterprise_data: Enterprise, ctx: SourceContext): "github_web_origin": ctx.github_web_origin, } + @app.transformer( name="enterprise_saml_service_provider", table_name="saml_service_provider", columns=SamlServiceProvider, - parallelized=True + parallelized=True, ) def enterprise_saml_service_provider(saml_provider: SamlProvider, ctx: SourceContext): yield { @@ -856,6 +1218,7 @@ def enterprise_saml_service_provider(saml_provider: SamlProvider, ctx: SourceCon "github_web_origin": saml_provider.get("github_web_origin"), } + @app.transformer( name="enterprise_saml_assertion_consumer_service", table_name="saml_assertion_consumer_service", @@ -874,11 +1237,12 @@ def enterprise_saml_assertion_consumer_service( "github_web_origin": saml_provider.get("github_web_origin"), } + @app.transformer( name="enterprise_saml_issuer", table_name="saml_issuer", columns=SamlIssuer, - parallelized=True + parallelized=True, ) def enterprise_saml_issuer(saml_provider: SamlProvider, ctx: SourceContext): issuer = saml_provider.get("issuer") @@ -895,15 +1259,14 @@ def enterprise_saml_issuer(saml_provider: SamlProvider, ctx: SourceContext): "github_web_origin": saml_provider.get("github_web_origin"), } + @app.transformer( name="enterprise_external_identity", table_name="external_identities", columns=ExternalIdentity, parallelized=True, ) -def enterprise_external_identity( - saml_provider: SamlProvider, ctx: SourceContext -): +def enterprise_external_identity(saml_provider: SamlProvider, ctx: SourceContext): client, graphql_path = _sso_graphql_client(ctx) if not client: logger.info( @@ -956,7 +1319,10 @@ def enterprise_external_identity( except Exception as e: logger.error( f"Error in resource 'enterprise_external_identity' processing enterprise '{ctx.enterprise_name}': {e}", - extra={"resource": "enterprise_external_identity", "phase": "resource_iteration"}, + extra={ + "resource": "enterprise_external_identity", + "phase": "resource_iteration", + }, ) return @@ -968,9 +1334,16 @@ def enterprise_resources(ctx: SourceContext): teams_resource = enterprise_teams(ctx) roles_resource = enterprise_roles(ctx) runner_groups_resource = enterprise_runner_groups(ctx) - scim_organizations_resource = enterprise_resource | enterprise_scim_organizations(ctx) + scim_organizations_resource = enterprise_resource | enterprise_scim_organizations( + ctx + ) + credential_inventory_resource = ( + enterprise_resource | enterprise_credential_inventory(ctx) + ) resources = [ enterprise_resource, + credential_inventory_resource, + credential_inventory_resource | classic_personal_access_tokens(), enterprise_resource | organizations_resource, enterprise_resource | members_resource | enterprise_users(ctx), enterprise_resource | members_resource | enterprise_managed_users(ctx), @@ -990,8 +1363,12 @@ def enterprise_resources(ctx: SourceContext): resources.extend( [ enterprise_resource | saml_resource, - enterprise_resource | saml_resource | enterprise_saml_service_provider(ctx), - enterprise_resource | saml_resource | enterprise_saml_assertion_consumer_service(ctx), + enterprise_resource + | saml_resource + | enterprise_saml_service_provider(ctx), + enterprise_resource + | saml_resource + | enterprise_saml_assertion_consumer_service(ctx), enterprise_resource | saml_resource | enterprise_saml_issuer(ctx), enterprise_resource | saml_resource | enterprise_external_identity(ctx), enterprise_resource | runner_groups_resource, diff --git a/tests/test_classic_personal_access_tokens.py b/tests/test_classic_personal_access_tokens.py new file mode 100644 index 0000000..b2f4a49 --- /dev/null +++ b/tests/test_classic_personal_access_tokens.py @@ -0,0 +1,649 @@ +import gzip +import io +import json +import re +from types import SimpleNamespace +from unittest.mock import MagicMock + +import dlt +import pytest +import requests +import duckdb +from dlt.destinations import filesystem + +from openhound_github.kinds import edges as ek +from openhound_github.lookup import GithubLookup +from openhound_github.models.classic_personal_access_token import ( + ClassicPersonalAccessToken, +) +from openhound_github.models.enterprise_member import BaseUser +from openhound_github.models.enterprise import Enterprise +from openhound_github.resources.enterprise import ( + SourceContext, + _download_enterprise_credential_inventory, + classic_personal_access_tokens, + enterprise_credential_inventory, +) + + +class Response: + def __init__(self, status_code, *, payload=None, content=b"", headers=None): + self.status_code = status_code + self.payload = payload or {} + self.content = content + self.raw = io.BytesIO(content) + self.headers = headers or {} + + def json(self): + return self.payload + + def raise_for_status(self): + if self.status_code >= 400: + error_response = requests.Response() + error_response.status_code = self.status_code + error_response._content = json.dumps(self.payload).encode() + error_response.headers.update(self.headers) + raise requests.HTTPError(response=error_response) + + def close(self): + self.raw.close() + + def iter_content(self, chunk_size): + for offset in range(0, len(self.content), chunk_size): + yield self.content[offset : offset + chunk_size] + + +def test_classic_pat_export_polls_one_job_and_deduplicates_org_rows(monkeypatch): + client = MagicMock() + client.post.return_value = Response( + 202, payload={"export_id": "export-1", "as_of": "2026-10-05T20:00:00Z"} + ) + client.get.side_effect = [ + Response(200, payload={"status": "started"}), + Response(302, headers={"Location": "https://example.test/signed.csv"}), + ] + csv_content = ( + "credential_type,credential_id,hashed_token,display_name,owner_id,owner,scopes," + "credential_state,expiry_status,enterprise_authorized,authorization_count,organization,created_at\n" + 'classic_pat,42,hash-42,"CI, deploy",7,octocat,"repo; read:org",active,expires,true,3,acme,2026-01-01T00:00:00Z\n' + 'classic_pat,42,hash-42,"CI, deploy",7,octocat,"repo; read:org",active,expires,true,3,ops,2026-01-01T00:00:00Z\n' + "classic_pat,43,hash-43,Personal,8,hubot,repo,expired,expires,false,0,,2026-01-01T00:00:00Z\n" + "fine_grained_pat,42,hash-fg,Other,8,hubot,,active,never,false,1,acme,2026-01-01T00:00:00Z\n" + ) + download = MagicMock(return_value=Response(200, content=csv_content.encode())) + monkeypatch.setattr("openhound_github.resources.enterprise.requests.get", download) + monkeypatch.setattr( + "openhound_github.resources.enterprise.time.sleep", lambda _: None + ) + + ctx = SourceContext( + client=client, enterprise_name="enterprise", deployment_type="ghec" + ) + inventories = list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + assert len(inventories) == 1 + inventory = inventories[0] + assert inventory["export_id"] == "export-1" + assert inventory["as_of"] == "2026-10-05T20:00:00Z" + assert len(inventory["rows"]) == 4 + assert inventory["rows"][3]["credential_type"] == "fine_grained_pat" + assert inventory["rows"][3]["hashed_token"] == "hash-fg" + assert "hashed_token" in inventory["columns"] + rows = list(classic_personal_access_tokens.__wrapped__(inventory)) + + assert len(rows) == 2 + assert rows[0]["credential_id"] == 42 + assert rows[0]["display_name"] == "CI, deploy" + assert rows[0]["scopes"] == ["repo", "read:org"] + assert rows[0]["enterprise_authorized"] is True + assert rows[0]["authorization_count"] == 3 + assert rows[0]["inventory_as_of"] == "2026-10-05T20:00:00Z" + assert rows[0]["authorized_organizations"] == ["acme", "ops"] + assert rows[1]["authorized_organizations"] == [] + assert rows[1]["enterprise_authorized"] is False + assert rows[1]["credential_state"] == "expired" + client.post.assert_called_once_with( + "/enterprises/enterprise/credentials/exports", + headers={"X-GitHub-Api-Version": "2026-03-10"}, + ) + assert client.get.call_count == 2 + assert client.get.call_args.kwargs["allow_redirects"] is False + download.assert_called_once_with( + "https://example.test/signed.csv", timeout=120, stream=True + ) + + +def test_classic_pat_parser_skips_malformed_numeric_rows(caplog): + inventory = { + "enterprise_node_id": "E_1", + "enterprise_name": "enterprise", + "rows": [ + {"credential_type": "classic_pat", "credential_id": "invalid"}, + { + "credential_type": "classic_pat", + "credential_id": "42", + "owner_id": "bad", + }, + { + "credential_type": "classic_pat", + "credential_id": "42", + "owner_id": "7", + "authorization_count": "bad", + }, + { + "credential_type": "classic_pat", + "credential_id": "42", + "owner_id": "7", + "authorization_count": "1", + "organization": "acme", + }, + { + "credential_type": "classic_pat", + "credential_id": "42", + "owner_id": "bad", + "organization": "ops", + }, + { + "credential_type": "classic_pat", + "credential_id": "42", + "authorization_count": "bad", + "organization": "ops", + }, + {"credential_type": "classic_pat", "credential_id": "43"}, + ], + } + + tokens = list(classic_personal_access_tokens.__wrapped__(inventory)) + + assert [token["credential_id"] for token in tokens] == [42, 43] + assert tokens[0]["owner_id"] == 7 + assert tokens[0]["authorization_count"] == 1 + assert tokens[0]["authorized_organizations"] == ["acme"] + assert [ + int(row_number) + for row_number in re.findall( + r"Skipping classic PAT inventory row (\d+)", caplog.text + ) + ] == [1, 2, 3, 5, 6] + + +def test_one_export_persists_raw_rows_and_models_only_classic_pats( + monkeypatch, tmp_path +): + calls = [] + + def inventory_rows(client, slug, fallback_client=None): + calls.append(slug) + return { + "export_id": "export-1", + "as_of": "2026-10-05T20:00:00Z", + "columns": ["credential_type", "credential_id", "hashed_token"], + "rows": [ + { + "credential_type": "classic_pat", + "credential_id": "42", + "hashed_token": "hash-classic", + }, + { + "credential_type": "oauth_app_user_token", + "credential_id": "42", + "hashed_token": "hash-oauth", + }, + ], + } + + monkeypatch.setattr( + "openhound_github.resources.enterprise._download_enterprise_credential_inventory", + inventory_rows, + ) + + @dlt.resource(name="probe_enterprise", columns=Enterprise) + def parent(): + yield Enterprise(id="E_1", slug="enterprise") + + @dlt.source + def source(): + enterprise = parent() + inventory = enterprise | enterprise_credential_inventory( + SourceContext( + client=object(), enterprise_name="enterprise", deployment_type="ghec" + ) + ) + return [enterprise, inventory, inventory | classic_personal_access_tokens()] + + pipeline = dlt.pipeline( + pipeline_name="credential_inventory_test", + destination=filesystem(bucket_url=str(tmp_path / "output")), + dataset_name="github_test", + pipelines_dir=str(tmp_path / "pipelines"), + ) + pipeline.run(source()) + + def stored_rows(table): + rows = [] + for path in (tmp_path / "output" / "github_test" / table).glob("*.gz"): + with gzip.open(path, "rt") as file: + rows.extend(json.loads(line) for line in file) + return rows + + raw = stored_rows("enterprise_credential_inventory") + modeled = stored_rows("classic_personal_access_tokens") + assert calls == ["enterprise"] + assert len(raw) == 1 + assert raw[0]["rows"][1]["credential_type"] == "oauth_app_user_token" + assert raw[0]["rows"][1]["hashed_token"] == "hash-oauth" + assert len(modeled) == 1 + assert modeled[0]["credential_id"] == 42 + + fallback_calls = [] + + def retry_inventory(client, slug, export_id=None, fallback_client=None): + fallback_calls.append((slug, export_id)) + return inventory_rows(client, slug) + + monkeypatch.setattr( + "openhound_github.resources.enterprise._download_enterprise_credential_inventory", + retry_inventory, + ) + pipeline.run(source()) + assert fallback_calls == [("enterprise", "export-1")] + + +def test_classic_pat_inventory_rate_limit_skips_without_starting_another_export(caplog): + client = MagicMock() + client.post.return_value = Response(429) + ctx = SourceContext( + client=client, enterprise_name="enterprise", deployment_type="ghec" + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + assert client.post.call_count == 1 + assert "export creation rejected (HTTP 429)" in caplog.text + + +@pytest.mark.parametrize("status_code", [403, 429]) +def test_classic_pat_inventory_secondary_limit_is_not_reported_as_daily_export_limit( + monkeypatch, caplog, status_code +): + client = MagicMock() + client.post.return_value = Response( + status_code, payload={"message": "You have exceeded a secondary rate limit."} + ) + ctx = SourceContext( + client=client, enterprise_name="enterprise", deployment_type="ghec" + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + client.get.assert_not_called() + assert "secondary rate limit reached" in caplog.text + assert "export creation rejected" not in caplog.text + + +def test_classic_pat_inventory_uses_pat_only_when_app_export_creation_is_denied( + monkeypatch, caplog +): + app_client = MagicMock() + app_client.post.return_value = Response(403) + pat_client = MagicMock() + pat_client.post.return_value = Response( + 202, payload={"export_id": "export-1", "as_of": "2026-10-07T21:00:00Z"} + ) + pat_client.get.return_value = Response( + 302, headers={"Location": "https://example.test/export.csv"} + ) + monkeypatch.setattr( + "openhound_github.resources.enterprise.requests.get", + lambda *_, **__: Response( + 200, content=b"credential_type,credential_id\nclassic_pat,42\n" + ), + ) + ctx = SourceContext( + client=app_client, + sso_client=pat_client, + enterprise_name="enterprise", + deployment_type="ghec", + ) + + inventory = list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + assert len(inventory) == 1 + assert inventory[0]["rows"][0]["credential_type"] == "classic_pat" + app_client.post.assert_called_once() + app_client.get.assert_not_called() + pat_client.post.assert_called_once() + pat_client.get.assert_called_once() + assert "retrying with configured classic PAT" in caplog.text + + +@pytest.mark.parametrize( + "response", + [Response(429), Response(403, payload={"message": "secondary rate limit"})], +) +def test_classic_pat_inventory_does_not_switch_credentials_on_rate_limit(response): + app_client = MagicMock() + app_client.post.return_value = response + pat_client = MagicMock() + ctx = SourceContext( + client=app_client, + sso_client=pat_client, + enterprise_name="enterprise", + deployment_type="ghec", + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + pat_client.post.assert_not_called() + + +def test_classic_pat_inventory_does_not_start_second_export_after_poll_failure(): + app_client = MagicMock() + app_client.post.return_value = Response(202, payload={"export_id": "export-1"}) + app_client.get.return_value = Response(403) + pat_client = MagicMock() + ctx = SourceContext( + client=app_client, + sso_client=pat_client, + enterprise_name="enterprise", + deployment_type="ghec", + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + app_client.post.assert_called_once() + pat_client.post.assert_not_called() + + +def test_classic_pat_inventory_does_not_retry_same_token_client(): + token_client = MagicMock() + token_client.post.return_value = Response(403) + ctx = SourceContext( + client=token_client, + sso_client=token_client, + enterprise_name="enterprise", + deployment_type="ghec", + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + token_client.post.assert_called_once() + + +def test_classic_pat_inventory_reuses_recent_export_before_creating(monkeypatch): + state = { + "last_export_id": "prior-1", + "last_export_enterprise": "enterprise", + "last_export_as_of": "2026-10-05T20:00:00Z", + "last_export_downloaded_at": 1000, + } + monkeypatch.setattr( + "openhound_github.resources.enterprise.dlt.current.resource_state", + lambda _: state, + ) + monkeypatch.setattr("openhound_github.resources.enterprise.time.time", lambda: 1001) + client = MagicMock() + client.post.return_value = Response(429) + client.get.return_value = Response( + 302, headers={"Location": "https://example.test/prior.csv"} + ) + pat_client = MagicMock() + monkeypatch.setattr( + "openhound_github.resources.enterprise.requests.get", + lambda *_, **__: Response( + 200, content=b"credential_type,credential_id\nclassic_pat,42\n" + ), + ) + ctx = SourceContext( + client=client, + sso_client=pat_client, + enterprise_name="enterprise", + deployment_type="ghec", + ) + + inventories = list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + assert len(inventories) == 1 + assert inventories[0]["export_id"] == "prior-1" + assert inventories[0]["as_of"] == "2026-10-05T20:00:00Z" + client.post.assert_not_called() + client.get.assert_called_once() + pat_client.get.assert_not_called() + + +def test_classic_pat_inventory_creates_export_when_recent_one_is_gone(monkeypatch): + monkeypatch.setattr( + "openhound_github.resources.enterprise.dlt.current.resource_state", + lambda _: { + "last_export_id": "prior-1", + "last_export_enterprise": "enterprise", + "last_export_downloaded_at": 1000, + }, + ) + monkeypatch.setattr("openhound_github.resources.enterprise.time.time", lambda: 1001) + client = MagicMock() + client.get.side_effect = [ + Response(404), + Response(302, headers={"Location": "https://example.test/new.csv"}), + ] + client.post.return_value = Response(202, payload={"export_id": "new-1"}) + monkeypatch.setattr( + "openhound_github.resources.enterprise.requests.get", + lambda *_, **__: Response(200, content=b"credential_type,credential_id\n"), + ) + ctx = SourceContext( + client=client, enterprise_name="enterprise", deployment_type="ghec" + ) + + inventories = list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + assert inventories[0]["export_id"] == "new-1" + client.post.assert_called_once() + + +def test_classic_pat_inventory_reuses_pat_created_export_with_pat(monkeypatch): + state = {} + monkeypatch.setattr( + "openhound_github.resources.enterprise.dlt.current.resource_state", + lambda _: state, + ) + monkeypatch.setattr("openhound_github.resources.enterprise.time.time", lambda: 1000) + app_client = MagicMock() + app_client.post.side_effect = [Response(403), Response(403)] + pat_client = MagicMock() + pat_client.post.side_effect = [ + Response( + 202, payload={"export_id": "pat-export", "as_of": "2026-10-07T21:00:00Z"} + ), + Response(429), + ] + pat_client.get.side_effect = [ + Response(302, headers={"Location": "https://example.test/export.csv"}), + Response(302, headers={"Location": "https://example.test/export.csv"}), + ] + monkeypatch.setattr( + "openhound_github.resources.enterprise.requests.get", + lambda *_, **__: Response( + 200, content=b"credential_type,credential_id\nclassic_pat,42\n" + ), + ) + ctx = SourceContext( + client=app_client, + sso_client=pat_client, + enterprise_name="enterprise", + deployment_type="ghec", + ) + + first = list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + second = list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + + assert state["last_export_created_with_fallback"] is True + assert first[0]["created_with_fallback"] is True + assert second[0]["created_with_fallback"] is True + assert second[0]["export_id"] == "pat-export" + assert second[0]["as_of"] == "2026-10-07T21:00:00Z" + app_client.get.assert_not_called() + assert pat_client.get.call_count == 2 + app_client.post.assert_called_once() + pat_client.post.assert_called_once() + + +def test_classic_pat_inventory_does_not_reuse_stale_export(monkeypatch): + monkeypatch.setattr( + "openhound_github.resources.enterprise.dlt.current.resource_state", + lambda _: { + "last_export_id": "prior-1", + "last_export_enterprise": "enterprise", + "last_export_downloaded_at": 1000, + }, + ) + monkeypatch.setattr( + "openhound_github.resources.enterprise.time.time", lambda: 1000 + 86401 + ) + client = MagicMock() + client.post.return_value = Response(429) + ctx = SourceContext( + client=client, enterprise_name="enterprise", deployment_type="ghec" + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + client.get.assert_not_called() + + +def test_classic_pat_collection_skips_ghes(): + client = MagicMock() + ctx = SourceContext( + client=client, enterprise_name="enterprise", deployment_type="ghes" + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + client.post.assert_not_called() + + +def test_classic_pat_export_rejects_non_https_download(monkeypatch): + client = MagicMock() + client.post.return_value = Response(202, payload={"export_id": "export-1"}) + client.get.return_value = Response( + 302, headers={"Location": "http://example.test/file.csv"} + ) + download = MagicMock() + monkeypatch.setattr("openhound_github.resources.enterprise.requests.get", download) + + with pytest.raises(ValueError, match="invalid HTTPS download URL"): + _download_enterprise_credential_inventory(client, "enterprise") + download.assert_not_called() + + +def test_classic_pat_failed_export_produces_no_partial_inventory(caplog): + client = MagicMock() + client.post.return_value = Response(202, payload={"export_id": "export-1"}) + client.get.return_value = Response(200, payload={"status": "failed"}) + ctx = SourceContext( + client=client, enterprise_name="enterprise", deployment_type="ghec" + ) + + assert ( + list( + enterprise_credential_inventory.__wrapped__(SimpleNamespace(id="E_1"), ctx) + ) + == [] + ) + assert "Credential export did not complete" in caplog.text + client.post.assert_called_once() + + +def test_classic_pat_model_emits_owner_and_authorization_edges(): + token = ClassicPersonalAccessToken( + credential_id=42, + display_name="CI deploy", + owner_id=7, + owner_login="octocat", + scopes=["repo"], + credential_state="active", + authorized_organizations=["acme", "ops"], + enterprise_node_id="E_1", + enterprise_name="enterprise", + ) + lookup = MagicMock() + lookup.enterprise_user_id_for_database_id.return_value = "U_1" + lookup.enterprise_organization_id_for_login.side_effect = ["O_1", None] + token._lookup = lookup + + assert token.as_node.properties.node_id == "GH_CLASSIC_PAT_E_1_42" + edges = list(token.edges) + assert [edge.kind for edge in edges] == [ + ek.CONTAINS, + ek.HAS_PERSONAL_ACCESS_TOKEN, + ek.AUTHORIZED_FOR_ORGANIZATION, + ] + assert edges[1].start.value == "U_1" + assert edges[2].end.value == "O_1" + + +def test_inventory_owner_and_organization_resolve_to_graph_ids(): + member = BaseUser.model_validate( + { + "__typename": "User", + "id": "U_1", + "databaseId": 7, + "login": "octocat", + "createdAt": "2026-01-01T00:00:00Z", + "updatedAt": "2026-01-01T00:00:00Z", + } + ) + assert member.model_dump()["database_id"] == 7 + + connection = duckdb.connect(":memory:") + connection.execute("CREATE SCHEMA github") + connection.execute( + "CREATE TABLE github.enterprise_users (id VARCHAR, database_id BIGINT)" + ) + connection.execute("INSERT INTO github.enterprise_users VALUES ('U_1', 7)") + connection.execute( + "CREATE TABLE github.enterprise_organizations (id VARCHAR, login VARCHAR)" + ) + connection.execute( + "INSERT INTO github.enterprise_organizations VALUES ('O_1', 'Acme')" + ) + lookup = GithubLookup(connection) + + assert lookup.enterprise_user_id_for_database_id(7) == "U_1" + assert lookup.enterprise_organization_id_for_login("acme") == "O_1" diff --git a/tests/test_helpers.py b/tests/test_helpers.py index be42929..5a2daca 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -100,3 +100,36 @@ def fake_send( assert response.json() == {"data": {"organization": {"repositories": {}}}} assert len(requests_seen) == 2 + + +def test_rate_limit_warning_logs_path_without_query(caplog) -> None: + response = graphql_response( + url="https://api.github.com/enterprises/example/credentials/exports?token=private", + body={"message": "You have exceeded a secondary rate limit"}, + ) + response.status_code = 403 + + should_retry = github_retry_policy(BearerTokenAuth(token="static-token"))( + response, None + ) + + assert should_retry is False + assert "POST /enterprises/example/credentials/exports" in caplog.text + assert "skipping export creation retry" in caplog.text + assert "token=private" not in caplog.text + + +def test_credential_export_429_is_not_retried(caplog) -> None: + response = graphql_response( + url="https://api.github.com/enterprises/example/credentials/exports", + body={"message": "Too many requests"}, + headers={"Retry-After": "60"}, + ) + response.status_code = 429 + + should_retry = github_retry_policy(BearerTokenAuth(token="static-token"))( + response, None + ) + + assert should_retry is False + assert "POST /enterprises/example/credentials/exports" in caplog.text