Skip to content

demo.adoc

Latest commit

 

History

History
189 lines (149 loc) · 11.9 KB

demo.adoc

File metadata and controls

189 lines (149 loc) · 11.9 KB

stackablectl demo

A demo is an end-to-end demonstration of the usage of the Stackable data platform. It is tied to a specific stack of the Stackable data platform, which will provide the required products for the demo.

General Usage

management:stackablectl:partial$commands/demo.adoc

Browse Available Demos

To list the available demos, run the following command:

$ stackablectl demo list
┌────┬───────────────────────┬─────────┬─────────────────────────────────────────────────────────┐
│ #  ┆ NAME                  ┆ STACK   ┆ DESCRIPTION                                             │
╞════╪═══════════════════════╪═════════╪═════════════════════════════════════════════════════════╡
│ 1  ┆ airflow-scheduled-job ┆ airflow ┆ Activate a simple Airflow DAG to run continuously at a  │
│    ┆                       ┆         ┆ set interval                                            │
├╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 2  ┆ ...                   ┆ ...     ┆ ...                                                     │
├╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 11 ┆ local-demo-test-env   ┆ test    ┆ Test demo. Provided with the STACKABLE_DEMO_FILES env   │
│    ┆                       ┆         ┆ var in the .env file                                    │
└────┴───────────────────────┴─────────┴─────────────────────────────────────────────────────────┘

Detailed information of a demo can be queried using the describe command:

$ stackablectl demo describe trino-taxi-data
 DEMO           trino-taxi-data
 DESCRIPTION    Demo loading 2.5 years of New York taxi data into S3 bucket, creating a Trino table and a Superset dashboard
 DOCUMENTATION  https://docs.stackable.tech/stackablectl/stable/demos/trino-taxi-data.html
 STACK          trino-superset-s3
 LABELS         trino, superset, minio, s3, ny-taxi-data

Installing a Demo

Using an Existing Kubernetes Cluster

If you want to access a Kubernetes cluster, make sure your kubectl Kubernetes client is configured to interact with the Kubernetes cluster. After that, run the following command:

$ stackablectl demo install trino-taxi-data

Using a Local Kubernetes Cluster

If you don’t have a Kubernetes cluster available, stackablectl can spin up a kind or minikube Kubernetes cluster for you. Based on the type of local cluster you want to use, ensure you have either kind or minikube installed on your system. stackablectl will perform a check to verify that these tools are available in your PATH and check if Docker is running.

Local Kind Kubernetes Cluster

$ stackablectl demo install trino-taxi-data -c kind
Creating cluster "stackable-data-platform" ...
 ✓ Ensuring node image (kindest/node:v1.26.3) 🖼
 ✓ Preparing nodes 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-stackable-data-platform"
You can now use your cluster with:

kubectl cluster-info --context kind-stackable-data-platform

Have a nice day! 👋

Installed demo trino-taxi-data

Use "stackablectl operator installed" to display the installed operators
Use "stackablectl stacklet list" to display the installed stacklets

Local Minikube Kubernetes Cluster

$ stackablectl demo install trino-taxi-data -c minikube
😄  [stackable-data-platform] minikube v1.30.1 on Ubuntu 22.04.2
✨  Using the docker driver based on user configuration
🎉  minikube 1.31.2 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.31.2
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

📌  Using Docker driver with root privileges
👍  Starting control plane node stackable-data-platform in cluster stackable-data-platform
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🔎  Verifying Kubernetes components...

👍  Starting worker node stackable-data-platform-m02 in cluster stackable-data-platform
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.58.2
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
    ▪ env NO_PROXY=192.168.58.2
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "stackable-data-platform" cluster and "default" namespace by default
Installed demo trino-taxi-data

Use "stackablectl operator installed" to display the installed operators
Use "stackablectl stacklet list" to display the installed stacklets

The demos create Kubernetes jobs that will populate test data and interact with the installed products to process the data. Until the products are ready, it is expected that the pods of these Jobs will fail with an error. They will get retried with an exponentially growing back-off time. After the products are ready, they should turn green, and everything should settle down.

Listing Deployed Stacklets

After installing your demo you can use the stackablectl stacklets command to list the installed stacklets as follows:

$ stackablectl stacklets list
┌──────────┬───────────────┬───────────┬─────────────────────────────────────────────┬────────────────────────────────────────────┐
│ PRODUCT  ┆ NAME          ┆ NAMESPACE ┆ ENDPOINTS                                   ┆ CONDITIONS                                 │
╞══════════╪═══════════════╪═══════════╪═════════════════════════════════════════════╪════════════════════════════════════════════╡
│ hive     ┆ hive          ┆ default   ┆                                             ┆ Available, Reconciling, Running            │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ opa      ┆ opa           ┆ default   ┆                                             ┆ Available, Reconciling, Running            │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ superset ┆ superset      ┆ default   ┆ external-superset http://192.168.58.2:30788 ┆ Available, Reconciling, Running            │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ trino    ┆ trino         ┆ default   ┆                                             ┆ Unavailable: See [1], Reconciling, Running │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ minio    ┆ minio-console ┆ default   ┆                                             ┆                                            │
└──────────┴───────────────┴───────────┴─────────────────────────────────────────────┴────────────────────────────────────────────┘

[1]: StatefulSet ["trino-coordinator-default", "trino-worker-default"] missing ready replicas.

Uninstalling a Demo

To uninstall a demo, you can run the following command, specifying the namespace the demo was installed in.

$ stackablectl demo uninstall trino-taxi-data -n <NAMESPACE>
Note

The uninstall command deletes the namespace the demo was installed in. Therefore it is not possible to uninstall demos in the default namespace.
Warning

Currently, some resources deployed by the demo are either still lingering after the deletion process or need to be deleted before running the uninstall command.

  • All demos

    • MutatingWebhookConfiguration restarter-sts-enricher.stackable.tech is not deleted

  • end-to-end-security, argo-cd-git-ops, hbase-hdfs-load-cycling-data, jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

    • ClusterroleBinding hdfs-clusterrolebinding-nodes is not deleted

  • argo-cd-git-ops

    • Delete the ArgoCD Applications airflow, airflow-postgres, minio, and sealed-secrets

    • Then you can delete the namespaces stackable-airflow, minio, and sealed-secrets

    • Delete the ArgoCD ApplicationSet stackable-operators and wait for ArgoCD to delete the Applications of that ApplicationSet

    • Only after deleting the above, run the stackablectl demo uninstall command, otherwise the deletion might get stuck

    • *.argoproj.io CRDs and sealedsecrets.bitnami.com CRD are not deleted