analytics-nginx | ||
api | ||
kubernetes | ||
mimir | ||
s3-endpoint | ||
spark-docker | ||
ui-nginx | ||
vizier-nginx-proxy | ||
.DS_Store | ||
build-images.sh | ||
Readme.md | ||
reset_microk8s.sh | ||
run-containers.sh | ||
run_containers_norn.sh |
Deploy a Containerized Vizier Instance
Vizier is a cloud-enabled tool that makes it easy to explore, validate, transform and debug data.
- vizier core platform is open-source. See: api, ui, mimir on GitHub
- for more details about Vizier see: https://vizierdb.info
Components
Vizier has a number of components that are not trivial to set up manually:
- [Web UI] - React user interface!
- [API Server] - python wsgi api server
- [MimirDB] - virtual probabalistic database
- [Proxy] - a reverse proxy that provides an endpoint for vizier services
- [Apache Spark] - distributed data processing
- [Hadoop] - evented I/O for the backend
- [S3] - optional data staging endpoint
- [Analytics] - optional vizier ui access tracking
Though instalation instructions for each of these components is availabel, it is time-consuming and difficult to install them manually. So is there an easier containerized deploment? Yes! Deployment to a kubernetes cluster is explained below.
Deploy The Vizier Stack to Kubernetes
If you already have a kubernetes cluster set up, good, make sure CoreDNS is enabled (we are using k8s v1.13.2). If not, you can get a single node cluster setup pretty fast using microk8s. See microk8s docs for more details, but basically you can just do the following:
sudo snap install microk8s --classic
microk8s.enable dns dashboard
Once your cluster is ready, get the yaml file for deploying Vizier and make the following adjustments:
- update the host paths for the persistant volumes if you would like them somewhere other than /mnt/ (YAML Line 15, 28, 41, 310)
- update the s3-credentials secret with your S3 access key id and secret - base64 encode them first: (YAML Line 330, 331)
echo "YOUR-S3-ACCESS-KEY-ID" | base64 echo "YOUR-S3-ACCESS-KEY-SECRET" | base64
- update the VIZIER_DOMAIN env variable for the vizier-proxy deployment to the domain you will use to access Vizier. You can use a real domain and DNS entries or the hosts file of a client. (YAML Line 622)
Deploy vizier
kubectl create -f vizier-deployment.yaml
You may need to do this to allow containers to access the internet
sudo iptables -P FORWARD ACCEPT
Find the ClusterIP or ExternalIP of the vizier-proxy service
kubectl get service vizier-proxy
After you have the IP of the vizier-proxy service you need to add the following entries to either DNS for a real domain or the hosts file of the client: so where VIZIER_DOMAIN=vizier.dev
IP Address | Host Name | Purpose |
---|---|---|
demo.vizier.dev | web ui for vizier | |
api.vizier.dev | web api for vizier | |
vizier.vizier.dev | supervisor ctl for api | |
mimir.vizier.dev | supervisor ctl for mimir | |
proxy.vizier.dev | supervisor ctl for proxy | |
analytics.vizier.dev | endpoint for access analytics and ui | |
spark.vizier.dev | web ui for spark master | |
driver.vizier.dev | web ui for spark driver | |
hdfs.vizier.dev | web ui for hadoop |
Now you should be able to access the Vizier UI from a web browser.
https://demo.<VIZIER_DOMAIN>/vizier-db
License
Apache License 2.0