This package and its sub-packages implement the functionality required to run Vanadium applications in a cluster environment.
In this kind of environment, the application is composed of one or more replicas. Each replica has its own Principal with Blessings that are extended from the application's Blessings.
If the application‘s Blessing is root/my-service, the replicas’ Blessings could be root/my-service/XXX and root/my-service/YYY.
The design was first proposed in this document.
The Cluster agent is in charge of storing the Blessings of the applications. When a replica starts, it fetches its Blessings from the cluster agent.
The Cluster agent interface is defined in service.vdl.
vkube is a command-line tool that helps manage Vanadium applications on Kubernetes.
Before vkube starts an application, it creates a new secret in the cluster agent and saves the Secret Key in a Kubernetes Secret object. This Secret Key will later be used by the replicas to get their Blessings.
Then, it takes the user-specified Deployment and adds a pod-agent container to its Pod template.
The pod-agent uses the Secret Key to get the blessings for the replica, and makes them available to the other container(s) via the agent protocol.
This walkthrough shows everything you need to start a test Vanadium application on Kubernetes.
It assumes that you are using the Container Engine on Google Cloud (aka GKE). The tools should work with Kubernetes on other platforms, but have not been tested outside of GKE.
This walkthrough assumes that the JIRI_ROOT is set and that $JIRI_ROOT/release/go/bin is in your PATH.
# Our work directory mkdir $HOME/cluster-walkthrough cd $HOME/cluster-walkthrough # For jiri and vanadium binaries export PATH=$JIRI_ROOT/.jiri_root/scripts:$JIRI_ROOT/release/go/bin:$PATH # Adjust these to your environment export PROJECT=my-project export ZONE=my-zone export CLUSTER=my-cluster
The following tools are required to work with GKE:
gcloud components install kubectl
jiri go install v.io/x/ref/services/cluster/vkube
You can create a kubernetes cluster from the cloud console, or with the gcloud container clusters create command.
This command creates a cluster named $CLUSTER.
gcloud container clusters create $CLUSTER --num-nodes 1 --project $PROJECT --zone $ZONE
gcloud compute disks create test-cluster-agent-disk --size 200GB --project $PROJECT --zone $ZONE
jiri go install v.io/x/ref/cmd/principal mkdir test-root principal create --with-passphrase=false test-root test-root export V23_CREDENTIALS=$(pwd)/test-root principal dump
By default, the vkube command looks for vkube.cfg in the current directory. If vkube.cfg is not in your current directory, you can specify where it is with --config.
cat - <<EOF >vkube.cfg
{
"project": "$PROJECT",
"zone": "$ZONE",
"cluster": "$CLUSTER",
"clusterAgent": {
"namespace": "default",
"image": "gcr.io/$PROJECT/cluster-agent:latest",
"cpu": "0.1",
"memory": "250M",
"blessing": "test-root:cluster-agent",
"admin": "test-root",
"persistentDisk": "test-cluster-agent-disk"
},
"podAgent": {
"image": "gcr.io/$PROJECT/pod-agent:latest"
}
}
EOF
The blessings and admin values should match the credentials that you created earlier.
This command will build two docker images and push them to your project's registry.
vkube build-docker-images -v
This command creates a Service and a Deployment for the cluster agent. Make sure that V23_CREDENTIALS is set and points at your test credentials.
vkube start-cluster-agent --wait
The first time that the cluster agent is started, it doesn't have any blessings. The claim-cluster-agent command sends the blessings defined in vkube.cfg to the cluster agent.
vkube claim-cluster-agent
In this example, we use fortuned as our test application.
Let‘s create a docker image that contains a statically linked fortuned binary, and push it to the project’s registry.
mkdir fortune-docker cat - <<EOF >fortune-docker/Dockerfile FROM busybox COPY fortuned /usr/local/bin/ EOF IMAGE=gcr.io/$PROJECT/fortuned jiri go build -o fortune-docker/fortuned -ldflags "-extldflags -static" \ v.io/x/ref/examples/fortune/fortuned docker build -t $IMAGE fortune-docker gcloud docker push $IMAGE
The Pod template contains only the fortuned container. There is no need to specify the pod-agent container. The vkube command adds it for us transparently.
cat - <<EOF >fortuned-deployment.json
{
"apiVersion": "extensions/v1beta1",
"kind": "Deployment",
"metadata": {
"name": "fortuned"
},
"spec": {
"template": {
"metadata": {
"labels": {
"application": "fortuned"
}
},
"spec": {
"containers": [
{
"name": "fortuned",
"image": "gcr.io/$PROJECT/fortuned:latest",
"command": [ "fortuned", "--v23.tcp.address=:2345" ],
"ports": [ { "containerPort": 2345 } ]
}
]
}
}
}
}
EOF
fortunedvkube start -f fortuned-deployment.json fortuned --wait
vkube kubectl get pods
cat - <<EOF >fortuned-service.json
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"name": "fortune"
},
"spec": {
"ports": [
{ "port": 2345, "targetPort": 2345 }
],
"selector": {
"application": "fortuned"
},
"type": "LoadBalancer"
}
}
EOF
vkube kubectl create -f fortuned-service.json
vkube kubectl get service fortune
The last command will show the service's external IP address after a minute or two.
jiri go install v.io/x/ref/cmd/vrpc
IPADDR=$(vkube kubectl get service fortune --template '{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}')
vrpc signature /$IPADDR:2345
vrpc identify /$IPADDR:2345
This should show something like
$ vrpc signature /$IPADDR:2345
// Fortune is the interface to a fortune-telling service.
type "v.io/x/ref/examples/fortune".Fortune interface {
// Adds a fortune to the set used by Get().
Add(fortune string) error
// Returns a random fortune.
Get() (fortune string | error)
// Returns whether or not a fortune exists.
Has(fortune string) (bool | error)
}
$ vrpc identify /$IPADDR:2345
PRESENTED: test-root:fortuned:10.64.0.4:54945
VALID: [test-root:fortuned:10.64.0.4:54945]
PUBLICKEY: 3d:92:fc:5f:b3:ee:72:1f:7b:bf:22:a2:d4:d8:86:1a
When you're done, these commands will turn off everything.
vkube stop -f fortuned-deployment.json vkube kubectl delete service fortune vkube stop-cluster-agent gcloud compute disks delete test-cluster-agent-disk --project $PROJECT --zone $ZONE
If you created a test cluster, you can delete it with gcloud container clusters create ...