tree: ec302ce5abea16cd77bf0e10d12ecd3aabc9afa6 [path history] [tgz]
  1. cluster_agent/
  2. cluster_agentd/
  3. diagrams/
  4. internal/
  5. vkube/
  6. cluster.vdl.go
  7. README.md
  8. service.vdl
services/cluster/README.md

Vanadium Cluster

This package and its sub-packages implement the functionality required to run Vanadium applications in a cluster environment.

In this kind of environment, the application is composed of one or more replicas. Each replica has its own Principal with Blessings that are extended from the application's Blessings.

If the application‘s Blessing is root/my-service, the replicas’ Blessings could be root/my-service/XXX and root/my-service/YYY.

The design was first proposed in this document.

Cluster agent

The Cluster agent is in charge of storing the Blessings of the applications. When a replica starts, it fetches its Blessings from the cluster agent.

The Cluster agent interface is defined in service.vdl.

vkube

vkube is a command-line tool that helps manage Vanadium applications on Kubernetes.

Before vkube starts an application, it creates a new secret in the cluster agent and saves the Secret Key in a Kubernetes Secret object. This Secret Key will later be used by the replicas to get their Blessings.

New Secret Diagram

Then, it takes the user-specified Deployment and adds a pod-agent container to its Pod template.

The pod-agent uses the Secret Key to get the blessings for the replica, and makes them available to the other container(s) via the agent protocol.

Pod Agent Diagram

Walkthrough for GKE

This walkthrough shows everything you need to start a test Vanadium application on Kubernetes.

It assumes that you are using the Container Engine on Google Cloud (aka GKE). The tools should work with Kubernetes on other platforms, but have not been tested outside of GKE.

Environment

This walkthrough assumes that the JIRI_ROOT is set and that $JIRI_ROOT/release/go/bin is in your PATH.

# Our work directory
mkdir $HOME/cluster-walkthrough
cd $HOME/cluster-walkthrough

# For jiri and vanadium binaries
export PATH=$JIRI_ROOT/.jiri_root/scripts:$JIRI_ROOT/release/go/bin:$PATH

# Adjust these to your environment
export PROJECT=my-project
export ZONE=my-zone
export CLUSTER=my-cluster

Tools

The following tools are required to work with GKE:

gcloud components install kubectl
  • vkube
jiri go install v.io/x/ref/services/cluster/vkube

Create a cluster

You can create a kubernetes cluster from the cloud console, or with the gcloud container clusters create command.

This command creates a cluster named $CLUSTER.

gcloud container clusters create $CLUSTER --num-nodes 1 --project $PROJECT --zone $ZONE

Create a persistent disk for the cluster agent

gcloud compute disks create test-cluster-agent-disk --size 200GB --project $PROJECT --zone $ZONE

Create test Vanadium credentials

jiri go install v.io/x/ref/cmd/principal
mkdir test-root
principal create --with-passphrase=false test-root test-root
export V23_CREDENTIALS=$(pwd)/test-root
principal dump

Create vkube.cfg

By default, the vkube command looks for vkube.cfg in the current directory. If vkube.cfg is not in your current directory, you can specify where it is with --config.

 cat - <<EOF >vkube.cfg
{
  "project": "$PROJECT",
  "zone": "$ZONE",
  "cluster": "$CLUSTER",
  "clusterAgent": {
    "namespace": "default",
    "image": "gcr.io/$PROJECT/cluster-agent:latest",
    "cpu": "0.1",
    "memory": "250M",
    "blessing": "test-root:cluster-agent",
    "admin": "test-root",
    "persistentDisk": "test-cluster-agent-disk"
  },
  "podAgent": {
    "image": "gcr.io/$PROJECT/pod-agent:latest"
  }
}
EOF

The blessings and admin values should match the credentials that you created earlier.

Build the cluster-agent and pod-agent images

This command will build two docker images and push them to your project's registry.

vkube build-docker-images -v

Start the cluster agent

This command creates a Service and a Deployment for the cluster agent. Make sure that V23_CREDENTIALS is set and points at your test credentials.

vkube start-cluster-agent --wait

Claim the cluster agent

The first time that the cluster agent is started, it doesn't have any blessings. The claim-cluster-agent command sends the blessings defined in vkube.cfg to the cluster agent.

vkube claim-cluster-agent

Build a docker image

In this example, we use fortuned as our test application.

Let‘s create a docker image that contains a statically linked fortuned binary, and push it to the project’s registry.

mkdir fortune-docker

 cat - <<EOF >fortune-docker/Dockerfile
FROM busybox
COPY fortuned /usr/local/bin/
EOF

IMAGE=gcr.io/$PROJECT/fortuned
jiri go build -o fortune-docker/fortuned -ldflags "-extldflags -static" \
  v.io/x/ref/examples/fortune/fortuned
docker build -t $IMAGE fortune-docker
gcloud docker push $IMAGE

Run your first application

Create fortuned-deployment.json

The Pod template contains only the fortuned container. There is no need to specify the pod-agent container. The vkube command adds it for us transparently.

 cat - <<EOF >fortuned-deployment.json
{
  "apiVersion": "extensions/v1beta1",
  "kind": "Deployment",
  "metadata": {
    "name": "fortuned"
  },
  "spec": {
    "template": {
      "metadata": {
        "labels": {
          "application": "fortuned"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "fortuned",
            "image": "gcr.io/$PROJECT/fortuned:latest",
            "command": [ "fortuned", "--v23.tcp.address=:2345" ],
            "ports": [ { "containerPort": 2345 } ]
          }
        ]
      }
    }
  }
}
EOF

Start the application with blessing extension fortuned

vkube start -f fortuned-deployment.json fortuned --wait

Verify that fortuned is running.

vkube kubectl get pods

Create the fortune service (load balancer)

 cat - <<EOF >fortuned-service.json
{
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {
    "name": "fortune"
  },
  "spec": {
    "ports": [
      { "port": 2345, "targetPort": 2345 }
    ],
    "selector": {
      "application": "fortuned"
    },
    "type": "LoadBalancer"
  }
}
EOF

vkube kubectl create -f fortuned-service.json

vkube kubectl get service fortune

The last command will show the service's external IP address after a minute or two.

Verify that the service is working

jiri go install v.io/x/ref/cmd/vrpc

IPADDR=$(vkube kubectl get service fortune --template '{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}')
vrpc signature /$IPADDR:2345
vrpc identify /$IPADDR:2345

This should show something like

$ vrpc signature /$IPADDR:2345
// Fortune is the interface to a fortune-telling service.
type "v.io/x/ref/examples/fortune".Fortune interface {
	// Adds a fortune to the set used by Get().
	Add(fortune string) error
	// Returns a random fortune.
	Get() (fortune string | error)
	// Returns whether or not a fortune exists.
	Has(fortune string) (bool | error)
}

$ vrpc identify /$IPADDR:2345
PRESENTED: test-root:fortuned:10.64.0.4:54945
VALID:     [test-root:fortuned:10.64.0.4:54945]
PUBLICKEY: 3d:92:fc:5f:b3:ee:72:1f:7b:bf:22:a2:d4:d8:86:1a

Clean up

When you're done, these commands will turn off everything.

vkube stop -f fortuned-deployment.json
vkube kubectl delete service fortune
vkube stop-cluster-agent
gcloud compute disks delete test-cluster-agent-disk --project $PROJECT --zone $ZONE

If you created a test cluster, you can delete it with gcloud container clusters create ...