Kops tasks and effort estimation ¶
List of open issues and features and their effort estimation as of .
TODO Issues listed below require proper labels to be assigned, especially P0.
Priorities
P0: Must have fixes and features, needed to make existing vSphere support in kops work.
P1: Important fixes and features, required to give vSphere users a more useful kubernetes cluster deployment experience, with multiple masters and HA.
P2: Rarely occurring issues and features that will bring vSphere support closer to AWS and GCE support in kops.
Notes:
- Effort estimation includes fix for an issue or implementation for a feature, testing and generating a PR.
- There are a few issues that are related to startup and base image. If we can resolve "Use PhotonOS for vSphere node template" issue first and replace init-cloud with guestinfo, those issues might get resolved automatically. But further investigation is needed and fixed issues will need verifications and testings.
| Priority | Task | Type (bug, feature, test | Effort estimate(in days) | Remarks |
|---|---|---|---|---|
| P0 | Kops for vSphere is broken kubernetes/kops#2729 | Bug | 1 | |
| P0 | AWS EBS is set as default volume provisioner, instead of vSphere kubernetes/kops#2732 | Bug | 1 | Looks like the fix is available, need to be tested again before committing https://github.com/vmware/kops/pull/70/files |
| P0 | Package installation through nodeup causing delay in cluster deployment kubernetes/kops#2742 | Bug | 2 | If we get the "Use PhotonOS for vSphere node template" done first, this one may just be avoided. Verification required. |
| P0 | Connection to api.clustername.skydns.local is failing kubernetes/kops#2744 | Bug | 1 | This might be fixed by Abrar's PR in kubernetes already. Just need to verify. |
| P0 | Make update command, that includes scale up and down, work kubernetes/kops#2738. There are two possible ways to implement this- without auto scaling group (ASG) or with auto scaling group | Feature | 4 days Assuming ASG is available. 4 days ASG is not available. | This effort estimation needs more analysis. |
| P0 | Make end-to-end CI/CD work on vmware/kops kubernetes/kops#2730 | Bug, Test | 3 | |
| P1 | Use PhotonOS for vSphere node template kubernetes/kops#2735. Use guestinfo instead of init-cloud kubernetes/kops#2726 | Feature | 5 | This was originally P2 issue. However several other issues might be affected by this one. So bring it to P1. Problem to solve: init-cloud on PhotonOS not working properly. Or, get rid of init-cloud and use guestinfo instead. |
| P1 | Default image name extraction for vSphere needs to be fixed kubernetes/kops#2740 | Bug | 2 | |
| P1 | Multi-master HA setup kubernetes/kops#2734 | Feature | 5 | |
| P1 | Add unit tests for vSphere workflows kubernetes/kops#2745 | Test | 8 | This effort estimation needs more analysis as estimator is not familiar with all the components for which tests need to be written. |
| P1 | Documentation of- 1) Existing commands and usage, blog kubernetes/kops#2739, 2) Behavior for all flags for ‘kops cluster create’ command kubernetes/kops#2741 | 3 | ||
| P2 | vCenter running out of HTTP sessions kubernetes/kops#2747 | Bug | 3 | |
| P2 | Long boot time for template VM kubernetes/kops#2746 | Bug | 3 | This estimation needs more analysis. Same here: if Use PhotonOS for vSphere node template can be resolved first, this might not be valid. Still, verification needed. |
| P2 | Improve methods that create user-data and meta-data for ISO kubernetes/kops#2748, or use guestinfo instead of cloud-init for passing in VM specific informations kubernetes/kops#2726 | Feature | 2 | We need decide If we should use guestinfo instead of cloud-init. |
| P2 | Support for rolling upgrade, normal upgrade | Feature | 7 | This estimation needs more analysis. Presence of ASG might simplify the implementation of this feature. Some changes might be needed in core kops code, as current rolling upgrade implementation is very much AWS specific. |
| P2 | Make ETCD volumes re-attachable for vSphere (AWS and GCE already support this) kubernetes/kops#2736 | Feature | 7 | This task needs more analysis for design and implementation. Estimate might change accordingly. |
| P2 | Security and isolation- 1) Networking for master, worker nodes kubernetes/kops#2731. 2) Credentials in plain text kubernetes/kops#2743 | Feature | -- | Don't have enough information on this to give an estimate for effort involved. |
| P2 | Explore vSphere DRS cluster’s anti-affinity rules to achieve better master HA and meaningful zone allotment for masters kubernetes/kops#2733 | Feature | 8 | This estimation needs more analysis. DRS anti-affinity rule needs to be explored, to see if it's even fit for this problem. |
| P2 | Enable user-defined dns zone name kubernetes/kops#2727 | Feature | 2 | |
| Total 67 |
Kops commands behavior for vSphere ¶
List of all kops commands and how they behave for vSphere cloud provider, as of .
Column explanation ¶
- Command, option and usage example are self-explanatory.
- vSphere support: whether or not the command is supported for vSphere cloud provider (Yes/No), followed by current status of that command and explanation of any failures.
- Graceful termination needed: If the command will not supported, does it need additional code to fail gracefully for vSphere provider?
- Remark: Miscellaneous comments about the command.
| Command | Option | Usage example | vSphere support | Graceful termination needed (if not fixed) | Remark |
|---|---|---|---|---|---|
| completion | bash | kops completion bash | Yes | No | Output shell completion code for the given shell (bash), which can easily be incorporated in a bash script to run kops commands as bash functions. |
| create | cluster | Yes. Supported/tested command flags: cloud, dns, dns-zone, image, networking, node-count, vsphere-server, vsphere-datacenter, vsphere-resource-pool, vsphere-datastore, vsphere-coredns-server, yes, zones. | Yes. Check for unsupported flags. Terminate command, if needed, with appropriate message. | Creates cluster spec and configs. If --yes is specified then creates resources as well. | |
| create | instancegroup | kops create ig --name=v1c1.skydns.local --role=Node --subnet=vmw-zone nodes2 | No. InstanceGroup spec gets created in object store. Command however shows this error even after setting 'image' value in spec: I0412 11:08:23.025842 80677 populate_instancegroup_spec.go:257] Cannot set default Image for CloudProvider="vsphere" | Yes. Either add a check for vSphere, or fix the issue causing the failure. | |
| create | secret | kops create secret sshpublickey test_key -i ~/.ssh/git_rsa.pub | Yes | No | Creates and delete secrets can be used in combination to replace existing secrets. Justin's explanation: "k8s in theory supports multiple certificates but it was not working until 1.5 so I don't think we actually enable it in kops This will be how we do certificate rotation though - add a certificate, roll that out, roll out a new key and switch to the new key" |
| create | -f FILENAME | Three yams files are required- cluster: kops create -f ~/kops.yaml, master IG: kops create -f ~/kops.nodeig.yaml, node IG: kops create -f ~/kops.masterig.yaml | Yes | No | |
| delete | cluster | kops delete cluster v2c1.skydns.local --yes | Yes | No | |
| delete | instancegroup | kops delete instancegroup --name=v2c1.skydns.local nodes.v2c1.skydns.local | No. No implementation available to list resources. Method corresponding to AWS is getting called and crashing with panic, without any useful message. | Yes | |
| delete | secret | Yes | - | ||
| delete | -f FILENAME | kops delete -f config.yaml --name=v2c1.skydns.local | No. Cluster deletion works. Instance group deletion is failing with error: panic: interface conversion: *vsphere.VSphereCloud is not awsup.AWSCloud: missing method AddAWSTags goroutine 1 [running]: panic(0x26fbd20, 0xc420770780) /usr/local/go/src/runtime/panic.go:500 +0x1a1 k8s.io/kops/upup/pkg/kutil.FindCloudInstanceGroups | Yes | Delete cluster, ig specified by the file. |
| describe | secrets | kops describe secrets | Yes | No | Describe secrets, based on the kubectl context. |
| edit | cluster | kops edit cluster --name=v2c1.skydns.local nodes | Yes. Edited spec gets updated in object store. | Yes | Edit works. But it would be a bad user experience if we allow users to edit the spec, followed by a failed 'kops update' and then no way to go back to the older spec. |
| edit | ig | kops edit ig --name=v2c1.skydns.local nodes | Yes. Edited spec gets updated in object store. | Yes | Edit works. But it would be a bad user experience if we allow users to edit the spec, followed by a failed 'kops update' and then no way to go back to the older spec. |
| edit | federation | No | Yes | Federation is a group of k8s clusters. This doesn't look an important goal for vSphere in near future. Q: "How is a federation getting created? I see update and edit methods for a federation but I am not clear how to get a federation in the first place." A: Justin's reply: I'm chatting with the federation folk about kubefed & kops and whether we should integrate them etc. The federation stuff was very alpha and I believe is (trivially) broken right now, but I'm debating integrating with kubefed vs fixing kops federation. kubefed worked fine when I tried it the other day. | |
| export | kubecfg | kops export kubecfg v1c1.skydns.local | Yes | - | Sets kubectl context to given cluster. |
| get | clusters | Yes | - | Gets list of clusters. If yaml output is specified, this output can be modified and used for 'kops replace' command. | |
| get | federations | Yes | - | Gets list of federations. For now empty list. | |
| get | instancesgroups | Yes | - | Gets list of intancegroups. If yaml output is specified, this output can be modified and used for 'kops replace' command. | |
| get | secrets | Yes | - | Gets list of secrets. | |
| import | cluster | kops import cluster --region=us-west-2 --name=v2c1.skydns.local nodes | No. Current implementation is very aws specific. Multiple aws services are queried to construct the api.Cluster object. | Yes | Imports spec for an existing cluster into the object store. While this functionality is good for importing and managing existing k8s clusters using kops, it doesn't seem like a high priority functionality at this point of time. |
| replace | kops replace -f FILENAME | No | Yes | Output of kops get cluster name -oyaml or kops get ig name -oyaml can be updated and passed to 'kops replace' command. |
|
| rolling-update | cluster | No. Current implementation is aws specific. | Yes | ||
| secrets | create | - | - | Legacy command, points to 'kops create secrets'. | |
| secrets | describe | - | - | Legacy command, points to 'kops describe secrets'. | |
| secrets | expose | - | - | Legacy command, points to 'kops get secrets -oplaintext'. | |
| secrets | get | - | - | Legacy command, point to 'kops get secret'. | |
| toolbox | dump | No. Current implementation is aws specific. | Yes | Dumps cloud information for the given cluster. This looks like a good to have functionality. Once resource listing is available for vsphere, which will anyways get used for deletion operation as well, this command should become easier to implement. | |
| toolbox | convert-imported | No. Current implementation is aws specific. | Yes | Doesn't look like a high priority functionality. | |
| update | cluster | kops update cluster --name=v2c1.skydns.local --yes | No. 1) Works for new cluster. 2) Existing cluster scale up: vSphere provisioning code tries to provision all master and node VMs from scratch. New nodes get created and registered successfully. Existing resources keep failing with 'already exists' error. 3) Existing cluster scale down: Won't work, no resource listing or deletion logic available for vSphere. On top of that all listed resources- masters and workers are attempted for creation and fail with 'already exists' error. | Yes | |
| update | federation | No | Yes | Federation is a group of k8s clusters. This doesn't look an important goal for vSphere in near future. | |
| upgrade | cluster | kops upgrade cluster --name=v1c1.skydns.local --yes | No. Seeing this error: W0413 11:48:52.216116 15456 upgrade_cluster.go:202] No matching images specified in channel; cannot prompt for upgrade | Yes | Find out more about 'channel' in context of kops. Note that no --channel argument is specified. |
| validate | cluster | kops validate cluster --name=v1c1.skydns.local | Yes. Not working right now. Failing with this error: cannot get nodes for "v1c1.skydns.local": Get https://api.v1c1.skydns.local/api/v1/nodes: dial tcp: lookup api.v1c1.skydns.local: no such host | - | Investigation is already going on- https://github.com/kubernetes/kops/issues/2744. This issue will most likely get fixed by a fix in cloud-provider code that is not returning appropriate internal and external IP for the node. |
| version | kops version | Yes | - | Prints client version information. Eg: Version 1.6.0-alpha.1 (git-500cb69) |