

Release notes for kops 1.12 series

Significant changes

• kops 1.12 enables etcd-manager by default. For kubernetes 1.12 (and later) we default to etcd3. We also enable TLS for etcd communications when using etcd-manager. The upgrade is therefore disruptive to the masters. More information is in the etcd migration documentation. This documentation is useful even if you are already using etcd3 with TLS.

• Components are no longer allowed to interact with etcd directly. Calico will be switched to use CRDs instead of directly with etcd. This is a disruptive upgrade, please read the calico notes in the etcd migration documentation

Required Actions

• Please back-up important data before upgrading, as the etcd2 to etcd3 migration is higher risk than most upgrades. The upgrade is disruptive to the masters, see notes above.
• Note that the upgrade for Calico users is disruptive, because it requires switching from direct-etcd-storage to CRD backed storage.

Full change list since 1.11.0 release

1.11.0 to 1.12.0-alpha.1

• machine-type generator: Warn if instance type not in ENI map @justinsb #6118
• Include name of unhealthy component in validation error @justinsb #6122
• Bump alpha channel kubernetes versions @justinsb #6123
• Add missing locking to awsmock LaunchConfigurations @justinsb #6124
• Add a1 and c5n instance types @justinsb #6117
• Simplify makefile for update-machine-types @justinsb #6121
• Update docs, removing brew --devel @mikesplain #6125
• machine-types: remove duplicate dedup @justinsb #6127
• Update amazon cni to 1.3.0 @mikesplain #6128
• Enable HPA tolerance configuration @rlees85 #6130
• Update addons dashboard version @jeefy #6136
• Spotinst: Bump controller image @liranp #6129
• Add cni to usage network option for kops create cluster @nak3 #6139
• Workspace updates for bazel / fix tests @mikesplain #6144
• Promote alpha channels to stable @mikesplain #6146
• Add GCE europe-north1-{a,b,c} @eetujalonen #6152
• Add self to security contacts @mikesplain #6147
• Fix missed stable channel upgrade path @mikesplain #6158
• Fix Calico upgrade job to use the correct version @tmjd #6156
• Fix for when node and master use the same SG. @rdrgmnzs #6175
• Add experimental and metrics flags for docker @rbtcollins #6171
• Add y flag for upgrade command for consistency @mikesplain #6177
• Add-ons spec example is missing "manifest". @qlikcoe #6170
• ExperimentalAllowedUnsafeSysctls has moved to AllowedUnsafeSysctls in k8s 1.11 @rdrgmnzs #6179
• Let a user specify the validation timeout when rotating a cluster. @rdrgmnzs #6185
• fix(docs): fix the compatibility matrics on hpa.md @Cryptophobia #6193
• bump prometheus-operator version and deploy file @zouyee #6196
• update heapster version and mark it retired @zouyee #6195
• Add Docker 18.06.1 for CentOS and RHEL 7 @bcorijn #6202
• Print --name with kops update cluster @joshbranham #6208
• Add --post-drain-delay to rolling-update cluster command @rifelpet #6211
• Adding kubernetes-dashboard v1.10.1 deployment to kops addons @schweizerbolzonello #6224
• Consider pending pods to be a validation failure @justinsb #6231
• Adding support for the new Stockholm region @liranp #6212
• Document how to update an existing vendored dependency @justinsb #6238
• Update to k8s 1.12 libraries @rdrgmnzs #5932
• Bump channels and bump alpha to latest @mikesplain #6239
• Automagically use curl instead of wget if that's what's available @eherot #6090
• cloudmock: replace unimplemented methods with interface embedding @justinsb #6243
• bazel: cleanup gobindata generation @justinsb #6235
• Update apimachinery for k8s 1.12 @justinsb #6245
• Bulk spelling fixes @justinsb #6242
• Don't panic when an etcd cluster is added @justinsb #6180
• Update aws-sdk-go to 1.16.9 @justinsb #6237
• Add p3dn.24xlarge @mikesplain #6253
• Rationalize deserialiation code @justinsb #6259
• Always log when a retry loop fails @justinsb #6260
• Update compatibility for v1.11.0 @mikesplain #6258
• AWS SDK v1.16.11 @gambol99 #6276
• nodeup: include underlying error in error message @andrestc #6279
• release process: add the relnotes command @justinsb #6269
• Fix missed error check in hasPlaceHolderIP @justinsb #6272
• Create dev-upload tasks, for a faster upload during dev builds @justinsb #6233
• Update recommended kubernetes version @justinsb #6271
• Release notes for 1.11 @justinsb #6270
• fixed the sentence mistake @abhijitio #6281
• update calico version to version 3.4.0 @chrisz100 #6263
• Remove duplicate Deployment for prometheus-operator @Smirl #6265
• Update aws-china.md @qqshfox #6262
• Recognize 2019 as a year @justinsb #6288
• Change jessie to stretch @abhijitio #6293
• Included type in SSL certificate documentation @walkafwalka #6289
• Update distroless @justinsb #6287
• Promote alpha kubernetes versions to stable @justinsb #6298
• Create prow-postsubmit target for release candidates @justinsb #6299
• Include windows build in distribution @justinsb #6300
• Fix kubelet api admin @gambol99 #6312
• GCE terraform: map source tags in firewallrule @justinsb #6295
• GCE terraform: support labels @justinsb #6296
• Add extra privilege to prometheus-k8s ClusterRole #6305
• Kubelet API RBAC Manifest @gambol99 #6317
• Upgrading coredns version to 1.3.0 @harshal-shah #6326
• Release 1.12.0-alpha.1 @justinsb #6257
• Retry Logging @gambol99 #6327
• Fix prow-postsubmit by copying prebuilt archive in bazel @justinsb #6328
• Remove Initializers from default admission plugins for 1.12+ @liggitt #6350
• include docker 18.06.1 missed dependency @nareshku #6338
• Fix alternative AWS partitions in custom instance profiles @rifelpet #6226
• Add doc regarding upgrading to CoreDNS @joshbranham #6344
• AWS: Enable ICMP Type 3 Code 4 for API server ELBs @davidarcher #6297
• Additional Storage & Volume Mounting @gambol99 #6066
• Kops for Openstack @jrperritt,@drekle,@wozniakjan,@marsavela #6351
• Update go version to 1.10.8 @justinsb #6401
• Suffix openstack subnet name with cluster name @wozniakjan #6380
• Update upgrade.md @ms4720 #6396
• minor grammar improvements to kops terraform docs @discdiver #6301
• Docs: Drop last DrainAndValidateRollingUpdate note @meeee #6374
• Allow users to set kubelet cpu-cfs-quota and cpu-cfs-quota-period flags @wndhydrnt #6375
• implement etcd status for openstack @zetaab #6381
• remove using deviceowner when filtering existing routerinterfaces @zetaab #6382
• ignore openstack managed volume tags @zetaab #6383
• kops version: Add --short flag, use it to get version in scripts @justinsb #6232
• find sshkey resource when updating cluster @zetaab #6384
• implement GetCloudGroups for openstack @zetaab #6386
• minor fixes to openstack @zetaab #6387
• fix openstack lb pool member logic @zetaab #6388
• Support "egress: External" to avoid configuring networking @justinsb,@cassandracomar,@moustafab #6218
• Bump alpha channels @mikesplain #6405
• Update bazel rules @mikesplain #6406
• implement delete cluster for openstack @zetaab #6385
• Openstack Floating IP Deletion @drekle #6425
• update openstack documentation @zetaab #6423
• Updated OWNERS file to include link to docs @rlenferink #6450
• [jjo] add docker-ce 18.06.2 for CVE-2019-5736 @jjo #6460
• Add permission for CreateTag on ENI to amazon-vpc-cni-k8s @nak3 #6389
• Document etcd3 migration process @justinsb #6408
• Normalize etcd cluster provider names @justinsb #6410
• Support etcd-manager v3, suitable for backporting @justinsb #6411
• Openstack loadbalancers erronous modification requests @drekle #6413
• fix typos for addon doc @fatsheep9146 #6416
• upgrade calico to 2.6.12 to fix TTA-2018-001 @mechpen #6422
• Use the forward plugin instead of proxy plugin in CoreDNS @rajansandeep #6424
• Update bazel workspace @mikesplain #6426
• Fix machine types and cleanup makefile @mikesplain #6427
• Add jessie patch @jjo,@mikesplain #6461
• Allow NodeAuthorizer to speak via HTTP Proxy if configured @KashifSaadat #6468
• Updated Canal manifest to v3.5.0 for k8s v1.12+ @KashifSaadat #6469
• Update document for GPU support @yujunz #6246
• Fixing kops-4049 @mmerrill3 #6210
• kube-apiserver: Add oidc-required-claim flag @jeyglk #6453
• add OWNERS file to openstack spesific folders @zetaab #6367
• Update Loadbalancer Pools @drekle #6433
• fix hostnames in kops openstack @zetaab #6442
• implement ig deletegroup for openstack @zetaab #6418
• Removing openstack credential file support @drekle #6480
• fix error when updating/creating lb in openstack @zetaab #6431
• recheck floatingip after server is active @zetaab #6432
• Ability to scale down instancegroup in openstack @zetaab #6421
• expose DryRunTarget changes and deletions @zetaab #6415
• support both octavia and old lbaasv2 api in openstack @zetaab #6438
• Guess SSH usernames for RHEL & Centos in toolbox dump @justinsb #6487
• Choose docker version 18.06.2 for k8s >= 1.12 @justinsb #6488
• Install kubelet config for default centos user @justinsb #6489
• Update the CoreDNS manifest @rajansandeep #6485
• docs: improve the queries for finding RHEL/CentOS images @justinsb #6486
• Workaround for overlay2 vs rhel-family docker bug @justinsb #6491
• retry l3floatingip list in fresh cluster @zetaab #6497
• Update 1.12 addon manifests to use apps/v1, rbac v1 @liggitt #6397
• Fix package name & version for container-selinux @justinsb #6492
• AWS Mixed Instances Policy / Fleet @gambol99 #6277
• Adding Comment @gambol99 #6508
• Kube Proxy Metrics Option @gambol99 #6513
• Sprig (Toolbox Templating) @gambol99 #6515
• Etcd memory and cpu requests @integrii #6313
• Map docker 18.06.3 @justinsb #6523
• Make docker 18.06.3 the default for k8s >= 1.12 @justinsb #6524
• Document strategy for cve_2019_5736 @justinsb #6522
• Try using chattr to mark docker-runc as immutable @justinsb #6506
• Simple mirror support @justinsb #6503
• Bump etcd-manager version to 3.0.20190224 @justinsb #6526
• update gophercloud vendor dependencies @zetaab #6478
• specify dns servers to openstack subnet @zetaab #6530
• possibility to specify floatingip subnet for resources in openstack @zetaab #6477
• Add Experimental Cluster Signing Duration flag @pgdagenais #6525
• set net.ipv4.ip_local_reserved_ports to the KubeAPIServer ServiceNodePortRange parameter on nodeup @sp-joseluis-ledesma #6343
• spread instances equally to all AZs @zetaab #6534
• update-machine-types: more metal instance types @justinsb #6551
• Add changelist for 1.11.1 @justinsb #6565
• Fix panic when using etcd-manager and resource requests are nil @KashifSaadat #6563
• Promote Kubernetes 1.11.7 to stable @olemarkus #6566
• Upgrade alpha to latest @mikesplain #6568
• implement delete instance, this is needed in rolling-update @zetaab #6576
• Stop setting deprecated --allow-privileged Kubelet flag in 1.14 @mtaufen #6340
• Openstack Security Group hardening @drekle #6521
• Update embargo doc link in SECURITY_CONTACTS and change PST to PSC @joelsmith #6601
• Instance LaunchConfig/Template Bug Fix @gambol99 #6590
• add docker.insecureRegistries flag @kimxogus #6586
• Add line breaks in example release cycle @MMeent #6591
• [jjo] Update Weave Net to version 2.5.1 @jjo #6370
• Adding installation guidelines for Windows @EchoDelta #6594
• Remove confusing comma in README @mattjmcnaughton #6607
• Add ServiceAccountKeyFile to KubeAPIServerConfig @Smirl #6578
• moving chrisz100 to approver level @chrisz100 #6434
• Fix dashboard yaml that returned 404 @mausch #6479
• Replace Y / N Markings of Compatibility Matrix in readme with ✔ / ❌ @compilenix #6539
• Rename addon.yml to addon.yaml @jsharpe #6323
• addons/cluster-autoscaler: Add jq installation for OSX environment @iBluemind #6567
• Update docs on authentication @flands #6575
• Omit IP-in-IP protocols in Openstack CNI Rules @marsavela #6614
• External out-of-tree CloudControllerManager support for openstack @zetaab #6444
• Use EnsureTask for create static pod directory @Smirl #6616
• Fix documentation about targetGroupArn key @phyrog #6611
• Update rolling_update.md @rj03hou,@Pharb #6247
• fix typo @zqm19941101 #6017
• Correcly handle CRLF in the manifest @gtrafimenkov #6570
• Add support for Docker 18.09.3. @tsuna #6347
• Fix confusing k8s upgrade docs for Terraform users @tspacek,@justinsb #6275
• Added Audit Webhook config @mbelangerupgrade,@jpbelangerupgrade #6361
• Spotinst: Avoid spurious changes @liranp #6028
• Fix amazon-vpc-routed-eni yaml template @tvi #6502
• Replace gcr.io URL with k8s.gcr.io vanity URL @justinsb #6623
• support gossip for AliCloud @LilyFaFa #6319
• add natGateways tasks for ALICloud @LilyFaFa #6402
• Fix some of the docker package names & versions @justinsb #6620
• Apply scope fix in #6502 to all manifest versions @tvi,@justinsb #6622
• Add --kubeconfig flag to kops export kubecfg @adamyy #5955
• add support to set cluster spec.kubelet @phedoreanu #6619
• Upgrade bazel gazelle @mikesplain #6609
• Fix typo @justinsb #6621
• Support g3s for gpu driver installation @reverson #6538
• Fix docker-healthcheck to work around Docker bug. @tsuna #6448
• docs: create checklist for new kubernetes version @justinsb #5818
• Fix metrics server addon @itskingori #6201
• Always create /var/lib/kubelet, even in bootstrap mode @justinsb #5982
• Launch Template Feature Flag @gambol99 #6512
• Remove docker-prestart hook @stevenjm #6564
• kops 1.12 configuration for calico: use CRDs @justinsb #6358
• Quick Clean @gambol99 #6634
• Sync data-types for webhook config with upstream @justinsb #6626
• Add manage security groups for loadbalancers @zetaab #6632
• Enable etcd-manager / etcd3 / etcd-tls in kops 1.12 @justinsb #6359
• Use EnsureTask for internal api route53 record @Smirl #6629
• Added reminder to publish conformance results in release process @chrisz100 #6640
• Update aws-china.md @qqshfox #6643
• Openstack server name collisions @drekle #6650
• tiny backslash arrangement @sevenfourk #6652
• Openstack environment escaping @drekle #6657
• Update upgrade.md @gamename #6654
• add ALI flags @LilyFaFa #6628
• Override volume zone name @zetaab #6655
• Updated Flannel manifest to 0.11.0 @gordonbondon #6660
• Update flannel version in bootstrapchannelbuilder @gordonbondon,@justinsb #6663
• Add flags for TLS Cipher suites customization for API Server, Kubelet and Controller-Manager @rochacon #6470
• If using etcd-backup and TLS is enabled, pass relevant options @KashifSaadat #6562
• Bump etcd-manager / etcd-backup to 3.0.20190325 @justinsb #6664
• 2048 - Add cloudLabels as tags to API ELB resource @ryan-dyer #6646
• Bump K8s 1.11 to 1.11.9 in the alpha channel @olemarkus #6665
• Upgrade rules go @mikesplain #6667
• Fix a missing dep lock @mikesplain #6668

1.12.0-alpha.1 to 1.12.0-alpha.2

• Support download protokube from mirror @justinsb #6673
• Promote alpha to stable and update alpha @mikesplain #6669
• Upload protokube to github as part of release @justinsb #6674
• Use CNI 0.7.5 @justinsb #6671
• Put 1.12 into stable channel, for users of kops 1.12-alphas @justinsb #6672
• Support mirrors with restricted characters @justinsb #6675

1.12.0-alpha.2 to 1.12.0-alpha.3

• Fix Key error change Overrides to Override @granular-ryanbonham #6691
• Add selector back to calico 1.12 deployment @justinsb #6682
• Update etcd-manager to 1.0.20190328 @justinsb #6695

1.12.0-alpha.3 to 1.12.0-beta.1

• Fix tagging and remove tagging elbs @mikesplain #6705
• Add DNS Resource Settings @granular-ryanbonham #6731
• Update instances types @mikesplain #6733
• Update kube-dns 1.3.0 to 1.3.3 @mikesplain #6734
• kube-dns-autoscaler: Add node watch to permissions @justinsb #6740
• Increase apiserver timeout to 45 seconds @justinsb #6743
• Fix issue #6700: User Data for launch templates & other terraform issues @rdrgmnzs #6732

1.12.0-beta.1 to 1.12.0-beta.2

• kube-dns: Update to 1.14.13 @justinsb #6741
• Launch Template use version number as well as name.@granular-ryanbonham #6755
• use dynamic s3 prefix in addAmazonVPCCNIPermissions func @bksteiny #6765

1.12.0-beta.2 to 1.12.0

• IAM Permission to Support Scaling from 0 with Lauch Templates @granular-ryanbonham #6861
• Avoid concurrent write corruption to /etc/hosts @justinsb,@granular-ryanbonham #6893
• Add i3en instance types @mikesplain #6898
• Add t3a family @mikesplain #6905
• Use existing SSHKeyName if no public key is created. @rralcala #6886
• bazel: fix distroless imports for latest bazel @justinsb #6910
• pkg/model: Fix dropped error @alrs #6911
• Add ability to specify cpuRequest for API Server @granular-ryanbonham #6706
• KubeAPIServer HTTP2 Stream Parameter @gambol99 #6913
• Add support for AWS ap-east-1 region @wxdao #6835
• Add min-resync-period for Controller Manager @maruina #6737
• Allow the AWS IAM Authenticator image name to be overridden @rifelpet #6730
• Add cpu management policy config @lynchc #5961
• Carry Provisioned IOPS to Terraform and CloudFormation templates @MathieuMailhos #6776
• update tolerations to openstack external cloud provider @zetaab #6821
• Fix typo in aws-iam-authenticator image field name @rifelpet #6840
• add the registry-qps kubelet flag @sp-joseluis-ledesma #6357
• Deep-copy proto state to prevent concurrent modification @justinsb #6707
• Publish utils.tar.gz to github releases also @justinsb #6680
• Allow uneven etcd zones @adammw #6641
• Add terraform support for additional CIDR blocks. @rdrgmnzs #6693
• Canal manifest updates for k8s v1.12+ @KashifSaadat #6823
• Update to etcd-manager 1.0.20190509 @justinsb #6917
• S3 VFS: Default to current region from metadata service @justinsb,@granular-ryanbonham #6943
• etcd-manager: Update to 3.0.20190513 @justinsb #6959
• Fix Docker not being installed on Ubuntu 16.04 @meeee #6965
• Issue #6945 @pkutishch,@mikesplain #6951

1.12.0 to 1.12.1

• Don't panic when deleting instancegroups @justinsb #7000
• etcd-manager: update to 3.0.20190516 @justinsb #7007
• Terraform: fix options field, should be spot_options @kimxogus #6988

1.12.1 to 1.12.2

• Mark ENI 0 as delete_on_termination for LaunchTemplates @granular-ryanbonham #7094

1.12.2 to 1.12.3

• Cherry pick of #7211: Use NodeAuthorizer config options instead of soely @jacksontj #7232
• Cherry pick of #7219: Make an actual deep-copy of the state @jacksontj #7235
• Upgrade Calico to 3.7.2 @asincu #7051
• Update canal to 3.6.4, for TTA-2019-002 @justinsb #7275
• Bumping calico to 3.7.4. @michalschott #7249
• Cherry pick of #7185: Replace behavior for aws hostnameOverride @jacksontj #7308
• Calico -> 3.7.4 for older versions @justinsb #7282
• Bump etcd-manager to 3.0.20190801 @justinsb #7349
• Warn/prevent if the version of etcd is unsupported with etcd-manager @justinsb #7340