kubespray自动化部署生产级别k8s集群

Deploy a Production Ready Kubernetes Cluster

Kubespray是Kubernetes incubator中的项目,目标是提供Production Ready Kubernetes部署方案,该项目基础是通过 Ansible-playbook 来定义系统与 Kubernetes 集群部署的任务

接上篇介绍,Terraform部署完k8s的所有虚拟机之后,接下来就是在deploy node上部署kubespray来自动化安装kubernetes.

在本篇教程中,我还将StorageClass和vSAN做联动,这样当应用申请PVC的时候,就会自动创建PV,并且将PV关联到vSAN上的vmfs.

具体环境如下:

ESXi HOST IP Remark
esxi-host01 10.0.0.11 #ESXi host01
esxi-host02 10.0.0.12 #ESXi host02
esxi-host03 10.0.0.13 #ESXi host03
esxi-host04 10.0.0.14 #ESXi host04
esxi-host05 10.0.0.15 #ESXi host05
vCenter 10.0.0.10 #vCenter
Terraform 10.0.0.5 #Terraform
k8s-deploy 10.0.0.9 #Deploy Node
k8s-master01 10.0.0.21 #Master Node
k8s-master02 10.0.0.22 #Master Node
k8s-master03 10.0.0.23 #Master Node
k8s-node01 10.0.0.31 #Workder Node
k8s-node02 10.0.0.32 #Workder Node
k8s-node03 10.0.0.33 #Workder Node
k8s-node04 10.0.0.34 #Workder Node
k8s-node05 10.0.0.35 #Workder Node

OS VERSION Kernel
Ubuntu 16.04 4.4.0-116

Components VERSION
kubernetes 1.9.5
etcd 3.2.4
calico 2.6.8
docker 17.03-ce

2.1 初始化Ubuntu

2.1.1 配置host表

[email protected]:~# cat /etc/hosts
127.0.0.1       localhost
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.0.0.21 k8s-master01
10.0.0.22 k8s-master02 
10.0.0.23 k8s-master03
10.0.0.31 k8s-node01 
10.0.0.32 k8s-node02 
10.0.0.33 k8s-node03 
10.0.0.34 k8s-node04 
10.0.0.35 k8s-node05

2.1.2 配置deploy节点免密登陆到k8s的所有节点

ssh-keygen
 ssh-copy-id -i k8s-master01
 ssh-copy-id -i k8s-master02
 ssh-copy-id -i k8s-master03
 ssh-copy-id -i k8s-node01
 ssh-copy-id -i k8s-node02
 ssh-copy-id -i k8s-node03
 ssh-copy-id -i k8s-node04
 ssh-copy-id -i k8s-node05

然后将deploy node SSH到每一台节点上,目的是提前将每台主机的指纹加入到deploy node的known_hosts里面

2.1.3 所有节点配置NTP

apt-get install ntp -y

ntp servers:
s1a.time.edu.cn
s1b.time.edu.cn
s1c.time.edu.cn
  
systemctl restart ntp

2.1.4 所有节点安装python

apt-get install python -y

2.2 在deploy node上安装ansible

apt-get install software-properties-common
apt-add-repository ppa:ansible/ansible
apt-get update
apt-get install ansible -y

安装完毕之后,请确保ansible是从PPA安装的,并且版本大于2.5.2。

[email protected]:~# ansible --version
ansible 2.5.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
 
[email protected]:~# ansible-playbook --version
ansible-playbook 2.5.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
[email protected]:~#

修改ansible默认的inventory

[email protected]:/opt# cat /etc/ansible/hosts 
[k8s-all]
k8s-master01
k8s-master02
k8s-master03
k8s-node01
k8s-node02
k8s-node03
k8s-node04
k8s-node05

[k8s-master]
k8s-master01
k8s-master02
k8s-master03


[k8s-node]
k8s-node01
k8s-node02
k8s-node03
k8s-node04
k8s-node05

最后测试一下

ansible all -m ping
ansible k8s-all -m copy -a 'src=/etc/hosts dest=/etc/hosts'
ansible k8s-all -m shell -a 'date'

2.3 在deploy node上安装kubespray并运行

kubespray的环境要求

  • Ansible v2.4
  • Jinja 2.9
  • Netaddr
  • Allow IPv4 forwarding
  • Ssh key must copied to inverotry
  • Access to internet
  • Memory SWAP is off
apt-get install python-pip -y
pip install --upgrade pip
hash -r
pip install jinja2
pip install netaddr
apt-get install python-netaddr -y
  
ansible all -m shell -a 'cat /proc/sys/net/ipv4/ip_forward'
ansible all -m shell -a 'echo 1 > /proc/sys/net/ipv4/ip_forward'
ansible all -m shell -a 'sysctl -p'
ansible all -m shell -a 'cat /proc/sys/net/ipv4/ip_forward'
ansible all -m shell -a 'swapoff -a'  #建议手动到每个节点上将/etc/fstab/的swap部分注释掉
ansible all -m shell -a 'free -h'

安装kubespray

git clone https://github.com/kubernetes-incubator/kubespray.git

修改默认的inventory

# Copy ``inventory/sample`` as ``inventory/mycluster``
cp -rfp inventory/sample inventory/mycluster

# Review and change parameters under ``inventory/mycluster/group_vars``
cat inventory/mycluster/group_vars/all.yml
cat inventory/mycluster/group_vars/k8s-cluster.yml

修改后的/inventory/mycluster/hosts.ini如下

k8s-master01 ansible_ssh_host=10.0.0.21
k8s-master02 ansible_ssh_host=10.0.0.22
k8s-master03 ansible_ssh_host=10.0.0.23
k8s-node01 ansible_ssh_host=10.0.0.31
k8s-node02 ansible_ssh_host=10.0.0.32
k8s-node03 ansible_ssh_host=10.0.0.33
k8s-node04 ansible_ssh_host=10.0.0.34
k8s-node05 ansible_ssh_host=10.0.0.35


[kube-master]
k8s-master01
k8s-master02
k8s-master03

[etcd]
k8s-master01
k8s-master02
k8s-master03

[kube-node]
k8s-node01
k8s-node02
k8s-node03
k8s-node04
k8s-node05

[kube-ingress]
k8s-node01
k8s-node02
k8s-node03
k8s-node04
k8s-node05

[k8s-cluster:children]
kube-master
kube-node
kube-ingress

修改后的/inventory/mycluster/group_vars/all.yml如下

# Valid bootstrap options (required): ubuntu, coreos, centos, none
bootstrap_os: none

#Directory where etcd data stored
etcd_data_dir: /var/lib/etcd

# Directory where the binaries will be installed
bin_dir: /usr/local/bin

## The access_ip variable is used to define how other nodes should access
## the node.  This is used in flannel to allow other flannel nodes to see
## this node for example.  The access_ip is really useful AWS and Google
## environments where the nodes are accessed remotely by the "public" ip,
## but don't know about that address themselves.
#access_ip: 1.1.1.1

### LOADBALANCING AND ACCESS MODES
## Enable multiaccess to configure etcd clients to access all of the etcd members directly
## as the "http://hostX:port, http://hostY:port, ..." and ignore the proxy loadbalancers.
## This may be the case if clients support and loadbalance multiple etcd servers natively.
#etcd_multiaccess: true

### ETCD: disable peer client cert authentication.
# This affects ETCD_PEER_CLIENT_CERT_AUTH variable
#etcd_peer_client_auth: true

## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
#loadbalancer_apiserver:
#  address: 1.2.3.4
#  port: 1234

## Internal loadbalancers for apiservers
#loadbalancer_apiserver_localhost: true

## Local loadbalancer should use this port instead, if defined.
## Defaults to kube_apiserver_port (6443)
#nginx_kube_apiserver_port: 8443

### OTHER OPTIONAL VARIABLES
## For some things, kubelet needs to load kernel modules.  For example, dynamic kernel services are needed
## for mounting persistent volumes into containers.  These may not be loaded by preinstall kubernetes
## processes.  For example, ceph and rbd backed volumes.  Set to true to allow kubelet to load kernel
## modules.
#kubelet_load_modules: false

## Internal network total size. This is the prefix of the
## entire network. Must be unused in your environment.
#kube_network_prefix: 18

## With calico it is possible to distributed routes with border routers of the datacenter.
## Warning : enabling router peering will disable calico's default behavior ('node mesh').
## The subnets of each nodes will be distributed by the datacenter router
#peer_with_router: false

## Upstream dns servers used by dnsmasq
#upstream_dns_servers:
#  - 8.8.8.8
#  - 8.8.4.4

## There are some changes specific to the cloud providers
## for instance we need to encapsulate packets with some network plugins
## If set the possible values are either 'gce', 'aws', 'azure', 'openstack', 'vsphere', or 'external'
## When openstack is used make sure to source in the openstack credentials
## like you would do when using nova-client before starting the playbook.
cloud_provider: vsphere

vsphere_vcenter_ip: "10.0.0.10"
vsphere_vcenter_port: 443
vsphere_insecure: 1
vsphere_user: "[email protected]"
vsphere_password: "CHANGEME"
vsphere_datacenter: "DataCenter"
vsphere_datastore: "vsanDatastore"
vsphere_working_dir: "k8s-nodes"
vsphere_scsi_controller_type: "pvscsi"
vsphere_resource_pool: ""k8s-cluster-RP"
#这些配置是为了StorageClass能够动态创建PV,k8s本身需要获取vCenter的权限

## When azure is used, you need to also set the following variables.
## see docs/azure.md for details on how to get these values
#azure_tenant_id:
#azure_subscription_id:
#azure_aad_client_id:
#azure_aad_client_secret:
#azure_resource_group:
#azure_location:
#azure_subnet_name:
#azure_security_group_name:
#azure_vnet_name:
#azure_vnet_resource_group:
#azure_route_table_name:

## When OpenStack is used, Cinder version can be explicitly specified if autodetection fails (Fixed in 1.9: https://github.com/kubernetes/kubernetes/issues/50461)
#openstack_blockstorage_version: "v1/v2/auto (default)"
## When OpenStack is used, if LBaaSv2 is available you can enable it with the following 2 variables.
#openstack_lbaas_enabled: True
#openstack_lbaas_subnet_id: "Neutron subnet ID (not network ID) to create LBaaS VIP"
## To enable automatic floating ip provisioning, specify a subnet.
#openstack_lbaas_floating_network_id: "Neutron network ID (not subnet ID) to get floating IP from, disabled by default"
## Override default LBaaS behavior
#openstack_lbaas_use_octavia: False
#openstack_lbaas_method: "ROUND_ROBIN"
#openstack_lbaas_provider: "haproxy"
#openstack_lbaas_create_monitor: "yes"
#openstack_lbaas_monitor_delay: "1m"
#openstack_lbaas_monitor_timeout: "30s"
#openstack_lbaas_monitor_max_retries: "3"

## Uncomment to enable experimental kubeadm deployment mode
#kubeadm_enabled: false
## Set these proxy values in order to update package manager and docker daemon to use proxies
#http_proxy: ""
#https_proxy: ""
## Refer to roles/kubespray-defaults/defaults/main.yml before modifying no_proxy
#no_proxy: ""

## Uncomment this if you want to force overlay/overlay2 as docker storage driver
## Please note that overlay2 is only supported on newer kernels
#docker_storage_options: -s overlay2

# Uncomment this if you have more than 3 nameservers, then we'll only use the first 3.
#docker_dns_servers_strict: false

## Default packages to install within the cluster, f.e:
#kpm_packages:
# - name: kube-system/grafana

## Certificate Management
## This setting determines whether certs are generated via scripts or whether a
## cluster of Hashicorp's Vault is started to issue certificates (using etcd
## as a backend). Options are "script" or "vault"
#cert_management: script

# Set to true to allow pre-checks to fail and continue deployment
#ignore_assert_errors: false

## Etcd auto compaction retention for mvcc key value store in hour
#etcd_compaction_retention: 0

## Set level of detail for etcd exported metrics, specify 'extensive' to include histogram metrics.
#etcd_metrics: basic

## Etcd is restricted by default to 512M on systems under 4GB RAM, 512MB is not enough for much more than testing.
## Set this if your etcd nodes have less than 4GB but you want more RAM for etcd. Set to 0 for unrestricted RAM.
#etcd_memory_limit: "512M"

# The read-only port for the Kubelet to serve on with no authentication/authorization. Uncomment to enable.
kube_read_only_port: 10255
# 开启这项的目的是可以让heapster可以读取kubelet的10255来取metric

修改后的/inventory/mycluster/group_vars/k8s-cluster.yml如下

# Kubernetes configuration dirs and system namespace.
# Those are where all the additional config stuff goes
# the kubernetes normally puts in /srv/kubernetes.
# This puts them in a sane location and namespace.
# Editing those values will almost surely break something.
kube_config_dir: /etc/kubernetes
kube_script_dir: "{{ bin_dir }}/kubernetes-scripts"
kube_manifest_dir: "{{ kube_config_dir }}/manifests"

# This is where all the cert scripts and certs will be located
kube_cert_dir: "{{ kube_config_dir }}/ssl"

# This is where all of the bearer tokens will be stored
kube_token_dir: "{{ kube_config_dir }}/tokens"

# This is where to save basic auth file
kube_users_dir: "{{ kube_config_dir }}/users"

kube_api_anonymous_auth: true

## Change this to use another Kubernetes version, e.g. a current beta release
kube_version: v1.9.5

# Where the binaries will be downloaded.
# Note: ensure that you've enough disk space (about 1G)
local_release_dir: "/tmp/releases"
# Random shifts for retrying failed ops like pushing/downloading
retry_stagger: 5

# This is the group that the cert creation scripts chgrp the
# cert files to. Not really changeable...
kube_cert_group: kube-cert

# Cluster Loglevel configuration
kube_log_level: 2

# Users to create for basic auth in Kubernetes API via HTTP
# Optionally add groups for user
kube_api_pwd: "{{ lookup('password', inventory_dir + '/credentials/kube_user.creds length=15 chars=ascii_letters,digits') }}"
kube_users:
  kube:
    pass: "{{kube_api_pwd}}"
    role: admin
    groups:
      - system:masters

## It is possible to activate / deactivate selected authentication methods (basic auth, static token auth)
#kube_oidc_auth: false
#kube_basic_auth: false
#kube_token_auth: false


## Variables for OpenID Connect Configuration https://kubernetes.io/docs/admin/authentication/
## To use OpenID you have to deploy additional an OpenID Provider (e.g Dex, Keycloak, ...)

# kube_oidc_url: https:// ...
# kube_oidc_client_id: kubernetes
## Optional settings for OIDC
# kube_oidc_ca_file: {{ kube_cert_dir }}/ca.pem
# kube_oidc_username_claim: sub
# kube_oidc_groups_claim: groups


# Choose network plugin (cilium, calico, contiv, weave or flannel)
# Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
kube_network_plugin: calico

# weave's network password for encryption
# if null then no network encryption
# you can use --extra-vars to pass the password in command line
weave_password: EnterPasswordHere

# Weave uses consensus mode by default
# Enabling seed mode allow to dynamically add or remove hosts
# https://www.weave.works/docs/net/latest/ipam/
weave_mode_seed: false

# This two variable are automatically changed by the weave's role, do not manually change these values
# To reset values :
# weave_seed: uninitialized
# weave_peers: uninitialized
weave_seed: uninitialized
weave_peers: uninitialized

# Set the MTU of Weave (default 1376, Jumbo Frames: 8916)
weave_mtu: 1376

# Enable kubernetes network policies
enable_network_policy: false

# Kubernetes internal network for services, unused block of space.
kube_service_addresses: 10.233.0.0/18

# internal network. When used, it will assign IP
# addresses from this range to individual pods.
# This network must be unused in your network infrastructure!
kube_pods_subnet: 10.233.64.0/18

# internal network node size allocation (optional). This is the size allocated
# to each node on your network.  With these defaults you should have
# room for 4096 nodes with 254 pods per node.
kube_network_node_prefix: 24

# The port the API Server will be listening on.
kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
kube_apiserver_port: 6443 # (https)
kube_apiserver_insecure_port: 8080 # (http)
# Set to 0 to disable insecure port - Requires RBAC in authorization_modes and kube_api_anonymous_auth: true
#kube_apiserver_insecure_port: 0 # (disabled)

# Kube-proxy proxyMode configuration.
# Can be ipvs, iptables
kube_proxy_mode: ipvs
# 这里我修改了默认的iptables为ipvs

## Encrypting Secret Data at Rest (experimental)
kube_encrypt_secret_data: false

# DNS configuration.
# Kubernetes cluster name, also will be used as DNS domain
cluster_name: cluster.local
# Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
ndots: 2
# Can be dnsmasq_kubedns, kubedns, coredns, coredns_dual, manual or none
dns_mode: coredns
# 这里我修改了默认的kubedns为coredns
# Set manual server if using a custom cluster DNS server
#manual_dns_server: 10.x.x.x

# Can be docker_dns, host_resolvconf or none
resolvconf_mode: docker_dns
# Deploy netchecker app to verify DNS resolve as an HTTP service
deploy_netchecker: false
# Ip address of the kubernetes skydns service
skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
skydns_server_secondary: "{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}"
dnsmasq_dns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(2)|ipaddr('address') }}"
dns_domain: "{{ cluster_name }}"

# Path used to store Docker data
docker_daemon_graph: "/var/lib/docker"

## A string of extra options to pass to the docker daemon.
## This string should be exactly as you wish it to appear.
## An obvious use case is allowing insecure-registry access
## to self hosted registries like so:

docker_options: "--insecure-registry={{ kube_service_addresses }} --graph={{ docker_daemon_graph }}  {{ docker_log_opts }}"
docker_bin_dir: "/usr/bin"

# Settings for containerized control plane (etcd/kubelet/secrets)
etcd_deployment_type: docker
kubelet_deployment_type: host
vault_deployment_type: docker
helm_deployment_type: host

# K8s image pull policy (imagePullPolicy)
k8s_image_pull_policy: IfNotPresent

# Kubernetes dashboard
# RBAC required. see docs/getting-started.md for access details.
dashboard_enabled: true

# Monitoring apps for k8s
efk_enabled: false

# Helm deployment
helm_enabled: false

# Istio deployment
istio_enabled: false

# Registry deployment
registry_enabled: false
# registry_namespace: "{{ system_namespace }}"
# registry_storage_class: ""
# registry_disk_size: "10Gi"

# Local volume provisioner deployment
local_volume_provisioner_enabled: false
# local_volume_provisioner_namespace: "{{ system_namespace }}"
# local_volume_provisioner_base_dir: /mnt/disks
# local_volume_provisioner_mount_dir: /mnt/disks
# local_volume_provisioner_storage_class: local-storage

# CephFS provisioner deployment
cephfs_provisioner_enabled: false
# cephfs_provisioner_namespace: "{{ system_namespace }}"
# cephfs_provisioner_cluster: ceph
# cephfs_provisioner_monitors:
#   - 172.24.0.1:6789
#   - 172.24.0.2:6789
#   - 172.24.0.3:6789
# cephfs_provisioner_admin_id: admin
# cephfs_provisioner_secret: secret
# cephfs_provisioner_storage_class: cephfs

# Nginx ingress controller deployment
ingress_nginx_enabled: true
# 将ingress打开
# ingress_nginx_host_network: false
# ingress_nginx_namespace: "ingress-nginx"
# ingress_nginx_insecure_port: 80
# ingress_nginx_secure_port: 443
# ingress_nginx_configmap:
#   map-hash-bucket-size: "128"
#   ssl-protocols: "SSLv2"
# ingress_nginx_configmap_tcp_services:
#   9000: "default/example-go:8080"
# ingress_nginx_configmap_udp_services:
#   53: "kube-system/kube-dns:53"

# Cert manager deployment
cert_manager_enabled: false
# cert_manager_namespace: "cert-manager"

# Add Persistent Volumes Storage Class for corresponding cloud provider ( OpenStack is only supported now )
persistent_volumes_enabled: false

# Make a copy of kubeconfig on the host that runs Ansible in {{ inventory_dir }}/artifacts
# kubeconfig_localhost: false
# Download kubectl onto the host that runs Ansible in {{ bin_dir }}
# kubectl_localhost: false

# dnsmasq
# dnsmasq_upstream_dns_servers:
#  - /resolvethiszone.with/10.0.4.250
#  - 8.8.8.8

#  Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (default true)
# kubelet_cgroups_per_qos: true

# A comma separated list of levels of node allocatable enforcement to be enforced by kubelet.
# Acceptable options are 'pods', 'system-reserved', 'kube-reserved' and ''. Default is "".
# kubelet_enforce_node_allocatable: pods

## Supplementary addresses that can be added in kubernetes ssl keys.
## That can be useful for example to setup a keepalived virtual IP
# supplementary_addresses_in_ssl_keys: [10.0.0.1, 10.0.0.2, 10.0.0.3]

## Running on top of openstack vms with cinder enabled may lead to unschedulable pods due to NoVolumeZoneConflict restriction in kube-scheduler.
## See https://github.com/kubernetes-incubator/kubespray/issues/2141
## Set this variable to true to get rid of this issue
volume_cross_zone_attachment: false

修改所有的grc.io和quay.io的image,我已经将这些image镜像到我的docker hub的仓库上。 在/kubespray目录下面创建change_registry.sh

#!/bin/bash
all_image_files=(
roles/download/defaults/main.yml
roles/kubernetes-apps/ansible/defaults/main.yml
)

for file in ${all_image_files[@]} ; do
    sed -i 's/gcr.io\/google-containers\//ottodeng\/gcr.io_google-containers_/g' $file
    sed -i 's/gcr.io\/google_containers\//ottodeng\/gcr.io_google_containers_/g' $file
    sed -i 's/quay.io\/coreos\//ottodeng\/quay.io_coreos_/g' $file
    sed -i 's/quay.io\/calico\//ottodeng\/quay.io_calico_/g' $file
    sed -i 's/quay.io\/external_storage\//ottodeng\/quay.io_external_storage_/g' $file
    sed -i 's/quay.io\/kubespray\//ottodeng\/quay.io_kubespray_/g' $file
    sed -i 's/quay.io\/kubernetes-ingress-controller\//ottodeng\/quay.io_kubernetes-ingress-controller_/g' $file
done
chmod +x change_registry.sh
./change_registry.sh

执行完毕之后,可以查看一下修改后的内容,确保已经修改成功

grep -irn "grc.io"
grep -irn "quay.io"

给kubespray的执行结果输出日志

cd ~/kubespray
echo 'log_path = /var/log/ansible.log' > ansible.cfg

开始执行kubespray

cd ~/kubespray
ansible-playbook -i inventory/mycluster/hosts.ini cluster.yml

当执行完毕之后,大概的样子如下图所示:

FINISH

如果没有failed,则证明已经部署成功,如果有failed,则可以查看/var/log/ansible.log的失败日志。

2.4 确认k8s集群运行正常,并且能够通过StorageClass动态创建PV

[email protected]:~# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-1               Healthy   {"health": "true"}  
etcd-2               Healthy   {"health": "true"}  
etcd-0               Healthy   {"health": "true"}  
[email protected]:~# kubectl get node
NAME             STATUS    ROLES          AGE       VERSION
k8s-master01   Ready     master         7m        v1.9.5
k8s-master02   Ready     master         7m        v1.9.5
k8s-master03   Ready     master         7m        v1.9.5
k8s-node01     Ready     ingress,node   7m        v1.9.5
k8s-node02     Ready     ingress,node   7m        v1.9.5
k8s-node03     Ready     ingress,node   7m        v1.9.5
k8s-node04     Ready     ingress,node   7m        v1.9.5
k8s-node05     Ready     ingress,node   7m        v1.9.5
[email protected]:~# kubectl cluster-info
Kubernetes master is running at https://10.0.0.21:6443
CoreDNS is running at https://10.0.0.21:6443/api/v1/namespaces/kube-system/services/coredns:dns/proxy
 
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
  
[email protected]:~# kubectl get pod --all-namespaces -o wide
NAMESPACE       NAME                                       READY     STATUS    RESTARTS   AGE       IP                NODE
ingress-nginx   ingress-nginx-controller-5pssh             1/1       Running   0          5m        10.233.90.129    k8s-node04
ingress-nginx   ingress-nginx-controller-76g64             1/1       Running   0          5m        10.233.64.65     k8s-node03
ingress-nginx   ingress-nginx-controller-9cxg8             1/1       Running   0          5m        10.233.97.66     k8s-node05
ingress-nginx   ingress-nginx-controller-z6qv6             1/1       Running   0          5m        10.233.121.129   k8s-node02
ingress-nginx   ingress-nginx-controller-z9bxw             1/1       Running   0          5m        10.233.65.65     k8s-node01
ingress-nginx   ingress-nginx-default-backend-v1.4-k2jr4   1/1       Running   0          5m        10.233.97.65     k8s-node05
kube-system     calico-node-7tcgx                          1/1       Running   0          5m        10.0.0.21      k8s-master01
kube-system     calico-node-cspmf                          1/1       Running   0          5m        10.0.0.22      k8s-master02
kube-system     calico-node-dqtxd                          1/1       Running   0          5m        10.0.0.32      k8s-node02
kube-system     calico-node-j46p2                          1/1       Running   0          5m        10.0.0.35      k8s-node05
kube-system     calico-node-mqxd9                          1/1       Running   0          5m        10.0.0.34      k8s-node04
kube-system     calico-node-nrng6                          1/1       Running   0          5m        10.0.0.23      k8s-master03
kube-system     calico-node-psk2q                          1/1       Running   0          5m        10.0.0.31      k8s-node01
kube-system     calico-node-s6thl                          1/1       Running   0          5m        10.0.0.33      k8s-node03
kube-system     coredns-7dfc584b6c-m4bs5                   1/1       Running   0          5m        10.233.121.130   k8s-node02
kube-system     coredns-7dfc584b6c-trwxp                   1/1       Running   0          5m        10.233.90.130    k8s-node04
kube-system     kube-apiserver-k8s-master01              1/1       Running   0          5m        10.0.0.21      k8s-master01
kube-system     kube-apiserver-k8s-master02              1/1       Running   0          5m        10.0.0.22      k8s-master02
kube-system     kube-apiserver-k8s-master03              1/1       Running   0          5m        10.0.0.23      k8s-master03
kube-system     kube-controller-manager-k8s-master01     1/1       Running   0          6m        10.0.0.21      k8s-master01
kube-system     kube-controller-manager-k8s-master02     1/1       Running   0          6m        10.0.0.22      k8s-master02
kube-system     kube-controller-manager-k8s-master03     1/1       Running   0          6m        10.0.0.23      k8s-master03
kube-system     kube-proxy-k8s-master01                  1/1       Running   0          5m        10.0.0.21      k8s-master01
kube-system     kube-proxy-k8s-master02                  1/1       Running   0          5m        10.0.0.22      k8s-master02
kube-system     kube-proxy-k8s-master03                  1/1       Running   0          5m        10.0.0.23      k8s-master03
kube-system     kube-proxy-k8s-node01                    1/1       Running   0          5m        10.0.0.31      k8s-node01
kube-system     kube-proxy-k8s-node02                    1/1       Running   0          6m        10.0.0.32      k8s-node02
kube-system     kube-proxy-k8s-node03                    1/1       Running   0          5m        10.0.0.33      k8s-node03
kube-system     kube-proxy-k8s-node04                    1/1       Running   0          6m        10.0.0.34      k8s-node04
kube-system     kube-proxy-k8s-node05                    1/1       Running   0          5m        10.0.0.35      k8s-node05
kube-system     kube-scheduler-k8s-master01              1/1       Running   0          6m        10.0.0.21      k8s-master01
kube-system     kube-scheduler-k8s-master02              1/1       Running   0          6m        10.0.0.22      k8s-master02
kube-system     kube-scheduler-k8s-master03              1/1       Running   0          6m        10.0.0.23      k8s-master03
kube-system     kubernetes-dashboard-66fc89bd85-k977x      1/1       Running   0          5m        10.233.121.131   k8s-node02
kube-system     nginx-proxy-k8s-node01                   1/1       Running   0          6m        10.0.0.31      k8s-node01
kube-system     nginx-proxy-k8s-node02                   1/1       Running   0          5m        10.0.0.32      k8s-node02
kube-system     nginx-proxy-k8s-node03                   1/1       Running   0          5m        10.0.0.33      k8s-node03
kube-system     nginx-proxy-k8s-node04                   1/1       Running   0          5m        10.0.0.34      k8s-node04
kube-system     nginx-proxy-k8s-node05                   1/1       Running   0          5m        10.0.0.35      k8s-node05
[email protected]:~#

[email protected]:~# calicoctl node status
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+------------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |   SINCE    |    INFO     |
+--------------+-------------------+-------+------------+-------------+
| 10.0.0.22    | node-to-node mesh | up    | 2018-05-07 | Established |
| 10.0.0.23    | node-to-node mesh | up    | 2018-05-07 | Established |
| 10.0.0.31    | node-to-node mesh | up    | 2018-05-07 | Established |
| 10.0.0.32    | node-to-node mesh | up    | 2018-05-07 | Established |
| 10.0.0.33    | node-to-node mesh | up    | 2018-05-07 | Established |
| 10.0.0.34    | node-to-node mesh | up    | 2018-05-07 | Established |
| 10.0.0.35    | node-to-node mesh | up    | 2018-05-07 | Established |
+--------------+-------------------+-------+------------+-------------+

IPv6 BGP status
No IPv6 peers found.

创建StorageClass和PV

cat vsphere-volume-sc-fast.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: 
  name: fast
provisioner: kubernetes.io/vsphere-volume
parameters: 
  datastore: vsanDatastore
  diskformat: thin
  fstype: ext3
cat vsphere-volume-pvcsc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvcsc001
  annotations:
    volume.beta.kubernetes.io/storage-class: fast
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
[email protected]:~# kubectl get pvc
NAME       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvcsc001   Bound     pvc-5adb155b-4f7c-11e8-aaf4-00505698233c   2Gi        RWO            fast           4s
[email protected]:~# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM              STORAGECLASS   REASON    AGE
pvc-5adb155b-4f7c-11e8-aaf4-00505698233c   2Gi        RWO            Delete           Bound     default/pvcsc001   fast                     7s
[email protected]:~#

可以查看到pvc已经正常的Bound到PV

在大型集群部署中,请注意提高master组件的pod limit或者直接取消limit,具体可以在role中找到对应的limit的输入值

详细项目请参考