编辑和创建接口被限制（开发一个禁止删除namespace的控制器）
编辑和创建接口被限制（开发一个禁止删除namespace的控制器）当 API 请求进入时，mutating 和 validating 控制器使用配置中的外部 webhooks 列表并发调用，规则如下：整个准入控制器的流程如下：准入控制器（Admission Controller）位于 API Server 中，在对象被持久化之前，准入控制器拦截对 API Server 的请求，一般用来做身份验证和授权。其中包含两个特殊的控制器：MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook。MutatingAdmissionWebhook ：用于变更请求对象，比如istio为每个Pod注入sidecar，就是通过它实现。ValidatingAdmissionWebhook：用于验证请求对象
大家好，我是乔克。
昨天收到一个朋友的信息，说不小心把集群的业务namespace干掉了，导致整个业务都停滞了，问我有没有禁止删除namespace的方案。
在我的记忆里，Kubernetes的准入里并没有这个控制器，所以我就给他说需要自己开发一个准入控制器来实现自己的目标。
作为人，何为正确！我不能只脱裤子，不放屁。所以这里也整理了一下如何自定义Kubernetes的准入控制器。
准入控制器（Admission Controller）位于 API Server 中，在对象被持久化之前，准入控制器拦截对 API Server 的请求，一般用来做身份验证和授权。其中包含两个特殊的控制器：MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook。
-
MutatingAdmissionWebhook ：用于变更请求对象，比如istio为每个Pod注入sidecar，就是通过它实现。
-
ValidatingAdmissionWebhook：用于验证请求对象
整个准入控制器的流程如下：
当 API 请求进入时，mutating 和 validating 控制器使用配置中的外部 webhooks 列表并发调用，规则如下：
-
如果所有的 webhooks 批准请求，准入控制链继续流转。
-
如果有任意一个 webhooks 阻止请求，那么准入控制请求终止，并返回第一个 webhook 阻止的原因。其中，多个 webhooks 阻止也只会返回第一个 webhook 阻止的原因。
-
如果在调用 webhook 过程中发生错误，那么请求会被终止或者忽略 webhook。
准入控制器是在 API Server 的启动参数中配置的。一个准入控制器可能属于以上两者中的一种，也可能两者都属于。
我们在部署 Kubernetes 集群的时候都会默认开启一系列准入控制器，如果没有设置这些准入控制器的话可以说你的 Kubernetes 集群就是在裸奔，应该叫管理员为集群添加准入控制器。
代码实现实现逻辑
在开发之前先大致了解一下准入控制器的Webhook的大致实现逻辑：
-
Webhook是一个标准的HTTP服务，接收HTTP请求
-
接收到的请求是一个AdmissionReview对象
-
然后我们自定义的Hook会处理这个AdmissionReview对象
-
处理完过后再返回一个AdmissionReview对象，这里面会包含处理结果
AdmissionReview的结构体如下：
// AdmissionReview describes an admission review Request/response.
type AdmissionReview struct {
metav1.TypeMeta `json:" inline"`
// Request describes the attributes for the admission request.
// optional
Request *AdmissionRequest `json:"request omitempty" protobuf:"bytes 1 opt name=request"`
// Response describes the attributes for the admission response.
// optional
Response *AdmissionResponse `json:"response omitempty" protobuf:"bytes 2 opt name=response"`
}
从代码的命名中可以很清晰的看出，在请求发送到 WebHook 时我们只需要关注内部的 AdmissionRequest(实际入参)，在我们编写的 WebHook 处理完成后只需要返回包含有 AdmissionResponse(实际返回体) 的 AdmissionReview 对象即可；总的来说 AdmissionReview 对象是个套壳，请求是里面的 AdmissionRequest，响应是里面的 AdmissionResponse。
具体实现
（1）首先创建一个HTTP Server，监听端口，接收请求
package main
import (
"context"
"flag"
"github.com/joker-bai/validate-namespace/http"
log "k8s.io/klog/v2"
"os"
"os/signal"
"syscall"
)
var (
tlscert tlskey port String
)
funcmain {
flag.StringVar(&tlscert "tlscert" "/etc/certs/cert.pem" "Path to the TLS Certificate")
flag.StringVar(&tlskey "tlskey" "/etc/certs/key.pem" "Path to the TLS key")
flag.StringVar(&port "port" "8443" "The port to listen")
flag.Parse
server := http.NewServer(port)
gofunc {
if err := server.ListenAndServeTLS(tlscert tlskey); err != nil {
log.errorf("Failed to listen and serve: %v" err)
}
}
log.Infof("Server running in port: %s" port)
// listen shutdown signal
signalChan := make(chan os.Signal 1)
signal.Notify(signalChan syscall.SIGINT syscall.SIGTERM)
<-signalChan
log.Info("Shutdown gracefully...")
if err := server.Shutdown(context.Background); err != nil {
log.Error(err)
}
}
由于准入控制器和Webhook之间需要使用TLS进行通信，所以上面监听的端口是TLS端口，通过server.ListenAndServeTLS实现，后续在部署服务的时候需要把证书挂到相应的目录中。
（2）定义Handler，将请求分发到具体的处理方法
package http
import (
"fmt"
"github.com/joker-bai/validate-namespace/namespace"
"net/http"
)
// NewServer creates and return a http.Server
func NewServer(port string) *http.Server {
// Instances hooks
nsValidation := namespace.NewValidationHook
// Routers
ah := newAdmissionHandler
mux := http.NewServeMux
mux.Handle("/healthz" healthz)
mux.Handle("/validate/delete-namespace" ah.Serve(nsValidation))
return &http.Server{
Addr: fmt.Sprintf(":%s" port)
Handler: mux
}
}
实现admissionHandler，主要作用是将http body的内容解析成AdmissionReview对象，然后调用具体的Hook处理，再将结果放到AdmissionReview中，返回给客户端。
package http
import (
"encoding/json"
"fmt"
"io"
"net/http"
"github.com/douglasmakey/admissioncontroller"
"k8s.io/api/admission/v1beta1"
admission "k8s.io/api/admission/v1beta1"
meta "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/serializer"
log "k8s.io/klog/v2"
)
// admissionHandler represents the HTTP handler for an admission webhook
type admissionHandler struct {
decoder runtime.Decoder
}
// newAdmissionHandler returns an instance of AdmissionHandler
func newAdmissionHandler *admissionHandler {
return &admissionHandler{
decoder: serializer.NewCodecFactory(runtime.NewScheme).UniversalDeserializer
}
}
// Serve returns a http.HandlerFunc for an admission webhook
func (h *admissionHandler) Serve(hook admissioncontroller.Hook) http.HandlerFunc {
returnfunc(w http.ResponseWriter r *http.Request) {
w.Header.Set("Content-Type" "application/json")
if r.Method != http.MethodPost {
http.Error(w fmt.Sprint("invalid method only POST requests are allowed") http.StatusMethodNotAllowed)
return
}
if contentType := r.Header.Get("Content-Type"); contentType != "application/json" {
http.Error(w fmt.Sprint("only content type 'application/json' is supported") http.StatusBadRequest)
return
}
body err := io.ReadAll(r.Body)
if err != nil {
http.Error(w fmt.Sprintf("could not read request body: %v" err) http.StatusBadRequest)
return
}
var review admission.AdmissionReview
if _ _ err := h.decoder.Decode(body nil &review); err != nil {
http.Error(w fmt.Sprintf("could not deserialize request: %v" err) http.StatusBadRequest)
return
}
if review.Request == nil {
http.Error(w "malformed admission review: request is nil" http.StatusBadRequest)
return
}
result err := hook.Execute(review.Request)
if err != nil {
log.Error(err)
w.WriteHeader(http.StatusInternalServerError)
return
}
admissionResponse := v1beta1.AdmissionReview{
Response: &v1beta1.AdmissionResponse{
UID: review.Request.UID
Allowed: result.Allowed
Result: &meta.Status{Message: result.Msg}
}
}
res err := json.Marshal(admissionResponse)
if err != nil {
log.Error(err)
http.Error(w fmt.Sprintf("could not marshal response: %v" err) http.StatusInternalServerError)
return
}
log.Infof("Webhook [%s - %s] - Allowed: %t" r.URL.Path review.Request.Operation result.Allowed)
w.WriteHeader(http.StatusOK)
w.Write(res)
}
}
func healthz http.HandlerFunc {
returnfunc(w http.ResponseWriter r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write(byte("ok"))
}
}
上面处理是通过hook.Execute来处理请求，这是admissionController内部实现的一个结构体，它为每个操作定义了一个方法，如下：
// AdmitFunc defines how to process an admission request
type AdmitFuncfunc(request *admission.AdmissionRequest) (*Result error)
// Hook represents the set of functions for each operation in an admission webhook.
type Hook struct {
Create AdmitFunc
Delete AdmitFunc
Update AdmitFunc
Connect AdmitFunc
}
我们就需要实现具体的AdmitFunc，并注册。
（3）将自己实现的方法注册到Hook中。
package namespace
import (
"github.com/douglasmakey/admissioncontroller"
)
// NewValidationHook delete namespace validation hook
func NewValidationHook admissioncontroller.Hook {
return admissioncontroller.Hook{
Delete: validateDelete
}
}
（4）实现具体的AdmitFunc
package namespace
import (
"github.com/douglasmakey/admissioncontroller"
log "k8s.io/klog/v2"
"k8s.io/api/admission/v1beta1"
)
func validateDelete admissioncontroller.AdmitFunc {
returnfunc(r *v1beta1.AdmissionRequest) (*admissioncontroller.Result error) {
if r.Kind.Kind == "Namespace" {
log.Info("You cannot delete namespace: " r.Name)
return &admissioncontroller.Result{Allowed: false} nil
} else {
return &admissioncontroller.Result{Allowed: true} nil
}
}
}
这里实现很简单，如果Kind为Namespace，就拒绝操作。
部署测试上面完成了业务逻辑开发，下面就把它部署到Kubernetes集群测试一番。
部署
（1）编写Dockerfile，将应用打包成镜像
FROM golang:1.17.5 AS build-env
ENV GOPROXY https://goproxy.cn
ADD . /go/src/app
WORKDIR /go/src/app
RUN go mod tidy
RUN cd cmd && GOOS=linux GOARCH=amd64 go build -v -a -ldflags '-extldflags "-static"' -o /go/src/app/app-server /go/src/app/cmd/main.go
FROM registry.cn-hangzhou.aliyuncs.com/coolops/ubuntu:22.04
ENV TZ=Asia/Shanghai
COPY --from=build-env /go/src/app/app-server /opt/app-server
WORKDIR /opt
EXPOSE 80
CMD [ "./app-server" ]
（2）创建TLS证书，使用脚本进行创建
#!/bin/bash
set -e
usage {
cat <<EOF
Generate certificate suitable for use with an sidecar-injector webhook service.
This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with sidecar-injector webhook
services. This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.
The server key/cert k8s CA cert are stored in a k8s secret.
usage: ${0} [OPTIONS]
The following flags are required.
--service Service name of webhook.
--namespace Namespace where webhook service and secret reside.
--secret Secret name for CA certificate and server certificate/key pair.
EOF
exit 1
}
while [[ $# -gt 0 ]]; do
case ${1} in
--service)
service="$2"
shift
;;
--secret)
secret="$2"
shift
;;
--namespace)
namespace="$2"
shift
;;
*)
usage
;;
esac
shift
done
[ -z ${service} ] && service=validate-delete-namespace
[ -z ${secret} ] && secret=validate-delete-namespace-tls
[ -z ${namespace} ] && namespace=default
if [ ! -x "$(command -v openssl)" ]; then
echo "openssl not found"
exit 1
fi
csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "
cat <<EOF >> ${tmpdir}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation digitalSignature keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF
openssl genrsa -out ${tmpdir}/server-key.pem 2048
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
# clean-up any previously created CSR for our service. Ignore errors if not present.
kubectl delete csr ${csrName} 2>/dev/ || true
# create server cert/key CSR and send to k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: ${csrName}
spec:
groups:
- system:authenticated
request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
# verify CSR has been created
while true; do
kubectl get csr ${csrName}
if [ "$?" -eq 0 ]; then
break
fi
done
# approve and fetch the signed certificate
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
if [[ ${serverCert} != '' ]]; then
break
fi
sleep 1
done
if [[ ${serverCert} == '' ]]; then
echo "ERROR: After approving csr ${csrName} the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
exit 1
fi
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem
# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
--from-file=key.pem=${tmpdir}/server-key.pem \
--from-file=cert.pem=${tmpdir}/server-cert.pem \
--dry-run -o yaml |
kubectl -n ${namespace} apply -f -
（3）编写Deployment部署服务
apiVersion: apps/v1
kind: Deployment
metadata:
name: validate-delete-namespace
labels:
app: validate-delete-namespace
spec:
replicas: 1
selector:
matchLabels:
app: validate-delete-namespace
template:
metadata:
labels:
app: validate-delete-namespace
spec:
containers:
- name: server
image: registry.cn-hangzhou.aliyuncs.com/coolops/validate-delete-namespace:latest
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /healthz
port: 8443
scheme: HTTPS
ports:
- containerPort: 8443
volumeMounts:
- name: tls-certs
mountPath: /etc/certs
readOnly:true
volumes:
- name: tls-certs
secret:
secretName: validate-delete-namespace-tls
---
apiVersion: v1
kind: Service
metadata:
name: validate-delete-namespace
spec:
selector:
app: validate-delete-namespace
ports:
- port: 443
targetPort: 8443
（4）部署Webhook
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
name: validate-delete-namespace
webhooks:
- name: validate-delete-namespace.default.svc.cluster.local
clientConfig:
service:
namespace: default
name: validate-delete-namespace
path: "/validate/delete-namespace"
caBundle: "${CA_BUNDLE}"
rules:
- operations:
- DELETE
apiGroups:
- ""
apiVersions:
- "v1"
resources:
- namespaces
failurePolicy: Ignore
这里有一个${CA_BUNDLE}占位符，在创建Webhook的时候要将其替换掉，使用如下命令：
cat ./validate-delete-namespace.yaml | sh ./patch-webhook-ca.sh > ./webhook.yaml
然后创建webhook.yaml即可。
kubectl apply -f webhook.yaml
上面的所有文件都在代码库里，可以直接使用脚本进行部署。
# sh deploy.sh
creating certs in tmpdir /tmp/tmp.SvMHWcPI6x
Generating RSA private key 2048 bit long modulus
..........................................
.............................................................
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default created
NAME AGE REQUESTOR CONDITION
validate-delete-namespace.default 0s kubernetes-admin Pending
certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default approved
secret/validate-delete-namespace-tls created
Creating k8s admission deployment
deployment.apps/validate-delete-namespace created
service/validate-delete-namespace created
validatingwebhookconfiguration.admissionregistration.k8s.io/validate-delete-namespace created
执行完成过后，可以查看具体的信息。
# kubectl get po
NAME READY STATUS RESTARTS AGE
validate-delete-namespace-74c9b8b7bd-5g9zv 1/1 Running 0 3s
# kubectl get secret
NAME TYPE DATA AGE
default-token-kx5wf kubernetes.io/service-account-token 3 72d
validate-delete-namespace-tls Opaque 2 53s
# kubectl get ValidatingWebhookConfiguration
NAME CREATED AT
validate-delete-namespace 2022-06-24T09:39:26Z
测试
（1）首先打开webhook的pod日志
# kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f
I0624 17:39:27.858753 1 main.go:30] Server running in port: 8443
（2）创建一个namespace并删除
# kubectl create ns joker
# kubectl get ns | grep joker
joker Active 4h5m
# kubectl delete ns joker
Error from server: admission webhook "validate-delete-namespace.default.svc.cluster.local" denied the request without explanation
# kubectl get ns | grep joker
joker Active 4h5m
可以发现我们的删除操作被拒绝了，并且查看namespace还存在。
我们也可以到日志中查看，如下：
# kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f
I0624 17:39:27.858753 1 main.go:30] Server running in port: 8443
2022/06/24 17:43:34 You cannot delete namespace: joker
I0624 17:43:34.664945 1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false
2022/06/24 17:43:34 You cannot delete namespace: joker
I0624 17:43:34.667043 1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false
上面就是简单的实现了一个准入控制器，
只要思想不滑坡，办法总比困难多。
感谢万能的百度，感谢牛逼的网友。
-
https://www.qikqiak.com/post/k8s-admission-webhook
-
https://github.com/douglasmakey/admissioncontroller
-
https://mritd.com/2020/08/19/write-a-dynamic-admission-control-webhook/
最后，求关注。如果你还想看更多优质原创文章，欢迎关注我们的公众号「运维开发故事」。
如果我的文章对你有所帮助，还请帮忙一下，你的支持会激励我输出更高质量的文章，非常感谢！
你还可以把我的公众号设为「星标」，这样当公众号文章更新时，你会在第一时间收到推送消息，避免错过我的文章更新。
