Table of Contents |
---|
JIRA Ticket
...
Istio is a service mesh which provides a dedicated infrastructure layer that you can add to your applications. It adds capabilities like observability, traffic management, and security, without adding them to your own code.
To populate its own service registry, Istio connects to a service discovery system. For example, if you’ve installed Istio on a Kubernetes cluster, then Istio automatically detects the services and endpoints in that cluster. Using this service registry, the Envoy proxies can then direct traffic to the relevant services.
Istio Ingress Gateway can be used as a API-Gateway to securely expose the APIs of your micro services. It can be easily configured to provide access control for the APIs i.e. allowing you to apply policies defining who can access the APIs, what operations they are allowed to perform and much more conditions.
The Istio API traffic management features available are: Virtual services: Configure request routing to services within the service mesh. Each virtual service can contain a series of routing rules, that are evaluated in order. Destination rules: Configures the destination of routing rules within a virtual service.
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: bookinfo-ratings spec: host: ratings.prod.svc.cluster.local trafficPolicy: loadBalancer: simple: LEAST_CONN subsets: - name: testversion labels: version: v3 trafficPolicy: loadBalancer: simple: ROUND_ROBIN |
Istio provisions the DNS names and secret names for the DNS certificates based on configuration you provide. The DNS certificates provisioned are signed by the Kubernetes CA and stored in the secrets following your configuration. Istio also manages the lifecycle of the DNS certificates, including their rotations and regenerations.
With Mutual TLS (mTLS) the client and server both verify each other’s certificates and use them to encrypt traffic using TLS.. With Istio, you can enforce mutual TLS automatically.
...
JWT can also contain information about the client that sent the request (client context).
We can use Istio's RequestAuthentication resource to configure JWT policies for your services.
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: httpbin namespace: foo spec: selector: matchLabels: app: httpbin jwtRules: - issuer: "issuer-foo" jwksUri: https://example.com/.well-known/jwks.json --- apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: httpbin namespace: foo spec: selector: matchLabels: app: httpbin rules: - from: - source: requestPrincipals: ["*"] |
...
- Kong can be used as an API gateway:
- Hiding internal microservice structure
- Could be used as R1 API front-end
Kong acts as the service registry, keeping a record of the available target instances for the upstream services. When a target comes online, it must register itself with Kong by sending a request to the Admin API. Each upstream service has its own ring balancer to distribute requests to the available targets.
With client-side discovery, the client or API gateway making the request is responsible for identifying the location of the service instance and routing the request to it. The client begins by querying the service registry to identify the location of the available instances of the service and then determines which instance to use. See https://konghq.com/learning-center/microservices/service-discovery-in-a-microservices-architecture/
Kong datastore
Kong uses an external datastore to store its configuration such as registered APIs, Consumers and Plugins. Plugins themselves can store every bit of information they need to be persisted, for example rate-limiting data or Consumer credentials. See https://konghq.com/faqs/#:~:text=PostgreSQL%20is%20an%20established%20SQL%20database%20for%20use,Cassandra%20or%20PostgreSQL%2C%20Kong%20maintains%20its%20own%20cache.
...
- Install ISTIO on minikube using instruction here: Istio Installation - Simplified Learning (waytoeasylearn.com)
cd to the istio directory and install the demo application
kubectl create ns foo
kubectl apply -f <(istioctl kube-inject -f samples/httpbin/httpbin.yaml) -n foo
Create a python script to generate a JWT token using the code from here: https://medium.com/intelligentmachines/istio-jwt-step-by-step-guide-for-micro-services-authentication-690b170348fc . Install python_jwt using pip if it's not already installed.
- Create jwt-example.yaml using the public key generated by the python script:
kubectl create -f jwt-example.yaml
Code Block language yml title jwt-example.yaml apiVersion: "security.istio.io/v1beta1" kind: "RequestAuthentication" metadata: name: "jwt-example" namespace: istio-system spec: selector: matchLabels: istio: ingressgateway jwtRules: - issuer: "ISSUER" jwks: | { "keys":[{"e":"AQAB","kty":"RSA","n":"x_yYl4uW5c6NHOA-bDDh0MThFggBWl-vYJr77b9F1LmAtTlJVM0rL5klTfv2DmlAmD9eZPrWeUOoOGhSpe58XiSAvxyeaOrZhtyUjT3aglrSys0YBsB19ItNGMuoIuzPpWOrdtKwHa9rPbrdc6q7vb93qu2UVaIz-3FJmGFtSA5t8FK_5bZKF-oOzRLwqeVQ3n0Bu_dFDuGeZjQWMZF32QupyA-GF-tDGGriPLy9sutlB1NQyZ4qiSZx5UMxcfLwsWfQxHemdwLeZXWKWNBov8RmbZy2Jz-dwg6XjHBWAjTnCGG9p-bp63nUlnELI3LcEGhGOugZBqcpNT5dEAQ0fQ"}]}
Export the JWT token generated by the python script as an environment variable:
export TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJBVURJRU5DRSIsImV4cCI6MTYzNzI1NDkxNSwiaWF0IjoxNjM3MjUxOTE1LCJpc3MiOiJJU1NVRVIiLCJqdGkiOiJCcmhDdEstcC00ZTF0RlBrZmpuSmhRIiwibmJmIjoxNjM3MjUxOTE1LCJwZXJtaXNzaW9uIjoicmVhZCIsInJvbGUiOiJ1c2VyIiwic3ViIjoiU1VCSkVDVCJ9.HrQCLPZXf0VkFe7JUVGXq-sHJQhVibqhToG4r63py-iwHWlUL02_WfoWRoxapgqGwImDdSlt1uG8RR-6VMqzWwGlcqBIRhFTG0nmzmtQjnOUs6QAKSUpA3PyWBIYHV0BwZbpo8Zq1Bo-sELy400fU-MCQ_054fSsG7JMBMmrnj8NyJmD2lNN0VSFGO53SPl2tQSVlc9OwAr8Uu0jfLPfUmh6yq43qFuxnVRfBGLLPNOt29aOfAetKLc72qlphtnbDx2a9teP5AIbkIWyIlhTytEnQRCwU4x8gDrEdkrHui4qCtzpl_uoITSwPe3AFsi7gQHB6rJoDj-j2zPc4rUTAA"export INGRESS_HOST=$(minikube ip)
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
Test the service:
curl --header "Authorization: Bearer $TOKEN" $INGRESS_HOST:$INGRESS_PORT/headers -s -o /dev/null -w "%{http_code}\n"You should get a response code of 200
Update the token to something invalid
The response will be 401
...
Retrieve public key using : http(s)://<hostname>/auth/realms/<realm name>
Anchor keycloak keycloak
Enable keycloak with Istio
Setup a new realm, user and client as shown here : https://www.keycloak.org/getting-started/getting-started-kube
Note the id of the new user, this will be used as the sub field in the token e.g. 81b2051b-52d9-4e4e-88a6-00ca04b7b73d"
The iss field is url of the realm e.g. http://192.168.49.2:30869/auth/realms/myrealm
Edit the jwt-pms RequestAuthentication definition above, replace the issuer with the keycloak iss and remove the jwks field and replace it with the jwksUri pointing to your keycloak certs
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: "jwt-pms" namespace: istio-nonrtric spec: selector: matchLabels: apptype: nonrtric-pms jwtRules: - issuer: "http://192.168.49.2:30869/auth/realms/myrealm" jwksUri: "http://192.168.49.2:30869/auth/realms/myrealm/protocol/openid-connect/certs" |
Modify the AuthorizationPolicy named pms-policy, change the issuer and subject to the keycloak iss/sub
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: "security.istio.io/v1beta1" kind: "AuthorizationPolicy" metadata: name: "pms-policy" namespace: istio-nonrtric spec: selector: matchLabels: apptype: nonrtric-pms action: ALLOW rules: - from: - source: requestPrincipals: ["http://192.168.49.2:30869/auth/realms/myrealm/81b2051b-52d9-4e4e-88a6-00ca04b7b73d"] |
Reapply the yaml file
to generate a token use the following command:
curl -X POST "$KEYCLOAK_URL" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "username=$USERNAME" \
-d "password=$PASSWORD" \
-d 'grant_type=password' \
-d "client_id=$CLIENT_ID" | jq -r '.access_token'
e.g.
curl -X POST http://192.168.49.2:30869/auth/realms/myrealm/protocol/openid-connect/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "username=user" \
-d "password=secret" \
-d 'grant_type=password' \
-d "client_id=myclient" | jq -r '.access_token'
Note: you may need to install the jq utility on your system for this to work - sudo apt-get install jq
Test the a1-policy service with your new token
TOKEN=$(curl -X POST http://192.168.49.2:30869/auth/realms/myrealm/protocol/openid-connect/token -H "Content-Type: application/x-www-form-urlencoded" -d username=user -d password=secret -d 'grant_type=password' -d client_id=myclient | jq -r '.access_token')
curl --header "Authorization: Bearer $TOKEN" $INGRESS_HOST:$INGRESS_PORT/a1-policy
Hello a1-policy
Note: The iss of the token will differ depending on how you retrieve it. If it's retrieved from within the cluster for URL will start with http://keycloak.default:8080/ otherwise it will be something like : http://192.168.49.2:31560/ (http://(minikube ip): (keycloak service nodePort))
Keycloak database
Keycloak uses the H2 database by default.
To configure keycloak to use a different database follow these steps.
- Install either postgres or mariadb using these yaml files: postgres.yaml or mariadb.yaml. These will setup the keycloak db along with the username and password. You just need to change the directory for your persistent storage to an appropiate directory on your host.
Update the keycloak installation script https://raw.githubusercontent.com/keycloak/keycloak-quickstarts/latest/kubernetes-examples/keycloak.yaml
Code Block language yml title Keycloak Environment env: - name: KEYCLOAK_USER value: "admin" - name: KEYCLOAK_PASSWORD value: "admin" - name: PROXY_ADDRESS_FORWARDING value: "true" - name: DB_VENDOR value: "postgres" - name: DB_ADDR value: "postgres" - name: DB_PORT value: "5432" - name: DB_DATABASE value: "keycloak" - name: DB_USER value: "keycloak" - name : DB_PASSWORD value: "keycloak"
...
Further details on authorization policies are avaiable here
Anchor | ||||
---|---|---|---|---|
|
Istio network policy is enforced at the pod level (in the Envoy proxy), in user-space, (layer 7), as opposed to Kubernetes network policy, which is in kernel-space (layer 4), and is enforced on the host. By operating at application layer, Istio has a richer set of attributes to express and enforce policy in the protocols it understands (e.g. HTTP headers).
Anchor grafana grafana
Grafana
Istio also comes with grafana, to start it run : istioctl dashboard grafana
...
For confidential clients you can also set the Client Authenticator to "X509 certificate" in the credentials tab.
You can then set your subject DN to something like: .*client@mail.com.* and turn on "Allow Regex Pattern Comparison"
The JWT can then be retrieved using a call like the following:
curl -k -X POST https://$HOST:$KEYCLOAK_PORT/auth/realms/$REALM/protocol/openid-connect/token \
--data "grant_type=password&scope=openid profile&client_id=$CLIENT" \
--cert client.pem
Alternatively you can use the -E option to entrypt the pem file
curl -k -X POST https://$HOST:$KEYCLOAK_PORT/auth/realms/$REALM/protocol/openid-connect/token \
--data "grant_type=password&scope=openid profile&client_id=$CLIENT" \
-E client.pem
For more informtion see: X.509 Client Certificate User Authentication
It is not possible to connect to a TLS server with curl using only a client certificate, without the client private key. https://stackoverflow.com/questions/36431179/using-curl-with-cert
Token can also be retrieved using go:
Code Block | ||||
---|---|---|---|---|
| ||||
package main import ( "crypto/tls" "crypto/x509" "fmt" "io/ioutil" "net/http" "net/url" ) func main() { caCert, _ := ioutil.ReadFile("/mnt/c/Users/ktimoney/keycloak-certs/rootCA.crt") caCertPool := x509.NewCertPool() caCertPool.AppendCertsFromPEM(caCert) cert, _ := tls.LoadX509KeyPair("/mnt/c/Users/ktimoney/keycloak-certs/client.crt", "/mnt/c/Users/ktimoney/keycloak-certs/client.key") client := &http.Client{ Transport: &http.Transport{ TLSClientConfig: &tls.Config{ RootCAs: caCertPool, Certificates: []tls.Certificate{cert}, }, }, } keycloakHost := "192.168.49.2" keycloakPort := "31561" realmName := "x509" keycloakUrl := "https://" + keycloakHost + ":" + keycloakPort + "/auth/realms/" + realmName + "/protocol/openid-connect/token" clientId := "x509client" scope := "openid profile" resp, err := client.PostForm(keycloakUrl, url.Values{"username": {""}, "password": {""}, "grant_type": {"password"}, "client_id": {clientId}, "scope": {scope}}) if err != nil { panic(err) } defer resp.Body.Close() fmt.Println("response Status:", resp.Status) fmt.Println("response Headers:", resp.Header) body, _ := ioutil.ReadAll(resp.Body) fmt.Println("response Body:", string(body)) } |
Java example available here: X.509 Authentication in Spring Security
Istio CA Certs
To allow istio to work with keycloak you must add your certificate to the istio certs when you're installing.
An istio operator file is used for this: istio.yaml
istioctl install --set profile=demo -f istio.yaml
Further instruction are available here: Custom CA Integration using Kubernetes CSR
Using istio-gateway to obtain JWT tokens.
You may want to avoid connecting directly to the keycloak server for security reasons.
You can connect to it through the istio ingress gateway instead if you wish.
You will need to setup a gateway in PASSTROUGH mode and virtual service that maps the keycloak host to the keycloak SNI host.
The following file can be used to do this for the CN used when creating certificates above: keycloak-gateway.yaml
You can test this using curl:
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/sh INGRESS_HOST=$(minikube ip) SECURE_INGRESS_PORT=$(kubectl -n default get service istio-ingressgateway -n istio-system -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}') curl -v --resolve "keycloak.est.tech:$SECURE_INGRESS_PORT:$INGRESS_HOST" --cacert rootCA.crt "https://keycloak.est.tech:$SECURE_INGRESS_PORT" CLIENT="myclient" REALM="x509" curl --resolve "keycloak.est.tech:$SECURE_INGRESS_PORT:$INGRESS_HOST" --cacert rootCA.crt \ -X POST https://keycloak.est.tech:$SECURE_INGRESS_PORT/auth/realms/$REALM/protocol/openid-connect/token \ --data "grant_type=password&scope=openid profile&client_id=$CLIENT" \ -E client.pem echo "" |
...
In the credentials section choose "Signed JWT" from the Client Authenticator dropdown and choose RS256 as the Signature Algorithm.
Then in the Keys tab import the client.crt we generated earlier.
We need to create a self-signed JWT assertion to use this.
Create a public key from your private key with the following command: openssl rsa -in client.key -outform PEM -pubout -out client_pub.key
See the following link on how to do this: Creating and validating a JWT RSA token in Golang
We need to make some modifications to the token.go code to enable it to work for us.
The following fields need to be added to the claims map:
claims["jti"] = "myJWTId" + fmt.Sprint(now.UnixNano())
claims["sub"] = "jwtclient"
claims["iss"] = "jwtclient"
claims["aud"] = "http://192.168.49.2:31560/auth/realms/x509"
Also modify main.go so it uses you public and private keys and only outputs the token value.
Compile and run the code to produce the self-signed JWT assertion.
This can then be used to obtain a JWT access token with the following curl command:
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/sh HOST=$(minikube ip) KEYCLOAK_PORT=$(kubectl -n default get service keycloak -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}') REALM="x509" CLIENT="jwtclient" JWT=$(./main) curl -k -X POST http://$HOST:$KEYCLOAK_PORT/auth/realms/$REALM/protocol/openid-connect/token \ -d "grant_type=client_credentials" -d "scope=openid profile" -d client_id=$CLIENT \ -d "client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer" \ -d client_assertion=$JWT echo "" |
...
No client secret is required for this authentication.
These self-signed JWT assertions are for one time use only, the jti claim value must have a unique id for every call.
Note: we can also call this using https with some small modifications.
Client authentication with signed JWT with client secret
...
You will need to copy the client secret into the cod code for this to work.
Download the go code: oauth2.go
...
curl -H "Authorization: Bearer $TOKEN" localhost:9000
Hello World OAuth2!
See also: golang oauth2
Keycloak Rest API
Documentation for the keycloak Rest API is available here: Keycloak Admin REST API
Below is some sample code that calls the clients rest api to create a new client:
Code Block | ||||
---|---|---|---|---|
| ||||
export ADMIN_TKN=$(curl -s -X POST --insecure https://$HOST:$KEYCLOAK_PORT/auth/realms/master/protocol/openid-connect/token \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "username=admin" \ -d 'password=admin' \ -d 'grant_type=password' \ -d 'client_id=admin-cli' | jq -r '.access_token') echo "ADMIN CLIENT TOKEN = $ADMIN_TKN" curl -X POST --insecure https://$HOST:$KEYCLOAK_PORT/auth/admin/realms/x509provider/clients \ -H "authorization: Bearer $ADMIN_TKN" \ -H "Content-Type: application/json" \ --data \ ' { "id": "x509Client", "name": "x509Client", "enabled": "true", "defaultClientScopes": ["email"], "redirectUris": ["*"], "attributes": {"use.refresh.tokens": "true", "client_credentials.use_refresh_token": "true"} } ' |
...
PUT request using service account requesting party token
PUT resources 1
OPA
The Open Policy Agent (OPA) is an open source, general-purpose policy engine that unifies policy enforcement across the stack. OPA provides a high-level declarative language that lets you specify policy as code and simple APIs to offload policy decision-making from your software.
OPA policies are expressed in a high-level declarative language called Rego. Rego is purpose-built for expressing policies over complex hierarchical data structures.
It even has a VSCode plugin that lets you highlight and evaluate rules and query policies right within the IDE.
Policy-based control for cloud native environments
...
Istio External Authorization with OPA
Dynamic Policy Composition for OPA
Open Policy Agent: Authorization in a Cloud Native World
Microservices Authorization using Open Policy Agent and Traefik (API Gateway)
A Study in Serverless Authorization with Open Policy Agent
Godaddy Opa Lambda Extension Plugin
GO
Go provides a library for opa.
Code Block | ||||
---|---|---|---|---|
| ||||
package main import ( "context" "encoding/json" "fmt" "github.com/open-policy-agent/opa/rego" "io/ioutil" "net/http" "net/url" "os" ) type Jwttoken struct { Access_token string Expires_in int Refresh_expires_in int Refresh_token string Token_type string Not_before_policy int Session_state string Scope string } var token Jwttoken var opaPolicy string = ` package authz import future.keywords.in default allow = false jwks := jwks_request("http://keycloak:8080/auth/realms/opa/protocol/openid-connect/certs").body filtered_jwks := [ key | some key in jwks.keys key.use == "sig" ] token_cert := json.marshal({"keys": filtered_jwks}) token = { "isValid": isValid, "header": header, "payload": payload } { [isValid, header, payload] := io.jwt.decode_verify(input, { "cert": token_cert, "aud": "account", "iss": "http://keycloak:808 0/auth/realms/opa"}) } allow { is_token_valid } is_token_valid { token.isValid now := time.now_ns() / 1000000000 token.payload.iat <= now now < token.payload.exp token.payload.clientRole == "[opa-client-role]" } jwks_request(url) = http.send({ "url": url, "method": "GET", "force_cache": true, "force_json_decode": true, "force_cache_duration_seconds": 3600 # Cache response for an hour }) ` func getToken() string { clientSecret := "63wkv0RUXkp01pbqtNTSwghhTxeMW55I" clientId := "opacli" realmName := "opa" keycloakHost := "keycloak" keycloakPort := "8080" keycloakUrl := "http://" + keycloakHost + ":" + keycloakPort + "/auth/realms/" + realmName + "/protocol/openid-connect/token" resp, err := http.PostForm(keycloakUrl, url.Values{"client_secret": {clientSecret}, "grant_type": {"client_credentials"}, "client_id": {clientId}}) if err != nil { fmt.Println(err) panic("Something wrong with the credentials or url ") } defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) json.Unmarshal([]byte(body), &token) return token.Access_token } func traceOpa(input string) { ctx := context.TODO() test := rego.New( rego.Query("x = data.authz.allow"), rego.Trace(true), rego.Module("example.rego", opaPolicy), rego.Input(input), ) test.Eval(ctx) rego.PrintTraceWithLocation(os.Stdout, test) } func evaluateOpa(input string) { ctx := context.TODO() query, err := rego.New( rego.Query("x = data.authz.allow"), rego.Module("example.rego", opaPolicy), ).PrepareForEval(ctx) if err != nil { // Handle error. fmt.Println(err.Error()) } results, err := query.Eval(ctx, rego.EvalInput(input)) // Inspect results. if err != nil { // Handle evaluation error. fmt.Println("Error: " + err.Error()) } else if len(results) == 0 { // Handle undefined result. fmt.Println("Results are empty") } else { // Handle result/decision. fmt.Printf("Results = %+v\n", results) //=> [{Expressions:[true] Bindings:map[x:true]}] } } func main() { tokenStr := getToken() traceOpa(tokenStr) evaluateOpa(tokenStr) } |
...
opa bench --data rbactest.rego 'data.rbactest.allow'
+-------------------------------------------------+------------+
| samples | 22605 |
| ns/op | 47760 |
| B/op | 6269 |
| allocs/op | 112 |
| histogram_timer_rego_external_resolve_ns_75% | 400 |
| histogram_timer_rego_external_resolve_ns_90% | 500 |
| histogram_timer_rego_external_resolve_ns_95% | 500 |
| histogram_timer_rego_external_resolve_ns_99% | 871 |
| histogram_timer_rego_external_resolve_ns_99.9% | 29394 |
| histogram_timer_rego_external_resolve_ns_99.99% | 29800 |
| histogram_timer_rego_external_resolve_ns_count | 22605 |
| histogram_timer_rego_external_resolve_ns_max | 29800 |
| histogram_timer_rego_external_resolve_ns_mean | 434 |
| histogram_timer_rego_external_resolve_ns_median | 400 |
| histogram_timer_rego_external_resolve_ns_min | 200 |
| histogram_timer_rego_external_resolve_ns_stddev | 1045 |
| histogram_timer_rego_query_eval_ns_75% | 31100 |
| histogram_timer_rego_query_eval_ns_90% | 37210 |
| histogram_timer_rego_query_eval_ns_95% | 47160 |
| histogram_timer_rego_query_eval_ns_99% | 91606 |
| histogram_timer_rego_query_eval_ns_99.9% | 630561 |
| histogram_timer_rego_query_eval_ns_99.99% | 631300 |
| histogram_timer_rego_query_eval_ns_count | 22605 |
| histogram_timer_rego_query_eval_ns_max | 631300 |
| histogram_timer_rego_query_eval_ns_mean | 29182 |
| histogram_timer_rego_query_eval_ns_median | 25300 |
| histogram_timer_rego_query_eval_ns_min | 15200 |
| histogram_timer_rego_query_eval_ns_stddev | 32411 |
+-------------------------------------------------+------------+
OPA & Minio
OPA & MinIO's Access Management Plugin
OPA Sidecar injection
First create a namespace for your apps and enable istio and opa
...
Ensure your istio mesh config has been setup to include grcp local authorizer
kubectl edit configmap istio -n istio-system
Code Block | ||||
---|---|---|---|---|
| ||||
extensionProviders: - envoyExtAuthzGrpc: port: "9191" service: local-opa-grpc.local name: opa-local |
Update your rapp-provider authorization policy to use this provider:
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: rapp-opa-provider-opa namespace: opa spec: selector: matchLabels: app: rapp-opa-provider action: CUSTOM provider: name: "opa-local" rules: - to: - operation: paths: ["/rapp-opa-provider"] notPaths: ["/health"] |
Run the opa_test.sh script above and you should see a message confirming your connection to the service.
Note: References to keycloak need to be updated to include the keycloak schema i.e keycloak.default
Basic Authentication
We can add basic authentication to our NGINX bubdle server by following these steps:
Create a password file using the following command: sudo htpasswd -c .htpasswd <user>, you will be prompted to input the password.
This will produce a file called .htpasswd containing the username and encrypted password
e.g. admin:$apr1$tPQCjrVW$sokcSj4QVkncEDna0Fc2o/
Add the following configmap definitions to your nginx.yaml
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: v1 kind: ConfigMap metadata: name: nginx-pwd-config namespace: default data: .htpasswd: | admin:$apr1$tPQCjrVW$sokcSj4QVkncEDna0Fc2o/ --- apiVersion: v1 kind: ConfigMap metadata: name: nginx-conf-config namespace: default data: default.conf: | server { server_name localhost; location ~ ^/bundles/(.*)$ { root /usr/share/nginx/html/bundles; try_files /$1 =404; auth_basic "Restricted"; auth_basic_user_file /etc/nginx/conf.d/conf/.htpasswd; } } --- |
Then update your volumes and volume mounts to include these files with your deployment
Code Block | ||||
---|---|---|---|---|
| ||||
volumeMounts: - name: bundlesdir mountPath: /usr/share/nginx/html/bundles readOnly: true - name: nginx-conf mountPath: /etc/nginx/conf.d/default.conf subPath: default.conf - name: nginx-pwd mountPath: /etc/nginx/conf.d/conf/.htpasswd subPath: .htpasswd volumes: - name: bundlesdir hostPath: # Ensure the file directory is created. path: /var/opa/bundles type: DirectoryOrCreate - name: nginx-conf configMap: name: nginx-conf-config defaultMode: 0644 - name: nginx-pwd configMap: name: nginx-pwd-config defaultMode: 0644 |
This will add basic authentication to your bundles directory.
Run echo -n <username>:<password> | base64 to encrpt your usename and password
e.g. echo -n admin:admin | base64
YWRtaW46YWRtaW4=
Update the opa-istio-config ConfigMap in the opa_inject.yaml file to include the encrypted string as a token in the cedentials section:
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: v1 kind: ConfigMap metadata: name: opa-istio-config namespace: opa data: config.yaml: | plugins: envoy_ext_authz_grpc: addr: :9191 path: policy/ingress/allow decision_logs: console: true services: - name: bundle-server url: http://bundle-server.default credentials: bearer: token: YWRtaW46YWRtaW4= scheme: Basic bundles: authz: service: bundle-server resource: bundles/opa-bundle.tar.gz persist: true polling: min_delay_seconds: 10 max_delay_seconds: 20 --- |
Your bundle is now protected with basic authentication.
JWT Injection
To enable automatic include a jwt token in our rapp request we need to enable some k8s objects:
1) MutatingWebhookConfiguration to inject jwt retrieval pod as a sidecar. The MutatingWebhookConfiguration uses a pod mutating serive to alter our pod and add the new sidecar.
Code Block | ||||
---|---|---|---|---|
| ||||
package main import ( "encoding/json" "errors" "flag" "fmt" "io/ioutil" "k8s.io/api/admission/v1beta1" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/serializer" "log" "net/http" "strconv" ) type ServerParameters struct { port int // webhook server port certFile string // path to the x509 cert keyFile string // path to the x509 private key } type patchOperation struct { Op string `json:"op"` Path string `json:"path"` Value interface{} `json:"value,omitempty"` } var parameters ServerParameters var ( universalDeserializer = serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer() ) func main() { flag.IntVar(¶meters.port, "port", 8443, "Webhook server port.") flag.StringVar(¶meters.certFile, "tlsCertFile", "/certs/tls.crt", "File containing the x509 certificate") flag.StringVar(¶meters.keyFile, "tlsKeyFile", "/certs/tls.key", "File containing the x509 private key") flag.Parse() http.HandleFunc("/inject-sidecar", HandleSideCarInjection) log.Fatal(http.ListenAndServeTLS(":"+strconv.Itoa(parameters.port), parameters.certFile, parameters.keyFile, nil)) } func HandleSideCarInjection(w http.ResponseWriter, r *http.Request) { body, err := ioutil.ReadAll(r.Body) err = ioutil.WriteFile("/tmp/request", body, 0644) if err != nil { panic(err.Error()) } var admissionReviewReq v1beta1.AdmissionReview if _, _, err := universalDeserializer.Decode(body, nil, &admissionReviewReq); err != nil { w.WriteHeader(http.StatusBadRequest) fmt.Errorf("Could not deserialize request: %v", err) } else if admissionReviewReq.Request == nil { w.WriteHeader(http.StatusBadRequest) errors.New("Malformed admission review - request is empty") } fmt.Printf("Received Admission Review Request - Type: %v \t Event: %v \t Name: %v \n", admissionReviewReq.Request.Kind, admissionReviewReq.Request.Operation, admissionReviewReq.Request.Name, ) var pod v1.Pod err = json.Unmarshal(admissionReviewReq.Request.Object.Raw, &pod) if err != nil { fmt.Errorf("Could not unmarshal pod from admission request: %v", err) } var patches []patchOperation labels := pod.ObjectMeta.Labels labels["sidecar-injection-webhook"] = "jwt-proxy" patches = append(patches, patchOperation{ Op: "add", Path: "/metadata/labels", Value: labels, }) var containers []v1.Container containers = append(containers, pod.Spec.Containers...) container := v1.Container{ Name: "jwt-proxy", Image: "ktimoney/rapps-jwt", ImagePullPolicy: v1.PullIfNotPresent, Ports: []v1.ContainerPort{ { Name: "http", Protocol: v1.ProtocolTCP, ContainerPort: 8888, }, }, VolumeMounts: []v1.VolumeMount{ { Name: "certsdir", MountPath: "/certs", ReadOnly: true, }, }, } containers = append(containers, container) fmt.Println(containers) patches = append(patches, patchOperation{ Op: "add", Path: "/spec/containers", Value: containers, }) pathType := v1.HostPathDirectoryOrCreate pathTypePtr := &pathType var volumes []v1.Volume volumes = append(volumes, pod.Spec.Volumes...) volume := v1.Volume{ Name: "certsdir", VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ Path: "/var/rapps/certs", Type: pathTypePtr, }, }, } volumes = append(volumes, volume) fmt.Println(volumes) patches = append(patches, patchOperation{ Op: "add", Path: "/spec/volumes", Value: volumes, }) fmt.Println(patches) patchBytes, err := json.Marshal(patches) if err != nil { fmt.Errorf("Error occurred when trying to marshal JSON patch: %v", err) } admissionReviewResponse := v1beta1.AdmissionReview{ Response: &v1beta1.AdmissionResponse{ UID: admissionReviewReq.Request.UID, Allowed: true, }, } admissionReviewResponse.Response.Patch = patchBytes bytes, err := json.Marshal(&admissionReviewResponse) if err != nil { fmt.Errorf("Error occurred when trying to marshal Aadmission Review response: %v", err) } w.Write(bytes) } |
MutatingWebhookConfiguration.yaml
Note: You'll need to configure your deployment to include a tls.crt and tls.key secret. Your MutatingWebhookConfiguration will need to include the corresponding ca bubdle.
2) Envoyfilter to update request header with jwt token.
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: RAPP-NAME-outbound-filter namespace: RAPP-NS spec: workloadSelector: labels: app.kubernetes.io/name: RAPP-NAME configPatches: # The first patch adds the lua filter to the listener/http connection manager - applyTo: HTTP_FILTER match: context: SIDECAR_OUTBOUND listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" subFilter: name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: # lua filter specification name: envoy.lua typed_config: "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua" inlineCode: | function envoy_on_request(request_handle) local uri = request_handle:headers():get(":path") local method = request_handle:headers():get(":method") if (method ~= "POST" and path ~= "/auth/realms/REALM-NAME/protocol/openid-connect/token") then -- Make an HTTP call to an upstream host with the following headers, body, and timeout. local headers, body = request_handle:httpCall( "jwt_cluster", { [":method"] = "GET", [":path"] = "/token", [":authority"] = "jwt-proxy", ["realm"] = "REALM-NAME", ["client"] = "CLIENT-NAME" }, "jwt call", 5000) if (headers["authorization"] ~= nil) then request_handle:headers():add("authorization", headers["authorization"]) end end end - applyTo: CLUSTER match: context: SIDECAR_OUTBOUND patch: operation: ADD value: # cluster specification name: jwt_cluster type: STRICT_DNS connect_timeout: 60s lb_policy: ROUND_ROBIN load_assignment: cluster_name: jwt_cluster endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 0.0.0.0 port_value: 8888 |
CFSSL
CFSSL is CloudFlare's PKI/TLS tool. It is both a command line tool and an HTTP API server for signing, verifying, and bundling TLS certificates.
To run this you first need to create an image with cfssl installed:
Code Block | ||||
---|---|---|---|---|
| ||||
FROM debian:latest RUN apt-get update && apt-get install -y curl && \ curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64 -o /usr/local/bin/cfssl && \ curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64 -o /usr/local/bin/cfssljson && \ chmod +x /usr/local/bin/cfssl && \ chmod +x /usr/local/bin/cfssljson RUN mkdir /config RUN mkdir /config RUN mkdir /certs WORKDIR /certs EXPOSE 8888 EXPOSE 8889 ENTRYPOINT ["cfssl version"] |
This will install cfssl on a debian image.
You can then use this image to create a cfssl service in your k8s cluster.
kubectl create -f rapps-cfssl.yaml
Note: If you want to use this with a postgres db you'll need to setup a new database and username/password and then create the tables.
Code Block | ||||
---|---|---|---|---|
| ||||
SELECT 'CREATE DATABASE cfssl' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'cfssl')\gexec DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'cfssl') THEN CREATE USER cfssl WITH PASSWORD 'cfssl'; GRANT ALL PRIVILEGES ON DATABASE cfssl TO cfssl; END IF; END $$; |
Login as the cfssl user then create the tables:
Code Block | ||||
---|---|---|---|---|
| ||||
CREATE TABLE IF NOT EXISTS certificates ( serial_number bytea NOT NULL, authority_key_identifier bytea NOT NULL, ca_label bytea, status bytea NOT NULL, reason int, expiry timestamptz, revoked_at timestamptz, pem bytea NOT NULL, issued_at timestamptz, not_before timestamptz, metadata jsonb, sans jsonb, common_name TEXT, PRIMARY KEY(serial_number, authority_key_identifier) ); CREATE TABLE IF NOT EXISTS ocsp_responses ( serial_number bytea NOT NULL, authority_key_identifier bytea NOT NULL, body bytea NOT NULL, expiry timestamptz, PRIMARY KEY(serial_number, authority_key_identifier), FOREIGN KEY(serial_number, authority_key_identifier) REFERENCES certificates(serial_number, authority_key_identifier) ); |
Once the pod is up and running you can connect to it by using port forwarding:
kubectl port-forward service/rapps-cfssl 8888:8888
You can generate signed certificates using a post request like the following:
curl -s -X POST -H "Content-Type: application/json" -d @./rapp-helloworld-provider-server.json http://127.0.0.1:8888/api/v1/cfssl/newcert
The rapp-helloworld-provider-server.json looks like this:
Code Block | ||||
---|---|---|---|---|
| ||||
{ "request":{ "hosts":[ "rapp-helloworld-provider" ], "names":[ { "C":"IE", "ST":"Ireland", "L":"Dublin", "O":"EST Rapp Provider", "OU":"EST Rapp Provider hosts" } ], "CN":"rapp-helloworld-provider", "key":{ "algo":"rsa", "size":2048 } }, "profile":"server" } |
To parse the response contents you can use the following method:
NEWCERT=$(curl -s -X POST -H "Content-Type: application/json" -d @./rapp-helloworld-provider-server.json http://127.0.0.1:8888/api/v1/cfssl/newcert)
echo $NEWCERT | jq -r .result.certificate
echo $NEWCERT | jq -r .result.private_key
OCSP & CRL
OCSP and CRL are ways of checking a certificates validity.
The cfssl crl endpoint run on port 8888: 127.0.0.1:8888/api/v1/cfssl/crl
For ocsp to run you need to run the following commands in your container:
cfssl ocsprefresh -db-config /config/db-pg.json -ca /certs/ca-server.pem -responder /certs/server-ocsp.pem -responder-key /certs/server-ocsp-key.pem
cfssl ocspdump -db-config /config/db-pg.json > ocspdump.txt
cfssl ocspserve -port=8889 -responses=/config/ocspdump.txt -loglevel=0
These commands will need to be re-run every time a new revoke request is received.
To revoke a certificate use:
curl -d '{ "serial": "708853190752997406197199597002842275021840632879", "authority_key_id": "dd2cdb5b5b8abbe2fba81fd6c04393938f802b03", "reason": "superseded" }' http://localhost:8888/api/v1/cfssl/revoke
You can obtain the serial and authority_key_id from your database using the following SQL:
select encode(authority_key_identifier,'escape') a_key, encode(serial_number,'escape') serial, encode(status,'escape') status from certificates;
You can also obtain the serial and authority_key_id from the certinfo endpoint
- Convert your certificate into a variable: pem=$(cat my.crt | sed -z 's/\n/\\n/g')
- curl -d '{"certificate": "'"$pem"'"}' http://localhost:8888/api/v1/cfssl/certinfo
- The serial number will come back as is but the authority_key_id needs to be converted: echo "22:8B:FA:ED:8B:FF:66:E7:05:A3:08:3A:41:33:D8:01:20:CA:CC:F4"| tr '[:upper:]' '[:lower:]' | sed s'/://'g =228bfaed8bff66e705a3083a4133d80120caccf4
You can check your certificate against ocsp and crl using the following code:
Code Block | ||||
---|---|---|---|---|
| ||||
package main import ( "bytes" "context" "crypto" "crypto/x509" "crypto/x509/pkix" "encoding/json" "encoding/pem" "flag" "fmt" "golang.org/x/crypto/ocsp" "io/ioutil" "log" "net/http" "net/url" "os" "time" ) type Cfinfo struct { Success string `json:"success,omitempty"` Result struct { Certificate string `json:"certificate"` Usages []string `json:"usages"` Expiry string `json:"expiry"` } Errors []string `json:"errors"` Messages []string `json:"messages"` } var cfinfo Cfinfo type Crlinfo struct { Success bool `json:"success,omitempty"` Result string `json:"result,omitempty"` Errors []string `json:"errors"` Messages []string `json:"messages"` } var crlinfo Crlinfo type ServerParameters struct { certFile string // path to the x509 cert issuerUrl string // webhook server port crlUrl string // webhook server port } var parameters ServerParameters func main() { flag.StringVar(¶meters.certFile, "certFile", "", "File containing the x509 certificate") flag.StringVar(¶meters.issuerUrl, "issuerUrl", "http://127.0.0.1:8888/api/v1/cfssl/info", "Url for retrieving the issuer certificate") flag.StringVar(¶meters.crlUrl, "crlUrl", "http://127.0.0.1:8888/api/v1/cfssl/crl", "Url for retrieving the crl") flag.Parse() if parameters.certFile == "" { flag.Usage() os.Exit(1) } // read x509 certificate from PEM encoded file cert_bytes := readFile(parameters.certFile) cert, err := decodeCert(cert_bytes) if err != nil { log.Fatal(err) } issuer_bytes, err := getIssuer(parameters.issuerUrl) //readCert(os.Args[2]) if err != nil { log.Fatal(err) } issuer, err := decodeCert(issuer_bytes) if err != nil { log.Fatal(err) } // Perform OCSP Check fmt.Println("Checing OCSP") status, err := checkOCSPStatus(cert, issuer) if err != nil { fmt.Println(err) } else { switch status { case ocsp.Good: fmt.Printf("[+] Certificate status is Good\n") case ocsp.Revoked: fmt.Printf("[-] Certificate status is Revoked\n") case ocsp.Unknown: fmt.Printf("[-] Certificate status is Unknown\n") } crl_bytes, err := getCrl(parameters.crlUrl) //readCert(os.Args[2]) if err != nil { log.Fatal(err) } crl, err := decodeCrl(crl_bytes) if err != nil { log.Fatal(err) } fmt.Println("\nChecing CRL") _, err = checkCRLStatus(cert, issuer, crl) if err != nil { log.Fatal(err) }else{ fmt.Println("[+] Certificate status is not Revoked\n") } } func checkCRLStatus(cert *x509.Certificate, issuer *x509.Certificate, crl *pkix.CertificateList) (bool, error) { var revoked = false // Check CRL signature err := issuer.CheckCRLSignature(crl) if err != nil { return revoked, err } // Check CRL validity if crl.TBSCertList.NextUpdate.Before(time.Now()) { return revoked, fmt.Errorf("CRL is outdated") } // Searching for our certificate in CRL for _, revokedCertificate := range crl.TBSCertList.RevokedCertificates { if revokedCertificate.SerialNumber.Cmp(cert.SerialNumber) == 0 { //Found validated certificate in list of revoked ones revoked = true return revoked, fmt.Errorf("[-] Certificate status is Revoked\n") } } return revoked, nil } // CheckOCSPStatus will make an OCSP request for the provided certificate. // If the status of the certificate is not good, then an error is returned. func checkOCSPStatus(cert *x509.Certificate, issuer *x509.Certificate) (int, error) { var ( ctx = context.Background() ocspURL = cert.OCSPServer[0] ) // Build OCSP request buffer, err := ocsp.CreateRequest(cert, issuer, &ocsp.RequestOptions{ Hash: crypto.SHA256, }) if err != nil { return ocsp.Unknown, fmt.Errorf("creating ocsp request body: %w", err) } req, err := http.NewRequest(http.MethodPost, ocspURL, bytes.NewBuffer(buffer)) if err != nil { return ocsp.Unknown, fmt.Errorf("creating http request: %w", err) } ocspUrl, err := url.Parse(ocspURL) if err != nil { return ocsp.Unknown, fmt.Errorf("parsing ocsp url: %w", err) } req.Header.Add("Content-Type", "application/ocsp-request") req.Header.Add("Accept", "application/ocsp-response") req.Header.Add("host", ocspUrl.Host) req = req.WithContext(ctx) // Make OCSP request httpResponse, err := http.DefaultClient.Do(req) if err != nil { return ocsp.Unknown, fmt.Errorf("making ocsp request: %w", err) } defer httpResponse.Body.Close() output, err := ioutil.ReadAll(httpResponse.Body) if err != nil { return ocsp.Unknown, fmt.Errorf("reading response body: %w", err) } // Parse response ocspResponse, err := ocsp.ParseResponse(output, issuer) if err != nil { return ocsp.Unknown, fmt.Errorf("parsing ocsp response: %w", err) } return ocspResponse.Status, nil } func readFile(file string) []byte { cert, err := ioutil.ReadFile(file) if err != nil { log.Fatalln(err) } return cert } func decodeCert(cert_bytes []byte) (*x509.Certificate, error) { b, _ := pem.Decode(cert_bytes) var cert *x509.Certificate cert, err := x509.ParseCertificate(b.Bytes) if err != nil { fmt.Println("Parse Error") return nil, fmt.Errorf("parsing certificate: %w", err) } return cert, err } func decodeCrl(cert_bytes []byte) (*pkix.CertificateList, error) { b, _ := pem.Decode(cert_bytes) crl, err := x509.ParseCRL(b.Bytes) if err != nil { return nil, fmt.Errorf("parsing crl: %w", err) } return crl, err } func getIssuer(issuerUrl string) ([]byte, error) { var resp = &http.Response{} values := map[string]string{"label": "intermediate"} jsonValue, _ := json.Marshal(values) resp, err := http.Post(issuerUrl, "application/json", bytes.NewBuffer(jsonValue)) if err != nil { fmt.Println(err) panic("Something wrong with the post request") } defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) json.Unmarshal([]byte(body), &cfinfo) return []byte(cfinfo.Result.Certificate), nil } func getCrl(crlUrl string) ([]byte, error) { resp, err := http.Get(crlUrl) if err != nil { return nil, err } defer resp.Body.Close() if resp.StatusCode >= 300 { return nil, fmt.Errorf("failed to retrieve CRL") } body, err := ioutil.ReadAll(resp.Body) json.Unmarshal([]byte(body), &crlinfo) crlString := "-----BEGIN X509 CRL-----\n" + crlinfo.Result + "\n-----END X509 CRL-----" return []byte(crlString), err } |
...
Integration of Istio with Kafka is limited.
The Kafka Protocol is a TCP level protocol (Layer 4)
Kafka is a binary wrapper over TCP protocol.
...
- ca.crt is obtained from the my-cluster-cluster-ca-cert secret, user.crt and user.key are obtained from the kowl secret.
Admission Controllers
An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.
Admission Controllers Reference
Kyverno
...