Notes from AWS security group last night

There are some concepts I learned from it. I will list it here as a note.

ASD TOP4 rules

  1. Targeted cyber intrusions remain the biggest threat to government ICT systems. Since opening in early 2010, the Cyber Security Operations Centre (CSOC) has detected and responded to thousands of these intrusions.
  2. You should never assume that your information is of little or no value. Adversaries are not just looking for classified information. A lot of activity observed by the CSOC has an economic focus, looking for information about Australia’s business dealings, its intellectual property, its scientific data and the government’s intentions.
  3. The threat is real, but there are things every organisation can do to significantly reduce the risk of a cyber intrusion. In 2009, based on our analysis of these intrusions, the Australian Signals Directorate produced Strategies to Mitigate Targeted Cyber Intrusions – a document that lists a variety of ways to protect an organisation’s ICT systems. At least 85% of the intrusions that ASD responded to in 2011 involved adversaries using unsophisticated techniques that would have been mitigated by implementing the top four mitigation strategies as a package.
  4. The top four mitigations are: application whitelisting; patching applications and operating systems and using the latest version; and minimising administrative privileges. This document is designed to help senior managers in organisations understand the effectiveness of implementing these strategies.

Now I begin to worry about our system…haha let’s backup the data now!

IDS/IPS

IDS (Intrusion Detection Systems)/ IPS ( Intrusion Prevention System)

The concept of IDS is trying to detect all possible trace of attack while IPS means deny the attack request(one example: as a firewall )

Security Trigger

One thing they mentioned last night is the trigger of getting the attack. Usually, we know which processes will be executed on the server. If the server has some other processes been setup, then we need to trigger the system to execute protecting operations.

CIS

It contains many best practices for security

Reading Notes – TCP tuning

TCP tuning

What is delay ack

TCP is a reliable protocol because of sending ack after receiving the request.

TCP delayed acknowledgment is a technique used by some implementations of the Transmission Control Protocol in an effort to improve network performance. In essence, several ACK responses may be combined together into a single response, reducing protocol overhead.

In one word, if delay ack is on, the ack of one request can be sent within another request together. Or if there are two ack packages, it will also be sent. Or if it’s over 40ms, the single ack will be sent.

Nagle

The logic of Nagle is like:

if there is new data to send
  if the window size >= MSS and available data is >= MSS
        send complete MSS segment now
  else
    if there is unconfirmed data still in the pipe
          enqueue data in the buffer until an acknowledge is received
    else
          send data immediately
    end if
  end if
end if

If the data size is larger than MSS, send it directly. Otherwise, see if there are any requests haven’t received ack. if not, we can wait until we receive the previous ack.

If the client is using Nagle and the Server side is using delay ack

Sometimes, we have to wait until the request reaches the delay ack timeout. example

Some HTTP requests will divide the header and content into two packages which can trigger this issue easier.

solution

Found online

struct curl_slist *list = NULL;
//合并post包
list = curl_slist_append(list, "Expect:");  

CURLcode code(CURLE_FAILED_INIT);
if (CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_URL, oss.str().c_str())) &&
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, timeout)) &&
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &write_callback)) &&
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L)) &&
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_POST, 1L)) &&
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, pooh.sizeleft)) &&
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback)) &&
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_READDATA, &pooh)) &&                
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L)) && //1000 ms curl bug
        CURLE_OK == (code = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list))                
        ) {

        //这里如果是小包就不开delay ack,实际不科学
        if (request.size() < 1024) {
                code = curl_easy_setopt(curl, CURLOPT_TCP_NODELAY, 1L);
        } else {
                code = curl_easy_setopt(curl, CURLOPT_TCP_NODELAY, 0L);
        }
        if(CURLE_OK == code) {
                code = curl_easy_perform(curl);
        }

Reading Note – Cache and Downgrade solution for distribution system

Cache

there is a good explanation about multi-level cache

  • (L1) Level 1 Cache(2KB – 64KB) – Instructions are first searched in this cache. L1 cache very small in comparison to others, thus making it faster than the rest.
  • (L2) Level 2 Cache(256KB – 512KB) – If the instructions are not present in the L1 cache then it looks in the L2 cache, which is a slightly larger pool of cache, thus accompanied by some latency.
  • (L3) Level 3 Cache (1MB -8MB) – With each cache miss, it proceeds to the next level cache. This is the largest among the all the cache, even though it is slower, it’s still faster than the RAM.

Downgrade solution for distribution system

Concept

In order to make sure the main service is still available during extreme situations such as high pressure, server error, or breaking workflow, we can design the downgrade service. The downgrade service will only provide limited services.

The server can automatically downgrade the service regarding key variables. We can also implement a manual solution to switch on/off downgrade service.

Some services can not be downgraded.

Category

  • automatically or manually
  • read service or write service
  • multiple level or single level

Scenario

  • normal: service is not stable because of upgrade and internet. Use automatical solution
  • warning: the stability range from 95% to 100%。 Use automatical solution or manual solution
  • error: availability less than 90%. It may be caused by DB connection pool is full or request is too large. Use automatical solution or manual solution
  • serious error: manual solution

source

Reading Note – Interface development for getting products distributed services

Cache Design

  • The product data will be saved to cache (redis)
  • Provide the interface for refreshing interface
  • During the refreshing of redis, the front end should still have data
  • Data cache should have a version number. Every time the cache update should generate a new cache key
  • Save the cache key to the list which can filter the latest version by current time

API Design

  • interface parameter: version and page index
  • response data: data and current version
  • use sorted set to get the latest version number
  • the first time request will response the current latest version number
  • the client will cache the version number. The next page request will need the cached version number
  • return 304 if the client request the first page and local version number equal to the current version number. So the client can avoid re-rendering the page
  • if no version number within the request, Use the latest one

Tips:

  • the infinite increase of version number: the version number will keep increasing. So we need to delete the version number of last day during the update. Use sorted set to filter the version number
  • when getting the latest version, capturing the version set for the last hour. Then use a filter to get the latest one. When getting the data for the product, find the version number which contains the data if the product does not exist in current version
  • SLB: Server Load Balancing

Structure

system design

Summary

Factors need to be considered:

  • The performance of the interface
  • The stability of the interface
  • Error Tolerance
  • Server side pressure: reduce the pressure of the server side
  • Service downgrade: during high pressure

Best practice for sass

The project front end is using vue vuex and sass. So here are sole rules about best practice:

  • Variable – can be used for standard layout, e.g. margin border font and coler
  • Nest – Regarding different modules, nest structure is easy to search and the structure is clear. It’s good for component dev
  • Reuse – reusing of the same include will lead to code redundancy

e.g.

@mixin clearfix{
 
  &:before,&:after{
 
    display:block;
    content:'';
    height:0;
    clear:both;
  }
 
}
 
.container{
    @include clearfix;   
}
 
.tab{
    @include clearfix;
}

instead of the above code, we should do:

@mixin clearfix{
 
  &:before,&:after{
 
    display:block;
    content:'';
    height:0;
    clear:both;
  }
}
 
.container,.tab{
    @include clearfix;   
}

WordPress Self-defined Plugin

Composer

PHP composer has a native support for wordpress plugin:

"package": {
    ...
   "type":"wordpress-plugin"
   ...
}

This will automatically install the composer package within wordpress plugin folder.

Plugins structure

Instead of generating multiple plugins for the wordpress, we can implement multiple extensions within one single plugin. The strucutre will look like:

Plugin
  |----woocommence extension
  |----thrid party auth
  |----analysis
  |---- ...

We can also implement a page to manage the plugin and turn on/off the extension.

Improve the performance

We can use plugin for caching the wordpress website. The cache can be used in front end, db and anywhere else.

Tips:

  • It’s not a good practice to change the function within the theme.php because it may affect the user access. Besides, once we change the theme it may cause issues. So we should use plugins instead of theme.
  • Security. We can hide and change the api url for admin and other security module.

Kubernetes (4)

Kubernetes (4)

Setup controller and service

Now, we need to create a Replication Controller for the application. Because if a standalone Pod dies, it won’t restart automatically.

# web-controller.yml
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: web
  name: web-controller
spec:
  replicas: 2
  template:
    metadata:
      labels:
        name: web
    spec:
      containers:
      - image: gcr.io/<YOUR-PROJECT-ID>/myapp
        name: web
        ports:
        - containerPort: 3000
          name: http-server
kubectl create -f web-controller.yml

Then we need to create a service as an interface for those pods.

This is just like the “link” command line option we used with Docker compose.

# web-service.yml
apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    name: web
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  selector:
    name: web

kubectl create -f web-service.yml
  • The type is LoadBalancer. This is a cool feature that will make Google Cloud Platform create an external network load balancer automatically for this service!
  • We map external port 80 to the internal port 3000, so we can serve HTTP traffic without messing with Firewalls.

We can use command to check pods status.

kubectl get pods

In order to find the IP address of our app, run this command:

$ gcloud compute forwarding-rules list
NAME     REGION        IP_ADDRESS       IP_PROTOCOL TARGET
abcdef   us-central1   104.197.XXX.XXX  TCP         us-xxxx

The ideal structure

So now we have two pods for the application and one web service which contains the extrenal ip.

Now we need to setup db service for our application.

MongoDB has a concept of Replica Set

A replica set is a group of mongod instances that maintain the same data set. A replica set contains several data bearing nodes and optionally one arbiter node. Of the data bearing nodes, one and only one member is deemed the primary node, while the other nodes are deemed secondary nodes.

When a primary does not communicate with the other members of the set for more than 10 seconds, an eligible secondary will hold an election to elect itself the new primary. The first secondary to hold an election and receive a majority of the members’ votes becomes primary.

We can follow another blog to set it up

The writer create a repository to auto config the mongodb Replica Set. I forked the repository

To setup:

git clone https://github.com/thesandlord/mongo-k8s-sidecar.git
cd /mongo-k8s-sidecar/example/StatefulSet/
kubectl apply -f googlecloud_ssd.yaml
kubectl apply -f mongo-statefulset.yaml

Tips:

  • Be careful about the zone difference

Rolling update

If we only need to update the container image:

kubectl rolling-update NAME [NEW_NAME] --image=IMAGE:TAG

Web UI

It’s better to setup a UI dashbaord for your cluster. All relevant operations can be done via that dashboard

Bind static IP to service external ip

  • create a service as usual
  • Once your app is up, make note of the External IP using
kubectl get services
  • Now go to the Google Cloud Platform Console -> Networking -> External IP Addresses.
  • Find the IP you were assigned earlier. Switch it from “Ephemeral” to “Static.” You will have to give it a name and it would be good to give it a description so you know why it is static.
  • Then modify your service (or service yaml file) to point to this static address. I’m going to modify the yaml.
apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    name: web
spec:
  type: LoadBalancer
  loadBalancerIP: 104.199.187.56
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    name: web
  • Once your yaml is modified you just need to run it; use
kubectl apply -f service.yml

Kubernetes (3)

Kubernetes (3)

Package Up The Image With Dockerfile

By following the tutorial, we need to first generate an independent image for production.

Later we can use google’s new fea ture (Build triggers) to trigger the branch update and automatically build and install staging image into the container registry.

Entire Process to Setup Application

Setup cluster via gcloud

We can setup new cluster via gcloud command or via gcloud GUI

Login to the gcloud cluster

gcloud container clusters get-credentials cluster-name --zone=asia-east1-a

Kubernetes (2)

Kubernetes (2)

Expose the Application Publicly

Structure

Master  
  |_ Deployment (Multi)  = Application (Deployed)
         |_ Node (Multi)
              |_ Pod (Multi) (Internal IP) 
                   |_ Container (Multi)
Service (External IP for public)
   |_ Pod (From different nodes) (optional)

Tips:

  • the kubernets needs at least 3 nodes to enable the auto update function for the application.

Scale Your Application

Scaling is accomplished by changing the number of replicas in a Deployment

Kubernetes also supports autoscaling of Pods

Services have an integrated load-balancer that will distribute network traffic to all Pods of an exposed Deployment. Services will monitor continuously the running Pods using endpoints, to ensure the traffic is sent only to available Pods.

Update Your Application

Rolling updates allow Deployments’ update to take place with zero downtime by incrementally updating Pods instances with new ones.

Object Management Using kubectl

There are 3 ways to manage the object

  • Imperative commands (directly via commands)
  • Imperative Management of Kubernetes Objects Using Configuration Files (use yaml files for config)
  • Declarative Management of Kubernetes Objects Using Configuration Files (using config files within the repository )

Usually we should use the third one. The basic concept is to create a config file for the project with everything prepared.

Deploy Real MEAN Stack application with Kubernetes

Link

Tips

  • COPY within dockerfile only works for copying file from outside to inside. If we want to move files within the container, we need
RUN cp A B

Kubernetes (1) & flask signal

Kubernetes

Cluster up and running

Using minikube to create a cluster

> minikube version
minikube version: v0.15.0-katacoda
> minikube start
Starting local Kubernetes cluster...

to check out kubectl

> kubectl version
Client Version: version.Info{Major:"1", Minor:"5", G
itVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07
b448ab3ed78d0520507", GitTreeState:"clean", BuildDat
e:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compi
ler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", G
itVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07
b448ab3ed78d0520507", GitTreeState:"clean", BuildDat
e:"1970-01-01T00:00:00Z", GoVersion:"go1.7.1", Compi
ler:"gc", Platform:"linux/amd64"}

check cluster details

> kubectl cluster-info
Kubernetes master is running at http://host01:8080
heapster is running at http://host01:8080/api/v1/pro
xy/namespaces/kube-system/services/heapster
kubernetes-dashboard is running at http://host01:808
0/api/v1/proxy/namespaces/kube-system/services/kuber
netes-dashboard
monitoring-grafana is running at http://host01:8080/
api/v1/proxy/namespaces/kube-system/services/monitor
ing-grafana
monitoring-influxdb is running at http://host01:8080
/api/v1/proxy/namespaces/kube-system/services/monito
ring-influxdb

We have a running master and a dashboard. The Kubernetes dashboard allows you to view your applications in a UI. During this tutorial, we’ll be focusing on the command line for deploying and exploring our application. To view the nodes in the cluster, run the kubectl get nodes command:

  > kubectl get nodes
NAME      STATUS    AGE
host01    Ready     5m

This command shows all nodes that can be used to host our applications. Now we have only one node, and we can see that it’s status is ready (it is ready to accept applications for deployment).

Deploy an App

Once you have a running Kubernetes cluster, you can deploy your containerized applications on top of it. To do so, you create a Kubernetes Deployment. The Deployment is responsible for creating and updating instances of your application.

If the Node hosting an instance goes down or is deleted, the Deployment controller replaces it. This provides a self-healing mechanism to address machine failure or maintenance.

When you create a Deployment, you’ll need to specify the container image for your application and the number of replicas that you want to run. You can change that information later by updating your Deployment;

Let’s run our first app on Kubernetes with the kubectl run command. The run command creates a new deployment. We need to provide the deployment name and app image location (include the full repository url for images hosted outside Docker hub). We want to run the app on a specific port so we add the –port parameter:

  > kubectl run kubernetes-bootcamp --image=docker.i
ort=8080lin/kubernetes-bootcamp:v1 --p
deployment "kubernetes-bootcamp" created

To list your deployments

 > kubectl get deployments
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   1         1         1            1           2m

To view the application output without exposing it externally, we’ll create a route between our terminal and the Kubernetes cluster using a proxy:

 > kubectl proxy
Starting to serve on 127.0.0.1:8001kubectl proxy

We now have a connection between our host (the online terminal) and the Kubernetes cluster. The started proxy enables direct access to the API. The app runs inside a Pod (we’ll cover the Pod concept in next module). Get the name of the Pod and store it in the POD_NAME environment variable:

export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

> echo Name of the Pod: $POD_NAME
Name of the Pod: kubernetes-bootcamp-390780338-k4k25

To see the output of our application, run a curl request:

curl http://localhost:8001/api/v1/proxy/namespaces/default/pods/$POD_NAME/

OD_NAME/ http://localhost:8001/api/v1/proxy/namespaces/default/pods/$P
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-390780338-k4k25
| v=1

Pods and Nodes

A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker or rkt), and some shared resources for those containers. Those resources include:

  • Shared storage, as Volumes
  • Networking, as a unique cluster IP address
  • Information about how to run each container, such as the container image version or specific ports to use

A Pod models an application-specific “logical host” and can contain different application containers which are relatively tightly coupled. For example, a Pod might include both the container with your Node.js app as well as a different container that feeds the data to be published by the Node.js webserver. The containers in a Pod share an IP Address and port space, are always co-located and co-scheduled, and run in a shared context on the same Node.

Pods are the atomic unit on the Kubernetes platform. When we create a Deployment on Kubernetes, that Deployment creates Pods with containers inside them (as opposed to creating containers directly). Each Pod is tied to the Node where it is scheduled, and remains there until termination (according to restart policy) or deletion. In case of a Node failure, identical Pods are scheduled on other available Nodes in the cluster.

A Pod always runs on a Node. A Node is a worker machine in Kubernetes and may be either a virtual or a physical machine, depending on the cluster. Each Node is managed by the Master. A Node can have multiple pods, and the Kubernetes master automatically handles scheduling the pods across the Nodes in the cluster. The Master’s automatic scheduling takes into account the available resources on each Node.

Every Kubernetes Node runs at least:

  • Kubelet, a process responsible for communication between the Kubernetes Master and the Nodes; it manages the Pods and the containers running on a machine.
  • A container runtime (like Docker, rkt) responsible for pulling the container image from a registry, unpacking the container, and running the application. Containers should only be scheduled together in a single Pod if they are tightly coupled and need to share resources such as disk.

We’ll use the kubectl get command and look for existing Pods:

 > kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
kubernetes-bootcamp-390780338-m454n   1/1       Running   0          10s

Next, to view what containers are inside that Pod and what images are used to build those containers we run the describe pods command:

> kubectl describe pods

To get logs from the container, we’ll use the kubectl logs command:

 > kubectl logs $POD_NAME
Kubernetes Bootcamp App Started At: 2017-05-05T06:46:41.845Z | Running On:  k
ubernetes-bootcamp-390780338-m454n

Running On: kubernetes-bootcamp-390780338-m454n | Total Requests: 1 | App Upt
ime: 230.702 seconds | Log Time: 2017-05-05T06:50:32.547Z

We can execute commands directly on the container.

  > kubectl exec $POD_NAME env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=kubernetes-bootcamp-390780338-m454n
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
NPM_CONFIG_LOGLEVEL=info
NODE_VERSION=6.3.1
HOME=/root

Next let’s start a bash session in the Pod’s container:

 > kubectl exec -ti $POD_NAME bash
root@kubernetes-bootcamp-390780338-m454n:/#

We have now an open console on the container where we run our NodeJS application. The source code of the app is in the server.js file:

cat server.js

You can check that the application is up by running a curl command:

curl localhost:8080

Note: here we used localhost because we executed the command inside the NodeJS container

To close your container connection type exit.

Although Pods each have a unique IP address, those IPs are not exposed outside the cluster without a Service. Services allow your applications to receive traffic. Services can be exposed in different ways by specifying a type in the ServiceSpec:

  • ClusterIP (default) – Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster.
  • NodePort – Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using :. Superset of ClusterIP.
  • LoadBalancer – Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort.
  • ExternalName – Exposes the Service using an arbitrary name (specified by externalName in the spec) by returning a CNAME record with the name. No proxy is used. This type requires v1.7 or higher of kube-dns.

Few new concepts for flask

Signal

Flask supports signal and signals relevant plugins. We can use the signal to decouple the big application.

For example, capturing the login and logout signal can be used for recording user activities.

API URL Design Rules

  • Only use noun for the url
  • Use HTTP Request Type to control the behavior
  • Return the next link within the request