Following rancher official website,I have created a script to backup. It will stop rancher, backup with busy_box to current working dir and then restart the container again.

1

$./backup_rancher.sh rancher/rancher:v2.3.2

The parameter rancher/rancher:v2.3.2 is depending on value on the field of IMAGE in your $docker ps

1
2
3

$ docker ps
CONTAINER ID        IMAGE                    COMMAND            ...
b9ad26b8ece5        rancher/rancher:v2.3.2   "entrypoint.sh"    ...

// backup_rancher.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

#!/bin/bash 
set -u
set -e 

function backup() {
  rancher_image=$1
  rancher_container_name=$2
  rancher_container_tag=$3
  rancher_backup_date="$(date '+%Y-%m-%d-%H-%M-%S')"

  rancher_backup_container_name="rancher-data-$rancher_backup_date"
  rancher_backup_filename="rancher-data-backup-$rancher_container_tag-$rancher_backup_date.tar.gz"

  echo "...Stoping target rancher container $rancher_container_name"
  docker stop "$rancher_container_name"
  
  echo "...Creating util container and starting backup"
  docker create --volumes-from "$rancher_container_name" --name "$rancher_backup_container_name" "$rancher_image"
  docker run --volumes-from "$rancher_backup_container_name" -v "/$PWD"://backup:z "$docker_repository/busybox" tar zcvf //backup/$rancher_backup_filename //var/lib/rancher
  
  echo "...Removing backup util container $rancher_backup_container_name"
  docker rm -f "$rancher_backup_container_name" 
  
  echo "...Restarting target rancher container $rancher_container_name"
  docker start "$rancher_container_name"
}

function main() {
  ancestor_filter=$1
  
  for rancher in $(docker ps --filter status=running --filter ancestor=$ancestor_filter --format '{{.Image}}|{{.Names}}') 
  do
    echo "Found running rancher container $rancher"
    IFS='|' read -ra rancherArr <<< $rancher
    IFS=' ' 
    docker_image="${rancherArr[0]}"
    docker_names="${rancherArr[1]}"
    
    echo "...Resolving docker image tag from $docker_image"
    docker_image_tag='latest'
    docker_image_repository="${docker_image}"
    if [[ "${docker_image}" =~ ^(.*):(.*) ]] 
    then 
      docker_image_repository="${BASH_REMATCH[1]}"
      docker_image_tag="${BASH_REMATCH[2]}"
      echo "Resolved image repository: $docker_image_repository, tag: $docker_image_tag"
    else 
      echo "No match tag found, use default tag $docker_image_tag"
    fi
    backup $docker_image $docker_names $docker_image_tag
  done
}


# Change to your proxy registry if you are behind a proxy
docker_repository="registry.hub.docker.com/library"

main "$@"

A story

Imagine the code you want to run is a king and node is the army of servants.
The day starts by one servant waking up the king and asking him if he needs anything. The king gives the servant a list of tasks and goes back to sleep a little longer. The servant now distributes those tasks among his colleagues and they get to work.
Once a servant finishes a task, he lines up outside the kings quarter to report. The king lets one servant in at a time, and listens to things he reports. Sometimes the king will give the servant more tasks on the way out.
Life is good, for the king’s servants carry out all of his tasks in parallel, but only report with one result at a time, so the king can focus.

Callback programming model in nodejs

Request an action and register a callback which will be called when the action has been done.
The first request is from node main.js,

1. all function calls will be done in the current phase
2. any async function call (await on async, async.then) will register a callback in the pending callbacks queue
3. any timer will be registered in the timers queue
4. any I/O events from OS level will be listened in poll phases, (timeout and count limitation)
5. any setImmediate will be run in check, right after next event loop
6. any close callbacks
7. checks if it is waiting for any asynchronous I/O or timers and shuts down cleanly if there are not any
8. nextTickQueue will be processed after the current operation is completed, regardless of the current phase of the event loop

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

   ┌───────────────────────────┐
┌─>│           timers          │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │     pending callbacks     │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │       idle, prepare       │
│  └─────────────┬─────────────┘      ┌───────────────┐
│  ┌─────────────┴─────────────┐      │   incoming:   │
│  │           poll            │<─────┤  connections, │
│  └─────────────┬─────────────┘      │   data, etc.  │
│  ┌─────────────┴─────────────┐      └───────────────┘
│  │           check           │
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
└──┤      close callbacks      │
   └───────────────────────────┘

nextTick, setImmediate and setTimeout

A function passed to process.nextTick() is going to be executed on the current iteration of the event loop, after the current operation ends.
Any function passed as the setImmediate() argument is a callback that’s executed in the next iteration of the event loop.
Any function passed as the setTimeout() argument is a callback that’s executed in the next iteration of the event loop if the time duration has been met

Show me the code

I create a js file and show u how it works.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

async function bunchWorkAsync() {
    for (let x = 0; x < 20; x++) {
        setImmediate(() => console.log("set immediate" + x));
        setTimeout(() => console.log("set timeout" + x), 0);
        new Promise((resolve, reject) => { console.log('current phase'); resolve('promise callback ' + x) }).then(v => console.log(v)).catch(r => console.log(r));
        process.nextTick(() => { console.log("set nexttick" + x) });
        console.log('register ' + x)
    }
    return 0;
}

console.log('current phase is started >>>>>> '); 
bunchWorkAsync().then(v => console.log(v+ ' is resolved in the end of callback queue'));
setImmediate(() => console.log("set immediate in the end"));
console.log('current phase is done >>>>>> ');

when I run the js file, event loop starts to work with my current function(node file.js will run all sync and async functions and register all async callbacks in some queues)
So the running sequence will be explained as following:

1. console.log
   console.log(‘current phase is started >>>>>> ‘);
2. bunchWorkAsync()
   we go into the loop,
3. 1 setImmediate(() => console.log(“set immediate” + x));
   we register a callback in the check phase in next event loop
4. 2 setTimeout(() => console.log(“set timeout” + x), 0);
   we register a callback in the timers in next event loop
5. 3 new Promise((resolve, reject) …)
   Run console.log(‘current phase’)
   we register a callback by Promise(same as a async function) in the callback phase
6. 4 process.nextTick(() => { console.log(“set nexttick” + x) });
   we register a callback right after the current operation is done, should be run in current event loop
7. 5 console.log(‘register ‘ + x)
   we run the log in current phase
8. 6 return 0, since we run the bunchWorkAsync with callback
   we register a callback in next event loop
9. setImmediate(() => console.log(“set immediate in the end”));
   we add a callback in the check phase in next event loop
10. console.log(‘current phase is done >>>>>> ‘);
    we run the command in current operation

Then we can think everything in the event loop as following
Current loop

1. current operation
   running all command
2. next tick queue
   20 callbacks setup by process.nextTick(() => { console.log(“set nexttick” + x) });

Next event loop

1. timers queue
   20 callbacks setup by setTimeout()
2. pending callbacks queue
   20 callbacks setup by new Promise().then()
   1 callback setup by bunchWorkAsync().then()
3. check
   20 callbacks setup by setImmediate()
   1 callback setup by setImmediate(() => console.log(“set immediate in the end”));

So in theory, in the end, you should expect the running sequence as following
1 current phase output
current phase is started >>>>>>
20 times ‘current phase’
20 times ‘register ‘ + x
current phase is done >>>>>>
2. nextTick
20 times ‘set nexttick’ + x
3. timers
queues stay
4. callbacks queue
20 times ‘promise callback’
v+ ‘ is resolved in the end of callback queue’
5. poll
hi OS, I tell u to print my logs, it takes 20ms;
OS said yes. it takes 2ms to get the message into node. (I think callback run in libuv?)
after 22ms …
Oh yeah, I need to check with timers, they may ask to do something now
Of course,
20 times ‘set timeout’ is in the queue for a while
jump into timers tasks

5. check phase
   20 times ‘set immediate’
   set immediate in the end

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125

$ node examples/event-loop.js
current phase is started >>>>>>
current phase
register 0
current phase
register 1
current phase
register 2
current phase
register 3
current phase
register 4
current phase
register 5
current phase
register 6
current phase
register 7
current phase
register 8
current phase
register 9
current phase
register 10
current phase
register 11
current phase
register 12
current phase
register 13
current phase
register 14
current phase
register 15
current phase
register 16
current phase
register 17
current phase
register 18
current phase
register 19
current phase is done >>>>>>
set nexttick0
set nexttick1
set nexttick2
set nexttick3
set nexttick4
set nexttick5
set nexttick6
set nexttick7
set nexttick8
set nexttick9
set nexttick10
set nexttick11
set nexttick12
set nexttick13
set nexttick14
set nexttick15
set nexttick16
set nexttick17
set nexttick18
set nexttick19
promise callback 0
promise callback 1
promise callback 2
promise callback 3
promise callback 4
promise callback 5
promise callback 6
promise callback 7
promise callback 8
promise callback 9
promise callback 10
promise callback 11
promise callback 12
promise callback 13
promise callback 14
promise callback 15
promise callback 16
promise callback 17
promise callback 18
promise callback 19
0 is resolved in the end of callback queue
set timeout0
set timeout1
set timeout2
set timeout3
set timeout4
set timeout5
set timeout6
set timeout7
set timeout8
set timeout9
set timeout10
set timeout11
set timeout12
set timeout13
set timeout14
set timeout15
set timeout16
set timeout17
set timeout18
set timeout19
set immediate0
set immediate1
set immediate2
set immediate3
set immediate4
set immediate5
set immediate6
set immediate7
set immediate8
set immediate9
set immediate10
set immediate11
set immediate12
set immediate13
set immediate14
set immediate15
set immediate16
set immediate17
set immediate18
set immediate19
set immediate in the end

References

• nodejs event loop
• a storty

I’ve never worked as a web frontend developer in any company, however I have to work with it with my own project all the time.

It’s so interesting when I try to search html editor that dreamweaver hasn’t been an option any more in most of the blogs. In 2009, when I learnt html tags, I thought everyone would use it in the future.

Anyway, it always helps to keep html definition in mind.

HTML is the standard markup language for documents designed to be displayed in a web browser. It can be assisted by technologies such as Cascading Style Sheets (CSS) and scripting languages such as JavaScript. Web browser receive html documents from a web browser.

Nowadays, it seems the sauce is better than the fish. JavaScript and CSS are taking more responsibilities than html itself, e.g. popular frameworks reactjs or angularjs.

I try to think about different ways of how a html page can be created and served. This is a simple

Initially in (1), html pages are static and be served from web server directly. Smart developers start to use template in the webserver and make it a lit bit reusable.

JavaScript is introduced in (2) to make web pages with animations or doing ajax request to render part of pages. It’s a big applause if u can submit a post request and update a page without refreshing the whole pages. Dynamically adding html elements with piece of data using jQuery in the browser is popular. Html still accounts for larger proportion of a project, I would say. Server template engine is heavily used to generate html pages with data.

JavaScript is dominant in (3), putting piece of data with js to be rendered is dissatisfied. JS developers create compelling frameworks which can render all data which is required by a page and replace template engine such as JSP. A index html page is sent to a client browser with a bundle of js files. When all files are fetched, browser will be ready to show a page as a Single-page Application(SPA). Data will be fetched and rendered when the user interacts. When I start to use ReactJS, I think it definitly belongs to this phase especially be using with nodejs app. It requires a client to have good network connection to do all the requests to different servers.

Empty pages or slow client loading, SEO rendering. Problems exist in phase (3) when it comes to those questions. Client has to wait for loading the who SPA in the first access. A blend solution of (2) and (3) which is called Isomorphic Web Application, an isomorphic app is a web app that blends a server-rendered web app with a single-page application. Someone defined it as codes can be run in both client and server side. I prefer to defined as a SPA with the feature that its first html page can be rendered in server side.
When server receives a page request, it shall request data by itself, render a page and send it to client, subsequent client request will be handled by the client it self as a SPA.

Good posts to explain Single Page Application and Isomorphic Web Application

• Toptal.com: client side vs server sdie pre rendering

• Medium ElyseKoGo

What is React JS

ReactJs is an javascript library for building interactive web user interfaces. It provides a framework for frontend developers to divide their web pages into different reusable components.

Back to the old days, we have html static page(or use template engine to create) to be served from an webserver, through http everyone having internet can access the content by using an browser.

Javascript gives the posibilities for web page developers to add more dynamic parts on a page, such as showing windows, showing dynamic content to html dom and sending ajax request to get extra data without refreshing whole pages.

As more and more contents and interactions grow, web pages become more complex. A lot of software developments will end up with a common question, how to reuse codes instead of copy/paste? In web page development, the question is how to break a page into different part so that we can reuse some of them in different projects.

ReactJS is trying to solve this problem by using component-based development.

What’s a component in reactjs?

As I mentioned above, how to reuse thing is one of purposes of React and component is the one which will be resued in react.

Function is the initial and basic idea in most of programme world when people don’t want to copy/paste pieces of codes around. I think generic functions will need a few parts in von neumann architecture

• input,
• output,
• control logic, how to react with input
• local state, where the function can store middle computation result.

In engineering, a function is interpreted as a specific process, action or task that a system is able to perform.

Not surprised that React will use similar idea. We can consider ReactJS as a system and a component as a function.

From reactjs offical website, a component takes in parameters, called props (short for “properties”), and returns a hierarchy of views to display via the render method. The function is something similar as following.

1
2
3
4
5
6
7

Component A {
    render(props) {
        // variables 
        //... control logic
        return htmlView;
    }
}

input is called props
output is the result of render
logic is determined by render method

The above component we defined based on the definition cannot use any other information except the props we give it. We can do a lot of simple things with it, such as rendering a list with different names

1
2
3
4
5
6
7
8
9

Component NameList {
    render(props) {
        return <ul><li>props.name</li></ul>
    }
}

ReactRenderEngine.render(NameList,{props={name:"kai"}})
ReactRenderEngine.render(NameList,{props={name:"Erik"}})
ReactRenderEngine.render(NameList,{props={name:"Fabio"}})

All of above is what we can imagine before going into real Reactjs implementation.

In practice, React does similar implementation. It has a render engine ReactDOM.render to render a component into html dom structure. React introduces element concept which is similar as html tags. The component specification is as following

1
2
3
4
5
6
7
8
9
10
11
12
13

// Definition function
function Welcome(props) {
  return <h1>Hello, {props.name}</h1>;
}

// ES6 Definition class
class Welcome extends React.Component {
  render() {
    return <h1>Hello, {this.props.name}</h1>;
  }
}

ReactDOM.render(<Welcome name="Sara" />, document.getElementById('root'))

One important thing about react component is that * React component is not the same as a function in javascript* even though we can define it by a function.

A function is finished and forget by system once it’s returned. However, a component can have runtime states and keeps track its states until it’s deleted by React. A component can have states by setting this.state in their constructors.

I would like to compare react component with a function with a global variable.

// javascript function with global variables
functionState=[]
function A() {
    functionState[count]++;
    return <div/>
}

// react component
class A extends React.Component {
  constructor(props) {
    super(props);
    this.state = {count: 0};
  }
  render() {
    return (
      this.state.count++;
      return <div/>
    );
  }
}

React is a framework to compose complex UIs from small and isolated pieces of code called “components”. It’s also a runtime to load, manage component states.

A component in react is a small service and it follows specific lifecycles.

I started to get involved into DevOps area since july of 2017. Jenkins, groovy and ssh are all my equipments to construct all the CI/CD pipeline realms.

I’ve heared a lot about Ansible since then but never get a determination to try it since I’m not sure what it is. Recently, I’m helping a team to build their CD pipeline. I think it’s a good time to use Ansible to do something. I start to get familier with the overview of the tool. In this post, I will give my opinions about those terms.

What is Ansible

Ansible is an open-source software provisioning, configuration management, and application-deployment tool.

I think it’s a definition that’s hard to understand for most of the developers if they’re not working with operations. But most likely, they have worked with those tasks in a single machine.

• ssh into a server
• install some missing libs or services (configuration management)
• add user or group (software provisioning)
• download a product (application-deployment)
• change permissions (application-deployment)
• start or stop a service (application-deployment)
• change some configurations for nginx(configuration management)
• …

Nowadays, Infrastructure as code is geting more and more popular, an experience operation might be able to provision a single server, debug the whole infrastructure and operate on a single product, however, it’s impossible to manage a cluster of servers at the same time.

• Manage microsevices running on different servers and each of them requires different configurations.
• Provision environments for testing, demo, staging and production.
  When
• Change open ports or load balancer configurations
• …

We do need a way to sit in a single host and shooting commands into different machines and it’s basically doing things as following:

1
2
3
4
5
6

kube_clusters=(server1 server2 server3 server4)
for serverName in $kube_clusters
  ssh -t username@serverName software_provision.sh 
  ssh -t username@serverName configuration_management.sh 
  ssh -t username@serverName application_deployment.sh 
done

Ansible is doing similar things but make the steps easier to understand and reusable in different systems. Ansible manages hosts by modeling all software packages, configs, and services as configurable resources. Via ssh, it transfers all tasks into remote servers and runs those tasks on the resources to bring them to the target state.

Concepts in Ansbile

Ansible works by connecting to your nodes and pushing out small programs, called “Ansible modules” to them. These programs are written to be resource models of the desired state of the system. Ansible then executes these modules (over SSH by default), and removes them when finished.

• Ansible server: The machine where Ansible is installed and from which all tasks and playbooks will be ran.

• Module: A module is units of codes executed directly on remote hosts. Ansible ships with a number of modules (called the ‘module library’). Modules can take args, return json values or notify other tasks.

1
2
3

ping
command -a "/sbin/reboot -t now"
service -a "name=httpd state=started"

-a to given arguments to ansible module

• Inventory: A file contains groups and hosts to be managed. Ansible works against multiple systems in your infrastructure at the same time. Inventory provides a way to defined names for hosts and group similiar ones. Such as our above bash array name kube_clusters. Following inventory file is in INI format used by ansible.

1
2
3
4
5
6
7
8
9
10

mail.example.com

[webservers]
foo.example.com
bar.example.com

[dbservers]
one.example.com
two.example.com
three.example.com

Running a module to see how it works

There’re more concepts in ansbile which is not covered yet, such as playbooks, roles and so on. Let’s have a initial look at ansible command before next post is ready.

1
2
3
4
5

kaichu:~$ ansible localhost -m ping -a data=helloansible
localhost | SUCCESS => {
    "changed": false,
    "ping": "helloansible"
}

In Ansible server where ansible is installed, we can use ansible to run a module ping with argument data=helloansible on a remote host localhost and see the returned result Json format in the console.

To understand the results of ping module, check following references, I will explain it in next post.

• Common module return values in Ansible
• Module ping in Ansible

Good references to distinguish provisioning and configuration management.

• Why configuration management and provisioning are different
• End to end application provisioning with ansible and terraform

To have a overview about ansible, check the official explanation.

• How ansible works
• All modules in ansible core

In the comming posts, I will introduce how to start with ansible and play some common modules to make you used to it.

Just start to use Jekyll to manage my personal blog, not sure if there’s a tool to create a blog in the basice theme minima, to make my life easier, I decide to add a simple bash scripts to add a post easily.

Steps to create a post in minima:

1. Add a file in _posts folder
2. Name the file with YYYY-MM-DD-my-title-is-here.markdown
3. Copy paste the front matter from an old blog
4. Change the title of to “My title is here”
5. Change the date and time
6. Start to think about my content

I think a basic bash will save my life and it’s worthy to add it, what I want the script to do is receiving a a list of words, finishing all above steps and generating a file with basic front matter

1

./new.sh my title is here

Date and time

We need a time format 2019-09-21 14:56:34 +0200 in the blog front matter, which can be generated using following bash date command

1

blogDate=`date "+%F %T %z"`

We need a date format 2019-09-21 which is also easy in bash, %F is refered by this turorial

1

filedate=`date +%F`

Title and filename

The input from the command will be all the words of the title. To generate a blog file, we need to convert all letters into lower case and add a hyphen in between, which is quite easy to be done by tr command

1
2

lowcase=$( echo $title | tr '[A-Z]' '[a-z]')
formatTitle=$( echo $lowcase | tr '[[:blank:]]' '-' )

However, in bash 3, seems we don’t have a good choice to make the first letter of a string upper case and welcome anyone to give me a suggest if you konw. I agree with Michael’s suggestion in stackoverflow and using following way to archive it. Basically, get the first letter in the string by using string manipulation and replace it with upper case. Then we combine it with the rest of other charaters in the string.

1

blogTitle="$( echo ${lowcase:0:1} | tr '[a-z]' '[A-Z]' )${lowcase:1}"

Then we can create the file and output the front matter with here document

1
2
3
4
5
6
7
8
9
10
11
12

# Create a file with the filename
touch $filename

# Add some common front matter
cat > $filename << EOF
---
layout: post
title:  "${blogTitle}"
date:   $blogDate
categories: 
---
EOF

To generate a new post

1
2
3

$ cd _posts/
$ ./new.sh my new post idea is ready
## > 2019-09-21-my-new-post-idea-is-ready.markdown has be generated

Script source file

I used to document things in two google docs, one of them is called engineering handbook and the other is programming handbook.
I think I need to share my ideas and experiences so that someone can find it and solve their problems. And also I need to learn some experiences from others.

You’ll find out a lot of topics in this website since I’m working as Developer, DevOps and fullstack CTO of Foodla AB