Category: LINUX

Linux and Docker on Ubuntu Series

May 11, 2022 /

Linux basic commands (Ubuntu)

Linux Kernel and hardware

Linux Runlevels

Linux Package Management for Ubuntu

Linux User Management commands

Linux Networking commands

Install Docker on Linux

Linux File Types

Create a file in Linux

How to check the file size in Linux

Compressing and Uncompressing Files in Linux

Searching files and directories in Linux

Search content with pattern in the file in Linux

Search content with pattern in the file in Linux

File Permissions in Linux

Check running services in Linux

DOCKER–

Docker FAQ’s

Install Docker on Ubuntu

Install Docker using install script on Ubuntu

Setup a Docker Swarm

Cache Busting and Version Pinning when building Docker images

Docker storage on Ubuntu

How to start docker in debug mode in Ubuntu

Docker Restart Policies

Use Docker image offline with Save and Load command in Ubuntu

Export Container and Import as Image using Docker in Ubuntu

Create a custom network in docker for communication between containers

Dcoker Security

Docker Best practice

Compressing and Uncompressing Files in Linux

May 3, 2022 / / 0 Comments

Compress a file in Linux using tar command

To compress multiple files us tar command with cf option

tar -cf test.tar <<file1>> <<file2>> <<file3>>

To see the content of the tar files use tf option

tar -tf test.tar

To extract the contents from tar file use xf option-

tar -f test.tar

Use zcf command to reduce the file size

tar -zcf testzcf.tar <<file1>> <<file2>>

See the size of test.tar and testzcf.tar file. The content in the tar is same but the size of the testzcf.tar is reduced with zcf command

There are other utlitlies to reduce the file size i.e. to compress files

Use gzip to compress file-

gzip <<file>>

This created filename.gz file. See the size of the firstfile.txt when comperessed

You can also use bzip2 and xz to compress files

Decompress a file

To decompress the gzip file use gunzip command-

You can use bunzip2 and unxz to decompress file

Use Docker Image offline with Save and Load command in Ubuntu

April 29, 2022 /

Docker Save and Load command

At times if you dont want to always pull the image from image registry which takes time to pull if the image’s are heavy, it makes sense to save the image and use it offline. This avoids to always pull the image from the registry.

Save the image once into tar file and reuse the images.

To Save image use follwoing command

First pull the image from repository

docker pull httpd
docker image save httpd -o httpdimage.tar

Get the image from the tar file instead pulling it from the registry

docker image load -i httpdimage.tar

Docker Security

April 28, 2022 /

Docker engine consists of Docker Daemon, Rest API and Docker CLI

To access the containers through Docker CLI the request is sent to Rest API and then to Docker Daemon to serve the request.

Docker Daemon service is accessible from within the host using unix socket which located in /var/run/docker.sock file

Applications can access the Docker daemon service from outside the host.

For accessing the docker daemon from outside the host securely configure /etc/docker/daemon.json when it is absolutely necessary

Setup the following in daemon.json file

{
   "hosts": ["tcp://hostip:2376"],
   "tls": "true",
   "tlscert": "/var/docker/server.pem",
   "tlskey": "/var/docker/serverkey.pem"
}

The above configuration help to connect to the Docker Daemon securely and in encrypted manner. On client run the docker command with tls set to true

docker --tls=true
OR 
export DOCKER_TLS=true
export DOCKER_HOST="tcp://hostip:2376"

Port 2376 allows to connect securely to Docker Daemon service.

But the above can be connected without authentication.

Access Docker Daemon using Certificate based Authentication

To access the Docker Daemon with certificate based authentication use following configuration-

{
   "hosts": ["tcp://hostip:2376"],
   "tls": "true",
   "tlscert": "/var/docker/server.pem",
   "tlskey": "/var/docker/serverkey.pem"
   "tlsverify": true,
   "tlscacert": "/var/docker/caserver.pem"
}

Here the tls_verify option enables certificate authentication based connection.

–tls will enable the connection with encryption

Clients with signed certificate will be able to access the host.

Client need to connect using following-

docker --tlsverify --tlscert=<<client.pem>> --tlskey=<<clientkey.pem>> --tlscacert=<<cacert.pem>>

Above can be also configured in ~/.docker file

Create a custom network in docker for communication between containers

April 21, 2022 /

When docker is installed it creates 3 networks. The default network is bridge.

To check the networks installed on host use following command-

docker network ls

To get the details of the network use inspect command

docker network inspect bridge

Now lets create a custom network for check the communication between the various containers

docker network create --driver=bridge --subnet=192.168.12.0/10 mycustomnetwork

Now lets create a container with the newly created network

docker container run -itd --net mycustomnetwork --name ubuntuone ubuntu

docker container run -itd --net mycustomnetwork --name ubuntutwo ubuntu

Here I have create 2 containers ubuntone and ubuntutwo

Inspect the containers to see the network details-

Lets ping the container e.g.- ubunttwo from ubuntuone-

docker exec -itd ubuntuone ping ubuntutwo

Error response from daemon: OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "ping": executable file not found in $PATH: unknown

If you receive above error you have to update repository and install ping utility

Install ping utility by interactively connecting to container and running the bash command

docker exec -it ubuntuone bash
apt-get udpate
apt-get install inetutils-ping

Execute the same for other container i.e. ubuntutwo

Now try to ping from ubuntuone to unbuntutwo

Here you should see the container is able to conect another-

docker exec -it ubuntuone ping ubuntutwo

For existing running container, you can use following command to connect to the network

docker network connect <<network name>> <<container name>>

Or if you want to move the container out of the network use network disconnect command-

docker network disconnect <<network name>> <<container name>>

To remove network – user rm command

docker network rm <<network name>>

To remove all unused network use prun command-

Prune will not remove OOTB network and will only remove user defined networks.

docker network prune

Docker Restart Policies

February 15, 2022 /

To setup restart policy to the container, use following command-

docker run --restart=<<policy option>> <<container>>

Following are the options for the container restart-

  1. no (default)
  2. on-failure
  3. always
  4. unless-specified

Following is the matrix for the restart policies-

* – this will start when the Docker daemon is started

Above is applicable if the container starts successfully

Live Restore

If you want to keep container running if the Docker daemon crashes or stops use the live restore option. This reeduces the container downtime due to daemon crashes or planned outages or upgrades.

Update the /etc/docker/daemon.json in Ubuntu system and add option live-restore:true

Docker FAQs

January 31, 2022 / / 0 Comments

Docker Engine and Architecture FAQ’s

Components of the Docker Engine – Docker Daemon, Rest API and Docker Cli

Component that manages Images, Containers, Volumes and Network – Docker Daemon

Component that manages containers in Docker Engine – LibContainer

Container can run with Docker – Yes

Component keeps alive container even if Docker Daemon is not working – Containerd-Shim

Docker engine objects- Images, Container, Volume and Network

In Container data is writable but not persistable – Yes

Dcoker looks for images in docker hub by default- Yes

Readonly component in Docker engine – Docker Images

Default directory where Docker data is stored (Ubuntu) – /var/lib/docker

Directory where the Docker config is stored(Ubuntu)- /etc/docker

OCI stands for – Open Container Initiative

OCI specification – runtime-spec and image-spec

View version of Docker engine – docker version

Stop the Docker service – systemctl stop docker or/and systemctl stop docker.socket

Start the Docker service – systemctl start docker.socket or/and systemctl start docker

Check Status of Docker service – systemctl status docker

Debug docker whilst starting the service – dockerd –debug

Where is the Daemon file located (Ubuntu) – /etc/docker/daemon.json

Where is the daemon socket located (Ubuntu) – /var/run/docker.sock

Port to connect the docker externaly with encrypted trafic – 2376

Port to connect the docker externaly with unencrypted trafic – 2375

Start the docker daemon manually – dockerd

Default docker daemon interface – Unix Socket

Default network driver – bridge

Stop Command signals running container on STOP command – SIGTERM followed by SIGKILL

Restart policies – no, on-failure, always and unless-stopped

Reduce container downtime due to daemon failure or restart- Enable Live Restore

Docker Images FAQ’s

Default Docker Image Registry – Docker Hub

Various Image Registry –

  • Docker Trusted Registry
  • Google Container Registry
  • Amazon Container Registry
  • Azure Container Registry

Types of Images in Docker Hub

  • Official Images
  • Verified Images
  • User Images

Base vs Parent Image –

Base Image are creatged from scratch, which means its empty. You cannot create a scratch image as it is always to be used. Any other images created from Base Image but used as parent to custom images are Parent Image. e.g. Ubuntu which is made from debian image. Here debian image is a Parent Image

Docker Swarm

What is the maximum and recommended number of mananger a swarm cluste can have? There is no max limit but recommended is 7 managers in swarm cluster

Cache Busting and Version Pinning when building Docker images

January 24, 2022 /
docker file – Layered Architecture

Docker uses Layered Architecture. When using Docker files it creates a new layer in the image which adds additional space to the image based on the instructions for that layer.

When a Docker build command is run it proceeds from the first instruction in Docker file to the last while caching each stage so as if the build fails next time build uses cache until it ran succesully and invalidated the stage that failed and the following stage. Layers repurpose the previous layers and don’t have to build all of them again.

In below example Docker file has 6 stages. Each stage will be cached when build command is ran.

Suppose a build fails at Stage 3 due to some reason or new package has to be added the Docker will invalidate the Stage 3 and the following stages

Next time when a issue is rectified the build command will repurpose the previuos layers and build the failed stages

docker file – Layered Architecture

But in this case the repository will not be update, so how to resolve or update the repository with the packages-

Cache Busting

In this case we can to combine the instructions so the repository is updated along with packages as below

docker file – Cache Busting and Version Pinning

Merging Stage 2 and Stage 3 from the previous docker file in to single instruction will ensure the repository is first udpated and pakages are installed

Merging these stages is called as Cache Busting

Version Pinning

You can also explicity mention the version of package to be installed

In stage 2 docker file is instrcuting to install python3-pip 21.3.1 version

Best Practice-

Instructions which are most frequently modified should be at the bottom of the file and the instructions which are least modified should be at the top of the docker file

Docker storage for Ubuntu

January 19, 2022 /

Docker uses storage drivers to store the read-only images and writable containers

It basically has 6 layers

Read-only/Image Layers

  1. Base Image e.g. Ubuntu OS
  2. Packages/Repositories e.g. apt etc
  3. Dependencies e.g. pip etc
  4. Custom Code e.g. python code etc
  5. Enrtypoint or command i.e. excutes the program

Writable Layer

6. Container Layer

Layers of a container based on the Ubuntu image

Data and files related to images and containers are store in /var/lib/docker folder in Ubuntu

To check the storage driver used by the docker, use following command-

docker info | more

Im my case it is overlay2

You can also use this command to get the storage driver

docker info | grep "Storage Driver"

How to change the storage driver

Stop the Docker service

systemctl stop docker.socket
systemctl stop docker

Check the docker service status

service docker status

Backup the docker folder

cp -au /var/lib/dovker /var/lib/docker.bk

Change the storage driver

echo '{ "storage-driver": "aufs" }' | sudo tee /etc/docker/dameon.json
service docker start

Image credit and reference links –

https://docs.docker.com/storage/storagedriver/

https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Create a file in Linux

December 7, 2021 /

Create a file with touch command

To create a new file, use touch command followed by the name of the file-

This should create a empty file.

touch thirdfile.txt

To create multiple files using touch command-

touch thirdfile-1.txt thirdfile-2.txt

Create a file with cat command

To create a new file with cat command use redirection operator followd by file name.

This will allow to add content to the file

cat > fourthfile.txt

Cratea a file using echo command

To create a new file using echo command use redirection operator followed by file name will create empty file or add content before redirection operator to add content while creating a file.

echo "This is fifth file." > fifthfile.txt

Create a empty file with echo command

echo > sixthfile.txt