SysAdmin.txt

@@@@@@@@@@@@@@@@@@@@@ ALL THINGS SYSADMIN/DEVOPS RELATED @@@@@@@@@@@@@@@@@@@@@@@

top - Process Activity
vmstat - System Activity, H/W info
w - Who logged on
ps - Displays processes
pgrep
free - Memory Usage
iostat - Average CPU Load, Disk Activity
sar - System Activity Reporter
mpstat - Multiprocessor Usage
pmap - Process Memory Usage
netstat - Networks stats
ss - Utility to investigate sockets, similar to netstat
iptraf - Realtime network traffic
tcpdump -
strace - System Calls
/proc - Various Kernel Stats
nagios - Server and Network Monitoring
FAN - Fully Automated Nagios

nmap - Port Scanning
lsof - list open files, network connections and more.
ntop
dig - Swiss Army Knife for DNS querying.

pidstat
mpstat
sar
dstat

ngrep
mitmproxy
opensnoop
eBPF
perf


cpuid - Dump detailed info about CPUs



@@@@@@@@@@@@@@@     PMAP  PROCESS MEMORY  @@@@@@@@@@@@@@@@@@
$ pmap PID



@@@@@@@@@@@@@@@     CRONTAB/CRONJOB       @@@@@@@@@@@@@@@@@@
Format of cron job:

    min(0-59) hour(0-24) day_of_month(1-31) month(1-12) day_of_week(0-6) cmd_to_run
    0 = Sunday
    6 = Saturday
    7 = Sunday again
- Asterisk means every/all instances.

- Manpages
$ man crontab

- Display contents of crontab file
$ crontab -l

- Edit current users cron jobs:
$ crontab -e

- To remove all cron jobs
$ crontab -r


Examples:
---------
- Run cron job every minute
* * * * * cmd_to_run

- Run cron job every 5th minute
*/5 * * * * cmd_to_run

- Every quarter hour
*/15 * * * * cmd_to_run

- Every half hour
30 * * * * cmd_to_run
or
*/30 * * * * cmd_to_run

- Run every hour, 5 mins and 10 mins past every hour
0,5,10 * * * * cmd_to_run

- Run every hour
0 * * * * cmd_to_run

- Run every 2 hours
0 */2 * * * cmd_to_run

- Run a job everyday (at 00:00)
0 0 * * * cmd_to_run

- Run a job everyday at 3am
0 3 * * * cmd_to_run

- Run a job every Sunday (at 00:00)
0 0 * * SUN cmd_to_run
or
0 0 * * 0 cmd_to_run

- Run a job every weekday only
0 0 * * 1-5 cmd_to_run

- Run a job every month (1st day)
0 0 1 * * cmd_to_run

- Run a job at 16:15 on 1st day of month
15 16 1 * * cmd_to_run

- Run a job every quarter i.e., 1st day of every 3rd month
0 0 1 */3 * cmd_to_run

- Run a job on specific month at a specific time
5 0 * 4 * cmd_to_run

- Run a job every 6 months
0 0 1 */6 * cmd_to_run

- Run a job every year
0 0 1 1 * cmd_to_run

Following strings can also be used to define job

We can also use the following strings to define job.
------
@reboot     Run once, at startup.
@yearly     Run once a year.
@annually   (same as @yearly).
@monthly    Run once a month.
@weekly     Run once a week.
@daily      Run once a day.
@midnight   (same as @daily).
@hourly     Run once an hour.

- Run a job everytime server is rebooted
@reboot cmd_to_run

Crontab Sytax Generators
----
https://crontab.guru/
https://crontab-generator.org/


[1] https://www.ostechnix.com/a-beginners-guide-to-cron-jobs/

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

+ How to check linux flavor/version
Method 1:
$ cat /etc/*-release

Method 2:
$ lsb_release -a            # displays LSB (Linux Standard Base) and
                            # distribution specific info.
Method 3:
$ cat /proc/version

---------------------------
+ How to find out kernel version
$ uname -a
$ uname -mrs

---------------------------
@@@@@@@@@@@@@@@@@  SYSTEM PERFORMANCE CHECK @@@@@@@@@@@@@@
// Notes from Brendan Gregg's videos

----- First 60 seconds investigation ------
uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1    // network throughput
sar -n TCP,ETCP 1
top

-------- Detailed -------------
- Performance is slow. What to do?

- 4 types of tools
    * Observability Tools: No impact on system.
        + Basic: vmstat, iostat, mpstat, ps, top ...
        + Intermediate: tcpdump, netstat, nicstat, pidstat, sar, ...
        + Advanced: ss, slaptop, perf_events, ...

    * Benchmarking Tools: Affects the system.
    * Tuning:
    * Static performance tuning:


Basic Tools >>>>>>>>

- uptime
    Get a quick overview on load averages.
- top (or htop or atop)
    %CPU is summed across all CPUs
    Can miss short-lived processes. 'atop' wont.
- ps
    has lots of flags and custom fields can be set.
- vmstat
    'r' is runnable tasks
    Gives high level CPU summary
- iostat (ATT Unix system)
    block I/O (disk) stats.

- mpstat ($ mpstat -P ALL 1)
    multi processor stats
    look for unbalanced (hot) CPUs, which means one thread is hogging CPU.

- free -m
    'buffers': block device I/O cache
    'cached': virtual page cache

Intermediate Tools >>>>>>>

- strace (system call tracer)
    $ strace -tttT -p 313
        * Translates syscall args. Very helpful for solving system usage issues.
    strace is based on ptrace. Has massive overhead.
    Can slow target by > 100x. Use extreme caution.

- tcpdump
    sniff packets for post analysis

- netstat. Note that `netstat` belongs to `net-tools` package in linux which is
  deprecated in favor or `ip` command suite. Use `nstat` which belongs to
  `iproute2` package.

    -s  Various protocol stats
    -i  Interface stats
    -r  route table
    -p  process details
    -c  per-second interval

- netstat statistics
    # All connections on port 80
    $ netstat -anp | grep :80

    # Help
    $ netstat -h

    netstat -s	Display statistics
    netstat -st	Display TCP statistics
    netstat -su	Display UDP statistics


- netstat listening
    netstat -ltunp	All Listening ports
    netstat -ltn	Listening TCP ports
    netstat -lun	Listening UDP ports
    netstat -lx	Listening Unix ports


- netstat connections
    netstat -a	All connections
    netstat -at	All TCP connections
    netstat -au	All UDP connections

- netstat networks
    netstat -i	    Show network interfaces
    netstat -ie	    Show network interfaces extended info


- netstat routing
    netstat -r	    Show routing table
    netstat -rn	    Show routing table, don't resolve hosts



- `nstat` cheat sheet. `nstat` is tool for monitoring kernel SNMP and network
  interface stats and part of iproute2 package.

  $ nstat
  # kernel
  IpInReceives                    12057              0.0
  IpInAddrErrors                  1                  0.0
  IpInDelivers                    12000              0.0
  IpOutRequests                   11060              0.0
  IpOutNoRoutes                   252                0.0
  TcpActiveOpens                  36                 0.0
  TcpEstabResets                  3                  0.0
  TcpInSegs                       11804              0.0
  TcpOutSegs                      10960              0.0
  TcpRetransSegs                  3                  0.0
  TcpOutRsts                      29                 0.0
  UdpInDatagrams                  196                0.0
  UdpOutDatagrams                 246                0.0
  Ip6InReceives                   18                 0.0
  Ip6OutRequests                  31                 0.0
  Ip6OutNoRoutes                  248                0.0
  Ip6InMcastPkts                  18                 0.0
  Ip6OutMcastPkts                 31                 0.0
  Ip6InOctets                     1715               0.0
  Ip6OutOctets                    2791               0.0
  Ip6InMcastOctets                1715               0.0
  Ip6OutMcastOctets               2791               0.0
  Ip6InNoECTPkts                  18                 0.0
  Icmp6OutMsgs                    13                 0.0
  Icmp6OutRouterSolicits          3                  0.0
  Icmp6OutNeighborSolicits        2                  0.0
  Icmp6OutMLDv2Reports            8                  0.0
  Icmp6OutType133                 3                  0.0
  Icmp6OutType135                 2                  0.0
  Icmp6OutType143                 8                  0.0
  Udp6OutDatagrams                18                 0.0
  TcpExtTW                        19                 0.0
  TcpExtDelayedACKs               37                 0.0
  TcpExtTCPHPHits                 10981              0.0
  TcpExtTCPPureAcks               142                0.0
  TcpExtTCPHPAcks                 267                0.0
  TcpExtTCPLossUndo               3                  0.0
  TcpExtTCPTimeouts               3                  0.0
  TcpExtTCPAbortOnClose           3                  0.0
  TcpExtTCPRcvCoalesce            327                0.0
  TcpExtTCPAutoCorking            5                  0.0
  TcpExtTCPSynRetrans             3                  0.0
  TcpExtTCPOrigDataSent           390                0.0
  TcpExtTCPHystartTrainDetect     1                  0.0
  TcpExtTCPHystartTrainCwnd       17                 0.0
  TcpExtTCPWinProbe               1                  0.0
  TcpExtTCPDelivered              424                0.0
  IpExtInMcastPkts                41                 0.0
  IpExtOutMcastPkts               47                 0.0
  IpExtInOctets                   54131913           0.0
  IpExtOutOctets                  829632             0.0
  IpExtInMcastOctets              3902               0.0
  IpExtOutMcastOctets             4142               0.0
  IpExtInNoECTPkts                48401              0.0


- This resource has details about the fields
    - https://www.kernel.org/doc/html/latest/networking/snmp_counter.html
    - https://loicpefferkorn.net/2018/09/linux-network-statistics-reference/

- nstat options
    -a, --ignore
        Dump absolute values of counters. The default calculates the increments
        since the previous run.
    -z, --zeroes
        Dump all zero counters. They are ignored by default.
    -r, --reset
        Resets the history, thus excluding averages since the last run.
    -d, --scan x
        Run in daemon mode to collect statistics. x is an interval between
        measurements in seconds.
    -t, --interval x
        Where x is the interval to average rates. Default value is 60s.



- pidstat
    Very useful process stats, eg, by-thread, disk I/O

- swapon -s
    Show swap device usage.

- lsof

- sar (System Activity Reporter)
    Archive or Live Mode

Advanced Tools >>>>>>>

- ss
    Info about socket stats

- iptraf
    Nice histogram about network pkt size per interface

- iotop

- slabtop
    Kernel slab allocator memory usage

- pcstat (available on Github)

- perf_events
    multi tool with many capabilities
        CPU profiling
        PMC profiling (Performance Monitoring Counters) - very low level
        static and dynamic tracing

- tiptop
    front end to PMC

- rdmsr
    * Model Specific Registers (MSRs), unlike PMCs, can be read by default in Xen
    guests: timestamp clock, temp, power ...
    * Use rdmsr from the msr-tools package to read them.

More Advanced Tools >>>>>>>
    ltrace      Library call tracer
    ethtool     Mostly interface tuning; some stats
    snmpget     SNMP network host statistics
    lldptool    Can get LLDP broadcast stats
    blktrace    Block I/O event tracer
    /proc       Many raw kernel counters
    pmu-tools   On- and off-core CPU counter tools


Benchmarking >>>>>>>
- lmbench
    * CPU, memory and kernel micro-benchmarks

- fio
    * File System or disk I/O micro-benchmarking

- pchar
    * traceroute with bandwidth per hop

- ethtool -S eth0
    root@webserver1-2:~# ethtool -S eth0
    NIC statistics:
         rx_queue_0_packets: 1383000
         rx_queue_0_bytes: 88404885
         rx_queue_0_drops: 0
         rx_queue_0_xdp_packets: 0
         rx_queue_0_xdp_tx: 0
         rx_queue_0_xdp_redirects: 0
         rx_queue_0_xdp_drops: 0
         rx_queue_0_kicks: 22
         tx_queue_0_packets: 6000
         tx_queue_0_bytes: 423757
         tx_queue_0_xdp_tx: 0
         tx_queue_0_xdp_tx_drops: 0
         tx_queue_0_kicks: 5719



root@webserver1-2:~# ip -s link show ens4
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:be:77:d2 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast
      88408725 1383060      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
        423757    6000      0       0       0       0
    altname enp0s4




Tuning Tools >>>>>>>
- Read Slides [4]

Profiling Tools >>>>>
- Read Slides [4]



[1] https://www.youtube.com/watch?v=ZdVpKx6Wmc8
[2] https://www.youtube.com/watch?v=FJW8nGV4jxY
[3]
[4]
http://cdn.oreillystatic.com/en/assets/1/event/122/Linux%20perf%20tools%20Presentation.pdf
[5] https://github.com/brendangregg/msr-cloud-tools

@@@@@@@@@@@@@@@   rsync   commands @@@@@@@@@@@@@@@@@@@@@@@

-v: verbose
-r: recursive copy of data. Doesn't keep permission or timestamp
-a: archive.
    Features of this flag:
    * Preserves timestamp, file permissions, user & group ownership
    * Copies files recursively
    * Copies symlinks
    + -a flag is superset of -r flag, looks like
-z: compress data
-h: Output numbers in human readable format

rsync [options] <src> <dst>

Copy/Sync a File on a Local Computer
$ rsync -zvh backup.tar /tmp/backups/

Copy/Sync a Directory on Local Computer
$ rsync -avzh /root/rpmpkgs /tmp/backups/

Copy a Directory from Local Server to a Remote Server
$ rsync -avz rpmpkgs/ root@192.168.0.101:/home/

Copy/Sync a Remote Directory to a Local Machine
$ rsync -avzh root@192.168.0.100:/home/tarunika/rpmpkgs /tmp/myrpms

Copy a File from a Remote Server to a Local Server with SSH
$ rsync -avzhe ssh --progress root@192.168.0.100:/root/install.log /tmp/

Excluding Dirs
---
To exclude directories, use --exclude=<relative-path> flag. Note that the
directory is relative to the source

$ rsync -a --exclude=/some/location/ /path/to/source /path/to/target

Excluded dir is here: /path/to/source/some/location/



Reference:
[1] https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/
[2]
https://unix.stackexchange.com/questions/2161/rsync-filter-copying-one-pattern-only/2503#2503
[3]

@@@@@@@@@@@@@@@@@@@@@ TAR commands @@@@@@@@@@@@@@@@@@@@@@@
+ 'tar' command examples [2]
// Note that '-' is a mess and may or may not be used.

1) Create tar Archive File
    $ tar -cvf example.tar /home/example/

    c - Create a new .tar archive file.
    v - Verbose
    f - File name is provided (example.tar)

2) Create tar.gz Archive File
    $ tar -cvzf example.tar.gz /home/example/
    OR
    $ tar -cvzf example.tgz /home/example/

3) Create a tar.bz2 Archive File
    $ tar -cvfj example.tar.bz2 /home/example/
    OR
    $ tar -cvfj example.tar.tbz /home/example/
    OR
    $ tar -cvfj example.tar.tb2 /home/example/

4) Untar tar (or tar.gz or tar.bz2) Archive File
    // NOTE: Untar in current directory
    $ tar -xvf example.tar
    OR
    $ tar -xvf example.tar -C /destination/directory/

    $ tar -xvf example.tar.gz

    $ tar -xvf example.tar.bz2

5) List Content of tar (or tar.gz or tar.bz2) Archive File
    $ tar -tvf example.tar
    $ tar -tvf example.tar.gz
    $ tar -tvf example.tar.bz2

6) Untar one or more files from tar Archive File
    $ tar -xvf example.tar 1.txt [2.txt ...]  //1.txt is part of the archive.
    OR
    $ tar --extract --file=example.tar 1.txt

7) Untar one or more files from tar.gz
    $ tar -zxvf example.tar.gz 1.txt
    OR
    $ tar --extract --file=example.tar.gz 1.txt

8) Untar one or more files from tar.bz2
    $ tar -jcvf example.tar.bz2 1.txt
    OR
    $ tar --extract --file=example.tar.bz2 1.txt

9) Extract group of files using Wildcard
    $ tar -xvf example.tar --wildcards '*.c'
    $ tar -zxvf example.tar.gz --wildcards '*.c'
    $ tar -jxvf example.tar.bz2 --wildcards '*.c'

10) Add files or directories to tar Archive
    $ tar -rvf example.tar 1.txt
    OR
    $ tar -rvf example.tar php   //php is a directory here

// tar command CANNOT add files/directories to .tar.gz and tar.bz2
// compressed files.

---------------------------
@@@@@@@@@@@@@@@@@@@@@ FIND commands @@@@@@@@@@@@@@@@@@@@@@@
Find Command Usage [3]

Part I – Basic Find Commands for Finding Files with Names

1. Find Files Using Name in Current Directory
Find all the files whose name is tecmint.txt in a current working
directory.
    $ find . -name tecmint.txt

2. Find Files Under Home Directory
Find all the files under /home directory with name tecmint.txt.
    $ find /home -name tecmint.txt

3. Find Files Using Name and Ignoring Case
Find all the files whose name is tecmint.txt and contains both capital and
small letters in /home directory.
    $ find /home -iname tecmint.txt

4. Find Directories Using Name
Find all directories whose name is Tecmint in / directory.
    $ find / -type d -name Tecmint

5. Find PHP Files Using Name
Find all php files whose name is tecmint.php in a current working
directory.
    $ find . -type f -name tecmint.php

6. Find all PHP Files in Directory
Find all php files in a directory.
    $ find . -type f -name "*.php"

Part II – Find Files Based on their Permissions

7. Find Files With 777 Permissions
Find all the files whose permissions are 777.
    $ find . -type f -perm 0777 -print

8. Find Files Without 777 Permissions
Find all the files without permission 777.
    $ find / -type f ! -perm 777

9. Find SGID Files with 644 Permissions
Find all the SGID bit files whose permissions set to 644.
    $ find / -perm 2644

10. Find Sticky Bit Files with 551 Permissions
Find all the Sticky Bit set files whose permission are 551.
    $ find / -perm 1551

11. Find SUID Files
Find all SUID set files.
    $ find / -perm /u=s

12. Find SGID Files
Find all SGID set files.
    $ find / -perm /g+s

13. Find Read Only Files
Find all Read Only files.
    $ find / -perm /u=r

14. Find Executable Files
Find all Executable files.
    $ find / -perm /a=x

15. Find Files with 777 Permissions and Chmod to 644
Find all 777 permission files and use chmod command to set permissions to
644.
    $ find / -type f -perm 0777 -print -exec chmod 644 {} \;

16. Find Directories with 777 Permissions and Chmod to 755
Find all 777 permission directories and use chmod command to set
permissions to 755.
    $ find / -type d -perm 777 -print -exec chmod 755 {} \;

17. Find and remove single File
To find a single file called tecmint.txt and remove it.
    $ find . -type f -name "tecmint.txt" -exec rm -f {} \;

18. Find and remove Multiple File
To find and remove multiple files such as .mp3 or .txt, then use.
    $ find . -type f -name "*.txt" -exec rm -f {} \;
    OR
    $ find . -type f -name "*.mp3" -exec rm -f {} \;

19. Find all Empty Files
To file all empty files under certain path.
    $ find /tmp -type f -empty

20. Find all Empty Directories
To file all empty directories under certain path.
    $ find /tmp -type d -empty

21. File all Hidden Files
To find all hidden files, use below command.
    $ find /tmp -type f -name ".*"


Part III – Search Files Based On Owners and Groups

22. Find Single File Based on User
To find all or single file called tecmint.txt under / root directory of
owner root.
    $ find / -user root -name tecmint.txt

23. Find all Files Based on User
To find all files that belongs to user Tecmint under /home directory.
    $ find /home -user tecmint

24. Find all Files Based on Group
To find all files that belongs to group Developer under /home directory.
    $ find /home -group developer

25. Find Particular Files of User
To find all .txt files of user Tecmint under /home directory.
    $ find /home -user tecmint -iname "*.txt"


Part IV – Find Files and Directories Based on Date and Time

26. Find Last 50 Days Modified Files
To find all the files which are modified 50 days back.
    $ find / -mtime 50

27. Find Last 50 Days Accessed Files
To find all the files which are accessed 50 days back.
    $ find / -atime 50

28. Find Last 50-100 Days Modified Files
To find all the files which are modified more than 50 days back and less
than 100 days.
    $ find / -mtime +50 –mtime -100

29. Find Changed Files in Last 1 Hour
To find all the files which are changed in last 1 hour.
    $ find / -cmin -60

30. Find Modified Files in Last 1 Hour
To find all the files which are modified in last 1 hour.
    $ find / -mmin -60

31. Find Accessed Files in Last 1 Hour
To find all the files which are accessed in last 1 hour.
    $ find / -amin -60


Part V – Find Files and Directories Based on Size

32. Find 50MB Files
To find all 50MB files, use.
    $ find / -size 50M

33. Find Size between 50MB – 100MB
To find all the files which are greater than 50MB and less than 100MB.
    $ find / -size +50M -size -100M

34. Find and Delete 100MB Files
To find all 100MB files and delete them using one single command.
    $ find / -size +100M -exec rm -rf {} \;

35. Find Specific Files and Delete
Find all .mp3 files with more than 10MB and delete them using one single
command.
    $ find / -type f -name *.mp3 -size +10M -exec rm {} \;

Part VI - Exclude directories during search [4]

@@@@@@@@@@@@@@@@@@@@@ CUT commands @@@@@@@@@@@@@@@@@@@@@@@
$ cut -c1 data.txt      // Select (or cut) 1st character/column of chars
$ cut -c1-4 data.txt    // Select 1 to 4 chars
$ cut -c-6 data.txt     // Select first 6 chars
$ cut -c4- data.txt     // Select from 4th char to end
$ cut -f 2 data.txt     // Select 2nd field (TAB is default delim)
$ cut -f 1 -d ':' data.txt  // Delim is ':' now
$ cut -f 1,4 data.txt   // Select 1 and 4th field
$ cut -f 1-4 data.txt   // 1 to 4 fields
$ cut -f 2- data.txt    // 2nd to end fields
$ cut -f 1,4-6,8 data.txt   // Get 1, 4 to 6 and 8th fields

Now it's intuitive. Playing around with chars (c) and field (f) flags


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                    LINUX NETWORKING COMMANDS
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

ipcalc
------
$ ipcalc 192.168.0.1/255.255.128.0
Address:   192.168.0.1          11000000.10101000.0 0000000.00000001
Netmask:   255.255.128.0 = 17   11111111.11111111.1 0000000.00000000
Wildcard:  0.0.127.255          00000000.00000000.0 1111111.11111111
=>
Network:   192.168.0.0/17       11000000.10101000.0 0000000.00000000
HostMin:   192.168.0.1          11000000.10101000.0 0000000.00000001
HostMax:   192.168.127.254      11000000.10101000.0 1111111.11111110
Broadcast: 192.168.127.255      11000000.10101000.0 1111111.11111111
Hosts/Net: 32766                 Class C, Private Internet


iproute2 commands [5]
--------
show all addresses
    $ ip address show

show address for a single intf
    $ ip address show ${interface name}

show running intf
    $ ip address show up

show statically configured address
    $ ip address show [dev ${interface}] permanent

show dynamically configured address
    $ ip address show [dev ${interface}] dynamic

Add an address to an interface
    $ ip address add ${address}/${mask} dev ${intf name}

Ex:
    ip address add 192.0.2.10/27 dev eth0
    ip address add 2001:db8:1::/48 dev tun10

Add an address with human-readable description
    $ ip address add ${address}/${mask} dev ${interface name} label ${interface name}:${description}

Ex:
    $ ip address add 192.0.2.1/24 dev eth0 label eth0:my_wan_address

Delete an address
    $ ip address delete ${address}/${prefix} dev ${interface name}

Remove all addresses from an interface
    $ ip address flush dev ${interface name}

View all routes
    $ ip route
    $ ip route show

View routes to a network and all its subnets
    $ ip route show to root ${address}/${mask}
    $ ip route show to root 192.168.0.0/24

View routes to a network and all its supernets
    $ ip route show to match ${address}/${mask}
    $ ip route show to match 192.168.0.0/24

View routes to exact subnet
    $ ip route show to exact ${address}/${mask}

View route actually used by kernel
    $ ip route get ${address}/${mask}

View route cache
    $ ip route show cached

Add a route via gateway
    $ ip route add ${address}/${mask} via ${next hop}
    $ ip route add 192.0.2.128/25 via 192.0.2.1

Add a route via interface
    $ ip route add ${address}/${mask} dev ${interface name}

Change/Replace/Delete a route
    $ ip route change ${rest of route statement}
    $ ip route replace ${rest of route statement}
    $ ip route delete ${rest of route statement}

Default route
    $ ip route add default via ${address}/${mask}
    $ ip route add default dev ${interface name}

Blackhole routes
    $ ip route add blackhole ${address}/${mask}

Special routes
    $ ip route add unreachable ${address}/${mask}
    $ ip route add prohibit ${address}/${mask}
    $ ip route add throw ${address}/${mask}

Routes with different metric
    $ ip route add ${address}/${mask} via ${gateway} metric ${number}
    $ ip route add 192.168.2.0/24 via 10.0.1.1 metric 5
    $ ip route add 192.168.2.0 dev ppp0 metric 10

Multipath routing
    $ ip route add ${addresss}/${mask} nexthop via ${gateway 1} weight ${number}
nexthop via ${gateway 2} weight ${number}

    $ ip route add default nexthop via 192.168.1.1 weight 1 nexthop dev ppp0 weight 10


Show info about all links
    $ ip link show
    $ ip link list

Show info about specific link
    $ ip link show dev ${intf name}

Bring up/down a link
    $ ip link set dev ${intf name} up
    $ ip link set dev ${intf name} down

Set human-readable link description
    $ ip link set dev ${intf name} alias "${description}"
    $ ip link set dev eth0 alias "LAN interface"

Rename an interface
    $ ip link set dev ${old name} name ${new name}

Change link layer address (usually MAC)
    $ ip link set dev ${intf name} address ${address}

Change link MTU
    $ ip link set dev ${intf name} mtu ${MTU value}

Delete link
    $ ip link delete dev ${intf name}


Enable or disable multicast on an interface
    $ ip link set ${interface name} multicast on
    $ ip link set ${interface name} multicast off


Enable or disable ARP on an interface
    $ ip link set ${interface name} arp on
    $ ip link set ${interface name} arp off


Create a VLAN intf
    $ ip link add name ${VLAN interface name} link ${parent interface name} type vlan id ${tag}

    $ ip link add name eth0.110 link eth0 type vlan id 110

QinQ
    $ ip link add name ${service interface} link ${physical interface} type vlan proto 802.1ad id ${service tag}

    $ ip link add name ${client interface} link ${service interface} type vlan proto 802.1q id ${client tag}

Ex:
    $ ip link add name eth0.100 link eth0 type vlan proto 802.1ad id 100 # Create service tag interface
    $ ip link add name eth0.100.200 link eth0.100 type vlan proto 802.1q id 200 # Create client tag interface


Create pseudo-ethernet (aka macvlan) interface
    $ ip link add name ${macvlan interface name} link ${parent interface} type macvlan
    $ ip link add name peth0 link eth0 type macvlan

Create dummy intf
    $ ip link add name ${dummy intf name} type dummy

Create a bridge intf
    $ ip link add name ${bridge name} type bridge

Add an intf to bridge
    $ ip link set dev ${intf name} master ${bridge name}

Remove intf from bridge
    $ ip link set dev ${intf name} nomaster

Create a bonding intf
    $ ip link add name ${name} type bond
    // read documentation as bonding is extensive topic

Create an Intermediate Functional Block interface (IFB)
    $ ip link add ${intf name} type ifb
    // RTFM for more. It's extensive topic.


Network Namespace Management
---------

Create a namespace
    $ ip netns add ${namespace name}

List existing namespaces
    $ ip netns list

Delete namespace
    $ ip netns delete ${namespace name}

Run a process inside a namespace
    $ ip netns exec ${namespace name} ${command}
    $ ip netns exec foo /bin/sh

List all processes assigned to a namespace
    $ ip netns pids ${namespace name}

Identify process' primary namespace
    $ ip netns identify ${pid}

Assign network intf to a namespace
    $ ip link set dev ${intf name} netns ${namespace name}

Connect one namespace to another
    # look up the reference link

Monitor network namespace subsystem events
    $ ip netns monitor


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                        PING/PING6
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
/* pinging a loopback addr. The -I is essential to ping loop-back intf */
$ ping -c 5 -I eth0 <ipv6-loop-back-addr>


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                    strace Examples/Notes
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
// Trace certain syscalls
$ strace -e trace=socket, sendmsg ./a.out

// Trace a running process
$ strace -p [PID]

// Trace child processes
$ strace -f ./a.out

// Longer output strings with -s [num]. Default is 32 chars
$ strace -s 2000 ./.a.out

// Count syscalls
$ strace -c ./a.out

// Time spent on each call
$ strace -T ./a.out

// Log output to a file
$ strace -o out.txt ./a.out



---------------------------
Ref:
[1] http://www.cyberciti.biz/faq/find-linux-distribution-name-version-number/
[2] http://www.tecmint.com/18-tar-command-examples-in-linux/
[3] http://www.tecmint.com/35-practical-examples-of-linux-find-command/
[4] http://www.theunixschool.com/2012/07/find-command-15-examples-to-exclude.html
[5] http://baturin.org/docs/iproute2/

@@@@@@@@@@@@@@@@@@@@@@ WORKING WITH "patch" COMMAND  @@@@@@@@@@@@@@@@@@@@@@@@@@
Generate Diff
-------------
diff -purN original.c changed.c > patch.txt

/* p = print relevant C function
 * u = Output in unified context
 * r = recursive compare subdirectories
 * N = treat absent files as empty
 * -x PAT = exclude files that match PAT
 * b = ignore space change
 * w = ignore all white spaces in comparison
 * B = ignore all blank line changes/diffs
 */

Apply Patch
-----------
// applying patch to original.c (or original directory)
patch --dry-run --ignore-whitespace -p0 -i patch.txt

patch --ignore-whitespace -p0 -i patch.txt
n
On Git
--------

Generating patch:
git format-patch [<commit criteria>] --stdout > patch.txt

Example:
// get the last committed changes
git format-patch -1 --stdout > patch.txt

// get the last two committed changes
git format-patch -2 --stdout > patch.txt

// Changes between origin & HEAD
git format-patch origin HEAD --stdout > patch.txt



Check the patch:
git apply --stat patch.txt

Dry run
git apply --check patch.txt

Applying the patch.
git am --signoff < patch.txt


@@@@@@@@@@@@@@@@@@@@@@@@@   ARP     NOTES   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
// Add static entry. To reach IP <ip>, use destination mac as <mac>
$ sudo arp -s <ip> <mac>

//
@@@@@@@@@@@@@@@@@@@@@@@@@   TCPDUMP NOTES   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
SYNTAX:   Protocol Direction  Host(s)   Value Operation   Other Expression
EXAMPLE:     tcp     dst      10.1.1.1    80     and      tcp dst 10.2.2.2

Protocol: ether, fddi, ip, arp, rarp, etc

Direction: src, dst, "src and dst", "src or dst". If nothing is specified,
"src or dst" is applied. For example, "host 2.2.2.2" indicates "src or dst
host 2.2.2.2".


---------------------------------------
$ tcpdump   // To display standard tcpdump output
$ tcpdump -v    // Verbose output
$ tcpdump -D    // Available interfaces for capture
$ tcpdump -n    // Display numerical addresses instead of DNS addresses
$ tcpdump -q    // Quick output
$ tcpdump -i eth0   // Traffic on eth0 interface
$ tcpdump udp   // Capture only UDP traffic
$ tcpdump port http // Capture port 80 http traffic
$ tcpdump -F file_name  // Capture traffic from filter stored in file_name
$ tcpdump -c 20 // stop the capture after 20 packets
$ tcpdump -w capture.log    // Send the capture in to the file
$ tcpdump -r capture.log    // Read the log file
$ tcpdump host www.openmaniak.com   // Display packets having
                                    // "www.openmaniak.com" as src or dst addr.

$ tcpdump src 192.168.1.100 and dst 192.168.1.2 and port ftp
// Display FTP packets with given src and dst addr

@@@@@@@@@@@@@@@@@@@@@@@@@   dig     NOTES   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

$ dig www.google.com // DNS record from a Name Server
$ dig +trace www.google.com // Trace of where all the Name Server got the record
                            // from. This bypasses all resolvers. Use this to
                            // ignore all caches and get directly from NS.

$ dig -x <ip-addr>          // Reverse lookup.
$ dig example.com NS +short // Find the authoritative nameservers.
$ dig @[nameserver] example.com       // Query directly the nameserver
$ dig @8.8.8.8 example.com  // Try Google's resolver
$ dig @1.1.1.1 example.com


@@@@@@@@@@@@@@@@@@@@@@@@@   nicstat notes   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
$ nicstat -l    // list interfaces
$ nicstat -i <intf-name>    // stats on that intf
$ nicstat -x    // extended output
$ nicstat -s    // a summary. Gives reads/s and writes/s. Concise.
$ nicstat 1     // print every second
$ nicstat 1 5   // print every sec for 5 times
$ nicstat -t    // TCP stats
$ nicstat -u    // UDP stats
$ nicstat -S [intf:speed[fd|hd]]    // read manpage



@@@@@@@@@@@@@@@@@@@@@@@@@   WIRESHARK    NOTES  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Display Filters
----
- Protocol pkts, just type their name: ssh, tcp, udp etc
    - dns or http: filter either packets
    - dns and http: filter packets having both protocols

- Multiple filters
    tcp and (ip.src==15.111.202.135 and ip.dst==15.116.49.238)
- tcp.analysis.flags
    * shows all bad TCP flows. Problem could be anything
    * Very useful filter
- ip.addr == 10.0.0.145 (source or destination IP addr)
- tcp.port == 443
- !(arp or dns or icmp)
    * Remove arp, dns and icmp from tracefiles
- Follow a TCP stream
    * Right click one packet in the stream
    * "Follow TCP stream"
- 'tcp contains facebook'
    * Searches all TCP streams with clear text containing 'facebook' text
    * Can do same for UDP stream also
- 'http.request'
    * All HTTP requests (GET. POST etc) are shown
    * 'http.response'
    * http.response.code == 200
- tcp.flags.syn == 1
- tcp.flags.reset == 1


[1] https://wiki.wireshark.org/DisplayFilters
[2] https://www.youtube.com/watch?v=68t07-KOH9Y

@@@@@@@@@@@@@@@@@@@@@@@@@   BASH   SHELL  NOTES  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- Arguments passed to bash scripts are called "Positional Parameters"
- Some special parameters exists, given below

$0      the first positional parameter, equivalent to argv[0] in C
$FUNCNAME   the function name (attention: inside a function, $0 is still the $0 of the shell, not the function name)
$1 … $9 the argument list elements from 1 to 9
${10} … ${N}    the argument list elements beyond 9 (note the parameter expansion syntax!)
$*  all positional parameters except $0
$@  all positional parameters except $0
$#  the number of arguments, not counting $0


@@@@@@@@@@@@@@@@@@@@@@@@@   WINDOWS   NETWORKING   CMDS   @@@@@@@@@@@@@@@@@@@@@@@
$ tracert <IP>      // traceroute equivalent command
$ route print       // ip route show equivalent command
$ ifconfig   /displaydns    // DNS cache maintained locally

@@@@@@@@@@@@@@@@@@@@@@@@@   MTR     NOTES   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
$ mtr - a network diagnostic tool


@@@@@@@@@@@@@@@@@@@@@@@@@   CURL    NOTES   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

[1] POST using JSON file
curl -v -H "Content-Type: application/json" -X POST --data @params.json -u your_username:your_password http://localhost:8000/env/add_server

[2]


References:
- StackOverflow


@@@@@@@@@@@@@@@@@@@    SSH     PORT     FORWARDING      @@@@@@@@@

- A simple SSH connection through a firewall:
$ ssh user1@HostB

                                      +--+
                                      |  |
    +----------------------+          |  |           +-----------------------+
    |  Host A              |          |  |           |   Host B              |
    |                      |          |  |           |                       |
    |+-------------+       |          |  |        22 | +-------------+       |
    || SSH Client  |       +--->------+--+-------->--+ | SSH Server  |       |
    ++-------------+-------+          |  |           +-+-------------+-------+
                                      |  |
                                      +--+

                                    Firewall




- Here, Imap client on Host C wants to connect to Imap Server on Host D, but
  the firewall is blocking them.

- SSH port forwarding can help:
$ ssh -g -L2001:HostD:143 user1@HostB
$ Client on Host C has to run IMAP HostA:2001


+----------------------+                         +-----------------------+
|   Host C             |                         |     Host D            |
|                      |                         |                       |
|                      |                    143  |                       |
|  IMAP Client         +->--+     +--+       +->-+---IMAP Server         |
|                      |    |     |  |       |   |                       |
+----------------------+    |     |  |       ^   +-----------------------+
                            |     |  |       |
                            v     |  |       |
                            |     |  |       |
+----------------------+    |     |  |       |   +-----------------------+
|  Host A              |    |     |  |       ^   |   Host B              |
|                      |    |     |  |       |   |                       |
|                      |2001|     |  |       |   |                       |
|      +------<--------+--<-+     |  |       +---+---<---+               |
|      |               |          |  |           |       |               |
|      v               |          |  |           |       ^               |
|      |               |          |  |        22 |       |               |
|  SSH Client----->----+---->------------->------+-- SSH Server          |
+----------------------+          |  |           +-----------------------+
                                  |  |
                                  +--+

                                Firewall


- Lets break this command down


$ ssh -g -L2001:HostD:143 user1@HostB

user1@HostB     Authenticate with HostB SSH server, as usual
-L              Enable local port forwarding
2001            Listening port on client
143             Target port to which to forward traffic
HostD           Target host to which to forward traffic
-g              Allow remote connections (Gateway Ports)
                This is the most liberal SSH port forwarding




- If you remove '-g' option then local port fowarding is limited as below
$ ssh -L2001:localhost:143 user1@HostB
$ IMAP Client on Host A has to run IMAP locahost:2001

                                  +--+
+----------------------+          |  |           +-----------------------+
|  Host A              |          |  |           |   Host B              |
|                      |          |  |       143 |                       |
|  IMAP client ---->---+->--+     |  |       +->-+---IMAP Server         |
|                      |    |     |  |       |   |                       |
|                      |    v     |  |       |   |                       |
|                      |    |     |  |       ^   |                       |
|                      |2001|     |  |       |   |                       |
|       +-----<--------+--<-+     |  |       +-<-+-----<---+             |
|       |              |          |  |           |         |             |
|       |              |          |  |           |         ^             |
|       |              |          |  |        22 |         |             |
|  SSH Client----->----+--->-------------->------+   SSH Server          |
+----------------------+          |  |           +-----------------------+
                                  +--+
                                Firewall


- Local Forwarding but separate destination host

# ssh -L2001:HostD:143 user1@HostB

                                                 +-----------------------+
                                                 |     Host D            |
                                            143  |                       |
                                  +--+       +->-+-- IMAP Server         |
                                  |  |       |   |                       |
                                  |  |       |   +-----------------------+
                                  |  |       ^
+----------------------+          |  |       |   +-----------------------+
|  Host A              |          |  |       |   |   Host B              |
|                      |          |  |       |   |                       |
|  IMAP Client --->----+->--+     |  |       |   |                       |
|                      |    |     |  |       ^   |                       |
|                      |    v     |  |       |   |                       |
|                      |2001|     |  |       |   |                       |
|      +------<--------+--<-+     |  |       +-<-+--<----+               |
|      |               |          |  |           |       |               |
|      |               |          |  |        22 |       |               |
|  SSH Client ---->----+----->------------>------+-- SSH Ser^er          |
+----------------------+          |  |           +-----------------------+
                                  |  |
                                  +--+

                                Firewall



Remote Port Forwarding
$ ssh -R2001:localhost:143 user1@HostB


                                  +--+
+----------------------+          |  |           +-----------------------+
|  Host A              |          |  |           |   Host B              |
|                      |  143     |  |           |                       |
|  IMAP Server ----<------<-+     |  |      +-<--+-<--  IMAP Client      |
|                      |    |     |  |      |    |                       |
|                      |    ^     |  |      v    |                       |
|                      |    |     |  |      |    |                       |
|                      |    |     |  |      |    |                       |
|       +------>--------->--+     |  |      +-->----->---+               |
|       |              |          |  |    2001   |       |               |
|       ^              |          |  |           |       |               |
|       |              |          |  |        22 |       v               |
|  SSH Client+----<----+---<--------------<------+--<-- SSH Server       |
+----------------------+          |  |           +-----------------------+
                                  |  |
                                  +--+

                                Firewall


- SSH Flags so far: -L, -R, -g
- Equivalent confis in ssh_config file:
	* LocalForward
	* RemoteForward
	* GatewayPorts