Table of Contents
- 1. cat line X to line Y on a huge file
- 2. /dev/null, 2>&1|tee
- 3. env, env -u, env -i
- 4. bash csh
- 5. xargs (128K limit on command lines)
- 6. rm: delete large number of files
- 7. Date and time: To get the current date and time in the helpful ISO 8601 format
- 8. run command until success
- 9. kill -STOP Stopping A Process, ^Z (Ctrl-Z)
- 10. hashbang, venv script Python shell
- 11. text process, remove comment
- 12. python venv
- 13. suspend commands in a whole
- 14. kill SSH bash session leader SIGHUP, controlling terminals
- 15. ssh SSH quoting
- 16. sed path, cut environment
- 17. tmux kill-all-session
- 18. cd
- 19. benchmark repeat
- 20. dd
- 21. device or resource busy
- 22. wget -c, aria2
- 23. Ref
1 cat line X to line Y on a huge file
less +320123 -j 10 filename SHELL=`which zsh` less +320123 -j 10 filename
cat line X to line Y on a huge file
Reading lines 50,000,000-50,000,010 https://unix.stackexchange.com/questions/47407/cat-line-x-to-line-y-on-a-huge-file
# tail+|head is faster by 10-15% than sed tail -n+50000000 test.in | head -n10
tail -n50000001 test.in | head -n10 sed -n '50000000,50000010p;57890010q' test.in
2 /dev/null, 2>&1|tee
cmd > /dev/null 2>&1 cmd 1> /dev/null 2>&1
find . 2>&1 |tee /tmp/output.txt
command | tee file.txt command > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2) (command 2> >(tee /tmp/log >&2) ) >/dev/null
3 env, env -u, env -i
env P4DIFF=vimdiff p4 diff
https://eklitzke.org/environment-variables
$USER
Fall back to the environment by calling getenv() with the string USER. if there's no variable declared called $USER
If you want to add a new variable to the environment you must direct the shell to do so with the export statement. You only need to use export once, and you only need to export variables that actually have meaning to subprocesses.
cat /proc/PID/environ # The values in the environment are separated by null bytes
print open('/proc/self/environ').read().split('\0')
python -c "print open('/proc/self/environ').read().split('\0')"
To update the environment of a running process, attach using GDB and call setenv().
env -i sh env -i bash --noprofile --norc env -i TERM="$TERM" bash --noprofile --norc exec env -i TERM="$TERM" HOME="$HOME" bash --noprofile --norc
# Warning: Tried to connect to session manager, Could not open network socket env -u SESSION_MANAGER vgp4 emulator
https://unix.stackexchange.com/questions/67096/xterm-warning-tried-to-connect-to-session-manager
https://man7.org/linux/man-pages/man5/proc.5.html /proc/self When a process accesses this magic symbolic link, it resolves to the process's own proc[pid] directory.
4 bash csh
csh -c 'source env.csh && make compil' bash -c 'source env.bash && make compil'
bash subshells
(cd /some/other/dir && other-command)
Brace expansion
mv foo.{txt,pdf} some-dir mv main.cc{,.bk} mkdir -p test-{a,b,c}/subtest-{1,2,3}
process substitution: treat output of a command like a file
diff /etc/hosts <(ssh somehost cat /etc/hosts) source <(grep TG_COMP env.csh)
Ignore slow i18n routines
export LC_ALL=C
4.1 csh shell non-interactive modenon-interactive mode
# Ignore this file in non-interactive mode. if ( ! ${?prompt} ) goto end # ... # ... # ... end:
csh -f hello.csh
$ csh --help tcsh 6.18.01 (Astron) 2012-02-14 (x86_64-unknown-linux-gnu) options wide,nls,dl,al,kan,sm,rh,color,filec -f start faster by ignoring the start-up file
#!/bin/csh -f echo ""; echo "[INFO]: Starting '$0' script";
5 xargs (128K limit on command lines)
find . -name '*.py' | xargs grep some_function
6 rm: delete large number of files
mkdir empty && rsync -r --delete empty/ bigDir && rmdir bigDir
7 Date and time: To get the current date and time in the helpful ISO 8601 format
use date -u +"%Y-%m-%dT%H:%M:%SZ" (other options are problematic).
date -u +"%Y-%m-%dT%H:%M:%SZ"
env TZ='Asia/Taipei' date Fri Jul 10 00:08:24 CST 2020 date Thu Jul 9 09:07:39 PDT 2020
dateutils: dateadd, datediff, strptime
/usr/bin/time -f "%E" sleep 10s 0:10.08
8 run command until success
while true do ping -c 1 google.com > /dev/null 2>&1 && break done;
9 kill -STOP Stopping A Process, ^Z (Ctrl-Z)
kill -STOP PID # stop the process kill -CONT PID # resume the process
Only two signals that cannot be caught: SIGSTOP is one, and SIGKILL is the other.
^Z sends a related (but distinct) signal called SIGTSTP. This also stops the process, but differs from SIGSTOP in that it can be caught.
https://eklitzke.org/stopping-a-process
``` kill -0 kill -15 kill -9 ```
``` pwd | curl -F "c=@-" "https://fars.ee/" ```
10 hashbang, venv script Python shell
hashbang bash
#!/usr/bin/env bash pwd
#!/path/to/venv/env/bin/python3 import subprocess as sp from io import StringIO cmd = "pwd" p = sp.run([cmd], stdout=sp.PIPE, shell=True) print(p.stdout.decode()) std_output =StringIO(p.stdout.decode()) print(std_output.read())
https://stackoverflow.com/questions/10376206/what-is-the-preferred-bash-shebang
11 text process, remove comment
cat test.txt | python -c "import sys; map(print, filter(lambda x: x.startswith('#'), sys.stdin))"
12 python venv
#conda
source activate py3_clean
python -m venv frida_env source frida_env/bin/activate
13 suspend commands in a whole
(command1; command2) Note the added parentheses.
14 kill SSH bash session leader SIGHUP, controlling terminals
pstree -p ps j -A
Process groups represented by PGID Sessions represented by SID
SSH bash session leader SIGHUP controlling terminals
Killing all processes in a process group
kill -SIGTERM -- -19701
If we pass a negative number, it kills the process group with that PGID.
Killing all processes in a session
pkill -s <SID>
15 ssh SSH quoting
KO
ssh user@machine.local bash -lc "cd /tmp;pwd"
SSH command-line parsing is not quite what you expect.
OK
ssh user@machine.local 'cd /tmp;pwd' ssh user@machine.local 'bash -lc "cd /tmp;pwd"'
https://www.chiark.greenend.org.uk/~cjwatson/blog/ssh-quoting.html https://news.ycombinator.com/item?id=27483077
16 sed path, cut environment
sed -i.bk s+/path1+/path2+g compile_commands.json
echo $PATH | tr ':' '\n' echo $LD_LIBRARY_PATH | tr ':' '\n'
17 tmux kill-all-session
tmux list-sessions | awk 'BEGIN{FS=":"}{print $1}' tmux list-sessions | awk 'BEGIN{FS=":"}{print $1}' | xargs -n 1 tmux kill-session -t
18 cd
mkdir -p make/new/directory cd !$
Form | Meaning |
$* | $1 $2 $3… |
$@ | $1 $2 $3… |
"$*" | "$1 $2 $3…" |
"$@" | "$1" "$2" "$3"… |
Use the quoted form "$@" if you're unsure. It's almost always what you want.
https://mhoffman.github.io/2015/05/21/how-to-navigate-directories-with-the-shell.html
19 benchmark repeat
time { repeat 1000 { bash empty.sh } } time { repeat 1000 { dash empty.sh } }
20 dd
# Obscure dd version dd if=image.iso of=/dev/sdb bs=4M # Equivalent cat version cat image.iso >/dev/sdb cat image.iso | pv >/dev/sdb
# Obscure dd version dd if=/dev/zero of=image.iso bs=4MB count=25 # Regular head version head -c 100MB /dev/zero >image.iso
https://eklitzke.org/the-cult-of-dd https://eklitzke.org/efficient-file-copying-on-linux
21 device or resource busy
ps lsof +D ./ | awk '{print $2}' | tail -n +2 lsof +D ./ | awk '{print $2}' | tail -n +2 | xargs -r kill -9
https://unix.stackexchange.com/questions/11238/how-to-get-over-device-or-resource-busy
22 wget -c, aria2
alias wget='aria2c -x 16 -s 16 --retry-wait=1'
https://medium.com/starbugs/become-a-command-line-master-together-47a0ad5dd41d
https://aria2.github.io/manual/en/html/aria2c.html -x, –max-connection-per-server=<NUM> -s, –split=<N> –retry-wait=<SEC>
https://medium.com/@evindu/aria2c-%E4%BD%BF%E7%94%A8%E6%89%8B%E6%9C%AD-d2b4fbc972b6
23 Ref
https://github.com/jlevy/the-art-of-command-line
https://blog.csdn.net/ithomer/article/details/9288353 strace, dup
https://zhuanlan.zhihu.com/p/31337815
https://mp.weixin.qq.com/s/AepY9Rxw_lYyuettsEHxZA
https://www.systutorials.com/how-to-make-tee-catch-the-stderr-only-in-linux/
http://morningcoffee.io/killing-a-process-and-all-of-its-descendants.html