passwd and shadow file

password and shadow file.

/etc/passwd file:

smithj:x:561:561:Joe Smith:/home/smithj:/bin/bash

Each field in a passwd entry is separated with ":" colon characters, and are as follows:

• Username, up to 8 characters. Case-sensitive, usually all lowercase
• An "x" in the password field. Passwords are stored in the ``/etc/shadow'' file.
• Numeric user id. This is assigned by the ``adduser'' script. Unix uses this field, plus the following group field, to identify which files belong to the user.
• Numeric group id. Red Hat uses group id's in a fairly unique manner for enhanced file security. Usually the group id will match the user id.
• Full name of user. I'm not sure what the maximum length for this field is, but try to keep it reasonable (under 30 characters).
• User's home directory. Usually /home/username (eg. /home/smithj). All user's personal files, web pages, mail forwarding, etc. will be stored here.
• User's "shell account". Often set to ``/bin/bash'' to provide access to the bash shell (my personal favorite shell).

Perhaps you do not wish to provide shell accounts for your users. You could create a script file called ``/bin/sorrysh'', for example, that would display some kind of error message and log the user off, and then set this script as their default shell.

Note: Note: If the account needs to provide "FTP" transfers to update web pages, etc. then the shell account will need to be set to ``/bin/bash'' -- and then special permissions will need to be set up in the user's home directory to prevent shell logins. See Section 7.1 for details on this.

The ``/etc/shadow'' file contains password and account expiration information for users, and looks like this:

smithj:Ep6mckrOLChF.:10063:0:99999:7:::

As with the passwd file, each field in the shadow file is also separated with ":" colon characters, and are as follows:

• Username, up to 8 characters. Case-sensitive, usually all lowercase. A direct match to the username in the /etc/passwd file.
• Password, 13 character encrypted. A blank entry (eg. ::) indicates a password is not required to log in (usually a bad idea), and a ``*'' entry (eg. :*:) indicates the account has been disabled.
• The number of days (since January 1, 1970) since the password was last changed.
• The number of days before password may be changed (0 indicates it may be changed at any time)
• The number of days after which password must be changed (99999 indicates user can keep his or her password unchanged for many, many years)
• The number of days to warn user of an expiring password (7 for a full week)
• The number of days after password expires that account is disabled
• The number of days since January 1, 1970 that an account has been disabled
• A reserved field for possible future use
  
  
  [ Original link: http://www.tldp.org/LDP/lame/LAME/linux-admin-made-easy/shadow-file-formats.html ]
  
  
  
  
  

understanding iostat

understanding iostat

 The "iostat" command shows average cpu time since the system was stared (similar to uptime0. It also creates a report of the activites of the disk sub system of the server in two parts:
( cpu utilization and device utilization. )



The CPU utilization report has four sections:

 %user: shows the percentage of cpu utilization that was taken up while executing at the user level applications.

 %nice: shows the percentages of cpu utilization that was taken up while executing at the user level with a nice priority.

 %sys: Shows the percentage of CPU utilization that was taken up while executing at the system level (kernel)

 %idel: shows the percentage of time the cpu was idle.


The Device utilization report has these sections:

 Device: The name of the block device.

 tps: The number of transfers per second (I/O requests per second) to the device. Multiple single I/O requests can be combined in a transfer request, because of transfer request can have different size.

 Blk_read/s, Blk_write/s: Blocks read and written per second indicate data read from or written to the device in seconds. Block can also have different sizes. Typical sizes are 1024, 2048 and 4096 bytes, depending on the partition size.

 # dumpe2fs -h /dev/sda1 | grep -F "Block size"

 Blk_read, Blk_wrtn: Indicates the total number of blocks read and written since the boot.


The "iostat" can use many options, the most useful one is "-x" option from the performance prespective.




#iostat -d -x sdb1

 rrpm/s, wrpm/s: The number of read/write requests merged per second that were issued to the device. Multiple single I/O requests can be mearged  in a transfer request, because a transfer request can have different size.


 r/s, w/s: The # of read/write requests that were issued to the device per seconds.

 rsec/s, wsec/s: The number of sectors read/write from the device per second.

 rkB/s, wkB/s: The number of kilobytes read/write from the device per second.

 avgrq-sz: The average size of the requests that were issued to the device. This value is displayed in sectors.

 avgqu-sz: The average queue leangth of the request that were issued to the device.

 await: show the percentage of cpu utilization that was used while executing at the system level (kernel)

 svctm: The average service time (In miliseconds) for I/O requests that were issued to the device.

 %utils: percentage of cpu time during which I/O requests were issued to the device (band width) utilization for the device). Device saturation occurs when this value is close to 100%

understanding vmstat

understanding vmstat

vmstat provides information about "processes", "memory", "paging", "block I/O", "traps" and "cpu" activity.

vmstat -n 2 10 [ generates vmstat 10 times with a sampling rate of two seconds. ]

process (procs):
 r: The number of processes waiting for runtime.
 b: The number of processes in uninterpretable sleep.

Memory:
 swpd: the amount of virtual memory used (KB)
 free: The amount of idle memory (KB)
 buff: the amount of memory used as buffers (KB)
 cache: The amount of memory used as cache (KB)

swap:
 si: Amount of memory swapped from the disk. (KB/s)
 so: Amount of memory swapped to the disk. (KB/s)

IO:
 bi: Blocks sent to a block device (blocks/s)
 bo: Blocks received from a block device (blocks/s)

system:
 in: The number of interrupts per seconds, including the clock.
 cs: The number of context switches per seconds.

cpu (% of total cpu time)
 us: Time spent running non-kernal code (user time, including nice time)
 sy: Time spent running kernel code  (system time)
 id: Time spent idle. (prior to linux 2.5.41, this included I/O-wait time. )
 wa: Time spent waiting for I/O. (prior to Linux 2.5.41, this appeared as zero)

some useful flags:

 -m: display the memory utilization of the kernal (slabs)
 -a: provides information about about active and inactive memory pages.
 -n: displays only one header line.
 -p: {partition} flag, vmstat also provides I/O statistics.

Understanding top

Understanding top

By default "top" displays the most cpu intensive task.
Update list in every five seconds

PID: process identification
USER: Name of the user who owns (and perhaps started) the process.
PRI: priority of the process.
NI: niceness level
SIZE: Amount of memory (code+data+stack)
RSS: Amount of physical RAM used, in kilobytes.
SHARE: Amount of memory shared with other processes, in kilobytes
STAT: state of process.
[S: sleeping, R: running, D: waiting (Mostly for IO), T: stopped(suspended), Z: Zombi(defunt)

%CPU: share of the cpu usage.
%MEM: share of physical memory.
Time: total cpu time used by the process.
command: command line used to start the task (including parameters)

Keys:

sort by: [A:age, N:pid, P:cpu usage, M:resident memory usage, T: time]
u: show only a specific user,
k: kill a task (with any signal)
r: renice a task
s: set the delay in seconds between update

s3cmd : A command line tools to work with the amazon s3

Some time we need to work with our "aws s3" bucket and need a command line tool to access amazon's s3 restful api. " s3cmd " one of the command that will help you working on this.

-> To install s3cmd at your ubuntu system:
sudo apt-get install s3cmd

Once you install your s3cmd you need to configure your system to have your access key and secrete key to work with your s3 bucket.

To configure your s3cmd having the config, run the following command:

s3cmd --configure

 You will be asked for the two keys, access key and secrete key
 
 NOTE: You use this key, whey you use the ec2-upload-bundle command.

sample_screenrc

startup_message off    # don't show start up message
autodetach on     # auto detach on exit
escape ^Xx     # change ctrl-a to ctrl-x


termcapinfo xterm* ti@:te@     # allow scrolling   [ Note: use * after xterm so that you can get scrolling for all the different screen ]
defscrollback        9000            # default: 100 # Set default lines of scrollback.
ignorecase on           # case insensitive search in scroll-back mode
bufferfile $HOME/.screen-exchange       # keep the buffer exchange file out of /tmp

# My status line
caption always "Screens: %{.bW}%-w%{.rW}%f%n %t%{-}%+w %=%{..G}  %{W}%Y-%m-%d %{W}%c %{g} %H"

sorendition "kg"        # makes screen messages stand out, black on green
altscreen on            # full-screen programs (less, Vim) should be cleared once quit
vbell off               # visual bells are hard to do right. screen's isn't good
defutf8 on              # allow utf characters

 hardstatus alwayslastline
 hardstatus string '%{= kw}[ %{= kb}%H%{= kw} ][%= %{= kw}%?%-Lw%?%{= kW}%n*%f %t%?%?%{= kw}%?%+Lw%?%?%= ][ %{r}%l%{w} ]%{w}[%{r} %d/%m/%y %C %A %{w}]%{w}'

Regular expressions matching - regex or regexp

Regular expressions matching:

/hello/ -> work hello should be there in the string

/^hello/ -> ^ mean, the match string should be in start
/hello$/ -> $ mean, the match string should be at the end

/hel+o/ -> + mean, once or more [ "helo", "hello", "helllllllo"] -> with one "l", or 2 "l" or multple "l"
/hel*o/ -> * mean, zero or more time. Almost same as above, with true for "heo" too. ]  -> [ "heo", "helo", "hello", "hellllllo" ]
/hel?o/ -> ? mean, zero or with only one. [ "heo" or "helo" only. ]. This will not match [ hello, hellllllo ]

/hello|goodbye/ -> either|or
/he..o/ -> . mean any character . [ hello, heazo ]
/\wello/ -> \w mean alphanmeric or _ (underscore)
/\bhello/ -> \b mean word boundary => space tab.


example of matching one of a email address:
/^[a-zA-Z0-9._-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}$/

Few more example of regex:

1. regex to grep the mac address:

egrep -io "([0-9A-F][0-9A-F]:)"{5}"[0-9A-F][0-9A-F]"

ifconfig | grep -i -P '([0-9A-F]{2}:){5}([0-9A-Fa-f][0-9A-F])'

-P for support of perl regex.

2. regex for ipaddress
ifconfig | egrep -o "(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])"

3.
 hostname  | egrep "(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])"

Notes on Regex:

*) for i in {1..9}; do touch $i.html; done

*) for i in `ls |grep html| cut -d "." -f1`; do mv ${i}.html ${i}.htm; done

/[crnld]ope/ -> [...] mean range of chars => mean will match ["cope" "rope", "nope", "lope" and "dope"] but not aope.


cat log1 |egrep -o -e "(client|appserver/webapp)/.*/" | perl -pe 's|(client\|appserver/webapp)|mydir|'

cat log1 |egrep -o -e "(client|appserver/webapp)/.*/" | perl -pe 's/(client|appserver\/webapp)/patch12/'

*** The above one liner will grep the above regex from the log1 file and do a search and replace, where ever "client" is there or "(appserver/webapp)" is there to "mydir" . ***


External links:
http://www.cheatography.com/davechild/cheat-sheets/regular-expressions/
http://www.cs.tut.fi/~jkorpela/perl/regexp.html
http://ult-tex.net/info/perl/
http://cloud.github.com/downloads/tartley/python-regex-cheatsheet/cheatsheet.pdf
http://www.regular-expressions.info/
http://overapi.com/regex/