Sunday 17 February 2013

Bash Notes

Bash Notes:


bash  [options] [arguments]
man bash [ for more informations ]
positional parameters $1, $2, etc. But its good to use the positional parameters as ${1}, ${2}
The name of the script is available as ${0}

Filename Metacharacters:

* -> Match any string of zero or more characters.
? -> Match any single character.
[abc...] -> Match any one of the enclosed characters; a hyphen can specify a range (e.g., a-z, A-Z, 0–9).
[!abc...] -> Match any character not enclosed as above.
~ -> Home directory of the current user.
~name -> Home directory of user name.
~+ -> Current working directory ($PWD).
~- -> Previous working directory ($OLDPWD).







Adding the given number:


i=0; for x in {1..10}; do i=$(( $i + $x)); echo $i; done
 


With the extglob option on:

?(pattern) Match zero or one instance of pattern.
*(pattern) Match zero or more instances of pattern.
+(pattern) Match one or more instances of pattern.
@(pattern) Match exactly one instance of pattern.
!(pattern) Match any strings that don’t match pattern.

This pattern can be a sequence of patterns separated by "|" meaning that the match applies to any of the patterns.
The extglob also can be used at egrep and awk.

Examples:

ls new* : List new and new.1
cat ch? : Match ch9 but not ch10
vi [D-R]* : Match files beginning with D through R
pr !(*.o|core) | lp : Print files non-object and non-core files

pre{X,Y[,Z...]}post : Expands to preXpost, preYpost, and so on.

# Expand textually; no sorting
$ echo hi{DDD,BBB,CCC,AAA}there
hiDDDthere hiBBBthere hiCCCthere hiAAAthere

# Expand, then match ch1, ch2, app1, app2
$ ls {ch,app}?

# Expands to mv info info.old
$ mv info{,.old}

# Simple numeric expansion
$ echo 1 to 10 is {1..10}
1 to 10 is 1 2 3 4 5 6 7 8 9 10

*) Renaming all the .html files to .htm file in a single directory:
rename 's/\.html/\.htm/' *.html
[ how about renameing .htm file to .php  -> rename 's/\.htm/\.php/' *.htm ]
[ rename is using sed type replacement, "\" is used for the escape of . ]

for i in *.html; do mv "${i}" "${i/.html}".htm; done   [ There should not be a space at "${i/.html}".htm ]

find /home/amitmund/test/rename_Test/ -type f -iname "*.html" -exec rename 's/\.html/\.htm/' *.htm {} \;

Quoting:

; Command separator.
& Background execution.
() Command grouping.
| Pipe.
< > & Redirection symbols.
* ? [ ] ~ + - @ ! Filename metacharacters.
" ' \ Used in quoting other characters.
` Command substitution.
$ Variable substitution (or command or
 arithmetic substitution).
# Start a comment that continues to the end of the line.
space tab newline Word separators.

" "  Everything between " and " is taken literally. [ Except few ]
$ Variable (or command and arithmetic) substitution will occur.
` Command substitution will occur.
" This marks the end of the double quoted string.
\ The character following a \ is taken literally. Also known as escape character.

Examples
$ echo 'Single quotes "protect" double quotes'
Single quotes "protect" double quotes
$ echo "Well, isn’t that \"special\"?"
Well, isn’t that "special"?
$ echo "You have `ls | wc -l` files in `pwd`"
You have
43 files in /home/bob
$ echo "The value of \$x is $x"
The value of $x is 100


cmd & -> Execute cmd in background.
cmd1 ; cmd2 -> Command sequence; execute multiple cmds on the same line.
{ cmd1 ; cmd2 ; } -> Execute commands as a group in the current shell.
(cmd1 ; cmd2) -> Execute commands as a group in a subshell.
cmd1 | cmd2 -> Pipe; use output from cmd1 as input to cmd2.
cmd1 `cmd2` -> Command substitution; use cmd2 output as arguments to cmd1.
cmd1 $(cmd2) -> POSIX shell command substitution; nesting is allowed.
cmd $((expression)) -> POSIX shell arithmetic substitution. Use the result of expression as argument to cmd.
cmd1 && cmd2 -> AND; execute cmd1 and then (if cmd1 succeeds) cmd2.
cmd1 || cmd2 -> OR; execute either cmd1 or (if cmd1 fails) cmd2.
! cmd -> NOT; execute cmd, and produce a zero exit status if cmd exits with a nonzero status.

Redirection Forms
File descriptor Name Common abbreviation Typical default
0 Standard input stdin Keyboard
1 Standard output stdout Screen
2 Standard error stderr Screen

Simple redirection
cmd > file -> Send output of cmd to file (overwrite).
cmd >> file -> Send output of cmd to file (append).
cmd < file -> Take input for cmd from file.
cmd <> file -> Open file for reading and writing on the standard input. The contents are not destroyed.*
cmd >| file -> Send output of cmd to file (overwrite), even if the shell’s noclobber option is set.


Redirection using file descriptors:
cmd >&n -> Send cmd output to file descriptor n.
cmd m>&n -> Same as previous, except that output that would normally go to file descriptor m is sent to file descriptor n instead.
cmd >&- -> Close standard output.
cmd <&n -> Take input for cmd from file descriptor n.

Multiple redirection
cmd 2>file -> Send standard error to file; standard output remains the same (e.g., the screen).
cmd > file 2>&1 -> Send both standard output and standard error to file.
cmd >& file -> Same as previous.
cmd &> file -> Same as previous. Preferred form.

cmd &>> file -> Append both standard output and standard error to file.
cmd > f1 2> f2 -> Send standard output to file f1 and standard error to file f2.
cmd | tee files -> Send output of cmd to standard output (usually the terminal) and to files. See tee(1).
cmd 2>&1 | tee files -> Send standard output and error output to screen and files.
cmd |& tee files ->Same as previous.

/dev/stdin -> A duplicate of file descriptor zero.
/dev/stdout -> A duplicate of file descriptor one.
/dev/stderr -> A duplicate of file descriptor two.

$ sed 's/^/XX /g' << EOF #Here document is sed's input
This is often how a shell archive is "wrapped",
bundling text for distribution. You would normally
run sed from a shell program, not from the command line.
EOF


${#var} Use the length of var.
${#*} Use the number of positional parameters.
${#@} Same as previous.

Built-in Shell Variables:

$# -> Number of command-line arguments.
$- -> Options currently in effect (supplied on command line or to set). The shell sets some options automatically.
$? -> Exit value of last executed command.
$$ -> Process number of the shell.
$! -> Process number of last background command.
$0 -> First word; that is, the command name.
$n -> Individual arguments on command line. If its more then 9 then we can use ${n}.
$*, $@ -> All arguments on command line ($1 $2 ...).
"$*" -> All arguments on command line as one string ("$1 $2..."). The values are separated by the first character in $IFS.
"$@" -> All arguments on command line, individually quoted ("$1" "$2" ...).
$_ -> Temporary variable;

BASHPID -> The process ID of the current Bash process. In some cases, this can differ from $$.
BASH -> The full pathname used to invoke this instance of Bash. [ e.g. echo $BASH ]
EUID -> Read-only variable with the numeric effective UID of the current user.
OLDPWD -> Previous working directory (set by cd).
PPID -> Process number of this shell’s parent.
PWD -> Current working directory (set by cd).
SECONDS[=n] -> Number of seconds since the shell was started.
BASH_ENV -> If set at startup, names a file to be processed for initialization commands.
IFS='chars' -> Input field separators; default is space, Tab, and newline.

PS1=string Primary prompt string; default is $.
PS2=string Secondary prompt (used in multiline commands); default is >.
PS3=string Prompt string in select loops; default is #?.

Array:

message=(how are you message?)
echo ${message[0]}
echo ${message[1]}
echo ${message[2]}
echo ${message[3]}
echo ${message[*]}
echo ${#message[*]}  -> 4  [ display total number of elements in the array message ]

${name[i]} Use element i of array name. i can be any arithmetic expression as described under let.
${name} Use element 0 of array name.
${name[*]} Use all elements of array name.
${name[@]} Same as previous.
${#name[*]} Use the number of elements in array name.
${#name[@]} Same as previous.


Special Prompt Strings:
Bash processes the values of PS1, PS2, and PS4 for the following
special escape sequences:
\a An ASCII BEL character (octal 07).
\A The current time in 24-hour HH:MM format.
\d The date in “weekday month day” format.
\D{format} The date as specified by the strftime(3) format format. The braces are required.
\e An ASCII Escape character (octal 033).
\h The hostname, up to the first period.
\H The full hostname.
\j The current number of jobs.
\l The basename of the shell’s terminal device.
\n A newline character.
\r A carriage return character.
\s The name of the shell (basename of $0).
\t The current time in 24-hour HH:MM:SS format.
\T The current time in 12-hour HH:MM:SS format.
\u The current user’s username.
\v The version of Bash.
\V The release (version plus patchlevel) of Bash.
\w The current directory, with $HOME abbreviated as ~.
\W The basename of the current directory, with $HOME abbreviated as ~.
\! The history number of this command (stored in the history).
\# The command number of this command (count of commands executed by the current shell).
\$ If the effective UID is 0, a #; otherwise, a $.
\@ The current time in 12-hour a.m./p.m. format. Variables | 35
\nnn The character represented by octal value nnn.
\\ A literal backslash.
\[ Start a sequence of nonprinting characters, such as for highlighting or changing colors on a terminal.
\] End a sequence of nonprinting characters.


Arithmetic Expressions:

$(( expr )) Use the value of the enclosed arithmetic expression.
++ -- Auto-increment and auto-decrement, both prefix and postfix
+ - Unary plus and minus
! ~ Logical negation and binary inversion (one’s complement)
** Exponentiationa
* / % Multiplication, division, modulus (remainder)
+ - Addition, subtraction
<< >> Bitwise left shift, bitwise right shift
< <= > >= Less than, less than or equal to, greater than, greater than or equal to
== != Equality, inequality (both evaluated left to right)
& Bitwise AND
^ Bitwise exclusive OR
| Bitwise OR
&& Logical AND (short circuit)
|| Logical OR (short circuit)
?: Inline conditional evaluation

Command editing keystrokes: [ Line edit mode ]

set -o vi or set -o emacs [ NOTE: First words as if its vi and second one if it set to emacs ]

h CTRL-b Move back one character.
l CTRL-f Move forward one character.
b ESC-b Move back one word.
w ESC-f Move forward one word.
X DEL Delete previous character.
x CTRL-d Delete character under cursor.
dw ESC-d Delete word forward.
db ESC-h Delete word backward.
xp CTRL-t Transpose two characters.

function name () { commands; }

amitmund@amitmundlaptop:~$ kill -l
 1) SIGHUP     2) SIGINT     3) SIGQUIT     4) SIGILL     5) SIGTRAP
 6) SIGABRT     7) SIGBUS     8) SIGFPE     9) SIGKILL    10) SIGUSR1
11) SIGSEGV    12) SIGUSR2    13) SIGPIPE    14) SIGALRM    15) SIGTERM
16) SIGSTKFLT    17) SIGCHLD    18) SIGCONT    19) SIGSTOP    20) SIGTSTP
21) SIGTTIN    22) SIGTTOU    23) SIGURG    24) SIGXCPU    25) SIGXFSZ
26) SIGVTALRM    27) SIGPROF    28) SIGWINCH    29) SIGIO    30) SIGPWR
31) SIGSYS    34) SIGRTMIN    35) SIGRTMIN+1    36) SIGRTMIN+2    37) SIGRTMIN+3
38) SIGRTMIN+4    39) SIGRTMIN+5    40) SIGRTMIN+6    41) SIGRTMIN+7    42) SIGRTMIN+8
43) SIGRTMIN+9    44) SIGRTMIN+10    45) SIGRTMIN+11    46) SIGRTMIN+12    47) SIGRTMIN+13
48) SIGRTMIN+14    49) SIGRTMIN+15    50) SIGRTMAX-14    51) SIGRTMAX-13    52) SIGRTMAX-12
53) SIGRTMAX-11    54) SIGRTMAX-10    55) SIGRTMAX-9    56) SIGRTMAX-8    57) SIGRTMAX-7
58) SIGRTMAX-6    59) SIGRTMAX-5    60) SIGRTMAX-4    61) SIGRTMAX-3    62) SIGRTMAX-2
63) SIGRTMAX-1    64) SIGRTMAX   


Some Commands:

times : Print accumulated CPU times.
ulimit : Manage various process limits.
unalias : Remove previously defined aliases.
until : Syntax for a loop that runs until a condition becomes true.
wait : Wait for a process or job to complete. ( e.g.: wait [ID] )

Tuesday 12 February 2013

Networking Fundamentals

Network Fundamentals:

The Seven OSI Layers:

7. Application
6. Presentation
5. Session
4. Transport
3. Network
2. Link
1. Physical

http://en.wikipedia.org/wiki/OSI_model


Private IP Address: -

10.0.0.0  -  10.255.255.255
172.16.0.0  -  172.31.255.255
192.168.0.0  - 192.168.255.255

Loopback address: 127.0.0.1

Network Class:

Class A: 00000000 - 01111111 -> ( 0 - 127 )
Class B: 10000000 - 10111111 -> ( 128 - 191 )
Class C: 11000000 - 11011111 ->  ( 192 - 223 )
Class D: 11100000 - 11101111 -> ( 224 - 239 )
Class E: 11110000 - 11110111 -> (240 - 247 )

0 -> 00000000
1 -> 00000001
2 -> 00000010
3 -> 00000011
4 -> 00000100
5 -> 00000101
6 -> 00000110
7 -> 00000111
8 -> 00001000













Saturday 9 February 2013

SettingUpAWSapi

Setting Up AWS api:  [ Amazon command line tools ] :

I am using ubuntu 12.10 and for the long time I was getting error while setting up AWS api in my system:

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
export EC2_HOME=~/ec2
export EC2_PRIVATE_KEY=<Private_key_path>
export EC2_CERT=<Cert_path>

For other aws api:
export AWS_ELB_HOME=~/elb

export AWS_CLOUDWATCH_HOME=~/cw

PATH=$PATH:$JAVA_HOME/bin:$EC2_HOME/bin

Note:

You can create different scripts, where you specify different EC2_CERT and EC2_PRIVATE_KEY to access your different aws account, example [dev, qa or prod]

For other api call: 
http://aws.amazon.com/developertools/

For AMI CLI: 
http://awsiammedia.s3.amazonaws.com/public/tools/cli/latest/IAMCli.zip

For ElasticBeanstalk CLI:
https://s3.amazonaws.com/elasticbeanstalk/cli/AWS-ElasticBeanstalk-CLI-2.6.0.zip

For ELB CLI:
http://ec2-downloads.s3.amazonaws.com/ElasticLoadBalancing.zip

For AutoScalling CLI:
http://ec2-downloads.s3.amazonaws.com/AutoScaling-2011-01-01.zip


For S3 CLI:
http://code.google.com/p/lits3/downloads/detail?name=LitS3-Commander-0.8.4.zip


Even after above setting I was getting java related error... and I stared trying solving the same issue, checked some on-line links and found lots of people have similar issue. [ I was my mistake that I was keeping all the aws command in a same directory. ]

I removed all the jre and java related package from my ubuntu system and reinstall those pkg again but the similar issue.

The error was something like:
ec2-describe-regions
Error: Could not find or load main class com.amazon.aes.webservices.client.cmd.DescribeRegions

Then I started looking around the JAVA CLASSPATH... again spent some more time. But the good part that I understand that in hard way... after working for long hours. As a SRE/DEVOPS person, when started from Linux System Administrator, understanding java error took some time :)

Started looking the aws command code:

amitmund@amitmundlaptop:~/ec2/bin$ cat ../ec2-api-tools-1.6.0.0/bin/ec2-describe-regions
#!/usr/bin/env bash

# Copyright 2006-2010 Amazon.com, Inc. or its affiliates.  All Rights Reserved.  Licensed under the
# Amazon Software License (the "License").  You may not use this file except in compliance with the License. A copy of the
# License is located at http://aws.amazon.com/asl or in the "license" file accompanying this file.  This file is distributed on an "AS
# IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

__ZIP_PREFIX__EC2_HOME="${EC2_HOME:?EC2_HOME is not set}"
__RPM_PREFIX__EC2_HOME=/usr/local/aes/cmdline
"${EC2_HOME}"/bin/ec2-cmd DescribeRegions "$@"

 And then followed the ec2-cmd:

amitmund@amitmundlaptop:~/ec2/bin$ cat ec2-cmd
#!/usr/bin/env bash

# Copyright 2006-2009 Amazon.com, Inc. or its affiliates.  All Rights Reserved.  Licensed under the
# Amazon Software License (the "License").  You may not use this file except in compliance with the License. A copy of the
# License is located at http://aws.amazon.com/asl or in the "license" file accompanying this file.  This file is distributed on an "AS
# IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

# This script "concentrates" all of our Java invocations into a single location
# for maintainability.

# 'Globals'
__ZIP_PREFIX__EC2_HOME="${EC2_HOME:-EC2_HOME is not set}"
__RPM_PREFIX__EC2_HOME=/usr/local/aes/cmdline
LIBDIR="${EC2_HOME}/lib"

# Check our Java env
JAVA_HOME=${JAVA_HOME:?JAVA_HOME is not set}

# If a classpath exists preserve it
CP="${CLASSPATH}"

# Check for cygwin bash so we use the correct path separator
case "`uname`" in
    CYGWIN*) cygwin=true;;
esac

# ---- Start of Cygwin test ----

cygprop=""

# And add our own libraries too
if [ "${cygwin}" == "true" ] ; then
    cygprop="-Dec2.cygwin=true"

    # Make sure that when using Cygwin we use Unix
    # Semantics for EC2_HOME
    if [ -n "${EC2_HOME}" ]
    then
        if echo "${EC2_HOME}"|egrep -q '[[:alpha:]]:\\'
        then
            echo
            echo " *INFO* Your EC2_HOME variable needs to specified as a Unix path under Cygwin"
            echo
        fi
    fi

# ---- End of Cygwin Tests ----

    for jar in "${LIBDIR}"/*.jar ; do
        cygjar=$(cygpath -w -a "${jar}")
        CP="${CP};${cygjar}"
    done
else
    for jar in "${LIBDIR}"/*.jar ; do
        CP="${CP}:${jar}"
    done
fi

CMD=$1
shift
"${JAVA_HOME}/bin/java" ${EC2_JVM_ARGS} ${cygprop} -classpath "${CP}" "com.amazon.aes.webservices.client.cmd.${CMD}" $EC2_DEFAULT_ARGS "$@"

 then I found it is looking for classpath and the jar files from the same directory's lib folder from were I have copied the aws commands.

 Later I created a "lib" directory in the same location where I created the "bin" directory for all the aws command line tools and started coping the "jar" files from all the original directory to this "lib" directory and its started working. :)





so.... where I was doing mistake?

I have downloaded all the aip from Amazon site... and it around few different zip files from windows and linux environment...

e.g:

AutoScaling-2010-08-01.zip
CloudWatch-2010-08-01.zip
ec2-api-tools.zip  RDSCli.zip
AWSCloudFormation-cli.zip 
ec2-ami-tools.zip        
IAMCli.zip

But what I did, to set the a single EC2_HOME directory, I extracted all these directory and copied all the aws command from those directories "bin" directory and copied them to ~/ec2/bin and not the related "lib" directory where all the related jar files were there.  Because of that I was getting the following error:

Error: Could not find or load main class com.amazon.aes.webservices.client.cmd.DescribeRegions







Thursday 7 February 2013

puppetlabsImpLinks



Important Links from Puppetlabs.com:


http://puppetlabs.com/resources/webinarshttp://puppetlabs.com/resources/webinars


http://docs.puppetlabs.com/
https://puppetlabs.com/misc/pdf-doc/
http://info.puppetlabs.com/download-learning-puppet-VM.html
http://docs.puppetlabs.com/references/latest/type.html
http://docs.puppetlabs.com/references/
https://puppetlabs.com/blog/
https://puppetlabs.com/resources/overview-2/
http://docs.puppetlabs.com/learning/ral.html
https://puppetlabs.com/blog/facter-part-1-facter-101/