Friday 30 May 2014

haproxy configuration

Haproxy my configuration few notes:


## Following is the ab [ apache benchmark output ] ##

ab -n 900 -c 30 http://localhost:60000/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 900 requests


Server Software:        Apache/2.2.22
Server Hostname:        localhost
Server Port:            60000

Document Path:          /
Document Length:        177 bytes

Concurrency Level:      30
Time taken for tests:   1.952 seconds
Complete requests:      900
Failed requests:        300
   (Connect: 0, Receive: 0, Length: 300, Exceptions: 0)
Write errors:           0
Total transferred:      521100 bytes
HTML transferred:       289800 bytes
Requests per second:    460.98 [#/sec] (mean)
Time per request:       65.079 [ms] (mean)
Time per request:       2.169 [ms] (mean, across all concurrent requests)
Transfer rate:          260.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       3
Processing:     1   63  91.8      1     375
Waiting:        0   63  91.9      1     375
Total:          1   64  91.8      1     375

Percentage of the requests served within a certain time (ms)
  50%      1
  66%     38
  75%    153
  80%    174
  90%    205
  95%    230
  98%    294
  99%    310
 100%    375 (longest request)


#####################################
#####################################
## Haproxy output status as cvs output ##
http://localhost:60000/haproxy?stats;csv

 pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,
MyWebServers,FRONTEND,,,1,47,2000,1590,190943,974595,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,1,0,492,,,,0,1589,0,0,0,0,,1,492,1590,,,
MyWebServers,lh1,0,0,0,12,,528,62976,239184,,0,,0,0,0,0,UP,1,1,0,0,0,335,0,,1,1,1,,528,,2,0,,164,L4OK,,0,0,528,0,0,0,0,0,,,,0,0,
MyWebServers,lh2,0,0,0,11,,527,62809,238731,,0,,0,0,0,0,UP,1,1,0,0,0,335,0,,1,1,2,,527,,2,0,,164,L4OK,,0,0,527,0,0,0,0,0,,,,0,0,
MyWebServers,ub1,0,0,0,30,,527,62809,437937,,0,,0,0,0,0,UP,1,1,0,0,0,335,0,,1,1,3,,527,,2,0,,164,L4OK,,1,0,527,0,0,0,0,0,,,,0,0,
MyWebServers,BACKEND,0,0,0,36,2000,1582,190943,974595,0,0,,0,0,0,0,UP,3,3,0,,0,335,0,,1,1,0,,1582,,1,0,,492,,,,0,1582,0,0,0,0,,,,,0,0,


user@Desktop:/etc/haproxy$ cat haproxy.cfg
# this config needs haproxy-1.1.28 or haproxy-1.2.1

global
    log 127.0.0.1    local0
    log 127.0.0.1    local1 notice
    #log loghost    local0 info
    maxconn 4096
    #chroot /usr/share/haproxy
#    user haproxy
#    group haproxy
    daemon
    #debug
    #quiet

defaults
    log    global
    mode    http
    option    httplog
    option    dontlognull
    retries    3
    option redispatch
    maxconn    2000
    contimeout    5000
    clitimeout    50000
    srvtimeout    50000


listen MyWebServers 0.0.0.0:60000
    mode http
    stats enable
    stats uri /haproxy?stats
     <--- using this we get the web output

    balance roundrobin
    server lh1 localhost:80 check
    server lh2 localhost:81 check
    server ub1 192.168.0.103:80 check


    option    httpclose        # disable keep-alive
    option  checkcache        # block response if set-cookie & cacheable

    rspidel ^Set-cookie:\ IP=    # do not let this cookie tell our internal IP address
  
    #errorloc    502    http://192.168.114.58/error502.html
    #errorfile    503    /etc/haproxy/errors/503.http
    errorfile    400    /etc/haproxy/errors/400.http
    errorfile    403    /etc/haproxy/errors/403.http
    errorfile    408    /etc/haproxy/errors/408.http
    errorfile    500    /etc/haproxy/errors/500.http
    errorfile    502    /etc/haproxy/errors/502.http
    errorfile    503    /etc/haproxy/errors/503.http
    errorfile    504    /etc/haproxy/errors/504.http





######################################
####################################
cat /etc/default/haproxy
# Set ENABLED to 1 if you want the init script to start haproxy.
ENABLED=1
# Add extra flags here.
#EXTRAOPTS="-de -m 16"



#################################
#################################
In browser:
http://localhost:60000/haproxy?stats

NOTE:
This value that we got is from the above haproxy config:

listen MyWebServers 0.0.0.0:60000
    mode http
    stats enable
    stats uri /haproxy?stats








##################################
##################################
# Another configuration note:

# Install the package:

sudo apt-get install haproxy


cat /etc/haproxy/haproxy.cfg    <-- Following is the default content.
# this config needs haproxy-1.1.28 or haproxy-1.2.1

global
    log 127.0.0.1    local0
    log 127.0.0.1    local1 notice
    #log loghost    local0 info
    maxconn 4096
    #chroot /usr/share/haproxy
    user haproxy
    group haproxy
    daemon
    #debug
    #quiet

defaults
    log    global
    mode    http
    option    httplog
    option    dontlognull
    retries    3
    option redispatch
    maxconn    2000
    contimeout    5000
    clitimeout    50000
    srvtimeout    50000

listen    appli1-rewrite 0.0.0.0:10001
    cookie    SERVERID rewrite
    balance    roundrobin
    server    app1_1 192.168.34.23:8080 cookie app1inst1 check inter 2000 rise 2 fall 5
    server    app1_2 192.168.34.32:8080 cookie app1inst2 check inter 2000 rise 2 fall 5
    server    app1_3 192.168.34.27:8080 cookie app1inst3 check inter 2000 rise 2 fall 5
    server    app1_4 192.168.34.42:8080 cookie app1inst4 check inter 2000 rise 2 fall 5

listen    appli2-insert 0.0.0.0:10002
    option    httpchk
    balance    roundrobin
    cookie    SERVERID insert indirect nocache
    server    inst1 192.168.114.56:80 cookie server01 check inter 2000 fall 3
    server    inst2 192.168.114.56:81 cookie server02 check inter 2000 fall 3
    capture cookie vgnvisitor= len 32

    option    httpclose        # disable keep-alive
    rspidel ^Set-cookie:\ IP=    # do not let this cookie tell our internal IP address
   
listen    appli3-relais 0.0.0.0:10003
    dispatch 192.168.135.17:80

listen    appli4-backup 0.0.0.0:10004
    option    httpchk /index.html
    option    persist
    balance    roundrobin
    server    inst1 192.168.114.56:80 check inter 2000 fall 3
    server    inst2 192.168.114.56:81 check inter 2000 fall 3 backup

listen    ssl-relay 0.0.0.0:8443
    option    ssl-hello-chk
    balance    source
    server    inst1 192.168.110.56:443 check inter 2000 fall 3
    server    inst2 192.168.110.57:443 check inter 2000 fall 3
    server    back1 192.168.120.58:443 backup

listen    appli5-backup 0.0.0.0:10005
    option    httpchk *
    balance    roundrobin
    cookie    SERVERID insert indirect nocache
    server    inst1 192.168.114.56:80 cookie server01 check inter 2000 fall 3
    server    inst2 192.168.114.56:81 cookie server02 check inter 2000 fall 3
    server    inst3 192.168.114.57:80 backup check inter 2000 fall 3
    capture cookie ASPSESSION len 32
    srvtimeout    20000

    option    httpclose        # disable keep-alive
    option  checkcache        # block response if set-cookie & cacheable

    rspidel ^Set-cookie:\ IP=    # do not let this cookie tell our internal IP address
   
    #errorloc    502    http://192.168.114.58/error502.html
    #errorfile    503    /etc/haproxy/errors/503.http
    errorfile    400    /etc/haproxy/errors/400.http
    errorfile    403    /etc/haproxy/errors/403.http
    errorfile    408    /etc/haproxy/errors/408.http
    errorfile    500    /etc/haproxy/errors/500.http
    errorfile    502    /etc/haproxy/errors/502.http
    errorfile    503    /etc/haproxy/errors/503.http
    errorfile    504    /etc/haproxy/errors/504.http



# You can update the following file [ /etc/haproxy/haproxy.cfg ] bashed as per your need.

02. Edit configurations file

    sudo vi /etc/haproxy/haproxy.cfg


Add following text to file and save it


global
maxconn 4096
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout
5000
clitimeout
50000
srvtimeout
50000
listen webcluster *:80
mode http
stats enable
stats auth us3r:passw0rd
balance roundrobin
option httpchk HEAD / HTTP/1.0
option forwardfor
cookie LSW_WEB insert
option httpclose
server web01 10.100.2.85:8280 cookie LSW_WEB01 check
server web02 10.100.2.85:8281 cookie LSW_WEB02 check


03. Setting startup parameter for HAProxy set enabled as 1 to start HAproxy
sudo vi /etc/default/haproxy

    # Set ENABLED to 1 if you want the init script to start haproxy.
    ENABLED=1


04. Start HAProxy form command line

    sudo /etc/init.d/haproxy start



05. Restart HAProxy after any configuration change form command line

sudo /etc/init.d/haproxy restart


06. Start 2 ESB servers (Set HTTP port as 8280 and 8281) and with default configurations

07. Access Web user interface from browser
Type following url in browser and provide stat credentials you provided in configurations file(haproxy.cfg)

http://10.100.2.85:80/haproxy?stats

You will see following User interface. 








External links:


http://haproxy.1wt.eu/download/1.3/doc/configuration.txt

https://www.digitalocean.com/community/articles/how-to-use-haproxy-to-set-up-http-load-balancing-on-an-ubuntu-vps

http://sanjeewamalalgoda.blogspot.in/2012/06/setup-haproxy-load-balancer-on-ubuntu.html

Wednesday 28 May 2014

tcp buffer tunning

A quick note on tcp buffer tunning:



/proc/sys/net/core/rmem_max – Maximum TCP Receive Window.
/proc/sys/net/core/wmem_max – Maximum TCP Send Window.
/proc/sys/net/ipv4/tcp_timestamps – Timestamp ( RFC 1323) add 12 bytes to the TCP headers.
/proc/sys/net/ipv4/tcp_sack – TCP selective acknowledgments.
/proc/sys/net/ipv4/tcp_window_scaling – support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535.

mem_default = Default Receive Window.
rmem_max = Maximum Receive Window.
wmem_default = Default Send Window.
wmem_max = Maximum Send Window.

Everything under /proc is volatile, so any changes made are lost after a reboot.
Applying TCP/IP Parameters at System Boot:

Place the following code in /etc/rc.local so that they get applied on the system while
booting:

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

python for mysql

# You might need the following module to be installed in the system:
MySQL-python

So you can try using " sudo pip install MySQL-python " or " sudo easy-install MySQL-python"

if you are getting error while installing MySQL-python [ apt-get install python-mysqldb ]

#!/usr/bin/env python
import MySQLdb

db = MySQLdb.connect(host="hostname", # your host, usually localhost
                     user="username", # your username
                      passwd="password", # your password
                      db="database") # name of the data base

# you must create a Cursor object. It will let
#  you execute all the queries you need
cur = db.cursor()

# Use all the SQL you like
cur.execute("select * from Country where Code2 = 'IN';")

# print all the first cell of all the rows
for row in cur.fetchall() :
#    print row[0]
    rows = row   
    print(type(rows))  
    MyList=list(rows)
    print(type(MyList))
    print('Following is in tuple format')
    print(rows)
    print('Following is the list format')
    print(MyList)
#    print row


############ There are other module for connecting the mysql, with different classes." ####

mysql-connector-python


you can try, in python prompt:

>>> dir(MySQLdb) ## to see the modules classes.

## to see installed python module: ## pip list


external link:
http://mysql-python.sourceforge.net/MySQLdb.html

Tuesday 27 May 2014

python string replace

Lets say we have a file, where we have hosts name and there true and false status. We checked the health status of the health and found we need to make it false, if its true, because of the health check is failing: how to do this:


bash: sed -i 's/true \(host1.*\)/false \1/g' 

bash: to print file from line 5 to 10:
 sed -n '5,10p' filename

file example:
true host1.example.com
true host2.example.com



cat true-false.py
#! /usr/bin/env python

myfile = open('haproxy-file.txt')
for line in myfile:
  if 'host1' in line:
    print (line.replace('true','false'))



# How I will do inline of a file and update the file?, with the new value.

#! /usr/bin/env python

myfile = open('haproxy-file.txt')
for line in myfile:
  if 'host1' in line:
    UpdateLine = line.replace('true','false')
    MyWriteFile = open('haproxy-file.txt','r+')
    MyWriteFile.write(UpdateLine)
    MyWriteFile.close()



In the above script, still some error are there: [will fix and update. ]

what is the error:
# Output:
false host1.example.com
rue host2.example.com #-> NOTE: 't' is missing over here

# Expecting Output:
false host1.example.com
true host2.example.com