avahi: for plug-n-play (such as Bonjour service). This daemon periodically send multicast message which is very annoying for whom work on network researches.
cups: printer related daemon
cron: periodic job launcher
atd: schedule a job (At-daemon)
rsync: remote file synchronization (very good for automatic backup, combine with cron)
...
Monday, October 17, 2011
[Linux Kernel] What does pdflush do?
Linux usually writes data out of the page cache using a process called pdflush.
At any moment, between 2 and 8 pdflush threads are running on the system. You can monitor how many are active by looking at /proc/sys/vm/nr_pdflush_threads.
As you write out data ultimately intended for disk, Linux caches this information in an area of memory called the page cache. You can find out basic info about the page cache using tools like free, vmstat or top
Old version of Linux (before 2.6) uses kupdated or bdflush instead.
Original Link: http://itmission.org/wiki/?n=Main.What-is-pdflush
At any moment, between 2 and 8 pdflush threads are running on the system. You can monitor how many are active by looking at /proc/sys/vm/nr_pdflush_threads.
As you write out data ultimately intended for disk, Linux caches this information in an area of memory called the page cache. You can find out basic info about the page cache using tools like free, vmstat or top
Old version of Linux (before 2.6) uses kupdated or bdflush instead.
Original Link: http://itmission.org/wiki/?n=Main.What-is-pdflush
Sunday, October 16, 2011
[PYTHON] openflow nox LAVI<->ENVI json message translate.py (allowing multiple ENVI connection)
I have edited the original translate.py to allow multiple ENVI connections by expanding the LAVI and ENVI connection sockets
#!/usr/bin/python
##
# Utility translate binary command to JSON equivalent for compatibility
# between ENVI and new LAVI in nox-destiny
#
# @author ykk
# @date June 2010
#
import socket
import select
import struct
import time
import simplejson
import getopt
import sys
def usage():
"""Display usage
"""
print "Usage "+sys.argv[0]+" [lavihost, default=localhost]"
print "\tConnect binary ENVI to JSON LAVI"
print "Options:"
print "-h/--help\n\tPrint this usage guide"
print "-e/--enviport\n\tSpecify port number for ENVI (default:2503)"
print "-l/--laviport\n\tSpecify port number for LAVI (default:2703)"
#Parse options and arguments
try:
opts, args = getopt.getopt(sys.argv[1:], "he:l:",
["help","laviport=","enviport="])
except getopt.GetoptError:
print "Option error!"
usage()
sys.exit(2)
#Parse options
noxhost = "localhost"
noxport = 2703
serveport = 2503
for opt,arg in opts:
if (opt in ("-h","--help")):
usage()
sys.exit(0)
elif (opt in ("-e","--enviport")):
serveport = int(arg)
elif (opt in ("-l","--laviport")):
noxport = int(arg)
#Parse host
if (len(args) >= 1):
noxhost = args[0]
def envi2lavi(binary):
jsonmsg = {}
(length, mtype, xid) = struct.unpack_from("!HBL",binary)
if (mtype == 0):
jsonmsg["type"] = "disconnect"
elif (mtype == 1):
jsonmsg["type"] = "ping"
jsonmsg["type"] = "tcp"
jsonmsg["type"] = "udp"
elif (mtype == 2):
jsonmsg["type"] = "echo"
elif (mtype == 16): #Node
(reqtype, nodetype) = struct.unpack_from("!BH",binary,7)
jsonmsg["type"] = "lavi"
if (reqtype == 1): #Req
jsonmsg["command"] = "request"
elif (reqtype == 2): #Sub
jsonmsg["command"] = "subscribe"
elif (reqtype == 3): #Unsub
jsonmsg["command"] = "unsubscribe"
else:
return {}
if (nodetype == 0): #Unknown
jsonmsg["node_type"] = "all"
elif (nodetype == 1): #OpenFlow
jsonmsg["node_type"] = "switch"
else:
return {}
elif (mtype == 19): #Link
(reqtype, linktype) = struct.unpack_from("!BH",binary,7)
jsonmsg["type"] = "lavi"
if (reqtype == 1): #Req
jsonmsg["command"] = "request"
elif (reqtype == 2): #Sub
jsonmsg["command"] = "subscribe"
elif (reqtype == 3): #Unsub
jsonmsg["command"] = "unsubscribe"
else:
return {}
if (linktype == 0): #Unknown
jsonmsg["link_type"] = "all"
elif (linktype == 1): #SW2SW
jsonmsg["link_type"] = "sw2sw"
else:
return {}
elif (mtype == 22): #Flow
(reqtype, flowtype) = struct.unpack_from("!BH",binary,7)
jsonmsg["type"] = "lavi"
if (reqtype == 1): #Req
jsonmsg["command"] = "request"
elif (reqtype == 2): #Sub
jsonmsg["command"] = "subscribe"
elif (reqtype == 3): #Unsub
jsonmsg["command"] = "unsubscribe"
else:
return {}
if (flowtype == 0): #Unknown
jsonmsg["flow_type"] = "all"
elif (flowtype == 1): #OpenFlow
jsonmsg["flow_type"] = "network"
else:
return {}
else:
return
return jsonmsg
def lavi2envi(jsonmsg):
envimsg = ""
msgtype = None
#Check for basic keys
if (not ("type" in jsonmsg and "command" in jsonmsg)):
print "Invalid message "+str(jsonmsg)
return ""
if (jsonmsg["type"] == "lavi"):
if ("node_type" in jsonmsg):
#Nodes
if (jsonmsg["node_type"] == "switch" or
jsonmsg["node_type"] == "host"):
#Switches
if (jsonmsg["command"] == "add"):
msgtype = 17
elif (jsonmsg["command"] == "delete"):
msgtype = 18
else:
return ""
for sw in jsonmsg["node_id"]:
envimsg += struct.pack("!HQ",1,int(sw,16))
else:
return ""
elif ("flow_type" in jsonmsg):
#Flows
enviaggmsg = ""
if (jsonmsg["flow_type"] == "network"):
if (jsonmsg["command"] == "add"):
msgtype = 23
for flow in jsonmsg["flows"]:
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, int(flow["src id"],16), int(flow["src port"],16)) + \
struct.pack("!HQH", 1, int(flow["dst id"],16), int(flow["dst port"],16)) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
elif (jsonmsg["command"] == "delete"):
msgtype = 24
#modified#######################################
# for flow in jsonmsg["flows"]:
################################################
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
elif (jsonmsg["flow_type"] == "host"):
if (jsonmsg["command"] == "add"):
msgtype = 23
for flow in jsonmsg["flows"]:
if ("dst port" in flow):
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, int(flow["src id"],16), 0) + \
struct.pack("!HQH", 1, int(flow["dst id"],16), int(flow["dst port"],16)) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
else:
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, int(flow["src id"],16), int(flow["src port"],16)) + \
struct.pack("!HQH", 1, int(flow["dst id"],16), 0) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
elif (jsonmsg["command"] == "delete"):
msgtype = 24
#modified####################################
# for flow in jsonmag["flows"]:
#############################################
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
return enviaggmsg
elif ("link_type" in jsonmsg):
#Links
if (jsonmsg["link_type"] == "sw2sw"):
if (jsonmsg["command"] == "add"):
msgtype = 20
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQHQ",
1,
1,int(link["src id"], 16),int(link["src port"], 16),
1,int(link["dst id"], 16),int(link["dst port"], 16),
1e7)
elif (jsonmsg["command"] == "delete"):
msgtype = 21
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQH",
1,
1,int(link["src id"], 16),int(link["src port"], 16),
1,int(link["dst id"], 16),int(link["dst port"], 16))
else:
return ""
elif (jsonmsg["link_type"] == "host2sw"):
#Host to switch
if (jsonmsg["command"] == "add"):
msgtype = 20
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQHQ",
1,
1, int(link["src id"], 16),0,
1, int(link["dst id"], 16),int(link["dst port"], 16),
1e7)
elif (jsonmsg["command"] == "delete"):
msgtype = 21
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQH",
1,
1,int(link["src id"], 16),0,
1, 0,int(link["src port"], 16),
1, int(link["dst id"], 16),int(link["dst port"], 16))
else:
return ""
else:
print "Unidentified message type"
print jsonmsg
return ""
else:
print "Not a LAVI message"
return ""
return struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
while True:
#Get ENVI connection
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(('',serveport))
server.listen(10)
print "Waiting for ENVI connection"
socks = [server]
ec_list = []
lc_list = []
#Poll and translate
lavibuffer = ""
eb={}
lb={}
lc={}
ec={}
connected_clients=0
while True:
(read, write, err) = select.select(socks,
[],
socks, 1)
for e in err:
print "Error:", e
for r in read:
if (server == r ):
(enviconn, saddr) = server.accept()
socks.append(enviconn)
ec_list.append(enviconn)
print "Got connection from "+str(saddr) + " SOCK: ", enviconn, " size: ", len(socks);
eb[str(enviconn)]=""
# create corresponding laviconn
# Get LAVI connection
laviconn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socks.append(laviconn)
lc_list.append(laviconn)
try:
laviconn.connect((noxhost, noxport))
print "Connected to NOX at "+str((noxhost,noxport)), " SOCK:", laviconn
except socket.error:
print "Cannot connect to LAVI @ "+noxhost
lb[str(laviconn)]=""
# cross reference
lc[str(enviconn)]=laviconn;
ec[str(laviconn)]=enviconn;
elif ( r in lc_list):
try:
data = r.recv(1)
if (not data):
print "NO DATA BREAK"
break
lb[str(r)] += data
try:
jsonmsg = simplejson.loads( lb[str(r)] )
binarymsg = lavi2envi(jsonmsg)
if (len(binarymsg) != 0):
# print "SEND MESSAGE TO ENVI to: ", r, " from: ", ec[str(r)]
ec[str(r)].send(binarymsg)
lb[str(r)] = ""
except ValueError:
pass
except socket.error:
print "SOCKET ERROR: BREAK"
break
elif ( r in ec_list):
try:
data = None
data = r.recv(1)
except: # this is probably socket error. delete this
r.close()
lc[str(r)].close()
socks.remove(lc[str(r)])
lc_list.remove(lc[str(r)])
socks.remove(r)
ec_list.remove(r)
print "A: delete socket: ", r, " size: ", len(socks);
continue
if (not data):
r.close()
lc[str(r)].close()
try:
socks.remove(lc[str(r)])
lc_list.remove(lc[str(r)])
socks.remove(r)
ec_list.remove(r)
except:
pass
print "B: delete socket: ", r, " size: ", len(socks);
else:
eb[str(r)] += data;
if (len(eb[str(r)]) > 2):
(length, ) = struct.unpack_from("!H",eb[str(r)])
if (length == len(eb[str(r)])):
jsonmsg = envi2lavi(eb[str(r)])
if (jsonmsg != None):
lc[str(r)].send(simplejson.dumps(jsonmsg))
eb[str(r)] = ""
else:
print "Error!: Unregistered socket found"
#Close
print "Closing connections"
if (clientok):
laviconn.shutdown(1)
laviconn.close()
time.sleep(1)
#!/usr/bin/python
##
# Utility translate binary command to JSON equivalent for compatibility
# between ENVI and new LAVI in nox-destiny
#
# @author ykk
# @date June 2010
#
import socket
import select
import struct
import time
import simplejson
import getopt
import sys
def usage():
"""Display usage
"""
print "Usage "+sys.argv[0]+"
print "\tConnect binary ENVI to JSON LAVI"
print "Options:"
print "-h/--help\n\tPrint this usage guide"
print "-e/--enviport\n\tSpecify port number for ENVI (default:2503)"
print "-l/--laviport\n\tSpecify port number for LAVI (default:2703)"
#Parse options and arguments
try:
opts, args = getopt.getopt(sys.argv[1:], "he:l:",
["help","laviport=","enviport="])
except getopt.GetoptError:
print "Option error!"
usage()
sys.exit(2)
#Parse options
noxhost = "localhost"
noxport = 2703
serveport = 2503
for opt,arg in opts:
if (opt in ("-h","--help")):
usage()
sys.exit(0)
elif (opt in ("-e","--enviport")):
serveport = int(arg)
elif (opt in ("-l","--laviport")):
noxport = int(arg)
#Parse host
if (len(args) >= 1):
noxhost = args[0]
def envi2lavi(binary):
jsonmsg = {}
(length, mtype, xid) = struct.unpack_from("!HBL",binary)
if (mtype == 0):
jsonmsg["type"] = "disconnect"
elif (mtype == 1):
jsonmsg["type"] = "ping"
jsonmsg["type"] = "tcp"
jsonmsg["type"] = "udp"
elif (mtype == 2):
jsonmsg["type"] = "echo"
elif (mtype == 16): #Node
(reqtype, nodetype) = struct.unpack_from("!BH",binary,7)
jsonmsg["type"] = "lavi"
if (reqtype == 1): #Req
jsonmsg["command"] = "request"
elif (reqtype == 2): #Sub
jsonmsg["command"] = "subscribe"
elif (reqtype == 3): #Unsub
jsonmsg["command"] = "unsubscribe"
else:
return {}
if (nodetype == 0): #Unknown
jsonmsg["node_type"] = "all"
elif (nodetype == 1): #OpenFlow
jsonmsg["node_type"] = "switch"
else:
return {}
elif (mtype == 19): #Link
(reqtype, linktype) = struct.unpack_from("!BH",binary,7)
jsonmsg["type"] = "lavi"
if (reqtype == 1): #Req
jsonmsg["command"] = "request"
elif (reqtype == 2): #Sub
jsonmsg["command"] = "subscribe"
elif (reqtype == 3): #Unsub
jsonmsg["command"] = "unsubscribe"
else:
return {}
if (linktype == 0): #Unknown
jsonmsg["link_type"] = "all"
elif (linktype == 1): #SW2SW
jsonmsg["link_type"] = "sw2sw"
else:
return {}
elif (mtype == 22): #Flow
(reqtype, flowtype) = struct.unpack_from("!BH",binary,7)
jsonmsg["type"] = "lavi"
if (reqtype == 1): #Req
jsonmsg["command"] = "request"
elif (reqtype == 2): #Sub
jsonmsg["command"] = "subscribe"
elif (reqtype == 3): #Unsub
jsonmsg["command"] = "unsubscribe"
else:
return {}
if (flowtype == 0): #Unknown
jsonmsg["flow_type"] = "all"
elif (flowtype == 1): #OpenFlow
jsonmsg["flow_type"] = "network"
else:
return {}
else:
return
return jsonmsg
def lavi2envi(jsonmsg):
envimsg = ""
msgtype = None
#Check for basic keys
if (not ("type" in jsonmsg and "command" in jsonmsg)):
print "Invalid message "+str(jsonmsg)
return ""
if (jsonmsg["type"] == "lavi"):
if ("node_type" in jsonmsg):
#Nodes
if (jsonmsg["node_type"] == "switch" or
jsonmsg["node_type"] == "host"):
#Switches
if (jsonmsg["command"] == "add"):
msgtype = 17
elif (jsonmsg["command"] == "delete"):
msgtype = 18
else:
return ""
for sw in jsonmsg["node_id"]:
envimsg += struct.pack("!HQ",1,int(sw,16))
else:
return ""
elif ("flow_type" in jsonmsg):
#Flows
enviaggmsg = ""
if (jsonmsg["flow_type"] == "network"):
if (jsonmsg["command"] == "add"):
msgtype = 23
for flow in jsonmsg["flows"]:
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, int(flow["src id"],16), int(flow["src port"],16)) + \
struct.pack("!HQH", 1, int(flow["dst id"],16), int(flow["dst port"],16)) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
elif (jsonmsg["command"] == "delete"):
msgtype = 24
#modified#######################################
# for flow in jsonmsg["flows"]:
################################################
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
elif (jsonmsg["flow_type"] == "host"):
if (jsonmsg["command"] == "add"):
msgtype = 23
for flow in jsonmsg["flows"]:
if ("dst port" in flow):
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, int(flow["src id"],16), 0) + \
struct.pack("!HQH", 1, int(flow["dst id"],16), int(flow["dst port"],16)) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
else:
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, int(flow["src id"],16), int(flow["src port"],16)) + \
struct.pack("!HQH", 1, int(flow["dst id"],16), 0) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
elif (jsonmsg["command"] == "delete"):
msgtype = 24
#modified####################################
# for flow in jsonmag["flows"]:
#############################################
envimsg = struct.pack("!LHL", 1, 0, int(jsonmsg["flow_id"], 16)) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!HQH", 1, 0, 0) + \
struct.pack("!H", 0)
enviaggmsg += struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
return enviaggmsg
elif ("link_type" in jsonmsg):
#Links
if (jsonmsg["link_type"] == "sw2sw"):
if (jsonmsg["command"] == "add"):
msgtype = 20
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQHQ",
1,
1,int(link["src id"], 16),int(link["src port"], 16),
1,int(link["dst id"], 16),int(link["dst port"], 16),
1e7)
elif (jsonmsg["command"] == "delete"):
msgtype = 21
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQH",
1,
1,int(link["src id"], 16),int(link["src port"], 16),
1,int(link["dst id"], 16),int(link["dst port"], 16))
else:
return ""
elif (jsonmsg["link_type"] == "host2sw"):
#Host to switch
if (jsonmsg["command"] == "add"):
msgtype = 20
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQHQ",
1,
1, int(link["src id"], 16),0,
1, int(link["dst id"], 16),int(link["dst port"], 16),
1e7)
elif (jsonmsg["command"] == "delete"):
msgtype = 21
for link in jsonmsg["links"]:
envimsg += struct.pack("!HHQHHQH",
1,
1,int(link["src id"], 16),0,
1, 0,int(link["src port"], 16),
1, int(link["dst id"], 16),int(link["dst port"], 16))
else:
return ""
else:
print "Unidentified message type"
print jsonmsg
return ""
else:
print "Not a LAVI message"
return ""
return struct.pack("!HBL",len(envimsg)+7,msgtype,0)+envimsg
while True:
#Get ENVI connection
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(('',serveport))
server.listen(10)
print "Waiting for ENVI connection"
socks = [server]
ec_list = []
lc_list = []
#Poll and translate
lavibuffer = ""
eb={}
lb={}
lc={}
ec={}
connected_clients=0
while True:
(read, write, err) = select.select(socks,
[],
socks, 1)
for e in err:
print "Error:", e
for r in read:
if (server == r ):
(enviconn, saddr) = server.accept()
socks.append(enviconn)
ec_list.append(enviconn)
print "Got connection from "+str(saddr) + " SOCK: ", enviconn, " size: ", len(socks);
eb[str(enviconn)]=""
# create corresponding laviconn
# Get LAVI connection
laviconn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socks.append(laviconn)
lc_list.append(laviconn)
try:
laviconn.connect((noxhost, noxport))
print "Connected to NOX at "+str((noxhost,noxport)), " SOCK:", laviconn
except socket.error:
print "Cannot connect to LAVI @ "+noxhost
lb[str(laviconn)]=""
# cross reference
lc[str(enviconn)]=laviconn;
ec[str(laviconn)]=enviconn;
elif ( r in lc_list):
try:
data = r.recv(1)
if (not data):
print "NO DATA BREAK"
break
lb[str(r)] += data
try:
jsonmsg = simplejson.loads( lb[str(r)] )
binarymsg = lavi2envi(jsonmsg)
if (len(binarymsg) != 0):
# print "SEND MESSAGE TO ENVI to: ", r, " from: ", ec[str(r)]
ec[str(r)].send(binarymsg)
lb[str(r)] = ""
except ValueError:
pass
except socket.error:
print "SOCKET ERROR: BREAK"
break
elif ( r in ec_list):
try:
data = None
data = r.recv(1)
except: # this is probably socket error. delete this
r.close()
lc[str(r)].close()
socks.remove(lc[str(r)])
lc_list.remove(lc[str(r)])
socks.remove(r)
ec_list.remove(r)
print "A: delete socket: ", r, " size: ", len(socks);
continue
if (not data):
r.close()
lc[str(r)].close()
try:
socks.remove(lc[str(r)])
lc_list.remove(lc[str(r)])
socks.remove(r)
ec_list.remove(r)
except:
pass
print "B: delete socket: ", r, " size: ", len(socks);
else:
eb[str(r)] += data;
if (len(eb[str(r)]) > 2):
(length, ) = struct.unpack_from("!H",eb[str(r)])
if (length == len(eb[str(r)])):
jsonmsg = envi2lavi(eb[str(r)])
if (jsonmsg != None):
lc[str(r)].send(simplejson.dumps(jsonmsg))
eb[str(r)] = ""
else:
print "Error!: Unregistered socket found"
#Close
print "Closing connections"
if (clientok):
laviconn.shutdown(1)
laviconn.close()
time.sleep(1)
Friday, October 14, 2011
[SOCKET] simple python echo server
#!/usr/bin/env python
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
[SOCKET] socket reuse example
1. bsd case
int on = 1;
if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) < 0)
{
perror("setsockopt(SO_REUSEADDR) failed");
}
2. python case
sock = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
# Enable the SO_REUSEADDR option
sock.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1 )
# Get the size of the socket's send buffer
bufsize = sock.getsockopt( socket.SOL_SOCKET, socket.SO_SNDBUF )
# Get the state of the SO_REUSEADDR option
state = sock.getsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR )
int on = 1;
if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) < 0)
{
perror("setsockopt(SO_REUSEADDR) failed");
}
2. python case
sock = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
# Enable the SO_REUSEADDR option
sock.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1 )
# Get the size of the socket's send buffer
bufsize = sock.getsockopt( socket.SOL_SOCKET, socket.SO_SNDBUF )
# Get the state of the SO_REUSEADDR option
state = sock.getsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR )
Thursday, October 13, 2011
ARP Request Unicasting???
Unicasting ARP Request sounds strange but indeed there are those things.
See Section 2.3.2.1 of RFC1122, "ARP Cache Validation." Specifically,
IMPLEMENTATION:
Four mechanisms have been used, sometimes in
combination, to flush out-of-date cache entries.
~~~
(2) Unicast Poll -- Actively poll the remote host by
periodically sending a point-to-point ARP Request
to it, and delete the entry if no ARP Reply is
received from N successive polls. Again, the
timeout should be on the order of a minute, and
typically N is 2.
See Section 2.3.2.1 of RFC1122, "ARP Cache Validation." Specifically,
IMPLEMENTATION:
Four mechanisms have been used, sometimes in
combination, to flush out-of-date cache entries.
~~~
(2) Unicast Poll -- Actively poll the remote host by
periodically sending a point-to-point ARP Request
to it, and delete the entry if no ARP Reply is
received from N successive polls. Again, the
timeout should be on the order of a minute, and
typically N is 2.
Tuesday, October 11, 2011
what is vsz and rss from the results of ps
Originally from: http://wiki.marandcustomsolutions.com/space/Linux/Understanding+memory+usage
What ps reports
The ps tool can output various pieces of information about a process, such as it's process id, current running state, and resource utilization. Two of the possible outputs are VSZ and RSS, which stand for "virtual set size" and "resident set size", which are commonly used by geeks around the world to see how much memory processes are taking up.
For example, here is the output of ps aux for KEdit:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dbunker 3468 0.0 2.7 25400 14452 ? S 20:19 0:00 kdeinit: kedit
According to ps, KEdit has a virtual size of about 25 megabytes and a resident size of about 14 megabytes (both numbers above are reported in kilobytes). It seems that most people like to randomly choose to accept one number or the other as representing the real memory usage of a process. I'm not going to explain the difference between VSZ and RSS right now but, needless to say, this is the wrong approach; neither number is an accurate picture of what the memory cost of running KEdit is.
Why ps is "wrong" Depending on how you look at it, ps is not reporting the real memory usage of processes. What it is really doing is showing how much real memory each process would take up if it were the only process running. Of course, a typical Linux machine has several dozen processes running at any given time, which means that the VSZ and RSS numbers reported by ps are almost definitely "wrong". In order to understand why, it is necessary to learn how Linux handles shared libraries in programs.
Most major programs on Linux use shared libraries to facilitate certain functionality. For example, a KDE text editing program will use several KDE shared libraries (to allow for interaction with other KDE components), several X libraries (to allow it to display images and copy and pasting), and several general system libraries (to allow it to perform basic operations). Many of these shared libraries, especially commonly used ones like libc, are used by many of the programs running on a Linux system. Due to this sharing, Linux is able to use a great trick: it will load a single copy of the shared libraries into memory and use that one copy for every program that references it.
For better or worse, many tools don't care very much about this very common trick; they simply report how much memory a process uses, regardless of whether that memory is shared with other processes as well. Two programs could therefore use a large shared library and yet have its size count towards both of their memory usage totals; the library is being double-counted, which can be very misleading if you don't know what is going on.
Unfortunately, a perfect representation of process memory usage isn't easy to obtain. Not only do you need to understand how the system really works, but you need to decide how you want to deal with some hard questions. Should a shared library that is only needed for one process be counted in that process's memory usage? If a shared library is used my multiple processes, should its memory usage be evenly distributed among the different processes, or just ignored? There isn't a hard and fast rule here; you might have different answers depending on the situation you're facing. It's easy to see why ps doesn't try harder to report "correct" memory usage totals, given the ambiguity.
Seeing a process's memory map Enough talk; let's see what the situation is with that "huge" KEdit process. To see what KEdit's memory looks like, we'll use the pmap program (with the -d flag):
Address Kbytes Mode Offset Device Mapping
08048000 40 r-x-- 0000000000000000 0fe:00000 kdeinit
08052000 4 rw--- 0000000000009000 0fe:00000 kdeinit
08053000 1164 rw--- 0000000008053000 000:00000 [ anon ]
40000000 84 r-x-- 0000000000000000 0fe:00000 ld-2.3.5.so
40015000 8 rw--- 0000000000014000 0fe:00000 ld-2.3.5.so
40017000 4 rw--- 0000000040017000 000:00000 [ anon ]
40018000 4 r-x-- 0000000000000000 0fe:00000 kedit.so
40019000 4 rw--- 0000000000000000 0fe:00000 kedit.so
40027000 252 r-x-- 0000000000000000 0fe:00000 libkparts.so.2.1.0
40066000 20 rw--- 000000000003e000 0fe:00000 libkparts.so.2.1.0
4006b000 3108 r-x-- 0000000000000000 0fe:00000 libkio.so.4.2.0
40374000 116 rw--- 0000000000309000 0fe:00000 libkio.so.4.2.0
40391000 8 rw--- 0000000040391000 000:00000 [ anon ]
40393000 2644 r-x-- 0000000000000000 0fe:00000 libkdeui.so.4.2.0
40628000 164 rw--- 0000000000295000 0fe:00000 libkdeui.so.4.2.0
40651000 4 rw--- 0000000040651000 000:00000 [ anon ]
40652000 100 r-x-- 0000000000000000 0fe:00000 libkdesu.so.4.2.0
4066b000 4 rw--- 0000000000019000 0fe:00000 libkdesu.so.4.2.0
4066c000 68 r-x-- 0000000000000000 0fe:00000 libkwalletclient.so.1.0.0
4067d000 4 rw--- 0000000000011000 0fe:00000 libkwalletclient.so.1.0.0
4067e000 4 rw--- 000000004067e000 000:00000 [ anon ]
4067f000 2148 r-x-- 0000000000000000 0fe:00000 libkdecore.so.4.2.0
40898000 64 rw--- 0000000000219000 0fe:00000 libkdecore.so.4.2.0
408a8000 8 rw--- 00000000408a8000 000:00000 [ anon ]
… (trimmed) …
mapped: 25404K writeable/private: 2432K shared: 0K
I cut out a lot of the output; the rest is similar to what is shown. Even without the complete output, we can see some very interesting things. One important thing to note about the output is that each shared library is listed twice; once for its code segment and once for its data segment. The code segments have a mode of "r-x--", while the data is set to "rw---". The Kbytes, Mode, and Mapping columns are the only ones we will care about, as the rest are unimportant to the discussion.
If you go through the output, you will find that the lines with the largest Kbytes number are usually the code segments of the included shared libraries (the ones that start with "lib" are the shared libraries). What is great about that is that they are the ones that can be shared between processes. If you factor out all of the parts that are shared between processes, you end up with the "writeable/private" total, which is shown at the bottom of the output. This is what can be considered the incremental cost of this process, factoring out the shared libraries. Therefore, the cost to run this instance of KEdit (assuming that all of the shared libraries were already loaded) is around 2 megabytes. That is quite a different story from the 14 or 25 megabytes that ps reported.
What does it all mean? The moral of this story is that process memory usage on Linux is a complex matter; you can't just run ps and know what is going on. This is especially true when you deal with programs that create a lot of identical children processes, like Apache. ps might report that each Apache process uses 10 megabytes of memory, when the reality might be that the marginal cost of each Apache process is 1 megabyte of memory. This information becomes critial when tuning Apache's MaxClients setting, which determines how many simultaneous requests your server can handle (although see one of my past postings for another way of increasing Apache's performance).
It also shows that it pays to stick with one desktop's software as much as possible. If you run KDE for your desktop, but mostly use Gnome applications, then you are paying a large price for a lot of redundant (but different) shared libraries. By sticking to just KDE or just Gnome apps as much as possible, you reduce your overall memory usage due to the reduced marginal memory cost of running new KDE or Gnome applications, which allows Linux to use more memory for other interesting things (like the file cache, which speeds up file accesses immensely).
What ps reports
The ps tool can output various pieces of information about a process, such as it's process id, current running state, and resource utilization. Two of the possible outputs are VSZ and RSS, which stand for "virtual set size" and "resident set size", which are commonly used by geeks around the world to see how much memory processes are taking up.
For example, here is the output of ps aux for KEdit:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dbunker 3468 0.0 2.7 25400 14452 ? S 20:19 0:00 kdeinit: kedit
According to ps, KEdit has a virtual size of about 25 megabytes and a resident size of about 14 megabytes (both numbers above are reported in kilobytes). It seems that most people like to randomly choose to accept one number or the other as representing the real memory usage of a process. I'm not going to explain the difference between VSZ and RSS right now but, needless to say, this is the wrong approach; neither number is an accurate picture of what the memory cost of running KEdit is.
Why ps is "wrong" Depending on how you look at it, ps is not reporting the real memory usage of processes. What it is really doing is showing how much real memory each process would take up if it were the only process running. Of course, a typical Linux machine has several dozen processes running at any given time, which means that the VSZ and RSS numbers reported by ps are almost definitely "wrong". In order to understand why, it is necessary to learn how Linux handles shared libraries in programs.
Most major programs on Linux use shared libraries to facilitate certain functionality. For example, a KDE text editing program will use several KDE shared libraries (to allow for interaction with other KDE components), several X libraries (to allow it to display images and copy and pasting), and several general system libraries (to allow it to perform basic operations). Many of these shared libraries, especially commonly used ones like libc, are used by many of the programs running on a Linux system. Due to this sharing, Linux is able to use a great trick: it will load a single copy of the shared libraries into memory and use that one copy for every program that references it.
For better or worse, many tools don't care very much about this very common trick; they simply report how much memory a process uses, regardless of whether that memory is shared with other processes as well. Two programs could therefore use a large shared library and yet have its size count towards both of their memory usage totals; the library is being double-counted, which can be very misleading if you don't know what is going on.
Unfortunately, a perfect representation of process memory usage isn't easy to obtain. Not only do you need to understand how the system really works, but you need to decide how you want to deal with some hard questions. Should a shared library that is only needed for one process be counted in that process's memory usage? If a shared library is used my multiple processes, should its memory usage be evenly distributed among the different processes, or just ignored? There isn't a hard and fast rule here; you might have different answers depending on the situation you're facing. It's easy to see why ps doesn't try harder to report "correct" memory usage totals, given the ambiguity.
Seeing a process's memory map Enough talk; let's see what the situation is with that "huge" KEdit process. To see what KEdit's memory looks like, we'll use the pmap program (with the -d flag):
Address Kbytes Mode Offset Device Mapping
08048000 40 r-x-- 0000000000000000 0fe:00000 kdeinit
08052000 4 rw--- 0000000000009000 0fe:00000 kdeinit
08053000 1164 rw--- 0000000008053000 000:00000 [ anon ]
40000000 84 r-x-- 0000000000000000 0fe:00000 ld-2.3.5.so
40015000 8 rw--- 0000000000014000 0fe:00000 ld-2.3.5.so
40017000 4 rw--- 0000000040017000 000:00000 [ anon ]
40018000 4 r-x-- 0000000000000000 0fe:00000 kedit.so
40019000 4 rw--- 0000000000000000 0fe:00000 kedit.so
40027000 252 r-x-- 0000000000000000 0fe:00000 libkparts.so.2.1.0
40066000 20 rw--- 000000000003e000 0fe:00000 libkparts.so.2.1.0
4006b000 3108 r-x-- 0000000000000000 0fe:00000 libkio.so.4.2.0
40374000 116 rw--- 0000000000309000 0fe:00000 libkio.so.4.2.0
40391000 8 rw--- 0000000040391000 000:00000 [ anon ]
40393000 2644 r-x-- 0000000000000000 0fe:00000 libkdeui.so.4.2.0
40628000 164 rw--- 0000000000295000 0fe:00000 libkdeui.so.4.2.0
40651000 4 rw--- 0000000040651000 000:00000 [ anon ]
40652000 100 r-x-- 0000000000000000 0fe:00000 libkdesu.so.4.2.0
4066b000 4 rw--- 0000000000019000 0fe:00000 libkdesu.so.4.2.0
4066c000 68 r-x-- 0000000000000000 0fe:00000 libkwalletclient.so.1.0.0
4067d000 4 rw--- 0000000000011000 0fe:00000 libkwalletclient.so.1.0.0
4067e000 4 rw--- 000000004067e000 000:00000 [ anon ]
4067f000 2148 r-x-- 0000000000000000 0fe:00000 libkdecore.so.4.2.0
40898000 64 rw--- 0000000000219000 0fe:00000 libkdecore.so.4.2.0
408a8000 8 rw--- 00000000408a8000 000:00000 [ anon ]
… (trimmed) …
mapped: 25404K writeable/private: 2432K shared: 0K
I cut out a lot of the output; the rest is similar to what is shown. Even without the complete output, we can see some very interesting things. One important thing to note about the output is that each shared library is listed twice; once for its code segment and once for its data segment. The code segments have a mode of "r-x--", while the data is set to "rw---". The Kbytes, Mode, and Mapping columns are the only ones we will care about, as the rest are unimportant to the discussion.
If you go through the output, you will find that the lines with the largest Kbytes number are usually the code segments of the included shared libraries (the ones that start with "lib" are the shared libraries). What is great about that is that they are the ones that can be shared between processes. If you factor out all of the parts that are shared between processes, you end up with the "writeable/private" total, which is shown at the bottom of the output. This is what can be considered the incremental cost of this process, factoring out the shared libraries. Therefore, the cost to run this instance of KEdit (assuming that all of the shared libraries were already loaded) is around 2 megabytes. That is quite a different story from the 14 or 25 megabytes that ps reported.
What does it all mean? The moral of this story is that process memory usage on Linux is a complex matter; you can't just run ps and know what is going on. This is especially true when you deal with programs that create a lot of identical children processes, like Apache. ps might report that each Apache process uses 10 megabytes of memory, when the reality might be that the marginal cost of each Apache process is 1 megabyte of memory. This information becomes critial when tuning Apache's MaxClients setting, which determines how many simultaneous requests your server can handle (although see one of my past postings for another way of increasing Apache's performance).
It also shows that it pays to stick with one desktop's software as much as possible. If you run KDE for your desktop, but mostly use Gnome applications, then you are paying a large price for a lot of redundant (but different) shared libraries. By sticking to just KDE or just Gnome apps as much as possible, you reduce your overall memory usage due to the reduced marginal memory cost of running new KDE or Gnome applications, which allows Linux to use more memory for other interesting things (like the file cache, which speeds up file accesses immensely).
Subscribe to:
Posts (Atom)