Hi everybody. Many times, I got some messages from visitors of this blog who cannot read french fluently and who were asking for articles in english. So I’ve decided to write this first post in english.
For many years, I’ve been using the whole set of linux tools to take care of my daily tasks as a sysadmin. Sed, awk, cut, along with regular expressions, are extremely powerful tools to process raw data (log files for instance) and present them in a suitable format.
But two years ago, as I was learning python, I noticed it was not just appropriate to developing complex programs but it is also perfectly suitable to achieve my sysadmin work, as it is much faster to complete some of your automated tasks than traditional linux tools.
In this post, let’s just compare both traditional way to deal with log files, vs a python approach to this. I hope you’ll be as convinced as I am.
Let’s take a quick example. I currently have an apache log file composed of 1 million lines. The logs are stored in combined format.
... 5.255.253.44 - - [23/Jul/2016:07:18:33 +0200] "GET /ul/4-air.png HTTP/1.1" 304 186 "-" "Mozilla... 107.23.92.82 - - [23/Jul/2016:07:18:57 +0200] "GET /feed/ HTTP/1.1" 200 7572 "-" "Mozilla... 5.255.253.44 - - [23/Jul/2016:07:20:38 +0200] "GET /ul/wrt54G-wireless-sec.jpg HTTP/1.1" 304 185 "-" "Mozilla... 52.30.101.176 - - [23/Jul/2016:07:20:53 +0200] "GET /robots.txt HTTP/1.1" 200 370 "-" "Mozilla... 146.185.251.139 - - [23/Jul/2016:07:21:49 +0200] "POST /xmlrpc.php HTTP/1.0" 404 8603 "-" "Mozilla... 5.196.4.212 - - [23/Jul/2016:07:22:02 +0200] "GET /feed/ HTTP/1.1" 304 177 "-" "Feed2Imap... 37.187.101.2 - - [23/Jul/2016:07:22:02 +0200] "GET /feed/ HTTP/1.1" 304 158 "-" "Tiny... 5.255.253.44 - - [23/Jul/2016:07:22:43 +0200] "GET /ul/5.jpg HTTP/1.1" 304 185 "-" "Mozilla... 5.255.253.44 - - [23/Jul/2016:07:24:48 +0200] "GET /ul/3-active-console.png HTTP/1.1" 304 186 "-" "Mozilla... ...
Let’s say we want to count the number of times a single IP address accessed a resource on the server, keeping just the 10 most frequent visitors. We would like to present the information in such a form, to create a CSV file for instance.
5.196.197.68;11820 104.236.104.28;7740 107.23.92.82;6940 ...
Bash approach
To do it in bash, we use several tools and we join them together with pipe character.
cat access1M.log | cut -d" " -f1 | sort | uniq -c | sed -e "s/^ *//" | sort -rn | \
awk -F" " '{print $2";"$1}'| head -n10
Eventually we are presented with this result :
91.134.167.121|22750 146.185.251.139|16070 146.185.251.137|13240 5.196.197.68|11820 104.236.104.28|7740 107.23.92.82|6940 37.187.109.125|6250 5.196.4.212|5940 37.187.101.2|5940 176.159.66.16|5270
Processing those 1 million lines of log took 17,6 seconds on my Macbook pro (3,1 GHz Core i7 16GB ram 1TB SSD)
real 0m17.671s user 0m18.353s sys 0m0.342s
Python approach
Now, let’s try to compute the same log file with python. I wrote a little python script that aims at doing the same processing than above.
#!/usr/bin/python
# ----------------------------------------------------
# Name : 10MostFrequentVisitors.py
# ----------------------------------------------------
import re
visits={}
replacements=[["\[|\]", ""]]
try:
fh=open("annexe/access1M.log","r")
except Exception as e:
print "error opening file"
else:
for line in fh:
for replacement in replacements:
src_str,dst_str=replacement
line=re.sub(src_str, dst_str,line)
values=line.split(" ")
ip=values[0]
if ip in visits:
visits[ip]+=1
else:
visits[ip]=1
visits_sorted=sorted(visits, key=visits.__getitem__, reverse=True)
for cpt in range(0,10):
ip=visits_sorted[cpt]
print "{0};{1}".format(ip,visits[ip])
Of course, I obtain the same output here. But in just 6,24s. Python was 2,83 times faster than bash processing the data.
real 0m6.244s user 0m5.980s sys 0m0.154s
Of course writing a script in python usually takes a longer time than combining ten commands with the use of pipe characters. But how many times are you doing the same tasks again and again ? Is it not worth writing a script once and for all ?
I hope you will be convinced now that sometimes, writing a python script can be a substantial gain of time when it comes to repetitive tasks, completed on a frequent (or even regular) basis.