In love with Python

Valentine's Day has come and gone, but I'm still swooning. I'm enamored with Python. Yes, Python the programming language, not the animal. Why am I vying for Python's attention?  Here's why...

import httplib

urlconn = httplib.HTTPConnection('www.somewhere');
urlconn.request("GET", "/somepath/to/login.asp");
response = urlconn.getresponse();
cookie = response.getheader('Set-Cookie');
print cookie;
urlconn.close();

If you run this code (with valid data, of course!), you would receive a response similar to the following:

ASPSESSIONID=AOBPFCCDFKASLMRTYWKNBCDS; path=/

With this tiny bit of code, we can gather session ID's from login pages for analysis.  This data can be used to further assess the security of the login page. We can (and as it sits, will) collect other cookie information from this code. What else could we do here?  A very simple change would allow us to collect other HTTP header information:

print response.getheader('Server');

This would return a response similar to the following:

Microsoft-IIS/6.0

We can also collect lots of other data, including the HTTP response code, other specific header data, or the full response itself.

Of course, this is a very rudimentary example that could be improved upon. We could implement a more flexible program by allowing the URL to be passed via command-line, implement error handling, and do something more useful with the output. For now, I'll leave this as an exercise for the reader. My point here is to illustrate how quickly one can create something useful and usable for a specific purpose.

Happy coding!

Four Linux Commands You Never Knew You Needed (Until Now)

If you're like me, you've probably used open source software such as Linux and GNU utilities for some time now.  In recent years, I've been involved in more research and development activities.  This has led to a combining of different schools of thought for me: that of the system administrator, and that of the developer or "power-user".  I've compiled a few useful but lesser-known (at least to me) commands that I'd like to share that seem to have overlapped projects during the course of my work. These have all saved me time in some way.

watch

Sadly, the watch command is a newer discovery for me. This useful ncurses-based utility allows you to monitor activity generated by the output of other commands over time. Here's a quick example:

watch 'ls -la /tmp'

This will display the contents of the /tmp directory, refreshing it every 2 seconds.  The refresh time is configurable via a parameter, i.e.:

watch -n 10 'ls -la /tmp'

The -n parameter takes input in seconds, thus the above command-line would generate output every 10 seconds. Using watch to monitor a directory for changes is useful in many scenarios. At times, I've found myself using this to monitor processes writing files.  However, here's an example that I use even more frequently:

watch "netstat -na |egrep '(SYN_SENT|ESTABLISHED|TIME_WAIT|FIN_WAIT_1|FIN_WAIT_2)'"

This effectively allows you to monitor the many states of open network sockets as they happen. While there are other tools that can do this (i.e. ntop), but the upside is that watch is included out-of-the-box in many Linux distros. Since we're only manipulating command-line parameters, the level of customization is very high. If you throw in an additional grep statement before grabbing the socket state, you can effectively filter on other criteria, such as IP addresses.  

Figure 1: monitoring network sockets using 'watch'.

It's worth noting that you'll not need to include all of these socket states in most situations. If you're not having issues with connectivity on either end, you're likely to only observe ESTABLISHED and TIME_WAIT.  For obvious reasons, I have found SYN_SENT useful for noting issues with remote hosts. The others are included for completeness.

 whatis

Put quite simply, the whatis utility displays short summaries of man pages on the command-line.

$ whatis whatis

whatis (1)           - display manual page descriptions

Simple, but useful, right?


 cal

As the name indicates, cal displays all or part of a calendar. While not an earth-shaking game-changer, the cal utility can be quite useful. Here are a few examples:

Display a full calendar for the year 2012: cal 2012

Display calendar for November 1955: cal 11 1955

Display calendar for the 8th month of the current year: cal -m 8
This can also be achieved via: cal 8 2013

Figure 2: Demonstrating the 'cal' command.

Perhaps it's my nature in being a "CLI guy", but I much prefer this to clicking through any of the wonky date & time widgets. I also find this to be much faster than most GUI, especially when you're in need of a date many months or years from today. What day of the week is the wife's birthday next year? cal N 2014 - done!

lsblk

List block devices (aka lsblk) is part of the util-linux software package. While it's a relatively new utility to me, it certainly is useful.  The default output of lsblk looks somewhat similar to df

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 666.1M  1 loop /rofs
sda      8:0    0 465.8G  0 disk
├─sda1   8:1    0  39.2M  0 part
├─sda2   8:2    0  14.8G  0 part
└─sda3   8:3    0 450.9G  0 part
sr0     11:0    1 695.3M  0 rom  /cdrom

$ df
Filesystem     1K-blocks   Used Available Use% Mounted on
/cow             1932460 395084   1537376  21% /
udev             1924792      4   1924788   1% /dev
tmpfs             772988    876    772112   1% /run
/dev/sr0          711980 711980         0 100% /cdrom
/dev/loop0        682240 682240         0 100% /rofs
tmpfs            1932460    156   1932304   1% /tmp
none                5120      4      5116   1% /run/lock
none             1932460    180   1932280   1% /run/shm

While the output of each utility is similar, there are some real differences. First, lsblk includes a very clear partition hierarchy in the NAME column. I find this to be very easy to read as well as convenient. Next, we have the MAJ and MIN columns, which refer to Linux device major and minor numbers. The RM and RO columns, which stand for 'removable' and 'read-only' respectively, are things I wish df had been telling me all along. The TYPE column is self explanatory, but is another example of output that I find to be useful.

While the default output of lsblk is great, it's also very customizable. The -o switch allows you to specify a comma-separated list of which columns are present in the output. Here's an example:

$ lsblk -o NAME,MODEL
NAME   MODEL
loop0 
sda    WDC WD5000AAKX-7
├─sda1
├─sda2
└─sda3
sr0    DVD+-RW GH70N  

This is definitely a convenient way to determine where physical disks are mapped. I no longer have to dig through dmesg output! Well... not for this reason, anyway.

If you're like me, you may have many physical systems for various reasons. Do you have a built-in multi-card reader on your PC? Do you remember which device maps to which type of reader? I don't.

$ lsblk -a -o NAME,MODEL
NAME   MODEL
ram0  
ram1  
ram2  
[ ... SNIP ... ]
loop7 
sda    WDC WD5000AAKX-7
├─sda1
├─sda2
└─sda3
sr0    DVD+-RW GH70N  
sdb    SD/MMC         
sdc    Compact Flash  
sdd    SM/xD Picture  
sde    MS/MS-Pro

Now I recall! The -a switch shows all devices. Note that the output has been trimmed.  There are a lot of other columns available, including STATE, OWNER, GROUP, and MODE.  Check lsblk --help for many more options.

That's all for now. I hope you find this information to be useful in your day-to-day CLI adventures.

Best Albums for Writing Code

If you're like me, you probably code on a somewhat regular basis. I find myself having a reason to code a new tool, script, or web page on a weekly basis. What helps you focus and get you though the process? For me, music is the answer. I've created a collection on Referly of the best albums for writing code.  Check it out here. Feel free to leave your comments about what music motivates you. I'd love to hear them.