Madhav Kobal's Blog

This blog will be dedicated to Linux, Open Source and Technology news, affairs, how-tos and virtually EVERYTHING in these domains.

Posts Tagged ‘Utility’

Convert a PDF file into a single Image using ImageMagick

Posted by madhavkobal on 18/06/2010

Don’t know when you might feel the necessity to convert a pdf into a .jpg or a .png file, but in case you have to, then ImageMagick will make this job really easy for you. ImageMagick is a great opensource command line utility for image manipulation.

Since, we are dealing with image conversion we will use the “convert” script, provided by ImageMagick.

[shredder12]$ convert file.pdf image.png

If the pdf file(file.pdf) has only one page then a single image.png file will be formed. In case of multiple pages, same number of image-[n].png files will be produced. For example, a three page pdf will result into three image files – image-0.png, image-1.png and image-2.png.

If you want to combine/stitch all these images together to form a single one, something similar to taking a snapshot of the whole pdf, then just run the following command.

[shredder12]$ convert image-* -append single_image.png

This will result in a single image file.

Advertisements

Posted in Uncategorized | Tagged: | Leave a Comment »

Desktop search engines for Linux

Posted by madhavkobal on 04/12/2009

The file system management is fine in Linux, but effectively searching the file system for specific files like mp3 (.mp3) , text (.txt), movie (.avi, .divx etc.), pictures (.jpeg, .bmp, .png) or documents (.doc) is one thing that most oflinux users miss from their Windows Desktop.

The inbuilt search interface in most of the linux distros is not so user-friendly and effective, so we here present some of the better alternative Desktop search engines for linux:
Beagle

http://beagle-project.org/

Beagle is one of the most useful and exhaustive desktop search replacement we could find for linux.
The list of supported files types is very large ranging from images, audio, video, documents, DEB and RPM packages, Email from Evolution and Thunderbird , IM logs and even RSS feeds.
You can even add a new file type by editing the file at : /etc/beagle/external-filters.xml

Beagle can be invoked and run either from Command line or Web interface, while Command line is more powerful as you can specify exact phrases, words or even logical operators (AND, OR) or even use metadata of audio and image files.

Beagle achieves efficiency by building non-redundant index of files, which is comparatively smaller than other engines.
For a 45 Gb partition, index was around 700 Mb.
Google Desktop Search

http://desktop.google.com/linux/

Google Desktop search for linux also supports many file types like Openoffice and MSoffice files, PDF, Html and others.
But downsides are:

1. It does not index zipped archives.
2. You cannot add new file types at all.
The noteworthy feature is you can point ot the exact directory where you are looking for a particular life. Otherwise, it is just useful for average office use and is useless for a power user like me.
DocFetcher

http://docfetcher.sourceforge.net/en/index.html

Supported file types: HTML, plain text, PDF, Microsoft Office (doc, xls, ppt), Microsoft Office 2007 (docx, xlsx, pptx), OpenOffice.org Writer, Calc, Draw, and Impress, RTF, AbiWord (abw, abw.gz, zabw), CHM, Visio, SVG.

DocFetcher is coded in Java, so naturally it is fast in indexing and easy on CPU resources.
It also supports regular expressions, phrase search and logical operators (AND, OR, NOT). Like GDS (Google Desktop search) it also does not search zipped archives.

One interesting query feature found is: Boosting terms, using which you can assign custom weights to words, thus increasing or decreasing the level of matching of words.
Tracker

http://projects.gnome.org/tracker

Tracker is a part of Gnome project.
It introduces the concept of file tags, which is in my view overcomplicates the file system management.

The list of file types supported is quite large, still there is no Bookmarks, EMails or Contact search support.
The biggest downside is that indexer is the slowest among all those we compared.

Posted in Uncategorized | Tagged: , | 3 Comments »

aria2c – Linux Command Line Download Utility

Posted by madhavkobal on 29/10/2009

aria2 is a utility for downloading files. The supported protocols are HTTP(S), FTP, BitTorrent, and Metalink. It has a powerful segmented downloading ability, downloading a file from multiple sources and multiple protocols and utilizing your download bandwidth to the maximum. It supports downloading a file from HTTP(S)/FTP and BitTorrent at the same time, while the data downloaded from HTTP(S)/FTP is uploaded to the BitTorrent swarm. Using Metalinks chunk checksums, aria2 automatically validates chunks of data while downloading a file like BitTorrent.

It continually trims out slow servers and favors faster ones, and if a file’s sources contain both HTTP(S) / FTP and BitTorrent URIs, content downloaded through HTTP(S) / FTP will be uploaded to the BitTorrent swarm.

aria2c [OPTIONS] [URL | TORRENT_FILE | METALINK_FILE]…


Examples

HTTP/FTP Segmented Download
Download a file using 1 connection
aria2c http://host/file.zip
Note
To pause a download, press Ctrl-C. You can resume the transfer by run aria2c with the same argument at the same directory. You can change URLs as long as they are pointing to the same file.
Download a file using 2 connections
aria2c -s2 http://host/file.zip
Download a file from 2 difference HTTP servers
aria2c http://host1/file.zip ftp://host2/file.zip
Download a file from HTTP and FTP servers
aria2c http://host1/file.zip ftp://host2/file.zip
Download files listed in a file concurrently
aria2c -ifiles.txt -j5
Note
-j option specifies the number of concurrent downloads.

Metalink Download

Download files with remote Metalink
aria2c --follow-metalink=mem http://host/file.metalink
Download using a local metalink file
aria2c -p -t10 --lowest-speed-limit=4000 -Mtest.metalink
Note
To pause a download, press Ctrl-C. You can resume the transfer by run aria2c with the same argument at the same directory.
Download only selected files using index
aria2c -Mtest.metalink --select-file=1-4,8
Note
The index is printed to the console using -S option.
Download a file using a local .metalink file with user preference
aria2c -Mtest.metalink --metalink-location=JP,US --metalink-version=1.1 --metalink-language=en-US

BitTorrent Download

Download files from remote BitTorrent file
aria2c --follow-bittorrent=mem http://host/file.torrent
Download using a local torrent file
aria2c --max-upload-limit=40K -Tfile.torrent
Note
--max-upload-limit specifies the max of upload rate.Note
To pause a download, press Ctrl-C. You can resume the transfer by run aria2c with the same argument at the same directory.

Download a file using torrent and HTTP/FTP server
aria2c -Ttest.torrent http://host1/file ftp://host2/file
Note
Downloading multi file torrent with HTTP/FTP is not supported.
Download only selected files using index(usually called "selectable download")
aria2c -Ttest.torrent --select-file=1-4,8
Note
The index is printed to the console using -S option.
Change the listening port for incoming peer
aria2c -Ttest.torrent --listen-port=7000-7001,8000
Note
Since aria2 doesn't configure firewall or router for port forwarding, it's up to you to do it manually.
Specify the condition to stop program after torrent download finished
aria2c -Ttest.torrent --seed-time=120 --seed-ratio=1.0
Note
In the above example, the program exists when the 120 minutes has elapsed since download completed or seed ratio reaches 1.0.
Throttle upload speed
aria2c -Ttest.torrent --max-upload-limit=100K
Enable DHT
aria2c -Ttest.torrent --enable-dht --dht-listen-port=6881
Note
DHT uses udp port. Since aria2 doesn't configure firewall or router for port forwarding, it's up to you to do it manually.

More advanced HTTP features

Load cookies
aria2c --load-cookies=cookies.txt http://host/file.zip
Note
You can use Firefox/Mozilla's cookie file without modification.
Resume download started by web browsers or another programs
aria2c -c -s2 http://host/partiallydownloadedfile.zip

And more advanced features

Throttle download speed
aria2c -Mtest.metalink --max-download-limit=100K
Repair a damaged download using --check-integrity option
aria2c -Mtest.metalink --check-integrity=true
Note
This option is only available used with BitTorrent or metalink with chunk checksums.
Drop connection if download speed is lower than specified value
aria2c -Mtest.metalink --lowest-speed-limit=10K
Parameterized URI support
You can specify set of parts:
aria2c -P http://{host1,host2,host3}/file.iso
You can specify numeric sequence:
aria2c -Z -P http://host/image[000-100].png
Note
-Z option is required if the all URIs don't point to the same file, such as the above example.You can specify step counter:

aria2c -Z -P http://host/image[A-Z:2].png
BitTorrent Encryption
Encrypt whole payload using ARC4:
aria2c -Tfile.torrent --bt-min-crypto-level=arc4 --bt-require-crypto=true

 

Posted in Uncategorized | Tagged: | Leave a Comment »

Wget Commands You Diden’t Hear About Before

Posted by madhavkobal on 22/10/2009

Wget is a free utility for non-interactive download of files from the Web. It supports http, https, and ftp protocols, as well as retrieval through http proxies. Every linux user use terminal, and using the terminal need from you to have some knowledge of command line, so today i choosed you a list of command line wget that perhaps you diden`t hear about them before.

1- Wget basic command list this surely you did hear see it and use it before:

cd picdir/ && wget -nd -pHEKk http://www.unixmen.com/me.jpg

Mean store the current browsable photo to the current picdir directory

wget -c http://www.unixmen/software.de

Download a file with the ability to stop the download and resume later

wget -r -nd -np -l1 -A ‘*.jpg’ http://www.unixmen.com/dir/

Download a site of files to the current directory

wget -r http://www.example.com
Download an entire website

echo ‘wget -c http://www.example.com/files.iso’ | at 09:00
Start a download at any given time

wget ftp://remote/filex.iso/

The usage of FTP is as simple. Wget will take care of login and password

wget –limit-rate=30k

Limit download of a link in 30 Kb/s

wget -nv –spider –force-html -i bookmarks.html

Check the links in a fi

wget –mirror http://www.example.com/

Update a local copy of a website

2- This wget command save a html page and convert it to a .pdf

wget $URL | htmldoc --webpage -f "$URL".pdf - ; xpdf "$URL".pdf &


3- Wget command to get photos from picasa Album :

wget 'link of a Picasa WebAlbum'
-O - |perl -e'while(<>){while(s/"media":{"content":\[{"url":"(.+?\.JPG)//){print
"$1\n"}}' |wget -w1 -i -


4-Check twitter if you can connect :

wget http://twitter.com/help/test.json -q -O -


5– Wget command to get all the Zips files and Pdf from a website :

wget --reject html,htm --accept pdf,zip -rl1 url

If the website use https then :

wget --reject html,htm --accept pdf,zip -rl1 --no-check-certificate https-url


6- Wget command to check if a remote file exist :

wget --spider -v http://www.server.com/path/file.ext


7- Wget command to download files from Rapideshare primium

wget -c -t 1 --load-cookies ~/.cookies/rapidshare <URL>


8- Wget command to extract a tarball file from a host without local saving :

wget -qO - "http://www.tarball.com/tarball.gz" | tar zxvf -


9- Block known dirty hosts from reaching your machine :

wget -qO - http://infiltrated.net/blacklisted|awk '!/#|[a-z]/&&/./{print "iptables -A INPUT -s "$1" -j DROP"}'

Blacklisted is a compiled list of all known dirty hosts (botnets, spammers, bruteforcers, etc.) which is updated on an hourly basis. This command will get the list and create the rules for you, if you want them automatically blocked, append |sh to the end of the command line. It’s a more practical solution to block all and allow in specifics however, there are many who don’t or can’t do this which is where this script will come in handy. For those using ipfw, a quick fix would be {print “add deny ip from “$1” to any}. Posted in the sample output are the top two entries. Be advised the blacklisted file itself filters out RFC1918 addresses (10.x.x.x, 172.16-31.x.x, 192.168.x.x) however, it is advisable you check/parse the list before you implement the rules


10- Wget command to download the entire website :

wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com

Original Author :  M. Zinoune -Zinovsky-

Posted in Uncategorized | Tagged: , , | Leave a Comment »

“top” like commands / apps

Posted by madhavkobal on 08/10/2009

You already know the Linux command called “top”, and also iotop and iftop (from previous Web Upd8 posts), but I bet you don’t know some of the other 14 top-like Linux apps:

htop – Undoubtedly the most famous of the top-like tools. It implements some extra options to the “top” command, like colors, ability to scroll horizontally and vertically, and a better interaction with the processes listed.

itop – Monitors the system breaks.

atop – Monitor for system resources. Displays swap, network, disk, cpu, activity, among other processes.

sntop – Pings a list of hosts and indicates whether they are active or inactive. Must be configured in /etc/sntoprc.

mtop, mytop – tools that monitor the performance and queries of a mysql database.

apachetop – Displays information about Apache, such as requests per second, bytes per second and the most popular URLs.

ptop – Similar to mtop, but specific to postgres.

dnstop – Displays information about the network’s DNS traffic.

virt-top – Displays details of virtualised systems: cpu, memory, uptime. You can monitor any system based on libvirt virtualisation such as Xen and KVM.

iptstate – Displays information about connections which pass through your firewall / gateway. It is basically a netstat that monitors the connections handled by iptables. Depends on module CONNTRACK.

pktstat, jnettop – Displays information on the network bandwidth, each with its peculiarity – From the same class as previously mentioned iftop.

nethogs – network information, but instead of displaying the traffic by protocol, nethogs shows the bandwidth usage by process. Very interesting.

You need to download and install this packages

Posted in Uncategorized | Tagged: | Leave a Comment »

SSH From Your Mobile Device

Posted by madhavkobal on 07/10/2009

Modern mobile phones and PDAs have increasingly sophisticated data/internet connectivity. This is particularly great for browsing the web on the train, but it’s also good for keeping an eye on your servers while you’re out and about. (I once fixed my web server from the middle of a muddy field at the Glastonbury Festival, which I thought was quite good going.) Here’s a quick roundup of SSH applications available for various platforms.

  • G1 Android: ConnectBot (or get from Marketplace). Includes support for SSH keys, which is useful on a mobile platform where you may need to reconnect occasionally.
  • Palm/Treo devices: pSSH. SSH2 for Palm OS 5 and up; TuSSH is another alternative if you want SSH1 or Palm OS 4. It does warn that it may not be entirely secure and shouldn’t be used for security-critical applications in part because it doesn’t use device-specific random number generation. It’s got a neat on-screen keyboard, and it can support SSH key auth.
  • Blackberry: MidpSSH. There’s a useful documentation blog. This should also work on other Java-compliant devices. It supports a predictive text option, which may be useful if you have a device that doesn’t have a full keyboard. It supports public key auth, however, there is no facility for a passphrase for the key. It also has macro support to make typing long/common strings easier.

Symbian devices: The well-known free SSH client PuTTY is available for Symbian. It supports public key authentication but only for keys created using PuTTYGen in Windows. The download comes with excellent documentation, which is also available online.

Original Author : Zinoune

Posted in Uncategorized | Tagged: , | Leave a Comment »

boot.kernel.org(BKO) : Booting your machine over HTTP

Posted by madhavkobal on 26/09/2009

Objective of this project is to let users boot their machines with bare minimal requirements on their side, and we will handle most of the problem that you may face in booting. All that user needs is Internet connectivity and a small program (gpxe) to boot the machine. This gpxe program provides network booting facility. You can download these program from our site (available at Downloads section). Also, you can find a list of distributions and tools that can be booted with BKO here. Your small 56KB gpxe program can boot any of these options!


Download:

USB Image Floppy Image CD ISO


Screenshots:



gpxe is booting the machine.

Menu provided by gpxe

Selecting which tool/distribution to boot from

Finalizing mirror and other options

DSL (Damm Small Linux) Booting

DSL Running.

How to use it?

Floppy Image CD ISO USB Image
  • Download an image file (see above)
Make a bootable floppy:

  • Linux: Insert a floppy disk into a drive, use something like:
    cat gpxe.dsk > /dev/fd0

    or

    dd if=gpxe.dsk of=/dev/fd0

    where fd0 is your floppy drive.

  • Windows: Insert a floppy into your drive, Use rawrite, or dd for windows
  • Mac: Mac has floppy drives?!
Make a bootable CD:

  • Linux: Insert a blank cd into a CD/DVD drive, use something like:
    cdrecord -sao
    You can refer this howto for details.
  • Windows: You can use nero or window’s inbuilt CD writer to burn the gpxe.iso on CD.
  • Mac: Just use the Finder to burn the ISO.
Make a bootable USB:
Warning: Backup your important data before using USB

  • Linux: Insert a USB disk, find it’s device name of USB. Then use following command:
    cat gpxe.dsk > /dev/sdX

    or

    dd if=gpxe.dsk of=/dev/sdX

    where sdX is your usb drive.

  • Windows:You can use dd for windows for Creating bootable USB disks.
  • Mac: the dd command mentioned above for linux should also work for MAC.
We recommend creating BKO enabled USB, which also allows you to store data, without wasting much space. You can refer why BKO enabled USB is better option.
Steps to follow

  • copy it in floppy, USB or in CDROM (check download section for instructions)
  • Make sure that you know what kind of network you have (Which entries will be needed)
  • Boot from the media in which you had copied the gpxe Image
  • In case of Static networking, you need to provide information about your connection, but if you are having DHCP, it should work out of the box.
  • You will be presented with menu showing available options to boot from, select one of them.
  • Your system should boot in the sytem you select
  • Once you are done with it, reboot the system and remvoe the media, you should get back your old system
  • BKO is still under development. If you encounter any problems anywhere, please refer troubleshooting and FAQ page for explanation. If you don’t find any solution to your problem, then feel free to contact us.
  • If you have any interesting experience with BKO, then shoot an email to us

Posted in Uncategorized | Tagged: , | Leave a Comment »

ss – Another Utility to Investigate Sockets

Posted by madhavkobal on 20/09/2009

After configuring services on the network in Linux system, it’s important for you to keep tabs on the ports that are actually listening on the system’s network interfaces. This is all more important because open ports are evidence of an intrusion. There are a Linux monitoring tools that will allow you the knowledge of open ports. There are some basic approaches for listing the ports that are listening on the network. Let’s have a look at one of the most easiest reliable ones.

We would like to show you the ss command to inspect the socket statistics. This is more reliable command that displays more TCP and state information than any other tools.

The ss command provides information about

  • All TCP sockets.
  • All UDP sockets.
  • All established ssh / ftp / http / https connections.
  • All local processes connected to X server.
  • All the tcp sockets in state FIN-WAIT-1 and much more.

ss  is  used  to  dump socket statistics. It allows showing information similar to netstat.  It can display more TCP and state informations than other tools

To display Socket Summary type this in the terminal

# ss -s

Sample Output

ss

Display all the open ports

# ss -l

Sample Output

ss2

To see the process names using open socket type this command

# ss -pl

Track who is responsible for opening socket/port

# ss -lp | grep

To show all the TCP sockets

# ss -t -a

To display all UDP sockets

# ss -u -a

Posted in Uncategorized | Tagged: , | 2 Comments »

Mastering the Bash History

Posted by madhavkobal on 06/09/2009

The Borne Again Shell, better known as bash, is the default for most Linux distributions. It’s extremely powerful as far as command shells go, and holds all kinds of nifty tricks for those willing to learn. One of the best features is, in my opinion, the command history system.

There are many options to use with the bash history, some of them I use nearly every time I open a command shell. Today I’ll be covering some of the most useful aspects of the history, but I certainly won’t be able to cover them all.

Up/Down Arrows

Many of you probably know this one, but for anyone who doesn’t, it’ll make your life a lot easier. Any time you’re in the command shell, simply use the up and down arrows on your keyboard to move through the list of previously entered commands. Next time you enter a long command that turns out to have a typo, you don’t have to retype the whole thing just hit the up arrow and make whatever edits are needed.

!!

A similar feature to the one listed above is the double bang (!!). Those characters entered in the shell represent the last typed command. Let’s say you’re trying to run a command only to find out that you need root privileges to do it. Instead of retyping the whole command, you can simply enter

sudo !!

Bash will substitute the !! for the previous command, as in the example below.

bash-history-bangbang

This also works further backward, you can do something like

!-5

to recall the command 5 entries back in the history.

!(word)

Perhaps a few days ago you typed a long, complex command into your shell, for example a series of options for “wterm”. You can find and re-enter than command by using the history’s built in search. You simply add the first few letters of the command after the ! and bash will find it.

Example of !(word)

Be careful with this one. You may end up running something other than what you expected. Don’t use this for potentially dangerous commands like “rm” or “fdisk”.

!?(word)

This is similar to the last feature in that it searches the history for the word entered, but unlike !(word) this will find the word anywhere in the command, not just the beginning.

bash-history-bangquestion

Be careful with this one. You may end up running something other than what you expected. Don’t use this for potentially dangerous commands like “rm” or “fdisk”.

Ctrl-R

This one may be my personal favorite history tool. It’s a bit like the !? item above, but interactive. In your command shell, hit Ctrl-R, and it will begin a search. As you type, bash will search the history and show you the results as you type. When it shows the command you want, simply hit enter and it will run that command. This can be safer than things like !? because you can see what the command will be before you run it, so you don’t have to guess or rely on memory.

Example of Ctrl-R in the shell

Manual Search

Perhaps you don’t like the way Ctrl-R works, or you want to see ALL the commands you typed, or just the ones that contain a particular word. The history command will show you your full command history, along with the history number of each command (hold that thought, we’ll cover those numbers in a moment).

You can also specify the number of items the history command will show. To see the last 10 entries in the history, you could type

history 10

To see all history entries that contain a particular word, you can use grep to filter the results, like below.

Example of using grep to filter history list

!(number)

When you use the history command to view your command history list, you’ll notice each item in the list has a number associated with it. You can use that to recall that particular command. For example, item 87 could be run again by entering

!87

into your shell.

Word replacement

This one’s pretty great. How many times have you written out some big long command only to realize you put, say, hdd when you wanted hdc? Well bash has you covered. You can replace a word in the previous command with another using “^”, as in the example below

Example of word replacement with "^"

Managing Your History

By default, the history is saved in ~/.bash_history. You can deal with this file however you see fit, but there are some useful things you could try to manage it automatically. Namely, setting the environment variables HISTSIZE and HISTFILESIZE. To set the number of entries that can be saved in the history file to 1000, enter

export HISTFILESIZE=1000

into your shell. This will limit the file to 1000 entries. If you want to make this change permanent, put that export command into your .bashrc or .bash_profile file.

original Author : Joshua Price

Posted in Uncategorized | Tagged: , | Leave a Comment »

Mount a remote file system using sshfs

Posted by madhavkobal on 26/08/2009

You can access a remote file system securely using sshfs and fuse which is a command to mount a remote filesystem encrypted through ssh.

This way you will be able to access remote files as if they were on you machine, just remember that if the connection between the computers is slow, the access will also be pretty slow.

Package needed

sshfs
fuse-utils

Next step is to mount the fuse module

modprobe fuse

Next create the mount point

mkdir /mnt/remote-fs
chown [your-user]:[your-group] /mnt/remote-fs/

Add yourself to the fuse group

adduser [your-user] fuse

Untill here all the command should be issued as root, now switch to your users and mount the remote filesystem.

sshfs remote-user@remote.server:/remote/directory /mnt/remote-fs/

It will now ask you accept the key if this is the first time you connect to that PC using ssh, and then the password, or only the password if this is not the first time you use ssh to connect to the remote.server

Posted in Uncategorized | Tagged: , , | Leave a Comment »