Tag Archives: prefix

How to Set Wget Target Directory Tips, Tricks and Tutorials 12 JUN 2014

If I ever need to pull something down from a website while working on an Ubuntu Server instance, then without a doubt, wget is my go to guy. From the man pages:

GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

Wget is non-interactive, meaning that it can work in the background, while the user is not logged on. This allows you to start a retrieval and disconnect from the system, letting Wget finish the work. By contrast, most of the Web browsers require constant user’s presence, which can be a great hindrance when transferring a lot of data.

Wget can follow links in HTML, XHTML, and CSS pages, to create local versions of remote web sites, fully recreating the directory structure of the original site (recursive downloading). Wget can be instructed to convert the links in downloaded files to point at the local files, for offline viewing.

Wget has been designed for robustness over slow or unstable network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved. If the server supports regetting, it will instruct the server to continue the download from where it left off.

So all in all, a VERY useful tool. The question being solved today is how to go about setting or specifying the target directory wget should be saving the downloaded files to. The short answer is that in the strictest sense you can’t – but you can specify a string prefix to add to all downloaded files, thereby allowing you to essentially set the wget target directory!

Again from the man pages:

-P prefix
Set directory prefix to prefix.  The directory prefix is the directory where all other files and sub-directories will be saved to, i.e. the top of the retrieval tree.  The default is . (the current directory).

In practice, if we want to download a file to /home/craiglotter then we would add -P /home/craiglotter/ to our command. In practice:

wget -P /home/craiglotter/ http://www.craiglotter.co.za/wp-uploads/image.jpg

The above would download the image.jpg file to /home/craiglotter/image.jpg.

Useful to know.

ubuntu black and orange logo

Related Link: http://manpages.ubuntu.com/manpages/trusty/man1/wget.1.html

Ubuntu: How to add a Suffix (or Prefix) to a Line of Text in a Text File using AWK CodeUnit 18 JUN 2012

If you are looking for a top notch tool to carry out quick text manipulations on text files under Ubuntu terminal, you could probably do no better than by looking up AWK (or I suppose SED if you are so inclined).

Anyway, back to AWK. The AWK utility is a data extraction and reporting tool that uses a data-driven scripting language consisting of a set of actions to be taken against textual data (either in files or data streams) for the purpose of producing formatted reports. The language used by awk extensively uses the string datatype, associative arrays (that is, arrays indexed by key strings), and regular expressions.

AWK was created at Bell Labs in the 1970s, and its name is derived from the family names of its authors – Alfred Aho, Peter Weinberger, and Brian Kernighan. ‘awk’, when written in all lowercase letters, refers to the Unix or Plan 9 program that runs other programs written in the AWK programming language.

Okay, now that we have some history, lets see just how easy it is to add either a prefix or suffix to each line contained in our text file using awk:

#add a prefix to each line in the text file
awk '{ printf("myprefix %sn", $l);}' sample-text-file.txt

#add a suffix to each line in the text file
awk '{ printf("%s mysuffixn", $l);}' sample-text-file.txt

#add both a prefix and a suffix to each line in the text file
awk '{ printf("myprefix %s mysuffixn", $l);}' sample-text-file.txt

The first example will result in each line in the sample text file being prepended with the word ‘myprefix’. The second example will result in each line in the sample text file being appended with the work ‘mysuffix’. I doubt that at this stage I still need to spell out what the third example does!

If you wish to save these changes to a text file, using the standard IO redirect functionality, i.e. the > sign:

#save output to a file
awk '{ printf("myprefix %s mysuffixn", $l);}' sample-text-file.txt > altered-text-file.txt

(Note that you shouldn’t direct the output at the input file as you’ll seriously screw things up. Rather save to a new file instead).


Related Link: http://en.wikipedia.org/wiki/AWK