"Some say God speaks to him in UNIX..."

SUBJECT		SUBJECT		SUBJECT		SUBJECT
VI Editor Tips		Shell Script (.ksh) Tips		awk & sed Tips		SQL Database Tips

02 April 2010: Join me in this tribute to the late Dr Henry Edward Roberts, inventor of the Altair 8800 computer.

This is an ongoing work, is maybe almost 1% done, so check back often for updates.

These script examples are actual, written-for-production, and used in: Farmland Industries, Sprint, USDA, and Qwest.

It was rewarding to receive these farewell emails on leaving Farmland, after 7 years as their sole, on-site "IT" guy.

As oldest technical person wherever I worked, it was easy to feel related to that cunningest of Dilbert characters, "Wally"

None of the script examples are newer than 2005 and most will be from the 1990s.

Most have had ".txt" appended to their runtime file names so they will be displayed OK by your web browser.

All the examples represent a fertile copy & paste resource of logically and syntactically correct programming dogma.

All of the script examples are based on the Bourne Shell (sh) and the Korn Shell (ksh). In 2009 the current default for Linux is the "Bourne Again" Shell (bash).

The bash shell may not handle the exampled sh and ksh scripts correctly due to likely changes in both syntax and capability in the intervening period.

Important: Many of the script examples begin with a line something like this:   #!/bin/ksh
This directive seeks to have the script interpreted (i.e., run) using the shell within the path indicated. This may or may not work as expected (or at all) in your environment. If the script fails to run, then edit line #1 to be this:   #!/bin/bash

Oracle & Informix sql (aka "Sequel") database scripts may be either standalone or embedded within sh or ksh scripts. Ditto for any isql (Informix) and sqlplus (Oracle) scripts.

Various UNIX/Linux ("U/L") brands and release levels may utilize directories, utility programs, and configuration (and log) files that may be of different names and in different locations than was the case in the U/L environment that was in effect at the time and place when the TT&E were commited to text.

The various U/L utility programs (e.g, tar  ls  and ftp) in your U/L environment may have more, less, and/or different modifying arguments available than for the same-named utilities in the U/L environment that was in effect at the time and place when the TT&E were commited to text.

There may be entirely new (and/or more powerful) utilities available in your environment (e.g., yum) than were available in the U/L environment that was in effect at the time and place when the TT&E were commited to text.

The web is an inexhaustible resource for all things U/L. Use it to your advantage. This page is embarassingly feeble.

BACK TO TOP

SHELL TIPS, TRICKS, and EXAMPLES

On Linux systems, Windows-like GUIs (e.g., KDE & Gnome) often are assigned to application-bound Users who are never expected (or allowed) to see or use a command-line prompt.

Those programmers who must use MS Windows for all or part of their computer life can find a rich UNIX-like programming environment with either the Mortice Kern Toolkit or the public cygwin (free).

The UNIX "shell" is a command-line user interface that has several popular versions. The earliest is sh followed by ksh, and finally bash (on most Linux-based systems). Every user may request (or be assigned) any of these three shells. A User is advised to accept and use the "standard" (i.e., default) shell for his system.

The interested reader is referred to this on-line bash reference manual. And here is the on-line Korn shell reference.

Effective use of variables and functions within UNIX/Linux shell scripts is an essential skill for U/L Admins and Users alike. A shell script can be thought of as an electronic circuit. The latter is composed of conductors, capacitors, resistors, transistors, etc. The job of the circuit is to do something useful, such as tune and play music from an FM station. Similarly, a shell script has a specific job to do. Here is a link for shell command-line examples: pixelbeat.org (thanks to Zach Lummis for the tip). Be mindful that the examples are for common operations and are based on linux.

A script variable can be likened to an electronic capacitor in the physical world. The capacitor is first energized to an appropriate voltage by the circuit, and this charge is held until needed later. Likewise, a variable in a script can be "charged" (populated) with a "voltage" (numeric or character content) by the programmer. At some later time, the running script can use or alter the variable content based on runtime events (such as the login id and current directory content).

Variables are often used within scripts to set up log file targets near the beginning of a script such as in this example:

lgdir="${HOME}/log"
tmpfi="/tmp/junk$$"
outfi="/tmp/`basename ${0} .ksh`.txt" # Output File
logfi="$lgdir/`basename ${0} .ksh`.log" # Log File

In the above code segment the script uses $ to evaluate (i.e., utilize the contents of) three variables: {HOME} {0} lgdir
The four variables lgdir tmpfi outfi logfi are each simultaneously created and populated in a single statement like this: tmpfi="/tmp/junk$$" Here, use of the $$ construct appends the Process Id Number to the word "junk".

Here is an excellent link to good web based reference for use of variables.

A script function is the equivalent of an integrated circuit in the electronic world. It is a virtual "black box" with a simple, dedicated purpose. The function has input and output just as its electronic counterpart.

As an example, below is the "_dang" function which I devised to intercept (with trap) a User kill-command (Control-C on the keyboard, and signal 9 to UNIX) and then perform an orderly, and logged, termination of the script:

function _dang
{ # Required Arguments:
# $1 = Name of this Program
# $2 = Exit Error-Code (non-zero)
# $3 = User-Abuse string
cat << BOX | tee -a $log_fi

#### ABNORMAL TERMINATION ALERT ####

SCRIPT ABENDED: `which $1`
AS OF: `date`
LOG FILE: $log_fi
EXIT CODE: $2
REASON: $3

BOX
exit $2
} # EOF _dang

And here is how the _dang function is accessed later within the same script:

# Trap and log User-termination:
trap "_dang \"$0\" 9 \"Terminated by User\"." 2

Here is an excellent link to good web based reference for use of functions.

This handy script-menu template is a working example of how and why to use variables, functions, loops, and conditionals in your scripts: Menu_DRM.ksh - I consider it to be the single most important example to study and analyze.
BACK TO TOP

---

VI EDITOR TIPS, TRICKS, and EXAMPLES

Before one can do much with UNIX/Linux one must become skillful with use of its native text editor, vi (short for "visual"). vi is found on every U/L machine and is used for creating and editing scripts, configuration files, and program source code.

Most of the U/L world has to co-exist with MS-Windows, sometimes to a very large extent. A good way to maintain vi skills while using a Windows machine is to download and install a vi-workalike text editor, called vim, onto the Windows platform. Thereafter, you can use vim on text (.txt) files and give up the simplistic (and weak) MS-Notepad text editor.

vim is more colorful and graphics-oriented than is vi, and it is also available for U/L based systems. Load vim on U/L at your own peril, because if you get addicted to vim then you will find yourself disappointed when working on a vi-only system (most of them).

Read or download this text file of vi commands (all work OK with vim, too).

A long, long time ago, the UNIX system for which I was Administrator had only the ed and vi editors available. The User production application that I wrote (a combination of shell and SQL scripts) occasionally required the User to input a short, descriptive paragraph that would be stored in the database (Informix). Since neither of the native editors were User-friendly, I had to create my own, simple text editor. I called it shed which was yettanuther Bourne Shell (sh) script. shed should still work OK (your results may vary) if you download it, and make it executable. Here it is: shed_1.4.sh

BACK TO TOP

---

AWK & SED TIPS, TRICKS, and EXAMPLES

awk is a powerful programming language available on all UNIX systems. Since awk is an interpreted language (i.e., uncompiled), an awk program written on HP/UX will run without modification on AIX, SOLARIS, LINUX, or any other UNIX-like system (provided that said system has the compiled awk-interpreter installed).

In addition to conditional branching (e.g., if/then/else) and loop processing, awk offers the power of multi-dimensional array processing plus a wealth of built-in string and math functions. It can read from and write to multiple files, and execute system commands. Using variable contents and system utilities, awk can write - on the fly - shell and SQL scripts to control subsequent data processing. The possibilities are endless. You have to understand at least some basic awk commands to be conversant with UNIX/Linux.

The awk files and examples below show how a programmer can integrate an interactive .ksh (or .sh) shell script and an embedded awk program to produce formatted output (e.g., a report) from an input data file. Much use is made of variables and functions to provide real-time values during processing. The context of these examples is working scripts ("production") for a wireless telephone company. Line-numberd versions of the .ksh example script are included to facilitate discussion as suggested at the bottom of this awk overview file: awk_fyi.txt.

Be sure you visit The AWK Programming Language, Second Edition

Read or download (right-click on link) these other text files:

DESCRIPTION	FILE NAME		DESCRIPTION	FILE NAME		DESCRIPTION	FILE NAME
awk example #1:	ckurf.ksh		awk example #2:	ckurf_num.txt		awk OVERVIEW:	awk_fyi.txt
awk example #3:	reseq.ksh		awk example #4:	reseq_num.txt		awk TUTORIAL #1:	tutorialspoint.com
awk example #5:	see_edf.ksh		awk example #6:	see_edf_num.txt		awk TUTORIAL #2:	grymoire.com
awk example #7:	copy_tables.ksh		awk example #8:	see_ct_dat.ksh		awk report example:	see_ct_dat.out.txt
awk example #9 (Bourne):	pmpopp.sh.txt		#9 calls this awk file:	pop_all.awk.txt"		#9 awk and sh fyi:	readme.awk.txt

The sed files and other examples below usually show how a programmer can integrate an interactive .ksh shell script, an embedded awk program, and an embedded sed call to produce formatted output (e.g., a report) from an input data file.

Read or download (right-click on link) these other text files, too:

DESCRIPTION	FILE NAME		DESCRIPTION	FILE NAME		DESCRIPTION	FILE NAME
sed example #1:	procit.ksh
awk array fyi	https://datafix.com.au

BACK TO TOP

---

SQL TIPS, TRICKS, and EXAMPLES

Probably all industrial-strength relational databases (e.g., Oracle) are standardized on the Structured Query Language (SQL for short, pronounced sequel). The examples here are all character (non-gui) based that Users ran on Wyse 50 serial terminals (no Windows). Nothing here is point-&-click, click-&-drag, or any other type of effete foofaw. This all comes from back in the day when real men were real programmers who wrote real code by hand (not by mouse). You might say we lived to write printf() statements!

Read or download (right-click on link) these text files:

DESCRIPTION	FILE NAME		DESCRIPTION	FILE NAME		DESCRIPTION	FILE NAME
sh & sql example #1:	batA96.sh		#1 calls this awk file:	batA96.awk		All-in-one: ksh, sql, & awk:	anc_order.ksh
sh & sql example #2:	pmpopp.sh		#2 calls this awk file:	pop_all.awk		Some Stored Procedures "readme":	README.txt
Stored Procedures ex #1:	rcup_spl.sql		Stored Procedures ex #2:	ven_hist.sql

I did a lot of work using Informix Standard Engine ("SE"), including its Stored Procedures language. My roots with Informix (originally named Relational Database Systems) go deep, beginning with when I met founders Roger Sippl and Laura King at a Usenix convention in 1985. At that time, they were just introducing their SQL-compliant database. All Informix products were eventually purchased and are now (or rather, were) marketed by IBM (eval) copies are available free). At one time, Informix SE and ESQLC were available for Linux. Highly recommended by almost everyone (but I've never used it - yet) is MySQL. It is available for free download for almost every type of system, including even [ugh] Windows. Give it a whirl.

BACK TO TOP

"Raise your glass to the hard working people.
Let's drink to the uncounted heads
Let's think of the wavering millions
Who want leaders but get gamblers instead.

Spare a thought for the stay-at-home voter,
His empty eyes gaze at strange beauty shows,
And a parade of gray-suited grafters
...A choice of cancer or polio."

From "The Salt of the Earth"
C. 1969 Mick Jagger and Keith Richard

---

Return to Dan & Sheryl's Home Page