Jawk - Compiler Module

Jawk accepts the -z argument to compile the script and the -Z to compile and execute the script. You may provide the -o argument to specify the output class name. By default, the name used is "AwkScript". Also by default, Jawk assumes no package name and writes the class file into the present working directory. To override this behavior, use the -d argument. This results in the assignment of a package name to the script, along with creation of the script class in the package-named directory.

For the compiler to work, you must have the Apache Byte Code Engineering Library (BCEL) JAR file in your classpath. Click here and follow the links to download the JAR file. Remember that the JAR file is imbedded within the ZIP file you download from the site. Then, augment your classpath to contain the JAR file. Without the BCEL JAR file in your path, you will get a NoClassDefFoundError because the Awk Compiler implementation is looking for the BCEL support classes and services, which it cannot find.

The script class relies on the Jawk runtime environment for execution. As a result, you must include jrt.jar in the classpath when executing the script class. (Note: jawk.jar includes classes contained in jrt.jar. Therefore, either jar file will do. However, jrt.jar is smaller, and thus, a more favorable choice.)

Compiler Usage

For Mac/Unix:
java -cp "jawk.jar:bcel.jar:$CLASSPATH" org.jawk.Awk -Z -f script.awk

For Windows:
java -cp "jawk.jar;bcel.jar;%classpath%" org.jawk.Awk -Z -f script.awk

For Mac/Unix:
java -cp "jrt.jar:$CLASSPATH" AwkScript

For Windows:
java -cp "jrt.jar;%classpath%" AwkScript

Extensions, such as _sleep, _dump, _INTEGER, etc are supported. Keep in mind, however, that arguments which enable most extensions must be provided at compile time. For example, to compile scripts which contain the _dump and _sleep keywords, use:

java -cp "jawk.jar;bcel.jar;%classpath%" org.jawk.Awk -z -x script.awk

java -cp "jrt.jar;%classpath%" AwkScript -h

Note: the -t extension is the only runtime, not compile-time, extension. Therefore, to use sorted key maps for associative arrays, the -t argument is needed along with the -Z argument, or when executing AwkScript itself.

Command Line Arguments

• [-v variable-name]* - One or more initial variable initializations.
• -F fs_val - Initialization of the FS variable
• -t - Employ key sorting in associative arrays.
• -h or -? - A help (usage) screen.
• [name=value | input_filename]+ - One or more parameters passed to ARGV/ARGC.

Architecture

import org.jawk.jrt.*;
import org.jawk.util.AwkParameters;

import java.util.*;
import java.util.regex.*;
import java.io.*;

public class AwkScript implements VariableManager {

	// use this field as the third argument to the AwkParameters
	// constructor when executing ScriptMain directly

	public static final String EXTENSION_DESCRIPTION = extension-description-string;

	private static final Integer ZERO = new Integer(0);
	private static final Integer ONE = new Integer(1);
	private static final Integer MINUS_ONE = new Integer(-1);

	public static void main(String args[]) {
		AwkScript as = new AwkScript();
		// this is why org.jawk.util.AwkParameters is in jrt.jar ...
		AwkParameters ap = new AwkParameters(AwkScript.class, args, EXTENSION_DESCRIPTION);
		// to send the error code back to the calling process
		System.exit(as.ScriptMain(ap));
	}

	// to satisfy the VariableManager interface

	// Note: field names here correspond to a global_N field
	// which are assigned upon compilation of the script.

	public final Object getARGC() { if (argc_field == null) return ""; else return argc_field; }
	public final Object getCONVFMT() { if (convfmt_field == null) return ""; else return convfmt_field; }
	public final Object getFS() { if (fs_field == null) return ""; else return fs_field; }
	public final Object getARGV() { return argv_field; }
	public final Object getOFS() { if (ofs_field == null) return ""; else return ofs_field; }
	public final Object getRS() { if (rs_field == null) return ""; else return rs_field; }
	public final void setFILENAME(String arg) { filename_field = arg; }
	public final void setNF(String arg) { nf_field = arg; }

	private final Object getNR() { if (nr_field == null) return ""; else return nr_field; }
	private final Object getFNR() { if (fnr_field == null) return ""; else return fnr_field; }

	public final void incNR() { nr_field = (int) JRT.toDouble(JRT.inc(getNR())); }
	public final void incFNR() { fnr_field = (int) JRT.toDouble(JRT.inc(getFNR())); }
	public final void resetFNR() { fnr_field = ZERO; }

	public final void assignField(String name, Object value) {
		if (name.equals("scalar1")) scalar1_field = value;
		else if (name.equals("scalar2")) scalar2_field = value;
		else if (name.equals("scalar3")) scalar3_field = value;
		...
		else if (name.equals("scalarN")) scalarN_field = value;
		else if (name.equals("funcName1")) throw an exception;
		else if (name.equals("funcName2")) throw an exception;
		...
		else if (name.equals("funcNameX")) throw an exception;
		else if (name.equals("assocArrayName1")) throw an exception;
		else if (name.equals("assocArrayName2")) throw an exception;
		...
		else if (name.equals("assocArrayNameM")) throw an exception;
	}

	private JRT input_runtime;
	private HashMap regexps;
	private HashMap pattern_pairs;
	private int exit_code;

	private int oldseed;

	private Random random_number_generator;

	// global_N refers to all the global variables,
	// those which are defined by default
	// (i.e., ARGC, ARGV, ENVIRON, NF, etc.)
	// and vars declared by the script.
	// The _SET_NUM_GLOBALS_ opcode allocates
	// these fields.

	private Object global_0;
	private Object global_1;
	private Object global_2;
	...
	private Object global_N;

	// Call this method to invoke the Jawk script.
	// Refer to the static main method implementation
	// and Javadocs on how to build the AwkParameters.
	// Use the public static String EXTENSION_DESCRIPTION
	// field (within AwkScript) as the third parameter
	// to the AwkParameters constructor to ensure proper
	// extension description in the usage statement.

	public final int ScriptMain(AwkParameters awk_parameters) {

		// local variables

		double dregister;
		StringBuffer sb = new StringBuffer();

		// Field Allocation
		// ----------------
		// Could have be done in the class constructor,
		// but placed here to ensure proper repeat initialization
		// if repeat execution is required
		// within the same JVM instance.  Because
		// if these were within the class constructor, each of these
		// data structures / int values would have to be
		// reinitialized in some way anyway.

		input_runtime = JRT(this);	// this = VariableManager
		regexps = new HashMap();
		pattern_pairs = new HashMap();
		oldseed = 0;
		random_number_generator = new Random(null);
		exit_code = 0;

		// script execution

		try {
			///
			/// Compiled BEGIN and input rule blocks code here.
			/// (EndException is thrown when exit() is encountered.)
			///
		} catch (EndException ee) {
			// do nothing
		}

		try {
			runEndBlocks();
		} catch (EndException ee) {
			// do nothing
		}

		return exit_code;
	}

	public void runEndBlocks() {
		double dregister;
		StringBuffer sb = new StringBuffer();

		///
		/// Compiled END blocks code here.
		/// (EndException is thrown when exit() is encountered.)
		///
	}

	// One of these exists for every function definition.
	// Arguments are reversed from its Jawk source.
	public Object FUNC_function_name(Object oN, Object oN-1, ... Object o2, Object o1) {
		Object _return_value_ = null;
		StringBuffer sb = new StringBuffer();
		double dregister = 0.0;

		///
		/// Compiled function function_name code here.
		/// (A return() sets the _return_value_ and falls out of this
		///  function code block.)
		/// (EndException is thrown when exit() is encountered.)
		///

		return _return_value_;
	}

	// The following is created for every optarg version of the function
	// call that exists within the script for this function_name.
	// X > 0 && N > X
	public final Object FUNC_function_name(Object oX, Object oX-1, ... Object o2, Object o1) {
		return FUNC_function_name(null, null, ..., null, oX, oX-1, ..., o2, o1);
	}
}

Here are several points that should be considered:

• The org.jawk.jrt.JRT class contains helper functions which deals with AWK file / command IO management and most of the built-in function (i.e., rand()) functionality. Most of this (if not all) could have been inlined into the compiled result. However, this would have crippled maintainability, and thus, is kept within a separate Java service class. jrt.jar contains this and other service classes refered to by AwkScript.
• The extension-description-string refers to the extension description provided by AwkParameters upon compilation (not at runtime), constructed by the extensionDescription() method in org.jawk.util.AwkParameters.
• Implementing the VariableManager is necessary to provide variable access to the JRT.
• Class instance fields, such as input_runtime, regexps, exit_code, and oldseed, could have (should have?) been allocated/assigned within the class constructor. However, allocation/assignment is done within ScriptMain() to ensure proper reinitialization of these fields in the event of repeat execution within the same class instance.
• Parameters to FUNC_ function calls are reversed. This is because Jawk uses recursive decent parsing.
• AwkScript is not declared final. Therefore, it can be subclassed. And, all FUNC_ (full-arg version) definitions can be overridden. Note, however, that all FUNC_ parameters are of type Object. Therefore, the overriding class must police itself (during runtime) to ensure correct typecasting.
  Note: Associative arrays are of type org.jawk.jrt.AssocArray. It does not subclass a Collections class, nor does it implement a Collections interface. Refer to the Javadocs for its API. (It does, however, implement the java.lang.Comparator interface.)
• AwkScript is not thread-safe. Therefore, it is not recommended that multiple threads access the same AwkScript instance.

Benchmarks

For the following AWK script:

BEGIN { print fib(30) }
function fib(i) {
	if (i<2) return i
	return fib(i-1) + fib(i-2)
}

The bottom line is that, on average, you'll see a noticeable increase in speed in the compiled version. If the script spends most of its time doing computations and calling AWK functions, then you'll see large gains in execution efficiency. If the script spends most of its time doing IO, then you wont see much gain.

Compilation vs. Interpretation

• Resultant scripts are difficult to revert back to AWK source, thus providing an improved level of obfuscation (than just using the -c argument).
• The compiled result is optimized by JIT compilation rather than the Jawk interpreter running your script. Thus, enhancements and improvements to JIT compilation has more of a direct impact to efficiency improvements on your script over its interpreted counterpart.
• AWK scripts can now be used with tools and services that operate on Java bytecode, such as profiling and debugging tools.
• AWK scripts can now take advantage of a commercial-grade, highly optimized, fully configurable, cutting edge garbage collector. This probably has a profound impact on memory management and execution efficiency for scripts that execute over a long period of time.

Changes to Jawk

Package Reorganization

Intermediate Code Changes

PositionForCompilation

Better Runtime Error Reporting

Interpreter Stack Leaks

Other Bug Fixes

• Incorrect right-associativity for addition, subtraction, multiplication, division, etc. It should be left-associative.
• Empty function bodies cause assertion errors.
• Passing in associative arrays as name=value parameters (either via -v or ARGV) caused assertion errors. Passing in function names in this manner was ignored.
• The reader responsible for partitioning input (at script runtime) with a regular expression (via the RS variable) will not always apply greedy input matching to the RS regexp, depending on where the regexp input exists on the input buffer. This resulted in extra records to be produced. However, greedy regex consumption assurance is not always desired (i.e., interactive input). Therefore, to enable, enable with -Djawk.forceGreedyRS=true, or tailor the RS regex to not accept input in an ambiguous manner. Greedy RS consumption is turned off by default.
• Many of the initial values and behavior of builtin global variables did not follow spec.

• a = 09; causes a NumberFormatException in its attempt to treat 09 as an octal number.
• BEGIN{print}END{print} causes a ParserException (i.e., the erroneous requirement of a statement terminator [;] between BEGIN/END/input-rules).