# Wawk

> AWK Extended on WebAssembly -- by Ailur Labs.

Wawk extends AWK, the classic pattern-scanning and text-processing language created at Bell Labs, on WebAssembly with plugin extensibility. It runs identically across browsers, servers, edge runtimes, and AI agents.

Wawk is an open-source project by [Ailur Labs](https://ailur.ai), dual-licensed under MIT and Apache 2.0.

## What Wawk Does

- **POSIX-compliant AWK interpreter**: Implements the full AWK language specification with pattern-action rules, associative arrays, regular expressions, and built-in functions
- **WebAssembly-native**: Compiles to .wasm modules (wasm32-wasip1, wasm32-unknown-unknown) -- runs in browsers, servers, and edge runtimes with zero OS dependencies
- **Plugin extensibility**: Custom functions via WIT Component Model plugins -- build once as a .wasm component, loaded via `--plugin` flag in any host
- **PropertyTree-native access**: Automatically parses structured data (JSON, XML, YAML, CSV) input, enabling field access via `$.field` notation (e.g., `$.name`, `$.address.city`) and array positional access via `$1`, `$2`, `$3`
- **Cross-platform consistency**: Identical output for the same input across all environments -- browser, server, edge, AI agents
- **Security hardening**: Output size limits (16MB), record size limits, regex safety (NFA-based, no ReDoS), execution timeouts, and configurable resource limits

## Architecture

Wawk consists of three components sharing the same engine code:

- **wawk-core**: Shared AWK engine written in Rust -- the core interpreter used by all targets
- **wawk-wasi**: Pure WASI command-line module -- runs under wasmtime or any WASI-compatible runtime
- **wawk-bindgen**: JavaScript-importable library via wasm-pack -- runs in browsers and Node.js

All three produce identical output for the same input, ensuring consistency across deployment targets.

## PropertyTree-Native Features

Wawk extends AWK with a universal PropertyTree data model for structured data (JSON, XML, YAML, CSV):

```awk
# Auto-detect JSON input -- no import needed
# Access object fields with $.field notation
{ print $.name, $.age }

# Nested JSON access with dot chaining
{ print $.name, $.address.city, $.address.zip }

# Array positional access with $1, $2, $3
{ print $1, $2, $3 }

# JSON object literal output
{ print {name: $.name, city: $.address.city} }

# JSON array literal output
BEGIN { print [1, 2, 3] }

# typeof() works on JSON fields
{ print typeof($.name), typeof($.age) }

# NF returns array/object length
{ print NF }
```

Structured data records are auto-detected on input (e.g., JSON lines starting with `{` or `[`). Object fields are accessible via `$.field` syntax. Array elements are accessible via `$1`, `$2`, `$3` positional syntax. The `print` statement auto-serializes PropertyTree values back to the output format.

## Plugin System

Plugins extend AWK with custom functions using the WIT Component Model. Each plugin exports two interfaces: `external-functions` (custom AWK-callable functions) and `format-handler` (format detection/parsing). The plugin lifecycle follows: `__init__` (initialize) -> `__meta__` (metadata: name, version, dependencies) -> function dispatch.

```awk
# Load plugin via --plugin flag
wawk --plugin ./my_plugin.wasm 'BEGIN { print greet("World") }'

# Plugin functions work like built-in functions
{ print transform($0) }
```

Plugins can be written in any language that compiles to WebAssembly:

- **Rust**: Using `wit-bindgen` with `cargo build --target wasm32-unknown-unknown`
- **C**: Using `wit-bindgen` C code generation with `clang --target=wasm32`
- **Go**: Using `wit-bindgen` Go bindings with TinyGo (`tinygo build -target=wasm-unknown`)
- **TypeScript**: Using AssemblyScript with `npx asc plugin.ts --target release`

All languages produce compatible .wasm components from the same WIT definition. The WIT interface is:

```wit
package wawk:plugins;

interface external-functions {
    call: func(name: string, args: list<string>) -> option<string>;
}

interface format-handler {
    detect:   func(input: string) -> string;
    parse:    func(input: string) -> string;
    serialize: func(tree-json: string) -> string;
}

world wawk-plugin {
    export external-functions;
    export format-handler;
}
```

## Optimizations

Key optimizations: zero-allocation hot paths, format auto-detection skip, FxHashMap for O(1) array access, regex compilation cache with LRU eviction, reusable buffers for print output, fast integer formatting via itoa, fast float formatting via ryu, single-pass whitespace splitting, leaf expression depth check bypass.

## Security Features

- **Output size limit**: 64MB max with amortized checks every 1024 records
- **Record size limit**: 64MB max to prevent memory exhaustion
- **Regex safety**: Pattern length limit, cache cap 512, NFA-based engine (no ReDoS vulnerability)
- **Execution timeout**: Configurable timeout with amortized checks
- **Printf safety**: Width/precision caps at 10000
- **Array size limit**: Max 1,000,000 entries
- **Object key limit**: Max 10,000 keys per PropertyTree
- **Nesting depth limit**: Max 64 levels for structured data
- **Field limit**: Max 100,000 fields per record
- **Open file limit**: Max 256 simultaneous targets
- **Audit log cap**: Max 1,024 entries
- **Loop iteration limit**: 100M iterations
- **Recursion limit**: Max 256 nested function calls
- **Sandboxed execution**: Wasm runtime isolation -- no network, filesystem, or system command access

## How To Use Wawk

### Install and Build

```bash
# Install Wasmtime (for wawk-wasi)
curl https://wasmtime.dev/install.sh -sSf | bash

# Install wasm-pack (for wawk-bindgen)
cargo install wasm-pack

# Build all Wasm targets
make build-wasm
```

### Run from Command Line

```bash
# Sum a column
printf '{ sum += $1 } END { print sum }\n1\n2\n3\n' | wasmtime wawk-wasi.wasm

# Filter lines matching a pattern
printf '/error/ { print $0 }\ninfo: ok\nerror: disk full\n' | wasmtime wawk-wasi.wasm

# Process JSON input
printf '{"name":"Alice","age":30}\n' | wasmtime wawk-wasi.wasm '{ print $.name, $.age }'
```

### Use in Node.js

```javascript
import init, { exec_awk } from './wawk/wawk.js';

await init();
const result = exec_awk('{ print $.name }', '{"name":"Alice"}');
console.log(result); // "Alice"
```

### Use in Browser

```html
<script type="module">
import init, { exec_awk, set_plugin_dispatch } from './wawk/wawk.js';

await init();

// Optional: register plugin dispatch for WIT plugin support
set_plugin_dispatch((name, args) => {
  if (name === 'greet') return `Hello, ${args[0]}!`;
  return null; // unknown function
});

const result = exec_awk('{ print $.name }', '{"name":"Alice"}');
console.log(result); // "Alice"
</script>
```

### Build a Plugin (Rust)

```rust
// src/lib.rs
wit_bindgen::generate!({
    world: "wawk-plugin",
    path: "wit",
});

struct Component;

impl exports::wawk::plugins::external_functions::Guest for Component {
    fn call(name: String, args: Vec<String>) -> Option<String> {
        match name.as_str() {
            "__init__" => Some("ok".into()),
            "__meta__" => Some(r#"{"name":"my-plugin","version":"0.1.0"}"#.into()),
            "greet" => {
                let who = args.first().map(|s| s.as_str()).unwrap_or("World");
                Some(format!("Hello, {}!", who))
            }
            _ => None,
        }
    }
}

// Format handler: required by WIT world (stub if not a format plugin)
impl exports::wawk::plugins::format_handler::Guest for Component {
    fn detect(_input: String) -> String { String::new() }
    fn parse(_input: String) -> String { r#"{"error":"not a format plugin"}"#.into() }
    fn serialize(_tree: String) -> String { r#"{"error":"not a format plugin"}"#.into() }
}

export!(Component);
```

```bash
cargo build --target wasm32-unknown-unknown --release
wawk --plugin ./target/wasm32-unknown-unknown/release/my_plugin.wasm 'BEGIN { print greet("World") }'
```

## Open-Source Repositories

- [wawk-rs](https://github.com/ailurlabs/wawk-rs) -- Core AWK engine (wawk-core, wawk-wasi, wawk-bindgen) in Rust
- [wawk-hello](https://github.com/ailurlabs/wawk-hello) -- Example plugin demonstrating the WIT plugin architecture with greet() and greet_lang() functions
- [wawk-website](https://github.com/ailurlabs/wawk-website) -- Official website with interactive reference and playground

## Key Concepts

- **AWK**: A pattern-scanning and text-processing language created at Bell Labs in 1977, part of the POSIX standard, shipped with every Unix system since the late 1970s
- **WebAssembly (WASM)**: A binary instruction format for a stack-based virtual machine, enabling near-native speed in browsers, servers, and edge environments
- **WIT Component Model**: Plugin system using WebAssembly Interface Types for language-agnostic extensibility
- **wawk-core**: Shared AWK engine written in Rust, used by wawk-wasi (CLI) and wawk-bindgen (JS library)
- **wawk-wasi**: Pure WASI command-line module, runs under wasmtime or any WASI-compatible runtime
- **wawk-bindgen**: JavaScript-importable library via wasm-pack, runs in browsers and Node.js
- **PropertyTree-native access**: Wawk extension that auto-parses structured data (JSON, XML, YAML, CSV) for `$.field` and `$1, $2, $3` positional access
- **WIT (WebAssembly Interface Types)**: Standard for defining interfaces between WebAssembly modules and their hosts

## Who It's For

- **Developers** needing portable, sandboxed text processing without OS dependencies
- **Edge computing** and serverless deployments requiring fast startup and low memory
- **AI agent workflows** requiring data transformation in sandboxed environments
- **Browser-based applications** needing client-side text processing
- **Teams** wanting identical AWK output across all platforms (browser, server, edge)
- **Structured data processing** (JSON, XML, YAML, CSV) with AWK's concise pattern-action syntax
- **Plugin developers** wanting to extend AWK with custom functions in Rust, C, Go, or TypeScript

## Comparison with Other AWK Implementations

| Feature | Wawk | gawk | mawk |
|---------|------|------|------|
| POSIX compliant | Yes | Yes | Yes |
| WebAssembly | Yes | No | No |
| Browser execution | Yes | No | No |
| Edge runtime | Yes | No | No |
| Plugin system | WIT Component Model | C extensions | None |
| PropertyTree-native | Yes | No | No |
| Cross-platform | Identical output | Varies by OS | Varies by OS |
| OS dependencies | None | libc, libm | libc |
| Binary size | Compact Wasm | ~2.5MB | ~200KB |

## Resources

- Website: [https://wawk.ailur.ai](https://wawk.ailur.ai)
- Language Specification & Tutorial: [https://wawk.ailur.ai/specification](https://wawk.ailur.ai/specification) -- Complete POSIX AWK reference plus Wawk extensions: patterns, fields, variables, regex, BEGIN/END, arrays, built-in functions, user functions, control flow, output/formatting, PropertyTree-native records, nested JSON, @plugin directive, plugin system, I/O, security features, performance optimizations
- Plugin Development Guide: [https://wawk.ailur.ai/plugin](https://wawk.ailur.ai/plugin) -- Step-by-step guide for Rust, C, Go, and TypeScript
- Interactive Playground: [https://wawk.ailur.ai/playground](https://wawk.ailur.ai/playground) -- Run AWK and Wawk scripts in your browser
- Parent Company: [https://ailur.ai](https://ailur.ai)
- GitHub Organization: [https://github.com/ailurlabs](https://github.com/ailurlabs)
- Contact: [https://ailur.ai/#contact](https://ailur.ai/#contact)