What I Kept From Stata

TabDat-Explore is my attempt to preserve a concise statistical command model while rebuilding the data and execution assumptions underneath it.

Routine Analysis Can Carry Too Much Interface Overhead

Imagine you need to explore a 10G Parquet file containing longitudinal clinical trial records across twelve hospital sites. Before fitting multi-level survival curves or running causal inference pipelines, you need to answer a few usual questions about the data:

• How many patient records and variables are present?
• What proportion of laboratory values or follow-up days are missing?
• What are the mean, median, and interquartile ranges for continuous variables like age, BMI, and total inpatient cost?
• How are patients distributed across treatment arms and clinical sites?
• Does average inpatient cost differ between treatment arms within each site?
• When we adjust for age and baseline BMI in a linear regression, does the treatment coefficient change noticeably?

None of these questions involve complex mathematics. They represent the standard preliminary inspection that every quantitative researcher, epidemiologist, or data analyst performs before doing serious work.

Yet the friction of answering them often comes from the representational machinery of the tools we use. In a Python notebook, answering these questions typically requires importing multiple libraries, configuring display options, calling DataFrame methods, managing method-chain syntax, and handling index resets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import polars as pl
import statsmodels.formula.api as smf

df = pl.read_parquet("cohort.parquet")

print(df.shape)
print(df.select(["age", "bmi", "cost"]).describe())
print(df.group_by(["site", "treatment"]).agg(pl.col("cost").mean()))

model_data = df.select(["cost", "treatment", "age", "bmi"]).to_pandas()
fit = smf.ols("cost ~ treatment + age + bmi", data=model_data).fit()
print(fit.summary())

If we instead turn to an embedded relational engine using SQL, the grouped query becomes explicit and scalable, but calculating basic descriptive statistics across several columns quickly turns into repetitive aggregate clauses:

1
2
3
4
5
6
7
8
9

SELECT 
    site,
    treatment,
    COUNT(*) AS n,
    AVG(cost) AS mean_cost,
    STDDEV(cost) AS sd_cost
FROM 'cohort.parquet'
GROUP BY site, treatment
ORDER BY site, treatment;

Neither approach is flawed. Python and R provide general-purpose programming environments that can express virtually any algorithm. SQL provides a standardized declarative language for relational operations. But for routine exploratory statistics, both require writing considerable mechanical code around simple intentions.

I spent years using an older statistical environment where these same operations required four brief lines in an interactive console:

1
2
3
4

describe
summarize age bmi cost
by site treatment: summarize cost
regress cost treatment age bmi

This contrast is about representational overhead: how much general-purpose syntax, environment setup, and state management must surround a domain-specific operation.

In my earlier essay, Why I Design CLI-First Software, I described how command models provide a shared contract for humans, scripts, and automated agents. That essay focused on the general architecture of command-driven systems and mentioned Stata as an early influence on my thinking.

TabDat-Explore takes that observation into statistical computing. It explores what happens when we take the analytical ergonomics of older statistical environments and rebuild the underlying storage, query execution, and semantic contracts for modern tabular data.

Stata Taught Me to Think in Analytical Commands

For readers who have worked primarily in Python, R, or SQL environments, Stata is a long-established statistical package widely used in economics, epidemiology, public health, and social science research.

Much of Stata’s interaction design revolves around a few consistent conventions:

1. An active dataset: Commands operate by default on a single in-memory table loaded into the current session.
2. First-class statistical verbs: Operations like summarize, tabulate, and regress directly express statistical intentions rather than generic array manipulations.
3. Prefix modifiers: Modifiers such as by and if compose with almost any analytical command without altering the core command grammar.
4. Post-estimation workflows: Fitting a model creates an internal estimation state that downstream commands like predict or test query directly.
5. Interactive continuity: Every command typed at the prompt can be copied directly into a script (a .do file) with identical execution behavior.

Consider how these conventions work in practice:

1

summarize age bmi

This command asks the environment to inspect the active dataset, check types, compute observations, means, standard deviations, minima, and maxima for the specified variables, and present them in a standardized tabular format.

Prefixing that operation with a grouping modifier preserves the exact same syntax:

1

by treatment: summarize outcome

The system splits the dataset by treatment arm, computes the summary table within each stratum, and formats the comparisons side by side.

When estimation is required, the verb changes to reflect the statistical model:

1
2
3

regress cost treatment age bmi
predict fitted_cost, xb
predict residuals, residuals

The regress command estimates ordinary least squares coefficients, reports standard errors, t-statistics, p-values, confidence intervals, and variance decompositions, and stores the fitted model in session state. The subsequent predict commands know how to extract linear predictions or residuals from that active estimation result without requiring the user to manually pass model objects between functions.

These commands felt direct because they matched how researchers think about analysis. The interface operated at the level of statistical intentions rather than data-structure mechanics.

An Interface Can Age Differently From Its Substrate

When software is described as “legacy,” the label often bundles several distinct design layers into a single judgment. A system’s user interface, statistical definitions, computational engine, storage format, and integration model are treated as an inseparable package.

In practice, these layers evolve and age at very different speeds.

Decomposing an analytical environment into these five layers clarifies why older tools remain compelling in some dimensions while becoming restrictive in others:

• Interaction model: The vocabulary of verbs (summarize, regress, tabulate), prefix modifiers (by, if), and interactive-to-script continuity. This layer addresses human cognitive bandwidth and can remain effective across decades.
• Statistical semantics: The behavioral rules governing estimation samples, missing-value propagation, post-estimation matrices, and random-seed reproducibility. These rules require formal precision and consistency.
• Execution engine: The computational substrate responsible for scanning columns, aggregating values, and solving linear systems. In-memory matrix loops from thirty years ago cannot match contemporary vectorized query engines.
• Storage model: The data formats supported for reading and writing. Proprietary binary formats tied to single software licenses do not fit modern data pipelines built around open standards like Apache Parquet.
• Integration and distribution: How the software is deployed, called headlessly, integrated into automated workflows, and consumed by other programs through structured outputs.

Older statistical software frequently combined brilliant interaction ergonomics with storage and execution assumptions that reflected the hardware constraints of the 1980s and 1990s: entire datasets had to fit into system RAM, data lived in proprietary row-oriented files, and the software functioned as a self-contained silo.

Modern data engineering inverted those technical assumptions. We now have open columnar formats, out-of-core vectorized execution, and ubiquitous scripting environments. Yet modern stacks often discarded the concise analytical command model along with the old substrate.

Selective modernization means recognizing that we can replace the bottom three layers (storage, execution, and integration) while preserving and refining the top two (interaction ergonomics and explicit semantics).

What TabDat Tries to Preserve

TabDat-Explore is an experimental analytical tool designed around four specific ergonomic principles inherited from traditional statistical environments.

1. Statistical Verbs as First-Class Language Constructs

In general-purpose programming languages, computing basic summary statistics often requires combining several lower-level functions. TabDat treats common statistical operations as primary language verbs:

1
2
3
4

tabdat> describe
tabdat> summarize age bmi cost
tabdat> tabulate treatment site
tabdat> regress cost treatment age bmi

The user specifies the analytical operation directly. The command names the intent, the arguments name the variables, and optional flags refine the behavior. This design keeps the user’s mental model focused on statistical exploration rather than syntax construction.

2. A Cognitively Cheap Active Dataset

Managing multiple DataFrame variables (df1, df_clean, df_merged, temp_df) in a notebook introduces cognitive tracking overhead. For exploratory analysis, maintaining one active working dataset in session memory reduces friction:

1
2

tabdat> use cohort.parquet
tabdat> summarize age bmi if site == 3

Commands assume the active dataset unless explicitly told otherwise. When an analysis requires working with multiple tables, TabDat allows loading named datasets or querying them through relational escape hatches. But the default interactive posture remains lightweight: one table in focus at a time.

3. Direct Continuity Between Interactive Commands and Scripts

Every line typed interactively in TabDat can be saved directly into a .td script without modification:

1
2
3
4
5
6

# cohort_baseline.td
use cohort.parquet
summarize age bmi cost
by treatment: summarize cost
regress cost treatment age bmi
predict fitted_cost, xb

Running this script from the terminal reproduces the entire interactive session:

1

tabdat run cohort_baseline.td

There is no mental translation between an “exploratory mode” and a “scripting mode.” The interactive prompt and the batch execution runner interpret the exact same grammar.

4. Terminal Ergonomics Designed for Discovery

A command language only feels responsive when the terminal environment actively supports discovery. TabDat includes interactive shell capabilities:

• Context-aware autocompletion for commands, options, file paths, and active dataset column names.
• Inline syntax highlighting that visually distinguishes verbs, variable names, strings, and numeric constants.
• Persistent cross-session command history.
• Immediate tabular rendering with column alignment and sensible precision defaults.

These ergonomics ensure that exploring an unfamiliar dataset from the command line does not require memorizing exact column spellings or looking up documentation for routine flags.

What TabDat Replaces

While TabDat borrows interaction patterns from older tools, its technical substrate is built entirely on modern data infrastructure.

Parquet-Native Storage

Traditional statistical software relied on proprietary binary formats (such as .dta or .sas7bdat) that required dedicated import/export steps to interface with broader data pipelines.

TabDat operates directly on Apache Parquet files. Parquet provides columnar compression, dictionary encoding, and partition awareness. It is supported across Python, R, Julia, Rust, Go, and major cloud query engines. A dataset produced by an upstream Spark or dbt pipeline can be opened immediately in TabDat without conversion, and datasets saved by TabDat remain readable by any modern tool.

DuckDB as the Embedded Analytical Engine

Rather than implementing a custom in-memory query engine, TabDat uses DuckDB as its execution backbone. DuckDB provides:

• Fast vectorized query execution against local files.
• Projection and filter pushdown, reading only the columns and row groups requested by a command.
• Native out-of-core processing, enabling analysis of datasets that exceed physical RAM.
• A battle-tested relational execution planner.

When a user executes summarize age bmi if site == 4, TabDat translates the filtering and aggregation logic into optimized expressions executed directly by DuckDB over the underlying Parquet file.

Eager vs. Lazy Execution Boundaries

Traditional tools assumed that opening a dataset meant reading all rows and columns into memory immediately. TabDat supports explicit lazy evaluation:

1
2
3

tabdat> use large_registry.parquet, lazy
tabdat> describe
tabdat> summarize age if followup_days > 365

In lazy mode, TabDat inspects Parquet metadata to extract column schemas, data types, and row counts without materializing the dataset into memory. Descriptive commands query the metadata or stream summary aggregates through DuckDB. The full dataset is only materialized when an estimator or transformation strictly requires raw arrays.

SQL Works Better as an Escape Hatch

Because DuckDB includes full SQL support, an obvious architectural question arises: Why build a statistical command language at all when DuckDB already provides SQL?

The answer lies in recognizing that statistical commands and SQL queries express fundamentally different types of user intent.

Consider calculating basic summary statistics for an outcome variable across treatment groups. In SQL, this requires explicitly naming each aggregate function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

SELECT 
    treatment,
    COUNT(outcome) AS n_obs,
    AVG(outcome) AS mean,
    STDDEV(outcome) AS std_dev,
    MIN(outcome) AS min_val,
    MAX(outcome) AS max_val
FROM cohort
GROUP BY treatment
ORDER BY treatment;

In TabDat, the identical statistical intention is expressed as:

1

tabdat> by treatment: summarize outcome

The statistical verb encapsulates standard distributional metrics, missing-value counts, and formatting conventions that would require dozens of lines of repetitive SQL.

However, the reverse is also true. For complex relational transformations (multi-table joins, window functions, conditional reshaping, and CTEs), SQL is vastly superior to any specialized statistical syntax. Attempting to expand a statistical DSL to handle arbitrary relational algebra inevitably results in an awkward, incomplete dialect of SQL.

TabDat resolves this boundary by providing an explicit SQL escape hatch:

1

tabdat> sql SELECT site, AVG(cost) as mean_cost FROM active GROUP BY site

The active dataset is exposed to DuckDB as a virtual table named active. The user can execute arbitrary SQL queries against it, create new tables, or assign the output of a SQL query back to the active session state.

Statistical commands handle the descriptive and modeling tasks where their ergonomics shine; SQL handles complex relational manipulation.

One Language, Several Ways to Call It

A well-designed command language should not be locked into an interactive terminal. The same command model that powers human exploration can serve scripts, batch pipelines, and automated processes.

TabDat provides three primary modes of interaction over a single semantic core:

1. Interactive Terminal

1

tabdat> summarize age bmi

Designed for fast feedback, rich terminal formatting, and exploratory discovery.

2. Script Execution

1

tabdat run analysis.td

Designed for repeatable batch execution in research pipelines or CI workflows.

3. Machine-Readable Noninteractive Invocation

1

tabdat --json -c "use cohort.parquet" -c "summarize age bmi"

When invoked with --json, TabDat produces structured JSON output on standard output while routing diagnostic logs and progress to standard error:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

{
  "status": "success",
  "command": "summarize",
  "variables": ["age", "bmi"],
  "results": {
    "age": {
      "obs": 24150,
      "missing": 12,
      "mean": 54.32,
      "std_dev": 11.45,
      "min": 18.0,
      "max": 89.0
    },
    "bmi": {
      "obs": 23980,
      "missing": 182,
      "mean": 27.84,
      "std_dev": 5.12,
      "min": 15.2,
      "max": 52.6
    }
  }
}

This headless JSON interface allows other programs, automated pipelines, or coding agents to execute statistical checks and parse results deterministically without scraping terminal text.

The essential property is that all three modes share the exact same command parser, statistical solvers, and execution semantics. The presentation layer changes, but the analytical contract remains identical.

The Hard Problem Eventually Became Semantics

Early in TabDat’s development, adding new commands felt straightforward. Implementing a parser rule for a new statistical verb and mapping it to a library function could be done in an afternoon. Within a few months, the repository had accumulated support for linear models, generalized linear models, fixed-effects estimators, instrumental variables, and survival models.

However, as the command surface expanded, it became clear that defining syntax was the easy part of the problem. The difficult challenge was defining and maintaining statistical semantics.

Consider what a seemingly simple command like regress cost treatment age bmi actually entails when built as a dependable analytical system:

• Estimation sample construction: How are rows with missing values across the dependent or independent variables identified and dropped? Is listwise deletion applied consistently across all estimators?
• Degrees of freedom and covariance corrections: Which small-sample degree-of-freedom corrections are applied by default? When robust standard errors (robust or cluster(site)) are requested, does the covariance calculation match established HC1, HC2, or HC3 formulations?
• Post-estimation state lifecycle: If a regression is fit on a filtered subset (regress cost treatment age bmi if site == 1), how does a subsequent predict fitted_cost command behave for observations outside that subset? Does it predict only for the estimation sample, or for all rows where the independent variables are non-missing?
• Deterministic behavior: Are tie-breaking rules in median and quantile calculations explicit and consistent between in-memory and lazy DuckDB execution paths?
• Error taxonomy: When a model fails to converge or exhibits severe multicollinearity, does the system fail gracefully with structured error codes, or does it emit raw linear-algebra exceptions?

A statistical command is not defined merely by its name and output formatting. It is defined by its semantic contract: the precise mathematical and operational rules governing its behavior.

This realization prompted a deliberate shift in TabDat’s development roadmap. Rather than continuing to add new estimator families, priority shifted toward stabilizing core semantics, building deterministic reference-validation suites against established statistical baselines (comparing outputs against R and Stata reference fixtures), and ensuring strict equivalence between eager and lazy execution modes.

In statistical software, semantic precision and reference validation are foundational product requirements.

What I Borrowed, and From Where

Clear software design requires acknowledging where ideas come from and separating direct inspiration from architectural substrates:

Direct Interaction Lineage: Stata

TabDat borrows Stata’s interaction vocabulary: concise statistical verbs (summarize, tabulate, regress), prefix modifiers (by, if), an active dataset workflow, and post-estimation command chaining. TabDat is Stata-inspired, not Stata-compatible; it borrows these ergonomic ideas without attempting to replicate Stata’s proprietary macro language, licensing model, or legacy idiosyncrasies.

Enabling Execution Substrate: DuckDB

DuckDB provides the core query execution engine, vectorized Parquet reader, out-of-core execution capabilities, and the embedded SQL environment. DuckDB is the technical enabler that allows TabDat’s concise command language to operate efficiently over large datasets.

Ecosystem Libraries

TabDat builds upon the Python scientific and systems ecosystem:

• prompt-toolkit for the interactive terminal interface, autocompletion, and syntax highlighting.
• pyarrow and polars for columnar memory structures and format interoperability.
• statsmodels and scipy for specialized statistical estimation and numerical solvers.

Neighboring and Convergent Tools

Other terminal-native tools have explored related problems from different angles. VisiData provides an interactive spreadsheet-like terminal interface for exploring tabular data via keyboard shortcuts. Miller brings Unix-pipeline ergonomics (sed, awk, cut) to structured tabular formats like CSV and JSON. TabDat occupies a complementary niche: bringing a disciplined statistical modeling and estimation command language to columnar Parquet datasets.

Selective Modernization Is Still a Tradeoff

Preserving older interaction ergonomics within a modernized architecture involves real tradeoffs. TabDat is not a universal solution for all tabular data tasks, and its design reflects explicit choices:

1. A Domain-Specific Language Requires Learning: A command language requires learning a specialized vocabulary. For users already fluent in Python or R DataFrame APIs, learning another set of verbs represents a nonzero investment.
2. General-Purpose Programming Remains Essential: TabDat is designed for exploratory data analysis, data inspection, and standard econometric modeling. For specialized machine learning, deep learning, custom optimization routines, or complex multi-stage data pipelines, general-purpose programming languages like Python or Julia remain indispensable.
3. Active Dataset State Requires Scripting Discipline: An active dataset reduces cognitive friction during quick interactive sessions, but implicit session state can create reproducibility hazards if analyses are not recorded into clean .td scripts.
4. Statistical Validation Is Demanding: Replicating the numerical precision, edge-case handling, and robust covariance estimators of mature statistical software requires extensive validation testing. Supporting a wide statistical surface demands long-term maintenance discipline.
5. Terminal-First Is an Intentional Choice: A command-line interface will not suit every workflow. Visual dashboarding, spatial GIS analysis, and interactive chart brushing naturally benefit from graphical environments.

TabDat does not attempt to replace the general-purpose scientific stack. It is a focused experiment in analytical ergonomics: providing a fast, expressive, and lightweight tool for the initial exploration and modeling stages of quantitative research.

Decomposing Software Before Rebuilding It

When modernizing analytical tools, it is easy to assume that adopting a new technology stack requires discarding everything that came before it. If we move to Parquet and vectorized query engines, we might assume we must interact with our data exclusively through DataFrame APIs, SQL consoles, or Jupyter notebooks.

TabDat-Explore suggests a different approach. Software interfaces and computational substrates age at different rates. An older statistical environment can contain interaction patterns that remain remarkably effective, even when its storage formats and execution engines no longer fit contemporary workflows.

By decomposing software into its interaction model, statistical semantics, execution engine, storage format, and integration contracts, we can make deliberate choices about what to keep and what to rebuild. We can preserve the concise analytical verbs and session ergonomics that made older tools productive, while anchoring the execution substrate firmly in the open, vectorized, and reproducible data ecosystem of today.

References

1. S. Park, Why I Design CLI-First Software (2026).
2. StataCorp, Stata Statistical Software: Release 18 (2023), College Station, TX: StataCorp LLC.
3. M. Raasveldt and H. Mühleisen, DuckDB: an Embeddable Analytical Database (2019), Proceedings of the 2019 ACM SIGMOD International Conference on Management of Data.
4. Apache Arrow & Parquet Projects, Apache Parquet Format Documentation (2024).
5. S. Seabold and J. Perktold, Statsmodels: Econometric and Statistical Modeling with Python (2010), Proceedings of the 9th Python in Science Conference.
6. S. Saul, VisiData: A Terminal Interface for Exploring and Arranging Data (2024).
7. J. Kerl, Miller: Like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON (2024).