Letting an LLM write sandboxed Pandas code: Zero-hallucination tabular Q&A in csv-qa-agent

1. The Tabular LLM Hallucination Problem

When users paste raw CSV rows into an LLM prompt and ask for grouped aggregations, medians, or standard deviations, three failure modes occur: arithmetic guessing, context-window exhaustion, and sensitive data leakage.

Autoregressive token prediction is not an arithmetic ALU—floating-point aggregations over hundreds of rows drift when predicted as text tokens. In csv-qa-agent (built for the Rooman AI Challenge), the LLM never sees raw dataset rows and never computes numbers in natural language.

2. Zero-Row Schema Extraction for Privacy & Token Efficiency

Instead of serializing rows, csv-qa-agent inspects the in-memory Pandas DataFrame and extracts only column names and dtypes. This compact schema descriptor is sent to Groq llama-3.3-70b-versatile with instructions to synthesize a deterministic Pandas expression assigned to result.

import pandas as pd

def build_schema_prompt_context(df: pd.DataFrame) -> str:
    schema_lines = [
        f"- {col} (dtype: {dtype})"
        for col, dtype in zip(df.columns, df.dtypes)
    ]
    return (
        "DataFrame `df` columns and types (no raw rows transmitted):\n"
        + "\n".join(schema_lines)
    )

3. Static Verification with Python's ast.NodeVisitor

Executing LLM-generated Python code directly with exec() is a critical security risk if a prompt injection attempts import os or open('/etc/passwd'). Before any generated snippet runs, csv-qa-agent parses the code into an Abstract Syntax Tree (ast.parse) and walks every node using a custom security guard.

Any import of os, sys, subprocess, socket, requests, or call to eval, exec, open, __import__, or dunder attributes (__class__, __subclasses__) is rejected statically before execution.

import ast

FORBIDDEN_MODULES = {"os", "sys", "subprocess", "socket", "shutil", "pathlib", "requests", "urllib"}
FORBIDDEN_BUILTINS = {"eval", "exec", "open", "compile", "__import__", "globals", "locals", "getattr"}

class SandboxSecurityVisitor(ast.NodeVisitor):
    def visit_Import(self, node: ast.Import) -> None:
        for alias in node.names:
            root_mod = alias.name.split(".")[0]
            if root_mod in FORBIDDEN_MODULES:
                raise PermissionError(f"Blocked unsafe import: {root_mod}")
        self.generic_visit(node)

    def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
        if node.module and node.module.split(".")[0] in FORBIDDEN_MODULES:
            raise PermissionError(f"Blocked unsafe from-import: {node.module}")
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call) -> None:
        if isinstance(node.func, ast.Name) and node.func.id in FORBIDDEN_BUILTINS:
            raise PermissionError(f"Blocked unsafe builtin call: {node.func.id}()")
        self.generic_visit(node)

    def visit_Attribute(self, node: ast.Attribute) -> None:
        if node.attr.startswith("__"):
            raise PermissionError(f"Blocked dunder attribute access: {node.attr}")
        self.generic_visit(node)

4. Restricted Namespace Execution & 66 Pytest Tests

Only after SandboxSecurityVisitor passes with zero violations does csv-qa-agent evaluate the expression inside a restricted dictionary scope with __builtins__ stripped down to safe primitives.

The entire pipeline—including adversarial prompt-injection attempts, dunder attribute escapes, and complex groupby aggregations—is verified by 66 automated pytest unit and security tests.