Skip to content

6 min read Cloud Written for CloudQuery

How We Accidentally Invented a Cloud Search Language

Visitors looking at a cast of the Rosetta Stone

Blog How we turned our search bar into a domain-specific language using Peggy instead of regex, solving complex parsing challenges for cloud resource queries.

TL;DR A "simple" search bar turned into a full domain-specific language (DSL) for cloud queries. Regex melted. Peggy rescued us. Along the way we learned more about Chomsky, compilers, and a few surprises about UX.

On this page
  1. Regex experiment - unreadable and fragile
  2. Peggy
  3. Chomsky and Compilers
  4. Lessons Learned
  5. Final Thoughts

Co-authored with Christopher Duflo at CloudQuery.

All we wanted was a box where you could type something like:

(type:ec2 AND region:us-west-2) OR "production database"

…and instantly see matching assets across AWS, Azure, and GCP. Easy, right?

Turning that vision into reality meant building more than a search bar. It meant inventing a mini language - one that was intuitive for users to write, easy to validate, and flexible enough to support full-text search (FTS) and logical expressions. We thought we could hack it together with some regular expressions.

We were wrong.

Users see a search bar. Underneath, it’s a small programming language.

We wanted FTS + structured queries for our asset inventory, but what looked simple quickly became a deep parsing challenge.

Regex experiment - unreadable and fragile

At first, it worked fine for simple filters like: region:us-east-1

However, this quickly exploded the complexity:

  • How do we validate nested parentheses?
  • What happens when a user forgets a closing )?
  • How do we differentiate “quoted strings” from field pairs?
  • What about AND, OR, and NOT precedence?
  • Nested logic: (type:ec2 AND region:us-west-2)?

Plus, it was nearly impossible for human beings to read. Every tweak to support a new requirement - nested groups, quoted strings, AND/OR precedence - felt like defusing a bomb. One change would fix one edge case and break five others.

Regex excels at straight-line patterns. It shines when you want to match an email or a simple field:value pair. But once you ask it to balance tokens or obey operator precedence, things crumble. We ran into three core headaches:

  • Nesting: Regex can’t reliably match balanced parentheses without insane hacks. Once you try, maintenance becomes a nightmare.
  • Logical operators: Handling AND, OR, NOT in the right order forced us into nested lookarounds that barely worked.
  • Opaque errors: A malformed input like (type:ec2 AND would trigger a generic “no match” response. Zero guidance on what to fix.

Peggy

That’s when we discovered Peggy, a parser generator that lets you define a grammar using Parsing Expression Grammars (PEGs) and compiles it into a working parser.

Peggy clicked immediately.

Instead of juggling brittle regex patterns, we wrote formal grammar rules like:

Expression = head:Term tail:(_ ("AND" / "OR") _ Term)* { ... }

Want proof? This is the AST for (type:ec2 OR type:rds) AND region=us-west-2:

{
	"kind": "and",
	"left": { "kind": "or", "...": "..." },
	"right": { "kind": "comparison", "field": "region", "value": "us-west-2" }
}

It was easier to write and easier to read. Peggy gave us:

  • A structured grammar we could reason about
  • Syntax error feedback for invalid expressions
  • An Abstract Syntax Tree (AST) to programmatically analyze and transform user input

Here’s a fundamental part of our grammar - how we handle comparison expressions like region="us-west-2" or instance_count>5:

ComparisonExpression = identifier:Identifier _ op:ComparisonOperator _ value:Value? finalSpace:_ {
    const isListValue = value && (value.kind === 'list-expression');
    const isQuotedValue = value && (value.kind === 'quoted-string');
    const hasTrailingSpace = finalSpace && finalSpace.length > 0;

    let isValueComplete = false;

    if (isListValue) {
        isValueComplete = value.complete;
    } else if (isQuotedValue) {
        isValueComplete = value.complete;
    } else {
        isValueComplete = (!!value && hasTrailingSpace);
    }

    return {
        kind: 'comparison-expression',
        identifier,
        identifierLocation: identifier.location,
        op: op.value,
        opLocation: op.location,
        value: value === null ? undefined : value,
        valueLocation: value ? value.location : undefined,
        position: location(),
        text: text(),
        complete: isValueComplete,
        context: 'value'
    }
}

This rule matches expressions like region="us-west-2" and builds a structured object with the field, operator, and value. It also tracks position information for syntax highlighting and validation, as well as completeness for conversion to a chip in the filter bar.

INPUT STRING you type a query LEXER splits it into tokens PEG ENGINE Peggy applies the grammar AST a tree of meaning PLANNER works out how to run it QUERY PLAN often SQL
INPUT STRING you type a query LEXER splits it into tokens PEG ENGINE Peggy applies the grammar AST a tree of meaning PLANNER works out how to run it QUERY PLAN often SQL
  1. Input String - You type your query
  2. Lexer - Breaks the string into tokens - identifiers, operators, parentheses, quoted strings
  3. PEG Engine - Peggy applies grammar rules, recognizes structures, enforces operator precedence
  4. AST (Abstract Syntax Tree) - Tree representation where each node is a chunk of meaning
  5. Planner - Walks the AST and figures out how to execute it efficiently
  6. Query Plan - Concrete plan (often SQL) that the engine will run

Chomsky and Compilers

By this point, it was clear: we weren’t just building a search bar. We were designing a domain-specific language (DSL) for querying cloud infrastructure.

It turns out our journey from regex to formal grammar has deep theoretical roots. In the 1950s, linguist Noam Chomsky developed a hierarchy of formal grammars that still shapes how we think about parsing and language processing today.

LevelNicknameWhat it can describeExamples
Type 3RegularSingle-layer patterns, no nestingRegex, log filters
Type 2Context-freeBalanced or nested structuresSQL, most programming languages, our PEG.js parser
Type 1Context-sensitivePatterns that depend on nearby contextSome natural-language processors
Type 0UnrestrictedAll computable structuresFull Turing-complete languages

We jumped from Type 3 (regex) to Type 2 (PEG.js) so we could support nested parentheses, operator precedence, and clean error handling without hand-written hacks.

TYPE 0: UNRESTRICTED TYPE 1: CONTEXT-SENSITIVE TYPE 2: CONTEXT-FREE TYPE 3: REGULAR regex PEG.js From regex to PEG.js: one level up, so queries can nest parentheses and respect precedence.
TYPE 0: UNRESTRICTED TYPE 1: CONTEXT-SENSITIVE TYPE 2: CONTEXT-FREE TYPE 3: REGULAR regex PEG.js From regex to PEG.js: one level up, so queries can nest parentheses and respect operator precedence.

Our first approach with regex operated at Type 3 (Regular Grammar) level, which worked fine for simple field:value matching. But as soon as our users needed nested parentheses and logical operators with precedence rules, we hit a wall.

Lessons Learned

What surprised us most was that a problem that started as “let’s build a better search bar” led us straight into the intersection of computer science, linguistics, and cognitive science.

This isn’t a coincidence. The fundamental challenge of cloud infrastructure - managing complexity through abstraction - mirrors the challenges of human language itself. Both require systems that can express complex ideas through the composition of simpler elements.

Final Thoughts

What started as a UI improvement turned into a system design problem. Along the way, we ended up relearning a lot of language theory.

  1. Regex is not a parser. Don’t try to write your own unless you’re doing something trivially simple.
  2. Use a parser generator. Tools like Peggy make it easier to define a grammar and maintain it.
  3. Design like a language. If your UI input is complex, it’s probably a DSL - treat it as such.
  4. Humanities are not optional. Understanding syntax, ambiguity, and user language is key to building better tools.

Our implementation handles various expression types:

  • Simple field queries: region=us-west-2
  • Text searches: "production database"
  • Logical operators: ec2 AND production
  • Parenthesized groups: (type:rds OR type:aurora) AND environment:prod
  • List operations: region IN (us-west-1, us-west-2, us-east-1)
  • Negation: NOT type:s3

The next time you’re tempted to reach for regex to parse complex input, remember: you might actually be building a language. Give parser generators like Peggy a look - your future self will thank you.

Works on My Machine

Get the next post by email. My newsletter covers homelab builds, AI tools and open source, and it goes out when I have something to say.

It's free, and you can unsubscribe anytime. You'll get an email to confirm first.

If you liked this post, consider sharing it on Hacker News, Bluesky, LinkedIn, or X.

Joe Karlsson at his desk, chin on hand, with a microphone in front of him

Joe
Karlsson

I write about my homelab, AI tools, databases, and the weird stuff I build on the side.

More about me →

BlueskyLinkedInGitHub