Re: Factor

Factor: the language, the theory, and the practice.

Shlex

Tuesday, September 29, 2026

#parsing

Something I’ve always respected about Python is their infamous batteries included philosophy:

The Python source distribution has long maintained the philosophy of “batteries included” – having a rich and versatile standard library which is immediately available, without making the user download separate packages. This gives the Python language a head start in many projects.

We have a similar approach in Factor, which leads us to our extensive vocabulary index. Sometimes we see new things that we could add. One recent example is Python’s shlex module for splitting a command into arguments in a shell-like manner, keeping quoted strings together.

I’m happy to say that we now have a shlex vocabulary in the main repository.

We can use it to read a command with a quoted argument:

USING: io prettyprint shlex ;

IN: scratchpad "echo -n 'hello world'" parse-shlex .
{ "echo" "-n" "hello world" }

This is also useful for options containing spaces or an empty value:

IN: scratchpad "--title='My Project' --label ''" parse-shlex .
{ "--title=My Project" "--label" "" }

For command-like lines in a configuration file, we might want to allow comments. The two flags to shlex-split enable comments and POSIX mode (although perhaps instead of word arguments, maybe these should be dynamic variables…):

IN: scratchpad "copy 'annual report.txt' archive # keep a backup" t t shlex-split .
{ "copy" "annual report.txt" "archive" }

IN: scratchpad "echo '#hello' # a comment" t t shlex-split .
{ "echo" "#hello" }

Going in the other direction, we can quote a filename as one argument for a POSIX shell:

IN: scratchpad "annual report.txt" shlex-quote print
'annual report.txt'

Or turn a whole sequence of arguments into a command string, ready to copy into a terminal:

IN: scratchpad { "cp" "annual report.txt" "backup copy.txt" } shlex-join print
cp 'annual report.txt' 'backup copy.txt'

Joining and splitting preserves the original arguments, including empty strings:

IN: scratchpad { "echo" "" "hello world" } shlex-join .
"echo '' 'hello world'"

IN: scratchpad { "echo" "" "hello world" } shlex-join parse-shlex .
{ "echo" "" "hello world" }

These words work with strings; they do not execute commands or expand variables and wildcards. The quoting is intended for POSIX-style shells.

This is now available in the development version of Factor!