Syntax Highlighting

Thursday, October 28, 2010

Update: I modified the code examples below to use the new colors.hex vocabulary.

It’s a sometimes overlooked feature that Factor contains a syntax highlighting vocabulary that supports a decent number of programming languages. The vocabulary is called xmode, and is used to highlight entries submitted to Factor’s pastebin.

I thought it would be fun to use xmode to implement syntax highlighting in the listener. When we’re done, it will look something like this:

First, we will need to use several vocabularies:

USING: accessors assocs colors io io.encodings.utf8 io.files
io.styles kernel literals locals math math.parser sequences
xmode.catalog xmode.marker ;

We define named styles that will apply to the tokens that are parsed by the syntax highlighter. For convenience, we will use the colors vocabulary to convert hexadecimal values to color tuples to build the stylesheet. In the stylesheet below, we use the same colors that are used in the pastebin.

CONSTANT: STYLES H{
    { "NULL"     H{ { foreground COLOR: #000000 } } }
    { "COMMENT1" H{ { foreground COLOR: #cc0000 } } }
    { "COMMENT2" H{ { foreground COLOR: #ff8400 } } }
    { "COMMENT3" H{ { foreground COLOR: #6600cc } } }
    { "COMMENT4" H{ { foreground COLOR: #cc6600 } } }
    { "DIGIT"    H{ { foreground COLOR: #ff0000 } } }
    { "FUNCTION" H{ { foreground COLOR: #9966ff } } }
    { "INVALID"  H{ { background COLOR: #ffffcc }
                    { foreground COLOR: #ff0066 } } }
    { "KEYWORD1" H{ { foreground COLOR: #006699 }
                    { font-style bold } } }
    { "KEYWORD2" H{ { foreground COLOR: #009966 }
                    { font-style bold } } }
    { "KEYWORD3" H{ { foreground COLOR: #0099ff }
                    { font-style bold } } }
    { "KEYWORD4" H{ { foreground COLOR: #66ccff }
                    { font-style bold } } }
    { "LABEL"    H{ { foreground COLOR: #02b902 } } }
    { "LITERAL1" H{ { foreground COLOR: #ff00cc } } }
    { "LITERAL2" H{ { foreground COLOR: #cc00cc } } }
    { "LITERAL3" H{ { foreground COLOR: #9900cc } } }
    { "LITERAL4" H{ { foreground COLOR: #6600cc } } }
    { "MARKUP"   H{ { foreground COLOR: #0000ff } } }
    { "OPERATOR" H{ { foreground COLOR: #000000 }
                    { font-style bold } } }
}

The xmode.catalog vocabulary provides support for looking up the type (or “mode”) of the file. The xmode.marker vocabulary then provides support for converting each line into a stream of tokens. Each token allows access to a named style. Once we have the name of the appropriate style, we can then look it up and format the output.

Putting that all together, we can implement the highlight. word.

: highlight-tokens ( tokens -- )
    [
        [ str>> ] [ id>> ] bi
        [ name>> STYLES at ] [ f ] if* format
    ] each nl ;

: highlight-lines ( lines mode -- )
    [ f ] 2dip load-mode [
        tokenize-line highlight-tokens
    ] curry each drop ;

:: highlight. ( path -- )
    path utf8 file-lines [
        path over first find-mode highlight-lines
    ] unless-empty ;

You should be able to paste the above code examples into your listener, to try it for yourself.