Adventures in Vim: Lee and Jim figure out how to change comment colors

One fine Monday morning, Ars Technica Senior Technology Editor Lee Hutchinson came to me with a problem: the colors in his text editor, in his humble opinion, had Begun To Suck.

In Lee's 20 years or so of Vim usage, he'd gotten accustomed to comment lines in his code and configuration files being rendered in dark blue. But after upgrading a machine to Ubuntu 20.04, Vim started rendering comments in cyan—and since the "Identifier" syntax category also rendered in cyan, he was unhappy enough about it to decide to change the defaults.

At first blush, Vim seems to adhere to roughly the same configuration standard that many if not most Unix-like systems and applications do—there's a set of systemwide configurations in /etc, which can be overridden individually per user by changes made in an optional configuration file in that user's home directory. In Vim's case, that's ~/.vimrc—just like Bash configurations can be overridden in ~/.bashrc.

But when Lee tried to make his One Simple Change to Vim's syntax highlighting—turn comments from the new cyan back into the dark blue, which he preferred—things got interesting.

The Hutchinson way to configure comment highlighting

After a little googling, the command Lee found to change comment color seemed to be pretty simple: highlight comment ctermcfg=19, where 19 is the color code Vim uses for dark blue. The problem is, making the change in ~/.vimrc didn't actually work.

To be more specific, it did work—briefly—but almost immediately after opening the file, the comments changed from dark blue back to cyan again. On a local, fast machine, the change happened too quickly to notice; but Lee was ssh'ing into a remote machine, and that gave just enough delay to see his color preference applied initially but quickly reverted.

After significant googling, Lee discovered an ugly workaround. There's a very old joke that Vim isn't actually a text editor at all—it's an operating system in its own right, which simply masquerades as a text editor. Like most good jokes, this one's a bit over the top but has a kernel of truth to it—Vim config files don't simply assign values to configuration variables; they can actually run code in their own right.

function DelayedSetVariables(timer)
    highlight comment ctermfg=19
endfunction

let timer=timer_start(16,'DelayedSetVariables')

Sure enough, the ugly hack worked: now, instead of seeing dark blue comments initially that then flashed back to the hated cyan, Hutchinson saw cyan comments that then flashed to his preferred dark blue.

This worked well enough for his purposes... but what's the point of being a senior technology editor if you can't run a problem past a technology reporter who reports to you?

The wrong way... actually, several wrong ways

When Lee brought his kinda-solved problem to me, it certainly sounded like a bug—I might not be a Vim user myself, but with more than 20 years of Unix-like OS experience under my own belt, I also expected a user-profile configuration file to cleanly overwrite a system-wide configuration. The unhinged ranting coherent, focused problem report Lee offered me included a warning: there were, in his words, "about 20 different places where Vim configuration changes get applied," so tracking down the problem was unusually sticky.

I'm not a Vim user myself—I'm one of those heathens who never saw any particular reason to learn more about Vim than the :q! needed to get the hell out of it—but my immediate suspicion was that a bug was causing Vim configuration files to be applied out of order. So I googled how to check what configurations had been applied to a running Vim instance: turns out there's a special command :scriptnames that will provide you with exactly that.

  1: /usr/share/vim/vimrc
  2: /usr/share/vim/vim81/debian.vim
  3: /usr/share/vim/vim81/syntax/syntax.vim
  4: /usr/share/vim/vim81/syntax/synload.vim
  5: /usr/share/vim/vim81/syntax/syncolor.vim
  6: /usr/share/vim/vim81/filetype.vim
  7: ~/.vimrc
  8: /usr/share/vim/vim81/plugin/getscriptPlugin.vim
  9: /usr/share/vim/vim81/plugin/gzip.vim
 10: /usr/share/vim/vim81/plugin/logiPat.vim
 11: /usr/share/vim/vim81/plugin/manpager.vim
 12: /usr/share/vim/vim81/plugin/matchparen.vim
 13: /usr/share/vim/vim81/plugin/netrwPlugin.vim
 14: /usr/share/vim/vim81/plugin/rrhelper.vim
 15: /usr/share/vim/vim81/plugin/spellfile.vim
 16: /usr/share/vim/vim81/plugin/tarPlugin.vim
 17: /usr/share/vim/vim81/plugin/tohtml.vim
 18: /usr/share/vim/vim81/plugin/vimballPlugin.vim
 19: /usr/share/vim/vim81/plugin/zipPlugin.vim
 20: /usr/share/vim/vim81/scripts.vim
 21: /usr/share/vim/vim81/syntax/perl.vim
 22: /usr/share/vim/vim81/syntax/pod.vim
Press ENTER or type command to continue

Lee hadn't been kidding about the vast array of configuration files to look through: my system loaded 22 separate configuration files, 15 of which took effect after the .vimrc in my home directory! Thus began the start of a long, winding, and ultimately fruitless primrose path: I wanted to find instances of the comment color being changed somewhere after my ~/.vimrc, and it turned out that just wasn't happening.

The only place I could find where comment color was set to Cyan was in /usr/share/vim/vim81/syncolor.vim, a couple of spaces ahead of my personal .vimrc. In theory, the change in ~/.vimrc should have overridden the one in syncolor.vim—but in practice, without Lee's ugly timer hack, the only way I could find to change the comment color was within syncolor.vim itself.

" Many terminals can only use six different colors (plus black and white).
" Therefore the number of colors used is kept low. It doesn't look nice with
" too many colors anyway.
" Careful with "cterm=bold", it changes the color to bright for some terminals.
" There are two sets of defaults: for a dark and a light background.
if &background == "dark"
  SynColor Comment      term=bold cterm=NONE ctermfg=Cyan ctermbg=NONE gui=NONE guifg=#80a0ff guibg=NONE

Changing ctermfg=Cyan inside syncolor.vim to ctermfg=19—or, better yet, ctermfg=DarkBlue, which produced an easier-to-read shade of blue—worked as expected, and it produced the output Lee wanted without the god-awful timer hack. But it applied the change systemwide, not just to Lee's own user account—and more importantly, it didn't explain how or why the original change in ~/.vimrc refused to work as expected.

I still smelled an out-of-order bug, so I dug further.

" Vim syntax support file
" Maintainer:   Bram Moolenaar 
" Last Change:  2001 Sep 12

" This file sets up the default methods for highlighting.
" It is loaded from "synload.vim" and from Vim for ":syntax reset".
" Also used from init_highlight().

According to the comments at the top of syncolor.vim, the changes within that file were applied in three cases—when synload.vim is parsed during Vim initialization, when the user issues the command :syntax reset, and within the Vim function init_highlight(). I knew neither Lee nor I was calling for :syntax reset, so I proceeded to find the invocation of syncolor.vim from within synload.vim.

" Set the default highlighting colors.  Use a color scheme if specified.
if exists("colors_name")
  exe "colors " . colors_name
else
  runtime! syntax/syncolor.vim
endif

If I put the simple highlight comment ctermfg=19 back into my ~/.vimrc, and commented out the runtime! syntax/syncolor.vim in synload.vim, I believed everything should work properly: this would still qualify as an ugly hack, of course, but it would narrow down where the problem behavior was coming from and allow me to write a more exact bug report to file with the Vim project.

Unfortunately, it didn't work that way: even with runtime! syntax/syncolor.vim commented out, the Cyan comments that file specified overrode the simple setting in my ~/.vimrc. This meant the configurations there were being called by Vim's init_highlight() function after it parsed ~/.vimrc.

On the one hand, this certainly still smelled like a bug to me: I couldn't override a simple configuration setting from my user-level rc file. On the other hand, did I mention the 20+ years of open source experience? I needed to make certain I wasn't missing something obvious that would cause a bug report to just get rejected with a #WONTFIX because I'd missed some deliberate Vim idiosyncrasy.

Finding the right way

Since Vim's configuration files had self-documenting comments, the time had come to read them more thoroughly. I'd already learned that the contents of syncolor.vim were applied by init_highlight() and synload.vim—but I needed to dig further.

I couldn't get any further with the documentation comments at the top of synload.vim or syncolor.vim, but the next clue came from the code in syncolor.vim itself:

if syntax_cmd == "enable"
    " ":syntax enable" keeps any existing colors
    command -nargs=* SynColor hi def 
    command -nargs=* SynLink hi def link 
  elseif syntax_cmd == "reset"
    " ":syntax reset" resets all colors to the default
    command -nargs=* SynColor hi 
    command -nargs=* SynLink hi! link 
  else
    " User defined syncolor file has already set the colors.
    finish
  endif

Clearly, there was some proper way to set user-defined colors, since this if block specifically avoided setting them up if a "user defined syncolor file" already had. So the next step was to Google "vim user defined syncolor file." The top search result was the source for syncolor.vim itself on Github, but the second result brought me to Vim documentation at SourceForge.

Performing a ctrl-F syncolor in-browser search on this 5,128-line document eventually got me to the information I needed, about 90 percent of the way down the page:

If you want to use different colors for syntax highlighting, you can add a Vim
script file to set these colors.  Put this file in a directory in
'runtimepath' which comes after $VIMRUNTIME, so that your settings overrule
the default colors.  This way these colors will be used after the ":syntax
reset" command.

For Unix you can use the file ~/.vim/after/syntax/syncolor.vim.

Finally, I'd found the right answer to the deceptively simple question "How do I change comment color within Vim?": after creating ~/.vim, ~/.vim/after, and ~/.vim/after/syntax, you can finally create the file ~/.vim/after/syntax/syncolor.vim—and changes made to syntax highlight colors there applied the way that Lee and I expected them to.

Petting the shaggy dog

Hopefully, you've learned something along the way as you read this god-awful shaggy dog story of configuring a Linux application. Maybe you, too, just wanted to change some colors in a text editor—in which case I've led you down an absurdly long path just to get to a relatively short answer.

But more importantly, I hope the exercise in full can serve as a broader exercise in troubleshooting. Happy Linux-ing!