I did some tests and decided to go with a mix of 50% grout and 50% N scale ballast. To use only grout, which is much like a fine powder, was more or less impossible. It gave you nearly no control, and the stuff ended up everywhere. A mix of ballast and grout was easier to control, but still gave a dirt look. Here is a picture of a spur ballasted withe the grout and ballast mix (the middle track). The mix is kept in place with a water and white glue mix.
Once the glue had dried I applied weathering powders to obtain a dirt look. I also used some rust powder along the rails, and some black powder in between the rails, simulating oil and whatnot. I gave the mainline a light rust and oil treatment as well.
And a close up...
The dirt is a little to brown maybe. I'll aim for a slightly more gray tone when I do the next stretch of track.