Should we use parentheses or brackets around functions? Is there a widely accepted rule/style?
For example, which of the following is the preferred way of writing formulas?
$(f(x) + g(x)) * h(x)$
vs
$[f(x) + g(x)] * h(x)$
$\endgroup$ 111 Answer
$\begingroup$From Mathematical Writing, by (the!) Donald Knuth, (the!) Tracy Larrabee, and (the!) Paul M. Roberts (1987):
In some cases your audience may expect nested parentheses. [... S]hould the outer pair be changed to brackets (or curly-braces)? This was once the prevailing convention, but it is now not only obsolete but potentially dangerous; brackets and curly braces have semantic content for many scientific professionals. [...] Typographers help by using slightly larger parentheses for the outer pair in a nested set.
This passage occurs as part of a discussion about parenthetical remarks in text, but I believe the sentiment applies (and is intended to apply) to displayed formulas.
As the creator of $\TeX$, Knuth may have been a little biased about what counted as "obsolete" typographic practice. Nevertheless, the opinion seems fairly widespread these days, and I suspect that this is the thinking behind Wikipedia's no-brackets guideline.
A not-unrelated passage from later in the book (bear in mind that the anecdotes date back more than three decades) ...
Don [Knuth] showed us [Larrabee and Roberts, as note-takers in Knuth's Mathematical Writing class] two examples where mathematical journals had trouble presenting programs, algorithms, or concrete mathematics in papers he wrote. [...] Don had to convince the typesetters at Acta Arithmetica to create "floor" and "ceiling" functions by carving off small pieces of the metal type for square brackets. [The second example had to do with "$:=$".]
[...]
He didn't get his way on everything, though. Brackets were used interchangeably with parentheses in a mathematical formula, despite Don's protest that the former have special meanings.
I'll note that I don't always find that "slightly larger parentheses" are enough to make nested groups legible; I sometimes feather my nests with spaces. I might consider formatting the example in the question as$$\big(\,f(x)+g(x)\,\big)\cdot h(x) \quad\text{rather than}\quad \big(f(x)+g(x)\big)\cdot h(x)$$In this case, though, it doesn't seem to make an appreciable difference.
Incidentally, the Journal of Machine Learning Research makes a PDF of Mathematical Writing available.
$\endgroup$
