Because it's not "operation mode", it's "mode of operation". "Of" something makes it genitive case, and for 2nd (?) Declension singular, the ending is "i".

The nominative plural has the same ending, but if "operandi" were an adjective modifying a plural noun, "modus" would also be nominative plural, or "modi", and the phrase would be "modi operandi". But "operandus" isn't an adjective.

So we have a nominative singular "mode" and a gentive singular "operation", and it's "mode of operation".