In my experience, that's not the case. In the case of English, the main difficulty was (is) learning the pronunciation of every word, because the English spelling system isn't very helpful (to put it kindly). As you advance, the problem is then to get a good grasp on usage/pragmatics; that is, there are tons of syntactically correct ways of saying the same thing, but only a few of them sound "natural". Grammar was never a big problem for me.
In the case of Japanese too, the problem is not the grammar. Japanese morphology is simple (e.g. nouns are the same in singular and plural, verb conjugation doesn't vary with person) and very regular. The big problems are 1) acquiring the vocabulary in the first place, which is hard because the writing system is even less helpful than English's 2) and remembering all those words, which is hard because for the most part they don't resemble either my native language or English.
Oh yes, English is terrible at denoting the pronunciation with its script. Not so for French (or from what I hear, German). Most of the time you are able to predict the pronunciation based on a set of characters. Words in Indian languages which use the Devanagari script (or scripts derived from it) can be directly mapped to their pronunciations. There are no exceptions.
In the case of Japanese too, the problem is not the grammar. Japanese morphology is simple (e.g. nouns are the same in singular and plural, verb conjugation doesn't vary with person) and very regular. The big problems are 1) acquiring the vocabulary in the first place, which is hard because the writing system is even less helpful than English's 2) and remembering all those words, which is hard because for the most part they don't resemble either my native language or English.