Apparently it doesn't affect the results significantly.
More info:
https://github.com/huggingface/transformers/pull/17901