You probably have some fun disk caching turned on in the OS or your disk has a built in cache that it’s not properly flushing. I’ve seen this before with cheap SSDs.
Maybe a dumb question, but are you on windows? Check that Defender isn't scanning every single time data is written to your veracrypt drive. When I was on windows, the most painful slowdowns were almost invariably caused by Windows Defender.
