The heart of the problem is simply the limited timing and computational resources in a multitasking non-realtime OS.
The smaller the audio buffers are, the more prone they'll be to starvation. Only if the application can be guaranteed to receive interrupt service and/or thread timeslices at a 100 Hz rate or better is it possible to achieve audio latency of 10 milliseconds. That's a difficult thing to guarantee in a modern consumer OS. It can be done, but it won't happen by accident, only by design.
The penalty for failing to service your 10-ms buffer is a dropout that sounds much worse than slightly higher latency, so there's an incentive to use larger buffers than necessary at every link in the signal chain. From the point of view of the OS vendor, musicians might complain about latency, but everyone will complain about dropouts.
My guess is that the large buffers are covering up for scheduling issues. On my devices (Moto X, Nexus 7) I can regularly get Android audio to stutter by connecting the power or switching apps. (And it'll happen on its own sometimes when it picks, as it so often does, a terrible time to update apps.)
The easy fix to that is to make the buffers bigger. Doing that, though, increases latency. The hard fix is to do what Apple has done and build for audio from the ground up.
Android's audio stack is "pushing" audio down to the audio driver, with the audio threads sleeping between every push. In other words, the audio driver's interrupts are not scheduling the actions.
Having smaller period sizes increases the risk of not "guessing" the sleep scheduling right, resulting in many audio dropouts.
Using a "pull" method is required for low latency audio, where the audio driver's interrupts are scheduling when audio is passed to and pulled from applications.
Another downside of halving the period size is CPU load/battery drain. There are simply too many layers in Android's audio stack, and it has quite a few unoptimised code as well (such as converting between audio sample formats with plain C code).
Given that each sample means a kernel mode-user mode trip and various context switches, I'm going to guess that halving the sample size globally would make your phone's audio handling consume more CPU.
