Might be useful to split it up to use the same speech features as some off the shelf vocoder, such as FARGAN used by RADE (https://freedv.org/radio-autoencoder/) and DRED (https://jmvalin.ca/papers/valin_dred_journal.pdf).