This, I just pretty much ended up using the basic LLMChain and do my own custom flow. The built in agents are close to useless for anything but a toy project; it is simply way too unreliable.