I have usually resisted KPIs but I have them now like "Got mistaken for an animal (ex. hunters and dogs)", "Heart rate (low) for adjusted gaits", "People laughed", etc. I pass out more business cards now in a week than I used to do in three months and from a KPI perspective I'm doing about 50% of what I could.
It's pretty easy. SBERT + classical classifiers from scikit-learn, don't forget the probability calibration. I get diversity by clustering on k-Means and taking the best N/k from k=20 clusters and I also blend in about 30% random items to keep the system honest.
It's on my agenda to make a general-purpose text classifier with a "better" model (better sensitivity to word order) but I don't think a better AUC-ROC would really make a difference in my case and a recommender model can't be that accurate anyway because I'm fickle and my judgements depend on how I'm feeling and how many articles about the same subject I've seen lately.
Fact is that I should change the status of that because even though I use it everyday I've only patched it twice in the last year. It spins like a top.
Whatever you do don't screw around with fine-tuned BERT. With noisy judgements you won't really get better accuracy than BERT+SVM and there's something to say for a fast model trainer that makes a good model 100% of the time without manual intervention. I haven't seen a training recipe I can believe in for that kind of model and "catastrophic forgetting" seems to eat you alive if you have 5000+ samples. For a general classifier I am thinking of selection between
