Not quite what I am asking. I am looking for system architecture not specific algorithms.Btw, We do use annoy and various other algorithms for our recommendations.
Not really sure what your central question then is: The general theme for building a recommender system architecture is:
1. Decide whether you are okay with a batch approach or an online learning approach or a hybrid.
2. Start simple with a batch approach (similar to what you are doing):
a) Get features ready from your dataset (assuming you have interaction data) : Pre-processing via some big data framework (Map Reduce, Data flow etc)
b) Build a vector space and nearest neighbors datastructures.
c) Stick both into a database optimized for reads
d) Stick a service in front of it and serve.
Once you are happy with 2, you can try out variations involving either online updates to your recommender system which involves changes to the type of database you might want to optimize. etc
I guess I should elaborate a little more on what I am looking for. I already have a hybrid approach working where batch processing informs and improves the ALS models and these models are stored in memory to do some (Near)real time recommendations. The attractive bit about using something like Solr is that the user behavior on the app/front-end is easily modeled in terms or query parameters that could help serve better recommendations and also improve the model. It also seems to be the most commonly used approach based on what I have seen. I am wondering if there are other ways of doing this. In the broader sense, I guess my question is what are the next steps to build on this basic recommendation system? What is a good way to serve recommendations based on user behavior in near-real time and how do these systems take feedback to improve the models.
> In the broader sense, I guess my question is what are the next steps to build on this basic recommendation system? What is a good way to serve recommendations based on user behavior in near-real time and how do these systems take feedback to improve the models.
In the past, I have helped build Lambda architectures where we use a batch model to build a content vector space, build estimates of users in batch, update those in realtime (using PubSub/Kafka) based on user feedback.
Other online mechanisms could be to use Contextual Bandits: e.g. use context in terms of user interactions with the several arms of the bandits being recommendation choices etc. This interaction data can be used to continuously improve your policy. Of course, the key benefit over a Matrix Factorization setup where the interaction matrix is continuously rebuilt over time based on new data, is the in built exploration which minimizes regret.
Would you mind sharing this information? I live in south bay and have struggled to find a good therapist/MD and have had very little success finding any recommendations.
Wow, thanks for all the responses. I think I would like to share more of what troubles me the most. I usually back away when the task seems even slightly complex. For example, I need to try to join 2 collections in mongo and want to try out map-reduce on it and see how long it takes.. I am just postponing it although I know if I stay with it, I will probably get it done in an hour or two. My mind refuses to start anything that seems complex. I get overwhelemed and anxious and delay everything to the last minute and then work in panic mode and do a half-assed job sometimes. Even though I have evidence that when I just start and stay with the problem, I more often than not do it well, I can't seem to internalize this and repeat this. What do you do when you have to do something seemingly complex. I always have trouble breaking it down into smaller tasks unless I have someone to talk to.. then somehow it gets very easy.. I have found that when I work in a group or in pairs, I am way more effective and my speed goes up tremendously. So much so that others often need time to catch up to what I saying. What advice would you give for someone struggling to work on his own?
Try this book: One Small Step Can Change Your Life; basically the author is a cognitive psychologist who studied various business processes and came to the conclusion that the ones that were most effective were not necessarily the best, they just worked because they were put into actual practice ... because they were non-threatening. His research then led him to the following insight: big planning is perceived as a threat by the brain because in the past it meant 1 of 2 things: (1) no more food, gather the group and get ready to move or (2) some other group doesn't like us, there is going to be some armed conflict, gather the group and ... Basically "big picture planning" while necessary if engaged too long activates the "fight or flight" response and you get all agitated and can't concentrate and wind up doing nothing all day. In other words, it's not that you "have trouble breaking things into pieces" it's because you don't break things into pieces that you activate parts of your brain and nervous system that make it impossible for you to calmly focus on the only reality you control - little pieces. So he suggest getting going by sidestepping/bypassing the activation of your "big changes are coming along with danger" systems by sneaking by them with the teeny, tiniest thing you can imagine doing successfully. Don't make it a goal to finish this part of your project or code for 4 hours or whatever. Make it a goal to sit at the computer for five minutes, open up your environment and not waste time with facebook/hacker news/email for 5 minutes. That's it - don't code, don't plan, just open up environment and then walk away after five minutes and get a small treat - a glass of water or something. Then instead of being overwhelmed your brain will be stimulated but not "activated" ... kind of like in the middle of the bell curve - not too aroused ... and you'll think, "Well, that wasn't too bad, I guess I could actually do a little bit more." Come back and just say, "If I could work on a teeny tiny problem, not a whole lot, but something that's been keeping me back and if I finished it and it became a building block I never had to rebuild and could move forward to, what would that thing be - and work on only that for the day. Don't look at other stuff until that's over. If it gets too frustrating, walk away and then ask yourself, can you go at it again for 15 minutes without distractions? Work up to a pomodoro (30 minutes of work of which 5 are a break). In other words, you have to trick your brain and keep it "asleep" so as to not hyper-arouse it too quick. Teeny tiny problems. So small they're laughable. Almost an funny not to get them done. So doable it's almost beneath you. But string together a couple of thousand of those over a few months and you'll be very productive. But don't plan them. Find one, do it, see where that leads you, take the next step, focus only on that until you did it. Make it so small it's not possible to get distracted. So small it's not possible to get overwhelmed. Make your problem being underwhelmed, so underwhelmed you quiet your hyperactive mind down enough to let you actually get work done. There are no complex anythings. Just strings of simples, some strings longer than others, some strings connect to more strings, but everything is an individual bead and you can handle each and every bead. Can you make a cup of tea? Sure! Can you do it while contemplating your place in the universe, and the meaning of life? No, because you're using part of your brain that needs to shut down the tea making part. Just focus on putting water in the pot and lighting the flame. You can do it. Really there is no secret other than this. You may forget to do it, but deep down you know this to be reality. You even say it ("I can't seem to internalize this and repeat it.") You don't have to internalize it. Here's another trick: everybody falls of their paths all the time, then they have to "remember" what works and just repeat it. It's so simple it's boring and that's why we forget - because we get bored and want to forget it's that simple. But you seem to know this. Just be okay with falling off, get back and apply it again. Until you forget again. Then you'll remember you forgot and keep repeating this. You got this. That's for the by yourself thing. As for pairs or groups - stop fighting it and just pair up with someone with similar goals and workstyles.
I remember being in an applied crypto class with you at CMU taught by Virgil Gligor if I am not mistaken and was thoroughly impressed with your knowledge of cryptosystems. I am happy for you! Way to go.
